2024-02-05T06:16:53.392 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-05T06:16:53.392 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_gabrioux@teuthology --name gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi --verbose --archive /home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546690 --description orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} -- /home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546690/orig.config.yaml 2024-02-05T06:16:53.425 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-05T06:16:53.501 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546690 branch: wip-guits-testing-3-2024-02-04-2254 description: orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} email: gabrioux@ibm.com first_in_suite: false job_id: '7546690' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-guits-testing-3-2024-02-04-2254 ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms bind msgr2: false ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) mon_bind_msgr2: false sha1: 164d3be4a527be775f4d3c343ea591122f1b8921 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 164d3be4a527be775f4d3c343ea591122f1b8921 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-guits-testing-3-2024-02-04-2254 sha1: 164d3be4a527be775f4d3c343ea591122f1b8921 owner: scheduled_gabrioux@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 7317 sha1: 164d3be4a527be775f4d3c343ea591122f1b8921 sleep_before_teardown: 0 subset: 1/8 suite: orch:cephadm suite_branch: wip-guits-testing-3-2024-02-04-2254 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_164d3be4a527be775f4d3c343ea591122f1b8921/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 164d3be4a527be775f4d3c343ea591122f1b8921 targets: smithi028.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBCA1FTL+IwOoJywhuWQEDUqUgZopUvLNHlNJXrX6CVXKNNh0+YvrNKCMk86c2yo6jBhtiVkzZ9WkUsbEM3HDBE= smithi167.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMWP3yae/jbJ9gmk1hgW2izKA6ESL/12AEm5YAxTgVzCcWcHh690xyocMv//tt4sJMuVJXmzPpahdEBJiqLXN7c= tasks: - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-02-05_05:57:01 tube: smithi user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-05T06:16:53.502 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_164d3be4a527be775f4d3c343ea591122f1b8921/qa; will attempt to use it 2024-02-05T06:16:53.502 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_164d3be4a527be775f4d3c343ea591122f1b8921/qa/tasks 2024-02-05T06:16:53.502 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-05T06:16:53.504 INFO:teuthology.task.internal:Checking packages... 2024-02-05T06:16:53.528 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '164d3be4a527be775f4d3c343ea591122f1b8921' 2024-02-05T06:16:53.528 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-05T06:16:53.528 INFO:teuthology.packaging:ref: None 2024-02-05T06:16:53.528 INFO:teuthology.packaging:tag: None 2024-02-05T06:16:53.528 INFO:teuthology.packaging:branch: wip-guits-testing-3-2024-02-04-2254 2024-02-05T06:16:53.528 INFO:teuthology.packaging:sha1: 164d3be4a527be775f4d3c343ea591122f1b8921 2024-02-05T06:16:53.529 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-guits-testing-3-2024-02-04-2254 2024-02-05T06:16:53.672 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1241.g164d3be4 2024-02-05T06:16:53.674 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-05T06:16:53.679 INFO:teuthology.task.internal:no buildpackages task found 2024-02-05T06:16:53.680 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-05T06:16:53.686 INFO:teuthology.task.internal:Saving configuration 2024-02-05T06:16:53.699 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-05T06:16:53.707 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-05T06:16:53.733 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546690', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-05 06:11:36.392400', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBCA1FTL+IwOoJywhuWQEDUqUgZopUvLNHlNJXrX6CVXKNNh0+YvrNKCMk86c2yo6jBhtiVkzZ9WkUsbEM3HDBE='} 2024-02-05T06:16:53.758 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi167.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546690', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-05 06:11:36.394302', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMWP3yae/jbJ9gmk1hgW2izKA6ESL/12AEm5YAxTgVzCcWcHh690xyocMv//tt4sJMuVJXmzPpahdEBJiqLXN7c='} 2024-02-05T06:16:53.758 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-05T06:16:53.763 INFO:teuthology.task.internal:roles: ubuntu@smithi028.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-02-05T06:16:53.763 INFO:teuthology.task.internal:roles: ubuntu@smithi167.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-02-05T06:16:53.764 INFO:teuthology.run_tasks:Running task console_log... 2024-02-05T06:16:53.831 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f765af80f70>, signals=[15]) 2024-02-05T06:16:53.832 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-05T06:16:53.839 INFO:teuthology.task.internal:Opening connections... 2024-02-05T06:16:53.839 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi028.front.sepia.ceph.com 2024-02-05T06:16:53.842 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T06:16:53.957 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi167.front.sepia.ceph.com 2024-02-05T06:16:53.958 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T06:16:54.052 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-05T06:16:54.059 DEBUG:teuthology.orchestra.run.smithi028:> uname -m 2024-02-05T06:16:54.076 INFO:teuthology.orchestra.run.smithi028.stdout:x86_64 2024-02-05T06:16:54.077 DEBUG:teuthology.orchestra.run.smithi028:> cat /etc/os-release 2024-02-05T06:16:54.135 INFO:teuthology.orchestra.run.smithi028.stdout:NAME="CentOS Stream" 2024-02-05T06:16:54.135 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION="9" 2024-02-05T06:16:54.136 INFO:teuthology.orchestra.run.smithi028.stdout:ID="centos" 2024-02-05T06:16:54.136 INFO:teuthology.orchestra.run.smithi028.stdout:ID_LIKE="rhel fedora" 2024-02-05T06:16:54.136 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_ID="9" 2024-02-05T06:16:54.136 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM_ID="platform:el9" 2024-02-05T06:16:54.136 INFO:teuthology.orchestra.run.smithi028.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-05T06:16:54.136 INFO:teuthology.orchestra.run.smithi028.stdout:ANSI_COLOR="0;31" 2024-02-05T06:16:54.137 INFO:teuthology.orchestra.run.smithi028.stdout:LOGO="fedora-logo-icon" 2024-02-05T06:16:54.137 INFO:teuthology.orchestra.run.smithi028.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-05T06:16:54.137 INFO:teuthology.orchestra.run.smithi028.stdout:HOME_URL="https://centos.org/" 2024-02-05T06:16:54.137 INFO:teuthology.orchestra.run.smithi028.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-05T06:16:54.137 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-05T06:16:54.137 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-05T06:16:54.138 INFO:teuthology.lock.ops:Updating smithi028.front.sepia.ceph.com on lock server 2024-02-05T06:16:54.163 DEBUG:teuthology.orchestra.run.smithi167:> uname -m 2024-02-05T06:16:54.181 INFO:teuthology.orchestra.run.smithi167.stdout:x86_64 2024-02-05T06:16:54.181 DEBUG:teuthology.orchestra.run.smithi167:> cat /etc/os-release 2024-02-05T06:16:54.238 INFO:teuthology.orchestra.run.smithi167.stdout:NAME="CentOS Stream" 2024-02-05T06:16:54.238 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION="9" 2024-02-05T06:16:54.238 INFO:teuthology.orchestra.run.smithi167.stdout:ID="centos" 2024-02-05T06:16:54.238 INFO:teuthology.orchestra.run.smithi167.stdout:ID_LIKE="rhel fedora" 2024-02-05T06:16:54.238 INFO:teuthology.orchestra.run.smithi167.stdout:VERSION_ID="9" 2024-02-05T06:16:54.238 INFO:teuthology.orchestra.run.smithi167.stdout:PLATFORM_ID="platform:el9" 2024-02-05T06:16:54.238 INFO:teuthology.orchestra.run.smithi167.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-05T06:16:54.238 INFO:teuthology.orchestra.run.smithi167.stdout:ANSI_COLOR="0;31" 2024-02-05T06:16:54.238 INFO:teuthology.orchestra.run.smithi167.stdout:LOGO="fedora-logo-icon" 2024-02-05T06:16:54.238 INFO:teuthology.orchestra.run.smithi167.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-05T06:16:54.238 INFO:teuthology.orchestra.run.smithi167.stdout:HOME_URL="https://centos.org/" 2024-02-05T06:16:54.238 INFO:teuthology.orchestra.run.smithi167.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-05T06:16:54.238 INFO:teuthology.orchestra.run.smithi167.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-05T06:16:54.239 INFO:teuthology.orchestra.run.smithi167.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-05T06:16:54.239 INFO:teuthology.lock.ops:Updating smithi167.front.sepia.ceph.com on lock server 2024-02-05T06:16:54.254 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-05T06:16:54.266 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-05T06:16:54.274 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-05T06:16:54.274 DEBUG:teuthology.orchestra.run.smithi028:> test '!' -e /home/ubuntu/cephtest 2024-02-05T06:16:54.277 DEBUG:teuthology.orchestra.run.smithi167:> test '!' -e /home/ubuntu/cephtest 2024-02-05T06:16:54.294 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-05T06:16:54.300 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-05T06:16:54.300 DEBUG:teuthology.orchestra.run.smithi028:> test -z $(ls -A /var/lib/ceph) 2024-02-05T06:16:54.335 DEBUG:teuthology.orchestra.run.smithi167:> test -z $(ls -A /var/lib/ceph) 2024-02-05T06:16:54.456 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-05T06:16:54.499 INFO:teuthology.run_tasks:Running task kernel... 2024-02-05T06:16:54.518 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-05T06:16:54.518 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-05T06:16:54.518 DEBUG:teuthology.orchestra.run.smithi028:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-05T06:16:54.519 DEBUG:teuthology.orchestra.run.smithi167:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-05T06:16:54.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:16:54.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:16:54.536 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-02-05T06:16:54.537 DEBUG:teuthology.orchestra.run.smithi167:> uname -r 2024-02-05T06:16:54.593 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-361.el9.x86_64 2024-02-05T06:16:54.594 INFO:teuthology.task.kernel:Running kernel on smithi028: 5.14.0-361.el9.x86_64 2024-02-05T06:16:54.594 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum install -y kernel 2024-02-05T06:16:54.595 INFO:teuthology.orchestra.run.smithi167.stdout:5.14.0-361.el9.x86_64 2024-02-05T06:16:54.595 INFO:teuthology.task.kernel:Running kernel on smithi167: 5.14.0-361.el9.x86_64 2024-02-05T06:16:54.595 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum install -y kernel 2024-02-05T06:16:56.733 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS-9 - AppStream 13 MB/s | 6.6 MB 00:00 2024-02-05T06:16:56.843 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS-9 - AppStream 15 MB/s | 6.6 MB 00:00 2024-02-05T06:16:58.192 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.0 MB 00:00 2024-02-05T06:16:58.627 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS-9 - BaseOS 2.9 MB/s | 2.0 MB 00:00 2024-02-05T06:16:59.074 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-02-05T06:17:01.677 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - BaseOS 3.0 MB/s | 8.0 MB 00:02 2024-02-05T06:17:05.636 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - AppStream 6.5 MB/s | 19 MB 00:02 2024-02-05T06:17:09.618 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - CRB 15 MB/s | 5.9 MB 00:00 2024-02-05T06:17:12.038 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 15 kB 00:01 2024-02-05T06:17:31.591 INFO:teuthology.orchestra.run.smithi167.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 20 MB 00:19 2024-02-05T06:17:36.447 INFO:teuthology.orchestra.run.smithi167.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-02-05T06:17:38.629 INFO:teuthology.orchestra.run.smithi167.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-05T06:17:38.630 INFO:teuthology.orchestra.run.smithi167.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-05T06:17:38.713 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-02-05T06:17:38.722 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:17:38.722 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-02-05T06:17:38.722 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:17:38.722 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-02-05T06:17:38.722 INFO:teuthology.orchestra.run.smithi167.stdout: kernel x86_64 5.14.0-416.el9 CentOS-BaseOS 6.0 M 2024-02-05T06:17:38.722 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-02-05T06:17:38.723 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-core x86_64 5.14.0-416.el9 CentOS-BaseOS 21 M 2024-02-05T06:17:38.723 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-modules x86_64 5.14.0-416.el9 CentOS-BaseOS 39 M 2024-02-05T06:17:38.723 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-modules-core x86_64 5.14.0-416.el9 CentOS-BaseOS 33 M 2024-02-05T06:17:38.723 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:17:38.723 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-02-05T06:17:38.723 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:17:38.723 INFO:teuthology.orchestra.run.smithi167.stdout:Install 4 Packages 2024-02-05T06:17:38.723 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:17:38.724 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 99 M 2024-02-05T06:17:38.724 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 122 M 2024-02-05T06:17:38.724 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-02-05T06:17:39.165 INFO:teuthology.orchestra.run.smithi167.stdout:(1/4): kernel-5.14.0-416.el9.x86_64.rpm 14 MB/s | 6.0 MB 00:00 2024-02-05T06:17:39.649 INFO:teuthology.orchestra.run.smithi167.stdout:(2/4): kernel-core-5.14.0-416.el9.x86_64.rpm 23 MB/s | 21 MB 00:00 2024-02-05T06:17:40.591 INFO:teuthology.orchestra.run.smithi167.stdout:(3/4): kernel-modules-5.14.0-416.el9.x86_64.rpm 21 MB/s | 39 MB 00:01 2024-02-05T06:17:40.933 INFO:teuthology.orchestra.run.smithi167.stdout:(4/4): kernel-modules-core-5.14.0-416.el9.x86_6 19 MB/s | 33 MB 00:01 2024-02-05T06:17:40.936 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:17:40.936 INFO:teuthology.orchestra.run.smithi167.stdout:Total 45 MB/s | 99 MB 00:02 2024-02-05T06:17:40.937 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-02-05T06:17:41.532 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-02-05T06:17:41.532 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-02-05T06:17:41.807 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 461 kB/s | 19 MB 00:41 2024-02-05T06:17:42.553 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-02-05T06:17:42.554 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-02-05T06:17:44.960 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-02-05T06:17:45.665 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : kernel-modules-core-5.14.0-416.el9.x86_64 1/4 2024-02-05T06:17:45.793 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T06:17:46.236 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 7.0 MB/s | 5.9 MB 00:00 2024-02-05T06:17:48.060 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 15 kB 00:00 2024-02-05T06:17:48.545 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T06:17:48.721 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T06:17:50.053 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 11 MB/s | 20 MB 00:01 2024-02-05T06:17:53.463 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T06:17:53.609 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : kernel-5.14.0-416.el9.x86_64 4/4 2024-02-05T06:17:55.086 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-02-05T06:17:57.385 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-05T06:17:57.385 INFO:teuthology.orchestra.run.smithi028.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-05T06:17:57.485 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-02-05T06:17:57.496 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:17:57.496 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-02-05T06:17:57.496 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:17:57.496 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-02-05T06:17:57.497 INFO:teuthology.orchestra.run.smithi028.stdout: kernel x86_64 5.14.0-416.el9 CentOS-BaseOS 6.0 M 2024-02-05T06:17:57.497 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-02-05T06:17:57.497 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-core x86_64 5.14.0-416.el9 CentOS-BaseOS 21 M 2024-02-05T06:17:57.497 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules x86_64 5.14.0-416.el9 CentOS-BaseOS 39 M 2024-02-05T06:17:57.497 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-core x86_64 5.14.0-416.el9 CentOS-BaseOS 33 M 2024-02-05T06:17:57.497 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:17:57.497 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-02-05T06:17:57.497 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:17:57.497 INFO:teuthology.orchestra.run.smithi028.stdout:Install 4 Packages 2024-02-05T06:17:57.497 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:17:57.498 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 99 M 2024-02-05T06:17:57.498 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 122 M 2024-02-05T06:17:57.498 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-02-05T06:17:57.971 INFO:teuthology.orchestra.run.smithi028.stdout:(1/4): kernel-5.14.0-416.el9.x86_64.rpm 13 MB/s | 6.0 MB 00:00 2024-02-05T06:17:58.366 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-modules-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T06:17:58.405 INFO:teuthology.orchestra.run.smithi028.stdout:(2/4): kernel-core-5.14.0-416.el9.x86_64.rpm 23 MB/s | 21 MB 00:00 2024-02-05T06:17:58.955 INFO:teuthology.orchestra.run.smithi028.stdout:(3/4): kernel-modules-5.14.0-416.el9.x86_64.rpm 27 MB/s | 39 MB 00:01 2024-02-05T06:17:59.513 INFO:teuthology.orchestra.run.smithi028.stdout:(4/4): kernel-modules-core-5.14.0-416.el9.x86_6 22 MB/s | 33 MB 00:01 2024-02-05T06:17:59.516 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:17:59.516 INFO:teuthology.orchestra.run.smithi028.stdout:Total 49 MB/s | 99 MB 00:02 2024-02-05T06:17:59.517 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-02-05T06:17:59.944 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-02-05T06:17:59.945 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-02-05T06:18:01.021 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-02-05T06:18:01.021 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-02-05T06:18:03.568 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-02-05T06:18:04.264 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-modules-core-5.14.0-416.el9.x86_64 1/4 2024-02-05T06:18:04.406 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T06:18:07.273 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T06:18:07.451 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T06:18:12.507 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T06:18:12.613 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : kernel-5.14.0-416.el9.x86_64 4/4 2024-02-05T06:18:17.692 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T06:18:27.996 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T06:18:27.996 INFO:teuthology.orchestra.run.smithi167.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-05T06:18:27.996 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:18:28.070 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-modules-5.14.0-416.el9.x86_64 4/4 2024-02-05T06:18:28.517 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: kernel-5.14.0-416.el9.x86_64 4/4 2024-02-05T06:18:28.517 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : kernel-5.14.0-416.el9.x86_64 1/4 2024-02-05T06:18:28.517 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T06:18:28.517 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T06:18:28.958 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : kernel-modules-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T06:18:28.958 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:18:28.958 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-02-05T06:18:28.958 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-5.14.0-416.el9.x86_64 2024-02-05T06:18:28.958 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-core-5.14.0-416.el9.x86_64 2024-02-05T06:18:28.958 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-modules-5.14.0-416.el9.x86_64 2024-02-05T06:18:28.959 INFO:teuthology.orchestra.run.smithi167.stdout: kernel-modules-core-5.14.0-416.el9.x86_64 2024-02-05T06:18:28.959 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:18:28.959 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-02-05T06:18:29.468 DEBUG:teuthology.orchestra.run.smithi167:> rpm -q kernel | sort -rV | head -n 1 2024-02-05T06:18:29.617 INFO:teuthology.orchestra.run.smithi167.stdout:kernel-5.14.0-416.el9.x86_64 2024-02-05T06:18:29.617 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-416.el9.x86_64 2024-02-05T06:18:29.617 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-416.el9.x86_64 2024-02-05T06:18:29.618 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-05T06:18:29.618 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-05T06:18:29.618 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-02-05T06:18:29.618 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi167.front.sepia.ceph.com, path=None, version=distro) 2024-02-05T06:18:29.618 DEBUG:teuthology.orchestra.run.smithi167:> rpm -q kernel | sort -rV | head -n 1 2024-02-05T06:18:29.726 INFO:teuthology.orchestra.run.smithi167.stdout:kernel-5.14.0-416.el9.x86_64 2024-02-05T06:18:29.726 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-416.el9.x86_64 2024-02-05T06:18:29.727 DEBUG:teuthology.orchestra.run.smithi167:> sudo rpm -qi grub2-tools 2024-02-05T06:18:29.765 INFO:teuthology.orchestra.run.smithi167.stdout:Name : grub2-tools 2024-02-05T06:18:29.765 INFO:teuthology.orchestra.run.smithi167.stdout:Epoch : 1 2024-02-05T06:18:29.765 INFO:teuthology.orchestra.run.smithi167.stdout:Version : 2.06 2024-02-05T06:18:29.765 INFO:teuthology.orchestra.run.smithi167.stdout:Release : 68.el9 2024-02-05T06:18:29.765 INFO:teuthology.orchestra.run.smithi167.stdout:Architecture: x86_64 2024-02-05T06:18:29.765 INFO:teuthology.orchestra.run.smithi167.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-05T06:18:29.765 INFO:teuthology.orchestra.run.smithi167.stdout:Group : Unspecified 2024-02-05T06:18:29.765 INFO:teuthology.orchestra.run.smithi167.stdout:Size : 8264254 2024-02-05T06:18:29.766 INFO:teuthology.orchestra.run.smithi167.stdout:License : GPLv3+ 2024-02-05T06:18:29.766 INFO:teuthology.orchestra.run.smithi167.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-05T06:18:29.766 INFO:teuthology.orchestra.run.smithi167.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-05T06:18:29.766 INFO:teuthology.orchestra.run.smithi167.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-05T06:18:29.766 INFO:teuthology.orchestra.run.smithi167.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-05T06:18:29.766 INFO:teuthology.orchestra.run.smithi167.stdout:Packager : builder@centos.org 2024-02-05T06:18:29.766 INFO:teuthology.orchestra.run.smithi167.stdout:Vendor : CentOS 2024-02-05T06:18:29.766 INFO:teuthology.orchestra.run.smithi167.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-05T06:18:29.766 INFO:teuthology.orchestra.run.smithi167.stdout:Summary : Support tools for GRUB. 2024-02-05T06:18:29.766 INFO:teuthology.orchestra.run.smithi167.stdout:Description : 2024-02-05T06:18:29.766 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:18:29.766 INFO:teuthology.orchestra.run.smithi167.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-05T06:18:29.766 INFO:teuthology.orchestra.run.smithi167.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-05T06:18:29.766 INFO:teuthology.orchestra.run.smithi167.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-05T06:18:29.766 INFO:teuthology.orchestra.run.smithi167.stdout:hardware devices. 2024-02-05T06:18:29.767 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:18:29.767 INFO:teuthology.orchestra.run.smithi167.stdout:This subpackage provides tools for support of all platforms. 2024-02-05T06:18:29.767 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-05T06:18:29.767 INFO:teuthology.task.kernel:Updating grub on smithi167 to boot 5.14.0-416.el9.x86_64 2024-02-05T06:18:29.768 DEBUG:teuthology.orchestra.run.smithi167:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-05T06:18:30.442 INFO:teuthology.orchestra.run.smithi167.stderr:Generating grub configuration file ... 2024-02-05T06:18:31.945 INFO:teuthology.orchestra.run.smithi167.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-05T06:18:31.992 INFO:teuthology.orchestra.run.smithi167.stderr:done 2024-02-05T06:18:31.992 DEBUG:teuthology.orchestra.run.smithi167:> mktemp 2024-02-05T06:18:32.007 INFO:teuthology.orchestra.run.smithi167.stdout:/tmp/tmp.9cTkq9HZnV 2024-02-05T06:18:32.008 DEBUG:teuthology.orchestra.run.smithi167:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.9cTkq9HZnV 2024-02-05T06:18:32.076 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0666 /tmp/tmp.9cTkq9HZnV 2024-02-05T06:18:32.221 DEBUG:teuthology.orchestra.remote:smithi167:/tmp/tmp.9cTkq9HZnV is 6KB 2024-02-05T06:18:32.270 DEBUG:teuthology.orchestra.run.smithi167:> rm -fr /tmp/tmp.9cTkq9HZnV 2024-02-05T06:18:32.285 DEBUG:teuthology.orchestra.run.smithi167:> sudo /bin/ls /boot/loader/entries || true 2024-02-05T06:18:32.349 INFO:teuthology.orchestra.run.smithi167.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-05T06:18:32.349 INFO:teuthology.orchestra.run.smithi167.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-05T06:18:32.349 INFO:teuthology.orchestra.run.smithi167.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-05T06:18:32.350 INFO:teuthology.orchestra.run.smithi167.stdout:15949e3c88704300882482eddfd7baae-5.14.0-416.el9.x86_64.conf 2024-02-05T06:18:32.351 DEBUG:teuthology.orchestra.run.smithi167:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-416.el9.x86_64 2024-02-05T06:18:32.498 DEBUG:teuthology.orchestra.run.smithi167:> sudo shutdown -r now 2024-02-05T06:18:48.443 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T06:18:48.443 INFO:teuthology.orchestra.run.smithi028.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-05T06:18:48.443 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:18:48.516 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-modules-5.14.0-416.el9.x86_64 4/4 2024-02-05T06:18:48.980 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: kernel-5.14.0-416.el9.x86_64 4/4 2024-02-05T06:18:48.981 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-5.14.0-416.el9.x86_64 1/4 2024-02-05T06:18:48.981 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-core-5.14.0-416.el9.x86_64 2/4 2024-02-05T06:18:48.981 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-modules-5.14.0-416.el9.x86_64 3/4 2024-02-05T06:18:49.416 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : kernel-modules-core-5.14.0-416.el9.x86_64 4/4 2024-02-05T06:18:49.416 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:18:49.416 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-02-05T06:18:49.416 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-5.14.0-416.el9.x86_64 2024-02-05T06:18:49.416 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-core-5.14.0-416.el9.x86_64 2024-02-05T06:18:49.416 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-5.14.0-416.el9.x86_64 2024-02-05T06:18:49.416 INFO:teuthology.orchestra.run.smithi028.stdout: kernel-modules-core-5.14.0-416.el9.x86_64 2024-02-05T06:18:49.416 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:18:49.416 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-02-05T06:18:49.922 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q kernel | sort -rV | head -n 1 2024-02-05T06:18:50.053 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-5.14.0-416.el9.x86_64 2024-02-05T06:18:50.053 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-416.el9.x86_64 2024-02-05T06:18:50.053 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-416.el9.x86_64 2024-02-05T06:18:50.053 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-05T06:18:50.054 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-05T06:18:50.054 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-02-05T06:18:50.054 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi028.front.sepia.ceph.com, path=None, version=distro) 2024-02-05T06:18:50.054 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q kernel | sort -rV | head -n 1 2024-02-05T06:18:50.166 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-5.14.0-416.el9.x86_64 2024-02-05T06:18:50.166 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-416.el9.x86_64 2024-02-05T06:18:50.166 DEBUG:teuthology.orchestra.run.smithi028:> sudo rpm -qi grub2-tools 2024-02-05T06:18:50.212 INFO:teuthology.orchestra.run.smithi028.stdout:Name : grub2-tools 2024-02-05T06:18:50.212 INFO:teuthology.orchestra.run.smithi028.stdout:Epoch : 1 2024-02-05T06:18:50.212 INFO:teuthology.orchestra.run.smithi028.stdout:Version : 2.06 2024-02-05T06:18:50.212 INFO:teuthology.orchestra.run.smithi028.stdout:Release : 68.el9 2024-02-05T06:18:50.212 INFO:teuthology.orchestra.run.smithi028.stdout:Architecture: x86_64 2024-02-05T06:18:50.212 INFO:teuthology.orchestra.run.smithi028.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-05T06:18:50.212 INFO:teuthology.orchestra.run.smithi028.stdout:Group : Unspecified 2024-02-05T06:18:50.212 INFO:teuthology.orchestra.run.smithi028.stdout:Size : 8264254 2024-02-05T06:18:50.212 INFO:teuthology.orchestra.run.smithi028.stdout:License : GPLv3+ 2024-02-05T06:18:50.213 INFO:teuthology.orchestra.run.smithi028.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-05T06:18:50.213 INFO:teuthology.orchestra.run.smithi028.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-05T06:18:50.213 INFO:teuthology.orchestra.run.smithi028.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-05T06:18:50.213 INFO:teuthology.orchestra.run.smithi028.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-05T06:18:50.213 INFO:teuthology.orchestra.run.smithi028.stdout:Packager : builder@centos.org 2024-02-05T06:18:50.213 INFO:teuthology.orchestra.run.smithi028.stdout:Vendor : CentOS 2024-02-05T06:18:50.213 INFO:teuthology.orchestra.run.smithi028.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-05T06:18:50.213 INFO:teuthology.orchestra.run.smithi028.stdout:Summary : Support tools for GRUB. 2024-02-05T06:18:50.213 INFO:teuthology.orchestra.run.smithi028.stdout:Description : 2024-02-05T06:18:50.213 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:18:50.213 INFO:teuthology.orchestra.run.smithi028.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-05T06:18:50.213 INFO:teuthology.orchestra.run.smithi028.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-05T06:18:50.213 INFO:teuthology.orchestra.run.smithi028.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-05T06:18:50.213 INFO:teuthology.orchestra.run.smithi028.stdout:hardware devices. 2024-02-05T06:18:50.213 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:18:50.214 INFO:teuthology.orchestra.run.smithi028.stdout:This subpackage provides tools for support of all platforms. 2024-02-05T06:18:50.214 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-05T06:18:50.214 INFO:teuthology.task.kernel:Updating grub on smithi028 to boot 5.14.0-416.el9.x86_64 2024-02-05T06:18:50.215 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-05T06:18:50.810 INFO:teuthology.orchestra.run.smithi028.stderr:Generating grub configuration file ... 2024-02-05T06:18:52.203 INFO:teuthology.orchestra.run.smithi028.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-05T06:18:52.247 INFO:teuthology.orchestra.run.smithi028.stderr:done 2024-02-05T06:18:52.249 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2024-02-05T06:18:52.264 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.rSlOcY4XDK 2024-02-05T06:18:52.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.rSlOcY4XDK 2024-02-05T06:18:52.337 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.rSlOcY4XDK 2024-02-05T06:18:52.479 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.rSlOcY4XDK is 6KB 2024-02-05T06:18:52.528 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.rSlOcY4XDK 2024-02-05T06:18:52.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo /bin/ls /boot/loader/entries || true 2024-02-05T06:18:52.611 INFO:teuthology.orchestra.run.smithi028.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-05T06:18:52.611 INFO:teuthology.orchestra.run.smithi028.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-05T06:18:52.612 INFO:teuthology.orchestra.run.smithi028.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-05T06:18:52.612 INFO:teuthology.orchestra.run.smithi028.stdout:15949e3c88704300882482eddfd7baae-5.14.0-416.el9.x86_64.conf 2024-02-05T06:18:52.613 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-416.el9.x86_64 2024-02-05T06:18:52.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo shutdown -r now 2024-02-05T06:19:02.508 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-02-05T06:19:02.508 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-02-05T06:19:02.509 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T06:19:22.764 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-02-05T06:19:22.764 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-02-05T06:19:22.765 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T06:19:37.152 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2024-02-05T06:19:46.162 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-02-05T06:19:46.163 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T06:20:04.540 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2024-02-05T06:20:16.552 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-02-05T06:20:16.553 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T06:20:19.616 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.167 2024-02-05T06:20:22.769 DEBUG:teuthology.orchestra.remote:timed out 2024-02-05T06:20:31.779 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-02-05T06:20:31.780 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T06:20:32.057 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-02-05T06:20:32.446 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-02-05T06:20:32.446 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-416.el9.x86_64"... 2024-02-05T06:20:32.446 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-02-05T06:20:32.462 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-416.el9.x86_64 2024-02-05T06:20:32.462 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-416.el9.x86_64 vs 5.14.0-416.el9.x86_64 2024-02-05T06:20:32.462 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-05T06:20:32.462 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-05T06:20:33.463 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-02-05T06:20:33.463 DEBUG:teuthology.orchestra.run.smithi028:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-05T06:20:33.548 INFO:teuthology.orchestra.run.smithi028.stdout:ttyS1 2024-02-05T06:20:33.586 DEBUG:teuthology.parallel:result is None 2024-02-05T06:20:34.617 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-02-05T06:20:34.617 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T06:20:34.901 DEBUG:teuthology.orchestra.run.smithi167:> true 2024-02-05T06:20:35.295 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-02-05T06:20:35.295 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-416.el9.x86_64"... 2024-02-05T06:20:35.296 DEBUG:teuthology.orchestra.run.smithi167:> uname -r 2024-02-05T06:20:35.311 INFO:teuthology.orchestra.run.smithi167.stdout:5.14.0-416.el9.x86_64 2024-02-05T06:20:35.311 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-416.el9.x86_64 vs 5.14.0-416.el9.x86_64 2024-02-05T06:20:35.311 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-05T06:20:35.311 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-05T06:20:36.312 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-02-05T06:20:36.312 DEBUG:teuthology.orchestra.run.smithi167:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-05T06:20:36.374 INFO:teuthology.orchestra.run.smithi167.stdout:ttyS1 2024-02-05T06:20:36.419 DEBUG:teuthology.parallel:result is None 2024-02-05T06:20:36.419 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-05T06:20:36.426 INFO:teuthology.task.internal:Creating test directory... 2024-02-05T06:20:36.426 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-05T06:20:36.429 DEBUG:teuthology.orchestra.run.smithi167:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-05T06:20:36.446 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-05T06:20:36.453 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-05T06:20:36.460 INFO:teuthology.task.internal:Creating archive directory... 2024-02-05T06:20:36.460 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-05T06:20:36.487 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-05T06:20:36.519 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-05T06:20:36.526 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-05T06:20:36.527 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-05T06:20:36.556 DEBUG:teuthology.orchestra.run.smithi167:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-05T06:20:36.589 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:20:36.591 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:20:36.603 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:20:36.605 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-05T06:20:36.607 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-05T06:20:36.614 INFO:teuthology.task.internal:Configuring sudo... 2024-02-05T06:20:36.614 DEBUG:teuthology.orchestra.run.smithi028:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-05T06:20:36.648 DEBUG:teuthology.orchestra.run.smithi167:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-05T06:20:36.678 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-05T06:20:36.688 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-05T06:20:36.688 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-05T06:20:36.717 DEBUG:teuthology.orchestra.run.smithi167:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-05T06:20:36.734 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-05T06:20:36.811 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-05T06:20:36.862 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-05T06:20:36.944 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-05T06:20:36.985 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-05T06:20:36.986 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-05T06:20:37.052 DEBUG:teuthology.orchestra.run.smithi167:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-05T06:20:37.095 DEBUG:teuthology.orchestra.run.smithi167:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-05T06:20:37.152 DEBUG:teuthology.orchestra.run.smithi167:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-05T06:20:37.234 DEBUG:teuthology.orchestra.run.smithi167:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-05T06:20:37.280 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-02-05T06:20:37.280 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-05T06:20:37.349 DEBUG:teuthology.orchestra.run.smithi028:> sudo service rsyslog restart 2024-02-05T06:20:37.351 DEBUG:teuthology.orchestra.run.smithi167:> sudo service rsyslog restart 2024-02-05T06:20:37.392 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-05T06:20:37.436 INFO:teuthology.orchestra.run.smithi167.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-05T06:20:37.949 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-05T06:20:37.957 INFO:teuthology.task.internal:Starting timer... 2024-02-05T06:20:37.957 INFO:teuthology.run_tasks:Running task pcp... 2024-02-05T06:20:37.967 INFO:teuthology.run_tasks:Running task selinux... 2024-02-05T06:20:37.974 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-05T06:20:37.974 DEBUG:teuthology.orchestra.run.smithi028:> sudo service auditd rotate 2024-02-05T06:20:38.018 INFO:teuthology.orchestra.run.smithi028.stdout:Rotating logs: 2024-02-05T06:20:38.019 DEBUG:teuthology.orchestra.run.smithi167:> sudo service auditd rotate 2024-02-05T06:20:38.073 INFO:teuthology.orchestra.run.smithi167.stdout:Rotating logs: 2024-02-05T06:20:38.075 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-05T06:20:38.075 DEBUG:teuthology.orchestra.run.smithi028:> /usr/sbin/getenforce 2024-02-05T06:20:38.097 INFO:teuthology.orchestra.run.smithi028.stdout:Permissive 2024-02-05T06:20:38.118 DEBUG:teuthology.orchestra.run.smithi167:> /usr/sbin/getenforce 2024-02-05T06:20:38.142 INFO:teuthology.orchestra.run.smithi167.stdout:Permissive 2024-02-05T06:20:38.142 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi028.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi167.front.sepia.ceph.com': 'permissive'} 2024-02-05T06:20:38.142 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-05T06:20:38.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:20:38.169 DEBUG:teuthology.orchestra.run.smithi167:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-05T06:20:38.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:20:38.212 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-05T06:20:38.212 DEBUG:teuthology.orchestra.run.smithi028:> sudo /usr/sbin/setenforce permissive 2024-02-05T06:20:38.249 DEBUG:teuthology.orchestra.run.smithi167:> sudo /usr/sbin/setenforce permissive 2024-02-05T06:20:38.286 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-05T06:20:38.301 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-02-05T06:20:38.306 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-02-05T06:20:38.484 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-05T06:20:38.499 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-05T06:20:38.500 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi028.front.sepia.ceph.com,smithi167.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-05T06:24:44.586 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi028.front.sepia.ceph.com'), Remote(name='ubuntu@smithi167.front.sepia.ceph.com')] 2024-02-05T06:24:44.587 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-02-05T06:24:44.588 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T06:24:44.681 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-02-05T06:24:44.755 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-02-05T06:24:44.755 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-02-05T06:24:44.756 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi167.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-05T06:24:44.844 DEBUG:teuthology.orchestra.run.smithi167:> true 2024-02-05T06:24:44.930 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi167.front.sepia.ceph.com' 2024-02-05T06:24:44.931 INFO:teuthology.run_tasks:Running task clock... 2024-02-05T06:24:44.942 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-05T06:24:44.943 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-05T06:24:44.943 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-05T06:24:44.945 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-05T06:24:44.946 DEBUG:teuthology.orchestra.run.smithi167:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-05T06:24:44.976 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-05T06:24:44.993 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-05T06:24:45.002 INFO:teuthology.orchestra.run.smithi167.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-05T06:24:45.020 INFO:teuthology.orchestra.run.smithi167.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-05T06:24:45.023 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: ntpd: command not found 2024-02-05T06:24:45.037 INFO:teuthology.orchestra.run.smithi028.stdout:506 Cannot talk to daemon 2024-02-05T06:24:45.051 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: ntpd: command not found 2024-02-05T06:24:45.053 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-05T06:24:45.065 INFO:teuthology.orchestra.run.smithi167.stdout:506 Cannot talk to daemon 2024-02-05T06:24:45.069 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-05T06:24:45.082 INFO:teuthology.orchestra.run.smithi167.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-05T06:24:45.100 INFO:teuthology.orchestra.run.smithi167.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-05T06:24:45.127 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-02-05T06:24:45.131 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-05T06:24:45.131 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-02-05T06:24:45.131 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T06:24:45.131 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T06:24:45.131 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T06:24:45.131 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T06:24:45.160 INFO:teuthology.orchestra.run.smithi167.stderr:bash: line 1: ntpq: command not found 2024-02-05T06:24:45.163 INFO:teuthology.orchestra.run.smithi167.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-05T06:24:45.163 INFO:teuthology.orchestra.run.smithi167.stdout:=============================================================================== 2024-02-05T06:24:45.163 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T06:24:45.163 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T06:24:45.163 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T06:24:45.163 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T06:24:45.164 INFO:teuthology.run_tasks:Running task install... 2024-02-05T06:24:45.172 DEBUG:teuthology.task.install:project ceph 2024-02-05T06:24:45.172 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '164d3be4a527be775f4d3c343ea591122f1b8921'}} 2024-02-05T06:24:45.173 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '164d3be4a527be775f4d3c343ea591122f1b8921'} 2024-02-05T06:24:45.173 INFO:teuthology.task.install:Using flavor: default 2024-02-05T06:24:45.178 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-05T06:24:45.178 INFO:teuthology.task.install:extra packages: [] 2024-02-05T06:24:45.179 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '164d3be4a527be775f4d3c343ea591122f1b8921', 'tag': None, 'wait_for_package': False} 2024-02-05T06:24:45.179 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=164d3be4a527be775f4d3c343ea591122f1b8921 2024-02-05T06:24:45.181 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '164d3be4a527be775f4d3c343ea591122f1b8921', 'tag': None, 'wait_for_package': False} 2024-02-05T06:24:45.182 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=164d3be4a527be775f4d3c343ea591122f1b8921 2024-02-05T06:24:45.333 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/flavors/default/ 2024-02-05T06:24:45.333 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1241.g164d3be4 2024-02-05T06:24:45.337 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/flavors/default/ 2024-02-05T06:24:45.337 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1241.g164d3be4 2024-02-05T06:24:45.448 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-05T06:24:45.448 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-05T06:24:45.448 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-05T06:24:45.476 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-05T06:24:45.476 DEBUG:teuthology.orchestra.run.smithi028:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/164d3be4a527be775f4d3c343ea591122f1b8921/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-05T06:24:45.506 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-05T06:24:45.506 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-02-05T06:24:45.506 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-05T06:24:45.534 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-05T06:24:45.535 DEBUG:teuthology.orchestra.run.smithi167:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/164d3be4a527be775f4d3c343ea591122f1b8921/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-05T06:24:45.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-05T06:24:45.603 DEBUG:teuthology.orchestra.run.smithi167:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-05T06:24:45.623 DEBUG:teuthology.orchestra.run.smithi028:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-05T06:24:45.682 DEBUG:teuthology.orchestra.run.smithi167:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-05T06:24:45.689 INFO:teuthology.orchestra.run.smithi028.stdout:check_obsoletes = 1 2024-02-05T06:24:45.691 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum clean all 2024-02-05T06:24:45.748 INFO:teuthology.orchestra.run.smithi167.stdout:check_obsoletes = 1 2024-02-05T06:24:45.750 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum clean all 2024-02-05T06:24:45.937 INFO:teuthology.orchestra.run.smithi028.stdout:74 files removed 2024-02-05T06:24:45.964 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-radosgw 2024-02-05T06:24:45.994 INFO:teuthology.orchestra.run.smithi167.stdout:74 files removed 2024-02-05T06:24:46.021 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-radosgw 2024-02-05T06:24:46.696 INFO:teuthology.orchestra.run.smithi028.stdout:ceph packages for x86_64 196 kB/s | 80 kB 00:00 2024-02-05T06:24:46.730 INFO:teuthology.orchestra.run.smithi167.stdout:ceph packages for x86_64 204 kB/s | 80 kB 00:00 2024-02-05T06:24:47.098 INFO:teuthology.orchestra.run.smithi167.stdout:ceph noarch packages 31 kB/s | 10 kB 00:00 2024-02-05T06:24:47.122 INFO:teuthology.orchestra.run.smithi028.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-02-05T06:24:47.448 INFO:teuthology.orchestra.run.smithi167.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-02-05T06:24:47.497 INFO:teuthology.orchestra.run.smithi028.stdout:ceph source packages 5.6 kB/s | 1.9 kB 00:00 2024-02-05T06:24:47.942 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS-9 - AppStream 14 MB/s | 6.6 MB 00:00 2024-02-05T06:24:48.058 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS-9 - AppStream 13 MB/s | 6.6 MB 00:00 2024-02-05T06:24:49.418 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS-9 - BaseOS 5.4 MB/s | 2.0 MB 00:00 2024-02-05T06:24:49.558 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS-9 - BaseOS 5.6 MB/s | 2.0 MB 00:00 2024-02-05T06:24:50.451 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-02-05T06:24:50.482 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-02-05T06:24:52.617 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:01 2024-02-05T06:24:53.893 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - AppStream 7.9 MB/s | 19 MB 00:02 2024-02-05T06:24:58.327 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 2.9 MB/s | 5.9 MB 00:02 2024-02-05T06:25:00.729 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 15 kB 00:01 2024-02-05T06:25:02.481 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 13 MB/s | 20 MB 00:01 2024-02-05T06:25:07.529 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2024-02-05T06:25:10.177 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-02-05T06:25:10.192 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:25:10.192 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-02-05T06:25:10.192 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:25:10.192 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-02-05T06:25:10.192 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 9.7 M 2024-02-05T06:25:10.192 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrading: 2024-02-05T06:25:10.192 INFO:teuthology.orchestra.run.smithi028.stdout: librados2 x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 3.4 M 2024-02-05T06:25:10.192 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1 x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 3.1 M 2024-02-05T06:25:10.192 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-02-05T06:25:10.193 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options 2024-02-05T06:25:10.193 INFO:teuthology.orchestra.run.smithi028.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-02-05T06:25:10.193 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 5.1 M 2024-02-05T06:25:10.193 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 20 M 2024-02-05T06:25:10.193 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 25 k 2024-02-05T06:25:10.193 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-05T06:25:10.193 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-02-05T06:25:10.193 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-02-05T06:25:10.193 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2 x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 743 k 2024-02-05T06:25:10.193 INFO:teuthology.orchestra.run.smithi028.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-05T06:25:10.193 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-05T06:25:10.193 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-05T06:25:10.193 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1 x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 498 k 2024-02-05T06:25:10.194 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-05T06:25:10.194 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2 x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 5.1 M 2024-02-05T06:25:10.194 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-05T06:25:10.194 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-05T06:25:10.194 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-02-05T06:25:10.194 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse 2024-02-05T06:25:10.194 INFO:teuthology.orchestra.run.smithi028.stdout: x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 45 k 2024-02-05T06:25:10.194 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 131 k 2024-02-05T06:25:10.194 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 162 k 2024-02-05T06:25:10.194 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-05T06:25:10.194 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 320 k 2024-02-05T06:25:10.194 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 298 k 2024-02-05T06:25:10.195 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 99 k 2024-02-05T06:25:10.195 INFO:teuthology.orchestra.run.smithi028.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-02-05T06:25:10.195 INFO:teuthology.orchestra.run.smithi028.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-05T06:25:10.195 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:25:10.195 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-02-05T06:25:10.195 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:25:10.195 INFO:teuthology.orchestra.run.smithi028.stdout:Install 27 Packages 2024-02-05T06:25:10.195 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrade 2 Packages 2024-02-05T06:25:10.195 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:25:10.196 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 57 M 2024-02-05T06:25:10.196 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-02-05T06:25:10.947 INFO:teuthology.orchestra.run.smithi028.stdout:(1/29): ceph-base-19.0.0-1241.g164d3be4.el9.x86 7.3 MB/s | 5.1 MB 00:00 2024-02-05T06:25:10.989 INFO:teuthology.orchestra.run.smithi028.stdout:(2/29): ceph-selinux-19.0.0-1241.g164d3be4.el9. 606 kB/s | 25 kB 00:00 2024-02-05T06:25:11.106 INFO:teuthology.orchestra.run.smithi028.stdout:(3/29): libcephfs2-19.0.0-1241.g164d3be4.el9.x8 6.2 MB/s | 743 kB 00:00 2024-02-05T06:25:11.181 INFO:teuthology.orchestra.run.smithi028.stdout:(4/29): libradosstriper1-19.0.0-1241.g164d3be4. 6.5 MB/s | 498 kB 00:00 2024-02-05T06:25:11.324 INFO:teuthology.orchestra.run.smithi028.stdout:(5/29): ceph-radosgw-19.0.0-1241.g164d3be4.el9. 9.0 MB/s | 9.7 MB 00:01 2024-02-05T06:25:11.390 INFO:teuthology.orchestra.run.smithi028.stdout:(6/29): python3-ceph-argparse-19.0.0-1241.g164d 678 kB/s | 45 kB 00:00 2024-02-05T06:25:11.457 INFO:teuthology.orchestra.run.smithi028.stdout:(7/29): python3-ceph-common-19.0.0-1241.g164d3b 1.9 MB/s | 131 kB 00:00 2024-02-05T06:25:11.525 INFO:teuthology.orchestra.run.smithi028.stdout:(8/29): python3-cephfs-19.0.0-1241.g164d3be4.el 2.4 MB/s | 162 kB 00:00 2024-02-05T06:25:11.624 INFO:teuthology.orchestra.run.smithi028.stdout:(9/29): librgw2-19.0.0-1241.g164d3be4.el9.x86_6 11 MB/s | 5.1 MB 00:00 2024-02-05T06:25:11.650 INFO:teuthology.orchestra.run.smithi028.stdout:(10/29): python3-rados-19.0.0-1241.g164d3be4.el 2.5 MB/s | 320 kB 00:00 2024-02-05T06:25:11.692 INFO:teuthology.orchestra.run.smithi028.stdout:(11/29): python3-rbd-19.0.0-1241.g164d3be4.el9. 4.4 MB/s | 298 kB 00:00 2024-02-05T06:25:11.717 INFO:teuthology.orchestra.run.smithi028.stdout:(12/29): python3-rgw-19.0.0-1241.g164d3be4.el9. 1.5 MB/s | 99 kB 00:00 2024-02-05T06:25:11.884 INFO:teuthology.orchestra.run.smithi028.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 552 kB/s | 106 kB 00:00 2024-02-05T06:25:11.909 INFO:teuthology.orchestra.run.smithi028.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 828 kB/s | 159 kB 00:00 2024-02-05T06:25:11.934 INFO:teuthology.orchestra.run.smithi028.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 884 kB/s | 44 kB 00:00 2024-02-05T06:25:11.943 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - CRB 417 kB/s | 5.9 MB 00:14 2024-02-05T06:25:12.018 INFO:teuthology.orchestra.run.smithi028.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 661 kB 00:00 2024-02-05T06:25:12.044 INFO:teuthology.orchestra.run.smithi028.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.5 MB/s | 282 kB 00:00 2024-02-05T06:25:12.068 INFO:teuthology.orchestra.run.smithi028.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 828 kB/s | 41 kB 00:00 2024-02-05T06:25:12.444 INFO:teuthology.orchestra.run.smithi028.stdout:(19/29): ceph-common-19.0.0-1241.g164d3be4.el9. 9.0 MB/s | 20 MB 00:02 2024-02-05T06:25:12.470 INFO:teuthology.orchestra.run.smithi028.stdout:(20/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 726 kB/s | 309 kB 00:00 2024-02-05T06:25:12.536 INFO:teuthology.orchestra.run.smithi028.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 1.2 MB/s | 49 kB 00:00 2024-02-05T06:25:12.578 INFO:teuthology.orchestra.run.smithi028.stdout:(22/29): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-02-05T06:25:12.603 INFO:teuthology.orchestra.run.smithi028.stdout:(23/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 163 kB/s | 26 kB 00:00 2024-02-05T06:25:12.695 INFO:teuthology.orchestra.run.smithi028.stdout:(24/29): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-02-05T06:25:12.795 INFO:teuthology.orchestra.run.smithi028.stdout:(25/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.8 MB/s | 838 kB 00:00 2024-02-05T06:25:13.029 INFO:teuthology.orchestra.run.smithi028.stdout:(26/29): librados2-19.0.0-1241.g164d3be4.el9.x8 14 MB/s | 3.4 MB 00:00 2024-02-05T06:25:13.163 INFO:teuthology.orchestra.run.smithi028.stdout:(27/29): thrift-0.15.0-2.el9.x86_64.rpm 3.4 MB/s | 1.6 MB 00:00 2024-02-05T06:25:13.221 INFO:teuthology.orchestra.run.smithi028.stdout:(28/29): librbd1-19.0.0-1241.g164d3be4.el9.x86_ 16 MB/s | 3.1 MB 00:00 2024-02-05T06:25:13.396 INFO:teuthology.orchestra.run.smithi028.stdout:(29/29): libarrow-9.0.0-10.el9.x86_64.rpm 3.3 MB/s | 4.4 MB 00:01 2024-02-05T06:25:13.403 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:25:13.403 INFO:teuthology.orchestra.run.smithi028.stdout:Total 18 MB/s | 57 MB 00:03 2024-02-05T06:25:13.403 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-02-05T06:25:13.440 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-02-05T06:25:13.440 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-02-05T06:25:13.552 INFO:teuthology.orchestra.run.smithi167.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 15 kB 00:00 2024-02-05T06:25:13.951 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-02-05T06:25:13.951 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-02-05T06:25:14.437 INFO:teuthology.orchestra.run.smithi167.stdout:Extra Packages for Enterprise Linux 27 MB/s | 20 MB 00:00 2024-02-05T06:25:15.507 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-02-05T06:25:15.602 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2024-02-05T06:25:15.635 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2024-02-05T06:25:15.687 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2024-02-05T06:25:16.081 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2024-02-05T06:25:16.117 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librados2-2:19.0.0-1241.g164d3be4.el9.x86_64 5/31 2024-02-05T06:25:16.201 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:19.0.0-1241.g164d3be4.el9.x86_64 5/31 2024-02-05T06:25:16.292 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rados-2:19.0.0-1241.g164d3be4.el9.x86_64 6/31 2024-02-05T06:25:16.327 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs2-2:19.0.0-1241.g164d3be4.el9.x86_64 7/31 2024-02-05T06:25:16.389 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephfs2-2:19.0.0-1241.g164d3be4.el9.x86_64 7/31 2024-02-05T06:25:16.418 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2024-02-05T06:25:16.463 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-argparse-2:19.0.0-1241.g164d3be4.el9. 9/31 2024-02-05T06:25:16.533 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cephfs-2:19.0.0-1241.g164d3be4.el9.x86_64 10/31 2024-02-05T06:25:16.554 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libradosstriper1-2:19.0.0-1241.g164d3be4.el9.x86_6 11/31 2024-02-05T06:25:16.620 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1241.g164d3be4.el9.x86_6 11/31 2024-02-05T06:25:16.729 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2024-02-05T06:25:16.762 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2024-02-05T06:25:16.803 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2024-02-05T06:25:16.827 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2024-02-05T06:25:16.872 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2024-02-05T06:25:16.905 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2024-02-05T06:25:17.246 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2024-02-05T06:25:17.277 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librbd1-2:19.0.0-1241.g164d3be4.el9.x86_64 19/31 2024-02-05T06:25:17.367 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:19.0.0-1241.g164d3be4.el9.x86_64 19/31 2024-02-05T06:25:17.403 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rbd-2:19.0.0-1241.g164d3be4.el9.x86_64 20/31 2024-02-05T06:25:17.448 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2024-02-05T06:25:17.930 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2024-02-05T06:25:17.954 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librgw2-2:19.0.0-1241.g164d3be4.el9.x86_64 23/31 2024-02-05T06:25:18.025 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librgw2-2:19.0.0-1241.g164d3be4.el9.x86_64 23/31 2024-02-05T06:25:18.103 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rgw-2:19.0.0-1241.g164d3be4.el9.x86_64 24/31 2024-02-05T06:25:18.131 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-common-2:19.0.0-1241.g164d3be4.el9.x8 25/31 2024-02-05T06:25:19.279 INFO:teuthology.orchestra.run.smithi167.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-02-05T06:25:20.445 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 26/31 2024-02-05T06:25:20.478 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 26/31 2024-02-05T06:25:21.062 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 26/31 2024-02-05T06:25:21.106 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-base-2:19.0.0-1241.g164d3be4.el9.x86_64 27/31 2024-02-05T06:25:21.180 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-base-2:19.0.0-1241.g164d3be4.el9.x86_64 27/31 2024-02-05T06:25:21.180 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-05T06:25:21.181 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-05T06:25:21.181 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:25:21.216 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-selinux-2:19.0.0-1241.g164d3be4.el9.x86_64 28/31 2024-02-05T06:25:21.987 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-02-05T06:25:22.001 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:25:22.001 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-02-05T06:25:22.001 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:25:22.001 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-02-05T06:25:22.001 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-radosgw x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 9.7 M 2024-02-05T06:25:22.001 INFO:teuthology.orchestra.run.smithi167.stdout:Upgrading: 2024-02-05T06:25:22.001 INFO:teuthology.orchestra.run.smithi167.stdout: librados2 x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 3.4 M 2024-02-05T06:25:22.001 INFO:teuthology.orchestra.run.smithi167.stdout: librbd1 x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 3.1 M 2024-02-05T06:25:22.001 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-02-05T06:25:22.001 INFO:teuthology.orchestra.run.smithi167.stdout: boost-program-options 2024-02-05T06:25:22.001 INFO:teuthology.orchestra.run.smithi167.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-02-05T06:25:22.001 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-base x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 5.1 M 2024-02-05T06:25:22.002 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-common x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 20 M 2024-02-05T06:25:22.002 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-selinux x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 25 k 2024-02-05T06:25:22.002 INFO:teuthology.orchestra.run.smithi167.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-05T06:25:22.002 INFO:teuthology.orchestra.run.smithi167.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-02-05T06:25:22.002 INFO:teuthology.orchestra.run.smithi167.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-02-05T06:25:22.002 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs2 x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 743 k 2024-02-05T06:25:22.002 INFO:teuthology.orchestra.run.smithi167.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-05T06:25:22.002 INFO:teuthology.orchestra.run.smithi167.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-05T06:25:22.002 INFO:teuthology.orchestra.run.smithi167.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-05T06:25:22.002 INFO:teuthology.orchestra.run.smithi167.stdout: libradosstriper1 x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 498 k 2024-02-05T06:25:22.002 INFO:teuthology.orchestra.run.smithi167.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-05T06:25:22.002 INFO:teuthology.orchestra.run.smithi167.stdout: librgw2 x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 5.1 M 2024-02-05T06:25:22.002 INFO:teuthology.orchestra.run.smithi167.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-05T06:25:22.002 INFO:teuthology.orchestra.run.smithi167.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-05T06:25:22.003 INFO:teuthology.orchestra.run.smithi167.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-02-05T06:25:22.003 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-argparse 2024-02-05T06:25:22.003 INFO:teuthology.orchestra.run.smithi167.stdout: x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 45 k 2024-02-05T06:25:22.003 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-common x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 131 k 2024-02-05T06:25:22.003 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cephfs x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 162 k 2024-02-05T06:25:22.003 INFO:teuthology.orchestra.run.smithi167.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-05T06:25:22.003 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rados x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 320 k 2024-02-05T06:25:22.003 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rbd x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 298 k 2024-02-05T06:25:22.003 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rgw x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 99 k 2024-02-05T06:25:22.003 INFO:teuthology.orchestra.run.smithi167.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-02-05T06:25:22.003 INFO:teuthology.orchestra.run.smithi167.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-05T06:25:22.003 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:25:22.003 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-02-05T06:25:22.003 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:25:22.003 INFO:teuthology.orchestra.run.smithi167.stdout:Install 27 Packages 2024-02-05T06:25:22.004 INFO:teuthology.orchestra.run.smithi167.stdout:Upgrade 2 Packages 2024-02-05T06:25:22.004 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:25:22.004 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 57 M 2024-02-05T06:25:22.004 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-02-05T06:25:22.556 INFO:teuthology.orchestra.run.smithi167.stdout:(1/29): ceph-base-19.0.0-1241.g164d3be4.el9.x86 10 MB/s | 5.1 MB 00:00 2024-02-05T06:25:22.606 INFO:teuthology.orchestra.run.smithi167.stdout:(2/29): ceph-selinux-19.0.0-1241.g164d3be4.el9. 505 kB/s | 25 kB 00:00 2024-02-05T06:25:22.681 INFO:teuthology.orchestra.run.smithi167.stdout:(3/29): libcephfs2-19.0.0-1241.g164d3be4.el9.x8 9.7 MB/s | 743 kB 00:00 2024-02-05T06:25:22.739 INFO:teuthology.orchestra.run.smithi167.stdout:(4/29): libradosstriper1-19.0.0-1241.g164d3be4. 8.4 MB/s | 498 kB 00:00 2024-02-05T06:25:23.056 INFO:teuthology.orchestra.run.smithi167.stdout:(5/29): librgw2-19.0.0-1241.g164d3be4.el9.x86_6 16 MB/s | 5.1 MB 00:00 2024-02-05T06:25:23.107 INFO:teuthology.orchestra.run.smithi167.stdout:(6/29): python3-ceph-argparse-19.0.0-1241.g164d 901 kB/s | 45 kB 00:00 2024-02-05T06:25:23.157 INFO:teuthology.orchestra.run.smithi167.stdout:(7/29): python3-ceph-common-19.0.0-1241.g164d3b 2.6 MB/s | 131 kB 00:00 2024-02-05T06:25:23.215 INFO:teuthology.orchestra.run.smithi167.stdout:(8/29): python3-cephfs-19.0.0-1241.g164d3be4.el 2.7 MB/s | 162 kB 00:00 2024-02-05T06:25:23.282 INFO:teuthology.orchestra.run.smithi167.stdout:(9/29): python3-rados-19.0.0-1241.g164d3be4.el9 4.7 MB/s | 320 kB 00:00 2024-02-05T06:25:23.349 INFO:teuthology.orchestra.run.smithi167.stdout:(10/29): python3-rbd-19.0.0-1241.g164d3be4.el9. 4.4 MB/s | 298 kB 00:00 2024-02-05T06:25:23.416 INFO:teuthology.orchestra.run.smithi167.stdout:(11/29): python3-rgw-19.0.0-1241.g164d3be4.el9. 1.5 MB/s | 99 kB 00:00 2024-02-05T06:25:23.550 INFO:teuthology.orchestra.run.smithi167.stdout:(12/29): boost-program-options-1.75.0-8.el9.x86 793 kB/s | 106 kB 00:00 2024-02-05T06:25:23.609 INFO:teuthology.orchestra.run.smithi167.stdout:(13/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-02-05T06:25:23.642 INFO:teuthology.orchestra.run.smithi167.stdout:(14/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-02-05T06:25:23.717 INFO:teuthology.orchestra.run.smithi167.stdout:(15/29): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-02-05T06:25:23.768 INFO:teuthology.orchestra.run.smithi167.stdout:(16/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-02-05T06:25:23.801 INFO:teuthology.orchestra.run.smithi167.stdout:(17/29): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-02-05T06:25:23.960 INFO:teuthology.orchestra.run.smithi167.stdout:(18/29): ceph-radosgw-19.0.0-1241.g164d3be4.el9 5.1 MB/s | 9.7 MB 00:01 2024-02-05T06:25:24.044 INFO:teuthology.orchestra.run.smithi167.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.2 MB/s | 309 kB 00:00 2024-02-05T06:25:24.102 INFO:teuthology.orchestra.run.smithi167.stdout:(20/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 453 kB/s | 26 kB 00:00 2024-02-05T06:25:24.194 INFO:teuthology.orchestra.run.smithi167.stdout:(21/29): libarrow-9.0.0-10.el9.x86_64.rpm 19 MB/s | 4.4 MB 00:00 2024-02-05T06:25:24.211 INFO:teuthology.orchestra.run.smithi167.stdout:(22/29): liboath-2.6.7-2.el9.x86_64.rpm 451 kB/s | 49 kB 00:00 2024-02-05T06:25:24.236 INFO:teuthology.orchestra.run.smithi167.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-02-05T06:25:24.279 INFO:teuthology.orchestra.run.smithi167.stdout:(24/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2024-02-05T06:25:24.304 INFO:teuthology.orchestra.run.smithi167.stdout:(25/29): re2-20211101-3.el9.x86_64.rpm 2.9 MB/s | 192 kB 00:00 2024-02-05T06:25:24.379 INFO:teuthology.orchestra.run.smithi167.stdout:(26/29): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-02-05T06:25:24.755 INFO:teuthology.orchestra.run.smithi167.stdout:(27/29): ceph-common-19.0.0-1241.g164d3be4.el9. 7.3 MB/s | 20 MB 00:02 2024-02-05T06:25:24.805 INFO:teuthology.orchestra.run.smithi167.stdout:(28/29): librados2-19.0.0-1241.g164d3be4.el9.x8 6.7 MB/s | 3.4 MB 00:00 2024-02-05T06:25:25.113 INFO:teuthology.orchestra.run.smithi167.stdout:(29/29): librbd1-19.0.0-1241.g164d3be4.el9.x86_ 4.3 MB/s | 3.1 MB 00:00 2024-02-05T06:25:25.118 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:25:25.118 INFO:teuthology.orchestra.run.smithi167.stdout:Total 18 MB/s | 57 MB 00:03 2024-02-05T06:25:25.118 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-02-05T06:25:25.153 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-02-05T06:25:25.153 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-02-05T06:25:25.641 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-02-05T06:25:25.641 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-02-05T06:25:27.022 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-02-05T06:25:27.122 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2024-02-05T06:25:27.159 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2024-02-05T06:25:27.211 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2024-02-05T06:25:27.604 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2024-02-05T06:25:27.650 INFO:teuthology.orchestra.run.smithi167.stdout: Upgrading : librados2-2:19.0.0-1241.g164d3be4.el9.x86_64 5/31 2024-02-05T06:25:27.742 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librados2-2:19.0.0-1241.g164d3be4.el9.x86_64 5/31 2024-02-05T06:25:27.829 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rados-2:19.0.0-1241.g164d3be4.el9.x86_64 6/31 2024-02-05T06:25:27.860 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libcephfs2-2:19.0.0-1241.g164d3be4.el9.x86_64 7/31 2024-02-05T06:25:27.922 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libcephfs2-2:19.0.0-1241.g164d3be4.el9.x86_64 7/31 2024-02-05T06:25:27.951 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2024-02-05T06:25:28.151 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-ceph-argparse-2:19.0.0-1241.g164d3be4.el9. 9/31 2024-02-05T06:25:28.222 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cephfs-2:19.0.0-1241.g164d3be4.el9.x86_64 10/31 2024-02-05T06:25:28.254 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libradosstriper1-2:19.0.0-1241.g164d3be4.el9.x86_6 11/31 2024-02-05T06:25:28.320 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1241.g164d3be4.el9.x86_6 11/31 2024-02-05T06:25:28.427 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2024-02-05T06:25:28.462 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2024-02-05T06:25:28.502 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2024-02-05T06:25:28.535 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2024-02-05T06:25:28.563 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2024-02-05T06:25:28.596 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2024-02-05T06:25:28.905 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2024-02-05T06:25:28.934 INFO:teuthology.orchestra.run.smithi167.stdout: Upgrading : librbd1-2:19.0.0-1241.g164d3be4.el9.x86_64 19/31 2024-02-05T06:25:29.044 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librbd1-2:19.0.0-1241.g164d3be4.el9.x86_64 19/31 2024-02-05T06:25:29.073 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rbd-2:19.0.0-1241.g164d3be4.el9.x86_64 20/31 2024-02-05T06:25:29.113 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2024-02-05T06:25:29.565 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2024-02-05T06:25:29.587 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librgw2-2:19.0.0-1241.g164d3be4.el9.x86_64 23/31 2024-02-05T06:25:29.659 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librgw2-2:19.0.0-1241.g164d3be4.el9.x86_64 23/31 2024-02-05T06:25:29.734 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rgw-2:19.0.0-1241.g164d3be4.el9.x86_64 24/31 2024-02-05T06:25:29.765 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-ceph-common-2:19.0.0-1241.g164d3be4.el9.x8 25/31 2024-02-05T06:25:31.163 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1241.g164d3be4.el9.x86_64 28/31 2024-02-05T06:25:31.163 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /sys 2024-02-05T06:25:31.163 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /proc 2024-02-05T06:25:31.163 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /mnt 2024-02-05T06:25:31.163 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /var/tmp 2024-02-05T06:25:31.163 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /home 2024-02-05T06:25:31.164 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /root 2024-02-05T06:25:31.164 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /tmp 2024-02-05T06:25:31.164 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:25:31.977 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 26/31 2024-02-05T06:25:32.011 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 26/31 2024-02-05T06:25:32.117 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-radosgw-2:19.0.0-1241.g164d3be4.el9.x86_64 29/31 2024-02-05T06:25:32.144 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1241.g164d3be4.el9.x86_64 29/31 2024-02-05T06:25:32.144 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T06:25:32.144 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-05T06:25:32.144 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-05T06:25:32.144 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-05T06:25:32.144 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:25:32.146 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-02-05T06:25:32.236 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-02-05T06:25:32.236 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T06:25:32.541 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 26/31 2024-02-05T06:25:32.590 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-base-2:19.0.0-1241.g164d3be4.el9.x86_64 27/31 2024-02-05T06:25:32.682 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-base-2:19.0.0-1241.g164d3be4.el9.x86_64 27/31 2024-02-05T06:25:32.683 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-05T06:25:32.683 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-05T06:25:32.683 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:25:32.716 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-selinux-2:19.0.0-1241.g164d3be4.el9.x86_64 28/31 2024-02-05T06:25:33.287 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T06:25:33.287 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-base-2:19.0.0-1241.g164d3be4.el9.x86_64 1/31 2024-02-05T06:25:33.287 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 2/31 2024-02-05T06:25:33.287 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-radosgw-2:19.0.0-1241.g164d3be4.el9.x86_64 3/31 2024-02-05T06:25:33.288 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-selinux-2:19.0.0-1241.g164d3be4.el9.x86_64 4/31 2024-02-05T06:25:33.288 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs2-2:19.0.0-1241.g164d3be4.el9.x86_64 5/31 2024-02-05T06:25:33.288 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libradosstriper1-2:19.0.0-1241.g164d3be4.el9.x86_6 6/31 2024-02-05T06:25:33.288 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librgw2-2:19.0.0-1241.g164d3be4.el9.x86_64 7/31 2024-02-05T06:25:33.288 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1241.g164d3be4.el9. 8/31 2024-02-05T06:25:33.288 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-common-2:19.0.0-1241.g164d3be4.el9.x8 9/31 2024-02-05T06:25:33.288 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cephfs-2:19.0.0-1241.g164d3be4.el9.x86_64 10/31 2024-02-05T06:25:33.288 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rados-2:19.0.0-1241.g164d3be4.el9.x86_64 11/31 2024-02-05T06:25:33.288 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rbd-2:19.0.0-1241.g164d3be4.el9.x86_64 12/31 2024-02-05T06:25:33.288 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rgw-2:19.0.0-1241.g164d3be4.el9.x86_64 13/31 2024-02-05T06:25:33.288 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2024-02-05T06:25:33.288 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2024-02-05T06:25:33.288 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2024-02-05T06:25:33.289 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2024-02-05T06:25:33.289 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2024-02-05T06:25:33.289 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2024-02-05T06:25:33.289 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2024-02-05T06:25:33.289 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2024-02-05T06:25:33.289 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2024-02-05T06:25:33.289 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2024-02-05T06:25:33.289 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2024-02-05T06:25:33.289 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2024-02-05T06:25:33.289 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2024-02-05T06:25:33.289 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2024-02-05T06:25:33.290 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:19.0.0-1241.g164d3be4.el9.x86_64 28/31 2024-02-05T06:25:33.290 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2024-02-05T06:25:33.290 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:19.0.0-1241.g164d3be4.el9.x86_64 30/31 2024-02-05T06:25:34.451 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T06:25:34.451 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:25:34.451 INFO:teuthology.orchestra.run.smithi028.stdout:Upgraded: 2024-02-05T06:25:34.452 INFO:teuthology.orchestra.run.smithi028.stdout: librados2-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:34.452 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:34.452 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-02-05T06:25:34.452 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-02-05T06:25:34.452 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:34.452 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:34.452 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:34.452 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:34.452 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-05T06:25:34.452 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-9.0.0-10.el9.x86_64 2024-02-05T06:25:34.452 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-02-05T06:25:34.452 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:34.452 INFO:teuthology.orchestra.run.smithi028.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-05T06:25:34.453 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-05T06:25:34.453 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-05T06:25:34.453 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:34.453 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-05T06:25:34.453 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:34.453 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-05T06:25:34.453 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-05T06:25:34.453 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-02-05T06:25:34.453 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:34.453 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:34.453 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:34.453 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-05T06:25:34.453 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:34.453 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:34.453 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:34.454 INFO:teuthology.orchestra.run.smithi028.stdout: re2-1:20211101-3.el9.x86_64 2024-02-05T06:25:34.454 INFO:teuthology.orchestra.run.smithi028.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-05T06:25:34.454 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:25:34.454 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-02-05T06:25:34.789 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-test 2024-02-05T06:25:35.360 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:28 ago on Mon 05 Feb 2024 06:25:07 AM UTC. 2024-02-05T06:25:35.525 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-02-05T06:25:35.526 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:25:35.526 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-02-05T06:25:35.526 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:25:35.526 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-02-05T06:25:35.526 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 47 M 2024-02-05T06:25:35.526 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-02-05T06:25:35.526 INFO:teuthology.orchestra.run.smithi028.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-05T06:25:35.526 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 163 k 2024-02-05T06:25:35.526 INFO:teuthology.orchestra.run.smithi028.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-05T06:25:35.526 INFO:teuthology.orchestra.run.smithi028.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-05T06:25:35.526 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-05T06:25:35.526 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:25:35.527 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-02-05T06:25:35.527 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:25:35.527 INFO:teuthology.orchestra.run.smithi028.stdout:Install 6 Packages 2024-02-05T06:25:35.527 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:25:35.527 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 48 M 2024-02-05T06:25:35.527 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 204 M 2024-02-05T06:25:35.527 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-02-05T06:25:35.814 INFO:teuthology.orchestra.run.smithi028.stdout:(1/6): libcephsqlite-19.0.0-1241.g164d3be4.el9. 571 kB/s | 163 kB 00:00 2024-02-05T06:25:35.873 INFO:teuthology.orchestra.run.smithi028.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 871 kB/s | 300 kB 00:00 2024-02-05T06:25:35.898 INFO:teuthology.orchestra.run.smithi028.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 751 kB/s | 63 kB 00:00 2024-02-05T06:25:35.965 INFO:teuthology.orchestra.run.smithi028.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 2.0 MB/s | 186 kB 00:00 2024-02-05T06:25:36.006 INFO:teuthology.orchestra.run.smithi028.stdout:(5/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.9 MB/s | 217 kB 00:00 2024-02-05T06:25:37.915 INFO:teuthology.orchestra.run.smithi028.stdout:(6/6): ceph-test-19.0.0-1241.g164d3be4.el9.x86_ 20 MB/s | 47 MB 00:02 2024-02-05T06:25:37.918 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:25:37.918 INFO:teuthology.orchestra.run.smithi028.stdout:Total 20 MB/s | 48 MB 00:02 2024-02-05T06:25:37.919 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-02-05T06:25:37.941 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-02-05T06:25:37.942 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-02-05T06:25:38.349 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-02-05T06:25:38.349 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-02-05T06:25:38.930 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-02-05T06:25:39.004 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-05T06:25:39.046 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-05T06:25:39.092 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-05T06:25:39.143 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-05T06:25:39.168 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephsqlite-2:19.0.0-1241.g164d3be4.el9.x86_64 5/6 2024-02-05T06:25:42.349 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1241.g164d3be4.el9.x86_64 28/31 2024-02-05T06:25:42.350 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /sys 2024-02-05T06:25:42.350 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /proc 2024-02-05T06:25:42.350 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /mnt 2024-02-05T06:25:42.350 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /var/tmp 2024-02-05T06:25:42.350 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /home 2024-02-05T06:25:42.350 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /root 2024-02-05T06:25:42.350 INFO:teuthology.orchestra.run.smithi167.stdout:skipping the directory /tmp 2024-02-05T06:25:42.350 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:25:43.283 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-radosgw-2:19.0.0-1241.g164d3be4.el9.x86_64 29/31 2024-02-05T06:25:43.308 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1241.g164d3be4.el9.x86_64 29/31 2024-02-05T06:25:43.308 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T06:25:43.308 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-05T06:25:43.308 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-05T06:25:43.308 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-05T06:25:43.308 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:25:43.310 INFO:teuthology.orchestra.run.smithi167.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-02-05T06:25:43.394 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-02-05T06:25:43.394 INFO:teuthology.orchestra.run.smithi167.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T06:25:43.907 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1241.g164d3be4.el9.x86_64 5/6 2024-02-05T06:25:43.987 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-test-2:19.0.0-1241.g164d3be4.el9.x86_64 6/6 2024-02-05T06:25:44.500 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T06:25:44.500 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-base-2:19.0.0-1241.g164d3be4.el9.x86_64 1/31 2024-02-05T06:25:44.500 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 2/31 2024-02-05T06:25:44.501 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-radosgw-2:19.0.0-1241.g164d3be4.el9.x86_64 3/31 2024-02-05T06:25:44.501 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-selinux-2:19.0.0-1241.g164d3be4.el9.x86_64 4/31 2024-02-05T06:25:44.501 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libcephfs2-2:19.0.0-1241.g164d3be4.el9.x86_64 5/31 2024-02-05T06:25:44.501 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libradosstriper1-2:19.0.0-1241.g164d3be4.el9.x86_6 6/31 2024-02-05T06:25:44.501 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librgw2-2:19.0.0-1241.g164d3be4.el9.x86_64 7/31 2024-02-05T06:25:44.501 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1241.g164d3be4.el9. 8/31 2024-02-05T06:25:44.501 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-ceph-common-2:19.0.0-1241.g164d3be4.el9.x8 9/31 2024-02-05T06:25:44.501 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cephfs-2:19.0.0-1241.g164d3be4.el9.x86_64 10/31 2024-02-05T06:25:44.501 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rados-2:19.0.0-1241.g164d3be4.el9.x86_64 11/31 2024-02-05T06:25:44.501 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rbd-2:19.0.0-1241.g164d3be4.el9.x86_64 12/31 2024-02-05T06:25:44.501 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rgw-2:19.0.0-1241.g164d3be4.el9.x86_64 13/31 2024-02-05T06:25:44.501 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2024-02-05T06:25:44.501 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2024-02-05T06:25:44.501 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2024-02-05T06:25:44.502 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2024-02-05T06:25:44.502 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2024-02-05T06:25:44.502 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2024-02-05T06:25:44.502 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2024-02-05T06:25:44.502 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2024-02-05T06:25:44.502 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2024-02-05T06:25:44.502 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2024-02-05T06:25:44.502 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2024-02-05T06:25:44.502 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2024-02-05T06:25:44.502 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2024-02-05T06:25:44.502 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2024-02-05T06:25:44.502 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librados2-2:19.0.0-1241.g164d3be4.el9.x86_64 28/31 2024-02-05T06:25:44.502 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2024-02-05T06:25:44.502 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librbd1-2:19.0.0-1241.g164d3be4.el9.x86_64 30/31 2024-02-05T06:25:45.579 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2024-02-05T06:25:45.580 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:25:45.580 INFO:teuthology.orchestra.run.smithi167.stdout:Upgraded: 2024-02-05T06:25:45.580 INFO:teuthology.orchestra.run.smithi167.stdout: librados2-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:45.580 INFO:teuthology.orchestra.run.smithi167.stdout: librbd1-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:45.580 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-02-05T06:25:45.580 INFO:teuthology.orchestra.run.smithi167.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-02-05T06:25:45.580 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-base-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:45.580 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:45.580 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-radosgw-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:45.580 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-selinux-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:45.580 INFO:teuthology.orchestra.run.smithi167.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-05T06:25:45.580 INFO:teuthology.orchestra.run.smithi167.stdout: libarrow-9.0.0-10.el9.x86_64 2024-02-05T06:25:45.580 INFO:teuthology.orchestra.run.smithi167.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-02-05T06:25:45.581 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs2-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:45.581 INFO:teuthology.orchestra.run.smithi167.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-05T06:25:45.581 INFO:teuthology.orchestra.run.smithi167.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-05T06:25:45.581 INFO:teuthology.orchestra.run.smithi167.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-05T06:25:45.581 INFO:teuthology.orchestra.run.smithi167.stdout: libradosstriper1-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:45.581 INFO:teuthology.orchestra.run.smithi167.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-05T06:25:45.581 INFO:teuthology.orchestra.run.smithi167.stdout: librgw2-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:45.581 INFO:teuthology.orchestra.run.smithi167.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-05T06:25:45.581 INFO:teuthology.orchestra.run.smithi167.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-05T06:25:45.581 INFO:teuthology.orchestra.run.smithi167.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-02-05T06:25:45.581 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-argparse-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:45.581 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ceph-common-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:45.581 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cephfs-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:45.581 INFO:teuthology.orchestra.run.smithi167.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-05T06:25:45.581 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rados-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:45.582 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rbd-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:45.582 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rgw-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:45.582 INFO:teuthology.orchestra.run.smithi167.stdout: re2-1:20211101-3.el9.x86_64 2024-02-05T06:25:45.582 INFO:teuthology.orchestra.run.smithi167.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-05T06:25:45.582 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:25:45.582 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-02-05T06:25:45.904 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-test 2024-02-05T06:25:46.392 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-test-2:19.0.0-1241.g164d3be4.el9.x86_64 6/6 2024-02-05T06:25:46.392 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-test-2:19.0.0-1241.g164d3be4.el9.x86_64 1/6 2024-02-05T06:25:46.392 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephsqlite-2:19.0.0-1241.g164d3be4.el9.x86_64 2/6 2024-02-05T06:25:46.393 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-05T06:25:46.393 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-05T06:25:46.393 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-05T06:25:46.474 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:00:27 ago on Mon 05 Feb 2024 06:25:19 AM UTC. 2024-02-05T06:25:46.630 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-02-05T06:25:46.631 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:25:46.631 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-02-05T06:25:46.631 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:25:46.631 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-02-05T06:25:46.631 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-test x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 47 M 2024-02-05T06:25:46.631 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-02-05T06:25:46.631 INFO:teuthology.orchestra.run.smithi167.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-05T06:25:46.631 INFO:teuthology.orchestra.run.smithi167.stdout: libcephsqlite x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 163 k 2024-02-05T06:25:46.631 INFO:teuthology.orchestra.run.smithi167.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-05T06:25:46.632 INFO:teuthology.orchestra.run.smithi167.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-05T06:25:46.632 INFO:teuthology.orchestra.run.smithi167.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-05T06:25:46.632 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:25:46.632 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-02-05T06:25:46.632 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:25:46.632 INFO:teuthology.orchestra.run.smithi167.stdout:Install 6 Packages 2024-02-05T06:25:46.632 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:25:46.632 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 48 M 2024-02-05T06:25:46.633 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 204 M 2024-02-05T06:25:46.633 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-02-05T06:25:46.806 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-05T06:25:46.806 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:25:46.806 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-02-05T06:25:46.807 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:46.807 INFO:teuthology.orchestra.run.smithi028.stdout: jq-1.6-16.el9.x86_64 2024-02-05T06:25:46.807 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:46.807 INFO:teuthology.orchestra.run.smithi028.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-05T06:25:46.807 INFO:teuthology.orchestra.run.smithi028.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-05T06:25:46.807 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-05T06:25:46.807 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:25:46.807 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-02-05T06:25:46.831 INFO:teuthology.orchestra.run.smithi167.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.5 MB/s | 300 kB 00:00 2024-02-05T06:25:46.873 INFO:teuthology.orchestra.run.smithi167.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-02-05T06:25:46.923 INFO:teuthology.orchestra.run.smithi167.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 3.6 MB/s | 186 kB 00:00 2024-02-05T06:25:46.974 INFO:teuthology.orchestra.run.smithi167.stdout:(4/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-02-05T06:25:47.007 INFO:teuthology.orchestra.run.smithi167.stdout:(5/6): libcephsqlite-19.0.0-1241.g164d3be4.el9. 436 kB/s | 163 kB 00:00 2024-02-05T06:25:47.088 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph 2024-02-05T06:25:47.662 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:40 ago on Mon 05 Feb 2024 06:25:07 AM UTC. 2024-02-05T06:25:47.836 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-02-05T06:25:47.838 INFO:teuthology.orchestra.run.smithi028.stdout:========================================================================================= 2024-02-05T06:25:47.838 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-02-05T06:25:47.839 INFO:teuthology.orchestra.run.smithi028.stdout:========================================================================================= 2024-02-05T06:25:47.839 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-02-05T06:25:47.839 INFO:teuthology.orchestra.run.smithi028.stdout: ceph x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 6.6 k 2024-02-05T06:25:47.839 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-02-05T06:25:47.839 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 2.1 M 2024-02-05T06:25:47.839 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 1.5 M 2024-02-05T06:25:47.839 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 247 k 2024-02-05T06:25:47.839 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 4.7 M 2024-02-05T06:25:47.839 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 17 M 2024-02-05T06:25:47.839 INFO:teuthology.orchestra.run.smithi028.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-02-05T06:25:47.839 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-02-05T06:25:47.839 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-02-05T06:25:47.839 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-05T06:25:47.839 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-05T06:25:47.840 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-05T06:25:47.840 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-05T06:25:47.840 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-05T06:25:47.840 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-05T06:25:47.840 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-05T06:25:47.840 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-05T06:25:47.840 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-05T06:25:47.840 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-05T06:25:47.840 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-05T06:25:47.840 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-05T06:25:47.840 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-05T06:25:47.840 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-05T06:25:47.841 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-05T06:25:47.841 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-05T06:25:47.841 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-05T06:25:47.841 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-05T06:25:47.841 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-05T06:25:47.841 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-05T06:25:47.841 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-05T06:25:47.841 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-05T06:25:47.841 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-05T06:25:47.841 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-05T06:25:47.841 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-05T06:25:47.841 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-05T06:25:47.841 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-05T06:25:47.841 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:25:47.841 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-02-05T06:25:47.841 INFO:teuthology.orchestra.run.smithi028.stdout:========================================================================================= 2024-02-05T06:25:47.841 INFO:teuthology.orchestra.run.smithi028.stdout:Install 36 Packages 2024-02-05T06:25:47.842 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:25:47.842 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 30 M 2024-02-05T06:25:47.842 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 107 M 2024-02-05T06:25:47.842 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-02-05T06:25:48.366 INFO:teuthology.orchestra.run.smithi028.stdout:(1/36): ceph-19.0.0-1241.g164d3be4.el9.x86_64.r 21 kB/s | 6.6 kB 00:00 2024-02-05T06:25:48.699 INFO:teuthology.orchestra.run.smithi028.stdout:(2/36): ceph-mgr-19.0.0-1241.g164d3be4.el9.x86_ 2.3 MB/s | 1.5 MB 00:00 2024-02-05T06:25:48.908 INFO:teuthology.orchestra.run.smithi028.stdout:(3/36): ceph-mds-19.0.0-1241.g164d3be4.el9.x86_ 2.5 MB/s | 2.1 MB 00:00 2024-02-05T06:25:48.983 INFO:teuthology.orchestra.run.smithi167.stdout:(6/6): ceph-test-19.0.0-1241.g164d3be4.el9.x86_ 20 MB/s | 47 MB 00:02 2024-02-05T06:25:48.983 INFO:teuthology.orchestra.run.smithi028.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1241.g164d 3.2 MB/s | 247 kB 00:00 2024-02-05T06:25:48.988 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:25:48.988 INFO:teuthology.orchestra.run.smithi167.stdout:Total 20 MB/s | 48 MB 00:02 2024-02-05T06:25:48.988 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-02-05T06:25:49.007 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-02-05T06:25:49.007 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-02-05T06:25:49.142 INFO:teuthology.orchestra.run.smithi028.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-02-05T06:25:49.192 INFO:teuthology.orchestra.run.smithi028.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-02-05T06:25:49.234 INFO:teuthology.orchestra.run.smithi028.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 760 kB/s | 32 kB 00:00 2024-02-05T06:25:49.276 INFO:teuthology.orchestra.run.smithi028.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-02-05T06:25:49.317 INFO:teuthology.orchestra.run.smithi028.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 930 kB/s | 39 kB 00:00 2024-02-05T06:25:49.368 INFO:teuthology.orchestra.run.smithi028.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-02-05T06:25:49.391 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-02-05T06:25:49.391 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-02-05T06:25:49.452 INFO:teuthology.orchestra.run.smithi028.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-02-05T06:25:49.494 INFO:teuthology.orchestra.run.smithi028.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-02-05T06:25:49.536 INFO:teuthology.orchestra.run.smithi028.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-02-05T06:25:49.578 INFO:teuthology.orchestra.run.smithi028.stdout:(14/36): python3-requests-2.25.1-8.el9.noarch.r 2.6 MB/s | 113 kB 00:00 2024-02-05T06:25:49.619 INFO:teuthology.orchestra.run.smithi028.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-02-05T06:25:49.887 INFO:teuthology.orchestra.run.smithi028.stdout:(16/36): ceph-osd-19.0.0-1241.g164d3be4.el9.x86 14 MB/s | 17 MB 00:01 2024-02-05T06:25:49.903 INFO:teuthology.orchestra.run.smithi028.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 79 kB/s | 22 kB 00:00 2024-02-05T06:25:49.970 INFO:teuthology.orchestra.run.smithi028.stdout:(18/36): ceph-mon-19.0.0-1241.g164d3be4.el9.x86 2.9 MB/s | 4.7 MB 00:01 2024-02-05T06:25:49.974 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-02-05T06:25:50.045 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-05T06:25:50.054 INFO:teuthology.orchestra.run.smithi028.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 290 kB/s | 43 kB 00:00 2024-02-05T06:25:50.074 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-05T06:25:50.086 INFO:teuthology.orchestra.run.smithi028.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 759 kB/s | 151 kB 00:00 2024-02-05T06:25:50.126 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-05T06:25:50.128 INFO:teuthology.orchestra.run.smithi028.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-02-05T06:25:50.153 INFO:teuthology.orchestra.run.smithi028.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 160 kB/s | 11 kB 00:00 2024-02-05T06:25:50.170 INFO:teuthology.orchestra.run.smithi028.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 420 kB/s | 18 kB 00:00 2024-02-05T06:25:50.179 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-05T06:25:50.196 INFO:teuthology.orchestra.run.smithi028.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 549 kB/s | 23 kB 00:00 2024-02-05T06:25:50.211 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libcephsqlite-2:19.0.0-1241.g164d3be4.el9.x86_64 5/6 2024-02-05T06:25:50.229 INFO:teuthology.orchestra.run.smithi028.stdout:(25/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.0 MB/s | 358 kB 00:00 2024-02-05T06:25:50.246 INFO:teuthology.orchestra.run.smithi028.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 257 kB/s | 19 kB 00:00 2024-02-05T06:25:50.263 INFO:teuthology.orchestra.run.smithi028.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 293 kB/s | 20 kB 00:00 2024-02-05T06:25:50.280 INFO:teuthology.orchestra.run.smithi028.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 919 kB/s | 46 kB 00:00 2024-02-05T06:25:50.297 INFO:teuthology.orchestra.run.smithi028.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-02-05T06:25:50.330 INFO:teuthology.orchestra.run.smithi028.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-02-05T06:25:50.356 INFO:teuthology.orchestra.run.smithi028.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-02-05T06:25:50.381 INFO:teuthology.orchestra.run.smithi028.stdout:(32/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-02-05T06:25:50.406 INFO:teuthology.orchestra.run.smithi028.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 474 kB/s | 36 kB 00:00 2024-02-05T06:25:50.432 INFO:teuthology.orchestra.run.smithi028.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-02-05T06:25:50.457 INFO:teuthology.orchestra.run.smithi028.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 398 kB/s | 20 kB 00:00 2024-02-05T06:25:50.482 INFO:teuthology.orchestra.run.smithi028.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-02-05T06:25:50.486 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:25:50.486 INFO:teuthology.orchestra.run.smithi028.stdout:Total 11 MB/s | 30 MB 00:02 2024-02-05T06:25:50.507 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-02-05T06:25:50.554 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-02-05T06:25:50.555 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-02-05T06:25:50.986 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-02-05T06:25:50.987 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-02-05T06:25:52.370 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-02-05T06:25:52.443 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-02-05T06:25:52.485 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-02-05T06:25:52.546 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-02-05T06:25:52.622 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-02-05T06:25:52.679 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-02-05T06:25:52.711 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-02-05T06:25:52.809 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-02-05T06:25:52.892 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-02-05T06:25:52.940 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-02-05T06:25:53.017 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-02-05T06:25:53.073 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-02-05T06:25:53.111 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-02-05T06:25:53.178 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-02-05T06:25:53.248 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-02-05T06:25:53.442 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-02-05T06:25:53.490 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-02-05T06:25:53.538 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-02-05T06:25:53.571 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-02-05T06:25:53.601 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-02-05T06:25:53.680 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-02-05T06:25:53.712 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-05T06:25:53.745 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-02-05T06:25:53.785 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-02-05T06:25:53.812 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-02-05T06:25:53.838 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-02-05T06:25:53.931 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-02-05T06:25:53.968 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-02-05T06:25:54.043 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-02-05T06:25:54.182 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-02-05T06:25:54.318 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-05T06:25:54.484 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1241.g164d3be4.el9. 31/36 2024-02-05T06:25:54.515 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-2:19.0.0-1241.g164d3be4.el9.x86_64 32/36 2024-02-05T06:25:54.539 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1241.g164d3be4.el9.x86_64 32/36 2024-02-05T06:25:54.539 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T06:25:54.539 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-05T06:25:54.539 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-05T06:25:54.539 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-05T06:25:54.540 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:25:54.658 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1241.g164d3be4.el9.x86_64 5/6 2024-02-05T06:25:54.735 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-test-2:19.0.0-1241.g164d3be4.el9.x86_64 6/6 2024-02-05T06:25:56.058 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-osd-2:19.0.0-1241.g164d3be4.el9.x86_64 33/36 2024-02-05T06:25:56.085 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-osd-2:19.0.0-1241.g164d3be4.el9.x86_64 33/36 2024-02-05T06:25:56.085 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T06:25:56.085 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-05T06:25:56.085 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-05T06:25:56.085 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-05T06:25:56.085 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:25:56.559 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mon-2:19.0.0-1241.g164d3be4.el9.x86_64 34/36 2024-02-05T06:25:56.593 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mon-2:19.0.0-1241.g164d3be4.el9.x86_64 34/36 2024-02-05T06:25:56.593 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T06:25:56.593 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-05T06:25:56.593 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-05T06:25:56.593 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-05T06:25:56.594 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:25:56.801 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mds-2:19.0.0-1241.g164d3be4.el9.x86_64 35/36 2024-02-05T06:25:56.823 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mds-2:19.0.0-1241.g164d3be4.el9.x86_64 35/36 2024-02-05T06:25:56.824 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T06:25:56.824 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-05T06:25:56.824 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-05T06:25:56.824 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-05T06:25:56.824 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:25:56.892 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-2:19.0.0-1241.g164d3be4.el9.x86_64 36/36 2024-02-05T06:25:57.283 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-test-2:19.0.0-1241.g164d3be4.el9.x86_64 6/6 2024-02-05T06:25:57.284 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-test-2:19.0.0-1241.g164d3be4.el9.x86_64 1/6 2024-02-05T06:25:57.284 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libcephsqlite-2:19.0.0-1241.g164d3be4.el9.x86_64 2/6 2024-02-05T06:25:57.284 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-05T06:25:57.284 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-05T06:25:57.284 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-05T06:25:57.671 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-05T06:25:57.671 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:25:57.671 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-02-05T06:25:57.671 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-test-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:57.671 INFO:teuthology.orchestra.run.smithi167.stdout: jq-1.6-16.el9.x86_64 2024-02-05T06:25:57.671 INFO:teuthology.orchestra.run.smithi167.stdout: libcephsqlite-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:57.671 INFO:teuthology.orchestra.run.smithi167.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-05T06:25:57.671 INFO:teuthology.orchestra.run.smithi167.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-05T06:25:57.671 INFO:teuthology.orchestra.run.smithi167.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-05T06:25:57.671 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:25:57.672 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-02-05T06:25:57.951 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph 2024-02-05T06:25:58.520 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:00:39 ago on Mon 05 Feb 2024 06:25:19 AM UTC. 2024-02-05T06:25:58.651 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-2:19.0.0-1241.g164d3be4.el9.x86_64 36/36 2024-02-05T06:25:58.651 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-2:19.0.0-1241.g164d3be4.el9.x86_64 1/36 2024-02-05T06:25:58.651 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mds-2:19.0.0-1241.g164d3be4.el9.x86_64 2/36 2024-02-05T06:25:58.652 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-2:19.0.0-1241.g164d3be4.el9.x86_64 3/36 2024-02-05T06:25:58.652 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mon-2:19.0.0-1241.g164d3be4.el9.x86_64 4/36 2024-02-05T06:25:58.652 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-osd-2:19.0.0-1241.g164d3be4.el9.x86_64 5/36 2024-02-05T06:25:58.652 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1241.g164d3be4.el9. 6/36 2024-02-05T06:25:58.652 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-02-05T06:25:58.652 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-02-05T06:25:58.652 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-02-05T06:25:58.652 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-02-05T06:25:58.652 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-02-05T06:25:58.652 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-02-05T06:25:58.652 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-02-05T06:25:58.652 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-02-05T06:25:58.653 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-02-05T06:25:58.653 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/36 2024-02-05T06:25:58.653 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-02-05T06:25:58.653 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-02-05T06:25:58.653 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-02-05T06:25:58.653 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-02-05T06:25:58.653 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-05T06:25:58.653 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-02-05T06:25:58.653 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-02-05T06:25:58.653 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-02-05T06:25:58.653 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-02-05T06:25:58.653 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-02-05T06:25:58.653 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-02-05T06:25:58.653 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-02-05T06:25:58.654 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-02-05T06:25:58.654 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-05T06:25:58.654 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-02-05T06:25:58.654 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-02-05T06:25:58.654 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-02-05T06:25:58.654 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-02-05T06:25:58.654 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-02-05T06:25:58.683 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-02-05T06:25:58.685 INFO:teuthology.orchestra.run.smithi167.stdout:========================================================================================= 2024-02-05T06:25:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-02-05T06:25:58.685 INFO:teuthology.orchestra.run.smithi167.stdout:========================================================================================= 2024-02-05T06:25:58.686 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-02-05T06:25:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: ceph x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 6.6 k 2024-02-05T06:25:58.686 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-02-05T06:25:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mds x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 2.1 M 2024-02-05T06:25:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 1.5 M 2024-02-05T06:25:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 247 k 2024-02-05T06:25:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mon x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 4.7 M 2024-02-05T06:25:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-osd x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 17 M 2024-02-05T06:25:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-02-05T06:25:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-02-05T06:25:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-02-05T06:25:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-05T06:25:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-05T06:25:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-05T06:25:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-05T06:25:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-05T06:25:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-05T06:25:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-05T06:25:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-05T06:25:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-05T06:25:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-05T06:25:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-05T06:25:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-05T06:25:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-05T06:25:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-05T06:25:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-05T06:25:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-05T06:25:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-05T06:25:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-05T06:25:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-05T06:25:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-05T06:25:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-05T06:25:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-05T06:25:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-05T06:25:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-05T06:25:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-05T06:25:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-05T06:25:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-05T06:25:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:25:58.688 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-02-05T06:25:58.688 INFO:teuthology.orchestra.run.smithi167.stdout:========================================================================================= 2024-02-05T06:25:58.688 INFO:teuthology.orchestra.run.smithi167.stdout:Install 36 Packages 2024-02-05T06:25:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:25:58.689 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 30 M 2024-02-05T06:25:58.689 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 107 M 2024-02-05T06:25:58.689 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-02-05T06:25:59.143 INFO:teuthology.orchestra.run.smithi167.stdout:(1/36): ceph-19.0.0-1241.g164d3be4.el9.x86_64.r 38 kB/s | 6.6 kB 00:00 2024-02-05T06:25:59.585 INFO:teuthology.orchestra.run.smithi167.stdout:(2/36): ceph-mgr-19.0.0-1241.g164d3be4.el9.x86_ 2.4 MB/s | 1.5 MB 00:00 2024-02-05T06:25:59.652 INFO:teuthology.orchestra.run.smithi167.stdout:(3/36): ceph-mon-19.0.0-1241.g164d3be4.el9.x86_ 9.2 MB/s | 4.7 MB 00:00 2024-02-05T06:25:59.702 INFO:teuthology.orchestra.run.smithi167.stdout:(4/36): ceph-mds-19.0.0-1241.g164d3be4.el9.x86_ 2.9 MB/s | 2.1 MB 00:00 2024-02-05T06:25:59.727 INFO:teuthology.orchestra.run.smithi167.stdout:(5/36): ceph-mgr-modules-core-19.0.0-1241.g164d 3.2 MB/s | 247 kB 00:00 2024-02-05T06:25:59.732 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-02-05T06:25:59.732 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:25:59.732 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-02-05T06:25:59.732 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:59.732 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:59.732 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:59.733 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T06:25:59.733 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:59.733 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:25:59.733 INFO:teuthology.orchestra.run.smithi028.stdout: lua-5.4.4-4.el9.x86_64 2024-02-05T06:25:59.733 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-02-05T06:25:59.733 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks-3.9.2-1.el9.noarch 2024-02-05T06:25:59.733 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-05T06:25:59.733 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-05T06:25:59.733 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-05T06:25:59.733 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-05T06:25:59.733 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-05T06:25:59.733 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-05T06:25:59.733 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-05T06:25:59.733 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-05T06:25:59.734 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-05T06:25:59.734 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-05T06:25:59.734 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-05T06:25:59.734 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-05T06:25:59.734 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-05T06:25:59.734 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-05T06:25:59.734 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-05T06:25:59.734 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply-3.11-14.el9.noarch 2024-02-05T06:25:59.734 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-05T06:25:59.734 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-05T06:25:59.734 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-05T06:25:59.734 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-05T06:25:59.734 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-05T06:25:59.734 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-05T06:25:59.734 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-05T06:25:59.734 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-05T06:25:59.735 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-05T06:25:59.735 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-05T06:25:59.735 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-05T06:25:59.735 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:25:59.735 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-02-05T06:25:59.886 INFO:teuthology.orchestra.run.smithi167.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-02-05T06:25:59.911 INFO:teuthology.orchestra.run.smithi167.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 899 kB/s | 144 kB 00:00 2024-02-05T06:25:59.937 INFO:teuthology.orchestra.run.smithi167.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 628 kB/s | 32 kB 00:00 2024-02-05T06:25:59.962 INFO:teuthology.orchestra.run.smithi167.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 944 kB/s | 47 kB 00:00 2024-02-05T06:25:59.987 INFO:teuthology.orchestra.run.smithi167.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 769 kB/s | 39 kB 00:00 2024-02-05T06:26:00.012 INFO:teuthology.orchestra.run.smithi167.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-02-05T06:26:00.060 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-base 2024-02-05T06:26:00.079 INFO:teuthology.orchestra.run.smithi167.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 12 MB/s | 1.1 MB 00:00 2024-02-05T06:26:00.123 INFO:teuthology.orchestra.run.smithi167.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 933 kB/s | 103 kB 00:00 2024-02-05T06:26:00.154 INFO:teuthology.orchestra.run.smithi167.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 1.6 MB/s | 124 kB 00:00 2024-02-05T06:26:00.179 INFO:teuthology.orchestra.run.smithi167.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 2.0 MB/s | 113 kB 00:00 2024-02-05T06:26:00.205 INFO:teuthology.orchestra.run.smithi167.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-02-05T06:26:00.330 INFO:teuthology.orchestra.run.smithi167.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 149 kB/s | 22 kB 00:00 2024-02-05T06:26:00.589 INFO:teuthology.orchestra.run.smithi167.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 394 kB/s | 151 kB 00:00 2024-02-05T06:26:00.631 INFO:teuthology.orchestra.run.smithi167.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 145 kB/s | 43 kB 00:00 2024-02-05T06:26:00.635 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:53 ago on Mon 05 Feb 2024 06:25:07 AM UTC. 2024-02-05T06:26:00.743 INFO:teuthology.orchestra.run.smithi028.stdout:Package ceph-base-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T06:26:00.756 INFO:teuthology.orchestra.run.smithi167.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-02-05T06:26:00.797 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-02-05T06:26:00.799 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-02-05T06:26:00.799 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-02-05T06:26:00.856 INFO:teuthology.orchestra.run.smithi167.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-02-05T06:26:00.859 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install cephadm 2024-02-05T06:26:01.023 INFO:teuthology.orchestra.run.smithi167.stdout:(22/36): ceph-osd-19.0.0-1241.g164d3be4.el9.x86 12 MB/s | 17 MB 00:01 2024-02-05T06:26:01.048 INFO:teuthology.orchestra.run.smithi167.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 92 kB/s | 18 kB 00:00 2024-02-05T06:26:01.076 INFO:teuthology.orchestra.run.smithi167.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 809 kB/s | 358 kB 00:00 2024-02-05T06:26:01.149 INFO:teuthology.orchestra.run.smithi167.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 193 kB/s | 19 kB 00:00 2024-02-05T06:26:01.224 INFO:teuthology.orchestra.run.smithi167.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 131 kB/s | 20 kB 00:00 2024-02-05T06:26:01.250 INFO:teuthology.orchestra.run.smithi167.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 461 kB/s | 46 kB 00:00 2024-02-05T06:26:01.275 INFO:teuthology.orchestra.run.smithi167.stdout:(28/36): python3-jaraco-collections-3.0.0-8.el9 92 kB/s | 23 kB 00:00 2024-02-05T06:26:01.342 INFO:teuthology.orchestra.run.smithi167.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 672 kB/s | 79 kB 00:00 2024-02-05T06:26:01.359 INFO:teuthology.orchestra.run.smithi167.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-02-05T06:26:01.409 INFO:teuthology.orchestra.run.smithi167.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 123 kB/s | 16 kB 00:00 2024-02-05T06:26:01.428 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:54 ago on Mon 05 Feb 2024 06:25:07 AM UTC. 2024-02-05T06:26:01.434 INFO:teuthology.orchestra.run.smithi167.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 974 kB/s | 90 kB 00:00 2024-02-05T06:26:01.452 INFO:teuthology.orchestra.run.smithi167.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 386 kB/s | 36 kB 00:00 2024-02-05T06:26:01.595 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-02-05T06:26:01.595 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:26:01.595 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-02-05T06:26:01.596 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:26:01.596 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-02-05T06:26:01.596 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 748 k 2024-02-05T06:26:01.596 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:01.596 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-02-05T06:26:01.596 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:26:01.596 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-02-05T06:26:01.596 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:01.596 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 748 k 2024-02-05T06:26:01.596 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 754 k 2024-02-05T06:26:01.596 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-02-05T06:26:01.618 INFO:teuthology.orchestra.run.smithi167.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 120 kB/s | 20 kB 00:00 2024-02-05T06:26:01.660 INFO:teuthology.orchestra.run.smithi167.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 914 kB/s | 230 kB 00:00 2024-02-05T06:26:01.744 INFO:teuthology.orchestra.run.smithi167.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.3 MB/s | 427 kB 00:00 2024-02-05T06:26:01.747 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:26:01.748 INFO:teuthology.orchestra.run.smithi167.stdout:Total 9.7 MB/s | 30 MB 00:03 2024-02-05T06:26:01.766 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-02-05T06:26:01.809 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-02-05T06:26:01.809 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-02-05T06:26:02.068 INFO:teuthology.orchestra.run.smithi028.stdout:cephadm-19.0.0-1241.g164d3be4.el9.noarch.rpm 1.6 MB/s | 748 kB 00:00 2024-02-05T06:26:02.069 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:26:02.069 INFO:teuthology.orchestra.run.smithi028.stdout:Total 1.5 MB/s | 748 kB 00:00 2024-02-05T06:26:02.069 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-02-05T06:26:02.074 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-02-05T06:26:02.074 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-02-05T06:26:02.080 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-02-05T06:26:02.081 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-02-05T06:26:02.153 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-02-05T06:26:02.232 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-02-05T06:26:02.232 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-02-05T06:26:02.617 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 1/1 2024-02-05T06:26:02.691 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 1/1 2024-02-05T06:26:03.048 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 1/1 2024-02-05T06:26:03.361 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 1/1 2024-02-05T06:26:03.362 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:03.362 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-02-05T06:26:03.362 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T06:26:03.362 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:03.362 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-02-05T06:26:03.572 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-immutable-object-cache 2024-02-05T06:26:03.607 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-02-05T06:26:03.672 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-02-05T06:26:03.696 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-02-05T06:26:03.724 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-02-05T06:26:03.794 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-02-05T06:26:03.832 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-02-05T06:26:03.864 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-02-05T06:26:03.956 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-02-05T06:26:04.026 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-02-05T06:26:04.067 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-02-05T06:26:04.139 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-02-05T06:26:04.153 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:57 ago on Mon 05 Feb 2024 06:25:07 AM UTC. 2024-02-05T06:26:04.193 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-02-05T06:26:04.231 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-02-05T06:26:04.280 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-02-05T06:26:04.319 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-02-05T06:26:04.320 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:26:04.320 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repo Size 2024-02-05T06:26:04.321 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:26:04.321 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-02-05T06:26:04.321 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 145 k 2024-02-05T06:26:04.321 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:04.321 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-02-05T06:26:04.321 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:26:04.321 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-02-05T06:26:04.321 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:04.321 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 145 k 2024-02-05T06:26:04.321 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 436 k 2024-02-05T06:26:04.321 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-02-05T06:26:04.348 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-02-05T06:26:04.528 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-02-05T06:26:04.578 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-02-05T06:26:04.608 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-02-05T06:26:04.642 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-02-05T06:26:04.673 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-02-05T06:26:04.673 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-immutable-object-cache-19.0.0-1241.g164d3b 414 kB/s | 145 kB 00:00 2024-02-05T06:26:04.673 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:26:04.673 INFO:teuthology.orchestra.run.smithi028.stdout:Total 412 kB/s | 145 kB 00:00 2024-02-05T06:26:04.674 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-02-05T06:26:04.682 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-02-05T06:26:04.682 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-02-05T06:26:04.736 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-02-05T06:26:04.736 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-02-05T06:26:04.748 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-02-05T06:26:04.792 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-05T06:26:04.825 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-02-05T06:26:04.857 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-02-05T06:26:04.889 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-02-05T06:26:04.892 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-02-05T06:26:04.926 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-02-05T06:26:04.990 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1241.g164d3be4. 1/1 2024-02-05T06:26:05.015 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1241.g164d3be4. 1/1 2024-02-05T06:26:05.015 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T06:26:05.016 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-05T06:26:05.016 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:05.016 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-02-05T06:26:05.058 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-02-05T06:26:05.130 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-02-05T06:26:05.268 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-02-05T06:26:05.392 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-05T06:26:05.558 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1241.g164d3be4.el9. 31/36 2024-02-05T06:26:05.588 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-2:19.0.0-1241.g164d3be4.el9.x86_64 32/36 2024-02-05T06:26:05.612 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1241.g164d3be4.el9.x86_64 32/36 2024-02-05T06:26:05.612 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T06:26:05.612 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-05T06:26:05.612 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-05T06:26:05.613 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-05T06:26:05.613 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:06.100 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1241.g164d3be4. 1/1 2024-02-05T06:26:06.100 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:06.100 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-02-05T06:26:06.100 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:26:06.100 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:06.101 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-02-05T06:26:06.345 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-mgr 2024-02-05T06:26:06.918 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:59 ago on Mon 05 Feb 2024 06:25:07 AM UTC. 2024-02-05T06:26:07.027 INFO:teuthology.orchestra.run.smithi028.stdout:Package ceph-mgr-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T06:26:07.082 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-02-05T06:26:07.083 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-02-05T06:26:07.083 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-02-05T06:26:07.089 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-osd-2:19.0.0-1241.g164d3be4.el9.x86_64 33/36 2024-02-05T06:26:07.117 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-osd-2:19.0.0-1241.g164d3be4.el9.x86_64 33/36 2024-02-05T06:26:07.117 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T06:26:07.117 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-05T06:26:07.117 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-05T06:26:07.117 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-05T06:26:07.117 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:07.140 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-mgr-dashboard 2024-02-05T06:26:07.565 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mon-2:19.0.0-1241.g164d3be4.el9.x86_64 34/36 2024-02-05T06:26:07.589 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mon-2:19.0.0-1241.g164d3be4.el9.x86_64 34/36 2024-02-05T06:26:07.589 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T06:26:07.589 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-05T06:26:07.589 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-05T06:26:07.589 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-05T06:26:07.589 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:07.707 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:00 ago on Mon 05 Feb 2024 06:25:07 AM UTC. 2024-02-05T06:26:07.799 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mds-2:19.0.0-1241.g164d3be4.el9.x86_64 35/36 2024-02-05T06:26:07.823 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mds-2:19.0.0-1241.g164d3be4.el9.x86_64 35/36 2024-02-05T06:26:07.823 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T06:26:07.823 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-05T06:26:07.823 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-05T06:26:07.823 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-05T06:26:07.823 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:07.873 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-02-05T06:26:07.874 INFO:teuthology.orchestra.run.smithi028.stdout:====================================================================================== 2024-02-05T06:26:07.874 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-02-05T06:26:07.874 INFO:teuthology.orchestra.run.smithi028.stdout:====================================================================================== 2024-02-05T06:26:07.874 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-02-05T06:26:07.874 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 1.8 M 2024-02-05T06:26:07.875 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-02-05T06:26:07.875 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-02-05T06:26:07.875 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 28 k 2024-02-05T06:26:07.875 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 15 k 2024-02-05T06:26:07.875 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-02-05T06:26:07.875 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-02-05T06:26:07.875 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-02-05T06:26:07.875 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-02-05T06:26:07.875 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-02-05T06:26:07.875 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-02-05T06:26:07.875 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-05T06:26:07.875 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-05T06:26:07.875 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:07.875 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-02-05T06:26:07.876 INFO:teuthology.orchestra.run.smithi028.stdout:====================================================================================== 2024-02-05T06:26:07.876 INFO:teuthology.orchestra.run.smithi028.stdout:Install 12 Packages 2024-02-05T06:26:07.876 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:07.876 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 6.8 M 2024-02-05T06:26:07.876 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 26 M 2024-02-05T06:26:07.876 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-02-05T06:26:07.888 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-2:19.0.0-1241.g164d3be4.el9.x86_64 36/36 2024-02-05T06:26:08.192 INFO:teuthology.orchestra.run.smithi028.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1241.g16 157 kB/s | 28 kB 00:00 2024-02-05T06:26:08.284 INFO:teuthology.orchestra.run.smithi028.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1241.g164 56 kB/s | 15 kB 00:00 2024-02-05T06:26:08.460 INFO:teuthology.orchestra.run.smithi028.stdout:(3/12): ceph-mgr-dashboard-19.0.0-1241.g164d3be 3.9 MB/s | 1.8 MB 00:00 2024-02-05T06:26:08.543 INFO:teuthology.orchestra.run.smithi028.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.9 MB/s | 1.0 MB 00:00 2024-02-05T06:26:08.568 INFO:teuthology.orchestra.run.smithi028.stdout:(5/12): python3-protobuf-3.14.0-13.el9.noarch.r 838 kB/s | 238 kB 00:00 2024-02-05T06:26:08.619 INFO:teuthology.orchestra.run.smithi028.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 5.3 MB/s | 863 kB 00:00 2024-02-05T06:26:08.645 INFO:teuthology.orchestra.run.smithi028.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 258 kB/s | 19 kB 00:00 2024-02-05T06:26:08.736 INFO:teuthology.orchestra.run.smithi028.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.8 MB/s | 548 kB 00:00 2024-02-05T06:26:08.761 INFO:teuthology.orchestra.run.smithi028.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.2 MB/s | 144 kB 00:00 2024-02-05T06:26:08.778 INFO:teuthology.orchestra.run.smithi028.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 735 kB/s | 31 kB 00:00 2024-02-05T06:26:08.820 INFO:teuthology.orchestra.run.smithi028.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.2 MB/s | 188 kB 00:00 2024-02-05T06:26:08.887 INFO:teuthology.orchestra.run.smithi028.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 7.6 MB/s | 2.0 MB 00:00 2024-02-05T06:26:08.889 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:26:08.890 INFO:teuthology.orchestra.run.smithi028.stdout:Total 6.7 MB/s | 6.8 MB 00:01 2024-02-05T06:26:08.907 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-02-05T06:26:08.926 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-02-05T06:26:08.926 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-02-05T06:26:09.097 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-02-05T06:26:09.098 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-02-05T06:26:09.616 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-02-05T06:26:09.675 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-2:19.0.0-1241.g164d3be4.el9.x86_64 36/36 2024-02-05T06:26:09.675 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-2:19.0.0-1241.g164d3be4.el9.x86_64 1/36 2024-02-05T06:26:09.675 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mds-2:19.0.0-1241.g164d3be4.el9.x86_64 2/36 2024-02-05T06:26:09.675 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-2:19.0.0-1241.g164d3be4.el9.x86_64 3/36 2024-02-05T06:26:09.675 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mon-2:19.0.0-1241.g164d3be4.el9.x86_64 4/36 2024-02-05T06:26:09.675 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-osd-2:19.0.0-1241.g164d3be4.el9.x86_64 5/36 2024-02-05T06:26:09.675 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1241.g164d3be4.el9. 6/36 2024-02-05T06:26:09.676 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-02-05T06:26:09.676 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-02-05T06:26:09.676 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-02-05T06:26:09.676 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-02-05T06:26:09.676 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-02-05T06:26:09.676 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-02-05T06:26:09.676 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-02-05T06:26:09.676 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-02-05T06:26:09.676 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-02-05T06:26:09.676 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/36 2024-02-05T06:26:09.676 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-02-05T06:26:09.676 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-02-05T06:26:09.676 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-02-05T06:26:09.676 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-02-05T06:26:09.677 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-02-05T06:26:09.677 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-02-05T06:26:09.677 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-02-05T06:26:09.677 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-02-05T06:26:09.677 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-02-05T06:26:09.677 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-02-05T06:26:09.677 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-02-05T06:26:09.677 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-02-05T06:26:09.677 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-02-05T06:26:09.677 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-02-05T06:26:09.677 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-02-05T06:26:09.677 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-02-05T06:26:09.677 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-02-05T06:26:09.677 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-02-05T06:26:09.678 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-02-05T06:26:09.688 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-02-05T06:26:09.731 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-02-05T06:26:09.779 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-02-05T06:26:09.808 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-02-05T06:26:09.917 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-02-05T06:26:10.059 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-02-05T06:26:10.160 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-02-05T06:26:10.202 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-02-05T06:26:10.229 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-02-05T06:26:10.259 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1241.g164d3be4.el9 10/12 2024-02-05T06:26:10.513 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1241.g164d3be4.el 11/12 2024-02-05T06:26:10.552 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1241.g164d3be4.el9.noa 12/12 2024-02-05T06:26:10.629 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-02-05T06:26:10.629 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:10.629 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-02-05T06:26:10.629 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:26:10.629 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mds-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:26:10.629 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:26:10.629 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-modules-core-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T06:26:10.629 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mon-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:26:10.630 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-osd-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:26:10.630 INFO:teuthology.orchestra.run.smithi167.stdout: lua-5.4.4-4.el9.x86_64 2024-02-05T06:26:10.630 INFO:teuthology.orchestra.run.smithi167.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-02-05T06:26:10.630 INFO:teuthology.orchestra.run.smithi167.stdout: luarocks-3.9.2-1.el9.noarch 2024-02-05T06:26:10.630 INFO:teuthology.orchestra.run.smithi167.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-05T06:26:10.630 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-05T06:26:10.630 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-05T06:26:10.630 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-05T06:26:10.630 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-05T06:26:10.630 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-05T06:26:10.630 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-05T06:26:10.630 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-05T06:26:10.630 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-05T06:26:10.630 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-05T06:26:10.631 INFO:teuthology.orchestra.run.smithi167.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-05T06:26:10.631 INFO:teuthology.orchestra.run.smithi167.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-05T06:26:10.631 INFO:teuthology.orchestra.run.smithi167.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-05T06:26:10.631 INFO:teuthology.orchestra.run.smithi167.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-05T06:26:10.631 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-05T06:26:10.631 INFO:teuthology.orchestra.run.smithi167.stdout: python3-ply-3.11-14.el9.noarch 2024-02-05T06:26:10.631 INFO:teuthology.orchestra.run.smithi167.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-05T06:26:10.631 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-05T06:26:10.631 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-05T06:26:10.631 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-05T06:26:10.631 INFO:teuthology.orchestra.run.smithi167.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-05T06:26:10.631 INFO:teuthology.orchestra.run.smithi167.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-05T06:26:10.631 INFO:teuthology.orchestra.run.smithi167.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-05T06:26:10.631 INFO:teuthology.orchestra.run.smithi167.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-05T06:26:10.632 INFO:teuthology.orchestra.run.smithi167.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-05T06:26:10.632 INFO:teuthology.orchestra.run.smithi167.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-05T06:26:10.632 INFO:teuthology.orchestra.run.smithi167.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-05T06:26:10.632 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:10.632 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-02-05T06:26:11.006 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-base 2024-02-05T06:26:11.311 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1241.g164d3be4.el9.noa 12/12 2024-02-05T06:26:11.311 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1241.g164d3be4.el 1/12 2024-02-05T06:26:11.311 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1241.g164d3be4.el9.noa 2/12 2024-02-05T06:26:11.311 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1241.g164d3be4.el9 3/12 2024-02-05T06:26:11.311 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-02-05T06:26:11.311 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-02-05T06:26:11.311 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-02-05T06:26:11.312 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-02-05T06:26:11.312 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-02-05T06:26:11.312 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-02-05T06:26:11.312 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-02-05T06:26:11.312 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-02-05T06:26:11.605 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:00:52 ago on Mon 05 Feb 2024 06:25:19 AM UTC. 2024-02-05T06:26:11.668 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-02-05T06:26:11.668 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:11.668 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-02-05T06:26:11.668 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-02-05T06:26:11.668 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T06:26:11.669 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T06:26:11.669 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T06:26:11.669 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data-1.46.7-10.el9.noarch 2024-02-05T06:26:11.669 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-3.14.0-13.el9.x86_64 2024-02-05T06:26:11.669 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-02-05T06:26:11.669 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-02-05T06:26:11.669 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-02-05T06:26:11.669 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-02-05T06:26:11.669 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-05T06:26:11.669 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-05T06:26:11.669 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:11.669 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-02-05T06:26:11.711 INFO:teuthology.orchestra.run.smithi167.stdout:Package ceph-base-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T06:26:11.763 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-02-05T06:26:11.764 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-02-05T06:26:11.764 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-02-05T06:26:11.836 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install cephadm 2024-02-05T06:26:11.923 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-05T06:26:12.401 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:00:53 ago on Mon 05 Feb 2024 06:25:19 AM UTC. 2024-02-05T06:26:12.496 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:05 ago on Mon 05 Feb 2024 06:25:07 AM UTC. 2024-02-05T06:26:12.558 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-02-05T06:26:12.558 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:26:12.559 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-02-05T06:26:12.559 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:26:12.559 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-02-05T06:26:12.559 INFO:teuthology.orchestra.run.smithi167.stdout: cephadm noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 748 k 2024-02-05T06:26:12.559 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:12.559 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-02-05T06:26:12.559 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:26:12.559 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-02-05T06:26:12.559 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:12.559 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 748 k 2024-02-05T06:26:12.559 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 754 k 2024-02-05T06:26:12.559 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-02-05T06:26:12.659 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-02-05T06:26:12.660 INFO:teuthology.orchestra.run.smithi028.stdout:============================================================================================ 2024-02-05T06:26:12.660 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-02-05T06:26:12.660 INFO:teuthology.orchestra.run.smithi028.stdout:============================================================================================ 2024-02-05T06:26:12.660 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-02-05T06:26:12.660 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 7.4 M 2024-02-05T06:26:12.660 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-02-05T06:26:12.661 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-05T06:26:12.661 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-05T06:26:12.661 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-05T06:26:12.661 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-05T06:26:12.661 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-05T06:26:12.661 INFO:teuthology.orchestra.run.smithi028.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-05T06:26:12.661 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-05T06:26:12.661 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-05T06:26:12.661 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-05T06:26:12.661 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-05T06:26:12.661 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-05T06:26:12.661 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:12.661 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-02-05T06:26:12.661 INFO:teuthology.orchestra.run.smithi028.stdout:============================================================================================ 2024-02-05T06:26:12.662 INFO:teuthology.orchestra.run.smithi028.stdout:Install 12 Packages 2024-02-05T06:26:12.662 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:12.662 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 37 M 2024-02-05T06:26:12.662 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 209 M 2024-02-05T06:26:12.662 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-02-05T06:26:12.858 INFO:teuthology.orchestra.run.smithi167.stdout:cephadm-19.0.0-1241.g164d3be4.el9.noarch.rpm 2.4 MB/s | 748 kB 00:00 2024-02-05T06:26:12.859 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:26:12.859 INFO:teuthology.orchestra.run.smithi167.stdout:Total 2.4 MB/s | 748 kB 00:00 2024-02-05T06:26:12.859 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-02-05T06:26:12.864 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-02-05T06:26:12.864 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-02-05T06:26:12.870 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-02-05T06:26:12.870 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-02-05T06:26:12.943 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-02-05T06:26:13.098 INFO:teuthology.orchestra.run.smithi028.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 186 kB/s | 32 kB 00:00 2024-02-05T06:26:13.139 INFO:teuthology.orchestra.run.smithi028.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 414 kB/s | 17 kB 00:00 2024-02-05T06:26:13.181 INFO:teuthology.orchestra.run.smithi028.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 821 kB/s | 34 kB 00:00 2024-02-05T06:26:13.290 INFO:teuthology.orchestra.run.smithi028.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.2 MB/s | 3.0 MB 00:00 2024-02-05T06:26:13.349 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 1/1 2024-02-05T06:26:13.432 INFO:teuthology.orchestra.run.smithi028.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 19 MB/s | 4.7 MB 00:00 2024-02-05T06:26:13.437 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 1/1 2024-02-05T06:26:13.532 INFO:teuthology.orchestra.run.smithi028.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-02-05T06:26:13.558 INFO:teuthology.orchestra.run.smithi028.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.7 MB/s | 217 kB 00:00 2024-02-05T06:26:13.716 INFO:teuthology.orchestra.run.smithi028.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 5.0 MB/s | 802 kB 00:00 2024-02-05T06:26:13.809 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 1/1 2024-02-05T06:26:13.900 INFO:teuthology.orchestra.run.smithi028.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-02-05T06:26:13.925 INFO:teuthology.orchestra.run.smithi028.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 942 kB/s | 194 kB 00:00 2024-02-05T06:26:14.017 INFO:teuthology.orchestra.run.smithi028.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-1 6.8 MB/s | 7.4 MB 00:01 2024-02-05T06:26:14.152 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 1/1 2024-02-05T06:26:14.152 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:14.152 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-02-05T06:26:14.152 INFO:teuthology.orchestra.run.smithi167.stdout: cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T06:26:14.152 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:14.152 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-02-05T06:26:14.380 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-immutable-object-cache 2024-02-05T06:26:14.550 INFO:teuthology.orchestra.run.smithi028.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 378 kB/s | 245 kB 00:00 2024-02-05T06:26:14.555 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:26:14.555 INFO:teuthology.orchestra.run.smithi028.stdout:Total 20 MB/s | 37 MB 00:01 2024-02-05T06:26:14.569 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-02-05T06:26:14.597 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-02-05T06:26:14.597 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-02-05T06:26:14.959 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:00:55 ago on Mon 05 Feb 2024 06:25:19 AM UTC. 2024-02-05T06:26:14.966 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-02-05T06:26:14.966 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-02-05T06:26:15.121 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-02-05T06:26:15.122 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:26:15.122 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-02-05T06:26:15.122 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:26:15.122 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-02-05T06:26:15.122 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 145 k 2024-02-05T06:26:15.122 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:15.122 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-02-05T06:26:15.123 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:26:15.123 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-02-05T06:26:15.123 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:15.123 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 145 k 2024-02-05T06:26:15.123 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 436 k 2024-02-05T06:26:15.123 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-02-05T06:26:15.332 INFO:teuthology.orchestra.run.smithi167.stdout:ceph-immutable-object-cache-19.0.0-1241.g164d3b 692 kB/s | 145 kB 00:00 2024-02-05T06:26:15.333 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:26:15.333 INFO:teuthology.orchestra.run.smithi167.stdout:Total 688 kB/s | 145 kB 00:00 2024-02-05T06:26:15.333 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-02-05T06:26:15.341 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-02-05T06:26:15.341 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-02-05T06:26:15.392 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-02-05T06:26:15.393 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-02-05T06:26:15.538 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-02-05T06:26:15.592 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1241.g164d3be4. 1/1 2024-02-05T06:26:15.616 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1241.g164d3be4. 1/1 2024-02-05T06:26:15.617 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T06:26:15.617 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-05T06:26:15.617 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:15.834 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-02-05T06:26:15.908 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-05T06:26:15.938 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-05T06:26:16.037 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-05T06:26:16.083 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-05T06:26:16.269 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-05T06:26:16.298 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-05T06:26:16.378 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-05T06:26:16.649 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1241.g164d3be4. 1/1 2024-02-05T06:26:16.649 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:16.649 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-02-05T06:26:16.649 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-immutable-object-cache-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:26:16.649 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:16.649 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-02-05T06:26:16.928 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mgr 2024-02-05T06:26:17.080 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-05T06:26:17.198 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-05T06:26:17.514 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:00:58 ago on Mon 05 Feb 2024 06:25:19 AM UTC. 2024-02-05T06:26:17.618 INFO:teuthology.orchestra.run.smithi167.stdout:Package ceph-mgr-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T06:26:17.669 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-02-05T06:26:17.670 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-02-05T06:26:17.670 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-02-05T06:26:17.742 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mgr-dashboard 2024-02-05T06:26:18.310 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:00:59 ago on Mon 05 Feb 2024 06:25:19 AM UTC. 2024-02-05T06:26:18.467 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-02-05T06:26:18.468 INFO:teuthology.orchestra.run.smithi167.stdout:====================================================================================== 2024-02-05T06:26:18.468 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-02-05T06:26:18.468 INFO:teuthology.orchestra.run.smithi167.stdout:====================================================================================== 2024-02-05T06:26:18.468 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-02-05T06:26:18.468 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 1.8 M 2024-02-05T06:26:18.468 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-02-05T06:26:18.468 INFO:teuthology.orchestra.run.smithi167.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-02-05T06:26:18.469 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 28 k 2024-02-05T06:26:18.469 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 15 k 2024-02-05T06:26:18.469 INFO:teuthology.orchestra.run.smithi167.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-02-05T06:26:18.469 INFO:teuthology.orchestra.run.smithi167.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-02-05T06:26:18.469 INFO:teuthology.orchestra.run.smithi167.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-02-05T06:26:18.469 INFO:teuthology.orchestra.run.smithi167.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-02-05T06:26:18.469 INFO:teuthology.orchestra.run.smithi167.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-02-05T06:26:18.469 INFO:teuthology.orchestra.run.smithi167.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-02-05T06:26:18.469 INFO:teuthology.orchestra.run.smithi167.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-05T06:26:18.469 INFO:teuthology.orchestra.run.smithi167.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-05T06:26:18.469 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:18.469 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-02-05T06:26:18.469 INFO:teuthology.orchestra.run.smithi167.stdout:====================================================================================== 2024-02-05T06:26:18.470 INFO:teuthology.orchestra.run.smithi167.stdout:Install 12 Packages 2024-02-05T06:26:18.470 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:18.470 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 6.8 M 2024-02-05T06:26:18.470 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 26 M 2024-02-05T06:26:18.470 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-02-05T06:26:18.484 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-05T06:26:19.130 INFO:teuthology.orchestra.run.smithi167.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1241.g164 83 kB/s | 15 kB 00:00 2024-02-05T06:26:19.155 INFO:teuthology.orchestra.run.smithi167.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1241.g16 135 kB/s | 28 kB 00:00 2024-02-05T06:26:19.314 INFO:teuthology.orchestra.run.smithi167.stdout:(3/12): ceph-mgr-dashboard-19.0.0-1241.g164d3be 4.8 MB/s | 1.8 MB 00:00 2024-02-05T06:26:19.339 INFO:teuthology.orchestra.run.smithi167.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 238 kB 00:00 2024-02-05T06:26:19.389 INFO:teuthology.orchestra.run.smithi167.stdout:(5/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.9 MB/s | 1.0 MB 00:00 2024-02-05T06:26:19.523 INFO:teuthology.orchestra.run.smithi167.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 146 kB/s | 19 kB 00:00 2024-02-05T06:26:19.560 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-05T06:26:19.588 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1241.g164d3 12/12 2024-02-05T06:26:19.773 INFO:teuthology.orchestra.run.smithi167.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.2 MB/s | 548 kB 00:00 2024-02-05T06:26:20.007 INFO:teuthology.orchestra.run.smithi167.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.2 MB/s | 863 kB 00:00 2024-02-05T06:26:20.032 INFO:teuthology.orchestra.run.smithi167.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 558 kB/s | 144 kB 00:00 2024-02-05T06:26:20.083 INFO:teuthology.orchestra.run.smithi167.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.6 MB/s | 2.0 MB 00:00 2024-02-05T06:26:20.108 INFO:teuthology.orchestra.run.smithi167.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 306 kB/s | 31 kB 00:00 2024-02-05T06:26:20.258 INFO:teuthology.orchestra.run.smithi167.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 837 kB/s | 188 kB 00:00 2024-02-05T06:26:20.262 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:26:20.263 INFO:teuthology.orchestra.run.smithi167.stdout:Total 3.8 MB/s | 6.8 MB 00:01 2024-02-05T06:26:20.281 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-02-05T06:26:20.298 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-02-05T06:26:20.298 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-02-05T06:26:20.477 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-02-05T06:26:20.477 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-02-05T06:26:20.984 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-02-05T06:26:21.075 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-02-05T06:26:21.110 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-02-05T06:26:21.166 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-02-05T06:26:21.204 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-02-05T06:26:21.306 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-02-05T06:26:21.437 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-02-05T06:26:21.523 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-02-05T06:26:21.608 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-02-05T06:26:21.637 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-02-05T06:26:21.666 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1241.g164d3be4.el9 10/12 2024-02-05T06:26:21.928 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1241.g164d3be4.el 11/12 2024-02-05T06:26:21.968 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1241.g164d3be4.el9.noa 12/12 2024-02-05T06:26:22.049 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1241.g164d3 12/12 2024-02-05T06:26:22.049 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1241.g164d3 1/12 2024-02-05T06:26:22.050 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-05T06:26:22.050 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-05T06:26:22.050 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-05T06:26:22.050 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-05T06:26:22.050 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-05T06:26:22.050 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-05T06:26:22.050 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-05T06:26:22.050 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-05T06:26:22.050 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-05T06:26:22.050 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-05T06:26:22.496 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-05T06:26:22.496 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:22.496 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-02-05T06:26:22.496 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T06:26:22.496 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-05T06:26:22.497 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-05T06:26:22.497 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-05T06:26:22.497 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-05T06:26:22.497 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-05T06:26:22.497 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-05T06:26:22.497 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-05T06:26:22.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-05T06:26:22.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-05T06:26:22.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-05T06:26:22.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-05T06:26:22.497 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:22.497 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-02-05T06:26:22.788 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1241.g164d3be4.el9.noa 12/12 2024-02-05T06:26:22.788 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1241.g164d3be4.el 1/12 2024-02-05T06:26:22.788 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1241.g164d3be4.el9.noa 2/12 2024-02-05T06:26:22.789 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1241.g164d3be4.el9 3/12 2024-02-05T06:26:22.789 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-02-05T06:26:22.789 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-02-05T06:26:22.789 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-02-05T06:26:22.789 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-02-05T06:26:22.789 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-02-05T06:26:22.789 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-02-05T06:26:22.789 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-02-05T06:26:22.789 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-02-05T06:26:22.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-mgr-rook 2024-02-05T06:26:23.159 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-02-05T06:26:23.160 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:23.160 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-02-05T06:26:23.160 INFO:teuthology.orchestra.run.smithi167.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-02-05T06:26:23.160 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-grafana-dashboards-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T06:26:23.160 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-dashboard-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T06:26:23.160 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-prometheus-alerts-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T06:26:23.160 INFO:teuthology.orchestra.run.smithi167.stdout: grpc-data-1.46.7-10.el9.noarch 2024-02-05T06:26:23.160 INFO:teuthology.orchestra.run.smithi167.stdout: protobuf-3.14.0-13.el9.x86_64 2024-02-05T06:26:23.160 INFO:teuthology.orchestra.run.smithi167.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-02-05T06:26:23.160 INFO:teuthology.orchestra.run.smithi167.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-02-05T06:26:23.160 INFO:teuthology.orchestra.run.smithi167.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-02-05T06:26:23.160 INFO:teuthology.orchestra.run.smithi167.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-02-05T06:26:23.160 INFO:teuthology.orchestra.run.smithi167.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-05T06:26:23.161 INFO:teuthology.orchestra.run.smithi167.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-05T06:26:23.161 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:23.161 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-02-05T06:26:23.408 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:16 ago on Mon 05 Feb 2024 06:25:07 AM UTC. 2024-02-05T06:26:23.436 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-05T06:26:23.575 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-02-05T06:26:23.576 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================== 2024-02-05T06:26:23.577 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-02-05T06:26:23.577 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================== 2024-02-05T06:26:23.577 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-02-05T06:26:23.577 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 52 k 2024-02-05T06:26:23.577 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-02-05T06:26:23.577 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-05T06:26:23.577 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-05T06:26:23.577 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-05T06:26:23.577 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-05T06:26:23.577 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-05T06:26:23.577 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-05T06:26:23.577 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-05T06:26:23.577 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-05T06:26:23.578 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-05T06:26:23.578 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-05T06:26:23.578 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-05T06:26:23.578 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-05T06:26:23.578 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:23.578 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-02-05T06:26:23.578 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================== 2024-02-05T06:26:23.578 INFO:teuthology.orchestra.run.smithi028.stdout:Install 13 Packages 2024-02-05T06:26:23.578 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:23.579 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 2.0 M 2024-02-05T06:26:23.579 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 26 M 2024-02-05T06:26:23.579 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-02-05T06:26:23.997 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:05 ago on Mon 05 Feb 2024 06:25:19 AM UTC. 2024-02-05T06:26:24.053 INFO:teuthology.orchestra.run.smithi028.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 130 kB/s | 18 kB 00:00 2024-02-05T06:26:24.079 INFO:teuthology.orchestra.run.smithi028.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 148 kB/s | 24 kB 00:00 2024-02-05T06:26:24.146 INFO:teuthology.orchestra.run.smithi028.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 174 kB 00:00 2024-02-05T06:26:24.153 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-02-05T06:26:24.154 INFO:teuthology.orchestra.run.smithi167.stdout:============================================================================================ 2024-02-05T06:26:24.154 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-02-05T06:26:24.154 INFO:teuthology.orchestra.run.smithi167.stdout:============================================================================================ 2024-02-05T06:26:24.154 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-02-05T06:26:24.154 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 7.4 M 2024-02-05T06:26:24.155 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-02-05T06:26:24.155 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-05T06:26:24.155 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-05T06:26:24.155 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-05T06:26:24.155 INFO:teuthology.orchestra.run.smithi167.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-05T06:26:24.155 INFO:teuthology.orchestra.run.smithi167.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-05T06:26:24.155 INFO:teuthology.orchestra.run.smithi167.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-05T06:26:24.155 INFO:teuthology.orchestra.run.smithi167.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-05T06:26:24.155 INFO:teuthology.orchestra.run.smithi167.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-05T06:26:24.155 INFO:teuthology.orchestra.run.smithi167.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-05T06:26:24.155 INFO:teuthology.orchestra.run.smithi167.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-05T06:26:24.155 INFO:teuthology.orchestra.run.smithi167.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-05T06:26:24.155 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:24.155 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-02-05T06:26:24.156 INFO:teuthology.orchestra.run.smithi167.stdout:============================================================================================ 2024-02-05T06:26:24.156 INFO:teuthology.orchestra.run.smithi167.stdout:Install 12 Packages 2024-02-05T06:26:24.156 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:24.156 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 37 M 2024-02-05T06:26:24.156 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 209 M 2024-02-05T06:26:24.156 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-02-05T06:26:24.171 INFO:teuthology.orchestra.run.smithi028.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2024-02-05T06:26:24.196 INFO:teuthology.orchestra.run.smithi028.stdout:(5/13): ceph-mgr-rook-19.0.0-1241.g164d3be4.el9 185 kB/s | 52 kB 00:00 2024-02-05T06:26:24.221 INFO:teuthology.orchestra.run.smithi028.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 211 kB 00:00 2024-02-05T06:26:24.247 INFO:teuthology.orchestra.run.smithi028.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 565 kB/s | 43 kB 00:00 2024-02-05T06:26:24.305 INFO:teuthology.orchestra.run.smithi028.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 180 kB/s | 15 kB 00:00 2024-02-05T06:26:24.330 INFO:teuthology.orchestra.run.smithi028.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 240 kB/s | 32 kB 00:00 2024-02-05T06:26:24.406 INFO:teuthology.orchestra.run.smithi028.stdout:(10/13): python3-google-auth-2.27.0-1.el9.noarc 1.2 MB/s | 203 kB 00:00 2024-02-05T06:26:24.431 INFO:teuthology.orchestra.run.smithi028.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 591 kB/s | 59 kB 00:00 2024-02-05T06:26:24.481 INFO:teuthology.orchestra.run.smithi028.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-02-05T06:26:24.515 INFO:teuthology.orchestra.run.smithi028.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 4.9 MB/s | 1.0 MB 00:00 2024-02-05T06:26:24.516 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:26:24.516 INFO:teuthology.orchestra.run.smithi028.stdout:Total 2.2 MB/s | 2.0 MB 00:00 2024-02-05T06:26:24.517 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-02-05T06:26:24.531 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-02-05T06:26:24.531 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-02-05T06:26:24.581 INFO:teuthology.orchestra.run.smithi167.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 207 kB/s | 32 kB 00:00 2024-02-05T06:26:24.623 INFO:teuthology.orchestra.run.smithi167.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 422 kB/s | 17 kB 00:00 2024-02-05T06:26:24.637 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-02-05T06:26:24.638 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-02-05T06:26:24.673 INFO:teuthology.orchestra.run.smithi167.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 696 kB/s | 34 kB 00:00 2024-02-05T06:26:24.724 INFO:teuthology.orchestra.run.smithi167.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-02-05T06:26:24.941 INFO:teuthology.orchestra.run.smithi167.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2024-02-05T06:26:25.016 INFO:teuthology.orchestra.run.smithi167.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 4.8 MB 00:00 2024-02-05T06:26:25.041 INFO:teuthology.orchestra.run.smithi167.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-02-05T06:26:25.182 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-02-05T06:26:25.362 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-05T06:26:25.383 INFO:teuthology.orchestra.run.smithi167.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 802 kB 00:00 2024-02-05T06:26:25.417 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-05T06:26:25.473 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-05T06:26:25.509 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-05T06:26:25.534 INFO:teuthology.orchestra.run.smithi167.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 30 MB/s | 16 MB 00:00 2024-02-05T06:26:25.550 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-05T06:26:25.559 INFO:teuthology.orchestra.run.smithi167.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 194 kB 00:00 2024-02-05T06:26:25.648 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-05T06:26:25.765 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-05T06:26:25.819 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-05T06:26:26.068 INFO:teuthology.orchestra.run.smithi167.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 460 kB/s | 245 kB 00:00 2024-02-05T06:26:26.577 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-05T06:26:26.632 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-05T06:26:26.666 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-05T06:26:26.714 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-05T06:26:26.743 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-rook-2:19.0.0-1241.g164d3be4.el9.noarch 13/13 2024-02-05T06:26:27.598 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1241.g164d3be4.el9.noarch 13/13 2024-02-05T06:26:27.598 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1241.g164d3be4.el9.noarch 1/13 2024-02-05T06:26:27.598 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-05T06:26:27.598 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-05T06:26:27.598 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-05T06:26:27.599 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-05T06:26:27.599 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-05T06:26:27.599 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-05T06:26:27.599 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-05T06:26:27.599 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-05T06:26:27.599 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-05T06:26:27.599 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-05T06:26:27.599 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-05T06:26:27.893 INFO:teuthology.orchestra.run.smithi167.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 2.1 MB/s | 7.4 MB 00:03 2024-02-05T06:26:27.896 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:26:27.896 INFO:teuthology.orchestra.run.smithi167.stdout:Total 9.9 MB/s | 37 MB 00:03 2024-02-05T06:26:27.910 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-02-05T06:26:27.935 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-02-05T06:26:27.936 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-02-05T06:26:27.977 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-05T06:26:27.977 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:27.977 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-02-05T06:26:27.977 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T06:26:27.977 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-05T06:26:27.977 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-05T06:26:27.977 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-05T06:26:27.977 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-05T06:26:27.977 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-05T06:26:27.977 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-05T06:26:27.977 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-05T06:26:27.978 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-05T06:26:27.978 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-05T06:26:27.978 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-05T06:26:27.978 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-05T06:26:27.978 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-05T06:26:27.978 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:27.978 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-02-05T06:26:28.220 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-mgr-cephadm 2024-02-05T06:26:28.287 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-02-05T06:26:28.288 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-02-05T06:26:28.794 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:21 ago on Mon 05 Feb 2024 06:25:07 AM UTC. 2024-02-05T06:26:28.960 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-02-05T06:26:28.961 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================== 2024-02-05T06:26:28.961 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-02-05T06:26:28.961 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================== 2024-02-05T06:26:28.961 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-02-05T06:26:28.961 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 138 k 2024-02-05T06:26:28.961 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-02-05T06:26:28.961 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-02-05T06:26:28.961 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-05T06:26:28.961 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-02-05T06:26:28.961 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-05T06:26:28.961 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-05T06:26:28.961 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:28.962 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-02-05T06:26:28.962 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================== 2024-02-05T06:26:28.962 INFO:teuthology.orchestra.run.smithi028.stdout:Install 6 Packages 2024-02-05T06:26:28.962 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:28.962 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 6.8 M 2024-02-05T06:26:28.962 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 33 M 2024-02-05T06:26:28.962 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-02-05T06:26:29.014 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-02-05T06:26:29.091 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-05T06:26:29.122 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-05T06:26:29.222 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-05T06:26:29.232 INFO:teuthology.orchestra.run.smithi028.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.1 MB/s | 228 kB 00:00 2024-02-05T06:26:29.268 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-05T06:26:29.416 INFO:teuthology.orchestra.run.smithi028.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 5.8 MB 00:00 2024-02-05T06:26:29.444 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-05T06:26:29.475 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-05T06:26:29.483 INFO:teuthology.orchestra.run.smithi028.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1241.g164d3be4.e 307 kB/s | 138 kB 00:00 2024-02-05T06:26:29.516 INFO:teuthology.orchestra.run.smithi028.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 576 kB/s | 58 kB 00:00 2024-02-05T06:26:29.552 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-05T06:26:29.592 INFO:teuthology.orchestra.run.smithi028.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 470 kB/s | 51 kB 00:00 2024-02-05T06:26:29.625 INFO:teuthology.orchestra.run.smithi028.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.4 MB/s | 548 kB 00:00 2024-02-05T06:26:29.627 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:26:29.627 INFO:teuthology.orchestra.run.smithi028.stdout:Total 10 MB/s | 6.8 MB 00:00 2024-02-05T06:26:29.628 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-02-05T06:26:29.637 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-02-05T06:26:29.637 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-02-05T06:26:29.731 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-02-05T06:26:29.731 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-02-05T06:26:30.030 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-02-05T06:26:30.196 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-05T06:26:30.203 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-05T06:26:30.252 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-02-05T06:26:30.323 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-05T06:26:30.795 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-05T06:26:30.890 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-05T06:26:30.951 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2024-02-05T06:26:30.988 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 6/6 2024-02-05T06:26:31.569 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-05T06:26:31.769 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 6/6 2024-02-05T06:26:31.769 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 1/6 2024-02-05T06:26:31.769 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-05T06:26:31.769 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2024-02-05T06:26:31.770 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-02-05T06:26:31.770 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-05T06:26:32.082 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-05T06:26:32.082 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:32.082 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-02-05T06:26:32.082 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T06:26:32.083 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-02-05T06:26:32.083 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-05T06:26:32.083 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-02-05T06:26:32.083 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-05T06:26:32.083 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-05T06:26:32.083 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:32.083 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-02-05T06:26:32.330 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-fuse 2024-02-05T06:26:32.598 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-05T06:26:32.626 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1241.g164d3 12/12 2024-02-05T06:26:32.906 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:25 ago on Mon 05 Feb 2024 06:25:07 AM UTC. 2024-02-05T06:26:33.070 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-02-05T06:26:33.071 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:26:33.071 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-02-05T06:26:33.071 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:26:33.071 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-02-05T06:26:33.071 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 885 k 2024-02-05T06:26:33.071 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-02-05T06:26:33.071 INFO:teuthology.orchestra.run.smithi028.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-05T06:26:33.071 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:33.071 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-02-05T06:26:33.071 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:26:33.072 INFO:teuthology.orchestra.run.smithi028.stdout:Install 2 Packages 2024-02-05T06:26:33.072 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:33.072 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 964 k 2024-02-05T06:26:33.072 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 2.8 M 2024-02-05T06:26:33.072 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-02-05T06:26:33.301 INFO:teuthology.orchestra.run.smithi028.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 343 kB/s | 79 kB 00:00 2024-02-05T06:26:33.860 INFO:teuthology.orchestra.run.smithi028.stdout:(2/2): ceph-fuse-19.0.0-1241.g164d3be4.el9.x86_ 1.1 MB/s | 885 kB 00:00 2024-02-05T06:26:33.861 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:26:33.861 INFO:teuthology.orchestra.run.smithi028.stdout:Total 1.2 MB/s | 964 kB 00:00 2024-02-05T06:26:33.862 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-02-05T06:26:33.885 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-02-05T06:26:33.885 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-02-05T06:26:33.965 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-02-05T06:26:33.965 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-02-05T06:26:34.156 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-02-05T06:26:34.290 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-05T06:26:34.362 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 2/2 2024-02-05T06:26:35.030 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 2/2 2024-02-05T06:26:35.030 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 1/2 2024-02-05T06:26:35.249 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1241.g164d3 12/12 2024-02-05T06:26:35.249 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1241.g164d3 1/12 2024-02-05T06:26:35.249 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-05T06:26:35.249 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-05T06:26:35.249 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-05T06:26:35.249 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-05T06:26:35.249 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-05T06:26:35.249 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-05T06:26:35.249 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-05T06:26:35.250 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-05T06:26:35.250 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-05T06:26:35.250 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-05T06:26:35.365 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-05T06:26:35.365 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:35.365 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-02-05T06:26:35.366 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-05T06:26:35.366 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:35.366 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-02-05T06:26:35.630 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-volume 2024-02-05T06:26:35.672 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-05T06:26:35.673 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:35.673 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-02-05T06:26:35.673 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T06:26:35.673 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-05T06:26:35.673 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-05T06:26:35.673 INFO:teuthology.orchestra.run.smithi167.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-05T06:26:35.673 INFO:teuthology.orchestra.run.smithi167.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-05T06:26:35.673 INFO:teuthology.orchestra.run.smithi167.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-05T06:26:35.673 INFO:teuthology.orchestra.run.smithi167.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-05T06:26:35.673 INFO:teuthology.orchestra.run.smithi167.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-05T06:26:35.673 INFO:teuthology.orchestra.run.smithi167.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-05T06:26:35.673 INFO:teuthology.orchestra.run.smithi167.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-05T06:26:35.673 INFO:teuthology.orchestra.run.smithi167.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-05T06:26:35.673 INFO:teuthology.orchestra.run.smithi167.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-05T06:26:35.674 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:35.674 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-02-05T06:26:36.036 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mgr-rook 2024-02-05T06:26:36.217 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:29 ago on Mon 05 Feb 2024 06:25:07 AM UTC. 2024-02-05T06:26:36.381 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-02-05T06:26:36.381 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:26:36.381 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-02-05T06:26:36.381 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:26:36.381 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-02-05T06:26:36.381 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 263 k 2024-02-05T06:26:36.382 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-02-05T06:26:36.382 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-02-05T06:26:36.382 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:36.382 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-02-05T06:26:36.382 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:26:36.382 INFO:teuthology.orchestra.run.smithi028.stdout:Install 2 Packages 2024-02-05T06:26:36.382 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:36.382 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 332 k 2024-02-05T06:26:36.383 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 1.5 M 2024-02-05T06:26:36.383 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-02-05T06:26:36.522 INFO:teuthology.orchestra.run.smithi028.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 496 kB/s | 69 kB 00:00 2024-02-05T06:26:36.631 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:17 ago on Mon 05 Feb 2024 06:25:19 AM UTC. 2024-02-05T06:26:36.763 INFO:teuthology.orchestra.run.smithi028.stdout:(2/2): ceph-volume-19.0.0-1241.g164d3be4.el9.no 692 kB/s | 263 kB 00:00 2024-02-05T06:26:36.764 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:26:36.765 INFO:teuthology.orchestra.run.smithi028.stdout:Total 867 kB/s | 332 kB 00:00 2024-02-05T06:26:36.765 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-02-05T06:26:36.774 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-02-05T06:26:36.775 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-02-05T06:26:36.798 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-02-05T06:26:36.799 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================== 2024-02-05T06:26:36.799 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-02-05T06:26:36.799 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================== 2024-02-05T06:26:36.799 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-02-05T06:26:36.799 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-rook noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 52 k 2024-02-05T06:26:36.799 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-02-05T06:26:36.799 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-05T06:26:36.800 INFO:teuthology.orchestra.run.smithi167.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-05T06:26:36.800 INFO:teuthology.orchestra.run.smithi167.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-05T06:26:36.800 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-05T06:26:36.800 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-05T06:26:36.800 INFO:teuthology.orchestra.run.smithi167.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-05T06:26:36.800 INFO:teuthology.orchestra.run.smithi167.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-05T06:26:36.800 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-05T06:26:36.800 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-05T06:26:36.800 INFO:teuthology.orchestra.run.smithi167.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-05T06:26:36.800 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-05T06:26:36.800 INFO:teuthology.orchestra.run.smithi167.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-05T06:26:36.800 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:36.800 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-02-05T06:26:36.801 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================== 2024-02-05T06:26:36.801 INFO:teuthology.orchestra.run.smithi167.stdout:Install 13 Packages 2024-02-05T06:26:36.801 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:36.801 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 2.0 M 2024-02-05T06:26:36.801 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 26 M 2024-02-05T06:26:36.801 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-02-05T06:26:36.822 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-02-05T06:26:36.823 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-02-05T06:26:36.979 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-02-05T06:26:37.077 INFO:teuthology.orchestra.run.smithi167.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 179 kB/s | 24 kB 00:00 2024-02-05T06:26:37.094 INFO:teuthology.orchestra.run.smithi167.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 118 kB/s | 18 kB 00:00 2024-02-05T06:26:37.153 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-02-05T06:26:37.178 INFO:teuthology.orchestra.run.smithi167.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2024-02-05T06:26:37.182 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-volume-2:19.0.0-1241.g164d3be4.el9.noarch 2/2 2024-02-05T06:26:37.197 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-volume-2:19.0.0-1241.g164d3be4.el9.noarch 2/2 2024-02-05T06:26:37.198 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T06:26:37.198 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-02-05T06:26:37.198 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:37.203 INFO:teuthology.orchestra.run.smithi167.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 132 kB 00:00 2024-02-05T06:26:37.229 INFO:teuthology.orchestra.run.smithi167.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-02-05T06:26:37.246 INFO:teuthology.orchestra.run.smithi167.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-02-05T06:26:37.263 INFO:teuthology.orchestra.run.smithi167.stdout:(7/13): ceph-mgr-rook-19.0.0-1241.g164d3be4.el9 162 kB/s | 52 kB 00:00 2024-02-05T06:26:37.363 INFO:teuthology.orchestra.run.smithi167.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 240 kB/s | 32 kB 00:00 2024-02-05T06:26:37.397 INFO:teuthology.orchestra.run.smithi167.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 99 kB/s | 15 kB 00:00 2024-02-05T06:26:37.505 INFO:teuthology.orchestra.run.smithi167.stdout:(10/13): python3-google-auth-2.27.0-1.el9.noarc 1.0 MB/s | 203 kB 00:00 2024-02-05T06:26:37.539 INFO:teuthology.orchestra.run.smithi167.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 416 kB/s | 59 kB 00:00 2024-02-05T06:26:37.647 INFO:teuthology.orchestra.run.smithi167.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 635 kB/s | 90 kB 00:00 2024-02-05T06:26:37.689 INFO:teuthology.orchestra.run.smithi167.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.1 MB/s | 1.0 MB 00:00 2024-02-05T06:26:37.693 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:26:37.693 INFO:teuthology.orchestra.run.smithi167.stdout:Total 2.3 MB/s | 2.0 MB 00:00 2024-02-05T06:26:37.694 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-02-05T06:26:37.707 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-02-05T06:26:37.707 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-02-05T06:26:37.807 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-02-05T06:26:37.807 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-02-05T06:26:37.903 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-volume-2:19.0.0-1241.g164d3be4.el9.noarch 1/2 2024-02-05T06:26:38.252 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-02-05T06:26:38.252 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:38.253 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-02-05T06:26:38.253 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T06:26:38.253 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging-20.9-5.el9.noarch 2024-02-05T06:26:38.253 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:38.253 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-02-05T06:26:38.330 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-02-05T06:26:38.487 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-05T06:26:38.549 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-05T06:26:38.553 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install librados-devel 2024-02-05T06:26:38.604 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-05T06:26:38.633 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-05T06:26:38.674 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-05T06:26:38.768 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-05T06:26:38.879 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-05T06:26:38.937 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-05T06:26:39.131 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:32 ago on Mon 05 Feb 2024 06:25:07 AM UTC. 2024-02-05T06:26:39.294 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-02-05T06:26:39.294 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:26:39.294 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repo Size 2024-02-05T06:26:39.294 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:26:39.295 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-02-05T06:26:39.295 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 127 k 2024-02-05T06:26:39.295 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:39.295 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-02-05T06:26:39.295 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:26:39.295 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-02-05T06:26:39.295 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:39.295 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 127 k 2024-02-05T06:26:39.295 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 456 k 2024-02-05T06:26:39.295 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-02-05T06:26:39.654 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-05T06:26:39.709 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-05T06:26:39.719 INFO:teuthology.orchestra.run.smithi028.stdout:librados-devel-19.0.0-1241.g164d3be4.el9.x86_64 299 kB/s | 127 kB 00:00 2024-02-05T06:26:39.719 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:26:39.719 INFO:teuthology.orchestra.run.smithi028.stdout:Total 298 kB/s | 127 kB 00:00 2024-02-05T06:26:39.720 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-02-05T06:26:39.726 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-02-05T06:26:39.726 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-02-05T06:26:39.743 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-05T06:26:39.785 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-02-05T06:26:39.785 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-02-05T06:26:39.791 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-05T06:26:39.813 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-rook-2:19.0.0-1241.g164d3be4.el9.noarch 13/13 2024-02-05T06:26:39.941 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-02-05T06:26:40.047 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librados-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T06:26:40.573 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T06:26:40.673 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1241.g164d3be4.el9.noarch 13/13 2024-02-05T06:26:40.673 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1241.g164d3be4.el9.noarch 1/13 2024-02-05T06:26:40.673 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-05T06:26:40.673 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-05T06:26:40.673 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-05T06:26:40.673 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-05T06:26:40.673 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-05T06:26:40.673 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-05T06:26:40.673 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-05T06:26:40.673 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-05T06:26:40.674 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-05T06:26:40.674 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-05T06:26:40.674 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-05T06:26:40.877 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T06:26:40.877 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:40.877 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-02-05T06:26:40.878 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:26:40.878 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:40.878 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-02-05T06:26:41.053 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-05T06:26:41.053 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:41.053 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-02-05T06:26:41.053 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-rook-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T06:26:41.053 INFO:teuthology.orchestra.run.smithi167.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-05T06:26:41.053 INFO:teuthology.orchestra.run.smithi167.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-05T06:26:41.053 INFO:teuthology.orchestra.run.smithi167.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-05T06:26:41.053 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-05T06:26:41.053 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-05T06:26:41.054 INFO:teuthology.orchestra.run.smithi167.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-05T06:26:41.054 INFO:teuthology.orchestra.run.smithi167.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-05T06:26:41.054 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-05T06:26:41.054 INFO:teuthology.orchestra.run.smithi167.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-05T06:26:41.054 INFO:teuthology.orchestra.run.smithi167.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-05T06:26:41.054 INFO:teuthology.orchestra.run.smithi167.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-05T06:26:41.054 INFO:teuthology.orchestra.run.smithi167.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-05T06:26:41.054 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:41.054 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-02-05T06:26:41.146 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install libcephfs2 2024-02-05T06:26:41.326 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-mgr-cephadm 2024-02-05T06:26:41.732 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:34 ago on Mon 05 Feb 2024 06:25:07 AM UTC. 2024-02-05T06:26:41.840 INFO:teuthology.orchestra.run.smithi028.stdout:Package libcephfs2-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T06:26:41.893 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:22 ago on Mon 05 Feb 2024 06:25:19 AM UTC. 2024-02-05T06:26:41.895 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-02-05T06:26:41.896 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-02-05T06:26:41.896 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-02-05T06:26:41.960 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install libcephfs-devel 2024-02-05T06:26:42.049 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-02-05T06:26:42.050 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================== 2024-02-05T06:26:42.050 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-02-05T06:26:42.050 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================== 2024-02-05T06:26:42.050 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-02-05T06:26:42.051 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 138 k 2024-02-05T06:26:42.051 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-02-05T06:26:42.051 INFO:teuthology.orchestra.run.smithi167.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-02-05T06:26:42.051 INFO:teuthology.orchestra.run.smithi167.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-05T06:26:42.051 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-02-05T06:26:42.051 INFO:teuthology.orchestra.run.smithi167.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-05T06:26:42.051 INFO:teuthology.orchestra.run.smithi167.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-05T06:26:42.051 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:42.051 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-02-05T06:26:42.051 INFO:teuthology.orchestra.run.smithi167.stdout:======================================================================================== 2024-02-05T06:26:42.051 INFO:teuthology.orchestra.run.smithi167.stdout:Install 6 Packages 2024-02-05T06:26:42.051 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:42.052 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 6.8 M 2024-02-05T06:26:42.052 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 33 M 2024-02-05T06:26:42.052 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-02-05T06:26:42.395 INFO:teuthology.orchestra.run.smithi167.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.1 MB/s | 228 kB 00:00 2024-02-05T06:26:42.479 INFO:teuthology.orchestra.run.smithi167.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1241.g164d3be4.e 491 kB/s | 138 kB 00:00 2024-02-05T06:26:42.535 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:35 ago on Mon 05 Feb 2024 06:25:07 AM UTC. 2024-02-05T06:26:42.571 INFO:teuthology.orchestra.run.smithi167.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 5.8 MB 00:00 2024-02-05T06:26:42.696 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-02-05T06:26:42.697 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:26:42.697 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repo Size 2024-02-05T06:26:42.697 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:26:42.697 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-02-05T06:26:42.697 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 31 k 2024-02-05T06:26:42.697 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:42.697 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-02-05T06:26:42.698 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:26:42.698 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-02-05T06:26:42.698 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:42.698 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 31 k 2024-02-05T06:26:42.698 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 140 k 2024-02-05T06:26:42.698 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-02-05T06:26:42.813 INFO:teuthology.orchestra.run.smithi167.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 174 kB/s | 58 kB 00:00 2024-02-05T06:26:42.888 INFO:teuthology.orchestra.run.smithi167.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 161 kB/s | 51 kB 00:00 2024-02-05T06:26:42.938 INFO:teuthology.orchestra.run.smithi167.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.0 MB/s | 548 kB 00:00 2024-02-05T06:26:42.940 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:26:42.940 INFO:teuthology.orchestra.run.smithi167.stdout:Total 7.6 MB/s | 6.8 MB 00:00 2024-02-05T06:26:42.940 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-02-05T06:26:42.946 INFO:teuthology.orchestra.run.smithi028.stdout:libcephfs-devel-19.0.0-1241.g164d3be4.el9.x86_6 126 kB/s | 31 kB 00:00 2024-02-05T06:26:42.947 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:26:42.947 INFO:teuthology.orchestra.run.smithi028.stdout:Total 125 kB/s | 31 kB 00:00 2024-02-05T06:26:42.947 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-02-05T06:26:42.949 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-02-05T06:26:42.949 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-02-05T06:26:42.951 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-02-05T06:26:42.951 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-02-05T06:26:42.982 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-02-05T06:26:42.982 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-02-05T06:26:43.040 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-02-05T06:26:43.040 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-02-05T06:26:43.094 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-02-05T06:26:43.223 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T06:26:43.351 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-02-05T06:26:43.520 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-05T06:26:43.574 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-02-05T06:26:43.606 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T06:26:43.864 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T06:26:43.865 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:43.865 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-02-05T06:26:43.865 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:26:43.865 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:43.865 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-02-05T06:26:44.056 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install librados2 2024-02-05T06:26:44.088 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-05T06:26:44.176 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-05T06:26:44.238 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2024-02-05T06:26:44.261 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 6/6 2024-02-05T06:26:44.635 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:37 ago on Mon 05 Feb 2024 06:25:07 AM UTC. 2024-02-05T06:26:44.743 INFO:teuthology.orchestra.run.smithi028.stdout:Package librados2-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T06:26:44.800 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-02-05T06:26:44.801 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-02-05T06:26:44.801 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-02-05T06:26:44.863 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install librbd1 2024-02-05T06:26:45.100 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 6/6 2024-02-05T06:26:45.100 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 1/6 2024-02-05T06:26:45.100 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-05T06:26:45.100 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2024-02-05T06:26:45.101 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-02-05T06:26:45.101 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-05T06:26:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-05T06:26:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:45.417 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-02-05T06:26:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-mgr-cephadm-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T06:26:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-02-05T06:26:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-05T06:26:45.417 INFO:teuthology.orchestra.run.smithi167.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-02-05T06:26:45.418 INFO:teuthology.orchestra.run.smithi167.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-05T06:26:45.418 INFO:teuthology.orchestra.run.smithi167.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-05T06:26:45.418 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:45.418 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-02-05T06:26:45.434 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:38 ago on Mon 05 Feb 2024 06:25:07 AM UTC. 2024-02-05T06:26:45.543 INFO:teuthology.orchestra.run.smithi028.stdout:Package librbd1-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T06:26:45.598 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-02-05T06:26:45.600 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-02-05T06:26:45.600 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-02-05T06:26:45.657 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-fuse 2024-02-05T06:26:45.662 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install python3-rados 2024-02-05T06:26:46.225 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:27 ago on Mon 05 Feb 2024 06:25:19 AM UTC. 2024-02-05T06:26:46.240 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:39 ago on Mon 05 Feb 2024 06:25:07 AM UTC. 2024-02-05T06:26:46.348 INFO:teuthology.orchestra.run.smithi028.stdout:Package python3-rados-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T06:26:46.382 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-02-05T06:26:46.383 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:26:46.383 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-02-05T06:26:46.383 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:26:46.383 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-02-05T06:26:46.383 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-fuse x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 885 k 2024-02-05T06:26:46.383 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-02-05T06:26:46.383 INFO:teuthology.orchestra.run.smithi167.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-05T06:26:46.383 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:46.383 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-02-05T06:26:46.384 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:26:46.384 INFO:teuthology.orchestra.run.smithi167.stdout:Install 2 Packages 2024-02-05T06:26:46.384 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:46.384 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 964 k 2024-02-05T06:26:46.384 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 2.8 M 2024-02-05T06:26:46.384 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-02-05T06:26:46.403 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-02-05T06:26:46.404 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-02-05T06:26:46.404 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-02-05T06:26:46.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install python3-rgw 2024-02-05T06:26:46.533 INFO:teuthology.orchestra.run.smithi167.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 532 kB/s | 79 kB 00:00 2024-02-05T06:26:47.033 INFO:teuthology.orchestra.run.smithi167.stdout:(2/2): ceph-fuse-19.0.0-1241.g164d3be4.el9.x86_ 1.3 MB/s | 885 kB 00:00 2024-02-05T06:26:47.034 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:26:47.035 INFO:teuthology.orchestra.run.smithi167.stdout:Total 1.4 MB/s | 964 kB 00:00 2024-02-05T06:26:47.035 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-02-05T06:26:47.047 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:40 ago on Mon 05 Feb 2024 06:25:07 AM UTC. 2024-02-05T06:26:47.055 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-02-05T06:26:47.055 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-02-05T06:26:47.132 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-02-05T06:26:47.132 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-02-05T06:26:47.154 INFO:teuthology.orchestra.run.smithi028.stdout:Package python3-rgw-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T06:26:47.209 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-02-05T06:26:47.210 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-02-05T06:26:47.211 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-02-05T06:26:47.266 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install python3-cephfs 2024-02-05T06:26:47.321 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-02-05T06:26:47.467 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-05T06:26:47.545 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 2/2 2024-02-05T06:26:47.837 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:40 ago on Mon 05 Feb 2024 06:25:07 AM UTC. 2024-02-05T06:26:47.943 INFO:teuthology.orchestra.run.smithi028.stdout:Package python3-cephfs-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T06:26:47.999 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-02-05T06:26:48.000 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-02-05T06:26:48.000 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-02-05T06:26:48.061 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install python3-rbd 2024-02-05T06:26:48.300 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 2/2 2024-02-05T06:26:48.300 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 1/2 2024-02-05T06:26:48.625 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-05T06:26:48.625 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:48.625 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-02-05T06:26:48.625 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-05T06:26:48.625 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:48.625 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-02-05T06:26:48.638 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:41 ago on Mon 05 Feb 2024 06:25:07 AM UTC. 2024-02-05T06:26:48.745 INFO:teuthology.orchestra.run.smithi028.stdout:Package python3-rbd-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T06:26:48.800 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-02-05T06:26:48.801 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-02-05T06:26:48.801 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-02-05T06:26:48.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install rbd-fuse 2024-02-05T06:26:48.905 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install ceph-volume 2024-02-05T06:26:49.443 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:42 ago on Mon 05 Feb 2024 06:25:07 AM UTC. 2024-02-05T06:26:49.496 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:30 ago on Mon 05 Feb 2024 06:25:19 AM UTC. 2024-02-05T06:26:49.609 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-02-05T06:26:49.610 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:26:49.610 INFO:teuthology.orchestra.run.smithi028.stdout: Package Architecture Version Repository Size 2024-02-05T06:26:49.610 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:26:49.610 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-02-05T06:26:49.610 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 85 k 2024-02-05T06:26:49.610 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:49.610 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-02-05T06:26:49.610 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:26:49.610 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-02-05T06:26:49.610 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:49.610 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 85 k 2024-02-05T06:26:49.610 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 231 k 2024-02-05T06:26:49.610 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-02-05T06:26:49.652 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-02-05T06:26:49.652 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:26:49.653 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repository Size 2024-02-05T06:26:49.653 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:26:49.653 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-02-05T06:26:49.653 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-volume noarch 2:19.0.0-1241.g164d3be4.el9 ceph-noarch 263 k 2024-02-05T06:26:49.653 INFO:teuthology.orchestra.run.smithi167.stdout:Installing dependencies: 2024-02-05T06:26:49.653 INFO:teuthology.orchestra.run.smithi167.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-02-05T06:26:49.653 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:49.653 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-02-05T06:26:49.653 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:26:49.653 INFO:teuthology.orchestra.run.smithi167.stdout:Install 2 Packages 2024-02-05T06:26:49.653 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:49.654 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 332 k 2024-02-05T06:26:49.654 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 1.5 M 2024-02-05T06:26:49.654 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-02-05T06:26:49.806 INFO:teuthology.orchestra.run.smithi167.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 452 kB/s | 69 kB 00:00 2024-02-05T06:26:49.941 INFO:teuthology.orchestra.run.smithi028.stdout:rbd-fuse-19.0.0-1241.g164d3be4.el9.x86_64.rpm 258 kB/s | 85 kB 00:00 2024-02-05T06:26:49.942 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:26:49.942 INFO:teuthology.orchestra.run.smithi028.stdout:Total 257 kB/s | 85 kB 00:00 2024-02-05T06:26:49.942 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-02-05T06:26:49.950 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-02-05T06:26:49.950 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-02-05T06:26:50.007 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-02-05T06:26:50.008 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-02-05T06:26:50.014 INFO:teuthology.orchestra.run.smithi167.stdout:(2/2): ceph-volume-19.0.0-1241.g164d3be4.el9.no 730 kB/s | 263 kB 00:00 2024-02-05T06:26:50.016 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:26:50.016 INFO:teuthology.orchestra.run.smithi167.stdout:Total 915 kB/s | 332 kB 00:00 2024-02-05T06:26:50.016 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-02-05T06:26:50.025 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-02-05T06:26:50.025 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-02-05T06:26:50.066 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-02-05T06:26:50.066 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-02-05T06:26:50.148 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-02-05T06:26:50.220 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-02-05T06:26:50.234 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T06:26:50.404 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-02-05T06:26:50.442 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : ceph-volume-2:19.0.0-1241.g164d3be4.el9.noarch 2/2 2024-02-05T06:26:50.459 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: ceph-volume-2:19.0.0-1241.g164d3be4.el9.noarch 2/2 2024-02-05T06:26:50.459 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T06:26:50.459 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-02-05T06:26:50.459 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:50.647 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T06:26:50.979 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T06:26:50.980 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:50.980 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-02-05T06:26:50.980 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:26:50.980 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:50.980 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-02-05T06:26:51.162 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : ceph-volume-2:19.0.0-1241.g164d3be4.el9.noarch 1/2 2024-02-05T06:26:51.236 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install rbd-mirror 2024-02-05T06:26:51.516 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-02-05T06:26:51.516 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:51.516 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-02-05T06:26:51.516 INFO:teuthology.orchestra.run.smithi167.stdout: ceph-volume-2:19.0.0-1241.g164d3be4.el9.noarch 2024-02-05T06:26:51.516 INFO:teuthology.orchestra.run.smithi167.stdout: python3-packaging-20.9-5.el9.noarch 2024-02-05T06:26:51.516 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:51.516 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-02-05T06:26:51.822 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:44 ago on Mon 05 Feb 2024 06:25:07 AM UTC. 2024-02-05T06:26:51.856 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install librados-devel 2024-02-05T06:26:51.984 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-02-05T06:26:51.985 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:26:51.985 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repo Size 2024-02-05T06:26:51.985 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:26:51.985 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-02-05T06:26:51.985 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 3.1 M 2024-02-05T06:26:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:51.986 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-02-05T06:26:51.986 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:26:51.986 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-02-05T06:26:51.986 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:51.986 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 3.1 M 2024-02-05T06:26:51.986 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 13 M 2024-02-05T06:26:51.986 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-02-05T06:26:52.443 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:33 ago on Mon 05 Feb 2024 06:25:19 AM UTC. 2024-02-05T06:26:52.602 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-02-05T06:26:52.603 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:26:52.603 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-02-05T06:26:52.603 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:26:52.603 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-02-05T06:26:52.603 INFO:teuthology.orchestra.run.smithi167.stdout: librados-devel x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 127 k 2024-02-05T06:26:52.603 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:52.603 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-02-05T06:26:52.603 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:26:52.603 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-02-05T06:26:52.603 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:52.604 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 127 k 2024-02-05T06:26:52.604 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 456 k 2024-02-05T06:26:52.604 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-02-05T06:26:52.964 INFO:teuthology.orchestra.run.smithi167.stdout:librados-devel-19.0.0-1241.g164d3be4.el9.x86_64 352 kB/s | 127 kB 00:00 2024-02-05T06:26:52.965 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:26:52.965 INFO:teuthology.orchestra.run.smithi167.stdout:Total 350 kB/s | 127 kB 00:00 2024-02-05T06:26:52.965 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-02-05T06:26:52.970 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-02-05T06:26:52.971 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-02-05T06:26:53.026 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-02-05T06:26:53.026 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-02-05T06:26:53.170 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-02-05T06:26:53.263 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : librados-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T06:26:53.318 INFO:teuthology.orchestra.run.smithi028.stdout:rbd-mirror-19.0.0-1241.g164d3be4.el9.x86_64.rpm 2.3 MB/s | 3.1 MB 00:01 2024-02-05T06:26:53.319 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:26:53.319 INFO:teuthology.orchestra.run.smithi028.stdout:Total 2.3 MB/s | 3.1 MB 00:01 2024-02-05T06:26:53.320 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-02-05T06:26:53.332 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-02-05T06:26:53.332 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-02-05T06:26:53.414 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-02-05T06:26:53.414 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-02-05T06:26:53.721 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: librados-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T06:26:53.858 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-02-05T06:26:53.919 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-mirror-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T06:26:53.942 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T06:26:53.943 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T06:26:53.943 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-05T06:26:53.943 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-05T06:26:53.943 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-05T06:26:53.943 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:53.998 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : librados-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T06:26:53.998 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:53.998 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-02-05T06:26:53.998 INFO:teuthology.orchestra.run.smithi167.stdout: librados-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:26:53.998 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:53.998 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-02-05T06:26:54.248 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install libcephfs2 2024-02-05T06:26:54.943 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:35 ago on Mon 05 Feb 2024 06:25:19 AM UTC. 2024-02-05T06:26:54.943 INFO:teuthology.orchestra.run.smithi167.stdout:Package libcephfs2-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T06:26:54.999 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-02-05T06:26:54.999 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-02-05T06:26:55.000 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-02-05T06:26:55.028 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-mirror-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T06:26:55.028 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:55.028 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-02-05T06:26:55.028 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:26:55.029 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:55.029 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-02-05T06:26:55.076 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install libcephfs-devel 2024-02-05T06:26:55.263 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install rbd-nbd 2024-02-05T06:26:55.645 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:36 ago on Mon 05 Feb 2024 06:25:19 AM UTC. 2024-02-05T06:26:55.800 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-02-05T06:26:55.801 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:26:55.801 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-02-05T06:26:55.801 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:26:55.801 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-02-05T06:26:55.801 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs-devel x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 31 k 2024-02-05T06:26:55.801 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:55.801 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-02-05T06:26:55.801 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:26:55.802 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-02-05T06:26:55.802 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:55.802 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 31 k 2024-02-05T06:26:55.802 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 140 k 2024-02-05T06:26:55.802 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-02-05T06:26:55.838 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:01:48 ago on Mon 05 Feb 2024 06:25:07 AM UTC. 2024-02-05T06:26:56.006 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-02-05T06:26:56.006 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:26:56.006 INFO:teuthology.orchestra.run.smithi028.stdout: Package Architecture Version Repository Size 2024-02-05T06:26:56.007 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:26:56.007 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-02-05T06:26:56.007 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 171 k 2024-02-05T06:26:56.007 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:56.007 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-02-05T06:26:56.007 INFO:teuthology.orchestra.run.smithi028.stdout:================================================================================ 2024-02-05T06:26:56.007 INFO:teuthology.orchestra.run.smithi028.stdout:Install 1 Package 2024-02-05T06:26:56.007 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:56.007 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 171 k 2024-02-05T06:26:56.007 INFO:teuthology.orchestra.run.smithi028.stdout:Installed size: 494 k 2024-02-05T06:26:56.007 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-02-05T06:26:56.052 INFO:teuthology.orchestra.run.smithi167.stdout:libcephfs-devel-19.0.0-1241.g164d3be4.el9.x86_6 125 kB/s | 31 kB 00:00 2024-02-05T06:26:56.052 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:26:56.053 INFO:teuthology.orchestra.run.smithi167.stdout:Total 124 kB/s | 31 kB 00:00 2024-02-05T06:26:56.053 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-02-05T06:26:56.056 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-02-05T06:26:56.057 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-02-05T06:26:56.089 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-02-05T06:26:56.089 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-02-05T06:26:56.178 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-02-05T06:26:56.229 INFO:teuthology.orchestra.run.smithi028.stdout:rbd-nbd-19.0.0-1241.g164d3be4.el9.x86_64.rpm 771 kB/s | 171 kB 00:00 2024-02-05T06:26:56.229 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:26:56.230 INFO:teuthology.orchestra.run.smithi028.stdout:Total 767 kB/s | 171 kB 00:00 2024-02-05T06:26:56.230 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-02-05T06:26:56.238 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-02-05T06:26:56.238 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-02-05T06:26:56.276 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : libcephfs-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T06:26:56.297 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-02-05T06:26:56.298 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-02-05T06:26:56.482 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-02-05T06:26:56.580 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-nbd-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T06:26:56.693 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T06:26:56.973 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : libcephfs-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T06:26:56.973 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:56.973 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-02-05T06:26:56.973 INFO:teuthology.orchestra.run.smithi167.stdout: libcephfs-devel-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:26:56.974 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:26:56.974 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-02-05T06:26:57.039 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T06:26:57.170 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install librados2 2024-02-05T06:26:57.357 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-nbd-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T06:26:57.358 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:57.358 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-02-05T06:26:57.358 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:26:57.358 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:26:57.358 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-02-05T06:26:57.637 DEBUG:teuthology.parallel:result is None 2024-02-05T06:26:57.736 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:38 ago on Mon 05 Feb 2024 06:25:19 AM UTC. 2024-02-05T06:26:57.841 INFO:teuthology.orchestra.run.smithi167.stdout:Package librados2-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T06:26:57.894 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-02-05T06:26:57.895 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-02-05T06:26:57.895 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-02-05T06:26:57.964 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install librbd1 2024-02-05T06:26:58.530 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:39 ago on Mon 05 Feb 2024 06:25:19 AM UTC. 2024-02-05T06:26:58.636 INFO:teuthology.orchestra.run.smithi167.stdout:Package librbd1-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T06:26:58.688 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-02-05T06:26:58.689 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-02-05T06:26:58.689 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-02-05T06:26:58.759 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install python3-rados 2024-02-05T06:26:59.332 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:40 ago on Mon 05 Feb 2024 06:25:19 AM UTC. 2024-02-05T06:26:59.436 INFO:teuthology.orchestra.run.smithi167.stdout:Package python3-rados-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T06:26:59.489 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-02-05T06:26:59.490 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-02-05T06:26:59.490 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-02-05T06:26:59.557 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install python3-rgw 2024-02-05T06:27:00.133 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:41 ago on Mon 05 Feb 2024 06:25:19 AM UTC. 2024-02-05T06:27:00.236 INFO:teuthology.orchestra.run.smithi167.stdout:Package python3-rgw-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T06:27:00.289 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-02-05T06:27:00.289 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-02-05T06:27:00.290 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-02-05T06:27:00.358 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install python3-cephfs 2024-02-05T06:27:00.932 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:41 ago on Mon 05 Feb 2024 06:25:19 AM UTC. 2024-02-05T06:27:01.035 INFO:teuthology.orchestra.run.smithi167.stdout:Package python3-cephfs-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T06:27:01.087 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-02-05T06:27:01.088 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-02-05T06:27:01.088 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-02-05T06:27:01.155 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install python3-rbd 2024-02-05T06:27:01.730 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:42 ago on Mon 05 Feb 2024 06:25:19 AM UTC. 2024-02-05T06:27:01.834 INFO:teuthology.orchestra.run.smithi167.stdout:Package python3-rbd-2:19.0.0-1241.g164d3be4.el9.x86_64 is already installed. 2024-02-05T06:27:01.886 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-02-05T06:27:01.887 INFO:teuthology.orchestra.run.smithi167.stdout:Nothing to do. 2024-02-05T06:27:01.887 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-02-05T06:27:01.959 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install rbd-fuse 2024-02-05T06:27:02.531 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:43 ago on Mon 05 Feb 2024 06:25:19 AM UTC. 2024-02-05T06:27:02.689 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-02-05T06:27:02.689 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:27:02.689 INFO:teuthology.orchestra.run.smithi167.stdout: Package Architecture Version Repository Size 2024-02-05T06:27:02.689 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:27:02.690 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-02-05T06:27:02.690 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-fuse x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 85 k 2024-02-05T06:27:02.690 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:27:02.690 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-02-05T06:27:02.690 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:27:02.690 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-02-05T06:27:02.690 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:27:02.690 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 85 k 2024-02-05T06:27:02.690 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 231 k 2024-02-05T06:27:02.690 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-02-05T06:27:02.898 INFO:teuthology.orchestra.run.smithi167.stdout:rbd-fuse-19.0.0-1241.g164d3be4.el9.x86_64.rpm 411 kB/s | 85 kB 00:00 2024-02-05T06:27:02.898 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:27:02.899 INFO:teuthology.orchestra.run.smithi167.stdout:Total 408 kB/s | 85 kB 00:00 2024-02-05T06:27:02.899 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-02-05T06:27:02.906 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-02-05T06:27:02.906 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-02-05T06:27:02.960 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-02-05T06:27:02.960 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-02-05T06:27:03.096 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-02-05T06:27:03.196 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : rbd-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T06:27:03.596 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T06:27:03.889 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : rbd-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T06:27:03.889 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:27:03.889 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-02-05T06:27:03.890 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-fuse-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:27:03.890 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:27:03.890 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-02-05T06:27:04.126 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install rbd-mirror 2024-02-05T06:27:04.710 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:45 ago on Mon 05 Feb 2024 06:25:19 AM UTC. 2024-02-05T06:27:04.874 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-02-05T06:27:04.875 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:27:04.875 INFO:teuthology.orchestra.run.smithi167.stdout: Package Arch Version Repo Size 2024-02-05T06:27:04.875 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:27:04.875 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-02-05T06:27:04.875 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-mirror x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 3.1 M 2024-02-05T06:27:04.876 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:27:04.876 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-02-05T06:27:04.876 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:27:04.876 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-02-05T06:27:04.876 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:27:04.876 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 3.1 M 2024-02-05T06:27:04.876 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 13 M 2024-02-05T06:27:04.876 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-02-05T06:27:05.485 INFO:teuthology.orchestra.run.smithi167.stdout:rbd-mirror-19.0.0-1241.g164d3be4.el9.x86_64.rpm 5.1 MB/s | 3.1 MB 00:00 2024-02-05T06:27:05.486 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:27:05.486 INFO:teuthology.orchestra.run.smithi167.stdout:Total 5.1 MB/s | 3.1 MB 00:00 2024-02-05T06:27:05.486 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-02-05T06:27:05.495 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-02-05T06:27:05.496 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-02-05T06:27:05.569 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-02-05T06:27:05.570 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-02-05T06:27:05.998 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-02-05T06:27:06.061 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : rbd-mirror-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T06:27:06.085 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T06:27:06.085 INFO:teuthology.orchestra.run.smithi167.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-05T06:27:06.085 INFO:teuthology.orchestra.run.smithi167.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-05T06:27:06.085 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-05T06:27:06.085 INFO:teuthology.orchestra.run.smithi167.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-05T06:27:06.085 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:27:07.234 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : rbd-mirror-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T06:27:07.234 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:27:07.234 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-02-05T06:27:07.234 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-mirror-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:27:07.234 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:27:07.234 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-02-05T06:27:07.514 DEBUG:teuthology.orchestra.run.smithi167:> sudo yum -y install rbd-nbd 2024-02-05T06:27:08.099 INFO:teuthology.orchestra.run.smithi167.stdout:Last metadata expiration check: 0:01:49 ago on Mon 05 Feb 2024 06:25:19 AM UTC. 2024-02-05T06:27:08.260 INFO:teuthology.orchestra.run.smithi167.stdout:Dependencies resolved. 2024-02-05T06:27:08.260 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:27:08.260 INFO:teuthology.orchestra.run.smithi167.stdout: Package Architecture Version Repository Size 2024-02-05T06:27:08.260 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:27:08.260 INFO:teuthology.orchestra.run.smithi167.stdout:Installing: 2024-02-05T06:27:08.261 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-nbd x86_64 2:19.0.0-1241.g164d3be4.el9 ceph 171 k 2024-02-05T06:27:08.261 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:27:08.261 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction Summary 2024-02-05T06:27:08.261 INFO:teuthology.orchestra.run.smithi167.stdout:================================================================================ 2024-02-05T06:27:08.261 INFO:teuthology.orchestra.run.smithi167.stdout:Install 1 Package 2024-02-05T06:27:08.261 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:27:08.261 INFO:teuthology.orchestra.run.smithi167.stdout:Total download size: 171 k 2024-02-05T06:27:08.261 INFO:teuthology.orchestra.run.smithi167.stdout:Installed size: 494 k 2024-02-05T06:27:08.261 INFO:teuthology.orchestra.run.smithi167.stdout:Downloading Packages: 2024-02-05T06:27:08.474 INFO:teuthology.orchestra.run.smithi167.stdout:rbd-nbd-19.0.0-1241.g164d3be4.el9.x86_64.rpm 789 kB/s | 171 kB 00:00 2024-02-05T06:27:08.475 INFO:teuthology.orchestra.run.smithi167.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:27:08.475 INFO:teuthology.orchestra.run.smithi167.stdout:Total 783 kB/s | 171 kB 00:00 2024-02-05T06:27:08.475 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction check 2024-02-05T06:27:08.483 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction check succeeded. 2024-02-05T06:27:08.483 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction test 2024-02-05T06:27:08.538 INFO:teuthology.orchestra.run.smithi167.stdout:Transaction test succeeded. 2024-02-05T06:27:08.539 INFO:teuthology.orchestra.run.smithi167.stdout:Running transaction 2024-02-05T06:27:08.723 INFO:teuthology.orchestra.run.smithi167.stdout: Preparing : 1/1 2024-02-05T06:27:08.814 INFO:teuthology.orchestra.run.smithi167.stdout: Installing : rbd-nbd-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T06:27:09.243 INFO:teuthology.orchestra.run.smithi167.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T06:27:09.529 INFO:teuthology.orchestra.run.smithi167.stdout: Verifying : rbd-nbd-2:19.0.0-1241.g164d3be4.el9.x86_64 1/1 2024-02-05T06:27:09.529 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:27:09.529 INFO:teuthology.orchestra.run.smithi167.stdout:Installed: 2024-02-05T06:27:09.529 INFO:teuthology.orchestra.run.smithi167.stdout: rbd-nbd-2:19.0.0-1241.g164d3be4.el9.x86_64 2024-02-05T06:27:09.529 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:27:09.529 INFO:teuthology.orchestra.run.smithi167.stdout:Complete! 2024-02-05T06:27:09.760 DEBUG:teuthology.parallel:result is None 2024-02-05T06:27:09.761 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=164d3be4a527be775f4d3c343ea591122f1b8921 2024-02-05T06:27:09.980 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-05T06:27:10.005 INFO:teuthology.orchestra.run.smithi028.stdout:19.0.0-1241.g164d3be4.el9 2024-02-05T06:27:10.006 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1241.g164d3be4.el9 2024-02-05T06:27:10.006 INFO:teuthology.task.install:The correct ceph version 19.0.0-1241.g164d3be4 is installed. 2024-02-05T06:27:10.007 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=164d3be4a527be775f4d3c343ea591122f1b8921 2024-02-05T06:27:10.224 DEBUG:teuthology.orchestra.run.smithi167:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-05T06:27:10.246 INFO:teuthology.orchestra.run.smithi167.stdout:19.0.0-1241.g164d3be4.el9 2024-02-05T06:27:10.246 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1241.g164d3be4.el9 2024-02-05T06:27:10.246 INFO:teuthology.task.install:The correct ceph version 19.0.0-1241.g164d3be4 is installed. 2024-02-05T06:27:10.248 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-05T06:27:10.248 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-05T06:27:10.249 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-05T06:27:10.282 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-02-05T06:27:10.282 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-05T06:27:10.314 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-05T06:27:10.315 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-05T06:27:10.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/daemon-helper 2024-02-05T06:27:10.352 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-05T06:27:10.420 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-02-05T06:27:10.421 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/usr/bin/daemon-helper 2024-02-05T06:27:10.448 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-05T06:27:10.516 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-05T06:27:10.517 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-05T06:27:10.517 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-05T06:27:10.546 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-05T06:27:10.615 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-02-05T06:27:10.615 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-05T06:27:10.642 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-05T06:27:10.709 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-05T06:27:10.709 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-05T06:27:10.709 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/stdin-killer 2024-02-05T06:27:10.738 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-05T06:27:10.807 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-02-05T06:27:10.807 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/usr/bin/stdin-killer 2024-02-05T06:27:10.833 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-05T06:27:10.900 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-05T06:27:10.991 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'mon_bind_msgr2': False, 'sha1': '164d3be4a527be775f4d3c343ea591122f1b8921', 'cephadm_mode': 'root'} 2024-02-05T06:27:10.992 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 2024-02-05T06:27:10.992 INFO:tasks.cephadm:Cluster fsid is 9801e956-c3ef-11ee-95b6-87774f69a715 2024-02-05T06:27:10.992 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-05T06:27:10.993 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '[v1:172.21.15.28:6789]', 'mon.c': '[v1:172.21.15.28:6790]', 'mon.b': '[v1:172.21.15.167:6789]'} 2024-02-05T06:27:10.993 INFO:tasks.cephadm:First mon is mon.a on smithi028 2024-02-05T06:27:10.993 INFO:tasks.cephadm:First mgr is y 2024-02-05T06:27:10.993 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-05T06:27:10.993 DEBUG:teuthology.orchestra.run.smithi028:> sudo hostname $(hostname -s) 2024-02-05T06:27:11.023 DEBUG:teuthology.orchestra.run.smithi167:> sudo hostname $(hostname -s) 2024-02-05T06:27:11.050 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-02-05T06:27:11.050 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=164d3be4a527be775f4d3c343ea591122f1b8921 2024-02-05T06:27:11.398 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '164d3be4a527be775f4d3c343ea591122f1b8921', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/76505/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1241-g164d3be4', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1241.g164d3be4'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-02-05 00:32:51.050045', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-guits-testing-3-2024-02-04-2254', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-02-05T06:27:11.728 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-guits-testing-3-2024-02-04-2254, sha1 164d3be4a527be775f4d3c343ea591122f1b8921 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=164d3be4a527be775f4d3c343ea591122f1b8921 2024-02-05T06:27:11.729 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/x86_64/flavors/default/cephadm 2024-02-05T06:27:11.729 DEBUG:teuthology.orchestra.run.smithi028:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-05T06:27:12.194 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r--. 1 ubuntu ubuntu 766928 Feb 5 06:27 /home/ubuntu/cephtest/cephadm 2024-02-05T06:27:12.194 DEBUG:teuthology.orchestra.run.smithi167:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-guits-testing-3-2024-02-04-2254/164d3be4a527be775f4d3c343ea591122f1b8921/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-05T06:27:13.138 INFO:teuthology.orchestra.run.smithi167.stdout:-rw-r--r--. 1 ubuntu ubuntu 766928 Feb 5 06:27 /home/ubuntu/cephtest/cephadm 2024-02-05T06:27:13.139 DEBUG:teuthology.orchestra.run.smithi028:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-05T06:27:13.157 DEBUG:teuthology.orchestra.run.smithi167:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-05T06:27:13.189 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 on all hosts... 2024-02-05T06:27:13.189 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 pull 2024-02-05T06:27:13.200 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 pull 2024-02-05T06:27:14.038 INFO:teuthology.orchestra.run.smithi028.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921... 2024-02-05T06:27:14.153 INFO:teuthology.orchestra.run.smithi167.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921... 2024-02-05T06:27:47.409 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:27:47.410 INFO:teuthology.orchestra.run.smithi167.stdout: "ceph_version": "ceph version 19.0.0-1241-g164d3be4 (164d3be4a527be775f4d3c343ea591122f1b8921) squid (dev)", 2024-02-05T06:27:47.410 INFO:teuthology.orchestra.run.smithi167.stdout: "image_id": "8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1", 2024-02-05T06:27:47.410 INFO:teuthology.orchestra.run.smithi167.stdout: "repo_digests": [ 2024-02-05T06:27:47.410 INFO:teuthology.orchestra.run.smithi167.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0" 2024-02-05T06:27:47.410 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:27:47.410 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:27:49.216 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:27:49.271 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 19.0.0-1241-g164d3be4 (164d3be4a527be775f4d3c343ea591122f1b8921) squid (dev)", 2024-02-05T06:27:49.272 INFO:teuthology.orchestra.run.smithi028.stdout: "image_id": "8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1", 2024-02-05T06:27:49.272 INFO:teuthology.orchestra.run.smithi028.stdout: "repo_digests": [ 2024-02-05T06:27:49.272 INFO:teuthology.orchestra.run.smithi028.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0" 2024-02-05T06:27:49.272 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:27:49.272 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:27:49.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /etc/ceph 2024-02-05T06:27:49.302 DEBUG:teuthology.orchestra.run.smithi167:> sudo mkdir -p /etc/ceph 2024-02-05T06:27:49.338 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 777 /etc/ceph 2024-02-05T06:27:49.368 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 777 /etc/ceph 2024-02-05T06:27:49.408 INFO:tasks.cephadm:Writing seed config... 2024-02-05T06:27:49.409 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-05T06:27:49.409 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-05T06:27:49.409 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-02-05T06:27:49.409 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-02-05T06:27:49.409 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-02-05T06:27:49.409 INFO:tasks.cephadm: override: [global] ms bind msgr2 = False 2024-02-05T06:27:49.409 INFO:tasks.cephadm: override: [global] ms type = async 2024-02-05T06:27:49.409 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-05T06:27:49.410 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-05T06:27:49.410 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-05T06:27:49.410 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-02-05T06:27:49.410 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-02-05T06:27:49.410 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-02-05T06:27:49.410 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-02-05T06:27:49.410 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-02-05T06:27:49.410 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-05T06:27:49.410 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-05T06:27:49.410 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-02-05T06:27:49.410 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-02-05T06:27:49.410 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-02-05T06:27:49.410 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-02-05T06:27:49.410 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-02-05T06:27:49.410 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-02-05T06:27:49.410 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-02-05T06:27:49.411 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-05T06:27:49.411 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-05T06:27:49.425 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 9801e956-c3ef-11ee-95b6-87774f69a715 ms bind msgr2 = False ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-02-05T06:27:49.425 DEBUG:teuthology.orchestra.run.smithi028:mon.a> sudo journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@mon.a.service 2024-02-05T06:27:49.467 DEBUG:teuthology.orchestra.run.smithi028:mgr.y> sudo journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@mgr.y.service 2024-02-05T06:27:49.510 INFO:tasks.cephadm:Bootstrapping... 2024-02-05T06:27:49.510 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 -v bootstrap --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.28:6789]' --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-05T06:27:49.706 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-02-05T06:27:49.706 INFO:teuthology.orchestra.run.smithi028.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921', '-v', 'bootstrap', '--fsid', '9801e956-c3ef-11ee-95b6-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-addrv', '[v1:172.21.15.28:6789]', '--skip-admin-label'] 2024-02-05T06:27:49.732 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 4.9.0 2024-02-05T06:27:49.733 INFO:teuthology.orchestra.run.smithi028.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-02-05T06:27:49.733 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying podman|docker is present... 2024-02-05T06:27:49.756 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 4.9.0 2024-02-05T06:27:49.756 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying lvm2 is present... 2024-02-05T06:27:49.756 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying time synchronization is in place... 2024-02-05T06:27:49.763 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-05T06:27:49.763 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-05T06:27:49.769 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-05T06:27:49.769 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-02-05T06:27:49.775 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout enabled 2024-02-05T06:27:49.781 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout active 2024-02-05T06:27:49.781 INFO:teuthology.orchestra.run.smithi028.stdout:Unit chronyd.service is enabled and running 2024-02-05T06:27:49.781 INFO:teuthology.orchestra.run.smithi028.stdout:Repeating the final host check... 2024-02-05T06:27:49.804 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 4.9.0 2024-02-05T06:27:49.805 INFO:teuthology.orchestra.run.smithi028.stdout:podman (/bin/podman) version 4.9.0 is present 2024-02-05T06:27:49.805 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl is present 2024-02-05T06:27:49.805 INFO:teuthology.orchestra.run.smithi028.stdout:lvcreate is present 2024-02-05T06:27:49.811 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-05T06:27:49.811 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-05T06:27:49.817 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-05T06:27:49.817 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-02-05T06:27:49.823 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout enabled 2024-02-05T06:27:49.829 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout active 2024-02-05T06:27:49.829 INFO:teuthology.orchestra.run.smithi028.stdout:Unit chronyd.service is enabled and running 2024-02-05T06:27:49.829 INFO:teuthology.orchestra.run.smithi028.stdout:Host looks OK 2024-02-05T06:27:49.829 INFO:teuthology.orchestra.run.smithi028.stdout:Cluster fsid: 9801e956-c3ef-11ee-95b6-87774f69a715 2024-02-05T06:27:49.830 INFO:teuthology.orchestra.run.smithi028.stdout:Acquiring lock 139868325864016 on /run/cephadm/9801e956-c3ef-11ee-95b6-87774f69a715.lock 2024-02-05T06:27:49.830 INFO:teuthology.orchestra.run.smithi028.stdout:Lock 139868325864016 acquired on /run/cephadm/9801e956-c3ef-11ee-95b6-87774f69a715.lock 2024-02-05T06:27:49.830 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying IP 172.21.15.28 port 6789 ... 2024-02-05T06:27:49.830 INFO:teuthology.orchestra.run.smithi028.stdout:Base mon IP(s) is [172.21.15.28:6789], mon addrv is [v1:172.21.15.28:6789] 2024-02-05T06:27:49.833 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.28 metric 100 2024-02-05T06:27:49.834 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.28 metric 100 2024-02-05T06:27:49.836 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-02-05T06:27:49.836 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-02-05T06:27:49.836 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-02-05T06:27:49.838 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-05T06:27:49.838 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-02-05T06:27:49.838 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-05T06:27:49.838 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-02-05T06:27:49.838 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link noprefixroute 2024-02-05T06:27:49.838 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-05T06:27:49.839 INFO:teuthology.orchestra.run.smithi028.stdout:Mon IP `172.21.15.28` is in CIDR network `172.21.0.0/20` 2024-02-05T06:27:49.839 INFO:teuthology.orchestra.run.smithi028.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20'] 2024-02-05T06:27:49.839 INFO:teuthology.orchestra.run.smithi028.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-05T06:27:49.840 INFO:teuthology.orchestra.run.smithi028.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921... 2024-02-05T06:27:50.326 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 2024-02-05T06:27:50.327 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921... 2024-02-05T06:27:50.327 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Getting image source signatures 2024-02-05T06:27:50.327 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying blob sha256:48ac04de0ee67de328b3ea21477b81735e6e7c0783ebcda067ca4695a8266750 2024-02-05T06:27:50.327 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying blob sha256:58df4e03ba066accf3a7e07426bab5f1c867e27a139801c642bdafb42b9a2d9a 2024-02-05T06:27:50.327 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying config sha256:8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 2024-02-05T06:27:50.327 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Writing manifest to image destination 2024-02-05T06:27:51.181 INFO:teuthology.orchestra.run.smithi028.stdout:ceph: stdout ceph version 19.0.0-1241-g164d3be4 (164d3be4a527be775f4d3c343ea591122f1b8921) squid (dev) 2024-02-05T06:27:51.181 INFO:teuthology.orchestra.run.smithi028.stdout:Ceph version: ceph version 19.0.0-1241-g164d3be4 (164d3be4a527be775f4d3c343ea591122f1b8921) squid (dev) 2024-02-05T06:27:51.181 INFO:teuthology.orchestra.run.smithi028.stdout:Extracting ceph user uid/gid from container image... 2024-02-05T06:27:51.977 INFO:teuthology.orchestra.run.smithi028.stdout:stat: stdout 167 167 2024-02-05T06:27:51.977 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial keys... 2024-02-05T06:27:52.908 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQDof8Bl8T8DEhAAL0Fkk1mwcgj64A7sW6LrNg== 2024-02-05T06:27:55.646 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQDrf8BlMqFWBRAA3H0Utng8QS2NkMZa9qRR6A== 2024-02-05T06:27:56.684 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQDsf8BlYssWARAA0tqelSJ7a5RBi411O+WyCA== 2024-02-05T06:27:56.685 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial monmap... 2024-02-05T06:27:57.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-05T06:27:57.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-02-05T06:27:57.523 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 9801e956-c3ef-11ee-95b6-87774f69a715 2024-02-05T06:27:57.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-05T06:27:57.524 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool for a [v1:172.21.15.28:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-05T06:27:57.524 INFO:teuthology.orchestra.run.smithi028.stdout:setting min_mon_release = quincy 2024-02-05T06:27:57.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: set fsid to 9801e956-c3ef-11ee-95b6-87774f69a715 2024-02-05T06:27:57.524 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-05T06:27:57.524 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:27:57.524 INFO:teuthology.orchestra.run.smithi028.stdout:Creating mon... 2024-02-05T06:27:58.438 INFO:teuthology.orchestra.run.smithi028.stdout:create mon.a on 2024-02-05T06:27:58.644 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-02-05T06:27:58.789 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-05T06:27:58.975 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-9801e956-c3ef-11ee-95b6-87774f69a715.target → /etc/systemd/system/ceph-9801e956-c3ef-11ee-95b6-87774f69a715.target. 2024-02-05T06:27:58.975 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-9801e956-c3ef-11ee-95b6-87774f69a715.target → /etc/systemd/system/ceph-9801e956-c3ef-11ee-95b6-87774f69a715.target. 2024-02-05T06:27:59.211 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-9801e956-c3ef-11ee-95b6-87774f69a715@mon.a 2024-02-05T06:27:59.211 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to reset failed state of unit ceph-9801e956-c3ef-11ee-95b6-87774f69a715@mon.a.service: Unit ceph-9801e956-c3ef-11ee-95b6-87774f69a715@mon.a.service not loaded. 2024-02-05T06:27:59.379 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-9801e956-c3ef-11ee-95b6-87774f69a715.target.wants/ceph-9801e956-c3ef-11ee-95b6-87774f69a715@mon.a.service → /etc/systemd/system/ceph-9801e956-c3ef-11ee-95b6-87774f69a715@.service. 2024-02-05T06:27:59.946 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-05T06:27:59.946 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-02-05T06:27:59.954 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-05T06:27:59.954 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-02-05T06:27:59.954 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-02-05T06:27:59.954 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to enable service . firewalld.service is not available 2024-02-05T06:27:59.955 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mon to start... 2024-02-05T06:27:59.955 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mon... 2024-02-05T06:28:01.043 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout cluster: 2024-02-05T06:28:01.043 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout id: 9801e956-c3ef-11ee-95b6-87774f69a715 2024-02-05T06:28:01.043 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-02-05T06:28:01.043 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-02-05T06:28:01.043 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout services: 2024-02-05T06:28:01.043 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.558982s) 2024-02-05T06:28:01.043 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-02-05T06:28:01.043 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-02-05T06:28:01.043 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-02-05T06:28:01.043 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout data: 2024-02-05T06:28:01.044 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-02-05T06:28:01.044 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-02-05T06:28:01.044 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-02-05T06:28:01.044 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout pgs: 2024-02-05T06:28:01.044 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-02-05T06:28:01.044 INFO:teuthology.orchestra.run.smithi028.stdout:mon is available 2024-02-05T06:28:01.044 INFO:teuthology.orchestra.run.smithi028.stdout:Assimilating anything we can from ceph.conf... 2024-02-05T06:28:02.145 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-02-05T06:28:02.146 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [global] 2024-02-05T06:28:02.146 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout fsid = 9801e956-c3ef-11ee-95b6-87774f69a715 2024-02-05T06:28:02.146 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.28:6789] 2024-02-05T06:28:02.146 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-05T06:28:02.146 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-05T06:28:02.146 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-05T06:28:02.146 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-05T06:28:02.146 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-02-05T06:28:02.146 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-05T06:28:02.146 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-05T06:28:02.146 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-02-05T06:28:02.146 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [osd] 2024-02-05T06:28:02.146 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-05T06:28:02.146 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-05T06:28:02.147 INFO:teuthology.orchestra.run.smithi028.stdout:Generating new minimal ceph.conf... 2024-02-05T06:28:03.173 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:02 smithi028 ceph-mon[29072]: from='client.? v1:172.21.15.28:0/1455890159' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:28:03.176 INFO:teuthology.orchestra.run.smithi028.stdout:Restarting the monitor... 2024-02-05T06:28:03.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:03 smithi028 systemd[1]: Stopping Ceph mon.a for 9801e956-c3ef-11ee-95b6-87774f69a715... 2024-02-05T06:28:03.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:03 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-a[29068]: 2024-02-05T06:28:03.439+0000 7f736505a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T06:28:03.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:03 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-a[29068]: 2024-02-05T06:28:03.439+0000 7f736505a700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-02-05T06:28:03.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:03 smithi028 podman[29316]: 2024-02-05 06:28:03.481269645 +0000 UTC m=+0.129636431 container died fa31816d0f78c93a47418e50b94dadfdcc1fe8da11fca313c52e83cfbb7d0ae6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-a, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2) 2024-02-05T06:28:03.985 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:03 smithi028 podman[29316]: 2024-02-05 06:28:03.780901594 +0000 UTC m=+0.429268380 container cleanup fa31816d0f78c93a47418e50b94dadfdcc1fe8da11fca313c52e83cfbb7d0ae6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-a, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-05T06:28:03.985 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:03 smithi028 bash[29316]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-a 2024-02-05T06:28:04.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:03 smithi028 podman[29328]: 2024-02-05 06:28:03.990318477 +0000 UTC m=+0.504215702 container remove fa31816d0f78c93a47418e50b94dadfdcc1fe8da11fca313c52e83cfbb7d0ae6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-02-05T06:28:04.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 systemd[1]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715@mon.a.service: Deactivated successfully. 2024-02-05T06:28:04.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 systemd[1]: Stopped Ceph mon.a for 9801e956-c3ef-11ee-95b6-87774f69a715. 2024-02-05T06:28:04.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 systemd[1]: Starting Ceph mon.a for 9801e956-c3ef-11ee-95b6-87774f69a715... 2024-02-05T06:28:04.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 podman[29423]: 2024-02-05 06:28:04.324531409 +0000 UTC m=+0.109826199 container create edf115e6318f8233e18189108e4694ccd68c6d1df7ab9fb20a5065836e92af46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-02-05T06:28:04.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 podman[29423]: 2024-02-05 06:28:04.232930601 +0000 UTC m=+0.018225325 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 2024-02-05T06:28:04.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 podman[29423]: 2024-02-05 06:28:04.518274683 +0000 UTC m=+0.303569401 container init edf115e6318f8233e18189108e4694ccd68c6d1df7ab9fb20a5065836e92af46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-a, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-02-05T06:28:04.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 podman[29423]: 2024-02-05 06:28:04.520993862 +0000 UTC m=+0.306288583 container start edf115e6318f8233e18189108e4694ccd68c6d1df7ab9fb20a5065836e92af46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-02-05T06:28:04.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: set uid:gid to 167:167 (ceph:ceph) 2024-02-05T06:28:04.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: ceph version 19.0.0-1241-g164d3be4 (164d3be4a527be775f4d3c343ea591122f1b8921) squid (dev), process ceph-mon, pid 2 2024-02-05T06:28:04.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: pidfile_write: ignore empty --pid-file 2024-02-05T06:28:04.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: load: jerasure load: lrc 2024-02-05T06:28:04.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: RocksDB version: 7.9.2 2024-02-05T06:28:04.605 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Git sha 0 2024-02-05T06:28:04.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Compile date 2024-02-04 23:11:06 2024-02-05T06:28:04.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: DB SUMMARY 2024-02-05T06:28:04.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: DB Session ID: 3WG4J7V60VKA9OAE9W02 2024-02-05T06:28:04.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: CURRENT file: CURRENT 2024-02-05T06:28:04.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: IDENTITY file: IDENTITY 2024-02-05T06:28:04.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-02-05T06:28:04.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-02-05T06:28:04.606 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 89550 ; 2024-02-05T06:28:04.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.error_if_exists: 0 2024-02-05T06:28:04.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.create_if_missing: 0 2024-02-05T06:28:04.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.paranoid_checks: 1 2024-02-05T06:28:04.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-05T06:28:04.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-05T06:28:04.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-05T06:28:04.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.env: 0x55b5b0515ae0 2024-02-05T06:28:04.607 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.fs: PosixFileSystem 2024-02-05T06:28:04.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.info_log: 0x55b5b2c75900 2024-02-05T06:28:04.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.max_file_opening_threads: 16 2024-02-05T06:28:04.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.statistics: (nil) 2024-02-05T06:28:04.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.use_fsync: 0 2024-02-05T06:28:04.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.max_log_file_size: 0 2024-02-05T06:28:04.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-05T06:28:04.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-05T06:28:04.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.keep_log_file_num: 1000 2024-02-05T06:28:04.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.recycle_log_file_num: 0 2024-02-05T06:28:04.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.allow_fallocate: 1 2024-02-05T06:28:04.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.allow_mmap_reads: 0 2024-02-05T06:28:04.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.allow_mmap_writes: 0 2024-02-05T06:28:04.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.use_direct_reads: 0 2024-02-05T06:28:04.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-05T06:28:04.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.create_missing_column_families: 0 2024-02-05T06:28:04.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.db_log_dir: 2024-02-05T06:28:04.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.wal_dir: 2024-02-05T06:28:04.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-05T06:28:04.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-05T06:28:04.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-05T06:28:04.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-05T06:28:04.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-05T06:28:04.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-05T06:28:04.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.advise_random_on_open: 1 2024-02-05T06:28:04.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.db_write_buffer_size: 0 2024-02-05T06:28:04.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.write_buffer_manager: 0x55b5b2c49400 2024-02-05T06:28:04.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-05T06:28:04.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-05T06:28:04.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-05T06:28:04.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.rate_limiter: (nil) 2024-02-05T06:28:04.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-05T06:28:04.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.wal_recovery_mode: 2 2024-02-05T06:28:04.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.enable_thread_tracking: 0 2024-02-05T06:28:04.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.enable_pipelined_write: 0 2024-02-05T06:28:04.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.unordered_write: 0 2024-02-05T06:28:04.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-05T06:28:04.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-05T06:28:04.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-05T06:28:04.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-05T06:28:04.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.row_cache: None 2024-02-05T06:28:04.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.wal_filter: None 2024-02-05T06:28:04.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-05T06:28:04.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.allow_ingest_behind: 0 2024-02-05T06:28:04.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.two_write_queues: 0 2024-02-05T06:28:04.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.manual_wal_flush: 0 2024-02-05T06:28:04.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.wal_compression: 0 2024-02-05T06:28:04.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.atomic_flush: 0 2024-02-05T06:28:04.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-05T06:28:04.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-05T06:28:04.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-05T06:28:04.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.log_readahead_size: 0 2024-02-05T06:28:04.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-05T06:28:04.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.best_efforts_recovery: 0 2024-02-05T06:28:04.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-05T06:28:04.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-05T06:28:04.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.allow_data_in_errors: 0 2024-02-05T06:28:04.612 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.db_host_id: __hostname__ 2024-02-05T06:28:04.612 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-05T06:28:04.612 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.max_background_jobs: 2 2024-02-05T06:28:04.612 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.max_background_compactions: -1 2024-02-05T06:28:04.612 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.max_subcompactions: 1 2024-02-05T06:28:04.612 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-05T06:28:04.612 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-05T06:28:04.612 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-05T06:28:04.612 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.max_total_wal_size: 0 2024-02-05T06:28:04.612 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-05T06:28:04.612 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-05T06:28:04.612 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-05T06:28:04.612 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-05T06:28:04.612 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.max_open_files: -1 2024-02-05T06:28:04.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.bytes_per_sync: 0 2024-02-05T06:28:04.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-05T06:28:04.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-05T06:28:04.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.compaction_readahead_size: 0 2024-02-05T06:28:04.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.max_background_flushes: -1 2024-02-05T06:28:04.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Compression algorithms supported: 2024-02-05T06:28:04.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-05T06:28:04.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: kZSTD supported: 0 2024-02-05T06:28:04.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: kXpressCompression supported: 0 2024-02-05T06:28:04.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: kLZ4HCCompression supported: 1 2024-02-05T06:28:04.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: kZlibCompression supported: 1 2024-02-05T06:28:04.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: kSnappyCompression supported: 1 2024-02-05T06:28:04.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: kLZ4Compression supported: 1 2024-02-05T06:28:04.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: kBZip2Compression supported: 0 2024-02-05T06:28:04.613 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-05T06:28:04.614 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-05T06:28:04.614 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-02-05T06:28:04.614 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-05T06:28:04.614 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-05T06:28:04.614 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.merge_operator: 2024-02-05T06:28:04.614 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.compaction_filter: None 2024-02-05T06:28:04.614 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.compaction_filter_factory: None 2024-02-05T06:28:04.614 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.sst_partitioner_factory: None 2024-02-05T06:28:04.614 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-05T06:28:04.614 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-05T06:28:04.614 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b5b2c75a00) 2024-02-05T06:28:04.614 INFO:journalctl@ceph.mon.a.smithi028.stdout: cache_index_and_filter_blocks: 1 2024-02-05T06:28:04.614 INFO:journalctl@ceph.mon.a.smithi028.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-05T06:28:04.614 INFO:journalctl@ceph.mon.a.smithi028.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-05T06:28:04.614 INFO:journalctl@ceph.mon.a.smithi028.stdout: pin_top_level_index_and_filter: 1 2024-02-05T06:28:04.615 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_type: 0 2024-02-05T06:28:04.615 INFO:journalctl@ceph.mon.a.smithi028.stdout: data_block_index_type: 0 2024-02-05T06:28:04.615 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_shortening: 1 2024-02-05T06:28:04.615 INFO:journalctl@ceph.mon.a.smithi028.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-05T06:28:04.615 INFO:journalctl@ceph.mon.a.smithi028.stdout: checksum: 4 2024-02-05T06:28:04.615 INFO:journalctl@ceph.mon.a.smithi028.stdout: no_block_cache: 0 2024-02-05T06:28:04.615 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache: 0x55b5b2c79090 2024-02-05T06:28:04.615 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_name: BinnedLRUCache 2024-02-05T06:28:04.615 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_options: 2024-02-05T06:28:04.615 INFO:journalctl@ceph.mon.a.smithi028.stdout: capacity : 536870912 2024-02-05T06:28:04.615 INFO:journalctl@ceph.mon.a.smithi028.stdout: num_shard_bits : 4 2024-02-05T06:28:04.615 INFO:journalctl@ceph.mon.a.smithi028.stdout: strict_capacity_limit : 0 2024-02-05T06:28:04.615 INFO:journalctl@ceph.mon.a.smithi028.stdout: high_pri_pool_ratio: 0.000 2024-02-05T06:28:04.615 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_compressed: (nil) 2024-02-05T06:28:04.616 INFO:journalctl@ceph.mon.a.smithi028.stdout: persistent_cache: (nil) 2024-02-05T06:28:04.616 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_size: 4096 2024-02-05T06:28:04.616 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_size_deviation: 10 2024-02-05T06:28:04.616 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_restart_interval: 16 2024-02-05T06:28:04.616 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_block_restart_interval: 1 2024-02-05T06:28:04.616 INFO:journalctl@ceph.mon.a.smithi028.stdout: metadata_block_size: 4096 2024-02-05T06:28:04.616 INFO:journalctl@ceph.mon.a.smithi028.stdout: partition_filters: 0 2024-02-05T06:28:04.616 INFO:journalctl@ceph.mon.a.smithi028.stdout: use_delta_encoding: 1 2024-02-05T06:28:04.616 INFO:journalctl@ceph.mon.a.smithi028.stdout: filter_policy: bloomfilter 2024-02-05T06:28:04.616 INFO:journalctl@ceph.mon.a.smithi028.stdout: whole_key_filtering: 1 2024-02-05T06:28:04.616 INFO:journalctl@ceph.mon.a.smithi028.stdout: verify_compression: 0 2024-02-05T06:28:04.616 INFO:journalctl@ceph.mon.a.smithi028.stdout: read_amp_bytes_per_bit: 0 2024-02-05T06:28:04.616 INFO:journalctl@ceph.mon.a.smithi028.stdout: format_version: 5 2024-02-05T06:28:04.616 INFO:journalctl@ceph.mon.a.smithi028.stdout: enable_index_compression: 1 2024-02-05T06:28:04.616 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_align: 0 2024-02-05T06:28:04.617 INFO:journalctl@ceph.mon.a.smithi028.stdout: max_auto_readahead_size: 262144 2024-02-05T06:28:04.617 INFO:journalctl@ceph.mon.a.smithi028.stdout: prepopulate_block_cache: 0 2024-02-05T06:28:04.617 INFO:journalctl@ceph.mon.a.smithi028.stdout: initial_auto_readahead_size: 8192 2024-02-05T06:28:04.617 INFO:journalctl@ceph.mon.a.smithi028.stdout: num_file_reads_for_auto_readahead: 2 2024-02-05T06:28:04.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.write_buffer_size: 33554432 2024-02-05T06:28:04.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.max_write_buffer_number: 2 2024-02-05T06:28:04.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.compression: NoCompression 2024-02-05T06:28:04.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.bottommost_compression: Disabled 2024-02-05T06:28:04.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.prefix_extractor: nullptr 2024-02-05T06:28:04.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-05T06:28:04.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.num_levels: 7 2024-02-05T06:28:04.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-05T06:28:04.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-05T06:28:04.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-05T06:28:04.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-05T06:28:04.617 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-05T06:28:04.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-05T06:28:04.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-05T06:28:04.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-05T06:28:04.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-05T06:28:04.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-05T06:28:04.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-05T06:28:04.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-05T06:28:04.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-05T06:28:04.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.compression_opts.level: 32767 2024-02-05T06:28:04.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.compression_opts.strategy: 0 2024-02-05T06:28:04.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-05T06:28:04.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-05T06:28:04.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-05T06:28:04.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-05T06:28:04.618 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.compression_opts.enabled: false 2024-02-05T06:28:04.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-05T06:28:04.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-05T06:28:04.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-05T06:28:04.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-05T06:28:04.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.target_file_size_base: 67108864 2024-02-05T06:28:04.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-05T06:28:04.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-05T06:28:04.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-05T06:28:04.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-05T06:28:04.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-05T06:28:04.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-05T06:28:04.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-05T06:28:04.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-05T06:28:04.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-05T06:28:04.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-05T06:28:04.620 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-05T06:28:04.620 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-05T06:28:04.620 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-05T06:28:04.620 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-05T06:28:04.620 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.arena_block_size: 1048576 2024-02-05T06:28:04.620 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-05T06:28:04.620 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-05T06:28:04.620 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.disable_auto_compactions: 0 2024-02-05T06:28:04.620 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-05T06:28:04.620 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-05T06:28:04.620 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-05T06:28:04.620 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-05T06:28:04.620 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-05T06:28:04.620 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-05T06:28:04.620 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-05T06:28:04.621 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-05T06:28:04.621 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-05T06:28:04.621 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-05T06:28:04.621 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.table_properties_collectors: 2024-02-05T06:28:04.621 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.inplace_update_support: 0 2024-02-05T06:28:04.621 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-05T06:28:04.621 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-05T06:28:04.621 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-05T06:28:04.622 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-05T06:28:04.622 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.bloom_locality: 0 2024-02-05T06:28:04.622 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.max_successive_merges: 0 2024-02-05T06:28:04.622 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-05T06:28:04.622 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.paranoid_file_checks: 0 2024-02-05T06:28:04.622 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.force_consistency_checks: 1 2024-02-05T06:28:04.622 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.report_bg_io_stats: 0 2024-02-05T06:28:04.622 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.ttl: 2592000 2024-02-05T06:28:04.622 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-05T06:28:04.622 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-05T06:28:04.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-05T06:28:04.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.enable_blob_files: false 2024-02-05T06:28:04.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.min_blob_size: 0 2024-02-05T06:28:04.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.blob_file_size: 268435456 2024-02-05T06:28:04.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-05T06:28:04.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-05T06:28:04.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-05T06:28:04.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-05T06:28:04.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-05T06:28:04.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.blob_file_starting_level: 0 2024-02-05T06:28:04.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-05T06:28:04.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-02-05T06:28:04.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-02-05T06:28:04.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d684d2e1-4d9f-4aaa-a9a3-c5acf5c4c770 2024-02-05T06:28:04.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707114484549781, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-02-05T06:28:04.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-02-05T06:28:04.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707114484550722, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 85487, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 280, "table_properties": {"data_size": 83595, "index_size": 238, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11691, "raw_average_key_size": 48, "raw_value_size": 77150, "raw_average_value_size": 322, "num_data_blocks": 11, "num_entries": 239, "num_filter_entries": 239, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707114484, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d684d2e1-4d9f-4aaa-a9a3-c5acf5c4c770", "db_session_id": "3WG4J7V60VKA9OAE9W02", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-02-05T06:28:04.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707114484550807, "job": 1, "event": "recovery_finished"} 2024-02-05T06:28:04.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-02-05T06:28:04.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-05T06:28:04.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55b5b2d6a000 2024-02-05T06:28:04.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: DB pointer 0x55b5b2d54000 2024-02-05T06:28:04.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-05T06:28:04.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-05T06:28:04.624 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** DB Stats ** 2024-02-05T06:28:04.624 INFO:journalctl@ceph.mon.a.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-05T06:28:04.624 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-05T06:28:04.624 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-05T06:28:04.625 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-05T06:28:04.625 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-05T06:28:04.625 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-05T06:28:04.625 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-05T06:28:04.625 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-02-05T06:28:04.625 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-02-05T06:28:04.625 INFO:journalctl@ceph.mon.a.smithi028.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-05T06:28:04.625 INFO:journalctl@ceph.mon.a.smithi028.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-05T06:28:04.625 INFO:journalctl@ceph.mon.a.smithi028.stdout: L0 2/0 85.28 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 131.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-05T06:28:04.625 INFO:journalctl@ceph.mon.a.smithi028.stdout: Sum 2/0 85.28 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 131.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-05T06:28:04.625 INFO:journalctl@ceph.mon.a.smithi028.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 131.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-05T06:28:04.625 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-02-05T06:28:04.625 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-02-05T06:28:04.625 INFO:journalctl@ceph.mon.a.smithi028.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-05T06:28:04.625 INFO:journalctl@ceph.mon.a.smithi028.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-05T06:28:04.626 INFO:journalctl@ceph.mon.a.smithi028.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 131.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-05T06:28:04.626 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-02-05T06:28:04.626 INFO:journalctl@ceph.mon.a.smithi028.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-05T06:28:04.626 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-02-05T06:28:04.626 INFO:journalctl@ceph.mon.a.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-05T06:28:04.626 INFO:journalctl@ceph.mon.a.smithi028.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-05T06:28:04.626 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-05T06:28:04.626 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-05T06:28:04.626 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-05T06:28:04.626 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-05T06:28:04.626 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative compaction: 0.00 GB write, 27.18 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-05T06:28:04.626 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval compaction: 0.00 GB write, 27.18 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-05T06:28:04.626 INFO:journalctl@ceph.mon.a.smithi028.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-05T06:28:04.626 INFO:journalctl@ceph.mon.a.smithi028.stdout: Block cache BinnedLRUCache@0x55b5b2c79090#2 capacity: 512.00 MB usage: 1.25 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-02-05T06:28:04.626 INFO:journalctl@ceph.mon.a.smithi028.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.42 KB,8.04663e-05%) Misc(1,0.00 KB,0%) 2024-02-05T06:28:04.627 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-02-05T06:28:04.627 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-05T06:28:04.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: starting mon.a rank 0 at public addrs v1:172.21.15.28:6789/0 at bind addrs v1:172.21.15.28:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid 9801e956-c3ef-11ee-95b6-87774f69a715 2024-02-05T06:28:04.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: mon.a@-1(???) e1 preinit fsid 9801e956-c3ef-11ee-95b6-87774f69a715 2024-02-05T06:28:04.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: mon.a@-1(???).mds e1 new map 2024-02-05T06:28:04.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: mon.a@-1(???).mds e1 print_map 2024-02-05T06:28:04.627 INFO:journalctl@ceph.mon.a.smithi028.stdout: e1 2024-02-05T06:28:04.627 INFO:journalctl@ceph.mon.a.smithi028.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-05T06:28:04.627 INFO:journalctl@ceph.mon.a.smithi028.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-05T06:28:04.627 INFO:journalctl@ceph.mon.a.smithi028.stdout: legacy client fscid: -1 2024-02-05T06:28:04.627 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-02-05T06:28:04.627 INFO:journalctl@ceph.mon.a.smithi028.stdout: No filesystems configured 2024-02-05T06:28:04.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-05T06:28:04.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-05T06:28:04.627 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-05T06:28:04.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-05T06:28:04.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-05T06:28:04.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 bash[29423]: edf115e6318f8233e18189108e4694ccd68c6d1df7ab9fb20a5065836e92af46 2024-02-05T06:28:04.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 systemd[1]: Started Ceph mon.a for 9801e956-c3ef-11ee-95b6-87774f69a715. 2024-02-05T06:28:04.628 INFO:teuthology.orchestra.run.smithi028.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-02-05T06:28:04.907 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-05T06:28:04.907 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: monmap e1: 1 mons at {a=v1:172.21.15.28:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-02-05T06:28:04.907 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: fsmap 2024-02-05T06:28:04.907 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: osdmap e1: 0 total, 0 up, 0 in 2024-02-05T06:28:04.907 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:04 smithi028 ceph-mon[29442]: mgrmap e1: no daemons active 2024-02-05T06:28:05.630 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote config to /etc/ceph/ceph.conf 2024-02-05T06:28:05.660 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-05T06:28:05.660 INFO:teuthology.orchestra.run.smithi028.stdout:Creating mgr... 2024-02-05T06:28:05.661 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying port 0.0.0.0:9283 ... 2024-02-05T06:28:05.661 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying port 0.0.0.0:8765 ... 2024-02-05T06:28:05.903 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-9801e956-c3ef-11ee-95b6-87774f69a715@mgr.y 2024-02-05T06:28:05.903 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to reset failed state of unit ceph-9801e956-c3ef-11ee-95b6-87774f69a715@mgr.y.service: Unit ceph-9801e956-c3ef-11ee-95b6-87774f69a715@mgr.y.service not loaded. 2024-02-05T06:28:06.055 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-9801e956-c3ef-11ee-95b6-87774f69a715.target.wants/ceph-9801e956-c3ef-11ee-95b6-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-9801e956-c3ef-11ee-95b6-87774f69a715@.service. 2024-02-05T06:28:06.072 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:06 smithi028 systemd[1]: Starting Ceph mgr.y for 9801e956-c3ef-11ee-95b6-87774f69a715... 2024-02-05T06:28:06.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:06 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/199405931' entity='client.admin' 2024-02-05T06:28:06.384 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:06 smithi028 podman[29660]: 2024-02-05 06:28:06.292842053 +0000 UTC m=+0.113473194 container create c706bd03e503cf878164bd837c9a67fd401c9a94b2327f9e48f4eebcc8042e1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-02-05T06:28:06.384 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:06 smithi028 podman[29660]: 2024-02-05 06:28:06.196240061 +0000 UTC m=+0.016871209 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 2024-02-05T06:28:06.586 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-05T06:28:06.586 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-02-05T06:28:06.593 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-05T06:28:06.593 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-02-05T06:28:06.593 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-02-05T06:28:06.593 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to enable service . firewalld.service is not available 2024-02-05T06:28:06.601 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-05T06:28:06.601 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-02-05T06:28:06.608 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-05T06:28:06.608 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-02-05T06:28:06.608 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-02-05T06:28:06.608 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-02-05T06:28:06.608 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr to start... 2024-02-05T06:28:06.608 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr... 2024-02-05T06:28:06.687 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:06 smithi028 podman[29660]: 2024-02-05 06:28:06.494908788 +0000 UTC m=+0.315539929 container init c706bd03e503cf878164bd837c9a67fd401c9a94b2327f9e48f4eebcc8042e1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2) 2024-02-05T06:28:06.687 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:06 smithi028 podman[29660]: 2024-02-05 06:28:06.497373877 +0000 UTC m=+0.318005020 container start c706bd03e503cf878164bd837c9a67fd401c9a94b2327f9e48f4eebcc8042e1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-02-05T06:28:06.688 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:06 smithi028 bash[29660]: c706bd03e503cf878164bd837c9a67fd401c9a94b2327f9e48f4eebcc8042e1d 2024-02-05T06:28:06.688 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:06 smithi028 systemd[1]: Started Ceph mgr.y for 9801e956-c3ef-11ee-95b6-87774f69a715. 2024-02-05T06:28:07.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:07 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/4102492574' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-05T06:28:07.354 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:07 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:07.057+0000 7f7c11530200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-05T06:28:07.354 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:07 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:07.157+0000 7f7c11530200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-05T06:28:07.598 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-02-05T06:28:07.601 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-02-05T06:28:07.601 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "9801e956-c3ef-11ee-95b6-87774f69a715", 2024-02-05T06:28:07.601 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-02-05T06:28:07.601 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-05T06:28:07.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-05T06:28:07.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-05T06:28:07.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-02-05T06:28:07.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-05T06:28:07.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-05T06:28:07.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-02-05T06:28:07.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-02-05T06:28:07.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-05T06:28:07.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-02-05T06:28:07.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-02-05T06:28:07.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-02-05T06:28:07.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-05T06:28:07.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T06:28:07.602 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-05T06:28:07.603 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-05T06:28:07.603 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-02-05T06:28:07.603 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-05T06:28:07.603 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T06:28:07.603 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-05T06:28:07.603 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-05T06:28:07.603 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-05T06:28:07.603 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-05T06:28:07.603 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-05T06:28:07.603 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-05T06:28:07.607 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-02-05T06:28:07.607 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-05T06:28:07.607 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-05T06:28:07.607 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-05T06:28:07.607 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-05T06:28:07.607 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-05T06:28:07.607 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-05T06:28:07.607 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-05T06:28:07.607 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-05T06:28:07.607 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-05T06:28:07.607 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-02-05T06:28:07.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-05T06:28:07.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T06:28:07.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-05T06:28:07.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-05T06:28:07.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-02-05T06:28:07.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-05T06:28:07.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-05T06:28:07.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-05T06:28:07.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-05T06:28:07.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-05T06:28:07.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-05T06:28:07.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-02-05T06:28:07.608 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-02-05T06:28:07.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-05T06:28:07.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-02-05T06:28:07.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-05T06:28:07.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T06:28:07.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-02-05T06:27:59.919721+0000", 2024-02-05T06:28:07.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-05T06:28:07.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-02-05T06:28:07.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-05T06:28:07.609 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-02-05T06:28:07.609 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (1/15)... 2024-02-05T06:28:08.153 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:07 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:07.824+0000 7f7c11530200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-05T06:28:08.153 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:07 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:07.918+0000 7f7c11530200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-05T06:28:08.153 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:08 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:08.060+0000 7f7c11530200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-05T06:28:08.153 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:08 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:08.157+0000 7f7c11530200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-05T06:28:08.437 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:08 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:08.241+0000 7f7c11530200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-05T06:28:08.811 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:08 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:08.516+0000 7f7c11530200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-05T06:28:08.811 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:08 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:08.629+0000 7f7c11530200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-05T06:28:08.811 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:08 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:08.717+0000 7f7c11530200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-05T06:28:08.812 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:08 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:08.812+0000 7f7c11530200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-05T06:28:09.187 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:08 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:08.990+0000 7f7c11530200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-05T06:28:10.437 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:10 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:10.013+0000 7f7c11530200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-05T06:28:10.437 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:10 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:10.155+0000 7f7c11530200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-05T06:28:10.779 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:10 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2025566148' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-05T06:28:10.780 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:10 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:10.492+0000 7f7c11530200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-05T06:28:10.780 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:10 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:10.584+0000 7f7c11530200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-05T06:28:10.780 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:10 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:10.783+0000 7f7c11530200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-05T06:28:11.187 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:10 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:10.970+0000 7f7c11530200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-05T06:28:11.517 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:11.257+0000 7f7c11530200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-05T06:28:11.937 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:11.521+0000 7f7c11530200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-05T06:28:11.937 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:11.763+0000 7f7c11530200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-05T06:28:12.937 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:12 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:12.589+0000 7f7c11530200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-05T06:28:13.437 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:13 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:13.173+0000 7f7c11530200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-05T06:28:13.438 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:13 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:13.273+0000 7f7c11530200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-05T06:28:13.937 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:13 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:13.492+0000 7f7c11530200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-05T06:28:13.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:13 smithi028 ceph-mon[29442]: Activating manager daemon y 2024-02-05T06:28:13.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:13 smithi028 ceph-mon[29442]: mgrmap e2: y(active, starting, since 0.00299157s) 2024-02-05T06:28:13.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:13 smithi028 ceph-mon[29442]: from='mgr.14100 v1:172.21.15.28:0/2671948786' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-05T06:28:13.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:13 smithi028 ceph-mon[29442]: from='mgr.14100 v1:172.21.15.28:0/2671948786' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-05T06:28:13.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:13 smithi028 ceph-mon[29442]: from='mgr.14100 v1:172.21.15.28:0/2671948786' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-05T06:28:13.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:13 smithi028 ceph-mon[29442]: from='mgr.14100 v1:172.21.15.28:0/2671948786' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-05T06:28:13.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:13 smithi028 ceph-mon[29442]: from='mgr.14100 v1:172.21.15.28:0/2671948786' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-05T06:28:13.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:13 smithi028 ceph-mon[29442]: from='mgr.14100 v1:172.21.15.28:0/2671948786' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-05T06:28:13.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:13 smithi028 ceph-mon[29442]: from='mgr.14100 v1:172.21.15.28:0/2671948786' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T06:28:13.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:13 smithi028 ceph-mon[29442]: from='mgr.14100 v1:172.21.15.28:0/2671948786' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-05T06:28:13.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:13 smithi028 ceph-mon[29442]: Manager daemon y is now available 2024-02-05T06:28:13.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:13 smithi028 ceph-mon[29442]: from='mgr.14100 v1:172.21.15.28:0/2671948786' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-05T06:28:13.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:13 smithi028 ceph-mon[29442]: from='mgr.14100 v1:172.21.15.28:0/2671948786' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-05T06:28:13.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:13 smithi028 ceph-mon[29442]: from='mgr.14100 v1:172.21.15.28:0/2671948786' entity='mgr.y' 2024-02-05T06:28:13.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:13 smithi028 ceph-mon[29442]: from='mgr.14100 v1:172.21.15.28:0/2671948786' entity='mgr.y' 2024-02-05T06:28:13.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:13 smithi028 ceph-mon[29442]: from='mgr.14100 v1:172.21.15.28:0/2671948786' entity='mgr.y' 2024-02-05T06:28:14.677 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-02-05T06:28:14.678 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-02-05T06:28:14.678 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "9801e956-c3ef-11ee-95b6-87774f69a715", 2024-02-05T06:28:14.678 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-02-05T06:28:14.678 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-05T06:28:14.678 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-05T06:28:14.678 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-05T06:28:14.678 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-02-05T06:28:14.678 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-05T06:28:14.678 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-05T06:28:14.678 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-02-05T06:28:14.678 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-02-05T06:28:14.678 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-05T06:28:14.678 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-02-05T06:28:14.679 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-02-05T06:28:14.679 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-02-05T06:28:14.679 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-05T06:28:14.679 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T06:28:14.679 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-05T06:28:14.679 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-05T06:28:14.679 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-02-05T06:28:14.679 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-05T06:28:14.679 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T06:28:14.679 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-05T06:28:14.679 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-05T06:28:14.679 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-05T06:28:14.679 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-05T06:28:14.679 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-05T06:28:14.679 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-05T06:28:14.683 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-02-05T06:28:14.683 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-05T06:28:14.683 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-05T06:28:14.683 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-05T06:28:14.683 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-05T06:28:14.683 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-05T06:28:14.683 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-05T06:28:14.683 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-05T06:28:14.683 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-05T06:28:14.683 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-05T06:28:14.683 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-02-05T06:28:14.683 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-05T06:28:14.684 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T06:28:14.684 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-05T06:28:14.684 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-05T06:28:14.684 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-02-05T06:28:14.684 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-05T06:28:14.684 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-05T06:28:14.684 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-05T06:28:14.684 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-05T06:28:14.684 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-05T06:28:14.684 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-05T06:28:14.684 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-02-05T06:28:14.684 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-02-05T06:28:14.684 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-05T06:28:14.684 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-02-05T06:28:14.684 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-05T06:28:14.685 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T06:28:14.685 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-02-05T06:27:59.919721+0000", 2024-02-05T06:28:14.685 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-05T06:28:14.685 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-02-05T06:28:14.685 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-05T06:28:14.685 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-02-05T06:28:14.685 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (2/15)... 2024-02-05T06:28:15.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:15 smithi028 ceph-mon[29442]: mgrmap e3: y(active, since 1.0071s) 2024-02-05T06:28:17.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:17 smithi028 ceph-mon[29442]: mgrmap e4: y(active, since 3s) 2024-02-05T06:28:17.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:17 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3380185106' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-05T06:28:17.968 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-02-05T06:28:17.968 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-02-05T06:28:17.968 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "9801e956-c3ef-11ee-95b6-87774f69a715", 2024-02-05T06:28:17.968 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-02-05T06:28:17.969 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-05T06:28:17.969 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-05T06:28:17.969 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-05T06:28:17.969 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-02-05T06:28:17.969 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-05T06:28:17.969 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-05T06:28:17.969 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-02-05T06:28:17.969 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-02-05T06:28:17.969 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-05T06:28:17.969 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-02-05T06:28:17.969 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-02-05T06:28:17.969 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-02-05T06:28:17.969 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-05T06:28:17.969 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T06:28:17.969 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-05T06:28:17.970 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-05T06:28:17.970 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-02-05T06:28:17.970 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-05T06:28:17.970 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T06:28:17.970 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-05T06:28:17.970 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-05T06:28:17.970 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-05T06:28:17.970 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-05T06:28:17.970 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-05T06:28:17.970 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-05T06:28:17.973 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-02-05T06:28:17.974 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-05T06:28:17.974 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-05T06:28:17.974 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-05T06:28:17.974 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-05T06:28:17.974 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-05T06:28:17.974 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-05T06:28:17.974 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-05T06:28:17.974 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-05T06:28:17.974 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-05T06:28:17.974 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-02-05T06:28:17.974 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-05T06:28:17.974 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T06:28:17.974 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-05T06:28:17.974 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-05T06:28:17.975 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-02-05T06:28:17.975 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-05T06:28:17.975 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-05T06:28:17.975 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-05T06:28:17.975 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-05T06:28:17.975 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-05T06:28:17.975 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-05T06:28:17.975 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-02-05T06:28:17.975 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-02-05T06:28:17.975 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-05T06:28:17.975 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-02-05T06:28:17.975 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-05T06:28:17.975 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-05T06:28:17.975 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-02-05T06:27:59.919721+0000", 2024-02-05T06:28:17.975 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-05T06:28:17.975 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-02-05T06:28:17.976 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-05T06:28:17.976 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-02-05T06:28:17.976 INFO:teuthology.orchestra.run.smithi028.stdout:mgr is available 2024-02-05T06:28:18.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:18 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1419878868' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-05T06:28:19.081 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-02-05T06:28:19.081 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [global] 2024-02-05T06:28:19.081 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout fsid = 9801e956-c3ef-11ee-95b6-87774f69a715 2024-02-05T06:28:19.081 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-05T06:28:19.081 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-05T06:28:19.081 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-05T06:28:19.081 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-05T06:28:19.081 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-02-05T06:28:19.082 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-05T06:28:19.082 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-05T06:28:19.082 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-02-05T06:28:19.082 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [osd] 2024-02-05T06:28:19.082 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-05T06:28:19.082 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-05T06:28:19.082 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling cephadm module... 2024-02-05T06:28:20.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:20 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3057022053' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-05T06:28:20.938 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:20 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ignoring --setuser ceph since I am not root 2024-02-05T06:28:20.938 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:20 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ignoring --setgroup ceph since I am not root 2024-02-05T06:28:21.437 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:21 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:21.128+0000 7f01de43a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-05T06:28:21.438 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:21 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:21.224+0000 7f01de43a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-05T06:28:21.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:21 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3057022053' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-05T06:28:21.876 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:21 smithi028 ceph-mon[29442]: mgrmap e5: y(active, since 7s) 2024-02-05T06:28:22.187 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:21 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:21.880+0000 7f01de43a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-05T06:28:22.188 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:21 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:21.976+0000 7f01de43a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-05T06:28:22.188 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:22 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:22.119+0000 7f01de43a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-05T06:28:22.566 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:22 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:22.217+0000 7f01de43a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-05T06:28:22.566 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:22 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:22.302+0000 7f01de43a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-05T06:28:22.566 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:22 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:22.571+0000 7f01de43a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-05T06:28:22.883 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:22 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:22.682+0000 7f01de43a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-05T06:28:22.883 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:22 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:22.770+0000 7f01de43a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-05T06:28:23.187 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:22 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:22.867+0000 7f01de43a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-05T06:28:23.188 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:23 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:23.045+0000 7f01de43a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-05T06:28:24.437 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:24 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:24.033+0000 7f01de43a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-05T06:28:24.438 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:24 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:24.167+0000 7f01de43a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-05T06:28:24.795 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:24 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:24.505+0000 7f01de43a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-05T06:28:24.795 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:24 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:24.597+0000 7f01de43a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-05T06:28:24.795 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:24 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:24.799+0000 7f01de43a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-05T06:28:25.101 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:24 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:24.988+0000 7f01de43a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-05T06:28:25.353 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:25 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:25.289+0000 7f01de43a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-05T06:28:25.667 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:25 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:25.556+0000 7f01de43a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-05T06:28:25.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:25 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2551563310' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-05T06:28:25.938 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:25 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:25.801+0000 7f01de43a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-05T06:28:26.185 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-02-05T06:28:26.185 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-02-05T06:28:26.185 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-05T06:28:26.185 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-02-05T06:28:26.186 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-05T06:28:26.186 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-02-05T06:28:26.186 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for the mgr to restart... 2024-02-05T06:28:26.186 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr epoch 5... 2024-02-05T06:28:26.937 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:26 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:26.639+0000 7f01de43a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-05T06:28:27.535 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:27 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:27.219+0000 7f01de43a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-05T06:28:27.535 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:27 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:27.319+0000 7f01de43a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-05T06:28:27.535 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:27 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:27.539+0000 7f01de43a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-05T06:28:27.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:27 smithi028 ceph-mon[29442]: Active manager daemon y restarted 2024-02-05T06:28:27.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:27 smithi028 ceph-mon[29442]: Activating manager daemon y 2024-02-05T06:28:27.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:27 smithi028 ceph-mon[29442]: osdmap e2: 0 total, 0 up, 0 in 2024-02-05T06:28:27.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:27 smithi028 ceph-mon[29442]: mgrmap e6: y(active, starting, since 0.0032292s) 2024-02-05T06:28:27.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:27 smithi028 ceph-mon[29442]: from='mgr.14118 v1:172.21.15.28:0/3725412590' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T06:28:27.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:27 smithi028 ceph-mon[29442]: from='mgr.14118 v1:172.21.15.28:0/3725412590' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-05T06:28:27.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:27 smithi028 ceph-mon[29442]: from='mgr.14118 v1:172.21.15.28:0/3725412590' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-05T06:28:27.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:27 smithi028 ceph-mon[29442]: from='mgr.14118 v1:172.21.15.28:0/3725412590' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-05T06:28:27.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:27 smithi028 ceph-mon[29442]: from='mgr.14118 v1:172.21.15.28:0/3725412590' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-05T06:28:27.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:27 smithi028 ceph-mon[29442]: Manager daemon y is now available 2024-02-05T06:28:27.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:27 smithi028 ceph-mon[29442]: from='mgr.14118 v1:172.21.15.28:0/3725412590' entity='mgr.y' 2024-02-05T06:28:27.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:27 smithi028 ceph-mon[29442]: from='mgr.14118 v1:172.21.15.28:0/3725412590' entity='mgr.y' 2024-02-05T06:28:27.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:27 smithi028 ceph-mon[29442]: from='mgr.14118 v1:172.21.15.28:0/3725412590' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:28:27.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:27 smithi028 ceph-mon[29442]: from='mgr.14118 v1:172.21.15.28:0/3725412590' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:28:28.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:28 smithi028 ceph-mon[29442]: Found migration_current of "None". Setting to last migration. 2024-02-05T06:28:28.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:28 smithi028 ceph-mon[29442]: from='mgr.14118 v1:172.21.15.28:0/3725412590' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:28:28.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:28 smithi028 ceph-mon[29442]: from='mgr.14118 v1:172.21.15.28:0/3725412590' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-05T06:28:28.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:28 smithi028 ceph-mon[29442]: from='mgr.14118 v1:172.21.15.28:0/3725412590' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-05T06:28:28.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:28 smithi028 ceph-mon[29442]: from='mgr.14118 v1:172.21.15.28:0/3725412590' entity='mgr.y' 2024-02-05T06:28:28.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:28 smithi028 ceph-mon[29442]: from='mgr.14118 v1:172.21.15.28:0/3725412590' entity='mgr.y' 2024-02-05T06:28:28.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:28 smithi028 ceph-mon[29442]: mgrmap e7: y(active, since 1.00533s) 2024-02-05T06:28:29.233 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-02-05T06:28:29.234 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-02-05T06:28:29.234 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-05T06:28:29.234 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-02-05T06:28:29.234 INFO:teuthology.orchestra.run.smithi028.stdout:mgr epoch 5 is available 2024-02-05T06:28:29.234 INFO:teuthology.orchestra.run.smithi028.stdout:Setting orchestrator backend to cephadm... 2024-02-05T06:28:29.888 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:29 smithi028 ceph-mon[29442]: [05/Feb/2024:06:28:28] ENGINE Bus STARTING 2024-02-05T06:28:29.888 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:29 smithi028 ceph-mon[29442]: from='client.14122 v1:172.21.15.28:0/768350442' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-05T06:28:29.888 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:29 smithi028 ceph-mon[29442]: from='client.14122 v1:172.21.15.28:0/768350442' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-05T06:28:29.888 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:29 smithi028 ceph-mon[29442]: [05/Feb/2024:06:28:28] ENGINE Serving on http://172.21.15.28:8765 2024-02-05T06:28:29.889 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:29 smithi028 ceph-mon[29442]: from='mgr.14118 v1:172.21.15.28:0/3725412590' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:28:30.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:30 smithi028 ceph-mon[29442]: [05/Feb/2024:06:28:28] ENGINE Serving on https://172.21.15.28:7150 2024-02-05T06:28:30.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:30 smithi028 ceph-mon[29442]: [05/Feb/2024:06:28:28] ENGINE Bus STARTED 2024-02-05T06:28:30.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:30 smithi028 ceph-mon[29442]: from='mgr.14118 v1:172.21.15.28:0/3725412590' entity='mgr.y' 2024-02-05T06:28:30.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:30 smithi028 ceph-mon[29442]: from='mgr.14118 v1:172.21.15.28:0/3725412590' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:28:30.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:30 smithi028 ceph-mon[29442]: mgrmap e8: y(active, since 3s) 2024-02-05T06:28:31.586 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout value unchanged 2024-02-05T06:28:31.587 INFO:teuthology.orchestra.run.smithi028.stdout:Generating ssh key... 2024-02-05T06:28:31.894 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:31 smithi028 ceph-mon[29442]: from='client.14130 v1:172.21.15.28:0/1781157813' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:28:32.897 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:32 smithi028 ceph-mon[29442]: from='client.14132 v1:172.21.15.28:0/3930269425' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:28:32.898 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: Generating public/private rsa key pair. 2024-02-05T06:28:32.898 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: Your identification has been saved in /tmp/tmp1h8mvsi_/key. 2024-02-05T06:28:32.898 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: Your public key has been saved in /tmp/tmp1h8mvsi_/key.pub. 2024-02-05T06:28:32.898 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: The key fingerprint is: 2024-02-05T06:28:32.898 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: SHA256:O4qZafV2rix0fXbwQTJVZWQJM3sgyfdRon+fpyrVwdQ ceph-9801e956-c3ef-11ee-95b6-87774f69a715 2024-02-05T06:28:32.898 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: The key's randomart image is: 2024-02-05T06:28:32.898 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: +---[RSA 3072]----+ 2024-02-05T06:28:32.898 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: | ...*++@| 2024-02-05T06:28:32.898 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: | o+oB*E| 2024-02-05T06:28:32.898 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: | oB...| 2024-02-05T06:28:32.898 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: | ..=. | 2024-02-05T06:28:32.898 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: | S. +.o.| 2024-02-05T06:28:32.898 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: | o ... + +.o| 2024-02-05T06:28:32.898 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: | o oo + . .o| 2024-02-05T06:28:32.899 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: | .=.oo.o ..| 2024-02-05T06:28:32.899 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: | .= .oo+..... | 2024-02-05T06:28:32.899 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: +----[SHA256]-----+ 2024-02-05T06:28:33.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:33 smithi028 ceph-mon[29442]: from='client.14134 v1:172.21.15.28:0/1324316137' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:28:33.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:33 smithi028 ceph-mon[29442]: Generating ssh key... 2024-02-05T06:28:33.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:33 smithi028 ceph-mon[29442]: from='mgr.14118 v1:172.21.15.28:0/3725412590' entity='mgr.y' 2024-02-05T06:28:33.786 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:33 smithi028 ceph-mon[29442]: from='mgr.14118 v1:172.21.15.28:0/3725412590' entity='mgr.y' 2024-02-05T06:28:34.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAMM8XFmW7Z5Ha+8HCCnWuxUSEsyFJnjeKphhxJEMguQ1akw2M9qPfksTVeb8Pg8yliJvidMFum1pq6nEoOhlJY5ORAsohE0MgO2fYfiBydYO+cgKIK5SgKNM6KSa+TOaQdATznKm89KVj+ZCPrSRYZ2fkoEsDL3QIV6BzzJCvvEP1MtCLvNCTRZt3MoQ7uGEJBQuoB6CMgHwBSFf8GYM6W0i/M9QJUIHt+WOu5QPuDFJ5LgJj8G8esmGaAcGSDC43K/oGKhNh1KM0ddTXz6dh1bU6uvma+NxY+N94kbS9bZKYq1mGNh6Ew0pA/CSrqZZVXoPLYJfDqibJPchXfjbPDCUTFUg//XoOwLd5TqOLkN5ORJN2GTa7UiOTNp5SfaeuE7Pdi7YB3JGbjpKheeKPonQ0v+zgw4Byr5JyDh3tzOMZ+nZv/W+4HPwDf7RyztJ38E98mxV1g8SqwaOwckK2A8A7famQuonz6Gg9pWhkqs1Ir4jcDdptZDpPbOnXdJU= ceph-9801e956-c3ef-11ee-95b6-87774f69a715 2024-02-05T06:28:34.248 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-05T06:28:34.248 INFO:teuthology.orchestra.run.smithi028.stdout:Adding key to root@localhost authorized_keys... 2024-02-05T06:28:34.248 INFO:teuthology.orchestra.run.smithi028.stdout:Adding host smithi028... 2024-02-05T06:28:35.746 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:35 smithi028 ceph-mon[29442]: from='client.14136 v1:172.21.15.28:0/3517518036' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:28:36.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:36 smithi028 ceph-mon[29442]: from='client.14138 v1:172.21.15.28:0/3781251082' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi028", "addr": "172.21.15.28", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:28:36.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:36 smithi028 ceph-mon[29442]: Deploying cephadm binary to smithi028 2024-02-05T06:28:39.805 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Added host 'smithi028' with addr '172.21.15.28' 2024-02-05T06:28:39.805 INFO:teuthology.orchestra.run.smithi028.stdout:Deploying unmanaged mon service... 2024-02-05T06:28:40.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:40 smithi028 ceph-mon[29442]: from='mgr.14118 v1:172.21.15.28:0/3725412590' entity='mgr.y' 2024-02-05T06:28:40.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:40 smithi028 ceph-mon[29442]: Added host smithi028 2024-02-05T06:28:40.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:40 smithi028 ceph-mon[29442]: from='mgr.14118 v1:172.21.15.28:0/3725412590' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:28:41.350 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-02-05T06:28:41.351 INFO:teuthology.orchestra.run.smithi028.stdout:Deploying unmanaged mgr service... 2024-02-05T06:28:41.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:41 smithi028 ceph-mon[29442]: from='client.14140 v1:172.21.15.28:0/3831179490' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:28:41.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:41 smithi028 ceph-mon[29442]: Saving service mon spec with placement count:5 2024-02-05T06:28:41.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:41 smithi028 ceph-mon[29442]: from='mgr.14118 v1:172.21.15.28:0/3725412590' entity='mgr.y' 2024-02-05T06:28:43.176 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-02-05T06:28:43.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:43 smithi028 ceph-mon[29442]: from='mgr.14118 v1:172.21.15.28:0/3725412590' entity='mgr.y' 2024-02-05T06:28:43.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:43 smithi028 ceph-mon[29442]: from='client.14142 v1:172.21.15.28:0/4112584847' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:28:43.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:43 smithi028 ceph-mon[29442]: from='mgr.14118 v1:172.21.15.28:0/3725412590' entity='mgr.y' 2024-02-05T06:28:44.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:44 smithi028 ceph-mon[29442]: Saving service mgr spec with placement count:2 2024-02-05T06:28:44.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:44 smithi028 ceph-mon[29442]: from='mgr.14118 v1:172.21.15.28:0/3725412590' entity='mgr.y' 2024-02-05T06:28:44.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:44 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3684915738' entity='client.admin' 2024-02-05T06:28:45.809 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling the dashboard module... 2024-02-05T06:28:46.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:46 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2784185557' entity='client.admin' 2024-02-05T06:28:46.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:46 smithi028 ceph-mon[29442]: from='mgr.14118 v1:172.21.15.28:0/3725412590' entity='mgr.y' 2024-02-05T06:28:47.278 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:47 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1636238497' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-05T06:28:47.278 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:47 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ignoring --setuser ceph since I am not root 2024-02-05T06:28:47.278 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:47 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ignoring --setgroup ceph since I am not root 2024-02-05T06:28:47.828 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:47 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:47.631+0000 7f1a88b29200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-05T06:28:47.828 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:47 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:47.727+0000 7f1a88b29200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-05T06:28:48.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:48 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1636238497' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-05T06:28:48.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:48 smithi028 ceph-mon[29442]: mgrmap e9: y(active, since 19s) 2024-02-05T06:28:48.489 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:48 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:48.397+0000 7f1a88b29200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-05T06:28:48.489 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:48 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:48.492+0000 7f1a88b29200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-05T06:28:48.817 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:48 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:48.636+0000 7f1a88b29200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-05T06:28:48.817 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:48 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:48.734+0000 7f1a88b29200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-05T06:28:48.817 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:48 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:48.820+0000 7f1a88b29200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-05T06:28:48.921 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-02-05T06:28:48.922 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-02-05T06:28:48.922 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-05T06:28:48.922 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-02-05T06:28:48.923 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-05T06:28:48.923 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-02-05T06:28:48.923 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for the mgr to restart... 2024-02-05T06:28:48.923 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr epoch 9... 2024-02-05T06:28:49.091 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:49 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2076377472' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-05T06:28:49.407 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:49 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:49.094+0000 7f1a88b29200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-05T06:28:49.407 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:49 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:49.215+0000 7f1a88b29200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-05T06:28:49.407 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:49 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:49.308+0000 7f1a88b29200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-05T06:28:49.689 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:49 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:49.412+0000 7f1a88b29200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-05T06:28:49.689 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:49 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:49.597+0000 7f1a88b29200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-05T06:28:50.688 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:50 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:50.627+0000 7f1a88b29200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-05T06:28:51.094 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:50 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:50.760+0000 7f1a88b29200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-05T06:28:51.094 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:51 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:51.097+0000 7f1a88b29200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-05T06:28:51.388 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:51 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:51.189+0000 7f1a88b29200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-05T06:28:51.388 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:51 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:51.388+0000 7f1a88b29200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-05T06:28:51.688 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:51 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:51.575+0000 7f1a88b29200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-05T06:28:52.123 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:51 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:51.865+0000 7f1a88b29200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-05T06:28:52.123 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:52 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:52.126+0000 7f1a88b29200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-05T06:28:52.438 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:52 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:52.367+0000 7f1a88b29200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-05T06:28:53.438 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:53 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:53.189+0000 7f1a88b29200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-05T06:28:54.088 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:53 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:53.771+0000 7f1a88b29200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-05T06:28:54.088 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:53 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:53.871+0000 7f1a88b29200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-05T06:28:54.088 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:28:54 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:28:54.091+0000 7f1a88b29200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-05T06:28:54.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:54 smithi028 ceph-mon[29442]: Active manager daemon y restarted 2024-02-05T06:28:54.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:54 smithi028 ceph-mon[29442]: Activating manager daemon y 2024-02-05T06:28:54.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:54 smithi028 ceph-mon[29442]: osdmap e3: 0 total, 0 up, 0 in 2024-02-05T06:28:54.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:54 smithi028 ceph-mon[29442]: mgrmap e10: y(active, starting, since 0.00343437s) 2024-02-05T06:28:54.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:54 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T06:28:54.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:54 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-05T06:28:54.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:54 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-05T06:28:54.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:54 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-05T06:28:54.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:54 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-05T06:28:54.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:54 smithi028 ceph-mon[29442]: Manager daemon y is now available 2024-02-05T06:28:54.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:54 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:28:54.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:54 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:28:55.412 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:55 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-05T06:28:55.412 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:55 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-05T06:28:55.412 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:55 smithi028 ceph-mon[29442]: mgrmap e11: y(active, since 1.00638s) 2024-02-05T06:28:55.729 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-02-05T06:28:55.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-02-05T06:28:55.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-05T06:28:55.730 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-02-05T06:28:55.730 INFO:teuthology.orchestra.run.smithi028.stdout:mgr epoch 9 is available 2024-02-05T06:28:55.730 INFO:teuthology.orchestra.run.smithi028.stdout:Generating a dashboard self-signed certificate... 2024-02-05T06:28:57.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:57 smithi028 ceph-mon[29442]: [05/Feb/2024:06:28:55] ENGINE Bus STARTING 2024-02-05T06:28:57.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:57 smithi028 ceph-mon[29442]: [05/Feb/2024:06:28:55] ENGINE Serving on https://172.21.15.28:7150 2024-02-05T06:28:57.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:57 smithi028 ceph-mon[29442]: [05/Feb/2024:06:28:55] ENGINE Serving on http://172.21.15.28:8765 2024-02-05T06:28:57.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:57 smithi028 ceph-mon[29442]: [05/Feb/2024:06:28:55] ENGINE Bus STARTED 2024-02-05T06:28:57.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:57 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:28:57.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:57 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:28:57.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:57 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:28:57.428 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-02-05T06:28:57.429 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial admin user... 2024-02-05T06:28:58.335 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:58 smithi028 ceph-mon[29442]: from='client.14162 v1:172.21.15.28:0/1062951504' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:28:58.335 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:58 smithi028 ceph-mon[29442]: mgrmap e12: y(active, since 2s) 2024-02-05T06:28:58.335 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:58 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:28:58.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$A81H5fVA8OwoU4aU.YQonuHm/s6b0k0kuByAht31cfE7GyMJViPgG", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1707114538, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-05T06:28:58.852 INFO:teuthology.orchestra.run.smithi028.stdout:Fetching dashboard port number... 2024-02-05T06:28:59.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:59 smithi028 ceph-mon[29442]: from='client.14164 v1:172.21.15.28:0/4021900158' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:28:59.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:28:59 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:00.051 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 8443 2024-02-05T06:29:00.060 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-05T06:29:00.061 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-02-05T06:29:00.068 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-05T06:29:00.068 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-02-05T06:29:00.068 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-02-05T06:29:00.068 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-05T06:29:00.069 INFO:teuthology.orchestra.run.smithi028.stdout:Ceph Dashboard is now available at: 2024-02-05T06:29:00.069 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:29:00.069 INFO:teuthology.orchestra.run.smithi028.stdout: URL: https://smithi028.front.sepia.ceph.com:8443/ 2024-02-05T06:29:00.070 INFO:teuthology.orchestra.run.smithi028.stdout: User: admin 2024-02-05T06:29:00.070 INFO:teuthology.orchestra.run.smithi028.stdout: Password: ewaqadvcwu 2024-02-05T06:29:00.070 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:29:00.070 INFO:teuthology.orchestra.run.smithi028.stdout:Saving cluster configuration to /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/config directory 2024-02-05T06:29:00.070 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling autotune for osd_memory_target 2024-02-05T06:29:00.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:00 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/14135400' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-05T06:29:02.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:02 smithi028 ceph-mon[29442]: mgrmap e13: y(active, since 6s) 2024-02-05T06:29:02.927 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-02-05T06:29:02.927 INFO:teuthology.orchestra.run.smithi028.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-02-05T06:29:02.927 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:29:02.927 INFO:teuthology.orchestra.run.smithi028.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-05T06:29:02.927 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:29:02.927 INFO:teuthology.orchestra.run.smithi028.stdout:Or, if you are only running a single cluster on this host: 2024-02-05T06:29:02.927 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:29:02.927 INFO:teuthology.orchestra.run.smithi028.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-02-05T06:29:02.927 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:29:02.928 INFO:teuthology.orchestra.run.smithi028.stdout:Please consider enabling telemetry to help improve Ceph: 2024-02-05T06:29:02.928 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:29:02.928 INFO:teuthology.orchestra.run.smithi028.stdout: ceph telemetry on 2024-02-05T06:29:02.928 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:29:02.928 INFO:teuthology.orchestra.run.smithi028.stdout:For more information see: 2024-02-05T06:29:02.928 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:29:02.928 INFO:teuthology.orchestra.run.smithi028.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-02-05T06:29:02.928 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:29:02.928 INFO:teuthology.orchestra.run.smithi028.stdout:Bootstrap complete. 2024-02-05T06:29:02.972 INFO:tasks.cephadm:Fetching config... 2024-02-05T06:29:02.972 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-05T06:29:02.972 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-05T06:29:02.987 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-05T06:29:02.988 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-05T06:29:02.988 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-05T06:29:03.042 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-05T06:29:03.043 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-05T06:29:03.043 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/keyring of=/dev/stdout 2024-02-05T06:29:03.109 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-05T06:29:03.109 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-05T06:29:03.109 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-05T06:29:03.163 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-05T06:29:03.164 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAMM8XFmW7Z5Ha+8HCCnWuxUSEsyFJnjeKphhxJEMguQ1akw2M9qPfksTVeb8Pg8yliJvidMFum1pq6nEoOhlJY5ORAsohE0MgO2fYfiBydYO+cgKIK5SgKNM6KSa+TOaQdATznKm89KVj+ZCPrSRYZ2fkoEsDL3QIV6BzzJCvvEP1MtCLvNCTRZt3MoQ7uGEJBQuoB6CMgHwBSFf8GYM6W0i/M9QJUIHt+WOu5QPuDFJ5LgJj8G8esmGaAcGSDC43K/oGKhNh1KM0ddTXz6dh1bU6uvma+NxY+N94kbS9bZKYq1mGNh6Ew0pA/CSrqZZVXoPLYJfDqibJPchXfjbPDCUTFUg//XoOwLd5TqOLkN5ORJN2GTa7UiOTNp5SfaeuE7Pdi7YB3JGbjpKheeKPonQ0v+zgw4Byr5JyDh3tzOMZ+nZv/W+4HPwDf7RyztJ38E98mxV1g8SqwaOwckK2A8A7famQuonz6Gg9pWhkqs1Ir4jcDdptZDpPbOnXdJU= ceph-9801e956-c3ef-11ee-95b6-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-05T06:29:03.250 INFO:teuthology.orchestra.run.smithi028.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAMM8XFmW7Z5Ha+8HCCnWuxUSEsyFJnjeKphhxJEMguQ1akw2M9qPfksTVeb8Pg8yliJvidMFum1pq6nEoOhlJY5ORAsohE0MgO2fYfiBydYO+cgKIK5SgKNM6KSa+TOaQdATznKm89KVj+ZCPrSRYZ2fkoEsDL3QIV6BzzJCvvEP1MtCLvNCTRZt3MoQ7uGEJBQuoB6CMgHwBSFf8GYM6W0i/M9QJUIHt+WOu5QPuDFJ5LgJj8G8esmGaAcGSDC43K/oGKhNh1KM0ddTXz6dh1bU6uvma+NxY+N94kbS9bZKYq1mGNh6Ew0pA/CSrqZZVXoPLYJfDqibJPchXfjbPDCUTFUg//XoOwLd5TqOLkN5ORJN2GTa7UiOTNp5SfaeuE7Pdi7YB3JGbjpKheeKPonQ0v+zgw4Byr5JyDh3tzOMZ+nZv/W+4HPwDf7RyztJ38E98mxV1g8SqwaOwckK2A8A7famQuonz6Gg9pWhkqs1Ir4jcDdptZDpPbOnXdJU= ceph-9801e956-c3ef-11ee-95b6-87774f69a715 2024-02-05T06:29:03.265 DEBUG:teuthology.orchestra.run.smithi167:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAMM8XFmW7Z5Ha+8HCCnWuxUSEsyFJnjeKphhxJEMguQ1akw2M9qPfksTVeb8Pg8yliJvidMFum1pq6nEoOhlJY5ORAsohE0MgO2fYfiBydYO+cgKIK5SgKNM6KSa+TOaQdATznKm89KVj+ZCPrSRYZ2fkoEsDL3QIV6BzzJCvvEP1MtCLvNCTRZt3MoQ7uGEJBQuoB6CMgHwBSFf8GYM6W0i/M9QJUIHt+WOu5QPuDFJ5LgJj8G8esmGaAcGSDC43K/oGKhNh1KM0ddTXz6dh1bU6uvma+NxY+N94kbS9bZKYq1mGNh6Ew0pA/CSrqZZVXoPLYJfDqibJPchXfjbPDCUTFUg//XoOwLd5TqOLkN5ORJN2GTa7UiOTNp5SfaeuE7Pdi7YB3JGbjpKheeKPonQ0v+zgw4Byr5JyDh3tzOMZ+nZv/W+4HPwDf7RyztJ38E98mxV1g8SqwaOwckK2A8A7famQuonz6Gg9pWhkqs1Ir4jcDdptZDpPbOnXdJU= ceph-9801e956-c3ef-11ee-95b6-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-05T06:29:03.311 INFO:teuthology.orchestra.run.smithi167.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAMM8XFmW7Z5Ha+8HCCnWuxUSEsyFJnjeKphhxJEMguQ1akw2M9qPfksTVeb8Pg8yliJvidMFum1pq6nEoOhlJY5ORAsohE0MgO2fYfiBydYO+cgKIK5SgKNM6KSa+TOaQdATznKm89KVj+ZCPrSRYZ2fkoEsDL3QIV6BzzJCvvEP1MtCLvNCTRZt3MoQ7uGEJBQuoB6CMgHwBSFf8GYM6W0i/M9QJUIHt+WOu5QPuDFJ5LgJj8G8esmGaAcGSDC43K/oGKhNh1KM0ddTXz6dh1bU6uvma+NxY+N94kbS9bZKYq1mGNh6Ew0pA/CSrqZZVXoPLYJfDqibJPchXfjbPDCUTFUg//XoOwLd5TqOLkN5ORJN2GTa7UiOTNp5SfaeuE7Pdi7YB3JGbjpKheeKPonQ0v+zgw4Byr5JyDh3tzOMZ+nZv/W+4HPwDf7RyztJ38E98mxV1g8SqwaOwckK2A8A7famQuonz6Gg9pWhkqs1Ir4jcDdptZDpPbOnXdJU= ceph-9801e956-c3ef-11ee-95b6-87774f69a715 2024-02-05T06:29:03.326 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-05T06:29:03.568 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:29:03.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:03 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2711103958' entity='client.admin' 2024-02-05T06:29:06.079 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:06 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:06.079 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:06 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:06.079 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:06 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi028", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:29:06.079 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:06 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:06.079 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:06 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:29:06.079 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:06 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:06.673 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-05T06:29:06.673 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-05T06:29:06.912 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:29:07.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:07 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3126465422' entity='client.admin' 2024-02-05T06:29:07.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:07 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:29:07.330 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:07 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:09.303 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi167 2024-02-05T06:29:09.303 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-02-05T06:29:09.303 DEBUG:teuthology.orchestra.run.smithi167:> dd of=/etc/ceph/ceph.conf 2024-02-05T06:29:09.321 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-02-05T06:29:09.321 DEBUG:teuthology.orchestra.run.smithi167:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-05T06:29:09.379 INFO:tasks.cephadm:Adding host smithi167 to orchestrator... 2024-02-05T06:29:09.379 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph orch host add smithi167 2024-02-05T06:29:09.617 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:29:09.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:09 smithi028 ceph-mon[29442]: from='client.14174 v1:172.21.15.28:0/3701377158' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:29:09.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:09 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:09.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:09 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:29:09.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:09 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:09.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:09 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:29:09.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:09 smithi028 ceph-mon[29442]: Updating smithi028:/etc/ceph/ceph.conf 2024-02-05T06:29:09.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:09 smithi028 ceph-mon[29442]: Updating smithi028:/var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T06:29:09.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:09 smithi028 ceph-mon[29442]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-02-05T06:29:09.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:09 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:09.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:09 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:09.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:09 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:10.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:10 smithi028 ceph-mon[29442]: Updating smithi028:/var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/config/ceph.client.admin.keyring 2024-02-05T06:29:12.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:12 smithi028 ceph-mon[29442]: from='client.14176 v1:172.21.15.28:0/1945395919' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi167", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:29:12.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:12 smithi028 ceph-mon[29442]: Deploying cephadm binary to smithi167 2024-02-05T06:29:15.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:15 smithi028 ceph-mon[29442]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:15.780 INFO:teuthology.orchestra.run.smithi028.stdout:Added host 'smithi167' with addr '172.21.15.167' 2024-02-05T06:29:16.413 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph orch host ls --format=json 2024-02-05T06:29:16.656 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:29:17.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:16 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:17.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:16 smithi028 ceph-mon[29442]: Added host smithi167 2024-02-05T06:29:17.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:16 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:29:17.076 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:16 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:18.099 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:17 smithi028 ceph-mon[29442]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:18.099 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:17 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:18.336 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:29:18.336 INFO:teuthology.orchestra.run.smithi028.stdout:[{"addr": "172.21.15.28", "hostname": "smithi028", "labels": [], "status": ""}, {"addr": "172.21.15.167", "hostname": "smithi167", "labels": [], "status": ""}] 2024-02-05T06:29:18.945 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-05T06:29:18.946 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd crush tunables default 2024-02-05T06:29:19.181 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:29:19.581 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:19 smithi028 ceph-mon[29442]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:19.581 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:19 smithi028 ceph-mon[29442]: from='client.14178 v1:172.21.15.28:0/406606638' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:29:19.581 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:19 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:21.573 INFO:teuthology.orchestra.run.smithi028.stderr:adjusted tunables profile to default 2024-02-05T06:29:21.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:21 smithi028 ceph-mon[29442]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:21.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:21 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3147000898' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-05T06:29:22.255 INFO:tasks.cephadm:Adding mon.a on smithi028 2024-02-05T06:29:22.255 INFO:tasks.cephadm:Adding mon.c on smithi028 2024-02-05T06:29:22.255 INFO:tasks.cephadm:Adding mon.b on smithi167 2024-02-05T06:29:22.255 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph orch apply mon '3;smithi028:[v1:172.21.15.28:6789]=a;smithi028:[v1:172.21.15.28:6790]=c;smithi167:[v1:172.21.15.167:6789]=b' 2024-02-05T06:29:22.490 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T06:29:22.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:22 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3147000898' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-05T06:29:22.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:22 smithi028 ceph-mon[29442]: osdmap e4: 0 total, 0 up, 0 in 2024-02-05T06:29:22.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:22 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:22.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:22 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:22.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:22 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:22.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:22 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:22.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:22 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi167", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:29:22.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:22 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:22.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:22 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:29:22.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:22 smithi028 ceph-mon[29442]: Updating smithi167:/etc/ceph/ceph.conf 2024-02-05T06:29:22.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:22 smithi028 ceph-mon[29442]: Updating smithi167:/var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T06:29:23.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:23 smithi028 ceph-mon[29442]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:23.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:23 smithi028 ceph-mon[29442]: Updating smithi167:/etc/ceph/ceph.client.admin.keyring 2024-02-05T06:29:23.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:23 smithi028 ceph-mon[29442]: Updating smithi167:/var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/config/ceph.client.admin.keyring 2024-02-05T06:29:23.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:23 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:23.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:23 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:23.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:23 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:24.084 INFO:teuthology.orchestra.run.smithi167.stdout:Scheduled mon update... 2024-02-05T06:29:24.720 DEBUG:teuthology.orchestra.run.smithi028:mon.c> sudo journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@mon.c.service 2024-02-05T06:29:24.723 DEBUG:teuthology.orchestra.run.smithi167:mon.b> sudo journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@mon.b.service 2024-02-05T06:29:24.725 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-05T06:29:24.725 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph mon dump -f json 2024-02-05T06:29:25.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:25 smithi028 ceph-mon[29442]: from='client.14182 v1:172.21.15.167:0/2948621679' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:[v1:172.21.15.28:6789]=a;smithi028:[v1:172.21.15.28:6790]=c;smithi167:[v1:172.21.15.167:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:29:25.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:25 smithi028 ceph-mon[29442]: Saving service mon spec with placement smithi028:[v1:172.21.15.28:6789]=a;smithi028:[v1:172.21.15.28:6790]=c;smithi167:[v1:172.21.15.167:6789]=b;count:3 2024-02-05T06:29:25.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:25 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:25.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:25 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:29:25.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:25 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:25.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:25 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:29:25.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:25 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:25.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:25 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T06:29:25.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:25 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:25.483 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T06:29:26.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:26 smithi028 ceph-mon[29442]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:26.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:26 smithi028 ceph-mon[29442]: Deploying daemon mon.b on smithi167 2024-02-05T06:29:27.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:27 smithi028 ceph-mon[29442]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:28.436 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:29:28.436 INFO:teuthology.orchestra.run.smithi167.stdout:{"epoch":1,"fsid":"9801e956-c3ef-11ee-95b6-87774f69a715","modified":"2024-02-05T06:27:56.979391Z","created":"2024-02-05T06:27:56.979391Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-05T06:29:28.438 INFO:teuthology.orchestra.run.smithi167.stderr:dumped monmap epoch 1 2024-02-05T06:29:28.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:28 smithi167 systemd[1]: Starting Ceph mon.b for 9801e956-c3ef-11ee-95b6-87774f69a715... 2024-02-05T06:29:29.412 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 podman[32755]: 2024-02-05 06:29:29.207407634 +0000 UTC m=+0.018166373 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T06:29:29.412 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 podman[32755]: 2024-02-05 06:29:29.308144535 +0000 UTC m=+0.118903277 container create 7cbbabf936992d710f8e034d38403e33c1c62092063512166f4a96f0df2efce2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-b, org.label-schema.build-date=20240131, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.vendor=CentOS) 2024-02-05T06:29:29.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:29 smithi028 ceph-mon[29442]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:29.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:29 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.167:0/156664896' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T06:29:29.724 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 podman[32755]: 2024-02-05 06:29:29.544164302 +0000 UTC m=+0.354923043 container init 7cbbabf936992d710f8e034d38403e33c1c62092063512166f4a96f0df2efce2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-b, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_CLEAN=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-05T06:29:29.724 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 podman[32755]: 2024-02-05 06:29:29.547325733 +0000 UTC m=+0.358084475 container start 7cbbabf936992d710f8e034d38403e33c1c62092063512166f4a96f0df2efce2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True) 2024-02-05T06:29:29.725 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: set uid:gid to 167:167 (ceph:ceph) 2024-02-05T06:29:29.725 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: ceph version 19.0.0-1241-g164d3be4 (164d3be4a527be775f4d3c343ea591122f1b8921) squid (dev), process ceph-mon, pid 2 2024-02-05T06:29:29.725 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: pidfile_write: ignore empty --pid-file 2024-02-05T06:29:29.725 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: load: jerasure load: lrc 2024-02-05T06:29:29.725 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: RocksDB version: 7.9.2 2024-02-05T06:29:29.725 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Git sha 0 2024-02-05T06:29:29.725 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Compile date 2024-02-04 23:11:06 2024-02-05T06:29:29.725 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: DB SUMMARY 2024-02-05T06:29:29.725 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: DB Session ID: FDHP9LNG4CFT435OQW8Q 2024-02-05T06:29:29.725 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: CURRENT file: CURRENT 2024-02-05T06:29:29.725 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: IDENTITY file: IDENTITY 2024-02-05T06:29:29.725 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-02-05T06:29:29.725 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-02-05T06:29:29.725 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 476 ; 2024-02-05T06:29:29.725 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.error_if_exists: 0 2024-02-05T06:29:29.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.create_if_missing: 0 2024-02-05T06:29:29.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.paranoid_checks: 1 2024-02-05T06:29:29.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-05T06:29:29.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-05T06:29:29.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-05T06:29:29.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.env: 0x557a2e1e6ae0 2024-02-05T06:29:29.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.fs: PosixFileSystem 2024-02-05T06:29:29.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.info_log: 0x557a30955920 2024-02-05T06:29:29.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.max_file_opening_threads: 16 2024-02-05T06:29:29.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.statistics: (nil) 2024-02-05T06:29:29.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.use_fsync: 0 2024-02-05T06:29:29.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.max_log_file_size: 0 2024-02-05T06:29:29.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-05T06:29:29.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-05T06:29:29.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.keep_log_file_num: 1000 2024-02-05T06:29:29.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.recycle_log_file_num: 0 2024-02-05T06:29:29.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.allow_fallocate: 1 2024-02-05T06:29:29.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.allow_mmap_reads: 0 2024-02-05T06:29:29.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.allow_mmap_writes: 0 2024-02-05T06:29:29.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.use_direct_reads: 0 2024-02-05T06:29:29.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-05T06:29:29.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.create_missing_column_families: 0 2024-02-05T06:29:29.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.db_log_dir: 2024-02-05T06:29:29.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.wal_dir: 2024-02-05T06:29:29.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-05T06:29:29.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-05T06:29:29.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-05T06:29:29.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-05T06:29:29.728 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-05T06:29:29.728 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-05T06:29:29.728 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.advise_random_on_open: 1 2024-02-05T06:29:29.728 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.db_write_buffer_size: 0 2024-02-05T06:29:29.728 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.write_buffer_manager: 0x557a30929400 2024-02-05T06:29:29.728 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-05T06:29:29.728 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-05T06:29:29.728 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-05T06:29:29.728 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.rate_limiter: (nil) 2024-02-05T06:29:29.728 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-05T06:29:29.728 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.wal_recovery_mode: 2 2024-02-05T06:29:29.728 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.enable_thread_tracking: 0 2024-02-05T06:29:29.728 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.enable_pipelined_write: 0 2024-02-05T06:29:29.728 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.unordered_write: 0 2024-02-05T06:29:29.728 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-05T06:29:29.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-05T06:29:29.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-05T06:29:29.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-05T06:29:29.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.row_cache: None 2024-02-05T06:29:29.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.wal_filter: None 2024-02-05T06:29:29.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-05T06:29:29.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.allow_ingest_behind: 0 2024-02-05T06:29:29.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.two_write_queues: 0 2024-02-05T06:29:29.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.manual_wal_flush: 0 2024-02-05T06:29:29.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.wal_compression: 0 2024-02-05T06:29:29.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.atomic_flush: 0 2024-02-05T06:29:29.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-05T06:29:29.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-05T06:29:29.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-05T06:29:29.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.log_readahead_size: 0 2024-02-05T06:29:29.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-05T06:29:29.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.best_efforts_recovery: 0 2024-02-05T06:29:29.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-05T06:29:29.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-05T06:29:29.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.allow_data_in_errors: 0 2024-02-05T06:29:29.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.db_host_id: __hostname__ 2024-02-05T06:29:29.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-05T06:29:29.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.max_background_jobs: 2 2024-02-05T06:29:29.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.max_background_compactions: -1 2024-02-05T06:29:29.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.max_subcompactions: 1 2024-02-05T06:29:29.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-05T06:29:29.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-05T06:29:29.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-05T06:29:29.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.max_total_wal_size: 0 2024-02-05T06:29:29.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-05T06:29:29.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-05T06:29:29.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-05T06:29:29.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-05T06:29:29.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.max_open_files: -1 2024-02-05T06:29:29.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.bytes_per_sync: 0 2024-02-05T06:29:29.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-05T06:29:29.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-05T06:29:29.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.compaction_readahead_size: 0 2024-02-05T06:29:29.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.max_background_flushes: -1 2024-02-05T06:29:29.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Compression algorithms supported: 2024-02-05T06:29:29.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-05T06:29:29.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: kZSTD supported: 0 2024-02-05T06:29:29.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: kXpressCompression supported: 0 2024-02-05T06:29:29.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: kLZ4HCCompression supported: 1 2024-02-05T06:29:29.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: kZlibCompression supported: 1 2024-02-05T06:29:29.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: kSnappyCompression supported: 1 2024-02-05T06:29:29.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: kLZ4Compression supported: 1 2024-02-05T06:29:29.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: kBZip2Compression supported: 0 2024-02-05T06:29:29.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-05T06:29:29.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-05T06:29:29.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-02-05T06:29:29.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-05T06:29:29.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-05T06:29:29.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.merge_operator: 2024-02-05T06:29:29.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.compaction_filter: None 2024-02-05T06:29:29.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.compaction_filter_factory: None 2024-02-05T06:29:29.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.sst_partitioner_factory: None 2024-02-05T06:29:29.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-05T06:29:29.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-05T06:29:29.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557a30955a20) 2024-02-05T06:29:29.734 INFO:journalctl@ceph.mon.b.smithi167.stdout: cache_index_and_filter_blocks: 1 2024-02-05T06:29:29.734 INFO:journalctl@ceph.mon.b.smithi167.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-05T06:29:29.734 INFO:journalctl@ceph.mon.b.smithi167.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-05T06:29:29.734 INFO:journalctl@ceph.mon.b.smithi167.stdout: pin_top_level_index_and_filter: 1 2024-02-05T06:29:29.734 INFO:journalctl@ceph.mon.b.smithi167.stdout: index_type: 0 2024-02-05T06:29:29.734 INFO:journalctl@ceph.mon.b.smithi167.stdout: data_block_index_type: 0 2024-02-05T06:29:29.734 INFO:journalctl@ceph.mon.b.smithi167.stdout: index_shortening: 1 2024-02-05T06:29:29.734 INFO:journalctl@ceph.mon.b.smithi167.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-05T06:29:29.734 INFO:journalctl@ceph.mon.b.smithi167.stdout: checksum: 4 2024-02-05T06:29:29.734 INFO:journalctl@ceph.mon.b.smithi167.stdout: no_block_cache: 0 2024-02-05T06:29:29.734 INFO:journalctl@ceph.mon.b.smithi167.stdout: block_cache: 0x557a30959090 2024-02-05T06:29:29.734 INFO:journalctl@ceph.mon.b.smithi167.stdout: block_cache_name: BinnedLRUCache 2024-02-05T06:29:29.734 INFO:journalctl@ceph.mon.b.smithi167.stdout: block_cache_options: 2024-02-05T06:29:29.734 INFO:journalctl@ceph.mon.b.smithi167.stdout: capacity : 536870912 2024-02-05T06:29:29.734 INFO:journalctl@ceph.mon.b.smithi167.stdout: num_shard_bits : 4 2024-02-05T06:29:29.734 INFO:journalctl@ceph.mon.b.smithi167.stdout: strict_capacity_limit : 0 2024-02-05T06:29:29.734 INFO:journalctl@ceph.mon.b.smithi167.stdout: high_pri_pool_ratio: 0.000 2024-02-05T06:29:29.735 INFO:journalctl@ceph.mon.b.smithi167.stdout: block_cache_compressed: (nil) 2024-02-05T06:29:29.735 INFO:journalctl@ceph.mon.b.smithi167.stdout: persistent_cache: (nil) 2024-02-05T06:29:29.735 INFO:journalctl@ceph.mon.b.smithi167.stdout: block_size: 4096 2024-02-05T06:29:29.735 INFO:journalctl@ceph.mon.b.smithi167.stdout: block_size_deviation: 10 2024-02-05T06:29:29.735 INFO:journalctl@ceph.mon.b.smithi167.stdout: block_restart_interval: 16 2024-02-05T06:29:29.735 INFO:journalctl@ceph.mon.b.smithi167.stdout: index_block_restart_interval: 1 2024-02-05T06:29:29.735 INFO:journalctl@ceph.mon.b.smithi167.stdout: metadata_block_size: 4096 2024-02-05T06:29:29.735 INFO:journalctl@ceph.mon.b.smithi167.stdout: partition_filters: 0 2024-02-05T06:29:29.735 INFO:journalctl@ceph.mon.b.smithi167.stdout: use_delta_encoding: 1 2024-02-05T06:29:29.735 INFO:journalctl@ceph.mon.b.smithi167.stdout: filter_policy: bloomfilter 2024-02-05T06:29:29.735 INFO:journalctl@ceph.mon.b.smithi167.stdout: whole_key_filtering: 1 2024-02-05T06:29:29.735 INFO:journalctl@ceph.mon.b.smithi167.stdout: verify_compression: 0 2024-02-05T06:29:29.735 INFO:journalctl@ceph.mon.b.smithi167.stdout: read_amp_bytes_per_bit: 0 2024-02-05T06:29:29.735 INFO:journalctl@ceph.mon.b.smithi167.stdout: format_version: 5 2024-02-05T06:29:29.735 INFO:journalctl@ceph.mon.b.smithi167.stdout: enable_index_compression: 1 2024-02-05T06:29:29.735 INFO:journalctl@ceph.mon.b.smithi167.stdout: block_align: 0 2024-02-05T06:29:29.735 INFO:journalctl@ceph.mon.b.smithi167.stdout: max_auto_readahead_size: 262144 2024-02-05T06:29:29.736 INFO:journalctl@ceph.mon.b.smithi167.stdout: prepopulate_block_cache: 0 2024-02-05T06:29:29.736 INFO:journalctl@ceph.mon.b.smithi167.stdout: initial_auto_readahead_size: 8192 2024-02-05T06:29:29.736 INFO:journalctl@ceph.mon.b.smithi167.stdout: num_file_reads_for_auto_readahead: 2 2024-02-05T06:29:29.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.write_buffer_size: 33554432 2024-02-05T06:29:29.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.max_write_buffer_number: 2 2024-02-05T06:29:29.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.compression: NoCompression 2024-02-05T06:29:29.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.bottommost_compression: Disabled 2024-02-05T06:29:29.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.prefix_extractor: nullptr 2024-02-05T06:29:29.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-05T06:29:29.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.num_levels: 7 2024-02-05T06:29:29.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-05T06:29:29.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-05T06:29:29.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-05T06:29:29.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-05T06:29:29.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-05T06:29:29.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-05T06:29:29.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-05T06:29:29.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-05T06:29:29.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-05T06:29:29.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-05T06:29:29.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-05T06:29:29.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-05T06:29:29.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-05T06:29:29.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.compression_opts.level: 32767 2024-02-05T06:29:29.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.compression_opts.strategy: 0 2024-02-05T06:29:29.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-05T06:29:29.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-05T06:29:29.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-05T06:29:29.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-05T06:29:29.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.compression_opts.enabled: false 2024-02-05T06:29:29.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-05T06:29:29.738 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-05T06:29:29.738 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-05T06:29:29.738 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-05T06:29:29.738 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.target_file_size_base: 67108864 2024-02-05T06:29:29.738 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-05T06:29:29.738 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-05T06:29:29.738 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-05T06:29:29.738 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-05T06:29:29.738 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-05T06:29:29.738 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-05T06:29:29.738 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-05T06:29:29.738 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-05T06:29:29.738 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-05T06:29:29.738 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-05T06:29:29.738 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-05T06:29:29.738 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-05T06:29:29.738 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-05T06:29:29.739 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-05T06:29:29.739 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.arena_block_size: 1048576 2024-02-05T06:29:29.739 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-05T06:29:29.739 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-05T06:29:29.739 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.disable_auto_compactions: 0 2024-02-05T06:29:29.739 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-05T06:29:29.739 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-05T06:29:29.739 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-05T06:29:29.739 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-05T06:29:29.739 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-05T06:29:29.739 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-05T06:29:29.739 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-05T06:29:29.739 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-05T06:29:29.739 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-05T06:29:29.739 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-05T06:29:29.739 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.table_properties_collectors: 2024-02-05T06:29:29.739 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.inplace_update_support: 0 2024-02-05T06:29:29.740 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-05T06:29:29.740 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-05T06:29:29.740 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-05T06:29:29.740 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-05T06:29:29.740 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.bloom_locality: 0 2024-02-05T06:29:29.740 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.max_successive_merges: 0 2024-02-05T06:29:29.740 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-05T06:29:29.740 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.paranoid_file_checks: 0 2024-02-05T06:29:29.740 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.force_consistency_checks: 1 2024-02-05T06:29:29.740 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.report_bg_io_stats: 0 2024-02-05T06:29:29.740 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.ttl: 2592000 2024-02-05T06:29:29.740 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-05T06:29:29.740 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-05T06:29:29.740 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-05T06:29:29.741 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.enable_blob_files: false 2024-02-05T06:29:29.741 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.min_blob_size: 0 2024-02-05T06:29:29.741 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.blob_file_size: 268435456 2024-02-05T06:29:29.741 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-05T06:29:29.741 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-05T06:29:29.741 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-05T06:29:29.741 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-05T06:29:29.741 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-05T06:29:29.741 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.blob_file_starting_level: 0 2024-02-05T06:29:29.741 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-05T06:29:29.741 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-05T06:29:29.741 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-02-05T06:29:29.741 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 0f2f6168-d9eb-4c36-88cd-67bae8ebfed1 2024-02-05T06:29:29.741 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707114569576617, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-02-05T06:29:29.741 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-02-05T06:29:29.742 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707114569577031, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1587, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 488, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 366, "raw_average_value_size": 73, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707114569, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "0f2f6168-d9eb-4c36-88cd-67bae8ebfed1", "db_session_id": "FDHP9LNG4CFT435OQW8Q", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-02-05T06:29:29.742 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707114569577111, "job": 1, "event": "recovery_finished"} 2024-02-05T06:29:29.742 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-02-05T06:29:29.742 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-05T06:29:29.742 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x557a30a4a000 2024-02-05T06:29:29.742 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: DB pointer 0x557a30a34000 2024-02-05T06:29:29.742 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-05T06:29:29.742 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-05T06:29:29.742 INFO:journalctl@ceph.mon.b.smithi167.stdout: ** DB Stats ** 2024-02-05T06:29:29.742 INFO:journalctl@ceph.mon.b.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-05T06:29:29.742 INFO:journalctl@ceph.mon.b.smithi167.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-05T06:29:29.742 INFO:journalctl@ceph.mon.b.smithi167.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-05T06:29:29.742 INFO:journalctl@ceph.mon.b.smithi167.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-05T06:29:29.742 INFO:journalctl@ceph.mon.b.smithi167.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-05T06:29:29.742 INFO:journalctl@ceph.mon.b.smithi167.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-05T06:29:29.742 INFO:journalctl@ceph.mon.b.smithi167.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-05T06:29:29.743 INFO:journalctl@ceph.mon.b.smithi167.stdout: 2024-02-05T06:29:29.743 INFO:journalctl@ceph.mon.b.smithi167.stdout: ** Compaction Stats [default] ** 2024-02-05T06:29:29.743 INFO:journalctl@ceph.mon.b.smithi167.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-05T06:29:29.743 INFO:journalctl@ceph.mon.b.smithi167.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-05T06:29:29.743 INFO:journalctl@ceph.mon.b.smithi167.stdout: L0 1/0 1.55 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-05T06:29:29.743 INFO:journalctl@ceph.mon.b.smithi167.stdout: Sum 1/0 1.55 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-05T06:29:29.743 INFO:journalctl@ceph.mon.b.smithi167.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-05T06:29:29.743 INFO:journalctl@ceph.mon.b.smithi167.stdout: 2024-02-05T06:29:29.743 INFO:journalctl@ceph.mon.b.smithi167.stdout: ** Compaction Stats [default] ** 2024-02-05T06:29:29.743 INFO:journalctl@ceph.mon.b.smithi167.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-05T06:29:29.743 INFO:journalctl@ceph.mon.b.smithi167.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-05T06:29:29.743 INFO:journalctl@ceph.mon.b.smithi167.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-05T06:29:29.743 INFO:journalctl@ceph.mon.b.smithi167.stdout: 2024-02-05T06:29:29.743 INFO:journalctl@ceph.mon.b.smithi167.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-05T06:29:29.744 INFO:journalctl@ceph.mon.b.smithi167.stdout: 2024-02-05T06:29:29.744 INFO:journalctl@ceph.mon.b.smithi167.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-05T06:29:29.744 INFO:journalctl@ceph.mon.b.smithi167.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-05T06:29:29.744 INFO:journalctl@ceph.mon.b.smithi167.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-05T06:29:29.744 INFO:journalctl@ceph.mon.b.smithi167.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-05T06:29:29.744 INFO:journalctl@ceph.mon.b.smithi167.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-05T06:29:29.744 INFO:journalctl@ceph.mon.b.smithi167.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-05T06:29:29.744 INFO:journalctl@ceph.mon.b.smithi167.stdout: Cumulative compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-05T06:29:29.744 INFO:journalctl@ceph.mon.b.smithi167.stdout: Interval compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-05T06:29:29.744 INFO:journalctl@ceph.mon.b.smithi167.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-05T06:29:29.744 INFO:journalctl@ceph.mon.b.smithi167.stdout: Block cache BinnedLRUCache@0x557a30959090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-02-05T06:29:29.744 INFO:journalctl@ceph.mon.b.smithi167.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-02-05T06:29:29.744 INFO:journalctl@ceph.mon.b.smithi167.stdout: 2024-02-05T06:29:29.744 INFO:journalctl@ceph.mon.b.smithi167.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-05T06:29:29.744 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-02-05T06:29:29.745 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: using public_addrv v1:172.21.15.167:6789/0 2024-02-05T06:29:29.745 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: starting mon.b rank -1 at public addrs v1:172.21.15.167:6789/0 at bind addrs v1:172.21.15.167:6789/0 mon_data /var/lib/ceph/mon/ceph-b fsid 9801e956-c3ef-11ee-95b6-87774f69a715 2024-02-05T06:29:29.745 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: mon.b@-1(???) e0 preinit fsid 9801e956-c3ef-11ee-95b6-87774f69a715 2024-02-05T06:29:29.745 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: mon.b@-1(synchronizing).mds e1 new map 2024-02-05T06:29:29.745 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: mon.b@-1(synchronizing).mds e1 print_map 2024-02-05T06:29:29.745 INFO:journalctl@ceph.mon.b.smithi167.stdout: e1 2024-02-05T06:29:29.745 INFO:journalctl@ceph.mon.b.smithi167.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-05T06:29:29.745 INFO:journalctl@ceph.mon.b.smithi167.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-05T06:29:29.745 INFO:journalctl@ceph.mon.b.smithi167.stdout: legacy client fscid: -1 2024-02-05T06:29:29.745 INFO:journalctl@ceph.mon.b.smithi167.stdout: 2024-02-05T06:29:29.745 INFO:journalctl@ceph.mon.b.smithi167.stdout: No filesystems configured 2024-02-05T06:29:29.745 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-05T06:29:29.745 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-05T06:29:29.746 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-05T06:29:29.746 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-05T06:29:29.746 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-05T06:29:29.746 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-05T06:29:29.746 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-05T06:29:29.746 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-05T06:29:29.746 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-05T06:29:29.746 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-05T06:29:29.746 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3147000898' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-05T06:29:29.746 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: osdmap e4: 0 total, 0 up, 0 in 2024-02-05T06:29:29.746 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:29.746 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:29.746 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:29.746 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:29.746 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi167", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:29:29.746 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:29.746 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:29:29.747 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: Updating smithi167:/etc/ceph/ceph.conf 2024-02-05T06:29:29.747 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: Updating smithi167:/var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T06:29:29.747 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:29.747 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: Updating smithi167:/etc/ceph/ceph.client.admin.keyring 2024-02-05T06:29:29.747 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: Updating smithi167:/var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/config/ceph.client.admin.keyring 2024-02-05T06:29:29.747 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:29.747 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:29.747 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:29.747 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: from='client.14182 v1:172.21.15.167:0/2948621679' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:[v1:172.21.15.28:6789]=a;smithi028:[v1:172.21.15.28:6790]=c;smithi167:[v1:172.21.15.167:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:29:29.747 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: Saving service mon spec with placement smithi028:[v1:172.21.15.28:6789]=a;smithi028:[v1:172.21.15.28:6790]=c;smithi167:[v1:172.21.15.167:6789]=b;count:3 2024-02-05T06:29:29.747 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:29.747 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:29:29.747 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:29.747 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:29:29.747 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:29.747 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T06:29:29.748 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:29.748 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:29.748 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: Deploying daemon mon.b on smithi167 2024-02-05T06:29:29.748 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:29.748 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:29.748 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.167:0/156664896' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T06:29:29.748 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 ceph-mon[32774]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-02-05T06:29:29.748 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 bash[32755]: 7cbbabf936992d710f8e034d38403e33c1c62092063512166f4a96f0df2efce2 2024-02-05T06:29:29.748 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:29 smithi167 systemd[1]: Started Ceph mon.b for 9801e956-c3ef-11ee-95b6-87774f69a715. 2024-02-05T06:29:30.126 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-05T06:29:30.127 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph mon dump -f json 2024-02-05T06:29:30.361 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:29:34.932 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:29:34.933 INFO:teuthology.orchestra.run.smithi167.stdout:{"epoch":2,"fsid":"9801e956-c3ef-11ee-95b6-87774f69a715","modified":"2024-02-05T06:29:29.597119Z","created":"2024-02-05T06:27:56.979391Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6789","nonce":0}]},"addr":"172.21.15.167:6789/0","public_addr":"172.21.15.167:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-05T06:29:34.935 INFO:teuthology.orchestra.run.smithi167.stderr:dumped monmap epoch 2 2024-02-05T06:29:34.945 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:34 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T06:29:34.945 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:34 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T06:29:34.945 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:34 smithi167 ceph-mon[32774]: mon.a calling monitor election 2024-02-05T06:29:34.945 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:34 smithi167 ceph-mon[32774]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:34.945 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:34 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T06:29:34.946 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:34 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T06:29:34.946 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:34 smithi167 ceph-mon[32774]: mon.b calling monitor election 2024-02-05T06:29:34.946 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:34 smithi167 ceph-mon[32774]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:34.946 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:34 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T06:29:34.946 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:34 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T06:29:34.946 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:34 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T06:29:34.946 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:34 smithi167 ceph-mon[32774]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-05T06:29:34.946 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:34 smithi167 ceph-mon[32774]: monmap e2: 2 mons at {a=v1:172.21.15.28:6789/0,b=v1:172.21.15.167:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-02-05T06:29:34.946 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:34 smithi167 ceph-mon[32774]: fsmap 2024-02-05T06:29:34.946 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:34 smithi167 ceph-mon[32774]: osdmap e4: 0 total, 0 up, 0 in 2024-02-05T06:29:34.946 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:34 smithi167 ceph-mon[32774]: mgrmap e13: y(active, since 40s) 2024-02-05T06:29:34.946 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:34 smithi167 ceph-mon[32774]: overall HEALTH_OK 2024-02-05T06:29:34.946 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:34 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:34.947 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:34 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:34.947 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:34 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:34.947 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:34 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T06:29:34.947 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:34 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:35.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:34 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T06:29:35.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:34 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T06:29:35.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:34 smithi028 ceph-mon[29442]: mon.a calling monitor election 2024-02-05T06:29:35.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:34 smithi028 ceph-mon[29442]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:35.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:34 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T06:29:35.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:34 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T06:29:35.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:34 smithi028 ceph-mon[29442]: mon.b calling monitor election 2024-02-05T06:29:35.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:34 smithi028 ceph-mon[29442]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:35.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:34 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T06:29:35.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:34 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T06:29:35.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:34 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T06:29:35.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:34 smithi028 ceph-mon[29442]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-02-05T06:29:35.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:34 smithi028 ceph-mon[29442]: monmap e2: 2 mons at {a=v1:172.21.15.28:6789/0,b=v1:172.21.15.167:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-02-05T06:29:35.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:34 smithi028 ceph-mon[29442]: fsmap 2024-02-05T06:29:35.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:34 smithi028 ceph-mon[29442]: osdmap e4: 0 total, 0 up, 0 in 2024-02-05T06:29:35.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:34 smithi028 ceph-mon[29442]: mgrmap e13: y(active, since 40s) 2024-02-05T06:29:35.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:34 smithi028 ceph-mon[29442]: overall HEALTH_OK 2024-02-05T06:29:35.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:34 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:35.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:34 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:35.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:34 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:35.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:34 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T06:29:35.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:34 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:35.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:35 smithi028 ceph-mon[29442]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:35.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:35 smithi028 ceph-mon[29442]: Deploying daemon mon.c on smithi028 2024-02-05T06:29:35.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:35 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.167:0/1876045232' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T06:29:35.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:35 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T06:29:35.973 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:35 smithi167 ceph-mon[32774]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:35.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:35 smithi167 ceph-mon[32774]: Deploying daemon mon.c on smithi028 2024-02-05T06:29:35.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:35 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.167:0/1876045232' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T06:29:35.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:35 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T06:29:36.572 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-05T06:29:36.573 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph mon dump -f json 2024-02-05T06:29:36.810 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:29:36.939 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:29:36 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:29:36.597+0000 7f1a3ed2d700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-02-05T06:29:37.732 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:37 smithi028 ceph-mon[29442]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:37.919 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:37 smithi167 ceph-mon[32774]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:38.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:38 smithi028 systemd[1]: Started Ceph mon.c for 9801e956-c3ef-11ee-95b6-87774f69a715. 2024-02-05T06:29:43.208 INFO:teuthology.orchestra.run.smithi167.stdout: 2024-02-05T06:29:43.209 INFO:teuthology.orchestra.run.smithi167.stdout:{"epoch":3,"fsid":"9801e956-c3ef-11ee-95b6-87774f69a715","modified":"2024-02-05T06:29:38.180108Z","created":"2024-02-05T06:27:56.979391Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6789","nonce":0}]},"addr":"172.21.15.167:6789/0","public_addr":"172.21.15.167:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6790","nonce":0}]},"addr":"172.21.15.28:6790/0","public_addr":"172.21.15.28:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-02-05T06:29:43.212 INFO:teuthology.orchestra.run.smithi167.stderr:dumped monmap epoch 3 2024-02-05T06:29:43.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:43 smithi167 ceph-mon[32774]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:43.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:43 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T06:29:43.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:43 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T06:29:43.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:43 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:29:43.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:43 smithi167 ceph-mon[32774]: mon.a calling monitor election 2024-02-05T06:29:43.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:43 smithi167 ceph-mon[32774]: mon.b calling monitor election 2024-02-05T06:29:43.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:43 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.167:0/3153842032' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T06:29:43.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:43 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:29:43.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:43 smithi167 ceph-mon[32774]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:43.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:43 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:29:43.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:43 smithi167 ceph-mon[32774]: mon.c calling monitor election 2024-02-05T06:29:43.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:43 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:29:43.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:43 smithi167 ceph-mon[32774]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:43.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:43 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:29:43.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:43 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:29:43.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:43 smithi167 ceph-mon[32774]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-05T06:29:43.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:43 smithi167 ceph-mon[32774]: monmap e3: 3 mons at {a=v1:172.21.15.28:6789/0,b=v1:172.21.15.167:6789/0,c=v1:172.21.15.28:6790/0} removed_ranks: {} disallowed_leaders: {} 2024-02-05T06:29:43.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:43 smithi167 ceph-mon[32774]: fsmap 2024-02-05T06:29:43.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:43 smithi167 ceph-mon[32774]: osdmap e4: 0 total, 0 up, 0 in 2024-02-05T06:29:43.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:43 smithi167 ceph-mon[32774]: mgrmap e13: y(active, since 49s) 2024-02-05T06:29:43.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:43 smithi167 ceph-mon[32774]: overall HEALTH_OK 2024-02-05T06:29:43.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:43 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:43.476 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:43 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:43.476 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:43 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:43.476 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:43 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:43.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[29442]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:43.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T06:29:43.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T06:29:43.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:29:43.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[29442]: mon.a calling monitor election 2024-02-05T06:29:43.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[29442]: mon.b calling monitor election 2024-02-05T06:29:43.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.167:0/3153842032' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T06:29:43.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:29:43.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[29442]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:43.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:29:43.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[29442]: mon.c calling monitor election 2024-02-05T06:29:43.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:29:43.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[29442]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:43.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:29:43.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:29:43.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[29442]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-05T06:29:43.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[29442]: monmap e3: 3 mons at {a=v1:172.21.15.28:6789/0,b=v1:172.21.15.167:6789/0,c=v1:172.21.15.28:6790/0} removed_ranks: {} disallowed_leaders: {} 2024-02-05T06:29:43.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[29442]: fsmap 2024-02-05T06:29:43.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[29442]: osdmap e4: 0 total, 0 up, 0 in 2024-02-05T06:29:43.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[29442]: mgrmap e13: y(active, since 49s) 2024-02-05T06:29:43.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[29442]: overall HEALTH_OK 2024-02-05T06:29:43.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:43.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:43.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:43.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:43.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[37984]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:43.691 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T06:29:43.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T06:29:43.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:29:43.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[37984]: mon.a calling monitor election 2024-02-05T06:29:43.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[37984]: mon.b calling monitor election 2024-02-05T06:29:43.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.167:0/3153842032' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T06:29:43.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:29:43.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[37984]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:43.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:29:43.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[37984]: mon.c calling monitor election 2024-02-05T06:29:43.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:29:43.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[37984]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:43.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:29:43.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:29:43.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[37984]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-02-05T06:29:43.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[37984]: monmap e3: 3 mons at {a=v1:172.21.15.28:6789/0,b=v1:172.21.15.167:6789/0,c=v1:172.21.15.28:6790/0} removed_ranks: {} disallowed_leaders: {} 2024-02-05T06:29:43.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[37984]: fsmap 2024-02-05T06:29:43.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[37984]: osdmap e4: 0 total, 0 up, 0 in 2024-02-05T06:29:43.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[37984]: mgrmap e13: y(active, since 49s) 2024-02-05T06:29:43.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[37984]: overall HEALTH_OK 2024-02-05T06:29:43.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:43.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:43.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:43.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:43 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:43.833 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-05T06:29:43.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph config generate-minimal-conf 2024-02-05T06:29:44.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:44 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:29:44.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:44 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:44.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:44 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:29:44.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:44 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:29:44.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:44 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:44.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:44 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:29:44.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:44 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:29:44.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:44 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:44.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:44 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:29:44.758 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:29:45.939 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[29442]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:45.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:45.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:29:45.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[29442]: Updating smithi028:/etc/ceph/ceph.conf 2024-02-05T06:29:45.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[29442]: Updating smithi167:/etc/ceph/ceph.conf 2024-02-05T06:29:45.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[29442]: Updating smithi028:/var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T06:29:45.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[29442]: Updating smithi167:/var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T06:29:45.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T06:29:45.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-05T06:29:45.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:45.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[37984]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:45.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:45.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:29:45.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[37984]: Updating smithi028:/etc/ceph/ceph.conf 2024-02-05T06:29:45.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[37984]: Updating smithi167:/etc/ceph/ceph.conf 2024-02-05T06:29:45.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[37984]: Updating smithi028:/var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T06:29:45.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[37984]: Updating smithi167:/var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T06:29:45.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T06:29:45.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-05T06:29:45.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:45 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:45.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:45 smithi167 ceph-mon[32774]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:45.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:45 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:45 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.975 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:45 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.975 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:45 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.975 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:45 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:45.975 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:45 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:29:45.975 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:45 smithi167 ceph-mon[32774]: Updating smithi028:/etc/ceph/ceph.conf 2024-02-05T06:29:45.975 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:45 smithi167 ceph-mon[32774]: Updating smithi167:/etc/ceph/ceph.conf 2024-02-05T06:29:45.975 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:45 smithi167 ceph-mon[32774]: Updating smithi028:/var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T06:29:45.975 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:45 smithi167 ceph-mon[32774]: Updating smithi167:/var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T06:29:45.975 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:45 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.975 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:45 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.976 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:45 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.976 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:45 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.976 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:45 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.976 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:45 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.976 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:45 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.976 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:45 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.976 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:45 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:45.976 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:45 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T06:29:45.976 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:45 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-05T06:29:45.976 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:45 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:46.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:46 smithi028 ceph-mon[37984]: Reconfiguring mon.a (unknown last config time)... 2024-02-05T06:29:46.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:46 smithi028 ceph-mon[37984]: Reconfiguring daemon mon.a on smithi028 2024-02-05T06:29:46.905 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:46 smithi028 ceph-mon[29442]: Reconfiguring mon.a (unknown last config time)... 2024-02-05T06:29:46.905 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:46 smithi028 ceph-mon[29442]: Reconfiguring daemon mon.a on smithi028 2024-02-05T06:29:46.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:46 smithi167 ceph-mon[32774]: Reconfiguring mon.a (unknown last config time)... 2024-02-05T06:29:46.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:46 smithi167 ceph-mon[32774]: Reconfiguring daemon mon.a on smithi028 2024-02-05T06:29:47.149 INFO:teuthology.orchestra.run.smithi028.stdout:# minimal ceph.conf for 9801e956-c3ef-11ee-95b6-87774f69a715 2024-02-05T06:29:47.149 INFO:teuthology.orchestra.run.smithi028.stdout:[global] 2024-02-05T06:29:47.149 INFO:teuthology.orchestra.run.smithi028.stdout: fsid = 9801e956-c3ef-11ee-95b6-87774f69a715 2024-02-05T06:29:47.149 INFO:teuthology.orchestra.run.smithi028.stdout: mon_host = 172.21.15.28:6789/0 172.21.15.167:6789/0 v1:172.21.15.28:6790/0 2024-02-05T06:29:47.765 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-05T06:29:47.765 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-05T06:29:47.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.conf 2024-02-05T06:29:47.795 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-05T06:29:47.795 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-05T06:29:47.862 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-02-05T06:29:47.862 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/ceph/ceph.conf 2024-02-05T06:29:47.896 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-02-05T06:29:47.896 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-05T06:29:47.966 INFO:tasks.cephadm:Adding mgr.y on smithi028 2024-02-05T06:29:47.966 INFO:tasks.cephadm:Adding mgr.x on smithi167 2024-02-05T06:29:47.967 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph orch apply mgr '2;smithi028=y;smithi167=x' 2024-02-05T06:29:48.043 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:47 smithi028 ceph-mon[29442]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:48.043 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:47 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3374903960' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:48.043 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:47 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:48.043 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:47 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:48.043 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:47 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T06:29:48.043 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:47 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-05T06:29:48.043 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:47 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:48.044 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:47 smithi028 ceph-mon[37984]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:48.044 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:47 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3374903960' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:48.044 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:47 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:48.044 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:47 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:48.044 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:47 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T06:29:48.044 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:47 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-05T06:29:48.044 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:47 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:48.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:47 smithi167 ceph-mon[32774]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:48.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:47 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3374903960' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:48.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:47 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:48.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:47 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:48.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:47 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T06:29:48.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:47 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-05T06:29:48.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:47 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:48.242 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:29:49.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:48 smithi028 ceph-mon[37984]: Reconfiguring mon.c (monmap changed)... 2024-02-05T06:29:49.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:48 smithi028 ceph-mon[37984]: Reconfiguring daemon mon.c on smithi028 2024-02-05T06:29:49.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:48 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:49.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:48 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:49.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:48 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T06:29:49.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:48 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-05T06:29:49.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:48 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:49.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:48 smithi028 ceph-mon[29442]: Reconfiguring mon.c (monmap changed)... 2024-02-05T06:29:49.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:48 smithi028 ceph-mon[29442]: Reconfiguring daemon mon.c on smithi028 2024-02-05T06:29:49.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:48 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:49.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:48 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:49.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:48 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T06:29:49.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:48 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-05T06:29:49.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:48 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:49.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:48 smithi167 ceph-mon[32774]: Reconfiguring mon.c (monmap changed)... 2024-02-05T06:29:49.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:48 smithi167 ceph-mon[32774]: Reconfiguring daemon mon.c on smithi028 2024-02-05T06:29:49.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:48 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:49.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:48 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:49.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:48 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-05T06:29:49.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:48 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-05T06:29:49.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:48 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:50.145 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:49 smithi028 ceph-mon[29442]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:50.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:49 smithi028 ceph-mon[29442]: Reconfiguring mon.b (monmap changed)... 2024-02-05T06:29:50.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:49 smithi028 ceph-mon[29442]: Reconfiguring daemon mon.b on smithi167 2024-02-05T06:29:50.146 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:49 smithi028 ceph-mon[37984]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:50.146 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:49 smithi028 ceph-mon[37984]: Reconfiguring mon.b (monmap changed)... 2024-02-05T06:29:50.146 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:49 smithi028 ceph-mon[37984]: Reconfiguring daemon mon.b on smithi167 2024-02-05T06:29:50.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:49 smithi167 ceph-mon[32774]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:50.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:49 smithi167 ceph-mon[32774]: Reconfiguring mon.b (monmap changed)... 2024-02-05T06:29:50.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:49 smithi167 ceph-mon[32774]: Reconfiguring daemon mon.b on smithi167 2024-02-05T06:29:51.028 INFO:teuthology.orchestra.run.smithi167.stdout:Scheduled mgr update... 2024-02-05T06:29:51.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[37984]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:51.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:51.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:51.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:29:51.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:51.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:29:51.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:51.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[37984]: from='client.24109 v1:172.21.15.167:0/2433633999' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi028=y;smithi167=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:29:51.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[37984]: Saving service mgr spec with placement smithi028=y;smithi167=x;count:2 2024-02-05T06:29:51.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:51.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:29:51.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:51.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:29:51.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:51.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-05T06:29:51.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-05T06:29:51.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-05T06:29:51.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:51.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[37984]: Deploying daemon mgr.x on smithi167 2024-02-05T06:29:51.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[29442]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:51.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:51.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:51.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:29:51.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:51.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:29:51.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:51.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[29442]: from='client.24109 v1:172.21.15.167:0/2433633999' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi028=y;smithi167=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:29:51.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[29442]: Saving service mgr spec with placement smithi028=y;smithi167=x;count:2 2024-02-05T06:29:51.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:51.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:29:51.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:51.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:29:51.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:51.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-05T06:29:51.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-05T06:29:51.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-05T06:29:51.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:51.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:51 smithi028 ceph-mon[29442]: Deploying daemon mgr.x on smithi167 2024-02-05T06:29:51.656 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:51 smithi167 ceph-mon[32774]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:51.657 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:51 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:51.657 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:51 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:51.657 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:51 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:29:51.657 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:51 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:51.657 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:51 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:29:51.657 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:51 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:51.657 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:51 smithi167 ceph-mon[32774]: from='client.24109 v1:172.21.15.167:0/2433633999' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi028=y;smithi167=x", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:29:51.657 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:51 smithi167 ceph-mon[32774]: Saving service mgr spec with placement smithi028=y;smithi167=x;count:2 2024-02-05T06:29:51.657 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:51 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:51.657 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:51 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:29:51.657 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:51 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:51.657 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:51 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:29:51.657 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:51 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:51.657 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:51 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-05T06:29:51.658 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:51 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-05T06:29:51.658 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:51 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-05T06:29:51.658 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:51 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:51.658 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:51 smithi167 ceph-mon[32774]: Deploying daemon mgr.x on smithi167 2024-02-05T06:29:51.681 DEBUG:teuthology.orchestra.run.smithi167:mgr.x> sudo journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@mgr.x.service 2024-02-05T06:29:51.683 INFO:tasks.cephadm:Deploying OSDs... 2024-02-05T06:29:51.684 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-05T06:29:51.684 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/scratch_devs of=/dev/stdout 2024-02-05T06:29:51.703 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-05T06:29:51.703 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_1 2024-02-05T06:29:51.761 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-05T06:29:51.761 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T06:29:51.761 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-02-05T06:29:51.761 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T06:29:51.761 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T06:29:51.761 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-02-05 06:29:37.614037377 +0000 2024-02-05T06:29:51.761 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-02-05 06:24:22.647494134 +0000 2024-02-05T06:29:51.761 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-02-05 06:24:22.647494134 +0000 2024-02-05T06:29:51.761 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-02-05 06:24:22.647494134 +0000 2024-02-05T06:29:51.762 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-05T06:29:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-02-05T06:29:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-02-05T06:29:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000116757 s, 4.4 MB/s 2024-02-05T06:29:51.833 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-05T06:29:51.891 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_2 2024-02-05T06:29:51.948 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-05T06:29:51.948 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T06:29:51.948 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-02-05T06:29:51.948 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T06:29:51.948 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T06:29:51.948 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-02-05 06:29:37.614037377 +0000 2024-02-05T06:29:51.948 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-02-05 06:24:22.987484483 +0000 2024-02-05T06:29:51.948 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-02-05 06:24:22.987484483 +0000 2024-02-05T06:29:51.949 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-02-05 06:24:22.987484483 +0000 2024-02-05T06:29:51.949 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-05T06:29:52.017 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-02-05T06:29:52.017 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-02-05T06:29:52.017 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000104347 s, 4.9 MB/s 2024-02-05T06:29:52.018 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-05T06:29:52.075 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_3 2024-02-05T06:29:52.132 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-05T06:29:52.132 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T06:29:52.132 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-02-05T06:29:52.132 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T06:29:52.132 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T06:29:52.132 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-02-05 06:29:37.614037377 +0000 2024-02-05T06:29:52.132 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-02-05 06:24:23.323474946 +0000 2024-02-05T06:29:52.132 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-02-05 06:24:23.323474946 +0000 2024-02-05T06:29:52.132 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-02-05 06:24:23.323474946 +0000 2024-02-05T06:29:52.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-05T06:29:52.199 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-02-05T06:29:52.199 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-02-05T06:29:52.199 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000110686 s, 4.6 MB/s 2024-02-05T06:29:52.200 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-05T06:29:52.257 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_4 2024-02-05T06:29:52.313 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-05T06:29:52.313 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T06:29:52.313 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-02-05T06:29:52.313 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T06:29:52.313 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T06:29:52.313 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-02-05 06:29:37.615037348 +0000 2024-02-05T06:29:52.313 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-02-05 06:24:23.664465266 +0000 2024-02-05T06:29:52.313 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-02-05 06:24:23.664465266 +0000 2024-02-05T06:29:52.313 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-02-05 06:24:23.664465266 +0000 2024-02-05T06:29:52.314 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-05T06:29:52.381 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-02-05T06:29:52.382 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-02-05T06:29:52.382 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000110492 s, 4.6 MB/s 2024-02-05T06:29:52.383 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-05T06:29:52.443 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-02-05T06:29:52.444 DEBUG:teuthology.orchestra.run.smithi167:> dd if=/scratch_devs of=/dev/stdout 2024-02-05T06:29:52.461 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-05T06:29:52.461 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_1 2024-02-05T06:29:52.516 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-05T06:29:52.516 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T06:29:52.516 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-02-05T06:29:52.517 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T06:29:52.517 INFO:teuthology.orchestra.run.smithi167.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T06:29:52.517 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2024-02-05 06:29:28.834280424 +0000 2024-02-05T06:29:52.517 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2024-02-05 06:24:22.202402573 +0000 2024-02-05T06:29:52.517 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2024-02-05 06:24:22.202402573 +0000 2024-02-05T06:29:52.517 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: 2024-02-05 06:24:22.202402573 +0000 2024-02-05T06:29:52.517 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-05T06:29:52.582 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2024-02-05T06:29:52.582 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2024-02-05T06:29:52.582 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000192414 s, 2.7 MB/s 2024-02-05T06:29:52.584 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-05T06:29:52.639 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_2 2024-02-05T06:29:52.693 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-05T06:29:52.694 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T06:29:52.694 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-02-05T06:29:52.694 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T06:29:52.694 INFO:teuthology.orchestra.run.smithi167.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T06:29:52.694 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2024-02-05 06:29:28.834280424 +0000 2024-02-05T06:29:52.694 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2024-02-05 06:24:22.546396562 +0000 2024-02-05T06:29:52.694 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2024-02-05 06:24:22.546396562 +0000 2024-02-05T06:29:52.694 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: 2024-02-05 06:24:22.546396562 +0000 2024-02-05T06:29:52.694 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-05T06:29:52.760 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2024-02-05T06:29:52.760 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2024-02-05T06:29:52.760 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000182389 s, 2.8 MB/s 2024-02-05T06:29:52.761 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-05T06:29:52.816 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_3 2024-02-05T06:29:52.877 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-05T06:29:52.877 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T06:29:52.878 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-02-05T06:29:52.878 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T06:29:52.878 INFO:teuthology.orchestra.run.smithi167.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T06:29:52.878 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2024-02-05 06:29:28.835280406 +0000 2024-02-05T06:29:52.878 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2024-02-05 06:24:22.885390639 +0000 2024-02-05T06:29:52.878 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2024-02-05 06:24:22.885390639 +0000 2024-02-05T06:29:52.878 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: 2024-02-05 06:24:22.885390639 +0000 2024-02-05T06:29:52.878 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-05T06:29:52.980 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2024-02-05T06:29:52.980 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2024-02-05T06:29:52.981 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.000212015 s, 2.4 MB/s 2024-02-05T06:29:52.982 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-05T06:29:52.996 DEBUG:teuthology.orchestra.run.smithi167:> stat /dev/vg_nvme/lv_4 2024-02-05T06:29:53.054 INFO:teuthology.orchestra.run.smithi167.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-05T06:29:53.055 INFO:teuthology.orchestra.run.smithi167.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-05T06:29:53.055 INFO:teuthology.orchestra.run.smithi167.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-02-05T06:29:53.055 INFO:teuthology.orchestra.run.smithi167.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-05T06:29:53.055 INFO:teuthology.orchestra.run.smithi167.stdout:Context: system_u:object_r:device_t:s0 2024-02-05T06:29:53.055 INFO:teuthology.orchestra.run.smithi167.stdout:Access: 2024-02-05 06:29:52.937870021 +0000 2024-02-05T06:29:53.055 INFO:teuthology.orchestra.run.smithi167.stdout:Modify: 2024-02-05 06:24:23.228384646 +0000 2024-02-05T06:29:53.055 INFO:teuthology.orchestra.run.smithi167.stdout:Change: 2024-02-05 06:24:23.228384646 +0000 2024-02-05T06:29:53.055 INFO:teuthology.orchestra.run.smithi167.stdout: Birth: 2024-02-05 06:24:23.228384646 +0000 2024-02-05T06:29:53.055 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-05T06:29:53.153 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records in 2024-02-05T06:29:53.153 INFO:teuthology.orchestra.run.smithi167.stderr:1+0 records out 2024-02-05T06:29:53.153 INFO:teuthology.orchestra.run.smithi167.stderr:512 bytes copied, 0.00019061 s, 2.7 MB/s 2024-02-05T06:29:53.154 DEBUG:teuthology.orchestra.run.smithi167:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-05T06:29:53.169 INFO:tasks.cephadm:Deploying osd.0 on smithi028 with /dev/vg_nvme/lv_4... 2024-02-05T06:29:53.169 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-05T06:29:53.324 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:29:53 smithi167 systemd[1]: Starting Ceph mgr.x for 9801e956-c3ef-11ee-95b6-87774f69a715... 2024-02-05T06:29:53.418 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:29:53.540 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:53 smithi028 ceph-mon[37984]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:53.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:53 smithi028 ceph-mon[29442]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:53.609 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:53 smithi167 ceph-mon[32774]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:53.609 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:29:53 smithi167 podman[34645]: 2024-02-05 06:29:53.291457522 +0000 UTC m=+0.018633481 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T06:29:53.610 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:29:53 smithi167 podman[34645]: 2024-02-05 06:29:53.41038584 +0000 UTC m=+0.137561834 container create 06698b169813f574c848694bbc169f8c27760194bb96bc4942224fc5feb3147c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-02-05T06:29:53.610 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:29:53 smithi167 podman[34645]: 2024-02-05 06:29:53.612437359 +0000 UTC m=+0.339613309 container init 06698b169813f574c848694bbc169f8c27760194bb96bc4942224fc5feb3147c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-02-05T06:29:53.974 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:29:53 smithi167 podman[34645]: 2024-02-05 06:29:53.61504888 +0000 UTC m=+0.342224835 container start 06698b169813f574c848694bbc169f8c27760194bb96bc4942224fc5feb3147c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-05T06:29:53.974 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:29:53 smithi167 bash[34645]: 06698b169813f574c848694bbc169f8c27760194bb96bc4942224fc5feb3147c 2024-02-05T06:29:53.974 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:29:53 smithi167 systemd[1]: Started Ceph mgr.x for 9801e956-c3ef-11ee-95b6-87774f69a715. 2024-02-05T06:29:54.474 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:29:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:29:54.186+0000 7facd5c84200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-05T06:29:54.474 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:29:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:29:54.282+0000 7facd5c84200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-05T06:29:54.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:54 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:54.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:54 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:54.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:54 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:54.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:54 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:54.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:54 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:29:54.975 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:54 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:55.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:54 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:55.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:54 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:55.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:54 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:55.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:54 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:55.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:54 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:29:55.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:54 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:55.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:54 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:55.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:54 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:55.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:54 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:55.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:54 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:55.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:54 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:29:55.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:54 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:55.227 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:29:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:29:54.977+0000 7facd5c84200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-05T06:29:55.227 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:29:55 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:29:55.073+0000 7facd5c84200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-05T06:29:55.389 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-05T06:29:55.389 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-05T06:29:55.389 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-02-05T06:29:55.389 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-02-05T06:29:55.389 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00864741 s, 1.2 GB/s 2024-02-05T06:29:55.389 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-02-05T06:29:55.693 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:29:55 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:29:55.229+0000 7facd5c84200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-05T06:29:55.693 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:29:55 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:29:55.331+0000 7facd5c84200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-05T06:29:55.693 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:29:55 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:29:55.419+0000 7facd5c84200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-05T06:29:55.693 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:29:55 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:29:55.694+0000 7facd5c84200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-05T06:29:55.907 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:55 smithi028 ceph-mon[37984]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:55.908 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:55 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:55.908 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:55 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:55.908 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:55 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:55.908 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:55 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:29:55.908 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:55 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:55.908 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:55 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-05T06:29:55.908 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:55 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-05T06:29:55.908 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:55 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:55.908 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:55 smithi028 ceph-mon[29442]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:55.909 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:55 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:55.909 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:55 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:55.909 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:55 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:55.909 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:55 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:29:55.909 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:55 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:55.909 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:55 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-05T06:29:55.909 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:55 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-05T06:29:55.909 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:55 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:55.935 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph orch daemon add osd smithi028:vg_nvme/lv_4 2024-02-05T06:29:55.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:55 smithi167 ceph-mon[32774]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:55.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:55 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:55.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:55 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:55.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:55 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:55.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:55 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:29:55.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:55 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:55.974 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:55 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-05T06:29:55.975 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:55 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-05T06:29:55.975 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:55 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:55.975 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:29:55 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:29:55.809+0000 7facd5c84200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-05T06:29:55.975 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:29:55 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:29:55.898+0000 7facd5c84200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-05T06:29:56.263 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:29:55 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:29:55.995+0000 7facd5c84200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-05T06:29:56.263 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:29:56 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:29:56.178+0000 7facd5c84200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-05T06:29:56.627 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:29:57.079 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:56 smithi028 ceph-mon[29442]: Reconfiguring mgr.y (unknown last config time)... 2024-02-05T06:29:57.079 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:56 smithi028 ceph-mon[29442]: Reconfiguring daemon mgr.y on smithi028 2024-02-05T06:29:57.079 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:56 smithi028 ceph-mon[37984]: Reconfiguring mgr.y (unknown last config time)... 2024-02-05T06:29:57.079 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:56 smithi028 ceph-mon[37984]: Reconfiguring daemon mgr.y on smithi028 2024-02-05T06:29:57.163 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:56 smithi167 ceph-mon[32774]: Reconfiguring mgr.y (unknown last config time)... 2024-02-05T06:29:57.163 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:56 smithi167 ceph-mon[32774]: Reconfiguring daemon mgr.y on smithi028 2024-02-05T06:29:57.474 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:29:57 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:29:57.165+0000 7facd5c84200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-05T06:29:57.474 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:29:57 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:29:57.297+0000 7facd5c84200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-05T06:29:57.919 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:29:57 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:29:57.632+0000 7facd5c84200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-05T06:29:57.920 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:29:57 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:29:57.722+0000 7facd5c84200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-05T06:29:57.921 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:29:57 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:29:57.919+0000 7facd5c84200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-05T06:29:58.224 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:29:58 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:29:58.104+0000 7facd5c84200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-05T06:29:58.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:58 smithi167 ceph-mon[32774]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:58.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:58 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:58.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:58 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:58.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:58 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:29:58.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:58 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:58.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:58 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:29:58.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:58 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:58.250 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:58 smithi028 ceph-mon[29442]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:58.250 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:58 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:58.250 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:58 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:58.250 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:58 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:29:58.250 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:58 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:58.250 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:58 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:29:58.250 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:58 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:58.251 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:58 smithi028 ceph-mon[37984]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:29:58.251 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:58 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:58.251 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:58 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:58.251 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:58 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:29:58.251 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:58 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:58.251 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:58 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:29:58.251 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:58 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:29:58.646 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:29:58 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:29:58.386+0000 7facd5c84200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-05T06:29:58.646 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:29:58 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:29:58.648+0000 7facd5c84200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-05T06:29:58.974 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:29:58 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:29:58.881+0000 7facd5c84200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-05T06:29:59.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:59 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T06:29:59.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:59 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T06:29:59.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:29:59 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:59.292 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:59 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T06:29:59.292 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:59 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T06:29:59.292 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:29:59 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:59.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:59 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T06:29:59.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:59 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T06:29:59.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:29:59 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:29:59.974 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:29:59 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:29:59.679+0000 7facd5c84200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-05T06:30:00.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:00 smithi028 ceph-mon[37984]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:00.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:00 smithi028 ceph-mon[37984]: from='client.44114 v1:172.21.15.28:0/158246757' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:30:00.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:00 smithi028 ceph-mon[37984]: overall HEALTH_OK 2024-02-05T06:30:00.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:00 smithi028 ceph-mon[29442]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:00.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:00 smithi028 ceph-mon[29442]: from='client.44114 v1:172.21.15.28:0/158246757' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:30:00.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:00 smithi028 ceph-mon[29442]: overall HEALTH_OK 2024-02-05T06:30:00.347 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:00 smithi167 ceph-mon[32774]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:00.347 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:00 smithi167 ceph-mon[32774]: from='client.44114 v1:172.21.15.28:0/158246757' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:30:00.347 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:00 smithi167 ceph-mon[32774]: overall HEALTH_OK 2024-02-05T06:30:00.347 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:30:00 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:30:00.252+0000 7facd5c84200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-05T06:30:00.347 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:30:00 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:30:00.349+0000 7facd5c84200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-05T06:30:00.724 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:30:00 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:30:00.564+0000 7facd5c84200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-05T06:30:01.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:01 smithi028 ceph-mon[29442]: Standby manager daemon x started 2024-02-05T06:30:01.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:01 smithi028 ceph-mon[29442]: from='mgr.? v1:172.21.15.167:0/2403698933' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-05T06:30:01.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:01 smithi028 ceph-mon[29442]: from='mgr.? v1:172.21.15.167:0/2403698933' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-05T06:30:01.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:01 smithi028 ceph-mon[29442]: from='mgr.? v1:172.21.15.167:0/2403698933' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-05T06:30:01.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:01 smithi028 ceph-mon[29442]: from='mgr.? v1:172.21.15.167:0/2403698933' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-05T06:30:01.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:01 smithi028 ceph-mon[37984]: Standby manager daemon x started 2024-02-05T06:30:01.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:01 smithi028 ceph-mon[37984]: from='mgr.? v1:172.21.15.167:0/2403698933' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-05T06:30:01.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:01 smithi028 ceph-mon[37984]: from='mgr.? v1:172.21.15.167:0/2403698933' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-05T06:30:01.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:01 smithi028 ceph-mon[37984]: from='mgr.? v1:172.21.15.167:0/2403698933' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-05T06:30:01.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:01 smithi028 ceph-mon[37984]: from='mgr.? v1:172.21.15.167:0/2403698933' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-05T06:30:01.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:01 smithi167 ceph-mon[32774]: Standby manager daemon x started 2024-02-05T06:30:01.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:01 smithi167 ceph-mon[32774]: from='mgr.? v1:172.21.15.167:0/2403698933' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-05T06:30:01.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:01 smithi167 ceph-mon[32774]: from='mgr.? v1:172.21.15.167:0/2403698933' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-05T06:30:01.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:01 smithi167 ceph-mon[32774]: from='mgr.? v1:172.21.15.167:0/2403698933' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-05T06:30:01.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:01 smithi167 ceph-mon[32774]: from='mgr.? v1:172.21.15.167:0/2403698933' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-05T06:30:02.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:02 smithi028 ceph-mon[29442]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:02.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:02 smithi028 ceph-mon[29442]: mgrmap e14: y(active, since 66s), standbys: x 2024-02-05T06:30:02.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:02 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-05T06:30:02.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:02 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/567085864' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c19fdd5b-09f6-45d4-a519-0a938cf1f2a6"}]: dispatch 2024-02-05T06:30:02.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:02 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/567085864' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c19fdd5b-09f6-45d4-a519-0a938cf1f2a6"}]': finished 2024-02-05T06:30:02.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:02 smithi028 ceph-mon[29442]: osdmap e5: 1 total, 0 up, 1 in 2024-02-05T06:30:02.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:02 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T06:30:02.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:02 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1847271240' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T06:30:02.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:02 smithi028 ceph-mon[37984]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:02.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:02 smithi028 ceph-mon[37984]: mgrmap e14: y(active, since 66s), standbys: x 2024-02-05T06:30:02.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:02 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-05T06:30:02.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:02 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/567085864' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c19fdd5b-09f6-45d4-a519-0a938cf1f2a6"}]: dispatch 2024-02-05T06:30:02.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:02 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/567085864' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c19fdd5b-09f6-45d4-a519-0a938cf1f2a6"}]': finished 2024-02-05T06:30:02.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:02 smithi028 ceph-mon[37984]: osdmap e5: 1 total, 0 up, 1 in 2024-02-05T06:30:02.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:02 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T06:30:02.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:02 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1847271240' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T06:30:02.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:02 smithi167 ceph-mon[32774]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:02.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:02 smithi167 ceph-mon[32774]: mgrmap e14: y(active, since 66s), standbys: x 2024-02-05T06:30:02.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:02 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-05T06:30:02.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:02 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/567085864' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c19fdd5b-09f6-45d4-a519-0a938cf1f2a6"}]: dispatch 2024-02-05T06:30:02.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:02 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/567085864' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c19fdd5b-09f6-45d4-a519-0a938cf1f2a6"}]': finished 2024-02-05T06:30:02.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:02 smithi167 ceph-mon[32774]: osdmap e5: 1 total, 0 up, 1 in 2024-02-05T06:30:02.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:02 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T06:30:02.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:02 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1847271240' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T06:30:04.276 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:04 smithi167 ceph-mon[32774]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:04.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:04 smithi028 ceph-mon[29442]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:04.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:04 smithi028 ceph-mon[37984]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:06.414 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:06 smithi028 ceph-mon[29442]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:06.415 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:06 smithi028 ceph-mon[37984]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:06.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:06 smithi167 ceph-mon[32774]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:08.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:08 smithi028 ceph-mon[29442]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:08.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:08 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-05T06:30:08.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:08 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:30:08.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:08 smithi028 ceph-mon[37984]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:08.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:08 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-05T06:30:08.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:08 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:30:08.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:08 smithi167 ceph-mon[32774]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:08.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:08 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-05T06:30:08.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:08 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:30:09.110 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:09 smithi028 ceph-mon[29442]: Deploying daemon osd.0 on smithi028 2024-02-05T06:30:09.110 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:09 smithi028 ceph-mon[37984]: Deploying daemon osd.0 on smithi028 2024-02-05T06:30:09.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:09 smithi167 ceph-mon[32774]: Deploying daemon osd.0 on smithi028 2024-02-05T06:30:10.333 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:10 smithi028 ceph-mon[29442]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:10.334 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:10 smithi028 ceph-mon[37984]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:10.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:10 smithi167 ceph-mon[32774]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:12.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:12 smithi028 ceph-mon[29442]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:12.293 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:12 smithi028 ceph-mon[37984]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:12.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:12 smithi167 ceph-mon[32774]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:14.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:13 smithi028 ceph-mon[29442]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:14.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:13 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:14.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:13 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:14.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:13 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:30:14.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:13 smithi028 ceph-mon[37984]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:14.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:13 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:14.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:13 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:14.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:13 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:30:14.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:13 smithi167 ceph-mon[32774]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:14.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:13 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:14.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:13 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:14.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:13 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:30:15.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:15 smithi028 ceph-mon[37984]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:15.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:15 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:15.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:15 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:15.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:15 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:30:15.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:15 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:30:15.937 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:15 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:15.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:15 smithi028 ceph-mon[29442]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:15.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:15 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:15.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:15 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:15.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:15 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:30:15.937 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:15 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:30:15.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:15 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:16.193 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 0 on host 'smithi028' 2024-02-05T06:30:16.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:15 smithi167 ceph-mon[32774]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:16.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:15 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:16.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:15 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:16.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:15 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:30:16.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:15 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:30:16.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:15 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:16.813 DEBUG:teuthology.orchestra.run.smithi028:osd.0> sudo journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@osd.0.service 2024-02-05T06:30:16.815 INFO:tasks.cephadm:Deploying osd.1 on smithi028 with /dev/vg_nvme/lv_3... 2024-02-05T06:30:16.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-05T06:30:17.083 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:16 smithi028 ceph-mon[29442]: from='osd.0 v1:172.21.15.28:6801/301279106' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-05T06:30:17.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:16 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:17.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:16 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:17.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:16 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:30:17.084 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:16 smithi028 ceph-mon[37984]: from='osd.0 v1:172.21.15.28:6801/301279106' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-05T06:30:17.084 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:16 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:17.084 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:16 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:17.084 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:16 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:30:17.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:16 smithi167 ceph-mon[32774]: from='osd.0 v1:172.21.15.28:6801/301279106' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-05T06:30:17.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:16 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:17.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:16 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:17.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:16 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:30:17.495 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:30:18.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:17 smithi028 ceph-mon[37984]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:18.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:17 smithi028 ceph-mon[37984]: from='osd.0 v1:172.21.15.28:6801/301279106' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-05T06:30:18.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:17 smithi028 ceph-mon[37984]: osdmap e6: 1 total, 0 up, 1 in 2024-02-05T06:30:18.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:17 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T06:30:18.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:17 smithi028 ceph-mon[37984]: from='osd.0 v1:172.21.15.28:6801/301279106' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-05T06:30:18.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:17 smithi028 ceph-mon[29442]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:18.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:17 smithi028 ceph-mon[29442]: from='osd.0 v1:172.21.15.28:6801/301279106' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-05T06:30:18.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:17 smithi028 ceph-mon[29442]: osdmap e6: 1 total, 0 up, 1 in 2024-02-05T06:30:18.135 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:17 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T06:30:18.135 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:17 smithi028 ceph-mon[29442]: from='osd.0 v1:172.21.15.28:6801/301279106' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-05T06:30:18.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:17 smithi167 ceph-mon[32774]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:18.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:17 smithi167 ceph-mon[32774]: from='osd.0 v1:172.21.15.28:6801/301279106' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-05T06:30:18.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:17 smithi167 ceph-mon[32774]: osdmap e6: 1 total, 0 up, 1 in 2024-02-05T06:30:18.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:17 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T06:30:18.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:17 smithi167 ceph-mon[32774]: from='osd.0 v1:172.21.15.28:6801/301279106' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-05T06:30:18.440 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 05 06:30:18 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-0[42047]: 2024-02-05T06:30:18.135+0000 7f6d29c56700 -1 osd.0 0 waiting for initial osdmap 2024-02-05T06:30:18.440 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 05 06:30:18 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-0[42047]: 2024-02-05T06:30:18.137+0000 7f6d21247700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T06:30:19.140 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:18 smithi028 ceph-mon[29442]: from='osd.0 v1:172.21.15.28:6801/301279106' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-02-05T06:30:19.140 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:18 smithi028 ceph-mon[29442]: osdmap e7: 1 total, 0 up, 1 in 2024-02-05T06:30:19.140 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:18 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T06:30:19.140 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:18 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T06:30:19.140 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:18 smithi028 ceph-mon[29442]: from='osd.0 v1:172.21.15.28:6801/301279106' entity='osd.0' 2024-02-05T06:30:19.140 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:18 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T06:30:19.140 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:18 smithi028 ceph-mon[37984]: from='osd.0 v1:172.21.15.28:6801/301279106' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-02-05T06:30:19.141 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:18 smithi028 ceph-mon[37984]: osdmap e7: 1 total, 0 up, 1 in 2024-02-05T06:30:19.141 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:18 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T06:30:19.141 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:18 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T06:30:19.141 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:18 smithi028 ceph-mon[37984]: from='osd.0 v1:172.21.15.28:6801/301279106' entity='osd.0' 2024-02-05T06:30:19.141 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:18 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T06:30:19.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:18 smithi167 ceph-mon[32774]: from='osd.0 v1:172.21.15.28:6801/301279106' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-02-05T06:30:19.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:18 smithi167 ceph-mon[32774]: osdmap e7: 1 total, 0 up, 1 in 2024-02-05T06:30:19.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:18 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T06:30:19.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:18 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T06:30:19.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:18 smithi167 ceph-mon[32774]: from='osd.0 v1:172.21.15.28:6801/301279106' entity='osd.0' 2024-02-05T06:30:19.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:18 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T06:30:19.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:19 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:30:19.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:19 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:30:19.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:19 smithi028 ceph-mon[29442]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:19.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:19 smithi028 ceph-mon[29442]: osd.0 v1:172.21.15.28:6801/301279106 boot 2024-02-05T06:30:19.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:19 smithi028 ceph-mon[29442]: osdmap e8: 1 total, 1 up, 1 in 2024-02-05T06:30:19.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:19 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T06:30:19.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:19 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:30:19.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:19 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:30:19.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:19 smithi028 ceph-mon[37984]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:19.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:19 smithi028 ceph-mon[37984]: osd.0 v1:172.21.15.28:6801/301279106 boot 2024-02-05T06:30:19.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:19 smithi028 ceph-mon[37984]: osdmap e8: 1 total, 1 up, 1 in 2024-02-05T06:30:19.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:19 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T06:30:20.081 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-05T06:30:20.082 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-05T06:30:20.082 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-02-05T06:30:20.082 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-02-05T06:30:20.082 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0102885 s, 1.0 GB/s 2024-02-05T06:30:20.082 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-02-05T06:30:20.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:19 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:30:20.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:19 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:30:20.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:19 smithi167 ceph-mon[32774]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-05T06:30:20.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:19 smithi167 ceph-mon[32774]: osd.0 v1:172.21.15.28:6801/301279106 boot 2024-02-05T06:30:20.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:19 smithi167 ceph-mon[32774]: osdmap e8: 1 total, 1 up, 1 in 2024-02-05T06:30:20.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:19 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T06:30:21.195 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph orch daemon add osd smithi028:vg_nvme/lv_3 2024-02-05T06:30:21.436 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:30:22.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:21 smithi028 ceph-mon[29442]: pgmap v41: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:22.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:21 smithi028 ceph-mon[29442]: Detected new or changed devices on smithi028 2024-02-05T06:30:22.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:21 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:22.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:21 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:22.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:21 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:30:22.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:21 smithi028 ceph-mon[29442]: Adjusting osd_memory_target on smithi028 to 16069M 2024-02-05T06:30:22.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:21 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:22.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:21 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:30:22.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:21 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:30:22.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:21 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:22.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:21 smithi028 ceph-mon[29442]: osdmap e9: 1 total, 1 up, 1 in 2024-02-05T06:30:22.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:21 smithi028 ceph-mon[37984]: pgmap v41: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:22.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:21 smithi028 ceph-mon[37984]: Detected new or changed devices on smithi028 2024-02-05T06:30:22.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:21 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:22.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:21 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:22.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:21 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:30:22.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:21 smithi028 ceph-mon[37984]: Adjusting osd_memory_target on smithi028 to 16069M 2024-02-05T06:30:22.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:21 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:22.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:21 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:30:22.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:21 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:30:22.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:21 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:22.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:21 smithi028 ceph-mon[37984]: osdmap e9: 1 total, 1 up, 1 in 2024-02-05T06:30:22.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:21 smithi167 ceph-mon[32774]: pgmap v41: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:22.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:21 smithi167 ceph-mon[32774]: Detected new or changed devices on smithi028 2024-02-05T06:30:22.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:21 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:22.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:21 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:22.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:21 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:30:22.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:21 smithi167 ceph-mon[32774]: Adjusting osd_memory_target on smithi028 to 16069M 2024-02-05T06:30:22.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:21 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:22.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:21 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:30:22.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:21 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:30:22.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:21 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:22.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:21 smithi167 ceph-mon[32774]: osdmap e9: 1 total, 1 up, 1 in 2024-02-05T06:30:23.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:23 smithi028 ceph-mon[29442]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:23.940 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:23 smithi028 ceph-mon[29442]: from='client.14241 v1:172.21.15.28:0/2863183840' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:30:23.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:23 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T06:30:23.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:23 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T06:30:23.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:23 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:30:23.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:23 smithi028 ceph-mon[37984]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:23.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:23 smithi028 ceph-mon[37984]: from='client.14241 v1:172.21.15.28:0/2863183840' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:30:23.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:23 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T06:30:23.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:23 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T06:30:23.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:23 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:30:24.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:23 smithi167 ceph-mon[32774]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:24.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:23 smithi167 ceph-mon[32774]: from='client.14241 v1:172.21.15.28:0/2863183840' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:30:24.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:23 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T06:30:24.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:23 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T06:30:24.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:23 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:30:26.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:25 smithi028 ceph-mon[29442]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:26.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:25 smithi028 ceph-mon[37984]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:26.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:25 smithi167 ceph-mon[32774]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:27.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:26 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/393024548' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52245f6e-eb96-4d0d-985b-dbb6fd3ce8de"}]: dispatch 2024-02-05T06:30:27.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:26 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/393024548' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "52245f6e-eb96-4d0d-985b-dbb6fd3ce8de"}]': finished 2024-02-05T06:30:27.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:26 smithi028 ceph-mon[29442]: osdmap e10: 2 total, 1 up, 2 in 2024-02-05T06:30:27.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:26 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T06:30:27.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:26 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/616392184' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T06:30:27.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:26 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/393024548' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52245f6e-eb96-4d0d-985b-dbb6fd3ce8de"}]: dispatch 2024-02-05T06:30:27.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:26 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/393024548' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "52245f6e-eb96-4d0d-985b-dbb6fd3ce8de"}]': finished 2024-02-05T06:30:27.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:26 smithi028 ceph-mon[37984]: osdmap e10: 2 total, 1 up, 2 in 2024-02-05T06:30:27.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:26 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T06:30:27.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:26 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/616392184' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T06:30:27.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:26 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/393024548' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "52245f6e-eb96-4d0d-985b-dbb6fd3ce8de"}]: dispatch 2024-02-05T06:30:27.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:26 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/393024548' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "52245f6e-eb96-4d0d-985b-dbb6fd3ce8de"}]': finished 2024-02-05T06:30:27.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:26 smithi167 ceph-mon[32774]: osdmap e10: 2 total, 1 up, 2 in 2024-02-05T06:30:27.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:26 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T06:30:27.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:26 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/616392184' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T06:30:28.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:27 smithi028 ceph-mon[29442]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:28.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:27 smithi028 ceph-mon[37984]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:28.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:27 smithi167 ceph-mon[32774]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:30.137 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:29 smithi028 ceph-mon[29442]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:30.138 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:29 smithi028 ceph-mon[37984]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:30.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:29 smithi167 ceph-mon[32774]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:32.140 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:31 smithi028 ceph-mon[29442]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:32.141 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:31 smithi028 ceph-mon[37984]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:32.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:31 smithi167 ceph-mon[32774]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:33.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:32 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-05T06:30:33.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:32 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:30:33.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:32 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-05T06:30:33.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:32 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:30:33.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:32 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-05T06:30:33.255 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:32 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:30:34.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:33 smithi028 ceph-mon[37984]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:34.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:33 smithi028 ceph-mon[37984]: Deploying daemon osd.1 on smithi028 2024-02-05T06:30:34.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:33 smithi028 ceph-mon[29442]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:34.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:33 smithi028 ceph-mon[29442]: Deploying daemon osd.1 on smithi028 2024-02-05T06:30:34.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:33 smithi167 ceph-mon[32774]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:34.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:33 smithi167 ceph-mon[32774]: Deploying daemon osd.1 on smithi028 2024-02-05T06:30:36.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:35 smithi167 ceph-mon[32774]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:36.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:35 smithi028 ceph-mon[29442]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:36.264 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:35 smithi028 ceph-mon[37984]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:38.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:37 smithi028 ceph-mon[37984]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:38.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:37 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:38.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:37 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:38.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:37 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:30:38.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:37 smithi028 ceph-mon[29442]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:38.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:37 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:38.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:37 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:38.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:37 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:30:38.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:37 smithi167 ceph-mon[32774]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:38.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:37 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:38.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:37 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:38.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:37 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:30:40.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:39 smithi167 ceph-mon[32774]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:40.242 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:39 smithi028 ceph-mon[29442]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:40.242 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:39 smithi028 ceph-mon[37984]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:41.703 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 1 on host 'smithi028' 2024-02-05T06:30:41.940 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:41 smithi028 ceph-mon[37984]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:41.940 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:41 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:41.940 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:41 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:41.940 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:41 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:30:41.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:41 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:30:41.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:41 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:41.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:41 smithi028 ceph-mon[37984]: from='osd.1 v1:172.21.15.28:6805/1795224649' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-05T06:30:41.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:41 smithi028 ceph-mon[37984]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-05T06:30:41.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:41 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:41.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:41 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:41.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:41 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:30:41.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:41 smithi028 ceph-mon[29442]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:41.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:41 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:41.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:41 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:41.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:41 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:30:41.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:41 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:30:41.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:41 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:41.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:41 smithi028 ceph-mon[29442]: from='osd.1 v1:172.21.15.28:6805/1795224649' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-05T06:30:41.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:41 smithi028 ceph-mon[29442]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-05T06:30:41.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:41 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:41.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:41 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:41.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:41 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:30:42.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:41 smithi167 ceph-mon[32774]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:42.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:41 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:42.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:41 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:42.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:41 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:30:42.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:41 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:30:42.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:41 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:42.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:41 smithi167 ceph-mon[32774]: from='osd.1 v1:172.21.15.28:6805/1795224649' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-05T06:30:42.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:41 smithi167 ceph-mon[32774]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-05T06:30:42.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:41 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:42.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:41 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:42.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:41 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:30:42.350 DEBUG:teuthology.orchestra.run.smithi028:osd.1> sudo journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@osd.1.service 2024-02-05T06:30:42.353 INFO:tasks.cephadm:Deploying osd.2 on smithi028 with /dev/vg_nvme/lv_2... 2024-02-05T06:30:42.353 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-05T06:30:43.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:42 smithi028 ceph-mon[37984]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-05T06:30:43.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:42 smithi028 ceph-mon[37984]: from='osd.1 v1:172.21.15.28:6805/1795224649' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-05T06:30:43.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:42 smithi028 ceph-mon[37984]: osdmap e11: 2 total, 1 up, 2 in 2024-02-05T06:30:43.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:42 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T06:30:43.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:42 smithi028 ceph-mon[37984]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-05T06:30:43.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:42 smithi028 ceph-mon[29442]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-05T06:30:43.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:42 smithi028 ceph-mon[29442]: from='osd.1 v1:172.21.15.28:6805/1795224649' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-05T06:30:43.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:42 smithi028 ceph-mon[29442]: osdmap e11: 2 total, 1 up, 2 in 2024-02-05T06:30:43.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:42 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T06:30:43.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:42 smithi028 ceph-mon[29442]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-05T06:30:43.192 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 05 06:30:42 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-1[48053]: 2024-02-05T06:30:42.964+0000 7f2924298700 -1 osd.1 0 waiting for initial osdmap 2024-02-05T06:30:43.192 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 05 06:30:42 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-1[48053]: 2024-02-05T06:30:42.966+0000 7f2920092700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T06:30:43.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:42 smithi167 ceph-mon[32774]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-05T06:30:43.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:42 smithi167 ceph-mon[32774]: from='osd.1 v1:172.21.15.28:6805/1795224649' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-05T06:30:43.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:42 smithi167 ceph-mon[32774]: osdmap e11: 2 total, 1 up, 2 in 2024-02-05T06:30:43.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:42 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T06:30:43.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:42 smithi167 ceph-mon[32774]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-05T06:30:43.561 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:30:44.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:43 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:30:44.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:43 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:30:44.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:43 smithi028 ceph-mon[29442]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:44.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:43 smithi028 ceph-mon[29442]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-02-05T06:30:44.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:43 smithi028 ceph-mon[29442]: osdmap e12: 2 total, 1 up, 2 in 2024-02-05T06:30:44.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:43 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T06:30:44.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:43 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T06:30:44.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:43 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:30:44.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:43 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:30:44.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:43 smithi028 ceph-mon[37984]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:44.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:43 smithi028 ceph-mon[37984]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-02-05T06:30:44.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:43 smithi028 ceph-mon[37984]: osdmap e12: 2 total, 1 up, 2 in 2024-02-05T06:30:44.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:43 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T06:30:44.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:43 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T06:30:44.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:43 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:30:44.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:43 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:30:44.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:43 smithi167 ceph-mon[32774]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:44.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:43 smithi167 ceph-mon[32774]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-02-05T06:30:44.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:43 smithi167 ceph-mon[32774]: osdmap e12: 2 total, 1 up, 2 in 2024-02-05T06:30:44.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:43 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T06:30:44.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:43 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T06:30:45.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:44 smithi028 ceph-mon[29442]: osd.1 v1:172.21.15.28:6805/1795224649 boot 2024-02-05T06:30:45.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:44 smithi028 ceph-mon[29442]: osdmap e13: 2 total, 2 up, 2 in 2024-02-05T06:30:45.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:44 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T06:30:45.147 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:44 smithi028 ceph-mon[37984]: osd.1 v1:172.21.15.28:6805/1795224649 boot 2024-02-05T06:30:45.147 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:44 smithi028 ceph-mon[37984]: osdmap e13: 2 total, 2 up, 2 in 2024-02-05T06:30:45.147 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:44 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T06:30:45.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:44 smithi167 ceph-mon[32774]: osd.1 v1:172.21.15.28:6805/1795224649 boot 2024-02-05T06:30:45.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:44 smithi167 ceph-mon[32774]: osdmap e13: 2 total, 2 up, 2 in 2024-02-05T06:30:45.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:44 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T06:30:45.738 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-05T06:30:45.739 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-05T06:30:45.739 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-02-05T06:30:45.739 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-02-05T06:30:45.739 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012236 s, 857 MB/s 2024-02-05T06:30:45.739 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-02-05T06:30:46.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:45 smithi028 ceph-mon[29442]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:46.117 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:45 smithi028 ceph-mon[37984]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:46.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:45 smithi167 ceph-mon[32774]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-05T06:30:46.792 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph orch daemon add osd smithi028:vg_nvme/lv_2 2024-02-05T06:30:47.038 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:30:47.147 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:46 smithi028 ceph-mon[37984]: osdmap e14: 2 total, 2 up, 2 in 2024-02-05T06:30:47.147 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:46 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:47.147 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:46 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:47.147 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:46 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:30:47.147 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:46 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:30:47.147 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:46 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:47.147 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:46 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:30:47.147 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:46 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:30:47.147 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:46 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:47.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:46 smithi028 ceph-mon[29442]: osdmap e14: 2 total, 2 up, 2 in 2024-02-05T06:30:47.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:46 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:47.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:46 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:47.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:46 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:30:47.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:46 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:30:47.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:46 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:47.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:46 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:30:47.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:46 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:30:47.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:46 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:47.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:46 smithi167 ceph-mon[32774]: osdmap e14: 2 total, 2 up, 2 in 2024-02-05T06:30:47.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:46 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:47.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:46 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:47.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:46 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:30:47.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:46 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:30:47.476 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:46 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:47.476 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:46 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:30:47.476 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:46 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:30:47.476 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:46 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:30:48.069 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:47 smithi028 ceph-mon[29442]: pgmap v60: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:30:48.070 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:47 smithi028 ceph-mon[29442]: Detected new or changed devices on smithi028 2024-02-05T06:30:48.070 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:47 smithi028 ceph-mon[29442]: Adjusting osd_memory_target on smithi028 to 8034M 2024-02-05T06:30:48.070 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:47 smithi028 ceph-mon[37984]: pgmap v60: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:30:48.070 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:47 smithi028 ceph-mon[37984]: Detected new or changed devices on smithi028 2024-02-05T06:30:48.070 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:47 smithi028 ceph-mon[37984]: Adjusting osd_memory_target on smithi028 to 8034M 2024-02-05T06:30:48.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:47 smithi167 ceph-mon[32774]: pgmap v60: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:30:48.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:47 smithi167 ceph-mon[32774]: Detected new or changed devices on smithi028 2024-02-05T06:30:48.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:47 smithi167 ceph-mon[32774]: Adjusting osd_memory_target on smithi028 to 8034M 2024-02-05T06:30:49.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:48 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T06:30:49.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:48 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T06:30:49.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:48 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:30:49.434 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:48 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T06:30:49.434 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:48 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T06:30:49.434 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:48 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:30:49.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:48 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T06:30:49.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:48 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T06:30:49.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:48 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:30:50.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:49 smithi028 ceph-mon[29442]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:30:50.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:49 smithi028 ceph-mon[29442]: from='client.44168 v1:172.21.15.28:0/1549793198' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:30:50.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:49 smithi028 ceph-mon[37984]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:30:50.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:49 smithi028 ceph-mon[37984]: from='client.44168 v1:172.21.15.28:0/1549793198' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:30:50.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:49 smithi167 ceph-mon[32774]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:30:50.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:49 smithi167 ceph-mon[32774]: from='client.44168 v1:172.21.15.28:0/1549793198' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:30:52.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:51 smithi028 ceph-mon[29442]: pgmap v62: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:30:52.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:51 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1494256173' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e18b630-afdf-4de9-a1a2-37bdfdd01c67"}]: dispatch 2024-02-05T06:30:52.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:51 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1494256173' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e18b630-afdf-4de9-a1a2-37bdfdd01c67"}]': finished 2024-02-05T06:30:52.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:51 smithi028 ceph-mon[29442]: osdmap e15: 3 total, 2 up, 3 in 2024-02-05T06:30:52.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:51 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T06:30:52.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:51 smithi028 ceph-mon[37984]: pgmap v62: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:30:52.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:51 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1494256173' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e18b630-afdf-4de9-a1a2-37bdfdd01c67"}]: dispatch 2024-02-05T06:30:52.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:51 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1494256173' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e18b630-afdf-4de9-a1a2-37bdfdd01c67"}]': finished 2024-02-05T06:30:52.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:51 smithi028 ceph-mon[37984]: osdmap e15: 3 total, 2 up, 3 in 2024-02-05T06:30:52.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:51 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T06:30:52.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:51 smithi167 ceph-mon[32774]: pgmap v62: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:30:52.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:51 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1494256173' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e18b630-afdf-4de9-a1a2-37bdfdd01c67"}]: dispatch 2024-02-05T06:30:52.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:51 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1494256173' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e18b630-afdf-4de9-a1a2-37bdfdd01c67"}]': finished 2024-02-05T06:30:52.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:51 smithi167 ceph-mon[32774]: osdmap e15: 3 total, 2 up, 3 in 2024-02-05T06:30:52.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:51 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T06:30:53.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:52 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1414174185' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T06:30:53.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:52 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1414174185' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T06:30:53.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:52 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1414174185' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T06:30:54.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:53 smithi028 ceph-mon[29442]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:30:54.356 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:53 smithi028 ceph-mon[37984]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:30:54.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:53 smithi167 ceph-mon[32774]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:30:56.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:55 smithi028 ceph-mon[29442]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:30:56.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:55 smithi028 ceph-mon[37984]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:30:56.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:55 smithi167 ceph-mon[32774]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:30:58.190 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:57 smithi028 ceph-mon[29442]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:30:58.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:57 smithi028 ceph-mon[37984]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:30:58.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:57 smithi167 ceph-mon[32774]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:30:59.190 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:59 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-05T06:30:59.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:59 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:30:59.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:30:59 smithi028 ceph-mon[37984]: Deploying daemon osd.2 on smithi028 2024-02-05T06:30:59.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:59 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-05T06:30:59.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:59 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:30:59.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:30:59 smithi028 ceph-mon[29442]: Deploying daemon osd.2 on smithi028 2024-02-05T06:30:59.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:59 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-05T06:30:59.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:59 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:30:59.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:30:59 smithi167 ceph-mon[32774]: Deploying daemon osd.2 on smithi028 2024-02-05T06:31:00.267 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:00 smithi028 ceph-mon[29442]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:31:00.267 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:00 smithi028 ceph-mon[37984]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:31:00.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:00 smithi167 ceph-mon[32774]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:31:02.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:02 smithi028 ceph-mon[29442]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:31:02.166 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:02 smithi028 ceph-mon[37984]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:31:02.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:02 smithi167 ceph-mon[32774]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:31:04.277 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:04 smithi167 ceph-mon[32774]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:31:04.278 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:04 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:04.278 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:04 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:04.278 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:04 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:31:04.298 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:04 smithi028 ceph-mon[29442]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:31:04.298 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:04 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:04.298 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:04 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:04.298 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:04 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:31:04.298 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:04 smithi028 ceph-mon[37984]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:31:04.298 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:04 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:04.298 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:04 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:04.298 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:04 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:31:06.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:06 smithi028 ceph-mon[29442]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:31:06.434 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:06 smithi028 ceph-mon[37984]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:31:06.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:06 smithi167 ceph-mon[32774]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:31:07.160 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 2 on host 'smithi028' 2024-02-05T06:31:07.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:07 smithi028 ceph-mon[29442]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:31:07.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:07 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:07.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:07 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:07.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:07 smithi028 ceph-mon[29442]: from='osd.2 v1:172.21.15.28:6809/1993280018' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-05T06:31:07.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:07 smithi028 ceph-mon[29442]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-05T06:31:07.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:07 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:07.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:07 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:07.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:07 smithi028 ceph-mon[37984]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:31:07.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:07 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:07.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:07 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:07.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:07 smithi028 ceph-mon[37984]: from='osd.2 v1:172.21.15.28:6809/1993280018' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-05T06:31:07.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:07 smithi028 ceph-mon[37984]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-05T06:31:07.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:07 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:07.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:07 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:07.725 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:07 smithi167 ceph-mon[32774]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:31:07.725 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:07 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:07.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:07 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:07.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:07 smithi167 ceph-mon[32774]: from='osd.2 v1:172.21.15.28:6809/1993280018' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-05T06:31:07.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:07 smithi167 ceph-mon[32774]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-05T06:31:07.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:07 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:07.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:07 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:07.772 DEBUG:teuthology.orchestra.run.smithi028:osd.2> sudo journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@osd.2.service 2024-02-05T06:31:07.774 INFO:tasks.cephadm:Deploying osd.3 on smithi028 with /dev/vg_nvme/lv_1... 2024-02-05T06:31:07.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-05T06:31:08.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:08 smithi028 ceph-mon[29442]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-05T06:31:08.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:08 smithi028 ceph-mon[29442]: osdmap e16: 3 total, 2 up, 3 in 2024-02-05T06:31:08.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:08 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T06:31:08.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:08 smithi028 ceph-mon[29442]: from='osd.2 v1:172.21.15.28:6809/1993280018' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-05T06:31:08.604 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:08 smithi028 ceph-mon[29442]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-05T06:31:08.605 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:08 smithi028 ceph-mon[37984]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-05T06:31:08.605 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:08 smithi028 ceph-mon[37984]: osdmap e16: 3 total, 2 up, 3 in 2024-02-05T06:31:08.605 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:08 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T06:31:08.605 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:08 smithi028 ceph-mon[37984]: from='osd.2 v1:172.21.15.28:6809/1993280018' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-05T06:31:08.605 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:08 smithi028 ceph-mon[37984]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-05T06:31:08.725 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:08 smithi167 ceph-mon[32774]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-05T06:31:08.725 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:08 smithi167 ceph-mon[32774]: osdmap e16: 3 total, 2 up, 3 in 2024-02-05T06:31:08.725 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:08 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T06:31:08.725 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:08 smithi167 ceph-mon[32774]: from='osd.2 v1:172.21.15.28:6809/1993280018' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-05T06:31:08.725 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:08 smithi167 ceph-mon[32774]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-05T06:31:08.864 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 05 06:31:08 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-2[53688]: 2024-02-05T06:31:08.603+0000 7fd6d5a83700 -1 osd.2 0 waiting for initial osdmap 2024-02-05T06:31:08.864 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 05 06:31:08 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-2[53688]: 2024-02-05T06:31:08.606+0000 7fd6cd875700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T06:31:09.340 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:31:09.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:09 smithi028 ceph-mon[29442]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:31:09.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:09 smithi028 ceph-mon[29442]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-02-05T06:31:09.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:09 smithi028 ceph-mon[29442]: osdmap e17: 3 total, 2 up, 3 in 2024-02-05T06:31:09.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:09 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T06:31:09.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:09 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T06:31:09.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:09 smithi028 ceph-mon[29442]: from='osd.2 ' entity='osd.2' 2024-02-05T06:31:09.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:09 smithi028 ceph-mon[37984]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:31:09.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:09 smithi028 ceph-mon[37984]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-02-05T06:31:09.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:09 smithi028 ceph-mon[37984]: osdmap e17: 3 total, 2 up, 3 in 2024-02-05T06:31:09.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:09 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T06:31:09.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:09 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T06:31:09.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:09 smithi028 ceph-mon[37984]: from='osd.2 ' entity='osd.2' 2024-02-05T06:31:09.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:09 smithi167 ceph-mon[32774]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:31:09.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:09 smithi167 ceph-mon[32774]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-02-05T06:31:09.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:09 smithi167 ceph-mon[32774]: osdmap e17: 3 total, 2 up, 3 in 2024-02-05T06:31:09.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:09 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T06:31:09.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:09 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T06:31:09.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:09 smithi167 ceph-mon[32774]: from='osd.2 ' entity='osd.2' 2024-02-05T06:31:10.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:10 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:31:10.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:10 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:31:10.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:10 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T06:31:10.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:10 smithi028 ceph-mon[29442]: osd.2 v1:172.21.15.28:6809/1993280018 boot 2024-02-05T06:31:10.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:10 smithi028 ceph-mon[29442]: osdmap e18: 3 total, 3 up, 3 in 2024-02-05T06:31:10.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:10 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T06:31:10.679 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:10 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:31:10.679 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:10 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:31:10.679 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:10 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T06:31:10.679 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:10 smithi028 ceph-mon[37984]: osd.2 v1:172.21.15.28:6809/1993280018 boot 2024-02-05T06:31:10.679 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:10 smithi028 ceph-mon[37984]: osdmap e18: 3 total, 3 up, 3 in 2024-02-05T06:31:10.679 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:10 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T06:31:10.725 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:10 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:31:10.725 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:10 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:31:10.725 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:10 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T06:31:10.725 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:10 smithi167 ceph-mon[32774]: osd.2 v1:172.21.15.28:6809/1993280018 boot 2024-02-05T06:31:10.725 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:10 smithi167 ceph-mon[32774]: osdmap e18: 3 total, 3 up, 3 in 2024-02-05T06:31:10.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:10 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T06:31:11.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[29442]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:31:11.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-05T06:31:11.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[29442]: Detected new or changed devices on smithi028 2024-02-05T06:31:11.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:11.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:11.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:31:11.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:31:11.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:31:11.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[29442]: Adjusting osd_memory_target on smithi028 to 5356M 2024-02-05T06:31:11.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:11.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:31:11.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:31:11.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:11.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-05T06:31:11.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[29442]: osdmap e19: 3 total, 3 up, 3 in 2024-02-05T06:31:11.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T06:31:11.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:31:11.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:31:11.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:31:11.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:11.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[37984]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:31:11.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-05T06:31:11.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[37984]: Detected new or changed devices on smithi028 2024-02-05T06:31:11.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:11.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:11.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:31:11.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:31:11.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:31:11.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[37984]: Adjusting osd_memory_target on smithi028 to 5356M 2024-02-05T06:31:11.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:11.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:31:11.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:31:11.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:11.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-05T06:31:11.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[37984]: osdmap e19: 3 total, 3 up, 3 in 2024-02-05T06:31:11.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T06:31:11.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:31:11.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:31:11.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:31:11.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:11 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:11.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:11 smithi167 ceph-mon[32774]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-05T06:31:11.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:11 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-05T06:31:11.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:11 smithi167 ceph-mon[32774]: Detected new or changed devices on smithi028 2024-02-05T06:31:11.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:11 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:11.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:11 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:11.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:11 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:31:11.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:11 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:31:11.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:11 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:31:11.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:11 smithi167 ceph-mon[32774]: Adjusting osd_memory_target on smithi028 to 5356M 2024-02-05T06:31:11.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:11 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:11.726 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:11 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:31:11.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:11 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:31:11.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:11 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:11.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:11 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-05T06:31:11.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:11 smithi167 ceph-mon[32774]: osdmap e19: 3 total, 3 up, 3 in 2024-02-05T06:31:11.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:11 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T06:31:11.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:11 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:31:11.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:11 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:31:11.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:11 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:31:11.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:11 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:11.824 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-05T06:31:11.825 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-05T06:31:11.825 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-02-05T06:31:11.825 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-02-05T06:31:11.825 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0084611 s, 1.2 GB/s 2024-02-05T06:31:11.825 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-02-05T06:31:12.191 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 05 06:31:11 smithi028 sudo[57427]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-05T06:31:12.191 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 05 06:31:11 smithi028 sudo[57427]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T06:31:12.191 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 05 06:31:11 smithi028 sudo[57427]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T06:31:12.191 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 05 06:31:11 smithi028 sudo[57427]: pam_unix(sudo:session): session closed for user root 2024-02-05T06:31:12.511 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 05 06:31:12 smithi028 sudo[57432]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-05T06:31:12.511 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 05 06:31:12 smithi028 sudo[57432]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T06:31:12.511 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 05 06:31:12 smithi028 sudo[57432]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T06:31:12.511 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 05 06:31:12 smithi028 sudo[57432]: pam_unix(sudo:session): session closed for user root 2024-02-05T06:31:12.539 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph orch daemon add osd smithi028:vg_nvme/lv_1 2024-02-05T06:31:12.786 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:31:12.791 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:12 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-05T06:31:12.791 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:12 smithi028 ceph-mon[29442]: osdmap e20: 3 total, 3 up, 3 in 2024-02-05T06:31:12.791 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:12 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:12.792 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:12 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-05T06:31:12.792 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:12 smithi028 ceph-mon[37984]: osdmap e20: 3 total, 3 up, 3 in 2024-02-05T06:31:12.792 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:12 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:12.792 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 05 06:31:12 smithi028 sudo[57435]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-05T06:31:12.792 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 05 06:31:12 smithi028 sudo[57435]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T06:31:12.792 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 05 06:31:12 smithi028 sudo[57435]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T06:31:12.792 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 05 06:31:12 smithi028 sudo[57435]: pam_unix(sudo:session): session closed for user root 2024-02-05T06:31:12.975 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:12 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-05T06:31:12.976 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:12 smithi167 ceph-mon[32774]: osdmap e20: 3 total, 3 up, 3 in 2024-02-05T06:31:12.976 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:12 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:13.044 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 05 06:31:12 smithi028 sudo[57498]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-05T06:31:13.044 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 05 06:31:12 smithi028 sudo[57498]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T06:31:13.044 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 05 06:31:12 smithi028 sudo[57498]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T06:31:13.044 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 05 06:31:12 smithi028 sudo[57498]: pam_unix(sudo:session): session closed for user root 2024-02-05T06:31:13.323 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 05 06:31:13 smithi028 sudo[57518]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-05T06:31:13.324 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 05 06:31:13 smithi028 sudo[57518]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T06:31:13.324 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 05 06:31:13 smithi028 sudo[57518]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T06:31:13.324 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 05 06:31:13 smithi028 sudo[57518]: pam_unix(sudo:session): session closed for user root 2024-02-05T06:31:13.580 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 05 06:31:13 smithi028 sudo[57534]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-05T06:31:13.580 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 05 06:31:13 smithi028 sudo[57534]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T06:31:13.580 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 05 06:31:13 smithi028 sudo[57534]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T06:31:13.580 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 05 06:31:13 smithi028 sudo[57534]: pam_unix(sudo:session): session closed for user root 2024-02-05T06:31:13.850 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:13 smithi028 sudo[57541]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-05T06:31:13.850 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:13 smithi028 sudo[57541]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T06:31:13.850 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:13 smithi028 sudo[57541]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T06:31:13.850 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:13 smithi028 sudo[57541]: pam_unix(sudo:session): session closed for user root 2024-02-05T06:31:14.117 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:13 smithi167 ceph-mon[32774]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:14.117 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:13 smithi167 ceph-mon[32774]: osdmap e21: 3 total, 3 up, 3 in 2024-02-05T06:31:14.117 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:13 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T06:31:14.117 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:13 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T06:31:14.118 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:13 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:31:14.118 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:14 smithi167 sudo[35360]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-05T06:31:14.118 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:14 smithi167 sudo[35360]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T06:31:14.118 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:14 smithi167 sudo[35360]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T06:31:14.118 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:14 smithi167 sudo[35360]: pam_unix(sudo:session): session closed for user root 2024-02-05T06:31:14.141 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:13 smithi028 ceph-mon[37984]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:14.141 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:13 smithi028 ceph-mon[37984]: osdmap e21: 3 total, 3 up, 3 in 2024-02-05T06:31:14.141 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:13 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T06:31:14.141 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:13 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T06:31:14.142 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:13 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:31:14.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:13 smithi028 sudo[57544]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-05T06:31:14.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:13 smithi028 sudo[57544]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T06:31:14.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:13 smithi028 sudo[57544]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T06:31:14.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:13 smithi028 sudo[57544]: pam_unix(sudo:session): session closed for user root 2024-02-05T06:31:14.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:13 smithi028 ceph-mon[29442]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:14.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:13 smithi028 ceph-mon[29442]: osdmap e21: 3 total, 3 up, 3 in 2024-02-05T06:31:14.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:13 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T06:31:14.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:13 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T06:31:14.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:13 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:31:14.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:14 smithi167 sudo[35363]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-05T06:31:14.476 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:14 smithi167 sudo[35363]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T06:31:14.476 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:14 smithi167 sudo[35363]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T06:31:14.476 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:14 smithi167 sudo[35363]: pam_unix(sudo:session): session closed for user root 2024-02-05T06:31:14.899 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:14 smithi028 sudo[57629]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-05T06:31:14.899 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:14 smithi028 sudo[57629]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T06:31:14.899 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:14 smithi028 sudo[57629]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T06:31:14.899 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:14 smithi028 sudo[57629]: pam_unix(sudo:session): session closed for user root 2024-02-05T06:31:15.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:14 smithi028 ceph-mon[29442]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-05T06:31:15.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:14 smithi028 ceph-mon[29442]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-05T06:31:15.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:14 smithi028 ceph-mon[29442]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-05T06:31:15.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:14 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T06:31:15.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:14 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T06:31:15.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:14 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:31:15.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:14 smithi028 ceph-mon[29442]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-05T06:31:15.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:14 smithi028 ceph-mon[29442]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-05T06:31:15.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:14 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:15.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:14 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T06:31:15.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:14 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T06:31:15.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:14 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:31:15.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:14 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T06:31:15.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:14 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T06:31:15.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:14 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:31:15.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:14 smithi028 sudo[57664]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-05T06:31:15.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:14 smithi028 sudo[57664]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-05T06:31:15.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:14 smithi028 sudo[57664]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-05T06:31:15.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:14 smithi028 sudo[57664]: pam_unix(sudo:session): session closed for user root 2024-02-05T06:31:15.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:14 smithi028 ceph-mon[37984]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-05T06:31:15.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:14 smithi028 ceph-mon[37984]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-05T06:31:15.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:14 smithi028 ceph-mon[37984]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-05T06:31:15.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:14 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T06:31:15.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:14 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T06:31:15.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:14 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:31:15.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:14 smithi028 ceph-mon[37984]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-05T06:31:15.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:14 smithi028 ceph-mon[37984]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-05T06:31:15.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:14 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:15.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:14 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T06:31:15.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:14 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T06:31:15.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:14 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:31:15.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:14 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T06:31:15.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:14 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T06:31:15.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:14 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:31:15.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:14 smithi167 ceph-mon[32774]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-05T06:31:15.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:14 smithi167 ceph-mon[32774]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-05T06:31:15.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:14 smithi167 ceph-mon[32774]: mgrmap e15: y(active, since 2m), standbys: x 2024-02-05T06:31:15.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:14 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T06:31:15.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:14 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T06:31:15.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:14 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:31:15.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:14 smithi167 ceph-mon[32774]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-05T06:31:15.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:14 smithi167 ceph-mon[32774]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-05T06:31:15.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:14 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:15.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:14 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T06:31:15.227 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:14 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T06:31:15.227 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:14 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:31:15.227 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:14 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T06:31:15.227 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:14 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T06:31:15.227 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:14 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:31:16.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:15 smithi028 ceph-mon[29442]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:16.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:15 smithi028 ceph-mon[29442]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-05T06:31:16.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:15 smithi028 ceph-mon[29442]: from='client.44201 v1:172.21.15.28:0/3357199784' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:31:16.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:15 smithi028 ceph-mon[29442]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-05T06:31:16.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:15 smithi028 ceph-mon[37984]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:16.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:15 smithi028 ceph-mon[37984]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-05T06:31:16.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:15 smithi028 ceph-mon[37984]: from='client.44201 v1:172.21.15.28:0/3357199784' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:31:16.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:15 smithi028 ceph-mon[37984]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-05T06:31:16.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:15 smithi167 ceph-mon[32774]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:16.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:15 smithi167 ceph-mon[32774]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-05T06:31:16.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:15 smithi167 ceph-mon[32774]: from='client.44201 v1:172.21.15.28:0/3357199784' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:31:16.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:15 smithi167 ceph-mon[32774]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-05T06:31:18.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:17 smithi028 ceph-mon[29442]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:18.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:17 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/307751557' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ae795af-3d59-41a6-95eb-b8447ff61282"}]: dispatch 2024-02-05T06:31:18.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:17 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/307751557' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ae795af-3d59-41a6-95eb-b8447ff61282"}]': finished 2024-02-05T06:31:18.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:17 smithi028 ceph-mon[29442]: osdmap e22: 4 total, 3 up, 4 in 2024-02-05T06:31:18.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:17 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T06:31:18.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:17 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2676627903' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T06:31:18.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:17 smithi028 ceph-mon[37984]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:18.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:17 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/307751557' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ae795af-3d59-41a6-95eb-b8447ff61282"}]: dispatch 2024-02-05T06:31:18.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:17 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/307751557' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ae795af-3d59-41a6-95eb-b8447ff61282"}]': finished 2024-02-05T06:31:18.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:17 smithi028 ceph-mon[37984]: osdmap e22: 4 total, 3 up, 4 in 2024-02-05T06:31:18.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:17 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T06:31:18.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:17 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2676627903' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T06:31:18.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:17 smithi167 ceph-mon[32774]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:18.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:17 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/307751557' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ae795af-3d59-41a6-95eb-b8447ff61282"}]: dispatch 2024-02-05T06:31:18.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:17 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/307751557' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ae795af-3d59-41a6-95eb-b8447ff61282"}]': finished 2024-02-05T06:31:18.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:17 smithi167 ceph-mon[32774]: osdmap e22: 4 total, 3 up, 4 in 2024-02-05T06:31:18.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:17 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T06:31:18.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:17 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2676627903' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T06:31:20.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:19 smithi028 ceph-mon[29442]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:20.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:19 smithi028 ceph-mon[37984]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:20.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:19 smithi167 ceph-mon[32774]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:22.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:21 smithi028 ceph-mon[29442]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:22.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:21 smithi028 ceph-mon[37984]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:22.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:21 smithi167 ceph-mon[32774]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:23.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:22 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:31:23.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:22 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:31:23.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:22 smithi028 ceph-mon[29442]: osdmap e23: 4 total, 3 up, 4 in 2024-02-05T06:31:23.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:22 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T06:31:23.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:22 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:31:23.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:22 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:31:23.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:22 smithi028 ceph-mon[37984]: osdmap e23: 4 total, 3 up, 4 in 2024-02-05T06:31:23.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:22 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T06:31:23.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:22 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:31:23.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:22 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:31:23.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:22 smithi167 ceph-mon[32774]: osdmap e23: 4 total, 3 up, 4 in 2024-02-05T06:31:23.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:22 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T06:31:24.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:23 smithi028 ceph-mon[29442]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:24.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:23 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-05T06:31:24.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:23 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:31:24.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:23 smithi028 ceph-mon[37984]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:24.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:23 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-05T06:31:24.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:23 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:31:24.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:23 smithi167 ceph-mon[32774]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:24.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:23 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-05T06:31:24.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:23 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:31:24.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:24 smithi028 ceph-mon[29442]: Deploying daemon osd.3 on smithi028 2024-02-05T06:31:24.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:24 smithi028 ceph-mon[37984]: Deploying daemon osd.3 on smithi028 2024-02-05T06:31:25.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:24 smithi167 ceph-mon[32774]: Deploying daemon osd.3 on smithi028 2024-02-05T06:31:26.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:25 smithi028 ceph-mon[29442]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:26.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:25 smithi028 ceph-mon[37984]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:26.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:25 smithi167 ceph-mon[32774]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:28.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:27 smithi028 ceph-mon[37984]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:28.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:27 smithi028 ceph-mon[29442]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:28.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:27 smithi167 ceph-mon[32774]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:30.075 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:29 smithi028 ceph-mon[29442]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:30.075 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:29 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:30.075 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:29 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:30.075 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:29 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:31:30.076 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:29 smithi028 ceph-mon[37984]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:30.076 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:29 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:30.076 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:29 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:30.076 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:29 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:31:30.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:29 smithi167 ceph-mon[32774]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:30.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:29 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:30.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:29 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:30.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:29 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:31:32.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:31 smithi028 ceph-mon[29442]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:32.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:31 smithi028 ceph-mon[37984]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:32.225 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:31 smithi167 ceph-mon[32774]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:32.899 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 3 on host 'smithi028' 2024-02-05T06:31:33.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:33 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:33.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:33 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:33.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:33 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:31:33.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:33 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:31:33.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:33 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:33.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:33 smithi028 ceph-mon[29442]: from='osd.3 v1:172.21.15.28:6813/1294621398' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-05T06:31:33.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:33 smithi028 ceph-mon[29442]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-05T06:31:33.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:33 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:33.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:33 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:33.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:33 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:31:33.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:33 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:33.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:33 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:33.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:33 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:31:33.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:33 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:31:33.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:33 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:33.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:33 smithi028 ceph-mon[37984]: from='osd.3 v1:172.21.15.28:6813/1294621398' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-05T06:31:33.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:33 smithi028 ceph-mon[37984]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-05T06:31:33.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:33 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:33.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:33 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:33.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:33 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:31:33.476 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:33 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:33.476 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:33 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:33.476 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:33 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:31:33.476 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:33 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:31:33.476 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:33 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:33.476 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:33 smithi167 ceph-mon[32774]: from='osd.3 v1:172.21.15.28:6813/1294621398' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-05T06:31:33.476 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:33 smithi167 ceph-mon[32774]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-05T06:31:33.476 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:33 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:33.476 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:33 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:33.477 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:33 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:31:33.518 DEBUG:teuthology.orchestra.run.smithi028:osd.3> sudo journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@osd.3.service 2024-02-05T06:31:33.520 INFO:tasks.cephadm:Deploying osd.4 on smithi167 with /dev/vg_nvme/lv_4... 2024-02-05T06:31:33.521 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-05T06:31:33.764 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:31:34.152 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:34 smithi167 ceph-mon[32774]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:34.152 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:34 smithi167 ceph-mon[32774]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-05T06:31:34.153 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:34 smithi167 ceph-mon[32774]: osdmap e24: 4 total, 3 up, 4 in 2024-02-05T06:31:34.153 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:34 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T06:31:34.153 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:34 smithi167 ceph-mon[32774]: from='osd.3 v1:172.21.15.28:6813/1294621398' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-05T06:31:34.153 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:34 smithi167 ceph-mon[32774]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-05T06:31:34.315 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:34 smithi028 ceph-mon[37984]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:34.315 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:34 smithi028 ceph-mon[37984]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-05T06:31:34.315 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:34 smithi028 ceph-mon[37984]: osdmap e24: 4 total, 3 up, 4 in 2024-02-05T06:31:34.315 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:34 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T06:31:34.315 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:34 smithi028 ceph-mon[37984]: from='osd.3 v1:172.21.15.28:6813/1294621398' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-05T06:31:34.315 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:34 smithi028 ceph-mon[37984]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-05T06:31:34.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:34 smithi028 ceph-mon[29442]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:34.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:34 smithi028 ceph-mon[29442]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-05T06:31:34.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:34 smithi028 ceph-mon[29442]: osdmap e24: 4 total, 3 up, 4 in 2024-02-05T06:31:34.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:34 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T06:31:34.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:34 smithi028 ceph-mon[29442]: from='osd.3 v1:172.21.15.28:6813/1294621398' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-05T06:31:34.316 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:34 smithi028 ceph-mon[29442]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-02-05T06:31:34.316 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 05 06:31:34 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-3[59489]: 2024-02-05T06:31:34.047+0000 7f4bb9f87700 -1 osd.3 0 waiting for initial osdmap 2024-02-05T06:31:34.316 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 05 06:31:34 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-3[59489]: 2024-02-05T06:31:34.050+0000 7f4bb5d81700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T06:31:35.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:35 smithi028 ceph-mon[29442]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-02-05T06:31:35.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:35 smithi028 ceph-mon[29442]: osdmap e25: 4 total, 3 up, 4 in 2024-02-05T06:31:35.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:35 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T06:31:35.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:35 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T06:31:35.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:35 smithi028 ceph-mon[37984]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-02-05T06:31:35.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:35 smithi028 ceph-mon[37984]: osdmap e25: 4 total, 3 up, 4 in 2024-02-05T06:31:35.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:35 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T06:31:35.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:35 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T06:31:35.476 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:35 smithi167 ceph-mon[32774]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-02-05T06:31:35.476 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:35 smithi167 ceph-mon[32774]: osdmap e25: 4 total, 3 up, 4 in 2024-02-05T06:31:35.476 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:35 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T06:31:35.476 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:35 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T06:31:35.626 INFO:teuthology.orchestra.run.smithi167.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-05T06:31:35.626 INFO:teuthology.orchestra.run.smithi167.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-05T06:31:35.626 INFO:teuthology.orchestra.run.smithi167.stderr: stderr: 10+0 records in 2024-02-05T06:31:35.626 INFO:teuthology.orchestra.run.smithi167.stderr:10+0 records out 2024-02-05T06:31:35.626 INFO:teuthology.orchestra.run.smithi167.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112988 s, 928 MB/s 2024-02-05T06:31:35.626 INFO:teuthology.orchestra.run.smithi167.stderr:--> Zapping successful for: 2024-02-05T06:31:36.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:36 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:31:36.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:36 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:31:36.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:36 smithi167 ceph-mon[32774]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:36.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:36 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T06:31:36.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:36 smithi167 ceph-mon[32774]: osd.3 v1:172.21.15.28:6813/1294621398 boot 2024-02-05T06:31:36.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:36 smithi167 ceph-mon[32774]: osdmap e26: 4 total, 4 up, 4 in 2024-02-05T06:31:36.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:36 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T06:31:36.273 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph orch daemon add osd smithi167:vg_nvme/lv_4 2024-02-05T06:31:36.330 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:36 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:31:36.331 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:36 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:31:36.331 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:36 smithi028 ceph-mon[37984]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:36.331 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:36 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T06:31:36.331 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:36 smithi028 ceph-mon[37984]: osd.3 v1:172.21.15.28:6813/1294621398 boot 2024-02-05T06:31:36.331 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:36 smithi028 ceph-mon[37984]: osdmap e26: 4 total, 4 up, 4 in 2024-02-05T06:31:36.331 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:36 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T06:31:36.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:36 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:31:36.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:36 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:31:36.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:36 smithi028 ceph-mon[29442]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-02-05T06:31:36.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:36 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T06:31:36.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:36 smithi028 ceph-mon[29442]: osd.3 v1:172.21.15.28:6813/1294621398 boot 2024-02-05T06:31:36.331 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:36 smithi028 ceph-mon[29442]: osdmap e26: 4 total, 4 up, 4 in 2024-02-05T06:31:36.332 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:36 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T06:31:36.516 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:31:37.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:37 smithi167 ceph-mon[32774]: osdmap e27: 4 total, 4 up, 4 in 2024-02-05T06:31:37.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:37 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:37.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:37 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:37.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:37 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:31:37.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:37 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:31:37.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:37 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:31:37.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:37 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:31:37.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:37 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:37.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:37 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:31:37.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:37 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:31:37.227 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:37 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:37.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:37 smithi028 ceph-mon[37984]: osdmap e27: 4 total, 4 up, 4 in 2024-02-05T06:31:37.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:37 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:37.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:37 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:37.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:37 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:31:37.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:37 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:31:37.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:37 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:31:37.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:37 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:31:37.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:37 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:37.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:37 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:31:37.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:37 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:31:37.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:37 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:37.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:37 smithi028 ceph-mon[29442]: osdmap e27: 4 total, 4 up, 4 in 2024-02-05T06:31:37.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:37 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:37.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:37 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:37.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:37 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:31:37.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:37 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:31:37.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:37 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:31:37.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:37 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:31:37.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:37 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:37.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:37 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:31:37.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:37 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:31:37.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:37 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:38.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:38 smithi167 ceph-mon[32774]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:38.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:38 smithi167 ceph-mon[32774]: Detected new or changed devices on smithi028 2024-02-05T06:31:38.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:38 smithi167 ceph-mon[32774]: Adjusting osd_memory_target on smithi028 to 4017M 2024-02-05T06:31:38.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:38 smithi028 ceph-mon[29442]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:38.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:38 smithi028 ceph-mon[29442]: Detected new or changed devices on smithi028 2024-02-05T06:31:38.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:38 smithi028 ceph-mon[29442]: Adjusting osd_memory_target on smithi028 to 4017M 2024-02-05T06:31:38.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:38 smithi028 ceph-mon[37984]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:38.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:38 smithi028 ceph-mon[37984]: Detected new or changed devices on smithi028 2024-02-05T06:31:38.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:38 smithi028 ceph-mon[37984]: Adjusting osd_memory_target on smithi028 to 4017M 2024-02-05T06:31:39.169 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:39 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T06:31:39.169 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:39 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T06:31:39.169 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:39 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:31:39.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:39 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T06:31:39.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:39 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T06:31:39.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:39 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:31:39.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:39 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T06:31:39.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:39 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T06:31:39.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:39 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:31:40.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:40 smithi167 ceph-mon[32774]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:40.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:40 smithi167 ceph-mon[32774]: from='client.24235 v1:172.21.15.167:0/1622903059' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:31:40.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:40 smithi028 ceph-mon[37984]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:40.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:40 smithi028 ceph-mon[37984]: from='client.24235 v1:172.21.15.167:0/1622903059' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:31:40.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:40 smithi028 ceph-mon[29442]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:40.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:40 smithi028 ceph-mon[29442]: from='client.24235 v1:172.21.15.167:0/1622903059' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:31:41.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:41 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.167:0/4134134704' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc6bd83d-570c-48c0-964d-e09b4608bc32"}]: dispatch 2024-02-05T06:31:41.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:41 smithi028 ceph-mon[37984]: from='client.24241 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc6bd83d-570c-48c0-964d-e09b4608bc32"}]: dispatch 2024-02-05T06:31:41.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:41 smithi028 ceph-mon[37984]: from='client.24241 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc6bd83d-570c-48c0-964d-e09b4608bc32"}]': finished 2024-02-05T06:31:41.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:41 smithi028 ceph-mon[37984]: osdmap e28: 5 total, 4 up, 5 in 2024-02-05T06:31:41.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:41 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T06:31:41.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:41 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.167:0/4134134704' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc6bd83d-570c-48c0-964d-e09b4608bc32"}]: dispatch 2024-02-05T06:31:41.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:41 smithi028 ceph-mon[29442]: from='client.24241 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc6bd83d-570c-48c0-964d-e09b4608bc32"}]: dispatch 2024-02-05T06:31:41.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:41 smithi028 ceph-mon[29442]: from='client.24241 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc6bd83d-570c-48c0-964d-e09b4608bc32"}]': finished 2024-02-05T06:31:41.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:41 smithi028 ceph-mon[29442]: osdmap e28: 5 total, 4 up, 5 in 2024-02-05T06:31:41.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:41 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T06:31:41.476 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:41 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.167:0/4134134704' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc6bd83d-570c-48c0-964d-e09b4608bc32"}]: dispatch 2024-02-05T06:31:41.476 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:41 smithi167 ceph-mon[32774]: from='client.24241 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc6bd83d-570c-48c0-964d-e09b4608bc32"}]: dispatch 2024-02-05T06:31:41.476 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:41 smithi167 ceph-mon[32774]: from='client.24241 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc6bd83d-570c-48c0-964d-e09b4608bc32"}]': finished 2024-02-05T06:31:41.476 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:41 smithi167 ceph-mon[32774]: osdmap e28: 5 total, 4 up, 5 in 2024-02-05T06:31:41.476 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:41 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T06:31:42.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:42 smithi028 ceph-mon[29442]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:42.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:42 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.167:0/3066434608' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T06:31:42.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:42 smithi028 ceph-mon[37984]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:42.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:42 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.167:0/3066434608' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T06:31:42.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:42 smithi167 ceph-mon[32774]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:42.475 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:42 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.167:0/3066434608' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T06:31:44.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:44 smithi167 ceph-mon[32774]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:44.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:44 smithi028 ceph-mon[29442]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:44.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:44 smithi028 ceph-mon[37984]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:46.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:46 smithi028 ceph-mon[37984]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:46.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:46 smithi028 ceph-mon[29442]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:46.476 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:46 smithi167 ceph-mon[32774]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:48.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:48 smithi167 ceph-mon[32774]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:48.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:48 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-05T06:31:48.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:48 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:31:48.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:48 smithi028 ceph-mon[37984]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:48.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:48 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-05T06:31:48.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:48 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:31:48.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:48 smithi028 ceph-mon[29442]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:48.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:48 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-05T06:31:48.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:48 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:31:49.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:49 smithi167 ceph-mon[32774]: Deploying daemon osd.4 on smithi167 2024-02-05T06:31:49.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:49 smithi028 ceph-mon[29442]: Deploying daemon osd.4 on smithi167 2024-02-05T06:31:49.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:49 smithi028 ceph-mon[37984]: Deploying daemon osd.4 on smithi167 2024-02-05T06:31:50.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:50 smithi167 ceph-mon[32774]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:50.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:50 smithi028 ceph-mon[29442]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:50.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:50 smithi028 ceph-mon[37984]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:52.341 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:52 smithi167 ceph-mon[32774]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:52.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:52 smithi028 ceph-mon[29442]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:52.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:52 smithi028 ceph-mon[37984]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:53.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:53 smithi028 ceph-mon[29442]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:53.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:53 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:53.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:53 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:53.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:53 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:31:53.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:53 smithi028 ceph-mon[37984]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:53.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:53 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:53.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:53 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:53.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:53 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:31:53.976 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:53 smithi167 ceph-mon[32774]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:53.976 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:53 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:53.976 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:53 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:53.976 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:53 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:31:55.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:55 smithi028 ceph-mon[29442]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:55.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:55 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:55.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:55 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:55.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:55 smithi028 ceph-mon[37984]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:55.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:55 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:55.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:55 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:55.954 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:55 smithi167 ceph-mon[32774]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:55.954 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:55 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:55.954 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:55 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:56.307 INFO:teuthology.orchestra.run.smithi167.stdout:Created osd(s) 4 on host 'smithi167' 2024-02-05T06:31:56.875 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:56 smithi167 ceph-mon[32774]: from='osd.4 v1:172.21.15.167:6800/4052090595' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-05T06:31:56.875 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:56 smithi167 ceph-mon[32774]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-05T06:31:56.875 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:56 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:31:56.875 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:56 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:31:56.875 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:56 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:56.875 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:56 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:56.875 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:56 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:56.876 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:56 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:31:56.907 DEBUG:teuthology.orchestra.run.smithi167:osd.4> sudo journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@osd.4.service 2024-02-05T06:31:56.909 INFO:tasks.cephadm:Deploying osd.5 on smithi167 with /dev/vg_nvme/lv_3... 2024-02-05T06:31:56.909 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-05T06:31:56.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:56 smithi028 ceph-mon[29442]: from='osd.4 v1:172.21.15.167:6800/4052090595' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-05T06:31:56.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:56 smithi028 ceph-mon[29442]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-05T06:31:56.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:56 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:31:56.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:56 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:31:56.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:56 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:56.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:56 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:56.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:56 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:56.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:56 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:31:56.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:56 smithi028 ceph-mon[37984]: from='osd.4 v1:172.21.15.167:6800/4052090595' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-05T06:31:56.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:56 smithi028 ceph-mon[37984]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-05T06:31:56.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:56 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:31:56.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:56 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:31:56.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:56 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:56.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:56 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:56.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:56 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:31:56.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:56 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:31:57.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:57 smithi028 ceph-mon[37984]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:57.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:57 smithi028 ceph-mon[37984]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-05T06:31:57.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:57 smithi028 ceph-mon[37984]: from='osd.4 v1:172.21.15.167:6800/4052090595' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]: dispatch 2024-02-05T06:31:57.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:57 smithi028 ceph-mon[37984]: osdmap e29: 5 total, 4 up, 5 in 2024-02-05T06:31:57.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:57 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T06:31:57.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:57 smithi028 ceph-mon[37984]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]: dispatch 2024-02-05T06:31:57.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:57 smithi028 ceph-mon[29442]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:57.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:57 smithi028 ceph-mon[29442]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-05T06:31:57.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:57 smithi028 ceph-mon[29442]: from='osd.4 v1:172.21.15.167:6800/4052090595' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]: dispatch 2024-02-05T06:31:57.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:57 smithi028 ceph-mon[29442]: osdmap e29: 5 total, 4 up, 5 in 2024-02-05T06:31:57.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:57 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T06:31:57.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:57 smithi028 ceph-mon[29442]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]: dispatch 2024-02-05T06:31:57.976 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:57 smithi167 ceph-mon[32774]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:57.976 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:57 smithi167 ceph-mon[32774]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-05T06:31:57.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:57 smithi167 ceph-mon[32774]: from='osd.4 v1:172.21.15.167:6800/4052090595' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]: dispatch 2024-02-05T06:31:57.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:57 smithi167 ceph-mon[32774]: osdmap e29: 5 total, 4 up, 5 in 2024-02-05T06:31:57.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:57 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T06:31:57.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:57 smithi167 ceph-mon[32774]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]: dispatch 2024-02-05T06:31:57.977 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:31:57 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 2024-02-05T06:31:57.618+0000 7fd67bbda700 -1 osd.4 0 waiting for initial osdmap 2024-02-05T06:31:57.977 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:31:57 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 2024-02-05T06:31:57.623+0000 7fd6779d4700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T06:31:58.309 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:31:58.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:58 smithi028 ceph-mon[29442]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]': finished 2024-02-05T06:31:58.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:58 smithi028 ceph-mon[29442]: osdmap e30: 5 total, 4 up, 5 in 2024-02-05T06:31:58.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:58 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T06:31:58.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:58 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T06:31:58.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:58 smithi028 ceph-mon[37984]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]': finished 2024-02-05T06:31:58.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:58 smithi028 ceph-mon[37984]: osdmap e30: 5 total, 4 up, 5 in 2024-02-05T06:31:58.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:58 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T06:31:58.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:58 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T06:31:58.955 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:58 smithi167 ceph-mon[32774]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]': finished 2024-02-05T06:31:58.955 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:58 smithi167 ceph-mon[32774]: osdmap e30: 5 total, 4 up, 5 in 2024-02-05T06:31:58.955 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:58 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T06:31:58.955 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:58 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T06:31:59.913 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:59 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:31:59.913 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:59 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:31:59.913 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:59 smithi167 ceph-mon[32774]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:59.913 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:59 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T06:31:59.913 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:59 smithi167 ceph-mon[32774]: osd.4 v1:172.21.15.167:6800/4052090595 boot 2024-02-05T06:31:59.913 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:59 smithi167 ceph-mon[32774]: osdmap e31: 5 total, 5 up, 5 in 2024-02-05T06:31:59.913 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:59 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T06:31:59.913 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:31:59 smithi167 ceph-mon[32774]: osdmap e32: 5 total, 5 up, 5 in 2024-02-05T06:31:59.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:59 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:31:59.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:59 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:31:59.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:59 smithi028 ceph-mon[29442]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:59.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:59 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T06:31:59.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:59 smithi028 ceph-mon[29442]: osd.4 v1:172.21.15.167:6800/4052090595 boot 2024-02-05T06:31:59.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:59 smithi028 ceph-mon[29442]: osdmap e31: 5 total, 5 up, 5 in 2024-02-05T06:31:59.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:59 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T06:31:59.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:31:59 smithi028 ceph-mon[29442]: osdmap e32: 5 total, 5 up, 5 in 2024-02-05T06:31:59.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:59 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:31:59.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:59 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:31:59.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:59 smithi028 ceph-mon[37984]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-02-05T06:31:59.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:59 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T06:31:59.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:59 smithi028 ceph-mon[37984]: osd.4 v1:172.21.15.167:6800/4052090595 boot 2024-02-05T06:31:59.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:59 smithi028 ceph-mon[37984]: osdmap e31: 5 total, 5 up, 5 in 2024-02-05T06:31:59.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:59 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T06:31:59.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:31:59 smithi028 ceph-mon[37984]: osdmap e32: 5 total, 5 up, 5 in 2024-02-05T06:32:01.044 INFO:teuthology.orchestra.run.smithi167.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-05T06:32:01.045 INFO:teuthology.orchestra.run.smithi167.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-05T06:32:01.045 INFO:teuthology.orchestra.run.smithi167.stderr: stderr: 10+0 records in 2024-02-05T06:32:01.045 INFO:teuthology.orchestra.run.smithi167.stderr:10+0 records out 2024-02-05T06:32:01.045 INFO:teuthology.orchestra.run.smithi167.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113878 s, 921 MB/s 2024-02-05T06:32:01.045 INFO:teuthology.orchestra.run.smithi167.stderr:--> Zapping successful for: 2024-02-05T06:32:01.069 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:00 smithi167 ceph-mon[32774]: Detected new or changed devices on smithi167 2024-02-05T06:32:01.069 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:00 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:01.069 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:00 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:01.069 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:00 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:32:01.069 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:00 smithi167 ceph-mon[32774]: Adjusting osd_memory_target on smithi167 to 17088M 2024-02-05T06:32:01.069 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:00 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:01.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:00 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:32:01.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:00 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:32:01.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:00 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:01.070 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:00 smithi167 ceph-mon[32774]: osdmap e33: 5 total, 5 up, 5 in 2024-02-05T06:32:01.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:00 smithi028 ceph-mon[29442]: Detected new or changed devices on smithi167 2024-02-05T06:32:01.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:00 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:01.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:00 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:01.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:00 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:32:01.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:00 smithi028 ceph-mon[29442]: Adjusting osd_memory_target on smithi167 to 17088M 2024-02-05T06:32:01.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:00 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:01.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:00 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:32:01.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:00 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:32:01.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:00 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:01.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:00 smithi028 ceph-mon[29442]: osdmap e33: 5 total, 5 up, 5 in 2024-02-05T06:32:01.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:00 smithi028 ceph-mon[37984]: Detected new or changed devices on smithi167 2024-02-05T06:32:01.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:00 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:01.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:00 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:01.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:00 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:32:01.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:00 smithi028 ceph-mon[37984]: Adjusting osd_memory_target on smithi167 to 17088M 2024-02-05T06:32:01.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:00 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:01.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:00 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:32:01.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:00 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:32:01.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:00 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:01.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:00 smithi028 ceph-mon[37984]: osdmap e33: 5 total, 5 up, 5 in 2024-02-05T06:32:01.698 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph orch daemon add osd smithi167:vg_nvme/lv_3 2024-02-05T06:32:01.935 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:32:01.976 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:01 smithi167 ceph-mon[32774]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-05T06:32:01.976 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:01 smithi167 ceph-mon[32774]: osdmap e34: 5 total, 5 up, 5 in 2024-02-05T06:32:02.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:01 smithi028 ceph-mon[29442]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-05T06:32:02.191 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:01 smithi028 ceph-mon[29442]: osdmap e34: 5 total, 5 up, 5 in 2024-02-05T06:32:02.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:01 smithi028 ceph-mon[37984]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-05T06:32:02.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:01 smithi028 ceph-mon[37984]: osdmap e34: 5 total, 5 up, 5 in 2024-02-05T06:32:03.083 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:02 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:32:03.083 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:02 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:32:03.083 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:02 smithi167 ceph-mon[32774]: osdmap e35: 5 total, 5 up, 5 in 2024-02-05T06:32:03.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:02 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:32:03.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:02 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:32:03.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:02 smithi028 ceph-mon[29442]: osdmap e35: 5 total, 5 up, 5 in 2024-02-05T06:32:03.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:02 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:32:03.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:02 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:32:03.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:02 smithi028 ceph-mon[37984]: osdmap e35: 5 total, 5 up, 5 in 2024-02-05T06:32:04.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:03 smithi028 ceph-mon[29442]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-05T06:32:04.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:03 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T06:32:04.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:03 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T06:32:04.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:03 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:32:04.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:03 smithi028 ceph-mon[37984]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-05T06:32:04.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:03 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T06:32:04.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:03 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T06:32:04.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:03 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:32:04.208 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:03 smithi167 ceph-mon[32774]: pgmap v118: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-05T06:32:04.208 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:03 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T06:32:04.208 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:03 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T06:32:04.209 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:03 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:32:05.099 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:04 smithi167 ceph-mon[32774]: from='client.24262 v1:172.21.15.167:0/1356216153' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:32:05.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:04 smithi028 ceph-mon[29442]: from='client.24262 v1:172.21.15.167:0/1356216153' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:32:05.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:04 smithi028 ceph-mon[37984]: from='client.24262 v1:172.21.15.167:0/1356216153' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:32:06.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:05 smithi028 ceph-mon[29442]: pgmap v120: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-05T06:32:06.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:05 smithi028 ceph-mon[37984]: pgmap v120: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-05T06:32:06.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:05 smithi167 ceph-mon[32774]: pgmap v120: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-05T06:32:07.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:06 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.167:0/1613011345' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "895bf465-2531-4d9f-bc23-1e75fd969f07"}]: dispatch 2024-02-05T06:32:07.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:06 smithi028 ceph-mon[37984]: from='client.24268 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "895bf465-2531-4d9f-bc23-1e75fd969f07"}]: dispatch 2024-02-05T06:32:07.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:06 smithi028 ceph-mon[37984]: from='client.24268 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "895bf465-2531-4d9f-bc23-1e75fd969f07"}]': finished 2024-02-05T06:32:07.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:06 smithi028 ceph-mon[37984]: osdmap e36: 6 total, 5 up, 6 in 2024-02-05T06:32:07.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:06 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T06:32:07.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:06 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.167:0/1613011345' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "895bf465-2531-4d9f-bc23-1e75fd969f07"}]: dispatch 2024-02-05T06:32:07.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:06 smithi028 ceph-mon[29442]: from='client.24268 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "895bf465-2531-4d9f-bc23-1e75fd969f07"}]: dispatch 2024-02-05T06:32:07.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:06 smithi028 ceph-mon[29442]: from='client.24268 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "895bf465-2531-4d9f-bc23-1e75fd969f07"}]': finished 2024-02-05T06:32:07.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:06 smithi028 ceph-mon[29442]: osdmap e36: 6 total, 5 up, 6 in 2024-02-05T06:32:07.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:06 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T06:32:07.227 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:06 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.167:0/1613011345' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "895bf465-2531-4d9f-bc23-1e75fd969f07"}]: dispatch 2024-02-05T06:32:07.227 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:06 smithi167 ceph-mon[32774]: from='client.24268 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "895bf465-2531-4d9f-bc23-1e75fd969f07"}]: dispatch 2024-02-05T06:32:07.227 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:06 smithi167 ceph-mon[32774]: from='client.24268 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "895bf465-2531-4d9f-bc23-1e75fd969f07"}]': finished 2024-02-05T06:32:07.227 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:06 smithi167 ceph-mon[32774]: osdmap e36: 6 total, 5 up, 6 in 2024-02-05T06:32:07.227 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:06 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T06:32:08.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:07 smithi028 ceph-mon[29442]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-05T06:32:08.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:07 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.167:0/4253365539' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T06:32:08.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:07 smithi028 ceph-mon[37984]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-05T06:32:08.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:07 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.167:0/4253365539' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T06:32:08.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:07 smithi167 ceph-mon[32774]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-05T06:32:08.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:07 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.167:0/4253365539' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T06:32:10.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:09 smithi028 ceph-mon[37984]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-02-05T06:32:10.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:09 smithi028 ceph-mon[29442]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-02-05T06:32:10.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:09 smithi167 ceph-mon[32774]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-02-05T06:32:12.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:11 smithi028 ceph-mon[37984]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-05T06:32:12.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:11 smithi028 ceph-mon[29442]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-05T06:32:12.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:11 smithi167 ceph-mon[32774]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-05T06:32:13.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:12 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-05T06:32:13.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:12 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:32:13.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:12 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-05T06:32:13.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:12 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:32:13.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:12 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-05T06:32:13.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:12 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:32:14.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:13 smithi028 ceph-mon[37984]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-02-05T06:32:14.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:13 smithi028 ceph-mon[37984]: Deploying daemon osd.5 on smithi167 2024-02-05T06:32:14.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:13 smithi028 ceph-mon[29442]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-02-05T06:32:14.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:13 smithi028 ceph-mon[29442]: Deploying daemon osd.5 on smithi167 2024-02-05T06:32:14.192 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:13 smithi167 ceph-mon[32774]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-02-05T06:32:14.193 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:13 smithi167 ceph-mon[32774]: Deploying daemon osd.5 on smithi167 2024-02-05T06:32:16.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:15 smithi028 ceph-mon[29442]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-05T06:32:16.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:15 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:32:16.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:15 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:32:16.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:15 smithi028 ceph-mon[29442]: osdmap e37: 6 total, 5 up, 6 in 2024-02-05T06:32:16.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:15 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T06:32:16.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:15 smithi028 ceph-mon[37984]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-05T06:32:16.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:15 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:32:16.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:15 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:32:16.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:15 smithi028 ceph-mon[37984]: osdmap e37: 6 total, 5 up, 6 in 2024-02-05T06:32:16.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:15 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T06:32:16.196 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:15 smithi167 ceph-mon[32774]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-05T06:32:16.197 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:15 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:32:16.197 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:15 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:32:16.197 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:15 smithi167 ceph-mon[32774]: osdmap e37: 6 total, 5 up, 6 in 2024-02-05T06:32:16.197 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:15 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T06:32:18.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:17 smithi028 ceph-mon[29442]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-05T06:32:18.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:17 smithi028 ceph-mon[37984]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-05T06:32:18.226 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:17 smithi167 ceph-mon[32774]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-05T06:32:19.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:19 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:19.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:19 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:19.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:19 smithi028 ceph-mon[29442]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-05T06:32:19.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:19 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:32:19.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:19 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:19.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:19 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:19.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:19 smithi028 ceph-mon[37984]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-05T06:32:19.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:19 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:32:19.477 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:19 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:19.477 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:19 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:19.477 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:19 smithi167 ceph-mon[32774]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-05T06:32:19.477 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:19 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:32:21.929 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:21 smithi167 ceph-mon[32774]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-05T06:32:21.929 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:21 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:21.929 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:21 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:21.929 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:21 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:32:21.929 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:21 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:32:21.929 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:21 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:21.929 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:21 smithi167 ceph-mon[32774]: from='osd.5 v1:172.21.15.167:6804/1631080500' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-05T06:32:21.929 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:21 smithi167 ceph-mon[32774]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-05T06:32:21.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:21 smithi028 ceph-mon[29442]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-05T06:32:21.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:21 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:21.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:21 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:21.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:21 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:32:21.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:21 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:32:21.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:21 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:21.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:21 smithi028 ceph-mon[29442]: from='osd.5 v1:172.21.15.167:6804/1631080500' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-05T06:32:21.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:21 smithi028 ceph-mon[29442]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-05T06:32:21.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:21 smithi028 ceph-mon[37984]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-05T06:32:21.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:21 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:21.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:21 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:21.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:21 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:32:21.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:21 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:32:21.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:21 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:21.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:21 smithi028 ceph-mon[37984]: from='osd.5 v1:172.21.15.167:6804/1631080500' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-05T06:32:21.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:21 smithi028 ceph-mon[37984]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-05T06:32:21.970 INFO:teuthology.orchestra.run.smithi167.stdout:Created osd(s) 5 on host 'smithi167' 2024-02-05T06:32:22.583 DEBUG:teuthology.orchestra.run.smithi167:osd.5> sudo journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@osd.5.service 2024-02-05T06:32:22.586 INFO:tasks.cephadm:Deploying osd.6 on smithi167 with /dev/vg_nvme/lv_2... 2024-02-05T06:32:22.586 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-05T06:32:23.128 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:22 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:23.128 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:22 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:23.128 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:22 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:32:23.129 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:22 smithi167 ceph-mon[32774]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-05T06:32:23.129 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:22 smithi167 ceph-mon[32774]: osdmap e38: 6 total, 5 up, 6 in 2024-02-05T06:32:23.129 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:22 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T06:32:23.129 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:22 smithi167 ceph-mon[32774]: from='osd.5 v1:172.21.15.167:6804/1631080500' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]: dispatch 2024-02-05T06:32:23.129 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:22 smithi167 ceph-mon[32774]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]: dispatch 2024-02-05T06:32:23.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:22 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:23.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:22 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:23.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:22 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:32:23.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:22 smithi028 ceph-mon[29442]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-05T06:32:23.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:22 smithi028 ceph-mon[29442]: osdmap e38: 6 total, 5 up, 6 in 2024-02-05T06:32:23.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:22 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T06:32:23.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:22 smithi028 ceph-mon[29442]: from='osd.5 v1:172.21.15.167:6804/1631080500' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]: dispatch 2024-02-05T06:32:23.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:22 smithi028 ceph-mon[29442]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]: dispatch 2024-02-05T06:32:23.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:22 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:23.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:22 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:23.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:22 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:32:23.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:22 smithi028 ceph-mon[37984]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-05T06:32:23.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:22 smithi028 ceph-mon[37984]: osdmap e38: 6 total, 5 up, 6 in 2024-02-05T06:32:23.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:22 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T06:32:23.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:22 smithi028 ceph-mon[37984]: from='osd.5 v1:172.21.15.167:6804/1631080500' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]: dispatch 2024-02-05T06:32:23.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:22 smithi028 ceph-mon[37984]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]: dispatch 2024-02-05T06:32:23.477 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:32:23 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:32:23.158+0000 7f56c195a700 -1 osd.5 0 waiting for initial osdmap 2024-02-05T06:32:23.477 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:32:23 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:32:23.162+0000 7f56bd754700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T06:32:23.854 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:32:24.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:23 smithi167 ceph-mon[32774]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-05T06:32:24.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:23 smithi167 ceph-mon[32774]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]': finished 2024-02-05T06:32:24.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:23 smithi167 ceph-mon[32774]: osdmap e39: 6 total, 5 up, 6 in 2024-02-05T06:32:24.071 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:23 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T06:32:24.072 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:23 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T06:32:24.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:23 smithi028 ceph-mon[29442]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-05T06:32:24.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:23 smithi028 ceph-mon[29442]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]': finished 2024-02-05T06:32:24.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:23 smithi028 ceph-mon[29442]: osdmap e39: 6 total, 5 up, 6 in 2024-02-05T06:32:24.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:23 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T06:32:24.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:23 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T06:32:24.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:23 smithi028 ceph-mon[37984]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-05T06:32:24.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:23 smithi028 ceph-mon[37984]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]': finished 2024-02-05T06:32:24.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:23 smithi028 ceph-mon[37984]: osdmap e39: 6 total, 5 up, 6 in 2024-02-05T06:32:24.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:23 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T06:32:24.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:23 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T06:32:25.410 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:25 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:32:25.410 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:25 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:32:25.410 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:25 smithi167 ceph-mon[32774]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-05T06:32:25.410 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:25 smithi167 ceph-mon[32774]: osd.5 v1:172.21.15.167:6804/1631080500 boot 2024-02-05T06:32:25.410 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:25 smithi167 ceph-mon[32774]: osdmap e40: 6 total, 6 up, 6 in 2024-02-05T06:32:25.410 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:25 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T06:32:25.410 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:25 smithi167 ceph-mon[32774]: osdmap e41: 6 total, 6 up, 6 in 2024-02-05T06:32:25.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:25 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:32:25.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:25 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:32:25.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:25 smithi028 ceph-mon[37984]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-05T06:32:25.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:25 smithi028 ceph-mon[37984]: osd.5 v1:172.21.15.167:6804/1631080500 boot 2024-02-05T06:32:25.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:25 smithi028 ceph-mon[37984]: osdmap e40: 6 total, 6 up, 6 in 2024-02-05T06:32:25.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:25 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T06:32:25.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:25 smithi028 ceph-mon[37984]: osdmap e41: 6 total, 6 up, 6 in 2024-02-05T06:32:25.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:25 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:32:25.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:25 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:32:25.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:25 smithi028 ceph-mon[29442]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-05T06:32:25.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:25 smithi028 ceph-mon[29442]: osd.5 v1:172.21.15.167:6804/1631080500 boot 2024-02-05T06:32:25.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:25 smithi028 ceph-mon[29442]: osdmap e40: 6 total, 6 up, 6 in 2024-02-05T06:32:25.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:25 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T06:32:25.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:25 smithi028 ceph-mon[29442]: osdmap e41: 6 total, 6 up, 6 in 2024-02-05T06:32:26.463 INFO:teuthology.orchestra.run.smithi167.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-05T06:32:26.463 INFO:teuthology.orchestra.run.smithi167.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-05T06:32:26.463 INFO:teuthology.orchestra.run.smithi167.stderr: stderr: 10+0 records in 2024-02-05T06:32:26.463 INFO:teuthology.orchestra.run.smithi167.stderr:10+0 records out 2024-02-05T06:32:26.463 INFO:teuthology.orchestra.run.smithi167.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0203145 s, 516 MB/s 2024-02-05T06:32:26.463 INFO:teuthology.orchestra.run.smithi167.stderr:--> Zapping successful for: 2024-02-05T06:32:26.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:26 smithi167 ceph-mon[32774]: osdmap e42: 6 total, 6 up, 6 in 2024-02-05T06:32:26.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:26 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:26.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:26 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:26.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:26 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:32:26.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:26 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:32:26.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:26 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:26.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:26 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:32:26.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:26 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:32:26.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:26 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:26.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:26 smithi028 ceph-mon[29442]: osdmap e42: 6 total, 6 up, 6 in 2024-02-05T06:32:26.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:26 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:26.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:26 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:26.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:26 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:32:26.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:26 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:32:26.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:26 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:26.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:26 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:32:26.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:26 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:32:26.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:26 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:26.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:26 smithi028 ceph-mon[37984]: osdmap e42: 6 total, 6 up, 6 in 2024-02-05T06:32:26.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:26 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:26.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:26 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:26.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:26 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:32:26.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:26 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:32:26.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:26 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:26.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:26 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:32:26.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:26 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:32:26.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:26 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:27.138 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph orch daemon add osd smithi167:vg_nvme/lv_2 2024-02-05T06:32:27.372 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:32:27.743 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:27 smithi167 ceph-mon[32774]: pgmap v138: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-05T06:32:27.744 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:27 smithi167 ceph-mon[32774]: Detected new or changed devices on smithi167 2024-02-05T06:32:27.744 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:27 smithi167 ceph-mon[32774]: Adjusting osd_memory_target on smithi167 to 8544M 2024-02-05T06:32:27.744 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:27 smithi167 ceph-mon[32774]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-05T06:32:27.744 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:27 smithi167 ceph-mon[32774]: osdmap e43: 6 total, 6 up, 6 in 2024-02-05T06:32:27.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:27 smithi028 ceph-mon[29442]: pgmap v138: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-05T06:32:27.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:27 smithi028 ceph-mon[29442]: Detected new or changed devices on smithi167 2024-02-05T06:32:27.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:27 smithi028 ceph-mon[29442]: Adjusting osd_memory_target on smithi167 to 8544M 2024-02-05T06:32:27.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:27 smithi028 ceph-mon[29442]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-05T06:32:27.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:27 smithi028 ceph-mon[29442]: osdmap e43: 6 total, 6 up, 6 in 2024-02-05T06:32:27.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:27 smithi028 ceph-mon[37984]: pgmap v138: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-05T06:32:27.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:27 smithi028 ceph-mon[37984]: Detected new or changed devices on smithi167 2024-02-05T06:32:27.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:27 smithi028 ceph-mon[37984]: Adjusting osd_memory_target on smithi167 to 8544M 2024-02-05T06:32:27.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:27 smithi028 ceph-mon[37984]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-02-05T06:32:27.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:27 smithi028 ceph-mon[37984]: osdmap e43: 6 total, 6 up, 6 in 2024-02-05T06:32:28.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:28 smithi028 ceph-mon[29442]: osdmap e44: 6 total, 6 up, 6 in 2024-02-05T06:32:28.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:28 smithi028 ceph-mon[37984]: osdmap e44: 6 total, 6 up, 6 in 2024-02-05T06:32:28.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:28 smithi167 ceph-mon[32774]: osdmap e44: 6 total, 6 up, 6 in 2024-02-05T06:32:29.925 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:29 smithi167 ceph-mon[32774]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-05T06:32:29.926 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:29 smithi167 ceph-mon[32774]: 1.0 scrub starts 2024-02-05T06:32:29.926 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:29 smithi167 ceph-mon[32774]: 1.0 scrub ok 2024-02-05T06:32:29.926 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:29 smithi167 ceph-mon[32774]: from='client.24289 v1:172.21.15.167:0/248134804' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:32:29.926 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:29 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T06:32:29.926 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:29 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T06:32:29.926 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:29 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:32:29.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:29 smithi028 ceph-mon[29442]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-05T06:32:29.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:29 smithi028 ceph-mon[29442]: 1.0 scrub starts 2024-02-05T06:32:29.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:29 smithi028 ceph-mon[29442]: 1.0 scrub ok 2024-02-05T06:32:29.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:29 smithi028 ceph-mon[29442]: from='client.24289 v1:172.21.15.167:0/248134804' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:32:29.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:29 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T06:32:29.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:29 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T06:32:29.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:29 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:32:29.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:29 smithi028 ceph-mon[37984]: pgmap v141: 1 pgs: 1 remapped+peering; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-05T06:32:29.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:29 smithi028 ceph-mon[37984]: 1.0 scrub starts 2024-02-05T06:32:29.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:29 smithi028 ceph-mon[37984]: 1.0 scrub ok 2024-02-05T06:32:29.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:29 smithi028 ceph-mon[37984]: from='client.24289 v1:172.21.15.167:0/248134804' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:32:29.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:29 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T06:32:29.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:29 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T06:32:29.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:29 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:32:30.877 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:30 smithi167 ceph-mon[32774]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-05T06:32:30.877 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:30 smithi167 ceph-mon[32774]: Cluster is now healthy 2024-02-05T06:32:30.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:30 smithi028 ceph-mon[37984]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-05T06:32:30.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:30 smithi028 ceph-mon[37984]: Cluster is now healthy 2024-02-05T06:32:30.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:30 smithi028 ceph-mon[29442]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-05T06:32:30.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:30 smithi028 ceph-mon[29442]: Cluster is now healthy 2024-02-05T06:32:31.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:31 smithi028 ceph-mon[29442]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-05T06:32:31.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:31 smithi028 ceph-mon[37984]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-05T06:32:31.976 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:31 smithi167 ceph-mon[32774]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-05T06:32:32.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:32 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.167:0/3506871121' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce0a9300-87a4-4e37-b702-d667f168c0e0"}]: dispatch 2024-02-05T06:32:32.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:32 smithi028 ceph-mon[29442]: from='client.24295 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce0a9300-87a4-4e37-b702-d667f168c0e0"}]: dispatch 2024-02-05T06:32:32.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:32 smithi028 ceph-mon[29442]: from='client.24295 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ce0a9300-87a4-4e37-b702-d667f168c0e0"}]': finished 2024-02-05T06:32:32.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:32 smithi028 ceph-mon[29442]: osdmap e45: 7 total, 6 up, 7 in 2024-02-05T06:32:32.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:32 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T06:32:32.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:32 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.167:0/1176009711' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T06:32:32.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:32 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.167:0/3506871121' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce0a9300-87a4-4e37-b702-d667f168c0e0"}]: dispatch 2024-02-05T06:32:32.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:32 smithi028 ceph-mon[37984]: from='client.24295 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce0a9300-87a4-4e37-b702-d667f168c0e0"}]: dispatch 2024-02-05T06:32:32.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:32 smithi028 ceph-mon[37984]: from='client.24295 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ce0a9300-87a4-4e37-b702-d667f168c0e0"}]': finished 2024-02-05T06:32:32.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:32 smithi028 ceph-mon[37984]: osdmap e45: 7 total, 6 up, 7 in 2024-02-05T06:32:32.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:32 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T06:32:32.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:32 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.167:0/1176009711' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T06:32:32.976 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:32 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.167:0/3506871121' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce0a9300-87a4-4e37-b702-d667f168c0e0"}]: dispatch 2024-02-05T06:32:32.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:32 smithi167 ceph-mon[32774]: from='client.24295 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ce0a9300-87a4-4e37-b702-d667f168c0e0"}]: dispatch 2024-02-05T06:32:32.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:32 smithi167 ceph-mon[32774]: from='client.24295 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ce0a9300-87a4-4e37-b702-d667f168c0e0"}]': finished 2024-02-05T06:32:32.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:32 smithi167 ceph-mon[32774]: osdmap e45: 7 total, 6 up, 7 in 2024-02-05T06:32:32.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:32 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T06:32:32.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:32 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.167:0/1176009711' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T06:32:33.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:33 smithi028 ceph-mon[29442]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-05T06:32:33.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:33 smithi028 ceph-mon[37984]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-05T06:32:33.976 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:33 smithi167 ceph-mon[32774]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-05T06:32:35.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:35 smithi028 ceph-mon[29442]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-02-05T06:32:35.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:35 smithi028 ceph-mon[37984]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-02-05T06:32:35.967 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:35 smithi167 ceph-mon[32774]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-02-05T06:32:37.861 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:37 smithi167 ceph-mon[32774]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-02-05T06:32:37.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:37 smithi028 ceph-mon[29442]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-02-05T06:32:37.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:37 smithi028 ceph-mon[37984]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-02-05T06:32:38.961 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:38 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-05T06:32:38.961 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:38 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:32:39.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:38 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-05T06:32:39.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:38 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:32:39.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:38 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-05T06:32:39.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:38 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:32:39.827 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:39 smithi167 ceph-mon[32774]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-05T06:32:39.828 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:39 smithi167 ceph-mon[32774]: Deploying daemon osd.6 on smithi167 2024-02-05T06:32:40.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:39 smithi028 ceph-mon[37984]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-05T06:32:40.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:39 smithi028 ceph-mon[37984]: Deploying daemon osd.6 on smithi167 2024-02-05T06:32:40.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:39 smithi028 ceph-mon[29442]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-05T06:32:40.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:39 smithi028 ceph-mon[29442]: Deploying daemon osd.6 on smithi167 2024-02-05T06:32:40.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:40 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:32:40.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:40 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:32:40.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:40 smithi167 ceph-mon[32774]: osdmap e46: 7 total, 6 up, 7 in 2024-02-05T06:32:40.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:40 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T06:32:41.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:40 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:32:41.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:40 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:32:41.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:40 smithi028 ceph-mon[37984]: osdmap e46: 7 total, 6 up, 7 in 2024-02-05T06:32:41.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:40 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T06:32:41.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:40 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:32:41.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:40 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:32:41.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:40 smithi028 ceph-mon[29442]: osdmap e46: 7 total, 6 up, 7 in 2024-02-05T06:32:41.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:40 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T06:32:41.953 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:41 smithi167 ceph-mon[32774]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-05T06:32:42.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:41 smithi028 ceph-mon[37984]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-05T06:32:42.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:41 smithi028 ceph-mon[29442]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-05T06:32:43.976 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:43 smithi167 ceph-mon[32774]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-05T06:32:44.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:43 smithi028 ceph-mon[29442]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-05T06:32:44.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:43 smithi028 ceph-mon[37984]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-05T06:32:45.170 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:44 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:45.170 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:44 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:45.170 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:44 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:32:45.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:44 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:45.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:44 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:45.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:44 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:32:45.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:44 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:45.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:44 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:45.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:44 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:32:46.076 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:45 smithi167 ceph-mon[32774]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-05T06:32:46.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:45 smithi028 ceph-mon[37984]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-05T06:32:46.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:45 smithi028 ceph-mon[29442]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-05T06:32:47.681 INFO:teuthology.orchestra.run.smithi167.stdout:Created osd(s) 6 on host 'smithi167' 2024-02-05T06:32:47.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:47 smithi167 ceph-mon[32774]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-05T06:32:47.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:47 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:47.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:47 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:47.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:47 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:32:47.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:47 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:32:47.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:47 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:47.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:47 smithi167 ceph-mon[32774]: from='osd.6 v1:172.21.15.167:6808/1644050617' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-05T06:32:47.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:47 smithi167 ceph-mon[32774]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-05T06:32:47.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:47 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:47.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:47 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:47.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:47 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:32:48.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:47 smithi028 ceph-mon[29442]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-05T06:32:48.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:47 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:48.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:47 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:48.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:47 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:32:48.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:47 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:32:48.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:47 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:48.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:47 smithi028 ceph-mon[29442]: from='osd.6 v1:172.21.15.167:6808/1644050617' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-05T06:32:48.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:47 smithi028 ceph-mon[29442]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-05T06:32:48.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:47 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:48.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:47 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:48.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:47 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:32:48.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:47 smithi028 ceph-mon[37984]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-05T06:32:48.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:47 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:48.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:47 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:48.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:47 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:32:48.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:47 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:32:48.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:47 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:48.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:47 smithi028 ceph-mon[37984]: from='osd.6 v1:172.21.15.167:6808/1644050617' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-05T06:32:48.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:47 smithi028 ceph-mon[37984]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-05T06:32:48.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:47 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:48.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:47 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:48.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:47 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:32:48.295 DEBUG:teuthology.orchestra.run.smithi167:osd.6> sudo journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@osd.6.service 2024-02-05T06:32:48.298 INFO:tasks.cephadm:Deploying osd.7 on smithi167 with /dev/vg_nvme/lv_1... 2024-02-05T06:32:48.298 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-05T06:32:48.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:48 smithi167 ceph-mon[32774]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-05T06:32:48.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:48 smithi167 ceph-mon[32774]: from='osd.6 v1:172.21.15.167:6808/1644050617' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]: dispatch 2024-02-05T06:32:48.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:48 smithi167 ceph-mon[32774]: osdmap e47: 7 total, 6 up, 7 in 2024-02-05T06:32:48.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:48 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T06:32:48.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:48 smithi167 ceph-mon[32774]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]: dispatch 2024-02-05T06:32:49.091 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:32:49.094 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:48 smithi028 ceph-mon[29442]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-05T06:32:49.094 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:48 smithi028 ceph-mon[29442]: from='osd.6 v1:172.21.15.167:6808/1644050617' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]: dispatch 2024-02-05T06:32:49.094 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:48 smithi028 ceph-mon[29442]: osdmap e47: 7 total, 6 up, 7 in 2024-02-05T06:32:49.094 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:48 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T06:32:49.094 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:48 smithi028 ceph-mon[29442]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]: dispatch 2024-02-05T06:32:49.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:48 smithi028 ceph-mon[37984]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-05T06:32:49.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:48 smithi028 ceph-mon[37984]: from='osd.6 v1:172.21.15.167:6808/1644050617' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]: dispatch 2024-02-05T06:32:49.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:48 smithi028 ceph-mon[37984]: osdmap e47: 7 total, 6 up, 7 in 2024-02-05T06:32:49.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:48 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T06:32:49.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:48 smithi028 ceph-mon[37984]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]: dispatch 2024-02-05T06:32:49.643 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:32:49 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:32:49.297+0000 7fd863624700 -1 osd.6 0 waiting for initial osdmap 2024-02-05T06:32:49.643 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:32:49 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:32:49.304+0000 7fd85b416700 -1 osd.6 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T06:32:49.949 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:49 smithi167 ceph-mon[32774]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-05T06:32:49.950 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:49 smithi167 ceph-mon[32774]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]': finished 2024-02-05T06:32:49.950 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:49 smithi167 ceph-mon[32774]: osdmap e48: 7 total, 6 up, 7 in 2024-02-05T06:32:49.950 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:49 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T06:32:49.950 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:49 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T06:32:49.950 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:49 smithi167 ceph-mon[32774]: from='osd.6 ' entity='osd.6' 2024-02-05T06:32:49.950 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:49 smithi167 ceph-mon[32774]: osd.6 v1:172.21.15.167:6808/1644050617 boot 2024-02-05T06:32:49.950 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:49 smithi167 ceph-mon[32774]: osdmap e49: 7 total, 7 up, 7 in 2024-02-05T06:32:49.950 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:49 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T06:32:50.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:49 smithi028 ceph-mon[29442]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-05T06:32:50.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:49 smithi028 ceph-mon[29442]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]': finished 2024-02-05T06:32:50.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:49 smithi028 ceph-mon[29442]: osdmap e48: 7 total, 6 up, 7 in 2024-02-05T06:32:50.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:49 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T06:32:50.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:49 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T06:32:50.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:49 smithi028 ceph-mon[29442]: from='osd.6 ' entity='osd.6' 2024-02-05T06:32:50.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:49 smithi028 ceph-mon[29442]: osd.6 v1:172.21.15.167:6808/1644050617 boot 2024-02-05T06:32:50.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:49 smithi028 ceph-mon[29442]: osdmap e49: 7 total, 7 up, 7 in 2024-02-05T06:32:50.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:49 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T06:32:50.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:49 smithi028 ceph-mon[37984]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-02-05T06:32:50.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:49 smithi028 ceph-mon[37984]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]': finished 2024-02-05T06:32:50.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:49 smithi028 ceph-mon[37984]: osdmap e48: 7 total, 6 up, 7 in 2024-02-05T06:32:50.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:49 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T06:32:50.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:49 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T06:32:50.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:49 smithi028 ceph-mon[37984]: from='osd.6 ' entity='osd.6' 2024-02-05T06:32:50.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:49 smithi028 ceph-mon[37984]: osd.6 v1:172.21.15.167:6808/1644050617 boot 2024-02-05T06:32:50.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:49 smithi028 ceph-mon[37984]: osdmap e49: 7 total, 7 up, 7 in 2024-02-05T06:32:50.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:49 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T06:32:51.152 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:50 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:32:51.152 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:50 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:32:51.152 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:50 smithi167 ceph-mon[32774]: osdmap e50: 7 total, 7 up, 7 in 2024-02-05T06:32:51.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:50 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:32:51.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:50 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:32:51.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:50 smithi028 ceph-mon[37984]: osdmap e50: 7 total, 7 up, 7 in 2024-02-05T06:32:51.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:50 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:32:51.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:50 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:32:51.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:50 smithi028 ceph-mon[29442]: osdmap e50: 7 total, 7 up, 7 in 2024-02-05T06:32:51.939 INFO:teuthology.orchestra.run.smithi167.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-05T06:32:51.939 INFO:teuthology.orchestra.run.smithi167.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-05T06:32:51.939 INFO:teuthology.orchestra.run.smithi167.stderr: stderr: 10+0 records in 2024-02-05T06:32:51.939 INFO:teuthology.orchestra.run.smithi167.stderr:10+0 records out 2024-02-05T06:32:51.939 INFO:teuthology.orchestra.run.smithi167.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0137719 s, 761 MB/s 2024-02-05T06:32:51.939 INFO:teuthology.orchestra.run.smithi167.stderr:--> Zapping successful for: 2024-02-05T06:32:51.964 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:51 smithi167 ceph-mon[32774]: pgmap v157: 1 pgs: 1 remapped+peering; 577 KiB data, 341 MiB used, 625 GiB / 626 GiB avail 2024-02-05T06:32:51.964 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:51 smithi167 ceph-mon[32774]: osdmap e51: 7 total, 7 up, 7 in 2024-02-05T06:32:52.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:51 smithi028 ceph-mon[37984]: pgmap v157: 1 pgs: 1 remapped+peering; 577 KiB data, 341 MiB used, 625 GiB / 626 GiB avail 2024-02-05T06:32:52.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:51 smithi028 ceph-mon[37984]: osdmap e51: 7 total, 7 up, 7 in 2024-02-05T06:32:52.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:51 smithi028 ceph-mon[29442]: pgmap v157: 1 pgs: 1 remapped+peering; 577 KiB data, 341 MiB used, 625 GiB / 626 GiB avail 2024-02-05T06:32:52.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:51 smithi028 ceph-mon[29442]: osdmap e51: 7 total, 7 up, 7 in 2024-02-05T06:32:53.037 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph orch daemon add osd smithi167:vg_nvme/lv_1 2024-02-05T06:32:53.271 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:32:53.722 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:53 smithi167 ceph-mon[32774]: pgmap v160: 1 pgs: 1 remapped+peering; 577 KiB data, 341 MiB used, 625 GiB / 626 GiB avail 2024-02-05T06:32:53.723 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:53 smithi167 ceph-mon[32774]: osdmap e52: 7 total, 7 up, 7 in 2024-02-05T06:32:53.723 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:53 smithi167 ceph-mon[32774]: Detected new or changed devices on smithi167 2024-02-05T06:32:53.723 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:53 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:53.723 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:53 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:53.723 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:53 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:32:53.723 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:53 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:32:53.723 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:53 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:32:53.723 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:53 smithi167 ceph-mon[32774]: Adjusting osd_memory_target on smithi167 to 5696M 2024-02-05T06:32:53.723 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:53 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:53.723 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:53 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:32:53.723 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:53 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:32:53.723 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:53 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:53.723 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:53 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:32:53.724 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:53 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:32:53.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:53 smithi028 ceph-mon[37984]: pgmap v160: 1 pgs: 1 remapped+peering; 577 KiB data, 341 MiB used, 625 GiB / 626 GiB avail 2024-02-05T06:32:53.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:53 smithi028 ceph-mon[37984]: osdmap e52: 7 total, 7 up, 7 in 2024-02-05T06:32:53.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:53 smithi028 ceph-mon[37984]: Detected new or changed devices on smithi167 2024-02-05T06:32:53.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:53 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:53.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:53 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:53.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:53 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:32:53.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:53 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:32:53.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:53 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:32:53.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:53 smithi028 ceph-mon[37984]: Adjusting osd_memory_target on smithi167 to 5696M 2024-02-05T06:32:53.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:53 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:53.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:53 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:32:53.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:53 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:32:53.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:53 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:53.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:53 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:32:53.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:53 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:32:53.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:53 smithi028 ceph-mon[29442]: pgmap v160: 1 pgs: 1 remapped+peering; 577 KiB data, 341 MiB used, 625 GiB / 626 GiB avail 2024-02-05T06:32:53.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:53 smithi028 ceph-mon[29442]: osdmap e52: 7 total, 7 up, 7 in 2024-02-05T06:32:53.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:53 smithi028 ceph-mon[29442]: Detected new or changed devices on smithi167 2024-02-05T06:32:53.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:53 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:53.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:53 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:53.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:53 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:32:53.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:53 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:32:53.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:53 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:32:53.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:53 smithi028 ceph-mon[29442]: Adjusting osd_memory_target on smithi167 to 5696M 2024-02-05T06:32:53.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:53 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:53.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:53 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:32:53.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:53 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:32:53.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:53 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:32:53.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:53 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:32:53.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:53 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:32:54.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:54 smithi028 ceph-mon[29442]: osdmap e53: 7 total, 7 up, 7 in 2024-02-05T06:32:54.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:54 smithi028 ceph-mon[37984]: osdmap e53: 7 total, 7 up, 7 in 2024-02-05T06:32:54.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:54 smithi167 ceph-mon[32774]: osdmap e53: 7 total, 7 up, 7 in 2024-02-05T06:32:55.917 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:55 smithi167 ceph-mon[32774]: pgmap v163: 1 pgs: 1 remapped+peering; 577 KiB data, 341 MiB used, 625 GiB / 626 GiB avail 2024-02-05T06:32:55.917 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:55 smithi167 ceph-mon[32774]: from='client.24316 v1:172.21.15.167:0/2574700147' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:32:55.917 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:55 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T06:32:55.917 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:55 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T06:32:55.917 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:55 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:32:55.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:55 smithi028 ceph-mon[29442]: pgmap v163: 1 pgs: 1 remapped+peering; 577 KiB data, 341 MiB used, 625 GiB / 626 GiB avail 2024-02-05T06:32:55.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:55 smithi028 ceph-mon[29442]: from='client.24316 v1:172.21.15.167:0/2574700147' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:32:55.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:55 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T06:32:55.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:55 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T06:32:55.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:55 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:32:55.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:55 smithi028 ceph-mon[37984]: pgmap v163: 1 pgs: 1 remapped+peering; 577 KiB data, 341 MiB used, 625 GiB / 626 GiB avail 2024-02-05T06:32:55.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:55 smithi028 ceph-mon[37984]: from='client.24316 v1:172.21.15.167:0/2574700147' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi167:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:32:55.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:55 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-05T06:32:55.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:55 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-05T06:32:55.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:55 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:32:57.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:57 smithi028 ceph-mon[29442]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-02-05T06:32:57.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:57 smithi028 ceph-mon[37984]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-02-05T06:32:57.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:57 smithi167 ceph-mon[32774]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-02-05T06:32:58.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:58 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.167:0/4094540214' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c545cbc-0873-4232-8807-4af9299ef95e"}]: dispatch 2024-02-05T06:32:58.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:58 smithi028 ceph-mon[29442]: from='client.24322 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c545cbc-0873-4232-8807-4af9299ef95e"}]: dispatch 2024-02-05T06:32:58.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:58 smithi028 ceph-mon[29442]: from='client.24322 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c545cbc-0873-4232-8807-4af9299ef95e"}]': finished 2024-02-05T06:32:58.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:58 smithi028 ceph-mon[29442]: osdmap e54: 8 total, 7 up, 8 in 2024-02-05T06:32:58.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:58 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T06:32:58.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:58 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.167:0/3882923527' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T06:32:58.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:58 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.167:0/4094540214' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c545cbc-0873-4232-8807-4af9299ef95e"}]: dispatch 2024-02-05T06:32:58.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:58 smithi028 ceph-mon[37984]: from='client.24322 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c545cbc-0873-4232-8807-4af9299ef95e"}]: dispatch 2024-02-05T06:32:58.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:58 smithi028 ceph-mon[37984]: from='client.24322 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c545cbc-0873-4232-8807-4af9299ef95e"}]': finished 2024-02-05T06:32:58.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:58 smithi028 ceph-mon[37984]: osdmap e54: 8 total, 7 up, 8 in 2024-02-05T06:32:58.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:58 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T06:32:58.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:58 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.167:0/3882923527' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T06:32:58.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:58 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.167:0/4094540214' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c545cbc-0873-4232-8807-4af9299ef95e"}]: dispatch 2024-02-05T06:32:58.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:58 smithi167 ceph-mon[32774]: from='client.24322 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c545cbc-0873-4232-8807-4af9299ef95e"}]: dispatch 2024-02-05T06:32:58.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:58 smithi167 ceph-mon[32774]: from='client.24322 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c545cbc-0873-4232-8807-4af9299ef95e"}]': finished 2024-02-05T06:32:58.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:58 smithi167 ceph-mon[32774]: osdmap e54: 8 total, 7 up, 8 in 2024-02-05T06:32:58.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:58 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T06:32:58.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:58 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.167:0/3882923527' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-05T06:33:00.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:32:59 smithi028 ceph-mon[29442]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-05T06:33:00.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:32:59 smithi028 ceph-mon[37984]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-05T06:33:00.227 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:32:59 smithi167 ceph-mon[32774]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-05T06:33:02.133 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:01 smithi167 ceph-mon[32774]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-02-05T06:33:02.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:01 smithi028 ceph-mon[29442]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-02-05T06:33:02.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:01 smithi028 ceph-mon[37984]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-02-05T06:33:03.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:03 smithi167 ceph-mon[32774]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-02-05T06:33:04.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:03 smithi028 ceph-mon[29442]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-02-05T06:33:04.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:03 smithi028 ceph-mon[37984]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-02-05T06:33:05.138 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:04 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:33:05.138 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:04 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:33:05.139 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:04 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-05T06:33:05.139 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:04 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:33:05.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:04 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:33:05.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:04 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:33:05.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:04 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-05T06:33:05.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:04 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:33:05.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:04 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:33:05.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:04 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:33:05.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:04 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-05T06:33:05.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:04 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:33:06.018 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:05 smithi167 ceph-mon[32774]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-05T06:33:06.018 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:05 smithi167 ceph-mon[32774]: Deploying daemon osd.7 on smithi167 2024-02-05T06:33:06.018 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:05 smithi167 ceph-mon[32774]: osdmap e55: 8 total, 7 up, 8 in 2024-02-05T06:33:06.018 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:05 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T06:33:06.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:05 smithi028 ceph-mon[29442]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-05T06:33:06.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:05 smithi028 ceph-mon[29442]: Deploying daemon osd.7 on smithi167 2024-02-05T06:33:06.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:05 smithi028 ceph-mon[29442]: osdmap e55: 8 total, 7 up, 8 in 2024-02-05T06:33:06.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:05 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T06:33:06.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:05 smithi028 ceph-mon[37984]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-05T06:33:06.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:05 smithi028 ceph-mon[37984]: Deploying daemon osd.7 on smithi167 2024-02-05T06:33:06.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:05 smithi028 ceph-mon[37984]: osdmap e55: 8 total, 7 up, 8 in 2024-02-05T06:33:06.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:05 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T06:33:08.123 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:07 smithi167 ceph-mon[32774]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-05T06:33:08.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:07 smithi028 ceph-mon[29442]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-05T06:33:08.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:07 smithi028 ceph-mon[37984]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-05T06:33:10.177 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:09 smithi167 ceph-mon[32774]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-05T06:33:10.177 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:09 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:10.177 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:09 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:10.177 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:09 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:33:10.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:09 smithi028 ceph-mon[29442]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-05T06:33:10.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:09 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:10.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:09 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:10.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:09 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:33:10.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:09 smithi028 ceph-mon[37984]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-05T06:33:10.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:09 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:10.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:09 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:10.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:09 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:33:11.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:11 smithi167 ceph-mon[32774]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-05T06:33:12.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:11 smithi028 ceph-mon[29442]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-05T06:33:12.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:11 smithi028 ceph-mon[37984]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-05T06:33:13.542 INFO:teuthology.orchestra.run.smithi167.stdout:Created osd(s) 7 on host 'smithi167' 2024-02-05T06:33:14.184 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:13 smithi167 ceph-mon[32774]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-05T06:33:14.184 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:13 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:14.184 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:13 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:14.184 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:13 smithi167 ceph-mon[32774]: from='osd.7 v1:172.21.15.167:6812/4031475764' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-05T06:33:14.184 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:13 smithi167 ceph-mon[32774]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-05T06:33:14.184 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:13 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:14.185 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:13 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:14.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:13 smithi028 ceph-mon[29442]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-05T06:33:14.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:13 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:14.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:13 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:14.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:13 smithi028 ceph-mon[29442]: from='osd.7 v1:172.21.15.167:6812/4031475764' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-05T06:33:14.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:13 smithi028 ceph-mon[29442]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-05T06:33:14.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:13 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:14.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:13 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:14.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:13 smithi028 ceph-mon[37984]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-05T06:33:14.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:13 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:14.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:13 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:14.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:13 smithi028 ceph-mon[37984]: from='osd.7 v1:172.21.15.167:6812/4031475764' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-05T06:33:14.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:13 smithi028 ceph-mon[37984]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-05T06:33:14.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:13 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:14.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:13 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:14.212 DEBUG:teuthology.orchestra.run.smithi167:osd.7> sudo journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@osd.7.service 2024-02-05T06:33:14.215 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-02-05T06:33:14.215 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd stat -f json 2024-02-05T06:33:14.472 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:33:14.916 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:14 smithi028 ceph-mon[37984]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-05T06:33:14.917 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:14 smithi028 ceph-mon[37984]: osdmap e56: 8 total, 7 up, 8 in 2024-02-05T06:33:14.917 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:14 smithi028 ceph-mon[37984]: from='osd.7 v1:172.21.15.167:6812/4031475764' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]: dispatch 2024-02-05T06:33:14.917 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:14 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T06:33:14.917 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:14 smithi028 ceph-mon[37984]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]: dispatch 2024-02-05T06:33:14.917 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:14 smithi028 ceph-mon[29442]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-05T06:33:14.917 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:14 smithi028 ceph-mon[29442]: osdmap e56: 8 total, 7 up, 8 in 2024-02-05T06:33:14.918 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:14 smithi028 ceph-mon[29442]: from='osd.7 v1:172.21.15.167:6812/4031475764' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]: dispatch 2024-02-05T06:33:14.918 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:14 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T06:33:14.918 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:14 smithi028 ceph-mon[29442]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]: dispatch 2024-02-05T06:33:15.171 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:14 smithi167 ceph-mon[32774]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-05T06:33:15.171 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:14 smithi167 ceph-mon[32774]: osdmap e56: 8 total, 7 up, 8 in 2024-02-05T06:33:15.171 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:14 smithi167 ceph-mon[32774]: from='osd.7 v1:172.21.15.167:6812/4031475764' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]: dispatch 2024-02-05T06:33:15.171 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:14 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T06:33:15.171 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:14 smithi167 ceph-mon[32774]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]: dispatch 2024-02-05T06:33:15.478 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:33:15 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:33:15.262+0000 7f3c1d9c7700 -1 osd.7 0 waiting for initial osdmap 2024-02-05T06:33:15.478 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:33:15 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:33:15.268+0000 7f3c157b9700 -1 osd.7 57 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-05T06:33:15.904 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:15 smithi028 ceph-mon[37984]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-05T06:33:15.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:15 smithi028 ceph-mon[37984]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]': finished 2024-02-05T06:33:15.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:15 smithi028 ceph-mon[37984]: osdmap e57: 8 total, 7 up, 8 in 2024-02-05T06:33:15.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:15 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T06:33:15.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:15 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T06:33:15.905 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:15 smithi028 ceph-mon[37984]: from='osd.7 ' entity='osd.7' 2024-02-05T06:33:15.906 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:15 smithi028 ceph-mon[29442]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-05T06:33:15.906 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:15 smithi028 ceph-mon[29442]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]': finished 2024-02-05T06:33:15.906 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:15 smithi028 ceph-mon[29442]: osdmap e57: 8 total, 7 up, 8 in 2024-02-05T06:33:15.906 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:15 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T06:33:15.906 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:15 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T06:33:15.906 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:15 smithi028 ceph-mon[29442]: from='osd.7 ' entity='osd.7' 2024-02-05T06:33:16.117 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:15 smithi167 ceph-mon[32774]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-02-05T06:33:16.118 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:15 smithi167 ceph-mon[32774]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi167", "root=default"]}]': finished 2024-02-05T06:33:16.118 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:15 smithi167 ceph-mon[32774]: osdmap e57: 8 total, 7 up, 8 in 2024-02-05T06:33:16.118 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:15 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T06:33:16.118 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:15 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T06:33:16.118 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:15 smithi167 ceph-mon[32774]: from='osd.7 ' entity='osd.7' 2024-02-05T06:33:16.452 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:33:17.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:16 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:33:17.035 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:16 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:33:17.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:16 smithi028 ceph-mon[29442]: osd.7 v1:172.21.15.167:6812/4031475764 boot 2024-02-05T06:33:17.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:16 smithi028 ceph-mon[29442]: osdmap e58: 8 total, 8 up, 8 in 2024-02-05T06:33:17.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:16 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T06:33:17.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:16 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1502296948' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-05T06:33:17.036 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:16 smithi028 ceph-mon[29442]: osdmap e59: 8 total, 8 up, 8 in 2024-02-05T06:33:17.036 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:16 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:33:17.036 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:16 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:33:17.036 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:16 smithi028 ceph-mon[37984]: osd.7 v1:172.21.15.167:6812/4031475764 boot 2024-02-05T06:33:17.036 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:16 smithi028 ceph-mon[37984]: osdmap e58: 8 total, 8 up, 8 in 2024-02-05T06:33:17.036 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:16 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T06:33:17.037 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:16 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1502296948' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-05T06:33:17.037 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:16 smithi028 ceph-mon[37984]: osdmap e59: 8 total, 8 up, 8 in 2024-02-05T06:33:17.061 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":58,"num_osds":8,"num_up_osds":8,"osd_up_since":1707114795,"num_in_osds":8,"osd_in_since":1707114777,"num_remapped_pgs":1} 2024-02-05T06:33:17.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd dump --format=json 2024-02-05T06:33:17.227 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:16 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:33:17.227 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:16 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:33:17.227 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:16 smithi167 ceph-mon[32774]: osd.7 v1:172.21.15.167:6812/4031475764 boot 2024-02-05T06:33:17.227 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:16 smithi167 ceph-mon[32774]: osdmap e58: 8 total, 8 up, 8 in 2024-02-05T06:33:17.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:16 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T06:33:17.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:16 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1502296948' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-05T06:33:17.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:16 smithi167 ceph-mon[32774]: osdmap e59: 8 total, 8 up, 8 in 2024-02-05T06:33:17.306 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:33:17.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:17 smithi167 ceph-mon[32774]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 715 GiB / 715 GiB avail 2024-02-05T06:33:17.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:17 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:17.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:17 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:17.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:17 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:33:17.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:17 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:33:17.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:17 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:33:17.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:17 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:33:17.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:17 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:17.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:17 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:33:17.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:17 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:33:17.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:17 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:17.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:17 smithi167 ceph-mon[32774]: osdmap e60: 8 total, 8 up, 8 in 2024-02-05T06:33:18.109 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:17 smithi028 ceph-mon[37984]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 715 GiB / 715 GiB avail 2024-02-05T06:33:18.109 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:17 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:18.109 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:17 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:18.109 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:17 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:33:18.109 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:17 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:33:18.110 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:17 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:33:18.110 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:17 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:33:18.110 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:17 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:18.110 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:17 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:33:18.110 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:17 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:33:18.110 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:17 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:18.110 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:17 smithi028 ceph-mon[37984]: osdmap e60: 8 total, 8 up, 8 in 2024-02-05T06:33:18.110 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:17 smithi028 ceph-mon[29442]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 715 GiB / 715 GiB avail 2024-02-05T06:33:18.110 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:17 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:18.110 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:17 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:18.110 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:17 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:33:18.111 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:17 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:33:18.111 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:17 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:33:18.111 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:17 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:33:18.111 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:17 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:18.111 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:17 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:33:18.111 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:17 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:33:18.111 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:17 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:18.111 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:17 smithi028 ceph-mon[29442]: osdmap e60: 8 total, 8 up, 8 in 2024-02-05T06:33:19.171 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:33:19.171 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":61,"fsid":"9801e956-c3ef-11ee-95b6-87774f69a715","created":"2024-02-05T06:27:59.919226+0000","modified":"2024-02-05T06:33:18.851403+0000","last_up_change":"2024-02-05T06:33:15.833573+0000","last_in_change":"2024-02-05T06:32:57.809784+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-05T06:31:10.441424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c19fdd5b-09f6-45d4-a519-0a938cf1f2a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6801","nonce":301279106}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6802","nonce":301279106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6804","nonce":301279106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6803","nonce":301279106}]},"public_addr":"172.21.15.28:6801/301279106","cluster_addr":"172.21.15.28:6802/301279106","heartbeat_back_addr":"172.21.15.28:6804/301279106","heartbeat_front_addr":"172.21.15.28:6803/301279106","state":["exists","up"]},{"osd":1,"uuid":"52245f6e-eb96-4d0d-985b-dbb6fd3ce8de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6805","nonce":1795224649}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6806","nonce":1795224649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6808","nonce":1795224649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6807","nonce":1795224649}]},"public_addr":"172.21.15.28:6805/1795224649","cluster_addr":"172.21.15.28:6806/1795224649","heartbeat_back_addr":"172.21.15.28:6808/1795224649","heartbeat_front_addr":"172.21.15.28:6807/1795224649","state":["exists","up"]},{"osd":2,"uuid":"1e18b630-afdf-4de9-a1a2-37bdfdd01c67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6809","nonce":1993280018}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6810","nonce":1993280018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6812","nonce":1993280018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6811","nonce":1993280018}]},"public_addr":"172.21.15.28:6809/1993280018","cluster_addr":"172.21.15.28:6810/1993280018","heartbeat_back_addr":"172.21.15.28:6812/1993280018","heartbeat_front_addr":"172.21.15.28:6811/1993280018","state":["exists","up"]},{"osd":3,"uuid":"3ae795af-3d59-41a6-95eb-b8447ff61282","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6813","nonce":1294621398}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6814","nonce":1294621398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6816","nonce":1294621398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6815","nonce":1294621398}]},"public_addr":"172.21.15.28:6813/1294621398","cluster_addr":"172.21.15.28:6814/1294621398","heartbeat_back_addr":"172.21.15.28:6816/1294621398","heartbeat_front_addr":"172.21.15.28:6815/1294621398","state":["exists","up"]},{"osd":4,"uuid":"fc6bd83d-570c-48c0-964d-e09b4608bc32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6800","nonce":4052090595}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6801","nonce":4052090595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6803","nonce":4052090595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6802","nonce":4052090595}]},"public_addr":"172.21.15.167:6800/4052090595","cluster_addr":"172.21.15.167:6801/4052090595","heartbeat_back_addr":"172.21.15.167:6803/4052090595","heartbeat_front_addr":"172.21.15.167:6802/4052090595","state":["exists","up"]},{"osd":5,"uuid":"895bf465-2531-4d9f-bc23-1e75fd969f07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6804","nonce":1631080500}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6805","nonce":1631080500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6807","nonce":1631080500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6806","nonce":1631080500}]},"public_addr":"172.21.15.167:6804/1631080500","cluster_addr":"172.21.15.167:6805/1631080500","heartbeat_back_addr":"172.21.15.167:6807/1631080500","heartbeat_front_addr":"172.21.15.167:6806/1631080500","state":["exists","up"]},{"osd":6,"uuid":"ce0a9300-87a4-4e37-b702-d667f168c0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6808","nonce":1644050617}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6809","nonce":1644050617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6811","nonce":1644050617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6810","nonce":1644050617}]},"public_addr":"172.21.15.167:6808/1644050617","cluster_addr":"172.21.15.167:6809/1644050617","heartbeat_back_addr":"172.21.15.167:6811/1644050617","heartbeat_front_addr":"172.21.15.167:6810/1644050617","state":["exists","up"]},{"osd":7,"uuid":"0c545cbc-0873-4232-8807-4af9299ef95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6812","nonce":4031475764}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6813","nonce":4031475764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6815","nonce":4031475764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6814","nonce":4031475764}]},"public_addr":"172.21.15.167:6812/4031475764","cluster_addr":"172.21.15.167:6813/4031475764","heartbeat_back_addr":"172.21.15.167:6815/4031475764","heartbeat_front_addr":"172.21.15.167:6814/4031475764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:32:39.576355+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:33:03.970578+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:32:14.803510+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:32:52.894807+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:31:56.698279+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:32:22.224066+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:32:47.949781+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:33:13.931286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/4125764599":"2024-02-06T06:28:54.105170+0000","172.21.15.28:0/1081543653":"2024-02-06T06:28:54.105170+0000","172.21.15.28:0/3794420554":"2024-02-06T06:28:27.553034+0000","172.21.15.28:6800/4266601437":"2024-02-06T06:28:27.553034+0000","172.21.15.28:0/388893831":"2024-02-06T06:28:27.553034+0000","172.21.15.28:0/460817910":"2024-02-06T06:28:27.553034+0000","172.21.15.28:0/2728305743":"2024-02-06T06:28:54.105170+0000","172.21.15.28:0/3605013594":"2024-02-06T06:28:54.105170+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-05T06:33:19.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:18 smithi028 ceph-mon[29442]: Detected new or changed devices on smithi167 2024-02-05T06:33:19.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:18 smithi028 ceph-mon[29442]: Adjusting osd_memory_target on smithi167 to 4272M 2024-02-05T06:33:19.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:18 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:33:19.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:18 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:33:19.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:18 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:33:19.172 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:18 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:19.172 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:18 smithi028 ceph-mon[37984]: Detected new or changed devices on smithi167 2024-02-05T06:33:19.172 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:18 smithi028 ceph-mon[37984]: Adjusting osd_memory_target on smithi167 to 4272M 2024-02-05T06:33:19.172 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:18 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:33:19.172 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:18 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:33:19.172 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:18 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:33:19.172 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:18 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:19.227 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:18 smithi167 ceph-mon[32774]: Detected new or changed devices on smithi167 2024-02-05T06:33:19.227 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:18 smithi167 ceph-mon[32774]: Adjusting osd_memory_target on smithi167 to 4272M 2024-02-05T06:33:19.227 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:18 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:33:19.227 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:18 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:33:19.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:18 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:33:19.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:18 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:19.788 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-05T06:31:10.441424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-05T06:33:19.788 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-02-05T06:33:20.033 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:33:20.151 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:19 smithi028 ceph-mon[29442]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 715 GiB / 715 GiB avail 2024-02-05T06:33:20.151 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:19 smithi028 ceph-mon[29442]: osdmap e61: 8 total, 8 up, 8 in 2024-02-05T06:33:20.151 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:19 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/389398316' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T06:33:20.151 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:19 smithi028 ceph-mon[37984]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 715 GiB / 715 GiB avail 2024-02-05T06:33:20.152 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:19 smithi028 ceph-mon[37984]: osdmap e61: 8 total, 8 up, 8 in 2024-02-05T06:33:20.152 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:19 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/389398316' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T06:33:20.227 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:19 smithi167 ceph-mon[32774]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 715 GiB / 715 GiB avail 2024-02-05T06:33:20.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:19 smithi167 ceph-mon[32774]: osdmap e61: 8 total, 8 up, 8 in 2024-02-05T06:33:20.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:19 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/389398316' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T06:33:21.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:20 smithi028 ceph-mon[29442]: osdmap e62: 8 total, 8 up, 8 in 2024-02-05T06:33:21.132 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:20 smithi028 ceph-mon[37984]: osdmap e62: 8 total, 8 up, 8 in 2024-02-05T06:33:21.227 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:20 smithi167 ceph-mon[32774]: osdmap e62: 8 total, 8 up, 8 in 2024-02-05T06:33:21.832 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-02-05T06:33:22.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:21 smithi028 ceph-mon[29442]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 715 GiB / 715 GiB avail 2024-02-05T06:33:22.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:21 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2065350177' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-05T06:33:22.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:21 smithi028 ceph-mon[37984]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 715 GiB / 715 GiB avail 2024-02-05T06:33:22.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:21 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2065350177' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-05T06:33:22.227 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:21 smithi167 ceph-mon[32774]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 316 MiB used, 715 GiB / 715 GiB avail 2024-02-05T06:33:22.227 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:21 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2065350177' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-05T06:33:22.408 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi028 2024-02-05T06:33:22.408 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi028=foo.a' 2024-02-05T06:33:22.660 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:33:24.077 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:23 smithi167 ceph-mon[32774]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-02-05T06:33:24.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:23 smithi028 ceph-mon[29442]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-02-05T06:33:24.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:23 smithi028 ceph-mon[37984]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-02-05T06:33:24.523 INFO:teuthology.orchestra.run.smithi167.stdout:Scheduled rgw.foo.a update... 2024-02-05T06:33:25.182 DEBUG:teuthology.orchestra.run.smithi028:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@rgw.foo.a.service 2024-02-05T06:33:25.185 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi167 2024-02-05T06:33:25.185 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-02-05T06:33:25.422 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:33:25.773 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:25 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:33:25.773 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:25 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:33:25.774 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:25 smithi028 ceph-mon[29442]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-02-05T06:33:25.774 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:25 smithi028 ceph-mon[29442]: from='client.24361 v1:172.21.15.167:0/3222404559' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi028=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:33:25.774 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:25 smithi028 ceph-mon[29442]: Saving service rgw.foo.a spec with placement smithi028=foo.a;count:1 2024-02-05T06:33:25.774 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:25 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:25.774 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:25 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:33:25.774 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:25 smithi028 ceph-mon[29442]: osdmap e63: 8 total, 8 up, 8 in 2024-02-05T06:33:25.774 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:25 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:33:25.774 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:25 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:33:25.774 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:25 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:25.774 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:25 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-05T06:33:25.774 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:25 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-05T06:33:25.774 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:25 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:25.774 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:25 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:33:25.775 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:25 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:33:25.775 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:25 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:33:25.775 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:25 smithi028 ceph-mon[37984]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-02-05T06:33:25.775 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:25 smithi028 ceph-mon[37984]: from='client.24361 v1:172.21.15.167:0/3222404559' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi028=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:33:25.775 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:25 smithi028 ceph-mon[37984]: Saving service rgw.foo.a spec with placement smithi028=foo.a;count:1 2024-02-05T06:33:25.775 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:25 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:25.775 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:25 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:33:25.775 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:25 smithi028 ceph-mon[37984]: osdmap e63: 8 total, 8 up, 8 in 2024-02-05T06:33:25.775 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:25 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:33:25.775 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:25 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:33:25.775 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:25 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:25.775 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:25 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-05T06:33:25.775 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:25 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-05T06:33:25.775 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:25 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:25.776 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:25 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:33:25.850 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:25 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:33:25.850 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:25 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:33:25.850 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:25 smithi167 ceph-mon[32774]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-02-05T06:33:25.850 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:25 smithi167 ceph-mon[32774]: from='client.24361 v1:172.21.15.167:0/3222404559' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi028=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:33:25.850 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:25 smithi167 ceph-mon[32774]: Saving service rgw.foo.a spec with placement smithi028=foo.a;count:1 2024-02-05T06:33:25.850 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:25 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:25.850 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:25 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:33:25.851 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:25 smithi167 ceph-mon[32774]: osdmap e63: 8 total, 8 up, 8 in 2024-02-05T06:33:25.851 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:25 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:33:25.851 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:25 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:33:25.851 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:25 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:25.851 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:25 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-05T06:33:25.851 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:25 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-05T06:33:25.851 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:25 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:25.851 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:25 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:33:26.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:26 smithi028 ceph-mon[29442]: Deploying daemon rgw.foo.a on smithi028 2024-02-05T06:33:26.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:26 smithi028 ceph-mon[37984]: Deploying daemon rgw.foo.a on smithi028 2024-02-05T06:33:26.840 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:26 smithi167 ceph-mon[32774]: Deploying daemon rgw.foo.a on smithi028 2024-02-05T06:33:27.443 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Feb 05 06:33:27 smithi028 systemd[1]: Starting Ceph rgw.foo.a for 9801e956-c3ef-11ee-95b6-87774f69a715... 2024-02-05T06:33:27.532 INFO:teuthology.orchestra.run.smithi167.stderr:pool 'datapool' created 2024-02-05T06:33:27.804 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:27 smithi028 ceph-mon[29442]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-02-05T06:33:27.804 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:27 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.167:0/2496614123' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-05T06:33:27.804 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:27 smithi028 ceph-mon[29442]: from='client.24367 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-05T06:33:27.804 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:27 smithi028 ceph-mon[37984]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-02-05T06:33:27.804 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:27 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.167:0/2496614123' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-05T06:33:27.804 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:27 smithi028 ceph-mon[37984]: from='client.24367 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-05T06:33:27.805 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Feb 05 06:33:27 smithi028 podman[64615]: 2024-02-05 06:33:27.438038122 +0000 UTC m=+0.017915111 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T06:33:27.805 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Feb 05 06:33:27 smithi028 podman[64615]: 2024-02-05 06:33:27.550979895 +0000 UTC m=+0.130856879 container create 0c8a74bcd665cf060f92be62f0e89b552fec5fd25d64d7e0b28b71ad94fdd6f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-rgw-foo-a, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-02-05T06:33:27.805 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Feb 05 06:33:27 smithi028 podman[64615]: 2024-02-05 06:33:27.803628443 +0000 UTC m=+0.383505427 container init 0c8a74bcd665cf060f92be62f0e89b552fec5fd25d64d7e0b28b71ad94fdd6f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-rgw-foo-a, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_CLEAN=True) 2024-02-05T06:33:27.851 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:27 smithi167 ceph-mon[32774]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-02-05T06:33:27.852 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:27 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.167:0/2496614123' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-05T06:33:27.852 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:27 smithi167 ceph-mon[32774]: from='client.24367 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-02-05T06:33:28.142 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- rbd pool init datapool 2024-02-05T06:33:28.192 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Feb 05 06:33:27 smithi028 podman[64615]: 2024-02-05 06:33:27.806557164 +0000 UTC m=+0.386434157 container start 0c8a74bcd665cf060f92be62f0e89b552fec5fd25d64d7e0b28b71ad94fdd6f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-rgw-foo-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2) 2024-02-05T06:33:28.193 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Feb 05 06:33:27 smithi028 bash[64615]: 0c8a74bcd665cf060f92be62f0e89b552fec5fd25d64d7e0b28b71ad94fdd6f8 2024-02-05T06:33:28.193 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Feb 05 06:33:27 smithi028 systemd[1]: Started Ceph rgw.foo.a for 9801e956-c3ef-11ee-95b6-87774f69a715. 2024-02-05T06:33:28.385 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:33:28.847 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:28 smithi167 ceph-mon[32774]: from='client.24367 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-05T06:33:28.848 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:28 smithi167 ceph-mon[32774]: osdmap e64: 8 total, 8 up, 8 in 2024-02-05T06:33:28.848 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:28 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:28.848 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:28 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:28.848 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:28 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:28.848 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:28 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:28.848 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:28 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:28.848 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:28 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:33:28.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:28 smithi028 ceph-mon[37984]: from='client.24367 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-05T06:33:28.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:28 smithi028 ceph-mon[37984]: osdmap e64: 8 total, 8 up, 8 in 2024-02-05T06:33:28.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:28 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:28.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:28 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:28.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:28 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:28.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:28 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:28.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:28 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:28.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:28 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:33:28.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:28 smithi028 ceph-mon[29442]: from='client.24367 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-02-05T06:33:28.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:28 smithi028 ceph-mon[29442]: osdmap e64: 8 total, 8 up, 8 in 2024-02-05T06:33:28.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:28 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:28.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:28 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:28.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:28 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:28.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:28 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:28.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:28 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:28.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:28 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:33:29.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:29 smithi167 ceph-mon[32774]: Saving service rgw.foo.a spec with placement smithi028=foo.a;count:1 2024-02-05T06:33:29.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:29 smithi167 ceph-mon[32774]: pgmap v191: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-05T06:33:29.727 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:29 smithi167 ceph-mon[32774]: osdmap e65: 8 total, 8 up, 8 in 2024-02-05T06:33:29.728 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:29 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/111672690' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-05T06:33:29.728 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:29 smithi167 ceph-mon[32774]: from='client.44366 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-05T06:33:29.728 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:29 smithi167 ceph-mon[32774]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T06:33:29.728 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:29 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:29.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:29 smithi028 ceph-mon[37984]: Saving service rgw.foo.a spec with placement smithi028=foo.a;count:1 2024-02-05T06:33:29.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:29 smithi028 ceph-mon[37984]: pgmap v191: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-05T06:33:29.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:29 smithi028 ceph-mon[37984]: osdmap e65: 8 total, 8 up, 8 in 2024-02-05T06:33:29.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:29 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/111672690' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-05T06:33:29.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:29 smithi028 ceph-mon[37984]: from='client.44366 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-05T06:33:29.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:29 smithi028 ceph-mon[37984]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T06:33:29.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:29 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:29.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:29 smithi028 ceph-mon[29442]: Saving service rgw.foo.a spec with placement smithi028=foo.a;count:1 2024-02-05T06:33:29.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:29 smithi028 ceph-mon[29442]: pgmap v191: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-05T06:33:29.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:29 smithi028 ceph-mon[29442]: osdmap e65: 8 total, 8 up, 8 in 2024-02-05T06:33:29.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:29 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/111672690' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-05T06:33:29.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:29 smithi028 ceph-mon[29442]: from='client.44366 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-05T06:33:29.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:29 smithi028 ceph-mon[29442]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T06:33:29.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:29 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:30.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:30 smithi028 ceph-mon[29442]: from='client.44366 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-05T06:33:30.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:30 smithi028 ceph-mon[29442]: osdmap e66: 8 total, 8 up, 8 in 2024-02-05T06:33:30.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:30 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:30.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:30 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:30.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:30 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:33:30.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:30 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:33:30.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:30 smithi028 ceph-mon[29442]: Checking dashboard <-> RGW credentials 2024-02-05T06:33:30.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:30 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.167:0/983376340' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-05T06:33:30.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:30 smithi028 ceph-mon[29442]: from='client.24391 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-05T06:33:30.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:30 smithi028 ceph-mon[37984]: from='client.44366 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-05T06:33:30.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:30 smithi028 ceph-mon[37984]: osdmap e66: 8 total, 8 up, 8 in 2024-02-05T06:33:30.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:30 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:30.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:30 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:30.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:30 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:33:30.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:30 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:33:30.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:30 smithi028 ceph-mon[37984]: Checking dashboard <-> RGW credentials 2024-02-05T06:33:30.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:30 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.167:0/983376340' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-05T06:33:30.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:30 smithi028 ceph-mon[37984]: from='client.24391 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-05T06:33:30.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:30 smithi167 ceph-mon[32774]: from='client.44366 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-05T06:33:30.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:30 smithi167 ceph-mon[32774]: osdmap e66: 8 total, 8 up, 8 in 2024-02-05T06:33:30.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:30 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:30.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:30 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:30.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:30 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:33:30.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:30 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:33:30.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:30 smithi167 ceph-mon[32774]: Checking dashboard <-> RGW credentials 2024-02-05T06:33:30.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:30 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.167:0/983376340' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-05T06:33:30.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:30 smithi167 ceph-mon[32774]: from='client.24391 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-02-05T06:33:31.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:31 smithi028 ceph-mon[37984]: pgmap v194: 36 pgs: 5 creating+peering, 17 active+clean, 14 unknown; 578 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 387 B/s rd, 775 B/s wr, 1 op/s 2024-02-05T06:33:31.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:31 smithi028 ceph-mon[37984]: from='client.24391 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-02-05T06:33:31.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:31 smithi028 ceph-mon[37984]: osdmap e67: 8 total, 8 up, 8 in 2024-02-05T06:33:31.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:31 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1937500867' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T06:33:31.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:31 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3445405819' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T06:33:31.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:31 smithi028 ceph-mon[37984]: from='client.44375 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T06:33:31.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:31 smithi028 ceph-mon[37984]: from='client.44411 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T06:33:31.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:31 smithi028 ceph-mon[29442]: pgmap v194: 36 pgs: 5 creating+peering, 17 active+clean, 14 unknown; 578 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 387 B/s rd, 775 B/s wr, 1 op/s 2024-02-05T06:33:31.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:31 smithi028 ceph-mon[29442]: from='client.24391 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-02-05T06:33:31.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:31 smithi028 ceph-mon[29442]: osdmap e67: 8 total, 8 up, 8 in 2024-02-05T06:33:31.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:31 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1937500867' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T06:33:31.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:31 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3445405819' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T06:33:31.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:31 smithi028 ceph-mon[29442]: from='client.44375 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T06:33:31.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:31 smithi028 ceph-mon[29442]: from='client.44411 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T06:33:31.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:31 smithi167 ceph-mon[32774]: pgmap v194: 36 pgs: 5 creating+peering, 17 active+clean, 14 unknown; 578 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 387 B/s rd, 775 B/s wr, 1 op/s 2024-02-05T06:33:31.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:31 smithi167 ceph-mon[32774]: from='client.24391 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-02-05T06:33:31.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:31 smithi167 ceph-mon[32774]: osdmap e67: 8 total, 8 up, 8 in 2024-02-05T06:33:31.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:31 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1937500867' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T06:33:31.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:31 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3445405819' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T06:33:31.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:31 smithi167 ceph-mon[32774]: from='client.44375 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T06:33:31.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:31 smithi167 ceph-mon[32774]: from='client.44411 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-05T06:33:32.883 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:32 smithi167 ceph-mon[32774]: from='client.44375 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-05T06:33:32.884 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:32 smithi167 ceph-mon[32774]: from='client.44411 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-05T06:33:32.884 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:32 smithi167 ceph-mon[32774]: osdmap e68: 8 total, 8 up, 8 in 2024-02-05T06:33:32.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:32 smithi028 ceph-mon[29442]: from='client.44375 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-05T06:33:32.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:32 smithi028 ceph-mon[29442]: from='client.44411 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-05T06:33:32.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:32 smithi028 ceph-mon[29442]: osdmap e68: 8 total, 8 up, 8 in 2024-02-05T06:33:32.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:32 smithi028 ceph-mon[37984]: from='client.44375 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-05T06:33:32.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:32 smithi028 ceph-mon[37984]: from='client.44411 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-05T06:33:32.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:32 smithi028 ceph-mon[37984]: osdmap e68: 8 total, 8 up, 8 in 2024-02-05T06:33:33.173 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.167 --placement '1;smithi167=iscsi.a' 2024-02-05T06:33:33.417 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:33:33.870 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:33 smithi167 ceph-mon[32774]: pgmap v197: 68 pgs: 12 creating+peering, 31 active+clean, 25 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-05T06:33:33.871 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:33 smithi167 ceph-mon[32774]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T06:33:33.871 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:33 smithi167 ceph-mon[32774]: Cluster is now healthy 2024-02-05T06:33:33.871 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:33 smithi167 ceph-mon[32774]: osdmap e69: 8 total, 8 up, 8 in 2024-02-05T06:33:33.871 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:33 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1937500867' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T06:33:33.871 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:33 smithi167 ceph-mon[32774]: from='client.44375 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T06:33:33.871 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:33 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3445405819' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T06:33:33.871 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:33 smithi167 ceph-mon[32774]: from='client.44411 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T06:33:33.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:33 smithi028 ceph-mon[29442]: pgmap v197: 68 pgs: 12 creating+peering, 31 active+clean, 25 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-05T06:33:33.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:33 smithi028 ceph-mon[29442]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T06:33:33.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:33 smithi028 ceph-mon[29442]: Cluster is now healthy 2024-02-05T06:33:33.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:33 smithi028 ceph-mon[29442]: osdmap e69: 8 total, 8 up, 8 in 2024-02-05T06:33:33.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:33 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1937500867' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T06:33:33.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:33 smithi028 ceph-mon[29442]: from='client.44375 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T06:33:33.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:33 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3445405819' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T06:33:33.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:33 smithi028 ceph-mon[29442]: from='client.44411 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T06:33:33.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:33 smithi028 ceph-mon[37984]: pgmap v197: 68 pgs: 12 creating+peering, 31 active+clean, 25 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-02-05T06:33:33.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:33 smithi028 ceph-mon[37984]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T06:33:33.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:33 smithi028 ceph-mon[37984]: Cluster is now healthy 2024-02-05T06:33:33.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:33 smithi028 ceph-mon[37984]: osdmap e69: 8 total, 8 up, 8 in 2024-02-05T06:33:33.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:33 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1937500867' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T06:33:33.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:33 smithi028 ceph-mon[37984]: from='client.44375 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T06:33:33.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:33 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3445405819' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T06:33:33.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:33 smithi028 ceph-mon[37984]: from='client.44411 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-05T06:33:34.864 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:34 smithi167 ceph-mon[32774]: from='client.44375 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-05T06:33:34.864 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:34 smithi167 ceph-mon[32774]: from='client.44411 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-05T06:33:34.864 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:34 smithi167 ceph-mon[32774]: osdmap e70: 8 total, 8 up, 8 in 2024-02-05T06:33:34.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:34 smithi028 ceph-mon[29442]: from='client.44375 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-05T06:33:34.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:34 smithi028 ceph-mon[29442]: from='client.44411 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-05T06:33:34.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:34 smithi028 ceph-mon[29442]: osdmap e70: 8 total, 8 up, 8 in 2024-02-05T06:33:34.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:34 smithi028 ceph-mon[37984]: from='client.44375 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-05T06:33:34.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:34 smithi028 ceph-mon[37984]: from='client.44411 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-05T06:33:34.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:34 smithi028 ceph-mon[37984]: osdmap e70: 8 total, 8 up, 8 in 2024-02-05T06:33:35.307 INFO:teuthology.orchestra.run.smithi167.stdout:Scheduled iscsi.datapool update... 2024-02-05T06:33:35.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:35 smithi028 ceph-mon[37984]: pgmap v200: 100 pgs: 12 creating+peering, 31 active+clean, 57 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-05T06:33:35.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:35 smithi028 ceph-mon[37984]: osdmap e71: 8 total, 8 up, 8 in 2024-02-05T06:33:35.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:35 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1937500867' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T06:33:35.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:35 smithi028 ceph-mon[37984]: from='client.44375 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T06:33:35.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:35 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3445405819' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T06:33:35.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:35 smithi028 ceph-mon[37984]: from='client.44411 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T06:33:35.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:35 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:35.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:35 smithi028 ceph-mon[29442]: pgmap v200: 100 pgs: 12 creating+peering, 31 active+clean, 57 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-05T06:33:35.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:35 smithi028 ceph-mon[29442]: osdmap e71: 8 total, 8 up, 8 in 2024-02-05T06:33:35.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:35 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1937500867' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T06:33:35.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:35 smithi028 ceph-mon[29442]: from='client.44375 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T06:33:35.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:35 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3445405819' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T06:33:35.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:35 smithi028 ceph-mon[29442]: from='client.44411 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T06:33:35.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:35 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:35.966 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:35 smithi167 ceph-mon[32774]: pgmap v200: 100 pgs: 12 creating+peering, 31 active+clean, 57 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-02-05T06:33:35.966 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:35 smithi167 ceph-mon[32774]: osdmap e71: 8 total, 8 up, 8 in 2024-02-05T06:33:35.966 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:35 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1937500867' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T06:33:35.966 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:35 smithi167 ceph-mon[32774]: from='client.44375 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T06:33:35.966 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:35 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3445405819' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T06:33:35.966 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:35 smithi167 ceph-mon[32774]: from='client.44411 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-05T06:33:35.966 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:35 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:35.995 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-02-05T06:33:35.995 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-05T06:33:35.995 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-02-05T06:33:36.031 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-02-05T06:33:36.031 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-02-05T06:33:36.061 DEBUG:teuthology.orchestra.run.smithi167:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@iscsi.iscsi.a.service 2024-02-05T06:33:36.103 INFO:tasks.cephadm:Adding prometheus.a on smithi167 2024-02-05T06:33:36.104 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph orch apply prometheus '1;smithi167=a' 2024-02-05T06:33:36.382 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:33:36.778 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:36 smithi167 ceph-mon[32774]: from='client.24424 v1:172.21.15.167:0/3717764512' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.167", "placement": "1;smithi167=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:33:36.778 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:36 smithi167 ceph-mon[32774]: Saving service iscsi.datapool spec with placement smithi167=iscsi.a;count:1 2024-02-05T06:33:36.778 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:36 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:33:36.779 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:36 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:33:36.779 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:36 smithi167 ceph-mon[32774]: from='client.44375 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-05T06:33:36.779 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:36 smithi167 ceph-mon[32774]: from='client.44411 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-05T06:33:36.779 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:36 smithi167 ceph-mon[32774]: osdmap e72: 8 total, 8 up, 8 in 2024-02-05T06:33:36.779 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:36 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3445405819' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T06:33:36.779 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:36 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1937500867' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T06:33:36.779 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:36 smithi167 ceph-mon[32774]: from='client.44411 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T06:33:36.779 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:36 smithi167 ceph-mon[32774]: from='client.44375 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T06:33:36.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:36 smithi028 ceph-mon[29442]: from='client.24424 v1:172.21.15.167:0/3717764512' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.167", "placement": "1;smithi167=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:33:36.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:36 smithi028 ceph-mon[29442]: Saving service iscsi.datapool spec with placement smithi167=iscsi.a;count:1 2024-02-05T06:33:36.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:36 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:33:36.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:36 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:33:36.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:36 smithi028 ceph-mon[29442]: from='client.44375 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-05T06:33:36.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:36 smithi028 ceph-mon[29442]: from='client.44411 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-05T06:33:36.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:36 smithi028 ceph-mon[29442]: osdmap e72: 8 total, 8 up, 8 in 2024-02-05T06:33:36.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:36 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3445405819' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T06:33:36.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:36 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1937500867' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T06:33:36.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:36 smithi028 ceph-mon[29442]: from='client.44411 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T06:33:36.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:36 smithi028 ceph-mon[29442]: from='client.44375 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T06:33:36.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:36 smithi028 ceph-mon[37984]: from='client.24424 v1:172.21.15.167:0/3717764512' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.167", "placement": "1;smithi167=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:33:36.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:36 smithi028 ceph-mon[37984]: Saving service iscsi.datapool spec with placement smithi167=iscsi.a;count:1 2024-02-05T06:33:36.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:36 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:33:36.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:36 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:33:36.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:36 smithi028 ceph-mon[37984]: from='client.44375 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-05T06:33:36.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:36 smithi028 ceph-mon[37984]: from='client.44411 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-05T06:33:36.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:36 smithi028 ceph-mon[37984]: osdmap e72: 8 total, 8 up, 8 in 2024-02-05T06:33:36.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:36 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3445405819' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T06:33:36.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:36 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1937500867' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T06:33:36.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:36 smithi028 ceph-mon[37984]: from='client.44411 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T06:33:36.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:36 smithi028 ceph-mon[37984]: from='client.44375 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-05T06:33:37.811 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:37 smithi167 ceph-mon[32774]: pgmap v203: 132 pgs: 15 creating+peering, 113 active+clean, 4 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1023 B/s wr, 5 op/s 2024-02-05T06:33:37.811 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:37 smithi167 ceph-mon[32774]: from='client.44411 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-05T06:33:37.811 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:37 smithi167 ceph-mon[32774]: from='client.44375 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-05T06:33:37.811 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:37 smithi167 ceph-mon[32774]: osdmap e73: 8 total, 8 up, 8 in 2024-02-05T06:33:37.811 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:37 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:37.812 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:37 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:37.812 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:37 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:37.812 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:37 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-05T06:33:37.812 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:37 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-05T06:33:37.812 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:37 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:33:37.812 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:37 smithi167 ceph-mon[32774]: Deploying daemon iscsi.iscsi.a on smithi167 2024-02-05T06:33:37.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:37 smithi028 ceph-mon[29442]: pgmap v203: 132 pgs: 15 creating+peering, 113 active+clean, 4 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1023 B/s wr, 5 op/s 2024-02-05T06:33:37.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:37 smithi028 ceph-mon[29442]: from='client.44411 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-05T06:33:37.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:37 smithi028 ceph-mon[29442]: from='client.44375 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-05T06:33:37.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:37 smithi028 ceph-mon[29442]: osdmap e73: 8 total, 8 up, 8 in 2024-02-05T06:33:37.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:37 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:37.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:37 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:37.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:37 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:37.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:37 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-05T06:33:37.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:37 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-05T06:33:37.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:37 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:33:37.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:37 smithi028 ceph-mon[29442]: Deploying daemon iscsi.iscsi.a on smithi167 2024-02-05T06:33:37.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:37 smithi028 ceph-mon[37984]: pgmap v203: 132 pgs: 15 creating+peering, 113 active+clean, 4 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1023 B/s wr, 5 op/s 2024-02-05T06:33:37.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:37 smithi028 ceph-mon[37984]: from='client.44411 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-05T06:33:37.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:37 smithi028 ceph-mon[37984]: from='client.44375 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-05T06:33:37.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:37 smithi028 ceph-mon[37984]: osdmap e73: 8 total, 8 up, 8 in 2024-02-05T06:33:37.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:37 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:37.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:37 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:37.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:37 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:37.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:37 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-02-05T06:33:37.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:37 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-02-05T06:33:37.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:37 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:33:37.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:37 smithi028 ceph-mon[37984]: Deploying daemon iscsi.iscsi.a on smithi167 2024-02-05T06:33:38.276 INFO:teuthology.orchestra.run.smithi167.stdout:Scheduled prometheus update... 2024-02-05T06:33:39.372 DEBUG:teuthology.orchestra.run.smithi167:prometheus.a> sudo journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@prometheus.a.service 2024-02-05T06:33:39.374 INFO:tasks.cephadm:Adding node-exporter.a on smithi028 2024-02-05T06:33:39.374 INFO:tasks.cephadm:Adding node-exporter.b on smithi167 2024-02-05T06:33:39.375 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph orch apply node-exporter '2;smithi028=a;smithi167=b' 2024-02-05T06:33:39.538 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:39 smithi167 ceph-mon[32774]: pgmap v205: 132 pgs: 15 creating+peering, 113 active+clean, 4 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 893 B/s wr, 4 op/s 2024-02-05T06:33:39.539 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:39 smithi167 ceph-mon[32774]: from='client.24445 v1:172.21.15.167:0/3201773859' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi167=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:33:39.539 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:39 smithi167 ceph-mon[32774]: Saving service prometheus spec with placement smithi167=a;count:1 2024-02-05T06:33:39.539 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:39 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:39.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:39 smithi028 ceph-mon[29442]: pgmap v205: 132 pgs: 15 creating+peering, 113 active+clean, 4 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 893 B/s wr, 4 op/s 2024-02-05T06:33:39.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:39 smithi028 ceph-mon[29442]: from='client.24445 v1:172.21.15.167:0/3201773859' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi167=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:33:39.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:39 smithi028 ceph-mon[29442]: Saving service prometheus spec with placement smithi167=a;count:1 2024-02-05T06:33:39.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:39 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:39.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:39 smithi028 ceph-mon[37984]: pgmap v205: 132 pgs: 15 creating+peering, 113 active+clean, 4 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 893 B/s wr, 4 op/s 2024-02-05T06:33:39.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:39 smithi028 ceph-mon[37984]: from='client.24445 v1:172.21.15.167:0/3201773859' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi167=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:33:39.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:39 smithi028 ceph-mon[37984]: Saving service prometheus spec with placement smithi167=a;count:1 2024-02-05T06:33:39.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:39 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:39.987 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:33:40.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:40 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:33:40.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:40 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:33:40.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:40 smithi028 ceph-mon[29442]: osdmap e74: 8 total, 8 up, 8 in 2024-02-05T06:33:40.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:40 smithi028 ceph-mon[29442]: osdmap e75: 8 total, 8 up, 8 in 2024-02-05T06:33:40.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:40 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:33:40.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:40 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:33:40.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:40 smithi028 ceph-mon[37984]: osdmap e74: 8 total, 8 up, 8 in 2024-02-05T06:33:40.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:40 smithi028 ceph-mon[37984]: osdmap e75: 8 total, 8 up, 8 in 2024-02-05T06:33:40.728 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:40 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:33:40.728 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:40 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:33:40.728 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:40 smithi167 ceph-mon[32774]: osdmap e74: 8 total, 8 up, 8 in 2024-02-05T06:33:40.728 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:40 smithi167 ceph-mon[32774]: osdmap e75: 8 total, 8 up, 8 in 2024-02-05T06:33:41.725 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:33:41 smithi167 podman[60231]: 2024-02-05 06:33:41.58466367 +0000 UTC m=+0.360102441 container create d9968e61e530f0c517bc3150699205f2c9a7f47dbb44f66010c26b32260181e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-05T06:33:41.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:41 smithi167 ceph-mon[32774]: pgmap v208: 132 pgs: 2 creating+peering, 130 active+clean; 581 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 5.7 KiB/s wr, 185 op/s 2024-02-05T06:33:42.057 INFO:teuthology.orchestra.run.smithi167.stdout:Scheduled node-exporter update... 2024-02-05T06:33:42.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:41 smithi028 ceph-mon[29442]: pgmap v208: 132 pgs: 2 creating+peering, 130 active+clean; 581 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 5.7 KiB/s wr, 185 op/s 2024-02-05T06:33:42.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:41 smithi028 ceph-mon[37984]: pgmap v208: 132 pgs: 2 creating+peering, 130 active+clean; 581 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 5.7 KiB/s wr, 185 op/s 2024-02-05T06:33:42.311 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:33:42 smithi167 podman[60231]: 2024-02-05 06:33:42.038506574 +0000 UTC m=+0.813945348 container init d9968e61e530f0c517bc3150699205f2c9a7f47dbb44f66010c26b32260181e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.vendor=CentOS) 2024-02-05T06:33:42.311 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:33:42 smithi167 podman[60231]: 2024-02-05 06:33:42.041668076 +0000 UTC m=+0.817106851 container start d9968e61e530f0c517bc3150699205f2c9a7f47dbb44f66010c26b32260181e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_CLEAN=True) 2024-02-05T06:33:42.311 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:33:42 smithi167 bash[60231]: d9968e61e530f0c517bc3150699205f2c9a7f47dbb44f66010c26b32260181e7 2024-02-05T06:33:42.311 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:33:42 smithi167 systemd[1]: Started Ceph iscsi.iscsi.a for 9801e956-c3ef-11ee-95b6-87774f69a715. 2024-02-05T06:33:42.656 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:33:42 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug Started the configuration object watcher 2024-02-05T06:33:42.656 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:33:42 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug Checking for config object changes every 1s 2024-02-05T06:33:42.656 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:33:42 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug Processing osd blocklist entries for this node 2024-02-05T06:33:42.656 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:33:42 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug Reading the configuration object to update local LIO configuration 2024-02-05T06:33:42.656 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:33:42 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug Configuration does not have an entry for this host(smithi167.front.sepia.ceph.com) - nothing to define to LIO 2024-02-05T06:33:42.754 DEBUG:teuthology.orchestra.run.smithi028:node-exporter.a> sudo journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@node-exporter.a.service 2024-02-05T06:33:42.757 DEBUG:teuthology.orchestra.run.smithi167:node-exporter.b> sudo journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@node-exporter.b.service 2024-02-05T06:33:42.759 INFO:tasks.cephadm:Adding alertmanager.a on smithi028 2024-02-05T06:33:42.759 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph orch apply alertmanager '1;smithi028=a' 2024-02-05T06:33:42.970 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:33:42 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-02-05T06:33:43.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:43 smithi167 ceph-mon[32774]: from='client.24451 v1:172.21.15.167:0/2785057831' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi028=a;smithi167=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:33:43.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:43 smithi167 ceph-mon[32774]: Saving service node-exporter spec with placement smithi028=a;smithi167=b;count:2 2024-02-05T06:33:43.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:43 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:43.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:43 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:43.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:43 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:43.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:43 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:43.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:43 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:43.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:43 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:33:43.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:43 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.167:0/3765583281' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-05T06:33:43.369 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:33:43.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:43 smithi028 ceph-mon[29442]: from='client.24451 v1:172.21.15.167:0/2785057831' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi028=a;smithi167=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:33:43.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:43 smithi028 ceph-mon[29442]: Saving service node-exporter spec with placement smithi028=a;smithi167=b;count:2 2024-02-05T06:33:43.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:43 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:43.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:43 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:43.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:43 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:43.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:43 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:43.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:43 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:43.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:43 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:33:43.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:43 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.167:0/3765583281' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-05T06:33:43.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:43 smithi028 ceph-mon[37984]: from='client.24451 v1:172.21.15.167:0/2785057831' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi028=a;smithi167=b", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:33:43.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:43 smithi028 ceph-mon[37984]: Saving service node-exporter spec with placement smithi028=a;smithi167=b;count:2 2024-02-05T06:33:43.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:43 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:43.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:43 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:43.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:43 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:43.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:43 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:43.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:43 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:43.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:43 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:33:43.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:43 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.167:0/3765583281' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-05T06:33:44.346 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:44 smithi167 ceph-mon[32774]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 5.7 KiB/s wr, 185 op/s 2024-02-05T06:33:44.346 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:44 smithi167 ceph-mon[32774]: Checking pool "datapool" exists for service iscsi.datapool 2024-02-05T06:33:44.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:44 smithi028 ceph-mon[29442]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 5.7 KiB/s wr, 185 op/s 2024-02-05T06:33:44.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:44 smithi028 ceph-mon[29442]: Checking pool "datapool" exists for service iscsi.datapool 2024-02-05T06:33:44.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:44 smithi028 ceph-mon[37984]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 5.7 KiB/s wr, 185 op/s 2024-02-05T06:33:44.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:44 smithi028 ceph-mon[37984]: Checking pool "datapool" exists for service iscsi.datapool 2024-02-05T06:33:45.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:45 smithi167 ceph-mon[32774]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-05T06:33:45.224 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:45 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:45.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:45 smithi028 ceph-mon[29442]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-05T06:33:45.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:45 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:45.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:45 smithi028 ceph-mon[37984]: mgrmap e16: y(active, since 4m), standbys: x 2024-02-05T06:33:45.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:45 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:46.020 INFO:teuthology.orchestra.run.smithi167.stdout:Scheduled alertmanager update... 2024-02-05T06:33:46.425 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:46 smithi167 ceph-mon[32774]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 4.5 KiB/s wr, 147 op/s 2024-02-05T06:33:46.425 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:46 smithi167 ceph-mon[32774]: 1.0 scrub starts 2024-02-05T06:33:46.425 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:46 smithi167 ceph-mon[32774]: 1.0 scrub ok 2024-02-05T06:33:46.425 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:46 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:46.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:46 smithi028 ceph-mon[29442]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 4.5 KiB/s wr, 147 op/s 2024-02-05T06:33:46.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:46 smithi028 ceph-mon[29442]: 1.0 scrub starts 2024-02-05T06:33:46.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:46 smithi028 ceph-mon[29442]: 1.0 scrub ok 2024-02-05T06:33:46.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:46 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:46.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:46 smithi028 ceph-mon[37984]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 4.5 KiB/s wr, 147 op/s 2024-02-05T06:33:46.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:46 smithi028 ceph-mon[37984]: 1.0 scrub starts 2024-02-05T06:33:46.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:46 smithi028 ceph-mon[37984]: 1.0 scrub ok 2024-02-05T06:33:46.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:46 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:46.711 DEBUG:teuthology.orchestra.run.smithi028:alertmanager.a> sudo journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@alertmanager.a.service 2024-02-05T06:33:46.713 INFO:tasks.cephadm:Adding grafana.a on smithi167 2024-02-05T06:33:46.714 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph orch apply grafana '1;smithi167=a' 2024-02-05T06:33:47.084 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:33:47.334 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:47 smithi167 ceph-mon[32774]: from='client.24487 v1:172.21.15.167:0/978662954' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi028=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:33:47.334 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:47 smithi167 ceph-mon[32774]: Saving service alertmanager spec with placement smithi028=a;count:1 2024-02-05T06:33:47.334 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:47 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:47.334 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:47 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:47.334 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:47 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:33:47.334 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:47 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:33:47.334 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:47 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:47.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:47 smithi028 ceph-mon[37984]: from='client.24487 v1:172.21.15.167:0/978662954' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi028=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:33:47.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:47 smithi028 ceph-mon[37984]: Saving service alertmanager spec with placement smithi028=a;count:1 2024-02-05T06:33:47.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:47 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:47.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:47 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:47.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:47 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:33:47.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:47 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:33:47.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:47 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:47.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:47 smithi028 ceph-mon[29442]: from='client.24487 v1:172.21.15.167:0/978662954' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi028=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:33:47.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:47 smithi028 ceph-mon[29442]: Saving service alertmanager spec with placement smithi028=a;count:1 2024-02-05T06:33:47.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:47 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:47.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:47 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:47.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:47 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:33:47.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:47 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:33:47.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:47 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:48.379 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:48 smithi167 ceph-mon[32774]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 140 op/s 2024-02-05T06:33:48.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:48 smithi028 ceph-mon[29442]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 140 op/s 2024-02-05T06:33:48.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:48 smithi028 ceph-mon[37984]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.4 KiB/s wr, 140 op/s 2024-02-05T06:33:48.997 INFO:teuthology.orchestra.run.smithi167.stdout:Scheduled grafana update... 2024-02-05T06:33:49.308 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:49 smithi167 ceph-mon[32774]: Deploying daemon prometheus.a on smithi167 2024-02-05T06:33:49.308 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:49 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:49.344 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:49 smithi028 ceph-mon[29442]: Deploying daemon prometheus.a on smithi167 2024-02-05T06:33:49.344 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:49 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:49.344 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:49 smithi028 ceph-mon[37984]: Deploying daemon prometheus.a on smithi167 2024-02-05T06:33:49.344 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:49 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:49.797 DEBUG:teuthology.orchestra.run.smithi167:grafana.a> sudo journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@grafana.a.service 2024-02-05T06:33:49.800 INFO:tasks.cephadm:Setting up client nodes... 2024-02-05T06:33:49.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-05T06:33:50.046 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:33:50.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:50 smithi028 ceph-mon[37984]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 3.9 KiB/s wr, 123 op/s 2024-02-05T06:33:50.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:50 smithi028 ceph-mon[37984]: from='client.24493 v1:172.21.15.167:0/2749495106' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi167=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:33:50.438 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:50 smithi028 ceph-mon[37984]: Saving service grafana spec with placement smithi167=a;count:1 2024-02-05T06:33:50.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:50 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:50.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:50 smithi028 ceph-mon[29442]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 3.9 KiB/s wr, 123 op/s 2024-02-05T06:33:50.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:50 smithi028 ceph-mon[29442]: from='client.24493 v1:172.21.15.167:0/2749495106' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi167=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:33:50.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:50 smithi028 ceph-mon[29442]: Saving service grafana spec with placement smithi167=a;count:1 2024-02-05T06:33:50.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:50 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:50.477 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:50 smithi167 ceph-mon[32774]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 3.9 KiB/s wr, 123 op/s 2024-02-05T06:33:50.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:50 smithi167 ceph-mon[32774]: from='client.24493 v1:172.21.15.167:0/2749495106' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi167=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:33:50.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:50 smithi167 ceph-mon[32774]: Saving service grafana spec with placement smithi167=a;count:1 2024-02-05T06:33:50.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:50 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:33:52.036 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-02-05T06:33:52.036 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBQgcBlbKYIAhAAWrjJDb4rrX/JCviOwzUOeA== 2024-02-05T06:33:52.390 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:52 smithi028 ceph-mon[29442]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 2.2 KiB/s wr, 64 op/s 2024-02-05T06:33:52.390 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:52 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2064009259' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T06:33:52.390 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:52 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2064009259' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-05T06:33:52.390 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:52 smithi028 ceph-mon[37984]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 2.2 KiB/s wr, 64 op/s 2024-02-05T06:33:52.390 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:52 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2064009259' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T06:33:52.391 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:52 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2064009259' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-05T06:33:52.478 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:33:52 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:33:52.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:52 smithi167 ceph-mon[32774]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 2.2 KiB/s wr, 64 op/s 2024-02-05T06:33:52.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:52 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2064009259' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T06:33:52.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:52 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2064009259' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-05T06:33:52.685 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-02-05T06:33:52.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-05T06:33:52.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-05T06:33:52.729 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-05T06:33:52.978 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:33:54.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:54 smithi028 ceph-mon[29442]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 9.8 KiB/s rd, 767 B/s wr, 23 op/s 2024-02-05T06:33:54.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:54 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:33:54.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:54 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:33:54.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:54 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:33:54.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:54 smithi028 ceph-mon[37984]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 9.8 KiB/s rd, 767 B/s wr, 23 op/s 2024-02-05T06:33:54.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:54 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:33:54.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:54 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:33:54.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:54 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:33:54.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:54 smithi167 ceph-mon[32774]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 9.8 KiB/s rd, 767 B/s wr, 23 op/s 2024-02-05T06:33:54.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:54 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:33:54.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:54 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:33:54.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:54 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:33:55.356 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:55 smithi167 ceph-mon[32774]: osdmap e76: 8 total, 8 up, 8 in 2024-02-05T06:33:55.356 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:55 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-02-05T06:33:55.357 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:55 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-02-05T06:33:55.357 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:55 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-02-05T06:33:55.357 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:55 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-02-05T06:33:55.357 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:55 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2024-02-05T06:33:55.357 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:55 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-02-05T06:33:55.357 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:55 smithi167 ceph-mon[32774]: osdmap e77: 8 total, 8 up, 8 in 2024-02-05T06:33:55.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:55 smithi028 ceph-mon[29442]: osdmap e76: 8 total, 8 up, 8 in 2024-02-05T06:33:55.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:55 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-02-05T06:33:55.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:55 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-02-05T06:33:55.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:55 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-02-05T06:33:55.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:55 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-02-05T06:33:55.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:55 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2024-02-05T06:33:55.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:55 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-02-05T06:33:55.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:55 smithi028 ceph-mon[29442]: osdmap e77: 8 total, 8 up, 8 in 2024-02-05T06:33:55.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:55 smithi028 ceph-mon[37984]: osdmap e76: 8 total, 8 up, 8 in 2024-02-05T06:33:55.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:55 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-02-05T06:33:55.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:55 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]: dispatch 2024-02-05T06:33:55.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:55 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-02-05T06:33:55.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:55 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-02-05T06:33:55.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:55 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 2]}]': finished 2024-02-05T06:33:55.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:55 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-02-05T06:33:55.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:55 smithi028 ceph-mon[37984]: osdmap e77: 8 total, 8 up, 8 in 2024-02-05T06:33:56.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:56 smithi167 ceph-mon[32774]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 102 B/s wr, 1 op/s 2024-02-05T06:33:56.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:56 smithi167 ceph-mon[32774]: osdmap e78: 8 total, 8 up, 8 in 2024-02-05T06:33:56.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:56 smithi028 ceph-mon[29442]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 102 B/s wr, 1 op/s 2024-02-05T06:33:56.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:56 smithi028 ceph-mon[29442]: osdmap e78: 8 total, 8 up, 8 in 2024-02-05T06:33:56.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:56 smithi028 ceph-mon[37984]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 102 B/s wr, 1 op/s 2024-02-05T06:33:56.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:56 smithi028 ceph-mon[37984]: osdmap e78: 8 total, 8 up, 8 in 2024-02-05T06:33:57.367 INFO:teuthology.orchestra.run.smithi167.stdout:[client.1] 2024-02-05T06:33:57.368 INFO:teuthology.orchestra.run.smithi167.stdout: key = AQBVgcBlxqmqFRAATGrz+jJb6AjwcywB9P/l2Q== 2024-02-05T06:33:58.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:58 smithi028 ceph-mon[29442]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 36 B/s, 0 objects/s recovering 2024-02-05T06:33:58.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:58 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.167:0/2702598900' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T06:33:58.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:58 smithi028 ceph-mon[29442]: from='client.24505 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T06:33:58.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:33:58 smithi028 ceph-mon[29442]: from='client.24505 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-05T06:33:58.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:58 smithi028 ceph-mon[37984]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 36 B/s, 0 objects/s recovering 2024-02-05T06:33:58.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:58 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.167:0/2702598900' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T06:33:58.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:58 smithi028 ceph-mon[37984]: from='client.24505 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T06:33:58.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:33:58 smithi028 ceph-mon[37984]: from='client.24505 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-05T06:33:58.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:58 smithi167 ceph-mon[32774]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 36 B/s, 0 objects/s recovering 2024-02-05T06:33:58.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:58 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.167:0/2702598900' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T06:33:58.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:58 smithi167 ceph-mon[32774]: from='client.24505 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-05T06:33:58.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:33:58 smithi167 ceph-mon[32774]: from='client.24505 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-05T06:33:58.697 DEBUG:teuthology.orchestra.run.smithi167:> set -ex 2024-02-05T06:33:58.697 DEBUG:teuthology.orchestra.run.smithi167:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-02-05T06:33:58.697 DEBUG:teuthology.orchestra.run.smithi167:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-05T06:33:58.741 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-05T06:33:58.741 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-05T06:33:58.742 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph mgr dump --format=json 2024-02-05T06:33:58.995 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:34:00.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:00 smithi028 ceph-mon[29442]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 36 B/s, 0 objects/s recovering 2024-02-05T06:34:00.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:00 smithi028 ceph-mon[29442]: osdmap e79: 8 total, 8 up, 8 in 2024-02-05T06:34:00.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:00 smithi028 ceph-mon[37984]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 36 B/s, 0 objects/s recovering 2024-02-05T06:34:00.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:00 smithi028 ceph-mon[37984]: osdmap e79: 8 total, 8 up, 8 in 2024-02-05T06:34:00.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:00 smithi167 ceph-mon[32774]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 36 B/s, 0 objects/s recovering 2024-02-05T06:34:00.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:00 smithi167 ceph-mon[32774]: osdmap e79: 8 total, 8 up, 8 in 2024-02-05T06:34:01.069 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:34:01.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:01 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3943652631' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-05T06:34:01.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:01 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3943652631' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-05T06:34:01.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:01 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3943652631' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-05T06:34:01.799 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":16,"flags":0,"active_gid":14150,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6800","nonce":1001442173}]},"active_addr":"172.21.15.28:6800/1001442173","active_change":"2024-02-05T06:28:54.105328+0000","active_mgr_features":4540701272860131327,"available":true,"standbys":[{"gid":24115,"name":"x","mgr_features":4540701272860131327,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.28:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":3160026720}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":1560471317}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":558171292}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":2832437766}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":3149394045}]}]} 2024-02-05T06:34:01.803 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-05T06:34:01.803 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-05T06:34:01.804 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd dump --format=json 2024-02-05T06:34:01.978 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:34:01 smithi167 systemd[1]: Started Ceph prometheus.a for 9801e956-c3ef-11ee-95b6-87774f69a715. 2024-02-05T06:34:02.047 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:34:02.425 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:02 smithi028 ceph-mon[29442]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 114 B/s, 3 objects/s recovering 2024-02-05T06:34:02.425 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:02 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:34:02.425 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:02 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:34:02.426 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:02 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:34:02.426 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:02 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-05T06:34:02.426 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:02 smithi028 ceph-mon[37984]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 114 B/s, 3 objects/s recovering 2024-02-05T06:34:02.426 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:02 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:34:02.426 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:02 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:34:02.426 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:02 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:34:02.426 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:02 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-05T06:34:02.478 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:34:02 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:34:02.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:02 smithi167 ceph-mon[32774]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 114 B/s, 3 objects/s recovering 2024-02-05T06:34:02.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:02 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:34:02.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:02 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:34:02.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:02 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' 2024-02-05T06:34:02.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:02 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-05T06:34:03.193 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:02 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ignoring --setuser ceph since I am not root 2024-02-05T06:34:03.193 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:02 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ignoring --setgroup ceph since I am not root 2024-02-05T06:34:03.228 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:02 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: ignoring --setuser ceph since I am not root 2024-02-05T06:34:03.247 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:02 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: ignoring --setgroup ceph since I am not root 2024-02-05T06:34:03.624 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:03 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:34:03.293+0000 7f65778ff200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-05T06:34:03.624 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:03 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:34:03.395+0000 7f65778ff200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-05T06:34:03.713 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:03 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:34:03.277+0000 7f585587f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-05T06:34:03.713 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:03 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:34:03.370+0000 7f585587f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-05T06:34:03.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:03 smithi028 ceph-mon[29442]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-05T06:34:03.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:03 smithi028 ceph-mon[29442]: mgrmap e17: y(active, since 5m), standbys: x 2024-02-05T06:34:03.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:03 smithi028 ceph-mon[37984]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-05T06:34:03.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:03 smithi028 ceph-mon[37984]: mgrmap e17: y(active, since 5m), standbys: x 2024-02-05T06:34:03.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:03 smithi167 ceph-mon[32774]: from='mgr.14150 v1:172.21.15.28:0/4013953509' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-05T06:34:03.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:03 smithi167 ceph-mon[32774]: mgrmap e17: y(active, since 5m), standbys: x 2024-02-05T06:34:04.085 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:34:04.086 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":79,"fsid":"9801e956-c3ef-11ee-95b6-87774f69a715","created":"2024-02-05T06:27:59.919226+0000","modified":"2024-02-05T06:33:59.874654+0000","last_up_change":"2024-02-05T06:33:15.833573+0000","last_in_change":"2024-02-05T06:32:57.809784+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-05T06:31:10.441424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-02-05T06:33:27.397416+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":69,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-02-05T06:33:28.090506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-05T06:33:29.866249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-05T06:33:31.826006+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-05T06:33:33.890835+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c19fdd5b-09f6-45d4-a519-0a938cf1f2a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6801","nonce":301279106}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6802","nonce":301279106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6804","nonce":301279106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6803","nonce":301279106}]},"public_addr":"172.21.15.28:6801/301279106","cluster_addr":"172.21.15.28:6802/301279106","heartbeat_back_addr":"172.21.15.28:6804/301279106","heartbeat_front_addr":"172.21.15.28:6803/301279106","state":["exists","up"]},{"osd":1,"uuid":"52245f6e-eb96-4d0d-985b-dbb6fd3ce8de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6805","nonce":1795224649}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6806","nonce":1795224649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6808","nonce":1795224649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6807","nonce":1795224649}]},"public_addr":"172.21.15.28:6805/1795224649","cluster_addr":"172.21.15.28:6806/1795224649","heartbeat_back_addr":"172.21.15.28:6808/1795224649","heartbeat_front_addr":"172.21.15.28:6807/1795224649","state":["exists","up"]},{"osd":2,"uuid":"1e18b630-afdf-4de9-a1a2-37bdfdd01c67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6809","nonce":1993280018}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6810","nonce":1993280018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6812","nonce":1993280018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6811","nonce":1993280018}]},"public_addr":"172.21.15.28:6809/1993280018","cluster_addr":"172.21.15.28:6810/1993280018","heartbeat_back_addr":"172.21.15.28:6812/1993280018","heartbeat_front_addr":"172.21.15.28:6811/1993280018","state":["exists","up"]},{"osd":3,"uuid":"3ae795af-3d59-41a6-95eb-b8447ff61282","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6813","nonce":1294621398}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6814","nonce":1294621398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6816","nonce":1294621398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6815","nonce":1294621398}]},"public_addr":"172.21.15.28:6813/1294621398","cluster_addr":"172.21.15.28:6814/1294621398","heartbeat_back_addr":"172.21.15.28:6816/1294621398","heartbeat_front_addr":"172.21.15.28:6815/1294621398","state":["exists","up"]},{"osd":4,"uuid":"fc6bd83d-570c-48c0-964d-e09b4608bc32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6800","nonce":4052090595}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6801","nonce":4052090595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6803","nonce":4052090595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6802","nonce":4052090595}]},"public_addr":"172.21.15.167:6800/4052090595","cluster_addr":"172.21.15.167:6801/4052090595","heartbeat_back_addr":"172.21.15.167:6803/4052090595","heartbeat_front_addr":"172.21.15.167:6802/4052090595","state":["exists","up"]},{"osd":5,"uuid":"895bf465-2531-4d9f-bc23-1e75fd969f07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6804","nonce":1631080500}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6805","nonce":1631080500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6807","nonce":1631080500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6806","nonce":1631080500}]},"public_addr":"172.21.15.167:6804/1631080500","cluster_addr":"172.21.15.167:6805/1631080500","heartbeat_back_addr":"172.21.15.167:6807/1631080500","heartbeat_front_addr":"172.21.15.167:6806/1631080500","state":["exists","up"]},{"osd":6,"uuid":"ce0a9300-87a4-4e37-b702-d667f168c0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6808","nonce":1644050617}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6809","nonce":1644050617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6811","nonce":1644050617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6810","nonce":1644050617}]},"public_addr":"172.21.15.167:6808/1644050617","cluster_addr":"172.21.15.167:6809/1644050617","heartbeat_back_addr":"172.21.15.167:6811/1644050617","heartbeat_front_addr":"172.21.15.167:6810/1644050617","state":["exists","up"]},{"osd":7,"uuid":"0c545cbc-0873-4232-8807-4af9299ef95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6812","nonce":4031475764}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6813","nonce":4031475764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6815","nonce":4031475764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6814","nonce":4031475764}]},"public_addr":"172.21.15.167:6812/4031475764","cluster_addr":"172.21.15.167:6813/4031475764","heartbeat_back_addr":"172.21.15.167:6815/4031475764","heartbeat_front_addr":"172.21.15.167:6814/4031475764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:33:53.455356+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:33:03.970578+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:33:38.637484+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:32:52.894807+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:33:23.934837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:33:35.448241+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:32:47.949781+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:33:13.931286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.e","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/4125764599":"2024-02-06T06:28:54.105170+0000","172.21.15.28:0/1081543653":"2024-02-06T06:28:54.105170+0000","172.21.15.28:0/3794420554":"2024-02-06T06:28:27.553034+0000","172.21.15.28:6800/4266601437":"2024-02-06T06:28:27.553034+0000","172.21.15.28:0/388893831":"2024-02-06T06:28:27.553034+0000","172.21.15.28:0/460817910":"2024-02-06T06:28:27.553034+0000","172.21.15.28:0/2728305743":"2024-02-06T06:28:54.105170+0000","172.21.15.28:0/3605013594":"2024-02-06T06:28:54.105170+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-05T06:34:04.327 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:04 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:34:04.075+0000 7f65778ff200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-05T06:34:04.328 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:04 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:34:04.179+0000 7f65778ff200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-05T06:34:04.328 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:04 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:34:04.325+0000 7f65778ff200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-05T06:34:04.342 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:04 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:34:04.012+0000 7f585587f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-05T06:34:04.342 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:04 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:34:04.103+0000 7f585587f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-05T06:34:04.342 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:04 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:34:04.245+0000 7f585587f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-05T06:34:04.342 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:04 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:34:04.341+0000 7f585587f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-05T06:34:04.689 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:04 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:34:04.425+0000 7f585587f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-05T06:34:04.689 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:04 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:34:04.687+0000 7f585587f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-05T06:34:04.693 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:04 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:34:04.426+0000 7f65778ff200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-05T06:34:04.693 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:04 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:34:04.511+0000 7f65778ff200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-05T06:34:04.831 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-05T06:34:04.832 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd dump --format=json 2024-02-05T06:34:04.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:04 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2996945457' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T06:34:04.978 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:04 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:34:04.799+0000 7f585587f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-05T06:34:04.978 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:04 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:34:04.886+0000 7f585587f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-05T06:34:04.999 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:04 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2996945457' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T06:34:04.999 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:04 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2996945457' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T06:34:05.000 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:04 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:34:04.783+0000 7f65778ff200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-05T06:34:05.000 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:04 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:34:04.906+0000 7f65778ff200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-05T06:34:05.082 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:34:05.291 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:04 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:34:04.997+0000 7f65778ff200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-05T06:34:05.292 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:05 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:34:05.105+0000 7f65778ff200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-05T06:34:05.292 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:05 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:34:05.289+0000 7f65778ff200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-05T06:34:05.360 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:04 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:34:04.982+0000 7f585587f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-05T06:34:05.361 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:34:05.160+0000 7f585587f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-05T06:34:05.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:05 smithi167 ceph-mon[32774]: osdmap e80: 8 total, 8 up, 8 in 2024-02-05T06:34:06.144 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:05 smithi028 ceph-mon[29442]: osdmap e80: 8 total, 8 up, 8 in 2024-02-05T06:34:06.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:05 smithi028 ceph-mon[37984]: osdmap e80: 8 total, 8 up, 8 in 2024-02-05T06:34:06.443 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:06 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:34:06.306+0000 7f65778ff200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-05T06:34:06.478 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:34:06.180+0000 7f585587f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-05T06:34:06.478 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:34:06.315+0000 7f585587f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-05T06:34:06.723 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:06 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:34:06.443+0000 7f65778ff200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-05T06:34:06.886 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:34:06.886 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":80,"fsid":"9801e956-c3ef-11ee-95b6-87774f69a715","created":"2024-02-05T06:27:59.919226+0000","modified":"2024-02-05T06:34:04.876198+0000","last_up_change":"2024-02-05T06:33:15.833573+0000","last_in_change":"2024-02-05T06:32:57.809784+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-05T06:31:10.441424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-02-05T06:33:27.397416+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":69,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-02-05T06:33:28.090506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-05T06:33:29.866249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-05T06:33:31.826006+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-05T06:33:33.890835+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c19fdd5b-09f6-45d4-a519-0a938cf1f2a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6801","nonce":301279106}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6802","nonce":301279106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6804","nonce":301279106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6803","nonce":301279106}]},"public_addr":"172.21.15.28:6801/301279106","cluster_addr":"172.21.15.28:6802/301279106","heartbeat_back_addr":"172.21.15.28:6804/301279106","heartbeat_front_addr":"172.21.15.28:6803/301279106","state":["exists","up"]},{"osd":1,"uuid":"52245f6e-eb96-4d0d-985b-dbb6fd3ce8de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6805","nonce":1795224649}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6806","nonce":1795224649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6808","nonce":1795224649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6807","nonce":1795224649}]},"public_addr":"172.21.15.28:6805/1795224649","cluster_addr":"172.21.15.28:6806/1795224649","heartbeat_back_addr":"172.21.15.28:6808/1795224649","heartbeat_front_addr":"172.21.15.28:6807/1795224649","state":["exists","up"]},{"osd":2,"uuid":"1e18b630-afdf-4de9-a1a2-37bdfdd01c67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6809","nonce":1993280018}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6810","nonce":1993280018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6812","nonce":1993280018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6811","nonce":1993280018}]},"public_addr":"172.21.15.28:6809/1993280018","cluster_addr":"172.21.15.28:6810/1993280018","heartbeat_back_addr":"172.21.15.28:6812/1993280018","heartbeat_front_addr":"172.21.15.28:6811/1993280018","state":["exists","up"]},{"osd":3,"uuid":"3ae795af-3d59-41a6-95eb-b8447ff61282","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6813","nonce":1294621398}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6814","nonce":1294621398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6816","nonce":1294621398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6815","nonce":1294621398}]},"public_addr":"172.21.15.28:6813/1294621398","cluster_addr":"172.21.15.28:6814/1294621398","heartbeat_back_addr":"172.21.15.28:6816/1294621398","heartbeat_front_addr":"172.21.15.28:6815/1294621398","state":["exists","up"]},{"osd":4,"uuid":"fc6bd83d-570c-48c0-964d-e09b4608bc32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6800","nonce":4052090595}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6801","nonce":4052090595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6803","nonce":4052090595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6802","nonce":4052090595}]},"public_addr":"172.21.15.167:6800/4052090595","cluster_addr":"172.21.15.167:6801/4052090595","heartbeat_back_addr":"172.21.15.167:6803/4052090595","heartbeat_front_addr":"172.21.15.167:6802/4052090595","state":["exists","up"]},{"osd":5,"uuid":"895bf465-2531-4d9f-bc23-1e75fd969f07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":77,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6804","nonce":1631080500}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6805","nonce":1631080500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6807","nonce":1631080500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6806","nonce":1631080500}]},"public_addr":"172.21.15.167:6804/1631080500","cluster_addr":"172.21.15.167:6805/1631080500","heartbeat_back_addr":"172.21.15.167:6807/1631080500","heartbeat_front_addr":"172.21.15.167:6806/1631080500","state":["exists","up"]},{"osd":6,"uuid":"ce0a9300-87a4-4e37-b702-d667f168c0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6808","nonce":1644050617}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6809","nonce":1644050617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6811","nonce":1644050617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6810","nonce":1644050617}]},"public_addr":"172.21.15.167:6808/1644050617","cluster_addr":"172.21.15.167:6809/1644050617","heartbeat_back_addr":"172.21.15.167:6811/1644050617","heartbeat_front_addr":"172.21.15.167:6810/1644050617","state":["exists","up"]},{"osd":7,"uuid":"0c545cbc-0873-4232-8807-4af9299ef95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6812","nonce":4031475764}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6813","nonce":4031475764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6815","nonce":4031475764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6814","nonce":4031475764}]},"public_addr":"172.21.15.167:6812/4031475764","cluster_addr":"172.21.15.167:6813/4031475764","heartbeat_back_addr":"172.21.15.167:6815/4031475764","heartbeat_front_addr":"172.21.15.167:6814/4031475764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:33:53.455356+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:33:03.970578+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:33:38.637484+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:32:52.894807+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:33:23.934837+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:33:35.448241+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:32:47.949781+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:33:13.931286+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.e","mappings":[{"from":1,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/4125764599":"2024-02-06T06:28:54.105170+0000","172.21.15.28:0/1081543653":"2024-02-06T06:28:54.105170+0000","172.21.15.28:0/3794420554":"2024-02-06T06:28:27.553034+0000","172.21.15.28:6800/4266601437":"2024-02-06T06:28:27.553034+0000","172.21.15.28:0/388893831":"2024-02-06T06:28:27.553034+0000","172.21.15.28:0/460817910":"2024-02-06T06:28:27.553034+0000","172.21.15.28:0/2728305743":"2024-02-06T06:28:54.105170+0000","172.21.15.28:0/3605013594":"2024-02-06T06:28:54.105170+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-05T06:34:06.937 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:34:06.658+0000 7f585587f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-05T06:34:06.937 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:34:06.749+0000 7f585587f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-05T06:34:07.096 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:06 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2683383271' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T06:34:07.096 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:06 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2683383271' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T06:34:07.097 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:06 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:34:06.793+0000 7f65778ff200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-05T06:34:07.097 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:06 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:34:06.888+0000 7f65778ff200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-05T06:34:07.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:06 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2683383271' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T06:34:07.228 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:34:06.947+0000 7f585587f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-05T06:34:07.228 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:07 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:34:07.134+0000 7f585587f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-05T06:34:07.356 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:07 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:34:07.094+0000 7f65778ff200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-05T06:34:07.356 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:07 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:34:07.289+0000 7f65778ff200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-05T06:34:07.607 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-05T06:34:07.608 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-05T06:34:07.608 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-05T06:34:07.608 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-05T06:34:07.609 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-05T06:34:07.609 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-05T06:34:07.610 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-05T06:34:07.610 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-05T06:34:07.647 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:07 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:34:07.586+0000 7f65778ff200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-05T06:34:07.687 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:07 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:34:07.422+0000 7f585587f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-05T06:34:07.978 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:07 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:34:07.686+0000 7f585587f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-05T06:34:07.978 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:07 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:34:07.924+0000 7f585587f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-05T06:34:08.075 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:34:08.143 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:34:08.144 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:34:08.145 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:34:08.162 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:34:08.277 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:07 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:34:07.995+0000 7f65778ff200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-05T06:34:08.547 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:08 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:34:08.320+0000 7f65778ff200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-05T06:34:09.228 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:08 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:34:08.728+0000 7f585587f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-05T06:34:09.443 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:09 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:34:09.172+0000 7f65778ff200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-05T06:34:09.628 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:09 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:34:09.307+0000 7f585587f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-05T06:34:09.628 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:09 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:34:09.408+0000 7f585587f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-05T06:34:09.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:09 smithi028 ceph-mon[29442]: Standby manager daemon x restarted 2024-02-05T06:34:09.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:09 smithi028 ceph-mon[29442]: Standby manager daemon x started 2024-02-05T06:34:09.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:09 smithi028 ceph-mon[29442]: from='mgr.? v1:172.21.15.167:0/563774493' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-05T06:34:09.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:09 smithi028 ceph-mon[29442]: from='mgr.? v1:172.21.15.167:0/563774493' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-05T06:34:09.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:09 smithi028 ceph-mon[29442]: from='mgr.? v1:172.21.15.167:0/563774493' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-05T06:34:09.785 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:09 smithi028 ceph-mon[29442]: from='mgr.? v1:172.21.15.167:0/563774493' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-05T06:34:09.785 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:09 smithi028 ceph-mon[37984]: Standby manager daemon x restarted 2024-02-05T06:34:09.785 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:09 smithi028 ceph-mon[37984]: Standby manager daemon x started 2024-02-05T06:34:09.785 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:09 smithi028 ceph-mon[37984]: from='mgr.? v1:172.21.15.167:0/563774493' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-05T06:34:09.785 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:09 smithi028 ceph-mon[37984]: from='mgr.? v1:172.21.15.167:0/563774493' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-05T06:34:09.785 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:09 smithi028 ceph-mon[37984]: from='mgr.? v1:172.21.15.167:0/563774493' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-05T06:34:09.785 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:09 smithi028 ceph-mon[37984]: from='mgr.? v1:172.21.15.167:0/563774493' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-05T06:34:09.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:09 smithi167 ceph-mon[32774]: Standby manager daemon x restarted 2024-02-05T06:34:09.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:09 smithi167 ceph-mon[32774]: Standby manager daemon x started 2024-02-05T06:34:09.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:09 smithi167 ceph-mon[32774]: from='mgr.? v1:172.21.15.167:0/563774493' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-05T06:34:09.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:09 smithi167 ceph-mon[32774]: from='mgr.? v1:172.21.15.167:0/563774493' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-05T06:34:09.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:09 smithi167 ceph-mon[32774]: from='mgr.? v1:172.21.15.167:0/563774493' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-05T06:34:09.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:09 smithi167 ceph-mon[32774]: from='mgr.? v1:172.21.15.167:0/563774493' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-05T06:34:09.979 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:09 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:34:09.626+0000 7f585587f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-05T06:34:09.979 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:09 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: [05/Feb/2024:06:34:09] ENGINE Bus STARTING 2024-02-05T06:34:09.980 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:09 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: CherryPy Checker: 2024-02-05T06:34:09.980 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:09 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: The Application mounted at '' has an empty config. 2024-02-05T06:34:09.980 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:09 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: 2024-02-05T06:34:09.980 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:09 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: [05/Feb/2024:06:34:09] ENGINE Serving on http://:::9283 2024-02-05T06:34:09.980 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:34:09 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x[34660]: [05/Feb/2024:06:34:09] ENGINE Bus STARTED 2024-02-05T06:34:10.013 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:34:10.100 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:09 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:34:09.782+0000 7f65778ff200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-05T06:34:10.100 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:09 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:34:09.886+0000 7f65778ff200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-05T06:34:10.357 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:34:10.112+0000 7f65778ff200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-05T06:34:10.357 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:34:10] ENGINE Bus STARTING 2024-02-05T06:34:10.693 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: CherryPy Checker: 2024-02-05T06:34:10.693 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: The Application mounted at '' has an empty config. 2024-02-05T06:34:10.693 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:34:10.694 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:34:10] ENGINE Serving on http://:::9283 2024-02-05T06:34:10.694 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:34:10] ENGINE Bus STARTED 2024-02-05T06:34:10.701 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:10 smithi167 ceph-mon[32774]: mgrmap e18: y(active, since 5m), standbys: x 2024-02-05T06:34:10.701 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:10 smithi167 ceph-mon[32774]: osdmap e81: 8 total, 8 up, 8 in 2024-02-05T06:34:10.721 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:34:10.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:10 smithi167 ceph-mon[32774]: Active manager daemon y restarted 2024-02-05T06:34:10.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:10 smithi167 ceph-mon[32774]: Activating manager daemon y 2024-02-05T06:34:10.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:10 smithi167 ceph-mon[32774]: osdmap e82: 8 total, 8 up, 8 in 2024-02-05T06:34:10.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:10 smithi167 ceph-mon[32774]: mgrmap e19: y(active, starting, since 0.00841961s), standbys: x 2024-02-05T06:34:10.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:10 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T06:34:10.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:10 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T06:34:10.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:10 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:34:10.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:10 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-05T06:34:10.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:10 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-05T06:34:10.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:10 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T06:34:10.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:10 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T06:34:10.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:10 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T06:34:10.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:10 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T06:34:10.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:10 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T06:34:10.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:10 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T06:34:10.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:10 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T06:34:10.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:10 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T06:34:10.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:10 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-05T06:34:10.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:10 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-05T06:34:10.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:10 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-05T06:34:10.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:10 smithi167 ceph-mon[32774]: Manager daemon y is now available 2024-02-05T06:34:10.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:10 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:34:10.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:10 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:34:10.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:10 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:10.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:10 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:34:10.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:10 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-05T06:34:10.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:10 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-05T06:34:10.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:10 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-05T06:34:10.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:10 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-05T06:34:11.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[29442]: mgrmap e18: y(active, since 5m), standbys: x 2024-02-05T06:34:11.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[29442]: osdmap e81: 8 total, 8 up, 8 in 2024-02-05T06:34:11.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[29442]: Active manager daemon y restarted 2024-02-05T06:34:11.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[29442]: Activating manager daemon y 2024-02-05T06:34:11.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[29442]: osdmap e82: 8 total, 8 up, 8 in 2024-02-05T06:34:11.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[29442]: mgrmap e19: y(active, starting, since 0.00841961s), standbys: x 2024-02-05T06:34:11.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T06:34:11.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T06:34:11.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:34:11.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-05T06:34:11.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-05T06:34:11.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T06:34:11.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T06:34:11.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T06:34:11.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T06:34:11.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T06:34:11.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T06:34:11.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T06:34:11.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T06:34:11.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-05T06:34:11.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-05T06:34:11.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-05T06:34:11.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[29442]: Manager daemon y is now available 2024-02-05T06:34:11.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:34:11.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:34:11.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:11.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:34:11.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-05T06:34:11.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-05T06:34:11.135 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-05T06:34:11.135 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-05T06:34:11.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[37984]: mgrmap e18: y(active, since 5m), standbys: x 2024-02-05T06:34:11.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[37984]: osdmap e81: 8 total, 8 up, 8 in 2024-02-05T06:34:11.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[37984]: Active manager daemon y restarted 2024-02-05T06:34:11.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[37984]: Activating manager daemon y 2024-02-05T06:34:11.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[37984]: osdmap e82: 8 total, 8 up, 8 in 2024-02-05T06:34:11.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[37984]: mgrmap e19: y(active, starting, since 0.00841961s), standbys: x 2024-02-05T06:34:11.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-05T06:34:11.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-05T06:34:11.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-05T06:34:11.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-05T06:34:11.136 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-05T06:34:11.136 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-05T06:34:11.136 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-05T06:34:11.136 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-05T06:34:11.136 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-05T06:34:11.136 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-05T06:34:11.136 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-05T06:34:11.136 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-05T06:34:11.136 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-05T06:34:11.136 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-05T06:34:11.136 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-05T06:34:11.136 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-05T06:34:11.136 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[37984]: Manager daemon y is now available 2024-02-05T06:34:11.137 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:34:11.137 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:34:11.137 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:11.137 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:34:11.137 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-05T06:34:11.137 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-05T06:34:11.137 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-05T06:34:11.137 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:10 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-05T06:34:11.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:11 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:34:11.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:11 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:34:11.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:11 smithi028 ceph-mon[29442]: mgrmap e20: y(active, since 1.01214s), standbys: x 2024-02-05T06:34:11.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:11 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:34:11.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:11 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:34:11.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:11 smithi028 ceph-mon[37984]: mgrmap e20: y(active, since 1.01214s), standbys: x 2024-02-05T06:34:11.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:11 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:34:11.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:11 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:34:11.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:11 smithi167 ceph-mon[32774]: mgrmap e20: y(active, since 1.01214s), standbys: x 2024-02-05T06:34:11.980 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:34:12.618 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:12 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:34:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:34:12.707 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:34:12 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:34:12.943 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:12 smithi028 ceph-mon[37984]: [05/Feb/2024:06:34:11] ENGINE Bus STARTING 2024-02-05T06:34:12.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:12 smithi028 ceph-mon[37984]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-02-05T06:34:12.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:12 smithi028 ceph-mon[37984]: [05/Feb/2024:06:34:11] ENGINE Serving on https://172.21.15.28:7150 2024-02-05T06:34:12.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:12 smithi028 ceph-mon[37984]: [05/Feb/2024:06:34:11] ENGINE Serving on http://172.21.15.28:8765 2024-02-05T06:34:12.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:12 smithi028 ceph-mon[37984]: [05/Feb/2024:06:34:11] ENGINE Bus STARTED 2024-02-05T06:34:12.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:12 smithi028 ceph-mon[37984]: osdmap e83: 8 total, 8 up, 8 in 2024-02-05T06:34:12.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:12 smithi028 ceph-mon[29442]: [05/Feb/2024:06:34:11] ENGINE Bus STARTING 2024-02-05T06:34:12.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:12 smithi028 ceph-mon[29442]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-02-05T06:34:12.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:12 smithi028 ceph-mon[29442]: [05/Feb/2024:06:34:11] ENGINE Serving on https://172.21.15.28:7150 2024-02-05T06:34:12.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:12 smithi028 ceph-mon[29442]: [05/Feb/2024:06:34:11] ENGINE Serving on http://172.21.15.28:8765 2024-02-05T06:34:12.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:12 smithi028 ceph-mon[29442]: [05/Feb/2024:06:34:11] ENGINE Bus STARTED 2024-02-05T06:34:12.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:12 smithi028 ceph-mon[29442]: osdmap e83: 8 total, 8 up, 8 in 2024-02-05T06:34:12.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:12 smithi167 ceph-mon[32774]: [05/Feb/2024:06:34:11] ENGINE Bus STARTING 2024-02-05T06:34:12.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:12 smithi167 ceph-mon[32774]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-02-05T06:34:12.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:12 smithi167 ceph-mon[32774]: [05/Feb/2024:06:34:11] ENGINE Serving on https://172.21.15.28:7150 2024-02-05T06:34:12.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:12 smithi167 ceph-mon[32774]: [05/Feb/2024:06:34:11] ENGINE Serving on http://172.21.15.28:8765 2024-02-05T06:34:12.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:12 smithi167 ceph-mon[32774]: [05/Feb/2024:06:34:11] ENGINE Bus STARTED 2024-02-05T06:34:12.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:12 smithi167 ceph-mon[32774]: osdmap e83: 8 total, 8 up, 8 in 2024-02-05T06:34:13.963 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:13 smithi167 ceph-mon[32774]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-02-05T06:34:13.963 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:13 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:34:13.963 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:13 smithi167 ceph-mon[32774]: mgrmap e21: y(active, since 3s), standbys: x 2024-02-05T06:34:14.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:13 smithi028 ceph-mon[37984]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-02-05T06:34:14.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:13 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:34:14.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:13 smithi028 ceph-mon[37984]: mgrmap e21: y(active, since 3s), standbys: x 2024-02-05T06:34:14.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:13 smithi028 ceph-mon[29442]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-02-05T06:34:14.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:13 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:34:14.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:13 smithi028 ceph-mon[29442]: mgrmap e21: y(active, since 3s), standbys: x 2024-02-05T06:34:15.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:15 smithi028 ceph-mon[29442]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-02-05T06:34:15.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:15 smithi028 ceph-mon[29442]: mgrmap e22: y(active, since 4s), standbys: x 2024-02-05T06:34:15.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:15 smithi028 ceph-mon[29442]: osdmap e84: 8 total, 8 up, 8 in 2024-02-05T06:34:15.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:15 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:15.813 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:15 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:15.813 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:15 smithi028 ceph-mon[37984]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-02-05T06:34:15.813 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:15 smithi028 ceph-mon[37984]: mgrmap e22: y(active, since 4s), standbys: x 2024-02-05T06:34:15.813 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:15 smithi028 ceph-mon[37984]: osdmap e84: 8 total, 8 up, 8 in 2024-02-05T06:34:15.813 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:15 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:15.813 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:15 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:15.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:15 smithi167 ceph-mon[32774]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-02-05T06:34:15.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:15 smithi167 ceph-mon[32774]: mgrmap e22: y(active, since 4s), standbys: x 2024-02-05T06:34:15.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:15 smithi167 ceph-mon[32774]: osdmap e84: 8 total, 8 up, 8 in 2024-02-05T06:34:15.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:15 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:15.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:15 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:17.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:16 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:17.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:16 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:17.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:16 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:17.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:16 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:17.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:16 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:17.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:16 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:17.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:16 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:17.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:16 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:17.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:16 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:17.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:16 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:17.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:16 smithi028 ceph-mon[37984]: Adjusting osd_memory_target on smithi167 to 3760M 2024-02-05T06:34:17.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:16 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:17.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:16 smithi028 ceph-mon[37984]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-02-05T06:34:17.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:16 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:17.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:16 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:17.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:16 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:17.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:16 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:17.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:16 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:17.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:16 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:17.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:16 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:17.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:16 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:17.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:16 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:17.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:16 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:17.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:16 smithi028 ceph-mon[29442]: Adjusting osd_memory_target on smithi167 to 3760M 2024-02-05T06:34:17.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:16 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:17.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:16 smithi028 ceph-mon[29442]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-02-05T06:34:17.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:16 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:17.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:16 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:17.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:16 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:17.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:16 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:17.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:16 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:17.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:16 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:17.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:16 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:17.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:16 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:17.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:16 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:17.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:16 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:17.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:16 smithi167 ceph-mon[32774]: Adjusting osd_memory_target on smithi167 to 3760M 2024-02-05T06:34:17.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:16 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:17.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:16 smithi167 ceph-mon[32774]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-02-05T06:34:17.779 INFO:teuthology.orchestra.run.smithi028.stdout:55834574891 2024-02-05T06:34:17.779 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-05T06:34:18.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:18 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:34:18.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:18 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:34:18.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:18 smithi028 ceph-mon[29442]: osdmap e85: 8 total, 8 up, 8 in 2024-02-05T06:34:18.069 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:18 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:34:18.069 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:18 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:34:18.069 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:18 smithi028 ceph-mon[37984]: osdmap e85: 8 total, 8 up, 8 in 2024-02-05T06:34:18.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:18 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:34:18.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:18 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:34:18.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:18 smithi167 ceph-mon[32774]: osdmap e85: 8 total, 8 up, 8 in 2024-02-05T06:34:19.297 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:19 smithi028 ceph-mon[37984]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 18 op/s 2024-02-05T06:34:19.298 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:19 smithi028 ceph-mon[29442]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 18 op/s 2024-02-05T06:34:19.324 INFO:teuthology.orchestra.run.smithi028.stdout:77309411367 2024-02-05T06:34:19.325 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-05T06:34:19.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:19 smithi167 ceph-mon[32774]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 18 op/s 2024-02-05T06:34:20.582 INFO:teuthology.orchestra.run.smithi028.stdout:133143986204 2024-02-05T06:34:20.583 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-05T06:34:21.000 INFO:teuthology.orchestra.run.smithi028.stdout:111669149730 2024-02-05T06:34:21.001 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-05T06:34:21.133 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:20 smithi028 ceph-mon[37984]: osdmap e86: 8 total, 8 up, 8 in 2024-02-05T06:34:21.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:20 smithi028 ceph-mon[37984]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 20 op/s 2024-02-05T06:34:21.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:20 smithi028 ceph-mon[29442]: osdmap e86: 8 total, 8 up, 8 in 2024-02-05T06:34:21.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:20 smithi028 ceph-mon[29442]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 20 op/s 2024-02-05T06:34:21.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:20 smithi167 ceph-mon[32774]: osdmap e86: 8 total, 8 up, 8 in 2024-02-05T06:34:21.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:20 smithi167 ceph-mon[32774]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 20 op/s 2024-02-05T06:34:22.393 INFO:teuthology.orchestra.run.smithi028.stdout:210453397523 2024-02-05T06:34:22.393 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-05T06:34:22.693 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:22 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:34:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:34:22.728 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:34:22 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:34:23.031 INFO:teuthology.orchestra.run.smithi028.stdout:171798691864 2024-02-05T06:34:23.032 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-05T06:34:23.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:23 smithi167 ceph-mon[32774]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-05T06:34:23.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:23 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:34:23.482 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:23 smithi028 ceph-mon[29442]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-05T06:34:23.482 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:23 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:34:23.482 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:23 smithi028 ceph-mon[37984]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-02-05T06:34:23.482 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:23 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:34:23.781 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:34:23.782 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:34:23.782 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:34:23.824 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:34:24.378 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:24 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:34:24.379 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:24 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:34:24.379 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:24 smithi028 ceph-mon[29442]: osdmap e87: 8 total, 8 up, 8 in 2024-02-05T06:34:24.379 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:24 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:34:24.379 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:24 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:34:24.379 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:24 smithi028 ceph-mon[37984]: osdmap e87: 8 total, 8 up, 8 in 2024-02-05T06:34:24.463 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:34:24.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:24 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:34:24.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:24 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:34:24.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:24 smithi167 ceph-mon[32774]: osdmap e87: 8 total, 8 up, 8 in 2024-02-05T06:34:25.388 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:25 smithi028 ceph-mon[29442]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 717 B/s rd, 0 op/s 2024-02-05T06:34:25.388 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:25 smithi028 ceph-mon[29442]: osdmap e88: 8 total, 8 up, 8 in 2024-02-05T06:34:25.389 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:25 smithi028 ceph-mon[37984]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 717 B/s rd, 0 op/s 2024-02-05T06:34:25.389 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:25 smithi028 ceph-mon[37984]: osdmap e88: 8 total, 8 up, 8 in 2024-02-05T06:34:25.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:25 smithi167 ceph-mon[32774]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 717 B/s rd, 0 op/s 2024-02-05T06:34:25.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:25 smithi167 ceph-mon[32774]: osdmap e88: 8 total, 8 up, 8 in 2024-02-05T06:34:26.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:26 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:34:26.635 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:26 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:34:26.636 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:26 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:34:27.350 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:27 smithi028 ceph-mon[29442]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 817 B/s rd, 0 op/s 2024-02-05T06:34:27.350 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:27 smithi028 ceph-mon[37984]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 817 B/s rd, 0 op/s 2024-02-05T06:34:27.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:27 smithi167 ceph-mon[32774]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 817 B/s rd, 0 op/s 2024-02-05T06:34:29.018 INFO:teuthology.orchestra.run.smithi028.stdout:111669149732 2024-02-05T06:34:29.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:29 smithi028 ceph-mon[29442]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:34:29.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:29 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2783442205' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-05T06:34:29.431 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:29 smithi028 ceph-mon[37984]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:34:29.431 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:29 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2783442205' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-05T06:34:29.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:29 smithi167 ceph-mon[32774]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:34:29.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:29 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2783442205' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-05T06:34:30.191 INFO:teuthology.orchestra.run.smithi028.stdout:77309411369 2024-02-05T06:34:31.077 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:30 smithi028 ceph-mon[37984]: 3.b scrub starts 2024-02-05T06:34:31.077 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:30 smithi028 ceph-mon[37984]: 3.b scrub ok 2024-02-05T06:34:31.077 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:30 smithi028 ceph-mon[37984]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:34:31.077 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:30 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3180145345' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-05T06:34:31.077 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:30 smithi028 ceph-mon[29442]: 3.b scrub starts 2024-02-05T06:34:31.077 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:30 smithi028 ceph-mon[29442]: 3.b scrub ok 2024-02-05T06:34:31.078 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:30 smithi028 ceph-mon[29442]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:34:31.078 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:30 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3180145345' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-05T06:34:31.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:30 smithi167 ceph-mon[32774]: 3.b scrub starts 2024-02-05T06:34:31.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:30 smithi167 ceph-mon[32774]: 3.b scrub ok 2024-02-05T06:34:31.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:30 smithi167 ceph-mon[32774]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:34:31.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:30 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3180145345' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-05T06:34:31.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:31 smithi028 ceph-mon[37984]: 3.10 scrub starts 2024-02-05T06:34:31.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:31 smithi028 ceph-mon[37984]: 3.10 scrub ok 2024-02-05T06:34:31.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:31 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2064102915' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-05T06:34:31.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:31 smithi028 ceph-mon[29442]: 3.10 scrub starts 2024-02-05T06:34:31.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:31 smithi028 ceph-mon[29442]: 3.10 scrub ok 2024-02-05T06:34:31.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:31 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2064102915' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-05T06:34:32.082 INFO:teuthology.orchestra.run.smithi028.stdout:34359738420 2024-02-05T06:34:32.082 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-05T06:34:32.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:31 smithi167 ceph-mon[32774]: 3.10 scrub starts 2024-02-05T06:34:32.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:31 smithi167 ceph-mon[32774]: 3.10 scrub ok 2024-02-05T06:34:32.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:31 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2064102915' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-05T06:34:32.641 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:34:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:34:32.667 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149730 got 111669149732 for osd.3 2024-02-05T06:34:32.667 DEBUG:teuthology.parallel:result is None 2024-02-05T06:34:32.728 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:34:32 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:34:33.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:32 smithi028 ceph-mon[29442]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:34:33.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:32 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:34:33.153 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:32 smithi028 ceph-mon[37984]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:34:33.153 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:32 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:34:33.184 INFO:teuthology.orchestra.run.smithi028.stdout:249108103184 2024-02-05T06:34:33.185 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-05T06:34:33.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:32 smithi167 ceph-mon[32774]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:34:33.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:32 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:34:33.481 INFO:teuthology.orchestra.run.smithi028.stdout:55834574894 2024-02-05T06:34:34.170 INFO:teuthology.orchestra.run.smithi028.stdout:133143986207 2024-02-05T06:34:34.385 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574891 got 55834574894 for osd.1 2024-02-05T06:34:34.386 DEBUG:teuthology.parallel:result is None 2024-02-05T06:34:34.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:34 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/123977139' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-05T06:34:34.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:34 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/123977139' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-05T06:34:34.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:34 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/123977139' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-05T06:34:34.949 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986204 got 133143986207 for osd.4 2024-02-05T06:34:34.949 DEBUG:teuthology.parallel:result is None 2024-02-05T06:34:35.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:35 smithi028 ceph-mon[29442]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:34:35.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:35 smithi028 ceph-mon[29442]: 5.1b scrub starts 2024-02-05T06:34:35.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:35 smithi028 ceph-mon[29442]: 5.1b scrub ok 2024-02-05T06:34:35.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:35 smithi028 ceph-mon[29442]: 5.6 scrub starts 2024-02-05T06:34:35.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:35 smithi028 ceph-mon[29442]: 5.6 scrub ok 2024-02-05T06:34:35.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:35 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:35.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:35 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:35.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:35 smithi028 ceph-mon[37984]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:34:35.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:35 smithi028 ceph-mon[37984]: 5.1b scrub starts 2024-02-05T06:34:35.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:35 smithi028 ceph-mon[37984]: 5.1b scrub ok 2024-02-05T06:34:35.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:35 smithi028 ceph-mon[37984]: 5.6 scrub starts 2024-02-05T06:34:35.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:35 smithi028 ceph-mon[37984]: 5.6 scrub ok 2024-02-05T06:34:35.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:35 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:35.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:35 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:35.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:35 smithi167 ceph-mon[32774]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:34:35.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:35 smithi167 ceph-mon[32774]: 5.1b scrub starts 2024-02-05T06:34:35.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:35 smithi167 ceph-mon[32774]: 5.1b scrub ok 2024-02-05T06:34:35.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:35 smithi167 ceph-mon[32774]: 5.6 scrub starts 2024-02-05T06:34:35.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:35 smithi167 ceph-mon[32774]: 5.6 scrub ok 2024-02-05T06:34:35.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:35 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:35.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:35 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:36.013 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411367 got 77309411369 for osd.2 2024-02-05T06:34:36.013 DEBUG:teuthology.parallel:result is None 2024-02-05T06:34:36.083 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:34:36.121 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:34:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:34:36.246 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:36 smithi028 ceph-mon[37984]: 3.1e scrub starts 2024-02-05T06:34:36.246 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:36 smithi028 ceph-mon[37984]: 3.1e scrub ok 2024-02-05T06:34:36.246 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:36 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:34:36.246 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:36 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:34:36.246 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:36 smithi028 ceph-mon[37984]: 4.1c deep-scrub starts 2024-02-05T06:34:36.246 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:36 smithi028 ceph-mon[37984]: 4.1c deep-scrub ok 2024-02-05T06:34:36.246 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:36 smithi028 ceph-mon[37984]: osdmap e89: 8 total, 8 up, 8 in 2024-02-05T06:34:36.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:36 smithi028 ceph-mon[29442]: 3.1e scrub starts 2024-02-05T06:34:36.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:36 smithi028 ceph-mon[29442]: 3.1e scrub ok 2024-02-05T06:34:36.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:36 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:34:36.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:36 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:34:36.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:36 smithi028 ceph-mon[29442]: 4.1c deep-scrub starts 2024-02-05T06:34:36.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:36 smithi028 ceph-mon[29442]: 4.1c deep-scrub ok 2024-02-05T06:34:36.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:36 smithi028 ceph-mon[29442]: osdmap e89: 8 total, 8 up, 8 in 2024-02-05T06:34:36.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:36 smithi167 ceph-mon[32774]: 3.1e scrub starts 2024-02-05T06:34:36.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:36 smithi167 ceph-mon[32774]: 3.1e scrub ok 2024-02-05T06:34:36.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:36 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:34:36.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:36 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:34:36.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:36 smithi167 ceph-mon[32774]: 4.1c deep-scrub starts 2024-02-05T06:34:36.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:36 smithi167 ceph-mon[32774]: 4.1c deep-scrub ok 2024-02-05T06:34:36.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:36 smithi167 ceph-mon[32774]: osdmap e89: 8 total, 8 up, 8 in 2024-02-05T06:34:37.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:37 smithi167 ceph-mon[32774]: 4.19 deep-scrub starts 2024-02-05T06:34:37.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:37 smithi167 ceph-mon[32774]: 4.19 deep-scrub ok 2024-02-05T06:34:37.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:37 smithi167 ceph-mon[32774]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-05T06:34:37.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:37 smithi167 ceph-mon[32774]: 4.15 scrub starts 2024-02-05T06:34:37.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:37 smithi167 ceph-mon[32774]: 4.15 scrub ok 2024-02-05T06:34:37.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:37 smithi167 ceph-mon[32774]: 3.4 scrub starts 2024-02-05T06:34:37.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:37 smithi167 ceph-mon[32774]: 3.4 scrub ok 2024-02-05T06:34:37.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:37 smithi167 ceph-mon[32774]: 4.b scrub starts 2024-02-05T06:34:37.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:37 smithi167 ceph-mon[32774]: 4.b scrub ok 2024-02-05T06:34:37.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:37 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:37.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:37 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:37.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:37 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:37.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:37 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:37.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:37 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:37.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:37 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:37.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:37 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:37.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:37 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:37.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:37 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:37.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:37 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:37.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:37 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:37.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:37 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:34:37.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:37 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:34:37.543 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[29442]: 4.19 deep-scrub starts 2024-02-05T06:34:37.543 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[29442]: 4.19 deep-scrub ok 2024-02-05T06:34:37.543 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[29442]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-05T06:34:37.544 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[29442]: 4.15 scrub starts 2024-02-05T06:34:37.544 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[29442]: 4.15 scrub ok 2024-02-05T06:34:37.544 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[29442]: 3.4 scrub starts 2024-02-05T06:34:37.544 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[29442]: 3.4 scrub ok 2024-02-05T06:34:37.544 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[29442]: 4.b scrub starts 2024-02-05T06:34:37.544 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[29442]: 4.b scrub ok 2024-02-05T06:34:37.544 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:37.544 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:37.544 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:37.544 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:37.544 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:37.544 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:37.544 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:37.544 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:37.545 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:37.545 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:37.545 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:37.545 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:34:37.545 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:34:37.545 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[37984]: 4.19 deep-scrub starts 2024-02-05T06:34:37.545 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[37984]: 4.19 deep-scrub ok 2024-02-05T06:34:37.545 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[37984]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-05T06:34:37.545 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[37984]: 4.15 scrub starts 2024-02-05T06:34:37.545 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[37984]: 4.15 scrub ok 2024-02-05T06:34:37.545 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[37984]: 3.4 scrub starts 2024-02-05T06:34:37.545 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[37984]: 3.4 scrub ok 2024-02-05T06:34:37.546 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[37984]: 4.b scrub starts 2024-02-05T06:34:37.546 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[37984]: 4.b scrub ok 2024-02-05T06:34:37.546 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:37.546 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:37.546 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:37.546 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:37.546 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:37.546 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:37.546 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:37.546 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:37.546 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:37.546 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:34:37.546 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:37.546 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:34:37.547 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:37 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:34:38.426 INFO:teuthology.orchestra.run.smithi028.stdout:210453397527 2024-02-05T06:34:38.844 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:38 smithi028 ceph-mon[29442]: Adjusting osd_memory_target on smithi028 to 3761M 2024-02-05T06:34:38.844 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:38 smithi028 ceph-mon[29442]: Updating smithi028:/etc/ceph/ceph.conf 2024-02-05T06:34:38.844 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:38 smithi028 ceph-mon[29442]: Updating smithi167:/etc/ceph/ceph.conf 2024-02-05T06:34:38.844 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:38 smithi028 ceph-mon[29442]: Updating smithi028:/var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T06:34:38.844 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:38 smithi028 ceph-mon[29442]: Updating smithi167:/var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T06:34:38.844 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:38 smithi028 ceph-mon[29442]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-02-05T06:34:38.844 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:38 smithi028 ceph-mon[29442]: Updating smithi167:/etc/ceph/ceph.client.admin.keyring 2024-02-05T06:34:38.844 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:38 smithi028 ceph-mon[29442]: Updating smithi028:/var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/config/ceph.client.admin.keyring 2024-02-05T06:34:38.844 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:38 smithi028 ceph-mon[29442]: Updating smithi167:/var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/config/ceph.client.admin.keyring 2024-02-05T06:34:38.845 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:38 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:38.845 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:38 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:38.845 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:38 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:38.845 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:38 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:38.845 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:38 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:38.845 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:38 smithi028 ceph-mon[29442]: Deploying daemon alertmanager.a on smithi028 2024-02-05T06:34:38.845 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:38 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/825233354' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-05T06:34:38.845 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:38 smithi028 ceph-mon[37984]: Adjusting osd_memory_target on smithi028 to 3761M 2024-02-05T06:34:38.845 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:38 smithi028 ceph-mon[37984]: Updating smithi028:/etc/ceph/ceph.conf 2024-02-05T06:34:38.845 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:38 smithi028 ceph-mon[37984]: Updating smithi167:/etc/ceph/ceph.conf 2024-02-05T06:34:38.845 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:38 smithi028 ceph-mon[37984]: Updating smithi028:/var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T06:34:38.845 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:38 smithi028 ceph-mon[37984]: Updating smithi167:/var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T06:34:38.845 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:38 smithi028 ceph-mon[37984]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-02-05T06:34:38.845 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:38 smithi028 ceph-mon[37984]: Updating smithi167:/etc/ceph/ceph.client.admin.keyring 2024-02-05T06:34:38.846 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:38 smithi028 ceph-mon[37984]: Updating smithi028:/var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/config/ceph.client.admin.keyring 2024-02-05T06:34:38.846 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:38 smithi028 ceph-mon[37984]: Updating smithi167:/var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/config/ceph.client.admin.keyring 2024-02-05T06:34:38.846 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:38 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:38.846 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:38 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:38.846 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:38 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:38.846 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:38 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:38.846 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:38 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:38.846 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:38 smithi028 ceph-mon[37984]: Deploying daemon alertmanager.a on smithi028 2024-02-05T06:34:38.846 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:38 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/825233354' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-05T06:34:38.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:38 smithi167 ceph-mon[32774]: Adjusting osd_memory_target on smithi028 to 3761M 2024-02-05T06:34:38.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:38 smithi167 ceph-mon[32774]: Updating smithi028:/etc/ceph/ceph.conf 2024-02-05T06:34:38.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:38 smithi167 ceph-mon[32774]: Updating smithi167:/etc/ceph/ceph.conf 2024-02-05T06:34:38.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:38 smithi167 ceph-mon[32774]: Updating smithi028:/var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T06:34:38.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:38 smithi167 ceph-mon[32774]: Updating smithi167:/var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/config/ceph.conf 2024-02-05T06:34:38.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:38 smithi167 ceph-mon[32774]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-02-05T06:34:38.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:38 smithi167 ceph-mon[32774]: Updating smithi167:/etc/ceph/ceph.client.admin.keyring 2024-02-05T06:34:38.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:38 smithi167 ceph-mon[32774]: Updating smithi028:/var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/config/ceph.client.admin.keyring 2024-02-05T06:34:38.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:38 smithi167 ceph-mon[32774]: Updating smithi167:/var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/config/ceph.client.admin.keyring 2024-02-05T06:34:38.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:38 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:38.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:38 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:38.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:38 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:38.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:38 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:38.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:38 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:38.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:38 smithi167 ceph-mon[32774]: Deploying daemon alertmanager.a on smithi028 2024-02-05T06:34:38.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:38 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/825233354' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-05T06:34:39.689 INFO:teuthology.orchestra.run.smithi028.stdout:249108103185 2024-02-05T06:34:39.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:39 smithi028 ceph-mon[29442]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:34:39.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:39 smithi028 ceph-mon[29442]: 3.1a scrub starts 2024-02-05T06:34:39.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:39 smithi028 ceph-mon[29442]: 3.1a scrub ok 2024-02-05T06:34:39.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:39 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1440770548' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-05T06:34:39.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:39 smithi028 ceph-mon[37984]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:34:39.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:39 smithi028 ceph-mon[37984]: 3.1a scrub starts 2024-02-05T06:34:39.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:39 smithi028 ceph-mon[37984]: 3.1a scrub ok 2024-02-05T06:34:39.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:39 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1440770548' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-05T06:34:39.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:39 smithi167 ceph-mon[32774]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:34:39.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:39 smithi167 ceph-mon[32774]: 3.1a scrub starts 2024-02-05T06:34:39.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:39 smithi167 ceph-mon[32774]: 3.1a scrub ok 2024-02-05T06:34:39.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:39 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1440770548' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-05T06:34:40.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:40 smithi028 ceph-mon[37984]: osdmap e90: 8 total, 8 up, 8 in 2024-02-05T06:34:40.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:40 smithi028 ceph-mon[37984]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-05T06:34:40.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:40 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:34:40.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:40 smithi028 ceph-mon[37984]: 5.8 scrub starts 2024-02-05T06:34:40.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:40 smithi028 ceph-mon[37984]: 5.8 scrub ok 2024-02-05T06:34:40.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:40 smithi028 ceph-mon[37984]: 5.e scrub starts 2024-02-05T06:34:40.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:40 smithi028 ceph-mon[37984]: 5.e scrub ok 2024-02-05T06:34:40.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:40 smithi028 ceph-mon[29442]: osdmap e90: 8 total, 8 up, 8 in 2024-02-05T06:34:40.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:40 smithi028 ceph-mon[29442]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-05T06:34:40.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:40 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:34:40.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:40 smithi028 ceph-mon[29442]: 5.8 scrub starts 2024-02-05T06:34:40.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:40 smithi028 ceph-mon[29442]: 5.8 scrub ok 2024-02-05T06:34:40.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:40 smithi028 ceph-mon[29442]: 5.e scrub starts 2024-02-05T06:34:40.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:40 smithi028 ceph-mon[29442]: 5.e scrub ok 2024-02-05T06:34:41.079 INFO:tasks.cephadm.ceph_manager.ceph:need seq 249108103184 got 249108103185 for osd.7 2024-02-05T06:34:41.079 DEBUG:teuthology.parallel:result is None 2024-02-05T06:34:41.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:40 smithi167 ceph-mon[32774]: osdmap e90: 8 total, 8 up, 8 in 2024-02-05T06:34:41.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:40 smithi167 ceph-mon[32774]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-05T06:34:41.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:40 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:34:41.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:40 smithi167 ceph-mon[32774]: 5.8 scrub starts 2024-02-05T06:34:41.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:40 smithi167 ceph-mon[32774]: 5.8 scrub ok 2024-02-05T06:34:41.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:40 smithi167 ceph-mon[32774]: 5.e scrub starts 2024-02-05T06:34:41.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:40 smithi167 ceph-mon[32774]: 5.e scrub ok 2024-02-05T06:34:41.628 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453397523 got 210453397527 for osd.6 2024-02-05T06:34:41.628 DEBUG:teuthology.parallel:result is None 2024-02-05T06:34:42.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:41 smithi028 ceph-mon[29442]: 5.12 scrub starts 2024-02-05T06:34:42.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:41 smithi028 ceph-mon[29442]: 5.12 scrub ok 2024-02-05T06:34:42.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:41 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:34:42.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:41 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:34:42.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:41 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:34:42.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:41 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:34:42.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:41 smithi028 ceph-mon[37984]: 5.12 scrub starts 2024-02-05T06:34:42.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:41 smithi028 ceph-mon[37984]: 5.12 scrub ok 2024-02-05T06:34:42.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:41 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:34:42.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:41 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:34:42.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:41 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:34:42.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:41 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:34:42.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:41 smithi167 ceph-mon[32774]: 5.12 scrub starts 2024-02-05T06:34:42.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:41 smithi167 ceph-mon[32774]: 5.12 scrub ok 2024-02-05T06:34:42.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:41 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:34:42.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:41 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:34:42.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:41 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:34:42.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:41 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:34:42.694 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:42 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:34:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:34:42.728 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:34:42 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:34:43.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:42 smithi028 ceph-mon[29442]: 6.3 scrub starts 2024-02-05T06:34:43.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:42 smithi028 ceph-mon[29442]: 6.3 scrub ok 2024-02-05T06:34:43.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:42 smithi028 ceph-mon[29442]: 4.14 scrub starts 2024-02-05T06:34:43.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:42 smithi028 ceph-mon[29442]: 4.14 scrub ok 2024-02-05T06:34:43.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:42 smithi028 ceph-mon[29442]: osdmap e91: 8 total, 8 up, 8 in 2024-02-05T06:34:43.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:42 smithi028 ceph-mon[29442]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-05T06:34:43.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:42 smithi028 ceph-mon[29442]: 5.b scrub starts 2024-02-05T06:34:43.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:42 smithi028 ceph-mon[29442]: 5.b scrub ok 2024-02-05T06:34:43.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:42 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:34:43.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:42 smithi028 ceph-mon[37984]: 6.3 scrub starts 2024-02-05T06:34:43.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:42 smithi028 ceph-mon[37984]: 6.3 scrub ok 2024-02-05T06:34:43.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:42 smithi028 ceph-mon[37984]: 4.14 scrub starts 2024-02-05T06:34:43.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:42 smithi028 ceph-mon[37984]: 4.14 scrub ok 2024-02-05T06:34:43.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:42 smithi028 ceph-mon[37984]: osdmap e91: 8 total, 8 up, 8 in 2024-02-05T06:34:43.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:42 smithi028 ceph-mon[37984]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-05T06:34:43.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:42 smithi028 ceph-mon[37984]: 5.b scrub starts 2024-02-05T06:34:43.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:42 smithi028 ceph-mon[37984]: 5.b scrub ok 2024-02-05T06:34:43.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:42 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:34:43.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:42 smithi167 ceph-mon[32774]: 6.3 scrub starts 2024-02-05T06:34:43.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:42 smithi167 ceph-mon[32774]: 6.3 scrub ok 2024-02-05T06:34:43.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:42 smithi167 ceph-mon[32774]: 4.14 scrub starts 2024-02-05T06:34:43.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:42 smithi167 ceph-mon[32774]: 4.14 scrub ok 2024-02-05T06:34:43.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:42 smithi167 ceph-mon[32774]: osdmap e91: 8 total, 8 up, 8 in 2024-02-05T06:34:43.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:42 smithi167 ceph-mon[32774]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-05T06:34:43.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:42 smithi167 ceph-mon[32774]: 5.b scrub starts 2024-02-05T06:34:43.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:42 smithi167 ceph-mon[32774]: 5.b scrub ok 2024-02-05T06:34:43.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:42 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:34:43.988 INFO:teuthology.orchestra.run.smithi028.stdout:171798691868 2024-02-05T06:34:44.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:43 smithi028 ceph-mon[29442]: 2.0 scrub starts 2024-02-05T06:34:44.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:43 smithi028 ceph-mon[29442]: 2.0 scrub ok 2024-02-05T06:34:44.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:43 smithi028 ceph-mon[29442]: 5.17 scrub starts 2024-02-05T06:34:44.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:43 smithi028 ceph-mon[29442]: 5.17 scrub ok 2024-02-05T06:34:44.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:43 smithi028 ceph-mon[29442]: 3.5 scrub starts 2024-02-05T06:34:44.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:43 smithi028 ceph-mon[29442]: 3.5 scrub ok 2024-02-05T06:34:44.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:43 smithi028 ceph-mon[29442]: 6.14 scrub starts 2024-02-05T06:34:44.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:43 smithi028 ceph-mon[29442]: 6.14 scrub ok 2024-02-05T06:34:44.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:43 smithi028 ceph-mon[37984]: 2.0 scrub starts 2024-02-05T06:34:44.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:43 smithi028 ceph-mon[37984]: 2.0 scrub ok 2024-02-05T06:34:44.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:43 smithi028 ceph-mon[37984]: 5.17 scrub starts 2024-02-05T06:34:44.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:43 smithi028 ceph-mon[37984]: 5.17 scrub ok 2024-02-05T06:34:44.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:43 smithi028 ceph-mon[37984]: 3.5 scrub starts 2024-02-05T06:34:44.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:43 smithi028 ceph-mon[37984]: 3.5 scrub ok 2024-02-05T06:34:44.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:43 smithi028 ceph-mon[37984]: 6.14 scrub starts 2024-02-05T06:34:44.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:43 smithi028 ceph-mon[37984]: 6.14 scrub ok 2024-02-05T06:34:44.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:43 smithi167 ceph-mon[32774]: 2.0 scrub starts 2024-02-05T06:34:44.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:43 smithi167 ceph-mon[32774]: 2.0 scrub ok 2024-02-05T06:34:44.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:43 smithi167 ceph-mon[32774]: 5.17 scrub starts 2024-02-05T06:34:44.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:43 smithi167 ceph-mon[32774]: 5.17 scrub ok 2024-02-05T06:34:44.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:43 smithi167 ceph-mon[32774]: 3.5 scrub starts 2024-02-05T06:34:44.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:43 smithi167 ceph-mon[32774]: 3.5 scrub ok 2024-02-05T06:34:44.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:43 smithi167 ceph-mon[32774]: 6.14 scrub starts 2024-02-05T06:34:44.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:43 smithi167 ceph-mon[32774]: 6.14 scrub ok 2024-02-05T06:34:44.917 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:44 smithi028 ceph-mon[29442]: 5.10 scrub starts 2024-02-05T06:34:44.917 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:44 smithi028 ceph-mon[29442]: 5.10 scrub ok 2024-02-05T06:34:44.917 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:44 smithi028 ceph-mon[29442]: 4.16 scrub starts 2024-02-05T06:34:44.917 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:44 smithi028 ceph-mon[29442]: 4.16 scrub ok 2024-02-05T06:34:44.917 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:44 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1555931175' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-05T06:34:44.917 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:44 smithi028 ceph-mon[29442]: 3.1d deep-scrub starts 2024-02-05T06:34:44.917 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:44 smithi028 ceph-mon[29442]: 3.1d deep-scrub ok 2024-02-05T06:34:44.917 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:44 smithi028 ceph-mon[29442]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:34:44.917 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:44 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:34:44.918 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:44 smithi028 ceph-mon[37984]: 5.10 scrub starts 2024-02-05T06:34:44.918 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:44 smithi028 ceph-mon[37984]: 5.10 scrub ok 2024-02-05T06:34:44.918 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:44 smithi028 ceph-mon[37984]: 4.16 scrub starts 2024-02-05T06:34:44.918 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:44 smithi028 ceph-mon[37984]: 4.16 scrub ok 2024-02-05T06:34:44.918 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:44 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1555931175' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-05T06:34:44.918 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:44 smithi028 ceph-mon[37984]: 3.1d deep-scrub starts 2024-02-05T06:34:44.918 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:44 smithi028 ceph-mon[37984]: 3.1d deep-scrub ok 2024-02-05T06:34:44.918 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:44 smithi028 ceph-mon[37984]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:34:44.918 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:44 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:34:45.183 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:44 smithi167 ceph-mon[32774]: 5.10 scrub starts 2024-02-05T06:34:45.183 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:44 smithi167 ceph-mon[32774]: 5.10 scrub ok 2024-02-05T06:34:45.183 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:44 smithi167 ceph-mon[32774]: 4.16 scrub starts 2024-02-05T06:34:45.183 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:44 smithi167 ceph-mon[32774]: 4.16 scrub ok 2024-02-05T06:34:45.183 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:44 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1555931175' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-05T06:34:45.183 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:44 smithi167 ceph-mon[32774]: 3.1d deep-scrub starts 2024-02-05T06:34:45.183 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:44 smithi167 ceph-mon[32774]: 3.1d deep-scrub ok 2024-02-05T06:34:45.183 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:44 smithi167 ceph-mon[32774]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:34:45.183 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:44 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:34:45.183 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:44 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:34:45.184 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:44 smithi167 ceph-mon[32774]: osdmap e92: 8 total, 8 up, 8 in 2024-02-05T06:34:45.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:44 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:34:45.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:44 smithi028 ceph-mon[29442]: osdmap e92: 8 total, 8 up, 8 in 2024-02-05T06:34:45.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:44 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:34:45.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:44 smithi028 ceph-mon[37984]: osdmap e92: 8 total, 8 up, 8 in 2024-02-05T06:34:45.306 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691864 got 171798691868 for osd.5 2024-02-05T06:34:45.306 DEBUG:teuthology.parallel:result is None 2024-02-05T06:34:46.138 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:45 smithi028 ceph-mon[29442]: 4.a scrub starts 2024-02-05T06:34:46.138 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:45 smithi028 ceph-mon[29442]: 4.a scrub ok 2024-02-05T06:34:46.138 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:45 smithi028 ceph-mon[29442]: 3.1 scrub starts 2024-02-05T06:34:46.138 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:45 smithi028 ceph-mon[29442]: 3.1 scrub ok 2024-02-05T06:34:46.138 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:45 smithi028 ceph-mon[29442]: 3.2 scrub starts 2024-02-05T06:34:46.138 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:45 smithi028 ceph-mon[29442]: 3.2 scrub ok 2024-02-05T06:34:46.138 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:45 smithi028 ceph-mon[29442]: 5.16 deep-scrub starts 2024-02-05T06:34:46.138 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:45 smithi028 ceph-mon[29442]: 5.16 deep-scrub ok 2024-02-05T06:34:46.138 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:45 smithi028 ceph-mon[29442]: 5.a scrub starts 2024-02-05T06:34:46.138 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:45 smithi028 ceph-mon[29442]: 5.a scrub ok 2024-02-05T06:34:46.138 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:45 smithi028 ceph-mon[29442]: 6.1e scrub starts 2024-02-05T06:34:46.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:45 smithi028 ceph-mon[29442]: 6.1e scrub ok 2024-02-05T06:34:46.139 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:45 smithi028 ceph-mon[37984]: 4.a scrub starts 2024-02-05T06:34:46.139 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:45 smithi028 ceph-mon[37984]: 4.a scrub ok 2024-02-05T06:34:46.139 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:45 smithi028 ceph-mon[37984]: 3.1 scrub starts 2024-02-05T06:34:46.139 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:45 smithi028 ceph-mon[37984]: 3.1 scrub ok 2024-02-05T06:34:46.139 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:45 smithi028 ceph-mon[37984]: 3.2 scrub starts 2024-02-05T06:34:46.139 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:45 smithi028 ceph-mon[37984]: 3.2 scrub ok 2024-02-05T06:34:46.139 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:45 smithi028 ceph-mon[37984]: 5.16 deep-scrub starts 2024-02-05T06:34:46.139 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:45 smithi028 ceph-mon[37984]: 5.16 deep-scrub ok 2024-02-05T06:34:46.139 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:45 smithi028 ceph-mon[37984]: 5.a scrub starts 2024-02-05T06:34:46.139 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:45 smithi028 ceph-mon[37984]: 5.a scrub ok 2024-02-05T06:34:46.139 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:45 smithi028 ceph-mon[37984]: 6.1e scrub starts 2024-02-05T06:34:46.139 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:45 smithi028 ceph-mon[37984]: 6.1e scrub ok 2024-02-05T06:34:46.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:45 smithi167 ceph-mon[32774]: 4.a scrub starts 2024-02-05T06:34:46.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:45 smithi167 ceph-mon[32774]: 4.a scrub ok 2024-02-05T06:34:46.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:45 smithi167 ceph-mon[32774]: 3.1 scrub starts 2024-02-05T06:34:46.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:45 smithi167 ceph-mon[32774]: 3.1 scrub ok 2024-02-05T06:34:46.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:45 smithi167 ceph-mon[32774]: 3.2 scrub starts 2024-02-05T06:34:46.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:45 smithi167 ceph-mon[32774]: 3.2 scrub ok 2024-02-05T06:34:46.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:45 smithi167 ceph-mon[32774]: 5.16 deep-scrub starts 2024-02-05T06:34:46.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:45 smithi167 ceph-mon[32774]: 5.16 deep-scrub ok 2024-02-05T06:34:46.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:45 smithi167 ceph-mon[32774]: 5.a scrub starts 2024-02-05T06:34:46.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:45 smithi167 ceph-mon[32774]: 5.a scrub ok 2024-02-05T06:34:46.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:45 smithi167 ceph-mon[32774]: 6.1e scrub starts 2024-02-05T06:34:46.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:45 smithi167 ceph-mon[32774]: 6.1e scrub ok 2024-02-05T06:34:46.505 INFO:teuthology.orchestra.run.smithi028.stdout:34359738423 2024-02-05T06:34:47.129 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:46 smithi028 ceph-mon[29442]: 4.1f deep-scrub starts 2024-02-05T06:34:47.129 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:46 smithi028 ceph-mon[29442]: 4.1f deep-scrub ok 2024-02-05T06:34:47.129 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:46 smithi028 ceph-mon[29442]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-05T06:34:47.129 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:46 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/623060284' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-05T06:34:47.129 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:46 smithi028 ceph-mon[29442]: 4.1a deep-scrub starts 2024-02-05T06:34:47.129 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:46 smithi028 ceph-mon[29442]: 4.1a deep-scrub ok 2024-02-05T06:34:47.130 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:46 smithi028 ceph-mon[37984]: 4.1f deep-scrub starts 2024-02-05T06:34:47.130 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:46 smithi028 ceph-mon[37984]: 4.1f deep-scrub ok 2024-02-05T06:34:47.130 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:46 smithi028 ceph-mon[37984]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-05T06:34:47.130 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:46 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/623060284' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-05T06:34:47.130 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:46 smithi028 ceph-mon[37984]: 4.1a deep-scrub starts 2024-02-05T06:34:47.130 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:46 smithi028 ceph-mon[37984]: 4.1a deep-scrub ok 2024-02-05T06:34:47.157 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738420 got 34359738423 for osd.0 2024-02-05T06:34:47.157 DEBUG:teuthology.parallel:result is None 2024-02-05T06:34:47.157 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-05T06:34:47.158 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T06:34:47.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:46 smithi167 ceph-mon[32774]: 4.1f deep-scrub starts 2024-02-05T06:34:47.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:46 smithi167 ceph-mon[32774]: 4.1f deep-scrub ok 2024-02-05T06:34:47.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:46 smithi167 ceph-mon[32774]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-02-05T06:34:47.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:46 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/623060284' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-05T06:34:47.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:46 smithi167 ceph-mon[32774]: 4.1a deep-scrub starts 2024-02-05T06:34:47.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:46 smithi167 ceph-mon[32774]: 4.1a deep-scrub ok 2024-02-05T06:34:47.998 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:47 smithi028 ceph-mon[29442]: 6.4 scrub starts 2024-02-05T06:34:47.998 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:47 smithi028 ceph-mon[29442]: 6.4 scrub ok 2024-02-05T06:34:47.999 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:47 smithi028 ceph-mon[37984]: 6.4 scrub starts 2024-02-05T06:34:47.999 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:47 smithi028 ceph-mon[37984]: 6.4 scrub ok 2024-02-05T06:34:48.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:47 smithi167 ceph-mon[32774]: 6.4 scrub starts 2024-02-05T06:34:48.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:47 smithi167 ceph-mon[32774]: 6.4 scrub ok 2024-02-05T06:34:49.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:48 smithi028 ceph-mon[29442]: 5.14 scrub starts 2024-02-05T06:34:49.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:48 smithi028 ceph-mon[29442]: 5.14 scrub ok 2024-02-05T06:34:49.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:48 smithi028 ceph-mon[29442]: 2.2 scrub starts 2024-02-05T06:34:49.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:48 smithi028 ceph-mon[29442]: 2.2 scrub ok 2024-02-05T06:34:49.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:48 smithi028 ceph-mon[29442]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:34:49.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:48 smithi028 ceph-mon[29442]: 3.14 scrub starts 2024-02-05T06:34:49.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:48 smithi028 ceph-mon[29442]: 3.14 scrub ok 2024-02-05T06:34:49.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:48 smithi028 ceph-mon[37984]: 5.14 scrub starts 2024-02-05T06:34:49.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:48 smithi028 ceph-mon[37984]: 5.14 scrub ok 2024-02-05T06:34:49.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:48 smithi028 ceph-mon[37984]: 2.2 scrub starts 2024-02-05T06:34:49.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:48 smithi028 ceph-mon[37984]: 2.2 scrub ok 2024-02-05T06:34:49.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:48 smithi028 ceph-mon[37984]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:34:49.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:48 smithi028 ceph-mon[37984]: 3.14 scrub starts 2024-02-05T06:34:49.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:48 smithi028 ceph-mon[37984]: 3.14 scrub ok 2024-02-05T06:34:49.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:48 smithi167 ceph-mon[32774]: 5.14 scrub starts 2024-02-05T06:34:49.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:48 smithi167 ceph-mon[32774]: 5.14 scrub ok 2024-02-05T06:34:49.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:48 smithi167 ceph-mon[32774]: 2.2 scrub starts 2024-02-05T06:34:49.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:48 smithi167 ceph-mon[32774]: 2.2 scrub ok 2024-02-05T06:34:49.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:48 smithi167 ceph-mon[32774]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:34:49.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:48 smithi167 ceph-mon[32774]: 3.14 scrub starts 2024-02-05T06:34:49.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:48 smithi167 ceph-mon[32774]: 3.14 scrub ok 2024-02-05T06:34:49.232 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:34:50.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:49 smithi028 ceph-mon[37984]: 3.19 scrub starts 2024-02-05T06:34:50.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:49 smithi028 ceph-mon[37984]: 3.19 scrub ok 2024-02-05T06:34:50.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:49 smithi028 ceph-mon[37984]: 6.11 scrub starts 2024-02-05T06:34:50.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:49 smithi028 ceph-mon[37984]: 6.11 scrub ok 2024-02-05T06:34:50.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:49 smithi028 ceph-mon[37984]: 6.f scrub starts 2024-02-05T06:34:50.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:49 smithi028 ceph-mon[37984]: 6.f scrub ok 2024-02-05T06:34:50.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:49 smithi028 ceph-mon[37984]: 5.1 scrub starts 2024-02-05T06:34:50.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:49 smithi028 ceph-mon[37984]: 5.1 scrub ok 2024-02-05T06:34:50.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:49 smithi028 ceph-mon[29442]: 3.19 scrub starts 2024-02-05T06:34:50.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:49 smithi028 ceph-mon[29442]: 3.19 scrub ok 2024-02-05T06:34:50.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:49 smithi028 ceph-mon[29442]: 6.11 scrub starts 2024-02-05T06:34:50.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:49 smithi028 ceph-mon[29442]: 6.11 scrub ok 2024-02-05T06:34:50.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:49 smithi028 ceph-mon[29442]: 6.f scrub starts 2024-02-05T06:34:50.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:49 smithi028 ceph-mon[29442]: 6.f scrub ok 2024-02-05T06:34:50.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:49 smithi028 ceph-mon[29442]: 5.1 scrub starts 2024-02-05T06:34:50.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:49 smithi028 ceph-mon[29442]: 5.1 scrub ok 2024-02-05T06:34:50.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:49 smithi167 ceph-mon[32774]: 3.19 scrub starts 2024-02-05T06:34:50.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:49 smithi167 ceph-mon[32774]: 3.19 scrub ok 2024-02-05T06:34:50.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:49 smithi167 ceph-mon[32774]: 6.11 scrub starts 2024-02-05T06:34:50.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:49 smithi167 ceph-mon[32774]: 6.11 scrub ok 2024-02-05T06:34:50.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:49 smithi167 ceph-mon[32774]: 6.f scrub starts 2024-02-05T06:34:50.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:49 smithi167 ceph-mon[32774]: 6.f scrub ok 2024-02-05T06:34:50.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:49 smithi167 ceph-mon[32774]: 5.1 scrub starts 2024-02-05T06:34:50.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:49 smithi167 ceph-mon[32774]: 5.1 scrub ok 2024-02-05T06:34:50.741 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:34:50 smithi028 podman[72643]: 2024-02-05 06:34:50.575503088 +0000 UTC m=+0.018783146 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-02-05T06:34:51.003 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:34:50 smithi028 podman[72643]: 2024-02-05 06:34:50.925047225 +0000 UTC m=+0.368327281 volume create 9fd603b0d2390056e86a7366ec69b895bc0888d085fa641345ee9b3f1395825d 2024-02-05T06:34:51.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:50 smithi028 ceph-mon[29442]: 5.9 scrub starts 2024-02-05T06:34:51.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:50 smithi028 ceph-mon[29442]: 5.9 scrub ok 2024-02-05T06:34:51.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:50 smithi028 ceph-mon[29442]: 3.1c deep-scrub starts 2024-02-05T06:34:51.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:50 smithi028 ceph-mon[29442]: 3.1c deep-scrub ok 2024-02-05T06:34:51.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:50 smithi028 ceph-mon[29442]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 620 B/s rd, 0 op/s 2024-02-05T06:34:51.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:50 smithi028 ceph-mon[29442]: 2.1 scrub starts 2024-02-05T06:34:51.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:50 smithi028 ceph-mon[29442]: 2.1 scrub ok 2024-02-05T06:34:51.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:50 smithi028 ceph-mon[29442]: 4.e scrub starts 2024-02-05T06:34:51.005 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:50 smithi028 ceph-mon[29442]: 4.e scrub ok 2024-02-05T06:34:51.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:50 smithi028 ceph-mon[37984]: 5.9 scrub starts 2024-02-05T06:34:51.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:50 smithi028 ceph-mon[37984]: 5.9 scrub ok 2024-02-05T06:34:51.005 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:50 smithi028 ceph-mon[37984]: 3.1c deep-scrub starts 2024-02-05T06:34:51.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:50 smithi028 ceph-mon[37984]: 3.1c deep-scrub ok 2024-02-05T06:34:51.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:50 smithi028 ceph-mon[37984]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 620 B/s rd, 0 op/s 2024-02-05T06:34:51.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:50 smithi028 ceph-mon[37984]: 2.1 scrub starts 2024-02-05T06:34:51.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:50 smithi028 ceph-mon[37984]: 2.1 scrub ok 2024-02-05T06:34:51.006 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:50 smithi028 ceph-mon[37984]: 4.e scrub starts 2024-02-05T06:34:51.007 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:50 smithi028 ceph-mon[37984]: 4.e scrub ok 2024-02-05T06:34:51.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:50 smithi167 ceph-mon[32774]: 5.9 scrub starts 2024-02-05T06:34:51.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:50 smithi167 ceph-mon[32774]: 5.9 scrub ok 2024-02-05T06:34:51.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:50 smithi167 ceph-mon[32774]: 3.1c deep-scrub starts 2024-02-05T06:34:51.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:50 smithi167 ceph-mon[32774]: 3.1c deep-scrub ok 2024-02-05T06:34:51.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:50 smithi167 ceph-mon[32774]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 620 B/s rd, 0 op/s 2024-02-05T06:34:51.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:50 smithi167 ceph-mon[32774]: 2.1 scrub starts 2024-02-05T06:34:51.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:50 smithi167 ceph-mon[32774]: 2.1 scrub ok 2024-02-05T06:34:51.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:50 smithi167 ceph-mon[32774]: 4.e scrub starts 2024-02-05T06:34:51.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:50 smithi167 ceph-mon[32774]: 4.e scrub ok 2024-02-05T06:34:51.259 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:34:51 smithi028 podman[72643]: 2024-02-05 06:34:51.031616104 +0000 UTC m=+0.474896161 container create 74971e2a5fe0bc7301558a3dae1627f6798275f02305ce3d0fa2e73270d2ab10 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T06:34:51.694 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:34:51] ENGINE Bus STOPPING 2024-02-05T06:34:51.694 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:34:51] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-05T06:34:51.694 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:34:51] ENGINE Bus STOPPED 2024-02-05T06:34:51.694 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:34:51] ENGINE Bus STARTING 2024-02-05T06:34:51.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:34:51 smithi028 podman[72643]: 2024-02-05 06:34:51.378797813 +0000 UTC m=+0.822077879 container init 74971e2a5fe0bc7301558a3dae1627f6798275f02305ce3d0fa2e73270d2ab10 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T06:34:51.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:34:51 smithi028 podman[72643]: 2024-02-05 06:34:51.381452313 +0000 UTC m=+0.824732360 container start 74971e2a5fe0bc7301558a3dae1627f6798275f02305ce3d0fa2e73270d2ab10 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T06:34:51.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a[72676]: ts=2024-02-05T06:34:51.392Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-02-05T06:34:51.695 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a[72676]: ts=2024-02-05T06:34:51.392Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-02-05T06:34:51.695 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a[72676]: ts=2024-02-05T06:34:51.394Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.28 port=9094 2024-02-05T06:34:51.695 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a[72676]: ts=2024-02-05T06:34:51.394Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-05T06:34:51.695 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a[72676]: ts=2024-02-05T06:34:51.417Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-05T06:34:51.695 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a[72676]: ts=2024-02-05T06:34:51.418Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-05T06:34:51.695 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a[72676]: ts=2024-02-05T06:34:51.421Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-02-05T06:34:51.695 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a[72676]: ts=2024-02-05T06:34:51.421Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-02-05T06:34:51.695 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:34:51 smithi028 bash[72643]: 74971e2a5fe0bc7301558a3dae1627f6798275f02305ce3d0fa2e73270d2ab10 2024-02-05T06:34:51.695 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:34:51 smithi028 systemd[1]: Started Ceph alertmanager.a for 9801e956-c3ef-11ee-95b6-87774f69a715. 2024-02-05T06:34:51.949 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:34:51.952 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-05T06:34:51.995 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:34:51] ENGINE Serving on http://:::9283 2024-02-05T06:34:51.995 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:34:51] ENGINE Bus STARTED 2024-02-05T06:34:51.996 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[37984]: 3.11 scrub starts 2024-02-05T06:34:51.996 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[37984]: 3.11 scrub ok 2024-02-05T06:34:51.996 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[37984]: 5.2 scrub starts 2024-02-05T06:34:51.996 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[37984]: 5.2 scrub ok 2024-02-05T06:34:51.996 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[37984]: 4.1e deep-scrub starts 2024-02-05T06:34:51.996 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[37984]: 4.1e deep-scrub ok 2024-02-05T06:34:51.996 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:51.996 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:51.996 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:51.996 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:51.996 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[37984]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-05T06:34:51.996 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:51.997 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:51.997 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-05T06:34:51.997 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[37984]: from='mon.? v1:172.21.15.28:0/1757070553' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-05T06:34:51.997 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:51.997 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[37984]: Deploying daemon grafana.a on smithi167 2024-02-05T06:34:51.997 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[29442]: 3.11 scrub starts 2024-02-05T06:34:51.997 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[29442]: 3.11 scrub ok 2024-02-05T06:34:51.997 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[29442]: 5.2 scrub starts 2024-02-05T06:34:51.997 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[29442]: 5.2 scrub ok 2024-02-05T06:34:51.997 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[29442]: 4.1e deep-scrub starts 2024-02-05T06:34:51.997 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[29442]: 4.1e deep-scrub ok 2024-02-05T06:34:51.997 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:51.997 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:51.998 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:51.998 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:51.998 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[29442]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-05T06:34:51.998 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:51.998 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:51.998 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-05T06:34:51.998 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[29442]: from='mon.? v1:172.21.15.28:0/1757070553' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-05T06:34:51.998 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:51.998 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:51 smithi028 ceph-mon[29442]: Deploying daemon grafana.a on smithi167 2024-02-05T06:34:52.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:51 smithi167 ceph-mon[32774]: 3.11 scrub starts 2024-02-05T06:34:52.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:51 smithi167 ceph-mon[32774]: 3.11 scrub ok 2024-02-05T06:34:52.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:51 smithi167 ceph-mon[32774]: 5.2 scrub starts 2024-02-05T06:34:52.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:51 smithi167 ceph-mon[32774]: 5.2 scrub ok 2024-02-05T06:34:52.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:51 smithi167 ceph-mon[32774]: 4.1e deep-scrub starts 2024-02-05T06:34:52.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:51 smithi167 ceph-mon[32774]: 4.1e deep-scrub ok 2024-02-05T06:34:52.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:51 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:52.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:51 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:52.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:51 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:52.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:51 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:52.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:51 smithi167 ceph-mon[32774]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-05T06:34:52.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:51 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:52.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:51 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:52.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:51 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-05T06:34:52.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:51 smithi167 ceph-mon[32774]: from='mon.? v1:172.21.15.28:0/1757070553' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-05T06:34:52.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:51 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:52.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:51 smithi167 ceph-mon[32774]: Deploying daemon grafana.a on smithi167 2024-02-05T06:34:52.694 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:34:52 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:34:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:34:52.724 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-02-05T06:34:50.151191+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1003,"num_read_kb":849,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1771792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":620,"ondisk_log_size":620,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":241276,"kb_used_data":8484,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749719940,"statfs":{"total":767960285184,"available":767713218560,"internally_reserved":0,"allocated":8687616,"data_stored":4795714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.255121"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":50,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.892759+0000","last_change":"2024-02-05T06:33:35.594797+0000","last_active":"2024-02-05T06:34:44.892759+0000","last_peered":"2024-02-05T06:34:44.892759+0000","last_clean":"2024-02-05T06:34:44.892759+0000","last_became_active":"2024-02-05T06:33:35.594467+0000","last_became_peered":"2024-02-05T06:33:35.594467+0000","last_unstale":"2024-02-05T06:34:44.892759+0000","last_undegraded":"2024-02-05T06:34:44.892759+0000","last_fullsized":"2024-02-05T06:34:44.892759+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:46.736983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":54,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890147+0000","last_change":"2024-02-05T06:33:33.586212+0000","last_active":"2024-02-05T06:34:44.890147+0000","last_peered":"2024-02-05T06:34:44.890147+0000","last_clean":"2024-02-05T06:34:44.890147+0000","last_became_active":"2024-02-05T06:33:33.585848+0000","last_became_peered":"2024-02-05T06:33:33.585848+0000","last_unstale":"2024-02-05T06:34:44.890147+0000","last_undegraded":"2024-02-05T06:34:44.890147+0000","last_fullsized":"2024-02-05T06:34:44.890147+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:51.677322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890647+0000","last_change":"2024-02-05T06:33:29.549864+0000","last_active":"2024-02-05T06:34:44.890647+0000","last_peered":"2024-02-05T06:34:44.890647+0000","last_clean":"2024-02-05T06:34:44.890647+0000","last_became_active":"2024-02-05T06:33:29.549592+0000","last_became_peered":"2024-02-05T06:33:29.549592+0000","last_unstale":"2024-02-05T06:34:44.890647+0000","last_undegraded":"2024-02-05T06:34:44.890647+0000","last_fullsized":"2024-02-05T06:34:44.890647+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:52.202922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":74,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.892665+0000","last_change":"2024-02-05T06:33:31.566905+0000","last_active":"2024-02-05T06:34:44.892665+0000","last_peered":"2024-02-05T06:34:44.892665+0000","last_clean":"2024-02-05T06:34:44.892665+0000","last_became_active":"2024-02-05T06:33:31.566571+0000","last_became_peered":"2024-02-05T06:33:31.566571+0000","last_unstale":"2024-02-05T06:34:44.892665+0000","last_undegraded":"2024-02-05T06:34:44.892665+0000","last_fullsized":"2024-02-05T06:34:44.892665+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:48.692724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096953+0000","last_change":"2024-02-05T06:33:35.668346+0000","last_active":"2024-02-05T06:34:45.096953+0000","last_peered":"2024-02-05T06:34:45.096953+0000","last_clean":"2024-02-05T06:34:45.096953+0000","last_became_active":"2024-02-05T06:33:35.667778+0000","last_became_peered":"2024-02-05T06:33:35.667778+0000","last_unstale":"2024-02-05T06:34:45.096953+0000","last_undegraded":"2024-02-05T06:34:45.096953+0000","last_fullsized":"2024-02-05T06:34:45.096953+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:59.948542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":54,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.892667+0000","last_change":"2024-02-05T06:33:33.583029+0000","last_active":"2024-02-05T06:34:44.892667+0000","last_peered":"2024-02-05T06:34:44.892667+0000","last_clean":"2024-02-05T06:34:44.892667+0000","last_became_active":"2024-02-05T06:33:33.582731+0000","last_became_peered":"2024-02-05T06:33:33.582731+0000","last_unstale":"2024-02-05T06:34:44.892667+0000","last_undegraded":"2024-02-05T06:34:44.892667+0000","last_fullsized":"2024-02-05T06:34:44.892667+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:56.016226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":71,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097695+0000","last_change":"2024-02-05T06:34:34.821722+0000","last_active":"2024-02-05T06:34:45.097695+0000","last_peered":"2024-02-05T06:34:45.097695+0000","last_clean":"2024-02-05T06:34:45.097695+0000","last_became_active":"2024-02-05T06:33:29.553864+0000","last_became_peered":"2024-02-05T06:33:29.553864+0000","last_unstale":"2024-02-05T06:34:45.097695+0000","last_undegraded":"2024-02-05T06:34:45.097695+0000","last_fullsized":"2024-02-05T06:34:45.097695+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:34.821603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:34.821603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:39.571760+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"88'22","reported_seq":99,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097275+0000","last_change":"2024-02-05T06:34:35.832461+0000","last_active":"2024-02-05T06:34:45.097275+0000","last_peered":"2024-02-05T06:34:45.097275+0000","last_clean":"2024-02-05T06:34:45.097275+0000","last_became_active":"2024-02-05T06:33:31.572544+0000","last_became_peered":"2024-02-05T06:33:31.572544+0000","last_unstale":"2024-02-05T06:34:45.097275+0000","last_undegraded":"2024-02-05T06:34:45.097275+0000","last_fullsized":"2024-02-05T06:34:45.097275+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'22","last_scrub_stamp":"2024-02-05T06:34:35.832368+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:34:35.832368+0000","last_clean_scrub_stamp":"2024-02-05T06:34:35.832368+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:43.028037+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.889644+0000","last_change":"2024-02-05T06:33:35.595744+0000","last_active":"2024-02-05T06:34:44.889644+0000","last_peered":"2024-02-05T06:34:44.889644+0000","last_clean":"2024-02-05T06:34:44.889644+0000","last_became_active":"2024-02-05T06:33:35.594860+0000","last_became_peered":"2024-02-05T06:33:35.594860+0000","last_unstale":"2024-02-05T06:34:44.889644+0000","last_undegraded":"2024-02-05T06:34:44.889644+0000","last_fullsized":"2024-02-05T06:34:44.889644+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095518+0000","last_change":"2024-02-05T06:34:34.187757+0000","last_active":"2024-02-05T06:34:45.095518+0000","last_peered":"2024-02-05T06:34:45.095518+0000","last_clean":"2024-02-05T06:34:45.095518+0000","last_became_active":"2024-02-05T06:33:33.585459+0000","last_became_peered":"2024-02-05T06:33:33.585459+0000","last_unstale":"2024-02-05T06:34:45.095518+0000","last_undegraded":"2024-02-05T06:34:45.095518+0000","last_fullsized":"2024-02-05T06:34:45.095518+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:34.187643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:34.187643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:35.727918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095719+0000","last_change":"2024-02-05T06:34:44.081424+0000","last_active":"2024-02-05T06:34:45.095719+0000","last_peered":"2024-02-05T06:34:45.095719+0000","last_clean":"2024-02-05T06:34:45.095719+0000","last_became_active":"2024-02-05T06:33:29.553251+0000","last_became_peered":"2024-02-05T06:33:29.553251+0000","last_unstale":"2024-02-05T06:34:45.095719+0000","last_undegraded":"2024-02-05T06:34:45.095719+0000","last_fullsized":"2024-02-05T06:34:45.095719+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:44.081318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:44.081318+0000","last_clean_scrub_stamp":"2024-02-05T06:34:44.081318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:49.865346+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'9","reported_seq":74,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.893086+0000","last_change":"2024-02-05T06:33:31.571199+0000","last_active":"2024-02-05T06:34:44.893086+0000","last_peered":"2024-02-05T06:34:44.893086+0000","last_clean":"2024-02-05T06:34:44.893086+0000","last_became_active":"2024-02-05T06:33:31.570995+0000","last_became_peered":"2024-02-05T06:33:31.570995+0000","last_unstale":"2024-02-05T06:34:44.893086+0000","last_undegraded":"2024-02-05T06:34:44.893086+0000","last_fullsized":"2024-02-05T06:34:44.893086+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095188+0000","last_change":"2024-02-05T06:33:35.600855+0000","last_active":"2024-02-05T06:34:45.095188+0000","last_peered":"2024-02-05T06:34:45.095188+0000","last_clean":"2024-02-05T06:34:45.095188+0000","last_became_active":"2024-02-05T06:33:35.600399+0000","last_became_peered":"2024-02-05T06:33:35.600399+0000","last_unstale":"2024-02-05T06:34:45.095188+0000","last_undegraded":"2024-02-05T06:34:45.095188+0000","last_fullsized":"2024-02-05T06:34:45.095188+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:59.096165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":55,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.922297+0000","last_change":"2024-02-05T06:33:33.584454+0000","last_active":"2024-02-05T06:34:45.922297+0000","last_peered":"2024-02-05T06:34:45.922297+0000","last_clean":"2024-02-05T06:34:45.922297+0000","last_became_active":"2024-02-05T06:33:33.584151+0000","last_became_peered":"2024-02-05T06:33:33.584151+0000","last_unstale":"2024-02-05T06:34:45.922297+0000","last_undegraded":"2024-02-05T06:34:45.922297+0000","last_fullsized":"2024-02-05T06:34:45.922297+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:48.436451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095937+0000","last_change":"2024-02-05T06:33:29.549247+0000","last_active":"2024-02-05T06:34:45.095937+0000","last_peered":"2024-02-05T06:34:45.095937+0000","last_clean":"2024-02-05T06:34:45.095937+0000","last_became_active":"2024-02-05T06:33:29.548931+0000","last_became_peered":"2024-02-05T06:33:29.548931+0000","last_unstale":"2024-02-05T06:34:45.095937+0000","last_undegraded":"2024-02-05T06:34:45.095937+0000","last_fullsized":"2024-02-05T06:34:45.095937+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":68,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.893390+0000","last_change":"2024-02-05T06:33:31.570226+0000","last_active":"2024-02-05T06:34:44.893390+0000","last_peered":"2024-02-05T06:34:44.893390+0000","last_clean":"2024-02-05T06:34:44.893390+0000","last_became_active":"2024-02-05T06:33:31.569654+0000","last_became_peered":"2024-02-05T06:33:31.569654+0000","last_unstale":"2024-02-05T06:34:44.893390+0000","last_undegraded":"2024-02-05T06:34:44.893390+0000","last_fullsized":"2024-02-05T06:34:44.893390+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:45.133474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":58,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.573515+0000","last_change":"2024-02-05T06:34:45.573557+0000","last_active":"2024-02-05T06:34:45.573515+0000","last_peered":"2024-02-05T06:34:45.573515+0000","last_clean":"2024-02-05T06:34:45.573515+0000","last_became_active":"2024-02-05T06:33:35.664772+0000","last_became_peered":"2024-02-05T06:33:35.664772+0000","last_unstale":"2024-02-05T06:34:45.573515+0000","last_undegraded":"2024-02-05T06:34:45.573515+0000","last_fullsized":"2024-02-05T06:34:45.573515+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:45.573432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:45.573432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:10.154762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":54,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890929+0000","last_change":"2024-02-05T06:33:33.588208+0000","last_active":"2024-02-05T06:34:44.890929+0000","last_peered":"2024-02-05T06:34:44.890929+0000","last_clean":"2024-02-05T06:34:44.890929+0000","last_became_active":"2024-02-05T06:33:33.587920+0000","last_became_peered":"2024-02-05T06:33:33.587920+0000","last_unstale":"2024-02-05T06:34:44.890929+0000","last_undegraded":"2024-02-05T06:34:44.890929+0000","last_fullsized":"2024-02-05T06:34:44.890929+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:54.461651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"66'1","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891771+0000","last_change":"2024-02-05T06:33:29.552104+0000","last_active":"2024-02-05T06:34:44.891771+0000","last_peered":"2024-02-05T06:34:44.891771+0000","last_clean":"2024-02-05T06:34:44.891771+0000","last_became_active":"2024-02-05T06:33:29.551820+0000","last_became_peered":"2024-02-05T06:33:29.551820+0000","last_unstale":"2024-02-05T06:34:44.891771+0000","last_undegraded":"2024-02-05T06:34:44.891771+0000","last_fullsized":"2024-02-05T06:34:44.891771+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"88'22","reported_seq":97,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.297835+0000","last_change":"2024-02-05T06:34:35.234726+0000","last_active":"2024-02-05T06:34:45.297835+0000","last_peered":"2024-02-05T06:34:45.297835+0000","last_clean":"2024-02-05T06:34:45.297835+0000","last_became_active":"2024-02-05T06:33:31.569763+0000","last_became_peered":"2024-02-05T06:33:31.569763+0000","last_unstale":"2024-02-05T06:34:45.297835+0000","last_undegraded":"2024-02-05T06:34:45.297835+0000","last_fullsized":"2024-02-05T06:34:45.297835+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'22","last_scrub_stamp":"2024-02-05T06:34:35.234636+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:34:35.234636+0000","last_clean_scrub_stamp":"2024-02-05T06:34:35.234636+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:53.234677+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097830+0000","last_change":"2024-02-05T06:33:35.665217+0000","last_active":"2024-02-05T06:34:45.097830+0000","last_peered":"2024-02-05T06:34:45.097830+0000","last_clean":"2024-02-05T06:34:45.097830+0000","last_became_active":"2024-02-05T06:33:35.664852+0000","last_became_peered":"2024-02-05T06:33:35.664852+0000","last_unstale":"2024-02-05T06:34:45.097830+0000","last_undegraded":"2024-02-05T06:34:45.097830+0000","last_fullsized":"2024-02-05T06:34:45.097830+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:47.798779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":54,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.893413+0000","last_change":"2024-02-05T06:33:33.587913+0000","last_active":"2024-02-05T06:34:44.893413+0000","last_peered":"2024-02-05T06:34:44.893413+0000","last_clean":"2024-02-05T06:34:44.893413+0000","last_became_active":"2024-02-05T06:33:33.587595+0000","last_became_peered":"2024-02-05T06:33:33.587595+0000","last_unstale":"2024-02-05T06:34:44.893413+0000","last_undegraded":"2024-02-05T06:34:44.893413+0000","last_fullsized":"2024-02-05T06:34:44.893413+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:53.273088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":70,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.892937+0000","last_change":"2024-02-05T06:34:38.623297+0000","last_active":"2024-02-05T06:34:44.892937+0000","last_peered":"2024-02-05T06:34:44.892937+0000","last_clean":"2024-02-05T06:34:44.892937+0000","last_became_active":"2024-02-05T06:33:29.551340+0000","last_became_peered":"2024-02-05T06:33:29.551340+0000","last_unstale":"2024-02-05T06:34:44.892937+0000","last_undegraded":"2024-02-05T06:34:44.892937+0000","last_fullsized":"2024-02-05T06:34:44.892937+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:38.623203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:38.623203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:49.053281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":100,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097390+0000","last_change":"2024-02-05T06:33:55.892463+0000","last_active":"2024-02-05T06:34:45.097390+0000","last_peered":"2024-02-05T06:34:45.097390+0000","last_clean":"2024-02-05T06:34:45.097390+0000","last_became_active":"2024-02-05T06:33:55.885324+0000","last_became_peered":"2024-02-05T06:33:55.885324+0000","last_unstale":"2024-02-05T06:34:45.097390+0000","last_undegraded":"2024-02-05T06:34:45.097390+0000","last_fullsized":"2024-02-05T06:34:45.097390+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:46.592833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"73'1","reported_seq":52,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.923186+0000","last_change":"2024-02-05T06:33:35.598794+0000","last_active":"2024-02-05T06:34:45.923186+0000","last_peered":"2024-02-05T06:34:45.923186+0000","last_clean":"2024-02-05T06:34:45.923186+0000","last_became_active":"2024-02-05T06:33:35.598501+0000","last_became_peered":"2024-02-05T06:33:35.598501+0000","last_unstale":"2024-02-05T06:34:45.923186+0000","last_undegraded":"2024-02-05T06:34:45.923186+0000","last_fullsized":"2024-02-05T06:34:45.923186+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:02.621665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"73'8","reported_seq":84,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.266561+0000","last_change":"2024-02-05T06:33:33.586794+0000","last_active":"2024-02-05T06:34:45.266561+0000","last_peered":"2024-02-05T06:34:45.266561+0000","last_clean":"2024-02-05T06:34:45.266561+0000","last_became_active":"2024-02-05T06:33:33.586389+0000","last_became_peered":"2024-02-05T06:33:33.586389+0000","last_unstale":"2024-02-05T06:34:45.266561+0000","last_undegraded":"2024-02-05T06:34:45.266561+0000","last_fullsized":"2024-02-05T06:34:45.266561+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:54.474866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890237+0000","last_change":"2024-02-05T06:33:29.546492+0000","last_active":"2024-02-05T06:34:44.890237+0000","last_peered":"2024-02-05T06:34:44.890237+0000","last_clean":"2024-02-05T06:34:44.890237+0000","last_became_active":"2024-02-05T06:33:29.546201+0000","last_became_peered":"2024-02-05T06:33:29.546201+0000","last_unstale":"2024-02-05T06:34:44.890237+0000","last_undegraded":"2024-02-05T06:34:44.890237+0000","last_fullsized":"2024-02-05T06:34:44.890237+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'10","reported_seq":74,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891064+0000","last_change":"2024-02-05T06:33:31.666735+0000","last_active":"2024-02-05T06:34:44.891064+0000","last_peered":"2024-02-05T06:34:44.891064+0000","last_clean":"2024-02-05T06:34:44.891064+0000","last_became_active":"2024-02-05T06:33:31.666481+0000","last_became_peered":"2024-02-05T06:33:31.666481+0000","last_unstale":"2024-02-05T06:34:44.891064+0000","last_undegraded":"2024-02-05T06:34:44.891064+0000","last_fullsized":"2024-02-05T06:34:44.891064+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":50,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891060+0000","last_change":"2024-02-05T06:33:35.594280+0000","last_active":"2024-02-05T06:34:44.891060+0000","last_peered":"2024-02-05T06:34:44.891060+0000","last_clean":"2024-02-05T06:34:44.891060+0000","last_became_active":"2024-02-05T06:33:35.593925+0000","last_became_peered":"2024-02-05T06:33:35.593925+0000","last_unstale":"2024-02-05T06:34:44.891060+0000","last_undegraded":"2024-02-05T06:34:44.891060+0000","last_fullsized":"2024-02-05T06:34:44.891060+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:48.199426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":55,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891396+0000","last_change":"2024-02-05T06:33:33.584707+0000","last_active":"2024-02-05T06:34:44.891396+0000","last_peered":"2024-02-05T06:34:44.891396+0000","last_clean":"2024-02-05T06:34:44.891396+0000","last_became_active":"2024-02-05T06:33:33.584390+0000","last_became_peered":"2024-02-05T06:33:33.584390+0000","last_unstale":"2024-02-05T06:34:44.891396+0000","last_undegraded":"2024-02-05T06:34:44.891396+0000","last_fullsized":"2024-02-05T06:34:44.891396+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096561+0000","last_change":"2024-02-05T06:33:29.550640+0000","last_active":"2024-02-05T06:34:45.096561+0000","last_peered":"2024-02-05T06:34:45.096561+0000","last_clean":"2024-02-05T06:34:45.096561+0000","last_became_active":"2024-02-05T06:33:29.550347+0000","last_became_peered":"2024-02-05T06:33:29.550347+0000","last_unstale":"2024-02-05T06:34:45.096561+0000","last_undegraded":"2024-02-05T06:34:45.096561+0000","last_fullsized":"2024-02-05T06:34:45.096561+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:47.916348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"92'16","reported_seq":90,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.270243+0000","last_change":"2024-02-05T06:34:45.270307+0000","last_active":"2024-02-05T06:34:45.270243+0000","last_peered":"2024-02-05T06:34:45.270243+0000","last_clean":"2024-02-05T06:34:45.270243+0000","last_became_active":"2024-02-05T06:33:31.573088+0000","last_became_peered":"2024-02-05T06:33:31.573088+0000","last_unstale":"2024-02-05T06:34:45.270243+0000","last_undegraded":"2024-02-05T06:34:45.270243+0000","last_fullsized":"2024-02-05T06:34:45.270243+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'16","last_scrub_stamp":"2024-02-05T06:34:45.270135+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:34:45.270135+0000","last_clean_scrub_stamp":"2024-02-05T06:34:45.270135+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:04.519387+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.922744+0000","last_change":"2024-02-05T06:33:35.597741+0000","last_active":"2024-02-05T06:34:45.922744+0000","last_peered":"2024-02-05T06:34:45.922744+0000","last_clean":"2024-02-05T06:34:45.922744+0000","last_became_active":"2024-02-05T06:33:35.597453+0000","last_became_peered":"2024-02-05T06:33:35.597453+0000","last_unstale":"2024-02-05T06:34:45.922744+0000","last_undegraded":"2024-02-05T06:34:45.922744+0000","last_fullsized":"2024-02-05T06:34:45.922744+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:49.858193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":54,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.265743+0000","last_change":"2024-02-05T06:33:33.585405+0000","last_active":"2024-02-05T06:34:45.265743+0000","last_peered":"2024-02-05T06:34:45.265743+0000","last_clean":"2024-02-05T06:34:45.265743+0000","last_became_active":"2024-02-05T06:33:33.585125+0000","last_became_peered":"2024-02-05T06:33:33.585125+0000","last_unstale":"2024-02-05T06:34:45.265743+0000","last_undegraded":"2024-02-05T06:34:45.265743+0000","last_fullsized":"2024-02-05T06:34:45.265743+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:50.106397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890564+0000","last_change":"2024-02-05T06:33:29.549815+0000","last_active":"2024-02-05T06:34:44.890564+0000","last_peered":"2024-02-05T06:34:44.890564+0000","last_clean":"2024-02-05T06:34:44.890564+0000","last_became_active":"2024-02-05T06:33:29.549204+0000","last_became_peered":"2024-02-05T06:33:29.549204+0000","last_unstale":"2024-02-05T06:34:44.890564+0000","last_undegraded":"2024-02-05T06:34:44.890564+0000","last_fullsized":"2024-02-05T06:34:44.890564+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:54.367234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"73'4","reported_seq":65,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096579+0000","last_change":"2024-02-05T06:33:31.573713+0000","last_active":"2024-02-05T06:34:45.096579+0000","last_peered":"2024-02-05T06:34:45.096579+0000","last_clean":"2024-02-05T06:34:45.096579+0000","last_became_active":"2024-02-05T06:33:31.573410+0000","last_became_peered":"2024-02-05T06:33:31.573410+0000","last_unstale":"2024-02-05T06:34:45.096579+0000","last_undegraded":"2024-02-05T06:34:45.096579+0000","last_fullsized":"2024-02-05T06:34:45.096579+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:58.822517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096768+0000","last_change":"2024-02-05T06:33:35.667939+0000","last_active":"2024-02-05T06:34:45.096768+0000","last_peered":"2024-02-05T06:34:45.096768+0000","last_clean":"2024-02-05T06:34:45.096768+0000","last_became_active":"2024-02-05T06:33:35.667602+0000","last_became_peered":"2024-02-05T06:33:35.667602+0000","last_unstale":"2024-02-05T06:34:45.096768+0000","last_undegraded":"2024-02-05T06:34:45.096768+0000","last_fullsized":"2024-02-05T06:34:45.096768+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.922284+0000","last_change":"2024-02-05T06:34:43.035728+0000","last_active":"2024-02-05T06:34:45.922284+0000","last_peered":"2024-02-05T06:34:45.922284+0000","last_clean":"2024-02-05T06:34:45.922284+0000","last_became_active":"2024-02-05T06:33:33.589428+0000","last_became_peered":"2024-02-05T06:33:33.589428+0000","last_unstale":"2024-02-05T06:34:45.922284+0000","last_undegraded":"2024-02-05T06:34:45.922284+0000","last_fullsized":"2024-02-05T06:34:45.922284+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:43.035635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:43.035635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:52.189018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095696+0000","last_change":"2024-02-05T06:33:29.551576+0000","last_active":"2024-02-05T06:34:45.095696+0000","last_peered":"2024-02-05T06:34:45.095696+0000","last_clean":"2024-02-05T06:34:45.095696+0000","last_became_active":"2024-02-05T06:33:29.551278+0000","last_became_peered":"2024-02-05T06:33:29.551278+0000","last_unstale":"2024-02-05T06:34:45.095696+0000","last_undegraded":"2024-02-05T06:34:45.095696+0000","last_fullsized":"2024-02-05T06:34:45.095696+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:58.094774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"73'11","reported_seq":78,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097095+0000","last_change":"2024-02-05T06:33:31.665074+0000","last_active":"2024-02-05T06:34:45.097095+0000","last_peered":"2024-02-05T06:34:45.097095+0000","last_clean":"2024-02-05T06:34:45.097095+0000","last_became_active":"2024-02-05T06:33:31.664743+0000","last_became_peered":"2024-02-05T06:33:31.664743+0000","last_unstale":"2024-02-05T06:34:45.097095+0000","last_undegraded":"2024-02-05T06:34:45.097095+0000","last_fullsized":"2024-02-05T06:34:45.097095+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:56.675510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"73'1","reported_seq":52,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.889401+0000","last_change":"2024-02-05T06:33:35.595450+0000","last_active":"2024-02-05T06:34:44.889401+0000","last_peered":"2024-02-05T06:34:44.889401+0000","last_clean":"2024-02-05T06:34:44.889401+0000","last_became_active":"2024-02-05T06:33:35.595031+0000","last_became_peered":"2024-02-05T06:33:35.595031+0000","last_unstale":"2024-02-05T06:34:44.889401+0000","last_undegraded":"2024-02-05T06:34:44.889401+0000","last_fullsized":"2024-02-05T06:34:44.889401+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":55,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097083+0000","last_change":"2024-02-05T06:33:33.594194+0000","last_active":"2024-02-05T06:34:45.097083+0000","last_peered":"2024-02-05T06:34:45.097083+0000","last_clean":"2024-02-05T06:34:45.097083+0000","last_became_active":"2024-02-05T06:33:33.593901+0000","last_became_peered":"2024-02-05T06:33:33.593901+0000","last_unstale":"2024-02-05T06:34:45.097083+0000","last_undegraded":"2024-02-05T06:34:45.097083+0000","last_fullsized":"2024-02-05T06:34:45.097083+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:46.645657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.923423+0000","last_change":"2024-02-05T06:33:29.559294+0000","last_active":"2024-02-05T06:34:45.923423+0000","last_peered":"2024-02-05T06:34:45.923423+0000","last_clean":"2024-02-05T06:34:45.923423+0000","last_became_active":"2024-02-05T06:33:29.559006+0000","last_became_peered":"2024-02-05T06:33:29.559006+0000","last_unstale":"2024-02-05T06:34:45.923423+0000","last_undegraded":"2024-02-05T06:34:45.923423+0000","last_fullsized":"2024-02-05T06:34:45.923423+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:52.625563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":74,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891566+0000","last_change":"2024-02-05T06:33:31.568831+0000","last_active":"2024-02-05T06:34:44.891566+0000","last_peered":"2024-02-05T06:34:44.891566+0000","last_clean":"2024-02-05T06:34:44.891566+0000","last_became_active":"2024-02-05T06:33:31.568529+0000","last_became_peered":"2024-02-05T06:33:31.568529+0000","last_unstale":"2024-02-05T06:34:44.891566+0000","last_undegraded":"2024-02-05T06:34:44.891566+0000","last_fullsized":"2024-02-05T06:34:44.891566+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096601+0000","last_change":"2024-02-05T06:33:35.597747+0000","last_active":"2024-02-05T06:34:45.096601+0000","last_peered":"2024-02-05T06:34:45.096601+0000","last_clean":"2024-02-05T06:34:45.096601+0000","last_became_active":"2024-02-05T06:33:35.597399+0000","last_became_peered":"2024-02-05T06:33:35.597399+0000","last_unstale":"2024-02-05T06:34:45.096601+0000","last_undegraded":"2024-02-05T06:34:45.096601+0000","last_fullsized":"2024-02-05T06:34:45.096601+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":62,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891472+0000","last_change":"2024-02-05T06:34:40.361349+0000","last_active":"2024-02-05T06:34:44.891472+0000","last_peered":"2024-02-05T06:34:44.891472+0000","last_clean":"2024-02-05T06:34:44.891472+0000","last_became_active":"2024-02-05T06:33:33.588461+0000","last_became_peered":"2024-02-05T06:33:33.588461+0000","last_unstale":"2024-02-05T06:34:44.891472+0000","last_undegraded":"2024-02-05T06:34:44.891472+0000","last_fullsized":"2024-02-05T06:34:44.891472+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:40.361273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:40.361273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:50.089342+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":62,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.894203+0000","last_change":"2024-02-05T06:33:29.552783+0000","last_active":"2024-02-05T06:34:44.894203+0000","last_peered":"2024-02-05T06:34:44.894203+0000","last_clean":"2024-02-05T06:34:44.894203+0000","last_became_active":"2024-02-05T06:33:29.552336+0000","last_became_peered":"2024-02-05T06:33:29.552336+0000","last_unstale":"2024-02-05T06:34:44.894203+0000","last_undegraded":"2024-02-05T06:34:44.894203+0000","last_fullsized":"2024-02-05T06:34:44.894203+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"73'11","reported_seq":77,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.893567+0000","last_change":"2024-02-05T06:33:31.569835+0000","last_active":"2024-02-05T06:34:44.893567+0000","last_peered":"2024-02-05T06:34:44.893567+0000","last_clean":"2024-02-05T06:34:44.893567+0000","last_became_active":"2024-02-05T06:33:31.569488+0000","last_became_peered":"2024-02-05T06:33:31.569488+0000","last_unstale":"2024-02-05T06:34:44.893567+0000","last_undegraded":"2024-02-05T06:34:44.893567+0000","last_fullsized":"2024-02-05T06:34:44.893567+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:51.980754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890759+0000","last_change":"2024-02-05T06:33:35.600109+0000","last_active":"2024-02-05T06:34:44.890759+0000","last_peered":"2024-02-05T06:34:44.890759+0000","last_clean":"2024-02-05T06:34:44.890759+0000","last_became_active":"2024-02-05T06:33:35.599808+0000","last_became_peered":"2024-02-05T06:33:35.599808+0000","last_unstale":"2024-02-05T06:34:44.890759+0000","last_undegraded":"2024-02-05T06:34:44.890759+0000","last_fullsized":"2024-02-05T06:34:44.890759+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:50.168225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":77,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096042+0000","last_change":"2024-02-05T06:33:33.585921+0000","last_active":"2024-02-05T06:34:45.096042+0000","last_peered":"2024-02-05T06:34:45.096042+0000","last_clean":"2024-02-05T06:34:45.096042+0000","last_became_active":"2024-02-05T06:33:33.585642+0000","last_became_peered":"2024-02-05T06:33:33.585642+0000","last_unstale":"2024-02-05T06:34:45.096042+0000","last_undegraded":"2024-02-05T06:34:45.096042+0000","last_fullsized":"2024-02-05T06:34:45.096042+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:48.217760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.922923+0000","last_change":"2024-02-05T06:33:29.557136+0000","last_active":"2024-02-05T06:34:45.922923+0000","last_peered":"2024-02-05T06:34:45.922923+0000","last_clean":"2024-02-05T06:34:45.922923+0000","last_became_active":"2024-02-05T06:33:29.556832+0000","last_became_peered":"2024-02-05T06:33:29.556832+0000","last_unstale":"2024-02-05T06:34:45.922923+0000","last_undegraded":"2024-02-05T06:34:45.922923+0000","last_fullsized":"2024-02-05T06:34:45.922923+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:52.324493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'10","reported_seq":82,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097179+0000","last_change":"2024-02-05T06:34:41.873963+0000","last_active":"2024-02-05T06:34:45.097179+0000","last_peered":"2024-02-05T06:34:45.097179+0000","last_clean":"2024-02-05T06:34:45.097179+0000","last_became_active":"2024-02-05T06:33:31.666808+0000","last_became_peered":"2024-02-05T06:33:31.666808+0000","last_unstale":"2024-02-05T06:34:45.097179+0000","last_undegraded":"2024-02-05T06:34:45.097179+0000","last_fullsized":"2024-02-05T06:34:45.097179+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-02-05T06:34:41.873876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:41.873876+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:08.435407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":50,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.893956+0000","last_change":"2024-02-05T06:33:35.597787+0000","last_active":"2024-02-05T06:34:44.893956+0000","last_peered":"2024-02-05T06:34:44.893956+0000","last_clean":"2024-02-05T06:34:44.893956+0000","last_became_active":"2024-02-05T06:33:35.597494+0000","last_became_peered":"2024-02-05T06:33:35.597494+0000","last_unstale":"2024-02-05T06:34:44.893956+0000","last_undegraded":"2024-02-05T06:34:44.893956+0000","last_fullsized":"2024-02-05T06:34:44.893956+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:54.313659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":79,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097598+0000","last_change":"2024-02-05T06:33:33.586465+0000","last_active":"2024-02-05T06:34:45.097598+0000","last_peered":"2024-02-05T06:34:45.097598+0000","last_clean":"2024-02-05T06:34:45.097598+0000","last_became_active":"2024-02-05T06:33:33.586043+0000","last_became_peered":"2024-02-05T06:33:33.586043+0000","last_unstale":"2024-02-05T06:34:45.097598+0000","last_undegraded":"2024-02-05T06:34:45.097598+0000","last_fullsized":"2024-02-05T06:34:45.097598+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.889924+0000","last_change":"2024-02-05T06:33:29.552610+0000","last_active":"2024-02-05T06:34:44.889924+0000","last_peered":"2024-02-05T06:34:44.889924+0000","last_clean":"2024-02-05T06:34:44.889924+0000","last_became_active":"2024-02-05T06:33:29.552170+0000","last_became_peered":"2024-02-05T06:33:29.552170+0000","last_unstale":"2024-02-05T06:34:44.889924+0000","last_undegraded":"2024-02-05T06:34:44.889924+0000","last_fullsized":"2024-02-05T06:34:44.889924+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"73'9","reported_seq":81,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096168+0000","last_change":"2024-02-05T06:34:36.172574+0000","last_active":"2024-02-05T06:34:45.096168+0000","last_peered":"2024-02-05T06:34:45.096168+0000","last_clean":"2024-02-05T06:34:45.096168+0000","last_became_active":"2024-02-05T06:33:31.666678+0000","last_became_peered":"2024-02-05T06:33:31.666678+0000","last_unstale":"2024-02-05T06:34:45.096168+0000","last_undegraded":"2024-02-05T06:34:45.096168+0000","last_fullsized":"2024-02-05T06:34:45.096168+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:34:36.172444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:36.172444+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:50.358190+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.298026+0000","last_change":"2024-02-05T06:34:43.228873+0000","last_active":"2024-02-05T06:34:45.298026+0000","last_peered":"2024-02-05T06:34:45.298026+0000","last_clean":"2024-02-05T06:34:45.298026+0000","last_became_active":"2024-02-05T06:33:35.601508+0000","last_became_peered":"2024-02-05T06:33:35.601508+0000","last_unstale":"2024-02-05T06:34:45.298026+0000","last_undegraded":"2024-02-05T06:34:45.298026+0000","last_fullsized":"2024-02-05T06:34:45.298026+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:43.228774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:43.228774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:57.334710+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097815+0000","last_change":"2024-02-05T06:34:42.866516+0000","last_active":"2024-02-05T06:34:45.097815+0000","last_peered":"2024-02-05T06:34:45.097815+0000","last_clean":"2024-02-05T06:34:45.097815+0000","last_became_active":"2024-02-05T06:33:33.593025+0000","last_became_peered":"2024-02-05T06:33:33.593025+0000","last_unstale":"2024-02-05T06:34:45.097815+0000","last_undegraded":"2024-02-05T06:34:45.097815+0000","last_fullsized":"2024-02-05T06:34:45.097815+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:42.866394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:42.866394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:43.991930+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.923191+0000","last_change":"2024-02-05T06:33:29.554219+0000","last_active":"2024-02-05T06:34:45.923191+0000","last_peered":"2024-02-05T06:34:45.923191+0000","last_clean":"2024-02-05T06:34:45.923191+0000","last_became_active":"2024-02-05T06:33:29.553880+0000","last_became_peered":"2024-02-05T06:33:29.553880+0000","last_unstale":"2024-02-05T06:34:45.923191+0000","last_undegraded":"2024-02-05T06:34:45.923191+0000","last_fullsized":"2024-02-05T06:34:45.923191+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:47.111162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"73'9","reported_seq":83,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891187+0000","last_change":"2024-02-05T06:34:43.760971+0000","last_active":"2024-02-05T06:34:44.891187+0000","last_peered":"2024-02-05T06:34:44.891187+0000","last_clean":"2024-02-05T06:34:44.891187+0000","last_became_active":"2024-02-05T06:33:31.665766+0000","last_became_peered":"2024-02-05T06:33:31.665766+0000","last_unstale":"2024-02-05T06:34:44.891187+0000","last_undegraded":"2024-02-05T06:34:44.891187+0000","last_fullsized":"2024-02-05T06:34:44.891187+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:34:43.760888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:43.760888+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:44.661663+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.923416+0000","last_change":"2024-02-05T06:33:35.666076+0000","last_active":"2024-02-05T06:34:45.923416+0000","last_peered":"2024-02-05T06:34:45.923416+0000","last_clean":"2024-02-05T06:34:45.923416+0000","last_became_active":"2024-02-05T06:33:35.665596+0000","last_became_peered":"2024-02-05T06:33:35.665596+0000","last_unstale":"2024-02-05T06:34:45.923416+0000","last_undegraded":"2024-02-05T06:34:45.923416+0000","last_fullsized":"2024-02-05T06:34:45.923416+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:51.044976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096424+0000","last_change":"2024-02-05T06:34:45.096449+0000","last_active":"2024-02-05T06:34:45.096424+0000","last_peered":"2024-02-05T06:34:45.096424+0000","last_clean":"2024-02-05T06:34:45.096424+0000","last_became_active":"2024-02-05T06:33:33.586768+0000","last_became_peered":"2024-02-05T06:33:33.586768+0000","last_unstale":"2024-02-05T06:34:45.096424+0000","last_undegraded":"2024-02-05T06:34:45.096424+0000","last_fullsized":"2024-02-05T06:34:45.096424+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:45.096340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:45.096340+0000","last_clean_scrub_stamp":"2024-02-05T06:34:45.096340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:12.226977+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":70,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.265669+0000","last_change":"2024-02-05T06:34:30.204064+0000","last_active":"2024-02-05T06:34:45.265669+0000","last_peered":"2024-02-05T06:34:45.265669+0000","last_clean":"2024-02-05T06:34:45.265669+0000","last_became_active":"2024-02-05T06:33:29.554427+0000","last_became_peered":"2024-02-05T06:33:29.554427+0000","last_unstale":"2024-02-05T06:34:45.265669+0000","last_undegraded":"2024-02-05T06:34:45.265669+0000","last_fullsized":"2024-02-05T06:34:45.265669+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:30.203967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:30.203967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:35.531249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":68,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097892+0000","last_change":"2024-02-05T06:33:31.666151+0000","last_active":"2024-02-05T06:34:45.097892+0000","last_peered":"2024-02-05T06:34:45.097892+0000","last_clean":"2024-02-05T06:34:45.097892+0000","last_became_active":"2024-02-05T06:33:31.665847+0000","last_became_peered":"2024-02-05T06:33:31.665847+0000","last_unstale":"2024-02-05T06:34:45.097892+0000","last_undegraded":"2024-02-05T06:34:45.097892+0000","last_fullsized":"2024-02-05T06:34:45.097892+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095574+0000","last_change":"2024-02-05T06:33:35.665928+0000","last_active":"2024-02-05T06:34:45.095574+0000","last_peered":"2024-02-05T06:34:45.095574+0000","last_clean":"2024-02-05T06:34:45.095574+0000","last_became_active":"2024-02-05T06:33:35.665581+0000","last_became_peered":"2024-02-05T06:33:35.665581+0000","last_unstale":"2024-02-05T06:34:45.095574+0000","last_undegraded":"2024-02-05T06:34:45.095574+0000","last_fullsized":"2024-02-05T06:34:45.095574+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:47.218454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":75,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.923730+0000","last_change":"2024-02-05T06:33:29.558514+0000","last_active":"2024-02-05T06:34:45.923730+0000","last_peered":"2024-02-05T06:34:45.923730+0000","last_clean":"2024-02-05T06:34:45.923730+0000","last_became_active":"2024-02-05T06:33:29.557522+0000","last_became_peered":"2024-02-05T06:33:29.557522+0000","last_unstale":"2024-02-05T06:34:45.923730+0000","last_undegraded":"2024-02-05T06:34:45.923730+0000","last_fullsized":"2024-02-05T06:34:45.923730+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:51.837286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"73'15","reported_seq":82,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095250+0000","last_change":"2024-02-05T06:33:31.666410+0000","last_active":"2024-02-05T06:34:45.095250+0000","last_peered":"2024-02-05T06:34:45.095250+0000","last_clean":"2024-02-05T06:34:45.095250+0000","last_became_active":"2024-02-05T06:33:31.666056+0000","last_became_peered":"2024-02-05T06:33:31.666056+0000","last_unstale":"2024-02-05T06:34:45.095250+0000","last_undegraded":"2024-02-05T06:34:45.095250+0000","last_fullsized":"2024-02-05T06:34:45.095250+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:56.860240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"73'8","reported_seq":79,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.922293+0000","last_change":"2024-02-05T06:33:33.590132+0000","last_active":"2024-02-05T06:34:45.922293+0000","last_peered":"2024-02-05T06:34:45.922293+0000","last_clean":"2024-02-05T06:34:45.922293+0000","last_became_active":"2024-02-05T06:33:33.589535+0000","last_became_peered":"2024-02-05T06:33:33.589535+0000","last_unstale":"2024-02-05T06:34:45.922293+0000","last_undegraded":"2024-02-05T06:34:45.922293+0000","last_fullsized":"2024-02-05T06:34:45.922293+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096781+0000","last_change":"2024-02-05T06:33:35.601680+0000","last_active":"2024-02-05T06:34:45.096781+0000","last_peered":"2024-02-05T06:34:45.096781+0000","last_clean":"2024-02-05T06:34:45.096781+0000","last_became_active":"2024-02-05T06:33:35.601030+0000","last_became_peered":"2024-02-05T06:33:35.601030+0000","last_unstale":"2024-02-05T06:34:45.096781+0000","last_undegraded":"2024-02-05T06:34:45.096781+0000","last_fullsized":"2024-02-05T06:34:45.096781+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:53.348191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.923545+0000","last_change":"2024-02-05T06:33:29.558929+0000","last_active":"2024-02-05T06:34:45.923545+0000","last_peered":"2024-02-05T06:34:45.923545+0000","last_clean":"2024-02-05T06:34:45.923545+0000","last_became_active":"2024-02-05T06:33:29.557284+0000","last_became_peered":"2024-02-05T06:33:29.557284+0000","last_unstale":"2024-02-05T06:34:45.923545+0000","last_undegraded":"2024-02-05T06:34:45.923545+0000","last_fullsized":"2024-02-05T06:34:45.923545+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:54.175804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'12","reported_seq":81,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.893212+0000","last_change":"2024-02-05T06:33:31.570242+0000","last_active":"2024-02-05T06:34:44.893212+0000","last_peered":"2024-02-05T06:34:44.893212+0000","last_clean":"2024-02-05T06:34:44.893212+0000","last_became_active":"2024-02-05T06:33:31.569924+0000","last_became_peered":"2024-02-05T06:33:31.569924+0000","last_unstale":"2024-02-05T06:34:44.893212+0000","last_undegraded":"2024-02-05T06:34:44.893212+0000","last_fullsized":"2024-02-05T06:34:44.893212+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:47.971041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.297549+0000","last_change":"2024-02-05T06:34:40.272941+0000","last_active":"2024-02-05T06:34:45.297549+0000","last_peered":"2024-02-05T06:34:45.297549+0000","last_clean":"2024-02-05T06:34:45.297549+0000","last_became_active":"2024-02-05T06:33:33.592546+0000","last_became_peered":"2024-02-05T06:33:33.592546+0000","last_unstale":"2024-02-05T06:34:45.297549+0000","last_undegraded":"2024-02-05T06:34:45.297549+0000","last_fullsized":"2024-02-05T06:34:45.297549+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:40.272852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:40.272852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:07.055905+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.923123+0000","last_change":"2024-02-05T06:33:35.599830+0000","last_active":"2024-02-05T06:34:45.923123+0000","last_peered":"2024-02-05T06:34:45.923123+0000","last_clean":"2024-02-05T06:34:45.923123+0000","last_became_active":"2024-02-05T06:33:35.599527+0000","last_became_peered":"2024-02-05T06:33:35.599527+0000","last_unstale":"2024-02-05T06:34:45.923123+0000","last_undegraded":"2024-02-05T06:34:45.923123+0000","last_fullsized":"2024-02-05T06:34:45.923123+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:58.253161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.297822+0000","last_change":"2024-02-05T06:34:42.265315+0000","last_active":"2024-02-05T06:34:45.297822+0000","last_peered":"2024-02-05T06:34:45.297822+0000","last_clean":"2024-02-05T06:34:45.297822+0000","last_became_active":"2024-02-05T06:33:33.589203+0000","last_became_peered":"2024-02-05T06:33:33.589203+0000","last_unstale":"2024-02-05T06:34:45.297822+0000","last_undegraded":"2024-02-05T06:34:45.297822+0000","last_fullsized":"2024-02-05T06:34:45.297822+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:42.265214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:42.265214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:53.433345+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.923587+0000","last_change":"2024-02-05T06:33:29.556849+0000","last_active":"2024-02-05T06:34:45.923587+0000","last_peered":"2024-02-05T06:34:45.923587+0000","last_clean":"2024-02-05T06:34:45.923587+0000","last_became_active":"2024-02-05T06:33:29.556541+0000","last_became_peered":"2024-02-05T06:33:29.556541+0000","last_unstale":"2024-02-05T06:34:45.923587+0000","last_undegraded":"2024-02-05T06:34:45.923587+0000","last_fullsized":"2024-02-05T06:34:45.923587+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:49.995627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"73'19","reported_seq":97,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.265984+0000","last_change":"2024-02-05T06:34:44.243017+0000","last_active":"2024-02-05T06:34:45.265984+0000","last_peered":"2024-02-05T06:34:45.265984+0000","last_clean":"2024-02-05T06:34:45.265984+0000","last_became_active":"2024-02-05T06:33:31.664699+0000","last_became_peered":"2024-02-05T06:33:31.664699+0000","last_unstale":"2024-02-05T06:34:45.265984+0000","last_undegraded":"2024-02-05T06:34:45.265984+0000","last_fullsized":"2024-02-05T06:34:45.265984+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-02-05T06:34:44.242897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:44.242897+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:01.187407+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.889844+0000","last_change":"2024-02-05T06:33:35.598200+0000","last_active":"2024-02-05T06:34:44.889844+0000","last_peered":"2024-02-05T06:34:44.889844+0000","last_clean":"2024-02-05T06:34:44.889844+0000","last_became_active":"2024-02-05T06:33:35.597863+0000","last_became_peered":"2024-02-05T06:33:35.597863+0000","last_unstale":"2024-02-05T06:34:44.889844+0000","last_undegraded":"2024-02-05T06:34:44.889844+0000","last_fullsized":"2024-02-05T06:34:44.889844+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:55.272938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.299123+0000","last_change":"2024-02-05T06:34:45.299145+0000","last_active":"2024-02-05T06:34:45.299123+0000","last_peered":"2024-02-05T06:34:45.299123+0000","last_clean":"2024-02-05T06:34:45.299123+0000","last_became_active":"2024-02-05T06:33:33.586895+0000","last_became_peered":"2024-02-05T06:33:33.586895+0000","last_unstale":"2024-02-05T06:34:45.299123+0000","last_undegraded":"2024-02-05T06:34:45.299123+0000","last_fullsized":"2024-02-05T06:34:45.299123+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:45.299055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:45.299055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:03.861496+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095649+0000","last_change":"2024-02-05T06:33:29.556254+0000","last_active":"2024-02-05T06:34:45.095649+0000","last_peered":"2024-02-05T06:34:45.095649+0000","last_clean":"2024-02-05T06:34:45.095649+0000","last_became_active":"2024-02-05T06:33:29.555950+0000","last_became_peered":"2024-02-05T06:33:29.555950+0000","last_unstale":"2024-02-05T06:34:45.095649+0000","last_undegraded":"2024-02-05T06:34:45.095649+0000","last_fullsized":"2024-02-05T06:34:45.095649+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:52.533616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"73'9","reported_seq":83,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891706+0000","last_change":"2024-02-05T06:34:36.779561+0000","last_active":"2024-02-05T06:34:44.891706+0000","last_peered":"2024-02-05T06:34:44.891706+0000","last_clean":"2024-02-05T06:34:44.891706+0000","last_became_active":"2024-02-05T06:33:31.564572+0000","last_became_peered":"2024-02-05T06:33:31.564572+0000","last_unstale":"2024-02-05T06:34:44.891706+0000","last_undegraded":"2024-02-05T06:34:44.891706+0000","last_fullsized":"2024-02-05T06:34:44.891706+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:34:36.779466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:36.779466+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:02.144960+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097537+0000","last_change":"2024-02-05T06:33:35.598477+0000","last_active":"2024-02-05T06:34:45.097537+0000","last_peered":"2024-02-05T06:34:45.097537+0000","last_clean":"2024-02-05T06:34:45.097537+0000","last_became_active":"2024-02-05T06:33:35.597750+0000","last_became_peered":"2024-02-05T06:33:35.597750+0000","last_unstale":"2024-02-05T06:34:45.097537+0000","last_undegraded":"2024-02-05T06:34:45.097537+0000","last_fullsized":"2024-02-05T06:34:45.097537+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:53.938845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":72,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095334+0000","last_change":"2024-02-05T06:33:32.571415+0000","last_active":"2024-02-05T06:34:45.095334+0000","last_peered":"2024-02-05T06:34:45.095334+0000","last_clean":"2024-02-05T06:34:45.095334+0000","last_became_active":"2024-02-05T06:33:28.542228+0000","last_became_peered":"2024-02-05T06:33:28.542228+0000","last_unstale":"2024-02-05T06:34:45.095334+0000","last_undegraded":"2024-02-05T06:34:45.095334+0000","last_fullsized":"2024-02-05T06:34:45.095334+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:33:27.524724+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001220348,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":62,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.892994+0000","last_change":"2024-02-05T06:33:29.554533+0000","last_active":"2024-02-05T06:34:44.892994+0000","last_peered":"2024-02-05T06:34:44.892994+0000","last_clean":"2024-02-05T06:34:44.892994+0000","last_became_active":"2024-02-05T06:33:29.553882+0000","last_became_peered":"2024-02-05T06:33:29.553882+0000","last_unstale":"2024-02-05T06:34:44.892994+0000","last_undegraded":"2024-02-05T06:34:44.892994+0000","last_fullsized":"2024-02-05T06:34:44.892994+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:56.794367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":71,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095445+0000","last_change":"2024-02-05T06:33:55.896865+0000","last_active":"2024-02-05T06:34:45.095445+0000","last_peered":"2024-02-05T06:34:45.095445+0000","last_clean":"2024-02-05T06:34:45.095445+0000","last_became_active":"2024-02-05T06:33:55.884601+0000","last_became_peered":"2024-02-05T06:33:55.884601+0000","last_unstale":"2024-02-05T06:34:45.095445+0000","last_undegraded":"2024-02-05T06:34:45.095445+0000","last_fullsized":"2024-02-05T06:34:45.095445+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:55.551394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":55,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890270+0000","last_change":"2024-02-05T06:33:33.589668+0000","last_active":"2024-02-05T06:34:44.890270+0000","last_peered":"2024-02-05T06:34:44.890270+0000","last_clean":"2024-02-05T06:34:44.890270+0000","last_became_active":"2024-02-05T06:33:33.589348+0000","last_became_peered":"2024-02-05T06:33:33.589348+0000","last_unstale":"2024-02-05T06:34:44.890270+0000","last_undegraded":"2024-02-05T06:34:44.890270+0000","last_fullsized":"2024-02-05T06:34:44.890270+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.923731+0000","last_change":"2024-02-05T06:33:35.666590+0000","last_active":"2024-02-05T06:34:45.923731+0000","last_peered":"2024-02-05T06:34:45.923731+0000","last_clean":"2024-02-05T06:34:45.923731+0000","last_became_active":"2024-02-05T06:33:35.666324+0000","last_became_peered":"2024-02-05T06:33:35.666324+0000","last_unstale":"2024-02-05T06:34:45.923731+0000","last_undegraded":"2024-02-05T06:34:45.923731+0000","last_fullsized":"2024-02-05T06:34:45.923731+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:51.202340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":62,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890767+0000","last_change":"2024-02-05T06:33:29.552003+0000","last_active":"2024-02-05T06:34:44.890767+0000","last_peered":"2024-02-05T06:34:44.890767+0000","last_clean":"2024-02-05T06:34:44.890767+0000","last_became_active":"2024-02-05T06:33:29.551549+0000","last_became_peered":"2024-02-05T06:33:29.551549+0000","last_unstale":"2024-02-05T06:34:44.890767+0000","last_undegraded":"2024-02-05T06:34:44.890767+0000","last_fullsized":"2024-02-05T06:34:44.890767+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:56.472943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":70,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.297568+0000","last_change":"2024-02-05T06:33:32.575511+0000","last_active":"2024-02-05T06:34:45.297568+0000","last_peered":"2024-02-05T06:34:45.297568+0000","last_clean":"2024-02-05T06:34:45.297568+0000","last_became_active":"2024-02-05T06:33:28.542977+0000","last_became_peered":"2024-02-05T06:33:28.542977+0000","last_unstale":"2024-02-05T06:34:45.297568+0000","last_undegraded":"2024-02-05T06:34:45.297568+0000","last_fullsized":"2024-02-05T06:34:45.297568+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:33:27.524724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091640799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'13","reported_seq":85,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891357+0000","last_change":"2024-02-05T06:33:31.562891+0000","last_active":"2024-02-05T06:34:44.891357+0000","last_peered":"2024-02-05T06:34:44.891357+0000","last_clean":"2024-02-05T06:34:44.891357+0000","last_became_active":"2024-02-05T06:33:31.560082+0000","last_became_peered":"2024-02-05T06:33:31.560082+0000","last_unstale":"2024-02-05T06:34:44.891357+0000","last_undegraded":"2024-02-05T06:34:44.891357+0000","last_fullsized":"2024-02-05T06:34:44.891357+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:50.831110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.298136+0000","last_change":"2024-02-05T06:34:34.240413+0000","last_active":"2024-02-05T06:34:45.298136+0000","last_peered":"2024-02-05T06:34:45.298136+0000","last_clean":"2024-02-05T06:34:45.298136+0000","last_became_active":"2024-02-05T06:33:33.587593+0000","last_became_peered":"2024-02-05T06:33:33.587593+0000","last_unstale":"2024-02-05T06:34:45.298136+0000","last_undegraded":"2024-02-05T06:34:45.298136+0000","last_fullsized":"2024-02-05T06:34:45.298136+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:34.240331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:34.240331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:49.655873+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095927+0000","last_change":"2024-02-05T06:33:35.600278+0000","last_active":"2024-02-05T06:34:45.095927+0000","last_peered":"2024-02-05T06:34:45.095927+0000","last_clean":"2024-02-05T06:34:45.095927+0000","last_became_active":"2024-02-05T06:33:35.600004+0000","last_became_peered":"2024-02-05T06:33:35.600004+0000","last_unstale":"2024-02-05T06:34:45.095927+0000","last_undegraded":"2024-02-05T06:34:45.095927+0000","last_fullsized":"2024-02-05T06:34:45.095927+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:59.635646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":317,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.923419+0000","last_change":"2024-02-05T06:33:45.752222+0000","last_active":"2024-02-05T06:34:45.923419+0000","last_peered":"2024-02-05T06:34:45.923419+0000","last_clean":"2024-02-05T06:34:45.923419+0000","last_became_active":"2024-02-05T06:33:19.862877+0000","last_became_peered":"2024-02-05T06:33:19.862877+0000","last_unstale":"2024-02-05T06:34:45.923419+0000","last_undegraded":"2024-02-05T06:34:45.923419+0000","last_fullsized":"2024-02-05T06:34:45.923419+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-02-05T06:33:45.752124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:33:45.752124+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:54.331188+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":71,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097177+0000","last_change":"2024-02-05T06:34:44.894533+0000","last_active":"2024-02-05T06:34:45.097177+0000","last_peered":"2024-02-05T06:34:45.097177+0000","last_clean":"2024-02-05T06:34:45.097177+0000","last_became_active":"2024-02-05T06:33:29.552698+0000","last_became_peered":"2024-02-05T06:33:29.552698+0000","last_unstale":"2024-02-05T06:34:45.097177+0000","last_undegraded":"2024-02-05T06:34:45.097177+0000","last_fullsized":"2024-02-05T06:34:45.097177+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:44.894443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:44.894443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:54.598610+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"73'16","reported_seq":96,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.266439+0000","last_change":"2024-02-05T06:33:31.666149+0000","last_active":"2024-02-05T06:34:45.266439+0000","last_peered":"2024-02-05T06:34:45.266439+0000","last_clean":"2024-02-05T06:34:45.266439+0000","last_became_active":"2024-02-05T06:33:31.665797+0000","last_became_peered":"2024-02-05T06:33:31.665797+0000","last_unstale":"2024-02-05T06:34:45.266439+0000","last_undegraded":"2024-02-05T06:34:45.266439+0000","last_fullsized":"2024-02-05T06:34:45.266439+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:00.032828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":55,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.922387+0000","last_change":"2024-02-05T06:33:33.591727+0000","last_active":"2024-02-05T06:34:45.922387+0000","last_peered":"2024-02-05T06:34:45.922387+0000","last_clean":"2024-02-05T06:34:45.922387+0000","last_became_active":"2024-02-05T06:33:33.591484+0000","last_became_peered":"2024-02-05T06:33:33.591484+0000","last_unstale":"2024-02-05T06:34:45.922387+0000","last_undegraded":"2024-02-05T06:34:45.922387+0000","last_fullsized":"2024-02-05T06:34:45.922387+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:56.324621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":50,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890553+0000","last_change":"2024-02-05T06:33:35.600982+0000","last_active":"2024-02-05T06:34:44.890553+0000","last_peered":"2024-02-05T06:34:44.890553+0000","last_clean":"2024-02-05T06:34:44.890553+0000","last_became_active":"2024-02-05T06:33:35.600656+0000","last_became_peered":"2024-02-05T06:33:35.600656+0000","last_unstale":"2024-02-05T06:34:44.890553+0000","last_undegraded":"2024-02-05T06:34:44.890553+0000","last_fullsized":"2024-02-05T06:34:44.890553+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":71,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890127+0000","last_change":"2024-02-05T06:34:44.781805+0000","last_active":"2024-02-05T06:34:44.890127+0000","last_peered":"2024-02-05T06:34:44.890127+0000","last_clean":"2024-02-05T06:34:44.890127+0000","last_became_active":"2024-02-05T06:33:29.544399+0000","last_became_peered":"2024-02-05T06:33:29.544399+0000","last_unstale":"2024-02-05T06:34:44.890127+0000","last_undegraded":"2024-02-05T06:34:44.890127+0000","last_fullsized":"2024-02-05T06:34:44.890127+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:44.781704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:44.781704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:06.382435+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":152,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.923254+0000","last_change":"2024-02-05T06:34:41.991897+0000","last_active":"2024-02-05T06:34:45.923254+0000","last_peered":"2024-02-05T06:34:45.923254+0000","last_clean":"2024-02-05T06:34:45.923254+0000","last_became_active":"2024-02-05T06:33:28.544852+0000","last_became_peered":"2024-02-05T06:33:28.544852+0000","last_unstale":"2024-02-05T06:34:45.923254+0000","last_undegraded":"2024-02-05T06:34:45.923254+0000","last_fullsized":"2024-02-05T06:34:45.923254+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T06:34:41.991777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:34:41.991777+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:10.879161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00074151600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":69,"num_read_kb":64,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":77,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890848+0000","last_change":"2024-02-05T06:33:31.564936+0000","last_active":"2024-02-05T06:34:44.890848+0000","last_peered":"2024-02-05T06:34:44.890848+0000","last_clean":"2024-02-05T06:34:44.890848+0000","last_became_active":"2024-02-05T06:33:31.564604+0000","last_became_peered":"2024-02-05T06:33:31.564604+0000","last_unstale":"2024-02-05T06:34:44.890848+0000","last_undegraded":"2024-02-05T06:34:44.890848+0000","last_fullsized":"2024-02-05T06:34:44.890848+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":53,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096253+0000","last_change":"2024-02-05T06:33:33.589962+0000","last_active":"2024-02-05T06:34:45.096253+0000","last_peered":"2024-02-05T06:34:45.096253+0000","last_clean":"2024-02-05T06:34:45.096253+0000","last_became_active":"2024-02-05T06:33:33.589388+0000","last_became_peered":"2024-02-05T06:33:33.589388+0000","last_unstale":"2024-02-05T06:34:45.096253+0000","last_undegraded":"2024-02-05T06:34:45.096253+0000","last_fullsized":"2024-02-05T06:34:45.096253+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:01.833071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":50,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890056+0000","last_change":"2024-02-05T06:33:35.665208+0000","last_active":"2024-02-05T06:34:44.890056+0000","last_peered":"2024-02-05T06:34:44.890056+0000","last_clean":"2024-02-05T06:34:44.890056+0000","last_became_active":"2024-02-05T06:33:35.664858+0000","last_became_peered":"2024-02-05T06:33:35.664858+0000","last_unstale":"2024-02-05T06:34:44.890056+0000","last_undegraded":"2024-02-05T06:34:44.890056+0000","last_fullsized":"2024-02-05T06:34:44.890056+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:53.295103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"66'1","reported_seq":76,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890784+0000","last_change":"2024-02-05T06:34:36.457175+0000","last_active":"2024-02-05T06:34:44.890784+0000","last_peered":"2024-02-05T06:34:44.890784+0000","last_clean":"2024-02-05T06:34:44.890784+0000","last_became_active":"2024-02-05T06:33:29.548227+0000","last_became_peered":"2024-02-05T06:33:29.548227+0000","last_unstale":"2024-02-05T06:34:44.890784+0000","last_undegraded":"2024-02-05T06:34:44.890784+0000","last_fullsized":"2024-02-05T06:34:44.890784+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:34:36.457059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:36.457059+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:45.403222+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'19","reported_seq":95,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891811+0000","last_change":"2024-02-05T06:33:31.667636+0000","last_active":"2024-02-05T06:34:44.891811+0000","last_peered":"2024-02-05T06:34:44.891811+0000","last_clean":"2024-02-05T06:34:44.891811+0000","last_became_active":"2024-02-05T06:33:31.667340+0000","last_became_peered":"2024-02-05T06:33:31.667340+0000","last_unstale":"2024-02-05T06:34:44.891811+0000","last_undegraded":"2024-02-05T06:34:44.891811+0000","last_fullsized":"2024-02-05T06:34:44.891811+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:45.776316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":54,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.266072+0000","last_change":"2024-02-05T06:33:33.588431+0000","last_active":"2024-02-05T06:34:45.266072+0000","last_peered":"2024-02-05T06:34:45.266072+0000","last_clean":"2024-02-05T06:34:45.266072+0000","last_became_active":"2024-02-05T06:33:33.588163+0000","last_became_peered":"2024-02-05T06:33:33.588163+0000","last_unstale":"2024-02-05T06:34:45.266072+0000","last_undegraded":"2024-02-05T06:34:45.266072+0000","last_fullsized":"2024-02-05T06:34:45.266072+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:48.917974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.889406+0000","last_change":"2024-02-05T06:33:35.600831+0000","last_active":"2024-02-05T06:34:44.889406+0000","last_peered":"2024-02-05T06:34:44.889406+0000","last_clean":"2024-02-05T06:34:44.889406+0000","last_became_active":"2024-02-05T06:33:35.600485+0000","last_became_peered":"2024-02-05T06:33:35.600485+0000","last_unstale":"2024-02-05T06:34:44.889406+0000","last_undegraded":"2024-02-05T06:34:44.889406+0000","last_fullsized":"2024-02-05T06:34:44.889406+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:51.006304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095967+0000","last_change":"2024-02-05T06:34:43.098655+0000","last_active":"2024-02-05T06:34:45.095967+0000","last_peered":"2024-02-05T06:34:45.095967+0000","last_clean":"2024-02-05T06:34:45.095967+0000","last_became_active":"2024-02-05T06:33:29.554940+0000","last_became_peered":"2024-02-05T06:33:29.554940+0000","last_unstale":"2024-02-05T06:34:45.095967+0000","last_undegraded":"2024-02-05T06:34:45.095967+0000","last_fullsized":"2024-02-05T06:34:45.095967+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:43.098546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:43.098546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:01.990483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"73'10","reported_seq":73,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891541+0000","last_change":"2024-02-05T06:33:31.570032+0000","last_active":"2024-02-05T06:34:44.891541+0000","last_peered":"2024-02-05T06:34:44.891541+0000","last_clean":"2024-02-05T06:34:44.891541+0000","last_became_active":"2024-02-05T06:33:31.569680+0000","last_became_peered":"2024-02-05T06:33:31.569680+0000","last_unstale":"2024-02-05T06:34:44.891541+0000","last_undegraded":"2024-02-05T06:34:44.891541+0000","last_fullsized":"2024-02-05T06:34:44.891541+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:57.891117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"73'8","reported_seq":79,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891588+0000","last_change":"2024-02-05T06:33:33.582537+0000","last_active":"2024-02-05T06:34:44.891588+0000","last_peered":"2024-02-05T06:34:44.891588+0000","last_clean":"2024-02-05T06:34:44.891588+0000","last_became_active":"2024-02-05T06:33:33.582187+0000","last_became_peered":"2024-02-05T06:33:33.582187+0000","last_unstale":"2024-02-05T06:34:44.891588+0000","last_undegraded":"2024-02-05T06:34:44.891588+0000","last_fullsized":"2024-02-05T06:34:44.891588+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":59,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.923213+0000","last_change":"2024-02-05T06:34:41.001666+0000","last_active":"2024-02-05T06:34:45.923213+0000","last_peered":"2024-02-05T06:34:45.923213+0000","last_clean":"2024-02-05T06:34:45.923213+0000","last_became_active":"2024-02-05T06:33:35.666315+0000","last_became_peered":"2024-02-05T06:33:35.666315+0000","last_unstale":"2024-02-05T06:34:45.923213+0000","last_undegraded":"2024-02-05T06:34:45.923213+0000","last_fullsized":"2024-02-05T06:34:45.923213+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:41.001570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:41.001570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:53.416960+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":64,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890941+0000","last_change":"2024-02-05T06:33:29.546683+0000","last_active":"2024-02-05T06:34:44.890941+0000","last_peered":"2024-02-05T06:34:44.890941+0000","last_clean":"2024-02-05T06:34:44.890941+0000","last_became_active":"2024-02-05T06:33:29.546405+0000","last_became_peered":"2024-02-05T06:33:29.546405+0000","last_unstale":"2024-02-05T06:34:44.890941+0000","last_undegraded":"2024-02-05T06:34:44.890941+0000","last_fullsized":"2024-02-05T06:34:44.890941+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:45.453250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'14","reported_seq":79,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.893000+0000","last_change":"2024-02-05T06:33:31.564209+0000","last_active":"2024-02-05T06:34:44.893000+0000","last_peered":"2024-02-05T06:34:44.893000+0000","last_clean":"2024-02-05T06:34:44.893000+0000","last_became_active":"2024-02-05T06:33:31.563836+0000","last_became_peered":"2024-02-05T06:33:31.563836+0000","last_unstale":"2024-02-05T06:34:44.893000+0000","last_undegraded":"2024-02-05T06:34:44.893000+0000","last_fullsized":"2024-02-05T06:34:44.893000+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:47.565469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"73'8","reported_seq":79,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096844+0000","last_change":"2024-02-05T06:33:33.591940+0000","last_active":"2024-02-05T06:34:45.096844+0000","last_peered":"2024-02-05T06:34:45.096844+0000","last_clean":"2024-02-05T06:34:45.096844+0000","last_became_active":"2024-02-05T06:33:33.591646+0000","last_became_peered":"2024-02-05T06:33:33.591646+0000","last_unstale":"2024-02-05T06:34:45.096844+0000","last_undegraded":"2024-02-05T06:34:45.096844+0000","last_fullsized":"2024-02-05T06:34:45.096844+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":50,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.892762+0000","last_change":"2024-02-05T06:33:35.600296+0000","last_active":"2024-02-05T06:34:44.892762+0000","last_peered":"2024-02-05T06:34:44.892762+0000","last_clean":"2024-02-05T06:34:44.892762+0000","last_became_active":"2024-02-05T06:33:35.599805+0000","last_became_peered":"2024-02-05T06:33:35.599805+0000","last_unstale":"2024-02-05T06:34:44.892762+0000","last_undegraded":"2024-02-05T06:34:44.892762+0000","last_fullsized":"2024-02-05T06:34:44.892762+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:03.912813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097711+0000","last_change":"2024-02-05T06:33:29.557121+0000","last_active":"2024-02-05T06:34:45.097711+0000","last_peered":"2024-02-05T06:34:45.097711+0000","last_clean":"2024-02-05T06:34:45.097711+0000","last_became_active":"2024-02-05T06:33:29.556002+0000","last_became_peered":"2024-02-05T06:33:29.556002+0000","last_unstale":"2024-02-05T06:34:45.097711+0000","last_undegraded":"2024-02-05T06:34:45.097711+0000","last_fullsized":"2024-02-05T06:34:45.097711+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:48.546981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"73'18","reported_seq":91,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096701+0000","last_change":"2024-02-05T06:33:31.666298+0000","last_active":"2024-02-05T06:34:45.096701+0000","last_peered":"2024-02-05T06:34:45.096701+0000","last_clean":"2024-02-05T06:34:45.096701+0000","last_became_active":"2024-02-05T06:33:31.665995+0000","last_became_peered":"2024-02-05T06:33:31.665995+0000","last_unstale":"2024-02-05T06:34:45.096701+0000","last_undegraded":"2024-02-05T06:34:45.096701+0000","last_fullsized":"2024-02-05T06:34:45.096701+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:47.201122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":54,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.892715+0000","last_change":"2024-02-05T06:33:33.587254+0000","last_active":"2024-02-05T06:34:44.892715+0000","last_peered":"2024-02-05T06:34:44.892715+0000","last_clean":"2024-02-05T06:34:44.892715+0000","last_became_active":"2024-02-05T06:33:33.586935+0000","last_became_peered":"2024-02-05T06:33:33.586935+0000","last_unstale":"2024-02-05T06:34:44.892715+0000","last_undegraded":"2024-02-05T06:34:44.892715+0000","last_fullsized":"2024-02-05T06:34:44.892715+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095645+0000","last_change":"2024-02-05T06:33:35.593461+0000","last_active":"2024-02-05T06:34:45.095645+0000","last_peered":"2024-02-05T06:34:45.095645+0000","last_clean":"2024-02-05T06:34:45.095645+0000","last_became_active":"2024-02-05T06:33:35.593210+0000","last_became_peered":"2024-02-05T06:33:35.593210+0000","last_unstale":"2024-02-05T06:34:45.095645+0000","last_undegraded":"2024-02-05T06:34:45.095645+0000","last_fullsized":"2024-02-05T06:34:45.095645+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":86,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.893963+0000","last_change":"2024-02-05T06:34:40.619333+0000","last_active":"2024-02-05T06:34:44.893963+0000","last_peered":"2024-02-05T06:34:44.893963+0000","last_clean":"2024-02-05T06:34:44.893963+0000","last_became_active":"2024-02-05T06:33:33.586753+0000","last_became_peered":"2024-02-05T06:33:33.586753+0000","last_unstale":"2024-02-05T06:34:44.893963+0000","last_undegraded":"2024-02-05T06:34:44.893963+0000","last_fullsized":"2024-02-05T06:34:44.893963+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:34:40.619192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:40.619192+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:48.417494+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096676+0000","last_change":"2024-02-05T06:33:29.557073+0000","last_active":"2024-02-05T06:34:45.096676+0000","last_peered":"2024-02-05T06:34:45.096676+0000","last_clean":"2024-02-05T06:34:45.096676+0000","last_became_active":"2024-02-05T06:33:29.555919+0000","last_became_peered":"2024-02-05T06:33:29.555919+0000","last_unstale":"2024-02-05T06:34:45.096676+0000","last_undegraded":"2024-02-05T06:34:45.096676+0000","last_fullsized":"2024-02-05T06:34:45.096676+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:54.542025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"73'15","reported_seq":83,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891508+0000","last_change":"2024-02-05T06:33:31.572905+0000","last_active":"2024-02-05T06:34:44.891508+0000","last_peered":"2024-02-05T06:34:44.891508+0000","last_clean":"2024-02-05T06:34:44.891508+0000","last_became_active":"2024-02-05T06:33:31.572603+0000","last_became_peered":"2024-02-05T06:33:31.572603+0000","last_unstale":"2024-02-05T06:34:44.891508+0000","last_undegraded":"2024-02-05T06:34:44.891508+0000","last_fullsized":"2024-02-05T06:34:44.891508+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:49.104679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096917+0000","last_change":"2024-02-05T06:33:35.668457+0000","last_active":"2024-02-05T06:34:45.096917+0000","last_peered":"2024-02-05T06:34:45.096917+0000","last_clean":"2024-02-05T06:34:45.096917+0000","last_became_active":"2024-02-05T06:33:35.668020+0000","last_became_peered":"2024-02-05T06:33:35.668020+0000","last_unstale":"2024-02-05T06:34:45.096917+0000","last_undegraded":"2024-02-05T06:34:45.096917+0000","last_fullsized":"2024-02-05T06:34:45.096917+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:55.572974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":53,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096246+0000","last_change":"2024-02-05T06:33:33.583518+0000","last_active":"2024-02-05T06:34:45.096246+0000","last_peered":"2024-02-05T06:34:45.096246+0000","last_clean":"2024-02-05T06:34:45.096246+0000","last_became_active":"2024-02-05T06:33:33.583029+0000","last_became_peered":"2024-02-05T06:33:33.583029+0000","last_unstale":"2024-02-05T06:34:45.096246+0000","last_undegraded":"2024-02-05T06:34:45.096246+0000","last_fullsized":"2024-02-05T06:34:45.096246+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:49.769832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":62,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.892790+0000","last_change":"2024-02-05T06:33:29.553248+0000","last_active":"2024-02-05T06:34:44.892790+0000","last_peered":"2024-02-05T06:34:44.892790+0000","last_clean":"2024-02-05T06:34:44.892790+0000","last_became_active":"2024-02-05T06:33:29.552415+0000","last_became_peered":"2024-02-05T06:33:29.552415+0000","last_unstale":"2024-02-05T06:34:44.892790+0000","last_undegraded":"2024-02-05T06:34:44.892790+0000","last_fullsized":"2024-02-05T06:34:44.892790+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:56.521015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"73'11","reported_seq":95,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.893832+0000","last_change":"2024-02-05T06:33:55.891989+0000","last_active":"2024-02-05T06:34:44.893832+0000","last_peered":"2024-02-05T06:34:44.893832+0000","last_clean":"2024-02-05T06:34:44.893832+0000","last_became_active":"2024-02-05T06:33:55.885523+0000","last_became_peered":"2024-02-05T06:33:55.885523+0000","last_unstale":"2024-02-05T06:34:44.893832+0000","last_undegraded":"2024-02-05T06:34:44.893832+0000","last_fullsized":"2024-02-05T06:34:44.893832+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.298252+0000","last_change":"2024-02-05T06:33:35.598926+0000","last_active":"2024-02-05T06:34:45.298252+0000","last_peered":"2024-02-05T06:34:45.298252+0000","last_clean":"2024-02-05T06:34:45.298252+0000","last_became_active":"2024-02-05T06:33:35.598593+0000","last_became_peered":"2024-02-05T06:33:35.598593+0000","last_unstale":"2024-02-05T06:34:45.298252+0000","last_undegraded":"2024-02-05T06:34:45.298252+0000","last_fullsized":"2024-02-05T06:34:45.298252+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":54,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890955+0000","last_change":"2024-02-05T06:33:33.586971+0000","last_active":"2024-02-05T06:34:44.890955+0000","last_peered":"2024-02-05T06:34:44.890955+0000","last_clean":"2024-02-05T06:34:44.890955+0000","last_became_active":"2024-02-05T06:33:33.586598+0000","last_became_peered":"2024-02-05T06:33:33.586598+0000","last_unstale":"2024-02-05T06:34:44.890955+0000","last_undegraded":"2024-02-05T06:34:44.890955+0000","last_fullsized":"2024-02-05T06:34:44.890955+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:49.228193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":62,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.266181+0000","last_change":"2024-02-05T06:33:29.550887+0000","last_active":"2024-02-05T06:34:45.266181+0000","last_peered":"2024-02-05T06:34:45.266181+0000","last_clean":"2024-02-05T06:34:45.266181+0000","last_became_active":"2024-02-05T06:33:29.550612+0000","last_became_peered":"2024-02-05T06:33:29.550612+0000","last_unstale":"2024-02-05T06:34:45.266181+0000","last_undegraded":"2024-02-05T06:34:45.266181+0000","last_fullsized":"2024-02-05T06:34:45.266181+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:55.717410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"73'17","reported_seq":86,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.893580+0000","last_change":"2024-02-05T06:33:31.566941+0000","last_active":"2024-02-05T06:34:44.893580+0000","last_peered":"2024-02-05T06:34:44.893580+0000","last_clean":"2024-02-05T06:34:44.893580+0000","last_became_active":"2024-02-05T06:33:31.566576+0000","last_became_peered":"2024-02-05T06:33:31.566576+0000","last_unstale":"2024-02-05T06:34:44.893580+0000","last_undegraded":"2024-02-05T06:34:44.893580+0000","last_fullsized":"2024-02-05T06:34:44.893580+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:00.400128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":50,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.893991+0000","last_change":"2024-02-05T06:33:35.597953+0000","last_active":"2024-02-05T06:34:44.893991+0000","last_peered":"2024-02-05T06:34:44.893991+0000","last_clean":"2024-02-05T06:34:44.893991+0000","last_became_active":"2024-02-05T06:33:35.597460+0000","last_became_peered":"2024-02-05T06:33:35.597460+0000","last_unstale":"2024-02-05T06:34:44.893991+0000","last_undegraded":"2024-02-05T06:34:44.893991+0000","last_fullsized":"2024-02-05T06:34:44.893991+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:00.813671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":77,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.297914+0000","last_change":"2024-02-05T06:33:33.588787+0000","last_active":"2024-02-05T06:34:45.297914+0000","last_peered":"2024-02-05T06:34:45.297914+0000","last_clean":"2024-02-05T06:34:45.297914+0000","last_became_active":"2024-02-05T06:33:33.588478+0000","last_became_peered":"2024-02-05T06:33:33.588478+0000","last_unstale":"2024-02-05T06:34:45.297914+0000","last_undegraded":"2024-02-05T06:34:45.297914+0000","last_fullsized":"2024-02-05T06:34:45.297914+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:46.947053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":71,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097232+0000","last_change":"2024-02-05T06:34:29.822718+0000","last_active":"2024-02-05T06:34:45.097232+0000","last_peered":"2024-02-05T06:34:45.097232+0000","last_clean":"2024-02-05T06:34:45.097232+0000","last_became_active":"2024-02-05T06:33:29.545073+0000","last_became_peered":"2024-02-05T06:33:29.545073+0000","last_unstale":"2024-02-05T06:34:45.097232+0000","last_undegraded":"2024-02-05T06:34:45.097232+0000","last_fullsized":"2024-02-05T06:34:45.097232+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:29.822634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:29.822634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:37.035501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"73'10","reported_seq":73,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.893424+0000","last_change":"2024-02-05T06:33:31.568457+0000","last_active":"2024-02-05T06:34:44.893424+0000","last_peered":"2024-02-05T06:34:44.893424+0000","last_clean":"2024-02-05T06:34:44.893424+0000","last_became_active":"2024-02-05T06:33:31.568074+0000","last_became_peered":"2024-02-05T06:33:31.568074+0000","last_unstale":"2024-02-05T06:34:44.893424+0000","last_undegraded":"2024-02-05T06:34:44.893424+0000","last_fullsized":"2024-02-05T06:34:44.893424+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:52.367398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":413,"ondisk_log_size":413,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":64,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103187,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32604,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712548,"statfs":{"total":95995035648,"available":95961649152,"internally_reserved":0,"allocated":1458176,"data_stored":970343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.78300000000000003}]},{"osd":1,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.84199999999999997}]},{"osd":2,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.73099999999999998}]},{"osd":3,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.748},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.85999999999999999}]},{"osd":4,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.53900000000000003}]},{"osd":5,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.56299999999999994}]},{"osd":6,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.61899999999999999}]}]},{"osd":6,"up_from":49,"seq":210453397529,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32564,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712588,"statfs":{"total":95995035648,"available":95961690112,"internally_reserved":0,"allocated":1421312,"data_stored":962385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:33:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.376}]},{"osd":1,"last update":"Mon Feb 5 06:33:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.40400000000000003}]},{"osd":2,"last update":"Mon Feb 5 06:33:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.45000000000000001}]},{"osd":3,"last update":"Mon Feb 5 06:33:51 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.58999999999999997}]},{"osd":4,"last update":"Mon Feb 5 06:33:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.355,"5min":1.355,"15min":1.355},"last":0.497},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.57399999999999995}]},{"osd":5,"last update":"Mon Feb 5 06:33:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.61499999999999999}]},{"osd":7,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.60599999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27884,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717268,"statfs":{"total":95995035648,"available":95966482432,"internally_reserved":0,"allocated":884736,"data_stored":380510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:33:47 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.35799999999999998,"15min":0.33700000000000002},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.375,"15min":0.34599999999999997},"min":{"1min":0.26100000000000001,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.36099999999999999}]},{"osd":2,"last update":"Mon Feb 5 06:34:13 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.39900000000000002,"15min":0.39200000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.59999999999999998,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.38700000000000001,"15min":0.377},"min":{"1min":0.26900000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.32700000000000001}]},{"osd":3,"last update":"Mon Feb 5 06:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.42999999999999999,"15min":0.42199999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.48799999999999999,"15min":0.47799999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.83399999999999996,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.312}]},{"osd":4,"last update":"Mon Feb 5 06:34:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56699999999999995,"15min":0.56100000000000005},"min":{"1min":0.434,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.62,"5min":0.58399999999999996,"15min":0.57799999999999996},"min":{"1min":0.39400000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.69199999999999995}]},{"osd":5,"last update":"Mon Feb 5 06:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.92800000000000005,"15min":0.98499999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86499999999999999,"5min":8.2509999999999994,"15min":8.2509999999999994},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.624,"15min":0.63400000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.73599999999999999}]},{"osd":6,"last update":"Mon Feb 5 06:33:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.67500000000000004}]},{"osd":7,"last update":"Mon Feb 5 06:34:16 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.65900000000000003},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":8.2219999999999995,"5min":8.2219999999999995,"15min":8.2219999999999995},"last":0.58899999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32612,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712540,"statfs":{"total":95995035648,"available":95961640960,"internally_reserved":0,"allocated":1466368,"data_stored":970099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 06:33:46 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.33700000000000002,"15min":0.318},"min":{"1min":0.26800000000000002,"5min":0.188,"15min":0.188},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.35499999999999998,"15min":0.33200000000000002},"min":{"1min":0.32300000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.60399999999999998,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.30199999999999999}]},{"osd":2,"last update":"Mon Feb 5 06:34:11 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.435,"15min":0.41299999999999998},"min":{"1min":0.315,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.42599999999999999},"min":{"1min":0.28999999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.42499999999999999}]},{"osd":3,"last update":"Mon Feb 5 06:34:37 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.47499999999999998,"15min":0.46200000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":4.3289999999999997,"5min":4.3289999999999997,"15min":4.3289999999999997},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45100000000000001},"min":{"1min":0.28499999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.68500000000000005,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.38400000000000001}]},{"osd":4,"last update":"Mon Feb 5 06:34:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.59799999999999998,"15min":0.59499999999999997},"min":{"1min":0.441,"5min":0.372,"15min":0.372},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.60899999999999999,"15min":0.60499999999999998},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.46500000000000002}]},{"osd":5,"last update":"Mon Feb 5 06:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.628,"15min":0.63600000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.73399999999999999,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.64200000000000002,"15min":0.64700000000000002},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.77900000000000003,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.48499999999999999}]},{"osd":6,"last update":"Mon Feb 5 06:33:49 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.47599999999999998}]},{"osd":7,"last update":"Mon Feb 5 06:34:16 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.442},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":5.2670000000000003,"5min":5.2670000000000003,"15min":5.2670000000000003},"last":0.52700000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31996,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713156,"statfs":{"total":95995035648,"available":95962271744,"internally_reserved":0,"allocated":839680,"data_stored":373335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:34:15 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.38100000000000001,"15min":0.36599999999999999},"min":{"1min":0.27000000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.59999999999999998,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.41199999999999998,"15min":0.38400000000000001},"min":{"1min":0.32000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.58099999999999996,"5min":0.752,"15min":0.752},"last":0.23899999999999999}]},{"osd":1,"last update":"Mon Feb 5 06:34:15 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39900000000000002,"15min":0.38900000000000001},"min":{"1min":0.26200000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.56000000000000005,"5min":0.626,"15min":0.626},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.44800000000000001,"15min":0.42899999999999999},"min":{"1min":0.33600000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.68300000000000005,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.32300000000000001}]},{"osd":3,"last update":"Mon Feb 5 06:34:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.47999999999999998,"15min":0.46999999999999997},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72499999999999998,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.47999999999999998,"15min":0.48199999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.66200000000000003,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.33800000000000002}]},{"osd":4,"last update":"Mon Feb 5 06:34:00 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.65000000000000002,"15min":0.64700000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0009999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.495,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.90000000000000002,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.47399999999999998}]},{"osd":5,"last update":"Mon Feb 5 06:34:30 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.69199999999999995,"15min":0.70299999999999996},"min":{"1min":0.47199999999999998,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.89600000000000002,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.66600000000000004,"15min":0.68000000000000005},"min":{"1min":0.42399999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.85299999999999998,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.55600000000000005}]},{"osd":6,"last update":"Mon Feb 5 06:33:51 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.7210000000000001,"15min":1.7210000000000001},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":17.143999999999998,"5min":17.143999999999998,"15min":17.143999999999998},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.59499999999999997}]},{"osd":7,"last update":"Mon Feb 5 06:34:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.502},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.61199999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149736,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27876,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717276,"statfs":{"total":95995035648,"available":95966490624,"internally_reserved":0,"allocated":876544,"data_stored":379417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:34:44 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.38600000000000001,"15min":0.39300000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.52000000000000002,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.44500000000000001,"15min":0.45400000000000001},"min":{"1min":0.316,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.55000000000000004,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.34000000000000002}]},{"osd":1,"last update":"Mon Feb 5 06:34:44 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.45800000000000002,"15min":0.47199999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.57099999999999995,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.44500000000000001,"15min":0.45300000000000001},"min":{"1min":0.27700000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.57599999999999996,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.40100000000000002}]},{"osd":2,"last update":"Mon Feb 5 06:34:44 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.48099999999999998,"15min":0.495},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.64600000000000002,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.46000000000000002,"15min":0.46899999999999997},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.64800000000000002,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.32400000000000001}]},{"osd":4,"last update":"Mon Feb 5 06:34:05 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.59899999999999998,"15min":0.61499999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.627,"5min":1.008,"15min":1.008},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.59099999999999997,"15min":0.60399999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.67900000000000005,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.56499999999999995}]},{"osd":5,"last update":"Mon Feb 5 06:34:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.60099999999999998,"15min":0.60699999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.73999999999999999,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.61699999999999999,"15min":0.627},"min":{"1min":0.44,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.748,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.51200000000000001}]},{"osd":6,"last update":"Mon Feb 5 06:33:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.49199999999999999}]},{"osd":7,"last update":"Mon Feb 5 06:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.439},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.54900000000000004}]}]},{"osd":4,"up_from":31,"seq":133143986210,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":872448,"data_stored":379529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:34:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.57899999999999996,"15min":0.58899999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.71699999999999997,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.54300000000000004,"15min":0.54100000000000004},"min":{"1min":0.42099999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.70399999999999996,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.47599999999999998}]},{"osd":1,"last update":"Mon Feb 5 06:34:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.55900000000000005,"15min":0.56200000000000006},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.69599999999999995,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.58999999999999997,"15min":0.59999999999999998},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.68600000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.38900000000000001}]},{"osd":2,"last update":"Mon Feb 5 06:34:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.76900000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58099999999999996,"15min":0.58699999999999997},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.80600000000000005,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.54700000000000004}]},{"osd":3,"last update":"Mon Feb 5 06:34:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59799999999999998,"15min":0.60099999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.375,"5min":1.375,"15min":1.375},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.58099999999999996,"15min":0.58699999999999997},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.746,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.496}]},{"osd":5,"last update":"Mon Feb 5 06:34:28 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.40999999999999998,"15min":0.40200000000000002},"min":{"1min":0.27500000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.442,"15min":0.439},"min":{"1min":0.317,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.64100000000000001}]},{"osd":6,"last update":"Mon Feb 5 06:33:51 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.65800000000000003}]},{"osd":7,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.61699999999999999}]}]},{"osd":5,"up_from":40,"seq":171798691869,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27868,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717284,"statfs":{"total":95995035648,"available":95966498816,"internally_reserved":0,"allocated":868352,"data_stored":380096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.53100000000000003,"15min":0.53400000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.41499999999999998},{"interface":"front","average":{"1min":1.45,"5min":0.72999999999999998,"15min":0.60999999999999999},"min":{"1min":0.35399999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":17.509,"5min":17.509,"15min":17.509},"last":0.27800000000000002}]},{"osd":1,"last update":"Mon Feb 5 06:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.56100000000000005,"15min":0.56200000000000006},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.752,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.54100000000000004,"15min":0.54400000000000004},"min":{"1min":0.37,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.51900000000000002}]},{"osd":2,"last update":"Mon Feb 5 06:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54800000000000004,"15min":0.55000000000000004},"min":{"1min":0.41499999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.53700000000000003,"15min":0.53600000000000003},"min":{"1min":0.40300000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.70299999999999996,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.66500000000000004}]},{"osd":3,"last update":"Mon Feb 5 06:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.55300000000000005,"15min":0.55900000000000005},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.66900000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.57699999999999996,"15min":0.58099999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.72599999999999998,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.72199999999999998}]},{"osd":4,"last update":"Mon Feb 5 06:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.39500000000000002,"15min":0.40100000000000002},"min":{"1min":0.29799999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.45300000000000001,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.32300000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.53600000000000003,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.68000000000000005}]},{"osd":6,"last update":"Mon Feb 5 06:33:50 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.69799999999999995}]},{"osd":7,"last update":"Mon Feb 5 06:34:16 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.79700000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T06:34:52.732 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T06:34:52.733 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:34:52 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:34:52.984 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:34:53.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:52 smithi028 ceph-mon[29442]: 5.11 scrub starts 2024-02-05T06:34:53.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:52 smithi028 ceph-mon[29442]: 5.11 scrub ok 2024-02-05T06:34:53.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:52 smithi028 ceph-mon[29442]: from='client.14754 v1:172.21.15.28:0/3928824356' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:34:53.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:52 smithi028 ceph-mon[29442]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:34:53.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:52 smithi028 ceph-mon[29442]: 5.d deep-scrub starts 2024-02-05T06:34:53.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:52 smithi028 ceph-mon[29442]: 5.d deep-scrub ok 2024-02-05T06:34:53.106 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:52 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:34:53.106 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:52 smithi028 ceph-mon[37984]: 5.11 scrub starts 2024-02-05T06:34:53.106 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:52 smithi028 ceph-mon[37984]: 5.11 scrub ok 2024-02-05T06:34:53.107 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:52 smithi028 ceph-mon[37984]: from='client.14754 v1:172.21.15.28:0/3928824356' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:34:53.107 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:52 smithi028 ceph-mon[37984]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:34:53.107 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:52 smithi028 ceph-mon[37984]: 5.d deep-scrub starts 2024-02-05T06:34:53.107 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:52 smithi028 ceph-mon[37984]: 5.d deep-scrub ok 2024-02-05T06:34:53.107 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:52 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:34:53.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:52 smithi167 ceph-mon[32774]: 5.11 scrub starts 2024-02-05T06:34:53.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:52 smithi167 ceph-mon[32774]: 5.11 scrub ok 2024-02-05T06:34:53.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:52 smithi167 ceph-mon[32774]: from='client.14754 v1:172.21.15.28:0/3928824356' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:34:53.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:52 smithi167 ceph-mon[32774]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:34:53.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:52 smithi167 ceph-mon[32774]: 5.d deep-scrub starts 2024-02-05T06:34:53.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:52 smithi167 ceph-mon[32774]: 5.d deep-scrub ok 2024-02-05T06:34:53.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:52 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:34:53.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:34:53 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a[72676]: ts=2024-02-05T06:34:53.395Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000444418s 2024-02-05T06:34:54.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:53 smithi028 ceph-mon[37984]: 4.12 scrub starts 2024-02-05T06:34:54.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:53 smithi028 ceph-mon[37984]: 4.12 scrub ok 2024-02-05T06:34:54.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:53 smithi028 ceph-mon[37984]: 4.6 scrub starts 2024-02-05T06:34:54.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:53 smithi028 ceph-mon[37984]: 4.6 scrub ok 2024-02-05T06:34:54.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:53 smithi028 ceph-mon[37984]: 6.d scrub starts 2024-02-05T06:34:54.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:53 smithi028 ceph-mon[37984]: 6.d scrub ok 2024-02-05T06:34:54.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:53 smithi028 ceph-mon[37984]: 5.1a scrub starts 2024-02-05T06:34:54.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:53 smithi028 ceph-mon[37984]: 5.1a scrub ok 2024-02-05T06:34:54.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:53 smithi028 ceph-mon[37984]: 4.1b scrub starts 2024-02-05T06:34:54.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:53 smithi028 ceph-mon[37984]: 4.1b scrub ok 2024-02-05T06:34:54.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:53 smithi028 ceph-mon[29442]: 4.12 scrub starts 2024-02-05T06:34:54.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:53 smithi028 ceph-mon[29442]: 4.12 scrub ok 2024-02-05T06:34:54.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:53 smithi028 ceph-mon[29442]: 4.6 scrub starts 2024-02-05T06:34:54.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:53 smithi028 ceph-mon[29442]: 4.6 scrub ok 2024-02-05T06:34:54.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:53 smithi028 ceph-mon[29442]: 6.d scrub starts 2024-02-05T06:34:54.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:53 smithi028 ceph-mon[29442]: 6.d scrub ok 2024-02-05T06:34:54.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:53 smithi028 ceph-mon[29442]: 5.1a scrub starts 2024-02-05T06:34:54.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:53 smithi028 ceph-mon[29442]: 5.1a scrub ok 2024-02-05T06:34:54.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:53 smithi028 ceph-mon[29442]: 4.1b scrub starts 2024-02-05T06:34:54.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:53 smithi028 ceph-mon[29442]: 4.1b scrub ok 2024-02-05T06:34:54.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:53 smithi167 ceph-mon[32774]: 4.12 scrub starts 2024-02-05T06:34:54.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:53 smithi167 ceph-mon[32774]: 4.12 scrub ok 2024-02-05T06:34:54.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:53 smithi167 ceph-mon[32774]: 4.6 scrub starts 2024-02-05T06:34:54.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:53 smithi167 ceph-mon[32774]: 4.6 scrub ok 2024-02-05T06:34:54.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:53 smithi167 ceph-mon[32774]: 6.d scrub starts 2024-02-05T06:34:54.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:53 smithi167 ceph-mon[32774]: 6.d scrub ok 2024-02-05T06:34:54.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:53 smithi167 ceph-mon[32774]: 5.1a scrub starts 2024-02-05T06:34:54.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:53 smithi167 ceph-mon[32774]: 5.1a scrub ok 2024-02-05T06:34:54.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:53 smithi167 ceph-mon[32774]: 4.1b scrub starts 2024-02-05T06:34:54.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:53 smithi167 ceph-mon[32774]: 4.1b scrub ok 2024-02-05T06:34:54.976 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:34:54.978 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-05T06:34:55.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:54 smithi167 ceph-mon[32774]: 3.12 scrub starts 2024-02-05T06:34:55.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:54 smithi167 ceph-mon[32774]: 3.12 scrub ok 2024-02-05T06:34:55.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:54 smithi167 ceph-mon[32774]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:34:55.283 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:54 smithi028 ceph-mon[37984]: 3.12 scrub starts 2024-02-05T06:34:55.284 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:54 smithi028 ceph-mon[37984]: 3.12 scrub ok 2024-02-05T06:34:55.284 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:54 smithi028 ceph-mon[37984]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:34:55.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:54 smithi028 ceph-mon[29442]: 3.12 scrub starts 2024-02-05T06:34:55.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:54 smithi028 ceph-mon[29442]: 3.12 scrub ok 2024-02-05T06:34:55.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:54 smithi028 ceph-mon[29442]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:34:55.679 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-02-05T06:34:54.152437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1008,"num_read_kb":854,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1771792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":629,"ondisk_log_size":629,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":241276,"kb_used_data":8508,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749719940,"statfs":{"total":767960285184,"available":767713218560,"internally_reserved":0,"allocated":8712192,"data_stored":4808732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002605"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":50,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.892759+0000","last_change":"2024-02-05T06:33:35.594797+0000","last_active":"2024-02-05T06:34:44.892759+0000","last_peered":"2024-02-05T06:34:44.892759+0000","last_clean":"2024-02-05T06:34:44.892759+0000","last_became_active":"2024-02-05T06:33:35.594467+0000","last_became_peered":"2024-02-05T06:33:35.594467+0000","last_unstale":"2024-02-05T06:34:44.892759+0000","last_undegraded":"2024-02-05T06:34:44.892759+0000","last_fullsized":"2024-02-05T06:34:44.892759+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:46.736983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":54,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890147+0000","last_change":"2024-02-05T06:33:33.586212+0000","last_active":"2024-02-05T06:34:44.890147+0000","last_peered":"2024-02-05T06:34:44.890147+0000","last_clean":"2024-02-05T06:34:44.890147+0000","last_became_active":"2024-02-05T06:33:33.585848+0000","last_became_peered":"2024-02-05T06:33:33.585848+0000","last_unstale":"2024-02-05T06:34:44.890147+0000","last_undegraded":"2024-02-05T06:34:44.890147+0000","last_fullsized":"2024-02-05T06:34:44.890147+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:51.677322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890647+0000","last_change":"2024-02-05T06:33:29.549864+0000","last_active":"2024-02-05T06:34:44.890647+0000","last_peered":"2024-02-05T06:34:44.890647+0000","last_clean":"2024-02-05T06:34:44.890647+0000","last_became_active":"2024-02-05T06:33:29.549592+0000","last_became_peered":"2024-02-05T06:33:29.549592+0000","last_unstale":"2024-02-05T06:34:44.890647+0000","last_undegraded":"2024-02-05T06:34:44.890647+0000","last_fullsized":"2024-02-05T06:34:44.890647+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:52.202922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":74,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.892665+0000","last_change":"2024-02-05T06:33:31.566905+0000","last_active":"2024-02-05T06:34:44.892665+0000","last_peered":"2024-02-05T06:34:44.892665+0000","last_clean":"2024-02-05T06:34:44.892665+0000","last_became_active":"2024-02-05T06:33:31.566571+0000","last_became_peered":"2024-02-05T06:33:31.566571+0000","last_unstale":"2024-02-05T06:34:44.892665+0000","last_undegraded":"2024-02-05T06:34:44.892665+0000","last_fullsized":"2024-02-05T06:34:44.892665+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:48.692724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096953+0000","last_change":"2024-02-05T06:33:35.668346+0000","last_active":"2024-02-05T06:34:45.096953+0000","last_peered":"2024-02-05T06:34:45.096953+0000","last_clean":"2024-02-05T06:34:45.096953+0000","last_became_active":"2024-02-05T06:33:35.667778+0000","last_became_peered":"2024-02-05T06:33:35.667778+0000","last_unstale":"2024-02-05T06:34:45.096953+0000","last_undegraded":"2024-02-05T06:34:45.096953+0000","last_fullsized":"2024-02-05T06:34:45.096953+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:59.948542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":54,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.892667+0000","last_change":"2024-02-05T06:33:33.583029+0000","last_active":"2024-02-05T06:34:44.892667+0000","last_peered":"2024-02-05T06:34:44.892667+0000","last_clean":"2024-02-05T06:34:44.892667+0000","last_became_active":"2024-02-05T06:33:33.582731+0000","last_became_peered":"2024-02-05T06:33:33.582731+0000","last_unstale":"2024-02-05T06:34:44.892667+0000","last_undegraded":"2024-02-05T06:34:44.892667+0000","last_fullsized":"2024-02-05T06:34:44.892667+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:56.016226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":71,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097695+0000","last_change":"2024-02-05T06:34:34.821722+0000","last_active":"2024-02-05T06:34:45.097695+0000","last_peered":"2024-02-05T06:34:45.097695+0000","last_clean":"2024-02-05T06:34:45.097695+0000","last_became_active":"2024-02-05T06:33:29.553864+0000","last_became_peered":"2024-02-05T06:33:29.553864+0000","last_unstale":"2024-02-05T06:34:45.097695+0000","last_undegraded":"2024-02-05T06:34:45.097695+0000","last_fullsized":"2024-02-05T06:34:45.097695+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:34.821603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:34.821603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:39.571760+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"88'22","reported_seq":99,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097275+0000","last_change":"2024-02-05T06:34:35.832461+0000","last_active":"2024-02-05T06:34:45.097275+0000","last_peered":"2024-02-05T06:34:45.097275+0000","last_clean":"2024-02-05T06:34:45.097275+0000","last_became_active":"2024-02-05T06:33:31.572544+0000","last_became_peered":"2024-02-05T06:33:31.572544+0000","last_unstale":"2024-02-05T06:34:45.097275+0000","last_undegraded":"2024-02-05T06:34:45.097275+0000","last_fullsized":"2024-02-05T06:34:45.097275+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'22","last_scrub_stamp":"2024-02-05T06:34:35.832368+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:34:35.832368+0000","last_clean_scrub_stamp":"2024-02-05T06:34:35.832368+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:43.028037+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.889644+0000","last_change":"2024-02-05T06:33:35.595744+0000","last_active":"2024-02-05T06:34:44.889644+0000","last_peered":"2024-02-05T06:34:44.889644+0000","last_clean":"2024-02-05T06:34:44.889644+0000","last_became_active":"2024-02-05T06:33:35.594860+0000","last_became_peered":"2024-02-05T06:33:35.594860+0000","last_unstale":"2024-02-05T06:34:44.889644+0000","last_undegraded":"2024-02-05T06:34:44.889644+0000","last_fullsized":"2024-02-05T06:34:44.889644+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095518+0000","last_change":"2024-02-05T06:34:34.187757+0000","last_active":"2024-02-05T06:34:45.095518+0000","last_peered":"2024-02-05T06:34:45.095518+0000","last_clean":"2024-02-05T06:34:45.095518+0000","last_became_active":"2024-02-05T06:33:33.585459+0000","last_became_peered":"2024-02-05T06:33:33.585459+0000","last_unstale":"2024-02-05T06:34:45.095518+0000","last_undegraded":"2024-02-05T06:34:45.095518+0000","last_fullsized":"2024-02-05T06:34:45.095518+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:34.187643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:34.187643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:35.727918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095719+0000","last_change":"2024-02-05T06:34:44.081424+0000","last_active":"2024-02-05T06:34:45.095719+0000","last_peered":"2024-02-05T06:34:45.095719+0000","last_clean":"2024-02-05T06:34:45.095719+0000","last_became_active":"2024-02-05T06:33:29.553251+0000","last_became_peered":"2024-02-05T06:33:29.553251+0000","last_unstale":"2024-02-05T06:34:45.095719+0000","last_undegraded":"2024-02-05T06:34:45.095719+0000","last_fullsized":"2024-02-05T06:34:45.095719+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:44.081318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:44.081318+0000","last_clean_scrub_stamp":"2024-02-05T06:34:44.081318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:49.865346+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"92'13","reported_seq":86,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:46.542965+0000","last_change":"2024-02-05T06:34:46.543006+0000","last_active":"2024-02-05T06:34:46.542965+0000","last_peered":"2024-02-05T06:34:46.542965+0000","last_clean":"2024-02-05T06:34:46.542965+0000","last_became_active":"2024-02-05T06:33:31.570995+0000","last_became_peered":"2024-02-05T06:33:31.570995+0000","last_unstale":"2024-02-05T06:34:46.542965+0000","last_undegraded":"2024-02-05T06:34:46.542965+0000","last_fullsized":"2024-02-05T06:34:46.542965+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'13","last_scrub_stamp":"2024-02-05T06:34:46.542893+0000","last_deep_scrub":"92'13","last_deep_scrub_stamp":"2024-02-05T06:34:46.542893+0000","last_clean_scrub_stamp":"2024-02-05T06:34:46.542893+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:00.864899+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095188+0000","last_change":"2024-02-05T06:33:35.600855+0000","last_active":"2024-02-05T06:34:45.095188+0000","last_peered":"2024-02-05T06:34:45.095188+0000","last_clean":"2024-02-05T06:34:45.095188+0000","last_became_active":"2024-02-05T06:33:35.600399+0000","last_became_peered":"2024-02-05T06:33:35.600399+0000","last_unstale":"2024-02-05T06:34:45.095188+0000","last_undegraded":"2024-02-05T06:34:45.095188+0000","last_fullsized":"2024-02-05T06:34:45.095188+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:59.096165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":55,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.922297+0000","last_change":"2024-02-05T06:33:33.584454+0000","last_active":"2024-02-05T06:34:45.922297+0000","last_peered":"2024-02-05T06:34:45.922297+0000","last_clean":"2024-02-05T06:34:45.922297+0000","last_became_active":"2024-02-05T06:33:33.584151+0000","last_became_peered":"2024-02-05T06:33:33.584151+0000","last_unstale":"2024-02-05T06:34:45.922297+0000","last_undegraded":"2024-02-05T06:34:45.922297+0000","last_fullsized":"2024-02-05T06:34:45.922297+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:48.436451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:50.055329+0000","last_change":"2024-02-05T06:34:50.055371+0000","last_active":"2024-02-05T06:34:50.055329+0000","last_peered":"2024-02-05T06:34:50.055329+0000","last_clean":"2024-02-05T06:34:50.055329+0000","last_became_active":"2024-02-05T06:33:29.548931+0000","last_became_peered":"2024-02-05T06:33:29.548931+0000","last_unstale":"2024-02-05T06:34:50.055329+0000","last_undegraded":"2024-02-05T06:34:50.055329+0000","last_fullsized":"2024-02-05T06:34:50.055329+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:50.055245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:50.055245+0000","last_clean_scrub_stamp":"2024-02-05T06:34:50.055245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:54.332822+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":68,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.893390+0000","last_change":"2024-02-05T06:33:31.570226+0000","last_active":"2024-02-05T06:34:44.893390+0000","last_peered":"2024-02-05T06:34:44.893390+0000","last_clean":"2024-02-05T06:34:44.893390+0000","last_became_active":"2024-02-05T06:33:31.569654+0000","last_became_peered":"2024-02-05T06:33:31.569654+0000","last_unstale":"2024-02-05T06:34:44.893390+0000","last_undegraded":"2024-02-05T06:34:44.893390+0000","last_fullsized":"2024-02-05T06:34:44.893390+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:45.133474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":58,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.573515+0000","last_change":"2024-02-05T06:34:45.573557+0000","last_active":"2024-02-05T06:34:45.573515+0000","last_peered":"2024-02-05T06:34:45.573515+0000","last_clean":"2024-02-05T06:34:45.573515+0000","last_became_active":"2024-02-05T06:33:35.664772+0000","last_became_peered":"2024-02-05T06:33:35.664772+0000","last_unstale":"2024-02-05T06:34:45.573515+0000","last_undegraded":"2024-02-05T06:34:45.573515+0000","last_fullsized":"2024-02-05T06:34:45.573515+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:45.573432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:45.573432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:10.154762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":54,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890929+0000","last_change":"2024-02-05T06:33:33.588208+0000","last_active":"2024-02-05T06:34:44.890929+0000","last_peered":"2024-02-05T06:34:44.890929+0000","last_clean":"2024-02-05T06:34:44.890929+0000","last_became_active":"2024-02-05T06:33:33.587920+0000","last_became_peered":"2024-02-05T06:33:33.587920+0000","last_unstale":"2024-02-05T06:34:44.890929+0000","last_undegraded":"2024-02-05T06:34:44.890929+0000","last_fullsized":"2024-02-05T06:34:44.890929+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:54.461651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"66'1","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891771+0000","last_change":"2024-02-05T06:33:29.552104+0000","last_active":"2024-02-05T06:34:44.891771+0000","last_peered":"2024-02-05T06:34:44.891771+0000","last_clean":"2024-02-05T06:34:44.891771+0000","last_became_active":"2024-02-05T06:33:29.551820+0000","last_became_peered":"2024-02-05T06:33:29.551820+0000","last_unstale":"2024-02-05T06:34:44.891771+0000","last_undegraded":"2024-02-05T06:34:44.891771+0000","last_fullsized":"2024-02-05T06:34:44.891771+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"88'22","reported_seq":97,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.297835+0000","last_change":"2024-02-05T06:34:35.234726+0000","last_active":"2024-02-05T06:34:45.297835+0000","last_peered":"2024-02-05T06:34:45.297835+0000","last_clean":"2024-02-05T06:34:45.297835+0000","last_became_active":"2024-02-05T06:33:31.569763+0000","last_became_peered":"2024-02-05T06:33:31.569763+0000","last_unstale":"2024-02-05T06:34:45.297835+0000","last_undegraded":"2024-02-05T06:34:45.297835+0000","last_fullsized":"2024-02-05T06:34:45.297835+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'22","last_scrub_stamp":"2024-02-05T06:34:35.234636+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:34:35.234636+0000","last_clean_scrub_stamp":"2024-02-05T06:34:35.234636+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:53.234677+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097830+0000","last_change":"2024-02-05T06:33:35.665217+0000","last_active":"2024-02-05T06:34:45.097830+0000","last_peered":"2024-02-05T06:34:45.097830+0000","last_clean":"2024-02-05T06:34:45.097830+0000","last_became_active":"2024-02-05T06:33:35.664852+0000","last_became_peered":"2024-02-05T06:33:35.664852+0000","last_unstale":"2024-02-05T06:34:45.097830+0000","last_undegraded":"2024-02-05T06:34:45.097830+0000","last_fullsized":"2024-02-05T06:34:45.097830+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:47.798779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":54,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.893413+0000","last_change":"2024-02-05T06:33:33.587913+0000","last_active":"2024-02-05T06:34:44.893413+0000","last_peered":"2024-02-05T06:34:44.893413+0000","last_clean":"2024-02-05T06:34:44.893413+0000","last_became_active":"2024-02-05T06:33:33.587595+0000","last_became_peered":"2024-02-05T06:33:33.587595+0000","last_unstale":"2024-02-05T06:34:44.893413+0000","last_undegraded":"2024-02-05T06:34:44.893413+0000","last_fullsized":"2024-02-05T06:34:44.893413+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:53.273088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":70,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.892937+0000","last_change":"2024-02-05T06:34:38.623297+0000","last_active":"2024-02-05T06:34:44.892937+0000","last_peered":"2024-02-05T06:34:44.892937+0000","last_clean":"2024-02-05T06:34:44.892937+0000","last_became_active":"2024-02-05T06:33:29.551340+0000","last_became_peered":"2024-02-05T06:33:29.551340+0000","last_unstale":"2024-02-05T06:34:44.892937+0000","last_undegraded":"2024-02-05T06:34:44.892937+0000","last_fullsized":"2024-02-05T06:34:44.892937+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:38.623203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:38.623203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:49.053281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":100,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097390+0000","last_change":"2024-02-05T06:33:55.892463+0000","last_active":"2024-02-05T06:34:45.097390+0000","last_peered":"2024-02-05T06:34:45.097390+0000","last_clean":"2024-02-05T06:34:45.097390+0000","last_became_active":"2024-02-05T06:33:55.885324+0000","last_became_peered":"2024-02-05T06:33:55.885324+0000","last_unstale":"2024-02-05T06:34:45.097390+0000","last_undegraded":"2024-02-05T06:34:45.097390+0000","last_fullsized":"2024-02-05T06:34:45.097390+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:46.592833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"73'1","reported_seq":52,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.923186+0000","last_change":"2024-02-05T06:33:35.598794+0000","last_active":"2024-02-05T06:34:45.923186+0000","last_peered":"2024-02-05T06:34:45.923186+0000","last_clean":"2024-02-05T06:34:45.923186+0000","last_became_active":"2024-02-05T06:33:35.598501+0000","last_became_peered":"2024-02-05T06:33:35.598501+0000","last_unstale":"2024-02-05T06:34:45.923186+0000","last_undegraded":"2024-02-05T06:34:45.923186+0000","last_fullsized":"2024-02-05T06:34:45.923186+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:02.621665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"73'8","reported_seq":85,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.266607+0000","last_change":"2024-02-05T06:33:33.586794+0000","last_active":"2024-02-05T06:34:45.266607+0000","last_peered":"2024-02-05T06:34:45.266607+0000","last_clean":"2024-02-05T06:34:45.266607+0000","last_became_active":"2024-02-05T06:33:33.586389+0000","last_became_peered":"2024-02-05T06:33:33.586389+0000","last_unstale":"2024-02-05T06:34:45.266607+0000","last_undegraded":"2024-02-05T06:34:45.266607+0000","last_fullsized":"2024-02-05T06:34:45.266607+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:54.474866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":71,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:48.387541+0000","last_change":"2024-02-05T06:34:48.387568+0000","last_active":"2024-02-05T06:34:48.387541+0000","last_peered":"2024-02-05T06:34:48.387541+0000","last_clean":"2024-02-05T06:34:48.387541+0000","last_became_active":"2024-02-05T06:33:29.546201+0000","last_became_peered":"2024-02-05T06:33:29.546201+0000","last_unstale":"2024-02-05T06:34:48.387541+0000","last_undegraded":"2024-02-05T06:34:48.387541+0000","last_fullsized":"2024-02-05T06:34:48.387541+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:34:48.387474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:48.387474+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:58.746172+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"92'15","reported_seq":87,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:50.718830+0000","last_change":"2024-02-05T06:34:50.718858+0000","last_active":"2024-02-05T06:34:50.718830+0000","last_peered":"2024-02-05T06:34:50.718830+0000","last_clean":"2024-02-05T06:34:50.718830+0000","last_became_active":"2024-02-05T06:33:31.666481+0000","last_became_peered":"2024-02-05T06:33:31.666481+0000","last_unstale":"2024-02-05T06:34:50.718830+0000","last_undegraded":"2024-02-05T06:34:50.718830+0000","last_fullsized":"2024-02-05T06:34:50.718830+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'15","last_scrub_stamp":"2024-02-05T06:34:50.718768+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-02-05T06:34:50.718768+0000","last_clean_scrub_stamp":"2024-02-05T06:34:50.718768+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:51.271274+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":50,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891060+0000","last_change":"2024-02-05T06:33:35.594280+0000","last_active":"2024-02-05T06:34:44.891060+0000","last_peered":"2024-02-05T06:34:44.891060+0000","last_clean":"2024-02-05T06:34:44.891060+0000","last_became_active":"2024-02-05T06:33:35.593925+0000","last_became_peered":"2024-02-05T06:33:35.593925+0000","last_unstale":"2024-02-05T06:34:44.891060+0000","last_undegraded":"2024-02-05T06:34:44.891060+0000","last_fullsized":"2024-02-05T06:34:44.891060+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:48.199426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":55,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891396+0000","last_change":"2024-02-05T06:33:33.584707+0000","last_active":"2024-02-05T06:34:44.891396+0000","last_peered":"2024-02-05T06:34:44.891396+0000","last_clean":"2024-02-05T06:34:44.891396+0000","last_became_active":"2024-02-05T06:33:33.584390+0000","last_became_peered":"2024-02-05T06:33:33.584390+0000","last_unstale":"2024-02-05T06:34:44.891396+0000","last_undegraded":"2024-02-05T06:34:44.891396+0000","last_fullsized":"2024-02-05T06:34:44.891396+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096561+0000","last_change":"2024-02-05T06:33:29.550640+0000","last_active":"2024-02-05T06:34:45.096561+0000","last_peered":"2024-02-05T06:34:45.096561+0000","last_clean":"2024-02-05T06:34:45.096561+0000","last_became_active":"2024-02-05T06:33:29.550347+0000","last_became_peered":"2024-02-05T06:33:29.550347+0000","last_unstale":"2024-02-05T06:34:45.096561+0000","last_undegraded":"2024-02-05T06:34:45.096561+0000","last_fullsized":"2024-02-05T06:34:45.096561+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:47.916348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"92'16","reported_seq":90,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.270243+0000","last_change":"2024-02-05T06:34:45.270307+0000","last_active":"2024-02-05T06:34:45.270243+0000","last_peered":"2024-02-05T06:34:45.270243+0000","last_clean":"2024-02-05T06:34:45.270243+0000","last_became_active":"2024-02-05T06:33:31.573088+0000","last_became_peered":"2024-02-05T06:33:31.573088+0000","last_unstale":"2024-02-05T06:34:45.270243+0000","last_undegraded":"2024-02-05T06:34:45.270243+0000","last_fullsized":"2024-02-05T06:34:45.270243+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'16","last_scrub_stamp":"2024-02-05T06:34:45.270135+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:34:45.270135+0000","last_clean_scrub_stamp":"2024-02-05T06:34:45.270135+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:04.519387+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.922744+0000","last_change":"2024-02-05T06:33:35.597741+0000","last_active":"2024-02-05T06:34:45.922744+0000","last_peered":"2024-02-05T06:34:45.922744+0000","last_clean":"2024-02-05T06:34:45.922744+0000","last_became_active":"2024-02-05T06:33:35.597453+0000","last_became_peered":"2024-02-05T06:33:35.597453+0000","last_unstale":"2024-02-05T06:34:45.922744+0000","last_undegraded":"2024-02-05T06:34:45.922744+0000","last_fullsized":"2024-02-05T06:34:45.922744+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:49.858193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":54,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.265743+0000","last_change":"2024-02-05T06:33:33.585405+0000","last_active":"2024-02-05T06:34:45.265743+0000","last_peered":"2024-02-05T06:34:45.265743+0000","last_clean":"2024-02-05T06:34:45.265743+0000","last_became_active":"2024-02-05T06:33:33.585125+0000","last_became_peered":"2024-02-05T06:33:33.585125+0000","last_unstale":"2024-02-05T06:34:45.265743+0000","last_undegraded":"2024-02-05T06:34:45.265743+0000","last_fullsized":"2024-02-05T06:34:45.265743+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:50.106397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890564+0000","last_change":"2024-02-05T06:33:29.549815+0000","last_active":"2024-02-05T06:34:44.890564+0000","last_peered":"2024-02-05T06:34:44.890564+0000","last_clean":"2024-02-05T06:34:44.890564+0000","last_became_active":"2024-02-05T06:33:29.549204+0000","last_became_peered":"2024-02-05T06:33:29.549204+0000","last_unstale":"2024-02-05T06:34:44.890564+0000","last_undegraded":"2024-02-05T06:34:44.890564+0000","last_fullsized":"2024-02-05T06:34:44.890564+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:54.367234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"73'4","reported_seq":65,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096579+0000","last_change":"2024-02-05T06:33:31.573713+0000","last_active":"2024-02-05T06:34:45.096579+0000","last_peered":"2024-02-05T06:34:45.096579+0000","last_clean":"2024-02-05T06:34:45.096579+0000","last_became_active":"2024-02-05T06:33:31.573410+0000","last_became_peered":"2024-02-05T06:33:31.573410+0000","last_unstale":"2024-02-05T06:34:45.096579+0000","last_undegraded":"2024-02-05T06:34:45.096579+0000","last_fullsized":"2024-02-05T06:34:45.096579+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:58.822517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096768+0000","last_change":"2024-02-05T06:33:35.667939+0000","last_active":"2024-02-05T06:34:45.096768+0000","last_peered":"2024-02-05T06:34:45.096768+0000","last_clean":"2024-02-05T06:34:45.096768+0000","last_became_active":"2024-02-05T06:33:35.667602+0000","last_became_peered":"2024-02-05T06:33:35.667602+0000","last_unstale":"2024-02-05T06:34:45.096768+0000","last_undegraded":"2024-02-05T06:34:45.096768+0000","last_fullsized":"2024-02-05T06:34:45.096768+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.922284+0000","last_change":"2024-02-05T06:34:43.035728+0000","last_active":"2024-02-05T06:34:45.922284+0000","last_peered":"2024-02-05T06:34:45.922284+0000","last_clean":"2024-02-05T06:34:45.922284+0000","last_became_active":"2024-02-05T06:33:33.589428+0000","last_became_peered":"2024-02-05T06:33:33.589428+0000","last_unstale":"2024-02-05T06:34:45.922284+0000","last_undegraded":"2024-02-05T06:34:45.922284+0000","last_fullsized":"2024-02-05T06:34:45.922284+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:43.035635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:43.035635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:52.189018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095696+0000","last_change":"2024-02-05T06:33:29.551576+0000","last_active":"2024-02-05T06:34:45.095696+0000","last_peered":"2024-02-05T06:34:45.095696+0000","last_clean":"2024-02-05T06:34:45.095696+0000","last_became_active":"2024-02-05T06:33:29.551278+0000","last_became_peered":"2024-02-05T06:33:29.551278+0000","last_unstale":"2024-02-05T06:34:45.095696+0000","last_undegraded":"2024-02-05T06:34:45.095696+0000","last_fullsized":"2024-02-05T06:34:45.095696+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:58.094774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"73'11","reported_seq":78,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097095+0000","last_change":"2024-02-05T06:33:31.665074+0000","last_active":"2024-02-05T06:34:45.097095+0000","last_peered":"2024-02-05T06:34:45.097095+0000","last_clean":"2024-02-05T06:34:45.097095+0000","last_became_active":"2024-02-05T06:33:31.664743+0000","last_became_peered":"2024-02-05T06:33:31.664743+0000","last_unstale":"2024-02-05T06:34:45.097095+0000","last_undegraded":"2024-02-05T06:34:45.097095+0000","last_fullsized":"2024-02-05T06:34:45.097095+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:56.675510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"73'1","reported_seq":52,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.889401+0000","last_change":"2024-02-05T06:33:35.595450+0000","last_active":"2024-02-05T06:34:44.889401+0000","last_peered":"2024-02-05T06:34:44.889401+0000","last_clean":"2024-02-05T06:34:44.889401+0000","last_became_active":"2024-02-05T06:33:35.595031+0000","last_became_peered":"2024-02-05T06:33:35.595031+0000","last_unstale":"2024-02-05T06:34:44.889401+0000","last_undegraded":"2024-02-05T06:34:44.889401+0000","last_fullsized":"2024-02-05T06:34:44.889401+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":55,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097083+0000","last_change":"2024-02-05T06:33:33.594194+0000","last_active":"2024-02-05T06:34:45.097083+0000","last_peered":"2024-02-05T06:34:45.097083+0000","last_clean":"2024-02-05T06:34:45.097083+0000","last_became_active":"2024-02-05T06:33:33.593901+0000","last_became_peered":"2024-02-05T06:33:33.593901+0000","last_unstale":"2024-02-05T06:34:45.097083+0000","last_undegraded":"2024-02-05T06:34:45.097083+0000","last_fullsized":"2024-02-05T06:34:45.097083+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:46.645657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.923423+0000","last_change":"2024-02-05T06:33:29.559294+0000","last_active":"2024-02-05T06:34:45.923423+0000","last_peered":"2024-02-05T06:34:45.923423+0000","last_clean":"2024-02-05T06:34:45.923423+0000","last_became_active":"2024-02-05T06:33:29.559006+0000","last_became_peered":"2024-02-05T06:33:29.559006+0000","last_unstale":"2024-02-05T06:34:45.923423+0000","last_undegraded":"2024-02-05T06:34:45.923423+0000","last_fullsized":"2024-02-05T06:34:45.923423+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:52.625563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":74,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891566+0000","last_change":"2024-02-05T06:33:31.568831+0000","last_active":"2024-02-05T06:34:44.891566+0000","last_peered":"2024-02-05T06:34:44.891566+0000","last_clean":"2024-02-05T06:34:44.891566+0000","last_became_active":"2024-02-05T06:33:31.568529+0000","last_became_peered":"2024-02-05T06:33:31.568529+0000","last_unstale":"2024-02-05T06:34:44.891566+0000","last_undegraded":"2024-02-05T06:34:44.891566+0000","last_fullsized":"2024-02-05T06:34:44.891566+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":59,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:48.882631+0000","last_change":"2024-02-05T06:34:48.882657+0000","last_active":"2024-02-05T06:34:48.882631+0000","last_peered":"2024-02-05T06:34:48.882631+0000","last_clean":"2024-02-05T06:34:48.882631+0000","last_became_active":"2024-02-05T06:33:35.597399+0000","last_became_peered":"2024-02-05T06:33:35.597399+0000","last_unstale":"2024-02-05T06:34:48.882631+0000","last_undegraded":"2024-02-05T06:34:48.882631+0000","last_fullsized":"2024-02-05T06:34:48.882631+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:48.882556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:48.882556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:52.368291+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":62,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891472+0000","last_change":"2024-02-05T06:34:40.361349+0000","last_active":"2024-02-05T06:34:44.891472+0000","last_peered":"2024-02-05T06:34:44.891472+0000","last_clean":"2024-02-05T06:34:44.891472+0000","last_became_active":"2024-02-05T06:33:33.588461+0000","last_became_peered":"2024-02-05T06:33:33.588461+0000","last_unstale":"2024-02-05T06:34:44.891472+0000","last_undegraded":"2024-02-05T06:34:44.891472+0000","last_fullsized":"2024-02-05T06:34:44.891472+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:40.361273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:40.361273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:50.089342+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":70,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:48.561101+0000","last_change":"2024-02-05T06:34:48.561146+0000","last_active":"2024-02-05T06:34:48.561101+0000","last_peered":"2024-02-05T06:34:48.561101+0000","last_clean":"2024-02-05T06:34:48.561101+0000","last_became_active":"2024-02-05T06:33:29.552336+0000","last_became_peered":"2024-02-05T06:33:29.552336+0000","last_unstale":"2024-02-05T06:34:48.561101+0000","last_undegraded":"2024-02-05T06:34:48.561101+0000","last_fullsized":"2024-02-05T06:34:48.561101+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:48.561010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:48.561010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:06.806058+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"73'11","reported_seq":77,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.893567+0000","last_change":"2024-02-05T06:33:31.569835+0000","last_active":"2024-02-05T06:34:44.893567+0000","last_peered":"2024-02-05T06:34:44.893567+0000","last_clean":"2024-02-05T06:34:44.893567+0000","last_became_active":"2024-02-05T06:33:31.569488+0000","last_became_peered":"2024-02-05T06:33:31.569488+0000","last_unstale":"2024-02-05T06:34:44.893567+0000","last_undegraded":"2024-02-05T06:34:44.893567+0000","last_fullsized":"2024-02-05T06:34:44.893567+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:51.980754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890759+0000","last_change":"2024-02-05T06:33:35.600109+0000","last_active":"2024-02-05T06:34:44.890759+0000","last_peered":"2024-02-05T06:34:44.890759+0000","last_clean":"2024-02-05T06:34:44.890759+0000","last_became_active":"2024-02-05T06:33:35.599808+0000","last_became_peered":"2024-02-05T06:33:35.599808+0000","last_unstale":"2024-02-05T06:34:44.890759+0000","last_undegraded":"2024-02-05T06:34:44.890759+0000","last_fullsized":"2024-02-05T06:34:44.890759+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:50.168225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":78,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096114+0000","last_change":"2024-02-05T06:33:33.585921+0000","last_active":"2024-02-05T06:34:45.096114+0000","last_peered":"2024-02-05T06:34:45.096114+0000","last_clean":"2024-02-05T06:34:45.096114+0000","last_became_active":"2024-02-05T06:33:33.585642+0000","last_became_peered":"2024-02-05T06:33:33.585642+0000","last_unstale":"2024-02-05T06:34:45.096114+0000","last_undegraded":"2024-02-05T06:34:45.096114+0000","last_fullsized":"2024-02-05T06:34:45.096114+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.922923+0000","last_change":"2024-02-05T06:33:29.557136+0000","last_active":"2024-02-05T06:34:45.922923+0000","last_peered":"2024-02-05T06:34:45.922923+0000","last_clean":"2024-02-05T06:34:45.922923+0000","last_became_active":"2024-02-05T06:33:29.556832+0000","last_became_peered":"2024-02-05T06:33:29.556832+0000","last_unstale":"2024-02-05T06:34:45.922923+0000","last_undegraded":"2024-02-05T06:34:45.922923+0000","last_fullsized":"2024-02-05T06:34:45.922923+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:52.324493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'10","reported_seq":82,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097179+0000","last_change":"2024-02-05T06:34:41.873963+0000","last_active":"2024-02-05T06:34:45.097179+0000","last_peered":"2024-02-05T06:34:45.097179+0000","last_clean":"2024-02-05T06:34:45.097179+0000","last_became_active":"2024-02-05T06:33:31.666808+0000","last_became_peered":"2024-02-05T06:33:31.666808+0000","last_unstale":"2024-02-05T06:34:45.097179+0000","last_undegraded":"2024-02-05T06:34:45.097179+0000","last_fullsized":"2024-02-05T06:34:45.097179+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-02-05T06:34:41.873876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:41.873876+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:08.435407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":50,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.893956+0000","last_change":"2024-02-05T06:33:35.597787+0000","last_active":"2024-02-05T06:34:44.893956+0000","last_peered":"2024-02-05T06:34:44.893956+0000","last_clean":"2024-02-05T06:34:44.893956+0000","last_became_active":"2024-02-05T06:33:35.597494+0000","last_became_peered":"2024-02-05T06:33:35.597494+0000","last_unstale":"2024-02-05T06:34:44.893956+0000","last_undegraded":"2024-02-05T06:34:44.893956+0000","last_fullsized":"2024-02-05T06:34:44.893956+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:54.313659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":88,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:47.925702+0000","last_change":"2024-02-05T06:34:47.925702+0000","last_active":"2024-02-05T06:34:47.925702+0000","last_peered":"2024-02-05T06:34:47.925702+0000","last_clean":"2024-02-05T06:34:47.925702+0000","last_became_active":"2024-02-05T06:33:33.586043+0000","last_became_peered":"2024-02-05T06:33:33.586043+0000","last_unstale":"2024-02-05T06:34:47.925702+0000","last_undegraded":"2024-02-05T06:34:47.925702+0000","last_fullsized":"2024-02-05T06:34:47.925702+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:34:47.925577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:47.925577+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:48.049254+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.889924+0000","last_change":"2024-02-05T06:33:29.552610+0000","last_active":"2024-02-05T06:34:44.889924+0000","last_peered":"2024-02-05T06:34:44.889924+0000","last_clean":"2024-02-05T06:34:44.889924+0000","last_became_active":"2024-02-05T06:33:29.552170+0000","last_became_peered":"2024-02-05T06:33:29.552170+0000","last_unstale":"2024-02-05T06:34:44.889924+0000","last_undegraded":"2024-02-05T06:34:44.889924+0000","last_fullsized":"2024-02-05T06:34:44.889924+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"73'9","reported_seq":81,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096168+0000","last_change":"2024-02-05T06:34:36.172574+0000","last_active":"2024-02-05T06:34:45.096168+0000","last_peered":"2024-02-05T06:34:45.096168+0000","last_clean":"2024-02-05T06:34:45.096168+0000","last_became_active":"2024-02-05T06:33:31.666678+0000","last_became_peered":"2024-02-05T06:33:31.666678+0000","last_unstale":"2024-02-05T06:34:45.096168+0000","last_undegraded":"2024-02-05T06:34:45.096168+0000","last_fullsized":"2024-02-05T06:34:45.096168+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:34:36.172444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:36.172444+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:50.358190+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.298026+0000","last_change":"2024-02-05T06:34:43.228873+0000","last_active":"2024-02-05T06:34:45.298026+0000","last_peered":"2024-02-05T06:34:45.298026+0000","last_clean":"2024-02-05T06:34:45.298026+0000","last_became_active":"2024-02-05T06:33:35.601508+0000","last_became_peered":"2024-02-05T06:33:35.601508+0000","last_unstale":"2024-02-05T06:34:45.298026+0000","last_undegraded":"2024-02-05T06:34:45.298026+0000","last_fullsized":"2024-02-05T06:34:45.298026+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:43.228774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:43.228774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:57.334710+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097815+0000","last_change":"2024-02-05T06:34:42.866516+0000","last_active":"2024-02-05T06:34:45.097815+0000","last_peered":"2024-02-05T06:34:45.097815+0000","last_clean":"2024-02-05T06:34:45.097815+0000","last_became_active":"2024-02-05T06:33:33.593025+0000","last_became_peered":"2024-02-05T06:33:33.593025+0000","last_unstale":"2024-02-05T06:34:45.097815+0000","last_undegraded":"2024-02-05T06:34:45.097815+0000","last_fullsized":"2024-02-05T06:34:45.097815+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:42.866394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:42.866394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:43.991930+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":71,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:49.998414+0000","last_change":"2024-02-05T06:34:49.998454+0000","last_active":"2024-02-05T06:34:49.998414+0000","last_peered":"2024-02-05T06:34:49.998414+0000","last_clean":"2024-02-05T06:34:49.998414+0000","last_became_active":"2024-02-05T06:33:29.553880+0000","last_became_peered":"2024-02-05T06:33:29.553880+0000","last_unstale":"2024-02-05T06:34:49.998414+0000","last_undegraded":"2024-02-05T06:34:49.998414+0000","last_fullsized":"2024-02-05T06:34:49.998414+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:49.998323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:49.998323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:50.575718+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"73'9","reported_seq":83,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891187+0000","last_change":"2024-02-05T06:34:43.760971+0000","last_active":"2024-02-05T06:34:44.891187+0000","last_peered":"2024-02-05T06:34:44.891187+0000","last_clean":"2024-02-05T06:34:44.891187+0000","last_became_active":"2024-02-05T06:33:31.665766+0000","last_became_peered":"2024-02-05T06:33:31.665766+0000","last_unstale":"2024-02-05T06:34:44.891187+0000","last_undegraded":"2024-02-05T06:34:44.891187+0000","last_fullsized":"2024-02-05T06:34:44.891187+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:34:43.760888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:43.760888+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:44.661663+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.923416+0000","last_change":"2024-02-05T06:33:35.666076+0000","last_active":"2024-02-05T06:34:45.923416+0000","last_peered":"2024-02-05T06:34:45.923416+0000","last_clean":"2024-02-05T06:34:45.923416+0000","last_became_active":"2024-02-05T06:33:35.665596+0000","last_became_peered":"2024-02-05T06:33:35.665596+0000","last_unstale":"2024-02-05T06:34:45.923416+0000","last_undegraded":"2024-02-05T06:34:45.923416+0000","last_fullsized":"2024-02-05T06:34:45.923416+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:51.044976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096424+0000","last_change":"2024-02-05T06:34:45.096449+0000","last_active":"2024-02-05T06:34:45.096424+0000","last_peered":"2024-02-05T06:34:45.096424+0000","last_clean":"2024-02-05T06:34:45.096424+0000","last_became_active":"2024-02-05T06:33:33.586768+0000","last_became_peered":"2024-02-05T06:33:33.586768+0000","last_unstale":"2024-02-05T06:34:45.096424+0000","last_undegraded":"2024-02-05T06:34:45.096424+0000","last_fullsized":"2024-02-05T06:34:45.096424+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:45.096340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:45.096340+0000","last_clean_scrub_stamp":"2024-02-05T06:34:45.096340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:12.226977+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":70,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.265669+0000","last_change":"2024-02-05T06:34:30.204064+0000","last_active":"2024-02-05T06:34:45.265669+0000","last_peered":"2024-02-05T06:34:45.265669+0000","last_clean":"2024-02-05T06:34:45.265669+0000","last_became_active":"2024-02-05T06:33:29.554427+0000","last_became_peered":"2024-02-05T06:33:29.554427+0000","last_unstale":"2024-02-05T06:34:45.265669+0000","last_undegraded":"2024-02-05T06:34:45.265669+0000","last_fullsized":"2024-02-05T06:34:45.265669+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:30.203967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:30.203967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:35.531249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":68,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097892+0000","last_change":"2024-02-05T06:33:31.666151+0000","last_active":"2024-02-05T06:34:45.097892+0000","last_peered":"2024-02-05T06:34:45.097892+0000","last_clean":"2024-02-05T06:34:45.097892+0000","last_became_active":"2024-02-05T06:33:31.665847+0000","last_became_peered":"2024-02-05T06:33:31.665847+0000","last_unstale":"2024-02-05T06:34:45.097892+0000","last_undegraded":"2024-02-05T06:34:45.097892+0000","last_fullsized":"2024-02-05T06:34:45.097892+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095574+0000","last_change":"2024-02-05T06:33:35.665928+0000","last_active":"2024-02-05T06:34:45.095574+0000","last_peered":"2024-02-05T06:34:45.095574+0000","last_clean":"2024-02-05T06:34:45.095574+0000","last_became_active":"2024-02-05T06:33:35.665581+0000","last_became_peered":"2024-02-05T06:33:35.665581+0000","last_unstale":"2024-02-05T06:34:45.095574+0000","last_undegraded":"2024-02-05T06:34:45.095574+0000","last_fullsized":"2024-02-05T06:34:45.095574+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:47.218454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":75,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.923730+0000","last_change":"2024-02-05T06:33:29.558514+0000","last_active":"2024-02-05T06:34:45.923730+0000","last_peered":"2024-02-05T06:34:45.923730+0000","last_clean":"2024-02-05T06:34:45.923730+0000","last_became_active":"2024-02-05T06:33:29.557522+0000","last_became_peered":"2024-02-05T06:33:29.557522+0000","last_unstale":"2024-02-05T06:34:45.923730+0000","last_undegraded":"2024-02-05T06:34:45.923730+0000","last_fullsized":"2024-02-05T06:34:45.923730+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:51.837286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"73'15","reported_seq":82,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095250+0000","last_change":"2024-02-05T06:33:31.666410+0000","last_active":"2024-02-05T06:34:45.095250+0000","last_peered":"2024-02-05T06:34:45.095250+0000","last_clean":"2024-02-05T06:34:45.095250+0000","last_became_active":"2024-02-05T06:33:31.666056+0000","last_became_peered":"2024-02-05T06:33:31.666056+0000","last_unstale":"2024-02-05T06:34:45.095250+0000","last_undegraded":"2024-02-05T06:34:45.095250+0000","last_fullsized":"2024-02-05T06:34:45.095250+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:56.860240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"73'8","reported_seq":88,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:49.007791+0000","last_change":"2024-02-05T06:34:49.007791+0000","last_active":"2024-02-05T06:34:49.007791+0000","last_peered":"2024-02-05T06:34:49.007791+0000","last_clean":"2024-02-05T06:34:49.007791+0000","last_became_active":"2024-02-05T06:33:33.589535+0000","last_became_peered":"2024-02-05T06:33:33.589535+0000","last_unstale":"2024-02-05T06:34:49.007791+0000","last_undegraded":"2024-02-05T06:34:49.007791+0000","last_fullsized":"2024-02-05T06:34:49.007791+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:34:49.007659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:49.007659+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:59.892182+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096781+0000","last_change":"2024-02-05T06:33:35.601680+0000","last_active":"2024-02-05T06:34:45.096781+0000","last_peered":"2024-02-05T06:34:45.096781+0000","last_clean":"2024-02-05T06:34:45.096781+0000","last_became_active":"2024-02-05T06:33:35.601030+0000","last_became_peered":"2024-02-05T06:33:35.601030+0000","last_unstale":"2024-02-05T06:34:45.096781+0000","last_undegraded":"2024-02-05T06:34:45.096781+0000","last_fullsized":"2024-02-05T06:34:45.096781+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:53.348191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.923545+0000","last_change":"2024-02-05T06:33:29.558929+0000","last_active":"2024-02-05T06:34:45.923545+0000","last_peered":"2024-02-05T06:34:45.923545+0000","last_clean":"2024-02-05T06:34:45.923545+0000","last_became_active":"2024-02-05T06:33:29.557284+0000","last_became_peered":"2024-02-05T06:33:29.557284+0000","last_unstale":"2024-02-05T06:34:45.923545+0000","last_undegraded":"2024-02-05T06:34:45.923545+0000","last_fullsized":"2024-02-05T06:34:45.923545+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:54.175804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'12","reported_seq":81,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.893212+0000","last_change":"2024-02-05T06:33:31.570242+0000","last_active":"2024-02-05T06:34:44.893212+0000","last_peered":"2024-02-05T06:34:44.893212+0000","last_clean":"2024-02-05T06:34:44.893212+0000","last_became_active":"2024-02-05T06:33:31.569924+0000","last_became_peered":"2024-02-05T06:33:31.569924+0000","last_unstale":"2024-02-05T06:34:44.893212+0000","last_undegraded":"2024-02-05T06:34:44.893212+0000","last_fullsized":"2024-02-05T06:34:44.893212+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:47.971041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.297549+0000","last_change":"2024-02-05T06:34:40.272941+0000","last_active":"2024-02-05T06:34:45.297549+0000","last_peered":"2024-02-05T06:34:45.297549+0000","last_clean":"2024-02-05T06:34:45.297549+0000","last_became_active":"2024-02-05T06:33:33.592546+0000","last_became_peered":"2024-02-05T06:33:33.592546+0000","last_unstale":"2024-02-05T06:34:45.297549+0000","last_undegraded":"2024-02-05T06:34:45.297549+0000","last_fullsized":"2024-02-05T06:34:45.297549+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:40.272852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:40.272852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:07.055905+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.923123+0000","last_change":"2024-02-05T06:33:35.599830+0000","last_active":"2024-02-05T06:34:45.923123+0000","last_peered":"2024-02-05T06:34:45.923123+0000","last_clean":"2024-02-05T06:34:45.923123+0000","last_became_active":"2024-02-05T06:33:35.599527+0000","last_became_peered":"2024-02-05T06:33:35.599527+0000","last_unstale":"2024-02-05T06:34:45.923123+0000","last_undegraded":"2024-02-05T06:34:45.923123+0000","last_fullsized":"2024-02-05T06:34:45.923123+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:58.253161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.297822+0000","last_change":"2024-02-05T06:34:42.265315+0000","last_active":"2024-02-05T06:34:45.297822+0000","last_peered":"2024-02-05T06:34:45.297822+0000","last_clean":"2024-02-05T06:34:45.297822+0000","last_became_active":"2024-02-05T06:33:33.589203+0000","last_became_peered":"2024-02-05T06:33:33.589203+0000","last_unstale":"2024-02-05T06:34:45.297822+0000","last_undegraded":"2024-02-05T06:34:45.297822+0000","last_fullsized":"2024-02-05T06:34:45.297822+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:42.265214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:42.265214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:53.433345+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.923587+0000","last_change":"2024-02-05T06:33:29.556849+0000","last_active":"2024-02-05T06:34:45.923587+0000","last_peered":"2024-02-05T06:34:45.923587+0000","last_clean":"2024-02-05T06:34:45.923587+0000","last_became_active":"2024-02-05T06:33:29.556541+0000","last_became_peered":"2024-02-05T06:33:29.556541+0000","last_unstale":"2024-02-05T06:34:45.923587+0000","last_undegraded":"2024-02-05T06:34:45.923587+0000","last_fullsized":"2024-02-05T06:34:45.923587+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:49.995627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"73'19","reported_seq":97,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.265984+0000","last_change":"2024-02-05T06:34:44.243017+0000","last_active":"2024-02-05T06:34:45.265984+0000","last_peered":"2024-02-05T06:34:45.265984+0000","last_clean":"2024-02-05T06:34:45.265984+0000","last_became_active":"2024-02-05T06:33:31.664699+0000","last_became_peered":"2024-02-05T06:33:31.664699+0000","last_unstale":"2024-02-05T06:34:45.265984+0000","last_undegraded":"2024-02-05T06:34:45.265984+0000","last_fullsized":"2024-02-05T06:34:45.265984+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-02-05T06:34:44.242897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:44.242897+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:01.187407+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.889844+0000","last_change":"2024-02-05T06:33:35.598200+0000","last_active":"2024-02-05T06:34:44.889844+0000","last_peered":"2024-02-05T06:34:44.889844+0000","last_clean":"2024-02-05T06:34:44.889844+0000","last_became_active":"2024-02-05T06:33:35.597863+0000","last_became_peered":"2024-02-05T06:33:35.597863+0000","last_unstale":"2024-02-05T06:34:44.889844+0000","last_undegraded":"2024-02-05T06:34:44.889844+0000","last_fullsized":"2024-02-05T06:34:44.889844+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:55.272938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.299123+0000","last_change":"2024-02-05T06:34:45.299145+0000","last_active":"2024-02-05T06:34:45.299123+0000","last_peered":"2024-02-05T06:34:45.299123+0000","last_clean":"2024-02-05T06:34:45.299123+0000","last_became_active":"2024-02-05T06:33:33.586895+0000","last_became_peered":"2024-02-05T06:33:33.586895+0000","last_unstale":"2024-02-05T06:34:45.299123+0000","last_undegraded":"2024-02-05T06:34:45.299123+0000","last_fullsized":"2024-02-05T06:34:45.299123+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:45.299055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:45.299055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:03.861496+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095649+0000","last_change":"2024-02-05T06:33:29.556254+0000","last_active":"2024-02-05T06:34:45.095649+0000","last_peered":"2024-02-05T06:34:45.095649+0000","last_clean":"2024-02-05T06:34:45.095649+0000","last_became_active":"2024-02-05T06:33:29.555950+0000","last_became_peered":"2024-02-05T06:33:29.555950+0000","last_unstale":"2024-02-05T06:34:45.095649+0000","last_undegraded":"2024-02-05T06:34:45.095649+0000","last_fullsized":"2024-02-05T06:34:45.095649+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:52.533616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"73'9","reported_seq":83,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891706+0000","last_change":"2024-02-05T06:34:36.779561+0000","last_active":"2024-02-05T06:34:44.891706+0000","last_peered":"2024-02-05T06:34:44.891706+0000","last_clean":"2024-02-05T06:34:44.891706+0000","last_became_active":"2024-02-05T06:33:31.564572+0000","last_became_peered":"2024-02-05T06:33:31.564572+0000","last_unstale":"2024-02-05T06:34:44.891706+0000","last_undegraded":"2024-02-05T06:34:44.891706+0000","last_fullsized":"2024-02-05T06:34:44.891706+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:34:36.779466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:36.779466+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:02.144960+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097537+0000","last_change":"2024-02-05T06:33:35.598477+0000","last_active":"2024-02-05T06:34:45.097537+0000","last_peered":"2024-02-05T06:34:45.097537+0000","last_clean":"2024-02-05T06:34:45.097537+0000","last_became_active":"2024-02-05T06:33:35.597750+0000","last_became_peered":"2024-02-05T06:33:35.597750+0000","last_unstale":"2024-02-05T06:34:45.097537+0000","last_undegraded":"2024-02-05T06:34:45.097537+0000","last_fullsized":"2024-02-05T06:34:45.097537+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:53.938845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":80,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:48.046085+0000","last_change":"2024-02-05T06:34:48.046115+0000","last_active":"2024-02-05T06:34:48.046085+0000","last_peered":"2024-02-05T06:34:48.046085+0000","last_clean":"2024-02-05T06:34:48.046085+0000","last_became_active":"2024-02-05T06:33:28.542228+0000","last_became_peered":"2024-02-05T06:33:28.542228+0000","last_unstale":"2024-02-05T06:34:48.046085+0000","last_undegraded":"2024-02-05T06:34:48.046085+0000","last_fullsized":"2024-02-05T06:34:48.046085+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-02-05T06:34:48.046017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:34:48.046017+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:04.358520+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001220348,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":62,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.892994+0000","last_change":"2024-02-05T06:33:29.554533+0000","last_active":"2024-02-05T06:34:44.892994+0000","last_peered":"2024-02-05T06:34:44.892994+0000","last_clean":"2024-02-05T06:34:44.892994+0000","last_became_active":"2024-02-05T06:33:29.553882+0000","last_became_peered":"2024-02-05T06:33:29.553882+0000","last_unstale":"2024-02-05T06:34:44.892994+0000","last_undegraded":"2024-02-05T06:34:44.892994+0000","last_fullsized":"2024-02-05T06:34:44.892994+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:56.794367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":72,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095506+0000","last_change":"2024-02-05T06:33:55.896865+0000","last_active":"2024-02-05T06:34:45.095506+0000","last_peered":"2024-02-05T06:34:45.095506+0000","last_clean":"2024-02-05T06:34:45.095506+0000","last_became_active":"2024-02-05T06:33:55.884601+0000","last_became_peered":"2024-02-05T06:33:55.884601+0000","last_unstale":"2024-02-05T06:34:45.095506+0000","last_undegraded":"2024-02-05T06:34:45.095506+0000","last_fullsized":"2024-02-05T06:34:45.095506+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:55.551394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":55,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890270+0000","last_change":"2024-02-05T06:33:33.589668+0000","last_active":"2024-02-05T06:34:44.890270+0000","last_peered":"2024-02-05T06:34:44.890270+0000","last_clean":"2024-02-05T06:34:44.890270+0000","last_became_active":"2024-02-05T06:33:33.589348+0000","last_became_peered":"2024-02-05T06:33:33.589348+0000","last_unstale":"2024-02-05T06:34:44.890270+0000","last_undegraded":"2024-02-05T06:34:44.890270+0000","last_fullsized":"2024-02-05T06:34:44.890270+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.923731+0000","last_change":"2024-02-05T06:33:35.666590+0000","last_active":"2024-02-05T06:34:45.923731+0000","last_peered":"2024-02-05T06:34:45.923731+0000","last_clean":"2024-02-05T06:34:45.923731+0000","last_became_active":"2024-02-05T06:33:35.666324+0000","last_became_peered":"2024-02-05T06:33:35.666324+0000","last_unstale":"2024-02-05T06:34:45.923731+0000","last_undegraded":"2024-02-05T06:34:45.923731+0000","last_fullsized":"2024-02-05T06:34:45.923731+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:51.202340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":62,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890767+0000","last_change":"2024-02-05T06:33:29.552003+0000","last_active":"2024-02-05T06:34:44.890767+0000","last_peered":"2024-02-05T06:34:44.890767+0000","last_clean":"2024-02-05T06:34:44.890767+0000","last_became_active":"2024-02-05T06:33:29.551549+0000","last_became_peered":"2024-02-05T06:33:29.551549+0000","last_unstale":"2024-02-05T06:34:44.890767+0000","last_undegraded":"2024-02-05T06:34:44.890767+0000","last_fullsized":"2024-02-05T06:34:44.890767+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:56.472943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":78,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:50.302189+0000","last_change":"2024-02-05T06:34:50.302211+0000","last_active":"2024-02-05T06:34:50.302189+0000","last_peered":"2024-02-05T06:34:50.302189+0000","last_clean":"2024-02-05T06:34:50.302189+0000","last_became_active":"2024-02-05T06:33:28.542977+0000","last_became_peered":"2024-02-05T06:33:28.542977+0000","last_unstale":"2024-02-05T06:34:50.302189+0000","last_undegraded":"2024-02-05T06:34:50.302189+0000","last_fullsized":"2024-02-05T06:34:50.302189+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:50.302133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:34:50.302133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:57.608189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091640799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'13","reported_seq":85,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891357+0000","last_change":"2024-02-05T06:33:31.562891+0000","last_active":"2024-02-05T06:34:44.891357+0000","last_peered":"2024-02-05T06:34:44.891357+0000","last_clean":"2024-02-05T06:34:44.891357+0000","last_became_active":"2024-02-05T06:33:31.560082+0000","last_became_peered":"2024-02-05T06:33:31.560082+0000","last_unstale":"2024-02-05T06:34:44.891357+0000","last_undegraded":"2024-02-05T06:34:44.891357+0000","last_fullsized":"2024-02-05T06:34:44.891357+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:50.831110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.298136+0000","last_change":"2024-02-05T06:34:34.240413+0000","last_active":"2024-02-05T06:34:45.298136+0000","last_peered":"2024-02-05T06:34:45.298136+0000","last_clean":"2024-02-05T06:34:45.298136+0000","last_became_active":"2024-02-05T06:33:33.587593+0000","last_became_peered":"2024-02-05T06:33:33.587593+0000","last_unstale":"2024-02-05T06:34:45.298136+0000","last_undegraded":"2024-02-05T06:34:45.298136+0000","last_fullsized":"2024-02-05T06:34:45.298136+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:34.240331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:34.240331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:49.655873+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095927+0000","last_change":"2024-02-05T06:33:35.600278+0000","last_active":"2024-02-05T06:34:45.095927+0000","last_peered":"2024-02-05T06:34:45.095927+0000","last_clean":"2024-02-05T06:34:45.095927+0000","last_became_active":"2024-02-05T06:33:35.600004+0000","last_became_peered":"2024-02-05T06:33:35.600004+0000","last_unstale":"2024-02-05T06:34:45.095927+0000","last_undegraded":"2024-02-05T06:34:45.095927+0000","last_fullsized":"2024-02-05T06:34:45.095927+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:59.635646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":317,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.923419+0000","last_change":"2024-02-05T06:33:45.752222+0000","last_active":"2024-02-05T06:34:45.923419+0000","last_peered":"2024-02-05T06:34:45.923419+0000","last_clean":"2024-02-05T06:34:45.923419+0000","last_became_active":"2024-02-05T06:33:19.862877+0000","last_became_peered":"2024-02-05T06:33:19.862877+0000","last_unstale":"2024-02-05T06:34:45.923419+0000","last_undegraded":"2024-02-05T06:34:45.923419+0000","last_fullsized":"2024-02-05T06:34:45.923419+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-02-05T06:33:45.752124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:33:45.752124+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:54.331188+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":71,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097177+0000","last_change":"2024-02-05T06:34:44.894533+0000","last_active":"2024-02-05T06:34:45.097177+0000","last_peered":"2024-02-05T06:34:45.097177+0000","last_clean":"2024-02-05T06:34:45.097177+0000","last_became_active":"2024-02-05T06:33:29.552698+0000","last_became_peered":"2024-02-05T06:33:29.552698+0000","last_unstale":"2024-02-05T06:34:45.097177+0000","last_undegraded":"2024-02-05T06:34:45.097177+0000","last_fullsized":"2024-02-05T06:34:45.097177+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:44.894443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:44.894443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:54.598610+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"73'16","reported_seq":97,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.266486+0000","last_change":"2024-02-05T06:33:31.666149+0000","last_active":"2024-02-05T06:34:45.266486+0000","last_peered":"2024-02-05T06:34:45.266486+0000","last_clean":"2024-02-05T06:34:45.266486+0000","last_became_active":"2024-02-05T06:33:31.665797+0000","last_became_peered":"2024-02-05T06:33:31.665797+0000","last_unstale":"2024-02-05T06:34:45.266486+0000","last_undegraded":"2024-02-05T06:34:45.266486+0000","last_fullsized":"2024-02-05T06:34:45.266486+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:00.032828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":55,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.922387+0000","last_change":"2024-02-05T06:33:33.591727+0000","last_active":"2024-02-05T06:34:45.922387+0000","last_peered":"2024-02-05T06:34:45.922387+0000","last_clean":"2024-02-05T06:34:45.922387+0000","last_became_active":"2024-02-05T06:33:33.591484+0000","last_became_peered":"2024-02-05T06:33:33.591484+0000","last_unstale":"2024-02-05T06:34:45.922387+0000","last_undegraded":"2024-02-05T06:34:45.922387+0000","last_fullsized":"2024-02-05T06:34:45.922387+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:56.324621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":58,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:46.361083+0000","last_change":"2024-02-05T06:34:46.361122+0000","last_active":"2024-02-05T06:34:46.361083+0000","last_peered":"2024-02-05T06:34:46.361083+0000","last_clean":"2024-02-05T06:34:46.361083+0000","last_became_active":"2024-02-05T06:33:35.600656+0000","last_became_peered":"2024-02-05T06:33:35.600656+0000","last_unstale":"2024-02-05T06:34:46.361083+0000","last_undegraded":"2024-02-05T06:34:46.361083+0000","last_fullsized":"2024-02-05T06:34:46.361083+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:46.361013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:46.361013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:07.997590+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":71,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890127+0000","last_change":"2024-02-05T06:34:44.781805+0000","last_active":"2024-02-05T06:34:44.890127+0000","last_peered":"2024-02-05T06:34:44.890127+0000","last_clean":"2024-02-05T06:34:44.890127+0000","last_became_active":"2024-02-05T06:33:29.544399+0000","last_became_peered":"2024-02-05T06:33:29.544399+0000","last_unstale":"2024-02-05T06:34:44.890127+0000","last_undegraded":"2024-02-05T06:34:44.890127+0000","last_fullsized":"2024-02-05T06:34:44.890127+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:44.781704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:44.781704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:06.382435+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":157,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:49.422071+0000","last_change":"2024-02-05T06:34:41.991897+0000","last_active":"2024-02-05T06:34:49.422071+0000","last_peered":"2024-02-05T06:34:49.422071+0000","last_clean":"2024-02-05T06:34:49.422071+0000","last_became_active":"2024-02-05T06:33:28.544852+0000","last_became_peered":"2024-02-05T06:33:28.544852+0000","last_unstale":"2024-02-05T06:34:49.422071+0000","last_undegraded":"2024-02-05T06:34:49.422071+0000","last_fullsized":"2024-02-05T06:34:49.422071+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T06:34:41.991777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:34:41.991777+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:10.879161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00074151600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":74,"num_read_kb":69,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":77,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890848+0000","last_change":"2024-02-05T06:33:31.564936+0000","last_active":"2024-02-05T06:34:44.890848+0000","last_peered":"2024-02-05T06:34:44.890848+0000","last_clean":"2024-02-05T06:34:44.890848+0000","last_became_active":"2024-02-05T06:33:31.564604+0000","last_became_peered":"2024-02-05T06:33:31.564604+0000","last_unstale":"2024-02-05T06:34:44.890848+0000","last_undegraded":"2024-02-05T06:34:44.890848+0000","last_fullsized":"2024-02-05T06:34:44.890848+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":53,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096253+0000","last_change":"2024-02-05T06:33:33.589962+0000","last_active":"2024-02-05T06:34:45.096253+0000","last_peered":"2024-02-05T06:34:45.096253+0000","last_clean":"2024-02-05T06:34:45.096253+0000","last_became_active":"2024-02-05T06:33:33.589388+0000","last_became_peered":"2024-02-05T06:33:33.589388+0000","last_unstale":"2024-02-05T06:34:45.096253+0000","last_undegraded":"2024-02-05T06:34:45.096253+0000","last_fullsized":"2024-02-05T06:34:45.096253+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:01.833071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":50,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890056+0000","last_change":"2024-02-05T06:33:35.665208+0000","last_active":"2024-02-05T06:34:44.890056+0000","last_peered":"2024-02-05T06:34:44.890056+0000","last_clean":"2024-02-05T06:34:44.890056+0000","last_became_active":"2024-02-05T06:33:35.664858+0000","last_became_peered":"2024-02-05T06:33:35.664858+0000","last_unstale":"2024-02-05T06:34:44.890056+0000","last_undegraded":"2024-02-05T06:34:44.890056+0000","last_fullsized":"2024-02-05T06:34:44.890056+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:53.295103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"66'1","reported_seq":76,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890784+0000","last_change":"2024-02-05T06:34:36.457175+0000","last_active":"2024-02-05T06:34:44.890784+0000","last_peered":"2024-02-05T06:34:44.890784+0000","last_clean":"2024-02-05T06:34:44.890784+0000","last_became_active":"2024-02-05T06:33:29.548227+0000","last_became_peered":"2024-02-05T06:33:29.548227+0000","last_unstale":"2024-02-05T06:34:44.890784+0000","last_undegraded":"2024-02-05T06:34:44.890784+0000","last_fullsized":"2024-02-05T06:34:44.890784+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:34:36.457059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:36.457059+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:45.403222+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'19","reported_seq":95,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891811+0000","last_change":"2024-02-05T06:33:31.667636+0000","last_active":"2024-02-05T06:34:44.891811+0000","last_peered":"2024-02-05T06:34:44.891811+0000","last_clean":"2024-02-05T06:34:44.891811+0000","last_became_active":"2024-02-05T06:33:31.667340+0000","last_became_peered":"2024-02-05T06:33:31.667340+0000","last_unstale":"2024-02-05T06:34:44.891811+0000","last_undegraded":"2024-02-05T06:34:44.891811+0000","last_fullsized":"2024-02-05T06:34:44.891811+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:45.776316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":62,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:50.353126+0000","last_change":"2024-02-05T06:34:50.353163+0000","last_active":"2024-02-05T06:34:50.353126+0000","last_peered":"2024-02-05T06:34:50.353126+0000","last_clean":"2024-02-05T06:34:50.353126+0000","last_became_active":"2024-02-05T06:33:33.588163+0000","last_became_peered":"2024-02-05T06:33:33.588163+0000","last_unstale":"2024-02-05T06:34:50.353126+0000","last_undegraded":"2024-02-05T06:34:50.353126+0000","last_fullsized":"2024-02-05T06:34:50.353126+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:50.353040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:50.353040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:10.822758+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.889406+0000","last_change":"2024-02-05T06:33:35.600831+0000","last_active":"2024-02-05T06:34:44.889406+0000","last_peered":"2024-02-05T06:34:44.889406+0000","last_clean":"2024-02-05T06:34:44.889406+0000","last_became_active":"2024-02-05T06:33:35.600485+0000","last_became_peered":"2024-02-05T06:33:35.600485+0000","last_unstale":"2024-02-05T06:34:44.889406+0000","last_undegraded":"2024-02-05T06:34:44.889406+0000","last_fullsized":"2024-02-05T06:34:44.889406+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:51.006304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095967+0000","last_change":"2024-02-05T06:34:43.098655+0000","last_active":"2024-02-05T06:34:45.095967+0000","last_peered":"2024-02-05T06:34:45.095967+0000","last_clean":"2024-02-05T06:34:45.095967+0000","last_became_active":"2024-02-05T06:33:29.554940+0000","last_became_peered":"2024-02-05T06:33:29.554940+0000","last_unstale":"2024-02-05T06:34:45.095967+0000","last_undegraded":"2024-02-05T06:34:45.095967+0000","last_fullsized":"2024-02-05T06:34:45.095967+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:43.098546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:43.098546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:01.990483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"73'10","reported_seq":73,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891541+0000","last_change":"2024-02-05T06:33:31.570032+0000","last_active":"2024-02-05T06:34:44.891541+0000","last_peered":"2024-02-05T06:34:44.891541+0000","last_clean":"2024-02-05T06:34:44.891541+0000","last_became_active":"2024-02-05T06:33:31.569680+0000","last_became_peered":"2024-02-05T06:33:31.569680+0000","last_unstale":"2024-02-05T06:34:44.891541+0000","last_undegraded":"2024-02-05T06:34:44.891541+0000","last_fullsized":"2024-02-05T06:34:44.891541+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:57.891117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"73'8","reported_seq":80,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891632+0000","last_change":"2024-02-05T06:33:33.582537+0000","last_active":"2024-02-05T06:34:44.891632+0000","last_peered":"2024-02-05T06:34:44.891632+0000","last_clean":"2024-02-05T06:34:44.891632+0000","last_became_active":"2024-02-05T06:33:33.582187+0000","last_became_peered":"2024-02-05T06:33:33.582187+0000","last_unstale":"2024-02-05T06:34:44.891632+0000","last_undegraded":"2024-02-05T06:34:44.891632+0000","last_fullsized":"2024-02-05T06:34:44.891632+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":59,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.923213+0000","last_change":"2024-02-05T06:34:41.001666+0000","last_active":"2024-02-05T06:34:45.923213+0000","last_peered":"2024-02-05T06:34:45.923213+0000","last_clean":"2024-02-05T06:34:45.923213+0000","last_became_active":"2024-02-05T06:33:35.666315+0000","last_became_peered":"2024-02-05T06:33:35.666315+0000","last_unstale":"2024-02-05T06:34:45.923213+0000","last_undegraded":"2024-02-05T06:34:45.923213+0000","last_fullsized":"2024-02-05T06:34:45.923213+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:41.001570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:41.001570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:53.416960+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":64,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890941+0000","last_change":"2024-02-05T06:33:29.546683+0000","last_active":"2024-02-05T06:34:44.890941+0000","last_peered":"2024-02-05T06:34:44.890941+0000","last_clean":"2024-02-05T06:34:44.890941+0000","last_became_active":"2024-02-05T06:33:29.546405+0000","last_became_peered":"2024-02-05T06:33:29.546405+0000","last_unstale":"2024-02-05T06:34:44.890941+0000","last_undegraded":"2024-02-05T06:34:44.890941+0000","last_fullsized":"2024-02-05T06:34:44.890941+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:45.453250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'14","reported_seq":79,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.893000+0000","last_change":"2024-02-05T06:33:31.564209+0000","last_active":"2024-02-05T06:34:44.893000+0000","last_peered":"2024-02-05T06:34:44.893000+0000","last_clean":"2024-02-05T06:34:44.893000+0000","last_became_active":"2024-02-05T06:33:31.563836+0000","last_became_peered":"2024-02-05T06:33:31.563836+0000","last_unstale":"2024-02-05T06:34:44.893000+0000","last_undegraded":"2024-02-05T06:34:44.893000+0000","last_fullsized":"2024-02-05T06:34:44.893000+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:47.565469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"73'8","reported_seq":80,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096985+0000","last_change":"2024-02-05T06:33:33.591940+0000","last_active":"2024-02-05T06:34:45.096985+0000","last_peered":"2024-02-05T06:34:45.096985+0000","last_clean":"2024-02-05T06:34:45.096985+0000","last_became_active":"2024-02-05T06:33:33.591646+0000","last_became_peered":"2024-02-05T06:33:33.591646+0000","last_unstale":"2024-02-05T06:34:45.096985+0000","last_undegraded":"2024-02-05T06:34:45.096985+0000","last_fullsized":"2024-02-05T06:34:45.096985+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":50,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.892762+0000","last_change":"2024-02-05T06:33:35.600296+0000","last_active":"2024-02-05T06:34:44.892762+0000","last_peered":"2024-02-05T06:34:44.892762+0000","last_clean":"2024-02-05T06:34:44.892762+0000","last_became_active":"2024-02-05T06:33:35.599805+0000","last_became_peered":"2024-02-05T06:33:35.599805+0000","last_unstale":"2024-02-05T06:34:44.892762+0000","last_undegraded":"2024-02-05T06:34:44.892762+0000","last_fullsized":"2024-02-05T06:34:44.892762+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:03.912813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097711+0000","last_change":"2024-02-05T06:33:29.557121+0000","last_active":"2024-02-05T06:34:45.097711+0000","last_peered":"2024-02-05T06:34:45.097711+0000","last_clean":"2024-02-05T06:34:45.097711+0000","last_became_active":"2024-02-05T06:33:29.556002+0000","last_became_peered":"2024-02-05T06:33:29.556002+0000","last_unstale":"2024-02-05T06:34:45.097711+0000","last_undegraded":"2024-02-05T06:34:45.097711+0000","last_fullsized":"2024-02-05T06:34:45.097711+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:48.546981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"73'18","reported_seq":91,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096701+0000","last_change":"2024-02-05T06:33:31.666298+0000","last_active":"2024-02-05T06:34:45.096701+0000","last_peered":"2024-02-05T06:34:45.096701+0000","last_clean":"2024-02-05T06:34:45.096701+0000","last_became_active":"2024-02-05T06:33:31.665995+0000","last_became_peered":"2024-02-05T06:33:31.665995+0000","last_unstale":"2024-02-05T06:34:45.096701+0000","last_undegraded":"2024-02-05T06:34:45.096701+0000","last_fullsized":"2024-02-05T06:34:45.096701+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:47.201122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":62,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:49.579564+0000","last_change":"2024-02-05T06:34:49.579591+0000","last_active":"2024-02-05T06:34:49.579564+0000","last_peered":"2024-02-05T06:34:49.579564+0000","last_clean":"2024-02-05T06:34:49.579564+0000","last_became_active":"2024-02-05T06:33:33.586935+0000","last_became_peered":"2024-02-05T06:33:33.586935+0000","last_unstale":"2024-02-05T06:34:49.579564+0000","last_undegraded":"2024-02-05T06:34:49.579564+0000","last_fullsized":"2024-02-05T06:34:49.579564+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:49.579493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:49.579493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:00.372885+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095645+0000","last_change":"2024-02-05T06:33:35.593461+0000","last_active":"2024-02-05T06:34:45.095645+0000","last_peered":"2024-02-05T06:34:45.095645+0000","last_clean":"2024-02-05T06:34:45.095645+0000","last_became_active":"2024-02-05T06:33:35.593210+0000","last_became_peered":"2024-02-05T06:33:35.593210+0000","last_unstale":"2024-02-05T06:34:45.095645+0000","last_undegraded":"2024-02-05T06:34:45.095645+0000","last_fullsized":"2024-02-05T06:34:45.095645+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":87,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.894006+0000","last_change":"2024-02-05T06:34:40.619333+0000","last_active":"2024-02-05T06:34:44.894006+0000","last_peered":"2024-02-05T06:34:44.894006+0000","last_clean":"2024-02-05T06:34:44.894006+0000","last_became_active":"2024-02-05T06:33:33.586753+0000","last_became_peered":"2024-02-05T06:33:33.586753+0000","last_unstale":"2024-02-05T06:34:44.894006+0000","last_undegraded":"2024-02-05T06:34:44.894006+0000","last_fullsized":"2024-02-05T06:34:44.894006+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:34:40.619192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:40.619192+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:48.417494+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096676+0000","last_change":"2024-02-05T06:33:29.557073+0000","last_active":"2024-02-05T06:34:45.096676+0000","last_peered":"2024-02-05T06:34:45.096676+0000","last_clean":"2024-02-05T06:34:45.096676+0000","last_became_active":"2024-02-05T06:33:29.555919+0000","last_became_peered":"2024-02-05T06:33:29.555919+0000","last_unstale":"2024-02-05T06:34:45.096676+0000","last_undegraded":"2024-02-05T06:34:45.096676+0000","last_fullsized":"2024-02-05T06:34:45.096676+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:54.542025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"73'15","reported_seq":83,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891508+0000","last_change":"2024-02-05T06:33:31.572905+0000","last_active":"2024-02-05T06:34:44.891508+0000","last_peered":"2024-02-05T06:34:44.891508+0000","last_clean":"2024-02-05T06:34:44.891508+0000","last_became_active":"2024-02-05T06:33:31.572603+0000","last_became_peered":"2024-02-05T06:33:31.572603+0000","last_unstale":"2024-02-05T06:34:44.891508+0000","last_undegraded":"2024-02-05T06:34:44.891508+0000","last_fullsized":"2024-02-05T06:34:44.891508+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:49.104679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096917+0000","last_change":"2024-02-05T06:33:35.668457+0000","last_active":"2024-02-05T06:34:45.096917+0000","last_peered":"2024-02-05T06:34:45.096917+0000","last_clean":"2024-02-05T06:34:45.096917+0000","last_became_active":"2024-02-05T06:33:35.668020+0000","last_became_peered":"2024-02-05T06:33:35.668020+0000","last_unstale":"2024-02-05T06:34:45.096917+0000","last_undegraded":"2024-02-05T06:34:45.096917+0000","last_fullsized":"2024-02-05T06:34:45.096917+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:55.572974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":53,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096246+0000","last_change":"2024-02-05T06:33:33.583518+0000","last_active":"2024-02-05T06:34:45.096246+0000","last_peered":"2024-02-05T06:34:45.096246+0000","last_clean":"2024-02-05T06:34:45.096246+0000","last_became_active":"2024-02-05T06:33:33.583029+0000","last_became_peered":"2024-02-05T06:33:33.583029+0000","last_unstale":"2024-02-05T06:34:45.096246+0000","last_undegraded":"2024-02-05T06:34:45.096246+0000","last_fullsized":"2024-02-05T06:34:45.096246+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:49.769832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":62,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.892790+0000","last_change":"2024-02-05T06:33:29.553248+0000","last_active":"2024-02-05T06:34:44.892790+0000","last_peered":"2024-02-05T06:34:44.892790+0000","last_clean":"2024-02-05T06:34:44.892790+0000","last_became_active":"2024-02-05T06:33:29.552415+0000","last_became_peered":"2024-02-05T06:33:29.552415+0000","last_unstale":"2024-02-05T06:34:44.892790+0000","last_undegraded":"2024-02-05T06:34:44.892790+0000","last_fullsized":"2024-02-05T06:34:44.892790+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:56.521015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"73'11","reported_seq":103,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:50.592882+0000","last_change":"2024-02-05T06:34:50.592920+0000","last_active":"2024-02-05T06:34:50.592882+0000","last_peered":"2024-02-05T06:34:50.592882+0000","last_clean":"2024-02-05T06:34:50.592882+0000","last_became_active":"2024-02-05T06:33:55.885523+0000","last_became_peered":"2024-02-05T06:33:55.885523+0000","last_unstale":"2024-02-05T06:34:50.592882+0000","last_undegraded":"2024-02-05T06:34:50.592882+0000","last_fullsized":"2024-02-05T06:34:50.592882+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-02-05T06:34:50.592793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:50.592793+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:56.891427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:49.298317+0000","last_change":"2024-02-05T06:34:49.298355+0000","last_active":"2024-02-05T06:34:49.298317+0000","last_peered":"2024-02-05T06:34:49.298317+0000","last_clean":"2024-02-05T06:34:49.298317+0000","last_became_active":"2024-02-05T06:33:35.598593+0000","last_became_peered":"2024-02-05T06:33:35.598593+0000","last_unstale":"2024-02-05T06:34:49.298317+0000","last_undegraded":"2024-02-05T06:34:49.298317+0000","last_fullsized":"2024-02-05T06:34:49.298317+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:49.298233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:49.298233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:07.506646+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":54,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890955+0000","last_change":"2024-02-05T06:33:33.586971+0000","last_active":"2024-02-05T06:34:44.890955+0000","last_peered":"2024-02-05T06:34:44.890955+0000","last_clean":"2024-02-05T06:34:44.890955+0000","last_became_active":"2024-02-05T06:33:33.586598+0000","last_became_peered":"2024-02-05T06:33:33.586598+0000","last_unstale":"2024-02-05T06:34:44.890955+0000","last_undegraded":"2024-02-05T06:34:44.890955+0000","last_fullsized":"2024-02-05T06:34:44.890955+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:49.228193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":62,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.266181+0000","last_change":"2024-02-05T06:33:29.550887+0000","last_active":"2024-02-05T06:34:45.266181+0000","last_peered":"2024-02-05T06:34:45.266181+0000","last_clean":"2024-02-05T06:34:45.266181+0000","last_became_active":"2024-02-05T06:33:29.550612+0000","last_became_peered":"2024-02-05T06:33:29.550612+0000","last_unstale":"2024-02-05T06:34:45.266181+0000","last_undegraded":"2024-02-05T06:34:45.266181+0000","last_fullsized":"2024-02-05T06:34:45.266181+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:55.717410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"73'17","reported_seq":86,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.893580+0000","last_change":"2024-02-05T06:33:31.566941+0000","last_active":"2024-02-05T06:34:44.893580+0000","last_peered":"2024-02-05T06:34:44.893580+0000","last_clean":"2024-02-05T06:34:44.893580+0000","last_became_active":"2024-02-05T06:33:31.566576+0000","last_became_peered":"2024-02-05T06:33:31.566576+0000","last_unstale":"2024-02-05T06:34:44.893580+0000","last_undegraded":"2024-02-05T06:34:44.893580+0000","last_fullsized":"2024-02-05T06:34:44.893580+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:00.400128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":50,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.893991+0000","last_change":"2024-02-05T06:33:35.597953+0000","last_active":"2024-02-05T06:34:44.893991+0000","last_peered":"2024-02-05T06:34:44.893991+0000","last_clean":"2024-02-05T06:34:44.893991+0000","last_became_active":"2024-02-05T06:33:35.597460+0000","last_became_peered":"2024-02-05T06:33:35.597460+0000","last_unstale":"2024-02-05T06:34:44.893991+0000","last_undegraded":"2024-02-05T06:34:44.893991+0000","last_fullsized":"2024-02-05T06:34:44.893991+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:00.813671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":78,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.297964+0000","last_change":"2024-02-05T06:33:33.588787+0000","last_active":"2024-02-05T06:34:45.297964+0000","last_peered":"2024-02-05T06:34:45.297964+0000","last_clean":"2024-02-05T06:34:45.297964+0000","last_became_active":"2024-02-05T06:33:33.588478+0000","last_became_peered":"2024-02-05T06:33:33.588478+0000","last_unstale":"2024-02-05T06:34:45.297964+0000","last_undegraded":"2024-02-05T06:34:45.297964+0000","last_fullsized":"2024-02-05T06:34:45.297964+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":71,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097232+0000","last_change":"2024-02-05T06:34:29.822718+0000","last_active":"2024-02-05T06:34:45.097232+0000","last_peered":"2024-02-05T06:34:45.097232+0000","last_clean":"2024-02-05T06:34:45.097232+0000","last_became_active":"2024-02-05T06:33:29.545073+0000","last_became_peered":"2024-02-05T06:33:29.545073+0000","last_unstale":"2024-02-05T06:34:45.097232+0000","last_undegraded":"2024-02-05T06:34:45.097232+0000","last_fullsized":"2024-02-05T06:34:45.097232+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:29.822634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:29.822634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:37.035501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"73'10","reported_seq":73,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.893424+0000","last_change":"2024-02-05T06:33:31.568457+0000","last_active":"2024-02-05T06:34:44.893424+0000","last_peered":"2024-02-05T06:34:44.893424+0000","last_clean":"2024-02-05T06:34:44.893424+0000","last_became_active":"2024-02-05T06:33:31.568074+0000","last_became_peered":"2024-02-05T06:33:31.568074+0000","last_unstale":"2024-02-05T06:34:44.893424+0000","last_undegraded":"2024-02-05T06:34:44.893424+0000","last_fullsized":"2024-02-05T06:34:44.893424+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:52.367398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":422,"ondisk_log_size":422,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":69,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103188,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32600,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712552,"statfs":{"total":95995035648,"available":95961653248,"internally_reserved":0,"allocated":1458176,"data_stored":970343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.55700000000000005}]},{"osd":1,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.46600000000000003}]},{"osd":2,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.57899999999999996}]},{"osd":3,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.503}]},{"osd":4,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.59799999999999998}]},{"osd":5,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.66400000000000003}]},{"osd":6,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.64600000000000002}]}]},{"osd":6,"up_from":49,"seq":210453397530,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32576,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712576,"statfs":{"total":95995035648,"available":95961677824,"internally_reserved":0,"allocated":1433600,"data_stored":968894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:33:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.68000000000000005}]},{"osd":1,"last update":"Mon Feb 5 06:33:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.51600000000000001}]},{"osd":2,"last update":"Mon Feb 5 06:33:51 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.54900000000000004}]},{"osd":3,"last update":"Mon Feb 5 06:33:51 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.59199999999999997}]},{"osd":4,"last update":"Mon Feb 5 06:33:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.355,"5min":1.355,"15min":1.355},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.379}]},{"osd":5,"last update":"Mon Feb 5 06:33:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.53200000000000003}]},{"osd":7,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.432}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":884736,"data_stored":380510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.36699999999999999,"15min":0.34000000000000002},"min":{"1min":0.26800000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.77600000000000002,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.38700000000000001,"15min":0.34999999999999998},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52800000000000002,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.34699999999999998}]},{"osd":2,"last update":"Mon Feb 5 06:34:13 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.39900000000000002,"15min":0.39200000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.59999999999999998,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.38700000000000001,"15min":0.377},"min":{"1min":0.26900000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.36799999999999999}]},{"osd":3,"last update":"Mon Feb 5 06:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.42999999999999999,"15min":0.42199999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.503,"5min":0.48799999999999999,"15min":0.47799999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.83399999999999996,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.45500000000000002}]},{"osd":4,"last update":"Mon Feb 5 06:34:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56699999999999995,"15min":0.56100000000000005},"min":{"1min":0.434,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.62,"5min":0.58399999999999996,"15min":0.57799999999999996},"min":{"1min":0.39400000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.55700000000000005}]},{"osd":5,"last update":"Mon Feb 5 06:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.92800000000000005,"15min":0.98499999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86499999999999999,"5min":8.2509999999999994,"15min":8.2509999999999994},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.624,"15min":0.63400000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.60799999999999998}]},{"osd":6,"last update":"Mon Feb 5 06:33:52 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.66700000000000004}]},{"osd":7,"last update":"Mon Feb 5 06:34:16 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.68000000000000005},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":8.2219999999999995,"5min":8.2219999999999995,"15min":8.2219999999999995},"last":0.58199999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32608,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712544,"statfs":{"total":95995035648,"available":95961645056,"internally_reserved":0,"allocated":1466368,"data_stored":970099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 06:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.38900000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.188,"15min":0.188},"max":{"1min":4.2729999999999997,"5min":4.2729999999999997,"15min":4.2729999999999997},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.377,"5min":0.36599999999999999,"15min":0.33600000000000002},"min":{"1min":0.26800000000000002,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.629,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.45700000000000002}]},{"osd":2,"last update":"Mon Feb 5 06:34:11 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.435,"15min":0.41299999999999998},"min":{"1min":0.315,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.42599999999999999},"min":{"1min":0.28999999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.437}]},{"osd":3,"last update":"Mon Feb 5 06:34:37 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.47499999999999998,"15min":0.46200000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":4.3289999999999997,"5min":4.3289999999999997,"15min":4.3289999999999997},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45100000000000001},"min":{"1min":0.28499999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.68500000000000005,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.503}]},{"osd":4,"last update":"Mon Feb 5 06:34:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.59799999999999998,"15min":0.59499999999999997},"min":{"1min":0.441,"5min":0.372,"15min":0.372},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.60899999999999999,"15min":0.60499999999999998},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.86899999999999999}]},{"osd":5,"last update":"Mon Feb 5 06:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.628,"15min":0.63600000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.73399999999999999,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.64200000000000002,"15min":0.64700000000000002},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.77900000000000003,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.78200000000000003}]},{"osd":6,"last update":"Mon Feb 5 06:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.68100000000000005,"15min":0.65200000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":5.2220000000000004,"5min":5.2220000000000004,"15min":5.2220000000000004},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.624,"5min":0.64800000000000002,"15min":0.65200000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.95399999999999996}]},{"osd":7,"last update":"Mon Feb 5 06:34:16 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":5.2670000000000003,"5min":5.2670000000000003,"15min":5.2670000000000003},"last":0.90800000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32008,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713144,"statfs":{"total":95995035648,"available":95962259456,"internally_reserved":0,"allocated":851968,"data_stored":379844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:34:15 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.38100000000000001,"15min":0.36599999999999999},"min":{"1min":0.27000000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.59999999999999998,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.317},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.41199999999999998,"15min":0.38400000000000001},"min":{"1min":0.32000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.58099999999999996,"5min":0.752,"15min":0.752},"last":0.38600000000000001}]},{"osd":1,"last update":"Mon Feb 5 06:34:15 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39900000000000002,"15min":0.38900000000000001},"min":{"1min":0.26200000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.56000000000000005,"5min":0.626,"15min":0.626},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.44800000000000001,"15min":0.42899999999999999},"min":{"1min":0.33600000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.68300000000000005,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.36699999999999999}]},{"osd":3,"last update":"Mon Feb 5 06:34:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.47999999999999998,"15min":0.46999999999999997},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72499999999999998,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.47999999999999998,"15min":0.48199999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.66200000000000003,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.34399999999999997}]},{"osd":4,"last update":"Mon Feb 5 06:34:00 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.65000000000000002,"15min":0.64700000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0009999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.495,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.90000000000000002,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.61599999999999999}]},{"osd":5,"last update":"Mon Feb 5 06:34:30 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.69199999999999995,"15min":0.70299999999999996},"min":{"1min":0.47199999999999998,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.89600000000000002,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.66600000000000004,"15min":0.68000000000000005},"min":{"1min":0.42399999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.85299999999999998,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.79400000000000004}]},{"osd":6,"last update":"Mon Feb 5 06:33:51 2024","interfaces":[{"interface":"back","average":{"1min":1.7210000000000001,"5min":1.7210000000000001,"15min":1.7210000000000001},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":17.143999999999998,"5min":17.143999999999998,"15min":17.143999999999998},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.77500000000000002}]},{"osd":7,"last update":"Mon Feb 5 06:34:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.70899999999999996}]}]},{"osd":3,"up_from":26,"seq":111669149737,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":876544,"data_stored":379417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:34:44 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.38600000000000001,"15min":0.39300000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.52000000000000002,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.44500000000000001,"15min":0.45400000000000001},"min":{"1min":0.316,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.55000000000000004,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.59399999999999997}]},{"osd":1,"last update":"Mon Feb 5 06:34:44 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.45800000000000002,"15min":0.47199999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.57099999999999995,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.44500000000000001,"15min":0.45300000000000001},"min":{"1min":0.27700000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.57599999999999996,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.33500000000000002}]},{"osd":2,"last update":"Mon Feb 5 06:34:44 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.48099999999999998,"15min":0.495},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.64600000000000002,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.46000000000000002,"15min":0.46899999999999997},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.64800000000000002,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.66900000000000004}]},{"osd":4,"last update":"Mon Feb 5 06:34:05 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.59899999999999998,"15min":0.61499999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.627,"5min":1.008,"15min":1.008},"last":0.626},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.59099999999999997,"15min":0.60399999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.67900000000000005,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.51500000000000001}]},{"osd":5,"last update":"Mon Feb 5 06:34:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.60099999999999998,"15min":0.60699999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.73999999999999999,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.61699999999999999,"15min":0.627},"min":{"1min":0.44,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.748,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.73799999999999999}]},{"osd":6,"last update":"Mon Feb 5 06:33:52 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.755}]},{"osd":7,"last update":"Mon Feb 5 06:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.72599999999999998}]}]},{"osd":4,"up_from":31,"seq":133143986211,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27868,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717284,"statfs":{"total":95995035648,"available":95966498816,"internally_reserved":0,"allocated":872448,"data_stored":379529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:34:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.57899999999999996,"15min":0.58899999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.71699999999999997,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.54300000000000004,"15min":0.54100000000000004},"min":{"1min":0.42099999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.70399999999999996,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.56399999999999995}]},{"osd":1,"last update":"Mon Feb 5 06:34:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.55900000000000005,"15min":0.56200000000000006},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.69599999999999995,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.58999999999999997,"15min":0.59999999999999998},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.68600000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.746}]},{"osd":2,"last update":"Mon Feb 5 06:34:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.76900000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58099999999999996,"15min":0.58699999999999997},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.80600000000000005,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.80300000000000005}]},{"osd":3,"last update":"Mon Feb 5 06:34:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59799999999999998,"15min":0.60099999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.375,"5min":1.375,"15min":1.375},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.58099999999999996,"15min":0.58699999999999997},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.746,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.75700000000000001}]},{"osd":5,"last update":"Mon Feb 5 06:34:28 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.40999999999999998,"15min":0.40200000000000002},"min":{"1min":0.27500000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.442,"15min":0.439},"min":{"1min":0.317,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.70599999999999996}]},{"osd":6,"last update":"Mon Feb 5 06:33:51 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.68500000000000005}]},{"osd":7,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.628},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.64800000000000002}]}]},{"osd":5,"up_from":40,"seq":171798691870,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":868352,"data_stored":380096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.53100000000000003,"15min":0.53400000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.45,"5min":0.72999999999999998,"15min":0.60999999999999999},"min":{"1min":0.35399999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":17.509,"5min":17.509,"15min":17.509},"last":0.65400000000000003}]},{"osd":1,"last update":"Mon Feb 5 06:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.56100000000000005,"15min":0.56200000000000006},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.752,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.54100000000000004,"15min":0.54400000000000004},"min":{"1min":0.37,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.751}]},{"osd":2,"last update":"Mon Feb 5 06:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54800000000000004,"15min":0.55000000000000004},"min":{"1min":0.41499999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.53700000000000003,"15min":0.53600000000000003},"min":{"1min":0.40300000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.70299999999999996,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.75900000000000001}]},{"osd":3,"last update":"Mon Feb 5 06:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.55300000000000005,"15min":0.55900000000000005},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.66900000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.57699999999999996,"15min":0.58099999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.72599999999999998,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.72599999999999998}]},{"osd":4,"last update":"Mon Feb 5 06:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.39500000000000002,"15min":0.40100000000000002},"min":{"1min":0.29799999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.45300000000000001,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.32300000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.53600000000000003,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.66900000000000004}]},{"osd":6,"last update":"Mon Feb 5 06:33:50 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.70299999999999996}]},{"osd":7,"last update":"Mon Feb 5 06:34:16 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.78400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T06:34:55.686 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-05T06:34:55.686 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-05T06:34:55.686 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-05T06:34:55.686 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph health --format=json 2024-02-05T06:34:55.933 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:34:56.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:55 smithi167 ceph-mon[32774]: 5.3 scrub starts 2024-02-05T06:34:56.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:55 smithi167 ceph-mon[32774]: 5.3 scrub ok 2024-02-05T06:34:56.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:55 smithi167 ceph-mon[32774]: 6.13 scrub starts 2024-02-05T06:34:56.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:55 smithi167 ceph-mon[32774]: 6.13 scrub ok 2024-02-05T06:34:56.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:55 smithi167 ceph-mon[32774]: from='client.14760 v1:172.21.15.28:0/3670500659' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:34:56.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:55 smithi167 ceph-mon[32774]: 6.12 scrub starts 2024-02-05T06:34:56.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:55 smithi167 ceph-mon[32774]: 6.12 scrub ok 2024-02-05T06:34:56.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:55 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:56.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:55 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:34:56.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:55 smithi167 ceph-mon[32774]: 6.1a scrub starts 2024-02-05T06:34:56.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:55 smithi167 ceph-mon[32774]: 6.1a scrub ok 2024-02-05T06:34:56.262 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:55 smithi028 ceph-mon[29442]: 5.3 scrub starts 2024-02-05T06:34:56.262 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:55 smithi028 ceph-mon[29442]: 5.3 scrub ok 2024-02-05T06:34:56.262 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:55 smithi028 ceph-mon[29442]: 6.13 scrub starts 2024-02-05T06:34:56.262 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:55 smithi028 ceph-mon[29442]: 6.13 scrub ok 2024-02-05T06:34:56.262 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:55 smithi028 ceph-mon[29442]: from='client.14760 v1:172.21.15.28:0/3670500659' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:34:56.262 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:55 smithi028 ceph-mon[29442]: 6.12 scrub starts 2024-02-05T06:34:56.263 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:55 smithi028 ceph-mon[29442]: 6.12 scrub ok 2024-02-05T06:34:56.263 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:55 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:56.263 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:55 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:34:56.263 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:55 smithi028 ceph-mon[29442]: 6.1a scrub starts 2024-02-05T06:34:56.263 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:55 smithi028 ceph-mon[29442]: 6.1a scrub ok 2024-02-05T06:34:56.263 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:55 smithi028 ceph-mon[37984]: 5.3 scrub starts 2024-02-05T06:34:56.263 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:55 smithi028 ceph-mon[37984]: 5.3 scrub ok 2024-02-05T06:34:56.263 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:55 smithi028 ceph-mon[37984]: 6.13 scrub starts 2024-02-05T06:34:56.263 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:55 smithi028 ceph-mon[37984]: 6.13 scrub ok 2024-02-05T06:34:56.263 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:55 smithi028 ceph-mon[37984]: from='client.14760 v1:172.21.15.28:0/3670500659' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:34:56.263 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:55 smithi028 ceph-mon[37984]: 6.12 scrub starts 2024-02-05T06:34:56.264 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:55 smithi028 ceph-mon[37984]: 6.12 scrub ok 2024-02-05T06:34:56.264 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:55 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:34:56.264 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:55 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:34:56.264 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:55 smithi028 ceph-mon[37984]: 6.1a scrub starts 2024-02-05T06:34:56.264 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:55 smithi028 ceph-mon[37984]: 6.1a scrub ok 2024-02-05T06:34:57.066 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:56 smithi028 ceph-mon[37984]: 5.1f scrub starts 2024-02-05T06:34:57.067 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:56 smithi028 ceph-mon[37984]: 5.1f scrub ok 2024-02-05T06:34:57.067 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:56 smithi028 ceph-mon[37984]: 6.1d scrub starts 2024-02-05T06:34:57.067 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:56 smithi028 ceph-mon[37984]: 6.1d scrub ok 2024-02-05T06:34:57.067 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:56 smithi028 ceph-mon[37984]: 4.17 scrub starts 2024-02-05T06:34:57.067 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:56 smithi028 ceph-mon[37984]: 4.17 scrub ok 2024-02-05T06:34:57.067 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:56 smithi028 ceph-mon[37984]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T06:34:57.067 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:56 smithi028 ceph-mon[37984]: 4.1 scrub starts 2024-02-05T06:34:57.067 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:56 smithi028 ceph-mon[37984]: 4.1 scrub ok 2024-02-05T06:34:57.067 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:56 smithi028 ceph-mon[29442]: 5.1f scrub starts 2024-02-05T06:34:57.067 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:56 smithi028 ceph-mon[29442]: 5.1f scrub ok 2024-02-05T06:34:57.067 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:56 smithi028 ceph-mon[29442]: 6.1d scrub starts 2024-02-05T06:34:57.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:56 smithi028 ceph-mon[29442]: 6.1d scrub ok 2024-02-05T06:34:57.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:56 smithi028 ceph-mon[29442]: 4.17 scrub starts 2024-02-05T06:34:57.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:56 smithi028 ceph-mon[29442]: 4.17 scrub ok 2024-02-05T06:34:57.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:56 smithi028 ceph-mon[29442]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T06:34:57.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:56 smithi028 ceph-mon[29442]: 4.1 scrub starts 2024-02-05T06:34:57.068 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:56 smithi028 ceph-mon[29442]: 4.1 scrub ok 2024-02-05T06:34:57.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:56 smithi167 ceph-mon[32774]: 5.1f scrub starts 2024-02-05T06:34:57.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:56 smithi167 ceph-mon[32774]: 5.1f scrub ok 2024-02-05T06:34:57.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:56 smithi167 ceph-mon[32774]: 6.1d scrub starts 2024-02-05T06:34:57.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:56 smithi167 ceph-mon[32774]: 6.1d scrub ok 2024-02-05T06:34:57.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:56 smithi167 ceph-mon[32774]: 4.17 scrub starts 2024-02-05T06:34:57.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:56 smithi167 ceph-mon[32774]: 4.17 scrub ok 2024-02-05T06:34:57.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:56 smithi167 ceph-mon[32774]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T06:34:57.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:56 smithi167 ceph-mon[32774]: 4.1 scrub starts 2024-02-05T06:34:57.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:56 smithi167 ceph-mon[32774]: 4.1 scrub ok 2024-02-05T06:34:57.818 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:34:57.818 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-05T06:34:58.147 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:57 smithi028 ceph-mon[29442]: 5.0 scrub starts 2024-02-05T06:34:58.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:57 smithi028 ceph-mon[29442]: 5.0 scrub ok 2024-02-05T06:34:58.148 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:57 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3842728928' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-05T06:34:58.148 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:57 smithi028 ceph-mon[37984]: 5.0 scrub starts 2024-02-05T06:34:58.148 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:57 smithi028 ceph-mon[37984]: 5.0 scrub ok 2024-02-05T06:34:58.148 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:57 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3842728928' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-05T06:34:58.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:57 smithi167 ceph-mon[32774]: 5.0 scrub starts 2024-02-05T06:34:58.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:57 smithi167 ceph-mon[32774]: 5.0 scrub ok 2024-02-05T06:34:58.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:57 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3842728928' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-05T06:34:58.546 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-05T06:34:58.546 INFO:tasks.cephadm:Setup complete, yielding 2024-02-05T06:34:58.546 INFO:teuthology.run_tasks:Running task thrashosds... 2024-02-05T06:34:58.558 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-02-05T06:34:58.558 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-guits-testing-3-2024-02-04-2254'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'mon_bind_msgr2': False, 'sha1': '164d3be4a527be775f4d3c343ea591122f1b8921'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '164d3be4a527be775f4d3c343ea591122f1b8921'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-guits-testing-3-2024-02-04-2254', 'sha1': '164d3be4a527be775f4d3c343ea591122f1b8921'}} 2024-02-05T06:34:58.558 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-02-05T06:34:58.558 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-02-05T06:34:58.807 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:34:59.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:58 smithi028 ceph-mon[29442]: 3.a scrub starts 2024-02-05T06:34:59.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:58 smithi028 ceph-mon[29442]: 3.a scrub ok 2024-02-05T06:34:59.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:34:58 smithi028 ceph-mon[29442]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:34:59.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:58 smithi028 ceph-mon[37984]: 3.a scrub starts 2024-02-05T06:34:59.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:58 smithi028 ceph-mon[37984]: 3.a scrub ok 2024-02-05T06:34:59.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:34:58 smithi028 ceph-mon[37984]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:34:59.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:58 smithi167 ceph-mon[32774]: 3.a scrub starts 2024-02-05T06:34:59.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:58 smithi167 ceph-mon[32774]: 3.a scrub ok 2024-02-05T06:34:59.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:34:58 smithi167 ceph-mon[32774]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:35:00.897 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:35:00 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:35:00] ENGINE Bus STOPPING 2024-02-05T06:35:00.898 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:35:00 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:35:00] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-05T06:35:00.898 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:35:00 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:35:00] ENGINE Bus STOPPED 2024-02-05T06:35:00.898 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:35:00 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:35:00] ENGINE Bus STARTING 2024-02-05T06:35:00.898 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:35:00 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:35:00] ENGINE Serving on http://:::9283 2024-02-05T06:35:00.898 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:35:00 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:35:00] ENGINE Bus STARTED 2024-02-05T06:35:01.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:00 smithi028 ceph-mon[29442]: 3.d scrub starts 2024-02-05T06:35:01.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:00 smithi028 ceph-mon[29442]: 3.d scrub ok 2024-02-05T06:35:01.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:00 smithi028 ceph-mon[29442]: 4.1d scrub starts 2024-02-05T06:35:01.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:00 smithi028 ceph-mon[29442]: 4.1d scrub ok 2024-02-05T06:35:01.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:00 smithi028 ceph-mon[29442]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:35:01.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:00 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1281856999' entity='client.admin' 2024-02-05T06:35:01.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:00 smithi028 ceph-mon[29442]: 4.9 scrub starts 2024-02-05T06:35:01.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:00 smithi028 ceph-mon[29442]: 4.9 scrub ok 2024-02-05T06:35:01.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:00 smithi028 ceph-mon[37984]: 3.d scrub starts 2024-02-05T06:35:01.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:00 smithi028 ceph-mon[37984]: 3.d scrub ok 2024-02-05T06:35:01.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:00 smithi028 ceph-mon[37984]: 4.1d scrub starts 2024-02-05T06:35:01.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:00 smithi028 ceph-mon[37984]: 4.1d scrub ok 2024-02-05T06:35:01.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:00 smithi028 ceph-mon[37984]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:35:01.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:00 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1281856999' entity='client.admin' 2024-02-05T06:35:01.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:00 smithi028 ceph-mon[37984]: 4.9 scrub starts 2024-02-05T06:35:01.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:00 smithi028 ceph-mon[37984]: 4.9 scrub ok 2024-02-05T06:35:01.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:00 smithi167 ceph-mon[32774]: 3.d scrub starts 2024-02-05T06:35:01.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:00 smithi167 ceph-mon[32774]: 3.d scrub ok 2024-02-05T06:35:01.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:00 smithi167 ceph-mon[32774]: 4.1d scrub starts 2024-02-05T06:35:01.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:00 smithi167 ceph-mon[32774]: 4.1d scrub ok 2024-02-05T06:35:01.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:00 smithi167 ceph-mon[32774]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:35:01.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:00 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1281856999' entity='client.admin' 2024-02-05T06:35:01.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:00 smithi167 ceph-mon[32774]: 4.9 scrub starts 2024-02-05T06:35:01.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:00 smithi167 ceph-mon[32774]: 4.9 scrub ok 2024-02-05T06:35:01.387 INFO:tasks.thrashosds:Beginning thrashosds... 2024-02-05T06:35:01.387 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-05T06:35:01.387 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T06:35:01.638 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:35:01.694 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:35:01 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a[72676]: ts=2024-02-05T06:35:01.396Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.001908123s 2024-02-05T06:35:02.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:01 smithi028 ceph-mon[29442]: 6.15 scrub starts 2024-02-05T06:35:02.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:01 smithi028 ceph-mon[29442]: 6.15 scrub ok 2024-02-05T06:35:02.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:01 smithi028 ceph-mon[29442]: 4.18 scrub starts 2024-02-05T06:35:02.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:01 smithi028 ceph-mon[29442]: 4.18 scrub ok 2024-02-05T06:35:02.040 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:01 smithi028 ceph-mon[37984]: 6.15 scrub starts 2024-02-05T06:35:02.040 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:01 smithi028 ceph-mon[37984]: 6.15 scrub ok 2024-02-05T06:35:02.040 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:01 smithi028 ceph-mon[37984]: 4.18 scrub starts 2024-02-05T06:35:02.040 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:01 smithi028 ceph-mon[37984]: 4.18 scrub ok 2024-02-05T06:35:02.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:01 smithi167 ceph-mon[32774]: 6.15 scrub starts 2024-02-05T06:35:02.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:01 smithi167 ceph-mon[32774]: 6.15 scrub ok 2024-02-05T06:35:02.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:01 smithi167 ceph-mon[32774]: 4.18 scrub starts 2024-02-05T06:35:02.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:01 smithi167 ceph-mon[32774]: 4.18 scrub ok 2024-02-05T06:35:02.661 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:35:02 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:35:02.678 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:35:02 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:35:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:35:02.929 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:02 smithi167 ceph-mon[32774]: 6.a scrub starts 2024-02-05T06:35:02.929 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:02 smithi167 ceph-mon[32774]: 6.a scrub ok 2024-02-05T06:35:02.929 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:02 smithi167 ceph-mon[32774]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:35:02.929 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:02 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:35:02.929 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:02 smithi167 ceph-mon[32774]: 4.13 deep-scrub starts 2024-02-05T06:35:02.929 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:02 smithi167 ceph-mon[32774]: 4.13 deep-scrub ok 2024-02-05T06:35:02.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:02 smithi028 ceph-mon[37984]: 6.a scrub starts 2024-02-05T06:35:02.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:02 smithi028 ceph-mon[37984]: 6.a scrub ok 2024-02-05T06:35:02.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:02 smithi028 ceph-mon[37984]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:35:02.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:02 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:35:02.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:02 smithi028 ceph-mon[37984]: 4.13 deep-scrub starts 2024-02-05T06:35:02.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:02 smithi028 ceph-mon[37984]: 4.13 deep-scrub ok 2024-02-05T06:35:02.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:02 smithi028 ceph-mon[29442]: 6.a scrub starts 2024-02-05T06:35:02.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:02 smithi028 ceph-mon[29442]: 6.a scrub ok 2024-02-05T06:35:02.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:02 smithi028 ceph-mon[29442]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:35:02.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:02 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:35:02.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:02 smithi028 ceph-mon[29442]: 4.13 deep-scrub starts 2024-02-05T06:35:02.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:02 smithi028 ceph-mon[29442]: 4.13 deep-scrub ok 2024-02-05T06:35:03.462 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:35:03.465 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-05T06:35:03.976 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:03 smithi167 ceph-mon[32774]: 6.5 scrub starts 2024-02-05T06:35:03.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:03 smithi167 ceph-mon[32774]: 6.5 scrub ok 2024-02-05T06:35:03.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:03 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:35:03.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:03 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:35:03.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:03 smithi167 ceph-mon[32774]: 4.5 scrub starts 2024-02-05T06:35:03.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:03 smithi167 ceph-mon[32774]: 4.5 scrub ok 2024-02-05T06:35:03.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:03 smithi167 ceph-mon[32774]: 4.f scrub starts 2024-02-05T06:35:03.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:03 smithi167 ceph-mon[32774]: 4.f scrub ok 2024-02-05T06:35:03.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:03 smithi167 ceph-mon[32774]: osdmap e93: 8 total, 8 up, 8 in 2024-02-05T06:35:03.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:03 smithi167 ceph-mon[32774]: 5.15 scrub starts 2024-02-05T06:35:03.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:03 smithi167 ceph-mon[32774]: 5.15 scrub ok 2024-02-05T06:35:03.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:03 smithi167 ceph-mon[32774]: 5.13 scrub starts 2024-02-05T06:35:03.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:03 smithi167 ceph-mon[32774]: 5.13 scrub ok 2024-02-05T06:35:03.978 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:03 smithi167 ceph-mon[32774]: from='client.14778 v1:172.21.15.28:0/4291140485' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:35:04.112 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:03 smithi028 ceph-mon[37984]: 6.5 scrub starts 2024-02-05T06:35:04.112 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:03 smithi028 ceph-mon[37984]: 6.5 scrub ok 2024-02-05T06:35:04.112 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:03 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:35:04.112 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:03 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:35:04.112 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:03 smithi028 ceph-mon[37984]: 4.5 scrub starts 2024-02-05T06:35:04.113 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:03 smithi028 ceph-mon[37984]: 4.5 scrub ok 2024-02-05T06:35:04.113 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:03 smithi028 ceph-mon[37984]: 4.f scrub starts 2024-02-05T06:35:04.113 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:03 smithi028 ceph-mon[37984]: 4.f scrub ok 2024-02-05T06:35:04.113 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:03 smithi028 ceph-mon[37984]: osdmap e93: 8 total, 8 up, 8 in 2024-02-05T06:35:04.113 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:03 smithi028 ceph-mon[37984]: 5.15 scrub starts 2024-02-05T06:35:04.113 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:03 smithi028 ceph-mon[37984]: 5.15 scrub ok 2024-02-05T06:35:04.113 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:03 smithi028 ceph-mon[37984]: 5.13 scrub starts 2024-02-05T06:35:04.114 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:03 smithi028 ceph-mon[37984]: 5.13 scrub ok 2024-02-05T06:35:04.114 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:03 smithi028 ceph-mon[37984]: from='client.14778 v1:172.21.15.28:0/4291140485' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:35:04.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:03 smithi028 ceph-mon[29442]: 6.5 scrub starts 2024-02-05T06:35:04.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:03 smithi028 ceph-mon[29442]: 6.5 scrub ok 2024-02-05T06:35:04.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:03 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:35:04.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:03 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:35:04.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:03 smithi028 ceph-mon[29442]: 4.5 scrub starts 2024-02-05T06:35:04.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:03 smithi028 ceph-mon[29442]: 4.5 scrub ok 2024-02-05T06:35:04.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:03 smithi028 ceph-mon[29442]: 4.f scrub starts 2024-02-05T06:35:04.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:03 smithi028 ceph-mon[29442]: 4.f scrub ok 2024-02-05T06:35:04.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:03 smithi028 ceph-mon[29442]: osdmap e93: 8 total, 8 up, 8 in 2024-02-05T06:35:04.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:03 smithi028 ceph-mon[29442]: 5.15 scrub starts 2024-02-05T06:35:04.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:03 smithi028 ceph-mon[29442]: 5.15 scrub ok 2024-02-05T06:35:04.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:03 smithi028 ceph-mon[29442]: 5.13 scrub starts 2024-02-05T06:35:04.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:03 smithi028 ceph-mon[29442]: 5.13 scrub ok 2024-02-05T06:35:04.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:03 smithi028 ceph-mon[29442]: from='client.14778 v1:172.21.15.28:0/4291140485' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:35:04.143 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-02-05T06:35:02.902679+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1018,"num_read_kb":864,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1771792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":630,"ondisk_log_size":630,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":241276,"kb_used_data":8508,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749719940,"statfs":{"total":767960285184,"available":767713218560,"internally_reserved":0,"allocated":8712192,"data_stored":4808732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.751105"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":58,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:55.600812+0000","last_change":"2024-02-05T06:34:55.600835+0000","last_active":"2024-02-05T06:34:55.600812+0000","last_peered":"2024-02-05T06:34:55.600812+0000","last_clean":"2024-02-05T06:34:55.600812+0000","last_became_active":"2024-02-05T06:33:35.594467+0000","last_became_peered":"2024-02-05T06:33:35.594467+0000","last_unstale":"2024-02-05T06:34:55.600812+0000","last_undegraded":"2024-02-05T06:34:55.600812+0000","last_fullsized":"2024-02-05T06:34:55.600812+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:55.600744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:55.600744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:09.045705+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":54,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890147+0000","last_change":"2024-02-05T06:33:33.586212+0000","last_active":"2024-02-05T06:34:44.890147+0000","last_peered":"2024-02-05T06:34:44.890147+0000","last_clean":"2024-02-05T06:34:44.890147+0000","last_became_active":"2024-02-05T06:33:33.585848+0000","last_became_peered":"2024-02-05T06:33:33.585848+0000","last_unstale":"2024-02-05T06:34:44.890147+0000","last_undegraded":"2024-02-05T06:34:44.890147+0000","last_fullsized":"2024-02-05T06:34:44.890147+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:51.677322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890647+0000","last_change":"2024-02-05T06:33:29.549864+0000","last_active":"2024-02-05T06:34:44.890647+0000","last_peered":"2024-02-05T06:34:44.890647+0000","last_clean":"2024-02-05T06:34:44.890647+0000","last_became_active":"2024-02-05T06:33:29.549592+0000","last_became_peered":"2024-02-05T06:33:29.549592+0000","last_unstale":"2024-02-05T06:34:44.890647+0000","last_undegraded":"2024-02-05T06:34:44.890647+0000","last_fullsized":"2024-02-05T06:34:44.890647+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:52.202922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":74,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.892665+0000","last_change":"2024-02-05T06:33:31.566905+0000","last_active":"2024-02-05T06:34:44.892665+0000","last_peered":"2024-02-05T06:34:44.892665+0000","last_clean":"2024-02-05T06:34:44.892665+0000","last_became_active":"2024-02-05T06:33:31.566571+0000","last_became_peered":"2024-02-05T06:33:31.566571+0000","last_unstale":"2024-02-05T06:34:44.892665+0000","last_undegraded":"2024-02-05T06:34:44.892665+0000","last_fullsized":"2024-02-05T06:34:44.892665+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:48.692724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096953+0000","last_change":"2024-02-05T06:33:35.668346+0000","last_active":"2024-02-05T06:34:45.096953+0000","last_peered":"2024-02-05T06:34:45.096953+0000","last_clean":"2024-02-05T06:34:45.096953+0000","last_became_active":"2024-02-05T06:33:35.667778+0000","last_became_peered":"2024-02-05T06:33:35.667778+0000","last_unstale":"2024-02-05T06:34:45.096953+0000","last_undegraded":"2024-02-05T06:34:45.096953+0000","last_fullsized":"2024-02-05T06:34:45.096953+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:59.948542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":54,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.892667+0000","last_change":"2024-02-05T06:33:33.583029+0000","last_active":"2024-02-05T06:34:44.892667+0000","last_peered":"2024-02-05T06:34:44.892667+0000","last_clean":"2024-02-05T06:34:44.892667+0000","last_became_active":"2024-02-05T06:33:33.582731+0000","last_became_peered":"2024-02-05T06:33:33.582731+0000","last_unstale":"2024-02-05T06:34:44.892667+0000","last_undegraded":"2024-02-05T06:34:44.892667+0000","last_fullsized":"2024-02-05T06:34:44.892667+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:56.016226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":71,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097695+0000","last_change":"2024-02-05T06:34:34.821722+0000","last_active":"2024-02-05T06:34:45.097695+0000","last_peered":"2024-02-05T06:34:45.097695+0000","last_clean":"2024-02-05T06:34:45.097695+0000","last_became_active":"2024-02-05T06:33:29.553864+0000","last_became_peered":"2024-02-05T06:33:29.553864+0000","last_unstale":"2024-02-05T06:34:45.097695+0000","last_undegraded":"2024-02-05T06:34:45.097695+0000","last_fullsized":"2024-02-05T06:34:45.097695+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:34.821603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:34.821603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:39.571760+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"88'22","reported_seq":99,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097275+0000","last_change":"2024-02-05T06:34:35.832461+0000","last_active":"2024-02-05T06:34:45.097275+0000","last_peered":"2024-02-05T06:34:45.097275+0000","last_clean":"2024-02-05T06:34:45.097275+0000","last_became_active":"2024-02-05T06:33:31.572544+0000","last_became_peered":"2024-02-05T06:33:31.572544+0000","last_unstale":"2024-02-05T06:34:45.097275+0000","last_undegraded":"2024-02-05T06:34:45.097275+0000","last_fullsized":"2024-02-05T06:34:45.097275+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'22","last_scrub_stamp":"2024-02-05T06:34:35.832368+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:34:35.832368+0000","last_clean_scrub_stamp":"2024-02-05T06:34:35.832368+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:43.028037+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.889644+0000","last_change":"2024-02-05T06:33:35.595744+0000","last_active":"2024-02-05T06:34:44.889644+0000","last_peered":"2024-02-05T06:34:44.889644+0000","last_clean":"2024-02-05T06:34:44.889644+0000","last_became_active":"2024-02-05T06:33:35.594860+0000","last_became_peered":"2024-02-05T06:33:35.594860+0000","last_unstale":"2024-02-05T06:34:44.889644+0000","last_undegraded":"2024-02-05T06:34:44.889644+0000","last_fullsized":"2024-02-05T06:34:44.889644+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095518+0000","last_change":"2024-02-05T06:34:34.187757+0000","last_active":"2024-02-05T06:34:45.095518+0000","last_peered":"2024-02-05T06:34:45.095518+0000","last_clean":"2024-02-05T06:34:45.095518+0000","last_became_active":"2024-02-05T06:33:33.585459+0000","last_became_peered":"2024-02-05T06:33:33.585459+0000","last_unstale":"2024-02-05T06:34:45.095518+0000","last_undegraded":"2024-02-05T06:34:45.095518+0000","last_fullsized":"2024-02-05T06:34:45.095518+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:34.187643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:34.187643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:35.727918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095719+0000","last_change":"2024-02-05T06:34:44.081424+0000","last_active":"2024-02-05T06:34:45.095719+0000","last_peered":"2024-02-05T06:34:45.095719+0000","last_clean":"2024-02-05T06:34:45.095719+0000","last_became_active":"2024-02-05T06:33:29.553251+0000","last_became_peered":"2024-02-05T06:33:29.553251+0000","last_unstale":"2024-02-05T06:34:45.095719+0000","last_undegraded":"2024-02-05T06:34:45.095719+0000","last_fullsized":"2024-02-05T06:34:45.095719+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:44.081318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:44.081318+0000","last_clean_scrub_stamp":"2024-02-05T06:34:44.081318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:49.865346+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"92'13","reported_seq":86,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:46.542965+0000","last_change":"2024-02-05T06:34:46.543006+0000","last_active":"2024-02-05T06:34:46.542965+0000","last_peered":"2024-02-05T06:34:46.542965+0000","last_clean":"2024-02-05T06:34:46.542965+0000","last_became_active":"2024-02-05T06:33:31.570995+0000","last_became_peered":"2024-02-05T06:33:31.570995+0000","last_unstale":"2024-02-05T06:34:46.542965+0000","last_undegraded":"2024-02-05T06:34:46.542965+0000","last_fullsized":"2024-02-05T06:34:46.542965+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'13","last_scrub_stamp":"2024-02-05T06:34:46.542893+0000","last_deep_scrub":"92'13","last_deep_scrub_stamp":"2024-02-05T06:34:46.542893+0000","last_clean_scrub_stamp":"2024-02-05T06:34:46.542893+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:00.864899+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095188+0000","last_change":"2024-02-05T06:33:35.600855+0000","last_active":"2024-02-05T06:34:45.095188+0000","last_peered":"2024-02-05T06:34:45.095188+0000","last_clean":"2024-02-05T06:34:45.095188+0000","last_became_active":"2024-02-05T06:33:35.600399+0000","last_became_peered":"2024-02-05T06:33:35.600399+0000","last_unstale":"2024-02-05T06:34:45.095188+0000","last_undegraded":"2024-02-05T06:34:45.095188+0000","last_fullsized":"2024-02-05T06:34:45.095188+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:59.096165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:53.073787+0000","last_change":"2024-02-05T06:34:53.073825+0000","last_active":"2024-02-05T06:34:53.073787+0000","last_peered":"2024-02-05T06:34:53.073787+0000","last_clean":"2024-02-05T06:34:53.073787+0000","last_became_active":"2024-02-05T06:33:33.584151+0000","last_became_peered":"2024-02-05T06:33:33.584151+0000","last_unstale":"2024-02-05T06:34:53.073787+0000","last_undegraded":"2024-02-05T06:34:53.073787+0000","last_fullsized":"2024-02-05T06:34:53.073787+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:53.073710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:53.073710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:08.940750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:50.055329+0000","last_change":"2024-02-05T06:34:50.055371+0000","last_active":"2024-02-05T06:34:50.055329+0000","last_peered":"2024-02-05T06:34:50.055329+0000","last_clean":"2024-02-05T06:34:50.055329+0000","last_became_active":"2024-02-05T06:33:29.548931+0000","last_became_peered":"2024-02-05T06:33:29.548931+0000","last_unstale":"2024-02-05T06:34:50.055329+0000","last_undegraded":"2024-02-05T06:34:50.055329+0000","last_fullsized":"2024-02-05T06:34:50.055329+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:50.055245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:50.055245+0000","last_clean_scrub_stamp":"2024-02-05T06:34:50.055245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:54.332822+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":76,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:53.638800+0000","last_change":"2024-02-05T06:34:53.638858+0000","last_active":"2024-02-05T06:34:53.638800+0000","last_peered":"2024-02-05T06:34:53.638800+0000","last_clean":"2024-02-05T06:34:53.638800+0000","last_became_active":"2024-02-05T06:33:31.569654+0000","last_became_peered":"2024-02-05T06:33:31.569654+0000","last_unstale":"2024-02-05T06:34:53.638800+0000","last_undegraded":"2024-02-05T06:34:53.638800+0000","last_fullsized":"2024-02-05T06:34:53.638800+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-02-05T06:34:53.638704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:53.638704+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:12.883735+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":58,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.573515+0000","last_change":"2024-02-05T06:34:45.573557+0000","last_active":"2024-02-05T06:34:45.573515+0000","last_peered":"2024-02-05T06:34:45.573515+0000","last_clean":"2024-02-05T06:34:45.573515+0000","last_became_active":"2024-02-05T06:33:35.664772+0000","last_became_peered":"2024-02-05T06:33:35.664772+0000","last_unstale":"2024-02-05T06:34:45.573515+0000","last_undegraded":"2024-02-05T06:34:45.573515+0000","last_fullsized":"2024-02-05T06:34:45.573515+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:45.573432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:45.573432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:10.154762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":54,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890929+0000","last_change":"2024-02-05T06:33:33.588208+0000","last_active":"2024-02-05T06:34:44.890929+0000","last_peered":"2024-02-05T06:34:44.890929+0000","last_clean":"2024-02-05T06:34:44.890929+0000","last_became_active":"2024-02-05T06:33:33.587920+0000","last_became_peered":"2024-02-05T06:33:33.587920+0000","last_unstale":"2024-02-05T06:34:44.890929+0000","last_undegraded":"2024-02-05T06:34:44.890929+0000","last_fullsized":"2024-02-05T06:34:44.890929+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:54.461651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"66'1","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891771+0000","last_change":"2024-02-05T06:33:29.552104+0000","last_active":"2024-02-05T06:34:44.891771+0000","last_peered":"2024-02-05T06:34:44.891771+0000","last_clean":"2024-02-05T06:34:44.891771+0000","last_became_active":"2024-02-05T06:33:29.551820+0000","last_became_peered":"2024-02-05T06:33:29.551820+0000","last_unstale":"2024-02-05T06:34:44.891771+0000","last_undegraded":"2024-02-05T06:34:44.891771+0000","last_fullsized":"2024-02-05T06:34:44.891771+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"88'22","reported_seq":97,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.297835+0000","last_change":"2024-02-05T06:34:35.234726+0000","last_active":"2024-02-05T06:34:45.297835+0000","last_peered":"2024-02-05T06:34:45.297835+0000","last_clean":"2024-02-05T06:34:45.297835+0000","last_became_active":"2024-02-05T06:33:31.569763+0000","last_became_peered":"2024-02-05T06:33:31.569763+0000","last_unstale":"2024-02-05T06:34:45.297835+0000","last_undegraded":"2024-02-05T06:34:45.297835+0000","last_fullsized":"2024-02-05T06:34:45.297835+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'22","last_scrub_stamp":"2024-02-05T06:34:35.234636+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:34:35.234636+0000","last_clean_scrub_stamp":"2024-02-05T06:34:35.234636+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:53.234677+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097830+0000","last_change":"2024-02-05T06:33:35.665217+0000","last_active":"2024-02-05T06:34:45.097830+0000","last_peered":"2024-02-05T06:34:45.097830+0000","last_clean":"2024-02-05T06:34:45.097830+0000","last_became_active":"2024-02-05T06:33:35.664852+0000","last_became_peered":"2024-02-05T06:33:35.664852+0000","last_unstale":"2024-02-05T06:34:45.097830+0000","last_undegraded":"2024-02-05T06:34:45.097830+0000","last_fullsized":"2024-02-05T06:34:45.097830+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:47.798779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":54,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.893413+0000","last_change":"2024-02-05T06:33:33.587913+0000","last_active":"2024-02-05T06:34:44.893413+0000","last_peered":"2024-02-05T06:34:44.893413+0000","last_clean":"2024-02-05T06:34:44.893413+0000","last_became_active":"2024-02-05T06:33:33.587595+0000","last_became_peered":"2024-02-05T06:33:33.587595+0000","last_unstale":"2024-02-05T06:34:44.893413+0000","last_undegraded":"2024-02-05T06:34:44.893413+0000","last_fullsized":"2024-02-05T06:34:44.893413+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:53.273088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":70,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.892937+0000","last_change":"2024-02-05T06:34:38.623297+0000","last_active":"2024-02-05T06:34:44.892937+0000","last_peered":"2024-02-05T06:34:44.892937+0000","last_clean":"2024-02-05T06:34:44.892937+0000","last_became_active":"2024-02-05T06:33:29.551340+0000","last_became_peered":"2024-02-05T06:33:29.551340+0000","last_unstale":"2024-02-05T06:34:44.892937+0000","last_undegraded":"2024-02-05T06:34:44.892937+0000","last_fullsized":"2024-02-05T06:34:44.892937+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:38.623203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:38.623203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:49.053281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":108,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:59.999796+0000","last_change":"2024-02-05T06:34:59.999830+0000","last_active":"2024-02-05T06:34:59.999796+0000","last_peered":"2024-02-05T06:34:59.999796+0000","last_clean":"2024-02-05T06:34:59.999796+0000","last_became_active":"2024-02-05T06:33:55.885324+0000","last_became_peered":"2024-02-05T06:33:55.885324+0000","last_unstale":"2024-02-05T06:34:59.999796+0000","last_undegraded":"2024-02-05T06:34:59.999796+0000","last_fullsized":"2024-02-05T06:34:59.999796+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:34:59.999726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:59.999726+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:23.143025+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"73'1","reported_seq":52,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.923186+0000","last_change":"2024-02-05T06:33:35.598794+0000","last_active":"2024-02-05T06:34:45.923186+0000","last_peered":"2024-02-05T06:34:45.923186+0000","last_clean":"2024-02-05T06:34:45.923186+0000","last_became_active":"2024-02-05T06:33:35.598501+0000","last_became_peered":"2024-02-05T06:33:35.598501+0000","last_unstale":"2024-02-05T06:34:45.923186+0000","last_undegraded":"2024-02-05T06:34:45.923186+0000","last_fullsized":"2024-02-05T06:34:45.923186+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:02.621665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"73'8","reported_seq":95,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:55.312940+0000","last_change":"2024-02-05T06:34:55.312940+0000","last_active":"2024-02-05T06:34:55.312940+0000","last_peered":"2024-02-05T06:34:55.312940+0000","last_clean":"2024-02-05T06:34:55.312940+0000","last_became_active":"2024-02-05T06:33:33.586389+0000","last_became_peered":"2024-02-05T06:33:33.586389+0000","last_unstale":"2024-02-05T06:34:55.312940+0000","last_undegraded":"2024-02-05T06:34:55.312940+0000","last_fullsized":"2024-02-05T06:34:55.312940+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:34:55.312853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:55.312853+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:18.905872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":71,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:48.387541+0000","last_change":"2024-02-05T06:34:48.387568+0000","last_active":"2024-02-05T06:34:48.387541+0000","last_peered":"2024-02-05T06:34:48.387541+0000","last_clean":"2024-02-05T06:34:48.387541+0000","last_became_active":"2024-02-05T06:33:29.546201+0000","last_became_peered":"2024-02-05T06:33:29.546201+0000","last_unstale":"2024-02-05T06:34:48.387541+0000","last_undegraded":"2024-02-05T06:34:48.387541+0000","last_fullsized":"2024-02-05T06:34:48.387541+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:34:48.387474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:48.387474+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:58.746172+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"92'15","reported_seq":87,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:50.718830+0000","last_change":"2024-02-05T06:34:50.718858+0000","last_active":"2024-02-05T06:34:50.718830+0000","last_peered":"2024-02-05T06:34:50.718830+0000","last_clean":"2024-02-05T06:34:50.718830+0000","last_became_active":"2024-02-05T06:33:31.666481+0000","last_became_peered":"2024-02-05T06:33:31.666481+0000","last_unstale":"2024-02-05T06:34:50.718830+0000","last_undegraded":"2024-02-05T06:34:50.718830+0000","last_fullsized":"2024-02-05T06:34:50.718830+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'15","last_scrub_stamp":"2024-02-05T06:34:50.718768+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-02-05T06:34:50.718768+0000","last_clean_scrub_stamp":"2024-02-05T06:34:50.718768+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:51.271274+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":58,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:55.554642+0000","last_change":"2024-02-05T06:34:55.554666+0000","last_active":"2024-02-05T06:34:55.554642+0000","last_peered":"2024-02-05T06:34:55.554642+0000","last_clean":"2024-02-05T06:34:55.554642+0000","last_became_active":"2024-02-05T06:33:35.593925+0000","last_became_peered":"2024-02-05T06:33:35.593925+0000","last_unstale":"2024-02-05T06:34:55.554642+0000","last_undegraded":"2024-02-05T06:34:55.554642+0000","last_fullsized":"2024-02-05T06:34:55.554642+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:55.554581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:55.554581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:56.554973+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":55,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891396+0000","last_change":"2024-02-05T06:33:33.584707+0000","last_active":"2024-02-05T06:34:44.891396+0000","last_peered":"2024-02-05T06:34:44.891396+0000","last_clean":"2024-02-05T06:34:44.891396+0000","last_became_active":"2024-02-05T06:33:33.584390+0000","last_became_peered":"2024-02-05T06:33:33.584390+0000","last_unstale":"2024-02-05T06:34:44.891396+0000","last_undegraded":"2024-02-05T06:34:44.891396+0000","last_fullsized":"2024-02-05T06:34:44.891396+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096561+0000","last_change":"2024-02-05T06:33:29.550640+0000","last_active":"2024-02-05T06:34:45.096561+0000","last_peered":"2024-02-05T06:34:45.096561+0000","last_clean":"2024-02-05T06:34:45.096561+0000","last_became_active":"2024-02-05T06:33:29.550347+0000","last_became_peered":"2024-02-05T06:33:29.550347+0000","last_unstale":"2024-02-05T06:34:45.096561+0000","last_undegraded":"2024-02-05T06:34:45.096561+0000","last_fullsized":"2024-02-05T06:34:45.096561+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:47.916348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"92'16","reported_seq":90,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.270243+0000","last_change":"2024-02-05T06:34:45.270307+0000","last_active":"2024-02-05T06:34:45.270243+0000","last_peered":"2024-02-05T06:34:45.270243+0000","last_clean":"2024-02-05T06:34:45.270243+0000","last_became_active":"2024-02-05T06:33:31.573088+0000","last_became_peered":"2024-02-05T06:33:31.573088+0000","last_unstale":"2024-02-05T06:34:45.270243+0000","last_undegraded":"2024-02-05T06:34:45.270243+0000","last_fullsized":"2024-02-05T06:34:45.270243+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'16","last_scrub_stamp":"2024-02-05T06:34:45.270135+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:34:45.270135+0000","last_clean_scrub_stamp":"2024-02-05T06:34:45.270135+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:04.519387+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":59,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:55.062081+0000","last_change":"2024-02-05T06:34:55.062101+0000","last_active":"2024-02-05T06:34:55.062081+0000","last_peered":"2024-02-05T06:34:55.062081+0000","last_clean":"2024-02-05T06:34:55.062081+0000","last_became_active":"2024-02-05T06:33:35.597453+0000","last_became_peered":"2024-02-05T06:33:35.597453+0000","last_unstale":"2024-02-05T06:34:55.062081+0000","last_undegraded":"2024-02-05T06:34:55.062081+0000","last_fullsized":"2024-02-05T06:34:55.062081+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:55.062013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:55.062013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:18.704482+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":62,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:51.334645+0000","last_change":"2024-02-05T06:34:51.334669+0000","last_active":"2024-02-05T06:34:51.334645+0000","last_peered":"2024-02-05T06:34:51.334645+0000","last_clean":"2024-02-05T06:34:51.334645+0000","last_became_active":"2024-02-05T06:33:33.585125+0000","last_became_peered":"2024-02-05T06:33:33.585125+0000","last_unstale":"2024-02-05T06:34:51.334645+0000","last_undegraded":"2024-02-05T06:34:51.334645+0000","last_fullsized":"2024-02-05T06:34:51.334645+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:51.334570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:51.334570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:19.957468+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890564+0000","last_change":"2024-02-05T06:33:29.549815+0000","last_active":"2024-02-05T06:34:44.890564+0000","last_peered":"2024-02-05T06:34:44.890564+0000","last_clean":"2024-02-05T06:34:44.890564+0000","last_became_active":"2024-02-05T06:33:29.549204+0000","last_became_peered":"2024-02-05T06:33:29.549204+0000","last_unstale":"2024-02-05T06:34:44.890564+0000","last_undegraded":"2024-02-05T06:34:44.890564+0000","last_fullsized":"2024-02-05T06:34:44.890564+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:54.367234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"73'4","reported_seq":65,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096579+0000","last_change":"2024-02-05T06:33:31.573713+0000","last_active":"2024-02-05T06:34:45.096579+0000","last_peered":"2024-02-05T06:34:45.096579+0000","last_clean":"2024-02-05T06:34:45.096579+0000","last_became_active":"2024-02-05T06:33:31.573410+0000","last_became_peered":"2024-02-05T06:33:31.573410+0000","last_unstale":"2024-02-05T06:34:45.096579+0000","last_undegraded":"2024-02-05T06:34:45.096579+0000","last_fullsized":"2024-02-05T06:34:45.096579+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:58.822517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":59,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:54.902064+0000","last_change":"2024-02-05T06:34:54.902085+0000","last_active":"2024-02-05T06:34:54.902064+0000","last_peered":"2024-02-05T06:34:54.902064+0000","last_clean":"2024-02-05T06:34:54.902064+0000","last_became_active":"2024-02-05T06:33:35.667602+0000","last_became_peered":"2024-02-05T06:33:35.667602+0000","last_unstale":"2024-02-05T06:34:54.902064+0000","last_undegraded":"2024-02-05T06:34:54.902064+0000","last_fullsized":"2024-02-05T06:34:54.902064+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:54.902000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:54.902000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:57.025191+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.922284+0000","last_change":"2024-02-05T06:34:43.035728+0000","last_active":"2024-02-05T06:34:45.922284+0000","last_peered":"2024-02-05T06:34:45.922284+0000","last_clean":"2024-02-05T06:34:45.922284+0000","last_became_active":"2024-02-05T06:33:33.589428+0000","last_became_peered":"2024-02-05T06:33:33.589428+0000","last_unstale":"2024-02-05T06:34:45.922284+0000","last_undegraded":"2024-02-05T06:34:45.922284+0000","last_fullsized":"2024-02-05T06:34:45.922284+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:43.035635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:43.035635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:52.189018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095696+0000","last_change":"2024-02-05T06:33:29.551576+0000","last_active":"2024-02-05T06:34:45.095696+0000","last_peered":"2024-02-05T06:34:45.095696+0000","last_clean":"2024-02-05T06:34:45.095696+0000","last_became_active":"2024-02-05T06:33:29.551278+0000","last_became_peered":"2024-02-05T06:33:29.551278+0000","last_unstale":"2024-02-05T06:34:45.095696+0000","last_undegraded":"2024-02-05T06:34:45.095696+0000","last_fullsized":"2024-02-05T06:34:45.095696+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:58.094774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"73'11","reported_seq":78,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097095+0000","last_change":"2024-02-05T06:33:31.665074+0000","last_active":"2024-02-05T06:34:45.097095+0000","last_peered":"2024-02-05T06:34:45.097095+0000","last_clean":"2024-02-05T06:34:45.097095+0000","last_became_active":"2024-02-05T06:33:31.664743+0000","last_became_peered":"2024-02-05T06:33:31.664743+0000","last_unstale":"2024-02-05T06:34:45.097095+0000","last_undegraded":"2024-02-05T06:34:45.097095+0000","last_fullsized":"2024-02-05T06:34:45.097095+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:56.675510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"73'1","reported_seq":52,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.889401+0000","last_change":"2024-02-05T06:33:35.595450+0000","last_active":"2024-02-05T06:34:44.889401+0000","last_peered":"2024-02-05T06:34:44.889401+0000","last_clean":"2024-02-05T06:34:44.889401+0000","last_became_active":"2024-02-05T06:33:35.595031+0000","last_became_peered":"2024-02-05T06:33:35.595031+0000","last_unstale":"2024-02-05T06:34:44.889401+0000","last_undegraded":"2024-02-05T06:34:44.889401+0000","last_fullsized":"2024-02-05T06:34:44.889401+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":55,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097083+0000","last_change":"2024-02-05T06:33:33.594194+0000","last_active":"2024-02-05T06:34:45.097083+0000","last_peered":"2024-02-05T06:34:45.097083+0000","last_clean":"2024-02-05T06:34:45.097083+0000","last_became_active":"2024-02-05T06:33:33.593901+0000","last_became_peered":"2024-02-05T06:33:33.593901+0000","last_unstale":"2024-02-05T06:34:45.097083+0000","last_undegraded":"2024-02-05T06:34:45.097083+0000","last_fullsized":"2024-02-05T06:34:45.097083+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:46.645657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.923423+0000","last_change":"2024-02-05T06:33:29.559294+0000","last_active":"2024-02-05T06:34:45.923423+0000","last_peered":"2024-02-05T06:34:45.923423+0000","last_clean":"2024-02-05T06:34:45.923423+0000","last_became_active":"2024-02-05T06:33:29.559006+0000","last_became_peered":"2024-02-05T06:33:29.559006+0000","last_unstale":"2024-02-05T06:34:45.923423+0000","last_undegraded":"2024-02-05T06:34:45.923423+0000","last_fullsized":"2024-02-05T06:34:45.923423+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:52.625563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":82,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:52.486138+0000","last_change":"2024-02-05T06:34:52.486190+0000","last_active":"2024-02-05T06:34:52.486138+0000","last_peered":"2024-02-05T06:34:52.486138+0000","last_clean":"2024-02-05T06:34:52.486138+0000","last_became_active":"2024-02-05T06:33:31.568529+0000","last_became_peered":"2024-02-05T06:33:31.568529+0000","last_unstale":"2024-02-05T06:34:52.486138+0000","last_undegraded":"2024-02-05T06:34:52.486138+0000","last_fullsized":"2024-02-05T06:34:52.486138+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:34:52.486064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:52.486064+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:20.775162+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":59,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:48.882631+0000","last_change":"2024-02-05T06:34:48.882657+0000","last_active":"2024-02-05T06:34:48.882631+0000","last_peered":"2024-02-05T06:34:48.882631+0000","last_clean":"2024-02-05T06:34:48.882631+0000","last_became_active":"2024-02-05T06:33:35.597399+0000","last_became_peered":"2024-02-05T06:33:35.597399+0000","last_unstale":"2024-02-05T06:34:48.882631+0000","last_undegraded":"2024-02-05T06:34:48.882631+0000","last_fullsized":"2024-02-05T06:34:48.882631+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:48.882556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:48.882556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:52.368291+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":62,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891472+0000","last_change":"2024-02-05T06:34:40.361349+0000","last_active":"2024-02-05T06:34:44.891472+0000","last_peered":"2024-02-05T06:34:44.891472+0000","last_clean":"2024-02-05T06:34:44.891472+0000","last_became_active":"2024-02-05T06:33:33.588461+0000","last_became_peered":"2024-02-05T06:33:33.588461+0000","last_unstale":"2024-02-05T06:34:44.891472+0000","last_undegraded":"2024-02-05T06:34:44.891472+0000","last_fullsized":"2024-02-05T06:34:44.891472+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:40.361273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:40.361273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:50.089342+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":70,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:48.561101+0000","last_change":"2024-02-05T06:34:48.561146+0000","last_active":"2024-02-05T06:34:48.561101+0000","last_peered":"2024-02-05T06:34:48.561101+0000","last_clean":"2024-02-05T06:34:48.561101+0000","last_became_active":"2024-02-05T06:33:29.552336+0000","last_became_peered":"2024-02-05T06:33:29.552336+0000","last_unstale":"2024-02-05T06:34:48.561101+0000","last_undegraded":"2024-02-05T06:34:48.561101+0000","last_fullsized":"2024-02-05T06:34:48.561101+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:48.561010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:48.561010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:06.806058+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"73'11","reported_seq":77,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.893567+0000","last_change":"2024-02-05T06:33:31.569835+0000","last_active":"2024-02-05T06:34:44.893567+0000","last_peered":"2024-02-05T06:34:44.893567+0000","last_clean":"2024-02-05T06:34:44.893567+0000","last_became_active":"2024-02-05T06:33:31.569488+0000","last_became_peered":"2024-02-05T06:33:31.569488+0000","last_unstale":"2024-02-05T06:34:44.893567+0000","last_undegraded":"2024-02-05T06:34:44.893567+0000","last_fullsized":"2024-02-05T06:34:44.893567+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:51.980754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890759+0000","last_change":"2024-02-05T06:33:35.600109+0000","last_active":"2024-02-05T06:34:44.890759+0000","last_peered":"2024-02-05T06:34:44.890759+0000","last_clean":"2024-02-05T06:34:44.890759+0000","last_became_active":"2024-02-05T06:33:35.599808+0000","last_became_peered":"2024-02-05T06:33:35.599808+0000","last_unstale":"2024-02-05T06:34:44.890759+0000","last_undegraded":"2024-02-05T06:34:44.890759+0000","last_fullsized":"2024-02-05T06:34:44.890759+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:50.168225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":80,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:54.609822+0000","last_change":"2024-02-05T06:33:33.585921+0000","last_active":"2024-02-05T06:34:54.609822+0000","last_peered":"2024-02-05T06:34:54.609822+0000","last_clean":"2024-02-05T06:34:54.609822+0000","last_became_active":"2024-02-05T06:33:33.585642+0000","last_became_peered":"2024-02-05T06:33:33.585642+0000","last_unstale":"2024-02-05T06:34:54.609822+0000","last_undegraded":"2024-02-05T06:34:54.609822+0000","last_fullsized":"2024-02-05T06:34:54.609822+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.922923+0000","last_change":"2024-02-05T06:33:29.557136+0000","last_active":"2024-02-05T06:34:45.922923+0000","last_peered":"2024-02-05T06:34:45.922923+0000","last_clean":"2024-02-05T06:34:45.922923+0000","last_became_active":"2024-02-05T06:33:29.556832+0000","last_became_peered":"2024-02-05T06:33:29.556832+0000","last_unstale":"2024-02-05T06:34:45.922923+0000","last_undegraded":"2024-02-05T06:34:45.922923+0000","last_fullsized":"2024-02-05T06:34:45.922923+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:52.324493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'10","reported_seq":82,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097179+0000","last_change":"2024-02-05T06:34:41.873963+0000","last_active":"2024-02-05T06:34:45.097179+0000","last_peered":"2024-02-05T06:34:45.097179+0000","last_clean":"2024-02-05T06:34:45.097179+0000","last_became_active":"2024-02-05T06:33:31.666808+0000","last_became_peered":"2024-02-05T06:33:31.666808+0000","last_unstale":"2024-02-05T06:34:45.097179+0000","last_undegraded":"2024-02-05T06:34:45.097179+0000","last_fullsized":"2024-02-05T06:34:45.097179+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-02-05T06:34:41.873876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:41.873876+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:08.435407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":50,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.893956+0000","last_change":"2024-02-05T06:33:35.597787+0000","last_active":"2024-02-05T06:34:44.893956+0000","last_peered":"2024-02-05T06:34:44.893956+0000","last_clean":"2024-02-05T06:34:44.893956+0000","last_became_active":"2024-02-05T06:33:35.597494+0000","last_became_peered":"2024-02-05T06:33:35.597494+0000","last_unstale":"2024-02-05T06:34:44.893956+0000","last_undegraded":"2024-02-05T06:34:44.893956+0000","last_fullsized":"2024-02-05T06:34:44.893956+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:54.313659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":90,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:54.609901+0000","last_change":"2024-02-05T06:34:47.925702+0000","last_active":"2024-02-05T06:34:54.609901+0000","last_peered":"2024-02-05T06:34:54.609901+0000","last_clean":"2024-02-05T06:34:54.609901+0000","last_became_active":"2024-02-05T06:33:33.586043+0000","last_became_peered":"2024-02-05T06:33:33.586043+0000","last_unstale":"2024-02-05T06:34:54.609901+0000","last_undegraded":"2024-02-05T06:34:54.609901+0000","last_fullsized":"2024-02-05T06:34:54.609901+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:34:47.925577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:47.925577+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:48.049254+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":71,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:53.792693+0000","last_change":"2024-02-05T06:34:53.792716+0000","last_active":"2024-02-05T06:34:53.792693+0000","last_peered":"2024-02-05T06:34:53.792693+0000","last_clean":"2024-02-05T06:34:53.792693+0000","last_became_active":"2024-02-05T06:33:29.552170+0000","last_became_peered":"2024-02-05T06:33:29.552170+0000","last_unstale":"2024-02-05T06:34:53.792693+0000","last_undegraded":"2024-02-05T06:34:53.792693+0000","last_fullsized":"2024-02-05T06:34:53.792693+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:53.792621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:53.792621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:58.017125+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"73'9","reported_seq":81,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096168+0000","last_change":"2024-02-05T06:34:36.172574+0000","last_active":"2024-02-05T06:34:45.096168+0000","last_peered":"2024-02-05T06:34:45.096168+0000","last_clean":"2024-02-05T06:34:45.096168+0000","last_became_active":"2024-02-05T06:33:31.666678+0000","last_became_peered":"2024-02-05T06:33:31.666678+0000","last_unstale":"2024-02-05T06:34:45.096168+0000","last_undegraded":"2024-02-05T06:34:45.096168+0000","last_fullsized":"2024-02-05T06:34:45.096168+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:34:36.172444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:36.172444+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:50.358190+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.298026+0000","last_change":"2024-02-05T06:34:43.228873+0000","last_active":"2024-02-05T06:34:45.298026+0000","last_peered":"2024-02-05T06:34:45.298026+0000","last_clean":"2024-02-05T06:34:45.298026+0000","last_became_active":"2024-02-05T06:33:35.601508+0000","last_became_peered":"2024-02-05T06:33:35.601508+0000","last_unstale":"2024-02-05T06:34:45.298026+0000","last_undegraded":"2024-02-05T06:34:45.298026+0000","last_fullsized":"2024-02-05T06:34:45.298026+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:43.228774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:43.228774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:57.334710+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097815+0000","last_change":"2024-02-05T06:34:42.866516+0000","last_active":"2024-02-05T06:34:45.097815+0000","last_peered":"2024-02-05T06:34:45.097815+0000","last_clean":"2024-02-05T06:34:45.097815+0000","last_became_active":"2024-02-05T06:33:33.593025+0000","last_became_peered":"2024-02-05T06:33:33.593025+0000","last_unstale":"2024-02-05T06:34:45.097815+0000","last_undegraded":"2024-02-05T06:34:45.097815+0000","last_fullsized":"2024-02-05T06:34:45.097815+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:42.866394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:42.866394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:43.991930+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":71,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:49.998414+0000","last_change":"2024-02-05T06:34:49.998454+0000","last_active":"2024-02-05T06:34:49.998414+0000","last_peered":"2024-02-05T06:34:49.998414+0000","last_clean":"2024-02-05T06:34:49.998414+0000","last_became_active":"2024-02-05T06:33:29.553880+0000","last_became_peered":"2024-02-05T06:33:29.553880+0000","last_unstale":"2024-02-05T06:34:49.998414+0000","last_undegraded":"2024-02-05T06:34:49.998414+0000","last_fullsized":"2024-02-05T06:34:49.998414+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:49.998323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:49.998323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:50.575718+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"73'9","reported_seq":83,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891187+0000","last_change":"2024-02-05T06:34:43.760971+0000","last_active":"2024-02-05T06:34:44.891187+0000","last_peered":"2024-02-05T06:34:44.891187+0000","last_clean":"2024-02-05T06:34:44.891187+0000","last_became_active":"2024-02-05T06:33:31.665766+0000","last_became_peered":"2024-02-05T06:33:31.665766+0000","last_unstale":"2024-02-05T06:34:44.891187+0000","last_undegraded":"2024-02-05T06:34:44.891187+0000","last_fullsized":"2024-02-05T06:34:44.891187+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:34:43.760888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:43.760888+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:44.661663+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":59,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:35:00.009684+0000","last_change":"2024-02-05T06:35:00.009704+0000","last_active":"2024-02-05T06:35:00.009684+0000","last_peered":"2024-02-05T06:35:00.009684+0000","last_clean":"2024-02-05T06:35:00.009684+0000","last_became_active":"2024-02-05T06:33:35.665596+0000","last_became_peered":"2024-02-05T06:33:35.665596+0000","last_unstale":"2024-02-05T06:35:00.009684+0000","last_undegraded":"2024-02-05T06:35:00.009684+0000","last_fullsized":"2024-02-05T06:35:00.009684+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:00.009630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:00.009630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:05.139245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096424+0000","last_change":"2024-02-05T06:34:45.096449+0000","last_active":"2024-02-05T06:34:45.096424+0000","last_peered":"2024-02-05T06:34:45.096424+0000","last_clean":"2024-02-05T06:34:45.096424+0000","last_became_active":"2024-02-05T06:33:33.586768+0000","last_became_peered":"2024-02-05T06:33:33.586768+0000","last_unstale":"2024-02-05T06:34:45.096424+0000","last_undegraded":"2024-02-05T06:34:45.096424+0000","last_fullsized":"2024-02-05T06:34:45.096424+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:45.096340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:45.096340+0000","last_clean_scrub_stamp":"2024-02-05T06:34:45.096340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:12.226977+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":70,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.265669+0000","last_change":"2024-02-05T06:34:30.204064+0000","last_active":"2024-02-05T06:34:45.265669+0000","last_peered":"2024-02-05T06:34:45.265669+0000","last_clean":"2024-02-05T06:34:45.265669+0000","last_became_active":"2024-02-05T06:33:29.554427+0000","last_became_peered":"2024-02-05T06:33:29.554427+0000","last_unstale":"2024-02-05T06:34:45.265669+0000","last_undegraded":"2024-02-05T06:34:45.265669+0000","last_fullsized":"2024-02-05T06:34:45.265669+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:30.203967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:30.203967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:35.531249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":76,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:55.912349+0000","last_change":"2024-02-05T06:34:55.912380+0000","last_active":"2024-02-05T06:34:55.912349+0000","last_peered":"2024-02-05T06:34:55.912349+0000","last_clean":"2024-02-05T06:34:55.912349+0000","last_became_active":"2024-02-05T06:33:31.665847+0000","last_became_peered":"2024-02-05T06:33:31.665847+0000","last_unstale":"2024-02-05T06:34:55.912349+0000","last_undegraded":"2024-02-05T06:34:55.912349+0000","last_fullsized":"2024-02-05T06:34:55.912349+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'6","last_scrub_stamp":"2024-02-05T06:34:55.912282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:55.912282+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:02.248574+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095574+0000","last_change":"2024-02-05T06:33:35.665928+0000","last_active":"2024-02-05T06:34:45.095574+0000","last_peered":"2024-02-05T06:34:45.095574+0000","last_clean":"2024-02-05T06:34:45.095574+0000","last_became_active":"2024-02-05T06:33:35.665581+0000","last_became_peered":"2024-02-05T06:33:35.665581+0000","last_unstale":"2024-02-05T06:34:45.095574+0000","last_undegraded":"2024-02-05T06:34:45.095574+0000","last_fullsized":"2024-02-05T06:34:45.095574+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:47.218454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":75,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.923730+0000","last_change":"2024-02-05T06:33:29.558514+0000","last_active":"2024-02-05T06:34:45.923730+0000","last_peered":"2024-02-05T06:34:45.923730+0000","last_clean":"2024-02-05T06:34:45.923730+0000","last_became_active":"2024-02-05T06:33:29.557522+0000","last_became_peered":"2024-02-05T06:33:29.557522+0000","last_unstale":"2024-02-05T06:34:45.923730+0000","last_undegraded":"2024-02-05T06:34:45.923730+0000","last_fullsized":"2024-02-05T06:34:45.923730+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:51.837286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"73'15","reported_seq":82,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095250+0000","last_change":"2024-02-05T06:33:31.666410+0000","last_active":"2024-02-05T06:34:45.095250+0000","last_peered":"2024-02-05T06:34:45.095250+0000","last_clean":"2024-02-05T06:34:45.095250+0000","last_became_active":"2024-02-05T06:33:31.666056+0000","last_became_peered":"2024-02-05T06:33:31.666056+0000","last_unstale":"2024-02-05T06:34:45.095250+0000","last_undegraded":"2024-02-05T06:34:45.095250+0000","last_fullsized":"2024-02-05T06:34:45.095250+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:56.860240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"73'8","reported_seq":90,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:54.609874+0000","last_change":"2024-02-05T06:34:49.007791+0000","last_active":"2024-02-05T06:34:54.609874+0000","last_peered":"2024-02-05T06:34:54.609874+0000","last_clean":"2024-02-05T06:34:54.609874+0000","last_became_active":"2024-02-05T06:33:33.589535+0000","last_became_peered":"2024-02-05T06:33:33.589535+0000","last_unstale":"2024-02-05T06:34:54.609874+0000","last_undegraded":"2024-02-05T06:34:54.609874+0000","last_fullsized":"2024-02-05T06:34:54.609874+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:34:49.007659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:49.007659+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:59.892182+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096781+0000","last_change":"2024-02-05T06:33:35.601680+0000","last_active":"2024-02-05T06:34:45.096781+0000","last_peered":"2024-02-05T06:34:45.096781+0000","last_clean":"2024-02-05T06:34:45.096781+0000","last_became_active":"2024-02-05T06:33:35.601030+0000","last_became_peered":"2024-02-05T06:33:35.601030+0000","last_unstale":"2024-02-05T06:34:45.096781+0000","last_undegraded":"2024-02-05T06:34:45.096781+0000","last_fullsized":"2024-02-05T06:34:45.096781+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:53.348191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.923545+0000","last_change":"2024-02-05T06:33:29.558929+0000","last_active":"2024-02-05T06:34:45.923545+0000","last_peered":"2024-02-05T06:34:45.923545+0000","last_clean":"2024-02-05T06:34:45.923545+0000","last_became_active":"2024-02-05T06:33:29.557284+0000","last_became_peered":"2024-02-05T06:33:29.557284+0000","last_unstale":"2024-02-05T06:34:45.923545+0000","last_undegraded":"2024-02-05T06:34:45.923545+0000","last_fullsized":"2024-02-05T06:34:45.923545+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:54.175804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'12","reported_seq":89,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:35:00.650602+0000","last_change":"2024-02-05T06:35:00.650630+0000","last_active":"2024-02-05T06:35:00.650602+0000","last_peered":"2024-02-05T06:35:00.650602+0000","last_clean":"2024-02-05T06:35:00.650602+0000","last_became_active":"2024-02-05T06:33:31.569924+0000","last_became_peered":"2024-02-05T06:33:31.569924+0000","last_unstale":"2024-02-05T06:35:00.650602+0000","last_undegraded":"2024-02-05T06:35:00.650602+0000","last_fullsized":"2024-02-05T06:35:00.650602+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:35:00.650542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:35:00.650542+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:28.714634+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.297549+0000","last_change":"2024-02-05T06:34:40.272941+0000","last_active":"2024-02-05T06:34:45.297549+0000","last_peered":"2024-02-05T06:34:45.297549+0000","last_clean":"2024-02-05T06:34:45.297549+0000","last_became_active":"2024-02-05T06:33:33.592546+0000","last_became_peered":"2024-02-05T06:33:33.592546+0000","last_unstale":"2024-02-05T06:34:45.297549+0000","last_undegraded":"2024-02-05T06:34:45.297549+0000","last_fullsized":"2024-02-05T06:34:45.297549+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:40.272852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:40.272852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:07.055905+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.923123+0000","last_change":"2024-02-05T06:33:35.599830+0000","last_active":"2024-02-05T06:34:45.923123+0000","last_peered":"2024-02-05T06:34:45.923123+0000","last_clean":"2024-02-05T06:34:45.923123+0000","last_became_active":"2024-02-05T06:33:35.599527+0000","last_became_peered":"2024-02-05T06:33:35.599527+0000","last_unstale":"2024-02-05T06:34:45.923123+0000","last_undegraded":"2024-02-05T06:34:45.923123+0000","last_fullsized":"2024-02-05T06:34:45.923123+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:58.253161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.297822+0000","last_change":"2024-02-05T06:34:42.265315+0000","last_active":"2024-02-05T06:34:45.297822+0000","last_peered":"2024-02-05T06:34:45.297822+0000","last_clean":"2024-02-05T06:34:45.297822+0000","last_became_active":"2024-02-05T06:33:33.589203+0000","last_became_peered":"2024-02-05T06:33:33.589203+0000","last_unstale":"2024-02-05T06:34:45.297822+0000","last_undegraded":"2024-02-05T06:34:45.297822+0000","last_fullsized":"2024-02-05T06:34:45.297822+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:42.265214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:42.265214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:53.433345+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":71,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:59.028807+0000","last_change":"2024-02-05T06:34:59.028827+0000","last_active":"2024-02-05T06:34:59.028807+0000","last_peered":"2024-02-05T06:34:59.028807+0000","last_clean":"2024-02-05T06:34:59.028807+0000","last_became_active":"2024-02-05T06:33:29.556541+0000","last_became_peered":"2024-02-05T06:33:29.556541+0000","last_unstale":"2024-02-05T06:34:59.028807+0000","last_undegraded":"2024-02-05T06:34:59.028807+0000","last_fullsized":"2024-02-05T06:34:59.028807+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:59.028748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:59.028748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:24.209293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"73'19","reported_seq":97,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.265984+0000","last_change":"2024-02-05T06:34:44.243017+0000","last_active":"2024-02-05T06:34:45.265984+0000","last_peered":"2024-02-05T06:34:45.265984+0000","last_clean":"2024-02-05T06:34:45.265984+0000","last_became_active":"2024-02-05T06:33:31.664699+0000","last_became_peered":"2024-02-05T06:33:31.664699+0000","last_unstale":"2024-02-05T06:34:45.265984+0000","last_undegraded":"2024-02-05T06:34:45.265984+0000","last_fullsized":"2024-02-05T06:34:45.265984+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-02-05T06:34:44.242897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:44.242897+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:01.187407+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.889844+0000","last_change":"2024-02-05T06:33:35.598200+0000","last_active":"2024-02-05T06:34:44.889844+0000","last_peered":"2024-02-05T06:34:44.889844+0000","last_clean":"2024-02-05T06:34:44.889844+0000","last_became_active":"2024-02-05T06:33:35.597863+0000","last_became_peered":"2024-02-05T06:33:35.597863+0000","last_unstale":"2024-02-05T06:34:44.889844+0000","last_undegraded":"2024-02-05T06:34:44.889844+0000","last_fullsized":"2024-02-05T06:34:44.889844+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:55.272938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.299123+0000","last_change":"2024-02-05T06:34:45.299145+0000","last_active":"2024-02-05T06:34:45.299123+0000","last_peered":"2024-02-05T06:34:45.299123+0000","last_clean":"2024-02-05T06:34:45.299123+0000","last_became_active":"2024-02-05T06:33:33.586895+0000","last_became_peered":"2024-02-05T06:33:33.586895+0000","last_unstale":"2024-02-05T06:34:45.299123+0000","last_undegraded":"2024-02-05T06:34:45.299123+0000","last_fullsized":"2024-02-05T06:34:45.299123+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:45.299055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:45.299055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:03.861496+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095649+0000","last_change":"2024-02-05T06:33:29.556254+0000","last_active":"2024-02-05T06:34:45.095649+0000","last_peered":"2024-02-05T06:34:45.095649+0000","last_clean":"2024-02-05T06:34:45.095649+0000","last_became_active":"2024-02-05T06:33:29.555950+0000","last_became_peered":"2024-02-05T06:33:29.555950+0000","last_unstale":"2024-02-05T06:34:45.095649+0000","last_undegraded":"2024-02-05T06:34:45.095649+0000","last_fullsized":"2024-02-05T06:34:45.095649+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:52.533616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"73'9","reported_seq":83,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891706+0000","last_change":"2024-02-05T06:34:36.779561+0000","last_active":"2024-02-05T06:34:44.891706+0000","last_peered":"2024-02-05T06:34:44.891706+0000","last_clean":"2024-02-05T06:34:44.891706+0000","last_became_active":"2024-02-05T06:33:31.564572+0000","last_became_peered":"2024-02-05T06:33:31.564572+0000","last_unstale":"2024-02-05T06:34:44.891706+0000","last_undegraded":"2024-02-05T06:34:44.891706+0000","last_fullsized":"2024-02-05T06:34:44.891706+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:34:36.779466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:36.779466+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:02.144960+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097537+0000","last_change":"2024-02-05T06:33:35.598477+0000","last_active":"2024-02-05T06:34:45.097537+0000","last_peered":"2024-02-05T06:34:45.097537+0000","last_clean":"2024-02-05T06:34:45.097537+0000","last_became_active":"2024-02-05T06:33:35.597750+0000","last_became_peered":"2024-02-05T06:33:35.597750+0000","last_unstale":"2024-02-05T06:34:45.097537+0000","last_undegraded":"2024-02-05T06:34:45.097537+0000","last_fullsized":"2024-02-05T06:34:45.097537+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:53.938845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":80,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:48.046085+0000","last_change":"2024-02-05T06:34:48.046115+0000","last_active":"2024-02-05T06:34:48.046085+0000","last_peered":"2024-02-05T06:34:48.046085+0000","last_clean":"2024-02-05T06:34:48.046085+0000","last_became_active":"2024-02-05T06:33:28.542228+0000","last_became_peered":"2024-02-05T06:33:28.542228+0000","last_unstale":"2024-02-05T06:34:48.046085+0000","last_undegraded":"2024-02-05T06:34:48.046085+0000","last_fullsized":"2024-02-05T06:34:48.046085+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-02-05T06:34:48.046017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:34:48.046017+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:04.358520+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001220348,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":62,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.892994+0000","last_change":"2024-02-05T06:33:29.554533+0000","last_active":"2024-02-05T06:34:44.892994+0000","last_peered":"2024-02-05T06:34:44.892994+0000","last_clean":"2024-02-05T06:34:44.892994+0000","last_became_active":"2024-02-05T06:33:29.553882+0000","last_became_peered":"2024-02-05T06:33:29.553882+0000","last_unstale":"2024-02-05T06:34:44.892994+0000","last_undegraded":"2024-02-05T06:34:44.892994+0000","last_fullsized":"2024-02-05T06:34:44.892994+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:56.794367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":74,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:54.610076+0000","last_change":"2024-02-05T06:33:55.896865+0000","last_active":"2024-02-05T06:34:54.610076+0000","last_peered":"2024-02-05T06:34:54.610076+0000","last_clean":"2024-02-05T06:34:54.610076+0000","last_became_active":"2024-02-05T06:33:55.884601+0000","last_became_peered":"2024-02-05T06:33:55.884601+0000","last_unstale":"2024-02-05T06:34:54.610076+0000","last_undegraded":"2024-02-05T06:34:54.610076+0000","last_fullsized":"2024-02-05T06:34:54.610076+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":55,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890270+0000","last_change":"2024-02-05T06:33:33.589668+0000","last_active":"2024-02-05T06:34:44.890270+0000","last_peered":"2024-02-05T06:34:44.890270+0000","last_clean":"2024-02-05T06:34:44.890270+0000","last_became_active":"2024-02-05T06:33:33.589348+0000","last_became_peered":"2024-02-05T06:33:33.589348+0000","last_unstale":"2024-02-05T06:34:44.890270+0000","last_undegraded":"2024-02-05T06:34:44.890270+0000","last_fullsized":"2024-02-05T06:34:44.890270+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.923731+0000","last_change":"2024-02-05T06:33:35.666590+0000","last_active":"2024-02-05T06:34:45.923731+0000","last_peered":"2024-02-05T06:34:45.923731+0000","last_clean":"2024-02-05T06:34:45.923731+0000","last_became_active":"2024-02-05T06:33:35.666324+0000","last_became_peered":"2024-02-05T06:33:35.666324+0000","last_unstale":"2024-02-05T06:34:45.923731+0000","last_undegraded":"2024-02-05T06:34:45.923731+0000","last_fullsized":"2024-02-05T06:34:45.923731+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:51.202340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":62,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890767+0000","last_change":"2024-02-05T06:33:29.552003+0000","last_active":"2024-02-05T06:34:44.890767+0000","last_peered":"2024-02-05T06:34:44.890767+0000","last_clean":"2024-02-05T06:34:44.890767+0000","last_became_active":"2024-02-05T06:33:29.551549+0000","last_became_peered":"2024-02-05T06:33:29.551549+0000","last_unstale":"2024-02-05T06:34:44.890767+0000","last_undegraded":"2024-02-05T06:34:44.890767+0000","last_fullsized":"2024-02-05T06:34:44.890767+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:56.472943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":78,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:50.302189+0000","last_change":"2024-02-05T06:34:50.302211+0000","last_active":"2024-02-05T06:34:50.302189+0000","last_peered":"2024-02-05T06:34:50.302189+0000","last_clean":"2024-02-05T06:34:50.302189+0000","last_became_active":"2024-02-05T06:33:28.542977+0000","last_became_peered":"2024-02-05T06:33:28.542977+0000","last_unstale":"2024-02-05T06:34:50.302189+0000","last_undegraded":"2024-02-05T06:34:50.302189+0000","last_fullsized":"2024-02-05T06:34:50.302189+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:50.302133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:34:50.302133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:57.608189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091640799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'13","reported_seq":85,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891357+0000","last_change":"2024-02-05T06:33:31.562891+0000","last_active":"2024-02-05T06:34:44.891357+0000","last_peered":"2024-02-05T06:34:44.891357+0000","last_clean":"2024-02-05T06:34:44.891357+0000","last_became_active":"2024-02-05T06:33:31.560082+0000","last_became_peered":"2024-02-05T06:33:31.560082+0000","last_unstale":"2024-02-05T06:34:44.891357+0000","last_undegraded":"2024-02-05T06:34:44.891357+0000","last_fullsized":"2024-02-05T06:34:44.891357+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:50.831110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.298136+0000","last_change":"2024-02-05T06:34:34.240413+0000","last_active":"2024-02-05T06:34:45.298136+0000","last_peered":"2024-02-05T06:34:45.298136+0000","last_clean":"2024-02-05T06:34:45.298136+0000","last_became_active":"2024-02-05T06:33:33.587593+0000","last_became_peered":"2024-02-05T06:33:33.587593+0000","last_unstale":"2024-02-05T06:34:45.298136+0000","last_undegraded":"2024-02-05T06:34:45.298136+0000","last_fullsized":"2024-02-05T06:34:45.298136+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:34.240331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:34.240331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:49.655873+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095927+0000","last_change":"2024-02-05T06:33:35.600278+0000","last_active":"2024-02-05T06:34:45.095927+0000","last_peered":"2024-02-05T06:34:45.095927+0000","last_clean":"2024-02-05T06:34:45.095927+0000","last_became_active":"2024-02-05T06:33:35.600004+0000","last_became_peered":"2024-02-05T06:33:35.600004+0000","last_unstale":"2024-02-05T06:34:45.095927+0000","last_undegraded":"2024-02-05T06:34:45.095927+0000","last_fullsized":"2024-02-05T06:34:45.095927+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:59.635646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":317,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.923419+0000","last_change":"2024-02-05T06:33:45.752222+0000","last_active":"2024-02-05T06:34:45.923419+0000","last_peered":"2024-02-05T06:34:45.923419+0000","last_clean":"2024-02-05T06:34:45.923419+0000","last_became_active":"2024-02-05T06:33:19.862877+0000","last_became_peered":"2024-02-05T06:33:19.862877+0000","last_unstale":"2024-02-05T06:34:45.923419+0000","last_undegraded":"2024-02-05T06:34:45.923419+0000","last_fullsized":"2024-02-05T06:34:45.923419+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-02-05T06:33:45.752124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:33:45.752124+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:54.331188+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":71,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097177+0000","last_change":"2024-02-05T06:34:44.894533+0000","last_active":"2024-02-05T06:34:45.097177+0000","last_peered":"2024-02-05T06:34:45.097177+0000","last_clean":"2024-02-05T06:34:45.097177+0000","last_became_active":"2024-02-05T06:33:29.552698+0000","last_became_peered":"2024-02-05T06:33:29.552698+0000","last_unstale":"2024-02-05T06:34:45.097177+0000","last_undegraded":"2024-02-05T06:34:45.097177+0000","last_fullsized":"2024-02-05T06:34:45.097177+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:44.894443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:44.894443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:54.598610+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"73'16","reported_seq":99,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:54.609856+0000","last_change":"2024-02-05T06:33:31.666149+0000","last_active":"2024-02-05T06:34:54.609856+0000","last_peered":"2024-02-05T06:34:54.609856+0000","last_clean":"2024-02-05T06:34:54.609856+0000","last_became_active":"2024-02-05T06:33:31.665797+0000","last_became_peered":"2024-02-05T06:33:31.665797+0000","last_unstale":"2024-02-05T06:34:54.609856+0000","last_undegraded":"2024-02-05T06:34:54.609856+0000","last_fullsized":"2024-02-05T06:34:54.609856+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:00.032828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":55,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.922387+0000","last_change":"2024-02-05T06:33:33.591727+0000","last_active":"2024-02-05T06:34:45.922387+0000","last_peered":"2024-02-05T06:34:45.922387+0000","last_clean":"2024-02-05T06:34:45.922387+0000","last_became_active":"2024-02-05T06:33:33.591484+0000","last_became_peered":"2024-02-05T06:33:33.591484+0000","last_unstale":"2024-02-05T06:34:45.922387+0000","last_undegraded":"2024-02-05T06:34:45.922387+0000","last_fullsized":"2024-02-05T06:34:45.922387+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:56.324621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":58,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:46.361083+0000","last_change":"2024-02-05T06:34:46.361122+0000","last_active":"2024-02-05T06:34:46.361083+0000","last_peered":"2024-02-05T06:34:46.361083+0000","last_clean":"2024-02-05T06:34:46.361083+0000","last_became_active":"2024-02-05T06:33:35.600656+0000","last_became_peered":"2024-02-05T06:33:35.600656+0000","last_unstale":"2024-02-05T06:34:46.361083+0000","last_undegraded":"2024-02-05T06:34:46.361083+0000","last_fullsized":"2024-02-05T06:34:46.361083+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:46.361013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:46.361013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:07.997590+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":71,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890127+0000","last_change":"2024-02-05T06:34:44.781805+0000","last_active":"2024-02-05T06:34:44.890127+0000","last_peered":"2024-02-05T06:34:44.890127+0000","last_clean":"2024-02-05T06:34:44.890127+0000","last_became_active":"2024-02-05T06:33:29.544399+0000","last_became_peered":"2024-02-05T06:33:29.544399+0000","last_unstale":"2024-02-05T06:34:44.890127+0000","last_undegraded":"2024-02-05T06:34:44.890127+0000","last_fullsized":"2024-02-05T06:34:44.890127+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:44.781704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:44.781704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:06.382435+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":167,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:59.434203+0000","last_change":"2024-02-05T06:34:41.991897+0000","last_active":"2024-02-05T06:34:59.434203+0000","last_peered":"2024-02-05T06:34:59.434203+0000","last_clean":"2024-02-05T06:34:59.434203+0000","last_became_active":"2024-02-05T06:33:28.544852+0000","last_became_peered":"2024-02-05T06:33:28.544852+0000","last_unstale":"2024-02-05T06:34:59.434203+0000","last_undegraded":"2024-02-05T06:34:59.434203+0000","last_fullsized":"2024-02-05T06:34:59.434203+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T06:34:41.991777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:34:41.991777+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:10.879161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00074151600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":84,"num_read_kb":79,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":85,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:52.743768+0000","last_change":"2024-02-05T06:34:52.743798+0000","last_active":"2024-02-05T06:34:52.743768+0000","last_peered":"2024-02-05T06:34:52.743768+0000","last_clean":"2024-02-05T06:34:52.743768+0000","last_became_active":"2024-02-05T06:33:31.564604+0000","last_became_peered":"2024-02-05T06:33:31.564604+0000","last_unstale":"2024-02-05T06:34:52.743768+0000","last_undegraded":"2024-02-05T06:34:52.743768+0000","last_fullsized":"2024-02-05T06:34:52.743768+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:34:52.743708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:52.743708+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:55.426621+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":53,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096253+0000","last_change":"2024-02-05T06:33:33.589962+0000","last_active":"2024-02-05T06:34:45.096253+0000","last_peered":"2024-02-05T06:34:45.096253+0000","last_clean":"2024-02-05T06:34:45.096253+0000","last_became_active":"2024-02-05T06:33:33.589388+0000","last_became_peered":"2024-02-05T06:33:33.589388+0000","last_unstale":"2024-02-05T06:34:45.096253+0000","last_undegraded":"2024-02-05T06:34:45.096253+0000","last_fullsized":"2024-02-05T06:34:45.096253+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:01.833071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":50,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890056+0000","last_change":"2024-02-05T06:33:35.665208+0000","last_active":"2024-02-05T06:34:44.890056+0000","last_peered":"2024-02-05T06:34:44.890056+0000","last_clean":"2024-02-05T06:34:44.890056+0000","last_became_active":"2024-02-05T06:33:35.664858+0000","last_became_peered":"2024-02-05T06:33:35.664858+0000","last_unstale":"2024-02-05T06:34:44.890056+0000","last_undegraded":"2024-02-05T06:34:44.890056+0000","last_fullsized":"2024-02-05T06:34:44.890056+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:53.295103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"66'1","reported_seq":76,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890784+0000","last_change":"2024-02-05T06:34:36.457175+0000","last_active":"2024-02-05T06:34:44.890784+0000","last_peered":"2024-02-05T06:34:44.890784+0000","last_clean":"2024-02-05T06:34:44.890784+0000","last_became_active":"2024-02-05T06:33:29.548227+0000","last_became_peered":"2024-02-05T06:33:29.548227+0000","last_unstale":"2024-02-05T06:34:44.890784+0000","last_undegraded":"2024-02-05T06:34:44.890784+0000","last_fullsized":"2024-02-05T06:34:44.890784+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:34:36.457059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:36.457059+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:45.403222+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'19","reported_seq":95,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891811+0000","last_change":"2024-02-05T06:33:31.667636+0000","last_active":"2024-02-05T06:34:44.891811+0000","last_peered":"2024-02-05T06:34:44.891811+0000","last_clean":"2024-02-05T06:34:44.891811+0000","last_became_active":"2024-02-05T06:33:31.667340+0000","last_became_peered":"2024-02-05T06:33:31.667340+0000","last_unstale":"2024-02-05T06:34:44.891811+0000","last_undegraded":"2024-02-05T06:34:44.891811+0000","last_fullsized":"2024-02-05T06:34:44.891811+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:45.776316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":62,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:50.353126+0000","last_change":"2024-02-05T06:34:50.353163+0000","last_active":"2024-02-05T06:34:50.353126+0000","last_peered":"2024-02-05T06:34:50.353126+0000","last_clean":"2024-02-05T06:34:50.353126+0000","last_became_active":"2024-02-05T06:33:33.588163+0000","last_became_peered":"2024-02-05T06:33:33.588163+0000","last_unstale":"2024-02-05T06:34:50.353126+0000","last_undegraded":"2024-02-05T06:34:50.353126+0000","last_fullsized":"2024-02-05T06:34:50.353126+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:50.353040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:50.353040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:10.822758+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.889406+0000","last_change":"2024-02-05T06:33:35.600831+0000","last_active":"2024-02-05T06:34:44.889406+0000","last_peered":"2024-02-05T06:34:44.889406+0000","last_clean":"2024-02-05T06:34:44.889406+0000","last_became_active":"2024-02-05T06:33:35.600485+0000","last_became_peered":"2024-02-05T06:33:35.600485+0000","last_unstale":"2024-02-05T06:34:44.889406+0000","last_undegraded":"2024-02-05T06:34:44.889406+0000","last_fullsized":"2024-02-05T06:34:44.889406+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:51.006304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.095967+0000","last_change":"2024-02-05T06:34:43.098655+0000","last_active":"2024-02-05T06:34:45.095967+0000","last_peered":"2024-02-05T06:34:45.095967+0000","last_clean":"2024-02-05T06:34:45.095967+0000","last_became_active":"2024-02-05T06:33:29.554940+0000","last_became_peered":"2024-02-05T06:33:29.554940+0000","last_unstale":"2024-02-05T06:34:45.095967+0000","last_undegraded":"2024-02-05T06:34:45.095967+0000","last_fullsized":"2024-02-05T06:34:45.095967+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:43.098546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:43.098546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:01.990483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"73'10","reported_seq":73,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891541+0000","last_change":"2024-02-05T06:33:31.570032+0000","last_active":"2024-02-05T06:34:44.891541+0000","last_peered":"2024-02-05T06:34:44.891541+0000","last_clean":"2024-02-05T06:34:44.891541+0000","last_became_active":"2024-02-05T06:33:31.569680+0000","last_became_peered":"2024-02-05T06:33:31.569680+0000","last_unstale":"2024-02-05T06:34:44.891541+0000","last_undegraded":"2024-02-05T06:34:44.891541+0000","last_fullsized":"2024-02-05T06:34:44.891541+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:57.891117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"73'8","reported_seq":90,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:54.764491+0000","last_change":"2024-02-05T06:34:54.764491+0000","last_active":"2024-02-05T06:34:54.764491+0000","last_peered":"2024-02-05T06:34:54.764491+0000","last_clean":"2024-02-05T06:34:54.764491+0000","last_became_active":"2024-02-05T06:33:33.582187+0000","last_became_peered":"2024-02-05T06:33:33.582187+0000","last_unstale":"2024-02-05T06:34:54.764491+0000","last_undegraded":"2024-02-05T06:34:54.764491+0000","last_fullsized":"2024-02-05T06:34:54.764491+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:34:54.764410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:54.764410+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:15.024154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":59,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.923213+0000","last_change":"2024-02-05T06:34:41.001666+0000","last_active":"2024-02-05T06:34:45.923213+0000","last_peered":"2024-02-05T06:34:45.923213+0000","last_clean":"2024-02-05T06:34:45.923213+0000","last_became_active":"2024-02-05T06:33:35.666315+0000","last_became_peered":"2024-02-05T06:33:35.666315+0000","last_unstale":"2024-02-05T06:34:45.923213+0000","last_undegraded":"2024-02-05T06:34:45.923213+0000","last_fullsized":"2024-02-05T06:34:45.923213+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:41.001570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:41.001570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:53.416960+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":64,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890941+0000","last_change":"2024-02-05T06:33:29.546683+0000","last_active":"2024-02-05T06:34:44.890941+0000","last_peered":"2024-02-05T06:34:44.890941+0000","last_clean":"2024-02-05T06:34:44.890941+0000","last_became_active":"2024-02-05T06:33:29.546405+0000","last_became_peered":"2024-02-05T06:33:29.546405+0000","last_unstale":"2024-02-05T06:34:44.890941+0000","last_undegraded":"2024-02-05T06:34:44.890941+0000","last_fullsized":"2024-02-05T06:34:44.890941+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:45.453250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'14","reported_seq":87,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:56.595969+0000","last_change":"2024-02-05T06:34:56.595997+0000","last_active":"2024-02-05T06:34:56.595969+0000","last_peered":"2024-02-05T06:34:56.595969+0000","last_clean":"2024-02-05T06:34:56.595969+0000","last_became_active":"2024-02-05T06:33:31.563836+0000","last_became_peered":"2024-02-05T06:33:31.563836+0000","last_unstale":"2024-02-05T06:34:56.595969+0000","last_undegraded":"2024-02-05T06:34:56.595969+0000","last_fullsized":"2024-02-05T06:34:56.595969+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'14","last_scrub_stamp":"2024-02-05T06:34:56.595906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:56.595906+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:05.623306+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"73'8","reported_seq":90,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:56.941927+0000","last_change":"2024-02-05T06:34:56.941927+0000","last_active":"2024-02-05T06:34:56.941927+0000","last_peered":"2024-02-05T06:34:56.941927+0000","last_clean":"2024-02-05T06:34:56.941927+0000","last_became_active":"2024-02-05T06:33:33.591646+0000","last_became_peered":"2024-02-05T06:33:33.591646+0000","last_unstale":"2024-02-05T06:34:56.941927+0000","last_undegraded":"2024-02-05T06:34:56.941927+0000","last_fullsized":"2024-02-05T06:34:56.941927+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:34:56.941840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:56.941840+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:25.481310+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":50,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.892762+0000","last_change":"2024-02-05T06:33:35.600296+0000","last_active":"2024-02-05T06:34:44.892762+0000","last_peered":"2024-02-05T06:34:44.892762+0000","last_clean":"2024-02-05T06:34:44.892762+0000","last_became_active":"2024-02-05T06:33:35.599805+0000","last_became_peered":"2024-02-05T06:33:35.599805+0000","last_unstale":"2024-02-05T06:34:44.892762+0000","last_undegraded":"2024-02-05T06:34:44.892762+0000","last_fullsized":"2024-02-05T06:34:44.892762+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:03.912813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097711+0000","last_change":"2024-02-05T06:33:29.557121+0000","last_active":"2024-02-05T06:34:45.097711+0000","last_peered":"2024-02-05T06:34:45.097711+0000","last_clean":"2024-02-05T06:34:45.097711+0000","last_became_active":"2024-02-05T06:33:29.556002+0000","last_became_peered":"2024-02-05T06:33:29.556002+0000","last_unstale":"2024-02-05T06:34:45.097711+0000","last_undegraded":"2024-02-05T06:34:45.097711+0000","last_fullsized":"2024-02-05T06:34:45.097711+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:48.546981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"73'18","reported_seq":91,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096701+0000","last_change":"2024-02-05T06:33:31.666298+0000","last_active":"2024-02-05T06:34:45.096701+0000","last_peered":"2024-02-05T06:34:45.096701+0000","last_clean":"2024-02-05T06:34:45.096701+0000","last_became_active":"2024-02-05T06:33:31.665995+0000","last_became_peered":"2024-02-05T06:33:31.665995+0000","last_unstale":"2024-02-05T06:34:45.096701+0000","last_undegraded":"2024-02-05T06:34:45.096701+0000","last_fullsized":"2024-02-05T06:34:45.096701+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:47.201122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":62,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:49.579564+0000","last_change":"2024-02-05T06:34:49.579591+0000","last_active":"2024-02-05T06:34:49.579564+0000","last_peered":"2024-02-05T06:34:49.579564+0000","last_clean":"2024-02-05T06:34:49.579564+0000","last_became_active":"2024-02-05T06:33:33.586935+0000","last_became_peered":"2024-02-05T06:33:33.586935+0000","last_unstale":"2024-02-05T06:34:49.579564+0000","last_undegraded":"2024-02-05T06:34:49.579564+0000","last_fullsized":"2024-02-05T06:34:49.579564+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:49.579493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:49.579493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:00.372885+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:53.061776+0000","last_change":"2024-02-05T06:34:53.061797+0000","last_active":"2024-02-05T06:34:53.061776+0000","last_peered":"2024-02-05T06:34:53.061776+0000","last_clean":"2024-02-05T06:34:53.061776+0000","last_became_active":"2024-02-05T06:33:35.593210+0000","last_became_peered":"2024-02-05T06:33:35.593210+0000","last_unstale":"2024-02-05T06:34:53.061776+0000","last_undegraded":"2024-02-05T06:34:53.061776+0000","last_fullsized":"2024-02-05T06:34:53.061776+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:53.061717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:53.061717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:58.799900+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":89,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:54.609828+0000","last_change":"2024-02-05T06:34:40.619333+0000","last_active":"2024-02-05T06:34:54.609828+0000","last_peered":"2024-02-05T06:34:54.609828+0000","last_clean":"2024-02-05T06:34:54.609828+0000","last_became_active":"2024-02-05T06:33:33.586753+0000","last_became_peered":"2024-02-05T06:33:33.586753+0000","last_unstale":"2024-02-05T06:34:54.609828+0000","last_undegraded":"2024-02-05T06:34:54.609828+0000","last_fullsized":"2024-02-05T06:34:54.609828+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:34:40.619192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:40.619192+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:48.417494+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":63,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096676+0000","last_change":"2024-02-05T06:33:29.557073+0000","last_active":"2024-02-05T06:34:45.096676+0000","last_peered":"2024-02-05T06:34:45.096676+0000","last_clean":"2024-02-05T06:34:45.096676+0000","last_became_active":"2024-02-05T06:33:29.555919+0000","last_became_peered":"2024-02-05T06:33:29.555919+0000","last_unstale":"2024-02-05T06:34:45.096676+0000","last_undegraded":"2024-02-05T06:34:45.096676+0000","last_fullsized":"2024-02-05T06:34:45.096676+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:54.542025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"73'15","reported_seq":83,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.891508+0000","last_change":"2024-02-05T06:33:31.572905+0000","last_active":"2024-02-05T06:34:44.891508+0000","last_peered":"2024-02-05T06:34:44.891508+0000","last_clean":"2024-02-05T06:34:44.891508+0000","last_became_active":"2024-02-05T06:33:31.572603+0000","last_became_peered":"2024-02-05T06:33:31.572603+0000","last_unstale":"2024-02-05T06:34:44.891508+0000","last_undegraded":"2024-02-05T06:34:44.891508+0000","last_fullsized":"2024-02-05T06:34:44.891508+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:49.104679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096917+0000","last_change":"2024-02-05T06:33:35.668457+0000","last_active":"2024-02-05T06:34:45.096917+0000","last_peered":"2024-02-05T06:34:45.096917+0000","last_clean":"2024-02-05T06:34:45.096917+0000","last_became_active":"2024-02-05T06:33:35.668020+0000","last_became_peered":"2024-02-05T06:33:35.668020+0000","last_unstale":"2024-02-05T06:34:45.096917+0000","last_undegraded":"2024-02-05T06:34:45.096917+0000","last_fullsized":"2024-02-05T06:34:45.096917+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:55.572974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":53,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.096246+0000","last_change":"2024-02-05T06:33:33.583518+0000","last_active":"2024-02-05T06:34:45.096246+0000","last_peered":"2024-02-05T06:34:45.096246+0000","last_clean":"2024-02-05T06:34:45.096246+0000","last_became_active":"2024-02-05T06:33:33.583029+0000","last_became_peered":"2024-02-05T06:33:33.583029+0000","last_unstale":"2024-02-05T06:34:45.096246+0000","last_undegraded":"2024-02-05T06:34:45.096246+0000","last_fullsized":"2024-02-05T06:34:45.096246+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:49.769832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":62,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.892790+0000","last_change":"2024-02-05T06:33:29.553248+0000","last_active":"2024-02-05T06:34:44.892790+0000","last_peered":"2024-02-05T06:34:44.892790+0000","last_clean":"2024-02-05T06:34:44.892790+0000","last_became_active":"2024-02-05T06:33:29.552415+0000","last_became_peered":"2024-02-05T06:33:29.552415+0000","last_unstale":"2024-02-05T06:34:44.892790+0000","last_undegraded":"2024-02-05T06:34:44.892790+0000","last_fullsized":"2024-02-05T06:34:44.892790+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:56.521015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"73'11","reported_seq":103,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:50.592882+0000","last_change":"2024-02-05T06:34:50.592920+0000","last_active":"2024-02-05T06:34:50.592882+0000","last_peered":"2024-02-05T06:34:50.592882+0000","last_clean":"2024-02-05T06:34:50.592882+0000","last_became_active":"2024-02-05T06:33:55.885523+0000","last_became_peered":"2024-02-05T06:33:55.885523+0000","last_unstale":"2024-02-05T06:34:50.592882+0000","last_undegraded":"2024-02-05T06:34:50.592882+0000","last_fullsized":"2024-02-05T06:34:50.592882+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-02-05T06:34:50.592793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:50.592793+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:56.891427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:49.298317+0000","last_change":"2024-02-05T06:34:49.298355+0000","last_active":"2024-02-05T06:34:49.298317+0000","last_peered":"2024-02-05T06:34:49.298317+0000","last_clean":"2024-02-05T06:34:49.298317+0000","last_became_active":"2024-02-05T06:33:35.598593+0000","last_became_peered":"2024-02-05T06:33:35.598593+0000","last_unstale":"2024-02-05T06:34:49.298317+0000","last_undegraded":"2024-02-05T06:34:49.298317+0000","last_fullsized":"2024-02-05T06:34:49.298317+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:49.298233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:49.298233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:07.506646+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":54,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.890955+0000","last_change":"2024-02-05T06:33:33.586971+0000","last_active":"2024-02-05T06:34:44.890955+0000","last_peered":"2024-02-05T06:34:44.890955+0000","last_clean":"2024-02-05T06:34:44.890955+0000","last_became_active":"2024-02-05T06:33:33.586598+0000","last_became_peered":"2024-02-05T06:33:33.586598+0000","last_unstale":"2024-02-05T06:34:44.890955+0000","last_undegraded":"2024-02-05T06:34:44.890955+0000","last_fullsized":"2024-02-05T06:34:44.890955+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:49.228193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":70,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:57.290996+0000","last_change":"2024-02-05T06:34:57.291015+0000","last_active":"2024-02-05T06:34:57.290996+0000","last_peered":"2024-02-05T06:34:57.290996+0000","last_clean":"2024-02-05T06:34:57.290996+0000","last_became_active":"2024-02-05T06:33:29.550612+0000","last_became_peered":"2024-02-05T06:33:29.550612+0000","last_unstale":"2024-02-05T06:34:57.290996+0000","last_undegraded":"2024-02-05T06:34:57.290996+0000","last_fullsized":"2024-02-05T06:34:57.290996+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:57.290944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:57.290944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:23.780005+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"73'17","reported_seq":86,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.893580+0000","last_change":"2024-02-05T06:33:31.566941+0000","last_active":"2024-02-05T06:34:44.893580+0000","last_peered":"2024-02-05T06:34:44.893580+0000","last_clean":"2024-02-05T06:34:44.893580+0000","last_became_active":"2024-02-05T06:33:31.566576+0000","last_became_peered":"2024-02-05T06:33:31.566576+0000","last_unstale":"2024-02-05T06:34:44.893580+0000","last_undegraded":"2024-02-05T06:34:44.893580+0000","last_fullsized":"2024-02-05T06:34:44.893580+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:00.400128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":50,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.893991+0000","last_change":"2024-02-05T06:33:35.597953+0000","last_active":"2024-02-05T06:34:44.893991+0000","last_peered":"2024-02-05T06:34:44.893991+0000","last_clean":"2024-02-05T06:34:44.893991+0000","last_became_active":"2024-02-05T06:33:35.597460+0000","last_became_peered":"2024-02-05T06:33:35.597460+0000","last_unstale":"2024-02-05T06:34:44.893991+0000","last_undegraded":"2024-02-05T06:34:44.893991+0000","last_fullsized":"2024-02-05T06:34:44.893991+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:00.813671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"92'9","reported_seq":89,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:54.609831+0000","last_change":"2024-02-05T06:34:52.283858+0000","last_active":"2024-02-05T06:34:54.609831+0000","last_peered":"2024-02-05T06:34:54.609831+0000","last_clean":"2024-02-05T06:34:54.609831+0000","last_became_active":"2024-02-05T06:33:33.588478+0000","last_became_peered":"2024-02-05T06:33:33.588478+0000","last_unstale":"2024-02-05T06:34:54.609831+0000","last_undegraded":"2024-02-05T06:34:54.609831+0000","last_fullsized":"2024-02-05T06:34:54.609831+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'9","last_scrub_stamp":"2024-02-05T06:34:52.283780+0000","last_deep_scrub":"92'9","last_deep_scrub_stamp":"2024-02-05T06:34:52.283780+0000","last_clean_scrub_stamp":"2024-02-05T06:34:52.283780+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:54.817931+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":71,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:45.097232+0000","last_change":"2024-02-05T06:34:29.822718+0000","last_active":"2024-02-05T06:34:45.097232+0000","last_peered":"2024-02-05T06:34:45.097232+0000","last_clean":"2024-02-05T06:34:45.097232+0000","last_became_active":"2024-02-05T06:33:29.545073+0000","last_became_peered":"2024-02-05T06:33:29.545073+0000","last_unstale":"2024-02-05T06:34:45.097232+0000","last_undegraded":"2024-02-05T06:34:45.097232+0000","last_fullsized":"2024-02-05T06:34:45.097232+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:29.822634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:29.822634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:37.035501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"73'10","reported_seq":73,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-02-05T06:34:44.893424+0000","last_change":"2024-02-05T06:33:31.568457+0000","last_active":"2024-02-05T06:34:44.893424+0000","last_peered":"2024-02-05T06:34:44.893424+0000","last_clean":"2024-02-05T06:34:44.893424+0000","last_became_active":"2024-02-05T06:33:31.568074+0000","last_became_peered":"2024-02-05T06:33:31.568074+0000","last_unstale":"2024-02-05T06:34:44.893424+0000","last_undegraded":"2024-02-05T06:34:44.893424+0000","last_fullsized":"2024-02-05T06:34:44.893424+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:34:52.367398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":422,"ondisk_log_size":422,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":79,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103190,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32600,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712552,"statfs":{"total":95995035648,"available":95961653248,"internally_reserved":0,"allocated":1458176,"data_stored":970343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.58199999999999996}]},{"osd":1,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.501},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.51100000000000001}]},{"osd":2,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.53800000000000003}]},{"osd":3,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.56100000000000005}]},{"osd":4,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.29999999999999999}]},{"osd":5,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.376},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.47199999999999998}]},{"osd":6,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.441}]}]},{"osd":6,"up_from":49,"seq":210453397532,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32576,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712576,"statfs":{"total":95995035648,"available":95961677824,"internally_reserved":0,"allocated":1433600,"data_stored":968894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:34:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.42999999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.752,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.44},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.35099999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.71299999999999997,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.54300000000000004}]},{"osd":1,"last update":"Mon Feb 5 06:34:51 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.51900000000000002,"15min":0.52200000000000002},"min":{"1min":0.378,"5min":0.309,"15min":0.309},"max":{"1min":0.746,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.501},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57999999999999996,"15min":0.58399999999999996},"min":{"1min":0.39100000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.76000000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.67000000000000004}]},{"osd":2,"last update":"Mon Feb 5 06:34:51 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.58099999999999996,"15min":0.58199999999999996},"min":{"1min":0.36399999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.77400000000000002,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.55100000000000005,"15min":0.55200000000000005},"min":{"1min":0.38300000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.65600000000000003}]},{"osd":3,"last update":"Mon Feb 5 06:34:51 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.55100000000000005,"15min":0.54700000000000004},"min":{"1min":0.42499999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.64500000000000002}]},{"osd":4,"last update":"Mon Feb 5 06:34:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.626,"5min":1.355,"15min":1.355},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.432,"15min":0.42199999999999999},"min":{"1min":0.28499999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.70999999999999996,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.38300000000000001}]},{"osd":5,"last update":"Mon Feb 5 06:34:51 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.46100000000000002,"15min":0.45600000000000002},"min":{"1min":0.33000000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.45900000000000002,"15min":0.45100000000000001},"min":{"1min":0.38600000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.65800000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.45100000000000001}]},{"osd":7,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.40300000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":884736,"data_stored":380510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.36699999999999999,"15min":0.34000000000000002},"min":{"1min":0.26800000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.77600000000000002,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.38700000000000001,"15min":0.34999999999999998},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52800000000000002,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.25700000000000001}]},{"osd":2,"last update":"Mon Feb 5 06:34:13 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.39900000000000002,"15min":0.39200000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.59999999999999998,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.38700000000000001,"15min":0.377},"min":{"1min":0.26900000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.35599999999999998}]},{"osd":3,"last update":"Mon Feb 5 06:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.42999999999999999,"15min":0.42199999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.31},{"interface":"front","average":{"1min":0.503,"5min":0.48799999999999999,"15min":0.47799999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.83399999999999996,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.39700000000000002}]},{"osd":4,"last update":"Mon Feb 5 06:34:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56699999999999995,"15min":0.56100000000000005},"min":{"1min":0.434,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.498},{"interface":"front","average":{"1min":0.62,"5min":0.58399999999999996,"15min":0.57799999999999996},"min":{"1min":0.39400000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.46899999999999997}]},{"osd":5,"last update":"Mon Feb 5 06:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.92800000000000005,"15min":0.98499999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86499999999999999,"5min":8.2509999999999994,"15min":8.2509999999999994},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.624,"15min":0.63400000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.41299999999999998}]},{"osd":6,"last update":"Mon Feb 5 06:34:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.63800000000000001,"15min":0.64800000000000002},"min":{"1min":0.42099999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.78700000000000003,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.67100000000000004,"15min":0.67800000000000005},"min":{"1min":0.51000000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.80600000000000005,"5min":1.014,"15min":1.014},"last":0.53400000000000003}]},{"osd":7,"last update":"Mon Feb 5 06:34:16 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.54500000000000004},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":8.2219999999999995,"5min":8.2219999999999995,"15min":8.2219999999999995},"last":0.42799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738427,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32608,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712544,"statfs":{"total":95995035648,"available":95961645056,"internally_reserved":0,"allocated":1466368,"data_stored":970099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 06:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.38900000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.188,"15min":0.188},"max":{"1min":4.2729999999999997,"5min":4.2729999999999997,"15min":4.2729999999999997},"last":0.438},{"interface":"front","average":{"1min":0.377,"5min":0.36599999999999999,"15min":0.33600000000000002},"min":{"1min":0.26800000000000002,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.629,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.40300000000000002}]},{"osd":2,"last update":"Mon Feb 5 06:34:11 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.435,"15min":0.41299999999999998},"min":{"1min":0.315,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.42599999999999999},"min":{"1min":0.28999999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.33400000000000002}]},{"osd":3,"last update":"Mon Feb 5 06:34:37 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.47499999999999998,"15min":0.46200000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":4.3289999999999997,"5min":4.3289999999999997,"15min":4.3289999999999997},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45100000000000001},"min":{"1min":0.28499999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.68500000000000005,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.27900000000000003}]},{"osd":4,"last update":"Mon Feb 5 06:34:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.59799999999999998,"15min":0.59499999999999997},"min":{"1min":0.441,"5min":0.372,"15min":0.372},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.60899999999999999,"15min":0.60499999999999998},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.67400000000000004}]},{"osd":5,"last update":"Mon Feb 5 06:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.628,"15min":0.63600000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.73399999999999999,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.64200000000000002,"15min":0.64700000000000002},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.77900000000000003,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.50700000000000001}]},{"osd":6,"last update":"Mon Feb 5 06:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.68100000000000005,"15min":0.65200000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":5.2220000000000004,"5min":5.2220000000000004,"15min":5.2220000000000004},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.624,"5min":0.64800000000000002,"15min":0.65200000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.47999999999999998}]},{"osd":7,"last update":"Mon Feb 5 06:34:16 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":5.2670000000000003,"5min":5.2670000000000003,"15min":5.2670000000000003},"last":0.73399999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411376,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32008,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713144,"statfs":{"total":95995035648,"available":95962259456,"internally_reserved":0,"allocated":851968,"data_stored":379844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:34:15 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.38100000000000001,"15min":0.36599999999999999},"min":{"1min":0.27000000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.59999999999999998,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.41199999999999998,"15min":0.38400000000000001},"min":{"1min":0.32000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.58099999999999996,"5min":0.752,"15min":0.752},"last":0.47099999999999997}]},{"osd":1,"last update":"Mon Feb 5 06:34:15 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39900000000000002,"15min":0.38900000000000001},"min":{"1min":0.26200000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.56000000000000005,"5min":0.626,"15min":0.626},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.44800000000000001,"15min":0.42899999999999999},"min":{"1min":0.33600000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.68300000000000005,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.57999999999999996}]},{"osd":3,"last update":"Mon Feb 5 06:34:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.47999999999999998,"15min":0.46999999999999997},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72499999999999998,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.47999999999999998,"15min":0.48199999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.66200000000000003,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.50800000000000001}]},{"osd":4,"last update":"Mon Feb 5 06:34:00 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.65000000000000002,"15min":0.64700000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0009999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.495,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.90000000000000002,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.64900000000000002}]},{"osd":5,"last update":"Mon Feb 5 06:34:30 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.69199999999999995,"15min":0.70299999999999996},"min":{"1min":0.47199999999999998,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.89600000000000002,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.66600000000000004,"15min":0.68000000000000005},"min":{"1min":0.42399999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.85299999999999998,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.59599999999999997}]},{"osd":6,"last update":"Mon Feb 5 06:34:54 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":1.51,"15min":1.6499999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.97999999999999998,"5min":17.143999999999998,"15min":17.143999999999998},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.69699999999999995,"15min":0.70099999999999996},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.96099999999999997,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.68799999999999994}]},{"osd":7,"last update":"Mon Feb 5 06:34:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.85899999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149739,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":876544,"data_stored":379417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:34:44 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.38600000000000001,"15min":0.39300000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.52000000000000002,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.44500000000000001,"15min":0.45400000000000001},"min":{"1min":0.316,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.55000000000000004,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.39900000000000002}]},{"osd":1,"last update":"Mon Feb 5 06:34:44 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.45800000000000002,"15min":0.47199999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.57099999999999995,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.44500000000000001,"15min":0.45300000000000001},"min":{"1min":0.27700000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.57599999999999996,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.246}]},{"osd":2,"last update":"Mon Feb 5 06:34:44 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.48099999999999998,"15min":0.495},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.64600000000000002,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.46000000000000002,"15min":0.46899999999999997},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.64800000000000002,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.36299999999999999}]},{"osd":4,"last update":"Mon Feb 5 06:34:05 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.59899999999999998,"15min":0.61499999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.627,"5min":1.008,"15min":1.008},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.59099999999999997,"15min":0.60399999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.67900000000000005,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.46700000000000003}]},{"osd":5,"last update":"Mon Feb 5 06:34:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.60099999999999998,"15min":0.60699999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.73999999999999999,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.61699999999999999,"15min":0.627},"min":{"1min":0.44,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.748,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.53400000000000003}]},{"osd":6,"last update":"Mon Feb 5 06:34:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.497},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63100000000000001,"15min":0.63500000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.88700000000000001,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.41599999999999998}]},{"osd":7,"last update":"Mon Feb 5 06:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.45800000000000002}]}]},{"osd":4,"up_from":31,"seq":133143986213,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27868,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717284,"statfs":{"total":95995035648,"available":95966498816,"internally_reserved":0,"allocated":872448,"data_stored":379529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:34:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.57899999999999996,"15min":0.58899999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.71699999999999997,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.54300000000000004,"15min":0.54100000000000004},"min":{"1min":0.42099999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.70399999999999996,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.503}]},{"osd":1,"last update":"Mon Feb 5 06:34:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.55900000000000005,"15min":0.56200000000000006},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.69599999999999995,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.58999999999999997,"15min":0.59999999999999998},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.68600000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.54500000000000004}]},{"osd":2,"last update":"Mon Feb 5 06:34:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.76900000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.58099999999999996,"15min":0.58699999999999997},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.80600000000000005,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.55500000000000005}]},{"osd":3,"last update":"Mon Feb 5 06:34:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59799999999999998,"15min":0.60099999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.375,"5min":1.375,"15min":1.375},"last":0.62},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.58099999999999996,"15min":0.58699999999999997},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.746,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.49099999999999999}]},{"osd":5,"last update":"Mon Feb 5 06:34:28 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.40999999999999998,"15min":0.40200000000000002},"min":{"1min":0.27500000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.442,"15min":0.439},"min":{"1min":0.317,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.379}]},{"osd":6,"last update":"Mon Feb 5 06:34:52 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.504,"15min":0.49299999999999999},"min":{"1min":0.371,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.79600000000000004,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.48899999999999999,"15min":0.47499999999999998},"min":{"1min":0.32300000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.40200000000000002}]},{"osd":7,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.47199999999999998}]}]},{"osd":5,"up_from":40,"seq":171798691872,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":868352,"data_stored":380096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.53100000000000003,"15min":0.53400000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.47499999999999998},{"interface":"front","average":{"1min":1.45,"5min":0.72999999999999998,"15min":0.60999999999999999},"min":{"1min":0.35399999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":17.509,"5min":17.509,"15min":17.509},"last":0.36899999999999999}]},{"osd":1,"last update":"Mon Feb 5 06:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.56100000000000005,"15min":0.56200000000000006},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.752,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.54100000000000004,"15min":0.54400000000000004},"min":{"1min":0.37,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.44400000000000001}]},{"osd":2,"last update":"Mon Feb 5 06:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54800000000000004,"15min":0.55000000000000004},"min":{"1min":0.41499999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.435},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.53700000000000003,"15min":0.53600000000000003},"min":{"1min":0.40300000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.70299999999999996,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.41999999999999998}]},{"osd":3,"last update":"Mon Feb 5 06:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.55300000000000005,"15min":0.55900000000000005},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.66900000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.57699999999999996,"15min":0.58099999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.72599999999999998,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.54600000000000004}]},{"osd":4,"last update":"Mon Feb 5 06:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.39500000000000002,"15min":0.40100000000000002},"min":{"1min":0.29799999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.45300000000000001,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.32300000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.53600000000000003,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.38600000000000001}]},{"osd":6,"last update":"Mon Feb 5 06:34:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.46000000000000002,"15min":0.45000000000000001},"min":{"1min":0.33900000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.47899999999999998,"15min":0.47099999999999997},"min":{"1min":0.373,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.50700000000000001}]},{"osd":7,"last update":"Mon Feb 5 06:34:16 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.314},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.48899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T06:35:04.150 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T06:35:04.399 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:35:04.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:04 smithi167 ceph-mon[32774]: 5.f scrub starts 2024-02-05T06:35:04.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:04 smithi167 ceph-mon[32774]: 5.f scrub ok 2024-02-05T06:35:04.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:04 smithi167 ceph-mon[32774]: 4.0 scrub starts 2024-02-05T06:35:04.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:04 smithi167 ceph-mon[32774]: 4.0 scrub ok 2024-02-05T06:35:04.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:04 smithi167 ceph-mon[32774]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:35:04.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:04 smithi167 ceph-mon[32774]: 4.c deep-scrub starts 2024-02-05T06:35:04.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:04 smithi167 ceph-mon[32774]: 4.c deep-scrub ok 2024-02-05T06:35:05.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:04 smithi028 ceph-mon[37984]: 5.f scrub starts 2024-02-05T06:35:05.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:04 smithi028 ceph-mon[37984]: 5.f scrub ok 2024-02-05T06:35:05.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:04 smithi028 ceph-mon[37984]: 4.0 scrub starts 2024-02-05T06:35:05.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:04 smithi028 ceph-mon[37984]: 4.0 scrub ok 2024-02-05T06:35:05.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:04 smithi028 ceph-mon[37984]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:35:05.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:04 smithi028 ceph-mon[37984]: 4.c deep-scrub starts 2024-02-05T06:35:05.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:04 smithi028 ceph-mon[37984]: 4.c deep-scrub ok 2024-02-05T06:35:05.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:04 smithi028 ceph-mon[29442]: 5.f scrub starts 2024-02-05T06:35:05.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:04 smithi028 ceph-mon[29442]: 5.f scrub ok 2024-02-05T06:35:05.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:04 smithi028 ceph-mon[29442]: 4.0 scrub starts 2024-02-05T06:35:05.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:04 smithi028 ceph-mon[29442]: 4.0 scrub ok 2024-02-05T06:35:05.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:04 smithi028 ceph-mon[29442]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:35:05.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:04 smithi028 ceph-mon[29442]: 4.c deep-scrub starts 2024-02-05T06:35:05.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:04 smithi028 ceph-mon[29442]: 4.c deep-scrub ok 2024-02-05T06:35:05.479 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 systemd[1]: Starting Ceph grafana.a for 9801e956-c3ef-11ee-95b6-87774f69a715... 2024-02-05T06:35:05.771 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 podman[63650]: 2024-02-05 06:35:05.400760596 +0000 UTC m=+0.017275802 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-02-05T06:35:05.771 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 podman[63650]: 2024-02-05 06:35:05.508880001 +0000 UTC m=+0.125395247 container create 5fe0a0c56ac4123c85388e99ddc8501f9940bbfd67dc30b0516653c75a70ede0 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, maintainer=Paul Cuzner , vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=239.1651231664, description=Ceph Grafana Container, io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, version=8.5, com.redhat.component=ubi8-container, distribution-scope=public, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, build-date=2022-04-29T11:48:11.055897, vcs-type=git) 2024-02-05T06:35:05.771 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 podman[63650]: 2024-02-05 06:35:05.769700304 +0000 UTC m=+0.386215507 container init 5fe0a0c56ac4123c85388e99ddc8501f9940bbfd67dc30b0516653c75a70ede0 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a, build-date=2022-04-29T11:48:11.055897, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, distribution-scope=public, maintainer=Paul Cuzner , vendor=Red Hat, Inc., description=Ceph Grafana Container, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, version=8.5, name=ubi8, io.buildah.version=1.32.0, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-02-05T06:35:06.021 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-mon[32774]: 5.c scrub starts 2024-02-05T06:35:06.021 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-mon[32774]: 5.c scrub ok 2024-02-05T06:35:06.021 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-mon[32774]: 6.1f scrub starts 2024-02-05T06:35:06.021 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-mon[32774]: 6.1f scrub ok 2024-02-05T06:35:06.021 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-mon[32774]: 5.1c scrub starts 2024-02-05T06:35:06.021 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-mon[32774]: 5.1c scrub ok 2024-02-05T06:35:06.022 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 podman[63650]: 2024-02-05 06:35:05.773098939 +0000 UTC m=+0.389614144 container start 5fe0a0c56ac4123c85388e99ddc8501f9940bbfd67dc30b0516653c75a70ede0 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.32.0, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, description=Ceph Grafana Container, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, release=239.1651231664, name=ubi8, distribution-scope=public, maintainer=Paul Cuzner , vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-04-29T11:48:11.055897, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=) 2024-02-05T06:35:06.022 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 bash[63650]: 5fe0a0c56ac4123c85388e99ddc8501f9940bbfd67dc30b0516653c75a70ede0 2024-02-05T06:35:06.022 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 systemd[1]: Started Ceph grafana.a for 9801e956-c3ef-11ee-95b6-87774f69a715. 2024-02-05T06:35:06.022 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=settings t=2024-02-05T06:35:05.925225378Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-02-05T06:35:06.022 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=settings t=2024-02-05T06:35:05.925419836Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-02-05T06:35:06.022 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=settings t=2024-02-05T06:35:05.925438962Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-02-05T06:35:06.023 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=settings t=2024-02-05T06:35:05.925446161Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-02-05T06:35:06.023 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=settings t=2024-02-05T06:35:05.925452217Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-02-05T06:35:06.023 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=settings t=2024-02-05T06:35:05.925458441Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-02-05T06:35:06.023 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=settings t=2024-02-05T06:35:05.925464141Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-02-05T06:35:06.023 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=settings t=2024-02-05T06:35:05.925469997Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-02-05T06:35:06.023 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=settings t=2024-02-05T06:35:05.925475965Z level=info msg="Path Home" path=/usr/share/grafana 2024-02-05T06:35:06.023 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=settings t=2024-02-05T06:35:05.92548167Z level=info msg="Path Data" path=/var/lib/grafana 2024-02-05T06:35:06.023 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=settings t=2024-02-05T06:35:05.925487101Z level=info msg="Path Logs" path=/var/log/grafana 2024-02-05T06:35:06.023 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=settings t=2024-02-05T06:35:05.925493075Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-02-05T06:35:06.023 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=settings t=2024-02-05T06:35:05.925498803Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-02-05T06:35:06.023 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=settings t=2024-02-05T06:35:05.925504614Z level=info msg="App mode production" 2024-02-05T06:35:06.023 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=sqlstore t=2024-02-05T06:35:05.925548619Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-02-05T06:35:06.023 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=sqlstore t=2024-02-05T06:35:05.925564195Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-02-05T06:35:06.023 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:05.926402571Z level=info msg="Starting DB migrations" 2024-02-05T06:35:06.024 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:05.92647792Z level=info msg="Executing migration" id="create migration_log table" 2024-02-05T06:35:06.024 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:05.926987402Z level=info msg="Executing migration" id="create user table" 2024-02-05T06:35:06.024 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:05.927402583Z level=info msg="Executing migration" id="add unique index user.login" 2024-02-05T06:35:06.024 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:05.930466805Z level=info msg="Executing migration" id="add unique index user.email" 2024-02-05T06:35:06.024 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:05.93341933Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-02-05T06:35:06.024 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:05.936058022Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-02-05T06:35:06.024 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:05.938746785Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-02-05T06:35:06.024 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:05.941581376Z level=info msg="Executing migration" id="create user table v2" 2024-02-05T06:35:06.024 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:05.944438687Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-02-05T06:35:06.024 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:05.947354786Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-02-05T06:35:06.024 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:05.95025202Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-02-05T06:35:06.024 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:05.952711408Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-02-05T06:35:06.024 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:05.955649544Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-02-05T06:35:06.025 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:05.958455494Z level=info msg="Executing migration" id="Update user table charset" 2024-02-05T06:35:06.025 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:05.958936185Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-02-05T06:35:06.025 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:05.961694808Z level=info msg="Executing migration" id="Add missing user data" 2024-02-05T06:35:06.025 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:05.964542003Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-02-05T06:35:06.025 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:05.967431972Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-02-05T06:35:06.025 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:05.967950601Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-02-05T06:35:06.025 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:05.970824036Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-02-05T06:35:06.025 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:05.973651147Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-02-05T06:35:06.025 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:05.976601832Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-02-05T06:35:06.025 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:05.977109638Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-02-05T06:35:06.025 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:05.979926849Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-02-05T06:35:06.025 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:05.982359326Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-02-05T06:35:06.025 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:05.98508959Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-02-05T06:35:06.026 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:05.987763112Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-02-05T06:35:06.026 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:05.99047595Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-02-05T06:35:06.026 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:05.992971757Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-02-05T06:35:06.026 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:05.995643892Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-02-05T06:35:06.026 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:05.998374012Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-02-05T06:35:06.026 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.001049103Z level=info msg="Executing migration" id="create temp_user v2" 2024-02-05T06:35:06.026 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.003763883Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-02-05T06:35:06.026 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.00629148Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-02-05T06:35:06.026 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.008966435Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-02-05T06:35:06.026 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.011881152Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-02-05T06:35:06.026 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.014461047Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-02-05T06:35:06.026 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.016807737Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-02-05T06:35:06.026 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.017742282Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-02-05T06:35:06.026 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.01830757Z level=info msg="Executing migration" id="create star table" 2024-02-05T06:35:06.026 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.019080217Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-02-05T06:35:06.027 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.019549554Z level=info msg="Executing migration" id="create org table v1" 2024-02-05T06:35:06.027 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.019992773Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-02-05T06:35:06.027 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.020850111Z level=info msg="Executing migration" id="create org_user table v1" 2024-02-05T06:35:06.170 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:05 smithi028 ceph-mon[29442]: 5.c scrub starts 2024-02-05T06:35:06.170 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:05 smithi028 ceph-mon[29442]: 5.c scrub ok 2024-02-05T06:35:06.170 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:05 smithi028 ceph-mon[29442]: 6.1f scrub starts 2024-02-05T06:35:06.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:05 smithi028 ceph-mon[29442]: 6.1f scrub ok 2024-02-05T06:35:06.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:05 smithi028 ceph-mon[29442]: 5.1c scrub starts 2024-02-05T06:35:06.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:05 smithi028 ceph-mon[29442]: 5.1c scrub ok 2024-02-05T06:35:06.171 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:05 smithi028 ceph-mon[37984]: 5.c scrub starts 2024-02-05T06:35:06.171 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:05 smithi028 ceph-mon[37984]: 5.c scrub ok 2024-02-05T06:35:06.171 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:05 smithi028 ceph-mon[37984]: 6.1f scrub starts 2024-02-05T06:35:06.171 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:05 smithi028 ceph-mon[37984]: 6.1f scrub ok 2024-02-05T06:35:06.171 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:05 smithi028 ceph-mon[37984]: 5.1c scrub starts 2024-02-05T06:35:06.171 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:05 smithi028 ceph-mon[37984]: 5.1c scrub ok 2024-02-05T06:35:06.274 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.021677179Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-02-05T06:35:06.274 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.022496236Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-02-05T06:35:06.274 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.023262934Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-02-05T06:35:06.274 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.023651736Z level=info msg="Executing migration" id="Update org table charset" 2024-02-05T06:35:06.275 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.024501754Z level=info msg="Executing migration" id="Update org_user table charset" 2024-02-05T06:35:06.275 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.02479218Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-02-05T06:35:06.275 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.025396512Z level=info msg="Executing migration" id="create dashboard table" 2024-02-05T06:35:06.275 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.026201995Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-02-05T06:35:06.275 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.026590064Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-02-05T06:35:06.275 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.027468721Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-02-05T06:35:06.275 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.028303278Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-02-05T06:35:06.275 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.029061814Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-02-05T06:35:06.275 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.031732912Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-02-05T06:35:06.275 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.034449079Z level=info msg="Executing migration" id="create dashboard v2" 2024-02-05T06:35:06.275 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.037377417Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-02-05T06:35:06.275 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.040249766Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-02-05T06:35:06.276 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.040742491Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-02-05T06:35:06.276 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.043614558Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-02-05T06:35:06.276 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.044054284Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-02-05T06:35:06.276 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.044348607Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-02-05T06:35:06.276 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.046997668Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-02-05T06:35:06.276 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.047715057Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-02-05T06:35:06.276 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.049772091Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-02-05T06:35:06.276 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.052449411Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-02-05T06:35:06.276 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.05524423Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-02-05T06:35:06.276 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.055716336Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-02-05T06:35:06.276 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.058528271Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-02-05T06:35:06.276 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.061220061Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-02-05T06:35:06.276 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.061551235Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-02-05T06:35:06.276 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.062143609Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-02-05T06:35:06.277 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.064990505Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-02-05T06:35:06.277 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.067755609Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-02-05T06:35:06.277 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.070401021Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-02-05T06:35:06.277 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.073259236Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-02-05T06:35:06.277 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.076053516Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-02-05T06:35:06.277 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.078897865Z level=info msg="Executing migration" id="Update dashboard title length" 2024-02-05T06:35:06.277 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.081578028Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-02-05T06:35:06.277 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.082034679Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-02-05T06:35:06.278 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.084749928Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-02-05T06:35:06.278 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.087509426Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-02-05T06:35:06.278 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.090234837Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-02-05T06:35:06.278 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.090674463Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-02-05T06:35:06.279 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.091668717Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-02-05T06:35:06.279 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.09239979Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-02-05T06:35:06.279 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.093326111Z level=info msg="Executing migration" id="Add check_sum column" 2024-02-05T06:35:06.279 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.094008055Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-02-05T06:35:06.279 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.094803874Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-02-05T06:35:06.279 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.095625256Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-02-05T06:35:06.279 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.096501765Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-02-05T06:35:06.279 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.097169537Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-02-05T06:35:06.279 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.097762279Z level=info msg="Executing migration" id="create data_source table" 2024-02-05T06:35:06.279 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.098600976Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-02-05T06:35:06.279 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.099384064Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-02-05T06:35:06.279 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.099865891Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-02-05T06:35:06.280 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.100315796Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-02-05T06:35:06.280 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.10110251Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-02-05T06:35:06.281 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.102133946Z level=info msg="Executing migration" id="create data_source table v2" 2024-02-05T06:35:06.281 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.102553689Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-02-05T06:35:06.281 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.103422489Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-02-05T06:35:06.281 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.103909809Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-02-05T06:35:06.281 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.104734294Z level=info msg="Executing migration" id="Add column with_credentials" 2024-02-05T06:35:06.281 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.105642001Z level=info msg="Executing migration" id="Add secure json data column" 2024-02-05T06:35:06.281 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.106452498Z level=info msg="Executing migration" id="Update data_source table charset" 2024-02-05T06:35:06.281 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.106825632Z level=info msg="Executing migration" id="Update initial version to 1" 2024-02-05T06:35:06.281 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.107752848Z level=info msg="Executing migration" id="Add read_only data column" 2024-02-05T06:35:06.281 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.108607114Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-02-05T06:35:06.281 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.109397965Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-02-05T06:35:06.281 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.109752847Z level=info msg="Executing migration" id="Add uid column" 2024-02-05T06:35:06.281 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.110716664Z level=info msg="Executing migration" id="Update uid value" 2024-02-05T06:35:06.282 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.11146593Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-02-05T06:35:06.282 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.112284068Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-02-05T06:35:06.282 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.11276446Z level=info msg="Executing migration" id="create api_key table" 2024-02-05T06:35:06.282 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.113230258Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-02-05T06:35:06.282 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.114034193Z level=info msg="Executing migration" id="add index api_key.key" 2024-02-05T06:35:06.282 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.114918009Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-02-05T06:35:06.282 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.115753815Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-02-05T06:35:06.282 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.116227487Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-02-05T06:35:06.282 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.117069158Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-02-05T06:35:06.282 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.12000952Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-02-05T06:35:06.282 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.12301211Z level=info msg="Executing migration" id="create api_key table v2" 2024-02-05T06:35:06.282 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.125558034Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-02-05T06:35:06.282 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.128321463Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-02-05T06:35:06.282 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.13102839Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-02-05T06:35:06.283 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.133950242Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-02-05T06:35:06.283 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.137777597Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-02-05T06:35:06.283 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.13843547Z level=info msg="Executing migration" id="Update api_key table charset" 2024-02-05T06:35:06.283 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.141223568Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-02-05T06:35:06.283 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.141822947Z level=info msg="Executing migration" id="Add service account foreign key" 2024-02-05T06:35:06.283 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.144319995Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-02-05T06:35:06.283 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.147201828Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-02-05T06:35:06.283 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.150092004Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-02-05T06:35:06.283 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.153059233Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-02-05T06:35:06.283 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.155899382Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-02-05T06:35:06.283 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.158693166Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-02-05T06:35:06.283 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.161772191Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-02-05T06:35:06.283 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.164467161Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-02-05T06:35:06.283 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.167311022Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-02-05T06:35:06.284 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.17014077Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-02-05T06:35:06.284 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.172913048Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-02-05T06:35:06.284 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.175507893Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-02-05T06:35:06.284 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.178358076Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-02-05T06:35:06.284 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.180953411Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-02-05T06:35:06.284 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.183754899Z level=info msg="Executing migration" id="create quota table v1" 2024-02-05T06:35:06.284 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.186575892Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-02-05T06:35:06.284 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.189212258Z level=info msg="Executing migration" id="Update quota table charset" 2024-02-05T06:35:06.284 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.191943902Z level=info msg="Executing migration" id="create plugin_setting table" 2024-02-05T06:35:06.284 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.192427952Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-02-05T06:35:06.284 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.195398028Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-02-05T06:35:06.284 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.198126887Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-02-05T06:35:06.284 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.200819017Z level=info msg="Executing migration" id="create session table" 2024-02-05T06:35:06.285 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.201391911Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-02-05T06:35:06.285 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.201719182Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-02-05T06:35:06.285 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.204021132Z level=info msg="Executing migration" id="create playlist table v2" 2024-02-05T06:35:06.285 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.204502864Z level=info msg="Executing migration" id="create playlist item table v2" 2024-02-05T06:35:06.285 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.207270061Z level=info msg="Executing migration" id="Update playlist table charset" 2024-02-05T06:35:06.285 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.209944255Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-02-05T06:35:06.285 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.210267384Z level=info msg="Executing migration" id="drop preferences table v2" 2024-02-05T06:35:06.285 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.210560064Z level=info msg="Executing migration" id="drop preferences table v3" 2024-02-05T06:35:06.285 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.210860576Z level=info msg="Executing migration" id="create preferences table v3" 2024-02-05T06:35:06.285 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.211304082Z level=info msg="Executing migration" id="Update preferences table charset" 2024-02-05T06:35:06.285 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.211630079Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-02-05T06:35:06.285 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.214028535Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-02-05T06:35:06.285 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.216691048Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-02-05T06:35:06.285 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.217384055Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-02-05T06:35:06.285 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.218258554Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-02-05T06:35:06.286 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.219026847Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-02-05T06:35:06.286 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.219495323Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-02-05T06:35:06.286 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.220523858Z level=info msg="Executing migration" id="create alert table v1" 2024-02-05T06:35:06.286 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.221233996Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-02-05T06:35:06.286 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.2220491Z level=info msg="Executing migration" id="add index alert state" 2024-02-05T06:35:06.286 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.222913712Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-02-05T06:35:06.286 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.223357182Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-02-05T06:35:06.286 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.223729375Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-02-05T06:35:06.286 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.224234959Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-02-05T06:35:06.286 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.224648396Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-02-05T06:35:06.286 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.226044914Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-02-05T06:35:06.286 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.226440366Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-02-05T06:35:06.286 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.226886531Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-02-05T06:35:06.286 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.227235355Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-02-05T06:35:06.287 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.227581288Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-02-05T06:35:06.287 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.227990611Z level=info msg="Executing migration" id="Add column is_default" 2024-02-05T06:35:06.287 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.22863264Z level=info msg="Executing migration" id="Add column frequency" 2024-02-05T06:35:06.287 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.229333072Z level=info msg="Executing migration" id="Add column send_reminder" 2024-02-05T06:35:06.287 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.230006553Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-02-05T06:35:06.287 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.230655342Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-02-05T06:35:06.287 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.23110702Z level=info msg="Executing migration" id="Update alert table charset" 2024-02-05T06:35:06.287 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.231393101Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-02-05T06:35:06.287 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.231651485Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-02-05T06:35:06.287 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.232048136Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-02-05T06:35:06.287 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.232483502Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-02-05T06:35:06.287 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.232852955Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-02-05T06:35:06.287 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.23325938Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-02-05T06:35:06.287 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.233720743Z level=info msg="Executing migration" id="Add for to alert table" 2024-02-05T06:35:06.288 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.234401838Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-02-05T06:35:06.288 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.235031814Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-02-05T06:35:06.288 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.235949408Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-02-05T06:35:06.288 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.236736192Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-02-05T06:35:06.288 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.237696309Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-02-05T06:35:06.288 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.23847004Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-02-05T06:35:06.288 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.239449427Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-02-05T06:35:06.288 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.240347846Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-02-05T06:35:06.288 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.241059886Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-02-05T06:35:06.288 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.241936515Z level=info msg="Executing migration" id="create annotation table v5" 2024-02-05T06:35:06.288 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.242757608Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-02-05T06:35:06.288 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.243581346Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-02-05T06:35:06.288 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.244606146Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-02-05T06:35:06.288 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.245502376Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-02-05T06:35:06.289 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.2462138Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-02-05T06:35:06.289 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.247093622Z level=info msg="Executing migration" id="Update annotation table charset" 2024-02-05T06:35:06.289 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.250040522Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-02-05T06:35:06.289 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.252896371Z level=info msg="Executing migration" id="Drop category_id index" 2024-02-05T06:35:06.289 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.255637682Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-02-05T06:35:06.289 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.258413209Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-02-05T06:35:06.289 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.261147969Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-02-05T06:35:06.289 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.263947023Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-02-05T06:35:06.289 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.266746951Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-02-05T06:35:06.289 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.269860348Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-02-05T06:35:06.289 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.272629543Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-02-05T06:35:06.289 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.275271789Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-02-05T06:35:06.429 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:35:06.431 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-05T06:35:06.525 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.278050095Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-02-05T06:35:06.525 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.280790733Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-02-05T06:35:06.525 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.283632106Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-02-05T06:35:06.525 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.286480847Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-02-05T06:35:06.526 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.289194884Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-02-05T06:35:06.526 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.292089676Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-02-05T06:35:06.526 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.295028933Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-02-05T06:35:06.526 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.297832539Z level=info msg="Executing migration" id="Add epoch_end column" 2024-02-05T06:35:06.526 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.300584047Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-02-05T06:35:06.526 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.303418802Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-02-05T06:35:06.526 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.306073356Z level=info msg="Executing migration" id="Move region to single row" 2024-02-05T06:35:06.526 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.308510338Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-02-05T06:35:06.526 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.31132713Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-02-05T06:35:06.526 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.314017421Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-02-05T06:35:06.526 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.316710592Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-02-05T06:35:06.526 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.319358219Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-02-05T06:35:06.526 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.321863812Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-02-05T06:35:06.526 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.324671381Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-02-05T06:35:06.527 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.327500798Z level=info msg="Executing migration" id="create test_data table" 2024-02-05T06:35:06.528 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.330150636Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-02-05T06:35:06.528 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.332691224Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-02-05T06:35:06.528 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.335250827Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-02-05T06:35:06.528 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.338058614Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-02-05T06:35:06.528 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.340942567Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-02-05T06:35:06.528 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.343590894Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-02-05T06:35:06.528 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.344366708Z level=info msg="Executing migration" id="create team table" 2024-02-05T06:35:06.528 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.345239275Z level=info msg="Executing migration" id="add index team.org_id" 2024-02-05T06:35:06.528 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.34603447Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-02-05T06:35:06.528 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.346952484Z level=info msg="Executing migration" id="create team member table" 2024-02-05T06:35:06.528 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.347720132Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-02-05T06:35:06.528 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.348717972Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-02-05T06:35:06.529 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.349468114Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-02-05T06:35:06.529 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.350342502Z level=info msg="Executing migration" id="Add column email to team table" 2024-02-05T06:35:06.529 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.351178615Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-02-05T06:35:06.529 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.352015995Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-02-05T06:35:06.529 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.352892804Z level=info msg="Executing migration" id="create dashboard acl table" 2024-02-05T06:35:06.529 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.35376863Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-02-05T06:35:06.529 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.354504532Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-02-05T06:35:06.529 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.355444181Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-02-05T06:35:06.529 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.35620276Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-02-05T06:35:06.530 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.357091076Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-02-05T06:35:06.530 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.357877937Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-02-05T06:35:06.530 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.358892864Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-02-05T06:35:06.530 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.359650948Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-02-05T06:35:06.530 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.36063077Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-02-05T06:35:06.530 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.361232048Z level=info msg="Executing migration" id="create tag table" 2024-02-05T06:35:06.530 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.362040301Z level=info msg="Executing migration" id="add index tag.key_value" 2024-02-05T06:35:06.530 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.362887882Z level=info msg="Executing migration" id="create login attempt table" 2024-02-05T06:35:06.530 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.363722082Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-02-05T06:35:06.530 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.364199078Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-02-05T06:35:06.531 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.365137847Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-02-05T06:35:06.531 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.367086926Z level=info msg="Executing migration" id="create login_attempt v2" 2024-02-05T06:35:06.531 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.367985024Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-02-05T06:35:06.531 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.368588337Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-02-05T06:35:06.531 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.36936634Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-02-05T06:35:06.531 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.369806328Z level=info msg="Executing migration" id="create user auth table" 2024-02-05T06:35:06.531 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.370345882Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-02-05T06:35:06.531 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.371183753Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-02-05T06:35:06.531 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.371565629Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-02-05T06:35:06.531 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.37238679Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-02-05T06:35:06.531 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.375270179Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-02-05T06:35:06.531 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.378188937Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-02-05T06:35:06.531 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.379268661Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-02-05T06:35:06.531 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.382056997Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-02-05T06:35:06.532 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.384886513Z level=info msg="Executing migration" id="create server_lock table" 2024-02-05T06:35:06.532 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.387618394Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-02-05T06:35:06.532 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.388245714Z level=info msg="Executing migration" id="create user auth token table" 2024-02-05T06:35:06.532 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.391016649Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-02-05T06:35:06.532 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.393815341Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-02-05T06:35:06.532 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.396581062Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-02-05T06:35:06.532 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.397041554Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-02-05T06:35:06.532 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.400078185Z level=info msg="Executing migration" id="create cache_data table" 2024-02-05T06:35:06.532 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.402554548Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-02-05T06:35:06.532 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.40522005Z level=info msg="Executing migration" id="create short_url table v1" 2024-02-05T06:35:06.533 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.405751587Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-02-05T06:35:06.533 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.408700795Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-02-05T06:35:06.533 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.411550998Z level=info msg="Executing migration" id="delete alert_definition table" 2024-02-05T06:35:06.533 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.414366114Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-02-05T06:35:06.533 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.414869944Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-02-05T06:35:06.533 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.417745556Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-02-05T06:35:06.533 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.420430481Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-02-05T06:35:06.533 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.422656257Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-02-05T06:35:06.533 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.423099542Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-02-05T06:35:06.533 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.425766685Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-02-05T06:35:06.533 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.428589407Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-02-05T06:35:06.533 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.4312445Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-02-05T06:35:06.533 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.432097796Z level=info msg="Executing migration" id="drop alert_definition table" 2024-02-05T06:35:06.533 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.434904944Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-02-05T06:35:06.534 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.437600544Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-02-05T06:35:06.534 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.440239437Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-02-05T06:35:06.534 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.440691586Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-02-05T06:35:06.534 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.443611101Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-02-05T06:35:06.534 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.446256267Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-02-05T06:35:06.534 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.448953188Z level=info msg="Executing migration" id="create alert_instance table" 2024-02-05T06:35:06.534 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.44950138Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-02-05T06:35:06.534 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.450275487Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-02-05T06:35:06.534 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.45072826Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-02-05T06:35:06.535 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.451520654Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-02-05T06:35:06.535 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.452366204Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-02-05T06:35:06.535 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.45278747Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-02-05T06:35:06.535 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.456126773Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-02-05T06:35:06.535 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.459433618Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-02-05T06:35:06.535 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.460271595Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-02-05T06:35:06.535 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.460705142Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-02-05T06:35:06.535 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.461579531Z level=info msg="Executing migration" id="create alert_rule table" 2024-02-05T06:35:06.535 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.462012474Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-02-05T06:35:06.535 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.462439404Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-02-05T06:35:06.536 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.462874422Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-02-05T06:35:06.536 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.463300722Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-02-05T06:35:06.536 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.463564638Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-02-05T06:35:06.536 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.46440466Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-02-05T06:35:06.536 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.465227335Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-02-05T06:35:06.536 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.466035773Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-02-05T06:35:06.536 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.466441996Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-02-05T06:35:06.536 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.466855862Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-02-05T06:35:06.537 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.467678888Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-02-05T06:35:06.537 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.468515795Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-02-05T06:35:06.537 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.468955177Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-02-05T06:35:06.537 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.469769992Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-02-05T06:35:06.537 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.470602733Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-02-05T06:35:06.537 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.47089497Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-02-05T06:35:06.537 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.471350098Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-02-05T06:35:06.537 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.47177777Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-02-05T06:35:06.537 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.472212882Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-02-05T06:35:06.537 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.472495715Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-02-05T06:35:06.537 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.473392577Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-02-05T06:35:06.537 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.474262696Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-02-05T06:35:06.537 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.475092248Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-02-05T06:35:06.537 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.475959265Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-02-05T06:35:06.538 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.476833474Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-02-05T06:35:06.539 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.477144042Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-02-05T06:35:06.539 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.477549171Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-02-05T06:35:06.539 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.478426667Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-02-05T06:35:06.539 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.478745602Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-02-05T06:35:06.539 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.47964844Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-02-05T06:35:06.539 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.480520387Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-02-05T06:35:06.539 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.481402671Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-02-05T06:35:06.539 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.482293557Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-02-05T06:35:06.539 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.482756481Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-02-05T06:35:06.539 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.483624547Z level=info msg="Executing migration" id="create provenance_type table" 2024-02-05T06:35:06.539 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.484478224Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-02-05T06:35:06.540 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.485321724Z level=info msg="Executing migration" id="create alert_image table" 2024-02-05T06:35:06.540 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.485770824Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-02-05T06:35:06.540 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.486652378Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-02-05T06:35:06.540 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.487437439Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-02-05T06:35:06.540 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.487867829Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-02-05T06:35:06.540 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.488290106Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-02-05T06:35:06.540 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.488365211Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-02-05T06:35:06.540 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.491204941Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-02-05T06:35:06.540 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.491779875Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-02-05T06:35:06.540 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.493963111Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-02-05T06:35:06.540 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.494090585Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-02-05T06:35:06.541 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.494235066Z level=info msg="alerts found to migrate" alerts=0 2024-02-05T06:35:06.541 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.49684408Z level=info msg="Executing migration" id="create library_element table v1" 2024-02-05T06:35:06.541 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.497297046Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-02-05T06:35:06.541 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.500362266Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-02-05T06:35:06.541 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.503080337Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-02-05T06:35:06.541 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.506719665Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-02-05T06:35:06.541 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.507256828Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-02-05T06:35:06.541 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.510153456Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-02-05T06:35:06.541 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.512881604Z level=info msg="Executing migration" id="create data_keys table" 2024-02-05T06:35:06.541 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.515901057Z level=info msg="Executing migration" id="create secrets table" 2024-02-05T06:35:06.541 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.51639652Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-02-05T06:35:06.541 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.520533372Z level=info msg="Executing migration" id="add name column into data_keys" 2024-02-05T06:35:06.541 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.523545145Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-02-05T06:35:06.542 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.526345251Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-02-05T06:35:06.912 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.530687424Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-02-05T06:35:06.912 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.535010076Z level=info msg="Executing migration" id="create kv_store table v1" 2024-02-05T06:35:06.913 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.537808963Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-02-05T06:35:06.913 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.540548627Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-02-05T06:35:06.913 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.540997268Z level=info msg="Executing migration" id="create permission table" 2024-02-05T06:35:06.913 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.543921585Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-02-05T06:35:06.913 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.546594543Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-02-05T06:35:06.913 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.549250786Z level=info msg="Executing migration" id="create role table" 2024-02-05T06:35:06.913 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.549728208Z level=info msg="Executing migration" id="add column display_name" 2024-02-05T06:35:06.914 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.552470008Z level=info msg="Executing migration" id="add column group_name" 2024-02-05T06:35:06.914 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.555265063Z level=info msg="Executing migration" id="add index role.org_id" 2024-02-05T06:35:06.914 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.558049589Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-02-05T06:35:06.914 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.560557525Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-02-05T06:35:06.914 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.56338662Z level=info msg="Executing migration" id="create team role table" 2024-02-05T06:35:06.914 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.566131236Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-02-05T06:35:06.914 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.568747853Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-02-05T06:35:06.914 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.569265635Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-02-05T06:35:06.915 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.572061261Z level=info msg="Executing migration" id="create user role table" 2024-02-05T06:35:06.915 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.572775678Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-02-05T06:35:06.915 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.573746791Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-02-05T06:35:06.915 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.574272078Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-02-05T06:35:06.915 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.575122046Z level=info msg="Executing migration" id="create builtin role table" 2024-02-05T06:35:06.915 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.575962175Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-02-05T06:35:06.915 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.576951658Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-02-05T06:35:06.915 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.577527868Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-02-05T06:35:06.916 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.578485237Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-02-05T06:35:06.916 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.578996817Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-02-05T06:35:06.916 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.579493156Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-02-05T06:35:06.916 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.57991816Z level=info msg="Executing migration" id="add unique index role.uid" 2024-02-05T06:35:06.916 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.580390048Z level=info msg="Executing migration" id="create seed assignment table" 2024-02-05T06:35:06.916 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.581200093Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-02-05T06:35:06.916 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.581670368Z level=info msg="Executing migration" id="add column hidden to role table" 2024-02-05T06:35:06.916 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.582653133Z level=info msg="Executing migration" id="create query_history table v1" 2024-02-05T06:35:06.916 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.58310346Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-02-05T06:35:06.917 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.583587302Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-02-05T06:35:06.917 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.583884361Z level=info msg="Executing migration" id="teams permissions migration" 2024-02-05T06:35:06.917 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.584368235Z level=info msg="Executing migration" id="dashboard permissions" 2024-02-05T06:35:06.917 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.584835715Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-02-05T06:35:06.918 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.585281771Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-02-05T06:35:06.918 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.585590917Z level=info msg="Executing migration" id="alerting notification permissions" 2024-02-05T06:35:06.918 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.585955318Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-02-05T06:35:06.918 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.586424634Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-02-05T06:35:06.918 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.586923311Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-02-05T06:35:06.918 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.587940209Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-02-05T06:35:06.918 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.588283654Z level=info msg="Executing migration" id="create correlation table v1" 2024-02-05T06:35:06.918 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.588748963Z level=info msg="Executing migration" id="add index correlations.uid" 2024-02-05T06:35:06.918 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.589244239Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-02-05T06:35:06.918 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.589728781Z level=info msg="Executing migration" id="add correlation config column" 2024-02-05T06:35:06.918 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.590748936Z level=info msg="Executing migration" id="create entity_events table" 2024-02-05T06:35:06.918 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.591242419Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-02-05T06:35:06.918 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.592055325Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-02-05T06:35:06.919 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.592167973Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-02-05T06:35:06.919 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.592502649Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-05T06:35:06.919 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.592602746Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-05T06:35:06.919 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.592886188Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-02-05T06:35:06.919 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.593333396Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-02-05T06:35:06.919 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.59373175Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-02-05T06:35:06.919 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.594378779Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-05T06:35:06.919 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.594831962Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-02-05T06:35:06.919 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.595277616Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-02-05T06:35:06.919 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.595684095Z level=info msg="Executing migration" id="Drop public config table" 2024-02-05T06:35:06.919 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.596123497Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-02-05T06:35:06.919 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.596561291Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-02-05T06:35:06.920 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.597018169Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-02-05T06:35:06.920 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.597485666Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-02-05T06:35:06.920 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.597932831Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-02-05T06:35:06.920 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.600908463Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-02-05T06:35:06.920 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.601918436Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-02-05T06:35:06.920 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.602909842Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-02-05T06:35:06.920 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.603240743Z level=info msg="Executing migration" id="add share column" 2024-02-05T06:35:06.920 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.604334143Z level=info msg="Executing migration" id="create default alerting folders" 2024-02-05T06:35:06.920 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.604785023Z level=info msg="Executing migration" id="create file table" 2024-02-05T06:35:06.920 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.605252577Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-02-05T06:35:06.920 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.605700424Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-02-05T06:35:06.920 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.606194634Z level=info msg="Executing migration" id="create file_meta table" 2024-02-05T06:35:06.920 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.607093156Z level=info msg="Executing migration" id="file table idx: path key" 2024-02-05T06:35:06.921 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.607902597Z level=info msg="Executing migration" id="set path collation in file table" 2024-02-05T06:35:06.921 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.608730567Z level=info msg="Executing migration" id="managed permissions migration" 2024-02-05T06:35:06.921 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.609189939Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-02-05T06:35:06.921 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.609983484Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-02-05T06:35:06.921 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.610446776Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-02-05T06:35:06.921 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.611537938Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-02-05T06:35:06.921 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.612501469Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-02-05T06:35:06.921 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.613136767Z level=info msg="Executing migration" id="update group index for alert rules" 2024-02-05T06:35:06.921 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.613948362Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-02-05T06:35:06.921 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.614881835Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-02-05T06:35:06.921 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.615535882Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-02-05T06:35:06.921 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.616620909Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-02-05T06:35:06.921 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.617690071Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-02-05T06:35:06.921 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.618546082Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-02-05T06:35:06.922 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.628252367Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-02-05T06:35:06.922 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.628762342Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-02-05T06:35:06.922 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.629623292Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-02-05T06:35:06.922 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.633017816Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-02-05T06:35:06.922 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.633827536Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-02-05T06:35:06.922 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.634223501Z level=info msg="Executing migration" id="create folder table" 2024-02-05T06:35:06.922 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.635043943Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-02-05T06:35:06.922 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.635513129Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-02-05T06:35:06.922 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.635978828Z level=info msg="Executing migration" id="Update folder title length" 2024-02-05T06:35:06.922 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.63680435Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-02-05T06:35:06.922 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=migrator t=2024-02-05T06:35:06.637311688Z level=info msg="migrations completed" performed=485 skipped=0 duration=710.855933ms 2024-02-05T06:35:06.922 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=sqlstore t=2024-02-05T06:35:06.637680011Z level=info msg="Created default organization" 2024-02-05T06:35:06.922 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=plugin.loader t=2024-02-05T06:35:06.694530749Z level=info msg="Plugin registered" pluginID=input 2024-02-05T06:35:06.922 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=plugin.loader t=2024-02-05T06:35:06.722917807Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-02-05T06:35:06.922 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=plugin.loader t=2024-02-05T06:35:06.722956058Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-02-05T06:35:06.923 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=secrets t=2024-02-05T06:35:06.723004437Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-02-05T06:35:06.923 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=query_data t=2024-02-05T06:35:06.725168966Z level=info msg="Query Service initialization" 2024-02-05T06:35:06.923 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=live.push_http t=2024-02-05T06:35:06.727711383Z level=info msg="Live Push Gateway initialization" 2024-02-05T06:35:06.981 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:06 smithi028 ceph-mon[29442]: 3.f scrub starts 2024-02-05T06:35:06.981 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:06 smithi028 ceph-mon[29442]: 3.f scrub ok 2024-02-05T06:35:06.981 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:06 smithi028 ceph-mon[29442]: 4.7 scrub starts 2024-02-05T06:35:06.981 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:06 smithi028 ceph-mon[29442]: 4.7 scrub ok 2024-02-05T06:35:06.981 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:06 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:06.981 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:06 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:06.981 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:06 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:06.981 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:06 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:06.981 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:06 smithi028 ceph-mon[29442]: Deploying daemon node-exporter.a on smithi028 2024-02-05T06:35:06.981 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:06 smithi028 ceph-mon[29442]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:35:06.982 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:06 smithi028 ceph-mon[29442]: from='client.14784 v1:172.21.15.28:0/4189782476' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:35:06.982 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:06 smithi028 ceph-mon[37984]: 3.f scrub starts 2024-02-05T06:35:06.982 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:06 smithi028 ceph-mon[37984]: 3.f scrub ok 2024-02-05T06:35:06.982 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:06 smithi028 ceph-mon[37984]: 4.7 scrub starts 2024-02-05T06:35:06.982 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:06 smithi028 ceph-mon[37984]: 4.7 scrub ok 2024-02-05T06:35:06.982 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:06 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:06.982 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:06 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:06.982 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:06 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:06.982 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:06 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:06.982 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:06 smithi028 ceph-mon[37984]: Deploying daemon node-exporter.a on smithi028 2024-02-05T06:35:06.982 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:06 smithi028 ceph-mon[37984]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:35:06.982 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:06 smithi028 ceph-mon[37984]: from='client.14784 v1:172.21.15.28:0/4189782476' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:35:07.195 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-mon[32774]: 3.f scrub starts 2024-02-05T06:35:07.195 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-mon[32774]: 3.f scrub ok 2024-02-05T06:35:07.196 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-mon[32774]: 4.7 scrub starts 2024-02-05T06:35:07.196 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-mon[32774]: 4.7 scrub ok 2024-02-05T06:35:07.196 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:07.196 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:07.196 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:07.196 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:07.196 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-mon[32774]: Deploying daemon node-exporter.a on smithi028 2024-02-05T06:35:07.197 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-mon[32774]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:35:07.197 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:06 smithi167 ceph-mon[32774]: from='client.14784 v1:172.21.15.28:0/4189782476' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:35:07.243 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-02-05T06:35:06.155409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1023,"num_read_kb":869,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1771792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":640,"ondisk_log_size":640,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240784,"kb_used_data":7644,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749720432,"statfs":{"total":767960285184,"available":767713722368,"internally_reserved":0,"allocated":7827456,"data_stored":4526644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002596"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":60,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.904407+0000","last_change":"2024-02-05T06:34:55.600835+0000","last_active":"2024-02-05T06:35:02.904407+0000","last_peered":"2024-02-05T06:35:02.904407+0000","last_clean":"2024-02-05T06:35:02.904407+0000","last_became_active":"2024-02-05T06:33:35.594467+0000","last_became_peered":"2024-02-05T06:33:35.594467+0000","last_unstale":"2024-02-05T06:35:02.904407+0000","last_undegraded":"2024-02-05T06:35:02.904407+0000","last_fullsized":"2024-02-05T06:35:02.904407+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:55.600744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:55.600744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:09.045705+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":56,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.969228+0000","last_change":"2024-02-05T06:33:33.586212+0000","last_active":"2024-02-05T06:35:02.969228+0000","last_peered":"2024-02-05T06:35:02.969228+0000","last_clean":"2024-02-05T06:35:02.969228+0000","last_became_active":"2024-02-05T06:33:33.585848+0000","last_became_peered":"2024-02-05T06:33:33.585848+0000","last_unstale":"2024-02-05T06:35:02.969228+0000","last_undegraded":"2024-02-05T06:35:02.969228+0000","last_fullsized":"2024-02-05T06:35:02.969228+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.905286+0000","last_change":"2024-02-05T06:33:29.549864+0000","last_active":"2024-02-05T06:35:02.905286+0000","last_peered":"2024-02-05T06:35:02.905286+0000","last_clean":"2024-02-05T06:35:02.905286+0000","last_became_active":"2024-02-05T06:33:29.549592+0000","last_became_peered":"2024-02-05T06:33:29.549592+0000","last_unstale":"2024-02-05T06:35:02.905286+0000","last_undegraded":"2024-02-05T06:35:02.905286+0000","last_fullsized":"2024-02-05T06:35:02.905286+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":84,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.904244+0000","last_change":"2024-02-05T06:35:01.601968+0000","last_active":"2024-02-05T06:35:02.904244+0000","last_peered":"2024-02-05T06:35:02.904244+0000","last_clean":"2024-02-05T06:35:02.904244+0000","last_became_active":"2024-02-05T06:33:31.566571+0000","last_became_peered":"2024-02-05T06:33:31.566571+0000","last_unstale":"2024-02-05T06:35:02.904244+0000","last_undegraded":"2024-02-05T06:35:02.904244+0000","last_fullsized":"2024-02-05T06:35:02.904244+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:35:01.601838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:35:01.601838+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:05.249000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":53,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.026729+0000","last_change":"2024-02-05T06:33:35.668346+0000","last_active":"2024-02-05T06:35:03.026729+0000","last_peered":"2024-02-05T06:35:03.026729+0000","last_clean":"2024-02-05T06:35:03.026729+0000","last_became_active":"2024-02-05T06:33:35.667778+0000","last_became_peered":"2024-02-05T06:33:35.667778+0000","last_unstale":"2024-02-05T06:35:03.026729+0000","last_undegraded":"2024-02-05T06:35:03.026729+0000","last_fullsized":"2024-02-05T06:35:03.026729+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":56,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.904268+0000","last_change":"2024-02-05T06:33:33.583029+0000","last_active":"2024-02-05T06:35:02.904268+0000","last_peered":"2024-02-05T06:35:02.904268+0000","last_clean":"2024-02-05T06:35:02.904268+0000","last_became_active":"2024-02-05T06:33:33.582731+0000","last_became_peered":"2024-02-05T06:33:33.582731+0000","last_unstale":"2024-02-05T06:35:02.904268+0000","last_undegraded":"2024-02-05T06:35:02.904268+0000","last_fullsized":"2024-02-05T06:35:02.904268+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":73,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.027240+0000","last_change":"2024-02-05T06:34:34.821722+0000","last_active":"2024-02-05T06:35:03.027240+0000","last_peered":"2024-02-05T06:35:03.027240+0000","last_clean":"2024-02-05T06:35:03.027240+0000","last_became_active":"2024-02-05T06:33:29.553864+0000","last_became_peered":"2024-02-05T06:33:29.553864+0000","last_unstale":"2024-02-05T06:35:03.027240+0000","last_undegraded":"2024-02-05T06:35:03.027240+0000","last_fullsized":"2024-02-05T06:35:03.027240+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:34.821603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:34.821603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:39.571760+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"88'22","reported_seq":101,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.026543+0000","last_change":"2024-02-05T06:34:35.832461+0000","last_active":"2024-02-05T06:35:03.026543+0000","last_peered":"2024-02-05T06:35:03.026543+0000","last_clean":"2024-02-05T06:35:03.026543+0000","last_became_active":"2024-02-05T06:33:31.572544+0000","last_became_peered":"2024-02-05T06:33:31.572544+0000","last_unstale":"2024-02-05T06:35:03.026543+0000","last_undegraded":"2024-02-05T06:35:03.026543+0000","last_fullsized":"2024-02-05T06:35:03.026543+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'22","last_scrub_stamp":"2024-02-05T06:34:35.832368+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:34:35.832368+0000","last_clean_scrub_stamp":"2024-02-05T06:34:35.832368+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:43.028037+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":53,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.904164+0000","last_change":"2024-02-05T06:33:35.595744+0000","last_active":"2024-02-05T06:35:02.904164+0000","last_peered":"2024-02-05T06:35:02.904164+0000","last_clean":"2024-02-05T06:35:02.904164+0000","last_became_active":"2024-02-05T06:33:35.594860+0000","last_became_peered":"2024-02-05T06:33:35.594860+0000","last_unstale":"2024-02-05T06:35:02.904164+0000","last_undegraded":"2024-02-05T06:35:02.904164+0000","last_fullsized":"2024-02-05T06:35:02.904164+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.968408+0000","last_change":"2024-02-05T06:34:34.187757+0000","last_active":"2024-02-05T06:35:02.968408+0000","last_peered":"2024-02-05T06:35:02.968408+0000","last_clean":"2024-02-05T06:35:02.968408+0000","last_became_active":"2024-02-05T06:33:33.585459+0000","last_became_peered":"2024-02-05T06:33:33.585459+0000","last_unstale":"2024-02-05T06:35:02.968408+0000","last_undegraded":"2024-02-05T06:35:02.968408+0000","last_fullsized":"2024-02-05T06:35:02.968408+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:34.187643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:34.187643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:35.727918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":71,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.968548+0000","last_change":"2024-02-05T06:34:44.081424+0000","last_active":"2024-02-05T06:35:02.968548+0000","last_peered":"2024-02-05T06:35:02.968548+0000","last_clean":"2024-02-05T06:35:02.968548+0000","last_became_active":"2024-02-05T06:33:29.553251+0000","last_became_peered":"2024-02-05T06:33:29.553251+0000","last_unstale":"2024-02-05T06:35:02.968548+0000","last_undegraded":"2024-02-05T06:35:02.968548+0000","last_fullsized":"2024-02-05T06:35:02.968548+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:44.081318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:44.081318+0000","last_clean_scrub_stamp":"2024-02-05T06:34:44.081318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:49.865346+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"92'13","reported_seq":88,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.904578+0000","last_change":"2024-02-05T06:34:46.543006+0000","last_active":"2024-02-05T06:35:02.904578+0000","last_peered":"2024-02-05T06:35:02.904578+0000","last_clean":"2024-02-05T06:35:02.904578+0000","last_became_active":"2024-02-05T06:33:31.570995+0000","last_became_peered":"2024-02-05T06:33:31.570995+0000","last_unstale":"2024-02-05T06:35:02.904578+0000","last_undegraded":"2024-02-05T06:35:02.904578+0000","last_fullsized":"2024-02-05T06:35:02.904578+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'13","last_scrub_stamp":"2024-02-05T06:34:46.542893+0000","last_deep_scrub":"92'13","last_deep_scrub_stamp":"2024-02-05T06:34:46.542893+0000","last_clean_scrub_stamp":"2024-02-05T06:34:46.542893+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:00.864899+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.968164+0000","last_change":"2024-02-05T06:33:35.600855+0000","last_active":"2024-02-05T06:35:02.968164+0000","last_peered":"2024-02-05T06:35:02.968164+0000","last_clean":"2024-02-05T06:35:02.968164+0000","last_became_active":"2024-02-05T06:33:35.600399+0000","last_became_peered":"2024-02-05T06:33:35.600399+0000","last_unstale":"2024-02-05T06:35:02.968164+0000","last_undegraded":"2024-02-05T06:35:02.968164+0000","last_fullsized":"2024-02-05T06:35:02.968164+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.225910+0000","last_change":"2024-02-05T06:34:53.073825+0000","last_active":"2024-02-05T06:35:03.225910+0000","last_peered":"2024-02-05T06:35:03.225910+0000","last_clean":"2024-02-05T06:35:03.225910+0000","last_became_active":"2024-02-05T06:33:33.584151+0000","last_became_peered":"2024-02-05T06:33:33.584151+0000","last_unstale":"2024-02-05T06:35:03.225910+0000","last_undegraded":"2024-02-05T06:35:03.225910+0000","last_fullsized":"2024-02-05T06:35:03.225910+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:53.073710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:53.073710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:08.940750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":71,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.968619+0000","last_change":"2024-02-05T06:34:50.055371+0000","last_active":"2024-02-05T06:35:02.968619+0000","last_peered":"2024-02-05T06:35:02.968619+0000","last_clean":"2024-02-05T06:35:02.968619+0000","last_became_active":"2024-02-05T06:33:29.548931+0000","last_became_peered":"2024-02-05T06:33:29.548931+0000","last_unstale":"2024-02-05T06:35:02.968619+0000","last_undegraded":"2024-02-05T06:35:02.968619+0000","last_fullsized":"2024-02-05T06:35:02.968619+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:50.055245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:50.055245+0000","last_clean_scrub_stamp":"2024-02-05T06:34:50.055245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:54.332822+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":78,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.904546+0000","last_change":"2024-02-05T06:34:53.638858+0000","last_active":"2024-02-05T06:35:02.904546+0000","last_peered":"2024-02-05T06:35:02.904546+0000","last_clean":"2024-02-05T06:35:02.904546+0000","last_became_active":"2024-02-05T06:33:31.569654+0000","last_became_peered":"2024-02-05T06:33:31.569654+0000","last_unstale":"2024-02-05T06:35:02.904546+0000","last_undegraded":"2024-02-05T06:35:02.904546+0000","last_fullsized":"2024-02-05T06:35:02.904546+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-02-05T06:34:53.638704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:53.638704+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:12.883735+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":60,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.905099+0000","last_change":"2024-02-05T06:34:45.573557+0000","last_active":"2024-02-05T06:35:02.905099+0000","last_peered":"2024-02-05T06:35:02.905099+0000","last_clean":"2024-02-05T06:35:02.905099+0000","last_became_active":"2024-02-05T06:33:35.664772+0000","last_became_peered":"2024-02-05T06:33:35.664772+0000","last_unstale":"2024-02-05T06:35:02.905099+0000","last_undegraded":"2024-02-05T06:35:02.905099+0000","last_fullsized":"2024-02-05T06:35:02.905099+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:45.573432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:45.573432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:10.154762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":56,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.969702+0000","last_change":"2024-02-05T06:33:33.588208+0000","last_active":"2024-02-05T06:35:02.969702+0000","last_peered":"2024-02-05T06:35:02.969702+0000","last_clean":"2024-02-05T06:35:02.969702+0000","last_became_active":"2024-02-05T06:33:33.587920+0000","last_became_peered":"2024-02-05T06:33:33.587920+0000","last_unstale":"2024-02-05T06:35:02.969702+0000","last_undegraded":"2024-02-05T06:35:02.969702+0000","last_fullsized":"2024-02-05T06:35:02.969702+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"66'1","reported_seq":71,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.904984+0000","last_change":"2024-02-05T06:33:29.552104+0000","last_active":"2024-02-05T06:35:02.904984+0000","last_peered":"2024-02-05T06:35:02.904984+0000","last_clean":"2024-02-05T06:35:02.904984+0000","last_became_active":"2024-02-05T06:33:29.551820+0000","last_became_peered":"2024-02-05T06:33:29.551820+0000","last_unstale":"2024-02-05T06:35:02.904984+0000","last_undegraded":"2024-02-05T06:35:02.904984+0000","last_fullsized":"2024-02-05T06:35:02.904984+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"88'22","reported_seq":99,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.903836+0000","last_change":"2024-02-05T06:34:35.234726+0000","last_active":"2024-02-05T06:35:02.903836+0000","last_peered":"2024-02-05T06:35:02.903836+0000","last_clean":"2024-02-05T06:35:02.903836+0000","last_became_active":"2024-02-05T06:33:31.569763+0000","last_became_peered":"2024-02-05T06:33:31.569763+0000","last_unstale":"2024-02-05T06:35:02.903836+0000","last_undegraded":"2024-02-05T06:35:02.903836+0000","last_fullsized":"2024-02-05T06:35:02.903836+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'22","last_scrub_stamp":"2024-02-05T06:34:35.234636+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:34:35.234636+0000","last_clean_scrub_stamp":"2024-02-05T06:34:35.234636+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:53.234677+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:05.077327+0000","last_change":"2024-02-05T06:35:05.077348+0000","last_active":"2024-02-05T06:35:05.077327+0000","last_peered":"2024-02-05T06:35:05.077327+0000","last_clean":"2024-02-05T06:35:05.077327+0000","last_became_active":"2024-02-05T06:33:35.664852+0000","last_became_peered":"2024-02-05T06:33:35.664852+0000","last_unstale":"2024-02-05T06:35:05.077327+0000","last_undegraded":"2024-02-05T06:35:05.077327+0000","last_fullsized":"2024-02-05T06:35:05.077327+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:05.077268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:05.077268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:21.542392+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:05.581025+0000","last_change":"2024-02-05T06:35:05.581047+0000","last_active":"2024-02-05T06:35:05.581025+0000","last_peered":"2024-02-05T06:35:05.581025+0000","last_clean":"2024-02-05T06:35:05.581025+0000","last_became_active":"2024-02-05T06:33:33.587595+0000","last_became_peered":"2024-02-05T06:33:33.587595+0000","last_unstale":"2024-02-05T06:35:05.581025+0000","last_undegraded":"2024-02-05T06:35:05.581025+0000","last_fullsized":"2024-02-05T06:35:05.581025+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:05.580971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:35:05.580971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:34.724446+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":72,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.904651+0000","last_change":"2024-02-05T06:34:38.623297+0000","last_active":"2024-02-05T06:35:02.904651+0000","last_peered":"2024-02-05T06:35:02.904651+0000","last_clean":"2024-02-05T06:35:02.904651+0000","last_became_active":"2024-02-05T06:33:29.551340+0000","last_became_peered":"2024-02-05T06:33:29.551340+0000","last_unstale":"2024-02-05T06:35:02.904651+0000","last_undegraded":"2024-02-05T06:35:02.904651+0000","last_fullsized":"2024-02-05T06:35:02.904651+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:38.623203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:38.623203+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:49.053281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":110,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.026883+0000","last_change":"2024-02-05T06:34:59.999830+0000","last_active":"2024-02-05T06:35:03.026883+0000","last_peered":"2024-02-05T06:35:03.026883+0000","last_clean":"2024-02-05T06:35:03.026883+0000","last_became_active":"2024-02-05T06:33:55.885324+0000","last_became_peered":"2024-02-05T06:33:55.885324+0000","last_unstale":"2024-02-05T06:35:03.026883+0000","last_undegraded":"2024-02-05T06:35:03.026883+0000","last_fullsized":"2024-02-05T06:35:03.026883+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:34:59.999726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:59.999726+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:23.143025+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"73'1","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.226166+0000","last_change":"2024-02-05T06:33:35.598794+0000","last_active":"2024-02-05T06:35:03.226166+0000","last_peered":"2024-02-05T06:35:03.226166+0000","last_clean":"2024-02-05T06:35:03.226166+0000","last_became_active":"2024-02-05T06:33:35.598501+0000","last_became_peered":"2024-02-05T06:33:35.598501+0000","last_unstale":"2024-02-05T06:35:03.226166+0000","last_undegraded":"2024-02-05T06:35:03.226166+0000","last_fullsized":"2024-02-05T06:35:03.226166+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"73'8","reported_seq":98,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.227151+0000","last_change":"2024-02-05T06:34:55.312940+0000","last_active":"2024-02-05T06:35:03.227151+0000","last_peered":"2024-02-05T06:35:03.227151+0000","last_clean":"2024-02-05T06:35:03.227151+0000","last_became_active":"2024-02-05T06:33:33.586389+0000","last_became_peered":"2024-02-05T06:33:33.586389+0000","last_unstale":"2024-02-05T06:35:03.227151+0000","last_undegraded":"2024-02-05T06:35:03.227151+0000","last_fullsized":"2024-02-05T06:35:03.227151+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:34:55.312853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:55.312853+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:18.905872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":73,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.969318+0000","last_change":"2024-02-05T06:34:48.387568+0000","last_active":"2024-02-05T06:35:02.969318+0000","last_peered":"2024-02-05T06:35:02.969318+0000","last_clean":"2024-02-05T06:35:02.969318+0000","last_became_active":"2024-02-05T06:33:29.546201+0000","last_became_peered":"2024-02-05T06:33:29.546201+0000","last_unstale":"2024-02-05T06:35:02.969318+0000","last_undegraded":"2024-02-05T06:35:02.969318+0000","last_fullsized":"2024-02-05T06:35:02.969318+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:34:48.387474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:48.387474+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:58.746172+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"92'15","reported_seq":89,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.905175+0000","last_change":"2024-02-05T06:34:50.718858+0000","last_active":"2024-02-05T06:35:02.905175+0000","last_peered":"2024-02-05T06:35:02.905175+0000","last_clean":"2024-02-05T06:35:02.905175+0000","last_became_active":"2024-02-05T06:33:31.666481+0000","last_became_peered":"2024-02-05T06:33:31.666481+0000","last_unstale":"2024-02-05T06:35:02.905175+0000","last_undegraded":"2024-02-05T06:35:02.905175+0000","last_fullsized":"2024-02-05T06:35:02.905175+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'15","last_scrub_stamp":"2024-02-05T06:34:50.718768+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-02-05T06:34:50.718768+0000","last_clean_scrub_stamp":"2024-02-05T06:34:50.718768+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:51.271274+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":60,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.969944+0000","last_change":"2024-02-05T06:34:55.554666+0000","last_active":"2024-02-05T06:35:02.969944+0000","last_peered":"2024-02-05T06:35:02.969944+0000","last_clean":"2024-02-05T06:35:02.969944+0000","last_became_active":"2024-02-05T06:33:35.593925+0000","last_became_peered":"2024-02-05T06:33:35.593925+0000","last_unstale":"2024-02-05T06:35:02.969944+0000","last_undegraded":"2024-02-05T06:35:02.969944+0000","last_fullsized":"2024-02-05T06:35:02.969944+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:55.554581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:55.554581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:56.554973+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":57,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.905348+0000","last_change":"2024-02-05T06:33:33.584707+0000","last_active":"2024-02-05T06:35:02.905348+0000","last_peered":"2024-02-05T06:35:02.905348+0000","last_clean":"2024-02-05T06:35:02.905348+0000","last_became_active":"2024-02-05T06:33:33.584390+0000","last_became_peered":"2024-02-05T06:33:33.584390+0000","last_unstale":"2024-02-05T06:35:02.905348+0000","last_undegraded":"2024-02-05T06:35:02.905348+0000","last_fullsized":"2024-02-05T06:35:02.905348+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.026085+0000","last_change":"2024-02-05T06:33:29.550640+0000","last_active":"2024-02-05T06:35:03.026085+0000","last_peered":"2024-02-05T06:35:03.026085+0000","last_clean":"2024-02-05T06:35:03.026085+0000","last_became_active":"2024-02-05T06:33:29.550347+0000","last_became_peered":"2024-02-05T06:33:29.550347+0000","last_unstale":"2024-02-05T06:35:03.026085+0000","last_undegraded":"2024-02-05T06:35:03.026085+0000","last_fullsized":"2024-02-05T06:35:03.026085+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"92'16","reported_seq":92,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.227084+0000","last_change":"2024-02-05T06:34:45.270307+0000","last_active":"2024-02-05T06:35:03.227084+0000","last_peered":"2024-02-05T06:35:03.227084+0000","last_clean":"2024-02-05T06:35:03.227084+0000","last_became_active":"2024-02-05T06:33:31.573088+0000","last_became_peered":"2024-02-05T06:33:31.573088+0000","last_unstale":"2024-02-05T06:35:03.227084+0000","last_undegraded":"2024-02-05T06:35:03.227084+0000","last_fullsized":"2024-02-05T06:35:03.227084+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'16","last_scrub_stamp":"2024-02-05T06:34:45.270135+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:34:45.270135+0000","last_clean_scrub_stamp":"2024-02-05T06:34:45.270135+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:04.519387+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.225778+0000","last_change":"2024-02-05T06:34:55.062101+0000","last_active":"2024-02-05T06:35:03.225778+0000","last_peered":"2024-02-05T06:35:03.225778+0000","last_clean":"2024-02-05T06:35:03.225778+0000","last_became_active":"2024-02-05T06:33:35.597453+0000","last_became_peered":"2024-02-05T06:33:35.597453+0000","last_unstale":"2024-02-05T06:35:03.225778+0000","last_undegraded":"2024-02-05T06:35:03.225778+0000","last_fullsized":"2024-02-05T06:35:03.225778+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:55.062013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:55.062013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:18.704482+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.225583+0000","last_change":"2024-02-05T06:34:51.334669+0000","last_active":"2024-02-05T06:35:03.225583+0000","last_peered":"2024-02-05T06:35:03.225583+0000","last_clean":"2024-02-05T06:35:03.225583+0000","last_became_active":"2024-02-05T06:33:33.585125+0000","last_became_peered":"2024-02-05T06:33:33.585125+0000","last_unstale":"2024-02-05T06:35:03.225583+0000","last_undegraded":"2024-02-05T06:35:03.225583+0000","last_fullsized":"2024-02-05T06:35:03.225583+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:51.334570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:51.334570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:19.957468+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.905252+0000","last_change":"2024-02-05T06:33:29.549815+0000","last_active":"2024-02-05T06:35:02.905252+0000","last_peered":"2024-02-05T06:35:02.905252+0000","last_clean":"2024-02-05T06:35:02.905252+0000","last_became_active":"2024-02-05T06:33:29.549204+0000","last_became_peered":"2024-02-05T06:33:29.549204+0000","last_unstale":"2024-02-05T06:35:02.905252+0000","last_undegraded":"2024-02-05T06:35:02.905252+0000","last_fullsized":"2024-02-05T06:35:02.905252+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"73'4","reported_seq":67,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.026184+0000","last_change":"2024-02-05T06:33:31.573713+0000","last_active":"2024-02-05T06:35:03.026184+0000","last_peered":"2024-02-05T06:35:03.026184+0000","last_clean":"2024-02-05T06:35:03.026184+0000","last_became_active":"2024-02-05T06:33:31.573410+0000","last_became_peered":"2024-02-05T06:33:31.573410+0000","last_unstale":"2024-02-05T06:35:03.026184+0000","last_undegraded":"2024-02-05T06:35:03.026184+0000","last_fullsized":"2024-02-05T06:35:03.026184+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.026607+0000","last_change":"2024-02-05T06:34:54.902085+0000","last_active":"2024-02-05T06:35:03.026607+0000","last_peered":"2024-02-05T06:35:03.026607+0000","last_clean":"2024-02-05T06:35:03.026607+0000","last_became_active":"2024-02-05T06:33:35.667602+0000","last_became_peered":"2024-02-05T06:33:35.667602+0000","last_unstale":"2024-02-05T06:35:03.026607+0000","last_undegraded":"2024-02-05T06:35:03.026607+0000","last_fullsized":"2024-02-05T06:35:03.026607+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:54.902000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:54.902000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:57.025191+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.225991+0000","last_change":"2024-02-05T06:34:43.035728+0000","last_active":"2024-02-05T06:35:03.225991+0000","last_peered":"2024-02-05T06:35:03.225991+0000","last_clean":"2024-02-05T06:35:03.225991+0000","last_became_active":"2024-02-05T06:33:33.589428+0000","last_became_peered":"2024-02-05T06:33:33.589428+0000","last_unstale":"2024-02-05T06:35:03.225991+0000","last_undegraded":"2024-02-05T06:35:03.225991+0000","last_fullsized":"2024-02-05T06:35:03.225991+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:43.035635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:43.035635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:52.189018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.968772+0000","last_change":"2024-02-05T06:33:29.551576+0000","last_active":"2024-02-05T06:35:02.968772+0000","last_peered":"2024-02-05T06:35:02.968772+0000","last_clean":"2024-02-05T06:35:02.968772+0000","last_became_active":"2024-02-05T06:33:29.551278+0000","last_became_peered":"2024-02-05T06:33:29.551278+0000","last_unstale":"2024-02-05T06:35:02.968772+0000","last_undegraded":"2024-02-05T06:35:02.968772+0000","last_fullsized":"2024-02-05T06:35:02.968772+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"73'11","reported_seq":80,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.026528+0000","last_change":"2024-02-05T06:33:31.665074+0000","last_active":"2024-02-05T06:35:03.026528+0000","last_peered":"2024-02-05T06:35:03.026528+0000","last_clean":"2024-02-05T06:35:03.026528+0000","last_became_active":"2024-02-05T06:33:31.664743+0000","last_became_peered":"2024-02-05T06:33:31.664743+0000","last_unstale":"2024-02-05T06:35:03.026528+0000","last_undegraded":"2024-02-05T06:35:03.026528+0000","last_fullsized":"2024-02-05T06:35:03.026528+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"73'1","reported_seq":54,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.903403+0000","last_change":"2024-02-05T06:33:35.595450+0000","last_active":"2024-02-05T06:35:02.903403+0000","last_peered":"2024-02-05T06:35:02.903403+0000","last_clean":"2024-02-05T06:35:02.903403+0000","last_became_active":"2024-02-05T06:33:35.595031+0000","last_became_peered":"2024-02-05T06:33:35.595031+0000","last_unstale":"2024-02-05T06:35:02.903403+0000","last_undegraded":"2024-02-05T06:35:02.903403+0000","last_fullsized":"2024-02-05T06:35:02.903403+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.027360+0000","last_change":"2024-02-05T06:35:03.027397+0000","last_active":"2024-02-05T06:35:03.027360+0000","last_peered":"2024-02-05T06:35:03.027360+0000","last_clean":"2024-02-05T06:35:03.027360+0000","last_became_active":"2024-02-05T06:33:33.593901+0000","last_became_peered":"2024-02-05T06:33:33.593901+0000","last_unstale":"2024-02-05T06:35:03.027360+0000","last_undegraded":"2024-02-05T06:35:03.027360+0000","last_fullsized":"2024-02-05T06:35:03.027360+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:03.027270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:35:03.027270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:31.482593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.226454+0000","last_change":"2024-02-05T06:33:29.559294+0000","last_active":"2024-02-05T06:35:03.226454+0000","last_peered":"2024-02-05T06:35:03.226454+0000","last_clean":"2024-02-05T06:35:03.226454+0000","last_became_active":"2024-02-05T06:33:29.559006+0000","last_became_peered":"2024-02-05T06:33:29.559006+0000","last_unstale":"2024-02-05T06:35:03.226454+0000","last_undegraded":"2024-02-05T06:35:03.226454+0000","last_fullsized":"2024-02-05T06:35:03.226454+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":84,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.969799+0000","last_change":"2024-02-05T06:34:52.486190+0000","last_active":"2024-02-05T06:35:02.969799+0000","last_peered":"2024-02-05T06:35:02.969799+0000","last_clean":"2024-02-05T06:35:02.969799+0000","last_became_active":"2024-02-05T06:33:31.568529+0000","last_became_peered":"2024-02-05T06:33:31.568529+0000","last_unstale":"2024-02-05T06:35:02.969799+0000","last_undegraded":"2024-02-05T06:35:02.969799+0000","last_fullsized":"2024-02-05T06:35:02.969799+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:34:52.486064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:52.486064+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:20.775162+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.026101+0000","last_change":"2024-02-05T06:34:48.882657+0000","last_active":"2024-02-05T06:35:03.026101+0000","last_peered":"2024-02-05T06:35:03.026101+0000","last_clean":"2024-02-05T06:35:03.026101+0000","last_became_active":"2024-02-05T06:33:35.597399+0000","last_became_peered":"2024-02-05T06:33:35.597399+0000","last_unstale":"2024-02-05T06:35:03.026101+0000","last_undegraded":"2024-02-05T06:35:03.026101+0000","last_fullsized":"2024-02-05T06:35:03.026101+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:48.882556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:48.882556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:52.368291+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.969517+0000","last_change":"2024-02-05T06:34:40.361349+0000","last_active":"2024-02-05T06:35:02.969517+0000","last_peered":"2024-02-05T06:35:02.969517+0000","last_clean":"2024-02-05T06:35:02.969517+0000","last_became_active":"2024-02-05T06:33:33.588461+0000","last_became_peered":"2024-02-05T06:33:33.588461+0000","last_unstale":"2024-02-05T06:35:02.969517+0000","last_undegraded":"2024-02-05T06:35:02.969517+0000","last_fullsized":"2024-02-05T06:35:02.969517+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:40.361273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:40.361273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:50.089342+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":72,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.904986+0000","last_change":"2024-02-05T06:34:48.561146+0000","last_active":"2024-02-05T06:35:02.904986+0000","last_peered":"2024-02-05T06:35:02.904986+0000","last_clean":"2024-02-05T06:35:02.904986+0000","last_became_active":"2024-02-05T06:33:29.552336+0000","last_became_peered":"2024-02-05T06:33:29.552336+0000","last_unstale":"2024-02-05T06:35:02.904986+0000","last_undegraded":"2024-02-05T06:35:02.904986+0000","last_fullsized":"2024-02-05T06:35:02.904986+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:48.561010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:48.561010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:06.806058+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"92'16","reported_seq":92,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.904739+0000","last_change":"2024-02-05T06:35:02.589330+0000","last_active":"2024-02-05T06:35:02.904739+0000","last_peered":"2024-02-05T06:35:02.904739+0000","last_clean":"2024-02-05T06:35:02.904739+0000","last_became_active":"2024-02-05T06:33:31.569488+0000","last_became_peered":"2024-02-05T06:33:31.569488+0000","last_unstale":"2024-02-05T06:35:02.904739+0000","last_undegraded":"2024-02-05T06:35:02.904739+0000","last_fullsized":"2024-02-05T06:35:02.904739+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'16","last_scrub_stamp":"2024-02-05T06:35:02.589229+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:35:02.589229+0000","last_clean_scrub_stamp":"2024-02-05T06:35:02.589229+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:23.268662+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":53,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.904980+0000","last_change":"2024-02-05T06:33:35.600109+0000","last_active":"2024-02-05T06:35:02.904980+0000","last_peered":"2024-02-05T06:35:02.904980+0000","last_clean":"2024-02-05T06:35:02.904980+0000","last_became_active":"2024-02-05T06:33:35.599808+0000","last_became_peered":"2024-02-05T06:33:35.599808+0000","last_unstale":"2024-02-05T06:35:02.904980+0000","last_undegraded":"2024-02-05T06:35:02.904980+0000","last_fullsized":"2024-02-05T06:35:02.904980+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":91,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.969499+0000","last_change":"2024-02-05T06:35:02.969499+0000","last_active":"2024-02-05T06:35:02.969499+0000","last_peered":"2024-02-05T06:35:02.969499+0000","last_clean":"2024-02-05T06:35:02.969499+0000","last_became_active":"2024-02-05T06:33:33.585642+0000","last_became_peered":"2024-02-05T06:33:33.585642+0000","last_unstale":"2024-02-05T06:35:02.969499+0000","last_undegraded":"2024-02-05T06:35:02.969499+0000","last_fullsized":"2024-02-05T06:35:02.969499+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:35:02.969404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:35:02.969404+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:18.064705+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":73,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:05.967372+0000","last_change":"2024-02-05T06:35:05.967392+0000","last_active":"2024-02-05T06:35:05.967372+0000","last_peered":"2024-02-05T06:35:05.967372+0000","last_clean":"2024-02-05T06:35:05.967372+0000","last_became_active":"2024-02-05T06:33:29.556832+0000","last_became_peered":"2024-02-05T06:33:29.556832+0000","last_unstale":"2024-02-05T06:35:05.967372+0000","last_undegraded":"2024-02-05T06:35:05.967372+0000","last_fullsized":"2024-02-05T06:35:05.967372+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:05.967303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:35:05.967303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:13.776318+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'10","reported_seq":84,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.026821+0000","last_change":"2024-02-05T06:34:41.873963+0000","last_active":"2024-02-05T06:35:03.026821+0000","last_peered":"2024-02-05T06:35:03.026821+0000","last_clean":"2024-02-05T06:35:03.026821+0000","last_became_active":"2024-02-05T06:33:31.666808+0000","last_became_peered":"2024-02-05T06:33:31.666808+0000","last_unstale":"2024-02-05T06:35:03.026821+0000","last_undegraded":"2024-02-05T06:35:03.026821+0000","last_fullsized":"2024-02-05T06:35:03.026821+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-02-05T06:34:41.873876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:41.873876+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:08.435407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":52,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.905298+0000","last_change":"2024-02-05T06:33:35.597787+0000","last_active":"2024-02-05T06:35:02.905298+0000","last_peered":"2024-02-05T06:35:02.905298+0000","last_clean":"2024-02-05T06:35:02.905298+0000","last_became_active":"2024-02-05T06:33:35.597494+0000","last_became_peered":"2024-02-05T06:33:35.597494+0000","last_unstale":"2024-02-05T06:35:02.905298+0000","last_undegraded":"2024-02-05T06:35:02.905298+0000","last_fullsized":"2024-02-05T06:35:02.905298+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":93,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.027123+0000","last_change":"2024-02-05T06:34:47.925702+0000","last_active":"2024-02-05T06:35:03.027123+0000","last_peered":"2024-02-05T06:35:03.027123+0000","last_clean":"2024-02-05T06:35:03.027123+0000","last_became_active":"2024-02-05T06:33:33.586043+0000","last_became_peered":"2024-02-05T06:33:33.586043+0000","last_unstale":"2024-02-05T06:35:03.027123+0000","last_undegraded":"2024-02-05T06:35:03.027123+0000","last_fullsized":"2024-02-05T06:35:03.027123+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:34:47.925577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:47.925577+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:48.049254+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":73,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.904775+0000","last_change":"2024-02-05T06:34:53.792716+0000","last_active":"2024-02-05T06:35:02.904775+0000","last_peered":"2024-02-05T06:35:02.904775+0000","last_clean":"2024-02-05T06:35:02.904775+0000","last_became_active":"2024-02-05T06:33:29.552170+0000","last_became_peered":"2024-02-05T06:33:29.552170+0000","last_unstale":"2024-02-05T06:35:02.904775+0000","last_undegraded":"2024-02-05T06:35:02.904775+0000","last_fullsized":"2024-02-05T06:35:02.904775+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:53.792621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:53.792621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:58.017125+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"73'9","reported_seq":83,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.968790+0000","last_change":"2024-02-05T06:34:36.172574+0000","last_active":"2024-02-05T06:35:02.968790+0000","last_peered":"2024-02-05T06:35:02.968790+0000","last_clean":"2024-02-05T06:35:02.968790+0000","last_became_active":"2024-02-05T06:33:31.666678+0000","last_became_peered":"2024-02-05T06:33:31.666678+0000","last_unstale":"2024-02-05T06:35:02.968790+0000","last_undegraded":"2024-02-05T06:35:02.968790+0000","last_fullsized":"2024-02-05T06:35:02.968790+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:34:36.172444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:36.172444+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:50.358190+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":59,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.904040+0000","last_change":"2024-02-05T06:34:43.228873+0000","last_active":"2024-02-05T06:35:02.904040+0000","last_peered":"2024-02-05T06:35:02.904040+0000","last_clean":"2024-02-05T06:35:02.904040+0000","last_became_active":"2024-02-05T06:33:35.601508+0000","last_became_peered":"2024-02-05T06:33:35.601508+0000","last_unstale":"2024-02-05T06:35:02.904040+0000","last_undegraded":"2024-02-05T06:35:02.904040+0000","last_fullsized":"2024-02-05T06:35:02.904040+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:43.228774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:43.228774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:57.334710+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.027384+0000","last_change":"2024-02-05T06:34:42.866516+0000","last_active":"2024-02-05T06:35:03.027384+0000","last_peered":"2024-02-05T06:35:03.027384+0000","last_clean":"2024-02-05T06:35:03.027384+0000","last_became_active":"2024-02-05T06:33:33.593025+0000","last_became_peered":"2024-02-05T06:33:33.593025+0000","last_unstale":"2024-02-05T06:35:03.027384+0000","last_undegraded":"2024-02-05T06:35:03.027384+0000","last_fullsized":"2024-02-05T06:35:03.027384+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:42.866394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:42.866394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:43.991930+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":73,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.225657+0000","last_change":"2024-02-05T06:34:49.998454+0000","last_active":"2024-02-05T06:35:03.225657+0000","last_peered":"2024-02-05T06:35:03.225657+0000","last_clean":"2024-02-05T06:35:03.225657+0000","last_became_active":"2024-02-05T06:33:29.553880+0000","last_became_peered":"2024-02-05T06:33:29.553880+0000","last_unstale":"2024-02-05T06:35:03.225657+0000","last_undegraded":"2024-02-05T06:35:03.225657+0000","last_fullsized":"2024-02-05T06:35:03.225657+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:49.998323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:49.998323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:50.575718+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"73'9","reported_seq":85,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.905197+0000","last_change":"2024-02-05T06:34:43.760971+0000","last_active":"2024-02-05T06:35:02.905197+0000","last_peered":"2024-02-05T06:35:02.905197+0000","last_clean":"2024-02-05T06:35:02.905197+0000","last_became_active":"2024-02-05T06:33:31.665766+0000","last_became_peered":"2024-02-05T06:33:31.665766+0000","last_unstale":"2024-02-05T06:35:02.905197+0000","last_undegraded":"2024-02-05T06:35:02.905197+0000","last_fullsized":"2024-02-05T06:35:02.905197+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:34:43.760888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:43.760888+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:44.661663+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.226312+0000","last_change":"2024-02-05T06:35:00.009704+0000","last_active":"2024-02-05T06:35:03.226312+0000","last_peered":"2024-02-05T06:35:03.226312+0000","last_clean":"2024-02-05T06:35:03.226312+0000","last_became_active":"2024-02-05T06:33:35.665596+0000","last_became_peered":"2024-02-05T06:33:35.665596+0000","last_unstale":"2024-02-05T06:35:03.226312+0000","last_undegraded":"2024-02-05T06:35:03.226312+0000","last_fullsized":"2024-02-05T06:35:03.226312+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:00.009630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:00.009630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:05.139245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.968764+0000","last_change":"2024-02-05T06:34:45.096449+0000","last_active":"2024-02-05T06:35:02.968764+0000","last_peered":"2024-02-05T06:35:02.968764+0000","last_clean":"2024-02-05T06:35:02.968764+0000","last_became_active":"2024-02-05T06:33:33.586768+0000","last_became_peered":"2024-02-05T06:33:33.586768+0000","last_unstale":"2024-02-05T06:35:02.968764+0000","last_undegraded":"2024-02-05T06:35:02.968764+0000","last_fullsized":"2024-02-05T06:35:02.968764+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:45.096340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:45.096340+0000","last_clean_scrub_stamp":"2024-02-05T06:34:45.096340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:12.226977+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":72,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.225494+0000","last_change":"2024-02-05T06:34:30.204064+0000","last_active":"2024-02-05T06:35:03.225494+0000","last_peered":"2024-02-05T06:35:03.225494+0000","last_clean":"2024-02-05T06:35:03.225494+0000","last_became_active":"2024-02-05T06:33:29.554427+0000","last_became_peered":"2024-02-05T06:33:29.554427+0000","last_unstale":"2024-02-05T06:35:03.225494+0000","last_undegraded":"2024-02-05T06:35:03.225494+0000","last_fullsized":"2024-02-05T06:35:03.225494+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:30.203967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:30.203967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:35.531249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":78,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.027475+0000","last_change":"2024-02-05T06:34:55.912380+0000","last_active":"2024-02-05T06:35:03.027475+0000","last_peered":"2024-02-05T06:35:03.027475+0000","last_clean":"2024-02-05T06:35:03.027475+0000","last_became_active":"2024-02-05T06:33:31.665847+0000","last_became_peered":"2024-02-05T06:33:31.665847+0000","last_unstale":"2024-02-05T06:35:03.027475+0000","last_undegraded":"2024-02-05T06:35:03.027475+0000","last_fullsized":"2024-02-05T06:35:03.027475+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'6","last_scrub_stamp":"2024-02-05T06:34:55.912282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:55.912282+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:02.248574+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":59,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.968528+0000","last_change":"2024-02-05T06:35:01.967337+0000","last_active":"2024-02-05T06:35:02.968528+0000","last_peered":"2024-02-05T06:35:02.968528+0000","last_clean":"2024-02-05T06:35:02.968528+0000","last_became_active":"2024-02-05T06:33:35.665581+0000","last_became_peered":"2024-02-05T06:33:35.665581+0000","last_unstale":"2024-02-05T06:35:02.968528+0000","last_undegraded":"2024-02-05T06:35:02.968528+0000","last_fullsized":"2024-02-05T06:35:02.968528+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:01.967242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:01.967242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:22.767376+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":85,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:05.001969+0000","last_change":"2024-02-05T06:35:05.001996+0000","last_active":"2024-02-05T06:35:05.001969+0000","last_peered":"2024-02-05T06:35:05.001969+0000","last_clean":"2024-02-05T06:35:05.001969+0000","last_became_active":"2024-02-05T06:33:29.557522+0000","last_became_peered":"2024-02-05T06:33:29.557522+0000","last_unstale":"2024-02-05T06:35:05.001969+0000","last_undegraded":"2024-02-05T06:35:05.001969+0000","last_fullsized":"2024-02-05T06:35:05.001969+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-02-05T06:35:05.001912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:35:05.001912+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:21.485133+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"73'15","reported_seq":84,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.968164+0000","last_change":"2024-02-05T06:33:31.666410+0000","last_active":"2024-02-05T06:35:02.968164+0000","last_peered":"2024-02-05T06:35:02.968164+0000","last_clean":"2024-02-05T06:35:02.968164+0000","last_became_active":"2024-02-05T06:33:31.666056+0000","last_became_peered":"2024-02-05T06:33:31.666056+0000","last_unstale":"2024-02-05T06:35:02.968164+0000","last_undegraded":"2024-02-05T06:35:02.968164+0000","last_fullsized":"2024-02-05T06:35:02.968164+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"73'8","reported_seq":93,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.225950+0000","last_change":"2024-02-05T06:34:49.007791+0000","last_active":"2024-02-05T06:35:03.225950+0000","last_peered":"2024-02-05T06:35:03.225950+0000","last_clean":"2024-02-05T06:35:03.225950+0000","last_became_active":"2024-02-05T06:33:33.589535+0000","last_became_peered":"2024-02-05T06:33:33.589535+0000","last_unstale":"2024-02-05T06:35:03.225950+0000","last_undegraded":"2024-02-05T06:35:03.225950+0000","last_fullsized":"2024-02-05T06:35:03.225950+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:34:49.007659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:49.007659+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:59.892182+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":53,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.026619+0000","last_change":"2024-02-05T06:33:35.601680+0000","last_active":"2024-02-05T06:35:03.026619+0000","last_peered":"2024-02-05T06:35:03.026619+0000","last_clean":"2024-02-05T06:35:03.026619+0000","last_became_active":"2024-02-05T06:33:35.601030+0000","last_became_peered":"2024-02-05T06:33:35.601030+0000","last_unstale":"2024-02-05T06:35:03.026619+0000","last_undegraded":"2024-02-05T06:35:03.026619+0000","last_fullsized":"2024-02-05T06:35:03.026619+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.226867+0000","last_change":"2024-02-05T06:33:29.558929+0000","last_active":"2024-02-05T06:35:03.226867+0000","last_peered":"2024-02-05T06:35:03.226867+0000","last_clean":"2024-02-05T06:35:03.226867+0000","last_became_active":"2024-02-05T06:33:29.557284+0000","last_became_peered":"2024-02-05T06:33:29.557284+0000","last_unstale":"2024-02-05T06:35:03.226867+0000","last_undegraded":"2024-02-05T06:35:03.226867+0000","last_fullsized":"2024-02-05T06:35:03.226867+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'12","reported_seq":91,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.904686+0000","last_change":"2024-02-05T06:35:00.650630+0000","last_active":"2024-02-05T06:35:02.904686+0000","last_peered":"2024-02-05T06:35:02.904686+0000","last_clean":"2024-02-05T06:35:02.904686+0000","last_became_active":"2024-02-05T06:33:31.569924+0000","last_became_peered":"2024-02-05T06:33:31.569924+0000","last_unstale":"2024-02-05T06:35:02.904686+0000","last_undegraded":"2024-02-05T06:35:02.904686+0000","last_fullsized":"2024-02-05T06:35:02.904686+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:35:00.650542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:35:00.650542+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:28.714634+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.903541+0000","last_change":"2024-02-05T06:34:40.272941+0000","last_active":"2024-02-05T06:35:02.903541+0000","last_peered":"2024-02-05T06:35:02.903541+0000","last_clean":"2024-02-05T06:35:02.903541+0000","last_became_active":"2024-02-05T06:33:33.592546+0000","last_became_peered":"2024-02-05T06:33:33.592546+0000","last_unstale":"2024-02-05T06:35:02.903541+0000","last_undegraded":"2024-02-05T06:35:02.903541+0000","last_fullsized":"2024-02-05T06:35:02.903541+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:40.272852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:40.272852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:07.055905+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":53,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.225386+0000","last_change":"2024-02-05T06:33:35.599830+0000","last_active":"2024-02-05T06:35:03.225386+0000","last_peered":"2024-02-05T06:35:03.225386+0000","last_clean":"2024-02-05T06:35:03.225386+0000","last_became_active":"2024-02-05T06:33:35.599527+0000","last_became_peered":"2024-02-05T06:33:35.599527+0000","last_unstale":"2024-02-05T06:35:03.225386+0000","last_undegraded":"2024-02-05T06:35:03.225386+0000","last_fullsized":"2024-02-05T06:35:03.225386+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.903830+0000","last_change":"2024-02-05T06:34:42.265315+0000","last_active":"2024-02-05T06:35:02.903830+0000","last_peered":"2024-02-05T06:35:02.903830+0000","last_clean":"2024-02-05T06:35:02.903830+0000","last_became_active":"2024-02-05T06:33:33.589203+0000","last_became_peered":"2024-02-05T06:33:33.589203+0000","last_unstale":"2024-02-05T06:35:02.903830+0000","last_undegraded":"2024-02-05T06:35:02.903830+0000","last_fullsized":"2024-02-05T06:35:02.903830+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:42.265214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:42.265214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:53.433345+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":73,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.227070+0000","last_change":"2024-02-05T06:34:59.028827+0000","last_active":"2024-02-05T06:35:03.227070+0000","last_peered":"2024-02-05T06:35:03.227070+0000","last_clean":"2024-02-05T06:35:03.227070+0000","last_became_active":"2024-02-05T06:33:29.556541+0000","last_became_peered":"2024-02-05T06:33:29.556541+0000","last_unstale":"2024-02-05T06:35:03.227070+0000","last_undegraded":"2024-02-05T06:35:03.227070+0000","last_fullsized":"2024-02-05T06:35:03.227070+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:59.028748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:59.028748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:24.209293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"73'19","reported_seq":99,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.226213+0000","last_change":"2024-02-05T06:34:44.243017+0000","last_active":"2024-02-05T06:35:03.226213+0000","last_peered":"2024-02-05T06:35:03.226213+0000","last_clean":"2024-02-05T06:35:03.226213+0000","last_became_active":"2024-02-05T06:33:31.664699+0000","last_became_peered":"2024-02-05T06:33:31.664699+0000","last_unstale":"2024-02-05T06:35:03.226213+0000","last_undegraded":"2024-02-05T06:35:03.226213+0000","last_fullsized":"2024-02-05T06:35:03.226213+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-02-05T06:34:44.242897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:44.242897+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:01.187407+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":53,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.904222+0000","last_change":"2024-02-05T06:33:35.598200+0000","last_active":"2024-02-05T06:35:02.904222+0000","last_peered":"2024-02-05T06:35:02.904222+0000","last_clean":"2024-02-05T06:35:02.904222+0000","last_became_active":"2024-02-05T06:33:35.597863+0000","last_became_peered":"2024-02-05T06:33:35.597863+0000","last_unstale":"2024-02-05T06:35:02.904222+0000","last_undegraded":"2024-02-05T06:35:02.904222+0000","last_fullsized":"2024-02-05T06:35:02.904222+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.903758+0000","last_change":"2024-02-05T06:34:45.299145+0000","last_active":"2024-02-05T06:35:02.903758+0000","last_peered":"2024-02-05T06:35:02.903758+0000","last_clean":"2024-02-05T06:35:02.903758+0000","last_became_active":"2024-02-05T06:33:33.586895+0000","last_became_peered":"2024-02-05T06:33:33.586895+0000","last_unstale":"2024-02-05T06:35:02.903758+0000","last_undegraded":"2024-02-05T06:35:02.903758+0000","last_fullsized":"2024-02-05T06:35:02.903758+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:45.299055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:45.299055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:03.861496+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.968495+0000","last_change":"2024-02-05T06:33:29.556254+0000","last_active":"2024-02-05T06:35:02.968495+0000","last_peered":"2024-02-05T06:35:02.968495+0000","last_clean":"2024-02-05T06:35:02.968495+0000","last_became_active":"2024-02-05T06:33:29.555950+0000","last_became_peered":"2024-02-05T06:33:29.555950+0000","last_unstale":"2024-02-05T06:35:02.968495+0000","last_undegraded":"2024-02-05T06:35:02.968495+0000","last_fullsized":"2024-02-05T06:35:02.968495+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"73'9","reported_seq":85,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.904887+0000","last_change":"2024-02-05T06:34:36.779561+0000","last_active":"2024-02-05T06:35:02.904887+0000","last_peered":"2024-02-05T06:35:02.904887+0000","last_clean":"2024-02-05T06:35:02.904887+0000","last_became_active":"2024-02-05T06:33:31.564572+0000","last_became_peered":"2024-02-05T06:33:31.564572+0000","last_unstale":"2024-02-05T06:35:02.904887+0000","last_undegraded":"2024-02-05T06:35:02.904887+0000","last_fullsized":"2024-02-05T06:35:02.904887+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:34:36.779466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:36.779466+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:02.144960+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":53,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.027116+0000","last_change":"2024-02-05T06:33:35.598477+0000","last_active":"2024-02-05T06:35:03.027116+0000","last_peered":"2024-02-05T06:35:03.027116+0000","last_clean":"2024-02-05T06:35:03.027116+0000","last_became_active":"2024-02-05T06:33:35.597750+0000","last_became_peered":"2024-02-05T06:33:35.597750+0000","last_unstale":"2024-02-05T06:35:03.027116+0000","last_undegraded":"2024-02-05T06:35:03.027116+0000","last_fullsized":"2024-02-05T06:35:03.027116+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":82,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.968305+0000","last_change":"2024-02-05T06:34:48.046115+0000","last_active":"2024-02-05T06:35:02.968305+0000","last_peered":"2024-02-05T06:35:02.968305+0000","last_clean":"2024-02-05T06:35:02.968305+0000","last_became_active":"2024-02-05T06:33:28.542228+0000","last_became_peered":"2024-02-05T06:33:28.542228+0000","last_unstale":"2024-02-05T06:35:02.968305+0000","last_undegraded":"2024-02-05T06:35:02.968305+0000","last_fullsized":"2024-02-05T06:35:02.968305+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-02-05T06:34:48.046017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:34:48.046017+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:04.358520+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001220348,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.904454+0000","last_change":"2024-02-05T06:33:29.554533+0000","last_active":"2024-02-05T06:35:02.904454+0000","last_peered":"2024-02-05T06:35:02.904454+0000","last_clean":"2024-02-05T06:35:02.904454+0000","last_became_active":"2024-02-05T06:33:29.553882+0000","last_became_peered":"2024-02-05T06:33:29.553882+0000","last_unstale":"2024-02-05T06:35:02.904454+0000","last_undegraded":"2024-02-05T06:35:02.904454+0000","last_fullsized":"2024-02-05T06:35:02.904454+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":77,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.968446+0000","last_change":"2024-02-05T06:33:55.896865+0000","last_active":"2024-02-05T06:35:02.968446+0000","last_peered":"2024-02-05T06:35:02.968446+0000","last_clean":"2024-02-05T06:35:02.968446+0000","last_became_active":"2024-02-05T06:33:55.884601+0000","last_became_peered":"2024-02-05T06:33:55.884601+0000","last_unstale":"2024-02-05T06:35:02.968446+0000","last_undegraded":"2024-02-05T06:35:02.968446+0000","last_fullsized":"2024-02-05T06:35:02.968446+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":57,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.905047+0000","last_change":"2024-02-05T06:33:33.589668+0000","last_active":"2024-02-05T06:35:02.905047+0000","last_peered":"2024-02-05T06:35:02.905047+0000","last_clean":"2024-02-05T06:35:02.905047+0000","last_became_active":"2024-02-05T06:33:33.589348+0000","last_became_peered":"2024-02-05T06:33:33.589348+0000","last_unstale":"2024-02-05T06:35:02.905047+0000","last_undegraded":"2024-02-05T06:35:02.905047+0000","last_fullsized":"2024-02-05T06:35:02.905047+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.226785+0000","last_change":"2024-02-05T06:35:01.995683+0000","last_active":"2024-02-05T06:35:03.226785+0000","last_peered":"2024-02-05T06:35:03.226785+0000","last_clean":"2024-02-05T06:35:03.226785+0000","last_became_active":"2024-02-05T06:33:35.666324+0000","last_became_peered":"2024-02-05T06:33:35.666324+0000","last_unstale":"2024-02-05T06:35:03.226785+0000","last_undegraded":"2024-02-05T06:35:03.226785+0000","last_fullsized":"2024-02-05T06:35:03.226785+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:01.995584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:01.995584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:05.585562+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.969476+0000","last_change":"2024-02-05T06:33:29.552003+0000","last_active":"2024-02-05T06:35:02.969476+0000","last_peered":"2024-02-05T06:35:02.969476+0000","last_clean":"2024-02-05T06:35:02.969476+0000","last_became_active":"2024-02-05T06:33:29.551549+0000","last_became_peered":"2024-02-05T06:33:29.551549+0000","last_unstale":"2024-02-05T06:35:02.969476+0000","last_undegraded":"2024-02-05T06:35:02.969476+0000","last_fullsized":"2024-02-05T06:35:02.969476+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":80,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.904083+0000","last_change":"2024-02-05T06:34:50.302211+0000","last_active":"2024-02-05T06:35:02.904083+0000","last_peered":"2024-02-05T06:35:02.904083+0000","last_clean":"2024-02-05T06:35:02.904083+0000","last_became_active":"2024-02-05T06:33:28.542977+0000","last_became_peered":"2024-02-05T06:33:28.542977+0000","last_unstale":"2024-02-05T06:35:02.904083+0000","last_undegraded":"2024-02-05T06:35:02.904083+0000","last_fullsized":"2024-02-05T06:35:02.904083+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:50.302133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:34:50.302133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:57.608189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091640799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'13","reported_seq":95,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:05.548029+0000","last_change":"2024-02-05T06:35:05.548084+0000","last_active":"2024-02-05T06:35:05.548029+0000","last_peered":"2024-02-05T06:35:05.548029+0000","last_clean":"2024-02-05T06:35:05.548029+0000","last_became_active":"2024-02-05T06:33:31.560082+0000","last_became_peered":"2024-02-05T06:33:31.560082+0000","last_unstale":"2024-02-05T06:35:05.548029+0000","last_undegraded":"2024-02-05T06:35:05.548029+0000","last_fullsized":"2024-02-05T06:35:05.548029+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-02-05T06:35:05.547906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:35:05.547906+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:07.037641+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.904407+0000","last_change":"2024-02-05T06:34:34.240413+0000","last_active":"2024-02-05T06:35:02.904407+0000","last_peered":"2024-02-05T06:35:02.904407+0000","last_clean":"2024-02-05T06:35:02.904407+0000","last_became_active":"2024-02-05T06:33:33.587593+0000","last_became_peered":"2024-02-05T06:33:33.587593+0000","last_unstale":"2024-02-05T06:35:02.904407+0000","last_undegraded":"2024-02-05T06:35:02.904407+0000","last_fullsized":"2024-02-05T06:35:02.904407+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:34.240331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:34.240331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:49.655873+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":51,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.968835+0000","last_change":"2024-02-05T06:33:35.600278+0000","last_active":"2024-02-05T06:35:02.968835+0000","last_peered":"2024-02-05T06:35:02.968835+0000","last_clean":"2024-02-05T06:35:02.968835+0000","last_became_active":"2024-02-05T06:33:35.600004+0000","last_became_peered":"2024-02-05T06:33:35.600004+0000","last_unstale":"2024-02-05T06:35:02.968835+0000","last_undegraded":"2024-02-05T06:35:02.968835+0000","last_fullsized":"2024-02-05T06:35:02.968835+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":319,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.225489+0000","last_change":"2024-02-05T06:33:45.752222+0000","last_active":"2024-02-05T06:35:03.225489+0000","last_peered":"2024-02-05T06:35:03.225489+0000","last_clean":"2024-02-05T06:35:03.225489+0000","last_became_active":"2024-02-05T06:33:19.862877+0000","last_became_peered":"2024-02-05T06:33:19.862877+0000","last_unstale":"2024-02-05T06:35:03.225489+0000","last_undegraded":"2024-02-05T06:35:03.225489+0000","last_fullsized":"2024-02-05T06:35:03.225489+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-02-05T06:33:45.752124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:33:45.752124+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":73,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.026679+0000","last_change":"2024-02-05T06:34:44.894533+0000","last_active":"2024-02-05T06:35:03.026679+0000","last_peered":"2024-02-05T06:35:03.026679+0000","last_clean":"2024-02-05T06:35:03.026679+0000","last_became_active":"2024-02-05T06:33:29.552698+0000","last_became_peered":"2024-02-05T06:33:29.552698+0000","last_unstale":"2024-02-05T06:35:03.026679+0000","last_undegraded":"2024-02-05T06:35:03.026679+0000","last_fullsized":"2024-02-05T06:35:03.026679+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:44.894443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:44.894443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:54.598610+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"73'16","reported_seq":110,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.226726+0000","last_change":"2024-02-05T06:35:02.410154+0000","last_active":"2024-02-05T06:35:03.226726+0000","last_peered":"2024-02-05T06:35:03.226726+0000","last_clean":"2024-02-05T06:35:03.226726+0000","last_became_active":"2024-02-05T06:33:31.665797+0000","last_became_peered":"2024-02-05T06:33:31.665797+0000","last_unstale":"2024-02-05T06:35:03.226726+0000","last_undegraded":"2024-02-05T06:35:03.226726+0000","last_fullsized":"2024-02-05T06:35:03.226726+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-02-05T06:35:02.410049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:35:02.410049+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:20.009808+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":57,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.226429+0000","last_change":"2024-02-05T06:33:33.591727+0000","last_active":"2024-02-05T06:35:03.226429+0000","last_peered":"2024-02-05T06:35:03.226429+0000","last_clean":"2024-02-05T06:35:03.226429+0000","last_became_active":"2024-02-05T06:33:33.591484+0000","last_became_peered":"2024-02-05T06:33:33.591484+0000","last_unstale":"2024-02-05T06:35:03.226429+0000","last_undegraded":"2024-02-05T06:35:03.226429+0000","last_fullsized":"2024-02-05T06:35:03.226429+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":60,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.969459+0000","last_change":"2024-02-05T06:34:46.361122+0000","last_active":"2024-02-05T06:35:02.969459+0000","last_peered":"2024-02-05T06:35:02.969459+0000","last_clean":"2024-02-05T06:35:02.969459+0000","last_became_active":"2024-02-05T06:33:35.600656+0000","last_became_peered":"2024-02-05T06:33:35.600656+0000","last_unstale":"2024-02-05T06:35:02.969459+0000","last_undegraded":"2024-02-05T06:35:02.969459+0000","last_fullsized":"2024-02-05T06:35:02.969459+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:46.361013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:46.361013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:07.997590+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":73,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.904497+0000","last_change":"2024-02-05T06:34:44.781805+0000","last_active":"2024-02-05T06:35:02.904497+0000","last_peered":"2024-02-05T06:35:02.904497+0000","last_clean":"2024-02-05T06:35:02.904497+0000","last_became_active":"2024-02-05T06:33:29.544399+0000","last_became_peered":"2024-02-05T06:33:29.544399+0000","last_unstale":"2024-02-05T06:35:02.904497+0000","last_undegraded":"2024-02-05T06:35:02.904497+0000","last_fullsized":"2024-02-05T06:35:02.904497+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:44.781704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:44.781704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:06.382435+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":174,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:04.441391+0000","last_change":"2024-02-05T06:34:41.991897+0000","last_active":"2024-02-05T06:35:04.441391+0000","last_peered":"2024-02-05T06:35:04.441391+0000","last_clean":"2024-02-05T06:35:04.441391+0000","last_became_active":"2024-02-05T06:33:28.544852+0000","last_became_peered":"2024-02-05T06:33:28.544852+0000","last_unstale":"2024-02-05T06:35:04.441391+0000","last_undegraded":"2024-02-05T06:35:04.441391+0000","last_fullsized":"2024-02-05T06:35:04.441391+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T06:34:41.991777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:34:41.991777+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:10.879161+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00074151600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":89,"num_read_kb":84,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":87,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.905060+0000","last_change":"2024-02-05T06:34:52.743798+0000","last_active":"2024-02-05T06:35:02.905060+0000","last_peered":"2024-02-05T06:35:02.905060+0000","last_clean":"2024-02-05T06:35:02.905060+0000","last_became_active":"2024-02-05T06:33:31.564604+0000","last_became_peered":"2024-02-05T06:33:31.564604+0000","last_unstale":"2024-02-05T06:35:02.905060+0000","last_undegraded":"2024-02-05T06:35:02.905060+0000","last_fullsized":"2024-02-05T06:35:02.905060+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:34:52.743708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:52.743708+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:55.426621+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":55,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.969025+0000","last_change":"2024-02-05T06:33:33.589962+0000","last_active":"2024-02-05T06:35:02.969025+0000","last_peered":"2024-02-05T06:35:02.969025+0000","last_clean":"2024-02-05T06:35:02.969025+0000","last_became_active":"2024-02-05T06:33:33.589388+0000","last_became_peered":"2024-02-05T06:33:33.589388+0000","last_unstale":"2024-02-05T06:35:02.969025+0000","last_undegraded":"2024-02-05T06:35:02.969025+0000","last_fullsized":"2024-02-05T06:35:02.969025+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:33:32.561974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":52,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.969179+0000","last_change":"2024-02-05T06:33:35.665208+0000","last_active":"2024-02-05T06:35:02.969179+0000","last_peered":"2024-02-05T06:35:02.969179+0000","last_clean":"2024-02-05T06:35:02.969179+0000","last_became_active":"2024-02-05T06:33:35.664858+0000","last_became_peered":"2024-02-05T06:33:35.664858+0000","last_unstale":"2024-02-05T06:35:02.969179+0000","last_undegraded":"2024-02-05T06:35:02.969179+0000","last_fullsized":"2024-02-05T06:35:02.969179+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"66'1","reported_seq":78,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.969697+0000","last_change":"2024-02-05T06:34:36.457175+0000","last_active":"2024-02-05T06:35:02.969697+0000","last_peered":"2024-02-05T06:35:02.969697+0000","last_clean":"2024-02-05T06:35:02.969697+0000","last_became_active":"2024-02-05T06:33:29.548227+0000","last_became_peered":"2024-02-05T06:33:29.548227+0000","last_unstale":"2024-02-05T06:35:02.969697+0000","last_undegraded":"2024-02-05T06:35:02.969697+0000","last_fullsized":"2024-02-05T06:35:02.969697+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:34:36.457059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:36.457059+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:45.403222+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'19","reported_seq":97,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.905151+0000","last_change":"2024-02-05T06:33:31.667636+0000","last_active":"2024-02-05T06:35:02.905151+0000","last_peered":"2024-02-05T06:35:02.905151+0000","last_clean":"2024-02-05T06:35:02.905151+0000","last_became_active":"2024-02-05T06:33:31.667340+0000","last_became_peered":"2024-02-05T06:33:31.667340+0000","last_unstale":"2024-02-05T06:35:02.905151+0000","last_undegraded":"2024-02-05T06:35:02.905151+0000","last_fullsized":"2024-02-05T06:35:02.905151+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.226240+0000","last_change":"2024-02-05T06:34:50.353163+0000","last_active":"2024-02-05T06:35:03.226240+0000","last_peered":"2024-02-05T06:35:03.226240+0000","last_clean":"2024-02-05T06:35:03.226240+0000","last_became_active":"2024-02-05T06:33:33.588163+0000","last_became_peered":"2024-02-05T06:33:33.588163+0000","last_unstale":"2024-02-05T06:35:03.226240+0000","last_undegraded":"2024-02-05T06:35:03.226240+0000","last_fullsized":"2024-02-05T06:35:03.226240+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:50.353040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:50.353040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:10.822758+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":53,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.903414+0000","last_change":"2024-02-05T06:33:35.600831+0000","last_active":"2024-02-05T06:35:02.903414+0000","last_peered":"2024-02-05T06:35:02.903414+0000","last_clean":"2024-02-05T06:35:02.903414+0000","last_became_active":"2024-02-05T06:33:35.600485+0000","last_became_peered":"2024-02-05T06:33:35.600485+0000","last_unstale":"2024-02-05T06:35:02.903414+0000","last_undegraded":"2024-02-05T06:35:02.903414+0000","last_fullsized":"2024-02-05T06:35:02.903414+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":71,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.968674+0000","last_change":"2024-02-05T06:34:43.098655+0000","last_active":"2024-02-05T06:35:02.968674+0000","last_peered":"2024-02-05T06:35:02.968674+0000","last_clean":"2024-02-05T06:35:02.968674+0000","last_became_active":"2024-02-05T06:33:29.554940+0000","last_became_peered":"2024-02-05T06:33:29.554940+0000","last_unstale":"2024-02-05T06:35:02.968674+0000","last_undegraded":"2024-02-05T06:35:02.968674+0000","last_fullsized":"2024-02-05T06:35:02.968674+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:43.098546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:43.098546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:01.990483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"73'10","reported_seq":75,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.969599+0000","last_change":"2024-02-05T06:33:31.570032+0000","last_active":"2024-02-05T06:35:02.969599+0000","last_peered":"2024-02-05T06:35:02.969599+0000","last_clean":"2024-02-05T06:35:02.969599+0000","last_became_active":"2024-02-05T06:33:31.569680+0000","last_became_peered":"2024-02-05T06:33:31.569680+0000","last_unstale":"2024-02-05T06:35:02.969599+0000","last_undegraded":"2024-02-05T06:35:02.969599+0000","last_fullsized":"2024-02-05T06:35:02.969599+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"73'8","reported_seq":93,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.904780+0000","last_change":"2024-02-05T06:34:54.764491+0000","last_active":"2024-02-05T06:35:02.904780+0000","last_peered":"2024-02-05T06:35:02.904780+0000","last_clean":"2024-02-05T06:35:02.904780+0000","last_became_active":"2024-02-05T06:33:33.582187+0000","last_became_peered":"2024-02-05T06:33:33.582187+0000","last_unstale":"2024-02-05T06:35:02.904780+0000","last_undegraded":"2024-02-05T06:35:02.904780+0000","last_fullsized":"2024-02-05T06:35:02.904780+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:34:54.764410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:54.764410+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:15.024154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":61,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.226225+0000","last_change":"2024-02-05T06:34:41.001666+0000","last_active":"2024-02-05T06:35:03.226225+0000","last_peered":"2024-02-05T06:35:03.226225+0000","last_clean":"2024-02-05T06:35:03.226225+0000","last_became_active":"2024-02-05T06:33:35.666315+0000","last_became_peered":"2024-02-05T06:33:35.666315+0000","last_unstale":"2024-02-05T06:35:03.226225+0000","last_undegraded":"2024-02-05T06:35:03.226225+0000","last_fullsized":"2024-02-05T06:35:03.226225+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:41.001570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:41.001570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:53.416960+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":66,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.905099+0000","last_change":"2024-02-05T06:33:29.546683+0000","last_active":"2024-02-05T06:35:02.905099+0000","last_peered":"2024-02-05T06:35:02.905099+0000","last_clean":"2024-02-05T06:35:02.905099+0000","last_became_active":"2024-02-05T06:33:29.546405+0000","last_became_peered":"2024-02-05T06:33:29.546405+0000","last_unstale":"2024-02-05T06:35:02.905099+0000","last_undegraded":"2024-02-05T06:35:02.905099+0000","last_fullsized":"2024-02-05T06:35:02.905099+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'14","reported_seq":89,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.904572+0000","last_change":"2024-02-05T06:34:56.595997+0000","last_active":"2024-02-05T06:35:02.904572+0000","last_peered":"2024-02-05T06:35:02.904572+0000","last_clean":"2024-02-05T06:35:02.904572+0000","last_became_active":"2024-02-05T06:33:31.563836+0000","last_became_peered":"2024-02-05T06:33:31.563836+0000","last_unstale":"2024-02-05T06:35:02.904572+0000","last_undegraded":"2024-02-05T06:35:02.904572+0000","last_fullsized":"2024-02-05T06:35:02.904572+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'14","last_scrub_stamp":"2024-02-05T06:34:56.595906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:56.595906+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:05.623306+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"73'8","reported_seq":93,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.026478+0000","last_change":"2024-02-05T06:34:56.941927+0000","last_active":"2024-02-05T06:35:03.026478+0000","last_peered":"2024-02-05T06:35:03.026478+0000","last_clean":"2024-02-05T06:35:03.026478+0000","last_became_active":"2024-02-05T06:33:33.591646+0000","last_became_peered":"2024-02-05T06:33:33.591646+0000","last_unstale":"2024-02-05T06:35:03.026478+0000","last_undegraded":"2024-02-05T06:35:03.026478+0000","last_fullsized":"2024-02-05T06:35:03.026478+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:34:56.941840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:56.941840+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:25.481310+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":52,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.904448+0000","last_change":"2024-02-05T06:33:35.600296+0000","last_active":"2024-02-05T06:35:02.904448+0000","last_peered":"2024-02-05T06:35:02.904448+0000","last_clean":"2024-02-05T06:35:02.904448+0000","last_became_active":"2024-02-05T06:33:35.599805+0000","last_became_peered":"2024-02-05T06:33:35.599805+0000","last_unstale":"2024-02-05T06:35:02.904448+0000","last_undegraded":"2024-02-05T06:35:02.904448+0000","last_fullsized":"2024-02-05T06:35:02.904448+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:03.912813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.027272+0000","last_change":"2024-02-05T06:33:29.557121+0000","last_active":"2024-02-05T06:35:03.027272+0000","last_peered":"2024-02-05T06:35:03.027272+0000","last_clean":"2024-02-05T06:35:03.027272+0000","last_became_active":"2024-02-05T06:33:29.556002+0000","last_became_peered":"2024-02-05T06:33:29.556002+0000","last_unstale":"2024-02-05T06:35:03.027272+0000","last_undegraded":"2024-02-05T06:35:03.027272+0000","last_fullsized":"2024-02-05T06:35:03.027272+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"73'18","reported_seq":101,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:04.034731+0000","last_change":"2024-02-05T06:35:04.034763+0000","last_active":"2024-02-05T06:35:04.034731+0000","last_peered":"2024-02-05T06:35:04.034731+0000","last_clean":"2024-02-05T06:35:04.034731+0000","last_became_active":"2024-02-05T06:33:31.665995+0000","last_became_peered":"2024-02-05T06:33:31.665995+0000","last_unstale":"2024-02-05T06:35:04.034731+0000","last_undegraded":"2024-02-05T06:35:04.034731+0000","last_fullsized":"2024-02-05T06:35:04.034731+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-02-05T06:35:04.034664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:35:04.034664+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:09.685656+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.904354+0000","last_change":"2024-02-05T06:34:49.579591+0000","last_active":"2024-02-05T06:35:02.904354+0000","last_peered":"2024-02-05T06:35:02.904354+0000","last_clean":"2024-02-05T06:35:02.904354+0000","last_became_active":"2024-02-05T06:33:33.586935+0000","last_became_peered":"2024-02-05T06:33:33.586935+0000","last_unstale":"2024-02-05T06:35:02.904354+0000","last_undegraded":"2024-02-05T06:35:02.904354+0000","last_fullsized":"2024-02-05T06:35:02.904354+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:49.579493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:49.579493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:00.372885+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":59,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.968508+0000","last_change":"2024-02-05T06:34:53.061797+0000","last_active":"2024-02-05T06:35:02.968508+0000","last_peered":"2024-02-05T06:35:02.968508+0000","last_clean":"2024-02-05T06:35:02.968508+0000","last_became_active":"2024-02-05T06:33:35.593210+0000","last_became_peered":"2024-02-05T06:33:35.593210+0000","last_unstale":"2024-02-05T06:35:02.968508+0000","last_undegraded":"2024-02-05T06:35:02.968508+0000","last_fullsized":"2024-02-05T06:35:02.968508+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:53.061717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:53.061717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:58.799900+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":92,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.905196+0000","last_change":"2024-02-05T06:34:40.619333+0000","last_active":"2024-02-05T06:35:02.905196+0000","last_peered":"2024-02-05T06:35:02.905196+0000","last_clean":"2024-02-05T06:35:02.905196+0000","last_became_active":"2024-02-05T06:33:33.586753+0000","last_became_peered":"2024-02-05T06:33:33.586753+0000","last_unstale":"2024-02-05T06:35:02.905196+0000","last_undegraded":"2024-02-05T06:35:02.905196+0000","last_fullsized":"2024-02-05T06:35:02.905196+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:34:40.619192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:40.619192+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:48.417494+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":65,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.026290+0000","last_change":"2024-02-05T06:33:29.557073+0000","last_active":"2024-02-05T06:35:03.026290+0000","last_peered":"2024-02-05T06:35:03.026290+0000","last_clean":"2024-02-05T06:35:03.026290+0000","last_became_active":"2024-02-05T06:33:29.555919+0000","last_became_peered":"2024-02-05T06:33:29.555919+0000","last_unstale":"2024-02-05T06:35:03.026290+0000","last_undegraded":"2024-02-05T06:35:03.026290+0000","last_fullsized":"2024-02-05T06:35:03.026290+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"73'15","reported_seq":93,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.970031+0000","last_change":"2024-02-05T06:35:02.573015+0000","last_active":"2024-02-05T06:35:02.970031+0000","last_peered":"2024-02-05T06:35:02.970031+0000","last_clean":"2024-02-05T06:35:02.970031+0000","last_became_active":"2024-02-05T06:33:31.572603+0000","last_became_peered":"2024-02-05T06:33:31.572603+0000","last_unstale":"2024-02-05T06:35:02.970031+0000","last_undegraded":"2024-02-05T06:35:02.970031+0000","last_fullsized":"2024-02-05T06:35:02.970031+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-02-05T06:35:02.572925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:35:02.572925+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:17.108298+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":53,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.026810+0000","last_change":"2024-02-05T06:33:35.668457+0000","last_active":"2024-02-05T06:35:03.026810+0000","last_peered":"2024-02-05T06:35:03.026810+0000","last_clean":"2024-02-05T06:35:03.026810+0000","last_became_active":"2024-02-05T06:33:35.668020+0000","last_became_peered":"2024-02-05T06:33:35.668020+0000","last_unstale":"2024-02-05T06:35:03.026810+0000","last_undegraded":"2024-02-05T06:35:03.026810+0000","last_fullsized":"2024-02-05T06:35:03.026810+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":63,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.999883+0000","last_change":"2024-02-05T06:35:03.999907+0000","last_active":"2024-02-05T06:35:03.999883+0000","last_peered":"2024-02-05T06:35:03.999883+0000","last_clean":"2024-02-05T06:35:03.999883+0000","last_became_active":"2024-02-05T06:33:33.583029+0000","last_became_peered":"2024-02-05T06:33:33.583029+0000","last_unstale":"2024-02-05T06:35:03.999883+0000","last_undegraded":"2024-02-05T06:35:03.999883+0000","last_fullsized":"2024-02-05T06:35:03.999883+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:03.999803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:35:03.999803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:33.176083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.904357+0000","last_change":"2024-02-05T06:33:29.553248+0000","last_active":"2024-02-05T06:35:02.904357+0000","last_peered":"2024-02-05T06:35:02.904357+0000","last_clean":"2024-02-05T06:35:02.904357+0000","last_became_active":"2024-02-05T06:33:29.552415+0000","last_became_peered":"2024-02-05T06:33:29.552415+0000","last_unstale":"2024-02-05T06:35:02.904357+0000","last_undegraded":"2024-02-05T06:35:02.904357+0000","last_fullsized":"2024-02-05T06:35:02.904357+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:33:28.530487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"73'11","reported_seq":105,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.905032+0000","last_change":"2024-02-05T06:34:50.592920+0000","last_active":"2024-02-05T06:35:02.905032+0000","last_peered":"2024-02-05T06:35:02.905032+0000","last_clean":"2024-02-05T06:35:02.905032+0000","last_became_active":"2024-02-05T06:33:55.885523+0000","last_became_peered":"2024-02-05T06:33:55.885523+0000","last_unstale":"2024-02-05T06:35:02.905032+0000","last_undegraded":"2024-02-05T06:35:02.905032+0000","last_fullsized":"2024-02-05T06:35:02.905032+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-02-05T06:34:50.592793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:50.592793+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:56.891427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":59,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.904536+0000","last_change":"2024-02-05T06:34:49.298355+0000","last_active":"2024-02-05T06:35:02.904536+0000","last_peered":"2024-02-05T06:35:02.904536+0000","last_clean":"2024-02-05T06:35:02.904536+0000","last_became_active":"2024-02-05T06:33:35.598593+0000","last_became_peered":"2024-02-05T06:33:35.598593+0000","last_unstale":"2024-02-05T06:35:02.904536+0000","last_undegraded":"2024-02-05T06:35:02.904536+0000","last_fullsized":"2024-02-05T06:35:02.904536+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:49.298233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:49.298233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:07.506646+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":64,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:04.582759+0000","last_change":"2024-02-05T06:35:04.582794+0000","last_active":"2024-02-05T06:35:04.582759+0000","last_peered":"2024-02-05T06:35:04.582759+0000","last_clean":"2024-02-05T06:35:04.582759+0000","last_became_active":"2024-02-05T06:33:33.586598+0000","last_became_peered":"2024-02-05T06:33:33.586598+0000","last_unstale":"2024-02-05T06:35:04.582759+0000","last_undegraded":"2024-02-05T06:35:04.582759+0000","last_fullsized":"2024-02-05T06:35:04.582759+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:04.582689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:35:04.582689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:18.560488+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":72,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.226326+0000","last_change":"2024-02-05T06:34:57.291015+0000","last_active":"2024-02-05T06:35:03.226326+0000","last_peered":"2024-02-05T06:35:03.226326+0000","last_clean":"2024-02-05T06:35:03.226326+0000","last_became_active":"2024-02-05T06:33:29.550612+0000","last_became_peered":"2024-02-05T06:33:29.550612+0000","last_unstale":"2024-02-05T06:35:03.226326+0000","last_undegraded":"2024-02-05T06:35:03.226326+0000","last_fullsized":"2024-02-05T06:35:03.226326+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:57.290944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:57.290944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:23.780005+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"73'17","reported_seq":88,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.905192+0000","last_change":"2024-02-05T06:33:31.566941+0000","last_active":"2024-02-05T06:35:02.905192+0000","last_peered":"2024-02-05T06:35:02.905192+0000","last_clean":"2024-02-05T06:35:02.905192+0000","last_became_active":"2024-02-05T06:33:31.566576+0000","last_became_peered":"2024-02-05T06:33:31.566576+0000","last_unstale":"2024-02-05T06:35:02.905192+0000","last_undegraded":"2024-02-05T06:35:02.905192+0000","last_fullsized":"2024-02-05T06:35:02.905192+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:33:30.545651+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":52,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.905221+0000","last_change":"2024-02-05T06:33:35.597953+0000","last_active":"2024-02-05T06:35:02.905221+0000","last_peered":"2024-02-05T06:35:02.905221+0000","last_clean":"2024-02-05T06:35:02.905221+0000","last_became_active":"2024-02-05T06:33:35.597460+0000","last_became_peered":"2024-02-05T06:33:35.597460+0000","last_unstale":"2024-02-05T06:35:02.905221+0000","last_undegraded":"2024-02-05T06:35:02.905221+0000","last_fullsized":"2024-02-05T06:35:02.905221+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"92'9","reported_seq":92,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:02.903875+0000","last_change":"2024-02-05T06:34:52.283858+0000","last_active":"2024-02-05T06:35:02.903875+0000","last_peered":"2024-02-05T06:35:02.903875+0000","last_clean":"2024-02-05T06:35:02.903875+0000","last_became_active":"2024-02-05T06:33:33.588478+0000","last_became_peered":"2024-02-05T06:33:33.588478+0000","last_unstale":"2024-02-05T06:35:02.903875+0000","last_undegraded":"2024-02-05T06:35:02.903875+0000","last_fullsized":"2024-02-05T06:35:02.903875+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'9","last_scrub_stamp":"2024-02-05T06:34:52.283780+0000","last_deep_scrub":"92'9","last_deep_scrub_stamp":"2024-02-05T06:34:52.283780+0000","last_clean_scrub_stamp":"2024-02-05T06:34:52.283780+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:54.817931+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":73,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:03.026863+0000","last_change":"2024-02-05T06:34:29.822718+0000","last_active":"2024-02-05T06:35:03.026863+0000","last_peered":"2024-02-05T06:35:03.026863+0000","last_clean":"2024-02-05T06:35:03.026863+0000","last_became_active":"2024-02-05T06:33:29.545073+0000","last_became_peered":"2024-02-05T06:33:29.545073+0000","last_unstale":"2024-02-05T06:35:03.026863+0000","last_undegraded":"2024-02-05T06:35:03.026863+0000","last_fullsized":"2024-02-05T06:35:03.026863+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:29.822634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:29.822634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:35:37.035501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"93'15","reported_seq":88,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-02-05T06:35:04.584725+0000","last_change":"2024-02-05T06:35:04.584760+0000","last_active":"2024-02-05T06:35:04.584725+0000","last_peered":"2024-02-05T06:35:04.584725+0000","last_clean":"2024-02-05T06:35:04.584725+0000","last_became_active":"2024-02-05T06:33:31.568074+0000","last_became_peered":"2024-02-05T06:33:31.568074+0000","last_unstale":"2024-02-05T06:35:04.584725+0000","last_undegraded":"2024-02-05T06:35:04.584725+0000","last_fullsized":"2024-02-05T06:35:04.584725+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'15","last_scrub_stamp":"2024-02-05T06:35:04.584657+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-02-05T06:35:04.584657+0000","last_clean_scrub_stamp":"2024-02-05T06:35:04.584657+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:14.826196+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":432,"ondisk_log_size":432,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":89,"num_read_kb":84,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103191,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32492,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712660,"statfs":{"total":95995035648,"available":95961763840,"internally_reserved":0,"allocated":1347584,"data_stored":935082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.49299999999999999}]},{"osd":1,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.52200000000000002}]},{"osd":2,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.47899999999999998}]},{"osd":3,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.44}]},{"osd":4,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.34100000000000003}]},{"osd":5,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.45200000000000001}]},{"osd":6,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.46800000000000003}]}]},{"osd":6,"up_from":49,"seq":210453397533,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32468,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712684,"statfs":{"total":95995035648,"available":95961788416,"internally_reserved":0,"allocated":1323008,"data_stored":933633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:34:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.42999999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.752,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.35099999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.71299999999999997,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.45100000000000001}]},{"osd":1,"last update":"Mon Feb 5 06:34:51 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.51900000000000002,"15min":0.52200000000000002},"min":{"1min":0.378,"5min":0.309,"15min":0.309},"max":{"1min":0.746,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57999999999999996,"15min":0.58399999999999996},"min":{"1min":0.39100000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.76000000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.40799999999999997}]},{"osd":2,"last update":"Mon Feb 5 06:34:51 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.58099999999999996,"15min":0.58199999999999996},"min":{"1min":0.36399999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.77400000000000002,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.55100000000000005,"15min":0.55200000000000005},"min":{"1min":0.38300000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.495}]},{"osd":3,"last update":"Mon Feb 5 06:34:51 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.55100000000000005,"15min":0.54700000000000004},"min":{"1min":0.42499999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.54300000000000004}]},{"osd":4,"last update":"Mon Feb 5 06:34:51 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.626,"5min":1.355,"15min":1.355},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.432,"15min":0.42199999999999999},"min":{"1min":0.28499999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.70999999999999996,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.373}]},{"osd":5,"last update":"Mon Feb 5 06:34:51 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.46100000000000002,"15min":0.45600000000000002},"min":{"1min":0.33000000000000002,"5min":0.318,"15min":0.318},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.504,"5min":0.45900000000000002,"15min":0.45100000000000001},"min":{"1min":0.38600000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.65800000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.34000000000000002}]},{"osd":7,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.38800000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27772,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717380,"statfs":{"total":95995035648,"available":95966597120,"internally_reserved":0,"allocated":774144,"data_stored":345249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.36699999999999999,"15min":0.34000000000000002},"min":{"1min":0.26800000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.77600000000000002,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.38700000000000001,"15min":0.34999999999999998},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.52800000000000002,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.40500000000000003}]},{"osd":2,"last update":"Mon Feb 5 06:34:13 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.39900000000000002,"15min":0.39200000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.59999999999999998,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.38700000000000001,"15min":0.377},"min":{"1min":0.26900000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.31900000000000001}]},{"osd":3,"last update":"Mon Feb 5 06:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.42999999999999999,"15min":0.42199999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.48799999999999999,"15min":0.47799999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.83399999999999996,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.38600000000000001}]},{"osd":4,"last update":"Mon Feb 5 06:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56799999999999995,"15min":0.56200000000000006},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.78600000000000003,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.59299999999999997,"15min":0.58099999999999996},"min":{"1min":0.443,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.72199999999999998}]},{"osd":5,"last update":"Mon Feb 5 06:34:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.92800000000000005,"15min":0.98499999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86499999999999999,"5min":8.2509999999999994,"15min":8.2509999999999994},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.624,"15min":0.63400000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.75700000000000001}]},{"osd":6,"last update":"Mon Feb 5 06:34:54 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.63800000000000001,"15min":0.64800000000000002},"min":{"1min":0.42099999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.78700000000000003,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.629,"5min":0.67100000000000004,"15min":0.67800000000000005},"min":{"1min":0.51000000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.80600000000000005,"5min":1.014,"15min":1.014},"last":0.70799999999999996}]},{"osd":7,"last update":"Mon Feb 5 06:34:16 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.61499999999999999},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":8.2219999999999995,"5min":8.2219999999999995,"15min":8.2219999999999995},"last":0.68300000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738428,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32500,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712652,"statfs":{"total":95995035648,"available":95961755648,"internally_reserved":0,"allocated":1355776,"data_stored":934838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 06:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.38900000000000001,"15min":0.33500000000000002},"min":{"1min":0.24099999999999999,"5min":0.188,"15min":0.188},"max":{"1min":4.2729999999999997,"5min":4.2729999999999997,"15min":4.2729999999999997},"last":0.245},{"interface":"front","average":{"1min":0.377,"5min":0.36599999999999999,"15min":0.33600000000000002},"min":{"1min":0.26800000000000002,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.629,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.34799999999999998}]},{"osd":2,"last update":"Mon Feb 5 06:34:11 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.435,"15min":0.41299999999999998},"min":{"1min":0.315,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.44800000000000001,"15min":0.42599999999999999},"min":{"1min":0.28999999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.29499999999999998}]},{"osd":3,"last update":"Mon Feb 5 06:34:37 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.47499999999999998,"15min":0.46200000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":4.3289999999999997,"5min":4.3289999999999997,"15min":4.3289999999999997},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45100000000000001},"min":{"1min":0.28499999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.68500000000000005,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.36899999999999999}]},{"osd":4,"last update":"Mon Feb 5 06:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59799999999999998,"15min":0.59499999999999997},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.65800000000000003,"15min":0.621},"min":{"1min":0.438,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":4.367,"5min":4.367,"15min":4.367},"last":0.753}]},{"osd":5,"last update":"Mon Feb 5 06:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.628,"15min":0.63600000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.73399999999999999,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.64200000000000002,"15min":0.64700000000000002},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.77900000000000003,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.629}]},{"osd":6,"last update":"Mon Feb 5 06:34:49 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.68100000000000005,"15min":0.65200000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":5.2220000000000004,"5min":5.2220000000000004,"15min":5.2220000000000004},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.64800000000000002,"15min":0.65200000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.66600000000000004}]},{"osd":7,"last update":"Mon Feb 5 06:34:16 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":5.2670000000000003,"5min":5.2670000000000003,"15min":5.2670000000000003},"last":0.69899999999999995}]}]},{"osd":2,"up_from":18,"seq":77309411377,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32024,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713128,"statfs":{"total":95995035648,"available":95962243072,"internally_reserved":0,"allocated":741376,"data_stored":344583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:34:15 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.38100000000000001,"15min":0.36599999999999999},"min":{"1min":0.27000000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.59999999999999998,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.41199999999999998,"15min":0.38400000000000001},"min":{"1min":0.32000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.58099999999999996,"5min":0.752,"15min":0.752},"last":0.54000000000000004}]},{"osd":1,"last update":"Mon Feb 5 06:34:15 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.39900000000000002,"15min":0.38900000000000001},"min":{"1min":0.26200000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.56000000000000005,"5min":0.626,"15min":0.626},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.44800000000000001,"15min":0.42899999999999999},"min":{"1min":0.33600000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.68300000000000005,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.61599999999999999}]},{"osd":3,"last update":"Mon Feb 5 06:34:37 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.47999999999999998,"15min":0.46999999999999997},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.72499999999999998,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.47999999999999998,"15min":0.48199999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.66200000000000003,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.63300000000000001}]},{"osd":4,"last update":"Mon Feb 5 06:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.64300000000000002,"15min":0.64500000000000002},"min":{"1min":0.41499999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.95599999999999996,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.65900000000000003,"15min":0.66400000000000003},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.86899999999999999,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.86099999999999999}]},{"osd":5,"last update":"Mon Feb 5 06:34:30 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.69199999999999995,"15min":0.70299999999999996},"min":{"1min":0.47199999999999998,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.89600000000000002,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.66600000000000004,"15min":0.68000000000000005},"min":{"1min":0.42399999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.85299999999999998,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.874}]},{"osd":6,"last update":"Mon Feb 5 06:34:54 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":1.51,"15min":1.6499999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.97999999999999998,"5min":17.143999999999998,"15min":17.143999999999998},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.69699999999999995,"15min":0.70099999999999996},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.96099999999999997,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.91900000000000004}]},{"osd":7,"last update":"Mon Feb 5 06:34:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.90200000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149740,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":765952,"data_stored":344156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:34:44 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.38600000000000001,"15min":0.39300000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.52000000000000002,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.44500000000000001,"15min":0.45400000000000001},"min":{"1min":0.316,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.55000000000000004,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.33500000000000002}]},{"osd":1,"last update":"Mon Feb 5 06:34:44 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.45800000000000002,"15min":0.47199999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.57099999999999995,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.44500000000000001,"15min":0.45300000000000001},"min":{"1min":0.27700000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.57599999999999996,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.28299999999999997}]},{"osd":2,"last update":"Mon Feb 5 06:34:44 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.48099999999999998,"15min":0.495},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.64600000000000002,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.378},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.46000000000000002,"15min":0.46899999999999997},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.64800000000000002,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.30499999999999999}]},{"osd":4,"last update":"Mon Feb 5 06:34:05 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.59899999999999998,"15min":0.61499999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.627,"5min":1.008,"15min":1.008},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.59099999999999997,"15min":0.60399999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.67900000000000005,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.58799999999999997}]},{"osd":5,"last update":"Mon Feb 5 06:34:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.60099999999999998,"15min":0.60699999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.73999999999999999,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.61699999999999999,"15min":0.627},"min":{"1min":0.44,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.748,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.52700000000000002}]},{"osd":6,"last update":"Mon Feb 5 06:34:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.5},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63100000000000001,"15min":0.63500000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.88700000000000001,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.60499999999999998}]},{"osd":7,"last update":"Mon Feb 5 06:34:21 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.57599999999999996}]}]},{"osd":4,"up_from":31,"seq":133143986214,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27884,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717268,"statfs":{"total":95995035648,"available":95966482432,"internally_reserved":0,"allocated":761856,"data_stored":344268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.57099999999999995,"15min":0.58699999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.78100000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.55200000000000005,"15min":0.54400000000000004},"min":{"1min":0.40699999999999997,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.44}]},{"osd":1,"last update":"Mon Feb 5 06:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55700000000000005,"15min":0.56100000000000005},"min":{"1min":0.40799999999999997,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.58499999999999996,"15min":0.59799999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.40799999999999997}]},{"osd":2,"last update":"Mon Feb 5 06:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.56399999999999995,"15min":0.56699999999999995},"min":{"1min":0.45800000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1,"5min":1,"15min":1},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.58299999999999996,"15min":0.58799999999999997},"min":{"1min":0.45400000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.94899999999999995,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.48599999999999999}]},{"osd":3,"last update":"Mon Feb 5 06:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.59299999999999997,"15min":0.59899999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.83999999999999997,"5min":1.375,"15min":1.375},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57599999999999996,"15min":0.58499999999999996},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.81000000000000005,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.55400000000000005}]},{"osd":5,"last update":"Mon Feb 5 06:34:28 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.40999999999999998,"15min":0.40200000000000002},"min":{"1min":0.27500000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.442,"15min":0.439},"min":{"1min":0.317,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.45900000000000002}]},{"osd":6,"last update":"Mon Feb 5 06:34:52 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.504,"15min":0.49299999999999999},"min":{"1min":0.371,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.79600000000000004,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.5},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.48899999999999999,"15min":0.47499999999999998},"min":{"1min":0.32300000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.51100000000000001}]},{"osd":7,"last update":"Mon Feb 5 06:34:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.56100000000000005}]}]},{"osd":5,"up_from":40,"seq":171798691873,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27756,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717396,"statfs":{"total":95995035648,"available":95966613504,"internally_reserved":0,"allocated":757760,"data_stored":344835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.53100000000000003,"15min":0.53400000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.51600000000000001},{"interface":"front","average":{"1min":1.45,"5min":0.72999999999999998,"15min":0.60999999999999999},"min":{"1min":0.35399999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":17.509,"5min":17.509,"15min":17.509},"last":0.64900000000000002}]},{"osd":1,"last update":"Mon Feb 5 06:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.56100000000000005,"15min":0.56200000000000006},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.752,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.54100000000000004,"15min":0.54400000000000004},"min":{"1min":0.37,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.66400000000000003}]},{"osd":2,"last update":"Mon Feb 5 06:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54800000000000004,"15min":0.55000000000000004},"min":{"1min":0.41499999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70299999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.53700000000000003,"15min":0.53600000000000003},"min":{"1min":0.40300000000000002,"5min":0.375,"15min":0.375},"max":{"1min":0.70299999999999996,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.53100000000000003}]},{"osd":3,"last update":"Mon Feb 5 06:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.55300000000000005,"15min":0.55900000000000005},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.66900000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.57699999999999996,"15min":0.58099999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.72599999999999998,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.58099999999999996}]},{"osd":4,"last update":"Mon Feb 5 06:34:27 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.39500000000000002,"15min":0.40100000000000002},"min":{"1min":0.29799999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.45300000000000001,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.41499999999999998,"15min":0.41899999999999998},"min":{"1min":0.32300000000000001,"5min":0.313,"15min":0.313},"max":{"1min":0.53600000000000003,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.63600000000000001}]},{"osd":6,"last update":"Mon Feb 5 06:34:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.46000000000000002,"15min":0.45000000000000001},"min":{"1min":0.33900000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.47899999999999998,"15min":0.47099999999999997},"min":{"1min":0.373,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.38700000000000001}]},{"osd":7,"last update":"Mon Feb 5 06:34:16 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.5},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.40899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T06:35:07.253 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-05T06:35:07.253 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd dump 2024-02-05T06:35:07.479 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:07 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=infra.usagestats.collector t=2024-02-05T06:35:07.387447804Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-02-05T06:35:07.479 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:07 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=provisioning.datasources t=2024-02-05T06:35:07.388483173Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-02-05T06:35:07.479 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:07 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=provisioning.datasources t=2024-02-05T06:35:07.399674534Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-02-05T06:35:07.479 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:07 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=provisioning.alerting t=2024-02-05T06:35:07.410737508Z level=info msg="starting to provision alerting" 2024-02-05T06:35:07.479 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:07 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=provisioning.alerting t=2024-02-05T06:35:07.410775949Z level=info msg="finished to provision alerting" 2024-02-05T06:35:07.479 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:07 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=ngalert.state.manager t=2024-02-05T06:35:07.410864005Z level=info msg="Warming state cache for startup" 2024-02-05T06:35:07.479 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:07 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=ngalert.state.manager t=2024-02-05T06:35:07.411301851Z level=info msg="State cache has been initialized" states=0 duration=437.006µs 2024-02-05T06:35:07.479 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:07 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=ticker t=2024-02-05T06:35:07.411426056Z level=info msg=starting first_tick=2024-02-05T06:35:10Z 2024-02-05T06:35:07.479 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:07 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=grafanaStorageLogger t=2024-02-05T06:35:07.410955154Z level=info msg="storage starting" 2024-02-05T06:35:07.479 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:07 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=ngalert.multiorg.alertmanager t=2024-02-05T06:35:07.411465525Z level=info msg="starting MultiOrg Alertmanager" 2024-02-05T06:35:07.480 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:07 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=http.server t=2024-02-05T06:35:07.412791719Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-02-05T06:35:07.480 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:07 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=sqlstore.transactions t=2024-02-05T06:35:07.423585441Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-02-05T06:35:07.480 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:35:07 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=sqlstore.transactions t=2024-02-05T06:35:07.438495841Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-02-05T06:35:07.531 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:35:07.653 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:07 smithi028 bash[73807]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-02-05T06:35:08.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:07 smithi028 ceph-mon[29442]: 3.13 scrub starts 2024-02-05T06:35:08.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:07 smithi028 ceph-mon[29442]: 3.13 scrub ok 2024-02-05T06:35:08.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:07 smithi028 ceph-mon[29442]: 5.19 deep-scrub starts 2024-02-05T06:35:08.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:07 smithi028 ceph-mon[29442]: 5.19 deep-scrub ok 2024-02-05T06:35:08.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:07 smithi028 ceph-mon[29442]: 3.18 scrub starts 2024-02-05T06:35:08.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:07 smithi028 ceph-mon[29442]: 3.18 scrub ok 2024-02-05T06:35:08.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:07 smithi028 ceph-mon[37984]: 3.13 scrub starts 2024-02-05T06:35:08.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:07 smithi028 ceph-mon[37984]: 3.13 scrub ok 2024-02-05T06:35:08.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:07 smithi028 ceph-mon[37984]: 5.19 deep-scrub starts 2024-02-05T06:35:08.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:07 smithi028 ceph-mon[37984]: 5.19 deep-scrub ok 2024-02-05T06:35:08.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:07 smithi028 ceph-mon[37984]: 3.18 scrub starts 2024-02-05T06:35:08.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:07 smithi028 ceph-mon[37984]: 3.18 scrub ok 2024-02-05T06:35:08.195 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:07 smithi028 bash[73807]: Getting image source signatures 2024-02-05T06:35:08.196 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:07 smithi028 bash[73807]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-02-05T06:35:08.196 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:07 smithi028 bash[73807]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-02-05T06:35:08.196 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:07 smithi028 bash[73807]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-02-05T06:35:08.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:07 smithi167 ceph-mon[32774]: 3.13 scrub starts 2024-02-05T06:35:08.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:07 smithi167 ceph-mon[32774]: 3.13 scrub ok 2024-02-05T06:35:08.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:07 smithi167 ceph-mon[32774]: 5.19 deep-scrub starts 2024-02-05T06:35:08.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:07 smithi167 ceph-mon[32774]: 5.19 deep-scrub ok 2024-02-05T06:35:08.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:07 smithi167 ceph-mon[32774]: 3.18 scrub starts 2024-02-05T06:35:08.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:07 smithi167 ceph-mon[32774]: 3.18 scrub ok 2024-02-05T06:35:09.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:08 smithi028 ceph-mon[29442]: 3.15 scrub starts 2024-02-05T06:35:09.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:08 smithi028 ceph-mon[29442]: 3.15 scrub ok 2024-02-05T06:35:09.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:08 smithi028 ceph-mon[29442]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:35:09.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:08 smithi028 ceph-mon[37984]: 3.15 scrub starts 2024-02-05T06:35:09.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:08 smithi028 ceph-mon[37984]: 3.15 scrub ok 2024-02-05T06:35:09.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:08 smithi028 ceph-mon[37984]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:35:09.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:08 smithi167 ceph-mon[32774]: 3.15 scrub starts 2024-02-05T06:35:09.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:08 smithi167 ceph-mon[32774]: 3.15 scrub ok 2024-02-05T06:35:09.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:08 smithi167 ceph-mon[32774]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:35:10.049 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:09 smithi028 ceph-mon[37984]: 3.e scrub starts 2024-02-05T06:35:10.049 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:09 smithi028 ceph-mon[37984]: 3.e scrub ok 2024-02-05T06:35:10.050 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:09 smithi028 ceph-mon[37984]: 5.1e deep-scrub starts 2024-02-05T06:35:10.050 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:09 smithi028 ceph-mon[37984]: 5.1e deep-scrub ok 2024-02-05T06:35:10.050 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:09 smithi028 ceph-mon[37984]: 3.c scrub starts 2024-02-05T06:35:10.050 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:09 smithi028 ceph-mon[37984]: 3.c scrub ok 2024-02-05T06:35:10.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:09 smithi028 ceph-mon[29442]: 3.e scrub starts 2024-02-05T06:35:10.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:09 smithi028 ceph-mon[29442]: 3.e scrub ok 2024-02-05T06:35:10.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:09 smithi028 ceph-mon[29442]: 5.1e deep-scrub starts 2024-02-05T06:35:10.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:09 smithi028 ceph-mon[29442]: 5.1e deep-scrub ok 2024-02-05T06:35:10.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:09 smithi028 ceph-mon[29442]: 3.c scrub starts 2024-02-05T06:35:10.050 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:09 smithi028 ceph-mon[29442]: 3.c scrub ok 2024-02-05T06:35:10.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:09 smithi167 ceph-mon[32774]: 3.e scrub starts 2024-02-05T06:35:10.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:09 smithi167 ceph-mon[32774]: 3.e scrub ok 2024-02-05T06:35:10.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:09 smithi167 ceph-mon[32774]: 5.1e deep-scrub starts 2024-02-05T06:35:10.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:09 smithi167 ceph-mon[32774]: 5.1e deep-scrub ok 2024-02-05T06:35:10.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:09 smithi167 ceph-mon[32774]: 3.c scrub starts 2024-02-05T06:35:10.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:09 smithi167 ceph-mon[32774]: 3.c scrub ok 2024-02-05T06:35:10.392 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 93 2024-02-05T06:35:10.392 INFO:teuthology.orchestra.run.smithi028.stdout:fsid 9801e956-c3ef-11ee-95b6-87774f69a715 2024-02-05T06:35:10.392 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-02-05T06:27:59.919226+0000 2024-02-05T06:35:10.393 INFO:teuthology.orchestra.run.smithi028.stdout:modified 2024-02-05T06:35:02.897514+0000 2024-02-05T06:35:10.393 INFO:teuthology.orchestra.run.smithi028.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-02-05T06:35:10.393 INFO:teuthology.orchestra.run.smithi028.stdout:crush_version 18 2024-02-05T06:35:10.393 INFO:teuthology.orchestra.run.smithi028.stdout:full_ratio 0.95 2024-02-05T06:35:10.393 INFO:teuthology.orchestra.run.smithi028.stdout:backfillfull_ratio 0.9 2024-02-05T06:35:10.394 INFO:teuthology.orchestra.run.smithi028.stdout:nearfull_ratio 0.85 2024-02-05T06:35:10.394 INFO:teuthology.orchestra.run.smithi028.stdout:require_min_compat_client luminous 2024-02-05T06:35:10.394 INFO:teuthology.orchestra.run.smithi028.stdout:min_compat_client luminous 2024-02-05T06:35:10.394 INFO:teuthology.orchestra.run.smithi028.stdout:require_osd_release squid 2024-02-05T06:35:10.394 INFO:teuthology.orchestra.run.smithi028.stdout:stretch_mode_enabled false 2024-02-05T06:35:10.394 INFO:teuthology.orchestra.run.smithi028.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-02-05T06:35:10.395 INFO:teuthology.orchestra.run.smithi028.stdout:pool 2 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 69 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-02-05T06:35:10.395 INFO:teuthology.orchestra.run.smithi028.stdout:pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 67 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-02-05T06:35:10.395 INFO:teuthology.orchestra.run.smithi028.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 69 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-02-05T06:35:10.395 INFO:teuthology.orchestra.run.smithi028.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 71 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-02-05T06:35:10.395 INFO:teuthology.orchestra.run.smithi028.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 73 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-02-05T06:35:10.396 INFO:teuthology.orchestra.run.smithi028.stdout:max_osd 8 2024-02-05T06:35:10.396 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 up in weight 1 up_from 8 up_thru 71 down_at 0 last_clean_interval [0,0) v1:172.21.15.28:6801/301279106 v1:172.21.15.28:6802/301279106 exists,up c19fdd5b-09f6-45d4-a519-0a938cf1f2a6 2024-02-05T06:35:10.396 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 up in weight 1 up_from 13 up_thru 71 down_at 0 last_clean_interval [0,0) v1:172.21.15.28:6805/1795224649 v1:172.21.15.28:6806/1795224649 exists,up 52245f6e-eb96-4d0d-985b-dbb6fd3ce8de 2024-02-05T06:35:10.396 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 up in weight 1 up_from 18 up_thru 71 down_at 0 last_clean_interval [0,0) v1:172.21.15.28:6809/1993280018 v1:172.21.15.28:6810/1993280018 exists,up 1e18b630-afdf-4de9-a1a2-37bdfdd01c67 2024-02-05T06:35:10.396 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 up in weight 1 up_from 26 up_thru 77 down_at 0 last_clean_interval [0,0) v1:172.21.15.28:6813/1294621398 v1:172.21.15.28:6814/1294621398 exists,up 3ae795af-3d59-41a6-95eb-b8447ff61282 2024-02-05T06:35:10.397 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 up in weight 1 up_from 31 up_thru 77 down_at 0 last_clean_interval [0,0) v1:172.21.15.167:6800/4052090595 v1:172.21.15.167:6801/4052090595 exists,up fc6bd83d-570c-48c0-964d-e09b4608bc32 2024-02-05T06:35:10.397 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 up in weight 1 up_from 40 up_thru 77 down_at 0 last_clean_interval [0,0) v1:172.21.15.167:6804/1631080500 v1:172.21.15.167:6805/1631080500 exists,up 895bf465-2531-4d9f-bc23-1e75fd969f07 2024-02-05T06:35:10.397 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 up in weight 1 up_from 49 up_thru 69 down_at 0 last_clean_interval [0,0) v1:172.21.15.167:6808/1644050617 v1:172.21.15.167:6809/1644050617 exists,up ce0a9300-87a4-4e37-b702-d667f168c0e0 2024-02-05T06:35:10.397 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 up in weight 1 up_from 58 up_thru 71 down_at 0 last_clean_interval [0,0) v1:172.21.15.167:6812/4031475764 v1:172.21.15.167:6813/4031475764 exists,up 0c545cbc-0873-4232-8807-4af9299ef95e 2024-02-05T06:35:10.397 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 4.4 [1,5] 2024-02-05T06:35:10.398 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 4.e [1,2] 2024-02-05T06:35:10.398 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 4.1d [1,2] 2024-02-05T06:35:10.398 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6800/1001442173 expires 2024-02-06T06:34:10.127914+0000 2024-02-05T06:35:10.398 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/3160026720 expires 2024-02-06T06:34:10.127914+0000 2024-02-05T06:35:10.398 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/558171292 expires 2024-02-06T06:34:10.127914+0000 2024-02-05T06:35:10.399 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/2832437766 expires 2024-02-06T06:34:10.127914+0000 2024-02-05T06:35:10.399 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6800/4125764599 expires 2024-02-06T06:28:54.105170+0000 2024-02-05T06:35:10.399 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/1081543653 expires 2024-02-06T06:28:54.105170+0000 2024-02-05T06:35:10.399 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/3794420554 expires 2024-02-06T06:28:27.553034+0000 2024-02-05T06:35:10.399 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/1560471317 expires 2024-02-06T06:34:10.127914+0000 2024-02-05T06:35:10.399 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6800/4266601437 expires 2024-02-06T06:28:27.553034+0000 2024-02-05T06:35:10.400 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/388893831 expires 2024-02-06T06:28:27.553034+0000 2024-02-05T06:35:10.400 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/460817910 expires 2024-02-06T06:28:27.553034+0000 2024-02-05T06:35:10.400 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/2728305743 expires 2024-02-06T06:28:54.105170+0000 2024-02-05T06:35:10.400 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/3149394045 expires 2024-02-06T06:34:10.127914+0000 2024-02-05T06:35:10.400 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/3605013594 expires 2024-02-06T06:28:54.105170+0000 2024-02-05T06:35:10.694 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:10 smithi028 bash[73807]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-02-05T06:35:11.194 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:10 smithi028 bash[73807]: Writing manifest to image destination 2024-02-05T06:35:11.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:10 smithi028 ceph-mon[37984]: 6.10 scrub starts 2024-02-05T06:35:11.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:10 smithi028 ceph-mon[37984]: 6.10 scrub ok 2024-02-05T06:35:11.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:10 smithi028 ceph-mon[37984]: 3.7 scrub starts 2024-02-05T06:35:11.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:10 smithi028 ceph-mon[37984]: 3.7 scrub ok 2024-02-05T06:35:11.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:10 smithi028 ceph-mon[37984]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:35:11.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:10 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:11.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:10 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:35:11.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:10 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/4236876281' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-05T06:35:11.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:10 smithi028 ceph-mon[29442]: 6.10 scrub starts 2024-02-05T06:35:11.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:10 smithi028 ceph-mon[29442]: 6.10 scrub ok 2024-02-05T06:35:11.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:10 smithi028 ceph-mon[29442]: 3.7 scrub starts 2024-02-05T06:35:11.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:10 smithi028 ceph-mon[29442]: 3.7 scrub ok 2024-02-05T06:35:11.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:10 smithi028 ceph-mon[29442]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:35:11.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:10 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:11.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:10 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:35:11.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:10 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/4236876281' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-05T06:35:11.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:10 smithi167 ceph-mon[32774]: 6.10 scrub starts 2024-02-05T06:35:11.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:10 smithi167 ceph-mon[32774]: 6.10 scrub ok 2024-02-05T06:35:11.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:10 smithi167 ceph-mon[32774]: 3.7 scrub starts 2024-02-05T06:35:11.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:10 smithi167 ceph-mon[32774]: 3.7 scrub ok 2024-02-05T06:35:11.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:10 smithi167 ceph-mon[32774]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:35:11.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:10 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:11.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:10 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:35:11.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:10 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/4236876281' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-05T06:35:11.621 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 71 down_at 0 last_clean_interval [0,0) v1:172.21.15.28:6801/301279106 v1:172.21.15.28:6802/301279106 exists,up c19fdd5b-09f6-45d4-a519-0a938cf1f2a6', 'osd.1 up in weight 1 up_from 13 up_thru 71 down_at 0 last_clean_interval [0,0) v1:172.21.15.28:6805/1795224649 v1:172.21.15.28:6806/1795224649 exists,up 52245f6e-eb96-4d0d-985b-dbb6fd3ce8de', 'osd.2 up in weight 1 up_from 18 up_thru 71 down_at 0 last_clean_interval [0,0) v1:172.21.15.28:6809/1993280018 v1:172.21.15.28:6810/1993280018 exists,up 1e18b630-afdf-4de9-a1a2-37bdfdd01c67', 'osd.3 up in weight 1 up_from 26 up_thru 77 down_at 0 last_clean_interval [0,0) v1:172.21.15.28:6813/1294621398 v1:172.21.15.28:6814/1294621398 exists,up 3ae795af-3d59-41a6-95eb-b8447ff61282', 'osd.4 up in weight 1 up_from 31 up_thru 77 down_at 0 last_clean_interval [0,0) v1:172.21.15.167:6800/4052090595 v1:172.21.15.167:6801/4052090595 exists,up fc6bd83d-570c-48c0-964d-e09b4608bc32', 'osd.5 up in weight 1 up_from 40 up_thru 77 down_at 0 last_clean_interval [0,0) v1:172.21.15.167:6804/1631080500 v1:172.21.15.167:6805/1631080500 exists,up 895bf465-2531-4d9f-bc23-1e75fd969f07', 'osd.6 up in weight 1 up_from 49 up_thru 69 down_at 0 last_clean_interval [0,0) v1:172.21.15.167:6808/1644050617 v1:172.21.15.167:6809/1644050617 exists,up ce0a9300-87a4-4e37-b702-d667f168c0e0', 'osd.7 up in weight 1 up_from 58 up_thru 71 down_at 0 last_clean_interval [0,0) v1:172.21.15.167:6812/4031475764 v1:172.21.15.167:6813/4031475764 exists,up 0c545cbc-0873-4232-8807-4af9299ef95e'] 2024-02-05T06:35:11.621 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon mon.a config show 2024-02-05T06:35:11.813 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 podman[73807]: 2024-02-05 06:35:11.478238055 +0000 UTC m=+4.160616752 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-02-05T06:35:11.813 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 podman[73807]: 2024-02-05 06:35:11.685165272 +0000 UTC m=+4.367543966 container create f4251001d32cc3f587827207435d243e11ea8e4ae21e1dca8cf7818871588f33 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-05T06:35:12.067 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:35:12.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-mon[29442]: 4.4 scrub starts 2024-02-05T06:35:12.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-mon[29442]: 4.4 scrub ok 2024-02-05T06:35:12.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-mon[37984]: 4.4 scrub starts 2024-02-05T06:35:12.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-mon[37984]: 4.4 scrub ok 2024-02-05T06:35:12.193 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 podman[73807]: 2024-02-05 06:35:11.912372267 +0000 UTC m=+4.594750968 container init f4251001d32cc3f587827207435d243e11ea8e4ae21e1dca8cf7818871588f33 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-05T06:35:12.193 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 podman[73807]: 2024-02-05 06:35:11.91603152 +0000 UTC m=+4.598410214 container start f4251001d32cc3f587827207435d243e11ea8e4ae21e1dca8cf7818871588f33 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-05T06:35:12.193 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.921Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-02-05T06:35:12.193 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.921Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-02-05T06:35:12.193 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-02-05T06:35:12.193 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-02-05T06:35:12.193 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-02-05T06:35:12.193 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-05T06:35:12.194 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-02-05T06:35:12.194 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=node_exporter.go:117 level=info collector=arp 2024-02-05T06:35:12.194 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=node_exporter.go:117 level=info collector=bcache 2024-02-05T06:35:12.194 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=node_exporter.go:117 level=info collector=bonding 2024-02-05T06:35:12.194 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=node_exporter.go:117 level=info collector=btrfs 2024-02-05T06:35:12.194 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=node_exporter.go:117 level=info collector=conntrack 2024-02-05T06:35:12.194 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=node_exporter.go:117 level=info collector=cpu 2024-02-05T06:35:12.194 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-02-05T06:35:12.194 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=node_exporter.go:117 level=info collector=diskstats 2024-02-05T06:35:12.194 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=node_exporter.go:117 level=info collector=dmi 2024-02-05T06:35:12.194 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=node_exporter.go:117 level=info collector=edac 2024-02-05T06:35:12.194 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=node_exporter.go:117 level=info collector=entropy 2024-02-05T06:35:12.194 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-02-05T06:35:12.194 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=node_exporter.go:117 level=info collector=filefd 2024-02-05T06:35:12.194 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=node_exporter.go:117 level=info collector=filesystem 2024-02-05T06:35:12.195 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=node_exporter.go:117 level=info collector=hwmon 2024-02-05T06:35:12.195 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=node_exporter.go:117 level=info collector=infiniband 2024-02-05T06:35:12.195 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=node_exporter.go:117 level=info collector=ipvs 2024-02-05T06:35:12.195 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=node_exporter.go:117 level=info collector=loadavg 2024-02-05T06:35:12.195 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=node_exporter.go:117 level=info collector=mdadm 2024-02-05T06:35:12.195 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=node_exporter.go:117 level=info collector=meminfo 2024-02-05T06:35:12.195 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=node_exporter.go:117 level=info collector=netclass 2024-02-05T06:35:12.195 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=node_exporter.go:117 level=info collector=netdev 2024-02-05T06:35:12.195 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=node_exporter.go:117 level=info collector=netstat 2024-02-05T06:35:12.195 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=node_exporter.go:117 level=info collector=nfs 2024-02-05T06:35:12.195 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=node_exporter.go:117 level=info collector=nfsd 2024-02-05T06:35:12.195 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=node_exporter.go:117 level=info collector=nvme 2024-02-05T06:35:12.195 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=node_exporter.go:117 level=info collector=os 2024-02-05T06:35:12.195 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-02-05T06:35:12.195 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=node_exporter.go:117 level=info collector=pressure 2024-02-05T06:35:12.196 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=node_exporter.go:117 level=info collector=rapl 2024-02-05T06:35:12.196 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.922Z caller=node_exporter.go:117 level=info collector=schedstat 2024-02-05T06:35:12.196 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.923Z caller=node_exporter.go:117 level=info collector=selinux 2024-02-05T06:35:12.196 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.923Z caller=node_exporter.go:117 level=info collector=sockstat 2024-02-05T06:35:12.196 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.923Z caller=node_exporter.go:117 level=info collector=softnet 2024-02-05T06:35:12.196 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.923Z caller=node_exporter.go:117 level=info collector=stat 2024-02-05T06:35:12.196 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.923Z caller=node_exporter.go:117 level=info collector=tapestats 2024-02-05T06:35:12.196 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.923Z caller=node_exporter.go:117 level=info collector=textfile 2024-02-05T06:35:12.196 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.923Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-02-05T06:35:12.196 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.923Z caller=node_exporter.go:117 level=info collector=time 2024-02-05T06:35:12.196 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.923Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-02-05T06:35:12.196 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.923Z caller=node_exporter.go:117 level=info collector=uname 2024-02-05T06:35:12.196 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.923Z caller=node_exporter.go:117 level=info collector=vmstat 2024-02-05T06:35:12.197 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.923Z caller=node_exporter.go:117 level=info collector=xfs 2024-02-05T06:35:12.197 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.923Z caller=node_exporter.go:117 level=info collector=zfs 2024-02-05T06:35:12.197 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.923Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-02-05T06:35:12.197 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a[74027]: ts=2024-02-05T06:35:11.923Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-02-05T06:35:12.197 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:12 smithi028 bash[73807]: f4251001d32cc3f587827207435d243e11ea8e4ae21e1dca8cf7818871588f33 2024-02-05T06:35:12.197 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:35:12 smithi028 systemd[1]: Started Ceph node-exporter.a for 9801e956-c3ef-11ee-95b6-87774f69a715. 2024-02-05T06:35:12.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:11 smithi167 ceph-mon[32774]: 4.4 scrub starts 2024-02-05T06:35:12.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:11 smithi167 ceph-mon[32774]: 4.4 scrub ok 2024-02-05T06:35:12.695 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:35:12 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:35:12.944 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:35:12 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:35:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:35:13.366 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:13 smithi028 ceph-mon[37984]: 1.0 scrub starts 2024-02-05T06:35:13.366 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:13 smithi028 ceph-mon[37984]: 1.0 scrub ok 2024-02-05T06:35:13.366 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:13 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:13.366 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:13 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:13.366 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:13 smithi028 ceph-mon[37984]: 6.b scrub starts 2024-02-05T06:35:13.366 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:13 smithi028 ceph-mon[37984]: 6.b scrub ok 2024-02-05T06:35:13.366 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:13 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:13.366 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:13 smithi028 ceph-mon[37984]: Deploying daemon node-exporter.b on smithi167 2024-02-05T06:35:13.366 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:13 smithi028 ceph-mon[37984]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:35:13.367 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:13 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:35:13.367 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:13 smithi028 ceph-mon[37984]: 6.17 scrub starts 2024-02-05T06:35:13.367 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:13 smithi028 ceph-mon[37984]: 6.17 scrub ok 2024-02-05T06:35:13.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:13 smithi028 ceph-mon[29442]: 1.0 scrub starts 2024-02-05T06:35:13.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:13 smithi028 ceph-mon[29442]: 1.0 scrub ok 2024-02-05T06:35:13.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:13 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:13.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:13 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:13.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:13 smithi028 ceph-mon[29442]: 6.b scrub starts 2024-02-05T06:35:13.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:13 smithi028 ceph-mon[29442]: 6.b scrub ok 2024-02-05T06:35:13.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:13 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:13.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:13 smithi028 ceph-mon[29442]: Deploying daemon node-exporter.b on smithi167 2024-02-05T06:35:13.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:13 smithi028 ceph-mon[29442]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:35:13.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:13 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:35:13.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:13 smithi028 ceph-mon[29442]: 6.17 scrub starts 2024-02-05T06:35:13.368 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:13 smithi028 ceph-mon[29442]: 6.17 scrub ok 2024-02-05T06:35:13.425 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:13 smithi167 ceph-mon[32774]: 1.0 scrub starts 2024-02-05T06:35:13.425 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:13 smithi167 ceph-mon[32774]: 1.0 scrub ok 2024-02-05T06:35:13.426 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:13 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:13.426 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:13 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:13.426 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:13 smithi167 ceph-mon[32774]: 6.b scrub starts 2024-02-05T06:35:13.426 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:13 smithi167 ceph-mon[32774]: 6.b scrub ok 2024-02-05T06:35:13.426 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:13 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:13.426 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:13 smithi167 ceph-mon[32774]: Deploying daemon node-exporter.b on smithi167 2024-02-05T06:35:13.426 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:13 smithi167 ceph-mon[32774]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:35:13.426 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:13 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:35:13.426 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:13 smithi167 ceph-mon[32774]: 6.17 scrub starts 2024-02-05T06:35:13.426 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:13 smithi167 ceph-mon[32774]: 6.17 scrub ok 2024-02-05T06:35:13.426 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:13 smithi167 systemd[1]: Starting Ceph node-exporter.b for 9801e956-c3ef-11ee-95b6-87774f69a715... 2024-02-05T06:35:13.729 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:13 smithi167 bash[63932]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-02-05T06:35:13.913 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:35:13.914 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mon.a", 2024-02-05T06:35:13.914 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster": "ceph", 2024-02-05T06:35:13.914 INFO:teuthology.orchestra.run.smithi028.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-02-05T06:35:13.914 INFO:teuthology.orchestra.run.smithi028.stdout: "admin_socket_mode": "", 2024-02-05T06:35:13.914 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-02-05T06:35:13.914 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_client_required": "cephx, none", 2024-02-05T06:35:13.914 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_cluster_required": "cephx", 2024-02-05T06:35:13.914 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_debug": "true", 2024-02-05T06:35:13.914 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-02-05T06:35:13.914 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-02-05T06:35:13.914 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_service_required": "cephx", 2024-02-05T06:35:13.914 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_service_ticket_ttl": "240.000000", 2024-02-05T06:35:13.914 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_supported": "", 2024-02-05T06:35:13.914 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_aio": "true", 2024-02-05T06:35:13.915 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_aio_max_queue_depth": "1024", 2024-02-05T06:35:13.915 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_aio_poll_ms": "250", 2024-02-05T06:35:13.915 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_aio_reap_max": "16", 2024-02-05T06:35:13.915 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_async_discard": "false", 2024-02-05T06:35:13.915 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_block_size": "4096", 2024-02-05T06:35:13.915 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_debug_aio": "false", 2024-02-05T06:35:13.915 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-02-05T06:35:13.915 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-02-05T06:35:13.915 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_debug_inflight_ios": "false", 2024-02-05T06:35:13.915 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_enable_discard": "false", 2024-02-05T06:35:13.915 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_flock_retry": "3", 2024-02-05T06:35:13.915 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_flock_retry_interval": "0.100000", 2024-02-05T06:35:13.915 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_inject_crash": "0", 2024-02-05T06:35:13.916 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_inject_crash_flush_delay": "2", 2024-02-05T06:35:13.916 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_ioring": "false", 2024-02-05T06:35:13.916 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_ioring_hipri": "false", 2024-02-05T06:35:13.916 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_ioring_sqthread_poll": "false", 2024-02-05T06:35:13.916 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-02-05T06:35:13.916 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_read_buffer_alignment": "4096", 2024-02-05T06:35:13.916 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-02-05T06:35:13.916 INFO:teuthology.orchestra.run.smithi028.stdout: "bdev_type": "", 2024-02-05T06:35:13.916 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_alloc_size": "1048576", 2024-02-05T06:35:13.916 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_allocator": "hybrid", 2024-02-05T06:35:13.916 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_buffered_io": "true", 2024-02-05T06:35:13.916 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_check_for_zeros": "false", 2024-02-05T06:35:13.916 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_check_volume_selector_often": "false", 2024-02-05T06:35:13.917 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-02-05T06:35:13.917 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_compact_log_sync": "false", 2024-02-05T06:35:13.917 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-02-05T06:35:13.917 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-02-05T06:35:13.917 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_log_compact_min_size": "16777216", 2024-02-05T06:35:13.917 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_log_replay_check_allocations": "true", 2024-02-05T06:35:13.917 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_max_log_runway": "4194304", 2024-02-05T06:35:13.917 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_max_prefetch": "1048576", 2024-02-05T06:35:13.917 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_min_flush_size": "524288", 2024-02-05T06:35:13.917 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_min_log_runway": "1048576", 2024-02-05T06:35:13.917 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_replay_recovery": "false", 2024-02-05T06:35:13.917 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-02-05T06:35:13.918 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_shared_alloc_size": "65536", 2024-02-05T06:35:13.918 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_sync_write": "false", 2024-02-05T06:35:13.918 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-02-05T06:35:13.918 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-02-05T06:35:13.918 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-02-05T06:35:13.918 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_allocation_from_file": "true", 2024-02-05T06:35:13.918 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_allocator": "hybrid", 2024-02-05T06:35:13.918 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-02-05T06:35:13.918 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-02-05T06:35:13.918 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-02-05T06:35:13.918 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-02-05T06:35:13.918 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-02-05T06:35:13.918 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-02-05T06:35:13.919 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blobid_prealloc": "10240", 2024-02-05T06:35:13.919 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_create": "true", 2024-02-05T06:35:13.919 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_db_create": "false", 2024-02-05T06:35:13.919 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_db_path": "", 2024-02-05T06:35:13.919 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_db_size": "0", 2024-02-05T06:35:13.919 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_path": "", 2024-02-05T06:35:13.919 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_preallocate_file": "false", 2024-02-05T06:35:13.919 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_size": "107374182400", 2024-02-05T06:35:13.919 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_wal_create": "false", 2024-02-05T06:35:13.919 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_wal_path": "", 2024-02-05T06:35:13.919 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_block_wal_size": "100663296", 2024-02-05T06:35:13.919 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_bluefs": "true", 2024-02-05T06:35:13.919 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-02-05T06:35:13.919 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_bluefs_env_mirror": "false", 2024-02-05T06:35:13.920 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-02-05T06:35:13.920 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-02-05T06:35:13.920 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-02-05T06:35:13.920 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-02-05T06:35:13.920 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-02-05T06:35:13.920 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-02-05T06:35:13.920 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_autotune": "true", 2024-02-05T06:35:13.920 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-02-05T06:35:13.920 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-02-05T06:35:13.920 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-02-05T06:35:13.920 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-02-05T06:35:13.920 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_size": "0", 2024-02-05T06:35:13.920 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-02-05T06:35:13.921 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-02-05T06:35:13.921 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-02-05T06:35:13.921 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-02-05T06:35:13.921 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_type": "2q", 2024-02-05T06:35:13.921 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-02-05T06:35:13.921 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_clone_cow": "true", 2024-02-05T06:35:13.921 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_algorithm": "snappy", 2024-02-05T06:35:13.921 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_max_blob_size": "0", 2024-02-05T06:35:13.921 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-02-05T06:35:13.921 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-02-05T06:35:13.921 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_min_blob_size": "0", 2024-02-05T06:35:13.921 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-02-05T06:35:13.921 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-02-05T06:35:13.921 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_mode": "none", 2024-02-05T06:35:13.922 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-02-05T06:35:13.922 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_csum_type": "crc32c", 2024-02-05T06:35:13.922 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_enforce_settings": "default", 2024-02-05T06:35:13.922 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_freelist": "false", 2024-02-05T06:35:13.922 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_fsck_abort": "false", 2024-02-05T06:35:13.922 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-02-05T06:35:13.922 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-02-05T06:35:13.922 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_inject_read_err": "false", 2024-02-05T06:35:13.922 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_legacy_omap": "false", 2024-02-05T06:35:13.922 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-02-05T06:35:13.922 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-02-05T06:35:13.922 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-02-05T06:35:13.922 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-02-05T06:35:13.922 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_prefill": "0.000000", 2024-02-05T06:35:13.923 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-02-05T06:35:13.923 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-02-05T06:35:13.923 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-02-05T06:35:13.923 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_small_allocations": "0", 2024-02-05T06:35:13.923 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-02-05T06:35:13.923 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_default_buffered_read": "true", 2024-02-05T06:35:13.923 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_default_buffered_write": "false", 2024-02-05T06:35:13.923 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_deferred_batch_ops": "0", 2024-02-05T06:35:13.923 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-02-05T06:35:13.923 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-02-05T06:35:13.923 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_elastic_shared_blobs": "true", 2024-02-05T06:35:13.923 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-02-05T06:35:13.923 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-02-05T06:35:13.924 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-02-05T06:35:13.924 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-02-05T06:35:13.924 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-02-05T06:35:13.924 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fail_eio": "false", 2024-02-05T06:35:13.924 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-02-05T06:35:13.924 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-02-05T06:35:13.924 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-02-05T06:35:13.924 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-02-05T06:35:13.924 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_on_mkfs": "true", 2024-02-05T06:35:13.924 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-02-05T06:35:13.924 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_on_mount": "false", 2024-02-05T06:35:13.924 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-02-05T06:35:13.924 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_on_umount": "false", 2024-02-05T06:35:13.924 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-02-05T06:35:13.925 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-02-05T06:35:13.925 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-02-05T06:35:13.925 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-02-05T06:35:13.925 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-02-05T06:35:13.925 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-02-05T06:35:13.925 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-02-05T06:35:13.925 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-02-05T06:35:13.925 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_ignore_data_csum": "false", 2024-02-05T06:35:13.925 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-02-05T06:35:13.925 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_kvbackend": "rocksdb", 2024-02-05T06:35:13.925 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-02-05T06:35:13.925 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-02-05T06:35:13.925 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_log_op_age": "5.000000", 2024-02-05T06:35:13.925 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_max_alloc_size": "0", 2024-02-05T06:35:13.925 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_max_blob_size": "0", 2024-02-05T06:35:13.926 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-02-05T06:35:13.926 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-02-05T06:35:13.926 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_max_defer_interval": "3.000000", 2024-02-05T06:35:13.926 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_max_deferred_txc": "32", 2024-02-05T06:35:13.926 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_min_alloc_size": "0", 2024-02-05T06:35:13.926 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-02-05T06:35:13.926 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-02-05T06:35:13.926 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_nid_prealloc": "1024", 2024-02-05T06:35:13.926 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_prefer_deferred_size": "0", 2024-02-05T06:35:13.926 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-02-05T06:35:13.926 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-02-05T06:35:13.926 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_qfsck_on_mount": "true", 2024-02-05T06:35:13.926 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_retry_disk_reads": "3", 2024-02-05T06:35:13.926 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_rocksdb_cf": "true", 2024-02-05T06:35:13.926 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-02-05T06:35:13.926 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-02-05T06:35:13.927 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_rocksdb_options_annex": "", 2024-02-05T06:35:13.927 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_spdk_coremask": "0x1", 2024-02-05T06:35:13.927 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_spdk_io_sleep": "5", 2024-02-05T06:35:13.927 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_spdk_max_io_completion": "0", 2024-02-05T06:35:13.927 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_spdk_mem": "512", 2024-02-05T06:35:13.927 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_sync_submit_transaction": "false", 2024-02-05T06:35:13.927 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_throttle_bytes": "67108864", 2024-02-05T06:35:13.927 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_throttle_cost_per_io": "0", 2024-02-05T06:35:13.927 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-02-05T06:35:13.927 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-02-05T06:35:13.927 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-02-05T06:35:13.927 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-02-05T06:35:13.927 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_tracing": "false", 2024-02-05T06:35:13.928 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-02-05T06:35:13.928 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-02-05T06:35:13.928 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_volume_selection_reserved": "0", 2024-02-05T06:35:13.928 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-02-05T06:35:13.928 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-02-05T06:35:13.928 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-02-05T06:35:13.928 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-02-05T06:35:13.928 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-02-05T06:35:13.928 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-02-05T06:35:13.928 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_zero_block_detection": "false", 2024-02-05T06:35:13.928 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-02-05T06:35:13.928 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_action_update_interval": "2", 2024-02-05T06:35:13.928 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-02-05T06:35:13.929 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-02-05T06:35:13.929 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-02-05T06:35:13.929 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-02-05T06:35:13.929 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_mount_timeout": "10", 2024-02-05T06:35:13.929 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-02-05T06:35:13.929 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-02-05T06:35:13.929 INFO:teuthology.orchestra.run.smithi028.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-02-05T06:35:13.929 INFO:teuthology.orchestra.run.smithi028.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-02-05T06:35:13.929 INFO:teuthology.orchestra.run.smithi028.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-02-05T06:35:13.929 INFO:teuthology.orchestra.run.smithi028.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-02-05T06:35:13.929 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_cluster_require_signatures": "false", 2024-02-05T06:35:13.929 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_cluster_require_version": "2", 2024-02-05T06:35:13.929 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_require_signatures": "false", 2024-02-05T06:35:13.929 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_require_version": "2", 2024-02-05T06:35:13.930 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_service_require_signatures": "false", 2024-02-05T06:35:13.930 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_service_require_version": "2", 2024-02-05T06:35:13.930 INFO:teuthology.orchestra.run.smithi028.stdout: "cephx_sign_messages": "true", 2024-02-05T06:35:13.930 INFO:teuthology.orchestra.run.smithi028.stdout: "chdir": "", 2024-02-05T06:35:13.930 INFO:teuthology.orchestra.run.smithi028.stdout: "client_acl_type": "", 2024-02-05T06:35:13.930 INFO:teuthology.orchestra.run.smithi028.stdout: "client_asio_thread_count": "2", 2024-02-05T06:35:13.930 INFO:teuthology.orchestra.run.smithi028.stdout: "client_cache_mid": "0.750000", 2024-02-05T06:35:13.930 INFO:teuthology.orchestra.run.smithi028.stdout: "client_cache_size": "16384", 2024-02-05T06:35:13.930 INFO:teuthology.orchestra.run.smithi028.stdout: "client_caps_release_delay": "5", 2024-02-05T06:35:13.930 INFO:teuthology.orchestra.run.smithi028.stdout: "client_check_pool_perm": "true", 2024-02-05T06:35:13.930 INFO:teuthology.orchestra.run.smithi028.stdout: "client_collect_and_send_global_metrics": "false", 2024-02-05T06:35:13.930 INFO:teuthology.orchestra.run.smithi028.stdout: "client_debug_force_sync_read": "false", 2024-02-05T06:35:13.930 INFO:teuthology.orchestra.run.smithi028.stdout: "client_debug_getattr_caps": "false", 2024-02-05T06:35:13.931 INFO:teuthology.orchestra.run.smithi028.stdout: "client_debug_inject_tick_delay": "0", 2024-02-05T06:35:13.931 INFO:teuthology.orchestra.run.smithi028.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-02-05T06:35:13.931 INFO:teuthology.orchestra.run.smithi028.stdout: "client_die_on_failed_remount": "false", 2024-02-05T06:35:13.931 INFO:teuthology.orchestra.run.smithi028.stdout: "client_dirsize_rbytes": "true", 2024-02-05T06:35:13.931 INFO:teuthology.orchestra.run.smithi028.stdout: "client_force_lazyio": "false", 2024-02-05T06:35:13.931 INFO:teuthology.orchestra.run.smithi028.stdout: "client_fs": "", 2024-02-05T06:35:13.931 INFO:teuthology.orchestra.run.smithi028.stdout: "client_inject_fixed_oldest_tid": "false", 2024-02-05T06:35:13.931 INFO:teuthology.orchestra.run.smithi028.stdout: "client_inject_release_failure": "false", 2024-02-05T06:35:13.931 INFO:teuthology.orchestra.run.smithi028.stdout: "client_max_inline_size": "4096", 2024-02-05T06:35:13.931 INFO:teuthology.orchestra.run.smithi028.stdout: "client_max_retries_on_remount_failure": "5", 2024-02-05T06:35:13.931 INFO:teuthology.orchestra.run.smithi028.stdout: "client_mds_namespace": "", 2024-02-05T06:35:13.931 INFO:teuthology.orchestra.run.smithi028.stdout: "client_metadata": "", 2024-02-05T06:35:13.931 INFO:teuthology.orchestra.run.smithi028.stdout: "client_mount_gid": "-1", 2024-02-05T06:35:13.932 INFO:teuthology.orchestra.run.smithi028.stdout: "client_mount_timeout": "300", 2024-02-05T06:35:13.932 INFO:teuthology.orchestra.run.smithi028.stdout: "client_mount_uid": "-1", 2024-02-05T06:35:13.932 INFO:teuthology.orchestra.run.smithi028.stdout: "client_mountpoint": "/", 2024-02-05T06:35:13.932 INFO:teuthology.orchestra.run.smithi028.stdout: "client_notify_timeout": "10", 2024-02-05T06:35:13.932 INFO:teuthology.orchestra.run.smithi028.stdout: "client_oc": "true", 2024-02-05T06:35:13.932 INFO:teuthology.orchestra.run.smithi028.stdout: "client_oc_max_dirty": "104857600", 2024-02-05T06:35:13.932 INFO:teuthology.orchestra.run.smithi028.stdout: "client_oc_max_dirty_age": "5.000000", 2024-02-05T06:35:13.932 INFO:teuthology.orchestra.run.smithi028.stdout: "client_oc_max_objects": "1000", 2024-02-05T06:35:13.932 INFO:teuthology.orchestra.run.smithi028.stdout: "client_oc_size": "209715200", 2024-02-05T06:35:13.932 INFO:teuthology.orchestra.run.smithi028.stdout: "client_oc_target_dirty": "8388608", 2024-02-05T06:35:13.932 INFO:teuthology.orchestra.run.smithi028.stdout: "client_permissions": "true", 2024-02-05T06:35:13.932 INFO:teuthology.orchestra.run.smithi028.stdout: "client_quota": "true", 2024-02-05T06:35:13.933 INFO:teuthology.orchestra.run.smithi028.stdout: "client_quota_df": "true", 2024-02-05T06:35:13.933 INFO:teuthology.orchestra.run.smithi028.stdout: "client_readahead_max_bytes": "0", 2024-02-05T06:35:13.933 INFO:teuthology.orchestra.run.smithi028.stdout: "client_readahead_max_periods": "4", 2024-02-05T06:35:13.933 INFO:teuthology.orchestra.run.smithi028.stdout: "client_readahead_min": "131072", 2024-02-05T06:35:13.933 INFO:teuthology.orchestra.run.smithi028.stdout: "client_reconnect_stale": "false", 2024-02-05T06:35:13.933 INFO:teuthology.orchestra.run.smithi028.stdout: "client_shutdown_timeout": "30", 2024-02-05T06:35:13.933 INFO:teuthology.orchestra.run.smithi028.stdout: "client_snapdir": ".snap", 2024-02-05T06:35:13.933 INFO:teuthology.orchestra.run.smithi028.stdout: "client_tick_interval": "1", 2024-02-05T06:35:13.933 INFO:teuthology.orchestra.run.smithi028.stdout: "client_trace": "", 2024-02-05T06:35:13.933 INFO:teuthology.orchestra.run.smithi028.stdout: "client_try_dentry_invalidate": "false", 2024-02-05T06:35:13.933 INFO:teuthology.orchestra.run.smithi028.stdout: "client_use_faked_inos": "false", 2024-02-05T06:35:13.933 INFO:teuthology.orchestra.run.smithi028.stdout: "client_use_random_mds": "false", 2024-02-05T06:35:13.933 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_graylog": "false", 2024-02-05T06:35:13.934 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-02-05T06:35:13.934 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_graylog_port": "12201", 2024-02-05T06:35:13.934 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_monitors": "default=true", 2024-02-05T06:35:13.934 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_syslog": "false", 2024-02-05T06:35:13.934 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-02-05T06:35:13.934 INFO:teuthology.orchestra.run.smithi028.stdout: "clog_to_syslog_level": "info", 2024-02-05T06:35:13.934 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "-", 2024-02-05T06:35:13.934 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_network": "", 2024-02-05T06:35:13.934 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_network_interface": "", 2024-02-05T06:35:13.934 INFO:teuthology.orchestra.run.smithi028.stdout: "compressor_zlib_isal": "false", 2024-02-05T06:35:13.934 INFO:teuthology.orchestra.run.smithi028.stdout: "compressor_zlib_level": "5", 2024-02-05T06:35:13.934 INFO:teuthology.orchestra.run.smithi028.stdout: "compressor_zlib_winsize": "-15", 2024-02-05T06:35:13.934 INFO:teuthology.orchestra.run.smithi028.stdout: "compressor_zstd_level": "1", 2024-02-05T06:35:13.935 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0", 2024-02-05T06:35:13.935 INFO:teuthology.orchestra.run.smithi028.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-02-05T06:35:13.935 INFO:teuthology.orchestra.run.smithi028.stdout: "crimson_alien_op_num_threads": "6", 2024-02-05T06:35:13.935 INFO:teuthology.orchestra.run.smithi028.stdout: "crimson_alien_thread_cpu_cores": "", 2024-02-05T06:35:13.935 INFO:teuthology.orchestra.run.smithi028.stdout: "crimson_osd_obc_lru_size": "512", 2024-02-05T06:35:13.935 INFO:teuthology.orchestra.run.smithi028.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-02-05T06:35:13.935 INFO:teuthology.orchestra.run.smithi028.stdout: "crimson_seastar_cpu_cores": "", 2024-02-05T06:35:13.935 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location": "", 2024-02-05T06:35:13.935 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location_hook": "", 2024-02-05T06:35:13.935 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_location_hook_timeout": "10", 2024-02-05T06:35:13.935 INFO:teuthology.orchestra.run.smithi028.stdout: "daemonize": "false", 2024-02-05T06:35:13.935 INFO:teuthology.orchestra.run.smithi028.stdout: "daos_pool": "tank", 2024-02-05T06:35:13.935 INFO:teuthology.orchestra.run.smithi028.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-02-05T06:35:13.935 INFO:teuthology.orchestra.run.smithi028.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-02-05T06:35:13.936 INFO:teuthology.orchestra.run.smithi028.stdout: "dbstore_db_name_prefix": "dbstore", 2024-02-05T06:35:13.936 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_alienstore": "0/5", 2024-02-05T06:35:13.936 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_allow_any_pool_priority": "false", 2024-02-05T06:35:13.936 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_asok": "1/5", 2024-02-05T06:35:13.936 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_asok_assert_abort": "false", 2024-02-05T06:35:13.936 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_asserts_on_shutdown": "true", 2024-02-05T06:35:13.936 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_auth": "1/5", 2024-02-05T06:35:13.936 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_bdev": "1/3", 2024-02-05T06:35:13.936 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_bluefs": "1/5", 2024-02-05T06:35:13.936 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_bluestore": "1/5", 2024-02-05T06:35:13.936 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_buffer": "0/1", 2024-02-05T06:35:13.936 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_ceph_exporter": "1/5", 2024-02-05T06:35:13.937 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_cephfs_mirror": "0/5", 2024-02-05T06:35:13.937 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_cephsqlite": "0/5", 2024-02-05T06:35:13.937 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_client": "0/5", 2024-02-05T06:35:13.937 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_compressor": "1/5", 2024-02-05T06:35:13.937 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_context": "0/1", 2024-02-05T06:35:13.937 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_crimson_interrupt": "0/5", 2024-02-05T06:35:13.937 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_crush": "1/1", 2024-02-05T06:35:13.937 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_crypto": "1/5", 2024-02-05T06:35:13.937 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_cyanstore": "0/5", 2024-02-05T06:35:13.937 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_deliberately_leak_memory": "false", 2024-02-05T06:35:13.937 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_disable_randomized_ping": "false", 2024-02-05T06:35:13.937 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_dpdk": "1/5", 2024-02-05T06:35:13.937 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_eventtrace": "1/5", 2024-02-05T06:35:13.938 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_filer": "0/1", 2024-02-05T06:35:13.938 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_filestore": "1/3", 2024-02-05T06:35:13.938 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_finisher": "1/1", 2024-02-05T06:35:13.938 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_fuse": "1/5", 2024-02-05T06:35:13.938 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_heartbeat_testing_span": "0", 2024-02-05T06:35:13.938 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_heartbeatmap": "1/5", 2024-02-05T06:35:13.938 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_immutable_obj_cache": "0/5", 2024-02-05T06:35:13.938 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_javaclient": "1/5", 2024-02-05T06:35:13.938 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_journal": "1/3", 2024-02-05T06:35:13.938 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_journaler": "0/5", 2024-02-05T06:35:13.938 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_kstore": "1/5", 2024-02-05T06:35:13.938 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_lockdep": "0/1", 2024-02-05T06:35:13.938 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mclock": "1/5", 2024-02-05T06:35:13.938 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mds": "1/5", 2024-02-05T06:35:13.939 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mds_balancer": "1/5", 2024-02-05T06:35:13.939 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mds_locker": "1/5", 2024-02-05T06:35:13.939 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mds_log": "1/5", 2024-02-05T06:35:13.939 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mds_log_expire": "1/5", 2024-02-05T06:35:13.939 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mds_migrator": "1/5", 2024-02-05T06:35:13.939 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_memstore": "1/5", 2024-02-05T06:35:13.939 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mgr": "2/5", 2024-02-05T06:35:13.939 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mgrc": "1/5", 2024-02-05T06:35:13.939 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_mon": "20/20", 2024-02-05T06:35:13.939 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_monc": "0/10", 2024-02-05T06:35:13.939 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_ms": "1/1", 2024-02-05T06:35:13.939 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_none": "0/5", 2024-02-05T06:35:13.939 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_objclass": "0/5", 2024-02-05T06:35:13.939 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_objectcacher": "0/5", 2024-02-05T06:35:13.939 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_objecter": "0/1", 2024-02-05T06:35:13.940 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_optracker": "0/5", 2024-02-05T06:35:13.940 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_osd": "1/5", 2024-02-05T06:35:13.940 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_paxos": "20/20", 2024-02-05T06:35:13.940 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_perfcounter": "1/5", 2024-02-05T06:35:13.940 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_prioritycache": "1/5", 2024-02-05T06:35:13.940 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rados": "0/5", 2024-02-05T06:35:13.940 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rbd": "0/5", 2024-02-05T06:35:13.940 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rbd_mirror": "0/5", 2024-02-05T06:35:13.940 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rbd_pwl": "0/5", 2024-02-05T06:35:13.940 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rbd_replay": "0/5", 2024-02-05T06:35:13.940 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_refs": "0/0", 2024-02-05T06:35:13.940 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_reserver": "1/1", 2024-02-05T06:35:13.940 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rgw": "1/5", 2024-02-05T06:35:13.941 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rgw_access": "1/5", 2024-02-05T06:35:13.941 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rgw_datacache": "1/5", 2024-02-05T06:35:13.941 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rgw_dbstore": "1/5", 2024-02-05T06:35:13.941 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rgw_flight": "1/5", 2024-02-05T06:35:13.941 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rgw_lifecycle": "1/5", 2024-02-05T06:35:13.941 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rgw_sync": "1/5", 2024-02-05T06:35:13.941 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_rocksdb": "4/5", 2024-02-05T06:35:13.941 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore": "0/5", 2024-02-05T06:35:13.941 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_backref": "0/5", 2024-02-05T06:35:13.941 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_cache": "0/5", 2024-02-05T06:35:13.941 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_cleaner": "0/5", 2024-02-05T06:35:13.941 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_device": "0/5", 2024-02-05T06:35:13.941 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_epm": "0/5", 2024-02-05T06:35:13.941 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-02-05T06:35:13.941 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_journal": "0/5", 2024-02-05T06:35:13.941 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_lba": "0/5", 2024-02-05T06:35:13.942 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_odata": "0/5", 2024-02-05T06:35:13.942 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_omap": "0/5", 2024-02-05T06:35:13.942 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_onode": "0/5", 2024-02-05T06:35:13.942 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_t": "0/5", 2024-02-05T06:35:13.942 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_seastore_tm": "0/5", 2024-02-05T06:35:13.942 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_striper": "0/1", 2024-02-05T06:35:13.942 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_test": "0/5", 2024-02-05T06:35:13.942 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_throttle": "1/1", 2024-02-05T06:35:13.942 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_timer": "0/1", 2024-02-05T06:35:13.942 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_tp": "0/5", 2024-02-05T06:35:13.942 INFO:teuthology.orchestra.run.smithi028.stdout: "debug_trace": "1/5", 2024-02-05T06:35:13.943 INFO:teuthology.orchestra.run.smithi028.stdout: "defer_client_eviction_on_laggy_osds": "true", 2024-02-05T06:35:13.944 INFO:teuthology.orchestra.run.smithi028.stdout: "device_failure_prediction_mode": "none", 2024-02-05T06:35:13.944 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-02-05T06:35:13.944 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-02-05T06:35:13.944 INFO:teuthology.orchestra.run.smithi028.stdout: "err_to_graylog": "false", 2024-02-05T06:35:13.944 INFO:teuthology.orchestra.run.smithi028.stdout: "err_to_journald": "false", 2024-02-05T06:35:13.944 INFO:teuthology.orchestra.run.smithi028.stdout: "err_to_stderr": "true", 2024-02-05T06:35:13.944 INFO:teuthology.orchestra.run.smithi028.stdout: "err_to_syslog": "false", 2024-02-05T06:35:13.944 INFO:teuthology.orchestra.run.smithi028.stdout: "event_tracing": "false", 2024-02-05T06:35:13.944 INFO:teuthology.orchestra.run.smithi028.stdout: "exporter_addr": "0.0.0.0", 2024-02-05T06:35:13.944 INFO:teuthology.orchestra.run.smithi028.stdout: "exporter_http_port": "9926", 2024-02-05T06:35:13.944 INFO:teuthology.orchestra.run.smithi028.stdout: "exporter_prio_limit": "5", 2024-02-05T06:35:13.944 INFO:teuthology.orchestra.run.smithi028.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-02-05T06:35:13.944 INFO:teuthology.orchestra.run.smithi028.stdout: "exporter_sort_metrics": "true", 2024-02-05T06:35:13.945 INFO:teuthology.orchestra.run.smithi028.stdout: "exporter_stats_period": "5", 2024-02-05T06:35:13.945 INFO:teuthology.orchestra.run.smithi028.stdout: "fake_statfs_for_testing": "0", 2024-02-05T06:35:13.945 INFO:teuthology.orchestra.run.smithi028.stdout: "fatal_signal_handlers": "true", 2024-02-05T06:35:13.945 INFO:teuthology.orchestra.run.smithi028.stdout: "filer_max_purge_ops": "10", 2024-02-05T06:35:13.945 INFO:teuthology.orchestra.run.smithi028.stdout: "filer_max_truncate_ops": "128", 2024-02-05T06:35:13.945 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_apply_finisher_threads": "1", 2024-02-05T06:35:13.945 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_blackhole": "false", 2024-02-05T06:35:13.945 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_btrfs_clone_range": "true", 2024-02-05T06:35:13.945 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_btrfs_snap": "true", 2024-02-05T06:35:13.945 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_caller_concurrency": "10", 2024-02-05T06:35:13.945 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_collect_device_partition_information": "true", 2024-02-05T06:35:13.945 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_commit_timeout": "600.000000", 2024-02-05T06:35:13.945 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_debug_inject_read_err": "false", 2024-02-05T06:35:13.945 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_debug_omap_check": "false", 2024-02-05T06:35:13.946 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_debug_random_read_err": "0.000000", 2024-02-05T06:35:13.946 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_debug_verify_split": "false", 2024-02-05T06:35:13.946 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_dump_file": "", 2024-02-05T06:35:13.946 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-02-05T06:35:13.946 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-02-05T06:35:13.946 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fadvise": "true", 2024-02-05T06:35:13.946 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fail_eio": "true", 2024-02-05T06:35:13.946 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fd_cache_shards": "16", 2024-02-05T06:35:13.946 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fd_cache_size": "128", 2024-02-05T06:35:13.946 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fiemap": "false", 2024-02-05T06:35:13.946 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fiemap_threshold": "4096", 2024-02-05T06:35:13.946 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-02-05T06:35:13.946 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_index_retry_probability": "0.000000", 2024-02-05T06:35:13.946 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_inject_stall": "0", 2024-02-05T06:35:13.946 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_journal_parallel": "false", 2024-02-05T06:35:13.947 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_journal_trailing": "false", 2024-02-05T06:35:13.947 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_journal_writeahead": "false", 2024-02-05T06:35:13.947 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_kill_at": "0", 2024-02-05T06:35:13.947 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-02-05T06:35:13.947 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattr_size": "0", 2024-02-05T06:35:13.947 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-02-05T06:35:13.947 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-02-05T06:35:13.947 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-02-05T06:35:13.947 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattrs": "0", 2024-02-05T06:35:13.947 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-02-05T06:35:13.947 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattrs_other": "2", 2024-02-05T06:35:13.947 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-02-05T06:35:13.947 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_sync_interval": "5.000000", 2024-02-05T06:35:13.947 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_xattr_value_size": "0", 2024-02-05T06:35:13.947 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-02-05T06:35:13.948 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-02-05T06:35:13.948 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-02-05T06:35:13.948 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_merge_threshold": "-10", 2024-02-05T06:35:13.948 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_min_sync_interval": "0.010000", 2024-02-05T06:35:13.948 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_odsync_write": "false", 2024-02-05T06:35:13.948 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_omap_backend": "rocksdb", 2024-02-05T06:35:13.948 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_omap_backend_path": "", 2024-02-05T06:35:13.948 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_omap_header_cache_size": "1024", 2024-02-05T06:35:13.948 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_ondisk_finisher_threads": "1", 2024-02-05T06:35:13.948 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-02-05T06:35:13.948 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_op_thread_timeout": "60", 2024-02-05T06:35:13.948 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_op_threads": "2", 2024-02-05T06:35:13.948 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_punch_hole": "false", 2024-02-05T06:35:13.949 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-02-05T06:35:13.949 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-02-05T06:35:13.949 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-02-05T06:35:13.949 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-02-05T06:35:13.949 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-02-05T06:35:13.949 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_max_bytes": "104857600", 2024-02-05T06:35:13.949 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-02-05T06:35:13.949 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-02-05T06:35:13.949 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-02-05T06:35:13.949 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_queue_max_ops": "50", 2024-02-05T06:35:13.949 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-02-05T06:35:13.949 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_seek_data_hole": "false", 2024-02-05T06:35:13.949 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_sloppy_crc": "false", 2024-02-05T06:35:13.949 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-02-05T06:35:13.949 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_splice": "false", 2024-02-05T06:35:13.950 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_split_multiple": "2", 2024-02-05T06:35:13.950 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_split_rand_factor": "20", 2024-02-05T06:35:13.950 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_update_to": "1000", 2024-02-05T06:35:13.950 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-02-05T06:35:13.950 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-02-05T06:35:13.950 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-02-05T06:35:13.950 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-02-05T06:35:13.950 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-02-05T06:35:13.950 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-02-05T06:35:13.950 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_enable": "true", 2024-02-05T06:35:13.950 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-02-05T06:35:13.950 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-02-05T06:35:13.950 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-02-05T06:35:13.950 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-02-05T06:35:13.950 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-02-05T06:35:13.951 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-02-05T06:35:13.951 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_xfs_extsize": "false", 2024-02-05T06:35:13.951 INFO:teuthology.orchestra.run.smithi028.stdout: "filestore_zfs_snap": "false", 2024-02-05T06:35:13.951 INFO:teuthology.orchestra.run.smithi028.stdout: "fio_dir": "/tmp/fio", 2024-02-05T06:35:13.951 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "9801e956-c3ef-11ee-95b6-87774f69a715", 2024-02-05T06:35:13.951 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_allow_other": "true", 2024-02-05T06:35:13.951 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_atomic_o_trunc": "true", 2024-02-05T06:35:13.951 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_big_writes": "true", 2024-02-05T06:35:13.951 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_debug": "false", 2024-02-05T06:35:13.951 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_default_permissions": "false", 2024-02-05T06:35:13.951 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_disable_pagecache": "false", 2024-02-05T06:35:13.951 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_max_write": "0", 2024-02-05T06:35:13.951 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_multithreaded": "true", 2024-02-05T06:35:13.951 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_require_active_mds": "true", 2024-02-05T06:35:13.952 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_set_user_groups": "true", 2024-02-05T06:35:13.952 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_splice_move": "true", 2024-02-05T06:35:13.952 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_splice_read": "true", 2024-02-05T06:35:13.952 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_splice_write": "true", 2024-02-05T06:35:13.952 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_syncfs_on_mksnap": "true", 2024-02-05T06:35:13.952 INFO:teuthology.orchestra.run.smithi028.stdout: "fuse_use_invalidate_cb": "true", 2024-02-05T06:35:13.952 INFO:teuthology.orchestra.run.smithi028.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-02-05T06:35:13.952 INFO:teuthology.orchestra.run.smithi028.stdout: "gss_target_name": "ceph", 2024-02-05T06:35:13.952 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_file": "", 2024-02-05T06:35:13.952 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_inject_failure": "0", 2024-02-05T06:35:13.952 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_interval": "5", 2024-02-05T06:35:13.952 INFO:teuthology.orchestra.run.smithi028.stdout: "host": "smithi028", 2024-02-05T06:35:13.952 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-02-05T06:35:13.953 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-02-05T06:35:13.953 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_max_size": "1073741824", 2024-02-05T06:35:13.953 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-02-05T06:35:13.953 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-02-05T06:35:13.953 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-02-05T06:35:13.953 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-02-05T06:35:13.953 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-02-05T06:35:13.953 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-02-05T06:35:13.953 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-02-05T06:35:13.953 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-02-05T06:35:13.953 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-02-05T06:35:13.953 INFO:teuthology.orchestra.run.smithi028.stdout: "immutable_object_cache_watermark": "0.900000", 2024-02-05T06:35:13.953 INFO:teuthology.orchestra.run.smithi028.stdout: "inject_early_sigterm": "false", 2024-02-05T06:35:13.954 INFO:teuthology.orchestra.run.smithi028.stdout: "jaeger_agent_port": "6799", 2024-02-05T06:35:13.954 INFO:teuthology.orchestra.run.smithi028.stdout: "jaeger_tracing_enable": "false", 2024-02-05T06:35:13.954 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_aio": "true", 2024-02-05T06:35:13.954 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_align_min_size": "65536", 2024-02-05T06:35:13.954 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_block_align": "true", 2024-02-05T06:35:13.954 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_block_size": "4096", 2024-02-05T06:35:13.954 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_dio": "true", 2024-02-05T06:35:13.954 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_discard": "false", 2024-02-05T06:35:13.954 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_force_aio": "false", 2024-02-05T06:35:13.954 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_ignore_corruption": "false", 2024-02-05T06:35:13.954 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_max_write_bytes": "10485760", 2024-02-05T06:35:13.954 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_max_write_entries": "100", 2024-02-05T06:35:13.955 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_replay_from": "0", 2024-02-05T06:35:13.955 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_throttle_high_multiple": "0.000000", 2024-02-05T06:35:13.955 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-02-05T06:35:13.955 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-02-05T06:35:13.955 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_throttle_max_multiple": "0.000000", 2024-02-05T06:35:13.955 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_write_header_frequency": "0", 2024-02-05T06:35:13.955 INFO:teuthology.orchestra.run.smithi028.stdout: "journal_zero_on_create": "false", 2024-02-05T06:35:13.955 INFO:teuthology.orchestra.run.smithi028.stdout: "journaler_prefetch_periods": "10", 2024-02-05T06:35:13.955 INFO:teuthology.orchestra.run.smithi028.stdout: "journaler_prezero_periods": "5", 2024-02-05T06:35:13.955 INFO:teuthology.orchestra.run.smithi028.stdout: "journaler_write_head_interval": "15", 2024-02-05T06:35:13.955 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "", 2024-02-05T06:35:13.955 INFO:teuthology.orchestra.run.smithi028.stdout: "keyfile": "", 2024-02-05T06:35:13.955 INFO:teuthology.orchestra.run.smithi028.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-02-05T06:35:13.955 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_backend": "rocksdb", 2024-02-05T06:35:13.956 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_default_stripe_size": "65536", 2024-02-05T06:35:13.956 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_fsck_on_mount": "false", 2024-02-05T06:35:13.956 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_fsck_on_mount_deep": "true", 2024-02-05T06:35:13.956 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_max_bytes": "67108864", 2024-02-05T06:35:13.956 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_max_ops": "512", 2024-02-05T06:35:13.956 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_nid_prealloc": "1024", 2024-02-05T06:35:13.956 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_onode_map_size": "1024", 2024-02-05T06:35:13.956 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-02-05T06:35:13.956 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_sync_submit_transaction": "false", 2024-02-05T06:35:13.956 INFO:teuthology.orchestra.run.smithi028.stdout: "kstore_sync_transaction": "false", 2024-02-05T06:35:13.956 INFO:teuthology.orchestra.run.smithi028.stdout: "librados_thread_count": "2", 2024-02-05T06:35:13.956 INFO:teuthology.orchestra.run.smithi028.stdout: "lockdep": "false", 2024-02-05T06:35:13.956 INFO:teuthology.orchestra.run.smithi028.stdout: "lockdep_force_backtrace": "false", 2024-02-05T06:35:13.956 INFO:teuthology.orchestra.run.smithi028.stdout: "log_coarse_timestamps": "true", 2024-02-05T06:35:13.957 INFO:teuthology.orchestra.run.smithi028.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-02-05T06:35:13.957 INFO:teuthology.orchestra.run.smithi028.stdout: "log_flush_on_exit": "false", 2024-02-05T06:35:13.957 INFO:teuthology.orchestra.run.smithi028.stdout: "log_graylog_host": "127.0.0.1", 2024-02-05T06:35:13.957 INFO:teuthology.orchestra.run.smithi028.stdout: "log_graylog_port": "12201", 2024-02-05T06:35:13.957 INFO:teuthology.orchestra.run.smithi028.stdout: "log_max_new": "1000", 2024-02-05T06:35:13.957 INFO:teuthology.orchestra.run.smithi028.stdout: "log_max_recent": "10000", 2024-02-05T06:35:13.957 INFO:teuthology.orchestra.run.smithi028.stdout: "log_stderr_prefix": "", 2024-02-05T06:35:13.957 INFO:teuthology.orchestra.run.smithi028.stdout: "log_stop_at_utilization": "0.970000", 2024-02-05T06:35:13.957 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": "true", 2024-02-05T06:35:13.957 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_graylog": "false", 2024-02-05T06:35:13.957 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_journald": "false", 2024-02-05T06:35:13.957 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_stderr": "false", 2024-02-05T06:35:13.957 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_syslog": "false", 2024-02-05T06:35:13.957 INFO:teuthology.orchestra.run.smithi028.stdout: "mandatory_topic_permissions": "false", 2024-02-05T06:35:13.958 INFO:teuthology.orchestra.run.smithi028.stdout: "max_rotating_auth_attempts": "10", 2024-02-05T06:35:13.958 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-02-05T06:35:13.958 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_action_on_write_error": "1", 2024-02-05T06:35:13.958 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_alternate_name_max": "8192", 2024-02-05T06:35:13.958 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_asio_thread_count": "2", 2024-02-05T06:35:13.958 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_export_pin": "true", 2024-02-05T06:35:13.958 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_fragment_dirs": "true", 2024-02-05T06:35:13.958 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-02-05T06:35:13.958 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_fragment_interval": "5", 2024-02-05T06:35:13.958 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_fragment_size_max": "100000", 2024-02-05T06:35:13.958 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_idle_threshold": "0.000000", 2024-02-05T06:35:13.958 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_interval": "10", 2024-02-05T06:35:13.959 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_max": "-1", 2024-02-05T06:35:13.959 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_max_until": "-1", 2024-02-05T06:35:13.959 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_merge_size": "50", 2024-02-05T06:35:13.959 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_midchunk": "0.300000", 2024-02-05T06:35:13.959 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_min_rebalance": "0.100000", 2024-02-05T06:35:13.959 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_min_start": "0.200000", 2024-02-05T06:35:13.959 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_minchunk": "0.001000", 2024-02-05T06:35:13.959 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_mode": "0", 2024-02-05T06:35:13.959 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_need_max": "1.200000", 2024-02-05T06:35:13.959 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_need_min": "0.800000", 2024-02-05T06:35:13.959 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_overload_epochs": "2", 2024-02-05T06:35:13.959 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-02-05T06:35:13.959 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_sample_interval": "3.000000", 2024-02-05T06:35:13.960 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_split_bits": "3", 2024-02-05T06:35:13.960 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_split_rd": "25000.000000", 2024-02-05T06:35:13.960 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_split_size": "10000", 2024-02-05T06:35:13.960 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_split_wr": "10000.000000", 2024-02-05T06:35:13.960 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_target_decay": "10.000000", 2024-02-05T06:35:13.960 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-02-05T06:35:13.960 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_beacon_grace": "15.000000", 2024-02-05T06:35:13.960 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_beacon_interval": "4.000000", 2024-02-05T06:35:13.960 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_beacon_mon_down_grace": "60", 2024-02-05T06:35:13.960 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_memory_limit": "4294967296", 2024-02-05T06:35:13.960 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_mid": "0.700000", 2024-02-05T06:35:13.960 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_release_free_interval": "10", 2024-02-05T06:35:13.960 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_reservation": "0.050000", 2024-02-05T06:35:13.961 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-02-05T06:35:13.961 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_trim_interval": "1", 2024-02-05T06:35:13.961 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache_trim_threshold": "262144", 2024-02-05T06:35:13.961 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-02-05T06:35:13.961 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-02-05T06:35:13.961 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_client_delegate_inos_pct": "50", 2024-02-05T06:35:13.961 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_client_prealloc_inos": "1000", 2024-02-05T06:35:13.961 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-02-05T06:35:13.961 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_connect_bootstrapping": "false", 2024-02-05T06:35:13.961 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_damage_table_max_entries": "10000", 2024-02-05T06:35:13.961 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-02-05T06:35:13.961 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_debug_auth_pins": "false", 2024-02-05T06:35:13.961 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_debug_frag": "false", 2024-02-05T06:35:13.961 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_debug_scatterstat": "false", 2024-02-05T06:35:13.962 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_debug_subtrees": "false", 2024-02-05T06:35:13.962 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_decay_halflife": "5.000000", 2024-02-05T06:35:13.962 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_default_dir_hash": "2", 2024-02-05T06:35:13.962 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_defer_session_stale": "true", 2024-02-05T06:35:13.962 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_deny_all_reconnect": "false", 2024-02-05T06:35:13.962 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dir_keys_per_op": "16384", 2024-02-05T06:35:13.962 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dir_max_commit_size": "10", 2024-02-05T06:35:13.962 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dir_max_entries": "0", 2024-02-05T06:35:13.962 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dir_prefetch": "true", 2024-02-05T06:35:13.962 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dirstat_min_interval": "1.000000", 2024-02-05T06:35:13.962 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dump_cache_after_rejoin": "false", 2024-02-05T06:35:13.962 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dump_cache_on_map": "false", 2024-02-05T06:35:13.963 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dump_cache_threshold_file": "0", 2024-02-05T06:35:13.963 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-02-05T06:35:13.963 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_early_reply": "true", 2024-02-05T06:35:13.963 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_enable_op_tracker": "true", 2024-02-05T06:35:13.963 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_enforce_unique_name": "true", 2024-02-05T06:35:13.963 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_export_ephemeral_distributed": "true", 2024-02-05T06:35:13.963 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-02-05T06:35:13.963 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_export_ephemeral_random": "true", 2024-02-05T06:35:13.963 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-02-05T06:35:13.963 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-02-05T06:35:13.963 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_forward_all_requests_to_auth": "false", 2024-02-05T06:35:13.963 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-02-05T06:35:13.963 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-02-05T06:35:13.964 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-02-05T06:35:13.964 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-02-05T06:35:13.964 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_health_cache_threshold": "1.500000", 2024-02-05T06:35:13.964 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_health_summarize_threshold": "10", 2024-02-05T06:35:13.964 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_heartbeat_grace": "15.000000", 2024-02-05T06:35:13.964 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_heartbeat_reset_grace": "1000", 2024-02-05T06:35:13.964 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_inject_health_dummy": "false", 2024-02-05T06:35:13.964 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-02-05T06:35:13.964 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_inject_migrator_session_race": "false", 2024-02-05T06:35:13.964 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-02-05T06:35:13.964 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-02-05T06:35:13.964 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-02-05T06:35:13.964 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_join_fs": "", 2024-02-05T06:35:13.965 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_journal_format": "1", 2024-02-05T06:35:13.965 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_create_at": "0", 2024-02-05T06:35:13.965 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_export_at": "0", 2024-02-05T06:35:13.965 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_import_at": "0", 2024-02-05T06:35:13.965 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_journal_at": "0", 2024-02-05T06:35:13.965 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_journal_expire_at": "0", 2024-02-05T06:35:13.965 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_journal_replay_at": "0", 2024-02-05T06:35:13.965 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_link_at": "0", 2024-02-05T06:35:13.965 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_mdstable_at": "0", 2024-02-05T06:35:13.965 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_openc_at": "0", 2024-02-05T06:35:13.965 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_rename_at": "0", 2024-02-05T06:35:13.965 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_shutdown_at": "0", 2024-02-05T06:35:13.965 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-02-05T06:35:13.965 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_event_large_threshold": "524288", 2024-02-05T06:35:13.966 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_events_per_segment": "1024", 2024-02-05T06:35:13.966 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_major_segment_event_ratio": "12", 2024-02-05T06:35:13.966 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_max_events": "-1", 2024-02-05T06:35:13.966 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_max_segments": "128", 2024-02-05T06:35:13.966 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_pause": "false", 2024-02-05T06:35:13.966 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_segment_size": "0", 2024-02-05T06:35:13.966 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_skip_corrupt_events": "false", 2024-02-05T06:35:13.966 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_skip_unbounded_events": "false", 2024-02-05T06:35:13.966 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-02-05T06:35:13.966 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_trim_threshold": "128", 2024-02-05T06:35:13.966 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-02-05T06:35:13.966 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log_warn_factor": "2.000000", 2024-02-05T06:35:13.966 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_caps_per_client": "1048576", 2024-02-05T06:35:13.967 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_completed_flushes": "100000", 2024-02-05T06:35:13.967 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_completed_requests": "100000", 2024-02-05T06:35:13.967 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_export_size": "20971520", 2024-02-05T06:35:13.967 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_file_recover": "32", 2024-02-05T06:35:13.967 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_purge_files": "64", 2024-02-05T06:35:13.967 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_purge_ops": "8192", 2024-02-05T06:35:13.967 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-02-05T06:35:13.967 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-02-05T06:35:13.967 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_max_snaps_per_dir": "100", 2024-02-05T06:35:13.967 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_metrics_update_interval": "2", 2024-02-05T06:35:13.967 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_min_caps_per_client": "100", 2024-02-05T06:35:13.967 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_min_caps_working_set": "10000", 2024-02-05T06:35:13.968 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-02-05T06:35:13.968 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_numa_node": "-1", 2024-02-05T06:35:13.968 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-02-05T06:35:13.968 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_op_complaint_time": "30.000000", 2024-02-05T06:35:13.968 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_op_history_duration": "600", 2024-02-05T06:35:13.968 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_op_history_size": "20", 2024-02-05T06:35:13.968 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_op_history_slow_op_size": "20", 2024-02-05T06:35:13.968 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_op_history_slow_op_threshold": "10", 2024-02-05T06:35:13.968 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_op_log_threshold": "5", 2024-02-05T06:35:13.968 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_ping_grace": "15", 2024-02-05T06:35:13.968 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_ping_interval": "5", 2024-02-05T06:35:13.968 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-02-05T06:35:13.968 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-02-05T06:35:13.969 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_recall_max_caps": "30000", 2024-02-05T06:35:13.969 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-02-05T06:35:13.969 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_recall_max_decay_threshold": "131072", 2024-02-05T06:35:13.969 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-02-05T06:35:13.969 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_recall_warning_threshold": "262144", 2024-02-05T06:35:13.969 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_reconnect_timeout": "45.000000", 2024-02-05T06:35:13.969 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_replay_interval": "1.000000", 2024-02-05T06:35:13.969 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-02-05T06:35:13.969 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-02-05T06:35:13.969 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_root_ino_gid": "0", 2024-02-05T06:35:13.969 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_root_ino_uid": "0", 2024-02-05T06:35:13.970 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-02-05T06:35:13.970 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_blocklist_on_evict": "true", 2024-02-05T06:35:13.970 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_blocklist_on_timeout": "true", 2024-02-05T06:35:13.970 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-02-05T06:35:13.970 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-02-05T06:35:13.970 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-02-05T06:35:13.970 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-02-05T06:35:13.970 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-02-05T06:35:13.970 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_session_metadata_threshold": "16777216", 2024-02-05T06:35:13.970 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-02-05T06:35:13.970 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_shutdown_check": "0", 2024-02-05T06:35:13.970 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_skip_ino": "0", 2024-02-05T06:35:13.970 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_sleep_rank_change": "0.000000", 2024-02-05T06:35:13.971 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_snap_max_uid": "4294967294", 2024-02-05T06:35:13.971 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_snap_min_uid": "0", 2024-02-05T06:35:13.971 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_snap_rstat": "false", 2024-02-05T06:35:13.971 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_standby_replay_damaged": "false", 2024-02-05T06:35:13.971 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_symlink_recovery": "true", 2024-02-05T06:35:13.971 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_task_status_update_interval": "2.000000", 2024-02-05T06:35:13.971 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_thrash_exports": "0", 2024-02-05T06:35:13.971 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_thrash_fragments": "0", 2024-02-05T06:35:13.971 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_tick_interval": "5.000000", 2024-02-05T06:35:13.971 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_valgrind_exit": "false", 2024-02-05T06:35:13.971 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_verify_backtrace": "1", 2024-02-05T06:35:13.971 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_verify_scatter": "false", 2024-02-05T06:35:13.971 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_wipe_ino_prealloc": "false", 2024-02-05T06:35:13.972 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_wipe_sessions": "false", 2024-02-05T06:35:13.972 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool_debug": "false", 2024-02-05T06:35:13.972 INFO:teuthology.orchestra.run.smithi028.stdout: "memstore_debug_omit_block_device_write": "false", 2024-02-05T06:35:13.972 INFO:teuthology.orchestra.run.smithi028.stdout: "memstore_device_bytes": "1073741824", 2024-02-05T06:35:13.972 INFO:teuthology.orchestra.run.smithi028.stdout: "memstore_page_set": "false", 2024-02-05T06:35:13.972 INFO:teuthology.orchestra.run.smithi028.stdout: "memstore_page_size": "65536", 2024-02-05T06:35:13.972 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_client_bytes": "134217728", 2024-02-05T06:35:13.972 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_client_messages": "512", 2024-02-05T06:35:13.972 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-02-05T06:35:13.972 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_connect_retry_interval": "1.000000", 2024-02-05T06:35:13.972 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-02-05T06:35:13.972 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-02-05T06:35:13.972 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_disabled_modules": "", 2024-02-05T06:35:13.973 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-02-05T06:35:13.973 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_max_pg_num_change": "128", 2024-02-05T06:35:13.973 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_mds_bytes": "134217728", 2024-02-05T06:35:13.973 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_mds_messages": "128", 2024-02-05T06:35:13.973 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-02-05T06:35:13.973 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_mon_bytes": "134217728", 2024-02-05T06:35:13.973 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_mon_messages": "128", 2024-02-05T06:35:13.973 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_osd_bytes": "536870912", 2024-02-05T06:35:13.973 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_osd_messages": "8192", 2024-02-05T06:35:13.973 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_pool": "true", 2024-02-05T06:35:13.973 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_service_beacon_grace": "60.000000", 2024-02-05T06:35:13.973 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_standby_modules": "true", 2024-02-05T06:35:13.973 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_stats_period": "5", 2024-02-05T06:35:13.974 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_stats_threshold": "5", 2024-02-05T06:35:13.974 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_tick_period": "2", 2024-02-05T06:35:13.974 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-02-05T06:35:13.974 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_accept_timeout_factor": "2.000000", 2024-02-05T06:35:13.974 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_allow_pool_delete": "true", 2024-02-05T06:35:13.974 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_allow_pool_size_one": "false", 2024-02-05T06:35:13.974 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_auth_validate_all_caps": "true", 2024-02-05T06:35:13.974 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-02-05T06:35:13.974 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-02-05T06:35:13.974 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_bytes": "104857600", 2024-02-05T06:35:13.974 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_directed_command_retry": "2", 2024-02-05T06:35:13.974 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_hunt_interval": "3.000000", 2024-02-05T06:35:13.975 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-02-05T06:35:13.975 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-02-05T06:35:13.975 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-02-05T06:35:13.975 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_hunt_parallel": "3", 2024-02-05T06:35:13.975 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_log_interval": "1.000000", 2024-02-05T06:35:13.975 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-02-05T06:35:13.975 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_ping_interval": "10.000000", 2024-02-05T06:35:13.975 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_ping_timeout": "30.000000", 2024-02-05T06:35:13.975 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_client_target_rank": "-1", 2024-02-05T06:35:13.975 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_clock_drift_allowed": "1.000000", 2024-02-05T06:35:13.975 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-02-05T06:35:13.975 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-02-05T06:35:13.975 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_file_level": "debug", 2024-02-05T06:35:13.976 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_file": "true", 2024-02-05T06:35:13.976 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_graylog": "false", 2024-02-05T06:35:13.976 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-02-05T06:35:13.976 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-02-05T06:35:13.976 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_journald": "true", 2024-02-05T06:35:13.976 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_stderr": "false", 2024-02-05T06:35:13.976 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-02-05T06:35:13.976 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-02-05T06:35:13.976 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-02-05T06:35:13.976 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_compact_on_bootstrap": "false", 2024-02-05T06:35:13.976 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_compact_on_start": "false", 2024-02-05T06:35:13.977 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_compact_on_trim": "true", 2024-02-05T06:35:13.977 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_con_tracker_persist_interval": "10", 2024-02-05T06:35:13.977 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_con_tracker_score_halflife": "43200", 2024-02-05T06:35:13.977 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_config_key_max_entry_size": "65536", 2024-02-05T06:35:13.977 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_cpu_threads": "4", 2024-02-05T06:35:13.977 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_crush_min_required_version": "hammer", 2024-02-05T06:35:13.977 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_daemon_bytes": "419430400", 2024-02-05T06:35:13.977 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-02-05T06:35:13.977 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_data_avail_crit": "5", 2024-02-05T06:35:13.977 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_data_avail_warn": "5", 2024-02-05T06:35:13.977 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_data_size_warn": "16106127360", 2024-02-05T06:35:13.977 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_block_osdmap_trim": "false", 2024-02-05T06:35:13.977 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-02-05T06:35:13.978 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_dump_json": "false", 2024-02-05T06:35:13.978 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-02-05T06:35:13.978 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_dump_transactions": "false", 2024-02-05T06:35:13.978 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_extra_checks": "false", 2024-02-05T06:35:13.978 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-02-05T06:35:13.978 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-02-05T06:35:13.978 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_no_require_reef": "false", 2024-02-05T06:35:13.978 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_no_require_squid": "false", 2024-02-05T06:35:13.978 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-02-05T06:35:13.978 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_delta_reset_interval": "10.000000", 2024-02-05T06:35:13.978 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_dns_srv_name": "ceph-mon", 2024-02-05T06:35:13.978 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_down_mkfs_grace": "60", 2024-02-05T06:35:13.978 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_election_default_strategy": "1", 2024-02-05T06:35:13.978 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_election_timeout": "5.000000", 2024-02-05T06:35:13.979 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-02-05T06:35:13.979 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_elector_ping_divisor": "2", 2024-02-05T06:35:13.979 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_elector_ping_timeout": "2.000000", 2024-02-05T06:35:13.979 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_enable_op_tracker": "true", 2024-02-05T06:35:13.979 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_fake_pool_delete": "false", 2024-02-05T06:35:13.979 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_force_quorum_join": "false", 2024-02-05T06:35:13.979 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_globalid_prealloc": "10000", 2024-02-05T06:35:13.979 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_health_detail_to_clog": "true", 2024-02-05T06:35:13.979 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_health_log_update_period": "5", 2024-02-05T06:35:13.979 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_health_max_detail": "50", 2024-02-05T06:35:13.979 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_health_to_clog": "true", 2024-02-05T06:35:13.979 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_health_to_clog_interval": "600", 2024-02-05T06:35:13.979 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-02-05T06:35:13.980 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_host": "172.21.15.28:6789/0", 2024-02-05T06:35:13.980 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_host_override": "", 2024-02-05T06:35:13.980 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_initial_members": "", 2024-02-05T06:35:13.980 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-02-05T06:35:13.980 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-02-05T06:35:13.980 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-02-05T06:35:13.980 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-02-05T06:35:13.980 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_keyvaluedb": "rocksdb", 2024-02-05T06:35:13.980 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_lease": "5.000000", 2024-02-05T06:35:13.980 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-02-05T06:35:13.980 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-02-05T06:35:13.981 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_log_full_interval": "50", 2024-02-05T06:35:13.981 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_log_max": "10000", 2024-02-05T06:35:13.981 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_log_max_summary": "50", 2024-02-05T06:35:13.981 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_log_entries_per_event": "4096", 2024-02-05T06:35:13.981 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_log_epochs": "500", 2024-02-05T06:35:13.981 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_mdsmap_epochs": "500", 2024-02-05T06:35:13.981 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_mgrmap_epochs": "500", 2024-02-05T06:35:13.981 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_osd": "10000", 2024-02-05T06:35:13.981 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_pg_per_osd": "10000", 2024-02-05T06:35:13.981 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_pool_pg_num": "65536", 2024-02-05T06:35:13.981 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-02-05T06:35:13.981 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-02-05T06:35:13.981 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mds_force_trim_to": "0", 2024-02-05T06:35:13.981 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mds_skip_sanity": "false", 2024-02-05T06:35:13.982 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_memory_autotune": "true", 2024-02-05T06:35:13.982 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_memory_target": "2147483648", 2024-02-05T06:35:13.982 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mgr_beacon_grace": "30", 2024-02-05T06:35:13.982 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-02-05T06:35:13.982 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mgr_digest_period": "5", 2024-02-05T06:35:13.982 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mgr_inactive_grace": "60", 2024-02-05T06:35:13.982 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mgr_mkfs_grace": "240", 2024-02-05T06:35:13.982 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-02-05T06:35:13.982 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_min_osdmap_epochs": "50", 2024-02-05T06:35:13.982 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_op_complaint_time": "30", 2024-02-05T06:35:13.982 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_op_history_duration": "600", 2024-02-05T06:35:13.982 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_op_history_size": "20", 2024-02-05T06:35:13.982 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_op_history_slow_op_size": "20", 2024-02-05T06:35:13.983 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_op_history_slow_op_threshold": "10", 2024-02-05T06:35:13.983 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_op_log_threshold": "5", 2024-02-05T06:35:13.983 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-02-05T06:35:13.983 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-02-05T06:35:13.983 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-02-05T06:35:13.983 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_auto_mark_in": "false", 2024-02-05T06:35:13.983 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_auto_mark_new_in": "true", 2024-02-05T06:35:13.983 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-02-05T06:35:13.983 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-02-05T06:35:13.983 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_cache_size": "500", 2024-02-05T06:35:13.983 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_cache_size_min": "134217728", 2024-02-05T06:35:13.983 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_crush_smoke_test": "true", 2024-02-05T06:35:13.983 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_destroyed_out_interval": "600", 2024-02-05T06:35:13.983 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_down_out_interval": "600", 2024-02-05T06:35:13.983 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-02-05T06:35:13.983 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-02-05T06:35:13.984 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_force_trim_to": "0", 2024-02-05T06:35:13.984 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_full_ratio": "0.950000", 2024-02-05T06:35:13.984 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-02-05T06:35:13.984 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_laggy_halflife": "3600", 2024-02-05T06:35:13.984 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_laggy_max_interval": "300", 2024-02-05T06:35:13.984 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_laggy_weight": "0.300000", 2024-02-05T06:35:13.984 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-02-05T06:35:13.984 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_max_creating_pgs": "1024", 2024-02-05T06:35:13.984 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_max_initial_pgs": "1024", 2024-02-05T06:35:13.984 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_min_down_reporters": "2", 2024-02-05T06:35:13.984 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-02-05T06:35:13.984 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-02-05T06:35:13.984 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-02-05T06:35:13.985 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_prime_pg_temp": "true", 2024-02-05T06:35:13.985 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-02-05T06:35:13.985 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-02-05T06:35:13.985 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_report_timeout": "900", 2024-02-05T06:35:13.985 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-02-05T06:35:13.985 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-02-05T06:35:13.985 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_warn_num_repaired": "10", 2024-02-05T06:35:13.985 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osd_warn_op_age": "32.000000", 2024-02-05T06:35:13.985 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-02-05T06:35:13.985 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osdmap_full_prune_interval": "2", 2024-02-05T06:35:13.985 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osdmap_full_prune_min": "15", 2024-02-05T06:35:13.985 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-02-05T06:35:13.985 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-02-05T06:35:13.985 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pg_stuck_threshold": "60", 2024-02-05T06:35:13.985 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-02-05T06:35:13.985 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pg_warn_min_objects": "10000", 2024-02-05T06:35:13.985 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pg_warn_min_per_osd": "0", 2024-02-05T06:35:13.986 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-02-05T06:35:13.986 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pool_quota_crit_threshold": "0", 2024-02-05T06:35:13.986 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_pool_quota_warn_threshold": "0", 2024-02-05T06:35:13.986 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_probe_timeout": "2.000000", 2024-02-05T06:35:13.986 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_reweight_max_change": "0.050000", 2024-02-05T06:35:13.986 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_reweight_max_osds": "4", 2024-02-05T06:35:13.986 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-02-05T06:35:13.986 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-02-05T06:35:13.986 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-02-05T06:35:13.986 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-02-05T06:35:13.986 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-02-05T06:35:13.986 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_scrub_interval": "86400", 2024-02-05T06:35:13.986 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_scrub_max_keys": "100", 2024-02-05T06:35:13.987 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_scrub_timeout": "300", 2024-02-05T06:35:13.987 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_session_timeout": "300", 2024-02-05T06:35:13.987 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_smart_report_timeout": "5", 2024-02-05T06:35:13.987 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_stat_smooth_intervals": "6", 2024-02-05T06:35:13.987 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-02-05T06:35:13.987 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_stretch_pool_min_size": "2", 2024-02-05T06:35:13.987 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_stretch_pool_size": "4", 2024-02-05T06:35:13.987 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-02-05T06:35:13.987 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_subscribe_interval": "86400.000000", 2024-02-05T06:35:13.987 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_sync_debug": "false", 2024-02-05T06:35:13.987 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_sync_max_payload_keys": "2000", 2024-02-05T06:35:13.987 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_sync_max_payload_size": "1048576", 2024-02-05T06:35:13.987 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_sync_provider_kill_at": "0", 2024-02-05T06:35:13.988 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_sync_requester_kill_at": "0", 2024-02-05T06:35:13.988 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_sync_timeout": "60.000000", 2024-02-05T06:35:13.988 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_target_pg_per_osd": "100", 2024-02-05T06:35:13.988 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_tick_interval": "5", 2024-02-05T06:35:13.988 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_timecheck_interval": "300.000000", 2024-02-05T06:35:13.988 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-02-05T06:35:13.988 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_use_min_delay_socket": "false", 2024-02-05T06:35:13.988 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_older_version_delay": "604800", 2024-02-05T06:35:13.988 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-02-05T06:35:13.988 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-02-05T06:35:13.989 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-02-05T06:35:13.989 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_filestore_osds": "true", 2024-02-05T06:35:13.989 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-02-05T06:35:13.989 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-02-05T06:35:13.989 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-02-05T06:35:13.989 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_misplaced": "false", 2024-02-05T06:35:13.989 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-02-05T06:35:13.989 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_older_version": "true", 2024-02-05T06:35:13.989 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-02-05T06:35:13.989 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_pool_no_app": "true", 2024-02-05T06:35:13.989 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-02-05T06:35:13.989 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-02-05T06:35:13.990 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-02-05T06:35:13.990 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-02-05T06:35:13.990 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_on_too_few_osds": "false", 2024-02-05T06:35:13.990 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-02-05T06:35:13.990 INFO:teuthology.orchestra.run.smithi028.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-02-05T06:35:13.990 INFO:teuthology.orchestra.run.smithi028.stdout: "monmap": "", 2024-02-05T06:35:13.990 INFO:teuthology.orchestra.run.smithi028.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-02-05T06:35:13.990 INFO:teuthology.orchestra.run.smithi028.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-02-05T06:35:13.990 INFO:teuthology.orchestra.run.smithi028.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-02-05T06:35:13.990 INFO:teuthology.orchestra.run.smithi028.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-02-05T06:35:13.990 INFO:teuthology.orchestra.run.smithi028.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-02-05T06:35:13.990 INFO:teuthology.orchestra.run.smithi028.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-02-05T06:35:13.990 INFO:teuthology.orchestra.run.smithi028.stdout: "motr_tracing_enabled": "false", 2024-02-05T06:35:13.991 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_op_threads": "3", 2024-02-05T06:35:13.991 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_buffer_size": "131072", 2024-02-05T06:35:13.991 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_cm": "false", 2024-02-05T06:35:13.991 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_device_name": "", 2024-02-05T06:35:13.991 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_dscp": "96", 2024-02-05T06:35:13.991 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-02-05T06:35:13.991 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_gid_idx": "0", 2024-02-05T06:35:13.991 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_local_gid": "", 2024-02-05T06:35:13.991 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_polling_us": "1000", 2024-02-05T06:35:13.991 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_port_num": "1", 2024-02-05T06:35:13.991 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-02-05T06:35:13.991 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-02-05T06:35:13.991 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_roce_ver": "1", 2024-02-05T06:35:13.992 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_send_buffers": "1024", 2024-02-05T06:35:13.992 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_sl": "3", 2024-02-05T06:35:13.992 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_support_srq": "true", 2024-02-05T06:35:13.992 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_rdma_type": "ib", 2024-02-05T06:35:13.992 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_async_reap_threshold": "5", 2024-02-05T06:35:13.992 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_before_connect": "false", 2024-02-05T06:35:13.992 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_ipv4": "true", 2024-02-05T06:35:13.992 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_ipv6": "false", 2024-02-05T06:35:13.992 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_msgr1": "true", 2024-02-05T06:35:13.992 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_msgr2": "false", 2024-02-05T06:35:13.992 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_port_max": "7568", 2024-02-05T06:35:13.992 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_port_min": "6800", 2024-02-05T06:35:13.992 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_prefer_ipv4": "false", 2024-02-05T06:35:13.992 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_retry_count": "3", 2024-02-05T06:35:13.993 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_bind_retry_delay": "5", 2024-02-05T06:35:13.993 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_blackhole_client": "false", 2024-02-05T06:35:13.993 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_blackhole_mds": "false", 2024-02-05T06:35:13.993 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_blackhole_mgr": "false", 2024-02-05T06:35:13.993 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_blackhole_mon": "false", 2024-02-05T06:35:13.993 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_blackhole_osd": "false", 2024-02-05T06:35:13.993 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_client_mode": "crc secure", 2024-02-05T06:35:13.993 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_cluster_mode": "crc secure", 2024-02-05T06:35:13.993 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_cluster_type": "", 2024-02-05T06:35:13.993 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_compress_secure": "false", 2024-02-05T06:35:13.993 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_connection_idle_timeout": "900", 2024-02-05T06:35:13.993 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_connection_ready_timeout": "10", 2024-02-05T06:35:13.993 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_crc_data": "true", 2024-02-05T06:35:13.993 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_crc_header": "true", 2024-02-05T06:35:13.993 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_die_on_bad_msg": "false", 2024-02-05T06:35:13.993 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_die_on_bug": "true", 2024-02-05T06:35:13.994 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_die_on_old_message": "true", 2024-02-05T06:35:13.994 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_die_on_skipped_message": "false", 2024-02-05T06:35:13.994 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_die_on_unhandled_msg": "false", 2024-02-05T06:35:13.994 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-02-05T06:35:13.994 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_coremask": "0xF", 2024-02-05T06:35:13.994 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-02-05T06:35:13.994 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_devs_allowlist": "", 2024-02-05T06:35:13.994 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_enable_tso": "true", 2024-02-05T06:35:13.994 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-02-05T06:35:13.994 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-02-05T06:35:13.994 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_hugepages": "", 2024-02-05T06:35:13.994 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_hw_flow_control": "true", 2024-02-05T06:35:13.994 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-02-05T06:35:13.994 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_lro": "true", 2024-02-05T06:35:13.994 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_memory_channel": "4", 2024-02-05T06:35:13.994 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-02-05T06:35:13.995 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_pmd": "", 2024-02-05T06:35:13.995 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_port_id": "0", 2024-02-05T06:35:13.995 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-02-05T06:35:13.995 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dump_corrupt_message_level": "1", 2024-02-05T06:35:13.995 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_dump_on_send": "false", 2024-02-05T06:35:13.995 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_initial_backoff": "0.200000", 2024-02-05T06:35:13.995 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_inject_delay_max": "1.000000", 2024-02-05T06:35:13.995 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_inject_delay_probability": "0.000000", 2024-02-05T06:35:13.995 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_inject_delay_type": "", 2024-02-05T06:35:13.995 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_inject_internal_delays": "0.000000", 2024-02-05T06:35:13.995 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_inject_network_congestion": "0", 2024-02-05T06:35:13.995 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_inject_socket_failures": "0", 2024-02-05T06:35:13.995 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_learn_addr_from_peer": "true", 2024-02-05T06:35:13.995 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_max_accept_failures": "4", 2024-02-05T06:35:13.995 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_max_backoff": "15.000000", 2024-02-05T06:35:13.995 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_mon_client_mode": "secure crc", 2024-02-05T06:35:13.995 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_mon_cluster_mode": "secure crc", 2024-02-05T06:35:13.996 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_mon_service_mode": "secure crc", 2024-02-05T06:35:13.996 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_osd_compress_min_size": "1024", 2024-02-05T06:35:13.996 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_osd_compress_mode": "none", 2024-02-05T06:35:13.996 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_osd_compression_algorithm": "snappy", 2024-02-05T06:35:13.996 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-02-05T06:35:13.996 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_pq_min_cost": "65536", 2024-02-05T06:35:13.996 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_public_type": "", 2024-02-05T06:35:13.996 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_service_mode": "crc secure", 2024-02-05T06:35:13.996 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_tcp_listen_backlog": "512", 2024-02-05T06:35:13.996 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_tcp_nodelay": "true", 2024-02-05T06:35:13.996 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-02-05T06:35:13.996 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_tcp_rcvbuf": "0", 2024-02-05T06:35:13.996 INFO:teuthology.orchestra.run.smithi028.stdout: "ms_type": "async+posix", 2024-02-05T06:35:13.996 INFO:teuthology.orchestra.run.smithi028.stdout: "no_config_file": "false", 2024-02-05T06:35:13.996 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_completion_locks_per_session": "32", 2024-02-05T06:35:13.997 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_debug_inject_relock_delay": "false", 2024-02-05T06:35:13.997 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_inflight_op_bytes": "104857600", 2024-02-05T06:35:13.997 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_inflight_ops": "1024", 2024-02-05T06:35:13.997 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_inject_no_watch_ping": "false", 2024-02-05T06:35:13.997 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-02-05T06:35:13.997 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_tick_interval": "5.000000", 2024-02-05T06:35:13.997 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter_timeout": "10.000000", 2024-02-05T06:35:13.997 INFO:teuthology.orchestra.run.smithi028.stdout: "objectstore_blackhole": "false", 2024-02-05T06:35:13.997 INFO:teuthology.orchestra.run.smithi028.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-02-05T06:35:13.997 INFO:teuthology.orchestra.run.smithi028.stdout: "openssl_engine_opts": "", 2024-02-05T06:35:13.997 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_delay_time": "5.000000", 2024-02-05T06:35:13.997 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_hist_halflife": "1000", 2024-02-05T06:35:13.997 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_max_low_ops": "2", 2024-02-05T06:35:13.997 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_max_ops": "4", 2024-02-05T06:35:13.997 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-02-05T06:35:13.998 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_quantize_effort": "0.100000", 2024-02-05T06:35:13.998 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_agent_slop": "0.020000", 2024-02-05T06:35:13.998 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-02-05T06:35:13.998 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_allow_recovery_below_min_size": "true", 2024-02-05T06:35:13.998 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_asio_thread_count": "2", 2024-02-05T06:35:13.998 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_async_recovery_min_cost": "100", 2024-02-05T06:35:13.998 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_auto_mark_unfound_lost": "false", 2024-02-05T06:35:13.998 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_backfill_retry_interval": "30.000000", 2024-02-05T06:35:13.998 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_backfill_scan_max": "512", 2024-02-05T06:35:13.998 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_backfill_scan_min": "64", 2024-02-05T06:35:13.998 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_backoff_on_degraded": "false", 2024-02-05T06:35:13.998 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_backoff_on_peering": "false", 2024-02-05T06:35:13.998 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_backoff_on_unfound": "true", 2024-02-05T06:35:13.998 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_beacon_report_interval": "300", 2024-02-05T06:35:13.998 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bench_duration": "30", 2024-02-05T06:35:13.998 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-02-05T06:35:13.998 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bench_max_block_size": "67108864", 2024-02-05T06:35:13.999 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bench_small_size_max_iops": "100", 2024-02-05T06:35:13.999 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_blkin_trace_all": "false", 2024-02-05T06:35:13.999 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_blocked_scrub_grace_period": "120", 2024-02-05T06:35:13.999 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-02-05T06:35:13.999 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-02-05T06:35:13.999 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-02-05T06:35:13.999 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_check_for_log_corruption": "false", 2024-02-05T06:35:13.999 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-02-05T06:35:13.999 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-02-05T06:35:13.999 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-02-05T06:35:13.999 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-02-05T06:35:13.999 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_class_update_on_start": "true", 2024-02-05T06:35:13.999 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_client_message_cap": "256", 2024-02-05T06:35:13.999 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_client_message_size_cap": "524288000", 2024-02-05T06:35:13.999 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_client_op_priority": "63", 2024-02-05T06:35:13.999 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_client_watch_timeout": "30", 2024-02-05T06:35:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_command_max_records": "256", 2024-02-05T06:35:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_compact_on_start": "false", 2024-02-05T06:35:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-02-05T06:35:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_crush_chooseleaf_type": "1", 2024-02-05T06:35:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_crush_initial_weight": "-1.000000", 2024-02-05T06:35:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_crush_update_on_start": "true", 2024-02-05T06:35:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_crush_update_weight_set": "true", 2024-02-05T06:35:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-02-05T06:35:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-02-05T06:35:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-02-05T06:35:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_drop_ping_duration": "0", 2024-02-05T06:35:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-02-05T06:35:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_feed_pullee": "-1", 2024-02-05T06:35:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-02-05T06:35:14.000 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-02-05T06:35:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-02-05T06:35:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_misdirected_ops": "false", 2024-02-05T06:35:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_no_acting_change": "false", 2024-02-05T06:35:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_no_purge_strays": "false", 2024-02-05T06:35:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_op_order": "false", 2024-02-05T06:35:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_pg_log_writeout": "false", 2024-02-05T06:35:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_pretend_recovery_active": "false", 2024-02-05T06:35:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-02-05T06:35:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-02-05T06:35:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_shutdown": "false", 2024-02-05T06:35:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-02-05T06:35:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-02-05T06:35:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_trim_objects": "false", 2024-02-05T06:35:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_verify_cached_snaps": "false", 2024-02-05T06:35:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_verify_missing_on_start": "false", 2024-02-05T06:35:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_verify_snaps": "false", 2024-02-05T06:35:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-02-05T06:35:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-02-05T06:35:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_keys": "1024", 2024-02-05T06:35:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-02-05T06:35:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-02-05T06:35:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-02-05T06:35:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_stride": "524288", 2024-02-05T06:35:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-02-05T06:35:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_default_data_pool_replay_window": "45", 2024-02-05T06:35:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_default_notify_timeout": "30", 2024-02-05T06:35:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_delete_sleep": "0.000000", 2024-02-05T06:35:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-02-05T06:35:14.002 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-02-05T06:35:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-02-05T06:35:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_discard_disconnected_ops": "true", 2024-02-05T06:35:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_enable_op_tracker": "true", 2024-02-05T06:35:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-02-05T06:35:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_extblkdev_plugins": "vdo", 2024-02-05T06:35:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-02-05T06:35:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-02-05T06:35:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_fast_info": "true", 2024-02-05T06:35:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_fast_shutdown": "true", 2024-02-05T06:35:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-02-05T06:35:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_fast_shutdown_timeout": "15", 2024-02-05T06:35:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-02-05T06:35:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-02-05T06:35:14.003 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-02-05T06:35:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_function_tracing": "false", 2024-02-05T06:35:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_grace": "20", 2024-02-05T06:35:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_interval": "6", 2024-02-05T06:35:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-02-05T06:35:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_min_peers": "10", 2024-02-05T06:35:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_min_size": "2000", 2024-02-05T06:35:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_stale": "600", 2024-02-05T06:35:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-02-05T06:35:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_hit_set_max_size": "100000", 2024-02-05T06:35:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_hit_set_min_size": "1000", 2024-02-05T06:35:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-02-05T06:35:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-02-05T06:35:14.004 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-02-05T06:35:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-02-05T06:35:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-02-05T06:35:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_journal_flush_on_shutdown": "true", 2024-02-05T06:35:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_journal_size": "5120", 2024-02-05T06:35:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_kill_backfill_at": "0", 2024-02-05T06:35:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_loop_before_reset_tphandle": "64", 2024-02-05T06:35:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_map_cache_size": "50", 2024-02-05T06:35:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_map_dedup": "true", 2024-02-05T06:35:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_map_message_max": "40", 2024-02-05T06:35:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_map_message_max_bytes": "10485760", 2024-02-05T06:35:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_map_share_max_epochs": "40", 2024-02-05T06:35:14.005 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_attr_name_len": "100", 2024-02-05T06:35:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_attr_size": "0", 2024-02-05T06:35:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_backfills": "1", 2024-02-05T06:35:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_markdown_count": "5", 2024-02-05T06:35:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_markdown_period": "600", 2024-02-05T06:35:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_object_name_len": "2048", 2024-02-05T06:35:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_object_namespace_len": "256", 2024-02-05T06:35:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_object_size": "134217728", 2024-02-05T06:35:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-02-05T06:35:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_omap_entries_per_request": "1024", 2024-02-05T06:35:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_pg_blocked_by": "16", 2024-02-05T06:35:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_pg_log_entries": "10000", 2024-02-05T06:35:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-02-05T06:35:14.006 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_pgls": "1024", 2024-02-05T06:35:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_push_cost": "8388608", 2024-02-05T06:35:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_push_objects": "10", 2024-02-05T06:35:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_scrubs": "3", 2024-02-05T06:35:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-02-05T06:35:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_trimming_pgs": "2", 2024-02-05T06:35:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_write_op_reply_len": "64", 2024-02-05T06:35:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_max_write_size": "90", 2024-02-05T06:35:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-02-05T06:35:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-02-05T06:35:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-02-05T06:35:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-02-05T06:35:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-02-05T06:35:14.007 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-02-05T06:35:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-02-05T06:35:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_override_recovery_settings": "false", 2024-02-05T06:35:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_profile": "balanced", 2024-02-05T06:35:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-02-05T06:35:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-02-05T06:35:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-02-05T06:35:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-02-05T06:35:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-02-05T06:35:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-02-05T06:35:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-02-05T06:35:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-02-05T06:35:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-02-05T06:35:14.008 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-02-05T06:35:14.009 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mclock_skip_benchmark": "false", 2024-02-05T06:35:14.009 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_base": "805306368", 2024-02-05T06:35:14.009 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_cache_min": "134217728", 2024-02-05T06:35:14.009 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-02-05T06:35:14.009 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-02-05T06:35:14.009 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_target": "4294967296", 2024-02-05T06:35:14.009 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_target_autotune": "false", 2024-02-05T06:35:14.009 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-02-05T06:35:14.009 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_min_pg_log_entries": "250", 2024-02-05T06:35:14.009 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_min_recovery_priority": "0", 2024-02-05T06:35:14.009 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mon_heartbeat_interval": "30", 2024-02-05T06:35:14.009 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-02-05T06:35:14.009 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mon_report_interval": "5", 2024-02-05T06:35:14.009 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mon_report_max_in_flight": "2", 2024-02-05T06:35:14.010 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-02-05T06:35:14.010 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_num_cache_shards": "32", 2024-02-05T06:35:14.010 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_num_op_tracker_shard": "32", 2024-02-05T06:35:14.010 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_numa_auto_affinity": "true", 2024-02-05T06:35:14.010 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_numa_node": "-1", 2024-02-05T06:35:14.010 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_numa_prefer_iface": "true", 2024-02-05T06:35:14.010 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-02-05T06:35:14.010 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_objecter_finishers": "1", 2024-02-05T06:35:14.010 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_objectstore": "bluestore", 2024-02-05T06:35:14.010 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_objectstore_fuse": "false", 2024-02-05T06:35:14.010 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_objectstore_tracing": "false", 2024-02-05T06:35:14.010 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_complaint_time": "30.000000", 2024-02-05T06:35:14.010 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_history_duration": "600", 2024-02-05T06:35:14.010 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_history_size": "20", 2024-02-05T06:35:14.011 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_history_slow_op_size": "20", 2024-02-05T06:35:14.011 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-02-05T06:35:14.011 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_log_threshold": "5", 2024-02-05T06:35:14.011 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_num_shards": "0", 2024-02-05T06:35:14.011 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_num_shards_hdd": "5", 2024-02-05T06:35:14.011 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_num_shards_ssd": "8", 2024-02-05T06:35:14.011 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_num_threads_per_shard": "0", 2024-02-05T06:35:14.011 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-02-05T06:35:14.011 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-02-05T06:35:14.011 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-02-05T06:35:14.011 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_pq_min_cost": "65536", 2024-02-05T06:35:14.011 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_queue": "mclock_scheduler", 2024-02-05T06:35:14.011 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_queue_cut_off": "high", 2024-02-05T06:35:14.011 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_thread_suicide_timeout": "150", 2024-02-05T06:35:14.011 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_op_thread_timeout": "15", 2024-02-05T06:35:14.012 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_open_classes_on_start": "true", 2024-02-05T06:35:14.012 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_os_flags": "0", 2024-02-05T06:35:14.012 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_peering_op_priority": "255", 2024-02-05T06:35:14.012 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_delete_cost": "1048576", 2024-02-05T06:35:14.012 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_delete_priority": "5", 2024-02-05T06:35:14.012 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-02-05T06:35:14.012 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-02-05T06:35:14.012 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_log_dups_tracked": "3000", 2024-02-05T06:35:14.012 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_log_trim_max": "10000", 2024-02-05T06:35:14.012 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_log_trim_min": "100", 2024-02-05T06:35:14.012 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-02-05T06:35:14.012 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_object_context_cache_count": "64", 2024-02-05T06:35:14.013 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-02-05T06:35:14.013 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pg_stat_report_interval_max_seconds": "500", 2024-02-05T06:35:14.013 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-02-05T06:35:14.013 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-02-05T06:35:14.013 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-02-05T06:35:14.013 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-02-05T06:35:14.013 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-02-05T06:35:14.013 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-02-05T06:35:14.013 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_crimson": "false", 2024-02-05T06:35:14.013 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_crush_rule": "-1", 2024-02-05T06:35:14.013 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_ec_fast_read": "false", 2024-02-05T06:35:14.013 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-02-05T06:35:14.013 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_flag_bulk": "false", 2024-02-05T06:35:14.013 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-02-05T06:35:14.013 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_flag_nodelete": "false", 2024-02-05T06:35:14.013 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-02-05T06:35:14.014 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-02-05T06:35:14.014 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_flags": "0", 2024-02-05T06:35:14.014 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-02-05T06:35:14.014 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_min_size": "0", 2024-02-05T06:35:14.014 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-02-05T06:35:14.014 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_pg_num": "32", 2024-02-05T06:35:14.014 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_pgp_num": "0", 2024-02-05T06:35:14.014 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-02-05T06:35:14.014 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_read_ratio": "70", 2024-02-05T06:35:14.014 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_size": "3", 2024-02-05T06:35:14.014 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_default_type": "replicated", 2024-02-05T06:35:14.014 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-02-05T06:35:14.014 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pool_use_gmt_hitset": "true", 2024-02-05T06:35:14.014 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_push_per_object_cost": "1000", 2024-02-05T06:35:14.015 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_read_ec_check_for_errors": "false", 2024-02-05T06:35:14.015 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recover_clone_overlap": "true", 2024-02-05T06:35:14.016 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recover_clone_overlap_limit": "10", 2024-02-05T06:35:14.016 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_cost": "20971520", 2024-02-05T06:35:14.016 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_delay_start": "0.000000", 2024-02-05T06:35:14.016 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_max_active": "0", 2024-02-05T06:35:14.016 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_max_active_hdd": "3", 2024-02-05T06:35:14.016 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_max_active_ssd": "10", 2024-02-05T06:35:14.016 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_max_chunk": "8388608", 2024-02-05T06:35:14.016 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-02-05T06:35:14.016 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_max_single_start": "1", 2024-02-05T06:35:14.016 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_op_priority": "3", 2024-02-05T06:35:14.016 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_op_warn_multiple": "16", 2024-02-05T06:35:14.016 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_priority": "5", 2024-02-05T06:35:14.017 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_retry_interval": "30.000000", 2024-02-05T06:35:14.017 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_sleep": "0.000000", 2024-02-05T06:35:14.017 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-02-05T06:35:14.017 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-02-05T06:35:14.017 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-02-05T06:35:14.017 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_repair_during_recovery": "false", 2024-02-05T06:35:14.017 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_requested_scrub_priority": "120", 2024-02-05T06:35:14.017 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-02-05T06:35:14.017 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_rollback_to_cluster_snap": "", 2024-02-05T06:35:14.017 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_auto_repair": "false", 2024-02-05T06:35:14.017 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-02-05T06:35:14.017 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-02-05T06:35:14.017 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_begin_hour": "0", 2024-02-05T06:35:14.017 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_begin_week_day": "0", 2024-02-05T06:35:14.018 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_chunk_max": "25", 2024-02-05T06:35:14.018 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_chunk_min": "5", 2024-02-05T06:35:14.018 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_cost": "52428800", 2024-02-05T06:35:14.018 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-02-05T06:35:14.018 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_during_recovery": "false", 2024-02-05T06:35:14.018 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_end_hour": "0", 2024-02-05T06:35:14.018 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_end_week_day": "0", 2024-02-05T06:35:14.018 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_event_cost": "4096", 2024-02-05T06:35:14.018 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-02-05T06:35:14.018 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-02-05T06:35:14.018 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_invalid_stats": "true", 2024-02-05T06:35:14.018 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_load_threshold": "0.500000", 2024-02-05T06:35:14.019 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_max_interval": "604800.000000", 2024-02-05T06:35:14.019 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_max_preemptions": "5", 2024-02-05T06:35:14.019 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_min_interval": "86400.000000", 2024-02-05T06:35:14.019 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_priority": "5", 2024-02-05T06:35:14.019 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_reservation_timeout": "300000", 2024-02-05T06:35:14.019 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_sleep": "0.000000", 2024-02-05T06:35:14.019 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_scrub_slow_reservation_response": "30000", 2024-02-05T06:35:14.019 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-02-05T06:35:14.019 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-02-05T06:35:14.019 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_shutdown_pgref_assert": "false", 2024-02-05T06:35:14.019 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_skip_data_digest": "false", 2024-02-05T06:35:14.019 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_smart_report_timeout": "5", 2024-02-05T06:35:14.019 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_snap_trim_cost": "1048576", 2024-02-05T06:35:14.020 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_snap_trim_priority": "5", 2024-02-05T06:35:14.020 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_snap_trim_sleep": "0.000000", 2024-02-05T06:35:14.020 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-02-05T06:35:14.020 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-02-05T06:35:14.020 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-02-05T06:35:14.020 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-02-05T06:35:14.020 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_stats_update_period_scrubbing": "15", 2024-02-05T06:35:14.020 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-02-05T06:35:14.020 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_target_transaction_size": "30", 2024-02-05T06:35:14.020 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-02-05T06:35:14.020 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-02-05T06:35:14.020 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-02-05T06:35:14.020 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-02-05T06:35:14.021 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-02-05T06:35:14.021 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-02-05T06:35:14.021 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-02-05T06:35:14.021 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_default_cache_mode": "writeback", 2024-02-05T06:35:14.021 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-02-05T06:35:14.021 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-02-05T06:35:14.021 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_tracing": "false", 2024-02-05T06:35:14.021 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_use_stale_snap": "false", 2024-02-05T06:35:14.021 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-02-05T06:35:14.021 INFO:teuthology.orchestra.run.smithi028.stdout: "osdc_blkin_trace_all": "false", 2024-02-05T06:35:14.021 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_kill_at": "0", 2024-02-05T06:35:14.021 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_max_join_drift": "10", 2024-02-05T06:35:14.021 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_min": "500", 2024-02-05T06:35:14.022 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_min_wait": "0.050000", 2024-02-05T06:35:14.022 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_propose_interval": "1.000000", 2024-02-05T06:35:14.022 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_service_trim_max": "500", 2024-02-05T06:35:14.022 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_service_trim_max_multiplier": "20", 2024-02-05T06:35:14.022 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_service_trim_min": "10", 2024-02-05T06:35:14.022 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_stash_full_interval": "25", 2024-02-05T06:35:14.022 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_trim_max": "500", 2024-02-05T06:35:14.022 INFO:teuthology.orchestra.run.smithi028.stdout: "paxos_trim_min": "250", 2024-02-05T06:35:14.022 INFO:teuthology.orchestra.run.smithi028.stdout: "perf": "true", 2024-02-05T06:35:14.022 INFO:teuthology.orchestra.run.smithi028.stdout: "pid_file": "", 2024-02-05T06:35:14.022 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-02-05T06:35:14.022 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-02-05T06:35:14.022 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "-", 2024-02-05T06:35:14.023 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrv": "", 2024-02-05T06:35:14.023 INFO:teuthology.orchestra.run.smithi028.stdout: "public_bind_addr": "-", 2024-02-05T06:35:14.023 INFO:teuthology.orchestra.run.smithi028.stdout: "public_network": "", 2024-02-05T06:35:14.023 INFO:teuthology.orchestra.run.smithi028.stdout: "public_network_interface": "", 2024-02-05T06:35:14.023 INFO:teuthology.orchestra.run.smithi028.stdout: "qat_compressor_enabled": "false", 2024-02-05T06:35:14.023 INFO:teuthology.orchestra.run.smithi028.stdout: "qat_compressor_session_max_number": "256", 2024-02-05T06:35:14.023 INFO:teuthology.orchestra.run.smithi028.stdout: "rados_mon_op_timeout": "0", 2024-02-05T06:35:14.023 INFO:teuthology.orchestra.run.smithi028.stdout: "rados_osd_op_timeout": "0", 2024-02-05T06:35:14.023 INFO:teuthology.orchestra.run.smithi028.stdout: "rados_tracing": "false", 2024-02-05T06:35:14.023 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_atime_update_interval": "60", 2024-02-05T06:35:14.024 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-02-05T06:35:14.024 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_balance_parent_reads": "false", 2024-02-05T06:35:14.024 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_balance_snap_reads": "false", 2024-02-05T06:35:14.024 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_blkin_trace_all": "false", 2024-02-05T06:35:14.024 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_blocklist_expire_seconds": "0", 2024-02-05T06:35:14.024 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_blocklist_on_break_lock": "true", 2024-02-05T06:35:14.024 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache": "true", 2024-02-05T06:35:14.024 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_block_writes_upfront": "false", 2024-02-05T06:35:14.024 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_max_dirty": "25165824", 2024-02-05T06:35:14.024 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-02-05T06:35:14.024 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_max_dirty_object": "0", 2024-02-05T06:35:14.024 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_policy": "writearound", 2024-02-05T06:35:14.025 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_size": "33554432", 2024-02-05T06:35:14.025 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_target_dirty": "16777216", 2024-02-05T06:35:14.025 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-02-05T06:35:14.025 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_clone_copy_on_read": "false", 2024-02-05T06:35:14.025 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_compression_hint": "none", 2024-02-05T06:35:14.025 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_concurrent_management_ops": "10", 2024-02-05T06:35:14.025 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-02-05T06:35:14.025 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_clone_format": "auto", 2024-02-05T06:35:14.025 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_data_pool": "", 2024-02-05T06:35:14.025 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_features": "61", 2024-02-05T06:35:14.025 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_format": "2", 2024-02-05T06:35:14.025 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_map_options": "", 2024-02-05T06:35:14.025 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_order": "22", 2024-02-05T06:35:14.025 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_pool": "rbd", 2024-02-05T06:35:14.026 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-02-05T06:35:14.026 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_stripe_count": "0", 2024-02-05T06:35:14.026 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_default_stripe_unit": "0", 2024-02-05T06:35:14.026 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_disable_zero_copy_writes": "true", 2024-02-05T06:35:14.026 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_discard_granularity_bytes": "65536", 2024-02-05T06:35:14.026 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-02-05T06:35:14.026 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_enable_alloc_hint": "true", 2024-02-05T06:35:14.026 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-02-05T06:35:14.026 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_io_scheduler": "simple", 2024-02-05T06:35:14.026 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-02-05T06:35:14.026 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_commit_age": "5.000000", 2024-02-05T06:35:14.026 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-02-05T06:35:14.026 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-02-05T06:35:14.026 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-02-05T06:35:14.027 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-02-05T06:35:14.027 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_object_flush_interval": "0", 2024-02-05T06:35:14.027 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-02-05T06:35:14.027 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-02-05T06:35:14.027 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_order": "24", 2024-02-05T06:35:14.027 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_pool": "", 2024-02-05T06:35:14.027 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_journal_splay_width": "4", 2024-02-05T06:35:14.027 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_localize_parent_reads": "false", 2024-02-05T06:35:14.027 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_localize_snap_reads": "false", 2024-02-05T06:35:14.027 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-02-05T06:35:14.027 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-02-05T06:35:14.027 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-02-05T06:35:14.027 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-02-05T06:35:14.027 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-02-05T06:35:14.027 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-02-05T06:35:14.028 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_image_policy_type": "simple", 2024-02-05T06:35:14.028 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-02-05T06:35:14.028 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-02-05T06:35:14.028 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-02-05T06:35:14.028 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-02-05T06:35:14.028 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-02-05T06:35:14.028 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-02-05T06:35:14.028 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-02-05T06:35:14.028 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_autotune": "true", 2024-02-05T06:35:14.028 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_base": "805306368", 2024-02-05T06:35:14.028 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-02-05T06:35:14.028 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-02-05T06:35:14.028 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-02-05T06:35:14.028 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-02-05T06:35:14.028 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_memory_target": "4294967296", 2024-02-05T06:35:14.029 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-02-05T06:35:14.029 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-02-05T06:35:14.029 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-02-05T06:35:14.029 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirroring_delete_delay": "0", 2024-02-05T06:35:14.029 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-02-05T06:35:14.029 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirroring_replay_delay": "0", 2024-02-05T06:35:14.029 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-02-05T06:35:14.029 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-02-05T06:35:14.029 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_move_to_trash_on_remove": "false", 2024-02-05T06:35:14.029 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-02-05T06:35:14.029 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_mtime_update_interval": "60", 2024-02-05T06:35:14.029 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_non_blocking_aio": "true", 2024-02-05T06:35:14.029 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_op_thread_timeout": "60", 2024-02-05T06:35:14.030 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_op_threads": "1", 2024-02-05T06:35:14.030 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_parent_cache_enabled": "false", 2024-02-05T06:35:14.030 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_persistent_cache_mode": "disabled", 2024-02-05T06:35:14.030 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_persistent_cache_path": "/tmp", 2024-02-05T06:35:14.030 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_persistent_cache_size": "1073741824", 2024-02-05T06:35:14.030 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_plugins": "", 2024-02-05T06:35:14.030 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_bps_burst": "0", 2024-02-05T06:35:14.030 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-02-05T06:35:14.030 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_bps_limit": "0", 2024-02-05T06:35:14.030 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_exclude_ops": "0", 2024-02-05T06:35:14.030 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_iops_burst": "0", 2024-02-05T06:35:14.030 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-02-05T06:35:14.031 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_iops_limit": "0", 2024-02-05T06:35:14.031 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_read_bps_burst": "0", 2024-02-05T06:35:14.031 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-02-05T06:35:14.031 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_read_bps_limit": "0", 2024-02-05T06:35:14.031 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_read_iops_burst": "0", 2024-02-05T06:35:14.031 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-02-05T06:35:14.031 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_read_iops_limit": "0", 2024-02-05T06:35:14.031 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_schedule_tick_min": "50", 2024-02-05T06:35:14.031 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_write_bps_burst": "0", 2024-02-05T06:35:14.031 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-02-05T06:35:14.031 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_write_bps_limit": "0", 2024-02-05T06:35:14.031 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_write_iops_burst": "0", 2024-02-05T06:35:14.032 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-02-05T06:35:14.032 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_qos_write_iops_limit": "0", 2024-02-05T06:35:14.032 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_quiesce_notification_attempts": "10", 2024-02-05T06:35:14.032 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_read_from_replica_policy": "default", 2024-02-05T06:35:14.032 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-02-05T06:35:14.032 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_readahead_max_bytes": "524288", 2024-02-05T06:35:14.032 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_readahead_trigger_requests": "10", 2024-02-05T06:35:14.032 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_request_timed_out_seconds": "30", 2024-02-05T06:35:14.032 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_skip_partial_discard": "true", 2024-02-05T06:35:14.032 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-02-05T06:35:14.032 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_tracing": "false", 2024-02-05T06:35:14.032 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_validate_names": "true", 2024-02-05T06:35:14.032 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_validate_pool": "true", 2024-02-05T06:35:14.033 INFO:teuthology.orchestra.run.smithi028.stdout: "restapi_base_url": "", 2024-02-05T06:35:14.033 INFO:teuthology.orchestra.run.smithi028.stdout: "restapi_log_level": "", 2024-02-05T06:35:14.033 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_acl_grants_max_num": "100", 2024-02-05T06:35:14.033 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_admin_entry": "admin", 2024-02-05T06:35:14.033 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-02-05T06:35:14.033 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_backend_store": "rados", 2024-02-05T06:35:14.033 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_barbican_url": "", 2024-02-05T06:35:14.033 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_beast_enable_async": "true", 2024-02-05T06:35:14.033 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_counters_cache": "false", 2024-02-05T06:35:14.033 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-02-05T06:35:14.033 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-02-05T06:35:14.033 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-02-05T06:35:14.033 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_index_max_aio": "128", 2024-02-05T06:35:14.033 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-02-05T06:35:14.033 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-02-05T06:35:14.034 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_quota_ttl": "600", 2024-02-05T06:35:14.034 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-02-05T06:35:14.034 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_cache_enabled": "true", 2024-02-05T06:35:14.034 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_cache_expiry_interval": "900", 2024-02-05T06:35:14.034 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_cache_lru_size": "10000", 2024-02-05T06:35:14.034 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_config_store": "rados", 2024-02-05T06:35:14.034 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_content_length_compat": "false", 2024-02-05T06:35:14.034 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_copy_obj_progress": "true", 2024-02-05T06:35:14.034 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-02-05T06:35:14.034 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_cors_rules_max_num": "100", 2024-02-05T06:35:14.034 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_cross_domain_policy": "", 2024-02-05T06:35:14.034 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_default_encryption_key": "", 2024-02-05T06:35:14.034 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_addr": "", 2024-02-05T06:35:14.034 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_ca_path": "", 2024-02-05T06:35:14.034 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_client_cert": "", 2024-02-05T06:35:14.035 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_client_key": "", 2024-02-05T06:35:14.035 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-02-05T06:35:14.035 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_password": "", 2024-02-05T06:35:14.035 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-02-05T06:35:14.035 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_kmip_username": "", 2024-02-05T06:35:14.035 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_require_ssl": "true", 2024-02-05T06:35:14.035 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-02-05T06:35:14.035 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-02-05T06:35:14.035 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-02-05T06:35:14.035 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-02-05T06:35:14.035 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-02-05T06:35:14.035 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-02-05T06:35:14.035 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-02-05T06:35:14.035 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-02-05T06:35:14.036 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-02-05T06:35:14.036 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-02-05T06:35:14.036 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-02-05T06:35:14.036 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-02-05T06:35:14.036 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-02-05T06:35:14.036 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-02-05T06:35:14.036 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_suppress_logs": "true", 2024-02-05T06:35:14.036 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_addr": "", 2024-02-05T06:35:14.036 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_auth": "token", 2024-02-05T06:35:14.036 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_namespace": "", 2024-02-05T06:35:14.036 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_prefix": "", 2024-02-05T06:35:14.036 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-02-05T06:35:14.036 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-02-05T06:35:14.037 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-02-05T06:35:14.037 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-02-05T06:35:14.037 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_token_file": "", 2024-02-05T06:35:14.037 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-02-05T06:35:14.037 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_curl_buffersize": "524288", 2024-02-05T06:35:14.037 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_curl_low_speed_limit": "1024", 2024-02-05T06:35:14.037 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_curl_low_speed_time": "300", 2024-02-05T06:35:14.037 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_curl_tcp_keepalive": "0", 2024-02-05T06:35:14.037 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-02-05T06:35:14.037 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-02-05T06:35:14.037 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-02-05T06:35:14.037 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-02-05T06:35:14.037 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-02-05T06:35:14.037 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_l1_fadvise": "4", 2024-02-05T06:35:14.038 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-02-05T06:35:14.038 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-02-05T06:35:14.038 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-02-05T06:35:14.038 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d4n_host": "127.0.0.1", 2024-02-05T06:35:14.038 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_d4n_port": "6379", 2024-02-05T06:35:14.038 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-02-05T06:35:14.038 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_log_changes_size": "1000", 2024-02-05T06:35:14.038 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_log_num_shards": "128", 2024-02-05T06:35:14.038 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-02-05T06:35:14.038 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_log_window": "30", 2024-02-05T06:35:14.038 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_notify_interval_msec": "0", 2024-02-05T06:35:14.038 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_sync_poll_interval": "20", 2024-02-05T06:35:14.038 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_data_sync_spawn_window": "20", 2024-02-05T06:35:14.038 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-02-05T06:35:14.039 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-02-05T06:35:14.039 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-02-05T06:35:14.039 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_default_data_log_backing": "fifo", 2024-02-05T06:35:14.039 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-02-05T06:35:14.039 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_default_region_info_oid": "default.region", 2024-02-05T06:35:14.039 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-02-05T06:35:14.039 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-02-05T06:35:14.039 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_defer_to_bucket_acls": "", 2024-02-05T06:35:14.039 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-02-05T06:35:14.039 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-02-05T06:35:14.039 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-02-05T06:35:14.039 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-02-05T06:35:14.040 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-02-05T06:35:14.040 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-02-05T06:35:14.040 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-02-05T06:35:14.040 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-02-05T06:35:14.040 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_data_res": "500.000000", 2024-02-05T06:35:14.040 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-02-05T06:35:14.040 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-02-05T06:35:14.040 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-02-05T06:35:14.040 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-02-05T06:35:14.040 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dns_name": "", 2024-02-05T06:35:14.040 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dns_s3website_name": "", 2024-02-05T06:35:14.040 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_dynamic_resharding": "true", 2024-02-05T06:35:14.041 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-02-05T06:35:14.041 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_gc_threads": "true", 2024-02-05T06:35:14.041 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_lc_threads": "true", 2024-02-05T06:35:14.041 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_ops_log": "false", 2024-02-05T06:35:14.041 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_quota_threads": "true", 2024-02-05T06:35:14.041 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_static_website": "false", 2024-02-05T06:35:14.041 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enable_usage_log": "false", 2024-02-05T06:35:14.041 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_enforce_swift_acls": "true", 2024-02-05T06:35:14.041 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_exit_timeout_secs": "120", 2024-02-05T06:35:14.041 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_expose_bucket": "false", 2024-02-05T06:35:14.041 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_extended_http_attrs": "", 2024-02-05T06:35:14.041 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_filter": "none", 2024-02-05T06:35:14.041 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-02-05T06:35:14.041 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_frontends": "beast port=7480", 2024-02-05T06:35:14.041 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_max_concurrent_io": "10", 2024-02-05T06:35:14.042 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_max_deferred": "50", 2024-02-05T06:35:14.042 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-02-05T06:35:14.042 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_max_objs": "32", 2024-02-05T06:35:14.042 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_max_queue_size": "134213632", 2024-02-05T06:35:14.042 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_max_trim_chunk": "16", 2024-02-05T06:35:14.042 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_obj_min_wait": "7200", 2024-02-05T06:35:14.042 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_processor_max_time": "3600", 2024-02-05T06:35:14.042 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_gc_processor_period": "3600", 2024-02-05T06:35:14.042 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-02-05T06:35:14.042 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_get_obj_window_size": "16777216", 2024-02-05T06:35:14.042 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_healthcheck_disabling_path": "", 2024-02-05T06:35:14.042 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ignore_get_invalid_range": "false", 2024-02-05T06:35:14.042 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_init_timeout": "300", 2024-02-05T06:35:14.042 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-02-05T06:35:14.043 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-02-05T06:35:14.043 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_kafka_connection_idle": "30", 2024-02-05T06:35:14.043 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_kafka_sleep_timeout": "10", 2024-02-05T06:35:14.043 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-02-05T06:35:14.043 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-02-05T06:35:14.043 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-02-05T06:35:14.043 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_domain": "", 2024-02-05T06:35:14.043 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_password": "", 2024-02-05T06:35:14.043 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_password_path": "", 2024-02-05T06:35:14.043 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_project": "", 2024-02-05T06:35:14.043 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_tenant": "", 2024-02-05T06:35:14.043 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_token": "", 2024-02-05T06:35:14.043 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_token_path": "", 2024-02-05T06:35:14.043 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_admin_user": "", 2024-02-05T06:35:14.044 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_api_version": "2", 2024-02-05T06:35:14.044 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_barbican_domain": "", 2024-02-05T06:35:14.044 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_barbican_password": "", 2024-02-05T06:35:14.044 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_barbican_project": "", 2024-02-05T06:35:14.044 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_barbican_tenant": "", 2024-02-05T06:35:14.044 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_barbican_user": "", 2024-02-05T06:35:14.044 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-02-05T06:35:14.044 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_implicit_tenants": "false", 2024-02-05T06:35:14.044 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-02-05T06:35:14.044 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_service_token_enabled": "false", 2024-02-05T06:35:14.044 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_token_cache_size": "10000", 2024-02-05T06:35:14.044 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_url": "", 2024-02-05T06:35:14.044 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_keystone_verify_ssl": "true", 2024-02-05T06:35:14.044 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_debug_interval": "-1", 2024-02-05T06:35:14.045 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_lock_max_time": "90", 2024-02-05T06:35:14.045 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_max_objs": "32", 2024-02-05T06:35:14.045 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_max_rules": "1000", 2024-02-05T06:35:14.045 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_max_worker": "3", 2024-02-05T06:35:14.045 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_max_wp_worker": "3", 2024-02-05T06:35:14.045 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lc_thread_delay": "0", 2024-02-05T06:35:14.045 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-02-05T06:35:14.045 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ldap_dnattr": "uid", 2024-02-05T06:35:14.045 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-02-05T06:35:14.045 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ldap_searchfilter": "", 2024-02-05T06:35:14.045 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-02-05T06:35:14.045 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ldap_uri": "ldaps://", 2024-02-05T06:35:14.045 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-02-05T06:35:14.045 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-02-05T06:35:14.045 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-02-05T06:35:14.046 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_log_http_headers": "", 2024-02-05T06:35:14.046 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_log_nonexistent_bucket": "false", 2024-02-05T06:35:14.046 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-02-05T06:35:14.046 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_log_object_name_utc": "false", 2024-02-05T06:35:14.046 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-02-05T06:35:14.046 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-02-05T06:35:14.046 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_attr_name_len": "0", 2024-02-05T06:35:14.046 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_attr_size": "0", 2024-02-05T06:35:14.046 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_attrs_num_in_req": "0", 2024-02-05T06:35:14.046 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_chunk_size": "4194304", 2024-02-05T06:35:14.046 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_concurrent_requests": "1024", 2024-02-05T06:35:14.046 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-02-05T06:35:14.046 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_dynamic_shards": "1999", 2024-02-05T06:35:14.047 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_listing_results": "1000", 2024-02-05T06:35:14.047 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_notify_retries": "10", 2024-02-05T06:35:14.047 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_objs_per_shard": "100000", 2024-02-05T06:35:14.047 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_put_param_size": "1048576", 2024-02-05T06:35:14.047 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_put_size": "5368709120", 2024-02-05T06:35:14.047 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_max_slo_entries": "1000", 2024-02-05T06:35:14.047 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_md_log_max_shards": "64", 2024-02-05T06:35:14.047 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_md_notify_interval_msec": "200", 2024-02-05T06:35:14.047 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_meta_sync_poll_interval": "20", 2024-02-05T06:35:14.047 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_meta_sync_spawn_window": "20", 2024-02-05T06:35:14.047 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-02-05T06:35:14.047 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_mp_lock_max_time": "600", 2024-02-05T06:35:14.048 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-02-05T06:35:14.048 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_multipart_min_part_size": "5242880", 2024-02-05T06:35:14.048 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-02-05T06:35:14.048 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-02-05T06:35:14.048 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_fhcache_size": "2017", 2024-02-05T06:35:14.048 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-02-05T06:35:14.048 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-02-05T06:35:14.048 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_lru_lanes": "5", 2024-02-05T06:35:14.048 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_max_gc": "300", 2024-02-05T06:35:14.048 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-02-05T06:35:14.048 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_run_gc_threads": "false", 2024-02-05T06:35:14.048 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_run_lc_threads": "false", 2024-02-05T06:35:14.049 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_run_quota_threads": "false", 2024-02-05T06:35:14.049 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_run_sync_thread": "false", 2024-02-05T06:35:14.049 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-02-05T06:35:14.049 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-02-05T06:35:14.049 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_num_async_rados_threads": "32", 2024-02-05T06:35:14.049 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_num_control_oids": "8", 2024-02-05T06:35:14.049 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_numa_node": "-1", 2024-02-05T06:35:14.049 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_obj_stripe_size": "4194304", 2024-02-05T06:35:14.049 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-02-05T06:35:14.049 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_objexp_chunk_size": "100", 2024-02-05T06:35:14.049 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_objexp_gc_interval": "600", 2024-02-05T06:35:14.049 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_objexp_hints_num_shards": "127", 2024-02-05T06:35:14.050 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-02-05T06:35:14.050 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-02-05T06:35:14.050 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_op_thread_timeout": "600", 2024-02-05T06:35:14.050 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_op_tracing": "false", 2024-02-05T06:35:14.050 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_opa_token": "", 2024-02-05T06:35:14.050 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_opa_url": "", 2024-02-05T06:35:14.050 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_opa_verify_ssl": "true", 2024-02-05T06:35:14.050 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-02-05T06:35:14.050 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-02-05T06:35:14.050 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ops_log_rados": "false", 2024-02-05T06:35:14.050 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_ops_log_socket_path": "", 2024-02-05T06:35:14.050 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-02-05T06:35:14.050 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-02-05T06:35:14.050 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-02-05T06:35:14.051 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_period_push_interval": "2.000000", 2024-02-05T06:35:14.051 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_period_push_interval_max": "30.000000", 2024-02-05T06:35:14.051 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_period_root_pool": ".rgw.root", 2024-02-05T06:35:14.051 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-02-05T06:35:14.051 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-02-05T06:35:14.051 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_posix_cache_lanes": "3", 2024-02-05T06:35:14.051 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-02-05T06:35:14.051 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_posix_cache_max_buckets": "100", 2024-02-05T06:35:14.051 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_posix_cache_partitions": "3", 2024-02-05T06:35:14.051 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-02-05T06:35:14.051 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_print_continue": "true", 2024-02-05T06:35:14.052 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_print_prohibited_content_length": "false", 2024-02-05T06:35:14.052 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-02-05T06:35:14.052 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-02-05T06:35:14.052 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-02-05T06:35:14.052 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-02-05T06:35:14.052 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_rados_tracing": "false", 2024-02-05T06:35:14.052 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_realm": "", 2024-02-05T06:35:14.052 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_realm_id": "", 2024-02-05T06:35:14.052 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-02-05T06:35:14.052 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_region": "", 2024-02-05T06:35:14.052 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_region_root_pool": ".rgw.root", 2024-02-05T06:35:14.052 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_relaxed_region_enforcement": "false", 2024-02-05T06:35:14.052 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-02-05T06:35:14.053 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-02-05T06:35:14.053 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_request_uri": "", 2024-02-05T06:35:14.053 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_reshard_batch_size": "64", 2024-02-05T06:35:14.053 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-02-05T06:35:14.053 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_reshard_max_aio": "128", 2024-02-05T06:35:14.053 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_reshard_num_logs": "16", 2024-02-05T06:35:14.053 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_reshard_thread_interval": "600", 2024-02-05T06:35:14.053 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_resolve_cname": "false", 2024-02-05T06:35:14.053 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_rest_getusage_op_compat": "false", 2024-02-05T06:35:14.053 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_run_sync_thread": "true", 2024-02-05T06:35:14.053 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-02-05T06:35:14.053 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_auth_use_keystone": "false", 2024-02-05T06:35:14.054 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_auth_use_ldap": "false", 2024-02-05T06:35:14.054 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_auth_use_rados": "true", 2024-02-05T06:35:14.054 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_auth_use_sts": "false", 2024-02-05T06:35:14.054 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-02-05T06:35:14.054 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_s3_success_create_obj_status": "0", 2024-02-05T06:35:14.054 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-02-05T06:35:14.054 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_scheduler_type": "throttler", 2024-02-05T06:35:14.054 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_script_uri": "", 2024-02-05T06:35:14.054 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_service_provider_name": "", 2024-02-05T06:35:14.054 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-02-05T06:35:14.054 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_client_id": "", 2024-02-05T06:35:14.054 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_client_secret": "", 2024-02-05T06:35:14.054 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_entry": "sts", 2024-02-05T06:35:14.055 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_key": "sts", 2024-02-05T06:35:14.055 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_max_session_duration": "43200", 2024-02-05T06:35:14.055 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_min_session_duration": "900", 2024-02-05T06:35:14.055 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sts_token_introspection_url": "", 2024-02-05T06:35:14.055 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_account_in_url": "false", 2024-02-05T06:35:14.055 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_auth_entry": "auth", 2024-02-05T06:35:14.055 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_auth_url": "", 2024-02-05T06:35:14.055 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_custom_header": "", 2024-02-05T06:35:14.055 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_enforce_content_length": "false", 2024-02-05T06:35:14.055 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_need_stats": "true", 2024-02-05T06:35:14.055 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_tenant_name": "", 2024-02-05T06:35:14.055 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_token_expiration": "86400", 2024-02-05T06:35:14.055 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_url": "", 2024-02-05T06:35:14.055 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_url_prefix": "swift", 2024-02-05T06:35:14.056 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_swift_versioning_enabled": "false", 2024-02-05T06:35:14.056 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-02-05T06:35:14.056 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-02-05T06:35:14.056 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_lease_period": "120", 2024-02-05T06:35:14.056 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-02-05T06:35:14.056 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_log_trim_interval": "1200", 2024-02-05T06:35:14.056 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-02-05T06:35:14.056 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-02-05T06:35:14.056 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-02-05T06:35:14.056 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_obj_etag_verify": "false", 2024-02-05T06:35:14.056 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_trace_history_size": "4096", 2024-02-05T06:35:14.056 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-02-05T06:35:14.056 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-02-05T06:35:14.057 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_thread_pool_size": "512", 2024-02-05T06:35:14.057 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_topic_persistency_max_retries": "0", 2024-02-05T06:35:14.057 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-02-05T06:35:14.057 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-02-05T06:35:14.057 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_comment": "", 2024-02-05T06:35:14.057 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_createby": "", 2024-02-05T06:35:14.057 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_encoding": "", 2024-02-05T06:35:14.057 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_flag": "false", 2024-02-05T06:35:14.057 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_max_size": "5368709120", 2024-02-05T06:35:14.057 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_origin": "", 2024-02-05T06:35:14.057 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_sha_unit": "524288", 2024-02-05T06:35:14.057 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_torrent_tracker": "", 2024-02-05T06:35:14.057 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_trust_forwarded_https": "false", 2024-02-05T06:35:14.057 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-02-05T06:35:14.058 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_usage_log_tick_interval": "30", 2024-02-05T06:35:14.058 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_usage_max_shards": "32", 2024-02-05T06:35:14.058 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_usage_max_user_shards": "1", 2024-02-05T06:35:14.058 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_use_opa_authz": "false", 2024-02-05T06:35:14.058 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_counters_cache": "false", 2024-02-05T06:35:14.058 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_counters_cache_size": "10000", 2024-02-05T06:35:14.058 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-02-05T06:35:14.058 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_default_quota_max_size": "-1", 2024-02-05T06:35:14.058 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_max_buckets": "1000", 2024-02-05T06:35:14.058 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_policies_max_num": "100", 2024-02-05T06:35:14.058 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-02-05T06:35:14.058 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-02-05T06:35:14.058 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_quota_sync_interval": "86400", 2024-02-05T06:35:14.058 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-02-05T06:35:14.059 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_user_unique_email": "true", 2024-02-05T06:35:14.059 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_verify_ssl": "true", 2024-02-05T06:35:14.059 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_website_routing_rules_max_num": "50", 2024-02-05T06:35:14.059 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_zone": "", 2024-02-05T06:35:14.059 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_zone_id": "", 2024-02-05T06:35:14.059 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-02-05T06:35:14.059 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_zonegroup": "", 2024-02-05T06:35:14.059 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_zonegroup_id": "", 2024-02-05T06:35:14.059 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-02-05T06:35:14.059 INFO:teuthology.orchestra.run.smithi028.stdout: "rgwlc_auto_session_clear": "true", 2024-02-05T06:35:14.059 INFO:teuthology.orchestra.run.smithi028.stdout: "rgwlc_skip_bucket_step": "false", 2024-02-05T06:35:14.059 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_block_size": "4096", 2024-02-05T06:35:14.059 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-02-05T06:35:14.060 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-02-05T06:35:14.060 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-02-05T06:35:14.060 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-02-05T06:35:14.060 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cache_shard_bits": "4", 2024-02-05T06:35:14.060 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cache_size": "536870912", 2024-02-05T06:35:14.060 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cache_type": "binned_lru", 2024-02-05T06:35:14.060 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-02-05T06:35:14.060 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-02-05T06:35:14.060 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-02-05T06:35:14.060 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_collect_compaction_stats": "false", 2024-02-05T06:35:14.060 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_collect_extended_stats": "false", 2024-02-05T06:35:14.060 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_collect_memory_stats": "false", 2024-02-05T06:35:14.060 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-02-05T06:35:14.061 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_index_type": "binary_search", 2024-02-05T06:35:14.061 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_log_to_ceph_log": "true", 2024-02-05T06:35:14.061 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_metadata_block_size": "4096", 2024-02-05T06:35:14.061 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_partition_filters": "false", 2024-02-05T06:35:14.061 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_perf": "false", 2024-02-05T06:35:14.061 INFO:teuthology.orchestra.run.smithi028.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-02-05T06:35:14.061 INFO:teuthology.orchestra.run.smithi028.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-02-05T06:35:14.061 INFO:teuthology.orchestra.run.smithi028.stdout: "rotating_keys_renewal_timeout": "10", 2024-02-05T06:35:14.061 INFO:teuthology.orchestra.run.smithi028.stdout: "run_dir": "/var/run/ceph", 2024-02-05T06:35:14.061 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_block_create": "true", 2024-02-05T06:35:14.061 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_cache_lru_size": "67108864", 2024-02-05T06:35:14.061 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_cbjournal_size": "5368709120", 2024-02-05T06:35:14.061 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_data_delta_based_overwrite": "0", 2024-02-05T06:35:14.061 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_default_max_object_size": "16777216", 2024-02-05T06:35:14.061 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-02-05T06:35:14.061 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_device_size": "53687091200", 2024-02-05T06:35:14.062 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_journal_batch_capacity": "16", 2024-02-05T06:35:14.062 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-02-05T06:35:14.062 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-02-05T06:35:14.062 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_journal_iodepth_limit": "5", 2024-02-05T06:35:14.062 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_main_device_type": "SSD", 2024-02-05T06:35:14.062 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_max_concurrent_transactions": "8", 2024-02-05T06:35:14.062 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-02-05T06:35:14.062 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-02-05T06:35:14.062 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-02-05T06:35:14.062 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-02-05T06:35:14.062 INFO:teuthology.orchestra.run.smithi028.stdout: "seastore_segment_size": "67108864", 2024-02-05T06:35:14.062 INFO:teuthology.orchestra.run.smithi028.stdout: "set_keepcaps": "false", 2024-02-05T06:35:14.062 INFO:teuthology.orchestra.run.smithi028.stdout: "setgroup": "ceph", 2024-02-05T06:35:14.063 INFO:teuthology.orchestra.run.smithi028.stdout: "setuser": "ceph", 2024-02-05T06:35:14.063 INFO:teuthology.orchestra.run.smithi028.stdout: "setuser_match_path": "", 2024-02-05T06:35:14.063 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_misplaced_ratio": "0.050000", 2024-02-05T06:35:14.063 INFO:teuthology.orchestra.run.smithi028.stdout: "thp": "false", 2024-02-05T06:35:14.063 INFO:teuthology.orchestra.run.smithi028.stdout: "threadpool_default_timeout": "60", 2024-02-05T06:35:14.063 INFO:teuthology.orchestra.run.smithi028.stdout: "threadpool_empty_queue_max_wait": "2", 2024-02-05T06:35:14.063 INFO:teuthology.orchestra.run.smithi028.stdout: "throttler_perf_counter": "true" 2024-02-05T06:35:14.063 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:35:14.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:14 smithi028 ceph-mon[29442]: 5.18 scrub starts 2024-02-05T06:35:14.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:14 smithi028 ceph-mon[29442]: 5.18 scrub ok 2024-02-05T06:35:14.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:14 smithi028 ceph-mon[37984]: 5.18 scrub starts 2024-02-05T06:35:14.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:14 smithi028 ceph-mon[37984]: 5.18 scrub ok 2024-02-05T06:35:14.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:14 smithi167 ceph-mon[32774]: 5.18 scrub starts 2024-02-05T06:35:14.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:14 smithi167 ceph-mon[32774]: 5.18 scrub ok 2024-02-05T06:35:14.479 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:14 smithi167 bash[63932]: Getting image source signatures 2024-02-05T06:35:14.479 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:14 smithi167 bash[63932]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-02-05T06:35:14.479 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:14 smithi167 bash[63932]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-02-05T06:35:14.479 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:14 smithi167 bash[63932]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-02-05T06:35:15.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-02-05T06:35:15.343 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:35:15.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:15 smithi028 ceph-mon[37984]: 5.4 scrub starts 2024-02-05T06:35:15.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:15 smithi028 ceph-mon[37984]: 5.4 scrub ok 2024-02-05T06:35:15.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:15 smithi028 ceph-mon[37984]: 6.6 scrub starts 2024-02-05T06:35:15.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:15 smithi028 ceph-mon[37984]: 6.6 scrub ok 2024-02-05T06:35:15.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:15 smithi028 ceph-mon[37984]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2024-02-05T06:35:15.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:15 smithi028 ceph-mon[37984]: 6.1 scrub starts 2024-02-05T06:35:15.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:15 smithi028 ceph-mon[37984]: 6.1 scrub ok 2024-02-05T06:35:15.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:15 smithi028 ceph-mon[37984]: 5.5 scrub starts 2024-02-05T06:35:15.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:15 smithi028 ceph-mon[37984]: 5.5 scrub ok 2024-02-05T06:35:15.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:15 smithi028 ceph-mon[29442]: 5.4 scrub starts 2024-02-05T06:35:15.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:15 smithi028 ceph-mon[29442]: 5.4 scrub ok 2024-02-05T06:35:15.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:15 smithi028 ceph-mon[29442]: 6.6 scrub starts 2024-02-05T06:35:15.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:15 smithi028 ceph-mon[29442]: 6.6 scrub ok 2024-02-05T06:35:15.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:15 smithi028 ceph-mon[29442]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2024-02-05T06:35:15.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:15 smithi028 ceph-mon[29442]: 6.1 scrub starts 2024-02-05T06:35:15.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:15 smithi028 ceph-mon[29442]: 6.1 scrub ok 2024-02-05T06:35:15.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:15 smithi028 ceph-mon[29442]: 5.5 scrub starts 2024-02-05T06:35:15.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:15 smithi028 ceph-mon[29442]: 5.5 scrub ok 2024-02-05T06:35:15.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:15 smithi167 ceph-mon[32774]: 5.4 scrub starts 2024-02-05T06:35:15.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:15 smithi167 ceph-mon[32774]: 5.4 scrub ok 2024-02-05T06:35:15.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:15 smithi167 ceph-mon[32774]: 6.6 scrub starts 2024-02-05T06:35:15.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:15 smithi167 ceph-mon[32774]: 6.6 scrub ok 2024-02-05T06:35:15.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:15 smithi167 ceph-mon[32774]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2024-02-05T06:35:15.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:15 smithi167 ceph-mon[32774]: 6.1 scrub starts 2024-02-05T06:35:15.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:15 smithi167 ceph-mon[32774]: 6.1 scrub ok 2024-02-05T06:35:15.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:15 smithi167 ceph-mon[32774]: 5.5 scrub starts 2024-02-05T06:35:15.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:15 smithi167 ceph-mon[32774]: 5.5 scrub ok 2024-02-05T06:35:17.208 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a: {} 2024-02-05T06:35:17.219 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b: {} 2024-02-05T06:35:17.229 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c: {} 2024-02-05T06:35:17.230 INFO:teuthology.orchestra.run.smithi028.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-05T06:35:17.231 INFO:teuthology.orchestra.run.smithi028.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-05T06:35:17.231 INFO:teuthology.orchestra.run.smithi028.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-05T06:35:17.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:17 smithi028 ceph-mon[29442]: 4.8 scrub starts 2024-02-05T06:35:17.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:17 smithi028 ceph-mon[29442]: 4.8 scrub ok 2024-02-05T06:35:17.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:17 smithi028 ceph-mon[29442]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:35:17.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:17 smithi028 ceph-mon[29442]: 5.1d scrub starts 2024-02-05T06:35:17.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:17 smithi028 ceph-mon[29442]: 5.1d scrub ok 2024-02-05T06:35:17.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:17 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3338015529' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T06:35:17.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:17 smithi028 ceph-mon[37984]: 4.8 scrub starts 2024-02-05T06:35:17.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:17 smithi028 ceph-mon[37984]: 4.8 scrub ok 2024-02-05T06:35:17.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:17 smithi028 ceph-mon[37984]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:35:17.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:17 smithi028 ceph-mon[37984]: 5.1d scrub starts 2024-02-05T06:35:17.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:17 smithi028 ceph-mon[37984]: 5.1d scrub ok 2024-02-05T06:35:17.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:17 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3338015529' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T06:35:17.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:17 smithi167 ceph-mon[32774]: 4.8 scrub starts 2024-02-05T06:35:17.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:17 smithi167 ceph-mon[32774]: 4.8 scrub ok 2024-02-05T06:35:17.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:17 smithi167 ceph-mon[32774]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:35:17.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:17 smithi167 ceph-mon[32774]: 5.1d scrub starts 2024-02-05T06:35:17.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:17 smithi167 ceph-mon[32774]: 5.1d scrub ok 2024-02-05T06:35:17.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:17 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3338015529' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T06:35:17.906 INFO:teuthology.run_tasks:Running task full_sequential... 2024-02-05T06:35:17.915 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-05T06:35:17.916 INFO:tasks.radosbench:Beginning radosbench... 2024-02-05T06:35:17.916 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-02-05T06:35:17.917 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd pool create unique_pool_0 16 2024-02-05T06:35:17.917 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-02-05T06:35:17.917 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T06:35:17.917 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T06:35:17.918 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T06:35:17.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd out 4 2024-02-05T06:35:17.918 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-02-05T06:35:17.918 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:17.919 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-02-05T06:35:17.919 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:35:17.919 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-02-05T06:35:17.919 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-05T06:35:17.920 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-02-05T06:35:17.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T06:35:17.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:17.950 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:17.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:18.051 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:18.086 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:18.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:18.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:18.188 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:35:18.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:18.216 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:18.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:35:18.236 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:35:18.236 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:35:18.241 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:35:18.317 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:18.340 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:18.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:18.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:18.442 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:18.466 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:18.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:18.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:18.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:18 smithi028 ceph-mon[29442]: 3.16 scrub starts 2024-02-05T06:35:18.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:18 smithi028 ceph-mon[29442]: 3.16 scrub ok 2024-02-05T06:35:18.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:18 smithi028 ceph-mon[29442]: 3.8 scrub starts 2024-02-05T06:35:18.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:18 smithi028 ceph-mon[29442]: 3.8 scrub ok 2024-02-05T06:35:18.477 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:18 smithi028 ceph-mon[37984]: 3.16 scrub starts 2024-02-05T06:35:18.477 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:18 smithi028 ceph-mon[37984]: 3.16 scrub ok 2024-02-05T06:35:18.477 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:18 smithi028 ceph-mon[37984]: 3.8 scrub starts 2024-02-05T06:35:18.477 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:18 smithi028 ceph-mon[37984]: 3.8 scrub ok 2024-02-05T06:35:18.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:18 smithi167 ceph-mon[32774]: 3.16 scrub starts 2024-02-05T06:35:18.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:18 smithi167 ceph-mon[32774]: 3.16 scrub ok 2024-02-05T06:35:18.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:18 smithi167 ceph-mon[32774]: 3.8 scrub starts 2024-02-05T06:35:18.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:18 smithi167 ceph-mon[32774]: 3.8 scrub ok 2024-02-05T06:35:18.568 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:18.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:18.593 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:18.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:18.694 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:18.718 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:18.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:18.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:18.820 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:18.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:18.844 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:18.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:18.945 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:18.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:18.974 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:18.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:19.075 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:19.102 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:19.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:19.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:19.204 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:19.229 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:19.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:19.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:19.331 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:35:19.359 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:19.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:19.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:35:19.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:19 smithi028 ceph-mon[29442]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:35:19.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:19 smithi028 ceph-mon[37984]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:35:19.462 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:19.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:19 smithi167 ceph-mon[32774]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:35:19.486 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:19.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:19.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:19.588 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:19.617 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:19.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:19.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:19.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:19.743 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:19.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:19.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:19.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:19.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:19.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:19.969 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:19.993 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:19.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:19.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:20.095 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:20.118 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:20.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:20.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:20.220 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:20.243 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:20.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:20.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:20.345 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:20.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:20.369 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:20.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:20.472 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:35:20.496 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:20.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:20.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:35:20.597 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:35:20.619 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:35:20.619 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:35:20.621 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:20.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:20.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:35:20.723 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:20.728 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:20 smithi167 bash[63932]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-02-05T06:35:20.728 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:20 smithi167 bash[63932]: Writing manifest to image destination 2024-02-05T06:35:20.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:20.748 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:20.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:20.849 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:20.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:20.873 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:20.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:20.974 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:20.999 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:21.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:21.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:21.077 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:20 smithi167 ceph-mon[32774]: 6.8 scrub starts 2024-02-05T06:35:21.077 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:20 smithi167 ceph-mon[32774]: 6.8 scrub ok 2024-02-05T06:35:21.077 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:20 smithi167 ceph-mon[32774]: 6.c deep-scrub starts 2024-02-05T06:35:21.078 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:20 smithi167 ceph-mon[32774]: 6.c deep-scrub ok 2024-02-05T06:35:21.078 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:20 smithi167 ceph-mon[32774]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:35:21.078 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:20 smithi167 podman[63932]: 2024-02-05 06:35:20.733018631 +0000 UTC m=+7.402716221 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-02-05T06:35:21.078 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:20 smithi167 podman[63932]: 2024-02-05 06:35:20.840330481 +0000 UTC m=+7.510028069 container create 201f32f85bdf13c94d1df168b9acca0e666c49ab8286b26572587a9410b90f2b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-05T06:35:21.078 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 podman[63932]: 2024-02-05 06:35:21.076166911 +0000 UTC m=+7.745864499 container init 201f32f85bdf13c94d1df168b9acca0e666c49ab8286b26572587a9410b90f2b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-05T06:35:21.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:21.126 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:21.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:21.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:21.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:20 smithi028 ceph-mon[37984]: 6.8 scrub starts 2024-02-05T06:35:21.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:20 smithi028 ceph-mon[37984]: 6.8 scrub ok 2024-02-05T06:35:21.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:20 smithi028 ceph-mon[37984]: 6.c deep-scrub starts 2024-02-05T06:35:21.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:20 smithi028 ceph-mon[37984]: 6.c deep-scrub ok 2024-02-05T06:35:21.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:20 smithi028 ceph-mon[37984]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:35:21.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:20 smithi028 ceph-mon[29442]: 6.8 scrub starts 2024-02-05T06:35:21.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:20 smithi028 ceph-mon[29442]: 6.8 scrub ok 2024-02-05T06:35:21.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:20 smithi028 ceph-mon[29442]: 6.c deep-scrub starts 2024-02-05T06:35:21.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:20 smithi028 ceph-mon[29442]: 6.c deep-scrub ok 2024-02-05T06:35:21.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:20 smithi028 ceph-mon[29442]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:35:21.228 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:21.252 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:21.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:21.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:21.354 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:21.377 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:21.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:21.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:21.479 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 podman[63932]: 2024-02-05 06:35:21.078742188 +0000 UTC m=+7.748439784 container start 201f32f85bdf13c94d1df168b9acca0e666c49ab8286b26572587a9410b90f2b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-05T06:35:21.480 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.082Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-02-05T06:35:21.480 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.082Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-02-05T06:35:21.480 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-02-05T06:35:21.480 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-02-05T06:35:21.480 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-02-05T06:35:21.480 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-05T06:35:21.480 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-02-05T06:35:21.480 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=arp 2024-02-05T06:35:21.480 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=bcache 2024-02-05T06:35:21.480 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=bonding 2024-02-05T06:35:21.480 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=btrfs 2024-02-05T06:35:21.480 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=conntrack 2024-02-05T06:35:21.481 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=cpu 2024-02-05T06:35:21.481 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-02-05T06:35:21.481 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=diskstats 2024-02-05T06:35:21.481 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=dmi 2024-02-05T06:35:21.481 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=edac 2024-02-05T06:35:21.481 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=entropy 2024-02-05T06:35:21.481 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-02-05T06:35:21.481 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=filefd 2024-02-05T06:35:21.481 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=filesystem 2024-02-05T06:35:21.481 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=hwmon 2024-02-05T06:35:21.481 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=infiniband 2024-02-05T06:35:21.481 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=ipvs 2024-02-05T06:35:21.481 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=loadavg 2024-02-05T06:35:21.481 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=mdadm 2024-02-05T06:35:21.482 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=meminfo 2024-02-05T06:35:21.482 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=netclass 2024-02-05T06:35:21.482 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=netdev 2024-02-05T06:35:21.482 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=netstat 2024-02-05T06:35:21.482 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=nfs 2024-02-05T06:35:21.482 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=nfsd 2024-02-05T06:35:21.482 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=nvme 2024-02-05T06:35:21.482 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=os 2024-02-05T06:35:21.482 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-02-05T06:35:21.482 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=pressure 2024-02-05T06:35:21.482 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=rapl 2024-02-05T06:35:21.482 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=schedstat 2024-02-05T06:35:21.482 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=selinux 2024-02-05T06:35:21.482 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:21.483 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=sockstat 2024-02-05T06:35:21.483 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=softnet 2024-02-05T06:35:21.483 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=stat 2024-02-05T06:35:21.483 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=tapestats 2024-02-05T06:35:21.483 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=textfile 2024-02-05T06:35:21.484 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-02-05T06:35:21.484 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=time 2024-02-05T06:35:21.484 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-02-05T06:35:21.484 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=uname 2024-02-05T06:35:21.484 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=vmstat 2024-02-05T06:35:21.484 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=xfs 2024-02-05T06:35:21.484 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.083Z caller=node_exporter.go:117 level=info collector=zfs 2024-02-05T06:35:21.484 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.084Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-02-05T06:35:21.484 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b[64189]: ts=2024-02-05T06:35:21.084Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-02-05T06:35:21.484 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 bash[63932]: 201f32f85bdf13c94d1df168b9acca0e666c49ab8286b26572587a9410b90f2b 2024-02-05T06:35:21.484 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:35:21 smithi167 systemd[1]: Started Ceph node-exporter.b for 9801e956-c3ef-11ee-95b6-87774f69a715. 2024-02-05T06:35:21.507 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:21.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:21.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:21.609 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:21.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:21.633 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:21.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:21.734 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:35:21.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:21.760 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:21.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:35:21.861 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:21.885 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:21.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:21.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:21.987 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:22.011 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:22.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:22.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:22.113 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:22.136 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:22.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:22.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:22.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:21 smithi028 ceph-mon[37984]: 6.1c scrub starts 2024-02-05T06:35:22.191 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:21 smithi028 ceph-mon[37984]: 6.1c scrub ok 2024-02-05T06:35:22.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:21 smithi028 ceph-mon[37984]: 6.18 scrub starts 2024-02-05T06:35:22.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:21 smithi028 ceph-mon[37984]: 6.18 scrub ok 2024-02-05T06:35:22.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:21 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:22.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:21 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:22.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:21 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:22.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:21 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:22.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:21 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:22.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:21 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:35:22.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:21 smithi028 ceph-mon[37984]: 3.9 scrub starts 2024-02-05T06:35:22.192 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:21 smithi028 ceph-mon[37984]: 3.9 scrub ok 2024-02-05T06:35:22.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:21 smithi028 ceph-mon[29442]: 6.1c scrub starts 2024-02-05T06:35:22.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:21 smithi028 ceph-mon[29442]: 6.1c scrub ok 2024-02-05T06:35:22.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:21 smithi028 ceph-mon[29442]: 6.18 scrub starts 2024-02-05T06:35:22.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:21 smithi028 ceph-mon[29442]: 6.18 scrub ok 2024-02-05T06:35:22.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:21 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:22.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:21 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:22.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:21 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:22.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:21 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:22.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:21 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:22.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:21 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:35:22.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:21 smithi028 ceph-mon[29442]: 3.9 scrub starts 2024-02-05T06:35:22.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:21 smithi028 ceph-mon[29442]: 3.9 scrub ok 2024-02-05T06:35:22.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-mon[32774]: 6.1c scrub starts 2024-02-05T06:35:22.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-mon[32774]: 6.1c scrub ok 2024-02-05T06:35:22.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-mon[32774]: 6.18 scrub starts 2024-02-05T06:35:22.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-mon[32774]: 6.18 scrub ok 2024-02-05T06:35:22.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:22.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:22.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:22.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:22.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:22.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:35:22.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-mon[32774]: 3.9 scrub starts 2024-02-05T06:35:22.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:21 smithi167 ceph-mon[32774]: 3.9 scrub ok 2024-02-05T06:35:22.238 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:22.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:22.263 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:22.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:22.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:22.388 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:22.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:22.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:22.490 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:22.514 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:22.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:22.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:22.616 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:22.640 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:22.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:22.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:22.729 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:35:22 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:35:22.741 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:22.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:22.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:22.782 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:35:22 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:35:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:35:22.868 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:22.893 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:22.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:22.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:22.995 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:23.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:23.020 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:23.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:23.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:22 smithi028 ceph-mon[29442]: 3.0 scrub starts 2024-02-05T06:35:23.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:22 smithi028 ceph-mon[29442]: 3.0 scrub ok 2024-02-05T06:35:23.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:22 smithi028 ceph-mon[29442]: 6.19 deep-scrub starts 2024-02-05T06:35:23.107 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:22 smithi028 ceph-mon[29442]: 6.19 deep-scrub ok 2024-02-05T06:35:23.108 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:22 smithi028 ceph-mon[29442]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:35:23.108 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:22 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:35:23.108 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:22 smithi028 ceph-mon[37984]: 3.0 scrub starts 2024-02-05T06:35:23.108 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:22 smithi028 ceph-mon[37984]: 3.0 scrub ok 2024-02-05T06:35:23.108 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:22 smithi028 ceph-mon[37984]: 6.19 deep-scrub starts 2024-02-05T06:35:23.108 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:22 smithi028 ceph-mon[37984]: 6.19 deep-scrub ok 2024-02-05T06:35:23.108 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:22 smithi028 ceph-mon[37984]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:35:23.108 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:22 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:35:23.122 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:23.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:23.146 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:23.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:23.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:22 smithi167 ceph-mon[32774]: 3.0 scrub starts 2024-02-05T06:35:23.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:22 smithi167 ceph-mon[32774]: 3.0 scrub ok 2024-02-05T06:35:23.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:22 smithi167 ceph-mon[32774]: 6.19 deep-scrub starts 2024-02-05T06:35:23.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:22 smithi167 ceph-mon[32774]: 6.19 deep-scrub ok 2024-02-05T06:35:23.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:22 smithi167 ceph-mon[32774]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:35:23.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:22 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:35:23.247 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:23.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:23.272 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:23.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:23.324 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:35:23.325 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-02-05T06:35:23.325 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-02-05T06:35:23.325 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:35:23.372 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:23.397 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:23.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:23.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:23.498 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:23.522 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:23.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:23.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:23.624 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:23.648 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:23.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:23.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:23.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:23 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-a[29438]: 2024-02-05T06:35:23.437+0000 7fa733293700 -1 mon.a@0(leader).osd e93 definitely_dead 0 2024-02-05T06:35:23.751 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:23.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:23.777 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:23.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:23.878 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:23.901 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:23.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:23.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:23.925 INFO:teuthology.orchestra.run.smithi028.stderr:marked out osd.4. 2024-02-05T06:35:23.989 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-05T06:35:24.003 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:24.056 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:24.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:24.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:24.158 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:24.183 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:24.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:24.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:24.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:23 smithi028 ceph-mon[29442]: 3.1b scrub starts 2024-02-05T06:35:24.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:23 smithi028 ceph-mon[29442]: 3.1b scrub ok 2024-02-05T06:35:24.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:23 smithi028 ceph-mon[29442]: 4.11 deep-scrub starts 2024-02-05T06:35:24.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:23 smithi028 ceph-mon[29442]: 4.11 deep-scrub ok 2024-02-05T06:35:24.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:23 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1369751071' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-02-05T06:35:24.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:23 smithi028 ceph-mon[29442]: Client client.admin marked osd.4 out, while it was still marked up 2024-02-05T06:35:24.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:23 smithi028 ceph-mon[37984]: 3.1b scrub starts 2024-02-05T06:35:24.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:23 smithi028 ceph-mon[37984]: 3.1b scrub ok 2024-02-05T06:35:24.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:23 smithi028 ceph-mon[37984]: 4.11 deep-scrub starts 2024-02-05T06:35:24.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:23 smithi028 ceph-mon[37984]: 4.11 deep-scrub ok 2024-02-05T06:35:24.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:23 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1369751071' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-02-05T06:35:24.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:23 smithi028 ceph-mon[37984]: Client client.admin marked osd.4 out, while it was still marked up 2024-02-05T06:35:24.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:23 smithi167 ceph-mon[32774]: 3.1b scrub starts 2024-02-05T06:35:24.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:23 smithi167 ceph-mon[32774]: 3.1b scrub ok 2024-02-05T06:35:24.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:23 smithi167 ceph-mon[32774]: 4.11 deep-scrub starts 2024-02-05T06:35:24.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:23 smithi167 ceph-mon[32774]: 4.11 deep-scrub ok 2024-02-05T06:35:24.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:23 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1369751071' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-02-05T06:35:24.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:23 smithi167 ceph-mon[32774]: Client client.admin marked osd.4 out, while it was still marked up 2024-02-05T06:35:24.285 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:24.309 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:24.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:24.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:24.411 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:24.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:24.436 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:24.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:24.537 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:24.561 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:24.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:24.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:24.663 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:24.686 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:24.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:24.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:24.788 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:24.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:24.814 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:24.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:24.915 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:24.957 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:24.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:24.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:25.059 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:35:25.083 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:25.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:25.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:35:25.184 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:25.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:24 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1369751071' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-02-05T06:35:25.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:24 smithi028 ceph-mon[29442]: osdmap e94: 8 total, 8 up, 7 in 2024-02-05T06:35:25.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:24 smithi028 ceph-mon[29442]: 5.7 scrub starts 2024-02-05T06:35:25.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:24 smithi028 ceph-mon[29442]: 5.7 scrub ok 2024-02-05T06:35:25.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:24 smithi028 ceph-mon[29442]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 208 MiB used, 626 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:35:25.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:24 smithi028 ceph-mon[29442]: osdmap e95: 8 total, 8 up, 7 in 2024-02-05T06:35:25.186 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:24 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1369751071' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-02-05T06:35:25.186 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:24 smithi028 ceph-mon[37984]: osdmap e94: 8 total, 8 up, 7 in 2024-02-05T06:35:25.186 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:24 smithi028 ceph-mon[37984]: 5.7 scrub starts 2024-02-05T06:35:25.187 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:24 smithi028 ceph-mon[37984]: 5.7 scrub ok 2024-02-05T06:35:25.187 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:24 smithi028 ceph-mon[37984]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 208 MiB used, 626 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:35:25.187 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:24 smithi028 ceph-mon[37984]: osdmap e95: 8 total, 8 up, 7 in 2024-02-05T06:35:25.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:25.210 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:25.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:25.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:24 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1369751071' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-02-05T06:35:25.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:24 smithi167 ceph-mon[32774]: osdmap e94: 8 total, 8 up, 7 in 2024-02-05T06:35:25.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:24 smithi167 ceph-mon[32774]: 5.7 scrub starts 2024-02-05T06:35:25.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:24 smithi167 ceph-mon[32774]: 5.7 scrub ok 2024-02-05T06:35:25.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:24 smithi167 ceph-mon[32774]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 208 MiB used, 626 GiB / 626 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:35:25.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:24 smithi167 ceph-mon[32774]: osdmap e95: 8 total, 8 up, 7 in 2024-02-05T06:35:25.311 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:25.336 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:25.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:25.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:25.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:25.462 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:25.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:25.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:25.563 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:25.588 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:25.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:25.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:25.694 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:25.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:25.721 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:25.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:25.822 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:25.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:25.846 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:25.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:25.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:25.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:25.972 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:25.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:26.074 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:26.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:26.098 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:26.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:26.199 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:35:26.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:26.224 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:26.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:35:26.325 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:26.348 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:26.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:26.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:26.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:26 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:26.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:26 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:35:26.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:26 smithi028 ceph-mon[29442]: 3.6 deep-scrub starts 2024-02-05T06:35:26.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:26 smithi028 ceph-mon[29442]: 3.6 deep-scrub ok 2024-02-05T06:35:26.383 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:26 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:26.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:26 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:26.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:26 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3382004656' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-02-05T06:35:26.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:26 smithi028 ceph-mon[29442]: from='client.44708 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-02-05T06:35:26.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:26 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:26.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:26 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:35:26.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:26 smithi028 ceph-mon[37984]: 3.6 deep-scrub starts 2024-02-05T06:35:26.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:26 smithi028 ceph-mon[37984]: 3.6 deep-scrub ok 2024-02-05T06:35:26.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:26 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:26.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:26 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:26.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:26 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3382004656' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-02-05T06:35:26.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:26 smithi028 ceph-mon[37984]: from='client.44708 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-02-05T06:35:26.450 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:26.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:26.475 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:26.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:26.576 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:26.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:26.601 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:26.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:26.702 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:26.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:26.726 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:26.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:26.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:26 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:26.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:26 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:35:26.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:26 smithi167 ceph-mon[32774]: 3.6 deep-scrub starts 2024-02-05T06:35:26.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:26 smithi167 ceph-mon[32774]: 3.6 deep-scrub ok 2024-02-05T06:35:26.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:26 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:26.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:26 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:26.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:26 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3382004656' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-02-05T06:35:26.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:26 smithi167 ceph-mon[32774]: from='client.44708 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-02-05T06:35:26.827 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:26.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:26.851 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:26.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:26.931 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'unique_pool_0' created 2024-02-05T06:35:26.951 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:26.974 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:26.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:26.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:27.076 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:27.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:27.102 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:27.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:27.135 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:35:27.156 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:35:27.175 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:35:27.194 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:35:27.203 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:27.214 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:35:27.226 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:27.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:27.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:27.234 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:35:27.255 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:35:27.275 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:35:27.277 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:35:27.277 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:35:27.277 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:35:27.277 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:35:27.277 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:35:27.277 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:35:27.277 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:35:27.277 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:35:27.328 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:27.339 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:35:27.355 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:27.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:27.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:27.457 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:27.484 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:27.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:27.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:27.585 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:27.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:27.611 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:27.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:27.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:27 smithi028 ceph-mon[29442]: 6.2 scrub starts 2024-02-05T06:35:27.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:27 smithi028 ceph-mon[29442]: 6.2 scrub ok 2024-02-05T06:35:27.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:27 smithi028 ceph-mon[29442]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 243 B/s, 9 objects/s recovering 2024-02-05T06:35:27.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:27 smithi028 ceph-mon[29442]: 4.d deep-scrub starts 2024-02-05T06:35:27.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:27 smithi028 ceph-mon[29442]: 4.d deep-scrub ok 2024-02-05T06:35:27.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:27 smithi028 ceph-mon[29442]: from='client.44708 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-02-05T06:35:27.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:27 smithi028 ceph-mon[29442]: osdmap e96: 8 total, 8 up, 7 in 2024-02-05T06:35:27.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:27 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/4097420809' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:35:27.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:27 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3663496563' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:35:27.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:27 smithi028 ceph-mon[37984]: 6.2 scrub starts 2024-02-05T06:35:27.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:27 smithi028 ceph-mon[37984]: 6.2 scrub ok 2024-02-05T06:35:27.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:27 smithi028 ceph-mon[37984]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 243 B/s, 9 objects/s recovering 2024-02-05T06:35:27.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:27 smithi028 ceph-mon[37984]: 4.d deep-scrub starts 2024-02-05T06:35:27.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:27 smithi028 ceph-mon[37984]: 4.d deep-scrub ok 2024-02-05T06:35:27.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:27 smithi028 ceph-mon[37984]: from='client.44708 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-02-05T06:35:27.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:27 smithi028 ceph-mon[37984]: osdmap e96: 8 total, 8 up, 7 in 2024-02-05T06:35:27.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:27 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/4097420809' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:35:27.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:27 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3663496563' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:35:27.712 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:27.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:27 smithi167 ceph-mon[32774]: 6.2 scrub starts 2024-02-05T06:35:27.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:27 smithi167 ceph-mon[32774]: 6.2 scrub ok 2024-02-05T06:35:27.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:27 smithi167 ceph-mon[32774]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 243 B/s, 9 objects/s recovering 2024-02-05T06:35:27.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:27 smithi167 ceph-mon[32774]: 4.d deep-scrub starts 2024-02-05T06:35:27.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:27 smithi167 ceph-mon[32774]: 4.d deep-scrub ok 2024-02-05T06:35:27.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:27 smithi167 ceph-mon[32774]: from='client.44708 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-02-05T06:35:27.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:27 smithi167 ceph-mon[32774]: osdmap e96: 8 total, 8 up, 7 in 2024-02-05T06:35:27.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:27 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/4097420809' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:35:27.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:27 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3663496563' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:35:27.740 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:27.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:27.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:27.759 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-02-05T06:35:27.842 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:27.869 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:27.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:27.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:27.930 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-02-05T06:35:27.972 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:27.996 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:27.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:27.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:28.097 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:28.121 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:28.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:28.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:28.223 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:28.252 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:28.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:28.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:28.339 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:28 smithi028 ceph-mon[29442]: 6.e scrub starts 2024-02-05T06:35:28.339 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:28 smithi028 ceph-mon[29442]: 6.e scrub ok 2024-02-05T06:35:28.340 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:28 smithi028 ceph-mon[29442]: 4.3 scrub starts 2024-02-05T06:35:28.340 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:28 smithi028 ceph-mon[29442]: 4.3 scrub ok 2024-02-05T06:35:28.340 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:28 smithi028 ceph-mon[29442]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:35:28.340 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:28 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3663496563' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:35:28.340 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:28 smithi028 ceph-mon[29442]: osdmap e97: 8 total, 8 up, 7 in 2024-02-05T06:35:28.340 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:28 smithi028 ceph-mon[37984]: 6.e scrub starts 2024-02-05T06:35:28.340 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:28 smithi028 ceph-mon[37984]: 6.e scrub ok 2024-02-05T06:35:28.340 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:28 smithi028 ceph-mon[37984]: 4.3 scrub starts 2024-02-05T06:35:28.340 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:28 smithi028 ceph-mon[37984]: 4.3 scrub ok 2024-02-05T06:35:28.340 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:28 smithi028 ceph-mon[37984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:35:28.340 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:28 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3663496563' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:35:28.341 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:28 smithi028 ceph-mon[37984]: osdmap e97: 8 total, 8 up, 7 in 2024-02-05T06:35:28.354 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:28.381 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:28.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:28.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:28.483 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:28.510 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:28.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:28.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:28.612 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:28.638 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:28.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:28.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:28.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:28 smithi167 ceph-mon[32774]: 6.e scrub starts 2024-02-05T06:35:28.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:28 smithi167 ceph-mon[32774]: 6.e scrub ok 2024-02-05T06:35:28.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:28 smithi167 ceph-mon[32774]: 4.3 scrub starts 2024-02-05T06:35:28.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:28 smithi167 ceph-mon[32774]: 4.3 scrub ok 2024-02-05T06:35:28.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:28 smithi167 ceph-mon[32774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:35:28.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:28 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3663496563' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:35:28.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:28 smithi167 ceph-mon[32774]: osdmap e97: 8 total, 8 up, 7 in 2024-02-05T06:35:28.740 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:28.766 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:28.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:28.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:28.867 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:28.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:28.897 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:28.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:28.998 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:29.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:29.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:29.125 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:29.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:29.150 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:29.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:29.251 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:29.276 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:29.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:29.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:29.378 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:29.403 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:29.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:29.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:29.505 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:29.529 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:29 smithi028 ceph-mon[29442]: pgmap v57: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 457 B/s, 18 objects/s recovering 2024-02-05T06:35:29.529 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:29 smithi028 ceph-mon[29442]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T06:35:29.529 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:29 smithi028 ceph-mon[29442]: osdmap e98: 8 total, 8 up, 7 in 2024-02-05T06:35:29.530 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:29 smithi028 ceph-mon[37984]: pgmap v57: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 457 B/s, 18 objects/s recovering 2024-02-05T06:35:29.530 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:29 smithi028 ceph-mon[37984]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T06:35:29.530 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:29 smithi028 ceph-mon[37984]: osdmap e98: 8 total, 8 up, 7 in 2024-02-05T06:35:29.534 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:29.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:29.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:29.603 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T06:35:29.603 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T06:35:29.603 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-05T06:35:29.604 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-05T06:35:29.636 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:29.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:29.672 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:29.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:29.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:29 smithi167 ceph-mon[32774]: pgmap v57: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 457 B/s, 18 objects/s recovering 2024-02-05T06:35:29.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:29 smithi167 ceph-mon[32774]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T06:35:29.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:29 smithi167 ceph-mon[32774]: osdmap e98: 8 total, 8 up, 7 in 2024-02-05T06:35:29.773 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:29.801 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:29.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:29.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:29.903 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:29.931 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:29.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:29.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:30.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:30.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:30.057 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:30.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:30.158 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:30.185 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:30.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:30.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:30.287 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:35:30.314 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:30.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:30.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:35:30.416 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:30.439 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:30.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:30.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:30.541 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:35:30.569 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:30.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:30.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:35:30.671 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:35:30.699 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:30.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:30.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:35:30.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T06:35:30.801 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:30.828 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:30.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:30.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:30.930 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:35:30.958 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:30.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:30.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:35:31.060 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:31.084 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:31.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:31.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:31.141 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:30 smithi028 ceph-mon[37984]: pgmap v59: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 370 B/s, 14 objects/s recovering 2024-02-05T06:35:31.141 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:30 smithi028 ceph-mon[29442]: pgmap v59: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 370 B/s, 14 objects/s recovering 2024-02-05T06:35:31.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:31.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:31.212 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:31.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:31.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:30 smithi167 ceph-mon[32774]: pgmap v59: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 370 B/s, 14 objects/s recovering 2024-02-05T06:35:31.313 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:31.341 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:31.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:31.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:31.424 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:35:31.432 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:35:31.443 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:31.447 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:35:31.447 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:35:31.479 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:31.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:31.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:31.582 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:31.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:31.607 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:31.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:31.708 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:31.736 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:31.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:31.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:31.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:31.862 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:31.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:31.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:31.891 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:35:31.964 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:31.992 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:31.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:31.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:32.094 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:32.112 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:31 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:35:32.112 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:31 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:35:32.112 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:31 smithi028 ceph-mon[29442]: osdmap e99: 8 total, 8 up, 7 in 2024-02-05T06:35:32.112 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:31 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:35:32.112 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:31 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:35:32.112 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:31 smithi028 ceph-mon[37984]: osdmap e99: 8 total, 8 up, 7 in 2024-02-05T06:35:32.122 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:32.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:32.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:32.224 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:32.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:31 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:35:32.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:31 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:35:32.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:31 smithi167 ceph-mon[32774]: osdmap e99: 8 total, 8 up, 7 in 2024-02-05T06:35:32.251 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:32.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:32.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:32.353 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:32.381 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:32.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:32.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:32.483 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:32.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:32.509 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:32.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:32.610 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:35:32.638 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:32.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:32.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:35:32.729 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:35:32 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:35:32.740 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:32.767 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:32.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:32.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:32.825 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:35:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:35:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:35:32.869 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:32.897 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:32.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:32.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:32.900 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:35:32.999 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:33.023 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:33.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:33.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:33.109 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:32 smithi028 ceph-mon[29442]: pgmap v61: 148 pgs: 148 active+clean; 582 KiB data, 211 MiB used, 626 GiB / 626 GiB avail; 976 B/s rd, 0 op/s 2024-02-05T06:35:33.109 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:32 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:35:33.109 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:32 smithi028 ceph-mon[37984]: pgmap v61: 148 pgs: 148 active+clean; 582 KiB data, 211 MiB used, 626 GiB / 626 GiB avail; 976 B/s rd, 0 op/s 2024-02-05T06:35:33.109 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:32 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:35:33.125 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:33.147 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:33.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:33.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:33.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:32 smithi167 ceph-mon[32774]: pgmap v61: 148 pgs: 148 active+clean; 582 KiB data, 211 MiB used, 626 GiB / 626 GiB avail; 976 B/s rd, 0 op/s 2024-02-05T06:35:33.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:32 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:35:33.249 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:35:33.278 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:33.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:33.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:35:33.379 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:33.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:33.404 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:33.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:33.505 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:33.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:33.529 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:33.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:33.630 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:33.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:33.655 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:33.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:33.756 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:33.779 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:33.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:33.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:33.881 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:33.904 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:33.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:33.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:33.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:33 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:35:33.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:33 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:35:33.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:33 smithi028 ceph-mon[29442]: osdmap e100: 8 total, 8 up, 7 in 2024-02-05T06:35:33.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:33 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:35:33.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:33 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:35:33.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:33 smithi028 ceph-mon[37984]: osdmap e100: 8 total, 8 up, 7 in 2024-02-05T06:35:34.006 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:35:34.034 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:34.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:34.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:35:34.136 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:34.159 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:34.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:34.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:34.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:33 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:35:34.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:33 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:35:34.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:33 smithi167 ceph-mon[32774]: osdmap e100: 8 total, 8 up, 7 in 2024-02-05T06:35:34.261 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:34.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:34.285 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:34.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:34.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:34.410 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:34.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:34.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:34.512 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:34.540 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:34.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:34.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:34.642 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:34.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:34.669 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:34.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:34.740 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-05T06:35:34.769 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:34.808 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:34.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:34.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:34.910 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:34.932 INFO:teuthology.orchestra.run.smithi028.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-02-05T06:35:34.937 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:34.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:34.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:35.039 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:35.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:35.064 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:35.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:35.125 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:34 smithi028 ceph-mon[37984]: pgmap v63: 148 pgs: 148 active+clean; 582 KiB data, 211 MiB used, 626 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:35:35.125 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:34 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/273040649' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T06:35:35.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:34 smithi028 ceph-mon[29442]: pgmap v63: 148 pgs: 148 active+clean; 582 KiB data, 211 MiB used, 626 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:35:35.125 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:34 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/273040649' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T06:35:35.165 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:35.190 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:35.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:35.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:35.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:34 smithi167 ceph-mon[32774]: pgmap v63: 148 pgs: 148 active+clean; 582 KiB data, 211 MiB used, 626 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:35:35.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:34 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/273040649' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T06:35:35.292 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:35.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:35.317 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:35.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:35.417 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:35.441 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:35.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:35.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:35.543 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:35.566 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:35.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:35.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:35.668 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:35.692 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:35.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:35.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:35.794 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:35.818 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:35.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:35.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:35.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:35.947 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:35.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:35.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:35.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:35 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/273040649' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T06:35:35.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:35 smithi167 ceph-mon[32774]: osdmap e101: 8 total, 8 up, 7 in 2024-02-05T06:35:36.049 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:36.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:36.074 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:36.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:36.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:35 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/273040649' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T06:35:36.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:35 smithi028 ceph-mon[29442]: osdmap e101: 8 total, 8 up, 7 in 2024-02-05T06:35:36.114 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:35 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/273040649' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T06:35:36.115 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:35 smithi028 ceph-mon[37984]: osdmap e101: 8 total, 8 up, 7 in 2024-02-05T06:35:36.175 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:36.199 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:36.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:36.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:36.300 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:36.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:36.325 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:36.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:36.426 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:36.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:36.451 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:36.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:36.552 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:35:36.576 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:36.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:36.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:35:36.678 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:36.702 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:36.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:36.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:36.804 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:36.828 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:36.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:36.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:36.846 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:35:36.847 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:35:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 101, 2024-02-05T06:35:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "9801e956-c3ef-11ee-95b6-87774f69a715", 2024-02-05T06:35:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-02-05T06:27:59.919226+0000", 2024-02-05T06:35:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-02-05T06:35:34.918304+0000", 2024-02-05T06:35:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: "last_up_change": "2024-02-05T06:33:15.833573+0000", 2024-02-05T06:35:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: "last_in_change": "2024-02-05T06:35:23.906253+0000", 2024-02-05T06:35:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-05T06:35:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_num": 5801984, 2024-02-05T06:35:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_set": [ 2024-02-05T06:35:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: "noscrub", 2024-02-05T06:35:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: "pglog_hardlimit", 2024-02-05T06:35:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "purged_snapdirs", 2024-02-05T06:35:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_deletes", 2024-02-05T06:35:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "sortbitwise" 2024-02-05T06:35:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:35:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_version": 18, 2024-02-05T06:35:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "full_ratio": 0.94999998807907104, 2024-02-05T06:35:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-05T06:35:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-05T06:35:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_snapshot": "", 2024-02-05T06:35:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_max": 7, 2024-02-05T06:35:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd": 8, 2024-02-05T06:35:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "require_min_compat_client": "luminous", 2024-02-05T06:35:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "min_compat_client": "luminous", 2024-02-05T06:35:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "require_osd_release": "squid", 2024-02-05T06:35:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_crimson": false, 2024-02-05T06:35:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "pools": [ 2024-02-05T06:35:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 1, 2024-02-05T06:35:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".mgr", 2024-02-05T06:35:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:31:10.441424+0000", 2024-02-05T06:35:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-02-05T06:35:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-02-05T06:35:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:35:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:35:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:35:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:35:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:35:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:35:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:35:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:35:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:35:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:35:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 1, 2024-02-05T06:35:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 1, 2024-02-05T06:35:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 1, 2024-02-05T06:35:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 1, 2024-02-05T06:35:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 1, 2024-02-05T06:35:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:35:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:35:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:35:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:35:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:35:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:35:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:35:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "21", 2024-02-05T06:35:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:35:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:35:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T06:35:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:35:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:35:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-02-05T06:35:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-02-05T06:35:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:35:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:35:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:35:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:35:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:35:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:35:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:35:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:35:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:35:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:35:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:35:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:35:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:35:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:35:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:35:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:35:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:35:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:35:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:35:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:35:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:35:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:35:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:35:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:35:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:35:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:35:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:35:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:35:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:35:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:35:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-02-05T06:35:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_max": 32, 2024-02-05T06:35:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_min": 1 2024-02-05T06:35:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:35:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": {} 2024-02-05T06:35:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:35:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:35:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 7.8899998664855957, 2024-02-05T06:35:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 7.8899998664855957, 2024-02-05T06:35:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.37999999523162842, 2024-02-05T06:35:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 3, 2024-02-05T06:35:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 3, 2024-02-05T06:35:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-02-05T06:35:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-02-05T06:35:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-02-05T06:35:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 2, 2024-02-05T06:35:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "datapool", 2024-02-05T06:35:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:33:27.397416+0000", 2024-02-05T06:35:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-02-05T06:35:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-05T06:35:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:35:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:35:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:35:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:35:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:35:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:35:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:35:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:35:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:35:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:35:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 3, 2024-02-05T06:35:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 3, 2024-02-05T06:35:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 3, 2024-02-05T06:35:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 3, 2024-02-05T06:35:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 3, 2024-02-05T06:35:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:35:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:35:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:35:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:35:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:35:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:35:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:35:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "69", 2024-02-05T06:35:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:35:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:35:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T06:35:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:35:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:35:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 2, 2024-02-05T06:35:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 69, 2024-02-05T06:35:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:35:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:35:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:35:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:35:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:35:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:35:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:35:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:35:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:35:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:35:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:35:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:35:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:35:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:35:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:35:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:35:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:35:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:35:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:35:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:35:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:35:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:35:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:35:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:35:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:35:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:35:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:35:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:35:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:35:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:35:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-02-05T06:35:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:35:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd": {} 2024-02-05T06:35:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:35:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:35:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.6500000953674316, 2024-02-05T06:35:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.6500000953674316, 2024-02-05T06:35:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T06:35:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2.3299999237060547, 2024-02-05T06:35:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2.3299999237060547, 2024-02-05T06:35:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-02-05T06:35:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-02-05T06:35:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-02-05T06:35:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 3, 2024-02-05T06:35:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".rgw.root", 2024-02-05T06:35:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:33:28.090506+0000", 2024-02-05T06:35:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-02-05T06:35:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-02-05T06:35:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:35:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:35:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:35:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:35:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:35:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:35:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:35:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:35:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:35:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:35:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-02-05T06:35:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-02-05T06:35:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-02-05T06:35:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-02-05T06:35:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-02-05T06:35:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:35:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:35:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:35:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:35:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:35:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:35:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:35:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "67", 2024-02-05T06:35:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:35:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:35:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T06:35:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:35:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:35:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-02-05T06:35:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-02-05T06:35:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:35:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:35:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:35:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:35:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:35:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:35:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:35:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:35:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:35:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:35:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:35:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:35:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:35:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:35:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:35:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:35:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:35:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:35:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:35:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:35:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:35:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:35:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:35:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:35:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:35:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:35:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:35:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:35:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:35:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:35:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-02-05T06:35:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:35:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-02-05T06:35:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:35:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:35:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.7400000095367432, 2024-02-05T06:35:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.7400000095367432, 2024-02-05T06:35:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T06:35:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.5299999713897705, 2024-02-05T06:35:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.5299999713897705, 2024-02-05T06:35:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-02-05T06:35:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-02-05T06:35:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-02-05T06:35:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 4, 2024-02-05T06:35:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.log", 2024-02-05T06:35:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:33:29.866249+0000", 2024-02-05T06:35:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-02-05T06:35:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-02-05T06:35:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:35:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:35:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:35:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:35:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:35:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:35:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:35:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:35:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:35:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:35:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-02-05T06:35:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-02-05T06:35:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-02-05T06:35:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-02-05T06:35:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-02-05T06:35:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:35:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:35:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:35:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:35:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:35:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:35:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:35:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "69", 2024-02-05T06:35:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:35:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:35:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T06:35:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:35:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:35:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-02-05T06:35:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-02-05T06:35:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:35:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:35:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:35:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:35:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:35:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:35:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:35:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:35:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:35:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:35:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:35:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:35:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:35:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:35:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:35:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:35:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:35:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:35:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:35:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:35:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:35:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:35:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:35:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:35:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:35:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:35:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:35:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:35:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:35:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:35:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-02-05T06:35:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:35:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-02-05T06:35:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:35:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:35:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.4900000095367432, 2024-02-05T06:35:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.4900000095367432, 2024-02-05T06:35:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T06:35:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2.190000057220459, 2024-02-05T06:35:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2.190000057220459, 2024-02-05T06:35:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-02-05T06:35:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-02-05T06:35:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-02-05T06:35:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 5, 2024-02-05T06:35:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.control", 2024-02-05T06:35:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:33:31.826006+0000", 2024-02-05T06:35:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-02-05T06:35:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-02-05T06:35:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:35:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:35:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:35:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:35:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:35:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:35:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:35:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:35:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:35:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:35:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-02-05T06:35:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-02-05T06:35:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-02-05T06:35:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-02-05T06:35:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-02-05T06:35:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:35:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:35:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:35:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:35:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:35:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:35:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:35:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "71", 2024-02-05T06:35:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:35:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:35:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T06:35:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:35:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:35:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-02-05T06:35:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-02-05T06:35:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:35:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:35:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:35:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:35:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:35:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:35:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:35:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:35:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:35:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:35:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:35:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:35:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:35:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:35:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:35:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:35:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:35:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:35:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:35:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:35:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:35:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:35:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:35:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:35:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:35:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:35:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:35:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:35:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:35:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:35:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-02-05T06:35:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:35:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-02-05T06:35:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:35:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:35:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.4900000095367432, 2024-02-05T06:35:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.4900000095367432, 2024-02-05T06:35:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T06:35:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.309999942779541, 2024-02-05T06:35:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.309999942779541, 2024-02-05T06:35:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-02-05T06:35:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-02-05T06:35:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-02-05T06:35:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 6, 2024-02-05T06:35:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.meta", 2024-02-05T06:35:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:33:33.890835+0000", 2024-02-05T06:35:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-02-05T06:35:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-02-05T06:35:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:35:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:35:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:35:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:35:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:35:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:35:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:35:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:35:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:35:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:35:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-02-05T06:35:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-02-05T06:35:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-02-05T06:35:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-02-05T06:35:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-02-05T06:35:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:35:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:35:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:35:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:35:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:35:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:35:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:35:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "73", 2024-02-05T06:35:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:35:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:35:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T06:35:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:35:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:35:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-02-05T06:35:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-02-05T06:35:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:35:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:35:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:35:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:35:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:35:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:35:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:35:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:35:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:35:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:35:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:35:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:35:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:35:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:35:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:35:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:35:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:35:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:35:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:35:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:35:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:35:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:35:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:35:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:35:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:35:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:35:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:35:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:35:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:35:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:35:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-02-05T06:35:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_bias": 4 2024-02-05T06:35:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:35:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-02-05T06:35:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:35:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:35:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.9900000095367432, 2024-02-05T06:35:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.9900000095367432, 2024-02-05T06:35:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T06:35:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.75, 2024-02-05T06:35:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.75, 2024-02-05T06:35:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-02-05T06:35:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-02-05T06:35:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-02-05T06:35:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 7, 2024-02-05T06:35:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "unique_pool_0", 2024-02-05T06:35:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:35:26.183571+0000", 2024-02-05T06:35:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-02-05T06:35:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-02-05T06:35:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:35:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:35:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:35:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:35:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:35:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:35:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:35:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:35:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:35:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:35:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 16, 2024-02-05T06:35:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 16, 2024-02-05T06:35:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 16, 2024-02-05T06:35:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 16, 2024-02-05T06:35:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 16, 2024-02-05T06:35:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:35:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:35:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:35:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:35:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:35:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:35:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:35:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "101", 2024-02-05T06:35:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:35:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:35:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T06:35:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:35:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:35:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-02-05T06:35:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-02-05T06:35:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:35:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:35:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:35:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:35:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:35:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:35:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:35:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:35:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:35:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:35:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:35:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:35:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:35:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:35:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:35:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:35:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:35:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:35:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:35:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:35:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:35:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:35:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:35:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:35:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:35:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:35:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:35:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:35:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:35:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:35:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-02-05T06:35:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:35:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: "rados": {} 2024-02-05T06:35:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:35:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:35:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.9900000095367432, 2024-02-05T06:35:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.9900000095367432, 2024-02-05T06:35:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T06:35:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.75, 2024-02-05T06:35:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.75, 2024-02-05T06:35:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-02-05T06:35:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-02-05T06:35:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-02-05T06:35:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:35:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-02-05T06:35:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-02-05T06:35:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "c19fdd5b-09f6-45d4-a519-0a938cf1f2a6", 2024-02-05T06:35:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:35:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:35:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:35:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:35:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:35:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:35:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 8, 2024-02-05T06:35:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 96, 2024-02-05T06:35:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:35:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:35:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:35:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:35:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:35:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-02-05T06:35:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 301279106 2024-02-05T06:35:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:35:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:35:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:35:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6802", 2024-02-05T06:35:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 301279106 2024-02-05T06:35:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:35:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:35:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:35:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6804", 2024-02-05T06:35:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 301279106 2024-02-05T06:35:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:35:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:35:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:35:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6803", 2024-02-05T06:35:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 301279106 2024-02-05T06:35:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6801/301279106", 2024-02-05T06:35:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6802/301279106", 2024-02-05T06:35:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6804/301279106", 2024-02-05T06:35:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6803/301279106", 2024-02-05T06:35:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:35:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:35:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:35:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-02-05T06:35:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "52245f6e-eb96-4d0d-985b-dbb6fd3ce8de", 2024-02-05T06:35:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:35:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:35:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:35:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:35:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:35:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:35:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-02-05T06:35:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 96, 2024-02-05T06:35:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:35:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:35:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:35:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:35:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:35:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6805", 2024-02-05T06:35:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1795224649 2024-02-05T06:35:36.899 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.899 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.899 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:35:36.899 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:35:36.899 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.899 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:35:36.899 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6806", 2024-02-05T06:35:36.899 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1795224649 2024-02-05T06:35:36.899 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.899 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.899 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:35:36.899 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:35:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:35:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6808", 2024-02-05T06:35:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1795224649 2024-02-05T06:35:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:35:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:35:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:35:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6807", 2024-02-05T06:35:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1795224649 2024-02-05T06:35:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6805/1795224649", 2024-02-05T06:35:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6806/1795224649", 2024-02-05T06:35:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6808/1795224649", 2024-02-05T06:35:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6807/1795224649", 2024-02-05T06:35:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:35:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:35:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:35:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-02-05T06:35:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "1e18b630-afdf-4de9-a1a2-37bdfdd01c67", 2024-02-05T06:35:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:35:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:35:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:35:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:35:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:35:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:35:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 18, 2024-02-05T06:35:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 96, 2024-02-05T06:35:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:35:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:35:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:35:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:35:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:35:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6809", 2024-02-05T06:35:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1993280018 2024-02-05T06:35:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:35:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:35:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:35:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6810", 2024-02-05T06:35:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1993280018 2024-02-05T06:35:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:35:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:35:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:35:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6812", 2024-02-05T06:35:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1993280018 2024-02-05T06:35:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:35:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:35:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:35:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6811", 2024-02-05T06:35:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1993280018 2024-02-05T06:35:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6809/1993280018", 2024-02-05T06:35:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6810/1993280018", 2024-02-05T06:35:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6812/1993280018", 2024-02-05T06:35:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6811/1993280018", 2024-02-05T06:35:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:35:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:35:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:35:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-02-05T06:35:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "3ae795af-3d59-41a6-95eb-b8447ff61282", 2024-02-05T06:35:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:35:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:35:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:35:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:35:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:35:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:35:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 26, 2024-02-05T06:35:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 96, 2024-02-05T06:35:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:35:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:35:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:35:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:35:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:35:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6813", 2024-02-05T06:35:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1294621398 2024-02-05T06:35:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:35:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:35:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:35:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6814", 2024-02-05T06:35:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1294621398 2024-02-05T06:35:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:35:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:35:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:35:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6816", 2024-02-05T06:35:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1294621398 2024-02-05T06:35:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:35:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:35:36.909 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.909 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:35:36.909 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6815", 2024-02-05T06:35:36.909 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1294621398 2024-02-05T06:35:36.909 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.909 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.909 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6813/1294621398", 2024-02-05T06:35:36.909 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6814/1294621398", 2024-02-05T06:35:36.909 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6816/1294621398", 2024-02-05T06:35:36.909 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6815/1294621398", 2024-02-05T06:35:36.909 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:35:36.909 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:35:36.909 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:35:36.909 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.910 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-02-05T06:35:36.910 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "fc6bd83d-570c-48c0-964d-e09b4608bc32", 2024-02-05T06:35:36.910 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:35:36.910 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 0, 2024-02-05T06:35:36.910 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-02-05T06:35:36.910 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:35:36.910 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:35:36.910 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:35:36.910 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 31, 2024-02-05T06:35:36.910 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 77, 2024-02-05T06:35:36.910 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:35:36.911 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:35:36.911 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:35:36.911 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:35:36.911 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.911 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:35:36.911 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6800", 2024-02-05T06:35:36.911 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4052090595 2024-02-05T06:35:36.911 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.911 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.911 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:35:36.911 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:35:36.911 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.911 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:35:36.912 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6801", 2024-02-05T06:35:36.912 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4052090595 2024-02-05T06:35:36.912 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.912 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.912 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:35:36.912 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:35:36.912 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.912 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:35:36.912 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6803", 2024-02-05T06:35:36.912 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4052090595 2024-02-05T06:35:36.912 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.912 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.913 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:35:36.913 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:35:36.913 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.913 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:35:36.913 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6802", 2024-02-05T06:35:36.913 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4052090595 2024-02-05T06:35:36.913 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.913 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.913 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.167:6800/4052090595", 2024-02-05T06:35:36.913 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.167:6801/4052090595", 2024-02-05T06:35:36.913 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.167:6803/4052090595", 2024-02-05T06:35:36.913 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.167:6802/4052090595", 2024-02-05T06:35:36.913 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:35:36.914 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:35:36.914 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:35:36.914 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.914 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.914 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-02-05T06:35:36.914 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "895bf465-2531-4d9f-bc23-1e75fd969f07", 2024-02-05T06:35:36.914 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:35:36.914 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:35:36.914 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:35:36.914 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:35:36.914 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:35:36.914 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:35:36.914 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 40, 2024-02-05T06:35:36.915 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 96, 2024-02-05T06:35:36.915 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:35:36.915 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:35:36.915 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:35:36.915 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:35:36.915 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.915 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:35:36.915 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6804", 2024-02-05T06:35:36.915 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1631080500 2024-02-05T06:35:36.915 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.915 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.915 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:35:36.915 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:35:36.915 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.916 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:35:36.916 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6805", 2024-02-05T06:35:36.916 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1631080500 2024-02-05T06:35:36.916 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.916 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.916 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:35:36.916 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:35:36.916 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.916 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:35:36.917 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6807", 2024-02-05T06:35:36.917 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1631080500 2024-02-05T06:35:36.917 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.917 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.918 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.918 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:35:36.918 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:35:36.918 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.918 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:35:36.918 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6806", 2024-02-05T06:35:36.918 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1631080500 2024-02-05T06:35:36.918 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.918 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.918 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.918 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.167:6804/1631080500", 2024-02-05T06:35:36.918 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.167:6805/1631080500", 2024-02-05T06:35:36.918 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.167:6807/1631080500", 2024-02-05T06:35:36.919 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.167:6806/1631080500", 2024-02-05T06:35:36.919 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:35:36.919 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:35:36.919 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:35:36.919 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.919 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.919 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-02-05T06:35:36.919 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "ce0a9300-87a4-4e37-b702-d667f168c0e0", 2024-02-05T06:35:36.919 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:35:36.919 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:35:36.919 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:35:36.919 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:35:36.919 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:35:36.920 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:35:36.920 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 49, 2024-02-05T06:35:36.920 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 96, 2024-02-05T06:35:36.920 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:35:36.920 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:35:36.920 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:35:36.920 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:35:36.920 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.920 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:35:36.920 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6808", 2024-02-05T06:35:36.920 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1644050617 2024-02-05T06:35:36.920 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.920 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.921 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:35:36.921 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:35:36.921 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.921 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:35:36.921 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6809", 2024-02-05T06:35:36.921 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1644050617 2024-02-05T06:35:36.921 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.921 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.921 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:35:36.921 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:35:36.921 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.921 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:35:36.922 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6811", 2024-02-05T06:35:36.922 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1644050617 2024-02-05T06:35:36.922 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.922 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.922 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:35:36.922 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:35:36.922 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.922 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:35:36.922 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6810", 2024-02-05T06:35:36.922 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1644050617 2024-02-05T06:35:36.922 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.922 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.923 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.167:6808/1644050617", 2024-02-05T06:35:36.923 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.167:6809/1644050617", 2024-02-05T06:35:36.923 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.167:6811/1644050617", 2024-02-05T06:35:36.923 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.167:6810/1644050617", 2024-02-05T06:35:36.923 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:35:36.923 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:35:36.923 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:35:36.923 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.923 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.923 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-02-05T06:35:36.923 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "0c545cbc-0873-4232-8807-4af9299ef95e", 2024-02-05T06:35:36.923 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:35:36.923 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:35:36.923 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:35:36.923 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:35:36.924 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:35:36.924 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:35:36.924 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 58, 2024-02-05T06:35:36.924 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 96, 2024-02-05T06:35:36.924 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:35:36.924 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:35:36.924 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:35:36.924 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:35:36.924 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.924 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:35:36.924 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6812", 2024-02-05T06:35:36.924 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4031475764 2024-02-05T06:35:36.925 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.925 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.925 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:35:36.925 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:35:36.925 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.925 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:35:36.925 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6813", 2024-02-05T06:35:36.925 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4031475764 2024-02-05T06:35:36.925 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.925 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.925 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:35:36.925 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:35:36.925 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.926 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:35:36.926 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6815", 2024-02-05T06:35:36.926 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4031475764 2024-02-05T06:35:36.926 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.926 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.926 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:35:36.926 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:35:36.926 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.926 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:35:36.926 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6814", 2024-02-05T06:35:36.926 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4031475764 2024-02-05T06:35:36.926 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.927 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.927 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.167:6812/4031475764", 2024-02-05T06:35:36.927 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.167:6813/4031475764", 2024-02-05T06:35:36.927 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.167:6815/4031475764", 2024-02-05T06:35:36.927 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.167:6814/4031475764", 2024-02-05T06:35:36.927 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:35:36.927 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:35:36.927 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:35:36.927 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.927 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.927 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:35:36.927 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_xinfo": [ 2024-02-05T06:35:36.928 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.928 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-02-05T06:35:36.928 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:35:36.928 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:35:36.928 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:35:36.928 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:35:36.928 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:35:36.928 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:35:02.377925+0000", 2024-02-05T06:35:36.928 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:35:36.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.928 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.928 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-02-05T06:35:36.928 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:35:36.929 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:35:36.929 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:35:36.929 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:35:36.929 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:35:36.929 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:35:32.713008+0000", 2024-02-05T06:35:36.930 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:35:36.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.930 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.930 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-02-05T06:35:36.930 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:35:36.930 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:35:36.930 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:35:36.930 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:35:36.930 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:35:36.930 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:34:44.457603+0000", 2024-02-05T06:35:36.930 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:35:36.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.930 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.930 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-02-05T06:35:36.931 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:35:36.931 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:35:36.931 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:35:36.931 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:35:36.931 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:35:36.931 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:34:16.357616+0000", 2024-02-05T06:35:36.931 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:35:36.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.931 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-02-05T06:35:36.931 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:35:36.931 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:35:36.931 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:35:36.931 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:35:36.931 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 65536, 2024-02-05T06:35:36.932 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:34:41.009118+0000", 2024-02-05T06:35:36.932 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:35:36.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.932 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.932 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-02-05T06:35:36.932 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:35:36.932 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:35:36.932 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:35:36.932 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:35:36.932 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:35:36.932 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:34:41.266888+0000", 2024-02-05T06:35:36.932 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:35:36.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.933 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-02-05T06:35:36.933 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:35:36.933 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:35:36.933 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:35:36.933 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:35:36.933 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:35:36.933 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:35:30.750818+0000", 2024-02-05T06:35:36.933 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:35:36.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.933 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-02-05T06:35:36.933 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:35:36.933 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:35:36.933 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:35:36.934 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:35:36.934 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:35:36.934 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:34:35.017723+0000", 2024-02-05T06:35:36.934 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:35:36.934 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.934 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:35:36.934 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap": [], 2024-02-05T06:35:36.934 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_items": [ 2024-02-05T06:35:36.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.934 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.4", 2024-02-05T06:35:36.934 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:35:36.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.934 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-02-05T06:35:36.934 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 5 2024-02-05T06:35:36.934 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.935 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.935 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.e", 2024-02-05T06:35:36.935 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:35:36.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.935 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-02-05T06:35:36.935 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-02-05T06:35:36.935 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.935 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.935 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.1d", 2024-02-05T06:35:36.935 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:35:36.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:35:36.936 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-02-05T06:35:36.936 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-02-05T06:35:36.936 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.936 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:35:36.936 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.936 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:35:36.936 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_primaries": [], 2024-02-05T06:35:36.936 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_temp": [], 2024-02-05T06:35:36.936 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_temp": [], 2024-02-05T06:35:36.936 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklist": { 2024-02-05T06:35:36.936 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/1001442173": "2024-02-06T06:34:10.127914+0000", 2024-02-05T06:35:36.936 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3160026720": "2024-02-06T06:34:10.127914+0000", 2024-02-05T06:35:36.936 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/558171292": "2024-02-06T06:34:10.127914+0000", 2024-02-05T06:35:36.937 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2832437766": "2024-02-06T06:34:10.127914+0000", 2024-02-05T06:35:36.937 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/4125764599": "2024-02-06T06:28:54.105170+0000", 2024-02-05T06:35:36.937 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1081543653": "2024-02-06T06:28:54.105170+0000", 2024-02-05T06:35:36.937 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3794420554": "2024-02-06T06:28:27.553034+0000", 2024-02-05T06:35:36.937 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1560471317": "2024-02-06T06:34:10.127914+0000", 2024-02-05T06:35:36.937 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/4266601437": "2024-02-06T06:28:27.553034+0000", 2024-02-05T06:35:36.937 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/388893831": "2024-02-06T06:28:27.553034+0000", 2024-02-05T06:35:36.937 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/460817910": "2024-02-06T06:28:27.553034+0000", 2024-02-05T06:35:36.937 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2728305743": "2024-02-06T06:28:54.105170+0000", 2024-02-05T06:35:36.937 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3149394045": "2024-02-06T06:34:10.127914+0000", 2024-02-05T06:35:36.937 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3605013594": "2024-02-06T06:28:54.105170+0000" 2024-02-05T06:35:36.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.937 INFO:teuthology.orchestra.run.smithi028.stdout: "range_blocklist": {}, 2024-02-05T06:35:36.937 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profiles": { 2024-02-05T06:35:36.938 INFO:teuthology.orchestra.run.smithi028.stdout: "default": { 2024-02-05T06:35:36.938 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-02-05T06:35:36.938 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-02-05T06:35:36.938 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-02-05T06:35:36.938 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-02-05T06:35:36.938 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van" 2024-02-05T06:35:36.938 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:35:36.938 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps_queue": [], 2024-02-05T06:35:36.938 INFO:teuthology.orchestra.run.smithi028.stdout: "new_removed_snaps": [], 2024-02-05T06:35:36.938 INFO:teuthology.orchestra.run.smithi028.stdout: "new_purged_snaps": [], 2024-02-05T06:35:36.938 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_node_flags": {}, 2024-02-05T06:35:36.938 INFO:teuthology.orchestra.run.smithi028.stdout: "device_class_flags": {}, 2024-02-05T06:35:36.938 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": { 2024-02-05T06:35:36.939 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_enabled": false, 2024-02-05T06:35:36.939 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_bucket_count": 0, 2024-02-05T06:35:36.939 INFO:teuthology.orchestra.run.smithi028.stdout: "degraded_stretch_mode": 0, 2024-02-05T06:35:36.939 INFO:teuthology.orchestra.run.smithi028.stdout: "recovering_stretch_mode": 0, 2024-02-05T06:35:36.939 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_bucket": 0 2024-02-05T06:35:36.939 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:35:36.939 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:35:36.939 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:36.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:36.963 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:36.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:37.064 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:37.088 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:37.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:37.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:37.140 DEBUG:teuthology.orchestra.run.smithi028:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-02-05T06:35:37.143 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-05T06:35:37.143 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-05T06:35:37.174 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:36 smithi028 ceph-mon[37984]: pgmap v65: 148 pgs: 148 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-05T06:35:37.174 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:36 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3253046441' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T06:35:37.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:36 smithi028 ceph-mon[29442]: pgmap v65: 148 pgs: 148 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-05T06:35:37.174 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:36 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3253046441' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T06:35:37.191 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:37.194 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:hints = 1 2024-02-05T06:35:37.194 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-05T06:35:37.194 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Object prefix: benchmark_data_smithi028_77509 2024-02-05T06:35:37.194 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T06:35:37.194 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 0 0 0 0 0 0 - 0 2024-02-05T06:35:37.228 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:36 smithi167 ceph-mon[32774]: pgmap v65: 148 pgs: 148 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-05T06:35:37.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:36 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3253046441' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T06:35:37.237 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:37.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:37.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:37.340 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:35:37.371 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:37.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:37.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:35:37.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:37.512 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:37.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:37.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:37.614 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:35:37.642 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:37.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:37.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:35:37.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:37.786 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:37.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:37.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:37.888 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:37.930 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:37.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:37.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:38.032 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:38.061 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:38.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:38.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:38.163 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:38.193 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:38.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:38.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:38.194 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1 16 3249 3233 202.052 202.062 0.00629049 0.00492836 2024-02-05T06:35:38.194 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:37 smithi028 ceph-mon[29442]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T06:35:38.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:37 smithi028 ceph-mon[37984]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T06:35:38.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:37 smithi167 ceph-mon[32774]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T06:35:38.295 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:38.325 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:38.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:38.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:38.393 INFO:tasks.thrashosds.thrasher:j is {'epoch': 101, 'fsid': '9801e956-c3ef-11ee-95b6-87774f69a715', 'created': '2024-02-05T06:27:59.919226+0000', 'modified': '2024-02-05T06:35:34.918304+0000', 'last_up_change': '2024-02-05T06:33:15.833573+0000', 'last_in_change': '2024-02-05T06:35:23.906253+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-05T06:31:10.441424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-02-05T06:33:27.397416+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-02-05T06:33:28.090506+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-05T06:33:29.866249+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.490000009536743, 'score_stable': 2.490000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-05T06:33:31.826006+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-05T06:33:33.890835+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-02-05T06:35:26.183571+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'c19fdd5b-09f6-45d4-a519-0a938cf1f2a6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6801', 'nonce': 301279106}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6802', 'nonce': 301279106}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6804', 'nonce': 301279106}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6803', 'nonce': 301279106}]}, 'public_addr': '172.21.15.28:6801/301279106', 'cluster_addr': '172.21.15.28:6802/301279106', 'heartbeat_back_addr': '172.21.15.28:6804/301279106', 'heartbeat_front_addr': '172.21.15.28:6803/301279106', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '52245f6e-eb96-4d0d-985b-dbb6fd3ce8de', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6805', 'nonce': 1795224649}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6806', 'nonce': 1795224649}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6808', 'nonce': 1795224649}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6807', 'nonce': 1795224649}]}, 'public_addr': '172.21.15.28:6805/1795224649', 'cluster_addr': '172.21.15.28:6806/1795224649', 'heartbeat_back_addr': '172.21.15.28:6808/1795224649', 'heartbeat_front_addr': '172.21.15.28:6807/1795224649', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '1e18b630-afdf-4de9-a1a2-37bdfdd01c67', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6809', 'nonce': 1993280018}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6810', 'nonce': 1993280018}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6812', 'nonce': 1993280018}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6811', 'nonce': 1993280018}]}, 'public_addr': '172.21.15.28:6809/1993280018', 'cluster_addr': '172.21.15.28:6810/1993280018', 'heartbeat_back_addr': '172.21.15.28:6812/1993280018', 'heartbeat_front_addr': '172.21.15.28:6811/1993280018', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '3ae795af-3d59-41a6-95eb-b8447ff61282', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6813', 'nonce': 1294621398}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6814', 'nonce': 1294621398}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6816', 'nonce': 1294621398}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6815', 'nonce': 1294621398}]}, 'public_addr': '172.21.15.28:6813/1294621398', 'cluster_addr': '172.21.15.28:6814/1294621398', 'heartbeat_back_addr': '172.21.15.28:6816/1294621398', 'heartbeat_front_addr': '172.21.15.28:6815/1294621398', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'fc6bd83d-570c-48c0-964d-e09b4608bc32', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 77, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6800', 'nonce': 4052090595}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6801', 'nonce': 4052090595}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6803', 'nonce': 4052090595}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6802', 'nonce': 4052090595}]}, 'public_addr': '172.21.15.167:6800/4052090595', 'cluster_addr': '172.21.15.167:6801/4052090595', 'heartbeat_back_addr': '172.21.15.167:6803/4052090595', 'heartbeat_front_addr': '172.21.15.167:6802/4052090595', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '895bf465-2531-4d9f-bc23-1e75fd969f07', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6804', 'nonce': 1631080500}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6805', 'nonce': 1631080500}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6807', 'nonce': 1631080500}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6806', 'nonce': 1631080500}]}, 'public_addr': '172.21.15.167:6804/1631080500', 'cluster_addr': '172.21.15.167:6805/1631080500', 'heartbeat_back_addr': '172.21.15.167:6807/1631080500', 'heartbeat_front_addr': '172.21.15.167:6806/1631080500', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ce0a9300-87a4-4e37-b702-d667f168c0e0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6808', 'nonce': 1644050617}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6809', 'nonce': 1644050617}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6811', 'nonce': 1644050617}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6810', 'nonce': 1644050617}]}, 'public_addr': '172.21.15.167:6808/1644050617', 'cluster_addr': '172.21.15.167:6809/1644050617', 'heartbeat_back_addr': '172.21.15.167:6811/1644050617', 'heartbeat_front_addr': '172.21.15.167:6810/1644050617', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '0c545cbc-0873-4232-8807-4af9299ef95e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6812', 'nonce': 4031475764}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6813', 'nonce': 4031475764}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6815', 'nonce': 4031475764}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6814', 'nonce': 4031475764}]}, 'public_addr': '172.21.15.167:6812/4031475764', 'cluster_addr': '172.21.15.167:6813/4031475764', 'heartbeat_back_addr': '172.21.15.167:6815/4031475764', 'heartbeat_front_addr': '172.21.15.167:6814/4031475764', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:35:02.377925+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:35:32.713008+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:34:44.457603+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:34:16.357616+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-05T06:34:41.009118+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:34:41.266888+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:35:30.750818+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:34:35.017723+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.e', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.28:6800/1001442173': '2024-02-06T06:34:10.127914+0000', '172.21.15.28:0/3160026720': '2024-02-06T06:34:10.127914+0000', '172.21.15.28:0/558171292': '2024-02-06T06:34:10.127914+0000', '172.21.15.28:0/2832437766': '2024-02-06T06:34:10.127914+0000', '172.21.15.28:6800/4125764599': '2024-02-06T06:28:54.105170+0000', '172.21.15.28:0/1081543653': '2024-02-06T06:28:54.105170+0000', '172.21.15.28:0/3794420554': '2024-02-06T06:28:27.553034+0000', '172.21.15.28:0/1560471317': '2024-02-06T06:34:10.127914+0000', '172.21.15.28:6800/4266601437': '2024-02-06T06:28:27.553034+0000', '172.21.15.28:0/388893831': '2024-02-06T06:28:27.553034+0000', '172.21.15.28:0/460817910': '2024-02-06T06:28:27.553034+0000', '172.21.15.28:0/2728305743': '2024-02-06T06:28:54.105170+0000', '172.21.15.28:0/3149394045': '2024-02-06T06:34:10.127914+0000', '172.21.15.28:0/3605013594': '2024-02-06T06:28:54.105170+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-05T06:35:38.393 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 4.e 2024-02-05T06:35:38.394 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd rm-pg-upmap-items 4.e 2024-02-05T06:35:38.427 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:38.433 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:35:38.482 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:38.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:38.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:38.583 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:35:38.612 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:38.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:38.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:35:38.714 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:38.743 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:38.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:38.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:38.845 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:38.868 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:38.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:38.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:38.970 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:39.011 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:39.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:39.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:39.113 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:39.143 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:39.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:39.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:39.194 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2 16 5987 5971 186.579 171.125 0.00466426 0.00535028 2024-02-05T06:35:39.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:38 smithi167 ceph-mon[32774]: pgmap v66: 148 pgs: 148 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-05T06:35:39.231 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:38 smithi028 ceph-mon[29442]: pgmap v66: 148 pgs: 148 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-05T06:35:39.232 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:38 smithi028 ceph-mon[37984]: pgmap v66: 148 pgs: 148 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-05T06:35:39.244 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:39.273 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:39.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:39.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:39.376 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:39.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:39.422 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:39.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:39.523 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:35:39.552 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:39.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:39.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:35:39.654 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:35:39.687 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:39.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:39.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:35:39.788 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:39.829 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:39.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:39.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:39.930 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:39.960 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:39.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:39.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:35:40.062 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:35:40.093 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:40.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:40.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:35:40.194 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3 16 8839 8823 183.777 178.25 0.00634596 0.00543264 2024-02-05T06:35:40.195 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:40.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:40.225 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:40.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:40.326 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:40.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:40.372 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:40.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:40.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:40.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:40.516 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:40.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:40.617 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:40.648 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:40.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:40.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:40.751 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:40.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:40.792 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:40.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:40.893 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:35:40.923 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:40.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:40.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:35:41.025 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:41.055 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:41.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:41.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:41.157 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:41.186 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:41.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:41.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:41.194 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4 16 11394 11378 177.751 159.688 0.0104922 0.00561945 2024-02-05T06:35:41.288 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:41.316 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:41.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:41.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:41.418 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:41.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:41 smithi028 ceph-mon[29442]: pgmap v67: 148 pgs: 148 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:35:41.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:41 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:41.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:41 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:35:41.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:41 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:41.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:41 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:41.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:41 smithi028 ceph-mon[37984]: pgmap v67: 148 pgs: 148 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:35:41.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:41 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:41.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:41 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:35:41.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:41 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:41.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:41 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:41.487 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:41.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:41.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:41.589 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:41.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:41.641 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:41.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:41.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:41 smithi167 ceph-mon[32774]: pgmap v67: 148 pgs: 148 active+clean; 582 KiB data, 212 MiB used, 626 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:35:41.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:41 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:41.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:41 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:35:41.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:41 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:41.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:41 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:41.742 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:41.788 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:41.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:41.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:41.890 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:41.929 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:41.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:41.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:42.031 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:42.061 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:42.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:42.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:42.163 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:42.190 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:42.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:42.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:42.195 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5 16 13775 13759 171.917 148.812 0.0138707 0.00580801 2024-02-05T06:35:42.292 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:42.319 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:42.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:42.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:42.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:42.449 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:35:42.475 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:42.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:42.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:42.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:42.587 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:35:42 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:35:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:35:42.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:42.622 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:42.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:42.723 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:42.729 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:35:42 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:35:42.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:42.790 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:42.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:42.891 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:42.935 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:42.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:42.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:43.037 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:43.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:43.065 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:43.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:43.166 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:43.196 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 6 16 15698 15682 163.296 120.188 0.00660923 0.00611598 2024-02-05T06:35:43.214 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:43.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:43.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:43.316 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:43.374 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:43.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:43.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:43.392 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-02-05T06:35:43.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:43 smithi028 ceph-mon[29442]: pgmap v68: 148 pgs: 148 active+clean; 705 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-02-05T06:35:43.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:43 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:35:43.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:43 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:35:43.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:43 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:43.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:43 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:35:43.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:43 smithi028 ceph-mon[29442]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-05T06:35:43.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:43 smithi028 ceph-mon[29442]: Reconfiguring daemon alertmanager.a on smithi028 2024-02-05T06:35:43.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:43 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:35:43.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:43 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:35:43.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:43 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/416449085' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:35:43.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:43 smithi028 ceph-mon[29442]: from='client.24757 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:35:43.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:43 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:35:43.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:43 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:35:43.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:43 smithi028 ceph-mon[37984]: pgmap v68: 148 pgs: 148 active+clean; 705 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-02-05T06:35:43.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:43 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:35:43.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:43 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:35:43.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:43 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:43.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:43 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:35:43.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:43 smithi028 ceph-mon[37984]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-05T06:35:43.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:43 smithi028 ceph-mon[37984]: Reconfiguring daemon alertmanager.a on smithi028 2024-02-05T06:35:43.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:43 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:35:43.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:43 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:35:43.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:43 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/416449085' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:35:43.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:43 smithi028 ceph-mon[37984]: from='client.24757 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:35:43.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:43 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:35:43.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:43 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:35:43.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:43.521 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:43.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:43.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:43.622 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:43.652 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:43.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:43.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:43.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:43 smithi167 ceph-mon[32774]: pgmap v68: 148 pgs: 148 active+clean; 705 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-02-05T06:35:43.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:43 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:35:43.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:43 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:35:43.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:43 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:35:43.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:43 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:35:43.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:43 smithi167 ceph-mon[32774]: Reconfiguring alertmanager.a (dependencies changed)... 2024-02-05T06:35:43.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:43 smithi167 ceph-mon[32774]: Reconfiguring daemon alertmanager.a on smithi028 2024-02-05T06:35:43.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:43 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:35:43.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:43 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:35:43.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:43 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/416449085' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:35:43.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:43 smithi167 ceph-mon[32774]: from='client.24757 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:35:43.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:43 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:35:43.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:43 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:35:43.754 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:43.784 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:43.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:43.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:43.886 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:43.931 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:43.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:43.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:44.033 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:44.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:44.064 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:44.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:44.165 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:44.195 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:44.196 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 7 16 18088 18072 161.305 149.375 0.0056572 0.006193 2024-02-05T06:35:44.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:44.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:44.298 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:44.338 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:44.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:44.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:44.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:44.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:44.478 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:44.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:44.579 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:44.616 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:44.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:44.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:44.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:44 smithi028 ceph-mon[29442]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:35:44.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:44 smithi028 ceph-mon[29442]: from='client.24757 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:35:44.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:44 smithi028 ceph-mon[29442]: osdmap e102: 8 total, 8 up, 7 in 2024-02-05T06:35:44.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:44 smithi028 ceph-mon[37984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:35:44.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:44 smithi028 ceph-mon[37984]: from='client.24757 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:35:44.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:44 smithi028 ceph-mon[37984]: osdmap e102: 8 total, 8 up, 7 in 2024-02-05T06:35:44.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:44.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:44 smithi167 ceph-mon[32774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:35:44.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:44 smithi167 ceph-mon[32774]: from='client.24757 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:35:44.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:44 smithi167 ceph-mon[32774]: osdmap e102: 8 total, 8 up, 7 in 2024-02-05T06:35:44.759 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:44.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:44.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:44.861 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:44.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:44.892 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:44.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:44.993 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:45.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:45.024 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:45.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:45.125 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:45.164 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:45.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:45.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:45.196 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 8 16 20434 20418 159.469 146.625 0.0104343 0.00626483 2024-02-05T06:35:45.267 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:35:45.298 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:45.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:45.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:35:45.401 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:45.440 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:45.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:45.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:45.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:45 smithi167 ceph-mon[32774]: pgmap v70: 148 pgs: 148 active+clean; 705 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-02-05T06:35:45.541 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:45.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:45.585 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:45.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:45.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:45 smithi028 ceph-mon[29442]: pgmap v70: 148 pgs: 148 active+clean; 705 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-02-05T06:35:45.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:45 smithi028 ceph-mon[37984]: pgmap v70: 148 pgs: 148 active+clean; 705 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-02-05T06:35:45.686 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:45.715 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:45.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:45.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:45.816 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:45.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:45.859 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:45.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:45.960 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:45.989 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:45.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:45.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:46.091 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:46.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:46.133 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:46.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:46.196 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 9 16 22442 22426 155.694 125.5 0.00465922 0.00639762 2024-02-05T06:35:46.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:46.267 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:46.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:46.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:46.370 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:46.400 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:46.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:46.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:46.502 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:46.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:46.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:46.644 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:46.673 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:46.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:46.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:46.775 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:46.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:46.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:46.917 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:46.945 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:46.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:46.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:47.047 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:47.087 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:47.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:47.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:47.189 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:47.198 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 10 15 24512 24497 153.042 129.438 0.00416038 0.00652843 2024-02-05T06:35:47.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:47.226 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:47.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:47.327 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:47.368 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:47.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:47.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:47.470 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:47.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:47.496 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:47.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:47.597 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:47.628 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:47.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:47.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:47.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:47 smithi028 ceph-mon[37984]: pgmap v71: 148 pgs: 148 active+clean; 1.4 GiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 140 MiB/s wr, 2.24k op/s 2024-02-05T06:35:47.664 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:47 smithi028 ceph-mon[29442]: pgmap v71: 148 pgs: 148 active+clean; 1.4 GiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 140 MiB/s wr, 2.24k op/s 2024-02-05T06:35:47.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:47 smithi167 ceph-mon[32774]: pgmap v71: 148 pgs: 148 active+clean; 1.4 GiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 140 MiB/s wr, 2.24k op/s 2024-02-05T06:35:47.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:47.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:47.758 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:47.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:47.859 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:47.889 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:47.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:47.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:47.991 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:35:48.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:48.016 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:48.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:35:48.117 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:48.154 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:48.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:48.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:48.198 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 11 16 25855 25839 146.755 83.875 0.00213825 0.00678012 2024-02-05T06:35:48.255 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:48.286 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:48.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:48.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:48.388 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:48.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:48.430 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:48.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:48.531 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:48.554 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:48.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:48.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:48.655 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:48.695 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:48.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:48.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:48.797 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:48.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:48.821 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:48.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:48.921 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:48.951 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:48.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:48.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:49.053 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:49.083 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:49.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:49.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:49.184 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:49.198 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 12 16 27356 27340 142.342 93.8125 0.00967284 0.0070189 2024-02-05T06:35:49.228 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:49.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:49.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:49.330 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:49.369 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:49.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:49.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:49.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:49.503 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:49.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:49.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:49.605 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:49.634 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:49.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:49.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:49.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:49 smithi028 ceph-mon[29442]: pgmap v72: 148 pgs: 148 active+clean; 1.4 GiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 140 MiB/s wr, 2.24k op/s 2024-02-05T06:35:49.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:49 smithi028 ceph-mon[37984]: pgmap v72: 148 pgs: 148 active+clean; 1.4 GiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 140 MiB/s wr, 2.24k op/s 2024-02-05T06:35:49.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:49 smithi167 ceph-mon[32774]: pgmap v72: 148 pgs: 148 active+clean; 1.4 GiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 140 MiB/s wr, 2.24k op/s 2024-02-05T06:35:49.735 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:49.754 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:35:49.755 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:35:49.765 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:49.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:49.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:49.867 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:49.892 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:49.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:49.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:49.994 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:50.023 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:50.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:50.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:50.125 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:50.166 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:50.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:50.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:50.198 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 13 15 29207 29192 140.296 115.75 0.00998593 0.00712085 2024-02-05T06:35:50.268 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:50.311 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:50.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:50.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:50.413 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:50.449 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:50.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:50.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:50.551 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:50.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:50.590 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:50.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:50.690 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:50.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:50.732 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:50.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:50.833 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:50.862 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:50.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:50.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:50.964 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:51.004 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:51.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:51.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:51.106 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:51.136 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:51.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:51.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:51.152 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:50 smithi028 ceph-mon[29442]: pgmap v73: 148 pgs: 148 active+clean; 1.4 GiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 140 MiB/s wr, 2.24k op/s 2024-02-05T06:35:51.152 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:50 smithi028 ceph-mon[37984]: pgmap v73: 148 pgs: 148 active+clean; 1.4 GiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 140 MiB/s wr, 2.24k op/s 2024-02-05T06:35:51.198 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 14 16 30763 30747 137.217 97.1875 0.0105309 0.00728152 2024-02-05T06:35:51.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:50 smithi167 ceph-mon[32774]: pgmap v73: 148 pgs: 148 active+clean; 1.4 GiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 140 MiB/s wr, 2.24k op/s 2024-02-05T06:35:51.237 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:51.295 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:51.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:51.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:51.397 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:51.426 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:51.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:51.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:51.528 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:51.569 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:51.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:51.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:51.671 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:51.699 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:51.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:51.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:51.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:51.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:51.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:51.903 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-05T06:35:51.941 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:51.994 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:51.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:51.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:52.095 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:52.121 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:52.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:52.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:52.200 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 15 16 32356 32340 134.695 99.5625 0.0108226 0.00741761 2024-02-05T06:35:52.224 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:52.250 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:52.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:52.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:52.351 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:52.380 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:52.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:52.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:52.482 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:35:52.509 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:52.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:52.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:35:52.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:52.655 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:52.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:52.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:52.729 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:35:52 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:35:52.756 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:52.786 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:52.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:52.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:52.853 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:35:52 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:35:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:35:52.887 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:52.915 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:52.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:52.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:53.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:53.057 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:53.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:53.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:53.093 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:35:53.137 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:35:53.159 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:53.194 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:35:53.199 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:53.200 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 16 16 33990 33974 132.66 102.125 0.00416103 0.00753031 2024-02-05T06:35:53.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:53.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:53.257 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:35:53.260 INFO:teuthology.orchestra.run.smithi028.stderr:clear 4.e pg_upmap_items mapping 2024-02-05T06:35:53.301 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:35:53.301 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:53.344 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:35:53.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:53.354 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:53.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:53.389 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:35:53.389 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:53 smithi028 ceph-mon[37984]: pgmap v74: 148 pgs: 148 active+clean; 1.9 GiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 1022 B/s rd, 121 MiB/s wr, 1.94k op/s 2024-02-05T06:35:53.389 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:53 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:35:53.389 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:53 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/130373027' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:35:53.389 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:53 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/905238783' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.e"}]: dispatch 2024-02-05T06:35:53.389 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:53 smithi028 ceph-mon[37984]: from='client.24763 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.e"}]: dispatch 2024-02-05T06:35:53.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:53 smithi028 ceph-mon[29442]: pgmap v74: 148 pgs: 148 active+clean; 1.9 GiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 1022 B/s rd, 121 MiB/s wr, 1.94k op/s 2024-02-05T06:35:53.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:53 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:35:53.390 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:53 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/130373027' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:35:53.390 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:53 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/905238783' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.e"}]: dispatch 2024-02-05T06:35:53.390 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:53 smithi028 ceph-mon[29442]: from='client.24763 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.e"}]: dispatch 2024-02-05T06:35:53.432 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:35:53.442 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:35:53.443 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:35:53.443 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:35:53.443 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:35:53.443 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:35:53.443 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:35:53.443 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:35:53.443 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:35:53.455 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:53.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T06:35:53.505 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:53.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:53.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:53.607 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:53.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:53.653 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:53.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:53.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:53 smithi167 ceph-mon[32774]: pgmap v74: 148 pgs: 148 active+clean; 1.9 GiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 1022 B/s rd, 121 MiB/s wr, 1.94k op/s 2024-02-05T06:35:53.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:53 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:35:53.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:53 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/130373027' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:35:53.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:53 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/905238783' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.e"}]: dispatch 2024-02-05T06:35:53.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:53 smithi167 ceph-mon[32774]: from='client.24763 ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.e"}]: dispatch 2024-02-05T06:35:53.754 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:53.794 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:53.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:53.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:53.898 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:35:53.928 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:53.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:53.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:35:54.030 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:54.059 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:54.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:54.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:54.161 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:54.200 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 17 16 35658 35642 130.988 104.25 0.00998177 0.00762895 2024-02-05T06:35:54.202 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:54.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:54.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:54.304 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:54.331 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:54.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:54.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:54.433 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:54.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:54.463 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:54.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:54.558 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:54 smithi028 ceph-mon[37984]: from='client.24763 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.e"}]': finished 2024-02-05T06:35:54.558 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:54 smithi028 ceph-mon[37984]: osdmap e103: 8 total, 8 up, 7 in 2024-02-05T06:35:54.558 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:54 smithi028 ceph-mon[29442]: from='client.24763 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.e"}]': finished 2024-02-05T06:35:54.558 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:54 smithi028 ceph-mon[29442]: osdmap e103: 8 total, 8 up, 7 in 2024-02-05T06:35:54.564 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:54.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:54.612 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:54.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:54.713 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:54.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:54 smithi167 ceph-mon[32774]: from='client.24763 ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.e"}]': finished 2024-02-05T06:35:54.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:54 smithi167 ceph-mon[32774]: osdmap e103: 8 total, 8 up, 7 in 2024-02-05T06:35:54.755 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:54.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:54.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:54.857 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:54.886 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:54.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:54.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:54.988 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:55.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:55.027 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:55.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:55.128 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:55.159 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:55.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:55.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:55.200 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 18 16 37434 37418 129.878 111 0.0176286 0.00769356 2024-02-05T06:35:55.261 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:55.290 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:55.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:55.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:55.392 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:55.421 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:55.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:55.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:55.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:55.553 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:55.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:55.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:55.655 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:55.683 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:55.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:55.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:55.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:55 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:35:55.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:55 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:35:55.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:55 smithi028 ceph-mon[37984]: pgmap v76: 148 pgs: 148 active+clean; 1.9 GiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 1022 B/s rd, 121 MiB/s wr, 1.94k op/s 2024-02-05T06:35:55.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:55 smithi028 ceph-mon[37984]: osdmap e104: 8 total, 8 up, 7 in 2024-02-05T06:35:55.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:55 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:35:55.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:55 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:35:55.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:55 smithi028 ceph-mon[29442]: pgmap v76: 148 pgs: 148 active+clean; 1.9 GiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 1022 B/s rd, 121 MiB/s wr, 1.94k op/s 2024-02-05T06:35:55.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:55 smithi028 ceph-mon[29442]: osdmap e104: 8 total, 8 up, 7 in 2024-02-05T06:35:55.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:55 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:35:55.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:55 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:35:55.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:55 smithi167 ceph-mon[32774]: pgmap v76: 148 pgs: 148 active+clean; 1.9 GiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 1022 B/s rd, 121 MiB/s wr, 1.94k op/s 2024-02-05T06:35:55.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:55 smithi167 ceph-mon[32774]: osdmap e104: 8 total, 8 up, 7 in 2024-02-05T06:35:55.785 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:55.811 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:55.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:55.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:55.913 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:55.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:55.939 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:55.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:56.040 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:56.079 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:56.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:56.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:56.180 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:56.200 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 19 16 39043 39027 128.335 100.562 0.00855857 0.00778748 2024-02-05T06:35:56.209 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:56.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:56.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:56.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:35:56.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:56.352 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:56.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:35:56.453 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:56.481 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:56.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:56.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:56.583 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:56.612 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:56.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:56.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:56.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:56 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:35:56.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:56 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:35:56.714 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:35:56.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:56 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:35:56.753 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:56.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:56.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:35:56.855 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:56.895 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:56.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:56.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:56.997 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:57.036 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:57.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:57.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:57.138 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:57.165 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:57.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:57.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:57.194 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:35:56 smithi028 systemd[1]: Stopping Ceph alertmanager.a for 9801e956-c3ef-11ee-95b6-87774f69a715... 2024-02-05T06:35:57.200 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-02-05T06:35:57.198856+0000 min lat: 0.0013072 max lat: 0.142299 avg lat: 0.00785131 2024-02-05T06:35:57.201 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T06:35:57.201 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 20 16 40756 40740 127.271 107.062 0.00430336 0.00785131 2024-02-05T06:35:57.240 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:35:57.244 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:35:57.266 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:57.294 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:57.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:57.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:57.395 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:57.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:57.425 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:57.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:57.526 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:57.535 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:57 smithi028 ceph-mon[29442]: pgmap v78: 148 pgs: 148 active+clean; 2.4 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 129 MiB/s wr, 2.07k op/s; 13 B/s, 0 objects/s recovering 2024-02-05T06:35:57.536 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:57 smithi028 ceph-mon[37984]: pgmap v78: 148 pgs: 148 active+clean; 2.4 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 129 MiB/s wr, 2.07k op/s; 13 B/s, 0 objects/s recovering 2024-02-05T06:35:57.553 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:57.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:57.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:57.655 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:57.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:57.683 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:57.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:57.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:57 smithi167 ceph-mon[32774]: pgmap v78: 148 pgs: 148 active+clean; 2.4 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 129 MiB/s wr, 2.07k op/s; 13 B/s, 0 objects/s recovering 2024-02-05T06:35:57.784 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:57.817 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:57.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:57.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:57.919 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:57.961 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:57.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:57.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:58.063 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:58.092 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:58.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:58.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:58.106 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:35:57 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a[72676]: ts=2024-02-05T06:35:57.907Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-02-05T06:35:58.195 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:58.200 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 21 16 42359 42343 125.979 100.188 0.00930681 0.00793156 2024-02-05T06:35:58.237 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:58.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:58.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:58.339 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:35:58.367 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:58.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:58.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:35:58.469 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:58.497 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:58.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:58.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:58.599 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:58.622 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:58.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:58.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:58.724 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:58.764 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:58.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:58.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:58.866 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:58.890 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:58.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:58.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:58.992 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:35:59.021 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:59.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:59.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:35:59.049 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T06:35:59.049 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T06:35:59.049 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-05T06:35:59.049 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-05T06:35:59.124 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:35:59.153 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:59.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:59.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:35:59.201 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 22 16 43734 43718 124.159 85.9375 0.011021 0.00804776 2024-02-05T06:35:59.254 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:59.291 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:35:59.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:59.304 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:59.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:59.344 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:35:59 smithi028 ceph-mon[29442]: pgmap v79: 148 pgs: 148 active+clean; 2.4 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 129 MiB/s wr, 2.07k op/s; 13 B/s, 0 objects/s recovering 2024-02-05T06:35:59.344 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:35:59 smithi028 ceph-mon[37984]: pgmap v79: 148 pgs: 148 active+clean; 2.4 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 129 MiB/s wr, 2.07k op/s; 13 B/s, 0 objects/s recovering 2024-02-05T06:35:59.405 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:35:59.432 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:59.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:59.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:35:59.534 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:35:59.588 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:59.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:59.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:35:59.691 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:35:59.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:35:59 smithi167 ceph-mon[32774]: pgmap v79: 148 pgs: 148 active+clean; 2.4 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 129 MiB/s wr, 2.07k op/s; 13 B/s, 0 objects/s recovering 2024-02-05T06:35:59.736 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:35:59.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:59.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:35:59.838 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:59.867 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:59.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:59.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:35:59.969 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:35:59.998 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:35:59.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:35:59.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:00.100 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:00.128 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:00.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:00.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:00.201 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 23 16 45183 45167 122.698 90.5625 0.00815465 0.0081451 2024-02-05T06:36:00.230 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:00.258 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:00.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:00.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:00.360 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:00.388 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:00.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:00.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:00.490 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:00.518 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:00.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:00.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:00.594 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:36:00 smithi028 podman[79725]: 2024-02-05 06:36:00.289118102 +0000 UTC m=+2.731160991 container died 74971e2a5fe0bc7301558a3dae1627f6798275f02305ce3d0fa2e73270d2ab10 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T06:36:00.621 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:00.648 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:00.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:00.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:00.750 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:00.778 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:00.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:00.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:00.880 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:00.908 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:00.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:00.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:00.918 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:36:00 smithi028 podman[79725]: 2024-02-05 06:36:00.773100495 +0000 UTC m=+3.215143381 container cleanup 74971e2a5fe0bc7301558a3dae1627f6798275f02305ce3d0fa2e73270d2ab10 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T06:36:00.919 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:36:00 smithi028 bash[79725]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a 2024-02-05T06:36:01.010 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:01.038 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:01.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:01.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:01.140 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:01.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:01.186 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:01.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:01.189 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:00 smithi028 ceph-mon[29442]: pgmap v80: 148 pgs: 148 active+clean; 2.4 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 65 MiB/s wr, 1.03k op/s; 13 B/s, 0 objects/s recovering 2024-02-05T06:36:01.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:00 smithi028 ceph-mon[37984]: pgmap v80: 148 pgs: 148 active+clean; 2.4 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 65 MiB/s wr, 1.03k op/s; 13 B/s, 0 objects/s recovering 2024-02-05T06:36:01.189 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:36:00 smithi028 podman[79756]: 2024-02-05 06:36:00.975271306 +0000 UTC m=+3.019624267 container remove 74971e2a5fe0bc7301558a3dae1627f6798275f02305ce3d0fa2e73270d2ab10 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T06:36:01.190 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:36:01 smithi028 podman[79756]: 2024-02-05 06:36:01.067795532 +0000 UTC m=+3.112148496 volume remove 9fd603b0d2390056e86a7366ec69b895bc0888d085fa641345ee9b3f1395825d 2024-02-05T06:36:01.202 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 24 16 46776 46760 121.732 99.5625 0.00993438 0.00820766 2024-02-05T06:36:01.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:00 smithi167 ceph-mon[32774]: pgmap v80: 148 pgs: 148 active+clean; 2.4 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 65 MiB/s wr, 1.03k op/s; 13 B/s, 0 objects/s recovering 2024-02-05T06:36:01.287 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:01.329 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:01.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:01.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:01.431 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:01.474 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:01.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:01.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:01.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:01.625 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:01.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:01.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:01.727 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:01.755 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:01.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:01.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:01.856 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:01.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:01.899 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:01.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:02.000 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:02.028 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:02.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:02.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:02.129 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:02.166 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:02.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:02.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:02.201 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 25 16 48206 48190 120.437 89.375 0.00885301 0.00829789 2024-02-05T06:36:02.268 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:02.308 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:02.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:02.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:02.410 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:02.453 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:02.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:02.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:02.555 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:02.561 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:36:02 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:36:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:36:02.561 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:36:02 smithi028 systemd[1]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-05T06:36:02.561 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:36:02 smithi028 systemd[1]: Stopped Ceph alertmanager.a for 9801e956-c3ef-11ee-95b6-87774f69a715. 2024-02-05T06:36:02.561 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:36:02 smithi028 systemd[1]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715@alertmanager.a.service: Consumed 1.195s CPU time. 2024-02-05T06:36:02.561 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:36:02 smithi028 systemd[1]: Starting Ceph alertmanager.a for 9801e956-c3ef-11ee-95b6-87774f69a715... 2024-02-05T06:36:02.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:02.605 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:02.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:02.680 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:36:02.706 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:02.729 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:36:02 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:36:02.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:02.761 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:02.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:02.787 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:36:02.787 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-02-05T06:36:02.787 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:02.787 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14847.0:48946 7.2 e105/96 7:467483fd:::benchmark_data_smithi028_77509_object48945:head v 105'3076, mlcod=105'3074)", 2024-02-05T06:36:02.787 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:36:02.778824+0000", 2024-02-05T06:36:02.787 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0051289680000000002, 2024-02-05T06:36:02.788 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051623529999999997, 2024-02-05T06:36:02.788 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:36:02.788 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-02-05T06:36:02.788 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:36:02.788 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:02.788 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:36:02.788 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:02.778824+0000", 2024-02-05T06:36:02.788 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:02.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:02.788 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:02.788 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:36:02.788 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:02.778824+0000", 2024-02-05T06:36:02.788 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:02.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:02.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:02.789 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:36:02.789 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:02.778825+0000", 2024-02-05T06:36:02.789 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8800000000000002e-07 2024-02-05T06:36:02.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:02.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:02.789 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:36:02.789 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:02.778864+0000", 2024-02-05T06:36:02.789 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9557000000000001e-05 2024-02-05T06:36:02.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:02.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:02.789 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:36:02.789 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:02.778872+0000", 2024-02-05T06:36:02.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6079999999999998e-06 2024-02-05T06:36:02.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:02.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:02.790 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:36:02.790 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:02.778878+0000", 2024-02-05T06:36:02.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0070000000000003e-06 2024-02-05T06:36:02.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:02.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:02.790 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:36:02.790 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:02.778934+0000", 2024-02-05T06:36:02.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6467e-05 2024-02-05T06:36:02.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:02.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:02.790 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:36:02.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:02.778952+0000", 2024-02-05T06:36:02.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7476e-05 2024-02-05T06:36:02.791 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:02.791 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:02.791 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:02.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:02.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:02.791 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14847.0:48953 7.2 e105/96 7:461e4539:::benchmark_data_smithi028_77509_object48952:head v 105'3077, mlcod=105'3074)", 2024-02-05T06:36:02.791 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:36:02.780200+0000", 2024-02-05T06:36:02.791 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0037525840000000002, 2024-02-05T06:36:02.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003854049, 2024-02-05T06:36:02.791 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:36:02.791 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-02-05T06:36:02.791 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:36:02.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:02.792 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:36:02.792 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:02.780200+0000", 2024-02-05T06:36:02.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:02.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:02.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:02.792 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:36:02.792 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:02.780200+0000", 2024-02-05T06:36:02.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:02.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:02.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:02.792 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:36:02.792 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:02.780202+0000", 2024-02-05T06:36:02.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.218e-06 2024-02-05T06:36:02.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:02.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:02.793 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:36:02.793 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:02.780254+0000", 2024-02-05T06:36:02.793 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2061000000000002e-05 2024-02-05T06:36:02.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:02.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:02.793 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:36:02.793 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:02.780286+0000", 2024-02-05T06:36:02.793 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2645000000000003e-05 2024-02-05T06:36:02.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:02.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:02.793 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:36:02.793 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:02.780295+0000", 2024-02-05T06:36:02.794 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8100000000000004e-06 2024-02-05T06:36:02.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:02.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:02.794 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:36:02.794 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:02.780424+0000", 2024-02-05T06:36:02.794 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012873100000000001 2024-02-05T06:36:02.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:02.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:02.794 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:36:02.794 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:02.780448+0000", 2024-02-05T06:36:02.794 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4226000000000001e-05 2024-02-05T06:36:02.794 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:02.794 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:02.794 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:02.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:02.795 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:02.795 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:48952 7.b 7.42e8dcdb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-02-05T06:36:02.795 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:36:02.784028+0000", 2024-02-05T06:36:02.795 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4294967295.9999247, 2024-02-05T06:36:02.795 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8819000000000005e-05, 2024-02-05T06:36:02.795 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:36:02.795 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "queued for pg", 2024-02-05T06:36:02.795 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:36:02.795 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:36:02.795 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:36:02.795 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48952 2024-02-05T06:36:02.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:02.795 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:36:02.795 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:02.796 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:36:02.796 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:02.784028+0000", 2024-02-05T06:36:02.796 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:02.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:02.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:02.796 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:36:02.796 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:02.784028+0000", 2024-02-05T06:36:02.796 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:02.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:02.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:02.796 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:36:02.796 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:02.784031+0000", 2024-02-05T06:36:02.796 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2359999999999999e-06 2024-02-05T06:36:02.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:02.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:02.797 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:36:02.797 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:02.784056+0000", 2024-02-05T06:36:02.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5092e-05 2024-02-05T06:36:02.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:02.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:02.797 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:36:02.797 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:02.784068+0000", 2024-02-05T06:36:02.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2082999999999999e-05 2024-02-05T06:36:02.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:02.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:02.797 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:36:02.797 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:02.784076+0000", 2024-02-05T06:36:02.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5290000000000008e-06 2024-02-05T06:36:02.798 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:02.798 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:02.798 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:02.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:02.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:02.798 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:48957 7.b 7.a558ba7b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-02-05T06:36:02.798 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:36:02.784095+0000", 2024-02-05T06:36:02.798 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4294967295.9998579, 2024-02-05T06:36:02.798 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0222000000000005e-05, 2024-02-05T06:36:02.798 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:36:02.798 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "queued for pg", 2024-02-05T06:36:02.798 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:36:02.798 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:36:02.798 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:36:02.798 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 48957 2024-02-05T06:36:02.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:02.799 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:36:02.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:02.799 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:36:02.799 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:02.784095+0000", 2024-02-05T06:36:02.799 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:02.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:02.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:02.799 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:36:02.799 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:02.784095+0000", 2024-02-05T06:36:02.799 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:02.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:02.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:02.799 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:36:02.799 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:02.784096+0000", 2024-02-05T06:36:02.800 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2200000000000003e-07 2024-02-05T06:36:02.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:02.800 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:02.800 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:36:02.800 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:02.784118+0000", 2024-02-05T06:36:02.800 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2166000000000001e-05 2024-02-05T06:36:02.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:02.800 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:02.800 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:36:02.800 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:02.784127+0000", 2024-02-05T06:36:02.800 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6440000000000006e-06 2024-02-05T06:36:02.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:02.800 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:02.800 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:36:02.801 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:02.784134+0000", 2024-02-05T06:36:02.801 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2370000000000003e-06 2024-02-05T06:36:02.801 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:02.801 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:02.801 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:02.801 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:02.801 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:36:02.801 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 4 2024-02-05T06:36:02.801 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:36:02.863 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:02.911 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:02.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:02.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:03.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:03.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:03.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:03.166 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:03.194 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:03.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:03.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:03.201 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 26 16 49535 49519 119 83.0625 0.0097347 0.0083986 2024-02-05T06:36:03.296 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:03.324 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:03.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:03.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:03.426 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:03.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:03.453 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:03.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:03.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:03 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:36:03.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:03 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:36:03.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:03 smithi167 ceph-mon[32774]: pgmap v81: 148 pgs: 148 active+clean; 2.9 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 112 MiB/s wr, 1.79k op/s; 12 B/s, 0 objects/s recovering 2024-02-05T06:36:03.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:03 smithi167 ceph-mon[32774]: osdmap e105: 8 total, 8 up, 7 in 2024-02-05T06:36:03.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:03 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:36:03.554 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:03.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:03.602 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:03.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:03.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:03 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:36:03.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:03 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:36:03.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:03 smithi028 ceph-mon[37984]: pgmap v81: 148 pgs: 148 active+clean; 2.9 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 112 MiB/s wr, 1.79k op/s; 12 B/s, 0 objects/s recovering 2024-02-05T06:36:03.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:03 smithi028 ceph-mon[37984]: osdmap e105: 8 total, 8 up, 7 in 2024-02-05T06:36:03.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:03 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:36:03.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:03 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:36:03.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:03 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:36:03.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:03 smithi028 ceph-mon[29442]: pgmap v81: 148 pgs: 148 active+clean; 2.9 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 112 MiB/s wr, 1.79k op/s; 12 B/s, 0 objects/s recovering 2024-02-05T06:36:03.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:03 smithi028 ceph-mon[29442]: osdmap e105: 8 total, 8 up, 7 in 2024-02-05T06:36:03.648 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:03 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:36:03.704 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:03.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:03.745 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:03.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:03.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:03.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:03.888 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:03.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:03.943 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:36:03.985 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-05T06:36:03.990 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:04.015 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:04.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:04.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:04.117 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:04.142 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:04.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:04.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:04.202 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 27 15 50652 50637 117.18 69.875 0.0157724 0.00852731 2024-02-05T06:36:04.244 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:04.245 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-02-05T06:36:04.305 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:04.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:04.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:04.407 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:04.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:04.456 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:04.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:04.558 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:04.586 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:04.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:04.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:04.688 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:04.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:04 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/922166159' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:36:04.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:04 smithi028 ceph-mon[37984]: from='client.24775 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:36:04.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:04 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/922166159' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:36:04.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:04 smithi028 ceph-mon[29442]: from='client.24775 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:36:04.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:04 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/922166159' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:36:04.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:04 smithi167 ceph-mon[32774]: from='client.24775 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:36:04.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:04.730 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:04.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:04.831 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:04.862 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:04.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:04.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:04.963 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:05.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:05.001 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:05.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:05.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:05.143 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:05.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:05.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:05.202 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 28 16 51886 51870 115.748 77.0625 0.0129249 0.00863495 2024-02-05T06:36:05.245 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:05.273 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:05.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:05.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:05.333 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:05 smithi028 ceph-mon[29442]: pgmap v83: 148 pgs: 148 active+clean; 2.9 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.0 KiB/s rd, 101 MiB/s wr, 1.62k op/s; 11 B/s, 0 objects/s recovering 2024-02-05T06:36:05.333 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:05 smithi028 ceph-mon[29442]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:36:05.333 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:05 smithi028 ceph-mon[29442]: from='client.24775 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:36:05.333 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:05 smithi028 ceph-mon[29442]: osdmap e106: 8 total, 8 up, 7 in 2024-02-05T06:36:05.334 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:05 smithi028 ceph-mon[37984]: pgmap v83: 148 pgs: 148 active+clean; 2.9 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.0 KiB/s rd, 101 MiB/s wr, 1.62k op/s; 11 B/s, 0 objects/s recovering 2024-02-05T06:36:05.334 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:05 smithi028 ceph-mon[37984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:36:05.334 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:05 smithi028 ceph-mon[37984]: from='client.24775 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:36:05.334 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:05 smithi028 ceph-mon[37984]: osdmap e106: 8 total, 8 up, 7 in 2024-02-05T06:36:05.374 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:05.402 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:05.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:05.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:05.504 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:05.542 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:05.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:05.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:05.644 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:05.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:05.689 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:05.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:05.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:05 smithi167 ceph-mon[32774]: pgmap v83: 148 pgs: 148 active+clean; 2.9 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 1.0 KiB/s rd, 101 MiB/s wr, 1.62k op/s; 11 B/s, 0 objects/s recovering 2024-02-05T06:36:05.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:05 smithi167 ceph-mon[32774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:36:05.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:05 smithi167 ceph-mon[32774]: from='client.24775 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:36:05.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:05 smithi167 ceph-mon[32774]: osdmap e106: 8 total, 8 up, 7 in 2024-02-05T06:36:05.790 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:05.818 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:05.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:05.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:05.919 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:05.958 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:05.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:05.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:06.060 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:06.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:06.084 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:06.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:06.185 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:06.202 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 29 16 53275 53259 114.75 86.8125 0.00938099 0.00870882 2024-02-05T06:36:06.213 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:06.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:06.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:06.315 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:06.344 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:06.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:06.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:06.446 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:06.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:06.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:06.591 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:06.619 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:06.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:06.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:06.721 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:06.751 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:06.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:06.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:06.853 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:06.891 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:06.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:06.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:06.993 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:07.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:07.042 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:07.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:07.143 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:07.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:07.191 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:07.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:07.203 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 30 16 54659 54643 113.808 86.5 0.0222397 0.00878 2024-02-05T06:36:07.262 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:36:07 smithi028 podman[80704]: 2024-02-05 06:36:07.0399305 +0000 UTC m=+0.368007619 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-02-05T06:36:07.292 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:07.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:07.349 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:07.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:07.450 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:07.488 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:07.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:07.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:07.520 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:07 smithi028 ceph-mon[37984]: 6.7 scrub starts 2024-02-05T06:36:07.520 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:07 smithi028 ceph-mon[37984]: 6.7 scrub ok 2024-02-05T06:36:07.520 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:07 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:36:07.520 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:07 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:36:07.520 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:07 smithi028 ceph-mon[37984]: 6.16 scrub starts 2024-02-05T06:36:07.520 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:07 smithi028 ceph-mon[37984]: 6.16 scrub ok 2024-02-05T06:36:07.520 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:07 smithi028 ceph-mon[37984]: pgmap v85: 148 pgs: 148 active+clean; 3.1 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-02-05T06:36:07.520 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:07 smithi028 ceph-mon[37984]: 5.12 scrub starts 2024-02-05T06:36:07.520 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:07 smithi028 ceph-mon[37984]: 5.12 scrub ok 2024-02-05T06:36:07.520 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:07 smithi028 ceph-mon[37984]: osdmap e107: 8 total, 8 up, 7 in 2024-02-05T06:36:07.520 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:07 smithi028 ceph-mon[37984]: 5.10 scrub starts 2024-02-05T06:36:07.521 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:07 smithi028 ceph-mon[37984]: 5.10 scrub ok 2024-02-05T06:36:07.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:07 smithi028 ceph-mon[29442]: 6.7 scrub starts 2024-02-05T06:36:07.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:07 smithi028 ceph-mon[29442]: 6.7 scrub ok 2024-02-05T06:36:07.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:07 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:36:07.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:07 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:36:07.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:07 smithi028 ceph-mon[29442]: 6.16 scrub starts 2024-02-05T06:36:07.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:07 smithi028 ceph-mon[29442]: 6.16 scrub ok 2024-02-05T06:36:07.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:07 smithi028 ceph-mon[29442]: pgmap v85: 148 pgs: 148 active+clean; 3.1 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-02-05T06:36:07.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:07 smithi028 ceph-mon[29442]: 5.12 scrub starts 2024-02-05T06:36:07.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:07 smithi028 ceph-mon[29442]: 5.12 scrub ok 2024-02-05T06:36:07.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:07 smithi028 ceph-mon[29442]: osdmap e107: 8 total, 8 up, 7 in 2024-02-05T06:36:07.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:07 smithi028 ceph-mon[29442]: 5.10 scrub starts 2024-02-05T06:36:07.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:07 smithi028 ceph-mon[29442]: 5.10 scrub ok 2024-02-05T06:36:07.522 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:36:07 smithi028 podman[80704]: 2024-02-05 06:36:07.38157953 +0000 UTC m=+0.709656635 volume create 415ec61802d55f8d9c6de3bc0717252bed01ba231e2a17d0e2605a8be5e27827 2024-02-05T06:36:07.590 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:07.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:07.620 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:07.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:07.722 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:07.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:07 smithi167 ceph-mon[32774]: 6.7 scrub starts 2024-02-05T06:36:07.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:07 smithi167 ceph-mon[32774]: 6.7 scrub ok 2024-02-05T06:36:07.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:07 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:36:07.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:07 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:36:07.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:07 smithi167 ceph-mon[32774]: 6.16 scrub starts 2024-02-05T06:36:07.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:07 smithi167 ceph-mon[32774]: 6.16 scrub ok 2024-02-05T06:36:07.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:07 smithi167 ceph-mon[32774]: pgmap v85: 148 pgs: 148 active+clean; 3.1 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-02-05T06:36:07.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:07 smithi167 ceph-mon[32774]: 5.12 scrub starts 2024-02-05T06:36:07.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:07 smithi167 ceph-mon[32774]: 5.12 scrub ok 2024-02-05T06:36:07.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:07 smithi167 ceph-mon[32774]: osdmap e107: 8 total, 8 up, 7 in 2024-02-05T06:36:07.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:07 smithi167 ceph-mon[32774]: 5.10 scrub starts 2024-02-05T06:36:07.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:07 smithi167 ceph-mon[32774]: 5.10 scrub ok 2024-02-05T06:36:07.747 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:07.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:07.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:07.849 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:07.875 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:07.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:07.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:07.945 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:36:07 smithi028 podman[80704]: 2024-02-05 06:36:07.626461707 +0000 UTC m=+0.954538861 container create 696060c3ff8b7a9d0f55bed2f60178446b4a783b14be24ba5edb33e52b210ede (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T06:36:07.977 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:08.005 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:08.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:08.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:08.107 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:08.122 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:36:08.122 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:36:08.122 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 107, 2024-02-05T06:36:08.123 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "9801e956-c3ef-11ee-95b6-87774f69a715", 2024-02-05T06:36:08.123 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-02-05T06:27:59.919226+0000", 2024-02-05T06:36:08.123 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-02-05T06:36:06.246435+0000", 2024-02-05T06:36:08.123 INFO:teuthology.orchestra.run.smithi028.stdout: "last_up_change": "2024-02-05T06:33:15.833573+0000", 2024-02-05T06:36:08.123 INFO:teuthology.orchestra.run.smithi028.stdout: "last_in_change": "2024-02-05T06:35:23.906253+0000", 2024-02-05T06:36:08.123 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-05T06:36:08.123 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_num": 5804032, 2024-02-05T06:36:08.123 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_set": [ 2024-02-05T06:36:08.123 INFO:teuthology.orchestra.run.smithi028.stdout: "nodeep-scrub", 2024-02-05T06:36:08.123 INFO:teuthology.orchestra.run.smithi028.stdout: "pglog_hardlimit", 2024-02-05T06:36:08.123 INFO:teuthology.orchestra.run.smithi028.stdout: "purged_snapdirs", 2024-02-05T06:36:08.123 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_deletes", 2024-02-05T06:36:08.123 INFO:teuthology.orchestra.run.smithi028.stdout: "sortbitwise" 2024-02-05T06:36:08.123 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:36:08.124 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_version": 18, 2024-02-05T06:36:08.124 INFO:teuthology.orchestra.run.smithi028.stdout: "full_ratio": 0.94999998807907104, 2024-02-05T06:36:08.124 INFO:teuthology.orchestra.run.smithi028.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-05T06:36:08.124 INFO:teuthology.orchestra.run.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-05T06:36:08.124 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_snapshot": "", 2024-02-05T06:36:08.124 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_max": 7, 2024-02-05T06:36:08.124 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd": 8, 2024-02-05T06:36:08.124 INFO:teuthology.orchestra.run.smithi028.stdout: "require_min_compat_client": "luminous", 2024-02-05T06:36:08.124 INFO:teuthology.orchestra.run.smithi028.stdout: "min_compat_client": "luminous", 2024-02-05T06:36:08.124 INFO:teuthology.orchestra.run.smithi028.stdout: "require_osd_release": "squid", 2024-02-05T06:36:08.124 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_crimson": false, 2024-02-05T06:36:08.124 INFO:teuthology.orchestra.run.smithi028.stdout: "pools": [ 2024-02-05T06:36:08.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.124 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 1, 2024-02-05T06:36:08.125 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".mgr", 2024-02-05T06:36:08.125 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:31:10.441424+0000", 2024-02-05T06:36:08.125 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-02-05T06:36:08.125 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-02-05T06:36:08.125 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:36:08.125 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:36:08.125 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:36:08.125 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:36:08.125 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:36:08.125 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:36:08.125 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:36:08.125 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:36:08.125 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:36:08.126 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:36:08.126 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 1, 2024-02-05T06:36:08.126 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 1, 2024-02-05T06:36:08.126 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 1, 2024-02-05T06:36:08.126 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 1, 2024-02-05T06:36:08.126 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 1, 2024-02-05T06:36:08.126 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:36:08.126 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:36:08.126 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:36:08.126 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:36:08.126 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:36:08.126 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:36:08.126 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:36:08.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.127 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "21", 2024-02-05T06:36:08.127 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:36:08.127 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:36:08.127 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T06:36:08.127 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:36:08.127 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:36:08.127 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-02-05T06:36:08.127 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-02-05T06:36:08.127 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:36:08.127 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:36:08.127 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:36:08.127 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:36:08.127 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:36:08.127 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:36:08.127 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:36:08.128 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:36:08.128 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:36:08.128 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:36:08.128 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:36:08.128 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:36:08.128 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:36:08.128 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:36:08.128 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:36:08.128 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:36:08.128 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:36:08.128 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:36:08.128 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:36:08.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.129 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:36:08.129 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:36:08.129 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:36:08.129 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:36:08.129 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:36:08.129 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:36:08.129 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:36:08.129 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:36:08.129 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:36:08.129 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:36:08.129 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:36:08.129 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-02-05T06:36:08.129 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_max": 32, 2024-02-05T06:36:08.130 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_min": 1 2024-02-05T06:36:08.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.130 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:36:08.130 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": {} 2024-02-05T06:36:08.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.130 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:36:08.130 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:36:08.130 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 7.8899998664855957, 2024-02-05T06:36:08.130 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 7.8899998664855957, 2024-02-05T06:36:08.130 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.37999999523162842, 2024-02-05T06:36:08.130 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 3, 2024-02-05T06:36:08.130 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 3, 2024-02-05T06:36:08.130 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-02-05T06:36:08.130 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-02-05T06:36:08.130 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-02-05T06:36:08.131 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.131 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 2, 2024-02-05T06:36:08.131 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "datapool", 2024-02-05T06:36:08.131 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:33:27.397416+0000", 2024-02-05T06:36:08.131 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-02-05T06:36:08.131 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-05T06:36:08.131 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:36:08.131 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:36:08.131 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:36:08.131 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:36:08.131 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:36:08.131 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:36:08.131 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:36:08.132 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:36:08.132 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:36:08.132 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:36:08.132 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 3, 2024-02-05T06:36:08.132 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 3, 2024-02-05T06:36:08.132 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 3, 2024-02-05T06:36:08.132 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 3, 2024-02-05T06:36:08.132 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 3, 2024-02-05T06:36:08.132 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:36:08.132 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:36:08.132 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:36:08.132 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:36:08.132 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:36:08.132 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:36:08.133 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:36:08.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.133 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "69", 2024-02-05T06:36:08.133 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:36:08.133 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:36:08.133 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T06:36:08.133 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:36:08.133 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:36:08.133 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 2, 2024-02-05T06:36:08.133 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 69, 2024-02-05T06:36:08.133 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:36:08.133 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:36:08.133 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:36:08.133 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:36:08.133 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:36:08.134 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:36:08.134 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:36:08.134 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:36:08.134 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:36:08.134 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:36:08.134 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:36:08.134 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:36:08.134 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:36:08.134 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:36:08.134 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:36:08.134 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:36:08.134 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:36:08.134 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:36:08.134 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:36:08.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.135 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:36:08.135 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:36:08.135 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:36:08.135 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:36:08.135 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:36:08.135 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:36:08.135 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:36:08.135 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:36:08.135 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:36:08.135 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:36:08.135 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:36:08.135 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-02-05T06:36:08.135 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:36:08.135 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd": {} 2024-02-05T06:36:08.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.136 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:36:08.136 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:36:08.136 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.6500000953674316, 2024-02-05T06:36:08.136 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.6500000953674316, 2024-02-05T06:36:08.136 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T06:36:08.136 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2.3299999237060547, 2024-02-05T06:36:08.136 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2.3299999237060547, 2024-02-05T06:36:08.136 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-02-05T06:36:08.136 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-02-05T06:36:08.136 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-02-05T06:36:08.136 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.137 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 3, 2024-02-05T06:36:08.137 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".rgw.root", 2024-02-05T06:36:08.137 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:33:28.090506+0000", 2024-02-05T06:36:08.137 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-02-05T06:36:08.137 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-02-05T06:36:08.137 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:36:08.137 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:36:08.137 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:36:08.137 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:36:08.137 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:36:08.137 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:36:08.137 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:36:08.137 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:36:08.137 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:36:08.137 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:36:08.138 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-02-05T06:36:08.138 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-02-05T06:36:08.138 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-02-05T06:36:08.138 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-02-05T06:36:08.138 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-02-05T06:36:08.138 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:36:08.138 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:36:08.138 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:36:08.138 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:36:08.138 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:36:08.138 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:36:08.138 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:36:08.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.139 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "67", 2024-02-05T06:36:08.140 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:36:08.140 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:36:08.140 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T06:36:08.140 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:36:08.140 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:36:08.140 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-02-05T06:36:08.140 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-02-05T06:36:08.140 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:36:08.140 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:36:08.140 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:36:08.140 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:36:08.140 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:36:08.140 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:36:08.140 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:36:08.141 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:36:08.141 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:36:08.141 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:36:08.141 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:36:08.141 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:36:08.141 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:36:08.141 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:36:08.141 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:36:08.141 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:36:08.141 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:36:08.141 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:36:08.141 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:36:08.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.141 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:36:08.142 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:36:08.142 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:36:08.142 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:36:08.142 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:36:08.142 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:36:08.142 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:36:08.142 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:36:08.142 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:36:08.142 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:36:08.142 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:36:08.142 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-02-05T06:36:08.142 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:36:08.142 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-02-05T06:36:08.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.142 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:36:08.143 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:36:08.143 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.7400000095367432, 2024-02-05T06:36:08.143 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.7400000095367432, 2024-02-05T06:36:08.143 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T06:36:08.143 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.5299999713897705, 2024-02-05T06:36:08.143 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.5299999713897705, 2024-02-05T06:36:08.143 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-02-05T06:36:08.143 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-02-05T06:36:08.143 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-02-05T06:36:08.143 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.143 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 4, 2024-02-05T06:36:08.143 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.log", 2024-02-05T06:36:08.144 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:33:29.866249+0000", 2024-02-05T06:36:08.144 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-02-05T06:36:08.144 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-02-05T06:36:08.144 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:36:08.144 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:36:08.144 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:36:08.144 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:36:08.144 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:36:08.144 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:36:08.144 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:36:08.144 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:36:08.144 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:36:08.144 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:36:08.145 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-02-05T06:36:08.145 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-02-05T06:36:08.145 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-02-05T06:36:08.145 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-02-05T06:36:08.145 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-02-05T06:36:08.145 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:36:08.145 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:36:08.145 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:36:08.145 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:36:08.145 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:36:08.145 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:36:08.145 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:36:08.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.146 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "69", 2024-02-05T06:36:08.146 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:36:08.146 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:36:08.146 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T06:36:08.146 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:36:08.146 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:36:08.146 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-02-05T06:36:08.146 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-02-05T06:36:08.146 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:36:08.146 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:36:08.146 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:36:08.146 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:36:08.146 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:36:08.147 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:36:08.147 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:36:08.147 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:36:08.147 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:36:08.147 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:36:08.147 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:36:08.147 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:36:08.147 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:36:08.147 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:36:08.147 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:36:08.147 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:36:08.147 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:36:08.147 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:36:08.148 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:36:08.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.148 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:36:08.148 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:36:08.148 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:36:08.148 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:36:08.148 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:36:08.148 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:36:08.148 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:36:08.148 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:36:08.148 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:36:08.148 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:36:08.148 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:36:08.149 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-02-05T06:36:08.149 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:36:08.149 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-02-05T06:36:08.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.149 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:36:08.149 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:36:08.149 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.4900000095367432, 2024-02-05T06:36:08.149 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.4900000095367432, 2024-02-05T06:36:08.149 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T06:36:08.149 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2.190000057220459, 2024-02-05T06:36:08.149 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2.190000057220459, 2024-02-05T06:36:08.149 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-02-05T06:36:08.149 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-02-05T06:36:08.149 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-02-05T06:36:08.150 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.150 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 5, 2024-02-05T06:36:08.150 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.control", 2024-02-05T06:36:08.150 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:33:31.826006+0000", 2024-02-05T06:36:08.150 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-02-05T06:36:08.150 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-02-05T06:36:08.150 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:36:08.150 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:36:08.150 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:36:08.150 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:36:08.150 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:36:08.150 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:36:08.151 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:36:08.151 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:36:08.151 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:36:08.151 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:36:08.151 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-02-05T06:36:08.151 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-02-05T06:36:08.151 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-02-05T06:36:08.151 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-02-05T06:36:08.151 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-02-05T06:36:08.151 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:36:08.151 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:36:08.151 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:36:08.151 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:36:08.151 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:36:08.151 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:36:08.152 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:36:08.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.152 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "71", 2024-02-05T06:36:08.152 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:36:08.152 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:36:08.152 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T06:36:08.152 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:36:08.152 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:36:08.152 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-02-05T06:36:08.152 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-02-05T06:36:08.152 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:36:08.152 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:36:08.152 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:36:08.152 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:36:08.153 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:36:08.153 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:36:08.153 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:36:08.153 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:36:08.153 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:36:08.153 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:36:08.153 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:36:08.153 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:36:08.153 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:36:08.153 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:36:08.153 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:36:08.153 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:36:08.153 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:36:08.154 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:36:08.154 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:36:08.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.154 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:36:08.154 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:36:08.154 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:36:08.154 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:36:08.154 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:36:08.154 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:36:08.154 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:36:08.154 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:36:08.154 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:36:08.154 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:36:08.154 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:36:08.154 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-02-05T06:36:08.154 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:36:08.154 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-02-05T06:36:08.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.155 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:36:08.155 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:36:08.155 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.4900000095367432, 2024-02-05T06:36:08.155 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.4900000095367432, 2024-02-05T06:36:08.155 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T06:36:08.155 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.309999942779541, 2024-02-05T06:36:08.155 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.309999942779541, 2024-02-05T06:36:08.155 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-02-05T06:36:08.155 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-02-05T06:36:08.155 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-02-05T06:36:08.155 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.156 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 6, 2024-02-05T06:36:08.156 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.meta", 2024-02-05T06:36:08.156 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:33:33.890835+0000", 2024-02-05T06:36:08.156 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-02-05T06:36:08.156 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-02-05T06:36:08.156 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:36:08.156 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:36:08.156 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:36:08.156 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:36:08.156 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:36:08.156 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:36:08.156 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:36:08.156 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:36:08.156 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:36:08.157 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:36:08.157 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-02-05T06:36:08.157 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-02-05T06:36:08.157 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-02-05T06:36:08.157 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-02-05T06:36:08.157 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-02-05T06:36:08.157 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:36:08.157 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:36:08.157 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:36:08.157 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:36:08.157 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:36:08.157 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:36:08.157 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:36:08.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.158 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "73", 2024-02-05T06:36:08.158 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:36:08.158 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:36:08.158 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T06:36:08.158 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:36:08.158 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:36:08.158 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-02-05T06:36:08.158 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-02-05T06:36:08.158 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:36:08.158 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:36:08.158 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:36:08.158 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:36:08.158 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:36:08.158 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:36:08.159 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:36:08.159 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:36:08.159 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:36:08.159 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:36:08.159 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:36:08.159 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:36:08.159 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:36:08.159 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:36:08.159 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:36:08.159 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:36:08.159 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:36:08.159 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:36:08.159 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:36:08.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.160 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:36:08.160 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:36:08.160 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:36:08.160 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:36:08.160 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:36:08.160 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:36:08.160 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:36:08.160 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:36:08.160 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:36:08.160 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:36:08.160 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:36:08.160 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-02-05T06:36:08.160 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_bias": 4 2024-02-05T06:36:08.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.161 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:36:08.161 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-02-05T06:36:08.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.161 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:36:08.161 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:36:08.161 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.9900000095367432, 2024-02-05T06:36:08.161 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.9900000095367432, 2024-02-05T06:36:08.161 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T06:36:08.161 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.75, 2024-02-05T06:36:08.161 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.75, 2024-02-05T06:36:08.161 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-02-05T06:36:08.161 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-02-05T06:36:08.161 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-02-05T06:36:08.162 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.162 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 7, 2024-02-05T06:36:08.162 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "unique_pool_0", 2024-02-05T06:36:08.162 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:35:26.183571+0000", 2024-02-05T06:36:08.162 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-02-05T06:36:08.162 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-02-05T06:36:08.162 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:36:08.162 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:36:08.162 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:36:08.162 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:36:08.162 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:36:08.162 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:36:08.162 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:36:08.163 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:36:08.163 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:36:08.163 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:36:08.163 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 16, 2024-02-05T06:36:08.163 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 16, 2024-02-05T06:36:08.163 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 16, 2024-02-05T06:36:08.163 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 16, 2024-02-05T06:36:08.163 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 16, 2024-02-05T06:36:08.163 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:36:08.163 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:36:08.163 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:36:08.163 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:36:08.163 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:36:08.163 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:36:08.164 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:36:08.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.164 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "101", 2024-02-05T06:36:08.164 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:36:08.164 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:36:08.164 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T06:36:08.164 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:36:08.164 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:36:08.164 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-02-05T06:36:08.164 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-02-05T06:36:08.164 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:36:08.164 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:36:08.164 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:36:08.164 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:36:08.165 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:36:08.165 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:36:08.165 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:36:08.165 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:36:08.165 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:36:08.165 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:36:08.165 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:36:08.165 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:36:08.165 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:36:08.165 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:36:08.165 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:36:08.165 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:36:08.165 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:36:08.165 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:36:08.166 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:36:08.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.166 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:36:08.166 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:36:08.166 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:36:08.166 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:36:08.166 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:36:08.166 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:36:08.166 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:36:08.166 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:36:08.166 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:36:08.166 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:36:08.166 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:36:08.166 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-02-05T06:36:08.167 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:36:08.167 INFO:teuthology.orchestra.run.smithi028.stdout: "rados": {} 2024-02-05T06:36:08.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.167 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:36:08.167 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:36:08.167 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.9900000095367432, 2024-02-05T06:36:08.167 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.9900000095367432, 2024-02-05T06:36:08.167 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T06:36:08.167 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.75, 2024-02-05T06:36:08.167 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.75, 2024-02-05T06:36:08.167 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-02-05T06:36:08.167 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-02-05T06:36:08.167 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-02-05T06:36:08.167 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.168 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.168 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:36:08.168 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-02-05T06:36:08.168 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.168 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-02-05T06:36:08.168 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "c19fdd5b-09f6-45d4-a519-0a938cf1f2a6", 2024-02-05T06:36:08.168 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:36:08.168 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:36:08.168 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:36:08.168 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:36:08.168 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:36:08.168 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:36:08.168 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 8, 2024-02-05T06:36:08.168 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 96, 2024-02-05T06:36:08.169 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:36:08.169 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:36:08.169 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:36:08.169 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:36:08.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.169 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:36:08.169 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-02-05T06:36:08.169 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 301279106 2024-02-05T06:36:08.169 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.169 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.169 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:36:08.169 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:36:08.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.170 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:36:08.170 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6802", 2024-02-05T06:36:08.170 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 301279106 2024-02-05T06:36:08.170 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.170 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.170 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:36:08.170 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:36:08.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.170 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:36:08.170 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6804", 2024-02-05T06:36:08.170 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 301279106 2024-02-05T06:36:08.170 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.170 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.171 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:36:08.171 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:36:08.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.171 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:36:08.171 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6803", 2024-02-05T06:36:08.171 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 301279106 2024-02-05T06:36:08.171 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.171 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.171 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6801/301279106", 2024-02-05T06:36:08.171 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6802/301279106", 2024-02-05T06:36:08.171 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6804/301279106", 2024-02-05T06:36:08.171 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6803/301279106", 2024-02-05T06:36:08.171 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:36:08.172 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:36:08.172 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:36:08.172 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.172 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-02-05T06:36:08.172 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "52245f6e-eb96-4d0d-985b-dbb6fd3ce8de", 2024-02-05T06:36:08.172 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:36:08.172 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:36:08.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:08.173 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:08.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:08.174 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:36:08.174 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:36:08.174 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:36:08.174 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:36:08.174 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-02-05T06:36:08.174 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 96, 2024-02-05T06:36:08.174 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:36:08.174 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:36:08.174 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:36:08.174 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:36:08.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.174 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:36:08.175 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6805", 2024-02-05T06:36:08.175 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1795224649 2024-02-05T06:36:08.175 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.175 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.175 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:36:08.175 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:36:08.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.175 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:36:08.175 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6806", 2024-02-05T06:36:08.175 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1795224649 2024-02-05T06:36:08.175 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.175 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.176 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:36:08.176 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:36:08.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.176 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:36:08.176 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6808", 2024-02-05T06:36:08.176 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1795224649 2024-02-05T06:36:08.176 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.176 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.176 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:36:08.176 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:36:08.176 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.177 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:36:08.177 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6807", 2024-02-05T06:36:08.177 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1795224649 2024-02-05T06:36:08.177 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.177 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.177 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6805/1795224649", 2024-02-05T06:36:08.177 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6806/1795224649", 2024-02-05T06:36:08.177 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6808/1795224649", 2024-02-05T06:36:08.177 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6807/1795224649", 2024-02-05T06:36:08.177 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:36:08.177 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:36:08.177 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:36:08.177 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.178 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-02-05T06:36:08.178 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "1e18b630-afdf-4de9-a1a2-37bdfdd01c67", 2024-02-05T06:36:08.178 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:36:08.178 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:36:08.178 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:36:08.178 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:36:08.178 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:36:08.178 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:36:08.178 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 18, 2024-02-05T06:36:08.178 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 96, 2024-02-05T06:36:08.178 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:36:08.178 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:36:08.178 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:36:08.179 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:36:08.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.179 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:36:08.179 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6809", 2024-02-05T06:36:08.179 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1993280018 2024-02-05T06:36:08.179 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.179 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.179 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:36:08.179 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:36:08.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.179 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:36:08.179 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6810", 2024-02-05T06:36:08.179 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1993280018 2024-02-05T06:36:08.180 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.180 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.180 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:36:08.180 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:36:08.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.180 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:36:08.180 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6812", 2024-02-05T06:36:08.180 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1993280018 2024-02-05T06:36:08.180 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.180 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.180 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:36:08.180 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:36:08.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.181 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:36:08.181 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6811", 2024-02-05T06:36:08.181 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1993280018 2024-02-05T06:36:08.181 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.181 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.181 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6809/1993280018", 2024-02-05T06:36:08.181 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6810/1993280018", 2024-02-05T06:36:08.181 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6812/1993280018", 2024-02-05T06:36:08.181 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6811/1993280018", 2024-02-05T06:36:08.181 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:36:08.181 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:36:08.181 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:36:08.182 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.182 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-02-05T06:36:08.182 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "3ae795af-3d59-41a6-95eb-b8447ff61282", 2024-02-05T06:36:08.182 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:36:08.182 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:36:08.182 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:36:08.182 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:36:08.182 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:36:08.182 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:36:08.182 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 26, 2024-02-05T06:36:08.182 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 96, 2024-02-05T06:36:08.182 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:36:08.183 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:36:08.183 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:36:08.183 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:36:08.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.183 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:36:08.183 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6813", 2024-02-05T06:36:08.183 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1294621398 2024-02-05T06:36:08.183 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.183 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.183 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:36:08.183 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:36:08.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.184 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:36:08.184 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6814", 2024-02-05T06:36:08.184 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1294621398 2024-02-05T06:36:08.184 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.184 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.184 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:36:08.184 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:36:08.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.184 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:36:08.185 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6816", 2024-02-05T06:36:08.185 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1294621398 2024-02-05T06:36:08.185 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.185 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.185 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:36:08.185 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:36:08.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.185 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:36:08.185 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6815", 2024-02-05T06:36:08.185 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1294621398 2024-02-05T06:36:08.185 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.185 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.185 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6813/1294621398", 2024-02-05T06:36:08.186 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6814/1294621398", 2024-02-05T06:36:08.186 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6816/1294621398", 2024-02-05T06:36:08.186 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6815/1294621398", 2024-02-05T06:36:08.186 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:36:08.186 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:36:08.186 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:36:08.186 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.186 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-02-05T06:36:08.186 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "fc6bd83d-570c-48c0-964d-e09b4608bc32", 2024-02-05T06:36:08.186 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:36:08.186 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 0, 2024-02-05T06:36:08.186 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 0, 2024-02-05T06:36:08.187 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:36:08.187 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:36:08.187 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:36:08.187 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 31, 2024-02-05T06:36:08.187 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 77, 2024-02-05T06:36:08.187 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:36:08.187 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:36:08.187 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:36:08.187 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:36:08.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.187 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:36:08.187 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6800", 2024-02-05T06:36:08.187 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4052090595 2024-02-05T06:36:08.187 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.187 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.188 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:36:08.188 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:36:08.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.188 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:36:08.188 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6801", 2024-02-05T06:36:08.188 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4052090595 2024-02-05T06:36:08.188 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.188 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.188 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:36:08.188 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:36:08.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.188 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:36:08.189 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6803", 2024-02-05T06:36:08.189 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4052090595 2024-02-05T06:36:08.189 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.189 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.189 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:36:08.189 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:36:08.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.189 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:36:08.189 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6802", 2024-02-05T06:36:08.189 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4052090595 2024-02-05T06:36:08.189 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.189 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.190 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.167:6800/4052090595", 2024-02-05T06:36:08.190 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.167:6801/4052090595", 2024-02-05T06:36:08.190 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.167:6803/4052090595", 2024-02-05T06:36:08.190 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.167:6802/4052090595", 2024-02-05T06:36:08.190 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:36:08.190 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:36:08.190 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:36:08.190 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.190 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-02-05T06:36:08.190 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "895bf465-2531-4d9f-bc23-1e75fd969f07", 2024-02-05T06:36:08.190 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:36:08.191 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:36:08.191 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:36:08.191 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:36:08.191 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:36:08.191 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:36:08.191 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 40, 2024-02-05T06:36:08.191 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 96, 2024-02-05T06:36:08.191 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:36:08.191 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:36:08.191 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:36:08.191 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:36:08.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.191 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:36:08.191 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6804", 2024-02-05T06:36:08.192 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1631080500 2024-02-05T06:36:08.192 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.192 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.192 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:36:08.192 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:36:08.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.192 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:36:08.192 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6805", 2024-02-05T06:36:08.192 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1631080500 2024-02-05T06:36:08.192 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.192 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.192 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:36:08.193 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:36:08.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.193 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:36:08.193 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6807", 2024-02-05T06:36:08.193 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1631080500 2024-02-05T06:36:08.193 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.193 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.193 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:36:08.193 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:36:08.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.193 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:36:08.193 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6806", 2024-02-05T06:36:08.194 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1631080500 2024-02-05T06:36:08.194 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.194 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.194 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.167:6804/1631080500", 2024-02-05T06:36:08.194 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.167:6805/1631080500", 2024-02-05T06:36:08.194 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.167:6807/1631080500", 2024-02-05T06:36:08.194 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.167:6806/1631080500", 2024-02-05T06:36:08.194 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:36:08.194 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:36:08.194 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:36:08.194 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.195 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-02-05T06:36:08.195 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "ce0a9300-87a4-4e37-b702-d667f168c0e0", 2024-02-05T06:36:08.195 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:36:08.195 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:36:08.195 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:36:08.195 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:36:08.195 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:36:08.195 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:36:08.195 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 49, 2024-02-05T06:36:08.195 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 103, 2024-02-05T06:36:08.195 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:36:08.195 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:36:08.196 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:36:08.196 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:36:08.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.196 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:36:08.196 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6808", 2024-02-05T06:36:08.196 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1644050617 2024-02-05T06:36:08.196 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.196 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.196 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:36:08.196 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:36:08.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.196 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:36:08.197 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6809", 2024-02-05T06:36:08.197 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1644050617 2024-02-05T06:36:08.197 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.197 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.197 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:36:08.197 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:36:08.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.197 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:36:08.197 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6811", 2024-02-05T06:36:08.197 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1644050617 2024-02-05T06:36:08.198 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.198 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.198 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:36:08.198 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:36:08.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.198 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:36:08.198 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6810", 2024-02-05T06:36:08.198 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1644050617 2024-02-05T06:36:08.198 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.198 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.198 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.167:6808/1644050617", 2024-02-05T06:36:08.198 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.167:6809/1644050617", 2024-02-05T06:36:08.199 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.167:6811/1644050617", 2024-02-05T06:36:08.199 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.167:6810/1644050617", 2024-02-05T06:36:08.199 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:36:08.199 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:36:08.199 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:36:08.199 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.199 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-02-05T06:36:08.199 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "0c545cbc-0873-4232-8807-4af9299ef95e", 2024-02-05T06:36:08.199 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:36:08.200 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:36:08.200 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:36:08.200 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:36:08.200 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:36:08.200 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:36:08.200 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 58, 2024-02-05T06:36:08.200 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 96, 2024-02-05T06:36:08.200 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:36:08.200 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:36:08.200 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:36:08.201 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:36:08.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.201 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:36:08.201 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6812", 2024-02-05T06:36:08.201 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4031475764 2024-02-05T06:36:08.201 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.201 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.201 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:36:08.201 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:36:08.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.201 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:36:08.201 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6813", 2024-02-05T06:36:08.201 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4031475764 2024-02-05T06:36:08.202 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.202 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.202 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:36:08.202 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:36:08.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.202 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:36:08.202 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6815", 2024-02-05T06:36:08.202 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4031475764 2024-02-05T06:36:08.202 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.202 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.202 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:36:08.203 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:36:08.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.203 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:36:08.203 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6814", 2024-02-05T06:36:08.203 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4031475764 2024-02-05T06:36:08.203 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.203 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.203 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.167:6812/4031475764", 2024-02-05T06:36:08.203 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.167:6813/4031475764", 2024-02-05T06:36:08.203 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.167:6815/4031475764", 2024-02-05T06:36:08.203 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.167:6814/4031475764", 2024-02-05T06:36:08.204 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:36:08.204 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:36:08.204 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:36:08.204 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.204 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.204 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:36:08.204 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_xinfo": [ 2024-02-05T06:36:08.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.204 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-02-05T06:36:08.204 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:36:08.204 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:36:08.204 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:36:08.204 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:36:08.205 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:36:08.205 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:35:02.377925+0000", 2024-02-05T06:36:08.205 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:36:08.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.205 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-02-05T06:36:08.205 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:36:08.205 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:36:08.205 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:36:08.205 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:36:08.205 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:36:08.205 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:35:32.713008+0000", 2024-02-05T06:36:08.205 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:36:08.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.206 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.206 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-02-05T06:36:08.206 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:36:08.206 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:36:08.206 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:36:08.206 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:36:08.206 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:36:08.206 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:36:05.805349+0000", 2024-02-05T06:36:08.206 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:36:08.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.206 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.206 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-02-05T06:36:08.206 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:36:08.207 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:36:08.207 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:36:08.207 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:36:08.207 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:36:08.207 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:35:42.829117+0000", 2024-02-05T06:36:08.207 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:36:08.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.207 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.207 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-02-05T06:36:08.207 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:36:08.207 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:36:08.207 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:36:08.207 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:36:08.207 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 65536, 2024-02-05T06:36:08.207 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:35:54.077989+0000", 2024-02-05T06:36:08.208 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:36:08.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.208 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-02-05T06:36:08.208 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:36:08.208 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:36:08.208 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:36:08.208 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:36:08.208 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:36:08.208 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:36:02.166150+0000", 2024-02-05T06:36:08.208 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:36:08.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.208 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-02-05T06:36:08.209 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:36:08.209 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:36:08.209 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:36:08.209 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:36:08.209 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:36:08.209 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:35:30.750818+0000", 2024-02-05T06:36:08.209 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:36:08.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.209 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-02-05T06:36:08.209 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:36:08.209 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:36:08.209 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:36:08.209 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:36:08.209 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:36:08.210 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:35:42.874186+0000", 2024-02-05T06:36:08.210 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:36:08.210 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.210 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:36:08.210 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap": [], 2024-02-05T06:36:08.210 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_items": [ 2024-02-05T06:36:08.210 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.210 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.4", 2024-02-05T06:36:08.210 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:36:08.210 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.210 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-02-05T06:36:08.210 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 5 2024-02-05T06:36:08.210 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.210 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.211 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.1d", 2024-02-05T06:36:08.211 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:36:08.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:08.211 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-02-05T06:36:08.211 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-02-05T06:36:08.211 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.211 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:08.211 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.211 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:36:08.211 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_primaries": [], 2024-02-05T06:36:08.211 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_temp": [], 2024-02-05T06:36:08.211 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_temp": [], 2024-02-05T06:36:08.211 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklist": { 2024-02-05T06:36:08.212 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/1001442173": "2024-02-06T06:34:10.127914+0000", 2024-02-05T06:36:08.212 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3160026720": "2024-02-06T06:34:10.127914+0000", 2024-02-05T06:36:08.212 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/558171292": "2024-02-06T06:34:10.127914+0000", 2024-02-05T06:36:08.212 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2832437766": "2024-02-06T06:34:10.127914+0000", 2024-02-05T06:36:08.212 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/4125764599": "2024-02-06T06:28:54.105170+0000", 2024-02-05T06:36:08.212 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1081543653": "2024-02-06T06:28:54.105170+0000", 2024-02-05T06:36:08.212 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3794420554": "2024-02-06T06:28:27.553034+0000", 2024-02-05T06:36:08.212 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1560471317": "2024-02-06T06:34:10.127914+0000", 2024-02-05T06:36:08.212 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/4266601437": "2024-02-06T06:28:27.553034+0000", 2024-02-05T06:36:08.212 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/388893831": "2024-02-06T06:28:27.553034+0000", 2024-02-05T06:36:08.212 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/460817910": "2024-02-06T06:28:27.553034+0000", 2024-02-05T06:36:08.212 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2728305743": "2024-02-06T06:28:54.105170+0000", 2024-02-05T06:36:08.212 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3149394045": "2024-02-06T06:34:10.127914+0000", 2024-02-05T06:36:08.212 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3605013594": "2024-02-06T06:28:54.105170+0000" 2024-02-05T06:36:08.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.213 INFO:teuthology.orchestra.run.smithi028.stdout: "range_blocklist": {}, 2024-02-05T06:36:08.213 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profiles": { 2024-02-05T06:36:08.213 INFO:teuthology.orchestra.run.smithi028.stdout: "default": { 2024-02-05T06:36:08.213 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-02-05T06:36:08.213 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-02-05T06:36:08.213 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-02-05T06:36:08.213 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-02-05T06:36:08.213 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van" 2024-02-05T06:36:08.213 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:08.213 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps_queue": [], 2024-02-05T06:36:08.213 INFO:teuthology.orchestra.run.smithi028.stdout: "new_removed_snaps": [], 2024-02-05T06:36:08.213 INFO:teuthology.orchestra.run.smithi028.stdout: "new_purged_snaps": [], 2024-02-05T06:36:08.214 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_node_flags": {}, 2024-02-05T06:36:08.214 INFO:teuthology.orchestra.run.smithi028.stdout: "device_class_flags": {}, 2024-02-05T06:36:08.214 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": { 2024-02-05T06:36:08.214 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_enabled": false, 2024-02-05T06:36:08.214 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_bucket_count": 0, 2024-02-05T06:36:08.214 INFO:teuthology.orchestra.run.smithi028.stdout: "degraded_stretch_mode": 0, 2024-02-05T06:36:08.214 INFO:teuthology.orchestra.run.smithi028.stdout: "recovering_stretch_mode": 0, 2024-02-05T06:36:08.214 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_bucket": 0 2024-02-05T06:36:08.214 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:08.214 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:36:08.215 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 31 16 55945 55929 112.73 80.375 0.0196057 0.00886491 2024-02-05T06:36:08.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:08.314 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:08.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:08.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:08.416 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:08.458 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:08.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:08.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:08.560 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:08.588 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:08.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:08.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:08.690 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:08.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:08 smithi028 ceph-mon[37984]: 4.10 scrub starts 2024-02-05T06:36:08.694 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:08 smithi028 ceph-mon[37984]: 4.10 scrub ok 2024-02-05T06:36:08.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:08 smithi028 ceph-mon[37984]: 6.0 scrub starts 2024-02-05T06:36:08.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:08 smithi028 ceph-mon[37984]: 6.0 scrub ok 2024-02-05T06:36:08.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:08 smithi028 ceph-mon[37984]: 4.2 scrub starts 2024-02-05T06:36:08.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:08 smithi028 ceph-mon[37984]: 4.2 scrub ok 2024-02-05T06:36:08.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:08 smithi028 ceph-mon[37984]: 3.10 scrub starts 2024-02-05T06:36:08.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:08 smithi028 ceph-mon[37984]: 3.10 scrub ok 2024-02-05T06:36:08.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:08 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2165346596' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T06:36:08.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:08 smithi028 ceph-mon[29442]: 4.10 scrub starts 2024-02-05T06:36:08.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:08 smithi028 ceph-mon[29442]: 4.10 scrub ok 2024-02-05T06:36:08.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:08 smithi028 ceph-mon[29442]: 6.0 scrub starts 2024-02-05T06:36:08.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:08 smithi028 ceph-mon[29442]: 6.0 scrub ok 2024-02-05T06:36:08.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:08 smithi028 ceph-mon[29442]: 4.2 scrub starts 2024-02-05T06:36:08.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:08 smithi028 ceph-mon[29442]: 4.2 scrub ok 2024-02-05T06:36:08.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:08 smithi028 ceph-mon[29442]: 3.10 scrub starts 2024-02-05T06:36:08.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:08 smithi028 ceph-mon[29442]: 3.10 scrub ok 2024-02-05T06:36:08.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:08 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2165346596' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T06:36:08.728 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:08.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:08 smithi167 ceph-mon[32774]: 4.10 scrub starts 2024-02-05T06:36:08.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:08 smithi167 ceph-mon[32774]: 4.10 scrub ok 2024-02-05T06:36:08.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:08 smithi167 ceph-mon[32774]: 6.0 scrub starts 2024-02-05T06:36:08.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:08 smithi167 ceph-mon[32774]: 6.0 scrub ok 2024-02-05T06:36:08.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:08 smithi167 ceph-mon[32774]: 4.2 scrub starts 2024-02-05T06:36:08.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:08 smithi167 ceph-mon[32774]: 4.2 scrub ok 2024-02-05T06:36:08.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:08 smithi167 ceph-mon[32774]: 3.10 scrub starts 2024-02-05T06:36:08.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:08 smithi167 ceph-mon[32774]: 3.10 scrub ok 2024-02-05T06:36:08.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:08 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2165346596' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T06:36:08.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:08.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:08.831 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:08.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:08.873 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:08.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:08.975 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:09.002 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:09.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:09.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:09.063 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T06:36:09.104 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:09.131 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:09.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:09.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:09.202 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 32 16 57313 57297 111.878 85.5 0.0082176 0.00893369 2024-02-05T06:36:09.233 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:09.263 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:09.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:09.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:09.365 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:09.392 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:09.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:09.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:09.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:09 smithi028 ceph-mon[29442]: 5.1b scrub starts 2024-02-05T06:36:09.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:09 smithi028 ceph-mon[29442]: 5.1b scrub ok 2024-02-05T06:36:09.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:09 smithi028 ceph-mon[29442]: pgmap v87: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-02-05T06:36:09.395 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:09 smithi028 ceph-mon[37984]: 5.1b scrub starts 2024-02-05T06:36:09.395 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:09 smithi028 ceph-mon[37984]: 5.1b scrub ok 2024-02-05T06:36:09.395 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:09 smithi028 ceph-mon[37984]: pgmap v87: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-02-05T06:36:09.495 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:09.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:09.545 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:09.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:09.646 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:09.686 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:09.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:09.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:09.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:09 smithi167 ceph-mon[32774]: 5.1b scrub starts 2024-02-05T06:36:09.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:09 smithi167 ceph-mon[32774]: 5.1b scrub ok 2024-02-05T06:36:09.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:09 smithi167 ceph-mon[32774]: pgmap v87: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-02-05T06:36:09.788 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:09.814 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:09.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:09.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:09.916 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:09.960 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:09.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:09.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:10.056 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:36:10.056 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:36:09 smithi028 podman[80704]: 2024-02-05 06:36:09.713685756 +0000 UTC m=+3.041762873 container init 696060c3ff8b7a9d0f55bed2f60178446b4a783b14be24ba5edb33e52b210ede (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T06:36:10.057 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:36:09 smithi028 podman[80704]: 2024-02-05 06:36:09.724248505 +0000 UTC m=+3.052325611 container start 696060c3ff8b7a9d0f55bed2f60178446b4a783b14be24ba5edb33e52b210ede (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T06:36:10.057 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:36:09 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a[80834]: ts=2024-02-05T06:36:09.749Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-02-05T06:36:10.057 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:36:09 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a[80834]: ts=2024-02-05T06:36:09.750Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-02-05T06:36:10.057 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:36:09 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a[80834]: ts=2024-02-05T06:36:09.754Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.28 port=9094 2024-02-05T06:36:10.057 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:36:09 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a[80834]: ts=2024-02-05T06:36:09.757Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-05T06:36:10.057 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:36:09 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a[80834]: ts=2024-02-05T06:36:09.809Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-05T06:36:10.057 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:36:09 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a[80834]: ts=2024-02-05T06:36:09.810Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-05T06:36:10.057 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:36:09 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a[80834]: ts=2024-02-05T06:36:09.815Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-02-05T06:36:10.057 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:36:09 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a[80834]: ts=2024-02-05T06:36:09.816Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-02-05T06:36:10.062 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:10.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:10.087 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:10.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:10.112 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:36:10.152 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:36:10.188 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:10.194 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:36:10.202 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 33 16 58513 58497 110.761 75 0.00812961 0.00902234 2024-02-05T06:36:10.236 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:36:10.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:10.237 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:10.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:10.276 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:36:10.317 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:36:10.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[29442]: 6.14 scrub starts 2024-02-05T06:36:10.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[29442]: 6.14 scrub ok 2024-02-05T06:36:10.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[29442]: 3.1d scrub starts 2024-02-05T06:36:10.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[29442]: 3.1d scrub ok 2024-02-05T06:36:10.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[29442]: 3.1f scrub starts 2024-02-05T06:36:10.318 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[29442]: 3.1f scrub ok 2024-02-05T06:36:10.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[29442]: 3.4 scrub starts 2024-02-05T06:36:10.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[29442]: 3.4 scrub ok 2024-02-05T06:36:10.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[29442]: osdmap e108: 8 total, 8 up, 7 in 2024-02-05T06:36:10.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/524696014' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:36:10.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:10.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:10.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [3, 5]}]: dispatch 2024-02-05T06:36:10.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-02-05T06:36:10.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]: dispatch 2024-02-05T06:36:10.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1e", "id": [0, 3]}]: dispatch 2024-02-05T06:36:10.319 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [3, 5]}]: dispatch 2024-02-05T06:36:10.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-02-05T06:36:10.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]: dispatch 2024-02-05T06:36:10.320 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1e", "id": [0, 3]}]: dispatch 2024-02-05T06:36:10.320 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[37984]: 6.14 scrub starts 2024-02-05T06:36:10.320 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[37984]: 6.14 scrub ok 2024-02-05T06:36:10.320 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[37984]: 3.1d scrub starts 2024-02-05T06:36:10.320 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[37984]: 3.1d scrub ok 2024-02-05T06:36:10.320 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[37984]: 3.1f scrub starts 2024-02-05T06:36:10.320 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[37984]: 3.1f scrub ok 2024-02-05T06:36:10.321 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[37984]: 3.4 scrub starts 2024-02-05T06:36:10.321 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[37984]: 3.4 scrub ok 2024-02-05T06:36:10.321 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[37984]: osdmap e108: 8 total, 8 up, 7 in 2024-02-05T06:36:10.321 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/524696014' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:36:10.321 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:10.321 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:10.321 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [3, 5]}]: dispatch 2024-02-05T06:36:10.321 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-02-05T06:36:10.321 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]: dispatch 2024-02-05T06:36:10.321 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1e", "id": [0, 3]}]: dispatch 2024-02-05T06:36:10.321 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [3, 5]}]: dispatch 2024-02-05T06:36:10.321 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-02-05T06:36:10.321 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]: dispatch 2024-02-05T06:36:10.321 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:10 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1e", "id": [0, 3]}]: dispatch 2024-02-05T06:36:10.322 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:36:10 smithi028 bash[80704]: 696060c3ff8b7a9d0f55bed2f60178446b4a783b14be24ba5edb33e52b210ede 2024-02-05T06:36:10.322 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:36:10 smithi028 systemd[1]: Started Ceph alertmanager.a for 9801e956-c3ef-11ee-95b6-87774f69a715. 2024-02-05T06:36:10.338 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:10.359 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:36:10.364 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:10.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:10.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:10.372 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:36:10.373 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:36:10.373 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:36:10.373 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:36:10.373 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:36:10.373 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:36:10.373 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:36:10.373 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:36:10.466 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:10.497 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:10.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:10.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:10.599 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:10.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:10.633 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:10.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:10.708 INFO:tasks.thrashosds.thrasher:j is {'epoch': 107, 'fsid': '9801e956-c3ef-11ee-95b6-87774f69a715', 'created': '2024-02-05T06:27:59.919226+0000', 'modified': '2024-02-05T06:36:06.246435+0000', 'last_up_change': '2024-02-05T06:33:15.833573+0000', 'last_in_change': '2024-02-05T06:35:23.906253+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-05T06:31:10.441424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-02-05T06:33:27.397416+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-02-05T06:33:28.090506+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-05T06:33:29.866249+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.490000009536743, 'score_stable': 2.490000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-05T06:33:31.826006+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-05T06:33:33.890835+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-02-05T06:35:26.183571+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'c19fdd5b-09f6-45d4-a519-0a938cf1f2a6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6801', 'nonce': 301279106}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6802', 'nonce': 301279106}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6804', 'nonce': 301279106}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6803', 'nonce': 301279106}]}, 'public_addr': '172.21.15.28:6801/301279106', 'cluster_addr': '172.21.15.28:6802/301279106', 'heartbeat_back_addr': '172.21.15.28:6804/301279106', 'heartbeat_front_addr': '172.21.15.28:6803/301279106', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '52245f6e-eb96-4d0d-985b-dbb6fd3ce8de', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6805', 'nonce': 1795224649}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6806', 'nonce': 1795224649}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6808', 'nonce': 1795224649}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6807', 'nonce': 1795224649}]}, 'public_addr': '172.21.15.28:6805/1795224649', 'cluster_addr': '172.21.15.28:6806/1795224649', 'heartbeat_back_addr': '172.21.15.28:6808/1795224649', 'heartbeat_front_addr': '172.21.15.28:6807/1795224649', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '1e18b630-afdf-4de9-a1a2-37bdfdd01c67', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6809', 'nonce': 1993280018}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6810', 'nonce': 1993280018}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6812', 'nonce': 1993280018}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6811', 'nonce': 1993280018}]}, 'public_addr': '172.21.15.28:6809/1993280018', 'cluster_addr': '172.21.15.28:6810/1993280018', 'heartbeat_back_addr': '172.21.15.28:6812/1993280018', 'heartbeat_front_addr': '172.21.15.28:6811/1993280018', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '3ae795af-3d59-41a6-95eb-b8447ff61282', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6813', 'nonce': 1294621398}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6814', 'nonce': 1294621398}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6816', 'nonce': 1294621398}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6815', 'nonce': 1294621398}]}, 'public_addr': '172.21.15.28:6813/1294621398', 'cluster_addr': '172.21.15.28:6814/1294621398', 'heartbeat_back_addr': '172.21.15.28:6816/1294621398', 'heartbeat_front_addr': '172.21.15.28:6815/1294621398', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'fc6bd83d-570c-48c0-964d-e09b4608bc32', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 77, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6800', 'nonce': 4052090595}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6801', 'nonce': 4052090595}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6803', 'nonce': 4052090595}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6802', 'nonce': 4052090595}]}, 'public_addr': '172.21.15.167:6800/4052090595', 'cluster_addr': '172.21.15.167:6801/4052090595', 'heartbeat_back_addr': '172.21.15.167:6803/4052090595', 'heartbeat_front_addr': '172.21.15.167:6802/4052090595', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '895bf465-2531-4d9f-bc23-1e75fd969f07', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6804', 'nonce': 1631080500}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6805', 'nonce': 1631080500}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6807', 'nonce': 1631080500}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6806', 'nonce': 1631080500}]}, 'public_addr': '172.21.15.167:6804/1631080500', 'cluster_addr': '172.21.15.167:6805/1631080500', 'heartbeat_back_addr': '172.21.15.167:6807/1631080500', 'heartbeat_front_addr': '172.21.15.167:6806/1631080500', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ce0a9300-87a4-4e37-b702-d667f168c0e0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 103, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6808', 'nonce': 1644050617}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6809', 'nonce': 1644050617}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6811', 'nonce': 1644050617}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6810', 'nonce': 1644050617}]}, 'public_addr': '172.21.15.167:6808/1644050617', 'cluster_addr': '172.21.15.167:6809/1644050617', 'heartbeat_back_addr': '172.21.15.167:6811/1644050617', 'heartbeat_front_addr': '172.21.15.167:6810/1644050617', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '0c545cbc-0873-4232-8807-4af9299ef95e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6812', 'nonce': 4031475764}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6813', 'nonce': 4031475764}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6815', 'nonce': 4031475764}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6814', 'nonce': 4031475764}]}, 'public_addr': '172.21.15.167:6812/4031475764', 'cluster_addr': '172.21.15.167:6813/4031475764', 'heartbeat_back_addr': '172.21.15.167:6815/4031475764', 'heartbeat_front_addr': '172.21.15.167:6814/4031475764', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:35:02.377925+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:35:32.713008+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:36:05.805349+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:35:42.829117+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-02-05T06:35:54.077989+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:36:02.166150+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:35:30.750818+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:35:42.874186+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.28:6800/1001442173': '2024-02-06T06:34:10.127914+0000', '172.21.15.28:0/3160026720': '2024-02-06T06:34:10.127914+0000', '172.21.15.28:0/558171292': '2024-02-06T06:34:10.127914+0000', '172.21.15.28:0/2832437766': '2024-02-06T06:34:10.127914+0000', '172.21.15.28:6800/4125764599': '2024-02-06T06:28:54.105170+0000', '172.21.15.28:0/1081543653': '2024-02-06T06:28:54.105170+0000', '172.21.15.28:0/3794420554': '2024-02-06T06:28:27.553034+0000', '172.21.15.28:0/1560471317': '2024-02-06T06:34:10.127914+0000', '172.21.15.28:6800/4266601437': '2024-02-06T06:28:27.553034+0000', '172.21.15.28:0/388893831': '2024-02-06T06:28:27.553034+0000', '172.21.15.28:0/460817910': '2024-02-06T06:28:27.553034+0000', '172.21.15.28:0/2728305743': '2024-02-06T06:28:54.105170+0000', '172.21.15.28:0/3149394045': '2024-02-06T06:34:10.127914+0000', '172.21.15.28:0/3605013594': '2024-02-06T06:28:54.105170+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-05T06:36:10.708 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T06:36:10.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:10 smithi167 ceph-mon[32774]: 6.14 scrub starts 2024-02-05T06:36:10.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:10 smithi167 ceph-mon[32774]: 6.14 scrub ok 2024-02-05T06:36:10.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:10 smithi167 ceph-mon[32774]: 3.1d scrub starts 2024-02-05T06:36:10.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:10 smithi167 ceph-mon[32774]: 3.1d scrub ok 2024-02-05T06:36:10.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:10 smithi167 ceph-mon[32774]: 3.1f scrub starts 2024-02-05T06:36:10.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:10 smithi167 ceph-mon[32774]: 3.1f scrub ok 2024-02-05T06:36:10.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:10 smithi167 ceph-mon[32774]: 3.4 scrub starts 2024-02-05T06:36:10.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:10 smithi167 ceph-mon[32774]: 3.4 scrub ok 2024-02-05T06:36:10.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:10 smithi167 ceph-mon[32774]: osdmap e108: 8 total, 8 up, 7 in 2024-02-05T06:36:10.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:10 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/524696014' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:36:10.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:10 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:10.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:10 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:10.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:10 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [3, 5]}]: dispatch 2024-02-05T06:36:10.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:10 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-02-05T06:36:10.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:10 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]: dispatch 2024-02-05T06:36:10.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:10 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1e", "id": [0, 3]}]: dispatch 2024-02-05T06:36:10.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:10 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [3, 5]}]: dispatch 2024-02-05T06:36:10.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:10 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-02-05T06:36:10.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:10 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]: dispatch 2024-02-05T06:36:10.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:10 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1e", "id": [0, 3]}]: dispatch 2024-02-05T06:36:10.735 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:10.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:10.792 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:10.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:10.893 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:10.954 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:10.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:10.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:11.056 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:11.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:11.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:11.202 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 34 16 59734 59718 109.748 76.3125 0.0100179 0.00910698 2024-02-05T06:36:11.231 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:11.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:11.273 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:11.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:11.374 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:11.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:11.417 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:11.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:11.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:11 smithi028 ceph-mon[37984]: 5.6 scrub starts 2024-02-05T06:36:11.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:11 smithi028 ceph-mon[37984]: 5.6 scrub ok 2024-02-05T06:36:11.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:11 smithi028 ceph-mon[37984]: 4.15 scrub starts 2024-02-05T06:36:11.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:11 smithi028 ceph-mon[37984]: 4.15 scrub ok 2024-02-05T06:36:11.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:11 smithi028 ceph-mon[37984]: pgmap v89: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-02-05T06:36:11.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:11 smithi028 ceph-mon[37984]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-05T06:36:11.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:11 smithi028 ceph-mon[37984]: 3.19 scrub starts 2024-02-05T06:36:11.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:11 smithi028 ceph-mon[37984]: 3.19 scrub ok 2024-02-05T06:36:11.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:11 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:36:11.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:11 smithi028 ceph-mon[37984]: Reconfiguring daemon prometheus.a on smithi167 2024-02-05T06:36:11.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:11 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [3, 5]}]': finished 2024-02-05T06:36:11.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:11 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-02-05T06:36:11.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:11 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]': finished 2024-02-05T06:36:11.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:11 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1e", "id": [0, 3]}]': finished 2024-02-05T06:36:11.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:11 smithi028 ceph-mon[37984]: osdmap e109: 8 total, 8 up, 7 in 2024-02-05T06:36:11.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:11 smithi028 ceph-mon[29442]: 5.6 scrub starts 2024-02-05T06:36:11.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:11 smithi028 ceph-mon[29442]: 5.6 scrub ok 2024-02-05T06:36:11.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:11 smithi028 ceph-mon[29442]: 4.15 scrub starts 2024-02-05T06:36:11.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:11 smithi028 ceph-mon[29442]: 4.15 scrub ok 2024-02-05T06:36:11.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:11 smithi028 ceph-mon[29442]: pgmap v89: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-02-05T06:36:11.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:11 smithi028 ceph-mon[29442]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-05T06:36:11.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:11 smithi028 ceph-mon[29442]: 3.19 scrub starts 2024-02-05T06:36:11.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:11 smithi028 ceph-mon[29442]: 3.19 scrub ok 2024-02-05T06:36:11.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:11 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:36:11.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:11 smithi028 ceph-mon[29442]: Reconfiguring daemon prometheus.a on smithi167 2024-02-05T06:36:11.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:11 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [3, 5]}]': finished 2024-02-05T06:36:11.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:11 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-02-05T06:36:11.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:11 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]': finished 2024-02-05T06:36:11.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:11 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1e", "id": [0, 3]}]': finished 2024-02-05T06:36:11.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:11 smithi028 ceph-mon[29442]: osdmap e109: 8 total, 8 up, 7 in 2024-02-05T06:36:11.486 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:36:11.519 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:11.541 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:36:11.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:11.547 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:11.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:11.648 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:11.651 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:11 smithi167 ceph-mon[32774]: 5.6 scrub starts 2024-02-05T06:36:11.651 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:11 smithi167 ceph-mon[32774]: 5.6 scrub ok 2024-02-05T06:36:11.651 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:11 smithi167 ceph-mon[32774]: 4.15 scrub starts 2024-02-05T06:36:11.651 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:11 smithi167 ceph-mon[32774]: 4.15 scrub ok 2024-02-05T06:36:11.651 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:11 smithi167 ceph-mon[32774]: pgmap v89: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-02-05T06:36:11.651 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:11 smithi167 ceph-mon[32774]: Reconfiguring prometheus.a (dependencies changed)... 2024-02-05T06:36:11.651 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:11 smithi167 ceph-mon[32774]: 3.19 scrub starts 2024-02-05T06:36:11.651 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:11 smithi167 ceph-mon[32774]: 3.19 scrub ok 2024-02-05T06:36:11.652 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:11 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:36:11.652 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:11 smithi167 ceph-mon[32774]: Reconfiguring daemon prometheus.a on smithi167 2024-02-05T06:36:11.652 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:11 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [3, 5]}]': finished 2024-02-05T06:36:11.652 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:11 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-02-05T06:36:11.652 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:11 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]': finished 2024-02-05T06:36:11.652 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:11 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1e", "id": [0, 3]}]': finished 2024-02-05T06:36:11.652 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:11 smithi167 ceph-mon[32774]: osdmap e109: 8 total, 8 up, 7 in 2024-02-05T06:36:11.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:11.700 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:11.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:11.801 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:11.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:11.830 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:11.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:11.931 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:11.968 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:11.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:11.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:12.070 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:12.093 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:12.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:12.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:12.194 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:12.195 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:36:11 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a[80834]: ts=2024-02-05T06:36:11.757Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00032524s 2024-02-05T06:36:12.203 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 35 16 60843 60827 108.592 69.3125 0.0133208 0.00920313 2024-02-05T06:36:12.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:12.236 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:12.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:12.337 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:12.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:12.379 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:12.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:12.480 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:12.504 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:12.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:12.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:12.558 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:12 smithi167 ceph-mon[32774]: 4.1c scrub starts 2024-02-05T06:36:12.558 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:12 smithi167 ceph-mon[32774]: 4.1c scrub ok 2024-02-05T06:36:12.558 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:12 smithi167 ceph-mon[32774]: 5.e scrub starts 2024-02-05T06:36:12.558 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:12 smithi167 ceph-mon[32774]: 5.e scrub ok 2024-02-05T06:36:12.559 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:12 smithi167 ceph-mon[32774]: 6.3 scrub starts 2024-02-05T06:36:12.559 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:12 smithi167 ceph-mon[32774]: 6.3 scrub ok 2024-02-05T06:36:12.559 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:12 smithi167 ceph-mon[32774]: osdmap e110: 8 total, 8 up, 7 in 2024-02-05T06:36:12.559 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:36:12 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:36:12.606 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:12.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:12.646 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:12.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:12.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:12 smithi028 ceph-mon[29442]: 4.1c scrub starts 2024-02-05T06:36:12.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:12 smithi028 ceph-mon[29442]: 4.1c scrub ok 2024-02-05T06:36:12.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:12 smithi028 ceph-mon[29442]: 5.e scrub starts 2024-02-05T06:36:12.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:12 smithi028 ceph-mon[29442]: 5.e scrub ok 2024-02-05T06:36:12.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:12 smithi028 ceph-mon[29442]: 6.3 scrub starts 2024-02-05T06:36:12.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:12 smithi028 ceph-mon[29442]: 6.3 scrub ok 2024-02-05T06:36:12.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:12 smithi028 ceph-mon[29442]: osdmap e110: 8 total, 8 up, 7 in 2024-02-05T06:36:12.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:12 smithi028 ceph-mon[37984]: 4.1c scrub starts 2024-02-05T06:36:12.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:12 smithi028 ceph-mon[37984]: 4.1c scrub ok 2024-02-05T06:36:12.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:12 smithi028 ceph-mon[37984]: 5.e scrub starts 2024-02-05T06:36:12.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:12 smithi028 ceph-mon[37984]: 5.e scrub ok 2024-02-05T06:36:12.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:12 smithi028 ceph-mon[37984]: 6.3 scrub starts 2024-02-05T06:36:12.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:12 smithi028 ceph-mon[37984]: 6.3 scrub ok 2024-02-05T06:36:12.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:12 smithi028 ceph-mon[37984]: osdmap e110: 8 total, 8 up, 7 in 2024-02-05T06:36:12.696 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:36:12 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:36:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:36:12.748 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:12.772 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:12.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:12.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:12.874 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:12.915 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:12.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:12.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:13.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:13.055 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:13.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:13.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:13.157 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:13.186 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:13.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:13.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:13.203 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 36 16 62000 61984 107.584 72.3125 0.0112733 0.00928979 2024-02-05T06:36:13.287 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:13.321 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:13.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:13.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:13.423 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:13.450 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:13.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:13.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:13.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:13 smithi167 ceph-mon[32774]: pgmap v92: 148 pgs: 4 peering, 144 active+clean; 3.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 864 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-02-05T06:36:13.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:13 smithi167 ceph-mon[32774]: 6.1b scrub starts 2024-02-05T06:36:13.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:13 smithi167 ceph-mon[32774]: 6.1b scrub ok 2024-02-05T06:36:13.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:13 smithi167 ceph-mon[32774]: 3.1a scrub starts 2024-02-05T06:36:13.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:13 smithi167 ceph-mon[32774]: 3.1a scrub ok 2024-02-05T06:36:13.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:13 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:36:13.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:13 smithi167 ceph-mon[32774]: 6.d scrub starts 2024-02-05T06:36:13.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:13 smithi167 ceph-mon[32774]: 6.d scrub ok 2024-02-05T06:36:13.514 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:36:13.553 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:13.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:13 smithi028 ceph-mon[29442]: pgmap v92: 148 pgs: 4 peering, 144 active+clean; 3.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 864 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-02-05T06:36:13.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:13 smithi028 ceph-mon[29442]: 6.1b scrub starts 2024-02-05T06:36:13.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:13 smithi028 ceph-mon[29442]: 6.1b scrub ok 2024-02-05T06:36:13.559 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:13 smithi028 ceph-mon[29442]: 3.1a scrub starts 2024-02-05T06:36:13.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:13 smithi028 ceph-mon[29442]: 3.1a scrub ok 2024-02-05T06:36:13.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:13 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:36:13.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:13 smithi028 ceph-mon[29442]: 6.d scrub starts 2024-02-05T06:36:13.560 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:13 smithi028 ceph-mon[29442]: 6.d scrub ok 2024-02-05T06:36:13.560 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:13 smithi028 ceph-mon[37984]: pgmap v92: 148 pgs: 4 peering, 144 active+clean; 3.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 864 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-02-05T06:36:13.560 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:13 smithi028 ceph-mon[37984]: 6.1b scrub starts 2024-02-05T06:36:13.560 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:13 smithi028 ceph-mon[37984]: 6.1b scrub ok 2024-02-05T06:36:13.560 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:13 smithi028 ceph-mon[37984]: 3.1a scrub starts 2024-02-05T06:36:13.560 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:13 smithi028 ceph-mon[37984]: 3.1a scrub ok 2024-02-05T06:36:13.561 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:13 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:36:13.561 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:13 smithi028 ceph-mon[37984]: 6.d scrub starts 2024-02-05T06:36:13.561 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:13 smithi028 ceph-mon[37984]: 6.d scrub ok 2024-02-05T06:36:13.588 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:13.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:13.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:13.689 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:13.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:13.737 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:13.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:13.839 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:13.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:13.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:13.974 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:14.003 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:14.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:14.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:14.105 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:14.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:14.134 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:14.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:14.203 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 37 16 62994 62978 106.355 62.125 0.0163561 0.00939622 2024-02-05T06:36:14.235 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:14.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:14.269 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:14.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:14.370 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:14.399 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:14.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:14.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:14.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:14 smithi167 ceph-mon[32774]: 5.11 scrub starts 2024-02-05T06:36:14.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:14 smithi167 ceph-mon[32774]: 3.5 scrub starts 2024-02-05T06:36:14.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:14 smithi167 ceph-mon[32774]: 5.11 scrub ok 2024-02-05T06:36:14.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:14 smithi167 ceph-mon[32774]: 3.5 scrub ok 2024-02-05T06:36:14.480 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:14 smithi167 systemd[1]: Stopping Ceph prometheus.a for 9801e956-c3ef-11ee-95b6-87774f69a715... 2024-02-05T06:36:14.501 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:14.530 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:14.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:14.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:14.632 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:14.672 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:14.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:14.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:14.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:14 smithi028 ceph-mon[29442]: 5.11 scrub starts 2024-02-05T06:36:14.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:14 smithi028 ceph-mon[29442]: 3.5 scrub starts 2024-02-05T06:36:14.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:14 smithi028 ceph-mon[29442]: 5.11 scrub ok 2024-02-05T06:36:14.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:14 smithi028 ceph-mon[29442]: 3.5 scrub ok 2024-02-05T06:36:14.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:14 smithi028 ceph-mon[37984]: 5.11 scrub starts 2024-02-05T06:36:14.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:14 smithi028 ceph-mon[37984]: 3.5 scrub starts 2024-02-05T06:36:14.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:14 smithi028 ceph-mon[37984]: 5.11 scrub ok 2024-02-05T06:36:14.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:14 smithi028 ceph-mon[37984]: 3.5 scrub ok 2024-02-05T06:36:14.775 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:14.819 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:14.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:14.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:14.921 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:14.946 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:14.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:14.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:14.979 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:14 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[61757]: ts=2024-02-05T06:36:14.541Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-05T06:36:14.979 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:14 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[61757]: ts=2024-02-05T06:36:14.541Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-02-05T06:36:14.980 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:14 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[61757]: ts=2024-02-05T06:36:14.541Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-02-05T06:36:14.980 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:14 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[61757]: ts=2024-02-05T06:36:14.541Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-05T06:36:14.980 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:14 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[61757]: ts=2024-02-05T06:36:14.541Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-02-05T06:36:14.980 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:14 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[61757]: ts=2024-02-05T06:36:14.541Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-02-05T06:36:14.980 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:14 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[61757]: ts=2024-02-05T06:36:14.541Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-02-05T06:36:14.980 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:14 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[61757]: ts=2024-02-05T06:36:14.541Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-02-05T06:36:14.980 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:14 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[61757]: ts=2024-02-05T06:36:14.541Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-02-05T06:36:14.980 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:14 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[61757]: ts=2024-02-05T06:36:14.541Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-02-05T06:36:14.980 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:14 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[61757]: ts=2024-02-05T06:36:14.541Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-02-05T06:36:14.980 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:14 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[61757]: ts=2024-02-05T06:36:14.541Z caller=main.go:1147 level=info msg="See you next time!" 2024-02-05T06:36:14.980 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:14 smithi167 podman[68325]: 2024-02-05 06:36:14.552071418 +0000 UTC m=+0.145765823 container died 25a3a392977ed28b585d05a58abd7476624db1594930592b8d0c2f94d7788a5c (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-05T06:36:15.048 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:15.075 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:15.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:15.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:15.177 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:15.203 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 38 16 64158 64142 105.471 72.75 0.00592841 0.00947613 2024-02-05T06:36:15.204 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:15.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:15.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:15.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:15.343 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:15.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:15.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:15.421 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:36:15.445 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:15.474 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:15.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:15.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:15.576 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:15.618 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:15.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:15.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:15.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:15 smithi028 ceph-mon[29442]: pgmap v93: 148 pgs: 4 peering, 144 active+clean; 3.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-02-05T06:36:15.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:15 smithi028 ceph-mon[29442]: 4.7 scrub starts 2024-02-05T06:36:15.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:15 smithi028 ceph-mon[29442]: 4.7 scrub ok 2024-02-05T06:36:15.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:15 smithi028 ceph-mon[29442]: 4.a scrub starts 2024-02-05T06:36:15.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:15 smithi028 ceph-mon[29442]: 4.a scrub ok 2024-02-05T06:36:15.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:15 smithi028 ceph-mon[29442]: 2.2 scrub starts 2024-02-05T06:36:15.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:15 smithi028 ceph-mon[29442]: 2.2 scrub ok 2024-02-05T06:36:15.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:15 smithi028 ceph-mon[29442]: osdmap e111: 8 total, 8 up, 7 in 2024-02-05T06:36:15.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:15 smithi028 ceph-mon[37984]: pgmap v93: 148 pgs: 4 peering, 144 active+clean; 3.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-02-05T06:36:15.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:15 smithi028 ceph-mon[37984]: 4.7 scrub starts 2024-02-05T06:36:15.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:15 smithi028 ceph-mon[37984]: 4.7 scrub ok 2024-02-05T06:36:15.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:15 smithi028 ceph-mon[37984]: 4.a scrub starts 2024-02-05T06:36:15.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:15 smithi028 ceph-mon[37984]: 4.a scrub ok 2024-02-05T06:36:15.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:15 smithi028 ceph-mon[37984]: 2.2 scrub starts 2024-02-05T06:36:15.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:15 smithi028 ceph-mon[37984]: 2.2 scrub ok 2024-02-05T06:36:15.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:15 smithi028 ceph-mon[37984]: osdmap e111: 8 total, 8 up, 7 in 2024-02-05T06:36:15.721 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:15.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:15 smithi167 ceph-mon[32774]: pgmap v93: 148 pgs: 4 peering, 144 active+clean; 3.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-02-05T06:36:15.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:15 smithi167 ceph-mon[32774]: 4.7 scrub starts 2024-02-05T06:36:15.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:15 smithi167 ceph-mon[32774]: 4.7 scrub ok 2024-02-05T06:36:15.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:15 smithi167 ceph-mon[32774]: 4.a scrub starts 2024-02-05T06:36:15.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:15 smithi167 ceph-mon[32774]: 4.a scrub ok 2024-02-05T06:36:15.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:15 smithi167 ceph-mon[32774]: 2.2 scrub starts 2024-02-05T06:36:15.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:15 smithi167 ceph-mon[32774]: 2.2 scrub ok 2024-02-05T06:36:15.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:15 smithi167 ceph-mon[32774]: osdmap e111: 8 total, 8 up, 7 in 2024-02-05T06:36:15.730 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:15 smithi167 podman[68325]: 2024-02-05 06:36:15.549326431 +0000 UTC m=+1.143020835 container cleanup 25a3a392977ed28b585d05a58abd7476624db1594930592b8d0c2f94d7788a5c (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-05T06:36:15.730 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:15 smithi167 bash[68325]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a 2024-02-05T06:36:15.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:15.754 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:15.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:15.855 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:15.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:15.889 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:15.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:15.991 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:16.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:16.026 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:16.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:16.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:16.140 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:36:16.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:16.195 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:16.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:16.203 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 39 16 65438 65422 104.817 80 0.0223645 0.00953441 2024-02-05T06:36:16.229 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:15 smithi167 podman[68354]: 2024-02-05 06:36:15.83506333 +0000 UTC m=+1.282062228 container remove 25a3a392977ed28b585d05a58abd7476624db1594930592b8d0c2f94d7788a5c (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-05T06:36:16.229 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:15 smithi167 systemd[1]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-02-05T06:36:16.229 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:15 smithi167 systemd[1]: Stopped Ceph prometheus.a for 9801e956-c3ef-11ee-95b6-87774f69a715. 2024-02-05T06:36:16.229 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:15 smithi167 systemd[1]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715@prometheus.a.service: Consumed 1.276s CPU time. 2024-02-05T06:36:16.230 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:15 smithi167 systemd[1]: Starting Ceph prometheus.a for 9801e956-c3ef-11ee-95b6-87774f69a715... 2024-02-05T06:36:16.296 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:16.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:16.327 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:16.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:16.428 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:16.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:16.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:16.574 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:16.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:16.604 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:16.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:16.633 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:16 smithi028 ceph-mon[29442]: 3.3 scrub starts 2024-02-05T06:36:16.633 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:16 smithi028 ceph-mon[29442]: 3.3 scrub ok 2024-02-05T06:36:16.633 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:16 smithi028 ceph-mon[37984]: 3.3 scrub starts 2024-02-05T06:36:16.634 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:16 smithi028 ceph-mon[37984]: 3.3 scrub ok 2024-02-05T06:36:16.677 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:16 smithi167 ceph-mon[32774]: 3.3 scrub starts 2024-02-05T06:36:16.677 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:16 smithi167 ceph-mon[32774]: 3.3 scrub ok 2024-02-05T06:36:16.677 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:16 smithi167 podman[68545]: 2024-02-05 06:36:16.154963108 +0000 UTC m=+0.026780744 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-02-05T06:36:16.677 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:16 smithi167 podman[68545]: 2024-02-05 06:36:16.278267019 +0000 UTC m=+0.150084654 container create 89e77b8f00daed45e30629aef5555c38dc6a653976263cef60036bd21e2c6119 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-05T06:36:16.706 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:16.744 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:16.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:16.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:16.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:16.887 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:16.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:16.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:16.934 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:16 smithi167 podman[68545]: 2024-02-05 06:36:16.806613204 +0000 UTC m=+0.678430855 container init 89e77b8f00daed45e30629aef5555c38dc6a653976263cef60036bd21e2c6119 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-05T06:36:16.935 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:16 smithi167 podman[68545]: 2024-02-05 06:36:16.810546178 +0000 UTC m=+0.682363821 container start 89e77b8f00daed45e30629aef5555c38dc6a653976263cef60036bd21e2c6119 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-05T06:36:16.935 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:36:16.846Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-02-05T06:36:16.935 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:36:16.846Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-02-05T06:36:16.935 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:36:16.846Z caller=main.go:570 level=info host_details="(Linux 5.14.0-416.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Jan 31 22:52:48 UTC 2024 x86_64 smithi167 (none))" 2024-02-05T06:36:16.935 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:36:16.846Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-05T06:36:16.935 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:36:16.846Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-05T06:36:16.935 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:36:16.848Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-02-05T06:36:16.935 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:36:16.849Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-02-05T06:36:16.935 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:36:16.852Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-02-05T06:36:16.935 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:36:16.852Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-02-05T06:36:16.935 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:36:16.854Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-05T06:36:16.935 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:36:16.854Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.043µs 2024-02-05T06:36:16.936 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:36:16.854Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-05T06:36:16.936 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:36:16.855Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-02-05T06:36:16.936 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:36:16.855Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-02-05T06:36:16.936 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:36:16.855Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=20.653µs wal_replay_duration=814.62µs wbl_replay_duration=132ns total_replay_duration=854.157µs 2024-02-05T06:36:16.936 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:36:16.856Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-02-05T06:36:16.936 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:36:16.856Z caller=main.go:1029 level=info msg="TSDB started" 2024-02-05T06:36:16.936 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:36:16.856Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-05T06:36:16.936 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:36:16.880Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=23.459648ms db_storage=905ns remote_storage=1.573µs web_handler=478ns query_engine=829ns scrape=580.13µs scrape_sd=107.878µs notify=16.604µs notify_sd=16.727µs rules=22.3605ms tracing=7.544µs 2024-02-05T06:36:16.936 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:36:16.880Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-02-05T06:36:16.936 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:36:16.880Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-02-05T06:36:16.936 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:16 smithi167 bash[68545]: 89e77b8f00daed45e30629aef5555c38dc6a653976263cef60036bd21e2c6119 2024-02-05T06:36:16.936 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:36:16 smithi167 systemd[1]: Started Ceph prometheus.a for 9801e956-c3ef-11ee-95b6-87774f69a715. 2024-02-05T06:36:16.989 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:17.022 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:17.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:17.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:17.124 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:17.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:17.152 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:17.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:17.221 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-02-05T06:36:17.219532+0000 min lat: 0.0013072 max lat: 0.145154 avg lat: 0.0096013 2024-02-05T06:36:17.221 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T06:36:17.221 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 40 16 66631 66615 104.015 74.5625 0.0034827 0.0096013 2024-02-05T06:36:17.253 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:17.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:17.287 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:17.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:17.304 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:36:17] ENGINE Bus STOPPING 2024-02-05T06:36:17.304 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:36:17] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-05T06:36:17.304 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:36:17] ENGINE Bus STOPPED 2024-02-05T06:36:17.304 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:36:17] ENGINE Bus STARTING 2024-02-05T06:36:17.304 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:36:17] ENGINE Serving on http://:::9283 2024-02-05T06:36:17.304 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:36:17] ENGINE Bus STARTED 2024-02-05T06:36:17.304 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:36:17] ENGINE Bus STOPPING 2024-02-05T06:36:17.304 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:36:17] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-05T06:36:17.304 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:36:17] ENGINE Bus STOPPED 2024-02-05T06:36:17.304 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:36:17] ENGINE Bus STARTING 2024-02-05T06:36:17.304 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:36:17] ENGINE Serving on http://:::9283 2024-02-05T06:36:17.388 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:17.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:17.438 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:17.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:17.539 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:17.584 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:17.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:17.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:17.589 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-mon[29442]: pgmap v95: 148 pgs: 3 peering, 145 active+clean; 3.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.7 KiB/s rd, 97 MiB/s wr, 1.55k op/s; 18 B/s, 0 objects/s recovering 2024-02-05T06:36:17.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-mon[29442]: 6.5 scrub starts 2024-02-05T06:36:17.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-mon[29442]: 6.5 scrub ok 2024-02-05T06:36:17.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:17.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:17.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-05T06:36:17.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi028.front.sepia.ceph.com:9093"}]: dispatch 2024-02-05T06:36:17.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:17.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-05T06:36:17.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi167.front.sepia.ceph.com:9095"}]: dispatch 2024-02-05T06:36:17.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:17.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-05T06:36:17.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi167.front.sepia.ceph.com:3000"}]: dispatch 2024-02-05T06:36:17.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:17.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:36:17.591 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:36:17] ENGINE Bus STARTED 2024-02-05T06:36:17.591 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:36:17] ENGINE Bus STOPPING 2024-02-05T06:36:17.591 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:36:17] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-05T06:36:17.591 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:36:17] ENGINE Bus STOPPED 2024-02-05T06:36:17.591 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:36:17] ENGINE Bus STARTING 2024-02-05T06:36:17.591 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:36:17] ENGINE Serving on http://:::9283 2024-02-05T06:36:17.591 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:36:17] ENGINE Bus STARTED 2024-02-05T06:36:17.593 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-mon[37984]: pgmap v95: 148 pgs: 3 peering, 145 active+clean; 3.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.7 KiB/s rd, 97 MiB/s wr, 1.55k op/s; 18 B/s, 0 objects/s recovering 2024-02-05T06:36:17.593 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-mon[37984]: 6.5 scrub starts 2024-02-05T06:36:17.593 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-mon[37984]: 6.5 scrub ok 2024-02-05T06:36:17.593 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:17.593 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:17.593 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-05T06:36:17.593 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi028.front.sepia.ceph.com:9093"}]: dispatch 2024-02-05T06:36:17.593 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:17.593 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-05T06:36:17.593 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi167.front.sepia.ceph.com:9095"}]: dispatch 2024-02-05T06:36:17.593 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:17.593 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-05T06:36:17.593 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi167.front.sepia.ceph.com:3000"}]: dispatch 2024-02-05T06:36:17.593 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:17.594 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:17 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:36:17.686 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:17.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:17 smithi167 ceph-mon[32774]: pgmap v95: 148 pgs: 3 peering, 145 active+clean; 3.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.7 KiB/s rd, 97 MiB/s wr, 1.55k op/s; 18 B/s, 0 objects/s recovering 2024-02-05T06:36:17.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:17 smithi167 ceph-mon[32774]: 6.5 scrub starts 2024-02-05T06:36:17.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:17 smithi167 ceph-mon[32774]: 6.5 scrub ok 2024-02-05T06:36:17.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:17 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:17.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:17 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:17.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:17 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-05T06:36:17.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:17 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi028.front.sepia.ceph.com:9093"}]: dispatch 2024-02-05T06:36:17.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:17 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:17.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:17 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-05T06:36:17.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:17 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi167.front.sepia.ceph.com:9095"}]: dispatch 2024-02-05T06:36:17.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:17 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:17.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:17 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-05T06:36:17.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:17 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi167.front.sepia.ceph.com:3000"}]: dispatch 2024-02-05T06:36:17.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:17 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:17.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:17 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:36:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:17.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:17.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:17.834 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:17.873 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:17.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:17.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:17.975 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:18.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:18.019 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:18.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:18.120 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:18.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:18.150 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:18.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:18.221 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 41 16 67768 67752 103.212 71.0625 0.0134582 0.00968397 2024-02-05T06:36:18.252 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:18.291 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:18.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:18.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:18.393 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:18.432 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:18.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:18.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:18.534 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:18.565 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:18.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:18.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:18.577 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:18 smithi167 ceph-mon[32774]: 5.b scrub starts 2024-02-05T06:36:18.577 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:18 smithi167 ceph-mon[32774]: 5.b scrub ok 2024-02-05T06:36:18.577 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:18 smithi167 ceph-mon[32774]: from='mon.? v1:172.21.15.28:0/1757070553' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-05T06:36:18.577 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:18 smithi167 ceph-mon[32774]: from='mon.? v1:172.21.15.28:0/1757070553' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi028.front.sepia.ceph.com:9093"}]: dispatch 2024-02-05T06:36:18.577 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:18 smithi167 ceph-mon[32774]: from='mon.? v1:172.21.15.28:0/1757070553' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-05T06:36:18.577 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:18 smithi167 ceph-mon[32774]: from='mon.? v1:172.21.15.28:0/1757070553' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi167.front.sepia.ceph.com:9095"}]: dispatch 2024-02-05T06:36:18.577 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:18 smithi167 ceph-mon[32774]: from='mon.? v1:172.21.15.28:0/1757070553' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-05T06:36:18.577 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:18 smithi167 ceph-mon[32774]: from='mon.? v1:172.21.15.28:0/1757070553' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi167.front.sepia.ceph.com:3000"}]: dispatch 2024-02-05T06:36:18.577 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:18 smithi167 ceph-mon[32774]: 3.17 scrub starts 2024-02-05T06:36:18.578 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:18 smithi167 ceph-mon[32774]: 3.17 scrub ok 2024-02-05T06:36:18.578 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:18 smithi167 ceph-mon[32774]: 3.1e scrub starts 2024-02-05T06:36:18.578 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:18 smithi167 ceph-mon[32774]: 3.1e scrub ok 2024-02-05T06:36:18.578 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:18 smithi167 ceph-mon[32774]: 4.1 scrub starts 2024-02-05T06:36:18.578 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:18 smithi167 ceph-mon[32774]: 4.1 scrub ok 2024-02-05T06:36:18.667 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:18.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:18 smithi028 ceph-mon[29442]: 5.b scrub starts 2024-02-05T06:36:18.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:18 smithi028 ceph-mon[29442]: 5.b scrub ok 2024-02-05T06:36:18.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:18 smithi028 ceph-mon[29442]: from='mon.? v1:172.21.15.28:0/1757070553' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-05T06:36:18.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:18 smithi028 ceph-mon[29442]: from='mon.? v1:172.21.15.28:0/1757070553' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi028.front.sepia.ceph.com:9093"}]: dispatch 2024-02-05T06:36:18.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:18 smithi028 ceph-mon[29442]: from='mon.? v1:172.21.15.28:0/1757070553' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-05T06:36:18.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:18 smithi028 ceph-mon[29442]: from='mon.? v1:172.21.15.28:0/1757070553' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi167.front.sepia.ceph.com:9095"}]: dispatch 2024-02-05T06:36:18.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:18 smithi028 ceph-mon[29442]: from='mon.? v1:172.21.15.28:0/1757070553' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-05T06:36:18.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:18 smithi028 ceph-mon[29442]: from='mon.? v1:172.21.15.28:0/1757070553' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi167.front.sepia.ceph.com:3000"}]: dispatch 2024-02-05T06:36:18.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:18 smithi028 ceph-mon[29442]: 3.17 scrub starts 2024-02-05T06:36:18.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:18 smithi028 ceph-mon[29442]: 3.17 scrub ok 2024-02-05T06:36:18.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:18 smithi028 ceph-mon[29442]: 3.1e scrub starts 2024-02-05T06:36:18.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:18 smithi028 ceph-mon[29442]: 3.1e scrub ok 2024-02-05T06:36:18.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:18 smithi028 ceph-mon[29442]: 4.1 scrub starts 2024-02-05T06:36:18.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:18 smithi028 ceph-mon[29442]: 4.1 scrub ok 2024-02-05T06:36:18.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:18 smithi028 ceph-mon[37984]: 5.b scrub starts 2024-02-05T06:36:18.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:18 smithi028 ceph-mon[37984]: 5.b scrub ok 2024-02-05T06:36:18.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:18 smithi028 ceph-mon[37984]: from='mon.? v1:172.21.15.28:0/1757070553' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-05T06:36:18.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:18 smithi028 ceph-mon[37984]: from='mon.? v1:172.21.15.28:0/1757070553' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi028.front.sepia.ceph.com:9093"}]: dispatch 2024-02-05T06:36:18.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:18 smithi028 ceph-mon[37984]: from='mon.? v1:172.21.15.28:0/1757070553' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-05T06:36:18.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:18 smithi028 ceph-mon[37984]: from='mon.? v1:172.21.15.28:0/1757070553' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi167.front.sepia.ceph.com:9095"}]: dispatch 2024-02-05T06:36:18.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:18 smithi028 ceph-mon[37984]: from='mon.? v1:172.21.15.28:0/1757070553' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-05T06:36:18.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:18 smithi028 ceph-mon[37984]: from='mon.? v1:172.21.15.28:0/1757070553' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi167.front.sepia.ceph.com:3000"}]: dispatch 2024-02-05T06:36:18.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:18 smithi028 ceph-mon[37984]: 3.17 scrub starts 2024-02-05T06:36:18.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:18 smithi028 ceph-mon[37984]: 3.17 scrub ok 2024-02-05T06:36:18.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:18 smithi028 ceph-mon[37984]: 3.1e scrub starts 2024-02-05T06:36:18.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:18 smithi028 ceph-mon[37984]: 3.1e scrub ok 2024-02-05T06:36:18.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:18 smithi028 ceph-mon[37984]: 4.1 scrub starts 2024-02-05T06:36:18.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:18 smithi028 ceph-mon[37984]: 4.1 scrub ok 2024-02-05T06:36:18.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:18.699 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:18.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:18.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:18.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:18.851 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:18.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:18.953 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:19.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:19.008 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:19.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:19.109 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:19.130 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:36:19.131 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:36:19.136 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:19.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:19.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:19.221 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 42 16 68950 68934 102.513 73.875 0.0104073 0.00974938 2024-02-05T06:36:19.238 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:19.288 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:19.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:19.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:19.390 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:19.420 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:19.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:19.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:19.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:19 smithi028 ceph-mon[29442]: pgmap v96: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.4 KiB/s rd, 108 MiB/s wr, 1.73k op/s; 31 B/s, 1 objects/s recovering 2024-02-05T06:36:19.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:19 smithi028 ceph-mon[29442]: 3.11 scrub starts 2024-02-05T06:36:19.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:19 smithi028 ceph-mon[29442]: 3.11 scrub ok 2024-02-05T06:36:19.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:19 smithi028 ceph-mon[37984]: pgmap v96: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.4 KiB/s rd, 108 MiB/s wr, 1.73k op/s; 31 B/s, 1 objects/s recovering 2024-02-05T06:36:19.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:19 smithi028 ceph-mon[37984]: 3.11 scrub starts 2024-02-05T06:36:19.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:19 smithi028 ceph-mon[37984]: 3.11 scrub ok 2024-02-05T06:36:19.522 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:19.549 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:19.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:19.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:19.651 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:19.679 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:19.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:19.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:19.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:19 smithi167 ceph-mon[32774]: pgmap v96: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.4 KiB/s rd, 108 MiB/s wr, 1.73k op/s; 31 B/s, 1 objects/s recovering 2024-02-05T06:36:19.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:19 smithi167 ceph-mon[32774]: 3.11 scrub starts 2024-02-05T06:36:19.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:19 smithi167 ceph-mon[32774]: 3.11 scrub ok 2024-02-05T06:36:19.782 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:19.824 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:19.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:19.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:19.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:19.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:19.971 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:19.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:20.072 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:20.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:20.109 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:20.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:20.188 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:36:19 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a[80834]: ts=2024-02-05T06:36:19.760Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003131529s 2024-02-05T06:36:20.211 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:20.222 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 43 16 69976 69960 101.621 64.125 0.0155786 0.00983326 2024-02-05T06:36:20.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:20.270 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:20.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:20.371 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:20.399 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:20.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:20.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:20.501 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:20.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:20.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:20.639 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:20.666 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:20.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:20.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:20.768 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:20.795 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:20.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:20.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:20.897 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:20.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:20.942 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:20.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:21.043 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:21.086 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:21.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:21.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:21.188 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:21.194 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:20 smithi028 ceph-mon[37984]: 4.19 scrub starts 2024-02-05T06:36:21.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:21 smithi028 ceph-mon[37984]: 4.19 scrub ok 2024-02-05T06:36:21.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:21 smithi028 ceph-mon[37984]: osdmap e112: 8 total, 8 up, 7 in 2024-02-05T06:36:21.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:21 smithi028 ceph-mon[37984]: pgmap v98: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 45 MiB/s wr, 714 op/s; 27 B/s, 1 objects/s recovering 2024-02-05T06:36:21.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:20 smithi028 ceph-mon[29442]: 4.19 scrub starts 2024-02-05T06:36:21.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:21 smithi028 ceph-mon[29442]: 4.19 scrub ok 2024-02-05T06:36:21.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:21 smithi028 ceph-mon[29442]: osdmap e112: 8 total, 8 up, 7 in 2024-02-05T06:36:21.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:21 smithi028 ceph-mon[29442]: pgmap v98: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 45 MiB/s wr, 714 op/s; 27 B/s, 1 objects/s recovering 2024-02-05T06:36:21.221 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 44 16 70949 70933 100.694 60.8125 0.0167343 0.0099258 2024-02-05T06:36:21.227 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:21.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:21.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:21.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:20 smithi167 ceph-mon[32774]: 4.19 scrub starts 2024-02-05T06:36:21.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:20 smithi167 ceph-mon[32774]: 4.19 scrub ok 2024-02-05T06:36:21.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:20 smithi167 ceph-mon[32774]: osdmap e112: 8 total, 8 up, 7 in 2024-02-05T06:36:21.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:20 smithi167 ceph-mon[32774]: pgmap v98: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 639 B/s rd, 45 MiB/s wr, 714 op/s; 27 B/s, 1 objects/s recovering 2024-02-05T06:36:21.329 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:21.357 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:21.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:21.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:21.459 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:21.486 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:21.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:21.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:21.588 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:21.623 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:21.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:21.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:21.725 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:21.772 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:21.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:21.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:21.873 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:21.915 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:21.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:21.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:22.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:22.054 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:22.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:22.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:22.156 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:22.182 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:22.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:22.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:22.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:21 smithi028 ceph-mon[29442]: 4.1f scrub starts 2024-02-05T06:36:22.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:21 smithi028 ceph-mon[29442]: 4.1f scrub ok 2024-02-05T06:36:22.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:21 smithi028 ceph-mon[29442]: 5.16 scrub starts 2024-02-05T06:36:22.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:21 smithi028 ceph-mon[29442]: 5.16 scrub ok 2024-02-05T06:36:22.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:21 smithi028 ceph-mon[37984]: 4.1f scrub starts 2024-02-05T06:36:22.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:21 smithi028 ceph-mon[37984]: 4.1f scrub ok 2024-02-05T06:36:22.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:21 smithi028 ceph-mon[37984]: 5.16 scrub starts 2024-02-05T06:36:22.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:21 smithi028 ceph-mon[37984]: 5.16 scrub ok 2024-02-05T06:36:22.221 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 45 16 72072 72056 100.016 70.1875 0.0185388 0.00999344 2024-02-05T06:36:22.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:21 smithi167 ceph-mon[32774]: 4.1f scrub starts 2024-02-05T06:36:22.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:21 smithi167 ceph-mon[32774]: 4.1f scrub ok 2024-02-05T06:36:22.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:21 smithi167 ceph-mon[32774]: 5.16 scrub starts 2024-02-05T06:36:22.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:21 smithi167 ceph-mon[32774]: 5.16 scrub ok 2024-02-05T06:36:22.284 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:22.312 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:22.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:22.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:22.414 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:22.444 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:22.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:22.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:22.546 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:22.585 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:22.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:22.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:22.687 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:22.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:22.731 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:22.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:22.732 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:36:22 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:36:22.817 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:36:22 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:36:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:36:22.832 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:22.860 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:22.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:22.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:22.962 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:22.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:22.992 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:22.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:23.093 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:23.119 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:23.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:23.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:23.192 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:36:23.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:22 smithi028 ceph-mon[29442]: 5.d scrub starts 2024-02-05T06:36:23.192 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:22 smithi028 ceph-mon[29442]: 5.d scrub ok 2024-02-05T06:36:23.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:22 smithi028 ceph-mon[29442]: 6.9 scrub starts 2024-02-05T06:36:23.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:22 smithi028 ceph-mon[29442]: 6.9 scrub ok 2024-02-05T06:36:23.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:22 smithi028 ceph-mon[29442]: pgmap v99: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.40k op/s; 27 B/s, 1 objects/s recovering 2024-02-05T06:36:23.193 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:22 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:36:23.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:22 smithi028 ceph-mon[37984]: 5.d scrub starts 2024-02-05T06:36:23.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:22 smithi028 ceph-mon[37984]: 5.d scrub ok 2024-02-05T06:36:23.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:22 smithi028 ceph-mon[37984]: 6.9 scrub starts 2024-02-05T06:36:23.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:22 smithi028 ceph-mon[37984]: 6.9 scrub ok 2024-02-05T06:36:23.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:22 smithi028 ceph-mon[37984]: pgmap v99: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.40k op/s; 27 B/s, 1 objects/s recovering 2024-02-05T06:36:23.193 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:22 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:36:23.221 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:23.222 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 46 16 73196 73180 99.3681 70.25 0.0171942 0.0100575 2024-02-05T06:36:23.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:22 smithi167 ceph-mon[32774]: 5.d scrub starts 2024-02-05T06:36:23.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:22 smithi167 ceph-mon[32774]: 5.d scrub ok 2024-02-05T06:36:23.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:22 smithi167 ceph-mon[32774]: 6.9 scrub starts 2024-02-05T06:36:23.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:22 smithi167 ceph-mon[32774]: 6.9 scrub ok 2024-02-05T06:36:23.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:22 smithi167 ceph-mon[32774]: pgmap v99: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.40k op/s; 27 B/s, 1 objects/s recovering 2024-02-05T06:36:23.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:22 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:36:23.234 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:36:23.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:23.256 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:23.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:23.264 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:36:23.301 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:36:23.338 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:36:23.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:23.379 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:36:23.433 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:23.433 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:36:23.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:23.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:23.478 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:36:23.489 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:36:23.489 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:36:23.489 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:36:23.489 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:36:23.489 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:36:23.489 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:36:23.489 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:36:23.489 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:36:23.536 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:23.564 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:23.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:23.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:23.665 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:23.708 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:23.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:23.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:23.810 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:23.837 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:23.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:23.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:23.939 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:23.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:23.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:24.085 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:24.122 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:24.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:24.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:24.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:23 smithi028 ceph-mon[37984]: 6.4 scrub starts 2024-02-05T06:36:24.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:23 smithi028 ceph-mon[37984]: 6.4 scrub ok 2024-02-05T06:36:24.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:23 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2307189710' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:36:24.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:23 smithi028 ceph-mon[37984]: 4.16 scrub starts 2024-02-05T06:36:24.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:23 smithi028 ceph-mon[37984]: 4.16 scrub ok 2024-02-05T06:36:24.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:23 smithi028 ceph-mon[29442]: 6.4 scrub starts 2024-02-05T06:36:24.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:23 smithi028 ceph-mon[29442]: 6.4 scrub ok 2024-02-05T06:36:24.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:23 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2307189710' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:36:24.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:23 smithi028 ceph-mon[29442]: 4.16 scrub starts 2024-02-05T06:36:24.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:23 smithi028 ceph-mon[29442]: 4.16 scrub ok 2024-02-05T06:36:24.222 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 47 16 74274 74258 98.6876 67.375 0.00728906 0.010127 2024-02-05T06:36:24.224 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:24.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-05T06:36:24.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:23 smithi167 ceph-mon[32774]: 6.4 scrub starts 2024-02-05T06:36:24.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:23 smithi167 ceph-mon[32774]: 6.4 scrub ok 2024-02-05T06:36:24.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:23 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2307189710' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:36:24.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:23 smithi167 ceph-mon[32774]: 4.16 scrub starts 2024-02-05T06:36:24.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:23 smithi167 ceph-mon[32774]: 4.16 scrub ok 2024-02-05T06:36:24.266 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:24.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:24.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:24.369 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:24.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:24.430 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:24.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:24.531 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:24.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:24.561 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:24.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:24.662 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:24.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:24.717 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:24.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:24.818 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:24.845 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:24.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:24.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:24.947 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:24.973 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:24.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:24.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:25.075 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:25.118 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:25.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:25.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:25.120 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:24 smithi028 ceph-mon[37984]: 2.0 scrub starts 2024-02-05T06:36:25.120 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:24 smithi028 ceph-mon[37984]: 2.0 scrub ok 2024-02-05T06:36:25.120 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:24 smithi028 ceph-mon[37984]: pgmap v100: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 76 MiB/s wr, 1.21k op/s; 23 B/s, 1 objects/s recovering 2024-02-05T06:36:25.121 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:24 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:25.121 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:24 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:25.121 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:24 smithi028 ceph-mon[37984]: osdmap e113: 8 total, 8 up, 7 in 2024-02-05T06:36:25.121 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:24 smithi028 ceph-mon[29442]: 2.0 scrub starts 2024-02-05T06:36:25.121 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:24 smithi028 ceph-mon[29442]: 2.0 scrub ok 2024-02-05T06:36:25.121 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:24 smithi028 ceph-mon[29442]: pgmap v100: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 76 MiB/s wr, 1.21k op/s; 23 B/s, 1 objects/s recovering 2024-02-05T06:36:25.121 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:24 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:25.121 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:24 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:25.121 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:24 smithi028 ceph-mon[29442]: osdmap e113: 8 total, 8 up, 7 in 2024-02-05T06:36:25.220 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:25.223 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 48 16 75276 75260 97.9366 62.625 0.010222 0.0102047 2024-02-05T06:36:25.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:24 smithi167 ceph-mon[32774]: 2.0 scrub starts 2024-02-05T06:36:25.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:24 smithi167 ceph-mon[32774]: 2.0 scrub ok 2024-02-05T06:36:25.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:24 smithi167 ceph-mon[32774]: pgmap v100: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.1 KiB/s rd, 76 MiB/s wr, 1.21k op/s; 23 B/s, 1 objects/s recovering 2024-02-05T06:36:25.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:24 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:25.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:24 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:25.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:24 smithi167 ceph-mon[32774]: osdmap e113: 8 total, 8 up, 7 in 2024-02-05T06:36:25.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:25.267 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:25.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:25.368 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:25.413 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:25.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:25.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:25.515 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:25.543 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:25.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:25.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:25.644 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:25.671 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:25.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:25.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:25.773 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:25.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:25.813 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:25.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:25.914 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:25.943 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:25.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:25.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:25.952 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:36:25.959 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:36:25.967 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-05T06:36:25.982 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-02-05T06:36:26.047 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:26.074 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:26.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:26.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:26.176 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:26.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:25 smithi028 ceph-mon[37984]: 3.1 scrub starts 2024-02-05T06:36:26.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:25 smithi028 ceph-mon[37984]: 3.1 scrub ok 2024-02-05T06:36:26.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:25 smithi028 ceph-mon[37984]: 2.1 scrub starts 2024-02-05T06:36:26.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:25 smithi028 ceph-mon[37984]: 2.1 scrub ok 2024-02-05T06:36:26.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:25 smithi028 ceph-mon[37984]: 5.f scrub starts 2024-02-05T06:36:26.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:25 smithi028 ceph-mon[37984]: 5.f scrub ok 2024-02-05T06:36:26.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:25 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3496266202' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:36:26.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:25 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:36:26.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:25 smithi028 ceph-mon[29442]: 3.1 scrub starts 2024-02-05T06:36:26.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:25 smithi028 ceph-mon[29442]: 3.1 scrub ok 2024-02-05T06:36:26.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:25 smithi028 ceph-mon[29442]: 2.1 scrub starts 2024-02-05T06:36:26.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:25 smithi028 ceph-mon[29442]: 2.1 scrub ok 2024-02-05T06:36:26.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:25 smithi028 ceph-mon[29442]: 5.f scrub starts 2024-02-05T06:36:26.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:25 smithi028 ceph-mon[29442]: 5.f scrub ok 2024-02-05T06:36:26.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:25 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3496266202' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:36:26.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:25 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:36:26.218 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:26.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:26.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:26.223 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 49 16 76254 76238 97.1856 61.125 0.019492 0.0102844 2024-02-05T06:36:26.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:25 smithi167 ceph-mon[32774]: 3.1 scrub starts 2024-02-05T06:36:26.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:25 smithi167 ceph-mon[32774]: 3.1 scrub ok 2024-02-05T06:36:26.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:25 smithi167 ceph-mon[32774]: 2.1 scrub starts 2024-02-05T06:36:26.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:25 smithi167 ceph-mon[32774]: 2.1 scrub ok 2024-02-05T06:36:26.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:25 smithi167 ceph-mon[32774]: 5.f scrub starts 2024-02-05T06:36:26.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:25 smithi167 ceph-mon[32774]: 5.f scrub ok 2024-02-05T06:36:26.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:25 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3496266202' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:36:26.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:25 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:36:26.320 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:26.349 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:26.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:26.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:26.452 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:26.482 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:26.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:26.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:26.584 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:26.612 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:26.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:26.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:26.714 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:26.743 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:26.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:26.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:26.845 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:26.873 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:26.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:26.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:26.889 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:36:26.974 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:27.043 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:27.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:27.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:27.115 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:26 smithi028 ceph-mon[37984]: 5.17 scrub starts 2024-02-05T06:36:27.115 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:26 smithi028 ceph-mon[37984]: 5.17 scrub ok 2024-02-05T06:36:27.115 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:26 smithi028 ceph-mon[37984]: from='client.44792 v1:172.21.15.28:0/4240462423' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:36:27.115 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:26 smithi028 ceph-mon[37984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:36:27.115 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:26 smithi028 ceph-mon[37984]: Cluster is now healthy 2024-02-05T06:36:27.116 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:26 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3496266202' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:36:27.116 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:26 smithi028 ceph-mon[37984]: osdmap e114: 8 total, 8 up, 7 in 2024-02-05T06:36:27.116 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:26 smithi028 ceph-mon[37984]: pgmap v103: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.6 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-02-05T06:36:27.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:26 smithi028 ceph-mon[29442]: 5.17 scrub starts 2024-02-05T06:36:27.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:26 smithi028 ceph-mon[29442]: 5.17 scrub ok 2024-02-05T06:36:27.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:26 smithi028 ceph-mon[29442]: from='client.44792 v1:172.21.15.28:0/4240462423' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:36:27.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:26 smithi028 ceph-mon[29442]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:36:27.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:26 smithi028 ceph-mon[29442]: Cluster is now healthy 2024-02-05T06:36:27.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:26 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3496266202' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:36:27.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:26 smithi028 ceph-mon[29442]: osdmap e114: 8 total, 8 up, 7 in 2024-02-05T06:36:27.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:26 smithi028 ceph-mon[29442]: pgmap v103: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.6 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-02-05T06:36:27.145 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:27.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:27.202 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:27.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:27.223 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 50 16 77379 77363 96.6479 70.3125 0.0103707 0.0103407 2024-02-05T06:36:27.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:26 smithi167 ceph-mon[32774]: 5.17 scrub starts 2024-02-05T06:36:27.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:26 smithi167 ceph-mon[32774]: 5.17 scrub ok 2024-02-05T06:36:27.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:26 smithi167 ceph-mon[32774]: from='client.44792 v1:172.21.15.28:0/4240462423' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:36:27.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:26 smithi167 ceph-mon[32774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:36:27.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:26 smithi167 ceph-mon[32774]: Cluster is now healthy 2024-02-05T06:36:27.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:26 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3496266202' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:36:27.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:26 smithi167 ceph-mon[32774]: osdmap e114: 8 total, 8 up, 7 in 2024-02-05T06:36:27.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:26 smithi167 ceph-mon[32774]: pgmap v103: 148 pgs: 148 active+clean; 4.5 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.6 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-02-05T06:36:27.303 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:27.332 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:27.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:27.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:27.433 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:27.474 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:27.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:27.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:27.576 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:27.602 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:27.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:27.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:27.704 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:27.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:27.748 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:27.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:27.848 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:27.879 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:27.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:27.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:27.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:28.021 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:28.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:28.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:28.123 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:28.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:28.167 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:28.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:28.223 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 51 16 78564 78548 96.2051 74.0625 0.00778958 0.0103892 2024-02-05T06:36:28.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:27 smithi167 ceph-mon[32774]: 6.1d scrub starts 2024-02-05T06:36:28.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:27 smithi167 ceph-mon[32774]: 6.1d scrub ok 2024-02-05T06:36:28.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:27 smithi167 ceph-mon[32774]: 4.b scrub starts 2024-02-05T06:36:28.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:27 smithi167 ceph-mon[32774]: 4.b scrub ok 2024-02-05T06:36:28.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:27 smithi167 ceph-mon[32774]: 5.14 scrub starts 2024-02-05T06:36:28.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:27 smithi167 ceph-mon[32774]: 5.14 scrub ok 2024-02-05T06:36:28.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:27 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:36:28.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:27 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:36:28.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:27 smithi167 ceph-mon[32774]: osdmap e115: 8 total, 8 up, 7 in 2024-02-05T06:36:28.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:27 smithi167 ceph-mon[32774]: 3.14 scrub starts 2024-02-05T06:36:28.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:27 smithi167 ceph-mon[32774]: 3.14 scrub ok 2024-02-05T06:36:28.268 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:28.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:28.300 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:28.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:28.399 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:27 smithi028 ceph-mon[37984]: 6.1d scrub starts 2024-02-05T06:36:28.399 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:27 smithi028 ceph-mon[37984]: 6.1d scrub ok 2024-02-05T06:36:28.400 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:27 smithi028 ceph-mon[37984]: 4.b scrub starts 2024-02-05T06:36:28.400 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:27 smithi028 ceph-mon[37984]: 4.b scrub ok 2024-02-05T06:36:28.400 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:27 smithi028 ceph-mon[37984]: 5.14 scrub starts 2024-02-05T06:36:28.400 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:27 smithi028 ceph-mon[37984]: 5.14 scrub ok 2024-02-05T06:36:28.400 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:27 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:36:28.400 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:27 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:36:28.400 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:27 smithi028 ceph-mon[37984]: osdmap e115: 8 total, 8 up, 7 in 2024-02-05T06:36:28.400 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:27 smithi028 ceph-mon[37984]: 3.14 scrub starts 2024-02-05T06:36:28.400 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:27 smithi028 ceph-mon[37984]: 3.14 scrub ok 2024-02-05T06:36:28.400 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:28.401 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:27 smithi028 ceph-mon[29442]: 6.1d scrub starts 2024-02-05T06:36:28.401 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:27 smithi028 ceph-mon[29442]: 6.1d scrub ok 2024-02-05T06:36:28.401 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:27 smithi028 ceph-mon[29442]: 4.b scrub starts 2024-02-05T06:36:28.401 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:27 smithi028 ceph-mon[29442]: 4.b scrub ok 2024-02-05T06:36:28.401 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:27 smithi028 ceph-mon[29442]: 5.14 scrub starts 2024-02-05T06:36:28.402 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:27 smithi028 ceph-mon[29442]: 5.14 scrub ok 2024-02-05T06:36:28.402 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:27 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:36:28.402 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:27 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:36:28.402 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:27 smithi028 ceph-mon[29442]: osdmap e115: 8 total, 8 up, 7 in 2024-02-05T06:36:28.402 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:27 smithi028 ceph-mon[29442]: 3.14 scrub starts 2024-02-05T06:36:28.402 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:27 smithi028 ceph-mon[29442]: 3.14 scrub ok 2024-02-05T06:36:28.445 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:28.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:28.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:28.447 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-02-05T06:36:24.927779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4647295324,"num_objects":71101,"num_object_clones":0,"num_object_copies":213303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71101,"num_whiteouts":0,"num_read":1086,"num_read_kb":932,"num_write":71518,"num_write_kb":4540253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":1774068,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70562,"ondisk_log_size":70562,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":13996620,"kb_used_data":13542392,"kb_used_omap":10,"kb_used_meta":454133,"kb_avail":642219444,"statfs":{"total":671965249536,"available":657632710656,"internally_reserved":0,"allocated":13867409408,"data_stored":13864250336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11156,"internal_metadata":465032300},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":548405248,"num_objects":8368,"num_object_clones":0,"num_object_copies":25104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8368,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8368,"num_write_kb":535552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.745684"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":47,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.931450+0000","last_change":"2024-02-05T06:35:24.946381+0000","last_active":"2024-02-05T06:36:19.931450+0000","last_peered":"2024-02-05T06:36:19.931450+0000","last_clean":"2024-02-05T06:36:19.931450+0000","last_became_active":"2024-02-05T06:35:24.946016+0000","last_became_peered":"2024-02-05T06:35:24.946016+0000","last_unstale":"2024-02-05T06:36:19.931450+0000","last_undegraded":"2024-02-05T06:36:19.931450+0000","last_fullsized":"2024-02-05T06:36:19.931450+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:55.600744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:55.600744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:20.019533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":103,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.945557+0000","last_change":"2024-02-05T06:35:06.525091+0000","last_active":"2024-02-05T06:36:19.945557+0000","last_peered":"2024-02-05T06:36:19.945557+0000","last_clean":"2024-02-05T06:36:19.945557+0000","last_became_active":"2024-02-05T06:33:33.585848+0000","last_became_peered":"2024-02-05T06:33:33.585848+0000","last_unstale":"2024-02-05T06:36:19.945557+0000","last_undegraded":"2024-02-05T06:36:19.945557+0000","last_fullsized":"2024-02-05T06:36:19.945557+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:06.525012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:06.525012+0000","last_clean_scrub_stamp":"2024-02-05T06:35:06.525012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:25.190619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":112,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.937104+0000","last_change":"2024-02-05T06:36:09.154584+0000","last_active":"2024-02-05T06:36:19.937104+0000","last_peered":"2024-02-05T06:36:19.937104+0000","last_clean":"2024-02-05T06:36:19.937104+0000","last_became_active":"2024-02-05T06:33:29.549592+0000","last_became_peered":"2024-02-05T06:33:29.549592+0000","last_unstale":"2024-02-05T06:36:19.937104+0000","last_undegraded":"2024-02-05T06:36:19.937104+0000","last_fullsized":"2024-02-05T06:36:19.937104+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:09.154442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:09.154442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:14.894164+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":91,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.930351+0000","last_change":"2024-02-05T06:35:24.972897+0000","last_active":"2024-02-05T06:36:19.930351+0000","last_peered":"2024-02-05T06:36:19.930351+0000","last_clean":"2024-02-05T06:36:19.930351+0000","last_became_active":"2024-02-05T06:35:24.939708+0000","last_became_peered":"2024-02-05T06:35:24.939708+0000","last_unstale":"2024-02-05T06:36:19.930351+0000","last_undegraded":"2024-02-05T06:36:19.930351+0000","last_fullsized":"2024-02-05T06:36:19.930351+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:35:01.601838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:35:01.601838+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":7,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":100,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.945647+0000","last_change":"2024-02-05T06:36:12.220215+0000","last_active":"2024-02-05T06:36:19.945647+0000","last_peered":"2024-02-05T06:36:19.945647+0000","last_clean":"2024-02-05T06:36:19.945647+0000","last_became_active":"2024-02-05T06:33:35.667778+0000","last_became_peered":"2024-02-05T06:33:35.667778+0000","last_unstale":"2024-02-05T06:36:19.945647+0000","last_undegraded":"2024-02-05T06:36:19.945647+0000","last_fullsized":"2024-02-05T06:36:19.945647+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:12.220090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:12.220090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:20.586314+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":46,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.930252+0000","last_change":"2024-02-05T06:35:24.929640+0000","last_active":"2024-02-05T06:36:19.930252+0000","last_peered":"2024-02-05T06:36:19.930252+0000","last_clean":"2024-02-05T06:36:19.930252+0000","last_became_active":"2024-02-05T06:35:24.929339+0000","last_became_peered":"2024-02-05T06:35:24.929339+0000","last_unstale":"2024-02-05T06:36:19.930252+0000","last_undegraded":"2024-02-05T06:36:19.930252+0000","last_fullsized":"2024-02-05T06:36:19.930252+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:13.639311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:35:13.639311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:40.444462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":120,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.939394+0000","last_change":"2024-02-05T06:36:17.296405+0000","last_active":"2024-02-05T06:36:19.939394+0000","last_peered":"2024-02-05T06:36:19.939394+0000","last_clean":"2024-02-05T06:36:19.939394+0000","last_became_active":"2024-02-05T06:33:29.553864+0000","last_became_peered":"2024-02-05T06:33:29.553864+0000","last_unstale":"2024-02-05T06:36:19.939394+0000","last_undegraded":"2024-02-05T06:36:19.939394+0000","last_fullsized":"2024-02-05T06:36:19.939394+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:17.296252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:17.296252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:27.660356+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"88'22","reported_seq":148,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.939666+0000","last_change":"2024-02-05T06:36:19.337617+0000","last_active":"2024-02-05T06:36:19.939666+0000","last_peered":"2024-02-05T06:36:19.939666+0000","last_clean":"2024-02-05T06:36:19.939666+0000","last_became_active":"2024-02-05T06:33:31.572544+0000","last_became_peered":"2024-02-05T06:33:31.572544+0000","last_unstale":"2024-02-05T06:36:19.939666+0000","last_undegraded":"2024-02-05T06:36:19.939666+0000","last_fullsized":"2024-02-05T06:36:19.939666+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'22","last_scrub_stamp":"2024-02-05T06:36:19.337341+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:34:35.832368+0000","last_clean_scrub_stamp":"2024-02-05T06:36:19.337341+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:29.086209+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":100,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.931931+0000","last_change":"2024-02-05T06:35:20.603134+0000","last_active":"2024-02-05T06:36:19.931931+0000","last_peered":"2024-02-05T06:36:19.931931+0000","last_clean":"2024-02-05T06:36:19.931931+0000","last_became_active":"2024-02-05T06:33:35.594860+0000","last_became_peered":"2024-02-05T06:33:35.594860+0000","last_unstale":"2024-02-05T06:36:19.931931+0000","last_undegraded":"2024-02-05T06:36:19.931931+0000","last_fullsized":"2024-02-05T06:36:19.931931+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:20.603044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:20.603044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:27.778604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.932201+0000","last_change":"2024-02-05T06:36:07.749745+0000","last_active":"2024-02-05T06:36:19.932201+0000","last_peered":"2024-02-05T06:36:19.932201+0000","last_clean":"2024-02-05T06:36:19.932201+0000","last_became_active":"2024-02-05T06:33:33.585459+0000","last_became_peered":"2024-02-05T06:33:33.585459+0000","last_unstale":"2024-02-05T06:36:19.932201+0000","last_undegraded":"2024-02-05T06:36:19.932201+0000","last_fullsized":"2024-02-05T06:36:19.932201+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:07.749639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:07.749639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:10.266182+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":124,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.932780+0000","last_change":"2024-02-05T06:36:08.762314+0000","last_active":"2024-02-05T06:36:19.932780+0000","last_peered":"2024-02-05T06:36:19.932780+0000","last_clean":"2024-02-05T06:36:19.932780+0000","last_became_active":"2024-02-05T06:35:24.944607+0000","last_became_peered":"2024-02-05T06:35:24.944607+0000","last_unstale":"2024-02-05T06:36:19.932780+0000","last_undegraded":"2024-02-05T06:36:19.932780+0000","last_fullsized":"2024-02-05T06:36:19.932780+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:08.762241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:44.081318+0000","last_clean_scrub_stamp":"2024-02-05T06:36:08.762241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:12.272105+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"92'13","reported_seq":133,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.947755+0000","last_change":"2024-02-05T06:35:24.973365+0000","last_active":"2024-02-05T06:36:19.947755+0000","last_peered":"2024-02-05T06:36:19.947755+0000","last_clean":"2024-02-05T06:36:19.947755+0000","last_became_active":"2024-02-05T06:35:24.948962+0000","last_became_peered":"2024-02-05T06:35:24.948962+0000","last_unstale":"2024-02-05T06:36:19.947755+0000","last_undegraded":"2024-02-05T06:36:19.947755+0000","last_fullsized":"2024-02-05T06:36:19.947755+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'13","last_scrub_stamp":"2024-02-05T06:34:46.542893+0000","last_deep_scrub":"92'13","last_deep_scrub_stamp":"2024-02-05T06:34:46.542893+0000","last_clean_scrub_stamp":"2024-02-05T06:34:46.542893+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.931371+0000","last_change":"2024-02-05T06:35:21.998937+0000","last_active":"2024-02-05T06:36:19.931371+0000","last_peered":"2024-02-05T06:36:19.931371+0000","last_clean":"2024-02-05T06:36:19.931371+0000","last_became_active":"2024-02-05T06:33:35.600399+0000","last_became_peered":"2024-02-05T06:33:35.600399+0000","last_unstale":"2024-02-05T06:36:19.931371+0000","last_undegraded":"2024-02-05T06:36:19.931371+0000","last_fullsized":"2024-02-05T06:36:19.931371+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:21.998850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:21.998850+0000","last_clean_scrub_stamp":"2024-02-05T06:35:21.998850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:43.487447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.926947+0000","last_change":"2024-02-05T06:35:24.944663+0000","last_active":"2024-02-05T06:36:19.926947+0000","last_peered":"2024-02-05T06:36:19.926947+0000","last_clean":"2024-02-05T06:36:19.926947+0000","last_became_active":"2024-02-05T06:35:24.944329+0000","last_became_peered":"2024-02-05T06:35:24.944329+0000","last_unstale":"2024-02-05T06:36:19.926947+0000","last_undegraded":"2024-02-05T06:36:19.926947+0000","last_fullsized":"2024-02-05T06:36:19.926947+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:53.073710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:53.073710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":116,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.932420+0000","last_change":"2024-02-05T06:35:24.946317+0000","last_active":"2024-02-05T06:36:19.932420+0000","last_peered":"2024-02-05T06:36:19.932420+0000","last_clean":"2024-02-05T06:36:19.932420+0000","last_became_active":"2024-02-05T06:35:24.945985+0000","last_became_peered":"2024-02-05T06:35:24.945985+0000","last_unstale":"2024-02-05T06:36:19.932420+0000","last_undegraded":"2024-02-05T06:36:19.932420+0000","last_fullsized":"2024-02-05T06:36:19.932420+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:50.055245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:50.055245+0000","last_clean_scrub_stamp":"2024-02-05T06:34:50.055245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":81,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.945370+0000","last_change":"2024-02-05T06:35:24.972322+0000","last_active":"2024-02-05T06:36:19.945370+0000","last_peered":"2024-02-05T06:36:19.945370+0000","last_clean":"2024-02-05T06:36:19.945370+0000","last_became_active":"2024-02-05T06:35:24.947925+0000","last_became_peered":"2024-02-05T06:35:24.947925+0000","last_unstale":"2024-02-05T06:36:19.945370+0000","last_undegraded":"2024-02-05T06:36:19.945370+0000","last_fullsized":"2024-02-05T06:36:19.945370+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-02-05T06:34:53.638704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:53.638704+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":47,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.931118+0000","last_change":"2024-02-05T06:35:24.938509+0000","last_active":"2024-02-05T06:36:19.931118+0000","last_peered":"2024-02-05T06:36:19.931118+0000","last_clean":"2024-02-05T06:36:19.931118+0000","last_became_active":"2024-02-05T06:35:24.938214+0000","last_became_peered":"2024-02-05T06:35:24.938214+0000","last_unstale":"2024-02-05T06:36:19.931118+0000","last_undegraded":"2024-02-05T06:36:19.931118+0000","last_fullsized":"2024-02-05T06:36:19.931118+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:45.573432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:45.573432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":109,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.940377+0000","last_change":"2024-02-05T06:35:24.933350+0000","last_active":"2024-02-05T06:36:19.940377+0000","last_peered":"2024-02-05T06:36:19.940377+0000","last_clean":"2024-02-05T06:36:19.940377+0000","last_became_active":"2024-02-05T06:35:24.932567+0000","last_became_peered":"2024-02-05T06:35:24.932567+0000","last_unstale":"2024-02-05T06:36:19.940377+0000","last_undegraded":"2024-02-05T06:36:19.940377+0000","last_fullsized":"2024-02-05T06:36:19.940377+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:16.531347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:35:16.531347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:28.088364+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"66'1","reported_seq":128,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.935249+0000","last_change":"2024-02-05T06:35:24.954933+0000","last_active":"2024-02-05T06:36:19.935249+0000","last_peered":"2024-02-05T06:36:19.935249+0000","last_clean":"2024-02-05T06:36:19.935249+0000","last_became_active":"2024-02-05T06:35:24.934095+0000","last_became_peered":"2024-02-05T06:35:24.934095+0000","last_unstale":"2024-02-05T06:36:19.935249+0000","last_undegraded":"2024-02-05T06:36:19.935249+0000","last_fullsized":"2024-02-05T06:36:19.935249+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:35:22.640782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:35:22.640782+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:47.527372+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"88'22","reported_seq":146,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.922764+0000","last_change":"2024-02-05T06:36:10.714443+0000","last_active":"2024-02-05T06:36:19.922764+0000","last_peered":"2024-02-05T06:36:19.922764+0000","last_clean":"2024-02-05T06:36:19.922764+0000","last_became_active":"2024-02-05T06:33:31.569763+0000","last_became_peered":"2024-02-05T06:33:31.569763+0000","last_unstale":"2024-02-05T06:36:19.922764+0000","last_undegraded":"2024-02-05T06:36:19.922764+0000","last_fullsized":"2024-02-05T06:36:19.922764+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'22","last_scrub_stamp":"2024-02-05T06:36:10.714308+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:34:35.234636+0000","last_clean_scrub_stamp":"2024-02-05T06:36:10.714308+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:16.856897+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":100,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.960551+0000","last_change":"2024-02-05T06:35:05.077348+0000","last_active":"2024-02-05T06:36:19.960551+0000","last_peered":"2024-02-05T06:36:19.960551+0000","last_clean":"2024-02-05T06:36:19.960551+0000","last_became_active":"2024-02-05T06:33:35.664852+0000","last_became_peered":"2024-02-05T06:33:35.664852+0000","last_unstale":"2024-02-05T06:36:19.960551+0000","last_undegraded":"2024-02-05T06:36:19.960551+0000","last_fullsized":"2024-02-05T06:36:19.960551+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:05.077268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:05.077268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:21.542392+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":46,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.940050+0000","last_change":"2024-02-05T06:35:24.948760+0000","last_active":"2024-02-05T06:36:19.940050+0000","last_peered":"2024-02-05T06:36:19.940050+0000","last_clean":"2024-02-05T06:36:19.940050+0000","last_became_active":"2024-02-05T06:35:24.948472+0000","last_became_peered":"2024-02-05T06:35:24.948472+0000","last_unstale":"2024-02-05T06:36:19.940050+0000","last_undegraded":"2024-02-05T06:36:19.940050+0000","last_fullsized":"2024-02-05T06:36:19.940050+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:05.580971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:35:05.580971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:26.200012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":54,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.943760+0000","last_change":"2024-02-05T06:36:12.270896+0000","last_active":"2024-02-05T06:36:19.943760+0000","last_peered":"2024-02-05T06:36:19.943760+0000","last_clean":"2024-02-05T06:36:19.943760+0000","last_became_active":"2024-02-05T06:35:24.931114+0000","last_became_peered":"2024-02-05T06:35:24.931114+0000","last_unstale":"2024-02-05T06:36:19.943760+0000","last_undegraded":"2024-02-05T06:36:19.943760+0000","last_fullsized":"2024-02-05T06:36:19.943760+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:12.270778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:12.270778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:42.058255+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":167,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.946797+0000","last_change":"2024-02-05T06:35:24.966381+0000","last_active":"2024-02-05T06:36:19.946797+0000","last_peered":"2024-02-05T06:36:19.946797+0000","last_clean":"2024-02-05T06:36:19.946797+0000","last_became_active":"2024-02-05T06:35:24.946406+0000","last_became_peered":"2024-02-05T06:35:24.946406+0000","last_unstale":"2024-02-05T06:36:19.946797+0000","last_undegraded":"2024-02-05T06:36:19.946797+0000","last_fullsized":"2024-02-05T06:36:19.946797+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:34:59.999726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:59.999726+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:29.017474+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"73'1","reported_seq":101,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.927633+0000","last_change":"2024-02-05T06:35:19.985742+0000","last_active":"2024-02-05T06:36:19.927633+0000","last_peered":"2024-02-05T06:36:19.927633+0000","last_clean":"2024-02-05T06:36:19.927633+0000","last_became_active":"2024-02-05T06:33:35.598501+0000","last_became_peered":"2024-02-05T06:33:35.598501+0000","last_unstale":"2024-02-05T06:36:19.927633+0000","last_undegraded":"2024-02-05T06:36:19.927633+0000","last_fullsized":"2024-02-05T06:36:19.927633+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-02-05T06:35:19.985656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:19.985656+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:42.023888+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"95'9","reported_seq":163,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.932085+0000","last_change":"2024-02-05T06:35:24.943457+0000","last_active":"2024-02-05T06:36:19.932085+0000","last_peered":"2024-02-05T06:36:19.932085+0000","last_clean":"2024-02-05T06:36:19.932085+0000","last_became_active":"2024-02-05T06:35:24.925670+0000","last_became_peered":"2024-02-05T06:35:24.925670+0000","last_unstale":"2024-02-05T06:36:19.932085+0000","last_undegraded":"2024-02-05T06:36:19.932085+0000","last_fullsized":"2024-02-05T06:36:19.932085+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:34:55.312853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:55.312853+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":130,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.945478+0000","last_change":"2024-02-05T06:36:10.257515+0000","last_active":"2024-02-05T06:36:19.945478+0000","last_peered":"2024-02-05T06:36:19.945478+0000","last_clean":"2024-02-05T06:36:19.945478+0000","last_became_active":"2024-02-05T06:35:24.942726+0000","last_became_peered":"2024-02-05T06:35:24.942726+0000","last_unstale":"2024-02-05T06:36:19.945478+0000","last_undegraded":"2024-02-05T06:36:19.945478+0000","last_fullsized":"2024-02-05T06:36:19.945478+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:36:10.257008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:10.257008+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:21.065817+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"92'15","reported_seq":128,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.942299+0000","last_change":"2024-02-05T06:34:50.718858+0000","last_active":"2024-02-05T06:36:19.942299+0000","last_peered":"2024-02-05T06:36:19.942299+0000","last_clean":"2024-02-05T06:36:19.942299+0000","last_became_active":"2024-02-05T06:33:31.666481+0000","last_became_peered":"2024-02-05T06:33:31.666481+0000","last_unstale":"2024-02-05T06:36:19.942299+0000","last_undegraded":"2024-02-05T06:36:19.942299+0000","last_fullsized":"2024-02-05T06:36:19.942299+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'15","last_scrub_stamp":"2024-02-05T06:34:50.718768+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-02-05T06:34:50.718768+0000","last_clean_scrub_stamp":"2024-02-05T06:34:50.718768+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":105,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.940559+0000","last_change":"2024-02-05T06:35:24.933602+0000","last_active":"2024-02-05T06:36:19.940559+0000","last_peered":"2024-02-05T06:36:19.940559+0000","last_clean":"2024-02-05T06:36:19.940559+0000","last_became_active":"2024-02-05T06:35:24.933003+0000","last_became_peered":"2024-02-05T06:35:24.933003+0000","last_unstale":"2024-02-05T06:36:19.940559+0000","last_undegraded":"2024-02-05T06:36:19.940559+0000","last_fullsized":"2024-02-05T06:36:19.940559+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:55.554581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:55.554581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.939067+0000","last_change":"2024-02-05T06:36:12.175207+0000","last_active":"2024-02-05T06:36:19.939067+0000","last_peered":"2024-02-05T06:36:19.939067+0000","last_clean":"2024-02-05T06:36:19.939067+0000","last_became_active":"2024-02-05T06:36:12.174876+0000","last_became_peered":"2024-02-05T06:36:12.174876+0000","last_unstale":"2024-02-05T06:36:19.939067+0000","last_undegraded":"2024-02-05T06:36:19.939067+0000","last_fullsized":"2024-02-05T06:36:19.939067+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:08.713894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:08.713894+0000","last_clean_scrub_stamp":"2024-02-05T06:35:08.713894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:34.662477+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":112,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.946237+0000","last_change":"2024-02-05T06:35:07.017685+0000","last_active":"2024-02-05T06:36:19.946237+0000","last_peered":"2024-02-05T06:36:19.946237+0000","last_clean":"2024-02-05T06:36:19.946237+0000","last_became_active":"2024-02-05T06:33:29.550347+0000","last_became_peered":"2024-02-05T06:33:29.550347+0000","last_unstale":"2024-02-05T06:36:19.946237+0000","last_undegraded":"2024-02-05T06:36:19.946237+0000","last_fullsized":"2024-02-05T06:36:19.946237+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:07.017572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:35:07.017572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:24.011901+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"92'16","reported_seq":139,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:20.685161+0000","last_change":"2024-02-05T06:36:20.685208+0000","last_active":"2024-02-05T06:36:20.685161+0000","last_peered":"2024-02-05T06:36:20.685161+0000","last_clean":"2024-02-05T06:36:20.685161+0000","last_became_active":"2024-02-05T06:33:31.573088+0000","last_became_peered":"2024-02-05T06:33:31.573088+0000","last_unstale":"2024-02-05T06:36:20.685161+0000","last_undegraded":"2024-02-05T06:36:20.685161+0000","last_fullsized":"2024-02-05T06:36:20.685161+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'16","last_scrub_stamp":"2024-02-05T06:36:20.685104+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:34:45.270135+0000","last_clean_scrub_stamp":"2024-02-05T06:36:20.685104+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:21.458668+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":106,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.926853+0000","last_change":"2024-02-05T06:35:24.948951+0000","last_active":"2024-02-05T06:36:19.926853+0000","last_peered":"2024-02-05T06:36:19.926853+0000","last_clean":"2024-02-05T06:36:19.926853+0000","last_became_active":"2024-02-05T06:35:24.948569+0000","last_became_peered":"2024-02-05T06:35:24.948569+0000","last_unstale":"2024-02-05T06:36:19.926853+0000","last_undegraded":"2024-02-05T06:36:19.926853+0000","last_fullsized":"2024-02-05T06:36:19.926853+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:55.062013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:55.062013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":117,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.932056+0000","last_change":"2024-02-05T06:36:13.766662+0000","last_active":"2024-02-05T06:36:19.932056+0000","last_peered":"2024-02-05T06:36:19.932056+0000","last_clean":"2024-02-05T06:36:19.932056+0000","last_became_active":"2024-02-05T06:35:24.925746+0000","last_became_peered":"2024-02-05T06:35:24.925746+0000","last_unstale":"2024-02-05T06:36:19.932056+0000","last_undegraded":"2024-02-05T06:36:19.932056+0000","last_fullsized":"2024-02-05T06:36:19.932056+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:13.766558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:13.766558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:27.599342+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":112,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.937203+0000","last_change":"2024-02-05T06:36:17.221190+0000","last_active":"2024-02-05T06:36:19.937203+0000","last_peered":"2024-02-05T06:36:19.937203+0000","last_clean":"2024-02-05T06:36:19.937203+0000","last_became_active":"2024-02-05T06:33:29.549204+0000","last_became_peered":"2024-02-05T06:33:29.549204+0000","last_unstale":"2024-02-05T06:36:19.937203+0000","last_undegraded":"2024-02-05T06:36:19.937203+0000","last_fullsized":"2024-02-05T06:36:19.937203+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:17.220950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:17.220950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:24.661470+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"73'4","reported_seq":114,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.946329+0000","last_change":"2024-02-05T06:36:06.303548+0000","last_active":"2024-02-05T06:36:19.946329+0000","last_peered":"2024-02-05T06:36:19.946329+0000","last_clean":"2024-02-05T06:36:19.946329+0000","last_became_active":"2024-02-05T06:33:31.573410+0000","last_became_peered":"2024-02-05T06:33:31.573410+0000","last_unstale":"2024-02-05T06:36:19.946329+0000","last_undegraded":"2024-02-05T06:36:19.946329+0000","last_fullsized":"2024-02-05T06:36:19.946329+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'4","last_scrub_stamp":"2024-02-05T06:36:06.303366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:06.303366+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:09.135553+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":100,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.945918+0000","last_change":"2024-02-05T06:34:54.902085+0000","last_active":"2024-02-05T06:36:19.945918+0000","last_peered":"2024-02-05T06:36:19.945918+0000","last_clean":"2024-02-05T06:36:19.945918+0000","last_became_active":"2024-02-05T06:33:35.667602+0000","last_became_peered":"2024-02-05T06:33:35.667602+0000","last_unstale":"2024-02-05T06:36:19.945918+0000","last_undegraded":"2024-02-05T06:36:19.945918+0000","last_fullsized":"2024-02-05T06:36:19.945918+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:54.902000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:54.902000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.927009+0000","last_change":"2024-02-05T06:36:06.846785+0000","last_active":"2024-02-05T06:36:19.927009+0000","last_peered":"2024-02-05T06:36:19.927009+0000","last_clean":"2024-02-05T06:36:19.927009+0000","last_became_active":"2024-02-05T06:35:24.933650+0000","last_became_peered":"2024-02-05T06:35:24.933650+0000","last_unstale":"2024-02-05T06:36:19.927009+0000","last_undegraded":"2024-02-05T06:36:19.927009+0000","last_fullsized":"2024-02-05T06:36:19.927009+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:06.846705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:06.846705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:30.022995+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.933966+0000","last_change":"2024-02-05T06:35:16.915069+0000","last_active":"2024-02-05T06:36:19.933966+0000","last_peered":"2024-02-05T06:36:19.933966+0000","last_clean":"2024-02-05T06:36:19.933966+0000","last_became_active":"2024-02-05T06:33:29.551278+0000","last_became_peered":"2024-02-05T06:33:29.551278+0000","last_unstale":"2024-02-05T06:36:19.933966+0000","last_undegraded":"2024-02-05T06:36:19.933966+0000","last_fullsized":"2024-02-05T06:36:19.933966+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:16.914970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:35:16.914970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:41.383486+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"93'16","reported_seq":132,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.939573+0000","last_change":"2024-02-05T06:35:22.938794+0000","last_active":"2024-02-05T06:36:19.939573+0000","last_peered":"2024-02-05T06:36:19.939573+0000","last_clean":"2024-02-05T06:36:19.939573+0000","last_became_active":"2024-02-05T06:33:31.664743+0000","last_became_peered":"2024-02-05T06:33:31.664743+0000","last_unstale":"2024-02-05T06:36:19.939573+0000","last_undegraded":"2024-02-05T06:36:19.939573+0000","last_fullsized":"2024-02-05T06:36:19.939573+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'16","last_scrub_stamp":"2024-02-05T06:35:22.938675+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-02-05T06:35:22.938675+0000","last_clean_scrub_stamp":"2024-02-05T06:35:22.938675+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:28.629679+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"73'1","reported_seq":101,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.931651+0000","last_change":"2024-02-05T06:35:09.676357+0000","last_active":"2024-02-05T06:36:19.931651+0000","last_peered":"2024-02-05T06:36:19.931651+0000","last_clean":"2024-02-05T06:36:19.931651+0000","last_became_active":"2024-02-05T06:33:35.595031+0000","last_became_peered":"2024-02-05T06:33:35.595031+0000","last_unstale":"2024-02-05T06:36:19.931651+0000","last_undegraded":"2024-02-05T06:36:19.931651+0000","last_fullsized":"2024-02-05T06:36:19.931651+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-02-05T06:35:09.676267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:09.676267+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:25.968668+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":104,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.945556+0000","last_change":"2024-02-05T06:35:03.027397+0000","last_active":"2024-02-05T06:36:19.945556+0000","last_peered":"2024-02-05T06:36:19.945556+0000","last_clean":"2024-02-05T06:36:19.945556+0000","last_became_active":"2024-02-05T06:33:33.593901+0000","last_became_peered":"2024-02-05T06:33:33.593901+0000","last_unstale":"2024-02-05T06:36:19.945556+0000","last_undegraded":"2024-02-05T06:36:19.945556+0000","last_fullsized":"2024-02-05T06:36:19.945556+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:03.027270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:35:03.027270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:31.482593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.927205+0000","last_change":"2024-02-05T06:35:24.938191+0000","last_active":"2024-02-05T06:36:19.927205+0000","last_peered":"2024-02-05T06:36:19.927205+0000","last_clean":"2024-02-05T06:36:19.927205+0000","last_became_active":"2024-02-05T06:35:24.937858+0000","last_became_peered":"2024-02-05T06:35:24.937858+0000","last_unstale":"2024-02-05T06:36:19.927205+0000","last_undegraded":"2024-02-05T06:36:19.927205+0000","last_fullsized":"2024-02-05T06:36:19.927205+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:06.946130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:35:06.946130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:27.453860+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":123,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.944085+0000","last_change":"2024-02-05T06:34:52.486190+0000","last_active":"2024-02-05T06:36:19.944085+0000","last_peered":"2024-02-05T06:36:19.944085+0000","last_clean":"2024-02-05T06:36:19.944085+0000","last_became_active":"2024-02-05T06:33:31.568529+0000","last_became_peered":"2024-02-05T06:33:31.568529+0000","last_unstale":"2024-02-05T06:36:19.944085+0000","last_undegraded":"2024-02-05T06:36:19.944085+0000","last_fullsized":"2024-02-05T06:36:19.944085+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:34:52.486064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:52.486064+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:20.775162+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":100,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.939225+0000","last_change":"2024-02-05T06:34:48.882657+0000","last_active":"2024-02-05T06:36:19.939225+0000","last_peered":"2024-02-05T06:36:19.939225+0000","last_clean":"2024-02-05T06:36:19.939225+0000","last_became_active":"2024-02-05T06:33:35.597399+0000","last_became_peered":"2024-02-05T06:33:35.597399+0000","last_unstale":"2024-02-05T06:36:19.939225+0000","last_undegraded":"2024-02-05T06:36:19.939225+0000","last_fullsized":"2024-02-05T06:36:19.939225+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:48.882556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:48.882556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":111,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.943877+0000","last_change":"2024-02-05T06:36:06.224697+0000","last_active":"2024-02-05T06:36:19.943877+0000","last_peered":"2024-02-05T06:36:19.943877+0000","last_clean":"2024-02-05T06:36:19.943877+0000","last_became_active":"2024-02-05T06:33:33.588461+0000","last_became_peered":"2024-02-05T06:33:33.588461+0000","last_unstale":"2024-02-05T06:36:19.943877+0000","last_undegraded":"2024-02-05T06:36:19.943877+0000","last_fullsized":"2024-02-05T06:36:19.943877+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:06.224205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:06.224205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:32.495349+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":46,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.927515+0000","last_change":"2024-02-05T06:35:24.924189+0000","last_active":"2024-02-05T06:36:19.927515+0000","last_peered":"2024-02-05T06:36:19.927515+0000","last_clean":"2024-02-05T06:36:19.927515+0000","last_became_active":"2024-02-05T06:35:24.923865+0000","last_became_peered":"2024-02-05T06:35:24.923865+0000","last_unstale":"2024-02-05T06:36:19.927515+0000","last_undegraded":"2024-02-05T06:36:19.927515+0000","last_fullsized":"2024-02-05T06:36:19.927515+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:48.561010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:48.561010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"92'16","reported_seq":138,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.932389+0000","last_change":"2024-02-05T06:35:24.955519+0000","last_active":"2024-02-05T06:36:19.932389+0000","last_peered":"2024-02-05T06:36:19.932389+0000","last_clean":"2024-02-05T06:36:19.932389+0000","last_became_active":"2024-02-05T06:35:24.932061+0000","last_became_peered":"2024-02-05T06:35:24.932061+0000","last_unstale":"2024-02-05T06:36:19.932389+0000","last_undegraded":"2024-02-05T06:36:19.932389+0000","last_fullsized":"2024-02-05T06:36:19.932389+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'16","last_scrub_stamp":"2024-02-05T06:35:02.589229+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:35:02.589229+0000","last_clean_scrub_stamp":"2024-02-05T06:35:02.589229+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":100,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.934122+0000","last_change":"2024-02-05T06:36:06.138504+0000","last_active":"2024-02-05T06:36:19.934122+0000","last_peered":"2024-02-05T06:36:19.934122+0000","last_clean":"2024-02-05T06:36:19.934122+0000","last_became_active":"2024-02-05T06:33:35.599808+0000","last_became_peered":"2024-02-05T06:33:35.599808+0000","last_unstale":"2024-02-05T06:36:19.934122+0000","last_undegraded":"2024-02-05T06:36:19.934122+0000","last_fullsized":"2024-02-05T06:36:19.934122+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:06.137845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:06.137845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:31.437152+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":145,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.932801+0000","last_change":"2024-02-05T06:35:02.969499+0000","last_active":"2024-02-05T06:36:19.932801+0000","last_peered":"2024-02-05T06:36:19.932801+0000","last_clean":"2024-02-05T06:36:19.932801+0000","last_became_active":"2024-02-05T06:33:33.585642+0000","last_became_peered":"2024-02-05T06:33:33.585642+0000","last_unstale":"2024-02-05T06:36:19.932801+0000","last_undegraded":"2024-02-05T06:36:19.932801+0000","last_fullsized":"2024-02-05T06:36:19.932801+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:35:02.969404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:35:02.969404+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.927011+0000","last_change":"2024-02-05T06:35:24.924230+0000","last_active":"2024-02-05T06:36:19.927011+0000","last_peered":"2024-02-05T06:36:19.927011+0000","last_clean":"2024-02-05T06:36:19.927011+0000","last_became_active":"2024-02-05T06:35:24.923827+0000","last_became_peered":"2024-02-05T06:35:24.923827+0000","last_unstale":"2024-02-05T06:36:19.927011+0000","last_undegraded":"2024-02-05T06:36:19.927011+0000","last_fullsized":"2024-02-05T06:36:19.927011+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:05.967303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:35:05.967303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:20.395529+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'10","reported_seq":141,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.939987+0000","last_change":"2024-02-05T06:36:12.205255+0000","last_active":"2024-02-05T06:36:19.939987+0000","last_peered":"2024-02-05T06:36:19.939987+0000","last_clean":"2024-02-05T06:36:19.939987+0000","last_became_active":"2024-02-05T06:36:12.171679+0000","last_became_peered":"2024-02-05T06:36:12.171679+0000","last_unstale":"2024-02-05T06:36:19.939987+0000","last_undegraded":"2024-02-05T06:36:19.939987+0000","last_fullsized":"2024-02-05T06:36:19.939987+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-02-05T06:34:41.873876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:41.873876+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":47,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.923327+0000","last_change":"2024-02-05T06:35:24.926205+0000","last_active":"2024-02-05T06:36:19.923327+0000","last_peered":"2024-02-05T06:36:19.923327+0000","last_clean":"2024-02-05T06:36:19.923327+0000","last_became_active":"2024-02-05T06:35:24.925810+0000","last_became_peered":"2024-02-05T06:35:24.925810+0000","last_unstale":"2024-02-05T06:36:19.923327+0000","last_undegraded":"2024-02-05T06:36:19.923327+0000","last_fullsized":"2024-02-05T06:36:19.923327+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:12.594735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:12.594735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":147,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.946881+0000","last_change":"2024-02-05T06:34:47.925702+0000","last_active":"2024-02-05T06:36:19.946881+0000","last_peered":"2024-02-05T06:36:19.946881+0000","last_clean":"2024-02-05T06:36:19.946881+0000","last_became_active":"2024-02-05T06:33:33.586043+0000","last_became_peered":"2024-02-05T06:33:33.586043+0000","last_unstale":"2024-02-05T06:36:19.946881+0000","last_undegraded":"2024-02-05T06:36:19.946881+0000","last_fullsized":"2024-02-05T06:36:19.946881+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:34:47.925577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:47.925577+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":112,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.933306+0000","last_change":"2024-02-05T06:34:53.792716+0000","last_active":"2024-02-05T06:36:19.933306+0000","last_peered":"2024-02-05T06:36:19.933306+0000","last_clean":"2024-02-05T06:36:19.933306+0000","last_became_active":"2024-02-05T06:33:29.552170+0000","last_became_peered":"2024-02-05T06:33:29.552170+0000","last_unstale":"2024-02-05T06:36:19.933306+0000","last_undegraded":"2024-02-05T06:36:19.933306+0000","last_fullsized":"2024-02-05T06:36:19.933306+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:53.792621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:53.792621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"73'9","reported_seq":146,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.932910+0000","last_change":"2024-02-05T06:36:12.206355+0000","last_active":"2024-02-05T06:36:19.932910+0000","last_peered":"2024-02-05T06:36:19.932910+0000","last_clean":"2024-02-05T06:36:19.932910+0000","last_became_active":"2024-02-05T06:36:12.173982+0000","last_became_peered":"2024-02-05T06:36:12.173982+0000","last_unstale":"2024-02-05T06:36:19.932910+0000","last_undegraded":"2024-02-05T06:36:19.932910+0000","last_fullsized":"2024-02-05T06:36:19.932910+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:36:09.760345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:09.760345+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:38.330993+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":112,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.923068+0000","last_change":"2024-02-05T06:36:08.696237+0000","last_active":"2024-02-05T06:36:19.923068+0000","last_peered":"2024-02-05T06:36:19.923068+0000","last_clean":"2024-02-05T06:36:19.923068+0000","last_became_active":"2024-02-05T06:35:24.923753+0000","last_became_peered":"2024-02-05T06:35:24.923753+0000","last_unstale":"2024-02-05T06:36:19.923068+0000","last_undegraded":"2024-02-05T06:36:19.923068+0000","last_fullsized":"2024-02-05T06:36:19.923068+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:08.696108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:08.696108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:12.296120+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":104,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.960367+0000","last_change":"2024-02-05T06:34:42.866516+0000","last_active":"2024-02-05T06:36:19.960367+0000","last_peered":"2024-02-05T06:36:19.960367+0000","last_clean":"2024-02-05T06:36:19.960367+0000","last_became_active":"2024-02-05T06:33:33.593025+0000","last_became_peered":"2024-02-05T06:33:33.593025+0000","last_unstale":"2024-02-05T06:36:19.960367+0000","last_undegraded":"2024-02-05T06:36:19.960367+0000","last_fullsized":"2024-02-05T06:36:19.960367+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:42.866394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:42.866394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":126,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.927025+0000","last_change":"2024-02-05T06:36:18.895871+0000","last_active":"2024-02-05T06:36:19.927025+0000","last_peered":"2024-02-05T06:36:19.927025+0000","last_clean":"2024-02-05T06:36:19.927025+0000","last_became_active":"2024-02-05T06:35:24.932786+0000","last_became_peered":"2024-02-05T06:35:24.932786+0000","last_unstale":"2024-02-05T06:36:19.927025+0000","last_undegraded":"2024-02-05T06:36:19.927025+0000","last_fullsized":"2024-02-05T06:36:19.927025+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:18.895623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:18.895623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:28.031555+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"73'9","reported_seq":124,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.941577+0000","last_change":"2024-02-05T06:34:43.760971+0000","last_active":"2024-02-05T06:36:19.941577+0000","last_peered":"2024-02-05T06:36:19.941577+0000","last_clean":"2024-02-05T06:36:19.941577+0000","last_became_active":"2024-02-05T06:33:31.665766+0000","last_became_peered":"2024-02-05T06:33:31.665766+0000","last_unstale":"2024-02-05T06:36:19.941577+0000","last_undegraded":"2024-02-05T06:36:19.941577+0000","last_fullsized":"2024-02-05T06:36:19.941577+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:34:43.760888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:43.760888+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":106,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.927128+0000","last_change":"2024-02-05T06:35:24.927249+0000","last_active":"2024-02-05T06:36:19.927128+0000","last_peered":"2024-02-05T06:36:19.927128+0000","last_clean":"2024-02-05T06:36:19.927128+0000","last_became_active":"2024-02-05T06:35:24.926850+0000","last_became_peered":"2024-02-05T06:35:24.926850+0000","last_unstale":"2024-02-05T06:36:19.927128+0000","last_undegraded":"2024-02-05T06:36:19.927128+0000","last_fullsized":"2024-02-05T06:36:19.927128+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:00.009630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:00.009630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:20.832648+0000","last_change":"2024-02-05T06:36:20.832672+0000","last_active":"2024-02-05T06:36:20.832648+0000","last_peered":"2024-02-05T06:36:20.832648+0000","last_clean":"2024-02-05T06:36:20.832648+0000","last_became_active":"2024-02-05T06:33:33.586768+0000","last_became_peered":"2024-02-05T06:33:33.586768+0000","last_unstale":"2024-02-05T06:36:20.832648+0000","last_undegraded":"2024-02-05T06:36:20.832648+0000","last_fullsized":"2024-02-05T06:36:20.832648+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:20.832579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:45.096340+0000","last_clean_scrub_stamp":"2024-02-05T06:36:20.832579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:39.454034+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":119,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.930165+0000","last_change":"2024-02-05T06:36:07.655271+0000","last_active":"2024-02-05T06:36:19.930165+0000","last_peered":"2024-02-05T06:36:19.930165+0000","last_clean":"2024-02-05T06:36:19.930165+0000","last_became_active":"2024-02-05T06:33:29.554427+0000","last_became_peered":"2024-02-05T06:33:29.554427+0000","last_unstale":"2024-02-05T06:36:19.930165+0000","last_undegraded":"2024-02-05T06:36:19.930165+0000","last_fullsized":"2024-02-05T06:36:19.930165+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:07.655171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:07.655171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:15.608469+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":117,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.960654+0000","last_change":"2024-02-05T06:34:55.912380+0000","last_active":"2024-02-05T06:36:19.960654+0000","last_peered":"2024-02-05T06:36:19.960654+0000","last_clean":"2024-02-05T06:36:19.960654+0000","last_became_active":"2024-02-05T06:33:31.665847+0000","last_became_peered":"2024-02-05T06:33:31.665847+0000","last_unstale":"2024-02-05T06:36:19.960654+0000","last_undegraded":"2024-02-05T06:36:19.960654+0000","last_fullsized":"2024-02-05T06:36:19.960654+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'6","last_scrub_stamp":"2024-02-05T06:34:55.912282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:55.912282+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"112'4405","reported_seq":4447,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:21.164947+0000","last_change":"2024-02-05T06:35:27.937770+0000","last_active":"2024-02-05T06:36:21.164947+0000","last_peered":"2024-02-05T06:36:21.164947+0000","last_clean":"2024-02-05T06:36:21.164947+0000","last_became_active":"2024-02-05T06:35:27.937512+0000","last_became_peered":"2024-02-05T06:35:27.937512+0000","last_unstale":"2024-02-05T06:36:21.164947+0000","last_undegraded":"2024-02-05T06:36:21.164947+0000","last_fullsized":"2024-02-05T06:36:21.164947+0000","mapping_epoch":96,"log_start":"101'100","ondisk_log_start":"101'100","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":4305,"log_dups_size":0,"ondisk_log_size":4305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:48.180519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288686080,"num_objects":4405,"num_object_clones":0,"num_object_copies":13215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4405,"num_write_kb":281920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.932145+0000","last_change":"2024-02-05T06:35:01.967337+0000","last_active":"2024-02-05T06:36:19.932145+0000","last_peered":"2024-02-05T06:36:19.932145+0000","last_clean":"2024-02-05T06:36:19.932145+0000","last_became_active":"2024-02-05T06:33:35.665581+0000","last_became_peered":"2024-02-05T06:33:35.665581+0000","last_unstale":"2024-02-05T06:36:19.932145+0000","last_undegraded":"2024-02-05T06:36:19.932145+0000","last_fullsized":"2024-02-05T06:36:19.932145+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:01.967242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:01.967242+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:22.767376+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":136,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.927468+0000","last_change":"2024-02-05T06:35:24.966901+0000","last_active":"2024-02-05T06:36:19.927468+0000","last_peered":"2024-02-05T06:36:19.927468+0000","last_clean":"2024-02-05T06:36:19.927468+0000","last_became_active":"2024-02-05T06:35:24.944468+0000","last_became_peered":"2024-02-05T06:35:24.944468+0000","last_unstale":"2024-02-05T06:36:19.927468+0000","last_undegraded":"2024-02-05T06:36:19.927468+0000","last_fullsized":"2024-02-05T06:36:19.927468+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-02-05T06:35:05.001912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:35:05.001912+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:32.290720+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"73'15","reported_seq":131,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.931369+0000","last_change":"2024-02-05T06:35:15.959953+0000","last_active":"2024-02-05T06:36:19.931369+0000","last_peered":"2024-02-05T06:36:19.931369+0000","last_clean":"2024-02-05T06:36:19.931369+0000","last_became_active":"2024-02-05T06:33:31.666056+0000","last_became_peered":"2024-02-05T06:33:31.666056+0000","last_unstale":"2024-02-05T06:36:19.931369+0000","last_undegraded":"2024-02-05T06:36:19.931369+0000","last_fullsized":"2024-02-05T06:36:19.931369+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-02-05T06:35:15.959820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:35:15.959820+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:33.226627+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"95'9","reported_seq":158,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.927224+0000","last_change":"2024-02-05T06:35:24.963428+0000","last_active":"2024-02-05T06:36:19.927224+0000","last_peered":"2024-02-05T06:36:19.927224+0000","last_clean":"2024-02-05T06:36:19.927224+0000","last_became_active":"2024-02-05T06:35:24.945581+0000","last_became_peered":"2024-02-05T06:35:24.945581+0000","last_unstale":"2024-02-05T06:36:19.927224+0000","last_undegraded":"2024-02-05T06:36:19.927224+0000","last_fullsized":"2024-02-05T06:36:19.927224+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:34:49.007659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:49.007659+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"112'4429","reported_seq":4470,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:21.225744+0000","last_change":"2024-02-05T06:35:27.933615+0000","last_active":"2024-02-05T06:36:21.225744+0000","last_peered":"2024-02-05T06:36:21.225744+0000","last_clean":"2024-02-05T06:36:21.225744+0000","last_became_active":"2024-02-05T06:35:27.933340+0000","last_became_peered":"2024-02-05T06:35:27.933340+0000","last_unstale":"2024-02-05T06:36:21.225744+0000","last_undegraded":"2024-02-05T06:36:21.225744+0000","last_fullsized":"2024-02-05T06:36:21.225744+0000","mapping_epoch":96,"log_start":"101'200","ondisk_log_start":"101'200","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":4229,"log_dups_size":0,"ondisk_log_size":4229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:38.512709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290258944,"num_objects":4429,"num_object_clones":0,"num_object_copies":13287,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4429,"num_write_kb":283456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":100,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.945736+0000","last_change":"2024-02-05T06:35:12.099300+0000","last_active":"2024-02-05T06:36:19.945736+0000","last_peered":"2024-02-05T06:36:19.945736+0000","last_clean":"2024-02-05T06:36:19.945736+0000","last_became_active":"2024-02-05T06:33:35.601030+0000","last_became_peered":"2024-02-05T06:33:35.601030+0000","last_unstale":"2024-02-05T06:36:19.945736+0000","last_undegraded":"2024-02-05T06:36:19.945736+0000","last_fullsized":"2024-02-05T06:36:19.945736+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:12.099209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:12.099209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:41.577423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.927345+0000","last_change":"2024-02-05T06:35:24.945191+0000","last_active":"2024-02-05T06:36:19.927345+0000","last_peered":"2024-02-05T06:36:19.927345+0000","last_clean":"2024-02-05T06:36:19.927345+0000","last_became_active":"2024-02-05T06:35:24.944869+0000","last_became_peered":"2024-02-05T06:35:24.944869+0000","last_unstale":"2024-02-05T06:36:19.927345+0000","last_undegraded":"2024-02-05T06:36:19.927345+0000","last_fullsized":"2024-02-05T06:36:19.927345+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:07.995294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:35:07.995294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'12","reported_seq":100,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.945391+0000","last_change":"2024-02-05T06:35:24.959782+0000","last_active":"2024-02-05T06:36:19.945391+0000","last_peered":"2024-02-05T06:36:19.945391+0000","last_clean":"2024-02-05T06:36:19.945391+0000","last_became_active":"2024-02-05T06:35:24.942308+0000","last_became_peered":"2024-02-05T06:35:24.942308+0000","last_unstale":"2024-02-05T06:36:19.945391+0000","last_undegraded":"2024-02-05T06:36:19.945391+0000","last_fullsized":"2024-02-05T06:36:19.945391+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:35:00.650542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:35:00.650542+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:29.937311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":102,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.924353+0000","last_change":"2024-02-05T06:34:40.272941+0000","last_active":"2024-02-05T06:36:19.924353+0000","last_peered":"2024-02-05T06:36:19.924353+0000","last_clean":"2024-02-05T06:36:19.924353+0000","last_became_active":"2024-02-05T06:33:33.592546+0000","last_became_peered":"2024-02-05T06:33:33.592546+0000","last_unstale":"2024-02-05T06:36:19.924353+0000","last_undegraded":"2024-02-05T06:36:19.924353+0000","last_fullsized":"2024-02-05T06:36:19.924353+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:40.272852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:40.272852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"112'4370","reported_seq":4412,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:21.163140+0000","last_change":"2024-02-05T06:35:27.930015+0000","last_active":"2024-02-05T06:36:21.163140+0000","last_peered":"2024-02-05T06:36:21.163140+0000","last_clean":"2024-02-05T06:36:21.163140+0000","last_became_active":"2024-02-05T06:35:27.929725+0000","last_became_peered":"2024-02-05T06:35:27.929725+0000","last_unstale":"2024-02-05T06:36:21.163140+0000","last_undegraded":"2024-02-05T06:36:21.163140+0000","last_fullsized":"2024-02-05T06:36:21.163140+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":4370,"log_dups_size":0,"ondisk_log_size":4370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:38.602129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286392320,"num_objects":4370,"num_object_clones":0,"num_object_copies":13110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4370,"num_write_kb":279680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":100,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.926628+0000","last_change":"2024-02-05T06:35:18.964868+0000","last_active":"2024-02-05T06:36:19.926628+0000","last_peered":"2024-02-05T06:36:19.926628+0000","last_clean":"2024-02-05T06:36:19.926628+0000","last_became_active":"2024-02-05T06:33:35.599527+0000","last_became_peered":"2024-02-05T06:33:35.599527+0000","last_unstale":"2024-02-05T06:36:19.926628+0000","last_undegraded":"2024-02-05T06:36:19.926628+0000","last_fullsized":"2024-02-05T06:36:19.926628+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:18.964748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:18.964748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:33.396526+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.924968+0000","last_change":"2024-02-05T06:36:16.649604+0000","last_active":"2024-02-05T06:36:19.924968+0000","last_peered":"2024-02-05T06:36:19.924968+0000","last_clean":"2024-02-05T06:36:19.924968+0000","last_became_active":"2024-02-05T06:33:33.589203+0000","last_became_peered":"2024-02-05T06:33:33.589203+0000","last_unstale":"2024-02-05T06:36:19.924968+0000","last_undegraded":"2024-02-05T06:36:19.924968+0000","last_fullsized":"2024-02-05T06:36:19.924968+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:16.649483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:16.649483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:43.871278+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":112,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.927432+0000","last_change":"2024-02-05T06:34:59.028827+0000","last_active":"2024-02-05T06:36:19.927432+0000","last_peered":"2024-02-05T06:36:19.927432+0000","last_clean":"2024-02-05T06:36:19.927432+0000","last_became_active":"2024-02-05T06:33:29.556541+0000","last_became_peered":"2024-02-05T06:33:29.556541+0000","last_unstale":"2024-02-05T06:36:19.927432+0000","last_undegraded":"2024-02-05T06:36:19.927432+0000","last_fullsized":"2024-02-05T06:36:19.927432+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:59.028748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:59.028748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:24.209293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"73'19","reported_seq":146,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.930829+0000","last_change":"2024-02-05T06:36:14.733809+0000","last_active":"2024-02-05T06:36:19.930829+0000","last_peered":"2024-02-05T06:36:19.930829+0000","last_clean":"2024-02-05T06:36:19.930829+0000","last_became_active":"2024-02-05T06:33:31.664699+0000","last_became_peered":"2024-02-05T06:33:31.664699+0000","last_unstale":"2024-02-05T06:36:19.930829+0000","last_undegraded":"2024-02-05T06:36:19.930829+0000","last_fullsized":"2024-02-05T06:36:19.930829+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-02-05T06:36:14.733710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:14.733710+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:28.251949+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"112'4488","reported_seq":4530,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:21.120953+0000","last_change":"2024-02-05T06:35:27.937986+0000","last_active":"2024-02-05T06:36:21.120953+0000","last_peered":"2024-02-05T06:36:21.120953+0000","last_clean":"2024-02-05T06:36:21.120953+0000","last_became_active":"2024-02-05T06:35:27.937755+0000","last_became_peered":"2024-02-05T06:35:27.937755+0000","last_unstale":"2024-02-05T06:36:21.120953+0000","last_undegraded":"2024-02-05T06:36:21.120953+0000","last_fullsized":"2024-02-05T06:36:21.120953+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":4488,"log_dups_size":0,"ondisk_log_size":4488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:39.991517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294125568,"num_objects":4488,"num_object_clones":0,"num_object_copies":13464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4488,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4488,"num_write_kb":287232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":92,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.932725+0000","last_change":"2024-02-05T06:33:35.598200+0000","last_active":"2024-02-05T06:36:19.932725+0000","last_peered":"2024-02-05T06:36:19.932725+0000","last_clean":"2024-02-05T06:36:19.932725+0000","last_became_active":"2024-02-05T06:33:35.597863+0000","last_became_peered":"2024-02-05T06:33:35.597863+0000","last_unstale":"2024-02-05T06:36:19.932725+0000","last_undegraded":"2024-02-05T06:36:19.932725+0000","last_fullsized":"2024-02-05T06:36:19.932725+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:33:34.572983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":108,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.924646+0000","last_change":"2024-02-05T06:35:24.943221+0000","last_active":"2024-02-05T06:36:19.924646+0000","last_peered":"2024-02-05T06:36:19.924646+0000","last_clean":"2024-02-05T06:36:19.924646+0000","last_became_active":"2024-02-05T06:35:24.942763+0000","last_became_peered":"2024-02-05T06:35:24.942763+0000","last_unstale":"2024-02-05T06:36:19.924646+0000","last_undegraded":"2024-02-05T06:36:19.924646+0000","last_fullsized":"2024-02-05T06:36:19.924646+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:45.299055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:45.299055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.932219+0000","last_change":"2024-02-05T06:35:08.945210+0000","last_active":"2024-02-05T06:36:19.932219+0000","last_peered":"2024-02-05T06:36:19.932219+0000","last_clean":"2024-02-05T06:36:19.932219+0000","last_became_active":"2024-02-05T06:33:29.555950+0000","last_became_peered":"2024-02-05T06:33:29.555950+0000","last_unstale":"2024-02-05T06:36:19.932219+0000","last_undegraded":"2024-02-05T06:36:19.932219+0000","last_fullsized":"2024-02-05T06:36:19.932219+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:08.945107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:35:08.945107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:28.415976+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"73'9","reported_seq":156,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.935081+0000","last_change":"2024-02-05T06:36:12.184960+0000","last_active":"2024-02-05T06:36:19.935081+0000","last_peered":"2024-02-05T06:36:19.935081+0000","last_clean":"2024-02-05T06:36:19.935081+0000","last_became_active":"2024-02-05T06:36:12.164629+0000","last_became_peered":"2024-02-05T06:36:12.164629+0000","last_unstale":"2024-02-05T06:36:19.935081+0000","last_undegraded":"2024-02-05T06:36:19.935081+0000","last_fullsized":"2024-02-05T06:36:19.935081+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:34:36.779466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:36.779466+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"112'4392","reported_seq":4432,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:21.162278+0000","last_change":"2024-02-05T06:35:27.934389+0000","last_active":"2024-02-05T06:36:21.162278+0000","last_peered":"2024-02-05T06:36:21.162278+0000","last_clean":"2024-02-05T06:36:21.162278+0000","last_became_active":"2024-02-05T06:35:27.934093+0000","last_became_peered":"2024-02-05T06:35:27.934093+0000","last_unstale":"2024-02-05T06:36:21.162278+0000","last_undegraded":"2024-02-05T06:36:21.162278+0000","last_fullsized":"2024-02-05T06:36:21.162278+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":4392,"log_dups_size":0,"ondisk_log_size":4392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:30.327918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287834112,"num_objects":4392,"num_object_clones":0,"num_object_copies":13176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4392,"num_write_kb":281088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":106,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.939579+0000","last_change":"2024-02-05T06:35:24.941903+0000","last_active":"2024-02-05T06:36:19.939579+0000","last_peered":"2024-02-05T06:36:19.939579+0000","last_clean":"2024-02-05T06:36:19.939579+0000","last_became_active":"2024-02-05T06:35:24.941567+0000","last_became_peered":"2024-02-05T06:35:24.941567+0000","last_unstale":"2024-02-05T06:36:19.939579+0000","last_undegraded":"2024-02-05T06:36:19.939579+0000","last_fullsized":"2024-02-05T06:36:19.939579+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:14.082774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:14.082774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":129,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.931939+0000","last_change":"2024-02-05T06:36:14.780874+0000","last_active":"2024-02-05T06:36:19.931939+0000","last_peered":"2024-02-05T06:36:19.931939+0000","last_clean":"2024-02-05T06:36:19.931939+0000","last_became_active":"2024-02-05T06:33:28.542228+0000","last_became_peered":"2024-02-05T06:33:28.542228+0000","last_unstale":"2024-02-05T06:36:19.931939+0000","last_undegraded":"2024-02-05T06:36:19.931939+0000","last_fullsized":"2024-02-05T06:36:19.931939+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-02-05T06:36:14.780738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:36:14.780738+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:42.032501+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001220348,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":54,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.935170+0000","last_change":"2024-02-05T06:36:15.167101+0000","last_active":"2024-02-05T06:36:19.935170+0000","last_peered":"2024-02-05T06:36:19.935170+0000","last_clean":"2024-02-05T06:36:19.935170+0000","last_became_active":"2024-02-05T06:35:24.928934+0000","last_became_peered":"2024-02-05T06:35:24.928934+0000","last_unstale":"2024-02-05T06:36:19.935170+0000","last_undegraded":"2024-02-05T06:36:19.935170+0000","last_fullsized":"2024-02-05T06:36:19.935170+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:15.166979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:15.166979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:38.581745+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":139,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.932561+0000","last_change":"2024-02-05T06:35:10.942882+0000","last_active":"2024-02-05T06:36:19.932561+0000","last_peered":"2024-02-05T06:36:19.932561+0000","last_clean":"2024-02-05T06:36:19.932561+0000","last_became_active":"2024-02-05T06:33:55.884601+0000","last_became_peered":"2024-02-05T06:33:55.884601+0000","last_unstale":"2024-02-05T06:36:19.932561+0000","last_undegraded":"2024-02-05T06:36:19.932561+0000","last_fullsized":"2024-02-05T06:36:19.932561+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-02-05T06:35:10.942752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:35:10.942752+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:23.895209+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.934883+0000","last_change":"2024-02-05T06:35:24.923325+0000","last_active":"2024-02-05T06:36:19.934883+0000","last_peered":"2024-02-05T06:36:19.934883+0000","last_clean":"2024-02-05T06:36:19.934883+0000","last_became_active":"2024-02-05T06:35:24.922958+0000","last_became_peered":"2024-02-05T06:35:24.922958+0000","last_unstale":"2024-02-05T06:36:19.934883+0000","last_undegraded":"2024-02-05T06:36:19.934883+0000","last_fullsized":"2024-02-05T06:36:19.934883+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:14.694964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:35:14.694964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:21.059595+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"112'4446","reported_seq":4488,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:21.193044+0000","last_change":"2024-02-05T06:35:27.933116+0000","last_active":"2024-02-05T06:36:21.193044+0000","last_peered":"2024-02-05T06:36:21.193044+0000","last_clean":"2024-02-05T06:36:21.193044+0000","last_became_active":"2024-02-05T06:35:27.932819+0000","last_became_peered":"2024-02-05T06:35:27.932819+0000","last_unstale":"2024-02-05T06:36:21.193044+0000","last_undegraded":"2024-02-05T06:36:21.193044+0000","last_fullsized":"2024-02-05T06:36:21.193044+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":4446,"log_dups_size":0,"ondisk_log_size":4446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:48.582917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291373056,"num_objects":4446,"num_object_clones":0,"num_object_copies":13338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4446,"num_write_kb":284544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":108,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.927503+0000","last_change":"2024-02-05T06:36:16.887854+0000","last_active":"2024-02-05T06:36:19.927503+0000","last_peered":"2024-02-05T06:36:19.927503+0000","last_clean":"2024-02-05T06:36:19.927503+0000","last_became_active":"2024-02-05T06:33:35.666324+0000","last_became_peered":"2024-02-05T06:33:35.666324+0000","last_unstale":"2024-02-05T06:36:19.927503+0000","last_undegraded":"2024-02-05T06:36:19.927503+0000","last_fullsized":"2024-02-05T06:36:19.927503+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:16.887732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:16.887732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:25.658445+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":111,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.938792+0000","last_change":"2024-02-05T06:35:21.506625+0000","last_active":"2024-02-05T06:36:19.938792+0000","last_peered":"2024-02-05T06:36:19.938792+0000","last_clean":"2024-02-05T06:36:19.938792+0000","last_became_active":"2024-02-05T06:33:29.551549+0000","last_became_peered":"2024-02-05T06:33:29.551549+0000","last_unstale":"2024-02-05T06:36:19.938792+0000","last_undegraded":"2024-02-05T06:36:19.938792+0000","last_fullsized":"2024-02-05T06:36:19.938792+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:21.506519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:35:21.506519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:26.003784+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":119,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.922443+0000","last_change":"2024-02-05T06:34:50.302211+0000","last_active":"2024-02-05T06:36:19.922443+0000","last_peered":"2024-02-05T06:36:19.922443+0000","last_clean":"2024-02-05T06:36:19.922443+0000","last_became_active":"2024-02-05T06:33:28.542977+0000","last_became_peered":"2024-02-05T06:33:28.542977+0000","last_unstale":"2024-02-05T06:36:19.922443+0000","last_undegraded":"2024-02-05T06:36:19.922443+0000","last_fullsized":"2024-02-05T06:36:19.922443+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:50.302133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:34:50.302133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091640799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'13","reported_seq":142,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.940214+0000","last_change":"2024-02-05T06:36:14.209527+0000","last_active":"2024-02-05T06:36:19.940214+0000","last_peered":"2024-02-05T06:36:19.940214+0000","last_clean":"2024-02-05T06:36:19.940214+0000","last_became_active":"2024-02-05T06:33:31.560082+0000","last_became_peered":"2024-02-05T06:33:31.560082+0000","last_unstale":"2024-02-05T06:36:19.940214+0000","last_undegraded":"2024-02-05T06:36:19.940214+0000","last_fullsized":"2024-02-05T06:36:19.940214+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-02-05T06:36:14.209318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:14.209318+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:29.833690+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.925388+0000","last_change":"2024-02-05T06:36:09.727864+0000","last_active":"2024-02-05T06:36:19.925388+0000","last_peered":"2024-02-05T06:36:19.925388+0000","last_clean":"2024-02-05T06:36:19.925388+0000","last_became_active":"2024-02-05T06:33:33.587593+0000","last_became_peered":"2024-02-05T06:33:33.587593+0000","last_unstale":"2024-02-05T06:36:19.925388+0000","last_undegraded":"2024-02-05T06:36:19.925388+0000","last_fullsized":"2024-02-05T06:36:19.925388+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:09.726651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:09.726651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:27.634848+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"112'4355","reported_seq":4397,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:21.188235+0000","last_change":"2024-02-05T06:35:27.935172+0000","last_active":"2024-02-05T06:36:21.188235+0000","last_peered":"2024-02-05T06:36:21.188235+0000","last_clean":"2024-02-05T06:36:21.188235+0000","last_became_active":"2024-02-05T06:35:27.934809+0000","last_became_peered":"2024-02-05T06:35:27.934809+0000","last_unstale":"2024-02-05T06:36:21.188235+0000","last_undegraded":"2024-02-05T06:36:21.188235+0000","last_fullsized":"2024-02-05T06:36:21.188235+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":4355,"log_dups_size":0,"ondisk_log_size":4355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:29.107529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285409280,"num_objects":4355,"num_object_clones":0,"num_object_copies":13065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4355,"num_write_kb":278720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":104,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.932875+0000","last_change":"2024-02-05T06:36:05.712818+0000","last_active":"2024-02-05T06:36:19.932875+0000","last_peered":"2024-02-05T06:36:19.932875+0000","last_clean":"2024-02-05T06:36:19.932875+0000","last_became_active":"2024-02-05T06:35:24.945472+0000","last_became_peered":"2024-02-05T06:35:24.945472+0000","last_unstale":"2024-02-05T06:36:19.932875+0000","last_undegraded":"2024-02-05T06:36:19.932875+0000","last_fullsized":"2024-02-05T06:36:19.932875+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:05.712737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:05.712737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:16.188343+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":366,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.926742+0000","last_change":"2024-02-05T06:35:11.952583+0000","last_active":"2024-02-05T06:36:19.926742+0000","last_peered":"2024-02-05T06:36:19.926742+0000","last_clean":"2024-02-05T06:36:19.926742+0000","last_became_active":"2024-02-05T06:33:19.862877+0000","last_became_peered":"2024-02-05T06:33:19.862877+0000","last_unstale":"2024-02-05T06:36:19.926742+0000","last_undegraded":"2024-02-05T06:36:19.926742+0000","last_fullsized":"2024-02-05T06:36:19.926742+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:35:11.952460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:35:11.952460+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:33.128777+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":112,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.947510+0000","last_change":"2024-02-05T06:34:44.894533+0000","last_active":"2024-02-05T06:36:19.947510+0000","last_peered":"2024-02-05T06:36:19.947510+0000","last_clean":"2024-02-05T06:36:19.947510+0000","last_became_active":"2024-02-05T06:33:29.552698+0000","last_became_peered":"2024-02-05T06:33:29.552698+0000","last_unstale":"2024-02-05T06:36:19.947510+0000","last_undegraded":"2024-02-05T06:36:19.947510+0000","last_fullsized":"2024-02-05T06:36:19.947510+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:44.894443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:44.894443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"95'17","reported_seq":189,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.931880+0000","last_change":"2024-02-05T06:35:24.955484+0000","last_active":"2024-02-05T06:36:19.931880+0000","last_peered":"2024-02-05T06:36:19.931880+0000","last_clean":"2024-02-05T06:36:19.931880+0000","last_became_active":"2024-02-05T06:35:24.925908+0000","last_became_peered":"2024-02-05T06:35:24.925908+0000","last_unstale":"2024-02-05T06:36:19.931880+0000","last_undegraded":"2024-02-05T06:36:19.931880+0000","last_fullsized":"2024-02-05T06:36:19.931880+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-02-05T06:35:02.410049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:35:02.410049+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:30.101577+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":104,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.927624+0000","last_change":"2024-02-05T06:35:13.945600+0000","last_active":"2024-02-05T06:36:19.927624+0000","last_peered":"2024-02-05T06:36:19.927624+0000","last_clean":"2024-02-05T06:36:19.927624+0000","last_became_active":"2024-02-05T06:33:33.591484+0000","last_became_peered":"2024-02-05T06:33:33.591484+0000","last_unstale":"2024-02-05T06:36:19.927624+0000","last_undegraded":"2024-02-05T06:36:19.927624+0000","last_fullsized":"2024-02-05T06:36:19.927624+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:13.945483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:35:13.945483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:28.271497+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"112'4489","reported_seq":4531,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:21.142636+0000","last_change":"2024-02-05T06:35:27.933269+0000","last_active":"2024-02-05T06:36:21.142636+0000","last_peered":"2024-02-05T06:36:21.142636+0000","last_clean":"2024-02-05T06:36:21.142636+0000","last_became_active":"2024-02-05T06:35:27.932989+0000","last_became_peered":"2024-02-05T06:35:27.932989+0000","last_unstale":"2024-02-05T06:36:21.142636+0000","last_undegraded":"2024-02-05T06:36:21.142636+0000","last_fullsized":"2024-02-05T06:36:21.142636+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":4489,"log_dups_size":0,"ondisk_log_size":4489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:46.889235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294191104,"num_objects":4489,"num_object_clones":0,"num_object_copies":13467,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4489,"num_write_kb":287296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":99,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.943180+0000","last_change":"2024-02-05T06:34:46.361122+0000","last_active":"2024-02-05T06:36:19.943180+0000","last_peered":"2024-02-05T06:36:19.943180+0000","last_clean":"2024-02-05T06:36:19.943180+0000","last_became_active":"2024-02-05T06:33:35.600656+0000","last_became_peered":"2024-02-05T06:33:35.600656+0000","last_unstale":"2024-02-05T06:36:19.943180+0000","last_undegraded":"2024-02-05T06:36:19.943180+0000","last_fullsized":"2024-02-05T06:36:19.943180+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:46.361013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:46.361013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.932513+0000","last_change":"2024-02-05T06:35:24.939887+0000","last_active":"2024-02-05T06:36:19.932513+0000","last_peered":"2024-02-05T06:36:19.932513+0000","last_clean":"2024-02-05T06:36:19.932513+0000","last_became_active":"2024-02-05T06:35:24.938966+0000","last_became_peered":"2024-02-05T06:35:24.938966+0000","last_unstale":"2024-02-05T06:36:19.932513+0000","last_undegraded":"2024-02-05T06:36:19.932513+0000","last_fullsized":"2024-02-05T06:36:19.932513+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:44.781704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:44.781704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":288,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.926712+0000","last_change":"2024-02-05T06:34:41.991897+0000","last_active":"2024-02-05T06:36:19.926712+0000","last_peered":"2024-02-05T06:36:19.926712+0000","last_clean":"2024-02-05T06:36:19.926712+0000","last_became_active":"2024-02-05T06:33:28.544852+0000","last_became_peered":"2024-02-05T06:33:28.544852+0000","last_unstale":"2024-02-05T06:36:19.926712+0000","last_undegraded":"2024-02-05T06:36:19.926712+0000","last_fullsized":"2024-02-05T06:36:19.926712+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T06:34:41.991777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:34:41.991777+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00074151600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":164,"num_read_kb":159,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":126,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.934221+0000","last_change":"2024-02-05T06:34:52.743798+0000","last_active":"2024-02-05T06:36:19.934221+0000","last_peered":"2024-02-05T06:36:19.934221+0000","last_clean":"2024-02-05T06:36:19.934221+0000","last_became_active":"2024-02-05T06:33:31.564604+0000","last_became_peered":"2024-02-05T06:33:31.564604+0000","last_unstale":"2024-02-05T06:36:19.934221+0000","last_undegraded":"2024-02-05T06:36:19.934221+0000","last_fullsized":"2024-02-05T06:36:19.934221+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:34:52.743708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:52.743708+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":102,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.933059+0000","last_change":"2024-02-05T06:35:23.964960+0000","last_active":"2024-02-05T06:36:19.933059+0000","last_peered":"2024-02-05T06:36:19.933059+0000","last_clean":"2024-02-05T06:36:19.933059+0000","last_became_active":"2024-02-05T06:33:33.589388+0000","last_became_peered":"2024-02-05T06:33:33.589388+0000","last_unstale":"2024-02-05T06:36:19.933059+0000","last_undegraded":"2024-02-05T06:36:19.933059+0000","last_fullsized":"2024-02-05T06:36:19.933059+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:23.964888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:35:23.964888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:37.479309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"112'4416","reported_seq":4458,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:21.145448+0000","last_change":"2024-02-05T06:35:27.937863+0000","last_active":"2024-02-05T06:36:21.145448+0000","last_peered":"2024-02-05T06:36:21.145448+0000","last_clean":"2024-02-05T06:36:21.145448+0000","last_became_active":"2024-02-05T06:35:27.937618+0000","last_became_peered":"2024-02-05T06:35:27.937618+0000","last_unstale":"2024-02-05T06:36:21.145448+0000","last_undegraded":"2024-02-05T06:36:21.145448+0000","last_fullsized":"2024-02-05T06:36:21.145448+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":4416,"log_dups_size":0,"ondisk_log_size":4416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:45.314347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289406976,"num_objects":4416,"num_object_clones":0,"num_object_copies":13248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4416,"num_write_kb":282624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":99,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.945628+0000","last_change":"2024-02-05T06:35:14.534417+0000","last_active":"2024-02-05T06:36:19.945628+0000","last_peered":"2024-02-05T06:36:19.945628+0000","last_clean":"2024-02-05T06:36:19.945628+0000","last_became_active":"2024-02-05T06:33:35.664858+0000","last_became_peered":"2024-02-05T06:33:35.664858+0000","last_unstale":"2024-02-05T06:36:19.945628+0000","last_undegraded":"2024-02-05T06:36:19.945628+0000","last_fullsized":"2024-02-05T06:36:19.945628+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:14.534332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:14.534332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:21.874172+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"66'1","reported_seq":125,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.943419+0000","last_change":"2024-02-05T06:36:09.253329+0000","last_active":"2024-02-05T06:36:19.943419+0000","last_peered":"2024-02-05T06:36:19.943419+0000","last_clean":"2024-02-05T06:36:19.943419+0000","last_became_active":"2024-02-05T06:33:29.548227+0000","last_became_peered":"2024-02-05T06:33:29.548227+0000","last_unstale":"2024-02-05T06:36:19.943419+0000","last_undegraded":"2024-02-05T06:36:19.943419+0000","last_fullsized":"2024-02-05T06:36:19.943419+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:36:09.253045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:09.253045+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:28.032291+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'19","reported_seq":144,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.935317+0000","last_change":"2024-02-05T06:35:27.592730+0000","last_active":"2024-02-05T06:36:19.935317+0000","last_peered":"2024-02-05T06:36:19.935317+0000","last_clean":"2024-02-05T06:36:19.935317+0000","last_became_active":"2024-02-05T06:33:31.667340+0000","last_became_peered":"2024-02-05T06:33:31.667340+0000","last_unstale":"2024-02-05T06:36:19.935317+0000","last_undegraded":"2024-02-05T06:36:19.935317+0000","last_fullsized":"2024-02-05T06:36:19.935317+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-02-05T06:35:27.592613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:35:27.592613+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:53.337884+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":103,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.930894+0000","last_change":"2024-02-05T06:34:50.353163+0000","last_active":"2024-02-05T06:36:19.930894+0000","last_peered":"2024-02-05T06:36:19.930894+0000","last_clean":"2024-02-05T06:36:19.930894+0000","last_became_active":"2024-02-05T06:33:33.588163+0000","last_became_peered":"2024-02-05T06:33:33.588163+0000","last_unstale":"2024-02-05T06:36:19.930894+0000","last_undegraded":"2024-02-05T06:36:19.930894+0000","last_fullsized":"2024-02-05T06:36:19.930894+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:50.353040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:50.353040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"112'4529","reported_seq":4571,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:21.225921+0000","last_change":"2024-02-05T06:35:27.936946+0000","last_active":"2024-02-05T06:36:21.225921+0000","last_peered":"2024-02-05T06:36:21.225921+0000","last_clean":"2024-02-05T06:36:21.225921+0000","last_became_active":"2024-02-05T06:35:27.935733+0000","last_became_peered":"2024-02-05T06:35:27.935733+0000","last_unstale":"2024-02-05T06:36:21.225921+0000","last_undegraded":"2024-02-05T06:36:21.225921+0000","last_fullsized":"2024-02-05T06:36:21.225921+0000","mapping_epoch":96,"log_start":"101'300","ondisk_log_start":"101'300","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":4229,"log_dups_size":0,"ondisk_log_size":4229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:40.137655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296812544,"num_objects":4529,"num_object_clones":0,"num_object_copies":13587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4529,"num_write_kb":289856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":100,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.931752+0000","last_change":"2024-02-05T06:36:07.154940+0000","last_active":"2024-02-05T06:36:19.931752+0000","last_peered":"2024-02-05T06:36:19.931752+0000","last_clean":"2024-02-05T06:36:19.931752+0000","last_became_active":"2024-02-05T06:33:35.600485+0000","last_became_peered":"2024-02-05T06:33:35.600485+0000","last_unstale":"2024-02-05T06:36:19.931752+0000","last_undegraded":"2024-02-05T06:36:19.931752+0000","last_fullsized":"2024-02-05T06:36:19.931752+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:07.154815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:07.154815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:21.512385+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.932880+0000","last_change":"2024-02-05T06:36:13.768661+0000","last_active":"2024-02-05T06:36:19.932880+0000","last_peered":"2024-02-05T06:36:19.932880+0000","last_clean":"2024-02-05T06:36:19.932880+0000","last_became_active":"2024-02-05T06:33:29.554940+0000","last_became_peered":"2024-02-05T06:33:29.554940+0000","last_unstale":"2024-02-05T06:36:19.932880+0000","last_undegraded":"2024-02-05T06:36:19.932880+0000","last_fullsized":"2024-02-05T06:36:19.932880+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:13.768592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:13.768592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:36.460814+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"73'10","reported_seq":140,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.943951+0000","last_change":"2024-02-05T06:36:07.235710+0000","last_active":"2024-02-05T06:36:19.943951+0000","last_peered":"2024-02-05T06:36:19.943951+0000","last_clean":"2024-02-05T06:36:19.943951+0000","last_became_active":"2024-02-05T06:35:24.935899+0000","last_became_peered":"2024-02-05T06:35:24.935899+0000","last_unstale":"2024-02-05T06:36:19.943951+0000","last_undegraded":"2024-02-05T06:36:19.943951+0000","last_fullsized":"2024-02-05T06:36:19.943951+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-02-05T06:36:07.232535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:07.232535+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:12.201763+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"73'8","reported_seq":147,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.934922+0000","last_change":"2024-02-05T06:34:54.764491+0000","last_active":"2024-02-05T06:36:19.934922+0000","last_peered":"2024-02-05T06:36:19.934922+0000","last_clean":"2024-02-05T06:36:19.934922+0000","last_became_active":"2024-02-05T06:33:33.582187+0000","last_became_peered":"2024-02-05T06:33:33.582187+0000","last_unstale":"2024-02-05T06:36:19.934922+0000","last_undegraded":"2024-02-05T06:36:19.934922+0000","last_fullsized":"2024-02-05T06:36:19.934922+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:34:54.764410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:54.764410+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"112'4410","reported_seq":4452,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:21.111625+0000","last_change":"2024-02-05T06:35:27.936200+0000","last_active":"2024-02-05T06:36:21.111625+0000","last_peered":"2024-02-05T06:36:21.111625+0000","last_clean":"2024-02-05T06:36:21.111625+0000","last_became_active":"2024-02-05T06:35:27.935898+0000","last_became_peered":"2024-02-05T06:35:27.935898+0000","last_unstale":"2024-02-05T06:36:21.111625+0000","last_undegraded":"2024-02-05T06:36:21.111625+0000","last_fullsized":"2024-02-05T06:36:21.111625+0000","mapping_epoch":96,"log_start":"101'100","ondisk_log_start":"101'100","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":4310,"log_dups_size":0,"ondisk_log_size":4310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:29.726677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289013760,"num_objects":4410,"num_object_clones":0,"num_object_copies":13230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4410,"num_write_kb":282240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":108,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.927291+0000","last_change":"2024-02-05T06:36:11.928300+0000","last_active":"2024-02-05T06:36:19.927291+0000","last_peered":"2024-02-05T06:36:19.927291+0000","last_clean":"2024-02-05T06:36:19.927291+0000","last_became_active":"2024-02-05T06:33:35.666315+0000","last_became_peered":"2024-02-05T06:33:35.666315+0000","last_unstale":"2024-02-05T06:36:19.927291+0000","last_undegraded":"2024-02-05T06:36:19.927291+0000","last_fullsized":"2024-02-05T06:36:19.927291+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:11.928194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:11.928194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:16.039940+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"95'2","reported_seq":124,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.934331+0000","last_change":"2024-02-05T06:35:25.616790+0000","last_active":"2024-02-05T06:36:19.934331+0000","last_peered":"2024-02-05T06:36:19.934331+0000","last_clean":"2024-02-05T06:36:19.934331+0000","last_became_active":"2024-02-05T06:35:24.927524+0000","last_became_peered":"2024-02-05T06:35:24.927524+0000","last_unstale":"2024-02-05T06:36:19.934331+0000","last_undegraded":"2024-02-05T06:36:19.934331+0000","last_fullsized":"2024-02-05T06:36:19.934331+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-05T06:35:25.616708+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-05T06:35:25.616708+0000","last_clean_scrub_stamp":"2024-02-05T06:35:25.616708+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:36.057122+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'14","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.931474+0000","last_change":"2024-02-05T06:36:17.814905+0000","last_active":"2024-02-05T06:36:19.931474+0000","last_peered":"2024-02-05T06:36:19.931474+0000","last_clean":"2024-02-05T06:36:19.931474+0000","last_became_active":"2024-02-05T06:35:24.937181+0000","last_became_peered":"2024-02-05T06:35:24.937181+0000","last_unstale":"2024-02-05T06:36:19.931474+0000","last_undegraded":"2024-02-05T06:36:19.931474+0000","last_fullsized":"2024-02-05T06:36:19.931474+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'14","last_scrub_stamp":"2024-02-05T06:36:17.814774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:17.814774+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:28.842951+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"95'9","reported_seq":158,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.946668+0000","last_change":"2024-02-05T06:35:24.968765+0000","last_active":"2024-02-05T06:36:19.946668+0000","last_peered":"2024-02-05T06:36:19.946668+0000","last_clean":"2024-02-05T06:36:19.946668+0000","last_became_active":"2024-02-05T06:35:24.944879+0000","last_became_peered":"2024-02-05T06:35:24.944879+0000","last_unstale":"2024-02-05T06:36:19.946668+0000","last_undegraded":"2024-02-05T06:36:19.946668+0000","last_fullsized":"2024-02-05T06:36:19.946668+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:34:56.941840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:56.941840+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"112'4530","reported_seq":4571,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:21.208808+0000","last_change":"2024-02-05T06:35:27.933500+0000","last_active":"2024-02-05T06:36:21.208808+0000","last_peered":"2024-02-05T06:36:21.208808+0000","last_clean":"2024-02-05T06:36:21.208808+0000","last_became_active":"2024-02-05T06:35:27.933203+0000","last_became_peered":"2024-02-05T06:35:27.933203+0000","last_unstale":"2024-02-05T06:36:21.208808+0000","last_undegraded":"2024-02-05T06:36:21.208808+0000","last_fullsized":"2024-02-05T06:36:21.208808+0000","mapping_epoch":96,"log_start":"101'300","ondisk_log_start":"101'300","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":4230,"log_dups_size":0,"ondisk_log_size":4230,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:40.834456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296878080,"num_objects":4530,"num_object_clones":0,"num_object_copies":13590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4530,"num_write_kb":289920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":55,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.947894+0000","last_change":"2024-02-05T06:35:25.951612+0000","last_active":"2024-02-05T06:36:19.947894+0000","last_peered":"2024-02-05T06:36:19.947894+0000","last_clean":"2024-02-05T06:36:19.947894+0000","last_became_active":"2024-02-05T06:35:24.948450+0000","last_became_peered":"2024-02-05T06:35:24.948450+0000","last_unstale":"2024-02-05T06:36:19.947894+0000","last_undegraded":"2024-02-05T06:36:19.947894+0000","last_fullsized":"2024-02-05T06:36:19.947894+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:25.951483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:25.951483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:33.999942+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":112,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.951537+0000","last_change":"2024-02-05T06:35:10.061496+0000","last_active":"2024-02-05T06:36:19.951537+0000","last_peered":"2024-02-05T06:36:19.951537+0000","last_clean":"2024-02-05T06:36:19.951537+0000","last_became_active":"2024-02-05T06:33:29.556002+0000","last_became_peered":"2024-02-05T06:33:29.556002+0000","last_unstale":"2024-02-05T06:36:19.951537+0000","last_undegraded":"2024-02-05T06:36:19.951537+0000","last_fullsized":"2024-02-05T06:36:19.951537+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:10.061388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:35:10.061388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"73'18","reported_seq":140,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.946571+0000","last_change":"2024-02-05T06:35:04.034763+0000","last_active":"2024-02-05T06:36:19.946571+0000","last_peered":"2024-02-05T06:36:19.946571+0000","last_clean":"2024-02-05T06:36:19.946571+0000","last_became_active":"2024-02-05T06:33:31.665995+0000","last_became_peered":"2024-02-05T06:33:31.665995+0000","last_unstale":"2024-02-05T06:36:19.946571+0000","last_undegraded":"2024-02-05T06:36:19.946571+0000","last_fullsized":"2024-02-05T06:36:19.946571+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-02-05T06:35:04.034664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:35:04.034664+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":46,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.939493+0000","last_change":"2024-02-05T06:35:24.942861+0000","last_active":"2024-02-05T06:36:19.939493+0000","last_peered":"2024-02-05T06:36:19.939493+0000","last_clean":"2024-02-05T06:36:19.939493+0000","last_became_active":"2024-02-05T06:35:24.942358+0000","last_became_peered":"2024-02-05T06:35:24.942358+0000","last_unstale":"2024-02-05T06:36:19.939493+0000","last_undegraded":"2024-02-05T06:36:19.939493+0000","last_fullsized":"2024-02-05T06:36:19.939493+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:49.579493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:49.579493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.c","version":"112'4345","reported_seq":4387,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:21.143223+0000","last_change":"2024-02-05T06:35:27.930565+0000","last_active":"2024-02-05T06:36:21.143223+0000","last_peered":"2024-02-05T06:36:21.143223+0000","last_clean":"2024-02-05T06:36:21.143223+0000","last_became_active":"2024-02-05T06:35:27.930258+0000","last_became_peered":"2024-02-05T06:35:27.930258+0000","last_unstale":"2024-02-05T06:36:21.143223+0000","last_undegraded":"2024-02-05T06:36:21.143223+0000","last_fullsized":"2024-02-05T06:36:21.143223+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":4345,"log_dups_size":0,"ondisk_log_size":4345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:39.507081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284753920,"num_objects":4345,"num_object_clones":0,"num_object_copies":13035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4345,"num_write_kb":278080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":106,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.932706+0000","last_change":"2024-02-05T06:36:12.777606+0000","last_active":"2024-02-05T06:36:19.932706+0000","last_peered":"2024-02-05T06:36:19.932706+0000","last_clean":"2024-02-05T06:36:19.932706+0000","last_became_active":"2024-02-05T06:33:35.593210+0000","last_became_peered":"2024-02-05T06:33:35.593210+0000","last_unstale":"2024-02-05T06:36:19.932706+0000","last_undegraded":"2024-02-05T06:36:19.932706+0000","last_fullsized":"2024-02-05T06:36:19.932706+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:12.777526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:12.777526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:28.471089+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"95'9","reported_seq":96,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.933999+0000","last_change":"2024-02-05T06:36:11.784869+0000","last_active":"2024-02-05T06:36:19.933999+0000","last_peered":"2024-02-05T06:36:19.933999+0000","last_clean":"2024-02-05T06:36:19.933999+0000","last_became_active":"2024-02-05T06:35:24.931241+0000","last_became_peered":"2024-02-05T06:35:24.931241+0000","last_unstale":"2024-02-05T06:36:19.933999+0000","last_undegraded":"2024-02-05T06:36:19.933999+0000","last_fullsized":"2024-02-05T06:36:19.933999+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'9","last_scrub_stamp":"2024-02-05T06:36:11.784734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:11.784734+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:25.027365+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":112,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.946484+0000","last_change":"2024-02-05T06:35:17.058289+0000","last_active":"2024-02-05T06:36:19.946484+0000","last_peered":"2024-02-05T06:36:19.946484+0000","last_clean":"2024-02-05T06:36:19.946484+0000","last_became_active":"2024-02-05T06:33:29.555919+0000","last_became_peered":"2024-02-05T06:33:29.555919+0000","last_unstale":"2024-02-05T06:36:19.946484+0000","last_undegraded":"2024-02-05T06:36:19.946484+0000","last_fullsized":"2024-02-05T06:36:19.946484+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:17.058199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:35:17.058199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"73'15","reported_seq":154,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.940033+0000","last_change":"2024-02-05T06:35:24.961720+0000","last_active":"2024-02-05T06:36:19.940033+0000","last_peered":"2024-02-05T06:36:19.940033+0000","last_clean":"2024-02-05T06:36:19.940033+0000","last_became_active":"2024-02-05T06:35:24.940653+0000","last_became_peered":"2024-02-05T06:35:24.940653+0000","last_unstale":"2024-02-05T06:36:19.940033+0000","last_undegraded":"2024-02-05T06:36:19.940033+0000","last_fullsized":"2024-02-05T06:36:19.940033+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-02-05T06:35:02.572925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:35:02.572925+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"112'4382","reported_seq":4424,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:21.141925+0000","last_change":"2024-02-05T06:35:27.935929+0000","last_active":"2024-02-05T06:36:21.141925+0000","last_peered":"2024-02-05T06:36:21.141925+0000","last_clean":"2024-02-05T06:36:21.141925+0000","last_became_active":"2024-02-05T06:35:27.935655+0000","last_became_peered":"2024-02-05T06:35:27.935655+0000","last_unstale":"2024-02-05T06:36:21.141925+0000","last_undegraded":"2024-02-05T06:36:21.141925+0000","last_fullsized":"2024-02-05T06:36:21.141925+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":4382,"log_dups_size":0,"ondisk_log_size":4382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:50.391475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287178752,"num_objects":4382,"num_object_clones":0,"num_object_copies":13146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4382,"num_write_kb":280448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":100,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.939902+0000","last_change":"2024-02-05T06:35:20.042610+0000","last_active":"2024-02-05T06:36:19.939902+0000","last_peered":"2024-02-05T06:36:19.939902+0000","last_clean":"2024-02-05T06:36:19.939902+0000","last_became_active":"2024-02-05T06:33:35.668020+0000","last_became_peered":"2024-02-05T06:33:35.668020+0000","last_unstale":"2024-02-05T06:36:19.939902+0000","last_undegraded":"2024-02-05T06:36:19.939902+0000","last_fullsized":"2024-02-05T06:36:19.939902+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:20.042534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:20.042534+0000","last_clean_scrub_stamp":"2024-02-05T06:35:20.042534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:27.194068+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":108,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.933054+0000","last_change":"2024-02-05T06:35:24.930172+0000","last_active":"2024-02-05T06:36:19.933054+0000","last_peered":"2024-02-05T06:36:19.933054+0000","last_clean":"2024-02-05T06:36:19.933054+0000","last_became_active":"2024-02-05T06:35:24.929834+0000","last_became_peered":"2024-02-05T06:35:24.929834+0000","last_unstale":"2024-02-05T06:36:19.933054+0000","last_undegraded":"2024-02-05T06:36:19.933054+0000","last_fullsized":"2024-02-05T06:36:19.933054+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:03.999803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:35:03.999803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":46,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.922521+0000","last_change":"2024-02-05T06:35:24.947332+0000","last_active":"2024-02-05T06:36:19.922521+0000","last_peered":"2024-02-05T06:36:19.922521+0000","last_clean":"2024-02-05T06:36:19.922521+0000","last_became_active":"2024-02-05T06:35:24.946121+0000","last_became_peered":"2024-02-05T06:35:24.946121+0000","last_unstale":"2024-02-05T06:36:19.922521+0000","last_undegraded":"2024-02-05T06:36:19.922521+0000","last_fullsized":"2024-02-05T06:36:19.922521+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:21.638465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:35:21.638465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:43.702380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.e","version":"73'11","reported_seq":114,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.931207+0000","last_change":"2024-02-05T06:35:54.297314+0000","last_active":"2024-02-05T06:36:19.931207+0000","last_peered":"2024-02-05T06:36:19.931207+0000","last_clean":"2024-02-05T06:36:19.931207+0000","last_became_active":"2024-02-05T06:35:54.274362+0000","last_became_peered":"2024-02-05T06:35:54.274362+0000","last_unstale":"2024-02-05T06:36:19.931207+0000","last_undegraded":"2024-02-05T06:36:19.931207+0000","last_fullsized":"2024-02-05T06:36:19.931207+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-02-05T06:34:50.592793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:50.592793+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.e","version":"112'4457","reported_seq":4499,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:21.116020+0000","last_change":"2024-02-05T06:35:27.938681+0000","last_active":"2024-02-05T06:36:21.116020+0000","last_peered":"2024-02-05T06:36:21.116020+0000","last_clean":"2024-02-05T06:36:21.116020+0000","last_became_active":"2024-02-05T06:35:27.938387+0000","last_became_peered":"2024-02-05T06:35:27.938387+0000","last_unstale":"2024-02-05T06:36:21.116020+0000","last_undegraded":"2024-02-05T06:36:21.116020+0000","last_fullsized":"2024-02-05T06:36:21.116020+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":4457,"log_dups_size":0,"ondisk_log_size":4457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:31.522485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292093952,"num_objects":4457,"num_object_clones":0,"num_object_copies":13371,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4457,"num_write_kb":285248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":104,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.923464+0000","last_change":"2024-02-05T06:35:24.939716+0000","last_active":"2024-02-05T06:36:19.923464+0000","last_peered":"2024-02-05T06:36:19.923464+0000","last_clean":"2024-02-05T06:36:19.923464+0000","last_became_active":"2024-02-05T06:35:24.939248+0000","last_became_peered":"2024-02-05T06:35:24.939248+0000","last_unstale":"2024-02-05T06:36:19.923464+0000","last_undegraded":"2024-02-05T06:36:19.923464+0000","last_fullsized":"2024-02-05T06:36:19.923464+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:49.298233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:49.298233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":109,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.943510+0000","last_change":"2024-02-05T06:35:24.926419+0000","last_active":"2024-02-05T06:36:19.943510+0000","last_peered":"2024-02-05T06:36:19.943510+0000","last_clean":"2024-02-05T06:36:19.943510+0000","last_became_active":"2024-02-05T06:35:24.925851+0000","last_became_peered":"2024-02-05T06:35:24.925851+0000","last_unstale":"2024-02-05T06:36:19.943510+0000","last_undegraded":"2024-02-05T06:36:19.943510+0000","last_fullsized":"2024-02-05T06:36:19.943510+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:04.582689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:35:04.582689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":117,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.930931+0000","last_change":"2024-02-05T06:35:24.940640+0000","last_active":"2024-02-05T06:36:19.930931+0000","last_peered":"2024-02-05T06:36:19.930931+0000","last_clean":"2024-02-05T06:36:19.930931+0000","last_became_active":"2024-02-05T06:35:24.940276+0000","last_became_peered":"2024-02-05T06:35:24.940276+0000","last_unstale":"2024-02-05T06:36:19.930931+0000","last_undegraded":"2024-02-05T06:36:19.930931+0000","last_fullsized":"2024-02-05T06:36:19.930931+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:57.290944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:57.290944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"95'25","reported_seq":127,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.923648+0000","last_change":"2024-02-05T06:35:26.275081+0000","last_active":"2024-02-05T06:36:19.923648+0000","last_peered":"2024-02-05T06:36:19.923648+0000","last_clean":"2024-02-05T06:36:19.923648+0000","last_became_active":"2024-02-05T06:35:24.931545+0000","last_became_peered":"2024-02-05T06:35:24.931545+0000","last_unstale":"2024-02-05T06:36:19.923648+0000","last_undegraded":"2024-02-05T06:36:19.923648+0000","last_fullsized":"2024-02-05T06:36:19.923648+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'25","last_scrub_stamp":"2024-02-05T06:35:26.274944+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-02-05T06:35:26.274944+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.274944+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:45.121910+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"112'4460","reported_seq":4502,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:21.178125+0000","last_change":"2024-02-05T06:35:27.931906+0000","last_active":"2024-02-05T06:36:21.178125+0000","last_peered":"2024-02-05T06:36:21.178125+0000","last_clean":"2024-02-05T06:36:21.178125+0000","last_became_active":"2024-02-05T06:35:27.931422+0000","last_became_peered":"2024-02-05T06:35:27.931422+0000","last_unstale":"2024-02-05T06:36:21.178125+0000","last_undegraded":"2024-02-05T06:36:21.178125+0000","last_fullsized":"2024-02-05T06:36:21.178125+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":4460,"log_dups_size":0,"ondisk_log_size":4460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:29.238213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292290560,"num_objects":4460,"num_object_clones":0,"num_object_copies":13380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4460,"num_write_kb":285440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":55,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.939584+0000","last_change":"2024-02-05T06:35:27.459919+0000","last_active":"2024-02-05T06:36:19.939584+0000","last_peered":"2024-02-05T06:36:19.939584+0000","last_clean":"2024-02-05T06:36:19.939584+0000","last_became_active":"2024-02-05T06:35:24.942766+0000","last_became_peered":"2024-02-05T06:35:24.942766+0000","last_unstale":"2024-02-05T06:36:19.939584+0000","last_undegraded":"2024-02-05T06:36:19.939584+0000","last_fullsized":"2024-02-05T06:36:19.939584+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:27.459848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:27.459848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:41.497954+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"92'9","reported_seq":146,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.924076+0000","last_change":"2024-02-05T06:34:52.283858+0000","last_active":"2024-02-05T06:36:19.924076+0000","last_peered":"2024-02-05T06:36:19.924076+0000","last_clean":"2024-02-05T06:36:19.924076+0000","last_became_active":"2024-02-05T06:33:33.588478+0000","last_became_peered":"2024-02-05T06:33:33.588478+0000","last_unstale":"2024-02-05T06:36:19.924076+0000","last_undegraded":"2024-02-05T06:36:19.924076+0000","last_fullsized":"2024-02-05T06:36:19.924076+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'9","last_scrub_stamp":"2024-02-05T06:34:52.283780+0000","last_deep_scrub":"92'9","last_deep_scrub_stamp":"2024-02-05T06:34:52.283780+0000","last_clean_scrub_stamp":"2024-02-05T06:34:52.283780+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.945476+0000","last_change":"2024-02-05T06:35:24.945148+0000","last_active":"2024-02-05T06:36:19.945476+0000","last_peered":"2024-02-05T06:36:19.945476+0000","last_clean":"2024-02-05T06:36:19.945476+0000","last_became_active":"2024-02-05T06:35:24.944848+0000","last_became_peered":"2024-02-05T06:35:24.944848+0000","last_unstale":"2024-02-05T06:36:19.945476+0000","last_undegraded":"2024-02-05T06:36:19.945476+0000","last_fullsized":"2024-02-05T06:36:19.945476+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:29.822634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:29.822634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"93'15","reported_seq":136,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-02-05T06:36:19.940078+0000","last_change":"2024-02-05T06:35:24.967852+0000","last_active":"2024-02-05T06:36:19.940078+0000","last_peered":"2024-02-05T06:36:19.940078+0000","last_clean":"2024-02-05T06:36:19.940078+0000","last_became_active":"2024-02-05T06:35:24.944760+0000","last_became_peered":"2024-02-05T06:35:24.944760+0000","last_unstale":"2024-02-05T06:36:19.940078+0000","last_undegraded":"2024-02-05T06:36:19.940078+0000","last_fullsized":"2024-02-05T06:36:19.940078+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'15","last_scrub_stamp":"2024-02-05T06:35:04.584657+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-02-05T06:35:04.584657+0000","last_clean_scrub_stamp":"2024-02-05T06:35:04.584657+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:36:29.351479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":4646699008,"num_objects":70903,"num_object_clones":0,"num_object_copies":212709,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":70903,"num_write_kb":4537792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13940228096,"data_stored":13940228096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69903,"ondisk_log_size":69903,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":164,"num_read_kb":159,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":4950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":686,"num_read_kb":443,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":2382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":446,"ondisk_log_size":446,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103206,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1744924,"kb_used_data":1691536,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92000228,"statfs":{"total":95995035648,"available":94208233472,"internally_reserved":0,"allocated":1732132864,"data_stored":1731683158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56499999999999995,"15min":0.56599999999999995},"min":{"1min":0.31,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.77700000000000002,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.23999999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60199999999999998,"15min":0.59999999999999998},"min":{"1min":0.40799999999999997,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.26700000000000002}]},{"osd":1,"last update":"Mon Feb 5 06:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.55900000000000005,"15min":0.56299999999999994},"min":{"1min":0.32800000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.36099999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":1.3200000000000001}]},{"osd":2,"last update":"Mon Feb 5 06:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.57099999999999995,"15min":0.57299999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.82099999999999995,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.57399999999999995,"15min":0.56799999999999995},"min":{"1min":0.40699999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.73999999999999999,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.46800000000000003}]},{"osd":3,"last update":"Mon Feb 5 06:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.59699999999999998,"15min":0.59899999999999998},"min":{"1min":0.38500000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.75800000000000001,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":3.9279999999999999},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.60599999999999998,"15min":0.60399999999999998},"min":{"1min":0.434,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":5.3419999999999996}]},{"osd":4,"last update":"Mon Feb 5 06:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.44600000000000001,"15min":0.44400000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.5,"15min":0.498},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.69299999999999995,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.29299999999999998}]},{"osd":5,"last update":"Mon Feb 5 06:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.51000000000000001,"15min":0.504},"min":{"1min":0.35999999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.78000000000000003,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.499},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.54900000000000004,"15min":0.54500000000000004},"min":{"1min":0.39100000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.71699999999999997,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.48899999999999999}]},{"osd":6,"last update":"Mon Feb 5 06:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.53800000000000003,"15min":0.53000000000000003},"min":{"1min":0.376,"5min":0.316,"15min":0.316},"max":{"1min":0.76600000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.54600000000000004,"15min":0.54200000000000004},"min":{"1min":0.41599999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.45000000000000001}]}]},{"osd":6,"up_from":49,"seq":210453397548,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2048532,"kb_used_data":1972936,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91696620,"statfs":{"total":95995035648,"available":93897338880,"internally_reserved":0,"allocated":2020286464,"data_stored":2019843941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:35:56 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.58599999999999997,"15min":0.56999999999999995},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":3.4369999999999998,"5min":3.4369999999999998,"15min":3.4369999999999998},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.54700000000000004,"15min":0.54300000000000004},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":1.2509999999999999}]},{"osd":1,"last update":"Mon Feb 5 06:35:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.52800000000000002,"15min":0.52500000000000002},"min":{"1min":0.34799999999999998,"5min":0.309,"15min":0.309},"max":{"1min":1.637,"5min":1.637,"15min":1.637},"last":0.497},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.56399999999999995,"15min":0.57799999999999996},"min":{"1min":0.317,"5min":0.255,"15min":0.255},"max":{"1min":0.89800000000000002,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.81100000000000005}]},{"osd":2,"last update":"Mon Feb 5 06:35:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.56999999999999995,"15min":0.57799999999999996},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.55700000000000005,"15min":0.55400000000000005},"min":{"1min":0.312,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.59799999999999998}]},{"osd":3,"last update":"Mon Feb 5 06:35:56 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.57699999999999996,"15min":0.55600000000000005},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":2.1339999999999999,"5min":2.1339999999999999,"15min":2.1339999999999999},"last":2.5150000000000001},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.64100000000000001,"15min":0.59599999999999997},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":4.2039999999999997,"5min":4.2039999999999997,"15min":4.2039999999999997},"last":0.51300000000000001}]},{"osd":4,"last update":"Mon Feb 5 06:35:56 2024","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.442,"15min":0.44700000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.68000000000000005,"5min":1.355,"15min":1.355},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.44800000000000001,"15min":0.42799999999999999},"min":{"1min":0.28299999999999997,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.39300000000000002}]},{"osd":5,"last update":"Mon Feb 5 06:35:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.46200000000000002,"15min":0.45600000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.747,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.501,"5min":0.46899999999999997,"15min":0.45500000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.78000000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.36299999999999999}]},{"osd":7,"last update":"Mon Feb 5 06:36:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.51700000000000002,"15min":0.52100000000000002},"min":{"1min":0.32600000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.75800000000000001,"5min":0.872,"15min":0.872},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.80200000000000005,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.61899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2049268,"kb_used_data":1977832,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91695884,"statfs":{"total":95995035648,"available":93896585216,"internally_reserved":0,"allocated":2025299968,"data_stored":2024826121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:35:52 2024","interfaces":[{"interface":"back","average":{"1min":4.3479999999999999,"5min":1.171,"15min":0.60799999999999998},"min":{"1min":0.252,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":67.421000000000006,"5min":67.421000000000006,"15min":67.421000000000006},"last":1.2829999999999999},{"interface":"front","average":{"1min":4.0679999999999996,"5min":1.1339999999999999,"15min":0.59899999999999998},"min":{"1min":0.23100000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":67.885000000000005,"5min":67.885000000000005,"15min":67.885000000000005},"last":1.897}]},{"osd":2,"last update":"Mon Feb 5 06:36:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":0.58199999999999996,"15min":0.45300000000000001},"min":{"1min":0.246,"5min":0.214,"15min":0.214},"max":{"1min":4.7510000000000003,"5min":4.7510000000000003,"15min":4.7510000000000003},"last":1.552},{"interface":"front","average":{"1min":1.0880000000000001,"5min":0.52900000000000003,"15min":0.42499999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.7050000000000001,"5min":3.7050000000000001,"15min":3.7050000000000001},"last":4.6310000000000002}]},{"osd":3,"last update":"Mon Feb 5 06:35:43 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.42699999999999999,"15min":0.42099999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":1.7869999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.53300000000000003,"15min":0.49299999999999999},"min":{"1min":0.30399999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":4.9980000000000002,"5min":4.9980000000000002,"15min":4.9980000000000002},"last":0.96299999999999997}]},{"osd":4,"last update":"Mon Feb 5 06:36:05 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.61699999999999999,"15min":0.57799999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":3.75,"5min":3.75,"15min":3.75},"last":0.45900000000000002},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.69099999999999995,"15min":0.61399999999999999},"min":{"1min":0.42499999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":5.2169999999999996,"5min":5.2169999999999996,"15min":5.2169999999999996},"last":1.637}]},{"osd":5,"last update":"Mon Feb 5 06:35:31 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.84299999999999997,"15min":0.95699999999999996},"min":{"1min":0.44400000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.83699999999999997,"5min":8.2509999999999994,"15min":8.2509999999999994},"last":1.004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.61699999999999999,"15min":0.63100000000000001},"min":{"1min":0.40100000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76100000000000001,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.39400000000000002}]},{"osd":6,"last update":"Mon Feb 5 06:35:55 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.68899999999999995,"15min":0.66500000000000004},"min":{"1min":0.42199999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":5.2359999999999998,"5min":5.2359999999999998,"15min":5.2359999999999998},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.69099999999999995,"15min":0.68500000000000005},"min":{"1min":0.49099999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":2.738,"5min":2.738,"15min":2.738},"last":1.0349999999999999}]},{"osd":7,"last update":"Mon Feb 5 06:36:20 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":0.73199999999999998,"15min":0.63800000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":5.1980000000000004,"5min":5.1980000000000004,"15min":5.1980000000000004},"last":0.49199999999999999},{"interface":"front","average":{"1min":1.1479999999999999,"5min":0.97899999999999998,"15min":1.0209999999999999},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":4.4870000000000001,"5min":8.2219999999999995,"15min":8.2219999999999995},"last":1.6759999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738443,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2328480,"kb_used_data":2261204,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91416672,"statfs":{"total":95995035648,"available":93610672128,"internally_reserved":0,"allocated":2315472896,"data_stored":2315018848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 06:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.443,"15min":0.35299999999999998},"min":{"1min":0.23499999999999999,"5min":0.188,"15min":0.188},"max":{"1min":3.855,"5min":4.2729999999999997,"15min":4.2729999999999997},"last":2.1320000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.42299999999999999,"15min":0.35499999999999998},"min":{"1min":0.26600000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":2.4470000000000001,"5min":2.4470000000000001,"15min":2.4470000000000001},"last":4.4909999999999997}]},{"osd":2,"last update":"Mon Feb 5 06:36:14 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":0.629,"15min":0.47699999999999998},"min":{"1min":0.32200000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":3.6989999999999998,"5min":3.6989999999999998,"15min":3.6989999999999998},"last":1.177},{"interface":"front","average":{"1min":1.1850000000000001,"5min":0.59999999999999998,"15min":0.47699999999999998},"min":{"1min":0.33100000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.113,"5min":3.113,"15min":3.113},"last":0.33400000000000002}]},{"osd":3,"last update":"Mon Feb 5 06:35:39 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.46800000000000003,"15min":0.46000000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.1830000000000001,"5min":4.3289999999999997,"15min":4.3289999999999997},"last":2.5110000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.47999999999999998,"15min":0.45900000000000002},"min":{"1min":0.26100000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":3.8130000000000002,"5min":3.8130000000000002,"15min":3.8130000000000002},"last":0.73699999999999999}]},{"osd":4,"last update":"Mon Feb 5 06:36:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":0.69899999999999995,"15min":0.629},"min":{"1min":0.39900000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":2.5619999999999998,"5min":2.5619999999999998,"15min":2.5619999999999998},"last":1.0640000000000001},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.71399999999999997,"15min":0.64000000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":4.2590000000000003,"5min":4.367,"15min":4.367},"last":2.2050000000000001}]},{"osd":5,"last update":"Mon Feb 5 06:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.61599999999999999,"15min":0.63200000000000001},"min":{"1min":0.41799999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63400000000000001,"15min":0.64400000000000002},"min":{"1min":0.44,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":1.0029999999999999}]},{"osd":6,"last update":"Mon Feb 5 06:35:52 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.72099999999999997,"15min":0.66500000000000004},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":4.2830000000000004,"5min":5.2220000000000004,"15min":5.2220000000000004},"last":2.1680000000000001},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.68500000000000005,"15min":0.66400000000000003},"min":{"1min":0.44800000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":3.3029999999999999,"5min":3.3029999999999999,"15min":3.3029999999999999},"last":0.39800000000000002}]},{"osd":7,"last update":"Mon Feb 5 06:36:20 2024","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":0.73999999999999999,"15min":0.64400000000000002},"min":{"1min":0.433,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":3.294,"5min":3.294,"15min":3.294},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.208,"5min":0.94099999999999995,"15min":0.94399999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":4.2300000000000004,"5min":5.2670000000000003,"15min":5.2670000000000003},"last":2.2349999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411392,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1761804,"kb_used_data":1704256,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":91983348,"statfs":{"total":95995035648,"available":94190948352,"internally_reserved":0,"allocated":1745158144,"data_stored":1744724211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:36:16 2024","interfaces":[{"interface":"back","average":{"1min":4.1909999999999998,"5min":1.1519999999999999,"15min":0.623},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":50.755000000000003,"5min":50.755000000000003,"15min":50.755000000000003},"last":3.9399999999999999},{"interface":"front","average":{"1min":4.157,"5min":1.1739999999999999,"15min":0.63800000000000001},"min":{"1min":0.192,"5min":0.188,"15min":0.188},"max":{"1min":50.802999999999997,"5min":50.802999999999997,"15min":50.802999999999997},"last":5.6799999999999997}]},{"osd":1,"last update":"Mon Feb 5 06:36:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":0.61199999999999999,"15min":0.46000000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":10.243,"5min":10.243,"15min":10.243},"last":2.9209999999999998},{"interface":"front","average":{"1min":1.204,"5min":0.61099999999999999,"15min":0.48299999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":5.5609999999999999,"5min":5.5609999999999999,"15min":5.5609999999999999},"last":4.3719999999999999}]},{"osd":3,"last update":"Mon Feb 5 06:35:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.48799999999999999,"15min":0.47299999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":3.6040000000000001},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.51500000000000001,"15min":0.49399999999999999},"min":{"1min":0.30099999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.1200000000000001,"5min":4.1200000000000001,"15min":4.1200000000000001},"last":3.3999999999999999}]},{"osd":4,"last update":"Mon Feb 5 06:36:04 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.66300000000000003,"15min":0.65100000000000002},"min":{"1min":0.42199999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":2.2930000000000001,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":2.968},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.66200000000000003,"15min":0.66500000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.837,"5min":1.837,"15min":1.837},"last":3.2280000000000002}]},{"osd":5,"last update":"Mon Feb 5 06:35:33 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.66800000000000004,"15min":0.69499999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.83799999999999997,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":3.1469999999999998},{"interface":"front","average":{"1min":0.621,"5min":0.65300000000000002,"15min":0.67500000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.874,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.74199999999999999}]},{"osd":6,"last update":"Mon Feb 5 06:35:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":1.3160000000000001,"15min":1.5860000000000001},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.8089999999999999,"5min":17.143999999999998,"15min":17.143999999999998},"last":3.194},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.71699999999999997,"15min":0.70699999999999996},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.752,"5min":1.752,"15min":1.752},"last":0.77700000000000002}]},{"osd":7,"last update":"Mon Feb 5 06:35:22 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.64800000000000002,"15min":0.65000000000000002},"min":{"1min":0.45600000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.83699999999999997,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":2.9990000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149755,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2881120,"kb_used_data":2805524,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":90864032,"statfs":{"total":95995035648,"available":93044768768,"internally_reserved":0,"allocated":2872856576,"data_stored":2872401626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:35:47 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.41399999999999998,"15min":0.40200000000000002},"min":{"1min":0.22900000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.8860000000000001,"5min":2.8860000000000001,"15min":2.8860000000000001},"last":2.0409999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.46000000000000002,"15min":0.45900000000000002},"min":{"1min":0.27800000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.9510000000000001,"5min":1.9510000000000001,"15min":1.9510000000000001},"last":4.6070000000000002}]},{"osd":1,"last update":"Mon Feb 5 06:35:47 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.46600000000000003,"15min":0.47499999999999998},"min":{"1min":0.251,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.7909999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":4.7919999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.45500000000000002,"15min":0.45700000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":2.161,"5min":2.161,"15min":2.161},"last":2.202}]},{"osd":2,"last update":"Mon Feb 5 06:35:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.52600000000000002,"15min":0.51000000000000001},"min":{"1min":0.29799999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":3.089,"5min":3.089,"15min":3.089},"last":2.2349999999999999},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.45100000000000001,"15min":0.46600000000000003},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.90600000000000003,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":2.0110000000000001}]},{"osd":4,"last update":"Mon Feb 5 06:36:10 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.63600000000000001,"15min":0.627},"min":{"1min":0.41899999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":3.4830000000000001,"5min":3.4830000000000001,"15min":3.4830000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":1.631,"5min":0.78600000000000003,"15min":0.66900000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":8.4060000000000006,"5min":8.4060000000000006,"15min":8.4060000000000006},"last":4.6550000000000002}]},{"osd":5,"last update":"Mon Feb 5 06:35:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.38100000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.79200000000000004,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.871},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.60799999999999998,"15min":0.624},"min":{"1min":0.45100000000000001,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.82899999999999996,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.47299999999999998}]},{"osd":6,"last update":"Mon Feb 5 06:35:58 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":0.71299999999999997,"15min":0.61499999999999999},"min":{"1min":0.45500000000000002,"5min":0.434,"15min":0.434},"max":{"1min":5.6970000000000001,"5min":5.6970000000000001,"15min":5.6970000000000001},"last":4.6890000000000001},{"interface":"front","average":{"1min":1.161,"5min":0.73499999999999999,"15min":0.67000000000000004},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":4.2830000000000004,"5min":4.2830000000000004,"15min":4.2830000000000004},"last":0.91000000000000003}]},{"osd":7,"last update":"Mon Feb 5 06:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.58699999999999997,"15min":0.58499999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.45800000000000002,"5min":0.434,"15min":0.434},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.94299999999999995}]}]},{"osd":4,"up_from":0,"seq":133143986229,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":40,"seq":171798691888,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1182492,"kb_used_data":1129104,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92562660,"statfs":{"total":95995035648,"available":94784163840,"internally_reserved":0,"allocated":1156202496,"data_stored":1155752431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.52200000000000002,"15min":0.53100000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.71999999999999997,"15min":0.60599999999999998},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.66900000000000004,"5min":17.509,"15min":17.509},"last":1.853}]},{"osd":1,"last update":"Mon Feb 5 06:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.55700000000000005,"15min":0.56100000000000005},"min":{"1min":0.38800000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":6.5869999999999997},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.53600000000000003,"15min":0.54200000000000004},"min":{"1min":0.36699999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":3.5800000000000001}]},{"osd":2,"last update":"Mon Feb 5 06:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.54100000000000004,"15min":0.54700000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53800000000000003,"15min":0.53600000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.54800000000000004}]},{"osd":3,"last update":"Mon Feb 5 06:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.54000000000000004,"15min":0.55400000000000005},"min":{"1min":0.35499999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":2.351},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.57399999999999995,"15min":0.57999999999999996},"min":{"1min":0.35599999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.74099999999999999,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":2.4279999999999999}]},{"osd":4,"last update":"Mon Feb 5 06:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.42899999999999999,"15min":0.42299999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.33700000000000002}]},{"osd":6,"last update":"Mon Feb 5 06:35:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46400000000000002,"15min":0.45100000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.61099999999999999,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.48499999999999999,"15min":0.47299999999999998},"min":{"1min":0.35399999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.39500000000000002}]},{"osd":7,"last update":"Mon Feb 5 06:36:17 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.46899999999999997,"15min":0.46600000000000003},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.57599999999999996,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.499,"15min":0.498},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.622,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.626}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2317746176,"data_stored":2317746176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2028470272,"data_stored":2028470272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1756102656,"data_stored":1756102656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2908553216,"data_stored":2908553216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1159987200,"data_stored":1159987200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2032599040,"data_stored":2032599040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1736769536,"data_stored":1736769536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T06:36:28.455 INFO:tasks.thrashosds.thrasher:Setting 3.18 to [5, 2, 4] 2024-02-05T06:36:28.455 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.18', '5', '2', '4'] 2024-02-05T06:36:28.455 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd pg-upmap 3.18 5 2 4 2024-02-05T06:36:28.548 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:28.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:28.601 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:28.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:28.702 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:28.748 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:28.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:28.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:28.850 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:28.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:28.878 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:28.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:28.972 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:36:28.978 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:29.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:29.054 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:29.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:29.155 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:29.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:28 smithi028 ceph-mon[29442]: 5.15 scrub starts 2024-02-05T06:36:29.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:28 smithi028 ceph-mon[29442]: 5.15 scrub ok 2024-02-05T06:36:29.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:28 smithi028 ceph-mon[29442]: pgmap v105: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 55 MiB/s wr, 886 op/s 2024-02-05T06:36:29.158 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:28 smithi028 ceph-mon[37984]: 5.15 scrub starts 2024-02-05T06:36:29.158 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:28 smithi028 ceph-mon[37984]: 5.15 scrub ok 2024-02-05T06:36:29.159 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:28 smithi028 ceph-mon[37984]: pgmap v105: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 55 MiB/s wr, 886 op/s 2024-02-05T06:36:29.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:29.204 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:29.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:29.223 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 52 16 79574 79558 95.5686 63.125 0.00531737 0.0104583 2024-02-05T06:36:29.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:29.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:29.357 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:29.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:29.459 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:29.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:28 smithi167 ceph-mon[32774]: 5.15 scrub starts 2024-02-05T06:36:29.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:28 smithi167 ceph-mon[32774]: 5.15 scrub ok 2024-02-05T06:36:29.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:28 smithi167 ceph-mon[32774]: pgmap v105: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 55 MiB/s wr, 886 op/s 2024-02-05T06:36:29.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:29.507 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:29.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:29.556 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T06:36:29.608 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:29.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:29.682 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:29.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:29.783 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:29.811 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:29.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:29.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:29.912 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:29.940 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:29.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:29.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:30.042 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:30.057 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:36:30.058 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-02-05T06:36:30.058 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-02-05T06:36:30.058 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-02-05T06:36:30.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.058 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14847.0:50545 7.9 e105/96 7:99c4e475:::benchmark_data_smithi028_77509_object50544:head v 105'3115, mlcod=105'3112)", 2024-02-05T06:36:30.058 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:36:04.110579+0000", 2024-02-05T06:36:30.058 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 25.939388202, 2024-02-05T06:36:30.058 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025837764999999999, 2024-02-05T06:36:30.058 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:36:30.058 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:36:30.058 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:36:30.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.059 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:36:30.059 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.110579+0000", 2024-02-05T06:36:30.059 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.059 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:36:30.059 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.110579+0000", 2024-02-05T06:36:30.059 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.059 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:36:30.059 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.110580+0000", 2024-02-05T06:36:30.059 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.265e-06 2024-02-05T06:36:30.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.060 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:36:30.060 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.110608+0000", 2024-02-05T06:36:30.060 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7914000000000001e-05 2024-02-05T06:36:30.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.060 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.060 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:36:30.060 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.110620+0000", 2024-02-05T06:36:30.060 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1871e-05 2024-02-05T06:36:30.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.060 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.060 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:36:30.060 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.110627+0000", 2024-02-05T06:36:30.060 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4970000000000004e-06 2024-02-05T06:36:30.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.061 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:36:30.061 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.112432+0000", 2024-02-05T06:36:30.061 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018050609999999999 2024-02-05T06:36:30.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.061 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:36:30.061 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.112453+0000", 2024-02-05T06:36:30.061 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0401e-05 2024-02-05T06:36:30.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.061 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:36:30.061 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.136122+0000", 2024-02-05T06:36:30.061 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023669504000000001 2024-02-05T06:36:30.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.062 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:36:30.062 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.136416+0000", 2024-02-05T06:36:30.062 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000294252 2024-02-05T06:36:30.062 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.062 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:30.062 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.062 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14847.0:50665 7.4 e105/96 7:26705de4:::benchmark_data_smithi028_77509_object50664:head v 105'3207, mlcod=105'3204)", 2024-02-05T06:36:30.062 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:36:04.211427+0000", 2024-02-05T06:36:30.062 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 25.838539826999998, 2024-02-05T06:36:30.062 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.030161962, 2024-02-05T06:36:30.062 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:36:30.063 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:36:30.063 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:36:30.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.063 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:36:30.063 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.211427+0000", 2024-02-05T06:36:30.063 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.063 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:36:30.063 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.211427+0000", 2024-02-05T06:36:30.063 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.063 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:36:30.063 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.211427+0000", 2024-02-05T06:36:30.063 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.01e-07 2024-02-05T06:36:30.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.064 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:36:30.064 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.211449+0000", 2024-02-05T06:36:30.064 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1906999999999999e-05 2024-02-05T06:36:30.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.064 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:36:30.064 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.211455+0000", 2024-02-05T06:36:30.064 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2630000000000003e-06 2024-02-05T06:36:30.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.064 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:36:30.065 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.211459+0000", 2024-02-05T06:36:30.065 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4479999999999996e-06 2024-02-05T06:36:30.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.065 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:36:30.065 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.214995+0000", 2024-02-05T06:36:30.065 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035363030000000002 2024-02-05T06:36:30.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.065 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:36:30.065 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.215015+0000", 2024-02-05T06:36:30.065 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9485000000000001e-05 2024-02-05T06:36:30.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.066 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:36:30.066 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.241532+0000", 2024-02-05T06:36:30.066 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.026517246000000001 2024-02-05T06:36:30.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.066 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:36:30.066 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.241589+0000", 2024-02-05T06:36:30.066 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6909e-05 2024-02-05T06:36:30.066 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.066 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:30.066 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.066 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:51155 7.b 7:d3c6ade6:::benchmark_data_smithi028_77509_object51154:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-02-05T06:36:30.066 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:36:04.619811+0000", 2024-02-05T06:36:30.067 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 25.430155858999999, 2024-02-05T06:36:30.067 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.026034607000000001, 2024-02-05T06:36:30.067 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:36:30.067 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:36:30.067 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:36:30.067 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:36:30.067 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:36:30.067 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 51155 2024-02-05T06:36:30.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.067 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:36:30.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.067 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:36:30.067 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.619811+0000", 2024-02-05T06:36:30.067 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.068 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.068 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:36:30.068 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.619811+0000", 2024-02-05T06:36:30.068 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.068 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.068 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:36:30.068 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.619812+0000", 2024-02-05T06:36:30.068 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3480000000000001e-06 2024-02-05T06:36:30.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.068 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.068 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:36:30.068 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.619835+0000", 2024-02-05T06:36:30.069 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2391e-05 2024-02-05T06:36:30.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.069 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.069 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:36:30.069 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.619845+0000", 2024-02-05T06:36:30.069 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0117e-05 2024-02-05T06:36:30.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.069 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.069 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:36:30.069 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.619853+0000", 2024-02-05T06:36:30.069 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7039999999999994e-06 2024-02-05T06:36:30.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.069 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.070 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:36:30.070 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.635766+0000", 2024-02-05T06:36:30.070 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015913344999999999 2024-02-05T06:36:30.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.070 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.070 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:36:30.070 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.635878+0000", 2024-02-05T06:36:30.070 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111634 2024-02-05T06:36:30.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.070 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.070 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-02-05T06:36:30.070 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.636006+0000", 2024-02-05T06:36:30.070 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000128257 2024-02-05T06:36:30.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.071 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.071 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:36:30.071 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.640448+0000", 2024-02-05T06:36:30.071 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0044423140000000002 2024-02-05T06:36:30.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.071 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.071 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:36:30.071 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.640692+0000", 2024-02-05T06:36:30.072 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000243813 2024-02-05T06:36:30.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.072 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.072 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:36:30.072 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.643652+0000", 2024-02-05T06:36:30.072 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029600659999999999 2024-02-05T06:36:30.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.072 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.072 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:36:30.072 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.645556+0000", 2024-02-05T06:36:30.072 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001904296 2024-02-05T06:36:30.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.072 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.073 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:36:30.073 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.645846+0000", 2024-02-05T06:36:30.073 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028932200000000001 2024-02-05T06:36:30.073 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.073 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:30.073 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.073 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.073 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:51390 7.b 7:d4306004:::benchmark_data_smithi028_77509_object51389:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-02-05T06:36:30.073 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:36:04.823764+0000", 2024-02-05T06:36:30.073 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 25.226202553, 2024-02-05T06:36:30.073 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.043931784000000001, 2024-02-05T06:36:30.073 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:36:30.073 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:36:30.073 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:36:30.074 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:36:30.074 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:36:30.074 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 51390 2024-02-05T06:36:30.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.074 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:36:30.074 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.074 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:36:30.074 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.823764+0000", 2024-02-05T06:36:30.074 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.074 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.074 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:36:30.074 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.823764+0000", 2024-02-05T06:36:30.074 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.075 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.075 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:36:30.075 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.823766+0000", 2024-02-05T06:36:30.075 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8929999999999999e-06 2024-02-05T06:36:30.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.075 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.075 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:36:30.075 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.823791+0000", 2024-02-05T06:36:30.075 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5106e-05 2024-02-05T06:36:30.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.075 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.075 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:36:30.075 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.823802+0000", 2024-02-05T06:36:30.075 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0822000000000001e-05 2024-02-05T06:36:30.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.076 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.076 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:36:30.076 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.823810+0000", 2024-02-05T06:36:30.076 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9310000000000008e-06 2024-02-05T06:36:30.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.076 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.076 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:36:30.076 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.823961+0000", 2024-02-05T06:36:30.076 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015111399999999999 2024-02-05T06:36:30.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.076 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.076 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:36:30.076 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.824079+0000", 2024-02-05T06:36:30.077 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000118237 2024-02-05T06:36:30.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.077 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-02-05T06:36:30.077 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.824212+0000", 2024-02-05T06:36:30.077 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013208499999999999 2024-02-05T06:36:30.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.077 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:36:30.077 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.826116+0000", 2024-02-05T06:36:30.077 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001904191 2024-02-05T06:36:30.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.077 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:36:30.077 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.826199+0000", 2024-02-05T06:36:30.078 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3567999999999997e-05 2024-02-05T06:36:30.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.078 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:36:30.078 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.828211+0000", 2024-02-05T06:36:30.078 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020114149999999999 2024-02-05T06:36:30.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.078 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:36:30.078 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.867651+0000", 2024-02-05T06:36:30.078 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.039440308 2024-02-05T06:36:30.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.078 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:36:30.078 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.867696+0000", 2024-02-05T06:36:30.079 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5114000000000002e-05 2024-02-05T06:36:30.079 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.079 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:30.079 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.079 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(client.14847.0:51390 7.b e106/96 ondisk, result = 0)", 2024-02-05T06:36:30.079 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:36:04.828096+0000", 2024-02-05T06:36:30.079 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 25.221871392000001, 2024-02-05T06:36:30.079 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.039610697, 2024-02-05T06:36:30.079 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:36:30.079 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-02-05T06:36:30.079 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:36:30.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.079 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:36:30.080 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.828096+0000", 2024-02-05T06:36:30.080 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.080 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:36:30.080 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.828096+0000", 2024-02-05T06:36:30.080 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.080 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:36:30.080 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.828097+0000", 2024-02-05T06:36:30.080 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.435e-06 2024-02-05T06:36:30.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.081 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:36:30.081 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.828105+0000", 2024-02-05T06:36:30.081 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7149999999999998e-06 2024-02-05T06:36:30.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.081 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.081 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:36:30.081 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.828117+0000", 2024-02-05T06:36:30.081 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2514000000000001e-05 2024-02-05T06:36:30.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.081 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.081 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:36:30.081 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.828125+0000", 2024-02-05T06:36:30.081 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3710000000000002e-06 2024-02-05T06:36:30.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.082 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.082 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:36:30.082 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.828196+0000", 2024-02-05T06:36:30.082 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1932000000000002e-05 2024-02-05T06:36:30.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.082 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.082 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:36:30.082 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.828205+0000", 2024-02-05T06:36:30.082 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5310000000000001e-06 2024-02-05T06:36:30.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.082 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.082 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:36:30.082 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.867706+0000", 2024-02-05T06:36:30.082 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.039501199000000001 2024-02-05T06:36:30.082 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.082 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:30.083 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.083 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.083 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14847.0:51386 7.1 e106/96 7:8e1fb5ab:::benchmark_data_smithi028_77509_object51385:head v 106'3274, mlcod=106'3272)", 2024-02-05T06:36:30.083 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:36:04.828324+0000", 2024-02-05T06:36:30.083 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 25.221642707000001, 2024-02-05T06:36:30.083 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.044105518000000003, 2024-02-05T06:36:30.083 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:36:30.083 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:36:30.083 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:36:30.083 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.083 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:36:30.083 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.828324+0000", 2024-02-05T06:36:30.083 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.084 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:36:30.084 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.828324+0000", 2024-02-05T06:36:30.084 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.084 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:36:30.084 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.828325+0000", 2024-02-05T06:36:30.084 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2359999999999999e-06 2024-02-05T06:36:30.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.084 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:36:30.084 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.828369+0000", 2024-02-05T06:36:30.084 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3096999999999998e-05 2024-02-05T06:36:30.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.085 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:36:30.085 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.867618+0000", 2024-02-05T06:36:30.085 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.039249306999999997 2024-02-05T06:36:30.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.085 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:36:30.085 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.867630+0000", 2024-02-05T06:36:30.085 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1834999999999999e-05 2024-02-05T06:36:30.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.085 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:36:30.085 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.867755+0000", 2024-02-05T06:36:30.085 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000124845 2024-02-05T06:36:30.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.086 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:36:30.086 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.867774+0000", 2024-02-05T06:36:30.086 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9026e-05 2024-02-05T06:36:30.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.086 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:36:30.086 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.872357+0000", 2024-02-05T06:36:30.086 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0045837639999999997 2024-02-05T06:36:30.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.086 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:36:30.086 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.872430+0000", 2024-02-05T06:36:30.087 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2408000000000002e-05 2024-02-05T06:36:30.087 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.087 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:30.087 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.087 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14847.0:51396 7.4 e106/96 7:2a646d39:::benchmark_data_smithi028_77509_object51395:head v 106'3261, mlcod=106'3259)", 2024-02-05T06:36:30.087 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:36:04.828764+0000", 2024-02-05T06:36:30.087 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 25.221202452, 2024-02-05T06:36:30.087 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.043740988000000001, 2024-02-05T06:36:30.087 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:36:30.087 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:36:30.087 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:36:30.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.087 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:36:30.087 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.828764+0000", 2024-02-05T06:36:30.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.088 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:36:30.088 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.828764+0000", 2024-02-05T06:36:30.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.088 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:36:30.088 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.828766+0000", 2024-02-05T06:36:30.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4100000000000001e-06 2024-02-05T06:36:30.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.088 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:36:30.088 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.829904+0000", 2024-02-05T06:36:30.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001137711 2024-02-05T06:36:30.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.089 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:36:30.089 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.867614+0000", 2024-02-05T06:36:30.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.037710293999999998 2024-02-05T06:36:30.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.089 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:36:30.089 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.867629+0000", 2024-02-05T06:36:30.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5574e-05 2024-02-05T06:36:30.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.089 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:36:30.089 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.867694+0000", 2024-02-05T06:36:30.090 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4988999999999997e-05 2024-02-05T06:36:30.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.090 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:36:30.090 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.867721+0000", 2024-02-05T06:36:30.090 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6857000000000001e-05 2024-02-05T06:36:30.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.090 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:36:30.090 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.872465+0000", 2024-02-05T06:36:30.090 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047440770000000002 2024-02-05T06:36:30.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.090 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:36:30.090 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:04.872505+0000", 2024-02-05T06:36:30.090 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0076000000000001e-05 2024-02-05T06:36:30.091 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.091 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:30.091 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.091 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:52211 7.b 7:d8d40488:::benchmark_data_smithi028_77509_object52210:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-02-05T06:36:30.091 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:36:05.429277+0000", 2024-02-05T06:36:30.091 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 24.620690003, 2024-02-05T06:36:30.091 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.026271826000000002, 2024-02-05T06:36:30.091 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:36:30.091 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:36:30.091 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:36:30.091 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:36:30.091 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:36:30.092 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 52211 2024-02-05T06:36:30.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.092 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:36:30.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.092 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:36:30.092 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:05.429277+0000", 2024-02-05T06:36:30.092 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.092 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:36:30.092 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:05.429277+0000", 2024-02-05T06:36:30.092 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.092 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:36:30.093 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:05.429278+0000", 2024-02-05T06:36:30.093 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2200000000000003e-07 2024-02-05T06:36:30.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.093 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:36:30.093 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:05.429312+0000", 2024-02-05T06:36:30.093 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4100999999999998e-05 2024-02-05T06:36:30.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.093 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:36:30.093 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:05.429319+0000", 2024-02-05T06:36:30.093 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2620000000000001e-06 2024-02-05T06:36:30.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.093 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:36:30.094 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:05.429326+0000", 2024-02-05T06:36:30.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.776e-06 2024-02-05T06:36:30.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.094 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:36:30.094 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:05.441315+0000", 2024-02-05T06:36:30.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011989468 2024-02-05T06:36:30.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.094 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:36:30.094 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:05.441434+0000", 2024-02-05T06:36:30.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000118325 2024-02-05T06:36:30.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.095 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-02-05T06:36:30.095 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:05.441552+0000", 2024-02-05T06:36:30.095 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011842599999999999 2024-02-05T06:36:30.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.095 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:36:30.095 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:05.443676+0000", 2024-02-05T06:36:30.095 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021234259999999999 2024-02-05T06:36:30.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.095 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:36:30.095 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:05.449525+0000", 2024-02-05T06:36:30.095 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0058499149999999998 2024-02-05T06:36:30.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.096 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:36:30.096 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:05.455458+0000", 2024-02-05T06:36:30.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0059322389999999997 2024-02-05T06:36:30.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.096 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:36:30.096 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:05.455512+0000", 2024-02-05T06:36:30.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4475000000000003e-05 2024-02-05T06:36:30.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.096 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:36:30.096 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:05.455549+0000", 2024-02-05T06:36:30.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6591000000000003e-05 2024-02-05T06:36:30.096 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.097 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:30.097 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.097 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:73853 7.b 7:dc374d29:::benchmark_data_smithi028_77509_object73852:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-02-05T06:36:30.097 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:36:23.843674+0000", 2024-02-05T06:36:30.097 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 6.2062933820000001, 2024-02-05T06:36:30.097 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029808713000000001, 2024-02-05T06:36:30.097 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:36:30.097 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:36:30.097 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:36:30.097 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:36:30.097 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:36:30.097 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 73853 2024-02-05T06:36:30.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.098 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:36:30.098 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.098 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:36:30.098 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:23.843674+0000", 2024-02-05T06:36:30.098 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.099 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:36:30.099 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:23.843674+0000", 2024-02-05T06:36:30.099 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.099 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:36:30.099 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:23.843676+0000", 2024-02-05T06:36:30.099 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7099999999999999e-06 2024-02-05T06:36:30.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.099 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:36:30.100 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:23.843699+0000", 2024-02-05T06:36:30.100 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2410999999999999e-05 2024-02-05T06:36:30.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.100 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:36:30.100 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:23.843714+0000", 2024-02-05T06:36:30.100 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4929000000000001e-05 2024-02-05T06:36:30.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.100 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:36:30.100 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:23.843721+0000", 2024-02-05T06:36:30.100 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.294e-06 2024-02-05T06:36:30.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.101 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:36:30.101 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:23.843785+0000", 2024-02-05T06:36:30.101 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3663000000000004e-05 2024-02-05T06:36:30.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.101 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:36:30.101 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:23.843919+0000", 2024-02-05T06:36:30.101 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013410900000000001 2024-02-05T06:36:30.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.101 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-02-05T06:36:30.101 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:23.844077+0000", 2024-02-05T06:36:30.101 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000158159 2024-02-05T06:36:30.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.102 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:36:30.102 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:23.846416+0000", 2024-02-05T06:36:30.102 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002338803 2024-02-05T06:36:30.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.102 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:36:30.102 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:23.873118+0000", 2024-02-05T06:36:30.102 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.026702802000000001 2024-02-05T06:36:30.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.102 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:36:30.102 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:23.873375+0000", 2024-02-05T06:36:30.103 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00025619899999999998 2024-02-05T06:36:30.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.103 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:36:30.103 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:23.873437+0000", 2024-02-05T06:36:30.103 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2136000000000003e-05 2024-02-05T06:36:30.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.103 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:36:30.103 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:23.873482+0000", 2024-02-05T06:36:30.103 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5497999999999997e-05 2024-02-05T06:36:30.103 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.103 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:30.103 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.104 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:74420 7.b 7:d30695ad:::benchmark_data_smithi028_77509_object74419:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-02-05T06:36:30.104 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:36:24.353821+0000", 2024-02-05T06:36:30.104 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.6961462479999998, 2024-02-05T06:36:30.104 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027569079, 2024-02-05T06:36:30.104 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:36:30.104 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:36:30.104 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:36:30.104 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:36:30.104 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:36:30.104 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 74420 2024-02-05T06:36:30.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.104 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:36:30.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.105 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:36:30.105 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.353821+0000", 2024-02-05T06:36:30.105 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.105 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:36:30.105 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.353821+0000", 2024-02-05T06:36:30.105 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.105 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:36:30.105 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.353823+0000", 2024-02-05T06:36:30.106 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5490000000000001e-06 2024-02-05T06:36:30.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.106 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:36:30.106 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.353846+0000", 2024-02-05T06:36:30.106 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2336000000000001e-05 2024-02-05T06:36:30.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.106 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:36:30.106 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.353867+0000", 2024-02-05T06:36:30.106 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1399999999999998e-05 2024-02-05T06:36:30.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.106 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:36:30.106 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.353875+0000", 2024-02-05T06:36:30.107 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2609999999999993e-06 2024-02-05T06:36:30.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.107 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:36:30.107 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.363185+0000", 2024-02-05T06:36:30.107 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0093093780000000001 2024-02-05T06:36:30.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.107 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:36:30.107 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.363680+0000", 2024-02-05T06:36:30.107 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00049588400000000004 2024-02-05T06:36:30.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.107 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-02-05T06:36:30.107 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.364268+0000", 2024-02-05T06:36:30.108 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00058739000000000003 2024-02-05T06:36:30.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.108 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:36:30.108 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.368647+0000", 2024-02-05T06:36:30.108 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0043788350000000002 2024-02-05T06:36:30.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.108 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:36:30.108 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.368865+0000", 2024-02-05T06:36:30.108 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00021801599999999999 2024-02-05T06:36:30.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.108 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:36:30.109 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.380643+0000", 2024-02-05T06:36:30.109 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011778209 2024-02-05T06:36:30.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.109 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:36:30.109 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.381347+0000", 2024-02-05T06:36:30.109 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00070363399999999997 2024-02-05T06:36:30.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.109 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:36:30.109 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.381390+0000", 2024-02-05T06:36:30.109 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3186999999999998e-05 2024-02-05T06:36:30.109 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.109 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:30.109 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.110 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:74421 7.b 7:d58b04bc:::benchmark_data_smithi028_77509_object74420:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-02-05T06:36:30.110 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:36:24.353929+0000", 2024-02-05T06:36:30.110 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.696037993, 2024-02-05T06:36:30.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027581069, 2024-02-05T06:36:30.110 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:36:30.110 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:36:30.110 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:36:30.110 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:36:30.110 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:36:30.110 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 74421 2024-02-05T06:36:30.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.110 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:36:30.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.111 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:36:30.111 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.353929+0000", 2024-02-05T06:36:30.111 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.111 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:36:30.111 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.353929+0000", 2024-02-05T06:36:30.111 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.111 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:36:30.111 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.353930+0000", 2024-02-05T06:36:30.111 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2099999999999995e-07 2024-02-05T06:36:30.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.112 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:36:30.112 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.353951+0000", 2024-02-05T06:36:30.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1670000000000001e-05 2024-02-05T06:36:30.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.112 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:36:30.112 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.353961+0000", 2024-02-05T06:36:30.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3409999999999992e-06 2024-02-05T06:36:30.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.112 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:36:30.112 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.353967+0000", 2024-02-05T06:36:30.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4200000000000004e-06 2024-02-05T06:36:30.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.113 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:36:30.113 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.369029+0000", 2024-02-05T06:36:30.113 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015062157 2024-02-05T06:36:30.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.113 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:36:30.113 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.369633+0000", 2024-02-05T06:36:30.113 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00060390300000000004 2024-02-05T06:36:30.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.113 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-02-05T06:36:30.113 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.370180+0000", 2024-02-05T06:36:30.113 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00054626999999999996 2024-02-05T06:36:30.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.114 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:36:30.114 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.380418+0000", 2024-02-05T06:36:30.114 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010238229 2024-02-05T06:36:30.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.114 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:36:30.114 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.380462+0000", 2024-02-05T06:36:30.114 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4158e-05 2024-02-05T06:36:30.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.114 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:36:30.114 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.381398+0000", 2024-02-05T06:36:30.114 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00093574100000000003 2024-02-05T06:36:30.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:36:30.115 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.381475+0000", 2024-02-05T06:36:30.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7251999999999998e-05 2024-02-05T06:36:30.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:36:30.115 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.381510+0000", 2024-02-05T06:36:30.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5107000000000001e-05 2024-02-05T06:36:30.115 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.115 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:30.116 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.116 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14847.0:74468 7.9 e112/96 7:9a923895:::benchmark_data_smithi028_77509_object74467:head v 112'4604, mlcod=112'4600)", 2024-02-05T06:36:30.116 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:36:24.413915+0000", 2024-02-05T06:36:30.116 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 5.6360518759999998, 2024-02-05T06:36:30.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027579507999999999, 2024-02-05T06:36:30.116 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:36:30.116 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:36:30.116 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:36:30.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.116 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:36:30.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.413915+0000", 2024-02-05T06:36:30.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.117 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:36:30.117 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.413915+0000", 2024-02-05T06:36:30.117 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.117 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:36:30.117 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.413916+0000", 2024-02-05T06:36:30.117 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.325e-06 2024-02-05T06:36:30.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.117 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:36:30.117 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.413941+0000", 2024-02-05T06:36:30.118 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4422000000000001e-05 2024-02-05T06:36:30.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.118 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:36:30.118 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.413951+0000", 2024-02-05T06:36:30.118 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0526000000000001e-05 2024-02-05T06:36:30.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.118 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:36:30.118 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.413960+0000", 2024-02-05T06:36:30.118 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3129999999999999e-06 2024-02-05T06:36:30.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.118 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:36:30.118 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.414323+0000", 2024-02-05T06:36:30.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00036358400000000002 2024-02-05T06:36:30.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:36:30.119 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.414349+0000", 2024-02-05T06:36:30.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5514999999999999e-05 2024-02-05T06:36:30.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:36:30.119 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.441215+0000", 2024-02-05T06:36:30.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.026866734 2024-02-05T06:36:30.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.120 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:36:30.120 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:24.441495+0000", 2024-02-05T06:36:30.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000279089 2024-02-05T06:36:30.120 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.120 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:30.120 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.120 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14847.0:77144 7.2 e114/96 7:45f278fb:::benchmark_data_smithi028_77509_object77143:head v 114'4779, mlcod=114'4777)", 2024-02-05T06:36:30.120 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:36:26.977734+0000", 2024-02-05T06:36:30.120 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 3.072233368, 2024-02-05T06:36:30.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.030385731999999999, 2024-02-05T06:36:30.120 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:36:30.120 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:36:30.121 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:36:30.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:36:30.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:26.977734+0000", 2024-02-05T06:36:30.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:36:30.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:26.977734+0000", 2024-02-05T06:36:30.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:36:30.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:26.977736+0000", 2024-02-05T06:36:30.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0870000000000001e-06 2024-02-05T06:36:30.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.122 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:36:30.122 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:26.977761+0000", 2024-02-05T06:36:30.122 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5548999999999999e-05 2024-02-05T06:36:30.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.122 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:36:30.122 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:26.977773+0000", 2024-02-05T06:36:30.122 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1793999999999999e-05 2024-02-05T06:36:30.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.122 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:36:30.122 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:26.977781+0000", 2024-02-05T06:36:30.122 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3780000000000001e-06 2024-02-05T06:36:30.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:36:30.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:26.981595+0000", 2024-02-05T06:36:30.123 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0038131969999999999 2024-02-05T06:36:30.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:36:30.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:26.981621+0000", 2024-02-05T06:36:30.123 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6067e-05 2024-02-05T06:36:30.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:36:30.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:27.008024+0000", 2024-02-05T06:36:30.123 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.026403395 2024-02-05T06:36:30.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.124 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:36:30.124 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:27.008119+0000", 2024-02-05T06:36:30.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5265000000000004e-05 2024-02-05T06:36:30.124 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.124 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:30.124 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.124 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14847.0:79321 7.0 e115/96 7:08a0a285:::benchmark_data_smithi028_77509_object79320:head v 115'4919, mlcod=115'4917)", 2024-02-05T06:36:30.124 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:36:28.945531+0000", 2024-02-05T06:36:30.124 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 1.1044360360000001, 2024-02-05T06:36:30.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025360170000000001, 2024-02-05T06:36:30.124 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:36:30.125 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:36:30.125 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:36:30.125 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.125 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:36:30.125 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:28.945531+0000", 2024-02-05T06:36:30.125 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:30.126 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:30.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:30.127 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.127 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:36:30.127 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:28.945531+0000", 2024-02-05T06:36:30.127 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.127 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.127 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:36:30.128 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:28.945532+0000", 2024-02-05T06:36:30.128 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4300000000000001e-07 2024-02-05T06:36:30.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.128 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:36:30.128 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:28.945625+0000", 2024-02-05T06:36:30.128 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3657000000000001e-05 2024-02-05T06:36:30.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.128 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:36:30.128 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:28.945634+0000", 2024-02-05T06:36:30.128 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2039999999999997e-06 2024-02-05T06:36:30.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.131 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:36:30.131 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:28.945650+0000", 2024-02-05T06:36:30.131 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6019e-05 2024-02-05T06:36:30.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.131 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:36:30.132 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:28.945979+0000", 2024-02-05T06:36:30.132 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00032926 2024-02-05T06:36:30.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.132 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:36:30.132 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:28.945997+0000", 2024-02-05T06:36:30.132 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8346999999999999e-05 2024-02-05T06:36:30.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.132 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:36:30.132 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:28.970742+0000", 2024-02-05T06:36:30.132 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024744964000000001 2024-02-05T06:36:30.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.133 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:36:30.133 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:28.970891+0000", 2024-02-05T06:36:30.133 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014877600000000001 2024-02-05T06:36:30.133 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.133 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:30.133 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.133 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14847.0:79658 7.d e115/96 7:b41b1509:::benchmark_data_smithi028_77509_object79657:head v 115'4872, mlcod=115'4869)", 2024-02-05T06:36:30.133 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:36:29.293682+0000", 2024-02-05T06:36:30.133 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.75628483700000004, 2024-02-05T06:36:30.133 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.031096071999999999, 2024-02-05T06:36:30.133 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:36:30.133 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:36:30.134 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:36:30.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.134 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:36:30.134 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.293682+0000", 2024-02-05T06:36:30.134 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.134 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:36:30.134 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.293682+0000", 2024-02-05T06:36:30.134 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.134 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:36:30.134 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.293683+0000", 2024-02-05T06:36:30.135 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2810000000000001e-06 2024-02-05T06:36:30.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.135 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:36:30.135 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.293748+0000", 2024-02-05T06:36:30.135 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4422000000000001e-05 2024-02-05T06:36:30.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.135 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:36:30.135 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.293757+0000", 2024-02-05T06:36:30.135 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6670000000000008e-06 2024-02-05T06:36:30.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.135 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:36:30.135 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.293765+0000", 2024-02-05T06:36:30.136 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2969999999999998e-06 2024-02-05T06:36:30.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.136 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:36:30.136 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.296338+0000", 2024-02-05T06:36:30.136 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025735060000000001 2024-02-05T06:36:30.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.136 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:36:30.136 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.296358+0000", 2024-02-05T06:36:30.136 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9256e-05 2024-02-05T06:36:30.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.136 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:36:30.137 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.320838+0000", 2024-02-05T06:36:30.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.024480900999999999 2024-02-05T06:36:30.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.137 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:36:30.137 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.324778+0000", 2024-02-05T06:36:30.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039397420000000004 2024-02-05T06:36:30.137 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.137 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:30.137 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.137 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14847.0:79668 7.d e115/96 7:b75ebbbb:::benchmark_data_smithi028_77509_object79667:head v 115'4873, mlcod=115'4870)", 2024-02-05T06:36:30.137 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:36:29.301354+0000", 2024-02-05T06:36:30.137 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.74861322299999999, 2024-02-05T06:36:30.138 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.028386198000000001, 2024-02-05T06:36:30.138 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:36:30.138 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:36:30.138 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:36:30.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:36:30.138 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.301354+0000", 2024-02-05T06:36:30.138 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:36:30.138 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.301354+0000", 2024-02-05T06:36:30.138 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.139 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:36:30.139 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.301355+0000", 2024-02-05T06:36:30.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3880000000000001e-06 2024-02-05T06:36:30.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.139 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:36:30.139 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.301387+0000", 2024-02-05T06:36:30.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1690000000000003e-05 2024-02-05T06:36:30.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.139 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:36:30.139 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.301398+0000", 2024-02-05T06:36:30.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0954999999999999e-05 2024-02-05T06:36:30.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.140 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:36:30.140 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.301405+0000", 2024-02-05T06:36:30.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3239999999999997e-06 2024-02-05T06:36:30.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.140 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:36:30.140 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.301471+0000", 2024-02-05T06:36:30.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5617000000000006e-05 2024-02-05T06:36:30.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.140 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:36:30.140 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.301490+0000", 2024-02-05T06:36:30.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9718e-05 2024-02-05T06:36:30.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:36:30.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.327529+0000", 2024-02-05T06:36:30.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.026038133000000001 2024-02-05T06:36:30.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:36:30.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.329740+0000", 2024-02-05T06:36:30.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002211373 2024-02-05T06:36:30.141 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.141 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:30.141 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.142 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14847.0:79672 7.d e115/96 7:b02b92a8:::benchmark_data_smithi028_77509_object79671:head v 115'4874, mlcod=115'4870)", 2024-02-05T06:36:30.142 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:36:29.303112+0000", 2024-02-05T06:36:30.142 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.74685514099999994, 2024-02-05T06:36:30.142 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.026672313999999999, 2024-02-05T06:36:30.142 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:36:30.142 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:36:30.142 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:36:30.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:36:30.142 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.303112+0000", 2024-02-05T06:36:30.142 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:36:30.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.303112+0000", 2024-02-05T06:36:30.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:36:30.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.303114+0000", 2024-02-05T06:36:30.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9250000000000002e-06 2024-02-05T06:36:30.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:36:30.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.303160+0000", 2024-02-05T06:36:30.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6610999999999998e-05 2024-02-05T06:36:30.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.144 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:36:30.144 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.303176+0000", 2024-02-05T06:36:30.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5990000000000001e-05 2024-02-05T06:36:30.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.144 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:36:30.144 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.303185+0000", 2024-02-05T06:36:30.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8040000000000007e-06 2024-02-05T06:36:30.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.144 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:36:30.144 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.312123+0000", 2024-02-05T06:36:30.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0089383410000000007 2024-02-05T06:36:30.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.145 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:36:30.145 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.312145+0000", 2024-02-05T06:36:30.145 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1489e-05 2024-02-05T06:36:30.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.145 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:36:30.145 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.329747+0000", 2024-02-05T06:36:30.145 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017601694000000001 2024-02-05T06:36:30.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.145 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:36:30.145 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.329784+0000", 2024-02-05T06:36:30.145 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7459999999999997e-05 2024-02-05T06:36:30.146 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.146 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:30.146 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.146 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14847.0:80304 7.9 e115/96 7:9c3e536c:::benchmark_data_smithi028_77509_object80303:head v 115'4974, mlcod=115'4972)", 2024-02-05T06:36:30.146 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:36:29.920713+0000", 2024-02-05T06:36:30.146 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.12925376399999999, 2024-02-05T06:36:30.146 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032916097999999998, 2024-02-05T06:36:30.146 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:36:30.146 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:36:30.146 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:36:30.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.146 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:36:30.146 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.920713+0000", 2024-02-05T06:36:30.146 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.147 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:36:30.147 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.920713+0000", 2024-02-05T06:36:30.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.147 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:36:30.147 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.920715+0000", 2024-02-05T06:36:30.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6649999999999999e-06 2024-02-05T06:36:30.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.147 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:36:30.147 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.920744+0000", 2024-02-05T06:36:30.148 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9062999999999999e-05 2024-02-05T06:36:30.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.148 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:36:30.148 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.920761+0000", 2024-02-05T06:36:30.148 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7036000000000001e-05 2024-02-05T06:36:30.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.148 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:36:30.148 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.920771+0000", 2024-02-05T06:36:30.148 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.696e-06 2024-02-05T06:36:30.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.148 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:36:30.148 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.920983+0000", 2024-02-05T06:36:30.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00021233800000000001 2024-02-05T06:36:30.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:36:30.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.921041+0000", 2024-02-05T06:36:30.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8131000000000003e-05 2024-02-05T06:36:30.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:36:30.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.953394+0000", 2024-02-05T06:36:30.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.032353407000000001 2024-02-05T06:36:30.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:36:30.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.953629+0000", 2024-02-05T06:36:30.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00023476200000000001 2024-02-05T06:36:30.150 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.150 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:30.150 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.150 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14847.0:80305 7.9 e115/96 7:9c8ce5a9:::benchmark_data_smithi028_77509_object80304:head v 115'4975, mlcod=115'4972)", 2024-02-05T06:36:30.150 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:36:29.921992+0000", 2024-02-05T06:36:30.150 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.127974807, 2024-02-05T06:36:30.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.039755619999999998, 2024-02-05T06:36:30.150 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:36:30.150 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:36:30.150 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:36:30.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.150 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:36:30.151 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.921992+0000", 2024-02-05T06:36:30.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.151 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:36:30.151 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.921992+0000", 2024-02-05T06:36:30.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.151 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:36:30.151 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.921993+0000", 2024-02-05T06:36:30.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1850000000000001e-06 2024-02-05T06:36:30.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.151 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:36:30.152 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.922019+0000", 2024-02-05T06:36:30.152 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.584e-05 2024-02-05T06:36:30.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.152 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:36:30.152 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.922033+0000", 2024-02-05T06:36:30.152 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4212e-05 2024-02-05T06:36:30.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.152 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:36:30.152 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.922041+0000", 2024-02-05T06:36:30.152 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0809999999999994e-06 2024-02-05T06:36:30.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.153 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:36:30.153 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.923782+0000", 2024-02-05T06:36:30.153 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017400740000000001 2024-02-05T06:36:30.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.153 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:36:30.153 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.941238+0000", 2024-02-05T06:36:30.153 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.017456816 2024-02-05T06:36:30.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.153 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:36:30.153 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.961643+0000", 2024-02-05T06:36:30.153 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020405162000000001 2024-02-05T06:36:30.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.154 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:36:30.154 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.961748+0000", 2024-02-05T06:36:30.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010425 2024-02-05T06:36:30.154 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.154 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:30.154 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.154 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14847.0:80308 7.9 e115/96 7:9ce3ca33:::benchmark_data_smithi028_77509_object80307:head v 115'4976, mlcod=115'4972)", 2024-02-05T06:36:30.154 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:36:29.930967+0000", 2024-02-05T06:36:30.154 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.118999487, 2024-02-05T06:36:30.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.030919378000000001, 2024-02-05T06:36:30.154 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:36:30.154 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:36:30.155 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:36:30.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:36:30.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.930967+0000", 2024-02-05T06:36:30.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:36:30.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.930967+0000", 2024-02-05T06:36:30.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:36:30.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:36:30.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.930969+0000", 2024-02-05T06:36:30.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3209999999999999e-06 2024-02-05T06:36:30.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.156 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:36:30.156 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.930995+0000", 2024-02-05T06:36:30.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5919999999999999e-05 2024-02-05T06:36:30.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.156 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:36:30.156 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.931006+0000", 2024-02-05T06:36:30.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1103e-05 2024-02-05T06:36:30.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.156 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:36:30.156 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.931013+0000", 2024-02-05T06:36:30.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2699999999999999e-06 2024-02-05T06:36:30.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.157 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:36:30.157 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.950043+0000", 2024-02-05T06:36:30.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019030154000000001 2024-02-05T06:36:30.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.157 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:36:30.157 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.950066+0000", 2024-02-05T06:36:30.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2816999999999999e-05 2024-02-05T06:36:30.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.157 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:36:30.157 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.961829+0000", 2024-02-05T06:36:30.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011762761 2024-02-05T06:36:30.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:36:30.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:36:30.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:36:30.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.961887+0000", 2024-02-05T06:36:30.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8032000000000003e-05 2024-02-05T06:36:30.158 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.158 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:30.158 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.158 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:36:30.158 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:36:30.158 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:36:30.223 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 53 15 80603 80588 94.9802 64.375 0.0408893 0.0105228 2024-02-05T06:36:30.228 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:30.259 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:30.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:30.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:30.354 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:29 smithi028 ceph-mon[37984]: 4.f scrub starts 2024-02-05T06:36:30.355 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:29 smithi028 ceph-mon[37984]: 4.f scrub ok 2024-02-05T06:36:30.355 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:29 smithi028 ceph-mon[37984]: 4.1e scrub starts 2024-02-05T06:36:30.355 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:29 smithi028 ceph-mon[37984]: 4.1e scrub ok 2024-02-05T06:36:30.355 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:29 smithi028 ceph-mon[37984]: 6.15 scrub starts 2024-02-05T06:36:30.355 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:29 smithi028 ceph-mon[37984]: 6.15 scrub ok 2024-02-05T06:36:30.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:29 smithi028 ceph-mon[29442]: 4.f scrub starts 2024-02-05T06:36:30.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:29 smithi028 ceph-mon[29442]: 4.f scrub ok 2024-02-05T06:36:30.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:29 smithi028 ceph-mon[29442]: 4.1e scrub starts 2024-02-05T06:36:30.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:29 smithi028 ceph-mon[29442]: 4.1e scrub ok 2024-02-05T06:36:30.355 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:29 smithi028 ceph-mon[29442]: 6.15 scrub starts 2024-02-05T06:36:30.356 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:29 smithi028 ceph-mon[29442]: 6.15 scrub ok 2024-02-05T06:36:30.361 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:30.403 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:30.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:30.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:30.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:29 smithi167 ceph-mon[32774]: 4.f scrub starts 2024-02-05T06:36:30.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:29 smithi167 ceph-mon[32774]: 4.f scrub ok 2024-02-05T06:36:30.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:29 smithi167 ceph-mon[32774]: 4.1e scrub starts 2024-02-05T06:36:30.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:29 smithi167 ceph-mon[32774]: 4.1e scrub ok 2024-02-05T06:36:30.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:29 smithi167 ceph-mon[32774]: 6.15 scrub starts 2024-02-05T06:36:30.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:29 smithi167 ceph-mon[32774]: 6.15 scrub ok 2024-02-05T06:36:30.505 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:30.533 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:30.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:30.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:30.635 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:30.674 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:30.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:30.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:30.776 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:30.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:30.806 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:30.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:30.907 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:30.935 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:30.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:30.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:31.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:31.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:31.090 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:31.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:31.191 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:31.218 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:31.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:31.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:31.223 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 54 16 81731 81715 94.5258 70.4375 0.00570036 0.0105746 2024-02-05T06:36:31.252 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:31 smithi028 ceph-mon[29442]: 5.c scrub starts 2024-02-05T06:36:31.252 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:31 smithi028 ceph-mon[29442]: 5.c scrub ok 2024-02-05T06:36:31.252 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:31 smithi028 ceph-mon[29442]: 4.6 scrub starts 2024-02-05T06:36:31.252 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:31 smithi028 ceph-mon[29442]: 4.6 scrub ok 2024-02-05T06:36:31.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:31 smithi028 ceph-mon[29442]: pgmap v106: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 55 MiB/s wr, 886 op/s 2024-02-05T06:36:31.253 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:31 smithi028 ceph-mon[37984]: 5.c scrub starts 2024-02-05T06:36:31.253 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:31 smithi028 ceph-mon[37984]: 5.c scrub ok 2024-02-05T06:36:31.253 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:31 smithi028 ceph-mon[37984]: 4.6 scrub starts 2024-02-05T06:36:31.253 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:31 smithi028 ceph-mon[37984]: 4.6 scrub ok 2024-02-05T06:36:31.253 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:31 smithi028 ceph-mon[37984]: pgmap v106: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 55 MiB/s wr, 886 op/s 2024-02-05T06:36:31.320 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:31.348 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:31.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:31.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:31.363 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:36:31.450 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:31.475 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:31.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:31.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:31.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:31 smithi167 ceph-mon[32774]: 5.c scrub starts 2024-02-05T06:36:31.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:31 smithi167 ceph-mon[32774]: 5.c scrub ok 2024-02-05T06:36:31.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:31 smithi167 ceph-mon[32774]: 4.6 scrub starts 2024-02-05T06:36:31.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:31 smithi167 ceph-mon[32774]: 4.6 scrub ok 2024-02-05T06:36:31.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:31 smithi167 ceph-mon[32774]: pgmap v106: 148 pgs: 148 active+clean; 4.7 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 55 MiB/s wr, 886 op/s 2024-02-05T06:36:31.577 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:31.607 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:31.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:31.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:31.709 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:31.738 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:31.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:31.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:31.814 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-05T06:36:31.840 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:31.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:31.914 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:31.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:32.015 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:32.044 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:32.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:32.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:32.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:32 smithi028 ceph-mon[29442]: 4.12 scrub starts 2024-02-05T06:36:32.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:32 smithi028 ceph-mon[29442]: 4.12 scrub ok 2024-02-05T06:36:32.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:32 smithi028 ceph-mon[29442]: 6.11 scrub starts 2024-02-05T06:36:32.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:32 smithi028 ceph-mon[29442]: 6.11 scrub ok 2024-02-05T06:36:32.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:32 smithi028 ceph-mon[29442]: 6.f scrub starts 2024-02-05T06:36:32.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:32 smithi028 ceph-mon[29442]: 6.f scrub ok 2024-02-05T06:36:32.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:32 smithi028 ceph-mon[29442]: 3.1c scrub starts 2024-02-05T06:36:32.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:32 smithi028 ceph-mon[29442]: 3.1c scrub ok 2024-02-05T06:36:32.126 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:32 smithi028 ceph-mon[29442]: 3.12 deep-scrub starts 2024-02-05T06:36:32.127 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:32 smithi028 ceph-mon[37984]: 4.12 scrub starts 2024-02-05T06:36:32.127 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:32 smithi028 ceph-mon[37984]: 4.12 scrub ok 2024-02-05T06:36:32.127 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:32 smithi028 ceph-mon[37984]: 6.11 scrub starts 2024-02-05T06:36:32.127 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:32 smithi028 ceph-mon[37984]: 6.11 scrub ok 2024-02-05T06:36:32.127 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:32 smithi028 ceph-mon[37984]: 6.f scrub starts 2024-02-05T06:36:32.127 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:32 smithi028 ceph-mon[37984]: 6.f scrub ok 2024-02-05T06:36:32.127 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:32 smithi028 ceph-mon[37984]: 3.1c scrub starts 2024-02-05T06:36:32.127 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:32 smithi028 ceph-mon[37984]: 3.1c scrub ok 2024-02-05T06:36:32.127 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:32 smithi028 ceph-mon[37984]: 3.12 deep-scrub starts 2024-02-05T06:36:32.146 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:32.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:32.214 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:32.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:32.223 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 55 16 82773 82757 93.9913 65.125 0.0095935 0.0106339 2024-02-05T06:36:32.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:32.369 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:32.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:32.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:32.455 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:32 smithi167 ceph-mon[32774]: 4.12 scrub starts 2024-02-05T06:36:32.455 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:32 smithi167 ceph-mon[32774]: 4.12 scrub ok 2024-02-05T06:36:32.455 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:32 smithi167 ceph-mon[32774]: 6.11 scrub starts 2024-02-05T06:36:32.455 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:32 smithi167 ceph-mon[32774]: 6.11 scrub ok 2024-02-05T06:36:32.455 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:32 smithi167 ceph-mon[32774]: 6.f scrub starts 2024-02-05T06:36:32.455 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:32 smithi167 ceph-mon[32774]: 6.f scrub ok 2024-02-05T06:36:32.455 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:32 smithi167 ceph-mon[32774]: 3.1c scrub starts 2024-02-05T06:36:32.455 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:32 smithi167 ceph-mon[32774]: 3.1c scrub ok 2024-02-05T06:36:32.455 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:32 smithi167 ceph-mon[32774]: 3.12 deep-scrub starts 2024-02-05T06:36:32.470 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:32.497 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:32.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:32.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:32.503 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:36:32.533 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:36:32.578 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:36:32.599 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:32.628 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:36:32.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:32.656 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:32.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:32.674 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:36:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:36:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:36:32.674 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:36:32.721 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:36:32.729 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:36:32 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:36:32.758 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:32.759 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:36:32.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:32.785 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:32.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:32.798 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:36:32.836 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:36:32.848 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:36:32.848 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:36:32.848 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:36:32.848 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:36:32.848 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:36:32.848 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:36:32.848 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:36:32.848 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:36:32.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:32.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:32.930 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:32.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:33.031 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:33.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:33.060 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:33.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:33.161 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:33.189 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:33.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:33.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:33.224 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 56 16 83697 83681 93.3444 57.75 0.00924235 0.0107076 2024-02-05T06:36:33.291 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:33.300 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:33 smithi028 ceph-mon[29442]: 3.12 deep-scrub ok 2024-02-05T06:36:33.300 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:33 smithi028 ceph-mon[29442]: 5.a scrub starts 2024-02-05T06:36:33.300 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:33 smithi028 ceph-mon[29442]: 5.a scrub ok 2024-02-05T06:36:33.300 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:33 smithi028 ceph-mon[29442]: 6.1a scrub starts 2024-02-05T06:36:33.301 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:33 smithi028 ceph-mon[29442]: 6.1a scrub ok 2024-02-05T06:36:33.301 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:33 smithi028 ceph-mon[29442]: 5.3 scrub starts 2024-02-05T06:36:33.301 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:33 smithi028 ceph-mon[29442]: pgmap v107: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.4 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-02-05T06:36:33.301 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:33 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:36:33.301 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:33 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2734938664' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:36:33.301 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:33 smithi028 ceph-mon[29442]: 5.9 scrub starts 2024-02-05T06:36:33.301 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:33 smithi028 ceph-mon[29442]: 5.9 scrub ok 2024-02-05T06:36:33.301 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:33 smithi028 ceph-mon[37984]: 3.12 deep-scrub ok 2024-02-05T06:36:33.301 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:33 smithi028 ceph-mon[37984]: 5.a scrub starts 2024-02-05T06:36:33.301 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:33 smithi028 ceph-mon[37984]: 5.a scrub ok 2024-02-05T06:36:33.301 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:33 smithi028 ceph-mon[37984]: 6.1a scrub starts 2024-02-05T06:36:33.302 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:33 smithi028 ceph-mon[37984]: 6.1a scrub ok 2024-02-05T06:36:33.302 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:33 smithi028 ceph-mon[37984]: 5.3 scrub starts 2024-02-05T06:36:33.302 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:33 smithi028 ceph-mon[37984]: pgmap v107: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.4 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-02-05T06:36:33.302 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:33 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:36:33.302 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:33 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2734938664' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:36:33.302 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:33 smithi028 ceph-mon[37984]: 5.9 scrub starts 2024-02-05T06:36:33.302 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:33 smithi028 ceph-mon[37984]: 5.9 scrub ok 2024-02-05T06:36:33.336 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:33.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:33.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:33.438 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:33.466 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:33.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:33.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:33.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:33 smithi167 ceph-mon[32774]: 3.12 deep-scrub ok 2024-02-05T06:36:33.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:33 smithi167 ceph-mon[32774]: 5.a scrub starts 2024-02-05T06:36:33.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:33 smithi167 ceph-mon[32774]: 5.a scrub ok 2024-02-05T06:36:33.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:33 smithi167 ceph-mon[32774]: 6.1a scrub starts 2024-02-05T06:36:33.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:33 smithi167 ceph-mon[32774]: 6.1a scrub ok 2024-02-05T06:36:33.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:33 smithi167 ceph-mon[32774]: 5.3 scrub starts 2024-02-05T06:36:33.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:33 smithi167 ceph-mon[32774]: pgmap v107: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.4 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-02-05T06:36:33.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:33 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:36:33.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:33 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2734938664' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:36:33.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:33 smithi167 ceph-mon[32774]: 5.9 scrub starts 2024-02-05T06:36:33.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:33 smithi167 ceph-mon[32774]: 5.9 scrub ok 2024-02-05T06:36:33.568 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:33.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:33.612 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:33.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:33.713 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:33.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:33.754 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:33.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:33.855 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:33.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:33.897 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:33.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:33.999 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:34.026 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:34.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:34.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:34.129 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:34.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:34.181 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:34.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:34.224 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 57 16 84672 84656 92.776 60.9375 0.00258108 0.0107717 2024-02-05T06:36:34.282 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:34.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:34.319 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:34.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:34.381 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:34 smithi167 ceph-mon[32774]: 5.3 scrub ok 2024-02-05T06:36:34.381 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:34 smithi167 ceph-mon[32774]: 6.1 scrub starts 2024-02-05T06:36:34.381 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:34 smithi167 ceph-mon[32774]: 6.1 scrub ok 2024-02-05T06:36:34.381 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:34 smithi167 ceph-mon[32774]: 6.a scrub starts 2024-02-05T06:36:34.382 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:34 smithi167 ceph-mon[32774]: 6.a scrub ok 2024-02-05T06:36:34.382 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:34 smithi167 ceph-mon[32774]: 3.e scrub starts 2024-02-05T06:36:34.382 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:34 smithi167 ceph-mon[32774]: 3.e scrub ok 2024-02-05T06:36:34.420 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:34.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:34 smithi028 ceph-mon[37984]: 5.3 scrub ok 2024-02-05T06:36:34.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:34 smithi028 ceph-mon[37984]: 6.1 scrub starts 2024-02-05T06:36:34.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:34 smithi028 ceph-mon[37984]: 6.1 scrub ok 2024-02-05T06:36:34.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:34 smithi028 ceph-mon[37984]: 6.a scrub starts 2024-02-05T06:36:34.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:34 smithi028 ceph-mon[37984]: 6.a scrub ok 2024-02-05T06:36:34.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:34 smithi028 ceph-mon[37984]: 3.e scrub starts 2024-02-05T06:36:34.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:34 smithi028 ceph-mon[37984]: 3.e scrub ok 2024-02-05T06:36:34.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:34 smithi028 ceph-mon[29442]: 5.3 scrub ok 2024-02-05T06:36:34.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:34 smithi028 ceph-mon[29442]: 6.1 scrub starts 2024-02-05T06:36:34.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:34 smithi028 ceph-mon[29442]: 6.1 scrub ok 2024-02-05T06:36:34.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:34 smithi028 ceph-mon[29442]: 6.a scrub starts 2024-02-05T06:36:34.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:34 smithi028 ceph-mon[29442]: 6.a scrub ok 2024-02-05T06:36:34.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:34 smithi028 ceph-mon[29442]: 3.e scrub starts 2024-02-05T06:36:34.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:34 smithi028 ceph-mon[29442]: 3.e scrub ok 2024-02-05T06:36:34.460 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:34.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:34.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:34.562 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:34.592 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:34.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:34.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:34.694 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:34.741 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:34.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:34.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:34.843 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:34.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:34.883 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:34.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:34.984 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:35.024 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:35.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:35.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:35.126 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:35.156 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:35.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:35.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:35.224 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 58 16 85743 85727 92.3306 66.9375 0.0155964 0.010826 2024-02-05T06:36:35.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:35 smithi167 ceph-mon[32774]: pgmap v108: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-02-05T06:36:35.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:35 smithi167 ceph-mon[32774]: 4.e deep-scrub starts 2024-02-05T06:36:35.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:35 smithi167 ceph-mon[32774]: 4.e deep-scrub ok 2024-02-05T06:36:35.259 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:35.297 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:35.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:35.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:35.399 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:35.443 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:35.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:35.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:35.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:35 smithi028 ceph-mon[37984]: pgmap v108: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-02-05T06:36:35.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:35 smithi028 ceph-mon[37984]: 4.e deep-scrub starts 2024-02-05T06:36:35.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:35 smithi028 ceph-mon[37984]: 4.e deep-scrub ok 2024-02-05T06:36:35.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:35 smithi028 ceph-mon[29442]: pgmap v108: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s 2024-02-05T06:36:35.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:35 smithi028 ceph-mon[29442]: 4.e deep-scrub starts 2024-02-05T06:36:35.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:35 smithi028 ceph-mon[29442]: 4.e deep-scrub ok 2024-02-05T06:36:35.545 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:35.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:35.593 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:35.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:35.695 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:35.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:35.747 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:35.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:35.849 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:35.879 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:35.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:35.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:35.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:36.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:36.019 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:36.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:36.120 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:36.152 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:36.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:36.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:36.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:36 smithi028 ceph-mon[37984]: 4.4 scrub starts 2024-02-05T06:36:36.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:36 smithi028 ceph-mon[37984]: 4.4 scrub ok 2024-02-05T06:36:36.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:36 smithi028 ceph-mon[29442]: 4.4 scrub starts 2024-02-05T06:36:36.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:36 smithi028 ceph-mon[29442]: 4.4 scrub ok 2024-02-05T06:36:36.224 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 59 16 86858 86842 91.9467 69.6875 0.0126014 0.0108698 2024-02-05T06:36:36.253 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:36.280 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:36.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:36.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:36.382 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:36.409 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:36.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:36.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:36.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:36 smithi167 ceph-mon[32774]: 4.4 scrub starts 2024-02-05T06:36:36.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:36 smithi167 ceph-mon[32774]: 4.4 scrub ok 2024-02-05T06:36:36.511 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:36.538 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:36.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:36.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:36.639 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:36.669 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:36.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:36.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:36.771 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:36.813 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:36.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:36.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:36.914 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:36.944 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:36.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:36.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:37.045 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:37.084 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:37.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:37.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:37.116 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:37 smithi028 ceph-mon[37984]: 5.5 scrub starts 2024-02-05T06:36:37.116 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:37 smithi028 ceph-mon[37984]: 5.5 scrub ok 2024-02-05T06:36:37.117 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:37 smithi028 ceph-mon[37984]: pgmap v109: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-02-05T06:36:37.117 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:37 smithi028 ceph-mon[37984]: 4.13 scrub starts 2024-02-05T06:36:37.117 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:37 smithi028 ceph-mon[37984]: 4.13 scrub ok 2024-02-05T06:36:37.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:37 smithi028 ceph-mon[29442]: 5.5 scrub starts 2024-02-05T06:36:37.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:37 smithi028 ceph-mon[29442]: 5.5 scrub ok 2024-02-05T06:36:37.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:37 smithi028 ceph-mon[29442]: pgmap v109: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-02-05T06:36:37.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:37 smithi028 ceph-mon[29442]: 4.13 scrub starts 2024-02-05T06:36:37.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:37 smithi028 ceph-mon[29442]: 4.13 scrub ok 2024-02-05T06:36:37.188 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:37.214 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:37.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:37.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:37.224 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-02-05T06:36:37.222532+0000 min lat: 0.0013072 max lat: 0.145154 avg lat: 0.0109297 2024-02-05T06:36:37.224 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T06:36:37.224 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 60 16 87858 87842 91.456 62.5 0.0153799 0.0109297 2024-02-05T06:36:37.316 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:37.342 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:37.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:37.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:37.444 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:37.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:37 smithi167 ceph-mon[32774]: 5.5 scrub starts 2024-02-05T06:36:37.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:37 smithi167 ceph-mon[32774]: 5.5 scrub ok 2024-02-05T06:36:37.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:37 smithi167 ceph-mon[32774]: pgmap v109: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-02-05T06:36:37.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:37 smithi167 ceph-mon[32774]: 4.13 scrub starts 2024-02-05T06:36:37.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:37 smithi167 ceph-mon[32774]: 4.13 scrub ok 2024-02-05T06:36:37.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:37.490 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:37.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:37.591 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:37.616 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:37.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:37.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:37.719 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:37.745 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:37.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:37.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:37.847 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:37.890 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:37.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:37.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:37.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:38.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:38.030 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:38.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:38.081 INFO:teuthology.orchestra.run.smithi028.stderr:set 3.18 pg_upmap mapping to [5,2,4] 2024-02-05T06:36:38.087 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-02-05T06:36:38.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:38.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:38.176 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:38.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:38.224 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 61 16 88806 88790 90.9281 59.25 0.00703985 0.0109923 2024-02-05T06:36:38.277 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:38.305 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:38.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:38.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:38.407 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:38.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:38.436 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:38.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:38.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:38 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/829288098' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.18", "id": [5, 2, 4]}]: dispatch 2024-02-05T06:36:38.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:38 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:36:38.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:38 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:36:38.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:38 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2851588849' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:36:38.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:38 smithi028 ceph-mon[37984]: from='client.24808 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:36:38.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:38 smithi028 ceph-mon[37984]: 5.19 scrub starts 2024-02-05T06:36:38.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:38 smithi028 ceph-mon[37984]: 5.19 scrub ok 2024-02-05T06:36:38.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:38 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/829288098' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.18", "id": [5, 2, 4]}]: dispatch 2024-02-05T06:36:38.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:38 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:36:38.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:38 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:36:38.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:38 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2851588849' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:36:38.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:38 smithi028 ceph-mon[29442]: from='client.24808 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:36:38.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:38 smithi028 ceph-mon[29442]: 5.19 scrub starts 2024-02-05T06:36:38.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:38 smithi028 ceph-mon[29442]: 5.19 scrub ok 2024-02-05T06:36:38.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:38 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/829288098' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.18", "id": [5, 2, 4]}]: dispatch 2024-02-05T06:36:38.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:38 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:36:38.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:38 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:36:38.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:38 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2851588849' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:36:38.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:38 smithi167 ceph-mon[32774]: from='client.24808 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:36:38.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:38 smithi167 ceph-mon[32774]: 5.19 scrub starts 2024-02-05T06:36:38.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:38 smithi167 ceph-mon[32774]: 5.19 scrub ok 2024-02-05T06:36:38.537 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:38.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:38.593 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:38.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:38.694 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:38.724 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:38.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:38.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:38.826 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:38.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:38.855 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:38.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:38.956 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:38.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:38.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:39.100 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:39.132 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:39.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:39.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:39.224 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 62 16 89833 89817 90.4969 64.1875 0.0141767 0.0110448 2024-02-05T06:36:39.233 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:39.289 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:39.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:39.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:39.322 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:39 smithi028 ceph-mon[37984]: 3.b scrub starts 2024-02-05T06:36:39.322 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:39 smithi028 ceph-mon[37984]: 3.b scrub ok 2024-02-05T06:36:39.322 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:39 smithi028 ceph-mon[37984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:36:39.322 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:39 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/829288098' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.18", "id": [5, 2, 4]}]': finished 2024-02-05T06:36:39.322 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:39 smithi028 ceph-mon[37984]: from='client.24808 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:36:39.322 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:39 smithi028 ceph-mon[37984]: osdmap e116: 8 total, 8 up, 7 in 2024-02-05T06:36:39.322 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:39 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:39.322 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:39 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:39.323 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:39 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:36:39.323 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:39 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:36:39.323 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:39 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:39.323 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:39 smithi028 ceph-mon[37984]: pgmap v111: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-02-05T06:36:39.323 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:39 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:36:39.323 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:39 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:36:39.323 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:39 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:36:39.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:39 smithi028 ceph-mon[29442]: 3.b scrub starts 2024-02-05T06:36:39.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:39 smithi028 ceph-mon[29442]: 3.b scrub ok 2024-02-05T06:36:39.323 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:39 smithi028 ceph-mon[29442]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:36:39.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:39 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/829288098' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.18", "id": [5, 2, 4]}]': finished 2024-02-05T06:36:39.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:39 smithi028 ceph-mon[29442]: from='client.24808 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:36:39.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:39 smithi028 ceph-mon[29442]: osdmap e116: 8 total, 8 up, 7 in 2024-02-05T06:36:39.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:39 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:39.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:39 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:39.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:39 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:36:39.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:39 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:36:39.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:39 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:39.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:39 smithi028 ceph-mon[29442]: pgmap v111: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-02-05T06:36:39.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:39 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:36:39.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:39 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:36:39.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:39 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:36:39.341 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:39 smithi167 ceph-mon[32774]: 3.b scrub starts 2024-02-05T06:36:39.341 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:39 smithi167 ceph-mon[32774]: 3.b scrub ok 2024-02-05T06:36:39.341 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:39 smithi167 ceph-mon[32774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:36:39.341 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:39 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/829288098' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.18", "id": [5, 2, 4]}]': finished 2024-02-05T06:36:39.342 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:39 smithi167 ceph-mon[32774]: from='client.24808 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:36:39.342 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:39 smithi167 ceph-mon[32774]: osdmap e116: 8 total, 8 up, 7 in 2024-02-05T06:36:39.342 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:39 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:39.342 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:39 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:39.342 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:39 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:36:39.342 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:39 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:36:39.342 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:39 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:39.342 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:39 smithi167 ceph-mon[32774]: pgmap v111: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-02-05T06:36:39.342 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:39 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:36:39.342 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:39 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:36:39.342 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:39 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:36:39.391 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:39.434 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:39.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:39.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:39.536 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:39.578 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:39.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:39.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:39.680 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:39.708 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:39.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:39.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:39.810 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:39.834 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:39.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:39.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:39.936 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:39.978 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:39.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:39.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:40.080 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:40.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:40.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:40.224 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:40.225 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 63 16 90857 90841 90.0764 64 0.023159 0.011097 2024-02-05T06:36:40.253 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:40.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:40.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:40.354 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:40.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:40.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:40.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:40 smithi028 ceph-mon[29442]: osdmap e117: 8 total, 8 up, 7 in 2024-02-05T06:36:40.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:40 smithi028 ceph-mon[37984]: osdmap e117: 8 total, 8 up, 7 in 2024-02-05T06:36:40.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:40 smithi167 ceph-mon[32774]: osdmap e117: 8 total, 8 up, 7 in 2024-02-05T06:36:40.496 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:40.541 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:40.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:40.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:40.643 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:40.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:40.681 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:40.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:40.721 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:36:40.782 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:40.810 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:40.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:40.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:40.911 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:40.947 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:40.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:40.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:41.049 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:41.077 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:41.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:41.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:41.178 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:41.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:41.207 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:41.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:41.225 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 64 16 91975 91959 89.7603 69.875 0.012131 0.0111356 2024-02-05T06:36:41.308 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:41.339 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:41.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:41.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:41.401 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:41 smithi028 ceph-mon[29442]: pgmap v113: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 40 MiB/s wr, 642 op/s 2024-02-05T06:36:41.401 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:41 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:36:41.402 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:41 smithi028 ceph-mon[37984]: pgmap v113: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 40 MiB/s wr, 642 op/s 2024-02-05T06:36:41.402 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:41 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:36:41.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:41.479 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:41.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:41 smithi167 ceph-mon[32774]: pgmap v113: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 40 MiB/s wr, 642 op/s 2024-02-05T06:36:41.479 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:41 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:36:41.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:41.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:41.581 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:41.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:41.606 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:41.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:41.707 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:41.712 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T06:36:41.748 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:41.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:41.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:41.850 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:41.882 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:41.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:41.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:41.984 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:42.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:42.026 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:42.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:42.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:42.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:42.174 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:42.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:42.225 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 65 16 92965 92949 89.3314 61.875 0.00432349 0.0111891 2024-02-05T06:36:42.275 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:42.311 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:42.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:42.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:42.412 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:42.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:42.451 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:42.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:42.552 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:42.582 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:42.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:42.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:42.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:42.727 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:42.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:42.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:42.730 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:36:42 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:36:42.763 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:36:42.791 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:36:42 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:36:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:36:42.829 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:42.854 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:42.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:42.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:42.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:43.015 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:43.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:43.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:43.117 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:43.145 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:43.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:43.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:43.225 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 66 16 93891 93875 88.8549 57.875 0.0139375 0.0112484 2024-02-05T06:36:43.247 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:43.277 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:43.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:43.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:43.326 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:43 smithi167 ceph-mon[32774]: pgmap v114: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-02-05T06:36:43.326 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:43 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:36:43.379 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:43.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:43.416 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:43.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:43.517 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:43.550 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:43.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:43.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:43.651 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:43.680 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:43.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:43.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:43.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:43 smithi028 ceph-mon[29442]: pgmap v114: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-02-05T06:36:43.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:43 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:36:43.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:43 smithi028 ceph-mon[37984]: pgmap v114: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-02-05T06:36:43.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:43 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:36:43.782 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:43.813 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:43.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:43.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:43.915 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:43.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:43.941 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:43.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:44.042 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:44.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:44.069 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:44.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:44.074 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T06:36:44.074 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T06:36:44.074 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-02-05T06:36:44.074 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd in 4 2024-02-05T06:36:44.170 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:44.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:44.207 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:44.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:44.225 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 67 16 94872 94856 88.4438 61.3125 0.00620877 0.0112963 2024-02-05T06:36:44.309 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:44.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:44.363 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:44.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:44.464 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:44.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:44.504 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:44.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:44.605 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:44.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:44.629 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:44.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:44.730 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:44.757 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:44.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:44.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:44.859 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:44.901 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:44.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:44.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:44.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:44 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:44.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:44 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:44.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:44 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:44.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:44 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:44.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:44 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:44.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:44 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:45.003 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:45.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:45.032 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:45.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:45.133 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:45.167 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:45.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:45.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:45.225 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 68 16 95835 95819 88.0284 60.1875 0.00864307 0.0113551 2024-02-05T06:36:45.269 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:45.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:45.307 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:45.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:45.408 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:45.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:45.436 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:45.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:45.537 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:45.580 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:45.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:45.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:45.682 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:45.723 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:45.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:45.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:45.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:45 smithi167 ceph-mon[32774]: pgmap v115: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 62 MiB/s wr, 999 op/s 2024-02-05T06:36:45.825 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:45.851 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:45.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:45.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:45.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:45 smithi028 ceph-mon[37984]: pgmap v115: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 62 MiB/s wr, 999 op/s 2024-02-05T06:36:45.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:45 smithi028 ceph-mon[29442]: pgmap v115: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 62 MiB/s wr, 999 op/s 2024-02-05T06:36:45.953 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:45.980 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:45.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:45.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:46.082 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:46.110 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:46.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:46.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:46.213 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:46.225 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 69 16 96838 96822 87.6612 62.6875 0.0213843 0.0114021 2024-02-05T06:36:46.285 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:36:46.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:46.311 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:46.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:46.412 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:46.461 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:46.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:46.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:46.563 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:46.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:46.609 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:46.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:46.637 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:36:46.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:46.763 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:46.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:46.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:46.864 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:46.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:46.921 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:46.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:47.022 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:47.050 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:47.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:47.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:47.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:47.204 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:47.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:47.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:47.226 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 70 16 97894 97878 87.3517 66 0.0185085 0.0114429 2024-02-05T06:36:47.307 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:47.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:47.355 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:47.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:47.456 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:47.468 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:36:47.469 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:36:47.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:47.482 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:47.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:47.584 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:47.641 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:47.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:47.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:47.742 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:47.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:47.772 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:47.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:47.873 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:47.901 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:47.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:47.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:47.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:47 smithi028 ceph-mon[29442]: pgmap v116: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 56 MiB/s wr, 892 op/s 2024-02-05T06:36:47.923 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:47 smithi028 ceph-mon[37984]: pgmap v116: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 56 MiB/s wr, 892 op/s 2024-02-05T06:36:47.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:47 smithi167 ceph-mon[32774]: pgmap v116: 148 pgs: 148 active+clean; 5.7 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 56 MiB/s wr, 892 op/s 2024-02-05T06:36:48.003 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:48.045 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:48.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:48.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:48.147 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:48.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:48.176 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:48.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:48.226 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 71 16 98940 98924 87.0422 65.375 0.0164431 0.0114838 2024-02-05T06:36:48.277 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:48.305 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:48.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:48.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:48.407 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:48.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:48.437 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:48.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:48.538 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:48.576 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:48.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:48.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:48.677 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:48.703 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:48.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:48.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:48.805 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:48.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:48.852 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:48.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:48.956 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:48.984 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:48.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:48.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:49.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:49.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:49.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:49.177 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:36:49.226 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-05T06:36:49.227 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 72 16 100085 100069 86.8269 71.5625 0.0163674 0.0115119 2024-02-05T06:36:49.230 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:49.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:49.278 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:49.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:49.380 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:49.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:49.442 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:49.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:49.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:49.604 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:49.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:49.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:49.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:49 smithi028 ceph-mon[29442]: pgmap v117: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 62 MiB/s wr, 999 op/s 2024-02-05T06:36:49.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:49 smithi028 ceph-mon[37984]: pgmap v117: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 62 MiB/s wr, 999 op/s 2024-02-05T06:36:49.706 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:49.734 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:49.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:49.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:49.836 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:49.894 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:49.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:49.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:49.979 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:49 smithi167 ceph-mon[32774]: pgmap v117: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 62 MiB/s wr, 999 op/s 2024-02-05T06:36:49.996 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:50.025 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:50.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:50.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:50.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:50.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:50.204 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:50.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:50.227 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 73 16 100959 100943 86.3853 54.625 0.0112989 0.0115701 2024-02-05T06:36:50.307 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:50.356 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:50.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:50.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:50.458 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:50.497 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:50.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:50.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:50.598 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:36:50.599 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:50.626 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:50.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:50.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:50.728 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:50.769 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:50.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:50.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:50.870 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:50.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:50.928 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:50.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:51.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:51.080 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:51.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:51.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:51.182 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:51.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:50 smithi028 ceph-mon[29442]: pgmap v118: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 930 B/s rd, 57 MiB/s wr, 909 op/s 2024-02-05T06:36:51.195 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:50 smithi028 ceph-mon[37984]: pgmap v118: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 930 B/s rd, 57 MiB/s wr, 909 op/s 2024-02-05T06:36:51.210 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:51.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:51.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:51.227 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 74 16 101914 101898 86.0246 59.6875 0.01312 0.0116197 2024-02-05T06:36:51.229 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:50 smithi167 ceph-mon[32774]: pgmap v118: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 930 B/s rd, 57 MiB/s wr, 909 op/s 2024-02-05T06:36:51.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:51.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:51.358 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:51.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:51.460 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:51.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:51.498 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:51.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:51.599 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:51.629 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:51.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:51.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:51.731 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:51.770 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:51.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:51.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:51.872 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:51.907 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:51.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:51.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:52.009 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:52.036 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:52.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:52.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:52.138 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:52.185 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:52.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:52.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:52.228 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 75 16 102964 102948 85.7526 65.625 0.0142252 0.0116565 2024-02-05T06:36:52.287 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:52.332 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:52.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:52.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:52.434 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:52.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:52.473 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:52.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:52.574 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:52.615 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:52.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:52.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:52.716 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:52.730 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:36:52 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:36:52.744 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:52.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:52.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:52.761 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:36:52 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:36:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:36:52.846 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:52.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:52.876 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:52.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:52.977 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:53.003 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:53.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:53.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:53.105 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:53.132 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:53.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:53.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:53.227 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 76 16 104051 104035 85.5182 67.9375 0.0198677 0.0116883 2024-02-05T06:36:53.234 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:53.262 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:53.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:53.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:53.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:53.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:53.406 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:53.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:53.507 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:53.534 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:53.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:53.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:53.636 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:53.683 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:53.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:53.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:53.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:53 smithi028 ceph-mon[37984]: pgmap v119: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.26k op/s 2024-02-05T06:36:53.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:53 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:36:53.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:53 smithi028 ceph-mon[29442]: pgmap v119: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.26k op/s 2024-02-05T06:36:53.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:53 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:36:53.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:53 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-a[29438]: 2024-02-05T06:36:53.392+0000 7fa733293700 -1 mon.a@0(leader).osd e117 definitely_dead 0 2024-02-05T06:36:53.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:53 smithi167 ceph-mon[32774]: pgmap v119: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.26k op/s 2024-02-05T06:36:53.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:53 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:36:53.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:53.837 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:53.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:53.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:53.939 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:53.965 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:53.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:53.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:54.067 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:54.105 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:54.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:54.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:54.207 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:54.227 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 77 16 105123 105107 85.2776 67 0.0079447 0.0117213 2024-02-05T06:36:54.250 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:54.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:54.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:54.292 INFO:teuthology.orchestra.run.smithi028.stderr:marked in osd.4. 2024-02-05T06:36:54.354 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:54.377 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:54.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:54.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:54.479 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:54.502 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:54.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:54.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:54.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:54 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1573005625' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-05T06:36:54.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:54 smithi028 ceph-mon[29442]: from='client.24811 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-05T06:36:54.509 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:54 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1573005625' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-05T06:36:54.509 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:54 smithi028 ceph-mon[37984]: from='client.24811 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-05T06:36:54.604 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:54.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:54.629 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:54.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:54.729 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:54.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:54 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1573005625' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-05T06:36:54.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:54 smithi167 ceph-mon[32774]: from='client.24811 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-02-05T06:36:54.754 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:54.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:54.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:54.856 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:54.879 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:54.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:54.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:54.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:55.011 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:55.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:55.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:55.113 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:55.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:55.137 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:55.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:55.227 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 78 16 105207 105191 84.252 5.25 0.00283747 0.0117245 2024-02-05T06:36:55.238 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:55.261 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:55.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:55.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:55.363 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:55.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:55.387 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:55.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:55.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:55 smithi028 ceph-mon[29442]: pgmap v120: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 52 MiB/s wr, 828 op/s 2024-02-05T06:36:55.440 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:55 smithi028 ceph-mon[29442]: from='client.24811 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-02-05T06:36:55.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:55 smithi028 ceph-mon[29442]: osdmap e118: 8 total, 8 up, 8 in 2024-02-05T06:36:55.441 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:55 smithi028 ceph-mon[29442]: osdmap e119: 8 total, 8 up, 8 in 2024-02-05T06:36:55.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:55 smithi028 ceph-mon[37984]: pgmap v120: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 52 MiB/s wr, 828 op/s 2024-02-05T06:36:55.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:55 smithi028 ceph-mon[37984]: from='client.24811 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-02-05T06:36:55.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:55 smithi028 ceph-mon[37984]: osdmap e118: 8 total, 8 up, 8 in 2024-02-05T06:36:55.441 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:55 smithi028 ceph-mon[37984]: osdmap e119: 8 total, 8 up, 8 in 2024-02-05T06:36:55.476 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:36:55.476 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:36:55.488 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:55.517 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:55.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:55.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:55.619 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:36:55.643 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:55.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:55.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:36:55.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:55 smithi167 ceph-mon[32774]: pgmap v120: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 52 MiB/s wr, 828 op/s 2024-02-05T06:36:55.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:55 smithi167 ceph-mon[32774]: from='client.24811 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-02-05T06:36:55.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:55 smithi167 ceph-mon[32774]: osdmap e118: 8 total, 8 up, 8 in 2024-02-05T06:36:55.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:55 smithi167 ceph-mon[32774]: osdmap e119: 8 total, 8 up, 8 in 2024-02-05T06:36:55.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:55.769 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:55.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:55.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:55.871 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:55.877 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-02-05T06:36:55.900 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:55.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:55.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:56.002 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:56.029 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:56.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:56.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:56.131 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:56.161 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:56.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:56.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:56.227 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 79 16 105384 105368 83.3259 11.0625 0.021395 0.0119959 2024-02-05T06:36:56.263 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:56.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:56.295 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:56.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:56.397 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:56.427 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:56.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:56.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:56.476 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:56 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:36:56.476 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:56 smithi028 ceph-mon[37984]: 7.9 starting backfill to osd.4 from (0'0,0'0] MAX to 117'6562 2024-02-05T06:36:56.477 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:56 smithi028 ceph-mon[37984]: osdmap e120: 8 total, 8 up, 8 in 2024-02-05T06:36:56.477 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:56 smithi028 ceph-mon[37984]: 7.1 starting backfill to osd.4 from (0'0,0'0] MAX to 117'6706 2024-02-05T06:36:56.477 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:56 smithi028 ceph-mon[37984]: 7.4 starting backfill to osd.4 from (0'0,0'0] MAX to 117'6648 2024-02-05T06:36:56.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:56 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:36:56.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:56 smithi028 ceph-mon[29442]: 7.9 starting backfill to osd.4 from (0'0,0'0] MAX to 117'6562 2024-02-05T06:36:56.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:56 smithi028 ceph-mon[29442]: osdmap e120: 8 total, 8 up, 8 in 2024-02-05T06:36:56.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:56 smithi028 ceph-mon[29442]: 7.1 starting backfill to osd.4 from (0'0,0'0] MAX to 117'6706 2024-02-05T06:36:56.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:56 smithi028 ceph-mon[29442]: 7.4 starting backfill to osd.4 from (0'0,0'0] MAX to 117'6648 2024-02-05T06:36:56.528 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:56.558 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:56.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:56.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:56.660 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:56.692 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:56.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:56.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:56.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:56 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:36:56.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:56 smithi167 ceph-mon[32774]: 7.9 starting backfill to osd.4 from (0'0,0'0] MAX to 117'6562 2024-02-05T06:36:56.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:56 smithi167 ceph-mon[32774]: osdmap e120: 8 total, 8 up, 8 in 2024-02-05T06:36:56.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:56 smithi167 ceph-mon[32774]: 7.1 starting backfill to osd.4 from (0'0,0'0] MAX to 117'6706 2024-02-05T06:36:56.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:56 smithi167 ceph-mon[32774]: 7.4 starting backfill to osd.4 from (0'0,0'0] MAX to 117'6648 2024-02-05T06:36:56.795 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:56.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:56.844 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:56.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:56.945 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:56.973 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:56.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:56.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:56.994 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:36:57.049 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:36:57.075 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:57.098 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:36:57.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:57.118 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:57.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:57.158 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:36:57.201 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:36:57.220 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:57.227 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-02-05T06:36:57.225717+0000 min lat: 0.0013072 max lat: 1.79789 avg lat: 0.0120484 2024-02-05T06:36:57.228 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T06:36:57.228 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 80 16 106255 106239 82.9646 54.4375 0.00673204 0.0120484 2024-02-05T06:36:57.241 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:36:57.248 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:57.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:57.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:57.298 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:36:57.344 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:36:57.349 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:36:57.349 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:36:57.349 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:36:57.349 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:36:57.349 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:36:57.349 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:36:57.349 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:36:57.349 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:36:57.351 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:57.379 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:57.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:57.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:57.480 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:57.531 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:57.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:57.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:57.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:57 smithi028 ceph-mon[29442]: 7.8 starting backfill to osd.4 from (0'0,0'0] MAX to 117'6655 2024-02-05T06:36:57.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:57 smithi028 ceph-mon[29442]: 7.3 starting backfill to osd.4 from (0'0,0'0] MAX to 117'6624 2024-02-05T06:36:57.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:57 smithi028 ceph-mon[29442]: pgmap v124: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 114 B/s, 4 objects/s recovering 2024-02-05T06:36:57.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:57 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1297050529' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:36:57.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:57 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1382581202' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:36:57.612 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:57 smithi028 ceph-mon[37984]: 7.8 starting backfill to osd.4 from (0'0,0'0] MAX to 117'6655 2024-02-05T06:36:57.612 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:57 smithi028 ceph-mon[37984]: 7.3 starting backfill to osd.4 from (0'0,0'0] MAX to 117'6624 2024-02-05T06:36:57.612 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:57 smithi028 ceph-mon[37984]: pgmap v124: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 114 B/s, 4 objects/s recovering 2024-02-05T06:36:57.612 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:57 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1297050529' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:36:57.612 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:57 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1382581202' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:36:57.634 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:57.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:57.665 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:57.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:57.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:57 smithi167 ceph-mon[32774]: 7.8 starting backfill to osd.4 from (0'0,0'0] MAX to 117'6655 2024-02-05T06:36:57.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:57 smithi167 ceph-mon[32774]: 7.3 starting backfill to osd.4 from (0'0,0'0] MAX to 117'6624 2024-02-05T06:36:57.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:57 smithi167 ceph-mon[32774]: pgmap v124: 148 pgs: 148 active+clean; 6.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 114 B/s, 4 objects/s recovering 2024-02-05T06:36:57.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:57 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1297050529' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:36:57.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:57 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1382581202' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:36:57.766 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:57.796 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:57.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:57.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:57.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:57.953 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:57.954 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-02-05T06:36:57.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:57.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:58.058 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:58.087 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:58.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:58.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:58.189 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:36:58.228 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 81 16 107169 107153 82.6452 57.125 0.0168913 0.0120938 2024-02-05T06:36:58.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:58.231 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:58.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:36:58.332 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:58.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:58.364 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:58.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:58.465 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:58.504 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:58.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:58.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:58.606 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:58.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:58.639 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:58.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:58.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:58 smithi028 ceph-mon[29442]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:36:58.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:58 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1382581202' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:36:58.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:58 smithi028 ceph-mon[29442]: osdmap e121: 8 total, 8 up, 8 in 2024-02-05T06:36:58.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:58 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:58.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:58 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:58.645 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:58 smithi028 ceph-mon[37984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:36:58.645 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:58 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1382581202' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:36:58.645 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:58 smithi028 ceph-mon[37984]: osdmap e121: 8 total, 8 up, 8 in 2024-02-05T06:36:58.645 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:58 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:58.645 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:58 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:58.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:58 smithi167 ceph-mon[32774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:36:58.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:58 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1382581202' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:36:58.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:58 smithi167 ceph-mon[32774]: osdmap e121: 8 total, 8 up, 8 in 2024-02-05T06:36:58.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:58 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:58.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:58 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:36:58.740 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:58.794 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:58.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:58.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:58.896 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:58.944 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:58.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:58.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:59.043 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-05T06:36:59.045 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:36:59.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:59.135 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:59.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:36:59.229 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 82 16 108031 108015 82.2934 53.875 0.0287645 0.0121461 2024-02-05T06:36:59.236 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:36:59.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:59.294 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:36:59.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:36:59.395 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:59.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:59.426 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:59.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:36:59.527 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:36:59.556 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:59.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:59.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:36:59.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:59 smithi028 ceph-mon[29442]: pgmap v126: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 871 op/s; 33164/316767 objects misplaced (10.470%); 1.7 MiB/s, 40 objects/s recovering 2024-02-05T06:36:59.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:36:59 smithi028 ceph-mon[29442]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:36:59.571 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:59 smithi028 ceph-mon[37984]: pgmap v126: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 871 op/s; 33164/316767 objects misplaced (10.470%); 1.7 MiB/s, 40 objects/s recovering 2024-02-05T06:36:59.571 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:36:59 smithi028 ceph-mon[37984]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:36:59.658 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:36:59.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:59.689 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:59.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:36:59.689 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:36:59.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:59 smithi167 ceph-mon[32774]: pgmap v126: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 871 op/s; 33164/316767 objects misplaced (10.470%); 1.7 MiB/s, 40 objects/s recovering 2024-02-05T06:36:59.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:36:59 smithi167 ceph-mon[32774]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:36:59.790 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:36:59.822 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:59.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:59.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:36:59.924 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:36:59.955 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:36:59.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:36:59.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:00.056 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:00.103 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:00.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:00.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:00.205 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:00.229 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 83 16 108924 108908 81.9744 55.8125 0.0192949 0.0121939 2024-02-05T06:37:00.236 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:00.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:00.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:00.338 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:00.378 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:00.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:00.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:00.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:00 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:37:00.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:00 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:37:00.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:00 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:37:00.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:00 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:37:00.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:00 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:37:00.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:00 smithi028 ceph-mon[37984]: osdmap e122: 8 total, 8 up, 8 in 2024-02-05T06:37:00.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:00 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:37:00.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:00 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:37:00.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:00 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:37:00.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:00 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:37:00.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:00 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:37:00.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:00 smithi028 ceph-mon[29442]: osdmap e122: 8 total, 8 up, 8 in 2024-02-05T06:37:00.480 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:00.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:00.510 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:00.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:00.611 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:00.641 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:00.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:00.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:00.701 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:37:00.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:00 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:37:00.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:00 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:37:00.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:00 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:37:00.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:00 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:37:00.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:00 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:37:00.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:00 smithi167 ceph-mon[32774]: osdmap e122: 8 total, 8 up, 8 in 2024-02-05T06:37:00.742 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:00.771 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:00.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:00.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:00.873 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:00.878 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T06:37:00.878 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T06:37:00.878 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-02-05T06:37:00.879 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T06:37:00.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:00.948 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:00.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:01.049 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:01.097 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:01.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:01.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:01.199 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:01.222 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:01.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:01.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:01.231 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 84 15 109770 109755 81.6288 52.9375 0.048921 0.0122452 2024-02-05T06:37:01.325 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:01.366 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:01.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:01.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:01.468 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:01.497 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:01.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:01.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:01.548 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:01 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:37:01.548 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:01 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:37:01.548 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:01 smithi028 ceph-mon[37984]: pgmap v128: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 972 B/s rd, 41 MiB/s wr, 661 op/s; 33164/316767 objects misplaced (10.470%); 1.3 MiB/s, 30 objects/s recovering 2024-02-05T06:37:01.548 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:01 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:37:01.548 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:01 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:37:01.548 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:01 smithi028 ceph-mon[29442]: pgmap v128: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 972 B/s rd, 41 MiB/s wr, 661 op/s; 33164/316767 objects misplaced (10.470%); 1.3 MiB/s, 30 objects/s recovering 2024-02-05T06:37:01.572 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T06:37:01.599 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:01.628 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:01.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:01.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:01.729 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:01.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:01 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:37:01.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:01 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:37:01.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:01 smithi167 ceph-mon[32774]: pgmap v128: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 972 B/s rd, 41 MiB/s wr, 661 op/s; 33164/316767 objects misplaced (10.470%); 1.3 MiB/s, 30 objects/s recovering 2024-02-05T06:37:01.784 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:01.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:01.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:01.886 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:01.914 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:01.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:01.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:01.981 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:37:02.016 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:02.092 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:37:02.092 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-02-05T06:37:02.092 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-02-05T06:37:02.092 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-02-05T06:37:02.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.092 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:37954 7.8 7:1eea8f7d:::benchmark_data_smithi028_77509_object37953:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T06:37:02.092 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:35:55.482647+0000", 2024-02-05T06:37:02.093 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 66.593570665000001, 2024-02-05T06:37:02.093 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.052155044999999997, 2024-02-05T06:37:02.093 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:37:02.093 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:37:02.093 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:37:02.093 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:37:02.093 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:37:02.093 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 37954 2024-02-05T06:37:02.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.093 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:37:02.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.093 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:37:02.093 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.482647+0000", 2024-02-05T06:37:02.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.094 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:37:02.094 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.482647+0000", 2024-02-05T06:37:02.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.094 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:37:02.094 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.482649+0000", 2024-02-05T06:37:02.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2979999999999999e-06 2024-02-05T06:37:02.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.095 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:37:02.095 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.482672+0000", 2024-02-05T06:37:02.095 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3031999999999999e-05 2024-02-05T06:37:02.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.095 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:37:02.095 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.482684+0000", 2024-02-05T06:37:02.095 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1436e-05 2024-02-05T06:37:02.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.095 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:37:02.095 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.482692+0000", 2024-02-05T06:37:02.095 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9999999999999996e-06 2024-02-05T06:37:02.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.096 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:37:02.096 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.482751+0000", 2024-02-05T06:37:02.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8996000000000002e-05 2024-02-05T06:37:02.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.096 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:37:02.096 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.482862+0000", 2024-02-05T06:37:02.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111369 2024-02-05T06:37:02.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.096 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-02-05T06:37:02.096 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.482988+0000", 2024-02-05T06:37:02.097 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000126038 2024-02-05T06:37:02.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.097 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.097 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.484633+0000", 2024-02-05T06:37:02.097 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001644427 2024-02-05T06:37:02.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.097 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:37:02.097 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.484726+0000", 2024-02-05T06:37:02.097 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3177999999999995e-05 2024-02-05T06:37:02.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.097 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.098 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.534583+0000", 2024-02-05T06:37:02.098 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.049856969000000001 2024-02-05T06:37:02.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.098 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.098 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:37:02.098 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.534757+0000", 2024-02-05T06:37:02.098 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017454600000000001 2024-02-05T06:37:02.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.098 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.098 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:37:02.098 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.534802+0000", 2024-02-05T06:37:02.098 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4756000000000003e-05 2024-02-05T06:37:02.098 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.098 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:37:02.099 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.099 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:37960 7.8 7:1d33512d:::benchmark_data_smithi028_77509_object37959:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T06:37:02.099 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:35:55.485328+0000", 2024-02-05T06:37:02.099 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 66.590889935999996, 2024-02-05T06:37:02.099 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.057335959999999998, 2024-02-05T06:37:02.099 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:37:02.099 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:37:02.099 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:37:02.099 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:37:02.099 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:37:02.099 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 37960 2024-02-05T06:37:02.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.099 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:37:02.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.100 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:37:02.100 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.485328+0000", 2024-02-05T06:37:02.100 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.100 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:37:02.100 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.485328+0000", 2024-02-05T06:37:02.100 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.100 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:37:02.100 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.485330+0000", 2024-02-05T06:37:02.101 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2220000000000001e-06 2024-02-05T06:37:02.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.101 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:37:02.101 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.485353+0000", 2024-02-05T06:37:02.101 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2908000000000002e-05 2024-02-05T06:37:02.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.101 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:37:02.101 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.485364+0000", 2024-02-05T06:37:02.101 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1486e-05 2024-02-05T06:37:02.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.101 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:37:02.101 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.485372+0000", 2024-02-05T06:37:02.102 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4699999999999996e-06 2024-02-05T06:37:02.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.102 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:37:02.102 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.485423+0000", 2024-02-05T06:37:02.102 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1439e-05 2024-02-05T06:37:02.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.102 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:37:02.102 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.485546+0000", 2024-02-05T06:37:02.102 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000122736 2024-02-05T06:37:02.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.102 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-02-05T06:37:02.102 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.485683+0000", 2024-02-05T06:37:02.102 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000136878 2024-02-05T06:37:02.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.103 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:37:02.103 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.488946+0000", 2024-02-05T06:37:02.103 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032630379999999998 2024-02-05T06:37:02.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.103 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.103 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.489751+0000", 2024-02-05T06:37:02.103 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00080498400000000002 2024-02-05T06:37:02.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.103 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.104 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.542424+0000", 2024-02-05T06:37:02.104 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.052673273999999999 2024-02-05T06:37:02.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.104 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:37:02.104 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.542621+0000", 2024-02-05T06:37:02.104 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019644899999999999 2024-02-05T06:37:02.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.104 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:37:02.104 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.542664+0000", 2024-02-05T06:37:02.104 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3075999999999999e-05 2024-02-05T06:37:02.104 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.104 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:37:02.105 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.105 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:37968 7.8 7:17585fce:::benchmark_data_smithi028_77509_object37967:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T06:37:02.105 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:35:55.489439+0000", 2024-02-05T06:37:02.105 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 66.586778296999995, 2024-02-05T06:37:02.105 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.057672943999999997, 2024-02-05T06:37:02.105 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:37:02.105 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:37:02.105 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:37:02.105 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:37:02.105 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:37:02.105 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 37968 2024-02-05T06:37:02.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.106 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:37:02.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.106 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:37:02.106 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.489439+0000", 2024-02-05T06:37:02.106 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.106 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:37:02.106 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.489439+0000", 2024-02-05T06:37:02.106 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.106 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:37:02.107 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.489442+0000", 2024-02-05T06:37:02.107 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2989999999999999e-06 2024-02-05T06:37:02.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.107 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:37:02.107 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.489464+0000", 2024-02-05T06:37:02.107 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2118999999999999e-05 2024-02-05T06:37:02.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.107 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:37:02.107 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.489475+0000", 2024-02-05T06:37:02.107 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.153e-05 2024-02-05T06:37:02.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.108 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:37:02.108 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.489483+0000", 2024-02-05T06:37:02.108 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9130000000000003e-06 2024-02-05T06:37:02.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.108 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:37:02.108 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.489787+0000", 2024-02-05T06:37:02.108 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00030324399999999999 2024-02-05T06:37:02.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.108 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:37:02.108 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.489979+0000", 2024-02-05T06:37:02.108 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019279400000000001 2024-02-05T06:37:02.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.109 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-02-05T06:37:02.109 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.490120+0000", 2024-02-05T06:37:02.109 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000140333 2024-02-05T06:37:02.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.109 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:37:02.109 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.491699+0000", 2024-02-05T06:37:02.109 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0015797540000000001 2024-02-05T06:37:02.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.109 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.109 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.492007+0000", 2024-02-05T06:37:02.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00030748800000000002 2024-02-05T06:37:02.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.110 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.547014+0000", 2024-02-05T06:37:02.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.055006791999999999 2024-02-05T06:37:02.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:37:02.110 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.547073+0000", 2024-02-05T06:37:02.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9243000000000002e-05 2024-02-05T06:37:02.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:37:02.110 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.547112+0000", 2024-02-05T06:37:02.111 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9434999999999998e-05 2024-02-05T06:37:02.111 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.111 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:37:02.111 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.111 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:38370 7.8 7:18eb092b:::benchmark_data_smithi028_77509_object38369:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T06:37:02.111 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:35:55.750845+0000", 2024-02-05T06:37:02.111 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 66.325372449, 2024-02-05T06:37:02.111 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.048862445999999997, 2024-02-05T06:37:02.111 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:37:02.111 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:37:02.111 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:37:02.112 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:37:02.112 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:37:02.112 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 38370 2024-02-05T06:37:02.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.112 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:37:02.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.112 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:37:02.112 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.750845+0000", 2024-02-05T06:37:02.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.112 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:37:02.112 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.750845+0000", 2024-02-05T06:37:02.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.113 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:37:02.113 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.750848+0000", 2024-02-05T06:37:02.113 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5749999999999999e-06 2024-02-05T06:37:02.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.113 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:37:02.113 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.750872+0000", 2024-02-05T06:37:02.113 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.427e-05 2024-02-05T06:37:02.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.114 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:37:02.114 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.750895+0000", 2024-02-05T06:37:02.114 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3116e-05 2024-02-05T06:37:02.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.114 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:37:02.114 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.750906+0000", 2024-02-05T06:37:02.114 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0969e-05 2024-02-05T06:37:02.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.114 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:37:02.114 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.751384+0000", 2024-02-05T06:37:02.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00047777600000000001 2024-02-05T06:37:02.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:37:02.115 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.751503+0000", 2024-02-05T06:37:02.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011873 2024-02-05T06:37:02.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-02-05T06:37:02.115 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.751638+0000", 2024-02-05T06:37:02.115 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013539800000000001 2024-02-05T06:37:02.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.115 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.115 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.756284+0000", 2024-02-05T06:37:02.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0046457800000000004 2024-02-05T06:37:02.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.116 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:37:02.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.756295+0000", 2024-02-05T06:37:02.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1094000000000001e-05 2024-02-05T06:37:02.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.116 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.116 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.799585+0000", 2024-02-05T06:37:02.116 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.043290197000000002 2024-02-05T06:37:02.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.116 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:37:02.117 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.799653+0000", 2024-02-05T06:37:02.117 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7444e-05 2024-02-05T06:37:02.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.117 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:37:02.117 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.799708+0000", 2024-02-05T06:37:02.117 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5096999999999998e-05 2024-02-05T06:37:02.117 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.117 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:37:02.117 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.117 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:38371 7.8 7:17b402a9:::benchmark_data_smithi028_77509_object38370:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T06:37:02.117 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:35:55.750951+0000", 2024-02-05T06:37:02.117 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 66.325266341000003, 2024-02-05T06:37:02.118 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.048935029999999997, 2024-02-05T06:37:02.118 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:37:02.118 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:37:02.118 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:37:02.118 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:37:02.119 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:37:02.119 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 38371 2024-02-05T06:37:02.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.119 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:37:02.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:37:02.119 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.750951+0000", 2024-02-05T06:37:02.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.119 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:37:02.119 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.750951+0000", 2024-02-05T06:37:02.119 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.120 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:37:02.120 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.750953+0000", 2024-02-05T06:37:02.120 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:37:02.120 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1790000000000001e-06 2024-02-05T06:37:02.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:37:02.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.750979+0000", 2024-02-05T06:37:02.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6332000000000001e-05 2024-02-05T06:37:02.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:37:02.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.750988+0000", 2024-02-05T06:37:02.121 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9979999999999992e-06 2024-02-05T06:37:02.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.121 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:37:02.121 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.750995+0000", 2024-02-05T06:37:02.122 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8360000000000003e-06 2024-02-05T06:37:02.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.122 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:37:02.122 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.753154+0000", 2024-02-05T06:37:02.122 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021593599999999999 2024-02-05T06:37:02.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.122 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:37:02.122 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.753276+0000", 2024-02-05T06:37:02.122 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121958 2024-02-05T06:37:02.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-02-05T06:37:02.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.753404+0000", 2024-02-05T06:37:02.123 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000127763 2024-02-05T06:37:02.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:37:02.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.756300+0000", 2024-02-05T06:37:02.123 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028956799999999999 2024-02-05T06:37:02.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.123 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.123 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.756329+0000", 2024-02-05T06:37:02.123 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9598e-05 2024-02-05T06:37:02.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.124 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.124 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.799784+0000", 2024-02-05T06:37:02.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.043455186999999999 2024-02-05T06:37:02.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.124 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:37:02.124 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.799850+0000", 2024-02-05T06:37:02.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5387000000000003e-05 2024-02-05T06:37:02.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.124 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:37:02.124 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.799886+0000", 2024-02-05T06:37:02.124 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6752000000000003e-05 2024-02-05T06:37:02.124 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.125 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:37:02.125 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.125 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.125 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:38376 7.8 7:18326902:::benchmark_data_smithi028_77509_object38375:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T06:37:02.125 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:35:55.752062+0000", 2024-02-05T06:37:02.125 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 66.324155787999999, 2024-02-05T06:37:02.125 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.048147017, 2024-02-05T06:37:02.125 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:37:02.125 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:37:02.125 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:37:02.125 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:37:02.125 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:37:02.125 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 38376 2024-02-05T06:37:02.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.126 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:37:02.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.126 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:37:02.126 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.752062+0000", 2024-02-05T06:37:02.126 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.126 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:37:02.126 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.752062+0000", 2024-02-05T06:37:02.126 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.126 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.126 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:37:02.126 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.752063+0000", 2024-02-05T06:37:02.127 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4640000000000001e-06 2024-02-05T06:37:02.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.127 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.127 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:37:02.127 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.752086+0000", 2024-02-05T06:37:02.127 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2623e-05 2024-02-05T06:37:02.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.127 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.127 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:37:02.127 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.752095+0000", 2024-02-05T06:37:02.127 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1169999999999993e-06 2024-02-05T06:37:02.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.127 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.127 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:37:02.128 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.752102+0000", 2024-02-05T06:37:02.128 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3289999999999996e-06 2024-02-05T06:37:02.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.128 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:37:02.128 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.754801+0000", 2024-02-05T06:37:02.128 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026986060000000001 2024-02-05T06:37:02.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.128 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:37:02.128 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.754921+0000", 2024-02-05T06:37:02.128 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011971700000000001 2024-02-05T06:37:02.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.129 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-02-05T06:37:02.129 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.755052+0000", 2024-02-05T06:37:02.129 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000130937 2024-02-05T06:37:02.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.129 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.129 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.756417+0000", 2024-02-05T06:37:02.129 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001365687 2024-02-05T06:37:02.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.129 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:37:02.129 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.756432+0000", 2024-02-05T06:37:02.129 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4537e-05 2024-02-05T06:37:02.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.130 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.130 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.800036+0000", 2024-02-05T06:37:02.130 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.043604395999999997 2024-02-05T06:37:02.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.130 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:37:02.130 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.800138+0000", 2024-02-05T06:37:02.130 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101933 2024-02-05T06:37:02.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.130 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:37:02.130 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:55.800209+0000", 2024-02-05T06:37:02.130 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0671000000000002e-05 2024-02-05T06:37:02.130 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.130 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:37:02.131 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.131 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:42999 7.8 7:17489bc4:::benchmark_data_smithi028_77509_object42998:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T06:37:02.131 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:35:58.574506+0000", 2024-02-05T06:37:02.131 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 63.501711716000003, 2024-02-05T06:37:02.131 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.056629183, 2024-02-05T06:37:02.131 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:37:02.131 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:37:02.131 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:37:02.131 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:37:02.131 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:37:02.131 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 42999 2024-02-05T06:37:02.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.132 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:37:02.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.132 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:37:02.132 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.574506+0000", 2024-02-05T06:37:02.132 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.132 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:37:02.132 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.574506+0000", 2024-02-05T06:37:02.132 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.132 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:37:02.132 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.574508+0000", 2024-02-05T06:37:02.133 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9259999999999999e-06 2024-02-05T06:37:02.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.133 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:37:02.133 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.574531+0000", 2024-02-05T06:37:02.133 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2623e-05 2024-02-05T06:37:02.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.133 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:37:02.133 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.574542+0000", 2024-02-05T06:37:02.133 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.112e-05 2024-02-05T06:37:02.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.133 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:37:02.133 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.574549+0000", 2024-02-05T06:37:02.134 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4070000000000004e-06 2024-02-05T06:37:02.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:02.135 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:02.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:02.136 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:37:02.136 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.575019+0000", 2024-02-05T06:37:02.136 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00046950200000000001 2024-02-05T06:37:02.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.136 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:37:02.136 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.575094+0000", 2024-02-05T06:37:02.136 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5428999999999999e-05 2024-02-05T06:37:02.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.136 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-02-05T06:37:02.136 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.575189+0000", 2024-02-05T06:37:02.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4963000000000005e-05 2024-02-05T06:37:02.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.137 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.137 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.580373+0000", 2024-02-05T06:37:02.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051842839999999999 2024-02-05T06:37:02.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.137 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:37:02.137 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.580496+0000", 2024-02-05T06:37:02.137 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012258200000000001 2024-02-05T06:37:02.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.137 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.137 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.630334+0000", 2024-02-05T06:37:02.138 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.049837688999999998 2024-02-05T06:37:02.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:37:02.138 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.631093+0000", 2024-02-05T06:37:02.138 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00075959400000000005 2024-02-05T06:37:02.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.138 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:37:02.138 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.631135+0000", 2024-02-05T06:37:02.138 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2064000000000002e-05 2024-02-05T06:37:02.138 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.138 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:37:02.138 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.139 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:43001 7.8 7:13248b09:::benchmark_data_smithi028_77509_object43000:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T06:37:02.139 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:35:58.574562+0000", 2024-02-05T06:37:02.139 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 63.501655818000003, 2024-02-05T06:37:02.139 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.056788390000000001, 2024-02-05T06:37:02.139 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:37:02.139 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:37:02.139 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:37:02.139 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:37:02.139 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:37:02.139 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 43001 2024-02-05T06:37:02.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.139 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:37:02.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.140 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:37:02.140 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.574562+0000", 2024-02-05T06:37:02.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.140 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:37:02.140 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.574562+0000", 2024-02-05T06:37:02.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.140 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:37:02.140 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.574563+0000", 2024-02-05T06:37:02.140 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3e-07 2024-02-05T06:37:02.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:37:02.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.574582+0000", 2024-02-05T06:37:02.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9689000000000001e-05 2024-02-05T06:37:02.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:37:02.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.574589+0000", 2024-02-05T06:37:02.141 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4869999999999999e-06 2024-02-05T06:37:02.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.141 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:37:02.141 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.574594+0000", 2024-02-05T06:37:02.142 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5430000000000001e-06 2024-02-05T06:37:02.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:37:02.142 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.577183+0000", 2024-02-05T06:37:02.142 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025889279999999999 2024-02-05T06:37:02.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:37:02.142 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.577316+0000", 2024-02-05T06:37:02.142 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000132757 2024-02-05T06:37:02.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.142 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-02-05T06:37:02.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.577448+0000", 2024-02-05T06:37:02.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013230199999999999 2024-02-05T06:37:02.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:37:02.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.580501+0000", 2024-02-05T06:37:02.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003053142 2024-02-05T06:37:02.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.143 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.627417+0000", 2024-02-05T06:37:02.143 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.046915912999999997 2024-02-05T06:37:02.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.143 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.144 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.631180+0000", 2024-02-05T06:37:02.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0037629920000000002 2024-02-05T06:37:02.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.144 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:37:02.144 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.631318+0000", 2024-02-05T06:37:02.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013810299999999999 2024-02-05T06:37:02.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.144 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:37:02.144 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.631350+0000", 2024-02-05T06:37:02.144 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1903999999999998e-05 2024-02-05T06:37:02.144 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.144 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:37:02.145 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.145 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:43007 7.8 7:1cb9f07b:::benchmark_data_smithi028_77509_object43006:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T06:37:02.145 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:35:58.576245+0000", 2024-02-05T06:37:02.145 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 63.499972999999997, 2024-02-05T06:37:02.145 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.055242483000000002, 2024-02-05T06:37:02.145 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:37:02.145 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:37:02.145 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:37:02.145 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:37:02.145 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:37:02.145 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 43007 2024-02-05T06:37:02.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.145 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:37:02.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.146 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:37:02.146 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.576245+0000", 2024-02-05T06:37:02.146 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.146 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:37:02.146 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.576245+0000", 2024-02-05T06:37:02.146 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.146 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.146 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:37:02.146 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.576247+0000", 2024-02-05T06:37:02.146 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7600000000000001e-06 2024-02-05T06:37:02.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.147 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:37:02.147 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.576282+0000", 2024-02-05T06:37:02.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5830000000000001e-05 2024-02-05T06:37:02.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.147 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:37:02.147 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.576293+0000", 2024-02-05T06:37:02.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1164e-05 2024-02-05T06:37:02.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.147 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:37:02.147 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.576302+0000", 2024-02-05T06:37:02.147 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3089999999999996e-06 2024-02-05T06:37:02.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.148 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:37:02.148 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.578879+0000", 2024-02-05T06:37:02.148 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025769629999999998 2024-02-05T06:37:02.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.148 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:37:02.148 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.578983+0000", 2024-02-05T06:37:02.148 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000104312 2024-02-05T06:37:02.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.148 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-02-05T06:37:02.148 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.579105+0000", 2024-02-05T06:37:02.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000122266 2024-02-05T06:37:02.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.627527+0000", 2024-02-05T06:37:02.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.048422117000000001 2024-02-05T06:37:02.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:37:02.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.629204+0000", 2024-02-05T06:37:02.149 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0016767419999999999 2024-02-05T06:37:02.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.149 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.149 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.631405+0000", 2024-02-05T06:37:02.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022012669999999998 2024-02-05T06:37:02.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.150 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:37:02.150 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.631458+0000", 2024-02-05T06:37:02.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2027999999999997e-05 2024-02-05T06:37:02.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.150 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:37:02.150 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.631487+0000", 2024-02-05T06:37:02.150 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9725000000000001e-05 2024-02-05T06:37:02.150 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.150 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:37:02.150 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.151 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:43013 7.8 7:16097e05:::benchmark_data_smithi028_77509_object43012:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T06:37:02.151 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:35:58.580100+0000", 2024-02-05T06:37:02.151 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 63.496117663, 2024-02-05T06:37:02.151 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.052414623, 2024-02-05T06:37:02.151 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:37:02.151 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:37:02.151 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:37:02.151 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:37:02.151 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:37:02.151 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 43013 2024-02-05T06:37:02.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.151 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:37:02.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.152 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:37:02.152 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.580100+0000", 2024-02-05T06:37:02.152 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.152 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:37:02.152 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.580100+0000", 2024-02-05T06:37:02.152 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.152 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:37:02.152 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.580102+0000", 2024-02-05T06:37:02.152 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.84e-06 2024-02-05T06:37:02.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.153 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:37:02.153 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.580132+0000", 2024-02-05T06:37:02.153 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0491999999999999e-05 2024-02-05T06:37:02.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.153 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:37:02.153 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.580144+0000", 2024-02-05T06:37:02.153 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1899000000000001e-05 2024-02-05T06:37:02.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.153 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:37:02.153 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.580153+0000", 2024-02-05T06:37:02.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3860000000000007e-06 2024-02-05T06:37:02.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.154 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:37:02.154 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.625039+0000", 2024-02-05T06:37:02.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.044886083 2024-02-05T06:37:02.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.154 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:37:02.154 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.625184+0000", 2024-02-05T06:37:02.154 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014483599999999999 2024-02-05T06:37:02.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.154 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-02-05T06:37:02.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.625352+0000", 2024-02-05T06:37:02.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016864899999999999 2024-02-05T06:37:02.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.629191+0000", 2024-02-05T06:37:02.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003838546 2024-02-05T06:37:02.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:37:02.155 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.629212+0000", 2024-02-05T06:37:02.155 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1287000000000001e-05 2024-02-05T06:37:02.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.155 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.156 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.632373+0000", 2024-02-05T06:37:02.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031608869999999998 2024-02-05T06:37:02.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.156 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:37:02.156 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.632475+0000", 2024-02-05T06:37:02.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101727 2024-02-05T06:37:02.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.156 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:37:02.156 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.632515+0000", 2024-02-05T06:37:02.156 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9990999999999997e-05 2024-02-05T06:37:02.156 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.156 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:37:02.156 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.157 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:43262 7.e 7:716129e8:::benchmark_data_smithi028_77509_object43261:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T06:37:02.157 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:35:58.773977+0000", 2024-02-05T06:37:02.157 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 63.302240873000002, 2024-02-05T06:37:02.157 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.141569418, 2024-02-05T06:37:02.157 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:37:02.157 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:37:02.157 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:37:02.157 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:37:02.157 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:37:02.157 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 43262 2024-02-05T06:37:02.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.157 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:37:02.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:37:02.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.773977+0000", 2024-02-05T06:37:02.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:37:02.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.773977+0000", 2024-02-05T06:37:02.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.158 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.158 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.158 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:37:02.158 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.773979+0000", 2024-02-05T06:37:02.158 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0679999999999999e-06 2024-02-05T06:37:02.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.159 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:37:02.159 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.774005+0000", 2024-02-05T06:37:02.159 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6536000000000001e-05 2024-02-05T06:37:02.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.159 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:37:02.159 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.774018+0000", 2024-02-05T06:37:02.159 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2874000000000001e-05 2024-02-05T06:37:02.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.159 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:37:02.159 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.774028+0000", 2024-02-05T06:37:02.160 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.183e-06 2024-02-05T06:37:02.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.160 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:37:02.160 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.774141+0000", 2024-02-05T06:37:02.160 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000113007 2024-02-05T06:37:02.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.160 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:37:02.160 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.774272+0000", 2024-02-05T06:37:02.160 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013184700000000001 2024-02-05T06:37:02.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.160 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:37:02.161 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.774410+0000", 2024-02-05T06:37:02.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000137727 2024-02-05T06:37:02.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.161 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:37:02.161 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.776287+0000", 2024-02-05T06:37:02.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001876768 2024-02-05T06:37:02.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.161 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.161 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.778309+0000", 2024-02-05T06:37:02.161 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002022112 2024-02-05T06:37:02.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.162 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.162 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.915347+0000", 2024-02-05T06:37:02.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.137038099 2024-02-05T06:37:02.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.162 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:37:02.162 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.915502+0000", 2024-02-05T06:37:02.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000154456 2024-02-05T06:37:02.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.162 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.162 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:37:02.162 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.915546+0000", 2024-02-05T06:37:02.162 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4740999999999997e-05 2024-02-05T06:37:02.162 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.162 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:37:02.163 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.163 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.163 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:43272 7.e 7:71beb652:::benchmark_data_smithi028_77509_object43271:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T06:37:02.163 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:35:58.778181+0000", 2024-02-05T06:37:02.163 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 63.298036492000001, 2024-02-05T06:37:02.163 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14068628899999999, 2024-02-05T06:37:02.163 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:37:02.163 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:37:02.163 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:37:02.163 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:37:02.163 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:37:02.163 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 43272 2024-02-05T06:37:02.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.164 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:37:02.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.164 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:37:02.164 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.778181+0000", 2024-02-05T06:37:02.164 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.164 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:37:02.164 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.778181+0000", 2024-02-05T06:37:02.164 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.164 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.164 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:37:02.165 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.778183+0000", 2024-02-05T06:37:02.165 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8670000000000001e-06 2024-02-05T06:37:02.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.165 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:37:02.165 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.778207+0000", 2024-02-05T06:37:02.165 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3465999999999999e-05 2024-02-05T06:37:02.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.165 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:37:02.165 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.778219+0000", 2024-02-05T06:37:02.165 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2208000000000001e-05 2024-02-05T06:37:02.165 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:37:02.166 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.778227+0000", 2024-02-05T06:37:02.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2649999999999996e-06 2024-02-05T06:37:02.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:37:02.166 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.778339+0000", 2024-02-05T06:37:02.166 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011191 2024-02-05T06:37:02.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.166 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.166 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:37:02.166 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.778475+0000", 2024-02-05T06:37:02.167 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013620799999999999 2024-02-05T06:37:02.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.167 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:37:02.167 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.778619+0000", 2024-02-05T06:37:02.167 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014408199999999999 2024-02-05T06:37:02.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.167 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.167 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.782013+0000", 2024-02-05T06:37:02.167 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033937440000000002 2024-02-05T06:37:02.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.167 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:37:02.167 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.782033+0000", 2024-02-05T06:37:02.168 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0086999999999999e-05 2024-02-05T06:37:02.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.168 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.168 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.168 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.918777+0000", 2024-02-05T06:37:02.168 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.136743847 2024-02-05T06:37:02.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.168 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.168 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:37:02.168 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.918830+0000", 2024-02-05T06:37:02.168 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2927000000000003e-05 2024-02-05T06:37:02.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.168 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.169 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:37:02.169 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:58.918868+0000", 2024-02-05T06:37:02.169 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7678000000000001e-05 2024-02-05T06:37:02.169 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.169 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:37:02.169 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.169 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:44275 7.8 7:1568b933:::benchmark_data_smithi028_77509_object44274:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T06:37:02.169 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:35:59.598026+0000", 2024-02-05T06:37:02.169 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 62.478192004999997, 2024-02-05T06:37:02.169 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.063322819000000002, 2024-02-05T06:37:02.169 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:37:02.169 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:37:02.170 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:37:02.170 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:37:02.170 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:37:02.170 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 44275 2024-02-05T06:37:02.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.170 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:37:02.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.170 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:37:02.170 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:59.598026+0000", 2024-02-05T06:37:02.170 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.170 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:37:02.170 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:59.598026+0000", 2024-02-05T06:37:02.171 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.171 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:37:02.171 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:59.598027+0000", 2024-02-05T06:37:02.171 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.92e-07 2024-02-05T06:37:02.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.171 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:37:02.171 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:59.598048+0000", 2024-02-05T06:37:02.171 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1339e-05 2024-02-05T06:37:02.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.171 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:37:02.171 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:59.598055+0000", 2024-02-05T06:37:02.172 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5950000000000001e-06 2024-02-05T06:37:02.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.172 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:37:02.172 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:59.598061+0000", 2024-02-05T06:37:02.172 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8449999999999999e-06 2024-02-05T06:37:02.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.172 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:37:02.172 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:59.598108+0000", 2024-02-05T06:37:02.172 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6851000000000003e-05 2024-02-05T06:37:02.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.172 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:37:02.172 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:59.598221+0000", 2024-02-05T06:37:02.173 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000113278 2024-02-05T06:37:02.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.173 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-02-05T06:37:02.173 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:59.598356+0000", 2024-02-05T06:37:02.173 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000134905 2024-02-05T06:37:02.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.173 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.173 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:59.600283+0000", 2024-02-05T06:37:02.173 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001926739 2024-02-05T06:37:02.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.173 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:37:02.174 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:59.600834+0000", 2024-02-05T06:37:02.174 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00055042600000000002 2024-02-05T06:37:02.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.174 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.174 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:59.661181+0000", 2024-02-05T06:37:02.174 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.060347897999999997 2024-02-05T06:37:02.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.174 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:37:02.174 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:59.661307+0000", 2024-02-05T06:37:02.174 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000125251 2024-02-05T06:37:02.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.174 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.174 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:37:02.175 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:35:59.661349+0000", 2024-02-05T06:37:02.175 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1900000000000002e-05 2024-02-05T06:37:02.175 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.175 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:37:02.175 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.175 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.175 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:53118 7.e 7:7e6be95a:::benchmark_data_smithi028_77509_object53117:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-02-05T06:37:02.176 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:36:06.043603+0000", 2024-02-05T06:37:02.176 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 56.032614410999997, 2024-02-05T06:37:02.176 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.067175597000000004, 2024-02-05T06:37:02.176 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:37:02.177 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:37:02.177 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:37:02.177 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:37:02.177 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:37:02.177 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 53118 2024-02-05T06:37:02.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.177 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:37:02.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.177 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:37:02.177 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.043603+0000", 2024-02-05T06:37:02.177 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.177 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.177 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:37:02.178 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.043603+0000", 2024-02-05T06:37:02.178 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.178 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:37:02.178 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.043605+0000", 2024-02-05T06:37:02.178 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0650000000000001e-06 2024-02-05T06:37:02.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.178 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:37:02.178 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.043628+0000", 2024-02-05T06:37:02.178 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2751999999999998e-05 2024-02-05T06:37:02.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.178 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.179 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:37:02.179 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.043645+0000", 2024-02-05T06:37:02.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6381000000000001e-05 2024-02-05T06:37:02.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.179 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:37:02.179 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.043655+0000", 2024-02-05T06:37:02.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0078000000000001e-05 2024-02-05T06:37:02.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.179 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.179 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:37:02.179 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.043720+0000", 2024-02-05T06:37:02.179 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5384999999999998e-05 2024-02-05T06:37:02.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.180 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:37:02.180 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.043842+0000", 2024-02-05T06:37:02.180 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012194800000000001 2024-02-05T06:37:02.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.180 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:37:02.180 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.043980+0000", 2024-02-05T06:37:02.180 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013763600000000001 2024-02-05T06:37:02.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.180 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.180 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.180 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.050729+0000", 2024-02-05T06:37:02.180 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0067498000000000002 2024-02-05T06:37:02.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.181 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:37:02.181 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.053317+0000", 2024-02-05T06:37:02.181 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025876179999999999 2024-02-05T06:37:02.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.181 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.181 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.110664+0000", 2024-02-05T06:37:02.181 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.057346895000000002 2024-02-05T06:37:02.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.181 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.181 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:37:02.181 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.110735+0000", 2024-02-05T06:37:02.181 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1434999999999996e-05 2024-02-05T06:37:02.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.182 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:37:02.182 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.110779+0000", 2024-02-05T06:37:02.182 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3603999999999998e-05 2024-02-05T06:37:02.182 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.182 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:37:02.182 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.182 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.182 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:53120 7.e 7:7eab26fd:::benchmark_data_smithi028_77509_object53119:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-02-05T06:37:02.182 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:36:06.045399+0000", 2024-02-05T06:37:02.182 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 56.030819178000002, 2024-02-05T06:37:02.182 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.065558006000000002, 2024-02-05T06:37:02.183 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:37:02.183 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:37:02.183 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:37:02.183 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:37:02.183 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:37:02.183 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 53120 2024-02-05T06:37:02.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.183 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:37:02.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.183 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:37:02.183 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.045399+0000", 2024-02-05T06:37:02.183 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:37:02.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.045399+0000", 2024-02-05T06:37:02.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:37:02.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.045400+0000", 2024-02-05T06:37:02.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9069999999999999e-06 2024-02-05T06:37:02.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.184 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.184 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:37:02.184 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.045422+0000", 2024-02-05T06:37:02.184 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1889e-05 2024-02-05T06:37:02.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.185 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:37:02.185 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.045434+0000", 2024-02-05T06:37:02.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1623e-05 2024-02-05T06:37:02.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.185 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:37:02.185 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.045443+0000", 2024-02-05T06:37:02.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6840000000000002e-06 2024-02-05T06:37:02.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.185 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.185 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:37:02.185 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.046067+0000", 2024-02-05T06:37:02.185 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00062429800000000002 2024-02-05T06:37:02.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:37:02.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.046195+0000", 2024-02-05T06:37:02.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012797000000000001 2024-02-05T06:37:02.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:37:02.186 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.047234+0000", 2024-02-05T06:37:02.186 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010392520000000001 2024-02-05T06:37:02.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.186 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.186 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.050768+0000", 2024-02-05T06:37:02.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035334329999999999 2024-02-05T06:37:02.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:37:02.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.053322+0000", 2024-02-05T06:37:02.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002554654 2024-02-05T06:37:02.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.187 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.187 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.110862+0000", 2024-02-05T06:37:02.187 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.057539726999999999 2024-02-05T06:37:02.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.187 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.188 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:37:02.188 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.110920+0000", 2024-02-05T06:37:02.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8103999999999998e-05 2024-02-05T06:37:02.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.188 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:37:02.188 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.110957+0000", 2024-02-05T06:37:02.188 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6464999999999997e-05 2024-02-05T06:37:02.188 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.188 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:37:02.188 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.188 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:53131 7.e 7:72bea833:::benchmark_data_smithi028_77509_object53130:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-02-05T06:37:02.189 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:36:06.050073+0000", 2024-02-05T06:37:02.189 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 56.026145049, 2024-02-05T06:37:02.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.070007110999999997, 2024-02-05T06:37:02.189 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:37:02.189 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:37:02.189 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:37:02.189 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:37:02.189 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:37:02.189 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 53131 2024-02-05T06:37:02.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.189 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:37:02.189 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.189 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:37:02.189 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.050073+0000", 2024-02-05T06:37:02.189 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:37:02.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.050073+0000", 2024-02-05T06:37:02.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:37:02.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.050074+0000", 2024-02-05T06:37:02.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.635e-06 2024-02-05T06:37:02.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.190 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.190 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:37:02.190 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.050097+0000", 2024-02-05T06:37:02.190 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2697999999999999e-05 2024-02-05T06:37:02.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.191 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:37:02.191 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.050106+0000", 2024-02-05T06:37:02.191 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0750000000000004e-06 2024-02-05T06:37:02.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.191 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:37:02.191 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.050114+0000", 2024-02-05T06:37:02.191 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4959999999999999e-06 2024-02-05T06:37:02.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.191 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.191 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:37:02.192 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.051104+0000", 2024-02-05T06:37:02.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00099033400000000005 2024-02-05T06:37:02.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.192 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:37:02.192 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.051294+0000", 2024-02-05T06:37:02.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019023799999999999 2024-02-05T06:37:02.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.192 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.192 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:37:02.192 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.051432+0000", 2024-02-05T06:37:02.192 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000137773 2024-02-05T06:37:02.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:37:02.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.053545+0000", 2024-02-05T06:37:02.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021125839999999998 2024-02-05T06:37:02.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.055255+0000", 2024-02-05T06:37:02.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001710589 2024-02-05T06:37:02.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.193 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.193 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.119979+0000", 2024-02-05T06:37:02.193 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.064724300999999998 2024-02-05T06:37:02.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:37:02.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.120041+0000", 2024-02-05T06:37:02.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1851000000000001e-05 2024-02-05T06:37:02.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.194 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.194 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:37:02.194 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:06.120080+0000", 2024-02-05T06:37:02.194 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8537000000000001e-05 2024-02-05T06:37:02.194 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.194 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:37:02.194 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.194 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.195 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.195 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:79340 7.e 7:77ecd7d0:::benchmark_data_smithi028_77509_object79339:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-02-05T06:37:02.195 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:36:28.970353+0000", 2024-02-05T06:37:02.195 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 33.105865227000002, 2024-02-05T06:37:02.195 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.051647291999999997, 2024-02-05T06:37:02.195 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:37:02.195 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:37:02.195 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:37:02.195 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:37:02.195 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:37:02.195 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 79340 2024-02-05T06:37:02.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.196 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:37:02.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.196 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:37:02.196 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:28.970353+0000", 2024-02-05T06:37:02.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.196 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:37:02.196 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:28.970353+0000", 2024-02-05T06:37:02.196 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.196 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.196 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.196 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:37:02.196 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:28.970356+0000", 2024-02-05T06:37:02.197 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1020000000000001e-06 2024-02-05T06:37:02.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.197 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:37:02.197 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:28.970381+0000", 2024-02-05T06:37:02.197 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5471999999999999e-05 2024-02-05T06:37:02.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.197 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:37:02.197 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:28.970403+0000", 2024-02-05T06:37:02.197 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2317e-05 2024-02-05T06:37:02.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.197 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.197 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:37:02.198 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:28.970414+0000", 2024-02-05T06:37:02.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0617e-05 2024-02-05T06:37:02.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.198 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:37:02.198 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:28.972350+0000", 2024-02-05T06:37:02.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001935864 2024-02-05T06:37:02.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.198 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:37:02.198 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:28.972510+0000", 2024-02-05T06:37:02.198 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016042300000000001 2024-02-05T06:37:02.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.198 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.199 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:37:02.199 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:28.972686+0000", 2024-02-05T06:37:02.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017597700000000001 2024-02-05T06:37:02.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.199 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.199 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:28.975729+0000", 2024-02-05T06:37:02.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0030431059999999998 2024-02-05T06:37:02.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.199 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:37:02.199 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:28.976926+0000", 2024-02-05T06:37:02.199 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00119709 2024-02-05T06:37:02.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.200 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.200 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.021861+0000", 2024-02-05T06:37:02.200 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.044934953 2024-02-05T06:37:02.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.200 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:37:02.200 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.021959+0000", 2024-02-05T06:37:02.200 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7513e-05 2024-02-05T06:37:02.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.200 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.200 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:37:02.200 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:29.022000+0000", 2024-02-05T06:37:02.200 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0858000000000001e-05 2024-02-05T06:37:02.201 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.201 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:37:02.201 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.201 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.201 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:106374 7.8 7:142626a4:::benchmark_data_smithi028_77509_object106373:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-02-05T06:37:02.201 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:36:57.357586+0000", 2024-02-05T06:37:02.201 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.7186314669999998, 2024-02-05T06:37:02.201 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.056254534000000002, 2024-02-05T06:37:02.201 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:37:02.201 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:37:02.201 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:37:02.201 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:37:02.201 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:37:02.202 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 106374 2024-02-05T06:37:02.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.202 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:37:02.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.202 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:37:02.202 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:57.357586+0000", 2024-02-05T06:37:02.202 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.202 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:37:02.202 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:57.357586+0000", 2024-02-05T06:37:02.202 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.202 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.202 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:37:02.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:57.357587+0000", 2024-02-05T06:37:02.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.035e-06 2024-02-05T06:37:02.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:37:02.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:57.357609+0000", 2024-02-05T06:37:02.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1875999999999999e-05 2024-02-05T06:37:02.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:37:02.203 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:57.357619+0000", 2024-02-05T06:37:02.203 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6940000000000007e-06 2024-02-05T06:37:02.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.203 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.203 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:37:02.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:57.357626+0000", 2024-02-05T06:37:02.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2699999999999999e-06 2024-02-05T06:37:02.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:37:02.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:57.360168+0000", 2024-02-05T06:37:02.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025414600000000002 2024-02-05T06:37:02.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.204 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:37:02.204 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:57.360312+0000", 2024-02-05T06:37:02.204 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014452000000000001 2024-02-05T06:37:02.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.204 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.205 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4,7", 2024-02-05T06:37:02.205 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:57.360470+0000", 2024-02-05T06:37:02.205 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015807000000000001 2024-02-05T06:37:02.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.205 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.205 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:57.367790+0000", 2024-02-05T06:37:02.205 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073193149999999998 2024-02-05T06:37:02.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.205 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.205 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:57.367833+0000", 2024-02-05T06:37:02.205 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3386000000000001e-05 2024-02-05T06:37:02.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.205 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.206 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:37:02.206 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:57.371995+0000", 2024-02-05T06:37:02.206 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041616320000000002 2024-02-05T06:37:02.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.206 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.206 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.206 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:57.413683+0000", 2024-02-05T06:37:02.206 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.041688702000000001 2024-02-05T06:37:02.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.206 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.206 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:37:02.206 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:57.413725+0000", 2024-02-05T06:37:02.206 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2109e-05 2024-02-05T06:37:02.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.206 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.207 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:37:02.207 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:57.413841+0000", 2024-02-05T06:37:02.207 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011546500000000001 2024-02-05T06:37:02.207 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.207 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:37:02.207 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.207 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.207 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:106928 7.e 7:7ade8df6:::benchmark_data_smithi028_77509_object106927:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-02-05T06:37:02.207 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:36:57.921524+0000", 2024-02-05T06:37:02.207 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4.1546932630000004, 2024-02-05T06:37:02.207 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.055058906999999997, 2024-02-05T06:37:02.207 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:37:02.207 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:37:02.207 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:37:02.208 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:37:02.208 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:37:02.208 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 106928 2024-02-05T06:37:02.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.208 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:37:02.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.208 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:37:02.208 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:57.921524+0000", 2024-02-05T06:37:02.208 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.208 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.208 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:37:02.208 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:57.921524+0000", 2024-02-05T06:37:02.208 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.209 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:37:02.209 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:57.921526+0000", 2024-02-05T06:37:02.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3790000000000001e-06 2024-02-05T06:37:02.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.209 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:37:02.209 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:57.921549+0000", 2024-02-05T06:37:02.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2875999999999999e-05 2024-02-05T06:37:02.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.209 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.209 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:37:02.209 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:57.921584+0000", 2024-02-05T06:37:02.209 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4987000000000002e-05 2024-02-05T06:37:02.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.210 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.210 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:37:02.210 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:57.921593+0000", 2024-02-05T06:37:02.210 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9989999999999997e-06 2024-02-05T06:37:02.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.210 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.210 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:37:02.210 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:57.927500+0000", 2024-02-05T06:37:02.210 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0059072359999999997 2024-02-05T06:37:02.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.210 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.210 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:37:02.210 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:57.927654+0000", 2024-02-05T06:37:02.211 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000154034 2024-02-05T06:37:02.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.211 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:37:02.211 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:57.927804+0000", 2024-02-05T06:37:02.211 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014999600000000001 2024-02-05T06:37:02.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.211 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.211 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:57.932546+0000", 2024-02-05T06:37:02.211 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047417470000000001 2024-02-05T06:37:02.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.212 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:37:02.212 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:57.934941+0000", 2024-02-05T06:37:02.212 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002395561 2024-02-05T06:37:02.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.212 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.212 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.212 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:57.975986+0000", 2024-02-05T06:37:02.212 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.041044535 2024-02-05T06:37:02.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.212 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.212 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:37:02.212 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:57.976363+0000", 2024-02-05T06:37:02.212 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00037681900000000003 2024-02-05T06:37:02.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.213 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:37:02.213 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:57.976583+0000", 2024-02-05T06:37:02.213 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000220738 2024-02-05T06:37:02.213 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.213 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:37:02.213 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.213 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.213 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:108098 7.8 7:10a2e9c6:::benchmark_data_smithi028_77509_object108097:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-02-05T06:37:02.213 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:36:59.320249+0000", 2024-02-05T06:37:02.213 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 2.7559683650000002, 2024-02-05T06:37:02.213 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.047272145000000002, 2024-02-05T06:37:02.214 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:37:02.214 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:37:02.214 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:37:02.214 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:37:02.214 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:37:02.214 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 108098 2024-02-05T06:37:02.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.214 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:37:02.214 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.214 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:37:02.214 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:59.320249+0000", 2024-02-05T06:37:02.214 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.215 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:37:02.215 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:59.320249+0000", 2024-02-05T06:37:02.215 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:37:02.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.215 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:37:02.215 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:59.320252+0000", 2024-02-05T06:37:02.215 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.954e-06 2024-02-05T06:37:02.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.215 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.215 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:37:02.216 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:59.320284+0000", 2024-02-05T06:37:02.216 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2054000000000002e-05 2024-02-05T06:37:02.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.216 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:37:02.216 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:59.320314+0000", 2024-02-05T06:37:02.216 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9213999999999998e-05 2024-02-05T06:37:02.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.216 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.216 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:37:02.216 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:59.320325+0000", 2024-02-05T06:37:02.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1066999999999999e-05 2024-02-05T06:37:02.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.217 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:37:02.217 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:59.320508+0000", 2024-02-05T06:37:02.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018306900000000001 2024-02-05T06:37:02.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.217 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:37:02.217 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:59.320937+0000", 2024-02-05T06:37:02.217 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00042894100000000001 2024-02-05T06:37:02.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.217 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.218 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4,7", 2024-02-05T06:37:02.218 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:59.321486+0000", 2024-02-05T06:37:02.218 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00054969899999999996 2024-02-05T06:37:02.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.218 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.218 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.218 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:59.328248+0000", 2024-02-05T06:37:02.218 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0067618820000000003 2024-02-05T06:37:02.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.218 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.218 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.218 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:59.328297+0000", 2024-02-05T06:37:02.219 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8779000000000002e-05 2024-02-05T06:37:02.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.219 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:37:02.219 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:59.330704+0000", 2024-02-05T06:37:02.219 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024068879999999998 2024-02-05T06:37:02.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.219 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:37:02.219 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:59.367336+0000", 2024-02-05T06:37:02.219 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.036632262999999998 2024-02-05T06:37:02.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.219 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.219 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:37:02.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:59.367409+0000", 2024-02-05T06:37:02.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2784e-05 2024-02-05T06:37:02.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:37:02.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:37:02.220 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:37:02.220 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:36:59.367522+0000", 2024-02-05T06:37:02.220 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000112551 2024-02-05T06:37:02.220 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.220 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:37:02.220 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.220 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:37:02.221 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:37:02.221 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:37:02.230 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 85 15 110567 110552 81.2543 49.8125 0.0132574 0.0123025 2024-02-05T06:37:02.236 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:02.267 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:02.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:02.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:02.369 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:02.398 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:02.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:02.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:02.500 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:02.536 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:02.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:02.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:02.638 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:02.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:02.681 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:02.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:02.695 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:37:02 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:37:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:37:02.782 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:02.809 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:02.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:02.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:02.911 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:02.944 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:02.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:02.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:02.980 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:37:02 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:37:03.045 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:03.076 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:03.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:03.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:03.178 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:03.227 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:03.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:03.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:03.230 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 86 15 111509 111494 80.9941 58.875 0.0155543 0.0123415 2024-02-05T06:37:03.328 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:03.358 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:03.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:03.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:03.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:03 smithi028 ceph-mon[29442]: pgmap v129: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.6 KiB/s rd, 79 MiB/s wr, 1.26k op/s; 32272/329952 objects misplaced (9.781%); 23 MiB/s, 375 objects/s recovering 2024-02-05T06:37:03.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:03 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:37:03.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:03 smithi028 ceph-mon[37984]: pgmap v129: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.6 KiB/s rd, 79 MiB/s wr, 1.26k op/s; 32272/329952 objects misplaced (9.781%); 23 MiB/s, 375 objects/s recovering 2024-02-05T06:37:03.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:03 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:37:03.460 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:03.489 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:03.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:03.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:03.591 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:03.621 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:03.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:03.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:03.723 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:03.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:03 smithi167 ceph-mon[32774]: pgmap v129: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.6 KiB/s rd, 79 MiB/s wr, 1.26k op/s; 32272/329952 objects misplaced (9.781%); 23 MiB/s, 375 objects/s recovering 2024-02-05T06:37:03.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:03 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:37:03.752 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:03.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:03.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:03.854 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:03.883 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:03.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:03.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:03.985 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:04.021 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:04.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:04.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:04.122 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:04.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:04.153 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:04.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:04.230 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 87 16 112487 112471 80.765 61.0625 0.027478 0.0123765 2024-02-05T06:37:04.255 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:04.285 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:04.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:04.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:04.387 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:04.432 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:04.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:04.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:04.494 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-05T06:37:04.534 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:04.562 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:04.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:04.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:04.664 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:04.712 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:04.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:04.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:04.814 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:04.842 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:04.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:04.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:04.944 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:04.972 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:04.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:04.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:05.074 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:05.117 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:05.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:05.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:05.219 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:05.230 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 88 16 113390 113374 80.4886 56.4375 0.00989268 0.0124187 2024-02-05T06:37:05.254 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:37:05.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:05.290 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:05.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:05.391 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:05.453 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:05.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:05.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:05.530 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:05 smithi028 ceph-mon[37984]: pgmap v130: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 53 MiB/s wr, 848 op/s; 32272/329952 objects misplaced (9.781%); 18 MiB/s, 290 objects/s recovering 2024-02-05T06:37:05.530 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:05 smithi028 ceph-mon[29442]: pgmap v130: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 53 MiB/s wr, 848 op/s; 32272/329952 objects misplaced (9.781%); 18 MiB/s, 290 objects/s recovering 2024-02-05T06:37:05.555 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:05.603 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:05.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:05.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:05.705 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:05.718 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:05 smithi167 ceph-mon[32774]: pgmap v130: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 53 MiB/s wr, 848 op/s; 32272/329952 objects misplaced (9.781%); 18 MiB/s, 290 objects/s recovering 2024-02-05T06:37:05.736 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:05.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:05.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:05.839 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:05.870 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:05.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:05.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:05.972 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:06.006 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:06.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:06.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:06.108 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:06.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:06.144 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:06.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:06.230 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 89 16 114273 114257 80.2043 55.1875 0.016244 0.0124633 2024-02-05T06:37:06.246 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:06.284 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:06.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:06.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:06.386 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:06.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:06.418 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:06.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:06.519 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:06.551 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:06.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:06.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:06.653 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:06.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:06.700 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:06.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:06.704 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:37:06.801 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:06.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:06.848 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:06.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:06.949 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:06.986 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:06.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:06.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:07.088 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:07.124 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:07.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:07.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:07.227 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:07.230 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Total time run: 90.0126 2024-02-05T06:37:07.230 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Total writes made: 115117 2024-02-05T06:37:07.230 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Write size: 65536 2024-02-05T06:37:07.230 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Object size: 65536 2024-02-05T06:37:07.230 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Bandwidth (MB/sec): 79.9312 2024-02-05T06:37:07.230 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Stddev Bandwidth: 32.346 2024-02-05T06:37:07.230 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Max bandwidth (MB/sec): 202.062 2024-02-05T06:37:07.231 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Min bandwidth (MB/sec): 5.25 2024-02-05T06:37:07.231 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Average IOPS: 1278 2024-02-05T06:37:07.231 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Stddev IOPS: 517.536 2024-02-05T06:37:07.231 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Max IOPS: 3233 2024-02-05T06:37:07.231 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Min IOPS: 84 2024-02-05T06:37:07.231 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Average Latency(s): 0.0125069 2024-02-05T06:37:07.231 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Stddev Latency(s): 0.0223295 2024-02-05T06:37:07.231 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Max latency(s): 1.79789 2024-02-05T06:37:07.231 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Min latency(s): 0.0013072 2024-02-05T06:37:07.237 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Cleaning up (deleting benchmark objects) 2024-02-05T06:37:07.261 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:07.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:07.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:07.363 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:07.397 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:07.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:07.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:07.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:07.543 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:07.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:07.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:07.645 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:07.677 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:07.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:07.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:07.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:07 smithi028 ceph-mon[29442]: pgmap v131: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 44 MiB/s wr, 702 op/s; 32272/334140 objects misplaced (9.658%); 17 MiB/s, 265 objects/s recovering 2024-02-05T06:37:07.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:07 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:37:07.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:07 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:37:07.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:07 smithi028 ceph-mon[29442]: osdmap e123: 8 total, 8 up, 8 in 2024-02-05T06:37:07.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:07 smithi028 ceph-mon[37984]: pgmap v131: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 44 MiB/s wr, 702 op/s; 32272/334140 objects misplaced (9.658%); 17 MiB/s, 265 objects/s recovering 2024-02-05T06:37:07.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:07 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:37:07.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:07 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:37:07.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:07 smithi028 ceph-mon[37984]: osdmap e123: 8 total, 8 up, 8 in 2024-02-05T06:37:07.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:07 smithi167 ceph-mon[32774]: pgmap v131: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 44 MiB/s wr, 702 op/s; 32272/334140 objects misplaced (9.658%); 17 MiB/s, 265 objects/s recovering 2024-02-05T06:37:07.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:07 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:37:07.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:07 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:37:07.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:07 smithi167 ceph-mon[32774]: osdmap e123: 8 total, 8 up, 8 in 2024-02-05T06:37:07.779 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:07.803 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:07.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:07.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:07.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:07.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:07.945 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:07.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:08.046 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:08.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:08.072 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:08.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:08.173 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:08.201 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:08.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:08.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:08.303 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:08.326 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:08.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:08.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:08.428 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:08.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:08.469 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:08.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:08.570 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:08.613 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:08.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:08.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:08.715 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:08.760 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:08.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:08.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:08.862 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:08.890 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:08.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:08.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:08.993 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:09.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:09.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:09.143 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:09.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:09.171 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:09.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:09.272 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:09.315 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:09.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:09.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:09.417 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:09.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:09.457 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:09.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:09.558 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:09.586 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:09 smithi028 ceph-mon[37984]: pgmap v133: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.08k op/s; 31156/343473 objects misplaced (9.071%); 35 MiB/s, 556 objects/s recovering 2024-02-05T06:37:09.586 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:09 smithi028 ceph-mon[29442]: pgmap v133: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.08k op/s; 31156/343473 objects misplaced (9.071%); 35 MiB/s, 556 objects/s recovering 2024-02-05T06:37:09.595 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:09.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:09.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:09.697 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:09.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:09 smithi167 ceph-mon[32774]: pgmap v133: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.08k op/s; 31156/343473 objects misplaced (9.071%); 35 MiB/s, 556 objects/s recovering 2024-02-05T06:37:09.736 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:09.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:09.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:09.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:09.877 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:09.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:09.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:09.978 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:10.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:10.009 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:10.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:10.027 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:37:10.037 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-05T06:37:10.111 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:10.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:10.135 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:10.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:10.236 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:10.260 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:10.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:10.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:10.362 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:10.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:10.386 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:10.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:10.487 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:10.517 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:10.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:10.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:10.619 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:10.647 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:10.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:10.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:10.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:10 smithi028 ceph-mon[37984]: from='client.24826 v1:172.21.15.28:0/543223570' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:37:10.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:10 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:37:10.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:10 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:37:10.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:10 smithi028 ceph-mon[29442]: from='client.24826 v1:172.21.15.28:0/543223570' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:37:10.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:10 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:37:10.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:10 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:37:10.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:10 smithi167 ceph-mon[32774]: from='client.24826 v1:172.21.15.28:0/543223570' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:37:10.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:10 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:37:10.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:10 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:37:10.750 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:10.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:10.790 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:10.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:10.891 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:10.921 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:10.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:10.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:11.023 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:11.053 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:11.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:11.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:11.155 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:11.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:11.195 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:11.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:11.296 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:11.337 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:11.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:11.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:11.439 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:11.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:11.478 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:11.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:11.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:11 smithi028 ceph-mon[29442]: pgmap v134: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 890 op/s; 31156/343473 objects misplaced (9.071%); 29 MiB/s, 456 objects/s recovering 2024-02-05T06:37:11.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:11 smithi028 ceph-mon[37984]: pgmap v134: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 890 op/s; 31156/343473 objects misplaced (9.071%); 29 MiB/s, 456 objects/s recovering 2024-02-05T06:37:11.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:11 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:37:11.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:11 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:37:11.580 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:11.609 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:11.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:11.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:11.711 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:11.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:11 smithi167 ceph-mon[32774]: pgmap v134: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 7.0 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 890 op/s; 31156/343473 objects misplaced (9.071%); 29 MiB/s, 456 objects/s recovering 2024-02-05T06:37:11.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:11 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:37:11.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:11 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:37:11.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:11 smithi167 ceph-mon[32774]: osdmap e124: 8 total, 8 up, 8 in 2024-02-05T06:37:11.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:11 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1836082080' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:37:11.740 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:11.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:11.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:11.795 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:11 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:37:11.795 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:11 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:37:11.795 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:11 smithi028 ceph-mon[29442]: osdmap e124: 8 total, 8 up, 8 in 2024-02-05T06:37:11.795 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:11 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1836082080' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:37:11.795 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:11 smithi028 ceph-mon[37984]: osdmap e124: 8 total, 8 up, 8 in 2024-02-05T06:37:11.796 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:11 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1836082080' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:37:11.842 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:11.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:11.871 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:11.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:11.972 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:12.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:12.004 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:12.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:12.100 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:37:12.100 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:37:12.100 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:37:12.100 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:37:12.100 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:37:12.100 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:37:12.101 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:37:12.101 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:37:12.101 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:37:12.101 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:37:12.101 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:37:12.101 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:37:12.101 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:37:12.101 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:37:12.101 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:37:12.101 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:37:12.105 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:12.136 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:12.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:12.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:12.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:12.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:12.285 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:12.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:12.386 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:12.414 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:12.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:12.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:12.516 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:12.557 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:12.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:12.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:12.659 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:12.695 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:37:12 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:37:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:37:12.726 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:12.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:12.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:12.828 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:12.858 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:12.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:12.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:12.960 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:12.980 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:37:12 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:37:13.003 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:13.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:13.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:13.105 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:13.139 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:13.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:13.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:13.241 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:13.282 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:13.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:13.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:13.384 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:13.431 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:13.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:13.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:13.533 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:13.564 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:13.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:13.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:13.666 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:13.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:13 smithi028 ceph-mon[37984]: pgmap v136: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 42 MiB/s wr, 1.05k op/s; 28410/336729 objects misplaced (8.437%); 35 MiB/s, 567 objects/s recovering 2024-02-05T06:37:13.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:13 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:37:13.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:13 smithi028 ceph-mon[29442]: pgmap v136: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 42 MiB/s wr, 1.05k op/s; 28410/336729 objects misplaced (8.437%); 35 MiB/s, 567 objects/s recovering 2024-02-05T06:37:13.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:13 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:37:13.699 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:13.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:13.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:13.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:13 smithi167 ceph-mon[32774]: pgmap v136: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 42 MiB/s wr, 1.05k op/s; 28410/336729 objects misplaced (8.437%); 35 MiB/s, 567 objects/s recovering 2024-02-05T06:37:13.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:13 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:37:13.801 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:13.842 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:13.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:13.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:13.944 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:13.973 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:13.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:13.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:14.075 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:14.103 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:14.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:14.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:14.205 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:14.238 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:14.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:14.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:14.340 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:14.385 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:14.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:14.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:14.486 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:14.528 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:14.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:14.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:14.630 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:14.660 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:14.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:14.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:14.762 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:14.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:14.804 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:14.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:14.904 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:14.935 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:14.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:14.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:15.037 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:15.067 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:15.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:15.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:15.168 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:15.227 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:15.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:15.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:15.249 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2024-02-05T06:37:08.202966+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7490902364,"num_objects":114491,"num_object_clones":0,"num_object_copies":343473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":31156,"num_objects_unfound":0,"num_objects_dirty":114491,"num_whiteouts":0,"num_read":1131,"num_read_kb":977,"num_write":114908,"num_write_kb":7317213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4856,"num_bytes_recovered":308418628,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84562,"ondisk_log_size":84562,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":449,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":22760476,"kb_used_data":22155676,"kb_used_omap":12,"kb_used_meta":604723,"kb_avail":727200740,"statfs":{"total":767960285184,"available":744653557760,"internally_reserved":0,"allocated":22687412224,"data_stored":22684329780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":619236913},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,6,3,4,4,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":583401472,"num_objects":8902,"num_object_clones":0,"num_object_copies":26706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2008,"num_objects_unfound":0,"num_objects_dirty":8902,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8902,"num_write_kb":569728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4571,"num_bytes_recovered":299565056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.209283"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":17,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.988642+0000","last_change":"2024-02-05T06:36:54.954342+0000","last_active":"2024-02-05T06:36:59.988642+0000","last_peered":"2024-02-05T06:36:59.988642+0000","last_clean":"2024-02-05T06:36:59.988642+0000","last_became_active":"2024-02-05T06:36:54.952487+0000","last_became_peered":"2024-02-05T06:36:54.952487+0000","last_unstale":"2024-02-05T06:36:59.988642+0000","last_undegraded":"2024-02-05T06:36:59.988642+0000","last_fullsized":"2024-02-05T06:36:59.988642+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:32.017855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:32.017855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:45.407053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":131,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.996684+0000","last_change":"2024-02-05T06:36:37.918740+0000","last_active":"2024-02-05T06:36:59.996684+0000","last_peered":"2024-02-05T06:36:59.996684+0000","last_clean":"2024-02-05T06:36:59.996684+0000","last_became_active":"2024-02-05T06:33:33.585848+0000","last_became_peered":"2024-02-05T06:33:33.585848+0000","last_unstale":"2024-02-05T06:36:59.996684+0000","last_undegraded":"2024-02-05T06:36:59.996684+0000","last_fullsized":"2024-02-05T06:36:59.996684+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:37.918224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:06.525012+0000","last_clean_scrub_stamp":"2024-02-05T06:36:37.918224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:00.105543+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":134,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-02-05T06:37:06.321017+0000","last_change":"2024-02-05T06:36:09.154584+0000","last_active":"2024-02-05T06:37:06.321017+0000","last_peered":"2024-02-05T06:37:06.321017+0000","last_clean":"2024-02-05T06:37:06.321017+0000","last_became_active":"2024-02-05T06:33:29.549592+0000","last_became_peered":"2024-02-05T06:33:29.549592+0000","last_unstale":"2024-02-05T06:37:06.321017+0000","last_undegraded":"2024-02-05T06:37:06.321017+0000","last_fullsized":"2024-02-05T06:37:06.321017+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:09.154442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:09.154442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:14.894164+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":27,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.987530+0000","last_change":"2024-02-05T06:36:54.997747+0000","last_active":"2024-02-05T06:36:59.987530+0000","last_peered":"2024-02-05T06:36:59.987530+0000","last_clean":"2024-02-05T06:36:59.987530+0000","last_became_active":"2024-02-05T06:36:54.959330+0000","last_became_peered":"2024-02-05T06:36:54.959330+0000","last_unstale":"2024-02-05T06:36:59.987530+0000","last_undegraded":"2024-02-05T06:36:59.987530+0000","last_fullsized":"2024-02-05T06:36:59.987530+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:35:01.601838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:35:01.601838+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":7,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":120,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.998010+0000","last_change":"2024-02-05T06:36:12.220215+0000","last_active":"2024-02-05T06:36:59.998010+0000","last_peered":"2024-02-05T06:36:59.998010+0000","last_clean":"2024-02-05T06:36:59.998010+0000","last_became_active":"2024-02-05T06:33:35.667778+0000","last_became_peered":"2024-02-05T06:33:35.667778+0000","last_unstale":"2024-02-05T06:36:59.998010+0000","last_undegraded":"2024-02-05T06:36:59.998010+0000","last_fullsized":"2024-02-05T06:36:59.998010+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:12.220090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:12.220090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:20.586314+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":17,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.987435+0000","last_change":"2024-02-05T06:36:54.961637+0000","last_active":"2024-02-05T06:36:59.987435+0000","last_peered":"2024-02-05T06:36:59.987435+0000","last_clean":"2024-02-05T06:36:59.987435+0000","last_became_active":"2024-02-05T06:36:54.959640+0000","last_became_peered":"2024-02-05T06:36:54.959640+0000","last_unstale":"2024-02-05T06:36:59.987435+0000","last_undegraded":"2024-02-05T06:36:59.987435+0000","last_fullsized":"2024-02-05T06:36:59.987435+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:13.639311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:35:13.639311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":140,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.997372+0000","last_change":"2024-02-05T06:36:17.296405+0000","last_active":"2024-02-05T06:36:59.997372+0000","last_peered":"2024-02-05T06:36:59.997372+0000","last_clean":"2024-02-05T06:36:59.997372+0000","last_became_active":"2024-02-05T06:33:29.553864+0000","last_became_peered":"2024-02-05T06:33:29.553864+0000","last_unstale":"2024-02-05T06:36:59.997372+0000","last_undegraded":"2024-02-05T06:36:59.997372+0000","last_fullsized":"2024-02-05T06:36:59.997372+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:17.296252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:17.296252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:27.660356+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"88'22","reported_seq":168,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.994954+0000","last_change":"2024-02-05T06:36:19.337617+0000","last_active":"2024-02-05T06:36:59.994954+0000","last_peered":"2024-02-05T06:36:59.994954+0000","last_clean":"2024-02-05T06:36:59.994954+0000","last_became_active":"2024-02-05T06:33:31.572544+0000","last_became_peered":"2024-02-05T06:33:31.572544+0000","last_unstale":"2024-02-05T06:36:59.994954+0000","last_undegraded":"2024-02-05T06:36:59.994954+0000","last_fullsized":"2024-02-05T06:36:59.994954+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'22","last_scrub_stamp":"2024-02-05T06:36:19.337341+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:34:35.832368+0000","last_clean_scrub_stamp":"2024-02-05T06:36:19.337341+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:29.086209+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":122,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-02-05T06:37:06.320645+0000","last_change":"2024-02-05T06:35:20.603134+0000","last_active":"2024-02-05T06:37:06.320645+0000","last_peered":"2024-02-05T06:37:06.320645+0000","last_clean":"2024-02-05T06:37:06.320645+0000","last_became_active":"2024-02-05T06:33:35.594860+0000","last_became_peered":"2024-02-05T06:33:35.594860+0000","last_unstale":"2024-02-05T06:37:06.320645+0000","last_undegraded":"2024-02-05T06:37:06.320645+0000","last_fullsized":"2024-02-05T06:37:06.320645+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:20.603044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:20.603044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":130,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.996300+0000","last_change":"2024-02-05T06:36:07.749745+0000","last_active":"2024-02-05T06:36:59.996300+0000","last_peered":"2024-02-05T06:36:59.996300+0000","last_clean":"2024-02-05T06:36:59.996300+0000","last_became_active":"2024-02-05T06:33:33.585459+0000","last_became_peered":"2024-02-05T06:33:33.585459+0000","last_unstale":"2024-02-05T06:36:59.996300+0000","last_undegraded":"2024-02-05T06:36:59.996300+0000","last_fullsized":"2024-02-05T06:36:59.996300+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:07.749639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:07.749639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:10.266182+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":150,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.996742+0000","last_change":"2024-02-05T06:36:54.950822+0000","last_active":"2024-02-05T06:36:59.996742+0000","last_peered":"2024-02-05T06:36:59.996742+0000","last_clean":"2024-02-05T06:36:59.996742+0000","last_became_active":"2024-02-05T06:36:54.950515+0000","last_became_peered":"2024-02-05T06:36:54.950515+0000","last_unstale":"2024-02-05T06:36:59.996742+0000","last_undegraded":"2024-02-05T06:36:59.996742+0000","last_fullsized":"2024-02-05T06:36:59.996742+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:08.762241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:44.081318+0000","last_clean_scrub_stamp":"2024-02-05T06:36:08.762241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:27.412047+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"92'13","reported_seq":27,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.988480+0000","last_change":"2024-02-05T06:36:55.009049+0000","last_active":"2024-02-05T06:36:59.988480+0000","last_peered":"2024-02-05T06:36:59.988480+0000","last_clean":"2024-02-05T06:36:59.988480+0000","last_became_active":"2024-02-05T06:36:54.969454+0000","last_became_peered":"2024-02-05T06:36:54.969454+0000","last_unstale":"2024-02-05T06:36:59.988480+0000","last_undegraded":"2024-02-05T06:36:59.988480+0000","last_fullsized":"2024-02-05T06:36:59.988480+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'13","last_scrub_stamp":"2024-02-05T06:34:46.542893+0000","last_deep_scrub":"92'13","last_deep_scrub_stamp":"2024-02-05T06:34:46.542893+0000","last_clean_scrub_stamp":"2024-02-05T06:34:46.542893+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":118,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.996303+0000","last_change":"2024-02-05T06:35:21.998937+0000","last_active":"2024-02-05T06:36:59.996303+0000","last_peered":"2024-02-05T06:36:59.996303+0000","last_clean":"2024-02-05T06:36:59.996303+0000","last_became_active":"2024-02-05T06:33:35.600399+0000","last_became_peered":"2024-02-05T06:33:35.600399+0000","last_unstale":"2024-02-05T06:36:59.996303+0000","last_undegraded":"2024-02-05T06:36:59.996303+0000","last_fullsized":"2024-02-05T06:36:59.996303+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:21.998850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:21.998850+0000","last_clean_scrub_stamp":"2024-02-05T06:35:21.998850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":136,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:00.003013+0000","last_change":"2024-02-05T06:36:54.970411+0000","last_active":"2024-02-05T06:37:00.003013+0000","last_peered":"2024-02-05T06:37:00.003013+0000","last_clean":"2024-02-05T06:37:00.003013+0000","last_became_active":"2024-02-05T06:36:54.969671+0000","last_became_peered":"2024-02-05T06:36:54.969671+0000","last_unstale":"2024-02-05T06:37:00.003013+0000","last_undegraded":"2024-02-05T06:37:00.003013+0000","last_fullsized":"2024-02-05T06:37:00.003013+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:53.073710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:53.073710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":150,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.996892+0000","last_change":"2024-02-05T06:36:54.951354+0000","last_active":"2024-02-05T06:36:59.996892+0000","last_peered":"2024-02-05T06:36:59.996892+0000","last_clean":"2024-02-05T06:36:59.996892+0000","last_became_active":"2024-02-05T06:36:54.951044+0000","last_became_peered":"2024-02-05T06:36:54.951044+0000","last_unstale":"2024-02-05T06:36:59.996892+0000","last_undegraded":"2024-02-05T06:36:59.996892+0000","last_fullsized":"2024-02-05T06:36:59.996892+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:30.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:50.055245+0000","last_clean_scrub_stamp":"2024-02-05T06:36:30.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:56.891812+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":23,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.989042+0000","last_change":"2024-02-05T06:36:54.985263+0000","last_active":"2024-02-05T06:36:59.989042+0000","last_peered":"2024-02-05T06:36:59.989042+0000","last_clean":"2024-02-05T06:36:59.989042+0000","last_became_active":"2024-02-05T06:36:54.954385+0000","last_became_peered":"2024-02-05T06:36:54.954385+0000","last_unstale":"2024-02-05T06:36:59.989042+0000","last_undegraded":"2024-02-05T06:36:59.989042+0000","last_fullsized":"2024-02-05T06:36:59.989042+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-02-05T06:34:53.638704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:53.638704+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":17,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.988386+0000","last_change":"2024-02-05T06:36:54.963609+0000","last_active":"2024-02-05T06:36:59.988386+0000","last_peered":"2024-02-05T06:36:59.988386+0000","last_clean":"2024-02-05T06:36:59.988386+0000","last_became_active":"2024-02-05T06:36:54.961535+0000","last_became_peered":"2024-02-05T06:36:54.961535+0000","last_unstale":"2024-02-05T06:36:59.988386+0000","last_undegraded":"2024-02-05T06:36:59.988386+0000","last_fullsized":"2024-02-05T06:36:59.988386+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:45.573432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:45.573432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":135,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.996975+0000","last_change":"2024-02-05T06:36:54.958978+0000","last_active":"2024-02-05T06:36:59.996975+0000","last_peered":"2024-02-05T06:36:59.996975+0000","last_clean":"2024-02-05T06:36:59.996975+0000","last_became_active":"2024-02-05T06:36:54.958069+0000","last_became_peered":"2024-02-05T06:36:54.958069+0000","last_unstale":"2024-02-05T06:36:59.996975+0000","last_undegraded":"2024-02-05T06:36:59.996975+0000","last_fullsized":"2024-02-05T06:36:59.996975+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:16.531347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:35:16.531347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:00.034084+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"66'1","reported_seq":160,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-02-05T06:37:06.320733+0000","last_change":"2024-02-05T06:36:54.991286+0000","last_active":"2024-02-05T06:37:06.320733+0000","last_peered":"2024-02-05T06:37:06.320733+0000","last_clean":"2024-02-05T06:37:06.320733+0000","last_became_active":"2024-02-05T06:36:54.971810+0000","last_became_peered":"2024-02-05T06:36:54.971810+0000","last_unstale":"2024-02-05T06:37:06.320733+0000","last_undegraded":"2024-02-05T06:37:06.320733+0000","last_fullsized":"2024-02-05T06:37:06.320733+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:35:22.640782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:35:22.640782+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"88'22","reported_seq":166,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.999949+0000","last_change":"2024-02-05T06:36:10.714443+0000","last_active":"2024-02-05T06:36:59.999949+0000","last_peered":"2024-02-05T06:36:59.999949+0000","last_clean":"2024-02-05T06:36:59.999949+0000","last_became_active":"2024-02-05T06:33:31.569763+0000","last_became_peered":"2024-02-05T06:33:31.569763+0000","last_unstale":"2024-02-05T06:36:59.999949+0000","last_undegraded":"2024-02-05T06:36:59.999949+0000","last_fullsized":"2024-02-05T06:36:59.999949+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'22","last_scrub_stamp":"2024-02-05T06:36:10.714308+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:34:35.234636+0000","last_clean_scrub_stamp":"2024-02-05T06:36:10.714308+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:16.856897+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":120,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.995630+0000","last_change":"2024-02-05T06:35:05.077348+0000","last_active":"2024-02-05T06:36:59.995630+0000","last_peered":"2024-02-05T06:36:59.995630+0000","last_clean":"2024-02-05T06:36:59.995630+0000","last_became_active":"2024-02-05T06:33:35.664852+0000","last_became_peered":"2024-02-05T06:33:35.664852+0000","last_unstale":"2024-02-05T06:36:59.995630+0000","last_undegraded":"2024-02-05T06:36:59.995630+0000","last_fullsized":"2024-02-05T06:36:59.995630+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:05.077268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:05.077268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":17,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.988724+0000","last_change":"2024-02-05T06:36:54.963177+0000","last_active":"2024-02-05T06:36:59.988724+0000","last_peered":"2024-02-05T06:36:59.988724+0000","last_clean":"2024-02-05T06:36:59.988724+0000","last_became_active":"2024-02-05T06:36:54.959289+0000","last_became_peered":"2024-02-05T06:36:54.959289+0000","last_unstale":"2024-02-05T06:36:59.988724+0000","last_undegraded":"2024-02-05T06:36:59.988724+0000","last_fullsized":"2024-02-05T06:36:59.988724+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:05.580971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:35:05.580971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.988403+0000","last_change":"2024-02-05T06:36:54.961607+0000","last_active":"2024-02-05T06:36:59.988403+0000","last_peered":"2024-02-05T06:36:59.988403+0000","last_clean":"2024-02-05T06:36:59.988403+0000","last_became_active":"2024-02-05T06:36:54.959070+0000","last_became_peered":"2024-02-05T06:36:54.959070+0000","last_unstale":"2024-02-05T06:36:59.988403+0000","last_undegraded":"2024-02-05T06:36:59.988403+0000","last_fullsized":"2024-02-05T06:36:59.988403+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:12.270778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:12.270778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:35.687670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":205,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.993115+0000","last_change":"2024-02-05T06:36:54.987702+0000","last_active":"2024-02-05T06:36:59.993115+0000","last_peered":"2024-02-05T06:36:59.993115+0000","last_clean":"2024-02-05T06:36:59.993115+0000","last_became_active":"2024-02-05T06:36:54.954880+0000","last_became_peered":"2024-02-05T06:36:54.954880+0000","last_unstale":"2024-02-05T06:36:59.993115+0000","last_undegraded":"2024-02-05T06:36:59.993115+0000","last_fullsized":"2024-02-05T06:36:59.993115+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:34:59.999726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:59.999726+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:01.557788+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"73'1","reported_seq":121,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:00.003466+0000","last_change":"2024-02-05T06:35:19.985742+0000","last_active":"2024-02-05T06:37:00.003466+0000","last_peered":"2024-02-05T06:37:00.003466+0000","last_clean":"2024-02-05T06:37:00.003466+0000","last_became_active":"2024-02-05T06:33:35.598501+0000","last_became_peered":"2024-02-05T06:33:35.598501+0000","last_unstale":"2024-02-05T06:37:00.003466+0000","last_undegraded":"2024-02-05T06:37:00.003466+0000","last_fullsized":"2024-02-05T06:37:00.003466+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-02-05T06:35:19.985656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:19.985656+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"119'10","reported_seq":203,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:00.001373+0000","last_change":"2024-02-05T06:36:54.971918+0000","last_active":"2024-02-05T06:37:00.001373+0000","last_peered":"2024-02-05T06:37:00.001373+0000","last_clean":"2024-02-05T06:37:00.001373+0000","last_became_active":"2024-02-05T06:36:54.957934+0000","last_became_peered":"2024-02-05T06:36:54.957934+0000","last_unstale":"2024-02-05T06:37:00.001373+0000","last_undegraded":"2024-02-05T06:37:00.001373+0000","last_fullsized":"2024-02-05T06:37:00.001373+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:34:55.312853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:55.312853+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:08.646516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":160,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.996599+0000","last_change":"2024-02-05T06:36:54.977901+0000","last_active":"2024-02-05T06:36:59.996599+0000","last_peered":"2024-02-05T06:36:59.996599+0000","last_clean":"2024-02-05T06:36:59.996599+0000","last_became_active":"2024-02-05T06:36:54.958414+0000","last_became_peered":"2024-02-05T06:36:54.958414+0000","last_unstale":"2024-02-05T06:36:59.996599+0000","last_undegraded":"2024-02-05T06:36:59.996599+0000","last_fullsized":"2024-02-05T06:36:59.996599+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:36:10.257008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:10.257008+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:38.113604+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"92'15","reported_seq":158,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-02-05T06:37:06.321118+0000","last_change":"2024-02-05T06:36:28.242822+0000","last_active":"2024-02-05T06:37:06.321118+0000","last_peered":"2024-02-05T06:37:06.321118+0000","last_clean":"2024-02-05T06:37:06.321118+0000","last_became_active":"2024-02-05T06:33:31.666481+0000","last_became_peered":"2024-02-05T06:33:31.666481+0000","last_unstale":"2024-02-05T06:37:06.321118+0000","last_undegraded":"2024-02-05T06:37:06.321118+0000","last_fullsized":"2024-02-05T06:37:06.321118+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'15","last_scrub_stamp":"2024-02-05T06:36:28.242304+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-02-05T06:34:50.718768+0000","last_clean_scrub_stamp":"2024-02-05T06:36:28.242304+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:43.531566+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":139,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.997146+0000","last_change":"2024-02-05T06:36:54.959126+0000","last_active":"2024-02-05T06:36:59.997146+0000","last_peered":"2024-02-05T06:36:59.997146+0000","last_clean":"2024-02-05T06:36:59.997146+0000","last_became_active":"2024-02-05T06:36:54.958621+0000","last_became_peered":"2024-02-05T06:36:54.958621+0000","last_unstale":"2024-02-05T06:36:59.997146+0000","last_undegraded":"2024-02-05T06:36:59.997146+0000","last_fullsized":"2024-02-05T06:36:59.997146+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:26.060478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:26.060478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:37.316041+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":35,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.996219+0000","last_change":"2024-02-05T06:36:12.175207+0000","last_active":"2024-02-05T06:36:59.996219+0000","last_peered":"2024-02-05T06:36:59.996219+0000","last_clean":"2024-02-05T06:36:59.996219+0000","last_became_active":"2024-02-05T06:36:12.174876+0000","last_became_peered":"2024-02-05T06:36:12.174876+0000","last_unstale":"2024-02-05T06:36:59.996219+0000","last_undegraded":"2024-02-05T06:36:59.996219+0000","last_fullsized":"2024-02-05T06:36:59.996219+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:08.713894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:08.713894+0000","last_clean_scrub_stamp":"2024-02-05T06:35:08.713894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.996097+0000","last_change":"2024-02-05T06:36:54.950782+0000","last_active":"2024-02-05T06:36:59.996097+0000","last_peered":"2024-02-05T06:36:59.996097+0000","last_clean":"2024-02-05T06:36:59.996097+0000","last_became_active":"2024-02-05T06:36:54.950298+0000","last_became_peered":"2024-02-05T06:36:54.950298+0000","last_unstale":"2024-02-05T06:36:59.996097+0000","last_undegraded":"2024-02-05T06:36:59.996097+0000","last_fullsized":"2024-02-05T06:36:59.996097+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:07.017572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:35:07.017572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"92'16","reported_seq":159,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:00.001215+0000","last_change":"2024-02-05T06:36:20.685208+0000","last_active":"2024-02-05T06:37:00.001215+0000","last_peered":"2024-02-05T06:37:00.001215+0000","last_clean":"2024-02-05T06:37:00.001215+0000","last_became_active":"2024-02-05T06:33:31.573088+0000","last_became_peered":"2024-02-05T06:33:31.573088+0000","last_unstale":"2024-02-05T06:37:00.001215+0000","last_undegraded":"2024-02-05T06:37:00.001215+0000","last_fullsized":"2024-02-05T06:37:00.001215+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'16","last_scrub_stamp":"2024-02-05T06:36:20.685104+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:34:45.270135+0000","last_clean_scrub_stamp":"2024-02-05T06:36:20.685104+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:21.458668+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":132,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:00.002984+0000","last_change":"2024-02-05T06:36:54.969597+0000","last_active":"2024-02-05T06:37:00.002984+0000","last_peered":"2024-02-05T06:37:00.002984+0000","last_clean":"2024-02-05T06:37:00.002984+0000","last_became_active":"2024-02-05T06:36:54.969234+0000","last_became_peered":"2024-02-05T06:36:54.969234+0000","last_unstale":"2024-02-05T06:37:00.002984+0000","last_undegraded":"2024-02-05T06:37:00.002984+0000","last_fullsized":"2024-02-05T06:37:00.002984+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:55.062013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:55.062013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":143,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:00.000594+0000","last_change":"2024-02-05T06:36:54.969100+0000","last_active":"2024-02-05T06:37:00.000594+0000","last_peered":"2024-02-05T06:37:00.000594+0000","last_clean":"2024-02-05T06:37:00.000594+0000","last_became_active":"2024-02-05T06:36:54.968712+0000","last_became_peered":"2024-02-05T06:36:54.968712+0000","last_unstale":"2024-02-05T06:37:00.000594+0000","last_undegraded":"2024-02-05T06:37:00.000594+0000","last_fullsized":"2024-02-05T06:37:00.000594+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:13.766558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:13.766558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:18.678834+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":134,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-02-05T06:37:06.321204+0000","last_change":"2024-02-05T06:36:17.221190+0000","last_active":"2024-02-05T06:37:06.321204+0000","last_peered":"2024-02-05T06:37:06.321204+0000","last_clean":"2024-02-05T06:37:06.321204+0000","last_became_active":"2024-02-05T06:33:29.549204+0000","last_became_peered":"2024-02-05T06:33:29.549204+0000","last_unstale":"2024-02-05T06:37:06.321204+0000","last_undegraded":"2024-02-05T06:37:06.321204+0000","last_fullsized":"2024-02-05T06:37:06.321204+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:17.220950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:17.220950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:24.661470+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"73'4","reported_seq":134,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.993773+0000","last_change":"2024-02-05T06:36:06.303548+0000","last_active":"2024-02-05T06:36:59.993773+0000","last_peered":"2024-02-05T06:36:59.993773+0000","last_clean":"2024-02-05T06:36:59.993773+0000","last_became_active":"2024-02-05T06:33:31.573410+0000","last_became_peered":"2024-02-05T06:33:31.573410+0000","last_unstale":"2024-02-05T06:36:59.993773+0000","last_undegraded":"2024-02-05T06:36:59.993773+0000","last_fullsized":"2024-02-05T06:36:59.993773+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'4","last_scrub_stamp":"2024-02-05T06:36:06.303366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:06.303366+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:09.135553+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":120,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.998455+0000","last_change":"2024-02-05T06:34:54.902085+0000","last_active":"2024-02-05T06:36:59.998455+0000","last_peered":"2024-02-05T06:36:59.998455+0000","last_clean":"2024-02-05T06:36:59.998455+0000","last_became_active":"2024-02-05T06:33:35.667602+0000","last_became_peered":"2024-02-05T06:33:35.667602+0000","last_unstale":"2024-02-05T06:36:59.998455+0000","last_undegraded":"2024-02-05T06:36:59.998455+0000","last_fullsized":"2024-02-05T06:36:59.998455+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:54.902000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:54.902000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":144,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:00.003194+0000","last_change":"2024-02-05T06:36:54.969773+0000","last_active":"2024-02-05T06:37:00.003194+0000","last_peered":"2024-02-05T06:37:00.003194+0000","last_clean":"2024-02-05T06:37:00.003194+0000","last_became_active":"2024-02-05T06:36:54.969443+0000","last_became_peered":"2024-02-05T06:36:54.969443+0000","last_unstale":"2024-02-05T06:37:00.003194+0000","last_undegraded":"2024-02-05T06:37:00.003194+0000","last_fullsized":"2024-02-05T06:37:00.003194+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:06.846705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:06.846705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:13.549530+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":130,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.997247+0000","last_change":"2024-02-05T06:35:16.915069+0000","last_active":"2024-02-05T06:36:59.997247+0000","last_peered":"2024-02-05T06:36:59.997247+0000","last_clean":"2024-02-05T06:36:59.997247+0000","last_became_active":"2024-02-05T06:33:29.551278+0000","last_became_peered":"2024-02-05T06:33:29.551278+0000","last_unstale":"2024-02-05T06:36:59.997247+0000","last_undegraded":"2024-02-05T06:36:59.997247+0000","last_fullsized":"2024-02-05T06:36:59.997247+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:16.914970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:35:16.914970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"93'16","reported_seq":152,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.994863+0000","last_change":"2024-02-05T06:35:22.938794+0000","last_active":"2024-02-05T06:36:59.994863+0000","last_peered":"2024-02-05T06:36:59.994863+0000","last_clean":"2024-02-05T06:36:59.994863+0000","last_became_active":"2024-02-05T06:33:31.664743+0000","last_became_peered":"2024-02-05T06:33:31.664743+0000","last_unstale":"2024-02-05T06:36:59.994863+0000","last_undegraded":"2024-02-05T06:36:59.994863+0000","last_fullsized":"2024-02-05T06:36:59.994863+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'16","last_scrub_stamp":"2024-02-05T06:35:22.938675+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-02-05T06:35:22.938675+0000","last_clean_scrub_stamp":"2024-02-05T06:35:22.938675+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"73'1","reported_seq":123,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-02-05T06:37:06.319691+0000","last_change":"2024-02-05T06:35:09.676357+0000","last_active":"2024-02-05T06:37:06.319691+0000","last_peered":"2024-02-05T06:37:06.319691+0000","last_clean":"2024-02-05T06:37:06.319691+0000","last_became_active":"2024-02-05T06:33:35.595031+0000","last_became_peered":"2024-02-05T06:33:35.595031+0000","last_unstale":"2024-02-05T06:37:06.319691+0000","last_undegraded":"2024-02-05T06:37:06.319691+0000","last_fullsized":"2024-02-05T06:37:06.319691+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-02-05T06:35:09.676267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:09.676267+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":124,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.997924+0000","last_change":"2024-02-05T06:35:03.027397+0000","last_active":"2024-02-05T06:36:59.997924+0000","last_peered":"2024-02-05T06:36:59.997924+0000","last_clean":"2024-02-05T06:36:59.997924+0000","last_became_active":"2024-02-05T06:33:33.593901+0000","last_became_peered":"2024-02-05T06:33:33.593901+0000","last_unstale":"2024-02-05T06:36:59.997924+0000","last_undegraded":"2024-02-05T06:36:59.997924+0000","last_fullsized":"2024-02-05T06:36:59.997924+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:03.027270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:35:03.027270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":144,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:00.003570+0000","last_change":"2024-02-05T06:36:54.971137+0000","last_active":"2024-02-05T06:37:00.003570+0000","last_peered":"2024-02-05T06:37:00.003570+0000","last_clean":"2024-02-05T06:37:00.003570+0000","last_became_active":"2024-02-05T06:36:54.970247+0000","last_became_peered":"2024-02-05T06:36:54.970247+0000","last_unstale":"2024-02-05T06:37:00.003570+0000","last_undegraded":"2024-02-05T06:37:00.003570+0000","last_fullsized":"2024-02-05T06:37:00.003570+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:06.946130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:35:06.946130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":151,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.997938+0000","last_change":"2024-02-05T06:36:30.047255+0000","last_active":"2024-02-05T06:36:59.997938+0000","last_peered":"2024-02-05T06:36:59.997938+0000","last_clean":"2024-02-05T06:36:59.997938+0000","last_became_active":"2024-02-05T06:33:31.568529+0000","last_became_peered":"2024-02-05T06:33:31.568529+0000","last_unstale":"2024-02-05T06:36:59.997938+0000","last_undegraded":"2024-02-05T06:36:59.997938+0000","last_fullsized":"2024-02-05T06:36:59.997938+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:36:30.047048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:30.047048+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:59.807573+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":128,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.994512+0000","last_change":"2024-02-05T06:36:30.451097+0000","last_active":"2024-02-05T06:36:59.994512+0000","last_peered":"2024-02-05T06:36:59.994512+0000","last_clean":"2024-02-05T06:36:59.994512+0000","last_became_active":"2024-02-05T06:33:35.597399+0000","last_became_peered":"2024-02-05T06:33:35.597399+0000","last_unstale":"2024-02-05T06:36:59.994512+0000","last_undegraded":"2024-02-05T06:36:59.994512+0000","last_fullsized":"2024-02-05T06:36:59.994512+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:30.450988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:30.450988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:34.833906+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":131,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.995172+0000","last_change":"2024-02-05T06:36:06.224697+0000","last_active":"2024-02-05T06:36:59.995172+0000","last_peered":"2024-02-05T06:36:59.995172+0000","last_clean":"2024-02-05T06:36:59.995172+0000","last_became_active":"2024-02-05T06:33:33.588461+0000","last_became_peered":"2024-02-05T06:33:33.588461+0000","last_unstale":"2024-02-05T06:36:59.995172+0000","last_undegraded":"2024-02-05T06:36:59.995172+0000","last_fullsized":"2024-02-05T06:36:59.995172+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:06.224205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:06.224205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:32.495349+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.988638+0000","last_change":"2024-02-05T06:36:54.964023+0000","last_active":"2024-02-05T06:36:59.988638+0000","last_peered":"2024-02-05T06:36:59.988638+0000","last_clean":"2024-02-05T06:36:59.988638+0000","last_became_active":"2024-02-05T06:36:54.963719+0000","last_became_peered":"2024-02-05T06:36:54.963719+0000","last_unstale":"2024-02-05T06:36:59.988638+0000","last_undegraded":"2024-02-05T06:36:59.988638+0000","last_fullsized":"2024-02-05T06:36:59.988638+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:27.903162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:27.903162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:35.419758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"92'16","reported_seq":29,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.989114+0000","last_change":"2024-02-05T06:36:54.999613+0000","last_active":"2024-02-05T06:36:59.989114+0000","last_peered":"2024-02-05T06:36:59.989114+0000","last_clean":"2024-02-05T06:36:59.989114+0000","last_became_active":"2024-02-05T06:36:54.961610+0000","last_became_peered":"2024-02-05T06:36:54.961610+0000","last_unstale":"2024-02-05T06:36:59.989114+0000","last_undegraded":"2024-02-05T06:36:59.989114+0000","last_fullsized":"2024-02-05T06:36:59.989114+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'16","last_scrub_stamp":"2024-02-05T06:36:36.728552+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:35:02.589229+0000","last_clean_scrub_stamp":"2024-02-05T06:36:36.728552+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:58.641289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":122,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-02-05T06:37:06.320463+0000","last_change":"2024-02-05T06:36:06.138504+0000","last_active":"2024-02-05T06:37:06.320463+0000","last_peered":"2024-02-05T06:37:06.320463+0000","last_clean":"2024-02-05T06:37:06.320463+0000","last_became_active":"2024-02-05T06:33:35.599808+0000","last_became_peered":"2024-02-05T06:33:35.599808+0000","last_unstale":"2024-02-05T06:37:06.320463+0000","last_undegraded":"2024-02-05T06:37:06.320463+0000","last_fullsized":"2024-02-05T06:37:06.320463+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:06.137845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:06.137845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:31.437152+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":182,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.996873+0000","last_change":"2024-02-05T06:36:27.952699+0000","last_active":"2024-02-05T06:36:59.996873+0000","last_peered":"2024-02-05T06:36:59.996873+0000","last_clean":"2024-02-05T06:36:59.996873+0000","last_became_active":"2024-02-05T06:33:33.585642+0000","last_became_peered":"2024-02-05T06:33:33.585642+0000","last_unstale":"2024-02-05T06:36:59.996873+0000","last_undegraded":"2024-02-05T06:36:59.996873+0000","last_fullsized":"2024-02-05T06:36:59.996873+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:36:27.952573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:27.952573+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:28.467691+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":144,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:00.003245+0000","last_change":"2024-02-05T06:36:54.969609+0000","last_active":"2024-02-05T06:37:00.003245+0000","last_peered":"2024-02-05T06:37:00.003245+0000","last_clean":"2024-02-05T06:37:00.003245+0000","last_became_active":"2024-02-05T06:36:54.969248+0000","last_became_peered":"2024-02-05T06:36:54.969248+0000","last_unstale":"2024-02-05T06:37:00.003245+0000","last_undegraded":"2024-02-05T06:37:00.003245+0000","last_fullsized":"2024-02-05T06:37:00.003245+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:05.967303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:35:05.967303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'10","reported_seq":161,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.995755+0000","last_change":"2024-02-05T06:36:12.205255+0000","last_active":"2024-02-05T06:36:59.995755+0000","last_peered":"2024-02-05T06:36:59.995755+0000","last_clean":"2024-02-05T06:36:59.995755+0000","last_became_active":"2024-02-05T06:36:12.171679+0000","last_became_peered":"2024-02-05T06:36:12.171679+0000","last_unstale":"2024-02-05T06:36:59.995755+0000","last_undegraded":"2024-02-05T06:36:59.995755+0000","last_fullsized":"2024-02-05T06:36:59.995755+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-02-05T06:34:41.873876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:41.873876+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":17,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.988624+0000","last_change":"2024-02-05T06:36:54.963383+0000","last_active":"2024-02-05T06:36:59.988624+0000","last_peered":"2024-02-05T06:36:59.988624+0000","last_clean":"2024-02-05T06:36:59.988624+0000","last_became_active":"2024-02-05T06:36:54.961617+0000","last_became_peered":"2024-02-05T06:36:54.961617+0000","last_unstale":"2024-02-05T06:36:59.988624+0000","last_undegraded":"2024-02-05T06:36:59.988624+0000","last_fullsized":"2024-02-05T06:36:59.988624+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:12.594735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:12.594735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":184,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.996139+0000","last_change":"2024-02-05T06:36:26.411694+0000","last_active":"2024-02-05T06:36:59.996139+0000","last_peered":"2024-02-05T06:36:59.996139+0000","last_clean":"2024-02-05T06:36:59.996139+0000","last_became_active":"2024-02-05T06:33:33.586043+0000","last_became_peered":"2024-02-05T06:33:33.586043+0000","last_unstale":"2024-02-05T06:36:59.996139+0000","last_undegraded":"2024-02-05T06:36:59.996139+0000","last_fullsized":"2024-02-05T06:36:59.996139+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:36:26.411576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:26.411576+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:35.928030+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":142,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-02-05T06:37:06.320402+0000","last_change":"2024-02-05T06:36:31.169821+0000","last_active":"2024-02-05T06:37:06.320402+0000","last_peered":"2024-02-05T06:37:06.320402+0000","last_clean":"2024-02-05T06:37:06.320402+0000","last_became_active":"2024-02-05T06:33:29.552170+0000","last_became_peered":"2024-02-05T06:33:29.552170+0000","last_unstale":"2024-02-05T06:37:06.320402+0000","last_undegraded":"2024-02-05T06:37:06.320402+0000","last_fullsized":"2024-02-05T06:37:06.320402+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:31.169691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:36:31.169691+0000","last_clean_scrub_stamp":"2024-02-05T06:36:31.169691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:53.607772+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"73'9","reported_seq":166,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.997040+0000","last_change":"2024-02-05T06:36:12.206355+0000","last_active":"2024-02-05T06:36:59.997040+0000","last_peered":"2024-02-05T06:36:59.997040+0000","last_clean":"2024-02-05T06:36:59.997040+0000","last_became_active":"2024-02-05T06:36:12.173982+0000","last_became_peered":"2024-02-05T06:36:12.173982+0000","last_unstale":"2024-02-05T06:36:59.997040+0000","last_undegraded":"2024-02-05T06:36:59.997040+0000","last_fullsized":"2024-02-05T06:36:59.997040+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:36:09.760345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:09.760345+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:38.330993+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":138,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:00.000305+0000","last_change":"2024-02-05T06:36:54.958405+0000","last_active":"2024-02-05T06:37:00.000305+0000","last_peered":"2024-02-05T06:37:00.000305+0000","last_clean":"2024-02-05T06:37:00.000305+0000","last_became_active":"2024-02-05T06:36:54.958065+0000","last_became_peered":"2024-02-05T06:36:54.958065+0000","last_unstale":"2024-02-05T06:37:00.000305+0000","last_undegraded":"2024-02-05T06:37:00.000305+0000","last_fullsized":"2024-02-05T06:37:00.000305+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:08.696108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:08.696108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:23.357937+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":132,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.995491+0000","last_change":"2024-02-05T06:36:25.378909+0000","last_active":"2024-02-05T06:36:59.995491+0000","last_peered":"2024-02-05T06:36:59.995491+0000","last_clean":"2024-02-05T06:36:59.995491+0000","last_became_active":"2024-02-05T06:33:33.593025+0000","last_became_peered":"2024-02-05T06:33:33.593025+0000","last_unstale":"2024-02-05T06:36:59.995491+0000","last_undegraded":"2024-02-05T06:36:59.995491+0000","last_fullsized":"2024-02-05T06:36:59.995491+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:25.378404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:25.378404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:49.015657+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":152,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:00.003888+0000","last_change":"2024-02-05T06:36:54.972800+0000","last_active":"2024-02-05T06:37:00.003888+0000","last_peered":"2024-02-05T06:37:00.003888+0000","last_clean":"2024-02-05T06:37:00.003888+0000","last_became_active":"2024-02-05T06:36:54.972105+0000","last_became_peered":"2024-02-05T06:36:54.972105+0000","last_unstale":"2024-02-05T06:37:00.003888+0000","last_undegraded":"2024-02-05T06:37:00.003888+0000","last_fullsized":"2024-02-05T06:37:00.003888+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:18.895623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:18.895623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:39.492351+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"73'9","reported_seq":154,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-02-05T06:37:06.320964+0000","last_change":"2024-02-05T06:36:23.148667+0000","last_active":"2024-02-05T06:37:06.320964+0000","last_peered":"2024-02-05T06:37:06.320964+0000","last_clean":"2024-02-05T06:37:06.320964+0000","last_became_active":"2024-02-05T06:33:31.665766+0000","last_became_peered":"2024-02-05T06:33:31.665766+0000","last_unstale":"2024-02-05T06:37:06.320964+0000","last_undegraded":"2024-02-05T06:37:06.320964+0000","last_fullsized":"2024-02-05T06:37:06.320964+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:36:23.148541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:23.148541+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:40.199216+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":140,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:00.003469+0000","last_change":"2024-02-05T06:36:54.970154+0000","last_active":"2024-02-05T06:37:00.003469+0000","last_peered":"2024-02-05T06:37:00.003469+0000","last_clean":"2024-02-05T06:37:00.003469+0000","last_became_active":"2024-02-05T06:36:54.969661+0000","last_became_peered":"2024-02-05T06:36:54.969661+0000","last_unstale":"2024-02-05T06:37:00.003469+0000","last_undegraded":"2024-02-05T06:37:00.003469+0000","last_fullsized":"2024-02-05T06:37:00.003469+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:29.845506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:29.845506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:56.231851+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":130,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.997234+0000","last_change":"2024-02-05T06:36:20.832672+0000","last_active":"2024-02-05T06:36:59.997234+0000","last_peered":"2024-02-05T06:36:59.997234+0000","last_clean":"2024-02-05T06:36:59.997234+0000","last_became_active":"2024-02-05T06:33:33.586768+0000","last_became_peered":"2024-02-05T06:33:33.586768+0000","last_unstale":"2024-02-05T06:36:59.997234+0000","last_undegraded":"2024-02-05T06:36:59.997234+0000","last_fullsized":"2024-02-05T06:36:59.997234+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:20.832579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:45.096340+0000","last_clean_scrub_stamp":"2024-02-05T06:36:20.832579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:39.454034+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":139,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:00.000350+0000","last_change":"2024-02-05T06:36:07.655271+0000","last_active":"2024-02-05T06:37:00.000350+0000","last_peered":"2024-02-05T06:37:00.000350+0000","last_clean":"2024-02-05T06:37:00.000350+0000","last_became_active":"2024-02-05T06:33:29.554427+0000","last_became_peered":"2024-02-05T06:33:29.554427+0000","last_unstale":"2024-02-05T06:37:00.000350+0000","last_undegraded":"2024-02-05T06:37:00.000350+0000","last_fullsized":"2024-02-05T06:37:00.000350+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:07.655171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:07.655171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:15.608469+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":137,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.995718+0000","last_change":"2024-02-05T06:34:55.912380+0000","last_active":"2024-02-05T06:36:59.995718+0000","last_peered":"2024-02-05T06:36:59.995718+0000","last_clean":"2024-02-05T06:36:59.995718+0000","last_became_active":"2024-02-05T06:33:31.665847+0000","last_became_peered":"2024-02-05T06:33:31.665847+0000","last_unstale":"2024-02-05T06:36:59.995718+0000","last_undegraded":"2024-02-05T06:36:59.995718+0000","last_fullsized":"2024-02-05T06:36:59.995718+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'6","last_scrub_stamp":"2024-02-05T06:34:55.912282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:55.912282+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"122'7093","reported_seq":7155,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:06.211014+0000","last_change":"2024-02-05T06:35:27.937770+0000","last_active":"2024-02-05T06:37:06.211014+0000","last_peered":"2024-02-05T06:37:06.211014+0000","last_clean":"2024-02-05T06:37:06.211014+0000","last_became_active":"2024-02-05T06:35:27.937512+0000","last_became_peered":"2024-02-05T06:35:27.937512+0000","last_unstale":"2024-02-05T06:37:06.211014+0000","last_undegraded":"2024-02-05T06:37:06.211014+0000","last_fullsized":"2024-02-05T06:37:06.211014+0000","mapping_epoch":96,"log_start":"103'2200","ondisk_log_start":"103'2200","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":4893,"log_dups_size":0,"ondisk_log_size":4893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":464846848,"num_objects":7093,"num_object_clones":0,"num_object_copies":21279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7093,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7093,"num_write_kb":453952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":126,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.997791+0000","last_change":"2024-02-05T06:36:33.055072+0000","last_active":"2024-02-05T06:36:59.997791+0000","last_peered":"2024-02-05T06:36:59.997791+0000","last_clean":"2024-02-05T06:36:59.997791+0000","last_became_active":"2024-02-05T06:33:35.665581+0000","last_became_peered":"2024-02-05T06:33:35.665581+0000","last_unstale":"2024-02-05T06:36:59.997791+0000","last_undegraded":"2024-02-05T06:36:59.997791+0000","last_fullsized":"2024-02-05T06:36:59.997791+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:33.054967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:33.054967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:34.914953+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":168,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:00.003696+0000","last_change":"2024-02-05T06:36:54.989025+0000","last_active":"2024-02-05T06:37:00.003696+0000","last_peered":"2024-02-05T06:37:00.003696+0000","last_clean":"2024-02-05T06:37:00.003696+0000","last_became_active":"2024-02-05T06:36:54.970234+0000","last_became_peered":"2024-02-05T06:36:54.970234+0000","last_unstale":"2024-02-05T06:37:00.003696+0000","last_undegraded":"2024-02-05T06:37:00.003696+0000","last_fullsized":"2024-02-05T06:37:00.003696+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-02-05T06:35:05.001912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:35:05.001912+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"73'15","reported_seq":151,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.996265+0000","last_change":"2024-02-05T06:35:15.959953+0000","last_active":"2024-02-05T06:36:59.996265+0000","last_peered":"2024-02-05T06:36:59.996265+0000","last_clean":"2024-02-05T06:36:59.996265+0000","last_became_active":"2024-02-05T06:33:31.666056+0000","last_became_peered":"2024-02-05T06:33:31.666056+0000","last_unstale":"2024-02-05T06:36:59.996265+0000","last_undegraded":"2024-02-05T06:36:59.996265+0000","last_fullsized":"2024-02-05T06:36:59.996265+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-02-05T06:35:15.959820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:35:15.959820+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"119'10","reported_seq":206,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:00.004068+0000","last_change":"2024-02-05T06:36:54.985467+0000","last_active":"2024-02-05T06:37:00.004068+0000","last_peered":"2024-02-05T06:37:00.004068+0000","last_clean":"2024-02-05T06:37:00.004068+0000","last_became_active":"2024-02-05T06:36:54.970348+0000","last_became_peered":"2024-02-05T06:36:54.970348+0000","last_unstale":"2024-02-05T06:37:00.004068+0000","last_undegraded":"2024-02-05T06:37:00.004068+0000","last_fullsized":"2024-02-05T06:37:00.004068+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'9","last_scrub_stamp":"2024-02-05T06:36:32.863645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:32.863645+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:35.262516+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"122'7108","reported_seq":7170,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:06.309094+0000","last_change":"2024-02-05T06:35:27.933615+0000","last_active":"2024-02-05T06:37:06.309094+0000","last_peered":"2024-02-05T06:37:06.309094+0000","last_clean":"2024-02-05T06:37:06.309094+0000","last_became_active":"2024-02-05T06:35:27.933340+0000","last_became_peered":"2024-02-05T06:35:27.933340+0000","last_unstale":"2024-02-05T06:37:06.309094+0000","last_undegraded":"2024-02-05T06:37:06.309094+0000","last_fullsized":"2024-02-05T06:37:06.309094+0000","mapping_epoch":96,"log_start":"104'2400","ondisk_log_start":"104'2400","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":4708,"log_dups_size":0,"ondisk_log_size":4708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:07.525440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":465829888,"num_objects":7108,"num_object_clones":0,"num_object_copies":21324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7108,"num_write_kb":454912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":120,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.998101+0000","last_change":"2024-02-05T06:35:12.099300+0000","last_active":"2024-02-05T06:36:59.998101+0000","last_peered":"2024-02-05T06:36:59.998101+0000","last_clean":"2024-02-05T06:36:59.998101+0000","last_became_active":"2024-02-05T06:33:35.601030+0000","last_became_peered":"2024-02-05T06:33:35.601030+0000","last_unstale":"2024-02-05T06:36:59.998101+0000","last_undegraded":"2024-02-05T06:36:59.998101+0000","last_fullsized":"2024-02-05T06:36:59.998101+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:12.099209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:12.099209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":152,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:00.003522+0000","last_change":"2024-02-05T06:36:54.970978+0000","last_active":"2024-02-05T06:37:00.003522+0000","last_peered":"2024-02-05T06:37:00.003522+0000","last_clean":"2024-02-05T06:37:00.003522+0000","last_became_active":"2024-02-05T06:36:54.970400+0000","last_became_peered":"2024-02-05T06:36:54.970400+0000","last_unstale":"2024-02-05T06:37:00.003522+0000","last_undegraded":"2024-02-05T06:37:00.003522+0000","last_fullsized":"2024-02-05T06:37:00.003522+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:33.825033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:33.825033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:53.454150+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'12","reported_seq":29,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.988088+0000","last_change":"2024-02-05T06:36:54.986580+0000","last_active":"2024-02-05T06:36:59.988088+0000","last_peered":"2024-02-05T06:36:59.988088+0000","last_clean":"2024-02-05T06:36:59.988088+0000","last_became_active":"2024-02-05T06:36:54.954071+0000","last_became_peered":"2024-02-05T06:36:54.954071+0000","last_unstale":"2024-02-05T06:36:59.988088+0000","last_undegraded":"2024-02-05T06:36:59.988088+0000","last_fullsized":"2024-02-05T06:36:59.988088+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:35:00.650542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:35:00.650542+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":122,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:00.011027+0000","last_change":"2024-02-05T06:34:40.272941+0000","last_active":"2024-02-05T06:37:00.011027+0000","last_peered":"2024-02-05T06:37:00.011027+0000","last_clean":"2024-02-05T06:37:00.011027+0000","last_became_active":"2024-02-05T06:33:33.592546+0000","last_became_peered":"2024-02-05T06:33:33.592546+0000","last_unstale":"2024-02-05T06:37:00.011027+0000","last_undegraded":"2024-02-05T06:37:00.011027+0000","last_fullsized":"2024-02-05T06:37:00.011027+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:40.272852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:40.272852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"123'7128","reported_seq":10263,"reported_epoch":123,"state":"active+remapped+backfilling","last_fresh":"2024-02-05T06:37:06.376563+0000","last_change":"2024-02-05T06:36:56.039410+0000","last_active":"2024-02-05T06:37:06.376563+0000","last_peered":"2024-02-05T06:37:06.376563+0000","last_clean":"2024-02-05T06:36:54.251560+0000","last_became_active":"2024-02-05T06:36:56.030709+0000","last_became_peered":"2024-02-05T06:36:56.030709+0000","last_unstale":"2024-02-05T06:37:06.376563+0000","last_undegraded":"2024-02-05T06:37:06.376563+0000","last_fullsized":"2024-02-05T06:37:06.376563+0000","mapping_epoch":119,"log_start":"102'2100","ondisk_log_start":"102'2100","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":5028,"log_dups_size":0,"ondisk_log_size":5028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":467140608,"num_objects":7128,"num_object_clones":0,"num_object_copies":21384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5535,"num_objects_unfound":0,"num_objects_dirty":7128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7128,"num_write_kb":456192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1533,"num_bytes_recovered":100466688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,7],"avail_no_missing":["0","1","7"],"object_location_counts":[{"shards":"0,1,7","objects":7128}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":120,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:00.002930+0000","last_change":"2024-02-05T06:35:18.964868+0000","last_active":"2024-02-05T06:37:00.002930+0000","last_peered":"2024-02-05T06:37:00.002930+0000","last_clean":"2024-02-05T06:37:00.002930+0000","last_became_active":"2024-02-05T06:33:35.599527+0000","last_became_peered":"2024-02-05T06:33:35.599527+0000","last_unstale":"2024-02-05T06:37:00.002930+0000","last_undegraded":"2024-02-05T06:37:00.002930+0000","last_fullsized":"2024-02-05T06:37:00.002930+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:18.964748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:18.964748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":130,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:00.000228+0000","last_change":"2024-02-05T06:36:16.649604+0000","last_active":"2024-02-05T06:37:00.000228+0000","last_peered":"2024-02-05T06:37:00.000228+0000","last_clean":"2024-02-05T06:37:00.000228+0000","last_became_active":"2024-02-05T06:33:33.589203+0000","last_became_peered":"2024-02-05T06:33:33.589203+0000","last_unstale":"2024-02-05T06:37:00.000228+0000","last_undegraded":"2024-02-05T06:37:00.000228+0000","last_fullsized":"2024-02-05T06:37:00.000228+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:16.649483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:16.649483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:43.871278+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":132,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:00.003818+0000","last_change":"2024-02-05T06:34:59.028827+0000","last_active":"2024-02-05T06:37:00.003818+0000","last_peered":"2024-02-05T06:37:00.003818+0000","last_clean":"2024-02-05T06:37:00.003818+0000","last_became_active":"2024-02-05T06:33:29.556541+0000","last_became_peered":"2024-02-05T06:33:29.556541+0000","last_unstale":"2024-02-05T06:37:00.003818+0000","last_undegraded":"2024-02-05T06:37:00.003818+0000","last_fullsized":"2024-02-05T06:37:00.003818+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:59.028748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:59.028748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"73'19","reported_seq":166,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:00.000575+0000","last_change":"2024-02-05T06:36:14.733809+0000","last_active":"2024-02-05T06:37:00.000575+0000","last_peered":"2024-02-05T06:37:00.000575+0000","last_clean":"2024-02-05T06:37:00.000575+0000","last_became_active":"2024-02-05T06:33:31.664699+0000","last_became_peered":"2024-02-05T06:33:31.664699+0000","last_unstale":"2024-02-05T06:37:00.000575+0000","last_undegraded":"2024-02-05T06:37:00.000575+0000","last_fullsized":"2024-02-05T06:37:00.000575+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-02-05T06:36:14.733710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:14.733710+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:28.251949+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"122'7219","reported_seq":7287,"reported_epoch":122,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T06:37:06.165844+0000","last_change":"2024-02-05T06:36:56.062331+0000","last_active":"2024-02-05T06:37:06.165844+0000","last_peered":"2024-02-05T06:37:06.165844+0000","last_clean":"2024-02-05T06:36:54.240117+0000","last_became_active":"2024-02-05T06:36:56.042585+0000","last_became_peered":"2024-02-05T06:36:56.042585+0000","last_unstale":"2024-02-05T06:37:06.165844+0000","last_undegraded":"2024-02-05T06:37:06.165844+0000","last_fullsized":"2024-02-05T06:37:06.165844+0000","mapping_epoch":119,"log_start":"101'900","ondisk_log_start":"101'900","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":6319,"log_dups_size":0,"ondisk_log_size":6319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":473104384,"num_objects":7219,"num_object_clones":0,"num_object_copies":21657,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7219,"num_objects_unfound":0,"num_objects_dirty":7219,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7219,"num_write_kb":462016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,3],"avail_no_missing":["2","3","7"],"object_location_counts":[{"shards":"2,3,7","objects":7219}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":122,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-02-05T06:37:06.320401+0000","last_change":"2024-02-05T06:36:22.102208+0000","last_active":"2024-02-05T06:37:06.320401+0000","last_peered":"2024-02-05T06:37:06.320401+0000","last_clean":"2024-02-05T06:37:06.320401+0000","last_became_active":"2024-02-05T06:33:35.597863+0000","last_became_peered":"2024-02-05T06:33:35.597863+0000","last_unstale":"2024-02-05T06:37:06.320401+0000","last_undegraded":"2024-02-05T06:37:06.320401+0000","last_fullsized":"2024-02-05T06:37:06.320401+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:22.102133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:22.102133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:47.960776+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":142,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:00.003978+0000","last_change":"2024-02-05T06:36:54.967406+0000","last_active":"2024-02-05T06:37:00.003978+0000","last_peered":"2024-02-05T06:37:00.003978+0000","last_clean":"2024-02-05T06:37:00.003978+0000","last_became_active":"2024-02-05T06:36:54.966016+0000","last_became_peered":"2024-02-05T06:36:54.966016+0000","last_unstale":"2024-02-05T06:37:00.003978+0000","last_undegraded":"2024-02-05T06:37:00.003978+0000","last_fullsized":"2024-02-05T06:37:00.003978+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:31.737492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:31.737492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:36.708214+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":130,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.996714+0000","last_change":"2024-02-05T06:35:08.945210+0000","last_active":"2024-02-05T06:36:59.996714+0000","last_peered":"2024-02-05T06:36:59.996714+0000","last_clean":"2024-02-05T06:36:59.996714+0000","last_became_active":"2024-02-05T06:33:29.555950+0000","last_became_peered":"2024-02-05T06:33:29.555950+0000","last_unstale":"2024-02-05T06:36:59.996714+0000","last_undegraded":"2024-02-05T06:36:59.996714+0000","last_fullsized":"2024-02-05T06:36:59.996714+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:08.945107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:35:08.945107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"73'9","reported_seq":202,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-02-05T06:37:06.320654+0000","last_change":"2024-02-05T06:36:54.995496+0000","last_active":"2024-02-05T06:37:06.320654+0000","last_peered":"2024-02-05T06:37:06.320654+0000","last_clean":"2024-02-05T06:37:06.320654+0000","last_became_active":"2024-02-05T06:36:54.973958+0000","last_became_peered":"2024-02-05T06:36:54.973958+0000","last_unstale":"2024-02-05T06:37:06.320654+0000","last_undegraded":"2024-02-05T06:37:06.320654+0000","last_fullsized":"2024-02-05T06:37:06.320654+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:36:26.172612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:26.172612+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:52.634290+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"122'7143","reported_seq":7205,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:06.068114+0000","last_change":"2024-02-05T06:35:27.934389+0000","last_active":"2024-02-05T06:37:06.068114+0000","last_peered":"2024-02-05T06:37:06.068114+0000","last_clean":"2024-02-05T06:37:06.068114+0000","last_became_active":"2024-02-05T06:35:27.934093+0000","last_became_peered":"2024-02-05T06:35:27.934093+0000","last_unstale":"2024-02-05T06:37:06.068114+0000","last_undegraded":"2024-02-05T06:37:06.068114+0000","last_fullsized":"2024-02-05T06:37:06.068114+0000","mapping_epoch":96,"log_start":"102'1600","ondisk_log_start":"102'1600","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":5543,"log_dups_size":0,"ondisk_log_size":5543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:08.646646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":468123648,"num_objects":7143,"num_object_clones":0,"num_object_copies":21429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7143,"num_write_kb":457152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":132,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.997553+0000","last_change":"2024-02-05T06:36:54.955665+0000","last_active":"2024-02-05T06:36:59.997553+0000","last_peered":"2024-02-05T06:36:59.997553+0000","last_clean":"2024-02-05T06:36:59.997553+0000","last_became_active":"2024-02-05T06:36:54.955311+0000","last_became_peered":"2024-02-05T06:36:54.955311+0000","last_unstale":"2024-02-05T06:36:59.997553+0000","last_undegraded":"2024-02-05T06:36:59.997553+0000","last_fullsized":"2024-02-05T06:36:59.997553+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:14.082774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:14.082774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:01.557855+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":149,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.997690+0000","last_change":"2024-02-05T06:36:14.780874+0000","last_active":"2024-02-05T06:36:59.997690+0000","last_peered":"2024-02-05T06:36:59.997690+0000","last_clean":"2024-02-05T06:36:59.997690+0000","last_became_active":"2024-02-05T06:33:28.542228+0000","last_became_peered":"2024-02-05T06:33:28.542228+0000","last_unstale":"2024-02-05T06:36:59.997690+0000","last_undegraded":"2024-02-05T06:36:59.997690+0000","last_fullsized":"2024-02-05T06:36:59.997690+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-02-05T06:36:14.780738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:36:14.780738+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:42.032501+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001220348,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.989080+0000","last_change":"2024-02-05T06:36:54.968582+0000","last_active":"2024-02-05T06:36:59.989080+0000","last_peered":"2024-02-05T06:36:59.989080+0000","last_clean":"2024-02-05T06:36:59.989080+0000","last_became_active":"2024-02-05T06:36:54.968205+0000","last_became_peered":"2024-02-05T06:36:54.968205+0000","last_unstale":"2024-02-05T06:36:59.989080+0000","last_undegraded":"2024-02-05T06:36:59.989080+0000","last_fullsized":"2024-02-05T06:36:59.989080+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:15.166979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:15.166979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:28.840085+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":176,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.996557+0000","last_change":"2024-02-05T06:36:35.079298+0000","last_active":"2024-02-05T06:36:59.996557+0000","last_peered":"2024-02-05T06:36:59.996557+0000","last_clean":"2024-02-05T06:36:59.996557+0000","last_became_active":"2024-02-05T06:33:55.884601+0000","last_became_peered":"2024-02-05T06:33:55.884601+0000","last_unstale":"2024-02-05T06:36:59.996557+0000","last_undegraded":"2024-02-05T06:36:59.996557+0000","last_fullsized":"2024-02-05T06:36:59.996557+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-02-05T06:36:35.079170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:35.079170+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:39.455700+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":146,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-02-05T06:37:06.321210+0000","last_change":"2024-02-05T06:36:54.974218+0000","last_active":"2024-02-05T06:37:06.321210+0000","last_peered":"2024-02-05T06:37:06.321210+0000","last_clean":"2024-02-05T06:37:06.321210+0000","last_became_active":"2024-02-05T06:36:54.973387+0000","last_became_peered":"2024-02-05T06:36:54.973387+0000","last_unstale":"2024-02-05T06:37:06.321210+0000","last_undegraded":"2024-02-05T06:37:06.321210+0000","last_fullsized":"2024-02-05T06:37:06.321210+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:36.161606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:36.161606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:39.208518+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"123'7198","reported_seq":10483,"reported_epoch":123,"state":"active+remapped+backfilling","last_fresh":"2024-02-05T06:37:06.376831+0000","last_change":"2024-02-05T06:36:56.063948+0000","last_active":"2024-02-05T06:37:06.376831+0000","last_peered":"2024-02-05T06:37:06.376831+0000","last_clean":"2024-02-05T06:36:54.264686+0000","last_became_active":"2024-02-05T06:36:56.033693+0000","last_became_peered":"2024-02-05T06:36:56.033693+0000","last_unstale":"2024-02-05T06:37:06.376831+0000","last_undegraded":"2024-02-05T06:37:06.376831+0000","last_fullsized":"2024-02-05T06:37:06.376831+0000","mapping_epoch":119,"log_start":"103'2200","ondisk_log_start":"103'2200","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":4998,"log_dups_size":0,"ondisk_log_size":4998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":471728128,"num_objects":7198,"num_object_clones":0,"num_object_copies":21594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5527,"num_objects_unfound":0,"num_objects_dirty":7198,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7198,"num_write_kb":460672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1606,"num_bytes_recovered":105250816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,1,6],"avail_no_missing":["0","1","6"],"object_location_counts":[{"shards":"0,1,6","objects":7198}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":128,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:00.003981+0000","last_change":"2024-02-05T06:36:16.887854+0000","last_active":"2024-02-05T06:37:00.003981+0000","last_peered":"2024-02-05T06:37:00.003981+0000","last_clean":"2024-02-05T06:37:00.003981+0000","last_became_active":"2024-02-05T06:33:35.666324+0000","last_became_peered":"2024-02-05T06:33:35.666324+0000","last_unstale":"2024-02-05T06:37:00.003981+0000","last_undegraded":"2024-02-05T06:37:00.003981+0000","last_fullsized":"2024-02-05T06:37:00.003981+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:16.887732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:16.887732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:25.658445+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":131,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.993757+0000","last_change":"2024-02-05T06:35:21.506625+0000","last_active":"2024-02-05T06:36:59.993757+0000","last_peered":"2024-02-05T06:36:59.993757+0000","last_clean":"2024-02-05T06:36:59.993757+0000","last_became_active":"2024-02-05T06:33:29.551549+0000","last_became_peered":"2024-02-05T06:33:29.551549+0000","last_unstale":"2024-02-05T06:36:59.993757+0000","last_undegraded":"2024-02-05T06:36:59.993757+0000","last_fullsized":"2024-02-05T06:36:59.993757+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:21.506519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:35:21.506519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":147,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.999657+0000","last_change":"2024-02-05T06:36:24.634226+0000","last_active":"2024-02-05T06:36:59.999657+0000","last_peered":"2024-02-05T06:36:59.999657+0000","last_clean":"2024-02-05T06:36:59.999657+0000","last_became_active":"2024-02-05T06:33:28.542977+0000","last_became_peered":"2024-02-05T06:33:28.542977+0000","last_unstale":"2024-02-05T06:36:59.999657+0000","last_undegraded":"2024-02-05T06:36:59.999657+0000","last_fullsized":"2024-02-05T06:36:59.999657+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:24.634022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:36:24.634022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:27.516973+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0.00091640799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'13","reported_seq":162,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.994709+0000","last_change":"2024-02-05T06:36:14.209527+0000","last_active":"2024-02-05T06:36:59.994709+0000","last_peered":"2024-02-05T06:36:59.994709+0000","last_clean":"2024-02-05T06:36:59.994709+0000","last_became_active":"2024-02-05T06:33:31.560082+0000","last_became_peered":"2024-02-05T06:33:31.560082+0000","last_unstale":"2024-02-05T06:36:59.994709+0000","last_undegraded":"2024-02-05T06:36:59.994709+0000","last_fullsized":"2024-02-05T06:36:59.994709+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-02-05T06:36:14.209318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:14.209318+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:29.833690+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":130,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.999870+0000","last_change":"2024-02-05T06:36:09.727864+0000","last_active":"2024-02-05T06:36:59.999870+0000","last_peered":"2024-02-05T06:36:59.999870+0000","last_clean":"2024-02-05T06:36:59.999870+0000","last_became_active":"2024-02-05T06:33:33.587593+0000","last_became_peered":"2024-02-05T06:33:33.587593+0000","last_unstale":"2024-02-05T06:36:59.999870+0000","last_undegraded":"2024-02-05T06:36:59.999870+0000","last_fullsized":"2024-02-05T06:36:59.999870+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:09.726651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:09.726651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:27.634848+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"122'6982","reported_seq":7046,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-02-05T06:37:06.320868+0000","last_change":"2024-02-05T06:35:27.935172+0000","last_active":"2024-02-05T06:37:06.320868+0000","last_peered":"2024-02-05T06:37:06.320868+0000","last_clean":"2024-02-05T06:37:06.320868+0000","last_became_active":"2024-02-05T06:35:27.934809+0000","last_became_peered":"2024-02-05T06:35:27.934809+0000","last_unstale":"2024-02-05T06:37:06.320868+0000","last_undegraded":"2024-02-05T06:37:06.320868+0000","last_fullsized":"2024-02-05T06:37:06.320868+0000","mapping_epoch":96,"log_start":"102'2000","ondisk_log_start":"102'2000","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":4982,"log_dups_size":0,"ondisk_log_size":4982,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457572352,"num_objects":6982,"num_object_clones":0,"num_object_copies":20946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6982,"num_write_kb":446848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":130,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.997400+0000","last_change":"2024-02-05T06:36:54.951238+0000","last_active":"2024-02-05T06:36:59.997400+0000","last_peered":"2024-02-05T06:36:59.997400+0000","last_clean":"2024-02-05T06:36:59.997400+0000","last_became_active":"2024-02-05T06:36:54.950929+0000","last_became_peered":"2024-02-05T06:36:54.950929+0000","last_unstale":"2024-02-05T06:36:59.997400+0000","last_undegraded":"2024-02-05T06:36:59.997400+0000","last_fullsized":"2024-02-05T06:36:59.997400+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:05.712737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:05.712737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:21.033967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":386,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:00.003036+0000","last_change":"2024-02-05T06:35:11.952583+0000","last_active":"2024-02-05T06:37:00.003036+0000","last_peered":"2024-02-05T06:37:00.003036+0000","last_clean":"2024-02-05T06:37:00.003036+0000","last_became_active":"2024-02-05T06:33:19.862877+0000","last_became_peered":"2024-02-05T06:33:19.862877+0000","last_unstale":"2024-02-05T06:37:00.003036+0000","last_undegraded":"2024-02-05T06:37:00.003036+0000","last_fullsized":"2024-02-05T06:37:00.003036+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:35:11.952460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:35:11.952460+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":132,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.993753+0000","last_change":"2024-02-05T06:34:44.894533+0000","last_active":"2024-02-05T06:36:59.993753+0000","last_peered":"2024-02-05T06:36:59.993753+0000","last_clean":"2024-02-05T06:36:59.993753+0000","last_became_active":"2024-02-05T06:33:29.552698+0000","last_became_peered":"2024-02-05T06:33:29.552698+0000","last_unstale":"2024-02-05T06:36:59.993753+0000","last_undegraded":"2024-02-05T06:36:59.993753+0000","last_fullsized":"2024-02-05T06:36:59.993753+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:44.894443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:44.894443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"119'18","reported_seq":243,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:00.001093+0000","last_change":"2024-02-05T06:36:54.992929+0000","last_active":"2024-02-05T06:37:00.001093+0000","last_peered":"2024-02-05T06:37:00.001093+0000","last_clean":"2024-02-05T06:37:00.001093+0000","last_became_active":"2024-02-05T06:36:54.966277+0000","last_became_peered":"2024-02-05T06:36:54.966277+0000","last_unstale":"2024-02-05T06:37:00.001093+0000","last_undegraded":"2024-02-05T06:37:00.001093+0000","last_fullsized":"2024-02-05T06:37:00.001093+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-02-05T06:35:02.410049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:35:02.410049+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:08.646582+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":124,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:00.003549+0000","last_change":"2024-02-05T06:35:13.945600+0000","last_active":"2024-02-05T06:37:00.003549+0000","last_peered":"2024-02-05T06:37:00.003549+0000","last_clean":"2024-02-05T06:37:00.003549+0000","last_became_active":"2024-02-05T06:33:33.591484+0000","last_became_peered":"2024-02-05T06:33:33.591484+0000","last_unstale":"2024-02-05T06:37:00.003549+0000","last_undegraded":"2024-02-05T06:37:00.003549+0000","last_fullsized":"2024-02-05T06:37:00.003549+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:13.945483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:35:13.945483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"122'7209","reported_seq":7271,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:06.118222+0000","last_change":"2024-02-05T06:35:27.933269+0000","last_active":"2024-02-05T06:37:06.118222+0000","last_peered":"2024-02-05T06:37:06.118222+0000","last_clean":"2024-02-05T06:37:06.118222+0000","last_became_active":"2024-02-05T06:35:27.932989+0000","last_became_peered":"2024-02-05T06:35:27.932989+0000","last_unstale":"2024-02-05T06:37:06.118222+0000","last_undegraded":"2024-02-05T06:37:06.118222+0000","last_fullsized":"2024-02-05T06:37:06.118222+0000","mapping_epoch":96,"log_start":"103'2100","ondisk_log_start":"103'2100","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":5109,"log_dups_size":0,"ondisk_log_size":5109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:06.847193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":472449024,"num_objects":7209,"num_object_clones":0,"num_object_copies":21627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7209,"num_write_kb":461376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":127,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.996433+0000","last_change":"2024-02-05T06:36:22.097109+0000","last_active":"2024-02-05T06:36:59.996433+0000","last_peered":"2024-02-05T06:36:59.996433+0000","last_clean":"2024-02-05T06:36:59.996433+0000","last_became_active":"2024-02-05T06:33:35.600656+0000","last_became_peered":"2024-02-05T06:33:35.600656+0000","last_unstale":"2024-02-05T06:36:59.996433+0000","last_undegraded":"2024-02-05T06:36:59.996433+0000","last_fullsized":"2024-02-05T06:36:59.996433+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:22.096881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:22.096881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:50.866314+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":154,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-02-05T06:37:06.320169+0000","last_change":"2024-02-05T06:36:54.973030+0000","last_active":"2024-02-05T06:37:06.320169+0000","last_peered":"2024-02-05T06:37:06.320169+0000","last_clean":"2024-02-05T06:37:06.320169+0000","last_became_active":"2024-02-05T06:36:54.972264+0000","last_became_peered":"2024-02-05T06:36:54.972264+0000","last_unstale":"2024-02-05T06:37:06.320169+0000","last_undegraded":"2024-02-05T06:37:06.320169+0000","last_fullsized":"2024-02-05T06:37:06.320169+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:24.213373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:24.213373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:52.016463+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":361,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:04.580192+0000","last_change":"2024-02-05T06:36:23.894474+0000","last_active":"2024-02-05T06:37:04.580192+0000","last_peered":"2024-02-05T06:37:04.580192+0000","last_clean":"2024-02-05T06:37:04.580192+0000","last_became_active":"2024-02-05T06:33:28.544852+0000","last_became_peered":"2024-02-05T06:33:28.544852+0000","last_unstale":"2024-02-05T06:37:04.580192+0000","last_undegraded":"2024-02-05T06:37:04.580192+0000","last_fullsized":"2024-02-05T06:37:04.580192+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T06:36:23.894359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:36:23.894359+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:43.470659+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0.00074151600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":209,"num_read_kb":204,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":156,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-02-05T06:37:06.320559+0000","last_change":"2024-02-05T06:36:29.231478+0000","last_active":"2024-02-05T06:37:06.320559+0000","last_peered":"2024-02-05T06:37:06.320559+0000","last_clean":"2024-02-05T06:37:06.320559+0000","last_became_active":"2024-02-05T06:33:31.564604+0000","last_became_peered":"2024-02-05T06:33:31.564604+0000","last_unstale":"2024-02-05T06:37:06.320559+0000","last_undegraded":"2024-02-05T06:37:06.320559+0000","last_fullsized":"2024-02-05T06:37:06.320559+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:36:29.231352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:29.231352+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:57.204094+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":122,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.997574+0000","last_change":"2024-02-05T06:35:23.964960+0000","last_active":"2024-02-05T06:36:59.997574+0000","last_peered":"2024-02-05T06:36:59.997574+0000","last_clean":"2024-02-05T06:36:59.997574+0000","last_became_active":"2024-02-05T06:33:33.589388+0000","last_became_peered":"2024-02-05T06:33:33.589388+0000","last_unstale":"2024-02-05T06:36:59.997574+0000","last_undegraded":"2024-02-05T06:36:59.997574+0000","last_fullsized":"2024-02-05T06:36:59.997574+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:23.964888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:35:23.964888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"122'7116","reported_seq":7178,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:06.148795+0000","last_change":"2024-02-05T06:35:27.937863+0000","last_active":"2024-02-05T06:37:06.148795+0000","last_peered":"2024-02-05T06:37:06.148795+0000","last_clean":"2024-02-05T06:37:06.148795+0000","last_became_active":"2024-02-05T06:35:27.937618+0000","last_became_peered":"2024-02-05T06:35:27.937618+0000","last_unstale":"2024-02-05T06:37:06.148795+0000","last_undegraded":"2024-02-05T06:37:06.148795+0000","last_fullsized":"2024-02-05T06:37:06.148795+0000","mapping_epoch":96,"log_start":"102'1400","ondisk_log_start":"102'1400","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":5716,"log_dups_size":0,"ondisk_log_size":5716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466354176,"num_objects":7116,"num_object_clones":0,"num_object_copies":21348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7116,"num_write_kb":455424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":127,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.996760+0000","last_change":"2024-02-05T06:36:33.005959+0000","last_active":"2024-02-05T06:36:59.996760+0000","last_peered":"2024-02-05T06:36:59.996760+0000","last_clean":"2024-02-05T06:36:59.996760+0000","last_became_active":"2024-02-05T06:33:35.664858+0000","last_became_peered":"2024-02-05T06:33:35.664858+0000","last_unstale":"2024-02-05T06:36:59.996760+0000","last_undegraded":"2024-02-05T06:36:59.996760+0000","last_fullsized":"2024-02-05T06:36:59.996760+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:33.005841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:33.005841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:01.594701+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"66'1","reported_seq":145,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.996618+0000","last_change":"2024-02-05T06:36:09.253329+0000","last_active":"2024-02-05T06:36:59.996618+0000","last_peered":"2024-02-05T06:36:59.996618+0000","last_clean":"2024-02-05T06:36:59.996618+0000","last_became_active":"2024-02-05T06:33:29.548227+0000","last_became_peered":"2024-02-05T06:33:29.548227+0000","last_unstale":"2024-02-05T06:36:59.996618+0000","last_undegraded":"2024-02-05T06:36:59.996618+0000","last_fullsized":"2024-02-05T06:36:59.996618+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:36:09.253045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:09.253045+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:28.032291+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'19","reported_seq":166,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-02-05T06:37:06.320760+0000","last_change":"2024-02-05T06:35:27.592730+0000","last_active":"2024-02-05T06:37:06.320760+0000","last_peered":"2024-02-05T06:37:06.320760+0000","last_clean":"2024-02-05T06:37:06.320760+0000","last_became_active":"2024-02-05T06:33:31.667340+0000","last_became_peered":"2024-02-05T06:33:31.667340+0000","last_unstale":"2024-02-05T06:37:06.320760+0000","last_undegraded":"2024-02-05T06:37:06.320760+0000","last_fullsized":"2024-02-05T06:37:06.320760+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-02-05T06:35:27.592613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:35:27.592613+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":123,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:00.000573+0000","last_change":"2024-02-05T06:34:50.353163+0000","last_active":"2024-02-05T06:37:00.000573+0000","last_peered":"2024-02-05T06:37:00.000573+0000","last_clean":"2024-02-05T06:37:00.000573+0000","last_became_active":"2024-02-05T06:33:33.588163+0000","last_became_peered":"2024-02-05T06:33:33.588163+0000","last_unstale":"2024-02-05T06:37:00.000573+0000","last_undegraded":"2024-02-05T06:37:00.000573+0000","last_fullsized":"2024-02-05T06:37:00.000573+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:50.353040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:50.353040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:02.807751+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"122'7277","reported_seq":10420,"reported_epoch":122,"state":"active+remapped+backfilling","last_fresh":"2024-02-05T06:37:06.225107+0000","last_change":"2024-02-05T06:36:56.035140+0000","last_active":"2024-02-05T06:37:06.225107+0000","last_peered":"2024-02-05T06:37:06.225107+0000","last_clean":"2024-02-05T06:36:54.239648+0000","last_became_active":"2024-02-05T06:36:56.028436+0000","last_became_peered":"2024-02-05T06:36:56.028436+0000","last_unstale":"2024-02-05T06:37:06.225107+0000","last_undegraded":"2024-02-05T06:37:06.225107+0000","last_fullsized":"2024-02-05T06:37:06.225107+0000","mapping_epoch":119,"log_start":"104'2400","ondisk_log_start":"104'2400","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":4877,"log_dups_size":0,"ondisk_log_size":4877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":476905472,"num_objects":7277,"num_object_clones":0,"num_object_copies":21831,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5676,"num_objects_unfound":0,"num_objects_dirty":7277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7277,"num_write_kb":465728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1538,"num_bytes_recovered":100794368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[1,3,6],"avail_no_missing":["1","3","6"],"object_location_counts":[{"shards":"1,3,6","objects":7277}],"blocked_by":[],"up_primary":4,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":122,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-02-05T06:37:06.320541+0000","last_change":"2024-02-05T06:36:07.154940+0000","last_active":"2024-02-05T06:37:06.320541+0000","last_peered":"2024-02-05T06:37:06.320541+0000","last_clean":"2024-02-05T06:37:06.320541+0000","last_became_active":"2024-02-05T06:33:35.600485+0000","last_became_peered":"2024-02-05T06:33:35.600485+0000","last_unstale":"2024-02-05T06:37:06.320541+0000","last_undegraded":"2024-02-05T06:37:06.320541+0000","last_fullsized":"2024-02-05T06:37:06.320541+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:07.154815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:07.154815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:21.512385+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":138,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.996920+0000","last_change":"2024-02-05T06:36:13.768661+0000","last_active":"2024-02-05T06:36:59.996920+0000","last_peered":"2024-02-05T06:36:59.996920+0000","last_clean":"2024-02-05T06:36:59.996920+0000","last_became_active":"2024-02-05T06:33:29.554940+0000","last_became_peered":"2024-02-05T06:33:29.554940+0000","last_unstale":"2024-02-05T06:36:59.996920+0000","last_undegraded":"2024-02-05T06:36:59.996920+0000","last_fullsized":"2024-02-05T06:36:59.996920+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:13.768592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:13.768592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:36.460814+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"73'10","reported_seq":178,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.995280+0000","last_change":"2024-02-05T06:36:54.983122+0000","last_active":"2024-02-05T06:36:59.995280+0000","last_peered":"2024-02-05T06:36:59.995280+0000","last_clean":"2024-02-05T06:36:59.995280+0000","last_became_active":"2024-02-05T06:36:54.959010+0000","last_became_peered":"2024-02-05T06:36:54.959010+0000","last_unstale":"2024-02-05T06:36:59.995280+0000","last_undegraded":"2024-02-05T06:36:59.995280+0000","last_fullsized":"2024-02-05T06:36:59.995280+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-02-05T06:36:07.232535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:07.232535+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:12.679933+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"73'8","reported_seq":186,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-02-05T06:37:06.320569+0000","last_change":"2024-02-05T06:36:32.174510+0000","last_active":"2024-02-05T06:37:06.320569+0000","last_peered":"2024-02-05T06:37:06.320569+0000","last_clean":"2024-02-05T06:37:06.320569+0000","last_became_active":"2024-02-05T06:33:33.582187+0000","last_became_peered":"2024-02-05T06:33:33.582187+0000","last_unstale":"2024-02-05T06:37:06.320569+0000","last_undegraded":"2024-02-05T06:37:06.320569+0000","last_fullsized":"2024-02-05T06:37:06.320569+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:36:32.174396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:32.174396+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:34.259380+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"122'7140","reported_seq":7202,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:06.320999+0000","last_change":"2024-02-05T06:35:27.936200+0000","last_active":"2024-02-05T06:37:06.320999+0000","last_peered":"2024-02-05T06:37:06.320999+0000","last_clean":"2024-02-05T06:37:06.320999+0000","last_became_active":"2024-02-05T06:35:27.935898+0000","last_became_peered":"2024-02-05T06:35:27.935898+0000","last_unstale":"2024-02-05T06:37:06.320999+0000","last_undegraded":"2024-02-05T06:37:06.320999+0000","last_fullsized":"2024-02-05T06:37:06.320999+0000","mapping_epoch":96,"log_start":"104'2500","ondisk_log_start":"104'2500","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":4640,"log_dups_size":0,"ondisk_log_size":4640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:07.525357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":467927040,"num_objects":7140,"num_object_clones":0,"num_object_copies":21420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7140,"num_write_kb":456960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":128,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:00.003402+0000","last_change":"2024-02-05T06:36:11.928300+0000","last_active":"2024-02-05T06:37:00.003402+0000","last_peered":"2024-02-05T06:37:00.003402+0000","last_clean":"2024-02-05T06:37:00.003402+0000","last_became_active":"2024-02-05T06:33:35.666315+0000","last_became_peered":"2024-02-05T06:33:35.666315+0000","last_unstale":"2024-02-05T06:37:00.003402+0000","last_undegraded":"2024-02-05T06:37:00.003402+0000","last_fullsized":"2024-02-05T06:37:00.003402+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:11.928194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:11.928194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:16.039940+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"95'2","reported_seq":156,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-02-05T06:37:06.320650+0000","last_change":"2024-02-05T06:36:54.993662+0000","last_active":"2024-02-05T06:37:06.320650+0000","last_peered":"2024-02-05T06:37:06.320650+0000","last_clean":"2024-02-05T06:37:06.320650+0000","last_became_active":"2024-02-05T06:36:54.975093+0000","last_became_peered":"2024-02-05T06:36:54.975093+0000","last_unstale":"2024-02-05T06:37:06.320650+0000","last_undegraded":"2024-02-05T06:37:06.320650+0000","last_fullsized":"2024-02-05T06:37:06.320650+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-05T06:35:25.616708+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-05T06:35:25.616708+0000","last_clean_scrub_stamp":"2024-02-05T06:35:25.616708+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'14","reported_seq":33,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.988184+0000","last_change":"2024-02-05T06:36:55.007431+0000","last_active":"2024-02-05T06:36:59.988184+0000","last_peered":"2024-02-05T06:36:59.988184+0000","last_clean":"2024-02-05T06:36:59.988184+0000","last_became_active":"2024-02-05T06:36:54.965287+0000","last_became_peered":"2024-02-05T06:36:54.965287+0000","last_unstale":"2024-02-05T06:36:59.988184+0000","last_undegraded":"2024-02-05T06:36:59.988184+0000","last_fullsized":"2024-02-05T06:36:59.988184+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'14","last_scrub_stamp":"2024-02-05T06:36:17.814774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:17.814774+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:46.232951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"119'10","reported_seq":198,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.994010+0000","last_change":"2024-02-05T06:36:54.983684+0000","last_active":"2024-02-05T06:36:59.994010+0000","last_peered":"2024-02-05T06:36:59.994010+0000","last_clean":"2024-02-05T06:36:59.994010+0000","last_became_active":"2024-02-05T06:36:54.954819+0000","last_became_peered":"2024-02-05T06:36:54.954819+0000","last_unstale":"2024-02-05T06:36:59.994010+0000","last_undegraded":"2024-02-05T06:36:59.994010+0000","last_fullsized":"2024-02-05T06:36:59.994010+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:34:56.941840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:56.941840+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:07.524981+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"122'7199","reported_seq":7275,"reported_epoch":122,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T06:37:06.306415+0000","last_change":"2024-02-05T06:36:56.054360+0000","last_active":"2024-02-05T06:37:06.306415+0000","last_peered":"2024-02-05T06:37:06.306415+0000","last_clean":"2024-02-05T06:36:54.270379+0000","last_became_active":"2024-02-05T06:36:56.030721+0000","last_became_peered":"2024-02-05T06:36:56.030721+0000","last_unstale":"2024-02-05T06:37:06.306415+0000","last_undegraded":"2024-02-05T06:37:06.306415+0000","last_fullsized":"2024-02-05T06:37:06.306415+0000","mapping_epoch":119,"log_start":"104'2500","ondisk_log_start":"104'2500","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":4699,"log_dups_size":0,"ondisk_log_size":4699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":471793664,"num_objects":7199,"num_object_clones":0,"num_object_copies":21597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7199,"num_objects_unfound":0,"num_objects_dirty":7199,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7199,"num_write_kb":460736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,6],"avail_no_missing":["3","2","6"],"object_location_counts":[{"shards":"2,3,6","objects":7199}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":17,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.988559+0000","last_change":"2024-02-05T06:36:54.959371+0000","last_active":"2024-02-05T06:36:59.988559+0000","last_peered":"2024-02-05T06:36:59.988559+0000","last_clean":"2024-02-05T06:36:59.988559+0000","last_became_active":"2024-02-05T06:36:54.958644+0000","last_became_peered":"2024-02-05T06:36:54.958644+0000","last_unstale":"2024-02-05T06:36:59.988559+0000","last_undegraded":"2024-02-05T06:36:59.988559+0000","last_fullsized":"2024-02-05T06:36:59.988559+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:25.951483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:25.951483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":132,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.995293+0000","last_change":"2024-02-05T06:35:10.061496+0000","last_active":"2024-02-05T06:36:59.995293+0000","last_peered":"2024-02-05T06:36:59.995293+0000","last_clean":"2024-02-05T06:36:59.995293+0000","last_became_active":"2024-02-05T06:33:29.556002+0000","last_became_peered":"2024-02-05T06:33:29.556002+0000","last_unstale":"2024-02-05T06:36:59.995293+0000","last_undegraded":"2024-02-05T06:36:59.995293+0000","last_fullsized":"2024-02-05T06:36:59.995293+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:10.061388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:35:10.061388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"73'18","reported_seq":160,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.993874+0000","last_change":"2024-02-05T06:35:04.034763+0000","last_active":"2024-02-05T06:36:59.993874+0000","last_peered":"2024-02-05T06:36:59.993874+0000","last_clean":"2024-02-05T06:36:59.993874+0000","last_became_active":"2024-02-05T06:33:31.665995+0000","last_became_peered":"2024-02-05T06:33:31.665995+0000","last_unstale":"2024-02-05T06:36:59.993874+0000","last_undegraded":"2024-02-05T06:36:59.993874+0000","last_fullsized":"2024-02-05T06:36:59.993874+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-02-05T06:35:04.034664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:35:04.034664+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":17,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.988211+0000","last_change":"2024-02-05T06:36:54.965253+0000","last_active":"2024-02-05T06:36:59.988211+0000","last_peered":"2024-02-05T06:36:59.988211+0000","last_clean":"2024-02-05T06:36:59.988211+0000","last_became_active":"2024-02-05T06:36:54.963774+0000","last_became_peered":"2024-02-05T06:36:54.963774+0000","last_unstale":"2024-02-05T06:36:59.988211+0000","last_undegraded":"2024-02-05T06:36:59.988211+0000","last_fullsized":"2024-02-05T06:36:59.988211+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:49.579493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:49.579493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"122'7108","reported_seq":7170,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:06.138580+0000","last_change":"2024-02-05T06:35:27.930565+0000","last_active":"2024-02-05T06:37:06.138580+0000","last_peered":"2024-02-05T06:37:06.138580+0000","last_clean":"2024-02-05T06:37:06.138580+0000","last_became_active":"2024-02-05T06:35:27.930258+0000","last_became_peered":"2024-02-05T06:35:27.930258+0000","last_unstale":"2024-02-05T06:37:06.138580+0000","last_undegraded":"2024-02-05T06:37:06.138580+0000","last_fullsized":"2024-02-05T06:37:06.138580+0000","mapping_epoch":96,"log_start":"102'1600","ondisk_log_start":"102'1600","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":5508,"log_dups_size":0,"ondisk_log_size":5508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:08.647908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":465829888,"num_objects":7108,"num_object_clones":0,"num_object_copies":21324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7108,"num_write_kb":454912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":126,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.996664+0000","last_change":"2024-02-05T06:36:12.777606+0000","last_active":"2024-02-05T06:36:59.996664+0000","last_peered":"2024-02-05T06:36:59.996664+0000","last_clean":"2024-02-05T06:36:59.996664+0000","last_became_active":"2024-02-05T06:33:35.593210+0000","last_became_peered":"2024-02-05T06:33:35.593210+0000","last_unstale":"2024-02-05T06:36:59.996664+0000","last_undegraded":"2024-02-05T06:36:59.996664+0000","last_fullsized":"2024-02-05T06:36:59.996664+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:12.777526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:12.777526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:28.471089+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"119'10","reported_seq":25,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.988447+0000","last_change":"2024-02-05T06:36:55.004386+0000","last_active":"2024-02-05T06:36:59.988447+0000","last_peered":"2024-02-05T06:36:59.988447+0000","last_clean":"2024-02-05T06:36:59.988447+0000","last_became_active":"2024-02-05T06:36:54.968825+0000","last_became_peered":"2024-02-05T06:36:54.968825+0000","last_unstale":"2024-02-05T06:36:59.988447+0000","last_undegraded":"2024-02-05T06:36:59.988447+0000","last_fullsized":"2024-02-05T06:36:59.988447+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'9","last_scrub_stamp":"2024-02-05T06:36:11.784734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:11.784734+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:35.592218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":132,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.993823+0000","last_change":"2024-02-05T06:35:17.058289+0000","last_active":"2024-02-05T06:36:59.993823+0000","last_peered":"2024-02-05T06:36:59.993823+0000","last_clean":"2024-02-05T06:36:59.993823+0000","last_became_active":"2024-02-05T06:33:29.555919+0000","last_became_peered":"2024-02-05T06:33:29.555919+0000","last_unstale":"2024-02-05T06:36:59.993823+0000","last_undegraded":"2024-02-05T06:36:59.993823+0000","last_fullsized":"2024-02-05T06:36:59.993823+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:17.058199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:35:17.058199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"73'15","reported_seq":204,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.994560+0000","last_change":"2024-02-05T06:36:54.984654+0000","last_active":"2024-02-05T06:36:59.994560+0000","last_peered":"2024-02-05T06:36:59.994560+0000","last_clean":"2024-02-05T06:36:59.994560+0000","last_became_active":"2024-02-05T06:36:54.948371+0000","last_became_peered":"2024-02-05T06:36:54.948371+0000","last_unstale":"2024-02-05T06:36:59.994560+0000","last_undegraded":"2024-02-05T06:36:59.994560+0000","last_fullsized":"2024-02-05T06:36:59.994560+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-02-05T06:36:28.066781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:28.066781+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:54.946715+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"122'7088","reported_seq":7150,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:06.120166+0000","last_change":"2024-02-05T06:35:27.935929+0000","last_active":"2024-02-05T06:37:06.120166+0000","last_peered":"2024-02-05T06:37:06.120166+0000","last_clean":"2024-02-05T06:37:06.120166+0000","last_became_active":"2024-02-05T06:35:27.935655+0000","last_became_peered":"2024-02-05T06:35:27.935655+0000","last_unstale":"2024-02-05T06:37:06.120166+0000","last_undegraded":"2024-02-05T06:37:06.120166+0000","last_fullsized":"2024-02-05T06:37:06.120166+0000","mapping_epoch":96,"log_start":"102'1500","ondisk_log_start":"102'1500","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":5588,"log_dups_size":0,"ondisk_log_size":5588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:08.647950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":464519168,"num_objects":7088,"num_object_clones":0,"num_object_copies":21264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7088,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7088,"num_write_kb":453632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":120,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.995670+0000","last_change":"2024-02-05T06:35:20.042610+0000","last_active":"2024-02-05T06:36:59.995670+0000","last_peered":"2024-02-05T06:36:59.995670+0000","last_clean":"2024-02-05T06:36:59.995670+0000","last_became_active":"2024-02-05T06:33:35.668020+0000","last_became_peered":"2024-02-05T06:33:35.668020+0000","last_unstale":"2024-02-05T06:36:59.995670+0000","last_undegraded":"2024-02-05T06:36:59.995670+0000","last_fullsized":"2024-02-05T06:36:59.995670+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:20.042534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:20.042534+0000","last_clean_scrub_stamp":"2024-02-05T06:35:20.042534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":142,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.997549+0000","last_change":"2024-02-05T06:36:54.950791+0000","last_active":"2024-02-05T06:36:59.997549+0000","last_peered":"2024-02-05T06:36:59.997549+0000","last_clean":"2024-02-05T06:36:59.997549+0000","last_became_active":"2024-02-05T06:36:54.950484+0000","last_became_peered":"2024-02-05T06:36:54.950484+0000","last_unstale":"2024-02-05T06:36:59.997549+0000","last_undegraded":"2024-02-05T06:36:59.997549+0000","last_fullsized":"2024-02-05T06:36:59.997549+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:24.940878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:24.940878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:34.193410+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.988282+0000","last_change":"2024-02-05T06:36:54.964984+0000","last_active":"2024-02-05T06:36:59.988282+0000","last_peered":"2024-02-05T06:36:59.988282+0000","last_clean":"2024-02-05T06:36:59.988282+0000","last_became_active":"2024-02-05T06:36:54.963340+0000","last_became_peered":"2024-02-05T06:36:54.963340+0000","last_unstale":"2024-02-05T06:36:59.988282+0000","last_undegraded":"2024-02-05T06:36:59.988282+0000","last_fullsized":"2024-02-05T06:36:59.988282+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:21.638465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:35:21.638465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"115'16","reported_seq":29,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.988313+0000","last_change":"2024-02-05T06:36:54.992244+0000","last_active":"2024-02-05T06:36:59.988313+0000","last_peered":"2024-02-05T06:36:59.988313+0000","last_clean":"2024-02-05T06:36:59.988313+0000","last_became_active":"2024-02-05T06:36:54.958582+0000","last_became_peered":"2024-02-05T06:36:54.958582+0000","last_unstale":"2024-02-05T06:36:59.988313+0000","last_undegraded":"2024-02-05T06:36:59.988313+0000","last_fullsized":"2024-02-05T06:36:59.988313+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'16","last_scrub_stamp":"2024-02-05T06:36:34.793243+0000","last_deep_scrub":"115'16","last_deep_scrub_stamp":"2024-02-05T06:36:34.793243+0000","last_clean_scrub_stamp":"2024-02-05T06:36:34.793243+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:56.608681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"122'7146","reported_seq":7207,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:06.202309+0000","last_change":"2024-02-05T06:35:27.938681+0000","last_active":"2024-02-05T06:37:06.202309+0000","last_peered":"2024-02-05T06:37:06.202309+0000","last_clean":"2024-02-05T06:37:06.202309+0000","last_became_active":"2024-02-05T06:35:27.938387+0000","last_became_peered":"2024-02-05T06:35:27.938387+0000","last_unstale":"2024-02-05T06:37:06.202309+0000","last_undegraded":"2024-02-05T06:37:06.202309+0000","last_fullsized":"2024-02-05T06:37:06.202309+0000","mapping_epoch":96,"log_start":"101'900","ondisk_log_start":"101'900","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":6246,"log_dups_size":0,"ondisk_log_size":6246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":468320256,"num_objects":7146,"num_object_clones":0,"num_object_copies":21438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7146,"num_write_kb":457344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":138,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:00.000577+0000","last_change":"2024-02-05T06:36:54.959177+0000","last_active":"2024-02-05T06:37:00.000577+0000","last_peered":"2024-02-05T06:37:00.000577+0000","last_clean":"2024-02-05T06:37:00.000577+0000","last_became_active":"2024-02-05T06:36:54.958805+0000","last_became_peered":"2024-02-05T06:36:54.958805+0000","last_unstale":"2024-02-05T06:37:00.000577+0000","last_undegraded":"2024-02-05T06:37:00.000577+0000","last_fullsized":"2024-02-05T06:37:00.000577+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:30.754865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:30.754865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:50.126762+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":143,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.996703+0000","last_change":"2024-02-05T06:36:54.958628+0000","last_active":"2024-02-05T06:36:59.996703+0000","last_peered":"2024-02-05T06:36:59.996703+0000","last_clean":"2024-02-05T06:36:59.996703+0000","last_became_active":"2024-02-05T06:36:54.958082+0000","last_became_peered":"2024-02-05T06:36:54.958082+0000","last_unstale":"2024-02-05T06:36:59.996703+0000","last_undegraded":"2024-02-05T06:36:59.996703+0000","last_fullsized":"2024-02-05T06:36:59.996703+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:29.069747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:29.069747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:46.117534+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":143,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:00.000688+0000","last_change":"2024-02-05T06:36:54.966548+0000","last_active":"2024-02-05T06:37:00.000688+0000","last_peered":"2024-02-05T06:37:00.000688+0000","last_clean":"2024-02-05T06:37:00.000688+0000","last_became_active":"2024-02-05T06:36:54.966178+0000","last_became_peered":"2024-02-05T06:36:54.966178+0000","last_unstale":"2024-02-05T06:37:00.000688+0000","last_undegraded":"2024-02-05T06:37:00.000688+0000","last_fullsized":"2024-02-05T06:37:00.000688+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:57.290944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:57.290944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:02.807790+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"95'25","reported_seq":35,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.988126+0000","last_change":"2024-02-05T06:36:55.005133+0000","last_active":"2024-02-05T06:36:59.988126+0000","last_peered":"2024-02-05T06:36:59.988126+0000","last_clean":"2024-02-05T06:36:59.988126+0000","last_became_active":"2024-02-05T06:36:54.961719+0000","last_became_peered":"2024-02-05T06:36:54.961719+0000","last_unstale":"2024-02-05T06:36:59.988126+0000","last_undegraded":"2024-02-05T06:36:59.988126+0000","last_fullsized":"2024-02-05T06:36:59.988126+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'25","last_scrub_stamp":"2024-02-05T06:35:26.274944+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-02-05T06:35:26.274944+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.274944+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"123'7139","reported_seq":7203,"reported_epoch":123,"state":"active+clean","last_fresh":"2024-02-05T06:37:06.335736+0000","last_change":"2024-02-05T06:35:27.931906+0000","last_active":"2024-02-05T06:37:06.335736+0000","last_peered":"2024-02-05T06:37:06.335736+0000","last_clean":"2024-02-05T06:37:06.335736+0000","last_became_active":"2024-02-05T06:35:27.931422+0000","last_became_peered":"2024-02-05T06:35:27.931422+0000","last_unstale":"2024-02-05T06:37:06.335736+0000","last_undegraded":"2024-02-05T06:37:06.335736+0000","last_fullsized":"2024-02-05T06:37:06.335736+0000","mapping_epoch":96,"log_start":"102'2100","ondisk_log_start":"102'2100","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":5039,"log_dups_size":0,"ondisk_log_size":5039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":467861504,"num_objects":7139,"num_object_clones":0,"num_object_copies":21417,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7139,"num_write_kb":456896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":17,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.988210+0000","last_change":"2024-02-05T06:36:54.963613+0000","last_active":"2024-02-05T06:36:59.988210+0000","last_peered":"2024-02-05T06:36:59.988210+0000","last_clean":"2024-02-05T06:36:59.988210+0000","last_became_active":"2024-02-05T06:36:54.961560+0000","last_became_peered":"2024-02-05T06:36:54.961560+0000","last_unstale":"2024-02-05T06:36:59.988210+0000","last_undegraded":"2024-02-05T06:36:59.988210+0000","last_fullsized":"2024-02-05T06:36:59.988210+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:27.459848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:27.459848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"92'9","reported_seq":183,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:37:00.000348+0000","last_change":"2024-02-05T06:36:21.707926+0000","last_active":"2024-02-05T06:37:00.000348+0000","last_peered":"2024-02-05T06:37:00.000348+0000","last_clean":"2024-02-05T06:37:00.000348+0000","last_became_active":"2024-02-05T06:33:33.588478+0000","last_became_peered":"2024-02-05T06:33:33.588478+0000","last_unstale":"2024-02-05T06:37:00.000348+0000","last_undegraded":"2024-02-05T06:37:00.000348+0000","last_fullsized":"2024-02-05T06:37:00.000348+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'9","last_scrub_stamp":"2024-02-05T06:36:21.707759+0000","last_deep_scrub":"92'9","last_deep_scrub_stamp":"2024-02-05T06:34:52.283780+0000","last_clean_scrub_stamp":"2024-02-05T06:36:21.707759+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:38.768987+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":152,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.997838+0000","last_change":"2024-02-05T06:36:54.971857+0000","last_active":"2024-02-05T06:36:59.997838+0000","last_peered":"2024-02-05T06:36:59.997838+0000","last_clean":"2024-02-05T06:36:59.997838+0000","last_became_active":"2024-02-05T06:36:54.971304+0000","last_became_peered":"2024-02-05T06:36:54.971304+0000","last_unstale":"2024-02-05T06:36:59.997838+0000","last_undegraded":"2024-02-05T06:36:59.997838+0000","last_fullsized":"2024-02-05T06:36:59.997838+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:37.506830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:37.506830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:00.884497+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"93'15","reported_seq":29,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-02-05T06:36:59.988807+0000","last_change":"2024-02-05T06:36:55.003532+0000","last_active":"2024-02-05T06:36:59.988807+0000","last_peered":"2024-02-05T06:36:59.988807+0000","last_clean":"2024-02-05T06:36:59.988807+0000","last_became_active":"2024-02-05T06:36:54.963620+0000","last_became_peered":"2024-02-05T06:36:54.963620+0000","last_unstale":"2024-02-05T06:36:59.988807+0000","last_undegraded":"2024-02-05T06:36:59.988807+0000","last_fullsized":"2024-02-05T06:36:59.988807+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'15","last_scrub_stamp":"2024-02-05T06:35:04.584657+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-02-05T06:35:04.584657+0000","last_clean_scrub_stamp":"2024-02-05T06:35:04.584657+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":7490306048,"num_objects":114293,"num_object_clones":0,"num_object_copies":342879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":31156,"num_objects_unfound":0,"num_objects_dirty":114293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":114293,"num_write_kb":7314752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4679,"num_bytes_recovered":306642944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22783459328,"data_stored":22783459328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83893,"ondisk_log_size":83893,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":209,"num_read_kb":204,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":686,"num_read_kb":443,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":3416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":452,"ondisk_log_size":452,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103215,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2804532,"kb_used_data":2733100,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":90940620,"statfs":{"total":95995035648,"available":93123194880,"internally_reserved":0,"allocated":2798694400,"data_stored":2798309194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:36:23 2024","interfaces":[{"interface":"back","average":{"1min":3.8900000000000001,"5min":1.23,"15min":0.78800000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":68.921000000000006,"5min":68.921000000000006,"15min":68.921000000000006},"last":1.5069999999999999},{"interface":"front","average":{"1min":3.9119999999999999,"5min":1.264,"15min":0.82099999999999995},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":69.616,"5min":69.616,"15min":69.616},"last":1.0229999999999999}]},{"osd":1,"last update":"Mon Feb 5 06:36:23 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.59199999999999997,"15min":0.57399999999999995},"min":{"1min":0.254,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":3.4609999999999999,"5min":3.4609999999999999,"15min":3.4609999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.628,"15min":0.60799999999999998},"min":{"1min":0.32700000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":2.5750000000000002,"5min":2.5750000000000002,"15min":2.5750000000000002},"last":1.1850000000000001}]},{"osd":2,"last update":"Mon Feb 5 06:36:23 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.58499999999999996,"15min":0.57699999999999996},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.093,"5min":2.093,"15min":2.093},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.622,"5min":0.58499999999999996,"15min":0.57199999999999995},"min":{"1min":0.34699999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.49399999999999999}]},{"osd":3,"last update":"Mon Feb 5 06:36:23 2024","interfaces":[{"interface":"back","average":{"1min":5.0209999999999999,"5min":1.4810000000000001,"15min":0.89300000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":72.168999999999997,"5min":72.168999999999997,"15min":72.168999999999997},"last":0.58999999999999997},{"interface":"front","average":{"1min":5.0709999999999997,"5min":1.5,"15min":0.90200000000000002},"min":{"1min":0.40699999999999997,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":68.962999999999994,"5min":68.962999999999994,"15min":68.962999999999994},"last":0.624}]},{"osd":4,"last update":"Mon Feb 5 06:36:23 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.44400000000000001,"15min":0.443},"min":{"1min":0.29199999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.48399999999999999,"15min":0.49299999999999999},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.61299999999999999,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.86799999999999999}]},{"osd":5,"last update":"Mon Feb 5 06:36:23 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.49299999999999999,"15min":0.498},"min":{"1min":0.29599999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.65400000000000003,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.53500000000000003,"15min":0.54100000000000004},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.91900000000000004,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.60399999999999998}]},{"osd":6,"last update":"Mon Feb 5 06:36:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53700000000000003,"15min":0.53000000000000003},"min":{"1min":0.38500000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.53300000000000003,"15min":0.53800000000000003},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.67100000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":1.1339999999999999}]}]},{"osd":6,"up_from":49,"seq":210453397557,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3279580,"kb_used_data":3194260,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":90465572,"statfs":{"total":95995035648,"available":92636745728,"internally_reserved":0,"allocated":3270922240,"data_stored":3270560161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":87357894},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:36:59 2024","interfaces":[{"interface":"back","average":{"1min":1.8600000000000001,"5min":0.84499999999999997,"15min":0.65700000000000003},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":6.1120000000000001,"5min":6.1120000000000001,"15min":6.1120000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":1.0109999999999999,"5min":0.64100000000000001,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":4.2469999999999999,"5min":4.2469999999999999,"15min":4.2469999999999999},"last":0.41599999999999998}]},{"osd":1,"last update":"Mon Feb 5 06:36:59 2024","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":0.66300000000000003,"15min":0.56999999999999995},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":5.9660000000000002,"5min":5.9660000000000002,"15min":5.9660000000000002},"last":0.27800000000000002},{"interface":"front","average":{"1min":1.139,"5min":0.67500000000000004,"15min":0.61499999999999999},"min":{"1min":0.29199999999999998,"5min":0.255,"15min":0.255},"max":{"1min":5.5099999999999998,"5min":5.5099999999999998,"15min":5.5099999999999998},"last":0.437}]},{"osd":2,"last update":"Mon Feb 5 06:36:59 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.625,"15min":0.59699999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":2.956,"5min":2.956,"15min":2.956},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.59799999999999998,"15min":0.56799999999999995},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.6840000000000002,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":0.54900000000000004}]},{"osd":3,"last update":"Mon Feb 5 06:36:59 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":0.73699999999999999,"15min":0.60899999999999999},"min":{"1min":0.33000000000000002,"5min":0.315,"15min":0.315},"max":{"1min":3.48,"5min":3.48,"15min":3.48},"last":0.33000000000000002},{"interface":"front","average":{"1min":1.399,"5min":0.80600000000000005,"15min":0.65100000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":6.5419999999999998,"5min":6.5419999999999998,"15min":6.5419999999999998},"last":0.372}]},{"osd":4,"last update":"Mon Feb 5 06:36:59 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.44,"15min":0.44600000000000001},"min":{"1min":0.32700000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.98899999999999999,"5min":1.355,"15min":1.355},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45600000000000002,"15min":0.42999999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.39200000000000002}]},{"osd":5,"last update":"Mon Feb 5 06:36:59 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46500000000000002,"15min":0.45700000000000002},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.251},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.47799999999999998,"15min":0.45800000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.42699999999999999}]},{"osd":7,"last update":"Mon Feb 5 06:36:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.51700000000000002,"15min":0.52100000000000002},"min":{"1min":0.32600000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.75800000000000001,"5min":0.872,"15min":0.872},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.55000000000000004,"15min":0.55300000000000005},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.80200000000000005,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.38500000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3290184,"kb_used_data":3200704,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":90454968,"statfs":{"total":95995035648,"available":92625887232,"internally_reserved":0,"allocated":3277520896,"data_stored":3277115311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:36:57 2024","interfaces":[{"interface":"back","average":{"1min":2.5019999999999998,"5min":1.6060000000000001,"15min":0.753},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":5.7039999999999997,"5min":67.421000000000006,"15min":67.421000000000006},"last":3.3719999999999999},{"interface":"front","average":{"1min":2.0299999999999998,"5min":1.474,"15min":0.71199999999999997},"min":{"1min":0.215,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":4.415,"5min":67.885000000000005,"15min":67.885000000000005},"last":4.8949999999999996}]},{"osd":2,"last update":"Mon Feb 5 06:36:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":0.58199999999999996,"15min":0.45300000000000001},"min":{"1min":0.246,"5min":0.214,"15min":0.214},"max":{"1min":4.7510000000000003,"5min":4.7510000000000003,"15min":4.7510000000000003},"last":0.71899999999999997},{"interface":"front","average":{"1min":1.0880000000000001,"5min":0.52900000000000003,"15min":0.42499999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.7050000000000001,"5min":3.7050000000000001,"15min":3.7050000000000001},"last":6.141}]},{"osd":3,"last update":"Mon Feb 5 06:36:47 2024","interfaces":[{"interface":"back","average":{"1min":7.0549999999999997,"5min":1.754,"15min":0.86399999999999999},"min":{"1min":0.36299999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":69.798000000000002,"5min":69.798000000000002,"15min":69.798000000000002},"last":1.6799999999999999},{"interface":"front","average":{"1min":6.0720000000000001,"5min":1.6519999999999999,"15min":0.86599999999999999},"min":{"1min":0.55500000000000005,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":70.765000000000001,"5min":70.765000000000001,"15min":70.765000000000001},"last":4.5789999999999997}]},{"osd":4,"last update":"Mon Feb 5 06:36:05 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.61699999999999999,"15min":0.57799999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":3.75,"5min":3.75,"15min":3.75},"last":0.41499999999999998},{"interface":"front","average":{"1min":1.0680000000000001,"5min":0.69099999999999995,"15min":0.61399999999999999},"min":{"1min":0.42499999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":5.2169999999999996,"5min":5.2169999999999996,"15min":5.2169999999999996},"last":0.64000000000000001}]},{"osd":5,"last update":"Mon Feb 5 06:36:32 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":0.94199999999999995,"15min":0.98999999999999999},"min":{"1min":0.45500000000000002,"5min":0.377,"15min":0.377},"max":{"1min":5.3460000000000001,"5min":8.2509999999999994,"15min":8.2509999999999994},"last":0.75800000000000001},{"interface":"front","average":{"1min":1.575,"5min":0.80500000000000005,"15min":0.69399999999999995},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":6.032,"5min":6.032,"15min":6.032},"last":0.503}]},{"osd":6,"last update":"Mon Feb 5 06:36:57 2024","interfaces":[{"interface":"back","average":{"1min":2.387,"5min":1.036,"15min":0.78000000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":13.391,"5min":13.391,"15min":13.391},"last":0.45000000000000001},{"interface":"front","average":{"1min":1.496,"5min":0.85299999999999998,"15min":0.73899999999999999},"min":{"1min":0.57899999999999996,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":5.3099999999999996,"5min":5.3099999999999996,"15min":5.3099999999999996},"last":1.508}]},{"osd":7,"last update":"Mon Feb 5 06:36:20 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":0.73199999999999998,"15min":0.63800000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":5.1980000000000004,"5min":5.1980000000000004,"15min":5.1980000000000004},"last":0.47699999999999998},{"interface":"front","average":{"1min":1.1479999999999999,"5min":0.97899999999999998,"15min":1.0209999999999999},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":4.4870000000000001,"5min":8.2219999999999995,"15min":8.2219999999999995},"last":0.68000000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738452,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3739900,"kb_used_data":3650420,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":90005252,"statfs":{"total":95995035648,"available":92165378048,"internally_reserved":0,"allocated":3738030080,"data_stored":3737636438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 06:36:55 2024","interfaces":[{"interface":"back","average":{"1min":2.0800000000000001,"5min":0.79800000000000004,"15min":0.47199999999999998},"min":{"1min":0.32200000000000001,"5min":0.23300000000000001,"15min":0.188},"max":{"1min":12.031000000000001,"5min":12.031000000000001,"15min":12.031000000000001},"last":4.7400000000000002},{"interface":"front","average":{"1min":2.0630000000000002,"5min":0.77200000000000002,"15min":0.47099999999999997},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.20499999999999999},"max":{"1min":7.1120000000000001,"5min":7.1120000000000001,"15min":7.1120000000000001},"last":2.8849999999999998}]},{"osd":2,"last update":"Mon Feb 5 06:36:14 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":0.629,"15min":0.47699999999999998},"min":{"1min":0.32200000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":3.6989999999999998,"5min":3.6989999999999998,"15min":3.6989999999999998},"last":3.6520000000000001},{"interface":"front","average":{"1min":1.1850000000000001,"5min":0.59999999999999998,"15min":0.47699999999999998},"min":{"1min":0.33100000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.113,"5min":3.113,"15min":3.113},"last":4.0099999999999998}]},{"osd":3,"last update":"Mon Feb 5 06:36:40 2024","interfaces":[{"interface":"back","average":{"1min":2.5539999999999998,"5min":0.88700000000000001,"15min":0.59899999999999998},"min":{"1min":0.311,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":12.084,"5min":12.084,"15min":12.084},"last":4.649},{"interface":"front","average":{"1min":2.0990000000000002,"5min":0.81000000000000005,"15min":0.56899999999999995},"min":{"1min":0.438,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.8579999999999997,"5min":4.8579999999999997,"15min":4.8579999999999997},"last":4.1040000000000001}]},{"osd":4,"last update":"Mon Feb 5 06:36:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":0.69899999999999995,"15min":0.629},"min":{"1min":0.39900000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":2.5619999999999998,"5min":2.5619999999999998,"15min":2.5619999999999998},"last":2.9180000000000001},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.71399999999999997,"15min":0.64000000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":4.2590000000000003,"5min":4.367,"15min":4.367},"last":4.8070000000000004}]},{"osd":5,"last update":"Mon Feb 5 06:36:31 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":0.752,"15min":0.67700000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":4.7599999999999998,"5min":4.7599999999999998,"15min":4.7599999999999998},"last":0.311},{"interface":"front","average":{"1min":1.3109999999999999,"5min":0.76700000000000002,"15min":0.68899999999999995},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":3.246,"5min":3.246,"15min":3.246},"last":2.98}]},{"osd":6,"last update":"Mon Feb 5 06:36:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8069999999999999,"5min":0.95499999999999996,"15min":0.74299999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":12.119999999999999,"5min":12.119999999999999,"15min":12.119999999999999},"last":4.6879999999999997},{"interface":"front","average":{"1min":1.3680000000000001,"5min":0.82799999999999996,"15min":0.71199999999999997},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":4.8029999999999999,"5min":4.8029999999999999,"15min":4.8029999999999999},"last":0.34300000000000003}]},{"osd":7,"last update":"Mon Feb 5 06:36:20 2024","interfaces":[{"interface":"back","average":{"1min":1.302,"5min":0.73999999999999999,"15min":0.64400000000000002},"min":{"1min":0.433,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":3.294,"5min":3.294,"15min":3.294},"last":2.9460000000000002},{"interface":"front","average":{"1min":1.208,"5min":0.94099999999999995,"15min":0.94399999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":4.2300000000000004,"5min":5.2670000000000003,"15min":5.2670000000000003},"last":4.7750000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411401,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2802404,"kb_used_data":2735132,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":90942748,"statfs":{"total":95995035648,"available":93125373952,"internally_reserved":0,"allocated":2800775168,"data_stored":2800405671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:36:16 2024","interfaces":[{"interface":"back","average":{"1min":4.1909999999999998,"5min":1.1519999999999999,"15min":0.623},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":50.755000000000003,"5min":50.755000000000003,"15min":50.755000000000003},"last":1.1859999999999999},{"interface":"front","average":{"1min":4.157,"5min":1.1739999999999999,"15min":0.63800000000000001},"min":{"1min":0.192,"5min":0.188,"15min":0.188},"max":{"1min":50.802999999999997,"5min":50.802999999999997,"15min":50.802999999999997},"last":1.391}]},{"osd":1,"last update":"Mon Feb 5 06:36:16 2024","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":0.61199999999999999,"15min":0.46000000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":10.243,"5min":10.243,"15min":10.243},"last":0.34699999999999998},{"interface":"front","average":{"1min":1.204,"5min":0.61099999999999999,"15min":0.48299999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":5.5609999999999999,"5min":5.5609999999999999,"15min":5.5609999999999999},"last":1.222}]},{"osd":3,"last update":"Mon Feb 5 06:36:41 2024","interfaces":[{"interface":"back","average":{"1min":4.5,"5min":1.2949999999999999,"15min":0.74199999999999999},"min":{"1min":0.46700000000000003,"5min":0.254,"15min":0.254},"max":{"1min":53.127000000000002,"5min":53.127000000000002,"15min":53.127000000000002},"last":0.42099999999999999},{"interface":"front","average":{"1min":4.2430000000000003,"5min":1.2669999999999999,"15min":0.745},"min":{"1min":0.30499999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":52.607999999999997,"5min":52.607999999999997,"15min":52.607999999999997},"last":4.2190000000000003}]},{"osd":4,"last update":"Mon Feb 5 06:36:04 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.66300000000000003,"15min":0.65100000000000002},"min":{"1min":0.42199999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":2.2930000000000001,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.66200000000000003,"15min":0.66500000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.837,"5min":1.837,"15min":1.837},"last":0.66000000000000003}]},{"osd":5,"last update":"Mon Feb 5 06:36:35 2024","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.70199999999999996,"15min":0.70599999999999996},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":3.1469999999999998,"5min":3.1469999999999998,"15min":3.1469999999999998},"last":0.69899999999999995},{"interface":"front","average":{"1min":1.5629999999999999,"5min":0.82799999999999996,"15min":0.73399999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":7.3730000000000002,"5min":7.3730000000000002,"15min":7.3730000000000002},"last":0.90300000000000002}]},{"osd":6,"last update":"Mon Feb 5 06:37:00 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":1.1299999999999999,"15min":1.524},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":3.194,"5min":17.143999999999998,"15min":17.143999999999998},"last":0.84499999999999997},{"interface":"front","average":{"1min":1.9159999999999999,"5min":0.95999999999999996,"15min":0.78800000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":10.27,"5min":10.27,"15min":10.27},"last":0.94899999999999995}]},{"osd":7,"last update":"Mon Feb 5 06:36:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":0.79400000000000004,"15min":0.69899999999999995},"min":{"1min":0.48499999999999999,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":10.206,"5min":10.206,"15min":10.206},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.2190000000000001,"5min":0.77900000000000003,"15min":0.70699999999999996},"min":{"1min":0.47399999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":5.476,"5min":5.476,"15min":5.476},"last":1.0680000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149764,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4642808,"kb_used_data":4543600,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":89102344,"statfs":{"total":95995035648,"available":91240800256,"internally_reserved":0,"allocated":4652646400,"data_stored":4652255886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":101579207},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:36:49 2024","interfaces":[{"interface":"back","average":{"1min":1.7829999999999999,"5min":0.69099999999999995,"15min":0.495},"min":{"1min":0.32500000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":6.9729999999999999,"5min":6.9729999999999999,"15min":6.9729999999999999},"last":2.593},{"interface":"front","average":{"1min":3.1970000000000001,"5min":1.008,"15min":0.64200000000000002},"min":{"1min":0.68700000000000006,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.4580000000000002,"5min":8.4580000000000002,"15min":8.4580000000000002},"last":2.4900000000000002}]},{"osd":1,"last update":"Mon Feb 5 06:36:49 2024","interfaces":[{"interface":"back","average":{"1min":2.5630000000000002,"5min":0.88300000000000001,"15min":0.61399999999999999},"min":{"1min":0.248,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.6010000000000009,"5min":9.6010000000000009,"15min":9.6010000000000009},"last":2.4340000000000002},{"interface":"front","average":{"1min":2.4340000000000002,"5min":0.84999999999999998,"15min":0.58799999999999997},"min":{"1min":0.28000000000000003,"5min":0.246,"15min":0.246},"max":{"1min":7.6040000000000001,"5min":7.6040000000000001,"15min":7.6040000000000001},"last":0.50900000000000001}]},{"osd":2,"last update":"Mon Feb 5 06:36:49 2024","interfaces":[{"interface":"back","average":{"1min":2.4950000000000001,"5min":0.92500000000000004,"15min":0.64300000000000002},"min":{"1min":0.55800000000000005,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":6.6529999999999996,"5min":6.6529999999999996,"15min":6.6529999999999996},"last":0.54300000000000004},{"interface":"front","average":{"1min":1.9399999999999999,"5min":0.74399999999999999,"15min":0.56399999999999995},"min":{"1min":0.254,"5min":0.248,"15min":0.248},"max":{"1min":7.0030000000000001,"5min":7.0030000000000001,"15min":7.0030000000000001},"last":0.82499999999999996}]},{"osd":4,"last update":"Mon Feb 5 06:36:10 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.63600000000000001,"15min":0.627},"min":{"1min":0.41899999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":3.4830000000000001,"5min":3.4830000000000001,"15min":3.4830000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":1.631,"5min":0.78600000000000003,"15min":0.66900000000000004},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":8.4060000000000006,"5min":8.4060000000000006,"15min":8.4060000000000006},"last":2.383}]},{"osd":5,"last update":"Mon Feb 5 06:36:33 2024","interfaces":[{"interface":"back","average":{"1min":1.925,"5min":0.85499999999999998,"15min":0.69099999999999995},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":6.5640000000000001,"5min":6.5640000000000001,"15min":6.5640000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.67200000000000004,"15min":0.64500000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":3.5649999999999999,"5min":3.5649999999999999,"15min":3.5649999999999999},"last":0.48199999999999998}]},{"osd":6,"last update":"Mon Feb 5 06:37:00 2024","interfaces":[{"interface":"back","average":{"1min":1.9119999999999999,"5min":0.98199999999999998,"15min":0.70499999999999996},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":8.3610000000000007,"5min":8.3610000000000007,"15min":8.3610000000000007},"last":2.4630000000000001},{"interface":"front","average":{"1min":2.0070000000000001,"5min":1.0089999999999999,"15min":0.76100000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":6.601,"5min":6.601,"15min":6.601},"last":0.56799999999999995}]},{"osd":7,"last update":"Mon Feb 5 06:36:26 2024","interfaces":[{"interface":"back","average":{"1min":2.895,"5min":1.0489999999999999,"15min":0.73899999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":45.039000000000001,"5min":45.039000000000001,"15min":45.039000000000001},"last":0.77800000000000002},{"interface":"front","average":{"1min":3.3199999999999998,"5min":1.1579999999999999,"15min":0.79600000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":44.036999999999999,"5min":44.036999999999999,"15min":44.036999999999999},"last":0.621}]}]},{"osd":4,"up_from":31,"seq":133143986238,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":348280,"kb_used_data":290736,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93396872,"statfs":{"total":95995035648,"available":95638396928,"internally_reserved":0,"allocated":297713664,"data_stored":297323372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:36:04 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.58699999999999997,"15min":0.59199999999999997},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":18.062999999999999},{"interface":"front","average":{"1min":1.133,"5min":0.67100000000000004,"15min":0.58299999999999996},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":6.1740000000000004,"5min":6.1740000000000004,"15min":6.1740000000000004},"last":17.856000000000002}]},{"osd":1,"last update":"Mon Feb 5 06:36:04 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.59399999999999997,"15min":0.57399999999999995},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.629,"15min":0.61299999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":2.552,"5min":2.552,"15min":2.552},"last":0.58499999999999996}]},{"osd":2,"last update":"Mon Feb 5 06:36:04 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.59699999999999998,"15min":0.57799999999999996},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.649,"5min":2.649,"15min":2.649},"last":0.437},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.57899999999999996,"15min":0.58699999999999997},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.41599999999999998}]},{"osd":3,"last update":"Mon Feb 5 06:36:04 2024","interfaces":[{"interface":"back","average":{"1min":5.4050000000000002,"5min":1.554,"15min":0.92000000000000004},"min":{"1min":0.50600000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":82.620999999999995,"5min":82.620999999999995,"15min":82.620999999999995},"last":0.33700000000000002},{"interface":"front","average":{"1min":4.9470000000000001,"5min":1.4470000000000001,"15min":0.875},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":82.024000000000001,"5min":82.024000000000001,"15min":82.024000000000001},"last":0.67600000000000005}]},{"osd":5,"last update":"Mon Feb 5 06:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.42199999999999999,"15min":0.40600000000000003},"min":{"1min":0.28299999999999997,"5min":0.253,"15min":0.253},"max":{"1min":0.76000000000000001,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.442},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.69499999999999995,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.38200000000000001}]},{"osd":6,"last update":"Mon Feb 5 06:36:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.51800000000000002,"15min":0.498},"min":{"1min":0.33600000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.52700000000000002,"15min":0.48799999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.109,"5min":2.109,"15min":2.109},"last":0.52300000000000002}]},{"osd":7,"last update":"Mon Feb 5 06:36:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.50800000000000001,"15min":0.51000000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70999999999999996,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.52900000000000003,"15min":0.53100000000000003},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.749,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.36599999999999999}]}]},{"osd":5,"up_from":40,"seq":171798691897,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1852788,"kb_used_data":1807724,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":91892364,"statfs":{"total":95995035648,"available":94097780736,"internally_reserved":0,"allocated":1851109376,"data_stored":1850723747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:36:29 2024","interfaces":[{"interface":"back","average":{"1min":1.4199999999999999,"5min":0.69899999999999995,"15min":0.58999999999999997},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":10.663,"5min":10.663,"15min":10.663},"last":5.3360000000000003},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.78800000000000003,"15min":0.629},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":4.6399999999999997,"5min":17.509,"15min":17.509},"last":0.41099999999999998}]},{"osd":1,"last update":"Mon Feb 5 06:36:29 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":0.65700000000000003,"15min":0.59399999999999997},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":6.5869999999999997,"5min":6.5869999999999997,"15min":6.5869999999999997},"last":1.004},{"interface":"front","average":{"1min":0.874,"5min":0.60099999999999998,"15min":0.56399999999999995},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":3.5800000000000001,"5min":3.5800000000000001,"15min":3.5800000000000001},"last":0.53400000000000003}]},{"osd":2,"last update":"Mon Feb 5 06:36:29 2024","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":0.63800000000000001,"15min":0.57999999999999996},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":5.5350000000000001,"5min":5.5350000000000001,"15min":5.5350000000000001},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.52300000000000002,"15min":0.53100000000000003},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.48699999999999999}]},{"osd":3,"last update":"Mon Feb 5 06:36:29 2024","interfaces":[{"interface":"back","average":{"1min":1.6699999999999999,"5min":0.76100000000000001,"15min":0.628},"min":{"1min":0.33200000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":15.082000000000001,"5min":15.082000000000001,"15min":15.082000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":2.3650000000000002,"5min":0.93000000000000005,"15min":0.69899999999999995},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":15.978,"5min":15.978,"15min":15.978},"last":1.0640000000000001}]},{"osd":4,"last update":"Mon Feb 5 06:36:29 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.39900000000000002,"15min":0.40200000000000002},"min":{"1min":0.29199999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.42399999999999999,"15min":0.42199999999999999},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.59099999999999997,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.31}]},{"osd":6,"last update":"Mon Feb 5 06:36:56 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.46999999999999997,"15min":0.45300000000000001},"min":{"1min":0.33300000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.76100000000000001,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.443},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.48999999999999999,"15min":0.47399999999999998},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.5}]},{"osd":7,"last update":"Mon Feb 5 06:36:17 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.46899999999999997,"15min":0.46600000000000003},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.57599999999999996,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.499,"15min":0.498},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.622,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.375}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3740729344,"data_stored":3740729344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3278635008,"data_stored":3278635008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2818768896,"data_stored":2818768896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4683202560,"data_stored":4683202560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":313458688,"data_stored":313458688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1864630272,"data_stored":1864630272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3282239488,"data_stored":3282239488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2801795072,"data_stored":2801795072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T06:37:15.257 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-05T06:37:15.257 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd pool set .mgr pg_num 11 2024-02-05T06:37:15.329 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:15.377 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:15.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:15.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:15.479 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:15.506 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:15.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:15.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:15.608 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:15.634 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:15.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:15.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:15.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:15 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:37:15.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:15 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:37:15.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:15 smithi167 ceph-mon[32774]: pgmap v137: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 767 B/s rd, 31 MiB/s wr, 876 op/s; 28410/336729 objects misplaced (8.437%); 35 MiB/s, 567 objects/s recovering 2024-02-05T06:37:15.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:15 smithi167 ceph-mon[32774]: osdmap e125: 8 total, 8 up, 8 in 2024-02-05T06:37:15.736 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:15.781 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:15.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:15.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:15.883 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:15.906 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:15.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:15.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:15.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:15 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:37:15.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:15 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:37:15.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:15 smithi028 ceph-mon[37984]: pgmap v137: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 767 B/s rd, 31 MiB/s wr, 876 op/s; 28410/336729 objects misplaced (8.437%); 35 MiB/s, 567 objects/s recovering 2024-02-05T06:37:15.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:15 smithi028 ceph-mon[37984]: osdmap e125: 8 total, 8 up, 8 in 2024-02-05T06:37:15.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:15 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:37:15.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:15 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:37:15.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:15 smithi028 ceph-mon[29442]: pgmap v137: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 767 B/s rd, 31 MiB/s wr, 876 op/s; 28410/336729 objects misplaced (8.437%); 35 MiB/s, 567 objects/s recovering 2024-02-05T06:37:15.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:15 smithi028 ceph-mon[29442]: osdmap e125: 8 total, 8 up, 8 in 2024-02-05T06:37:16.008 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:16.043 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:16.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:16.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:16.149 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:16.186 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:16.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:16.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:16.287 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:16.329 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:16.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:16.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:16.430 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:16.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:16.464 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:16.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:16.565 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:16.603 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:16.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:16.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:16.705 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:16.738 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:16.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:16.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:16.811 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:37:16.839 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:16.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:16.921 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:16.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:17.022 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:17.051 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:17.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:17.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:17.153 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:17.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:17.187 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:17.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:17.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:17.320 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:17.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:17.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:17.422 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:17.453 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:17.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:17.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:17.555 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:17.581 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:17.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:17.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:17.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:17.729 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:17.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:17 smithi167 ceph-mon[32774]: pgmap v139: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 6.4 MiB/s wr, 663 op/s; 28410/332517 objects misplaced (8.544%); 17 MiB/s, 270 objects/s recovering 2024-02-05T06:37:17.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:17.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:17.832 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:17.862 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:17.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:17.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:17.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:17 smithi028 ceph-mon[37984]: pgmap v139: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 6.4 MiB/s wr, 663 op/s; 28410/332517 objects misplaced (8.544%); 17 MiB/s, 270 objects/s recovering 2024-02-05T06:37:17.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:17 smithi028 ceph-mon[29442]: pgmap v139: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 6.4 MiB/s wr, 663 op/s; 28410/332517 objects misplaced (8.544%); 17 MiB/s, 270 objects/s recovering 2024-02-05T06:37:17.964 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:17.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:17.995 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:17.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:18.096 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:18.128 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:18.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:18.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:18.230 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:18.254 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:18.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:18.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:18.356 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:18.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:18.393 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:18.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:18.494 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:18.532 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:18.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:18.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:18.633 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:18.670 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:18.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:18.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:18.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:18.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:18.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:18.919 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:18.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:18.950 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:18.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:19.051 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:19.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:19.096 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:19.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:19.197 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:19.238 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:19.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:19.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:19.340 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:19.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:19.383 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:19.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:19.484 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:19.516 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:19.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:19.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:19.618 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:19.662 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:19.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:19.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:19.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:19 smithi167 ceph-mon[32774]: pgmap v140: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 6.4 MiB/s wr, 1.05k op/s; 25220/323190 objects misplaced (7.803%); 33 MiB/s, 533 objects/s recovering 2024-02-05T06:37:19.763 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:19.792 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:19.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:19.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:19.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:19 smithi028 ceph-mon[29442]: pgmap v140: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 6.4 MiB/s wr, 1.05k op/s; 25220/323190 objects misplaced (7.803%); 33 MiB/s, 533 objects/s recovering 2024-02-05T06:37:19.809 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:19 smithi028 ceph-mon[37984]: pgmap v140: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 6.4 MiB/s wr, 1.05k op/s; 25220/323190 objects misplaced (7.803%); 33 MiB/s, 533 objects/s recovering 2024-02-05T06:37:19.894 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:19.923 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:19.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:19.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:20.024 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:20.051 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:20.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:20.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:20.155 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:20.183 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:20.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:20.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:20.284 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:20.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:20.331 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:20.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:20.432 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:20.458 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:37:20.458 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:37:20.477 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:37:20.494 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:20.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:20.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:20.596 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:20.625 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:20.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:20.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:20.728 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:20.756 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:20.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:20.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:20.858 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:20.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:20.906 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:20.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:20.973 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-02-05T06:37:21.008 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:21.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:21.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:21.151 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:20 smithi028 ceph-mon[37984]: pgmap v141: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.6 MiB/s wr, 912 op/s; 25220/323190 objects misplaced (7.803%); 29 MiB/s, 462 objects/s recovering 2024-02-05T06:37:21.151 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:20 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2283151023' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:37:21.151 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:20 smithi028 ceph-mon[29442]: pgmap v141: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.6 MiB/s wr, 912 op/s; 25220/323190 objects misplaced (7.803%); 29 MiB/s, 462 objects/s recovering 2024-02-05T06:37:21.151 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:20 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2283151023' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:37:21.154 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:21.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:21.199 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:21.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:21.203 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-05T06:37:21.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:20 smithi167 ceph-mon[32774]: pgmap v141: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.6 MiB/s wr, 912 op/s; 25220/323190 objects misplaced (7.803%); 29 MiB/s, 462 objects/s recovering 2024-02-05T06:37:21.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:20 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2283151023' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:37:21.301 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:21.328 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:21.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:21.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:21.430 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:21.499 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:21.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:21.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:21.601 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:21.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:21.655 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:21.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:21.756 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:21.802 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:21.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:21.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:21.904 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:21.932 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:21.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:21.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:22.034 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:22.077 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:22.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:22.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:22.162 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:21 smithi028 ceph-mon[29442]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:37:22.162 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:21 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2283151023' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:37:22.162 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:21 smithi028 ceph-mon[29442]: osdmap e126: 8 total, 8 up, 8 in 2024-02-05T06:37:22.162 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:21 smithi028 ceph-mon[37984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:37:22.162 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:21 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2283151023' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:37:22.162 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:21 smithi028 ceph-mon[37984]: osdmap e126: 8 total, 8 up, 8 in 2024-02-05T06:37:22.180 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:22.210 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:22.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:22.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:22.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:21 smithi167 ceph-mon[32774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:37:22.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:21 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2283151023' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:37:22.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:21 smithi167 ceph-mon[32774]: osdmap e126: 8 total, 8 up, 8 in 2024-02-05T06:37:22.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:22.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:22.353 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:22.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:22.454 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:22.496 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:22.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:22.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:22.509 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:37:22.598 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:22.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:22.651 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:22.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:22.752 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:22.786 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:22.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:22.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:22.888 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:22.917 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:22.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:22.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:22.926 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:37:22 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:37:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:37:22.962 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:37:22 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:37:23.019 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:23.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:23.050 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:23.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:23.151 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:23.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:22 smithi028 ceph-mon[29442]: 6.1b scrub starts 2024-02-05T06:37:23.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:22 smithi028 ceph-mon[29442]: 6.1b scrub ok 2024-02-05T06:37:23.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:22 smithi028 ceph-mon[29442]: pgmap v143: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 22147/311322 objects misplaced (7.114%); 33 MiB/s, 531 objects/s recovering 2024-02-05T06:37:23.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:22 smithi028 ceph-mon[29442]: 3.18 scrub starts 2024-02-05T06:37:23.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:22 smithi028 ceph-mon[29442]: 3.18 scrub ok 2024-02-05T06:37:23.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:22 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:37:23.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:22 smithi028 ceph-mon[37984]: 6.1b scrub starts 2024-02-05T06:37:23.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:22 smithi028 ceph-mon[37984]: 6.1b scrub ok 2024-02-05T06:37:23.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:22 smithi028 ceph-mon[37984]: pgmap v143: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 22147/311322 objects misplaced (7.114%); 33 MiB/s, 531 objects/s recovering 2024-02-05T06:37:23.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:22 smithi028 ceph-mon[37984]: 3.18 scrub starts 2024-02-05T06:37:23.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:22 smithi028 ceph-mon[37984]: 3.18 scrub ok 2024-02-05T06:37:23.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:22 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:37:23.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:23.200 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:23.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:23.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:22 smithi167 ceph-mon[32774]: 6.1b scrub starts 2024-02-05T06:37:23.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:22 smithi167 ceph-mon[32774]: 6.1b scrub ok 2024-02-05T06:37:23.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:22 smithi167 ceph-mon[32774]: pgmap v143: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 22147/311322 objects misplaced (7.114%); 33 MiB/s, 531 objects/s recovering 2024-02-05T06:37:23.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:22 smithi167 ceph-mon[32774]: 3.18 scrub starts 2024-02-05T06:37:23.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:22 smithi167 ceph-mon[32774]: 3.18 scrub ok 2024-02-05T06:37:23.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:22 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:37:23.301 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:23.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:23.348 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:23.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:23.449 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:23.501 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:23.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:23.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:23.603 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:23.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:23.644 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:23.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:23.745 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:23.774 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:23.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:23.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:23.876 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:23.904 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:23.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:23.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:23.929 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:37:23.981 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:37:24.006 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:24.026 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:37:24.047 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:24.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:24.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:24.072 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:37:24.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:23 smithi028 ceph-mon[29442]: 6.14 scrub starts 2024-02-05T06:37:24.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:23 smithi028 ceph-mon[29442]: 6.14 scrub ok 2024-02-05T06:37:24.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:23 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1168708312' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:37:24.072 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:23 smithi028 ceph-mon[37984]: 6.14 scrub starts 2024-02-05T06:37:24.073 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:23 smithi028 ceph-mon[37984]: 6.14 scrub ok 2024-02-05T06:37:24.073 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:23 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1168708312' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:37:24.111 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:37:24.149 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:24.152 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:37:24.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:24.191 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:24.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:24.198 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:37:24.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:23 smithi167 ceph-mon[32774]: 6.14 scrub starts 2024-02-05T06:37:24.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:23 smithi167 ceph-mon[32774]: 6.14 scrub ok 2024-02-05T06:37:24.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:23 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1168708312' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:37:24.237 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:37:24.248 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:37:24.248 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:37:24.248 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:37:24.248 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:37:24.248 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:37:24.248 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:37:24.249 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:37:24.249 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:37:24.292 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:24.321 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:24.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:24.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:24.423 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:24.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:24.462 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:24.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:24.563 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T06:37:24.564 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:24.610 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:24.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:24.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:24.711 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:24.741 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:24.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:24.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:24.843 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:24.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:24.873 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:24.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:24.974 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:25.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:25.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:25.124 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:25.166 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:25.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:25.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:25.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:24 smithi167 ceph-mon[32774]: pgmap v144: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 870 op/s; 22147/311322 objects misplaced (7.114%); 27 MiB/s, 437 objects/s recovering 2024-02-05T06:37:25.267 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:25.297 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:25.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:25.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:25.399 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:25.429 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:25.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:25.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:25.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:24 smithi028 ceph-mon[37984]: pgmap v144: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 870 op/s; 22147/311322 objects misplaced (7.114%); 27 MiB/s, 437 objects/s recovering 2024-02-05T06:37:25.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:24 smithi028 ceph-mon[29442]: pgmap v144: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 870 op/s; 22147/311322 objects misplaced (7.114%); 27 MiB/s, 437 objects/s recovering 2024-02-05T06:37:25.531 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:25.564 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:25.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:25.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:25.666 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:25.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:25.714 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:25.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:25.814 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:25.844 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:25.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:25.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:25.946 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:25.976 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:25.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:25.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:26.078 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:26.097 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:25 smithi028 ceph-mon[37984]: 5.1f scrub starts 2024-02-05T06:37:26.097 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:25 smithi028 ceph-mon[37984]: 5.1f scrub ok 2024-02-05T06:37:26.097 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:25 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:37:26.097 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:25 smithi028 ceph-mon[37984]: 5.8 scrub starts 2024-02-05T06:37:26.097 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:25 smithi028 ceph-mon[37984]: 5.8 scrub ok 2024-02-05T06:37:26.097 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:25 smithi028 ceph-mon[29442]: 5.1f scrub starts 2024-02-05T06:37:26.097 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:25 smithi028 ceph-mon[29442]: 5.1f scrub ok 2024-02-05T06:37:26.098 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:25 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:37:26.098 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:25 smithi028 ceph-mon[29442]: 5.8 scrub starts 2024-02-05T06:37:26.098 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:25 smithi028 ceph-mon[29442]: 5.8 scrub ok 2024-02-05T06:37:26.106 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:26.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:26.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:26.113 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:37:26.208 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:26.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:25 smithi167 ceph-mon[32774]: 5.1f scrub starts 2024-02-05T06:37:26.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:25 smithi167 ceph-mon[32774]: 5.1f scrub ok 2024-02-05T06:37:26.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:25 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:37:26.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:25 smithi167 ceph-mon[32774]: 5.8 scrub starts 2024-02-05T06:37:26.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:25 smithi167 ceph-mon[32774]: 5.8 scrub ok 2024-02-05T06:37:26.238 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:26.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:26.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:26.339 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:26.368 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:26.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:26.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:26.470 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:26.520 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:26.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:26.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:26.622 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:26.668 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:26.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:26.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:26.769 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:26.801 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:26.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:26.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:26.903 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:26.933 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:26.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:26.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:27.034 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:27.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:27.088 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:27.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:27.189 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:27.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:26 smithi028 ceph-mon[29442]: 5.2 scrub starts 2024-02-05T06:37:27.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:26 smithi028 ceph-mon[29442]: 5.2 scrub ok 2024-02-05T06:37:27.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:26 smithi028 ceph-mon[29442]: pgmap v145: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 839 op/s; 22147/307350 objects misplaced (7.206%); 27 MiB/s, 425 objects/s recovering 2024-02-05T06:37:27.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:26 smithi028 ceph-mon[29442]: 5.1a scrub starts 2024-02-05T06:37:27.219 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:26 smithi028 ceph-mon[29442]: 5.1a scrub ok 2024-02-05T06:37:27.219 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:26 smithi028 ceph-mon[37984]: 5.2 scrub starts 2024-02-05T06:37:27.220 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:26 smithi028 ceph-mon[37984]: 5.2 scrub ok 2024-02-05T06:37:27.220 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:26 smithi028 ceph-mon[37984]: pgmap v145: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 839 op/s; 22147/307350 objects misplaced (7.206%); 27 MiB/s, 425 objects/s recovering 2024-02-05T06:37:27.220 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:26 smithi028 ceph-mon[37984]: 5.1a scrub starts 2024-02-05T06:37:27.220 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:26 smithi028 ceph-mon[37984]: 5.1a scrub ok 2024-02-05T06:37:27.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:27.221 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:27.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:27.323 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:27.352 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:27.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:27.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:27.454 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:27.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:26 smithi167 ceph-mon[32774]: 5.2 scrub starts 2024-02-05T06:37:27.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:26 smithi167 ceph-mon[32774]: 5.2 scrub ok 2024-02-05T06:37:27.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:26 smithi167 ceph-mon[32774]: pgmap v145: 148 pgs: 2 active+remapped+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 839 op/s; 22147/307350 objects misplaced (7.206%); 27 MiB/s, 425 objects/s recovering 2024-02-05T06:37:27.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:26 smithi167 ceph-mon[32774]: 5.1a scrub starts 2024-02-05T06:37:27.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:26 smithi167 ceph-mon[32774]: 5.1a scrub ok 2024-02-05T06:37:27.484 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:27.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:27.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:27.589 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:27.617 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:27.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:27.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:27.719 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:27.732 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:37:27.747 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:27.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:27.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:27.849 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:27.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:27.880 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:27.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:27.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:28.023 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:28.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:28.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:28.125 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:28.155 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:37:28.158 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:27 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:37:28.158 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:27 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:37:28.158 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:27 smithi028 ceph-mon[37984]: 3.15 scrub starts 2024-02-05T06:37:28.158 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:27 smithi028 ceph-mon[37984]: 3.15 scrub ok 2024-02-05T06:37:28.158 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:27 smithi028 ceph-mon[37984]: 4.5 scrub starts 2024-02-05T06:37:28.159 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:27 smithi028 ceph-mon[37984]: 4.5 scrub ok 2024-02-05T06:37:28.159 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:27 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:37:28.159 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:27 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:37:28.159 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:27 smithi028 ceph-mon[29442]: 3.15 scrub starts 2024-02-05T06:37:28.159 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:27 smithi028 ceph-mon[29442]: 3.15 scrub ok 2024-02-05T06:37:28.159 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:27 smithi028 ceph-mon[29442]: 4.5 scrub starts 2024-02-05T06:37:28.159 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:27 smithi028 ceph-mon[29442]: 4.5 scrub ok 2024-02-05T06:37:28.175 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:28.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:28.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:28.277 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:28.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:28.347 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:28.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:28.448 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:28.472 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:28.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:28.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:28.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:27 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:37:28.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:27 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:37:28.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:27 smithi167 ceph-mon[32774]: 3.15 scrub starts 2024-02-05T06:37:28.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:27 smithi167 ceph-mon[32774]: 3.15 scrub ok 2024-02-05T06:37:28.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:27 smithi167 ceph-mon[32774]: 4.5 scrub starts 2024-02-05T06:37:28.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:27 smithi167 ceph-mon[32774]: 4.5 scrub ok 2024-02-05T06:37:28.574 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:28.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:28.629 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:28.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:28.730 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:28.758 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:28.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:28.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:28.860 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:28.888 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:28.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:28.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:28.990 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:29.018 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:29.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:29.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:29.120 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:29.148 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:29.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:29.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:29.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:29 smithi028 ceph-mon[37984]: 3.c scrub starts 2024-02-05T06:37:29.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:29 smithi028 ceph-mon[37984]: 3.c scrub ok 2024-02-05T06:37:29.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:29 smithi028 ceph-mon[37984]: osdmap e127: 8 total, 8 up, 8 in 2024-02-05T06:37:29.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:29 smithi028 ceph-mon[37984]: pgmap v147: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 18919/298338 objects misplaced (6.341%); 35 MiB/s, 561 objects/s recovering 2024-02-05T06:37:29.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:29 smithi028 ceph-mon[37984]: 3.f scrub starts 2024-02-05T06:37:29.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:29 smithi028 ceph-mon[37984]: 3.f scrub ok 2024-02-05T06:37:29.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:29 smithi028 ceph-mon[29442]: 3.c scrub starts 2024-02-05T06:37:29.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:29 smithi028 ceph-mon[29442]: 3.c scrub ok 2024-02-05T06:37:29.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:29 smithi028 ceph-mon[29442]: osdmap e127: 8 total, 8 up, 8 in 2024-02-05T06:37:29.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:29 smithi028 ceph-mon[29442]: pgmap v147: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 18919/298338 objects misplaced (6.341%); 35 MiB/s, 561 objects/s recovering 2024-02-05T06:37:29.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:29 smithi028 ceph-mon[29442]: 3.f scrub starts 2024-02-05T06:37:29.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:29 smithi028 ceph-mon[29442]: 3.f scrub ok 2024-02-05T06:37:29.251 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:29.298 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:29.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:29.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:29.400 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:29.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:29.441 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:29.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:29.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:29 smithi167 ceph-mon[32774]: 3.c scrub starts 2024-02-05T06:37:29.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:29 smithi167 ceph-mon[32774]: 3.c scrub ok 2024-02-05T06:37:29.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:29 smithi167 ceph-mon[32774]: osdmap e127: 8 total, 8 up, 8 in 2024-02-05T06:37:29.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:29 smithi167 ceph-mon[32774]: pgmap v147: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 18919/298338 objects misplaced (6.341%); 35 MiB/s, 561 objects/s recovering 2024-02-05T06:37:29.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:29 smithi167 ceph-mon[32774]: 3.f scrub starts 2024-02-05T06:37:29.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:29 smithi167 ceph-mon[32774]: 3.f scrub ok 2024-02-05T06:37:29.542 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:29.574 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:29.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:29.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:29.676 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:29.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:29.768 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:29.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:29.869 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:29.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:29.901 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:29.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:30.003 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:30.033 INFO:teuthology.orchestra.run.smithi028.stderr:set pool 1 pg_num to 11 2024-02-05T06:37:30.054 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:30.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:30.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:30.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:30.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:30 smithi028 ceph-mon[29442]: 4.8 scrub starts 2024-02-05T06:37:30.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:30 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3353061879' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-02-05T06:37:30.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:30 smithi028 ceph-mon[29442]: from='client.24850 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-02-05T06:37:30.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:30 smithi028 ceph-mon[29442]: 4.8 scrub ok 2024-02-05T06:37:30.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:30 smithi028 ceph-mon[29442]: 7.e scrub starts 2024-02-05T06:37:30.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:30 smithi028 ceph-mon[29442]: 6.12 scrub starts 2024-02-05T06:37:30.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:30 smithi028 ceph-mon[29442]: 6.12 scrub ok 2024-02-05T06:37:30.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:30 smithi028 ceph-mon[29442]: 3.a scrub starts 2024-02-05T06:37:30.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:30 smithi028 ceph-mon[29442]: 3.a scrub ok 2024-02-05T06:37:30.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:30 smithi028 ceph-mon[37984]: 4.8 scrub starts 2024-02-05T06:37:30.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:30 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3353061879' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-02-05T06:37:30.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:30 smithi028 ceph-mon[37984]: from='client.24850 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-02-05T06:37:30.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:30 smithi028 ceph-mon[37984]: 4.8 scrub ok 2024-02-05T06:37:30.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:30 smithi028 ceph-mon[37984]: 7.e scrub starts 2024-02-05T06:37:30.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:30 smithi028 ceph-mon[37984]: 6.12 scrub starts 2024-02-05T06:37:30.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:30 smithi028 ceph-mon[37984]: 6.12 scrub ok 2024-02-05T06:37:30.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:30 smithi028 ceph-mon[37984]: 3.a scrub starts 2024-02-05T06:37:30.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:30 smithi028 ceph-mon[37984]: 3.a scrub ok 2024-02-05T06:37:30.206 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:30.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:30.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:30.307 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:30.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:30.338 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:30.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:30.439 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:30.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:30.480 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:30.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:30.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:30 smithi167 ceph-mon[32774]: 4.8 scrub starts 2024-02-05T06:37:30.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:30 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3353061879' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-02-05T06:37:30.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:30 smithi167 ceph-mon[32774]: from='client.24850 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-02-05T06:37:30.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:30 smithi167 ceph-mon[32774]: 4.8 scrub ok 2024-02-05T06:37:30.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:30 smithi167 ceph-mon[32774]: 7.e scrub starts 2024-02-05T06:37:30.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:30 smithi167 ceph-mon[32774]: 6.12 scrub starts 2024-02-05T06:37:30.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:30 smithi167 ceph-mon[32774]: 6.12 scrub ok 2024-02-05T06:37:30.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:30 smithi167 ceph-mon[32774]: 3.a scrub starts 2024-02-05T06:37:30.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:30 smithi167 ceph-mon[32774]: 3.a scrub ok 2024-02-05T06:37:30.581 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:30.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:30.630 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:30.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:30.731 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:30.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:30.778 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:30.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:30.879 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:30.909 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:30.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:30.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:31.011 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:31.036 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:31.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:31.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:31.137 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:31.166 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:31.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:31.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:31.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:31 smithi028 ceph-mon[29442]: from='client.24850 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-02-05T06:37:31.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:31 smithi028 ceph-mon[29442]: osdmap e128: 8 total, 8 up, 8 in 2024-02-05T06:37:31.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:31 smithi028 ceph-mon[29442]: pgmap v149: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 541 op/s; 18919/298338 objects misplaced (6.341%); 18 MiB/s, 292 objects/s recovering 2024-02-05T06:37:31.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:31 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-05T06:37:31.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:31 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-05T06:37:31.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:31 smithi028 ceph-mon[29442]: 4.1c scrub starts 2024-02-05T06:37:31.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:31 smithi028 ceph-mon[29442]: 4.1c scrub ok 2024-02-05T06:37:31.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:31 smithi028 ceph-mon[29442]: 3.d scrub starts 2024-02-05T06:37:31.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:31 smithi028 ceph-mon[29442]: 3.d scrub ok 2024-02-05T06:37:31.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:31 smithi028 ceph-mon[29442]: 7.7 scrub starts 2024-02-05T06:37:31.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:31 smithi028 ceph-mon[37984]: from='client.24850 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-02-05T06:37:31.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:31 smithi028 ceph-mon[37984]: osdmap e128: 8 total, 8 up, 8 in 2024-02-05T06:37:31.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:31 smithi028 ceph-mon[37984]: pgmap v149: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 541 op/s; 18919/298338 objects misplaced (6.341%); 18 MiB/s, 292 objects/s recovering 2024-02-05T06:37:31.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:31 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-05T06:37:31.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:31 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-05T06:37:31.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:31 smithi028 ceph-mon[37984]: 4.1c scrub starts 2024-02-05T06:37:31.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:31 smithi028 ceph-mon[37984]: 4.1c scrub ok 2024-02-05T06:37:31.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:31 smithi028 ceph-mon[37984]: 3.d scrub starts 2024-02-05T06:37:31.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:31 smithi028 ceph-mon[37984]: 3.d scrub ok 2024-02-05T06:37:31.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:31 smithi028 ceph-mon[37984]: 7.7 scrub starts 2024-02-05T06:37:31.268 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:31.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:31.317 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:31.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:31.418 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:31.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:31.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:31.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:31 smithi167 ceph-mon[32774]: from='client.24850 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-02-05T06:37:31.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:31 smithi167 ceph-mon[32774]: osdmap e128: 8 total, 8 up, 8 in 2024-02-05T06:37:31.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:31 smithi167 ceph-mon[32774]: pgmap v149: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait+backfill_toofull, 143 active+clean, 3 active+remapped+backfilling; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 541 op/s; 18919/298338 objects misplaced (6.341%); 18 MiB/s, 292 objects/s recovering 2024-02-05T06:37:31.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:31 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-05T06:37:31.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:31 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-02-05T06:37:31.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:31 smithi167 ceph-mon[32774]: 4.1c scrub starts 2024-02-05T06:37:31.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:31 smithi167 ceph-mon[32774]: 4.1c scrub ok 2024-02-05T06:37:31.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:31 smithi167 ceph-mon[32774]: 3.d scrub starts 2024-02-05T06:37:31.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:31 smithi167 ceph-mon[32774]: 3.d scrub ok 2024-02-05T06:37:31.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:31 smithi167 ceph-mon[32774]: 7.7 scrub starts 2024-02-05T06:37:31.562 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:31.608 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:31.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:31.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:31.710 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:31.740 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:31.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:31.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:31.842 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:31.874 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:31.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:31.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:31.976 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:32.008 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:32.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:32.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:32.110 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:32.155 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:32.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:32.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:32.257 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:32.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:32.291 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:32.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:32.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:32.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:32 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-02-05T06:37:32.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:32 smithi028 ceph-mon[29442]: osdmap e129: 8 total, 8 up, 8 in 2024-02-05T06:37:32.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:32 smithi028 ceph-mon[29442]: 2.1 scrub starts 2024-02-05T06:37:32.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:32 smithi028 ceph-mon[29442]: 2.1 scrub ok 2024-02-05T06:37:32.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:32 smithi028 ceph-mon[29442]: 5.4 scrub starts 2024-02-05T06:37:32.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:32 smithi028 ceph-mon[29442]: 5.4 scrub ok 2024-02-05T06:37:32.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:32 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-02-05T06:37:32.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:32 smithi028 ceph-mon[37984]: osdmap e129: 8 total, 8 up, 8 in 2024-02-05T06:37:32.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:32 smithi028 ceph-mon[37984]: 2.1 scrub starts 2024-02-05T06:37:32.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:32 smithi028 ceph-mon[37984]: 2.1 scrub ok 2024-02-05T06:37:32.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:32 smithi028 ceph-mon[37984]: 5.4 scrub starts 2024-02-05T06:37:32.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:32 smithi028 ceph-mon[37984]: 5.4 scrub ok 2024-02-05T06:37:32.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:32.453 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:32.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:32.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:32 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-02-05T06:37:32.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:32 smithi167 ceph-mon[32774]: osdmap e129: 8 total, 8 up, 8 in 2024-02-05T06:37:32.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:32 smithi167 ceph-mon[32774]: 2.1 scrub starts 2024-02-05T06:37:32.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:32 smithi167 ceph-mon[32774]: 2.1 scrub ok 2024-02-05T06:37:32.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:32 smithi167 ceph-mon[32774]: 5.4 scrub starts 2024-02-05T06:37:32.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:32 smithi167 ceph-mon[32774]: 5.4 scrub ok 2024-02-05T06:37:32.554 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:32.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:32.607 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:32.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:32.694 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:37:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:37:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:37:32.708 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:32.739 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:32.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:32.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:32.841 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:32.871 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:32.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:32.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:32.973 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:32.980 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:37:32 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:37:33.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:33.006 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:33.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:33.108 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:33.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:33.149 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:33.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:33.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:33.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:33.295 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:33.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:33.396 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:33.428 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:33.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:33.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:33.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:33 smithi028 ceph-mon[37984]: 7.e scrub ok 2024-02-05T06:37:33.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:33 smithi028 ceph-mon[37984]: osdmap e130: 8 total, 8 up, 8 in 2024-02-05T06:37:33.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:33 smithi028 ceph-mon[37984]: pgmap v152: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+clean+scrubbing, 3 active+remapped+backfilling, 10 unknown, 140 active+clean, 1 peering, 1 active+remapped+backfill_wait+backfill_toofull; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 15975/285561 objects misplaced (5.594%); 33 MiB/s, 523 objects/s recovering 2024-02-05T06:37:33.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:33 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-05T06:37:33.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:33 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-05T06:37:33.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:33 smithi028 ceph-mon[37984]: 5.6 scrub starts 2024-02-05T06:37:33.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:33 smithi028 ceph-mon[37984]: 5.6 scrub ok 2024-02-05T06:37:33.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:33 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:37:33.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:33 smithi028 ceph-mon[37984]: 1.0 scrub starts 2024-02-05T06:37:33.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:33 smithi028 ceph-mon[37984]: 1.0 scrub ok 2024-02-05T06:37:33.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:33 smithi028 ceph-mon[37984]: 7.7 scrub ok 2024-02-05T06:37:33.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:33 smithi028 ceph-mon[37984]: 7.c scrub starts 2024-02-05T06:37:33.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:33 smithi028 ceph-mon[29442]: 7.e scrub ok 2024-02-05T06:37:33.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:33 smithi028 ceph-mon[29442]: osdmap e130: 8 total, 8 up, 8 in 2024-02-05T06:37:33.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:33 smithi028 ceph-mon[29442]: pgmap v152: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+clean+scrubbing, 3 active+remapped+backfilling, 10 unknown, 140 active+clean, 1 peering, 1 active+remapped+backfill_wait+backfill_toofull; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 15975/285561 objects misplaced (5.594%); 33 MiB/s, 523 objects/s recovering 2024-02-05T06:37:33.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:33 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-05T06:37:33.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:33 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-05T06:37:33.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:33 smithi028 ceph-mon[29442]: 5.6 scrub starts 2024-02-05T06:37:33.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:33 smithi028 ceph-mon[29442]: 5.6 scrub ok 2024-02-05T06:37:33.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:33 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:37:33.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:33 smithi028 ceph-mon[29442]: 1.0 scrub starts 2024-02-05T06:37:33.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:33 smithi028 ceph-mon[29442]: 1.0 scrub ok 2024-02-05T06:37:33.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:33 smithi028 ceph-mon[29442]: 7.7 scrub ok 2024-02-05T06:37:33.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:33 smithi028 ceph-mon[29442]: 7.c scrub starts 2024-02-05T06:37:33.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:33 smithi167 ceph-mon[32774]: 7.e scrub ok 2024-02-05T06:37:33.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:33 smithi167 ceph-mon[32774]: osdmap e130: 8 total, 8 up, 8 in 2024-02-05T06:37:33.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:33 smithi167 ceph-mon[32774]: pgmap v152: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+clean+scrubbing, 3 active+remapped+backfilling, 10 unknown, 140 active+clean, 1 peering, 1 active+remapped+backfill_wait+backfill_toofull; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s; 15975/285561 objects misplaced (5.594%); 33 MiB/s, 523 objects/s recovering 2024-02-05T06:37:33.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:33 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-05T06:37:33.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:33 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-02-05T06:37:33.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:33 smithi167 ceph-mon[32774]: 5.6 scrub starts 2024-02-05T06:37:33.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:33 smithi167 ceph-mon[32774]: 5.6 scrub ok 2024-02-05T06:37:33.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:33 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:37:33.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:33 smithi167 ceph-mon[32774]: 1.0 scrub starts 2024-02-05T06:37:33.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:33 smithi167 ceph-mon[32774]: 1.0 scrub ok 2024-02-05T06:37:33.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:33 smithi167 ceph-mon[32774]: 7.7 scrub ok 2024-02-05T06:37:33.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:33 smithi167 ceph-mon[32774]: 7.c scrub starts 2024-02-05T06:37:33.530 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:33.564 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:33.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:33.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:33.666 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:33.699 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:33.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:33.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:33.801 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:33.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:33.829 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:33.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:33.930 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:33.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:33.964 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:33.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:34.065 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:34.103 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:34.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:34.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:34.104 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-02-05T06:37:34.205 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:34.241 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:34.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:34.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:34.343 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:34.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:34.375 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:34.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:34.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:34 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-02-05T06:37:34.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:34 smithi028 ceph-mon[29442]: osdmap e131: 8 total, 8 up, 8 in 2024-02-05T06:37:34.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:34 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3070022296' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:37:34.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:34 smithi028 ceph-mon[29442]: from='client.44849 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:37:34.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:34 smithi028 ceph-mon[29442]: 7.d scrub starts 2024-02-05T06:37:34.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:34 smithi028 ceph-mon[29442]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:37:34.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:34 smithi028 ceph-mon[29442]: from='client.44849 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:37:34.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:34 smithi028 ceph-mon[29442]: osdmap e132: 8 total, 8 up, 8 in 2024-02-05T06:37:34.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:34 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-02-05T06:37:34.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:34 smithi028 ceph-mon[37984]: osdmap e131: 8 total, 8 up, 8 in 2024-02-05T06:37:34.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:34 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3070022296' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:37:34.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:34 smithi028 ceph-mon[37984]: from='client.44849 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:37:34.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:34 smithi028 ceph-mon[37984]: 7.d scrub starts 2024-02-05T06:37:34.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:34 smithi028 ceph-mon[37984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:37:34.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:34 smithi028 ceph-mon[37984]: from='client.44849 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:37:34.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:34 smithi028 ceph-mon[37984]: osdmap e132: 8 total, 8 up, 8 in 2024-02-05T06:37:34.476 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:34.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:34 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-02-05T06:37:34.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:34 smithi167 ceph-mon[32774]: osdmap e131: 8 total, 8 up, 8 in 2024-02-05T06:37:34.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:34 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3070022296' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:37:34.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:34 smithi167 ceph-mon[32774]: from='client.44849 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:37:34.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:34 smithi167 ceph-mon[32774]: 7.d scrub starts 2024-02-05T06:37:34.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:34 smithi167 ceph-mon[32774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:37:34.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:34 smithi167 ceph-mon[32774]: from='client.44849 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:37:34.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:34 smithi167 ceph-mon[32774]: osdmap e132: 8 total, 8 up, 8 in 2024-02-05T06:37:34.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:34.509 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:34.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:34.610 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:34.657 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:34.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:34.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:34.759 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:34.788 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:34.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:34.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:34.875 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:37:34.875 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-02-05T06:37:34.876 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-02-05T06:37:34.876 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-02-05T06:37:34.876 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:37:34.889 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:34.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:34.933 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:34.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:35.034 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:35.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:35.067 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:35.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:35.153 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:35 smithi028 ceph-mon[29442]: pgmap v155: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+clean+scrubbing, 3 active+remapped+backfilling, 10 unknown, 140 active+clean, 1 peering, 1 active+remapped+backfill_wait+backfill_toofull; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 15975/285561 objects misplaced (5.594%); 34 MiB/s, 550 objects/s recovering 2024-02-05T06:37:35.153 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:35 smithi028 ceph-mon[37984]: pgmap v155: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+clean+scrubbing, 3 active+remapped+backfilling, 10 unknown, 140 active+clean, 1 peering, 1 active+remapped+backfill_wait+backfill_toofull; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 15975/285561 objects misplaced (5.594%); 34 MiB/s, 550 objects/s recovering 2024-02-05T06:37:35.168 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:35.211 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:35.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:35.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:35.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:35.355 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:35.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:35.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:35.380 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:35 smithi167 ceph-mon[32774]: pgmap v155: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+clean+scrubbing, 3 active+remapped+backfilling, 10 unknown, 140 active+clean, 1 peering, 1 active+remapped+backfill_wait+backfill_toofull; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 15975/285561 objects misplaced (5.594%); 34 MiB/s, 550 objects/s recovering 2024-02-05T06:37:35.381 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:35 smithi167 ceph-mon[32774]: 7.c scrub ok 2024-02-05T06:37:35.381 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:35 smithi167 ceph-mon[32774]: 3.13 scrub starts 2024-02-05T06:37:35.381 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:35 smithi167 ceph-mon[32774]: 3.13 scrub ok 2024-02-05T06:37:35.381 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:35 smithi167 ceph-mon[32774]: osdmap e133: 8 total, 8 up, 8 in 2024-02-05T06:37:35.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:35 smithi028 ceph-mon[29442]: 7.c scrub ok 2024-02-05T06:37:35.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:35 smithi028 ceph-mon[29442]: 3.13 scrub starts 2024-02-05T06:37:35.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:35 smithi028 ceph-mon[29442]: 3.13 scrub ok 2024-02-05T06:37:35.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:35 smithi028 ceph-mon[29442]: osdmap e133: 8 total, 8 up, 8 in 2024-02-05T06:37:35.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:35 smithi028 ceph-mon[37984]: 7.c scrub ok 2024-02-05T06:37:35.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:35 smithi028 ceph-mon[37984]: 3.13 scrub starts 2024-02-05T06:37:35.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:35 smithi028 ceph-mon[37984]: 3.13 scrub ok 2024-02-05T06:37:35.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:35 smithi028 ceph-mon[37984]: osdmap e133: 8 total, 8 up, 8 in 2024-02-05T06:37:35.457 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:35.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:35.493 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:35.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:35.594 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:35.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:35.628 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:35.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:35.730 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:35.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:35.762 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:35.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:35.863 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:35.873 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T06:37:35.873 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T06:37:35.873 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-02-05T06:37:35.874 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T06:37:35.894 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:35.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:35.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:35.991 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-05T06:37:35.996 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:36.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:36.033 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:36.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:36.134 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:37:36.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:36 smithi028 ceph-mon[29442]: 7.d scrub ok 2024-02-05T06:37:36.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:36 smithi028 ceph-mon[29442]: 6.8 scrub starts 2024-02-05T06:37:36.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:36 smithi028 ceph-mon[29442]: 6.8 scrub ok 2024-02-05T06:37:36.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:36 smithi028 ceph-mon[29442]: 3.10 scrub starts 2024-02-05T06:37:36.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:36 smithi028 ceph-mon[29442]: 3.10 scrub ok 2024-02-05T06:37:36.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:36 smithi028 ceph-mon[37984]: 7.d scrub ok 2024-02-05T06:37:36.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:36 smithi028 ceph-mon[37984]: 6.8 scrub starts 2024-02-05T06:37:36.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:36 smithi028 ceph-mon[37984]: 6.8 scrub ok 2024-02-05T06:37:36.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:36 smithi028 ceph-mon[37984]: 3.10 scrub starts 2024-02-05T06:37:36.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:36 smithi028 ceph-mon[37984]: 3.10 scrub ok 2024-02-05T06:37:36.222 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:36.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:36.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:36.324 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:36.358 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:36.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:36.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:36.460 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:36.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:36 smithi167 ceph-mon[32774]: 7.d scrub ok 2024-02-05T06:37:36.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:36 smithi167 ceph-mon[32774]: 6.8 scrub starts 2024-02-05T06:37:36.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:36 smithi167 ceph-mon[32774]: 6.8 scrub ok 2024-02-05T06:37:36.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:36 smithi167 ceph-mon[32774]: 3.10 scrub starts 2024-02-05T06:37:36.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:36 smithi167 ceph-mon[32774]: 3.10 scrub ok 2024-02-05T06:37:36.503 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:36.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:36.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:36.605 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:36.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:36.644 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:36.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:36.745 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:36.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:36.805 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:36.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:36.906 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:36.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:36.982 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:36.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:37.083 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:37.114 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:37.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:37.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:37.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:37 smithi028 ceph-mon[29442]: osdmap e134: 8 total, 8 up, 8 in 2024-02-05T06:37:37.195 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:37 smithi028 ceph-mon[29442]: pgmap v158: 158 pgs: 8 peering, 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 3 active+remapped+backfilling, 2 unknown, 142 active+clean, 1 active+remapped+backfill_wait+backfill_toofull; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 327 op/s; 15975/281586 objects misplaced (5.673%); 25 KiB/s, 0 objects/s recovering 2024-02-05T06:37:37.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:37 smithi028 ceph-mon[29442]: 5.11 scrub starts 2024-02-05T06:37:37.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:37 smithi028 ceph-mon[29442]: 5.11 scrub ok 2024-02-05T06:37:37.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:37 smithi028 ceph-mon[37984]: osdmap e134: 8 total, 8 up, 8 in 2024-02-05T06:37:37.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:37 smithi028 ceph-mon[37984]: pgmap v158: 158 pgs: 8 peering, 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 3 active+remapped+backfilling, 2 unknown, 142 active+clean, 1 active+remapped+backfill_wait+backfill_toofull; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 327 op/s; 15975/281586 objects misplaced (5.673%); 25 KiB/s, 0 objects/s recovering 2024-02-05T06:37:37.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:37 smithi028 ceph-mon[37984]: 5.11 scrub starts 2024-02-05T06:37:37.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:37 smithi028 ceph-mon[37984]: 5.11 scrub ok 2024-02-05T06:37:37.215 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:37.261 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:37.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:37.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:37.363 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:37.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:37.405 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:37.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:37.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:37 smithi167 ceph-mon[32774]: osdmap e134: 8 total, 8 up, 8 in 2024-02-05T06:37:37.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:37 smithi167 ceph-mon[32774]: pgmap v158: 158 pgs: 8 peering, 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 3 active+remapped+backfilling, 2 unknown, 142 active+clean, 1 active+remapped+backfill_wait+backfill_toofull; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 0 B/s wr, 327 op/s; 15975/281586 objects misplaced (5.673%); 25 KiB/s, 0 objects/s recovering 2024-02-05T06:37:37.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:37 smithi167 ceph-mon[32774]: 5.11 scrub starts 2024-02-05T06:37:37.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:37 smithi167 ceph-mon[32774]: 5.11 scrub ok 2024-02-05T06:37:37.506 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:37.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:37.551 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:37.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:37.652 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:37.682 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:37.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:37.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:37.784 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:37.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:37.829 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:37.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:37.930 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:37.972 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:37.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:37.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:38.075 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:38.115 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:38.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:38.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:38.217 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:38.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:38.322 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:38.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:38.423 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:38.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:38 smithi028 ceph-mon[29442]: osdmap e135: 8 total, 8 up, 8 in 2024-02-05T06:37:38.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:38 smithi028 ceph-mon[29442]: 1.1 deep-scrub starts 2024-02-05T06:37:38.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:38 smithi028 ceph-mon[29442]: 1.1 deep-scrub ok 2024-02-05T06:37:38.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:38 smithi028 ceph-mon[29442]: 1.9 scrub starts 2024-02-05T06:37:38.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:38 smithi028 ceph-mon[29442]: 1.9 scrub ok 2024-02-05T06:37:38.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:38 smithi028 ceph-mon[29442]: 1.7 scrub starts 2024-02-05T06:37:38.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:38 smithi028 ceph-mon[29442]: 1.7 scrub ok 2024-02-05T06:37:38.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:38 smithi028 ceph-mon[37984]: osdmap e135: 8 total, 8 up, 8 in 2024-02-05T06:37:38.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:38 smithi028 ceph-mon[37984]: 1.1 deep-scrub starts 2024-02-05T06:37:38.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:38 smithi028 ceph-mon[37984]: 1.1 deep-scrub ok 2024-02-05T06:37:38.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:38 smithi028 ceph-mon[37984]: 1.9 scrub starts 2024-02-05T06:37:38.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:38 smithi028 ceph-mon[37984]: 1.9 scrub ok 2024-02-05T06:37:38.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:38 smithi028 ceph-mon[37984]: 1.7 scrub starts 2024-02-05T06:37:38.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:38 smithi028 ceph-mon[37984]: 1.7 scrub ok 2024-02-05T06:37:38.461 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:38.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:38.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:38.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:38 smithi167 ceph-mon[32774]: osdmap e135: 8 total, 8 up, 8 in 2024-02-05T06:37:38.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:38 smithi167 ceph-mon[32774]: 1.1 deep-scrub starts 2024-02-05T06:37:38.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:38 smithi167 ceph-mon[32774]: 1.1 deep-scrub ok 2024-02-05T06:37:38.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:38 smithi167 ceph-mon[32774]: 1.9 scrub starts 2024-02-05T06:37:38.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:38 smithi167 ceph-mon[32774]: 1.9 scrub ok 2024-02-05T06:37:38.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:38 smithi167 ceph-mon[32774]: 1.7 scrub starts 2024-02-05T06:37:38.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:38 smithi167 ceph-mon[32774]: 1.7 scrub ok 2024-02-05T06:37:38.563 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:38.590 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:38.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:38.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:38.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T06:37:38.692 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:38.732 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:38.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:38.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:38.834 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:38.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:38.886 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:38.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:38.942 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:37:38.987 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:39.048 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:37:39.049 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:39.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:39.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:39.151 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:39.180 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:39.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:39.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:39.281 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:39.341 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:39.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:39.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:39.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:39 smithi028 ceph-mon[29442]: pgmap v160: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 10 peering, 3 active+remapped+backfilling, 143 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 13217/272592 objects misplaced (4.849%); 31 MiB/s, 498 objects/s recovering 2024-02-05T06:37:39.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:39 smithi028 ceph-mon[29442]: 1.2 scrub starts 2024-02-05T06:37:39.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:39 smithi028 ceph-mon[29442]: 1.2 scrub ok 2024-02-05T06:37:39.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:39 smithi028 ceph-mon[29442]: 1.6 deep-scrub starts 2024-02-05T06:37:39.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:39 smithi028 ceph-mon[29442]: 1.6 deep-scrub ok 2024-02-05T06:37:39.375 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:39 smithi028 ceph-mon[37984]: pgmap v160: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 10 peering, 3 active+remapped+backfilling, 143 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 13217/272592 objects misplaced (4.849%); 31 MiB/s, 498 objects/s recovering 2024-02-05T06:37:39.375 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:39 smithi028 ceph-mon[37984]: 1.2 scrub starts 2024-02-05T06:37:39.375 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:39 smithi028 ceph-mon[37984]: 1.2 scrub ok 2024-02-05T06:37:39.375 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:39 smithi028 ceph-mon[37984]: 1.6 deep-scrub starts 2024-02-05T06:37:39.376 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:39 smithi028 ceph-mon[37984]: 1.6 deep-scrub ok 2024-02-05T06:37:39.443 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:39.471 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:39.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:39.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:39.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:39 smithi167 ceph-mon[32774]: pgmap v160: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 10 peering, 3 active+remapped+backfilling, 143 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 1.04k op/s; 13217/272592 objects misplaced (4.849%); 31 MiB/s, 498 objects/s recovering 2024-02-05T06:37:39.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:39 smithi167 ceph-mon[32774]: 1.2 scrub starts 2024-02-05T06:37:39.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:39 smithi167 ceph-mon[32774]: 1.2 scrub ok 2024-02-05T06:37:39.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:39 smithi167 ceph-mon[32774]: 1.6 deep-scrub starts 2024-02-05T06:37:39.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:39 smithi167 ceph-mon[32774]: 1.6 deep-scrub ok 2024-02-05T06:37:39.573 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:39.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:39.605 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:39.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:39.706 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:39.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:39.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:39.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:39.910 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:39.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:39.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:40.011 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:40.040 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:40.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:40.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:40.142 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:40.181 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:40.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:40.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:40.283 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:40.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:40.314 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:40.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:40.415 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:40.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:40 smithi028 ceph-mon[37984]: 1.a scrub starts 2024-02-05T06:37:40.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:40 smithi028 ceph-mon[37984]: 1.a scrub ok 2024-02-05T06:37:40.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:40 smithi028 ceph-mon[37984]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:37:40.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:40 smithi028 ceph-mon[37984]: osdmap e136: 8 total, 8 up, 8 in 2024-02-05T06:37:40.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:40 smithi028 ceph-mon[29442]: 1.a scrub starts 2024-02-05T06:37:40.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:40 smithi028 ceph-mon[29442]: 1.a scrub ok 2024-02-05T06:37:40.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:40 smithi028 ceph-mon[29442]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:37:40.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:40 smithi028 ceph-mon[29442]: osdmap e136: 8 total, 8 up, 8 in 2024-02-05T06:37:40.446 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:40.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:40.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:40.453 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:37:40.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:40 smithi167 ceph-mon[32774]: 1.a scrub starts 2024-02-05T06:37:40.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:40 smithi167 ceph-mon[32774]: 1.a scrub ok 2024-02-05T06:37:40.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:40 smithi167 ceph-mon[32774]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:37:40.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:40 smithi167 ceph-mon[32774]: osdmap e136: 8 total, 8 up, 8 in 2024-02-05T06:37:40.516 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:37:40.540 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:37:40.548 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:40.565 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:37:40.573 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:40.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:40.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:40.591 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:37:40.622 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:37:40.647 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:37:40.674 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:37:40.675 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:40.699 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:37:40.700 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:40.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:40.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:40.701 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:37:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:37:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:37:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:37:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:37:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:37:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:37:40.702 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:37:40.801 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:40.831 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:40.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:40.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:40.932 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:40.960 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:40.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:40.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:41.062 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:41.093 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:41.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:41.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:41.195 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:41.220 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:41.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:41.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:41.324 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:41.351 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:41.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:41.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:41.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:41 smithi028 ceph-mon[29442]: 1.8 scrub starts 2024-02-05T06:37:41.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:41 smithi028 ceph-mon[29442]: 1.8 scrub ok 2024-02-05T06:37:41.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:41 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:37:41.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:41 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:37:41.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:41 smithi028 ceph-mon[29442]: pgmap v162: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 10 peering, 3 active+remapped+backfilling, 143 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 838 op/s; 13217/272592 objects misplaced (4.849%); 25 MiB/s, 401 objects/s recovering 2024-02-05T06:37:41.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:41 smithi028 ceph-mon[29442]: 5.7 scrub starts 2024-02-05T06:37:41.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:41 smithi028 ceph-mon[29442]: 5.7 scrub ok 2024-02-05T06:37:41.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:41 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:37:41.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:41 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/308724572' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:37:41.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:41 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:37:41.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:41 smithi028 ceph-mon[29442]: osdmap e137: 8 total, 8 up, 8 in 2024-02-05T06:37:41.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:41 smithi028 ceph-mon[37984]: 1.8 scrub starts 2024-02-05T06:37:41.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:41 smithi028 ceph-mon[37984]: 1.8 scrub ok 2024-02-05T06:37:41.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:41 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:37:41.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:41 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:37:41.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:41 smithi028 ceph-mon[37984]: pgmap v162: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 10 peering, 3 active+remapped+backfilling, 143 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 838 op/s; 13217/272592 objects misplaced (4.849%); 25 MiB/s, 401 objects/s recovering 2024-02-05T06:37:41.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:41 smithi028 ceph-mon[37984]: 5.7 scrub starts 2024-02-05T06:37:41.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:41 smithi028 ceph-mon[37984]: 5.7 scrub ok 2024-02-05T06:37:41.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:41 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:37:41.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:41 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/308724572' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:37:41.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:41 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:37:41.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:41 smithi028 ceph-mon[37984]: osdmap e137: 8 total, 8 up, 8 in 2024-02-05T06:37:41.454 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:41.478 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:41.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:41.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:41.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:41 smithi167 ceph-mon[32774]: 1.8 scrub starts 2024-02-05T06:37:41.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:41 smithi167 ceph-mon[32774]: 1.8 scrub ok 2024-02-05T06:37:41.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:41 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:37:41.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:41 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:37:41.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:41 smithi167 ceph-mon[32774]: pgmap v162: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfill_wait, 10 peering, 3 active+remapped+backfilling, 143 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 838 op/s; 13217/272592 objects misplaced (4.849%); 25 MiB/s, 401 objects/s recovering 2024-02-05T06:37:41.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:41 smithi167 ceph-mon[32774]: 5.7 scrub starts 2024-02-05T06:37:41.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:41 smithi167 ceph-mon[32774]: 5.7 scrub ok 2024-02-05T06:37:41.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:41 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:37:41.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:41 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/308724572' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:37:41.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:41 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:37:41.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:41 smithi167 ceph-mon[32774]: osdmap e137: 8 total, 8 up, 8 in 2024-02-05T06:37:41.579 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:41.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:41.605 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:41.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:41.706 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:41.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:41.733 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:41.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:41.834 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:41.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:41.859 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:41.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:41.960 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:41.990 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:41.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:41.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:42.092 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:42.123 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:42.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:42.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:42.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:42.253 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:42.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:42.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:42.355 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:42.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:42.380 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:42.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:42.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:42 smithi028 ceph-mon[29442]: 5.1d scrub starts 2024-02-05T06:37:42.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:42 smithi028 ceph-mon[29442]: 5.1d scrub ok 2024-02-05T06:37:42.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:42 smithi028 ceph-mon[29442]: 6.10 scrub starts 2024-02-05T06:37:42.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:42 smithi028 ceph-mon[29442]: 6.10 scrub ok 2024-02-05T06:37:42.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:42 smithi028 ceph-mon[29442]: 5.a scrub starts 2024-02-05T06:37:42.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:42 smithi028 ceph-mon[29442]: 5.a scrub ok 2024-02-05T06:37:42.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:42 smithi028 ceph-mon[29442]: 6.1c scrub starts 2024-02-05T06:37:42.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:42 smithi028 ceph-mon[29442]: 6.1c scrub ok 2024-02-05T06:37:42.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:42 smithi028 ceph-mon[29442]: 4.1f scrub starts 2024-02-05T06:37:42.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:42 smithi028 ceph-mon[29442]: 4.1f scrub ok 2024-02-05T06:37:42.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:42 smithi028 ceph-mon[29442]: osdmap e138: 8 total, 8 up, 8 in 2024-02-05T06:37:42.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:42 smithi028 ceph-mon[37984]: 5.1d scrub starts 2024-02-05T06:37:42.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:42 smithi028 ceph-mon[37984]: 5.1d scrub ok 2024-02-05T06:37:42.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:42 smithi028 ceph-mon[37984]: 6.10 scrub starts 2024-02-05T06:37:42.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:42 smithi028 ceph-mon[37984]: 6.10 scrub ok 2024-02-05T06:37:42.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:42 smithi028 ceph-mon[37984]: 5.a scrub starts 2024-02-05T06:37:42.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:42 smithi028 ceph-mon[37984]: 5.a scrub ok 2024-02-05T06:37:42.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:42 smithi028 ceph-mon[37984]: 6.1c scrub starts 2024-02-05T06:37:42.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:42 smithi028 ceph-mon[37984]: 6.1c scrub ok 2024-02-05T06:37:42.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:42 smithi028 ceph-mon[37984]: 4.1f scrub starts 2024-02-05T06:37:42.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:42 smithi028 ceph-mon[37984]: 4.1f scrub ok 2024-02-05T06:37:42.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:42 smithi028 ceph-mon[37984]: osdmap e138: 8 total, 8 up, 8 in 2024-02-05T06:37:42.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:42 smithi167 ceph-mon[32774]: 5.1d scrub starts 2024-02-05T06:37:42.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:42 smithi167 ceph-mon[32774]: 5.1d scrub ok 2024-02-05T06:37:42.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:42 smithi167 ceph-mon[32774]: 6.10 scrub starts 2024-02-05T06:37:42.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:42 smithi167 ceph-mon[32774]: 6.10 scrub ok 2024-02-05T06:37:42.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:42 smithi167 ceph-mon[32774]: 5.a scrub starts 2024-02-05T06:37:42.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:42 smithi167 ceph-mon[32774]: 5.a scrub ok 2024-02-05T06:37:42.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:42 smithi167 ceph-mon[32774]: 6.1c scrub starts 2024-02-05T06:37:42.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:42 smithi167 ceph-mon[32774]: 6.1c scrub ok 2024-02-05T06:37:42.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:42 smithi167 ceph-mon[32774]: 4.1f scrub starts 2024-02-05T06:37:42.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:42 smithi167 ceph-mon[32774]: 4.1f scrub ok 2024-02-05T06:37:42.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:42 smithi167 ceph-mon[32774]: osdmap e138: 8 total, 8 up, 8 in 2024-02-05T06:37:42.481 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:42.509 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:42.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:42.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:42.611 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:42.636 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:42.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:42.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:42.738 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:42.766 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:37:42 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:37:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:37:42.769 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:42.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:42.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:42.871 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:42.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:42.898 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:42.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:42.980 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:37:42 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:37:42.999 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:43.030 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:43.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:43.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:43.132 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:43.163 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:43.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:43.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:43.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:43.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:43.312 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:43.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:43.413 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:43.445 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:43.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:43 smithi028 ceph-mon[29442]: 3.0 scrub starts 2024-02-05T06:37:43.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:43 smithi028 ceph-mon[29442]: 3.0 scrub ok 2024-02-05T06:37:43.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:43 smithi028 ceph-mon[29442]: pgmap v165: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 active+remapped+backfilling, 154 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 746 op/s; 9528/260823 objects misplaced (3.653%); 38 MiB/s, 618 objects/s recovering 2024-02-05T06:37:43.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:43 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:37:43.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:43 smithi028 ceph-mon[29442]: 7.5 deep-scrub starts 2024-02-05T06:37:43.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:43 smithi028 ceph-mon[29442]: 4.a scrub starts 2024-02-05T06:37:43.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:43 smithi028 ceph-mon[29442]: 4.a scrub ok 2024-02-05T06:37:43.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:43 smithi028 ceph-mon[29442]: osdmap e139: 8 total, 8 up, 8 in 2024-02-05T06:37:43.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:43 smithi028 ceph-mon[37984]: 3.0 scrub starts 2024-02-05T06:37:43.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:43 smithi028 ceph-mon[37984]: 3.0 scrub ok 2024-02-05T06:37:43.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:43 smithi028 ceph-mon[37984]: pgmap v165: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 active+remapped+backfilling, 154 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 746 op/s; 9528/260823 objects misplaced (3.653%); 38 MiB/s, 618 objects/s recovering 2024-02-05T06:37:43.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:43 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:37:43.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:43 smithi028 ceph-mon[37984]: 7.5 deep-scrub starts 2024-02-05T06:37:43.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:43 smithi028 ceph-mon[37984]: 4.a scrub starts 2024-02-05T06:37:43.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:43 smithi028 ceph-mon[37984]: 4.a scrub ok 2024-02-05T06:37:43.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:43 smithi028 ceph-mon[37984]: osdmap e139: 8 total, 8 up, 8 in 2024-02-05T06:37:43.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:43.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:43.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:43 smithi167 ceph-mon[32774]: 3.0 scrub starts 2024-02-05T06:37:43.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:43 smithi167 ceph-mon[32774]: 3.0 scrub ok 2024-02-05T06:37:43.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:43 smithi167 ceph-mon[32774]: pgmap v165: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 active+remapped+backfilling, 154 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 746 op/s; 9528/260823 objects misplaced (3.653%); 38 MiB/s, 618 objects/s recovering 2024-02-05T06:37:43.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:43 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:37:43.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:43 smithi167 ceph-mon[32774]: 7.5 deep-scrub starts 2024-02-05T06:37:43.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:43 smithi167 ceph-mon[32774]: 4.a scrub starts 2024-02-05T06:37:43.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:43 smithi167 ceph-mon[32774]: 4.a scrub ok 2024-02-05T06:37:43.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:43 smithi167 ceph-mon[32774]: osdmap e139: 8 total, 8 up, 8 in 2024-02-05T06:37:43.551 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:43.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:43.591 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:43.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:43.693 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:43.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:43.740 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:43.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:43.841 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:43.871 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:43.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:43.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:43.973 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:44.003 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:44.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:44.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:44.105 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:44.132 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:44.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:44.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:44.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:44.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:44.279 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:44.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:44.380 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:44.408 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:44.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:44.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:44.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:44 smithi028 ceph-mon[29442]: 1.3 scrub starts 2024-02-05T06:37:44.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:44 smithi028 ceph-mon[29442]: 1.3 scrub ok 2024-02-05T06:37:44.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:44 smithi028 ceph-mon[29442]: 3.16 scrub starts 2024-02-05T06:37:44.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:44 smithi028 ceph-mon[29442]: 3.16 scrub ok 2024-02-05T06:37:44.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:44 smithi028 ceph-mon[29442]: 5.10 scrub starts 2024-02-05T06:37:44.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:44 smithi028 ceph-mon[29442]: 5.10 scrub ok 2024-02-05T06:37:44.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:44 smithi028 ceph-mon[37984]: 1.3 scrub starts 2024-02-05T06:37:44.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:44 smithi028 ceph-mon[37984]: 1.3 scrub ok 2024-02-05T06:37:44.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:44 smithi028 ceph-mon[37984]: 3.16 scrub starts 2024-02-05T06:37:44.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:44 smithi028 ceph-mon[37984]: 3.16 scrub ok 2024-02-05T06:37:44.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:44 smithi028 ceph-mon[37984]: 5.10 scrub starts 2024-02-05T06:37:44.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:44 smithi028 ceph-mon[37984]: 5.10 scrub ok 2024-02-05T06:37:44.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:44 smithi167 ceph-mon[32774]: 1.3 scrub starts 2024-02-05T06:37:44.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:44 smithi167 ceph-mon[32774]: 1.3 scrub ok 2024-02-05T06:37:44.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:44 smithi167 ceph-mon[32774]: 3.16 scrub starts 2024-02-05T06:37:44.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:44 smithi167 ceph-mon[32774]: 3.16 scrub ok 2024-02-05T06:37:44.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:44 smithi167 ceph-mon[32774]: 5.10 scrub starts 2024-02-05T06:37:44.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:44 smithi167 ceph-mon[32774]: 5.10 scrub ok 2024-02-05T06:37:44.510 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:44.539 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:44.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:44.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:44.641 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:44.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:44.703 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:44.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:44.804 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:44.835 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:44.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:44.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:44.937 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:44.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:44.964 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:44.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:45.065 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:45.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:45.109 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:45.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:45.210 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:45.250 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:45.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:45.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:45.352 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:45.385 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:45.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:45.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:45.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:45 smithi167 ceph-mon[32774]: pgmap v167: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 active+remapped+backfilling, 154 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 892 op/s; 9528/260823 objects misplaced (3.653%); 46 MiB/s, 739 objects/s recovering 2024-02-05T06:37:45.488 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:45.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:45.514 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:45.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:45.615 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:45.634 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:45 smithi028 ceph-mon[29442]: pgmap v167: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 active+remapped+backfilling, 154 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 892 op/s; 9528/260823 objects misplaced (3.653%); 46 MiB/s, 739 objects/s recovering 2024-02-05T06:37:45.634 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:45 smithi028 ceph-mon[37984]: pgmap v167: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 active+remapped+backfilling, 154 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 0 B/s wr, 892 op/s; 9528/260823 objects misplaced (3.653%); 46 MiB/s, 739 objects/s recovering 2024-02-05T06:37:45.645 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:45.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:45.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:45.747 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:45.783 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:45.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:45.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:45.885 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:45.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:45.929 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:45.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:46.030 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:46.075 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:46.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:46.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:46.177 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:46.201 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:46.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:46.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:46.303 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:46.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:46.336 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:46.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:46.437 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:46.469 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:46.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:46.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:46.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:46 smithi167 ceph-mon[32774]: 7.b scrub starts 2024-02-05T06:37:46.571 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:46.613 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:46.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:46.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:46.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:46 smithi028 ceph-mon[29442]: 7.b scrub starts 2024-02-05T06:37:46.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:46 smithi028 ceph-mon[37984]: 7.b scrub starts 2024-02-05T06:37:46.715 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:46.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:46.744 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:46.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:46.845 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:46.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:46.886 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:46.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:46.987 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:47.026 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:47.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:47.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:47.128 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:47.172 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:47.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:47.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:47.246 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:47 smithi028 ceph-mon[37984]: 7.b scrub ok 2024-02-05T06:37:47.246 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:47 smithi028 ceph-mon[37984]: pgmap v168: 158 pgs: 1 active+clean+scrubbing+deep, 1 peering, 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 active+remapped+backfilling, 153 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 802 op/s; 9528/257796 objects misplaced (3.696%); 33 MiB/s, 525 objects/s recovering 2024-02-05T06:37:47.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:47 smithi028 ceph-mon[29442]: 7.b scrub ok 2024-02-05T06:37:47.247 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:47 smithi028 ceph-mon[29442]: pgmap v168: 158 pgs: 1 active+clean+scrubbing+deep, 1 peering, 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 active+remapped+backfilling, 153 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 802 op/s; 9528/257796 objects misplaced (3.696%); 33 MiB/s, 525 objects/s recovering 2024-02-05T06:37:47.274 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:47.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:47.318 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:47.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:47.419 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:47.462 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:47.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:47.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:47.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:47 smithi167 ceph-mon[32774]: 7.b scrub ok 2024-02-05T06:37:47.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:47 smithi167 ceph-mon[32774]: pgmap v168: 158 pgs: 1 active+clean+scrubbing+deep, 1 peering, 1 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 active+remapped+backfilling, 153 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 802 op/s; 9528/257796 objects misplaced (3.696%); 33 MiB/s, 525 objects/s recovering 2024-02-05T06:37:47.564 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:47.601 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:47.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:47.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:47.702 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:47.750 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:47.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:47.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:47.852 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:47.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:47.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:47.996 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:48.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:48.041 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:48.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:48.142 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:48.176 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:48.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:48.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:48.278 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:48.307 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:48.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:48.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:48.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:48.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:48 smithi028 ceph-mon[29442]: 3.1b scrub starts 2024-02-05T06:37:48.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:48 smithi028 ceph-mon[29442]: 3.1b scrub ok 2024-02-05T06:37:48.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:48 smithi028 ceph-mon[29442]: 5.d scrub starts 2024-02-05T06:37:48.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:48 smithi028 ceph-mon[29442]: 5.d scrub ok 2024-02-05T06:37:48.424 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:48 smithi028 ceph-mon[29442]: from='client.24865 v1:172.21.15.28:0/857666583' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:37:48.424 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:48 smithi028 ceph-mon[37984]: 3.1b scrub starts 2024-02-05T06:37:48.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:48 smithi028 ceph-mon[37984]: 3.1b scrub ok 2024-02-05T06:37:48.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:48 smithi028 ceph-mon[37984]: 5.d scrub starts 2024-02-05T06:37:48.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:48 smithi028 ceph-mon[37984]: 5.d scrub ok 2024-02-05T06:37:48.425 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:48 smithi028 ceph-mon[37984]: from='client.24865 v1:172.21.15.28:0/857666583' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:37:48.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:48.458 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:48.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:48.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:48 smithi167 ceph-mon[32774]: 3.1b scrub starts 2024-02-05T06:37:48.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:48 smithi167 ceph-mon[32774]: 3.1b scrub ok 2024-02-05T06:37:48.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:48 smithi167 ceph-mon[32774]: 5.d scrub starts 2024-02-05T06:37:48.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:48 smithi167 ceph-mon[32774]: 5.d scrub ok 2024-02-05T06:37:48.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:48 smithi167 ceph-mon[32774]: from='client.24865 v1:172.21.15.28:0/857666583' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:37:48.559 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:48.588 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:48.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:48.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:48.690 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:48.716 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:48.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:48.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:48.782 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:37:48.818 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:48.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:48.884 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:48.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:48.885 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:37:48.885 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:37:48.985 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:49.013 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:49.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:49.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:49.115 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:49.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:49.173 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:49.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:49.274 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:49.299 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:49.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:49.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:49.403 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:49.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:49.430 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:49.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:49.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:49 smithi167 ceph-mon[32774]: 4.2 scrub starts 2024-02-05T06:37:49.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:49 smithi167 ceph-mon[32774]: 4.2 scrub ok 2024-02-05T06:37:49.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:49 smithi167 ceph-mon[32774]: pgmap v169: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 155 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 935 op/s; 7603/251829 objects misplaced (3.019%); 41 MiB/s, 659 objects/s recovering 2024-02-05T06:37:49.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:49 smithi167 ceph-mon[32774]: 6.19 scrub starts 2024-02-05T06:37:49.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:49 smithi167 ceph-mon[32774]: 6.19 scrub ok 2024-02-05T06:37:49.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:49 smithi167 ceph-mon[32774]: 6.3 scrub starts 2024-02-05T06:37:49.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:49 smithi167 ceph-mon[32774]: 6.3 scrub ok 2024-02-05T06:37:49.532 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:49.561 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:49.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:49.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:49.664 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:49.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:49 smithi028 ceph-mon[37984]: 4.2 scrub starts 2024-02-05T06:37:49.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:49 smithi028 ceph-mon[37984]: 4.2 scrub ok 2024-02-05T06:37:49.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:49 smithi028 ceph-mon[37984]: pgmap v169: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 155 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 935 op/s; 7603/251829 objects misplaced (3.019%); 41 MiB/s, 659 objects/s recovering 2024-02-05T06:37:49.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:49 smithi028 ceph-mon[37984]: 6.19 scrub starts 2024-02-05T06:37:49.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:49 smithi028 ceph-mon[37984]: 6.19 scrub ok 2024-02-05T06:37:49.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:49 smithi028 ceph-mon[37984]: 6.3 scrub starts 2024-02-05T06:37:49.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:49 smithi028 ceph-mon[37984]: 6.3 scrub ok 2024-02-05T06:37:49.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:49 smithi028 ceph-mon[29442]: 4.2 scrub starts 2024-02-05T06:37:49.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:49 smithi028 ceph-mon[29442]: 4.2 scrub ok 2024-02-05T06:37:49.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:49 smithi028 ceph-mon[29442]: pgmap v169: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 155 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 935 op/s; 7603/251829 objects misplaced (3.019%); 41 MiB/s, 659 objects/s recovering 2024-02-05T06:37:49.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:49 smithi028 ceph-mon[29442]: 6.19 scrub starts 2024-02-05T06:37:49.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:49 smithi028 ceph-mon[29442]: 6.19 scrub ok 2024-02-05T06:37:49.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:49 smithi028 ceph-mon[29442]: 6.3 scrub starts 2024-02-05T06:37:49.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:49 smithi028 ceph-mon[29442]: 6.3 scrub ok 2024-02-05T06:37:49.704 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:49.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:49.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:49.747 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:37:49.753 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-05T06:37:49.807 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:49.835 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:49.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:49.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:49.937 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:49.974 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:49.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:49.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:50.076 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:50.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:50.107 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:50.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:50.208 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:50.235 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:50.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:50.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:50.246 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-02-05T06:37:50.337 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:50.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:50.370 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:50.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:50.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:50 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:37:50.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:50 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:37:50.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:50 smithi028 ceph-mon[29442]: osdmap e140: 8 total, 8 up, 8 in 2024-02-05T06:37:50.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:50 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2803281763' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:37:50.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:50 smithi028 ceph-mon[29442]: from='client.24871 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:37:50.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:50 smithi028 ceph-mon[29442]: 6.5 scrub starts 2024-02-05T06:37:50.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:50 smithi028 ceph-mon[29442]: 6.5 scrub ok 2024-02-05T06:37:50.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:50 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:37:50.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:50 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:37:50.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:50 smithi028 ceph-mon[37984]: osdmap e140: 8 total, 8 up, 8 in 2024-02-05T06:37:50.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:50 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2803281763' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:37:50.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:50 smithi028 ceph-mon[37984]: from='client.24871 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:37:50.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:50 smithi028 ceph-mon[37984]: 6.5 scrub starts 2024-02-05T06:37:50.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:50 smithi028 ceph-mon[37984]: 6.5 scrub ok 2024-02-05T06:37:50.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:50.513 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:50.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:50.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:50.615 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:50.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:50.645 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:50.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:50.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:50 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:37:50.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:50 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:37:50.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:50 smithi167 ceph-mon[32774]: osdmap e140: 8 total, 8 up, 8 in 2024-02-05T06:37:50.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:50 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2803281763' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:37:50.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:50 smithi167 ceph-mon[32774]: from='client.24871 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:37:50.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:50 smithi167 ceph-mon[32774]: 6.5 scrub starts 2024-02-05T06:37:50.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:50 smithi167 ceph-mon[32774]: 6.5 scrub ok 2024-02-05T06:37:50.747 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:50.776 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:50.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:50.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:50.878 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:50.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:50.916 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:50.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:51.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:51.056 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:51.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:51.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:51.157 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:51.191 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:51.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:51.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:51.293 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:51.325 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-05T06:37:51.334 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:51.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:51.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:51.436 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:51.470 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:51.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:51.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:51.530 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2024-02-05T06:37:46.221405+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5619093094,"num_objects":85932,"num_object_clones":0,"num_object_copies":257796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9528,"num_objects_unfound":0,"num_objects_dirty":85932,"num_whiteouts":0,"num_read":1014,"num_read_kb":793,"num_write":144986,"num_write_kb":7368194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21037,"num_bytes_recovered":1368027322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82812,"ondisk_log_size":82812,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":478,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":17960228,"kb_used_data":17037520,"kb_used_omap":12,"kb_used_meta":922611,"kb_avail":732000988,"statfs":{"total":767960285184,"available":749569011712,"internally_reserved":0,"allocated":17446420480,"data_stored":17443161478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":944754225},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,1,2,5,3,4,0,0,0,1,15],"upper_bound":2048},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-323271017,"num_objects":-4932,"num_object_clones":0,"num_object_copies":-14796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3689,"num_objects_unfound":0,"num_objects_dirty":-4932,"num_whiteouts":0,"num_read":-119,"num_read_kb":-180,"num_write":4817,"num_write_kb":-1537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3154,"num_bytes_recovered":205572935,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002000"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":51,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.967212+0000","last_change":"2024-02-05T06:36:54.954342+0000","last_active":"2024-02-05T06:37:42.967212+0000","last_peered":"2024-02-05T06:37:42.967212+0000","last_clean":"2024-02-05T06:37:42.967212+0000","last_became_active":"2024-02-05T06:36:54.952487+0000","last_became_peered":"2024-02-05T06:36:54.952487+0000","last_unstale":"2024-02-05T06:37:42.967212+0000","last_undegraded":"2024-02-05T06:37:42.967212+0000","last_fullsized":"2024-02-05T06:37:42.967212+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:32.017855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:32.017855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:45.407053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":161,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.968685+0000","last_change":"2024-02-05T06:36:37.918740+0000","last_active":"2024-02-05T06:37:40.968685+0000","last_peered":"2024-02-05T06:37:40.968685+0000","last_clean":"2024-02-05T06:37:40.968685+0000","last_became_active":"2024-02-05T06:33:33.585848+0000","last_became_peered":"2024-02-05T06:33:33.585848+0000","last_unstale":"2024-02-05T06:37:40.968685+0000","last_undegraded":"2024-02-05T06:37:40.968685+0000","last_fullsized":"2024-02-05T06:37:40.968685+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:37.918224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:06.525012+0000","last_clean_scrub_stamp":"2024-02-05T06:36:37.918224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:00.105543+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":163,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.965333+0000","last_change":"2024-02-05T06:36:09.154584+0000","last_active":"2024-02-05T06:37:40.965333+0000","last_peered":"2024-02-05T06:37:40.965333+0000","last_clean":"2024-02-05T06:37:40.965333+0000","last_became_active":"2024-02-05T06:33:29.549592+0000","last_became_peered":"2024-02-05T06:33:29.549592+0000","last_unstale":"2024-02-05T06:37:40.965333+0000","last_undegraded":"2024-02-05T06:37:40.965333+0000","last_fullsized":"2024-02-05T06:37:40.965333+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:09.154442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:09.154442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":61,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.966022+0000","last_change":"2024-02-05T06:36:54.997747+0000","last_active":"2024-02-05T06:37:42.966022+0000","last_peered":"2024-02-05T06:37:42.966022+0000","last_clean":"2024-02-05T06:37:42.966022+0000","last_became_active":"2024-02-05T06:36:54.959330+0000","last_became_peered":"2024-02-05T06:36:54.959330+0000","last_unstale":"2024-02-05T06:37:42.966022+0000","last_undegraded":"2024-02-05T06:37:42.966022+0000","last_fullsized":"2024-02-05T06:37:42.966022+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:35:01.601838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:35:01.601838+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":7,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":158,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.966051+0000","last_change":"2024-02-05T06:37:21.419916+0000","last_active":"2024-02-05T06:37:40.966051+0000","last_peered":"2024-02-05T06:37:40.966051+0000","last_clean":"2024-02-05T06:37:40.966051+0000","last_became_active":"2024-02-05T06:33:35.667778+0000","last_became_peered":"2024-02-05T06:33:35.667778+0000","last_unstale":"2024-02-05T06:37:40.966051+0000","last_undegraded":"2024-02-05T06:37:40.966051+0000","last_fullsized":"2024-02-05T06:37:40.966051+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:21.419504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:37:21.419504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:47.323614+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":51,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.965851+0000","last_change":"2024-02-05T06:36:54.961637+0000","last_active":"2024-02-05T06:37:42.965851+0000","last_peered":"2024-02-05T06:37:42.965851+0000","last_clean":"2024-02-05T06:37:42.965851+0000","last_became_active":"2024-02-05T06:36:54.959640+0000","last_became_peered":"2024-02-05T06:36:54.959640+0000","last_unstale":"2024-02-05T06:37:42.965851+0000","last_undegraded":"2024-02-05T06:37:42.965851+0000","last_fullsized":"2024-02-05T06:37:42.965851+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:13.639311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:35:13.639311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":171,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.966572+0000","last_change":"2024-02-05T06:36:17.296405+0000","last_active":"2024-02-05T06:37:40.966572+0000","last_peered":"2024-02-05T06:37:40.966572+0000","last_clean":"2024-02-05T06:37:40.966572+0000","last_became_active":"2024-02-05T06:33:29.553864+0000","last_became_peered":"2024-02-05T06:33:29.553864+0000","last_unstale":"2024-02-05T06:37:40.966572+0000","last_undegraded":"2024-02-05T06:37:40.966572+0000","last_fullsized":"2024-02-05T06:37:40.966572+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:17.296252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:17.296252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"88'22","reported_seq":198,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.964711+0000","last_change":"2024-02-05T06:36:19.337617+0000","last_active":"2024-02-05T06:37:40.964711+0000","last_peered":"2024-02-05T06:37:40.964711+0000","last_clean":"2024-02-05T06:37:40.964711+0000","last_became_active":"2024-02-05T06:33:31.572544+0000","last_became_peered":"2024-02-05T06:33:31.572544+0000","last_unstale":"2024-02-05T06:37:40.964711+0000","last_undegraded":"2024-02-05T06:37:40.964711+0000","last_fullsized":"2024-02-05T06:37:40.964711+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'22","last_scrub_stamp":"2024-02-05T06:36:19.337341+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:34:35.832368+0000","last_clean_scrub_stamp":"2024-02-05T06:36:19.337341+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":150,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.963409+0000","last_change":"2024-02-05T06:35:20.603134+0000","last_active":"2024-02-05T06:37:40.963409+0000","last_peered":"2024-02-05T06:37:40.963409+0000","last_clean":"2024-02-05T06:37:40.963409+0000","last_became_active":"2024-02-05T06:33:35.594860+0000","last_became_peered":"2024-02-05T06:33:35.594860+0000","last_unstale":"2024-02-05T06:37:40.963409+0000","last_undegraded":"2024-02-05T06:37:40.963409+0000","last_fullsized":"2024-02-05T06:37:40.963409+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:20.603044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:20.603044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":164,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.999253+0000","last_change":"2024-02-05T06:36:07.749745+0000","last_active":"2024-02-05T06:37:42.999253+0000","last_peered":"2024-02-05T06:37:42.999253+0000","last_clean":"2024-02-05T06:37:42.999253+0000","last_became_active":"2024-02-05T06:33:33.585459+0000","last_became_peered":"2024-02-05T06:33:33.585459+0000","last_unstale":"2024-02-05T06:37:42.999253+0000","last_undegraded":"2024-02-05T06:37:42.999253+0000","last_fullsized":"2024-02-05T06:37:42.999253+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:07.749639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:07.749639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":184,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.999413+0000","last_change":"2024-02-05T06:36:54.950822+0000","last_active":"2024-02-05T06:37:42.999413+0000","last_peered":"2024-02-05T06:37:42.999413+0000","last_clean":"2024-02-05T06:37:42.999413+0000","last_became_active":"2024-02-05T06:36:54.950515+0000","last_became_peered":"2024-02-05T06:36:54.950515+0000","last_unstale":"2024-02-05T06:37:42.999413+0000","last_undegraded":"2024-02-05T06:37:42.999413+0000","last_fullsized":"2024-02-05T06:37:42.999413+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:08.762241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:44.081318+0000","last_clean_scrub_stamp":"2024-02-05T06:36:08.762241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"92'13","reported_seq":61,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.967064+0000","last_change":"2024-02-05T06:36:55.009049+0000","last_active":"2024-02-05T06:37:42.967064+0000","last_peered":"2024-02-05T06:37:42.967064+0000","last_clean":"2024-02-05T06:37:42.967064+0000","last_became_active":"2024-02-05T06:36:54.969454+0000","last_became_peered":"2024-02-05T06:36:54.969454+0000","last_unstale":"2024-02-05T06:37:42.967064+0000","last_undegraded":"2024-02-05T06:37:42.967064+0000","last_fullsized":"2024-02-05T06:37:42.967064+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'13","last_scrub_stamp":"2024-02-05T06:34:46.542893+0000","last_deep_scrub":"92'13","last_deep_scrub_stamp":"2024-02-05T06:34:46.542893+0000","last_clean_scrub_stamp":"2024-02-05T06:34:46.542893+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":152,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.998346+0000","last_change":"2024-02-05T06:35:21.998937+0000","last_active":"2024-02-05T06:37:42.998346+0000","last_peered":"2024-02-05T06:37:42.998346+0000","last_clean":"2024-02-05T06:37:42.998346+0000","last_became_active":"2024-02-05T06:33:35.600399+0000","last_became_peered":"2024-02-05T06:33:35.600399+0000","last_unstale":"2024-02-05T06:37:42.998346+0000","last_undegraded":"2024-02-05T06:37:42.998346+0000","last_fullsized":"2024-02-05T06:37:42.998346+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:21.998850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:21.998850+0000","last_clean_scrub_stamp":"2024-02-05T06:35:21.998850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":178,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.970674+0000","last_change":"2024-02-05T06:37:26.781187+0000","last_active":"2024-02-05T06:37:42.970674+0000","last_peered":"2024-02-05T06:37:42.970674+0000","last_clean":"2024-02-05T06:37:42.970674+0000","last_became_active":"2024-02-05T06:36:54.969671+0000","last_became_peered":"2024-02-05T06:36:54.969671+0000","last_unstale":"2024-02-05T06:37:42.970674+0000","last_undegraded":"2024-02-05T06:37:42.970674+0000","last_fullsized":"2024-02-05T06:37:42.970674+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:26.781111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:37:26.781111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:47.145242+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":184,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.999117+0000","last_change":"2024-02-05T06:36:54.951354+0000","last_active":"2024-02-05T06:37:42.999117+0000","last_peered":"2024-02-05T06:37:42.999117+0000","last_clean":"2024-02-05T06:37:42.999117+0000","last_became_active":"2024-02-05T06:36:54.951044+0000","last_became_peered":"2024-02-05T06:36:54.951044+0000","last_unstale":"2024-02-05T06:37:42.999117+0000","last_undegraded":"2024-02-05T06:37:42.999117+0000","last_fullsized":"2024-02-05T06:37:42.999117+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:30.976268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:50.055245+0000","last_clean_scrub_stamp":"2024-02-05T06:36:30.976268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:56.891812+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":57,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.966889+0000","last_change":"2024-02-05T06:36:54.985263+0000","last_active":"2024-02-05T06:37:42.966889+0000","last_peered":"2024-02-05T06:37:42.966889+0000","last_clean":"2024-02-05T06:37:42.966889+0000","last_became_active":"2024-02-05T06:36:54.954385+0000","last_became_peered":"2024-02-05T06:36:54.954385+0000","last_unstale":"2024-02-05T06:37:42.966889+0000","last_undegraded":"2024-02-05T06:37:42.966889+0000","last_fullsized":"2024-02-05T06:37:42.966889+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-02-05T06:34:53.638704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:53.638704+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":51,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.966542+0000","last_change":"2024-02-05T06:36:54.963609+0000","last_active":"2024-02-05T06:37:42.966542+0000","last_peered":"2024-02-05T06:37:42.966542+0000","last_clean":"2024-02-05T06:37:42.966542+0000","last_became_active":"2024-02-05T06:36:54.961535+0000","last_became_peered":"2024-02-05T06:36:54.961535+0000","last_unstale":"2024-02-05T06:37:42.966542+0000","last_undegraded":"2024-02-05T06:37:42.966542+0000","last_fullsized":"2024-02-05T06:37:42.966542+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:45.573432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:45.573432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":173,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.968393+0000","last_change":"2024-02-05T06:37:40.344307+0000","last_active":"2024-02-05T06:37:40.968393+0000","last_peered":"2024-02-05T06:37:40.968393+0000","last_clean":"2024-02-05T06:37:40.968393+0000","last_became_active":"2024-02-05T06:36:54.958069+0000","last_became_peered":"2024-02-05T06:36:54.958069+0000","last_unstale":"2024-02-05T06:37:40.968393+0000","last_undegraded":"2024-02-05T06:37:40.968393+0000","last_fullsized":"2024-02-05T06:37:40.968393+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:40.344166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:37:40.344166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:39:08.368396+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"66'1","reported_seq":188,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.963979+0000","last_change":"2024-02-05T06:36:54.991286+0000","last_active":"2024-02-05T06:37:40.963979+0000","last_peered":"2024-02-05T06:37:40.963979+0000","last_clean":"2024-02-05T06:37:40.963979+0000","last_became_active":"2024-02-05T06:36:54.971810+0000","last_became_peered":"2024-02-05T06:36:54.971810+0000","last_unstale":"2024-02-05T06:37:40.963979+0000","last_undegraded":"2024-02-05T06:37:40.963979+0000","last_fullsized":"2024-02-05T06:37:40.963979+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:35:22.640782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:35:22.640782+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"88'22","reported_seq":204,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.996522+0000","last_change":"2024-02-05T06:37:30.483386+0000","last_active":"2024-02-05T06:37:40.996522+0000","last_peered":"2024-02-05T06:37:40.996522+0000","last_clean":"2024-02-05T06:37:40.996522+0000","last_became_active":"2024-02-05T06:33:31.569763+0000","last_became_peered":"2024-02-05T06:33:31.569763+0000","last_unstale":"2024-02-05T06:37:40.996522+0000","last_undegraded":"2024-02-05T06:37:40.996522+0000","last_fullsized":"2024-02-05T06:37:40.996522+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'22","last_scrub_stamp":"2024-02-05T06:37:30.483233+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:34:35.234636+0000","last_clean_scrub_stamp":"2024-02-05T06:37:30.483233+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:34.268799+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":150,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.966475+0000","last_change":"2024-02-05T06:35:05.077348+0000","last_active":"2024-02-05T06:37:40.966475+0000","last_peered":"2024-02-05T06:37:40.966475+0000","last_clean":"2024-02-05T06:37:40.966475+0000","last_became_active":"2024-02-05T06:33:35.664852+0000","last_became_peered":"2024-02-05T06:33:35.664852+0000","last_unstale":"2024-02-05T06:37:40.966475+0000","last_undegraded":"2024-02-05T06:37:40.966475+0000","last_fullsized":"2024-02-05T06:37:40.966475+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:05.077268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:05.077268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":51,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.966324+0000","last_change":"2024-02-05T06:36:54.963177+0000","last_active":"2024-02-05T06:37:42.966324+0000","last_peered":"2024-02-05T06:37:42.966324+0000","last_clean":"2024-02-05T06:37:42.966324+0000","last_became_active":"2024-02-05T06:36:54.959289+0000","last_became_peered":"2024-02-05T06:36:54.959289+0000","last_unstale":"2024-02-05T06:37:42.966324+0000","last_undegraded":"2024-02-05T06:37:42.966324+0000","last_fullsized":"2024-02-05T06:37:42.966324+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:05.580971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:35:05.580971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":51,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.966302+0000","last_change":"2024-02-05T06:36:54.961607+0000","last_active":"2024-02-05T06:37:42.966302+0000","last_peered":"2024-02-05T06:37:42.966302+0000","last_clean":"2024-02-05T06:37:42.966302+0000","last_became_active":"2024-02-05T06:36:54.959070+0000","last_became_peered":"2024-02-05T06:36:54.959070+0000","last_unstale":"2024-02-05T06:37:42.966302+0000","last_undegraded":"2024-02-05T06:37:42.966302+0000","last_fullsized":"2024-02-05T06:37:42.966302+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:12.270778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:12.270778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":235,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.966067+0000","last_change":"2024-02-05T06:36:54.987702+0000","last_active":"2024-02-05T06:37:40.966067+0000","last_peered":"2024-02-05T06:37:40.966067+0000","last_clean":"2024-02-05T06:37:40.966067+0000","last_became_active":"2024-02-05T06:36:54.954880+0000","last_became_peered":"2024-02-05T06:36:54.954880+0000","last_unstale":"2024-02-05T06:37:40.966067+0000","last_undegraded":"2024-02-05T06:37:40.966067+0000","last_fullsized":"2024-02-05T06:37:40.966067+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:34:59.999726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:59.999726+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"73'1","reported_seq":163,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.971365+0000","last_change":"2024-02-05T06:37:41.605448+0000","last_active":"2024-02-05T06:37:42.971365+0000","last_peered":"2024-02-05T06:37:42.971365+0000","last_clean":"2024-02-05T06:37:42.971365+0000","last_became_active":"2024-02-05T06:33:35.598501+0000","last_became_peered":"2024-02-05T06:33:35.598501+0000","last_unstale":"2024-02-05T06:37:42.971365+0000","last_undegraded":"2024-02-05T06:37:42.971365+0000","last_fullsized":"2024-02-05T06:37:42.971365+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-02-05T06:37:41.605375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:37:41.605375+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:52.286879+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"119'10","reported_seq":254,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:43.003752+0000","last_change":"2024-02-05T06:37:24.768633+0000","last_active":"2024-02-05T06:37:43.003752+0000","last_peered":"2024-02-05T06:37:43.003752+0000","last_clean":"2024-02-05T06:37:43.003752+0000","last_became_active":"2024-02-05T06:36:54.957934+0000","last_became_peered":"2024-02-05T06:36:54.957934+0000","last_unstale":"2024-02-05T06:37:43.003752+0000","last_undegraded":"2024-02-05T06:37:43.003752+0000","last_fullsized":"2024-02-05T06:37:43.003752+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-02-05T06:37:24.768516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:37:24.768516+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:28.752261+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":190,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.968079+0000","last_change":"2024-02-05T06:36:54.977901+0000","last_active":"2024-02-05T06:37:40.968079+0000","last_peered":"2024-02-05T06:37:40.968079+0000","last_clean":"2024-02-05T06:37:40.968079+0000","last_became_active":"2024-02-05T06:36:54.958414+0000","last_became_peered":"2024-02-05T06:36:54.958414+0000","last_unstale":"2024-02-05T06:37:40.968079+0000","last_undegraded":"2024-02-05T06:37:40.968079+0000","last_fullsized":"2024-02-05T06:37:40.968079+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:36:10.257008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:10.257008+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"92'15","reported_seq":187,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.965075+0000","last_change":"2024-02-05T06:36:28.242822+0000","last_active":"2024-02-05T06:37:40.965075+0000","last_peered":"2024-02-05T06:37:40.965075+0000","last_clean":"2024-02-05T06:37:40.965075+0000","last_became_active":"2024-02-05T06:33:31.666481+0000","last_became_peered":"2024-02-05T06:33:31.666481+0000","last_unstale":"2024-02-05T06:37:40.965075+0000","last_undegraded":"2024-02-05T06:37:40.965075+0000","last_fullsized":"2024-02-05T06:37:40.965075+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'15","last_scrub_stamp":"2024-02-05T06:36:28.242304+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-02-05T06:34:50.718768+0000","last_clean_scrub_stamp":"2024-02-05T06:36:28.242304+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:43.531566+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":169,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.968658+0000","last_change":"2024-02-05T06:36:54.959126+0000","last_active":"2024-02-05T06:37:40.968658+0000","last_peered":"2024-02-05T06:37:40.968658+0000","last_clean":"2024-02-05T06:37:40.968658+0000","last_became_active":"2024-02-05T06:36:54.958621+0000","last_became_peered":"2024-02-05T06:36:54.958621+0000","last_unstale":"2024-02-05T06:37:40.968658+0000","last_undegraded":"2024-02-05T06:37:40.968658+0000","last_fullsized":"2024-02-05T06:37:40.968658+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:26.060478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:26.060478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":65,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.965834+0000","last_change":"2024-02-05T06:36:12.175207+0000","last_active":"2024-02-05T06:37:40.965834+0000","last_peered":"2024-02-05T06:37:40.965834+0000","last_clean":"2024-02-05T06:37:40.965834+0000","last_became_active":"2024-02-05T06:36:12.174876+0000","last_became_peered":"2024-02-05T06:36:12.174876+0000","last_unstale":"2024-02-05T06:37:40.965834+0000","last_undegraded":"2024-02-05T06:37:40.965834+0000","last_fullsized":"2024-02-05T06:37:40.965834+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:08.713894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:08.713894+0000","last_clean_scrub_stamp":"2024-02-05T06:35:08.713894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":59,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.998465+0000","last_change":"2024-02-05T06:37:22.248009+0000","last_active":"2024-02-05T06:37:42.998465+0000","last_peered":"2024-02-05T06:37:42.998465+0000","last_clean":"2024-02-05T06:37:42.998465+0000","last_became_active":"2024-02-05T06:36:54.950298+0000","last_became_peered":"2024-02-05T06:36:54.950298+0000","last_unstale":"2024-02-05T06:37:42.998465+0000","last_undegraded":"2024-02-05T06:37:42.998465+0000","last_fullsized":"2024-02-05T06:37:42.998465+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:22.247897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:37:22.247897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:44.679577+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"92'16","reported_seq":202,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:43.003619+0000","last_change":"2024-02-05T06:37:41.849127+0000","last_active":"2024-02-05T06:37:43.003619+0000","last_peered":"2024-02-05T06:37:43.003619+0000","last_clean":"2024-02-05T06:37:43.003619+0000","last_became_active":"2024-02-05T06:33:31.573088+0000","last_became_peered":"2024-02-05T06:33:31.573088+0000","last_unstale":"2024-02-05T06:37:43.003619+0000","last_undegraded":"2024-02-05T06:37:43.003619+0000","last_fullsized":"2024-02-05T06:37:43.003619+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'16","last_scrub_stamp":"2024-02-05T06:37:41.849041+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:34:45.270135+0000","last_clean_scrub_stamp":"2024-02-05T06:37:41.849041+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:39:08.680999+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":174,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.970513+0000","last_change":"2024-02-05T06:37:29.706087+0000","last_active":"2024-02-05T06:37:42.970513+0000","last_peered":"2024-02-05T06:37:42.970513+0000","last_clean":"2024-02-05T06:37:42.970513+0000","last_became_active":"2024-02-05T06:36:54.969234+0000","last_became_peered":"2024-02-05T06:36:54.969234+0000","last_unstale":"2024-02-05T06:37:42.970513+0000","last_undegraded":"2024-02-05T06:37:42.970513+0000","last_fullsized":"2024-02-05T06:37:42.970513+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:29.705961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:37:29.705961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:36.260365+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":185,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:43.001930+0000","last_change":"2024-02-05T06:37:36.743191+0000","last_active":"2024-02-05T06:37:43.001930+0000","last_peered":"2024-02-05T06:37:43.001930+0000","last_clean":"2024-02-05T06:37:43.001930+0000","last_became_active":"2024-02-05T06:36:54.968712+0000","last_became_peered":"2024-02-05T06:36:54.968712+0000","last_unstale":"2024-02-05T06:37:43.001930+0000","last_undegraded":"2024-02-05T06:37:43.001930+0000","last_fullsized":"2024-02-05T06:37:43.001930+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:36.741703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:37:36.741703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:50.673100+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":162,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.965433+0000","last_change":"2024-02-05T06:36:17.221190+0000","last_active":"2024-02-05T06:37:40.965433+0000","last_peered":"2024-02-05T06:37:40.965433+0000","last_clean":"2024-02-05T06:37:40.965433+0000","last_became_active":"2024-02-05T06:33:29.549204+0000","last_became_peered":"2024-02-05T06:33:29.549204+0000","last_unstale":"2024-02-05T06:37:40.965433+0000","last_undegraded":"2024-02-05T06:37:40.965433+0000","last_fullsized":"2024-02-05T06:37:40.965433+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:17.220950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:17.220950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"73'4","reported_seq":164,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.964002+0000","last_change":"2024-02-05T06:36:06.303548+0000","last_active":"2024-02-05T06:37:40.964002+0000","last_peered":"2024-02-05T06:37:40.964002+0000","last_clean":"2024-02-05T06:37:40.964002+0000","last_became_active":"2024-02-05T06:33:31.573410+0000","last_became_peered":"2024-02-05T06:33:31.573410+0000","last_unstale":"2024-02-05T06:37:40.964002+0000","last_undegraded":"2024-02-05T06:37:40.964002+0000","last_fullsized":"2024-02-05T06:37:40.964002+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'4","last_scrub_stamp":"2024-02-05T06:36:06.303366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:06.303366+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":150,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.966675+0000","last_change":"2024-02-05T06:34:54.902085+0000","last_active":"2024-02-05T06:37:40.966675+0000","last_peered":"2024-02-05T06:37:40.966675+0000","last_clean":"2024-02-05T06:37:40.966675+0000","last_became_active":"2024-02-05T06:33:35.667602+0000","last_became_peered":"2024-02-05T06:33:35.667602+0000","last_unstale":"2024-02-05T06:37:40.966675+0000","last_undegraded":"2024-02-05T06:37:40.966675+0000","last_fullsized":"2024-02-05T06:37:40.966675+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:54.902000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:34:54.902000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":186,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:43.605065+0000","last_change":"2024-02-05T06:37:43.605103+0000","last_active":"2024-02-05T06:37:43.605065+0000","last_peered":"2024-02-05T06:37:43.605065+0000","last_clean":"2024-02-05T06:37:43.605065+0000","last_became_active":"2024-02-05T06:36:54.969443+0000","last_became_peered":"2024-02-05T06:36:54.969443+0000","last_unstale":"2024-02-05T06:37:43.605065+0000","last_undegraded":"2024-02-05T06:37:43.605065+0000","last_fullsized":"2024-02-05T06:37:43.605065+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:43.604994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:37:43.604994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:39:05.821301+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":172,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:43.243877+0000","last_change":"2024-02-05T06:37:43.243897+0000","last_active":"2024-02-05T06:37:43.243877+0000","last_peered":"2024-02-05T06:37:43.243877+0000","last_clean":"2024-02-05T06:37:43.243877+0000","last_became_active":"2024-02-05T06:33:29.551278+0000","last_became_peered":"2024-02-05T06:33:29.551278+0000","last_unstale":"2024-02-05T06:37:43.243877+0000","last_undegraded":"2024-02-05T06:37:43.243877+0000","last_fullsized":"2024-02-05T06:37:43.243877+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:43.243812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:37:43.243812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:49.602112+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"93'16","reported_seq":182,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.964629+0000","last_change":"2024-02-05T06:35:22.938794+0000","last_active":"2024-02-05T06:37:40.964629+0000","last_peered":"2024-02-05T06:37:40.964629+0000","last_clean":"2024-02-05T06:37:40.964629+0000","last_became_active":"2024-02-05T06:33:31.664743+0000","last_became_peered":"2024-02-05T06:33:31.664743+0000","last_unstale":"2024-02-05T06:37:40.964629+0000","last_undegraded":"2024-02-05T06:37:40.964629+0000","last_fullsized":"2024-02-05T06:37:40.964629+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'16","last_scrub_stamp":"2024-02-05T06:35:22.938675+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-02-05T06:35:22.938675+0000","last_clean_scrub_stamp":"2024-02-05T06:35:22.938675+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"73'1","reported_seq":159,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:41.335292+0000","last_change":"2024-02-05T06:37:41.335352+0000","last_active":"2024-02-05T06:37:41.335292+0000","last_peered":"2024-02-05T06:37:41.335292+0000","last_clean":"2024-02-05T06:37:41.335292+0000","last_became_active":"2024-02-05T06:33:35.595031+0000","last_became_peered":"2024-02-05T06:33:35.595031+0000","last_unstale":"2024-02-05T06:37:41.335292+0000","last_undegraded":"2024-02-05T06:37:41.335292+0000","last_fullsized":"2024-02-05T06:37:41.335292+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-02-05T06:37:41.335213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:37:41.335213+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:57.666297+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":154,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.965940+0000","last_change":"2024-02-05T06:35:03.027397+0000","last_active":"2024-02-05T06:37:40.965940+0000","last_peered":"2024-02-05T06:37:40.965940+0000","last_clean":"2024-02-05T06:37:40.965940+0000","last_became_active":"2024-02-05T06:33:33.593901+0000","last_became_peered":"2024-02-05T06:33:33.593901+0000","last_unstale":"2024-02-05T06:37:40.965940+0000","last_undegraded":"2024-02-05T06:37:40.965940+0000","last_fullsized":"2024-02-05T06:37:40.965940+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:03.027270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:35:03.027270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":186,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.986069+0000","last_change":"2024-02-05T06:37:27.744260+0000","last_active":"2024-02-05T06:37:42.986069+0000","last_peered":"2024-02-05T06:37:42.986069+0000","last_clean":"2024-02-05T06:37:42.986069+0000","last_became_active":"2024-02-05T06:36:54.970247+0000","last_became_peered":"2024-02-05T06:36:54.970247+0000","last_unstale":"2024-02-05T06:37:42.986069+0000","last_undegraded":"2024-02-05T06:37:42.986069+0000","last_fullsized":"2024-02-05T06:37:42.986069+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:27.744143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:37:27.744143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:49.607190+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":181,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.968395+0000","last_change":"2024-02-05T06:36:30.047255+0000","last_active":"2024-02-05T06:37:40.968395+0000","last_peered":"2024-02-05T06:37:40.968395+0000","last_clean":"2024-02-05T06:37:40.968395+0000","last_became_active":"2024-02-05T06:33:31.568529+0000","last_became_peered":"2024-02-05T06:33:31.568529+0000","last_unstale":"2024-02-05T06:37:40.968395+0000","last_undegraded":"2024-02-05T06:37:40.968395+0000","last_fullsized":"2024-02-05T06:37:40.968395+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:36:30.047048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:30.047048+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:59.807573+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":158,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.964357+0000","last_change":"2024-02-05T06:36:30.451097+0000","last_active":"2024-02-05T06:37:40.964357+0000","last_peered":"2024-02-05T06:37:40.964357+0000","last_clean":"2024-02-05T06:37:40.964357+0000","last_became_active":"2024-02-05T06:33:35.597399+0000","last_became_peered":"2024-02-05T06:33:35.597399+0000","last_unstale":"2024-02-05T06:37:40.964357+0000","last_undegraded":"2024-02-05T06:37:40.964357+0000","last_fullsized":"2024-02-05T06:37:40.964357+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:30.450988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:30.450988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":161,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.968206+0000","last_change":"2024-02-05T06:36:06.224697+0000","last_active":"2024-02-05T06:37:40.968206+0000","last_peered":"2024-02-05T06:37:40.968206+0000","last_clean":"2024-02-05T06:37:40.968206+0000","last_became_active":"2024-02-05T06:33:33.588461+0000","last_became_peered":"2024-02-05T06:33:33.588461+0000","last_unstale":"2024-02-05T06:37:40.968206+0000","last_undegraded":"2024-02-05T06:37:40.968206+0000","last_fullsized":"2024-02-05T06:37:40.968206+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:06.224205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:06.224205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":51,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.966303+0000","last_change":"2024-02-05T06:36:54.964023+0000","last_active":"2024-02-05T06:37:42.966303+0000","last_peered":"2024-02-05T06:37:42.966303+0000","last_clean":"2024-02-05T06:37:42.966303+0000","last_became_active":"2024-02-05T06:36:54.963719+0000","last_became_peered":"2024-02-05T06:36:54.963719+0000","last_unstale":"2024-02-05T06:37:42.966303+0000","last_undegraded":"2024-02-05T06:37:42.966303+0000","last_fullsized":"2024-02-05T06:37:42.966303+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:27.903162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:27.903162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"92'16","reported_seq":63,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.967058+0000","last_change":"2024-02-05T06:36:54.999613+0000","last_active":"2024-02-05T06:37:42.967058+0000","last_peered":"2024-02-05T06:37:42.967058+0000","last_clean":"2024-02-05T06:37:42.967058+0000","last_became_active":"2024-02-05T06:36:54.961610+0000","last_became_peered":"2024-02-05T06:36:54.961610+0000","last_unstale":"2024-02-05T06:37:42.967058+0000","last_undegraded":"2024-02-05T06:37:42.967058+0000","last_fullsized":"2024-02-05T06:37:42.967058+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'16","last_scrub_stamp":"2024-02-05T06:36:36.728552+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:35:02.589229+0000","last_clean_scrub_stamp":"2024-02-05T06:36:36.728552+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:58.641289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":150,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.964883+0000","last_change":"2024-02-05T06:36:06.138504+0000","last_active":"2024-02-05T06:37:40.964883+0000","last_peered":"2024-02-05T06:37:40.964883+0000","last_clean":"2024-02-05T06:37:40.964883+0000","last_became_active":"2024-02-05T06:33:35.599808+0000","last_became_peered":"2024-02-05T06:33:35.599808+0000","last_unstale":"2024-02-05T06:37:40.964883+0000","last_undegraded":"2024-02-05T06:37:40.964883+0000","last_fullsized":"2024-02-05T06:37:40.964883+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:06.137845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:06.137845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":224,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.999532+0000","last_change":"2024-02-05T06:36:27.952699+0000","last_active":"2024-02-05T06:37:42.999532+0000","last_peered":"2024-02-05T06:37:42.999532+0000","last_clean":"2024-02-05T06:37:42.999532+0000","last_became_active":"2024-02-05T06:33:33.585642+0000","last_became_peered":"2024-02-05T06:33:33.585642+0000","last_unstale":"2024-02-05T06:37:42.999532+0000","last_undegraded":"2024-02-05T06:37:42.999532+0000","last_fullsized":"2024-02-05T06:37:42.999532+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:36:27.952573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:27.952573+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":186,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.970814+0000","last_change":"2024-02-05T06:37:34.651569+0000","last_active":"2024-02-05T06:37:42.970814+0000","last_peered":"2024-02-05T06:37:42.970814+0000","last_clean":"2024-02-05T06:37:42.970814+0000","last_became_active":"2024-02-05T06:36:54.969248+0000","last_became_peered":"2024-02-05T06:36:54.969248+0000","last_unstale":"2024-02-05T06:37:42.970814+0000","last_undegraded":"2024-02-05T06:37:42.970814+0000","last_fullsized":"2024-02-05T06:37:42.970814+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:34.651493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:37:34.651493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:56.852181+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'10","reported_seq":191,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.966070+0000","last_change":"2024-02-05T06:36:12.205255+0000","last_active":"2024-02-05T06:37:40.966070+0000","last_peered":"2024-02-05T06:37:40.966070+0000","last_clean":"2024-02-05T06:37:40.966070+0000","last_became_active":"2024-02-05T06:36:12.171679+0000","last_became_peered":"2024-02-05T06:36:12.171679+0000","last_unstale":"2024-02-05T06:37:40.966070+0000","last_undegraded":"2024-02-05T06:37:40.966070+0000","last_fullsized":"2024-02-05T06:37:40.966070+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-02-05T06:34:41.873876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:41.873876+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":51,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.966942+0000","last_change":"2024-02-05T06:36:54.963383+0000","last_active":"2024-02-05T06:37:42.966942+0000","last_peered":"2024-02-05T06:37:42.966942+0000","last_clean":"2024-02-05T06:37:42.966942+0000","last_became_active":"2024-02-05T06:36:54.961617+0000","last_became_peered":"2024-02-05T06:36:54.961617+0000","last_unstale":"2024-02-05T06:37:42.966942+0000","last_undegraded":"2024-02-05T06:37:42.966942+0000","last_fullsized":"2024-02-05T06:37:42.966942+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:12.594735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:12.594735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":221,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.966469+0000","last_change":"2024-02-05T06:36:26.411694+0000","last_active":"2024-02-05T06:37:40.966469+0000","last_peered":"2024-02-05T06:37:40.966469+0000","last_clean":"2024-02-05T06:37:40.966469+0000","last_became_active":"2024-02-05T06:33:33.586043+0000","last_became_peered":"2024-02-05T06:33:33.586043+0000","last_unstale":"2024-02-05T06:37:40.966469+0000","last_undegraded":"2024-02-05T06:37:40.966469+0000","last_fullsized":"2024-02-05T06:37:40.966469+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:36:26.411576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:26.411576+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":170,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.963522+0000","last_change":"2024-02-05T06:36:31.169821+0000","last_active":"2024-02-05T06:37:40.963522+0000","last_peered":"2024-02-05T06:37:40.963522+0000","last_clean":"2024-02-05T06:37:40.963522+0000","last_became_active":"2024-02-05T06:33:29.552170+0000","last_became_peered":"2024-02-05T06:33:29.552170+0000","last_unstale":"2024-02-05T06:37:40.963522+0000","last_undegraded":"2024-02-05T06:37:40.963522+0000","last_fullsized":"2024-02-05T06:37:40.963522+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:31.169691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:36:31.169691+0000","last_clean_scrub_stamp":"2024-02-05T06:36:31.169691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:53.607772+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"73'9","reported_seq":200,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.999605+0000","last_change":"2024-02-05T06:36:12.206355+0000","last_active":"2024-02-05T06:37:42.999605+0000","last_peered":"2024-02-05T06:37:42.999605+0000","last_clean":"2024-02-05T06:37:42.999605+0000","last_became_active":"2024-02-05T06:36:12.173982+0000","last_became_peered":"2024-02-05T06:36:12.173982+0000","last_unstale":"2024-02-05T06:37:42.999605+0000","last_undegraded":"2024-02-05T06:37:42.999605+0000","last_fullsized":"2024-02-05T06:37:42.999605+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:36:09.760345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:09.760345+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":176,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.996846+0000","last_change":"2024-02-05T06:37:23.504416+0000","last_active":"2024-02-05T06:37:40.996846+0000","last_peered":"2024-02-05T06:37:40.996846+0000","last_clean":"2024-02-05T06:37:40.996846+0000","last_became_active":"2024-02-05T06:36:54.958065+0000","last_became_peered":"2024-02-05T06:36:54.958065+0000","last_unstale":"2024-02-05T06:37:40.996846+0000","last_undegraded":"2024-02-05T06:37:40.996846+0000","last_fullsized":"2024-02-05T06:37:40.996846+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:23.504303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:37:23.504303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:27.269693+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":162,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.966379+0000","last_change":"2024-02-05T06:36:25.378909+0000","last_active":"2024-02-05T06:37:40.966379+0000","last_peered":"2024-02-05T06:37:40.966379+0000","last_clean":"2024-02-05T06:37:40.966379+0000","last_became_active":"2024-02-05T06:33:33.593025+0000","last_became_peered":"2024-02-05T06:33:33.593025+0000","last_unstale":"2024-02-05T06:37:40.966379+0000","last_undegraded":"2024-02-05T06:37:40.966379+0000","last_fullsized":"2024-02-05T06:37:40.966379+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:25.378404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:25.378404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:49.015657+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":186,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.970619+0000","last_change":"2024-02-05T06:36:54.972800+0000","last_active":"2024-02-05T06:37:42.970619+0000","last_peered":"2024-02-05T06:37:42.970619+0000","last_clean":"2024-02-05T06:37:42.970619+0000","last_became_active":"2024-02-05T06:36:54.972105+0000","last_became_peered":"2024-02-05T06:36:54.972105+0000","last_unstale":"2024-02-05T06:37:42.970619+0000","last_undegraded":"2024-02-05T06:37:42.970619+0000","last_fullsized":"2024-02-05T06:37:42.970619+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:18.895623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:18.895623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"73'9","reported_seq":182,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.966985+0000","last_change":"2024-02-05T06:36:23.148667+0000","last_active":"2024-02-05T06:37:40.966985+0000","last_peered":"2024-02-05T06:37:40.966985+0000","last_clean":"2024-02-05T06:37:40.966985+0000","last_became_active":"2024-02-05T06:33:31.665766+0000","last_became_peered":"2024-02-05T06:33:31.665766+0000","last_unstale":"2024-02-05T06:37:40.966985+0000","last_undegraded":"2024-02-05T06:37:40.966985+0000","last_fullsized":"2024-02-05T06:37:40.966985+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:36:23.148541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:23.148541+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":174,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.985958+0000","last_change":"2024-02-05T06:36:54.970154+0000","last_active":"2024-02-05T06:37:42.985958+0000","last_peered":"2024-02-05T06:37:42.985958+0000","last_clean":"2024-02-05T06:37:42.985958+0000","last_became_active":"2024-02-05T06:36:54.969661+0000","last_became_peered":"2024-02-05T06:36:54.969661+0000","last_unstale":"2024-02-05T06:37:42.985958+0000","last_undegraded":"2024-02-05T06:37:42.985958+0000","last_fullsized":"2024-02-05T06:37:42.985958+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:29.845506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:29.845506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:56.231851+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":164,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:43.000333+0000","last_change":"2024-02-05T06:36:20.832672+0000","last_active":"2024-02-05T06:37:43.000333+0000","last_peered":"2024-02-05T06:37:43.000333+0000","last_clean":"2024-02-05T06:37:43.000333+0000","last_became_active":"2024-02-05T06:33:33.586768+0000","last_became_peered":"2024-02-05T06:33:33.586768+0000","last_unstale":"2024-02-05T06:37:43.000333+0000","last_undegraded":"2024-02-05T06:37:43.000333+0000","last_fullsized":"2024-02-05T06:37:43.000333+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:20.832579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:45.096340+0000","last_clean_scrub_stamp":"2024-02-05T06:36:20.832579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":181,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:43.002360+0000","last_change":"2024-02-05T06:37:35.737037+0000","last_active":"2024-02-05T06:37:43.002360+0000","last_peered":"2024-02-05T06:37:43.002360+0000","last_clean":"2024-02-05T06:37:43.002360+0000","last_became_active":"2024-02-05T06:33:29.554427+0000","last_became_peered":"2024-02-05T06:33:29.554427+0000","last_unstale":"2024-02-05T06:37:43.002360+0000","last_undegraded":"2024-02-05T06:37:43.002360+0000","last_fullsized":"2024-02-05T06:37:43.002360+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:35.736913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:37:35.736913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:49.460223+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":167,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.966575+0000","last_change":"2024-02-05T06:34:55.912380+0000","last_active":"2024-02-05T06:37:40.966575+0000","last_peered":"2024-02-05T06:37:40.966575+0000","last_clean":"2024-02-05T06:37:40.966575+0000","last_became_active":"2024-02-05T06:33:31.665847+0000","last_became_peered":"2024-02-05T06:33:31.665847+0000","last_unstale":"2024-02-05T06:37:40.966575+0000","last_undegraded":"2024-02-05T06:37:40.966575+0000","last_fullsized":"2024-02-05T06:37:40.966575+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'6","last_scrub_stamp":"2024-02-05T06:34:55.912282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:34:55.912282+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"137'8907","reported_seq":8999,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:41.146579+0000","last_change":"2024-02-05T06:35:27.937770+0000","last_active":"2024-02-05T06:37:41.146579+0000","last_peered":"2024-02-05T06:37:41.146579+0000","last_clean":"2024-02-05T06:37:41.146579+0000","last_became_active":"2024-02-05T06:35:27.937512+0000","last_became_peered":"2024-02-05T06:35:27.937512+0000","last_unstale":"2024-02-05T06:37:41.146579+0000","last_undegraded":"2024-02-05T06:37:41.146579+0000","last_fullsized":"2024-02-05T06:37:41.146579+0000","mapping_epoch":96,"log_start":"110'3966","ondisk_log_start":"110'3966","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":4941,"log_dups_size":0,"ondisk_log_size":4941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353173504,"num_objects":5389,"num_object_clones":0,"num_object_copies":16167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8907,"num_write_kb":457472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":160,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.999192+0000","last_change":"2024-02-05T06:36:33.055072+0000","last_active":"2024-02-05T06:37:42.999192+0000","last_peered":"2024-02-05T06:37:42.999192+0000","last_clean":"2024-02-05T06:37:42.999192+0000","last_became_active":"2024-02-05T06:33:35.665581+0000","last_became_peered":"2024-02-05T06:33:35.665581+0000","last_unstale":"2024-02-05T06:37:42.999192+0000","last_undegraded":"2024-02-05T06:37:42.999192+0000","last_fullsized":"2024-02-05T06:37:42.999192+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:33.054967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:33.054967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":210,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.970836+0000","last_change":"2024-02-05T06:37:28.708906+0000","last_active":"2024-02-05T06:37:42.970836+0000","last_peered":"2024-02-05T06:37:42.970836+0000","last_clean":"2024-02-05T06:37:42.970836+0000","last_became_active":"2024-02-05T06:36:54.970234+0000","last_became_peered":"2024-02-05T06:36:54.970234+0000","last_unstale":"2024-02-05T06:37:42.970836+0000","last_undegraded":"2024-02-05T06:37:42.970836+0000","last_fullsized":"2024-02-05T06:37:42.970836+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-02-05T06:37:28.708774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:37:28.708774+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:36.763830+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"73'15","reported_seq":193,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.998726+0000","last_change":"2024-02-05T06:37:29.261862+0000","last_active":"2024-02-05T06:37:42.998726+0000","last_peered":"2024-02-05T06:37:42.998726+0000","last_clean":"2024-02-05T06:37:42.998726+0000","last_became_active":"2024-02-05T06:33:31.666056+0000","last_became_peered":"2024-02-05T06:33:31.666056+0000","last_unstale":"2024-02-05T06:37:42.998726+0000","last_undegraded":"2024-02-05T06:37:42.998726+0000","last_fullsized":"2024-02-05T06:37:42.998726+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-02-05T06:37:29.261710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:37:29.261710+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:43.272587+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"119'10","reported_seq":248,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.970768+0000","last_change":"2024-02-05T06:36:54.985467+0000","last_active":"2024-02-05T06:37:42.970768+0000","last_peered":"2024-02-05T06:37:42.970768+0000","last_clean":"2024-02-05T06:37:42.970768+0000","last_became_active":"2024-02-05T06:36:54.970348+0000","last_became_peered":"2024-02-05T06:36:54.970348+0000","last_unstale":"2024-02-05T06:37:42.970768+0000","last_undegraded":"2024-02-05T06:37:42.970768+0000","last_fullsized":"2024-02-05T06:37:42.970768+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'9","last_scrub_stamp":"2024-02-05T06:36:32.863645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:32.863645+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"137'8925","reported_seq":9018,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:41.167705+0000","last_change":"2024-02-05T06:35:27.933615+0000","last_active":"2024-02-05T06:37:41.167705+0000","last_peered":"2024-02-05T06:37:41.167705+0000","last_clean":"2024-02-05T06:37:41.167705+0000","last_became_active":"2024-02-05T06:35:27.933340+0000","last_became_peered":"2024-02-05T06:35:27.933340+0000","last_unstale":"2024-02-05T06:37:41.167705+0000","last_undegraded":"2024-02-05T06:37:41.167705+0000","last_fullsized":"2024-02-05T06:37:41.167705+0000","mapping_epoch":96,"log_start":"112'4624","ondisk_log_start":"112'4624","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":4301,"log_dups_size":0,"ondisk_log_size":4301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353828864,"num_objects":5399,"num_object_clones":0,"num_object_copies":16197,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8925,"num_write_kb":458368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":150,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.966149+0000","last_change":"2024-02-05T06:35:12.099300+0000","last_active":"2024-02-05T06:37:40.966149+0000","last_peered":"2024-02-05T06:37:40.966149+0000","last_clean":"2024-02-05T06:37:40.966149+0000","last_became_active":"2024-02-05T06:33:35.601030+0000","last_became_peered":"2024-02-05T06:33:35.601030+0000","last_unstale":"2024-02-05T06:37:40.966149+0000","last_undegraded":"2024-02-05T06:37:40.966149+0000","last_fullsized":"2024-02-05T06:37:40.966149+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:12.099209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:12.099209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":186,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.971245+0000","last_change":"2024-02-05T06:36:54.970978+0000","last_active":"2024-02-05T06:37:42.971245+0000","last_peered":"2024-02-05T06:37:42.971245+0000","last_clean":"2024-02-05T06:37:42.971245+0000","last_became_active":"2024-02-05T06:36:54.970400+0000","last_became_peered":"2024-02-05T06:36:54.970400+0000","last_unstale":"2024-02-05T06:37:42.971245+0000","last_undegraded":"2024-02-05T06:37:42.971245+0000","last_fullsized":"2024-02-05T06:37:42.971245+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:33.825033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:33.825033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:53.454150+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'12","reported_seq":63,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.966163+0000","last_change":"2024-02-05T06:36:54.986580+0000","last_active":"2024-02-05T06:37:42.966163+0000","last_peered":"2024-02-05T06:37:42.966163+0000","last_clean":"2024-02-05T06:37:42.966163+0000","last_became_active":"2024-02-05T06:36:54.954071+0000","last_became_peered":"2024-02-05T06:36:54.954071+0000","last_unstale":"2024-02-05T06:37:42.966163+0000","last_undegraded":"2024-02-05T06:37:42.966163+0000","last_fullsized":"2024-02-05T06:37:42.966163+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:35:00.650542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:35:00.650542+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":160,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.995104+0000","last_change":"2024-02-05T06:37:25.496202+0000","last_active":"2024-02-05T06:37:40.995104+0000","last_peered":"2024-02-05T06:37:40.995104+0000","last_clean":"2024-02-05T06:37:40.995104+0000","last_became_active":"2024-02-05T06:33:33.592546+0000","last_became_peered":"2024-02-05T06:33:33.592546+0000","last_unstale":"2024-02-05T06:37:40.995104+0000","last_undegraded":"2024-02-05T06:37:40.995104+0000","last_fullsized":"2024-02-05T06:37:40.995104+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:25.496091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:37:25.496091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:55.131421+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"136'8937","reported_seq":21877,"reported_epoch":137,"state":"peering","last_fresh":"2024-02-05T06:37:40.966434+0000","last_change":"2024-02-05T06:37:40.966434+0000","last_active":"2024-02-05T06:37:40.966434+0000","last_peered":"2024-02-05T06:37:40.421833+0000","last_clean":"2024-02-05T06:37:40.421833+0000","last_became_active":"2024-02-05T06:36:56.030709+0000","last_became_peered":"2024-02-05T06:36:56.030709+0000","last_unstale":"2024-02-05T06:37:40.966434+0000","last_undegraded":"2024-02-05T06:37:40.966434+0000","last_fullsized":"2024-02-05T06:37:40.966434+0000","mapping_epoch":137,"log_start":"110'3864","ondisk_log_start":"110'3864","created":96,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":5073,"log_dups_size":0,"ondisk_log_size":5073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":354353152,"num_objects":5407,"num_object_clones":0,"num_object_copies":16221,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5407,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8937,"num_write_kb":459008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6420,"num_bytes_recovered":420741120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":162,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.970333+0000","last_change":"2024-02-05T06:37:35.617004+0000","last_active":"2024-02-05T06:37:42.970333+0000","last_peered":"2024-02-05T06:37:42.970333+0000","last_clean":"2024-02-05T06:37:42.970333+0000","last_became_active":"2024-02-05T06:33:35.599527+0000","last_became_peered":"2024-02-05T06:33:35.599527+0000","last_unstale":"2024-02-05T06:37:42.970333+0000","last_undegraded":"2024-02-05T06:37:42.970333+0000","last_fullsized":"2024-02-05T06:37:42.970333+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:35.616896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:37:35.616896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:53.773149+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":160,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.996339+0000","last_change":"2024-02-05T06:36:16.649604+0000","last_active":"2024-02-05T06:37:40.996339+0000","last_peered":"2024-02-05T06:37:40.996339+0000","last_clean":"2024-02-05T06:37:40.996339+0000","last_became_active":"2024-02-05T06:33:33.589203+0000","last_became_peered":"2024-02-05T06:33:33.589203+0000","last_unstale":"2024-02-05T06:37:40.996339+0000","last_undegraded":"2024-02-05T06:37:40.996339+0000","last_fullsized":"2024-02-05T06:37:40.996339+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:16.649483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:16.649483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:43.871278+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":174,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.986121+0000","last_change":"2024-02-05T06:37:30.671399+0000","last_active":"2024-02-05T06:37:42.986121+0000","last_peered":"2024-02-05T06:37:42.986121+0000","last_clean":"2024-02-05T06:37:42.986121+0000","last_became_active":"2024-02-05T06:33:29.556541+0000","last_became_peered":"2024-02-05T06:33:29.556541+0000","last_unstale":"2024-02-05T06:37:42.986121+0000","last_undegraded":"2024-02-05T06:37:42.986121+0000","last_fullsized":"2024-02-05T06:37:42.986121+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:30.671274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:37:30.671274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:55.485966+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"73'19","reported_seq":208,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:43.002578+0000","last_change":"2024-02-05T06:37:42.815719+0000","last_active":"2024-02-05T06:37:43.002578+0000","last_peered":"2024-02-05T06:37:43.002578+0000","last_clean":"2024-02-05T06:37:43.002578+0000","last_became_active":"2024-02-05T06:33:31.664699+0000","last_became_peered":"2024-02-05T06:33:31.664699+0000","last_unstale":"2024-02-05T06:37:43.002578+0000","last_undegraded":"2024-02-05T06:37:43.002578+0000","last_fullsized":"2024-02-05T06:37:43.002578+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-02-05T06:37:42.815623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:37:42.815623+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:39:11.398466+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"137'9053","reported_seq":9156,"reported_epoch":137,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T06:37:41.153795+0000","last_change":"2024-02-05T06:37:26.064778+0000","last_active":"2024-02-05T06:37:41.153795+0000","last_peered":"2024-02-05T06:37:41.153795+0000","last_clean":"2024-02-05T06:36:54.240117+0000","last_became_active":"2024-02-05T06:36:56.042585+0000","last_became_peered":"2024-02-05T06:36:56.042585+0000","last_unstale":"2024-02-05T06:37:41.153795+0000","last_undegraded":"2024-02-05T06:37:41.153795+0000","last_fullsized":"2024-02-05T06:37:41.153795+0000","mapping_epoch":119,"log_start":"105'3168","ondisk_log_start":"105'3168","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":5885,"log_dups_size":0,"ondisk_log_size":5885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359333888,"num_objects":5483,"num_object_clones":0,"num_object_copies":16449,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5483,"num_objects_unfound":0,"num_objects_dirty":5483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9053,"num_write_kb":465152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,3],"avail_no_missing":["2","3","7"],"object_location_counts":[{"shards":"2,3,7","objects":5483}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":150,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.964341+0000","last_change":"2024-02-05T06:36:22.102208+0000","last_active":"2024-02-05T06:37:40.964341+0000","last_peered":"2024-02-05T06:37:40.964341+0000","last_clean":"2024-02-05T06:37:40.964341+0000","last_became_active":"2024-02-05T06:33:35.597863+0000","last_became_peered":"2024-02-05T06:33:35.597863+0000","last_unstale":"2024-02-05T06:37:40.964341+0000","last_undegraded":"2024-02-05T06:37:40.964341+0000","last_fullsized":"2024-02-05T06:37:40.964341+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:22.102133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:22.102133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:47.960776+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":172,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.996889+0000","last_change":"2024-02-05T06:36:54.967406+0000","last_active":"2024-02-05T06:37:40.996889+0000","last_peered":"2024-02-05T06:37:40.996889+0000","last_clean":"2024-02-05T06:37:40.996889+0000","last_became_active":"2024-02-05T06:36:54.966016+0000","last_became_peered":"2024-02-05T06:36:54.966016+0000","last_unstale":"2024-02-05T06:37:40.996889+0000","last_undegraded":"2024-02-05T06:37:40.996889+0000","last_fullsized":"2024-02-05T06:37:40.996889+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:31.737492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:31.737492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":172,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.998925+0000","last_change":"2024-02-05T06:37:27.314428+0000","last_active":"2024-02-05T06:37:42.998925+0000","last_peered":"2024-02-05T06:37:42.998925+0000","last_clean":"2024-02-05T06:37:42.998925+0000","last_became_active":"2024-02-05T06:33:29.555950+0000","last_became_peered":"2024-02-05T06:33:29.555950+0000","last_unstale":"2024-02-05T06:37:42.998925+0000","last_undegraded":"2024-02-05T06:37:42.998925+0000","last_fullsized":"2024-02-05T06:37:42.998925+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:27.314329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:37:27.314329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:56.362407+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"73'9","reported_seq":230,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.963856+0000","last_change":"2024-02-05T06:36:54.995496+0000","last_active":"2024-02-05T06:37:40.963856+0000","last_peered":"2024-02-05T06:37:40.963856+0000","last_clean":"2024-02-05T06:37:40.963856+0000","last_became_active":"2024-02-05T06:36:54.973958+0000","last_became_peered":"2024-02-05T06:36:54.973958+0000","last_unstale":"2024-02-05T06:37:40.963856+0000","last_undegraded":"2024-02-05T06:37:40.963856+0000","last_fullsized":"2024-02-05T06:37:40.963856+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:36:26.172612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:26.172612+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:52.634290+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":328,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.998495+0000","last_change":"2024-02-05T06:37:37.282551+0000","last_active":"2024-02-05T06:37:42.998495+0000","last_peered":"2024-02-05T06:37:42.998495+0000","last_clean":"2024-02-05T06:37:42.998495+0000","last_became_active":"2024-02-05T06:37:37.136524+0000","last_became_peered":"2024-02-05T06:37:37.136524+0000","last_unstale":"2024-02-05T06:37:42.998495+0000","last_undegraded":"2024-02-05T06:37:42.998495+0000","last_fullsized":"2024-02-05T06:37:42.998495+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:37:37.282442+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:37.282442+0000","last_clean_scrub_stamp":"2024-02-05T06:37:37.282442+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:44.510220+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"139'9093","reported_seq":9258,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:46.149572+0000","last_change":"2024-02-05T06:37:32.797946+0000","last_active":"2024-02-05T06:37:46.149572+0000","last_peered":"2024-02-05T06:37:46.149572+0000","last_clean":"2024-02-05T06:37:46.149572+0000","last_became_active":"2024-02-05T06:35:27.934093+0000","last_became_peered":"2024-02-05T06:35:27.934093+0000","last_unstale":"2024-02-05T06:37:46.149572+0000","last_undegraded":"2024-02-05T06:37:46.149572+0000","last_fullsized":"2024-02-05T06:37:46.149572+0000","mapping_epoch":96,"log_start":"109'3732","ondisk_log_start":"109'3732","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'8513","last_scrub_stamp":"2024-02-05T06:37:32.797678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:37:32.797678+0000","objects_scrubbed":5923,"log_size":5361,"log_dups_size":0,"ondisk_log_size":5361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:51.272291+0000","scrub_duration":1978,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347799552,"num_objects":5307,"num_object_clones":0,"num_object_copies":15921,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9093,"num_write_kb":460800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":162,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.967093+0000","last_change":"2024-02-05T06:36:54.955665+0000","last_active":"2024-02-05T06:37:40.967093+0000","last_peered":"2024-02-05T06:37:40.967093+0000","last_clean":"2024-02-05T06:37:40.967093+0000","last_became_active":"2024-02-05T06:36:54.955311+0000","last_became_peered":"2024-02-05T06:36:54.955311+0000","last_unstale":"2024-02-05T06:37:40.967093+0000","last_undegraded":"2024-02-05T06:37:40.967093+0000","last_fullsized":"2024-02-05T06:37:40.967093+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:14.082774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:14.082774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":184,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.998954+0000","last_change":"2024-02-05T06:36:14.780874+0000","last_active":"2024-02-05T06:37:42.998954+0000","last_peered":"2024-02-05T06:37:42.998954+0000","last_clean":"2024-02-05T06:37:42.998954+0000","last_became_active":"2024-02-05T06:33:28.542228+0000","last_became_peered":"2024-02-05T06:33:28.542228+0000","last_unstale":"2024-02-05T06:37:42.998954+0000","last_undegraded":"2024-02-05T06:37:42.998954+0000","last_fullsized":"2024-02-05T06:37:42.998954+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-02-05T06:36:14.780738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:36:14.780738+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001220348,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":51,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.966978+0000","last_change":"2024-02-05T06:36:54.968582+0000","last_active":"2024-02-05T06:37:42.966978+0000","last_peered":"2024-02-05T06:37:42.966978+0000","last_clean":"2024-02-05T06:37:42.966978+0000","last_became_active":"2024-02-05T06:36:54.968205+0000","last_became_peered":"2024-02-05T06:36:54.968205+0000","last_unstale":"2024-02-05T06:37:42.966978+0000","last_undegraded":"2024-02-05T06:37:42.966978+0000","last_fullsized":"2024-02-05T06:37:42.966978+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:15.166979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:15.166979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"78'29","reported_seq":218,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.998745+0000","last_change":"2024-02-05T06:36:35.079298+0000","last_active":"2024-02-05T06:37:42.998745+0000","last_peered":"2024-02-05T06:37:42.998745+0000","last_clean":"2024-02-05T06:37:42.998745+0000","last_became_active":"2024-02-05T06:33:55.884601+0000","last_became_peered":"2024-02-05T06:33:55.884601+0000","last_unstale":"2024-02-05T06:37:42.998745+0000","last_undegraded":"2024-02-05T06:37:42.998745+0000","last_fullsized":"2024-02-05T06:37:42.998745+0000","mapping_epoch":77,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":78,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-02-05T06:36:35.079170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:35.079170+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":174,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.964675+0000","last_change":"2024-02-05T06:36:54.974218+0000","last_active":"2024-02-05T06:37:40.964675+0000","last_peered":"2024-02-05T06:37:40.964675+0000","last_clean":"2024-02-05T06:37:40.964675+0000","last_became_active":"2024-02-05T06:36:54.973387+0000","last_became_peered":"2024-02-05T06:36:54.973387+0000","last_unstale":"2024-02-05T06:37:40.964675+0000","last_undegraded":"2024-02-05T06:37:40.964675+0000","last_fullsized":"2024-02-05T06:37:40.964675+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:36.161606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:36.161606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":446,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.970576+0000","last_change":"2024-02-05T06:37:38.667533+0000","last_active":"2024-02-05T06:37:42.970576+0000","last_peered":"2024-02-05T06:37:42.970576+0000","last_clean":"2024-02-05T06:37:42.970576+0000","last_became_active":"2024-02-05T06:37:37.141918+0000","last_became_peered":"2024-02-05T06:37:37.141918+0000","last_unstale":"2024-02-05T06:37:42.970576+0000","last_undegraded":"2024-02-05T06:37:42.970576+0000","last_fullsized":"2024-02-05T06:37:42.970576+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:37:38.667413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:37:38.667413+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:46.950046+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"137'8987","reported_seq":21984,"reported_epoch":137,"state":"active+clean+remapped","last_fresh":"2024-02-05T06:37:41.191716+0000","last_change":"2024-02-05T06:37:41.191236+0000","last_active":"2024-02-05T06:37:41.191716+0000","last_peered":"2024-02-05T06:37:41.191716+0000","last_clean":"2024-02-05T06:37:41.191716+0000","last_became_active":"2024-02-05T06:36:56.033693+0000","last_became_peered":"2024-02-05T06:36:56.033693+0000","last_unstale":"2024-02-05T06:37:41.191716+0000","last_undegraded":"2024-02-05T06:37:41.191716+0000","last_fullsized":"2024-02-05T06:37:41.191716+0000","mapping_epoch":119,"log_start":"110'3931","ondisk_log_start":"110'3931","created":96,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":5056,"log_dups_size":0,"ondisk_log_size":5056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":359464960,"num_objects":5485,"num_object_clones":0,"num_object_copies":16455,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5485,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8987,"num_write_kb":463104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6444,"num_bytes_recovered":422313984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,1,6],"avail_no_missing":["0","1","4","6"],"object_location_counts":[{"shards":"0,1,6","objects":5485}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":163,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.986204+0000","last_change":"2024-02-05T06:36:16.887854+0000","last_active":"2024-02-05T06:37:42.986204+0000","last_peered":"2024-02-05T06:37:42.986204+0000","last_clean":"2024-02-05T06:37:42.986204+0000","last_became_active":"2024-02-05T06:33:35.666324+0000","last_became_peered":"2024-02-05T06:33:35.666324+0000","last_unstale":"2024-02-05T06:37:42.986204+0000","last_undegraded":"2024-02-05T06:37:42.986204+0000","last_fullsized":"2024-02-05T06:37:42.986204+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:16.887732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:16.887732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":162,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.967939+0000","last_change":"2024-02-05T06:35:21.506625+0000","last_active":"2024-02-05T06:37:40.967939+0000","last_peered":"2024-02-05T06:37:40.967939+0000","last_clean":"2024-02-05T06:37:40.967939+0000","last_became_active":"2024-02-05T06:33:29.551549+0000","last_became_peered":"2024-02-05T06:33:29.551549+0000","last_unstale":"2024-02-05T06:37:40.967939+0000","last_undegraded":"2024-02-05T06:37:40.967939+0000","last_fullsized":"2024-02-05T06:37:40.967939+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:21.506519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:35:21.506519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":186,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.994928+0000","last_change":"2024-02-05T06:37:31.522883+0000","last_active":"2024-02-05T06:37:40.994928+0000","last_peered":"2024-02-05T06:37:40.994928+0000","last_clean":"2024-02-05T06:37:40.994928+0000","last_became_active":"2024-02-05T06:33:28.542977+0000","last_became_peered":"2024-02-05T06:33:28.542977+0000","last_unstale":"2024-02-05T06:37:40.994928+0000","last_undegraded":"2024-02-05T06:37:40.994928+0000","last_fullsized":"2024-02-05T06:37:40.994928+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:31.522765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:37:31.522765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:59.449606+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0.00091640799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'13","reported_seq":192,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.968607+0000","last_change":"2024-02-05T06:36:14.209527+0000","last_active":"2024-02-05T06:37:40.968607+0000","last_peered":"2024-02-05T06:37:40.968607+0000","last_clean":"2024-02-05T06:37:40.968607+0000","last_became_active":"2024-02-05T06:33:31.560082+0000","last_became_peered":"2024-02-05T06:33:31.560082+0000","last_unstale":"2024-02-05T06:37:40.968607+0000","last_undegraded":"2024-02-05T06:37:40.968607+0000","last_fullsized":"2024-02-05T06:37:40.968607+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-02-05T06:36:14.209318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:14.209318+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":168,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.997197+0000","last_change":"2024-02-05T06:37:32.508703+0000","last_active":"2024-02-05T06:37:40.997197+0000","last_peered":"2024-02-05T06:37:40.997197+0000","last_clean":"2024-02-05T06:37:40.997197+0000","last_became_active":"2024-02-05T06:33:33.587593+0000","last_became_peered":"2024-02-05T06:33:33.587593+0000","last_unstale":"2024-02-05T06:37:40.997197+0000","last_undegraded":"2024-02-05T06:37:40.997197+0000","last_fullsized":"2024-02-05T06:37:40.997197+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:32.507579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:37:32.507579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:37.733629+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"137'8756","reported_seq":8849,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:41.171060+0000","last_change":"2024-02-05T06:35:27.935172+0000","last_active":"2024-02-05T06:37:41.171060+0000","last_peered":"2024-02-05T06:37:41.171060+0000","last_clean":"2024-02-05T06:37:41.171060+0000","last_became_active":"2024-02-05T06:35:27.934809+0000","last_became_peered":"2024-02-05T06:35:27.934809+0000","last_unstale":"2024-02-05T06:37:41.171060+0000","last_undegraded":"2024-02-05T06:37:41.171060+0000","last_fullsized":"2024-02-05T06:37:41.171060+0000","mapping_epoch":96,"log_start":"110'3717","ondisk_log_start":"110'3717","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":5039,"log_dups_size":0,"ondisk_log_size":5039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346423296,"num_objects":5286,"num_object_clones":0,"num_object_copies":15858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5286,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8756,"num_write_kb":449344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":164,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.999805+0000","last_change":"2024-02-05T06:36:54.951238+0000","last_active":"2024-02-05T06:37:42.999805+0000","last_peered":"2024-02-05T06:37:42.999805+0000","last_clean":"2024-02-05T06:37:42.999805+0000","last_became_active":"2024-02-05T06:36:54.950929+0000","last_became_peered":"2024-02-05T06:36:54.950929+0000","last_unstale":"2024-02-05T06:37:42.999805+0000","last_undegraded":"2024-02-05T06:37:42.999805+0000","last_fullsized":"2024-02-05T06:37:42.999805+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:05.712737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:05.712737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":438,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.970441+0000","last_change":"2024-02-05T06:37:32.595616+0000","last_active":"2024-02-05T06:37:42.970441+0000","last_peered":"2024-02-05T06:37:42.970441+0000","last_clean":"2024-02-05T06:37:42.970441+0000","last_became_active":"2024-02-05T06:37:32.073854+0000","last_became_peered":"2024-02-05T06:37:32.073854+0000","last_unstale":"2024-02-05T06:37:42.970441+0000","last_undegraded":"2024-02-05T06:37:42.970441+0000","last_fullsized":"2024-02-05T06:37:42.970441+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:37:32.595437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:37:32.595437+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:44.051403+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":163,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.965587+0000","last_change":"2024-02-05T06:34:44.894533+0000","last_active":"2024-02-05T06:37:40.965587+0000","last_peered":"2024-02-05T06:37:40.965587+0000","last_clean":"2024-02-05T06:37:40.965587+0000","last_became_active":"2024-02-05T06:33:29.552698+0000","last_became_peered":"2024-02-05T06:33:29.552698+0000","last_unstale":"2024-02-05T06:37:40.965587+0000","last_undegraded":"2024-02-05T06:37:40.965587+0000","last_fullsized":"2024-02-05T06:37:40.965587+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:44.894443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:34:44.894443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"119'18","reported_seq":294,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:44.647916+0000","last_change":"2024-02-05T06:37:27.842875+0000","last_active":"2024-02-05T06:37:44.647916+0000","last_peered":"2024-02-05T06:37:44.647916+0000","last_clean":"2024-02-05T06:37:44.647916+0000","last_became_active":"2024-02-05T06:36:54.966277+0000","last_became_peered":"2024-02-05T06:36:54.966277+0000","last_unstale":"2024-02-05T06:37:44.647916+0000","last_undegraded":"2024-02-05T06:37:44.647916+0000","last_fullsized":"2024-02-05T06:37:44.647916+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'18","last_scrub_stamp":"2024-02-05T06:37:27.842775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:37:27.842775+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:37.373698+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":167,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.971474+0000","last_change":"2024-02-05T06:37:31.636554+0000","last_active":"2024-02-05T06:37:42.971474+0000","last_peered":"2024-02-05T06:37:42.971474+0000","last_clean":"2024-02-05T06:37:42.971474+0000","last_became_active":"2024-02-05T06:33:33.591484+0000","last_became_peered":"2024-02-05T06:33:33.591484+0000","last_unstale":"2024-02-05T06:37:42.971474+0000","last_undegraded":"2024-02-05T06:37:42.971474+0000","last_fullsized":"2024-02-05T06:37:42.971474+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:31.636407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:37:31.636407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:34.034836+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":316,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.968038+0000","last_change":"2024-02-05T06:37:37.138461+0000","last_active":"2024-02-05T06:37:40.968038+0000","last_peered":"2024-02-05T06:37:40.968038+0000","last_clean":"2024-02-05T06:37:40.968038+0000","last_became_active":"2024-02-05T06:37:37.137754+0000","last_became_peered":"2024-02-05T06:37:37.137754+0000","last_unstale":"2024-02-05T06:37:40.968038+0000","last_undegraded":"2024-02-05T06:37:40.968038+0000","last_fullsized":"2024-02-05T06:37:40.968038+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:35:11.952460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:35:11.952460+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:42.333854+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"139'9946","reported_seq":10076,"reported_epoch":139,"state":"active+clean+scrubbing+deep","last_fresh":"2024-02-05T06:37:46.167457+0000","last_change":"2024-02-05T06:37:42.616510+0000","last_active":"2024-02-05T06:37:46.167457+0000","last_peered":"2024-02-05T06:37:46.167457+0000","last_clean":"2024-02-05T06:37:46.167457+0000","last_became_active":"2024-02-05T06:35:27.932989+0000","last_became_peered":"2024-02-05T06:35:27.932989+0000","last_unstale":"2024-02-05T06:37:46.167457+0000","last_undegraded":"2024-02-05T06:37:46.167457+0000","last_fullsized":"2024-02-05T06:37:46.167457+0000","mapping_epoch":96,"log_start":"115'5052","ondisk_log_start":"115'5052","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":774,"log_size":4894,"log_dups_size":0,"ondisk_log_size":4894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 3s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350617600,"num_objects":5350,"num_object_clones":0,"num_object_copies":16050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9172,"num_write_kb":464704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":157,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.968082+0000","last_change":"2024-02-05T06:36:22.097109+0000","last_active":"2024-02-05T06:37:40.968082+0000","last_peered":"2024-02-05T06:37:40.968082+0000","last_clean":"2024-02-05T06:37:40.968082+0000","last_became_active":"2024-02-05T06:33:35.600656+0000","last_became_peered":"2024-02-05T06:33:35.600656+0000","last_unstale":"2024-02-05T06:37:40.968082+0000","last_undegraded":"2024-02-05T06:37:40.968082+0000","last_fullsized":"2024-02-05T06:37:40.968082+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:22.096881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:22.096881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:50.866314+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":183,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.963379+0000","last_change":"2024-02-05T06:36:54.973030+0000","last_active":"2024-02-05T06:37:40.963379+0000","last_peered":"2024-02-05T06:37:40.963379+0000","last_clean":"2024-02-05T06:37:40.963379+0000","last_became_active":"2024-02-05T06:36:54.972264+0000","last_became_peered":"2024-02-05T06:36:54.972264+0000","last_unstale":"2024-02-05T06:37:40.963379+0000","last_undegraded":"2024-02-05T06:37:40.963379+0000","last_fullsized":"2024-02-05T06:37:40.963379+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:24.213373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:24.213373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:52.016463+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":436,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:44.626199+0000","last_change":"2024-02-05T06:36:23.894474+0000","last_active":"2024-02-05T06:37:44.626199+0000","last_peered":"2024-02-05T06:37:44.626199+0000","last_clean":"2024-02-05T06:37:44.626199+0000","last_became_active":"2024-02-05T06:33:28.544852+0000","last_became_peered":"2024-02-05T06:33:28.544852+0000","last_unstale":"2024-02-05T06:37:44.626199+0000","last_undegraded":"2024-02-05T06:37:44.626199+0000","last_fullsized":"2024-02-05T06:37:44.626199+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T06:36:23.894359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:36:23.894359+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0.00074151600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":249,"num_read_kb":244,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":184,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.964907+0000","last_change":"2024-02-05T06:36:29.231478+0000","last_active":"2024-02-05T06:37:40.964907+0000","last_peered":"2024-02-05T06:37:40.964907+0000","last_clean":"2024-02-05T06:37:40.964907+0000","last_became_active":"2024-02-05T06:33:31.564604+0000","last_became_peered":"2024-02-05T06:33:31.564604+0000","last_unstale":"2024-02-05T06:37:40.964907+0000","last_undegraded":"2024-02-05T06:37:40.964907+0000","last_fullsized":"2024-02-05T06:37:40.964907+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:36:29.231352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:29.231352+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:57.204094+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":164,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:43.000036+0000","last_change":"2024-02-05T06:37:40.273683+0000","last_active":"2024-02-05T06:37:43.000036+0000","last_peered":"2024-02-05T06:37:43.000036+0000","last_clean":"2024-02-05T06:37:43.000036+0000","last_became_active":"2024-02-05T06:33:33.589388+0000","last_became_peered":"2024-02-05T06:33:33.589388+0000","last_unstale":"2024-02-05T06:37:43.000036+0000","last_undegraded":"2024-02-05T06:37:43.000036+0000","last_fullsized":"2024-02-05T06:37:43.000036+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:40.273584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:37:40.273584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:52.880089+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":334,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:43.003091+0000","last_change":"2024-02-05T06:37:38.778736+0000","last_active":"2024-02-05T06:37:43.003091+0000","last_peered":"2024-02-05T06:37:43.003091+0000","last_clean":"2024-02-05T06:37:43.003091+0000","last_became_active":"2024-02-05T06:37:37.140011+0000","last_became_peered":"2024-02-05T06:37:37.140011+0000","last_unstale":"2024-02-05T06:37:43.003091+0000","last_undegraded":"2024-02-05T06:37:43.003091+0000","last_fullsized":"2024-02-05T06:37:43.003091+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:37:38.778568+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:38.778568+0000","last_clean_scrub_stamp":"2024-02-05T06:37:38.778568+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:39:02.374386+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"139'9191","reported_seq":9288,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:46.139295+0000","last_change":"2024-02-05T06:35:27.937863+0000","last_active":"2024-02-05T06:37:46.139295+0000","last_peered":"2024-02-05T06:37:46.139295+0000","last_clean":"2024-02-05T06:37:46.139295+0000","last_became_active":"2024-02-05T06:35:27.937618+0000","last_became_peered":"2024-02-05T06:35:27.937618+0000","last_unstale":"2024-02-05T06:37:46.139295+0000","last_undegraded":"2024-02-05T06:37:46.139295+0000","last_fullsized":"2024-02-05T06:37:46.139295+0000","mapping_epoch":96,"log_start":"110'3899","ondisk_log_start":"110'3899","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":5292,"log_dups_size":0,"ondisk_log_size":5292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337575936,"num_objects":5151,"num_object_clones":0,"num_object_copies":15453,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9191,"num_write_kb":458944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":158,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.968819+0000","last_change":"2024-02-05T06:36:33.005959+0000","last_active":"2024-02-05T06:37:40.968819+0000","last_peered":"2024-02-05T06:37:40.968819+0000","last_clean":"2024-02-05T06:37:40.968819+0000","last_became_active":"2024-02-05T06:33:35.664858+0000","last_became_peered":"2024-02-05T06:33:35.664858+0000","last_unstale":"2024-02-05T06:37:40.968819+0000","last_undegraded":"2024-02-05T06:37:40.968819+0000","last_fullsized":"2024-02-05T06:37:40.968819+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:33.005841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:33.005841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:01.594701+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"66'1","reported_seq":176,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.968442+0000","last_change":"2024-02-05T06:36:09.253329+0000","last_active":"2024-02-05T06:37:40.968442+0000","last_peered":"2024-02-05T06:37:40.968442+0000","last_clean":"2024-02-05T06:37:40.968442+0000","last_became_active":"2024-02-05T06:33:29.548227+0000","last_became_peered":"2024-02-05T06:33:29.548227+0000","last_unstale":"2024-02-05T06:37:40.968442+0000","last_undegraded":"2024-02-05T06:37:40.968442+0000","last_fullsized":"2024-02-05T06:37:40.968442+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:36:09.253045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:09.253045+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'19","reported_seq":194,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.964056+0000","last_change":"2024-02-05T06:35:27.592730+0000","last_active":"2024-02-05T06:37:40.964056+0000","last_peered":"2024-02-05T06:37:40.964056+0000","last_clean":"2024-02-05T06:37:40.964056+0000","last_became_active":"2024-02-05T06:33:31.667340+0000","last_became_peered":"2024-02-05T06:33:31.667340+0000","last_unstale":"2024-02-05T06:37:40.964056+0000","last_undegraded":"2024-02-05T06:37:40.964056+0000","last_fullsized":"2024-02-05T06:37:40.964056+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-02-05T06:35:27.592613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:35:27.592613+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":166,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:43.002702+0000","last_change":"2024-02-05T06:37:25.799278+0000","last_active":"2024-02-05T06:37:43.002702+0000","last_peered":"2024-02-05T06:37:43.002702+0000","last_clean":"2024-02-05T06:37:43.002702+0000","last_became_active":"2024-02-05T06:33:33.588163+0000","last_became_peered":"2024-02-05T06:33:33.588163+0000","last_unstale":"2024-02-05T06:37:43.002702+0000","last_undegraded":"2024-02-05T06:37:43.002702+0000","last_fullsized":"2024-02-05T06:37:43.002702+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:25.799164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:37:25.799164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:52.081396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":334,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:43.003525+0000","last_change":"2024-02-05T06:37:37.792761+0000","last_active":"2024-02-05T06:37:43.003525+0000","last_peered":"2024-02-05T06:37:43.003525+0000","last_clean":"2024-02-05T06:37:43.003525+0000","last_became_active":"2024-02-05T06:37:37.136785+0000","last_became_peered":"2024-02-05T06:37:37.136785+0000","last_unstale":"2024-02-05T06:37:43.003525+0000","last_undegraded":"2024-02-05T06:37:43.003525+0000","last_fullsized":"2024-02-05T06:37:43.003525+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:37:37.792683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:37:37.792683+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:48.306915+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"139'9317","reported_seq":22560,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:46.123315+0000","last_change":"2024-02-05T06:37:40.987056+0000","last_active":"2024-02-05T06:37:46.123315+0000","last_peered":"2024-02-05T06:37:46.123315+0000","last_clean":"2024-02-05T06:37:46.123315+0000","last_became_active":"2024-02-05T06:37:40.969634+0000","last_became_peered":"2024-02-05T06:37:40.969634+0000","last_unstale":"2024-02-05T06:37:46.123315+0000","last_undegraded":"2024-02-05T06:37:46.123315+0000","last_fullsized":"2024-02-05T06:37:46.123315+0000","mapping_epoch":136,"log_start":"112'4741","ondisk_log_start":"112'4741","created":96,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":4576,"log_dups_size":0,"ondisk_log_size":4576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350027776,"num_objects":5341,"num_object_clones":0,"num_object_copies":16023,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9317,"num_write_kb":469056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6567,"num_bytes_recovered":430374912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":151,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.963219+0000","last_change":"2024-02-05T06:36:07.154940+0000","last_active":"2024-02-05T06:37:40.963219+0000","last_peered":"2024-02-05T06:37:40.963219+0000","last_clean":"2024-02-05T06:37:40.963219+0000","last_became_active":"2024-02-05T06:33:35.600485+0000","last_became_peered":"2024-02-05T06:33:35.600485+0000","last_unstale":"2024-02-05T06:37:40.963219+0000","last_undegraded":"2024-02-05T06:37:40.963219+0000","last_fullsized":"2024-02-05T06:37:40.963219+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:07.154815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:07.154815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":173,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.999509+0000","last_change":"2024-02-05T06:36:13.768661+0000","last_active":"2024-02-05T06:37:42.999509+0000","last_peered":"2024-02-05T06:37:42.999509+0000","last_clean":"2024-02-05T06:37:42.999509+0000","last_became_active":"2024-02-05T06:33:29.554940+0000","last_became_peered":"2024-02-05T06:33:29.554940+0000","last_unstale":"2024-02-05T06:37:42.999509+0000","last_undegraded":"2024-02-05T06:37:42.999509+0000","last_fullsized":"2024-02-05T06:37:42.999509+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:13.768592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:13.768592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"73'10","reported_seq":208,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.968273+0000","last_change":"2024-02-05T06:36:54.983122+0000","last_active":"2024-02-05T06:37:40.968273+0000","last_peered":"2024-02-05T06:37:40.968273+0000","last_clean":"2024-02-05T06:37:40.968273+0000","last_became_active":"2024-02-05T06:36:54.959010+0000","last_became_peered":"2024-02-05T06:36:54.959010+0000","last_unstale":"2024-02-05T06:37:40.968273+0000","last_undegraded":"2024-02-05T06:37:40.968273+0000","last_fullsized":"2024-02-05T06:37:40.968273+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-02-05T06:36:07.232535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:07.232535+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"73'8","reported_seq":221,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.963718+0000","last_change":"2024-02-05T06:36:32.174510+0000","last_active":"2024-02-05T06:37:40.963718+0000","last_peered":"2024-02-05T06:37:40.963718+0000","last_clean":"2024-02-05T06:37:40.963718+0000","last_became_active":"2024-02-05T06:33:33.582187+0000","last_became_peered":"2024-02-05T06:33:33.582187+0000","last_unstale":"2024-02-05T06:37:40.963718+0000","last_undegraded":"2024-02-05T06:37:40.963718+0000","last_fullsized":"2024-02-05T06:37:40.963718+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:36:32.174396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:32.174396+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":316,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.965773+0000","last_change":"2024-02-05T06:37:37.138219+0000","last_active":"2024-02-05T06:37:40.965773+0000","last_peered":"2024-02-05T06:37:40.965773+0000","last_clean":"2024-02-05T06:37:40.965773+0000","last_became_active":"2024-02-05T06:37:37.137865+0000","last_became_peered":"2024-02-05T06:37:37.137865+0000","last_unstale":"2024-02-05T06:37:40.965773+0000","last_undegraded":"2024-02-05T06:37:40.965773+0000","last_fullsized":"2024-02-05T06:37:40.965773+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:35:11.952460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:35:11.952460+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:42.344934+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":488690,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"137'8997","reported_seq":9090,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:41.196087+0000","last_change":"2024-02-05T06:35:27.936200+0000","last_active":"2024-02-05T06:37:41.196087+0000","last_peered":"2024-02-05T06:37:41.196087+0000","last_clean":"2024-02-05T06:37:41.196087+0000","last_became_active":"2024-02-05T06:35:27.935898+0000","last_became_peered":"2024-02-05T06:35:27.935898+0000","last_unstale":"2024-02-05T06:37:41.196087+0000","last_undegraded":"2024-02-05T06:37:41.196087+0000","last_fullsized":"2024-02-05T06:37:41.196087+0000","mapping_epoch":96,"log_start":"113'4692","ondisk_log_start":"113'4692","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":4305,"log_dups_size":0,"ondisk_log_size":4305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353566720,"num_objects":5395,"num_object_clones":0,"num_object_copies":16185,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8997,"num_write_kb":460544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":163,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.971213+0000","last_change":"2024-02-05T06:36:11.928300+0000","last_active":"2024-02-05T06:37:42.971213+0000","last_peered":"2024-02-05T06:37:42.971213+0000","last_clean":"2024-02-05T06:37:42.971213+0000","last_became_active":"2024-02-05T06:33:35.666315+0000","last_became_peered":"2024-02-05T06:33:35.666315+0000","last_unstale":"2024-02-05T06:37:42.971213+0000","last_undegraded":"2024-02-05T06:37:42.971213+0000","last_fullsized":"2024-02-05T06:37:42.971213+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:11.928194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:11.928194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"95'2","reported_seq":184,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.964980+0000","last_change":"2024-02-05T06:36:54.993662+0000","last_active":"2024-02-05T06:37:40.964980+0000","last_peered":"2024-02-05T06:37:40.964980+0000","last_clean":"2024-02-05T06:37:40.964980+0000","last_became_active":"2024-02-05T06:36:54.975093+0000","last_became_peered":"2024-02-05T06:36:54.975093+0000","last_unstale":"2024-02-05T06:37:40.964980+0000","last_undegraded":"2024-02-05T06:37:40.964980+0000","last_fullsized":"2024-02-05T06:37:40.964980+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-05T06:35:25.616708+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-05T06:35:25.616708+0000","last_clean_scrub_stamp":"2024-02-05T06:35:25.616708+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'14","reported_seq":67,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.966275+0000","last_change":"2024-02-05T06:36:55.007431+0000","last_active":"2024-02-05T06:37:42.966275+0000","last_peered":"2024-02-05T06:37:42.966275+0000","last_clean":"2024-02-05T06:37:42.966275+0000","last_became_active":"2024-02-05T06:36:54.965287+0000","last_became_peered":"2024-02-05T06:36:54.965287+0000","last_unstale":"2024-02-05T06:37:42.966275+0000","last_undegraded":"2024-02-05T06:37:42.966275+0000","last_fullsized":"2024-02-05T06:37:42.966275+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'14","last_scrub_stamp":"2024-02-05T06:36:17.814774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:17.814774+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:46.232951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"119'10","reported_seq":236,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.964212+0000","last_change":"2024-02-05T06:36:54.983684+0000","last_active":"2024-02-05T06:37:40.964212+0000","last_peered":"2024-02-05T06:37:40.964212+0000","last_clean":"2024-02-05T06:37:40.964212+0000","last_became_active":"2024-02-05T06:36:54.954819+0000","last_became_peered":"2024-02-05T06:36:54.954819+0000","last_unstale":"2024-02-05T06:37:40.964212+0000","last_undegraded":"2024-02-05T06:37:40.964212+0000","last_fullsized":"2024-02-05T06:37:40.964212+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:34:56.941840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:56.941840+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":320,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.966138+0000","last_change":"2024-02-05T06:37:37.135467+0000","last_active":"2024-02-05T06:37:42.966138+0000","last_peered":"2024-02-05T06:37:42.966138+0000","last_clean":"2024-02-05T06:37:42.966138+0000","last_became_active":"2024-02-05T06:37:37.135073+0000","last_became_peered":"2024-02-05T06:37:37.135073+0000","last_unstale":"2024-02-05T06:37:42.966138+0000","last_undegraded":"2024-02-05T06:37:42.966138+0000","last_fullsized":"2024-02-05T06:37:42.966138+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:35:11.952460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:35:11.952460+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"137'9056","reported_seq":12024,"reported_epoch":137,"state":"active+remapped+backfilling","last_fresh":"2024-02-05T06:37:41.354014+0000","last_change":"2024-02-05T06:37:35.187450+0000","last_active":"2024-02-05T06:37:41.354014+0000","last_peered":"2024-02-05T06:37:41.354014+0000","last_clean":"2024-02-05T06:36:54.270379+0000","last_became_active":"2024-02-05T06:36:56.030721+0000","last_became_peered":"2024-02-05T06:36:56.030721+0000","last_unstale":"2024-02-05T06:37:41.354014+0000","last_undegraded":"2024-02-05T06:37:41.354014+0000","last_fullsized":"2024-02-05T06:37:41.354014+0000","mapping_epoch":119,"log_start":"112'4725","ondisk_log_start":"112'4725","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":4331,"log_dups_size":0,"ondisk_log_size":4331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356909056,"num_objects":5446,"num_object_clones":0,"num_object_copies":16338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4045,"num_objects_unfound":0,"num_objects_dirty":5446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9056,"num_write_kb":464064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1431,"num_bytes_recovered":93782016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,6],"avail_no_missing":["3","2","6"],"object_location_counts":[{"shards":"2,3,6","objects":5446}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":51,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.967164+0000","last_change":"2024-02-05T06:36:54.959371+0000","last_active":"2024-02-05T06:37:42.967164+0000","last_peered":"2024-02-05T06:37:42.967164+0000","last_clean":"2024-02-05T06:37:42.967164+0000","last_became_active":"2024-02-05T06:36:54.958644+0000","last_became_peered":"2024-02-05T06:36:54.958644+0000","last_unstale":"2024-02-05T06:37:42.967164+0000","last_undegraded":"2024-02-05T06:37:42.967164+0000","last_fullsized":"2024-02-05T06:37:42.967164+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:25.951483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:25.951483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":162,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.966240+0000","last_change":"2024-02-05T06:35:10.061496+0000","last_active":"2024-02-05T06:37:40.966240+0000","last_peered":"2024-02-05T06:37:40.966240+0000","last_clean":"2024-02-05T06:37:40.966240+0000","last_became_active":"2024-02-05T06:33:29.556002+0000","last_became_peered":"2024-02-05T06:33:29.556002+0000","last_unstale":"2024-02-05T06:37:40.966240+0000","last_undegraded":"2024-02-05T06:37:40.966240+0000","last_fullsized":"2024-02-05T06:37:40.966240+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:10.061388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:35:10.061388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"73'18","reported_seq":191,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.964114+0000","last_change":"2024-02-05T06:35:04.034763+0000","last_active":"2024-02-05T06:37:40.964114+0000","last_peered":"2024-02-05T06:37:40.964114+0000","last_clean":"2024-02-05T06:37:40.964114+0000","last_became_active":"2024-02-05T06:33:31.665995+0000","last_became_peered":"2024-02-05T06:33:31.665995+0000","last_unstale":"2024-02-05T06:37:40.964114+0000","last_undegraded":"2024-02-05T06:37:40.964114+0000","last_fullsized":"2024-02-05T06:37:40.964114+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-02-05T06:35:04.034664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:35:04.034664+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":51,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.966369+0000","last_change":"2024-02-05T06:36:54.965253+0000","last_active":"2024-02-05T06:37:42.966369+0000","last_peered":"2024-02-05T06:37:42.966369+0000","last_clean":"2024-02-05T06:37:42.966369+0000","last_became_active":"2024-02-05T06:36:54.963774+0000","last_became_peered":"2024-02-05T06:36:54.963774+0000","last_unstale":"2024-02-05T06:37:42.966369+0000","last_undegraded":"2024-02-05T06:37:42.966369+0000","last_fullsized":"2024-02-05T06:37:42.966369+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:34:49.579493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:34:49.579493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":328,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.998724+0000","last_change":"2024-02-05T06:37:38.307494+0000","last_active":"2024-02-05T06:37:42.998724+0000","last_peered":"2024-02-05T06:37:42.998724+0000","last_clean":"2024-02-05T06:37:42.998724+0000","last_became_active":"2024-02-05T06:37:37.140506+0000","last_became_peered":"2024-02-05T06:37:37.140506+0000","last_unstale":"2024-02-05T06:37:42.998724+0000","last_undegraded":"2024-02-05T06:37:42.998724+0000","last_fullsized":"2024-02-05T06:37:42.998724+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:37:38.307364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:37:38.307364+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:40.346345+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"139'9045","reported_seq":9205,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:46.155131+0000","last_change":"2024-02-05T06:37:34.244070+0000","last_active":"2024-02-05T06:37:46.155131+0000","last_peered":"2024-02-05T06:37:46.155131+0000","last_clean":"2024-02-05T06:37:46.155131+0000","last_became_active":"2024-02-05T06:35:27.930258+0000","last_became_peered":"2024-02-05T06:35:27.930258+0000","last_unstale":"2024-02-05T06:37:46.155131+0000","last_undegraded":"2024-02-05T06:37:46.155131+0000","last_fullsized":"2024-02-05T06:37:46.155131+0000","mapping_epoch":96,"log_start":"109'3658","ondisk_log_start":"109'3658","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'8533","last_scrub_stamp":"2024-02-05T06:37:34.243776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:37:34.243776+0000","objects_scrubbed":5815,"log_size":5387,"log_dups_size":0,"ondisk_log_size":5387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:44.970828+0000","scrub_duration":1414,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345440256,"num_objects":5271,"num_object_clones":0,"num_object_copies":15813,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9045,"num_write_kb":458112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":160,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.999328+0000","last_change":"2024-02-05T06:36:12.777606+0000","last_active":"2024-02-05T06:37:42.999328+0000","last_peered":"2024-02-05T06:37:42.999328+0000","last_clean":"2024-02-05T06:37:42.999328+0000","last_became_active":"2024-02-05T06:33:35.593210+0000","last_became_peered":"2024-02-05T06:33:35.593210+0000","last_unstale":"2024-02-05T06:37:42.999328+0000","last_undegraded":"2024-02-05T06:37:42.999328+0000","last_fullsized":"2024-02-05T06:37:42.999328+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:12.777526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:12.777526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"119'10","reported_seq":67,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.966647+0000","last_change":"2024-02-05T06:36:55.004386+0000","last_active":"2024-02-05T06:37:42.966647+0000","last_peered":"2024-02-05T06:37:42.966647+0000","last_clean":"2024-02-05T06:37:42.966647+0000","last_became_active":"2024-02-05T06:36:54.968825+0000","last_became_peered":"2024-02-05T06:36:54.968825+0000","last_unstale":"2024-02-05T06:37:42.966647+0000","last_undegraded":"2024-02-05T06:37:42.966647+0000","last_fullsized":"2024-02-05T06:37:42.966647+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'9","last_scrub_stamp":"2024-02-05T06:36:11.784734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:11.784734+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":162,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.964020+0000","last_change":"2024-02-05T06:35:17.058289+0000","last_active":"2024-02-05T06:37:40.964020+0000","last_peered":"2024-02-05T06:37:40.964020+0000","last_clean":"2024-02-05T06:37:40.964020+0000","last_became_active":"2024-02-05T06:33:29.555919+0000","last_became_peered":"2024-02-05T06:33:29.555919+0000","last_unstale":"2024-02-05T06:37:40.964020+0000","last_undegraded":"2024-02-05T06:37:40.964020+0000","last_fullsized":"2024-02-05T06:37:40.964020+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:17.058199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:35:17.058199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"73'15","reported_seq":234,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.968422+0000","last_change":"2024-02-05T06:36:54.984654+0000","last_active":"2024-02-05T06:37:40.968422+0000","last_peered":"2024-02-05T06:37:40.968422+0000","last_clean":"2024-02-05T06:37:40.968422+0000","last_became_active":"2024-02-05T06:36:54.948371+0000","last_became_peered":"2024-02-05T06:36:54.948371+0000","last_unstale":"2024-02-05T06:37:40.968422+0000","last_undegraded":"2024-02-05T06:37:40.968422+0000","last_fullsized":"2024-02-05T06:37:40.968422+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-02-05T06:36:28.066781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:28.066781+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:54.946715+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"139'9091","reported_seq":9253,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:46.169838+0000","last_change":"2024-02-05T06:37:35.556987+0000","last_active":"2024-02-05T06:37:46.169838+0000","last_peered":"2024-02-05T06:37:46.169838+0000","last_clean":"2024-02-05T06:37:46.169838+0000","last_became_active":"2024-02-05T06:35:27.935655+0000","last_became_peered":"2024-02-05T06:35:27.935655+0000","last_unstale":"2024-02-05T06:37:46.169838+0000","last_undegraded":"2024-02-05T06:37:46.169838+0000","last_fullsized":"2024-02-05T06:37:46.169838+0000","mapping_epoch":96,"log_start":"108'3613","ondisk_log_start":"108'3613","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'8624","last_scrub_stamp":"2024-02-05T06:37:35.556849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:37:35.556849+0000","objects_scrubbed":5720,"log_size":5478,"log_dups_size":0,"ondisk_log_size":5478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:59.964684+0000","scrub_duration":1761,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341704728,"num_objects":5215,"num_object_clones":0,"num_object_copies":15645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5215,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9091,"num_write_kb":457729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":150,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.965977+0000","last_change":"2024-02-05T06:35:20.042610+0000","last_active":"2024-02-05T06:37:40.965977+0000","last_peered":"2024-02-05T06:37:40.965977+0000","last_clean":"2024-02-05T06:37:40.965977+0000","last_became_active":"2024-02-05T06:33:35.668020+0000","last_became_peered":"2024-02-05T06:33:35.668020+0000","last_unstale":"2024-02-05T06:37:40.965977+0000","last_undegraded":"2024-02-05T06:37:40.965977+0000","last_fullsized":"2024-02-05T06:37:40.965977+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:20.042534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:20.042534+0000","last_clean_scrub_stamp":"2024-02-05T06:35:20.042534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":176,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:43.000002+0000","last_change":"2024-02-05T06:36:54.950791+0000","last_active":"2024-02-05T06:37:43.000002+0000","last_peered":"2024-02-05T06:37:43.000002+0000","last_clean":"2024-02-05T06:37:43.000002+0000","last_became_active":"2024-02-05T06:36:54.950484+0000","last_became_peered":"2024-02-05T06:36:54.950484+0000","last_unstale":"2024-02-05T06:37:43.000002+0000","last_undegraded":"2024-02-05T06:37:43.000002+0000","last_fullsized":"2024-02-05T06:37:43.000002+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:24.940878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:24.940878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":51,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.966458+0000","last_change":"2024-02-05T06:36:54.964984+0000","last_active":"2024-02-05T06:37:42.966458+0000","last_peered":"2024-02-05T06:37:42.966458+0000","last_clean":"2024-02-05T06:37:42.966458+0000","last_became_active":"2024-02-05T06:36:54.963340+0000","last_became_peered":"2024-02-05T06:36:54.963340+0000","last_unstale":"2024-02-05T06:37:42.966458+0000","last_undegraded":"2024-02-05T06:37:42.966458+0000","last_fullsized":"2024-02-05T06:37:42.966458+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:21.638465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:35:21.638465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"115'16","reported_seq":63,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.966496+0000","last_change":"2024-02-05T06:36:54.992244+0000","last_active":"2024-02-05T06:37:42.966496+0000","last_peered":"2024-02-05T06:37:42.966496+0000","last_clean":"2024-02-05T06:37:42.966496+0000","last_became_active":"2024-02-05T06:36:54.958582+0000","last_became_peered":"2024-02-05T06:36:54.958582+0000","last_unstale":"2024-02-05T06:37:42.966496+0000","last_undegraded":"2024-02-05T06:37:42.966496+0000","last_fullsized":"2024-02-05T06:37:42.966496+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'16","last_scrub_stamp":"2024-02-05T06:36:34.793243+0000","last_deep_scrub":"115'16","last_deep_scrub_stamp":"2024-02-05T06:36:34.793243+0000","last_clean_scrub_stamp":"2024-02-05T06:36:34.793243+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:56.608681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"83'128","reported_seq":328,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.998374+0000","last_change":"2024-02-05T06:37:39.307765+0000","last_active":"2024-02-05T06:37:42.998374+0000","last_peered":"2024-02-05T06:37:42.998374+0000","last_clean":"2024-02-05T06:37:42.998374+0000","last_became_active":"2024-02-05T06:37:37.144024+0000","last_became_peered":"2024-02-05T06:37:37.144024+0000","last_unstale":"2024-02-05T06:37:42.998374+0000","last_undegraded":"2024-02-05T06:37:42.998374+0000","last_fullsized":"2024-02-05T06:37:42.998374+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:37:39.307601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:37:39.307601+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:39:04.278501+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"137'8982","reported_seq":9143,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:41.200035+0000","last_change":"2024-02-05T06:37:32.002543+0000","last_active":"2024-02-05T06:37:41.200035+0000","last_peered":"2024-02-05T06:37:41.200035+0000","last_clean":"2024-02-05T06:37:41.200035+0000","last_became_active":"2024-02-05T06:35:27.938387+0000","last_became_peered":"2024-02-05T06:35:27.938387+0000","last_unstale":"2024-02-05T06:37:41.200035+0000","last_undegraded":"2024-02-05T06:37:41.200035+0000","last_fullsized":"2024-02-05T06:37:41.200035+0000","mapping_epoch":96,"log_start":"105'3086","ondisk_log_start":"105'3086","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'8496","last_scrub_stamp":"2024-02-05T06:37:32.002217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:37:32.002217+0000","objects_scrubbed":6000,"log_size":5896,"log_dups_size":0,"ondisk_log_size":5896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:55.946459+0000","scrub_duration":2534,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356515840,"num_objects":5440,"num_object_clones":0,"num_object_copies":16320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5440,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8982,"num_write_kb":461504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":168,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.997184+0000","last_change":"2024-02-05T06:36:54.959177+0000","last_active":"2024-02-05T06:37:40.997184+0000","last_peered":"2024-02-05T06:37:40.997184+0000","last_clean":"2024-02-05T06:37:40.997184+0000","last_became_active":"2024-02-05T06:36:54.958805+0000","last_became_peered":"2024-02-05T06:36:54.958805+0000","last_unstale":"2024-02-05T06:37:40.997184+0000","last_undegraded":"2024-02-05T06:37:40.997184+0000","last_fullsized":"2024-02-05T06:37:40.997184+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:30.754865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:30.754865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:50.126762+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":173,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.968505+0000","last_change":"2024-02-05T06:36:54.958628+0000","last_active":"2024-02-05T06:37:40.968505+0000","last_peered":"2024-02-05T06:37:40.968505+0000","last_clean":"2024-02-05T06:37:40.968505+0000","last_became_active":"2024-02-05T06:36:54.958082+0000","last_became_peered":"2024-02-05T06:36:54.958082+0000","last_unstale":"2024-02-05T06:37:40.968505+0000","last_undegraded":"2024-02-05T06:37:40.968505+0000","last_fullsized":"2024-02-05T06:37:40.968505+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:29.069747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:29.069747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:37:46.117534+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":185,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:43.001998+0000","last_change":"2024-02-05T06:37:29.801958+0000","last_active":"2024-02-05T06:37:43.001998+0000","last_peered":"2024-02-05T06:37:43.001998+0000","last_clean":"2024-02-05T06:37:43.001998+0000","last_became_active":"2024-02-05T06:36:54.966178+0000","last_became_peered":"2024-02-05T06:36:54.966178+0000","last_unstale":"2024-02-05T06:37:43.001998+0000","last_undegraded":"2024-02-05T06:37:43.001998+0000","last_fullsized":"2024-02-05T06:37:43.001998+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:29.801850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:37:29.801850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:47.802943+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"95'25","reported_seq":69,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.966686+0000","last_change":"2024-02-05T06:36:55.005133+0000","last_active":"2024-02-05T06:37:42.966686+0000","last_peered":"2024-02-05T06:37:42.966686+0000","last_clean":"2024-02-05T06:37:42.966686+0000","last_became_active":"2024-02-05T06:36:54.961719+0000","last_became_peered":"2024-02-05T06:36:54.961719+0000","last_unstale":"2024-02-05T06:37:42.966686+0000","last_undegraded":"2024-02-05T06:37:42.966686+0000","last_fullsized":"2024-02-05T06:37:42.966686+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'25","last_scrub_stamp":"2024-02-05T06:35:26.274944+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-02-05T06:35:26.274944+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.274944+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":446,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.970400+0000","last_change":"2024-02-05T06:37:37.689513+0000","last_active":"2024-02-05T06:37:42.970400+0000","last_peered":"2024-02-05T06:37:42.970400+0000","last_clean":"2024-02-05T06:37:42.970400+0000","last_became_active":"2024-02-05T06:37:37.134583+0000","last_became_peered":"2024-02-05T06:37:37.134583+0000","last_unstale":"2024-02-05T06:37:42.970400+0000","last_undegraded":"2024-02-05T06:37:42.970400+0000","last_fullsized":"2024-02-05T06:37:42.970400+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:37:37.689430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:37:37.689430+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:52.234565+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.f","version":"137'8992","reported_seq":9084,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:41.193508+0000","last_change":"2024-02-05T06:35:27.931906+0000","last_active":"2024-02-05T06:37:41.193508+0000","last_peered":"2024-02-05T06:37:41.193508+0000","last_clean":"2024-02-05T06:37:41.193508+0000","last_became_active":"2024-02-05T06:35:27.931422+0000","last_became_peered":"2024-02-05T06:35:27.931422+0000","last_unstale":"2024-02-05T06:37:41.193508+0000","last_undegraded":"2024-02-05T06:37:41.193508+0000","last_fullsized":"2024-02-05T06:37:41.193508+0000","mapping_epoch":96,"log_start":"110'3944","ondisk_log_start":"110'3944","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:26.912882+0000","last_clean_scrub_stamp":"2024-02-05T06:35:26.912882+0000","objects_scrubbed":0,"log_size":5048,"log_dups_size":0,"ondisk_log_size":5048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351928320,"num_objects":5370,"num_object_clones":0,"num_object_copies":16110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8992,"num_write_kb":459584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":51,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.966454+0000","last_change":"2024-02-05T06:36:54.963613+0000","last_active":"2024-02-05T06:37:42.966454+0000","last_peered":"2024-02-05T06:37:42.966454+0000","last_clean":"2024-02-05T06:37:42.966454+0000","last_became_active":"2024-02-05T06:36:54.961560+0000","last_became_peered":"2024-02-05T06:36:54.961560+0000","last_unstale":"2024-02-05T06:37:42.966454+0000","last_undegraded":"2024-02-05T06:37:42.966454+0000","last_fullsized":"2024-02-05T06:37:42.966454+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:35:27.459848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:35:27.459848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"92'9","reported_seq":220,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.996596+0000","last_change":"2024-02-05T06:36:21.707926+0000","last_active":"2024-02-05T06:37:40.996596+0000","last_peered":"2024-02-05T06:37:40.996596+0000","last_clean":"2024-02-05T06:37:40.996596+0000","last_became_active":"2024-02-05T06:33:33.588478+0000","last_became_peered":"2024-02-05T06:33:33.588478+0000","last_unstale":"2024-02-05T06:37:40.996596+0000","last_undegraded":"2024-02-05T06:37:40.996596+0000","last_fullsized":"2024-02-05T06:37:40.996596+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'9","last_scrub_stamp":"2024-02-05T06:36:21.707759+0000","last_deep_scrub":"92'9","last_deep_scrub_stamp":"2024-02-05T06:34:52.283780+0000","last_clean_scrub_stamp":"2024-02-05T06:36:21.707759+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":182,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-02-05T06:37:40.965856+0000","last_change":"2024-02-05T06:36:54.971857+0000","last_active":"2024-02-05T06:37:40.965856+0000","last_peered":"2024-02-05T06:37:40.965856+0000","last_clean":"2024-02-05T06:37:40.965856+0000","last_became_active":"2024-02-05T06:36:54.971304+0000","last_became_peered":"2024-02-05T06:36:54.971304+0000","last_unstale":"2024-02-05T06:37:40.965856+0000","last_undegraded":"2024-02-05T06:37:40.965856+0000","last_fullsized":"2024-02-05T06:37:40.965856+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:37.506830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:37.506830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:38:00.884497+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"93'15","reported_seq":63,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-02-05T06:37:42.966390+0000","last_change":"2024-02-05T06:36:55.003532+0000","last_active":"2024-02-05T06:37:42.966390+0000","last_peered":"2024-02-05T06:37:42.966390+0000","last_clean":"2024-02-05T06:37:42.966390+0000","last_became_active":"2024-02-05T06:36:54.963620+0000","last_became_peered":"2024-02-05T06:36:54.963620+0000","last_unstale":"2024-02-05T06:37:42.966390+0000","last_undegraded":"2024-02-05T06:37:42.966390+0000","last_fullsized":"2024-02-05T06:37:42.966390+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'15","last_scrub_stamp":"2024-02-05T06:35:04.584657+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-02-05T06:35:04.584657+0000","last_clean_scrub_stamp":"2024-02-05T06:35:04.584657+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":5618663448,"num_objects":85735,"num_object_clones":0,"num_object_copies":257205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9528,"num_objects_unfound":0,"num_objects_dirty":85735,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144501,"num_write_kb":7367489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20862,"num_bytes_recovered":1367212032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17398050816,"data_stored":17398038600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80863,"ondisk_log_size":80863,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":249,"num_read_kb":244,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":423698,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":54,"num_read_kb":81,"num_write":49,"num_write_kb":660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":810710,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1164,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":686,"num_read_kb":443,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":3416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":452,"ondisk_log_size":452,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103223,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1796868,"kb_used_data":1689340,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":91948284,"statfs":{"total":95995035648,"available":94155042816,"internally_reserved":0,"allocated":1729884160,"data_stored":1729475378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":110098890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:37:28 2024","interfaces":[{"interface":"back","average":{"1min":1.5860000000000001,"5min":1.4330000000000001,"15min":0.85499999999999998},"min":{"1min":0.27800000000000002,"5min":0.219,"15min":0.219},"max":{"1min":7.9370000000000003,"5min":68.921000000000006,"15min":68.921000000000006},"last":1.141},{"interface":"front","average":{"1min":1.9810000000000001,"5min":1.54,"15min":0.91300000000000003},"min":{"1min":0.27200000000000002,"5min":0.248,"15min":0.248},"max":{"1min":7.7199999999999998,"5min":69.616,"15min":69.616},"last":2.9580000000000002}]},{"osd":1,"last update":"Mon Feb 5 06:37:28 2024","interfaces":[{"interface":"back","average":{"1min":5.7169999999999996,"5min":1.6220000000000001,"15min":0.91700000000000004},"min":{"1min":0.26400000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":89.231999999999999,"5min":89.231999999999999,"15min":89.231999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":6.1280000000000001,"5min":1.734,"15min":0.97699999999999998},"min":{"1min":0.38900000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":87.995999999999995,"5min":87.995999999999995,"15min":87.995999999999995},"last":0.91200000000000003}]},{"osd":2,"last update":"Mon Feb 5 06:37:28 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":0.72299999999999998,"15min":0.623},"min":{"1min":0.27400000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.5990000000000002,"5min":6.5990000000000002,"15min":6.5990000000000002},"last":2.464},{"interface":"front","average":{"1min":1.3009999999999999,"5min":0.73199999999999998,"15min":0.621},"min":{"1min":0.30199999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":3.3210000000000002,"5min":3.3210000000000002,"15min":3.3210000000000002},"last":5.274}]},{"osd":3,"last update":"Mon Feb 5 06:37:28 2024","interfaces":[{"interface":"back","average":{"1min":6.0659999999999998,"5min":2.5739999999999998,"15min":1.258},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":89.481999999999999,"5min":89.481999999999999,"15min":89.481999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":6.5670000000000002,"5min":2.6920000000000002,"15min":1.3},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":92.293999999999997,"5min":92.293999999999997,"15min":92.293999999999997},"last":1.1779999999999999}]},{"osd":4,"last update":"Mon Feb 5 06:37:28 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.503,"15min":0.46300000000000002},"min":{"1min":0.29799999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":3.5409999999999999,"5min":3.5409999999999999,"15min":3.5409999999999999},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48299999999999998,"15min":0.49299999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.29199999999999998}]},{"osd":5,"last update":"Mon Feb 5 06:37:28 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.51300000000000001,"15min":0.505},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":2.3290000000000002,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.57399999999999995,"15min":0.55400000000000005},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":3.5990000000000002,"5min":3.5990000000000002,"15min":3.5990000000000002},"last":0.28000000000000003}]},{"osd":6,"last update":"Mon Feb 5 06:37:28 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.59599999999999997,"15min":0.55000000000000004},"min":{"1min":0.33000000000000002,"5min":0.316,"15min":0.316},"max":{"1min":3.5750000000000002,"5min":3.5750000000000002,"15min":3.5750000000000002},"last":0.314},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.56899999999999995,"15min":0.55000000000000004},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":3.2189999999999999,"5min":3.2189999999999999,"15min":3.2189999999999999},"last":0.33500000000000002}]}]},{"osd":6,"up_from":49,"seq":210453397565,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2142000,"kb_used_data":2030312,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":91603152,"statfs":{"total":95995035648,"available":93801627648,"internally_reserved":0,"allocated":2079039488,"data_stored":2078649761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":114358726},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:36:59 2024","interfaces":[{"interface":"back","average":{"1min":1.8600000000000001,"5min":0.84499999999999997,"15min":0.65700000000000003},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":6.1120000000000001,"5min":6.1120000000000001,"15min":6.1120000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":1.0109999999999999,"5min":0.64100000000000001,"15min":0.57499999999999996},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":4.2469999999999999,"5min":4.2469999999999999,"15min":4.2469999999999999},"last":0.32800000000000001}]},{"osd":1,"last update":"Mon Feb 5 06:36:59 2024","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":0.66300000000000003,"15min":0.56999999999999995},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":5.9660000000000002,"5min":5.9660000000000002,"15min":5.9660000000000002},"last":0.38900000000000001},{"interface":"front","average":{"1min":1.139,"5min":0.67500000000000004,"15min":0.61499999999999999},"min":{"1min":0.29199999999999998,"5min":0.255,"15min":0.255},"max":{"1min":5.5099999999999998,"5min":5.5099999999999998,"15min":5.5099999999999998},"last":0.41999999999999998}]},{"osd":2,"last update":"Mon Feb 5 06:36:59 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.625,"15min":0.59699999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":2.956,"5min":2.956,"15min":2.956},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.59799999999999998,"15min":0.56799999999999995},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.6840000000000002,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":0.45800000000000002}]},{"osd":3,"last update":"Mon Feb 5 06:36:59 2024","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":0.73699999999999999,"15min":0.60899999999999999},"min":{"1min":0.33000000000000002,"5min":0.315,"15min":0.315},"max":{"1min":3.48,"5min":3.48,"15min":3.48},"last":0.30499999999999999},{"interface":"front","average":{"1min":1.399,"5min":0.80600000000000005,"15min":0.65100000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":6.5419999999999998,"5min":6.5419999999999998,"15min":6.5419999999999998},"last":0.48399999999999999}]},{"osd":4,"last update":"Mon Feb 5 06:36:59 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.44,"15min":0.44600000000000001},"min":{"1min":0.32700000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.98899999999999999,"5min":1.355,"15min":1.355},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45600000000000002,"15min":0.42999999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.34999999999999998}]},{"osd":5,"last update":"Mon Feb 5 06:36:59 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46500000000000002,"15min":0.45700000000000002},"min":{"1min":0.29199999999999998,"5min":0.251,"15min":0.251},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.47799999999999998,"15min":0.45800000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.432}]},{"osd":7,"last update":"Mon Feb 5 06:37:21 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.55800000000000005,"15min":0.53400000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":6.0350000000000001,"5min":6.0350000000000001,"15min":6.0350000000000001},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.63600000000000001,"15min":0.58199999999999996},"min":{"1min":0.38500000000000001,"5min":0.315,"15min":0.315},"max":{"1min":8.5399999999999991,"5min":8.5399999999999991,"15min":8.5399999999999991},"last":0.442}]}]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2536640,"kb_used_data":2433268,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":91208512,"statfs":{"total":95995035648,"available":93397516288,"internally_reserved":0,"allocated":2491666432,"data_stored":2491245411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:36:57 2024","interfaces":[{"interface":"back","average":{"1min":2.5019999999999998,"5min":1.6060000000000001,"15min":0.753},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":5.7039999999999997,"5min":67.421000000000006,"15min":67.421000000000006},"last":0.19},{"interface":"front","average":{"1min":2.0299999999999998,"5min":1.474,"15min":0.71199999999999997},"min":{"1min":0.215,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":4.415,"5min":67.885000000000005,"15min":67.885000000000005},"last":4.3570000000000002}]},{"osd":2,"last update":"Mon Feb 5 06:37:16 2024","interfaces":[{"interface":"back","average":{"1min":2.0419999999999998,"5min":0.91200000000000003,"15min":0.56299999999999994},"min":{"1min":0.215,"5min":0.214,"15min":0.214},"max":{"1min":8.5630000000000006,"5min":8.5630000000000006,"15min":8.5630000000000006},"last":2.375},{"interface":"front","average":{"1min":4.0339999999999998,"5min":1.2609999999999999,"15min":0.66900000000000004},"min":{"1min":0.45200000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":11.859,"5min":11.859,"15min":11.859},"last":0.375}]},{"osd":3,"last update":"Mon Feb 5 06:36:47 2024","interfaces":[{"interface":"back","average":{"1min":7.0549999999999997,"5min":1.754,"15min":0.86399999999999999},"min":{"1min":0.36299999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":69.798000000000002,"5min":69.798000000000002,"15min":69.798000000000002},"last":4.399},{"interface":"front","average":{"1min":6.0720000000000001,"5min":1.6519999999999999,"15min":0.86599999999999999},"min":{"1min":0.55500000000000005,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":70.765000000000001,"5min":70.765000000000001,"15min":70.765000000000001},"last":0.24199999999999999}]},{"osd":4,"last update":"Mon Feb 5 06:37:10 2024","interfaces":[{"interface":"back","average":{"1min":2.5899999999999999,"5min":1.024,"15min":0.71399999999999997},"min":{"1min":0.41499999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":11.679,"5min":11.679,"15min":11.679},"last":0.41699999999999998},{"interface":"front","average":{"1min":1.319,"5min":0.83999999999999997,"15min":0.66400000000000003},"min":{"1min":0.33600000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":4.5199999999999996,"5min":5.2169999999999996,"15min":5.2169999999999996},"last":4.4660000000000002}]},{"osd":5,"last update":"Mon Feb 5 06:37:37 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.026,"15min":1.018},"min":{"1min":0.442,"5min":0.377,"15min":0.377},"max":{"1min":6.0469999999999997,"5min":8.2509999999999994,"15min":8.2509999999999994},"last":2.2170000000000001},{"interface":"front","average":{"1min":2.9089999999999998,"5min":1.2589999999999999,"15min":0.84499999999999997},"min":{"1min":0.39700000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":13.455,"5min":13.455,"15min":13.455},"last":0.44400000000000001}]},{"osd":6,"last update":"Mon Feb 5 06:36:57 2024","interfaces":[{"interface":"back","average":{"1min":2.387,"5min":1.036,"15min":0.78000000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":13.391,"5min":13.391,"15min":13.391},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.496,"5min":0.85299999999999998,"15min":0.73899999999999999},"min":{"1min":0.57899999999999996,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":5.3099999999999996,"5min":5.3099999999999996,"15min":5.3099999999999996},"last":2.3410000000000002}]},{"osd":7,"last update":"Mon Feb 5 06:37:22 2024","interfaces":[{"interface":"back","average":{"1min":3.157,"5min":1.2450000000000001,"15min":0.80900000000000005},"min":{"1min":0.42799999999999999,"5min":0.37,"15min":0.37},"max":{"1min":13.49,"5min":13.49,"15min":13.49},"last":0.49199999999999999},{"interface":"front","average":{"1min":1.0629999999999999,"5min":0.98299999999999998,"15min":1.0229999999999999},"min":{"1min":0.46300000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":3.4300000000000002,"5min":8.2219999999999995,"15min":8.2219999999999995},"last":4.4329999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2883136,"kb_used_data":2775600,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":90862016,"statfs":{"total":95995035648,"available":93042704384,"internally_reserved":0,"allocated":2842214400,"data_stored":2841803687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 06:36:55 2024","interfaces":[{"interface":"back","average":{"1min":2.0800000000000001,"5min":0.79800000000000004,"15min":0.47199999999999998},"min":{"1min":0.32200000000000001,"5min":0.23300000000000001,"15min":0.188},"max":{"1min":12.031000000000001,"5min":12.031000000000001,"15min":12.031000000000001},"last":3.875},{"interface":"front","average":{"1min":2.0630000000000002,"5min":0.77200000000000002,"15min":0.47099999999999997},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.20499999999999999},"max":{"1min":7.1120000000000001,"5min":7.1120000000000001,"15min":7.1120000000000001},"last":1.7110000000000001}]},{"osd":2,"last update":"Mon Feb 5 06:37:14 2024","interfaces":[{"interface":"back","average":{"1min":2.657,"5min":1.0800000000000001,"15min":0.628},"min":{"1min":0.31900000000000001,"5min":0.315,"15min":0.27000000000000002},"max":{"1min":7.3540000000000001,"5min":7.3540000000000001,"15min":7.3540000000000001},"last":1.7450000000000001},{"interface":"front","average":{"1min":1.8500000000000001,"5min":0.88700000000000001,"15min":0.57299999999999995},"min":{"1min":0.33400000000000002,"5min":0.28999999999999998,"15min":0.28000000000000003},"max":{"1min":4.8310000000000004,"5min":4.8310000000000004,"15min":4.8310000000000004},"last":1.768}]},{"osd":3,"last update":"Mon Feb 5 06:36:40 2024","interfaces":[{"interface":"back","average":{"1min":2.5539999999999998,"5min":0.88700000000000001,"15min":0.59899999999999998},"min":{"1min":0.311,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":12.084,"5min":12.084,"15min":12.084},"last":3.766},{"interface":"front","average":{"1min":2.0990000000000002,"5min":0.81000000000000005,"15min":0.56899999999999995},"min":{"1min":0.438,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.8579999999999997,"5min":4.8579999999999997,"15min":4.8579999999999997},"last":0.34999999999999998}]},{"osd":4,"last update":"Mon Feb 5 06:37:10 2024","interfaces":[{"interface":"back","average":{"1min":1.9139999999999999,"5min":0.96299999999999997,"15min":0.71699999999999997},"min":{"1min":0.51700000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":7.3849999999999998,"5min":7.3849999999999998,"15min":7.3849999999999998},"last":0.314},{"interface":"front","average":{"1min":2.0489999999999999,"5min":1.0029999999999999,"15min":0.73599999999999999},"min":{"1min":0.38700000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":12.186999999999999,"5min":12.186999999999999,"15min":12.186999999999999},"last":3.8439999999999999}]},{"osd":5,"last update":"Mon Feb 5 06:37:34 2024","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":0.93100000000000005,"15min":0.73699999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":6.7210000000000001,"5min":6.7210000000000001,"15min":6.7210000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.8,"5min":0.997,"15min":0.76500000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":7.3220000000000001,"5min":7.3220000000000001,"15min":7.3220000000000001},"last":0.61099999999999999}]},{"osd":6,"last update":"Mon Feb 5 06:36:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8069999999999999,"5min":0.95499999999999996,"15min":0.74299999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":12.119999999999999,"5min":12.119999999999999,"15min":12.119999999999999},"last":3.8100000000000001},{"interface":"front","average":{"1min":1.3680000000000001,"5min":0.82799999999999996,"15min":0.71199999999999997},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":4.8029999999999999,"5min":4.8029999999999999,"15min":4.8029999999999999},"last":0.56599999999999995}]},{"osd":7,"last update":"Mon Feb 5 06:37:23 2024","interfaces":[{"interface":"back","average":{"1min":1.7150000000000001,"5min":0.96299999999999997,"15min":0.71899999999999997},"min":{"1min":0.47199999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":7.2859999999999996,"5min":7.2859999999999996,"15min":7.2859999999999996},"last":0.65400000000000003},{"interface":"front","average":{"1min":2.1600000000000001,"5min":1.1830000000000001,"15min":1.0249999999999999},"min":{"1min":0.49199999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":12.156000000000001,"5min":12.156000000000001,"15min":12.156000000000001},"last":3.9049999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2208860,"kb_used_data":2097168,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":91536292,"statfs":{"total":95995035648,"available":93733163008,"internally_reserved":0,"allocated":2147500032,"data_stored":2147115619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:37:17 2024","interfaces":[{"interface":"back","average":{"1min":2.7949999999999999,"5min":1.639,"15min":0.78500000000000003},"min":{"1min":0.23200000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":18.120000000000001,"5min":50.755000000000003,"15min":50.755000000000003},"last":50.814},{"interface":"front","average":{"1min":2.0680000000000001,"5min":1.5129999999999999,"15min":0.751},"min":{"1min":0.19700000000000001,"5min":0.188,"15min":0.188},"max":{"1min":14.683999999999999,"5min":50.802999999999997,"15min":50.802999999999997},"last":47.122999999999998}]},{"osd":1,"last update":"Mon Feb 5 06:37:17 2024","interfaces":[{"interface":"back","average":{"1min":2.2410000000000001,"5min":0.98399999999999999,"15min":0.58399999999999996},"min":{"1min":0.25600000000000001,"5min":0.252,"15min":0.252},"max":{"1min":9.3019999999999996,"5min":10.243,"15min":10.243},"last":31.303000000000001},{"interface":"front","average":{"1min":1.8340000000000001,"5min":0.89400000000000002,"15min":0.57799999999999996},"min":{"1min":0.28899999999999998,"5min":0.22,"15min":0.22},"max":{"1min":9.2080000000000002,"5min":9.2080000000000002,"15min":9.2080000000000002},"last":31.425000000000001}]},{"osd":3,"last update":"Mon Feb 5 06:36:41 2024","interfaces":[{"interface":"back","average":{"1min":4.5,"5min":1.2949999999999999,"15min":0.74199999999999999},"min":{"1min":0.46700000000000003,"5min":0.254,"15min":0.254},"max":{"1min":53.127000000000002,"5min":53.127000000000002,"15min":53.127000000000002},"last":46.845999999999997},{"interface":"front","average":{"1min":4.2430000000000003,"5min":1.2669999999999999,"15min":0.745},"min":{"1min":0.30499999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":52.607999999999997,"5min":52.607999999999997,"15min":52.607999999999997},"last":46.481999999999999}]},{"osd":4,"last update":"Mon Feb 5 06:37:08 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":0.80800000000000005,"15min":0.69999999999999996},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":5.516,"5min":5.516,"15min":5.516},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.93200000000000005,"5min":0.71499999999999997,"15min":0.68300000000000005},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":3.2280000000000002,"5min":3.2280000000000002,"15min":3.2280000000000002},"last":0.73099999999999998}]},{"osd":5,"last update":"Mon Feb 5 06:37:35 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.75700000000000001,"15min":0.72499999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.8919999999999999,"5min":3.1469999999999998,"15min":3.1469999999999998},"last":0.62},{"interface":"front","average":{"1min":1.0309999999999999,"5min":0.89700000000000002,"15min":0.75700000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":3.8220000000000001,"5min":7.3730000000000002,"15min":7.3730000000000002},"last":0.76900000000000002}]},{"osd":6,"last update":"Mon Feb 5 06:37:00 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":1.1299999999999999,"15min":1.524},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":3.194,"5min":17.143999999999998,"15min":17.143999999999998},"last":0.75700000000000001},{"interface":"front","average":{"1min":1.9159999999999999,"5min":0.95999999999999996,"15min":0.78800000000000003},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":10.27,"5min":10.27,"15min":10.27},"last":0.78700000000000003}]},{"osd":7,"last update":"Mon Feb 5 06:37:27 2024","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":0.94699999999999995,"15min":0.75},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":5.6500000000000004,"5min":10.206,"15min":10.206},"last":0.74099999999999999},{"interface":"front","average":{"1min":1.0720000000000001,"5min":0.85999999999999999,"15min":0.73399999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":2.7930000000000001,"5min":5.476,"15min":5.476},"last":0.63900000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149771,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3612088,"kb_used_data":3464304,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":90133064,"statfs":{"total":95995035648,"available":92296257536,"internally_reserved":0,"allocated":3547447296,"data_stored":3547036679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":151321031},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:36:49 2024","interfaces":[{"interface":"back","average":{"1min":1.7829999999999999,"5min":0.69099999999999995,"15min":0.495},"min":{"1min":0.32500000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":6.9729999999999999,"5min":6.9729999999999999,"15min":6.9729999999999999},"last":2.6360000000000001},{"interface":"front","average":{"1min":3.1970000000000001,"5min":1.008,"15min":0.64200000000000002},"min":{"1min":0.68700000000000006,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":8.4580000000000002,"5min":8.4580000000000002,"15min":8.4580000000000002},"last":4.2359999999999998}]},{"osd":1,"last update":"Mon Feb 5 06:36:49 2024","interfaces":[{"interface":"back","average":{"1min":2.5630000000000002,"5min":0.88300000000000001,"15min":0.61399999999999999},"min":{"1min":0.248,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":9.6010000000000009,"5min":9.6010000000000009,"15min":9.6010000000000009},"last":2.335},{"interface":"front","average":{"1min":2.4340000000000002,"5min":0.84999999999999998,"15min":0.58799999999999997},"min":{"1min":0.28000000000000003,"5min":0.246,"15min":0.246},"max":{"1min":7.6040000000000001,"5min":7.6040000000000001,"15min":7.6040000000000001},"last":0.30299999999999999}]},{"osd":2,"last update":"Mon Feb 5 06:36:49 2024","interfaces":[{"interface":"back","average":{"1min":2.4950000000000001,"5min":0.92500000000000004,"15min":0.64300000000000002},"min":{"1min":0.55800000000000005,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":6.6529999999999996,"5min":6.6529999999999996,"15min":6.6529999999999996},"last":6.1319999999999997},{"interface":"front","average":{"1min":1.9399999999999999,"5min":0.74399999999999999,"15min":0.56399999999999995},"min":{"1min":0.254,"5min":0.248,"15min":0.248},"max":{"1min":7.0030000000000001,"5min":7.0030000000000001,"15min":7.0030000000000001},"last":6.1760000000000002}]},{"osd":4,"last update":"Mon Feb 5 06:37:11 2024","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":0.77300000000000002,"15min":0.67300000000000004},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":6.899,"5min":6.899,"15min":6.899},"last":2.5720000000000001},{"interface":"front","average":{"1min":2.1000000000000001,"5min":1.0840000000000001,"15min":0.76800000000000002},"min":{"1min":0.505,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":5.5700000000000003,"5min":8.4060000000000006,"15min":8.4060000000000006},"last":2.262}]},{"osd":5,"last update":"Mon Feb 5 06:37:35 2024","interfaces":[{"interface":"back","average":{"1min":1.7769999999999999,"5min":1.0880000000000001,"15min":0.76900000000000002},"min":{"1min":0.42999999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":5.968,"5min":6.5640000000000001,"15min":6.5640000000000001},"last":3.6179999999999999},{"interface":"front","average":{"1min":2.0699999999999998,"5min":0.95999999999999996,"15min":0.74099999999999999},"min":{"1min":0.36499999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":15.792999999999999,"5min":15.792999999999999,"15min":15.792999999999999},"last":2.5219999999999998}]},{"osd":6,"last update":"Mon Feb 5 06:37:00 2024","interfaces":[{"interface":"back","average":{"1min":1.9119999999999999,"5min":0.98199999999999998,"15min":0.70499999999999996},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":8.3610000000000007,"5min":8.3610000000000007,"15min":8.3610000000000007},"last":2.3119999999999998},{"interface":"front","average":{"1min":2.0070000000000001,"5min":1.0089999999999999,"15min":0.76100000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":6.601,"5min":6.601,"15min":6.601},"last":3.653}]},{"osd":7,"last update":"Mon Feb 5 06:37:26 2024","interfaces":[{"interface":"back","average":{"1min":2.0920000000000001,"5min":1.351,"15min":0.83999999999999997},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":16.524999999999999,"5min":45.039000000000001,"15min":45.039000000000001},"last":2.6080000000000001},{"interface":"front","average":{"1min":2.1480000000000001,"5min":1.4650000000000001,"15min":0.89800000000000002},"min":{"1min":0.49199999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":6.6269999999999998,"5min":44.036999999999999,"15min":44.036999999999999},"last":3.6779999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986246,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1370636,"kb_used_data":1200644,"kb_used_omap":1,"kb_used_meta":169982,"kb_avail":92374516,"statfs":{"total":95995035648,"available":94591504384,"internally_reserved":0,"allocated":1229459456,"data_stored":1229042036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":174062021},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1,15],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:37:07 2024","interfaces":[{"interface":"back","average":{"1min":3.0569999999999999,"5min":1.0800000000000001,"15min":0.75600000000000001},"min":{"1min":0.38900000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":18.062999999999999,"5min":18.062999999999999,"15min":18.062999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":3.1499999999999999,"5min":1.1930000000000001,"15min":0.75700000000000001},"min":{"1min":0.27000000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":17.856000000000002,"5min":17.856000000000002,"15min":17.856000000000002},"last":0.51400000000000001}]},{"osd":1,"last update":"Mon Feb 5 06:37:07 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.66900000000000004,"15min":0.59899999999999998},"min":{"1min":0.28000000000000003,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.871,"5min":2.871,"15min":2.871},"last":0.434},{"interface":"front","average":{"1min":0.877,"5min":0.68300000000000005,"15min":0.63100000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":2.7730000000000001,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":0.33300000000000002}]},{"osd":2,"last update":"Mon Feb 5 06:37:07 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.60499999999999998,"15min":0.58099999999999996},"min":{"1min":0.28499999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.3420000000000001,"5min":2.649,"15min":2.649},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.90700000000000003,"5min":0.64200000000000002,"15min":0.60799999999999998},"min":{"1min":0.32200000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":4.0599999999999996,"5min":4.0599999999999996,"15min":4.0599999999999996},"last":0.48699999999999999}]},{"osd":3,"last update":"Mon Feb 5 06:37:07 2024","interfaces":[{"interface":"back","average":{"1min":6.8029999999999999,"5min":2.794,"15min":1.333},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":99.546999999999997,"5min":99.546999999999997,"15min":99.546999999999997},"last":0.54100000000000004},{"interface":"front","average":{"1min":7.2720000000000002,"5min":2.7839999999999998,"15min":1.321},"min":{"1min":0.38300000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":99.245000000000005,"5min":99.245000000000005,"15min":99.245000000000005},"last":0.39200000000000002}]},{"osd":5,"last update":"Mon Feb 5 06:37:34 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.41799999999999998,"15min":0.40400000000000003},"min":{"1min":0.29099999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.57199999999999995,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.45200000000000001,"15min":0.442},"min":{"1min":0.29699999999999999,"5min":0.252,"15min":0.252},"max":{"1min":0.77000000000000002,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.36299999999999999}]},{"osd":6,"last update":"Mon Feb 5 06:36:57 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.51800000000000002,"15min":0.498},"min":{"1min":0.33600000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.52700000000000002,"15min":0.48799999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.109,"5min":2.109,"15min":2.109},"last":0.498}]},{"osd":7,"last update":"Mon Feb 5 06:37:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.52400000000000002,"15min":0.51500000000000001},"min":{"1min":0.34599999999999997,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":2.0230000000000001,"5min":2.0230000000000001,"15min":2.0230000000000001},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.52500000000000002,"15min":0.53000000000000003},"min":{"1min":0.316,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.44800000000000001}]}]},{"osd":5,"up_from":40,"seq":171798691905,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1410000,"kb_used_data":1346884,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92335152,"statfs":{"total":95995035648,"available":94551195648,"internally_reserved":0,"allocated":1379209216,"data_stored":1378792907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:37:30 2024","interfaces":[{"interface":"back","average":{"1min":2.1970000000000001,"5min":1.0309999999999999,"15min":0.70099999999999996},"min":{"1min":0.26700000000000002,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":11.904999999999999,"5min":11.904999999999999,"15min":11.904999999999999},"last":1.206},{"interface":"front","average":{"1min":3.1720000000000002,"5min":1.3129999999999999,"15min":0.80400000000000005},"min":{"1min":0.27800000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":38.808999999999997,"5min":38.808999999999997,"15min":38.808999999999997},"last":0.22900000000000001}]},{"osd":1,"last update":"Mon Feb 5 06:37:30 2024","interfaces":[{"interface":"back","average":{"1min":2.089,"5min":0.96199999999999997,"15min":0.69599999999999995},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":18.582000000000001,"5min":18.582000000000001,"15min":18.582000000000001},"last":1.244},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.69899999999999995,"15min":0.59699999999999998},"min":{"1min":0.246,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":3.6459999999999999,"5min":3.6459999999999999,"15min":3.6459999999999999},"last":0.34000000000000002}]},{"osd":2,"last update":"Mon Feb 5 06:37:30 2024","interfaces":[{"interface":"back","average":{"1min":1.97,"5min":0.92200000000000004,"15min":0.67400000000000004},"min":{"1min":0.35399999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":5.6500000000000004,"5min":5.6500000000000004,"15min":5.6500000000000004},"last":2.0880000000000001},{"interface":"front","average":{"1min":1.2130000000000001,"5min":0.65900000000000003,"15min":0.57699999999999996},"min":{"1min":0.30199999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":4.6559999999999997,"5min":4.6559999999999997,"15min":4.6559999999999997},"last":0.38900000000000001}]},{"osd":3,"last update":"Mon Feb 5 06:37:30 2024","interfaces":[{"interface":"back","average":{"1min":1.593,"5min":0.96799999999999997,"15min":0.69699999999999995},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":7.1139999999999999,"5min":15.082000000000001,"15min":15.082000000000001},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.113,"15min":0.76000000000000001},"min":{"1min":0.317,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":4.0629999999999997,"5min":15.978,"15min":15.978},"last":0.29399999999999998}]},{"osd":4,"last update":"Mon Feb 5 06:37:30 2024","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39800000000000002,"15min":0.40200000000000002},"min":{"1min":0.28499999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.628,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.443,"15min":0.42799999999999999},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":2.8260000000000001,"5min":2.8260000000000001,"15min":2.8260000000000001},"last":1.528}]},{"osd":6,"last update":"Mon Feb 5 06:36:56 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.46999999999999997,"15min":0.45300000000000001},"min":{"1min":0.33300000000000002,"5min":0.245,"15min":0.245},"max":{"1min":0.76100000000000001,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.48999999999999999,"15min":0.47399999999999998},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.70699999999999996}]},{"osd":7,"last update":"Mon Feb 5 06:37:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47199999999999998,"15min":0.46700000000000003},"min":{"1min":0.32400000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":1.6779999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.5,"15min":0.498},"min":{"1min":0.36499999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":2.798}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2841382912,"data_stored":2841378840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2484146176,"data_stored":2484142104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2141192192,"data_stored":2141192192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3546021888,"data_stored":3546021888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1229914112,"data_stored":1229914112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1365639168,"data_stored":1365639168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2071662592,"data_stored":2071658520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1718091776,"data_stored":1718091776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T06:37:51.539 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd pool set .mgr pgp_num 11 2024-02-05T06:37:51.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:51 smithi028 ceph-mon[29442]: 6.18 scrub starts 2024-02-05T06:37:51.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:51 smithi028 ceph-mon[29442]: 6.18 scrub ok 2024-02-05T06:37:51.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:51 smithi028 ceph-mon[29442]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:37:51.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:51 smithi028 ceph-mon[29442]: pgmap v171: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 155 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 372 op/s; 7603/251829 objects misplaced (3.019%); 13 MiB/s, 203 objects/s recovering 2024-02-05T06:37:51.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:51 smithi028 ceph-mon[29442]: from='client.24871 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:37:51.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:51 smithi028 ceph-mon[29442]: osdmap e141: 8 total, 8 up, 8 in 2024-02-05T06:37:51.540 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:51 smithi028 ceph-mon[37984]: 6.18 scrub starts 2024-02-05T06:37:51.540 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:51 smithi028 ceph-mon[37984]: 6.18 scrub ok 2024-02-05T06:37:51.540 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:51 smithi028 ceph-mon[37984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:37:51.540 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:51 smithi028 ceph-mon[37984]: pgmap v171: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 155 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 372 op/s; 7603/251829 objects misplaced (3.019%); 13 MiB/s, 203 objects/s recovering 2024-02-05T06:37:51.540 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:51 smithi028 ceph-mon[37984]: from='client.24871 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:37:51.540 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:51 smithi028 ceph-mon[37984]: osdmap e141: 8 total, 8 up, 8 in 2024-02-05T06:37:51.573 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:51.586 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:37:51.604 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:51.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:51.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:51.660 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:37:51.707 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:51.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:51 smithi167 ceph-mon[32774]: 6.18 scrub starts 2024-02-05T06:37:51.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:51 smithi167 ceph-mon[32774]: 6.18 scrub ok 2024-02-05T06:37:51.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:51 smithi167 ceph-mon[32774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:37:51.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:51 smithi167 ceph-mon[32774]: pgmap v171: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 155 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 0 B/s wr, 372 op/s; 7603/251829 objects misplaced (3.019%); 13 MiB/s, 203 objects/s recovering 2024-02-05T06:37:51.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:51 smithi167 ceph-mon[32774]: from='client.24871 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:37:51.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:51 smithi167 ceph-mon[32774]: osdmap e141: 8 total, 8 up, 8 in 2024-02-05T06:37:51.773 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:51.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:51.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:51.876 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:51.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:51.922 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:51.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:52.023 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:52.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:52.105 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:52.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:52.206 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:52.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:52.237 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:52.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:52.339 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:52.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:52.394 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:52.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:52.495 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:52.525 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:52.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:52.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:52.627 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:52.655 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:52.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:52.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:52.695 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:37:52 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:37:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:37:52.757 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:52.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:52.788 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:52.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:52.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:37:52 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:37:52.857 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:37:52.890 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:52.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:52.920 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:52.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:53.021 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:53.070 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:53.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:53.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:53.172 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:53.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:53.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:53.295 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:53 smithi028 ceph-mon[37984]: pgmap v173: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing+deep, 1 active+remapped+backfilling, 155 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 907 op/s; 6354/238977 objects misplaced (2.659%); 20 MiB/s, 312 objects/s recovering 2024-02-05T06:37:53.295 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:53 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:37:53.296 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:53 smithi028 ceph-mon[29442]: pgmap v173: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing+deep, 1 active+remapped+backfilling, 155 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 907 op/s; 6354/238977 objects misplaced (2.659%); 20 MiB/s, 312 objects/s recovering 2024-02-05T06:37:53.296 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:53 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:37:53.316 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:53.345 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:53.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:53.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:53.446 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:53.476 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:53.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:53.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:53.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:53 smithi167 ceph-mon[32774]: pgmap v173: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing+deep, 1 active+remapped+backfilling, 155 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 907 op/s; 6354/238977 objects misplaced (2.659%); 20 MiB/s, 312 objects/s recovering 2024-02-05T06:37:53.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:53 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:37:53.578 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:53.626 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:53.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:53.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:53.684 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:37:53.684 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:37:53.727 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:53.772 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:53.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:53.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:53.874 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:53.905 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:53.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:53.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:54.007 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:54.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:54.046 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:54.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:54.091 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T06:37:54.148 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:54.179 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:54.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:54.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:54.281 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:54.311 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:54.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:54.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:54.414 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:54.476 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:54.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:54.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:54.517 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-05T06:37:54.578 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:54.618 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:54.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:54.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:54.721 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:54.757 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:54.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:54.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:54.859 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:54.887 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:37:54.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:54.898 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:54.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:54.999 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:55.029 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:55.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:55.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:55.131 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:55.161 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:55.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:55.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:55.177 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:37:55.178 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:37:55.178 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:37:55.178 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:37:55.178 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:37:55.178 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:37:55.178 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:37:55.178 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:37:55.178 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:37:55.178 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:37:55.178 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:37:55.178 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:37:55.179 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:37:55.179 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:37:55.179 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:37:55.179 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:37:55.262 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:55.295 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:55.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:55.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:55.397 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:55.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:55.430 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:55.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:55.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:55 smithi028 ceph-mon[29442]: pgmap v174: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing+deep, 1 active+remapped+backfilling, 155 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 781 op/s; 6354/238977 objects misplaced (2.659%); 20 MiB/s, 312 objects/s recovering 2024-02-05T06:37:55.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:55 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2367754264' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:37:55.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:55 smithi028 ceph-mon[37984]: pgmap v174: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing+deep, 1 active+remapped+backfilling, 155 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 781 op/s; 6354/238977 objects misplaced (2.659%); 20 MiB/s, 312 objects/s recovering 2024-02-05T06:37:55.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:55 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2367754264' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:37:55.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:55 smithi167 ceph-mon[32774]: pgmap v174: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing+deep, 1 active+remapped+backfilling, 155 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 0 B/s wr, 781 op/s; 6354/238977 objects misplaced (2.659%); 20 MiB/s, 312 objects/s recovering 2024-02-05T06:37:55.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:55 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2367754264' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:37:55.532 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:37:55.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:55.566 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:55.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:37:55.667 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:55.699 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:55.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:55.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:55.801 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:55.832 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:55.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:55.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:55.934 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:55.989 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:55.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:55.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:56.091 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:56.133 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:56.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:56.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:56.234 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:56.264 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:56.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:56.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:56.366 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:56.403 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:56.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:56.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:56.505 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:56.537 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:56.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:56.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:56.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:56.676 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:56.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:56.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:56.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:56 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:37:56.695 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:56 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:37:56.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:56 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:37:56.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:56 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:37:56.762 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:56 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:37:56.762 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:56 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:37:56.779 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:56.807 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:56.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:56.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:56.844 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:37:56.908 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:56.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:56.969 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:56.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:57.070 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:57.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:57.107 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:57.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:57.208 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:57.217 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:37:57.217 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-02-05T06:37:57.217 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-02-05T06:37:57.218 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-02-05T06:37:57.218 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:37:57.240 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:57.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:57.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:57.341 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:57.369 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:57.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:57.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:57.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:57.518 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:57.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:57.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:57.620 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:57.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:57 smithi028 ceph-mon[29442]: pgmap v175: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 155 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 741 op/s; 6354/234057 objects misplaced (2.715%); 6.8 MiB/s, 108 objects/s recovering 2024-02-05T06:37:57.645 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:57 smithi028 ceph-mon[37984]: pgmap v175: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 155 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 741 op/s; 6354/234057 objects misplaced (2.715%); 6.8 MiB/s, 108 objects/s recovering 2024-02-05T06:37:57.657 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:57.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:57.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:57.759 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:57.798 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:57.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:57.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:57.842 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:37:57.881 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:57 smithi167 ceph-mon[32774]: pgmap v175: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 155 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 741 op/s; 6354/234057 objects misplaced (2.715%); 6.8 MiB/s, 108 objects/s recovering 2024-02-05T06:37:57.900 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:57.923 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-05T06:37:57.931 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:57.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:57.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:58.034 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:58.064 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:58.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:58.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:58.166 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:37:58.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:58.222 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:58.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:37:58.253 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:37:58.323 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:58.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:58.366 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:58.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:58.467 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:58.493 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:58.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:58.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:58.545 INFO:teuthology.orchestra.run.smithi028.stderr:set pool 1 pgp_num to 11 2024-02-05T06:37:58.596 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:58.626 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:58.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:58.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:58.659 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:58 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1151634008' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-02-05T06:37:58.659 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:58 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1151634008' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-02-05T06:37:58.668 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:58 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1151634008' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-02-05T06:37:58.728 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:37:58.758 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:58.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:58.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:37:58.860 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:37:58.890 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:58.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:58.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:37:58.993 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:59.020 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:59.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:59.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:59.122 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:37:59.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:59.146 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:59.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:37:59.247 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:59.272 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:59.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:59.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:59.374 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:59.398 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:59.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:59.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:37:59.500 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:37:59.525 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:59.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:59.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:37:59.627 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:59.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:59 smithi028 ceph-mon[29442]: pgmap v176: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 155 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 957 op/s; 5075/226005 objects misplaced (2.246%); 13 MiB/s, 202 objects/s recovering 2024-02-05T06:37:59.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:59 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1151634008' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-02-05T06:37:59.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:37:59 smithi028 ceph-mon[29442]: osdmap e142: 8 total, 8 up, 8 in 2024-02-05T06:37:59.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:59 smithi028 ceph-mon[37984]: pgmap v176: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 155 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 957 op/s; 5075/226005 objects misplaced (2.246%); 13 MiB/s, 202 objects/s recovering 2024-02-05T06:37:59.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:59 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1151634008' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-02-05T06:37:59.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:37:59 smithi028 ceph-mon[37984]: osdmap e142: 8 total, 8 up, 8 in 2024-02-05T06:37:59.687 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:59.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:59.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:59.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:59 smithi167 ceph-mon[32774]: pgmap v176: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 155 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 957 op/s; 5075/226005 objects misplaced (2.246%); 13 MiB/s, 202 objects/s recovering 2024-02-05T06:37:59.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:59 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1151634008' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-02-05T06:37:59.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:37:59 smithi167 ceph-mon[32774]: osdmap e142: 8 total, 8 up, 8 in 2024-02-05T06:37:59.765 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:37:59.789 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:37:59.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:59.843 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:37:59.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:37:59.944 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:37:59.978 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:37:59.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:37:59.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:00.080 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:00.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:00.166 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:00.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:00.267 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:00.314 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:00.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:00.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:00.327 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:38:00.328 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-02-05T06:38:00.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-02-05T06:38:00.328 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-02-05T06:38:00.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.328 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:153555 7.1 7:8a0efd98:::benchmark_data_smithi028_77509_object38435:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-05T06:38:00.328 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:37:54.582809+0000", 2024-02-05T06:38:00.328 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.7392954090000003, 2024-02-05T06:38:00.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018448177999999999, 2024-02-05T06:38:00.328 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:38:00.328 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:38:00.328 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:38:00.328 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:38:00.328 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:38:00.329 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 153555 2024-02-05T06:38:00.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.329 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:38:00.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:38:00.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.582809+0000", 2024-02-05T06:38:00.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:38:00.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.582809+0000", 2024-02-05T06:38:00.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:38:00.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.582810+0000", 2024-02-05T06:38:00.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.125e-06 2024-02-05T06:38:00.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:38:00.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.582815+0000", 2024-02-05T06:38:00.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9849999999999997e-06 2024-02-05T06:38:00.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:38:00.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.582825+0000", 2024-02-05T06:38:00.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3239999999999992e-06 2024-02-05T06:38:00.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:38:00.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.582832+0000", 2024-02-05T06:38:00.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.401e-06 2024-02-05T06:38:00.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:38:00.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.586547+0000", 2024-02-05T06:38:00.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037152790000000002 2024-02-05T06:38:00.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:38:00.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.586991+0000", 2024-02-05T06:38:00.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000443955 2024-02-05T06:38:00.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-02-05T06:38:00.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.587110+0000", 2024-02-05T06:38:00.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118373 2024-02-05T06:38:00.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:38:00.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.591388+0000", 2024-02-05T06:38:00.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0042783439999999999 2024-02-05T06:38:00.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.598772+0000", 2024-02-05T06:38:00.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0073834520000000004 2024-02-05T06:38:00.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.601152+0000", 2024-02-05T06:38:00.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023809450000000002 2024-02-05T06:38:00.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.333 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:38:00.333 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.601219+0000", 2024-02-05T06:38:00.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6340000000000005e-05 2024-02-05T06:38:00.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:38:00.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.601257+0000", 2024-02-05T06:38:00.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8655000000000002e-05 2024-02-05T06:38:00.334 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.334 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:38:00.334 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.334 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.14847.0:153698 7.4 e141/138 7:2c8f5b10:::benchmark_data_smithi028_77509_object38578:head v 141'9674, mlcod=141'9671)", 2024-02-05T06:38:00.334 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:37:54.760235+0000", 2024-02-05T06:38:00.334 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.5618700079999996, 2024-02-05T06:38:00.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016143347999999998, 2024-02-05T06:38:00.334 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:38:00.334 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:38:00.335 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:38:00.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:38:00.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.760235+0000", 2024-02-05T06:38:00.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:38:00.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.760235+0000", 2024-02-05T06:38:00.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:38:00.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.760235+0000", 2024-02-05T06:38:00.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9400000000000005e-07 2024-02-05T06:38:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:38:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.760240+0000", 2024-02-05T06:38:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4580000000000003e-06 2024-02-05T06:38:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:38:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.760245+0000", 2024-02-05T06:38:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1080000000000001e-06 2024-02-05T06:38:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:38:00.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.760249+0000", 2024-02-05T06:38:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3139999999999997e-06 2024-02-05T06:38:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:38:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.765536+0000", 2024-02-05T06:38:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0052864160000000004 2024-02-05T06:38:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:38:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.765605+0000", 2024-02-05T06:38:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9201000000000004e-05 2024-02-05T06:38:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:38:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.776348+0000", 2024-02-05T06:38:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010743145000000001 2024-02-05T06:38:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:38:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.776378+0000", 2024-02-05T06:38:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0111999999999999e-05 2024-02-05T06:38:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:38:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:154041 7.1 7:882597ca:::benchmark_data_smithi028_77509_object38921:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-05T06:38:00.338 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:37:55.145724+0000", 2024-02-05T06:38:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.1763809939999996, 2024-02-05T06:38:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017793862000000001, 2024-02-05T06:38:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:38:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:38:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:38:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:38:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:38:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 154041 2024-02-05T06:38:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:38:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:38:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.145724+0000", 2024-02-05T06:38:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:38:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.145724+0000", 2024-02-05T06:38:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:38:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.145724+0000", 2024-02-05T06:38:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7700000000000005e-07 2024-02-05T06:38:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:38:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.145726+0000", 2024-02-05T06:38:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1950000000000002e-06 2024-02-05T06:38:00.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:38:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.145730+0000", 2024-02-05T06:38:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6950000000000002e-06 2024-02-05T06:38:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:38:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.145733+0000", 2024-02-05T06:38:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3529999999999999e-06 2024-02-05T06:38:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:38:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.147441+0000", 2024-02-05T06:38:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0017073590000000001 2024-02-05T06:38:00.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:38:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.147564+0000", 2024-02-05T06:38:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012353099999999999 2024-02-05T06:38:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-02-05T06:38:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.147686+0000", 2024-02-05T06:38:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121357 2024-02-05T06:38:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:38:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.151149+0000", 2024-02-05T06:38:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003463739 2024-02-05T06:38:00.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.151719+0000", 2024-02-05T06:38:00.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00056961000000000002 2024-02-05T06:38:00.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.163426+0000", 2024-02-05T06:38:00.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011707232999999999 2024-02-05T06:38:00.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:38:00.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.163480+0000", 2024-02-05T06:38:00.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4199000000000001e-05 2024-02-05T06:38:00.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:38:00.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.163518+0000", 2024-02-05T06:38:00.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7113999999999997e-05 2024-02-05T06:38:00.344 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.344 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:38:00.344 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.344 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:154045 7.1 7:86339cef:::benchmark_data_smithi028_77509_object38925:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-05T06:38:00.344 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:37:55.148524+0000", 2024-02-05T06:38:00.344 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.1735803630000001, 2024-02-05T06:38:00.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.01558753, 2024-02-05T06:38:00.344 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:38:00.344 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:38:00.345 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:38:00.345 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:38:00.345 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:38:00.345 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 154045 2024-02-05T06:38:00.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.345 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:38:00.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:38:00.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.148524+0000", 2024-02-05T06:38:00.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:38:00.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.148524+0000", 2024-02-05T06:38:00.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:38:00.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.148525+0000", 2024-02-05T06:38:00.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0559999999999999e-06 2024-02-05T06:38:00.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:38:00.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.148529+0000", 2024-02-05T06:38:00.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.019e-06 2024-02-05T06:38:00.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:38:00.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.148536+0000", 2024-02-05T06:38:00.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.443e-06 2024-02-05T06:38:00.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:38:00.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.148542+0000", 2024-02-05T06:38:00.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9839999999999998e-06 2024-02-05T06:38:00.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:38:00.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.149705+0000", 2024-02-05T06:38:00.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001162819 2024-02-05T06:38:00.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:38:00.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.149817+0000", 2024-02-05T06:38:00.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000112855 2024-02-05T06:38:00.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-02-05T06:38:00.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.149938+0000", 2024-02-05T06:38:00.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012081 2024-02-05T06:38:00.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:38:00.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.151263+0000", 2024-02-05T06:38:00.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0013249080000000001 2024-02-05T06:38:00.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.151806+0000", 2024-02-05T06:38:00.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00054304099999999997 2024-02-05T06:38:00.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.163997+0000", 2024-02-05T06:38:00.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012190829 2024-02-05T06:38:00.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.349 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:38:00.349 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.164075+0000", 2024-02-05T06:38:00.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7960000000000006e-05 2024-02-05T06:38:00.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:38:00.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.164112+0000", 2024-02-05T06:38:00.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6806e-05 2024-02-05T06:38:00.350 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.350 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:38:00.350 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.350 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:154405 7.1 7:8cd67aa4:::benchmark_data_smithi028_77509_object39285:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-05T06:38:00.350 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:37:55.537509+0000", 2024-02-05T06:38:00.350 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.7845956569999997, 2024-02-05T06:38:00.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015492789999999999, 2024-02-05T06:38:00.350 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:38:00.350 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:38:00.350 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:38:00.351 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:38:00.351 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:38:00.351 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 154405 2024-02-05T06:38:00.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.351 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:38:00.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:38:00.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.537509+0000", 2024-02-05T06:38:00.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:38:00.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.537509+0000", 2024-02-05T06:38:00.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:38:00.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.537511+0000", 2024-02-05T06:38:00.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9010000000000001e-06 2024-02-05T06:38:00.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:38:00.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.537517+0000", 2024-02-05T06:38:00.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8379999999999998e-06 2024-02-05T06:38:00.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:38:00.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.537526+0000", 2024-02-05T06:38:00.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5550000000000008e-06 2024-02-05T06:38:00.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:38:00.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.537534+0000", 2024-02-05T06:38:00.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5510000000000002e-06 2024-02-05T06:38:00.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:38:00.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.537890+0000", 2024-02-05T06:38:00.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00035604199999999999 2024-02-05T06:38:00.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:38:00.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.538012+0000", 2024-02-05T06:38:00.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012209100000000001 2024-02-05T06:38:00.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-02-05T06:38:00.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.538259+0000", 2024-02-05T06:38:00.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000246939 2024-02-05T06:38:00.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:38:00.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.543822+0000", 2024-02-05T06:38:00.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055628170000000003 2024-02-05T06:38:00.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.545951+0000", 2024-02-05T06:38:00.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021289490000000002 2024-02-05T06:38:00.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.552799+0000", 2024-02-05T06:38:00.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.006847985 2024-02-05T06:38:00.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:38:00.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.552953+0000", 2024-02-05T06:38:00.356 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015465800000000001 2024-02-05T06:38:00.356 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.356 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.356 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:38:00.356 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.553002+0000", 2024-02-05T06:38:00.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8464000000000002e-05 2024-02-05T06:38:00.357 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.357 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:38:00.357 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.357 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:154753 7.1 7:8fe8923a:::benchmark_data_smithi028_77509_object39633:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-05T06:38:00.357 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:37:55.949189+0000", 2024-02-05T06:38:00.357 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.3729158909999999, 2024-02-05T06:38:00.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015790404000000001, 2024-02-05T06:38:00.357 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:38:00.357 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:38:00.357 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:38:00.357 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:38:00.357 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:38:00.358 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 154753 2024-02-05T06:38:00.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.358 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:38:00.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:38:00.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.949189+0000", 2024-02-05T06:38:00.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.358 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:38:00.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.949189+0000", 2024-02-05T06:38:00.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:38:00.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.949191+0000", 2024-02-05T06:38:00.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1509999999999998e-06 2024-02-05T06:38:00.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:38:00.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.949197+0000", 2024-02-05T06:38:00.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0939999999999996e-06 2024-02-05T06:38:00.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:38:00.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.949213+0000", 2024-02-05T06:38:00.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6376e-05 2024-02-05T06:38:00.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:38:00.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.949222+0000", 2024-02-05T06:38:00.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0340000000000002e-06 2024-02-05T06:38:00.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:38:00.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.949280+0000", 2024-02-05T06:38:00.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7484000000000001e-05 2024-02-05T06:38:00.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:38:00.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.949367+0000", 2024-02-05T06:38:00.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7126000000000006e-05 2024-02-05T06:38:00.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-02-05T06:38:00.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.949496+0000", 2024-02-05T06:38:00.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012874300000000001 2024-02-05T06:38:00.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:38:00.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.951474+0000", 2024-02-05T06:38:00.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019782089999999999 2024-02-05T06:38:00.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.952075+0000", 2024-02-05T06:38:00.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00060084600000000004 2024-02-05T06:38:00.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.964873+0000", 2024-02-05T06:38:00.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012798075000000001 2024-02-05T06:38:00.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:38:00.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.964932+0000", 2024-02-05T06:38:00.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8730999999999997e-05 2024-02-05T06:38:00.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:38:00.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.964979+0000", 2024-02-05T06:38:00.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7534999999999998e-05 2024-02-05T06:38:00.362 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.363 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:38:00.363 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.363 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:155498 7.1 7:8e3af501:::benchmark_data_smithi028_77509_object40378:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-05T06:38:00.363 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:37:56.825317+0000", 2024-02-05T06:38:00.363 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.496787205, 2024-02-05T06:38:00.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017317203999999999, 2024-02-05T06:38:00.363 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:38:00.363 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:38:00.363 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:38:00.363 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:38:00.363 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:38:00.363 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 155498 2024-02-05T06:38:00.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.364 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:38:00.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:38:00.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:56.825317+0000", 2024-02-05T06:38:00.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:38:00.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:56.825317+0000", 2024-02-05T06:38:00.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:38:00.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:56.825319+0000", 2024-02-05T06:38:00.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4789999999999999e-06 2024-02-05T06:38:00.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:38:00.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:56.825324+0000", 2024-02-05T06:38:00.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7319999999999998e-06 2024-02-05T06:38:00.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:38:00.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:56.825335+0000", 2024-02-05T06:38:00.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1787000000000001e-05 2024-02-05T06:38:00.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:38:00.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:56.825343+0000", 2024-02-05T06:38:00.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9279999999999993e-06 2024-02-05T06:38:00.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:38:00.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:56.825398+0000", 2024-02-05T06:38:00.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4784000000000003e-05 2024-02-05T06:38:00.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:38:00.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:56.825519+0000", 2024-02-05T06:38:00.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012061300000000001 2024-02-05T06:38:00.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.366 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-02-05T06:38:00.366 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:56.825636+0000", 2024-02-05T06:38:00.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000116747 2024-02-05T06:38:00.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:38:00.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:56.828061+0000", 2024-02-05T06:38:00.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024249800000000002 2024-02-05T06:38:00.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:56.828109+0000", 2024-02-05T06:38:00.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8797000000000001e-05 2024-02-05T06:38:00.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:56.842521+0000", 2024-02-05T06:38:00.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.014411396999999999 2024-02-05T06:38:00.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:38:00.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:56.842591+0000", 2024-02-05T06:38:00.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9840999999999998e-05 2024-02-05T06:38:00.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:38:00.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:56.842635+0000", 2024-02-05T06:38:00.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4119000000000002e-05 2024-02-05T06:38:00.368 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.368 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:38:00.368 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.368 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:155792 7.1 7:87dcae05:::benchmark_data_smithi028_77509_object40672:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-05T06:38:00.369 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:37:57.183572+0000", 2024-02-05T06:38:00.369 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 3.1385331569999999, 2024-02-05T06:38:00.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017610361000000001, 2024-02-05T06:38:00.369 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:38:00.369 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:38:00.369 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:38:00.369 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:38:00.369 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:38:00.369 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 155792 2024-02-05T06:38:00.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.369 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:38:00.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:38:00.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.183572+0000", 2024-02-05T06:38:00.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:38:00.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.183572+0000", 2024-02-05T06:38:00.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:38:00.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.183573+0000", 2024-02-05T06:38:00.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4139999999999999e-06 2024-02-05T06:38:00.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:38:00.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.183579+0000", 2024-02-05T06:38:00.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1510000000000001e-06 2024-02-05T06:38:00.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:38:00.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.183589+0000", 2024-02-05T06:38:00.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7860000000000008e-06 2024-02-05T06:38:00.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:38:00.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.183596+0000", 2024-02-05T06:38:00.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5159999999999997e-06 2024-02-05T06:38:00.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:38:00.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.183627+0000", 2024-02-05T06:38:00.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0562000000000003e-05 2024-02-05T06:38:00.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:38:00.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.183713+0000", 2024-02-05T06:38:00.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5573000000000002e-05 2024-02-05T06:38:00.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-02-05T06:38:00.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.183790+0000", 2024-02-05T06:38:00.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7919999999999999e-05 2024-02-05T06:38:00.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:38:00.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.185581+0000", 2024-02-05T06:38:00.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001790975 2024-02-05T06:38:00.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.186134+0000", 2024-02-05T06:38:00.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00055258800000000002 2024-02-05T06:38:00.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.201076+0000", 2024-02-05T06:38:00.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.014942202999999999 2024-02-05T06:38:00.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:38:00.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.201134+0000", 2024-02-05T06:38:00.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7451000000000003e-05 2024-02-05T06:38:00.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:38:00.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.201182+0000", 2024-02-05T06:38:00.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8222e-05 2024-02-05T06:38:00.374 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.374 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:38:00.374 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.374 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:155998 7.1 7:8b988ab9:::benchmark_data_smithi028_77509_object40878:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-05T06:38:00.374 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:37:57.443971+0000", 2024-02-05T06:38:00.374 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 2.8781332769999999, 2024-02-05T06:38:00.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016572631000000001, 2024-02-05T06:38:00.375 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:38:00.375 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:38:00.375 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:38:00.375 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:38:00.375 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:38:00.375 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 155998 2024-02-05T06:38:00.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.375 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:38:00.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:38:00.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.443971+0000", 2024-02-05T06:38:00.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:38:00.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.443971+0000", 2024-02-05T06:38:00.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:38:00.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.443973+0000", 2024-02-05T06:38:00.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2219999999999999e-06 2024-02-05T06:38:00.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:38:00.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.443977+0000", 2024-02-05T06:38:00.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3889999999999998e-06 2024-02-05T06:38:00.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:38:00.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.443987+0000", 2024-02-05T06:38:00.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7799999999999995e-06 2024-02-05T06:38:00.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:38:00.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.443994+0000", 2024-02-05T06:38:00.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9249999999999998e-06 2024-02-05T06:38:00.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:38:00.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.447876+0000", 2024-02-05T06:38:00.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038817909999999999 2024-02-05T06:38:00.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:38:00.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.448045+0000", 2024-02-05T06:38:00.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016918900000000001 2024-02-05T06:38:00.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-02-05T06:38:00.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.448282+0000", 2024-02-05T06:38:00.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023734 2024-02-05T06:38:00.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:38:00.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.451480+0000", 2024-02-05T06:38:00.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031979550000000002 2024-02-05T06:38:00.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.451575+0000", 2024-02-05T06:38:00.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4927999999999996e-05 2024-02-05T06:38:00.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.460429+0000", 2024-02-05T06:38:00.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088544069999999999 2024-02-05T06:38:00.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:38:00.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.460504+0000", 2024-02-05T06:38:00.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4314999999999996e-05 2024-02-05T06:38:00.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:38:00.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.460544+0000", 2024-02-05T06:38:00.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0389999999999998e-05 2024-02-05T06:38:00.380 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.380 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:38:00.380 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.380 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:156245 7.1 7:83f43ef0:::benchmark_data_smithi028_77509_object41125:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-05T06:38:00.380 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:37:57.735875+0000", 2024-02-05T06:38:00.380 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 2.5862296169999999, 2024-02-05T06:38:00.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021990206000000002, 2024-02-05T06:38:00.380 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:38:00.380 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:38:00.381 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:38:00.381 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:38:00.381 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:38:00.381 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 156245 2024-02-05T06:38:00.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.381 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:38:00.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:38:00.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.735875+0000", 2024-02-05T06:38:00.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.381 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:38:00.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.735875+0000", 2024-02-05T06:38:00.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:38:00.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.735877+0000", 2024-02-05T06:38:00.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9120000000000001e-06 2024-02-05T06:38:00.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:38:00.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.735881+0000", 2024-02-05T06:38:00.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4769999999999997e-06 2024-02-05T06:38:00.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:38:00.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.735892+0000", 2024-02-05T06:38:00.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1035e-05 2024-02-05T06:38:00.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:38:00.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.735901+0000", 2024-02-05T06:38:00.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1810000000000001e-06 2024-02-05T06:38:00.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:38:00.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.735955+0000", 2024-02-05T06:38:00.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4230000000000001e-05 2024-02-05T06:38:00.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:38:00.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.736078+0000", 2024-02-05T06:38:00.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012329299999999999 2024-02-05T06:38:00.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-02-05T06:38:00.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.736194+0000", 2024-02-05T06:38:00.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011620000000000001 2024-02-05T06:38:00.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:38:00.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.738548+0000", 2024-02-05T06:38:00.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023531300000000002 2024-02-05T06:38:00.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.738789+0000", 2024-02-05T06:38:00.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00024188899999999999 2024-02-05T06:38:00.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.757770+0000", 2024-02-05T06:38:00.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018980932999999998 2024-02-05T06:38:00.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:38:00.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.757815+0000", 2024-02-05T06:38:00.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4168999999999997e-05 2024-02-05T06:38:00.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:38:00.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.757865+0000", 2024-02-05T06:38:00.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0757000000000002e-05 2024-02-05T06:38:00.386 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.386 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:38:00.386 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.386 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:156251 7.1 7:8f6110a4:::benchmark_data_smithi028_77509_object41131:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-05T06:38:00.386 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:37:57.742657+0000", 2024-02-05T06:38:00.386 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 2.5794479470000002, 2024-02-05T06:38:00.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019485061000000001, 2024-02-05T06:38:00.386 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:38:00.386 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:38:00.387 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:38:00.387 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:38:00.387 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:38:00.387 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 156251 2024-02-05T06:38:00.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.387 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:38:00.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:38:00.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.742657+0000", 2024-02-05T06:38:00.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:38:00.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.742657+0000", 2024-02-05T06:38:00.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:38:00.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.742658+0000", 2024-02-05T06:38:00.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6449999999999999e-06 2024-02-05T06:38:00.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:38:00.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.742663+0000", 2024-02-05T06:38:00.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4410000000000003e-06 2024-02-05T06:38:00.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:38:00.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.742674+0000", 2024-02-05T06:38:00.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1199000000000001e-05 2024-02-05T06:38:00.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:38:00.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.742680+0000", 2024-02-05T06:38:00.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2249999999999997e-06 2024-02-05T06:38:00.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:38:00.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.742722+0000", 2024-02-05T06:38:00.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1814999999999999e-05 2024-02-05T06:38:00.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:38:00.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.742803+0000", 2024-02-05T06:38:00.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0591000000000001e-05 2024-02-05T06:38:00.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-02-05T06:38:00.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.742877+0000", 2024-02-05T06:38:00.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4505000000000006e-05 2024-02-05T06:38:00.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.750696+0000", 2024-02-05T06:38:00.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078193280000000004 2024-02-05T06:38:00.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:38:00.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.750711+0000", 2024-02-05T06:38:00.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4129e-05 2024-02-05T06:38:00.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.762002+0000", 2024-02-05T06:38:00.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011291759 2024-02-05T06:38:00.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:38:00.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.762099+0000", 2024-02-05T06:38:00.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6218000000000006e-05 2024-02-05T06:38:00.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:38:00.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.762142+0000", 2024-02-05T06:38:00.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3205999999999999e-05 2024-02-05T06:38:00.392 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.392 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:38:00.392 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.392 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:156252 7.1 7:883458fd:::benchmark_data_smithi028_77509_object41132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-05T06:38:00.392 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:37:57.743096+0000", 2024-02-05T06:38:00.392 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 2.5790088940000002, 2024-02-05T06:38:00.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019212177, 2024-02-05T06:38:00.392 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:38:00.392 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:38:00.392 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:38:00.392 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:38:00.392 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:38:00.392 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 156252 2024-02-05T06:38:00.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.393 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:38:00.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:38:00.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.743096+0000", 2024-02-05T06:38:00.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:38:00.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.743096+0000", 2024-02-05T06:38:00.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:38:00.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.743097+0000", 2024-02-05T06:38:00.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.116e-06 2024-02-05T06:38:00.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:38:00.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.743101+0000", 2024-02-05T06:38:00.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4009999999999999e-06 2024-02-05T06:38:00.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:38:00.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.743113+0000", 2024-02-05T06:38:00.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2078e-05 2024-02-05T06:38:00.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:38:00.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.743121+0000", 2024-02-05T06:38:00.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5789999999999998e-06 2024-02-05T06:38:00.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:38:00.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.745020+0000", 2024-02-05T06:38:00.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001898781 2024-02-05T06:38:00.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:38:00.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.745128+0000", 2024-02-05T06:38:00.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000108032 2024-02-05T06:38:00.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-02-05T06:38:00.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.745250+0000", 2024-02-05T06:38:00.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000122717 2024-02-05T06:38:00.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:38:00.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.750718+0000", 2024-02-05T06:38:00.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0054673150000000004 2024-02-05T06:38:00.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.753143+0000", 2024-02-05T06:38:00.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024252869999999999 2024-02-05T06:38:00.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.762185+0000", 2024-02-05T06:38:00.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090416999999999997 2024-02-05T06:38:00.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:38:00.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.762269+0000", 2024-02-05T06:38:00.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4033e-05 2024-02-05T06:38:00.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:38:00.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.762308+0000", 2024-02-05T06:38:00.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9137999999999997e-05 2024-02-05T06:38:00.397 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.397 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:38:00.397 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.398 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:156256 7.1 7:85f1e46b:::benchmark_data_smithi028_77509_object41136:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-05T06:38:00.398 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:37:57.745837+0000", 2024-02-05T06:38:00.398 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 2.5762673039999999, 2024-02-05T06:38:00.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016674610999999999, 2024-02-05T06:38:00.398 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:38:00.398 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:38:00.398 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:38:00.398 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:38:00.398 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:38:00.398 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 156256 2024-02-05T06:38:00.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.398 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:38:00.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:38:00.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.745837+0000", 2024-02-05T06:38:00.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:38:00.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.745837+0000", 2024-02-05T06:38:00.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:38:00.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.745854+0000", 2024-02-05T06:38:00.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6464e-05 2024-02-05T06:38:00.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:38:00.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.745859+0000", 2024-02-05T06:38:00.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1059999999999999e-06 2024-02-05T06:38:00.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:38:00.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.745872+0000", 2024-02-05T06:38:00.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3417e-05 2024-02-05T06:38:00.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:38:00.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.745880+0000", 2024-02-05T06:38:00.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7139999999999993e-06 2024-02-05T06:38:00.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:38:00.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.748094+0000", 2024-02-05T06:38:00.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00221383 2024-02-05T06:38:00.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:38:00.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.748215+0000", 2024-02-05T06:38:00.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012147899999999999 2024-02-05T06:38:00.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-02-05T06:38:00.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.748335+0000", 2024-02-05T06:38:00.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011992199999999999 2024-02-05T06:38:00.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:38:00.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.750722+0000", 2024-02-05T06:38:00.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023869960000000002 2024-02-05T06:38:00.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.753191+0000", 2024-02-05T06:38:00.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024684630000000002 2024-02-05T06:38:00.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.762387+0000", 2024-02-05T06:38:00.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0091964070000000002 2024-02-05T06:38:00.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:38:00.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.762474+0000", 2024-02-05T06:38:00.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7009999999999995e-05 2024-02-05T06:38:00.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:38:00.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.762512+0000", 2024-02-05T06:38:00.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7802999999999997e-05 2024-02-05T06:38:00.403 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.403 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:38:00.403 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.403 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:156257 7.1 7:8f871b44:::benchmark_data_smithi028_77509_object41137:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-05T06:38:00.404 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:37:57.745924+0000", 2024-02-05T06:38:00.404 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 2.576180301, 2024-02-05T06:38:00.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.016801769000000001, 2024-02-05T06:38:00.404 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:38:00.404 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:38:00.404 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:38:00.404 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:38:00.404 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:38:00.404 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 156257 2024-02-05T06:38:00.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.404 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:38:00.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:38:00.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.745924+0000", 2024-02-05T06:38:00.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:38:00.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.745924+0000", 2024-02-05T06:38:00.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:38:00.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.745925+0000", 2024-02-05T06:38:00.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7700000000000003e-07 2024-02-05T06:38:00.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:38:00.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.745928+0000", 2024-02-05T06:38:00.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.762e-06 2024-02-05T06:38:00.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.406 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:38:00.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.745935+0000", 2024-02-05T06:38:00.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3089999999999998e-06 2024-02-05T06:38:00.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.406 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:38:00.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.745941+0000", 2024-02-05T06:38:00.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.553e-06 2024-02-05T06:38:00.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.406 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:38:00.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.750747+0000", 2024-02-05T06:38:00.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048056640000000003 2024-02-05T06:38:00.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.407 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:38:00.407 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.750863+0000", 2024-02-05T06:38:00.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000116569 2024-02-05T06:38:00.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.407 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-02-05T06:38:00.407 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.750971+0000", 2024-02-05T06:38:00.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010755999999999999 2024-02-05T06:38:00.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.407 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:38:00.407 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.753407+0000", 2024-02-05T06:38:00.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024366090000000002 2024-02-05T06:38:00.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.753535+0000", 2024-02-05T06:38:00.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012770899999999999 2024-02-05T06:38:00.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.762608+0000", 2024-02-05T06:38:00.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090729790000000001 2024-02-05T06:38:00.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:38:00.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.762690+0000", 2024-02-05T06:38:00.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1545000000000006e-05 2024-02-05T06:38:00.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:38:00.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.762726+0000", 2024-02-05T06:38:00.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6633e-05 2024-02-05T06:38:00.409 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.409 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:38:00.409 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.409 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:156273 7.1 7:8896e771:::benchmark_data_smithi028_77509_object41153:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-05T06:38:00.409 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:37:57.764424+0000", 2024-02-05T06:38:00.409 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 2.557681069, 2024-02-05T06:38:00.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024082348999999999, 2024-02-05T06:38:00.410 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:38:00.410 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:38:00.410 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:38:00.410 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:38:00.410 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:38:00.410 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 156273 2024-02-05T06:38:00.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.410 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:38:00.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:38:00.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.764424+0000", 2024-02-05T06:38:00.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:38:00.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.764424+0000", 2024-02-05T06:38:00.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:38:00.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.764426+0000", 2024-02-05T06:38:00.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8989999999999999e-06 2024-02-05T06:38:00.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:38:00.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.764431+0000", 2024-02-05T06:38:00.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1669999999999998e-06 2024-02-05T06:38:00.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:38:00.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.764447+0000", 2024-02-05T06:38:00.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6753e-05 2024-02-05T06:38:00.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:38:00.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.764456+0000", 2024-02-05T06:38:00.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7269999999999996e-06 2024-02-05T06:38:00.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:38:00.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.764543+0000", 2024-02-05T06:38:00.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6703e-05 2024-02-05T06:38:00.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:38:00.413 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.764650+0000", 2024-02-05T06:38:00.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000107324 2024-02-05T06:38:00.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.413 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-02-05T06:38:00.413 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.764750+0000", 2024-02-05T06:38:00.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000100263 2024-02-05T06:38:00.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.413 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:38:00.413 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.769315+0000", 2024-02-05T06:38:00.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0045643899999999998 2024-02-05T06:38:00.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.413 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.769688+0000", 2024-02-05T06:38:00.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00037272999999999999 2024-02-05T06:38:00.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.788330+0000", 2024-02-05T06:38:00.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018642773000000001 2024-02-05T06:38:00.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:38:00.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.788463+0000", 2024-02-05T06:38:00.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013266299999999999 2024-02-05T06:38:00.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:38:00.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.788506+0000", 2024-02-05T06:38:00.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2957000000000002e-05 2024-02-05T06:38:00.415 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.415 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:38:00.415 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.415 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:156276 7.1 7:8b5c0612:::benchmark_data_smithi028_77509_object41156:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-05T06:38:00.415 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:37:57.764486+0000", 2024-02-05T06:38:00.415 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 2.5576189940000003, 2024-02-05T06:38:00.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024171775999999999, 2024-02-05T06:38:00.415 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:38:00.415 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:38:00.415 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:38:00.415 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:38:00.416 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:38:00.416 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 156276 2024-02-05T06:38:00.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.416 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:38:00.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:38:00.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.764486+0000", 2024-02-05T06:38:00.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:38:00.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.764486+0000", 2024-02-05T06:38:00.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:38:00.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.764487+0000", 2024-02-05T06:38:00.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.850000000000001e-07 2024-02-05T06:38:00.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:38:00.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.764491+0000", 2024-02-05T06:38:00.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3749999999999996e-06 2024-02-05T06:38:00.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:38:00.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.764504+0000", 2024-02-05T06:38:00.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3125999999999999e-05 2024-02-05T06:38:00.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:38:00.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.764512+0000", 2024-02-05T06:38:00.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4340000000000003e-06 2024-02-05T06:38:00.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:38:00.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.766984+0000", 2024-02-05T06:38:00.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024728060000000001 2024-02-05T06:38:00.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:38:00.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.767092+0000", 2024-02-05T06:38:00.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000107939 2024-02-05T06:38:00.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-02-05T06:38:00.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.767199+0000", 2024-02-05T06:38:00.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010702200000000001 2024-02-05T06:38:00.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:38:00.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.769417+0000", 2024-02-05T06:38:00.420 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:00.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022179510000000001 2024-02-05T06:38:00.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.769800+0000", 2024-02-05T06:38:00.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00038232400000000002 2024-02-05T06:38:00.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.788559+0000", 2024-02-05T06:38:00.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018759006000000002 2024-02-05T06:38:00.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:38:00.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.788617+0000", 2024-02-05T06:38:00.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8539000000000003e-05 2024-02-05T06:38:00.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:38:00.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.788657+0000", 2024-02-05T06:38:00.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0268999999999997e-05 2024-02-05T06:38:00.422 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.422 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:38:00.423 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.423 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:156364 7.1 7:887a57f2:::benchmark_data_smithi028_77509_object41244:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-05T06:38:00.423 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:37:57.859168+0000", 2024-02-05T06:38:00.423 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 2.4629362530000001, 2024-02-05T06:38:00.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.026949289000000001, 2024-02-05T06:38:00.423 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:38:00.423 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:38:00.423 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:38:00.423 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:38:00.423 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:38:00.423 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 156364 2024-02-05T06:38:00.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.424 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:38:00.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:38:00.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.859168+0000", 2024-02-05T06:38:00.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:38:00.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.859168+0000", 2024-02-05T06:38:00.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:38:00.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.859171+0000", 2024-02-05T06:38:00.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2110000000000001e-06 2024-02-05T06:38:00.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:38:00.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.859176+0000", 2024-02-05T06:38:00.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6659999999999996e-06 2024-02-05T06:38:00.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:38:00.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.859189+0000", 2024-02-05T06:38:00.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2944e-05 2024-02-05T06:38:00.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:38:00.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.859197+0000", 2024-02-05T06:38:00.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9629999999999998e-06 2024-02-05T06:38:00.426 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:38:00.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.859256+0000", 2024-02-05T06:38:00.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8356999999999997e-05 2024-02-05T06:38:00.426 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:38:00.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.859367+0000", 2024-02-05T06:38:00.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111298 2024-02-05T06:38:00.426 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-02-05T06:38:00.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.859472+0000", 2024-02-05T06:38:00.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010497099999999999 2024-02-05T06:38:00.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:38:00.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.861850+0000", 2024-02-05T06:38:00.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023785989999999999 2024-02-05T06:38:00.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.862307+0000", 2024-02-05T06:38:00.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00045621300000000002 2024-02-05T06:38:00.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.885973+0000", 2024-02-05T06:38:00.428 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023666423999999998 2024-02-05T06:38:00.428 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.428 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:38:00.428 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.886069+0000", 2024-02-05T06:38:00.428 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5946000000000006e-05 2024-02-05T06:38:00.428 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.428 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:38:00.428 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.886118+0000", 2024-02-05T06:38:00.428 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8696999999999998e-05 2024-02-05T06:38:00.428 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.428 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:38:00.428 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.428 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.429 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:156533 7.1 7:8a3346c8:::benchmark_data_smithi028_77509_object41413:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-05T06:38:00.429 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:37:58.073555+0000", 2024-02-05T06:38:00.429 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 2.2485497529999998, 2024-02-05T06:38:00.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.027279464, 2024-02-05T06:38:00.429 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:38:00.429 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:38:00.429 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:38:00.429 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:38:00.429 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:38:00.429 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 156533 2024-02-05T06:38:00.429 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.429 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:38:00.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.429 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:38:00.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.073555+0000", 2024-02-05T06:38:00.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.430 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:38:00.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.073555+0000", 2024-02-05T06:38:00.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.430 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:38:00.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.073557+0000", 2024-02-05T06:38:00.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8759999999999999e-06 2024-02-05T06:38:00.430 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:38:00.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.073562+0000", 2024-02-05T06:38:00.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1080000000000001e-06 2024-02-05T06:38:00.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:38:00.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.073573+0000", 2024-02-05T06:38:00.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1371e-05 2024-02-05T06:38:00.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:38:00.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.073580+0000", 2024-02-05T06:38:00.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5039999999999999e-06 2024-02-05T06:38:00.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:38:00.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.073644+0000", 2024-02-05T06:38:00.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4224e-05 2024-02-05T06:38:00.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.432 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.432 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:38:00.432 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.073791+0000", 2024-02-05T06:38:00.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000147078 2024-02-05T06:38:00.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.432 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.432 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-02-05T06:38:00.432 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.073946+0000", 2024-02-05T06:38:00.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000155336 2024-02-05T06:38:00.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.432 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.432 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:38:00.433 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.078520+0000", 2024-02-05T06:38:00.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0045734390000000003 2024-02-05T06:38:00.433 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.433 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.433 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.433 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.079082+0000", 2024-02-05T06:38:00.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00056171499999999996 2024-02-05T06:38:00.433 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.433 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.433 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.433 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.100736+0000", 2024-02-05T06:38:00.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021654577000000001 2024-02-05T06:38:00.433 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.433 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.433 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:38:00.433 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.100793+0000", 2024-02-05T06:38:00.434 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6703999999999998e-05 2024-02-05T06:38:00.434 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.434 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.434 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:38:00.434 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.100834+0000", 2024-02-05T06:38:00.434 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1532000000000001e-05 2024-02-05T06:38:00.434 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.434 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:38:00.434 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.434 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.434 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.434 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:156537 7.1 7:8cbc65d3:::benchmark_data_smithi028_77509_object41417:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-05T06:38:00.434 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:37:58.074414+0000", 2024-02-05T06:38:00.434 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 2.2476903579999998, 2024-02-05T06:38:00.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.026592665000000001, 2024-02-05T06:38:00.435 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:38:00.435 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:38:00.435 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:38:00.435 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:38:00.435 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:38:00.435 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 156537 2024-02-05T06:38:00.435 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.435 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:38:00.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.435 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:38:00.435 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.074414+0000", 2024-02-05T06:38:00.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.435 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:38:00.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.074414+0000", 2024-02-05T06:38:00.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:38:00.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.074416+0000", 2024-02-05T06:38:00.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.339e-06 2024-02-05T06:38:00.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:38:00.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.074421+0000", 2024-02-05T06:38:00.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0200000000000002e-06 2024-02-05T06:38:00.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:38:00.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.074432+0000", 2024-02-05T06:38:00.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1106e-05 2024-02-05T06:38:00.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:38:00.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.074439+0000", 2024-02-05T06:38:00.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5959999999999998e-06 2024-02-05T06:38:00.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:38:00.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.076161+0000", 2024-02-05T06:38:00.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00172179 2024-02-05T06:38:00.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:38:00.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.076276+0000", 2024-02-05T06:38:00.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011525499999999999 2024-02-05T06:38:00.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-02-05T06:38:00.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.076389+0000", 2024-02-05T06:38:00.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000112593 2024-02-05T06:38:00.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:38:00.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.078723+0000", 2024-02-05T06:38:00.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002333651 2024-02-05T06:38:00.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.079222+0000", 2024-02-05T06:38:00.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00049925399999999997 2024-02-05T06:38:00.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.100903+0000", 2024-02-05T06:38:00.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021681512999999999 2024-02-05T06:38:00.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:38:00.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.100965+0000", 2024-02-05T06:38:00.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1404999999999999e-05 2024-02-05T06:38:00.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.440 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:38:00.440 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.101007+0000", 2024-02-05T06:38:00.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2143e-05 2024-02-05T06:38:00.440 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.440 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:38:00.440 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.440 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:157521 7.1 7:8caa21e1:::benchmark_data_smithi028_77509_object42401:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-02-05T06:38:00.440 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:38:00.070121+0000", 2024-02-05T06:38:00.440 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.25198405499999998, 2024-02-05T06:38:00.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015298559, 2024-02-05T06:38:00.440 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:38:00.440 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:38:00.440 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:38:00.441 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:38:00.441 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:38:00.441 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 157521 2024-02-05T06:38:00.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.441 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:38:00.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.441 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:38:00.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.070121+0000", 2024-02-05T06:38:00.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.441 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:38:00.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.070121+0000", 2024-02-05T06:38:00.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:38:00.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:38:00.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.070122+0000", 2024-02-05T06:38:00.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.655e-06 2024-02-05T06:38:00.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:38:00.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.070126+0000", 2024-02-05T06:38:00.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.07e-06 2024-02-05T06:38:00.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:38:00.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.070136+0000", 2024-02-05T06:38:00.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9580000000000002e-06 2024-02-05T06:38:00.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:38:00.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.070142+0000", 2024-02-05T06:38:00.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4990000000000002e-06 2024-02-05T06:38:00.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:38:00.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.070193+0000", 2024-02-05T06:38:00.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0964999999999997e-05 2024-02-05T06:38:00.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:38:00.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.070325+0000", 2024-02-05T06:38:00.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013192299999999999 2024-02-05T06:38:00.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,6", 2024-02-05T06:38:00.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.070465+0000", 2024-02-05T06:38:00.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014058899999999999 2024-02-05T06:38:00.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:38:00.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.072338+0000", 2024-02-05T06:38:00.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001872425 2024-02-05T06:38:00.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.073514+0000", 2024-02-05T06:38:00.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0011764919999999999 2024-02-05T06:38:00.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:38:00.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.085312+0000", 2024-02-05T06:38:00.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011797446 2024-02-05T06:38:00.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:38:00.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.085371+0000", 2024-02-05T06:38:00.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9784000000000002e-05 2024-02-05T06:38:00.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:38:00.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:38:00.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:38:00.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.085419+0000", 2024-02-05T06:38:00.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7753000000000002e-05 2024-02-05T06:38:00.445 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.446 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:38:00.446 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.446 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:38:00.446 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:38:00.446 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:38:00.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:00.476 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:00.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:00.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:00.624 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:00.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:00.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:00.725 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:00.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:00 smithi167 ceph-mon[32774]: osdmap e143: 8 total, 8 up, 8 in 2024-02-05T06:38:00.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:00 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:38:00.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:00 smithi167 ceph-mon[32774]: osdmap e144: 8 total, 8 up, 8 in 2024-02-05T06:38:00.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:00 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2238434327' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:38:00.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:00 smithi167 ceph-mon[32774]: from='client.44879 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:38:00.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:00.757 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:00.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:00.858 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:00.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:00.887 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:00.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:00.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:00 smithi028 ceph-mon[37984]: osdmap e143: 8 total, 8 up, 8 in 2024-02-05T06:38:00.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:00 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:38:00.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:00 smithi028 ceph-mon[37984]: osdmap e144: 8 total, 8 up, 8 in 2024-02-05T06:38:00.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:00 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2238434327' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:38:00.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:00 smithi028 ceph-mon[37984]: from='client.44879 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:38:00.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:00 smithi028 ceph-mon[29442]: osdmap e143: 8 total, 8 up, 8 in 2024-02-05T06:38:00.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:00 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:38:00.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:00 smithi028 ceph-mon[29442]: osdmap e144: 8 total, 8 up, 8 in 2024-02-05T06:38:00.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:00 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2238434327' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:38:00.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:00 smithi028 ceph-mon[29442]: from='client.44879 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:38:00.987 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-02-05T06:38:00.988 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:01.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:01.031 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:01.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:01.131 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:01.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:01.195 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:01.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:01.296 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:01.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:01.329 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:01.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:01.429 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:01.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:01.459 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:01.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:01.560 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:01.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:01.590 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:01.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:01.691 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:01.695 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:01 smithi028 ceph-mon[37984]: pgmap v180: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 155 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 721 op/s; 5075/226005 objects misplaced (2.246%); 9.9 MiB/s, 158 objects/s recovering 2024-02-05T06:38:01.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:01 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:38:01.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:01 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:38:01.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:01 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:38:01.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:01 smithi028 ceph-mon[37984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:38:01.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:01 smithi028 ceph-mon[37984]: from='client.44879 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:38:01.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:01 smithi028 ceph-mon[37984]: osdmap e145: 8 total, 8 up, 8 in 2024-02-05T06:38:01.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:01 smithi028 ceph-mon[29442]: pgmap v180: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 155 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 721 op/s; 5075/226005 objects misplaced (2.246%); 9.9 MiB/s, 158 objects/s recovering 2024-02-05T06:38:01.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:01 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:38:01.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:01 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:38:01.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:01 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:38:01.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:01 smithi028 ceph-mon[29442]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:38:01.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:01 smithi028 ceph-mon[29442]: from='client.44879 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:38:01.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:01 smithi028 ceph-mon[29442]: osdmap e145: 8 total, 8 up, 8 in 2024-02-05T06:38:01.723 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:01.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:01.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:01.773 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-05T06:38:01.802 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:01 smithi167 ceph-mon[32774]: pgmap v180: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 155 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 721 op/s; 5075/226005 objects misplaced (2.246%); 9.9 MiB/s, 158 objects/s recovering 2024-02-05T06:38:01.803 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:01 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:38:01.803 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:01 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:38:01.803 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:01 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:38:01.803 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:01 smithi167 ceph-mon[32774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:38:01.803 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:01 smithi167 ceph-mon[32774]: from='client.44879 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:38:01.803 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:01 smithi167 ceph-mon[32774]: osdmap e145: 8 total, 8 up, 8 in 2024-02-05T06:38:01.824 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:01.857 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:01.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:01.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:01.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:01.995 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:01.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:01.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:02.097 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:02.108 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:38:02.138 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:02.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:02.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:02.240 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:02.267 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:02.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:02.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:02.369 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:02.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:02.402 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:02.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:02.503 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:02.532 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:02.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:02.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:02.634 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:02.663 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:02.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:02.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:02.721 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:38:02.762 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:38:02.762 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:02 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:38:02.762 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:02 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:38:02.762 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:02 smithi028 ceph-mon[37984]: osdmap e146: 8 total, 8 up, 8 in 2024-02-05T06:38:02.763 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:02 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:38:02.763 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:02 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:38:02.763 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:02 smithi028 ceph-mon[29442]: osdmap e146: 8 total, 8 up, 8 in 2024-02-05T06:38:02.763 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:38:02 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:38:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:38:02.764 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:02.792 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:02.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:02.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:02.815 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:38:02.859 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:38:02.892 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:02 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:38:02.892 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:02 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:38:02.892 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:02 smithi167 ceph-mon[32774]: osdmap e146: 8 total, 8 up, 8 in 2024-02-05T06:38:02.892 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:38:02 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:38:02.895 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:02.899 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:38:02.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:02.929 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:02.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:02.942 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:38:02.982 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:38:03.024 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:38:03.030 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:38:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:38:03.033 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:38:03.033 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:38:03.033 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:38:03.033 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:38:03.033 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:38:03.033 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:38:03.060 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:03.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:03.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:03.161 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:03.188 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:03.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:03.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:03.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:03.328 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:03.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:03.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:03.430 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:03.460 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:03.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:03.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:03.561 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:03.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:03.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:03.701 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:03.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:03.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:03.767 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T06:38:03.810 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:03 smithi028 ceph-mon[37984]: pgmap v183: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 959 op/s; 4486/215364 objects misplaced (2.083%); 5.7 MiB/s, 91 objects/s recovering 2024-02-05T06:38:03.810 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:03 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:38:03.810 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:03 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3728932577' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:38:03.810 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:03 smithi028 ceph-mon[29442]: pgmap v183: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 959 op/s; 4486/215364 objects misplaced (2.083%); 5.7 MiB/s, 91 objects/s recovering 2024-02-05T06:38:03.810 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:03 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:38:03.811 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:03 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3728932577' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:38:03.849 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:03.892 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:03.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:03.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:03.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:03 smithi167 ceph-mon[32774]: pgmap v183: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 959 op/s; 4486/215364 objects misplaced (2.083%); 5.7 MiB/s, 91 objects/s recovering 2024-02-05T06:38:03.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:03 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:38:03.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:03 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3728932577' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:38:03.995 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:04.022 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:04.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:04.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:04.124 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:04.184 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:04.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:04.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:04.223 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:38:04.286 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:04.344 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:04.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:04.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:04.413 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T06:38:04.413 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T06:38:04.414 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 1, 2, 3, 5, 6, 7, 4] 2024-02-05T06:38:04.414 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd out 7 2024-02-05T06:38:04.446 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:04.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:04.476 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:04.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:04.577 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:04.607 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:04.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:04.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:04.670 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:38:04.670 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:38:04.708 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:04.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:04.766 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:04.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:04.867 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:04.906 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:04.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:04.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:05.008 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:05.038 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:05.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:05.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:05.140 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:05.171 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:05.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:05.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:05.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:05.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:05.308 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:05.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:05.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:05.453 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:05.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:05.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:05.494 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-05T06:38:05.555 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:05.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:05.595 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:05.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:05.697 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:05.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:05.739 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:05.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:05.819 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:38:05.838 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:38:05.840 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:05.870 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:05.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:05.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:05.944 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:05 smithi167 ceph-mon[32774]: pgmap v184: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 756 op/s; 4486/215364 objects misplaced (2.083%); 4.5 MiB/s, 72 objects/s recovering 2024-02-05T06:38:05.944 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:05 smithi167 ceph-mon[32774]: osdmap e147: 8 total, 8 up, 8 in 2024-02-05T06:38:05.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:05 smithi028 ceph-mon[29442]: pgmap v184: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 756 op/s; 4486/215364 objects misplaced (2.083%); 4.5 MiB/s, 72 objects/s recovering 2024-02-05T06:38:05.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:05 smithi028 ceph-mon[29442]: osdmap e147: 8 total, 8 up, 8 in 2024-02-05T06:38:05.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:05 smithi028 ceph-mon[37984]: pgmap v184: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 756 op/s; 4486/215364 objects misplaced (2.083%); 4.5 MiB/s, 72 objects/s recovering 2024-02-05T06:38:05.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:05 smithi028 ceph-mon[37984]: osdmap e147: 8 total, 8 up, 8 in 2024-02-05T06:38:05.973 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:05.980 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:38:06.007 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:06.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:06.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:06.108 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:06.162 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:06.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:06.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:06.264 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:06.294 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:06.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:06.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:06.396 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:06.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:06.425 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:06.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:06.526 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:06.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:06.571 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:06.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:06.672 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:06.700 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:06.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:06.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:06.802 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:06.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:06.844 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:06.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:06.945 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:06.987 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:06.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:06.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:07.088 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:07.130 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:07.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:07.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:07.231 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:07.270 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:07.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:07.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:07.371 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:07.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:07.401 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:07.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:07.502 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:07.556 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:07.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:07.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:07.574 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-02-05T06:38:07.658 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:07.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:07.690 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:07.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:07.751 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:07 smithi167 ceph-mon[32774]: pgmap v186: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 920 op/s; 4486/209460 objects misplaced (2.142%); 3.5 MiB/s, 56 objects/s recovering 2024-02-05T06:38:07.752 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:07 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1212074036' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:38:07.791 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:07.827 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:07.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:07.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:07.929 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:07.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:07 smithi028 ceph-mon[29442]: pgmap v186: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 920 op/s; 4486/209460 objects misplaced (2.142%); 3.5 MiB/s, 56 objects/s recovering 2024-02-05T06:38:07.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:07 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1212074036' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:38:07.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:07 smithi028 ceph-mon[37984]: pgmap v186: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 920 op/s; 4486/209460 objects misplaced (2.142%); 3.5 MiB/s, 56 objects/s recovering 2024-02-05T06:38:07.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:07 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1212074036' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:38:07.964 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:07.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:07.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:08.066 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:08.105 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:08.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:08.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:08.207 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:08.241 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:08.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:08.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:08.343 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:08.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:08.380 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:08.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:08.481 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:08.512 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:08.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:08.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:08.614 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:08.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:08.645 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:08.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:08.746 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:08.771 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:08.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:08.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:08.847 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:08 smithi167 ceph-mon[32774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:38:08.847 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:08 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1212074036' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:38:08.847 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:08 smithi167 ceph-mon[32774]: osdmap e148: 8 total, 8 up, 8 in 2024-02-05T06:38:08.873 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:08.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:08.908 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:08.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:08.924 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:08 smithi028 ceph-mon[29442]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:38:08.924 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:08 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1212074036' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:38:08.924 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:08 smithi028 ceph-mon[29442]: osdmap e148: 8 total, 8 up, 8 in 2024-02-05T06:38:08.924 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:08 smithi028 ceph-mon[37984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:38:08.925 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:08 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1212074036' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:38:08.925 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:08 smithi028 ceph-mon[37984]: osdmap e148: 8 total, 8 up, 8 in 2024-02-05T06:38:09.009 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:09.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:09.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:09.150 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:09.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:09.192 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:09.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:09.293 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:09.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:09.328 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:09.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:09.429 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:09.458 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:09.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:09.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:09.560 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:09.592 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:09.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:09.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:09.694 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:09.732 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:09.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:09.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:09.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:09.874 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:09.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:09.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:09.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:09 smithi028 ceph-mon[29442]: pgmap v188: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 819 B/s rd, 0 B/s wr, 859 op/s; 4150/199278 objects misplaced (2.083%) 2024-02-05T06:38:09.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:09 smithi028 ceph-mon[29442]: 3.4 scrub starts 2024-02-05T06:38:09.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:09 smithi028 ceph-mon[29442]: 3.4 scrub ok 2024-02-05T06:38:09.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:09 smithi028 ceph-mon[29442]: 3.2 scrub starts 2024-02-05T06:38:09.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:09 smithi028 ceph-mon[29442]: 3.2 scrub ok 2024-02-05T06:38:09.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:09 smithi028 ceph-mon[37984]: pgmap v188: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 819 B/s rd, 0 B/s wr, 859 op/s; 4150/199278 objects misplaced (2.083%) 2024-02-05T06:38:09.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:09 smithi028 ceph-mon[37984]: 3.4 scrub starts 2024-02-05T06:38:09.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:09 smithi028 ceph-mon[37984]: 3.4 scrub ok 2024-02-05T06:38:09.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:09 smithi028 ceph-mon[37984]: 3.2 scrub starts 2024-02-05T06:38:09.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:09 smithi028 ceph-mon[37984]: 3.2 scrub ok 2024-02-05T06:38:09.976 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:09.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:09 smithi167 ceph-mon[32774]: pgmap v188: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 819 B/s rd, 0 B/s wr, 859 op/s; 4150/199278 objects misplaced (2.083%) 2024-02-05T06:38:09.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:09 smithi167 ceph-mon[32774]: 3.4 scrub starts 2024-02-05T06:38:09.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:09 smithi167 ceph-mon[32774]: 3.4 scrub ok 2024-02-05T06:38:09.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:09 smithi167 ceph-mon[32774]: 3.2 scrub starts 2024-02-05T06:38:09.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:09 smithi167 ceph-mon[32774]: 3.2 scrub ok 2024-02-05T06:38:09.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:10.000 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:10.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:10.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:10.140 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:38:10.140 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:38:10.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:10.142 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:10.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:10.243 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:10.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:10.272 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:10.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:10.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:10.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:10.398 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:10.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:10.499 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:10.522 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:10.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:10.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:10.624 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:10.654 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:10.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:10.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:10.756 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:10.797 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:10.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:10.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:10.898 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:10.933 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:10.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:10.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:10.945 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:10 smithi028 ceph-mon[29442]: 7.5 scrub starts 2024-02-05T06:38:10.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:10 smithi028 ceph-mon[29442]: 7.0 scrub starts 2024-02-05T06:38:10.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:10 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:38:10.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:10 smithi028 ceph-mon[37984]: 7.5 scrub starts 2024-02-05T06:38:10.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:10 smithi028 ceph-mon[37984]: 7.0 scrub starts 2024-02-05T06:38:10.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:10 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:38:10.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:10 smithi167 ceph-mon[32774]: 7.5 scrub starts 2024-02-05T06:38:10.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:10 smithi167 ceph-mon[32774]: 7.0 scrub starts 2024-02-05T06:38:10.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:10 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:38:11.035 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:11.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:11.069 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:11.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:11.171 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:11.207 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:11.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:11.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:11.309 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:11.346 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:11.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:11.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:11.447 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:11.469 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-05T06:38:11.487 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:11.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:11.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:11.589 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:11.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:11.622 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:11.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:11.723 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:11.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:11 smithi167 ceph-mon[32774]: 1.5 scrub starts 2024-02-05T06:38:11.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:11 smithi167 ceph-mon[32774]: 1.5 scrub ok 2024-02-05T06:38:11.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:11 smithi167 ceph-mon[32774]: 1.4 scrub starts 2024-02-05T06:38:11.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:11 smithi167 ceph-mon[32774]: 1.4 scrub ok 2024-02-05T06:38:11.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:11 smithi167 ceph-mon[32774]: pgmap v189: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 670 op/s; 4150/199278 objects misplaced (2.083%) 2024-02-05T06:38:11.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:11 smithi167 ceph-mon[32774]: 7.5 scrub ok 2024-02-05T06:38:11.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:11 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.9", "id": [4, 1]}]: dispatch 2024-02-05T06:38:11.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:11 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.9", "id": [4, 1]}]: dispatch 2024-02-05T06:38:11.763 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:11.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:11.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:11.818 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:38:11.865 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:11.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:11.892 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:11.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:11.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:11 smithi028 ceph-mon[29442]: 1.5 scrub starts 2024-02-05T06:38:11.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:11 smithi028 ceph-mon[29442]: 1.5 scrub ok 2024-02-05T06:38:11.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:11 smithi028 ceph-mon[29442]: 1.4 scrub starts 2024-02-05T06:38:11.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:11 smithi028 ceph-mon[29442]: 1.4 scrub ok 2024-02-05T06:38:11.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:11 smithi028 ceph-mon[29442]: pgmap v189: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 670 op/s; 4150/199278 objects misplaced (2.083%) 2024-02-05T06:38:11.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:11 smithi028 ceph-mon[29442]: 7.5 scrub ok 2024-02-05T06:38:11.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:11 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.9", "id": [4, 1]}]: dispatch 2024-02-05T06:38:11.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:11 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.9", "id": [4, 1]}]: dispatch 2024-02-05T06:38:11.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:11 smithi028 ceph-mon[37984]: 1.5 scrub starts 2024-02-05T06:38:11.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:11 smithi028 ceph-mon[37984]: 1.5 scrub ok 2024-02-05T06:38:11.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:11 smithi028 ceph-mon[37984]: 1.4 scrub starts 2024-02-05T06:38:11.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:11 smithi028 ceph-mon[37984]: 1.4 scrub ok 2024-02-05T06:38:11.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:11 smithi028 ceph-mon[37984]: pgmap v189: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 670 op/s; 4150/199278 objects misplaced (2.083%) 2024-02-05T06:38:11.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:11 smithi028 ceph-mon[37984]: 7.5 scrub ok 2024-02-05T06:38:11.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:11 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.9", "id": [4, 1]}]: dispatch 2024-02-05T06:38:11.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:11 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.9", "id": [4, 1]}]: dispatch 2024-02-05T06:38:11.993 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:12.021 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:12.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:12.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:12.123 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:12.147 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:12.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:12.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:12.249 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:12.275 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:12.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:12.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:12.377 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:12.404 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:12.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:12.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:12.506 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:12.536 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:12.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:12.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:12.638 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:12.673 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:12.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:12.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:12.745 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:38:12 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:38:12.745 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:12 smithi167 ceph-mon[32774]: 7.0 scrub ok 2024-02-05T06:38:12.746 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:12 smithi167 ceph-mon[32774]: 5.9 scrub starts 2024-02-05T06:38:12.746 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:12 smithi167 ceph-mon[32774]: 5.9 scrub ok 2024-02-05T06:38:12.746 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:12 smithi167 ceph-mon[32774]: 5.b scrub starts 2024-02-05T06:38:12.746 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:12 smithi167 ceph-mon[32774]: 5.b scrub ok 2024-02-05T06:38:12.746 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:12 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.9", "id": [4, 1]}]': finished 2024-02-05T06:38:12.746 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:12 smithi167 ceph-mon[32774]: osdmap e149: 8 total, 8 up, 8 in 2024-02-05T06:38:12.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:12 smithi028 ceph-mon[29442]: 7.0 scrub ok 2024-02-05T06:38:12.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:12 smithi028 ceph-mon[29442]: 5.9 scrub starts 2024-02-05T06:38:12.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:12 smithi028 ceph-mon[29442]: 5.9 scrub ok 2024-02-05T06:38:12.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:12 smithi028 ceph-mon[29442]: 5.b scrub starts 2024-02-05T06:38:12.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:12 smithi028 ceph-mon[29442]: 5.b scrub ok 2024-02-05T06:38:12.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:12 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.9", "id": [4, 1]}]': finished 2024-02-05T06:38:12.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:12 smithi028 ceph-mon[29442]: osdmap e149: 8 total, 8 up, 8 in 2024-02-05T06:38:12.757 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:38:12 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:38:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:38:12.758 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:12 smithi028 ceph-mon[37984]: 7.0 scrub ok 2024-02-05T06:38:12.758 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:12 smithi028 ceph-mon[37984]: 5.9 scrub starts 2024-02-05T06:38:12.758 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:12 smithi028 ceph-mon[37984]: 5.9 scrub ok 2024-02-05T06:38:12.758 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:12 smithi028 ceph-mon[37984]: 5.b scrub starts 2024-02-05T06:38:12.758 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:12 smithi028 ceph-mon[37984]: 5.b scrub ok 2024-02-05T06:38:12.758 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:12 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.9", "id": [4, 1]}]': finished 2024-02-05T06:38:12.759 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:12 smithi028 ceph-mon[37984]: osdmap e149: 8 total, 8 up, 8 in 2024-02-05T06:38:12.775 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:12.848 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:38:12.850 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:12.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:12.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:12.953 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:12.982 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:12.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:12.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:13.084 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:13.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:13.116 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:13.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:13.217 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:13.245 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:13.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:13.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:13.347 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:13.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:13.377 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:13.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:13.478 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:13.521 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:13.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:13.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:13.623 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:13.650 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:13.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:13.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:13.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:13 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:38:13.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:13 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:38:13.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:13 smithi028 ceph-mon[29442]: 5.1 scrub starts 2024-02-05T06:38:13.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:13 smithi028 ceph-mon[29442]: 5.1 scrub ok 2024-02-05T06:38:13.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:13 smithi028 ceph-mon[29442]: pgmap v191: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 156 active+clean; 3.7 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.42k op/s; 3848/184275 objects misplaced (2.088%) 2024-02-05T06:38:13.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:13 smithi028 ceph-mon[29442]: 4.7 scrub starts 2024-02-05T06:38:13.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:13 smithi028 ceph-mon[29442]: 4.7 scrub ok 2024-02-05T06:38:13.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:13 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:38:13.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:13 smithi028 ceph-mon[29442]: osdmap e150: 8 total, 8 up, 8 in 2024-02-05T06:38:13.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:13 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:38:13.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:13 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:38:13.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:13 smithi028 ceph-mon[37984]: 5.1 scrub starts 2024-02-05T06:38:13.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:13 smithi028 ceph-mon[37984]: 5.1 scrub ok 2024-02-05T06:38:13.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:13 smithi028 ceph-mon[37984]: pgmap v191: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 156 active+clean; 3.7 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.42k op/s; 3848/184275 objects misplaced (2.088%) 2024-02-05T06:38:13.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:13 smithi028 ceph-mon[37984]: 4.7 scrub starts 2024-02-05T06:38:13.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:13 smithi028 ceph-mon[37984]: 4.7 scrub ok 2024-02-05T06:38:13.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:13 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:38:13.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:13 smithi028 ceph-mon[37984]: osdmap e150: 8 total, 8 up, 8 in 2024-02-05T06:38:13.752 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:13.784 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:13.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:13.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:13.851 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:13 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:38:13.851 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:13 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:38:13.851 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:13 smithi167 ceph-mon[32774]: 5.1 scrub starts 2024-02-05T06:38:13.851 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:13 smithi167 ceph-mon[32774]: 5.1 scrub ok 2024-02-05T06:38:13.852 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:13 smithi167 ceph-mon[32774]: pgmap v191: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 156 active+clean; 3.7 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 1.42k op/s; 3848/184275 objects misplaced (2.088%) 2024-02-05T06:38:13.852 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:13 smithi167 ceph-mon[32774]: 4.7 scrub starts 2024-02-05T06:38:13.852 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:13 smithi167 ceph-mon[32774]: 4.7 scrub ok 2024-02-05T06:38:13.852 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:13 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:38:13.852 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:13 smithi167 ceph-mon[32774]: osdmap e150: 8 total, 8 up, 8 in 2024-02-05T06:38:13.886 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:13.915 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:13.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:13.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:14.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:14.060 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:14.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:14.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:14.162 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:14.212 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:14.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:14.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:14.314 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:14.339 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:38:14.339 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:38:14.366 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:14.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:14.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:14.468 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:14.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:14.507 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:14.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:14.584 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T06:38:14.608 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:14.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:14.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:14.791 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:14.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:14.855 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:14.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:14.898 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:14 smithi167 ceph-mon[32774]: 3.11 scrub starts 2024-02-05T06:38:14.898 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:14 smithi167 ceph-mon[32774]: 3.11 scrub ok 2024-02-05T06:38:14.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:14 smithi028 ceph-mon[37984]: 3.11 scrub starts 2024-02-05T06:38:14.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:14 smithi028 ceph-mon[37984]: 3.11 scrub ok 2024-02-05T06:38:14.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:14 smithi028 ceph-mon[29442]: 3.11 scrub starts 2024-02-05T06:38:14.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:14 smithi028 ceph-mon[29442]: 3.11 scrub ok 2024-02-05T06:38:14.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:14 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-a[29438]: 2024-02-05T06:38:14.681+0000 7fa733293700 -1 mon.a@0(leader).osd e150 definitely_dead 0 2024-02-05T06:38:14.956 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:14.987 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:14.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:14.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:15.090 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:15.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:15.138 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:15.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:15.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:15.278 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:15.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:15.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:15.380 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:15.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:15.426 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:15.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:15.497 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-05T06:38:15.505 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:38:15.527 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:15.547 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:38:15.567 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:15.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:15.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:15.597 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:38:15.606 INFO:teuthology.orchestra.run.smithi028.stderr:marked out osd.7. 2024-02-05T06:38:15.653 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:38:15.668 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:15.678 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:38:15.693 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:15.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:15.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:15.701 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:38:15.721 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:38:15.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:15 smithi167 ceph-mon[32774]: 4.1a scrub starts 2024-02-05T06:38:15.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:15 smithi167 ceph-mon[32774]: 4.1a scrub ok 2024-02-05T06:38:15.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:15 smithi167 ceph-mon[32774]: pgmap v193: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 156 active+clean; 3.7 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 744 op/s; 3848/184275 objects misplaced (2.088%) 2024-02-05T06:38:15.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:15 smithi167 ceph-mon[32774]: 3.9 scrub starts 2024-02-05T06:38:15.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:15 smithi167 ceph-mon[32774]: 3.9 scrub ok 2024-02-05T06:38:15.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:15 smithi167 ceph-mon[32774]: 2.0 scrub starts 2024-02-05T06:38:15.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:15 smithi167 ceph-mon[32774]: 2.0 scrub ok 2024-02-05T06:38:15.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:15 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1088009351' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-02-05T06:38:15.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:15 smithi167 ceph-mon[32774]: from='client.24898 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-02-05T06:38:15.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:15 smithi167 ceph-mon[32774]: Client client.admin marked osd.7 out, while it was still marked up 2024-02-05T06:38:15.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:15 smithi167 ceph-mon[32774]: 7.6 scrub starts 2024-02-05T06:38:15.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:15 smithi167 ceph-mon[32774]: 5.1b scrub starts 2024-02-05T06:38:15.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:15 smithi167 ceph-mon[32774]: 5.1b scrub ok 2024-02-05T06:38:15.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:15 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/994690174' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:38:15.741 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:38:15.762 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:38:15.764 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:38:15.764 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:38:15.764 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:38:15.764 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:38:15.764 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:38:15.764 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:38:15.764 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:38:15.764 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:38:15.777 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:38:15.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:15.799 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:15 smithi028 ceph-mon[29442]: 4.1a scrub starts 2024-02-05T06:38:15.799 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:15 smithi028 ceph-mon[29442]: 4.1a scrub ok 2024-02-05T06:38:15.799 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:15 smithi028 ceph-mon[29442]: pgmap v193: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 156 active+clean; 3.7 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 744 op/s; 3848/184275 objects misplaced (2.088%) 2024-02-05T06:38:15.799 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:15 smithi028 ceph-mon[29442]: 3.9 scrub starts 2024-02-05T06:38:15.799 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:15 smithi028 ceph-mon[29442]: 3.9 scrub ok 2024-02-05T06:38:15.800 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:15 smithi028 ceph-mon[29442]: 2.0 scrub starts 2024-02-05T06:38:15.800 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:15 smithi028 ceph-mon[29442]: 2.0 scrub ok 2024-02-05T06:38:15.800 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:15 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1088009351' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-02-05T06:38:15.800 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:15 smithi028 ceph-mon[29442]: from='client.24898 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-02-05T06:38:15.800 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:15 smithi028 ceph-mon[29442]: Client client.admin marked osd.7 out, while it was still marked up 2024-02-05T06:38:15.800 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:15 smithi028 ceph-mon[29442]: 7.6 scrub starts 2024-02-05T06:38:15.800 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:15 smithi028 ceph-mon[29442]: 5.1b scrub starts 2024-02-05T06:38:15.800 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:15 smithi028 ceph-mon[29442]: 5.1b scrub ok 2024-02-05T06:38:15.800 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:15 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/994690174' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:38:15.800 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:15 smithi028 ceph-mon[37984]: 4.1a scrub starts 2024-02-05T06:38:15.801 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:15 smithi028 ceph-mon[37984]: 4.1a scrub ok 2024-02-05T06:38:15.801 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:15 smithi028 ceph-mon[37984]: pgmap v193: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+clean+scrubbing, 156 active+clean; 3.7 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 0 B/s wr, 744 op/s; 3848/184275 objects misplaced (2.088%) 2024-02-05T06:38:15.801 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:15 smithi028 ceph-mon[37984]: 3.9 scrub starts 2024-02-05T06:38:15.801 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:15 smithi028 ceph-mon[37984]: 3.9 scrub ok 2024-02-05T06:38:15.801 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:15 smithi028 ceph-mon[37984]: 2.0 scrub starts 2024-02-05T06:38:15.801 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:15 smithi028 ceph-mon[37984]: 2.0 scrub ok 2024-02-05T06:38:15.801 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:15 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1088009351' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-02-05T06:38:15.801 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:15 smithi028 ceph-mon[37984]: from='client.24898 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-02-05T06:38:15.801 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:15 smithi028 ceph-mon[37984]: Client client.admin marked osd.7 out, while it was still marked up 2024-02-05T06:38:15.801 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:15 smithi028 ceph-mon[37984]: 7.6 scrub starts 2024-02-05T06:38:15.801 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:15 smithi028 ceph-mon[37984]: 5.1b scrub starts 2024-02-05T06:38:15.801 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:15 smithi028 ceph-mon[37984]: 5.1b scrub ok 2024-02-05T06:38:15.801 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:15 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/994690174' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:38:15.822 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:15.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:15.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:15.924 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:15.948 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:15.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:15.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:16.050 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:16.075 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:16.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:16.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:16.177 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:16.200 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:16.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:16.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:16.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:16.326 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:16.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:16.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:16.427 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:16.451 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:16.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:16.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:16.553 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:16.577 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:16.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:16.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:16.679 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:16.702 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:16.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:16.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:16.804 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:16.827 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:16.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:16.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:16.916 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:16 smithi167 ceph-mon[32774]: 6.13 scrub starts 2024-02-05T06:38:16.916 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:16 smithi167 ceph-mon[32774]: 6.13 scrub ok 2024-02-05T06:38:16.916 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:16 smithi167 ceph-mon[32774]: 5.12 scrub starts 2024-02-05T06:38:16.916 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:16 smithi167 ceph-mon[32774]: 5.12 scrub ok 2024-02-05T06:38:16.916 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:16 smithi167 ceph-mon[32774]: from='client.24898 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-02-05T06:38:16.916 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:16 smithi167 ceph-mon[32774]: 3.e scrub starts 2024-02-05T06:38:16.916 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:16 smithi167 ceph-mon[32774]: 3.e scrub ok 2024-02-05T06:38:16.917 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:16 smithi167 ceph-mon[32774]: osdmap e151: 8 total, 8 up, 7 in 2024-02-05T06:38:16.917 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:16 smithi167 ceph-mon[32774]: 7.4 scrub starts 2024-02-05T06:38:16.917 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:16 smithi167 ceph-mon[32774]: 7.4 scrub ok 2024-02-05T06:38:16.917 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:16 smithi167 ceph-mon[32774]: 4.1d scrub starts 2024-02-05T06:38:16.917 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:16 smithi167 ceph-mon[32774]: 4.1d scrub ok 2024-02-05T06:38:16.917 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:16 smithi167 ceph-mon[32774]: osdmap e152: 8 total, 8 up, 7 in 2024-02-05T06:38:16.917 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:16 smithi167 ceph-mon[32774]: Health check failed: Reduced data availability: 10 pgs inactive, 12 pgs peering (PG_AVAILABILITY) 2024-02-05T06:38:16.921 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:16 smithi028 ceph-mon[37984]: 6.13 scrub starts 2024-02-05T06:38:16.921 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:16 smithi028 ceph-mon[37984]: 6.13 scrub ok 2024-02-05T06:38:16.921 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:16 smithi028 ceph-mon[37984]: 5.12 scrub starts 2024-02-05T06:38:16.922 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:16 smithi028 ceph-mon[37984]: 5.12 scrub ok 2024-02-05T06:38:16.922 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:16 smithi028 ceph-mon[37984]: from='client.24898 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-02-05T06:38:16.922 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:16 smithi028 ceph-mon[37984]: 3.e scrub starts 2024-02-05T06:38:16.922 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:16 smithi028 ceph-mon[37984]: 3.e scrub ok 2024-02-05T06:38:16.922 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:16 smithi028 ceph-mon[37984]: osdmap e151: 8 total, 8 up, 7 in 2024-02-05T06:38:16.922 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:16 smithi028 ceph-mon[37984]: 7.4 scrub starts 2024-02-05T06:38:16.922 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:16 smithi028 ceph-mon[37984]: 7.4 scrub ok 2024-02-05T06:38:16.922 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:16 smithi028 ceph-mon[37984]: 4.1d scrub starts 2024-02-05T06:38:16.922 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:16 smithi028 ceph-mon[37984]: 4.1d scrub ok 2024-02-05T06:38:16.922 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:16 smithi028 ceph-mon[37984]: osdmap e152: 8 total, 8 up, 7 in 2024-02-05T06:38:16.922 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:16 smithi028 ceph-mon[37984]: Health check failed: Reduced data availability: 10 pgs inactive, 12 pgs peering (PG_AVAILABILITY) 2024-02-05T06:38:16.922 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:16 smithi028 ceph-mon[29442]: 6.13 scrub starts 2024-02-05T06:38:16.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:16 smithi028 ceph-mon[29442]: 6.13 scrub ok 2024-02-05T06:38:16.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:16 smithi028 ceph-mon[29442]: 5.12 scrub starts 2024-02-05T06:38:16.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:16 smithi028 ceph-mon[29442]: 5.12 scrub ok 2024-02-05T06:38:16.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:16 smithi028 ceph-mon[29442]: from='client.24898 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-02-05T06:38:16.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:16 smithi028 ceph-mon[29442]: 3.e scrub starts 2024-02-05T06:38:16.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:16 smithi028 ceph-mon[29442]: 3.e scrub ok 2024-02-05T06:38:16.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:16 smithi028 ceph-mon[29442]: osdmap e151: 8 total, 8 up, 7 in 2024-02-05T06:38:16.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:16 smithi028 ceph-mon[29442]: 7.4 scrub starts 2024-02-05T06:38:16.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:16 smithi028 ceph-mon[29442]: 7.4 scrub ok 2024-02-05T06:38:16.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:16 smithi028 ceph-mon[29442]: 4.1d scrub starts 2024-02-05T06:38:16.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:16 smithi028 ceph-mon[29442]: 4.1d scrub ok 2024-02-05T06:38:16.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:16 smithi028 ceph-mon[29442]: osdmap e152: 8 total, 8 up, 7 in 2024-02-05T06:38:16.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:16 smithi028 ceph-mon[29442]: Health check failed: Reduced data availability: 10 pgs inactive, 12 pgs peering (PG_AVAILABILITY) 2024-02-05T06:38:16.929 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:16.953 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:16.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:16.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:17.055 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:17.079 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:17.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:17.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:17.181 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:17.205 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:17.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:17.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:17.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:17.329 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:17.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:17.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:17.431 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:17.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:17.456 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:17.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:17.503 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:38:17.503 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-02-05T06:38:17.503 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-02-05T06:38:17.503 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:38:17.558 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:17.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:17.582 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:17.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:17.683 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:17.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:17.713 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:17.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:17.814 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:17.843 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:17.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:17.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:17.945 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:17.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:17 smithi028 ceph-mon[37984]: pgmap v195: 158 pgs: 25 peering, 1 active+remapped+backfill_toofull, 132 active+clean; 3.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s; 3848/180288 objects misplaced (2.134%) 2024-02-05T06:38:17.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:17 smithi028 ceph-mon[37984]: 6.15 scrub starts 2024-02-05T06:38:17.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:17 smithi028 ceph-mon[37984]: 6.15 scrub ok 2024-02-05T06:38:17.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:17 smithi028 ceph-mon[37984]: 6.7 scrub starts 2024-02-05T06:38:17.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:17 smithi028 ceph-mon[37984]: 6.7 scrub ok 2024-02-05T06:38:17.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:17 smithi028 ceph-mon[37984]: osdmap e153: 8 total, 8 up, 7 in 2024-02-05T06:38:17.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:17 smithi028 ceph-mon[29442]: pgmap v195: 158 pgs: 25 peering, 1 active+remapped+backfill_toofull, 132 active+clean; 3.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s; 3848/180288 objects misplaced (2.134%) 2024-02-05T06:38:17.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:17 smithi028 ceph-mon[29442]: 6.15 scrub starts 2024-02-05T06:38:17.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:17 smithi028 ceph-mon[29442]: 6.15 scrub ok 2024-02-05T06:38:17.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:17 smithi028 ceph-mon[29442]: 6.7 scrub starts 2024-02-05T06:38:17.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:17 smithi028 ceph-mon[29442]: 6.7 scrub ok 2024-02-05T06:38:17.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:17 smithi028 ceph-mon[29442]: osdmap e153: 8 total, 8 up, 7 in 2024-02-05T06:38:17.976 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:17.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:17.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:17.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:17 smithi167 ceph-mon[32774]: pgmap v195: 158 pgs: 25 peering, 1 active+remapped+backfill_toofull, 132 active+clean; 3.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 1.05k op/s; 3848/180288 objects misplaced (2.134%) 2024-02-05T06:38:17.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:17 smithi167 ceph-mon[32774]: 6.15 scrub starts 2024-02-05T06:38:17.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:17 smithi167 ceph-mon[32774]: 6.15 scrub ok 2024-02-05T06:38:17.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:17 smithi167 ceph-mon[32774]: 6.7 scrub starts 2024-02-05T06:38:17.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:17 smithi167 ceph-mon[32774]: 6.7 scrub ok 2024-02-05T06:38:17.980 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:17 smithi167 ceph-mon[32774]: osdmap e153: 8 total, 8 up, 7 in 2024-02-05T06:38:18.078 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:18.108 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:18.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:18.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:18.209 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:18.238 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:18.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:18.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:18.340 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:18.371 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-05T06:38:18.373 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:18.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:18.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:18.475 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:18.504 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:18.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:18.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:18.606 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:18.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:18.643 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:18.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:18.711 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:38:18.718 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:18 smithi028 ceph-mon[37984]: 6.f scrub starts 2024-02-05T06:38:18.718 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:18 smithi028 ceph-mon[37984]: 6.f scrub ok 2024-02-05T06:38:18.719 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:18 smithi028 ceph-mon[37984]: 7.f starting backfill to osd.4 from (0'0,0'0] MAX to 150'10851 2024-02-05T06:38:18.719 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:18 smithi028 ceph-mon[37984]: 7.c starting backfill to osd.2 from (0'0,0'0] MAX to 150'10692 2024-02-05T06:38:18.719 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:18 smithi028 ceph-mon[37984]: 7.6 starting backfill to osd.5 from (0'0,0'0] MAX to 150'10546 2024-02-05T06:38:18.719 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:18 smithi028 ceph-mon[37984]: 7.8 continuing backfill to osd.4 from (115'4968,150'10935] MIN to 150'10935 2024-02-05T06:38:18.719 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:18 smithi028 ceph-mon[37984]: 7.5 starting backfill to osd.1 from (0'0,0'0] MAX to 150'14634 2024-02-05T06:38:18.719 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:18 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:38:18.719 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:18 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:38:18.719 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:18 smithi028 ceph-mon[37984]: osdmap e154: 8 total, 8 up, 7 in 2024-02-05T06:38:18.719 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:18 smithi028 ceph-mon[29442]: 6.f scrub starts 2024-02-05T06:38:18.719 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:18 smithi028 ceph-mon[29442]: 6.f scrub ok 2024-02-05T06:38:18.719 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:18 smithi028 ceph-mon[29442]: 7.f starting backfill to osd.4 from (0'0,0'0] MAX to 150'10851 2024-02-05T06:38:18.719 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:18 smithi028 ceph-mon[29442]: 7.c starting backfill to osd.2 from (0'0,0'0] MAX to 150'10692 2024-02-05T06:38:18.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:18 smithi028 ceph-mon[29442]: 7.6 starting backfill to osd.5 from (0'0,0'0] MAX to 150'10546 2024-02-05T06:38:18.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:18 smithi028 ceph-mon[29442]: 7.8 continuing backfill to osd.4 from (115'4968,150'10935] MIN to 150'10935 2024-02-05T06:38:18.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:18 smithi028 ceph-mon[29442]: 7.5 starting backfill to osd.1 from (0'0,0'0] MAX to 150'14634 2024-02-05T06:38:18.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:18 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:38:18.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:18 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:38:18.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:18 smithi028 ceph-mon[29442]: osdmap e154: 8 total, 8 up, 7 in 2024-02-05T06:38:18.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:18 smithi167 ceph-mon[32774]: 6.f scrub starts 2024-02-05T06:38:18.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:18 smithi167 ceph-mon[32774]: 6.f scrub ok 2024-02-05T06:38:18.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:18 smithi167 ceph-mon[32774]: 7.f starting backfill to osd.4 from (0'0,0'0] MAX to 150'10851 2024-02-05T06:38:18.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:18 smithi167 ceph-mon[32774]: 7.c starting backfill to osd.2 from (0'0,0'0] MAX to 150'10692 2024-02-05T06:38:18.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:18 smithi167 ceph-mon[32774]: 7.6 starting backfill to osd.5 from (0'0,0'0] MAX to 150'10546 2024-02-05T06:38:18.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:18 smithi167 ceph-mon[32774]: 7.8 continuing backfill to osd.4 from (115'4968,150'10935] MIN to 150'10935 2024-02-05T06:38:18.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:18 smithi167 ceph-mon[32774]: 7.5 starting backfill to osd.1 from (0'0,0'0] MAX to 150'14634 2024-02-05T06:38:18.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:18 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:38:18.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:18 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:38:18.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:18 smithi167 ceph-mon[32774]: osdmap e154: 8 total, 8 up, 7 in 2024-02-05T06:38:18.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:18.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:18.792 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:18.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:18.893 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:18.923 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:18.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:18.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:18.949 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:38:19.024 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:19.053 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:19.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:19.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:19.155 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:19.182 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:19.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:19.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:19.284 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:19.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:19.343 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:19.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:19.414 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:38:19.444 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:19.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:19.514 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:19.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:19.597 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-02-05T06:38:19.615 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:19.644 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:19.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:19.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:19.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:19 smithi028 ceph-mon[29442]: pgmap v198: 158 pgs: 1 remapped+peering, 4 unknown, 57 peering, 96 active+clean; 3.5 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 125 KiB/s wr, 776 op/s; 90 KiB/s, 0 objects/s recovering 2024-02-05T06:38:19.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:19 smithi028 ceph-mon[29442]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-05T06:38:19.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:19 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1027591071' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:38:19.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:19 smithi028 ceph-mon[29442]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:38:19.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:19 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1027591071' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:38:19.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:19 smithi028 ceph-mon[29442]: osdmap e155: 8 total, 8 up, 7 in 2024-02-05T06:38:19.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:19 smithi028 ceph-mon[37984]: pgmap v198: 158 pgs: 1 remapped+peering, 4 unknown, 57 peering, 96 active+clean; 3.5 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 125 KiB/s wr, 776 op/s; 90 KiB/s, 0 objects/s recovering 2024-02-05T06:38:19.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:19 smithi028 ceph-mon[37984]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-05T06:38:19.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:19 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1027591071' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:38:19.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:19 smithi028 ceph-mon[37984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:38:19.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:19 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1027591071' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:38:19.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:19 smithi028 ceph-mon[37984]: osdmap e155: 8 total, 8 up, 7 in 2024-02-05T06:38:19.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:19.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:19.794 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:19.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:19.895 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:19.898 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:19 smithi167 ceph-mon[32774]: pgmap v198: 158 pgs: 1 remapped+peering, 4 unknown, 57 peering, 96 active+clean; 3.5 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 125 KiB/s wr, 776 op/s; 90 KiB/s, 0 objects/s recovering 2024-02-05T06:38:19.898 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:19 smithi167 ceph-mon[32774]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-05T06:38:19.898 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:19 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1027591071' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:38:19.899 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:19 smithi167 ceph-mon[32774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:38:19.899 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:19 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1027591071' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:38:19.899 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:19 smithi167 ceph-mon[32774]: osdmap e155: 8 total, 8 up, 7 in 2024-02-05T06:38:19.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:19.932 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:19.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:20.033 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:20.069 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:20.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:20.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:20.171 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:20.202 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:20.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:20.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:20.303 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:20.338 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:20.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:20.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:20.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:20.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:20.482 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:20.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:20.583 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:20.616 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:20.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:20.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:20.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:20.734 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:38:20.734 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-02-05T06:38:20.734 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-02-05T06:38:20.734 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-02-05T06:38:20.734 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:38:20.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:20.764 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:20.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:20.865 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:20.904 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:20.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:20.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:21.006 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:21.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:21.036 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:21.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:21.137 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:21.170 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:20 smithi028 ceph-mon[37984]: osdmap e156: 8 total, 8 up, 7 in 2024-02-05T06:38:21.170 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:20 smithi028 ceph-mon[37984]: 1.9 deep-scrub starts 2024-02-05T06:38:21.170 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:20 smithi028 ceph-mon[37984]: 1.9 deep-scrub ok 2024-02-05T06:38:21.170 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:20 smithi028 ceph-mon[37984]: pgmap v202: 158 pgs: 1 remapped+peering, 4 unknown, 57 peering, 96 active+clean; 3.5 GiB data, 10 GiB used, 615 GiB / 626 GiB avail 2024-02-05T06:38:21.170 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:20 smithi028 ceph-mon[29442]: osdmap e156: 8 total, 8 up, 7 in 2024-02-05T06:38:21.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:20 smithi028 ceph-mon[29442]: 1.9 deep-scrub starts 2024-02-05T06:38:21.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:20 smithi028 ceph-mon[29442]: 1.9 deep-scrub ok 2024-02-05T06:38:21.171 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:20 smithi028 ceph-mon[29442]: pgmap v202: 158 pgs: 1 remapped+peering, 4 unknown, 57 peering, 96 active+clean; 3.5 GiB data, 10 GiB used, 615 GiB / 626 GiB avail 2024-02-05T06:38:21.177 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:21.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:21.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:21.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:20 smithi167 ceph-mon[32774]: osdmap e156: 8 total, 8 up, 7 in 2024-02-05T06:38:21.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:20 smithi167 ceph-mon[32774]: 1.9 deep-scrub starts 2024-02-05T06:38:21.231 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:20 smithi167 ceph-mon[32774]: 1.9 deep-scrub ok 2024-02-05T06:38:21.231 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:20 smithi167 ceph-mon[32774]: pgmap v202: 158 pgs: 1 remapped+peering, 4 unknown, 57 peering, 96 active+clean; 3.5 GiB data, 10 GiB used, 615 GiB / 626 GiB avail 2024-02-05T06:38:21.279 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:21.309 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:21.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:21.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:21.412 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:21.451 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-05T06:38:21.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:21.457 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:21.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:21.558 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:21.602 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:21.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:21.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:21.704 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:21.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:21.760 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:21.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:21.774 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:38:21.861 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:21.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:21.906 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:21.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:22.008 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:22.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:22.039 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:22.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:22.140 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:22.168 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:22.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:22.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:22.172 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:21 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:38:22.172 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:21 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:38:22.172 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:21 smithi167 ceph-mon[32774]: osdmap e157: 8 total, 8 up, 7 in 2024-02-05T06:38:22.172 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:21 smithi167 ceph-mon[32774]: 4.14 scrub starts 2024-02-05T06:38:22.172 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:21 smithi167 ceph-mon[32774]: 4.14 scrub ok 2024-02-05T06:38:22.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:21 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:38:22.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:21 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:38:22.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:21 smithi028 ceph-mon[29442]: osdmap e157: 8 total, 8 up, 7 in 2024-02-05T06:38:22.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:21 smithi028 ceph-mon[29442]: 4.14 scrub starts 2024-02-05T06:38:22.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:21 smithi028 ceph-mon[29442]: 4.14 scrub ok 2024-02-05T06:38:22.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:21 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:38:22.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:21 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:38:22.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:21 smithi028 ceph-mon[37984]: osdmap e157: 8 total, 8 up, 7 in 2024-02-05T06:38:22.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:21 smithi028 ceph-mon[37984]: 4.14 scrub starts 2024-02-05T06:38:22.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:21 smithi028 ceph-mon[37984]: 4.14 scrub ok 2024-02-05T06:38:22.269 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:22.299 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:22.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:22.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:22.348 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:38:22.398 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:38:22.400 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:22.429 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:22.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:22.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:22.443 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:38:22.496 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:38:22.529 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T06:38:22.531 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:22.538 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:38:22.575 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:38:22.580 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:22.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:22.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:22.612 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:38:22.653 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:38:22.666 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:38:22.667 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:38:22.667 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:38:22.667 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:38:22.667 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:38:22.667 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:38:22.667 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:38:22.667 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:38:22.683 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:22.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:22.711 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:22.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:22.812 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:22.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:22.836 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:22.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:22.937 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:22.945 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:38:22 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:38:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:38:22.947 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:38:22 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:38:22.967 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:22.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:22.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:23.070 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:23.099 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:23.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:23.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:23.178 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 4] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T06:38:23.178 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T06:38:23.178 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-02-05T06:38:23.179 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph tell osd.0 'trim stale osdmaps' 2024-02-05T06:38:23.201 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:23.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:22 smithi167 ceph-mon[32774]: pgmap v204: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 3.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 887 op/s; 14951/161505 objects misplaced (9.257%); 29 MiB/s, 480 objects/s recovering 2024-02-05T06:38:23.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:22 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2789490461' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:38:23.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:22 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:38:23.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:23.259 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:23.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:23.360 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:23.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:23.404 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:23.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:23.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:22 smithi028 ceph-mon[37984]: pgmap v204: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 3.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 887 op/s; 14951/161505 objects misplaced (9.257%); 29 MiB/s, 480 objects/s recovering 2024-02-05T06:38:23.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:23 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2789490461' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:38:23.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:23 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:38:23.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:22 smithi028 ceph-mon[29442]: pgmap v204: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 3.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 887 op/s; 14951/161505 objects misplaced (9.257%); 29 MiB/s, 480 objects/s recovering 2024-02-05T06:38:23.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:23 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2789490461' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:38:23.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:23 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:38:23.505 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:23.537 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:23.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:23.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:23.639 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:23.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:23.664 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:23.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:23.765 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:23.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:23.797 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:23.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:23.899 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:23.937 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:23.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:23.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:23.962 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:38:23.963 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:38:24.039 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:24.073 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:24.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:24.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:24.175 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:24.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:24.201 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:24.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:24.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:23 smithi167 ceph-mon[32774]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:38:24.231 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:23 smithi167 ceph-mon[32774]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 14 pgs inactive, 17 pgs peering) 2024-02-05T06:38:24.231 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:23 smithi167 ceph-mon[32774]: 5.0 scrub starts 2024-02-05T06:38:24.231 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:23 smithi167 ceph-mon[32774]: 5.0 scrub ok 2024-02-05T06:38:24.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:24.343 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:24.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:24.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:24.445 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:24.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:23 smithi028 ceph-mon[29442]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:38:24.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:24 smithi028 ceph-mon[29442]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 14 pgs inactive, 17 pgs peering) 2024-02-05T06:38:24.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:24 smithi028 ceph-mon[29442]: 5.0 scrub starts 2024-02-05T06:38:24.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:24 smithi028 ceph-mon[29442]: 5.0 scrub ok 2024-02-05T06:38:24.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:23 smithi028 ceph-mon[37984]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:38:24.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:24 smithi028 ceph-mon[37984]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 14 pgs inactive, 17 pgs peering) 2024-02-05T06:38:24.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:24 smithi028 ceph-mon[37984]: 5.0 scrub starts 2024-02-05T06:38:24.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:24 smithi028 ceph-mon[37984]: 5.0 scrub ok 2024-02-05T06:38:24.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:24.497 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:24.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:24.598 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:24.632 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:24.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:24.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:24.734 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:24.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:24.777 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:24.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:24.792 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-05T06:38:24.878 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:24.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:24.937 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:24.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:25.038 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:25.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:25.073 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:25.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:25.174 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:25.197 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:38:25.214 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:25.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:25.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:25.300 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:24 smithi028 ceph-mon[29442]: pgmap v205: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 3.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 628 op/s; 14951/161505 objects misplaced (9.257%); 20 MiB/s, 340 objects/s recovering 2024-02-05T06:38:25.300 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:24 smithi028 ceph-mon[29442]: osdmap e158: 8 total, 8 up, 7 in 2024-02-05T06:38:25.300 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:25 smithi028 ceph-mon[37984]: pgmap v205: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 3.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 628 op/s; 14951/161505 objects misplaced (9.257%); 20 MiB/s, 340 objects/s recovering 2024-02-05T06:38:25.301 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:25 smithi028 ceph-mon[37984]: osdmap e158: 8 total, 8 up, 7 in 2024-02-05T06:38:25.316 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:25.333 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:24 smithi167 ceph-mon[32774]: pgmap v205: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 3.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 628 op/s; 14951/161505 objects misplaced (9.257%); 20 MiB/s, 340 objects/s recovering 2024-02-05T06:38:25.333 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:24 smithi167 ceph-mon[32774]: osdmap e158: 8 total, 8 up, 7 in 2024-02-05T06:38:25.346 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:25.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:25.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:25.429 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:38:25.433 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:38:25.448 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:25.478 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:25.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:25.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:25.580 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:25.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:25.625 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:25.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:25.726 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:25.757 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:25.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:25.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:25.858 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:25.887 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:25.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:25.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:25.989 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:26.018 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:26.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:26.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:26.120 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:26.148 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:26.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:26.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:26.250 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:26.279 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:26.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:26.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:26.381 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:26.418 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:26.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:26.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:26.520 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:26.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:26.551 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:26.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:26.653 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:26.686 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:26.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:26.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:26.788 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:26.819 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:26 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:38:26.819 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:26 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:38:26.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:26 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:38:26.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:26 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:38:26.820 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:26 smithi167 ceph-mon[32774]: osdmap e159: 8 total, 8 up, 7 in 2024-02-05T06:38:26.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:26.821 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:26.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:26.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:26 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:38:26.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:26 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:38:26.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:26 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:38:26.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:26 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:38:26.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:26 smithi028 ceph-mon[29442]: osdmap e159: 8 total, 8 up, 7 in 2024-02-05T06:38:26.880 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:26 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:38:26.880 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:26 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:38:26.880 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:26 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:38:26.880 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:26 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:38:26.880 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:26 smithi028 ceph-mon[37984]: osdmap e159: 8 total, 8 up, 7 in 2024-02-05T06:38:26.921 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:26.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:26.956 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:26.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:27.057 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:27.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:27.104 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:27.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:27.204 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:27.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:27.239 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:27.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:27.311 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:38:27.311 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:38:27.340 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:27.343 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:38:27.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:27.371 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:27.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:27.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:27.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:27.533 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:27.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:27.634 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:27.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:27.697 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:27.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:27.798 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:27.836 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:27.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:27.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:27.901 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:27 smithi167 ceph-mon[32774]: 5.1e scrub starts 2024-02-05T06:38:27.901 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:27 smithi167 ceph-mon[32774]: 5.1e scrub ok 2024-02-05T06:38:27.901 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:27 smithi167 ceph-mon[32774]: pgmap v208: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 3.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 818 op/s; 14684/157419 objects misplaced (9.328%); 19 MiB/s, 320 objects/s recovering 2024-02-05T06:38:27.938 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:27.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:27 smithi028 ceph-mon[29442]: 5.1e scrub starts 2024-02-05T06:38:27.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:27 smithi028 ceph-mon[29442]: 5.1e scrub ok 2024-02-05T06:38:27.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:27 smithi028 ceph-mon[29442]: pgmap v208: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 3.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 818 op/s; 14684/157419 objects misplaced (9.328%); 19 MiB/s, 320 objects/s recovering 2024-02-05T06:38:27.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:27 smithi028 ceph-mon[37984]: 5.1e scrub starts 2024-02-05T06:38:27.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:27 smithi028 ceph-mon[37984]: 5.1e scrub ok 2024-02-05T06:38:27.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:27 smithi028 ceph-mon[37984]: pgmap v208: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 3.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 818 op/s; 14684/157419 objects misplaced (9.328%); 19 MiB/s, 320 objects/s recovering 2024-02-05T06:38:27.965 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:27.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:27.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:28.068 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:28.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:28.130 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:28.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:28.231 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:28.262 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:28.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:28.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:28.349 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-05T06:38:28.364 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:28.428 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:28.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:28.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:28.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:28.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:28.611 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:28.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:28.687 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:38:28.712 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:28.746 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:28.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:28.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:28.848 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:28.901 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:28.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:28.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:29.003 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:29.050 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:29.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:29.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:29.117 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:38:29.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:29.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:29.219 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:29.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:29.320 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:29.364 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:29.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:29.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:29.466 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:29.517 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:29.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:29.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:29.619 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:29.648 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:29.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:29.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:29.750 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:29.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:29.797 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:29.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:29.881 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:29 smithi167 ceph-mon[32774]: 3.7 deep-scrub starts 2024-02-05T06:38:29.882 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:29 smithi167 ceph-mon[32774]: 3.7 deep-scrub ok 2024-02-05T06:38:29.882 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:29 smithi167 ceph-mon[32774]: pgmap v209: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 3.0 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.11k op/s; 11502/148029 objects misplaced (7.770%); 36 MiB/s, 579 objects/s recovering 2024-02-05T06:38:29.897 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:29.942 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:29.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:29.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:29.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:29 smithi028 ceph-mon[37984]: 3.7 deep-scrub starts 2024-02-05T06:38:29.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:29 smithi028 ceph-mon[37984]: 3.7 deep-scrub ok 2024-02-05T06:38:29.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:29 smithi028 ceph-mon[37984]: pgmap v209: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 3.0 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.11k op/s; 11502/148029 objects misplaced (7.770%); 36 MiB/s, 579 objects/s recovering 2024-02-05T06:38:29.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:29 smithi028 ceph-mon[29442]: 3.7 deep-scrub starts 2024-02-05T06:38:29.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:29 smithi028 ceph-mon[29442]: 3.7 deep-scrub ok 2024-02-05T06:38:29.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:29 smithi028 ceph-mon[29442]: pgmap v209: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 3.0 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 1.11k op/s; 11502/148029 objects misplaced (7.770%); 36 MiB/s, 579 objects/s recovering 2024-02-05T06:38:30.044 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:30.083 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:30.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:30.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:30.185 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:30.211 INFO:teuthology.orchestra.run.smithi028.stderr:no valid command found; 10 closest matches: 2024-02-05T06:38:30.211 INFO:teuthology.orchestra.run.smithi028.stderr:0 2024-02-05T06:38:30.211 INFO:teuthology.orchestra.run.smithi028.stderr:1 2024-02-05T06:38:30.211 INFO:teuthology.orchestra.run.smithi028.stderr:2 2024-02-05T06:38:30.211 INFO:teuthology.orchestra.run.smithi028.stderr:abort 2024-02-05T06:38:30.211 INFO:teuthology.orchestra.run.smithi028.stderr:assert 2024-02-05T06:38:30.212 INFO:teuthology.orchestra.run.smithi028.stderr:bench [] [] [] [] 2024-02-05T06:38:30.212 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs debug_inject_read_zeros 2024-02-05T06:38:30.212 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs files list 2024-02-05T06:38:30.212 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs stats 2024-02-05T06:38:30.212 INFO:teuthology.orchestra.run.smithi028.stderr:bluestore allocator dump block 2024-02-05T06:38:30.212 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: invalid command 2024-02-05T06:38:30.225 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:30.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:30.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:30.327 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:30.368 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:30.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:30.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:30.469 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:30.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:30.503 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:30.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:30.604 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:30.634 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:30.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:30.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:30.736 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:30.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:30.772 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:30.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:30.809 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:38:30.809 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:38:30.872 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:30.903 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:30.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:30.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:31.005 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:31.038 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:31.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:31.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:31.139 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:31.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:31.173 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:31.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:31.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:30 smithi167 ceph-mon[32774]: osdmap e160: 8 total, 8 up, 7 in 2024-02-05T06:38:31.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:30 smithi167 ceph-mon[32774]: pgmap v211: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 3.0 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 749 op/s; 11502/148029 objects misplaced (7.770%); 24 MiB/s, 379 objects/s recovering 2024-02-05T06:38:31.274 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:31.300 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:31.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:31.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:31.402 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:31.433 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:31.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:31.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:31.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:30 smithi028 ceph-mon[37984]: osdmap e160: 8 total, 8 up, 7 in 2024-02-05T06:38:31.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:30 smithi028 ceph-mon[37984]: pgmap v211: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 3.0 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 749 op/s; 11502/148029 objects misplaced (7.770%); 24 MiB/s, 379 objects/s recovering 2024-02-05T06:38:31.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:30 smithi028 ceph-mon[29442]: osdmap e160: 8 total, 8 up, 7 in 2024-02-05T06:38:31.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:30 smithi028 ceph-mon[29442]: pgmap v211: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 3.0 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 749 op/s; 11502/148029 objects misplaced (7.770%); 24 MiB/s, 379 objects/s recovering 2024-02-05T06:38:31.535 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:31.568 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:31.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:31.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:31.596 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-05T06:38:31.670 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:31.706 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:31.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:31.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:31.808 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:31.842 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:31.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:31.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:31.944 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:31.965 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:38:31.967 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:31.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:31.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:32.069 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:32.116 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:32.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:32.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:32.218 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:32.249 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:32.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:32.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:32.351 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:32.388 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:32.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:32.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:32.490 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:32.520 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:32.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:32.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:32.622 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:32.652 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:32.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:32.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:32.730 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:38:32 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:38:32.754 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:32.779 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:38:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:38:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:38:32.794 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:32.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:32.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:32.829 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-05T06:38:32.830 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-05T06:38:32.830 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph tell osd.1 'trim stale osdmaps' 2024-02-05T06:38:32.897 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:32.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:32.922 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:32.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:33.023 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:33.076 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:33.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:33.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:33.178 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:33.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:33.213 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:33.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:33.314 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:33.344 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:33.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:33.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:33.446 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:33.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:33 smithi167 ceph-mon[32774]: 4.17 scrub starts 2024-02-05T06:38:33.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:33 smithi167 ceph-mon[32774]: 4.17 scrub ok 2024-02-05T06:38:33.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:33 smithi167 ceph-mon[32774]: pgmap v212: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 2.7 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.27k op/s; 8058/133833 objects misplaced (6.021%); 41 MiB/s, 649 objects/s recovering 2024-02-05T06:38:33.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:33 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:38:33.481 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:33.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:33.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:33.583 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:33.589 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:33 smithi028 ceph-mon[37984]: 4.17 scrub starts 2024-02-05T06:38:33.589 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:33 smithi028 ceph-mon[37984]: 4.17 scrub ok 2024-02-05T06:38:33.589 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:33 smithi028 ceph-mon[37984]: pgmap v212: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 2.7 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.27k op/s; 8058/133833 objects misplaced (6.021%); 41 MiB/s, 649 objects/s recovering 2024-02-05T06:38:33.589 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:33 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:38:33.592 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:33 smithi028 ceph-mon[29442]: 4.17 scrub starts 2024-02-05T06:38:33.592 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:33 smithi028 ceph-mon[29442]: 4.17 scrub ok 2024-02-05T06:38:33.592 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:33 smithi028 ceph-mon[29442]: pgmap v212: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 2.7 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.27k op/s; 8058/133833 objects misplaced (6.021%); 41 MiB/s, 649 objects/s recovering 2024-02-05T06:38:33.592 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:33 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:38:33.629 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:33.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:33.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:33.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:33.785 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:33.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:33.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:33.887 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:33.915 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:33.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:33.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:34.017 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:34.045 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:34.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:34.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:34.146 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:34.174 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:34.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:34.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:34.276 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:34.316 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:34.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:34.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:34.326 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-02-05T06:38:34.418 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:34.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:34 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/534968701' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:38:34.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:34 smithi028 ceph-mon[37984]: from='client.44918 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:38:34.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:34 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/534968701' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:38:34.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:34 smithi028 ceph-mon[29442]: from='client.44918 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:38:34.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:34.451 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:34.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:34.552 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:34.592 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:34 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/534968701' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:38:34.592 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:34 smithi167 ceph-mon[32774]: from='client.44918 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:38:34.604 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:34.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:34.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:34.706 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:34.737 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:34.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:34.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:34.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:34.892 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:34.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:34.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:34.994 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:35.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:35.038 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:35.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:35.096 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:38:35.096 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:38:35.139 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:35.171 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:35.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:35.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:35.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:35.319 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:35.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:35.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:35.421 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:35.450 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:35.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:35.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:35.486 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:38:35.552 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:35.581 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:35.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:35.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:35.588 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:35 smithi028 ceph-mon[37984]: pgmap v213: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 2.7 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 8058/133833 objects misplaced (6.021%); 36 MiB/s, 573 objects/s recovering 2024-02-05T06:38:35.589 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:35 smithi028 ceph-mon[37984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:38:35.589 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:35 smithi028 ceph-mon[37984]: from='client.44918 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:38:35.589 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:35 smithi028 ceph-mon[37984]: osdmap e161: 8 total, 8 up, 7 in 2024-02-05T06:38:35.589 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:35 smithi028 ceph-mon[37984]: osdmap e162: 8 total, 8 up, 7 in 2024-02-05T06:38:35.589 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:35 smithi028 ceph-mon[29442]: pgmap v213: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 2.7 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 8058/133833 objects misplaced (6.021%); 36 MiB/s, 573 objects/s recovering 2024-02-05T06:38:35.589 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:35 smithi028 ceph-mon[29442]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:38:35.589 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:35 smithi028 ceph-mon[29442]: from='client.44918 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:38:35.589 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:35 smithi028 ceph-mon[29442]: osdmap e161: 8 total, 8 up, 7 in 2024-02-05T06:38:35.589 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:35 smithi028 ceph-mon[29442]: osdmap e162: 8 total, 8 up, 7 in 2024-02-05T06:38:35.645 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:35 smithi167 ceph-mon[32774]: pgmap v213: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 2.7 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 8058/133833 objects misplaced (6.021%); 36 MiB/s, 573 objects/s recovering 2024-02-05T06:38:35.645 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:35 smithi167 ceph-mon[32774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:38:35.645 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:35 smithi167 ceph-mon[32774]: from='client.44918 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:38:35.645 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:35 smithi167 ceph-mon[32774]: osdmap e161: 8 total, 8 up, 7 in 2024-02-05T06:38:35.645 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:35 smithi167 ceph-mon[32774]: osdmap e162: 8 total, 8 up, 7 in 2024-02-05T06:38:35.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:35.720 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:35.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:35.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:35.822 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:35.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:35.866 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:35.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:35.960 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-05T06:38:35.967 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:35.994 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:35.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:35.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:36.096 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:36.125 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:36.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:36.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:36.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:36.303 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:36.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:36.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:36.405 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:36.435 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:36.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:36.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:36.536 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:36.564 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:36.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:36.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:36.666 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:36.712 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:36.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:36.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:36.814 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:36.856 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:36.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:36.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:36.958 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:36.992 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:36.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:36.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:37.094 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:37.127 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:37.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:37.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:37.229 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:37.268 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:37.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:37.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:37.363 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:37 smithi028 ceph-mon[37984]: pgmap v216: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 2.6 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 815 B/s rd, 0 B/s wr, 990 op/s; 7742/129396 objects misplaced (5.983%); 24 MiB/s, 389 objects/s recovering 2024-02-05T06:38:37.363 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:37 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2182603960' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:38:37.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:37 smithi028 ceph-mon[29442]: pgmap v216: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 2.6 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 815 B/s rd, 0 B/s wr, 990 op/s; 7742/129396 objects misplaced (5.983%); 24 MiB/s, 389 objects/s recovering 2024-02-05T06:38:37.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:37 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2182603960' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:38:37.364 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:38:37.370 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:37.400 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:37.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:37.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:37.411 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:38:37.448 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T06:38:37.454 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:37 smithi167 ceph-mon[32774]: pgmap v216: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 2.6 GiB data, 8.8 GiB used, 617 GiB / 626 GiB avail; 815 B/s rd, 0 B/s wr, 990 op/s; 7742/129396 objects misplaced (5.983%); 24 MiB/s, 389 objects/s recovering 2024-02-05T06:38:37.454 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:37 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2182603960' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:38:37.454 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:38:37.502 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:38:37.502 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:37.546 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:38:37.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:37.572 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:37.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:37.589 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:38:37.633 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:38:37.673 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:37.677 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:38:37.687 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:38:37.687 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:38:37.687 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:38:37.687 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:38:37.687 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:38:37.687 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:38:37.687 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:38:37.687 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:38:37.701 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:37.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:37.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:37.803 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:37.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:37.859 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:37.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:37.960 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:37.991 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:37.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:37.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:38.092 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:38.125 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:38.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:38.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:38.227 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:38.255 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:38.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:38.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:38.357 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:38.390 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:38.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:38.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:38.492 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:38.521 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:38.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:38.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:38.622 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:38.654 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:38.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:38.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:38.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:38.781 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:38.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:38.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:38.883 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:38.897 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:38:38.907 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:38.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:38.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:38.942 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:38:38.971 INFO:teuthology.orchestra.run.smithi028.stderr:no valid command found; 10 closest matches: 2024-02-05T06:38:38.971 INFO:teuthology.orchestra.run.smithi028.stderr:0 2024-02-05T06:38:38.971 INFO:teuthology.orchestra.run.smithi028.stderr:1 2024-02-05T06:38:38.971 INFO:teuthology.orchestra.run.smithi028.stderr:2 2024-02-05T06:38:38.971 INFO:teuthology.orchestra.run.smithi028.stderr:abort 2024-02-05T06:38:38.971 INFO:teuthology.orchestra.run.smithi028.stderr:assert 2024-02-05T06:38:38.972 INFO:teuthology.orchestra.run.smithi028.stderr:bench [] [] [] [] 2024-02-05T06:38:38.972 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs debug_inject_read_zeros 2024-02-05T06:38:38.972 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs files list 2024-02-05T06:38:38.972 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs stats 2024-02-05T06:38:38.972 INFO:teuthology.orchestra.run.smithi028.stderr:bluestore allocator dump block 2024-02-05T06:38:38.972 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: invalid command 2024-02-05T06:38:39.009 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:39.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:39.034 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:39.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:39.135 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:39.159 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:39.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:39.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:39.261 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:39.288 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:39.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:39.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:39.391 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:39.424 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:39.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:39.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:39.526 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:39.552 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:39.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:39.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:39.653 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:39.677 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:39.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:39 smithi028 ceph-mon[37984]: pgmap v217: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 2.4 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 4823/119730 objects misplaced (4.028%); 39 MiB/s, 618 objects/s recovering 2024-02-05T06:38:39.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:39 smithi028 ceph-mon[37984]: osdmap e163: 8 total, 8 up, 7 in 2024-02-05T06:38:39.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:39 smithi028 ceph-mon[29442]: pgmap v217: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 2.4 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 4823/119730 objects misplaced (4.028%); 39 MiB/s, 618 objects/s recovering 2024-02-05T06:38:39.694 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:39 smithi028 ceph-mon[29442]: osdmap e163: 8 total, 8 up, 7 in 2024-02-05T06:38:39.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:39.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:39.722 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-05T06:38:39.723 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-05T06:38:39.723 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph tell osd.2 'trim stale osdmaps' 2024-02-05T06:38:39.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:39 smithi167 ceph-mon[32774]: pgmap v217: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 2.4 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 4823/119730 objects misplaced (4.028%); 39 MiB/s, 618 objects/s recovering 2024-02-05T06:38:39.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:39 smithi167 ceph-mon[32774]: osdmap e163: 8 total, 8 up, 7 in 2024-02-05T06:38:39.795 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:39.818 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:39.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:39.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:39.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:39.946 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:39.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:39.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:40.048 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:40.081 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:40.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:40.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:40.183 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:40.219 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:40.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:40.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:40.321 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:40.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:40.346 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:40.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:40.447 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:40.472 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:40.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:40.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:40.575 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:40.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:40.599 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:40.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:40.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:40 smithi028 ceph-mon[29442]: osdmap e164: 8 total, 8 up, 7 in 2024-02-05T06:38:40.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:40 smithi028 ceph-mon[29442]: osdmap e165: 8 total, 8 up, 7 in 2024-02-05T06:38:40.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:40 smithi028 ceph-mon[37984]: osdmap e164: 8 total, 8 up, 7 in 2024-02-05T06:38:40.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:40 smithi028 ceph-mon[37984]: osdmap e165: 8 total, 8 up, 7 in 2024-02-05T06:38:40.701 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:40.703 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:38:40.725 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:40.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:40.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:40.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:40 smithi167 ceph-mon[32774]: osdmap e164: 8 total, 8 up, 7 in 2024-02-05T06:38:40.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:40 smithi167 ceph-mon[32774]: osdmap e165: 8 total, 8 up, 7 in 2024-02-05T06:38:40.827 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:40.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:40.852 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:40.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:40.953 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:40.976 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:40.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:40.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:41.078 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:41.117 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:41.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:41.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:41.219 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:41.247 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:41.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:41.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:41.349 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:41.389 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:41.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:41.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:41.491 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:41.520 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:41.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:41.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:41.622 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:41.661 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:41.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:41.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:41.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:41 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:38:41.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:41 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:38:41.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:41 smithi028 ceph-mon[37984]: pgmap v221: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 2.4 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 970 B/s rd, 0 B/s wr, 892 op/s; 4823/119730 objects misplaced (4.028%); 30 MiB/s, 474 objects/s recovering 2024-02-05T06:38:41.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:41 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:38:41.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:41 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:38:41.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:41 smithi028 ceph-mon[37984]: osdmap e166: 8 total, 8 up, 7 in 2024-02-05T06:38:41.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:41 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:38:41.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:41 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:38:41.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:41 smithi028 ceph-mon[29442]: pgmap v221: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 2.4 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 970 B/s rd, 0 B/s wr, 892 op/s; 4823/119730 objects misplaced (4.028%); 30 MiB/s, 474 objects/s recovering 2024-02-05T06:38:41.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:41 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:38:41.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:41 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:38:41.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:41 smithi028 ceph-mon[29442]: osdmap e166: 8 total, 8 up, 7 in 2024-02-05T06:38:41.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:41 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:38:41.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:41 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:38:41.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:41 smithi167 ceph-mon[32774]: pgmap v221: 158 pgs: 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 153 active+clean; 2.4 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 970 B/s rd, 0 B/s wr, 892 op/s; 4823/119730 objects misplaced (4.028%); 30 MiB/s, 474 objects/s recovering 2024-02-05T06:38:41.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:41 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:38:41.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:41 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:38:41.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:41 smithi167 ceph-mon[32774]: osdmap e166: 8 total, 8 up, 7 in 2024-02-05T06:38:41.762 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:41.791 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:41.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:41.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:41.893 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:41.923 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:41.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:41.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:42.025 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:42.062 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:42.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:42.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:42.164 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:42.209 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:42.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:42.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:42.311 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:42.337 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:42.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:42.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:42.439 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:42.479 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:42.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:42.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:42.580 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:42.604 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:42.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:42.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:42.696 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:38:42 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:38:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:38:42.706 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:42.745 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:42.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:42.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:42.847 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:42.881 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:42.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:42.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:42.980 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:38:42 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:38:42.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:43.019 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:43.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:43.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:43.120 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:43.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:43.145 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:43.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:43.246 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:43.274 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:43.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:43.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:43.376 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:43.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:43.400 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:43.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:43.501 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:43.535 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:43.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:43.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:43.637 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:43.666 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:43.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:43.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:43.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:43 smithi028 ceph-mon[29442]: pgmap v223: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.3 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 710 op/s; 2325/111219 objects misplaced (2.090%); 34 MiB/s, 551 objects/s recovering 2024-02-05T06:38:43.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:43 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:38:43.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:43 smithi028 ceph-mon[37984]: pgmap v223: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.3 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 710 op/s; 2325/111219 objects misplaced (2.090%); 34 MiB/s, 551 objects/s recovering 2024-02-05T06:38:43.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:43 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:38:43.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:43 smithi167 ceph-mon[32774]: pgmap v223: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.3 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 710 op/s; 2325/111219 objects misplaced (2.090%); 34 MiB/s, 551 objects/s recovering 2024-02-05T06:38:43.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:43 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:38:43.767 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:43.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:43.792 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:43.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:43.893 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:43.931 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:43.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:43.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:44.033 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:44.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:44.058 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:44.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:44.159 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:44.197 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:44.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:44.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:44.299 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:44.337 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:44.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:44.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:44.439 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:44.463 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:44.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:44.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:44.565 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:44.592 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:44.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:44.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:44.694 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:44.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:44.730 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:44.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:44.831 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:44.859 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:44.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:44.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:44.962 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:44.985 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:44.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:44.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:45.087 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:45.115 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:45.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:45.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:45.216 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:45.243 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:45.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:45.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:45.345 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:45.368 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:45.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:45.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:45.470 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:45.496 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:45.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:45.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:45.598 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:45.632 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:45.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:45.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:45.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:45 smithi028 ceph-mon[29442]: pgmap v224: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.3 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 867 B/s rd, 0 B/s wr, 481 op/s; 2325/111219 objects misplaced (2.090%); 23 MiB/s, 373 objects/s recovering 2024-02-05T06:38:45.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:45 smithi028 ceph-mon[37984]: pgmap v224: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.3 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 867 B/s rd, 0 B/s wr, 481 op/s; 2325/111219 objects misplaced (2.090%); 23 MiB/s, 373 objects/s recovering 2024-02-05T06:38:45.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:45 smithi167 ceph-mon[32774]: pgmap v224: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.3 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 867 B/s rd, 0 B/s wr, 481 op/s; 2325/111219 objects misplaced (2.090%); 23 MiB/s, 373 objects/s recovering 2024-02-05T06:38:45.734 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:45.772 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:45.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:45.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:45.874 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:45.899 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:45.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:45.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:46.002 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:46.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:46.042 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:46.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:46.143 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:46.170 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:46.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:46.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:46.272 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:46.299 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:46.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:46.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:46.401 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:46.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:46.461 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:46.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:46.562 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:46.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:46.592 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:46.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:46.693 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:46.734 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:46.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:46.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:46.836 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:46.884 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:46.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:46.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:46.986 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:47.031 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:47.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:47.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:47.133 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:47.161 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:47.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:47.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:47.262 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:47.293 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:47.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:47.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:47.370 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:47 smithi028 ceph-mon[29442]: pgmap v225: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.2 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 742 B/s rd, 0 B/s wr, 632 op/s; 2007/106656 objects misplaced (1.882%); 20 MiB/s, 319 objects/s recovering 2024-02-05T06:38:47.394 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:47.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:47.440 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:47.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:47.541 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:47.567 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:47.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:47.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:47.668 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:47.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:47 smithi028 ceph-mon[37984]: pgmap v225: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.2 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 742 B/s rd, 0 B/s wr, 632 op/s; 2007/106656 objects misplaced (1.882%); 20 MiB/s, 319 objects/s recovering 2024-02-05T06:38:47.708 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:47.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:47.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:47.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:47 smithi167 ceph-mon[32774]: pgmap v225: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.2 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 742 B/s rd, 0 B/s wr, 632 op/s; 2007/106656 objects misplaced (1.882%); 20 MiB/s, 319 objects/s recovering 2024-02-05T06:38:47.810 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:47.811 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:38:47.861 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:47.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:47.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:38:47.924 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:38:47.924 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-02-05T06:38:47.924 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.925 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:199624 7.6 7:6a17c4d3:::benchmark_data_smithi028_77509_object84504:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-02-05T06:38:47.925 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:47.905856+0000", 2024-02-05T06:38:47.925 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.012221437, 2024-02-05T06:38:47.925 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012253178999999999, 2024-02-05T06:38:47.925 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:38:47.925 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "waiting for sub ops", 2024-02-05T06:38:47.925 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:38:47.925 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:38:47.925 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:38:47.925 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 199624 2024-02-05T06:38:47.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:38:47.925 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:38:47.925 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.925 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:38:47.926 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:47.905856+0000", 2024-02-05T06:38:47.926 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:38:47.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:38:47.926 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.926 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:38:47.926 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:47.905856+0000", 2024-02-05T06:38:47.926 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:38:47.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:38:47.926 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.926 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:38:47.926 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:47.905857+0000", 2024-02-05T06:38:47.926 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4470000000000001e-06 2024-02-05T06:38:47.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:38:47.926 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.927 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:38:47.927 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:47.905861+0000", 2024-02-05T06:38:47.927 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2570000000000001e-06 2024-02-05T06:38:47.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:38:47.927 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.927 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:38:47.927 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:47.905871+0000", 2024-02-05T06:38:47.927 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6379999999999999e-06 2024-02-05T06:38:47.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:38:47.927 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.927 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:38:47.927 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:47.905879+0000", 2024-02-05T06:38:47.927 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8590000000000005e-06 2024-02-05T06:38:47.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:38:47.927 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.928 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:38:47.928 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:47.914555+0000", 2024-02-05T06:38:47.928 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0086757029999999999 2024-02-05T06:38:47.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:38:47.928 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.928 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:38:47.928 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:47.914897+0000", 2024-02-05T06:38:47.928 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00034220099999999999 2024-02-05T06:38:47.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:38:47.928 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.928 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,5", 2024-02-05T06:38:47.928 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:47.915346+0000", 2024-02-05T06:38:47.928 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00044904899999999998 2024-02-05T06:38:47.929 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:38:47.929 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:38:47.929 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:38:47.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:38:47.929 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.929 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14847.0:199623 7.e e166/96 7:74cb29f4:::benchmark_data_smithi028_77509_object84503:head v 166'12569, mlcod=166'12567)", 2024-02-05T06:38:47.929 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:47.906300+0000", 2024-02-05T06:38:47.929 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.011777492000000001, 2024-02-05T06:38:47.929 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011900028999999999, 2024-02-05T06:38:47.929 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:38:47.929 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-02-05T06:38:47.929 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:38:47.929 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.929 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:38:47.930 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:47.906300+0000", 2024-02-05T06:38:47.930 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:38:47.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:38:47.930 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.930 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:38:47.930 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:47.906300+0000", 2024-02-05T06:38:47.930 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:38:47.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:38:47.930 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.930 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:38:47.930 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:47.906301+0000", 2024-02-05T06:38:47.930 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2559999999999999e-06 2024-02-05T06:38:47.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:38:47.930 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.931 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:38:47.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:47.906308+0000", 2024-02-05T06:38:47.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.951e-06 2024-02-05T06:38:47.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:38:47.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.931 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:38:47.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:47.906317+0000", 2024-02-05T06:38:47.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.611e-06 2024-02-05T06:38:47.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:38:47.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.931 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:38:47.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:47.906325+0000", 2024-02-05T06:38:47.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1720000000000001e-06 2024-02-05T06:38:47.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:38:47.932 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.932 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:38:47.932 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:47.910120+0000", 2024-02-05T06:38:47.932 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0037952559999999999 2024-02-05T06:38:47.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:38:47.932 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.932 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:38:47.932 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:47.910185+0000", 2024-02-05T06:38:47.932 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5395999999999995e-05 2024-02-05T06:38:47.932 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:38:47.932 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:38:47.932 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:38:47.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:38:47.932 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.932 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14847.0:199632 7.e e166/96 7:70765b14:::benchmark_data_smithi028_77509_object84512:head v 166'12570, mlcod=166'12567)", 2024-02-05T06:38:47.933 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:47.916512+0000", 2024-02-05T06:38:47.933 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.001564555, 2024-02-05T06:38:47.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018705779999999999, 2024-02-05T06:38:47.933 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:38:47.933 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-02-05T06:38:47.933 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:38:47.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.933 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:38:47.933 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:47.916512+0000", 2024-02-05T06:38:47.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:38:47.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:38:47.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.933 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:38:47.933 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:47.916512+0000", 2024-02-05T06:38:47.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:38:47.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:38:47.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.934 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:38:47.934 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:47.916514+0000", 2024-02-05T06:38:47.934 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7260000000000001e-06 2024-02-05T06:38:47.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:38:47.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.934 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:38:47.934 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:47.916523+0000", 2024-02-05T06:38:47.934 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.551e-06 2024-02-05T06:38:47.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:38:47.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.934 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:38:47.934 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:47.916538+0000", 2024-02-05T06:38:47.934 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5034999999999999e-05 2024-02-05T06:38:47.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:38:47.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.935 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:38:47.935 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:47.916554+0000", 2024-02-05T06:38:47.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6178999999999999e-05 2024-02-05T06:38:47.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:38:47.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.935 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:38:47.935 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:47.917967+0000", 2024-02-05T06:38:47.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0014134270000000001 2024-02-05T06:38:47.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:38:47.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.935 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:38:47.935 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:47.918060+0000", 2024-02-05T06:38:47.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2761000000000002e-05 2024-02-05T06:38:47.936 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:38:47.936 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:38:47.936 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:38:47.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:38:47.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.936 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(client.14847.0:199624 7.6 e166/164)", 2024-02-05T06:38:47.936 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:47.918293+0000", 2024-02-05T06:38:47.936 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4294967295.9997835, 2024-02-05T06:38:47.936 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000152576, 2024-02-05T06:38:47.936 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:38:47.936 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "queued for pg", 2024-02-05T06:38:47.936 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:38:47.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.936 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:38:47.937 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:47.918293+0000", 2024-02-05T06:38:47.937 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:38:47.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:38:47.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.937 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:38:47.937 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:47.918293+0000", 2024-02-05T06:38:47.937 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:38:47.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:38:47.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.937 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:38:47.937 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:47.918295+0000", 2024-02-05T06:38:47.937 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3790000000000001e-06 2024-02-05T06:38:47.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:38:47.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.937 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:38:47.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:47.918303+0000", 2024-02-05T06:38:47.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8930000000000005e-06 2024-02-05T06:38:47.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:38:47.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.938 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:38:47.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:47.918311+0000", 2024-02-05T06:38:47.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7280000000000001e-06 2024-02-05T06:38:47.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:38:47.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:38:47.938 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:38:47.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:47.918319+0000", 2024-02-05T06:38:47.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2019999999999998e-06 2024-02-05T06:38:47.938 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:38:47.938 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:38:47.939 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:38:47.939 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:38:47.939 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:38:47.939 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 4 2024-02-05T06:38:47.939 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:38:47.963 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:47.990 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:47.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:47.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:48.092 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:48.119 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:48.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:48.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:48.221 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:48.262 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:48.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:48.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:48.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:48.407 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-02-05T06:38:48.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:48.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:48.466 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:48 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:38:48.466 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:48 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:38:48.466 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:48 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1329079727' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:38:48.467 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:48 smithi028 ceph-mon[37984]: from='client.24940 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:38:48.467 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:48 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:38:48.467 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:48 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:38:48.467 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:48 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1329079727' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:38:48.467 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:48 smithi028 ceph-mon[29442]: from='client.24940 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:38:48.518 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:48.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:48.550 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:48.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:48.651 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:48.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:48.681 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:48.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:48.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:48 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:38:48.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:48 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:38:48.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:48 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1329079727' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:38:48.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:48 smithi167 ceph-mon[32774]: from='client.24940 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:38:48.782 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:48.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:48.810 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:48.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:48.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:48.948 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:48.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:48.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:49.050 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:49.091 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:49.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:49.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:49.193 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:49.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:49.236 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:49.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:49.337 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:49.338 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-05T06:38:49.364 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:49.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:49.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:49.466 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:49.495 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:49.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:49.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:49.596 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:49.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:49.649 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:49.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:49.668 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:49 smithi028 ceph-mon[29442]: pgmap v226: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.0 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 920 op/s; 2007/96909 objects misplaced (2.071%); 17 MiB/s, 266 objects/s recovering 2024-02-05T06:38:49.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:49 smithi028 ceph-mon[29442]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:38:49.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:49 smithi028 ceph-mon[29442]: from='client.24940 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:38:49.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:49 smithi028 ceph-mon[29442]: osdmap e167: 8 total, 8 up, 7 in 2024-02-05T06:38:49.669 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:49 smithi028 ceph-mon[37984]: pgmap v226: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.0 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 920 op/s; 2007/96909 objects misplaced (2.071%); 17 MiB/s, 266 objects/s recovering 2024-02-05T06:38:49.669 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:49 smithi028 ceph-mon[37984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:38:49.669 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:49 smithi028 ceph-mon[37984]: from='client.24940 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:38:49.669 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:49 smithi028 ceph-mon[37984]: osdmap e167: 8 total, 8 up, 7 in 2024-02-05T06:38:49.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:49 smithi167 ceph-mon[32774]: pgmap v226: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.0 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 920 op/s; 2007/96909 objects misplaced (2.071%); 17 MiB/s, 266 objects/s recovering 2024-02-05T06:38:49.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:49 smithi167 ceph-mon[32774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:38:49.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:49 smithi167 ceph-mon[32774]: from='client.24940 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:38:49.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:49 smithi167 ceph-mon[32774]: osdmap e167: 8 total, 8 up, 7 in 2024-02-05T06:38:49.750 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:49.778 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:49.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:49.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:49.879 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:49.920 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:49.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:49.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:50.022 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:50.052 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:50.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:50.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:50.154 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:50.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:50.193 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:50.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:50.294 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:50.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:50.324 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:50.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:50.425 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:50.450 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:50.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:50.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:50.552 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:50.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:50.582 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:50.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:50.683 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:50.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:50.711 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:50.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:50.812 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:50.851 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:50.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:50.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:50.953 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:50.993 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:50.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:50.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:51.095 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:51.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:51.135 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:51.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:51.230 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:50 smithi167 ceph-mon[32774]: osdmap e168: 8 total, 8 up, 7 in 2024-02-05T06:38:51.231 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:50 smithi167 ceph-mon[32774]: pgmap v229: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.0 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 596 op/s; 2007/96909 objects misplaced (2.071%) 2024-02-05T06:38:51.236 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:51.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:51.261 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:51.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:51.362 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:51.390 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:51.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:51.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:51.426 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:50 smithi028 ceph-mon[29442]: osdmap e168: 8 total, 8 up, 7 in 2024-02-05T06:38:51.426 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:50 smithi028 ceph-mon[29442]: pgmap v229: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.0 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 596 op/s; 2007/96909 objects misplaced (2.071%) 2024-02-05T06:38:51.426 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:50 smithi028 ceph-mon[37984]: osdmap e168: 8 total, 8 up, 7 in 2024-02-05T06:38:51.426 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:50 smithi028 ceph-mon[37984]: pgmap v229: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.0 GiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 596 op/s; 2007/96909 objects misplaced (2.071%) 2024-02-05T06:38:51.492 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:51.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:51.521 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:51.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:51.622 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:51.651 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:51.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:51.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:51.753 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:51.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:51.778 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:51.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:51.879 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:51.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:51.908 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:51.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:52.009 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:52.034 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:52.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:52.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:52.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:52.182 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:52.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:52.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:52.284 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:52.313 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:52.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:52.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:52.415 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:52.441 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:52.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:52.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:52.544 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:52.570 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:52.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:52.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:52.651 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:38:52.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:52.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:52.727 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:52.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:52.795 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:38:52 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:38:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:38:52.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:52.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:52.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:52.973 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:52.980 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:38:52 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:38:53.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:53.014 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:53.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:53.115 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:53.143 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:53.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:53.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:53.245 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:53.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:53.287 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:53.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:53.388 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:53.416 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:53.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:53.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:53.511 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:38:53.518 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:53.556 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:53.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:53.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:53.569 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:38:53.569 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:53 smithi028 ceph-mon[37984]: pgmap v230: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.7 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s; 1683/81729 objects misplaced (2.059%) 2024-02-05T06:38:53.569 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:53 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:38:53.570 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:53 smithi028 ceph-mon[29442]: pgmap v230: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.7 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s; 1683/81729 objects misplaced (2.059%) 2024-02-05T06:38:53.570 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:53 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:38:53.613 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:38:53.652 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:38:53.658 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:53.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:53.696 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:53.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:53.697 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:38:53.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:53 smithi167 ceph-mon[32774]: pgmap v230: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.7 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s; 1683/81729 objects misplaced (2.059%) 2024-02-05T06:38:53.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:53 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:38:53.739 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:38:53.778 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:38:53.798 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:53.816 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:38:53.820 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:38:53.820 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:38:53.821 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:38:53.821 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:38:53.821 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:38:53.821 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:38:53.821 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:38:53.821 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:38:53.824 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:53.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:53.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:53.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:53.963 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:53.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:53.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:53.995 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T06:38:54.065 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:54.093 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:54.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:54.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:54.195 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:54.224 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:54.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:54.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:54.326 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:54.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:54.388 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:54.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:54.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:54 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1019518564' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:38:54.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:54 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1019518564' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:38:54.490 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:54.517 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:54.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:54.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:54.619 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:54.662 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:54.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:54.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:54.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:54 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1019518564' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:38:54.764 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:54.791 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:54.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:54.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:54.893 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:54.949 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:54.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:54.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:55.051 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:55.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:55.097 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:55.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:55.198 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:55.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:55.239 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:55.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:55.340 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:55.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:55.397 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:55.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:55.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:55 smithi028 ceph-mon[37984]: pgmap v231: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.7 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 1683/81729 objects misplaced (2.059%) 2024-02-05T06:38:55.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:55 smithi028 ceph-mon[29442]: pgmap v231: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.7 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 1683/81729 objects misplaced (2.059%) 2024-02-05T06:38:55.498 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:55.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:55.530 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:55.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:55.631 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:55.658 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:55.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:55.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:55.705 INFO:teuthology.orchestra.run.smithi028.stderr:no valid command found; 10 closest matches: 2024-02-05T06:38:55.705 INFO:teuthology.orchestra.run.smithi028.stderr:0 2024-02-05T06:38:55.705 INFO:teuthology.orchestra.run.smithi028.stderr:1 2024-02-05T06:38:55.705 INFO:teuthology.orchestra.run.smithi028.stderr:2 2024-02-05T06:38:55.705 INFO:teuthology.orchestra.run.smithi028.stderr:abort 2024-02-05T06:38:55.705 INFO:teuthology.orchestra.run.smithi028.stderr:assert 2024-02-05T06:38:55.705 INFO:teuthology.orchestra.run.smithi028.stderr:bench [] [] [] [] 2024-02-05T06:38:55.705 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs debug_inject_read_zeros 2024-02-05T06:38:55.705 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs files list 2024-02-05T06:38:55.705 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs stats 2024-02-05T06:38:55.705 INFO:teuthology.orchestra.run.smithi028.stderr:bluestore allocator dump block 2024-02-05T06:38:55.705 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: invalid command 2024-02-05T06:38:55.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:55 smithi167 ceph-mon[32774]: pgmap v231: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.7 GiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 1683/81729 objects misplaced (2.059%) 2024-02-05T06:38:55.760 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:55.788 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:55.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:55.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:55.890 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:55.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:55.932 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:55.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:56.033 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:56.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:56.062 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:56.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:56.163 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:56.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:56.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:56.301 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:56.329 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:56.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:56.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:56.431 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:56.472 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:56.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:56.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:56.574 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:56.601 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:56.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:56.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:56.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:56 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:38:56.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:56 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:38:56.703 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:56.729 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:56.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:56.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:56.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:56 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:38:56.831 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:56.858 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:56.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:56.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:56.960 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:38:56.987 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:56.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:56.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:38:57.089 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:57.116 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:57.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:57.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:57.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:57.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:57.255 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:57.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:57.356 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:57.400 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:57.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:57.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:57.425 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:38:57.426 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:38:57.502 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:57.527 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:57 smithi028 ceph-mon[29442]: pgmap v232: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.5 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 848 op/s; 1321/76551 objects misplaced (1.726%) 2024-02-05T06:38:57.528 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:57 smithi028 ceph-mon[37984]: pgmap v232: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.5 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 848 op/s; 1321/76551 objects misplaced (1.726%) 2024-02-05T06:38:57.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:57.529 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:57.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:57.630 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:57.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:57.658 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:57.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:57.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:57 smithi167 ceph-mon[32774]: pgmap v232: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.5 GiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 848 op/s; 1321/76551 objects misplaced (1.726%) 2024-02-05T06:38:57.759 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:57.785 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:57.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:57.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:57.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:57.925 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:57.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:57.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:58.027 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:58.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:58.069 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:58.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:58.117 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:38:58.170 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:58.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-05T06:38:58.242 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:58.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:58.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:58.344 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:58.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:58.433 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:58.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:58.534 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:58.561 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:58.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:58.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:58.664 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:38:58.709 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:38:58.729 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:58.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:58.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:38:58.754 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-05T06:38:58.754 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-05T06:38:58.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph tell osd.3 'trim stale osdmaps' 2024-02-05T06:38:58.806 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:38:58.830 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:38:58.858 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:58.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:58.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:38:58.960 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:59.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:59.035 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:59.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:59.137 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:59.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:59.167 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:59.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:59.268 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:38:59.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:59.309 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:59.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:38:59.410 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:59.438 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:59.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:59.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:38:59.540 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:38:59.577 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:59.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:59.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:38:59.679 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:38:59.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:38:59 smithi028 ceph-mon[29442]: pgmap v233: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.3 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.07k op/s; 1321/65322 objects misplaced (2.022%) 2024-02-05T06:38:59.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:38:59 smithi028 ceph-mon[37984]: pgmap v233: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.3 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.07k op/s; 1321/65322 objects misplaced (2.022%) 2024-02-05T06:38:59.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:59.716 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:38:59.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:38:59.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:38:59 smithi167 ceph-mon[32774]: pgmap v233: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.3 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.07k op/s; 1321/65322 objects misplaced (2.022%) 2024-02-05T06:38:59.816 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:38:59.845 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:59.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:59.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:38:59.947 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:38:59.976 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:38:59.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:38:59.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:00.078 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:00.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:00.120 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:00.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:00.224 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:00.251 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:00.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:00.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:00.353 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:00.381 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:00.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:00.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:00.483 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:00.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:00.512 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:00.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:00.613 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:00.660 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:00.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:00.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:00.762 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:00.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:00.791 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:00.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:00.892 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:00.940 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:39:00.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:00.949 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:00.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:01.050 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:01.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:01.118 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:01.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:01.196 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:00 smithi028 ceph-mon[37984]: pgmap v234: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.3 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 996 B/s rd, 0 B/s wr, 1.02k op/s; 1321/65322 objects misplaced (2.022%) 2024-02-05T06:39:01.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:00 smithi028 ceph-mon[29442]: pgmap v234: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.3 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 996 B/s rd, 0 B/s wr, 1.02k op/s; 1321/65322 objects misplaced (2.022%) 2024-02-05T06:39:01.220 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:01.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:01.276 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:01.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:01.377 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:01.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:01.405 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:01.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:01.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:00 smithi167 ceph-mon[32774]: pgmap v234: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.3 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 996 B/s rd, 0 B/s wr, 1.02k op/s; 1321/65322 objects misplaced (2.022%) 2024-02-05T06:39:01.506 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:01.594 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:01.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:01.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:01.696 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:01.731 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:01.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:01.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:01.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:01.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:01.880 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:01.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:01.981 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:02.009 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:02.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:02.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:02.111 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:02.158 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:02.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:02.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:02.241 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:01 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:39:02.241 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:01 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:39:02.255 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:39:02.260 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:02.285 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:02.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:02.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:02.347 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:39:02.348 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:39:02.387 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:02.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:02.454 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:02.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:02.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:01 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:39:02.555 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:02.594 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:02.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:02.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:02.696 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:02.723 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:02.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:02.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:02.825 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:02.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:02.876 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:02.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:02.946 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:39:02 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:39:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:39:02.977 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:02.980 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:39:02 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:39:03.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:03.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:03.124 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:03.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:03.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:03.269 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:03.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:03.315 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:03.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:03.340 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:03 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:39:03.340 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:03 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:39:03.340 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:03 smithi028 ceph-mon[37984]: pgmap v235: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1007 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s; 1011/48897 objects misplaced (2.068%) 2024-02-05T06:39:03.340 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:03 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:39:03.340 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:03 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:39:03.340 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:03 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:39:03.340 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:03 smithi028 ceph-mon[29442]: pgmap v235: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1007 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s; 1011/48897 objects misplaced (2.068%) 2024-02-05T06:39:03.340 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:03 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:39:03.416 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:03.455 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:03.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:03.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:03.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:03 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:39:03.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:03 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:39:03.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:03 smithi167 ceph-mon[32774]: pgmap v235: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1007 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s; 1011/48897 objects misplaced (2.068%) 2024-02-05T06:39:03.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:03 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:39:03.557 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:03.586 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:03.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:03.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:03.687 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:03.717 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:03.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:03.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:03.811 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-05T06:39:03.820 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:03.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:03.848 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:03.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:03.949 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:04.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:04.027 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:04.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:04.128 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:04.177 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:04.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:04.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:04.280 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:04.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:04.309 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:04.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:04.329 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:04 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:39:04.329 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:04 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:39:04.329 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:04 smithi028 ceph-mon[29442]: osdmap e169: 8 total, 8 up, 7 in 2024-02-05T06:39:04.330 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:04 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:39:04.330 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:04 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:39:04.330 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:04 smithi028 ceph-mon[37984]: osdmap e169: 8 total, 8 up, 7 in 2024-02-05T06:39:04.410 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:04.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:04.460 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:04.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:04.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:04 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:39:04.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:04 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:39:04.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:04 smithi167 ceph-mon[32774]: osdmap e169: 8 total, 8 up, 7 in 2024-02-05T06:39:04.561 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:04.606 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:04.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:04.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:04.708 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:04.733 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:04.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:04.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:04.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:04.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:04.887 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:04.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:04.988 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:05.015 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:05.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:05.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:05.117 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:05.158 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:05.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:05.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:05.260 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:05.305 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:05.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:05.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:05.407 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:05.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:05 smithi028 ceph-mon[29442]: pgmap v237: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1007 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 1011/48897 objects misplaced (2.068%) 2024-02-05T06:39:05.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:05 smithi028 ceph-mon[29442]: osdmap e170: 8 total, 8 up, 7 in 2024-02-05T06:39:05.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:05 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/381099398' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:39:05.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:05 smithi028 ceph-mon[29442]: from='client.44939 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:39:05.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:05 smithi028 ceph-mon[37984]: pgmap v237: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1007 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 1011/48897 objects misplaced (2.068%) 2024-02-05T06:39:05.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:05 smithi028 ceph-mon[37984]: osdmap e170: 8 total, 8 up, 7 in 2024-02-05T06:39:05.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:05 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/381099398' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:39:05.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:05 smithi028 ceph-mon[37984]: from='client.44939 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:39:05.457 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:05.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:05.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:05.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:05 smithi167 ceph-mon[32774]: pgmap v237: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1007 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 1011/48897 objects misplaced (2.068%) 2024-02-05T06:39:05.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:05 smithi167 ceph-mon[32774]: osdmap e170: 8 total, 8 up, 7 in 2024-02-05T06:39:05.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:05 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/381099398' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:39:05.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:05 smithi167 ceph-mon[32774]: from='client.44939 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:39:05.559 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:05.599 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:05.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:05.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:05.701 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:05.744 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:05.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:05.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:05.847 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:05.889 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:05.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:05.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:05.991 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:06.014 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-02-05T06:39:06.037 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:06.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:06.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:06.139 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:06.168 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:06.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:06.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:06.270 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:06.311 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:06.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:06.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:06.404 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:06 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:39:06.404 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:06 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:39:06.405 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:06 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:39:06.405 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:06 smithi028 ceph-mon[29442]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:39:06.405 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:06 smithi028 ceph-mon[29442]: from='client.44939 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:39:06.405 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:06 smithi028 ceph-mon[29442]: osdmap e171: 8 total, 8 up, 7 in 2024-02-05T06:39:06.405 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:06 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:39:06.405 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:06 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:39:06.405 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:06 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:39:06.405 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:06 smithi028 ceph-mon[37984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:39:06.405 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:06 smithi028 ceph-mon[37984]: from='client.44939 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:39:06.405 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:06 smithi028 ceph-mon[37984]: osdmap e171: 8 total, 8 up, 7 in 2024-02-05T06:39:06.414 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:06.468 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:06.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:06.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:06.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:06 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:39:06.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:06 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:39:06.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:06 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:39:06.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:06 smithi167 ceph-mon[32774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:39:06.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:06 smithi167 ceph-mon[32774]: from='client.44939 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:39:06.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:06 smithi167 ceph-mon[32774]: osdmap e171: 8 total, 8 up, 7 in 2024-02-05T06:39:06.570 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:06.614 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:06.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:06.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:06.716 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:06.752 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:06.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:06.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:06.854 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:06.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:06.883 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:06.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:06.984 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:07.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:07.011 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:07.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:07.112 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:07.141 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:07.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:07.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:07.243 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:07.269 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:07.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:07.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:07.370 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:07.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:07.411 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:07.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:07.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:07 smithi028 ceph-mon[29442]: pgmap v240: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 903 MiB data, 3.9 GiB used, 622 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.19k op/s; 684/43920 objects misplaced (1.557%) 2024-02-05T06:39:07.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:07 smithi028 ceph-mon[29442]: 6.1f scrub starts 2024-02-05T06:39:07.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:07 smithi028 ceph-mon[29442]: 6.1f scrub ok 2024-02-05T06:39:07.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:07 smithi028 ceph-mon[37984]: pgmap v240: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 903 MiB data, 3.9 GiB used, 622 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.19k op/s; 684/43920 objects misplaced (1.557%) 2024-02-05T06:39:07.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:07 smithi028 ceph-mon[37984]: 6.1f scrub starts 2024-02-05T06:39:07.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:07 smithi028 ceph-mon[37984]: 6.1f scrub ok 2024-02-05T06:39:07.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:07 smithi167 ceph-mon[32774]: pgmap v240: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 903 MiB data, 3.9 GiB used, 622 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.19k op/s; 684/43920 objects misplaced (1.557%) 2024-02-05T06:39:07.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:07 smithi167 ceph-mon[32774]: 6.1f scrub starts 2024-02-05T06:39:07.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:07 smithi167 ceph-mon[32774]: 6.1f scrub ok 2024-02-05T06:39:07.512 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:07.537 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:07.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:07.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:07.639 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:07.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:07.668 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:07.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:07.769 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:07.810 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:07.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:07.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:07.912 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:07.956 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:07.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:07.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:08.058 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:08.086 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:08.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:08.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:08.187 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:08.232 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:08.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:08.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:08.334 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:08.362 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:08.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:08.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:08.464 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:08.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:08 smithi167 ceph-mon[32774]: 6.1d scrub starts 2024-02-05T06:39:08.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:08 smithi167 ceph-mon[32774]: 6.1d scrub ok 2024-02-05T06:39:08.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:08 smithi167 ceph-mon[32774]: 3.6 scrub starts 2024-02-05T06:39:08.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:08 smithi167 ceph-mon[32774]: 3.6 scrub ok 2024-02-05T06:39:08.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:08 smithi167 ceph-mon[32774]: 6.6 scrub starts 2024-02-05T06:39:08.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:08 smithi167 ceph-mon[32774]: 6.6 scrub ok 2024-02-05T06:39:08.490 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:08.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:08.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:08.592 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:08.621 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:08.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:08.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:08.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:08 smithi028 ceph-mon[29442]: 6.1d scrub starts 2024-02-05T06:39:08.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:08 smithi028 ceph-mon[29442]: 6.1d scrub ok 2024-02-05T06:39:08.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:08 smithi028 ceph-mon[29442]: 3.6 scrub starts 2024-02-05T06:39:08.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:08 smithi028 ceph-mon[29442]: 3.6 scrub ok 2024-02-05T06:39:08.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:08 smithi028 ceph-mon[29442]: 6.6 scrub starts 2024-02-05T06:39:08.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:08 smithi028 ceph-mon[29442]: 6.6 scrub ok 2024-02-05T06:39:08.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:08 smithi028 ceph-mon[37984]: 6.1d scrub starts 2024-02-05T06:39:08.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:08 smithi028 ceph-mon[37984]: 6.1d scrub ok 2024-02-05T06:39:08.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:08 smithi028 ceph-mon[37984]: 3.6 scrub starts 2024-02-05T06:39:08.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:08 smithi028 ceph-mon[37984]: 3.6 scrub ok 2024-02-05T06:39:08.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:08 smithi028 ceph-mon[37984]: 6.6 scrub starts 2024-02-05T06:39:08.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:08 smithi028 ceph-mon[37984]: 6.6 scrub ok 2024-02-05T06:39:08.723 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:08.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:08.762 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:08.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:08.863 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:08.909 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:08.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:08.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:09.010 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:09.028 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:39:09.036 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:09.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:09.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:09.041 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:39:09.082 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:39:09.108 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:39:09.126 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:39:09.137 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:09.167 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:39:09.194 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:09.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:09.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:09.208 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:39:09.246 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:39:09.293 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:09 smithi028 ceph-mon[29442]: 3.19 scrub starts 2024-02-05T06:39:09.293 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:09 smithi028 ceph-mon[29442]: 3.19 scrub ok 2024-02-05T06:39:09.294 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:09 smithi028 ceph-mon[29442]: 7.6 scrub starts 2024-02-05T06:39:09.294 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:09 smithi028 ceph-mon[29442]: 7.6 scrub ok 2024-02-05T06:39:09.294 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:09 smithi028 ceph-mon[29442]: pgmap v241: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 682 MiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 866 op/s; 684/33309 objects misplaced (2.053%) 2024-02-05T06:39:09.294 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:09 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1521132240' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:39:09.294 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:39:09.294 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:09 smithi028 ceph-mon[37984]: 3.19 scrub starts 2024-02-05T06:39:09.295 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:09 smithi028 ceph-mon[37984]: 3.19 scrub ok 2024-02-05T06:39:09.295 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:09 smithi028 ceph-mon[37984]: 7.6 scrub starts 2024-02-05T06:39:09.295 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:09 smithi028 ceph-mon[37984]: 7.6 scrub ok 2024-02-05T06:39:09.295 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:09 smithi028 ceph-mon[37984]: pgmap v241: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 682 MiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 866 op/s; 684/33309 objects misplaced (2.053%) 2024-02-05T06:39:09.295 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:09 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1521132240' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:39:09.295 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:09.307 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:39:09.307 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:39:09.308 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:39:09.308 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:39:09.308 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:39:09.308 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:39:09.308 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:39:09.308 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:39:09.324 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:09.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:09.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:09.426 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:09.452 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:09.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:09.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:09.480 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:09 smithi167 ceph-mon[32774]: 3.19 scrub starts 2024-02-05T06:39:09.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:09 smithi167 ceph-mon[32774]: 3.19 scrub ok 2024-02-05T06:39:09.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:09 smithi167 ceph-mon[32774]: 7.6 scrub starts 2024-02-05T06:39:09.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:09 smithi167 ceph-mon[32774]: 7.6 scrub ok 2024-02-05T06:39:09.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:09 smithi167 ceph-mon[32774]: pgmap v241: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 682 MiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 866 op/s; 684/33309 objects misplaced (2.053%) 2024-02-05T06:39:09.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:09 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1521132240' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:39:09.554 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:09.582 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:09.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:09.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:09.684 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:09.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:09.723 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:09.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:09.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:09.864 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:09.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:09.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:09.966 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:09.992 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:09.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:09.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:10.094 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:10.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:10.125 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:10.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:10.226 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:10.267 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:10 smithi028 ceph-mon[29442]: 3.1d scrub starts 2024-02-05T06:39:10.267 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:10 smithi028 ceph-mon[29442]: 3.1d scrub ok 2024-02-05T06:39:10.267 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:10 smithi028 ceph-mon[29442]: 7.8 scrub starts 2024-02-05T06:39:10.268 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:10 smithi028 ceph-mon[29442]: 1.0 scrub starts 2024-02-05T06:39:10.268 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:10 smithi028 ceph-mon[29442]: 1.0 scrub ok 2024-02-05T06:39:10.268 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:10 smithi028 ceph-mon[29442]: 4.1b scrub starts 2024-02-05T06:39:10.268 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:10 smithi028 ceph-mon[29442]: 4.1b scrub ok 2024-02-05T06:39:10.268 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:10 smithi028 ceph-mon[29442]: 7.8 scrub ok 2024-02-05T06:39:10.268 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:10 smithi028 ceph-mon[29442]: 4.5 scrub starts 2024-02-05T06:39:10.268 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:10 smithi028 ceph-mon[29442]: 4.5 scrub ok 2024-02-05T06:39:10.268 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:10 smithi028 ceph-mon[29442]: osdmap e172: 8 total, 8 up, 7 in 2024-02-05T06:39:10.268 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:10 smithi028 ceph-mon[37984]: 3.1d scrub starts 2024-02-05T06:39:10.268 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:10 smithi028 ceph-mon[37984]: 3.1d scrub ok 2024-02-05T06:39:10.268 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:10 smithi028 ceph-mon[37984]: 7.8 scrub starts 2024-02-05T06:39:10.268 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:10 smithi028 ceph-mon[37984]: 1.0 scrub starts 2024-02-05T06:39:10.269 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:10 smithi028 ceph-mon[37984]: 1.0 scrub ok 2024-02-05T06:39:10.269 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:10 smithi028 ceph-mon[37984]: 4.1b scrub starts 2024-02-05T06:39:10.269 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:10 smithi028 ceph-mon[37984]: 4.1b scrub ok 2024-02-05T06:39:10.269 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:10 smithi028 ceph-mon[37984]: 7.8 scrub ok 2024-02-05T06:39:10.269 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:10 smithi028 ceph-mon[37984]: 4.5 scrub starts 2024-02-05T06:39:10.269 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:10 smithi028 ceph-mon[37984]: 4.5 scrub ok 2024-02-05T06:39:10.269 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:10 smithi028 ceph-mon[37984]: osdmap e172: 8 total, 8 up, 7 in 2024-02-05T06:39:10.269 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:10.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:10.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:10.371 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:10.399 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:10.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:10.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:10.434 INFO:teuthology.orchestra.run.smithi028.stderr:no valid command found; 10 closest matches: 2024-02-05T06:39:10.434 INFO:teuthology.orchestra.run.smithi028.stderr:0 2024-02-05T06:39:10.434 INFO:teuthology.orchestra.run.smithi028.stderr:1 2024-02-05T06:39:10.434 INFO:teuthology.orchestra.run.smithi028.stderr:2 2024-02-05T06:39:10.434 INFO:teuthology.orchestra.run.smithi028.stderr:abort 2024-02-05T06:39:10.434 INFO:teuthology.orchestra.run.smithi028.stderr:assert 2024-02-05T06:39:10.434 INFO:teuthology.orchestra.run.smithi028.stderr:bench [] [] [] [] 2024-02-05T06:39:10.435 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs debug_inject_read_zeros 2024-02-05T06:39:10.435 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs files list 2024-02-05T06:39:10.435 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs stats 2024-02-05T06:39:10.435 INFO:teuthology.orchestra.run.smithi028.stderr:bluestore allocator dump block 2024-02-05T06:39:10.435 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: invalid command 2024-02-05T06:39:10.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:10 smithi167 ceph-mon[32774]: 3.1d scrub starts 2024-02-05T06:39:10.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:10 smithi167 ceph-mon[32774]: 3.1d scrub ok 2024-02-05T06:39:10.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:10 smithi167 ceph-mon[32774]: 7.8 scrub starts 2024-02-05T06:39:10.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:10 smithi167 ceph-mon[32774]: 1.0 scrub starts 2024-02-05T06:39:10.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:10 smithi167 ceph-mon[32774]: 1.0 scrub ok 2024-02-05T06:39:10.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:10 smithi167 ceph-mon[32774]: 4.1b scrub starts 2024-02-05T06:39:10.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:10 smithi167 ceph-mon[32774]: 4.1b scrub ok 2024-02-05T06:39:10.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:10 smithi167 ceph-mon[32774]: 7.8 scrub ok 2024-02-05T06:39:10.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:10 smithi167 ceph-mon[32774]: 4.5 scrub starts 2024-02-05T06:39:10.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:10 smithi167 ceph-mon[32774]: 4.5 scrub ok 2024-02-05T06:39:10.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:10 smithi167 ceph-mon[32774]: osdmap e172: 8 total, 8 up, 7 in 2024-02-05T06:39:10.501 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:10.544 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:10.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:10.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:10.646 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:10.687 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:10.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:10.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:10.789 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:10.816 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:10.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:10.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:10.918 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:10.945 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:10.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:10.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:10.969 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T06:39:11.047 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:11.073 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:11.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:11.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:11.175 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:11.202 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:11.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:11.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:11.210 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-05T06:39:11.211 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-05T06:39:11.211 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph tell osd.5 'trim stale osdmaps' 2024-02-05T06:39:11.273 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:11 smithi028 ceph-mon[29442]: 5.c scrub starts 2024-02-05T06:39:11.273 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:11 smithi028 ceph-mon[29442]: 5.c scrub ok 2024-02-05T06:39:11.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:11 smithi028 ceph-mon[29442]: 5.15 scrub starts 2024-02-05T06:39:11.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:11 smithi028 ceph-mon[29442]: 5.15 scrub ok 2024-02-05T06:39:11.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:11 smithi028 ceph-mon[29442]: 1.2 scrub starts 2024-02-05T06:39:11.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:11 smithi028 ceph-mon[29442]: 1.2 scrub ok 2024-02-05T06:39:11.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:11 smithi028 ceph-mon[29442]: 4.9 scrub starts 2024-02-05T06:39:11.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:11 smithi028 ceph-mon[29442]: 4.9 scrub ok 2024-02-05T06:39:11.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:11 smithi028 ceph-mon[29442]: pgmap v243: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 682 MiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 866 op/s; 684/33309 objects misplaced (2.053%) 2024-02-05T06:39:11.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:11 smithi028 ceph-mon[29442]: 5.1f scrub starts 2024-02-05T06:39:11.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:11 smithi028 ceph-mon[29442]: 5.1f scrub ok 2024-02-05T06:39:11.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:11 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:39:11.274 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:11 smithi028 ceph-mon[37984]: 5.c scrub starts 2024-02-05T06:39:11.274 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:11 smithi028 ceph-mon[37984]: 5.c scrub ok 2024-02-05T06:39:11.275 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:11 smithi028 ceph-mon[37984]: 5.15 scrub starts 2024-02-05T06:39:11.275 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:11 smithi028 ceph-mon[37984]: 5.15 scrub ok 2024-02-05T06:39:11.275 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:11 smithi028 ceph-mon[37984]: 1.2 scrub starts 2024-02-05T06:39:11.275 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:11 smithi028 ceph-mon[37984]: 1.2 scrub ok 2024-02-05T06:39:11.275 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:11 smithi028 ceph-mon[37984]: 4.9 scrub starts 2024-02-05T06:39:11.275 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:11 smithi028 ceph-mon[37984]: 4.9 scrub ok 2024-02-05T06:39:11.275 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:11 smithi028 ceph-mon[37984]: pgmap v243: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 682 MiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 866 op/s; 684/33309 objects misplaced (2.053%) 2024-02-05T06:39:11.275 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:11 smithi028 ceph-mon[37984]: 5.1f scrub starts 2024-02-05T06:39:11.275 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:11 smithi028 ceph-mon[37984]: 5.1f scrub ok 2024-02-05T06:39:11.275 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:11 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:39:11.304 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:11.329 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:11.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:11.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:11.431 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:11.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:11 smithi167 ceph-mon[32774]: 5.c scrub starts 2024-02-05T06:39:11.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:11 smithi167 ceph-mon[32774]: 5.c scrub ok 2024-02-05T06:39:11.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:11 smithi167 ceph-mon[32774]: 5.15 scrub starts 2024-02-05T06:39:11.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:11 smithi167 ceph-mon[32774]: 5.15 scrub ok 2024-02-05T06:39:11.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:11 smithi167 ceph-mon[32774]: 1.2 scrub starts 2024-02-05T06:39:11.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:11 smithi167 ceph-mon[32774]: 1.2 scrub ok 2024-02-05T06:39:11.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:11 smithi167 ceph-mon[32774]: 4.9 scrub starts 2024-02-05T06:39:11.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:11 smithi167 ceph-mon[32774]: 4.9 scrub ok 2024-02-05T06:39:11.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:11 smithi167 ceph-mon[32774]: pgmap v243: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 682 MiB data, 3.3 GiB used, 623 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 866 op/s; 684/33309 objects misplaced (2.053%) 2024-02-05T06:39:11.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:11 smithi167 ceph-mon[32774]: 5.1f scrub starts 2024-02-05T06:39:11.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:11 smithi167 ceph-mon[32774]: 5.1f scrub ok 2024-02-05T06:39:11.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:11 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:39:11.486 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:11.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:11.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:11.589 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:11.643 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:11.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:11.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:11.745 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:11.774 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:11.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:11.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:11.876 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:11.920 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:11.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:11.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:12.022 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:12.062 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:12.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:12.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:12.164 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:12.192 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:12.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:12.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:12.268 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:39:12.294 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:12.381 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:12.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:12.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:12.483 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:12.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[29442]: 6.4 scrub starts 2024-02-05T06:39:12.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[29442]: 6.4 scrub ok 2024-02-05T06:39:12.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[29442]: 6.1e scrub starts 2024-02-05T06:39:12.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[29442]: 6.1e scrub ok 2024-02-05T06:39:12.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[29442]: 4.11 scrub starts 2024-02-05T06:39:12.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[29442]: 4.11 scrub ok 2024-02-05T06:39:12.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [4, 2]}]: dispatch 2024-02-05T06:39:12.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-02-05T06:39:12.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [4, 2]}]: dispatch 2024-02-05T06:39:12.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 0]}]: dispatch 2024-02-05T06:39:12.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 0]}]: dispatch 2024-02-05T06:39:12.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [0, 3]}]: dispatch 2024-02-05T06:39:12.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-02-05T06:39:12.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 0]}]: dispatch 2024-02-05T06:39:12.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 0]}]: dispatch 2024-02-05T06:39:12.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [0, 3]}]: dispatch 2024-02-05T06:39:12.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [0, 2]}]: dispatch 2024-02-05T06:39:12.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [0, 2]}]: dispatch 2024-02-05T06:39:12.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1c", "id": [4, 6]}]: dispatch 2024-02-05T06:39:12.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1c", "id": [4, 6]}]: dispatch 2024-02-05T06:39:12.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.13", "id": [3, 1]}]: dispatch 2024-02-05T06:39:12.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.13", "id": [3, 1]}]: dispatch 2024-02-05T06:39:12.502 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[37984]: 6.4 scrub starts 2024-02-05T06:39:12.502 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[37984]: 6.4 scrub ok 2024-02-05T06:39:12.502 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[37984]: 6.1e scrub starts 2024-02-05T06:39:12.502 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[37984]: 6.1e scrub ok 2024-02-05T06:39:12.502 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[37984]: 4.11 scrub starts 2024-02-05T06:39:12.502 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[37984]: 4.11 scrub ok 2024-02-05T06:39:12.502 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [4, 2]}]: dispatch 2024-02-05T06:39:12.502 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-02-05T06:39:12.502 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [4, 2]}]: dispatch 2024-02-05T06:39:12.502 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 0]}]: dispatch 2024-02-05T06:39:12.502 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 0]}]: dispatch 2024-02-05T06:39:12.502 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [0, 3]}]: dispatch 2024-02-05T06:39:12.502 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-02-05T06:39:12.502 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 0]}]: dispatch 2024-02-05T06:39:12.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 0]}]: dispatch 2024-02-05T06:39:12.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [0, 3]}]: dispatch 2024-02-05T06:39:12.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [0, 2]}]: dispatch 2024-02-05T06:39:12.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [0, 2]}]: dispatch 2024-02-05T06:39:12.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1c", "id": [4, 6]}]: dispatch 2024-02-05T06:39:12.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1c", "id": [4, 6]}]: dispatch 2024-02-05T06:39:12.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.13", "id": [3, 1]}]: dispatch 2024-02-05T06:39:12.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.13", "id": [3, 1]}]: dispatch 2024-02-05T06:39:12.511 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:12.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:12.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:12.613 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:12.635 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:12 smithi167 ceph-mon[32774]: 6.4 scrub starts 2024-02-05T06:39:12.635 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:12 smithi167 ceph-mon[32774]: 6.4 scrub ok 2024-02-05T06:39:12.635 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:12 smithi167 ceph-mon[32774]: 6.1e scrub starts 2024-02-05T06:39:12.635 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:12 smithi167 ceph-mon[32774]: 6.1e scrub ok 2024-02-05T06:39:12.635 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:12 smithi167 ceph-mon[32774]: 4.11 scrub starts 2024-02-05T06:39:12.635 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:12 smithi167 ceph-mon[32774]: 4.11 scrub ok 2024-02-05T06:39:12.635 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:12 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [4, 2]}]: dispatch 2024-02-05T06:39:12.636 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:12 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-02-05T06:39:12.636 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:12 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [4, 2]}]: dispatch 2024-02-05T06:39:12.636 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:12 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 0]}]: dispatch 2024-02-05T06:39:12.636 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:12 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 0]}]: dispatch 2024-02-05T06:39:12.636 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:12 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [0, 3]}]: dispatch 2024-02-05T06:39:12.636 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:12 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-02-05T06:39:12.636 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:12 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 0]}]: dispatch 2024-02-05T06:39:12.636 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:12 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 0]}]: dispatch 2024-02-05T06:39:12.636 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:12 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [0, 3]}]: dispatch 2024-02-05T06:39:12.636 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:12 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [0, 2]}]: dispatch 2024-02-05T06:39:12.636 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:12 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [0, 2]}]: dispatch 2024-02-05T06:39:12.637 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:12 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1c", "id": [4, 6]}]: dispatch 2024-02-05T06:39:12.637 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:12 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1c", "id": [4, 6]}]: dispatch 2024-02-05T06:39:12.637 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:12 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.13", "id": [3, 1]}]: dispatch 2024-02-05T06:39:12.637 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:12 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.13", "id": [3, 1]}]: dispatch 2024-02-05T06:39:12.640 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:12.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:12.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:12.742 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:12.769 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:12.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:12.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:12.871 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:12.915 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:12.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:12.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:12.948 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:39:12 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:39:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:39:12.981 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:39:12 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:39:13.017 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:13.046 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:13.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:13.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:13.148 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:13.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:13.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:13.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:13.313 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:13.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:13.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:13.415 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:13.439 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:13.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:13.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:13.540 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:13.567 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:13.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:13.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:13.668 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:13.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[37984]: 4.f scrub starts 2024-02-05T06:39:13.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[37984]: 4.f scrub ok 2024-02-05T06:39:13.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[37984]: 6.d scrub starts 2024-02-05T06:39:13.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[37984]: 6.d scrub ok 2024-02-05T06:39:13.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[37984]: 5.18 scrub starts 2024-02-05T06:39:13.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[37984]: 5.18 scrub ok 2024-02-05T06:39:13.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[37984]: 4.1c scrub starts 2024-02-05T06:39:13.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[37984]: 4.1c scrub ok 2024-02-05T06:39:13.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [4, 2]}]': finished 2024-02-05T06:39:13.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-02-05T06:39:13.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 0]}]': finished 2024-02-05T06:39:13.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 0]}]': finished 2024-02-05T06:39:13.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [0, 3]}]': finished 2024-02-05T06:39:13.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [0, 2]}]': finished 2024-02-05T06:39:13.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1c", "id": [4, 6]}]': finished 2024-02-05T06:39:13.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.13", "id": [3, 1]}]': finished 2024-02-05T06:39:13.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[37984]: osdmap e173: 8 total, 8 up, 7 in 2024-02-05T06:39:13.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[37984]: pgmap v245: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 342 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 1.69k op/s; 362/17004 objects misplaced (2.129%) 2024-02-05T06:39:13.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[37984]: 4.0 scrub starts 2024-02-05T06:39:13.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[37984]: 4.0 scrub ok 2024-02-05T06:39:13.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[37984]: 1.6 scrub starts 2024-02-05T06:39:13.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[37984]: 1.6 scrub ok 2024-02-05T06:39:13.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:39:13.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[37984]: osdmap e174: 8 total, 8 up, 7 in 2024-02-05T06:39:13.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[29442]: 4.f scrub starts 2024-02-05T06:39:13.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[29442]: 4.f scrub ok 2024-02-05T06:39:13.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[29442]: 6.d scrub starts 2024-02-05T06:39:13.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[29442]: 6.d scrub ok 2024-02-05T06:39:13.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[29442]: 5.18 scrub starts 2024-02-05T06:39:13.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[29442]: 5.18 scrub ok 2024-02-05T06:39:13.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[29442]: 4.1c scrub starts 2024-02-05T06:39:13.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[29442]: 4.1c scrub ok 2024-02-05T06:39:13.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [4, 2]}]': finished 2024-02-05T06:39:13.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-02-05T06:39:13.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 0]}]': finished 2024-02-05T06:39:13.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 0]}]': finished 2024-02-05T06:39:13.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [0, 3]}]': finished 2024-02-05T06:39:13.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [0, 2]}]': finished 2024-02-05T06:39:13.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1c", "id": [4, 6]}]': finished 2024-02-05T06:39:13.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.13", "id": [3, 1]}]': finished 2024-02-05T06:39:13.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[29442]: osdmap e173: 8 total, 8 up, 7 in 2024-02-05T06:39:13.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[29442]: pgmap v245: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 342 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 1.69k op/s; 362/17004 objects misplaced (2.129%) 2024-02-05T06:39:13.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[29442]: 4.0 scrub starts 2024-02-05T06:39:13.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[29442]: 4.0 scrub ok 2024-02-05T06:39:13.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[29442]: 1.6 scrub starts 2024-02-05T06:39:13.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[29442]: 1.6 scrub ok 2024-02-05T06:39:13.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:39:13.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:13 smithi028 ceph-mon[29442]: osdmap e174: 8 total, 8 up, 7 in 2024-02-05T06:39:13.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:13.701 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:13.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:13.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:13 smithi167 ceph-mon[32774]: 4.f scrub starts 2024-02-05T06:39:13.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:13 smithi167 ceph-mon[32774]: 4.f scrub ok 2024-02-05T06:39:13.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:13 smithi167 ceph-mon[32774]: 6.d scrub starts 2024-02-05T06:39:13.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:13 smithi167 ceph-mon[32774]: 6.d scrub ok 2024-02-05T06:39:13.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:13 smithi167 ceph-mon[32774]: 5.18 scrub starts 2024-02-05T06:39:13.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:13 smithi167 ceph-mon[32774]: 5.18 scrub ok 2024-02-05T06:39:13.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:13 smithi167 ceph-mon[32774]: 4.1c scrub starts 2024-02-05T06:39:13.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:13 smithi167 ceph-mon[32774]: 4.1c scrub ok 2024-02-05T06:39:13.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:13 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [4, 2]}]': finished 2024-02-05T06:39:13.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:13 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-02-05T06:39:13.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:13 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 0]}]': finished 2024-02-05T06:39:13.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:13 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 0]}]': finished 2024-02-05T06:39:13.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:13 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [0, 3]}]': finished 2024-02-05T06:39:13.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:13 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [0, 2]}]': finished 2024-02-05T06:39:13.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:13 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1c", "id": [4, 6]}]': finished 2024-02-05T06:39:13.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:13 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.13", "id": [3, 1]}]': finished 2024-02-05T06:39:13.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:13 smithi167 ceph-mon[32774]: osdmap e173: 8 total, 8 up, 7 in 2024-02-05T06:39:13.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:13 smithi167 ceph-mon[32774]: pgmap v245: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 342 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 1.69k op/s; 362/17004 objects misplaced (2.129%) 2024-02-05T06:39:13.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:13 smithi167 ceph-mon[32774]: 4.0 scrub starts 2024-02-05T06:39:13.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:13 smithi167 ceph-mon[32774]: 4.0 scrub ok 2024-02-05T06:39:13.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:13 smithi167 ceph-mon[32774]: 1.6 scrub starts 2024-02-05T06:39:13.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:13 smithi167 ceph-mon[32774]: 1.6 scrub ok 2024-02-05T06:39:13.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:13 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:39:13.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:13 smithi167 ceph-mon[32774]: osdmap e174: 8 total, 8 up, 7 in 2024-02-05T06:39:13.803 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:13.831 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:13.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:13.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:13.932 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:13.972 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:13.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:13.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:14.002 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:39:14.069 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:39:14.073 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:14.101 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:14.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:14.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:14.203 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:14.220 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:39:14.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:14.261 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:14.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:14.362 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:14.391 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:14.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:14.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:14.493 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:14.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:14.520 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:14.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:14.621 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:14.663 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:14.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:14.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:14.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:14.807 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:14.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:14.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:14.909 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:14.935 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:14.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:14.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:15.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:15.083 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:15.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:15.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:15.184 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:15.211 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:15.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:15.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:15.313 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:15.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:15.358 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:15.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:15.459 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:15.501 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:15.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:15.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:15.603 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:15.630 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:15.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:15.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:15.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:15 smithi028 ceph-mon[29442]: pgmap v247: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 342 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 899 op/s; 362/17004 objects misplaced (2.129%) 2024-02-05T06:39:15.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:15 smithi028 ceph-mon[29442]: 5.1c scrub starts 2024-02-05T06:39:15.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:15 smithi028 ceph-mon[29442]: 5.1c scrub ok 2024-02-05T06:39:15.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:15 smithi028 ceph-mon[37984]: pgmap v247: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 342 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 899 op/s; 362/17004 objects misplaced (2.129%) 2024-02-05T06:39:15.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:15 smithi028 ceph-mon[37984]: 5.1c scrub starts 2024-02-05T06:39:15.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:15 smithi028 ceph-mon[37984]: 5.1c scrub ok 2024-02-05T06:39:15.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:15 smithi167 ceph-mon[32774]: pgmap v247: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 342 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 0 B/s wr, 899 op/s; 362/17004 objects misplaced (2.129%) 2024-02-05T06:39:15.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:15 smithi167 ceph-mon[32774]: 5.1c scrub starts 2024-02-05T06:39:15.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:15 smithi167 ceph-mon[32774]: 5.1c scrub ok 2024-02-05T06:39:15.732 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:15.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:15.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:15.875 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:15.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:15.922 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:15.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:16.023 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:16.049 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:16.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:16.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:16.151 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:16.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:16.188 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:16.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:16.289 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:16.317 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:16.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:16.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:16.419 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:16.458 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:16.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:16.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:16.560 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:16.588 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:16.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:16.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:16.622 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Removed 115117 objects 2024-02-05T06:39:16.622 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Clean up completed and total clean up time :129.384 2024-02-05T06:39:16.685 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:16 smithi028 ceph-mon[37984]: 4.18 scrub starts 2024-02-05T06:39:16.685 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:16 smithi028 ceph-mon[37984]: 4.18 scrub ok 2024-02-05T06:39:16.685 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:16 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:39:16.685 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:16 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:39:16.685 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:16 smithi028 ceph-mon[37984]: 5.19 scrub starts 2024-02-05T06:39:16.685 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:16 smithi028 ceph-mon[37984]: 5.19 scrub ok 2024-02-05T06:39:16.685 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:16 smithi028 ceph-mon[37984]: 5.11 scrub starts 2024-02-05T06:39:16.685 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:16 smithi028 ceph-mon[37984]: 5.11 scrub ok 2024-02-05T06:39:16.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:16 smithi028 ceph-mon[29442]: 4.18 scrub starts 2024-02-05T06:39:16.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:16 smithi028 ceph-mon[29442]: 4.18 scrub ok 2024-02-05T06:39:16.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:16 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:39:16.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:16 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:39:16.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:16 smithi028 ceph-mon[29442]: 5.19 scrub starts 2024-02-05T06:39:16.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:16 smithi028 ceph-mon[29442]: 5.19 scrub ok 2024-02-05T06:39:16.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:16 smithi028 ceph-mon[29442]: 5.11 scrub starts 2024-02-05T06:39:16.686 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:16 smithi028 ceph-mon[29442]: 5.11 scrub ok 2024-02-05T06:39:16.689 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:16.713 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:16.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:16.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:16.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:16 smithi167 ceph-mon[32774]: 4.18 scrub starts 2024-02-05T06:39:16.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:16 smithi167 ceph-mon[32774]: 4.18 scrub ok 2024-02-05T06:39:16.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:16 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:39:16.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:16 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:39:16.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:16 smithi167 ceph-mon[32774]: 5.19 scrub starts 2024-02-05T06:39:16.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:16 smithi167 ceph-mon[32774]: 5.19 scrub ok 2024-02-05T06:39:16.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:16 smithi167 ceph-mon[32774]: 5.11 scrub starts 2024-02-05T06:39:16.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:16 smithi167 ceph-mon[32774]: 5.11 scrub ok 2024-02-05T06:39:16.795 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-02-05T06:39:16.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-02-05T06:39:16.814 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:16.861 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:16.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:16.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:16.963 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:16.988 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:16.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:16.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:17.090 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:17.114 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:17.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:17.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:17.216 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:17.244 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:17.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:17.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:17.346 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:17.374 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:17.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:17.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:17.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:17.500 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:17.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:17.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:17.602 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:17.625 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:17.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:17.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:17.683 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:17 smithi028 ceph-mon[37984]: pgmap v248: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 240 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 1.12k op/s; 28/12075 objects misplaced (0.232%) 2024-02-05T06:39:17.684 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:17 smithi028 ceph-mon[37984]: osdmap e175: 8 total, 8 up, 7 in 2024-02-05T06:39:17.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:17 smithi028 ceph-mon[29442]: pgmap v248: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 240 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 1.12k op/s; 28/12075 objects misplaced (0.232%) 2024-02-05T06:39:17.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:17 smithi028 ceph-mon[29442]: osdmap e175: 8 total, 8 up, 7 in 2024-02-05T06:39:17.727 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:17.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:17 smithi167 ceph-mon[32774]: pgmap v248: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 240 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 1.12k op/s; 28/12075 objects misplaced (0.232%) 2024-02-05T06:39:17.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:17 smithi167 ceph-mon[32774]: osdmap e175: 8 total, 8 up, 7 in 2024-02-05T06:39:17.756 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:17.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:17.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:17.858 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:17.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:17.886 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:17.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:17.987 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:18.010 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:18.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:18.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:18.044 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:39:18.044 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-02-05T06:39:18.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:18.045 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14847.0:230015 7.2 e175/96)", 2024-02-05T06:39:18.045 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:39:16.416999+0000", 2024-02-05T06:39:18.045 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.00011299200000000001, 2024-02-05T06:39:18.045 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014259900000000001, 2024-02-05T06:39:18.045 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:39:18.045 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-02-05T06:39:18.045 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:39:18.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:18.045 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:39:18.045 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:39:16.416999+0000", 2024-02-05T06:39:18.045 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:18.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:18.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:18.046 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:39:18.046 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:39:16.416999+0000", 2024-02-05T06:39:18.046 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:18.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:18.046 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:18.046 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:39:18.046 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:39:16.417001+0000", 2024-02-05T06:39:18.046 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.764e-06 2024-02-05T06:39:18.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:18.046 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:18.046 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:39:18.046 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:39:16.417011+0000", 2024-02-05T06:39:18.046 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0368e-05 2024-02-05T06:39:18.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:18.046 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:18.047 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:39:18.047 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:39:16.417030+0000", 2024-02-05T06:39:18.047 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9321999999999999e-05 2024-02-05T06:39:18.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:18.047 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:18.047 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:39:18.047 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:39:16.417040+0000", 2024-02-05T06:39:18.047 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2310000000000002e-06 2024-02-05T06:39:18.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:18.047 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:18.047 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:39:18.047 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:39:16.417114+0000", 2024-02-05T06:39:18.047 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4334999999999999e-05 2024-02-05T06:39:18.047 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:18.047 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:39:18.047 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:18.048 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:18.048 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:39:18.048 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 1 2024-02-05T06:39:18.048 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:39:18.113 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:18.141 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:18.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:18.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:18.243 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:18.267 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:18.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:18.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:18.368 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:18.397 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:18.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:18.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:18.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:18 smithi028 ceph-mon[37984]: 4.15 scrub starts 2024-02-05T06:39:18.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:18 smithi028 ceph-mon[37984]: 4.15 scrub ok 2024-02-05T06:39:18.473 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:18 smithi028 ceph-mon[37984]: 4.c scrub starts 2024-02-05T06:39:18.474 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:18 smithi028 ceph-mon[37984]: 4.c scrub ok 2024-02-05T06:39:18.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:18 smithi028 ceph-mon[29442]: 4.15 scrub starts 2024-02-05T06:39:18.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:18 smithi028 ceph-mon[29442]: 4.15 scrub ok 2024-02-05T06:39:18.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:18 smithi028 ceph-mon[29442]: 4.c scrub starts 2024-02-05T06:39:18.474 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:18 smithi028 ceph-mon[29442]: 4.c scrub ok 2024-02-05T06:39:18.499 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:18.523 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:18.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:18.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:18.625 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:18.648 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:18.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:18.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:18.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:18 smithi167 ceph-mon[32774]: 4.15 scrub starts 2024-02-05T06:39:18.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:18 smithi167 ceph-mon[32774]: 4.15 scrub ok 2024-02-05T06:39:18.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:18 smithi167 ceph-mon[32774]: 4.c scrub starts 2024-02-05T06:39:18.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:18 smithi167 ceph-mon[32774]: 4.c scrub ok 2024-02-05T06:39:18.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-05T06:39:18.750 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:18.778 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:18.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:18.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:18.880 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:18.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:18.909 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:18.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:19.010 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:19.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:19.038 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:19.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:19.140 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:19.165 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:19.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:19.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:19.268 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:19.296 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:19.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:19.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:19.315 INFO:teuthology.orchestra.run.smithi028.stderr:no valid command found; 10 closest matches: 2024-02-05T06:39:19.315 INFO:teuthology.orchestra.run.smithi028.stderr:0 2024-02-05T06:39:19.315 INFO:teuthology.orchestra.run.smithi028.stderr:1 2024-02-05T06:39:19.315 INFO:teuthology.orchestra.run.smithi028.stderr:2 2024-02-05T06:39:19.315 INFO:teuthology.orchestra.run.smithi028.stderr:abort 2024-02-05T06:39:19.315 INFO:teuthology.orchestra.run.smithi028.stderr:assert 2024-02-05T06:39:19.315 INFO:teuthology.orchestra.run.smithi028.stderr:bench [] [] [] [] 2024-02-05T06:39:19.316 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs debug_inject_read_zeros 2024-02-05T06:39:19.316 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs files list 2024-02-05T06:39:19.316 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs stats 2024-02-05T06:39:19.316 INFO:teuthology.orchestra.run.smithi028.stderr:bluestore allocator dump block 2024-02-05T06:39:19.316 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: invalid command 2024-02-05T06:39:19.349 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:19 smithi028 ceph-mon[37984]: pgmap v250: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 13 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 848 B/s rd, 0 B/s wr, 874 op/s; 28/1200 objects misplaced (2.333%); 54 B/s, 2 objects/s recovering 2024-02-05T06:39:19.349 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:19 smithi028 ceph-mon[37984]: osdmap e176: 8 total, 8 up, 7 in 2024-02-05T06:39:19.349 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:19 smithi028 ceph-mon[37984]: 6.c scrub starts 2024-02-05T06:39:19.350 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:19 smithi028 ceph-mon[37984]: 6.c scrub ok 2024-02-05T06:39:19.350 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:19 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/606417600' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:39:19.350 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:19 smithi028 ceph-mon[29442]: pgmap v250: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 13 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 848 B/s rd, 0 B/s wr, 874 op/s; 28/1200 objects misplaced (2.333%); 54 B/s, 2 objects/s recovering 2024-02-05T06:39:19.350 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:19 smithi028 ceph-mon[29442]: osdmap e176: 8 total, 8 up, 7 in 2024-02-05T06:39:19.350 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:19 smithi028 ceph-mon[29442]: 6.c scrub starts 2024-02-05T06:39:19.350 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:19 smithi028 ceph-mon[29442]: 6.c scrub ok 2024-02-05T06:39:19.350 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:19 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/606417600' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:39:19.375 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:39:19.375 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:39:19.375 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:39:19.375 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:39:19.375 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:39:19.375 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:39:19.375 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:39:19.375 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:39:19.376 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:39:19.376 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:39:19.376 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:39:19.376 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:39:19.376 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:39:19.376 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:39:19.376 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:39:19.376 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:39:19.382 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:39:19.398 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:19.422 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:19.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:19.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:19.528 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:19.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:19.553 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:19.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:19.654 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:19.681 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:19.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:19.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:19.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:19 smithi167 ceph-mon[32774]: pgmap v250: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 13 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 848 B/s rd, 0 B/s wr, 874 op/s; 28/1200 objects misplaced (2.333%); 54 B/s, 2 objects/s recovering 2024-02-05T06:39:19.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:19 smithi167 ceph-mon[32774]: osdmap e176: 8 total, 8 up, 7 in 2024-02-05T06:39:19.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:19 smithi167 ceph-mon[32774]: 6.c scrub starts 2024-02-05T06:39:19.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:19 smithi167 ceph-mon[32774]: 6.c scrub ok 2024-02-05T06:39:19.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:19 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/606417600' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:39:19.782 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:19.807 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:19.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:19.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:19.908 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:19.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:19.937 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:19.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:20.038 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:20.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:20.062 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:20.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:20.163 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:20.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:20.188 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:20.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:20.289 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:20.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:20.317 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:20.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:20.418 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:20.442 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:20.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:20.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:20.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:20 smithi028 ceph-mon[29442]: 5.f scrub starts 2024-02-05T06:39:20.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:20 smithi028 ceph-mon[29442]: 5.f scrub ok 2024-02-05T06:39:20.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:20 smithi028 ceph-mon[29442]: 6.17 scrub starts 2024-02-05T06:39:20.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:20 smithi028 ceph-mon[29442]: 6.17 scrub ok 2024-02-05T06:39:20.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:20 smithi028 ceph-mon[29442]: 6.14 scrub starts 2024-02-05T06:39:20.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:20 smithi028 ceph-mon[29442]: 6.14 scrub ok 2024-02-05T06:39:20.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:20 smithi028 ceph-mon[29442]: osdmap e177: 8 total, 8 up, 7 in 2024-02-05T06:39:20.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:20 smithi028 ceph-mon[29442]: 7.c scrub starts 2024-02-05T06:39:20.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:20 smithi028 ceph-mon[29442]: 7.c scrub ok 2024-02-05T06:39:20.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:20 smithi028 ceph-mon[29442]: osdmap e178: 8 total, 8 up, 7 in 2024-02-05T06:39:20.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:20 smithi028 ceph-mon[37984]: 5.f scrub starts 2024-02-05T06:39:20.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:20 smithi028 ceph-mon[37984]: 5.f scrub ok 2024-02-05T06:39:20.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:20 smithi028 ceph-mon[37984]: 6.17 scrub starts 2024-02-05T06:39:20.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:20 smithi028 ceph-mon[37984]: 6.17 scrub ok 2024-02-05T06:39:20.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:20 smithi028 ceph-mon[37984]: 6.14 scrub starts 2024-02-05T06:39:20.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:20 smithi028 ceph-mon[37984]: 6.14 scrub ok 2024-02-05T06:39:20.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:20 smithi028 ceph-mon[37984]: osdmap e177: 8 total, 8 up, 7 in 2024-02-05T06:39:20.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:20 smithi028 ceph-mon[37984]: 7.c scrub starts 2024-02-05T06:39:20.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:20 smithi028 ceph-mon[37984]: 7.c scrub ok 2024-02-05T06:39:20.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:20 smithi028 ceph-mon[37984]: osdmap e178: 8 total, 8 up, 7 in 2024-02-05T06:39:20.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:20.571 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:20.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:20.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:20.673 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:20.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:20 smithi167 ceph-mon[32774]: 5.f scrub starts 2024-02-05T06:39:20.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:20 smithi167 ceph-mon[32774]: 5.f scrub ok 2024-02-05T06:39:20.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:20 smithi167 ceph-mon[32774]: 6.17 scrub starts 2024-02-05T06:39:20.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:20 smithi167 ceph-mon[32774]: 6.17 scrub ok 2024-02-05T06:39:20.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:20 smithi167 ceph-mon[32774]: 6.14 scrub starts 2024-02-05T06:39:20.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:20 smithi167 ceph-mon[32774]: 6.14 scrub ok 2024-02-05T06:39:20.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:20 smithi167 ceph-mon[32774]: osdmap e177: 8 total, 8 up, 7 in 2024-02-05T06:39:20.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:20 smithi167 ceph-mon[32774]: 7.c scrub starts 2024-02-05T06:39:20.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:20 smithi167 ceph-mon[32774]: 7.c scrub ok 2024-02-05T06:39:20.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:20 smithi167 ceph-mon[32774]: osdmap e178: 8 total, 8 up, 7 in 2024-02-05T06:39:20.742 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:20.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:20.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:20.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:20.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:20.870 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:20.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:20.971 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:20.997 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:20.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:20.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:21.099 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:21.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:21.125 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:21.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:21.226 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:21.250 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:21.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:21.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:21.352 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:21.381 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:21.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:21.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:21.482 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:21.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:21.507 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:21.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:21.608 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:21.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:21.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:21.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:21 smithi028 ceph-mon[29442]: 4.3 scrub starts 2024-02-05T06:39:21.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:21 smithi028 ceph-mon[29442]: 4.3 scrub ok 2024-02-05T06:39:21.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:21 smithi028 ceph-mon[29442]: 7.1 scrub starts 2024-02-05T06:39:21.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:21 smithi028 ceph-mon[29442]: 7.1 scrub ok 2024-02-05T06:39:21.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:21 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:39:21.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:21 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:39:21.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:21 smithi028 ceph-mon[29442]: pgmap v254: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 13 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 907 op/s; 28/1200 objects misplaced (2.333%); 82 B/s, 3 objects/s recovering 2024-02-05T06:39:21.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:21 smithi028 ceph-mon[29442]: 7.2 scrub starts 2024-02-05T06:39:21.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:21 smithi028 ceph-mon[29442]: 7.2 scrub ok 2024-02-05T06:39:21.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:21 smithi028 ceph-mon[29442]: osdmap e179: 8 total, 8 up, 7 in 2024-02-05T06:39:21.686 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:21 smithi028 ceph-mon[37984]: 4.3 scrub starts 2024-02-05T06:39:21.686 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:21 smithi028 ceph-mon[37984]: 4.3 scrub ok 2024-02-05T06:39:21.686 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:21 smithi028 ceph-mon[37984]: 7.1 scrub starts 2024-02-05T06:39:21.686 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:21 smithi028 ceph-mon[37984]: 7.1 scrub ok 2024-02-05T06:39:21.686 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:21 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:39:21.686 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:21 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:39:21.686 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:21 smithi028 ceph-mon[37984]: pgmap v254: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 13 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 907 op/s; 28/1200 objects misplaced (2.333%); 82 B/s, 3 objects/s recovering 2024-02-05T06:39:21.686 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:21 smithi028 ceph-mon[37984]: 7.2 scrub starts 2024-02-05T06:39:21.686 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:21 smithi028 ceph-mon[37984]: 7.2 scrub ok 2024-02-05T06:39:21.686 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:21 smithi028 ceph-mon[37984]: osdmap e179: 8 total, 8 up, 7 in 2024-02-05T06:39:21.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:21 smithi167 ceph-mon[32774]: 4.3 scrub starts 2024-02-05T06:39:21.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:21 smithi167 ceph-mon[32774]: 4.3 scrub ok 2024-02-05T06:39:21.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:21 smithi167 ceph-mon[32774]: 7.1 scrub starts 2024-02-05T06:39:21.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:21 smithi167 ceph-mon[32774]: 7.1 scrub ok 2024-02-05T06:39:21.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:21 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:39:21.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:21 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:39:21.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:21 smithi167 ceph-mon[32774]: pgmap v254: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 13 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 907 op/s; 28/1200 objects misplaced (2.333%); 82 B/s, 3 objects/s recovering 2024-02-05T06:39:21.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:21 smithi167 ceph-mon[32774]: 7.2 scrub starts 2024-02-05T06:39:21.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:21 smithi167 ceph-mon[32774]: 7.2 scrub ok 2024-02-05T06:39:21.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:21 smithi167 ceph-mon[32774]: osdmap e179: 8 total, 8 up, 7 in 2024-02-05T06:39:21.734 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:21.758 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:21.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:21.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:21.860 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:21.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:21.885 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:21.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:21.986 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:22.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:22.010 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:22.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:22.111 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:22.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:22.140 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:22.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:22.241 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:22.269 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:22.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:22.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:22.371 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:22.398 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:22.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:22.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:22.500 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:22.529 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:22.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:22.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:22.565 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-05T06:39:22.566 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-05T06:39:22.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph tell osd.6 'trim stale osdmaps' 2024-02-05T06:39:22.631 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:22.645 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:22 smithi167 ceph-mon[32774]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-05T06:39:22.657 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:22.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:22.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:22.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:22 smithi028 ceph-mon[29442]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-05T06:39:22.693 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:39:22 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:39:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:39:22.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:22 smithi028 ceph-mon[37984]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-05T06:39:22.759 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:22.782 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:22.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:22.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:22.884 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:22.912 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:22.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:22.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:22.981 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:39:22 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:39:22.991 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:39:23.014 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:23.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:23.039 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:23.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:23.140 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:23.168 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:23.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:23.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:23.270 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:23.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:23.294 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:23.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:23.395 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:23.404 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-02-05T06:39:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:23.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:23.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:23.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:23.546 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:23.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:23.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:23.644 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:23 smithi028 ceph-mon[37984]: pgmap v256: 158 pgs: 158 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 51 op/s 2024-02-05T06:39:23.644 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:23 smithi028 ceph-mon[37984]: 3.10 scrub starts 2024-02-05T06:39:23.645 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:23 smithi028 ceph-mon[37984]: 3.10 scrub ok 2024-02-05T06:39:23.645 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:23 smithi028 ceph-mon[37984]: 6.1 scrub starts 2024-02-05T06:39:23.645 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:23 smithi028 ceph-mon[37984]: 6.1 scrub ok 2024-02-05T06:39:23.645 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:23 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2456162483' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:39:23.645 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:23 smithi028 ceph-mon[37984]: from='client.44969 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:39:23.645 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:23 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:39:23.645 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:23 smithi028 ceph-mon[37984]: 6.a scrub starts 2024-02-05T06:39:23.645 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:23 smithi028 ceph-mon[37984]: 6.a scrub ok 2024-02-05T06:39:23.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:23 smithi028 ceph-mon[29442]: pgmap v256: 158 pgs: 158 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 51 op/s 2024-02-05T06:39:23.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:23 smithi028 ceph-mon[29442]: 3.10 scrub starts 2024-02-05T06:39:23.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:23 smithi028 ceph-mon[29442]: 3.10 scrub ok 2024-02-05T06:39:23.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:23 smithi028 ceph-mon[29442]: 6.1 scrub starts 2024-02-05T06:39:23.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:23 smithi028 ceph-mon[29442]: 6.1 scrub ok 2024-02-05T06:39:23.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:23 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2456162483' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:39:23.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:23 smithi028 ceph-mon[29442]: from='client.44969 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:39:23.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:23 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:39:23.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:23 smithi028 ceph-mon[29442]: 6.a scrub starts 2024-02-05T06:39:23.646 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:23 smithi028 ceph-mon[29442]: 6.a scrub ok 2024-02-05T06:39:23.648 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:23.676 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:23.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:23.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:23.710 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:39:23.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:23 smithi167 ceph-mon[32774]: pgmap v256: 158 pgs: 158 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 51 op/s 2024-02-05T06:39:23.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:23 smithi167 ceph-mon[32774]: 3.10 scrub starts 2024-02-05T06:39:23.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:23 smithi167 ceph-mon[32774]: 3.10 scrub ok 2024-02-05T06:39:23.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:23 smithi167 ceph-mon[32774]: 6.1 scrub starts 2024-02-05T06:39:23.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:23 smithi167 ceph-mon[32774]: 6.1 scrub ok 2024-02-05T06:39:23.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:23 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2456162483' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:39:23.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:23 smithi167 ceph-mon[32774]: from='client.44969 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:39:23.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:23 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:39:23.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:23 smithi167 ceph-mon[32774]: 6.a scrub starts 2024-02-05T06:39:23.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:23 smithi167 ceph-mon[32774]: 6.a scrub ok 2024-02-05T06:39:23.778 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:23.801 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:23.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:23.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:23.903 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:23.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:23.927 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:23.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:24.028 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:24.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:24.053 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:24.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:24.154 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:24.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:24.179 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:24.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:24.280 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:24.308 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:24.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:24.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:24.410 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:24.437 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:24.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:24.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:24.539 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:24.568 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:24.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:24.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:24.670 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:24.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:24 smithi028 ceph-mon[29442]: 7.9 scrub starts 2024-02-05T06:39:24.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:24 smithi028 ceph-mon[29442]: 7.9 scrub ok 2024-02-05T06:39:24.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:24 smithi028 ceph-mon[29442]: 3.8 scrub starts 2024-02-05T06:39:24.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:24 smithi028 ceph-mon[29442]: 3.8 scrub ok 2024-02-05T06:39:24.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:24 smithi028 ceph-mon[29442]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:39:24.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:24 smithi028 ceph-mon[29442]: Cluster is now healthy 2024-02-05T06:39:24.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:24 smithi028 ceph-mon[29442]: from='client.44969 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:39:24.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:24 smithi028 ceph-mon[29442]: osdmap e180: 8 total, 8 up, 7 in 2024-02-05T06:39:24.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:24 smithi028 ceph-mon[37984]: 7.9 scrub starts 2024-02-05T06:39:24.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:24 smithi028 ceph-mon[37984]: 7.9 scrub ok 2024-02-05T06:39:24.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:24 smithi028 ceph-mon[37984]: 3.8 scrub starts 2024-02-05T06:39:24.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:24 smithi028 ceph-mon[37984]: 3.8 scrub ok 2024-02-05T06:39:24.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:24 smithi028 ceph-mon[37984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:39:24.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:24 smithi028 ceph-mon[37984]: Cluster is now healthy 2024-02-05T06:39:24.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:24 smithi028 ceph-mon[37984]: from='client.44969 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:39:24.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:24 smithi028 ceph-mon[37984]: osdmap e180: 8 total, 8 up, 7 in 2024-02-05T06:39:24.698 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:24.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:24.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:24.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:24 smithi167 ceph-mon[32774]: 7.9 scrub starts 2024-02-05T06:39:24.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:24 smithi167 ceph-mon[32774]: 7.9 scrub ok 2024-02-05T06:39:24.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:24 smithi167 ceph-mon[32774]: 3.8 scrub starts 2024-02-05T06:39:24.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:24 smithi167 ceph-mon[32774]: 3.8 scrub ok 2024-02-05T06:39:24.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:24 smithi167 ceph-mon[32774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:39:24.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:24 smithi167 ceph-mon[32774]: Cluster is now healthy 2024-02-05T06:39:24.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:24 smithi167 ceph-mon[32774]: from='client.44969 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:39:24.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:24 smithi167 ceph-mon[32774]: osdmap e180: 8 total, 8 up, 7 in 2024-02-05T06:39:24.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:24.825 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:24.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:24.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:24.926 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:24.954 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:24.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:24.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:25.056 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:25.087 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:25.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:25.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:25.091 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:39:25.190 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:25.214 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:25.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:25.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:25.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:25.339 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:25.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:25.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:25.441 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:25.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:25.472 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:25.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:25.573 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:25.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:25.602 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:25.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:25.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:25 smithi028 ceph-mon[37984]: 6.2 deep-scrub starts 2024-02-05T06:39:25.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:25 smithi028 ceph-mon[37984]: 6.2 deep-scrub ok 2024-02-05T06:39:25.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:25 smithi028 ceph-mon[37984]: pgmap v258: 158 pgs: 158 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 42 op/s 2024-02-05T06:39:25.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:25 smithi028 ceph-mon[37984]: 7.7 deep-scrub starts 2024-02-05T06:39:25.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:25 smithi028 ceph-mon[37984]: 7.7 deep-scrub ok 2024-02-05T06:39:25.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:25 smithi028 ceph-mon[37984]: 1.3 scrub starts 2024-02-05T06:39:25.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:25 smithi028 ceph-mon[37984]: 1.3 scrub ok 2024-02-05T06:39:25.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:25 smithi028 ceph-mon[29442]: 6.2 deep-scrub starts 2024-02-05T06:39:25.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:25 smithi028 ceph-mon[29442]: 6.2 deep-scrub ok 2024-02-05T06:39:25.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:25 smithi028 ceph-mon[29442]: pgmap v258: 158 pgs: 158 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 42 op/s 2024-02-05T06:39:25.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:25 smithi028 ceph-mon[29442]: 7.7 deep-scrub starts 2024-02-05T06:39:25.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:25 smithi028 ceph-mon[29442]: 7.7 deep-scrub ok 2024-02-05T06:39:25.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:25 smithi028 ceph-mon[29442]: 1.3 scrub starts 2024-02-05T06:39:25.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:25 smithi028 ceph-mon[29442]: 1.3 scrub ok 2024-02-05T06:39:25.702 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:25.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:25.726 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:25.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:25.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:25 smithi167 ceph-mon[32774]: 6.2 deep-scrub starts 2024-02-05T06:39:25.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:25 smithi167 ceph-mon[32774]: 6.2 deep-scrub ok 2024-02-05T06:39:25.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:25 smithi167 ceph-mon[32774]: pgmap v258: 158 pgs: 158 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 42 op/s 2024-02-05T06:39:25.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:25 smithi167 ceph-mon[32774]: 7.7 deep-scrub starts 2024-02-05T06:39:25.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:25 smithi167 ceph-mon[32774]: 7.7 deep-scrub ok 2024-02-05T06:39:25.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:25 smithi167 ceph-mon[32774]: 1.3 scrub starts 2024-02-05T06:39:25.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:25 smithi167 ceph-mon[32774]: 1.3 scrub ok 2024-02-05T06:39:25.828 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:25.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:25.857 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:25.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:25.958 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:25.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:25.986 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:25.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:26.087 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:26.115 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:26.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:26.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:26.210 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:39:26.210 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:39:26.217 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:26.244 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:26.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:26.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:26.346 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:26.375 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:26.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:26.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:26.477 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:26.503 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:26.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:26.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:26.605 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:26.634 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:26.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:26.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:26.736 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:26.759 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:26.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:26.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:26.861 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:26.884 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:26.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:26.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:26.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:26 smithi028 ceph-mon[29442]: 4.2 scrub starts 2024-02-05T06:39:26.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:26 smithi028 ceph-mon[29442]: 4.2 scrub ok 2024-02-05T06:39:26.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:26 smithi028 ceph-mon[29442]: 5.13 scrub starts 2024-02-05T06:39:26.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:26 smithi028 ceph-mon[29442]: 5.13 scrub ok 2024-02-05T06:39:26.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:26 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:39:26.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:26 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:39:26.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:26 smithi028 ceph-mon[37984]: 4.2 scrub starts 2024-02-05T06:39:26.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:26 smithi028 ceph-mon[37984]: 4.2 scrub ok 2024-02-05T06:39:26.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:26 smithi028 ceph-mon[37984]: 5.13 scrub starts 2024-02-05T06:39:26.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:26 smithi028 ceph-mon[37984]: 5.13 scrub ok 2024-02-05T06:39:26.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:26 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:39:26.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:26 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:39:26.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:26 smithi167 ceph-mon[32774]: 4.2 scrub starts 2024-02-05T06:39:26.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:26 smithi167 ceph-mon[32774]: 4.2 scrub ok 2024-02-05T06:39:26.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:26 smithi167 ceph-mon[32774]: 5.13 scrub starts 2024-02-05T06:39:26.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:26 smithi167 ceph-mon[32774]: 5.13 scrub ok 2024-02-05T06:39:26.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:26 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:39:26.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:26 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:39:26.986 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:27.013 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:27.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:27.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:27.115 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:27.138 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:27.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:27.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:27.240 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:27.263 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:27.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:27.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:27.365 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:27.393 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:27.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:27.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:27.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-05T06:39:27.495 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:27.542 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:27.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:27.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:27.644 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:27.644 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T06:39:27.679 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'unique_pool_0' removed 2024-02-05T06:39:27.714 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:27.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:27.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:27.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:27 smithi028 ceph-mon[29442]: pgmap v259: 158 pgs: 158 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 815 B/s rd, 0 B/s wr, 32 op/s 2024-02-05T06:39:27.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:27 smithi028 ceph-mon[29442]: 6.b scrub starts 2024-02-05T06:39:27.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:27 smithi028 ceph-mon[29442]: 6.b scrub ok 2024-02-05T06:39:27.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:27 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3771120201' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T06:39:27.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:27 smithi028 ceph-mon[29442]: from='client.44975 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T06:39:27.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:27 smithi028 ceph-mon[29442]: 3.0 scrub starts 2024-02-05T06:39:27.720 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:27 smithi028 ceph-mon[29442]: 3.0 scrub ok 2024-02-05T06:39:27.721 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:27 smithi028 ceph-mon[37984]: pgmap v259: 158 pgs: 158 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 815 B/s rd, 0 B/s wr, 32 op/s 2024-02-05T06:39:27.721 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:27 smithi028 ceph-mon[37984]: 6.b scrub starts 2024-02-05T06:39:27.721 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:27 smithi028 ceph-mon[37984]: 6.b scrub ok 2024-02-05T06:39:27.721 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:27 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3771120201' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T06:39:27.721 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:27 smithi028 ceph-mon[37984]: from='client.44975 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T06:39:27.721 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:27 smithi028 ceph-mon[37984]: 3.0 scrub starts 2024-02-05T06:39:27.721 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:27 smithi028 ceph-mon[37984]: 3.0 scrub ok 2024-02-05T06:39:27.816 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:27.839 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:27.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:27.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:27.941 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:27.967 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:27.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:27.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:27.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:27 smithi167 ceph-mon[32774]: pgmap v259: 158 pgs: 158 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 815 B/s rd, 0 B/s wr, 32 op/s 2024-02-05T06:39:27.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:27 smithi167 ceph-mon[32774]: 6.b scrub starts 2024-02-05T06:39:27.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:27 smithi167 ceph-mon[32774]: 6.b scrub ok 2024-02-05T06:39:27.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:27 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3771120201' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T06:39:27.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:27 smithi167 ceph-mon[32774]: from='client.44975 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T06:39:27.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:27 smithi167 ceph-mon[32774]: 3.0 scrub starts 2024-02-05T06:39:27.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:27 smithi167 ceph-mon[32774]: 3.0 scrub ok 2024-02-05T06:39:28.069 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:28.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:28.095 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:28.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:28.196 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:28.219 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:28.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:28.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:28.321 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:28.347 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:28.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:28.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:28.449 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:28.473 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:28.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:28.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:28.575 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:28.591 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-05T06:39:28.592 INFO:tasks.radosbench:Beginning radosbench... 2024-02-05T06:39:28.592 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-02-05T06:39:28.592 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd pool create unique_pool_1 16 2024-02-05T06:39:28.598 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:28.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:28.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:28.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:28.724 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:28.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:28.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:28.826 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:28.850 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:28.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:28.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:28.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:28 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:39:28.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:28 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:39:28.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:28 smithi028 ceph-mon[29442]: 7.a scrub starts 2024-02-05T06:39:28.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:28 smithi028 ceph-mon[29442]: 7.a scrub ok 2024-02-05T06:39:28.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:28 smithi028 ceph-mon[29442]: from='client.44975 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T06:39:28.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:28 smithi028 ceph-mon[29442]: osdmap e181: 8 total, 8 up, 7 in 2024-02-05T06:39:28.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:28 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:39:28.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:28 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:39:28.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:28 smithi028 ceph-mon[37984]: 7.a scrub starts 2024-02-05T06:39:28.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:28 smithi028 ceph-mon[37984]: 7.a scrub ok 2024-02-05T06:39:28.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:28 smithi028 ceph-mon[37984]: from='client.44975 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T06:39:28.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:28 smithi028 ceph-mon[37984]: osdmap e181: 8 total, 8 up, 7 in 2024-02-05T06:39:28.952 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:28.975 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:28.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:28.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:28.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:28 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:39:28.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:28 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:39:28.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:28 smithi167 ceph-mon[32774]: 7.a scrub starts 2024-02-05T06:39:28.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:28 smithi167 ceph-mon[32774]: 7.a scrub ok 2024-02-05T06:39:28.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:28 smithi167 ceph-mon[32774]: from='client.44975 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T06:39:28.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:28 smithi167 ceph-mon[32774]: osdmap e181: 8 total, 8 up, 7 in 2024-02-05T06:39:29.034 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:39:29.077 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:29.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:29.103 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:29.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:29.204 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:29.228 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:29.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:29.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:29.330 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:29.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:29.356 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:29.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:29.460 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:29.485 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:29.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:29.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:29.587 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:29.614 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:29.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:29.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:29.619 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:39:29.622 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:39:29.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:29 smithi028 ceph-mon[37984]: pgmap v261: 142 pgs: 142 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:39:29.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:29 smithi028 ceph-mon[37984]: 3.5 deep-scrub starts 2024-02-05T06:39:29.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:29 smithi028 ceph-mon[37984]: 3.5 deep-scrub ok 2024-02-05T06:39:29.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:29 smithi028 ceph-mon[37984]: 3.1f scrub starts 2024-02-05T06:39:29.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:29 smithi028 ceph-mon[37984]: 3.1f scrub ok 2024-02-05T06:39:29.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:29 smithi028 ceph-mon[29442]: pgmap v261: 142 pgs: 142 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:39:29.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:29 smithi028 ceph-mon[29442]: 3.5 deep-scrub starts 2024-02-05T06:39:29.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:29 smithi028 ceph-mon[29442]: 3.5 deep-scrub ok 2024-02-05T06:39:29.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:29 smithi028 ceph-mon[29442]: 3.1f scrub starts 2024-02-05T06:39:29.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:29 smithi028 ceph-mon[29442]: 3.1f scrub ok 2024-02-05T06:39:29.716 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:29.743 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:29.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:29.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:29.845 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:29.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:29.869 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:29.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:29.970 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:29.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:29 smithi167 ceph-mon[32774]: pgmap v261: 142 pgs: 142 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:39:29.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:29 smithi167 ceph-mon[32774]: 3.5 deep-scrub starts 2024-02-05T06:39:29.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:29 smithi167 ceph-mon[32774]: 3.5 deep-scrub ok 2024-02-05T06:39:29.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:29 smithi167 ceph-mon[32774]: 3.1f scrub starts 2024-02-05T06:39:29.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:29 smithi167 ceph-mon[32774]: 3.1f scrub ok 2024-02-05T06:39:29.999 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:30.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:30.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:30.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:30.125 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:30.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:30.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:30.227 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:30.255 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:30.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:30.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:30.357 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:30.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:30.384 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:30.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:30.485 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:30.509 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:30.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:30.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:30.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:30.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:30.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:30.736 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:30.765 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:30.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:30.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:30.867 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:30.895 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:30.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:30.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:30.997 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:31.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:31.023 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:31.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:31.124 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:31.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:31.150 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:31.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:31.251 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:31.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:30 smithi028 ceph-mon[29442]: 5.16 scrub starts 2024-02-05T06:39:31.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:30 smithi028 ceph-mon[29442]: 5.16 scrub ok 2024-02-05T06:39:31.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:30 smithi028 ceph-mon[29442]: osdmap e182: 8 total, 8 up, 7 in 2024-02-05T06:39:31.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:30 smithi028 ceph-mon[29442]: 6.e deep-scrub starts 2024-02-05T06:39:31.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:30 smithi028 ceph-mon[29442]: 6.e deep-scrub ok 2024-02-05T06:39:31.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:30 smithi028 ceph-mon[29442]: pgmap v263: 142 pgs: 142 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 596 B/s rd, 0 op/s 2024-02-05T06:39:31.275 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:30 smithi028 ceph-mon[37984]: 5.16 scrub starts 2024-02-05T06:39:31.275 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:30 smithi028 ceph-mon[37984]: 5.16 scrub ok 2024-02-05T06:39:31.275 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:30 smithi028 ceph-mon[37984]: osdmap e182: 8 total, 8 up, 7 in 2024-02-05T06:39:31.275 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:30 smithi028 ceph-mon[37984]: 6.e deep-scrub starts 2024-02-05T06:39:31.275 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:30 smithi028 ceph-mon[37984]: 6.e deep-scrub ok 2024-02-05T06:39:31.275 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:30 smithi028 ceph-mon[37984]: pgmap v263: 142 pgs: 142 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 596 B/s rd, 0 op/s 2024-02-05T06:39:31.279 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:31.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:31.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:31.381 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:31.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:31.412 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:31.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:31.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:30 smithi167 ceph-mon[32774]: 5.16 scrub starts 2024-02-05T06:39:31.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:30 smithi167 ceph-mon[32774]: 5.16 scrub ok 2024-02-05T06:39:31.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:30 smithi167 ceph-mon[32774]: osdmap e182: 8 total, 8 up, 7 in 2024-02-05T06:39:31.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:30 smithi167 ceph-mon[32774]: 6.e deep-scrub starts 2024-02-05T06:39:31.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:30 smithi167 ceph-mon[32774]: 6.e deep-scrub ok 2024-02-05T06:39:31.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:30 smithi167 ceph-mon[32774]: pgmap v263: 142 pgs: 142 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 596 B/s rd, 0 op/s 2024-02-05T06:39:31.513 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:31.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:31.538 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:31.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:31.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:31.662 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:31.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:31.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:31.764 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:31.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:31.793 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:31.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:31.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:31.918 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:31.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:31.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:32.020 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:32.051 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:32.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:32.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:32.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:31 smithi028 ceph-mon[29442]: 4.4 scrub starts 2024-02-05T06:39:32.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:31 smithi028 ceph-mon[29442]: 4.4 scrub ok 2024-02-05T06:39:32.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:31 smithi028 ceph-mon[29442]: 1.7 scrub starts 2024-02-05T06:39:32.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:31 smithi028 ceph-mon[29442]: 1.7 scrub ok 2024-02-05T06:39:32.087 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:31 smithi028 ceph-mon[29442]: 6.0 scrub starts 2024-02-05T06:39:32.088 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:31 smithi028 ceph-mon[29442]: 6.0 scrub ok 2024-02-05T06:39:32.088 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:31 smithi028 ceph-mon[29442]: 4.d scrub starts 2024-02-05T06:39:32.088 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:31 smithi028 ceph-mon[29442]: 4.d scrub ok 2024-02-05T06:39:32.088 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:31 smithi028 ceph-mon[37984]: 4.4 scrub starts 2024-02-05T06:39:32.088 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:31 smithi028 ceph-mon[37984]: 4.4 scrub ok 2024-02-05T06:39:32.088 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:31 smithi028 ceph-mon[37984]: 1.7 scrub starts 2024-02-05T06:39:32.088 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:31 smithi028 ceph-mon[37984]: 1.7 scrub ok 2024-02-05T06:39:32.088 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:31 smithi028 ceph-mon[37984]: 6.0 scrub starts 2024-02-05T06:39:32.088 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:31 smithi028 ceph-mon[37984]: 6.0 scrub ok 2024-02-05T06:39:32.088 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:31 smithi028 ceph-mon[37984]: 4.d scrub starts 2024-02-05T06:39:32.089 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:31 smithi028 ceph-mon[37984]: 4.d scrub ok 2024-02-05T06:39:32.127 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:39:32.146 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:39:32.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:32.165 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:39:32.176 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:32.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:32.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:32.185 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:39:32.206 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:39:32.227 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:39:32.248 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:39:32.269 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:39:32.270 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:39:32.270 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:39:32.270 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:39:32.270 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:39:32.270 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:39:32.270 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:39:32.270 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:39:32.270 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:39:32.278 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:32.303 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:32.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:32.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:32.405 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:32.429 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:32.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:32.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:32.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:31 smithi167 ceph-mon[32774]: 4.4 scrub starts 2024-02-05T06:39:32.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:31 smithi167 ceph-mon[32774]: 4.4 scrub ok 2024-02-05T06:39:32.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:31 smithi167 ceph-mon[32774]: 1.7 scrub starts 2024-02-05T06:39:32.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:31 smithi167 ceph-mon[32774]: 1.7 scrub ok 2024-02-05T06:39:32.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:31 smithi167 ceph-mon[32774]: 6.0 scrub starts 2024-02-05T06:39:32.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:31 smithi167 ceph-mon[32774]: 6.0 scrub ok 2024-02-05T06:39:32.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:31 smithi167 ceph-mon[32774]: 4.d scrub starts 2024-02-05T06:39:32.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:31 smithi167 ceph-mon[32774]: 4.d scrub ok 2024-02-05T06:39:32.531 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:32.559 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:32.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:32.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:32.661 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:32.685 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:32.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:32.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:32.760 INFO:teuthology.orchestra.run.smithi028.stderr:no valid command found; 10 closest matches: 2024-02-05T06:39:32.760 INFO:teuthology.orchestra.run.smithi028.stderr:0 2024-02-05T06:39:32.760 INFO:teuthology.orchestra.run.smithi028.stderr:1 2024-02-05T06:39:32.760 INFO:teuthology.orchestra.run.smithi028.stderr:2 2024-02-05T06:39:32.760 INFO:teuthology.orchestra.run.smithi028.stderr:abort 2024-02-05T06:39:32.760 INFO:teuthology.orchestra.run.smithi028.stderr:assert 2024-02-05T06:39:32.760 INFO:teuthology.orchestra.run.smithi028.stderr:bench [] [] [] [] 2024-02-05T06:39:32.760 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs debug_inject_read_zeros 2024-02-05T06:39:32.761 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs files list 2024-02-05T06:39:32.761 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs stats 2024-02-05T06:39:32.761 INFO:teuthology.orchestra.run.smithi028.stderr:bluestore allocator dump block 2024-02-05T06:39:32.761 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: invalid command 2024-02-05T06:39:32.761 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:39:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:39:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:39:32.788 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:32.811 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:32.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:32.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:32.913 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:32.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:32.941 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:32.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:32.981 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:39:32 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:39:33.043 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:33.069 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:33.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:33.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:33.120 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:32 smithi028 ceph-mon[29442]: 6.16 scrub starts 2024-02-05T06:39:33.120 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:32 smithi028 ceph-mon[29442]: 6.16 scrub ok 2024-02-05T06:39:33.120 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:32 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3067050116' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:39:33.120 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:32 smithi028 ceph-mon[29442]: 5.1d scrub starts 2024-02-05T06:39:33.120 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:32 smithi028 ceph-mon[29442]: 5.1d scrub ok 2024-02-05T06:39:33.120 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:32 smithi028 ceph-mon[29442]: pgmap v264: 142 pgs: 142 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:39:33.121 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:32 smithi028 ceph-mon[29442]: 4.10 scrub starts 2024-02-05T06:39:33.121 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:32 smithi028 ceph-mon[29442]: 4.10 scrub ok 2024-02-05T06:39:33.121 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:32 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:39:33.121 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:32 smithi028 ceph-mon[37984]: 6.16 scrub starts 2024-02-05T06:39:33.121 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:32 smithi028 ceph-mon[37984]: 6.16 scrub ok 2024-02-05T06:39:33.121 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:32 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3067050116' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:39:33.121 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:32 smithi028 ceph-mon[37984]: 5.1d scrub starts 2024-02-05T06:39:33.121 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:32 smithi028 ceph-mon[37984]: 5.1d scrub ok 2024-02-05T06:39:33.121 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:32 smithi028 ceph-mon[37984]: pgmap v264: 142 pgs: 142 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:39:33.121 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:32 smithi028 ceph-mon[37984]: 4.10 scrub starts 2024-02-05T06:39:33.121 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:32 smithi028 ceph-mon[37984]: 4.10 scrub ok 2024-02-05T06:39:33.121 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:32 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:39:33.171 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:33.195 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:33.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:33.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:33.297 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:33.325 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:33.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:33.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:33.427 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:33.452 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:33.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:33.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:33.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:32 smithi167 ceph-mon[32774]: 6.16 scrub starts 2024-02-05T06:39:33.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:32 smithi167 ceph-mon[32774]: 6.16 scrub ok 2024-02-05T06:39:33.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:32 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3067050116' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:39:33.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:32 smithi167 ceph-mon[32774]: 5.1d scrub starts 2024-02-05T06:39:33.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:32 smithi167 ceph-mon[32774]: 5.1d scrub ok 2024-02-05T06:39:33.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:32 smithi167 ceph-mon[32774]: pgmap v264: 142 pgs: 142 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:39:33.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:32 smithi167 ceph-mon[32774]: 4.10 scrub starts 2024-02-05T06:39:33.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:32 smithi167 ceph-mon[32774]: 4.10 scrub ok 2024-02-05T06:39:33.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:32 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:39:33.556 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:33.584 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:33.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:33.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:33.686 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:33.715 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:33.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:33.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:33.817 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:33.845 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:33.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:33.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:33.947 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:33.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:33.975 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:33.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:34.076 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:34.100 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:34.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:34.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:39:34.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:33 smithi028 ceph-mon[29442]: 5.2 scrub starts 2024-02-05T06:39:34.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:33 smithi028 ceph-mon[29442]: 5.2 scrub ok 2024-02-05T06:39:34.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:33 smithi028 ceph-mon[29442]: 6.8 deep-scrub starts 2024-02-05T06:39:34.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:33 smithi028 ceph-mon[29442]: 6.8 deep-scrub ok 2024-02-05T06:39:34.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:33 smithi028 ceph-mon[29442]: 4.7 scrub starts 2024-02-05T06:39:34.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:33 smithi028 ceph-mon[29442]: 4.7 scrub ok 2024-02-05T06:39:34.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:33 smithi028 ceph-mon[37984]: 5.2 scrub starts 2024-02-05T06:39:34.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:33 smithi028 ceph-mon[37984]: 5.2 scrub ok 2024-02-05T06:39:34.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:33 smithi028 ceph-mon[37984]: 6.8 deep-scrub starts 2024-02-05T06:39:34.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:33 smithi028 ceph-mon[37984]: 6.8 deep-scrub ok 2024-02-05T06:39:34.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:33 smithi028 ceph-mon[37984]: 4.7 scrub starts 2024-02-05T06:39:34.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:33 smithi028 ceph-mon[37984]: 4.7 scrub ok 2024-02-05T06:39:34.201 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:34.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:34.230 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:34.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:34.331 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:34.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:34.356 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:34.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:34.457 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:34.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:33 smithi167 ceph-mon[32774]: 5.2 scrub starts 2024-02-05T06:39:34.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:33 smithi167 ceph-mon[32774]: 5.2 scrub ok 2024-02-05T06:39:34.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:33 smithi167 ceph-mon[32774]: 6.8 deep-scrub starts 2024-02-05T06:39:34.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:33 smithi167 ceph-mon[32774]: 6.8 deep-scrub ok 2024-02-05T06:39:34.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:33 smithi167 ceph-mon[32774]: 4.7 scrub starts 2024-02-05T06:39:34.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:33 smithi167 ceph-mon[32774]: 4.7 scrub ok 2024-02-05T06:39:34.485 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:34.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:34.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:34.587 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:34.611 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:34.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:34.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:34.714 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:34.742 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:34.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:34.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:34.844 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:34.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:34.872 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:34.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:34.973 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:35.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:35.001 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:35.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:35.102 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:35.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:35.128 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:35.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:35.229 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:35.256 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:35.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:35.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:35.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:35 smithi028 ceph-mon[29442]: 2.2 deep-scrub starts 2024-02-05T06:39:35.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:35 smithi028 ceph-mon[29442]: 2.2 deep-scrub ok 2024-02-05T06:39:35.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:35 smithi028 ceph-mon[29442]: 5.6 deep-scrub starts 2024-02-05T06:39:35.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:35 smithi028 ceph-mon[29442]: 5.6 deep-scrub ok 2024-02-05T06:39:35.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:35 smithi028 ceph-mon[29442]: 3.4 scrub starts 2024-02-05T06:39:35.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:35 smithi028 ceph-mon[29442]: 3.4 scrub ok 2024-02-05T06:39:35.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:35 smithi028 ceph-mon[29442]: pgmap v265: 142 pgs: 142 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:39:35.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:35 smithi028 ceph-mon[29442]: 3.1e scrub starts 2024-02-05T06:39:35.258 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:35 smithi028 ceph-mon[29442]: 3.1e scrub ok 2024-02-05T06:39:35.258 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:35 smithi028 ceph-mon[37984]: 2.2 deep-scrub starts 2024-02-05T06:39:35.258 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:35 smithi028 ceph-mon[37984]: 2.2 deep-scrub ok 2024-02-05T06:39:35.258 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:35 smithi028 ceph-mon[37984]: 5.6 deep-scrub starts 2024-02-05T06:39:35.259 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:35 smithi028 ceph-mon[37984]: 5.6 deep-scrub ok 2024-02-05T06:39:35.259 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:35 smithi028 ceph-mon[37984]: 3.4 scrub starts 2024-02-05T06:39:35.259 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:35 smithi028 ceph-mon[37984]: 3.4 scrub ok 2024-02-05T06:39:35.259 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:35 smithi028 ceph-mon[37984]: pgmap v265: 142 pgs: 142 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:39:35.259 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:35 smithi028 ceph-mon[37984]: 3.1e scrub starts 2024-02-05T06:39:35.259 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:35 smithi028 ceph-mon[37984]: 3.1e scrub ok 2024-02-05T06:39:35.359 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:35.382 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:35 smithi167 ceph-mon[32774]: 2.2 deep-scrub starts 2024-02-05T06:39:35.382 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:35 smithi167 ceph-mon[32774]: 2.2 deep-scrub ok 2024-02-05T06:39:35.382 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:35 smithi167 ceph-mon[32774]: 5.6 deep-scrub starts 2024-02-05T06:39:35.383 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:35 smithi167 ceph-mon[32774]: 5.6 deep-scrub ok 2024-02-05T06:39:35.383 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:35 smithi167 ceph-mon[32774]: 3.4 scrub starts 2024-02-05T06:39:35.383 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:35 smithi167 ceph-mon[32774]: 3.4 scrub ok 2024-02-05T06:39:35.383 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:35 smithi167 ceph-mon[32774]: pgmap v265: 142 pgs: 142 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:39:35.383 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:35 smithi167 ceph-mon[32774]: 3.1e scrub starts 2024-02-05T06:39:35.383 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:35 smithi167 ceph-mon[32774]: 3.1e scrub ok 2024-02-05T06:39:35.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:35.384 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:35.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:35.485 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:35.509 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:39:35.509 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-02-05T06:39:35.509 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-02-05T06:39:35.509 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-02-05T06:39:35.509 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.509 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14847.0:115896 7.0 e123/96 7:024728f6:::benchmark_data_smithi028_77509_object776:head v 123'7218, mlcod=123'7216)", 2024-02-05T06:39:35.509 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:37:08.050520+0000", 2024-02-05T06:39:35.509 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 147.45200089799999, 2024-02-05T06:39:35.510 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.081151329999999994, 2024-02-05T06:39:35.510 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:39:35.510 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:39:35.510 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:39:35.510 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.510 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:39:35.510 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.050520+0000", 2024-02-05T06:39:35.510 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.510 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.510 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.510 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:39:35.510 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.050520+0000", 2024-02-05T06:39:35.510 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.511 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.511 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.511 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:39:35.511 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.050521+0000", 2024-02-05T06:39:35.511 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.308e-06 2024-02-05T06:39:35.511 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.511 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.511 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:39:35.512 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.050528+0000", 2024-02-05T06:39:35.512 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7399999999999998e-06 2024-02-05T06:39:35.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.512 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.512 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:39:35.512 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.050538+0000", 2024-02-05T06:39:35.512 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.995e-06 2024-02-05T06:39:35.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.513 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.513 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:39:35.513 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.050545+0000", 2024-02-05T06:39:35.513 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2080000000000003e-06 2024-02-05T06:39:35.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.513 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.513 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:39:35.514 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.050598+0000", 2024-02-05T06:39:35.514 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2797000000000003e-05 2024-02-05T06:39:35.514 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.514 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.514 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:39:35.514 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.050617+0000", 2024-02-05T06:39:35.514 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.859e-05 2024-02-05T06:39:35.514 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.515 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.515 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:39:35.515 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.131617+0000", 2024-02-05T06:39:35.515 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.081000513999999996 2024-02-05T06:39:35.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.515 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.516 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:39:35.516 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.131671+0000", 2024-02-05T06:39:35.516 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4178000000000002e-05 2024-02-05T06:39:35.516 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.516 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:39:35.516 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.516 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.516 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.517 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:115897 7.1 7:83791f66:::benchmark_data_smithi028_77509_object777:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-02-05T06:39:35.517 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:37:08.052317+0000", 2024-02-05T06:39:35.517 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 147.45020423, 2024-02-05T06:39:35.517 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.096961541999999998, 2024-02-05T06:39:35.517 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:39:35.517 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:39:35.517 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:39:35.517 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:39:35.517 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:39:35.517 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 115897 2024-02-05T06:39:35.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.518 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:39:35.518 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.518 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:39:35.518 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.052317+0000", 2024-02-05T06:39:35.518 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.518 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.518 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:39:35.518 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.052317+0000", 2024-02-05T06:39:35.518 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.518 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.518 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:39:35.518 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.052319+0000", 2024-02-05T06:39:35.519 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.979e-06 2024-02-05T06:39:35.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.519 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.519 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:39:35.519 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.052324+0000", 2024-02-05T06:39:35.519 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2889999999999997e-06 2024-02-05T06:39:35.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.519 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.519 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:39:35.519 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.131518+0000", 2024-02-05T06:39:35.519 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.079193701000000005 2024-02-05T06:39:35.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.519 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.519 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:39:35.519 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.131529+0000", 2024-02-05T06:39:35.520 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1586000000000001e-05 2024-02-05T06:39:35.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.520 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.520 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:39:35.520 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.132457+0000", 2024-02-05T06:39:35.520 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00092798100000000003 2024-02-05T06:39:35.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.520 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.520 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:39:35.520 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.132540+0000", 2024-02-05T06:39:35.520 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2987999999999999e-05 2024-02-05T06:39:35.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.520 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.520 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4,6", 2024-02-05T06:39:35.521 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.132615+0000", 2024-02-05T06:39:35.521 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4407999999999996e-05 2024-02-05T06:39:35.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.521 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.521 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:39:35.521 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.137996+0000", 2024-02-05T06:39:35.521 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0053816740000000004 2024-02-05T06:39:35.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.521 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.521 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:39:35.521 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.138040+0000", 2024-02-05T06:39:35.521 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3729000000000001e-05 2024-02-05T06:39:35.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.521 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.522 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:39:35.522 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.144364+0000", 2024-02-05T06:39:35.522 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0063243070000000004 2024-02-05T06:39:35.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.522 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.522 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:39:35.522 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.148809+0000", 2024-02-05T06:39:35.522 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0044442650000000002 2024-02-05T06:39:35.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.522 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.522 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:39:35.522 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.149158+0000", 2024-02-05T06:39:35.522 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00034900799999999999 2024-02-05T06:39:35.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.522 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.523 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:39:35.523 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.149278+0000", 2024-02-05T06:39:35.523 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000120627 2024-02-05T06:39:35.523 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.523 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:39:35.523 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.523 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.523 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14847.0:116015 7.0 e123/96 7:09d99a81:::benchmark_data_smithi028_77509_object895:head v 123'7229, mlcod=123'7227)", 2024-02-05T06:39:35.523 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:37:08.258745+0000", 2024-02-05T06:39:35.523 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 147.24377630800001, 2024-02-05T06:39:35.523 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.080875104000000003, 2024-02-05T06:39:35.523 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:39:35.523 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:39:35.523 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:39:35.524 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.524 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:39:35.524 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.258745+0000", 2024-02-05T06:39:35.524 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.524 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.524 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:39:35.524 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.258745+0000", 2024-02-05T06:39:35.524 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.524 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.524 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:39:35.524 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.258754+0000", 2024-02-05T06:39:35.525 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3850000000000008e-06 2024-02-05T06:39:35.525 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.525 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.525 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:39:35.525 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.258782+0000", 2024-02-05T06:39:35.525 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8238e-05 2024-02-05T06:39:35.525 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.525 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.526 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:39:35.526 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.258804+0000", 2024-02-05T06:39:35.526 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1330000000000001e-05 2024-02-05T06:39:35.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.526 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.526 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:39:35.526 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.258811+0000", 2024-02-05T06:39:35.526 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8089999999999993e-06 2024-02-05T06:39:35.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.527 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.527 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:39:35.527 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.258884+0000", 2024-02-05T06:39:35.527 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2194999999999996e-05 2024-02-05T06:39:35.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.527 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.527 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:39:35.528 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.258903+0000", 2024-02-05T06:39:35.528 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9814999999999999e-05 2024-02-05T06:39:35.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.528 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.528 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:39:35.528 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.339581+0000", 2024-02-05T06:39:35.528 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.080677171000000006 2024-02-05T06:39:35.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.529 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.529 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:39:35.529 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.339620+0000", 2024-02-05T06:39:35.529 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9161e-05 2024-02-05T06:39:35.529 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.529 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:39:35.530 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.530 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.530 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14847.0:116018 7.0 e123/96 7:0719b86d:::benchmark_data_smithi028_77509_object898:head v 123'7230, mlcod=123'7227)", 2024-02-05T06:39:35.530 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:37:08.262083+0000", 2024-02-05T06:39:35.530 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 147.240438039, 2024-02-05T06:39:35.531 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.077561942999999994, 2024-02-05T06:39:35.531 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:39:35.531 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:39:35.531 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:39:35.531 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.531 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:39:35.531 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.262083+0000", 2024-02-05T06:39:35.532 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.532 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.532 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:39:35.532 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.262083+0000", 2024-02-05T06:39:35.532 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.533 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.533 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:39:35.533 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.262084+0000", 2024-02-05T06:39:35.533 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2399999999999997e-07 2024-02-05T06:39:35.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.533 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.533 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:39:35.534 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.262090+0000", 2024-02-05T06:39:35.534 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5549999999999999e-06 2024-02-05T06:39:35.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.534 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.534 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:39:35.534 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.262099+0000", 2024-02-05T06:39:35.534 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1249999999999999e-06 2024-02-05T06:39:35.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.535 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.535 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:39:35.535 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.262106+0000", 2024-02-05T06:39:35.535 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4690000000000002e-06 2024-02-05T06:39:35.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.536 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.536 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:39:35.536 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.265385+0000", 2024-02-05T06:39:35.536 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003278638 2024-02-05T06:39:35.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.536 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.537 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:39:35.537 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.265405+0000", 2024-02-05T06:39:35.537 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0687999999999999e-05 2024-02-05T06:39:35.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.537 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.537 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:39:35.538 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.339623+0000", 2024-02-05T06:39:35.538 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.074217394000000006 2024-02-05T06:39:35.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.538 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.538 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:39:35.538 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.339645+0000", 2024-02-05T06:39:35.539 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2249999999999999e-05 2024-02-05T06:39:35.539 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.539 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:39:35.539 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.539 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.539 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.539 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14847.0:116024 7.0 e123/96 7:0013e82d:::benchmark_data_smithi028_77509_object904:head v 123'7232, mlcod=123'7227)", 2024-02-05T06:39:35.539 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:37:08.267014+0000", 2024-02-05T06:39:35.539 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 147.23550670700001, 2024-02-05T06:39:35.539 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.078333861000000005, 2024-02-05T06:39:35.539 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:39:35.540 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:39:35.540 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:39:35.540 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.540 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:39:35.540 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.267014+0000", 2024-02-05T06:39:35.540 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.540 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.540 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:39:35.540 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.267014+0000", 2024-02-05T06:39:35.540 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.540 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.540 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:39:35.541 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.267015+0000", 2024-02-05T06:39:35.541 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.254e-06 2024-02-05T06:39:35.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.541 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.541 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:39:35.541 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.267022+0000", 2024-02-05T06:39:35.541 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1709999999999999e-06 2024-02-05T06:39:35.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.541 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.541 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:39:35.541 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.267030+0000", 2024-02-05T06:39:35.541 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7010000000000002e-06 2024-02-05T06:39:35.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.541 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.542 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:39:35.542 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.267037+0000", 2024-02-05T06:39:35.542 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0690000000000004e-06 2024-02-05T06:39:35.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.542 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.542 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:39:35.542 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.339848+0000", 2024-02-05T06:39:35.542 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.072810428999999996 2024-02-05T06:39:35.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.542 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.542 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:39:35.542 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.339867+0000", 2024-02-05T06:39:35.542 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9491e-05 2024-02-05T06:39:35.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.542 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.543 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:39:35.543 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.345279+0000", 2024-02-05T06:39:35.543 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0054114389999999997 2024-02-05T06:39:35.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.543 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.543 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:39:35.543 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:08.345348+0000", 2024-02-05T06:39:35.543 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9307000000000006e-05 2024-02-05T06:39:35.543 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.543 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:39:35.543 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.543 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.543 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14847.0:116795 7.2 e123/96 7:443c9f92:::benchmark_data_smithi028_77509_object1675:head v 123'7286, mlcod=123'7284)", 2024-02-05T06:39:35.544 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:37:09.307936+0000", 2024-02-05T06:39:35.544 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 146.194585012, 2024-02-05T06:39:35.544 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.092757275, 2024-02-05T06:39:35.544 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:39:35.544 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:39:35.544 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:39:35.544 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.544 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:39:35.544 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:09.307936+0000", 2024-02-05T06:39:35.544 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.544 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.544 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.544 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:39:35.544 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:09.307936+0000", 2024-02-05T06:39:35.545 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.545 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.545 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:39:35.545 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:09.307936+0000", 2024-02-05T06:39:35.545 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.51e-07 2024-02-05T06:39:35.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.545 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.545 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:39:35.545 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:09.307942+0000", 2024-02-05T06:39:35.545 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3539999999999999e-06 2024-02-05T06:39:35.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.545 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.545 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:39:35.545 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:09.307948+0000", 2024-02-05T06:39:35.546 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0599999999999996e-06 2024-02-05T06:39:35.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.546 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.546 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:39:35.546 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:09.307954+0000", 2024-02-05T06:39:35.546 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6389999999999997e-06 2024-02-05T06:39:35.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.546 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.546 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:39:35.546 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:09.308672+0000", 2024-02-05T06:39:35.546 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00071860299999999995 2024-02-05T06:39:35.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.546 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.546 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:39:35.546 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:09.308694+0000", 2024-02-05T06:39:35.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:35.548 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:35.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:35.549 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2130999999999999e-05 2024-02-05T06:39:35.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.549 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.549 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:39:35.549 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:09.399826+0000", 2024-02-05T06:39:35.549 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.091131456999999999 2024-02-05T06:39:35.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.550 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.550 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:39:35.550 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:09.400693+0000", 2024-02-05T06:39:35.550 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00086748000000000001 2024-02-05T06:39:35.550 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.550 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:39:35.550 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.550 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.550 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "MOSDPGPushReply(7.1 123/119 [PushReplyOp(7:84697eab:::benchmark_data_smithi028_77509_object79085:head)])", 2024-02-05T06:39:35.550 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:37:09.308907+0000", 2024-02-05T06:39:35.550 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 146.19361358099999, 2024-02-05T06:39:35.550 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.088655028999999996, 2024-02-05T06:39:35.550 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:39:35.551 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-02-05T06:39:35.551 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:39:35.551 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.551 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:39:35.551 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:09.308907+0000", 2024-02-05T06:39:35.551 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.551 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.551 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:39:35.551 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:09.308907+0000", 2024-02-05T06:39:35.551 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.551 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.551 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:39:35.552 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:09.308908+0000", 2024-02-05T06:39:35.552 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.39e-07 2024-02-05T06:39:35.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.552 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.552 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:39:35.552 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:09.308912+0000", 2024-02-05T06:39:35.552 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7840000000000001e-06 2024-02-05T06:39:35.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.552 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.552 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:39:35.552 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:09.308918+0000", 2024-02-05T06:39:35.553 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0299999999999999e-06 2024-02-05T06:39:35.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.553 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.553 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:39:35.553 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:09.308925+0000", 2024-02-05T06:39:35.553 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8979999999999999e-06 2024-02-05T06:39:35.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.553 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.553 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:39:35.553 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:09.309151+0000", 2024-02-05T06:39:35.553 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000226414 2024-02-05T06:39:35.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.553 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.554 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:39:35.554 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:09.397562+0000", 2024-02-05T06:39:35.554 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.088411463999999995 2024-02-05T06:39:35.554 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.554 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:39:35.554 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.554 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.554 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.554 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14847.0:116805 7.2 e123/96 7:43bee4d6:::benchmark_data_smithi028_77509_object1685:head v 123'7287, mlcod=123'7284)", 2024-02-05T06:39:35.554 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:37:09.310865+0000", 2024-02-05T06:39:35.554 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 146.191656421, 2024-02-05T06:39:35.554 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.089907580000000001, 2024-02-05T06:39:35.554 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:39:35.554 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:39:35.555 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:39:35.555 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.555 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:39:35.555 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:09.310865+0000", 2024-02-05T06:39:35.555 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.555 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.555 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:39:35.555 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:09.310865+0000", 2024-02-05T06:39:35.555 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.555 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.555 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:39:35.555 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:09.310866+0000", 2024-02-05T06:39:35.556 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.398e-06 2024-02-05T06:39:35.556 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.556 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.556 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:39:35.556 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:09.310873+0000", 2024-02-05T06:39:35.556 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8900000000000001e-06 2024-02-05T06:39:35.556 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.556 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.556 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:39:35.556 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:09.397356+0000", 2024-02-05T06:39:35.556 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.086483562999999999 2024-02-05T06:39:35.556 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.556 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.556 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:39:35.557 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:09.397376+0000", 2024-02-05T06:39:35.557 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9698999999999999e-05 2024-02-05T06:39:35.557 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.557 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.557 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:39:35.557 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:09.397453+0000", 2024-02-05T06:39:35.557 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6532999999999994e-05 2024-02-05T06:39:35.557 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.557 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.557 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:39:35.557 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:09.397482+0000", 2024-02-05T06:39:35.558 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9635999999999999e-05 2024-02-05T06:39:35.558 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.558 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.558 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:39:35.558 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:09.400699+0000", 2024-02-05T06:39:35.558 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032162969999999999 2024-02-05T06:39:35.558 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.558 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.558 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:39:35.559 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:09.400772+0000", 2024-02-05T06:39:35.559 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3564000000000002e-05 2024-02-05T06:39:35.559 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.559 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:39:35.559 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.559 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.559 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:117381 7.b 7:d649292c:::benchmark_data_smithi028_77509_object2261:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-02-05T06:39:35.560 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:37:10.308188+0000", 2024-02-05T06:39:35.560 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 145.194333435, 2024-02-05T06:39:35.560 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.134929304, 2024-02-05T06:39:35.560 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:39:35.560 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:39:35.560 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:39:35.560 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:39:35.560 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:39:35.561 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 117381 2024-02-05T06:39:35.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.561 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:39:35.561 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.561 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:39:35.561 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.308188+0000", 2024-02-05T06:39:35.561 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.562 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.562 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.562 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:39:35.562 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.308188+0000", 2024-02-05T06:39:35.562 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.562 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.563 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.563 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:39:35.563 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.308190+0000", 2024-02-05T06:39:35.563 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0470000000000001e-06 2024-02-05T06:39:35.563 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.563 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.563 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:39:35.564 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.308195+0000", 2024-02-05T06:39:35.564 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9450000000000001e-06 2024-02-05T06:39:35.564 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.564 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.564 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:39:35.564 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.308205+0000", 2024-02-05T06:39:35.564 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0735e-05 2024-02-05T06:39:35.565 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.565 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.565 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:39:35.565 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.308213+0000", 2024-02-05T06:39:35.565 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7819999999999994e-06 2024-02-05T06:39:35.565 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.565 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.565 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:39:35.566 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.309647+0000", 2024-02-05T06:39:35.566 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0014336539999999999 2024-02-05T06:39:35.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.566 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.566 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:39:35.566 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.309799+0000", 2024-02-05T06:39:35.566 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000151948 2024-02-05T06:39:35.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.567 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.567 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-02-05T06:39:35.567 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.309905+0000", 2024-02-05T06:39:35.567 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000106644 2024-02-05T06:39:35.567 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.567 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.567 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:39:35.568 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.315682+0000", 2024-02-05T06:39:35.568 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0057766980000000002 2024-02-05T06:39:35.568 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.568 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.568 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:39:35.568 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.442661+0000", 2024-02-05T06:39:35.569 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.12697856599999999 2024-02-05T06:39:35.569 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.569 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.569 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:39:35.569 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.443011+0000", 2024-02-05T06:39:35.569 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00035015799999999999 2024-02-05T06:39:35.570 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.570 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.570 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:39:35.570 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.443070+0000", 2024-02-05T06:39:35.570 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9723999999999999e-05 2024-02-05T06:39:35.570 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.570 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.570 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:39:35.570 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.443117+0000", 2024-02-05T06:39:35.570 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6403000000000003e-05 2024-02-05T06:39:35.571 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.571 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:39:35.571 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.571 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.571 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.571 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:117382 7.b 7:d29ec41a:::benchmark_data_smithi028_77509_object2262:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-02-05T06:39:35.571 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:37:10.308252+0000", 2024-02-05T06:39:35.572 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 145.19426853300001, 2024-02-05T06:39:35.572 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.135047902, 2024-02-05T06:39:35.572 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:39:35.572 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:39:35.572 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:39:35.572 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:39:35.572 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:39:35.572 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 117382 2024-02-05T06:39:35.572 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.572 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:39:35.573 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.573 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:39:35.573 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.308252+0000", 2024-02-05T06:39:35.573 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.573 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.573 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:39:35.573 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.308252+0000", 2024-02-05T06:39:35.573 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.573 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.573 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:39:35.574 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.308253+0000", 2024-02-05T06:39:35.574 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3399999999999997e-07 2024-02-05T06:39:35.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.574 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.574 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:39:35.574 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.308265+0000", 2024-02-05T06:39:35.574 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1881e-05 2024-02-05T06:39:35.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.574 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.574 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:39:35.574 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.308273+0000", 2024-02-05T06:39:35.574 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8820000000000001e-06 2024-02-05T06:39:35.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.574 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.574 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:39:35.575 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.308279+0000", 2024-02-05T06:39:35.575 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2360000000000001e-06 2024-02-05T06:39:35.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.575 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.575 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:39:35.575 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.312476+0000", 2024-02-05T06:39:35.575 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041967920000000004 2024-02-05T06:39:35.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.575 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.575 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:39:35.575 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.312590+0000", 2024-02-05T06:39:35.575 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011414800000000001 2024-02-05T06:39:35.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.575 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.575 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-02-05T06:39:35.576 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.312697+0000", 2024-02-05T06:39:35.576 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000107138 2024-02-05T06:39:35.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.576 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.576 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:39:35.576 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.315687+0000", 2024-02-05T06:39:35.576 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029894850000000001 2024-02-05T06:39:35.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.576 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.576 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:39:35.576 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.442714+0000", 2024-02-05T06:39:35.576 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.127026633 2024-02-05T06:39:35.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.577 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.577 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:39:35.577 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.443213+0000", 2024-02-05T06:39:35.577 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00049972299999999999 2024-02-05T06:39:35.577 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.577 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.577 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:39:35.577 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.443256+0000", 2024-02-05T06:39:35.577 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2378e-05 2024-02-05T06:39:35.577 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.577 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.578 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:39:35.578 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.443300+0000", 2024-02-05T06:39:35.578 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4672000000000002e-05 2024-02-05T06:39:35.578 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.578 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:39:35.578 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.579 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.579 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:117387 7.b 7:de710dfb:::benchmark_data_smithi028_77509_object2267:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-02-05T06:39:35.579 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:37:10.311855+0000", 2024-02-05T06:39:35.579 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 145.19066548199999, 2024-02-05T06:39:35.579 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.13160402399999999, 2024-02-05T06:39:35.579 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:39:35.579 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:39:35.579 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:39:35.579 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:39:35.579 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:39:35.580 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 117387 2024-02-05T06:39:35.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.580 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:39:35.580 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.580 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:39:35.580 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.311855+0000", 2024-02-05T06:39:35.580 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.580 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.580 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:39:35.580 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.311855+0000", 2024-02-05T06:39:35.580 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.581 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.581 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:39:35.581 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.311857+0000", 2024-02-05T06:39:35.581 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.629e-06 2024-02-05T06:39:35.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.581 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.581 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:39:35.581 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.311861+0000", 2024-02-05T06:39:35.581 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8820000000000003e-06 2024-02-05T06:39:35.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.582 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.582 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:39:35.582 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.311880+0000", 2024-02-05T06:39:35.582 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8938000000000001e-05 2024-02-05T06:39:35.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.582 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.582 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:39:35.582 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.311886+0000", 2024-02-05T06:39:35.582 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5189999999999998e-06 2024-02-05T06:39:35.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.582 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.582 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:39:35.582 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.315714+0000", 2024-02-05T06:39:35.582 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0038273019999999999 2024-02-05T06:39:35.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.583 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.583 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:39:35.583 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.315820+0000", 2024-02-05T06:39:35.583 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010644199999999999 2024-02-05T06:39:35.583 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.583 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.583 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-02-05T06:39:35.583 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.315922+0000", 2024-02-05T06:39:35.583 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010172 2024-02-05T06:39:35.583 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.583 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.584 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:39:35.584 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.442675+0000", 2024-02-05T06:39:35.584 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.126753012 2024-02-05T06:39:35.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.584 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.584 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:39:35.584 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.442749+0000", 2024-02-05T06:39:35.584 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3752999999999997e-05 2024-02-05T06:39:35.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.584 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.584 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:39:35.584 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.443386+0000", 2024-02-05T06:39:35.584 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00063715099999999997 2024-02-05T06:39:35.585 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.585 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.585 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:39:35.585 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.443428+0000", 2024-02-05T06:39:35.585 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2199e-05 2024-02-05T06:39:35.585 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.585 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.585 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:39:35.585 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.443459+0000", 2024-02-05T06:39:35.585 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1476999999999997e-05 2024-02-05T06:39:35.585 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.585 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:39:35.585 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.585 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.585 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.586 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(client.14847.0:117381 7.b e123/96 ondisk, result = 0)", 2024-02-05T06:39:35.586 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:37:10.318589+0000", 2024-02-05T06:39:35.586 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 145.18393173699999, 2024-02-05T06:39:35.586 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.12453316, 2024-02-05T06:39:35.586 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:39:35.586 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-02-05T06:39:35.586 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:39:35.586 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.586 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:39:35.586 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.318589+0000", 2024-02-05T06:39:35.587 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.587 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.587 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:39:35.587 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.318589+0000", 2024-02-05T06:39:35.587 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.587 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.587 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:39:35.587 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.318591+0000", 2024-02-05T06:39:35.587 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.409e-06 2024-02-05T06:39:35.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.587 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.587 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:39:35.588 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.318595+0000", 2024-02-05T06:39:35.588 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1450000000000001e-06 2024-02-05T06:39:35.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.588 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.588 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:39:35.588 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.442941+0000", 2024-02-05T06:39:35.588 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.124346717 2024-02-05T06:39:35.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.588 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.588 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:39:35.588 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.442950+0000", 2024-02-05T06:39:35.588 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7099999999999996e-06 2024-02-05T06:39:35.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.588 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.589 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:39:35.589 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.442998+0000", 2024-02-05T06:39:35.589 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8152000000000002e-05 2024-02-05T06:39:35.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.589 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.589 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:39:35.589 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.443005+0000", 2024-02-05T06:39:35.589 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1509999999999998e-06 2024-02-05T06:39:35.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.589 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.589 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:39:35.589 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.443122+0000", 2024-02-05T06:39:35.589 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000116876 2024-02-05T06:39:35.589 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.590 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:39:35.590 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.590 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.590 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(client.14847.0:117381 7.b e123/96 ondisk, result = 0)", 2024-02-05T06:39:35.590 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:37:10.318692+0000", 2024-02-05T06:39:35.590 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 145.18382942900001, 2024-02-05T06:39:35.590 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.123985584, 2024-02-05T06:39:35.590 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:39:35.590 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-02-05T06:39:35.590 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:39:35.590 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.590 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:39:35.590 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.318692+0000", 2024-02-05T06:39:35.590 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.591 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.591 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:39:35.591 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.318692+0000", 2024-02-05T06:39:35.591 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.591 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.591 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:39:35.592 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.318692+0000", 2024-02-05T06:39:35.592 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6700000000000003e-07 2024-02-05T06:39:35.592 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.592 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.592 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:39:35.592 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.318695+0000", 2024-02-05T06:39:35.592 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5359999999999999e-06 2024-02-05T06:39:35.592 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.592 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.592 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:39:35.592 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.442466+0000", 2024-02-05T06:39:35.593 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.123771245 2024-02-05T06:39:35.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.593 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.593 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:39:35.593 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.442489+0000", 2024-02-05T06:39:35.593 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3121000000000001e-05 2024-02-05T06:39:35.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.593 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.593 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:39:35.593 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.442642+0000", 2024-02-05T06:39:35.593 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015256199999999999 2024-02-05T06:39:35.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.593 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.593 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:39:35.594 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.442653+0000", 2024-02-05T06:39:35.594 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1531e-05 2024-02-05T06:39:35.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.594 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.594 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:39:35.594 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:10.442677+0000", 2024-02-05T06:39:35.594 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4022000000000001e-05 2024-02-05T06:39:35.594 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.594 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:39:35.594 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.594 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.594 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14847.0:130128 7.2 e126/96 7:4a5479fd:::benchmark_data_smithi028_77509_object15008:head v 126'8148, mlcod=126'8146)", 2024-02-05T06:39:35.594 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:37:25.291401+0000", 2024-02-05T06:39:35.594 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 130.211120212, 2024-02-05T06:39:35.595 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.077930872999999998, 2024-02-05T06:39:35.595 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:39:35.595 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:39:35.595 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:39:35.595 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.595 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:39:35.595 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:25.291401+0000", 2024-02-05T06:39:35.595 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.595 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.595 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:39:35.595 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:25.291401+0000", 2024-02-05T06:39:35.595 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.595 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.596 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:39:35.596 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:25.291402+0000", 2024-02-05T06:39:35.596 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3009999999999999e-06 2024-02-05T06:39:35.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.596 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.596 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:39:35.596 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:25.291409+0000", 2024-02-05T06:39:35.596 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3980000000000002e-06 2024-02-05T06:39:35.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.596 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.596 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:39:35.596 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:25.359220+0000", 2024-02-05T06:39:35.596 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.067810180999999997 2024-02-05T06:39:35.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.597 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.597 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:39:35.597 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:25.359246+0000", 2024-02-05T06:39:35.597 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6376e-05 2024-02-05T06:39:35.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.597 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.597 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:39:35.597 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:25.359420+0000", 2024-02-05T06:39:35.597 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000174022 2024-02-05T06:39:35.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.597 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.597 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:39:35.598 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:25.359481+0000", 2024-02-05T06:39:35.598 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0968999999999999e-05 2024-02-05T06:39:35.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.598 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.598 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:39:35.598 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:25.369293+0000", 2024-02-05T06:39:35.598 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0098116079999999994 2024-02-05T06:39:35.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.598 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.598 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:39:35.598 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:25.369332+0000", 2024-02-05T06:39:35.598 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9017999999999998e-05 2024-02-05T06:39:35.599 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.599 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:39:35.599 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.599 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.599 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:130135 7.1 7:8392266e:::benchmark_data_smithi028_77509_object15015:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-02-05T06:39:35.599 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:37:25.294795+0000", 2024-02-05T06:39:35.599 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 130.207726261, 2024-02-05T06:39:35.599 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.075811425000000002, 2024-02-05T06:39:35.599 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:39:35.599 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:39:35.599 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:39:35.599 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:39:35.599 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:39:35.599 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 130135 2024-02-05T06:39:35.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.600 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:39:35.600 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.600 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:39:35.600 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:25.294795+0000", 2024-02-05T06:39:35.600 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.600 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.600 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:39:35.600 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:25.294795+0000", 2024-02-05T06:39:35.600 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.600 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.601 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:39:35.601 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:25.294797+0000", 2024-02-05T06:39:35.601 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.018e-06 2024-02-05T06:39:35.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.601 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.601 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:39:35.601 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:25.294805+0000", 2024-02-05T06:39:35.601 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6389999999999998e-06 2024-02-05T06:39:35.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.601 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.601 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:39:35.601 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:25.361044+0000", 2024-02-05T06:39:35.601 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.066239078000000007 2024-02-05T06:39:35.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.601 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.602 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:39:35.602 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:25.361057+0000", 2024-02-05T06:39:35.602 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2322e-05 2024-02-05T06:39:35.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.602 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.602 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:39:35.602 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:25.362204+0000", 2024-02-05T06:39:35.602 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001147087 2024-02-05T06:39:35.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.602 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.602 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:39:35.602 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:25.362346+0000", 2024-02-05T06:39:35.602 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014173899999999999 2024-02-05T06:39:35.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.603 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.603 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4,6", 2024-02-05T06:39:35.603 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:25.362490+0000", 2024-02-05T06:39:35.603 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014443600000000001 2024-02-05T06:39:35.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.603 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.603 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:39:35.603 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:25.366699+0000", 2024-02-05T06:39:35.603 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042092930000000002 2024-02-05T06:39:35.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.603 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.603 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:39:35.603 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:25.366955+0000", 2024-02-05T06:39:35.603 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00025538399999999999 2024-02-05T06:39:35.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.604 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.604 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:39:35.604 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:25.367005+0000", 2024-02-05T06:39:35.604 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0587999999999997e-05 2024-02-05T06:39:35.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.604 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.604 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:39:35.604 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:25.370403+0000", 2024-02-05T06:39:35.604 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003397438 2024-02-05T06:39:35.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.604 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.604 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:39:35.604 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:25.370474+0000", 2024-02-05T06:39:35.604 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1771000000000001e-05 2024-02-05T06:39:35.605 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.605 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.605 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:39:35.605 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:25.370606+0000", 2024-02-05T06:39:35.605 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013163199999999999 2024-02-05T06:39:35.605 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.605 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:39:35.605 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.605 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.605 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.605 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:138497 7.1 7:886e25bc:::benchmark_data_smithi028_77509_object23377:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-02-05T06:39:35.605 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:37:35.084329+0000", 2024-02-05T06:39:35.605 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 120.41819221999999, 2024-02-05T06:39:35.605 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.085113805000000001, 2024-02-05T06:39:35.605 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:39:35.606 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:39:35.606 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:39:35.606 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:39:35.606 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:39:35.606 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 138497 2024-02-05T06:39:35.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.606 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:39:35.606 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.606 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:39:35.606 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:35.084329+0000", 2024-02-05T06:39:35.606 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.607 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:39:35.607 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:35.084329+0000", 2024-02-05T06:39:35.607 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.607 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:39:35.607 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:35.084331+0000", 2024-02-05T06:39:35.607 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1459999999999999e-06 2024-02-05T06:39:35.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.607 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:39:35.607 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:35.084337+0000", 2024-02-05T06:39:35.607 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5540000000000003e-06 2024-02-05T06:39:35.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.608 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.608 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:39:35.608 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:35.084352+0000", 2024-02-05T06:39:35.608 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4367e-05 2024-02-05T06:39:35.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.608 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.608 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:39:35.608 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:35.084495+0000", 2024-02-05T06:39:35.608 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014326700000000001 2024-02-05T06:39:35.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.608 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.608 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:39:35.608 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:35.085654+0000", 2024-02-05T06:39:35.609 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0011592760000000001 2024-02-05T06:39:35.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.609 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.609 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:39:35.609 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:35.085810+0000", 2024-02-05T06:39:35.609 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015578799999999999 2024-02-05T06:39:35.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.609 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.609 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4,6", 2024-02-05T06:39:35.609 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:35.150511+0000", 2024-02-05T06:39:35.609 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.064700522999999996 2024-02-05T06:39:35.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.609 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.609 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:39:35.609 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:35.158813+0000", 2024-02-05T06:39:35.610 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0083023439999999997 2024-02-05T06:39:35.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.610 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.610 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:39:35.610 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:35.164570+0000", 2024-02-05T06:39:35.610 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0057574150000000001 2024-02-05T06:39:35.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.610 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.610 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:39:35.610 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:35.164672+0000", 2024-02-05T06:39:35.610 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010113 2024-02-05T06:39:35.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.610 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.610 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:39:35.611 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:35.169222+0000", 2024-02-05T06:39:35.611 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0045500599999999999 2024-02-05T06:39:35.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.611 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:39:35.611 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:35.169305+0000", 2024-02-05T06:39:35.611 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3270999999999996e-05 2024-02-05T06:39:35.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.611 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:39:35.611 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:35.169443+0000", 2024-02-05T06:39:35.611 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013766399999999999 2024-02-05T06:39:35.611 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.611 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:39:35.611 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.612 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14568.0:273 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e151)", 2024-02-05T06:39:35.612 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:15.680881+0000", 2024-02-05T06:39:35.612 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 79.821640177000006, 2024-02-05T06:39:35.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.92211636600000002, 2024-02-05T06:39:35.612 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:39:35.612 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-02-05T06:39:35.612 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:39:35.612 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14568", 2024-02-05T06:39:35.612 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.167:0/939366379", 2024-02-05T06:39:35.612 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 273 2024-02-05T06:39:35.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.612 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:39:35.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:39:35.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.680881+0000", 2024-02-05T06:39:35.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:39:35.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.680881+0000", 2024-02-05T06:39:35.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:39:35.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.680883+0000", 2024-02-05T06:39:35.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3130000000000001e-06 2024-02-05T06:39:35.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:39:35.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.680891+0000", 2024-02-05T06:39:35.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1389999999999995e-06 2024-02-05T06:39:35.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:39:35.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.680906+0000", 2024-02-05T06:39:35.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5218000000000001e-05 2024-02-05T06:39:35.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:39:35.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.680918+0000", 2024-02-05T06:39:35.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1949e-05 2024-02-05T06:39:35.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.615 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:39:35.615 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.680990+0000", 2024-02-05T06:39:35.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1134e-05 2024-02-05T06:39:35.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.615 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:39:35.615 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.681002+0000", 2024-02-05T06:39:35.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2743e-05 2024-02-05T06:39:35.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.615 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:39:35.615 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:16.602683+0000", 2024-02-05T06:39:35.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.92168123499999999 2024-02-05T06:39:35.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.616 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:39:35.616 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:16.602846+0000", 2024-02-05T06:39:35.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016218800000000001 2024-02-05T06:39:35.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.616 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:39:35.616 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:16.602997+0000", 2024-02-05T06:39:35.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000151447 2024-02-05T06:39:35.616 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.616 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:39:35.616 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.617 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.617 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14847.0:197071 7.4 e166/138 7:2c33cc13:::benchmark_data_smithi028_77509_object81951:head v 166'12393, mlcod=166'12391)", 2024-02-05T06:39:35.617 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:45.303328+0000", 2024-02-05T06:39:35.617 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.199192474999997, 2024-02-05T06:39:35.617 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.091991096999999994, 2024-02-05T06:39:35.617 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:39:35.617 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:39:35.617 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:39:35.617 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.617 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:39:35.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:45.303328+0000", 2024-02-05T06:39:35.618 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.618 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:39:35.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:45.303328+0000", 2024-02-05T06:39:35.618 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.618 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:39:35.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:45.303337+0000", 2024-02-05T06:39:35.618 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0189999999999995e-06 2024-02-05T06:39:35.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:39:35.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:45.303345+0000", 2024-02-05T06:39:35.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.374e-06 2024-02-05T06:39:35.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:39:35.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:45.303355+0000", 2024-02-05T06:39:35.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0057999999999999e-05 2024-02-05T06:39:35.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:39:35.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:45.303362+0000", 2024-02-05T06:39:35.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2930000000000003e-06 2024-02-05T06:39:35.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:39:35.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:45.303441+0000", 2024-02-05T06:39:35.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9169e-05 2024-02-05T06:39:35.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:39:35.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:45.303464+0000", 2024-02-05T06:39:35.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2464999999999999e-05 2024-02-05T06:39:35.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.621 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:39:35.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:45.395246+0000", 2024-02-05T06:39:35.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.091782569999999994 2024-02-05T06:39:35.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.621 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:39:35.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:45.395320+0000", 2024-02-05T06:39:35.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3149e-05 2024-02-05T06:39:35.622 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.622 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:39:35.622 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.622 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14847.0:197075 7.9 e166/137 7:973f8045:::benchmark_data_smithi028_77509_object81955:head v 166'12254, mlcod=166'12252)", 2024-02-05T06:39:35.622 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:45.304607+0000", 2024-02-05T06:39:35.622 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.197914167, 2024-02-05T06:39:35.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.10754852299999999, 2024-02-05T06:39:35.622 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:39:35.622 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:39:35.623 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:39:35.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:39:35.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:45.304607+0000", 2024-02-05T06:39:35.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:39:35.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:45.304607+0000", 2024-02-05T06:39:35.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:39:35.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:45.304608+0000", 2024-02-05T06:39:35.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2890000000000001e-06 2024-02-05T06:39:35.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.624 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:39:35.625 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:45.304616+0000", 2024-02-05T06:39:35.625 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5830000000000001e-06 2024-02-05T06:39:35.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.625 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:39:35.625 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:45.304627+0000", 2024-02-05T06:39:35.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0869999999999999e-05 2024-02-05T06:39:35.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.626 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:39:35.626 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:45.304634+0000", 2024-02-05T06:39:35.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8900000000000007e-06 2024-02-05T06:39:35.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:39:35.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:45.304757+0000", 2024-02-05T06:39:35.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012225800000000001 2024-02-05T06:39:35.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:39:35.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:45.304777+0000", 2024-02-05T06:39:35.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0387e-05 2024-02-05T06:39:35.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:39:35.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:45.411992+0000", 2024-02-05T06:39:35.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.10721539400000001 2024-02-05T06:39:35.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:39:35.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:45.412155+0000", 2024-02-05T06:39:35.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016285200000000001 2024-02-05T06:39:35.629 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.629 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:39:35.629 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.629 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.14847.0:197077 7.5 e166/165 7:a4e377b2:::benchmark_data_smithi028_77509_object81957:head v 166'16233, mlcod=166'16231)", 2024-02-05T06:39:35.629 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:45.304977+0000", 2024-02-05T06:39:35.629 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 50.197543676000002, 2024-02-05T06:39:35.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.105419047, 2024-02-05T06:39:35.629 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:39:35.629 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:39:35.629 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:39:35.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:39:35.630 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:45.304977+0000", 2024-02-05T06:39:35.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:39:35.630 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:45.304977+0000", 2024-02-05T06:39:35.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:39:35.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:39:35.630 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:45.304979+0000", 2024-02-05T06:39:35.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.33e-06 2024-02-05T06:39:35.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:39:35.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:45.304985+0000", 2024-02-05T06:39:35.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6279999999999999e-06 2024-02-05T06:39:35.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:39:35.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:45.304995+0000", 2024-02-05T06:39:35.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0068000000000001e-05 2024-02-05T06:39:35.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:39:35.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:45.305003+0000", 2024-02-05T06:39:35.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8730000000000007e-06 2024-02-05T06:39:35.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:39:35.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:45.305053+0000", 2024-02-05T06:39:35.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0071999999999998e-05 2024-02-05T06:39:35.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:39:35.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:45.305073+0000", 2024-02-05T06:39:35.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0003000000000001e-05 2024-02-05T06:39:35.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:39:35.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:45.410249+0000", 2024-02-05T06:39:35.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.10517536199999999 2024-02-05T06:39:35.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:39:35.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:39:35.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:39:35.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:45.410396+0000", 2024-02-05T06:39:35.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000147711 2024-02-05T06:39:35.633 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.633 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:39:35.633 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.633 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:39:35.633 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:39:35.633 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:39:35.649 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:35.677 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:35.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:35.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:35.779 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:35.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:35.804 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:35.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:35.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:35.907 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-05T06:39:35.908 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-05T06:39:35.909 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph tell osd.4 'trim stale osdmaps' 2024-02-05T06:39:35.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:35.929 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:35.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:36.031 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:36.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:36.061 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:36.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:36.155 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:36 smithi028 ceph-mon[29442]: 3.1c scrub starts 2024-02-05T06:39:36.155 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:36 smithi028 ceph-mon[29442]: 3.1c scrub ok 2024-02-05T06:39:36.155 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:36 smithi028 ceph-mon[29442]: 4.a scrub starts 2024-02-05T06:39:36.155 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:36 smithi028 ceph-mon[29442]: 4.a scrub ok 2024-02-05T06:39:36.156 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:36 smithi028 ceph-mon[29442]: 5.a scrub starts 2024-02-05T06:39:36.156 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:36 smithi028 ceph-mon[29442]: 5.a scrub ok 2024-02-05T06:39:36.156 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:36 smithi028 ceph-mon[29442]: 3.3 scrub starts 2024-02-05T06:39:36.156 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:36 smithi028 ceph-mon[29442]: 3.3 scrub ok 2024-02-05T06:39:36.156 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:36 smithi028 ceph-mon[29442]: 5.12 scrub starts 2024-02-05T06:39:36.156 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:36 smithi028 ceph-mon[29442]: 5.12 scrub ok 2024-02-05T06:39:36.156 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:36 smithi028 ceph-mon[29442]: 4.19 deep-scrub starts 2024-02-05T06:39:36.156 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:36 smithi028 ceph-mon[29442]: 4.19 deep-scrub ok 2024-02-05T06:39:36.156 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:36 smithi028 ceph-mon[37984]: 3.1c scrub starts 2024-02-05T06:39:36.156 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:36 smithi028 ceph-mon[37984]: 3.1c scrub ok 2024-02-05T06:39:36.157 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:36 smithi028 ceph-mon[37984]: 4.a scrub starts 2024-02-05T06:39:36.157 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:36 smithi028 ceph-mon[37984]: 4.a scrub ok 2024-02-05T06:39:36.157 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:36 smithi028 ceph-mon[37984]: 5.a scrub starts 2024-02-05T06:39:36.157 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:36 smithi028 ceph-mon[37984]: 5.a scrub ok 2024-02-05T06:39:36.157 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:36 smithi028 ceph-mon[37984]: 3.3 scrub starts 2024-02-05T06:39:36.157 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:36 smithi028 ceph-mon[37984]: 3.3 scrub ok 2024-02-05T06:39:36.157 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:36 smithi028 ceph-mon[37984]: 5.12 scrub starts 2024-02-05T06:39:36.157 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:36 smithi028 ceph-mon[37984]: 5.12 scrub ok 2024-02-05T06:39:36.157 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:36 smithi028 ceph-mon[37984]: 4.19 deep-scrub starts 2024-02-05T06:39:36.157 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:36 smithi028 ceph-mon[37984]: 4.19 deep-scrub ok 2024-02-05T06:39:36.162 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:36.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:36.187 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:36.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:36.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:36.311 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:36.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:36.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:36.413 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:36.441 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:36.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:36.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:36.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:36 smithi167 ceph-mon[32774]: 3.1c scrub starts 2024-02-05T06:39:36.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:36 smithi167 ceph-mon[32774]: 3.1c scrub ok 2024-02-05T06:39:36.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:36 smithi167 ceph-mon[32774]: 4.a scrub starts 2024-02-05T06:39:36.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:36 smithi167 ceph-mon[32774]: 4.a scrub ok 2024-02-05T06:39:36.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:36 smithi167 ceph-mon[32774]: 5.a scrub starts 2024-02-05T06:39:36.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:36 smithi167 ceph-mon[32774]: 5.a scrub ok 2024-02-05T06:39:36.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:36 smithi167 ceph-mon[32774]: 3.3 scrub starts 2024-02-05T06:39:36.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:36 smithi167 ceph-mon[32774]: 3.3 scrub ok 2024-02-05T06:39:36.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:36 smithi167 ceph-mon[32774]: 5.12 scrub starts 2024-02-05T06:39:36.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:36 smithi167 ceph-mon[32774]: 5.12 scrub ok 2024-02-05T06:39:36.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:36 smithi167 ceph-mon[32774]: 4.19 deep-scrub starts 2024-02-05T06:39:36.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:36 smithi167 ceph-mon[32774]: 4.19 deep-scrub ok 2024-02-05T06:39:36.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:36.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:36.568 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:36.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:36.642 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:39:36.669 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:36.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:36.710 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:36.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:36.811 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:36.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:36.835 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:36.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:36.936 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:36.964 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:36.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:36.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:37.002 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-05T06:39:37.019 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-02-05T06:39:37.066 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:37.093 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:37.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:37.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:37.195 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:37.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:37.219 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:37.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:37.261 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:37 smithi028 ceph-mon[29442]: 3.a scrub starts 2024-02-05T06:39:37.261 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:37 smithi028 ceph-mon[29442]: 3.a scrub ok 2024-02-05T06:39:37.262 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:37 smithi028 ceph-mon[29442]: 5.8 scrub starts 2024-02-05T06:39:37.262 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:37 smithi028 ceph-mon[29442]: 5.8 scrub ok 2024-02-05T06:39:37.262 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:37 smithi028 ceph-mon[29442]: pgmap v266: 142 pgs: 142 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 593 B/s rd, 0 op/s 2024-02-05T06:39:37.262 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:37 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/646198156' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:39:37.262 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:37 smithi028 ceph-mon[37984]: 3.a scrub starts 2024-02-05T06:39:37.263 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:37 smithi028 ceph-mon[37984]: 3.a scrub ok 2024-02-05T06:39:37.263 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:37 smithi028 ceph-mon[37984]: 5.8 scrub starts 2024-02-05T06:39:37.263 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:37 smithi028 ceph-mon[37984]: 5.8 scrub ok 2024-02-05T06:39:37.263 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:37 smithi028 ceph-mon[37984]: pgmap v266: 142 pgs: 142 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 593 B/s rd, 0 op/s 2024-02-05T06:39:37.264 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:37 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/646198156' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:39:37.321 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:37.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:37.345 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:37.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:37.447 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:37.476 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:37.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:37.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:37.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:37 smithi167 ceph-mon[32774]: 3.a scrub starts 2024-02-05T06:39:37.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:37 smithi167 ceph-mon[32774]: 3.a scrub ok 2024-02-05T06:39:37.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:37 smithi167 ceph-mon[32774]: 5.8 scrub starts 2024-02-05T06:39:37.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:37 smithi167 ceph-mon[32774]: 5.8 scrub ok 2024-02-05T06:39:37.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:37 smithi167 ceph-mon[32774]: pgmap v266: 142 pgs: 142 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 593 B/s rd, 0 op/s 2024-02-05T06:39:37.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:37 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/646198156' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:39:37.578 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:37.606 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:37.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:37.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:37.708 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:37.734 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:37.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:37.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:37.836 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:37.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:37.861 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:37.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:37.962 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:37.987 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:37.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:37.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:38.088 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:38.117 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:38.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:38.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:38.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:38 smithi028 ceph-mon[29442]: 6.5 scrub starts 2024-02-05T06:39:38.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:38 smithi028 ceph-mon[29442]: 6.5 scrub ok 2024-02-05T06:39:38.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:38 smithi028 ceph-mon[29442]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:39:38.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:38 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/646198156' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:39:38.176 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:38 smithi028 ceph-mon[29442]: osdmap e183: 8 total, 8 up, 7 in 2024-02-05T06:39:38.176 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:38 smithi028 ceph-mon[37984]: 6.5 scrub starts 2024-02-05T06:39:38.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:38 smithi028 ceph-mon[37984]: 6.5 scrub ok 2024-02-05T06:39:38.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:38 smithi028 ceph-mon[37984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:39:38.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:38 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/646198156' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:39:38.177 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:38 smithi028 ceph-mon[37984]: osdmap e183: 8 total, 8 up, 7 in 2024-02-05T06:39:38.218 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:38.246 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:38.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:38.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:38.348 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:38.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:38.373 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:38.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:38.474 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:38.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:38 smithi167 ceph-mon[32774]: 6.5 scrub starts 2024-02-05T06:39:38.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:38 smithi167 ceph-mon[32774]: 6.5 scrub ok 2024-02-05T06:39:38.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:38 smithi167 ceph-mon[32774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:39:38.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:38 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/646198156' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:39:38.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:38 smithi167 ceph-mon[32774]: osdmap e183: 8 total, 8 up, 7 in 2024-02-05T06:39:38.499 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:38.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:38.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:38.600 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:38.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:38.625 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:38.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:38.679 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:39:38.683 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:39:38.684 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:39:38.726 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:38.755 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:38.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:38.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:38.856 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:38.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:38.888 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:38.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:38.989 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:39.016 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:39.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:39.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:39.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:39.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:39.143 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:39.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:39.164 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:39 smithi028 ceph-mon[29442]: 4.1f scrub starts 2024-02-05T06:39:39.164 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:39 smithi028 ceph-mon[29442]: pgmap v268: 142 pgs: 142 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:39:39.165 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:39 smithi028 ceph-mon[37984]: 4.1f scrub starts 2024-02-05T06:39:39.165 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:39 smithi028 ceph-mon[37984]: pgmap v268: 142 pgs: 142 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:39:39.244 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:39.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:39.271 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:39.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:39.372 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:39.396 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:39.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:39.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:39.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:39 smithi167 ceph-mon[32774]: 4.1f scrub starts 2024-02-05T06:39:39.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:39 smithi167 ceph-mon[32774]: pgmap v268: 142 pgs: 142 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:39:39.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:39.521 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:39.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:39.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:39.622 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:39.647 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:39.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:39.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:39.749 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:39.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:39.774 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:39.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:39.875 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:39.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:39.904 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:39.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:40.005 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:40.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:40.030 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:40.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:40.131 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:40.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:40.160 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:40.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:40.203 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T06:39:40.261 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:40.286 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:40.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:40.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:40.388 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:40.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:40.417 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:40.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:40.519 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:40.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:40.545 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:40.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:40.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:40.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:40.669 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:40.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:40.770 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:40.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:40.795 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:40.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:40.896 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:40.927 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:40.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:40.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:41.007 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'unique_pool_1' created 2024-02-05T06:39:41.029 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:41.055 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:41.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:41.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:41.103 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:40 smithi028 ceph-mon[37984]: osdmap e184: 8 total, 8 up, 7 in 2024-02-05T06:39:41.103 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:40 smithi028 ceph-mon[37984]: pgmap v270: 142 pgs: 142 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:39:41.103 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:40 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2156117691' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-05T06:39:41.103 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:40 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:39:41.103 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:40 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:39:41.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:40 smithi028 ceph-mon[29442]: osdmap e184: 8 total, 8 up, 7 in 2024-02-05T06:39:41.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:40 smithi028 ceph-mon[29442]: pgmap v270: 142 pgs: 142 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:39:41.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:40 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2156117691' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-05T06:39:41.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:40 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:39:41.104 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:40 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:39:41.157 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:41.206 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:41.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:41.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:41.308 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:41.336 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:41.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:41.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:41.438 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:41.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:41.466 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:41.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:41.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:40 smithi167 ceph-mon[32774]: osdmap e184: 8 total, 8 up, 7 in 2024-02-05T06:39:41.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:40 smithi167 ceph-mon[32774]: pgmap v270: 142 pgs: 142 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:39:41.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:40 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2156117691' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-02-05T06:39:41.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:40 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:39:41.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:40 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:39:41.568 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:41.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:41.596 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:41.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:41.697 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:41.723 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:41.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:41.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:41.825 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:41.854 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:41.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:41.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:41.956 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:41.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:41.985 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:41.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:42.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:42.110 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:42.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:42.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:42.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:41 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:39:42.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:41 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:39:42.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:41 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2156117691' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-02-05T06:39:42.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:41 smithi028 ceph-mon[29442]: osdmap e185: 8 total, 8 up, 7 in 2024-02-05T06:39:42.186 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:41 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:39:42.186 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:41 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:39:42.186 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:41 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2156117691' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-02-05T06:39:42.187 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:41 smithi028 ceph-mon[37984]: osdmap e185: 8 total, 8 up, 7 in 2024-02-05T06:39:42.212 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:42.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:42.237 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:42.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:42.338 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:42.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:42.367 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:42.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:42.468 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:42.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:41 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:39:42.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:41 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:39:42.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:41 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2156117691' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-02-05T06:39:42.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:41 smithi167 ceph-mon[32774]: osdmap e185: 8 total, 8 up, 7 in 2024-02-05T06:39:42.496 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:42.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:42.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:42.598 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:42.626 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:42.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:42.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:42.728 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:42.754 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:39:42.755 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:42.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:42.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:42.857 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:42.866 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:39:42 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:39:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:39:42.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:42.886 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:42.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:42.981 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:39:42 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:39:42.988 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:43.009 INFO:teuthology.orchestra.run.smithi028.stderr:no valid command found; 10 closest matches: 2024-02-05T06:39:43.009 INFO:teuthology.orchestra.run.smithi028.stderr:0 2024-02-05T06:39:43.009 INFO:teuthology.orchestra.run.smithi028.stderr:1 2024-02-05T06:39:43.009 INFO:teuthology.orchestra.run.smithi028.stderr:2 2024-02-05T06:39:43.009 INFO:teuthology.orchestra.run.smithi028.stderr:abort 2024-02-05T06:39:43.009 INFO:teuthology.orchestra.run.smithi028.stderr:assert 2024-02-05T06:39:43.010 INFO:teuthology.orchestra.run.smithi028.stderr:bench [] [] [] [] 2024-02-05T06:39:43.010 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs debug_inject_read_zeros 2024-02-05T06:39:43.010 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs files list 2024-02-05T06:39:43.010 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs stats 2024-02-05T06:39:43.010 INFO:teuthology.orchestra.run.smithi028.stderr:bluestore allocator dump block 2024-02-05T06:39:43.010 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: invalid command 2024-02-05T06:39:43.015 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:43.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:43.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:43.117 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:43.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:43 smithi028 ceph-mon[29442]: osdmap e186: 8 total, 8 up, 7 in 2024-02-05T06:39:43.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:43 smithi028 ceph-mon[29442]: pgmap v273: 158 pgs: 16 creating+peering, 142 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 973 B/s rd, 0 op/s 2024-02-05T06:39:43.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:43 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:39:43.142 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:43 smithi028 ceph-mon[37984]: osdmap e186: 8 total, 8 up, 7 in 2024-02-05T06:39:43.142 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:43 smithi028 ceph-mon[37984]: pgmap v273: 158 pgs: 16 creating+peering, 142 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 973 B/s rd, 0 op/s 2024-02-05T06:39:43.142 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:43 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:39:43.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:43.143 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:43.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:43.244 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:43.272 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:43.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:43.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:43.374 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:43.398 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:43.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:43.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:43.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:43 smithi167 ceph-mon[32774]: osdmap e186: 8 total, 8 up, 7 in 2024-02-05T06:39:43.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:43 smithi167 ceph-mon[32774]: pgmap v273: 158 pgs: 16 creating+peering, 142 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 973 B/s rd, 0 op/s 2024-02-05T06:39:43.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:43 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:39:43.500 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:43.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:43.524 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:43.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:43.625 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:43.653 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:43.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:43.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:43.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:43.756 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-05T06:39:43.757 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-05T06:39:43.821 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:43.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:43.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:43.923 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:43.947 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:43.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:43.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:44.049 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:44.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:44.075 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:44.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:44.176 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:44.184 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:39:44.184 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-02-05T06:39:44.184 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-02-05T06:39:44.185 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:39:44.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:44 smithi028 ceph-mon[29442]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T06:39:44.185 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:44 smithi028 ceph-mon[29442]: osdmap e187: 8 total, 8 up, 7 in 2024-02-05T06:39:44.185 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:44 smithi028 ceph-mon[37984]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T06:39:44.185 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:44 smithi028 ceph-mon[37984]: osdmap e187: 8 total, 8 up, 7 in 2024-02-05T06:39:44.201 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:44.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:44.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:44.302 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:44.330 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:44.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:44.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:44.433 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:44.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:44.457 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:44.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:44.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:44 smithi167 ceph-mon[32774]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T06:39:44.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:44 smithi167 ceph-mon[32774]: osdmap e187: 8 total, 8 up, 7 in 2024-02-05T06:39:44.558 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:44.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:44.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:44.684 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:44.708 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:44.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:44.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:44.809 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:44.838 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:44.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:44.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:44.882 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-05T06:39:44.940 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:44.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:44.969 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:44.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:45.070 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:45.098 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:45.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:45.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:45.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:45 smithi028 ceph-mon[29442]: pgmap v275: 158 pgs: 16 creating+peering, 142 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:39:45.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:45 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:39:45.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:45 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:39:45.196 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:45 smithi028 ceph-mon[29442]: osdmap e188: 8 total, 8 up, 7 in 2024-02-05T06:39:45.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:45 smithi028 ceph-mon[37984]: pgmap v275: 158 pgs: 16 creating+peering, 142 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:39:45.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:45 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:39:45.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:45 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:39:45.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:45 smithi028 ceph-mon[37984]: osdmap e188: 8 total, 8 up, 7 in 2024-02-05T06:39:45.199 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:45.227 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:45.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:45.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:45.329 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:45.356 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:45.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:45.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:45.458 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:45.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:45 smithi167 ceph-mon[32774]: pgmap v275: 158 pgs: 16 creating+peering, 142 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:39:45.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:45 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:39:45.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:45 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:39:45.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:45 smithi167 ceph-mon[32774]: osdmap e188: 8 total, 8 up, 7 in 2024-02-05T06:39:45.485 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:45.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:45.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:45.586 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:45.612 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:45.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:45.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:45.714 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:45.717 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-02-05T06:39:45.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:45.740 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:45.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:45.841 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:45.869 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:45.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:45.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:45.971 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:45.999 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:46.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:46.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:46.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:46.125 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:46.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:46.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:46.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:46.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:46.252 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:46.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:46.353 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:46.376 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:46.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:46.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:46.478 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:46.506 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:46.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:46.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:46.608 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:46.637 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:46.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:46.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:46.738 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:46.766 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:46.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:46.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:46.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:46.892 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:46.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:46.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:46.994 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:47.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:47.019 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:47.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:47.121 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:47.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:47.145 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:47.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:47.246 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:47.269 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:47.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:47.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:47.371 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:47.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:47.401 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:47.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:47.503 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:47.531 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:47.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:47.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:47.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:47 smithi028 ceph-mon[29442]: pgmap v277: 158 pgs: 10 creating+peering, 148 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 971 B/s rd, 0 op/s 2024-02-05T06:39:47.624 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:47 smithi028 ceph-mon[37984]: pgmap v277: 158 pgs: 10 creating+peering, 148 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 971 B/s rd, 0 op/s 2024-02-05T06:39:47.633 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:47.660 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:47.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:47.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:47.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:47 smithi167 ceph-mon[32774]: pgmap v277: 158 pgs: 10 creating+peering, 148 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 971 B/s rd, 0 op/s 2024-02-05T06:39:47.752 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:39:47.763 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:47.771 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:39:47.786 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:47.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:47.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:47.791 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:39:47.811 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:39:47.831 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:39:47.851 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:39:47.872 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:39:47.888 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:47.893 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:39:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:39:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:39:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:39:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:39:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:39:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:39:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:39:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:39:47.912 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:47.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:47.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:48.014 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:48.042 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:48.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:48.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:48.144 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:48.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:48.174 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:48.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:48.275 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:48.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:48.304 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:48.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:48.405 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:48.428 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:39:48.431 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:48.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:48.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:48.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:48 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2861640417' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:39:48.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:48 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2861640417' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:39:48.534 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:48.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:48.560 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:48.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:48.661 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:48.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:48.685 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:48.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:48.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:48 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2861640417' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:39:48.758 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 4] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T06:39:48.758 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T06:39:48.759 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-02-05T06:39:48.759 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-05T06:39:48.759 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-05T06:39:48.759 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-02-05T06:39:48.786 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:48.828 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:48.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:48.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:48.869 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:39:48.930 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:48.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:48.954 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:48.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:49.012 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:39:49.055 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:49.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:49.080 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:49.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:49.181 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:49.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:49.205 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:49.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:49.306 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:49.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:49.332 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:49.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:49.341 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-02-05T06:39:49.433 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:49.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:49 smithi028 ceph-mon[29442]: pgmap v278: 158 pgs: 158 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 816 B/s rd, 0 op/s 2024-02-05T06:39:49.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:49 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1233064873' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:39:49.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:49 smithi028 ceph-mon[29442]: from='client.45017 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:39:49.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:49 smithi028 ceph-mon[37984]: pgmap v278: 158 pgs: 158 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 816 B/s rd, 0 op/s 2024-02-05T06:39:49.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:49 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1233064873' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:39:49.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:49 smithi028 ceph-mon[37984]: from='client.45017 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:39:49.451 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:49 smithi167 ceph-mon[32774]: pgmap v278: 158 pgs: 158 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 816 B/s rd, 0 op/s 2024-02-05T06:39:49.452 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:49 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1233064873' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:39:49.452 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:49 smithi167 ceph-mon[32774]: from='client.45017 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:39:49.458 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:49.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:49.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:49.560 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:49.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:49.585 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:49.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:49.686 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:49.709 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:49.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:49.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:49.811 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:49.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:49.836 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:49.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:49.938 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:49.961 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:49.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:49.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:50.063 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:50.087 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:50.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:50.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:50.189 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:50.214 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:50.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:50.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:50.316 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:50.342 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:50.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:50.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:50.416 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:50 smithi167 ceph-mon[32774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:39:50.416 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:50 smithi167 ceph-mon[32774]: from='client.45017 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:39:50.416 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:50 smithi167 ceph-mon[32774]: osdmap e189: 8 total, 8 up, 7 in 2024-02-05T06:39:50.416 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:50 smithi167 ceph-mon[32774]: osdmap e190: 8 total, 8 up, 7 in 2024-02-05T06:39:50.444 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:50.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:50 smithi028 ceph-mon[29442]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:39:50.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:50 smithi028 ceph-mon[29442]: from='client.45017 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:39:50.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:50 smithi028 ceph-mon[29442]: osdmap e189: 8 total, 8 up, 7 in 2024-02-05T06:39:50.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:50 smithi028 ceph-mon[29442]: osdmap e190: 8 total, 8 up, 7 in 2024-02-05T06:39:50.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:50 smithi028 ceph-mon[37984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:39:50.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:50 smithi028 ceph-mon[37984]: from='client.45017 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:39:50.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:50 smithi028 ceph-mon[37984]: osdmap e189: 8 total, 8 up, 7 in 2024-02-05T06:39:50.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:50 smithi028 ceph-mon[37984]: osdmap e190: 8 total, 8 up, 7 in 2024-02-05T06:39:50.468 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:50.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:50.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:50.570 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:50.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:50.613 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:50.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:50.714 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:50.737 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:50.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:50.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:50.812 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:39:50.812 INFO:teuthology.orchestra.run.smithi167.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-02-05T06:39:50.812 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:39:50.840 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:50.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:50.864 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:50.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:50.884 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:39:50.965 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:50.992 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:50.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:50.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:51.094 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:51.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:51.119 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:51.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:51.220 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:51.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:51.245 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:51.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:51.346 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:51.366 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:51 smithi167 ceph-mon[32774]: pgmap v281: 158 pgs: 158 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:39:51.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:51.372 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:51.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:51.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:51.497 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:51.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:51.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:51.600 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:51.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:51.625 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:51.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:51.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:51 smithi028 ceph-mon[29442]: pgmap v281: 158 pgs: 158 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:39:51.660 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:51 smithi028 ceph-mon[37984]: pgmap v281: 158 pgs: 158 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:39:51.726 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:51.751 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:51.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:51.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:51.854 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:51.879 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:51.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:51.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:51.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:52.004 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:52.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:52.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:52.106 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:52.129 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:52.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:52.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:52.231 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:52.254 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:52.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:52.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:52.356 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:52.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:52.380 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:52.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:52.481 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:52.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:52.507 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:52.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:52.608 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:52.632 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:52.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:52.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:52.679 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:39:52.680 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:39:52.734 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:52.758 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:52.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:52.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:52.859 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:52.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:52.884 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:52.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:52.946 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:39:52 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:39:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:39:52.981 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:39:52 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:39:52.985 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:53.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:53.010 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:53.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:53.042 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:39:53.112 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:53.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:53.137 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:53.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:53.238 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:53.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:53.263 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:53.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:53.346 INFO:teuthology.orchestra.run.smithi028.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-02-05T06:39:53.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:53.388 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:53.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:53.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:53.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:53 smithi028 ceph-mon[37984]: pgmap v282: 158 pgs: 158 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-05T06:39:53.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:53 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1434569755' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T06:39:53.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:53 smithi028 ceph-mon[37984]: from='client.45023 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T06:39:53.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:53 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:39:53.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:53 smithi028 ceph-mon[29442]: pgmap v282: 158 pgs: 158 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-05T06:39:53.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:53 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1434569755' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T06:39:53.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:53 smithi028 ceph-mon[29442]: from='client.45023 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T06:39:53.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:53 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:39:53.490 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:53.514 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:53.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:53.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:53.573 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-05T06:39:53.616 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:53.640 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:53.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:53.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:53.687 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T06:39:53.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:53 smithi167 ceph-mon[32774]: pgmap v282: 158 pgs: 158 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-05T06:39:53.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:53 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1434569755' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T06:39:53.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:53 smithi167 ceph-mon[32774]: from='client.45023 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T06:39:53.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:53 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:39:53.742 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:53.766 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:53.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:53.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:53.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:53.894 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:53.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:53.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:53.996 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:54.024 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:54.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:54.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:54.126 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:54.151 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:54.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:54.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:54.253 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:54.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:54.282 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:54.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:54.383 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:54.410 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:54.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:54.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:54.512 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:54.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:54.540 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:54.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:54.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:54 smithi028 ceph-mon[29442]: from='client.45023 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T06:39:54.610 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:54 smithi028 ceph-mon[29442]: osdmap e191: 8 total, 8 up, 7 in 2024-02-05T06:39:54.610 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:54 smithi028 ceph-mon[37984]: from='client.45023 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T06:39:54.610 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:54 smithi028 ceph-mon[37984]: osdmap e191: 8 total, 8 up, 7 in 2024-02-05T06:39:54.641 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:54.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:54.669 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:54.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:54.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:54 smithi167 ceph-mon[32774]: from='client.45023 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T06:39:54.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:54 smithi167 ceph-mon[32774]: osdmap e191: 8 total, 8 up, 7 in 2024-02-05T06:39:54.771 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:54.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:54.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:54.887 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:39:54.898 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:54.921 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:54.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:54.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:55.023 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:55.047 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:55.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:55.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:55.149 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:55.159 DEBUG:teuthology.orchestra.run.smithi028:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-02-05T06:39:55.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:55.174 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:55.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:55.175 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-05T06:39:55.175 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-05T06:39:55.221 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:hints = 1 2024-02-05T06:39:55.221 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-05T06:39:55.221 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Object prefix: benchmark_data_smithi028_105698 2024-02-05T06:39:55.221 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T06:39:55.221 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 0 0 0 0 0 0 - 0 2024-02-05T06:39:55.275 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:55.313 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:55.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:55.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:55.415 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:55.430 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:39:55.445 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:55.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:55.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:55.547 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:55.595 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:55.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:55.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:55.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:55 smithi028 ceph-mon[29442]: pgmap v284: 158 pgs: 158 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:39:55.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:55 smithi028 ceph-mon[29442]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T06:39:55.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:55 smithi028 ceph-mon[29442]: osdmap e192: 8 total, 8 up, 7 in 2024-02-05T06:39:55.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:55 smithi028 ceph-mon[37984]: pgmap v284: 158 pgs: 158 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:39:55.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:55 smithi028 ceph-mon[37984]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T06:39:55.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:55 smithi028 ceph-mon[37984]: osdmap e192: 8 total, 8 up, 7 in 2024-02-05T06:39:55.696 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:55.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:55 smithi167 ceph-mon[32774]: pgmap v284: 158 pgs: 158 active+clean; 582 KiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:39:55.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:55 smithi167 ceph-mon[32774]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T06:39:55.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:55 smithi167 ceph-mon[32774]: osdmap e192: 8 total, 8 up, 7 in 2024-02-05T06:39:55.742 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:55.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:55.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:55.844 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:55.875 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:55.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:55.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:55.977 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:56.008 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:56.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:56.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:56.110 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:56.148 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:56.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:56.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:56.221 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1 15 3380 3365 210.302 210.312 0.00535155 0.00474243 2024-02-05T06:39:56.251 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:56.283 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:56.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:56.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:56.385 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:56.425 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:56.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:56.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:56.488 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 4] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T06:39:56.488 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T06:39:56.488 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 2, 3, 5, 6, 4] 2024-02-05T06:39:56.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd out 5 2024-02-05T06:39:56.527 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:56.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:56.560 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:56.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:56.661 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:56.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:56 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:39:56.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:56 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:39:56.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:56.713 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:56.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:56.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:56 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:39:56.814 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:56.846 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:56.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:56.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:56.948 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:56.980 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:56.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:56.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:57.082 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:57.123 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:57.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:57.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:57.221 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2 16 6500 6484 202.584 194.938 0.00393707 0.00492521 2024-02-05T06:39:57.225 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:57.257 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:57.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:57.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:57.359 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:57.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:57 smithi028 ceph-mon[29442]: pgmap v286: 158 pgs: 158 active+clean; 80 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 815 B/s rd, 13 MiB/s wr, 202 op/s 2024-02-05T06:39:57.389 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:57 smithi028 ceph-mon[37984]: pgmap v286: 158 pgs: 158 active+clean; 80 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 815 B/s rd, 13 MiB/s wr, 202 op/s 2024-02-05T06:39:57.403 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:57.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:57.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:57.505 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:39:57.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:57.547 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:57.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:39:57.648 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:57.681 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:57.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:57.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:57.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:57 smithi167 ceph-mon[32774]: pgmap v286: 158 pgs: 158 active+clean; 80 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 815 B/s rd, 13 MiB/s wr, 202 op/s 2024-02-05T06:39:57.784 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:57.826 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:57.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:57.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:57.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:57.980 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:57.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:57.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:58.082 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:58.113 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:58.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:58.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:58.215 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:39:58.221 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3 16 9530 9514 198.175 189.375 0.00625048 0.0050376 2024-02-05T06:39:58.244 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:58.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:58.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:39:58.346 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:58.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:58.396 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:58.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:58.497 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:39:58.537 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:58.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:58.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:39:58.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:39:58.677 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:58.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:58.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:39:58.780 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:58.811 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:58.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:58.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:58.914 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:58.944 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:58.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:58.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:58.990 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:39:59.039 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:39:59.046 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:39:59.077 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:59.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:59.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:39:59.081 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:39:59.126 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:39:59.166 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:39:59.178 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:59.212 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:39:59.223 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4 16 12149 12133 189.461 163.688 0.00375229 0.00526993 2024-02-05T06:39:59.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:59.234 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:59.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:39:59.253 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:39:59.294 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:39:59.298 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:39:59.298 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:39:59.298 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:39:59.298 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:39:59.298 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:39:59.299 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:39:59.299 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:39:59.299 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:39:59.335 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:59.365 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:59.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:59.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:59.467 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:39:59.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:59.501 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:59.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:39:59.598 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:59 smithi028 ceph-mon[37984]: pgmap v287: 158 pgs: 158 active+clean; 251 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 31 MiB/s wr, 502 op/s 2024-02-05T06:39:59.598 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:39:59 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/670100496' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:39:59.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:59 smithi028 ceph-mon[29442]: pgmap v287: 158 pgs: 158 active+clean; 251 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 31 MiB/s wr, 502 op/s 2024-02-05T06:39:59.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:39:59 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/670100496' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:39:59.602 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:59.608 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:39:59.629 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:59.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:59.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:59.731 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:39:59.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:59 smithi167 ceph-mon[32774]: pgmap v287: 158 pgs: 158 active+clean; 251 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 31 MiB/s wr, 502 op/s 2024-02-05T06:39:59.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:39:59 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/670100496' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:39:59.760 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:39:59.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:59.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:39:59.862 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:39:59.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:39:59.903 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:39:59.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:00.004 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:00.036 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:00.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:00.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:00.138 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:00.166 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:00.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:00.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:00.224 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5 16 14648 14632 182.807 156.188 0.0041995 0.0054623 2024-02-05T06:40:00.267 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:00.307 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:00.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:00.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:00.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:00 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2073245799' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:40:00.395 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:00 smithi028 ceph-mon[29442]: from='client.25057 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:40:00.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:00 smithi028 ceph-mon[29442]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-02-05T06:40:00.396 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:00 smithi028 ceph-mon[29442]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-02-05T06:40:00.396 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:00 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2073245799' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:40:00.396 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:00 smithi028 ceph-mon[37984]: from='client.25057 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:40:00.396 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:00 smithi028 ceph-mon[37984]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-02-05T06:40:00.396 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:00 smithi028 ceph-mon[37984]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-02-05T06:40:00.397 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-02-05T06:40:00.410 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:00.438 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:00.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:00.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:00.540 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:00.572 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:00.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:00.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:00.674 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:00.717 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:00.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:00.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:00.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:00 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2073245799' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:40:00.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:00 smithi167 ceph-mon[32774]: from='client.25057 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:40:00.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:00 smithi167 ceph-mon[32774]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-02-05T06:40:00.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:00 smithi167 ceph-mon[32774]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-02-05T06:40:00.819 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:00.849 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:00.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:00.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:00.869 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:40:00.869 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-02-05T06:40:00.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-02-05T06:40:00.870 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-02-05T06:40:00.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.870 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:192337 7.8 7:118a156b:::benchmark_data_smithi028_77509_object77217:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-02-05T06:40:00.870 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:38.337115+0000", 2024-02-05T06:40:00.870 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.422841676999994, 2024-02-05T06:40:00.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0591137100000001, 2024-02-05T06:40:00.870 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:40:00.870 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-02-05T06:40:00.870 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:40:00.870 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:40:00.870 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:40:00.870 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 192337 2024-02-05T06:40:00.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.871 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:40:00.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:40:00.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.337115+0000", 2024-02-05T06:40:00.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:40:00.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.337115+0000", 2024-02-05T06:40:00.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:40:00.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.337117+0000", 2024-02-05T06:40:00.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.73e-06 2024-02-05T06:40:00.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.872 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:40:00.872 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.337122+0000", 2024-02-05T06:40:00.872 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5310000000000001e-06 2024-02-05T06:40:00.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.872 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:40:00.872 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.337133+0000", 2024-02-05T06:40:00.872 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0570999999999999e-05 2024-02-05T06:40:00.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.872 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:40:00.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.337141+0000", 2024-02-05T06:40:00.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3979999999999999e-06 2024-02-05T06:40:00.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.873 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.337204+0000", 2024-02-05T06:40:00.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2786999999999994e-05 2024-02-05T06:40:00.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.873 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:40:00.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.337367+0000", 2024-02-05T06:40:00.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016232799999999999 2024-02-05T06:40:00.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4,7", 2024-02-05T06:40:00.874 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.337495+0000", 2024-02-05T06:40:00.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012881599999999999 2024-02-05T06:40:00.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.874 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.346118+0000", 2024-02-05T06:40:00.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0086228139999999995 2024-02-05T06:40:00.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:40:00.874 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.346131+0000", 2024-02-05T06:40:00.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2860999999999999e-05 2024-02-05T06:40:00.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.346165+0000", 2024-02-05T06:40:00.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4406999999999998e-05 2024-02-05T06:40:00.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.360213+0000", 2024-02-05T06:40:00.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014047484000000001 2024-02-05T06:40:00.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:40:00.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.360224+0000", 2024-02-05T06:40:00.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1392999999999999e-05 2024-02-05T06:40:00.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.876 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.876 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.396146+0000", 2024-02-05T06:40:00.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.03592211 2024-02-05T06:40:00.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.876 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:40:00.876 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.396229+0000", 2024-02-05T06:40:00.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2479999999999996e-05 2024-02-05T06:40:00.876 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.877 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:40:00.877 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.877 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:192338 7.8 7:16a7efbe:::benchmark_data_smithi028_77509_object77218:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-02-05T06:40:00.877 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:38.337628+0000", 2024-02-05T06:40:00.877 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.422328906000004, 2024-02-05T06:40:00.877 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0591757799999999, 2024-02-05T06:40:00.877 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:40:00.877 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-02-05T06:40:00.877 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:40:00.877 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:40:00.877 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:40:00.877 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 192338 2024-02-05T06:40:00.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.878 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:40:00.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:40:00.878 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.337628+0000", 2024-02-05T06:40:00.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:40:00.878 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.337628+0000", 2024-02-05T06:40:00.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:40:00.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.337629+0000", 2024-02-05T06:40:00.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1880000000000001e-06 2024-02-05T06:40:00.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:40:00.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.337634+0000", 2024-02-05T06:40:00.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5689999999999998e-06 2024-02-05T06:40:00.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:40:00.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.337643+0000", 2024-02-05T06:40:00.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5529999999999998e-06 2024-02-05T06:40:00.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.880 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:40:00.880 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.337651+0000", 2024-02-05T06:40:00.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8280000000000003e-06 2024-02-05T06:40:00.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.880 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.880 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.341287+0000", 2024-02-05T06:40:00.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0036361760000000001 2024-02-05T06:40:00.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.880 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:40:00.880 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.341412+0000", 2024-02-05T06:40:00.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000125112 2024-02-05T06:40:00.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.881 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4,7", 2024-02-05T06:40:00.881 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.341535+0000", 2024-02-05T06:40:00.881 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000122915 2024-02-05T06:40:00.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.881 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.881 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.346203+0000", 2024-02-05T06:40:00.881 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0046678359999999999 2024-02-05T06:40:00.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.881 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.881 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.346242+0000", 2024-02-05T06:40:00.881 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9008999999999999e-05 2024-02-05T06:40:00.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.882 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.882 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:40:00.882 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.351060+0000", 2024-02-05T06:40:00.882 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004818363 2024-02-05T06:40:00.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.882 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.882 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.882 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.360255+0000", 2024-02-05T06:40:00.882 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0091945059999999999 2024-02-05T06:40:00.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.882 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.882 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:40:00.882 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.360270+0000", 2024-02-05T06:40:00.883 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5324000000000001e-05 2024-02-05T06:40:00.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.883 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.883 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.883 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.396734+0000", 2024-02-05T06:40:00.883 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0364635360000001 2024-02-05T06:40:00.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.883 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.883 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:40:00.883 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.396804+0000", 2024-02-05T06:40:00.883 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9864999999999996e-05 2024-02-05T06:40:00.883 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.883 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:40:00.884 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.884 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.884 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:192347 7.8 7:11cd75fe:::benchmark_data_smithi028_77509_object77227:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-02-05T06:40:00.884 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:38.339579+0000", 2024-02-05T06:40:00.884 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.420378158999995, 2024-02-05T06:40:00.884 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.057352659, 2024-02-05T06:40:00.884 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:40:00.884 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-02-05T06:40:00.884 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:40:00.884 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:40:00.884 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:40:00.884 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 192347 2024-02-05T06:40:00.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.885 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:40:00.885 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.885 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:40:00.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.339579+0000", 2024-02-05T06:40:00.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.885 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.885 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:40:00.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.339579+0000", 2024-02-05T06:40:00.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.885 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.885 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:40:00.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.339580+0000", 2024-02-05T06:40:00.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.497e-06 2024-02-05T06:40:00.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.886 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.886 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:40:00.886 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.339585+0000", 2024-02-05T06:40:00.886 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0839999999999999e-06 2024-02-05T06:40:00.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.886 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.886 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:40:00.886 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.339596+0000", 2024-02-05T06:40:00.886 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0536e-05 2024-02-05T06:40:00.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.886 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.886 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:40:00.886 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.339604+0000", 2024-02-05T06:40:00.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2220000000000002e-06 2024-02-05T06:40:00.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.887 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.887 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.346356+0000", 2024-02-05T06:40:00.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0067518120000000003 2024-02-05T06:40:00.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.887 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:40:00.887 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.346473+0000", 2024-02-05T06:40:00.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011735600000000001 2024-02-05T06:40:00.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.887 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4,7", 2024-02-05T06:40:00.887 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.346587+0000", 2024-02-05T06:40:00.888 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000114293 2024-02-05T06:40:00.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.888 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.888 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.888 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.351674+0000", 2024-02-05T06:40:00.888 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050869640000000002 2024-02-05T06:40:00.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.888 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.888 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.888 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.351712+0000", 2024-02-05T06:40:00.888 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.786e-05 2024-02-05T06:40:00.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.889 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.889 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:40:00.889 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.354090+0000", 2024-02-05T06:40:00.889 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002377601 2024-02-05T06:40:00.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.889 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.889 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.889 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.360302+0000", 2024-02-05T06:40:00.889 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062116050000000003 2024-02-05T06:40:00.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.889 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.889 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:40:00.889 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.360310+0000", 2024-02-05T06:40:00.890 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9209999999999998e-06 2024-02-05T06:40:00.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.890 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.890 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.396838+0000", 2024-02-05T06:40:00.890 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0365271110000001 2024-02-05T06:40:00.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.890 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:40:00.890 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.396931+0000", 2024-02-05T06:40:00.890 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3796999999999997e-05 2024-02-05T06:40:00.890 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.890 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:40:00.890 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.891 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.891 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:192353 7.8 7:150e0f66:::benchmark_data_smithi028_77509_object77233:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-02-05T06:40:00.891 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:38.356556+0000", 2024-02-05T06:40:00.891 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.403401169999995, 2024-02-05T06:40:00.891 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0493035660000001, 2024-02-05T06:40:00.891 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:40:00.891 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:40:00.891 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:40:00.891 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:40:00.891 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:40:00.891 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 192353 2024-02-05T06:40:00.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.892 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:40:00.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.892 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:40:00.892 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.356556+0000", 2024-02-05T06:40:00.892 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.892 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:40:00.892 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.356556+0000", 2024-02-05T06:40:00.892 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.892 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:40:00.892 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.356557+0000", 2024-02-05T06:40:00.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5820000000000001e-06 2024-02-05T06:40:00.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.893 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:40:00.893 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.356563+0000", 2024-02-05T06:40:00.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2580000000000003e-06 2024-02-05T06:40:00.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.893 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:40:00.893 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.356572+0000", 2024-02-05T06:40:00.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7389999999999994e-06 2024-02-05T06:40:00.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.893 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:40:00.893 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.356581+0000", 2024-02-05T06:40:00.894 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7080000000000003e-06 2024-02-05T06:40:00.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.894 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.894 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.360338+0000", 2024-02-05T06:40:00.894 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0037565789999999999 2024-02-05T06:40:00.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.894 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:40:00.894 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.360345+0000", 2024-02-05T06:40:00.894 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7190000000000001e-06 2024-02-05T06:40:00.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.895 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.895 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.396962+0000", 2024-02-05T06:40:00.895 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0366170079999999 2024-02-05T06:40:00.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.895 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.895 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:40:00.895 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.397086+0000", 2024-02-05T06:40:00.895 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000123294 2024-02-05T06:40:00.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.895 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.895 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4", 2024-02-05T06:40:00.895 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.397186+0000", 2024-02-05T06:40:00.895 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000100381 2024-02-05T06:40:00.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.896 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.896 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:40:00.896 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.405647+0000", 2024-02-05T06:40:00.896 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0084607150000000006 2024-02-05T06:40:00.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.896 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.896 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.896 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.405705+0000", 2024-02-05T06:40:00.896 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8365000000000001e-05 2024-02-05T06:40:00.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.896 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.896 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.896 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.405747+0000", 2024-02-05T06:40:00.896 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1501000000000001e-05 2024-02-05T06:40:00.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.897 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:40:00.897 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.405799+0000", 2024-02-05T06:40:00.897 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1999000000000001e-05 2024-02-05T06:40:00.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.897 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:40:00.897 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.405859+0000", 2024-02-05T06:40:00.897 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0717999999999997e-05 2024-02-05T06:40:00.897 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.897 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:40:00.897 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.898 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.898 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:192358 7.8 7:1a63e476:::benchmark_data_smithi028_77509_object77238:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-02-05T06:40:00.898 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:38.366256+0000", 2024-02-05T06:40:00.898 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.393700727999999, 2024-02-05T06:40:00.898 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0455992860000001, 2024-02-05T06:40:00.898 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:40:00.898 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:40:00.898 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:40:00.898 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:40:00.898 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:40:00.898 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 192358 2024-02-05T06:40:00.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.898 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:40:00.899 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.899 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:40:00.899 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.366256+0000", 2024-02-05T06:40:00.899 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.899 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.899 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:40:00.899 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.366256+0000", 2024-02-05T06:40:00.899 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.899 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.899 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:40:00.899 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.366264+0000", 2024-02-05T06:40:00.899 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2290000000000003e-06 2024-02-05T06:40:00.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.900 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:40:00.900 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.366270+0000", 2024-02-05T06:40:00.900 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6119999999999998e-06 2024-02-05T06:40:00.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.900 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:40:00.900 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.366279+0000", 2024-02-05T06:40:00.900 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3999999999999998e-06 2024-02-05T06:40:00.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.900 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:40:00.901 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.366286+0000", 2024-02-05T06:40:00.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5799999999999997e-06 2024-02-05T06:40:00.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.901 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.901 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.366360+0000", 2024-02-05T06:40:00.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3857999999999997e-05 2024-02-05T06:40:00.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.901 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:40:00.901 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.366370+0000", 2024-02-05T06:40:00.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8309999999999996e-06 2024-02-05T06:40:00.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.902 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.902 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.400159+0000", 2024-02-05T06:40:00.902 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.033789549 2024-02-05T06:40:00.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.902 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:40:00.902 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.400280+0000", 2024-02-05T06:40:00.902 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012075500000000001 2024-02-05T06:40:00.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.902 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4", 2024-02-05T06:40:00.902 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.400401+0000", 2024-02-05T06:40:00.902 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121271 2024-02-05T06:40:00.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:40:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.405872+0000", 2024-02-05T06:40:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0054704489999999996 2024-02-05T06:40:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.405927+0000", 2024-02-05T06:40:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5216000000000001e-05 2024-02-05T06:40:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.411703+0000", 2024-02-05T06:40:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0057759270000000001 2024-02-05T06:40:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.903 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.904 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:40:00.904 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.411804+0000", 2024-02-05T06:40:00.904 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010108299999999999 2024-02-05T06:40:00.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.904 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.904 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:40:00.904 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.411855+0000", 2024-02-05T06:40:00.904 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1526000000000001e-05 2024-02-05T06:40:00.904 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.904 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:40:00.904 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.905 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:192376 7.8 7:12aca2d8:::benchmark_data_smithi028_77509_object77256:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-02-05T06:40:00.905 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:38.377362+0000", 2024-02-05T06:40:00.905 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.382594870999995, 2024-02-05T06:40:00.905 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.041133079, 2024-02-05T06:40:00.905 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:40:00.905 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:40:00.905 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:40:00.905 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:40:00.905 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:40:00.905 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 192376 2024-02-05T06:40:00.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.905 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:40:00.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.905 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:40:00.906 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.377362+0000", 2024-02-05T06:40:00.906 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.906 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:40:00.906 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.377362+0000", 2024-02-05T06:40:00.906 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.906 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:40:00.906 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.377363+0000", 2024-02-05T06:40:00.906 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.13e-06 2024-02-05T06:40:00.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.907 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:40:00.907 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.377368+0000", 2024-02-05T06:40:00.907 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7870000000000001e-06 2024-02-05T06:40:00.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.907 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.907 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:40:00.907 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.377378+0000", 2024-02-05T06:40:00.907 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0379e-05 2024-02-05T06:40:00.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.907 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.907 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:40:00.907 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.377386+0000", 2024-02-05T06:40:00.907 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7540000000000006e-06 2024-02-05T06:40:00.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.908 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.908 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.377461+0000", 2024-02-05T06:40:00.908 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5334999999999996e-05 2024-02-05T06:40:00.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.908 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:40:00.908 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.377473+0000", 2024-02-05T06:40:00.908 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1280999999999999e-05 2024-02-05T06:40:00.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.908 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.908 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.402958+0000", 2024-02-05T06:40:00.908 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0254855270000001 2024-02-05T06:40:00.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.909 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.909 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:40:00.909 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.403055+0000", 2024-02-05T06:40:00.909 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7029999999999998e-05 2024-02-05T06:40:00.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.909 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.909 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4", 2024-02-05T06:40:00.909 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.403163+0000", 2024-02-05T06:40:00.909 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108173 2024-02-05T06:40:00.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.909 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.909 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.909 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.405984+0000", 2024-02-05T06:40:00.910 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028201889999999999 2024-02-05T06:40:00.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.910 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:40:00.910 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.411965+0000", 2024-02-05T06:40:00.910 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0059810710000000001 2024-02-05T06:40:00.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.910 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.910 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.418351+0000", 2024-02-05T06:40:00.910 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0063859299999999997 2024-02-05T06:40:00.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.910 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:40:00.910 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.418452+0000", 2024-02-05T06:40:00.911 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101118 2024-02-05T06:40:00.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.911 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.911 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:40:00.911 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.418495+0000", 2024-02-05T06:40:00.911 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3374999999999997e-05 2024-02-05T06:40:00.911 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.911 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:40:00.911 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.911 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.911 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:192402 7.8 7:167c0a8c:::benchmark_data_smithi028_77509_object77282:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-02-05T06:40:00.911 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:38.406909+0000", 2024-02-05T06:40:00.912 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.353047625000002, 2024-02-05T06:40:00.912 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.011859187, 2024-02-05T06:40:00.912 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:40:00.912 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:40:00.912 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:40:00.912 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:40:00.912 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:40:00.912 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 192402 2024-02-05T06:40:00.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.912 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:40:00.912 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.912 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:40:00.912 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.406909+0000", 2024-02-05T06:40:00.912 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.913 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.913 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:40:00.913 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.406909+0000", 2024-02-05T06:40:00.913 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.913 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.913 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:40:00.913 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.406912+0000", 2024-02-05T06:40:00.913 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6699999999999998e-06 2024-02-05T06:40:00.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.913 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.913 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:40:00.913 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.406918+0000", 2024-02-05T06:40:00.914 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2190000000000001e-06 2024-02-05T06:40:00.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.914 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.914 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:40:00.914 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.406934+0000", 2024-02-05T06:40:00.914 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6104999999999999e-05 2024-02-05T06:40:00.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.914 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.914 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:40:00.914 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.406943+0000", 2024-02-05T06:40:00.914 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7099999999999996e-06 2024-02-05T06:40:00.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.914 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.407092+0000", 2024-02-05T06:40:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000148796 2024-02-05T06:40:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:40:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.407102+0000", 2024-02-05T06:40:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9629999999999993e-06 2024-02-05T06:40:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.406020+0000", 2024-02-05T06:40:00.916 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.99891851600000003 2024-02-05T06:40:00.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.916 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.916 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:40:00.916 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.406242+0000", 2024-02-05T06:40:00.916 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000222184 2024-02-05T06:40:00.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.916 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.916 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4", 2024-02-05T06:40:00.916 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.406396+0000", 2024-02-05T06:40:00.916 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015325000000000001 2024-02-05T06:40:00.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.916 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.916 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.917 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.411950+0000", 2024-02-05T06:40:00.917 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055542550000000001 2024-02-05T06:40:00.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.917 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.917 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:40:00.917 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.418304+0000", 2024-02-05T06:40:00.917 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0063543740000000003 2024-02-05T06:40:00.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.917 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.917 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.917 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.418627+0000", 2024-02-05T06:40:00.917 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00032294300000000001 2024-02-05T06:40:00.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.918 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.918 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:40:00.918 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.418710+0000", 2024-02-05T06:40:00.918 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3205e-05 2024-02-05T06:40:00.918 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.918 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.918 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:40:00.918 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.418768+0000", 2024-02-05T06:40:00.918 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7997000000000001e-05 2024-02-05T06:40:00.918 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.918 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:40:00.918 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.918 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.918 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.919 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:192404 7.8 7:12218d7a:::benchmark_data_smithi028_77509_object77284:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-02-05T06:40:00.919 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:38.407199+0000", 2024-02-05T06:40:00.919 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.352758055999999, 2024-02-05T06:40:00.919 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0118317029999999, 2024-02-05T06:40:00.919 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:40:00.919 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:40:00.919 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:40:00.919 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:40:00.919 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:40:00.919 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 192404 2024-02-05T06:40:00.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.919 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:40:00.919 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.920 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:40:00.920 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.407199+0000", 2024-02-05T06:40:00.920 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.920 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.920 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:40:00.920 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.407199+0000", 2024-02-05T06:40:00.920 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.920 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.920 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:40:00.920 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.407200+0000", 2024-02-05T06:40:00.921 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.319e-06 2024-02-05T06:40:00.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.921 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.921 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:40:00.921 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.407205+0000", 2024-02-05T06:40:00.921 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.69e-06 2024-02-05T06:40:00.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.921 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.921 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:40:00.921 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.407218+0000", 2024-02-05T06:40:00.921 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3698e-05 2024-02-05T06:40:00.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.921 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.922 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:40:00.922 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.407226+0000", 2024-02-05T06:40:00.922 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8420000000000005e-06 2024-02-05T06:40:00.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.922 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.922 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.922 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.407287+0000", 2024-02-05T06:40:00.922 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0912000000000002e-05 2024-02-05T06:40:00.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.922 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.922 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:40:00.922 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.407296+0000", 2024-02-05T06:40:00.922 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1400000000000006e-06 2024-02-05T06:40:00.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.923 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.923 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.923 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.408982+0000", 2024-02-05T06:40:00.923 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.001685299 2024-02-05T06:40:00.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.923 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.923 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:40:00.923 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.409096+0000", 2024-02-05T06:40:00.923 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000114498 2024-02-05T06:40:00.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.923 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.923 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4", 2024-02-05T06:40:00.923 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.409174+0000", 2024-02-05T06:40:00.924 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7632999999999993e-05 2024-02-05T06:40:00.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.924 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.924 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.924 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.412018+0000", 2024-02-05T06:40:00.924 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028445969999999999 2024-02-05T06:40:00.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.924 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.924 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:40:00.924 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.418510+0000", 2024-02-05T06:40:00.924 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0064918170000000004 2024-02-05T06:40:00.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.924 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.924 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.925 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.418824+0000", 2024-02-05T06:40:00.925 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00031352600000000001 2024-02-05T06:40:00.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.925 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.925 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:40:00.925 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.418989+0000", 2024-02-05T06:40:00.925 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016478900000000001 2024-02-05T06:40:00.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.925 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.925 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:40:00.925 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.419030+0000", 2024-02-05T06:40:00.925 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1943000000000001e-05 2024-02-05T06:40:00.925 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.926 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:40:00.926 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.926 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.926 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:192455 7.8 7:145ae87a:::benchmark_data_smithi028_77509_object77335:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-02-05T06:40:00.926 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:38.457684+0000", 2024-02-05T06:40:00.926 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.302273025000005, 2024-02-05T06:40:00.926 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.96165713600000002, 2024-02-05T06:40:00.926 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:40:00.926 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:40:00.926 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:40:00.926 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:40:00.926 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:40:00.926 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 192455 2024-02-05T06:40:00.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.927 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:40:00.927 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.927 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:40:00.927 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.457684+0000", 2024-02-05T06:40:00.927 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.927 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.927 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:40:00.927 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.457684+0000", 2024-02-05T06:40:00.927 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.928 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.928 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:40:00.928 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.457685+0000", 2024-02-05T06:40:00.928 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6819999999999999e-06 2024-02-05T06:40:00.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.928 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.928 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:40:00.928 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.457690+0000", 2024-02-05T06:40:00.928 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.357e-06 2024-02-05T06:40:00.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.928 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.928 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:40:00.928 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.457705+0000", 2024-02-05T06:40:00.928 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5199e-05 2024-02-05T06:40:00.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.929 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.929 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:40:00.929 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.457712+0000", 2024-02-05T06:40:00.929 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6680000000000004e-06 2024-02-05T06:40:00.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.929 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.929 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.929 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.457777+0000", 2024-02-05T06:40:00.929 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5244000000000001e-05 2024-02-05T06:40:00.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.929 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.930 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:40:00.930 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.457788+0000", 2024-02-05T06:40:00.930 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1168e-05 2024-02-05T06:40:00.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.930 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.930 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.930 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.412068+0000", 2024-02-05T06:40:00.930 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.95427959600000001 2024-02-05T06:40:00.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.931 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:40:00.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.412185+0000", 2024-02-05T06:40:00.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000117547 2024-02-05T06:40:00.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.931 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4", 2024-02-05T06:40:00.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.412852+0000", 2024-02-05T06:40:00.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00066681800000000001 2024-02-05T06:40:00.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.931 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:40:00.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.418516+0000", 2024-02-05T06:40:00.932 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0056640140000000002 2024-02-05T06:40:00.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.932 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.932 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.932 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.419112+0000", 2024-02-05T06:40:00.932 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00059562500000000002 2024-02-05T06:40:00.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.932 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.932 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.932 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.419152+0000", 2024-02-05T06:40:00.932 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0311000000000001e-05 2024-02-05T06:40:00.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.932 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.932 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:40:00.933 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.419208+0000", 2024-02-05T06:40:00.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6169999999999999e-05 2024-02-05T06:40:00.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.933 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:40:00.933 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.419341+0000", 2024-02-05T06:40:00.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013273700000000001 2024-02-05T06:40:00.933 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.933 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:40:00.933 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.934 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:192466 7.8 7:1913b8bb:::benchmark_data_smithi028_77509_object77346:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-02-05T06:40:00.934 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:38.468224+0000", 2024-02-05T06:40:00.934 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.291732854000003, 2024-02-05T06:40:00.934 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.95867438299999996, 2024-02-05T06:40:00.934 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:40:00.934 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:40:00.934 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:40:00.934 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:40:00.934 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:40:00.934 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 192466 2024-02-05T06:40:00.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.934 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:40:00.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.935 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:40:00.935 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.468224+0000", 2024-02-05T06:40:00.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.935 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:40:00.935 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.468224+0000", 2024-02-05T06:40:00.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.935 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:40:00.935 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.468225+0000", 2024-02-05T06:40:00.936 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3349999999999999e-06 2024-02-05T06:40:00.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.936 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:40:00.936 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.468230+0000", 2024-02-05T06:40:00.936 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6609999999999999e-06 2024-02-05T06:40:00.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.936 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:40:00.936 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.468240+0000", 2024-02-05T06:40:00.936 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5599999999999999e-06 2024-02-05T06:40:00.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.937 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:40:00.937 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.468247+0000", 2024-02-05T06:40:00.937 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5340000000000002e-06 2024-02-05T06:40:00.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.937 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.937 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.468315+0000", 2024-02-05T06:40:00.937 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8422000000000003e-05 2024-02-05T06:40:00.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.937 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:40:00.937 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.468325+0000", 2024-02-05T06:40:00.937 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5750000000000007e-06 2024-02-05T06:40:00.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.938 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.415713+0000", 2024-02-05T06:40:00.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.94738796999999997 2024-02-05T06:40:00.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.938 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:40:00.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.415856+0000", 2024-02-05T06:40:00.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014318499999999999 2024-02-05T06:40:00.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.938 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4", 2024-02-05T06:40:00.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.415981+0000", 2024-02-05T06:40:00.939 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012492699999999999 2024-02-05T06:40:00.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.939 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.939 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:40:00.939 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.418783+0000", 2024-02-05T06:40:00.939 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028021959999999999 2024-02-05T06:40:00.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.939 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.939 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.939 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.419382+0000", 2024-02-05T06:40:00.939 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00059883799999999997 2024-02-05T06:40:00.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.939 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.939 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.939 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.426789+0000", 2024-02-05T06:40:00.940 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0074069119999999999 2024-02-05T06:40:00.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.940 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.940 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:40:00.940 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.426847+0000", 2024-02-05T06:40:00.940 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7590000000000003e-05 2024-02-05T06:40:00.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.940 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.940 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:40:00.940 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.426898+0000", 2024-02-05T06:40:00.940 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1678000000000002e-05 2024-02-05T06:40:00.940 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.940 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:40:00.940 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.941 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.941 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:192477 7.8 7:1213b1e3:::benchmark_data_smithi028_77509_object77357:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-02-05T06:40:00.941 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:38.482175+0000", 2024-02-05T06:40:00.941 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.2777818, 2024-02-05T06:40:00.941 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.945012713, 2024-02-05T06:40:00.941 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:40:00.941 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:40:00.941 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:40:00.941 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:40:00.941 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:40:00.941 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 192477 2024-02-05T06:40:00.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.942 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:40:00.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.942 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:40:00.942 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.482175+0000", 2024-02-05T06:40:00.942 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.942 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:40:00.942 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.482175+0000", 2024-02-05T06:40:00.942 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.942 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:40:00.943 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.482177+0000", 2024-02-05T06:40:00.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.272e-06 2024-02-05T06:40:00.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.943 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:40:00.943 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.482183+0000", 2024-02-05T06:40:00.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6200000000000004e-06 2024-02-05T06:40:00.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.943 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:40:00.943 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.482194+0000", 2024-02-05T06:40:00.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1524e-05 2024-02-05T06:40:00.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.943 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:40:00.944 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.482202+0000", 2024-02-05T06:40:00.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7540000000000006e-06 2024-02-05T06:40:00.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.944 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.944 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.482278+0000", 2024-02-05T06:40:00.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6179000000000004e-05 2024-02-05T06:40:00.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.944 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:40:00.944 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.482290+0000", 2024-02-05T06:40:00.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1538e-05 2024-02-05T06:40:00.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.945 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.945 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.419422+0000", 2024-02-05T06:40:00.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.93713164000000004 2024-02-05T06:40:00.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.945 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:40:00.945 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.419530+0000", 2024-02-05T06:40:00.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108298 2024-02-05T06:40:00.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.945 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4", 2024-02-05T06:40:00.945 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.419631+0000", 2024-02-05T06:40:00.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000100874 2024-02-05T06:40:00.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.427061+0000", 2024-02-05T06:40:00.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0074300970000000001 2024-02-05T06:40:00.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.427098+0000", 2024-02-05T06:40:00.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7354999999999997e-05 2024-02-05T06:40:00.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:40:00.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.427109+0000", 2024-02-05T06:40:00.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0348e-05 2024-02-05T06:40:00.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:40:00.947 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.427152+0000", 2024-02-05T06:40:00.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3509e-05 2024-02-05T06:40:00.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:40:00.947 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.427188+0000", 2024-02-05T06:40:00.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5704999999999998e-05 2024-02-05T06:40:00.947 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.947 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:40:00.947 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.948 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:192486 7.8 7:1082e2c5:::benchmark_data_smithi028_77509_object77366:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-02-05T06:40:00.948 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:38.490414+0000", 2024-02-05T06:40:00.948 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.269543272999996, 2024-02-05T06:40:00.948 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.94319277599999995, 2024-02-05T06:40:00.948 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:40:00.948 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:40:00.948 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:40:00.948 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:40:00.948 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:40:00.948 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 192486 2024-02-05T06:40:00.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.948 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:40:00.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.948 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:40:00.948 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.490414+0000", 2024-02-05T06:40:00.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:40:00.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.490414+0000", 2024-02-05T06:40:00.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:40:00.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.490416+0000", 2024-02-05T06:40:00.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.711e-06 2024-02-05T06:40:00.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:40:00.950 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.490422+0000", 2024-02-05T06:40:00.950 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7450000000000001e-06 2024-02-05T06:40:00.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.950 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:40:00.950 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.490439+0000", 2024-02-05T06:40:00.950 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7091999999999998e-05 2024-02-05T06:40:00.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.950 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:40:00.950 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.490448+0000", 2024-02-05T06:40:00.950 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9050000000000003e-06 2024-02-05T06:40:00.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.490523+0000", 2024-02-05T06:40:00.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4840999999999997e-05 2024-02-05T06:40:00.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:40:00.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.490533+0000", 2024-02-05T06:40:00.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.031e-05 2024-02-05T06:40:00.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.423956+0000", 2024-02-05T06:40:00.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.93342235699999998 2024-02-05T06:40:00.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.952 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:40:00.952 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.424058+0000", 2024-02-05T06:40:00.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000102707 2024-02-05T06:40:00.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.952 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4", 2024-02-05T06:40:00.952 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.424167+0000", 2024-02-05T06:40:00.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108312 2024-02-05T06:40:00.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.952 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:40:00.952 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.433395+0000", 2024-02-05T06:40:00.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0092287649999999999 2024-02-05T06:40:00.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.953 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.433438+0000", 2024-02-05T06:40:00.953 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2302999999999998e-05 2024-02-05T06:40:00.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.953 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.433475+0000", 2024-02-05T06:40:00.953 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7749000000000001e-05 2024-02-05T06:40:00.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.954 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:40:00.954 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.433567+0000", 2024-02-05T06:40:00.954 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1310000000000005e-05 2024-02-05T06:40:00.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.954 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:40:00.954 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.433606+0000", 2024-02-05T06:40:00.954 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9669000000000003e-05 2024-02-05T06:40:00.954 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.954 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:40:00.954 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.954 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:192491 7.8 7:1ef0b3cf:::benchmark_data_smithi028_77509_object77371:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-02-05T06:40:00.955 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:38.496778+0000", 2024-02-05T06:40:00.955 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.263178843999995, 2024-02-05T06:40:00.955 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.94398407100000004, 2024-02-05T06:40:00.955 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:40:00.955 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:40:00.955 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:40:00.955 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:40:00.955 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:40:00.955 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 192491 2024-02-05T06:40:00.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.955 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:40:00.955 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.955 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:40:00.955 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.496778+0000", 2024-02-05T06:40:00.956 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.956 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:40:00.956 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.496778+0000", 2024-02-05T06:40:00.956 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.956 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:40:00.956 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.496781+0000", 2024-02-05T06:40:00.956 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7070000000000001e-06 2024-02-05T06:40:00.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.956 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:40:00.957 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.496787+0000", 2024-02-05T06:40:00.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0950000000000001e-06 2024-02-05T06:40:00.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.957 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:40:00.957 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.496813+0000", 2024-02-05T06:40:00.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5857e-05 2024-02-05T06:40:00.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.957 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:40:00.957 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.496822+0000", 2024-02-05T06:40:00.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6660000000000003e-06 2024-02-05T06:40:00.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.958 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.958 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.496921+0000", 2024-02-05T06:40:00.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8653000000000006e-05 2024-02-05T06:40:00.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.958 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:40:00.958 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.496932+0000", 2024-02-05T06:40:00.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0855e-05 2024-02-05T06:40:00.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.958 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.958 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.427215+0000", 2024-02-05T06:40:00.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.93028335699999998 2024-02-05T06:40:00.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.959 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:40:00.959 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.427323+0000", 2024-02-05T06:40:00.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108039 2024-02-05T06:40:00.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.959 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4", 2024-02-05T06:40:00.959 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.427428+0000", 2024-02-05T06:40:00.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000104916 2024-02-05T06:40:00.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.959 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.959 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.433644+0000", 2024-02-05T06:40:00.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062161480000000003 2024-02-05T06:40:00.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.960 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.960 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.433682+0000", 2024-02-05T06:40:00.960 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7267000000000001e-05 2024-02-05T06:40:00.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.960 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:40:00.960 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.440674+0000", 2024-02-05T06:40:00.960 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0069923160000000002 2024-02-05T06:40:00.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.960 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:40:00.960 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.440725+0000", 2024-02-05T06:40:00.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1471000000000002e-05 2024-02-05T06:40:00.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.961 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:40:00.961 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.440762+0000", 2024-02-05T06:40:00.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6724000000000003e-05 2024-02-05T06:40:00.961 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.961 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:40:00.961 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.961 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:192492 7.8 7:1c84ffe8:::benchmark_data_smithi028_77509_object77372:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-02-05T06:40:00.961 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:38.496842+0000", 2024-02-05T06:40:00.961 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.263114736000006, 2024-02-05T06:40:00.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.94407687900000004, 2024-02-05T06:40:00.962 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:40:00.962 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:40:00.962 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:40:00.962 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:40:00.962 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:40:00.962 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 192492 2024-02-05T06:40:00.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.962 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:40:00.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:40:00.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.496842+0000", 2024-02-05T06:40:00.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:40:00.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.496842+0000", 2024-02-05T06:40:00.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:40:00.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.496843+0000", 2024-02-05T06:40:00.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7000000000000003e-07 2024-02-05T06:40:00.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:40:00.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.496847+0000", 2024-02-05T06:40:00.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8920000000000002e-06 2024-02-05T06:40:00.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:40:00.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.496854+0000", 2024-02-05T06:40:00.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4250000000000001e-06 2024-02-05T06:40:00.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:40:00.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.496861+0000", 2024-02-05T06:40:00.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5710000000000003e-06 2024-02-05T06:40:00.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.496959+0000", 2024-02-05T06:40:00.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8307000000000006e-05 2024-02-05T06:40:00.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.965 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:40:00.965 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.496968+0000", 2024-02-05T06:40:00.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3939999999999996e-06 2024-02-05T06:40:00.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.965 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.965 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.433709+0000", 2024-02-05T06:40:00.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.93674168700000005 2024-02-05T06:40:00.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.966 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:40:00.966 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.433805+0000", 2024-02-05T06:40:00.966 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5612999999999994e-05 2024-02-05T06:40:00.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.966 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4", 2024-02-05T06:40:00.966 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.433911+0000", 2024-02-05T06:40:00.966 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000106332 2024-02-05T06:40:00.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.966 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.966 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.440801+0000", 2024-02-05T06:40:00.966 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.006890172 2024-02-05T06:40:00.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.967 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.967 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:40:00.967 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.440813+0000", 2024-02-05T06:40:00.967 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1260000000000001e-05 2024-02-05T06:40:00.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.967 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.967 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.967 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.440846+0000", 2024-02-05T06:40:00.967 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3423000000000002e-05 2024-02-05T06:40:00.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.967 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.967 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:40:00.968 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.440887+0000", 2024-02-05T06:40:00.968 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1118999999999997e-05 2024-02-05T06:40:00.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.968 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:40:00.968 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.440919+0000", 2024-02-05T06:40:00.968 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1813999999999997e-05 2024-02-05T06:40:00.968 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.968 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:40:00.968 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.968 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:192500 7.8 7:1c437fe4:::benchmark_data_smithi028_77509_object77380:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-02-05T06:40:00.968 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:38.511280+0000", 2024-02-05T06:40:00.969 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 82.248676899000003, 2024-02-05T06:40:00.969 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.93731256799999996, 2024-02-05T06:40:00.969 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:40:00.969 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:40:00.969 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:40:00.969 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:40:00.969 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:40:00.969 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 192500 2024-02-05T06:40:00.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.969 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:40:00.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.969 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:40:00.969 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.511280+0000", 2024-02-05T06:40:00.969 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.970 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:40:00.970 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.511280+0000", 2024-02-05T06:40:00.970 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.970 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:40:00.970 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.511282+0000", 2024-02-05T06:40:00.970 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9769999999999999e-06 2024-02-05T06:40:00.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.970 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:40:00.970 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.511287+0000", 2024-02-05T06:40:00.970 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2109999999999997e-06 2024-02-05T06:40:00.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.971 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:40:00.971 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.511299+0000", 2024-02-05T06:40:00.971 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1984000000000001e-05 2024-02-05T06:40:00.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.971 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:40:00.971 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.511307+0000", 2024-02-05T06:40:00.971 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3639999999999999e-06 2024-02-05T06:40:00.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.971 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.971 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.511376+0000", 2024-02-05T06:40:00.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8404000000000004e-05 2024-02-05T06:40:00.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.972 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:40:00.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.511387+0000", 2024-02-05T06:40:00.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1518000000000001e-05 2024-02-05T06:40:00.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.972 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.437898+0000", 2024-02-05T06:40:00.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.92651070000000002 2024-02-05T06:40:00.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.973 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:40:00.973 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.437993+0000", 2024-02-05T06:40:00.973 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5056000000000006e-05 2024-02-05T06:40:00.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.973 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4", 2024-02-05T06:40:00.973 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.438097+0000", 2024-02-05T06:40:00.973 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103434 2024-02-05T06:40:00.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.973 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.973 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.446548+0000", 2024-02-05T06:40:00.973 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0084514180000000005 2024-02-05T06:40:00.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.974 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:40:00.974 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.446574+0000", 2024-02-05T06:40:00.974 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.599e-05 2024-02-05T06:40:00.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.974 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.974 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.448481+0000", 2024-02-05T06:40:00.974 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001907377 2024-02-05T06:40:00.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.974 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:40:00.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.448538+0000", 2024-02-05T06:40:00.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6530000000000003e-05 2024-02-05T06:40:00.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:40:00.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.448593+0000", 2024-02-05T06:40:00.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4605000000000003e-05 2024-02-05T06:40:00.975 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.975 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:40:00.975 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.975 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:192607 7.5 7:ac582dbe:::benchmark_data_smithi028_77509_object77487:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-02-05T06:40:00.975 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:40.027506+0000", 2024-02-05T06:40:00.976 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 80.732450674999995, 2024-02-05T06:40:00.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.97921314100000001, 2024-02-05T06:40:00.976 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:40:00.976 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:40:00.976 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:40:00.976 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:40:00.976 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:40:00.976 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 192607 2024-02-05T06:40:00.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.976 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:40:00.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:40:00.976 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.027506+0000", 2024-02-05T06:40:00.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.977 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:40:00.977 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.027506+0000", 2024-02-05T06:40:00.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.977 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:40:00.977 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.027508+0000", 2024-02-05T06:40:00.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.53e-06 2024-02-05T06:40:00.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.977 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:40:00.977 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.027513+0000", 2024-02-05T06:40:00.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.865e-06 2024-02-05T06:40:00.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:40:00.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.027524+0000", 2024-02-05T06:40:00.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1026000000000001e-05 2024-02-05T06:40:00.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:40:00.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.027532+0000", 2024-02-05T06:40:00.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9689999999999994e-06 2024-02-05T06:40:00.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.027591+0000", 2024-02-05T06:40:00.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9491999999999999e-05 2024-02-05T06:40:00.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.979 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:40:00.979 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.027601+0000", 2024-02-05T06:40:00.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.91e-06 2024-02-05T06:40:00.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.979 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.979 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.998983+0000", 2024-02-05T06:40:00.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.97138245300000003 2024-02-05T06:40:00.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:40:00.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.999074+0000", 2024-02-05T06:40:00.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0660000000000003e-05 2024-02-05T06:40:00.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,1", 2024-02-05T06:40:00.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.999397+0000", 2024-02-05T06:40:00.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00032286800000000003 2024-02-05T06:40:00.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:40:00.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.006547+0000", 2024-02-05T06:40:00.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0071499659999999998 2024-02-05T06:40:00.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.981 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.981 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.006605+0000", 2024-02-05T06:40:00.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7776999999999999e-05 2024-02-05T06:40:00.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.981 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.981 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.006642+0000", 2024-02-05T06:40:00.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7698999999999999e-05 2024-02-05T06:40:00.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.981 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:40:00.981 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.006676+0000", 2024-02-05T06:40:00.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3389000000000002e-05 2024-02-05T06:40:00.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.982 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:40:00.982 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.006719+0000", 2024-02-05T06:40:00.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3537e-05 2024-02-05T06:40:00.982 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.982 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:40:00.982 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.982 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:192647 7.5 7:aae146c8:::benchmark_data_smithi028_77509_object77527:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-02-05T06:40:00.982 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:40.059890+0000", 2024-02-05T06:40:00.982 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 80.700067060999999, 2024-02-05T06:40:00.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.94709845400000003, 2024-02-05T06:40:00.982 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:40:00.982 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:40:00.982 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:40:00.983 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:40:00.983 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:40:00.983 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 192647 2024-02-05T06:40:00.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.983 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:40:00.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:40:00.983 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.059890+0000", 2024-02-05T06:40:00.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:40:00.983 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.059890+0000", 2024-02-05T06:40:00.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.984 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:40:00.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.059891+0000", 2024-02-05T06:40:00.984 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4759999999999999e-06 2024-02-05T06:40:00.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.984 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:40:00.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.059896+0000", 2024-02-05T06:40:00.984 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3850000000000004e-06 2024-02-05T06:40:00.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.984 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:40:00.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.059906+0000", 2024-02-05T06:40:00.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0496000000000001e-05 2024-02-05T06:40:00.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.985 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:40:00.985 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.059914+0000", 2024-02-05T06:40:00.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8599999999999993e-06 2024-02-05T06:40:00.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.985 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.985 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.059975+0000", 2024-02-05T06:40:00.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1257e-05 2024-02-05T06:40:00.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.985 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:40:00.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.059984+0000", 2024-02-05T06:40:00.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4230000000000005e-06 2024-02-05T06:40:00.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.002477+0000", 2024-02-05T06:40:00.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.94249373299999994 2024-02-05T06:40:00.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:40:00.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.002607+0000", 2024-02-05T06:40:00.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000129825 2024-02-05T06:40:00.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,1", 2024-02-05T06:40:00.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.002728+0000", 2024-02-05T06:40:00.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121049 2024-02-05T06:40:00.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:40:00.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.006799+0000", 2024-02-05T06:40:00.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004070901 2024-02-05T06:40:00.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.006852+0000", 2024-02-05T06:40:00.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2491000000000003e-05 2024-02-05T06:40:00.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.988 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.988 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.006921+0000", 2024-02-05T06:40:00.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9225000000000002e-05 2024-02-05T06:40:00.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.988 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:40:00.988 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.006967+0000", 2024-02-05T06:40:00.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.596e-05 2024-02-05T06:40:00.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.988 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:40:00.988 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.006988+0000", 2024-02-05T06:40:00.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1373e-05 2024-02-05T06:40:00.988 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.989 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:40:00.989 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.989 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:192659 7.5 7:a638279d:::benchmark_data_smithi028_77509_object77539:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-02-05T06:40:00.989 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:40.072812+0000", 2024-02-05T06:40:00.989 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 80.687144634999996, 2024-02-05T06:40:00.989 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.93731890299999998, 2024-02-05T06:40:00.989 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:40:00.989 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:40:00.989 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:40:00.989 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:40:00.989 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:40:00.989 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 192659 2024-02-05T06:40:00.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.990 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:40:00.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.990 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:40:00.990 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.072812+0000", 2024-02-05T06:40:00.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.990 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:40:00.990 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.072812+0000", 2024-02-05T06:40:00.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.990 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:40:00.991 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.072815+0000", 2024-02-05T06:40:00.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4940000000000002e-06 2024-02-05T06:40:00.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:40:00.991 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.072822+0000", 2024-02-05T06:40:00.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9029999999999998e-06 2024-02-05T06:40:00.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:40:00.991 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.072840+0000", 2024-02-05T06:40:00.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8465999999999999e-05 2024-02-05T06:40:00.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.992 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:40:00.992 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.072850+0000", 2024-02-05T06:40:00.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6560000000000004e-06 2024-02-05T06:40:00.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.992 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.992 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.072915+0000", 2024-02-05T06:40:00.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4986000000000004e-05 2024-02-05T06:40:00.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.992 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:40:00.992 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.072924+0000", 2024-02-05T06:40:00.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6099999999999995e-06 2024-02-05T06:40:00.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.004804+0000", 2024-02-05T06:40:00.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.93187985100000004 2024-02-05T06:40:00.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:40:00.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.004877+0000", 2024-02-05T06:40:00.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2350999999999999e-05 2024-02-05T06:40:00.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,1", 2024-02-05T06:40:00.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.004963+0000", 2024-02-05T06:40:00.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6347999999999994e-05 2024-02-05T06:40:00.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:40:00.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.006995+0000", 2024-02-05T06:40:00.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002032458 2024-02-05T06:40:00.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.009960+0000", 2024-02-05T06:40:00.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0029642700000000002 2024-02-05T06:40:00.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:00.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.010024+0000", 2024-02-05T06:40:00.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4772999999999998e-05 2024-02-05T06:40:00.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:40:00.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.010085+0000", 2024-02-05T06:40:00.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1055000000000004e-05 2024-02-05T06:40:00.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:40:00.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.010131+0000", 2024-02-05T06:40:00.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5682000000000001e-05 2024-02-05T06:40:00.995 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.995 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:40:00.995 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:00.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.995 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:192671 7.5 7:aaf7afeb:::benchmark_data_smithi028_77509_object77551:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-02-05T06:40:00.996 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:40.081633+0000", 2024-02-05T06:40:00.996 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 80.678323417000001, 2024-02-05T06:40:00.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.94033711799999997, 2024-02-05T06:40:00.996 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:40:00.996 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:40:00.996 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:40:00.996 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:40:00.996 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:40:00.996 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 192671 2024-02-05T06:40:00.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.996 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:40:00.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.996 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:40:00.997 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.081633+0000", 2024-02-05T06:40:00.997 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.997 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:40:00.997 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.081633+0000", 2024-02-05T06:40:00.997 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:00.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.997 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:40:00.997 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.081634+0000", 2024-02-05T06:40:00.997 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8599999999999996e-07 2024-02-05T06:40:00.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.997 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:40:00.997 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.081639+0000", 2024-02-05T06:40:00.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4859999999999999e-06 2024-02-05T06:40:00.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:40:00.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.081647+0000", 2024-02-05T06:40:00.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7749999999999993e-06 2024-02-05T06:40:00.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:40:00.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.081654+0000", 2024-02-05T06:40:00.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3320000000000003e-06 2024-02-05T06:40:00.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.081710+0000", 2024-02-05T06:40:00.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6174999999999997e-05 2024-02-05T06:40:00.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.999 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:40:00.999 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.081718+0000", 2024-02-05T06:40:00.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2910000000000007e-06 2024-02-05T06:40:00.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.999 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:00.999 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.007023+0000", 2024-02-05T06:40:00.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.92530491699999995 2024-02-05T06:40:00.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:00.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:00.999 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:40:01.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.007135+0000", 2024-02-05T06:40:01.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011166 2024-02-05T06:40:01.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:01.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:01.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,1", 2024-02-05T06:40:01.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.007226+0000", 2024-02-05T06:40:01.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0686000000000006e-05 2024-02-05T06:40:01.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:01.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:01.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:01.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.015290+0000", 2024-02-05T06:40:01.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0080642100000000005 2024-02-05T06:40:01.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:01.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:01.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:40:01.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.018460+0000", 2024-02-05T06:40:01.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031696139999999999 2024-02-05T06:40:01.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:01.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:01.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:01.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.021298+0000", 2024-02-05T06:40:01.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002838648 2024-02-05T06:40:01.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:01.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:01.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:40:01.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.021924+0000", 2024-02-05T06:40:01.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00062615800000000003 2024-02-05T06:40:01.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:01.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:01.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:40:01.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.021971+0000", 2024-02-05T06:40:01.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6180000000000002e-05 2024-02-05T06:40:01.002 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:01.002 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:40:01.002 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:01.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:01.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:01.002 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:192724 7.5 7:a4d7f24c:::benchmark_data_smithi028_77509_object77604:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-02-05T06:40:01.002 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:40.125512+0000", 2024-02-05T06:40:01.002 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 80.634445130000003, 2024-02-05T06:40:01.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.89672592699999998, 2024-02-05T06:40:01.003 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:40:01.003 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:40:01.003 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:40:01.003 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:40:01.003 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:40:01.003 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 192724 2024-02-05T06:40:01.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:01.003 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:40:01.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:01.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:40:01.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.125512+0000", 2024-02-05T06:40:01.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:01.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:01.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:01.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:40:01.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.125512+0000", 2024-02-05T06:40:01.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:40:01.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:01.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:01.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:40:01.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.125514+0000", 2024-02-05T06:40:01.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2089999999999999e-06 2024-02-05T06:40:01.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:01.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:01.005 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:40:01.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.125520+0000", 2024-02-05T06:40:01.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6790000000000002e-06 2024-02-05T06:40:01.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:01.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:01.005 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:40:01.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.125535+0000", 2024-02-05T06:40:01.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5367999999999999e-05 2024-02-05T06:40:01.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:01.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:01.005 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:40:01.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.125543+0000", 2024-02-05T06:40:01.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5269999999999998e-06 2024-02-05T06:40:01.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:01.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:01.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:01.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.125602+0000", 2024-02-05T06:40:01.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8804000000000001e-05 2024-02-05T06:40:01.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:01.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:01.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:40:01.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.125611+0000", 2024-02-05T06:40:01.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8340000000000005e-06 2024-02-05T06:40:01.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:01.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:01.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:40:01.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.010174+0000", 2024-02-05T06:40:01.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.88456316700000004 2024-02-05T06:40:01.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:01.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:01.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:40:01.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.010303+0000", 2024-02-05T06:40:01.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012899 2024-02-05T06:40:01.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:01.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:01.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,1", 2024-02-05T06:40:01.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.010420+0000", 2024-02-05T06:40:01.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000117052 2024-02-05T06:40:01.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:01.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:01.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:40:01.008 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:01.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.018465+0000", 2024-02-05T06:40:01.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0080445859999999994 2024-02-05T06:40:01.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:01.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:01.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:01.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.022028+0000", 2024-02-05T06:40:01.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035627010000000001 2024-02-05T06:40:01.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:01.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:01.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:40:01.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.022153+0000", 2024-02-05T06:40:01.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012538999999999999 2024-02-05T06:40:01.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:01.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:01.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:40:01.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.022204+0000", 2024-02-05T06:40:01.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0899000000000002e-05 2024-02-05T06:40:01.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:40:01.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:40:01.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:40:01.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.022238+0000", 2024-02-05T06:40:01.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3720999999999998e-05 2024-02-05T06:40:01.010 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:01.010 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:40:01.010 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:01.010 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:40:01.010 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:40:01.011 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:40:01.050 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:01.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:01.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:01.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:01.194 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:01.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:01.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:01.224 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 6 16 17053 17037 177.392 150.312 0.00532901 0.005631 2024-02-05T06:40:01.295 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:01.327 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:01.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:01.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:01.429 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:01.466 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:01.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:01.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:01.568 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:01.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:01.656 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:01.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:01.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:01 smithi028 ceph-mon[37984]: pgmap v288: 158 pgs: 158 active+clean; 251 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 31 MiB/s wr, 501 op/s 2024-02-05T06:40:01.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:01 smithi028 ceph-mon[37984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:40:01.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:01 smithi028 ceph-mon[37984]: from='client.25057 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:40:01.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:01 smithi028 ceph-mon[37984]: osdmap e193: 8 total, 8 up, 7 in 2024-02-05T06:40:01.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:01 smithi028 ceph-mon[37984]: 4.8 scrub starts 2024-02-05T06:40:01.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:01 smithi028 ceph-mon[37984]: 4.8 scrub ok 2024-02-05T06:40:01.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:01 smithi028 ceph-mon[29442]: pgmap v288: 158 pgs: 158 active+clean; 251 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 31 MiB/s wr, 501 op/s 2024-02-05T06:40:01.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:01 smithi028 ceph-mon[29442]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:40:01.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:01 smithi028 ceph-mon[29442]: from='client.25057 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:40:01.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:01 smithi028 ceph-mon[29442]: osdmap e193: 8 total, 8 up, 7 in 2024-02-05T06:40:01.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:01 smithi028 ceph-mon[29442]: 4.8 scrub starts 2024-02-05T06:40:01.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:01 smithi028 ceph-mon[29442]: 4.8 scrub ok 2024-02-05T06:40:01.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:01 smithi167 ceph-mon[32774]: pgmap v288: 158 pgs: 158 active+clean; 251 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 31 MiB/s wr, 501 op/s 2024-02-05T06:40:01.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:01 smithi167 ceph-mon[32774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:40:01.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:01 smithi167 ceph-mon[32774]: from='client.25057 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:40:01.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:01 smithi167 ceph-mon[32774]: osdmap e193: 8 total, 8 up, 7 in 2024-02-05T06:40:01.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:01 smithi167 ceph-mon[32774]: 4.8 scrub starts 2024-02-05T06:40:01.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:01 smithi167 ceph-mon[32774]: 4.8 scrub ok 2024-02-05T06:40:01.757 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:01.798 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:01.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:01.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:01.900 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:01.940 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:01.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:01.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:02.042 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:02.083 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:02.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:02.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:02.185 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:40:02.185 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:02.216 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:02.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:02.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:02.224 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 7 16 19564 19548 174.469 156.938 0.00972922 0.00572362 2024-02-05T06:40:02.317 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:02.363 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:02.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:02.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:02.465 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:02.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:02 smithi028 ceph-mon[29442]: 4.1f scrub starts 2024-02-05T06:40:02.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:02 smithi028 ceph-mon[29442]: 4.1f scrub ok 2024-02-05T06:40:02.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:02 smithi028 ceph-mon[29442]: 6.13 scrub starts 2024-02-05T06:40:02.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:02 smithi028 ceph-mon[29442]: 6.13 scrub ok 2024-02-05T06:40:02.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:02 smithi028 ceph-mon[37984]: 4.1f scrub starts 2024-02-05T06:40:02.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:02 smithi028 ceph-mon[37984]: 4.1f scrub ok 2024-02-05T06:40:02.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:02 smithi028 ceph-mon[37984]: 6.13 scrub starts 2024-02-05T06:40:02.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:02 smithi028 ceph-mon[37984]: 6.13 scrub ok 2024-02-05T06:40:02.504 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:40:02 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:40:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:40:02.511 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:02.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:02.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:02.613 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:02.656 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:02.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:02.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:02.688 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:02 smithi167 ceph-mon[32774]: 4.1f scrub starts 2024-02-05T06:40:02.688 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:02 smithi167 ceph-mon[32774]: 4.1f scrub ok 2024-02-05T06:40:02.688 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:02 smithi167 ceph-mon[32774]: 6.13 scrub starts 2024-02-05T06:40:02.688 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:02 smithi167 ceph-mon[32774]: 6.13 scrub ok 2024-02-05T06:40:02.758 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:02.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:02.800 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:02.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:02.901 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:02.933 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:02.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:02.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:02.981 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:40:02 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:40:03.035 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:03.076 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:03.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:03.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:03.178 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:03.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:03.210 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:03.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:03.224 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 8 16 21883 21867 170.777 144.938 0.00442734 0.00584984 2024-02-05T06:40:03.311 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:03.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:03.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:03.456 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:03.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:03.504 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:03.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:03.605 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:03.651 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:03.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:03.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:03.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:03 smithi028 ceph-mon[37984]: pgmap v290: 158 pgs: 158 active+clean; 1.1 GiB data, 4.3 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 137 MiB/s wr, 2.20k op/s 2024-02-05T06:40:03.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:03 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:40:03.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:03 smithi028 ceph-mon[37984]: 2.0 scrub starts 2024-02-05T06:40:03.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:03 smithi028 ceph-mon[37984]: 2.0 scrub ok 2024-02-05T06:40:03.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:03 smithi028 ceph-mon[37984]: 1.a scrub starts 2024-02-05T06:40:03.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:03 smithi028 ceph-mon[37984]: 1.a scrub ok 2024-02-05T06:40:03.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:03 smithi028 ceph-mon[29442]: pgmap v290: 158 pgs: 158 active+clean; 1.1 GiB data, 4.3 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 137 MiB/s wr, 2.20k op/s 2024-02-05T06:40:03.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:03 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:40:03.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:03 smithi028 ceph-mon[29442]: 2.0 scrub starts 2024-02-05T06:40:03.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:03 smithi028 ceph-mon[29442]: 2.0 scrub ok 2024-02-05T06:40:03.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:03 smithi028 ceph-mon[29442]: 1.a scrub starts 2024-02-05T06:40:03.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:03 smithi028 ceph-mon[29442]: 1.a scrub ok 2024-02-05T06:40:03.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:03 smithi167 ceph-mon[32774]: pgmap v290: 158 pgs: 158 active+clean; 1.1 GiB data, 4.3 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 137 MiB/s wr, 2.20k op/s 2024-02-05T06:40:03.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:03 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:40:03.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:03 smithi167 ceph-mon[32774]: 2.0 scrub starts 2024-02-05T06:40:03.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:03 smithi167 ceph-mon[32774]: 2.0 scrub ok 2024-02-05T06:40:03.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:03 smithi167 ceph-mon[32774]: 1.a scrub starts 2024-02-05T06:40:03.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:03 smithi167 ceph-mon[32774]: 1.a scrub ok 2024-02-05T06:40:03.752 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:03.784 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:03.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:03.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:03.886 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:03.917 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:03.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:03.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:03.945 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-05T06:40:04.019 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:04.049 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:04.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:04.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:04.151 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:04.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:04.194 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:04.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:04.224 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 9 16 23996 23980 166.475 132.062 0.00732458 0.00600006 2024-02-05T06:40:04.295 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:04.318 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:04.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:04.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:04.420 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:04.458 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:04.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:04.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:04.560 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:04.593 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:04.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:04.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:04.694 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:04.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:04 smithi028 ceph-mon[37984]: 5.1 scrub starts 2024-02-05T06:40:04.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:04 smithi028 ceph-mon[37984]: 5.1 scrub ok 2024-02-05T06:40:04.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:04 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:40:04.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:04 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:40:04.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:04 smithi028 ceph-mon[37984]: osdmap e194: 8 total, 8 up, 7 in 2024-02-05T06:40:04.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:04 smithi028 ceph-mon[37984]: 5.17 scrub starts 2024-02-05T06:40:04.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:04 smithi028 ceph-mon[37984]: 5.17 scrub ok 2024-02-05T06:40:04.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:04 smithi028 ceph-mon[37984]: 3.d scrub starts 2024-02-05T06:40:04.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:04 smithi028 ceph-mon[37984]: 3.d scrub ok 2024-02-05T06:40:04.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:04 smithi028 ceph-mon[29442]: 5.1 scrub starts 2024-02-05T06:40:04.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:04 smithi028 ceph-mon[29442]: 5.1 scrub ok 2024-02-05T06:40:04.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:04 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:40:04.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:04 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:40:04.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:04 smithi028 ceph-mon[29442]: osdmap e194: 8 total, 8 up, 7 in 2024-02-05T06:40:04.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:04 smithi028 ceph-mon[29442]: 5.17 scrub starts 2024-02-05T06:40:04.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:04 smithi028 ceph-mon[29442]: 5.17 scrub ok 2024-02-05T06:40:04.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:04 smithi028 ceph-mon[29442]: 3.d scrub starts 2024-02-05T06:40:04.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:04 smithi028 ceph-mon[29442]: 3.d scrub ok 2024-02-05T06:40:04.725 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:04.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:04.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:04.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:04 smithi167 ceph-mon[32774]: 5.1 scrub starts 2024-02-05T06:40:04.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:04 smithi167 ceph-mon[32774]: 5.1 scrub ok 2024-02-05T06:40:04.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:04 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:40:04.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:04 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:40:04.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:04 smithi167 ceph-mon[32774]: osdmap e194: 8 total, 8 up, 7 in 2024-02-05T06:40:04.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:04 smithi167 ceph-mon[32774]: 5.17 scrub starts 2024-02-05T06:40:04.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:04 smithi167 ceph-mon[32774]: 5.17 scrub ok 2024-02-05T06:40:04.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:04 smithi167 ceph-mon[32774]: 3.d scrub starts 2024-02-05T06:40:04.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:04 smithi167 ceph-mon[32774]: 3.d scrub ok 2024-02-05T06:40:04.827 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:04.860 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:04.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:04.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:04.962 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:04.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:04.986 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:04.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:05.087 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:05.126 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:05.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:05.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:05.224 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 10 16 25884 25868 161.627 118 0.00471589 0.00618133 2024-02-05T06:40:05.229 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:05.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:05.263 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:05.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:05.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:05.402 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:05.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:05.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:05.504 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:05.538 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:05.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:05.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:05.641 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:05.678 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:05.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:05.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:05.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:05 smithi028 ceph-mon[29442]: pgmap v292: 158 pgs: 158 active+clean; 1.1 GiB data, 4.3 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 127 MiB/s wr, 2.04k op/s 2024-02-05T06:40:05.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:05 smithi028 ceph-mon[29442]: 3.18 scrub starts 2024-02-05T06:40:05.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:05 smithi028 ceph-mon[29442]: 3.18 scrub ok 2024-02-05T06:40:05.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:05 smithi028 ceph-mon[37984]: pgmap v292: 158 pgs: 158 active+clean; 1.1 GiB data, 4.3 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 127 MiB/s wr, 2.04k op/s 2024-02-05T06:40:05.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:05 smithi028 ceph-mon[37984]: 3.18 scrub starts 2024-02-05T06:40:05.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:05 smithi028 ceph-mon[37984]: 3.18 scrub ok 2024-02-05T06:40:05.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:05 smithi167 ceph-mon[32774]: pgmap v292: 158 pgs: 158 active+clean; 1.1 GiB data, 4.3 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 127 MiB/s wr, 2.04k op/s 2024-02-05T06:40:05.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:05 smithi167 ceph-mon[32774]: 3.18 scrub starts 2024-02-05T06:40:05.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:05 smithi167 ceph-mon[32774]: 3.18 scrub ok 2024-02-05T06:40:05.781 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:05.825 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:05.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:05.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:05.927 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:05.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:05.984 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:05.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:06.085 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:06.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:06.135 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:06.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:06.224 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 11 16 27880 27864 158.274 124.75 0.00713006 0.00631131 2024-02-05T06:40:06.236 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:06.268 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:06.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:06.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:06.370 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:06.408 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:40:06.409 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:06.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:06.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:06.499 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:40:06.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:06.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:06.562 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:06.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:06.663 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:06.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:06 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:40:06.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:06 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:40:06.704 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:06.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:06.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:06.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:06 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:40:06.806 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:06.843 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:06.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:06.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:06.945 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:06.983 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:06.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:06.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:07.085 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:07.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:07.130 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:07.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:07.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:06 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-a[29438]: 2024-02-05T06:40:06.916+0000 7fa733293700 -1 mon.a@0(leader).osd e194 definitely_dead 0 2024-02-05T06:40:07.224 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 12 16 29545 29529 153.757 104.062 0.00345518 0.00649839 2024-02-05T06:40:07.231 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:07.269 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:07.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:07.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:07.371 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:07.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:07.395 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:07.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:07.496 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:07.538 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:07.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:07.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:07.568 INFO:teuthology.orchestra.run.smithi028.stderr:marked out osd.5. 2024-02-05T06:40:07.640 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:07.663 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:07.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:07.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:07.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:07 smithi028 ceph-mon[37984]: 3.17 scrub starts 2024-02-05T06:40:07.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:07 smithi028 ceph-mon[37984]: 3.17 scrub ok 2024-02-05T06:40:07.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:07 smithi028 ceph-mon[37984]: pgmap v293: 158 pgs: 158 active+clean; 1.3 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 138 MiB/s wr, 2.21k op/s 2024-02-05T06:40:07.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:07 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:40:07.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:07 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:40:07.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:07 smithi028 ceph-mon[37984]: 6.3 scrub starts 2024-02-05T06:40:07.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:07 smithi028 ceph-mon[37984]: 6.3 scrub ok 2024-02-05T06:40:07.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:07 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1670777194' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-02-05T06:40:07.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:07 smithi028 ceph-mon[37984]: from='client.45050 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-02-05T06:40:07.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:07 smithi028 ceph-mon[37984]: Client client.admin marked osd.5 out, while it was still marked up 2024-02-05T06:40:07.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:07 smithi028 ceph-mon[29442]: 3.17 scrub starts 2024-02-05T06:40:07.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:07 smithi028 ceph-mon[29442]: 3.17 scrub ok 2024-02-05T06:40:07.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:07 smithi028 ceph-mon[29442]: pgmap v293: 158 pgs: 158 active+clean; 1.3 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 138 MiB/s wr, 2.21k op/s 2024-02-05T06:40:07.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:07 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:40:07.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:07 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:40:07.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:07 smithi028 ceph-mon[29442]: 6.3 scrub starts 2024-02-05T06:40:07.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:07 smithi028 ceph-mon[29442]: 6.3 scrub ok 2024-02-05T06:40:07.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:07 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1670777194' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-02-05T06:40:07.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:07 smithi028 ceph-mon[29442]: from='client.45050 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-02-05T06:40:07.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:07 smithi028 ceph-mon[29442]: Client client.admin marked osd.5 out, while it was still marked up 2024-02-05T06:40:07.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:07 smithi167 ceph-mon[32774]: 3.17 scrub starts 2024-02-05T06:40:07.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:07 smithi167 ceph-mon[32774]: 3.17 scrub ok 2024-02-05T06:40:07.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:07 smithi167 ceph-mon[32774]: pgmap v293: 158 pgs: 158 active+clean; 1.3 GiB data, 5.2 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 138 MiB/s wr, 2.21k op/s 2024-02-05T06:40:07.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:07 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:40:07.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:07 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:40:07.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:07 smithi167 ceph-mon[32774]: 6.3 scrub starts 2024-02-05T06:40:07.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:07 smithi167 ceph-mon[32774]: 6.3 scrub ok 2024-02-05T06:40:07.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:07 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1670777194' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-02-05T06:40:07.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:07 smithi167 ceph-mon[32774]: from='client.45050 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-02-05T06:40:07.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:07 smithi167 ceph-mon[32774]: Client client.admin marked osd.5 out, while it was still marked up 2024-02-05T06:40:07.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:07.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:07.790 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:07.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:07.889 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T06:40:07.890 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:07.917 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:07.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:07.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:08.019 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:08.042 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:08.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:08.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:08.144 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:08.172 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:08.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:08.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:08.224 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 13 16 30042 30026 144.32 31.0625 0.00403349 0.00657991 2024-02-05T06:40:08.274 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:08.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:08.299 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:08.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:08.400 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:08.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:08.429 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:08.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:08.530 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:08.558 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:08.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:08.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:08.660 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:08.686 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:08.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:08.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:08.788 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:08.811 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:08.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:08.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:08.913 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:08.938 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:08.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:08.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:08.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:08 smithi028 ceph-mon[29442]: 5.e scrub starts 2024-02-05T06:40:08.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:08 smithi028 ceph-mon[29442]: 5.e scrub ok 2024-02-05T06:40:08.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:08 smithi028 ceph-mon[29442]: from='client.45050 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-02-05T06:40:08.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:08 smithi028 ceph-mon[29442]: osdmap e195: 8 total, 8 up, 6 in 2024-02-05T06:40:08.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:08 smithi028 ceph-mon[29442]: 4.1e scrub starts 2024-02-05T06:40:08.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:08 smithi028 ceph-mon[29442]: 4.1e scrub ok 2024-02-05T06:40:08.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:08 smithi028 ceph-mon[37984]: 5.e scrub starts 2024-02-05T06:40:08.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:08 smithi028 ceph-mon[37984]: 5.e scrub ok 2024-02-05T06:40:08.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:08 smithi028 ceph-mon[37984]: from='client.45050 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-02-05T06:40:08.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:08 smithi028 ceph-mon[37984]: osdmap e195: 8 total, 8 up, 6 in 2024-02-05T06:40:08.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:08 smithi028 ceph-mon[37984]: 4.1e scrub starts 2024-02-05T06:40:08.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:08 smithi028 ceph-mon[37984]: 4.1e scrub ok 2024-02-05T06:40:08.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:08 smithi167 ceph-mon[32774]: 5.e scrub starts 2024-02-05T06:40:08.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:08 smithi167 ceph-mon[32774]: 5.e scrub ok 2024-02-05T06:40:08.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:08 smithi167 ceph-mon[32774]: from='client.45050 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-02-05T06:40:08.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:08 smithi167 ceph-mon[32774]: osdmap e195: 8 total, 8 up, 6 in 2024-02-05T06:40:08.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:08 smithi167 ceph-mon[32774]: 4.1e scrub starts 2024-02-05T06:40:08.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:08 smithi167 ceph-mon[32774]: 4.1e scrub ok 2024-02-05T06:40:09.040 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:09.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:09.065 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:09.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:09.166 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:09.195 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:09.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:09.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:09.224 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 14 16 30042 30026 134.012 0 - 0.00657991 2024-02-05T06:40:09.297 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:09.320 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:09.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:09.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:09.422 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:09.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:09.446 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:09.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:09.547 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:09.576 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:09.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:09.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:09.678 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:09.716 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:09.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:09.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:09.817 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:09.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:09.861 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:09.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:09.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:09 smithi028 ceph-mon[37984]: 3.1a scrub starts 2024-02-05T06:40:09.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:09 smithi028 ceph-mon[37984]: 3.1a scrub ok 2024-02-05T06:40:09.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:09 smithi028 ceph-mon[37984]: pgmap v295: 158 pgs: 158 active+clean; 1.7 GiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 1.3 KiB/s rd, 191 MiB/s wr, 3.06k op/s 2024-02-05T06:40:09.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:09 smithi028 ceph-mon[37984]: osdmap e196: 8 total, 8 up, 6 in 2024-02-05T06:40:09.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:09 smithi028 ceph-mon[29442]: 3.1a scrub starts 2024-02-05T06:40:09.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:09 smithi028 ceph-mon[29442]: 3.1a scrub ok 2024-02-05T06:40:09.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:09 smithi028 ceph-mon[29442]: pgmap v295: 158 pgs: 158 active+clean; 1.7 GiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 1.3 KiB/s rd, 191 MiB/s wr, 3.06k op/s 2024-02-05T06:40:09.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:09 smithi028 ceph-mon[29442]: osdmap e196: 8 total, 8 up, 6 in 2024-02-05T06:40:09.963 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:09.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:09 smithi167 ceph-mon[32774]: 3.1a scrub starts 2024-02-05T06:40:09.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:09 smithi167 ceph-mon[32774]: 3.1a scrub ok 2024-02-05T06:40:09.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:09 smithi167 ceph-mon[32774]: pgmap v295: 158 pgs: 158 active+clean; 1.7 GiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 1.3 KiB/s rd, 191 MiB/s wr, 3.06k op/s 2024-02-05T06:40:09.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:09 smithi167 ceph-mon[32774]: osdmap e196: 8 total, 8 up, 6 in 2024-02-05T06:40:09.992 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:09.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:09.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:10.094 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:10.132 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:10.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:10.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:10.224 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 15 16 30465 30449 126.841 13.2188 0.0151943 0.00787638 2024-02-05T06:40:10.234 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:10.275 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:10.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:10.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:10.377 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:10.405 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:10.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:10.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:10.507 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:10.540 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:10.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:10.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:10.642 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:10.671 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:10.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:10.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:10.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:10 smithi028 ceph-mon[29442]: 6.1c scrub starts 2024-02-05T06:40:10.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:10 smithi028 ceph-mon[29442]: 6.1c scrub ok 2024-02-05T06:40:10.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:10 smithi028 ceph-mon[29442]: osdmap e197: 8 total, 8 up, 6 in 2024-02-05T06:40:10.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:10 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:40:10.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:10 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:40:10.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:10 smithi028 ceph-mon[29442]: osdmap e198: 8 total, 8 up, 6 in 2024-02-05T06:40:10.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:10 smithi028 ceph-mon[37984]: 6.1c scrub starts 2024-02-05T06:40:10.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:10 smithi028 ceph-mon[37984]: 6.1c scrub ok 2024-02-05T06:40:10.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:10 smithi028 ceph-mon[37984]: osdmap e197: 8 total, 8 up, 6 in 2024-02-05T06:40:10.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:10 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:40:10.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:10 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:40:10.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:10 smithi028 ceph-mon[37984]: osdmap e198: 8 total, 8 up, 6 in 2024-02-05T06:40:10.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:10.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:10.810 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:10.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:10.912 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:10.941 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:10.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:10.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:10.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:10 smithi167 ceph-mon[32774]: 6.1c scrub starts 2024-02-05T06:40:10.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:10 smithi167 ceph-mon[32774]: 6.1c scrub ok 2024-02-05T06:40:10.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:10 smithi167 ceph-mon[32774]: osdmap e197: 8 total, 8 up, 6 in 2024-02-05T06:40:10.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:10 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:40:10.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:10 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:40:10.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:10 smithi167 ceph-mon[32774]: osdmap e198: 8 total, 8 up, 6 in 2024-02-05T06:40:11.043 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:11.087 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:11.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:11.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:11.189 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:11.224 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 16 15 31383 31368 122.504 57.4375 0.020807 0.00815305 2024-02-05T06:40:11.232 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:11.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:11.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:11.335 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:11.363 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:11.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:11.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:11.465 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:11.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:11.529 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:11.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:11.630 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:11.658 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:11.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:11.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:11.760 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:11.787 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:11.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:11.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:11.860 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:11 smithi028 ceph-mon[29442]: pgmap v299: 158 pgs: 158 active+clean; 1.7 GiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.62k op/s 2024-02-05T06:40:11.860 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:11 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:40:11.861 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:11 smithi028 ceph-mon[29442]: osdmap e199: 8 total, 8 up, 6 in 2024-02-05T06:40:11.861 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:11 smithi028 ceph-mon[37984]: pgmap v299: 158 pgs: 158 active+clean; 1.7 GiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.62k op/s 2024-02-05T06:40:11.861 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:11 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:40:11.861 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:11 smithi028 ceph-mon[37984]: osdmap e199: 8 total, 8 up, 6 in 2024-02-05T06:40:11.889 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:11.918 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:11.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:11.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:11.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:11 smithi167 ceph-mon[32774]: pgmap v299: 158 pgs: 158 active+clean; 1.7 GiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.62k op/s 2024-02-05T06:40:11.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:11 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:40:11.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:11 smithi167 ceph-mon[32774]: osdmap e199: 8 total, 8 up, 6 in 2024-02-05T06:40:12.020 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:12.065 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:12.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:12.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:12.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:12.214 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:12.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:12.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:12.225 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 17 16 32272 32256 118.563 55.5 0.0125369 0.00842433 2024-02-05T06:40:12.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:12.360 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:12.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:12.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:12.462 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:12.506 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:12.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:12.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:12.607 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:12.634 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:12.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:12.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:12.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:12 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:40:12.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:12 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:40:12.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:12 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:40:12.670 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:40:12 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:40:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:40:12.670 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:12 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:40:12.670 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:12 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:40:12.670 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:12 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:40:12.736 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:12.765 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:12.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:12.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:12.867 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:12.908 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:12.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:12.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:12.981 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:40:12 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:40:12.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:12 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:40:12.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:12 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:40:12.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:12 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:40:13.010 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:13.038 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:13.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:13.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:13.140 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:13.169 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:13.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:13.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:13.225 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 18 16 33171 33155 115.097 56.1875 0.0207001 0.00868117 2024-02-05T06:40:13.271 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:13.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:13.320 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:13.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:13.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:13.458 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:13.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:13.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:13.560 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:13.603 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:13.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:13.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:13.705 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:13.735 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:13.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:13.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:13.837 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:13.881 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:13.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:13.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:13.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:13 smithi028 ceph-mon[29442]: pgmap v301: 158 pgs: 7 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 149 active+clean; 1.9 GiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 21 KiB/s rd, 54 MiB/s wr, 890 op/s; 1942/95469 objects degraded (2.034%); 14241/95469 objects misplaced (14.917%); 13 MiB/s, 219 objects/s recovering 2024-02-05T06:40:13.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:13 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:40:13.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:13 smithi028 ceph-mon[29442]: Health check failed: Degraded data redundancy: 1942/95469 objects degraded (2.034%), 2 pgs degraded (PG_DEGRADED) 2024-02-05T06:40:13.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:13 smithi028 ceph-mon[37984]: pgmap v301: 158 pgs: 7 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 149 active+clean; 1.9 GiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 21 KiB/s rd, 54 MiB/s wr, 890 op/s; 1942/95469 objects degraded (2.034%); 14241/95469 objects misplaced (14.917%); 13 MiB/s, 219 objects/s recovering 2024-02-05T06:40:13.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:13 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:40:13.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:13 smithi028 ceph-mon[37984]: Health check failed: Degraded data redundancy: 1942/95469 objects degraded (2.034%), 2 pgs degraded (PG_DEGRADED) 2024-02-05T06:40:13.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:13 smithi167 ceph-mon[32774]: pgmap v301: 158 pgs: 7 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 149 active+clean; 1.9 GiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 21 KiB/s rd, 54 MiB/s wr, 890 op/s; 1942/95469 objects degraded (2.034%); 14241/95469 objects misplaced (14.917%); 13 MiB/s, 219 objects/s recovering 2024-02-05T06:40:13.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:13 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:40:13.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:13 smithi167 ceph-mon[32774]: Health check failed: Degraded data redundancy: 1942/95469 objects degraded (2.034%), 2 pgs degraded (PG_DEGRADED) 2024-02-05T06:40:13.983 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:14.013 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:14.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:14.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:14.115 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:14.159 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:14.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:14.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:14.225 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 19 16 34128 34112 112.188 59.8125 0.014779 0.00890491 2024-02-05T06:40:14.261 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:14.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:14.304 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:14.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:14.405 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:14.433 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:14.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:14.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:14.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:14.590 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:14.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:14.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:14.692 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:14.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:14.762 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:14.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:14.826 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:40:14.863 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:14.868 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:40:14.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:14.910 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:14.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:14.918 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:40:15.011 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:15.038 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:15.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:15.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:15.057 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:40:15.096 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:40:15.134 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:40:15.140 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:15.168 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:15.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:15.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:15.175 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:40:15.202 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:40:15.203 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:40:15.203 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:40:15.203 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:40:15.203 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:40:15.204 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:40:15.204 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:40:15.204 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:40:15.204 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:40:15.225 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-02-05T06:40:15.221633+0000 min lat: 0.00125405 max lat: 2.2409 avg lat: 0.00916241 2024-02-05T06:40:15.225 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T06:40:15.225 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 20 16 34846 34830 108.822 44.875 0.0030866 0.00916241 2024-02-05T06:40:15.271 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:15.313 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:15.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:15.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:15.415 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:15.439 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:15.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:15.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:15.541 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:15.585 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:15.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:15.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:15.687 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:15.727 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:15.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:15.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:15.829 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:15.858 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:15.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:15.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:15.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:15 smithi028 ceph-mon[29442]: pgmap v302: 158 pgs: 7 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 149 active+clean; 1.9 GiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 15 KiB/s rd, 38 MiB/s wr, 622 op/s; 1942/95469 objects degraded (2.034%); 14241/95469 objects misplaced (14.917%); 9.2 MiB/s, 153 objects/s recovering 2024-02-05T06:40:15.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:15 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2414714639' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:40:15.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:15 smithi028 ceph-mon[29442]: osdmap e200: 8 total, 8 up, 6 in 2024-02-05T06:40:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:15 smithi028 ceph-mon[37984]: pgmap v302: 158 pgs: 7 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 149 active+clean; 1.9 GiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 15 KiB/s rd, 38 MiB/s wr, 622 op/s; 1942/95469 objects degraded (2.034%); 14241/95469 objects misplaced (14.917%); 9.2 MiB/s, 153 objects/s recovering 2024-02-05T06:40:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:15 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2414714639' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:40:15.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:15 smithi028 ceph-mon[37984]: osdmap e200: 8 total, 8 up, 6 in 2024-02-05T06:40:15.960 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:15.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:15 smithi167 ceph-mon[32774]: pgmap v302: 158 pgs: 7 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 149 active+clean; 1.9 GiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 15 KiB/s rd, 38 MiB/s wr, 622 op/s; 1942/95469 objects degraded (2.034%); 14241/95469 objects misplaced (14.917%); 9.2 MiB/s, 153 objects/s recovering 2024-02-05T06:40:15.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:15 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2414714639' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:40:15.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:15 smithi167 ceph-mon[32774]: osdmap e200: 8 total, 8 up, 6 in 2024-02-05T06:40:15.984 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:15.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:15.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:16.086 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:16.115 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:16.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:16.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:16.217 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:16.227 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 21 16 35651 35635 106.025 50.3125 0.0118323 0.00942151 2024-02-05T06:40:16.259 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:16.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:16.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:16.360 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:16.389 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:16.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:16.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:16.491 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:16.522 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:16.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:16.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:16.624 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:16.670 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:16.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:16.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:16.771 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:16.816 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:16.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:16.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:16.918 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:16.947 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:16.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:16.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:17.051 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:17.081 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:17.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:17.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:17.182 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:17.211 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:17.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:17.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:17.227 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 22 16 36615 36599 103.944 60.25 0.0131106 0.00961102 2024-02-05T06:40:17.312 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:40:17.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:17.363 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:40:17.363 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:40:17.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:17.370 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:17.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:17.459 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 4] out_osds: [7, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T06:40:17.460 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T06:40:17.460 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-02-05T06:40:17.460 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd in 7 2024-02-05T06:40:17.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:17.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:17.565 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:17.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:17.666 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:17.695 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:17.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:17.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:17.797 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:17.824 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:17.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:17.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:17.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:17.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:17 smithi028 ceph-mon[37984]: pgmap v304: 158 pgs: 7 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 149 active+clean; 2.0 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 13 KiB/s rd, 45 MiB/s wr, 734 op/s; 1942/98604 objects degraded (1.969%); 12705/98604 objects misplaced (12.885%); 24 MiB/s, 384 objects/s recovering 2024-02-05T06:40:17.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:17 smithi028 ceph-mon[29442]: pgmap v304: 158 pgs: 7 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 149 active+clean; 2.0 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 13 KiB/s rd, 45 MiB/s wr, 734 op/s; 1942/98604 objects degraded (1.969%); 12705/98604 objects misplaced (12.885%); 24 MiB/s, 384 objects/s recovering 2024-02-05T06:40:17.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:17.965 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:17.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:17.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:17 smithi167 ceph-mon[32774]: pgmap v304: 158 pgs: 7 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 149 active+clean; 2.0 GiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 13 KiB/s rd, 45 MiB/s wr, 734 op/s; 1942/98604 objects degraded (1.969%); 12705/98604 objects misplaced (12.885%); 24 MiB/s, 384 objects/s recovering 2024-02-05T06:40:18.066 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:18.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:18.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:18.191 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-05T06:40:18.210 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:18.227 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 23 15 37470 37455 101.751 53.5 0.023831 0.00982057 2024-02-05T06:40:18.238 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:18.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:18.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:18.339 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:18.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:18.395 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:18.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:18.496 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:18.524 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:18.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:18.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:18.626 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:18.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:18.669 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:18.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:18.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:18 smithi028 ceph-mon[37984]: Health check update: Degraded data redundancy: 1942/108396 objects degraded (1.792%), 2 pgs degraded (PG_DEGRADED) 2024-02-05T06:40:18.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:18 smithi028 ceph-mon[29442]: Health check update: Degraded data redundancy: 1942/108396 objects degraded (1.792%), 2 pgs degraded (PG_DEGRADED) 2024-02-05T06:40:18.770 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:18.796 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:18.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:18.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:18.898 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:18.926 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:18.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:18.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:18.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:18 smithi167 ceph-mon[32774]: Health check update: Degraded data redundancy: 1942/108396 objects degraded (1.792%), 2 pgs degraded (PG_DEGRADED) 2024-02-05T06:40:19.027 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:19.039 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:40:19.056 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:19.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:19.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:19.158 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:19.186 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:19.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:19.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:19.228 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 24 16 38353 38337 99.8085 55.125 0.0185584 0.0100106 2024-02-05T06:40:19.287 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:19.338 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:19.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:19.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:19.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:19.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:19.495 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:19.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:19.586 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:40:19.597 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:19.623 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:19.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:19.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:19.686 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:40:19.725 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:19.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:19.752 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:19.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:19.775 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:19 smithi028 ceph-mon[37984]: pgmap v305: 158 pgs: 7 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 149 active+clean; 2.2 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 11 KiB/s rd, 61 MiB/s wr, 984 op/s; 1942/108396 objects degraded (1.792%); 11634/108396 objects misplaced (10.733%); 27 MiB/s, 435 objects/s recovering 2024-02-05T06:40:19.777 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:19 smithi028 ceph-mon[29442]: pgmap v305: 158 pgs: 7 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 149 active+clean; 2.2 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 11 KiB/s rd, 61 MiB/s wr, 984 op/s; 1942/108396 objects degraded (1.792%); 11634/108396 objects misplaced (10.733%); 27 MiB/s, 435 objects/s recovering 2024-02-05T06:40:19.853 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:19.880 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:19.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:19.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:19.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:19 smithi167 ceph-mon[32774]: pgmap v305: 158 pgs: 7 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 149 active+clean; 2.2 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 11 KiB/s rd, 61 MiB/s wr, 984 op/s; 1942/108396 objects degraded (1.792%); 11634/108396 objects misplaced (10.733%); 27 MiB/s, 435 objects/s recovering 2024-02-05T06:40:19.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:20.030 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:20.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:20.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:20.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:20.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:20.176 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:20.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:20.228 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 25 16 39077 39061 97.6264 45.25 0.0147622 0.0102337 2024-02-05T06:40:20.277 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:20.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:20.325 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:20.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:20.426 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:20.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:20.462 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:20.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:20.563 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:20.588 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:20.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:20.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:20.690 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:20.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:20.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:20.832 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:20.861 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:20.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:20.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:20.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:20 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:40:20.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:20 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:40:20.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:20 smithi028 ceph-mon[37984]: osdmap e201: 8 total, 8 up, 6 in 2024-02-05T06:40:20.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:20 smithi028 ceph-mon[37984]: osdmap e202: 8 total, 8 up, 6 in 2024-02-05T06:40:20.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:20 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:40:20.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:20 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:40:20.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:20 smithi028 ceph-mon[29442]: osdmap e201: 8 total, 8 up, 6 in 2024-02-05T06:40:20.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:20 smithi028 ceph-mon[29442]: osdmap e202: 8 total, 8 up, 6 in 2024-02-05T06:40:20.963 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:20.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:20 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:40:20.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:20 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:40:20.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:20 smithi167 ceph-mon[32774]: osdmap e201: 8 total, 8 up, 6 in 2024-02-05T06:40:20.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:20 smithi167 ceph-mon[32774]: osdmap e202: 8 total, 8 up, 6 in 2024-02-05T06:40:20.990 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:20.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:20.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:21.092 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:21.119 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:21.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:21.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:21.221 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:21.228 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 26 16 39968 39952 96.0134 55.6875 0.013777 0.0104074 2024-02-05T06:40:21.248 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:21.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:21.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:21.350 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:21.406 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:21.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:21.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:21.508 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:21.535 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:21.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:21.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:21.637 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:21.681 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:21.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:21.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:21.783 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:21.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:21.824 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:21.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:21.886 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:40:21.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:21.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:21 smithi028 ceph-mon[29442]: pgmap v308: 158 pgs: 7 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 149 active+clean; 2.2 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 853 B/s rd, 45 MiB/s wr, 718 op/s; 1942/108396 objects degraded (1.792%); 11634/108396 objects misplaced (10.733%); 27 MiB/s, 434 objects/s recovering 2024-02-05T06:40:21.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:21 smithi028 ceph-mon[37984]: pgmap v308: 158 pgs: 7 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 149 active+clean; 2.2 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 853 B/s rd, 45 MiB/s wr, 718 op/s; 1942/108396 objects degraded (1.792%); 11634/108396 objects misplaced (10.733%); 27 MiB/s, 434 objects/s recovering 2024-02-05T06:40:21.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:21 smithi167 ceph-mon[32774]: pgmap v308: 158 pgs: 7 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 149 active+clean; 2.2 GiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 853 B/s rd, 45 MiB/s wr, 718 op/s; 1942/108396 objects degraded (1.792%); 11634/108396 objects misplaced (10.733%); 27 MiB/s, 434 objects/s recovering 2024-02-05T06:40:21.982 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:21.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:21.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:22.084 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:22.139 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:22.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:22.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:22.228 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 27 16 40883 40867 94.5755 57.1875 0.0134331 0.0105666 2024-02-05T06:40:22.242 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:22.270 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:22.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:22.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:22.372 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:22.398 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:22.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:22.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:22.500 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:22.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:22.559 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:22.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:22.660 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:22.687 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:22.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:22.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:22.696 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:40:22 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:40:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:40:22.789 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:22.845 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:22.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:22.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:22.948 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:22.973 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:22.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:22.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:22.981 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:40:22 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:40:23.076 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:23.103 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:23.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:23.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:23.205 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:23.228 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 28 16 41781 41765 93.2022 56.125 0.00834804 0.0107182 2024-02-05T06:40:23.230 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:23.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:23.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:23.331 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:23.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:23.358 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:23.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:23.459 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:23.505 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:23.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:23.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:23.607 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:23.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:23.650 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:23.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:23.697 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-02-05T06:40:23.752 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:23.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:23.778 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:23.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:23.880 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:23.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:23.926 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:23.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:23.945 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:23 smithi028 ceph-mon[37984]: pgmap v309: 158 pgs: 6 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 150 active+clean; 2.5 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 1.4 KiB/s rd, 74 MiB/s wr, 1.18k op/s; 1942/121209 objects degraded (1.602%); 8925/121209 objects misplaced (7.363%); 46 MiB/s, 730 objects/s recovering 2024-02-05T06:40:23.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:23 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:40:23.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:23 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1958266014' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:40:23.946 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:23 smithi028 ceph-mon[37984]: from='client.25075 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:40:23.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:23 smithi028 ceph-mon[29442]: pgmap v309: 158 pgs: 6 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 150 active+clean; 2.5 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 1.4 KiB/s rd, 74 MiB/s wr, 1.18k op/s; 1942/121209 objects degraded (1.602%); 8925/121209 objects misplaced (7.363%); 46 MiB/s, 730 objects/s recovering 2024-02-05T06:40:23.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:23 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:40:23.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:23 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1958266014' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:40:23.946 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:23 smithi028 ceph-mon[29442]: from='client.25075 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:40:23.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:23 smithi167 ceph-mon[32774]: pgmap v309: 158 pgs: 6 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 150 active+clean; 2.5 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 1.4 KiB/s rd, 74 MiB/s wr, 1.18k op/s; 1942/121209 objects degraded (1.602%); 8925/121209 objects misplaced (7.363%); 46 MiB/s, 730 objects/s recovering 2024-02-05T06:40:23.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:23 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:40:23.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:23 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1958266014' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:40:23.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:23 smithi167 ceph-mon[32774]: from='client.25075 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:40:24.027 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:24.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:24.056 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:24.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:24.156 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:24.181 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:24.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:24.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:24.228 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 29 16 42558 42542 91.6629 48.5625 0.0444528 0.010901 2024-02-05T06:40:24.284 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:24.324 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:24.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:24.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:24.427 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:24.470 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:24.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:24.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:24.572 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:24.621 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:24.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:24.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:24.723 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:24.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:24.775 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:24.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:24.876 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:24.902 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:24.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:24.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:24.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:24 smithi028 ceph-mon[29442]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:40:24.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:24 smithi028 ceph-mon[29442]: from='client.25075 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:40:24.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:24 smithi028 ceph-mon[29442]: osdmap e203: 8 total, 8 up, 6 in 2024-02-05T06:40:24.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:24 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/230704025' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:40:24.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:24 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:40:24.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:24 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:40:24.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:24 smithi028 ceph-mon[37984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:40:24.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:24 smithi028 ceph-mon[37984]: from='client.25075 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:40:24.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:24 smithi028 ceph-mon[37984]: osdmap e203: 8 total, 8 up, 6 in 2024-02-05T06:40:24.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:24 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/230704025' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:40:24.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:24 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:40:24.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:24 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:40:24.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:24 smithi167 ceph-mon[32774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:40:24.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:24 smithi167 ceph-mon[32774]: from='client.25075 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:40:24.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:24 smithi167 ceph-mon[32774]: osdmap e203: 8 total, 8 up, 6 in 2024-02-05T06:40:24.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:24 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/230704025' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:40:24.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:24 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:40:24.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:24 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:40:25.005 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:25.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:25.049 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:25.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:25.075 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:40:25.075 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:40:25.076 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:40:25.076 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:40:25.076 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:40:25.076 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:40:25.076 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:40:25.076 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:40:25.076 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:40:25.076 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:40:25.076 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:40:25.076 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:40:25.076 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:40:25.077 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:40:25.077 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:40:25.077 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:40:25.150 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:25.177 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:25.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:25.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:25.228 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 30 16 43423 43407 90.4096 54.0625 0.0181128 0.0110534 2024-02-05T06:40:25.279 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:25.305 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:25.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:25.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:25.407 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:25.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:25.458 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:25.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:25.560 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:25.603 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:25.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:25.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:25.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:25 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-a[29438]: 2024-02-05T06:40:25.472+0000 7fa733293700 -1 mon.a@0(leader).osd e204 definitely_dead 0 2024-02-05T06:40:25.704 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:25.743 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:25.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:25.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:25.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:25.887 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:25.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:25.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:25.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:25 smithi167 ceph-mon[32774]: pgmap v311: 158 pgs: 6 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 150 active+clean; 2.5 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 852 B/s rd, 44 MiB/s wr, 712 op/s; 1942/121209 objects degraded (1.602%); 8925/121209 objects misplaced (7.363%); 28 MiB/s, 451 objects/s recovering 2024-02-05T06:40:25.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:25 smithi167 ceph-mon[32774]: osdmap e204: 8 total, 8 up, 6 in 2024-02-05T06:40:25.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:25 smithi167 ceph-mon[32774]: Health check update: Degraded data redundancy: 1942/121209 objects degraded (1.602%), 2 pgs degraded (PG_DEGRADED) 2024-02-05T06:40:25.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:25 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/94459160' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-02-05T06:40:25.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:25 smithi167 ceph-mon[32774]: from='client.25087 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-02-05T06:40:25.989 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:26.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:26.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:26.059 INFO:teuthology.orchestra.run.smithi028.stderr:marked in osd.7. 2024-02-05T06:40:26.060 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:25 smithi028 ceph-mon[37984]: pgmap v311: 158 pgs: 6 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 150 active+clean; 2.5 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 852 B/s rd, 44 MiB/s wr, 712 op/s; 1942/121209 objects degraded (1.602%); 8925/121209 objects misplaced (7.363%); 28 MiB/s, 451 objects/s recovering 2024-02-05T06:40:26.060 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:25 smithi028 ceph-mon[37984]: osdmap e204: 8 total, 8 up, 6 in 2024-02-05T06:40:26.060 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:25 smithi028 ceph-mon[37984]: Health check update: Degraded data redundancy: 1942/121209 objects degraded (1.602%), 2 pgs degraded (PG_DEGRADED) 2024-02-05T06:40:26.060 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:25 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/94459160' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-02-05T06:40:26.060 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:25 smithi028 ceph-mon[37984]: from='client.25087 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-02-05T06:40:26.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:25 smithi028 ceph-mon[29442]: pgmap v311: 158 pgs: 6 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 150 active+clean; 2.5 GiB data, 8.0 GiB used, 528 GiB / 536 GiB avail; 852 B/s rd, 44 MiB/s wr, 712 op/s; 1942/121209 objects degraded (1.602%); 8925/121209 objects misplaced (7.363%); 28 MiB/s, 451 objects/s recovering 2024-02-05T06:40:26.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:25 smithi028 ceph-mon[29442]: osdmap e204: 8 total, 8 up, 6 in 2024-02-05T06:40:26.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:25 smithi028 ceph-mon[29442]: Health check update: Degraded data redundancy: 1942/121209 objects degraded (1.602%), 2 pgs degraded (PG_DEGRADED) 2024-02-05T06:40:26.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:25 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/94459160' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-02-05T06:40:26.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:25 smithi028 ceph-mon[29442]: from='client.25087 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-02-05T06:40:26.134 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:26.165 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:26.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:26.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:26.228 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 31 16 44159 44143 88.9771 46 0.00408825 0.0111752 2024-02-05T06:40:26.267 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:26.298 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:26.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:26.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:26.400 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:26.425 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:26.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:26.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:26.527 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:26.553 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:26.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:26.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:26.655 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:26.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:26.680 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:26.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:26.741 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T06:40:26.782 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:26.806 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:26.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:26.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:26.908 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:26.941 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:26.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:26.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:27.043 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:27.093 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:27.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:27.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:27.195 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:27.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:26 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:40:27.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:26 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:40:27.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:26 smithi028 ceph-mon[37984]: from='client.25087 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-02-05T06:40:27.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:26 smithi028 ceph-mon[37984]: osdmap e205: 8 total, 8 up, 7 in 2024-02-05T06:40:27.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:26 smithi028 ceph-mon[37984]: pgmap v314: 158 pgs: 29 peering, 3 unknown, 6 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.5 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 56 MiB/s wr, 888 op/s; 1942/124389 objects degraded (1.561%); 8147/124389 objects misplaced (6.550%); 36 MiB/s, 578 objects/s recovering 2024-02-05T06:40:27.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:26 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:40:27.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:26 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:40:27.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:26 smithi028 ceph-mon[29442]: from='client.25087 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-02-05T06:40:27.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:26 smithi028 ceph-mon[29442]: osdmap e205: 8 total, 8 up, 7 in 2024-02-05T06:40:27.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:26 smithi028 ceph-mon[29442]: pgmap v314: 158 pgs: 29 peering, 3 unknown, 6 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.5 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 56 MiB/s wr, 888 op/s; 1942/124389 objects degraded (1.561%); 8147/124389 objects misplaced (6.550%); 36 MiB/s, 578 objects/s recovering 2024-02-05T06:40:27.222 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:27.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:27.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:27.228 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 32 16 44172 44156 86.2223 0.8125 1.14103 0.0113578 2024-02-05T06:40:27.231 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:26 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:40:27.231 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:26 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:40:27.231 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:26 smithi167 ceph-mon[32774]: from='client.25087 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-02-05T06:40:27.231 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:26 smithi167 ceph-mon[32774]: osdmap e205: 8 total, 8 up, 7 in 2024-02-05T06:40:27.231 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:26 smithi167 ceph-mon[32774]: pgmap v314: 158 pgs: 29 peering, 3 unknown, 6 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 118 active+clean; 2.5 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 56 MiB/s wr, 888 op/s; 1942/124389 objects degraded (1.561%); 8147/124389 objects misplaced (6.550%); 36 MiB/s, 578 objects/s recovering 2024-02-05T06:40:27.324 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:27.349 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:27.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:27.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:27.450 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:27.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:27.475 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:27.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:27.576 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:27.601 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:27.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:27.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:27.703 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:27.727 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:27.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:27.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:27.829 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:27.854 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:27.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:27.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:27.956 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:27.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:27.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:28.089 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:28.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:28.125 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:28.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:28.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:27 smithi028 ceph-mon[37984]: Health check failed: Reduced data availability: 11 pgs inactive, 14 pgs peering (PG_AVAILABILITY) 2024-02-05T06:40:28.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:27 smithi028 ceph-mon[37984]: osdmap e206: 8 total, 8 up, 7 in 2024-02-05T06:40:28.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:27 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:40:28.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:27 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:40:28.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:27 smithi028 ceph-mon[29442]: Health check failed: Reduced data availability: 11 pgs inactive, 14 pgs peering (PG_AVAILABILITY) 2024-02-05T06:40:28.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:27 smithi028 ceph-mon[29442]: osdmap e206: 8 total, 8 up, 7 in 2024-02-05T06:40:28.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:27 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:40:28.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:27 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:40:28.226 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:28.230 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 33 16 44229 44213 83.7178 3.5625 0.0174419 0.0119362 2024-02-05T06:40:28.231 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:27 smithi167 ceph-mon[32774]: Health check failed: Reduced data availability: 11 pgs inactive, 14 pgs peering (PG_AVAILABILITY) 2024-02-05T06:40:28.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:27 smithi167 ceph-mon[32774]: osdmap e206: 8 total, 8 up, 7 in 2024-02-05T06:40:28.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:27 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:40:28.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:27 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:40:28.267 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:28.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:28.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:28.369 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:28.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:28.411 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:28.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:28.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:28.545 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:28.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:28.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:28.647 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:28.676 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:28.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:28.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:28.778 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:28.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:28.819 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:28.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:28.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:28.959 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:28.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:28.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:29.060 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:29.111 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:29.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:29.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:29.213 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:29.229 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 34 16 44992 44976 82.6582 47.6875 0.0079513 0.0120904 2024-02-05T06:40:29.250 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:29.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:29.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:29.352 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:29.376 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:29.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:29.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:29.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:29 smithi028 ceph-mon[29442]: osdmap e207: 8 total, 8 up, 7 in 2024-02-05T06:40:29.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:29 smithi028 ceph-mon[29442]: pgmap v317: 158 pgs: 3 remapped+peering, 61 peering, 3 unknown, 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 86 active+clean; 2.7 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 62 MiB/s wr, 985 op/s; 1/133044 objects degraded (0.001%); 5546/133044 objects misplaced (4.169%); 33 MiB/s, 526 objects/s recovering 2024-02-05T06:40:29.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:29 smithi028 ceph-mon[37984]: osdmap e207: 8 total, 8 up, 7 in 2024-02-05T06:40:29.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:29 smithi028 ceph-mon[37984]: pgmap v317: 158 pgs: 3 remapped+peering, 61 peering, 3 unknown, 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 86 active+clean; 2.7 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 62 MiB/s wr, 985 op/s; 1/133044 objects degraded (0.001%); 5546/133044 objects misplaced (4.169%); 33 MiB/s, 526 objects/s recovering 2024-02-05T06:40:29.478 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:29.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:29 smithi167 ceph-mon[32774]: osdmap e207: 8 total, 8 up, 7 in 2024-02-05T06:40:29.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:29 smithi167 ceph-mon[32774]: pgmap v317: 158 pgs: 3 remapped+peering, 61 peering, 3 unknown, 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 86 active+clean; 2.7 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 62 MiB/s wr, 985 op/s; 1/133044 objects degraded (0.001%); 5546/133044 objects misplaced (4.169%); 33 MiB/s, 526 objects/s recovering 2024-02-05T06:40:29.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:29.512 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:29.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:29.613 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:29.643 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:29.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:29.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:29.745 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:29.771 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:29.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:29.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:29.873 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:29.902 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:29.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:29.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:30.004 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:30.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:30.030 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:30.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:30.131 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:30.163 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:30.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:30.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:30.229 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 35 16 45679 45663 81.5232 42.9375 0.0220794 0.0122593 2024-02-05T06:40:30.266 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:30.295 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:30.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:30.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:30.397 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:30.433 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:30.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:30.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:30.446 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:30 smithi028 ceph-mon[37984]: osdmap e208: 8 total, 8 up, 7 in 2024-02-05T06:40:30.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:30 smithi028 ceph-mon[37984]: Health check update: Degraded data redundancy: 1/133044 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-02-05T06:40:30.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:30 smithi028 ceph-mon[29442]: osdmap e208: 8 total, 8 up, 7 in 2024-02-05T06:40:30.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:30 smithi028 ceph-mon[29442]: Health check update: Degraded data redundancy: 1/133044 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-02-05T06:40:30.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:30 smithi167 ceph-mon[32774]: osdmap e208: 8 total, 8 up, 7 in 2024-02-05T06:40:30.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:30 smithi167 ceph-mon[32774]: Health check update: Degraded data redundancy: 1/133044 objects degraded (0.001%), 1 pg degraded (PG_DEGRADED) 2024-02-05T06:40:30.536 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:30.575 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:30.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:30.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:30.676 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:30.706 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:30.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:30.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:30.808 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:30.841 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:30.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:30.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:30.944 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:30.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:30.992 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:30.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:31.093 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:31.123 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:31.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:31.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:31.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:31.229 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 36 16 46576 46560 80.8159 56.0625 0.004072 0.0123591 2024-02-05T06:40:31.261 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:31.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:31.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:31.363 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:31.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:31.392 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:31.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:31.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:31 smithi028 ceph-mon[29442]: osdmap e209: 8 total, 8 up, 7 in 2024-02-05T06:40:31.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:31 smithi028 ceph-mon[29442]: pgmap v320: 158 pgs: 3 remapped+peering, 61 peering, 3 unknown, 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 86 active+clean; 2.7 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 45 MiB/s wr, 720 op/s; 1/133044 objects degraded (0.001%); 5546/133044 objects misplaced (4.169%); 21 MiB/s, 336 objects/s recovering 2024-02-05T06:40:31.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:31 smithi028 ceph-mon[37984]: osdmap e209: 8 total, 8 up, 7 in 2024-02-05T06:40:31.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:31 smithi028 ceph-mon[37984]: pgmap v320: 158 pgs: 3 remapped+peering, 61 peering, 3 unknown, 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 86 active+clean; 2.7 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 45 MiB/s wr, 720 op/s; 1/133044 objects degraded (0.001%); 5546/133044 objects misplaced (4.169%); 21 MiB/s, 336 objects/s recovering 2024-02-05T06:40:31.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:31 smithi167 ceph-mon[32774]: osdmap e209: 8 total, 8 up, 7 in 2024-02-05T06:40:31.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:31 smithi167 ceph-mon[32774]: pgmap v320: 158 pgs: 3 remapped+peering, 61 peering, 3 unknown, 4 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 86 active+clean; 2.7 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 45 MiB/s wr, 720 op/s; 1/133044 objects degraded (0.001%); 5546/133044 objects misplaced (4.169%); 21 MiB/s, 336 objects/s recovering 2024-02-05T06:40:31.493 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:31.522 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:31.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:31.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:31.624 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:31.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:31.650 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:31.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:31.751 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:31.778 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:31.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:31.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:31.880 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:31.907 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:31.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:31.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:32.009 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:32.037 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:32.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:32.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:32.139 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:32.171 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:32.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:32.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:32.229 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 37 16 46858 46842 79.1081 17.625 0.00944115 0.012615 2024-02-05T06:40:32.273 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:32.305 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:32.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:32.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:32.408 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:32.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:32 smithi028 ceph-mon[29442]: osdmap e210: 8 total, 8 up, 7 in 2024-02-05T06:40:32.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:32 smithi028 ceph-mon[37984]: osdmap e210: 8 total, 8 up, 7 in 2024-02-05T06:40:32.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:32.449 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:32.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:32.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:32 smithi167 ceph-mon[32774]: osdmap e210: 8 total, 8 up, 7 in 2024-02-05T06:40:32.550 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:32.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:32.588 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:32.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:32.689 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:32.717 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:32.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:32.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:32.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:32.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:32.846 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:32.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:32.946 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:32.948 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:40:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:40:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:40:32.981 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:40:32 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:40:32.987 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:32.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:32.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:33.089 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:33.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:33.129 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:33.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:33.229 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 38 16 47709 47693 78.426 53.1875 0.00776052 0.0127439 2024-02-05T06:40:33.230 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:33.270 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:33.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:33.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:33.372 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:33.399 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:33.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:33.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:33.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:33 smithi028 ceph-mon[29442]: osdmap e211: 8 total, 8 up, 7 in 2024-02-05T06:40:33.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:33 smithi028 ceph-mon[29442]: pgmap v323: 158 pgs: 1 peering, 4 active+recovery_wait+degraded, 5 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.9 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 8.2 KiB/s rd, 40 MiB/s wr, 644 op/s; 5528/140655 objects degraded (3.930%); 10812/140655 objects misplaced (7.687%); 47 MiB/s, 764 objects/s recovering 2024-02-05T06:40:33.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:33 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:40:33.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:33 smithi028 ceph-mon[37984]: osdmap e211: 8 total, 8 up, 7 in 2024-02-05T06:40:33.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:33 smithi028 ceph-mon[37984]: pgmap v323: 158 pgs: 1 peering, 4 active+recovery_wait+degraded, 5 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.9 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 8.2 KiB/s rd, 40 MiB/s wr, 644 op/s; 5528/140655 objects degraded (3.930%); 10812/140655 objects misplaced (7.687%); 47 MiB/s, 764 objects/s recovering 2024-02-05T06:40:33.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:33 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:40:33.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:33 smithi167 ceph-mon[32774]: osdmap e211: 8 total, 8 up, 7 in 2024-02-05T06:40:33.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:33 smithi167 ceph-mon[32774]: pgmap v323: 158 pgs: 1 peering, 4 active+recovery_wait+degraded, 5 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.9 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 8.2 KiB/s rd, 40 MiB/s wr, 644 op/s; 5528/140655 objects degraded (3.930%); 10812/140655 objects misplaced (7.687%); 47 MiB/s, 764 objects/s recovering 2024-02-05T06:40:33.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:33 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:40:33.501 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:33.534 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:33.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:33.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:33.636 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:33.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:33.667 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:33.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:33.768 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:33.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:33.794 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:33.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:33.896 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:33.921 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:33.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:33.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:34.022 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:34.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:34.050 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:34.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:34.151 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:34.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:34.189 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:34.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:34.229 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 39 16 47937 47921 76.7806 14.25 0.00548337 0.0129769 2024-02-05T06:40:34.290 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:34.320 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:34.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:34.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:34.421 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:34.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:34 smithi028 ceph-mon[29442]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 12 pgs inactive, 15 pgs peering) 2024-02-05T06:40:34.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:34 smithi028 ceph-mon[29442]: osdmap e212: 8 total, 8 up, 7 in 2024-02-05T06:40:34.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:34 smithi028 ceph-mon[37984]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 12 pgs inactive, 15 pgs peering) 2024-02-05T06:40:34.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:34 smithi028 ceph-mon[37984]: osdmap e212: 8 total, 8 up, 7 in 2024-02-05T06:40:34.450 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:34.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:34.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:34.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:34 smithi167 ceph-mon[32774]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 12 pgs inactive, 15 pgs peering) 2024-02-05T06:40:34.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:34 smithi167 ceph-mon[32774]: osdmap e212: 8 total, 8 up, 7 in 2024-02-05T06:40:34.552 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:34.598 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:34.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:34.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:34.700 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:34.728 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:34.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:34.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:34.830 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:34.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:34.864 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:34.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:34.965 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:35.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:35.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:35.105 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:35.135 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:35.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:35.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:35.229 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-02-05T06:40:35.225812+0000 min lat: 0.00125405 max lat: 2.2409 avg lat: 0.0131108 2024-02-05T06:40:35.230 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T06:40:35.230 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 40 16 48818 48802 76.2376 55.0625 0.00948064 0.0131108 2024-02-05T06:40:35.237 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:35.268 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:35.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:35.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:35.370 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:35.382 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:35 smithi167 ceph-mon[32774]: osdmap e213: 8 total, 8 up, 7 in 2024-02-05T06:40:35.382 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:35 smithi167 ceph-mon[32774]: pgmap v326: 158 pgs: 1 peering, 4 active+recovery_wait+degraded, 5 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.9 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 8.2 KiB/s rd, 40 MiB/s wr, 644 op/s; 5528/140655 objects degraded (3.930%); 10812/140655 objects misplaced (7.687%); 47 MiB/s, 764 objects/s recovering 2024-02-05T06:40:35.382 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:35 smithi167 ceph-mon[32774]: Health check update: Degraded data redundancy: 5528/140655 objects degraded (3.930%), 9 pgs degraded (PG_DEGRADED) 2024-02-05T06:40:35.382 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:35 smithi167 ceph-mon[32774]: osdmap e214: 8 total, 8 up, 7 in 2024-02-05T06:40:35.400 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:35.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:35.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:35.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:35 smithi028 ceph-mon[37984]: osdmap e213: 8 total, 8 up, 7 in 2024-02-05T06:40:35.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:35 smithi028 ceph-mon[37984]: pgmap v326: 158 pgs: 1 peering, 4 active+recovery_wait+degraded, 5 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.9 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 8.2 KiB/s rd, 40 MiB/s wr, 644 op/s; 5528/140655 objects degraded (3.930%); 10812/140655 objects misplaced (7.687%); 47 MiB/s, 764 objects/s recovering 2024-02-05T06:40:35.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:35 smithi028 ceph-mon[37984]: Health check update: Degraded data redundancy: 5528/140655 objects degraded (3.930%), 9 pgs degraded (PG_DEGRADED) 2024-02-05T06:40:35.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:35 smithi028 ceph-mon[37984]: osdmap e214: 8 total, 8 up, 7 in 2024-02-05T06:40:35.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:35 smithi028 ceph-mon[29442]: osdmap e213: 8 total, 8 up, 7 in 2024-02-05T06:40:35.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:35 smithi028 ceph-mon[29442]: pgmap v326: 158 pgs: 1 peering, 4 active+recovery_wait+degraded, 5 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 143 active+clean; 2.9 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 8.2 KiB/s rd, 40 MiB/s wr, 644 op/s; 5528/140655 objects degraded (3.930%); 10812/140655 objects misplaced (7.687%); 47 MiB/s, 764 objects/s recovering 2024-02-05T06:40:35.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:35 smithi028 ceph-mon[29442]: Health check update: Degraded data redundancy: 5528/140655 objects degraded (3.930%), 9 pgs degraded (PG_DEGRADED) 2024-02-05T06:40:35.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:35 smithi028 ceph-mon[29442]: osdmap e214: 8 total, 8 up, 7 in 2024-02-05T06:40:35.502 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:35.545 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:35.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:35.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:35.647 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:35.685 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:35.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:35.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:35.787 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:35.811 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:35.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:35.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:35.913 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:35.952 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:35.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:35.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:36.054 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:36.092 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:36.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:36.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:36.195 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:36.221 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:36.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:36.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:36.229 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 41 16 49634 49618 75.6219 51 0.00510148 0.013199 2024-02-05T06:40:36.323 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:36.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:36.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:36.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:36.510 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:36.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:36.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:36.612 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:36.643 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:36.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:36.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:36.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:40:36.745 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:36.774 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:36.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:36.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:36.876 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:36.908 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:36.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:36.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:37.010 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:37.036 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:37.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:37.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:37.138 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:37.163 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:37.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:37.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:37.230 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 42 16 50008 49992 74.378 23.375 0.00329022 0.0132235 2024-02-05T06:40:37.265 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:37.292 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:37.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:37.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:37.394 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:37.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:37.422 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:37.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:37.523 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:37.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:37.557 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:37.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:37.658 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:37.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:37 smithi028 ceph-mon[29442]: osdmap e215: 8 total, 8 up, 7 in 2024-02-05T06:40:37.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:37 smithi028 ceph-mon[29442]: pgmap v329: 158 pgs: 4 active+recovery_wait+degraded, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 145 active+clean; 2.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 12 MiB/s wr, 194 op/s; 5528/142971 objects degraded (3.867%); 10034/142971 objects misplaced (7.018%); 12 MiB/s, 194 objects/s recovering 2024-02-05T06:40:37.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:37 smithi028 ceph-mon[37984]: osdmap e215: 8 total, 8 up, 7 in 2024-02-05T06:40:37.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:37 smithi028 ceph-mon[37984]: pgmap v329: 158 pgs: 4 active+recovery_wait+degraded, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 145 active+clean; 2.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 12 MiB/s wr, 194 op/s; 5528/142971 objects degraded (3.867%); 10034/142971 objects misplaced (7.018%); 12 MiB/s, 194 objects/s recovering 2024-02-05T06:40:37.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:37.699 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:37.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:37.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:37 smithi167 ceph-mon[32774]: osdmap e215: 8 total, 8 up, 7 in 2024-02-05T06:40:37.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:37 smithi167 ceph-mon[32774]: pgmap v329: 158 pgs: 4 active+recovery_wait+degraded, 4 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 145 active+clean; 2.9 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 12 MiB/s wr, 194 op/s; 5528/142971 objects degraded (3.867%); 10034/142971 objects misplaced (7.018%); 12 MiB/s, 194 objects/s recovering 2024-02-05T06:40:37.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:37.839 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:37.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:37.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:37.942 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:37.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:37.992 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:37.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:38.093 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:38.120 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:38.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:38.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:38.223 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:38.230 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 43 16 50777 50761 73.766 48.0625 0.0285711 0.0135486 2024-02-05T06:40:38.250 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:38.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:38.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:38.352 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:38.398 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:38.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:38.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:38.500 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:38.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:38.536 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:38.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:38.637 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:38.668 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:38.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:38.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:38.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:38 smithi028 ceph-mon[37984]: osdmap e216: 8 total, 8 up, 7 in 2024-02-05T06:40:38.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:38 smithi028 ceph-mon[29442]: osdmap e216: 8 total, 8 up, 7 in 2024-02-05T06:40:38.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:38 smithi167 ceph-mon[32774]: osdmap e216: 8 total, 8 up, 7 in 2024-02-05T06:40:38.770 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:38.814 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:38.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:38.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:38.916 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:38.940 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:38.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:38.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:39.041 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:39.084 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:39.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:39.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:39.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:39.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:39.211 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:39.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:39.230 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 44 16 51597 51581 73.2541 51.25 0.0181911 0.0136322 2024-02-05T06:40:39.312 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:39.344 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:39.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:39.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:39.446 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:39.478 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:39.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:39.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:39.580 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:39.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:39 smithi028 ceph-mon[29442]: pgmap v331: 158 pgs: 1 peering, 4 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 145 active+clean; 3.0 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 47 MiB/s wr, 750 op/s; 5528/150027 objects degraded (3.685%); 8236/150027 objects misplaced (5.490%); 39 MiB/s, 617 objects/s recovering 2024-02-05T06:40:39.598 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:39 smithi028 ceph-mon[37984]: pgmap v331: 158 pgs: 1 peering, 4 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 145 active+clean; 3.0 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 47 MiB/s wr, 750 op/s; 5528/150027 objects degraded (3.685%); 8236/150027 objects misplaced (5.490%); 39 MiB/s, 617 objects/s recovering 2024-02-05T06:40:39.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:39.609 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:39.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:39.710 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:39.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:39 smithi167 ceph-mon[32774]: pgmap v331: 158 pgs: 1 peering, 4 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 145 active+clean; 3.0 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 47 MiB/s wr, 750 op/s; 5528/150027 objects degraded (3.685%); 8236/150027 objects misplaced (5.490%); 39 MiB/s, 617 objects/s recovering 2024-02-05T06:40:39.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:39.735 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:39.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:39.836 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:39.875 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:39.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:39.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:39.978 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:40.026 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:40.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:40.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:40.128 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:40.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:40.177 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:40.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:40.230 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 45 16 52295 52279 72.5957 43.625 0.0220518 0.0137679 2024-02-05T06:40:40.278 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:40.309 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:40.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:40.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:40.410 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:40.442 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:40.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:40.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:40.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:40.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:40.585 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:40.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:40.686 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:40.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:40 smithi028 ceph-mon[37984]: Health check update: Degraded data redundancy: 5528/150027 objects degraded (3.685%), 9 pgs degraded (PG_DEGRADED) 2024-02-05T06:40:40.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:40 smithi028 ceph-mon[29442]: Health check update: Degraded data redundancy: 5528/150027 objects degraded (3.685%), 9 pgs degraded (PG_DEGRADED) 2024-02-05T06:40:40.716 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:40.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:40.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:40.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:40 smithi167 ceph-mon[32774]: Health check update: Degraded data redundancy: 5528/150027 objects degraded (3.685%), 9 pgs degraded (PG_DEGRADED) 2024-02-05T06:40:40.818 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:40.846 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:40.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:40.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:40.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:40.977 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:40.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:40.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:41.079 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:41.108 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:41.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:41.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:41.210 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:41.230 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 46 16 53024 53008 72.008 45.5625 0.0166893 0.0138786 2024-02-05T06:40:41.239 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:41.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:41.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:41.341 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:41.365 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:41.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:41.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:41.467 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:41.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:41.513 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:41.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:41.614 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:41.645 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:41.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:41.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:41.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:41 smithi028 ceph-mon[37984]: pgmap v332: 158 pgs: 1 peering, 4 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 145 active+clean; 3.0 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 33 MiB/s wr, 521 op/s; 5528/150027 objects degraded (3.685%); 8236/150027 objects misplaced (5.490%); 27 MiB/s, 429 objects/s recovering 2024-02-05T06:40:41.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:41 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:40:41.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:41 smithi028 ceph-mon[29442]: pgmap v332: 158 pgs: 1 peering, 4 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 145 active+clean; 3.0 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 33 MiB/s wr, 521 op/s; 5528/150027 objects degraded (3.685%); 8236/150027 objects misplaced (5.490%); 27 MiB/s, 429 objects/s recovering 2024-02-05T06:40:41.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:41 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:40:41.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:41 smithi167 ceph-mon[32774]: pgmap v332: 158 pgs: 1 peering, 4 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 145 active+clean; 3.0 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 853 B/s rd, 33 MiB/s wr, 521 op/s; 5528/150027 objects degraded (3.685%); 8236/150027 objects misplaced (5.490%); 27 MiB/s, 429 objects/s recovering 2024-02-05T06:40:41.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:41 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:40:41.747 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:41.794 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:41.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:41.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:41.897 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:41.938 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:41.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:41.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:42.040 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:42.081 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:42.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:42.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:42.183 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:42.222 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:42.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:42.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:42.230 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 47 16 54001 53985 71.775 61.0625 0.00714637 0.0139252 2024-02-05T06:40:42.325 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:42.358 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:42.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:42.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:42.460 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:42.494 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:42.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:42.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:42.596 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:42.624 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:42.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:42.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:42.726 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:42.750 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:42.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:42.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:42.852 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:42.883 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:42.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:42.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:42.947 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:40:42 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:40:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:40:42.981 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:40:42 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:40:42.986 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:43.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:43.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:43.130 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:43.154 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:43.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:43.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:43.230 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 48 16 54805 54789 71.3265 50.25 0.0128328 0.0140132 2024-02-05T06:40:43.256 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:43.304 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:43.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:43.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:43.406 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:43.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:43.447 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:43.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:43.548 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:43.572 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:43.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:43.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:43.674 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:43.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:43 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:40:43.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:43 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:40:43.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:43 smithi028 ceph-mon[29442]: pgmap v333: 158 pgs: 4 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 146 active+clean; 3.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.4 KiB/s rd, 57 MiB/s wr, 911 op/s; 5528/160539 objects degraded (3.443%); 6381/160539 objects misplaced (3.975%); 38 MiB/s, 608 objects/s recovering 2024-02-05T06:40:43.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:43 smithi028 ceph-mon[29442]: osdmap e217: 8 total, 8 up, 7 in 2024-02-05T06:40:43.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:43 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:40:43.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:43 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:40:43.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:43 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:40:43.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:43 smithi028 ceph-mon[37984]: pgmap v333: 158 pgs: 4 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 146 active+clean; 3.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.4 KiB/s rd, 57 MiB/s wr, 911 op/s; 5528/160539 objects degraded (3.443%); 6381/160539 objects misplaced (3.975%); 38 MiB/s, 608 objects/s recovering 2024-02-05T06:40:43.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:43 smithi028 ceph-mon[37984]: osdmap e217: 8 total, 8 up, 7 in 2024-02-05T06:40:43.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:43 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:40:43.700 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:43.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:43.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:43.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:43 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:40:43.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:43 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:40:43.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:43 smithi167 ceph-mon[32774]: pgmap v333: 158 pgs: 4 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 146 active+clean; 3.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.4 KiB/s rd, 57 MiB/s wr, 911 op/s; 5528/160539 objects degraded (3.443%); 6381/160539 objects misplaced (3.975%); 38 MiB/s, 608 objects/s recovering 2024-02-05T06:40:43.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:43 smithi167 ceph-mon[32774]: osdmap e217: 8 total, 8 up, 7 in 2024-02-05T06:40:43.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:43 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:40:43.800 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-02-05T06:40:43.802 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:43.829 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:43.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:43.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:43.931 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:43.957 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:43.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:43.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:44.060 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:44.087 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:44.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:44.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:44.189 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:44.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:44.217 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:44.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:44.230 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 49 16 55140 55124 70.2982 20.9375 0.00301941 0.0140034 2024-02-05T06:40:44.318 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:44.351 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:44.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:44.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:44.452 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:44.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:44.484 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:44.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:44.585 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:44.616 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:44.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:44.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:44.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:44 smithi028 ceph-mon[29442]: osdmap e218: 8 total, 8 up, 7 in 2024-02-05T06:40:44.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:44 smithi028 ceph-mon[37984]: osdmap e218: 8 total, 8 up, 7 in 2024-02-05T06:40:44.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:44.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:44 smithi167 ceph-mon[32774]: osdmap e218: 8 total, 8 up, 7 in 2024-02-05T06:40:44.754 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:44.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:44.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:44.856 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:44.895 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:44.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:44.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:44.997 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:45.027 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:45.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:45.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:45.129 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:45.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:45.161 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:45.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:45.231 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 50 16 55913 55897 69.8585 48.3125 0.00565705 0.0143092 2024-02-05T06:40:45.262 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:45.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:45.287 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:45.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:45.388 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:45.418 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:45.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:45.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:45.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:45 smithi028 ceph-mon[37984]: pgmap v336: 158 pgs: 4 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 146 active+clean; 3.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 730 B/s rd, 31 MiB/s wr, 500 op/s; 5528/160539 objects degraded (3.443%); 6381/160539 objects misplaced (3.975%); 17 MiB/s, 264 objects/s recovering 2024-02-05T06:40:45.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:45 smithi028 ceph-mon[37984]: osdmap e219: 8 total, 8 up, 7 in 2024-02-05T06:40:45.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:45 smithi028 ceph-mon[37984]: Health check update: Degraded data redundancy: 5528/160539 objects degraded (3.443%), 9 pgs degraded (PG_DEGRADED) 2024-02-05T06:40:45.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:45 smithi028 ceph-mon[37984]: osdmap e220: 8 total, 8 up, 7 in 2024-02-05T06:40:45.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:45 smithi028 ceph-mon[29442]: pgmap v336: 158 pgs: 4 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 146 active+clean; 3.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 730 B/s rd, 31 MiB/s wr, 500 op/s; 5528/160539 objects degraded (3.443%); 6381/160539 objects misplaced (3.975%); 17 MiB/s, 264 objects/s recovering 2024-02-05T06:40:45.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:45 smithi028 ceph-mon[29442]: osdmap e219: 8 total, 8 up, 7 in 2024-02-05T06:40:45.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:45 smithi028 ceph-mon[29442]: Health check update: Degraded data redundancy: 5528/160539 objects degraded (3.443%), 9 pgs degraded (PG_DEGRADED) 2024-02-05T06:40:45.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:45 smithi028 ceph-mon[29442]: osdmap e220: 8 total, 8 up, 7 in 2024-02-05T06:40:45.520 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:45.546 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:45.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:45.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:45.648 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:45.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:45.674 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:45.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:45.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:45 smithi167 ceph-mon[32774]: pgmap v336: 158 pgs: 4 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 146 active+clean; 3.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 730 B/s rd, 31 MiB/s wr, 500 op/s; 5528/160539 objects degraded (3.443%); 6381/160539 objects misplaced (3.975%); 17 MiB/s, 264 objects/s recovering 2024-02-05T06:40:45.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:45 smithi167 ceph-mon[32774]: osdmap e219: 8 total, 8 up, 7 in 2024-02-05T06:40:45.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:45 smithi167 ceph-mon[32774]: Health check update: Degraded data redundancy: 5528/160539 objects degraded (3.443%), 9 pgs degraded (PG_DEGRADED) 2024-02-05T06:40:45.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:45 smithi167 ceph-mon[32774]: osdmap e220: 8 total, 8 up, 7 in 2024-02-05T06:40:45.775 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:45.804 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:45.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:45.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:45.906 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:45.937 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:45.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:45.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:46.039 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:46.063 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:46.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:46.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:46.166 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:46.196 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:46.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:46.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:46.231 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 51 16 56606 56590 69.3379 43.3125 0.0174639 0.0144155 2024-02-05T06:40:46.298 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:46.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:46.340 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:46.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:46.441 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:46.469 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:46.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:46.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:46.571 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:46.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:46.628 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:46.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:46.729 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:46.754 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:46.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:46.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:46.856 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:46.880 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:46.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:46.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:46.934 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:40:46.934 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-02-05T06:40:46.934 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-02-05T06:40:46.934 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-02-05T06:40:46.934 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:40:46.982 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:47.005 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:47.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:47.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:47.107 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:47.136 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:47.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:47.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:47.231 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 52 16 56964 56948 68.4348 22.375 0.00337523 0.0144183 2024-02-05T06:40:47.238 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:47.266 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:47.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:47.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:47.368 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:47.401 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:47.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:47.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:47.403 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:40:47.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:47 smithi167 ceph-mon[32774]: pgmap v339: 158 pgs: 4 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 1 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 146 active+clean; 3.3 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 15 MiB/s wr, 233 op/s; 5528/163329 objects degraded (3.385%); 5240/163329 objects misplaced (3.208%); 18 MiB/s, 285 objects/s recovering 2024-02-05T06:40:47.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:47 smithi167 ceph-mon[32774]: osdmap e221: 8 total, 8 up, 7 in 2024-02-05T06:40:47.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:47 smithi167 ceph-mon[32774]: 3.14 scrub starts 2024-02-05T06:40:47.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:47 smithi167 ceph-mon[32774]: 3.14 scrub ok 2024-02-05T06:40:47.503 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:47.532 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:47.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:47.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:47.634 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:47.677 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:47.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:47.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:47.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:47 smithi028 ceph-mon[37984]: pgmap v339: 158 pgs: 4 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 1 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 146 active+clean; 3.3 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 15 MiB/s wr, 233 op/s; 5528/163329 objects degraded (3.385%); 5240/163329 objects misplaced (3.208%); 18 MiB/s, 285 objects/s recovering 2024-02-05T06:40:47.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:47 smithi028 ceph-mon[37984]: osdmap e221: 8 total, 8 up, 7 in 2024-02-05T06:40:47.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:47 smithi028 ceph-mon[37984]: 3.14 scrub starts 2024-02-05T06:40:47.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:47 smithi028 ceph-mon[37984]: 3.14 scrub ok 2024-02-05T06:40:47.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:47 smithi028 ceph-mon[29442]: pgmap v339: 158 pgs: 4 active+recovery_wait+degraded, 2 active+recovering+undersized+remapped, 1 active+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 active+recovery_wait+undersized+degraded+remapped, 146 active+clean; 3.3 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 15 MiB/s wr, 233 op/s; 5528/163329 objects degraded (3.385%); 5240/163329 objects misplaced (3.208%); 18 MiB/s, 285 objects/s recovering 2024-02-05T06:40:47.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:47 smithi028 ceph-mon[29442]: osdmap e221: 8 total, 8 up, 7 in 2024-02-05T06:40:47.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:47 smithi028 ceph-mon[29442]: 3.14 scrub starts 2024-02-05T06:40:47.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:47 smithi028 ceph-mon[29442]: 3.14 scrub ok 2024-02-05T06:40:47.779 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:47.806 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:47.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:47.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:47.907 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:47.950 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:47.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:47.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:48.005 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-05T06:40:48.052 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:48.081 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:48.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:48.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:48.183 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:48.211 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:48.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:48.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:48.231 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 53 16 57774 57758 68.0988 50.625 0.0191736 0.0146773 2024-02-05T06:40:48.313 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:48.365 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:48.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:48.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:48.467 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:48.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:48.514 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:48.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:48.615 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:48.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:48.657 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:48.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:48.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:48 smithi028 ceph-mon[29442]: osdmap e222: 8 total, 8 up, 7 in 2024-02-05T06:40:48.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:48 smithi028 ceph-mon[29442]: 4.1 scrub starts 2024-02-05T06:40:48.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:48 smithi028 ceph-mon[29442]: 4.1 scrub ok 2024-02-05T06:40:48.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:48 smithi028 ceph-mon[37984]: osdmap e222: 8 total, 8 up, 7 in 2024-02-05T06:40:48.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:48 smithi028 ceph-mon[37984]: 4.1 scrub starts 2024-02-05T06:40:48.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:48 smithi028 ceph-mon[37984]: 4.1 scrub ok 2024-02-05T06:40:48.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:48 smithi167 ceph-mon[32774]: osdmap e222: 8 total, 8 up, 7 in 2024-02-05T06:40:48.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:48 smithi167 ceph-mon[32774]: 4.1 scrub starts 2024-02-05T06:40:48.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:48 smithi167 ceph-mon[32774]: 4.1 scrub ok 2024-02-05T06:40:48.745 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:40:48.759 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:48.785 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:48.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:48.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:48.801 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 4, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T06:40:48.801 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T06:40:48.801 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_1 2024-02-05T06:40:48.801 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_1 by 10 min 8 2024-02-05T06:40:48.802 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T06:40:48.888 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:48.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:48.916 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:48.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:49.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:49.067 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:49.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:49.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:49.169 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:49.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:49.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:49.232 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 54 16 58648 58632 67.8492 54.625 0.0175857 0.0147319 2024-02-05T06:40:49.324 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:49.390 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:49.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:49.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:49.492 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:49.509 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:49 smithi028 ceph-mon[37984]: pgmap v342: 158 pgs: 1 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 150 active+clean; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 873 op/s; 2775/171006 objects degraded (1.623%); 6915/171006 objects misplaced (4.044%); 34 MiB/s, 554 objects/s recovering 2024-02-05T06:40:49.510 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:49 smithi028 ceph-mon[37984]: osdmap e223: 8 total, 8 up, 7 in 2024-02-05T06:40:49.510 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:49 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:40:49.510 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:49 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:40:49.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:49 smithi028 ceph-mon[29442]: pgmap v342: 158 pgs: 1 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 150 active+clean; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 873 op/s; 2775/171006 objects degraded (1.623%); 6915/171006 objects misplaced (4.044%); 34 MiB/s, 554 objects/s recovering 2024-02-05T06:40:49.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:49 smithi028 ceph-mon[29442]: osdmap e223: 8 total, 8 up, 7 in 2024-02-05T06:40:49.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:49 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:40:49.510 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:49 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:40:49.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:49.562 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:49.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:49.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:49.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:49.713 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:49.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:49.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:49 smithi167 ceph-mon[32774]: pgmap v342: 158 pgs: 1 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 150 active+clean; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 873 op/s; 2775/171006 objects degraded (1.623%); 6915/171006 objects misplaced (4.044%); 34 MiB/s, 554 objects/s recovering 2024-02-05T06:40:49.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:49 smithi167 ceph-mon[32774]: osdmap e223: 8 total, 8 up, 7 in 2024-02-05T06:40:49.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:49 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:40:49.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:49 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:40:49.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:49.859 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:49.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:49.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:49.961 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:49.990 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:49.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:49.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:50.093 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:50.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:50.121 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:50.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:50.223 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:50.231 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 55 16 59493 59477 67.5757 52.8125 0.0200171 0.0147898 2024-02-05T06:40:50.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:50.270 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:50.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:50.370 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:50.397 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:50.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:50.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:50.453 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:40:50.499 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:50.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:50.526 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:50.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:50.627 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:50.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:50.678 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:50.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:50.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:50 smithi028 ceph-mon[37984]: osdmap e224: 8 total, 8 up, 7 in 2024-02-05T06:40:50.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:50 smithi028 ceph-mon[37984]: Health check update: Degraded data redundancy: 2775/171006 objects degraded (1.623%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T06:40:50.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:50 smithi028 ceph-mon[29442]: osdmap e224: 8 total, 8 up, 7 in 2024-02-05T06:40:50.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:50 smithi028 ceph-mon[29442]: Health check update: Degraded data redundancy: 2775/171006 objects degraded (1.623%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T06:40:50.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:50 smithi167 ceph-mon[32774]: osdmap e224: 8 total, 8 up, 7 in 2024-02-05T06:40:50.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:50 smithi167 ceph-mon[32774]: Health check update: Degraded data redundancy: 2775/171006 objects degraded (1.623%), 3 pgs degraded (PG_DEGRADED) 2024-02-05T06:40:50.779 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:50.819 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:50.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:50.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:50.921 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:50.960 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:50.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:50.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:51.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:51.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:51.122 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:51.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:51.130 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:40:51.223 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:51.234 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 56 16 60367 60351 67.3435 54.625 0.0345625 0.014842 2024-02-05T06:40:51.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:51.279 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:51.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:51.381 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-02-05T06:40:51.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:51.428 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:51.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:51.532 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:51.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:51.557 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:51.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:51.658 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:51.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:51 smithi028 ceph-mon[37984]: pgmap v345: 158 pgs: 1 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 150 active+clean; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 40 MiB/s wr, 639 op/s; 2775/171006 objects degraded (1.623%); 6915/171006 objects misplaced (4.044%); 17 MiB/s, 269 objects/s recovering 2024-02-05T06:40:51.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:51 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2466580324' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:40:51.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:51 smithi028 ceph-mon[37984]: from='client.45068 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:40:51.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:51 smithi028 ceph-mon[29442]: pgmap v345: 158 pgs: 1 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 150 active+clean; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 40 MiB/s wr, 639 op/s; 2775/171006 objects degraded (1.623%); 6915/171006 objects misplaced (4.044%); 17 MiB/s, 269 objects/s recovering 2024-02-05T06:40:51.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:51 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2466580324' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:40:51.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:51 smithi028 ceph-mon[29442]: from='client.45068 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:40:51.701 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:51.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:51.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:51.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:51 smithi167 ceph-mon[32774]: pgmap v345: 158 pgs: 1 active+recovery_wait+degraded, 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 150 active+clean; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 40 MiB/s wr, 639 op/s; 2775/171006 objects degraded (1.623%); 6915/171006 objects misplaced (4.044%); 17 MiB/s, 269 objects/s recovering 2024-02-05T06:40:51.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:51 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2466580324' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:40:51.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:51 smithi167 ceph-mon[32774]: from='client.45068 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:40:51.803 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:51.846 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:51.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:51.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:51.948 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:51.995 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:51.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:51.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:52.097 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:52.124 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:52.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:52.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:52.226 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:52.233 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 57 16 61164 61148 67.0348 49.8125 0.0127695 0.0149099 2024-02-05T06:40:52.269 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:52.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:52.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:52.371 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:52.399 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:52.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:52.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:52.501 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:52.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:52.545 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:52.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:52.646 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:52.669 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:52.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:52.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:52.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:52 smithi028 ceph-mon[37984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:40:52.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:52 smithi028 ceph-mon[37984]: from='client.45068 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:40:52.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:52 smithi028 ceph-mon[37984]: osdmap e225: 8 total, 8 up, 7 in 2024-02-05T06:40:52.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:52 smithi028 ceph-mon[29442]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:40:52.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:52 smithi028 ceph-mon[29442]: from='client.45068 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:40:52.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:52 smithi028 ceph-mon[29442]: osdmap e225: 8 total, 8 up, 7 in 2024-02-05T06:40:52.698 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:40:52 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:40:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:40:52.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:52 smithi167 ceph-mon[32774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:40:52.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:52 smithi167 ceph-mon[32774]: from='client.45068 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:40:52.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:52 smithi167 ceph-mon[32774]: osdmap e225: 8 total, 8 up, 7 in 2024-02-05T06:40:52.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:52.819 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:52.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:52.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:52.921 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:52.968 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:52.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:52.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:53.070 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:53.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:53.119 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:53.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:53.220 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:53.231 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:40:52 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:40:53.233 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 58 16 61974 61958 66.7517 50.625 0.00356507 0.0149562 2024-02-05T06:40:53.244 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:53.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:53.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:53.346 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:53.399 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:53.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:53.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:53.501 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:53.537 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:53.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:53.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:53.639 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:53.665 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:53.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:53.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:53.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:53 smithi028 ceph-mon[37984]: pgmap v347: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 153 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.0 KiB/s rd, 48 MiB/s wr, 764 op/s; 2774/182355 objects degraded (1.521%); 5652/182355 objects misplaced (3.099%); 16 MiB/s, 255 objects/s recovering 2024-02-05T06:40:53.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:53 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:40:53.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:53 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3256930882' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:40:53.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:53 smithi028 ceph-mon[29442]: pgmap v347: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 153 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.0 KiB/s rd, 48 MiB/s wr, 764 op/s; 2774/182355 objects degraded (1.521%); 5652/182355 objects misplaced (3.099%); 16 MiB/s, 255 objects/s recovering 2024-02-05T06:40:53.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:53 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:40:53.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:53 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3256930882' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:40:53.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:53 smithi167 ceph-mon[32774]: pgmap v347: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 153 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.0 KiB/s rd, 48 MiB/s wr, 764 op/s; 2774/182355 objects degraded (1.521%); 5652/182355 objects misplaced (3.099%); 16 MiB/s, 255 objects/s recovering 2024-02-05T06:40:53.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:53 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:40:53.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:53 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3256930882' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:40:53.767 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:53.797 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:53.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:53.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:53.873 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:40:53.873 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:40:53.873 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:40:53.873 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:40:53.873 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:40:53.874 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:40:53.874 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:40:53.874 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:40:53.874 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:40:53.874 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:40:53.874 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:40:53.874 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:40:53.874 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:40:53.874 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:40:53.874 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:40:53.874 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:40:53.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:53.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:53.940 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:53.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:54.041 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:54.065 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:54.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:54.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:54.167 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:54.197 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:54.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:54.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:54.233 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 59 16 62877 62861 66.5769 56.4375 0.00271761 0.0150146 2024-02-05T06:40:54.299 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:54.328 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:54.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:54.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:54.430 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:54.458 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:54.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:54.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:54.560 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:54.605 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:54.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:54.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:54.706 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:54.749 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:54.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:54.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:54.851 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:54.878 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:54.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:54.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:54.980 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:55.006 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:55.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:55.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:55.108 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:55.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:55.157 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:55.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:55.233 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-02-05T06:40:55.229458+0000 min lat: 0.00125405 max lat: 2.2409 avg lat: 0.0150363 2024-02-05T06:40:55.233 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T06:40:55.234 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 60 16 63847 63831 66.4776 60.625 0.0208575 0.0150363 2024-02-05T06:40:55.258 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:55.287 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:55.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:55.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:55.388 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:55.419 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:55.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:55.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:55.521 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:55.564 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:55.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:55.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:55.666 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:55.695 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:55.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:55.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:55.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:55 smithi028 ceph-mon[29442]: pgmap v348: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 153 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 39 MiB/s wr, 631 op/s; 2774/182355 objects degraded (1.521%); 5652/182355 objects misplaced (3.099%); 13 MiB/s, 210 objects/s recovering 2024-02-05T06:40:55.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:55 smithi028 ceph-mon[29442]: Health check update: Degraded data redundancy: 2774/182355 objects degraded (1.521%), 2 pgs degraded (PG_DEGRADED) 2024-02-05T06:40:55.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:55 smithi028 ceph-mon[29442]: osdmap e226: 8 total, 8 up, 7 in 2024-02-05T06:40:55.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:55 smithi028 ceph-mon[37984]: pgmap v348: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 153 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 39 MiB/s wr, 631 op/s; 2774/182355 objects degraded (1.521%); 5652/182355 objects misplaced (3.099%); 13 MiB/s, 210 objects/s recovering 2024-02-05T06:40:55.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:55 smithi028 ceph-mon[37984]: Health check update: Degraded data redundancy: 2774/182355 objects degraded (1.521%), 2 pgs degraded (PG_DEGRADED) 2024-02-05T06:40:55.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:55 smithi028 ceph-mon[37984]: osdmap e226: 8 total, 8 up, 7 in 2024-02-05T06:40:55.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:55 smithi167 ceph-mon[32774]: pgmap v348: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 153 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 39 MiB/s wr, 631 op/s; 2774/182355 objects degraded (1.521%); 5652/182355 objects misplaced (3.099%); 13 MiB/s, 210 objects/s recovering 2024-02-05T06:40:55.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:55 smithi167 ceph-mon[32774]: Health check update: Degraded data redundancy: 2774/182355 objects degraded (1.521%), 2 pgs degraded (PG_DEGRADED) 2024-02-05T06:40:55.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:55 smithi167 ceph-mon[32774]: osdmap e226: 8 total, 8 up, 7 in 2024-02-05T06:40:55.797 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:55.833 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:55.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:55.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:55.936 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:55.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:55.981 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:55.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:56.082 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:56.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:56.127 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:56.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:56.228 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:56.233 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 61 16 64823 64807 66.3877 61 0.00375558 0.0150393 2024-02-05T06:40:56.258 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:56.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:56.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:56.360 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:56.385 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:56.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:56.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:56.486 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:56.513 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:56.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:56.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:56.615 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:56.644 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:56.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:56.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:56.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:40:56.773 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:56.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:56.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:40:56.875 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:40:56.881 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:40:56.903 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:56.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:56.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:40:57.006 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:57.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:57.031 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:57.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:57.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:57.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:57.185 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:57.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:57.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:57 smithi028 ceph-mon[29442]: osdmap e227: 8 total, 8 up, 7 in 2024-02-05T06:40:57.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:57 smithi028 ceph-mon[29442]: pgmap v351: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 153 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.7 KiB/s rd, 51 MiB/s wr, 816 op/s; 2774/185679 objects degraded (1.494%); 5652/185679 objects misplaced (3.044%); 13 MiB/s, 210 objects/s recovering 2024-02-05T06:40:57.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:57 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:40:57.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:57 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:40:57.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:57 smithi028 ceph-mon[37984]: osdmap e227: 8 total, 8 up, 7 in 2024-02-05T06:40:57.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:57 smithi028 ceph-mon[37984]: pgmap v351: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 153 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.7 KiB/s rd, 51 MiB/s wr, 816 op/s; 2774/185679 objects degraded (1.494%); 5652/185679 objects misplaced (3.044%); 13 MiB/s, 210 objects/s recovering 2024-02-05T06:40:57.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:57 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:40:57.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:57 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:40:57.235 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 62 16 64998 64982 65.4917 10.9375 0.00443821 0.0152635 2024-02-05T06:40:57.287 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:57.310 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:57.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:57.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:57.412 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:40:57.456 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:57.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:57.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:40:57.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:57 smithi167 ceph-mon[32774]: osdmap e227: 8 total, 8 up, 7 in 2024-02-05T06:40:57.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:57 smithi167 ceph-mon[32774]: pgmap v351: 158 pgs: 3 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 153 active+clean; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.7 KiB/s rd, 51 MiB/s wr, 816 op/s; 2774/185679 objects degraded (1.494%); 5652/185679 objects misplaced (3.044%); 13 MiB/s, 210 objects/s recovering 2024-02-05T06:40:57.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:57 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:40:57.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:57 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:40:57.557 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:57.589 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:57.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:57.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:57.691 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:57.721 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:57.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:57.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:57.823 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:57.870 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T06:40:57.888 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:57.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:57.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:57.990 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:58.019 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:58.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:58.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:58.121 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:40:58.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:58.150 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:58.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:40:58.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:58 smithi028 ceph-mon[29442]: osdmap e228: 8 total, 8 up, 7 in 2024-02-05T06:40:58.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:58 smithi028 ceph-mon[37984]: osdmap e228: 8 total, 8 up, 7 in 2024-02-05T06:40:58.235 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 63 16 65963 65947 65.4094 60.3125 0.0142056 0.015283 2024-02-05T06:40:58.252 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:58.280 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:58.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:58.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:58.381 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:58.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:58.451 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:58.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:58.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:58 smithi167 ceph-mon[32774]: osdmap e228: 8 total, 8 up, 7 in 2024-02-05T06:40:58.552 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:58.582 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:58.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:58.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:58.684 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:58.709 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:58.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:58.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:58.806 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:40:58.806 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:40:58.811 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:58.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:58.837 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:58.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:58.939 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:58.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:58.968 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:58.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:59.069 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:40:59.098 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:59.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:59.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:40:59.200 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:59.231 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:59.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:59.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:59.235 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 64 16 66859 66843 65.2623 56 0.0054877 0.0153058 2024-02-05T06:40:59.334 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:59.358 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:59.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:59.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:59.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:40:59 smithi028 ceph-mon[29442]: pgmap v353: 158 pgs: 1 peering, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 153 active+clean; 4.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 44 MiB/s wr, 712 op/s; 2774/195165 objects degraded (1.421%); 4396/195165 objects misplaced (2.252%); 13 MiB/s, 209 objects/s recovering 2024-02-05T06:40:59.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:40:59 smithi028 ceph-mon[37984]: pgmap v353: 158 pgs: 1 peering, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 153 active+clean; 4.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 44 MiB/s wr, 712 op/s; 2774/195165 objects degraded (1.421%); 4396/195165 objects misplaced (2.252%); 13 MiB/s, 209 objects/s recovering 2024-02-05T06:40:59.459 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:59.481 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:40:59 smithi167 ceph-mon[32774]: pgmap v353: 158 pgs: 1 peering, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 153 active+clean; 4.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 44 MiB/s wr, 712 op/s; 2774/195165 objects degraded (1.421%); 4396/195165 objects misplaced (2.252%); 13 MiB/s, 209 objects/s recovering 2024-02-05T06:40:59.486 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:59.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:59.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:59.588 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:40:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:40:59.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:59.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:40:59.682 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-05T06:40:59.715 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:40:59.750 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:59.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:59.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:40:59.851 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:40:59.878 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:40:59.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:40:59.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:40:59.941 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:40:59.946 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:40:59.946 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:40:59.980 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:00.004 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:00.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:00.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:00.106 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:00.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:00.150 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:00.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:00.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:00 smithi028 ceph-mon[29442]: osdmap e229: 8 total, 8 up, 7 in 2024-02-05T06:41:00.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:00 smithi028 ceph-mon[29442]: Health check update: Degraded data redundancy: 2774/195165 objects degraded (1.421%), 2 pgs degraded (PG_DEGRADED) 2024-02-05T06:41:00.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:00 smithi028 ceph-mon[29442]: osdmap e230: 8 total, 8 up, 7 in 2024-02-05T06:41:00.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:00 smithi028 ceph-mon[37984]: osdmap e229: 8 total, 8 up, 7 in 2024-02-05T06:41:00.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:00 smithi028 ceph-mon[37984]: Health check update: Degraded data redundancy: 2774/195165 objects degraded (1.421%), 2 pgs degraded (PG_DEGRADED) 2024-02-05T06:41:00.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:00 smithi028 ceph-mon[37984]: osdmap e230: 8 total, 8 up, 7 in 2024-02-05T06:41:00.235 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 65 16 67059 67043 64.4507 12.5 0.02891 0.0155091 2024-02-05T06:41:00.251 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:00.280 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:00.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:00.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:00.377 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:41:00.382 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:00.383 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-05T06:41:00.413 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:00 smithi167 ceph-mon[32774]: osdmap e229: 8 total, 8 up, 7 in 2024-02-05T06:41:00.413 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:00 smithi167 ceph-mon[32774]: Health check update: Degraded data redundancy: 2774/195165 objects degraded (1.421%), 2 pgs degraded (PG_DEGRADED) 2024-02-05T06:41:00.413 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:00 smithi167 ceph-mon[32774]: osdmap e230: 8 total, 8 up, 7 in 2024-02-05T06:41:00.425 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:00.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:00.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:00.527 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:00.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:00.572 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:00.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:00.673 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:00.715 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:00.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:00.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:00.818 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:00.863 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:00.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:00.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:00.965 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:00.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:00.997 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:00.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:01.099 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:01.127 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:01.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:01.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:01.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:01 smithi028 ceph-mon[29442]: pgmap v356: 158 pgs: 1 peering, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 153 active+clean; 4.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.00k op/s; 2774/195165 objects degraded (1.421%); 4396/195165 objects misplaced (2.252%); 18 MiB/s, 295 objects/s recovering 2024-02-05T06:41:01.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:01 smithi028 ceph-mon[29442]: from='client.25102 v1:172.21.15.28:0/2454904326' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:41:01.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:01 smithi028 ceph-mon[37984]: pgmap v356: 158 pgs: 1 peering, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 153 active+clean; 4.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.00k op/s; 2774/195165 objects degraded (1.421%); 4396/195165 objects misplaced (2.252%); 18 MiB/s, 295 objects/s recovering 2024-02-05T06:41:01.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:01 smithi028 ceph-mon[37984]: from='client.25102 v1:172.21.15.28:0/2454904326' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:41:01.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:01.236 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 66 16 68037 68021 64.4002 61.125 0.0310579 0.0155205 2024-02-05T06:41:01.274 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:01.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:01.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:01.376 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:01.405 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:01.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:01.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:01.466 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:01 smithi167 ceph-mon[32774]: pgmap v356: 158 pgs: 1 peering, 2 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 153 active+clean; 4.0 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.00k op/s; 2774/195165 objects degraded (1.421%); 4396/195165 objects misplaced (2.252%); 18 MiB/s, 295 objects/s recovering 2024-02-05T06:41:01.466 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:01 smithi167 ceph-mon[32774]: from='client.25102 v1:172.21.15.28:0/2454904326' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:41:01.508 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:01.537 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:01.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:01.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:01.638 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:01.681 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:01.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:01.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:01.783 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:01.826 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:01.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:01.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:01.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:01.956 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:41:01.956 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:41:01.973 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:01.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:01.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:02.075 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:02.108 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:02.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:02.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:02.210 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:02.236 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 67 16 69034 69018 64.3688 62.3125 0.00683969 0.0155283 2024-02-05T06:41:02.256 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:02.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:02.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:02.346 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:02 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:41:02.346 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:02 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:41:02.359 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:02.389 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:02.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:02.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:02.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:02 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:41:02.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:02 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:41:02.433 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:02 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:41:02.433 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:02 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:41:02.487 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2024-02-05T06:41:00.291944+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4251016659,"num_objects":65055,"num_object_clones":0,"num_object_copies":195165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2774,"num_objects_misplaced":4396,"num_objects_unfound":0,"num_objects_dirty":65055,"num_whiteouts":0,"num_read":807,"num_read_kb":594,"num_write":65358,"num_write_kb":4151770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20791,"num_bytes_recovered":1339818460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66422,"ondisk_log_size":66422,"up":474,"acting":470,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":478,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":14098452,"kb_used_data":12416960,"kb_used_omap":10,"kb_used_meta":1681397,"kb_avail":642117612,"statfs":{"total":671965249536,"available":657528434688,"internally_reserved":0,"allocated":12714967040,"data_stored":12708917876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":1721750635},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":279838720,"num_objects":4270,"num_object_clones":0,"num_object_copies":12810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1256,"num_objects_unfound":0,"num_objects_dirty":4270,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4270,"num_write_kb":273280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1256,"num_bytes_recovered":82313216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.249695"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":240,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.041777+0000","last_change":"2024-02-05T06:40:27.072854+0000","last_active":"2024-02-05T06:40:56.041777+0000","last_peered":"2024-02-05T06:40:56.041777+0000","last_clean":"2024-02-05T06:40:56.041777+0000","last_became_active":"2024-02-05T06:40:27.072478+0000","last_became_peered":"2024-02-05T06:40:27.072478+0000","last_unstale":"2024-02-05T06:40:56.041777+0000","last_undegraded":"2024-02-05T06:40:56.041777+0000","last_fullsized":"2024-02-05T06:40:56.041777+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:32.017855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:32.017855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:40:56.561156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":369,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.056436+0000","last_change":"2024-02-05T06:40:27.087768+0000","last_active":"2024-02-05T06:40:56.056436+0000","last_peered":"2024-02-05T06:40:56.056436+0000","last_clean":"2024-02-05T06:40:56.056436+0000","last_became_active":"2024-02-05T06:40:27.087211+0000","last_became_peered":"2024-02-05T06:40:27.087211+0000","last_unstale":"2024-02-05T06:40:56.056436+0000","last_undegraded":"2024-02-05T06:40:56.056436+0000","last_fullsized":"2024-02-05T06:40:56.056436+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:15.357465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:06.525012+0000","last_clean_scrub_stamp":"2024-02-05T06:39:15.357465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:00.318016+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":358,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.051098+0000","last_change":"2024-02-05T06:40:08.601821+0000","last_active":"2024-02-05T06:40:56.051098+0000","last_peered":"2024-02-05T06:40:56.051098+0000","last_clean":"2024-02-05T06:40:56.051098+0000","last_became_active":"2024-02-05T06:40:08.598443+0000","last_became_peered":"2024-02-05T06:40:08.598443+0000","last_unstale":"2024-02-05T06:40:56.051098+0000","last_undegraded":"2024-02-05T06:40:56.051098+0000","last_fullsized":"2024-02-05T06:40:56.051098+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:28.841777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:39:28.841777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":246,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.038121+0000","last_change":"2024-02-05T06:39:15.027055+0000","last_active":"2024-02-05T06:40:56.038121+0000","last_peered":"2024-02-05T06:40:56.038121+0000","last_clean":"2024-02-05T06:40:56.038121+0000","last_became_active":"2024-02-05T06:36:54.959330+0000","last_became_peered":"2024-02-05T06:36:54.959330+0000","last_unstale":"2024-02-05T06:40:56.038121+0000","last_undegraded":"2024-02-05T06:40:56.038121+0000","last_fullsized":"2024-02-05T06:40:56.038121+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:39:15.026945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:15.026945+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:40:56.561376+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":7,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":350,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.060475+0000","last_change":"2024-02-05T06:40:27.077900+0000","last_active":"2024-02-05T06:40:56.060475+0000","last_peered":"2024-02-05T06:40:56.060475+0000","last_clean":"2024-02-05T06:40:56.060475+0000","last_became_active":"2024-02-05T06:40:27.077485+0000","last_became_peered":"2024-02-05T06:40:27.077485+0000","last_unstale":"2024-02-05T06:40:56.060475+0000","last_undegraded":"2024-02-05T06:40:56.060475+0000","last_fullsized":"2024-02-05T06:40:56.060475+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:21.419504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:37:21.419504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":236,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.038235+0000","last_change":"2024-02-05T06:39:11.961722+0000","last_active":"2024-02-05T06:40:56.038235+0000","last_peered":"2024-02-05T06:40:56.038235+0000","last_clean":"2024-02-05T06:40:56.038235+0000","last_became_active":"2024-02-05T06:36:54.959640+0000","last_became_peered":"2024-02-05T06:36:54.959640+0000","last_unstale":"2024-02-05T06:40:56.038235+0000","last_undegraded":"2024-02-05T06:40:56.038235+0000","last_fullsized":"2024-02-05T06:40:56.038235+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:11.961635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:11.961635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:40:56.561392+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":359,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.057039+0000","last_change":"2024-02-05T06:39:34.395050+0000","last_active":"2024-02-05T06:40:56.057039+0000","last_peered":"2024-02-05T06:40:56.057039+0000","last_clean":"2024-02-05T06:40:56.057039+0000","last_became_active":"2024-02-05T06:33:29.553864+0000","last_became_peered":"2024-02-05T06:33:29.553864+0000","last_unstale":"2024-02-05T06:40:56.057039+0000","last_undegraded":"2024-02-05T06:40:56.057039+0000","last_fullsized":"2024-02-05T06:40:56.057039+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:34.394974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:39:34.394974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"88'22","reported_seq":387,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.052311+0000","last_change":"2024-02-05T06:39:35.433638+0000","last_active":"2024-02-05T06:40:56.052311+0000","last_peered":"2024-02-05T06:40:56.052311+0000","last_clean":"2024-02-05T06:40:56.052311+0000","last_became_active":"2024-02-05T06:33:31.572544+0000","last_became_peered":"2024-02-05T06:33:31.572544+0000","last_unstale":"2024-02-05T06:40:56.052311+0000","last_undegraded":"2024-02-05T06:40:56.052311+0000","last_fullsized":"2024-02-05T06:40:56.052311+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'22","last_scrub_stamp":"2024-02-05T06:39:35.433539+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:39:35.433539+0000","last_clean_scrub_stamp":"2024-02-05T06:39:35.433539+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":352,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.048578+0000","last_change":"2024-02-05T06:40:27.069177+0000","last_active":"2024-02-05T06:40:56.048578+0000","last_peered":"2024-02-05T06:40:56.048578+0000","last_clean":"2024-02-05T06:40:56.048578+0000","last_became_active":"2024-02-05T06:40:27.068888+0000","last_became_peered":"2024-02-05T06:40:27.068888+0000","last_unstale":"2024-02-05T06:40:56.048578+0000","last_undegraded":"2024-02-05T06:40:56.048578+0000","last_fullsized":"2024-02-05T06:40:56.048578+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:49.406914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:37:49.406914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":275,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.048659+0000","last_change":"2024-02-05T06:40:27.066310+0000","last_active":"2024-02-05T06:40:56.048659+0000","last_peered":"2024-02-05T06:40:56.048659+0000","last_clean":"2024-02-05T06:40:56.048659+0000","last_became_active":"2024-02-05T06:40:27.065918+0000","last_became_peered":"2024-02-05T06:40:27.065918+0000","last_unstale":"2024-02-05T06:40:56.048659+0000","last_undegraded":"2024-02-05T06:40:56.048659+0000","last_fullsized":"2024-02-05T06:40:56.048659+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:38:15.240524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:38:15.240524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":209,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.039108+0000","last_change":"2024-02-05T06:40:08.579505+0000","last_active":"2024-02-05T06:40:56.039108+0000","last_peered":"2024-02-05T06:40:56.039108+0000","last_clean":"2024-02-05T06:40:56.039108+0000","last_became_active":"2024-02-05T06:40:08.579165+0000","last_became_peered":"2024-02-05T06:40:08.579165+0000","last_unstale":"2024-02-05T06:40:56.039108+0000","last_undegraded":"2024-02-05T06:40:56.039108+0000","last_fullsized":"2024-02-05T06:40:56.039108+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:08.476236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:44.081318+0000","last_clean_scrub_stamp":"2024-02-05T06:39:08.476236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:40:56.561305+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"92'13","reported_seq":246,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.041677+0000","last_change":"2024-02-05T06:38:14.064011+0000","last_active":"2024-02-05T06:40:56.041677+0000","last_peered":"2024-02-05T06:40:56.041677+0000","last_clean":"2024-02-05T06:40:56.041677+0000","last_became_active":"2024-02-05T06:36:54.969454+0000","last_became_peered":"2024-02-05T06:36:54.969454+0000","last_unstale":"2024-02-05T06:40:56.041677+0000","last_undegraded":"2024-02-05T06:40:56.041677+0000","last_fullsized":"2024-02-05T06:40:56.041677+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'13","last_scrub_stamp":"2024-02-05T06:38:14.063877+0000","last_deep_scrub":"92'13","last_deep_scrub_stamp":"2024-02-05T06:34:46.542893+0000","last_clean_scrub_stamp":"2024-02-05T06:38:14.063877+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:40:56.561278+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":74,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.056612+0000","last_change":"2024-02-05T06:40:08.602038+0000","last_active":"2024-02-05T06:40:56.056612+0000","last_peered":"2024-02-05T06:40:56.056612+0000","last_clean":"2024-02-05T06:40:56.056612+0000","last_became_active":"2024-02-05T06:40:08.601685+0000","last_became_peered":"2024-02-05T06:40:08.601685+0000","last_unstale":"2024-02-05T06:40:56.056612+0000","last_undegraded":"2024-02-05T06:40:56.056612+0000","last_fullsized":"2024-02-05T06:40:56.056612+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:48.266135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:21.998850+0000","last_clean_scrub_stamp":"2024-02-05T06:37:48.266135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:00.317629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":53,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.055248+0000","last_change":"2024-02-05T06:40:27.072096+0000","last_active":"2024-02-05T06:40:56.055248+0000","last_peered":"2024-02-05T06:40:56.055248+0000","last_clean":"2024-02-05T06:40:56.055248+0000","last_became_active":"2024-02-05T06:40:27.071758+0000","last_became_peered":"2024-02-05T06:40:27.071758+0000","last_unstale":"2024-02-05T06:40:56.055248+0000","last_undegraded":"2024-02-05T06:40:56.055248+0000","last_fullsized":"2024-02-05T06:40:56.055248+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:26.781111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:37:26.781111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":209,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.038724+0000","last_change":"2024-02-05T06:40:08.579914+0000","last_active":"2024-02-05T06:40:56.038724+0000","last_peered":"2024-02-05T06:40:56.038724+0000","last_clean":"2024-02-05T06:40:56.038724+0000","last_became_active":"2024-02-05T06:40:08.578882+0000","last_became_peered":"2024-02-05T06:40:08.578882+0000","last_unstale":"2024-02-05T06:40:56.038724+0000","last_undegraded":"2024-02-05T06:40:56.038724+0000","last_fullsized":"2024-02-05T06:40:56.038724+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:34.388028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:50.055245+0000","last_clean_scrub_stamp":"2024-02-05T06:39:34.388028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:40:56.561443+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":242,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.039203+0000","last_change":"2024-02-05T06:39:08.998613+0000","last_active":"2024-02-05T06:40:56.039203+0000","last_peered":"2024-02-05T06:40:56.039203+0000","last_clean":"2024-02-05T06:40:56.039203+0000","last_became_active":"2024-02-05T06:36:54.954385+0000","last_became_peered":"2024-02-05T06:36:54.954385+0000","last_unstale":"2024-02-05T06:40:56.039203+0000","last_undegraded":"2024-02-05T06:40:56.039203+0000","last_fullsized":"2024-02-05T06:40:56.039203+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-02-05T06:39:08.998484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:08.998484+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:40:56.561409+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":243,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.039121+0000","last_change":"2024-02-05T06:40:08.582243+0000","last_active":"2024-02-05T06:40:56.039121+0000","last_peered":"2024-02-05T06:40:56.039121+0000","last_clean":"2024-02-05T06:40:56.039121+0000","last_became_active":"2024-02-05T06:40:08.581272+0000","last_became_peered":"2024-02-05T06:40:08.581272+0000","last_unstale":"2024-02-05T06:40:56.039121+0000","last_undegraded":"2024-02-05T06:40:56.039121+0000","last_fullsized":"2024-02-05T06:40:56.039121+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:10.958212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:10.958212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:40:56.561357+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":362,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.053024+0000","last_change":"2024-02-05T06:39:32.263466+0000","last_active":"2024-02-05T06:40:56.053024+0000","last_peered":"2024-02-05T06:40:56.053024+0000","last_clean":"2024-02-05T06:40:56.053024+0000","last_became_active":"2024-02-05T06:36:54.958069+0000","last_became_peered":"2024-02-05T06:36:54.958069+0000","last_unstale":"2024-02-05T06:40:56.053024+0000","last_undegraded":"2024-02-05T06:40:56.053024+0000","last_fullsized":"2024-02-05T06:40:56.053024+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:32.263391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:32.263391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:00.317984+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"66'1","reported_seq":397,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.049050+0000","last_change":"2024-02-05T06:40:27.118661+0000","last_active":"2024-02-05T06:40:56.049050+0000","last_peered":"2024-02-05T06:40:56.049050+0000","last_clean":"2024-02-05T06:40:56.049050+0000","last_became_active":"2024-02-05T06:40:27.073360+0000","last_became_peered":"2024-02-05T06:40:27.073360+0000","last_unstale":"2024-02-05T06:40:56.049050+0000","last_undegraded":"2024-02-05T06:40:56.049050+0000","last_fullsized":"2024-02-05T06:40:56.049050+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:37:47.363630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:37:47.363630+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"88'22","reported_seq":392,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.057947+0000","last_change":"2024-02-05T06:39:11.993130+0000","last_active":"2024-02-05T06:40:56.057947+0000","last_peered":"2024-02-05T06:40:56.057947+0000","last_clean":"2024-02-05T06:40:56.057947+0000","last_became_active":"2024-02-05T06:33:31.569763+0000","last_became_peered":"2024-02-05T06:33:31.569763+0000","last_unstale":"2024-02-05T06:40:56.057947+0000","last_undegraded":"2024-02-05T06:40:56.057947+0000","last_fullsized":"2024-02-05T06:40:56.057947+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'22","last_scrub_stamp":"2024-02-05T06:39:11.992880+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:34:35.234636+0000","last_clean_scrub_stamp":"2024-02-05T06:39:11.992880+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":345,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.055740+0000","last_change":"2024-02-05T06:40:08.606128+0000","last_active":"2024-02-05T06:40:56.055740+0000","last_peered":"2024-02-05T06:40:56.055740+0000","last_clean":"2024-02-05T06:40:56.055740+0000","last_became_active":"2024-02-05T06:40:08.605815+0000","last_became_peered":"2024-02-05T06:40:08.605815+0000","last_unstale":"2024-02-05T06:40:56.055740+0000","last_undegraded":"2024-02-05T06:40:56.055740+0000","last_fullsized":"2024-02-05T06:40:56.055740+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:06.387381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:06.387381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":124,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.057376+0000","last_change":"2024-02-05T06:39:14.482807+0000","last_active":"2024-02-05T06:40:56.057376+0000","last_peered":"2024-02-05T06:40:56.057376+0000","last_clean":"2024-02-05T06:40:56.057376+0000","last_became_active":"2024-02-05T06:39:13.248618+0000","last_became_peered":"2024-02-05T06:39:13.248618+0000","last_unstale":"2024-02-05T06:40:56.057376+0000","last_undegraded":"2024-02-05T06:40:56.057376+0000","last_fullsized":"2024-02-05T06:40:56.057376+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:14.482739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:14.482739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":236,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.041651+0000","last_change":"2024-02-05T06:40:07.898901+0000","last_active":"2024-02-05T06:40:56.041651+0000","last_peered":"2024-02-05T06:40:56.041651+0000","last_clean":"2024-02-05T06:40:56.041651+0000","last_became_active":"2024-02-05T06:36:54.959070+0000","last_became_peered":"2024-02-05T06:36:54.959070+0000","last_unstale":"2024-02-05T06:40:56.041651+0000","last_undegraded":"2024-02-05T06:40:56.041651+0000","last_fullsized":"2024-02-05T06:40:56.041651+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:40:07.898795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:40:07.898795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:18.962211+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":423,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.050166+0000","last_change":"2024-02-05T06:38:16.181378+0000","last_active":"2024-02-05T06:40:56.050166+0000","last_peered":"2024-02-05T06:40:56.050166+0000","last_clean":"2024-02-05T06:40:56.050166+0000","last_became_active":"2024-02-05T06:36:54.954880+0000","last_became_peered":"2024-02-05T06:36:54.954880+0000","last_unstale":"2024-02-05T06:40:56.050166+0000","last_undegraded":"2024-02-05T06:40:56.050166+0000","last_fullsized":"2024-02-05T06:40:56.050166+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:38:16.181294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:38:16.181294+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"73'1","reported_seq":57,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.058069+0000","last_change":"2024-02-05T06:40:46.449931+0000","last_active":"2024-02-05T06:40:56.058069+0000","last_peered":"2024-02-05T06:40:56.058069+0000","last_clean":"2024-02-05T06:40:56.058069+0000","last_became_active":"2024-02-05T06:40:27.185817+0000","last_became_peered":"2024-02-05T06:40:27.185817+0000","last_unstale":"2024-02-05T06:40:56.058069+0000","last_undegraded":"2024-02-05T06:40:56.058069+0000","last_fullsized":"2024-02-05T06:40:56.058069+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-02-05T06:40:09.298234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:40:09.298234+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:09.632403+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"208'12","reported_seq":496,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.057512+0000","last_change":"2024-02-05T06:40:27.068738+0000","last_active":"2024-02-05T06:40:56.057512+0000","last_peered":"2024-02-05T06:40:56.057512+0000","last_clean":"2024-02-05T06:40:56.057512+0000","last_became_active":"2024-02-05T06:40:27.063399+0000","last_became_peered":"2024-02-05T06:40:27.063399+0000","last_unstale":"2024-02-05T06:40:56.057512+0000","last_undegraded":"2024-02-05T06:40:56.057512+0000","last_fullsized":"2024-02-05T06:40:56.057512+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'11","last_scrub_stamp":"2024-02-05T06:39:10.519706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:10.519706+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":380,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.056332+0000","last_change":"2024-02-05T06:39:07.546967+0000","last_active":"2024-02-05T06:40:56.056332+0000","last_peered":"2024-02-05T06:40:56.056332+0000","last_clean":"2024-02-05T06:40:56.056332+0000","last_became_active":"2024-02-05T06:36:54.958414+0000","last_became_peered":"2024-02-05T06:36:54.958414+0000","last_unstale":"2024-02-05T06:40:56.056332+0000","last_undegraded":"2024-02-05T06:40:56.056332+0000","last_fullsized":"2024-02-05T06:40:56.056332+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:39:07.546646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:39:07.546646+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:00.317713+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"92'15","reported_seq":412,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.054011+0000","last_change":"2024-02-05T06:40:46.436058+0000","last_active":"2024-02-05T06:40:56.054011+0000","last_peered":"2024-02-05T06:40:56.054011+0000","last_clean":"2024-02-05T06:40:56.054011+0000","last_became_active":"2024-02-05T06:40:27.100610+0000","last_became_peered":"2024-02-05T06:40:27.100610+0000","last_unstale":"2024-02-05T06:40:56.054011+0000","last_undegraded":"2024-02-05T06:40:56.054011+0000","last_fullsized":"2024-02-05T06:40:56.054011+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'15","last_scrub_stamp":"2024-02-05T06:40:07.595540+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-02-05T06:34:50.718768+0000","last_clean_scrub_stamp":"2024-02-05T06:40:07.595540+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:33.345169+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":364,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.053141+0000","last_change":"2024-02-05T06:40:08.593331+0000","last_active":"2024-02-05T06:40:56.053141+0000","last_peered":"2024-02-05T06:40:56.053141+0000","last_clean":"2024-02-05T06:40:56.053141+0000","last_became_active":"2024-02-05T06:40:08.593007+0000","last_became_peered":"2024-02-05T06:40:08.593007+0000","last_unstale":"2024-02-05T06:40:56.053141+0000","last_undegraded":"2024-02-05T06:40:56.053141+0000","last_fullsized":"2024-02-05T06:40:56.053141+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:06.475715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:06.475715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:00.317682+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":266,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.056894+0000","last_change":"2024-02-05T06:40:27.097489+0000","last_active":"2024-02-05T06:40:56.056894+0000","last_peered":"2024-02-05T06:40:56.056894+0000","last_clean":"2024-02-05T06:40:56.056894+0000","last_became_active":"2024-02-05T06:40:27.097169+0000","last_became_peered":"2024-02-05T06:40:27.097169+0000","last_unstale":"2024-02-05T06:40:56.056894+0000","last_undegraded":"2024-02-05T06:40:56.056894+0000","last_fullsized":"2024-02-05T06:40:56.056894+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:38:26.171039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:08.713894+0000","last_clean_scrub_stamp":"2024-02-05T06:38:26.171039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":73,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.050373+0000","last_change":"2024-02-05T06:40:08.607982+0000","last_active":"2024-02-05T06:40:56.050373+0000","last_peered":"2024-02-05T06:40:56.050373+0000","last_clean":"2024-02-05T06:40:56.050373+0000","last_became_active":"2024-02-05T06:40:08.606962+0000","last_became_peered":"2024-02-05T06:40:08.606962+0000","last_unstale":"2024-02-05T06:40:56.050373+0000","last_undegraded":"2024-02-05T06:40:56.050373+0000","last_fullsized":"2024-02-05T06:40:56.050373+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:40:05.263073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:40:05.263073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:10.913645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"92'16","reported_seq":407,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.057693+0000","last_change":"2024-02-05T06:40:08.667187+0000","last_active":"2024-02-05T06:40:56.057693+0000","last_peered":"2024-02-05T06:40:56.057693+0000","last_clean":"2024-02-05T06:40:56.057693+0000","last_became_active":"2024-02-05T06:40:08.603702+0000","last_became_peered":"2024-02-05T06:40:08.603702+0000","last_unstale":"2024-02-05T06:40:56.057693+0000","last_undegraded":"2024-02-05T06:40:56.057693+0000","last_fullsized":"2024-02-05T06:40:56.057693+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'16","last_scrub_stamp":"2024-02-05T06:40:01.612165+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:34:45.270135+0000","last_clean_scrub_stamp":"2024-02-05T06:40:01.612165+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:11.450516+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":53,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.055362+0000","last_change":"2024-02-05T06:40:27.099270+0000","last_active":"2024-02-05T06:40:56.055362+0000","last_peered":"2024-02-05T06:40:56.055362+0000","last_clean":"2024-02-05T06:40:56.055362+0000","last_became_active":"2024-02-05T06:40:27.098348+0000","last_became_peered":"2024-02-05T06:40:27.098348+0000","last_unstale":"2024-02-05T06:40:56.055362+0000","last_undegraded":"2024-02-05T06:40:56.055362+0000","last_fullsized":"2024-02-05T06:40:56.055362+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:29.705961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:37:29.705961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":382,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.056835+0000","last_change":"2024-02-05T06:40:27.060697+0000","last_active":"2024-02-05T06:40:56.056835+0000","last_peered":"2024-02-05T06:40:56.056835+0000","last_clean":"2024-02-05T06:40:56.056835+0000","last_became_active":"2024-02-05T06:40:27.060112+0000","last_became_peered":"2024-02-05T06:40:27.060112+0000","last_unstale":"2024-02-05T06:40:56.056835+0000","last_undegraded":"2024-02-05T06:40:56.056835+0000","last_fullsized":"2024-02-05T06:40:56.056835+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:15.439354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:15.439354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":358,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.051297+0000","last_change":"2024-02-05T06:40:08.600081+0000","last_active":"2024-02-05T06:40:56.051297+0000","last_peered":"2024-02-05T06:40:56.051297+0000","last_clean":"2024-02-05T06:40:56.051297+0000","last_became_active":"2024-02-05T06:40:08.595986+0000","last_became_peered":"2024-02-05T06:40:08.595986+0000","last_unstale":"2024-02-05T06:40:56.051297+0000","last_undegraded":"2024-02-05T06:40:56.051297+0000","last_fullsized":"2024-02-05T06:40:56.051297+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:40:05.586211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:40:05.586211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:20.946734+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"73'4","reported_seq":353,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.062409+0000","last_change":"2024-02-05T06:39:32.415715+0000","last_active":"2024-02-05T06:40:56.062409+0000","last_peered":"2024-02-05T06:40:56.062409+0000","last_clean":"2024-02-05T06:40:56.062409+0000","last_became_active":"2024-02-05T06:33:31.573410+0000","last_became_peered":"2024-02-05T06:33:31.573410+0000","last_unstale":"2024-02-05T06:40:56.062409+0000","last_undegraded":"2024-02-05T06:40:56.062409+0000","last_fullsized":"2024-02-05T06:40:56.062409+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'4","last_scrub_stamp":"2024-02-05T06:39:32.415626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:32.415626+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":125,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.054282+0000","last_change":"2024-02-05T06:40:02.234701+0000","last_active":"2024-02-05T06:40:56.054282+0000","last_peered":"2024-02-05T06:40:56.054282+0000","last_clean":"2024-02-05T06:40:56.054282+0000","last_became_active":"2024-02-05T06:39:13.245684+0000","last_became_peered":"2024-02-05T06:39:13.245684+0000","last_unstale":"2024-02-05T06:40:56.054282+0000","last_undegraded":"2024-02-05T06:40:56.054282+0000","last_fullsized":"2024-02-05T06:40:56.054282+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:40:02.234576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:40:02.234576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:22.594196+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":53,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.055165+0000","last_change":"2024-02-05T06:40:27.063394+0000","last_active":"2024-02-05T06:40:56.055165+0000","last_peered":"2024-02-05T06:40:56.055165+0000","last_clean":"2024-02-05T06:40:56.055165+0000","last_became_active":"2024-02-05T06:40:27.062975+0000","last_became_peered":"2024-02-05T06:40:27.062975+0000","last_unstale":"2024-02-05T06:40:56.055165+0000","last_undegraded":"2024-02-05T06:40:56.055165+0000","last_fullsized":"2024-02-05T06:40:56.055165+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:43.604994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:37:43.604994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":53,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.056088+0000","last_change":"2024-02-05T06:40:27.094305+0000","last_active":"2024-02-05T06:40:56.056088+0000","last_peered":"2024-02-05T06:40:56.056088+0000","last_clean":"2024-02-05T06:40:56.056088+0000","last_became_active":"2024-02-05T06:40:27.093971+0000","last_became_peered":"2024-02-05T06:40:27.093971+0000","last_unstale":"2024-02-05T06:40:56.056088+0000","last_undegraded":"2024-02-05T06:40:56.056088+0000","last_fullsized":"2024-02-05T06:40:56.056088+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:43.243812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:37:43.243812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"93'16","reported_seq":425,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.052225+0000","last_change":"2024-02-05T06:40:27.124811+0000","last_active":"2024-02-05T06:40:56.052225+0000","last_peered":"2024-02-05T06:40:56.052225+0000","last_clean":"2024-02-05T06:40:56.052225+0000","last_became_active":"2024-02-05T06:40:27.080906+0000","last_became_peered":"2024-02-05T06:40:27.080906+0000","last_unstale":"2024-02-05T06:40:56.052225+0000","last_undegraded":"2024-02-05T06:40:56.052225+0000","last_fullsized":"2024-02-05T06:40:56.052225+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'16","last_scrub_stamp":"2024-02-05T06:39:11.291756+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-02-05T06:35:22.938675+0000","last_clean_scrub_stamp":"2024-02-05T06:39:11.291756+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"73'1","reported_seq":351,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.048238+0000","last_change":"2024-02-05T06:40:08.649622+0000","last_active":"2024-02-05T06:40:56.048238+0000","last_peered":"2024-02-05T06:40:56.048238+0000","last_clean":"2024-02-05T06:40:56.048238+0000","last_became_active":"2024-02-05T06:40:08.584043+0000","last_became_peered":"2024-02-05T06:40:08.584043+0000","last_unstale":"2024-02-05T06:40:56.048238+0000","last_undegraded":"2024-02-05T06:40:56.048238+0000","last_fullsized":"2024-02-05T06:40:56.048238+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-02-05T06:37:41.335213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:37:41.335213+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":343,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.060368+0000","last_change":"2024-02-05T06:39:25.369971+0000","last_active":"2024-02-05T06:40:56.060368+0000","last_peered":"2024-02-05T06:40:56.060368+0000","last_clean":"2024-02-05T06:40:56.060368+0000","last_became_active":"2024-02-05T06:33:33.593901+0000","last_became_peered":"2024-02-05T06:33:33.593901+0000","last_unstale":"2024-02-05T06:40:56.060368+0000","last_undegraded":"2024-02-05T06:40:56.060368+0000","last_fullsized":"2024-02-05T06:40:56.060368+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:25.369865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:25.369865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":53,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.055733+0000","last_change":"2024-02-05T06:40:27.078243+0000","last_active":"2024-02-05T06:40:56.055733+0000","last_peered":"2024-02-05T06:40:56.055733+0000","last_clean":"2024-02-05T06:40:56.055733+0000","last_became_active":"2024-02-05T06:40:27.077701+0000","last_became_peered":"2024-02-05T06:40:27.077701+0000","last_unstale":"2024-02-05T06:40:56.055733+0000","last_undegraded":"2024-02-05T06:40:56.055733+0000","last_fullsized":"2024-02-05T06:40:56.055733+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:27.744143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:37:27.744143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":127,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.050368+0000","last_change":"2024-02-05T06:39:13.292585+0000","last_active":"2024-02-05T06:40:56.050368+0000","last_peered":"2024-02-05T06:40:56.050368+0000","last_clean":"2024-02-05T06:40:56.050368+0000","last_became_active":"2024-02-05T06:39:13.244124+0000","last_became_peered":"2024-02-05T06:39:13.244124+0000","last_unstale":"2024-02-05T06:40:56.050368+0000","last_undegraded":"2024-02-05T06:40:56.050368+0000","last_fullsized":"2024-02-05T06:40:56.050368+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:36:30.047048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:30.047048+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":345,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.051972+0000","last_change":"2024-02-05T06:40:08.596936+0000","last_active":"2024-02-05T06:40:56.051972+0000","last_peered":"2024-02-05T06:40:56.051972+0000","last_clean":"2024-02-05T06:40:56.051972+0000","last_became_active":"2024-02-05T06:40:08.596289+0000","last_became_peered":"2024-02-05T06:40:08.596289+0000","last_unstale":"2024-02-05T06:40:56.051972+0000","last_undegraded":"2024-02-05T06:40:56.051972+0000","last_fullsized":"2024-02-05T06:40:56.051972+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:30.450988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:30.450988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":371,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.057014+0000","last_change":"2024-02-05T06:40:27.080858+0000","last_active":"2024-02-05T06:40:56.057014+0000","last_peered":"2024-02-05T06:40:56.057014+0000","last_clean":"2024-02-05T06:40:56.057014+0000","last_became_active":"2024-02-05T06:40:27.080531+0000","last_became_peered":"2024-02-05T06:40:27.080531+0000","last_unstale":"2024-02-05T06:40:56.057014+0000","last_undegraded":"2024-02-05T06:40:56.057014+0000","last_fullsized":"2024-02-05T06:40:56.057014+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:35.213313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:35.213313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:40:57.702622+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":248,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.038775+0000","last_change":"2024-02-05T06:40:46.639977+0000","last_active":"2024-02-05T06:40:56.038775+0000","last_peered":"2024-02-05T06:40:56.038775+0000","last_clean":"2024-02-05T06:40:56.038775+0000","last_became_active":"2024-02-05T06:40:27.065047+0000","last_became_peered":"2024-02-05T06:40:27.065047+0000","last_unstale":"2024-02-05T06:40:56.038775+0000","last_undegraded":"2024-02-05T06:40:56.038775+0000","last_fullsized":"2024-02-05T06:40:56.038775+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:40:46.639897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:40:46.639897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:51.971299+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"92'16","reported_seq":240,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.039369+0000","last_change":"2024-02-05T06:36:54.999613+0000","last_active":"2024-02-05T06:40:56.039369+0000","last_peered":"2024-02-05T06:40:56.039369+0000","last_clean":"2024-02-05T06:40:56.039369+0000","last_became_active":"2024-02-05T06:36:54.961610+0000","last_became_peered":"2024-02-05T06:36:54.961610+0000","last_unstale":"2024-02-05T06:40:56.039369+0000","last_undegraded":"2024-02-05T06:40:56.039369+0000","last_fullsized":"2024-02-05T06:40:56.039369+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'16","last_scrub_stamp":"2024-02-05T06:36:36.728552+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:35:02.589229+0000","last_clean_scrub_stamp":"2024-02-05T06:36:36.728552+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:40:56.561221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":352,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.048979+0000","last_change":"2024-02-05T06:40:27.124829+0000","last_active":"2024-02-05T06:40:56.048979+0000","last_peered":"2024-02-05T06:40:56.048979+0000","last_clean":"2024-02-05T06:40:56.048979+0000","last_became_active":"2024-02-05T06:40:27.096132+0000","last_became_peered":"2024-02-05T06:40:27.096132+0000","last_unstale":"2024-02-05T06:40:56.048979+0000","last_undegraded":"2024-02-05T06:40:56.048979+0000","last_fullsized":"2024-02-05T06:40:56.048979+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:31.831291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:31.831291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"197'9","reported_seq":113,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.052941+0000","last_change":"2024-02-05T06:40:08.659377+0000","last_active":"2024-02-05T06:40:56.052941+0000","last_peered":"2024-02-05T06:40:56.052941+0000","last_clean":"2024-02-05T06:40:56.052941+0000","last_became_active":"2024-02-05T06:40:08.596051+0000","last_became_peered":"2024-02-05T06:40:08.596051+0000","last_unstale":"2024-02-05T06:40:56.052941+0000","last_undegraded":"2024-02-05T06:40:56.052941+0000","last_fullsized":"2024-02-05T06:40:56.052941+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:39:09.504696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:09.504696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:00.317744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":53,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.055546+0000","last_change":"2024-02-05T06:40:27.098718+0000","last_active":"2024-02-05T06:40:56.055546+0000","last_peered":"2024-02-05T06:40:56.055546+0000","last_clean":"2024-02-05T06:40:56.055546+0000","last_became_active":"2024-02-05T06:40:27.098418+0000","last_became_peered":"2024-02-05T06:40:27.098418+0000","last_unstale":"2024-02-05T06:40:56.055546+0000","last_undegraded":"2024-02-05T06:40:56.055546+0000","last_fullsized":"2024-02-05T06:40:56.055546+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:34.651493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:37:34.651493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'10","reported_seq":416,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.057744+0000","last_change":"2024-02-05T06:40:27.198874+0000","last_active":"2024-02-05T06:40:56.057744+0000","last_peered":"2024-02-05T06:40:56.057744+0000","last_clean":"2024-02-05T06:40:56.057744+0000","last_became_active":"2024-02-05T06:40:27.093418+0000","last_became_peered":"2024-02-05T06:40:27.093418+0000","last_unstale":"2024-02-05T06:40:56.057744+0000","last_undegraded":"2024-02-05T06:40:56.057744+0000","last_fullsized":"2024-02-05T06:40:56.057744+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-02-05T06:38:21.205097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:38:21.205097+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":242,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.039239+0000","last_change":"2024-02-05T06:40:08.583062+0000","last_active":"2024-02-05T06:40:56.039239+0000","last_peered":"2024-02-05T06:40:56.039239+0000","last_clean":"2024-02-05T06:40:56.039239+0000","last_became_active":"2024-02-05T06:40:08.582586+0000","last_became_peered":"2024-02-05T06:40:08.582586+0000","last_unstale":"2024-02-05T06:40:56.039239+0000","last_undegraded":"2024-02-05T06:40:56.039239+0000","last_fullsized":"2024-02-05T06:40:56.039239+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:18.964707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:18.964707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:40:56.561340+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"208'10","reported_seq":461,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.058110+0000","last_change":"2024-02-05T06:40:27.202569+0000","last_active":"2024-02-05T06:40:56.058110+0000","last_peered":"2024-02-05T06:40:56.058110+0000","last_clean":"2024-02-05T06:40:56.058110+0000","last_became_active":"2024-02-05T06:40:27.096584+0000","last_became_peered":"2024-02-05T06:40:27.096584+0000","last_unstale":"2024-02-05T06:40:56.058110+0000","last_undegraded":"2024-02-05T06:40:56.058110+0000","last_fullsized":"2024-02-05T06:40:56.058110+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:36:26.411576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:26.411576+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":364,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.050769+0000","last_change":"2024-02-05T06:40:27.088637+0000","last_active":"2024-02-05T06:40:56.050769+0000","last_peered":"2024-02-05T06:40:56.050769+0000","last_clean":"2024-02-05T06:40:56.050769+0000","last_became_active":"2024-02-05T06:40:27.088096+0000","last_became_peered":"2024-02-05T06:40:27.088096+0000","last_unstale":"2024-02-05T06:40:56.050769+0000","last_undegraded":"2024-02-05T06:40:56.050769+0000","last_fullsized":"2024-02-05T06:40:56.050769+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:31.169691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:36:31.169691+0000","last_clean_scrub_stamp":"2024-02-05T06:36:31.169691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"73'9","reported_seq":63,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.055825+0000","last_change":"2024-02-05T06:40:27.381430+0000","last_active":"2024-02-05T06:40:56.055825+0000","last_peered":"2024-02-05T06:40:56.055825+0000","last_clean":"2024-02-05T06:40:56.055825+0000","last_became_active":"2024-02-05T06:40:27.140868+0000","last_became_peered":"2024-02-05T06:40:27.140868+0000","last_unstale":"2024-02-05T06:40:56.055825+0000","last_undegraded":"2024-02-05T06:40:56.055825+0000","last_fullsized":"2024-02-05T06:40:56.055825+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:39:16.394399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:16.394399+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":377,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.058305+0000","last_change":"2024-02-05T06:40:27.092829+0000","last_active":"2024-02-05T06:40:56.058305+0000","last_peered":"2024-02-05T06:40:56.058305+0000","last_clean":"2024-02-05T06:40:56.058305+0000","last_became_active":"2024-02-05T06:40:27.090632+0000","last_became_peered":"2024-02-05T06:40:27.090632+0000","last_unstale":"2024-02-05T06:40:56.058305+0000","last_undegraded":"2024-02-05T06:40:56.058305+0000","last_fullsized":"2024-02-05T06:40:56.058305+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:19.054356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:19.054356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":369,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.055665+0000","last_change":"2024-02-05T06:40:27.092004+0000","last_active":"2024-02-05T06:40:56.055665+0000","last_peered":"2024-02-05T06:40:56.055665+0000","last_clean":"2024-02-05T06:40:56.055665+0000","last_became_active":"2024-02-05T06:40:27.091544+0000","last_became_peered":"2024-02-05T06:40:27.091544+0000","last_unstale":"2024-02-05T06:40:56.055665+0000","last_undegraded":"2024-02-05T06:40:56.055665+0000","last_fullsized":"2024-02-05T06:40:56.055665+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:40:03.572450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:40:03.572450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:11.986980+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":53,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.055327+0000","last_change":"2024-02-05T06:40:27.064705+0000","last_active":"2024-02-05T06:40:56.055327+0000","last_peered":"2024-02-05T06:40:56.055327+0000","last_clean":"2024-02-05T06:40:56.055327+0000","last_became_active":"2024-02-05T06:40:27.064388+0000","last_became_peered":"2024-02-05T06:40:27.064388+0000","last_unstale":"2024-02-05T06:40:56.055327+0000","last_undegraded":"2024-02-05T06:40:56.055327+0000","last_fullsized":"2024-02-05T06:40:56.055327+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:38:13.558400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:38:13.558400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"73'9","reported_seq":396,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.054106+0000","last_change":"2024-02-05T06:40:46.447632+0000","last_active":"2024-02-05T06:40:56.054106+0000","last_peered":"2024-02-05T06:40:56.054106+0000","last_clean":"2024-02-05T06:40:56.054106+0000","last_became_active":"2024-02-05T06:40:27.101116+0000","last_became_peered":"2024-02-05T06:40:27.101116+0000","last_unstale":"2024-02-05T06:40:56.054106+0000","last_undegraded":"2024-02-05T06:40:56.054106+0000","last_fullsized":"2024-02-05T06:40:56.054106+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:36:23.148541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:23.148541+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":53,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.055919+0000","last_change":"2024-02-05T06:40:27.063878+0000","last_active":"2024-02-05T06:40:56.055919+0000","last_peered":"2024-02-05T06:40:56.055919+0000","last_clean":"2024-02-05T06:40:56.055919+0000","last_became_active":"2024-02-05T06:40:27.063350+0000","last_became_peered":"2024-02-05T06:40:27.063350+0000","last_unstale":"2024-02-05T06:40:56.055919+0000","last_undegraded":"2024-02-05T06:40:56.055919+0000","last_fullsized":"2024-02-05T06:40:56.055919+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:38:16.709585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:38:16.709585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":74,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.057167+0000","last_change":"2024-02-05T06:40:08.605454+0000","last_active":"2024-02-05T06:40:56.057167+0000","last_peered":"2024-02-05T06:40:56.057167+0000","last_clean":"2024-02-05T06:40:56.057167+0000","last_became_active":"2024-02-05T06:40:08.605110+0000","last_became_peered":"2024-02-05T06:40:08.605110+0000","last_unstale":"2024-02-05T06:40:56.057167+0000","last_undegraded":"2024-02-05T06:40:56.057167+0000","last_fullsized":"2024-02-05T06:40:56.057167+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:29.372079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:45.096340+0000","last_clean_scrub_stamp":"2024-02-05T06:39:29.372079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":372,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.056636+0000","last_change":"2024-02-05T06:40:08.590689+0000","last_active":"2024-02-05T06:40:56.056636+0000","last_peered":"2024-02-05T06:40:56.056636+0000","last_clean":"2024-02-05T06:40:56.056636+0000","last_became_active":"2024-02-05T06:40:08.590234+0000","last_became_peered":"2024-02-05T06:40:08.590234+0000","last_unstale":"2024-02-05T06:40:56.056636+0000","last_undegraded":"2024-02-05T06:40:56.056636+0000","last_fullsized":"2024-02-05T06:40:56.056636+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:22.309939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:39:22.309939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":384,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.055821+0000","last_change":"2024-02-05T06:40:27.110051+0000","last_active":"2024-02-05T06:40:56.055821+0000","last_peered":"2024-02-05T06:40:56.055821+0000","last_clean":"2024-02-05T06:40:56.055821+0000","last_became_active":"2024-02-05T06:40:27.091689+0000","last_became_peered":"2024-02-05T06:40:27.091689+0000","last_unstale":"2024-02-05T06:40:56.055821+0000","last_undegraded":"2024-02-05T06:40:56.055821+0000","last_fullsized":"2024-02-05T06:40:56.055821+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'6","last_scrub_stamp":"2024-02-05T06:38:32.122868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:38:32.122868+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"227'4065","reported_seq":7955,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.140503+0000","last_change":"2024-02-05T06:40:20.020947+0000","last_active":"2024-02-05T06:40:56.140503+0000","last_peered":"2024-02-05T06:40:56.140503+0000","last_clean":"2024-02-05T06:40:56.140503+0000","last_became_active":"2024-02-05T06:40:20.018366+0000","last_became_peered":"2024-02-05T06:40:20.018366+0000","last_unstale":"2024-02-05T06:40:56.140503+0000","last_undegraded":"2024-02-05T06:40:56.140503+0000","last_fullsized":"2024-02-05T06:40:56.140503+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":4065,"log_dups_size":0,"ondisk_log_size":4065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:40:56.561524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266403840,"num_objects":4065,"num_object_clones":0,"num_object_copies":12195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4065,"num_write_kb":260160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1894,"num_bytes_recovered":124125184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":74,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.056812+0000","last_change":"2024-02-05T06:40:08.591629+0000","last_active":"2024-02-05T06:40:56.056812+0000","last_peered":"2024-02-05T06:40:56.056812+0000","last_clean":"2024-02-05T06:40:56.056812+0000","last_became_active":"2024-02-05T06:40:08.590874+0000","last_became_peered":"2024-02-05T06:40:08.590874+0000","last_unstale":"2024-02-05T06:40:56.056812+0000","last_undegraded":"2024-02-05T06:40:56.056812+0000","last_fullsized":"2024-02-05T06:40:56.056812+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:23.325194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:23.325194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":59,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.056241+0000","last_change":"2024-02-05T06:40:27.105096+0000","last_active":"2024-02-05T06:40:56.056241+0000","last_peered":"2024-02-05T06:40:56.056241+0000","last_clean":"2024-02-05T06:40:56.056241+0000","last_became_active":"2024-02-05T06:40:27.068279+0000","last_became_peered":"2024-02-05T06:40:27.068279+0000","last_unstale":"2024-02-05T06:40:56.056241+0000","last_undegraded":"2024-02-05T06:40:56.056241+0000","last_fullsized":"2024-02-05T06:40:56.056241+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-02-05T06:37:28.708774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:37:28.708774+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"73'15","reported_seq":69,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.055030+0000","last_change":"2024-02-05T06:40:27.377323+0000","last_active":"2024-02-05T06:40:56.055030+0000","last_peered":"2024-02-05T06:40:56.055030+0000","last_clean":"2024-02-05T06:40:56.055030+0000","last_became_active":"2024-02-05T06:40:27.062755+0000","last_became_peered":"2024-02-05T06:40:27.062755+0000","last_unstale":"2024-02-05T06:40:56.055030+0000","last_undegraded":"2024-02-05T06:40:56.055030+0000","last_fullsized":"2024-02-05T06:40:56.055030+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-02-05T06:40:01.321025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:40:01.321025+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:09.556597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"208'12","reported_seq":63,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.055232+0000","last_change":"2024-02-05T06:40:27.091490+0000","last_active":"2024-02-05T06:40:56.055232+0000","last_peered":"2024-02-05T06:40:56.055232+0000","last_clean":"2024-02-05T06:40:56.055232+0000","last_became_active":"2024-02-05T06:40:27.064195+0000","last_became_peered":"2024-02-05T06:40:27.064195+0000","last_unstale":"2024-02-05T06:40:56.055232+0000","last_undegraded":"2024-02-05T06:40:56.055232+0000","last_fullsized":"2024-02-05T06:40:56.055232+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-02-05T06:38:11.542821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:38:11.542821+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"227'3992","reported_seq":9480,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.211826+0000","last_change":"2024-02-05T06:40:47.342532+0000","last_active":"2024-02-05T06:40:56.211826+0000","last_peered":"2024-02-05T06:40:56.211826+0000","last_clean":"2024-02-05T06:40:56.211826+0000","last_became_active":"2024-02-05T06:40:47.337156+0000","last_became_peered":"2024-02-05T06:40:47.337156+0000","last_unstale":"2024-02-05T06:40:56.211826+0000","last_undegraded":"2024-02-05T06:40:56.211826+0000","last_fullsized":"2024-02-05T06:40:56.211826+0000","mapping_epoch":221,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":3992,"log_dups_size":0,"ondisk_log_size":3992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261619712,"num_objects":3992,"num_object_clones":0,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3992,"num_write_kb":255488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2692,"num_bytes_recovered":176422912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":351,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.060554+0000","last_change":"2024-02-05T06:40:27.092917+0000","last_active":"2024-02-05T06:40:56.060554+0000","last_peered":"2024-02-05T06:40:56.060554+0000","last_clean":"2024-02-05T06:40:56.060554+0000","last_became_active":"2024-02-05T06:40:27.092613+0000","last_became_peered":"2024-02-05T06:40:27.092613+0000","last_unstale":"2024-02-05T06:40:56.060554+0000","last_undegraded":"2024-02-05T06:40:56.060554+0000","last_fullsized":"2024-02-05T06:40:56.060554+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:26.390356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:26.390356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":53,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.056130+0000","last_change":"2024-02-05T06:40:27.063835+0000","last_active":"2024-02-05T06:40:56.056130+0000","last_peered":"2024-02-05T06:40:56.056130+0000","last_clean":"2024-02-05T06:40:56.056130+0000","last_became_active":"2024-02-05T06:40:27.063300+0000","last_became_peered":"2024-02-05T06:40:27.063300+0000","last_unstale":"2024-02-05T06:40:56.056130+0000","last_undegraded":"2024-02-05T06:40:56.056130+0000","last_fullsized":"2024-02-05T06:40:56.056130+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:38:15.583491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:38:15.583491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'12","reported_seq":248,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.038453+0000","last_change":"2024-02-05T06:39:09.987283+0000","last_active":"2024-02-05T06:40:56.038453+0000","last_peered":"2024-02-05T06:40:56.038453+0000","last_clean":"2024-02-05T06:40:56.038453+0000","last_became_active":"2024-02-05T06:36:54.954071+0000","last_became_peered":"2024-02-05T06:36:54.954071+0000","last_unstale":"2024-02-05T06:40:56.038453+0000","last_undegraded":"2024-02-05T06:40:56.038453+0000","last_fullsized":"2024-02-05T06:40:56.038453+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:39:09.987177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:09.987177+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:40:56.561426+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":349,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.058471+0000","last_change":"2024-02-05T06:39:35.954431+0000","last_active":"2024-02-05T06:40:56.058471+0000","last_peered":"2024-02-05T06:40:56.058471+0000","last_clean":"2024-02-05T06:40:56.058471+0000","last_became_active":"2024-02-05T06:33:33.592546+0000","last_became_peered":"2024-02-05T06:33:33.592546+0000","last_unstale":"2024-02-05T06:40:56.058471+0000","last_undegraded":"2024-02-05T06:40:56.058471+0000","last_fullsized":"2024-02-05T06:40:56.058471+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:35.954359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:35.954359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"227'4007","reported_seq":4103,"reported_epoch":227,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-02-05T06:40:56.266968+0000","last_change":"2024-02-05T06:40:28.172749+0000","last_active":"2024-02-05T06:40:56.266968+0000","last_peered":"2024-02-05T06:40:56.266968+0000","last_clean":"2024-02-05T06:40:26.040508+0000","last_became_active":"2024-02-05T06:40:28.168410+0000","last_became_peered":"2024-02-05T06:40:28.168410+0000","last_unstale":"2024-02-05T06:40:56.266968+0000","last_undegraded":"2024-02-05T06:40:28.079056+0000","last_fullsized":"2024-02-05T06:40:28.058589+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":4007,"log_dups_size":0,"ondisk_log_size":4007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262602752,"num_objects":4007,"num_object_clones":0,"num_object_copies":12021,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4007,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4007,"num_write_kb":256448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6],"avail_no_missing":["0","2","6"],"object_location_counts":[{"shards":"0,2,6","objects":2773},{"shards":"0,6","objects":1234}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":53,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.055256+0000","last_change":"2024-02-05T06:40:27.218419+0000","last_active":"2024-02-05T06:40:56.055256+0000","last_peered":"2024-02-05T06:40:56.055256+0000","last_clean":"2024-02-05T06:40:56.055256+0000","last_became_active":"2024-02-05T06:40:27.218081+0000","last_became_peered":"2024-02-05T06:40:27.218081+0000","last_unstale":"2024-02-05T06:40:56.055256+0000","last_undegraded":"2024-02-05T06:40:56.055256+0000","last_fullsized":"2024-02-05T06:40:56.055256+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:32.925000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:32.925000+0000","last_clean_scrub_stamp":"2024-02-05T06:39:32.925000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:01.220130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":361,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.059004+0000","last_change":"2024-02-05T06:40:27.089797+0000","last_active":"2024-02-05T06:40:56.059004+0000","last_peered":"2024-02-05T06:40:56.059004+0000","last_clean":"2024-02-05T06:40:56.059004+0000","last_became_active":"2024-02-05T06:40:27.089489+0000","last_became_peered":"2024-02-05T06:40:27.089489+0000","last_unstale":"2024-02-05T06:40:56.059004+0000","last_undegraded":"2024-02-05T06:40:56.059004+0000","last_fullsized":"2024-02-05T06:40:56.059004+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:38:11.612313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:38:11.612313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":53,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.055740+0000","last_change":"2024-02-05T06:40:27.078034+0000","last_active":"2024-02-05T06:40:56.055740+0000","last_peered":"2024-02-05T06:40:56.055740+0000","last_clean":"2024-02-05T06:40:56.055740+0000","last_became_active":"2024-02-05T06:40:27.077534+0000","last_became_peered":"2024-02-05T06:40:27.077534+0000","last_unstale":"2024-02-05T06:40:56.055740+0000","last_undegraded":"2024-02-05T06:40:56.055740+0000","last_fullsized":"2024-02-05T06:40:56.055740+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:40:04.284398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:40:04.284398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:23.943119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"73'19","reported_seq":471,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.056924+0000","last_change":"2024-02-05T06:40:27.083505+0000","last_active":"2024-02-05T06:40:56.056924+0000","last_peered":"2024-02-05T06:40:56.056924+0000","last_clean":"2024-02-05T06:40:56.056924+0000","last_became_active":"2024-02-05T06:40:27.068417+0000","last_became_peered":"2024-02-05T06:40:27.068417+0000","last_unstale":"2024-02-05T06:40:56.056924+0000","last_undegraded":"2024-02-05T06:40:56.056924+0000","last_fullsized":"2024-02-05T06:40:56.056924+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-02-05T06:39:34.480345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:34.480345+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"227'3991","reported_seq":4085,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.240591+0000","last_change":"2024-02-05T06:39:42.012271+0000","last_active":"2024-02-05T06:40:56.240591+0000","last_peered":"2024-02-05T06:40:56.240591+0000","last_clean":"2024-02-05T06:40:56.240591+0000","last_became_active":"2024-02-05T06:39:42.011938+0000","last_became_peered":"2024-02-05T06:39:42.011938+0000","last_unstale":"2024-02-05T06:40:56.240591+0000","last_undegraded":"2024-02-05T06:40:56.240591+0000","last_fullsized":"2024-02-05T06:40:56.240591+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":3991,"log_dups_size":0,"ondisk_log_size":3991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:09.409984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":261554176,"num_objects":3991,"num_object_clones":0,"num_object_copies":11973,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3991,"num_write_kb":255424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":344,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.051024+0000","last_change":"2024-02-05T06:40:27.130950+0000","last_active":"2024-02-05T06:40:56.051024+0000","last_peered":"2024-02-05T06:40:56.051024+0000","last_clean":"2024-02-05T06:40:56.051024+0000","last_became_active":"2024-02-05T06:40:27.129625+0000","last_became_peered":"2024-02-05T06:40:27.129625+0000","last_unstale":"2024-02-05T06:40:56.051024+0000","last_undegraded":"2024-02-05T06:40:56.051024+0000","last_fullsized":"2024-02-05T06:40:56.051024+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:22.102133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:22.102133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":369,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.060347+0000","last_change":"2024-02-05T06:39:34.909566+0000","last_active":"2024-02-05T06:40:56.060347+0000","last_peered":"2024-02-05T06:40:56.060347+0000","last_clean":"2024-02-05T06:40:56.060347+0000","last_became_active":"2024-02-05T06:36:54.966016+0000","last_became_peered":"2024-02-05T06:36:54.966016+0000","last_unstale":"2024-02-05T06:40:56.060347+0000","last_undegraded":"2024-02-05T06:40:56.060347+0000","last_fullsized":"2024-02-05T06:40:56.060347+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:34.909485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:34.909485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:04.907278+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":75,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.057831+0000","last_change":"2024-02-05T06:40:08.606060+0000","last_active":"2024-02-05T06:40:56.057831+0000","last_peered":"2024-02-05T06:40:56.057831+0000","last_clean":"2024-02-05T06:40:56.057831+0000","last_became_active":"2024-02-05T06:40:08.605597+0000","last_became_peered":"2024-02-05T06:40:08.605597+0000","last_unstale":"2024-02-05T06:40:56.057831+0000","last_undegraded":"2024-02-05T06:40:56.057831+0000","last_fullsized":"2024-02-05T06:40:56.057831+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:27.314329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:37:27.314329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"73'9","reported_seq":444,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.048908+0000","last_change":"2024-02-05T06:40:08.611772+0000","last_active":"2024-02-05T06:40:56.048908+0000","last_peered":"2024-02-05T06:40:56.048908+0000","last_clean":"2024-02-05T06:40:56.048908+0000","last_became_active":"2024-02-05T06:40:08.584204+0000","last_became_peered":"2024-02-05T06:40:08.584204+0000","last_unstale":"2024-02-05T06:40:56.048908+0000","last_undegraded":"2024-02-05T06:40:56.048908+0000","last_fullsized":"2024-02-05T06:40:56.048908+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:36:26.172612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:26.172612+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"227'4175","reported_seq":7668,"reported_epoch":227,"state":"active+recovering+undersized+remapped","last_fresh":"2024-02-05T06:40:56.773553+0000","last_change":"2024-02-05T06:40:27.365455+0000","last_active":"2024-02-05T06:40:56.773553+0000","last_peered":"2024-02-05T06:40:56.773553+0000","last_clean":"2024-02-05T06:40:07.517588+0000","last_became_active":"2024-02-05T06:40:27.269349+0000","last_became_peered":"2024-02-05T06:40:27.269349+0000","last_unstale":"2024-02-05T06:40:56.773553+0000","last_undegraded":"2024-02-05T06:40:56.773553+0000","last_fullsized":"2024-02-05T06:40:27.052939+0000","mapping_epoch":205,"log_start":"192'200","ondisk_log_start":"192'200","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":3975,"log_dups_size":0,"ondisk_log_size":3975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273612800,"num_objects":4175,"num_object_clones":0,"num_object_copies":12525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1680,"num_objects_unfound":0,"num_objects_dirty":4175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4175,"num_write_kb":267200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1695,"num_bytes_recovered":111083520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,0],"avail_no_missing":["3","0","5"],"object_location_counts":[{"shards":"0,2,3","objects":875},{"shards":"0,3","objects":2495},{"shards":"0,3,5","objects":805}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":382,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.049048+0000","last_change":"2024-02-05T06:40:11.050188+0000","last_active":"2024-02-05T06:40:56.049048+0000","last_peered":"2024-02-05T06:40:56.049048+0000","last_clean":"2024-02-05T06:40:56.049048+0000","last_became_active":"2024-02-05T06:40:11.049515+0000","last_became_peered":"2024-02-05T06:40:11.049515+0000","last_unstale":"2024-02-05T06:40:56.049048+0000","last_undegraded":"2024-02-05T06:40:56.049048+0000","last_fullsized":"2024-02-05T06:40:56.049048+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:37:37.282442+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:37.282442+0000","last_clean_scrub_stamp":"2024-02-05T06:37:37.282442+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:40:58.650134+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":363,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.057246+0000","last_change":"2024-02-05T06:40:27.082866+0000","last_active":"2024-02-05T06:40:56.057246+0000","last_peered":"2024-02-05T06:40:56.057246+0000","last_clean":"2024-02-05T06:40:56.057246+0000","last_became_active":"2024-02-05T06:40:27.082440+0000","last_became_peered":"2024-02-05T06:40:27.082440+0000","last_unstale":"2024-02-05T06:40:56.057246+0000","last_undegraded":"2024-02-05T06:40:56.057246+0000","last_fullsized":"2024-02-05T06:40:56.057246+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:07.364172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:07.364172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"182'3","reported_seq":351,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.056709+0000","last_change":"2024-02-05T06:40:08.658727+0000","last_active":"2024-02-05T06:40:56.056709+0000","last_peered":"2024-02-05T06:40:56.056709+0000","last_clean":"2024-02-05T06:40:56.056709+0000","last_became_active":"2024-02-05T06:40:08.594540+0000","last_became_peered":"2024-02-05T06:40:08.594540+0000","last_unstale":"2024-02-05T06:40:56.056709+0000","last_undegraded":"2024-02-05T06:40:56.056709+0000","last_fullsized":"2024-02-05T06:40:56.056709+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'3","last_scrub_stamp":"2024-02-05T06:39:33.373415+0000","last_deep_scrub":"182'3","last_deep_scrub_stamp":"2024-02-05T06:39:33.373415+0000","last_clean_scrub_stamp":"2024-02-05T06:39:33.373415+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:01.646207+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001220348,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":236,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.039293+0000","last_change":"2024-02-05T06:39:34.999778+0000","last_active":"2024-02-05T06:40:56.039293+0000","last_peered":"2024-02-05T06:40:56.039293+0000","last_clean":"2024-02-05T06:40:56.039293+0000","last_became_active":"2024-02-05T06:36:54.968205+0000","last_became_peered":"2024-02-05T06:36:54.968205+0000","last_unstale":"2024-02-05T06:40:56.039293+0000","last_undegraded":"2024-02-05T06:40:56.039293+0000","last_fullsized":"2024-02-05T06:40:56.039293+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:34.999659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:39:34.999659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:40:56.561491+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"197'30","reported_seq":105,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.052266+0000","last_change":"2024-02-05T06:40:08.673390+0000","last_active":"2024-02-05T06:40:56.052266+0000","last_peered":"2024-02-05T06:40:56.052266+0000","last_clean":"2024-02-05T06:40:56.052266+0000","last_became_active":"2024-02-05T06:40:08.604483+0000","last_became_peered":"2024-02-05T06:40:08.604483+0000","last_unstale":"2024-02-05T06:40:56.052266+0000","last_undegraded":"2024-02-05T06:40:56.052266+0000","last_fullsized":"2024-02-05T06:40:56.052266+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-02-05T06:39:30.346384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:30.346384+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:40:56.141119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":356,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.050278+0000","last_change":"2024-02-05T06:36:54.974218+0000","last_active":"2024-02-05T06:40:56.050278+0000","last_peered":"2024-02-05T06:40:56.050278+0000","last_clean":"2024-02-05T06:40:56.050278+0000","last_became_active":"2024-02-05T06:36:54.973387+0000","last_became_peered":"2024-02-05T06:36:54.973387+0000","last_unstale":"2024-02-05T06:40:56.050278+0000","last_undegraded":"2024-02-05T06:40:56.050278+0000","last_fullsized":"2024-02-05T06:40:56.050278+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:36.161606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:36.161606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"227'4069","reported_seq":4162,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.250940+0000","last_change":"2024-02-05T06:39:42.011437+0000","last_active":"2024-02-05T06:40:56.250940+0000","last_peered":"2024-02-05T06:40:56.250940+0000","last_clean":"2024-02-05T06:40:56.250940+0000","last_became_active":"2024-02-05T06:39:42.011116+0000","last_became_peered":"2024-02-05T06:39:42.011116+0000","last_unstale":"2024-02-05T06:40:56.250940+0000","last_undegraded":"2024-02-05T06:40:56.250940+0000","last_fullsized":"2024-02-05T06:40:56.250940+0000","mapping_epoch":185,"log_start":"192'100","ondisk_log_start":"192'100","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":3969,"log_dups_size":0,"ondisk_log_size":3969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266665984,"num_objects":4069,"num_object_clones":0,"num_object_copies":12207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4069,"num_write_kb":260416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":474,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.055274+0000","last_change":"2024-02-05T06:40:30.082941+0000","last_active":"2024-02-05T06:40:56.055274+0000","last_peered":"2024-02-05T06:40:56.055274+0000","last_clean":"2024-02-05T06:40:56.055274+0000","last_became_active":"2024-02-05T06:40:30.082467+0000","last_became_peered":"2024-02-05T06:40:30.082467+0000","last_unstale":"2024-02-05T06:40:56.055274+0000","last_undegraded":"2024-02-05T06:40:56.055274+0000","last_fullsized":"2024-02-05T06:40:56.055274+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:39:09.930291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:39:09.930291+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":53,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.055643+0000","last_change":"2024-02-05T06:40:27.078458+0000","last_active":"2024-02-05T06:40:56.055643+0000","last_peered":"2024-02-05T06:40:56.055643+0000","last_clean":"2024-02-05T06:40:56.055643+0000","last_became_active":"2024-02-05T06:40:27.077966+0000","last_became_peered":"2024-02-05T06:40:27.077966+0000","last_unstale":"2024-02-05T06:40:56.055643+0000","last_undegraded":"2024-02-05T06:40:56.055643+0000","last_fullsized":"2024-02-05T06:40:56.055643+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:36.461211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:36.461211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":359,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.052529+0000","last_change":"2024-02-05T06:39:27.282286+0000","last_active":"2024-02-05T06:40:56.052529+0000","last_peered":"2024-02-05T06:40:56.052529+0000","last_clean":"2024-02-05T06:40:56.052529+0000","last_became_active":"2024-02-05T06:33:29.551549+0000","last_became_peered":"2024-02-05T06:33:29.551549+0000","last_unstale":"2024-02-05T06:40:56.052529+0000","last_undegraded":"2024-02-05T06:40:56.052529+0000","last_fullsized":"2024-02-05T06:40:56.052529+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:27.282189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:39:27.282189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:00.322439+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":366,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.060388+0000","last_change":"2024-02-05T06:37:31.522883+0000","last_active":"2024-02-05T06:40:56.060388+0000","last_peered":"2024-02-05T06:40:56.060388+0000","last_clean":"2024-02-05T06:40:56.060388+0000","last_became_active":"2024-02-05T06:33:28.542977+0000","last_became_peered":"2024-02-05T06:33:28.542977+0000","last_unstale":"2024-02-05T06:40:56.060388+0000","last_undegraded":"2024-02-05T06:40:56.060388+0000","last_fullsized":"2024-02-05T06:40:56.060388+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:31.522765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:37:31.522765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0.00091640799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'13","reported_seq":426,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.057542+0000","last_change":"2024-02-05T06:40:27.140182+0000","last_active":"2024-02-05T06:40:56.057542+0000","last_peered":"2024-02-05T06:40:56.057542+0000","last_clean":"2024-02-05T06:40:56.057542+0000","last_became_active":"2024-02-05T06:40:27.070536+0000","last_became_peered":"2024-02-05T06:40:27.070536+0000","last_unstale":"2024-02-05T06:40:56.057542+0000","last_undegraded":"2024-02-05T06:40:56.057542+0000","last_fullsized":"2024-02-05T06:40:56.057542+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-02-05T06:39:33.280115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:33.280115+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:00.317921+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":375,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.059727+0000","last_change":"2024-02-05T06:40:27.089066+0000","last_active":"2024-02-05T06:40:56.059727+0000","last_peered":"2024-02-05T06:40:56.059727+0000","last_clean":"2024-02-05T06:40:56.059727+0000","last_became_active":"2024-02-05T06:40:27.088773+0000","last_became_peered":"2024-02-05T06:40:27.088773+0000","last_unstale":"2024-02-05T06:40:56.059727+0000","last_undegraded":"2024-02-05T06:40:56.059727+0000","last_fullsized":"2024-02-05T06:40:56.059727+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:33.914422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:33.914422+0000","last_clean_scrub_stamp":"2024-02-05T06:39:33.914422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"227'4054","reported_seq":7946,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.230940+0000","last_change":"2024-02-05T06:40:32.112242+0000","last_active":"2024-02-05T06:40:56.230940+0000","last_peered":"2024-02-05T06:40:56.230940+0000","last_clean":"2024-02-05T06:40:56.230940+0000","last_became_active":"2024-02-05T06:40:32.107529+0000","last_became_peered":"2024-02-05T06:40:32.107529+0000","last_unstale":"2024-02-05T06:40:56.230940+0000","last_undegraded":"2024-02-05T06:40:56.230940+0000","last_fullsized":"2024-02-05T06:40:56.230940+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":4054,"log_dups_size":0,"ondisk_log_size":4054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:05.047824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265682944,"num_objects":4054,"num_object_clones":0,"num_object_copies":12162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4054,"num_write_kb":259456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1894,"num_bytes_recovered":124125184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":189,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.055584+0000","last_change":"2024-02-05T06:40:08.607944+0000","last_active":"2024-02-05T06:40:56.055584+0000","last_peered":"2024-02-05T06:40:56.055584+0000","last_clean":"2024-02-05T06:40:56.055584+0000","last_became_active":"2024-02-05T06:40:08.607210+0000","last_became_peered":"2024-02-05T06:40:08.607210+0000","last_unstale":"2024-02-05T06:40:56.055584+0000","last_undegraded":"2024-02-05T06:40:56.055584+0000","last_fullsized":"2024-02-05T06:40:56.055584+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:38:17.284669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:38:17.284669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":474,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.055394+0000","last_change":"2024-02-05T06:40:30.081724+0000","last_active":"2024-02-05T06:40:56.055394+0000","last_peered":"2024-02-05T06:40:56.055394+0000","last_clean":"2024-02-05T06:40:56.055394+0000","last_became_active":"2024-02-05T06:40:30.081178+0000","last_became_peered":"2024-02-05T06:40:30.081178+0000","last_unstale":"2024-02-05T06:40:56.055394+0000","last_undegraded":"2024-02-05T06:40:56.055394+0000","last_fullsized":"2024-02-05T06:40:56.055394+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:39:08.974109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:39:08.974109+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":357,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.063628+0000","last_change":"2024-02-05T06:40:08.591738+0000","last_active":"2024-02-05T06:40:56.063628+0000","last_peered":"2024-02-05T06:40:56.063628+0000","last_clean":"2024-02-05T06:40:56.063628+0000","last_became_active":"2024-02-05T06:40:08.591426+0000","last_became_peered":"2024-02-05T06:40:08.591426+0000","last_unstale":"2024-02-05T06:40:56.063628+0000","last_undegraded":"2024-02-05T06:40:56.063628+0000","last_fullsized":"2024-02-05T06:40:56.063628+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:38:09.201827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:38:09.201827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"208'20","reported_seq":564,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.057542+0000","last_change":"2024-02-05T06:40:27.080702+0000","last_active":"2024-02-05T06:40:56.057542+0000","last_peered":"2024-02-05T06:40:56.057542+0000","last_clean":"2024-02-05T06:40:56.057542+0000","last_became_active":"2024-02-05T06:40:27.061111+0000","last_became_peered":"2024-02-05T06:40:27.061111+0000","last_unstale":"2024-02-05T06:40:56.057542+0000","last_undegraded":"2024-02-05T06:40:56.057542+0000","last_fullsized":"2024-02-05T06:40:56.057542+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'19","last_scrub_stamp":"2024-02-05T06:39:09.538463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:09.538463+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":53,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.058056+0000","last_change":"2024-02-05T06:40:27.186122+0000","last_active":"2024-02-05T06:40:56.058056+0000","last_peered":"2024-02-05T06:40:56.058056+0000","last_clean":"2024-02-05T06:40:56.058056+0000","last_became_active":"2024-02-05T06:40:27.185780+0000","last_became_peered":"2024-02-05T06:40:27.185780+0000","last_unstale":"2024-02-05T06:40:56.058056+0000","last_undegraded":"2024-02-05T06:40:56.058056+0000","last_fullsized":"2024-02-05T06:40:56.058056+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:31.636407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:37:31.636407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"226'4009","reported_seq":7987,"reported_epoch":227,"state":"peering","last_fresh":"2024-02-05T06:40:56.056512+0000","last_change":"2024-02-05T06:40:56.056512+0000","last_active":"2024-02-05T06:40:56.056512+0000","last_peered":"2024-02-05T06:40:56.049542+0000","last_clean":"2024-02-05T06:40:07.559155+0000","last_became_active":"2024-02-05T06:40:09.777966+0000","last_became_peered":"2024-02-05T06:40:09.777966+0000","last_unstale":"2024-02-05T06:40:56.056512+0000","last_undegraded":"2024-02-05T06:40:56.056512+0000","last_fullsized":"2024-02-05T06:40:56.056512+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":4009,"log_dups_size":0,"ondisk_log_size":4009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262733824,"num_objects":4009,"num_object_clones":0,"num_object_copies":12027,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4009,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4009,"num_write_kb":256576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1941,"num_bytes_recovered":127205376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":514,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.054442+0000","last_change":"2024-02-05T06:39:24.339609+0000","last_active":"2024-02-05T06:40:56.054442+0000","last_peered":"2024-02-05T06:40:56.054442+0000","last_clean":"2024-02-05T06:40:56.054442+0000","last_became_active":"2024-02-05T06:37:37.137754+0000","last_became_peered":"2024-02-05T06:37:37.137754+0000","last_unstale":"2024-02-05T06:40:56.054442+0000","last_undegraded":"2024-02-05T06:40:56.054442+0000","last_fullsized":"2024-02-05T06:40:56.054442+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:39:24.339522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:39:24.339522+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:00.317803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":353,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.052362+0000","last_change":"2024-02-05T06:40:08.605617+0000","last_active":"2024-02-05T06:40:56.052362+0000","last_peered":"2024-02-05T06:40:56.052362+0000","last_clean":"2024-02-05T06:40:56.052362+0000","last_became_active":"2024-02-05T06:40:08.605015+0000","last_became_peered":"2024-02-05T06:40:08.605015+0000","last_unstale":"2024-02-05T06:40:56.052362+0000","last_undegraded":"2024-02-05T06:40:56.052362+0000","last_fullsized":"2024-02-05T06:40:56.052362+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:10.411120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:10.411120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:00.317771+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":364,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.049140+0000","last_change":"2024-02-05T06:36:54.973030+0000","last_active":"2024-02-05T06:40:56.049140+0000","last_peered":"2024-02-05T06:40:56.049140+0000","last_clean":"2024-02-05T06:40:56.049140+0000","last_became_active":"2024-02-05T06:36:54.972264+0000","last_became_peered":"2024-02-05T06:36:54.972264+0000","last_unstale":"2024-02-05T06:40:56.049140+0000","last_undegraded":"2024-02-05T06:40:56.049140+0000","last_fullsized":"2024-02-05T06:40:56.049140+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:24.213373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:24.213373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":87,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.055390+0000","last_change":"2024-02-05T06:40:27.193820+0000","last_active":"2024-02-05T06:40:56.055390+0000","last_peered":"2024-02-05T06:40:56.055390+0000","last_clean":"2024-02-05T06:40:56.055390+0000","last_became_active":"2024-02-05T06:40:27.071509+0000","last_became_peered":"2024-02-05T06:40:27.071509+0000","last_unstale":"2024-02-05T06:40:56.055390+0000","last_undegraded":"2024-02-05T06:40:56.055390+0000","last_fullsized":"2024-02-05T06:40:56.055390+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T06:40:03.205189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:40:03.205189+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:32.734819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074151600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":36,"num_read_kb":31,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":366,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.049064+0000","last_change":"2024-02-05T06:36:29.231478+0000","last_active":"2024-02-05T06:40:56.049064+0000","last_peered":"2024-02-05T06:40:56.049064+0000","last_clean":"2024-02-05T06:40:56.049064+0000","last_became_active":"2024-02-05T06:33:31.564604+0000","last_became_peered":"2024-02-05T06:33:31.564604+0000","last_unstale":"2024-02-05T06:40:56.049064+0000","last_undegraded":"2024-02-05T06:40:56.049064+0000","last_fullsized":"2024-02-05T06:40:56.049064+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:36:29.231352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:29.231352+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":74,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.057742+0000","last_change":"2024-02-05T06:40:08.590795+0000","last_active":"2024-02-05T06:40:56.057742+0000","last_peered":"2024-02-05T06:40:56.057742+0000","last_clean":"2024-02-05T06:40:56.057742+0000","last_became_active":"2024-02-05T06:40:08.589927+0000","last_became_peered":"2024-02-05T06:40:08.589927+0000","last_unstale":"2024-02-05T06:40:56.057742+0000","last_undegraded":"2024-02-05T06:40:56.057742+0000","last_fullsized":"2024-02-05T06:40:56.057742+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:40.273584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:37:40.273584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:00.317597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.f","version":"227'4043","reported_seq":4136,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.294674+0000","last_change":"2024-02-05T06:39:42.682421+0000","last_active":"2024-02-05T06:40:56.294674+0000","last_peered":"2024-02-05T06:40:56.294674+0000","last_clean":"2024-02-05T06:40:56.294674+0000","last_became_active":"2024-02-05T06:39:42.682109+0000","last_became_peered":"2024-02-05T06:39:42.682109+0000","last_unstale":"2024-02-05T06:40:56.294674+0000","last_undegraded":"2024-02-05T06:40:56.294674+0000","last_fullsized":"2024-02-05T06:40:56.294674+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":4043,"log_dups_size":0,"ondisk_log_size":4043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264962048,"num_objects":4043,"num_object_clones":0,"num_object_copies":12129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4043,"num_write_kb":258752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":519,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.057479+0000","last_change":"2024-02-05T06:39:12.528803+0000","last_active":"2024-02-05T06:40:56.057479+0000","last_peered":"2024-02-05T06:40:56.057479+0000","last_clean":"2024-02-05T06:40:56.057479+0000","last_became_active":"2024-02-05T06:37:37.140011+0000","last_became_peered":"2024-02-05T06:37:37.140011+0000","last_unstale":"2024-02-05T06:40:56.057479+0000","last_undegraded":"2024-02-05T06:40:56.057479+0000","last_fullsized":"2024-02-05T06:40:56.057479+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:39:12.528724+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:38.778568+0000","last_clean_scrub_stamp":"2024-02-05T06:39:12.528724+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":347,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.056523+0000","last_change":"2024-02-05T06:39:22.330018+0000","last_active":"2024-02-05T06:40:56.056523+0000","last_peered":"2024-02-05T06:40:56.056523+0000","last_clean":"2024-02-05T06:40:56.056523+0000","last_became_active":"2024-02-05T06:33:35.664858+0000","last_became_peered":"2024-02-05T06:33:35.664858+0000","last_unstale":"2024-02-05T06:40:56.056523+0000","last_undegraded":"2024-02-05T06:40:56.056523+0000","last_fullsized":"2024-02-05T06:40:56.056523+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:22.329914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:22.329914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:00.317859+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"66'1","reported_seq":383,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.055295+0000","last_change":"2024-02-05T06:40:08.656245+0000","last_active":"2024-02-05T06:40:56.055295+0000","last_peered":"2024-02-05T06:40:56.055295+0000","last_clean":"2024-02-05T06:40:56.055295+0000","last_became_active":"2024-02-05T06:40:08.594925+0000","last_became_peered":"2024-02-05T06:40:08.594925+0000","last_unstale":"2024-02-05T06:40:56.055295+0000","last_undegraded":"2024-02-05T06:40:56.055295+0000","last_fullsized":"2024-02-05T06:40:56.055295+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:39:34.241779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:39:34.241779+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:00.318052+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'19","reported_seq":456,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.049139+0000","last_change":"2024-02-05T06:40:27.127864+0000","last_active":"2024-02-05T06:40:56.049139+0000","last_peered":"2024-02-05T06:40:56.049139+0000","last_clean":"2024-02-05T06:40:56.049139+0000","last_became_active":"2024-02-05T06:40:27.073899+0000","last_became_peered":"2024-02-05T06:40:27.073899+0000","last_unstale":"2024-02-05T06:40:56.049139+0000","last_undegraded":"2024-02-05T06:40:56.049139+0000","last_fullsized":"2024-02-05T06:40:56.049139+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-02-05T06:39:19.843545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:19.843545+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":356,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.056932+0000","last_change":"2024-02-05T06:40:08.591742+0000","last_active":"2024-02-05T06:40:56.056932+0000","last_peered":"2024-02-05T06:40:56.056932+0000","last_clean":"2024-02-05T06:40:56.056932+0000","last_became_active":"2024-02-05T06:40:08.591312+0000","last_became_peered":"2024-02-05T06:40:08.591312+0000","last_unstale":"2024-02-05T06:40:56.056932+0000","last_undegraded":"2024-02-05T06:40:56.056932+0000","last_fullsized":"2024-02-05T06:40:56.056932+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:32.449390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:32.449390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"227'4194","reported_seq":4288,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.300786+0000","last_change":"2024-02-05T06:39:42.684749+0000","last_active":"2024-02-05T06:40:56.300786+0000","last_peered":"2024-02-05T06:40:56.300786+0000","last_clean":"2024-02-05T06:40:56.300786+0000","last_became_active":"2024-02-05T06:39:42.684454+0000","last_became_peered":"2024-02-05T06:39:42.684454+0000","last_unstale":"2024-02-05T06:40:56.300786+0000","last_undegraded":"2024-02-05T06:40:56.300786+0000","last_fullsized":"2024-02-05T06:40:56.300786+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":4194,"log_dups_size":0,"ondisk_log_size":4194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:40:58.473086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274857984,"num_objects":4194,"num_object_clones":0,"num_object_copies":12582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4194,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4194,"num_write_kb":268416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":546,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.057611+0000","last_change":"2024-02-05T06:40:30.085803+0000","last_active":"2024-02-05T06:40:56.057611+0000","last_peered":"2024-02-05T06:40:56.057611+0000","last_clean":"2024-02-05T06:40:56.057611+0000","last_became_active":"2024-02-05T06:40:30.085438+0000","last_became_peered":"2024-02-05T06:40:30.085438+0000","last_unstale":"2024-02-05T06:40:56.057611+0000","last_undegraded":"2024-02-05T06:40:56.057611+0000","last_fullsized":"2024-02-05T06:40:56.057611+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:39:30.440075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:39:30.440075+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":340,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.048428+0000","last_change":"2024-02-05T06:39:30.865329+0000","last_active":"2024-02-05T06:40:56.048428+0000","last_peered":"2024-02-05T06:40:56.048428+0000","last_clean":"2024-02-05T06:40:56.048428+0000","last_became_active":"2024-02-05T06:33:35.600485+0000","last_became_peered":"2024-02-05T06:33:35.600485+0000","last_unstale":"2024-02-05T06:40:56.048428+0000","last_undegraded":"2024-02-05T06:40:56.048428+0000","last_fullsized":"2024-02-05T06:40:56.048428+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:30.865200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:30.865200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":75,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.050087+0000","last_change":"2024-02-05T06:40:08.582459+0000","last_active":"2024-02-05T06:40:56.050087+0000","last_peered":"2024-02-05T06:40:56.050087+0000","last_clean":"2024-02-05T06:40:56.050087+0000","last_became_active":"2024-02-05T06:40:08.582079+0000","last_became_peered":"2024-02-05T06:40:08.582079+0000","last_unstale":"2024-02-05T06:40:56.050087+0000","last_undegraded":"2024-02-05T06:40:56.050087+0000","last_fullsized":"2024-02-05T06:40:56.050087+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:28.353956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:28.353956+0000","last_clean_scrub_stamp":"2024-02-05T06:39:28.353956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"73'10","reported_seq":424,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.057096+0000","last_change":"2024-02-05T06:40:08.667772+0000","last_active":"2024-02-05T06:40:56.057096+0000","last_peered":"2024-02-05T06:40:56.057096+0000","last_clean":"2024-02-05T06:40:56.057096+0000","last_became_active":"2024-02-05T06:40:08.595642+0000","last_became_peered":"2024-02-05T06:40:08.595642+0000","last_unstale":"2024-02-05T06:40:56.057096+0000","last_undegraded":"2024-02-05T06:40:56.057096+0000","last_fullsized":"2024-02-05T06:40:56.057096+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-02-05T06:39:25.306103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:25.306103+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:00.317832+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"197'9","reported_seq":452,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.048753+0000","last_change":"2024-02-05T06:40:08.656439+0000","last_active":"2024-02-05T06:40:56.048753+0000","last_peered":"2024-02-05T06:40:56.048753+0000","last_clean":"2024-02-05T06:40:56.048753+0000","last_became_active":"2024-02-05T06:40:08.600932+0000","last_became_peered":"2024-02-05T06:40:08.600932+0000","last_unstale":"2024-02-05T06:40:56.048753+0000","last_undegraded":"2024-02-05T06:40:56.048753+0000","last_fullsized":"2024-02-05T06:40:56.048753+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:36:32.174396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:32.174396+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"227'3952","reported_seq":10157,"reported_epoch":227,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-02-05T06:40:56.493165+0000","last_change":"2024-02-05T06:40:27.204722+0000","last_active":"2024-02-05T06:40:56.493165+0000","last_peered":"2024-02-05T06:40:56.493165+0000","last_clean":"2024-02-05T06:40:07.538450+0000","last_became_active":"2024-02-05T06:40:27.189477+0000","last_became_peered":"2024-02-05T06:40:27.189477+0000","last_unstale":"2024-02-05T06:40:56.493165+0000","last_undegraded":"2024-02-05T06:40:27.091019+0000","last_fullsized":"2024-02-05T06:40:27.056046+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":3952,"log_dups_size":0,"ondisk_log_size":3952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":258998272,"num_objects":3952,"num_object_clones":0,"num_object_copies":11856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":443,"num_objects_unfound":0,"num_objects_dirty":3952,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3952,"num_write_kb":252928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3055,"num_bytes_recovered":200212480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[2,6],"avail_no_missing":["2","5","6"],"object_location_counts":[{"shards":"1,2,6","objects":443},{"shards":"2,6","objects":3508}],"blocked_by":[],"up_primary":7,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":544,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.057535+0000","last_change":"2024-02-05T06:40:48.371703+0000","last_active":"2024-02-05T06:40:56.057535+0000","last_peered":"2024-02-05T06:40:56.057535+0000","last_clean":"2024-02-05T06:40:56.057535+0000","last_became_active":"2024-02-05T06:40:48.370713+0000","last_became_peered":"2024-02-05T06:40:48.370713+0000","last_unstale":"2024-02-05T06:40:56.057535+0000","last_undegraded":"2024-02-05T06:40:56.057535+0000","last_fullsized":"2024-02-05T06:40:56.057535+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:38:10.176507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:38:10.176507+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":53,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.055509+0000","last_change":"2024-02-05T06:40:27.099496+0000","last_active":"2024-02-05T06:40:56.055509+0000","last_peered":"2024-02-05T06:40:56.055509+0000","last_clean":"2024-02-05T06:40:56.055509+0000","last_became_active":"2024-02-05T06:40:27.099168+0000","last_became_peered":"2024-02-05T06:40:27.099168+0000","last_unstale":"2024-02-05T06:40:56.055509+0000","last_undegraded":"2024-02-05T06:40:56.055509+0000","last_fullsized":"2024-02-05T06:40:56.055509+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:40:06.734454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:40:06.734454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:18.186414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"95'2","reported_seq":374,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.053893+0000","last_change":"2024-02-05T06:39:06.955815+0000","last_active":"2024-02-05T06:40:56.053893+0000","last_peered":"2024-02-05T06:40:56.053893+0000","last_clean":"2024-02-05T06:40:56.053893+0000","last_became_active":"2024-02-05T06:36:54.975093+0000","last_became_peered":"2024-02-05T06:36:54.975093+0000","last_unstale":"2024-02-05T06:40:56.053893+0000","last_undegraded":"2024-02-05T06:40:56.053893+0000","last_fullsized":"2024-02-05T06:40:56.053893+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-05T06:39:06.955629+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-05T06:35:25.616708+0000","last_clean_scrub_stamp":"2024-02-05T06:39:06.955629+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'14","reported_seq":274,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.038461+0000","last_change":"2024-02-05T06:40:47.627775+0000","last_active":"2024-02-05T06:40:56.038461+0000","last_peered":"2024-02-05T06:40:56.038461+0000","last_clean":"2024-02-05T06:40:56.038461+0000","last_became_active":"2024-02-05T06:40:08.590366+0000","last_became_peered":"2024-02-05T06:40:08.590366+0000","last_unstale":"2024-02-05T06:40:56.038461+0000","last_undegraded":"2024-02-05T06:40:56.038461+0000","last_fullsized":"2024-02-05T06:40:56.038461+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'14","last_scrub_stamp":"2024-02-05T06:40:47.627644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:40:47.627644+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:05.903986+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"119'10","reported_seq":463,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.062646+0000","last_change":"2024-02-05T06:38:23.177410+0000","last_active":"2024-02-05T06:40:56.062646+0000","last_peered":"2024-02-05T06:40:56.062646+0000","last_clean":"2024-02-05T06:40:56.062646+0000","last_became_active":"2024-02-05T06:36:54.954819+0000","last_became_peered":"2024-02-05T06:36:54.954819+0000","last_unstale":"2024-02-05T06:40:56.062646+0000","last_undegraded":"2024-02-05T06:40:56.062646+0000","last_fullsized":"2024-02-05T06:40:56.062646+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-02-05T06:38:23.177286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:38:23.177286+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"227'4064","reported_seq":7361,"reported_epoch":227,"state":"active+recovering+undersized+remapped","last_fresh":"2024-02-05T06:40:56.822147+0000","last_change":"2024-02-05T06:40:46.468070+0000","last_active":"2024-02-05T06:40:56.822147+0000","last_peered":"2024-02-05T06:40:56.822147+0000","last_clean":"2024-02-05T06:40:07.546191+0000","last_became_active":"2024-02-05T06:40:27.192989+0000","last_became_peered":"2024-02-05T06:40:27.192989+0000","last_unstale":"2024-02-05T06:40:56.822147+0000","last_undegraded":"2024-02-05T06:40:56.822147+0000","last_fullsized":"2024-02-05T06:40:27.049428+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":4064,"log_dups_size":0,"ondisk_log_size":4064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266338304,"num_objects":4064,"num_object_clones":0,"num_object_copies":12192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2273,"num_objects_unfound":0,"num_objects_dirty":4064,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4064,"num_write_kb":260096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1597,"num_bytes_recovered":104660992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3","objects":1791},{"shards":"0,3,4","objects":901},{"shards":"0,3,4,5","objects":675},{"shards":"0,3,5","objects":697}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":506,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.039371+0000","last_change":"2024-02-05T06:38:10.076368+0000","last_active":"2024-02-05T06:40:56.039371+0000","last_peered":"2024-02-05T06:40:56.039371+0000","last_clean":"2024-02-05T06:40:56.039371+0000","last_became_active":"2024-02-05T06:37:37.135073+0000","last_became_peered":"2024-02-05T06:37:37.135073+0000","last_unstale":"2024-02-05T06:40:56.039371+0000","last_undegraded":"2024-02-05T06:40:56.039371+0000","last_fullsized":"2024-02-05T06:40:56.039371+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:38:10.076258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:38:10.076258+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:40:56.561244+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":236,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.041698+0000","last_change":"2024-02-05T06:39:24.090774+0000","last_active":"2024-02-05T06:40:56.041698+0000","last_peered":"2024-02-05T06:40:56.041698+0000","last_clean":"2024-02-05T06:40:56.041698+0000","last_became_active":"2024-02-05T06:36:54.958644+0000","last_became_peered":"2024-02-05T06:36:54.958644+0000","last_unstale":"2024-02-05T06:40:56.041698+0000","last_undegraded":"2024-02-05T06:40:56.041698+0000","last_fullsized":"2024-02-05T06:40:56.041698+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:24.090670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:24.090670+0000","last_clean_scrub_stamp":"2024-02-05T06:39:24.090670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":369,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.055499+0000","last_change":"2024-02-05T06:40:27.094061+0000","last_active":"2024-02-05T06:40:56.055499+0000","last_peered":"2024-02-05T06:40:56.055499+0000","last_clean":"2024-02-05T06:40:56.055499+0000","last_became_active":"2024-02-05T06:40:27.093739+0000","last_became_peered":"2024-02-05T06:40:27.093739+0000","last_unstale":"2024-02-05T06:40:56.055499+0000","last_undegraded":"2024-02-05T06:40:56.055499+0000","last_fullsized":"2024-02-05T06:40:56.055499+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:38:28.150695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:38:28.150695+0000","last_clean_scrub_stamp":"2024-02-05T06:38:28.150695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"73'18","reported_seq":427,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.062562+0000","last_change":"2024-02-05T06:40:46.408120+0000","last_active":"2024-02-05T06:40:56.062562+0000","last_peered":"2024-02-05T06:40:56.062562+0000","last_clean":"2024-02-05T06:40:56.062562+0000","last_became_active":"2024-02-05T06:40:27.277747+0000","last_became_peered":"2024-02-05T06:40:27.277747+0000","last_unstale":"2024-02-05T06:40:56.062562+0000","last_undegraded":"2024-02-05T06:40:56.062562+0000","last_fullsized":"2024-02-05T06:40:56.062562+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-02-05T06:39:12.382954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:12.382954+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":256,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.038570+0000","last_change":"2024-02-05T06:40:27.074218+0000","last_active":"2024-02-05T06:40:56.038570+0000","last_peered":"2024-02-05T06:40:56.038570+0000","last_clean":"2024-02-05T06:40:56.038570+0000","last_became_active":"2024-02-05T06:40:27.073889+0000","last_became_peered":"2024-02-05T06:40:27.073889+0000","last_unstale":"2024-02-05T06:40:56.038570+0000","last_undegraded":"2024-02-05T06:40:56.038570+0000","last_fullsized":"2024-02-05T06:40:56.038570+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:40:02.899805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:40:02.899805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:03.745839+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"227'4074","reported_seq":8032,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.293511+0000","last_change":"2024-02-05T06:40:37.438903+0000","last_active":"2024-02-05T06:40:56.293511+0000","last_peered":"2024-02-05T06:40:56.293511+0000","last_clean":"2024-02-05T06:40:56.293511+0000","last_became_active":"2024-02-05T06:40:37.405461+0000","last_became_peered":"2024-02-05T06:40:37.405461+0000","last_unstale":"2024-02-05T06:40:56.293511+0000","last_undegraded":"2024-02-05T06:40:56.293511+0000","last_fullsized":"2024-02-05T06:40:56.293511+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":4074,"log_dups_size":0,"ondisk_log_size":4074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:00.322403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266993664,"num_objects":4074,"num_object_clones":0,"num_object_copies":12222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4074,"num_write_kb":260736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1925,"num_bytes_recovered":126156800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":376,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.060181+0000","last_change":"2024-02-05T06:40:11.044717+0000","last_active":"2024-02-05T06:40:56.060181+0000","last_peered":"2024-02-05T06:40:56.060181+0000","last_clean":"2024-02-05T06:40:56.060181+0000","last_became_active":"2024-02-05T06:40:11.042179+0000","last_became_peered":"2024-02-05T06:40:11.042179+0000","last_unstale":"2024-02-05T06:40:56.060181+0000","last_undegraded":"2024-02-05T06:40:56.060181+0000","last_fullsized":"2024-02-05T06:40:56.060181+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:40:03.329947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:40:03.329947+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:00.592646+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":26,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":74,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.052804+0000","last_change":"2024-02-05T06:40:08.595889+0000","last_active":"2024-02-05T06:40:56.052804+0000","last_peered":"2024-02-05T06:40:56.052804+0000","last_clean":"2024-02-05T06:40:56.052804+0000","last_became_active":"2024-02-05T06:40:08.595517+0000","last_became_peered":"2024-02-05T06:40:08.595517+0000","last_unstale":"2024-02-05T06:40:56.052804+0000","last_undegraded":"2024-02-05T06:40:56.052804+0000","last_fullsized":"2024-02-05T06:40:56.052804+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:11.464973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:11.464973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:00.317954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"197'11","reported_seq":300,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.039174+0000","last_change":"2024-02-05T06:40:08.601333+0000","last_active":"2024-02-05T06:40:56.039174+0000","last_peered":"2024-02-05T06:40:56.039174+0000","last_clean":"2024-02-05T06:40:56.039174+0000","last_became_active":"2024-02-05T06:40:08.585067+0000","last_became_peered":"2024-02-05T06:40:08.585067+0000","last_unstale":"2024-02-05T06:40:56.039174+0000","last_undegraded":"2024-02-05T06:40:56.039174+0000","last_fullsized":"2024-02-05T06:40:56.039174+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-02-05T06:40:06.921141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:40:06.921141+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:08.150025+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":363,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.062489+0000","last_change":"2024-02-05T06:40:27.214932+0000","last_active":"2024-02-05T06:40:56.062489+0000","last_peered":"2024-02-05T06:40:56.062489+0000","last_clean":"2024-02-05T06:40:56.062489+0000","last_became_active":"2024-02-05T06:40:27.213439+0000","last_became_peered":"2024-02-05T06:40:27.213439+0000","last_unstale":"2024-02-05T06:40:56.062489+0000","last_undegraded":"2024-02-05T06:40:56.062489+0000","last_fullsized":"2024-02-05T06:40:56.062489+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:23.372627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:39:23.372627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"73'15","reported_seq":133,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.050993+0000","last_change":"2024-02-05T06:39:13.334762+0000","last_active":"2024-02-05T06:40:56.050993+0000","last_peered":"2024-02-05T06:40:56.050993+0000","last_clean":"2024-02-05T06:40:56.050993+0000","last_became_active":"2024-02-05T06:39:13.257978+0000","last_became_peered":"2024-02-05T06:39:13.257978+0000","last_unstale":"2024-02-05T06:40:56.050993+0000","last_undegraded":"2024-02-05T06:40:56.050993+0000","last_fullsized":"2024-02-05T06:40:56.050993+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-02-05T06:39:11.376185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:11.376185+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2","version":"227'4074","reported_seq":7900,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.191474+0000","last_change":"2024-02-05T06:40:34.130876+0000","last_active":"2024-02-05T06:40:56.191474+0000","last_peered":"2024-02-05T06:40:56.191474+0000","last_clean":"2024-02-05T06:40:56.191474+0000","last_became_active":"2024-02-05T06:40:34.127148+0000","last_became_peered":"2024-02-05T06:40:34.127148+0000","last_unstale":"2024-02-05T06:40:56.191474+0000","last_undegraded":"2024-02-05T06:40:56.191474+0000","last_fullsized":"2024-02-05T06:40:56.191474+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":4074,"log_dups_size":0,"ondisk_log_size":4074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:02.452284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266993664,"num_objects":4074,"num_object_clones":0,"num_object_copies":12222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4074,"num_write_kb":260736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1861,"num_bytes_recovered":121962496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":351,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.057668+0000","last_change":"2024-02-05T06:40:27.078125+0000","last_active":"2024-02-05T06:40:56.057668+0000","last_peered":"2024-02-05T06:40:56.057668+0000","last_clean":"2024-02-05T06:40:56.057668+0000","last_became_active":"2024-02-05T06:40:27.077570+0000","last_became_peered":"2024-02-05T06:40:27.077570+0000","last_unstale":"2024-02-05T06:40:56.057668+0000","last_undegraded":"2024-02-05T06:40:56.057668+0000","last_fullsized":"2024-02-05T06:40:56.057668+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:18.402553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:20.042534+0000","last_clean_scrub_stamp":"2024-02-05T06:39:18.402553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":201,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.038990+0000","last_change":"2024-02-05T06:40:08.583193+0000","last_active":"2024-02-05T06:40:56.038990+0000","last_peered":"2024-02-05T06:40:56.038990+0000","last_clean":"2024-02-05T06:40:56.038990+0000","last_became_active":"2024-02-05T06:40:08.582647+0000","last_became_peered":"2024-02-05T06:40:08.582647+0000","last_unstale":"2024-02-05T06:40:56.038990+0000","last_undegraded":"2024-02-05T06:40:56.038990+0000","last_fullsized":"2024-02-05T06:40:56.038990+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:18.379775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:18.379775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:40:56.561475+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":71,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.054797+0000","last_change":"2024-02-05T06:40:27.087535+0000","last_active":"2024-02-05T06:40:56.054797+0000","last_peered":"2024-02-05T06:40:56.054797+0000","last_clean":"2024-02-05T06:40:56.054797+0000","last_became_active":"2024-02-05T06:40:27.086808+0000","last_became_peered":"2024-02-05T06:40:27.086808+0000","last_unstale":"2024-02-05T06:40:56.054797+0000","last_undegraded":"2024-02-05T06:40:56.054797+0000","last_fullsized":"2024-02-05T06:40:56.054797+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:38:14.268605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:38:14.268605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:00.317656+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"115'16","reported_seq":240,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.038988+0000","last_change":"2024-02-05T06:36:54.992244+0000","last_active":"2024-02-05T06:40:56.038988+0000","last_peered":"2024-02-05T06:40:56.038988+0000","last_clean":"2024-02-05T06:40:56.038988+0000","last_became_active":"2024-02-05T06:36:54.958582+0000","last_became_peered":"2024-02-05T06:36:54.958582+0000","last_unstale":"2024-02-05T06:40:56.038988+0000","last_undegraded":"2024-02-05T06:40:56.038988+0000","last_fullsized":"2024-02-05T06:40:56.038988+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'16","last_scrub_stamp":"2024-02-05T06:36:34.793243+0000","last_deep_scrub":"115'16","last_deep_scrub_stamp":"2024-02-05T06:36:34.793243+0000","last_clean_scrub_stamp":"2024-02-05T06:36:34.793243+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:40:56.561195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"227'4020","reported_seq":4114,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.290473+0000","last_change":"2024-02-05T06:39:42.682964+0000","last_active":"2024-02-05T06:40:56.290473+0000","last_peered":"2024-02-05T06:40:56.290473+0000","last_clean":"2024-02-05T06:40:56.290473+0000","last_became_active":"2024-02-05T06:39:42.682661+0000","last_became_peered":"2024-02-05T06:39:42.682661+0000","last_unstale":"2024-02-05T06:40:56.290473+0000","last_undegraded":"2024-02-05T06:40:56.290473+0000","last_fullsized":"2024-02-05T06:40:56.290473+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":4020,"log_dups_size":0,"ondisk_log_size":4020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263454720,"num_objects":4020,"num_object_clones":0,"num_object_copies":12060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4020,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4020,"num_write_kb":257280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"83'128","reported_seq":380,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.058376+0000","last_change":"2024-02-05T06:40:11.077092+0000","last_active":"2024-02-05T06:40:56.058376+0000","last_peered":"2024-02-05T06:40:56.058376+0000","last_clean":"2024-02-05T06:40:56.058376+0000","last_became_active":"2024-02-05T06:40:11.067992+0000","last_became_peered":"2024-02-05T06:40:11.067992+0000","last_unstale":"2024-02-05T06:40:56.058376+0000","last_undegraded":"2024-02-05T06:40:56.058376+0000","last_fullsized":"2024-02-05T06:40:56.058376+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:37:39.307601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:37:39.307601+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:00.592669+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":423697,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":357,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.059275+0000","last_change":"2024-02-05T06:38:17.558520+0000","last_active":"2024-02-05T06:40:56.059275+0000","last_peered":"2024-02-05T06:40:56.059275+0000","last_clean":"2024-02-05T06:40:56.059275+0000","last_became_active":"2024-02-05T06:36:54.958805+0000","last_became_peered":"2024-02-05T06:36:54.958805+0000","last_unstale":"2024-02-05T06:40:56.059275+0000","last_undegraded":"2024-02-05T06:40:56.059275+0000","last_fullsized":"2024-02-05T06:40:56.059275+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:38:17.558441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:38:17.558441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":363,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.055390+0000","last_change":"2024-02-05T06:39:09.427061+0000","last_active":"2024-02-05T06:40:56.055390+0000","last_peered":"2024-02-05T06:40:56.055390+0000","last_clean":"2024-02-05T06:40:56.055390+0000","last_became_active":"2024-02-05T06:36:54.958082+0000","last_became_peered":"2024-02-05T06:36:54.958082+0000","last_unstale":"2024-02-05T06:40:56.055390+0000","last_undegraded":"2024-02-05T06:40:56.055390+0000","last_fullsized":"2024-02-05T06:40:56.055390+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:09.426862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:09.426862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:00.317887+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":376,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.057027+0000","last_change":"2024-02-05T06:39:35.429964+0000","last_active":"2024-02-05T06:40:56.057027+0000","last_peered":"2024-02-05T06:40:56.057027+0000","last_clean":"2024-02-05T06:40:56.057027+0000","last_became_active":"2024-02-05T06:39:13.240038+0000","last_became_peered":"2024-02-05T06:39:13.240038+0000","last_unstale":"2024-02-05T06:40:56.057027+0000","last_undegraded":"2024-02-05T06:40:56.057027+0000","last_fullsized":"2024-02-05T06:40:56.057027+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:35.429861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:39:35.429861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"95'25","reported_seq":254,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.039246+0000","last_change":"2024-02-05T06:39:31.037992+0000","last_active":"2024-02-05T06:40:56.039246+0000","last_peered":"2024-02-05T06:40:56.039246+0000","last_clean":"2024-02-05T06:40:56.039246+0000","last_became_active":"2024-02-05T06:36:54.961719+0000","last_became_peered":"2024-02-05T06:36:54.961719+0000","last_unstale":"2024-02-05T06:40:56.039246+0000","last_undegraded":"2024-02-05T06:40:56.039246+0000","last_fullsized":"2024-02-05T06:40:56.039246+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'25","last_scrub_stamp":"2024-02-05T06:39:31.037880+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-02-05T06:35:26.274944+0000","last_clean_scrub_stamp":"2024-02-05T06:39:31.037880+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:40:56.561460+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"227'4074","reported_seq":7899,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.277961+0000","last_change":"2024-02-05T06:40:44.347862+0000","last_active":"2024-02-05T06:40:56.277961+0000","last_peered":"2024-02-05T06:40:56.277961+0000","last_clean":"2024-02-05T06:40:56.277961+0000","last_became_active":"2024-02-05T06:40:44.342540+0000","last_became_peered":"2024-02-05T06:40:44.342540+0000","last_unstale":"2024-02-05T06:40:56.277961+0000","last_undegraded":"2024-02-05T06:40:56.277961+0000","last_fullsized":"2024-02-05T06:40:56.277961+0000","mapping_epoch":218,"log_start":"192'100","ondisk_log_start":"192'100","created":185,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":3974,"log_dups_size":0,"ondisk_log_size":3974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266993664,"num_objects":4074,"num_object_clones":0,"num_object_copies":12222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4074,"num_write_kb":260736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1861,"num_bytes_recovered":121962496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":463,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.055240+0000","last_change":"2024-02-05T06:40:30.095555+0000","last_active":"2024-02-05T06:40:56.055240+0000","last_peered":"2024-02-05T06:40:56.055240+0000","last_clean":"2024-02-05T06:40:56.055240+0000","last_became_active":"2024-02-05T06:40:30.095202+0000","last_became_peered":"2024-02-05T06:40:30.095202+0000","last_unstale":"2024-02-05T06:40:56.055240+0000","last_undegraded":"2024-02-05T06:40:56.055240+0000","last_fullsized":"2024-02-05T06:40:56.055240+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:38:20.178985+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:38:20.178985+0000","last_clean_scrub_stamp":"2024-02-05T06:38:20.178985+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:40:56.758261+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":236,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.038895+0000","last_change":"2024-02-05T06:39:30.057231+0000","last_active":"2024-02-05T06:40:56.038895+0000","last_peered":"2024-02-05T06:40:56.038895+0000","last_clean":"2024-02-05T06:40:56.038895+0000","last_became_active":"2024-02-05T06:36:54.961560+0000","last_became_peered":"2024-02-05T06:36:54.961560+0000","last_unstale":"2024-02-05T06:40:56.038895+0000","last_undegraded":"2024-02-05T06:40:56.038895+0000","last_fullsized":"2024-02-05T06:40:56.038895+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:30.057099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:30.057099+0000","last_clean_scrub_stamp":"2024-02-05T06:39:30.057099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:40:56.561507+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"208'12","reported_seq":478,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.059896+0000","last_change":"2024-02-05T06:40:27.207229+0000","last_active":"2024-02-05T06:40:56.059896+0000","last_peered":"2024-02-05T06:40:56.059896+0000","last_clean":"2024-02-05T06:40:56.059896+0000","last_became_active":"2024-02-05T06:40:27.102928+0000","last_became_peered":"2024-02-05T06:40:27.102928+0000","last_unstale":"2024-02-05T06:40:56.059896+0000","last_undegraded":"2024-02-05T06:40:56.059896+0000","last_fullsized":"2024-02-05T06:40:56.059896+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'9","last_scrub_stamp":"2024-02-05T06:37:47.417250+0000","last_deep_scrub":"92'9","last_deep_scrub_stamp":"2024-02-05T06:34:52.283780+0000","last_clean_scrub_stamp":"2024-02-05T06:37:47.417250+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":363,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.060347+0000","last_change":"2024-02-05T06:36:54.971857+0000","last_active":"2024-02-05T06:40:56.060347+0000","last_peered":"2024-02-05T06:40:56.060347+0000","last_clean":"2024-02-05T06:40:56.060347+0000","last_became_active":"2024-02-05T06:36:54.971304+0000","last_became_peered":"2024-02-05T06:36:54.971304+0000","last_unstale":"2024-02-05T06:40:56.060347+0000","last_undegraded":"2024-02-05T06:40:56.060347+0000","last_fullsized":"2024-02-05T06:40:56.060347+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:37.506830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:37.506830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"93'15","reported_seq":248,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-02-05T06:40:56.038834+0000","last_change":"2024-02-05T06:39:16.997591+0000","last_active":"2024-02-05T06:40:56.038834+0000","last_peered":"2024-02-05T06:40:56.038834+0000","last_clean":"2024-02-05T06:40:56.038834+0000","last_became_active":"2024-02-05T06:36:54.963620+0000","last_became_peered":"2024-02-05T06:36:54.963620+0000","last_unstale":"2024-02-05T06:40:56.038834+0000","last_undegraded":"2024-02-05T06:40:56.038834+0000","last_fullsized":"2024-02-05T06:40:56.038834+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'15","last_scrub_stamp":"2024-02-05T06:39:16.997494+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-02-05T06:35:04.584657+0000","last_clean_scrub_stamp":"2024-02-05T06:39:16.997494+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:40:56.561324+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":4250468352,"num_objects":64857,"num_object_clones":0,"num_object_copies":194571,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2774,"num_objects_misplaced":4396,"num_objects_unfound":0,"num_objects_dirty":64857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":64857,"num_write_kb":4150848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20415,"num_bytes_recovered":1337917440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13672120320,"data_stored":13672120320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64457,"ondisk_log_size":64457,"up":48,"acting":44,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":31,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":542359,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":108,"num_write":65,"num_write_kb":877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1889765,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":2742,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":5242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":684,"num_read_kb":441,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":339,"num_bytes_recovered":7354,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":483328,"data_stored":12784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":455,"ondisk_log_size":455,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103261,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":988268,"kb_used_data":804384,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":92756884,"statfs":{"total":95995035648,"available":94983049216,"internally_reserved":0,"allocated":823689216,"data_stored":822827722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":188283338},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":2.2149999999999999,"15min":1.115},"min":{"1min":0.24399999999999999,"5min":0.219,"15min":0.219},"max":{"1min":1.6379999999999999,"5min":68.921000000000006,"15min":68.921000000000006},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.1830000000000001,"5min":2.423,"15min":1.208},"min":{"1min":0.28799999999999998,"5min":0.248,"15min":0.248},"max":{"1min":3.3010000000000002,"5min":69.616,"15min":69.616},"last":0.51000000000000001}]},{"osd":1,"last update":"Mon Feb 5 06:40:39 2024","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":1.931,"15min":1.018},"min":{"1min":0.32400000000000001,"5min":0.254,"15min":0.23599999999999999},"max":{"1min":9.4939999999999998,"5min":89.231999999999999,"15min":89.231999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":2.7810000000000001,"5min":2.2149999999999999,"15min":1.137},"min":{"1min":0.33000000000000002,"5min":0.26300000000000001,"15min":0.26100000000000001},"max":{"1min":14.824,"5min":87.995999999999995,"15min":87.995999999999995},"last":0.61699999999999999}]},{"osd":2,"last update":"Mon Feb 5 06:40:39 2024","interfaces":[{"interface":"back","average":{"1min":1.8460000000000001,"5min":1.675,"15min":0.93999999999999995},"min":{"1min":0.28399999999999997,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":11.991,"5min":53.947000000000003,"15min":53.947000000000003},"last":0.373},{"interface":"front","average":{"1min":0.999,"5min":1.6120000000000001,"15min":0.91700000000000004},"min":{"1min":0.30499999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.8149999999999999,"5min":53.299999999999997,"15min":53.299999999999997},"last":3.5739999999999998}]},{"osd":3,"last update":"Mon Feb 5 06:40:39 2024","interfaces":[{"interface":"back","average":{"1min":3.2349999999999999,"5min":3.2130000000000001,"15min":1.47},"min":{"1min":0.36799999999999999,"5min":0.316,"15min":0.316},"max":{"1min":19.513000000000002,"5min":89.481999999999999,"15min":89.481999999999999},"last":1.847},{"interface":"front","average":{"1min":1.405,"5min":3.077,"15min":1.429},"min":{"1min":0.307,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":12.034000000000001,"5min":92.293999999999997,"15min":92.293999999999997},"last":4.8390000000000004}]},{"osd":4,"last update":"Mon Feb 5 06:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.53700000000000003,"15min":0.47499999999999998},"min":{"1min":0.26400000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78300000000000003,"5min":3.5409999999999999,"15min":3.5409999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.48299999999999998,"15min":0.49399999999999999},"min":{"1min":0.29099999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.73599999999999999,"5min":2.258,"15min":2.258},"last":0.40799999999999997}]},{"osd":5,"last update":"Mon Feb 5 06:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.52200000000000002,"15min":0.51100000000000001},"min":{"1min":0.312,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.3220000000000001,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59199999999999997,"15min":0.56200000000000006},"min":{"1min":0.35299999999999998,"5min":0.252,"15min":0.252},"max":{"1min":2.0529999999999999,"5min":3.5990000000000002,"15min":3.5990000000000002},"last":0.57699999999999996}]},{"osd":6,"last update":"Mon Feb 5 06:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66000000000000003,"15min":0.57499999999999996},"min":{"1min":0.35999999999999999,"5min":0.314,"15min":0.314},"max":{"1min":1.52,"5min":3.5750000000000002,"15min":3.5750000000000002},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57599999999999996,"15min":0.55400000000000005},"min":{"1min":0.35999999999999999,"5min":0.245,"15min":0.245},"max":{"1min":1.2310000000000001,"5min":3.2189999999999999,"15min":3.2189999999999999},"last":0.54400000000000004}]}]},{"osd":6,"up_from":49,"seq":210453397603,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1977300,"kb_used_data":1783688,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":91767852,"statfs":{"total":95995035648,"available":93970280448,"internally_reserved":0,"allocated":1826496512,"data_stored":1825658482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":198244806},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:40:06 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":1.365,"15min":0.82999999999999996},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.0179999999999998,"5min":6.6070000000000002,"15min":6.6070000000000002},"last":0.249},{"interface":"front","average":{"1min":0.78700000000000003,"5min":1.1890000000000001,"15min":0.75700000000000001},"min":{"1min":0.314,"5min":0.248,"15min":0.248},"max":{"1min":3.855,"5min":9.2629999999999999,"15min":9.2629999999999999},"last":1.2310000000000001}]},{"osd":1,"last update":"Mon Feb 5 06:40:06 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":1.0109999999999999,"15min":0.68400000000000005},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.3610000000000002,"5min":5.9660000000000002,"15min":5.9660000000000002},"last":2.0449999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.80300000000000005,"15min":0.65600000000000003},"min":{"1min":0.34000000000000002,"5min":0.26400000000000001,"15min":0.255},"max":{"1min":1.919,"5min":5.5099999999999998,"15min":5.5099999999999998},"last":0.36299999999999999}]},{"osd":2,"last update":"Mon Feb 5 06:40:06 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":1.077,"15min":0.747},"min":{"1min":0.29699999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":1.7270000000000001,"5min":11.898999999999999,"15min":11.898999999999999},"last":2.5649999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.96499999999999997,"15min":0.68999999999999995},"min":{"1min":0.311,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.1040000000000001,"5min":7.335,"15min":7.335},"last":1.179}]},{"osd":3,"last update":"Mon Feb 5 06:40:06 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":2.2400000000000002,"15min":1.1120000000000001},"min":{"1min":0.35699999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":2.1669999999999998,"5min":99.269000000000005,"15min":99.269000000000005},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.2290000000000001,"5min":2.48,"15min":1.2090000000000001},"min":{"1min":0.40999999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":5.5810000000000004,"5min":101.75700000000001,"15min":101.75700000000001},"last":0.28999999999999998}]},{"osd":4,"last update":"Mon Feb 5 06:40:06 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.47199999999999998,"15min":0.45700000000000002},"min":{"1min":0.30499999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.80000000000000004,"5min":1.8220000000000001,"15min":1.8220000000000001},"last":0.314},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.47599999999999998,"15min":0.442},"min":{"1min":0.32900000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.78300000000000003,"5min":1.2210000000000001,"15min":1.2210000000000001},"last":0.33000000000000002}]},{"osd":5,"last update":"Mon Feb 5 06:40:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.50700000000000001,"15min":0.47399999999999998},"min":{"1min":0.32600000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.84399999999999997,"5min":1.798,"15min":1.798},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.51100000000000001,"15min":0.47199999999999998},"min":{"1min":0.373,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.85799999999999998,"5min":1.276,"15min":1.276},"last":0.371}]},{"osd":7,"last update":"Mon Feb 5 06:40:25 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.54700000000000004,"15min":0.53100000000000003},"min":{"1min":0.34100000000000003,"5min":0.309,"15min":0.309},"max":{"1min":0.92900000000000005,"5min":6.0350000000000001,"15min":6.0350000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.66300000000000003,"15min":0.58999999999999997},"min":{"1min":0.39200000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.90300000000000002,"5min":8.5399999999999991,"15min":8.5399999999999991},"last":0.35099999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574972,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2134868,"kb_used_data":1919052,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":91610284,"statfs":{"total":95995035648,"available":93808930816,"internally_reserved":0,"allocated":1965109248,"data_stored":1964210686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":220985797},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:39:59 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":2.3439999999999999,"15min":1.0129999999999999},"min":{"1min":0.26400000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":3.2429999999999999,"5min":67.421000000000006,"15min":67.421000000000006},"last":0.69299999999999995},{"interface":"front","average":{"1min":1.7749999999999999,"5min":2.3879999999999999,"15min":1.0349999999999999},"min":{"1min":0.26500000000000001,"5min":0.20999999999999999,"15min":0.19600000000000001},"max":{"1min":6.3789999999999996,"5min":67.885000000000005,"15min":67.885000000000005},"last":1.766}]},{"osd":2,"last update":"Mon Feb 5 06:40:22 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":1.726,"15min":0.83799999999999997},"min":{"1min":0.27100000000000002,"5min":0.215,"15min":0.214},"max":{"1min":3.1000000000000001,"5min":9.7929999999999993,"15min":9.7929999999999993},"last":0.435},{"interface":"front","average":{"1min":0.94799999999999995,"5min":2.1469999999999998,"15min":0.96899999999999997},"min":{"1min":0.249,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.5890000000000004,"5min":11.859,"15min":11.859},"last":0.41099999999999998}]},{"osd":3,"last update":"Mon Feb 5 06:39:56 2024","interfaces":[{"interface":"back","average":{"1min":1.393,"5min":2.8919999999999999,"15min":1.2470000000000001},"min":{"1min":0.30199999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":6.4580000000000002,"5min":69.798000000000002,"15min":69.798000000000002},"last":3.532},{"interface":"front","average":{"1min":1.173,"5min":2.4060000000000001,"15min":1.123},"min":{"1min":0.29599999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":4.9729999999999999,"5min":70.765000000000001,"15min":70.765000000000001},"last":0.66700000000000004}]},{"osd":4,"last update":"Mon Feb 5 06:40:16 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":1.4510000000000001,"15min":0.85899999999999999},"min":{"1min":0.33600000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.4420000000000002,"5min":11.679,"15min":11.679},"last":0.49099999999999999},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.3919999999999999,"15min":0.85299999999999998},"min":{"1min":0.44900000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.0730000000000004,"5min":6.0730000000000004,"15min":6.0730000000000004},"last":0.26300000000000001}]},{"osd":5,"last update":"Mon Feb 5 06:40:44 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.1439999999999999,"15min":1},"min":{"1min":0.34599999999999997,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":3.4399999999999999,"5min":6.0469999999999997,"15min":8.2509999999999994},"last":0.63400000000000001},{"interface":"front","average":{"1min":1.081,"5min":1.6220000000000001,"15min":0.95899999999999996},"min":{"1min":0.42799999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":4.5590000000000002,"5min":13.455,"15min":13.455},"last":0.57599999999999996}]},{"osd":6,"last update":"Mon Feb 5 06:39:59 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":1.681,"15min":0.98999999999999999},"min":{"1min":0.35799999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":4.2220000000000004,"5min":13.391,"15min":13.391},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.90000000000000002,"5min":1.1950000000000001,"15min":0.84899999999999998},"min":{"1min":0.40400000000000003,"5min":0.379,"15min":0.379},"max":{"1min":2.9289999999999998,"5min":8.6099999999999994,"15min":8.6099999999999994},"last":0.47299999999999998}]},{"osd":7,"last update":"Mon Feb 5 06:40:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":1.663,"15min":0.94799999999999995},"min":{"1min":0.46000000000000002,"5min":0.37,"15min":0.37},"max":{"1min":4.5300000000000002,"5min":13.49,"15min":13.49},"last":0.54000000000000004},{"interface":"front","average":{"1min":1.6910000000000001,"5min":1.423,"15min":1.141},"min":{"1min":0.41599999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":6.1870000000000003,"5min":6.1870000000000003,"15min":8.2219999999999995},"last":0.55600000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738498,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2102284,"kb_used_data":1832316,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":91642868,"statfs":{"total":95995035648,"available":93842296832,"internally_reserved":0,"allocated":1876291584,"data_stored":1875409272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 06:39:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.722,"15min":0.80600000000000005},"min":{"1min":0.245,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":9.5109999999999992,"5min":12.031000000000001,"15min":12.031000000000001},"last":5.8700000000000001},{"interface":"front","average":{"1min":0.876,"5min":1.585,"15min":0.75700000000000001},"min":{"1min":0.25700000000000001,"5min":0.23999999999999999,"15min":0.20499999999999999},"max":{"1min":2.9900000000000002,"5min":7.6230000000000002,"15min":7.6230000000000002},"last":6.032}]},{"osd":2,"last update":"Mon Feb 5 06:40:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9019999999999999,"5min":2.0939999999999999,"15min":0.97799999999999998},"min":{"1min":0.27100000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":16.981999999999999,"5min":16.981999999999999,"15min":16.981999999999999},"last":2.1619999999999999},{"interface":"front","average":{"1min":1.6879999999999999,"5min":1.9059999999999999,"15min":0.92300000000000004},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.28000000000000003},"max":{"1min":16.998999999999999,"5min":16.998999999999999,"15min":16.998999999999999},"last":1.264}]},{"osd":3,"last update":"Mon Feb 5 06:40:49 2024","interfaces":[{"interface":"back","average":{"1min":2.3980000000000001,"5min":2.286,"15min":1.0700000000000001},"min":{"1min":0.32400000000000001,"5min":0.29799999999999999,"15min":0.29099999999999998},"max":{"1min":16.923999999999999,"5min":16.923999999999999,"15min":16.923999999999999},"last":1.6930000000000001},{"interface":"front","average":{"1min":3.202,"5min":2.5750000000000002,"15min":1.1679999999999999},"min":{"1min":0.26000000000000001,"5min":0.254,"15min":0.254},"max":{"1min":18.259,"5min":18.259,"15min":18.259},"last":2.399}]},{"osd":4,"last update":"Mon Feb 5 06:40:16 2024","interfaces":[{"interface":"back","average":{"1min":1.76,"5min":1.577,"15min":0.92300000000000004},"min":{"1min":0.40300000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":17.241,"5min":17.241,"15min":17.241},"last":2.246},{"interface":"front","average":{"1min":1.6839999999999999,"5min":1.528,"15min":0.92900000000000005},"min":{"1min":0.503,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":18.495999999999999,"5min":18.495999999999999,"15min":18.495999999999999},"last":1.615}]},{"osd":5,"last update":"Mon Feb 5 06:40:39 2024","interfaces":[{"interface":"back","average":{"1min":2.0449999999999999,"5min":1.508,"15min":0.92100000000000004},"min":{"1min":0.39400000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":18.170999999999999,"5min":18.170999999999999,"15min":18.170999999999999},"last":1.3300000000000001},{"interface":"front","average":{"1min":1.9370000000000001,"5min":1.569,"15min":0.95099999999999996},"min":{"1min":0.40000000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":17.152999999999999,"5min":17.152999999999999,"15min":17.152999999999999},"last":2.2189999999999999}]},{"osd":6,"last update":"Mon Feb 5 06:39:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.379,"15min":0.89900000000000002},"min":{"1min":0.50900000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":3.931,"5min":12.119999999999999,"15min":12.119999999999999},"last":1.657},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.2869999999999999,"15min":0.86299999999999999},"min":{"1min":0.497,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":2.7229999999999999,"5min":6.6820000000000004,"15min":6.6820000000000004},"last":1.3029999999999999}]},{"osd":7,"last update":"Mon Feb 5 06:40:31 2024","interfaces":[{"interface":"back","average":{"1min":1.827,"5min":1.5980000000000001,"15min":0.93100000000000005},"min":{"1min":0.48799999999999999,"5min":0.37,"15min":0.37},"max":{"1min":17.210000000000001,"5min":17.210000000000001,"15min":17.210000000000001},"last":2.2709999999999999},{"interface":"front","average":{"1min":1.7589999999999999,"5min":1.6040000000000001,"15min":1.1459999999999999},"min":{"1min":0.47899999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":18.533999999999999,"5min":18.533999999999999,"15min":18.533999999999999},"last":1.7210000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411447,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2273048,"kb_used_data":2066956,"kb_used_omap":1,"kb_used_meta":206078,"kb_avail":91472104,"statfs":{"total":95995035648,"available":93667434496,"internally_reserved":0,"allocated":2116562944,"data_stored":2115722013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":211024325},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:40:24 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":3.0630000000000002,"15min":1.2689999999999999},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.282,"5min":50.814,"15min":50.814},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.887,"5min":2.9489999999999998,"15min":1.2450000000000001},"min":{"1min":0.21199999999999999,"5min":0.192,"15min":0.188},"max":{"1min":10.026,"5min":50.802999999999997,"15min":50.802999999999997},"last":1.044}]},{"osd":1,"last update":"Mon Feb 5 06:40:24 2024","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":2.4180000000000001,"15min":1.0669999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":6.7430000000000003,"5min":31.303000000000001,"15min":31.303000000000001},"last":2.823},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.853,"15min":0.90900000000000003},"min":{"1min":0.28299999999999997,"5min":0.22,"15min":0.22},"max":{"1min":4.4770000000000003,"5min":31.425000000000001,"15min":31.425000000000001},"last":1.48}]},{"osd":3,"last update":"Mon Feb 5 06:40:54 2024","interfaces":[{"interface":"back","average":{"1min":2.6709999999999998,"5min":3.3050000000000002,"15min":1.4199999999999999},"min":{"1min":0.28299999999999997,"5min":0.254,"15min":0.254},"max":{"1min":11.409000000000001,"5min":53.127000000000002,"15min":53.127000000000002},"last":5.5270000000000001},{"interface":"front","average":{"1min":3.1150000000000002,"5min":3.1629999999999998,"15min":1.387},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.22500000000000001},"max":{"1min":11.388999999999999,"5min":52.607999999999997,"15min":52.607999999999997},"last":5.5469999999999997}]},{"osd":4,"last update":"Mon Feb 5 06:40:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":1.129,"15min":0.80600000000000005},"min":{"1min":0.33700000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":1.9850000000000001,"5min":6.4180000000000001,"15min":6.4180000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.753,"5min":1.0529999999999999,"15min":0.79300000000000004},"min":{"1min":0.30199999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":1.903,"5min":6.8780000000000001,"15min":6.8780000000000001},"last":1.1419999999999999}]},{"osd":5,"last update":"Mon Feb 5 06:40:41 2024","interfaces":[{"interface":"back","average":{"1min":7.0350000000000001,"5min":2.323,"15min":1.2330000000000001},"min":{"1min":0.48299999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":86.614999999999995,"5min":86.614999999999995,"15min":86.614999999999995},"last":1.1699999999999999},{"interface":"front","average":{"1min":7.2270000000000003,"5min":2.738,"15min":1.359},"min":{"1min":0.40500000000000003,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":86.543999999999997,"5min":86.543999999999997,"15min":86.543999999999997},"last":5.4640000000000004}]},{"osd":6,"last update":"Mon Feb 5 06:40:07 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":1.1060000000000001,"15min":1.446},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":1.8959999999999999,"5min":6.9139999999999997,"15min":17.143999999999998},"last":1.194},{"interface":"front","average":{"1min":0.90800000000000003,"5min":1.5269999999999999,"15min":0.97499999999999998},"min":{"1min":0.45500000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":3.8380000000000001,"5min":10.27,"15min":10.27},"last":2.8759999999999999}]},{"osd":7,"last update":"Mon Feb 5 06:40:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.6659999999999999,"15min":0.98899999999999999},"min":{"1min":0.46100000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":6.8609999999999998,"5min":10.206,"15min":10.206},"last":2.9060000000000001},{"interface":"front","average":{"1min":1.123,"5min":1.4079999999999999,"15min":0.91600000000000004},"min":{"1min":0.41399999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":4.6929999999999996,"5min":6.4749999999999996,"15min":6.4749999999999996},"last":0.68600000000000005}]}]},{"osd":3,"up_from":26,"seq":111669149810,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2637936,"kb_used_data":2327716,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":91107216,"statfs":{"total":95995035648,"available":93293789184,"internally_reserved":0,"allocated":2383581184,"data_stored":2382715509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":317651399},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:39:58 2024","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.569,"15min":0.78400000000000003},"min":{"1min":0.23799999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":6.4219999999999997,"5min":13.391999999999999,"15min":13.391999999999999},"last":1.1619999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":2.097,"15min":1},"min":{"1min":0.23100000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.3869999999999996,"5min":25.087,"15min":25.087},"last":0.82399999999999995}]},{"osd":1,"last update":"Mon Feb 5 06:39:58 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":1.6279999999999999,"15min":0.85499999999999998},"min":{"1min":0.248,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.3780000000000001,"5min":9.6010000000000009,"15min":9.6010000000000009},"last":2.2120000000000002},{"interface":"front","average":{"1min":1.0069999999999999,"5min":1.55,"15min":0.81699999999999995},"min":{"1min":0.26900000000000002,"5min":0.246,"15min":0.246},"max":{"1min":5.2779999999999996,"5min":8.3040000000000003,"15min":8.3040000000000003},"last":1.5920000000000001}]},{"osd":2,"last update":"Mon Feb 5 06:39:58 2024","interfaces":[{"interface":"back","average":{"1min":2.7919999999999998,"5min":2.048,"15min":1.01},"min":{"1min":0.29899999999999999,"5min":0.29799999999999999,"15min":0.27300000000000002},"max":{"1min":41.529000000000003,"5min":41.529000000000003,"15min":41.529000000000003},"last":7.2880000000000003},{"interface":"front","average":{"1min":3.2029999999999998,"5min":1.964,"15min":0.96599999999999997},"min":{"1min":0.32100000000000001,"5min":0.248,"15min":0.248},"max":{"1min":41.759999999999998,"5min":41.759999999999998,"15min":41.759999999999998},"last":1.1279999999999999}]},{"osd":4,"last update":"Mon Feb 5 06:40:16 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":1.2150000000000001,"15min":0.80600000000000005},"min":{"1min":0.39800000000000002,"5min":0.307,"15min":0.307},"max":{"1min":3.3090000000000002,"5min":16.577999999999999,"15min":16.577999999999999},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.89800000000000002,"5min":1.518,"15min":0.90300000000000002},"min":{"1min":0.314,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.0190000000000001,"5min":8.4060000000000006,"15min":8.4060000000000006},"last":0.92700000000000005}]},{"osd":5,"last update":"Mon Feb 5 06:40:40 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.7330000000000001,"15min":0.97799999999999998},"min":{"1min":0.441,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":5.2249999999999996,"5min":13.672000000000001,"15min":13.672000000000001},"last":1.2310000000000001},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.375,"15min":0.871},"min":{"1min":0.45100000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":6.2569999999999997,"5min":15.792999999999999,"15min":15.792999999999999},"last":1.9890000000000001}]},{"osd":6,"last update":"Mon Feb 5 06:40:06 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":1.498,"15min":0.877},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":3.3540000000000001,"5min":8.3610000000000007,"15min":8.3610000000000007},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.753,"5min":1.456,"15min":0.90800000000000003},"min":{"1min":0.318,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.821,"5min":6.601,"15min":6.601},"last":1.1970000000000001}]},{"osd":7,"last update":"Mon Feb 5 06:40:32 2024","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.732,"15min":0.96799999999999997},"min":{"1min":0.45000000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.7469999999999999,"5min":45.039000000000001,"15min":45.039000000000001},"last":2.1760000000000002},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.95,"15min":1.0609999999999999},"min":{"1min":0.46700000000000003,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":5.258,"5min":44.036999999999999,"15min":44.036999999999999},"last":1.262}]}]},{"osd":4,"up_from":31,"seq":133143986284,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1984748,"kb_used_data":1682848,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":91760404,"statfs":{"total":95995035648,"available":93962653696,"internally_reserved":0,"allocated":1723236352,"data_stored":1722374192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:40:14 2024","interfaces":[{"interface":"back","average":{"1min":2.4569999999999999,"5min":1.7250000000000001,"15min":0.96399999999999997},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":33.372,"5min":33.372,"15min":33.372},"last":2.468},{"interface":"front","average":{"1min":2.1869999999999998,"5min":2.008,"15min":1.0329999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":33.323999999999998,"5min":33.323999999999998,"15min":33.323999999999998},"last":0.64500000000000002}]},{"osd":1,"last update":"Mon Feb 5 06:40:14 2024","interfaces":[{"interface":"back","average":{"1min":2.4420000000000002,"5min":1.163,"15min":0.76100000000000001},"min":{"1min":0.25600000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":33.747999999999998,"5min":33.747999999999998,"15min":33.747999999999998},"last":0.45900000000000002},{"interface":"front","average":{"1min":2.3940000000000001,"5min":1.3280000000000001,"15min":0.83899999999999997},"min":{"1min":0.28299999999999997,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":33.691000000000003,"5min":33.691000000000003,"15min":33.691000000000003},"last":3.6760000000000002}]},{"osd":2,"last update":"Mon Feb 5 06:40:14 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.79900000000000004,"15min":0.64400000000000002},"min":{"1min":0.28199999999999997,"5min":0.245,"15min":0.245},"max":{"1min":1.0840000000000001,"5min":4.1239999999999997,"15min":4.1239999999999997},"last":2.1150000000000002},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.98799999999999999,"15min":0.72099999999999997},"min":{"1min":0.35999999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":4.298,"5min":5.7030000000000003,"15min":5.7030000000000003},"last":2.448}]},{"osd":3,"last update":"Mon Feb 5 06:40:15 2024","interfaces":[{"interface":"back","average":{"1min":2.7210000000000001,"5min":3.6070000000000002,"15min":1.601},"min":{"1min":0.37,"5min":0.31,"15min":0.31},"max":{"1min":41.512,"5min":99.546999999999997,"15min":99.546999999999997},"last":5.3170000000000002},{"interface":"front","average":{"1min":2.4820000000000002,"5min":3.6200000000000001,"15min":1.595},"min":{"1min":0.33400000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":38.527000000000001,"5min":99.245000000000005,"15min":99.245000000000005},"last":4.6310000000000002}]},{"osd":5,"last update":"Mon Feb 5 06:40:42 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.63,"15min":0.48499999999999999},"min":{"1min":0.28699999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1180000000000001,"5min":11.098000000000001,"15min":11.098000000000001},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.61399999999999999,"15min":0.5},"min":{"1min":0.25700000000000001,"5min":0.252,"15min":0.252},"max":{"1min":1.145,"5min":10.965999999999999,"15min":10.965999999999999},"last":0.58399999999999996}]},{"osd":6,"last update":"Mon Feb 5 06:40:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.67900000000000005,"15min":0.55700000000000005},"min":{"1min":0.316,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.1080000000000001,"5min":11.868,"15min":11.868},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.72399999999999998,"15min":0.56100000000000005},"min":{"1min":0.35599999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.1110000000000002,"5min":11.787000000000001,"15min":11.787000000000001},"last":0.80000000000000004}]},{"osd":7,"last update":"Mon Feb 5 06:40:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.54900000000000004,"15min":0.52500000000000002},"min":{"1min":0.29399999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":2.2719999999999998},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.56599999999999995,"15min":0.54700000000000004},"min":{"1min":0.29199999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.70299999999999996,"5min":2.9470000000000001,"15min":2.9470000000000001},"last":0.78400000000000003}]}]},{"osd":5,"up_from":0,"seq":171798691943,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1874329600,"data_stored":1874329600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1963327488,"data_stored":1963327488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2131886080,"data_stored":2131886080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2396192768,"data_stored":2396192768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1736114176,"data_stored":1736114176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":888864768,"data_stored":888864768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1843724288,"data_stored":1843724288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":837681152,"data_stored":837681152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T06:41:02.502 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-02-05T06:41:02.502 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:02.533 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:02.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:02.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:02.635 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:02.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:02.680 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:02.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:02.697 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:41:02 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:41:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:41:02.781 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:02.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:02.807 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:02.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:02.908 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:02.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:02.937 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:02.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:03.038 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:03.116 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:41:02 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:41:03.143 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:03.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:03.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:03.236 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 68 16 69989 69973 64.2999 59.6875 0.00865889 0.0155456 2024-02-05T06:41:03.244 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:03.283 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:03.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:03.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:03.354 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:03 smithi167 ceph-mon[32774]: osdmap e231: 8 total, 8 up, 7 in 2024-02-05T06:41:03.354 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:03 smithi167 ceph-mon[32774]: pgmap v358: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 974 B/s rd, 41 MiB/s wr, 663 op/s; 5886/205608 objects misplaced (2.863%); 15 MiB/s, 244 objects/s recovering 2024-02-05T06:41:03.354 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:03 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:41:03.355 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-05T06:41:03.385 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:03.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:03.430 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:03.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:03.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:03 smithi028 ceph-mon[29442]: osdmap e231: 8 total, 8 up, 7 in 2024-02-05T06:41:03.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:03 smithi028 ceph-mon[29442]: pgmap v358: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 974 B/s rd, 41 MiB/s wr, 663 op/s; 5886/205608 objects misplaced (2.863%); 15 MiB/s, 244 objects/s recovering 2024-02-05T06:41:03.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:03 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:41:03.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:03 smithi028 ceph-mon[37984]: osdmap e231: 8 total, 8 up, 7 in 2024-02-05T06:41:03.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:03 smithi028 ceph-mon[37984]: pgmap v358: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 974 B/s rd, 41 MiB/s wr, 663 op/s; 5886/205608 objects misplaced (2.863%); 15 MiB/s, 244 objects/s recovering 2024-02-05T06:41:03.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:03 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:41:03.531 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:03.563 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:03.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:03.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:03.665 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:03.681 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:41:03.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:03.712 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:03.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:03.813 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:03.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:03.841 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:03.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:03.942 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:03.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:03.984 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:03.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:04.085 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:04.119 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:04.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:04.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:04.221 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:04.238 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 69 16 70918 70902 64.2094 58.0625 0.0223146 0.0155684 2024-02-05T06:41:04.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:04.264 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:04.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:04.365 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:04.405 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:04.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:04.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:04.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:04 smithi028 ceph-mon[37984]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2774/195165 objects degraded (1.421%), 2 pgs degraded) 2024-02-05T06:41:04.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:04 smithi028 ceph-mon[29442]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2774/195165 objects degraded (1.421%), 2 pgs degraded) 2024-02-05T06:41:04.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:04 smithi167 ceph-mon[32774]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2774/195165 objects degraded (1.421%), 2 pgs degraded) 2024-02-05T06:41:04.507 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:04.547 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:04.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:04.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:04.649 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:04.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:04.695 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:04.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:04.796 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:04.824 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:04.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:04.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:04.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:04.970 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:04.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:04.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:05.072 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:05.095 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:05.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:05.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:05.133 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:05 smithi167 ceph-mon[32774]: pgmap v359: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 36 MiB/s wr, 580 op/s; 5886/205608 objects misplaced (2.863%); 13 MiB/s, 214 objects/s recovering 2024-02-05T06:41:05.133 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:05 smithi167 ceph-mon[32774]: osdmap e232: 8 total, 8 up, 7 in 2024-02-05T06:41:05.196 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:05.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:05.228 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:05.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:05.236 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 70 16 71797 71781 64.0769 54.9375 0.00452098 0.0155992 2024-02-05T06:41:05.330 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:05.359 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:05.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:05.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:05.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:05 smithi028 ceph-mon[37984]: pgmap v359: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 36 MiB/s wr, 580 op/s; 5886/205608 objects misplaced (2.863%); 13 MiB/s, 214 objects/s recovering 2024-02-05T06:41:05.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:05 smithi028 ceph-mon[37984]: osdmap e232: 8 total, 8 up, 7 in 2024-02-05T06:41:05.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:05 smithi028 ceph-mon[29442]: pgmap v359: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.2 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 36 MiB/s wr, 580 op/s; 5886/205608 objects misplaced (2.863%); 13 MiB/s, 214 objects/s recovering 2024-02-05T06:41:05.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:05 smithi028 ceph-mon[29442]: osdmap e232: 8 total, 8 up, 7 in 2024-02-05T06:41:05.461 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:05.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:05.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:05.548 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:41:05.548 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:41:05.605 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:05.633 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:05.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:05.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:05.735 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:05.764 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:05.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:05.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:05.866 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:05.895 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:05.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:05.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:05.996 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:06.028 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:06.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:06.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:06.131 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:06.181 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:06.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:06.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:06.236 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 71 16 72763 72747 64.0247 60.375 0.0124236 0.0156123 2024-02-05T06:41:06.283 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-05T06:41:06.283 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:06.317 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:06.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:06.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:06.419 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:06.452 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:06.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:06.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:06.554 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:06.590 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:06.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:06.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:06.627 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:41:06.692 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:06.721 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:06.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:06.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:06.823 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:06.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:06.874 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:06.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:06.975 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:07.017 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:07.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:07.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:07.118 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:07.146 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:07.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:07.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:07.236 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 72 16 73720 73704 63.9661 59.8125 0.00783441 0.015628 2024-02-05T06:41:07.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:07.296 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:07.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:07.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:07.398 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:07.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:07.445 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:07.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:07.503 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 4, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T06:41:07.503 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T06:41:07.503 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_1 2024-02-05T06:41:07.504 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T06:41:07.546 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:07.573 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:07.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:07.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:07.653 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:07 smithi028 ceph-mon[37984]: pgmap v361: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.6 KiB/s rd, 49 MiB/s wr, 793 op/s; 5886/210066 objects misplaced (2.802%); 13 MiB/s, 204 objects/s recovering 2024-02-05T06:41:07.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:07 smithi028 ceph-mon[29442]: pgmap v361: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.6 KiB/s rd, 49 MiB/s wr, 793 op/s; 5886/210066 objects misplaced (2.802%); 13 MiB/s, 204 objects/s recovering 2024-02-05T06:41:07.675 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:07.681 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:41:07.701 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:07.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:07.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:07.704 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:41:07.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:07 smithi167 ceph-mon[32774]: pgmap v361: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.3 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.6 KiB/s rd, 49 MiB/s wr, 793 op/s; 5886/210066 objects misplaced (2.802%); 13 MiB/s, 204 objects/s recovering 2024-02-05T06:41:07.744 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:41:07.790 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:41:07.803 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:07.831 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:07.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:07.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:07.839 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:41:07.884 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:41:07.926 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:41:07.932 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:07.968 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:41:07.975 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:41:07.977 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:41:07.977 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:41:07.977 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:41:07.977 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:41:07.977 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:41:07.977 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:41:07.977 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:41:07.977 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:41:07.983 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:07.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:07.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:08.085 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:08.109 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:08.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:08.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:08.210 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:08.236 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 73 16 74444 74428 63.7097 45.25 0.0176984 0.0156904 2024-02-05T06:41:08.244 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:08.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:08.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:08.346 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:08.383 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-02-05T06:41:08.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:08.390 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:08.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:08.491 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:08.537 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:08.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:08.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:08.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:08.679 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:08.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:08.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:08.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:08 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/580364309' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:41:08.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:08 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3988072782' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:41:08.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:08 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/580364309' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:41:08.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:08 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3988072782' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:41:08.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:08 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/580364309' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:41:08.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:08 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3988072782' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:41:08.781 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:08.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:08.819 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:08.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:08.920 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:08.951 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:08.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:08.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:09.053 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:09.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:09.084 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:09.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:09.149 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:41:09.149 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-02-05T06:41:09.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-02-05T06:41:09.150 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-02-05T06:41:09.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.150 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:30010 8.2 8:4c5236ed:::benchmark_data_smithi028_105698_object30009:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-02-05T06:41:09.150 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:07.547293+0000", 2024-02-05T06:41:09.150 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 61.595065411999997, 2024-02-05T06:41:09.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2328662420000001, 2024-02-05T06:41:09.150 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:09.150 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-02-05T06:41:09.150 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:09.150 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:41:09.150 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:41:09.150 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30010 2024-02-05T06:41:09.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.151 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:09.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:09.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.547293+0000", 2024-02-05T06:41:09.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:09.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.547293+0000", 2024-02-05T06:41:09.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:09.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.547295+0000", 2024-02-05T06:41:09.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0810000000000001e-06 2024-02-05T06:41:09.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:09.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.547338+0000", 2024-02-05T06:41:09.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2488999999999999e-05 2024-02-05T06:41:09.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:09.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.547347+0000", 2024-02-05T06:41:09.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2569999999999996e-06 2024-02-05T06:41:09.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:09.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.547355+0000", 2024-02-05T06:41:09.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7540000000000006e-06 2024-02-05T06:41:09.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.547424+0000", 2024-02-05T06:41:09.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9509000000000001e-05 2024-02-05T06:41:09.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:09.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.547552+0000", 2024-02-05T06:41:09.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012786699999999999 2024-02-05T06:41:09.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:41:09.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.547685+0000", 2024-02-05T06:41:09.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013314199999999999 2024-02-05T06:41:09.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.549231+0000", 2024-02-05T06:41:09.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015464529999999999 2024-02-05T06:41:09.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:09.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.549378+0000", 2024-02-05T06:41:09.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014604400000000001 2024-02-05T06:41:09.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.562199+0000", 2024-02-05T06:41:09.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012821572 2024-02-05T06:41:09.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:41:09.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.562214+0000", 2024-02-05T06:41:09.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4938e-05 2024-02-05T06:41:09.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:08.559178+0000", 2024-02-05T06:41:09.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99696352700000002 2024-02-05T06:41:09.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:41:09.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:08.559200+0000", 2024-02-05T06:41:09.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2819e-05 2024-02-05T06:41:09.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.780059+0000", 2024-02-05T06:41:09.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2208585730000001 2024-02-05T06:41:09.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:09.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.780159+0000", 2024-02-05T06:41:09.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010021700000000001 2024-02-05T06:41:09.157 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.157 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:09.157 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.158 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:30022 8.2 8:4849d65b:::benchmark_data_smithi028_105698_object30021:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-02-05T06:41:09.158 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:07.561554+0000", 2024-02-05T06:41:09.158 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 61.580804106999999, 2024-02-05T06:41:09.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2397414059999998, 2024-02-05T06:41:09.158 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:09.158 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:09.158 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:09.158 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:41:09.158 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:41:09.158 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30022 2024-02-05T06:41:09.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.158 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:09.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:09.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.561554+0000", 2024-02-05T06:41:09.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:09.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.561554+0000", 2024-02-05T06:41:09.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:09.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.561557+0000", 2024-02-05T06:41:09.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5349999999999999e-06 2024-02-05T06:41:09.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:09.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.561599+0000", 2024-02-05T06:41:09.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2616e-05 2024-02-05T06:41:09.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:09.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.561613+0000", 2024-02-05T06:41:09.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3122000000000001e-05 2024-02-05T06:41:09.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:09.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.561624+0000", 2024-02-05T06:41:09.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1398e-05 2024-02-05T06:41:09.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.562244+0000", 2024-02-05T06:41:09.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00061996200000000005 2024-02-05T06:41:09.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:41:09.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.562252+0000", 2024-02-05T06:41:09.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7659999999999998e-06 2024-02-05T06:41:09.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:08.559259+0000", 2024-02-05T06:41:09.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99700688299999995 2024-02-05T06:41:09.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:41:09.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:08.559271+0000", 2024-02-05T06:41:09.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2364999999999999e-05 2024-02-05T06:41:09.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.781169+0000", 2024-02-05T06:41:09.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.221897805 2024-02-05T06:41:09.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:09.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.781268+0000", 2024-02-05T06:41:09.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8818000000000002e-05 2024-02-05T06:41:09.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:41:09.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.781446+0000", 2024-02-05T06:41:09.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178495 2024-02-05T06:41:09.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:09.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.784625+0000", 2024-02-05T06:41:09.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003179472 2024-02-05T06:41:09.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.784675+0000", 2024-02-05T06:41:09.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9064999999999999e-05 2024-02-05T06:41:09.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.799907+0000", 2024-02-05T06:41:09.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015232757 2024-02-05T06:41:09.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:09.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.799974+0000", 2024-02-05T06:41:09.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6451000000000005e-05 2024-02-05T06:41:09.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:09.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.801296+0000", 2024-02-05T06:41:09.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0013218959999999999 2024-02-05T06:41:09.166 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.166 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:09.166 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.166 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:30023 8.2 8:4af7df9a:::benchmark_data_smithi028_105698_object30022:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-02-05T06:41:09.166 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:07.569012+0000", 2024-02-05T06:41:09.166 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 61.573346219999998, 2024-02-05T06:41:09.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.233291189, 2024-02-05T06:41:09.166 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:09.166 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:09.166 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:09.166 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:41:09.166 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:41:09.167 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30023 2024-02-05T06:41:09.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.167 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:09.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:09.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.569012+0000", 2024-02-05T06:41:09.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:09.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.569012+0000", 2024-02-05T06:41:09.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:09.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.569014+0000", 2024-02-05T06:41:09.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2699999999999999e-06 2024-02-05T06:41:09.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:09.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.569105+0000", 2024-02-05T06:41:09.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0138999999999999e-05 2024-02-05T06:41:09.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:09.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.569122+0000", 2024-02-05T06:41:09.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7382999999999999e-05 2024-02-05T06:41:09.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:09.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.569133+0000", 2024-02-05T06:41:09.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1004e-05 2024-02-05T06:41:09.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.569190+0000", 2024-02-05T06:41:09.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6972999999999998e-05 2024-02-05T06:41:09.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:41:09.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.569232+0000", 2024-02-05T06:41:09.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2197000000000003e-05 2024-02-05T06:41:09.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:08.559313+0000", 2024-02-05T06:41:09.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99008115900000004 2024-02-05T06:41:09.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:41:09.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:08.559318+0000", 2024-02-05T06:41:09.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2129999999999999e-06 2024-02-05T06:41:09.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.782618+0000", 2024-02-05T06:41:09.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.223299849 2024-02-05T06:41:09.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:09.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.782747+0000", 2024-02-05T06:41:09.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012904400000000001 2024-02-05T06:41:09.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:41:09.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.782939+0000", 2024-02-05T06:41:09.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019172500000000001 2024-02-05T06:41:09.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:09.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.785396+0000", 2024-02-05T06:41:09.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002457374 2024-02-05T06:41:09.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.791518+0000", 2024-02-05T06:41:09.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0061220119999999996 2024-02-05T06:41:09.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.801346+0000", 2024-02-05T06:41:09.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0098273000000000006 2024-02-05T06:41:09.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:09.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.801400+0000", 2024-02-05T06:41:09.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3910999999999999e-05 2024-02-05T06:41:09.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:09.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.802303+0000", 2024-02-05T06:41:09.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00090363599999999998 2024-02-05T06:41:09.173 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.173 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:09.173 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.173 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:30025 8.2 8:43cd8d1f:::benchmark_data_smithi028_105698_object30024:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-02-05T06:41:09.173 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:07.569160+0000", 2024-02-05T06:41:09.174 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 61.573198099000003, 2024-02-05T06:41:09.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2370548960000001, 2024-02-05T06:41:09.174 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:09.174 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:09.174 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:09.174 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:41:09.174 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:41:09.174 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30025 2024-02-05T06:41:09.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.174 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:09.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:09.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.569160+0000", 2024-02-05T06:41:09.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:09.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.569160+0000", 2024-02-05T06:41:09.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:09.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.569161+0000", 2024-02-05T06:41:09.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.207e-06 2024-02-05T06:41:09.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:09.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.569194+0000", 2024-02-05T06:41:09.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2512999999999999e-05 2024-02-05T06:41:09.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:09.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.569208+0000", 2024-02-05T06:41:09.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4239999999999999e-05 2024-02-05T06:41:09.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:09.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.569215+0000", 2024-02-05T06:41:09.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8600000000000004e-06 2024-02-05T06:41:09.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.569268+0000", 2024-02-05T06:41:09.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2688000000000001e-05 2024-02-05T06:41:09.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:41:09.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.569278+0000", 2024-02-05T06:41:09.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8050000000000001e-06 2024-02-05T06:41:09.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:08.560025+0000", 2024-02-05T06:41:09.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99074740500000003 2024-02-05T06:41:09.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:41:09.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:08.560035+0000", 2024-02-05T06:41:09.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0155e-05 2024-02-05T06:41:09.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.783783+0000", 2024-02-05T06:41:09.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.223747838 2024-02-05T06:41:09.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:09.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.783909+0000", 2024-02-05T06:41:09.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012643499999999999 2024-02-05T06:41:09.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:41:09.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.784006+0000", 2024-02-05T06:41:09.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7021999999999994e-05 2024-02-05T06:41:09.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:09.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.786832+0000", 2024-02-05T06:41:09.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028256000000000002 2024-02-05T06:41:09.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.792577+0000", 2024-02-05T06:41:09.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0057444920000000003 2024-02-05T06:41:09.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.805021+0000", 2024-02-05T06:41:09.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012444319000000001 2024-02-05T06:41:09.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:09.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.805087+0000", 2024-02-05T06:41:09.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6050000000000006e-05 2024-02-05T06:41:09.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:09.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.806215+0000", 2024-02-05T06:41:09.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0011282670000000001 2024-02-05T06:41:09.182 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.183 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:09.183 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.183 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:47685 8.2 8:43c465ad:::benchmark_data_smithi028_105698_object47684:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-02-05T06:41:09.183 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:33.205036+0000", 2024-02-05T06:41:09.183 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 35.937322592999998, 2024-02-05T06:41:09.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.93568001700000003, 2024-02-05T06:41:09.183 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:09.183 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:09.183 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:09.183 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:41:09.183 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:41:09.183 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 47685 2024-02-05T06:41:09.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.184 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:09.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:09.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.205036+0000", 2024-02-05T06:41:09.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:09.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.205036+0000", 2024-02-05T06:41:09.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:09.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.205039+0000", 2024-02-05T06:41:09.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9079999999999999e-06 2024-02-05T06:41:09.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:09.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.205081+0000", 2024-02-05T06:41:09.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2083999999999999e-05 2024-02-05T06:41:09.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:09.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.205098+0000", 2024-02-05T06:41:09.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6843000000000001e-05 2024-02-05T06:41:09.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:09.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.205109+0000", 2024-02-05T06:41:09.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1392e-05 2024-02-05T06:41:09.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.205175+0000", 2024-02-05T06:41:09.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5726000000000001e-05 2024-02-05T06:41:09.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:41:09.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.205185+0000", 2024-02-05T06:41:09.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9529999999999994e-06 2024-02-05T06:41:09.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.129215+0000", 2024-02-05T06:41:09.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.92403015600000005 2024-02-05T06:41:09.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:09.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.129317+0000", 2024-02-05T06:41:09.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010205900000000001 2024-02-05T06:41:09.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:41:09.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.129418+0000", 2024-02-05T06:41:09.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000100621 2024-02-05T06:41:09.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:09.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.135562+0000", 2024-02-05T06:41:09.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.006144465 2024-02-05T06:41:09.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.138545+0000", 2024-02-05T06:41:09.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002982574 2024-02-05T06:41:09.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.140655+0000", 2024-02-05T06:41:09.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021105450000000001 2024-02-05T06:41:09.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:09.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.140690+0000", 2024-02-05T06:41:09.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4999000000000002e-05 2024-02-05T06:41:09.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:09.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.140716+0000", 2024-02-05T06:41:09.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5692000000000001e-05 2024-02-05T06:41:09.189 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.189 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:09.189 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.190 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:47701 8.2 8:4643231d:::benchmark_data_smithi028_105698_object47700:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-02-05T06:41:09.190 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:33.218535+0000", 2024-02-05T06:41:09.190 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 35.923823558000002, 2024-02-05T06:41:09.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.92233781100000001, 2024-02-05T06:41:09.190 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:09.190 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:09.190 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:09.190 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:41:09.190 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:41:09.190 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 47701 2024-02-05T06:41:09.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.190 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:09.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:09.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.218535+0000", 2024-02-05T06:41:09.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:09.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.218535+0000", 2024-02-05T06:41:09.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:09.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.218537+0000", 2024-02-05T06:41:09.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6589999999999999e-06 2024-02-05T06:41:09.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:09.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.218587+0000", 2024-02-05T06:41:09.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9749000000000001e-05 2024-02-05T06:41:09.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:09.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.218601+0000", 2024-02-05T06:41:09.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3909e-05 2024-02-05T06:41:09.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:09.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.218608+0000", 2024-02-05T06:41:09.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8419999999999999e-06 2024-02-05T06:41:09.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.218680+0000", 2024-02-05T06:41:09.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1952000000000005e-05 2024-02-05T06:41:09.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:41:09.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.218692+0000", 2024-02-05T06:41:09.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1946999999999999e-05 2024-02-05T06:41:09.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.131026+0000", 2024-02-05T06:41:09.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.91233404299999998 2024-02-05T06:41:09.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:09.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.131112+0000", 2024-02-05T06:41:09.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6299999999999997e-05 2024-02-05T06:41:09.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:41:09.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.131199+0000", 2024-02-05T06:41:09.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7270999999999998e-05 2024-02-05T06:41:09.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:09.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.138459+0000", 2024-02-05T06:41:09.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0072594749999999996 2024-02-05T06:41:09.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.140769+0000", 2024-02-05T06:41:09.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023098010000000002 2024-02-05T06:41:09.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.140814+0000", 2024-02-05T06:41:09.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4956999999999997e-05 2024-02-05T06:41:09.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:09.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.140852+0000", 2024-02-05T06:41:09.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8281999999999997e-05 2024-02-05T06:41:09.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:09.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.140873+0000", 2024-02-05T06:41:09.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0624000000000001e-05 2024-02-05T06:41:09.196 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.196 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:09.196 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.196 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:47705 8.2 8:4e686c31:::benchmark_data_smithi028_105698_object47704:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-02-05T06:41:09.197 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:33.223911+0000", 2024-02-05T06:41:09.197 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 35.918447565000001, 2024-02-05T06:41:09.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.92633359699999995, 2024-02-05T06:41:09.197 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:09.197 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:09.197 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:09.197 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:41:09.197 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:41:09.197 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 47705 2024-02-05T06:41:09.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.197 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:09.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:09.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.223911+0000", 2024-02-05T06:41:09.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:09.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.223911+0000", 2024-02-05T06:41:09.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:09.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.223914+0000", 2024-02-05T06:41:09.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8899999999999999e-06 2024-02-05T06:41:09.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:09.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.223957+0000", 2024-02-05T06:41:09.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2905999999999999e-05 2024-02-05T06:41:09.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:09.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.223967+0000", 2024-02-05T06:41:09.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0798000000000001e-05 2024-02-05T06:41:09.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:09.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.223976+0000", 2024-02-05T06:41:09.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5760000000000006e-06 2024-02-05T06:41:09.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.224043+0000", 2024-02-05T06:41:09.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7403999999999994e-05 2024-02-05T06:41:09.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:41:09.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.224055+0000", 2024-02-05T06:41:09.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1664999999999999e-05 2024-02-05T06:41:09.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.132006+0000", 2024-02-05T06:41:09.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.90795076200000002 2024-02-05T06:41:09.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:09.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.132144+0000", 2024-02-05T06:41:09.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013802700000000001 2024-02-05T06:41:09.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:41:09.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.132295+0000", 2024-02-05T06:41:09.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015124800000000001 2024-02-05T06:41:09.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:09.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.138467+0000", 2024-02-05T06:41:09.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0061714439999999999 2024-02-05T06:41:09.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.141862+0000", 2024-02-05T06:41:09.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0033951900000000002 2024-02-05T06:41:09.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.150139+0000", 2024-02-05T06:41:09.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0082776719999999998 2024-02-05T06:41:09.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:09.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.150202+0000", 2024-02-05T06:41:09.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2496999999999995e-05 2024-02-05T06:41:09.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:09.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.150244+0000", 2024-02-05T06:41:09.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2518000000000002e-05 2024-02-05T06:41:09.203 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.203 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:09.203 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.203 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:47715 8.2 8:4f1625e2:::benchmark_data_smithi028_105698_object47714:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-02-05T06:41:09.203 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:33.233229+0000", 2024-02-05T06:41:09.203 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 35.909129096000001, 2024-02-05T06:41:09.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.91711963900000004, 2024-02-05T06:41:09.204 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:09.204 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:09.204 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:09.204 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:41:09.204 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:41:09.204 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 47715 2024-02-05T06:41:09.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.204 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:09.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:09.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.233229+0000", 2024-02-05T06:41:09.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:09.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.233229+0000", 2024-02-05T06:41:09.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:09.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.233238+0000", 2024-02-05T06:41:09.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2879999999999993e-06 2024-02-05T06:41:09.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:09.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.233317+0000", 2024-02-05T06:41:09.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8967000000000005e-05 2024-02-05T06:41:09.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:09.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.233334+0000", 2024-02-05T06:41:09.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7121000000000001e-05 2024-02-05T06:41:09.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:09.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.233343+0000", 2024-02-05T06:41:09.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1710000000000008e-06 2024-02-05T06:41:09.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.233423+0000", 2024-02-05T06:41:09.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0519999999999995e-05 2024-02-05T06:41:09.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:41:09.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.233438+0000", 2024-02-05T06:41:09.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.464e-05 2024-02-05T06:41:09.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.133226+0000", 2024-02-05T06:41:09.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.89978812100000005 2024-02-05T06:41:09.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:09.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.133340+0000", 2024-02-05T06:41:09.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011403799999999999 2024-02-05T06:41:09.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:41:09.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.133486+0000", 2024-02-05T06:41:09.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014550500000000001 2024-02-05T06:41:09.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:09.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.138483+0000", 2024-02-05T06:41:09.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049977900000000002 2024-02-05T06:41:09.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.141928+0000", 2024-02-05T06:41:09.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0034444779999999999 2024-02-05T06:41:09.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.150295+0000", 2024-02-05T06:41:09.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0083667529999999993 2024-02-05T06:41:09.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:09.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.150327+0000", 2024-02-05T06:41:09.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2446000000000001e-05 2024-02-05T06:41:09.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:09.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.150349+0000", 2024-02-05T06:41:09.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1801e-05 2024-02-05T06:41:09.210 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.210 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:09.210 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.210 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:47729 8.2 8:45730f30:::benchmark_data_smithi028_105698_object47728:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-02-05T06:41:09.210 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:33.251499+0000", 2024-02-05T06:41:09.210 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 35.890859501000001, 2024-02-05T06:41:09.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.89898365300000005, 2024-02-05T06:41:09.210 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:09.210 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:09.211 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:09.211 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:41:09.211 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:41:09.211 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 47729 2024-02-05T06:41:09.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.211 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:09.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:09.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.251499+0000", 2024-02-05T06:41:09.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:09.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.251499+0000", 2024-02-05T06:41:09.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:09.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.251501+0000", 2024-02-05T06:41:09.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6189999999999998e-06 2024-02-05T06:41:09.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:09.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.251542+0000", 2024-02-05T06:41:09.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0522000000000003e-05 2024-02-05T06:41:09.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:09.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.251556+0000", 2024-02-05T06:41:09.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4015999999999999e-05 2024-02-05T06:41:09.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:09.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.251565+0000", 2024-02-05T06:41:09.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5199999999999997e-06 2024-02-05T06:41:09.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.251631+0000", 2024-02-05T06:41:09.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6666000000000002e-05 2024-02-05T06:41:09.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:41:09.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.251644+0000", 2024-02-05T06:41:09.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3067e-05 2024-02-05T06:41:09.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.134398+0000", 2024-02-05T06:41:09.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.88275409299999996 2024-02-05T06:41:09.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:09.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.134509+0000", 2024-02-05T06:41:09.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000110549 2024-02-05T06:41:09.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:41:09.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.134646+0000", 2024-02-05T06:41:09.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013719900000000001 2024-02-05T06:41:09.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:09.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.139687+0000", 2024-02-05T06:41:09.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0050411060000000001 2024-02-05T06:41:09.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.145353+0000", 2024-02-05T06:41:09.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056655949999999998 2024-02-05T06:41:09.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.150433+0000", 2024-02-05T06:41:09.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0050798960000000004 2024-02-05T06:41:09.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:09.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.150462+0000", 2024-02-05T06:41:09.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.887e-05 2024-02-05T06:41:09.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:09.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.150483+0000", 2024-02-05T06:41:09.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0934999999999999e-05 2024-02-05T06:41:09.217 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.217 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:09.217 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.217 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:47751 8.2 8:4da41d20:::benchmark_data_smithi028_105698_object47750:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-02-05T06:41:09.217 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:33.277284+0000", 2024-02-05T06:41:09.217 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 35.865074685000003, 2024-02-05T06:41:09.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87330628499999996, 2024-02-05T06:41:09.217 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:09.217 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:09.218 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:09.218 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:41:09.218 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:41:09.218 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 47751 2024-02-05T06:41:09.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.218 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:09.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:09.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.277284+0000", 2024-02-05T06:41:09.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:09.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.277284+0000", 2024-02-05T06:41:09.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:09.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.277285+0000", 2024-02-05T06:41:09.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.658e-06 2024-02-05T06:41:09.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:09.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.277348+0000", 2024-02-05T06:41:09.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2583999999999996e-05 2024-02-05T06:41:09.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:09.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.277357+0000", 2024-02-05T06:41:09.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0089999999999996e-06 2024-02-05T06:41:09.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:09.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.277363+0000", 2024-02-05T06:41:09.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.477e-06 2024-02-05T06:41:09.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.277427+0000", 2024-02-05T06:41:09.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3110999999999999e-05 2024-02-05T06:41:09.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:41:09.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.277439+0000", 2024-02-05T06:41:09.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2318e-05 2024-02-05T06:41:09.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.135651+0000", 2024-02-05T06:41:09.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.85821182299999998 2024-02-05T06:41:09.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:09.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.135762+0000", 2024-02-05T06:41:09.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111431 2024-02-05T06:41:09.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:41:09.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.135890+0000", 2024-02-05T06:41:09.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000127441 2024-02-05T06:41:09.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:09.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.139694+0000", 2024-02-05T06:41:09.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00380431 2024-02-05T06:41:09.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.145414+0000", 2024-02-05T06:41:09.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005719933 2024-02-05T06:41:09.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.150544+0000", 2024-02-05T06:41:09.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0051303679999999997 2024-02-05T06:41:09.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:09.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.150572+0000", 2024-02-05T06:41:09.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7447999999999999e-05 2024-02-05T06:41:09.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:09.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.150590+0000", 2024-02-05T06:41:09.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8374000000000001e-05 2024-02-05T06:41:09.223 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.223 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:09.224 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.224 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:47754 8.2 8:4c1d8935:::benchmark_data_smithi028_105698_object47753:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-02-05T06:41:09.224 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:33.279349+0000", 2024-02-05T06:41:09.224 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 35.863009546000001, 2024-02-05T06:41:09.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.87133377099999998, 2024-02-05T06:41:09.224 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:09.224 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:09.224 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:09.224 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:41:09.224 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:41:09.224 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 47754 2024-02-05T06:41:09.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.225 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:09.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:09.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.279349+0000", 2024-02-05T06:41:09.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:09.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.279349+0000", 2024-02-05T06:41:09.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:09.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.279350+0000", 2024-02-05T06:41:09.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.297e-06 2024-02-05T06:41:09.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:09.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.279428+0000", 2024-02-05T06:41:09.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7438e-05 2024-02-05T06:41:09.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:09.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.279438+0000", 2024-02-05T06:41:09.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0900000000000001e-05 2024-02-05T06:41:09.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:09.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.279445+0000", 2024-02-05T06:41:09.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3400000000000003e-06 2024-02-05T06:41:09.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.279490+0000", 2024-02-05T06:41:09.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4950999999999997e-05 2024-02-05T06:41:09.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:41:09.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.279498+0000", 2024-02-05T06:41:09.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1899999999999995e-06 2024-02-05T06:41:09.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.137147+0000", 2024-02-05T06:41:09.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.85764903100000001 2024-02-05T06:41:09.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:09.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.137270+0000", 2024-02-05T06:41:09.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000122767 2024-02-05T06:41:09.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:41:09.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.137419+0000", 2024-02-05T06:41:09.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014935400000000001 2024-02-05T06:41:09.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:09.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.140730+0000", 2024-02-05T06:41:09.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0033108830000000001 2024-02-05T06:41:09.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.150396+0000", 2024-02-05T06:41:09.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0096660189999999997 2024-02-05T06:41:09.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.150619+0000", 2024-02-05T06:41:09.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022308199999999999 2024-02-05T06:41:09.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:09.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.150651+0000", 2024-02-05T06:41:09.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2141999999999998e-05 2024-02-05T06:41:09.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:09.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.150683+0000", 2024-02-05T06:41:09.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1377000000000001e-05 2024-02-05T06:41:09.230 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.230 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:09.230 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.231 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:47768 8.2 8:4b8ec012:::benchmark_data_smithi028_105698_object47767:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-02-05T06:41:09.231 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:33.291506+0000", 2024-02-05T06:41:09.231 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 35.850852336999999, 2024-02-05T06:41:09.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.85926484599999997, 2024-02-05T06:41:09.231 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:09.231 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:09.231 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:09.231 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:41:09.231 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:41:09.231 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 47768 2024-02-05T06:41:09.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.231 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:09.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:09.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.291506+0000", 2024-02-05T06:41:09.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:09.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.291506+0000", 2024-02-05T06:41:09.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:09.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.291515+0000", 2024-02-05T06:41:09.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1390000000000001e-06 2024-02-05T06:41:09.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:09.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.291560+0000", 2024-02-05T06:41:09.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4719999999999999e-05 2024-02-05T06:41:09.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:09.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.291574+0000", 2024-02-05T06:41:09.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4039000000000001e-05 2024-02-05T06:41:09.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:09.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.291583+0000", 2024-02-05T06:41:09.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1689999999999998e-06 2024-02-05T06:41:09.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.291642+0000", 2024-02-05T06:41:09.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8423e-05 2024-02-05T06:41:09.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:41:09.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.291650+0000", 2024-02-05T06:41:09.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3760000000000008e-06 2024-02-05T06:41:09.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.138594+0000", 2024-02-05T06:41:09.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.84694426899999997 2024-02-05T06:41:09.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:09.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.138709+0000", 2024-02-05T06:41:09.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000114875 2024-02-05T06:41:09.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:41:09.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.138853+0000", 2024-02-05T06:41:09.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014439499999999999 2024-02-05T06:41:09.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:09.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.141800+0000", 2024-02-05T06:41:09.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029470680000000002 2024-02-05T06:41:09.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.150515+0000", 2024-02-05T06:41:09.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0087146540000000005 2024-02-05T06:41:09.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.150710+0000", 2024-02-05T06:41:09.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000194988 2024-02-05T06:41:09.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:09.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.150737+0000", 2024-02-05T06:41:09.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7073999999999999e-05 2024-02-05T06:41:09.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:09.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.150771+0000", 2024-02-05T06:41:09.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3657e-05 2024-02-05T06:41:09.237 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.237 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:09.237 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.237 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:47779 8.2 8:4a5155c3:::benchmark_data_smithi028_105698_object47778:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-02-05T06:41:09.238 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:33.304878+0000", 2024-02-05T06:41:09.238 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 35.837480468000003, 2024-02-05T06:41:09.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.84648374000000004, 2024-02-05T06:41:09.238 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:09.238 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:09.238 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:09.238 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:41:09.238 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:41:09.238 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 47779 2024-02-05T06:41:09.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.238 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:09.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:09.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.304878+0000", 2024-02-05T06:41:09.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:09.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.304878+0000", 2024-02-05T06:41:09.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:09.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.304879+0000", 2024-02-05T06:41:09.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6020000000000001e-06 2024-02-05T06:41:09.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:09.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.304927+0000", 2024-02-05T06:41:09.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7960000000000002e-05 2024-02-05T06:41:09.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:09.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.304938+0000", 2024-02-05T06:41:09.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0779e-05 2024-02-05T06:41:09.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:09.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.304946+0000", 2024-02-05T06:41:09.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2899999999999997e-06 2024-02-05T06:41:09.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.305006+0000", 2024-02-05T06:41:09.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0611e-05 2024-02-05T06:41:09.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:41:09.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.305016+0000", 2024-02-05T06:41:09.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5100000000000004e-06 2024-02-05T06:41:09.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.139734+0000", 2024-02-05T06:41:09.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.83471860200000003 2024-02-05T06:41:09.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:09.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.139821+0000", 2024-02-05T06:41:09.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6747000000000001e-05 2024-02-05T06:41:09.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:41:09.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.139918+0000", 2024-02-05T06:41:09.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6663999999999995e-05 2024-02-05T06:41:09.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:09.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.141937+0000", 2024-02-05T06:41:09.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020195439999999999 2024-02-05T06:41:09.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.150807+0000", 2024-02-05T06:41:09.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088696139999999996 2024-02-05T06:41:09.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.151292+0000", 2024-02-05T06:41:09.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00048498399999999999 2024-02-05T06:41:09.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:09.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.151325+0000", 2024-02-05T06:41:09.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3389000000000002e-05 2024-02-05T06:41:09.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:09.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.151362+0000", 2024-02-05T06:41:09.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6443999999999999e-05 2024-02-05T06:41:09.244 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.244 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:09.244 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.244 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:47780 8.2 8:44ba1419:::benchmark_data_smithi028_105698_object47779:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-02-05T06:41:09.244 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:33.307061+0000", 2024-02-05T06:41:09.244 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 35.835297326999999, 2024-02-05T06:41:09.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.84650175500000002, 2024-02-05T06:41:09.244 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:09.245 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:09.245 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:09.245 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:41:09.245 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:41:09.245 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 47780 2024-02-05T06:41:09.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.245 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:09.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:09.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.307061+0000", 2024-02-05T06:41:09.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:09.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.307061+0000", 2024-02-05T06:41:09.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:09.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.307063+0000", 2024-02-05T06:41:09.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.559e-06 2024-02-05T06:41:09.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:09.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.307108+0000", 2024-02-05T06:41:09.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5618000000000003e-05 2024-02-05T06:41:09.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:09.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.307118+0000", 2024-02-05T06:41:09.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6059999999999992e-06 2024-02-05T06:41:09.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:09.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.307124+0000", 2024-02-05T06:41:09.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9460000000000003e-06 2024-02-05T06:41:09.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.307175+0000", 2024-02-05T06:41:09.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0883000000000001e-05 2024-02-05T06:41:09.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:41:09.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.307183+0000", 2024-02-05T06:41:09.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0800000000000006e-06 2024-02-05T06:41:09.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.140902+0000", 2024-02-05T06:41:09.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.83371917699999998 2024-02-05T06:41:09.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:09.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.140985+0000", 2024-02-05T06:41:09.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2748000000000001e-05 2024-02-05T06:41:09.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:41:09.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.141073+0000", 2024-02-05T06:41:09.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8128999999999996e-05 2024-02-05T06:41:09.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:09.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.145362+0000", 2024-02-05T06:41:09.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0042892679999999997 2024-02-05T06:41:09.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.150832+0000", 2024-02-05T06:41:09.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0054698660000000003 2024-02-05T06:41:09.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.153464+0000", 2024-02-05T06:41:09.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002632564 2024-02-05T06:41:09.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:09.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.153524+0000", 2024-02-05T06:41:09.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9423999999999999e-05 2024-02-05T06:41:09.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:09.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.153563+0000", 2024-02-05T06:41:09.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8887000000000002e-05 2024-02-05T06:41:09.251 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.251 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:09.251 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.251 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:47827 8.2 8:424b012f:::benchmark_data_smithi028_105698_object47826:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-02-05T06:41:09.251 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:33.380259+0000", 2024-02-05T06:41:09.251 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 35.762099655, 2024-02-05T06:41:09.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.79022138600000003, 2024-02-05T06:41:09.251 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:09.251 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:09.251 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:09.251 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:41:09.251 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:41:09.252 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 47827 2024-02-05T06:41:09.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.252 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:09.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:09.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.380259+0000", 2024-02-05T06:41:09.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:09.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.380259+0000", 2024-02-05T06:41:09.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:09.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.380262+0000", 2024-02-05T06:41:09.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9909999999999998e-06 2024-02-05T06:41:09.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:09.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.380305+0000", 2024-02-05T06:41:09.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3548999999999999e-05 2024-02-05T06:41:09.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:09.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.380322+0000", 2024-02-05T06:41:09.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.681e-05 2024-02-05T06:41:09.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:09.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.380333+0000", 2024-02-05T06:41:09.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0682999999999999e-05 2024-02-05T06:41:09.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.380391+0000", 2024-02-05T06:41:09.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7964999999999998e-05 2024-02-05T06:41:09.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:41:09.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.380400+0000", 2024-02-05T06:41:09.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.918e-06 2024-02-05T06:41:09.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.141969+0000", 2024-02-05T06:41:09.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.76156977199999998 2024-02-05T06:41:09.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:09.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.142078+0000", 2024-02-05T06:41:09.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000108786 2024-02-05T06:41:09.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:41:09.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.142206+0000", 2024-02-05T06:41:09.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000128152 2024-02-05T06:41:09.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:09.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.145368+0000", 2024-02-05T06:41:09.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031615089999999998 2024-02-05T06:41:09.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.152751+0000", 2024-02-05T06:41:09.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0073826830000000001 2024-02-05T06:41:09.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.170403+0000", 2024-02-05T06:41:09.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017652737000000002 2024-02-05T06:41:09.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:09.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.170442+0000", 2024-02-05T06:41:09.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8964000000000002e-05 2024-02-05T06:41:09.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:09.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.170480+0000", 2024-02-05T06:41:09.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7867000000000002e-05 2024-02-05T06:41:09.257 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.257 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:09.258 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.258 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:47849 8.2 8:4aed0be1:::benchmark_data_smithi028_105698_object47848:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-02-05T06:41:09.258 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:33.439371+0000", 2024-02-05T06:41:09.258 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 35.702987157999999, 2024-02-05T06:41:09.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.73127238299999997, 2024-02-05T06:41:09.258 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:09.258 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:09.258 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:09.258 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:41:09.258 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:41:09.258 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 47849 2024-02-05T06:41:09.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.259 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:09.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:09.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.439371+0000", 2024-02-05T06:41:09.259 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:09.260 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 74 16 75347 75331 63.6114 56.4375 0.00900523 0.0157145 2024-02-05T06:41:09.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:09.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.439371+0000", 2024-02-05T06:41:09.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:09.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.439373+0000", 2024-02-05T06:41:09.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.784e-06 2024-02-05T06:41:09.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:09.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.439416+0000", 2024-02-05T06:41:09.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2635000000000001e-05 2024-02-05T06:41:09.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:09.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.439429+0000", 2024-02-05T06:41:09.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3789e-05 2024-02-05T06:41:09.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:09.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.439436+0000", 2024-02-05T06:41:09.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8909999999999998e-06 2024-02-05T06:41:09.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.439503+0000", 2024-02-05T06:41:09.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6370000000000003e-05 2024-02-05T06:41:09.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:41:09.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.439514+0000", 2024-02-05T06:41:09.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1747e-05 2024-02-05T06:41:09.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.143032+0000", 2024-02-05T06:41:09.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.703517799 2024-02-05T06:41:09.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:09.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.143146+0000", 2024-02-05T06:41:09.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011404900000000001 2024-02-05T06:41:09.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:41:09.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.143256+0000", 2024-02-05T06:41:09.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000110003 2024-02-05T06:41:09.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:09.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.146339+0000", 2024-02-05T06:41:09.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030826220000000001 2024-02-05T06:41:09.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.152800+0000", 2024-02-05T06:41:09.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0064615489999999996 2024-02-05T06:41:09.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.170542+0000", 2024-02-05T06:41:09.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017741317999999999 2024-02-05T06:41:09.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:09.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.170602+0000", 2024-02-05T06:41:09.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0600999999999998e-05 2024-02-05T06:41:09.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:09.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.170644+0000", 2024-02-05T06:41:09.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1226000000000001e-05 2024-02-05T06:41:09.266 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.266 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:09.266 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.266 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:47851 8.2 8:438f1496:::benchmark_data_smithi028_105698_object47850:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-02-05T06:41:09.266 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:33.447583+0000", 2024-02-05T06:41:09.266 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 35.694774893000002, 2024-02-05T06:41:09.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.72314318, 2024-02-05T06:41:09.266 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:09.266 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:09.266 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:09.267 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:41:09.267 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:41:09.267 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 47851 2024-02-05T06:41:09.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.267 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:09.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:09.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.447583+0000", 2024-02-05T06:41:09.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:09.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.447583+0000", 2024-02-05T06:41:09.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:09.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.447585+0000", 2024-02-05T06:41:09.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5200000000000001e-06 2024-02-05T06:41:09.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:09.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.447634+0000", 2024-02-05T06:41:09.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8714999999999997e-05 2024-02-05T06:41:09.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:09.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.447647+0000", 2024-02-05T06:41:09.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3395e-05 2024-02-05T06:41:09.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:09.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.447656+0000", 2024-02-05T06:41:09.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8109999999999992e-06 2024-02-05T06:41:09.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.447720+0000", 2024-02-05T06:41:09.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4288000000000005e-05 2024-02-05T06:41:09.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:41:09.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.447735+0000", 2024-02-05T06:41:09.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4450999999999999e-05 2024-02-05T06:41:09.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.144109+0000", 2024-02-05T06:41:09.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.69637437199999996 2024-02-05T06:41:09.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:09.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.144210+0000", 2024-02-05T06:41:09.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101458 2024-02-05T06:41:09.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:41:09.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.144343+0000", 2024-02-05T06:41:09.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013213 2024-02-05T06:41:09.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:09.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.148805+0000", 2024-02-05T06:41:09.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.004462728 2024-02-05T06:41:09.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.162760+0000", 2024-02-05T06:41:09.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013954783 2024-02-05T06:41:09.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.170677+0000", 2024-02-05T06:41:09.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079168269999999995 2024-02-05T06:41:09.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:09.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.170706+0000", 2024-02-05T06:41:09.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9295e-05 2024-02-05T06:41:09.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:09.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.170727+0000", 2024-02-05T06:41:09.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0407e-05 2024-02-05T06:41:09.272 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.272 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:09.273 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.273 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:47854 8.2 8:4daf0ffa:::benchmark_data_smithi028_105698_object47853:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-02-05T06:41:09.273 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:33.448630+0000", 2024-02-05T06:41:09.273 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 35.693728518999997, 2024-02-05T06:41:09.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.72531150600000005, 2024-02-05T06:41:09.273 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:09.273 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:09.273 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:09.273 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:41:09.273 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:41:09.273 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 47854 2024-02-05T06:41:09.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.273 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:09.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:09.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.448630+0000", 2024-02-05T06:41:09.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:09.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.448630+0000", 2024-02-05T06:41:09.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:09.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.448631+0000", 2024-02-05T06:41:09.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2890000000000001e-06 2024-02-05T06:41:09.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:09.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.448693+0000", 2024-02-05T06:41:09.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2197000000000001e-05 2024-02-05T06:41:09.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:09.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.448703+0000", 2024-02-05T06:41:09.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2119999999999992e-06 2024-02-05T06:41:09.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:09.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.448708+0000", 2024-02-05T06:41:09.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4659999999999998e-06 2024-02-05T06:41:09.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.448766+0000", 2024-02-05T06:41:09.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8411e-05 2024-02-05T06:41:09.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:41:09.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.448776+0000", 2024-02-05T06:41:09.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3740000000000004e-06 2024-02-05T06:41:09.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.145460+0000", 2024-02-05T06:41:09.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.69668377999999997 2024-02-05T06:41:09.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:09.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.145530+0000", 2024-02-05T06:41:09.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0796000000000005e-05 2024-02-05T06:41:09.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:41:09.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.145614+0000", 2024-02-05T06:41:09.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3251000000000006e-05 2024-02-05T06:41:09.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:09.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.150256+0000", 2024-02-05T06:41:09.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0046418969999999999 2024-02-05T06:41:09.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.162796+0000", 2024-02-05T06:41:09.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0125408 2024-02-05T06:41:09.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.173848+0000", 2024-02-05T06:41:09.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011051523000000001 2024-02-05T06:41:09.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:09.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.173903+0000", 2024-02-05T06:41:09.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4789e-05 2024-02-05T06:41:09.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:09.279 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.173941+0000", 2024-02-05T06:41:09.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8720999999999997e-05 2024-02-05T06:41:09.279 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.279 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:09.279 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.279 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:47886 8.2 8:4fe7f2ad:::benchmark_data_smithi028_105698_object47885:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-02-05T06:41:09.279 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:33.539267+0000", 2024-02-05T06:41:09.279 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 35.603091743999997, 2024-02-05T06:41:09.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.70017919500000003, 2024-02-05T06:41:09.279 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:09.280 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:09.280 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:09.280 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:41:09.280 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:41:09.280 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 47886 2024-02-05T06:41:09.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.281 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:09.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:09.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.539267+0000", 2024-02-05T06:41:09.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:09.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.539267+0000", 2024-02-05T06:41:09.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:09.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.539269+0000", 2024-02-05T06:41:09.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4849999999999999e-06 2024-02-05T06:41:09.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:09.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.539305+0000", 2024-02-05T06:41:09.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5763999999999998e-05 2024-02-05T06:41:09.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:09.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.539321+0000", 2024-02-05T06:41:09.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5744999999999999e-05 2024-02-05T06:41:09.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:09.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.539330+0000", 2024-02-05T06:41:09.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9760000000000001e-06 2024-02-05T06:41:09.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.539391+0000", 2024-02-05T06:41:09.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0970000000000001e-05 2024-02-05T06:41:09.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:41:09.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.539399+0000", 2024-02-05T06:41:09.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6470000000000004e-06 2024-02-05T06:41:09.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.146373+0000", 2024-02-05T06:41:09.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.60697373300000002 2024-02-05T06:41:09.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:09.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.146468+0000", 2024-02-05T06:41:09.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4987000000000004e-05 2024-02-05T06:41:09.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:41:09.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.146577+0000", 2024-02-05T06:41:09.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000109196 2024-02-05T06:41:09.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:09.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.150260+0000", 2024-02-05T06:41:09.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0036826480000000002 2024-02-05T06:41:09.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.162829+0000", 2024-02-05T06:41:09.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012569366 2024-02-05T06:41:09.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.239354+0000", 2024-02-05T06:41:09.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076524691000000006 2024-02-05T06:41:09.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:09.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.239398+0000", 2024-02-05T06:41:09.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4135999999999999e-05 2024-02-05T06:41:09.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:09.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.239446+0000", 2024-02-05T06:41:09.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7851e-05 2024-02-05T06:41:09.287 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.287 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:09.287 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.287 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:47902 8.2 8:4046ff50:::benchmark_data_smithi028_105698_object47901:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-02-05T06:41:09.287 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:33.604502+0000", 2024-02-05T06:41:09.287 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 35.537856054999999, 2024-02-05T06:41:09.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.63510125200000001, 2024-02-05T06:41:09.287 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:09.288 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:09.288 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:09.288 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:41:09.288 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:41:09.288 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 47902 2024-02-05T06:41:09.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.288 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:09.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:09.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.604502+0000", 2024-02-05T06:41:09.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:09.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.604502+0000", 2024-02-05T06:41:09.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:09.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:09.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.604505+0000", 2024-02-05T06:41:09.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5509999999999998e-06 2024-02-05T06:41:09.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:09.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.604565+0000", 2024-02-05T06:41:09.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9701999999999999e-05 2024-02-05T06:41:09.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:09.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.604580+0000", 2024-02-05T06:41:09.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5265e-05 2024-02-05T06:41:09.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:09.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.604590+0000", 2024-02-05T06:41:09.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9140000000000003e-06 2024-02-05T06:41:09.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.604653+0000", 2024-02-05T06:41:09.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3486999999999997e-05 2024-02-05T06:41:09.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:41:09.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.604661+0000", 2024-02-05T06:41:09.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.242e-06 2024-02-05T06:41:09.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:09.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.147650+0000", 2024-02-05T06:41:09.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.54298826 2024-02-05T06:41:09.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:09.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.147783+0000", 2024-02-05T06:41:09.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013367199999999999 2024-02-05T06:41:09.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:41:09.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.147930+0000", 2024-02-05T06:41:09.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014656900000000001 2024-02-05T06:41:09.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:09.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.150405+0000", 2024-02-05T06:41:09.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0024745869999999999 2024-02-05T06:41:09.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.162870+0000", 2024-02-05T06:41:09.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012465425 2024-02-05T06:41:09.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:09.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.239504+0000", 2024-02-05T06:41:09.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076634498999999995 2024-02-05T06:41:09.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:09.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.239563+0000", 2024-02-05T06:41:09.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8431999999999999e-05 2024-02-05T06:41:09.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:09.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:09.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:09.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.239604+0000", 2024-02-05T06:41:09.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0646999999999999e-05 2024-02-05T06:41:09.293 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.293 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:09.294 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.294 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:09.294 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:09.294 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:41:09.321 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:09.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:09.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:09.423 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:09.463 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:09.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:09.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:09.565 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:09.603 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:09.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:09.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:09.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:09 smithi028 ceph-mon[37984]: pgmap v362: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 5223/219879 objects misplaced (2.375%); 15 MiB/s, 243 objects/s recovering 2024-02-05T06:41:09.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:09 smithi028 ceph-mon[37984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:41:09.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:09 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3988072782' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:41:09.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:09 smithi028 ceph-mon[37984]: osdmap e233: 8 total, 8 up, 7 in 2024-02-05T06:41:09.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:09 smithi028 ceph-mon[29442]: pgmap v362: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 5223/219879 objects misplaced (2.375%); 15 MiB/s, 243 objects/s recovering 2024-02-05T06:41:09.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:09 smithi028 ceph-mon[29442]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:41:09.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:09 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3988072782' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:41:09.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:09 smithi028 ceph-mon[29442]: osdmap e233: 8 total, 8 up, 7 in 2024-02-05T06:41:09.705 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:09.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:09 smithi167 ceph-mon[32774]: pgmap v362: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.03k op/s; 5223/219879 objects misplaced (2.375%); 15 MiB/s, 243 objects/s recovering 2024-02-05T06:41:09.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:09 smithi167 ceph-mon[32774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:41:09.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:09 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3988072782' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:41:09.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:09 smithi167 ceph-mon[32774]: osdmap e233: 8 total, 8 up, 7 in 2024-02-05T06:41:09.736 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:09.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:09.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:09.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:09.880 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:09.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:09.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:09.982 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:10.008 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:10.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:10.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:10.110 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:10.122 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-05T06:41:10.158 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:10.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:10.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:10.236 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 75 16 76275 76259 63.5365 58 0.0353309 0.0157311 2024-02-05T06:41:10.260 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:10.286 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:10.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:10.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:10.388 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:10.406 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:41:10.428 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:10.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:10.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:10.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:10.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:10.577 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:10.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:10.678 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:10.708 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:10.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:10.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:10.810 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:10.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:10.854 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:10.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:10.955 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:10.987 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:10.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:10.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:11.089 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:11.117 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:11.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:11.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:11.219 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:11.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:11 smithi167 ceph-mon[32774]: osdmap e234: 8 total, 8 up, 7 in 2024-02-05T06:41:11.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:11 smithi167 ceph-mon[32774]: pgmap v365: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 50 MiB/s wr, 793 op/s; 5223/219879 objects misplaced (2.375%); 6.9 MiB/s, 110 objects/s recovering 2024-02-05T06:41:11.236 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 76 16 77029 77013 63.3205 47.125 0.0086489 0.0157868 2024-02-05T06:41:11.264 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:11.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:11.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:11.367 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:11.396 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:11.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:11.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:11.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:11 smithi028 ceph-mon[29442]: osdmap e234: 8 total, 8 up, 7 in 2024-02-05T06:41:11.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:11 smithi028 ceph-mon[29442]: pgmap v365: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 50 MiB/s wr, 793 op/s; 5223/219879 objects misplaced (2.375%); 6.9 MiB/s, 110 objects/s recovering 2024-02-05T06:41:11.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:11 smithi028 ceph-mon[37984]: osdmap e234: 8 total, 8 up, 7 in 2024-02-05T06:41:11.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:11 smithi028 ceph-mon[37984]: pgmap v365: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 853 B/s rd, 50 MiB/s wr, 793 op/s; 5223/219879 objects misplaced (2.375%); 6.9 MiB/s, 110 objects/s recovering 2024-02-05T06:41:11.498 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:11.526 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:11.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:11.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:11.628 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:11.676 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:11.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:11.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:11.777 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:11.814 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:11.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:11.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:11.916 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:11.941 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:11.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:11.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:12.043 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:12.073 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:12.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:12.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:12.177 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:12.206 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:12.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:12.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:12.237 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 77 16 77895 77879 63.201 54.125 0.0224671 0.0158169 2024-02-05T06:41:12.308 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:12.358 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:12.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:12.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:12.374 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:41:12.374 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-02-05T06:41:12.374 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-02-05T06:41:12.374 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:41:12.460 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:12.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:12 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:41:12.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:12 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:41:12.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:12 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:41:12.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:12.517 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:12.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:12.618 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:12.649 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:12.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:12.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:12.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:12 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:41:12.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:12 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:41:12.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:12 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:41:12.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:12 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:41:12.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:12 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:41:12.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:12 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:41:12.698 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:41:12 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:41:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:41:12.751 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:12.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:12.781 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:12.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:12.882 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:12.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:12.912 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:12.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:13.013 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:13.044 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:13.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:13.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:13.087 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:41:12 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:41:13.125 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-05T06:41:13.147 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:13.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:13.192 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:13.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:13.237 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 78 16 78741 78725 63.0686 52.875 0.0174177 0.01585 2024-02-05T06:41:13.293 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:13.335 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:13.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:13.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:13.437 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:13.461 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:41:13.468 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:13.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:13.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:13.569 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:13.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:13.616 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:13.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:13.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:13 smithi028 ceph-mon[37984]: pgmap v366: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.4 KiB/s rd, 77 MiB/s wr, 1.23k op/s; 4563/232512 objects misplaced (1.962%); 11 MiB/s, 181 objects/s recovering 2024-02-05T06:41:13.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:13 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:41:13.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:13 smithi028 ceph-mon[29442]: pgmap v366: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.4 KiB/s rd, 77 MiB/s wr, 1.23k op/s; 4563/232512 objects misplaced (1.962%); 11 MiB/s, 181 objects/s recovering 2024-02-05T06:41:13.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:13 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:41:13.717 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:13.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:13 smithi167 ceph-mon[32774]: pgmap v366: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.4 KiB/s rd, 77 MiB/s wr, 1.23k op/s; 4563/232512 objects misplaced (1.962%); 11 MiB/s, 181 objects/s recovering 2024-02-05T06:41:13.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:13 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:41:13.745 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:13.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:13.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:13.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:13.888 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:13.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:13.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:13.990 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:14.030 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:14.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:14.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:14.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:14.181 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:14.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:14.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:14.237 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 79 16 79678 79662 63.0115 58.5625 0.0168263 0.0158637 2024-02-05T06:41:14.283 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:14.312 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:14.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:14.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:14.413 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:14.436 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:14.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:14.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:14.538 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:14.563 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:14.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:14.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:14.665 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:14.700 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:14.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:14.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:14.801 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:14.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:14.844 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:14.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:14.945 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:14.975 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:14.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:14.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:15.077 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:15.108 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:15.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:15.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:15.210 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:15.237 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-02-05T06:41:15.233108+0000 min lat: 0.00125405 max lat: 2.2409 avg lat: 0.0158848 2024-02-05T06:41:15.237 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T06:41:15.237 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 80 16 80582 80566 62.93 56.5 0.0212957 0.0158848 2024-02-05T06:41:15.252 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:15.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:15.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:15.353 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:15.395 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:15.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:15.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:15.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:15 smithi167 ceph-mon[32774]: pgmap v367: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 58 MiB/s wr, 935 op/s; 4563/232512 objects misplaced (1.962%); 10 MiB/s, 165 objects/s recovering 2024-02-05T06:41:15.497 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:15.528 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:15.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:15.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:15.630 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:15.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:15.662 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:15.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:15.685 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:15 smithi028 ceph-mon[37984]: pgmap v367: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 58 MiB/s wr, 935 op/s; 4563/232512 objects misplaced (1.962%); 10 MiB/s, 165 objects/s recovering 2024-02-05T06:41:15.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:15 smithi028 ceph-mon[29442]: pgmap v367: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.7 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 58 MiB/s wr, 935 op/s; 4563/232512 objects misplaced (1.962%); 10 MiB/s, 165 objects/s recovering 2024-02-05T06:41:15.763 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:15.805 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:15.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:15.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:15.906 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:15.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:15.961 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:15.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:16.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:16.115 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:16.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:16.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:16.217 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:16.238 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 81 16 81545 81529 62.8954 60.1875 0.0199364 0.0158933 2024-02-05T06:41:16.248 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:16.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:16.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:16.350 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:16.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:16.392 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:16.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:16.477 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:16 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:41:16.477 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:16 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:41:16.477 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:16 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:41:16.493 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:16.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:16.517 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:16.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:16.618 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:16.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:16.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:16.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:16 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:41:16.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:16 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:41:16.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:16 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:41:16.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:16 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:41:16.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:16 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:41:16.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:16 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:41:16.762 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:16.778 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:41:16.779 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-02-05T06:41:16.779 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-02-05T06:41:16.779 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-02-05T06:41:16.779 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:41:16.792 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:16.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:16.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:16.894 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:16.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:16.923 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:16.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:17.025 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:17.075 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:17.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:17.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:17.081 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:41:17.177 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:17.207 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:17.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:17.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:17.238 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 82 16 82418 82402 62.7937 54.5625 0.0173964 0.0159198 2024-02-05T06:41:17.311 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:17.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:17.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:17.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:17 smithi028 ceph-mon[29442]: pgmap v368: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.8 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 44 MiB/s wr, 699 op/s; 4563/236652 objects misplaced (1.928%); 5.2 MiB/s, 82 objects/s recovering 2024-02-05T06:41:17.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:17 smithi028 ceph-mon[37984]: pgmap v368: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.8 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 44 MiB/s wr, 699 op/s; 4563/236652 objects misplaced (1.928%); 5.2 MiB/s, 82 objects/s recovering 2024-02-05T06:41:17.467 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:17.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:17 smithi167 ceph-mon[32774]: pgmap v368: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 4.8 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 44 MiB/s wr, 699 op/s; 4563/236652 objects misplaced (1.928%); 5.2 MiB/s, 82 objects/s recovering 2024-02-05T06:41:17.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T06:41:17.492 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:17.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:17.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:17.503 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-05T06:41:17.594 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:17.646 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:17.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:17.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:17.748 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:17.783 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:17.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:17.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:17.871 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:41:17.885 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:17.927 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:17.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:17.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:18.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:18.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:18.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:18.209 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:18.239 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 83 16 83286 83270 62.6908 54.25 0.0193216 0.0159458 2024-02-05T06:41:18.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:18.263 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:18.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:18.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:18.419 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:18.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:18.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:18.588 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:18.634 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:18.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:18.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:18.671 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:41:18.673 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:41:18.736 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:18.793 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:18.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:18.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:18.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:18.944 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:18.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:18.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:19.046 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:19.093 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:19.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:19.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:19.194 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:19.220 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:41:19.224 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-05T06:41:19.236 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:19.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:19.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:19.240 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 84 16 84165 84149 62.5984 54.9375 0.0223389 0.0159678 2024-02-05T06:41:19.341 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:19.388 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:19.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:19.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:19.425 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:19 smithi028 ceph-mon[29442]: pgmap v369: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 55 MiB/s wr, 885 op/s; 3771/246243 objects misplaced (1.531%); 9.1 MiB/s, 146 objects/s recovering 2024-02-05T06:41:19.426 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:19 smithi028 ceph-mon[37984]: pgmap v369: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 55 MiB/s wr, 885 op/s; 3771/246243 objects misplaced (1.531%); 9.1 MiB/s, 146 objects/s recovering 2024-02-05T06:41:19.490 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:19.528 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:19.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:19.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:19.630 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:19.638 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:19 smithi167 ceph-mon[32774]: pgmap v369: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 55 MiB/s wr, 885 op/s; 3771/246243 objects misplaced (1.531%); 9.1 MiB/s, 146 objects/s recovering 2024-02-05T06:41:19.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:19.669 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:19.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:19.770 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:19.795 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:19.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:19.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:19.803 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:41:19.803 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-02-05T06:41:19.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-02-05T06:41:19.803 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-02-05T06:41:19.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.803 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.44375.0:936 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch reconnect cookie 94598155040256 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-02-05T06:41:19.803 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:33:54.875558+0000", 2024-02-05T06:41:19.803 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 444.92037226000002, 2024-02-05T06:41:19.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.012695831, 2024-02-05T06:41:19.804 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:19.804 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-02-05T06:41:19.804 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:19.804 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.44375", 2024-02-05T06:41:19.804 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/1937500867", 2024-02-05T06:41:19.804 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 936 2024-02-05T06:41:19.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.804 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:19.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:19.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:33:54.875558+0000", 2024-02-05T06:41:19.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:19.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:33:54.875558+0000", 2024-02-05T06:41:19.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:19.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:33:54.875562+0000", 2024-02-05T06:41:19.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4320000000000001e-06 2024-02-05T06:41:19.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:19.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:33:54.875576+0000", 2024-02-05T06:41:19.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3477e-05 2024-02-05T06:41:19.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:19.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:33:54.875603+0000", 2024-02-05T06:41:19.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7708999999999998e-05 2024-02-05T06:41:19.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:19.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:33:54.875615+0000", 2024-02-05T06:41:19.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.145e-05 2024-02-05T06:41:19.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:19.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:33:54.881088+0000", 2024-02-05T06:41:19.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0054734140000000002 2024-02-05T06:41:19.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.807 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:41:19.807 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:33:54.881097+0000", 2024-02-05T06:41:19.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4030000000000007e-06 2024-02-05T06:41:19.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:19.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:33:55.884992+0000", 2024-02-05T06:41:19.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0038959000000001 2024-02-05T06:41:19.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-02-05T06:41:19.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:33:55.885136+0000", 2024-02-05T06:41:19.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000143783 2024-02-05T06:41:19.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:19.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:33:55.886317+0000", 2024-02-05T06:41:19.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0011804610000000001 2024-02-05T06:41:19.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for rw locks", 2024-02-05T06:41:19.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:33:55.886441+0000", 2024-02-05T06:41:19.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000124623 2024-02-05T06:41:19.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:19.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:33:55.886765+0000", 2024-02-05T06:41:19.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032335699999999997 2024-02-05T06:41:19.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:19.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:33:55.886826+0000", 2024-02-05T06:41:19.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1723000000000005e-05 2024-02-05T06:41:19.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:19.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:33:55.888253+0000", 2024-02-05T06:41:19.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001427099 2024-02-05T06:41:19.811 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.811 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:19.811 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.811 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.44375.0:1111 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94598155037952 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-02-05T06:41:19.811 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:35:23.909001+0000", 2024-02-05T06:41:19.811 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 355.88692861499999, 2024-02-05T06:41:19.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.038390352, 2024-02-05T06:41:19.811 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:19.812 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-02-05T06:41:19.812 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:19.812 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.44375", 2024-02-05T06:41:19.812 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/1937500867", 2024-02-05T06:41:19.812 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1111 2024-02-05T06:41:19.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.812 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:19.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:19.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909001+0000", 2024-02-05T06:41:19.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:19.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909001+0000", 2024-02-05T06:41:19.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.813 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:19.813 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909003+0000", 2024-02-05T06:41:19.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0389999999999999e-06 2024-02-05T06:41:19.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:19.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909009+0000", 2024-02-05T06:41:19.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.361e-06 2024-02-05T06:41:19.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:19.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909023+0000", 2024-02-05T06:41:19.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4282e-05 2024-02-05T06:41:19.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:19.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909031+0000", 2024-02-05T06:41:19.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8860000000000004e-06 2024-02-05T06:41:19.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:19.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909101+0000", 2024-02-05T06:41:19.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0049999999999995e-05 2024-02-05T06:41:19.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-02-05T06:41:19.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909109+0000", 2024-02-05T06:41:19.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0600000000000008e-06 2024-02-05T06:41:19.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:19.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.917423+0000", 2024-02-05T06:41:19.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0083142839999999999 2024-02-05T06:41:19.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:41:19.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.917433+0000", 2024-02-05T06:41:19.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8470000000000008e-06 2024-02-05T06:41:19.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:19.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.931694+0000", 2024-02-05T06:41:19.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.014260489 2024-02-05T06:41:19.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for degraded object", 2024-02-05T06:41:19.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.932024+0000", 2024-02-05T06:41:19.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000330565 2024-02-05T06:41:19.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:19.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.942678+0000", 2024-02-05T06:41:19.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010653953000000001 2024-02-05T06:41:19.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:19.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.942792+0000", 2024-02-05T06:41:19.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011345000000000001 2024-02-05T06:41:19.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:19.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.947392+0000", 2024-02-05T06:41:19.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0046000859999999998 2024-02-05T06:41:19.819 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.819 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:19.819 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.819 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.44375.0:1120 5.e 5:7759931f:::notify.3:head [watch ping cookie 94598155037952 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-02-05T06:41:19.819 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:35:24.614910+0000", 2024-02-05T06:41:19.819 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 355.18101946500002, 2024-02-05T06:41:19.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.32873784299999997, 2024-02-05T06:41:19.820 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:19.820 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-02-05T06:41:19.821 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:19.821 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.44375", 2024-02-05T06:41:19.821 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/1937500867", 2024-02-05T06:41:19.821 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1120 2024-02-05T06:41:19.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.821 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:19.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:19.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.614910+0000", 2024-02-05T06:41:19.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:19.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.614910+0000", 2024-02-05T06:41:19.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:19.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.614911+0000", 2024-02-05T06:41:19.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9899999999999999e-07 2024-02-05T06:41:19.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:19.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.614915+0000", 2024-02-05T06:41:19.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3859999999999999e-06 2024-02-05T06:41:19.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:19.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.614922+0000", 2024-02-05T06:41:19.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4139999999999997e-06 2024-02-05T06:41:19.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:19.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.614933+0000", 2024-02-05T06:41:19.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1131999999999999e-05 2024-02-05T06:41:19.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:19.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.614999+0000", 2024-02-05T06:41:19.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5753000000000006e-05 2024-02-05T06:41:19.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:41:19.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.615010+0000", 2024-02-05T06:41:19.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0791999999999999e-05 2024-02-05T06:41:19.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:19.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.932060+0000", 2024-02-05T06:41:19.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.31705008800000001 2024-02-05T06:41:19.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for degraded object", 2024-02-05T06:41:19.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.932108+0000", 2024-02-05T06:41:19.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8208999999999999e-05 2024-02-05T06:41:19.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:19.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.943359+0000", 2024-02-05T06:41:19.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011250829 2024-02-05T06:41:19.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:19.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.943455+0000", 2024-02-05T06:41:19.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5778000000000004e-05 2024-02-05T06:41:19.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:19.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.943648+0000", 2024-02-05T06:41:19.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019366300000000001 2024-02-05T06:41:19.826 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.826 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:19.826 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.826 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:194790 7.0 7:0396bf8b:::benchmark_data_smithi028_77509_object79670:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-02-05T06:41:19.826 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:38:42.739592+0000", 2024-02-05T06:41:19.826 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 157.056338122, 2024-02-05T06:41:19.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.111845179, 2024-02-05T06:41:19.826 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:19.826 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:19.827 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:19.827 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:41:19.827 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:41:19.827 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 194790 2024-02-05T06:41:19.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.827 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:19.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:19.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:42.739592+0000", 2024-02-05T06:41:19.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:19.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:42.739592+0000", 2024-02-05T06:41:19.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:19.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:42.739593+0000", 2024-02-05T06:41:19.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7999999999999993e-07 2024-02-05T06:41:19.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:19.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:42.739596+0000", 2024-02-05T06:41:19.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5990000000000002e-06 2024-02-05T06:41:19.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:19.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:42.739605+0000", 2024-02-05T06:41:19.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.952e-06 2024-02-05T06:41:19.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:19.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:42.739611+0000", 2024-02-05T06:41:19.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2569999999999998e-06 2024-02-05T06:41:19.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:19.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:42.739640+0000", 2024-02-05T06:41:19.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9581e-05 2024-02-05T06:41:19.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:19.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:42.739711+0000", 2024-02-05T06:41:19.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0742999999999997e-05 2024-02-05T06:41:19.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:41:19.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:42.739788+0000", 2024-02-05T06:41:19.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6884000000000005e-05 2024-02-05T06:41:19.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:19.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:42.741370+0000", 2024-02-05T06:41:19.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00158256 2024-02-05T06:41:19.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:19.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:42.741805+0000", 2024-02-05T06:41:19.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00043510199999999998 2024-02-05T06:41:19.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:19.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:42.851376+0000", 2024-02-05T06:41:19.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.109570336 2024-02-05T06:41:19.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:19.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:42.851412+0000", 2024-02-05T06:41:19.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6572000000000002e-05 2024-02-05T06:41:19.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.832 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:19.832 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:42.851437+0000", 2024-02-05T06:41:19.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4613e-05 2024-02-05T06:41:19.832 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.832 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:19.832 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.832 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:195824 7.0 7:0029c504:::benchmark_data_smithi028_77509_object80704:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-02-05T06:41:19.832 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:38:43.931205+0000", 2024-02-05T06:41:19.832 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 155.86472461299999, 2024-02-05T06:41:19.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20595250700000001, 2024-02-05T06:41:19.832 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:19.832 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:19.833 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:19.833 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:41:19.833 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:41:19.833 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 195824 2024-02-05T06:41:19.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.833 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:19.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:19.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:43.931205+0000", 2024-02-05T06:41:19.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:19.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:43.931205+0000", 2024-02-05T06:41:19.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:19.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:43.931207+0000", 2024-02-05T06:41:19.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.248e-06 2024-02-05T06:41:19.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:19.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:43.931214+0000", 2024-02-05T06:41:19.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1210000000000001e-06 2024-02-05T06:41:19.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:19.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:43.931222+0000", 2024-02-05T06:41:19.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9109999999999993e-06 2024-02-05T06:41:19.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:19.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:43.931230+0000", 2024-02-05T06:41:19.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6200000000000005e-06 2024-02-05T06:41:19.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:19.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:43.934486+0000", 2024-02-05T06:41:19.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032558589999999998 2024-02-05T06:41:19.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:19.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:43.934556+0000", 2024-02-05T06:41:19.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9919999999999995e-05 2024-02-05T06:41:19.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:41:19.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:43.934625+0000", 2024-02-05T06:41:19.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9425999999999996e-05 2024-02-05T06:41:19.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:19.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:43.935911+0000", 2024-02-05T06:41:19.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00128513 2024-02-05T06:41:19.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:19.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:43.942273+0000", 2024-02-05T06:41:19.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.006362482 2024-02-05T06:41:19.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:19.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:44.137035+0000", 2024-02-05T06:41:19.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.19476223200000001 2024-02-05T06:41:19.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:19.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:44.137105+0000", 2024-02-05T06:41:19.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9675e-05 2024-02-05T06:41:19.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:19.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:44.137158+0000", 2024-02-05T06:41:19.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2883000000000002e-05 2024-02-05T06:41:19.838 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.838 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:19.838 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.839 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:226758 7.0 7:0424680d:::benchmark_data_smithi028_77509_object111638:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e174)", 2024-02-05T06:41:19.839 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:39:13.342211+0000", 2024-02-05T06:41:19.839 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 126.45371918399999, 2024-02-05T06:41:19.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.143588876, 2024-02-05T06:41:19.839 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:19.839 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:19.839 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:19.839 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:41:19.839 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:41:19.839 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 226758 2024-02-05T06:41:19.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.839 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:19.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:19.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:39:13.342211+0000", 2024-02-05T06:41:19.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:19.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:39:13.342211+0000", 2024-02-05T06:41:19.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:19.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:39:13.342213+0000", 2024-02-05T06:41:19.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0360000000000001e-06 2024-02-05T06:41:19.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:19.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:39:13.342218+0000", 2024-02-05T06:41:19.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.643e-06 2024-02-05T06:41:19.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:19.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:39:13.342231+0000", 2024-02-05T06:41:19.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2303000000000001e-05 2024-02-05T06:41:19.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:19.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:39:13.342238+0000", 2024-02-05T06:41:19.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5379999999999997e-06 2024-02-05T06:41:19.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:19.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:39:13.342285+0000", 2024-02-05T06:41:19.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7116000000000001e-05 2024-02-05T06:41:19.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:19.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:39:13.342369+0000", 2024-02-05T06:41:19.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3453000000000002e-05 2024-02-05T06:41:19.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:41:19.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:39:13.342438+0000", 2024-02-05T06:41:19.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9585000000000005e-05 2024-02-05T06:41:19.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:19.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:39:13.344433+0000", 2024-02-05T06:41:19.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019948119999999999 2024-02-05T06:41:19.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.844 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:19.844 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:39:13.344786+0000", 2024-02-05T06:41:19.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00035263200000000002 2024-02-05T06:41:19.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:19.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:39:13.485702+0000", 2024-02-05T06:41:19.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14091648500000001 2024-02-05T06:41:19.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:19.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:39:13.485747+0000", 2024-02-05T06:41:19.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4413000000000003e-05 2024-02-05T06:41:19.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:19.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:39:13.485800+0000", 2024-02-05T06:41:19.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2859999999999999e-05 2024-02-05T06:41:19.845 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.846 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:19.846 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.846 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:25583 8.4 8:2fcebb9c:::benchmark_data_smithi028_105698_object25582:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-02-05T06:41:19.846 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:04.948440+0000", 2024-02-05T06:41:19.846 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 74.847489714999995, 2024-02-05T06:41:19.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10085525300000001, 2024-02-05T06:41:19.846 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:19.847 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:19.847 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:19.847 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:41:19.847 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:41:19.847 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 25583 2024-02-05T06:41:19.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.847 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:19.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:19.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.948440+0000", 2024-02-05T06:41:19.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:19.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.948440+0000", 2024-02-05T06:41:19.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:19.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.948442+0000", 2024-02-05T06:41:19.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8300000000000001e-06 2024-02-05T06:41:19.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:19.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.948551+0000", 2024-02-05T06:41:19.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010851799999999999 2024-02-05T06:41:19.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:19.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.948570+0000", 2024-02-05T06:41:19.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9772e-05 2024-02-05T06:41:19.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:19.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.948583+0000", 2024-02-05T06:41:19.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2376e-05 2024-02-05T06:41:19.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.850 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:19.850 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.948652+0000", 2024-02-05T06:41:19.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8849000000000004e-05 2024-02-05T06:41:19.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.850 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:19.850 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.948781+0000", 2024-02-05T06:41:19.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000129572 2024-02-05T06:41:19.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.851 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:41:19.851 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.948925+0000", 2024-02-05T06:41:19.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014407499999999999 2024-02-05T06:41:19.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.851 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:19.851 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.950217+0000", 2024-02-05T06:41:19.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012917600000000001 2024-02-05T06:41:19.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:19.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.951107+0000", 2024-02-05T06:41:19.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00088959499999999997 2024-02-05T06:41:19.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:19.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.049190+0000", 2024-02-05T06:41:19.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098083341000000004 2024-02-05T06:41:19.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:19.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.049247+0000", 2024-02-05T06:41:19.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.753e-05 2024-02-05T06:41:19.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:19.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.049295+0000", 2024-02-05T06:41:19.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8034999999999997e-05 2024-02-05T06:41:19.854 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.854 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:19.854 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.854 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:25585 8.8 8:180409ea:::benchmark_data_smithi028_105698_object25584:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-02-05T06:41:19.854 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:04.948612+0000", 2024-02-05T06:41:19.854 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 74.847317918000002, 2024-02-05T06:41:19.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.102374772, 2024-02-05T06:41:19.854 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:19.854 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:19.855 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:19.855 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:41:19.855 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:41:19.855 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 25585 2024-02-05T06:41:19.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.855 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:19.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:19.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.948612+0000", 2024-02-05T06:41:19.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:19.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.948612+0000", 2024-02-05T06:41:19.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:19.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.948613+0000", 2024-02-05T06:41:19.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5999999999999991e-07 2024-02-05T06:41:19.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:19.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.948672+0000", 2024-02-05T06:41:19.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8693000000000002e-05 2024-02-05T06:41:19.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:19.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.948680+0000", 2024-02-05T06:41:19.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7399999999999993e-06 2024-02-05T06:41:19.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:19.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.948688+0000", 2024-02-05T06:41:19.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3309999999999998e-06 2024-02-05T06:41:19.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:19.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.948772+0000", 2024-02-05T06:41:19.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4673999999999996e-05 2024-02-05T06:41:19.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:19.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.948893+0000", 2024-02-05T06:41:19.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120473 2024-02-05T06:41:19.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:41:19.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.949032+0000", 2024-02-05T06:41:19.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000139092 2024-02-05T06:41:19.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:19.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.050668+0000", 2024-02-05T06:41:19.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.101636412 2024-02-05T06:41:19.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:19.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.050727+0000", 2024-02-05T06:41:19.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8193999999999999e-05 2024-02-05T06:41:19.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:19.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.050887+0000", 2024-02-05T06:41:19.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016072800000000001 2024-02-05T06:41:19.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:19.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.050944+0000", 2024-02-05T06:41:19.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6366000000000002e-05 2024-02-05T06:41:19.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:19.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.050987+0000", 2024-02-05T06:41:19.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3108999999999997e-05 2024-02-05T06:41:19.860 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.860 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:19.860 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.860 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:25586 8.8 8:18c07631:::benchmark_data_smithi028_105698_object25585:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-02-05T06:41:19.860 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:04.948704+0000", 2024-02-05T06:41:19.860 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 74.847225925999993, 2024-02-05T06:41:19.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10244455, 2024-02-05T06:41:19.861 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:19.861 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:19.861 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:19.861 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:41:19.861 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:41:19.861 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 25586 2024-02-05T06:41:19.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.861 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:19.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:19.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.948704+0000", 2024-02-05T06:41:19.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:19.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.948704+0000", 2024-02-05T06:41:19.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:19.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.948705+0000", 2024-02-05T06:41:19.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2699999999999998e-07 2024-02-05T06:41:19.862 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.862 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.862 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:19.862 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.948738+0000", 2024-02-05T06:41:19.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3423999999999997e-05 2024-02-05T06:41:19.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:19.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.948746+0000", 2024-02-05T06:41:19.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5989999999999996e-06 2024-02-05T06:41:19.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.863 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:19.863 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.948752+0000", 2024-02-05T06:41:19.863 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2539999999999998e-06 2024-02-05T06:41:19.863 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.863 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:19.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.950567+0000", 2024-02-05T06:41:19.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001815299 2024-02-05T06:41:19.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:19.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.950696+0000", 2024-02-05T06:41:19.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000128477 2024-02-05T06:41:19.864 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.864 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.864 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:41:19.864 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.950781+0000", 2024-02-05T06:41:19.864 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5462000000000003e-05 2024-02-05T06:41:19.865 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.865 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.865 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:19.865 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.050673+0000", 2024-02-05T06:41:19.865 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.099892065000000002 2024-02-05T06:41:19.866 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.866 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.866 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:19.866 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.050782+0000", 2024-02-05T06:41:19.866 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010823999999999999 2024-02-05T06:41:19.866 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:19.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.051055+0000", 2024-02-05T06:41:19.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027378999999999999 2024-02-05T06:41:19.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:19.867 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.051111+0000", 2024-02-05T06:41:19.867 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5829999999999999e-05 2024-02-05T06:41:19.867 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.867 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.867 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:19.868 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.051149+0000", 2024-02-05T06:41:19.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7283000000000002e-05 2024-02-05T06:41:19.868 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.868 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:19.868 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.868 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.868 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.868 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:25587 8.8 8:1318dd79:::benchmark_data_smithi028_105698_object25586:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-02-05T06:41:19.868 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:04.948761+0000", 2024-02-05T06:41:19.868 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 74.847168620000005, 2024-02-05T06:41:19.868 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.103847866, 2024-02-05T06:41:19.868 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:19.868 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:19.869 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:19.869 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:41:19.869 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:41:19.869 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 25587 2024-02-05T06:41:19.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.869 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:19.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:19.869 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.948761+0000", 2024-02-05T06:41:19.869 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.869 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.869 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.869 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:19.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.948761+0000", 2024-02-05T06:41:19.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:19.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.948762+0000", 2024-02-05T06:41:19.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0399999999999995e-07 2024-02-05T06:41:19.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:19.870 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.948837+0000", 2024-02-05T06:41:19.870 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4726000000000002e-05 2024-02-05T06:41:19.870 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.870 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.870 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:19.871 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.948848+0000", 2024-02-05T06:41:19.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1303e-05 2024-02-05T06:41:19.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.871 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:19.871 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.948854+0000", 2024-02-05T06:41:19.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9290000000000003e-06 2024-02-05T06:41:19.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.871 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:19.871 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.049337+0000", 2024-02-05T06:41:19.871 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.100483352 2024-02-05T06:41:19.871 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.871 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.871 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:19.872 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.049510+0000", 2024-02-05T06:41:19.872 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017253099999999999 2024-02-05T06:41:19.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:41:19.872 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.049644+0000", 2024-02-05T06:41:19.872 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013441000000000001 2024-02-05T06:41:19.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.872 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:19.872 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.052424+0000", 2024-02-05T06:41:19.872 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0027793509999999998 2024-02-05T06:41:19.872 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.872 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.873 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:19.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.052469+0000", 2024-02-05T06:41:19.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5624000000000002e-05 2024-02-05T06:41:19.873 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.873 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.873 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:19.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.052526+0000", 2024-02-05T06:41:19.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7101000000000001e-05 2024-02-05T06:41:19.873 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.873 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.873 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:19.873 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.052575+0000", 2024-02-05T06:41:19.873 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8778e-05 2024-02-05T06:41:19.874 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.874 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.874 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:19.874 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.052609+0000", 2024-02-05T06:41:19.874 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4057000000000003e-05 2024-02-05T06:41:19.874 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.874 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:19.874 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.874 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.874 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.874 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:25589 8.8 8:1723f1ba:::benchmark_data_smithi028_105698_object25588:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-02-05T06:41:19.874 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:04.950898+0000", 2024-02-05T06:41:19.875 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 74.845031970999997, 2024-02-05T06:41:19.875 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10725388600000001, 2024-02-05T06:41:19.875 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:19.875 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:19.875 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:19.875 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:41:19.875 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:41:19.876 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 25589 2024-02-05T06:41:19.876 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.876 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:19.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:19.876 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.950898+0000", 2024-02-05T06:41:19.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.876 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.876 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.876 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:19.876 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.950898+0000", 2024-02-05T06:41:19.876 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:19.877 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.950899+0000", 2024-02-05T06:41:19.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5060000000000001e-06 2024-02-05T06:41:19.877 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.877 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.877 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:19.877 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.950933+0000", 2024-02-05T06:41:19.877 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3083999999999997e-05 2024-02-05T06:41:19.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:19.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.950944+0000", 2024-02-05T06:41:19.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1027e-05 2024-02-05T06:41:19.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.878 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:19.878 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.950951+0000", 2024-02-05T06:41:19.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6249999999999998e-06 2024-02-05T06:41:19.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:19.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.051180+0000", 2024-02-05T06:41:19.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10022909300000001 2024-02-05T06:41:19.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:19.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.051295+0000", 2024-02-05T06:41:19.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000114663 2024-02-05T06:41:19.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.880 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:41:19.880 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.051430+0000", 2024-02-05T06:41:19.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013476800000000001 2024-02-05T06:41:19.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.880 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:19.880 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.054141+0000", 2024-02-05T06:41:19.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002711672 2024-02-05T06:41:19.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:19.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.057725+0000", 2024-02-05T06:41:19.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003583522 2024-02-05T06:41:19.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:19.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.058066+0000", 2024-02-05T06:41:19.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000340739 2024-02-05T06:41:19.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:19.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.058117+0000", 2024-02-05T06:41:19.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1178999999999999e-05 2024-02-05T06:41:19.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:19.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.058152+0000", 2024-02-05T06:41:19.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5008000000000001e-05 2024-02-05T06:41:19.883 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.883 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:19.883 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.883 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.15144.0:25585 8.8 e194/185 ondisk, result = 0)", 2024-02-05T06:41:19.884 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:04.951082+0000", 2024-02-05T06:41:19.884 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 74.844848145, 2024-02-05T06:41:19.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.099654320000000005, 2024-02-05T06:41:19.884 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:19.884 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-02-05T06:41:19.884 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:19.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:19.885 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.951082+0000", 2024-02-05T06:41:19.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:19.885 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.951082+0000", 2024-02-05T06:41:19.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.886 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:19.886 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.951083+0000", 2024-02-05T06:41:19.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.062e-06 2024-02-05T06:41:19.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.886 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:19.886 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.951087+0000", 2024-02-05T06:41:19.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2309999999999998e-06 2024-02-05T06:41:19.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:19.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.951095+0000", 2024-02-05T06:41:19.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1589999999999993e-06 2024-02-05T06:41:19.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:19.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.951102+0000", 2024-02-05T06:41:19.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3090000000000001e-06 2024-02-05T06:41:19.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:19.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.050714+0000", 2024-02-05T06:41:19.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.099612510000000001 2024-02-05T06:41:19.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:19.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.050721+0000", 2024-02-05T06:41:19.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9260000000000003e-06 2024-02-05T06:41:19.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:19.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.050736+0000", 2024-02-05T06:41:19.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5123000000000001e-05 2024-02-05T06:41:19.889 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.889 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:19.889 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.889 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.889 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.15144.0:25581 8.3 e194/185 8:c1ccb920:::benchmark_data_smithi028_105698_object25580:head v 194'1614, mlcod=194'1611)", 2024-02-05T06:41:19.889 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:04.951318+0000", 2024-02-05T06:41:19.889 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 74.844611545999996, 2024-02-05T06:41:19.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.097859081000000001, 2024-02-05T06:41:19.889 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:19.889 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:19.890 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:19.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.890 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:19.890 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.951318+0000", 2024-02-05T06:41:19.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.890 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:19.890 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.951318+0000", 2024-02-05T06:41:19.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.890 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:19.890 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.951319+0000", 2024-02-05T06:41:19.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4499999999999996e-07 2024-02-05T06:41:19.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.891 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:19.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.951387+0000", 2024-02-05T06:41:19.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8170000000000006e-05 2024-02-05T06:41:19.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.891 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:19.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.951395+0000", 2024-02-05T06:41:19.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3490000000000003e-06 2024-02-05T06:41:19.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.891 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:19.891 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.951400+0000", 2024-02-05T06:41:19.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6899999999999997e-06 2024-02-05T06:41:19.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:19.892 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.951439+0000", 2024-02-05T06:41:19.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.824e-05 2024-02-05T06:41:19.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:19.892 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.951451+0000", 2024-02-05T06:41:19.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2225000000000001e-05 2024-02-05T06:41:19.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:19.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.049128+0000", 2024-02-05T06:41:19.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.097677231000000003 2024-02-05T06:41:19.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:19.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.049177+0000", 2024-02-05T06:41:19.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9431000000000002e-05 2024-02-05T06:41:19.893 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.893 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:19.893 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.893 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.15144.0:25583 8.4 e194/185 ondisk, result = 0)", 2024-02-05T06:41:19.893 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:04.951575+0000", 2024-02-05T06:41:19.893 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 74.844354902000006, 2024-02-05T06:41:19.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.097728111000000006, 2024-02-05T06:41:19.894 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:19.894 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-02-05T06:41:19.894 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:19.894 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.894 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:19.894 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.951575+0000", 2024-02-05T06:41:19.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.894 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.894 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.894 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:19.894 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.951575+0000", 2024-02-05T06:41:19.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.894 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.895 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.895 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:19.895 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.951576+0000", 2024-02-05T06:41:19.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0359999999999999e-06 2024-02-05T06:41:19.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.895 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.895 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:19.895 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.951580+0000", 2024-02-05T06:41:19.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3209999999999998e-06 2024-02-05T06:41:19.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.895 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.895 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:19.895 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.049105+0000", 2024-02-05T06:41:19.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.097524949999999999 2024-02-05T06:41:19.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.896 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:19.896 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.049116+0000", 2024-02-05T06:41:19.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0416e-05 2024-02-05T06:41:19.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.896 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:19.896 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.049171+0000", 2024-02-05T06:41:19.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5219000000000001e-05 2024-02-05T06:41:19.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.896 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:19.896 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.049182+0000", 2024-02-05T06:41:19.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1306e-05 2024-02-05T06:41:19.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.897 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:19.897 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.049303+0000", 2024-02-05T06:41:19.897 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120863 2024-02-05T06:41:19.897 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.897 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:19.897 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.897 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.897 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.15144.0:25585 8.8 e194/185 ondisk, result = 0)", 2024-02-05T06:41:19.897 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:04.952003+0000", 2024-02-05T06:41:19.897 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 74.843927023000006, 2024-02-05T06:41:19.897 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098991784999999999, 2024-02-05T06:41:19.897 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:19.897 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-02-05T06:41:19.898 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:19.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:19.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.952003+0000", 2024-02-05T06:41:19.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:19.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.952003+0000", 2024-02-05T06:41:19.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:19.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.952004+0000", 2024-02-05T06:41:19.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5000000000000001e-07 2024-02-05T06:41:19.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:19.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.952007+0000", 2024-02-05T06:41:19.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2509999999999999e-06 2024-02-05T06:41:19.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:19.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.049083+0000", 2024-02-05T06:41:19.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.097075553999999994 2024-02-05T06:41:19.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:19.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.049100+0000", 2024-02-05T06:41:19.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7005000000000001e-05 2024-02-05T06:41:19.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.900 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:19.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.050818+0000", 2024-02-05T06:41:19.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0017184870000000001 2024-02-05T06:41:19.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.900 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:19.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.050879+0000", 2024-02-05T06:41:19.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1005000000000003e-05 2024-02-05T06:41:19.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.900 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:19.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.050995+0000", 2024-02-05T06:41:19.901 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011553300000000001 2024-02-05T06:41:19.901 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.901 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:19.901 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.901 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.901 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.901 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop_reply(client.15144.0:25586 8.8 e194/185 ondisk, result = 0)", 2024-02-05T06:41:19.901 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:04.954251+0000", 2024-02-05T06:41:19.901 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 74.841678876000003, 2024-02-05T06:41:19.901 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.096541701999999993, 2024-02-05T06:41:19.901 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:19.901 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-02-05T06:41:19.901 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:19.901 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.901 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:19.902 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.954251+0000", 2024-02-05T06:41:19.902 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.902 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.902 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:19.902 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.954251+0000", 2024-02-05T06:41:19.902 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.902 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.902 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:19.902 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.954252+0000", 2024-02-05T06:41:19.902 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5099999999999998e-07 2024-02-05T06:41:19.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.902 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.903 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:19.903 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:04.954255+0000", 2024-02-05T06:41:19.903 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3699999999999999e-06 2024-02-05T06:41:19.903 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.903 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:19.903 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.049086+0000", 2024-02-05T06:41:19.903 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094830708999999999 2024-02-05T06:41:19.903 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.903 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:19.903 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.049100+0000", 2024-02-05T06:41:19.903 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3634e-05 2024-02-05T06:41:19.903 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.904 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:19.904 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.050769+0000", 2024-02-05T06:41:19.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0016689980000000001 2024-02-05T06:41:19.904 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.904 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.904 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:19.904 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.050776+0000", 2024-02-05T06:41:19.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3499999999999999e-06 2024-02-05T06:41:19.904 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.904 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.904 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:19.904 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:05.050793+0000", 2024-02-05T06:41:19.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6690000000000001e-05 2024-02-05T06:41:19.904 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.904 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:19.905 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.905 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.905 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.905 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:29758 8.8 8:1b4d239d:::benchmark_data_smithi028_105698_object29757:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-02-05T06:41:19.905 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:07.329985+0000", 2024-02-05T06:41:19.905 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 72.465944653999998, 2024-02-05T06:41:19.905 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.102071096, 2024-02-05T06:41:19.905 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:19.905 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:19.905 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:19.905 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:41:19.905 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:41:19.905 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 29758 2024-02-05T06:41:19.905 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.906 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:19.906 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.906 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:19.906 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.329985+0000", 2024-02-05T06:41:19.906 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.906 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.906 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.906 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:19.906 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.329985+0000", 2024-02-05T06:41:19.906 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.906 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.906 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.906 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:19.906 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.329986+0000", 2024-02-05T06:41:19.906 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.23e-07 2024-02-05T06:41:19.907 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.907 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:19.907 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.330037+0000", 2024-02-05T06:41:19.907 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0550999999999998e-05 2024-02-05T06:41:19.907 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.907 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:19.907 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.330045+0000", 2024-02-05T06:41:19.907 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.354e-06 2024-02-05T06:41:19.907 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.907 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:19.907 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.330050+0000", 2024-02-05T06:41:19.908 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6149999999999999e-06 2024-02-05T06:41:19.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.908 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:19.908 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.330098+0000", 2024-02-05T06:41:19.908 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8004000000000003e-05 2024-02-05T06:41:19.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:19.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.330232+0000", 2024-02-05T06:41:19.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013396400000000001 2024-02-05T06:41:19.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:41:19.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.330387+0000", 2024-02-05T06:41:19.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000155675 2024-02-05T06:41:19.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:19.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.337078+0000", 2024-02-05T06:41:19.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0066908239999999997 2024-02-05T06:41:19.910 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.910 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:19.910 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.337117+0000", 2024-02-05T06:41:19.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9289000000000003e-05 2024-02-05T06:41:19.910 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.910 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:19.910 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.431967+0000", 2024-02-05T06:41:19.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094849622999999994 2024-02-05T06:41:19.910 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.910 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.910 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:19.910 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.432010+0000", 2024-02-05T06:41:19.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2644e-05 2024-02-05T06:41:19.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.911 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:19.911 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.432056+0000", 2024-02-05T06:41:19.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6730000000000002e-05 2024-02-05T06:41:19.911 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.911 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:19.911 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.911 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.911 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.911 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:29760 8.8 8:1ec74b53:::benchmark_data_smithi028_105698_object29759:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-02-05T06:41:19.911 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:07.330511+0000", 2024-02-05T06:41:19.911 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 72.465419191999999, 2024-02-05T06:41:19.911 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.101644178, 2024-02-05T06:41:19.911 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:19.912 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:19.912 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:19.912 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:41:19.912 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:41:19.912 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 29760 2024-02-05T06:41:19.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.912 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:19.912 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.912 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:19.912 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.330511+0000", 2024-02-05T06:41:19.912 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.912 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.913 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:19.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.330511+0000", 2024-02-05T06:41:19.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.913 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.913 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.913 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:19.913 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.330512+0000", 2024-02-05T06:41:19.913 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.093e-06 2024-02-05T06:41:19.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:19.914 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.330566+0000", 2024-02-05T06:41:19.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4367000000000003e-05 2024-02-05T06:41:19.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:19.914 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.330575+0000", 2024-02-05T06:41:19.914 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3209999999999994e-06 2024-02-05T06:41:19.914 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.914 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.914 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:19.915 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.330584+0000", 2024-02-05T06:41:19.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.799e-06 2024-02-05T06:41:19.915 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.915 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:19.915 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.331699+0000", 2024-02-05T06:41:19.915 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001114943 2024-02-05T06:41:19.915 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.915 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.916 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:19.916 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.331823+0000", 2024-02-05T06:41:19.916 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012341500000000001 2024-02-05T06:41:19.916 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.916 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.916 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:41:19.916 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.331967+0000", 2024-02-05T06:41:19.916 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014403300000000001 2024-02-05T06:41:19.916 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.916 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.916 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:19.917 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.337086+0000", 2024-02-05T06:41:19.917 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0051194389999999999 2024-02-05T06:41:19.917 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.917 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.917 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:19.917 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.431754+0000", 2024-02-05T06:41:19.917 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094668337000000005 2024-02-05T06:41:19.917 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.917 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.917 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:19.918 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.432096+0000", 2024-02-05T06:41:19.918 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034192100000000002 2024-02-05T06:41:19.918 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.918 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.918 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:19.918 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.432130+0000", 2024-02-05T06:41:19.918 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3738000000000001e-05 2024-02-05T06:41:19.919 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.919 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.919 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:19.919 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.432155+0000", 2024-02-05T06:41:19.919 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4771999999999999e-05 2024-02-05T06:41:19.919 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.919 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:19.920 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.920 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.920 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.920 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:29761 8.8 8:1148b940:::benchmark_data_smithi028_105698_object29760:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-02-05T06:41:19.920 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:07.330596+0000", 2024-02-05T06:41:19.920 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 72.465333520000001, 2024-02-05T06:41:19.920 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.108482866, 2024-02-05T06:41:19.920 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:19.921 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:19.921 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:19.921 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:41:19.921 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:41:19.921 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 29761 2024-02-05T06:41:19.921 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.922 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:19.922 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.922 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:19.922 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.330596+0000", 2024-02-05T06:41:19.922 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.922 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.923 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.923 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:19.923 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.330596+0000", 2024-02-05T06:41:19.923 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.923 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.923 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.923 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:19.923 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.330597+0000", 2024-02-05T06:41:19.923 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0500000000000003e-07 2024-02-05T06:41:19.923 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.923 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.923 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:19.923 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.330632+0000", 2024-02-05T06:41:19.924 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5264e-05 2024-02-05T06:41:19.924 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:19.924 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.330640+0000", 2024-02-05T06:41:19.924 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0970000000000006e-06 2024-02-05T06:41:19.924 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.924 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:19.924 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.330647+0000", 2024-02-05T06:41:19.924 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4579999999999998e-06 2024-02-05T06:41:19.924 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.924 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.925 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:19.925 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.333711+0000", 2024-02-05T06:41:19.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030638869999999999 2024-02-05T06:41:19.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.925 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.925 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:19.925 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.333853+0000", 2024-02-05T06:41:19.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014242899999999999 2024-02-05T06:41:19.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.925 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.925 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:41:19.925 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.333953+0000", 2024-02-05T06:41:19.925 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9648999999999994e-05 2024-02-05T06:41:19.925 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.925 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.925 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:19.926 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.337133+0000", 2024-02-05T06:41:19.926 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0031805689999999998 2024-02-05T06:41:19.926 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.926 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.926 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:19.926 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.431786+0000", 2024-02-05T06:41:19.926 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.094652871 2024-02-05T06:41:19.926 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.926 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.926 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:19.926 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.439025+0000", 2024-02-05T06:41:19.926 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0072390079999999999 2024-02-05T06:41:19.926 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.926 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.927 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:19.927 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.439058+0000", 2024-02-05T06:41:19.927 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3151000000000002e-05 2024-02-05T06:41:19.927 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.927 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.927 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:19.927 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.439079+0000", 2024-02-05T06:41:19.927 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0778e-05 2024-02-05T06:41:19.927 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.927 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:19.927 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.927 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.927 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.927 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:29767 8.8 8:1391243f:::benchmark_data_smithi028_105698_object29766:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-02-05T06:41:19.928 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:07.333631+0000", 2024-02-05T06:41:19.928 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 72.462299200999993, 2024-02-05T06:41:19.928 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10557051200000001, 2024-02-05T06:41:19.928 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:19.928 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:19.928 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:19.928 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:41:19.928 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:41:19.928 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 29767 2024-02-05T06:41:19.928 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.928 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:19.928 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.928 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:19.928 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.333631+0000", 2024-02-05T06:41:19.928 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.929 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.929 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.929 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:19.929 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.333631+0000", 2024-02-05T06:41:19.929 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:19.929 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.929 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.929 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:19.929 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.333633+0000", 2024-02-05T06:41:19.929 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0949999999999998e-06 2024-02-05T06:41:19.929 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.929 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.929 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:19.929 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.333733+0000", 2024-02-05T06:41:19.930 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000100237 2024-02-05T06:41:19.930 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.930 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.930 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:19.930 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.333744+0000", 2024-02-05T06:41:19.930 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1205e-05 2024-02-05T06:41:19.930 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.930 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.930 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:19.930 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.333752+0000", 2024-02-05T06:41:19.930 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.96e-06 2024-02-05T06:41:19.930 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.930 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.930 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:19.931 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.335142+0000", 2024-02-05T06:41:19.931 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001389698 2024-02-05T06:41:19.931 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.931 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.931 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:19.931 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.335440+0000", 2024-02-05T06:41:19.931 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029819500000000001 2024-02-05T06:41:19.931 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.931 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.931 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:41:19.931 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.335585+0000", 2024-02-05T06:41:19.931 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014485100000000001 2024-02-05T06:41:19.931 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.931 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.932 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:19.932 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.337204+0000", 2024-02-05T06:41:19.932 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0016193080000000001 2024-02-05T06:41:19.932 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.932 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.932 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:19.932 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.435246+0000", 2024-02-05T06:41:19.932 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098042156000000005 2024-02-05T06:41:19.932 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.932 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.932 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:19.932 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.439118+0000", 2024-02-05T06:41:19.932 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038717579999999999 2024-02-05T06:41:19.932 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.933 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.933 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:19.933 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.439168+0000", 2024-02-05T06:41:19.933 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9561999999999997e-05 2024-02-05T06:41:19.933 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:19.933 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:19.933 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:19.933 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.439201+0000", 2024-02-05T06:41:19.933 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3487e-05 2024-02-05T06:41:19.933 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.933 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:19.933 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.933 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:19.933 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:19.933 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:41:19.934 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:19.965 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:19.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:19.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:20.067 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:20.091 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:20.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:20.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:20.193 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:20.231 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:20.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:20.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:20.238 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 85 16 85080 85064 62.5347 57.1875 0.0142907 0.0159767 2024-02-05T06:41:20.333 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:20.365 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:20.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:20.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:20.467 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:20.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:20.497 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:20.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:20.598 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:20.622 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:20 smithi028 ceph-mon[29442]: from='client.15207 v1:172.21.15.28:0/744603915' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:41:20.622 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:20 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:41:20.622 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:20 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:41:20.622 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:20 smithi028 ceph-mon[29442]: osdmap e235: 8 total, 8 up, 7 in 2024-02-05T06:41:20.622 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:20 smithi028 ceph-mon[37984]: from='client.15207 v1:172.21.15.28:0/744603915' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:41:20.622 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:20 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:41:20.622 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:20 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:41:20.622 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:20 smithi028 ceph-mon[37984]: osdmap e235: 8 total, 8 up, 7 in 2024-02-05T06:41:20.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:20.623 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:20.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:20.725 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:20.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:20 smithi167 ceph-mon[32774]: from='client.15207 v1:172.21.15.28:0/744603915' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:41:20.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:20 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:41:20.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:20 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:41:20.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:20 smithi167 ceph-mon[32774]: osdmap e235: 8 total, 8 up, 7 in 2024-02-05T06:41:20.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:20.757 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:20.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:20.858 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:20.882 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:20.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:20.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:20.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:21.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:21.009 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:21.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:21.110 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:21.150 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:21.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:21.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:21.239 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 86 16 85331 85315 61.99 15.6875 0.00746711 0.0161251 2024-02-05T06:41:21.252 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:21.297 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-05T06:41:21.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:21.299 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:21.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:21.399 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:21.430 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:21.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:21.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:21.532 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:21.565 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:21.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:21.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:21.625 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:41:21.667 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:21.695 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:21.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:21.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:21.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:21 smithi028 ceph-mon[29442]: pgmap v371: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 55 MiB/s wr, 879 op/s; 3771/246243 objects misplaced (1.531%); 9.1 MiB/s, 145 objects/s recovering 2024-02-05T06:41:21.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:21 smithi028 ceph-mon[29442]: osdmap e236: 8 total, 8 up, 7 in 2024-02-05T06:41:21.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:21 smithi028 ceph-mon[37984]: pgmap v371: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 55 MiB/s wr, 879 op/s; 3771/246243 objects misplaced (1.531%); 9.1 MiB/s, 145 objects/s recovering 2024-02-05T06:41:21.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:21 smithi028 ceph-mon[37984]: osdmap e236: 8 total, 8 up, 7 in 2024-02-05T06:41:21.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:21 smithi167 ceph-mon[32774]: pgmap v371: 158 pgs: 3 active+recovering+undersized+remapped, 155 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 55 MiB/s wr, 879 op/s; 3771/246243 objects misplaced (1.531%); 9.1 MiB/s, 145 objects/s recovering 2024-02-05T06:41:21.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:21 smithi167 ceph-mon[32774]: osdmap e236: 8 total, 8 up, 7 in 2024-02-05T06:41:21.797 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:21.830 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:21.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:21.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:21.832 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2024-02-05T06:41:18.297187+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5366832595,"num_objects":82081,"num_object_clones":0,"num_object_copies":246243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3771,"num_objects_unfound":0,"num_objects_dirty":82081,"num_whiteouts":0,"num_read":827,"num_read_kb":614,"num_write":82384,"num_write_kb":5241434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24192,"num_bytes_recovered":1562706396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71648,"ondisk_log_size":71648,"up":474,"acting":471,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":478,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":17746560,"kb_used_data":15844360,"kb_used_omap":10,"kb_used_meta":1902133,"kb_avail":638469504,"statfs":{"total":671965249536,"available":653792772096,"internally_reserved":0,"allocated":16224624640,"data_stored":16218299020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":1947784299},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,5,8,6,6,14],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":575930368,"num_objects":8788,"num_object_clones":0,"num_object_copies":26364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1452,"num_objects_unfound":0,"num_objects_dirty":8788,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8788,"num_write_kb":562432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1452,"num_bytes_recovered":95158272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.932745"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":254,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.089284+0000","last_change":"2024-02-05T06:40:27.072854+0000","last_active":"2024-02-05T06:41:10.089284+0000","last_peered":"2024-02-05T06:41:10.089284+0000","last_clean":"2024-02-05T06:41:10.089284+0000","last_became_active":"2024-02-05T06:40:27.072478+0000","last_became_peered":"2024-02-05T06:40:27.072478+0000","last_unstale":"2024-02-05T06:41:10.089284+0000","last_undegraded":"2024-02-05T06:41:10.089284+0000","last_fullsized":"2024-02-05T06:41:10.089284+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:32.017855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:32.017855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":383,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.087128+0000","last_change":"2024-02-05T06:40:27.087768+0000","last_active":"2024-02-05T06:41:10.087128+0000","last_peered":"2024-02-05T06:41:10.087128+0000","last_clean":"2024-02-05T06:41:10.087128+0000","last_became_active":"2024-02-05T06:40:27.087211+0000","last_became_peered":"2024-02-05T06:40:27.087211+0000","last_unstale":"2024-02-05T06:41:10.087128+0000","last_undegraded":"2024-02-05T06:41:10.087128+0000","last_fullsized":"2024-02-05T06:41:10.087128+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:15.357465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:06.525012+0000","last_clean_scrub_stamp":"2024-02-05T06:39:15.357465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:10.206247+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":372,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.090657+0000","last_change":"2024-02-05T06:40:08.601821+0000","last_active":"2024-02-05T06:41:10.090657+0000","last_peered":"2024-02-05T06:41:10.090657+0000","last_clean":"2024-02-05T06:41:10.090657+0000","last_became_active":"2024-02-05T06:40:08.598443+0000","last_became_peered":"2024-02-05T06:40:08.598443+0000","last_unstale":"2024-02-05T06:41:10.090657+0000","last_undegraded":"2024-02-05T06:41:10.090657+0000","last_fullsized":"2024-02-05T06:41:10.090657+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:28.841777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:39:28.841777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":260,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.088634+0000","last_change":"2024-02-05T06:39:15.027055+0000","last_active":"2024-02-05T06:41:10.088634+0000","last_peered":"2024-02-05T06:41:10.088634+0000","last_clean":"2024-02-05T06:41:10.088634+0000","last_became_active":"2024-02-05T06:36:54.959330+0000","last_became_peered":"2024-02-05T06:36:54.959330+0000","last_unstale":"2024-02-05T06:41:10.088634+0000","last_undegraded":"2024-02-05T06:41:10.088634+0000","last_fullsized":"2024-02-05T06:41:10.088634+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:39:15.026945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:15.026945+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":7,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":365,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.090620+0000","last_change":"2024-02-05T06:40:27.077900+0000","last_active":"2024-02-05T06:41:10.090620+0000","last_peered":"2024-02-05T06:41:10.090620+0000","last_clean":"2024-02-05T06:41:10.090620+0000","last_became_active":"2024-02-05T06:40:27.077485+0000","last_became_peered":"2024-02-05T06:40:27.077485+0000","last_unstale":"2024-02-05T06:41:10.090620+0000","last_undegraded":"2024-02-05T06:41:10.090620+0000","last_fullsized":"2024-02-05T06:41:10.090620+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:21.419504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:37:21.419504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":250,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.088745+0000","last_change":"2024-02-05T06:39:11.961722+0000","last_active":"2024-02-05T06:41:10.088745+0000","last_peered":"2024-02-05T06:41:10.088745+0000","last_clean":"2024-02-05T06:41:10.088745+0000","last_became_active":"2024-02-05T06:36:54.959640+0000","last_became_peered":"2024-02-05T06:36:54.959640+0000","last_unstale":"2024-02-05T06:41:10.088745+0000","last_undegraded":"2024-02-05T06:41:10.088745+0000","last_fullsized":"2024-02-05T06:41:10.088745+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:11.961635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:11.961635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":373,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.093970+0000","last_change":"2024-02-05T06:39:34.395050+0000","last_active":"2024-02-05T06:41:10.093970+0000","last_peered":"2024-02-05T06:41:10.093970+0000","last_clean":"2024-02-05T06:41:10.093970+0000","last_became_active":"2024-02-05T06:33:29.553864+0000","last_became_peered":"2024-02-05T06:33:29.553864+0000","last_unstale":"2024-02-05T06:41:10.093970+0000","last_undegraded":"2024-02-05T06:41:10.093970+0000","last_fullsized":"2024-02-05T06:41:10.093970+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:34.394974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:39:34.394974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"88'22","reported_seq":401,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.089971+0000","last_change":"2024-02-05T06:39:35.433638+0000","last_active":"2024-02-05T06:41:10.089971+0000","last_peered":"2024-02-05T06:41:10.089971+0000","last_clean":"2024-02-05T06:41:10.089971+0000","last_became_active":"2024-02-05T06:33:31.572544+0000","last_became_peered":"2024-02-05T06:33:31.572544+0000","last_unstale":"2024-02-05T06:41:10.089971+0000","last_undegraded":"2024-02-05T06:41:10.089971+0000","last_fullsized":"2024-02-05T06:41:10.089971+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'22","last_scrub_stamp":"2024-02-05T06:39:35.433539+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:39:35.433539+0000","last_clean_scrub_stamp":"2024-02-05T06:39:35.433539+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":366,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.087132+0000","last_change":"2024-02-05T06:40:27.069177+0000","last_active":"2024-02-05T06:41:10.087132+0000","last_peered":"2024-02-05T06:41:10.087132+0000","last_clean":"2024-02-05T06:41:10.087132+0000","last_became_active":"2024-02-05T06:40:27.068888+0000","last_became_peered":"2024-02-05T06:40:27.068888+0000","last_unstale":"2024-02-05T06:41:10.087132+0000","last_undegraded":"2024-02-05T06:41:10.087132+0000","last_fullsized":"2024-02-05T06:41:10.087132+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:49.406914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:37:49.406914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":289,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.087018+0000","last_change":"2024-02-05T06:40:27.066310+0000","last_active":"2024-02-05T06:41:10.087018+0000","last_peered":"2024-02-05T06:41:10.087018+0000","last_clean":"2024-02-05T06:41:10.087018+0000","last_became_active":"2024-02-05T06:40:27.065918+0000","last_became_peered":"2024-02-05T06:40:27.065918+0000","last_unstale":"2024-02-05T06:41:10.087018+0000","last_undegraded":"2024-02-05T06:41:10.087018+0000","last_fullsized":"2024-02-05T06:41:10.087018+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:38:15.240524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:38:15.240524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":223,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.089752+0000","last_change":"2024-02-05T06:40:08.579505+0000","last_active":"2024-02-05T06:41:10.089752+0000","last_peered":"2024-02-05T06:41:10.089752+0000","last_clean":"2024-02-05T06:41:10.089752+0000","last_became_active":"2024-02-05T06:40:08.579165+0000","last_became_peered":"2024-02-05T06:40:08.579165+0000","last_unstale":"2024-02-05T06:41:10.089752+0000","last_undegraded":"2024-02-05T06:41:10.089752+0000","last_fullsized":"2024-02-05T06:41:10.089752+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:08.476236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:44.081318+0000","last_clean_scrub_stamp":"2024-02-05T06:39:08.476236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"92'13","reported_seq":260,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.089149+0000","last_change":"2024-02-05T06:38:14.064011+0000","last_active":"2024-02-05T06:41:10.089149+0000","last_peered":"2024-02-05T06:41:10.089149+0000","last_clean":"2024-02-05T06:41:10.089149+0000","last_became_active":"2024-02-05T06:36:54.969454+0000","last_became_peered":"2024-02-05T06:36:54.969454+0000","last_unstale":"2024-02-05T06:41:10.089149+0000","last_undegraded":"2024-02-05T06:41:10.089149+0000","last_fullsized":"2024-02-05T06:41:10.089149+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'13","last_scrub_stamp":"2024-02-05T06:38:14.063877+0000","last_deep_scrub":"92'13","last_deep_scrub_stamp":"2024-02-05T06:34:46.542893+0000","last_clean_scrub_stamp":"2024-02-05T06:38:14.063877+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":88,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.087361+0000","last_change":"2024-02-05T06:40:08.602038+0000","last_active":"2024-02-05T06:41:10.087361+0000","last_peered":"2024-02-05T06:41:10.087361+0000","last_clean":"2024-02-05T06:41:10.087361+0000","last_became_active":"2024-02-05T06:40:08.601685+0000","last_became_peered":"2024-02-05T06:40:08.601685+0000","last_unstale":"2024-02-05T06:41:10.087361+0000","last_undegraded":"2024-02-05T06:41:10.087361+0000","last_fullsized":"2024-02-05T06:41:10.087361+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:48.266135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:21.998850+0000","last_clean_scrub_stamp":"2024-02-05T06:37:48.266135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:10.205873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":67,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.094175+0000","last_change":"2024-02-05T06:40:27.072096+0000","last_active":"2024-02-05T06:41:10.094175+0000","last_peered":"2024-02-05T06:41:10.094175+0000","last_clean":"2024-02-05T06:41:10.094175+0000","last_became_active":"2024-02-05T06:40:27.071758+0000","last_became_peered":"2024-02-05T06:40:27.071758+0000","last_unstale":"2024-02-05T06:41:10.094175+0000","last_undegraded":"2024-02-05T06:41:10.094175+0000","last_fullsized":"2024-02-05T06:41:10.094175+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:26.781111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:37:26.781111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":223,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.089390+0000","last_change":"2024-02-05T06:40:08.579914+0000","last_active":"2024-02-05T06:41:10.089390+0000","last_peered":"2024-02-05T06:41:10.089390+0000","last_clean":"2024-02-05T06:41:10.089390+0000","last_became_active":"2024-02-05T06:40:08.578882+0000","last_became_peered":"2024-02-05T06:40:08.578882+0000","last_unstale":"2024-02-05T06:41:10.089390+0000","last_undegraded":"2024-02-05T06:41:10.089390+0000","last_fullsized":"2024-02-05T06:41:10.089390+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:34.388028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:50.055245+0000","last_clean_scrub_stamp":"2024-02-05T06:39:34.388028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":256,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.089230+0000","last_change":"2024-02-05T06:39:08.998613+0000","last_active":"2024-02-05T06:41:10.089230+0000","last_peered":"2024-02-05T06:41:10.089230+0000","last_clean":"2024-02-05T06:41:10.089230+0000","last_became_active":"2024-02-05T06:36:54.954385+0000","last_became_peered":"2024-02-05T06:36:54.954385+0000","last_unstale":"2024-02-05T06:41:10.089230+0000","last_undegraded":"2024-02-05T06:41:10.089230+0000","last_fullsized":"2024-02-05T06:41:10.089230+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-02-05T06:39:08.998484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:08.998484+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":257,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.089826+0000","last_change":"2024-02-05T06:40:08.582243+0000","last_active":"2024-02-05T06:41:10.089826+0000","last_peered":"2024-02-05T06:41:10.089826+0000","last_clean":"2024-02-05T06:41:10.089826+0000","last_became_active":"2024-02-05T06:40:08.581272+0000","last_became_peered":"2024-02-05T06:40:08.581272+0000","last_unstale":"2024-02-05T06:41:10.089826+0000","last_undegraded":"2024-02-05T06:41:10.089826+0000","last_fullsized":"2024-02-05T06:41:10.089826+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:10.958212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:10.958212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":377,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.089036+0000","last_change":"2024-02-05T06:39:32.263466+0000","last_active":"2024-02-05T06:41:10.089036+0000","last_peered":"2024-02-05T06:41:10.089036+0000","last_clean":"2024-02-05T06:41:10.089036+0000","last_became_active":"2024-02-05T06:36:54.958069+0000","last_became_peered":"2024-02-05T06:36:54.958069+0000","last_unstale":"2024-02-05T06:41:10.089036+0000","last_undegraded":"2024-02-05T06:41:10.089036+0000","last_fullsized":"2024-02-05T06:41:10.089036+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:32.263391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:32.263391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:10.206220+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"66'1","reported_seq":412,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.087626+0000","last_change":"2024-02-05T06:40:27.118661+0000","last_active":"2024-02-05T06:41:10.087626+0000","last_peered":"2024-02-05T06:41:10.087626+0000","last_clean":"2024-02-05T06:41:10.087626+0000","last_became_active":"2024-02-05T06:40:27.073360+0000","last_became_peered":"2024-02-05T06:40:27.073360+0000","last_unstale":"2024-02-05T06:41:10.087626+0000","last_undegraded":"2024-02-05T06:41:10.087626+0000","last_fullsized":"2024-02-05T06:41:10.087626+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:37:47.363630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:37:47.363630+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"88'22","reported_seq":407,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.094245+0000","last_change":"2024-02-05T06:39:11.993130+0000","last_active":"2024-02-05T06:41:10.094245+0000","last_peered":"2024-02-05T06:41:10.094245+0000","last_clean":"2024-02-05T06:41:10.094245+0000","last_became_active":"2024-02-05T06:33:31.569763+0000","last_became_peered":"2024-02-05T06:33:31.569763+0000","last_unstale":"2024-02-05T06:41:10.094245+0000","last_undegraded":"2024-02-05T06:41:10.094245+0000","last_fullsized":"2024-02-05T06:41:10.094245+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'22","last_scrub_stamp":"2024-02-05T06:39:11.992880+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:34:35.234636+0000","last_clean_scrub_stamp":"2024-02-05T06:39:11.992880+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:12.649595+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":359,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.096250+0000","last_change":"2024-02-05T06:40:08.606128+0000","last_active":"2024-02-05T06:41:10.096250+0000","last_peered":"2024-02-05T06:41:10.096250+0000","last_clean":"2024-02-05T06:41:10.096250+0000","last_became_active":"2024-02-05T06:40:08.605815+0000","last_became_peered":"2024-02-05T06:40:08.605815+0000","last_unstale":"2024-02-05T06:41:10.096250+0000","last_undegraded":"2024-02-05T06:41:10.096250+0000","last_fullsized":"2024-02-05T06:41:10.096250+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:06.387381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:06.387381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":138,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.028524+0000","last_change":"2024-02-05T06:39:14.482807+0000","last_active":"2024-02-05T06:41:10.028524+0000","last_peered":"2024-02-05T06:41:10.028524+0000","last_clean":"2024-02-05T06:41:10.028524+0000","last_became_active":"2024-02-05T06:39:13.248618+0000","last_became_peered":"2024-02-05T06:39:13.248618+0000","last_unstale":"2024-02-05T06:41:10.028524+0000","last_undegraded":"2024-02-05T06:41:10.028524+0000","last_fullsized":"2024-02-05T06:41:10.028524+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:14.482739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:14.482739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":250,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.089078+0000","last_change":"2024-02-05T06:40:07.898901+0000","last_active":"2024-02-05T06:41:10.089078+0000","last_peered":"2024-02-05T06:41:10.089078+0000","last_clean":"2024-02-05T06:41:10.089078+0000","last_became_active":"2024-02-05T06:36:54.959070+0000","last_became_peered":"2024-02-05T06:36:54.959070+0000","last_unstale":"2024-02-05T06:41:10.089078+0000","last_undegraded":"2024-02-05T06:41:10.089078+0000","last_fullsized":"2024-02-05T06:41:10.089078+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:40:07.898795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:40:07.898795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:18.962211+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":438,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.094863+0000","last_change":"2024-02-05T06:38:16.181378+0000","last_active":"2024-02-05T06:41:10.094863+0000","last_peered":"2024-02-05T06:41:10.094863+0000","last_clean":"2024-02-05T06:41:10.094863+0000","last_became_active":"2024-02-05T06:36:54.954880+0000","last_became_peered":"2024-02-05T06:36:54.954880+0000","last_unstale":"2024-02-05T06:41:10.094863+0000","last_undegraded":"2024-02-05T06:41:10.094863+0000","last_fullsized":"2024-02-05T06:41:10.094863+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:38:16.181294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:38:16.181294+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"73'1","reported_seq":71,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.095686+0000","last_change":"2024-02-05T06:40:46.449931+0000","last_active":"2024-02-05T06:41:10.095686+0000","last_peered":"2024-02-05T06:41:10.095686+0000","last_clean":"2024-02-05T06:41:10.095686+0000","last_became_active":"2024-02-05T06:40:27.185817+0000","last_became_peered":"2024-02-05T06:40:27.185817+0000","last_unstale":"2024-02-05T06:41:10.095686+0000","last_undegraded":"2024-02-05T06:41:10.095686+0000","last_fullsized":"2024-02-05T06:41:10.095686+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-02-05T06:40:09.298234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:40:09.298234+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"208'12","reported_seq":514,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.028905+0000","last_change":"2024-02-05T06:40:27.068738+0000","last_active":"2024-02-05T06:41:10.028905+0000","last_peered":"2024-02-05T06:41:10.028905+0000","last_clean":"2024-02-05T06:41:10.028905+0000","last_became_active":"2024-02-05T06:40:27.063399+0000","last_became_peered":"2024-02-05T06:40:27.063399+0000","last_unstale":"2024-02-05T06:41:10.028905+0000","last_undegraded":"2024-02-05T06:41:10.028905+0000","last_fullsized":"2024-02-05T06:41:10.028905+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'11","last_scrub_stamp":"2024-02-05T06:39:10.519706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:10.519706+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":394,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.087798+0000","last_change":"2024-02-05T06:39:07.546967+0000","last_active":"2024-02-05T06:41:10.087798+0000","last_peered":"2024-02-05T06:41:10.087798+0000","last_clean":"2024-02-05T06:41:10.087798+0000","last_became_active":"2024-02-05T06:36:54.958414+0000","last_became_peered":"2024-02-05T06:36:54.958414+0000","last_unstale":"2024-02-05T06:41:10.087798+0000","last_undegraded":"2024-02-05T06:41:10.087798+0000","last_fullsized":"2024-02-05T06:41:10.087798+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:39:07.546646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:39:07.546646+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:10.205959+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"92'15","reported_seq":426,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.099383+0000","last_change":"2024-02-05T06:40:46.436058+0000","last_active":"2024-02-05T06:41:10.099383+0000","last_peered":"2024-02-05T06:41:10.099383+0000","last_clean":"2024-02-05T06:41:10.099383+0000","last_became_active":"2024-02-05T06:40:27.100610+0000","last_became_peered":"2024-02-05T06:40:27.100610+0000","last_unstale":"2024-02-05T06:41:10.099383+0000","last_undegraded":"2024-02-05T06:41:10.099383+0000","last_fullsized":"2024-02-05T06:41:10.099383+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'15","last_scrub_stamp":"2024-02-05T06:40:07.595540+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-02-05T06:34:50.718768+0000","last_clean_scrub_stamp":"2024-02-05T06:40:07.595540+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:33.345169+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":379,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.089159+0000","last_change":"2024-02-05T06:40:08.593331+0000","last_active":"2024-02-05T06:41:10.089159+0000","last_peered":"2024-02-05T06:41:10.089159+0000","last_clean":"2024-02-05T06:41:10.089159+0000","last_became_active":"2024-02-05T06:40:08.593007+0000","last_became_peered":"2024-02-05T06:40:08.593007+0000","last_unstale":"2024-02-05T06:41:10.089159+0000","last_undegraded":"2024-02-05T06:41:10.089159+0000","last_fullsized":"2024-02-05T06:41:10.089159+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:06.475715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:06.475715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:10.205927+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":280,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.093832+0000","last_change":"2024-02-05T06:40:27.097489+0000","last_active":"2024-02-05T06:41:10.093832+0000","last_peered":"2024-02-05T06:41:10.093832+0000","last_clean":"2024-02-05T06:41:10.093832+0000","last_became_active":"2024-02-05T06:40:27.097169+0000","last_became_peered":"2024-02-05T06:40:27.097169+0000","last_unstale":"2024-02-05T06:41:10.093832+0000","last_undegraded":"2024-02-05T06:41:10.093832+0000","last_fullsized":"2024-02-05T06:41:10.093832+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:38:26.171039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:08.713894+0000","last_clean_scrub_stamp":"2024-02-05T06:38:26.171039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":87,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.091568+0000","last_change":"2024-02-05T06:40:08.607982+0000","last_active":"2024-02-05T06:41:10.091568+0000","last_peered":"2024-02-05T06:41:10.091568+0000","last_clean":"2024-02-05T06:41:10.091568+0000","last_became_active":"2024-02-05T06:40:08.606962+0000","last_became_peered":"2024-02-05T06:40:08.606962+0000","last_unstale":"2024-02-05T06:41:10.091568+0000","last_undegraded":"2024-02-05T06:41:10.091568+0000","last_fullsized":"2024-02-05T06:41:10.091568+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:40:05.263073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:40:05.263073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:10.913645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"92'16","reported_seq":421,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.028938+0000","last_change":"2024-02-05T06:40:08.667187+0000","last_active":"2024-02-05T06:41:10.028938+0000","last_peered":"2024-02-05T06:41:10.028938+0000","last_clean":"2024-02-05T06:41:10.028938+0000","last_became_active":"2024-02-05T06:40:08.603702+0000","last_became_peered":"2024-02-05T06:40:08.603702+0000","last_unstale":"2024-02-05T06:41:10.028938+0000","last_undegraded":"2024-02-05T06:41:10.028938+0000","last_fullsized":"2024-02-05T06:41:10.028938+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'16","last_scrub_stamp":"2024-02-05T06:40:01.612165+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:34:45.270135+0000","last_clean_scrub_stamp":"2024-02-05T06:40:01.612165+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:11.450516+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":67,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.094265+0000","last_change":"2024-02-05T06:40:27.099270+0000","last_active":"2024-02-05T06:41:10.094265+0000","last_peered":"2024-02-05T06:41:10.094265+0000","last_clean":"2024-02-05T06:41:10.094265+0000","last_became_active":"2024-02-05T06:40:27.098348+0000","last_became_peered":"2024-02-05T06:40:27.098348+0000","last_unstale":"2024-02-05T06:41:10.094265+0000","last_undegraded":"2024-02-05T06:41:10.094265+0000","last_fullsized":"2024-02-05T06:41:10.094265+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:29.705961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:37:29.705961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":396,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.028254+0000","last_change":"2024-02-05T06:40:27.060697+0000","last_active":"2024-02-05T06:41:10.028254+0000","last_peered":"2024-02-05T06:41:10.028254+0000","last_clean":"2024-02-05T06:41:10.028254+0000","last_became_active":"2024-02-05T06:40:27.060112+0000","last_became_peered":"2024-02-05T06:40:27.060112+0000","last_unstale":"2024-02-05T06:41:10.028254+0000","last_undegraded":"2024-02-05T06:41:10.028254+0000","last_fullsized":"2024-02-05T06:41:10.028254+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:15.439354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:15.439354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":372,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.090856+0000","last_change":"2024-02-05T06:40:08.600081+0000","last_active":"2024-02-05T06:41:10.090856+0000","last_peered":"2024-02-05T06:41:10.090856+0000","last_clean":"2024-02-05T06:41:10.090856+0000","last_became_active":"2024-02-05T06:40:08.595986+0000","last_became_peered":"2024-02-05T06:40:08.595986+0000","last_unstale":"2024-02-05T06:41:10.090856+0000","last_undegraded":"2024-02-05T06:41:10.090856+0000","last_fullsized":"2024-02-05T06:41:10.090856+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:40:05.586211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:40:05.586211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:20.946734+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"73'4","reported_seq":367,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.095671+0000","last_change":"2024-02-05T06:39:32.415715+0000","last_active":"2024-02-05T06:41:10.095671+0000","last_peered":"2024-02-05T06:41:10.095671+0000","last_clean":"2024-02-05T06:41:10.095671+0000","last_became_active":"2024-02-05T06:33:31.573410+0000","last_became_peered":"2024-02-05T06:33:31.573410+0000","last_unstale":"2024-02-05T06:41:10.095671+0000","last_undegraded":"2024-02-05T06:41:10.095671+0000","last_fullsized":"2024-02-05T06:41:10.095671+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'4","last_scrub_stamp":"2024-02-05T06:39:32.415626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:32.415626+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":139,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.089389+0000","last_change":"2024-02-05T06:40:02.234701+0000","last_active":"2024-02-05T06:41:10.089389+0000","last_peered":"2024-02-05T06:41:10.089389+0000","last_clean":"2024-02-05T06:41:10.089389+0000","last_became_active":"2024-02-05T06:39:13.245684+0000","last_became_peered":"2024-02-05T06:39:13.245684+0000","last_unstale":"2024-02-05T06:41:10.089389+0000","last_undegraded":"2024-02-05T06:41:10.089389+0000","last_fullsized":"2024-02-05T06:41:10.089389+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:40:02.234576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:40:02.234576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:22.594196+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":67,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.093390+0000","last_change":"2024-02-05T06:40:27.063394+0000","last_active":"2024-02-05T06:41:10.093390+0000","last_peered":"2024-02-05T06:41:10.093390+0000","last_clean":"2024-02-05T06:41:10.093390+0000","last_became_active":"2024-02-05T06:40:27.062975+0000","last_became_peered":"2024-02-05T06:40:27.062975+0000","last_unstale":"2024-02-05T06:41:10.093390+0000","last_undegraded":"2024-02-05T06:41:10.093390+0000","last_fullsized":"2024-02-05T06:41:10.093390+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:43.604994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:37:43.604994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":67,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.094932+0000","last_change":"2024-02-05T06:40:27.094305+0000","last_active":"2024-02-05T06:41:10.094932+0000","last_peered":"2024-02-05T06:41:10.094932+0000","last_clean":"2024-02-05T06:41:10.094932+0000","last_became_active":"2024-02-05T06:40:27.093971+0000","last_became_peered":"2024-02-05T06:40:27.093971+0000","last_unstale":"2024-02-05T06:41:10.094932+0000","last_undegraded":"2024-02-05T06:41:10.094932+0000","last_fullsized":"2024-02-05T06:41:10.094932+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:43.243812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:37:43.243812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"93'16","reported_seq":439,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.089886+0000","last_change":"2024-02-05T06:40:27.124811+0000","last_active":"2024-02-05T06:41:10.089886+0000","last_peered":"2024-02-05T06:41:10.089886+0000","last_clean":"2024-02-05T06:41:10.089886+0000","last_became_active":"2024-02-05T06:40:27.080906+0000","last_became_peered":"2024-02-05T06:40:27.080906+0000","last_unstale":"2024-02-05T06:41:10.089886+0000","last_undegraded":"2024-02-05T06:41:10.089886+0000","last_fullsized":"2024-02-05T06:41:10.089886+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'16","last_scrub_stamp":"2024-02-05T06:39:11.291756+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-02-05T06:35:22.938675+0000","last_clean_scrub_stamp":"2024-02-05T06:39:11.291756+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"73'1","reported_seq":365,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.086793+0000","last_change":"2024-02-05T06:40:08.649622+0000","last_active":"2024-02-05T06:41:10.086793+0000","last_peered":"2024-02-05T06:41:10.086793+0000","last_clean":"2024-02-05T06:41:10.086793+0000","last_became_active":"2024-02-05T06:40:08.584043+0000","last_became_peered":"2024-02-05T06:40:08.584043+0000","last_unstale":"2024-02-05T06:41:10.086793+0000","last_undegraded":"2024-02-05T06:41:10.086793+0000","last_fullsized":"2024-02-05T06:41:10.086793+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-02-05T06:37:41.335213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:37:41.335213+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":357,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.090542+0000","last_change":"2024-02-05T06:39:25.369971+0000","last_active":"2024-02-05T06:41:10.090542+0000","last_peered":"2024-02-05T06:41:10.090542+0000","last_clean":"2024-02-05T06:41:10.090542+0000","last_became_active":"2024-02-05T06:33:33.593901+0000","last_became_peered":"2024-02-05T06:33:33.593901+0000","last_unstale":"2024-02-05T06:41:10.090542+0000","last_undegraded":"2024-02-05T06:41:10.090542+0000","last_fullsized":"2024-02-05T06:41:10.090542+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:25.369865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:25.369865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":67,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.094985+0000","last_change":"2024-02-05T06:40:27.078243+0000","last_active":"2024-02-05T06:41:10.094985+0000","last_peered":"2024-02-05T06:41:10.094985+0000","last_clean":"2024-02-05T06:41:10.094985+0000","last_became_active":"2024-02-05T06:40:27.077701+0000","last_became_peered":"2024-02-05T06:40:27.077701+0000","last_unstale":"2024-02-05T06:41:10.094985+0000","last_undegraded":"2024-02-05T06:41:10.094985+0000","last_fullsized":"2024-02-05T06:41:10.094985+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:27.744143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:37:27.744143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":141,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.086756+0000","last_change":"2024-02-05T06:39:13.292585+0000","last_active":"2024-02-05T06:41:10.086756+0000","last_peered":"2024-02-05T06:41:10.086756+0000","last_clean":"2024-02-05T06:41:10.086756+0000","last_became_active":"2024-02-05T06:39:13.244124+0000","last_became_peered":"2024-02-05T06:39:13.244124+0000","last_unstale":"2024-02-05T06:41:10.086756+0000","last_undegraded":"2024-02-05T06:41:10.086756+0000","last_fullsized":"2024-02-05T06:41:10.086756+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:36:30.047048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:30.047048+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":359,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.089589+0000","last_change":"2024-02-05T06:40:08.596936+0000","last_active":"2024-02-05T06:41:10.089589+0000","last_peered":"2024-02-05T06:41:10.089589+0000","last_clean":"2024-02-05T06:41:10.089589+0000","last_became_active":"2024-02-05T06:40:08.596289+0000","last_became_peered":"2024-02-05T06:40:08.596289+0000","last_unstale":"2024-02-05T06:41:10.089589+0000","last_undegraded":"2024-02-05T06:41:10.089589+0000","last_fullsized":"2024-02-05T06:41:10.089589+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:30.450988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:30.450988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":385,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.090332+0000","last_change":"2024-02-05T06:40:27.080858+0000","last_active":"2024-02-05T06:41:10.090332+0000","last_peered":"2024-02-05T06:41:10.090332+0000","last_clean":"2024-02-05T06:41:10.090332+0000","last_became_active":"2024-02-05T06:40:27.080531+0000","last_became_peered":"2024-02-05T06:40:27.080531+0000","last_unstale":"2024-02-05T06:41:10.090332+0000","last_undegraded":"2024-02-05T06:41:10.090332+0000","last_fullsized":"2024-02-05T06:41:10.090332+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:35.213313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:35.213313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:10.206464+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":262,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.089496+0000","last_change":"2024-02-05T06:40:46.639977+0000","last_active":"2024-02-05T06:41:10.089496+0000","last_peered":"2024-02-05T06:41:10.089496+0000","last_clean":"2024-02-05T06:41:10.089496+0000","last_became_active":"2024-02-05T06:40:27.065047+0000","last_became_peered":"2024-02-05T06:40:27.065047+0000","last_unstale":"2024-02-05T06:41:10.089496+0000","last_undegraded":"2024-02-05T06:41:10.089496+0000","last_fullsized":"2024-02-05T06:41:10.089496+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:40:46.639897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:40:46.639897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:51.971299+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"92'16","reported_seq":254,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.089412+0000","last_change":"2024-02-05T06:36:54.999613+0000","last_active":"2024-02-05T06:41:10.089412+0000","last_peered":"2024-02-05T06:41:10.089412+0000","last_clean":"2024-02-05T06:41:10.089412+0000","last_became_active":"2024-02-05T06:36:54.961610+0000","last_became_peered":"2024-02-05T06:36:54.961610+0000","last_unstale":"2024-02-05T06:41:10.089412+0000","last_undegraded":"2024-02-05T06:41:10.089412+0000","last_fullsized":"2024-02-05T06:41:10.089412+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'16","last_scrub_stamp":"2024-02-05T06:36:36.728552+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:35:02.589229+0000","last_clean_scrub_stamp":"2024-02-05T06:36:36.728552+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":366,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.099064+0000","last_change":"2024-02-05T06:40:27.124829+0000","last_active":"2024-02-05T06:41:10.099064+0000","last_peered":"2024-02-05T06:41:10.099064+0000","last_clean":"2024-02-05T06:41:10.099064+0000","last_became_active":"2024-02-05T06:40:27.096132+0000","last_became_peered":"2024-02-05T06:40:27.096132+0000","last_unstale":"2024-02-05T06:41:10.099064+0000","last_undegraded":"2024-02-05T06:41:10.099064+0000","last_fullsized":"2024-02-05T06:41:10.099064+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:31.831291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:31.831291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"197'9","reported_seq":131,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.088987+0000","last_change":"2024-02-05T06:40:08.659377+0000","last_active":"2024-02-05T06:41:10.088987+0000","last_peered":"2024-02-05T06:41:10.088987+0000","last_clean":"2024-02-05T06:41:10.088987+0000","last_became_active":"2024-02-05T06:40:08.596051+0000","last_became_peered":"2024-02-05T06:40:08.596051+0000","last_unstale":"2024-02-05T06:41:10.088987+0000","last_undegraded":"2024-02-05T06:41:10.088987+0000","last_fullsized":"2024-02-05T06:41:10.088987+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:39:09.504696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:09.504696+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:18.214461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":67,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.094440+0000","last_change":"2024-02-05T06:40:27.098718+0000","last_active":"2024-02-05T06:41:10.094440+0000","last_peered":"2024-02-05T06:41:10.094440+0000","last_clean":"2024-02-05T06:41:10.094440+0000","last_became_active":"2024-02-05T06:40:27.098418+0000","last_became_peered":"2024-02-05T06:40:27.098418+0000","last_unstale":"2024-02-05T06:41:10.094440+0000","last_undegraded":"2024-02-05T06:41:10.094440+0000","last_fullsized":"2024-02-05T06:41:10.094440+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:34.651493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:37:34.651493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'10","reported_seq":430,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.092161+0000","last_change":"2024-02-05T06:40:27.198874+0000","last_active":"2024-02-05T06:41:10.092161+0000","last_peered":"2024-02-05T06:41:10.092161+0000","last_clean":"2024-02-05T06:41:10.092161+0000","last_became_active":"2024-02-05T06:40:27.093418+0000","last_became_peered":"2024-02-05T06:40:27.093418+0000","last_unstale":"2024-02-05T06:41:10.092161+0000","last_undegraded":"2024-02-05T06:41:10.092161+0000","last_fullsized":"2024-02-05T06:41:10.092161+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-02-05T06:38:21.205097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:38:21.205097+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":256,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.089769+0000","last_change":"2024-02-05T06:40:08.583062+0000","last_active":"2024-02-05T06:41:10.089769+0000","last_peered":"2024-02-05T06:41:10.089769+0000","last_clean":"2024-02-05T06:41:10.089769+0000","last_became_active":"2024-02-05T06:40:08.582586+0000","last_became_peered":"2024-02-05T06:40:08.582586+0000","last_unstale":"2024-02-05T06:41:10.089769+0000","last_undegraded":"2024-02-05T06:41:10.089769+0000","last_fullsized":"2024-02-05T06:41:10.089769+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:18.964707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:18.964707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"208'10","reported_seq":479,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.092706+0000","last_change":"2024-02-05T06:40:27.202569+0000","last_active":"2024-02-05T06:41:10.092706+0000","last_peered":"2024-02-05T06:41:10.092706+0000","last_clean":"2024-02-05T06:41:10.092706+0000","last_became_active":"2024-02-05T06:40:27.096584+0000","last_became_peered":"2024-02-05T06:40:27.096584+0000","last_unstale":"2024-02-05T06:41:10.092706+0000","last_undegraded":"2024-02-05T06:41:10.092706+0000","last_fullsized":"2024-02-05T06:41:10.092706+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:36:26.411576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:26.411576+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":378,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.086990+0000","last_change":"2024-02-05T06:40:27.088637+0000","last_active":"2024-02-05T06:41:10.086990+0000","last_peered":"2024-02-05T06:41:10.086990+0000","last_clean":"2024-02-05T06:41:10.086990+0000","last_became_active":"2024-02-05T06:40:27.088096+0000","last_became_peered":"2024-02-05T06:40:27.088096+0000","last_unstale":"2024-02-05T06:41:10.086990+0000","last_undegraded":"2024-02-05T06:41:10.086990+0000","last_fullsized":"2024-02-05T06:41:10.086990+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:31.169691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:36:31.169691+0000","last_clean_scrub_stamp":"2024-02-05T06:36:31.169691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"73'9","reported_seq":77,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.095085+0000","last_change":"2024-02-05T06:40:27.381430+0000","last_active":"2024-02-05T06:41:10.095085+0000","last_peered":"2024-02-05T06:41:10.095085+0000","last_clean":"2024-02-05T06:41:10.095085+0000","last_became_active":"2024-02-05T06:40:27.140868+0000","last_became_peered":"2024-02-05T06:40:27.140868+0000","last_unstale":"2024-02-05T06:41:10.095085+0000","last_undegraded":"2024-02-05T06:41:10.095085+0000","last_fullsized":"2024-02-05T06:41:10.095085+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:39:16.394399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:16.394399+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":391,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.094582+0000","last_change":"2024-02-05T06:40:27.092829+0000","last_active":"2024-02-05T06:41:10.094582+0000","last_peered":"2024-02-05T06:41:10.094582+0000","last_clean":"2024-02-05T06:41:10.094582+0000","last_became_active":"2024-02-05T06:40:27.090632+0000","last_became_peered":"2024-02-05T06:40:27.090632+0000","last_unstale":"2024-02-05T06:41:10.094582+0000","last_undegraded":"2024-02-05T06:41:10.094582+0000","last_fullsized":"2024-02-05T06:41:10.094582+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:19.054356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:19.054356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:12.649561+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":383,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.096173+0000","last_change":"2024-02-05T06:40:27.092004+0000","last_active":"2024-02-05T06:41:10.096173+0000","last_peered":"2024-02-05T06:41:10.096173+0000","last_clean":"2024-02-05T06:41:10.096173+0000","last_became_active":"2024-02-05T06:40:27.091544+0000","last_became_peered":"2024-02-05T06:40:27.091544+0000","last_unstale":"2024-02-05T06:41:10.096173+0000","last_undegraded":"2024-02-05T06:41:10.096173+0000","last_fullsized":"2024-02-05T06:41:10.096173+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:40:03.572450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:40:03.572450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:11.986980+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":67,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.093644+0000","last_change":"2024-02-05T06:40:27.064705+0000","last_active":"2024-02-05T06:41:10.093644+0000","last_peered":"2024-02-05T06:41:10.093644+0000","last_clean":"2024-02-05T06:41:10.093644+0000","last_became_active":"2024-02-05T06:40:27.064388+0000","last_became_peered":"2024-02-05T06:40:27.064388+0000","last_unstale":"2024-02-05T06:41:10.093644+0000","last_undegraded":"2024-02-05T06:41:10.093644+0000","last_fullsized":"2024-02-05T06:41:10.093644+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:38:13.558400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:38:13.558400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"73'9","reported_seq":410,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.099505+0000","last_change":"2024-02-05T06:40:46.447632+0000","last_active":"2024-02-05T06:41:10.099505+0000","last_peered":"2024-02-05T06:41:10.099505+0000","last_clean":"2024-02-05T06:41:10.099505+0000","last_became_active":"2024-02-05T06:40:27.101116+0000","last_became_peered":"2024-02-05T06:40:27.101116+0000","last_unstale":"2024-02-05T06:41:10.099505+0000","last_undegraded":"2024-02-05T06:41:10.099505+0000","last_fullsized":"2024-02-05T06:41:10.099505+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:36:23.148541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:23.148541+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":67,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.095094+0000","last_change":"2024-02-05T06:40:27.063878+0000","last_active":"2024-02-05T06:41:10.095094+0000","last_peered":"2024-02-05T06:41:10.095094+0000","last_clean":"2024-02-05T06:41:10.095094+0000","last_became_active":"2024-02-05T06:40:27.063350+0000","last_became_peered":"2024-02-05T06:40:27.063350+0000","last_unstale":"2024-02-05T06:41:10.095094+0000","last_undegraded":"2024-02-05T06:41:10.095094+0000","last_fullsized":"2024-02-05T06:41:10.095094+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:38:16.709585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:38:16.709585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":88,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.094095+0000","last_change":"2024-02-05T06:40:08.605454+0000","last_active":"2024-02-05T06:41:10.094095+0000","last_peered":"2024-02-05T06:41:10.094095+0000","last_clean":"2024-02-05T06:41:10.094095+0000","last_became_active":"2024-02-05T06:40:08.605110+0000","last_became_peered":"2024-02-05T06:40:08.605110+0000","last_unstale":"2024-02-05T06:41:10.094095+0000","last_undegraded":"2024-02-05T06:41:10.094095+0000","last_fullsized":"2024-02-05T06:41:10.094095+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:29.372079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:45.096340+0000","last_clean_scrub_stamp":"2024-02-05T06:39:29.372079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":386,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.027915+0000","last_change":"2024-02-05T06:40:08.590689+0000","last_active":"2024-02-05T06:41:10.027915+0000","last_peered":"2024-02-05T06:41:10.027915+0000","last_clean":"2024-02-05T06:41:10.027915+0000","last_became_active":"2024-02-05T06:40:08.590234+0000","last_became_peered":"2024-02-05T06:40:08.590234+0000","last_unstale":"2024-02-05T06:41:10.027915+0000","last_undegraded":"2024-02-05T06:41:10.027915+0000","last_fullsized":"2024-02-05T06:41:10.027915+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:22.309939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:39:22.309939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":398,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.096254+0000","last_change":"2024-02-05T06:40:27.110051+0000","last_active":"2024-02-05T06:41:10.096254+0000","last_peered":"2024-02-05T06:41:10.096254+0000","last_clean":"2024-02-05T06:41:10.096254+0000","last_became_active":"2024-02-05T06:40:27.091689+0000","last_became_peered":"2024-02-05T06:40:27.091689+0000","last_unstale":"2024-02-05T06:41:10.096254+0000","last_undegraded":"2024-02-05T06:41:10.096254+0000","last_fullsized":"2024-02-05T06:41:10.096254+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'6","last_scrub_stamp":"2024-02-05T06:38:32.122868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:38:32.122868+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"234'5124","reported_seq":9028,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:16.181749+0000","last_change":"2024-02-05T06:40:20.020947+0000","last_active":"2024-02-05T06:41:16.181749+0000","last_peered":"2024-02-05T06:41:16.181749+0000","last_clean":"2024-02-05T06:41:16.181749+0000","last_became_active":"2024-02-05T06:40:20.018366+0000","last_became_peered":"2024-02-05T06:40:20.018366+0000","last_unstale":"2024-02-05T06:41:16.181749+0000","last_undegraded":"2024-02-05T06:41:16.181749+0000","last_fullsized":"2024-02-05T06:41:16.181749+0000","mapping_epoch":201,"log_start":"192'900","ondisk_log_start":"192'900","created":185,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":4224,"log_dups_size":0,"ondisk_log_size":4224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:17.472863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335806464,"num_objects":5124,"num_object_clones":0,"num_object_copies":15372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5124,"num_write_kb":327936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1894,"num_bytes_recovered":124125184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":88,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.028013+0000","last_change":"2024-02-05T06:40:08.591629+0000","last_active":"2024-02-05T06:41:10.028013+0000","last_peered":"2024-02-05T06:41:10.028013+0000","last_clean":"2024-02-05T06:41:10.028013+0000","last_became_active":"2024-02-05T06:40:08.590874+0000","last_became_peered":"2024-02-05T06:40:08.590874+0000","last_unstale":"2024-02-05T06:41:10.028013+0000","last_undegraded":"2024-02-05T06:41:10.028013+0000","last_fullsized":"2024-02-05T06:41:10.028013+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:23.325194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:23.325194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":73,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.094968+0000","last_change":"2024-02-05T06:40:27.105096+0000","last_active":"2024-02-05T06:41:10.094968+0000","last_peered":"2024-02-05T06:41:10.094968+0000","last_clean":"2024-02-05T06:41:10.094968+0000","last_became_active":"2024-02-05T06:40:27.068279+0000","last_became_peered":"2024-02-05T06:40:27.068279+0000","last_unstale":"2024-02-05T06:41:10.094968+0000","last_undegraded":"2024-02-05T06:41:10.094968+0000","last_fullsized":"2024-02-05T06:41:10.094968+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-02-05T06:37:28.708774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:37:28.708774+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"73'15","reported_seq":83,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.093256+0000","last_change":"2024-02-05T06:40:27.377323+0000","last_active":"2024-02-05T06:41:10.093256+0000","last_peered":"2024-02-05T06:41:10.093256+0000","last_clean":"2024-02-05T06:41:10.093256+0000","last_became_active":"2024-02-05T06:40:27.062755+0000","last_became_peered":"2024-02-05T06:40:27.062755+0000","last_unstale":"2024-02-05T06:41:10.093256+0000","last_undegraded":"2024-02-05T06:41:10.093256+0000","last_fullsized":"2024-02-05T06:41:10.093256+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-02-05T06:40:01.321025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:40:01.321025+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"208'12","reported_seq":81,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.093611+0000","last_change":"2024-02-05T06:40:27.091490+0000","last_active":"2024-02-05T06:41:10.093611+0000","last_peered":"2024-02-05T06:41:10.093611+0000","last_clean":"2024-02-05T06:41:10.093611+0000","last_became_active":"2024-02-05T06:40:27.064195+0000","last_became_peered":"2024-02-05T06:40:27.064195+0000","last_unstale":"2024-02-05T06:41:10.093611+0000","last_undegraded":"2024-02-05T06:41:10.093611+0000","last_fullsized":"2024-02-05T06:41:10.093611+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-02-05T06:38:11.542821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:38:11.542821+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"234'4989","reported_seq":10491,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:16.204897+0000","last_change":"2024-02-05T06:40:47.342532+0000","last_active":"2024-02-05T06:41:16.204897+0000","last_peered":"2024-02-05T06:41:16.204897+0000","last_clean":"2024-02-05T06:41:16.204897+0000","last_became_active":"2024-02-05T06:40:47.337156+0000","last_became_peered":"2024-02-05T06:40:47.337156+0000","last_unstale":"2024-02-05T06:41:16.204897+0000","last_undegraded":"2024-02-05T06:41:16.204897+0000","last_fullsized":"2024-02-05T06:41:16.204897+0000","mapping_epoch":221,"log_start":"192'400","ondisk_log_start":"192'400","created":185,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":4589,"log_dups_size":0,"ondisk_log_size":4589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326959104,"num_objects":4989,"num_object_clones":0,"num_object_copies":14967,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4989,"num_write_kb":319296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2692,"num_bytes_recovered":176422912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":365,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.090943+0000","last_change":"2024-02-05T06:40:27.092917+0000","last_active":"2024-02-05T06:41:10.090943+0000","last_peered":"2024-02-05T06:41:10.090943+0000","last_clean":"2024-02-05T06:41:10.090943+0000","last_became_active":"2024-02-05T06:40:27.092613+0000","last_became_peered":"2024-02-05T06:40:27.092613+0000","last_unstale":"2024-02-05T06:41:10.090943+0000","last_undegraded":"2024-02-05T06:41:10.090943+0000","last_fullsized":"2024-02-05T06:41:10.090943+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:26.390356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:26.390356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":67,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.095017+0000","last_change":"2024-02-05T06:40:27.063835+0000","last_active":"2024-02-05T06:41:10.095017+0000","last_peered":"2024-02-05T06:41:10.095017+0000","last_clean":"2024-02-05T06:41:10.095017+0000","last_became_active":"2024-02-05T06:40:27.063300+0000","last_became_peered":"2024-02-05T06:40:27.063300+0000","last_unstale":"2024-02-05T06:41:10.095017+0000","last_undegraded":"2024-02-05T06:41:10.095017+0000","last_fullsized":"2024-02-05T06:41:10.095017+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:38:15.583491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:38:15.583491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'12","reported_seq":263,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.089023+0000","last_change":"2024-02-05T06:39:09.987283+0000","last_active":"2024-02-05T06:41:10.089023+0000","last_peered":"2024-02-05T06:41:10.089023+0000","last_clean":"2024-02-05T06:41:10.089023+0000","last_became_active":"2024-02-05T06:36:54.954071+0000","last_became_peered":"2024-02-05T06:36:54.954071+0000","last_unstale":"2024-02-05T06:41:10.089023+0000","last_undegraded":"2024-02-05T06:41:10.089023+0000","last_fullsized":"2024-02-05T06:41:10.089023+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:39:09.987177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:09.987177+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:17.472738+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":363,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.093799+0000","last_change":"2024-02-05T06:39:35.954431+0000","last_active":"2024-02-05T06:41:10.093799+0000","last_peered":"2024-02-05T06:41:10.093799+0000","last_clean":"2024-02-05T06:41:10.093799+0000","last_became_active":"2024-02-05T06:33:33.592546+0000","last_became_peered":"2024-02-05T06:33:33.592546+0000","last_unstale":"2024-02-05T06:41:10.093799+0000","last_undegraded":"2024-02-05T06:41:10.093799+0000","last_fullsized":"2024-02-05T06:41:10.093799+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:35.954359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:35.954359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:12.649624+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"234'5052","reported_seq":6433,"reported_epoch":234,"state":"active+recovering+undersized+remapped","last_fresh":"2024-02-05T06:41:16.893190+0000","last_change":"2024-02-05T06:40:58.811691+0000","last_active":"2024-02-05T06:41:16.893190+0000","last_peered":"2024-02-05T06:41:16.893190+0000","last_clean":"2024-02-05T06:40:26.040508+0000","last_became_active":"2024-02-05T06:40:28.168410+0000","last_became_peered":"2024-02-05T06:40:28.168410+0000","last_unstale":"2024-02-05T06:41:16.893190+0000","last_undegraded":"2024-02-05T06:41:16.893190+0000","last_fullsized":"2024-02-05T06:40:28.058589+0000","mapping_epoch":206,"log_start":"192'400","ondisk_log_start":"192'400","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":4652,"log_dups_size":0,"ondisk_log_size":4652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331087872,"num_objects":5052,"num_object_clones":0,"num_object_copies":15156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2138,"num_objects_unfound":0,"num_objects_dirty":5052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5052,"num_write_kb":323328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":635,"num_bytes_recovered":41615360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6],"avail_no_missing":["0","2","6"],"object_location_counts":[{"shards":"0,2,6","objects":2138},{"shards":"0,6","objects":2914}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":67,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.093463+0000","last_change":"2024-02-05T06:40:27.218419+0000","last_active":"2024-02-05T06:41:10.093463+0000","last_peered":"2024-02-05T06:41:10.093463+0000","last_clean":"2024-02-05T06:41:10.093463+0000","last_became_active":"2024-02-05T06:40:27.218081+0000","last_became_peered":"2024-02-05T06:40:27.218081+0000","last_unstale":"2024-02-05T06:41:10.093463+0000","last_undegraded":"2024-02-05T06:41:10.093463+0000","last_fullsized":"2024-02-05T06:41:10.093463+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:32.925000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:32.925000+0000","last_clean_scrub_stamp":"2024-02-05T06:39:32.925000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":375,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.096722+0000","last_change":"2024-02-05T06:40:27.089797+0000","last_active":"2024-02-05T06:41:10.096722+0000","last_peered":"2024-02-05T06:41:10.096722+0000","last_clean":"2024-02-05T06:41:10.096722+0000","last_became_active":"2024-02-05T06:40:27.089489+0000","last_became_peered":"2024-02-05T06:40:27.089489+0000","last_unstale":"2024-02-05T06:41:10.096722+0000","last_undegraded":"2024-02-05T06:41:10.096722+0000","last_fullsized":"2024-02-05T06:41:10.096722+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:38:11.612313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:38:11.612313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:12.649502+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":67,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.094992+0000","last_change":"2024-02-05T06:40:27.078034+0000","last_active":"2024-02-05T06:41:10.094992+0000","last_peered":"2024-02-05T06:41:10.094992+0000","last_clean":"2024-02-05T06:41:10.094992+0000","last_became_active":"2024-02-05T06:40:27.077534+0000","last_became_peered":"2024-02-05T06:40:27.077534+0000","last_unstale":"2024-02-05T06:41:10.094992+0000","last_undegraded":"2024-02-05T06:41:10.094992+0000","last_fullsized":"2024-02-05T06:41:10.094992+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:40:04.284398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:40:04.284398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:23.943119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"73'19","reported_seq":485,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.028239+0000","last_change":"2024-02-05T06:40:27.083505+0000","last_active":"2024-02-05T06:41:10.028239+0000","last_peered":"2024-02-05T06:41:10.028239+0000","last_clean":"2024-02-05T06:41:10.028239+0000","last_became_active":"2024-02-05T06:40:27.068417+0000","last_became_peered":"2024-02-05T06:40:27.068417+0000","last_unstale":"2024-02-05T06:41:10.028239+0000","last_undegraded":"2024-02-05T06:41:10.028239+0000","last_fullsized":"2024-02-05T06:41:10.028239+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-02-05T06:39:34.480345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:34.480345+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"234'4997","reported_seq":5105,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:16.464599+0000","last_change":"2024-02-05T06:39:42.012271+0000","last_active":"2024-02-05T06:41:16.464599+0000","last_peered":"2024-02-05T06:41:16.464599+0000","last_clean":"2024-02-05T06:41:16.464599+0000","last_became_active":"2024-02-05T06:39:42.011938+0000","last_became_peered":"2024-02-05T06:39:42.011938+0000","last_unstale":"2024-02-05T06:41:16.464599+0000","last_undegraded":"2024-02-05T06:41:16.464599+0000","last_fullsized":"2024-02-05T06:41:16.464599+0000","mapping_epoch":185,"log_start":"192'600","ondisk_log_start":"192'600","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":4397,"log_dups_size":0,"ondisk_log_size":4397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:18.217308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327483392,"num_objects":4997,"num_object_clones":0,"num_object_copies":14991,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4997,"num_write_kb":319808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":358,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.091485+0000","last_change":"2024-02-05T06:40:27.130950+0000","last_active":"2024-02-05T06:41:10.091485+0000","last_peered":"2024-02-05T06:41:10.091485+0000","last_clean":"2024-02-05T06:41:10.091485+0000","last_became_active":"2024-02-05T06:40:27.129625+0000","last_became_peered":"2024-02-05T06:40:27.129625+0000","last_unstale":"2024-02-05T06:41:10.091485+0000","last_undegraded":"2024-02-05T06:41:10.091485+0000","last_fullsized":"2024-02-05T06:41:10.091485+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:22.102133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:22.102133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":383,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.095817+0000","last_change":"2024-02-05T06:39:34.909566+0000","last_active":"2024-02-05T06:41:10.095817+0000","last_peered":"2024-02-05T06:41:10.095817+0000","last_clean":"2024-02-05T06:41:10.095817+0000","last_became_active":"2024-02-05T06:36:54.966016+0000","last_became_peered":"2024-02-05T06:36:54.966016+0000","last_unstale":"2024-02-05T06:41:10.095817+0000","last_undegraded":"2024-02-05T06:41:10.095817+0000","last_fullsized":"2024-02-05T06:41:10.095817+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:34.909485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:34.909485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:12.649683+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":89,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.092253+0000","last_change":"2024-02-05T06:40:08.606060+0000","last_active":"2024-02-05T06:41:10.092253+0000","last_peered":"2024-02-05T06:41:10.092253+0000","last_clean":"2024-02-05T06:41:10.092253+0000","last_became_active":"2024-02-05T06:40:08.605597+0000","last_became_peered":"2024-02-05T06:40:08.605597+0000","last_unstale":"2024-02-05T06:41:10.092253+0000","last_undegraded":"2024-02-05T06:41:10.092253+0000","last_fullsized":"2024-02-05T06:41:10.092253+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:27.314329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:37:27.314329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"73'9","reported_seq":458,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.087469+0000","last_change":"2024-02-05T06:40:08.611772+0000","last_active":"2024-02-05T06:41:10.087469+0000","last_peered":"2024-02-05T06:41:10.087469+0000","last_clean":"2024-02-05T06:41:10.087469+0000","last_became_active":"2024-02-05T06:40:08.584204+0000","last_became_peered":"2024-02-05T06:40:08.584204+0000","last_unstale":"2024-02-05T06:41:10.087469+0000","last_undegraded":"2024-02-05T06:41:10.087469+0000","last_fullsized":"2024-02-05T06:41:10.087469+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:36:26.172612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:26.172612+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"234'5274","reported_seq":11528,"reported_epoch":234,"state":"active+recovering+undersized+remapped","last_fresh":"2024-02-05T06:41:16.890506+0000","last_change":"2024-02-05T06:40:27.365455+0000","last_active":"2024-02-05T06:41:16.890506+0000","last_peered":"2024-02-05T06:41:16.890506+0000","last_clean":"2024-02-05T06:40:07.517588+0000","last_became_active":"2024-02-05T06:40:27.269349+0000","last_became_peered":"2024-02-05T06:40:27.269349+0000","last_unstale":"2024-02-05T06:41:16.890506+0000","last_undegraded":"2024-02-05T06:41:16.890506+0000","last_fullsized":"2024-02-05T06:40:27.052939+0000","mapping_epoch":205,"log_start":"193'1300","ondisk_log_start":"193'1300","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":3974,"log_dups_size":0,"ondisk_log_size":3974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":345636864,"num_objects":5274,"num_object_clones":0,"num_object_copies":15822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":306,"num_objects_unfound":0,"num_objects_dirty":5274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5274,"num_write_kb":337536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3069,"num_bytes_recovered":201129984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,0],"avail_no_missing":["3","0","5"],"object_location_counts":[{"shards":"0,2,3","objects":306},{"shards":"0,3","objects":4968}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":396,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.091163+0000","last_change":"2024-02-05T06:40:11.050188+0000","last_active":"2024-02-05T06:41:10.091163+0000","last_peered":"2024-02-05T06:41:10.091163+0000","last_clean":"2024-02-05T06:41:10.091163+0000","last_became_active":"2024-02-05T06:40:11.049515+0000","last_became_peered":"2024-02-05T06:40:11.049515+0000","last_unstale":"2024-02-05T06:41:10.091163+0000","last_undegraded":"2024-02-05T06:41:10.091163+0000","last_fullsized":"2024-02-05T06:41:10.091163+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:37:37.282442+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:37.282442+0000","last_clean_scrub_stamp":"2024-02-05T06:37:37.282442+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:14.694647+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":377,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.094168+0000","last_change":"2024-02-05T06:40:27.082866+0000","last_active":"2024-02-05T06:41:10.094168+0000","last_peered":"2024-02-05T06:41:10.094168+0000","last_clean":"2024-02-05T06:41:10.094168+0000","last_became_active":"2024-02-05T06:40:27.082440+0000","last_became_peered":"2024-02-05T06:40:27.082440+0000","last_unstale":"2024-02-05T06:41:10.094168+0000","last_undegraded":"2024-02-05T06:41:10.094168+0000","last_fullsized":"2024-02-05T06:41:10.094168+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:07.364172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:07.364172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"182'3","reported_seq":365,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.090514+0000","last_change":"2024-02-05T06:40:08.658727+0000","last_active":"2024-02-05T06:41:10.090514+0000","last_peered":"2024-02-05T06:41:10.090514+0000","last_clean":"2024-02-05T06:41:10.090514+0000","last_became_active":"2024-02-05T06:40:08.594540+0000","last_became_peered":"2024-02-05T06:40:08.594540+0000","last_unstale":"2024-02-05T06:41:10.090514+0000","last_undegraded":"2024-02-05T06:41:10.090514+0000","last_fullsized":"2024-02-05T06:41:10.090514+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'3","last_scrub_stamp":"2024-02-05T06:39:33.373415+0000","last_deep_scrub":"182'3","last_deep_scrub_stamp":"2024-02-05T06:39:33.373415+0000","last_clean_scrub_stamp":"2024-02-05T06:39:33.373415+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:10.206507+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001220348,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":251,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.089352+0000","last_change":"2024-02-05T06:39:34.999778+0000","last_active":"2024-02-05T06:41:10.089352+0000","last_peered":"2024-02-05T06:41:10.089352+0000","last_clean":"2024-02-05T06:41:10.089352+0000","last_became_active":"2024-02-05T06:36:54.968205+0000","last_became_peered":"2024-02-05T06:36:54.968205+0000","last_unstale":"2024-02-05T06:41:10.089352+0000","last_undegraded":"2024-02-05T06:41:10.089352+0000","last_fullsized":"2024-02-05T06:41:10.089352+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:34.999659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:39:34.999659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"197'30","reported_seq":123,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.089937+0000","last_change":"2024-02-05T06:40:08.673390+0000","last_active":"2024-02-05T06:41:10.089937+0000","last_peered":"2024-02-05T06:41:10.089937+0000","last_clean":"2024-02-05T06:41:10.089937+0000","last_became_active":"2024-02-05T06:40:08.604483+0000","last_became_peered":"2024-02-05T06:40:08.604483+0000","last_unstale":"2024-02-05T06:41:10.089937+0000","last_undegraded":"2024-02-05T06:41:10.089937+0000","last_fullsized":"2024-02-05T06:41:10.089937+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-02-05T06:39:30.346384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:30.346384+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:18.214856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":370,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.087222+0000","last_change":"2024-02-05T06:36:54.974218+0000","last_active":"2024-02-05T06:41:10.087222+0000","last_peered":"2024-02-05T06:41:10.087222+0000","last_clean":"2024-02-05T06:41:10.087222+0000","last_became_active":"2024-02-05T06:36:54.973387+0000","last_became_peered":"2024-02-05T06:36:54.973387+0000","last_unstale":"2024-02-05T06:41:10.087222+0000","last_undegraded":"2024-02-05T06:41:10.087222+0000","last_fullsized":"2024-02-05T06:41:10.087222+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:36.161606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:36.161606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"234'5178","reported_seq":5285,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:16.872152+0000","last_change":"2024-02-05T06:39:42.011437+0000","last_active":"2024-02-05T06:41:16.872152+0000","last_peered":"2024-02-05T06:41:16.872152+0000","last_clean":"2024-02-05T06:41:16.872152+0000","last_became_active":"2024-02-05T06:39:42.011116+0000","last_became_peered":"2024-02-05T06:39:42.011116+0000","last_unstale":"2024-02-05T06:41:16.872152+0000","last_undegraded":"2024-02-05T06:41:16.872152+0000","last_fullsized":"2024-02-05T06:41:16.872152+0000","mapping_epoch":185,"log_start":"193'1200","ondisk_log_start":"193'1200","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":3978,"log_dups_size":0,"ondisk_log_size":3978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339345408,"num_objects":5178,"num_object_clones":0,"num_object_copies":15534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5178,"num_write_kb":331392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":488,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.092877+0000","last_change":"2024-02-05T06:40:30.082941+0000","last_active":"2024-02-05T06:41:10.092877+0000","last_peered":"2024-02-05T06:41:10.092877+0000","last_clean":"2024-02-05T06:41:10.092877+0000","last_became_active":"2024-02-05T06:40:30.082467+0000","last_became_peered":"2024-02-05T06:40:30.082467+0000","last_unstale":"2024-02-05T06:41:10.092877+0000","last_undegraded":"2024-02-05T06:41:10.092877+0000","last_fullsized":"2024-02-05T06:41:10.092877+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:39:09.930291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:39:09.930291+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":67,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.094763+0000","last_change":"2024-02-05T06:40:27.078458+0000","last_active":"2024-02-05T06:41:10.094763+0000","last_peered":"2024-02-05T06:41:10.094763+0000","last_clean":"2024-02-05T06:41:10.094763+0000","last_became_active":"2024-02-05T06:40:27.077966+0000","last_became_peered":"2024-02-05T06:40:27.077966+0000","last_unstale":"2024-02-05T06:41:10.094763+0000","last_undegraded":"2024-02-05T06:41:10.094763+0000","last_fullsized":"2024-02-05T06:41:10.094763+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:36.461211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:36.461211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":373,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.087912+0000","last_change":"2024-02-05T06:39:27.282286+0000","last_active":"2024-02-05T06:41:10.087912+0000","last_peered":"2024-02-05T06:41:10.087912+0000","last_clean":"2024-02-05T06:41:10.087912+0000","last_became_active":"2024-02-05T06:33:29.551549+0000","last_became_peered":"2024-02-05T06:33:29.551549+0000","last_unstale":"2024-02-05T06:41:10.087912+0000","last_undegraded":"2024-02-05T06:41:10.087912+0000","last_fullsized":"2024-02-05T06:41:10.087912+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:27.282189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:39:27.282189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:10.206348+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":380,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.095229+0000","last_change":"2024-02-05T06:37:31.522883+0000","last_active":"2024-02-05T06:41:10.095229+0000","last_peered":"2024-02-05T06:41:10.095229+0000","last_clean":"2024-02-05T06:41:10.095229+0000","last_became_active":"2024-02-05T06:33:28.542977+0000","last_became_peered":"2024-02-05T06:33:28.542977+0000","last_unstale":"2024-02-05T06:41:10.095229+0000","last_undegraded":"2024-02-05T06:41:10.095229+0000","last_fullsized":"2024-02-05T06:41:10.095229+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:31.522765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:37:31.522765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:12.649437+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0.00091640799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'13","reported_seq":440,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.087161+0000","last_change":"2024-02-05T06:40:27.140182+0000","last_active":"2024-02-05T06:41:10.087161+0000","last_peered":"2024-02-05T06:41:10.087161+0000","last_clean":"2024-02-05T06:41:10.087161+0000","last_became_active":"2024-02-05T06:40:27.070536+0000","last_became_peered":"2024-02-05T06:40:27.070536+0000","last_unstale":"2024-02-05T06:41:10.087161+0000","last_undegraded":"2024-02-05T06:41:10.087161+0000","last_fullsized":"2024-02-05T06:41:10.087161+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-02-05T06:39:33.280115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:33.280115+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:10.206167+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":389,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.096408+0000","last_change":"2024-02-05T06:40:27.089066+0000","last_active":"2024-02-05T06:41:10.096408+0000","last_peered":"2024-02-05T06:41:10.096408+0000","last_clean":"2024-02-05T06:41:10.096408+0000","last_became_active":"2024-02-05T06:40:27.088773+0000","last_became_peered":"2024-02-05T06:40:27.088773+0000","last_unstale":"2024-02-05T06:41:10.096408+0000","last_undegraded":"2024-02-05T06:41:10.096408+0000","last_fullsized":"2024-02-05T06:41:10.096408+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:33.914422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:33.914422+0000","last_clean_scrub_stamp":"2024-02-05T06:39:33.914422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:12.649653+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"234'5039","reported_seq":8945,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:16.218336+0000","last_change":"2024-02-05T06:40:32.112242+0000","last_active":"2024-02-05T06:41:16.218336+0000","last_peered":"2024-02-05T06:41:16.218336+0000","last_clean":"2024-02-05T06:41:16.218336+0000","last_became_active":"2024-02-05T06:40:32.107529+0000","last_became_peered":"2024-02-05T06:40:32.107529+0000","last_unstale":"2024-02-05T06:41:16.218336+0000","last_undegraded":"2024-02-05T06:41:16.218336+0000","last_fullsized":"2024-02-05T06:41:16.218336+0000","mapping_epoch":210,"log_start":"192'400","ondisk_log_start":"192'400","created":185,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":4639,"log_dups_size":0,"ondisk_log_size":4639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330235904,"num_objects":5039,"num_object_clones":0,"num_object_copies":15117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5039,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5039,"num_write_kb":322496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1894,"num_bytes_recovered":124125184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":203,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.096087+0000","last_change":"2024-02-05T06:40:08.607944+0000","last_active":"2024-02-05T06:41:10.096087+0000","last_peered":"2024-02-05T06:41:10.096087+0000","last_clean":"2024-02-05T06:41:10.096087+0000","last_became_active":"2024-02-05T06:40:08.607210+0000","last_became_peered":"2024-02-05T06:40:08.607210+0000","last_unstale":"2024-02-05T06:41:10.096087+0000","last_undegraded":"2024-02-05T06:41:10.096087+0000","last_fullsized":"2024-02-05T06:41:10.096087+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:38:17.284669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:38:17.284669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":488,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.092864+0000","last_change":"2024-02-05T06:40:30.081724+0000","last_active":"2024-02-05T06:41:10.092864+0000","last_peered":"2024-02-05T06:41:10.092864+0000","last_clean":"2024-02-05T06:41:10.092864+0000","last_became_active":"2024-02-05T06:40:30.081178+0000","last_became_peered":"2024-02-05T06:40:30.081178+0000","last_unstale":"2024-02-05T06:41:10.092864+0000","last_undegraded":"2024-02-05T06:41:10.092864+0000","last_fullsized":"2024-02-05T06:41:10.092864+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:39:08.974109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:39:08.974109+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":371,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.091196+0000","last_change":"2024-02-05T06:40:08.591738+0000","last_active":"2024-02-05T06:41:10.091196+0000","last_peered":"2024-02-05T06:41:10.091196+0000","last_clean":"2024-02-05T06:41:10.091196+0000","last_became_active":"2024-02-05T06:40:08.591426+0000","last_became_peered":"2024-02-05T06:40:08.591426+0000","last_unstale":"2024-02-05T06:41:10.091196+0000","last_undegraded":"2024-02-05T06:41:10.091196+0000","last_fullsized":"2024-02-05T06:41:10.091196+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:38:09.201827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:38:09.201827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"208'20","reported_seq":582,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.029095+0000","last_change":"2024-02-05T06:40:27.080702+0000","last_active":"2024-02-05T06:41:10.029095+0000","last_peered":"2024-02-05T06:41:10.029095+0000","last_clean":"2024-02-05T06:41:10.029095+0000","last_became_active":"2024-02-05T06:40:27.061111+0000","last_became_peered":"2024-02-05T06:40:27.061111+0000","last_unstale":"2024-02-05T06:41:10.029095+0000","last_undegraded":"2024-02-05T06:41:10.029095+0000","last_fullsized":"2024-02-05T06:41:10.029095+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'19","last_scrub_stamp":"2024-02-05T06:39:09.538463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:09.538463+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":67,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.095667+0000","last_change":"2024-02-05T06:40:27.186122+0000","last_active":"2024-02-05T06:41:10.095667+0000","last_peered":"2024-02-05T06:41:10.095667+0000","last_clean":"2024-02-05T06:41:10.095667+0000","last_became_active":"2024-02-05T06:40:27.185780+0000","last_became_peered":"2024-02-05T06:40:27.185780+0000","last_unstale":"2024-02-05T06:41:10.095667+0000","last_undegraded":"2024-02-05T06:41:10.095667+0000","last_fullsized":"2024-02-05T06:41:10.095667+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:31.636407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:37:31.636407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"234'5066","reported_seq":9071,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:16.460002+0000","last_change":"2024-02-05T06:40:57.125381+0000","last_active":"2024-02-05T06:41:16.460002+0000","last_peered":"2024-02-05T06:41:16.460002+0000","last_clean":"2024-02-05T06:41:16.460002+0000","last_became_active":"2024-02-05T06:40:57.069768+0000","last_became_peered":"2024-02-05T06:40:57.069768+0000","last_unstale":"2024-02-05T06:41:16.460002+0000","last_undegraded":"2024-02-05T06:41:16.460002+0000","last_fullsized":"2024-02-05T06:41:16.460002+0000","mapping_epoch":227,"log_start":"192'700","ondisk_log_start":"192'700","created":185,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":4366,"log_dups_size":0,"ondisk_log_size":4366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:19.213842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332005376,"num_objects":5066,"num_object_clones":0,"num_object_copies":15198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5066,"num_write_kb":324224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1943,"num_bytes_recovered":127336448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":528,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.089636+0000","last_change":"2024-02-05T06:39:24.339609+0000","last_active":"2024-02-05T06:41:10.089636+0000","last_peered":"2024-02-05T06:41:10.089636+0000","last_clean":"2024-02-05T06:41:10.089636+0000","last_became_active":"2024-02-05T06:37:37.137754+0000","last_became_peered":"2024-02-05T06:37:37.137754+0000","last_unstale":"2024-02-05T06:41:10.089636+0000","last_undegraded":"2024-02-05T06:41:10.089636+0000","last_fullsized":"2024-02-05T06:41:10.089636+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:39:24.339522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:39:24.339522+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:10.206050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":367,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.089560+0000","last_change":"2024-02-05T06:40:08.605617+0000","last_active":"2024-02-05T06:41:10.089560+0000","last_peered":"2024-02-05T06:41:10.089560+0000","last_clean":"2024-02-05T06:41:10.089560+0000","last_became_active":"2024-02-05T06:40:08.605015+0000","last_became_peered":"2024-02-05T06:40:08.605015+0000","last_unstale":"2024-02-05T06:41:10.089560+0000","last_undegraded":"2024-02-05T06:41:10.089560+0000","last_fullsized":"2024-02-05T06:41:10.089560+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:10.411120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:10.411120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:10.206017+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":378,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.091282+0000","last_change":"2024-02-05T06:36:54.973030+0000","last_active":"2024-02-05T06:41:10.091282+0000","last_peered":"2024-02-05T06:41:10.091282+0000","last_clean":"2024-02-05T06:41:10.091282+0000","last_became_active":"2024-02-05T06:36:54.972264+0000","last_became_peered":"2024-02-05T06:36:54.972264+0000","last_unstale":"2024-02-05T06:41:10.091282+0000","last_undegraded":"2024-02-05T06:41:10.091282+0000","last_fullsized":"2024-02-05T06:41:10.091282+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:24.213373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:24.213373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":121,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:15.133213+0000","last_change":"2024-02-05T06:40:27.193820+0000","last_active":"2024-02-05T06:41:15.133213+0000","last_peered":"2024-02-05T06:41:15.133213+0000","last_clean":"2024-02-05T06:41:15.133213+0000","last_became_active":"2024-02-05T06:40:27.071509+0000","last_became_peered":"2024-02-05T06:40:27.071509+0000","last_unstale":"2024-02-05T06:41:15.133213+0000","last_undegraded":"2024-02-05T06:41:15.133213+0000","last_fullsized":"2024-02-05T06:41:15.133213+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T06:40:03.205189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:40:03.205189+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:32.734819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074151600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":56,"num_read_kb":51,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":380,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.099175+0000","last_change":"2024-02-05T06:36:29.231478+0000","last_active":"2024-02-05T06:41:10.099175+0000","last_peered":"2024-02-05T06:41:10.099175+0000","last_clean":"2024-02-05T06:41:10.099175+0000","last_became_active":"2024-02-05T06:33:31.564604+0000","last_became_peered":"2024-02-05T06:33:31.564604+0000","last_unstale":"2024-02-05T06:41:10.099175+0000","last_undegraded":"2024-02-05T06:41:10.099175+0000","last_fullsized":"2024-02-05T06:41:10.099175+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:36:29.231352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:29.231352+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":88,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.087366+0000","last_change":"2024-02-05T06:40:08.590795+0000","last_active":"2024-02-05T06:41:10.087366+0000","last_peered":"2024-02-05T06:41:10.087366+0000","last_clean":"2024-02-05T06:41:10.087366+0000","last_became_active":"2024-02-05T06:40:08.589927+0000","last_became_peered":"2024-02-05T06:40:08.589927+0000","last_unstale":"2024-02-05T06:41:10.087366+0000","last_undegraded":"2024-02-05T06:41:10.087366+0000","last_fullsized":"2024-02-05T06:41:10.087366+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:40.273584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:37:40.273584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:10.205843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.f","version":"234'5180","reported_seq":5287,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:16.891505+0000","last_change":"2024-02-05T06:39:42.682421+0000","last_active":"2024-02-05T06:41:16.891505+0000","last_peered":"2024-02-05T06:41:16.891505+0000","last_clean":"2024-02-05T06:41:16.891505+0000","last_became_active":"2024-02-05T06:39:42.682109+0000","last_became_peered":"2024-02-05T06:39:42.682109+0000","last_unstale":"2024-02-05T06:41:16.891505+0000","last_undegraded":"2024-02-05T06:41:16.891505+0000","last_fullsized":"2024-02-05T06:41:16.891505+0000","mapping_epoch":185,"log_start":"193'1200","ondisk_log_start":"193'1200","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":3980,"log_dups_size":0,"ondisk_log_size":3980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339476480,"num_objects":5180,"num_object_clones":0,"num_object_copies":15540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5180,"num_write_kb":331520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":533,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.028870+0000","last_change":"2024-02-05T06:39:12.528803+0000","last_active":"2024-02-05T06:41:10.028870+0000","last_peered":"2024-02-05T06:41:10.028870+0000","last_clean":"2024-02-05T06:41:10.028870+0000","last_became_active":"2024-02-05T06:37:37.140011+0000","last_became_peered":"2024-02-05T06:37:37.140011+0000","last_unstale":"2024-02-05T06:41:10.028870+0000","last_undegraded":"2024-02-05T06:41:10.028870+0000","last_fullsized":"2024-02-05T06:41:10.028870+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:39:12.528724+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:38.778568+0000","last_clean_scrub_stamp":"2024-02-05T06:39:12.528724+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":361,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.087227+0000","last_change":"2024-02-05T06:39:22.330018+0000","last_active":"2024-02-05T06:41:10.087227+0000","last_peered":"2024-02-05T06:41:10.087227+0000","last_clean":"2024-02-05T06:41:10.087227+0000","last_became_active":"2024-02-05T06:33:35.664858+0000","last_became_peered":"2024-02-05T06:33:35.664858+0000","last_unstale":"2024-02-05T06:41:10.087227+0000","last_undegraded":"2024-02-05T06:41:10.087227+0000","last_fullsized":"2024-02-05T06:41:10.087227+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:22.329914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:22.329914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:10.206108+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"66'1","reported_seq":397,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.089773+0000","last_change":"2024-02-05T06:40:08.656245+0000","last_active":"2024-02-05T06:41:10.089773+0000","last_peered":"2024-02-05T06:41:10.089773+0000","last_clean":"2024-02-05T06:41:10.089773+0000","last_became_active":"2024-02-05T06:40:08.594925+0000","last_became_peered":"2024-02-05T06:40:08.594925+0000","last_unstale":"2024-02-05T06:41:10.089773+0000","last_undegraded":"2024-02-05T06:41:10.089773+0000","last_fullsized":"2024-02-05T06:41:10.089773+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:39:34.241779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:39:34.241779+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:10.206289+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'19","reported_seq":470,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.087745+0000","last_change":"2024-02-05T06:40:27.127864+0000","last_active":"2024-02-05T06:41:10.087745+0000","last_peered":"2024-02-05T06:41:10.087745+0000","last_clean":"2024-02-05T06:41:10.087745+0000","last_became_active":"2024-02-05T06:40:27.073899+0000","last_became_peered":"2024-02-05T06:40:27.073899+0000","last_unstale":"2024-02-05T06:41:10.087745+0000","last_undegraded":"2024-02-05T06:41:10.087745+0000","last_fullsized":"2024-02-05T06:41:10.087745+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-02-05T06:39:19.843545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:19.843545+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":370,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.028248+0000","last_change":"2024-02-05T06:40:08.591742+0000","last_active":"2024-02-05T06:41:10.028248+0000","last_peered":"2024-02-05T06:41:10.028248+0000","last_clean":"2024-02-05T06:41:10.028248+0000","last_became_active":"2024-02-05T06:40:08.591312+0000","last_became_peered":"2024-02-05T06:40:08.591312+0000","last_unstale":"2024-02-05T06:41:10.028248+0000","last_undegraded":"2024-02-05T06:41:10.028248+0000","last_fullsized":"2024-02-05T06:41:10.028248+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:32.449390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:32.449390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"234'5262","reported_seq":5367,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:16.932829+0000","last_change":"2024-02-05T06:39:42.684749+0000","last_active":"2024-02-05T06:41:16.932829+0000","last_peered":"2024-02-05T06:41:16.932829+0000","last_clean":"2024-02-05T06:41:16.932829+0000","last_became_active":"2024-02-05T06:39:42.684454+0000","last_became_peered":"2024-02-05T06:39:42.684454+0000","last_unstale":"2024-02-05T06:41:16.932829+0000","last_undegraded":"2024-02-05T06:41:16.932829+0000","last_fullsized":"2024-02-05T06:41:16.932829+0000","mapping_epoch":185,"log_start":"192'600","ondisk_log_start":"192'600","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":4662,"log_dups_size":0,"ondisk_log_size":4662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344850432,"num_objects":5262,"num_object_clones":0,"num_object_copies":15786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5262,"num_write_kb":336768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":560,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.028822+0000","last_change":"2024-02-05T06:40:30.085803+0000","last_active":"2024-02-05T06:41:10.028822+0000","last_peered":"2024-02-05T06:41:10.028822+0000","last_clean":"2024-02-05T06:41:10.028822+0000","last_became_active":"2024-02-05T06:40:30.085438+0000","last_became_peered":"2024-02-05T06:40:30.085438+0000","last_unstale":"2024-02-05T06:41:10.028822+0000","last_undegraded":"2024-02-05T06:41:10.028822+0000","last_fullsized":"2024-02-05T06:41:10.028822+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:39:30.440075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:39:30.440075+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":354,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.086903+0000","last_change":"2024-02-05T06:39:30.865329+0000","last_active":"2024-02-05T06:41:10.086903+0000","last_peered":"2024-02-05T06:41:10.086903+0000","last_clean":"2024-02-05T06:41:10.086903+0000","last_became_active":"2024-02-05T06:33:35.600485+0000","last_became_peered":"2024-02-05T06:33:35.600485+0000","last_unstale":"2024-02-05T06:41:10.086903+0000","last_undegraded":"2024-02-05T06:41:10.086903+0000","last_fullsized":"2024-02-05T06:41:10.086903+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:30.865200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:30.865200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":89,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.094766+0000","last_change":"2024-02-05T06:40:08.582459+0000","last_active":"2024-02-05T06:41:10.094766+0000","last_peered":"2024-02-05T06:41:10.094766+0000","last_clean":"2024-02-05T06:41:10.094766+0000","last_became_active":"2024-02-05T06:40:08.582079+0000","last_became_peered":"2024-02-05T06:40:08.582079+0000","last_unstale":"2024-02-05T06:41:10.094766+0000","last_undegraded":"2024-02-05T06:41:10.094766+0000","last_fullsized":"2024-02-05T06:41:10.094766+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:28.353956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:28.353956+0000","last_clean_scrub_stamp":"2024-02-05T06:39:28.353956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"73'10","reported_seq":438,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.090418+0000","last_change":"2024-02-05T06:40:08.667772+0000","last_active":"2024-02-05T06:41:10.090418+0000","last_peered":"2024-02-05T06:41:10.090418+0000","last_clean":"2024-02-05T06:41:10.090418+0000","last_became_active":"2024-02-05T06:40:08.595642+0000","last_became_peered":"2024-02-05T06:40:08.595642+0000","last_unstale":"2024-02-05T06:41:10.090418+0000","last_undegraded":"2024-02-05T06:41:10.090418+0000","last_fullsized":"2024-02-05T06:41:10.090418+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-02-05T06:39:25.306103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:25.306103+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:10.206080+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"197'9","reported_seq":470,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.087352+0000","last_change":"2024-02-05T06:40:08.656439+0000","last_active":"2024-02-05T06:41:10.087352+0000","last_peered":"2024-02-05T06:41:10.087352+0000","last_clean":"2024-02-05T06:41:10.087352+0000","last_became_active":"2024-02-05T06:40:08.600932+0000","last_became_peered":"2024-02-05T06:40:08.600932+0000","last_unstale":"2024-02-05T06:41:10.087352+0000","last_undegraded":"2024-02-05T06:41:10.087352+0000","last_fullsized":"2024-02-05T06:41:10.087352+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:36:32.174396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:32.174396+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"234'5014","reported_seq":12129,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:16.234519+0000","last_change":"2024-02-05T06:41:00.031561+0000","last_active":"2024-02-05T06:41:16.234519+0000","last_peered":"2024-02-05T06:41:16.234519+0000","last_clean":"2024-02-05T06:41:16.234519+0000","last_became_active":"2024-02-05T06:41:00.026128+0000","last_became_peered":"2024-02-05T06:41:00.026128+0000","last_unstale":"2024-02-05T06:41:16.234519+0000","last_undegraded":"2024-02-05T06:41:16.234519+0000","last_fullsized":"2024-02-05T06:41:16.234519+0000","mapping_epoch":229,"log_start":"192'500","ondisk_log_start":"192'500","created":185,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":4514,"log_dups_size":0,"ondisk_log_size":4514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328597504,"num_objects":5014,"num_object_clones":0,"num_object_copies":15042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5014,"num_write_kb":320896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3499,"num_bytes_recovered":229310464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":558,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.091930+0000","last_change":"2024-02-05T06:40:48.371703+0000","last_active":"2024-02-05T06:41:10.091930+0000","last_peered":"2024-02-05T06:41:10.091930+0000","last_clean":"2024-02-05T06:41:10.091930+0000","last_became_active":"2024-02-05T06:40:48.370713+0000","last_became_peered":"2024-02-05T06:40:48.370713+0000","last_unstale":"2024-02-05T06:41:10.091930+0000","last_undegraded":"2024-02-05T06:41:10.091930+0000","last_fullsized":"2024-02-05T06:41:10.091930+0000","mapping_epoch":222,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:38:10.176507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:38:10.176507+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":67,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.094430+0000","last_change":"2024-02-05T06:40:27.099496+0000","last_active":"2024-02-05T06:41:10.094430+0000","last_peered":"2024-02-05T06:41:10.094430+0000","last_clean":"2024-02-05T06:41:10.094430+0000","last_became_active":"2024-02-05T06:40:27.099168+0000","last_became_peered":"2024-02-05T06:40:27.099168+0000","last_unstale":"2024-02-05T06:41:10.094430+0000","last_undegraded":"2024-02-05T06:41:10.094430+0000","last_fullsized":"2024-02-05T06:41:10.094430+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:40:06.734454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:40:06.734454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:18.186414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"95'2","reported_seq":388,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.099281+0000","last_change":"2024-02-05T06:39:06.955815+0000","last_active":"2024-02-05T06:41:10.099281+0000","last_peered":"2024-02-05T06:41:10.099281+0000","last_clean":"2024-02-05T06:41:10.099281+0000","last_became_active":"2024-02-05T06:36:54.975093+0000","last_became_peered":"2024-02-05T06:36:54.975093+0000","last_unstale":"2024-02-05T06:41:10.099281+0000","last_undegraded":"2024-02-05T06:41:10.099281+0000","last_fullsized":"2024-02-05T06:41:10.099281+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-05T06:39:06.955629+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-05T06:35:25.616708+0000","last_clean_scrub_stamp":"2024-02-05T06:39:06.955629+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'14","reported_seq":289,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.088984+0000","last_change":"2024-02-05T06:40:47.627775+0000","last_active":"2024-02-05T06:41:10.088984+0000","last_peered":"2024-02-05T06:41:10.088984+0000","last_clean":"2024-02-05T06:41:10.088984+0000","last_became_active":"2024-02-05T06:40:08.590366+0000","last_became_peered":"2024-02-05T06:40:08.590366+0000","last_unstale":"2024-02-05T06:41:10.088984+0000","last_undegraded":"2024-02-05T06:41:10.088984+0000","last_fullsized":"2024-02-05T06:41:10.088984+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'14","last_scrub_stamp":"2024-02-05T06:40:47.627644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:40:47.627644+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:05.903986+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"119'10","reported_seq":481,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.095989+0000","last_change":"2024-02-05T06:38:23.177410+0000","last_active":"2024-02-05T06:41:10.095989+0000","last_peered":"2024-02-05T06:41:10.095989+0000","last_clean":"2024-02-05T06:41:10.095989+0000","last_became_active":"2024-02-05T06:36:54.954819+0000","last_became_peered":"2024-02-05T06:36:54.954819+0000","last_unstale":"2024-02-05T06:41:10.095989+0000","last_undegraded":"2024-02-05T06:41:10.095989+0000","last_fullsized":"2024-02-05T06:41:10.095989+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-02-05T06:38:23.177286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:38:23.177286+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"234'5182","reported_seq":10385,"reported_epoch":234,"state":"active+recovering+undersized+remapped","last_fresh":"2024-02-05T06:41:16.891553+0000","last_change":"2024-02-05T06:40:46.468070+0000","last_active":"2024-02-05T06:41:16.891553+0000","last_peered":"2024-02-05T06:41:16.891553+0000","last_clean":"2024-02-05T06:40:07.546191+0000","last_became_active":"2024-02-05T06:40:27.192989+0000","last_became_peered":"2024-02-05T06:40:27.192989+0000","last_unstale":"2024-02-05T06:41:16.891553+0000","last_undegraded":"2024-02-05T06:41:16.891553+0000","last_fullsized":"2024-02-05T06:40:27.049428+0000","mapping_epoch":205,"log_start":"192'500","ondisk_log_start":"192'500","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":4682,"log_dups_size":0,"ondisk_log_size":4682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339607552,"num_objects":5182,"num_object_clones":0,"num_object_copies":15546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1327,"num_objects_unfound":0,"num_objects_dirty":5182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5182,"num_write_kb":331648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2543,"num_bytes_recovered":166658048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3","objects":3855},{"shards":"0,3,4","objects":901},{"shards":"0,3,5","objects":426}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":520,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.089315+0000","last_change":"2024-02-05T06:38:10.076368+0000","last_active":"2024-02-05T06:41:10.089315+0000","last_peered":"2024-02-05T06:41:10.089315+0000","last_clean":"2024-02-05T06:41:10.089315+0000","last_became_active":"2024-02-05T06:37:37.135073+0000","last_became_peered":"2024-02-05T06:37:37.135073+0000","last_unstale":"2024-02-05T06:41:10.089315+0000","last_undegraded":"2024-02-05T06:41:10.089315+0000","last_fullsized":"2024-02-05T06:41:10.089315+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:38:10.076258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:38:10.076258+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":251,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.089190+0000","last_change":"2024-02-05T06:39:24.090774+0000","last_active":"2024-02-05T06:41:10.089190+0000","last_peered":"2024-02-05T06:41:10.089190+0000","last_clean":"2024-02-05T06:41:10.089190+0000","last_became_active":"2024-02-05T06:36:54.958644+0000","last_became_peered":"2024-02-05T06:36:54.958644+0000","last_unstale":"2024-02-05T06:41:10.089190+0000","last_undegraded":"2024-02-05T06:41:10.089190+0000","last_fullsized":"2024-02-05T06:41:10.089190+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:24.090670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:24.090670+0000","last_clean_scrub_stamp":"2024-02-05T06:39:24.090670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":383,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.095946+0000","last_change":"2024-02-05T06:40:27.094061+0000","last_active":"2024-02-05T06:41:10.095946+0000","last_peered":"2024-02-05T06:41:10.095946+0000","last_clean":"2024-02-05T06:41:10.095946+0000","last_became_active":"2024-02-05T06:40:27.093739+0000","last_became_peered":"2024-02-05T06:40:27.093739+0000","last_unstale":"2024-02-05T06:41:10.095946+0000","last_undegraded":"2024-02-05T06:41:10.095946+0000","last_fullsized":"2024-02-05T06:41:10.095946+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:38:28.150695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:38:28.150695+0000","last_clean_scrub_stamp":"2024-02-05T06:38:28.150695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"73'18","reported_seq":441,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.095854+0000","last_change":"2024-02-05T06:40:46.408120+0000","last_active":"2024-02-05T06:41:10.095854+0000","last_peered":"2024-02-05T06:41:10.095854+0000","last_clean":"2024-02-05T06:41:10.095854+0000","last_became_active":"2024-02-05T06:40:27.277747+0000","last_became_peered":"2024-02-05T06:40:27.277747+0000","last_unstale":"2024-02-05T06:41:10.095854+0000","last_undegraded":"2024-02-05T06:41:10.095854+0000","last_fullsized":"2024-02-05T06:41:10.095854+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-02-05T06:39:12.382954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:12.382954+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":271,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.089106+0000","last_change":"2024-02-05T06:40:27.074218+0000","last_active":"2024-02-05T06:41:10.089106+0000","last_peered":"2024-02-05T06:41:10.089106+0000","last_clean":"2024-02-05T06:41:10.089106+0000","last_became_active":"2024-02-05T06:40:27.073889+0000","last_became_peered":"2024-02-05T06:40:27.073889+0000","last_unstale":"2024-02-05T06:41:10.089106+0000","last_undegraded":"2024-02-05T06:41:10.089106+0000","last_fullsized":"2024-02-05T06:41:10.089106+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:40:02.899805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:40:02.899805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"234'5191","reported_seq":9163,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:16.443832+0000","last_change":"2024-02-05T06:40:37.438903+0000","last_active":"2024-02-05T06:41:16.443832+0000","last_peered":"2024-02-05T06:41:16.443832+0000","last_clean":"2024-02-05T06:41:16.443832+0000","last_became_active":"2024-02-05T06:40:37.405461+0000","last_became_peered":"2024-02-05T06:40:37.405461+0000","last_unstale":"2024-02-05T06:41:16.443832+0000","last_undegraded":"2024-02-05T06:41:16.443832+0000","last_fullsized":"2024-02-05T06:41:16.443832+0000","mapping_epoch":215,"log_start":"192'800","ondisk_log_start":"192'800","created":185,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":4391,"log_dups_size":0,"ondisk_log_size":4391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:18.214771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340197376,"num_objects":5191,"num_object_clones":0,"num_object_copies":15573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5191,"num_write_kb":332224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1925,"num_bytes_recovered":126156800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":390,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.095615+0000","last_change":"2024-02-05T06:40:11.044717+0000","last_active":"2024-02-05T06:41:10.095615+0000","last_peered":"2024-02-05T06:41:10.095615+0000","last_clean":"2024-02-05T06:41:10.095615+0000","last_became_active":"2024-02-05T06:40:11.042179+0000","last_became_peered":"2024-02-05T06:40:11.042179+0000","last_unstale":"2024-02-05T06:41:10.095615+0000","last_undegraded":"2024-02-05T06:41:10.095615+0000","last_fullsized":"2024-02-05T06:41:10.095615+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:40:03.329947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:40:03.329947+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:11.606576+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":26,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,4],"acting":[2,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":88,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.088796+0000","last_change":"2024-02-05T06:40:08.595889+0000","last_active":"2024-02-05T06:41:10.088796+0000","last_peered":"2024-02-05T06:41:10.088796+0000","last_clean":"2024-02-05T06:41:10.088796+0000","last_became_active":"2024-02-05T06:40:08.595517+0000","last_became_peered":"2024-02-05T06:40:08.595517+0000","last_unstale":"2024-02-05T06:41:10.088796+0000","last_undegraded":"2024-02-05T06:41:10.088796+0000","last_fullsized":"2024-02-05T06:41:10.088796+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:11.464973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:11.464973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:10.206194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"197'11","reported_seq":318,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.089910+0000","last_change":"2024-02-05T06:40:08.601333+0000","last_active":"2024-02-05T06:41:10.089910+0000","last_peered":"2024-02-05T06:41:10.089910+0000","last_clean":"2024-02-05T06:41:10.089910+0000","last_became_active":"2024-02-05T06:40:08.585067+0000","last_became_peered":"2024-02-05T06:40:08.585067+0000","last_unstale":"2024-02-05T06:41:10.089910+0000","last_undegraded":"2024-02-05T06:41:10.089910+0000","last_fullsized":"2024-02-05T06:41:10.089910+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-02-05T06:40:06.921141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:40:06.921141+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:17.472928+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":377,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.095771+0000","last_change":"2024-02-05T06:40:27.214932+0000","last_active":"2024-02-05T06:41:10.095771+0000","last_peered":"2024-02-05T06:41:10.095771+0000","last_clean":"2024-02-05T06:41:10.095771+0000","last_became_active":"2024-02-05T06:40:27.213439+0000","last_became_peered":"2024-02-05T06:40:27.213439+0000","last_unstale":"2024-02-05T06:41:10.095771+0000","last_undegraded":"2024-02-05T06:41:10.095771+0000","last_fullsized":"2024-02-05T06:41:10.095771+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:23.372627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:39:23.372627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"73'15","reported_seq":147,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.087650+0000","last_change":"2024-02-05T06:39:13.334762+0000","last_active":"2024-02-05T06:41:10.087650+0000","last_peered":"2024-02-05T06:41:10.087650+0000","last_clean":"2024-02-05T06:41:10.087650+0000","last_became_active":"2024-02-05T06:39:13.257978+0000","last_became_peered":"2024-02-05T06:39:13.257978+0000","last_unstale":"2024-02-05T06:41:10.087650+0000","last_undegraded":"2024-02-05T06:41:10.087650+0000","last_fullsized":"2024-02-05T06:41:10.087650+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-02-05T06:39:11.376185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:11.376185+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2","version":"234'5113","reported_seq":8953,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:16.212800+0000","last_change":"2024-02-05T06:40:34.130876+0000","last_active":"2024-02-05T06:41:16.212800+0000","last_peered":"2024-02-05T06:41:16.212800+0000","last_clean":"2024-02-05T06:41:16.212800+0000","last_became_active":"2024-02-05T06:40:34.127148+0000","last_became_peered":"2024-02-05T06:40:34.127148+0000","last_unstale":"2024-02-05T06:41:16.212800+0000","last_undegraded":"2024-02-05T06:41:16.212800+0000","last_fullsized":"2024-02-05T06:41:16.212800+0000","mapping_epoch":212,"log_start":"192'900","ondisk_log_start":"192'900","created":185,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":4213,"log_dups_size":0,"ondisk_log_size":4213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:17.472895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335085568,"num_objects":5113,"num_object_clones":0,"num_object_copies":15339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5113,"num_write_kb":327232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1861,"num_bytes_recovered":121962496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":365,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.092076+0000","last_change":"2024-02-05T06:40:27.078125+0000","last_active":"2024-02-05T06:41:10.092076+0000","last_peered":"2024-02-05T06:41:10.092076+0000","last_clean":"2024-02-05T06:41:10.092076+0000","last_became_active":"2024-02-05T06:40:27.077570+0000","last_became_peered":"2024-02-05T06:40:27.077570+0000","last_unstale":"2024-02-05T06:41:10.092076+0000","last_undegraded":"2024-02-05T06:41:10.092076+0000","last_fullsized":"2024-02-05T06:41:10.092076+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:18.402553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:20.042534+0000","last_clean_scrub_stamp":"2024-02-05T06:39:18.402553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":215,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.089643+0000","last_change":"2024-02-05T06:40:08.583193+0000","last_active":"2024-02-05T06:41:10.089643+0000","last_peered":"2024-02-05T06:41:10.089643+0000","last_clean":"2024-02-05T06:41:10.089643+0000","last_became_active":"2024-02-05T06:40:08.582647+0000","last_became_peered":"2024-02-05T06:40:08.582647+0000","last_unstale":"2024-02-05T06:41:10.089643+0000","last_undegraded":"2024-02-05T06:41:10.089643+0000","last_fullsized":"2024-02-05T06:41:10.089643+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:18.379775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:18.379775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":85,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.086782+0000","last_change":"2024-02-05T06:40:27.087535+0000","last_active":"2024-02-05T06:41:10.086782+0000","last_peered":"2024-02-05T06:41:10.086782+0000","last_clean":"2024-02-05T06:41:10.086782+0000","last_became_active":"2024-02-05T06:40:27.086808+0000","last_became_peered":"2024-02-05T06:40:27.086808+0000","last_unstale":"2024-02-05T06:41:10.086782+0000","last_undegraded":"2024-02-05T06:41:10.086782+0000","last_fullsized":"2024-02-05T06:41:10.086782+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:38:14.268605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:38:14.268605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:10.205900+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"115'16","reported_seq":254,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.089767+0000","last_change":"2024-02-05T06:36:54.992244+0000","last_active":"2024-02-05T06:41:10.089767+0000","last_peered":"2024-02-05T06:41:10.089767+0000","last_clean":"2024-02-05T06:41:10.089767+0000","last_became_active":"2024-02-05T06:36:54.958582+0000","last_became_peered":"2024-02-05T06:36:54.958582+0000","last_unstale":"2024-02-05T06:41:10.089767+0000","last_undegraded":"2024-02-05T06:41:10.089767+0000","last_fullsized":"2024-02-05T06:41:10.089767+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'16","last_scrub_stamp":"2024-02-05T06:36:34.793243+0000","last_deep_scrub":"115'16","last_deep_scrub_stamp":"2024-02-05T06:36:34.793243+0000","last_clean_scrub_stamp":"2024-02-05T06:36:34.793243+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"234'5097","reported_seq":5205,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:16.458387+0000","last_change":"2024-02-05T06:39:42.682964+0000","last_active":"2024-02-05T06:41:16.458387+0000","last_peered":"2024-02-05T06:41:16.458387+0000","last_clean":"2024-02-05T06:41:16.458387+0000","last_became_active":"2024-02-05T06:39:42.682661+0000","last_became_peered":"2024-02-05T06:39:42.682661+0000","last_unstale":"2024-02-05T06:41:16.458387+0000","last_undegraded":"2024-02-05T06:41:16.458387+0000","last_fullsized":"2024-02-05T06:41:16.458387+0000","mapping_epoch":185,"log_start":"192'700","ondisk_log_start":"192'700","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":4397,"log_dups_size":0,"ondisk_log_size":4397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:18.214826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334036992,"num_objects":5097,"num_object_clones":0,"num_object_copies":15291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5097,"num_write_kb":326208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"83'128","reported_seq":394,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.093693+0000","last_change":"2024-02-05T06:40:11.077092+0000","last_active":"2024-02-05T06:41:10.093693+0000","last_peered":"2024-02-05T06:41:10.093693+0000","last_clean":"2024-02-05T06:41:10.093693+0000","last_became_active":"2024-02-05T06:40:11.067992+0000","last_became_peered":"2024-02-05T06:40:11.067992+0000","last_unstale":"2024-02-05T06:41:10.093693+0000","last_undegraded":"2024-02-05T06:41:10.093693+0000","last_fullsized":"2024-02-05T06:41:10.093693+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:37:39.307601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:37:39.307601+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:11.606611+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":423697,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":371,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.094553+0000","last_change":"2024-02-05T06:38:17.558520+0000","last_active":"2024-02-05T06:41:10.094553+0000","last_peered":"2024-02-05T06:41:10.094553+0000","last_clean":"2024-02-05T06:41:10.094553+0000","last_became_active":"2024-02-05T06:36:54.958805+0000","last_became_peered":"2024-02-05T06:36:54.958805+0000","last_unstale":"2024-02-05T06:41:10.094553+0000","last_undegraded":"2024-02-05T06:41:10.094553+0000","last_fullsized":"2024-02-05T06:41:10.094553+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:38:17.558441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:38:17.558441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:12.649532+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":377,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.089864+0000","last_change":"2024-02-05T06:39:09.427061+0000","last_active":"2024-02-05T06:41:10.089864+0000","last_peered":"2024-02-05T06:41:10.089864+0000","last_clean":"2024-02-05T06:41:10.089864+0000","last_became_active":"2024-02-05T06:36:54.958082+0000","last_became_peered":"2024-02-05T06:36:54.958082+0000","last_unstale":"2024-02-05T06:41:10.089864+0000","last_undegraded":"2024-02-05T06:41:10.089864+0000","last_fullsized":"2024-02-05T06:41:10.089864+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:09.426862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:09.426862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:10.206135+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":390,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.028347+0000","last_change":"2024-02-05T06:39:35.429964+0000","last_active":"2024-02-05T06:41:10.028347+0000","last_peered":"2024-02-05T06:41:10.028347+0000","last_clean":"2024-02-05T06:41:10.028347+0000","last_became_active":"2024-02-05T06:39:13.240038+0000","last_became_peered":"2024-02-05T06:39:13.240038+0000","last_unstale":"2024-02-05T06:41:10.028347+0000","last_undegraded":"2024-02-05T06:41:10.028347+0000","last_fullsized":"2024-02-05T06:41:10.028347+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:35.429861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:39:35.429861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"95'25","reported_seq":268,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.089895+0000","last_change":"2024-02-05T06:39:31.037992+0000","last_active":"2024-02-05T06:41:10.089895+0000","last_peered":"2024-02-05T06:41:10.089895+0000","last_clean":"2024-02-05T06:41:10.089895+0000","last_became_active":"2024-02-05T06:36:54.961719+0000","last_became_peered":"2024-02-05T06:36:54.961719+0000","last_unstale":"2024-02-05T06:41:10.089895+0000","last_undegraded":"2024-02-05T06:41:10.089895+0000","last_fullsized":"2024-02-05T06:41:10.089895+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'25","last_scrub_stamp":"2024-02-05T06:39:31.037880+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-02-05T06:35:26.274944+0000","last_clean_scrub_stamp":"2024-02-05T06:39:31.037880+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"234'5125","reported_seq":8963,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:16.864902+0000","last_change":"2024-02-05T06:40:44.347862+0000","last_active":"2024-02-05T06:41:16.864902+0000","last_peered":"2024-02-05T06:41:16.864902+0000","last_clean":"2024-02-05T06:41:16.864902+0000","last_became_active":"2024-02-05T06:40:44.342540+0000","last_became_peered":"2024-02-05T06:40:44.342540+0000","last_unstale":"2024-02-05T06:41:16.864902+0000","last_undegraded":"2024-02-05T06:41:16.864902+0000","last_fullsized":"2024-02-05T06:41:16.864902+0000","mapping_epoch":218,"log_start":"193'1100","ondisk_log_start":"193'1100","created":185,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":4025,"log_dups_size":0,"ondisk_log_size":4025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335872000,"num_objects":5125,"num_object_clones":0,"num_object_copies":15375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5125,"num_write_kb":328000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1861,"num_bytes_recovered":121962496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":477,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.093484+0000","last_change":"2024-02-05T06:40:30.095555+0000","last_active":"2024-02-05T06:41:10.093484+0000","last_peered":"2024-02-05T06:41:10.093484+0000","last_clean":"2024-02-05T06:41:10.093484+0000","last_became_active":"2024-02-05T06:40:30.095202+0000","last_became_peered":"2024-02-05T06:40:30.095202+0000","last_unstale":"2024-02-05T06:41:10.093484+0000","last_undegraded":"2024-02-05T06:41:10.093484+0000","last_fullsized":"2024-02-05T06:41:10.093484+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:38:20.178985+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:38:20.178985+0000","last_clean_scrub_stamp":"2024-02-05T06:38:20.178985+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:13.670069+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":250,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.089682+0000","last_change":"2024-02-05T06:39:30.057231+0000","last_active":"2024-02-05T06:41:10.089682+0000","last_peered":"2024-02-05T06:41:10.089682+0000","last_clean":"2024-02-05T06:41:10.089682+0000","last_became_active":"2024-02-05T06:36:54.961560+0000","last_became_peered":"2024-02-05T06:36:54.961560+0000","last_unstale":"2024-02-05T06:41:10.089682+0000","last_undegraded":"2024-02-05T06:41:10.089682+0000","last_fullsized":"2024-02-05T06:41:10.089682+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:30.057099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:30.057099+0000","last_clean_scrub_stamp":"2024-02-05T06:39:30.057099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"208'12","reported_seq":496,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.094035+0000","last_change":"2024-02-05T06:40:27.207229+0000","last_active":"2024-02-05T06:41:10.094035+0000","last_peered":"2024-02-05T06:41:10.094035+0000","last_clean":"2024-02-05T06:41:10.094035+0000","last_became_active":"2024-02-05T06:40:27.102928+0000","last_became_peered":"2024-02-05T06:40:27.102928+0000","last_unstale":"2024-02-05T06:41:10.094035+0000","last_undegraded":"2024-02-05T06:41:10.094035+0000","last_fullsized":"2024-02-05T06:41:10.094035+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'9","last_scrub_stamp":"2024-02-05T06:37:47.417250+0000","last_deep_scrub":"92'9","last_deep_scrub_stamp":"2024-02-05T06:34:52.283780+0000","last_clean_scrub_stamp":"2024-02-05T06:37:47.417250+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:41:19.536894+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":377,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.090466+0000","last_change":"2024-02-05T06:36:54.971857+0000","last_active":"2024-02-05T06:41:10.090466+0000","last_peered":"2024-02-05T06:41:10.090466+0000","last_clean":"2024-02-05T06:41:10.090466+0000","last_became_active":"2024-02-05T06:36:54.971304+0000","last_became_peered":"2024-02-05T06:36:54.971304+0000","last_unstale":"2024-02-05T06:41:10.090466+0000","last_undegraded":"2024-02-05T06:41:10.090466+0000","last_fullsized":"2024-02-05T06:41:10.090466+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:37.506830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:37.506830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"93'15","reported_seq":262,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-02-05T06:41:10.089492+0000","last_change":"2024-02-05T06:39:16.997591+0000","last_active":"2024-02-05T06:41:10.089492+0000","last_peered":"2024-02-05T06:41:10.089492+0000","last_clean":"2024-02-05T06:41:10.089492+0000","last_became_active":"2024-02-05T06:36:54.963620+0000","last_became_peered":"2024-02-05T06:36:54.963620+0000","last_unstale":"2024-02-05T06:41:10.089492+0000","last_undegraded":"2024-02-05T06:41:10.089492+0000","last_fullsized":"2024-02-05T06:41:10.089492+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'15","last_scrub_stamp":"2024-02-05T06:39:16.997494+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-02-05T06:35:04.584657+0000","last_clean_scrub_stamp":"2024-02-05T06:39:16.997494+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":5366284288,"num_objects":81883,"num_object_clones":0,"num_object_copies":245649,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3771,"num_objects_unfound":0,"num_objects_dirty":81883,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":81883,"num_write_kb":5240512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23816,"num_bytes_recovered":1560805376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16941056000,"data_stored":16941056000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69683,"ondisk_log_size":69683,"up":48,"acting":45,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":51,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":542359,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":108,"num_write":65,"num_write_kb":877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1889765,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":2742,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":5242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":684,"num_read_kb":441,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":339,"num_bytes_recovered":7354,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":483328,"data_stored":12784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":455,"ondisk_log_size":455,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103265,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1564488,"kb_used_data":1362560,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":92180664,"statfs":{"total":95995035648,"available":94392999936,"internally_reserved":0,"allocated":1395261440,"data_stored":1394359000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":206764490},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":2.2149999999999999,"15min":1.115},"min":{"1min":0.24399999999999999,"5min":0.219,"15min":0.219},"max":{"1min":1.6379999999999999,"5min":68.921000000000006,"15min":68.921000000000006},"last":3.2709999999999999},{"interface":"front","average":{"1min":1.1830000000000001,"5min":2.423,"15min":1.208},"min":{"1min":0.28799999999999998,"5min":0.248,"15min":0.248},"max":{"1min":3.3010000000000002,"5min":69.616,"15min":69.616},"last":2.4700000000000002}]},{"osd":1,"last update":"Mon Feb 5 06:40:39 2024","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":1.931,"15min":1.018},"min":{"1min":0.32400000000000001,"5min":0.254,"15min":0.23599999999999999},"max":{"1min":9.4939999999999998,"5min":89.231999999999999,"15min":89.231999999999999},"last":0.377},{"interface":"front","average":{"1min":2.7810000000000001,"5min":2.2149999999999999,"15min":1.137},"min":{"1min":0.33000000000000002,"5min":0.26300000000000001,"15min":0.26100000000000001},"max":{"1min":14.824,"5min":87.995999999999995,"15min":87.995999999999995},"last":2.012}]},{"osd":2,"last update":"Mon Feb 5 06:40:39 2024","interfaces":[{"interface":"back","average":{"1min":1.8460000000000001,"5min":1.675,"15min":0.93999999999999995},"min":{"1min":0.28399999999999997,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":11.991,"5min":53.947000000000003,"15min":53.947000000000003},"last":2.129},{"interface":"front","average":{"1min":0.999,"5min":1.6120000000000001,"15min":0.91700000000000004},"min":{"1min":0.30499999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.8149999999999999,"5min":53.299999999999997,"15min":53.299999999999997},"last":2.2949999999999999}]},{"osd":3,"last update":"Mon Feb 5 06:40:39 2024","interfaces":[{"interface":"back","average":{"1min":3.2349999999999999,"5min":3.2130000000000001,"15min":1.47},"min":{"1min":0.36799999999999999,"5min":0.316,"15min":0.316},"max":{"1min":19.513000000000002,"5min":89.481999999999999,"15min":89.481999999999999},"last":0.35399999999999998},{"interface":"front","average":{"1min":1.405,"5min":3.077,"15min":1.429},"min":{"1min":0.307,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":12.034000000000001,"5min":92.293999999999997,"15min":92.293999999999997},"last":0.45500000000000002}]},{"osd":4,"last update":"Mon Feb 5 06:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.53700000000000003,"15min":0.47499999999999998},"min":{"1min":0.26400000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78300000000000003,"5min":3.5409999999999999,"15min":3.5409999999999999},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.48299999999999998,"15min":0.49399999999999999},"min":{"1min":0.29099999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.73599999999999999,"5min":2.258,"15min":2.258},"last":0.39000000000000001}]},{"osd":5,"last update":"Mon Feb 5 06:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.52200000000000002,"15min":0.51100000000000001},"min":{"1min":0.312,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.3220000000000001,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.59199999999999997,"15min":0.56200000000000006},"min":{"1min":0.35299999999999998,"5min":0.252,"15min":0.252},"max":{"1min":2.0529999999999999,"5min":3.5990000000000002,"15min":3.5990000000000002},"last":0.56399999999999995}]},{"osd":6,"last update":"Mon Feb 5 06:40:39 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66000000000000003,"15min":0.57499999999999996},"min":{"1min":0.35999999999999999,"5min":0.314,"15min":0.314},"max":{"1min":1.52,"5min":3.5750000000000002,"15min":3.5750000000000002},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57599999999999996,"15min":0.55400000000000005},"min":{"1min":0.35999999999999999,"5min":0.245,"15min":0.245},"max":{"1min":1.2310000000000001,"5min":3.2189999999999999,"15min":3.2189999999999999},"last":0.52900000000000003}]}]},{"osd":6,"up_from":49,"seq":210453397607,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2501296,"kb_used_data":2245224,"kb_used_omap":1,"kb_used_meta":256062,"kb_avail":91243856,"statfs":{"total":95995035648,"available":93433708544,"internally_reserved":0,"allocated":2299109376,"data_stored":2298230400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":262207942},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:41:07 2024","interfaces":[{"interface":"back","average":{"1min":6.5650000000000004,"5min":2.5419999999999998,"15min":1.23},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":74.075000000000003,"5min":74.075000000000003,"15min":74.075000000000003},"last":1.7649999999999999},{"interface":"front","average":{"1min":6.4880000000000004,"5min":2.3719999999999999,"15min":1.1539999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":74.186000000000007,"5min":74.186000000000007,"15min":74.186000000000007},"last":0.29099999999999998}]},{"osd":1,"last update":"Mon Feb 5 06:41:07 2024","interfaces":[{"interface":"back","average":{"1min":4.6859999999999999,"5min":1.8340000000000001,"15min":0.96199999999999997},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":72.951999999999998,"5min":72.951999999999998,"15min":72.951999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":4.4880000000000004,"5min":1.599,"15min":0.91600000000000004},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.255},"max":{"1min":73.352999999999994,"5min":73.352999999999994,"15min":73.352999999999994},"last":0.82599999999999996}]},{"osd":2,"last update":"Mon Feb 5 06:41:07 2024","interfaces":[{"interface":"back","average":{"1min":8.7539999999999996,"5min":2.722,"15min":1.2909999999999999},"min":{"1min":0.31900000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":85.617000000000004,"5min":85.617000000000004,"15min":85.617000000000004},"last":0.505},{"interface":"front","average":{"1min":8.9700000000000006,"5min":2.6419999999999999,"15min":1.2509999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":85.335999999999999,"5min":85.335999999999999,"15min":85.335999999999999},"last":0.36699999999999999}]},{"osd":3,"last update":"Mon Feb 5 06:41:07 2024","interfaces":[{"interface":"back","average":{"1min":6.1180000000000003,"5min":3.3279999999999998,"15min":1.4830000000000001},"min":{"1min":0.308,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":72.921999999999997,"5min":99.269000000000005,"15min":99.269000000000005},"last":2.504},{"interface":"front","average":{"1min":6.5819999999999999,"5min":3.6139999999999999,"15min":1.609},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":73.302000000000007,"5min":101.75700000000001,"15min":101.75700000000001},"last":0.35299999999999998}]},{"osd":4,"last update":"Mon Feb 5 06:41:07 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48699999999999999,"15min":0.45900000000000002},"min":{"1min":0.26800000000000002,"5min":0.26700000000000002,"15min":0.23100000000000001},"max":{"1min":1.153,"5min":1.8220000000000001,"15min":1.8220000000000001},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.47099999999999997,"15min":0.44600000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.613,"5min":1.613,"15min":1.613},"last":0.38200000000000001}]},{"osd":5,"last update":"Mon Feb 5 06:41:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.51200000000000001,"15min":0.47599999999999998},"min":{"1min":0.29899999999999999,"5min":0.29199999999999998,"15min":0.251},"max":{"1min":1.1930000000000001,"5min":1.798,"15min":1.798},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.502,"15min":0.47299999999999998},"min":{"1min":0.31900000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.42599999999999999}]},{"osd":7,"last update":"Mon Feb 5 06:40:25 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.54700000000000004,"15min":0.53100000000000003},"min":{"1min":0.34100000000000003,"5min":0.309,"15min":0.309},"max":{"1min":0.92900000000000005,"5min":6.0350000000000001,"15min":6.0350000000000001},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.66300000000000003,"15min":0.58999999999999997},"min":{"1min":0.39200000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.90300000000000002,"5min":8.5399999999999991,"15min":8.5399999999999991},"last":0.437}]}]},{"osd":1,"up_from":13,"seq":55834574976,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2637416,"kb_used_data":2357728,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":91107736,"statfs":{"total":95995035648,"available":93294321664,"internally_reserved":0,"allocated":2414313472,"data_stored":2413379047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":286390725},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:41:01 2024","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.7549999999999999,"15min":1.0840000000000001},"min":{"1min":0.183,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":4.5330000000000004,"5min":10.457000000000001,"15min":67.421000000000006},"last":3.7749999999999999},{"interface":"front","average":{"1min":2.6800000000000002,"5min":2.1099999999999999,"15min":1.1919999999999999},"min":{"1min":0.34200000000000003,"5min":0.20999999999999999,"15min":0.19600000000000001},"max":{"1min":6.6280000000000001,"5min":7.2830000000000004,"15min":67.885000000000005},"last":7.6529999999999996}]},{"osd":2,"last update":"Mon Feb 5 06:40:22 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":1.726,"15min":0.83799999999999997},"min":{"1min":0.27100000000000002,"5min":0.215,"15min":0.214},"max":{"1min":3.1000000000000001,"5min":9.7929999999999993,"15min":9.7929999999999993},"last":5.1200000000000001},{"interface":"front","average":{"1min":0.94799999999999995,"5min":2.1469999999999998,"15min":0.96899999999999997},"min":{"1min":0.249,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":4.5890000000000004,"5min":11.859,"15min":11.859},"last":1.306}]},{"osd":3,"last update":"Mon Feb 5 06:40:56 2024","interfaces":[{"interface":"back","average":{"1min":4.1500000000000004,"5min":3.6400000000000001,"15min":1.4950000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.25600000000000001},"max":{"1min":14.16,"5min":69.798000000000002,"15min":69.798000000000002},"last":7.7110000000000003},{"interface":"front","average":{"1min":1.627,"5min":2.5920000000000001,"15min":1.1990000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.5640000000000001,"5min":70.765000000000001,"15min":70.765000000000001},"last":3.7970000000000002}]},{"osd":4,"last update":"Mon Feb 5 06:40:16 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":1.4510000000000001,"15min":0.85899999999999999},"min":{"1min":0.33600000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":4.4420000000000002,"5min":11.679,"15min":11.679},"last":0.35399999999999998},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.3919999999999999,"15min":0.85299999999999998},"min":{"1min":0.44900000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.0730000000000004,"5min":6.0730000000000004,"15min":6.0730000000000004},"last":0.51400000000000001}]},{"osd":5,"last update":"Mon Feb 5 06:40:44 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.1439999999999999,"15min":1},"min":{"1min":0.34599999999999997,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":3.4399999999999999,"5min":6.0469999999999997,"15min":8.2509999999999994},"last":3.7450000000000001},{"interface":"front","average":{"1min":1.081,"5min":1.6220000000000001,"15min":0.95899999999999996},"min":{"1min":0.42799999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":4.5590000000000002,"5min":13.455,"15min":13.455},"last":0.47399999999999998}]},{"osd":6,"last update":"Mon Feb 5 06:41:01 2024","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.73,"15min":1.024},"min":{"1min":0.45700000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":4.4909999999999997,"5min":13.391,"15min":13.391},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.2370000000000001,"15min":0.87},"min":{"1min":0.39500000000000002,"5min":0.379,"15min":0.379},"max":{"1min":3.2930000000000001,"5min":8.6099999999999994,"15min":8.6099999999999994},"last":3.0489999999999999}]},{"osd":7,"last update":"Mon Feb 5 06:40:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":1.663,"15min":0.94799999999999995},"min":{"1min":0.46000000000000002,"5min":0.37,"15min":0.37},"max":{"1min":4.5300000000000002,"5min":13.49,"15min":13.49},"last":0.41299999999999998},{"interface":"front","average":{"1min":1.6910000000000001,"5min":1.423,"15min":1.141},"min":{"1min":0.41599999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":6.1870000000000003,"5min":6.1870000000000003,"15min":8.2219999999999995},"last":0.55000000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738502,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2583256,"kb_used_data":2304976,"kb_used_omap":1,"kb_used_meta":278270,"kb_avail":91161896,"statfs":{"total":95995035648,"available":93349781504,"internally_reserved":0,"allocated":2360295424,"data_stored":2359377249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":284948933},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2,1,7],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 06:41:04 2024","interfaces":[{"interface":"back","average":{"1min":2.7999999999999998,"5min":2.165,"15min":0.97199999999999998},"min":{"1min":0.249,"5min":0.23200000000000001,"15min":0.188},"max":{"1min":18.59,"5min":18.59,"15min":18.59},"last":1.0580000000000001},{"interface":"front","average":{"1min":3.1230000000000002,"5min":2.0880000000000001,"15min":0.94399999999999995},"min":{"1min":0.30499999999999999,"5min":0.23999999999999999,"15min":0.20499999999999999},"max":{"1min":18.300999999999998,"5min":18.300999999999998,"15min":18.300999999999998},"last":4.5060000000000002}]},{"osd":2,"last update":"Mon Feb 5 06:40:22 2024","interfaces":[{"interface":"back","average":{"1min":1.9019999999999999,"5min":2.0939999999999999,"15min":0.97799999999999998},"min":{"1min":0.27100000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":16.981999999999999,"5min":16.981999999999999,"15min":16.981999999999999},"last":3.8660000000000001},{"interface":"front","average":{"1min":1.6879999999999999,"5min":1.9059999999999999,"15min":0.92300000000000004},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.28000000000000003},"max":{"1min":16.998999999999999,"5min":16.998999999999999,"15min":16.998999999999999},"last":3.7890000000000001}]},{"osd":3,"last update":"Mon Feb 5 06:40:49 2024","interfaces":[{"interface":"back","average":{"1min":2.3980000000000001,"5min":2.286,"15min":1.0700000000000001},"min":{"1min":0.32400000000000001,"5min":0.29799999999999999,"15min":0.29099999999999998},"max":{"1min":16.923999999999999,"5min":16.923999999999999,"15min":16.923999999999999},"last":0.94299999999999995},{"interface":"front","average":{"1min":3.202,"5min":2.5750000000000002,"15min":1.1679999999999999},"min":{"1min":0.26000000000000001,"5min":0.254,"15min":0.254},"max":{"1min":18.259,"5min":18.259,"15min":18.259},"last":3.8199999999999998}]},{"osd":4,"last update":"Mon Feb 5 06:40:16 2024","interfaces":[{"interface":"back","average":{"1min":1.76,"5min":1.577,"15min":0.92300000000000004},"min":{"1min":0.40300000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":17.241,"5min":17.241,"15min":17.241},"last":1.988},{"interface":"front","average":{"1min":1.6839999999999999,"5min":1.528,"15min":0.92900000000000005},"min":{"1min":0.503,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":18.495999999999999,"5min":18.495999999999999,"15min":18.495999999999999},"last":0.75600000000000001}]},{"osd":5,"last update":"Mon Feb 5 06:40:39 2024","interfaces":[{"interface":"back","average":{"1min":2.0449999999999999,"5min":1.508,"15min":0.92100000000000004},"min":{"1min":0.39400000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":18.170999999999999,"5min":18.170999999999999,"15min":18.170999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":1.9370000000000001,"5min":1.569,"15min":0.95099999999999996},"min":{"1min":0.40000000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":17.152999999999999,"5min":17.152999999999999,"15min":17.152999999999999},"last":2.0670000000000002}]},{"osd":6,"last update":"Mon Feb 5 06:41:04 2024","interfaces":[{"interface":"back","average":{"1min":2.2050000000000001,"5min":1.6519999999999999,"15min":1.004},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":18.562999999999999,"5min":18.562999999999999,"15min":18.562999999999999},"last":1.1080000000000001},{"interface":"front","average":{"1min":2.0310000000000001,"5min":1.5249999999999999,"15min":0.95399999999999996},"min":{"1min":0.438,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":18.213999999999999,"5min":18.213999999999999,"15min":18.213999999999999},"last":0.56599999999999995}]},{"osd":7,"last update":"Mon Feb 5 06:40:31 2024","interfaces":[{"interface":"back","average":{"1min":1.827,"5min":1.5980000000000001,"15min":0.93100000000000005},"min":{"1min":0.48799999999999999,"5min":0.37,"15min":0.37},"max":{"1min":17.210000000000001,"5min":17.210000000000001,"15min":17.210000000000001},"last":2.036},{"interface":"front","average":{"1min":1.7589999999999999,"5min":1.6040000000000001,"15min":1.1459999999999999},"min":{"1min":0.47899999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":18.533999999999999,"5min":18.533999999999999,"15min":18.533999999999999},"last":1.268}]}]},{"osd":2,"up_from":18,"seq":77309411451,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2810740,"kb_used_data":2540780,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":90934412,"statfs":{"total":95995035648,"available":93116837888,"internally_reserved":0,"allocated":2601758720,"data_stored":2600876843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:40:24 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":3.0630000000000002,"15min":1.2689999999999999},"min":{"1min":0.215,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.282,"5min":50.814,"15min":50.814},"last":2.8359999999999999},{"interface":"front","average":{"1min":1.887,"5min":2.9489999999999998,"15min":1.2450000000000001},"min":{"1min":0.21199999999999999,"5min":0.192,"15min":0.188},"max":{"1min":10.026,"5min":50.802999999999997,"15min":50.802999999999997},"last":1.345}]},{"osd":1,"last update":"Mon Feb 5 06:40:24 2024","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":2.4180000000000001,"15min":1.0669999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":6.7430000000000003,"5min":31.303000000000001,"15min":31.303000000000001},"last":2.9350000000000001},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.853,"15min":0.90900000000000003},"min":{"1min":0.28299999999999997,"5min":0.22,"15min":0.22},"max":{"1min":4.4770000000000003,"5min":31.425000000000001,"15min":31.425000000000001},"last":0.60799999999999998}]},{"osd":3,"last update":"Mon Feb 5 06:40:54 2024","interfaces":[{"interface":"back","average":{"1min":2.6709999999999998,"5min":3.3050000000000002,"15min":1.4199999999999999},"min":{"1min":0.28299999999999997,"5min":0.254,"15min":0.254},"max":{"1min":11.409000000000001,"5min":53.127000000000002,"15min":53.127000000000002},"last":0.27300000000000002},{"interface":"front","average":{"1min":3.1150000000000002,"5min":3.1629999999999998,"15min":1.387},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.22500000000000001},"max":{"1min":11.388999999999999,"5min":52.607999999999997,"15min":52.607999999999997},"last":1.2529999999999999}]},{"osd":4,"last update":"Mon Feb 5 06:41:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.2130000000000001,"15min":0.83999999999999997},"min":{"1min":0.29699999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":4.2270000000000003,"5min":6.4180000000000001,"15min":6.4180000000000001},"last":0.34000000000000002},{"interface":"front","average":{"1min":1.103,"5min":1.137,"15min":0.82199999999999995},"min":{"1min":0.36199999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":4.798,"5min":6.8780000000000001,"15min":6.8780000000000001},"last":1.1799999999999999}]},{"osd":5,"last update":"Mon Feb 5 06:40:41 2024","interfaces":[{"interface":"back","average":{"1min":7.0350000000000001,"5min":2.323,"15min":1.2330000000000001},"min":{"1min":0.48299999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":86.614999999999995,"5min":86.614999999999995,"15min":86.614999999999995},"last":1.2190000000000001},{"interface":"front","average":{"1min":7.2270000000000003,"5min":2.738,"15min":1.359},"min":{"1min":0.40500000000000003,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":86.543999999999997,"5min":86.543999999999997,"15min":86.543999999999997},"last":0.56999999999999995}]},{"osd":6,"last update":"Mon Feb 5 06:41:11 2024","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.2,"15min":1.4119999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":4.9489999999999998,"5min":6.9139999999999997,"15min":17.143999999999998},"last":1.9810000000000001},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.7210000000000001,"15min":1.0469999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":6.7919999999999998,"5min":10.27,"15min":10.27},"last":2.9079999999999999}]},{"osd":7,"last update":"Mon Feb 5 06:40:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.6659999999999999,"15min":0.98899999999999999},"min":{"1min":0.46100000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":6.8609999999999998,"5min":10.206,"15min":10.206},"last":2.871},{"interface":"front","average":{"1min":1.123,"5min":1.4079999999999999,"15min":0.91600000000000004},"min":{"1min":0.41399999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":4.6929999999999996,"5min":6.4749999999999996,"15min":6.4749999999999996},"last":0.77900000000000003}]}]},{"osd":3,"up_from":26,"seq":111669149814,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3262412,"kb_used_data":2956356,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":90482740,"statfs":{"total":95995035648,"available":92654325760,"internally_reserved":0,"allocated":3027308544,"data_stored":3026401923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":313391559},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,4,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:40:59 2024","interfaces":[{"interface":"back","average":{"1min":2.5129999999999999,"5min":1.964,"15min":0.92500000000000004},"min":{"1min":0.32700000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":7.2690000000000001,"5min":13.391999999999999,"15min":13.391999999999999},"last":1.7809999999999999},{"interface":"front","average":{"1min":1.9199999999999999,"5min":2.3740000000000001,"15min":1.097},"min":{"1min":0.23100000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":7.0659999999999998,"5min":25.087,"15min":25.087},"last":1.3089999999999999}]},{"osd":1,"last update":"Mon Feb 5 06:40:59 2024","interfaces":[{"interface":"back","average":{"1min":2.4209999999999998,"5min":2.008,"15min":0.98499999999999999},"min":{"1min":0.34999999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":5.8979999999999997,"5min":9.6010000000000009,"15min":9.6010000000000009},"last":1.587},{"interface":"front","average":{"1min":2.4449999999999998,"5min":1.9370000000000001,"15min":0.94999999999999996},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":6.9619999999999997,"5min":8.3040000000000003,"15min":8.3040000000000003},"last":3.4460000000000002}]},{"osd":2,"last update":"Mon Feb 5 06:40:59 2024","interfaces":[{"interface":"back","average":{"1min":2.3500000000000001,"5min":2.3730000000000002,"15min":1.133},"min":{"1min":0.38400000000000001,"5min":0.29899999999999999,"15min":0.27300000000000002},"max":{"1min":7.2880000000000003,"5min":41.529000000000003,"15min":41.529000000000003},"last":6.8099999999999996},{"interface":"front","average":{"1min":2.085,"5min":2.2949999999999999,"15min":1.073},"min":{"1min":0.33600000000000002,"5min":0.254,"15min":0.248},"max":{"1min":9.6769999999999996,"5min":41.759999999999998,"15min":41.759999999999998},"last":1.958}]},{"osd":4,"last update":"Mon Feb 5 06:40:16 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":1.2150000000000001,"15min":0.80600000000000005},"min":{"1min":0.39800000000000002,"5min":0.307,"15min":0.307},"max":{"1min":3.3090000000000002,"5min":16.577999999999999,"15min":16.577999999999999},"last":1.8799999999999999},{"interface":"front","average":{"1min":0.89800000000000002,"5min":1.518,"15min":0.90300000000000002},"min":{"1min":0.314,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.0190000000000001,"5min":8.4060000000000006,"15min":8.4060000000000006},"last":1.2529999999999999}]},{"osd":5,"last update":"Mon Feb 5 06:40:40 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.7330000000000001,"15min":0.97799999999999998},"min":{"1min":0.441,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":5.2249999999999996,"5min":13.672000000000001,"15min":13.672000000000001},"last":3.6179999999999999},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.375,"15min":0.871},"min":{"1min":0.45100000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":6.2569999999999997,"5min":15.792999999999999,"15min":15.792999999999999},"last":1.9179999999999999}]},{"osd":6,"last update":"Mon Feb 5 06:41:10 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.603,"15min":0.96099999999999997},"min":{"1min":0.34899999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":4.5439999999999996,"5min":8.3610000000000007,"15min":8.3610000000000007},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.6799999999999999,"5min":1.5600000000000001,"15min":0.97799999999999998},"min":{"1min":0.41699999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":5.1829999999999998,"5min":6.601,"15min":6.601},"last":3.552}]},{"osd":7,"last update":"Mon Feb 5 06:40:32 2024","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.732,"15min":0.96799999999999997},"min":{"1min":0.45000000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.7469999999999999,"5min":45.039000000000001,"15min":45.039000000000001},"last":6.6429999999999998},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.95,"15min":1.0609999999999999},"min":{"1min":0.46700000000000003,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":5.258,"5min":44.036999999999999,"15min":44.036999999999999},"last":3.5880000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986288,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2386952,"kb_used_data":2076736,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":91358200,"statfs":{"total":95995035648,"available":93550796800,"internally_reserved":0,"allocated":2126577664,"data_stored":2125674558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":317651397},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:40:14 2024","interfaces":[{"interface":"back","average":{"1min":2.4569999999999999,"5min":1.7250000000000001,"15min":0.96399999999999997},"min":{"1min":0.23699999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":33.372,"5min":33.372,"15min":33.372},"last":0.88600000000000001},{"interface":"front","average":{"1min":2.1869999999999998,"5min":2.008,"15min":1.0329999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":33.323999999999998,"5min":33.323999999999998,"15min":33.323999999999998},"last":0.94399999999999995}]},{"osd":1,"last update":"Mon Feb 5 06:40:14 2024","interfaces":[{"interface":"back","average":{"1min":2.4420000000000002,"5min":1.163,"15min":0.76100000000000001},"min":{"1min":0.25600000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":33.747999999999998,"5min":33.747999999999998,"15min":33.747999999999998},"last":1.7350000000000001},{"interface":"front","average":{"1min":2.3940000000000001,"5min":1.3280000000000001,"15min":0.83899999999999997},"min":{"1min":0.28299999999999997,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":33.691000000000003,"5min":33.691000000000003,"15min":33.691000000000003},"last":0.84999999999999998}]},{"osd":2,"last update":"Mon Feb 5 06:41:14 2024","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.82599999999999996,"15min":0.66400000000000003},"min":{"1min":0.29399999999999998,"5min":0.245,"15min":0.245},"max":{"1min":2.3260000000000001,"5min":4.1239999999999997,"15min":4.1239999999999997},"last":0.92300000000000004},{"interface":"front","average":{"1min":1.018,"5min":1.0780000000000001,"15min":0.749},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.766,"5min":5.7030000000000003,"15min":5.7030000000000003},"last":0.65400000000000003}]},{"osd":3,"last update":"Mon Feb 5 06:40:15 2024","interfaces":[{"interface":"back","average":{"1min":2.7210000000000001,"5min":3.6070000000000002,"15min":1.601},"min":{"1min":0.37,"5min":0.31,"15min":0.31},"max":{"1min":41.512,"5min":99.546999999999997,"15min":99.546999999999997},"last":0.89900000000000002},{"interface":"front","average":{"1min":2.4820000000000002,"5min":3.6200000000000001,"15min":1.595},"min":{"1min":0.33400000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":38.527000000000001,"5min":99.245000000000005,"15min":99.245000000000005},"last":1.6970000000000001}]},{"osd":5,"last update":"Mon Feb 5 06:40:42 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.63,"15min":0.48499999999999999},"min":{"1min":0.28699999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1180000000000001,"5min":11.098000000000001,"15min":11.098000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.61399999999999999,"15min":0.5},"min":{"1min":0.25700000000000001,"5min":0.252,"15min":0.252},"max":{"1min":1.145,"5min":10.965999999999999,"15min":10.965999999999999},"last":0.68300000000000005}]},{"osd":6,"last update":"Mon Feb 5 06:41:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.68600000000000005,"15min":0.56000000000000005},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":2.9630000000000001,"5min":11.868,"15min":11.868},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.73799999999999999,"15min":0.56799999999999995},"min":{"1min":0.29899999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":3.0350000000000001,"5min":11.787000000000001,"15min":11.787000000000001},"last":0.70099999999999996}]},{"osd":7,"last update":"Mon Feb 5 06:40:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.54900000000000004,"15min":0.52500000000000002},"min":{"1min":0.29399999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":2.2719999999999998},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.56599999999999995,"15min":0.54700000000000004},"min":{"1min":0.29199999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.70299999999999996,"5min":2.9470000000000001,"15min":2.9470000000000001},"last":0.97499999999999998}]}]},{"osd":5,"up_from":0,"seq":171798691947,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2367881216,"data_stored":2367881216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2420572160,"data_stored":2420572160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2625896448,"data_stored":2625896448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3032547328,"data_stored":3032547328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2142961664,"data_stored":2142961664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":623575040,"data_stored":623575040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2316763136,"data_stored":2316763136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1410859008,"data_stored":1410859008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T06:41:21.840 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd pool set unique_pool_1 pgp_num 16 2024-02-05T06:41:21.932 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:21.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:21.992 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:21.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:22.093 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:22.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:22.123 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:22.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:22.224 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:22.239 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 87 15 86305 86290 61.9779 60.9375 0.015899 0.0161284 2024-02-05T06:41:22.253 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:22.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:22.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:22.354 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:22.382 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:22.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:22.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:22.484 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:22.537 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:22.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:22.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:22.639 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:22.653 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:41:22 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:41:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:41:22.669 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:22.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:22.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:22.771 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:22.825 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:22.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:22.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:22.927 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:22.966 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:22.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:22.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:23.043 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:41:22 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:41:23.067 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:23.088 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:41:23.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:23.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:23.220 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:41:23.223 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:23.239 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 88 16 87268 87252 61.9567 60.125 0.0203045 0.016136 2024-02-05T06:41:23.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:23.256 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:23.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:23.262 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:41:23.299 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:41:23.335 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:41:23.356 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:23.378 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:41:23.389 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:23.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:23.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:23.393 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-02-05T06:41:23.428 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:41:23.471 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:41:23.491 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:23.512 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:41:23.513 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:23 smithi028 ceph-mon[29442]: pgmap v373: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.05k op/s; 2788/257748 objects misplaced (1.082%); 14 MiB/s, 221 objects/s recovering 2024-02-05T06:41:23.513 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:23 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2606109102' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:41:23.513 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:23 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:41:23.513 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:23 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3481900541' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:41:23.513 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:23 smithi028 ceph-mon[37984]: pgmap v373: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.05k op/s; 2788/257748 objects misplaced (1.082%); 14 MiB/s, 221 objects/s recovering 2024-02-05T06:41:23.513 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:23 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2606109102' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:41:23.513 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:23 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:41:23.513 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:23 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3481900541' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:41:23.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:23.525 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:23.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:23.527 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:41:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:41:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:41:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:41:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:41:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:41:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:41:23.528 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:41:23.539 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:41:23.539 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:41:23.626 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:23.655 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:23.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:23.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:23.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:23 smithi167 ceph-mon[32774]: pgmap v373: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.05k op/s; 2788/257748 objects misplaced (1.082%); 14 MiB/s, 221 objects/s recovering 2024-02-05T06:41:23.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:23 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2606109102' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:41:23.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:23 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:41:23.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:23 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3481900541' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:41:23.758 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:23.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:23.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:23.910 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:23.940 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:23.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:23.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:24.042 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:24.073 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:24.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:24.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:24.174 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:24.205 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:24.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:24.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:24.237 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-05T06:41:24.239 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 89 16 88301 88285 61.9859 64.5625 0.0252473 0.0161274 2024-02-05T06:41:24.307 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:24.345 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:24.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:24.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:24.447 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:24.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:24 smithi167 ceph-mon[32774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:41:24.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:24 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2606109102' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:41:24.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:24 smithi167 ceph-mon[32774]: osdmap e237: 8 total, 8 up, 7 in 2024-02-05T06:41:24.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:24.493 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:24.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:24.594 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:24.601 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:41:24.636 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:24.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:24.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:24.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:24 smithi028 ceph-mon[29442]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:41:24.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:24 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2606109102' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:41:24.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:24 smithi028 ceph-mon[29442]: osdmap e237: 8 total, 8 up, 7 in 2024-02-05T06:41:24.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:24 smithi028 ceph-mon[37984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:41:24.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:24 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2606109102' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:41:24.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:24 smithi028 ceph-mon[37984]: osdmap e237: 8 total, 8 up, 7 in 2024-02-05T06:41:24.738 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:24.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:24.772 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:24.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:24.873 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:24.905 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:24.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:24.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:25.007 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:25.052 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:25.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:25.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:25.155 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:25.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:25.197 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:25.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:25.239 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Total time run: 90.0111 2024-02-05T06:41:25.239 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Total writes made: 89272 2024-02-05T06:41:25.239 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Write size: 65536 2024-02-05T06:41:25.240 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Object size: 65536 2024-02-05T06:41:25.240 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Bandwidth (MB/sec): 61.9868 2024-02-05T06:41:25.240 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Stddev Bandwidth: 41.1139 2024-02-05T06:41:25.240 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Max bandwidth (MB/sec): 210.312 2024-02-05T06:41:25.240 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Min bandwidth (MB/sec): 0 2024-02-05T06:41:25.240 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Average IOPS: 991 2024-02-05T06:41:25.240 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Stddev IOPS: 657.83 2024-02-05T06:41:25.240 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Max IOPS: 3365 2024-02-05T06:41:25.240 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Min IOPS: 0 2024-02-05T06:41:25.240 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Average Latency(s): 0.0161275 2024-02-05T06:41:25.241 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Stddev Latency(s): 0.0511308 2024-02-05T06:41:25.241 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Max latency(s): 2.2409 2024-02-05T06:41:25.241 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Min latency(s): 0.00125405 2024-02-05T06:41:25.244 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Cleaning up (deleting benchmark objects) 2024-02-05T06:41:25.298 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:25.326 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:25.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:25.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:25.428 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:25.451 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:25.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:25.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:25.553 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:25.584 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:25.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:25.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:25.685 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:25.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:25 smithi028 ceph-mon[37984]: pgmap v375: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 40 MiB/s wr, 639 op/s; 2788/257748 objects misplaced (1.082%); 10 MiB/s, 163 objects/s recovering 2024-02-05T06:41:25.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:25 smithi028 ceph-mon[37984]: 6.1a scrub starts 2024-02-05T06:41:25.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:25 smithi028 ceph-mon[37984]: 6.1a scrub ok 2024-02-05T06:41:25.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:25 smithi028 ceph-mon[37984]: osdmap e238: 8 total, 8 up, 7 in 2024-02-05T06:41:25.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:25 smithi028 ceph-mon[29442]: pgmap v375: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 40 MiB/s wr, 639 op/s; 2788/257748 objects misplaced (1.082%); 10 MiB/s, 163 objects/s recovering 2024-02-05T06:41:25.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:25 smithi028 ceph-mon[29442]: 6.1a scrub starts 2024-02-05T06:41:25.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:25 smithi028 ceph-mon[29442]: 6.1a scrub ok 2024-02-05T06:41:25.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:25 smithi028 ceph-mon[29442]: osdmap e238: 8 total, 8 up, 7 in 2024-02-05T06:41:25.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:25.713 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:25.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:25.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:25 smithi167 ceph-mon[32774]: pgmap v375: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 40 MiB/s wr, 639 op/s; 2788/257748 objects misplaced (1.082%); 10 MiB/s, 163 objects/s recovering 2024-02-05T06:41:25.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:25 smithi167 ceph-mon[32774]: 6.1a scrub starts 2024-02-05T06:41:25.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:25 smithi167 ceph-mon[32774]: 6.1a scrub ok 2024-02-05T06:41:25.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:25 smithi167 ceph-mon[32774]: osdmap e238: 8 total, 8 up, 7 in 2024-02-05T06:41:25.814 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:25.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:25.845 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:25.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:25.946 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:25.975 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:25.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:25.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:26.077 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:26.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:26.110 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:26.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:26.212 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:26.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:26.235 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:26.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:26.336 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:26.373 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:26.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:26.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:26.475 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:26.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:26.499 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:26.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:26.500 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:26 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:41:26.600 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:26.641 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:26.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:26.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:26.650 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:41:26.650 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:41:26.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:26 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:41:26.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:26 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:41:26.743 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:26.783 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:26.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:26.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:26.885 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:26.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:26.932 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:26.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:27.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:27.071 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:27.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:27.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:27.173 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:27.205 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:27.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:27.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:27.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:27.352 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:27.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:27.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:27.370 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-05T06:41:27.453 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:27.495 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:27.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:27.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:27.597 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:27.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:27.632 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:27.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:27.696 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:41:27.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:27 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:41:27.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:27 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:41:27.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:27 smithi028 ceph-mon[29442]: pgmap v377: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.8 KiB/s rd, 53 MiB/s wr, 895 op/s; 2788/260571 objects misplaced (1.070%); 10 MiB/s, 163 objects/s recovering 2024-02-05T06:41:27.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:27 smithi028 ceph-mon[29442]: 4.e scrub starts 2024-02-05T06:41:27.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:27 smithi028 ceph-mon[29442]: 4.e scrub ok 2024-02-05T06:41:27.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:27 smithi028 ceph-mon[29442]: osdmap e239: 8 total, 8 up, 7 in 2024-02-05T06:41:27.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:27 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:41:27.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:27 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:41:27.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:27 smithi028 ceph-mon[37984]: pgmap v377: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.8 KiB/s rd, 53 MiB/s wr, 895 op/s; 2788/260571 objects misplaced (1.070%); 10 MiB/s, 163 objects/s recovering 2024-02-05T06:41:27.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:27 smithi028 ceph-mon[37984]: 4.e scrub starts 2024-02-05T06:41:27.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:27 smithi028 ceph-mon[37984]: 4.e scrub ok 2024-02-05T06:41:27.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:27 smithi028 ceph-mon[37984]: osdmap e239: 8 total, 8 up, 7 in 2024-02-05T06:41:27.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:27 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:41:27.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:27 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:41:27.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:27 smithi167 ceph-mon[32774]: pgmap v377: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 5.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.8 KiB/s rd, 53 MiB/s wr, 895 op/s; 2788/260571 objects misplaced (1.070%); 10 MiB/s, 163 objects/s recovering 2024-02-05T06:41:27.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:27 smithi167 ceph-mon[32774]: 4.e scrub starts 2024-02-05T06:41:27.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:27 smithi167 ceph-mon[32774]: 4.e scrub ok 2024-02-05T06:41:27.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:27 smithi167 ceph-mon[32774]: osdmap e239: 8 total, 8 up, 7 in 2024-02-05T06:41:27.732 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:27.792 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:27.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:27.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:27.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:27.934 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:27.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:27.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:28.036 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:28.066 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:28.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:28.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:28.168 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:28.201 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:28.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:28.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:28.302 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:28.332 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:28.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:28.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:28.434 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:28.465 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:28.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:28.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:28.567 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:28.596 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:28.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:28.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:28.698 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:28.805 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:28 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:41:28.805 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:28 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:41:28.805 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:28 smithi167 ceph-mon[32774]: 5.7 scrub starts 2024-02-05T06:41:28.805 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:28 smithi167 ceph-mon[32774]: 5.7 scrub ok 2024-02-05T06:41:28.805 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:28 smithi167 ceph-mon[32774]: 4.13 scrub starts 2024-02-05T06:41:28.805 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:28 smithi167 ceph-mon[32774]: 4.13 scrub ok 2024-02-05T06:41:28.805 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:28 smithi167 ceph-mon[32774]: osdmap e240: 8 total, 8 up, 7 in 2024-02-05T06:41:28.806 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:28 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:41:28.806 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:28 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:41:28.806 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:28 smithi028 ceph-mon[37984]: 5.7 scrub starts 2024-02-05T06:41:28.806 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:28 smithi028 ceph-mon[37984]: 5.7 scrub ok 2024-02-05T06:41:28.806 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:28 smithi028 ceph-mon[37984]: 4.13 scrub starts 2024-02-05T06:41:28.806 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:28 smithi028 ceph-mon[37984]: 4.13 scrub ok 2024-02-05T06:41:28.807 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:28 smithi028 ceph-mon[37984]: osdmap e240: 8 total, 8 up, 7 in 2024-02-05T06:41:28.807 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:28 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:41:28.807 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:28 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:41:28.807 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:28 smithi028 ceph-mon[29442]: 5.7 scrub starts 2024-02-05T06:41:28.807 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:28 smithi028 ceph-mon[29442]: 5.7 scrub ok 2024-02-05T06:41:28.807 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:28 smithi028 ceph-mon[29442]: 4.13 scrub starts 2024-02-05T06:41:28.807 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:28 smithi028 ceph-mon[29442]: 4.13 scrub ok 2024-02-05T06:41:28.807 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:28 smithi028 ceph-mon[29442]: osdmap e240: 8 total, 8 up, 7 in 2024-02-05T06:41:28.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:28.836 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:28.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:28.937 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:28.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:28.981 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:28.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:29.082 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:29.122 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:29.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:29.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:29.223 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:29.269 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:29.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:29.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:29.371 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:29.400 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:29.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:29.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:29.502 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:29.513 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:29 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:41:29.513 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:29 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:41:29.514 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:29 smithi167 ceph-mon[32774]: pgmap v380: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 45 MiB/s wr, 989 op/s; 2252/264483 objects misplaced (0.851%); 6.8 MiB/s, 108 objects/s recovering 2024-02-05T06:41:29.514 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:29 smithi167 ceph-mon[32774]: Health check failed: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-02-05T06:41:29.514 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:29 smithi167 ceph-mon[32774]: 1.a scrub starts 2024-02-05T06:41:29.514 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:29 smithi167 ceph-mon[32774]: 1.a scrub ok 2024-02-05T06:41:29.514 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:29 smithi167 ceph-mon[32774]: 1.5 scrub starts 2024-02-05T06:41:29.514 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:29 smithi167 ceph-mon[32774]: osdmap e241: 8 total, 8 up, 7 in 2024-02-05T06:41:29.514 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:29 smithi167 ceph-mon[32774]: 1.5 scrub ok 2024-02-05T06:41:29.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:29.536 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:29.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:29.574 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:41:29.574 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:41:29.637 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:29.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:29.664 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:29.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:29.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:29.805 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:29.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:29.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:29.906 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:29.943 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:29.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:29.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:29.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:29 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:41:29.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:29 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:41:29.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:29 smithi028 ceph-mon[37984]: pgmap v380: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 45 MiB/s wr, 989 op/s; 2252/264483 objects misplaced (0.851%); 6.8 MiB/s, 108 objects/s recovering 2024-02-05T06:41:29.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:29 smithi028 ceph-mon[37984]: Health check failed: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-02-05T06:41:29.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:29 smithi028 ceph-mon[37984]: 1.a scrub starts 2024-02-05T06:41:29.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:29 smithi028 ceph-mon[37984]: 1.a scrub ok 2024-02-05T06:41:29.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:29 smithi028 ceph-mon[37984]: 1.5 scrub starts 2024-02-05T06:41:29.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:29 smithi028 ceph-mon[37984]: osdmap e241: 8 total, 8 up, 7 in 2024-02-05T06:41:29.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:29 smithi028 ceph-mon[37984]: 1.5 scrub ok 2024-02-05T06:41:29.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:29 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:41:29.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:29 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:41:29.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:29 smithi028 ceph-mon[29442]: pgmap v380: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 45 MiB/s wr, 989 op/s; 2252/264483 objects misplaced (0.851%); 6.8 MiB/s, 108 objects/s recovering 2024-02-05T06:41:29.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:29 smithi028 ceph-mon[29442]: Health check failed: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-02-05T06:41:29.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:29 smithi028 ceph-mon[29442]: 1.a scrub starts 2024-02-05T06:41:29.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:29 smithi028 ceph-mon[29442]: 1.a scrub ok 2024-02-05T06:41:29.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:29 smithi028 ceph-mon[29442]: 1.5 scrub starts 2024-02-05T06:41:29.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:29 smithi028 ceph-mon[29442]: osdmap e241: 8 total, 8 up, 7 in 2024-02-05T06:41:29.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:29 smithi028 ceph-mon[29442]: 1.5 scrub ok 2024-02-05T06:41:30.045 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:30.053 INFO:teuthology.orchestra.run.smithi028.stderr:set pool 8 pgp_num to 16 2024-02-05T06:41:30.089 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:30.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:30.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:30.177 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T06:41:30.191 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:30.218 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:30.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:30.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:30.320 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:30.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:30.349 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:30.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:30.450 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:30.458 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:30 smithi167 ceph-mon[32774]: 1.8 scrub starts 2024-02-05T06:41:30.459 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:30 smithi167 ceph-mon[32774]: 1.8 scrub ok 2024-02-05T06:41:30.459 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:30 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3939730046' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-05T06:41:30.459 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:30 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3939730046' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-02-05T06:41:30.459 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:30 smithi167 ceph-mon[32774]: osdmap e242: 8 total, 8 up, 7 in 2024-02-05T06:41:30.478 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:30.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:30.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:30.580 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:30.629 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:30.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:30.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:30.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:30 smithi028 ceph-mon[29442]: 1.8 scrub starts 2024-02-05T06:41:30.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:30 smithi028 ceph-mon[29442]: 1.8 scrub ok 2024-02-05T06:41:30.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:30 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3939730046' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-05T06:41:30.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:30 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3939730046' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-02-05T06:41:30.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:30 smithi028 ceph-mon[29442]: osdmap e242: 8 total, 8 up, 7 in 2024-02-05T06:41:30.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:30 smithi028 ceph-mon[37984]: 1.8 scrub starts 2024-02-05T06:41:30.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:30 smithi028 ceph-mon[37984]: 1.8 scrub ok 2024-02-05T06:41:30.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:30 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3939730046' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-02-05T06:41:30.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:30 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3939730046' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-02-05T06:41:30.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:30 smithi028 ceph-mon[37984]: osdmap e242: 8 total, 8 up, 7 in 2024-02-05T06:41:30.732 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:30.776 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:30.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:30.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:30.809 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:41:30.878 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:30.902 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:30.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:30.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:31.004 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:31.020 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:41:31.036 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:31.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:31.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:31.056 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-05T06:41:31.139 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:31.173 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:31.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:31.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:31.275 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:31.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:31.323 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:31.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:31.414 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:41:31.424 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:31.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:31.451 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:31.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:31.552 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:31.592 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:31.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:31.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:31.694 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:31.718 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:31.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:31.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:31.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:31 smithi167 ceph-mon[32774]: 6.19 scrub starts 2024-02-05T06:41:31.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:31 smithi167 ceph-mon[32774]: 6.19 scrub ok 2024-02-05T06:41:31.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:31 smithi167 ceph-mon[32774]: pgmap v383: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 36 MiB/s wr, 833 op/s; 2252/264483 objects misplaced (0.851%); 8.4 MiB/s, 133 objects/s recovering 2024-02-05T06:41:31.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:31 smithi167 ceph-mon[32774]: 2.1 scrub starts 2024-02-05T06:41:31.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:31 smithi167 ceph-mon[32774]: 2.1 scrub ok 2024-02-05T06:41:31.820 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:31.854 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:31.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:31.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:31.874 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:31 smithi028 ceph-mon[37984]: 6.19 scrub starts 2024-02-05T06:41:31.874 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:31 smithi028 ceph-mon[37984]: 6.19 scrub ok 2024-02-05T06:41:31.875 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:31 smithi028 ceph-mon[37984]: pgmap v383: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 36 MiB/s wr, 833 op/s; 2252/264483 objects misplaced (0.851%); 8.4 MiB/s, 133 objects/s recovering 2024-02-05T06:41:31.875 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:31 smithi028 ceph-mon[37984]: 2.1 scrub starts 2024-02-05T06:41:31.875 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:31 smithi028 ceph-mon[37984]: 2.1 scrub ok 2024-02-05T06:41:31.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:31 smithi028 ceph-mon[29442]: 6.19 scrub starts 2024-02-05T06:41:31.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:31 smithi028 ceph-mon[29442]: 6.19 scrub ok 2024-02-05T06:41:31.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:31 smithi028 ceph-mon[29442]: pgmap v383: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 5.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 36 MiB/s wr, 833 op/s; 2252/264483 objects misplaced (0.851%); 8.4 MiB/s, 133 objects/s recovering 2024-02-05T06:41:31.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:31 smithi028 ceph-mon[29442]: 2.1 scrub starts 2024-02-05T06:41:31.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:31 smithi028 ceph-mon[29442]: 2.1 scrub ok 2024-02-05T06:41:31.956 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:32.001 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:32.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:32.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:32.103 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:32.135 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:32.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:32.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:32.237 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:32.266 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:32.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:32.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:32.368 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:32.395 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:32.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:32.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:32.496 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:41:32.497 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:32.555 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:32.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:32.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:32.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:32.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:32.691 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:32.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:32.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:32 smithi167 ceph-mon[32774]: 5.d scrub starts 2024-02-05T06:41:32.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:32 smithi167 ceph-mon[32774]: 5.d scrub ok 2024-02-05T06:41:32.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:32 smithi167 ceph-mon[32774]: 4.1a scrub starts 2024-02-05T06:41:32.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:32 smithi167 ceph-mon[32774]: 4.1a scrub ok 2024-02-05T06:41:32.791 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:32.818 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:32.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:32.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:32.881 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:32 smithi028 ceph-mon[29442]: 5.d scrub starts 2024-02-05T06:41:32.881 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:32 smithi028 ceph-mon[29442]: 5.d scrub ok 2024-02-05T06:41:32.881 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:32 smithi028 ceph-mon[29442]: 4.1a scrub starts 2024-02-05T06:41:32.881 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:32 smithi028 ceph-mon[29442]: 4.1a scrub ok 2024-02-05T06:41:32.881 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:41:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:41:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:41:32.881 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:32 smithi028 ceph-mon[37984]: 5.d scrub starts 2024-02-05T06:41:32.882 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:32 smithi028 ceph-mon[37984]: 5.d scrub ok 2024-02-05T06:41:32.882 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:32 smithi028 ceph-mon[37984]: 4.1a scrub starts 2024-02-05T06:41:32.882 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:32 smithi028 ceph-mon[37984]: 4.1a scrub ok 2024-02-05T06:41:32.919 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:32.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:32.946 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:32.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:33.047 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:33.072 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:33.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:33.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:33.115 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:41:32 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:41:33.173 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:33.200 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:33.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:33.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:33.249 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:41:33.250 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:41:33.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:33.326 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:33.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:33.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:33.428 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:33.452 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:33.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:33.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:33.554 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:33.594 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:33.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:33.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:33.670 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:33 smithi028 ceph-mon[37984]: pgmap v384: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 25 MiB/s wr, 1.48k op/s; 1516/248367 objects misplaced (0.610%); 13 MiB/s, 216 objects/s recovering 2024-02-05T06:41:33.670 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:33 smithi028 ceph-mon[37984]: osdmap e243: 8 total, 8 up, 7 in 2024-02-05T06:41:33.670 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:33 smithi028 ceph-mon[37984]: 5.b scrub starts 2024-02-05T06:41:33.670 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:33 smithi028 ceph-mon[37984]: 5.b scrub ok 2024-02-05T06:41:33.670 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:33 smithi028 ceph-mon[37984]: 3.1d scrub starts 2024-02-05T06:41:33.670 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:33 smithi028 ceph-mon[37984]: 3.1d scrub ok 2024-02-05T06:41:33.670 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:33 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:41:33.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:33 smithi028 ceph-mon[29442]: pgmap v384: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 25 MiB/s wr, 1.48k op/s; 1516/248367 objects misplaced (0.610%); 13 MiB/s, 216 objects/s recovering 2024-02-05T06:41:33.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:33 smithi028 ceph-mon[29442]: osdmap e243: 8 total, 8 up, 7 in 2024-02-05T06:41:33.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:33 smithi028 ceph-mon[29442]: 5.b scrub starts 2024-02-05T06:41:33.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:33 smithi028 ceph-mon[29442]: 5.b scrub ok 2024-02-05T06:41:33.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:33 smithi028 ceph-mon[29442]: 3.1d scrub starts 2024-02-05T06:41:33.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:33 smithi028 ceph-mon[29442]: 3.1d scrub ok 2024-02-05T06:41:33.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:33 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:41:33.696 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:33.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:33 smithi167 ceph-mon[32774]: pgmap v384: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 25 MiB/s wr, 1.48k op/s; 1516/248367 objects misplaced (0.610%); 13 MiB/s, 216 objects/s recovering 2024-02-05T06:41:33.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:33 smithi167 ceph-mon[32774]: osdmap e243: 8 total, 8 up, 7 in 2024-02-05T06:41:33.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:33 smithi167 ceph-mon[32774]: 5.b scrub starts 2024-02-05T06:41:33.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:33 smithi167 ceph-mon[32774]: 5.b scrub ok 2024-02-05T06:41:33.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:33 smithi167 ceph-mon[32774]: 3.1d scrub starts 2024-02-05T06:41:33.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:33 smithi167 ceph-mon[32774]: 3.1d scrub ok 2024-02-05T06:41:33.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:33 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:41:33.735 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:33.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:33.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:33.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:33.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:33.881 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:33.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:33.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:33.994 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-05T06:41:34.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:34.024 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:34.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:34.125 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:34.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:34.164 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:34.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:34.265 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:34.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:34.301 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:34.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:34.343 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:41:34.402 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:34.434 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:34.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:34.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:34.536 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:34.565 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:34.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:34.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:34.667 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:34.711 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:34.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:34.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:34.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:34 smithi167 ceph-mon[32774]: 3.9 scrub starts 2024-02-05T06:41:34.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:34 smithi167 ceph-mon[32774]: 3.9 scrub ok 2024-02-05T06:41:34.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:34 smithi167 ceph-mon[32774]: osdmap e244: 8 total, 8 up, 7 in 2024-02-05T06:41:34.801 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:34 smithi028 ceph-mon[29442]: 3.9 scrub starts 2024-02-05T06:41:34.802 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:34 smithi028 ceph-mon[29442]: 3.9 scrub ok 2024-02-05T06:41:34.802 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:34 smithi028 ceph-mon[29442]: osdmap e244: 8 total, 8 up, 7 in 2024-02-05T06:41:34.802 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:34 smithi028 ceph-mon[37984]: 3.9 scrub starts 2024-02-05T06:41:34.802 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:34 smithi028 ceph-mon[37984]: 3.9 scrub ok 2024-02-05T06:41:34.802 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:34 smithi028 ceph-mon[37984]: osdmap e244: 8 total, 8 up, 7 in 2024-02-05T06:41:34.812 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:34.845 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:34.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:34.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:34.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:34.989 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:34.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:34.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:35.091 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:35.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:35.132 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:35.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:35.233 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:35.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:35.277 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:35.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:35.378 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:35.425 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:35.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:35.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:35.527 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:35.554 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:35.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:35.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:35.656 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:35.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:35 smithi028 ceph-mon[37984]: 6.1d scrub starts 2024-02-05T06:41:35.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:35 smithi028 ceph-mon[37984]: 6.1d scrub ok 2024-02-05T06:41:35.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:35 smithi028 ceph-mon[37984]: pgmap v387: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 913 op/s; 1516/248367 objects misplaced (0.610%); 7.8 MiB/s, 125 objects/s recovering 2024-02-05T06:41:35.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:35 smithi028 ceph-mon[37984]: 6.f scrub starts 2024-02-05T06:41:35.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:35 smithi028 ceph-mon[37984]: 6.f scrub ok 2024-02-05T06:41:35.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:35 smithi028 ceph-mon[29442]: 6.1d scrub starts 2024-02-05T06:41:35.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:35 smithi028 ceph-mon[29442]: 6.1d scrub ok 2024-02-05T06:41:35.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:35 smithi028 ceph-mon[29442]: pgmap v387: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 913 op/s; 1516/248367 objects misplaced (0.610%); 7.8 MiB/s, 125 objects/s recovering 2024-02-05T06:41:35.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:35 smithi028 ceph-mon[29442]: 6.f scrub starts 2024-02-05T06:41:35.689 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:35 smithi028 ceph-mon[29442]: 6.f scrub ok 2024-02-05T06:41:35.690 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:35.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:35.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:35.744 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:41:35.789 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:41:35.792 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:35.819 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:35.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:35.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:35.837 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:41:35.856 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:35 smithi167 ceph-mon[32774]: 6.1d scrub starts 2024-02-05T06:41:35.856 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:35 smithi167 ceph-mon[32774]: 6.1d scrub ok 2024-02-05T06:41:35.856 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:35 smithi167 ceph-mon[32774]: pgmap v387: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 5.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 0 B/s wr, 913 op/s; 1516/248367 objects misplaced (0.610%); 7.8 MiB/s, 125 objects/s recovering 2024-02-05T06:41:35.856 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:35 smithi167 ceph-mon[32774]: 6.f scrub starts 2024-02-05T06:41:35.856 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:35 smithi167 ceph-mon[32774]: 6.f scrub ok 2024-02-05T06:41:35.878 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:41:35.898 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 4, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T06:41:35.898 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T06:41:35.898 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-02-05T06:41:35.898 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd in 5 2024-02-05T06:41:35.918 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:41:35.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:35.960 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:41:35.962 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:35.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:35.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:36.006 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:41:36.045 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:41:36.052 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:41:36.052 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:41:36.052 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:41:36.052 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:41:36.052 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:41:36.052 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:41:36.052 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:41:36.053 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:41:36.064 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:36.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:36.100 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:36.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:36.201 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:36.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:36.254 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:36.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:36.355 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:36.385 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:36.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:36.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:36.487 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:36.519 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-02-05T06:41:36.532 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:36.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:36.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:36.634 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:36.661 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:36.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:36.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:36.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:36 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/891607057' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:41:36.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:36 smithi028 ceph-mon[37984]: from='client.25135 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:41:36.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:36 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3171995429' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:41:36.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:36 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/891607057' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:41:36.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:36 smithi028 ceph-mon[29442]: from='client.25135 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:41:36.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:36 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3171995429' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:41:36.763 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:36.795 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:36.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:36.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:36.813 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:36 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/891607057' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:41:36.813 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:36 smithi167 ceph-mon[32774]: from='client.25135 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:41:36.813 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:36 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3171995429' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:41:36.873 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:41:36.873 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-02-05T06:41:36.873 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-02-05T06:41:36.873 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-02-05T06:41:36.873 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:41:36.897 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:36.939 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:36.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:36.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:37.041 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:37.067 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:37.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:37.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:37.169 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:37.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:37.214 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:37.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:37.315 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:37.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:37.345 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:37.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:37.446 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:37.473 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:37.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:37.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:37.563 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-05T06:41:37.571 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:41:37.575 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:37.636 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:37.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:37.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:37.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:37 smithi028 ceph-mon[37984]: pgmap v388: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 1.08k op/s; 1516/244065 objects misplaced (0.621%); 7.2 MiB/s, 117 objects/s recovering 2024-02-05T06:41:37.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:37 smithi028 ceph-mon[37984]: 6.14 scrub starts 2024-02-05T06:41:37.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:37 smithi028 ceph-mon[37984]: 6.14 scrub ok 2024-02-05T06:41:37.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:37 smithi028 ceph-mon[37984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:41:37.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:37 smithi028 ceph-mon[37984]: from='client.25135 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:41:37.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:37 smithi028 ceph-mon[37984]: osdmap e245: 8 total, 8 up, 7 in 2024-02-05T06:41:37.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:37 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:41:37.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:37 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:41:37.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:37 smithi028 ceph-mon[29442]: pgmap v388: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 1.08k op/s; 1516/244065 objects misplaced (0.621%); 7.2 MiB/s, 117 objects/s recovering 2024-02-05T06:41:37.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:37 smithi028 ceph-mon[29442]: 6.14 scrub starts 2024-02-05T06:41:37.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:37 smithi028 ceph-mon[29442]: 6.14 scrub ok 2024-02-05T06:41:37.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:37 smithi028 ceph-mon[29442]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:41:37.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:37 smithi028 ceph-mon[29442]: from='client.25135 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:41:37.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:37 smithi028 ceph-mon[29442]: osdmap e245: 8 total, 8 up, 7 in 2024-02-05T06:41:37.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:37 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:41:37.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:37 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:41:37.737 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:37.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:37.771 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:37.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:37.873 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:37.891 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:41:37.904 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:37.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:37.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:37.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:37 smithi167 ceph-mon[32774]: pgmap v388: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 1.08k op/s; 1516/244065 objects misplaced (0.621%); 7.2 MiB/s, 117 objects/s recovering 2024-02-05T06:41:37.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:37 smithi167 ceph-mon[32774]: 6.14 scrub starts 2024-02-05T06:41:37.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:37 smithi167 ceph-mon[32774]: 6.14 scrub ok 2024-02-05T06:41:37.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:37 smithi167 ceph-mon[32774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:41:37.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:37 smithi167 ceph-mon[32774]: from='client.25135 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:41:37.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:37 smithi167 ceph-mon[32774]: osdmap e245: 8 total, 8 up, 7 in 2024-02-05T06:41:37.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:37 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:41:37.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:37 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:41:38.006 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:38.050 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:38.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:38.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:38.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:38.193 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:38.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:38.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:38.295 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:38.330 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:38.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:38.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:38.432 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:38.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:38.473 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:38.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:38.574 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:38.606 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:38.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:38.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:38.707 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:38.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:38.739 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:38.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:38.791 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:41:38.834 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:38 smithi028 ceph-mon[29442]: 4.c scrub starts 2024-02-05T06:41:38.834 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:38 smithi028 ceph-mon[29442]: 4.c scrub ok 2024-02-05T06:41:38.834 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:38 smithi028 ceph-mon[29442]: osdmap e246: 8 total, 8 up, 7 in 2024-02-05T06:41:38.835 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:38 smithi028 ceph-mon[37984]: 4.c scrub starts 2024-02-05T06:41:38.835 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:38 smithi028 ceph-mon[37984]: 4.c scrub ok 2024-02-05T06:41:38.835 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:38 smithi028 ceph-mon[37984]: osdmap e246: 8 total, 8 up, 7 in 2024-02-05T06:41:38.840 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:38.866 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:38.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:38.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:38.968 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:38.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:38 smithi167 ceph-mon[32774]: 4.c scrub starts 2024-02-05T06:41:38.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:38 smithi167 ceph-mon[32774]: 4.c scrub ok 2024-02-05T06:41:38.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:38 smithi167 ceph-mon[32774]: osdmap e246: 8 total, 8 up, 7 in 2024-02-05T06:41:39.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:39.001 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:39.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:39.102 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:39.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:39.153 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:39.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:39.254 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:39.277 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:39.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:39.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:39.379 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:39.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:39.419 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:39.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:39.454 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T06:41:39.520 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:39.585 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:39.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:39.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:39.687 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:39.695 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:39 smithi167 ceph-mon[32774]: pgmap v391: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 878 B/s rd, 0 B/s wr, 808 op/s; 1074/234243 objects misplaced (0.458%); 4.7 MiB/s, 75 objects/s recovering 2024-02-05T06:41:39.695 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:39 smithi167 ceph-mon[32774]: 6.17 deep-scrub starts 2024-02-05T06:41:39.695 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:39 smithi167 ceph-mon[32774]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-02-05T06:41:39.696 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:39 smithi167 ceph-mon[32774]: 6.17 deep-scrub ok 2024-02-05T06:41:39.743 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:39.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:39.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:39.744 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:41:39.744 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-02-05T06:41:39.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-02-05T06:41:39.745 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-02-05T06:41:39.745 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.745 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.44375.0:1114 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94598155039104 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-02-05T06:41:39.745 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:35:23.908978+0000", 2024-02-05T06:41:39.745 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 375.827043076, 2024-02-05T06:41:39.745 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0444529549999999, 2024-02-05T06:41:39.745 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:39.745 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-02-05T06:41:39.745 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:39.745 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.44375", 2024-02-05T06:41:39.745 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/1937500867", 2024-02-05T06:41:39.745 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1114 2024-02-05T06:41:39.745 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.746 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:39.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:39.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.908978+0000", 2024-02-05T06:41:39.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:39.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.908978+0000", 2024-02-05T06:41:39.746 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.746 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.746 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.746 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:39.746 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.908981+0000", 2024-02-05T06:41:39.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0299999999999998e-06 2024-02-05T06:41:39.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:39.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.908989+0000", 2024-02-05T06:41:39.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8080000000000005e-06 2024-02-05T06:41:39.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.747 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.747 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:39.747 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909008+0000", 2024-02-05T06:41:39.747 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8580000000000002e-05 2024-02-05T06:41:39.747 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:39.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909021+0000", 2024-02-05T06:41:39.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2858e-05 2024-02-05T06:41:39.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:39.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909101+0000", 2024-02-05T06:41:39.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0309000000000006e-05 2024-02-05T06:41:39.748 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.748 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.748 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-02-05T06:41:39.748 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909109+0000", 2024-02-05T06:41:39.748 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9829999999999996e-06 2024-02-05T06:41:39.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.749 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:39.749 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.922952+0000", 2024-02-05T06:41:39.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013843519 2024-02-05T06:41:39.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.749 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:41:39.749 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.922964+0000", 2024-02-05T06:41:39.749 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1982e-05 2024-02-05T06:41:39.749 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.749 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.749 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:39.750 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.946185+0000", 2024-02-05T06:41:39.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0232209109999999 2024-02-05T06:41:39.750 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.750 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.750 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for degraded object", 2024-02-05T06:41:39.750 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.946501+0000", 2024-02-05T06:41:39.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00031523400000000001 2024-02-05T06:41:39.750 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.750 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.750 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:39.750 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.951484+0000", 2024-02-05T06:41:39.750 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049829180000000002 2024-02-05T06:41:39.750 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.750 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:39.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.951567+0000", 2024-02-05T06:41:39.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3858999999999998e-05 2024-02-05T06:41:39.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.751 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:39.751 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.953431+0000", 2024-02-05T06:41:39.751 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0018639640000000001 2024-02-05T06:41:39.751 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.751 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:39.751 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.751 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.751 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.752 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.44375.0:1124 5.9 5:93e5b521:::notify.7:head [watch ping cookie 94598155039104 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-02-05T06:41:39.752 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:35:24.614839+0000", 2024-02-05T06:41:39.752 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 375.12118275199998, 2024-02-05T06:41:39.752 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.33742624700000001, 2024-02-05T06:41:39.752 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:39.752 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-02-05T06:41:39.752 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:39.752 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.44375", 2024-02-05T06:41:39.752 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/1937500867", 2024-02-05T06:41:39.752 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1124 2024-02-05T06:41:39.752 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.752 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:39.752 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.752 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:39.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.614839+0000", 2024-02-05T06:41:39.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:39.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.614839+0000", 2024-02-05T06:41:39.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.753 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:39.753 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.614847+0000", 2024-02-05T06:41:39.753 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4840000000000004e-06 2024-02-05T06:41:39.753 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.753 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:39.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.614853+0000", 2024-02-05T06:41:39.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9619999999999998e-06 2024-02-05T06:41:39.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:39.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.614860+0000", 2024-02-05T06:41:39.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.635e-06 2024-02-05T06:41:39.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:39.754 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.614869+0000", 2024-02-05T06:41:39.754 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5640000000000002e-06 2024-02-05T06:41:39.754 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.754 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.754 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:39.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.614962+0000", 2024-02-05T06:41:39.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2708999999999996e-05 2024-02-05T06:41:39.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:41:39.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.614974+0000", 2024-02-05T06:41:39.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2187999999999999e-05 2024-02-05T06:41:39.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:39.755 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.946534+0000", 2024-02-05T06:41:39.755 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.33156012000000001 2024-02-05T06:41:39.755 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.755 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.755 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for degraded object", 2024-02-05T06:41:39.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.946585+0000", 2024-02-05T06:41:39.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0203e-05 2024-02-05T06:41:39.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:39.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.952028+0000", 2024-02-05T06:41:39.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0054438680000000001 2024-02-05T06:41:39.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.756 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:39.756 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.952100+0000", 2024-02-05T06:41:39.756 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1254999999999995e-05 2024-02-05T06:41:39.756 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.756 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.757 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:39.757 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.952265+0000", 2024-02-05T06:41:39.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016525900000000001 2024-02-05T06:41:39.757 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.757 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:39.757 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.757 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.757 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.757 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:43285 7.5 7:a60a1da4:::benchmark_data_smithi028_77509_object43284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-02-05T06:41:39.757 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:35:58.782552+0000", 2024-02-05T06:41:39.757 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 340.95346915800002, 2024-02-05T06:41:39.757 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.133745637, 2024-02-05T06:41:39.757 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:39.758 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:39.758 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:39.758 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:41:39.758 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:41:39.758 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 43285 2024-02-05T06:41:39.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.758 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:39.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.758 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:39.758 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:58.782552+0000", 2024-02-05T06:41:39.758 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.758 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.758 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:39.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:58.782552+0000", 2024-02-05T06:41:39.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:39.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:58.782554+0000", 2024-02-05T06:41:39.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0109999999999999e-06 2024-02-05T06:41:39.759 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.759 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.759 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:39.759 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:58.782605+0000", 2024-02-05T06:41:39.759 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0537000000000001e-05 2024-02-05T06:41:39.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:39.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:58.782619+0000", 2024-02-05T06:41:39.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4430000000000001e-05 2024-02-05T06:41:39.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:39.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:58.782627+0000", 2024-02-05T06:41:39.760 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9910000000000002e-06 2024-02-05T06:41:39.760 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.760 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.760 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:39.760 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:58.782687+0000", 2024-02-05T06:41:39.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9571999999999998e-05 2024-02-05T06:41:39.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:39.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:58.782815+0000", 2024-02-05T06:41:39.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012803599999999999 2024-02-05T06:41:39.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-02-05T06:41:39.761 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:58.782965+0000", 2024-02-05T06:41:39.761 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014986099999999999 2024-02-05T06:41:39.761 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.761 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.761 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:39.762 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:58.784278+0000", 2024-02-05T06:41:39.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0013136269999999999 2024-02-05T06:41:39.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.762 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:39.762 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:58.784728+0000", 2024-02-05T06:41:39.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00044963300000000001 2024-02-05T06:41:39.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.762 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:39.762 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:58.916227+0000", 2024-02-05T06:41:39.762 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13149888800000001 2024-02-05T06:41:39.762 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.762 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:39.763 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:58.916265+0000", 2024-02-05T06:41:39.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8498999999999999e-05 2024-02-05T06:41:39.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.763 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:39.763 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:58.916298+0000", 2024-02-05T06:41:39.763 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2552000000000003e-05 2024-02-05T06:41:39.763 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.763 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:39.763 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.763 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.763 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.763 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:117393 7.5 7:a0daa47f:::benchmark_data_smithi028_77509_object2273:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-02-05T06:41:39.764 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:37:10.315966+0000", 2024-02-05T06:41:39.764 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 269.42005543499999, 2024-02-05T06:41:39.764 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13498384799999999, 2024-02-05T06:41:39.764 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:39.764 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:39.764 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:39.764 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:41:39.764 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:41:39.764 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 117393 2024-02-05T06:41:39.764 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.764 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:39.764 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.764 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:39.764 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:10.315966+0000", 2024-02-05T06:41:39.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:39.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:10.315966+0000", 2024-02-05T06:41:39.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:39.765 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:10.315968+0000", 2024-02-05T06:41:39.765 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.243e-06 2024-02-05T06:41:39.765 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.765 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.765 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:39.766 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:10.315974+0000", 2024-02-05T06:41:39.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0789999999999997e-06 2024-02-05T06:41:39.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.766 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:39.766 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:10.316013+0000", 2024-02-05T06:41:39.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8328999999999998e-05 2024-02-05T06:41:39.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.766 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.766 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:39.766 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:10.316023+0000", 2024-02-05T06:41:39.766 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9969999999999993e-06 2024-02-05T06:41:39.766 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.767 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:39.767 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:10.316089+0000", 2024-02-05T06:41:39.767 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6727e-05 2024-02-05T06:41:39.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.767 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:39.767 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:10.316201+0000", 2024-02-05T06:41:39.767 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011192900000000001 2024-02-05T06:41:39.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.767 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.767 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-02-05T06:41:39.767 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:10.316316+0000", 2024-02-05T06:41:39.767 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000114954 2024-02-05T06:41:39.767 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:39.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:10.319096+0000", 2024-02-05T06:41:39.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0027797870000000001 2024-02-05T06:41:39.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:39.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:10.319169+0000", 2024-02-05T06:41:39.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2470000000000002e-05 2024-02-05T06:41:39.768 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.768 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.768 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:39.768 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:10.450853+0000", 2024-02-05T06:41:39.768 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13168428900000001 2024-02-05T06:41:39.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:39.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:10.450905+0000", 2024-02-05T06:41:39.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1847e-05 2024-02-05T06:41:39.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.769 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.769 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:39.769 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:10.450950+0000", 2024-02-05T06:41:39.769 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5197000000000001e-05 2024-02-05T06:41:39.769 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.769 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:39.769 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.769 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.770 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:153715 7.5 7:a0417651:::benchmark_data_smithi028_77509_object38595:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-05T06:41:39.770 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:37:54.776399+0000", 2024-02-05T06:41:39.770 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 224.95962206300001, 2024-02-05T06:41:39.770 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11539161100000001, 2024-02-05T06:41:39.770 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:39.770 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:39.770 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:39.770 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:41:39.770 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:41:39.770 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 153715 2024-02-05T06:41:39.770 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.770 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:39.770 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.770 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:39.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.776399+0000", 2024-02-05T06:41:39.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:39.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.776399+0000", 2024-02-05T06:41:39.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:39.771 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.776402+0000", 2024-02-05T06:41:39.771 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.176e-06 2024-02-05T06:41:39.771 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.771 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.771 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:39.772 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.776408+0000", 2024-02-05T06:41:39.772 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7139999999999996e-06 2024-02-05T06:41:39.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.773 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:39.773 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.776421+0000", 2024-02-05T06:41:39.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2357e-05 2024-02-05T06:41:39.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.773 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.773 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:39.773 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.776428+0000", 2024-02-05T06:41:39.773 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7349999999999996e-06 2024-02-05T06:41:39.773 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:39.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.776516+0000", 2024-02-05T06:41:39.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7979999999999995e-05 2024-02-05T06:41:39.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:39.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.776657+0000", 2024-02-05T06:41:39.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000140872 2024-02-05T06:41:39.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.774 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.774 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-02-05T06:41:39.774 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.776811+0000", 2024-02-05T06:41:39.774 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015373599999999999 2024-02-05T06:41:39.774 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:39.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.779835+0000", 2024-02-05T06:41:39.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030235689999999998 2024-02-05T06:41:39.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:39.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.810881+0000", 2024-02-05T06:41:39.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.031046601 2024-02-05T06:41:39.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.775 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.775 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:39.775 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.891656+0000", 2024-02-05T06:41:39.775 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.080774652000000002 2024-02-05T06:41:39.775 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:39.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.891722+0000", 2024-02-05T06:41:39.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5833000000000005e-05 2024-02-05T06:41:39.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.776 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:39.776 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:54.891791+0000", 2024-02-05T06:41:39.776 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9386000000000003e-05 2024-02-05T06:41:39.776 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.776 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:39.776 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.776 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.776 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.777 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:154015 7.5 7:ae2f9223:::benchmark_data_smithi028_77509_object38895:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-05T06:41:39.777 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:37:55.122255+0000", 2024-02-05T06:41:39.777 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 224.61376632899999, 2024-02-05T06:41:39.777 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.122266588, 2024-02-05T06:41:39.777 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:39.777 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:39.777 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:39.777 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:41:39.777 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:41:39.777 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 154015 2024-02-05T06:41:39.777 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.777 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:39.777 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.777 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:39.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.122255+0000", 2024-02-05T06:41:39.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:39.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.122255+0000", 2024-02-05T06:41:39.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.778 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:39.778 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.122257+0000", 2024-02-05T06:41:39.778 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1229999999999998e-06 2024-02-05T06:41:39.778 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.778 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:39.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.122263+0000", 2024-02-05T06:41:39.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1129999999999997e-06 2024-02-05T06:41:39.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:39.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.122275+0000", 2024-02-05T06:41:39.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1555e-05 2024-02-05T06:41:39.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.779 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.779 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:39.779 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.122283+0000", 2024-02-05T06:41:39.779 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3310000000000004e-06 2024-02-05T06:41:39.779 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:39.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.199076+0000", 2024-02-05T06:41:39.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.076793014000000007 2024-02-05T06:41:39.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:39.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.199196+0000", 2024-02-05T06:41:39.780 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119388 2024-02-05T06:41:39.780 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.780 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.780 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-02-05T06:41:39.780 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.199332+0000", 2024-02-05T06:41:39.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000136017 2024-02-05T06:41:39.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:39.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.205922+0000", 2024-02-05T06:41:39.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0065907550000000002 2024-02-05T06:41:39.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:39.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.231179+0000", 2024-02-05T06:41:39.781 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025256308000000002 2024-02-05T06:41:39.781 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.781 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.781 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:39.781 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.244399+0000", 2024-02-05T06:41:39.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013220562999999999 2024-02-05T06:41:39.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:39.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.244459+0000", 2024-02-05T06:41:39.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0155999999999998e-05 2024-02-05T06:41:39.782 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.782 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.782 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:39.782 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.244522+0000", 2024-02-05T06:41:39.782 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2265000000000001e-05 2024-02-05T06:41:39.782 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.782 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:39.783 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.783 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.783 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:154023 7.5 7:ab80c2f4:::benchmark_data_smithi028_77509_object38903:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-05T06:41:39.783 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:37:55.124027+0000", 2024-02-05T06:41:39.783 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 224.61199445299999, 2024-02-05T06:41:39.783 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.120633619, 2024-02-05T06:41:39.783 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:39.783 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:39.783 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:39.783 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:41:39.783 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:41:39.783 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 154023 2024-02-05T06:41:39.783 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.783 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:39.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:39.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.124027+0000", 2024-02-05T06:41:39.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:39.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.124027+0000", 2024-02-05T06:41:39.784 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.784 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.784 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.784 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:39.784 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.124029+0000", 2024-02-05T06:41:39.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.623e-06 2024-02-05T06:41:39.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:39.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.124036+0000", 2024-02-05T06:41:39.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6009999999999997e-06 2024-02-05T06:41:39.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:39.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.124048+0000", 2024-02-05T06:41:39.785 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1565e-05 2024-02-05T06:41:39.785 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.785 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.785 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:39.785 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.124055+0000", 2024-02-05T06:41:39.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6149999999999999e-06 2024-02-05T06:41:39.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.786 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:39.786 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.201264+0000", 2024-02-05T06:41:39.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.077208627000000002 2024-02-05T06:41:39.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.786 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:39.786 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.201371+0000", 2024-02-05T06:41:39.786 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106921 2024-02-05T06:41:39.786 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.786 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.786 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-02-05T06:41:39.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.201497+0000", 2024-02-05T06:41:39.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012559 2024-02-05T06:41:39.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:39.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.206010+0000", 2024-02-05T06:41:39.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0045134629999999997 2024-02-05T06:41:39.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.787 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.787 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:39.787 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.231320+0000", 2024-02-05T06:41:39.787 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.02531017 2024-02-05T06:41:39.787 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:39.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.244562+0000", 2024-02-05T06:41:39.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013241358 2024-02-05T06:41:39.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:39.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.244613+0000", 2024-02-05T06:41:39.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1329999999999998e-05 2024-02-05T06:41:39.788 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.788 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.788 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:39.788 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.244661+0000", 2024-02-05T06:41:39.788 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7756000000000002e-05 2024-02-05T06:41:39.789 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.789 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:39.789 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.789 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.789 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.789 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:154642 7.5 7:a5ea576f:::benchmark_data_smithi028_77509_object39522:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-05T06:41:39.789 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:37:55.797205+0000", 2024-02-05T06:41:39.789 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 223.93881683399999, 2024-02-05T06:41:39.789 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.120068486, 2024-02-05T06:41:39.789 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:39.789 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:39.789 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:39.789 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:41:39.789 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:41:39.790 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 154642 2024-02-05T06:41:39.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.790 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:39.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:39.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.797205+0000", 2024-02-05T06:41:39.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:39.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.797205+0000", 2024-02-05T06:41:39.790 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.790 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.790 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.790 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:39.790 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.797206+0000", 2024-02-05T06:41:39.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6190000000000001e-06 2024-02-05T06:41:39.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:39.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.797211+0000", 2024-02-05T06:41:39.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1560000000000003e-06 2024-02-05T06:41:39.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:39.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.797225+0000", 2024-02-05T06:41:39.791 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3439e-05 2024-02-05T06:41:39.791 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.791 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.791 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:39.791 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.797233+0000", 2024-02-05T06:41:39.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2470000000000008e-06 2024-02-05T06:41:39.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:39.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.851882+0000", 2024-02-05T06:41:39.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.054649009999999998 2024-02-05T06:41:39.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:39.792 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.851985+0000", 2024-02-05T06:41:39.792 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010306600000000001 2024-02-05T06:41:39.792 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.792 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.792 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-02-05T06:41:39.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.852097+0000", 2024-02-05T06:41:39.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011226999999999999 2024-02-05T06:41:39.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:39.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.854816+0000", 2024-02-05T06:41:39.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0027185320000000001 2024-02-05T06:41:39.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:39.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.883328+0000", 2024-02-05T06:41:39.793 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.028512062000000001 2024-02-05T06:41:39.793 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.793 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.793 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:39.793 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.917201+0000", 2024-02-05T06:41:39.794 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.033873249000000001 2024-02-05T06:41:39.794 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.794 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.794 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:39.794 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.917237+0000", 2024-02-05T06:41:39.794 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5893999999999999e-05 2024-02-05T06:41:39.794 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.794 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.794 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:39.794 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:55.917273+0000", 2024-02-05T06:41:39.794 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5942000000000002e-05 2024-02-05T06:41:39.794 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.794 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:39.794 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.794 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.795 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:155790 7.5 7:a5a5d249:::benchmark_data_smithi028_77509_object40670:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-05T06:41:39.795 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:37:57.181752+0000", 2024-02-05T06:41:39.795 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 222.55426989700001, 2024-02-05T06:41:39.795 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.114473717, 2024-02-05T06:41:39.795 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:39.795 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:39.795 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:39.795 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:41:39.795 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:41:39.795 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 155790 2024-02-05T06:41:39.795 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.795 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:39.795 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.796 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:39.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.181752+0000", 2024-02-05T06:41:39.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.796 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:39.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.181752+0000", 2024-02-05T06:41:39.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.796 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:39.796 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.181754+0000", 2024-02-05T06:41:39.796 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3470000000000001e-06 2024-02-05T06:41:39.796 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.796 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:39.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.181760+0000", 2024-02-05T06:41:39.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8150000000000002e-06 2024-02-05T06:41:39.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:39.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.181774+0000", 2024-02-05T06:41:39.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4080000000000001e-05 2024-02-05T06:41:39.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.797 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.797 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:39.797 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.181785+0000", 2024-02-05T06:41:39.797 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1077000000000001e-05 2024-02-05T06:41:39.797 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:39.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.226173+0000", 2024-02-05T06:41:39.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.044388550999999998 2024-02-05T06:41:39.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:39.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.226318+0000", 2024-02-05T06:41:39.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014475600000000001 2024-02-05T06:41:39.798 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.798 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.798 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-02-05T06:41:39.798 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.226473+0000", 2024-02-05T06:41:39.798 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015433100000000001 2024-02-05T06:41:39.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:39.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.229296+0000", 2024-02-05T06:41:39.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002823005 2024-02-05T06:41:39.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:39.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.269672+0000", 2024-02-05T06:41:39.799 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.040376173000000001 2024-02-05T06:41:39.799 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.799 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.799 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:39.799 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.296111+0000", 2024-02-05T06:41:39.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.026439190000000001 2024-02-05T06:41:39.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:39.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.296169+0000", 2024-02-05T06:41:39.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7744000000000001e-05 2024-02-05T06:41:39.800 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.800 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.800 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:39.800 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.296225+0000", 2024-02-05T06:41:39.800 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6647999999999997e-05 2024-02-05T06:41:39.800 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.800 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:39.800 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.801 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:156384 7.5 7:a087fe43:::benchmark_data_smithi028_77509_object41264:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-05T06:41:39.801 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:37:57.883589+0000", 2024-02-05T06:41:39.801 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 221.85243201, 2024-02-05T06:41:39.801 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.114714398, 2024-02-05T06:41:39.801 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:39.801 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:39.801 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:39.801 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:41:39.801 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:41:39.801 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 156384 2024-02-05T06:41:39.801 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.801 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:39.801 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:39.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.883589+0000", 2024-02-05T06:41:39.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:39.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.883589+0000", 2024-02-05T06:41:39.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.802 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.802 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.802 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:39.802 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.883591+0000", 2024-02-05T06:41:39.802 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9e-06 2024-02-05T06:41:39.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:39.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.883599+0000", 2024-02-05T06:41:39.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5440000000000001e-06 2024-02-05T06:41:39.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:39.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.883611+0000", 2024-02-05T06:41:39.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.203e-05 2024-02-05T06:41:39.803 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.803 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.803 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:39.803 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.883619+0000", 2024-02-05T06:41:39.803 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8990000000000001e-06 2024-02-05T06:41:39.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:39.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.939404+0000", 2024-02-05T06:41:39.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055785243999999998 2024-02-05T06:41:39.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.804 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:39.804 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.939489+0000", 2024-02-05T06:41:39.804 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5130000000000007e-05 2024-02-05T06:41:39.804 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.804 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-02-05T06:41:39.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.939570+0000", 2024-02-05T06:41:39.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1012000000000003e-05 2024-02-05T06:41:39.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:39.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.944723+0000", 2024-02-05T06:41:39.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0051528320000000004 2024-02-05T06:41:39.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.805 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:39.805 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.974999+0000", 2024-02-05T06:41:39.805 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.030276093 2024-02-05T06:41:39.805 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.805 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:39.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.998170+0000", 2024-02-05T06:41:39.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023171056999999998 2024-02-05T06:41:39.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:39.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.998226+0000", 2024-02-05T06:41:39.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5392000000000001e-05 2024-02-05T06:41:39.806 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.806 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.806 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:39.806 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.998304+0000", 2024-02-05T06:41:39.806 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8264999999999997e-05 2024-02-05T06:41:39.806 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.806 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:39.807 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.807 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.807 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:156385 7.5 7:a10ff918:::benchmark_data_smithi028_77509_object41265:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-05T06:41:39.807 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:37:57.883790+0000", 2024-02-05T06:41:39.807 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 221.85223140400001, 2024-02-05T06:41:39.807 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11466615500000001, 2024-02-05T06:41:39.807 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:39.807 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:39.807 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:39.807 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:41:39.807 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:41:39.807 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 156385 2024-02-05T06:41:39.807 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.808 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:39.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:39.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.883790+0000", 2024-02-05T06:41:39.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:39.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.883790+0000", 2024-02-05T06:41:39.808 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.808 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.808 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.808 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:39.808 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.883791+0000", 2024-02-05T06:41:39.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3200000000000005e-07 2024-02-05T06:41:39.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:39.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.883794+0000", 2024-02-05T06:41:39.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8140000000000001e-06 2024-02-05T06:41:39.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:39.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.883800+0000", 2024-02-05T06:41:39.809 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4960000000000001e-06 2024-02-05T06:41:39.809 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.809 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.809 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:39.809 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.883805+0000", 2024-02-05T06:41:39.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.327e-06 2024-02-05T06:41:39.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:39.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.941887+0000", 2024-02-05T06:41:39.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058081527000000001 2024-02-05T06:41:39.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:39.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.942024+0000", 2024-02-05T06:41:39.810 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013760399999999999 2024-02-05T06:41:39.810 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.810 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.810 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-02-05T06:41:39.810 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.942175+0000", 2024-02-05T06:41:39.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015072100000000001 2024-02-05T06:41:39.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:39.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.944734+0000", 2024-02-05T06:41:39.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025590980000000001 2024-02-05T06:41:39.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:39.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.975051+0000", 2024-02-05T06:41:39.811 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.030317196000000001 2024-02-05T06:41:39.811 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.811 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.811 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:39.811 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.998354+0000", 2024-02-05T06:41:39.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023302607999999999 2024-02-05T06:41:39.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:39.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.998397+0000", 2024-02-05T06:41:39.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2982000000000003e-05 2024-02-05T06:41:39.812 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.812 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.812 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:39.812 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.998456+0000", 2024-02-05T06:41:39.812 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9150000000000001e-05 2024-02-05T06:41:39.812 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.812 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:39.812 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.813 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.813 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:156428 7.5 7:a3515a3b:::benchmark_data_smithi028_77509_object41308:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-05T06:41:39.813 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:37:57.940631+0000", 2024-02-05T06:41:39.813 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 221.795390166, 2024-02-05T06:41:39.813 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.132758237, 2024-02-05T06:41:39.813 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:39.813 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:39.813 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:39.813 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:41:39.813 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:41:39.813 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 156428 2024-02-05T06:41:39.813 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.813 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:39.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:39.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.940631+0000", 2024-02-05T06:41:39.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:39.814 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.940631+0000", 2024-02-05T06:41:39.814 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.814 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.814 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.814 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:39.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.940633+0000", 2024-02-05T06:41:39.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0849999999999999e-06 2024-02-05T06:41:39.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:39.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.940639+0000", 2024-02-05T06:41:39.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.22e-06 2024-02-05T06:41:39.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:39.815 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.940651+0000", 2024-02-05T06:41:39.815 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2846e-05 2024-02-05T06:41:39.815 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.815 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.815 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:39.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.940660+0000", 2024-02-05T06:41:39.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.473e-06 2024-02-05T06:41:39.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:39.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.950249+0000", 2024-02-05T06:41:39.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0095890899999999998 2024-02-05T06:41:39.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.816 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:39.816 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.950381+0000", 2024-02-05T06:41:39.816 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013228 2024-02-05T06:41:39.816 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.816 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-02-05T06:41:39.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.950500+0000", 2024-02-05T06:41:39.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119097 2024-02-05T06:41:39.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:39.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.953175+0000", 2024-02-05T06:41:39.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002674282 2024-02-05T06:41:39.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.817 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.817 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:39.817 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:57.975217+0000", 2024-02-05T06:41:39.817 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022042161000000001 2024-02-05T06:41:39.817 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:39.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.073272+0000", 2024-02-05T06:41:39.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098055613 2024-02-05T06:41:39.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:39.818 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.073332+0000", 2024-02-05T06:41:39.818 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9815999999999998e-05 2024-02-05T06:41:39.818 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.818 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.818 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:39.819 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.073390+0000", 2024-02-05T06:41:39.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7274000000000001e-05 2024-02-05T06:41:39.819 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.819 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:39.819 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.819 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.819 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.819 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:156721 7.5 7:a2338077:::benchmark_data_smithi028_77509_object41601:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-05T06:41:39.819 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:37:58.283491+0000", 2024-02-05T06:41:39.819 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 221.45253043100001, 2024-02-05T06:41:39.819 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.113973083, 2024-02-05T06:41:39.819 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:39.820 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:39.820 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:39.820 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:41:39.820 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:41:39.820 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 156721 2024-02-05T06:41:39.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.820 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:39.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.820 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:39.820 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.283491+0000", 2024-02-05T06:41:39.820 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.820 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.820 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:39.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.283491+0000", 2024-02-05T06:41:39.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:39.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.283493+0000", 2024-02-05T06:41:39.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.306e-06 2024-02-05T06:41:39.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.821 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.821 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:39.821 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.283500+0000", 2024-02-05T06:41:39.821 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.161e-06 2024-02-05T06:41:39.821 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:39.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.283511+0000", 2024-02-05T06:41:39.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1593000000000001e-05 2024-02-05T06:41:39.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:39.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.283523+0000", 2024-02-05T06:41:39.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2177999999999999e-05 2024-02-05T06:41:39.822 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.822 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.822 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:39.822 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.337457+0000", 2024-02-05T06:41:39.822 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.053933285999999997 2024-02-05T06:41:39.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:39.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.337552+0000", 2024-02-05T06:41:39.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5509999999999999e-05 2024-02-05T06:41:39.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-02-05T06:41:39.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.337640+0000", 2024-02-05T06:41:39.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7959000000000003e-05 2024-02-05T06:41:39.823 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.823 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.823 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:39.823 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.339366+0000", 2024-02-05T06:41:39.823 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0017258270000000001 2024-02-05T06:41:39.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:39.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.385612+0000", 2024-02-05T06:41:39.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046246559 2024-02-05T06:41:39.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:39.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.397372+0000", 2024-02-05T06:41:39.824 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.011759639000000001 2024-02-05T06:41:39.824 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.824 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.824 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:39.824 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.397410+0000", 2024-02-05T06:41:39.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7743000000000001e-05 2024-02-05T06:41:39.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.825 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:39.825 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:58.397464+0000", 2024-02-05T06:41:39.825 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4321999999999999e-05 2024-02-05T06:41:39.825 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.825 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:39.825 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.825 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.825 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.825 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:157240 7.5 7:ad6fac87:::benchmark_data_smithi028_77509_object42120:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-02-05T06:41:39.825 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:37:59.612156+0000", 2024-02-05T06:41:39.825 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 220.123865757, 2024-02-05T06:41:39.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.131971483, 2024-02-05T06:41:39.826 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:39.826 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:39.826 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:39.826 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:41:39.826 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:41:39.826 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 157240 2024-02-05T06:41:39.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.826 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:39.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.826 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:39.826 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:59.612156+0000", 2024-02-05T06:41:39.826 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.826 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.826 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:39.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:59.612156+0000", 2024-02-05T06:41:39.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:39.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:59.612158+0000", 2024-02-05T06:41:39.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1280000000000002e-06 2024-02-05T06:41:39.827 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.827 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.827 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:39.827 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:59.612163+0000", 2024-02-05T06:41:39.827 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.113e-06 2024-02-05T06:41:39.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:39.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:59.612175+0000", 2024-02-05T06:41:39.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.188e-05 2024-02-05T06:41:39.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:39.828 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:59.612183+0000", 2024-02-05T06:41:39.828 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2530000000000004e-06 2024-02-05T06:41:39.828 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.828 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.828 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:39.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:59.635929+0000", 2024-02-05T06:41:39.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023746257 2024-02-05T06:41:39.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:39.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:59.636058+0000", 2024-02-05T06:41:39.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012825099999999999 2024-02-05T06:41:39.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.829 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-02-05T06:41:39.829 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:59.636197+0000", 2024-02-05T06:41:39.829 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013920600000000001 2024-02-05T06:41:39.829 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.829 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:39.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:59.639128+0000", 2024-02-05T06:41:39.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029314609999999998 2024-02-05T06:41:39.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:39.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:59.682887+0000", 2024-02-05T06:41:39.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.043758745000000002 2024-02-05T06:41:39.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.830 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.830 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:39.830 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:59.744012+0000", 2024-02-05T06:41:39.830 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.061124676000000003 2024-02-05T06:41:39.830 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:39.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:59.744070+0000", 2024-02-05T06:41:39.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8091999999999999e-05 2024-02-05T06:41:39.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.831 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:39.831 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:37:59.744127+0000", 2024-02-05T06:41:39.831 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7420999999999998e-05 2024-02-05T06:41:39.831 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.831 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:39.831 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.831 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.831 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.832 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:157506 7.5 7:a1c9f73b:::benchmark_data_smithi028_77509_object42386:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-02-05T06:41:39.832 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:38:00.039433+0000", 2024-02-05T06:41:39.832 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 219.696588488, 2024-02-05T06:41:39.832 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.118693295, 2024-02-05T06:41:39.832 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:39.832 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:39.832 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:39.832 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:41:39.832 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:41:39.832 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 157506 2024-02-05T06:41:39.832 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.832 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:39.832 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:39.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.039433+0000", 2024-02-05T06:41:39.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:39.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.039433+0000", 2024-02-05T06:41:39.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.833 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:39.833 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.039435+0000", 2024-02-05T06:41:39.833 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0600000000000002e-06 2024-02-05T06:41:39.833 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.833 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:39.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.039440+0000", 2024-02-05T06:41:39.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5310000000000003e-06 2024-02-05T06:41:39.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:39.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.039448+0000", 2024-02-05T06:41:39.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8920000000000006e-06 2024-02-05T06:41:39.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.834 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:39.834 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.039456+0000", 2024-02-05T06:41:39.834 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0849999999999999e-06 2024-02-05T06:41:39.834 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.834 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:39.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.074202+0000", 2024-02-05T06:41:39.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.034746698999999999 2024-02-05T06:41:39.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:39.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.074341+0000", 2024-02-05T06:41:39.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000138388 2024-02-05T06:41:39.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-02-05T06:41:39.835 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.074491+0000", 2024-02-05T06:41:39.835 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000150161 2024-02-05T06:41:39.835 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.835 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.835 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:39.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.079698+0000", 2024-02-05T06:41:39.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0052076630000000004 2024-02-05T06:41:39.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:39.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.117838+0000", 2024-02-05T06:41:39.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.038139978999999997 2024-02-05T06:41:39.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.836 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:39.836 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.157996+0000", 2024-02-05T06:41:39.836 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.040157625000000002 2024-02-05T06:41:39.836 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.836 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:39.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.158064+0000", 2024-02-05T06:41:39.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.792e-05 2024-02-05T06:41:39.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.837 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:39.837 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.158126+0000", 2024-02-05T06:41:39.837 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2292000000000006e-05 2024-02-05T06:41:39.837 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.837 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:39.837 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.837 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.837 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.837 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:157978 7.5 7:a474196b:::benchmark_data_smithi028_77509_object42858:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-02-05T06:41:39.837 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:38:00.653715+0000", 2024-02-05T06:41:39.837 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 219.082306695, 2024-02-05T06:41:39.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.149159445, 2024-02-05T06:41:39.838 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:39.838 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:39.838 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:39.838 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:41:39.838 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:41:39.838 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 157978 2024-02-05T06:41:39.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.838 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:39.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:39.838 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.653715+0000", 2024-02-05T06:41:39.838 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.838 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.838 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.838 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:39.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.653715+0000", 2024-02-05T06:41:39.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:39.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.653716+0000", 2024-02-05T06:41:39.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.269e-06 2024-02-05T06:41:39.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:39.839 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.653720+0000", 2024-02-05T06:41:39.839 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1370000000000004e-06 2024-02-05T06:41:39.839 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.839 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.839 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:39.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.653735+0000", 2024-02-05T06:41:39.840 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.452e-05 2024-02-05T06:41:39.840 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.840 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.840 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:39.840 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.653743+0000", 2024-02-05T06:41:39.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4710000000000007e-06 2024-02-05T06:41:39.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:39.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.675803+0000", 2024-02-05T06:41:39.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022060158 2024-02-05T06:41:39.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.841 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.841 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:39.841 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.675903+0000", 2024-02-05T06:41:39.841 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9516000000000001e-05 2024-02-05T06:41:39.841 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-02-05T06:41:39.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.675987+0000", 2024-02-05T06:41:39.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4429000000000001e-05 2024-02-05T06:41:39.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:41:39.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.677212+0000", 2024-02-05T06:41:39.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012252020000000001 2024-02-05T06:41:39.842 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.842 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.842 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:39.842 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.735136+0000", 2024-02-05T06:41:39.842 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057923184000000003 2024-02-05T06:41:39.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:41:39.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.802760+0000", 2024-02-05T06:41:39.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067624563999999998 2024-02-05T06:41:39.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:39.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.802819+0000", 2024-02-05T06:41:39.843 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8328999999999997e-05 2024-02-05T06:41:39.843 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.843 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.843 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:39.843 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.802874+0000", 2024-02-05T06:41:39.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5665999999999999e-05 2024-02-05T06:41:39.844 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.844 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:39.844 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.844 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.844 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.844 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14568.0:403 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e205)", 2024-02-05T06:41:39.844 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:26.898363+0000", 2024-02-05T06:41:39.844 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 72.837658176000005, 2024-02-05T06:41:39.844 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.18084518599999999, 2024-02-05T06:41:39.844 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:39.844 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-02-05T06:41:39.844 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:41:39.845 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14568", 2024-02-05T06:41:39.845 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.167:0/939366379", 2024-02-05T06:41:39.845 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 403 2024-02-05T06:41:39.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.845 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:39.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:39.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.898363+0000", 2024-02-05T06:41:39.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.845 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.845 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:39.845 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.898363+0000", 2024-02-05T06:41:39.845 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.845 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:39.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.898366+0000", 2024-02-05T06:41:39.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6520000000000002e-06 2024-02-05T06:41:39.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:39.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.898375+0000", 2024-02-05T06:41:39.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7390000000000005e-06 2024-02-05T06:41:39.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.846 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.846 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:39.846 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.898391+0000", 2024-02-05T06:41:39.846 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6427999999999999e-05 2024-02-05T06:41:39.846 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:39.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.898400+0000", 2024-02-05T06:41:39.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4610000000000008e-06 2024-02-05T06:41:39.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:39.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.898454+0000", 2024-02-05T06:41:39.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4317999999999997e-05 2024-02-05T06:41:39.847 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.847 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.847 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:41:39.847 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.898463+0000", 2024-02-05T06:41:39.847 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.704e-06 2024-02-05T06:41:39.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:39.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.071801+0000", 2024-02-05T06:41:39.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.17333821099999999 2024-02-05T06:41:39.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-02-05T06:41:39.848 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.071937+0000", 2024-02-05T06:41:39.848 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013592899999999999 2024-02-05T06:41:39.848 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.848 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.848 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:39.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.078945+0000", 2024-02-05T06:41:39.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0070086469999999998 2024-02-05T06:41:39.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:39.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.079046+0000", 2024-02-05T06:41:39.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010094099999999999 2024-02-05T06:41:39.849 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.849 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.849 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:39.849 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.079208+0000", 2024-02-05T06:41:39.849 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000162156 2024-02-05T06:41:39.849 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.849 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:39.849 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.850 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "MOSDPGPush(8.c 234/205 [PushOp(8:31c07e14:::benchmark_data_smithi028_105698_object21451:head, version: 193'1292, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:31c07e14:::benchmark_data_smithi028_105698_object21451:head@193'1292, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-05T06:41:39.850 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:41:14.343396+0000", 2024-02-05T06:41:39.850 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 25.392625771999999, 2024-02-05T06:41:39.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.12074657599999999, 2024-02-05T06:41:39.850 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:39.850 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-02-05T06:41:39.850 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:39.850 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.850 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:39.850 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:14.343396+0000", 2024-02-05T06:41:39.850 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.850 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.851 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:39.851 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:14.343396+0000", 2024-02-05T06:41:39.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.851 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:39.851 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:14.343408+0000", 2024-02-05T06:41:39.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2163e-05 2024-02-05T06:41:39.851 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.851 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.851 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:39.851 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:14.343449+0000", 2024-02-05T06:41:39.851 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0738000000000002e-05 2024-02-05T06:41:39.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:39.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:14.441291+0000", 2024-02-05T06:41:39.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.097842751000000006 2024-02-05T06:41:39.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:39.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:14.441360+0000", 2024-02-05T06:41:39.852 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8366000000000002e-05 2024-02-05T06:41:39.852 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.852 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.852 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:39.852 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:14.462495+0000", 2024-02-05T06:41:39.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021135412999999999 2024-02-05T06:41:39.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:39.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:14.462505+0000", 2024-02-05T06:41:39.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6269999999999995e-06 2024-02-05T06:41:39.853 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.853 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.853 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:39.853 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:14.464142+0000", 2024-02-05T06:41:39.853 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0016375179999999999 2024-02-05T06:41:39.853 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.853 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:39.853 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.854 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.15144.0:84987 8.6 e235/206 8:62c6998f:::benchmark_data_smithi028_105698_object84986:head v 235'5250, mlcod=192'730)", 2024-02-05T06:41:39.854 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:41:20.057107+0000", 2024-02-05T06:41:39.854 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.678914974000001, 2024-02-05T06:41:39.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.124286945, 2024-02-05T06:41:39.854 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:39.854 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:41:39.854 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:39.854 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.854 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:39.854 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:20.057107+0000", 2024-02-05T06:41:39.854 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.854 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:39.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:20.057107+0000", 2024-02-05T06:41:39.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:39.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:20.057119+0000", 2024-02-05T06:41:39.855 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2937e-05 2024-02-05T06:41:39.855 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.855 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.855 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:39.855 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:20.057188+0000", 2024-02-05T06:41:39.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8134999999999997e-05 2024-02-05T06:41:39.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:39.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:20.082780+0000", 2024-02-05T06:41:39.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025592012000000001 2024-02-05T06:41:39.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:39.856 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:20.082787+0000", 2024-02-05T06:41:39.856 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9689999999999997e-06 2024-02-05T06:41:39.856 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.856 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.856 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:39.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:20.082836+0000", 2024-02-05T06:41:39.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9546000000000003e-05 2024-02-05T06:41:39.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:39.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:20.082869+0000", 2024-02-05T06:41:39.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3009999999999997e-05 2024-02-05T06:41:39.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.857 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:41:39.857 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:20.181278+0000", 2024-02-05T06:41:39.857 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098409281000000001 2024-02-05T06:41:39.857 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.857 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.858 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:39.858 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:20.181393+0000", 2024-02-05T06:41:39.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115055 2024-02-05T06:41:39.858 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.858 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:39.858 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.858 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.858 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.858 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "MOSDPGPush(8.6 235/206 [PushOp(8:6ee13a37:::benchmark_data_smithi028_105698_object12677:head, version: 192'732, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:6ee13a37:::benchmark_data_smithi028_105698_object12677:head@192'732, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-02-05T06:41:39.858 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:41:20.082920+0000", 2024-02-05T06:41:39.858 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 19.653101511999999, 2024-02-05T06:41:39.858 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11510062, 2024-02-05T06:41:39.858 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:41:39.858 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-02-05T06:41:39.859 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:41:39.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:41:39.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:20.082920+0000", 2024-02-05T06:41:39.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:41:39.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:20.082920+0000", 2024-02-05T06:41:39.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:41:39.859 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.859 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.859 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:41:39.859 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:20.082921+0000", 2024-02-05T06:41:39.859 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5799999999999999e-07 2024-02-05T06:41:39.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:41:39.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:20.082957+0000", 2024-02-05T06:41:39.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6656000000000003e-05 2024-02-05T06:41:39.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:41:39.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:20.082971+0000", 2024-02-05T06:41:39.860 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3454e-05 2024-02-05T06:41:39.860 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.860 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.860 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:41:39.860 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:20.082983+0000", 2024-02-05T06:41:39.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.241e-05 2024-02-05T06:41:39.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:41:39.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:20.196515+0000", 2024-02-05T06:41:39.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.113532164 2024-02-05T06:41:39.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:41:39.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:20.196524+0000", 2024-02-05T06:41:39.861 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8009999999999993e-06 2024-02-05T06:41:39.861 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:41:39.861 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:41:39.861 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:41:39.861 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:20.198021+0000", 2024-02-05T06:41:39.862 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001496577 2024-02-05T06:41:39.862 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.862 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:39.862 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.862 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:41:39.862 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:41:39.862 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:41:39.862 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:39.893 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:39.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:39.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:39.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:39 smithi028 ceph-mon[37984]: pgmap v391: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 878 B/s rd, 0 B/s wr, 808 op/s; 1074/234243 objects misplaced (0.458%); 4.7 MiB/s, 75 objects/s recovering 2024-02-05T06:41:39.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:39 smithi028 ceph-mon[37984]: 6.17 deep-scrub starts 2024-02-05T06:41:39.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:39 smithi028 ceph-mon[37984]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-02-05T06:41:39.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:39 smithi028 ceph-mon[37984]: 6.17 deep-scrub ok 2024-02-05T06:41:39.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:39 smithi028 ceph-mon[29442]: pgmap v391: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 878 B/s rd, 0 B/s wr, 808 op/s; 1074/234243 objects misplaced (0.458%); 4.7 MiB/s, 75 objects/s recovering 2024-02-05T06:41:39.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:39 smithi028 ceph-mon[29442]: 6.17 deep-scrub starts 2024-02-05T06:41:39.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:39 smithi028 ceph-mon[29442]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-02-05T06:41:39.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:39 smithi028 ceph-mon[29442]: 6.17 deep-scrub ok 2024-02-05T06:41:39.995 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:40.018 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:40.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:40.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:40.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:40.163 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:40.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:40.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:40.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:40.307 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:40.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:40.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:40.409 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:40.436 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:40.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:40.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:40.538 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:40.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:40.566 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:40.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:40.584 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:40 smithi028 ceph-mon[29442]: 4.1c scrub starts 2024-02-05T06:41:40.584 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:40 smithi028 ceph-mon[29442]: 4.1c scrub ok 2024-02-05T06:41:40.584 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:40 smithi028 ceph-mon[29442]: 6.1e scrub starts 2024-02-05T06:41:40.584 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:40 smithi028 ceph-mon[29442]: 6.1e scrub ok 2024-02-05T06:41:40.584 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:40 smithi028 ceph-mon[37984]: 4.1c scrub starts 2024-02-05T06:41:40.584 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:40 smithi028 ceph-mon[37984]: 4.1c scrub ok 2024-02-05T06:41:40.584 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:40 smithi028 ceph-mon[37984]: 6.1e scrub starts 2024-02-05T06:41:40.584 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:40 smithi028 ceph-mon[37984]: 6.1e scrub ok 2024-02-05T06:41:40.668 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:40.697 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:40.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:40.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:40.754 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-05T06:41:40.799 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:40.824 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:40.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:40.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:40.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:40.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:40.974 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:40.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:40.980 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:41:40.980 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:41:40.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:40 smithi167 ceph-mon[32774]: 4.1c scrub starts 2024-02-05T06:41:40.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:40 smithi167 ceph-mon[32774]: 4.1c scrub ok 2024-02-05T06:41:40.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:40 smithi167 ceph-mon[32774]: 6.1e scrub starts 2024-02-05T06:41:40.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:40 smithi167 ceph-mon[32774]: 6.1e scrub ok 2024-02-05T06:41:41.076 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:41.105 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:41.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:41.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:41.207 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:41.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:41.239 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:41.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:41.340 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:41.368 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:41.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:41.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:41.470 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:41.509 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:41.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:41.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:41.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:41.651 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:41.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:41.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:41.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:41 smithi028 ceph-mon[37984]: pgmap v392: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 749 B/s rd, 0 B/s wr, 689 op/s; 1074/234243 objects misplaced (0.458%); 4.0 MiB/s, 64 objects/s recovering 2024-02-05T06:41:41.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:41 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:41:41.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:41 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:41:41.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:41 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2317522990' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-02-05T06:41:41.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:41 smithi028 ceph-mon[37984]: from='client.25147 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-02-05T06:41:41.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:41 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-a[29438]: 2024-02-05T06:41:41.490+0000 7fa733293700 -1 mon.a@0(leader).osd e246 definitely_dead 0 2024-02-05T06:41:41.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:41 smithi028 ceph-mon[29442]: pgmap v392: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 749 B/s rd, 0 B/s wr, 689 op/s; 1074/234243 objects misplaced (0.458%); 4.0 MiB/s, 64 objects/s recovering 2024-02-05T06:41:41.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:41 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:41:41.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:41 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:41:41.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:41 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2317522990' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-02-05T06:41:41.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:41 smithi028 ceph-mon[29442]: from='client.25147 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-02-05T06:41:41.754 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:41.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:41.786 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:41.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:41.888 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:41.916 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:41.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:41.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:41.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:41 smithi167 ceph-mon[32774]: pgmap v392: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 749 B/s rd, 0 B/s wr, 689 op/s; 1074/234243 objects misplaced (0.458%); 4.0 MiB/s, 64 objects/s recovering 2024-02-05T06:41:41.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:41 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:41:41.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:41 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:41:41.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:41 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2317522990' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-02-05T06:41:41.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:41 smithi167 ceph-mon[32774]: from='client.25147 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-02-05T06:41:42.018 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:42.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:42.073 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:42.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:42.174 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:42.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:42.218 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:42.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:42.319 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:42.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:42.357 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:42.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:42.409 INFO:teuthology.orchestra.run.smithi028.stderr:marked in osd.5. 2024-02-05T06:41:42.459 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:42.484 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:42.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:42.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:42.586 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:42.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:42.610 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:42.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:42.697 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:41:42 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:41:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:41:42.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:42.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:42.736 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:42.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:42.837 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:42.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:42.861 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:42.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:42.963 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:42.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:42.988 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:42.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:43.089 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:43.114 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:43.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:43.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:43.215 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:43.232 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:41:42 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:41:43.239 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:43.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:43.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:43.341 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:43.365 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:43.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:43.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:43.467 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:43.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:43.491 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:43.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:43.592 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:43.616 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:43.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:43.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:43.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:43 smithi028 ceph-mon[29442]: 5.14 scrub starts 2024-02-05T06:41:43.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:43 smithi028 ceph-mon[29442]: 5.14 scrub ok 2024-02-05T06:41:43.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:43 smithi028 ceph-mon[29442]: pgmap v393: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 798/218400 objects misplaced (0.365%); 5.6 MiB/s, 89 objects/s recovering 2024-02-05T06:41:43.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:43 smithi028 ceph-mon[29442]: 3.19 scrub starts 2024-02-05T06:41:43.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:43 smithi028 ceph-mon[29442]: 3.19 scrub ok 2024-02-05T06:41:43.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:43 smithi028 ceph-mon[29442]: from='client.25147 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-02-05T06:41:43.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:43 smithi028 ceph-mon[29442]: 5.8 scrub starts 2024-02-05T06:41:43.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:43 smithi028 ceph-mon[29442]: osdmap e247: 8 total, 8 up, 8 in 2024-02-05T06:41:43.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:43 smithi028 ceph-mon[29442]: 5.8 scrub ok 2024-02-05T06:41:43.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:43 smithi028 ceph-mon[29442]: 4.18 deep-scrub starts 2024-02-05T06:41:43.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:43 smithi028 ceph-mon[29442]: 4.18 deep-scrub ok 2024-02-05T06:41:43.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:43 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:41:43.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:43 smithi028 ceph-mon[37984]: 5.14 scrub starts 2024-02-05T06:41:43.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:43 smithi028 ceph-mon[37984]: 5.14 scrub ok 2024-02-05T06:41:43.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:43 smithi028 ceph-mon[37984]: pgmap v393: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 798/218400 objects misplaced (0.365%); 5.6 MiB/s, 89 objects/s recovering 2024-02-05T06:41:43.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:43 smithi028 ceph-mon[37984]: 3.19 scrub starts 2024-02-05T06:41:43.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:43 smithi028 ceph-mon[37984]: 3.19 scrub ok 2024-02-05T06:41:43.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:43 smithi028 ceph-mon[37984]: from='client.25147 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-02-05T06:41:43.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:43 smithi028 ceph-mon[37984]: 5.8 scrub starts 2024-02-05T06:41:43.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:43 smithi028 ceph-mon[37984]: osdmap e247: 8 total, 8 up, 8 in 2024-02-05T06:41:43.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:43 smithi028 ceph-mon[37984]: 5.8 scrub ok 2024-02-05T06:41:43.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:43 smithi028 ceph-mon[37984]: 4.18 deep-scrub starts 2024-02-05T06:41:43.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:43 smithi028 ceph-mon[37984]: 4.18 deep-scrub ok 2024-02-05T06:41:43.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:43 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:41:43.718 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:43.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:43 smithi167 ceph-mon[32774]: 5.14 scrub starts 2024-02-05T06:41:43.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:43 smithi167 ceph-mon[32774]: 5.14 scrub ok 2024-02-05T06:41:43.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:43 smithi167 ceph-mon[32774]: pgmap v393: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 798/218400 objects misplaced (0.365%); 5.6 MiB/s, 89 objects/s recovering 2024-02-05T06:41:43.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:43 smithi167 ceph-mon[32774]: 3.19 scrub starts 2024-02-05T06:41:43.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:43 smithi167 ceph-mon[32774]: 3.19 scrub ok 2024-02-05T06:41:43.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:43 smithi167 ceph-mon[32774]: from='client.25147 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-02-05T06:41:43.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:43 smithi167 ceph-mon[32774]: 5.8 scrub starts 2024-02-05T06:41:43.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:43 smithi167 ceph-mon[32774]: osdmap e247: 8 total, 8 up, 8 in 2024-02-05T06:41:43.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:43 smithi167 ceph-mon[32774]: 5.8 scrub ok 2024-02-05T06:41:43.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:43 smithi167 ceph-mon[32774]: 4.18 deep-scrub starts 2024-02-05T06:41:43.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:43 smithi167 ceph-mon[32774]: 4.18 deep-scrub ok 2024-02-05T06:41:43.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:43 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:41:43.742 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:43.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:43.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:43.843 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:43.868 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:43.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:43.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:43.970 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:43.994 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:43.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:43.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:44.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:44.120 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:44.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:44.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:44.222 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:44.248 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:44.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:44.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:44.252 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:41:44.253 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-02-05T06:41:44.350 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:44.374 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:44.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:44.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:44.476 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:44.514 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:44.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:44.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:44.525 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:44 smithi028 ceph-mon[29442]: osdmap e248: 8 total, 8 up, 8 in 2024-02-05T06:41:44.525 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:44 smithi028 ceph-mon[29442]: 5.a scrub starts 2024-02-05T06:41:44.525 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:44 smithi028 ceph-mon[29442]: 5.a scrub ok 2024-02-05T06:41:44.525 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:44 smithi028 ceph-mon[29442]: osdmap e249: 8 total, 8 up, 8 in 2024-02-05T06:41:44.525 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:44 smithi028 ceph-mon[37984]: osdmap e248: 8 total, 8 up, 8 in 2024-02-05T06:41:44.525 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:44 smithi028 ceph-mon[37984]: 5.a scrub starts 2024-02-05T06:41:44.525 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:44 smithi028 ceph-mon[37984]: 5.a scrub ok 2024-02-05T06:41:44.525 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:44 smithi028 ceph-mon[37984]: osdmap e249: 8 total, 8 up, 8 in 2024-02-05T06:41:44.616 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:44.652 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:44.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:44.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:44.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:44 smithi167 ceph-mon[32774]: osdmap e248: 8 total, 8 up, 8 in 2024-02-05T06:41:44.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:44 smithi167 ceph-mon[32774]: 5.a scrub starts 2024-02-05T06:41:44.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:44 smithi167 ceph-mon[32774]: 5.a scrub ok 2024-02-05T06:41:44.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:44 smithi167 ceph-mon[32774]: osdmap e249: 8 total, 8 up, 8 in 2024-02-05T06:41:44.754 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:44.801 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:44.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:44.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:44.903 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:44.939 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:44.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:44.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:45.041 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:45.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:45.087 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:45.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:45.188 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:45.220 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:45.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:45.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:45.323 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:45.355 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:45.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:45.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:45.457 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:45.485 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:45.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:45.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:45.587 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:45.614 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:45.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:45.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:45.708 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:45 smithi028 ceph-mon[37984]: pgmap v396: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 754 B/s rd, 0 B/s wr, 778 op/s; 798/218400 objects misplaced (0.365%); 2.5 MiB/s, 40 objects/s recovering 2024-02-05T06:41:45.708 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:45 smithi028 ceph-mon[37984]: 6.4 scrub starts 2024-02-05T06:41:45.708 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:45 smithi028 ceph-mon[37984]: 6.4 scrub ok 2024-02-05T06:41:45.708 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:45 smithi028 ceph-mon[37984]: 8.8 starting backfill to osd.5 from (0'0,194'1981] MIN to 246'6863 2024-02-05T06:41:45.708 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:45 smithi028 ceph-mon[37984]: 8.2 starting backfill to osd.5 from (0'0,0'0] MAX to 246'6755 2024-02-05T06:41:45.708 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:45 smithi028 ceph-mon[37984]: 8.0 starting backfill to osd.5 from (0'0,194'1860] MIN to 246'6681 2024-02-05T06:41:45.708 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:45 smithi028 ceph-mon[37984]: 8.9 starting backfill to osd.5 from (0'0,0'0] MAX to 246'6612 2024-02-05T06:41:45.709 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:45 smithi028 ceph-mon[37984]: 8.4 starting backfill to osd.5 from (0'0,0'0] MAX to 246'6685 2024-02-05T06:41:45.709 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:45 smithi028 ceph-mon[37984]: 8.d starting backfill to osd.5 from (0'0,194'1820] MIN to 246'6506 2024-02-05T06:41:45.709 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:45 smithi028 ceph-mon[37984]: 8.3 starting backfill to osd.5 from (0'0,194'1924] MIN to 246'6782 2024-02-05T06:41:45.709 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:45 smithi028 ceph-mon[37984]: 8.a starting backfill to osd.5 from (0'0,194'1941] MIN to 246'6684 2024-02-05T06:41:45.709 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:45 smithi028 ceph-mon[37984]: 5.6 scrub starts 2024-02-05T06:41:45.709 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:45 smithi028 ceph-mon[37984]: 5.6 scrub ok 2024-02-05T06:41:45.709 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:45 smithi028 ceph-mon[37984]: osdmap e250: 8 total, 8 up, 8 in 2024-02-05T06:41:45.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:45 smithi028 ceph-mon[29442]: pgmap v396: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 754 B/s rd, 0 B/s wr, 778 op/s; 798/218400 objects misplaced (0.365%); 2.5 MiB/s, 40 objects/s recovering 2024-02-05T06:41:45.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:45 smithi028 ceph-mon[29442]: 6.4 scrub starts 2024-02-05T06:41:45.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:45 smithi028 ceph-mon[29442]: 6.4 scrub ok 2024-02-05T06:41:45.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:45 smithi028 ceph-mon[29442]: 8.8 starting backfill to osd.5 from (0'0,194'1981] MIN to 246'6863 2024-02-05T06:41:45.709 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:45 smithi028 ceph-mon[29442]: 8.2 starting backfill to osd.5 from (0'0,0'0] MAX to 246'6755 2024-02-05T06:41:45.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:45 smithi028 ceph-mon[29442]: 8.0 starting backfill to osd.5 from (0'0,194'1860] MIN to 246'6681 2024-02-05T06:41:45.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:45 smithi028 ceph-mon[29442]: 8.9 starting backfill to osd.5 from (0'0,0'0] MAX to 246'6612 2024-02-05T06:41:45.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:45 smithi028 ceph-mon[29442]: 8.4 starting backfill to osd.5 from (0'0,0'0] MAX to 246'6685 2024-02-05T06:41:45.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:45 smithi028 ceph-mon[29442]: 8.d starting backfill to osd.5 from (0'0,194'1820] MIN to 246'6506 2024-02-05T06:41:45.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:45 smithi028 ceph-mon[29442]: 8.3 starting backfill to osd.5 from (0'0,194'1924] MIN to 246'6782 2024-02-05T06:41:45.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:45 smithi028 ceph-mon[29442]: 8.a starting backfill to osd.5 from (0'0,194'1941] MIN to 246'6684 2024-02-05T06:41:45.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:45 smithi028 ceph-mon[29442]: 5.6 scrub starts 2024-02-05T06:41:45.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:45 smithi028 ceph-mon[29442]: 5.6 scrub ok 2024-02-05T06:41:45.710 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:45 smithi028 ceph-mon[29442]: osdmap e250: 8 total, 8 up, 8 in 2024-02-05T06:41:45.715 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:45.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:45 smithi167 ceph-mon[32774]: pgmap v396: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 4.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 754 B/s rd, 0 B/s wr, 778 op/s; 798/218400 objects misplaced (0.365%); 2.5 MiB/s, 40 objects/s recovering 2024-02-05T06:41:45.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:45 smithi167 ceph-mon[32774]: 6.4 scrub starts 2024-02-05T06:41:45.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:45 smithi167 ceph-mon[32774]: 6.4 scrub ok 2024-02-05T06:41:45.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:45 smithi167 ceph-mon[32774]: 8.8 starting backfill to osd.5 from (0'0,194'1981] MIN to 246'6863 2024-02-05T06:41:45.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:45 smithi167 ceph-mon[32774]: 8.2 starting backfill to osd.5 from (0'0,0'0] MAX to 246'6755 2024-02-05T06:41:45.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:45 smithi167 ceph-mon[32774]: 8.0 starting backfill to osd.5 from (0'0,194'1860] MIN to 246'6681 2024-02-05T06:41:45.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:45 smithi167 ceph-mon[32774]: 8.9 starting backfill to osd.5 from (0'0,0'0] MAX to 246'6612 2024-02-05T06:41:45.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:45 smithi167 ceph-mon[32774]: 8.4 starting backfill to osd.5 from (0'0,0'0] MAX to 246'6685 2024-02-05T06:41:45.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:45 smithi167 ceph-mon[32774]: 8.d starting backfill to osd.5 from (0'0,194'1820] MIN to 246'6506 2024-02-05T06:41:45.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:45 smithi167 ceph-mon[32774]: 8.3 starting backfill to osd.5 from (0'0,194'1924] MIN to 246'6782 2024-02-05T06:41:45.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:45 smithi167 ceph-mon[32774]: 8.a starting backfill to osd.5 from (0'0,194'1941] MIN to 246'6684 2024-02-05T06:41:45.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:45 smithi167 ceph-mon[32774]: 5.6 scrub starts 2024-02-05T06:41:45.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:45 smithi167 ceph-mon[32774]: 5.6 scrub ok 2024-02-05T06:41:45.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:45 smithi167 ceph-mon[32774]: osdmap e250: 8 total, 8 up, 8 in 2024-02-05T06:41:45.742 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:45.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:45.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:45.843 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:45.870 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:45.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:45.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:45.972 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:46.000 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:46.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:46.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:46.102 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:46.145 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:46.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:46.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:46.247 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:46.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:46.317 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:46.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:46.418 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:46.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:46.469 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:46.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:46.570 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:46.608 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:46.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:46.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:46.710 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:46.752 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:46.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:46.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:46.853 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:46.886 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:46.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:46.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:46.935 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:41:46.935 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:41:46.988 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:47.029 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:47.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:47.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:47.131 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:47.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:47 smithi028 ceph-mon[29442]: osdmap e251: 8 total, 8 up, 8 in 2024-02-05T06:41:47.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:47 smithi028 ceph-mon[29442]: pgmap v400: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 155 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 101 op/s; 9008/217227 objects misplaced (4.147%); 10 MiB/s, 169 objects/s recovering 2024-02-05T06:41:47.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:47 smithi028 ceph-mon[29442]: 6.12 scrub starts 2024-02-05T06:41:47.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:47 smithi028 ceph-mon[29442]: 6.12 scrub ok 2024-02-05T06:41:47.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:47 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3590321939' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:41:47.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:47 smithi028 ceph-mon[37984]: osdmap e251: 8 total, 8 up, 8 in 2024-02-05T06:41:47.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:47 smithi028 ceph-mon[37984]: pgmap v400: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 155 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 101 op/s; 9008/217227 objects misplaced (4.147%); 10 MiB/s, 169 objects/s recovering 2024-02-05T06:41:47.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:47 smithi028 ceph-mon[37984]: 6.12 scrub starts 2024-02-05T06:41:47.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:47 smithi028 ceph-mon[37984]: 6.12 scrub ok 2024-02-05T06:41:47.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:47 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3590321939' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:41:47.162 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:47.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:47.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:47.263 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:47.308 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:47.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:47.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:47.411 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:47.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:47.453 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:47.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:47.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:47 smithi167 ceph-mon[32774]: osdmap e251: 8 total, 8 up, 8 in 2024-02-05T06:41:47.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:47 smithi167 ceph-mon[32774]: pgmap v400: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+remapped, 155 active+clean; 4.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 101 op/s; 9008/217227 objects misplaced (4.147%); 10 MiB/s, 169 objects/s recovering 2024-02-05T06:41:47.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:47 smithi167 ceph-mon[32774]: 6.12 scrub starts 2024-02-05T06:41:47.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:47 smithi167 ceph-mon[32774]: 6.12 scrub ok 2024-02-05T06:41:47.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:47 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3590321939' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:41:47.555 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:47.588 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:47.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:47.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:47.690 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:47.733 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:47.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:47.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:47.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:47.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:47.879 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:47.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:47.980 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:48.006 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:48.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:48.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:48.108 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:48.142 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:48.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:48.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:48.244 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:48.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:48.286 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:48.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:48.387 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:48.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:48.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:48.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:48 smithi028 ceph-mon[29442]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:41:48.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:48 smithi028 ceph-mon[29442]: 5.18 scrub starts 2024-02-05T06:41:48.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:48 smithi028 ceph-mon[29442]: 5.18 scrub ok 2024-02-05T06:41:48.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:48 smithi028 ceph-mon[37984]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:41:48.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:48 smithi028 ceph-mon[37984]: 5.18 scrub starts 2024-02-05T06:41:48.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:48 smithi028 ceph-mon[37984]: 5.18 scrub ok 2024-02-05T06:41:48.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:48 smithi167 ceph-mon[32774]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:41:48.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:48 smithi167 ceph-mon[32774]: 5.18 scrub starts 2024-02-05T06:41:48.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:48 smithi167 ceph-mon[32774]: 5.18 scrub ok 2024-02-05T06:41:48.526 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:48.570 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:48.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:48.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:48.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:48.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:48.714 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:48.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:48.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:48.856 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:48.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:48.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:48.958 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:49.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:49.000 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:49.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:49.101 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:49.134 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:49.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:49.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:49.236 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:49.255 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 4, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T06:41:49.255 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T06:41:49.255 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-05T06:41:49.255 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-05T06:41:49.270 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:49.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:49.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:49.371 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:49.423 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:49.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:49.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:49.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:49 smithi028 ceph-mon[29442]: pgmap v401: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 150 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 454 op/s; 33245/211710 objects misplaced (15.703%); 34 MiB/s, 557 objects/s recovering 2024-02-05T06:41:49.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:49 smithi028 ceph-mon[29442]: 4.1b scrub starts 2024-02-05T06:41:49.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:49 smithi028 ceph-mon[29442]: 4.1b scrub ok 2024-02-05T06:41:49.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:49 smithi028 ceph-mon[29442]: 5.4 scrub starts 2024-02-05T06:41:49.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:49 smithi028 ceph-mon[29442]: 5.4 scrub ok 2024-02-05T06:41:49.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:49 smithi028 ceph-mon[37984]: pgmap v401: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 150 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 454 op/s; 33245/211710 objects misplaced (15.703%); 34 MiB/s, 557 objects/s recovering 2024-02-05T06:41:49.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:49 smithi028 ceph-mon[37984]: 4.1b scrub starts 2024-02-05T06:41:49.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:49 smithi028 ceph-mon[37984]: 4.1b scrub ok 2024-02-05T06:41:49.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:49 smithi028 ceph-mon[37984]: 5.4 scrub starts 2024-02-05T06:41:49.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:49 smithi028 ceph-mon[37984]: 5.4 scrub ok 2024-02-05T06:41:49.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:49 smithi167 ceph-mon[32774]: pgmap v401: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 150 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 454 op/s; 33245/211710 objects misplaced (15.703%); 34 MiB/s, 557 objects/s recovering 2024-02-05T06:41:49.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:49 smithi167 ceph-mon[32774]: 4.1b scrub starts 2024-02-05T06:41:49.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:49 smithi167 ceph-mon[32774]: 4.1b scrub ok 2024-02-05T06:41:49.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:49 smithi167 ceph-mon[32774]: 5.4 scrub starts 2024-02-05T06:41:49.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:49 smithi167 ceph-mon[32774]: 5.4 scrub ok 2024-02-05T06:41:49.525 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:49.557 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:49.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:49.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:49.659 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:49.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:49.705 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:49.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:49.806 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:49.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:49.844 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:49.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:49.945 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:49.984 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:49.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:49.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:50.086 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:50.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:50.124 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:50.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:50.225 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:50.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:50.258 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:50.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:50.359 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:50.400 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:50.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:50.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:50.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:50 smithi028 ceph-mon[29442]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-02-05T06:41:50.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:50 smithi028 ceph-mon[29442]: 5.1a scrub starts 2024-02-05T06:41:50.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:50 smithi028 ceph-mon[29442]: 5.1a scrub ok 2024-02-05T06:41:50.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:50 smithi028 ceph-mon[29442]: osdmap e252: 8 total, 8 up, 8 in 2024-02-05T06:41:50.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:50 smithi028 ceph-mon[37984]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-02-05T06:41:50.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:50 smithi028 ceph-mon[37984]: 5.1a scrub starts 2024-02-05T06:41:50.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:50 smithi028 ceph-mon[37984]: 5.1a scrub ok 2024-02-05T06:41:50.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:50 smithi028 ceph-mon[37984]: osdmap e252: 8 total, 8 up, 8 in 2024-02-05T06:41:50.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:50 smithi167 ceph-mon[32774]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-02-05T06:41:50.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:50 smithi167 ceph-mon[32774]: 5.1a scrub starts 2024-02-05T06:41:50.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:50 smithi167 ceph-mon[32774]: 5.1a scrub ok 2024-02-05T06:41:50.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:50 smithi167 ceph-mon[32774]: osdmap e252: 8 total, 8 up, 8 in 2024-02-05T06:41:50.502 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:50.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:50.539 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:50.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:50.640 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:50.683 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:50.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:50.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:50.785 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:50.814 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:50.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:50.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:50.916 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:50.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:50.956 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:50.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:51.057 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:51.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:51.083 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:51.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:51.184 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:51.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:51.226 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:51.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:51.327 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:51.359 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:51.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:51.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:51.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:51 smithi028 ceph-mon[37984]: pgmap v403: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 150 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 378 op/s; 33245/211710 objects misplaced (15.703%); 29 MiB/s, 464 objects/s recovering 2024-02-05T06:41:51.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:51 smithi028 ceph-mon[37984]: 1.9 deep-scrub starts 2024-02-05T06:41:51.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:51 smithi028 ceph-mon[37984]: 1.9 deep-scrub ok 2024-02-05T06:41:51.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:51 smithi028 ceph-mon[29442]: pgmap v403: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 150 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 378 op/s; 33245/211710 objects misplaced (15.703%); 29 MiB/s, 464 objects/s recovering 2024-02-05T06:41:51.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:51 smithi028 ceph-mon[29442]: 1.9 deep-scrub starts 2024-02-05T06:41:51.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:51 smithi028 ceph-mon[29442]: 1.9 deep-scrub ok 2024-02-05T06:41:51.460 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:51.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:51 smithi167 ceph-mon[32774]: pgmap v403: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 150 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 378 op/s; 33245/211710 objects misplaced (15.703%); 29 MiB/s, 464 objects/s recovering 2024-02-05T06:41:51.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:51 smithi167 ceph-mon[32774]: 1.9 deep-scrub starts 2024-02-05T06:41:51.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:51 smithi167 ceph-mon[32774]: 1.9 deep-scrub ok 2024-02-05T06:41:51.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:51.484 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:51.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:51.585 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:51.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:51.619 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:51.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:51.720 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:51.762 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:51.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:51.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:51.864 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:51.907 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:51.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:51.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:51.943 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:41:51.943 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:41:51.943 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:41:51.943 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:41:51.943 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:41:51.943 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:41:51.944 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:41:51.944 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:41:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:41:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:41:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:41:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:41:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:41:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:41:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:41:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:41:52.009 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:52.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:52.060 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:52.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:52.161 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:52.205 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:52.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:52.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:52.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:52.349 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:52.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:52.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:52.450 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:52.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:52.487 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:52.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:52.588 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:52.617 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:52.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:52.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:52.719 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:52.748 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:52.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:52.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:52.850 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:52.880 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:52.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:52.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:52.947 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:41:52 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:41:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:41:52.982 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:53.013 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:53.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:53.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:53.115 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:53.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:53.156 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:53.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:53.232 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:41:52 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:41:53.257 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:53.292 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:53.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:53.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:53.394 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:53.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:53.436 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:53.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:53.537 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:53.568 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:53.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:53.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:53.649 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:53 smithi028 ceph-mon[37984]: pgmap v404: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 150 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 955 op/s; 28367/197523 objects misplaced (14.361%); 47 MiB/s, 754 objects/s recovering 2024-02-05T06:41:53.649 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:53 smithi028 ceph-mon[37984]: 4.9 scrub starts 2024-02-05T06:41:53.649 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:53 smithi028 ceph-mon[37984]: 4.9 scrub ok 2024-02-05T06:41:53.649 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:53 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:41:53.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:53 smithi028 ceph-mon[29442]: pgmap v404: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 150 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 955 op/s; 28367/197523 objects misplaced (14.361%); 47 MiB/s, 754 objects/s recovering 2024-02-05T06:41:53.649 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:53 smithi028 ceph-mon[29442]: 4.9 scrub starts 2024-02-05T06:41:53.650 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:53 smithi028 ceph-mon[29442]: 4.9 scrub ok 2024-02-05T06:41:53.650 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:53 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:41:53.670 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:53.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:53.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:53.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:53 smithi167 ceph-mon[32774]: pgmap v404: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 150 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 955 op/s; 28367/197523 objects misplaced (14.361%); 47 MiB/s, 754 objects/s recovering 2024-02-05T06:41:53.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:53 smithi167 ceph-mon[32774]: 4.9 scrub starts 2024-02-05T06:41:53.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:53 smithi167 ceph-mon[32774]: 4.9 scrub ok 2024-02-05T06:41:53.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:53 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:41:53.820 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:53.849 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:53.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:53.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:53.951 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:53.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:53.987 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:53.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:54.089 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:54.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:54.123 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:54.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:54.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:54.268 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:54.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:54.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:54.370 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:54.419 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:54.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:54.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:54.521 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:54.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-05T06:41:54.554 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:54.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:54.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:54.636 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:41:54.656 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:54.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:54 smithi028 ceph-mon[37984]: 3.15 scrub starts 2024-02-05T06:41:54.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:54 smithi028 ceph-mon[37984]: 3.15 scrub ok 2024-02-05T06:41:54.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:54 smithi028 ceph-mon[29442]: 3.15 scrub starts 2024-02-05T06:41:54.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:54 smithi028 ceph-mon[29442]: 3.15 scrub ok 2024-02-05T06:41:54.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:54 smithi167 ceph-mon[32774]: 3.15 scrub starts 2024-02-05T06:41:54.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:54 smithi167 ceph-mon[32774]: 3.15 scrub ok 2024-02-05T06:41:54.738 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:54.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:54.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:54.840 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:54.900 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:54.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:54.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:55.002 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:55.035 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:55.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:55.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:55.067 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-02-05T06:41:55.136 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:55.177 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:55.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:55.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:55.279 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:55.319 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:55.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:55.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:55.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:55.464 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:55.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:55.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:55.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:55.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:55.610 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:55.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:55.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:55 smithi028 ceph-mon[37984]: pgmap v405: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 150 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 839 op/s; 28367/197523 objects misplaced (14.361%); 41 MiB/s, 662 objects/s recovering 2024-02-05T06:41:55.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:55 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2679582310' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:41:55.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:55 smithi028 ceph-mon[37984]: from='client.45131 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:41:55.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:55 smithi028 ceph-mon[37984]: 4.d scrub starts 2024-02-05T06:41:55.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:55 smithi028 ceph-mon[37984]: 4.d scrub ok 2024-02-05T06:41:55.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:55 smithi028 ceph-mon[37984]: 3.f scrub starts 2024-02-05T06:41:55.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:55 smithi028 ceph-mon[37984]: 3.f scrub ok 2024-02-05T06:41:55.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:55 smithi028 ceph-mon[37984]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:41:55.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:55 smithi028 ceph-mon[37984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:41:55.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:55 smithi028 ceph-mon[37984]: from='client.45131 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:41:55.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:55 smithi028 ceph-mon[37984]: osdmap e253: 8 total, 8 up, 8 in 2024-02-05T06:41:55.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:55 smithi028 ceph-mon[29442]: pgmap v405: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 150 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 839 op/s; 28367/197523 objects misplaced (14.361%); 41 MiB/s, 662 objects/s recovering 2024-02-05T06:41:55.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:55 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2679582310' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:41:55.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:55 smithi028 ceph-mon[29442]: from='client.45131 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:41:55.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:55 smithi028 ceph-mon[29442]: 4.d scrub starts 2024-02-05T06:41:55.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:55 smithi028 ceph-mon[29442]: 4.d scrub ok 2024-02-05T06:41:55.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:55 smithi028 ceph-mon[29442]: 3.f scrub starts 2024-02-05T06:41:55.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:55 smithi028 ceph-mon[29442]: 3.f scrub ok 2024-02-05T06:41:55.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:55 smithi028 ceph-mon[29442]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:41:55.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:55 smithi028 ceph-mon[29442]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:41:55.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:55 smithi028 ceph-mon[29442]: from='client.45131 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:41:55.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:55 smithi028 ceph-mon[29442]: osdmap e253: 8 total, 8 up, 8 in 2024-02-05T06:41:55.712 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:55.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:55 smithi167 ceph-mon[32774]: pgmap v405: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 150 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 0 B/s wr, 839 op/s; 28367/197523 objects misplaced (14.361%); 41 MiB/s, 662 objects/s recovering 2024-02-05T06:41:55.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:55 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2679582310' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:41:55.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:55 smithi167 ceph-mon[32774]: from='client.45131 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:41:55.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:55 smithi167 ceph-mon[32774]: 4.d scrub starts 2024-02-05T06:41:55.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:55 smithi167 ceph-mon[32774]: 4.d scrub ok 2024-02-05T06:41:55.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:55 smithi167 ceph-mon[32774]: 3.f scrub starts 2024-02-05T06:41:55.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:55 smithi167 ceph-mon[32774]: 3.f scrub ok 2024-02-05T06:41:55.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:55 smithi167 ceph-mon[32774]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:41:55.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:55 smithi167 ceph-mon[32774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:41:55.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:55 smithi167 ceph-mon[32774]: from='client.45131 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:41:55.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:55 smithi167 ceph-mon[32774]: osdmap e253: 8 total, 8 up, 8 in 2024-02-05T06:41:55.745 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:55.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:55.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:55.847 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:55.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:55.880 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:55.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:55.981 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:56.014 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:56.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:56.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:56.116 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:56.148 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:56.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:56.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:56.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:56.277 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:56.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:56.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:56.379 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:56.415 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:56.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:56.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:56.436 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:41:56.517 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:56.547 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:56.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:56.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:56.649 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:56.662 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:41:56.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:56.731 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:56.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:41:56.832 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:56.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:56.883 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:56.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:56.984 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:57.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:57.027 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:57.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:57.128 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:57.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:57.184 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:57.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:57.286 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:41:57.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:57.319 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:57.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:41:57.420 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:57.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:57.454 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:57.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:57.555 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:57.584 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:57.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:57.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:57.686 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:57.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:57 smithi028 ceph-mon[37984]: pgmap v407: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 150 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 699 op/s; 26448/194862 objects misplaced (13.573%); 33 MiB/s, 533 objects/s recovering 2024-02-05T06:41:57.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:57 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:41:57.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:57 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:41:57.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:57 smithi028 ceph-mon[29442]: pgmap v407: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 150 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 699 op/s; 26448/194862 objects misplaced (13.573%); 33 MiB/s, 533 objects/s recovering 2024-02-05T06:41:57.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:57 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:41:57.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:57 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:41:57.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:57.729 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:57.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:57.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:57 smithi167 ceph-mon[32774]: pgmap v407: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 150 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 699 op/s; 26448/194862 objects misplaced (13.573%); 33 MiB/s, 533 objects/s recovering 2024-02-05T06:41:57.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:57 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:41:57.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:57 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:41:57.830 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:57.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:57.856 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:57.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:57.957 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:57.980 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:41:57.988 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:57.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:57.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:58.090 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:58.121 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:58.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:58.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:58.223 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:58.254 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:58.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:58.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:58.300 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T06:41:58.342 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:41:58.342 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:41:58.342 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 254, 2024-02-05T06:41:58.342 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "9801e956-c3ef-11ee-95b6-87774f69a715", 2024-02-05T06:41:58.342 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-02-05T06:27:59.919226+0000", 2024-02-05T06:41:58.342 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-02-05T06:41:57.415443+0000", 2024-02-05T06:41:58.342 INFO:teuthology.orchestra.run.smithi028.stdout: "last_up_change": "2024-02-05T06:33:15.833573+0000", 2024-02-05T06:41:58.342 INFO:teuthology.orchestra.run.smithi028.stdout: "last_in_change": "2024-02-05T06:41:42.383087+0000", 2024-02-05T06:41:58.343 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-05T06:41:58.343 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_num": 5801984, 2024-02-05T06:41:58.343 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_set": [ 2024-02-05T06:41:58.343 INFO:teuthology.orchestra.run.smithi028.stdout: "noscrub", 2024-02-05T06:41:58.343 INFO:teuthology.orchestra.run.smithi028.stdout: "pglog_hardlimit", 2024-02-05T06:41:58.343 INFO:teuthology.orchestra.run.smithi028.stdout: "purged_snapdirs", 2024-02-05T06:41:58.343 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_deletes", 2024-02-05T06:41:58.343 INFO:teuthology.orchestra.run.smithi028.stdout: "sortbitwise" 2024-02-05T06:41:58.343 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:41:58.343 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_version": 18, 2024-02-05T06:41:58.343 INFO:teuthology.orchestra.run.smithi028.stdout: "full_ratio": 0.94999998807907104, 2024-02-05T06:41:58.343 INFO:teuthology.orchestra.run.smithi028.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-05T06:41:58.343 INFO:teuthology.orchestra.run.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-05T06:41:58.343 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_snapshot": "", 2024-02-05T06:41:58.343 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_max": 8, 2024-02-05T06:41:58.344 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd": 8, 2024-02-05T06:41:58.344 INFO:teuthology.orchestra.run.smithi028.stdout: "require_min_compat_client": "luminous", 2024-02-05T06:41:58.344 INFO:teuthology.orchestra.run.smithi028.stdout: "min_compat_client": "luminous", 2024-02-05T06:41:58.344 INFO:teuthology.orchestra.run.smithi028.stdout: "require_osd_release": "squid", 2024-02-05T06:41:58.344 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_crimson": false, 2024-02-05T06:41:58.344 INFO:teuthology.orchestra.run.smithi028.stdout: "pools": [ 2024-02-05T06:41:58.344 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.344 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 1, 2024-02-05T06:41:58.344 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".mgr", 2024-02-05T06:41:58.344 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:31:10.441424+0000", 2024-02-05T06:41:58.344 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-02-05T06:41:58.344 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-02-05T06:41:58.344 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:41:58.345 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:41:58.345 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:41:58.345 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:41:58.345 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:41:58.345 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:41:58.345 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:41:58.345 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:41:58.345 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:41:58.345 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:41:58.345 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 11, 2024-02-05T06:41:58.345 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 11, 2024-02-05T06:41:58.345 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 11, 2024-02-05T06:41:58.345 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 11, 2024-02-05T06:41:58.345 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 11, 2024-02-05T06:41:58.346 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:41:58.346 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:41:58.346 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:41:58.346 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:41:58.346 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:41:58.346 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:41:58.346 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:41:58.346 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.346 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "142", 2024-02-05T06:41:58.346 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:41:58.346 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:41:58.346 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "129", 2024-02-05T06:41:58.346 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:41:58.346 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:41:58.347 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-02-05T06:41:58.347 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-02-05T06:41:58.347 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:41:58.347 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:41:58.347 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:41:58.347 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:41:58.347 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:41:58.347 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:41:58.347 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:41:58.347 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:41:58.347 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:41:58.347 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:41:58.347 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:41:58.347 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:41:58.347 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:41:58.348 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:41:58.348 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:41:58.348 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:41:58.348 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:41:58.348 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:41:58.348 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:41:58.348 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.348 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:41:58.348 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:41:58.348 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:41:58.348 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:41:58.348 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:41:58.349 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:41:58.349 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:41:58.349 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:41:58.349 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:41:58.349 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:41:58.349 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:41:58.349 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-02-05T06:41:58.349 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_max": 32, 2024-02-05T06:41:58.349 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_min": 1 2024-02-05T06:41:58.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.349 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:41:58.349 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": {} 2024-02-05T06:41:58.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.350 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:41:58.350 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:41:58.350 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.1800000667572021, 2024-02-05T06:41:58.350 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.1800000667572021, 2024-02-05T06:41:58.350 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-02-05T06:41:58.350 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2.1800000667572021, 2024-02-05T06:41:58.350 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2.1800000667572021, 2024-02-05T06:41:58.350 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-02-05T06:41:58.350 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-02-05T06:41:58.350 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-02-05T06:41:58.350 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.350 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.350 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.350 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 2, 2024-02-05T06:41:58.351 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "datapool", 2024-02-05T06:41:58.351 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:33:27.397416+0000", 2024-02-05T06:41:58.351 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-02-05T06:41:58.351 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-05T06:41:58.351 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:41:58.351 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:41:58.351 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:41:58.351 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:41:58.351 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:41:58.351 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:41:58.351 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:41:58.351 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:41:58.351 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:41:58.352 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:41:58.352 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 3, 2024-02-05T06:41:58.352 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 3, 2024-02-05T06:41:58.352 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 3, 2024-02-05T06:41:58.352 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 3, 2024-02-05T06:41:58.352 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 3, 2024-02-05T06:41:58.352 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:41:58.352 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:41:58.352 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:41:58.352 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:41:58.352 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:41:58.352 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:41:58.352 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:41:58.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.352 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "69", 2024-02-05T06:41:58.353 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:41:58.353 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:41:58.353 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T06:41:58.353 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:41:58.353 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:41:58.353 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 2, 2024-02-05T06:41:58.353 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 69, 2024-02-05T06:41:58.353 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:41:58.353 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:41:58.353 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:41:58.353 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:41:58.353 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:41:58.353 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:41:58.353 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:41:58.354 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:41:58.354 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:41:58.354 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:41:58.354 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:41:58.354 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:41:58.354 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:41:58.354 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:41:58.354 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:41:58.354 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:41:58.354 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:41:58.354 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:41:58.354 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:41:58.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.354 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:41:58.354 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:41:58.354 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:41:58.355 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:41:58.355 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:41:58.355 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:41:58.355 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:41:58.355 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:41:58.355 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:41:58.355 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:41:58.355 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:41:58.355 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-02-05T06:41:58.355 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:41:58.355 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd": {} 2024-02-05T06:41:58.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.355 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:41:58.355 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:41:58.355 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.6500000953674316, 2024-02-05T06:41:58.356 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.6500000953674316, 2024-02-05T06:41:58.356 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T06:41:58.356 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2.3299999237060547, 2024-02-05T06:41:58.356 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2.3299999237060547, 2024-02-05T06:41:58.356 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-02-05T06:41:58.356 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-02-05T06:41:58.356 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-02-05T06:41:58.356 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.356 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.356 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.356 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 3, 2024-02-05T06:41:58.356 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".rgw.root", 2024-02-05T06:41:58.356 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:33:28.090506+0000", 2024-02-05T06:41:58.357 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-02-05T06:41:58.357 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-02-05T06:41:58.357 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:41:58.357 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:41:58.357 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:41:58.357 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:41:58.357 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:41:58.357 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:41:58.357 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:41:58.357 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:41:58.357 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:41:58.357 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:41:58.357 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-02-05T06:41:58.357 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-02-05T06:41:58.357 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-02-05T06:41:58.357 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-02-05T06:41:58.358 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-02-05T06:41:58.358 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:41:58.358 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:41:58.358 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:41:58.358 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:41:58.358 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:41:58.358 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:41:58.358 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:41:58.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.358 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "67", 2024-02-05T06:41:58.358 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:41:58.358 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:41:58.358 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T06:41:58.358 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:41:58.358 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:41:58.359 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-02-05T06:41:58.359 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-02-05T06:41:58.359 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:41:58.359 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:41:58.359 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:41:58.359 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:41:58.359 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:41:58.359 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:41:58.359 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:41:58.359 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:41:58.359 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:41:58.359 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:41:58.359 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:41:58.359 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:41:58.360 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:41:58.360 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:41:58.360 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:41:58.360 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:41:58.360 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:41:58.360 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:41:58.360 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:41:58.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.360 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:41:58.360 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:41:58.360 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:41:58.360 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:41:58.360 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:41:58.360 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:41:58.361 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:41:58.361 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:41:58.361 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:41:58.361 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:41:58.361 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:41:58.361 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-02-05T06:41:58.361 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:41:58.361 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-02-05T06:41:58.361 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.361 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:41:58.361 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:41:58.361 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.5, 2024-02-05T06:41:58.361 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.5, 2024-02-05T06:41:58.361 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-02-05T06:41:58.362 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.5, 2024-02-05T06:41:58.362 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.5, 2024-02-05T06:41:58.362 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-02-05T06:41:58.362 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-02-05T06:41:58.362 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-02-05T06:41:58.362 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.362 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 4, 2024-02-05T06:41:58.362 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.log", 2024-02-05T06:41:58.362 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:33:29.866249+0000", 2024-02-05T06:41:58.362 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-02-05T06:41:58.362 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-02-05T06:41:58.362 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:41:58.362 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:41:58.363 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:41:58.363 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:41:58.363 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:41:58.363 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:41:58.363 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:41:58.363 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:41:58.363 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:41:58.363 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:41:58.363 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-02-05T06:41:58.363 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-02-05T06:41:58.363 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-02-05T06:41:58.363 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-02-05T06:41:58.363 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-02-05T06:41:58.363 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:41:58.363 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:41:58.364 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:41:58.364 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:41:58.364 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:41:58.364 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:41:58.364 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:41:58.364 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.364 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "69", 2024-02-05T06:41:58.364 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:41:58.364 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:41:58.364 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T06:41:58.364 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:41:58.364 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:41:58.364 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-02-05T06:41:58.364 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-02-05T06:41:58.364 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:41:58.365 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:41:58.365 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:41:58.365 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:41:58.365 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:41:58.365 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:41:58.365 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:41:58.365 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:41:58.365 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:41:58.365 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:41:58.365 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:41:58.365 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:41:58.365 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:41:58.365 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:41:58.365 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:41:58.366 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:41:58.366 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:41:58.366 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:41:58.366 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:41:58.366 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.366 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:41:58.366 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:41:58.366 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:41:58.366 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:41:58.366 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:41:58.366 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:41:58.366 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:41:58.366 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:41:58.366 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:41:58.366 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:41:58.366 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:41:58.367 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-02-05T06:41:58.367 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:41:58.367 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-02-05T06:41:58.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.367 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:41:58.367 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:41:58.367 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.25, 2024-02-05T06:41:58.367 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.25, 2024-02-05T06:41:58.367 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-02-05T06:41:58.367 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2.25, 2024-02-05T06:41:58.367 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2.25, 2024-02-05T06:41:58.367 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-02-05T06:41:58.367 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-02-05T06:41:58.367 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-02-05T06:41:58.367 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.368 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.368 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 5, 2024-02-05T06:41:58.368 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.control", 2024-02-05T06:41:58.368 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:33:31.826006+0000", 2024-02-05T06:41:58.368 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-02-05T06:41:58.368 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-02-05T06:41:58.368 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:41:58.368 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:41:58.368 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:41:58.368 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:41:58.368 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:41:58.368 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:41:58.368 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:41:58.368 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:41:58.369 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:41:58.369 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:41:58.369 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-02-05T06:41:58.369 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-02-05T06:41:58.369 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-02-05T06:41:58.369 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-02-05T06:41:58.369 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-02-05T06:41:58.369 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:41:58.369 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:41:58.369 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:41:58.369 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:41:58.369 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:41:58.369 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:41:58.369 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:41:58.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.370 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "71", 2024-02-05T06:41:58.370 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:41:58.370 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:41:58.370 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T06:41:58.370 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:41:58.370 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:41:58.370 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-02-05T06:41:58.370 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-02-05T06:41:58.370 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:41:58.370 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:41:58.370 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:41:58.370 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:41:58.370 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:41:58.370 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:41:58.371 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:41:58.371 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:41:58.371 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:41:58.371 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:41:58.371 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:41:58.371 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:41:58.371 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:41:58.371 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:41:58.371 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:41:58.371 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:41:58.371 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:41:58.371 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:41:58.371 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:41:58.371 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.371 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:41:58.372 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:41:58.372 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:41:58.372 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:41:58.372 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:41:58.372 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:41:58.372 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:41:58.372 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:41:58.372 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:41:58.372 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:41:58.372 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:41:58.372 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-02-05T06:41:58.372 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:41:58.372 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-02-05T06:41:58.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.373 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:41:58.373 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:41:58.373 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.25, 2024-02-05T06:41:58.373 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.25, 2024-02-05T06:41:58.373 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-02-05T06:41:58.373 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.25, 2024-02-05T06:41:58.373 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.25, 2024-02-05T06:41:58.373 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-02-05T06:41:58.373 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-02-05T06:41:58.373 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-02-05T06:41:58.373 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.373 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.373 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 6, 2024-02-05T06:41:58.374 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.meta", 2024-02-05T06:41:58.374 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:33:33.890835+0000", 2024-02-05T06:41:58.374 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-02-05T06:41:58.374 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:41:58.375 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-02-05T06:41:58.375 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:41:58.375 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:41:58.375 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:41:58.375 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:41:58.375 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:41:58.375 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:41:58.376 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:41:58.376 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:41:58.376 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:41:58.376 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:41:58.376 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-02-05T06:41:58.376 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-02-05T06:41:58.376 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-02-05T06:41:58.376 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-02-05T06:41:58.376 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-02-05T06:41:58.376 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:41:58.376 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:41:58.376 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:41:58.376 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:41:58.376 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:41:58.376 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:41:58.377 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:41:58.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.377 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "73", 2024-02-05T06:41:58.377 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:41:58.377 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:41:58.377 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T06:41:58.377 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:41:58.377 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:41:58.377 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-02-05T06:41:58.377 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-02-05T06:41:58.377 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:41:58.377 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:41:58.377 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:41:58.377 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:41:58.378 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:41:58.378 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:41:58.378 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:41:58.378 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:41:58.378 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:41:58.378 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:41:58.378 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:41:58.378 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:41:58.378 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:41:58.378 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:41:58.378 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:41:58.378 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:41:58.378 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:41:58.378 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:41:58.379 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:41:58.379 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.379 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:41:58.379 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:41:58.379 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:41:58.379 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:41:58.379 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:41:58.379 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:41:58.379 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:41:58.379 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:41:58.379 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:41:58.379 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:41:58.379 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:41:58.379 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-02-05T06:41:58.379 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_bias": 4 2024-02-05T06:41:58.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.380 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:41:58.380 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-02-05T06:41:58.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.380 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:41:58.380 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:41:58.380 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.5, 2024-02-05T06:41:58.380 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.5, 2024-02-05T06:41:58.380 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-02-05T06:41:58.380 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.5, 2024-02-05T06:41:58.380 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.5, 2024-02-05T06:41:58.380 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-02-05T06:41:58.380 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-02-05T06:41:58.380 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-02-05T06:41:58.381 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.381 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.381 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.381 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 8, 2024-02-05T06:41:58.381 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "unique_pool_1", 2024-02-05T06:41:58.381 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:39:40.813792+0000", 2024-02-05T06:41:58.381 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-02-05T06:41:58.381 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-02-05T06:41:58.381 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:41:58.381 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:41:58.381 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:41:58.381 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:41:58.381 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:41:58.381 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:41:58.381 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:41:58.382 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:41:58.382 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:41:58.382 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:41:58.382 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 16, 2024-02-05T06:41:58.382 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 16, 2024-02-05T06:41:58.382 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 16, 2024-02-05T06:41:58.382 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 16, 2024-02-05T06:41:58.382 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 16, 2024-02-05T06:41:58.382 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:41:58.382 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:41:58.382 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:41:58.382 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:41:58.382 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:41:58.382 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:41:58.383 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:41:58.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.383 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "242", 2024-02-05T06:41:58.383 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:41:58.383 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:41:58.383 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T06:41:58.383 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:41:58.383 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:41:58.383 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-02-05T06:41:58.383 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-02-05T06:41:58.383 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:41:58.383 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:41:58.383 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:41:58.383 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:41:58.383 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:41:58.384 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:41:58.384 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:41:58.384 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:41:58.384 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:41:58.384 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:41:58.384 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:41:58.384 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:41:58.384 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:41:58.384 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:41:58.384 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:41:58.384 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:41:58.384 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:41:58.384 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:41:58.384 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:41:58.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.385 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:41:58.385 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:41:58.385 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:41:58.385 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:41:58.385 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:41:58.385 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:41:58.385 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:41:58.385 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:41:58.385 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:41:58.385 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:41:58.385 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:41:58.385 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-02-05T06:41:58.385 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:41:58.385 INFO:teuthology.orchestra.run.smithi028.stdout: "rados": {} 2024-02-05T06:41:58.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.386 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:41:58.386 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:41:58.386 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2, 2024-02-05T06:41:58.386 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2, 2024-02-05T06:41:58.386 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-02-05T06:41:58.386 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-02-05T06:41:58.386 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-02-05T06:41:58.386 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-02-05T06:41:58.386 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-02-05T06:41:58.386 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-02-05T06:41:58.386 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.386 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.386 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:41:58.386 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-02-05T06:41:58.386 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.386 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-02-05T06:41:58.386 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "c19fdd5b-09f6-45d4-a519-0a938cf1f2a6", 2024-02-05T06:41:58.387 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:41:58.387 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:41:58.387 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:41:58.387 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:41:58.387 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:41:58.387 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:41:58.387 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 8, 2024-02-05T06:41:58.387 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 250, 2024-02-05T06:41:58.387 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:41:58.387 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:41:58.387 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:41:58.387 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:41:58.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.387 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:41:58.387 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-02-05T06:41:58.388 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 301279106 2024-02-05T06:41:58.388 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.388 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.388 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:41:58.388 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:41:58.388 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.388 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:41:58.388 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6802", 2024-02-05T06:41:58.388 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 301279106 2024-02-05T06:41:58.388 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.388 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.388 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:41:58.389 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:41:58.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.389 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:41:58.389 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6804", 2024-02-05T06:41:58.389 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 301279106 2024-02-05T06:41:58.389 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.389 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.389 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.389 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:41:58.389 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:41:58.389 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.389 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:41:58.389 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6803", 2024-02-05T06:41:58.389 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 301279106 2024-02-05T06:41:58.389 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.389 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.390 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6801/301279106", 2024-02-05T06:41:58.390 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6802/301279106", 2024-02-05T06:41:58.390 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6804/301279106", 2024-02-05T06:41:58.390 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6803/301279106", 2024-02-05T06:41:58.390 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:41:58.390 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:41:58.390 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:41:58.390 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.390 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.390 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-02-05T06:41:58.390 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "52245f6e-eb96-4d0d-985b-dbb6fd3ce8de", 2024-02-05T06:41:58.390 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:41:58.390 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:41:58.391 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:41:58.391 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:41:58.391 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:41:58.391 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:41:58.391 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-02-05T06:41:58.391 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 248, 2024-02-05T06:41:58.391 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:41:58.391 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:41:58.391 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:41:58.391 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:41:58.391 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.391 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:41:58.391 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6805", 2024-02-05T06:41:58.391 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1795224649 2024-02-05T06:41:58.391 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.392 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.392 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:41:58.392 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:41:58.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.392 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:41:58.392 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6806", 2024-02-05T06:41:58.392 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1795224649 2024-02-05T06:41:58.392 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.392 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.392 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:41:58.392 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:41:58.392 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.393 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:41:58.393 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6808", 2024-02-05T06:41:58.393 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1795224649 2024-02-05T06:41:58.393 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.393 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.393 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:41:58.393 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:41:58.393 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.393 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:41:58.393 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6807", 2024-02-05T06:41:58.393 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1795224649 2024-02-05T06:41:58.393 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.393 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.394 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6805/1795224649", 2024-02-05T06:41:58.394 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6806/1795224649", 2024-02-05T06:41:58.394 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6808/1795224649", 2024-02-05T06:41:58.394 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6807/1795224649", 2024-02-05T06:41:58.394 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:41:58.394 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:41:58.394 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:41:58.394 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.394 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.394 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-02-05T06:41:58.394 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "1e18b630-afdf-4de9-a1a2-37bdfdd01c67", 2024-02-05T06:41:58.394 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:41:58.394 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:41:58.395 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:41:58.395 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:41:58.395 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:41:58.395 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:41:58.395 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 18, 2024-02-05T06:41:58.395 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 249, 2024-02-05T06:41:58.395 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:41:58.395 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:41:58.395 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:41:58.395 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:41:58.395 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.395 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:41:58.395 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6809", 2024-02-05T06:41:58.395 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1993280018 2024-02-05T06:41:58.396 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.396 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.396 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:41:58.396 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:41:58.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.396 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:41:58.396 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6810", 2024-02-05T06:41:58.396 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1993280018 2024-02-05T06:41:58.396 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.396 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.396 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:41:58.396 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:41:58.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.397 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:41:58.397 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6812", 2024-02-05T06:41:58.397 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1993280018 2024-02-05T06:41:58.397 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.397 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.397 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:41:58.397 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:41:58.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.397 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:41:58.397 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6811", 2024-02-05T06:41:58.397 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1993280018 2024-02-05T06:41:58.397 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.397 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.398 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6809/1993280018", 2024-02-05T06:41:58.398 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6810/1993280018", 2024-02-05T06:41:58.398 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6812/1993280018", 2024-02-05T06:41:58.398 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6811/1993280018", 2024-02-05T06:41:58.398 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:41:58.398 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:41:58.398 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:41:58.398 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.398 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.398 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.398 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-02-05T06:41:58.398 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "3ae795af-3d59-41a6-95eb-b8447ff61282", 2024-02-05T06:41:58.399 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:41:58.399 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:41:58.399 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:41:58.399 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:41:58.399 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:41:58.399 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:41:58.399 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 26, 2024-02-05T06:41:58.399 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 250, 2024-02-05T06:41:58.400 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:41:58.400 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:41:58.400 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:41:58.400 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:41:58.400 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.400 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:41:58.401 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6813", 2024-02-05T06:41:58.401 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1294621398 2024-02-05T06:41:58.401 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.401 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.401 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:41:58.401 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:41:58.401 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.401 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:41:58.401 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6814", 2024-02-05T06:41:58.401 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1294621398 2024-02-05T06:41:58.401 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.401 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.402 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:41:58.402 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:41:58.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.402 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:41:58.402 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6816", 2024-02-05T06:41:58.402 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1294621398 2024-02-05T06:41:58.402 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.402 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.402 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.402 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:41:58.402 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:41:58.402 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.402 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:41:58.402 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6815", 2024-02-05T06:41:58.403 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1294621398 2024-02-05T06:41:58.403 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.403 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.403 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6813/1294621398", 2024-02-05T06:41:58.403 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6814/1294621398", 2024-02-05T06:41:58.403 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6816/1294621398", 2024-02-05T06:41:58.403 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6815/1294621398", 2024-02-05T06:41:58.403 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:41:58.403 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:41:58.403 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:41:58.403 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.403 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.404 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.404 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-02-05T06:41:58.404 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "fc6bd83d-570c-48c0-964d-e09b4608bc32", 2024-02-05T06:41:58.404 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:41:58.404 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:41:58.404 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:41:58.404 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:41:58.404 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:41:58.404 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:41:58.404 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 31, 2024-02-05T06:41:58.404 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 248, 2024-02-05T06:41:58.404 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:41:58.404 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:41:58.405 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:41:58.405 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:41:58.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.405 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:41:58.405 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6800", 2024-02-05T06:41:58.405 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4052090595 2024-02-05T06:41:58.405 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.405 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.405 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:41:58.405 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:41:58.405 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.405 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:41:58.405 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6801", 2024-02-05T06:41:58.406 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4052090595 2024-02-05T06:41:58.406 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.406 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.406 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:41:58.406 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:41:58.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.406 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:41:58.406 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6803", 2024-02-05T06:41:58.406 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4052090595 2024-02-05T06:41:58.406 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.406 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.406 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:41:58.406 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:41:58.406 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.407 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:41:58.407 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6802", 2024-02-05T06:41:58.407 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4052090595 2024-02-05T06:41:58.407 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.407 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.407 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.167:6800/4052090595", 2024-02-05T06:41:58.407 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.167:6801/4052090595", 2024-02-05T06:41:58.407 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.167:6803/4052090595", 2024-02-05T06:41:58.407 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.167:6802/4052090595", 2024-02-05T06:41:58.407 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:41:58.407 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:41:58.407 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:41:58.407 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.408 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.408 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-02-05T06:41:58.408 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "895bf465-2531-4d9f-bc23-1e75fd969f07", 2024-02-05T06:41:58.408 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:41:58.408 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:41:58.408 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:41:58.408 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:41:58.408 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:41:58.408 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:41:58.408 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 40, 2024-02-05T06:41:58.408 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 250, 2024-02-05T06:41:58.408 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:41:58.408 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:41:58.409 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:41:58.409 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:41:58.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.409 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:41:58.409 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6804", 2024-02-05T06:41:58.409 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1631080500 2024-02-05T06:41:58.409 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.409 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.409 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:41:58.409 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:41:58.409 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.409 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:41:58.409 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6805", 2024-02-05T06:41:58.410 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1631080500 2024-02-05T06:41:58.410 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.410 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.410 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:41:58.410 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:41:58.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.410 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:41:58.410 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6807", 2024-02-05T06:41:58.410 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1631080500 2024-02-05T06:41:58.410 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.410 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.410 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:41:58.410 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:41:58.410 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.411 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:41:58.412 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6806", 2024-02-05T06:41:58.412 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1631080500 2024-02-05T06:41:58.412 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.412 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.412 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.167:6804/1631080500", 2024-02-05T06:41:58.412 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.167:6805/1631080500", 2024-02-05T06:41:58.412 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.167:6807/1631080500", 2024-02-05T06:41:58.413 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.167:6806/1631080500", 2024-02-05T06:41:58.413 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:41:58.413 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:41:58.413 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:41:58.413 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.413 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.413 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-02-05T06:41:58.413 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "ce0a9300-87a4-4e37-b702-d667f168c0e0", 2024-02-05T06:41:58.413 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:41:58.413 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:41:58.413 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:41:58.413 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:41:58.413 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:41:58.413 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:41:58.414 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 49, 2024-02-05T06:41:58.414 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 253, 2024-02-05T06:41:58.414 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:41:58.414 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:41:58.414 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:41:58.414 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:41:58.414 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.414 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:41:58.414 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6808", 2024-02-05T06:41:58.414 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1644050617 2024-02-05T06:41:58.414 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.414 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.414 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:41:58.415 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:41:58.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.415 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:41:58.415 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6809", 2024-02-05T06:41:58.415 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1644050617 2024-02-05T06:41:58.415 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.415 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.415 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:41:58.415 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:41:58.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.415 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:41:58.415 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6811", 2024-02-05T06:41:58.415 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1644050617 2024-02-05T06:41:58.416 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.416 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.416 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:41:58.416 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:41:58.416 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.416 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:41:58.416 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6810", 2024-02-05T06:41:58.416 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1644050617 2024-02-05T06:41:58.416 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.416 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.416 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.167:6808/1644050617", 2024-02-05T06:41:58.416 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.167:6809/1644050617", 2024-02-05T06:41:58.417 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.167:6811/1644050617", 2024-02-05T06:41:58.417 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.167:6810/1644050617", 2024-02-05T06:41:58.417 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:41:58.417 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:41:58.417 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:41:58.417 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.417 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.417 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.417 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-02-05T06:41:58.417 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "0c545cbc-0873-4232-8807-4af9299ef95e", 2024-02-05T06:41:58.417 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:41:58.417 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:41:58.417 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:41:58.417 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:41:58.417 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:41:58.418 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:41:58.418 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 58, 2024-02-05T06:41:58.418 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 247, 2024-02-05T06:41:58.418 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:41:58.418 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:41:58.418 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:41:58.418 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:41:58.418 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.418 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:41:58.418 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6812", 2024-02-05T06:41:58.418 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4031475764 2024-02-05T06:41:58.418 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.418 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.419 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:41:58.419 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:41:58.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.419 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:41:58.419 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6813", 2024-02-05T06:41:58.419 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4031475764 2024-02-05T06:41:58.419 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.419 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.419 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.419 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:41:58.419 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:41:58.419 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.419 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:41:58.419 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6815", 2024-02-05T06:41:58.419 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4031475764 2024-02-05T06:41:58.420 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.420 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.420 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:41:58.420 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:41:58.420 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.420 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:41:58.420 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6814", 2024-02-05T06:41:58.420 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4031475764 2024-02-05T06:41:58.420 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.420 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.420 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.167:6812/4031475764", 2024-02-05T06:41:58.420 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.167:6813/4031475764", 2024-02-05T06:41:58.421 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.167:6815/4031475764", 2024-02-05T06:41:58.421 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.167:6814/4031475764", 2024-02-05T06:41:58.421 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:41:58.421 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:41:58.421 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:41:58.421 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.421 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.421 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:41:58.421 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_xinfo": [ 2024-02-05T06:41:58.421 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.421 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-02-05T06:41:58.421 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:41:58.421 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:41:58.421 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:41:58.422 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:41:58.422 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:41:58.422 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:41:26.679567+0000", 2024-02-05T06:41:58.422 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:41:58.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.422 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.422 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-02-05T06:41:58.422 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:41:58.422 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:41:58.422 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:41:58.422 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:41:58.422 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:41:58.422 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:41:28.210087+0000", 2024-02-05T06:41:58.423 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:41:58.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.423 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-02-05T06:41:58.423 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:41:58.423 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:41:58.423 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:41:58.423 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:41:58.423 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:41:58.423 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:41:01.104811+0000", 2024-02-05T06:41:58.423 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:41:58.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.423 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.423 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-02-05T06:41:58.424 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:41:58.424 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:41:58.424 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:41:58.424 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:41:58.424 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:41:58.424 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:41:36.898273+0000", 2024-02-05T06:41:58.424 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:41:58.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.424 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.424 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-02-05T06:41:58.424 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:41:58.424 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:41:58.424 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:41:58.424 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:41:58.425 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:41:58.425 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:40:48.443277+0000", 2024-02-05T06:41:58.425 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:41:58.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.425 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.425 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-02-05T06:41:58.425 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:41:58.425 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:41:58.425 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:41:58.425 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:41:58.425 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:41:58.425 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:40:41.894769+0000", 2024-02-05T06:41:58.425 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:41:58.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.425 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.426 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-02-05T06:41:58.426 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:41:58.426 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:41:58.426 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:41:58.426 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:41:58.426 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:41:58.426 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:41:19.366195+0000", 2024-02-05T06:41:58.426 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:41:58.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.426 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.426 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-02-05T06:41:58.426 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:41:58.426 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:41:58.426 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:41:58.426 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:41:58.427 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:41:58.427 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:41:26.190582+0000", 2024-02-05T06:41:58.427 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:41:58.427 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.427 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:41:58.427 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap": [ 2024-02-05T06:41:58.427 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.427 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "3.18", 2024-02-05T06:41:58.427 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-02-05T06:41:58.427 INFO:teuthology.orchestra.run.smithi028.stdout: 5, 2024-02-05T06:41:58.427 INFO:teuthology.orchestra.run.smithi028.stdout: 2, 2024-02-05T06:41:58.427 INFO:teuthology.orchestra.run.smithi028.stdout: 4 2024-02-05T06:41:58.427 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.427 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.427 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:41:58.428 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_items": [ 2024-02-05T06:41:58.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.428 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "3.9", 2024-02-05T06:41:58.428 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:41:58.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.428 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 4, 2024-02-05T06:41:58.428 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 1 2024-02-05T06:41:58.428 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.428 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.428 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "3.a", 2024-02-05T06:41:58.428 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:41:58.428 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.429 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 4, 2024-02-05T06:41:58.429 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-02-05T06:41:58.429 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.429 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.429 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.f", 2024-02-05T06:41:58.429 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:41:58.429 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.429 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-02-05T06:41:58.429 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 0 2024-02-05T06:41:58.429 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.429 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.430 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.12", 2024-02-05T06:41:58.430 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:41:58.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.430 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-02-05T06:41:58.430 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 0 2024-02-05T06:41:58.430 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.430 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.430 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.14", 2024-02-05T06:41:58.430 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:41:58.430 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.430 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-02-05T06:41:58.430 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-02-05T06:41:58.430 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.431 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.431 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.15", 2024-02-05T06:41:58.431 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:41:58.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.431 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 3, 2024-02-05T06:41:58.431 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-02-05T06:41:58.431 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.431 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.431 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.1d", 2024-02-05T06:41:58.431 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:41:58.431 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.432 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-02-05T06:41:58.432 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-02-05T06:41:58.432 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.432 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.432 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.432 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "5.1c", 2024-02-05T06:41:58.432 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:41:58.432 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.432 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 4, 2024-02-05T06:41:58.432 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 6 2024-02-05T06:41:58.432 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.432 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.433 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "5.1e", 2024-02-05T06:41:58.433 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:41:58.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.433 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 0, 2024-02-05T06:41:58.433 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 3 2024-02-05T06:41:58.433 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.433 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.433 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "6.13", 2024-02-05T06:41:58.433 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:41:58.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.433 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 3, 2024-02-05T06:41:58.434 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 1 2024-02-05T06:41:58.434 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.434 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.434 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.434 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:41:58.434 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_primaries": [], 2024-02-05T06:41:58.434 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_temp": [ 2024-02-05T06:41:58.434 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.434 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "8.0", 2024-02-05T06:41:58.434 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-02-05T06:41:58.434 INFO:teuthology.orchestra.run.smithi028.stdout: 3, 2024-02-05T06:41:58.434 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-02-05T06:41:58.434 INFO:teuthology.orchestra.run.smithi028.stdout: 4 2024-02-05T06:41:58.434 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.435 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.435 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "8.2", 2024-02-05T06:41:58.435 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-02-05T06:41:58.435 INFO:teuthology.orchestra.run.smithi028.stdout: 4, 2024-02-05T06:41:58.435 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-02-05T06:41:58.435 INFO:teuthology.orchestra.run.smithi028.stdout: 3 2024-02-05T06:41:58.435 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.435 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.435 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "8.3", 2024-02-05T06:41:58.435 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-02-05T06:41:58.435 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-02-05T06:41:58.435 INFO:teuthology.orchestra.run.smithi028.stdout: 6, 2024-02-05T06:41:58.436 INFO:teuthology.orchestra.run.smithi028.stdout: 2 2024-02-05T06:41:58.436 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.436 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "8.4", 2024-02-05T06:41:58.436 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-02-05T06:41:58.436 INFO:teuthology.orchestra.run.smithi028.stdout: 2, 2024-02-05T06:41:58.436 INFO:teuthology.orchestra.run.smithi028.stdout: 4, 2024-02-05T06:41:58.436 INFO:teuthology.orchestra.run.smithi028.stdout: 6 2024-02-05T06:41:58.436 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.436 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.436 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.436 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "8.8", 2024-02-05T06:41:58.436 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-02-05T06:41:58.437 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-02-05T06:41:58.437 INFO:teuthology.orchestra.run.smithi028.stdout: 3, 2024-02-05T06:41:58.437 INFO:teuthology.orchestra.run.smithi028.stdout: 7 2024-02-05T06:41:58.437 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.437 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.437 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "8.a", 2024-02-05T06:41:58.437 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-02-05T06:41:58.437 INFO:teuthology.orchestra.run.smithi028.stdout: 1, 2024-02-05T06:41:58.437 INFO:teuthology.orchestra.run.smithi028.stdout: 2, 2024-02-05T06:41:58.437 INFO:teuthology.orchestra.run.smithi028.stdout: 3 2024-02-05T06:41:58.437 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.438 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:41:58.438 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "8.d", 2024-02-05T06:41:58.438 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-02-05T06:41:58.438 INFO:teuthology.orchestra.run.smithi028.stdout: 2, 2024-02-05T06:41:58.438 INFO:teuthology.orchestra.run.smithi028.stdout: 7, 2024-02-05T06:41:58.438 INFO:teuthology.orchestra.run.smithi028.stdout: 6 2024-02-05T06:41:58.438 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:41:58.438 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.438 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:41:58.438 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_temp": [], 2024-02-05T06:41:58.438 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklist": { 2024-02-05T06:41:58.438 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/1001442173": "2024-02-06T06:34:10.127914+0000", 2024-02-05T06:41:58.438 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3160026720": "2024-02-06T06:34:10.127914+0000", 2024-02-05T06:41:58.439 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/558171292": "2024-02-06T06:34:10.127914+0000", 2024-02-05T06:41:58.439 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2832437766": "2024-02-06T06:34:10.127914+0000", 2024-02-05T06:41:58.439 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/4125764599": "2024-02-06T06:28:54.105170+0000", 2024-02-05T06:41:58.439 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1081543653": "2024-02-06T06:28:54.105170+0000", 2024-02-05T06:41:58.439 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3794420554": "2024-02-06T06:28:27.553034+0000", 2024-02-05T06:41:58.439 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1560471317": "2024-02-06T06:34:10.127914+0000", 2024-02-05T06:41:58.439 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/4266601437": "2024-02-06T06:28:27.553034+0000", 2024-02-05T06:41:58.439 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/388893831": "2024-02-06T06:28:27.553034+0000", 2024-02-05T06:41:58.439 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/460817910": "2024-02-06T06:28:27.553034+0000", 2024-02-05T06:41:58.439 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2728305743": "2024-02-06T06:28:54.105170+0000", 2024-02-05T06:41:58.439 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3149394045": "2024-02-06T06:34:10.127914+0000", 2024-02-05T06:41:58.439 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3605013594": "2024-02-06T06:28:54.105170+0000" 2024-02-05T06:41:58.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.440 INFO:teuthology.orchestra.run.smithi028.stdout: "range_blocklist": {}, 2024-02-05T06:41:58.440 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profiles": { 2024-02-05T06:41:58.440 INFO:teuthology.orchestra.run.smithi028.stdout: "default": { 2024-02-05T06:41:58.440 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-02-05T06:41:58.440 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-02-05T06:41:58.440 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-02-05T06:41:58.440 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-02-05T06:41:58.440 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van" 2024-02-05T06:41:58.440 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.440 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:41:58.440 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps_queue": [], 2024-02-05T06:41:58.440 INFO:teuthology.orchestra.run.smithi028.stdout: "new_removed_snaps": [], 2024-02-05T06:41:58.440 INFO:teuthology.orchestra.run.smithi028.stdout: "new_purged_snaps": [], 2024-02-05T06:41:58.440 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_node_flags": {}, 2024-02-05T06:41:58.441 INFO:teuthology.orchestra.run.smithi028.stdout: "device_class_flags": {}, 2024-02-05T06:41:58.441 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": { 2024-02-05T06:41:58.441 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_enabled": false, 2024-02-05T06:41:58.441 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_bucket_count": 0, 2024-02-05T06:41:58.441 INFO:teuthology.orchestra.run.smithi028.stdout: "degraded_stretch_mode": 0, 2024-02-05T06:41:58.441 INFO:teuthology.orchestra.run.smithi028.stdout: "recovering_stretch_mode": 0, 2024-02-05T06:41:58.441 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_bucket": 0 2024-02-05T06:41:58.441 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:41:58.441 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:41:58.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:58.442 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:58.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:41:58.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:58.595 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:58.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:58.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:58.697 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:58.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:58 smithi028 ceph-mon[29442]: osdmap e254: 8 total, 8 up, 8 in 2024-02-05T06:41:58.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:58 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2400513143' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T06:41:58.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:58 smithi028 ceph-mon[37984]: osdmap e254: 8 total, 8 up, 8 in 2024-02-05T06:41:58.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:58 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2400513143' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T06:41:58.729 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:58.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:58.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:58.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:58 smithi167 ceph-mon[32774]: osdmap e254: 8 total, 8 up, 8 in 2024-02-05T06:41:58.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:58 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2400513143' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T06:41:58.831 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:41:58.874 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:58.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:58.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:41:58.975 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:59.015 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:59.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:59.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:59.117 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:41:59.146 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:59.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:59.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:41:59.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:41:59.295 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:59.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:59.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:41:59.397 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:41:59.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:59.473 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:41:59.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:41:59.574 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:59.617 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:59.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:59.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:59.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:59 smithi028 ceph-mon[29442]: pgmap v409: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 150 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s; 23605/182973 objects misplaced (12.901%); 43 MiB/s, 685 objects/s recovering 2024-02-05T06:41:59.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:41:59 smithi028 ceph-mon[29442]: osdmap e255: 8 total, 8 up, 8 in 2024-02-05T06:41:59.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:59 smithi028 ceph-mon[37984]: pgmap v409: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 150 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s; 23605/182973 objects misplaced (12.901%); 43 MiB/s, 685 objects/s recovering 2024-02-05T06:41:59.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:41:59 smithi028 ceph-mon[37984]: osdmap e255: 8 total, 8 up, 8 in 2024-02-05T06:41:59.719 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:41:59.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:59 smithi167 ceph-mon[32774]: pgmap v409: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 150 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s; 23605/182973 objects misplaced (12.901%); 43 MiB/s, 685 objects/s recovering 2024-02-05T06:41:59.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:41:59 smithi167 ceph-mon[32774]: osdmap e255: 8 total, 8 up, 8 in 2024-02-05T06:41:59.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:59.768 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:59.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:41:59.869 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:41:59.911 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:41:59.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:41:59.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:00.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:00.055 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:00.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:00.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:00.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:00.185 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:00.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:00.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:00.286 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:00.323 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:00.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:00.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:00.425 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:00.450 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:00.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:00.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:00.552 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:00.592 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:00.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:00.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:00.694 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:00.721 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:00.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:00.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:00.823 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:00.867 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:00.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:00.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:00.969 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:01.000 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:01.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:01.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:01.102 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:01.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:01.149 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:01.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:01.250 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:01.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:01 smithi028 ceph-mon[29442]: osdmap e256: 8 total, 8 up, 8 in 2024-02-05T06:42:01.257 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:01 smithi028 ceph-mon[29442]: pgmap v412: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 150 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 972 B/s rd, 0 B/s wr, 921 op/s; 23605/182973 objects misplaced (12.901%); 33 MiB/s, 522 objects/s recovering 2024-02-05T06:42:01.258 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:01 smithi028 ceph-mon[37984]: osdmap e256: 8 total, 8 up, 8 in 2024-02-05T06:42:01.258 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:01 smithi028 ceph-mon[37984]: pgmap v412: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 150 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 972 B/s rd, 0 B/s wr, 921 op/s; 23605/182973 objects misplaced (12.901%); 33 MiB/s, 522 objects/s recovering 2024-02-05T06:42:01.279 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:01.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:01.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:01.380 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:01.415 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:01.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:01.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:01.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:01 smithi167 ceph-mon[32774]: osdmap e256: 8 total, 8 up, 8 in 2024-02-05T06:42:01.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:01 smithi167 ceph-mon[32774]: pgmap v412: 158 pgs: 2 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 150 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 972 B/s rd, 0 B/s wr, 921 op/s; 23605/182973 objects misplaced (12.901%); 33 MiB/s, 522 objects/s recovering 2024-02-05T06:42:01.517 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:01.554 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:01.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:01.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:01.656 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:01.715 INFO:tasks.thrashosds.thrasher:j is {'epoch': 254, 'fsid': '9801e956-c3ef-11ee-95b6-87774f69a715', 'created': '2024-02-05T06:27:59.919226+0000', 'modified': '2024-02-05T06:41:57.415443+0000', 'last_up_change': '2024-02-05T06:33:15.833573+0000', 'last_in_change': '2024-02-05T06:41:42.383087+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-05T06:31:10.441424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '142', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '129', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.180000066757202, 'score_stable': 2.180000066757202, 'optimal_score': 1, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-02-05T06:33:27.397416+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-02-05T06:33:28.090506+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-05T06:33:29.866249+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-05T06:33:31.826006+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-05T06:33:33.890835+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-02-05T06:39:40.813792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '242', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'c19fdd5b-09f6-45d4-a519-0a938cf1f2a6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 250, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6801', 'nonce': 301279106}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6802', 'nonce': 301279106}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6804', 'nonce': 301279106}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6803', 'nonce': 301279106}]}, 'public_addr': '172.21.15.28:6801/301279106', 'cluster_addr': '172.21.15.28:6802/301279106', 'heartbeat_back_addr': '172.21.15.28:6804/301279106', 'heartbeat_front_addr': '172.21.15.28:6803/301279106', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '52245f6e-eb96-4d0d-985b-dbb6fd3ce8de', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 248, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6805', 'nonce': 1795224649}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6806', 'nonce': 1795224649}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6808', 'nonce': 1795224649}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6807', 'nonce': 1795224649}]}, 'public_addr': '172.21.15.28:6805/1795224649', 'cluster_addr': '172.21.15.28:6806/1795224649', 'heartbeat_back_addr': '172.21.15.28:6808/1795224649', 'heartbeat_front_addr': '172.21.15.28:6807/1795224649', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '1e18b630-afdf-4de9-a1a2-37bdfdd01c67', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 249, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6809', 'nonce': 1993280018}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6810', 'nonce': 1993280018}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6812', 'nonce': 1993280018}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6811', 'nonce': 1993280018}]}, 'public_addr': '172.21.15.28:6809/1993280018', 'cluster_addr': '172.21.15.28:6810/1993280018', 'heartbeat_back_addr': '172.21.15.28:6812/1993280018', 'heartbeat_front_addr': '172.21.15.28:6811/1993280018', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '3ae795af-3d59-41a6-95eb-b8447ff61282', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 250, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6813', 'nonce': 1294621398}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6814', 'nonce': 1294621398}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6816', 'nonce': 1294621398}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6815', 'nonce': 1294621398}]}, 'public_addr': '172.21.15.28:6813/1294621398', 'cluster_addr': '172.21.15.28:6814/1294621398', 'heartbeat_back_addr': '172.21.15.28:6816/1294621398', 'heartbeat_front_addr': '172.21.15.28:6815/1294621398', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'fc6bd83d-570c-48c0-964d-e09b4608bc32', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 248, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6800', 'nonce': 4052090595}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6801', 'nonce': 4052090595}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6803', 'nonce': 4052090595}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6802', 'nonce': 4052090595}]}, 'public_addr': '172.21.15.167:6800/4052090595', 'cluster_addr': '172.21.15.167:6801/4052090595', 'heartbeat_back_addr': '172.21.15.167:6803/4052090595', 'heartbeat_front_addr': '172.21.15.167:6802/4052090595', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '895bf465-2531-4d9f-bc23-1e75fd969f07', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 250, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6804', 'nonce': 1631080500}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6805', 'nonce': 1631080500}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6807', 'nonce': 1631080500}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6806', 'nonce': 1631080500}]}, 'public_addr': '172.21.15.167:6804/1631080500', 'cluster_addr': '172.21.15.167:6805/1631080500', 'heartbeat_back_addr': '172.21.15.167:6807/1631080500', 'heartbeat_front_addr': '172.21.15.167:6806/1631080500', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ce0a9300-87a4-4e37-b702-d667f168c0e0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 253, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6808', 'nonce': 1644050617}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6809', 'nonce': 1644050617}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6811', 'nonce': 1644050617}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6810', 'nonce': 1644050617}]}, 'public_addr': '172.21.15.167:6808/1644050617', 'cluster_addr': '172.21.15.167:6809/1644050617', 'heartbeat_back_addr': '172.21.15.167:6811/1644050617', 'heartbeat_front_addr': '172.21.15.167:6810/1644050617', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '0c545cbc-0873-4232-8807-4af9299ef95e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 247, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6812', 'nonce': 4031475764}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6813', 'nonce': 4031475764}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6815', 'nonce': 4031475764}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6814', 'nonce': 4031475764}]}, 'public_addr': '172.21.15.167:6812/4031475764', 'cluster_addr': '172.21.15.167:6813/4031475764', 'heartbeat_back_addr': '172.21.15.167:6815/4031475764', 'heartbeat_front_addr': '172.21.15.167:6814/4031475764', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:41:26.679567+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:41:28.210087+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:41:01.104811+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:41:36.898273+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:40:48.443277+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:40:41.894769+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:41:19.366195+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:41:26.190582+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.18', 'osds': [5, 2, 4]}], 'pg_upmap_items': [{'pgid': '3.9', 'mappings': [{'from': 4, 'to': 1}]}, {'pgid': '3.a', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '4.14', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.15', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.1c', 'mappings': [{'from': 4, 'to': 6}]}, {'pgid': '5.1e', 'mappings': [{'from': 0, 'to': 3}]}, {'pgid': '6.13', 'mappings': [{'from': 3, 'to': 1}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '8.0', 'osds': [3, 1, 4]}, {'pgid': '8.2', 'osds': [4, 0, 3]}, {'pgid': '8.3', 'osds': [1, 6, 2]}, {'pgid': '8.4', 'osds': [2, 4, 6]}, {'pgid': '8.8', 'osds': [0, 3, 7]}, {'pgid': '8.a', 'osds': [1, 2, 3]}, {'pgid': '8.d', 'osds': [2, 7, 6]}], 'primary_temp': [], 'blocklist': {'172.21.15.28:6800/1001442173': '2024-02-06T06:34:10.127914+0000', '172.21.15.28:0/3160026720': '2024-02-06T06:34:10.127914+0000', '172.21.15.28:0/558171292': '2024-02-06T06:34:10.127914+0000', '172.21.15.28:0/2832437766': '2024-02-06T06:34:10.127914+0000', '172.21.15.28:6800/4125764599': '2024-02-06T06:28:54.105170+0000', '172.21.15.28:0/1081543653': '2024-02-06T06:28:54.105170+0000', '172.21.15.28:0/3794420554': '2024-02-06T06:28:27.553034+0000', '172.21.15.28:0/1560471317': '2024-02-06T06:34:10.127914+0000', '172.21.15.28:6800/4266601437': '2024-02-06T06:28:27.553034+0000', '172.21.15.28:0/388893831': '2024-02-06T06:28:27.553034+0000', '172.21.15.28:0/460817910': '2024-02-06T06:28:27.553034+0000', '172.21.15.28:0/2728305743': '2024-02-06T06:28:54.105170+0000', '172.21.15.28:0/3149394045': '2024-02-06T06:34:10.127914+0000', '172.21.15.28:0/3605013594': '2024-02-06T06:28:54.105170+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-05T06:42:01.716 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T06:42:01.722 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:01.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:01.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:01.824 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:01.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:01.857 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:01.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:01.958 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:01.986 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:01.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:01.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:02.088 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:02.130 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:02.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:02.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:02.232 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:02.312 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:02.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:02.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:02.415 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:02.469 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:02.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:02.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:02.571 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:02.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:02.603 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:02.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:02.697 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:42:02 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:42:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:42:02.704 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:02.738 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:02.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:02.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:02.779 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:42:02.840 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:02.873 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:02.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:02.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:02.975 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:03.006 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:03.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:03.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:03.108 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:03.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:03.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:03.232 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:42:02 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:42:03.266 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:03.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:03.296 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:03.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:03.397 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:03.444 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:03.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:03.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:03.546 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:03.577 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:03.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:03.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:03.678 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:03.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:03 smithi028 ceph-mon[29442]: pgmap v413: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 151 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.31k op/s; 19265/171354 objects misplaced (11.243%); 44 MiB/s, 704 objects/s recovering 2024-02-05T06:42:03.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:03 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:42:03.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:03 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:42:03.680 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:03 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:42:03.680 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:03 smithi028 ceph-mon[37984]: pgmap v413: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 151 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.31k op/s; 19265/171354 objects misplaced (11.243%); 44 MiB/s, 704 objects/s recovering 2024-02-05T06:42:03.680 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:03 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:42:03.680 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:03 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:42:03.680 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:03 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:42:03.707 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:03.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:03.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:03.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:03 smithi167 ceph-mon[32774]: pgmap v413: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 151 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.31k op/s; 19265/171354 objects misplaced (11.243%); 44 MiB/s, 704 objects/s recovering 2024-02-05T06:42:03.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:03 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:42:03.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:03 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:42:03.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:03 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:42:03.809 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:03.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:03.846 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:03.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:03.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:03.986 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:03.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:03.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:04.088 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:04.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:04.136 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:04.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:04.237 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:04.259 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:42:04.259 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:42:04.267 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:04.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:04.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:04.369 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:04.415 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:04.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:04.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:04.517 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:04.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:04 smithi028 ceph-mon[29442]: osdmap e257: 8 total, 8 up, 8 in 2024-02-05T06:42:04.540 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:04 smithi028 ceph-mon[37984]: osdmap e257: 8 total, 8 up, 8 in 2024-02-05T06:42:04.583 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:04.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:04.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:04.685 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:04.717 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:04.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:04.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:04.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:04 smithi167 ceph-mon[32774]: osdmap e257: 8 total, 8 up, 8 in 2024-02-05T06:42:04.819 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:04.856 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:04.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:04.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:04.872 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:42:04.958 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:04.986 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:04.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:04.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:05.088 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:05.126 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:05.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:05.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:05.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:05.268 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:05.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:05.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:05.370 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:05.408 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:05.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:05.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:05.510 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:05.539 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:05.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:05.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:05.620 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:05 smithi028 ceph-mon[37984]: pgmap v415: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 151 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 646 op/s; 19265/171354 objects misplaced (11.243%); 31 MiB/s, 501 objects/s recovering 2024-02-05T06:42:05.620 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:05 smithi028 ceph-mon[37984]: osdmap e258: 8 total, 8 up, 8 in 2024-02-05T06:42:05.620 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:05 smithi028 ceph-mon[29442]: pgmap v415: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 151 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 646 op/s; 19265/171354 objects misplaced (11.243%); 31 MiB/s, 501 objects/s recovering 2024-02-05T06:42:05.620 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:05 smithi028 ceph-mon[29442]: osdmap e258: 8 total, 8 up, 8 in 2024-02-05T06:42:05.641 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:05.678 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-05T06:42:05.689 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:05.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:05.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:05.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:05 smithi167 ceph-mon[32774]: pgmap v415: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 151 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 646 op/s; 19265/171354 objects misplaced (11.243%); 31 MiB/s, 501 objects/s recovering 2024-02-05T06:42:05.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:05 smithi167 ceph-mon[32774]: osdmap e258: 8 total, 8 up, 8 in 2024-02-05T06:42:05.792 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:05.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:05.836 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:05.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:05.937 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:05.972 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:05.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:05.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:06.074 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:06.114 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:06.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:06.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:06.215 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:06.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:06.279 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:06.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:06.380 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:06.411 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:06.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:06.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:06.513 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:06.541 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:06.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:06.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:06.622 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:42:06.643 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:06.669 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:42:06.676 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:06.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:06.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:06.716 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:42:06.760 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:42:06.779 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:06.798 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:42:06.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:06.848 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:06.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:06.853 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:42:06.893 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:42:06.941 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:42:06.948 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:42:06.948 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:42:06.948 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:42:06.948 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:42:06.948 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:42:06.948 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:42:06.949 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:42:06.949 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:42:06.949 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:06.980 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:06.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:06.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:07.082 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:07.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:07.114 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:07.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:07.215 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:07.257 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:07.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:07.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:07.359 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:07.391 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:07.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:07.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:07.493 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:07.532 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:07.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:07.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:07.634 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:07.665 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:07.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:07.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:07.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:07 smithi028 ceph-mon[29442]: pgmap v417: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 151 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 667 op/s; 18962/170445 objects misplaced (11.125%); 30 MiB/s, 480 objects/s recovering 2024-02-05T06:42:07.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:07 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:42:07.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:07 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:42:07.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:07 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3219225625' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:42:07.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:07 smithi028 ceph-mon[37984]: pgmap v417: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 151 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 667 op/s; 18962/170445 objects misplaced (11.125%); 30 MiB/s, 480 objects/s recovering 2024-02-05T06:42:07.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:07 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:42:07.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:07 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:42:07.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:07 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3219225625' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:42:07.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:07 smithi167 ceph-mon[32774]: pgmap v417: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 151 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 667 op/s; 18962/170445 objects misplaced (11.125%); 30 MiB/s, 480 objects/s recovering 2024-02-05T06:42:07.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:07 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:42:07.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:07 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:42:07.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:07 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3219225625' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:42:07.767 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:07.808 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:07.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:07.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:07.910 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:07.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:07.934 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:07.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:08.035 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:08.076 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:08.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:08.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:08.178 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:08.205 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:08.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:08.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:08.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:08.346 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:08.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:08.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:08.448 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:08.461 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-02-05T06:42:08.476 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:08.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:08.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:08.578 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:08.612 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:08.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:08.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:08.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:08 smithi028 ceph-mon[29442]: osdmap e259: 8 total, 8 up, 8 in 2024-02-05T06:42:08.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:08 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/595709886' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:42:08.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:08 smithi028 ceph-mon[37984]: osdmap e259: 8 total, 8 up, 8 in 2024-02-05T06:42:08.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:08 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/595709886' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:42:08.713 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:08.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:08 smithi167 ceph-mon[32774]: osdmap e259: 8 total, 8 up, 8 in 2024-02-05T06:42:08.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:08 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/595709886' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:42:08.746 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:08.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:08.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:08.848 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:08.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:08.889 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:08.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:08.990 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:09.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:09.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:09.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:09.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:09.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:09.274 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:09.317 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:09.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:09.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:09.419 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:09.452 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:09.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:09.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:09.554 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:09.591 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:09.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:09.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:09.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:09 smithi028 ceph-mon[29442]: pgmap v419: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 151 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 811 op/s; 15912/156750 objects misplaced (10.151%); 19 MiB/s, 303 objects/s recovering 2024-02-05T06:42:09.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:09 smithi028 ceph-mon[29442]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:42:09.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:09 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/595709886' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:42:09.609 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:09 smithi028 ceph-mon[29442]: osdmap e260: 8 total, 8 up, 8 in 2024-02-05T06:42:09.609 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:09 smithi028 ceph-mon[37984]: pgmap v419: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 151 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 811 op/s; 15912/156750 objects misplaced (10.151%); 19 MiB/s, 303 objects/s recovering 2024-02-05T06:42:09.609 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:09 smithi028 ceph-mon[37984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:42:09.609 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:09 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/595709886' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:42:09.609 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:09 smithi028 ceph-mon[37984]: osdmap e260: 8 total, 8 up, 8 in 2024-02-05T06:42:09.693 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:09.724 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:09.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:09.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:09.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:09 smithi167 ceph-mon[32774]: pgmap v419: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 151 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 811 op/s; 15912/156750 objects misplaced (10.151%); 19 MiB/s, 303 objects/s recovering 2024-02-05T06:42:09.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:09 smithi167 ceph-mon[32774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:42:09.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:09 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/595709886' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:42:09.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:09 smithi167 ceph-mon[32774]: osdmap e260: 8 total, 8 up, 8 in 2024-02-05T06:42:09.826 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:09.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:09.873 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:09.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:09.974 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:10.008 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:10.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:10.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:10.109 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:10.144 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:10.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:10.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:10.246 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:10.285 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:10.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:10.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:10.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:10.419 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:10.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:10.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:10.521 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:10.547 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:10.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:10.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:10.649 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:10.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:10.675 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:10.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:10.775 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:10.804 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:10.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:10.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:10.906 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:10.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:10.935 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:10.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:11.036 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:11.064 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:11.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:11.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:11.166 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:11.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:11.198 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:11.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:11.299 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:11.330 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:11.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:11.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:11.432 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:11.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:11 smithi028 ceph-mon[37984]: osdmap e261: 8 total, 8 up, 8 in 2024-02-05T06:42:11.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:11 smithi028 ceph-mon[37984]: pgmap v422: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 151 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 973 B/s rd, 0 B/s wr, 926 op/s; 15912/156750 objects misplaced (10.151%); 22 MiB/s, 345 objects/s recovering 2024-02-05T06:42:11.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:11 smithi028 ceph-mon[29442]: osdmap e261: 8 total, 8 up, 8 in 2024-02-05T06:42:11.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:11 smithi028 ceph-mon[29442]: pgmap v422: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 151 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 973 B/s rd, 0 B/s wr, 926 op/s; 15912/156750 objects misplaced (10.151%); 22 MiB/s, 345 objects/s recovering 2024-02-05T06:42:11.459 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:11.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:11.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:11.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:11 smithi167 ceph-mon[32774]: osdmap e261: 8 total, 8 up, 8 in 2024-02-05T06:42:11.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:11 smithi167 ceph-mon[32774]: pgmap v422: 158 pgs: 1 active+remapped+backfill_wait, 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 151 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 973 B/s rd, 0 B/s wr, 926 op/s; 15912/156750 objects misplaced (10.151%); 22 MiB/s, 345 objects/s recovering 2024-02-05T06:42:11.561 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:11.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:11.588 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:11.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:11.657 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:42:11.689 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:11.716 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:11.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:11.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:11.817 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:11.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:11.843 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:11.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:11.945 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:11.971 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:11.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:11.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:12.073 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:12.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:12.100 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:12.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:12.201 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:12.237 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:12.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:12.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:12.339 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:12.366 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:12 smithi028 ceph-mon[29442]: osdmap e262: 8 total, 8 up, 8 in 2024-02-05T06:42:12.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:12 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:42:12.367 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:12 smithi028 ceph-mon[37984]: osdmap e262: 8 total, 8 up, 8 in 2024-02-05T06:42:12.367 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:12 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:42:12.374 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:12.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:12.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:12.476 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:12.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:12 smithi167 ceph-mon[32774]: osdmap e262: 8 total, 8 up, 8 in 2024-02-05T06:42:12.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:12 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:42:12.507 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:12.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:12.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:12.609 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:12.649 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:12.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:12.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:12.698 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:42:12 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:42:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:42:12.751 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:12.797 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:12.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:12.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:12.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:12.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:12.942 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:12.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:13.043 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:13.073 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:13.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:13.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:13.175 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:13.214 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:13.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:13.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:13.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:13 smithi167 ceph-mon[32774]: osdmap e263: 8 total, 8 up, 8 in 2024-02-05T06:42:13.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:13 smithi167 ceph-mon[32774]: pgmap v425: 158 pgs: 1 peering, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 152 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 867 op/s; 12061/146355 objects misplaced (8.241%); 44 MiB/s, 709 objects/s recovering 2024-02-05T06:42:13.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:13 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:42:13.235 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:42:12 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:42:13.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:13.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:13.364 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:13.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:13.392 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:13 smithi028 ceph-mon[37984]: osdmap e263: 8 total, 8 up, 8 in 2024-02-05T06:42:13.392 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:13 smithi028 ceph-mon[37984]: pgmap v425: 158 pgs: 1 peering, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 152 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 867 op/s; 12061/146355 objects misplaced (8.241%); 44 MiB/s, 709 objects/s recovering 2024-02-05T06:42:13.392 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:13 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:42:13.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:13 smithi028 ceph-mon[29442]: osdmap e263: 8 total, 8 up, 8 in 2024-02-05T06:42:13.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:13 smithi028 ceph-mon[29442]: pgmap v425: 158 pgs: 1 peering, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 152 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 867 op/s; 12061/146355 objects misplaced (8.241%); 44 MiB/s, 709 objects/s recovering 2024-02-05T06:42:13.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:13 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:42:13.464 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:13.516 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:13.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:13.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:13.618 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:13.668 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:13.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:13.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:13.769 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:13.810 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:13.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:13.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:13.912 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:13.957 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:13.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:13.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:14.059 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:14.093 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:14.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:14.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:14.194 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:14.195 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:42:14.224 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:14.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:14.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:14.326 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:14.412 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:14.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:14.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:14.514 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:14.545 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:14.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:14.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:14.646 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:14.684 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:14.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:14.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:14.785 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:14.819 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:14.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:14.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:14.921 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:14.929 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:42:14.932 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-05T06:42:14.947 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:14.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:14.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:14.965 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T06:42:15.049 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:15.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:15.100 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:15.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:15.201 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:15.235 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:15.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:15.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:15.337 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:15.369 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:15.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:15.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:15.471 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:15.502 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:15 smithi028 ceph-mon[37984]: pgmap v426: 158 pgs: 1 peering, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 152 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 872 B/s rd, 0 B/s wr, 591 op/s; 12061/146355 objects misplaced (8.241%); 30 MiB/s, 483 objects/s recovering 2024-02-05T06:42:15.502 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:15 smithi028 ceph-mon[37984]: osdmap e264: 8 total, 8 up, 8 in 2024-02-05T06:42:15.502 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:15 smithi028 ceph-mon[29442]: pgmap v426: 158 pgs: 1 peering, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 152 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 872 B/s rd, 0 B/s wr, 591 op/s; 12061/146355 objects misplaced (8.241%); 30 MiB/s, 483 objects/s recovering 2024-02-05T06:42:15.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:15 smithi028 ceph-mon[29442]: osdmap e264: 8 total, 8 up, 8 in 2024-02-05T06:42:15.507 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:15.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:15.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:15.610 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:15.642 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:15.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:15.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:15.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:15 smithi167 ceph-mon[32774]: pgmap v426: 158 pgs: 1 peering, 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 152 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 872 B/s rd, 0 B/s wr, 591 op/s; 12061/146355 objects misplaced (8.241%); 30 MiB/s, 483 objects/s recovering 2024-02-05T06:42:15.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:15 smithi167 ceph-mon[32774]: osdmap e264: 8 total, 8 up, 8 in 2024-02-05T06:42:15.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:15.782 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:15.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:15.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:15.884 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:15.911 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:15.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:15.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:16.013 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:16.049 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:16.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:16.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:16.150 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:16.187 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:16.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:16.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:16.289 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:16.318 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:16.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:16.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:16.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:16.447 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:16.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:16.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:16.549 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:16.580 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:16.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:16.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:16.681 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:16.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:16 smithi028 ceph-mon[29442]: from='client.25180 v1:172.21.15.28:0/1972836333' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:42:16.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:16 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:42:16.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:16 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:42:16.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:16 smithi028 ceph-mon[29442]: osdmap e265: 8 total, 8 up, 8 in 2024-02-05T06:42:16.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:16 smithi028 ceph-mon[37984]: from='client.25180 v1:172.21.15.28:0/1972836333' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:42:16.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:16 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:42:16.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:16 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:42:16.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:16 smithi028 ceph-mon[37984]: osdmap e265: 8 total, 8 up, 8 in 2024-02-05T06:42:16.709 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:16.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:16.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:16.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:16 smithi167 ceph-mon[32774]: from='client.25180 v1:172.21.15.28:0/1972836333' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:42:16.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:16 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:42:16.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:16 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:42:16.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:16 smithi167 ceph-mon[32774]: osdmap e265: 8 total, 8 up, 8 in 2024-02-05T06:42:16.811 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:16.839 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:16.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:16.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:16.941 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:16.970 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:16.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:16.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:17.071 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:17.101 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:17.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:17.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:17.203 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:17.247 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:17.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:17.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:17.349 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:17.379 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:17.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:17.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:17.480 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:17.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:17.514 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:17.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:17.615 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:17.640 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:17.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:17.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:17.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:17 smithi028 ceph-mon[29442]: pgmap v429: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 825 op/s; 11300/143757 objects misplaced (7.860%); 40 MiB/s, 633 objects/s recovering 2024-02-05T06:42:17.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:17 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:42:17.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:17 smithi028 ceph-mon[29442]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:42:17.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:17 smithi028 ceph-mon[29442]: osdmap e266: 8 total, 8 up, 8 in 2024-02-05T06:42:17.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:17 smithi028 ceph-mon[37984]: pgmap v429: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 825 op/s; 11300/143757 objects misplaced (7.860%); 40 MiB/s, 633 objects/s recovering 2024-02-05T06:42:17.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:17 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:42:17.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:17 smithi028 ceph-mon[37984]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:42:17.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:17 smithi028 ceph-mon[37984]: osdmap e266: 8 total, 8 up, 8 in 2024-02-05T06:42:17.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:17 smithi167 ceph-mon[32774]: pgmap v429: 158 pgs: 3 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 153 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 825 op/s; 11300/143757 objects misplaced (7.860%); 40 MiB/s, 633 objects/s recovering 2024-02-05T06:42:17.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:17 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:42:17.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:17 smithi167 ceph-mon[32774]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:42:17.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:17 smithi167 ceph-mon[32774]: osdmap e266: 8 total, 8 up, 8 in 2024-02-05T06:42:17.742 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:17.775 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:17.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:17.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:17.877 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:17.915 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:17.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:17.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:18.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:18.040 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:18.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:18.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:18.141 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:18.183 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:18.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:18.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:18.285 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:18.319 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:18.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:18.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:18.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:18.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:18.462 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:18.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:18.563 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:18.605 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:18.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:18.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:18.707 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:18.740 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:18.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:18.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:18.842 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:18.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:18.892 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:18.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:18.993 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:19.047 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:19.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:19.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:19.149 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:19.190 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:19.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:19.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:19.292 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:19.331 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:19.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:19.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:19.433 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:19.473 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:19.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:19.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:19.575 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:19.613 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:19.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:19.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:19.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:19 smithi028 ceph-mon[37984]: pgmap v431: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 3 active+remapped+backfilling, 153 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 721 op/s; 8859/133371 objects misplaced (6.642%); 24 MiB/s, 376 objects/s recovering 2024-02-05T06:42:19.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:19 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:42:19.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:19 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:42:19.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:19 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:42:19.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:19 smithi028 ceph-mon[29442]: pgmap v431: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 3 active+remapped+backfilling, 153 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 721 op/s; 8859/133371 objects misplaced (6.642%); 24 MiB/s, 376 objects/s recovering 2024-02-05T06:42:19.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:19 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:42:19.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:19 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:42:19.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:19 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:42:19.716 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:19.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:19 smithi167 ceph-mon[32774]: pgmap v431: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 3 active+remapped+backfilling, 153 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 721 op/s; 8859/133371 objects misplaced (6.642%); 24 MiB/s, 376 objects/s recovering 2024-02-05T06:42:19.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:19 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:42:19.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:19 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:42:19.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:19 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:42:19.748 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:19.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:19.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:19.850 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:19.882 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:19.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:19.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:19.967 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":426,"stamp":"2024-02-05T06:42:14.317617+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3184716570,"num_objects":48785,"num_object_clones":0,"num_object_copies":146355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12061,"num_objects_unfound":0,"num_objects_dirty":48785,"num_whiteouts":0,"num_read":829,"num_read_kb":590,"num_write":130413,"num_write_kb":5713674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43063,"num_bytes_recovered":2796333945,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82327,"ondisk_log_size":82327,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":485,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":12419320,"kb_used_data":10081796,"kb_used_omap":12,"kb_used_meta":2337331,"kb_avail":737541896,"statfs":{"total":767960285184,"available":755242901504,"internally_reserved":0,"allocated":10323759104,"data_stored":10320103506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":2393427505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,3,2,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-227082240,"num_objects":-3465,"num_object_clones":0,"num_object_copies":-10395,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3851,"num_objects_unfound":0,"num_objects_dirty":-3465,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":3465,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2837,"num_bytes_recovered":185794560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.867986"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":325,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.116897+0000","last_change":"2024-02-05T06:41:43.412983+0000","last_active":"2024-02-05T06:42:11.116897+0000","last_peered":"2024-02-05T06:42:11.116897+0000","last_clean":"2024-02-05T06:42:11.116897+0000","last_became_active":"2024-02-05T06:41:43.412603+0000","last_became_peered":"2024-02-05T06:41:43.412603+0000","last_unstale":"2024-02-05T06:42:11.116897+0000","last_undegraded":"2024-02-05T06:42:11.116897+0000","last_fullsized":"2024-02-05T06:42:11.116897+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:41:24.362120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:41:24.362120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:28.671089+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":445,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.061248+0000","last_change":"2024-02-05T06:41:43.420644+0000","last_active":"2024-02-05T06:42:11.061248+0000","last_peered":"2024-02-05T06:42:11.061248+0000","last_clean":"2024-02-05T06:42:11.061248+0000","last_became_active":"2024-02-05T06:41:43.420342+0000","last_became_peered":"2024-02-05T06:41:43.420342+0000","last_unstale":"2024-02-05T06:42:11.061248+0000","last_undegraded":"2024-02-05T06:42:11.061248+0000","last_fullsized":"2024-02-05T06:42:11.061248+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:15.357465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:06.525012+0000","last_clean_scrub_stamp":"2024-02-05T06:39:15.357465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":435,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.062598+0000","last_change":"2024-02-05T06:41:43.421787+0000","last_active":"2024-02-05T06:42:11.062598+0000","last_peered":"2024-02-05T06:42:11.062598+0000","last_clean":"2024-02-05T06:42:11.062598+0000","last_became_active":"2024-02-05T06:41:43.421315+0000","last_became_peered":"2024-02-05T06:41:43.421315+0000","last_unstale":"2024-02-05T06:42:11.062598+0000","last_undegraded":"2024-02-05T06:42:11.062598+0000","last_fullsized":"2024-02-05T06:42:11.062598+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:28.841777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:39:28.841777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"247'13","reported_seq":329,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.115788+0000","last_change":"2024-02-05T06:41:42.473521+0000","last_active":"2024-02-05T06:42:11.115788+0000","last_peered":"2024-02-05T06:42:11.115788+0000","last_clean":"2024-02-05T06:42:11.115788+0000","last_became_active":"2024-02-05T06:36:54.959330+0000","last_became_peered":"2024-02-05T06:36:54.959330+0000","last_unstale":"2024-02-05T06:42:11.115788+0000","last_undegraded":"2024-02-05T06:42:11.115788+0000","last_fullsized":"2024-02-05T06:42:11.115788+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'13","last_scrub_stamp":"2024-02-05T06:41:42.473424+0000","last_deep_scrub":"247'13","last_deep_scrub_stamp":"2024-02-05T06:41:42.473424+0000","last_clean_scrub_stamp":"2024-02-05T06:41:42.473424+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:45.682620+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":7,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":421,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.067085+0000","last_change":"2024-02-05T06:40:27.077900+0000","last_active":"2024-02-05T06:42:11.067085+0000","last_peered":"2024-02-05T06:42:11.067085+0000","last_clean":"2024-02-05T06:42:11.067085+0000","last_became_active":"2024-02-05T06:40:27.077485+0000","last_became_peered":"2024-02-05T06:40:27.077485+0000","last_unstale":"2024-02-05T06:42:11.067085+0000","last_undegraded":"2024-02-05T06:42:11.067085+0000","last_fullsized":"2024-02-05T06:42:11.067085+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:21.419504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:37:21.419504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":315,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.115902+0000","last_change":"2024-02-05T06:41:47.546569+0000","last_active":"2024-02-05T06:42:11.115902+0000","last_peered":"2024-02-05T06:42:11.115902+0000","last_clean":"2024-02-05T06:42:11.115902+0000","last_became_active":"2024-02-05T06:36:54.959640+0000","last_became_peered":"2024-02-05T06:36:54.959640+0000","last_unstale":"2024-02-05T06:42:11.115902+0000","last_undegraded":"2024-02-05T06:42:11.115902+0000","last_fullsized":"2024-02-05T06:42:11.115902+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:41:47.546464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:41:47.546464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:43:04.216541+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":430,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.066927+0000","last_change":"2024-02-05T06:39:34.395050+0000","last_active":"2024-02-05T06:42:11.066927+0000","last_peered":"2024-02-05T06:42:11.066927+0000","last_clean":"2024-02-05T06:42:11.066927+0000","last_became_active":"2024-02-05T06:33:29.553864+0000","last_became_peered":"2024-02-05T06:33:29.553864+0000","last_unstale":"2024-02-05T06:42:11.066927+0000","last_undegraded":"2024-02-05T06:42:11.066927+0000","last_fullsized":"2024-02-05T06:42:11.066927+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:34.394974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:39:34.394974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"88'22","reported_seq":457,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.065417+0000","last_change":"2024-02-05T06:39:35.433638+0000","last_active":"2024-02-05T06:42:11.065417+0000","last_peered":"2024-02-05T06:42:11.065417+0000","last_clean":"2024-02-05T06:42:11.065417+0000","last_became_active":"2024-02-05T06:33:31.572544+0000","last_became_peered":"2024-02-05T06:33:31.572544+0000","last_unstale":"2024-02-05T06:42:11.065417+0000","last_undegraded":"2024-02-05T06:42:11.065417+0000","last_fullsized":"2024-02-05T06:42:11.065417+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'22","last_scrub_stamp":"2024-02-05T06:39:35.433539+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:39:35.433539+0000","last_clean_scrub_stamp":"2024-02-05T06:39:35.433539+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":422,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.060327+0000","last_change":"2024-02-05T06:40:27.069177+0000","last_active":"2024-02-05T06:42:11.060327+0000","last_peered":"2024-02-05T06:42:11.060327+0000","last_clean":"2024-02-05T06:42:11.060327+0000","last_became_active":"2024-02-05T06:40:27.068888+0000","last_became_peered":"2024-02-05T06:40:27.068888+0000","last_unstale":"2024-02-05T06:42:11.060327+0000","last_undegraded":"2024-02-05T06:42:11.060327+0000","last_fullsized":"2024-02-05T06:42:11.060327+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:49.406914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:37:49.406914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":39,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.059650+0000","last_change":"2024-02-05T06:41:43.418097+0000","last_active":"2024-02-05T06:42:11.059650+0000","last_peered":"2024-02-05T06:42:11.059650+0000","last_clean":"2024-02-05T06:42:11.059650+0000","last_became_active":"2024-02-05T06:41:43.417611+0000","last_became_peered":"2024-02-05T06:41:43.417611+0000","last_unstale":"2024-02-05T06:42:11.059650+0000","last_undegraded":"2024-02-05T06:42:11.059650+0000","last_fullsized":"2024-02-05T06:42:11.059650+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:38:15.240524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:38:15.240524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":39,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.060875+0000","last_change":"2024-02-05T06:41:43.404052+0000","last_active":"2024-02-05T06:42:11.060875+0000","last_peered":"2024-02-05T06:42:11.060875+0000","last_clean":"2024-02-05T06:42:11.060875+0000","last_became_active":"2024-02-05T06:41:43.403746+0000","last_became_peered":"2024-02-05T06:41:43.403746+0000","last_unstale":"2024-02-05T06:42:11.060875+0000","last_undegraded":"2024-02-05T06:42:11.060875+0000","last_fullsized":"2024-02-05T06:42:11.060875+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:41:32.584584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:44.081318+0000","last_clean_scrub_stamp":"2024-02-05T06:41:32.584584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:47.244919+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"92'13","reported_seq":325,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.116653+0000","last_change":"2024-02-05T06:41:31.544513+0000","last_active":"2024-02-05T06:42:11.116653+0000","last_peered":"2024-02-05T06:42:11.116653+0000","last_clean":"2024-02-05T06:42:11.116653+0000","last_became_active":"2024-02-05T06:36:54.969454+0000","last_became_peered":"2024-02-05T06:36:54.969454+0000","last_unstale":"2024-02-05T06:42:11.116653+0000","last_undegraded":"2024-02-05T06:42:11.116653+0000","last_fullsized":"2024-02-05T06:42:11.116653+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'13","last_scrub_stamp":"2024-02-05T06:41:31.544427+0000","last_deep_scrub":"92'13","last_deep_scrub_stamp":"2024-02-05T06:34:46.542893+0000","last_clean_scrub_stamp":"2024-02-05T06:41:31.544427+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:42.485599+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":39,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.059418+0000","last_change":"2024-02-05T06:41:43.408237+0000","last_active":"2024-02-05T06:42:11.059418+0000","last_peered":"2024-02-05T06:42:11.059418+0000","last_clean":"2024-02-05T06:42:11.059418+0000","last_became_active":"2024-02-05T06:41:43.407404+0000","last_became_peered":"2024-02-05T06:41:43.407404+0000","last_unstale":"2024-02-05T06:42:11.059418+0000","last_undegraded":"2024-02-05T06:42:11.059418+0000","last_fullsized":"2024-02-05T06:42:11.059418+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:41:30.216535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:21.998850+0000","last_clean_scrub_stamp":"2024-02-05T06:41:30.216535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:32.320162+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":131,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.061368+0000","last_change":"2024-02-05T06:41:49.773526+0000","last_active":"2024-02-05T06:42:11.061368+0000","last_peered":"2024-02-05T06:42:11.061368+0000","last_clean":"2024-02-05T06:42:11.061368+0000","last_became_active":"2024-02-05T06:40:27.071758+0000","last_became_peered":"2024-02-05T06:40:27.071758+0000","last_unstale":"2024-02-05T06:42:11.061368+0000","last_undegraded":"2024-02-05T06:42:11.061368+0000","last_fullsized":"2024-02-05T06:42:11.061368+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:41:49.773408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:41:49.773408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:43:05.292126+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":39,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.060086+0000","last_change":"2024-02-05T06:41:43.408791+0000","last_active":"2024-02-05T06:42:11.060086+0000","last_peered":"2024-02-05T06:42:11.060086+0000","last_clean":"2024-02-05T06:42:11.060086+0000","last_became_active":"2024-02-05T06:41:43.408428+0000","last_became_peered":"2024-02-05T06:41:43.408428+0000","last_unstale":"2024-02-05T06:42:11.060086+0000","last_undegraded":"2024-02-05T06:42:11.060086+0000","last_fullsized":"2024-02-05T06:42:11.060086+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:34.388028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:50.055245+0000","last_clean_scrub_stamp":"2024-02-05T06:39:34.388028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":321,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.116527+0000","last_change":"2024-02-05T06:41:48.543907+0000","last_active":"2024-02-05T06:42:11.116527+0000","last_peered":"2024-02-05T06:42:11.116527+0000","last_clean":"2024-02-05T06:42:11.116527+0000","last_became_active":"2024-02-05T06:36:54.954385+0000","last_became_peered":"2024-02-05T06:36:54.954385+0000","last_unstale":"2024-02-05T06:42:11.116527+0000","last_undegraded":"2024-02-05T06:42:11.116527+0000","last_fullsized":"2024-02-05T06:42:11.116527+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-02-05T06:41:48.543648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:41:48.543648+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:43:14.984806+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":327,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.117268+0000","last_change":"2024-02-05T06:41:43.412335+0000","last_active":"2024-02-05T06:42:11.117268+0000","last_peered":"2024-02-05T06:42:11.117268+0000","last_clean":"2024-02-05T06:42:11.117268+0000","last_became_active":"2024-02-05T06:41:43.411344+0000","last_became_peered":"2024-02-05T06:41:43.411344+0000","last_unstale":"2024-02-05T06:42:11.117268+0000","last_undegraded":"2024-02-05T06:42:11.117268+0000","last_fullsized":"2024-02-05T06:42:11.117268+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:41:39.552367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:41:39.552367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:52.614547+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":433,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.062325+0000","last_change":"2024-02-05T06:39:32.263466+0000","last_active":"2024-02-05T06:42:11.062325+0000","last_peered":"2024-02-05T06:42:11.062325+0000","last_clean":"2024-02-05T06:42:11.062325+0000","last_became_active":"2024-02-05T06:36:54.958069+0000","last_became_peered":"2024-02-05T06:36:54.958069+0000","last_unstale":"2024-02-05T06:42:11.062325+0000","last_undegraded":"2024-02-05T06:42:11.062325+0000","last_fullsized":"2024-02-05T06:42:11.062325+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:32.263391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:32.263391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"66'1","reported_seq":468,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.061346+0000","last_change":"2024-02-05T06:40:27.118661+0000","last_active":"2024-02-05T06:42:11.061346+0000","last_peered":"2024-02-05T06:42:11.061346+0000","last_clean":"2024-02-05T06:42:11.061346+0000","last_became_active":"2024-02-05T06:40:27.073360+0000","last_became_peered":"2024-02-05T06:40:27.073360+0000","last_unstale":"2024-02-05T06:42:11.061346+0000","last_undegraded":"2024-02-05T06:42:11.061346+0000","last_fullsized":"2024-02-05T06:42:11.061346+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:37:47.363630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:37:47.363630+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"88'22","reported_seq":471,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.068070+0000","last_change":"2024-02-05T06:41:39.454550+0000","last_active":"2024-02-05T06:42:11.068070+0000","last_peered":"2024-02-05T06:42:11.068070+0000","last_clean":"2024-02-05T06:42:11.068070+0000","last_became_active":"2024-02-05T06:33:31.569763+0000","last_became_peered":"2024-02-05T06:33:31.569763+0000","last_unstale":"2024-02-05T06:42:11.068070+0000","last_undegraded":"2024-02-05T06:42:11.068070+0000","last_fullsized":"2024-02-05T06:42:11.068070+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'22","last_scrub_stamp":"2024-02-05T06:41:39.454093+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:34:35.234636+0000","last_clean_scrub_stamp":"2024-02-05T06:41:39.454093+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:49.467085+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":422,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.068135+0000","last_change":"2024-02-05T06:41:43.411742+0000","last_active":"2024-02-05T06:42:11.068135+0000","last_peered":"2024-02-05T06:42:11.068135+0000","last_clean":"2024-02-05T06:42:11.068135+0000","last_became_active":"2024-02-05T06:41:43.411442+0000","last_became_peered":"2024-02-05T06:41:43.411442+0000","last_unstale":"2024-02-05T06:42:11.068135+0000","last_undegraded":"2024-02-05T06:42:11.068135+0000","last_fullsized":"2024-02-05T06:42:11.068135+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:06.387381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:06.387381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":195,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.067860+0000","last_change":"2024-02-05T06:39:14.482807+0000","last_active":"2024-02-05T06:42:11.067860+0000","last_peered":"2024-02-05T06:42:11.067860+0000","last_clean":"2024-02-05T06:42:11.067860+0000","last_became_active":"2024-02-05T06:39:13.248618+0000","last_became_peered":"2024-02-05T06:39:13.248618+0000","last_unstale":"2024-02-05T06:42:11.067860+0000","last_undegraded":"2024-02-05T06:42:11.067860+0000","last_fullsized":"2024-02-05T06:42:11.067860+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:14.482739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:14.482739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:14.227708+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":307,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.116546+0000","last_change":"2024-02-05T06:40:07.898901+0000","last_active":"2024-02-05T06:42:11.116546+0000","last_peered":"2024-02-05T06:42:11.116546+0000","last_clean":"2024-02-05T06:42:11.116546+0000","last_became_active":"2024-02-05T06:36:54.959070+0000","last_became_peered":"2024-02-05T06:36:54.959070+0000","last_unstale":"2024-02-05T06:42:11.116546+0000","last_undegraded":"2024-02-05T06:42:11.116546+0000","last_fullsized":"2024-02-05T06:42:11.116546+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:40:07.898795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:40:07.898795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":494,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.066026+0000","last_change":"2024-02-05T06:38:16.181378+0000","last_active":"2024-02-05T06:42:11.066026+0000","last_peered":"2024-02-05T06:42:11.066026+0000","last_clean":"2024-02-05T06:42:11.066026+0000","last_became_active":"2024-02-05T06:36:54.954880+0000","last_became_peered":"2024-02-05T06:36:54.954880+0000","last_unstale":"2024-02-05T06:42:11.066026+0000","last_undegraded":"2024-02-05T06:42:11.066026+0000","last_fullsized":"2024-02-05T06:42:11.066026+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:38:16.181294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:38:16.181294+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"73'1","reported_seq":137,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.061982+0000","last_change":"2024-02-05T06:41:43.446618+0000","last_active":"2024-02-05T06:42:11.061982+0000","last_peered":"2024-02-05T06:42:11.061982+0000","last_clean":"2024-02-05T06:42:11.061982+0000","last_became_active":"2024-02-05T06:41:43.418640+0000","last_became_peered":"2024-02-05T06:41:43.418640+0000","last_unstale":"2024-02-05T06:42:11.061982+0000","last_undegraded":"2024-02-05T06:42:11.061982+0000","last_fullsized":"2024-02-05T06:42:11.061982+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-02-05T06:40:09.298234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:40:09.298234+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:13.576904+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"208'12","reported_seq":582,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.066817+0000","last_change":"2024-02-05T06:40:27.068738+0000","last_active":"2024-02-05T06:42:11.066817+0000","last_peered":"2024-02-05T06:42:11.066817+0000","last_clean":"2024-02-05T06:42:11.066817+0000","last_became_active":"2024-02-05T06:40:27.063399+0000","last_became_peered":"2024-02-05T06:40:27.063399+0000","last_unstale":"2024-02-05T06:42:11.066817+0000","last_undegraded":"2024-02-05T06:42:11.066817+0000","last_fullsized":"2024-02-05T06:42:11.066817+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'11","last_scrub_stamp":"2024-02-05T06:39:10.519706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:10.519706+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:14.227580+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":458,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.061156+0000","last_change":"2024-02-05T06:41:42.377968+0000","last_active":"2024-02-05T06:42:11.061156+0000","last_peered":"2024-02-05T06:42:11.061156+0000","last_clean":"2024-02-05T06:42:11.061156+0000","last_became_active":"2024-02-05T06:36:54.958414+0000","last_became_peered":"2024-02-05T06:36:54.958414+0000","last_unstale":"2024-02-05T06:42:11.061156+0000","last_undegraded":"2024-02-05T06:42:11.061156+0000","last_fullsized":"2024-02-05T06:42:11.061156+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:41:42.377698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:41:42.377698+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:59.586309+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"92'15","reported_seq":483,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.062589+0000","last_change":"2024-02-05T06:40:46.436058+0000","last_active":"2024-02-05T06:42:11.062589+0000","last_peered":"2024-02-05T06:42:11.062589+0000","last_clean":"2024-02-05T06:42:11.062589+0000","last_became_active":"2024-02-05T06:40:27.100610+0000","last_became_peered":"2024-02-05T06:40:27.100610+0000","last_unstale":"2024-02-05T06:42:11.062589+0000","last_undegraded":"2024-02-05T06:42:11.062589+0000","last_fullsized":"2024-02-05T06:42:11.062589+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'15","last_scrub_stamp":"2024-02-05T06:40:07.595540+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-02-05T06:34:50.718768+0000","last_clean_scrub_stamp":"2024-02-05T06:40:07.595540+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":449,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.062452+0000","last_change":"2024-02-05T06:41:43.414961+0000","last_active":"2024-02-05T06:42:11.062452+0000","last_peered":"2024-02-05T06:42:11.062452+0000","last_clean":"2024-02-05T06:42:11.062452+0000","last_became_active":"2024-02-05T06:41:43.414638+0000","last_became_peered":"2024-02-05T06:41:43.414638+0000","last_unstale":"2024-02-05T06:42:11.062452+0000","last_undegraded":"2024-02-05T06:42:11.062452+0000","last_fullsized":"2024-02-05T06:42:11.062452+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:41:34.298647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:41:34.298647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:43:01.897056+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":337,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.066773+0000","last_change":"2024-02-05T06:40:27.097489+0000","last_active":"2024-02-05T06:42:11.066773+0000","last_peered":"2024-02-05T06:42:11.066773+0000","last_clean":"2024-02-05T06:42:11.066773+0000","last_became_active":"2024-02-05T06:40:27.097169+0000","last_became_peered":"2024-02-05T06:40:27.097169+0000","last_unstale":"2024-02-05T06:42:11.066773+0000","last_undegraded":"2024-02-05T06:42:11.066773+0000","last_fullsized":"2024-02-05T06:42:11.066773+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:38:26.171039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:08.713894+0000","last_clean_scrub_stamp":"2024-02-05T06:38:26.171039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":39,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.059286+0000","last_change":"2024-02-05T06:41:43.414726+0000","last_active":"2024-02-05T06:42:11.059286+0000","last_peered":"2024-02-05T06:42:11.059286+0000","last_clean":"2024-02-05T06:42:11.059286+0000","last_became_active":"2024-02-05T06:41:43.411043+0000","last_became_peered":"2024-02-05T06:41:43.411043+0000","last_unstale":"2024-02-05T06:42:11.059286+0000","last_undegraded":"2024-02-05T06:42:11.059286+0000","last_fullsized":"2024-02-05T06:42:11.059286+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:40:05.263073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:40:05.263073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"92'16","reported_seq":496,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.067827+0000","last_change":"2024-02-05T06:41:43.441700+0000","last_active":"2024-02-05T06:42:11.067827+0000","last_peered":"2024-02-05T06:42:11.067827+0000","last_clean":"2024-02-05T06:42:11.067827+0000","last_became_active":"2024-02-05T06:41:43.419769+0000","last_became_peered":"2024-02-05T06:41:43.419769+0000","last_unstale":"2024-02-05T06:42:11.067827+0000","last_undegraded":"2024-02-05T06:42:11.067827+0000","last_fullsized":"2024-02-05T06:42:11.067827+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'16","last_scrub_stamp":"2024-02-05T06:40:01.612165+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:34:45.270135+0000","last_clean_scrub_stamp":"2024-02-05T06:40:01.612165+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:14.227880+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":131,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.061459+0000","last_change":"2024-02-05T06:41:46.814211+0000","last_active":"2024-02-05T06:42:11.061459+0000","last_peered":"2024-02-05T06:42:11.061459+0000","last_clean":"2024-02-05T06:42:11.061459+0000","last_became_active":"2024-02-05T06:40:27.098348+0000","last_became_peered":"2024-02-05T06:40:27.098348+0000","last_unstale":"2024-02-05T06:42:11.061459+0000","last_undegraded":"2024-02-05T06:42:11.061459+0000","last_fullsized":"2024-02-05T06:42:11.061459+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:41:46.814088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:41:46.814088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:43:02.321371+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":452,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.066533+0000","last_change":"2024-02-05T06:40:27.060697+0000","last_active":"2024-02-05T06:42:11.066533+0000","last_peered":"2024-02-05T06:42:11.066533+0000","last_clean":"2024-02-05T06:42:11.066533+0000","last_became_active":"2024-02-05T06:40:27.060112+0000","last_became_peered":"2024-02-05T06:40:27.060112+0000","last_unstale":"2024-02-05T06:42:11.066533+0000","last_undegraded":"2024-02-05T06:42:11.066533+0000","last_fullsized":"2024-02-05T06:42:11.066533+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:15.439354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:15.439354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:14.227622+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":434,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.062794+0000","last_change":"2024-02-05T06:41:43.424786+0000","last_active":"2024-02-05T06:42:11.062794+0000","last_peered":"2024-02-05T06:42:11.062794+0000","last_clean":"2024-02-05T06:42:11.062794+0000","last_became_active":"2024-02-05T06:41:43.424459+0000","last_became_peered":"2024-02-05T06:41:43.424459+0000","last_unstale":"2024-02-05T06:42:11.062794+0000","last_undegraded":"2024-02-05T06:42:11.062794+0000","last_fullsized":"2024-02-05T06:42:11.062794+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:40:05.586211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:40:05.586211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"73'4","reported_seq":423,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.065870+0000","last_change":"2024-02-05T06:39:32.415715+0000","last_active":"2024-02-05T06:42:11.065870+0000","last_peered":"2024-02-05T06:42:11.065870+0000","last_clean":"2024-02-05T06:42:11.065870+0000","last_became_active":"2024-02-05T06:33:31.573410+0000","last_became_peered":"2024-02-05T06:33:31.573410+0000","last_unstale":"2024-02-05T06:42:11.065870+0000","last_undegraded":"2024-02-05T06:42:11.065870+0000","last_fullsized":"2024-02-05T06:42:11.065870+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'4","last_scrub_stamp":"2024-02-05T06:39:32.415626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:32.415626+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":196,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.062822+0000","last_change":"2024-02-05T06:40:02.234701+0000","last_active":"2024-02-05T06:42:11.062822+0000","last_peered":"2024-02-05T06:42:11.062822+0000","last_clean":"2024-02-05T06:42:11.062822+0000","last_became_active":"2024-02-05T06:39:13.245684+0000","last_became_peered":"2024-02-05T06:39:13.245684+0000","last_unstale":"2024-02-05T06:42:11.062822+0000","last_undegraded":"2024-02-05T06:42:11.062822+0000","last_fullsized":"2024-02-05T06:42:11.062822+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:40:02.234576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:40:02.234576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":123,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.062063+0000","last_change":"2024-02-05T06:40:27.063394+0000","last_active":"2024-02-05T06:42:11.062063+0000","last_peered":"2024-02-05T06:42:11.062063+0000","last_clean":"2024-02-05T06:42:11.062063+0000","last_became_active":"2024-02-05T06:40:27.062975+0000","last_became_peered":"2024-02-05T06:40:27.062975+0000","last_unstale":"2024-02-05T06:42:11.062063+0000","last_undegraded":"2024-02-05T06:42:11.062063+0000","last_fullsized":"2024-02-05T06:42:11.062063+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:43.604994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:37:43.604994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:13.576676+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":39,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.060576+0000","last_change":"2024-02-05T06:41:43.413331+0000","last_active":"2024-02-05T06:42:11.060576+0000","last_peered":"2024-02-05T06:42:11.060576+0000","last_clean":"2024-02-05T06:42:11.060576+0000","last_became_active":"2024-02-05T06:41:43.412654+0000","last_became_peered":"2024-02-05T06:41:43.412654+0000","last_unstale":"2024-02-05T06:42:11.060576+0000","last_undegraded":"2024-02-05T06:42:11.060576+0000","last_fullsized":"2024-02-05T06:42:11.060576+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:43.243812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:37:43.243812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"93'16","reported_seq":495,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.066809+0000","last_change":"2024-02-05T06:40:27.124811+0000","last_active":"2024-02-05T06:42:11.066809+0000","last_peered":"2024-02-05T06:42:11.066809+0000","last_clean":"2024-02-05T06:42:11.066809+0000","last_became_active":"2024-02-05T06:40:27.080906+0000","last_became_peered":"2024-02-05T06:40:27.080906+0000","last_unstale":"2024-02-05T06:42:11.066809+0000","last_undegraded":"2024-02-05T06:42:11.066809+0000","last_fullsized":"2024-02-05T06:42:11.066809+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'16","last_scrub_stamp":"2024-02-05T06:39:11.291756+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-02-05T06:35:22.938675+0000","last_clean_scrub_stamp":"2024-02-05T06:39:11.291756+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"73'1","reported_seq":431,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.059965+0000","last_change":"2024-02-05T06:41:43.427301+0000","last_active":"2024-02-05T06:42:11.059965+0000","last_peered":"2024-02-05T06:42:11.059965+0000","last_clean":"2024-02-05T06:42:11.059965+0000","last_became_active":"2024-02-05T06:41:43.409761+0000","last_became_peered":"2024-02-05T06:41:43.409761+0000","last_unstale":"2024-02-05T06:42:11.059965+0000","last_undegraded":"2024-02-05T06:42:11.059965+0000","last_fullsized":"2024-02-05T06:42:11.059965+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-02-05T06:37:41.335213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:37:41.335213+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":413,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.065808+0000","last_change":"2024-02-05T06:39:25.369971+0000","last_active":"2024-02-05T06:42:11.065808+0000","last_peered":"2024-02-05T06:42:11.065808+0000","last_clean":"2024-02-05T06:42:11.065808+0000","last_became_active":"2024-02-05T06:33:33.593901+0000","last_became_peered":"2024-02-05T06:33:33.593901+0000","last_unstale":"2024-02-05T06:42:11.065808+0000","last_undegraded":"2024-02-05T06:42:11.065808+0000","last_fullsized":"2024-02-05T06:42:11.065808+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:25.369865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:25.369865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":131,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.062664+0000","last_change":"2024-02-05T06:41:53.678428+0000","last_active":"2024-02-05T06:42:11.062664+0000","last_peered":"2024-02-05T06:42:11.062664+0000","last_clean":"2024-02-05T06:42:11.062664+0000","last_became_active":"2024-02-05T06:40:27.077701+0000","last_became_peered":"2024-02-05T06:40:27.077701+0000","last_unstale":"2024-02-05T06:42:11.062664+0000","last_undegraded":"2024-02-05T06:42:11.062664+0000","last_fullsized":"2024-02-05T06:42:11.062664+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:41:53.678210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:41:53.678210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:43:17.146051+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":198,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.060301+0000","last_change":"2024-02-05T06:39:13.292585+0000","last_active":"2024-02-05T06:42:11.060301+0000","last_peered":"2024-02-05T06:42:11.060301+0000","last_clean":"2024-02-05T06:42:11.060301+0000","last_became_active":"2024-02-05T06:39:13.244124+0000","last_became_peered":"2024-02-05T06:39:13.244124+0000","last_unstale":"2024-02-05T06:42:11.060301+0000","last_undegraded":"2024-02-05T06:42:11.060301+0000","last_fullsized":"2024-02-05T06:42:11.060301+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:36:30.047048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:30.047048+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":421,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.065036+0000","last_change":"2024-02-05T06:41:43.417497+0000","last_active":"2024-02-05T06:42:11.065036+0000","last_peered":"2024-02-05T06:42:11.065036+0000","last_clean":"2024-02-05T06:42:11.065036+0000","last_became_active":"2024-02-05T06:41:43.417155+0000","last_became_peered":"2024-02-05T06:41:43.417155+0000","last_unstale":"2024-02-05T06:42:11.065036+0000","last_undegraded":"2024-02-05T06:42:11.065036+0000","last_fullsized":"2024-02-05T06:42:11.065036+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:30.450988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:30.450988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":447,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.063818+0000","last_change":"2024-02-05T06:41:43.422033+0000","last_active":"2024-02-05T06:42:11.063818+0000","last_peered":"2024-02-05T06:42:11.063818+0000","last_clean":"2024-02-05T06:42:11.063818+0000","last_became_active":"2024-02-05T06:41:43.421433+0000","last_became_peered":"2024-02-05T06:41:43.421433+0000","last_unstale":"2024-02-05T06:42:11.063818+0000","last_undegraded":"2024-02-05T06:42:11.063818+0000","last_fullsized":"2024-02-05T06:42:11.063818+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:35.213313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:35.213313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":319,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.114877+0000","last_change":"2024-02-05T06:40:46.639977+0000","last_active":"2024-02-05T06:42:11.114877+0000","last_peered":"2024-02-05T06:42:11.114877+0000","last_clean":"2024-02-05T06:42:11.114877+0000","last_became_active":"2024-02-05T06:40:27.065047+0000","last_became_peered":"2024-02-05T06:40:27.065047+0000","last_unstale":"2024-02-05T06:42:11.114877+0000","last_undegraded":"2024-02-05T06:42:11.114877+0000","last_fullsized":"2024-02-05T06:42:11.114877+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:40:46.639897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:40:46.639897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"92'16","reported_seq":319,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.116744+0000","last_change":"2024-02-05T06:41:27.424684+0000","last_active":"2024-02-05T06:42:11.116744+0000","last_peered":"2024-02-05T06:42:11.116744+0000","last_clean":"2024-02-05T06:42:11.116744+0000","last_became_active":"2024-02-05T06:36:54.961610+0000","last_became_peered":"2024-02-05T06:36:54.961610+0000","last_unstale":"2024-02-05T06:42:11.116744+0000","last_undegraded":"2024-02-05T06:42:11.116744+0000","last_fullsized":"2024-02-05T06:42:11.116744+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'16","last_scrub_stamp":"2024-02-05T06:41:27.424553+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:35:02.589229+0000","last_clean_scrub_stamp":"2024-02-05T06:41:27.424553+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:37.195635+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":422,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.062441+0000","last_change":"2024-02-05T06:40:27.124829+0000","last_active":"2024-02-05T06:42:11.062441+0000","last_peered":"2024-02-05T06:42:11.062441+0000","last_clean":"2024-02-05T06:42:11.062441+0000","last_became_active":"2024-02-05T06:40:27.096132+0000","last_became_peered":"2024-02-05T06:40:27.096132+0000","last_unstale":"2024-02-05T06:42:11.062441+0000","last_undegraded":"2024-02-05T06:42:11.062441+0000","last_fullsized":"2024-02-05T06:42:11.062441+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:31.831291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:31.831291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"249'10","reported_seq":49,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.060719+0000","last_change":"2024-02-05T06:41:43.431253+0000","last_active":"2024-02-05T06:42:11.060719+0000","last_peered":"2024-02-05T06:42:11.060719+0000","last_clean":"2024-02-05T06:42:11.060719+0000","last_became_active":"2024-02-05T06:41:43.418753+0000","last_became_peered":"2024-02-05T06:41:43.418753+0000","last_unstale":"2024-02-05T06:42:11.060719+0000","last_undegraded":"2024-02-05T06:42:11.060719+0000","last_fullsized":"2024-02-05T06:42:11.060719+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:39:09.504696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:09.504696+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":123,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.061680+0000","last_change":"2024-02-05T06:40:27.098718+0000","last_active":"2024-02-05T06:42:11.061680+0000","last_peered":"2024-02-05T06:42:11.061680+0000","last_clean":"2024-02-05T06:42:11.061680+0000","last_became_active":"2024-02-05T06:40:27.098418+0000","last_became_peered":"2024-02-05T06:40:27.098418+0000","last_unstale":"2024-02-05T06:42:11.061680+0000","last_undegraded":"2024-02-05T06:42:11.061680+0000","last_fullsized":"2024-02-05T06:42:11.061680+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:34.651493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:37:34.651493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:13.576695+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'10","reported_seq":486,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.065508+0000","last_change":"2024-02-05T06:40:27.198874+0000","last_active":"2024-02-05T06:42:11.065508+0000","last_peered":"2024-02-05T06:42:11.065508+0000","last_clean":"2024-02-05T06:42:11.065508+0000","last_became_active":"2024-02-05T06:40:27.093418+0000","last_became_peered":"2024-02-05T06:40:27.093418+0000","last_unstale":"2024-02-05T06:42:11.065508+0000","last_undegraded":"2024-02-05T06:42:11.065508+0000","last_fullsized":"2024-02-05T06:42:11.065508+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-02-05T06:38:21.205097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:38:21.205097+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":327,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.116935+0000","last_change":"2024-02-05T06:41:43.413904+0000","last_active":"2024-02-05T06:42:11.116935+0000","last_peered":"2024-02-05T06:42:11.116935+0000","last_clean":"2024-02-05T06:42:11.116935+0000","last_became_active":"2024-02-05T06:41:43.413562+0000","last_became_peered":"2024-02-05T06:41:43.413562+0000","last_unstale":"2024-02-05T06:42:11.116935+0000","last_undegraded":"2024-02-05T06:42:11.116935+0000","last_fullsized":"2024-02-05T06:42:11.116935+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:41:38.559780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:41:38.559780+0000","last_clean_scrub_stamp":"2024-02-05T06:41:38.559780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:47.975454+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"208'10","reported_seq":554,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.065860+0000","last_change":"2024-02-05T06:41:42.168574+0000","last_active":"2024-02-05T06:42:11.065860+0000","last_peered":"2024-02-05T06:42:11.065860+0000","last_clean":"2024-02-05T06:42:11.065860+0000","last_became_active":"2024-02-05T06:40:27.096584+0000","last_became_peered":"2024-02-05T06:40:27.096584+0000","last_unstale":"2024-02-05T06:42:11.065860+0000","last_undegraded":"2024-02-05T06:42:11.065860+0000","last_fullsized":"2024-02-05T06:42:11.065860+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-02-05T06:41:42.168453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:41:42.168453+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:44.318369+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":434,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.061930+0000","last_change":"2024-02-05T06:40:27.088637+0000","last_active":"2024-02-05T06:42:11.061930+0000","last_peered":"2024-02-05T06:42:11.061930+0000","last_clean":"2024-02-05T06:42:11.061930+0000","last_became_active":"2024-02-05T06:40:27.088096+0000","last_became_peered":"2024-02-05T06:40:27.088096+0000","last_unstale":"2024-02-05T06:42:11.061930+0000","last_undegraded":"2024-02-05T06:42:11.061930+0000","last_fullsized":"2024-02-05T06:42:11.061930+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:31.169691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:36:31.169691+0000","last_clean_scrub_stamp":"2024-02-05T06:36:31.169691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"73'9","reported_seq":49,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.060881+0000","last_change":"2024-02-05T06:41:43.438677+0000","last_active":"2024-02-05T06:42:11.060881+0000","last_peered":"2024-02-05T06:42:11.060881+0000","last_clean":"2024-02-05T06:42:11.060881+0000","last_became_active":"2024-02-05T06:41:43.413708+0000","last_became_peered":"2024-02-05T06:41:43.413708+0000","last_unstale":"2024-02-05T06:42:11.060881+0000","last_undegraded":"2024-02-05T06:42:11.060881+0000","last_fullsized":"2024-02-05T06:42:11.060881+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:39:16.394399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:16.394399+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":455,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.068640+0000","last_change":"2024-02-05T06:41:36.438049+0000","last_active":"2024-02-05T06:42:11.068640+0000","last_peered":"2024-02-05T06:42:11.068640+0000","last_clean":"2024-02-05T06:42:11.068640+0000","last_became_active":"2024-02-05T06:40:27.090632+0000","last_became_peered":"2024-02-05T06:40:27.090632+0000","last_unstale":"2024-02-05T06:42:11.068640+0000","last_undegraded":"2024-02-05T06:42:11.068640+0000","last_fullsized":"2024-02-05T06:42:11.068640+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:41:36.437912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:41:36.437912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:43:00.510017+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":439,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.068052+0000","last_change":"2024-02-05T06:40:27.092004+0000","last_active":"2024-02-05T06:42:11.068052+0000","last_peered":"2024-02-05T06:42:11.068052+0000","last_clean":"2024-02-05T06:42:11.068052+0000","last_became_active":"2024-02-05T06:40:27.091544+0000","last_became_peered":"2024-02-05T06:40:27.091544+0000","last_unstale":"2024-02-05T06:42:11.068052+0000","last_undegraded":"2024-02-05T06:42:11.068052+0000","last_fullsized":"2024-02-05T06:42:11.068052+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:40:03.572450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:40:03.572450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":123,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.061640+0000","last_change":"2024-02-05T06:40:27.064705+0000","last_active":"2024-02-05T06:42:11.061640+0000","last_peered":"2024-02-05T06:42:11.061640+0000","last_clean":"2024-02-05T06:42:11.061640+0000","last_became_active":"2024-02-05T06:40:27.064388+0000","last_became_peered":"2024-02-05T06:40:27.064388+0000","last_unstale":"2024-02-05T06:42:11.061640+0000","last_undegraded":"2024-02-05T06:42:11.061640+0000","last_fullsized":"2024-02-05T06:42:11.061640+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:38:13.558400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:38:13.558400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:13.576728+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"73'9","reported_seq":466,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.062659+0000","last_change":"2024-02-05T06:40:46.447632+0000","last_active":"2024-02-05T06:42:11.062659+0000","last_peered":"2024-02-05T06:42:11.062659+0000","last_clean":"2024-02-05T06:42:11.062659+0000","last_became_active":"2024-02-05T06:40:27.101116+0000","last_became_peered":"2024-02-05T06:40:27.101116+0000","last_unstale":"2024-02-05T06:42:11.062659+0000","last_undegraded":"2024-02-05T06:42:11.062659+0000","last_fullsized":"2024-02-05T06:42:11.062659+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:36:23.148541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:23.148541+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":123,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.062859+0000","last_change":"2024-02-05T06:40:27.063878+0000","last_active":"2024-02-05T06:42:11.062859+0000","last_peered":"2024-02-05T06:42:11.062859+0000","last_clean":"2024-02-05T06:42:11.062859+0000","last_became_active":"2024-02-05T06:40:27.063350+0000","last_became_peered":"2024-02-05T06:40:27.063350+0000","last_unstale":"2024-02-05T06:42:11.062859+0000","last_undegraded":"2024-02-05T06:42:11.062859+0000","last_fullsized":"2024-02-05T06:42:11.062859+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:38:16.709585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:38:16.709585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:13.576743+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":39,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.060497+0000","last_change":"2024-02-05T06:41:43.406398+0000","last_active":"2024-02-05T06:42:11.060497+0000","last_peered":"2024-02-05T06:42:11.060497+0000","last_clean":"2024-02-05T06:42:11.060497+0000","last_became_active":"2024-02-05T06:41:43.406094+0000","last_became_peered":"2024-02-05T06:41:43.406094+0000","last_unstale":"2024-02-05T06:42:11.060497+0000","last_undegraded":"2024-02-05T06:42:11.060497+0000","last_fullsized":"2024-02-05T06:42:11.060497+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:29.372079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:45.096340+0000","last_clean_scrub_stamp":"2024-02-05T06:39:29.372079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":448,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.066641+0000","last_change":"2024-02-05T06:41:43.412454+0000","last_active":"2024-02-05T06:42:11.066641+0000","last_peered":"2024-02-05T06:42:11.066641+0000","last_clean":"2024-02-05T06:42:11.066641+0000","last_became_active":"2024-02-05T06:41:43.412130+0000","last_became_peered":"2024-02-05T06:41:43.412130+0000","last_unstale":"2024-02-05T06:42:11.066641+0000","last_undegraded":"2024-02-05T06:42:11.066641+0000","last_fullsized":"2024-02-05T06:42:11.066641+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:22.309939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:39:22.309939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:14.227757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":454,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.068222+0000","last_change":"2024-02-05T06:40:27.110051+0000","last_active":"2024-02-05T06:42:11.068222+0000","last_peered":"2024-02-05T06:42:11.068222+0000","last_clean":"2024-02-05T06:42:11.068222+0000","last_became_active":"2024-02-05T06:40:27.091689+0000","last_became_peered":"2024-02-05T06:40:27.091689+0000","last_unstale":"2024-02-05T06:42:11.068222+0000","last_undegraded":"2024-02-05T06:42:11.068222+0000","last_fullsized":"2024-02-05T06:42:11.068222+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'6","last_scrub_stamp":"2024-02-05T06:38:32.122868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:38:32.122868+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"262'8132","reported_seq":12097,"reported_epoch":262,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T06:42:11.420877+0000","last_change":"2024-02-05T06:41:44.545492+0000","last_active":"2024-02-05T06:42:11.420877+0000","last_peered":"2024-02-05T06:42:11.420877+0000","last_clean":"2024-02-05T06:41:42.347874+0000","last_became_active":"2024-02-05T06:41:44.525032+0000","last_became_peered":"2024-02-05T06:41:44.525032+0000","last_unstale":"2024-02-05T06:42:11.420877+0000","last_undegraded":"2024-02-05T06:42:11.420877+0000","last_fullsized":"2024-02-05T06:42:11.420877+0000","mapping_epoch":248,"log_start":"202'2500","ondisk_log_start":"202'2500","created":185,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":5632,"log_dups_size":0,"ondisk_log_size":5632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200146944,"num_objects":3054,"num_object_clones":0,"num_object_copies":9162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3054,"num_objects_unfound":0,"num_objects_dirty":3054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8132,"num_write_kb":357952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1894,"num_bytes_recovered":124125184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[2,4,6],"avail_no_missing":["2","4","6"],"object_location_counts":[{"shards":"2,4,6","objects":3054}],"blocked_by":[],"up_primary":5,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":39,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.059653+0000","last_change":"2024-02-05T06:41:43.421760+0000","last_active":"2024-02-05T06:42:11.059653+0000","last_peered":"2024-02-05T06:42:11.059653+0000","last_clean":"2024-02-05T06:42:11.059653+0000","last_became_active":"2024-02-05T06:41:43.421231+0000","last_became_peered":"2024-02-05T06:41:43.421231+0000","last_unstale":"2024-02-05T06:42:11.059653+0000","last_undegraded":"2024-02-05T06:42:11.059653+0000","last_fullsized":"2024-02-05T06:42:11.059653+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:23.325194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:23.325194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":137,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.062427+0000","last_change":"2024-02-05T06:41:54.668727+0000","last_active":"2024-02-05T06:42:11.062427+0000","last_peered":"2024-02-05T06:42:11.062427+0000","last_clean":"2024-02-05T06:42:11.062427+0000","last_became_active":"2024-02-05T06:40:27.068279+0000","last_became_peered":"2024-02-05T06:40:27.068279+0000","last_unstale":"2024-02-05T06:42:11.062427+0000","last_undegraded":"2024-02-05T06:42:11.062427+0000","last_fullsized":"2024-02-05T06:42:11.062427+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-02-05T06:41:54.668600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:41:54.668600+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:43:12.525098+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"73'15","reported_seq":55,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.059419+0000","last_change":"2024-02-05T06:41:43.444817+0000","last_active":"2024-02-05T06:42:11.059419+0000","last_peered":"2024-02-05T06:42:11.059419+0000","last_clean":"2024-02-05T06:42:11.059419+0000","last_became_active":"2024-02-05T06:41:43.412579+0000","last_became_peered":"2024-02-05T06:41:43.412579+0000","last_unstale":"2024-02-05T06:42:11.059419+0000","last_undegraded":"2024-02-05T06:42:11.059419+0000","last_fullsized":"2024-02-05T06:42:11.059419+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-02-05T06:40:01.321025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:40:01.321025+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"208'12","reported_seq":148,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.061510+0000","last_change":"2024-02-05T06:40:27.091490+0000","last_active":"2024-02-05T06:42:11.061510+0000","last_peered":"2024-02-05T06:42:11.061510+0000","last_clean":"2024-02-05T06:42:11.061510+0000","last_became_active":"2024-02-05T06:40:27.064195+0000","last_became_peered":"2024-02-05T06:40:27.064195+0000","last_unstale":"2024-02-05T06:42:11.061510+0000","last_undegraded":"2024-02-05T06:42:11.061510+0000","last_fullsized":"2024-02-05T06:42:11.061510+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-02-05T06:38:11.542821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:38:11.542821+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:13.576762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"262'7956","reported_seq":13515,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.306175+0000","last_change":"2024-02-05T06:40:47.342532+0000","last_active":"2024-02-05T06:42:11.306175+0000","last_peered":"2024-02-05T06:42:11.306175+0000","last_clean":"2024-02-05T06:42:11.306175+0000","last_became_active":"2024-02-05T06:40:47.337156+0000","last_became_peered":"2024-02-05T06:40:47.337156+0000","last_unstale":"2024-02-05T06:42:11.306175+0000","last_undegraded":"2024-02-05T06:42:11.306175+0000","last_fullsized":"2024-02-05T06:42:11.306175+0000","mapping_epoch":221,"log_start":"202'2500","ondisk_log_start":"202'2500","created":185,"last_epoch_clean":222,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":5456,"log_dups_size":0,"ondisk_log_size":5456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:14.227726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197525504,"num_objects":3014,"num_object_clones":0,"num_object_copies":9042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3014,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7956,"num_write_kb":351040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2692,"num_bytes_recovered":176422912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":421,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.067172+0000","last_change":"2024-02-05T06:40:27.092917+0000","last_active":"2024-02-05T06:42:11.067172+0000","last_peered":"2024-02-05T06:42:11.067172+0000","last_clean":"2024-02-05T06:42:11.067172+0000","last_became_active":"2024-02-05T06:40:27.092613+0000","last_became_peered":"2024-02-05T06:40:27.092613+0000","last_unstale":"2024-02-05T06:42:11.067172+0000","last_undegraded":"2024-02-05T06:42:11.067172+0000","last_fullsized":"2024-02-05T06:42:11.067172+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:26.390356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:26.390356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":123,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.062455+0000","last_change":"2024-02-05T06:40:27.063835+0000","last_active":"2024-02-05T06:42:11.062455+0000","last_peered":"2024-02-05T06:42:11.062455+0000","last_clean":"2024-02-05T06:42:11.062455+0000","last_became_active":"2024-02-05T06:40:27.063300+0000","last_became_peered":"2024-02-05T06:40:27.063300+0000","last_unstale":"2024-02-05T06:42:11.062455+0000","last_undegraded":"2024-02-05T06:42:11.062455+0000","last_fullsized":"2024-02-05T06:42:11.062455+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:38:15.583491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:38:15.583491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:13.576711+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'12","reported_seq":327,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.115861+0000","last_change":"2024-02-05T06:41:52.647443+0000","last_active":"2024-02-05T06:42:11.115861+0000","last_peered":"2024-02-05T06:42:11.115861+0000","last_clean":"2024-02-05T06:42:11.115861+0000","last_became_active":"2024-02-05T06:36:54.954071+0000","last_became_peered":"2024-02-05T06:36:54.954071+0000","last_unstale":"2024-02-05T06:42:11.115861+0000","last_undegraded":"2024-02-05T06:42:11.115861+0000","last_fullsized":"2024-02-05T06:42:11.115861+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:41:52.647335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:41:52.647335+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:43:16.018756+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":427,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.067052+0000","last_change":"2024-02-05T06:41:42.402797+0000","last_active":"2024-02-05T06:42:11.067052+0000","last_peered":"2024-02-05T06:42:11.067052+0000","last_clean":"2024-02-05T06:42:11.067052+0000","last_became_active":"2024-02-05T06:33:33.592546+0000","last_became_peered":"2024-02-05T06:33:33.592546+0000","last_unstale":"2024-02-05T06:42:11.067052+0000","last_undegraded":"2024-02-05T06:42:11.067052+0000","last_fullsized":"2024-02-05T06:42:11.067052+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:41:42.402682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:41:42.402682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:43:06.782219+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"262'8076","reported_seq":13798,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.571137+0000","last_change":"2024-02-05T06:41:46.064604+0000","last_active":"2024-02-05T06:42:11.571137+0000","last_peered":"2024-02-05T06:42:11.571137+0000","last_clean":"2024-02-05T06:42:11.571137+0000","last_became_active":"2024-02-05T06:41:46.052802+0000","last_became_peered":"2024-02-05T06:41:46.052802+0000","last_unstale":"2024-02-05T06:42:11.571137+0000","last_undegraded":"2024-02-05T06:42:11.571137+0000","last_fullsized":"2024-02-05T06:42:11.571137+0000","mapping_epoch":250,"log_start":"204'2700","ondisk_log_start":"204'2700","created":185,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":5376,"log_dups_size":0,"ondisk_log_size":5376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:16.551589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194510848,"num_objects":2968,"num_object_clones":0,"num_object_copies":8904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2968,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8076,"num_write_kb":353408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2773,"num_bytes_recovered":181731328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":123,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.062138+0000","last_change":"2024-02-05T06:40:27.218419+0000","last_active":"2024-02-05T06:42:11.062138+0000","last_peered":"2024-02-05T06:42:11.062138+0000","last_clean":"2024-02-05T06:42:11.062138+0000","last_became_active":"2024-02-05T06:40:27.218081+0000","last_became_peered":"2024-02-05T06:40:27.218081+0000","last_unstale":"2024-02-05T06:42:11.062138+0000","last_undegraded":"2024-02-05T06:42:11.062138+0000","last_fullsized":"2024-02-05T06:42:11.062138+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:32.925000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:32.925000+0000","last_clean_scrub_stamp":"2024-02-05T06:39:32.925000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:13.576829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":445,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.067860+0000","last_change":"2024-02-05T06:41:43.413486+0000","last_active":"2024-02-05T06:42:11.067860+0000","last_peered":"2024-02-05T06:42:11.067860+0000","last_clean":"2024-02-05T06:42:11.067860+0000","last_became_active":"2024-02-05T06:41:43.413095+0000","last_became_peered":"2024-02-05T06:41:43.413095+0000","last_unstale":"2024-02-05T06:42:11.067860+0000","last_undegraded":"2024-02-05T06:42:11.067860+0000","last_fullsized":"2024-02-05T06:42:11.067860+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:41:32.488776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:41:32.488776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:43.474463+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":129,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.062750+0000","last_change":"2024-02-05T06:41:43.419777+0000","last_active":"2024-02-05T06:42:11.062750+0000","last_peered":"2024-02-05T06:42:11.062750+0000","last_clean":"2024-02-05T06:42:11.062750+0000","last_became_active":"2024-02-05T06:41:43.418202+0000","last_became_peered":"2024-02-05T06:41:43.418202+0000","last_unstale":"2024-02-05T06:42:11.062750+0000","last_undegraded":"2024-02-05T06:42:11.062750+0000","last_fullsized":"2024-02-05T06:42:11.062750+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:40:04.284398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:40:04.284398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:13.576871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"73'19","reported_seq":541,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.067369+0000","last_change":"2024-02-05T06:40:27.083505+0000","last_active":"2024-02-05T06:42:11.067369+0000","last_peered":"2024-02-05T06:42:11.067369+0000","last_clean":"2024-02-05T06:42:11.067369+0000","last_became_active":"2024-02-05T06:40:27.068417+0000","last_became_peered":"2024-02-05T06:40:27.068417+0000","last_unstale":"2024-02-05T06:42:11.067369+0000","last_undegraded":"2024-02-05T06:42:11.067369+0000","last_fullsized":"2024-02-05T06:42:11.067369+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-02-05T06:39:34.480345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:34.480345+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:14.227802+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"262'8008","reported_seq":8174,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.474727+0000","last_change":"2024-02-05T06:39:42.012271+0000","last_active":"2024-02-05T06:42:11.474727+0000","last_peered":"2024-02-05T06:42:11.474727+0000","last_clean":"2024-02-05T06:42:11.474727+0000","last_became_active":"2024-02-05T06:39:42.011938+0000","last_became_peered":"2024-02-05T06:39:42.011938+0000","last_unstale":"2024-02-05T06:42:11.474727+0000","last_undegraded":"2024-02-05T06:42:11.474727+0000","last_fullsized":"2024-02-05T06:42:11.474727+0000","mapping_epoch":185,"log_start":"216'3200","ondisk_log_start":"216'3200","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":4808,"log_dups_size":0,"ondisk_log_size":4808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193855488,"num_objects":2958,"num_object_clones":0,"num_object_copies":8874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2958,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8008,"num_write_kb":350912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":414,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.060449+0000","last_change":"2024-02-05T06:40:27.130950+0000","last_active":"2024-02-05T06:42:11.060449+0000","last_peered":"2024-02-05T06:42:11.060449+0000","last_clean":"2024-02-05T06:42:11.060449+0000","last_became_active":"2024-02-05T06:40:27.129625+0000","last_became_peered":"2024-02-05T06:40:27.129625+0000","last_unstale":"2024-02-05T06:42:11.060449+0000","last_undegraded":"2024-02-05T06:42:11.060449+0000","last_fullsized":"2024-02-05T06:42:11.060449+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:22.102133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:36:22.102133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":447,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.067725+0000","last_change":"2024-02-05T06:41:43.419626+0000","last_active":"2024-02-05T06:42:11.067725+0000","last_peered":"2024-02-05T06:42:11.067725+0000","last_clean":"2024-02-05T06:42:11.067725+0000","last_became_active":"2024-02-05T06:36:54.966016+0000","last_became_peered":"2024-02-05T06:36:54.966016+0000","last_unstale":"2024-02-05T06:42:11.067725+0000","last_undegraded":"2024-02-05T06:42:11.067725+0000","last_fullsized":"2024-02-05T06:42:11.067725+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:41:43.419519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:41:43.419519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:43:05.611119+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":39,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.059995+0000","last_change":"2024-02-05T06:41:43.413435+0000","last_active":"2024-02-05T06:42:11.059995+0000","last_peered":"2024-02-05T06:42:11.059995+0000","last_clean":"2024-02-05T06:42:11.059995+0000","last_became_active":"2024-02-05T06:41:43.412611+0000","last_became_peered":"2024-02-05T06:41:43.412611+0000","last_unstale":"2024-02-05T06:42:11.059995+0000","last_undegraded":"2024-02-05T06:42:11.059995+0000","last_fullsized":"2024-02-05T06:42:11.059995+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:37:27.314329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:37:27.314329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"73'9","reported_seq":514,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.061187+0000","last_change":"2024-02-05T06:40:08.611772+0000","last_active":"2024-02-05T06:42:11.061187+0000","last_peered":"2024-02-05T06:42:11.061187+0000","last_clean":"2024-02-05T06:42:11.061187+0000","last_became_active":"2024-02-05T06:40:08.584204+0000","last_became_peered":"2024-02-05T06:40:08.584204+0000","last_unstale":"2024-02-05T06:42:11.061187+0000","last_undegraded":"2024-02-05T06:42:11.061187+0000","last_fullsized":"2024-02-05T06:42:11.061187+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:36:26.172612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:26.172612+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"261'8315","reported_seq":22820,"reported_epoch":262,"state":"peering","last_fresh":"2024-02-05T06:42:11.059944+0000","last_change":"2024-02-05T06:42:11.059944+0000","last_active":"2024-02-05T06:42:11.059944+0000","last_peered":"2024-02-05T06:42:10.598287+0000","last_clean":"2024-02-05T06:41:42.315324+0000","last_became_active":"2024-02-05T06:41:44.488768+0000","last_became_peered":"2024-02-05T06:41:44.488768+0000","last_unstale":"2024-02-05T06:42:11.059944+0000","last_undegraded":"2024-02-05T06:42:11.059944+0000","last_fullsized":"2024-02-05T06:42:11.059944+0000","mapping_epoch":262,"log_start":"207'2900","ondisk_log_start":"207'2900","created":185,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":5415,"log_dups_size":0,"ondisk_log_size":5415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202571776,"num_objects":3091,"num_object_clones":0,"num_object_copies":9273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8315,"num_write_kb":364992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7159,"num_bytes_recovered":469172224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":456,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.059553+0000","last_change":"2024-02-05T06:41:46.055318+0000","last_active":"2024-02-05T06:42:11.059553+0000","last_peered":"2024-02-05T06:42:11.059553+0000","last_clean":"2024-02-05T06:42:11.059553+0000","last_became_active":"2024-02-05T06:41:46.054968+0000","last_became_peered":"2024-02-05T06:41:46.054968+0000","last_unstale":"2024-02-05T06:42:11.059553+0000","last_undegraded":"2024-02-05T06:42:11.059553+0000","last_fullsized":"2024-02-05T06:42:11.059553+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:37:37.282442+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:37.282442+0000","last_clean_scrub_stamp":"2024-02-05T06:37:37.282442+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:12.858362+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":433,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.067147+0000","last_change":"2024-02-05T06:40:27.082866+0000","last_active":"2024-02-05T06:42:11.067147+0000","last_peered":"2024-02-05T06:42:11.067147+0000","last_clean":"2024-02-05T06:42:11.067147+0000","last_became_active":"2024-02-05T06:40:27.082440+0000","last_became_peered":"2024-02-05T06:40:27.082440+0000","last_unstale":"2024-02-05T06:42:11.067147+0000","last_undegraded":"2024-02-05T06:42:11.067147+0000","last_fullsized":"2024-02-05T06:42:11.067147+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:07.364172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:07.364172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"182'3","reported_seq":43,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.059742+0000","last_change":"2024-02-05T06:41:43.427965+0000","last_active":"2024-02-05T06:42:11.059742+0000","last_peered":"2024-02-05T06:42:11.059742+0000","last_clean":"2024-02-05T06:42:11.059742+0000","last_became_active":"2024-02-05T06:41:43.415806+0000","last_became_peered":"2024-02-05T06:41:43.415806+0000","last_unstale":"2024-02-05T06:42:11.059742+0000","last_undegraded":"2024-02-05T06:42:11.059742+0000","last_fullsized":"2024-02-05T06:42:11.059742+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'3","last_scrub_stamp":"2024-02-05T06:39:33.373415+0000","last_deep_scrub":"182'3","last_deep_scrub_stamp":"2024-02-05T06:39:33.373415+0000","last_clean_scrub_stamp":"2024-02-05T06:39:33.373415+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001220348,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":307,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.116630+0000","last_change":"2024-02-05T06:39:34.999778+0000","last_active":"2024-02-05T06:42:11.116630+0000","last_peered":"2024-02-05T06:42:11.116630+0000","last_clean":"2024-02-05T06:42:11.116630+0000","last_became_active":"2024-02-05T06:36:54.968205+0000","last_became_peered":"2024-02-05T06:36:54.968205+0000","last_unstale":"2024-02-05T06:42:11.116630+0000","last_undegraded":"2024-02-05T06:42:11.116630+0000","last_fullsized":"2024-02-05T06:42:11.116630+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:34.999659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:39:34.999659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"197'30","reported_seq":190,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.061505+0000","last_change":"2024-02-05T06:40:08.673390+0000","last_active":"2024-02-05T06:42:11.061505+0000","last_peered":"2024-02-05T06:42:11.061505+0000","last_clean":"2024-02-05T06:42:11.061505+0000","last_became_active":"2024-02-05T06:40:08.604483+0000","last_became_peered":"2024-02-05T06:40:08.604483+0000","last_unstale":"2024-02-05T06:42:11.061505+0000","last_undegraded":"2024-02-05T06:42:11.061505+0000","last_fullsized":"2024-02-05T06:42:11.061505+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'29","last_scrub_stamp":"2024-02-05T06:39:30.346384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:30.346384+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":427,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.061546+0000","last_change":"2024-02-05T06:36:54.974218+0000","last_active":"2024-02-05T06:42:11.061546+0000","last_peered":"2024-02-05T06:42:11.061546+0000","last_clean":"2024-02-05T06:42:11.061546+0000","last_became_active":"2024-02-05T06:36:54.973387+0000","last_became_peered":"2024-02-05T06:36:54.973387+0000","last_unstale":"2024-02-05T06:42:11.061546+0000","last_undegraded":"2024-02-05T06:42:11.061546+0000","last_fullsized":"2024-02-05T06:42:11.061546+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:36.161606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:36.161606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"262'8167","reported_seq":8332,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.525027+0000","last_change":"2024-02-05T06:39:42.011437+0000","last_active":"2024-02-05T06:42:11.525027+0000","last_peered":"2024-02-05T06:42:11.525027+0000","last_clean":"2024-02-05T06:42:11.525027+0000","last_became_active":"2024-02-05T06:39:42.011116+0000","last_became_peered":"2024-02-05T06:39:42.011116+0000","last_unstale":"2024-02-05T06:42:11.525027+0000","last_undegraded":"2024-02-05T06:42:11.525027+0000","last_fullsized":"2024-02-05T06:42:11.525027+0000","mapping_epoch":185,"log_start":"224'3700","ondisk_log_start":"224'3700","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":4467,"log_dups_size":0,"ondisk_log_size":4467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201261056,"num_objects":3071,"num_object_clones":0,"num_object_copies":9213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8167,"num_write_kb":359616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":544,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.061380+0000","last_change":"2024-02-05T06:40:30.082941+0000","last_active":"2024-02-05T06:42:11.061380+0000","last_peered":"2024-02-05T06:42:11.061380+0000","last_clean":"2024-02-05T06:42:11.061380+0000","last_became_active":"2024-02-05T06:40:30.082467+0000","last_became_peered":"2024-02-05T06:40:30.082467+0000","last_unstale":"2024-02-05T06:42:11.061380+0000","last_undegraded":"2024-02-05T06:42:11.061380+0000","last_fullsized":"2024-02-05T06:42:11.061380+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:39:09.930291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:39:09.930291+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:13.576798+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":123,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.062474+0000","last_change":"2024-02-05T06:40:27.078458+0000","last_active":"2024-02-05T06:42:11.062474+0000","last_peered":"2024-02-05T06:42:11.062474+0000","last_clean":"2024-02-05T06:42:11.062474+0000","last_became_active":"2024-02-05T06:40:27.077966+0000","last_became_peered":"2024-02-05T06:40:27.077966+0000","last_unstale":"2024-02-05T06:42:11.062474+0000","last_undegraded":"2024-02-05T06:42:11.062474+0000","last_fullsized":"2024-02-05T06:42:11.062474+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:36.461211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:36.461211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:13.576813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":430,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.064144+0000","last_change":"2024-02-05T06:39:27.282286+0000","last_active":"2024-02-05T06:42:11.064144+0000","last_peered":"2024-02-05T06:42:11.064144+0000","last_clean":"2024-02-05T06:42:11.064144+0000","last_became_active":"2024-02-05T06:33:29.551549+0000","last_became_peered":"2024-02-05T06:33:29.551549+0000","last_unstale":"2024-02-05T06:42:11.064144+0000","last_undegraded":"2024-02-05T06:42:11.064144+0000","last_fullsized":"2024-02-05T06:42:11.064144+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:27.282189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:39:27.282189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":445,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.068069+0000","last_change":"2024-02-05T06:41:30.515720+0000","last_active":"2024-02-05T06:42:11.068069+0000","last_peered":"2024-02-05T06:42:11.068069+0000","last_clean":"2024-02-05T06:42:11.068069+0000","last_became_active":"2024-02-05T06:33:28.542977+0000","last_became_peered":"2024-02-05T06:33:28.542977+0000","last_unstale":"2024-02-05T06:42:11.068069+0000","last_undegraded":"2024-02-05T06:42:11.068069+0000","last_fullsized":"2024-02-05T06:42:11.068069+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:41:30.515154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:41:30.515154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:55.243926+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00091640799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'13","reported_seq":514,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.063477+0000","last_change":"2024-02-05T06:41:43.445306+0000","last_active":"2024-02-05T06:42:11.063477+0000","last_peered":"2024-02-05T06:42:11.063477+0000","last_clean":"2024-02-05T06:42:11.063477+0000","last_became_active":"2024-02-05T06:41:43.420458+0000","last_became_peered":"2024-02-05T06:41:43.420458+0000","last_unstale":"2024-02-05T06:42:11.063477+0000","last_undegraded":"2024-02-05T06:42:11.063477+0000","last_fullsized":"2024-02-05T06:42:11.063477+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-02-05T06:39:33.280115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:33.280115+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":459,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.068506+0000","last_change":"2024-02-05T06:41:44.437195+0000","last_active":"2024-02-05T06:42:11.068506+0000","last_peered":"2024-02-05T06:42:11.068506+0000","last_clean":"2024-02-05T06:42:11.068506+0000","last_became_active":"2024-02-05T06:41:43.412387+0000","last_became_peered":"2024-02-05T06:41:43.412387+0000","last_unstale":"2024-02-05T06:42:11.068506+0000","last_undegraded":"2024-02-05T06:42:11.068506+0000","last_fullsized":"2024-02-05T06:42:11.068506+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:41:44.437101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:33.914422+0000","last_clean_scrub_stamp":"2024-02-05T06:41:44.437101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:50.390003+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"262'8070","reported_seq":20235,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.317086+0000","last_change":"2024-02-05T06:41:58.453233+0000","last_active":"2024-02-05T06:42:11.317086+0000","last_peered":"2024-02-05T06:42:11.317086+0000","last_clean":"2024-02-05T06:42:11.317086+0000","last_became_active":"2024-02-05T06:41:58.437685+0000","last_became_peered":"2024-02-05T06:41:58.437685+0000","last_unstale":"2024-02-05T06:42:11.317086+0000","last_undegraded":"2024-02-05T06:42:11.317086+0000","last_fullsized":"2024-02-05T06:42:11.317086+0000","mapping_epoch":254,"log_start":"204'2700","ondisk_log_start":"204'2700","created":185,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":5370,"log_dups_size":0,"ondisk_log_size":5370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:14.227742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196083712,"num_objects":2992,"num_object_clones":0,"num_object_copies":8976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8070,"num_write_kb":353984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5987,"num_bytes_recovered":392364032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":39,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.060892+0000","last_change":"2024-02-05T06:41:43.409089+0000","last_active":"2024-02-05T06:42:11.060892+0000","last_peered":"2024-02-05T06:42:11.060892+0000","last_clean":"2024-02-05T06:42:11.060892+0000","last_became_active":"2024-02-05T06:41:43.408773+0000","last_became_peered":"2024-02-05T06:41:43.408773+0000","last_unstale":"2024-02-05T06:42:11.060892+0000","last_undegraded":"2024-02-05T06:42:11.060892+0000","last_fullsized":"2024-02-05T06:42:11.060892+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:38:17.284669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:38:17.284669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":544,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.062262+0000","last_change":"2024-02-05T06:40:30.081724+0000","last_active":"2024-02-05T06:42:11.062262+0000","last_peered":"2024-02-05T06:42:11.062262+0000","last_clean":"2024-02-05T06:42:11.062262+0000","last_became_active":"2024-02-05T06:40:30.081178+0000","last_became_peered":"2024-02-05T06:40:30.081178+0000","last_unstale":"2024-02-05T06:42:11.062262+0000","last_undegraded":"2024-02-05T06:42:11.062262+0000","last_fullsized":"2024-02-05T06:42:11.062262+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:39:08.974109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:39:08.974109+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:13.576780+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":434,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.065312+0000","last_change":"2024-02-05T06:41:43.411652+0000","last_active":"2024-02-05T06:42:11.065312+0000","last_peered":"2024-02-05T06:42:11.065312+0000","last_clean":"2024-02-05T06:42:11.065312+0000","last_became_active":"2024-02-05T06:41:43.411335+0000","last_became_peered":"2024-02-05T06:41:43.411335+0000","last_unstale":"2024-02-05T06:42:11.065312+0000","last_undegraded":"2024-02-05T06:42:11.065312+0000","last_fullsized":"2024-02-05T06:42:11.065312+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:38:09.201827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:38:09.201827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"208'20","reported_seq":650,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.074855+0000","last_change":"2024-02-05T06:40:27.080702+0000","last_active":"2024-02-05T06:42:11.074855+0000","last_peered":"2024-02-05T06:42:11.074855+0000","last_clean":"2024-02-05T06:42:11.074855+0000","last_became_active":"2024-02-05T06:40:27.061111+0000","last_became_peered":"2024-02-05T06:40:27.061111+0000","last_unstale":"2024-02-05T06:42:11.074855+0000","last_undegraded":"2024-02-05T06:42:11.074855+0000","last_fullsized":"2024-02-05T06:42:11.074855+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'19","last_scrub_stamp":"2024-02-05T06:39:09.538463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:09.538463+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:14.227651+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":137,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.061894+0000","last_change":"2024-02-05T06:41:48.777261+0000","last_active":"2024-02-05T06:42:11.061894+0000","last_peered":"2024-02-05T06:42:11.061894+0000","last_clean":"2024-02-05T06:42:11.061894+0000","last_became_active":"2024-02-05T06:41:43.418198+0000","last_became_peered":"2024-02-05T06:41:43.418198+0000","last_unstale":"2024-02-05T06:42:11.061894+0000","last_undegraded":"2024-02-05T06:42:11.061894+0000","last_fullsized":"2024-02-05T06:42:11.061894+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:41:48.777030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:41:48.777030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:54.334201+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"262'8137","reported_seq":13367,"reported_epoch":262,"state":"active+remapped+backfilling","last_fresh":"2024-02-05T06:42:11.602330+0000","last_change":"2024-02-05T06:42:09.925676+0000","last_active":"2024-02-05T06:42:11.602330+0000","last_peered":"2024-02-05T06:42:11.602330+0000","last_clean":"2024-02-05T06:41:42.394363+0000","last_became_active":"2024-02-05T06:41:44.561618+0000","last_became_peered":"2024-02-05T06:41:44.561618+0000","last_unstale":"2024-02-05T06:42:11.602330+0000","last_undegraded":"2024-02-05T06:42:11.602330+0000","last_fullsized":"2024-02-05T06:42:11.602330+0000","mapping_epoch":248,"log_start":"216'3300","ondisk_log_start":"216'3300","created":185,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":4837,"log_dups_size":0,"ondisk_log_size":4837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194183168,"num_objects":2963,"num_object_clones":0,"num_object_copies":8889,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2392,"num_objects_unfound":0,"num_objects_dirty":2963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8137,"num_write_kb":355200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2523,"num_bytes_recovered":165281792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,3],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":2963}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":585,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.062964+0000","last_change":"2024-02-05T06:39:24.339609+0000","last_active":"2024-02-05T06:42:11.062964+0000","last_peered":"2024-02-05T06:42:11.062964+0000","last_clean":"2024-02-05T06:42:11.062964+0000","last_became_active":"2024-02-05T06:37:37.137754+0000","last_became_peered":"2024-02-05T06:37:37.137754+0000","last_unstale":"2024-02-05T06:42:11.062964+0000","last_undegraded":"2024-02-05T06:42:11.062964+0000","last_fullsized":"2024-02-05T06:42:11.062964+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:39:24.339522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:39:24.339522+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":438,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.061599+0000","last_change":"2024-02-05T06:41:44.353773+0000","last_active":"2024-02-05T06:42:11.061599+0000","last_peered":"2024-02-05T06:42:11.061599+0000","last_clean":"2024-02-05T06:42:11.061599+0000","last_became_active":"2024-02-05T06:41:43.410910+0000","last_became_peered":"2024-02-05T06:41:43.410910+0000","last_unstale":"2024-02-05T06:42:11.061599+0000","last_undegraded":"2024-02-05T06:42:11.061599+0000","last_fullsized":"2024-02-05T06:42:11.061599+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:41:44.353692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:41:44.353692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:58.496929+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":435,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.060235+0000","last_change":"2024-02-05T06:36:54.973030+0000","last_active":"2024-02-05T06:42:11.060235+0000","last_peered":"2024-02-05T06:42:11.060235+0000","last_clean":"2024-02-05T06:42:11.060235+0000","last_became_active":"2024-02-05T06:36:54.972264+0000","last_became_peered":"2024-02-05T06:36:54.972264+0000","last_unstale":"2024-02-05T06:42:11.060235+0000","last_undegraded":"2024-02-05T06:42:11.060235+0000","last_fullsized":"2024-02-05T06:42:11.060235+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:24.213373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:24.213373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":232,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.062303+0000","last_change":"2024-02-05T06:40:27.193820+0000","last_active":"2024-02-05T06:42:11.062303+0000","last_peered":"2024-02-05T06:42:11.062303+0000","last_clean":"2024-02-05T06:42:11.062303+0000","last_became_active":"2024-02-05T06:40:27.071509+0000","last_became_peered":"2024-02-05T06:40:27.071509+0000","last_unstale":"2024-02-05T06:42:11.062303+0000","last_undegraded":"2024-02-05T06:42:11.062303+0000","last_fullsized":"2024-02-05T06:42:11.062303+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T06:40:03.205189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:40:03.205189+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:13.576888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074151600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":111,"num_read_kb":106,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":437,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.062534+0000","last_change":"2024-02-05T06:36:29.231478+0000","last_active":"2024-02-05T06:42:11.062534+0000","last_peered":"2024-02-05T06:42:11.062534+0000","last_clean":"2024-02-05T06:42:11.062534+0000","last_became_active":"2024-02-05T06:33:31.564604+0000","last_became_peered":"2024-02-05T06:33:31.564604+0000","last_unstale":"2024-02-05T06:42:11.062534+0000","last_undegraded":"2024-02-05T06:42:11.062534+0000","last_fullsized":"2024-02-05T06:42:11.062534+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:36:29.231352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:36:29.231352+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:16.551453+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":39,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.060997+0000","last_change":"2024-02-05T06:41:43.418140+0000","last_active":"2024-02-05T06:42:11.060997+0000","last_peered":"2024-02-05T06:42:11.060997+0000","last_clean":"2024-02-05T06:42:11.060997+0000","last_became_active":"2024-02-05T06:41:43.417751+0000","last_became_peered":"2024-02-05T06:41:43.417751+0000","last_unstale":"2024-02-05T06:42:11.060997+0000","last_undegraded":"2024-02-05T06:42:11.060997+0000","last_fullsized":"2024-02-05T06:42:11.060997+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:41:27.147994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:41:27.147994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:32.006812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"262'8168","reported_seq":8333,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.408582+0000","last_change":"2024-02-05T06:39:42.682421+0000","last_active":"2024-02-05T06:42:11.408582+0000","last_peered":"2024-02-05T06:42:11.408582+0000","last_clean":"2024-02-05T06:42:11.408582+0000","last_became_active":"2024-02-05T06:39:42.682109+0000","last_became_peered":"2024-02-05T06:39:42.682109+0000","last_unstale":"2024-02-05T06:42:11.408582+0000","last_undegraded":"2024-02-05T06:42:11.408582+0000","last_fullsized":"2024-02-05T06:42:11.408582+0000","mapping_epoch":185,"log_start":"224'3700","ondisk_log_start":"224'3700","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":4468,"log_dups_size":0,"ondisk_log_size":4468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198574080,"num_objects":3030,"num_object_clones":0,"num_object_copies":9090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8168,"num_write_kb":358336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":590,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.067593+0000","last_change":"2024-02-05T06:39:12.528803+0000","last_active":"2024-02-05T06:42:11.067593+0000","last_peered":"2024-02-05T06:42:11.067593+0000","last_clean":"2024-02-05T06:42:11.067593+0000","last_became_active":"2024-02-05T06:37:37.140011+0000","last_became_peered":"2024-02-05T06:37:37.140011+0000","last_unstale":"2024-02-05T06:42:11.067593+0000","last_undegraded":"2024-02-05T06:42:11.067593+0000","last_fullsized":"2024-02-05T06:42:11.067593+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:39:12.528724+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:38.778568+0000","last_clean_scrub_stamp":"2024-02-05T06:39:12.528724+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:14.227688+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":418,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.061345+0000","last_change":"2024-02-05T06:39:22.330018+0000","last_active":"2024-02-05T06:42:11.061345+0000","last_peered":"2024-02-05T06:42:11.061345+0000","last_clean":"2024-02-05T06:42:11.061345+0000","last_became_active":"2024-02-05T06:33:35.664858+0000","last_became_peered":"2024-02-05T06:33:35.664858+0000","last_unstale":"2024-02-05T06:42:11.061345+0000","last_undegraded":"2024-02-05T06:42:11.061345+0000","last_fullsized":"2024-02-05T06:42:11.061345+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:22.329914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:22.329914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"66'1","reported_seq":464,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.061796+0000","last_change":"2024-02-05T06:41:43.426423+0000","last_active":"2024-02-05T06:42:11.061796+0000","last_peered":"2024-02-05T06:42:11.061796+0000","last_clean":"2024-02-05T06:42:11.061796+0000","last_became_active":"2024-02-05T06:41:43.411404+0000","last_became_peered":"2024-02-05T06:41:43.411404+0000","last_unstale":"2024-02-05T06:42:11.061796+0000","last_undegraded":"2024-02-05T06:42:11.061796+0000","last_fullsized":"2024-02-05T06:42:11.061796+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:39:34.241779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:39:34.241779+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'19","reported_seq":551,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.061437+0000","last_change":"2024-02-05T06:41:43.437079+0000","last_active":"2024-02-05T06:42:11.061437+0000","last_peered":"2024-02-05T06:42:11.061437+0000","last_clean":"2024-02-05T06:42:11.061437+0000","last_became_active":"2024-02-05T06:41:43.409664+0000","last_became_peered":"2024-02-05T06:41:43.409664+0000","last_unstale":"2024-02-05T06:42:11.061437+0000","last_undegraded":"2024-02-05T06:42:11.061437+0000","last_fullsized":"2024-02-05T06:42:11.061437+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-02-05T06:39:19.843545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:19.843545+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":433,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.067455+0000","last_change":"2024-02-05T06:41:43.411871+0000","last_active":"2024-02-05T06:42:11.067455+0000","last_peered":"2024-02-05T06:42:11.067455+0000","last_clean":"2024-02-05T06:42:11.067455+0000","last_became_active":"2024-02-05T06:41:43.411550+0000","last_became_peered":"2024-02-05T06:41:43.411550+0000","last_unstale":"2024-02-05T06:42:11.067455+0000","last_undegraded":"2024-02-05T06:42:11.067455+0000","last_fullsized":"2024-02-05T06:42:11.067455+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:32.449390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:32.449390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:14.227819+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"262'8321","reported_seq":8487,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.638751+0000","last_change":"2024-02-05T06:39:42.684749+0000","last_active":"2024-02-05T06:42:11.638751+0000","last_peered":"2024-02-05T06:42:11.638751+0000","last_clean":"2024-02-05T06:42:11.638751+0000","last_became_active":"2024-02-05T06:39:42.684454+0000","last_became_peered":"2024-02-05T06:39:42.684454+0000","last_unstale":"2024-02-05T06:42:11.638751+0000","last_undegraded":"2024-02-05T06:42:11.638751+0000","last_fullsized":"2024-02-05T06:42:11.638751+0000","mapping_epoch":185,"log_start":"207'2900","ondisk_log_start":"207'2900","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":5421,"log_dups_size":0,"ondisk_log_size":5421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:16.551644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200343552,"num_objects":3057,"num_object_clones":0,"num_object_copies":9171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8321,"num_write_kb":364096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":617,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.067745+0000","last_change":"2024-02-05T06:40:30.085803+0000","last_active":"2024-02-05T06:42:11.067745+0000","last_peered":"2024-02-05T06:42:11.067745+0000","last_clean":"2024-02-05T06:42:11.067745+0000","last_became_active":"2024-02-05T06:40:30.085438+0000","last_became_peered":"2024-02-05T06:40:30.085438+0000","last_unstale":"2024-02-05T06:42:11.067745+0000","last_undegraded":"2024-02-05T06:42:11.067745+0000","last_fullsized":"2024-02-05T06:42:11.067745+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:39:30.440075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:39:30.440075+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:14.227783+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":411,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.060084+0000","last_change":"2024-02-05T06:39:30.865329+0000","last_active":"2024-02-05T06:42:11.060084+0000","last_peered":"2024-02-05T06:42:11.060084+0000","last_clean":"2024-02-05T06:42:11.060084+0000","last_became_active":"2024-02-05T06:33:35.600485+0000","last_became_peered":"2024-02-05T06:33:35.600485+0000","last_unstale":"2024-02-05T06:42:11.060084+0000","last_undegraded":"2024-02-05T06:42:11.060084+0000","last_fullsized":"2024-02-05T06:42:11.060084+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:30.865200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:30.865200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":39,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.060783+0000","last_change":"2024-02-05T06:41:43.415244+0000","last_active":"2024-02-05T06:42:11.060783+0000","last_peered":"2024-02-05T06:42:11.060783+0000","last_clean":"2024-02-05T06:42:11.060783+0000","last_became_active":"2024-02-05T06:41:43.411732+0000","last_became_peered":"2024-02-05T06:41:43.411732+0000","last_unstale":"2024-02-05T06:42:11.060783+0000","last_undegraded":"2024-02-05T06:42:11.060783+0000","last_fullsized":"2024-02-05T06:42:11.060783+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:28.353956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:28.353956+0000","last_clean_scrub_stamp":"2024-02-05T06:39:28.353956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"73'10","reported_seq":513,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.063870+0000","last_change":"2024-02-05T06:41:43.448108+0000","last_active":"2024-02-05T06:42:11.063870+0000","last_peered":"2024-02-05T06:42:11.063870+0000","last_clean":"2024-02-05T06:42:11.063870+0000","last_became_active":"2024-02-05T06:41:43.421030+0000","last_became_peered":"2024-02-05T06:41:43.421030+0000","last_unstale":"2024-02-05T06:42:11.063870+0000","last_undegraded":"2024-02-05T06:42:11.063870+0000","last_fullsized":"2024-02-05T06:42:11.063870+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-02-05T06:39:25.306103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:25.306103+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"249'10","reported_seq":548,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.061037+0000","last_change":"2024-02-05T06:41:43.444768+0000","last_active":"2024-02-05T06:42:11.061037+0000","last_peered":"2024-02-05T06:42:11.061037+0000","last_clean":"2024-02-05T06:42:11.061037+0000","last_became_active":"2024-02-05T06:41:43.423638+0000","last_became_peered":"2024-02-05T06:41:43.423638+0000","last_unstale":"2024-02-05T06:42:11.061037+0000","last_undegraded":"2024-02-05T06:42:11.061037+0000","last_fullsized":"2024-02-05T06:42:11.061037+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-02-05T06:36:32.174396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:36:32.174396+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"262'7944","reported_seq":22581,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.169620+0000","last_change":"2024-02-05T06:42:11.136259+0000","last_active":"2024-02-05T06:42:11.169620+0000","last_peered":"2024-02-05T06:42:11.169620+0000","last_clean":"2024-02-05T06:42:11.169620+0000","last_became_active":"2024-02-05T06:42:11.077659+0000","last_became_peered":"2024-02-05T06:42:11.077659+0000","last_unstale":"2024-02-05T06:42:11.169620+0000","last_undegraded":"2024-02-05T06:42:11.169620+0000","last_fullsized":"2024-02-05T06:42:11.169620+0000","mapping_epoch":261,"log_start":"213'3006","ondisk_log_start":"213'3006","created":185,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":4938,"log_dups_size":0,"ondisk_log_size":4938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194314264,"num_objects":2966,"num_object_clones":0,"num_object_copies":8898,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2966,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7944,"num_write_kb":349057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7223,"num_bytes_recovered":473169944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":628,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.065245+0000","last_change":"2024-02-05T06:41:46.054285+0000","last_active":"2024-02-05T06:42:11.065245+0000","last_peered":"2024-02-05T06:42:11.065245+0000","last_clean":"2024-02-05T06:42:11.065245+0000","last_became_active":"2024-02-05T06:41:46.053837+0000","last_became_peered":"2024-02-05T06:41:46.053837+0000","last_unstale":"2024-02-05T06:42:11.065245+0000","last_undegraded":"2024-02-05T06:42:11.065245+0000","last_fullsized":"2024-02-05T06:42:11.065245+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:38:10.176507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:38:10.176507+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":123,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.061709+0000","last_change":"2024-02-05T06:40:27.099496+0000","last_active":"2024-02-05T06:42:11.061709+0000","last_peered":"2024-02-05T06:42:11.061709+0000","last_clean":"2024-02-05T06:42:11.061709+0000","last_became_active":"2024-02-05T06:40:27.099168+0000","last_became_peered":"2024-02-05T06:40:27.099168+0000","last_unstale":"2024-02-05T06:42:11.061709+0000","last_undegraded":"2024-02-05T06:42:11.061709+0000","last_fullsized":"2024-02-05T06:42:11.061709+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:40:06.734454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:40:06.734454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:13.576855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"95'2","reported_seq":445,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.062549+0000","last_change":"2024-02-05T06:39:06.955815+0000","last_active":"2024-02-05T06:42:11.062549+0000","last_peered":"2024-02-05T06:42:11.062549+0000","last_clean":"2024-02-05T06:42:11.062549+0000","last_became_active":"2024-02-05T06:36:54.975093+0000","last_became_peered":"2024-02-05T06:36:54.975093+0000","last_unstale":"2024-02-05T06:42:11.062549+0000","last_undegraded":"2024-02-05T06:42:11.062549+0000","last_fullsized":"2024-02-05T06:42:11.062549+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-05T06:39:06.955629+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-05T06:35:25.616708+0000","last_clean_scrub_stamp":"2024-02-05T06:39:06.955629+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'14","reported_seq":367,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.115950+0000","last_change":"2024-02-05T06:41:43.435953+0000","last_active":"2024-02-05T06:42:11.115950+0000","last_peered":"2024-02-05T06:42:11.115950+0000","last_clean":"2024-02-05T06:42:11.115950+0000","last_became_active":"2024-02-05T06:41:43.411540+0000","last_became_peered":"2024-02-05T06:41:43.411540+0000","last_unstale":"2024-02-05T06:42:11.115950+0000","last_undegraded":"2024-02-05T06:42:11.115950+0000","last_fullsized":"2024-02-05T06:42:11.115950+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'14","last_scrub_stamp":"2024-02-05T06:40:47.627644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:40:47.627644+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"119'10","reported_seq":549,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.066052+0000","last_change":"2024-02-05T06:38:23.177410+0000","last_active":"2024-02-05T06:42:11.066052+0000","last_peered":"2024-02-05T06:42:11.066052+0000","last_clean":"2024-02-05T06:42:11.066052+0000","last_became_active":"2024-02-05T06:36:54.954819+0000","last_became_peered":"2024-02-05T06:36:54.954819+0000","last_unstale":"2024-02-05T06:42:11.066052+0000","last_undegraded":"2024-02-05T06:42:11.066052+0000","last_fullsized":"2024-02-05T06:42:11.066052+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-02-05T06:38:23.177286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:38:23.177286+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"262'8142","reported_seq":16063,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.659872+0000","last_change":"2024-02-05T06:41:33.485142+0000","last_active":"2024-02-05T06:42:11.659872+0000","last_peered":"2024-02-05T06:42:11.659872+0000","last_clean":"2024-02-05T06:42:11.659872+0000","last_became_active":"2024-02-05T06:41:33.479274+0000","last_became_peered":"2024-02-05T06:41:33.479274+0000","last_unstale":"2024-02-05T06:42:11.659872+0000","last_undegraded":"2024-02-05T06:42:11.659872+0000","last_fullsized":"2024-02-05T06:42:11.659872+0000","mapping_epoch":243,"log_start":"204'2700","ondisk_log_start":"204'2700","created":185,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":5442,"log_dups_size":0,"ondisk_log_size":5442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:16.551677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203030528,"num_objects":3098,"num_object_clones":0,"num_object_copies":9294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8142,"num_write_kb":359680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3870,"num_bytes_recovered":253624320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":585,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.115815+0000","last_change":"2024-02-05T06:41:28.466905+0000","last_active":"2024-02-05T06:42:11.115815+0000","last_peered":"2024-02-05T06:42:11.115815+0000","last_clean":"2024-02-05T06:42:11.115815+0000","last_became_active":"2024-02-05T06:37:37.135073+0000","last_became_peered":"2024-02-05T06:37:37.135073+0000","last_unstale":"2024-02-05T06:42:11.115815+0000","last_undegraded":"2024-02-05T06:42:11.115815+0000","last_fullsized":"2024-02-05T06:42:11.115815+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:41:28.466576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:41:28.466576+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:55.243075+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":307,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.116746+0000","last_change":"2024-02-05T06:39:24.090774+0000","last_active":"2024-02-05T06:42:11.116746+0000","last_peered":"2024-02-05T06:42:11.116746+0000","last_clean":"2024-02-05T06:42:11.116746+0000","last_became_active":"2024-02-05T06:36:54.958644+0000","last_became_peered":"2024-02-05T06:36:54.958644+0000","last_unstale":"2024-02-05T06:42:11.116746+0000","last_undegraded":"2024-02-05T06:42:11.116746+0000","last_fullsized":"2024-02-05T06:42:11.116746+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:24.090670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:24.090670+0000","last_clean_scrub_stamp":"2024-02-05T06:39:24.090670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":440,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.068322+0000","last_change":"2024-02-05T06:40:27.094061+0000","last_active":"2024-02-05T06:42:11.068322+0000","last_peered":"2024-02-05T06:42:11.068322+0000","last_clean":"2024-02-05T06:42:11.068322+0000","last_became_active":"2024-02-05T06:40:27.093739+0000","last_became_peered":"2024-02-05T06:40:27.093739+0000","last_unstale":"2024-02-05T06:42:11.068322+0000","last_undegraded":"2024-02-05T06:42:11.068322+0000","last_fullsized":"2024-02-05T06:42:11.068322+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:38:28.150695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:38:28.150695+0000","last_clean_scrub_stamp":"2024-02-05T06:38:28.150695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"73'18","reported_seq":498,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.065992+0000","last_change":"2024-02-05T06:40:46.408120+0000","last_active":"2024-02-05T06:42:11.065992+0000","last_peered":"2024-02-05T06:42:11.065992+0000","last_clean":"2024-02-05T06:42:11.065992+0000","last_became_active":"2024-02-05T06:40:27.277747+0000","last_became_peered":"2024-02-05T06:40:27.277747+0000","last_unstale":"2024-02-05T06:42:11.065992+0000","last_undegraded":"2024-02-05T06:42:11.065992+0000","last_fullsized":"2024-02-05T06:42:11.065992+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-02-05T06:39:12.382954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:12.382954+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":327,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.114545+0000","last_change":"2024-02-05T06:40:27.074218+0000","last_active":"2024-02-05T06:42:11.114545+0000","last_peered":"2024-02-05T06:42:11.114545+0000","last_clean":"2024-02-05T06:42:11.114545+0000","last_became_active":"2024-02-05T06:40:27.073889+0000","last_became_peered":"2024-02-05T06:40:27.073889+0000","last_unstale":"2024-02-05T06:42:11.114545+0000","last_undegraded":"2024-02-05T06:42:11.114545+0000","last_fullsized":"2024-02-05T06:42:11.114545+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:40:02.899805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:40:02.899805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"262'8268","reported_seq":12319,"reported_epoch":262,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T06:42:11.541536+0000","last_change":"2024-02-05T06:41:44.607687+0000","last_active":"2024-02-05T06:42:11.541536+0000","last_peered":"2024-02-05T06:42:11.541536+0000","last_clean":"2024-02-05T06:41:42.393649+0000","last_became_active":"2024-02-05T06:41:44.535422+0000","last_became_peered":"2024-02-05T06:41:44.535422+0000","last_unstale":"2024-02-05T06:42:11.541536+0000","last_undegraded":"2024-02-05T06:42:11.541536+0000","last_fullsized":"2024-02-05T06:42:11.541536+0000","mapping_epoch":248,"log_start":"219'3500","ondisk_log_start":"219'3500","created":185,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":4768,"log_dups_size":0,"ondisk_log_size":4768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199753728,"num_objects":3048,"num_object_clones":0,"num_object_copies":9144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3045,"num_objects_unfound":0,"num_objects_dirty":3048,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8268,"num_write_kb":362112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1931,"num_bytes_recovered":126550016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,6,2],"avail_no_missing":["1","2","6"],"object_location_counts":[{"shards":"1,2,6","objects":3048}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":458,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.059824+0000","last_change":"2024-02-05T06:41:46.054326+0000","last_active":"2024-02-05T06:42:11.059824+0000","last_peered":"2024-02-05T06:42:11.059824+0000","last_clean":"2024-02-05T06:42:11.059824+0000","last_became_active":"2024-02-05T06:41:46.053993+0000","last_became_peered":"2024-02-05T06:41:46.053993+0000","last_unstale":"2024-02-05T06:42:11.059824+0000","last_undegraded":"2024-02-05T06:42:11.059824+0000","last_fullsized":"2024-02-05T06:42:11.059824+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:41:28.452682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:41:28.452682+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:56.394202+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":39,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.060755+0000","last_change":"2024-02-05T06:41:43.420562+0000","last_active":"2024-02-05T06:42:11.060755+0000","last_peered":"2024-02-05T06:42:11.060755+0000","last_clean":"2024-02-05T06:42:11.060755+0000","last_became_active":"2024-02-05T06:41:43.418066+0000","last_became_peered":"2024-02-05T06:41:43.418066+0000","last_unstale":"2024-02-05T06:42:11.060755+0000","last_undegraded":"2024-02-05T06:42:11.060755+0000","last_fullsized":"2024-02-05T06:42:11.060755+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:11.464973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:39:11.464973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"249'12","reported_seq":396,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.117356+0000","last_change":"2024-02-05T06:41:43.429561+0000","last_active":"2024-02-05T06:42:11.117356+0000","last_peered":"2024-02-05T06:42:11.117356+0000","last_clean":"2024-02-05T06:42:11.117356+0000","last_became_active":"2024-02-05T06:41:43.411973+0000","last_became_peered":"2024-02-05T06:41:43.411973+0000","last_unstale":"2024-02-05T06:42:11.117356+0000","last_undegraded":"2024-02-05T06:42:11.117356+0000","last_fullsized":"2024-02-05T06:42:11.117356+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-02-05T06:40:06.921141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:40:06.921141+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":434,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.066005+0000","last_change":"2024-02-05T06:40:27.214932+0000","last_active":"2024-02-05T06:42:11.066005+0000","last_peered":"2024-02-05T06:42:11.066005+0000","last_clean":"2024-02-05T06:42:11.066005+0000","last_became_active":"2024-02-05T06:40:27.213439+0000","last_became_peered":"2024-02-05T06:40:27.213439+0000","last_unstale":"2024-02-05T06:42:11.066005+0000","last_undegraded":"2024-02-05T06:42:11.066005+0000","last_fullsized":"2024-02-05T06:42:11.066005+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:23.372627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:39:23.372627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"73'15","reported_seq":204,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.062500+0000","last_change":"2024-02-05T06:39:13.334762+0000","last_active":"2024-02-05T06:42:11.062500+0000","last_peered":"2024-02-05T06:42:11.062500+0000","last_clean":"2024-02-05T06:42:11.062500+0000","last_became_active":"2024-02-05T06:39:13.257978+0000","last_became_peered":"2024-02-05T06:39:13.257978+0000","last_unstale":"2024-02-05T06:42:11.062500+0000","last_undegraded":"2024-02-05T06:42:11.062500+0000","last_fullsized":"2024-02-05T06:42:11.062500+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-02-05T06:39:11.376185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:39:11.376185+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2","version":"262'8135","reported_seq":12040,"reported_epoch":262,"state":"active+remapped+backfill_toofull","last_fresh":"2024-02-05T06:42:11.232433+0000","last_change":"2024-02-05T06:41:44.527048+0000","last_active":"2024-02-05T06:42:11.232433+0000","last_peered":"2024-02-05T06:42:11.232433+0000","last_clean":"2024-02-05T06:41:42.384775+0000","last_became_active":"2024-02-05T06:41:44.514311+0000","last_became_peered":"2024-02-05T06:41:44.514311+0000","last_unstale":"2024-02-05T06:42:11.232433+0000","last_undegraded":"2024-02-05T06:42:11.232433+0000","last_fullsized":"2024-02-05T06:42:11.232433+0000","mapping_epoch":248,"log_start":"220'3500","ondisk_log_start":"220'3500","created":185,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":4635,"log_dups_size":0,"ondisk_log_size":4635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206241792,"num_objects":3147,"num_object_clones":0,"num_object_copies":9441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3147,"num_objects_unfound":0,"num_objects_dirty":3147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8135,"num_write_kb":361024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1861,"num_bytes_recovered":121962496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,3],"avail_no_missing":["4","0","3"],"object_location_counts":[{"shards":"0,3,4","objects":3147}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":427,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.065413+0000","last_change":"2024-02-05T06:41:43.410912+0000","last_active":"2024-02-05T06:42:11.065413+0000","last_peered":"2024-02-05T06:42:11.065413+0000","last_clean":"2024-02-05T06:42:11.065413+0000","last_became_active":"2024-02-05T06:41:43.410600+0000","last_became_peered":"2024-02-05T06:41:43.410600+0000","last_unstale":"2024-02-05T06:42:11.065413+0000","last_undegraded":"2024-02-05T06:42:11.065413+0000","last_fullsized":"2024-02-05T06:42:11.065413+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:18.402553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:20.042534+0000","last_clean_scrub_stamp":"2024-02-05T06:39:18.402553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":39,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.060890+0000","last_change":"2024-02-05T06:41:43.418311+0000","last_active":"2024-02-05T06:42:11.060890+0000","last_peered":"2024-02-05T06:42:11.060890+0000","last_clean":"2024-02-05T06:42:11.060890+0000","last_became_active":"2024-02-05T06:41:43.417343+0000","last_became_peered":"2024-02-05T06:41:43.417343+0000","last_unstale":"2024-02-05T06:42:11.060890+0000","last_undegraded":"2024-02-05T06:42:11.060890+0000","last_fullsized":"2024-02-05T06:42:11.060890+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:18.379775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:18.379775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":149,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.060914+0000","last_change":"2024-02-05T06:41:33.228147+0000","last_active":"2024-02-05T06:42:11.060914+0000","last_peered":"2024-02-05T06:42:11.060914+0000","last_clean":"2024-02-05T06:42:11.060914+0000","last_became_active":"2024-02-05T06:40:27.086808+0000","last_became_peered":"2024-02-05T06:40:27.086808+0000","last_unstale":"2024-02-05T06:42:11.060914+0000","last_undegraded":"2024-02-05T06:42:11.060914+0000","last_fullsized":"2024-02-05T06:42:11.060914+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:41:33.228035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:41:33.228035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:37.434380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"115'16","reported_seq":319,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.117174+0000","last_change":"2024-02-05T06:41:26.381049+0000","last_active":"2024-02-05T06:42:11.117174+0000","last_peered":"2024-02-05T06:42:11.117174+0000","last_clean":"2024-02-05T06:42:11.117174+0000","last_became_active":"2024-02-05T06:36:54.958582+0000","last_became_peered":"2024-02-05T06:36:54.958582+0000","last_unstale":"2024-02-05T06:42:11.117174+0000","last_undegraded":"2024-02-05T06:42:11.117174+0000","last_fullsized":"2024-02-05T06:42:11.117174+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'16","last_scrub_stamp":"2024-02-05T06:41:26.380967+0000","last_deep_scrub":"115'16","last_deep_scrub_stamp":"2024-02-05T06:36:34.793243+0000","last_clean_scrub_stamp":"2024-02-05T06:41:26.380967+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:45.699274+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"262'8034","reported_seq":8200,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.447579+0000","last_change":"2024-02-05T06:39:42.682964+0000","last_active":"2024-02-05T06:42:11.447579+0000","last_peered":"2024-02-05T06:42:11.447579+0000","last_clean":"2024-02-05T06:42:11.447579+0000","last_became_active":"2024-02-05T06:39:42.682661+0000","last_became_peered":"2024-02-05T06:39:42.682661+0000","last_unstale":"2024-02-05T06:42:11.447579+0000","last_undegraded":"2024-02-05T06:42:11.447579+0000","last_fullsized":"2024-02-05T06:42:11.447579+0000","mapping_epoch":185,"log_start":"216'3200","ondisk_log_start":"216'3200","created":185,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":4834,"log_dups_size":0,"ondisk_log_size":4834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203816960,"num_objects":3110,"num_object_clones":0,"num_object_copies":9330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3110,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8034,"num_write_kb":356608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"83'128","reported_seq":462,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.059380+0000","last_change":"2024-02-05T06:41:46.053681+0000","last_active":"2024-02-05T06:42:11.059380+0000","last_peered":"2024-02-05T06:42:11.059380+0000","last_clean":"2024-02-05T06:42:11.059380+0000","last_became_active":"2024-02-05T06:41:46.053324+0000","last_became_peered":"2024-02-05T06:41:46.053324+0000","last_unstale":"2024-02-05T06:42:11.059380+0000","last_undegraded":"2024-02-05T06:42:11.059380+0000","last_fullsized":"2024-02-05T06:42:11.059380+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:41:29.496742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:41:29.496742+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:30.909388+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":435,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.068271+0000","last_change":"2024-02-05T06:41:34.538969+0000","last_active":"2024-02-05T06:42:11.068271+0000","last_peered":"2024-02-05T06:42:11.068271+0000","last_clean":"2024-02-05T06:42:11.068271+0000","last_became_active":"2024-02-05T06:36:54.958805+0000","last_became_peered":"2024-02-05T06:36:54.958805+0000","last_unstale":"2024-02-05T06:42:11.068271+0000","last_undegraded":"2024-02-05T06:42:11.068271+0000","last_fullsized":"2024-02-05T06:42:11.068271+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:41:34.538854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:41:34.538854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:38.425192+0000","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":433,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.061887+0000","last_change":"2024-02-05T06:39:09.427061+0000","last_active":"2024-02-05T06:42:11.061887+0000","last_peered":"2024-02-05T06:42:11.061887+0000","last_clean":"2024-02-05T06:42:11.061887+0000","last_became_active":"2024-02-05T06:36:54.958082+0000","last_became_peered":"2024-02-05T06:36:54.958082+0000","last_unstale":"2024-02-05T06:42:11.061887+0000","last_undegraded":"2024-02-05T06:42:11.061887+0000","last_fullsized":"2024-02-05T06:42:11.061887+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:09.426862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:39:09.426862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":447,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.067565+0000","last_change":"2024-02-05T06:39:35.429964+0000","last_active":"2024-02-05T06:42:11.067565+0000","last_peered":"2024-02-05T06:42:11.067565+0000","last_clean":"2024-02-05T06:42:11.067565+0000","last_became_active":"2024-02-05T06:39:13.240038+0000","last_became_peered":"2024-02-05T06:39:13.240038+0000","last_unstale":"2024-02-05T06:42:11.067565+0000","last_undegraded":"2024-02-05T06:42:11.067565+0000","last_fullsized":"2024-02-05T06:42:11.067565+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:35.429861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:39:35.429861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:14.227834+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"95'25","reported_seq":333,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.116302+0000","last_change":"2024-02-05T06:41:54.656081+0000","last_active":"2024-02-05T06:42:11.116302+0000","last_peered":"2024-02-05T06:42:11.116302+0000","last_clean":"2024-02-05T06:42:11.116302+0000","last_became_active":"2024-02-05T06:36:54.961719+0000","last_became_peered":"2024-02-05T06:36:54.961719+0000","last_unstale":"2024-02-05T06:42:11.116302+0000","last_undegraded":"2024-02-05T06:42:11.116302+0000","last_fullsized":"2024-02-05T06:42:11.116302+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'25","last_scrub_stamp":"2024-02-05T06:41:54.655986+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-02-05T06:35:26.274944+0000","last_clean_scrub_stamp":"2024-02-05T06:41:54.655986+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:54.821706+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"262'8087","reported_seq":17733,"reported_epoch":262,"state":"active+remapped+backfilling","last_fresh":"2024-02-05T06:42:12.010175+0000","last_change":"2024-02-05T06:41:57.031526+0000","last_active":"2024-02-05T06:42:12.010175+0000","last_peered":"2024-02-05T06:42:12.010175+0000","last_clean":"2024-02-05T06:41:42.381926+0000","last_became_active":"2024-02-05T06:41:44.549797+0000","last_became_peered":"2024-02-05T06:41:44.549797+0000","last_unstale":"2024-02-05T06:42:12.010175+0000","last_undegraded":"2024-02-05T06:42:12.010175+0000","last_fullsized":"2024-02-05T06:42:12.010175+0000","mapping_epoch":248,"log_start":"222'3600","ondisk_log_start":"222'3600","created":185,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:40.991488+0000","last_clean_scrub_stamp":"2024-02-05T06:39:40.991488+0000","objects_scrubbed":0,"log_size":4487,"log_dups_size":0,"ondisk_log_size":4487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197853184,"num_objects":3019,"num_object_clones":0,"num_object_copies":9057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":423,"num_objects_unfound":0,"num_objects_dirty":3019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8087,"num_write_kb":355392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4733,"num_bytes_recovered":310181888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,1,4],"avail_no_missing":["3","1","4"],"object_location_counts":[{"shards":"1,3,4","objects":3019}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":541,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.061490+0000","last_change":"2024-02-05T06:41:50.706286+0000","last_active":"2024-02-05T06:42:11.061490+0000","last_peered":"2024-02-05T06:42:11.061490+0000","last_clean":"2024-02-05T06:42:11.061490+0000","last_became_active":"2024-02-05T06:40:30.095202+0000","last_became_peered":"2024-02-05T06:40:30.095202+0000","last_unstale":"2024-02-05T06:42:11.061490+0000","last_undegraded":"2024-02-05T06:42:11.061490+0000","last_fullsized":"2024-02-05T06:42:11.061490+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:41:50.706159+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:41:50.706159+0000","last_clean_scrub_stamp":"2024-02-05T06:41:50.706159+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:42:55.020656+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":307,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.117034+0000","last_change":"2024-02-05T06:39:30.057231+0000","last_active":"2024-02-05T06:42:11.117034+0000","last_peered":"2024-02-05T06:42:11.117034+0000","last_clean":"2024-02-05T06:42:11.117034+0000","last_became_active":"2024-02-05T06:36:54.961560+0000","last_became_peered":"2024-02-05T06:36:54.961560+0000","last_unstale":"2024-02-05T06:42:11.117034+0000","last_undegraded":"2024-02-05T06:42:11.117034+0000","last_fullsized":"2024-02-05T06:42:11.117034+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:39:30.057099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:30.057099+0000","last_clean_scrub_stamp":"2024-02-05T06:39:30.057099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"249'13","reported_seq":581,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.068246+0000","last_change":"2024-02-05T06:41:43.442327+0000","last_active":"2024-02-05T06:42:11.068246+0000","last_peered":"2024-02-05T06:42:11.068246+0000","last_clean":"2024-02-05T06:42:11.068246+0000","last_became_active":"2024-02-05T06:41:43.413139+0000","last_became_peered":"2024-02-05T06:41:43.413139+0000","last_unstale":"2024-02-05T06:42:11.068246+0000","last_undegraded":"2024-02-05T06:42:11.068246+0000","last_fullsized":"2024-02-05T06:42:11.068246+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'12","last_scrub_stamp":"2024-02-05T06:41:31.480420+0000","last_deep_scrub":"92'9","last_deep_scrub_stamp":"2024-02-05T06:34:52.283780+0000","last_clean_scrub_stamp":"2024-02-05T06:41:31.480420+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:43:01.021266+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":433,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.065721+0000","last_change":"2024-02-05T06:36:54.971857+0000","last_active":"2024-02-05T06:42:11.065721+0000","last_peered":"2024-02-05T06:42:11.065721+0000","last_clean":"2024-02-05T06:42:11.065721+0000","last_became_active":"2024-02-05T06:36:54.971304+0000","last_became_peered":"2024-02-05T06:36:54.971304+0000","last_unstale":"2024-02-05T06:42:11.065721+0000","last_undegraded":"2024-02-05T06:42:11.065721+0000","last_fullsized":"2024-02-05T06:42:11.065721+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:36:37.506830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:36:37.506830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"93'15","reported_seq":327,"reported_epoch":262,"state":"active+clean","last_fresh":"2024-02-05T06:42:11.116476+0000","last_change":"2024-02-05T06:41:37.498015+0000","last_active":"2024-02-05T06:42:11.116476+0000","last_peered":"2024-02-05T06:42:11.116476+0000","last_clean":"2024-02-05T06:42:11.116476+0000","last_became_active":"2024-02-05T06:36:54.963620+0000","last_became_peered":"2024-02-05T06:36:54.963620+0000","last_unstale":"2024-02-05T06:42:11.116476+0000","last_undegraded":"2024-02-05T06:42:11.116476+0000","last_fullsized":"2024-02-05T06:42:11.116476+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'15","last_scrub_stamp":"2024-02-05T06:41:37.497882+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-02-05T06:35:04.584657+0000","last_clean_scrub_stamp":"2024-02-05T06:41:37.497882+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:43:01.944340+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":3184066584,"num_objects":48586,"num_object_clones":0,"num_object_copies":145758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12061,"num_objects_unfound":0,"num_objects_dirty":48586,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":129960,"num_write_kb":5713409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42646,"num_bytes_recovered":2794586136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10318262272,"data_stored":10318250056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80354,"ondisk_log_size":80354,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":111,"num_read_kb":106,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":644038,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1734418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":685,"num_read_kb":442,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":373,"num_bytes_recovered":8234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":459,"ondisk_log_size":459,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103276,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1209840,"kb_used_data":975968,"kb_used_omap":1,"kb_used_meta":233854,"kb_avail":92535312,"statfs":{"total":95995035648,"available":94756159488,"internally_reserved":0,"allocated":999391232,"data_stored":998938530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":239466954},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:41:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.657,"15min":1.151},"min":{"1min":0.29399999999999998,"5min":0.24399999999999999,"15min":0.219},"max":{"1min":3.298,"5min":52.890999999999998,"15min":68.921000000000006},"last":0.25},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.849,"15min":1.238},"min":{"1min":0.28199999999999997,"5min":0.26100000000000001,"15min":0.248},"max":{"1min":3.3919999999999999,"5min":53.353000000000002,"15min":69.616},"last":0.55300000000000005}]},{"osd":1,"last update":"Mon Feb 5 06:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":1.97,"15min":1.042},"min":{"1min":0.254,"5min":0.254,"15min":0.23599999999999999},"max":{"1min":3.9950000000000001,"5min":89.231999999999999,"15min":89.231999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":1.351,"5min":2.3359999999999999,"15min":1.1879999999999999},"min":{"1min":0.33800000000000002,"5min":0.26300000000000001,"15min":0.26100000000000001},"max":{"1min":9.2189999999999994,"5min":87.995999999999995,"15min":87.995999999999995},"last":3.4340000000000002}]},{"osd":2,"last update":"Mon Feb 5 06:41:39 2024","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.78,"15min":0.97899999999999998},"min":{"1min":0.31,"5min":0.27400000000000002,"15min":0.26200000000000001},"max":{"1min":3.891,"5min":53.947000000000003,"15min":53.947000000000003},"last":0.46800000000000003},{"interface":"front","average":{"1min":1.175,"5min":1.7230000000000001,"15min":0.95799999999999996},"min":{"1min":0.29499999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.5739999999999998,"5min":53.299999999999997,"15min":53.299999999999997},"last":0.60599999999999998}]},{"osd":3,"last update":"Mon Feb 5 06:41:39 2024","interfaces":[{"interface":"back","average":{"1min":1.635,"5min":2.536,"15min":1.5389999999999999},"min":{"1min":0.32700000000000001,"5min":0.316,"15min":0.316},"max":{"1min":8.5120000000000005,"5min":89.481999999999999,"15min":89.481999999999999},"last":0.38200000000000001},{"interface":"front","average":{"1min":1.468,"5min":2.3559999999999999,"15min":1.486},"min":{"1min":0.39400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":7.9199999999999999,"5min":92.293999999999997,"15min":92.293999999999997},"last":0.40000000000000002}]},{"osd":4,"last update":"Mon Feb 5 06:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.48599999999999999},"min":{"1min":0.28599999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":2.2349999999999999,"5min":3.5409999999999999,"15min":3.5409999999999999},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.50800000000000001,"15min":0.497},"min":{"1min":0.27100000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":1.546,"5min":2.258,"15min":2.258},"last":0.41699999999999998}]},{"osd":5,"last update":"Mon Feb 5 06:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.55600000000000005,"15min":0.51600000000000001},"min":{"1min":0.34399999999999997,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.3779999999999999,"5min":2.3290000000000002,"15min":2.3290000000000002},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.60999999999999999,"15min":0.56299999999999994},"min":{"1min":0.36699999999999999,"5min":0.252,"15min":0.252},"max":{"1min":1.1519999999999999,"5min":3.5990000000000002,"15min":3.5990000000000002},"last":0.44700000000000001}]},{"osd":6,"last update":"Mon Feb 5 06:41:39 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.72299999999999998,"15min":0.59499999999999997},"min":{"1min":0.32000000000000001,"5min":0.314,"15min":0.314},"max":{"1min":2.976,"5min":3.5750000000000002,"15min":3.5750000000000002},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.61799999999999999,"15min":0.56399999999999995},"min":{"1min":0.307,"5min":0.245,"15min":0.245},"max":{"1min":1.7190000000000001,"5min":3.2189999999999999,"15min":3.2189999999999999},"last":0.53600000000000003}]}]},{"osd":6,"up_from":49,"seq":210453397618,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1419748,"kb_used_data":1158032,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":92325404,"statfs":{"total":95995035648,"available":94541213696,"internally_reserved":0,"allocated":1185824768,"data_stored":1185394239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":267909574},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:42:09 2024","interfaces":[{"interface":"back","average":{"1min":6.2439999999999998,"5min":3.419,"15min":1.609},"min":{"1min":0.26300000000000001,"5min":0.224,"15min":0.224},"max":{"1min":62.537999999999997,"5min":74.075000000000003,"15min":74.075000000000003},"last":0.55000000000000004},{"interface":"front","average":{"1min":6.468,"5min":3.464,"15min":1.5489999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":61.411000000000001,"5min":74.186000000000007,"15min":74.186000000000007},"last":0.65900000000000003}]},{"osd":1,"last update":"Mon Feb 5 06:42:09 2024","interfaces":[{"interface":"back","average":{"1min":5.4329999999999998,"5min":2.6800000000000002,"15min":1.2889999999999999},"min":{"1min":0.26500000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":59.686,"5min":72.951999999999998,"15min":72.951999999999998},"last":1.4870000000000001},{"interface":"front","average":{"1min":5.9820000000000002,"5min":2.5680000000000001,"15min":1.276},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":60.220999999999997,"5min":73.352999999999994,"15min":73.352999999999994},"last":0.52000000000000002}]},{"osd":2,"last update":"Mon Feb 5 06:42:09 2024","interfaces":[{"interface":"back","average":{"1min":2.8450000000000002,"5min":3.1190000000000002,"15min":1.4419999999999999},"min":{"1min":0.29999999999999999,"5min":0.25700000000000001,"15min":0.22500000000000001},"max":{"1min":30.420000000000002,"5min":85.617000000000004,"15min":85.617000000000004},"last":0.435},{"interface":"front","average":{"1min":2.8119999999999998,"5min":3.0529999999999999,"15min":1.401},"min":{"1min":0.30299999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":30.718,"5min":85.335999999999999,"15min":85.335999999999999},"last":0.40699999999999997}]},{"osd":3,"last update":"Mon Feb 5 06:42:09 2024","interfaces":[{"interface":"back","average":{"1min":2.8260000000000001,"5min":3.6240000000000001,"15min":1.635},"min":{"1min":0.34000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":30.175000000000001,"5min":99.269000000000005,"15min":99.269000000000005},"last":1.8899999999999999},{"interface":"front","average":{"1min":2.8809999999999998,"5min":3.9100000000000001,"15min":1.7629999999999999},"min":{"1min":0.316,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":30.452000000000002,"5min":101.75700000000001,"15min":101.75700000000001},"last":0.48799999999999999}]},{"osd":4,"last update":"Mon Feb 5 06:42:09 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.49399999999999999,"15min":0.46100000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.23100000000000001},"max":{"1min":1.8480000000000001,"5min":1.8480000000000001,"15min":1.8480000000000001},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.48799999999999999,"15min":0.45400000000000001},"min":{"1min":0.24399999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":2.371,"5min":2.371,"15min":2.371},"last":0.86599999999999999}]},{"osd":5,"last update":"Mon Feb 5 06:42:09 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.52000000000000002,"15min":0.47899999999999998},"min":{"1min":0.314,"5min":0.29899999999999999,"15min":0.251},"max":{"1min":1.9319999999999999,"5min":1.9319999999999999,"15min":1.9319999999999999},"last":0.627},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.54000000000000004,"15min":0.48899999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.24299999999999999},"max":{"1min":2.4540000000000002,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":0.82299999999999995}]},{"osd":7,"last update":"Mon Feb 5 06:41:26 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.56000000000000005,"15min":0.53300000000000003},"min":{"1min":0.32200000000000001,"5min":0.309,"15min":0.309},"max":{"1min":1.534,"5min":6.0350000000000001,"15min":6.0350000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.67700000000000005,"15min":0.59399999999999997},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.8819999999999999,"5min":8.5399999999999991,"15min":8.5399999999999991},"last":0.80300000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574987,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1706072,"kb_used_data":1381960,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":92039080,"statfs":{"total":95995035648,"available":94248017920,"internally_reserved":0,"allocated":1415127040,"data_stored":1414641399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:42:06 2024","interfaces":[{"interface":"back","average":{"1min":6.1150000000000002,"5min":2.4769999999999999,"15min":1.47},"min":{"1min":0.21199999999999999,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":43.651000000000003,"5min":43.651000000000003,"15min":67.421000000000006},"last":0.34300000000000003},{"interface":"front","average":{"1min":6.0650000000000004,"5min":2.9169999999999998,"15min":1.5740000000000001},"min":{"1min":0.35499999999999998,"5min":0.20999999999999999,"15min":0.19600000000000001},"max":{"1min":45.670999999999999,"5min":45.670999999999999,"15min":67.885000000000005},"last":0.44500000000000001}]},{"osd":2,"last update":"Mon Feb 5 06:41:22 2024","interfaces":[{"interface":"back","average":{"1min":2.1429999999999998,"5min":1.8939999999999999,"15min":0.95499999999999996},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":9.8989999999999991,"5min":9.8989999999999991,"15min":9.8989999999999991},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.847,"5min":2.2989999999999999,"15min":1.0669999999999999},"min":{"1min":0.39100000000000001,"5min":0.249,"15min":0.20899999999999999},"max":{"1min":6.5220000000000002,"5min":11.859,"15min":11.859},"last":1.0269999999999999}]},{"osd":3,"last update":"Mon Feb 5 06:41:57 2024","interfaces":[{"interface":"back","average":{"1min":4.085,"5min":3.0459999999999998,"15min":1.74},"min":{"1min":0.34499999999999997,"5min":0.27900000000000003,"15min":0.25600000000000001},"max":{"1min":25.236999999999998,"5min":25.236999999999998,"15min":69.798000000000002},"last":0.83599999999999997},{"interface":"front","average":{"1min":5.5869999999999997,"5min":2.4950000000000001,"15min":1.54},"min":{"1min":0.41199999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":43.354999999999997,"5min":43.354999999999997,"15min":70.765000000000001},"last":0.72299999999999998}]},{"osd":4,"last update":"Mon Feb 5 06:41:16 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.4890000000000001,"15min":0.88800000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":3.5630000000000002,"5min":11.679,"15min":11.679},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.444,"15min":0.90300000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":3.6070000000000002,"5min":6.0730000000000004,"15min":6.0730000000000004},"last":0.47999999999999998}]},{"osd":5,"last update":"Mon Feb 5 06:41:44 2024","interfaces":[{"interface":"back","average":{"1min":3.4359999999999999,"5min":1.53,"15min":1.1619999999999999},"min":{"1min":0.35299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":43.414999999999999,"5min":43.414999999999999,"15min":43.414999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":1.53,"5min":1.613,"15min":1.0189999999999999},"min":{"1min":0.378,"5min":0.33500000000000002,"15min":0.33100000000000002},"max":{"1min":6.4560000000000004,"5min":13.455,"15min":13.455},"last":1.125}]},{"osd":6,"last update":"Mon Feb 5 06:42:06 2024","interfaces":[{"interface":"back","average":{"1min":1.9179999999999999,"5min":1.6359999999999999,"15min":1.1080000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":7.1260000000000003,"5min":11.805999999999999,"15min":13.391},"last":1.097},{"interface":"front","average":{"1min":3.7810000000000001,"5min":1.694,"15min":1.0760000000000001},"min":{"1min":0.38100000000000001,"5min":0.379,"15min":0.379},"max":{"1min":43.475000000000001,"5min":43.475000000000001,"15min":43.475000000000001},"last":0.64900000000000002}]},{"osd":7,"last update":"Mon Feb 5 06:41:27 2024","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.6699999999999999,"15min":0.997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":6.4299999999999997,"5min":13.49,"15min":13.49},"last":1.147},{"interface":"front","average":{"1min":1.462,"5min":1.486,"15min":1.169},"min":{"1min":0.41999999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":3.6389999999999998,"5min":6.1870000000000003,"15min":8.2219999999999995},"last":0.66900000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738513,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1680068,"kb_used_data":1374008,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":92065084,"statfs":{"total":95995035648,"available":94274646016,"internally_reserved":0,"allocated":1406984192,"data_stored":1406514128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":313391557},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 06:42:06 2024","interfaces":[{"interface":"back","average":{"1min":2.3330000000000002,"5min":2.2160000000000002,"15min":1.107},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":6.117,"5min":18.59,"15min":18.59},"last":0.24199999999999999},{"interface":"front","average":{"1min":2.1699999999999999,"5min":2.1099999999999999,"15min":1.0669999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.20499999999999999},"max":{"1min":6.0510000000000002,"5min":18.300999999999998,"15min":18.300999999999998},"last":0.69299999999999995}]},{"osd":2,"last update":"Mon Feb 5 06:41:23 2024","interfaces":[{"interface":"back","average":{"1min":2.0640000000000001,"5min":2.2349999999999999,"15min":1.0880000000000001},"min":{"1min":0.28699999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":5.5540000000000003,"5min":16.981999999999999,"15min":16.981999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":1.5880000000000001,"5min":1.9870000000000001,"15min":1.0009999999999999},"min":{"1min":0.32700000000000001,"5min":0.29199999999999998,"15min":0.28000000000000003},"max":{"1min":7.8529999999999998,"5min":16.998999999999999,"15min":16.998999999999999},"last":0.435}]},{"osd":3,"last update":"Mon Feb 5 06:41:50 2024","interfaces":[{"interface":"back","average":{"1min":2.1789999999999998,"5min":2.2109999999999999,"15min":1.1850000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":7.9379999999999997,"5min":16.923999999999999,"15min":16.923999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":2.3690000000000002,"5min":2.629,"15min":1.296},"min":{"1min":0.34300000000000003,"5min":0.254,"15min":0.254},"max":{"1min":9.4499999999999993,"5min":18.259,"15min":18.259},"last":0.378}]},{"osd":4,"last update":"Mon Feb 5 06:41:20 2024","interfaces":[{"interface":"back","average":{"1min":3.5840000000000001,"5min":2.0739999999999998,"15min":1.1220000000000001},"min":{"1min":0.33900000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":40.789999999999999,"5min":40.789999999999999,"15min":40.789999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":3.5539999999999998,"5min":2.0619999999999998,"15min":1.1259999999999999},"min":{"1min":0.32700000000000001,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":40.773000000000003,"5min":40.773000000000003,"15min":40.773000000000003},"last":0.53700000000000003}]},{"osd":5,"last update":"Mon Feb 5 06:41:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.502,"15min":0.96499999999999997},"min":{"1min":0.36399999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":4.7270000000000003,"5min":18.170999999999999,"15min":18.170999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":1.278,"5min":1.5629999999999999,"15min":0.99299999999999999},"min":{"1min":0.33300000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":4.5279999999999996,"5min":17.152999999999999,"15min":17.152999999999999},"last":0.41699999999999998}]},{"osd":6,"last update":"Mon Feb 5 06:42:06 2024","interfaces":[{"interface":"back","average":{"1min":1.8919999999999999,"5min":1.669,"15min":1.087},"min":{"1min":0.36799999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":5.992,"5min":18.562999999999999,"15min":18.562999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":1.409,"5min":1.5329999999999999,"15min":1.0049999999999999},"min":{"1min":0.379,"5min":0.33200000000000002,"15min":0.33000000000000002},"max":{"1min":4.7610000000000001,"5min":18.213999999999999,"15min":18.213999999999999},"last":0.55800000000000005}]},{"osd":7,"last update":"Mon Feb 5 06:41:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.605,"15min":0.97999999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":4.5570000000000004,"5min":17.210000000000001,"15min":17.210000000000001},"last":0.5},{"interface":"front","average":{"1min":1.76,"5min":1.714,"15min":1.2},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":5.7690000000000001,"5min":18.533999999999999,"15min":18.533999999999999},"last":0.64900000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411462,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1662900,"kb_used_data":1360996,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":92082252,"statfs":{"total":95995035648,"available":94292226048,"internally_reserved":0,"allocated":1393659904,"data_stored":1393210781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:41:25 2024","interfaces":[{"interface":"back","average":{"1min":2.0870000000000002,"5min":2.6419999999999999,"15min":1.3839999999999999},"min":{"1min":0.35399999999999998,"5min":0.215,"15min":0.20000000000000001},"max":{"1min":8.9339999999999993,"5min":50.814,"15min":50.814},"last":0.34899999999999998},{"interface":"front","average":{"1min":2.2360000000000002,"5min":2.5640000000000001,"15min":1.3700000000000001},"min":{"1min":0.44900000000000001,"5min":0.19700000000000001,"15min":0.188},"max":{"1min":5.6890000000000001,"5min":47.122999999999998,"15min":50.802999999999997},"last":0.26300000000000001}]},{"osd":1,"last update":"Mon Feb 5 06:41:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":2.4020000000000001,"15min":1.133},"min":{"1min":0.32300000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":5.1230000000000002,"5min":31.303000000000001,"15min":31.303000000000001},"last":0.24299999999999999},{"interface":"front","average":{"1min":1.042,"5min":1.821,"15min":0.94999999999999996},"min":{"1min":0.379,"5min":0.221,"15min":0.22},"max":{"1min":3.427,"5min":31.425000000000001,"15min":31.425000000000001},"last":0.441}]},{"osd":3,"last update":"Mon Feb 5 06:41:58 2024","interfaces":[{"interface":"back","average":{"1min":2.286,"5min":2.8620000000000001,"15min":1.5409999999999999},"min":{"1min":0.27300000000000002,"5min":0.254,"15min":0.254},"max":{"1min":13.504,"5min":46.845999999999997,"15min":53.127000000000002},"last":0.48599999999999999},{"interface":"front","average":{"1min":1.131,"5min":2.5409999999999999,"15min":1.4299999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.22500000000000001},"max":{"1min":4.4690000000000003,"5min":46.481999999999999,"15min":52.607999999999997},"last":0.30199999999999999}]},{"osd":4,"last update":"Mon Feb 5 06:41:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.2130000000000001,"15min":0.83999999999999997},"min":{"1min":0.29699999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":4.2270000000000003,"5min":6.4180000000000001,"15min":6.4180000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":1.103,"5min":1.137,"15min":0.82199999999999995},"min":{"1min":0.36199999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":4.798,"5min":6.8780000000000001,"15min":6.8780000000000001},"last":0.54300000000000004}]},{"osd":5,"last update":"Mon Feb 5 06:41:42 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":2.3559999999999999,"15min":1.2549999999999999},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":6.6079999999999997,"5min":86.614999999999995,"15min":86.614999999999995},"last":1.0309999999999999},{"interface":"front","average":{"1min":1.282,"5min":2.6819999999999999,"15min":1.399},"min":{"1min":0.33800000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":5.4640000000000004,"5min":86.543999999999997,"15min":86.543999999999997},"last":1.3}]},{"osd":6,"last update":"Mon Feb 5 06:41:11 2024","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.2,"15min":1.4119999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":4.9489999999999998,"5min":6.9139999999999997,"15min":17.143999999999998},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.7210000000000001,"15min":1.0469999999999999},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":6.7919999999999998,"5min":10.27,"15min":10.27},"last":0.96299999999999997}]},{"osd":7,"last update":"Mon Feb 5 06:41:32 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.6539999999999999,"15min":1.0329999999999999},"min":{"1min":0.39900000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":4.9089999999999998,"5min":7.8280000000000003,"15min":10.206},"last":0.56100000000000005},{"interface":"front","average":{"1min":1.21,"5min":1.4059999999999999,"15min":0.95199999999999996},"min":{"1min":0.45400000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":3.4990000000000001,"5min":6.4749999999999996,"15min":6.4749999999999996},"last":1.137}]}]},{"osd":3,"up_from":26,"seq":111669149825,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2103812,"kb_used_data":1769972,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":91641340,"statfs":{"total":95995035648,"available":93840732160,"internally_reserved":0,"allocated":1812451328,"data_stored":1811985260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":341834183},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:42:04 2024","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":1.819,"15min":0.96899999999999997},"min":{"1min":0.184,"5min":0.184,"15min":0.184},"max":{"1min":5.9109999999999996,"5min":13.391999999999999,"15min":13.391999999999999},"last":3.6190000000000002},{"interface":"front","average":{"1min":1.8220000000000001,"5min":2.0990000000000002,"15min":1.1879999999999999},"min":{"1min":0.23000000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.7670000000000003,"5min":25.087,"15min":25.087},"last":3.093}]},{"osd":1,"last update":"Mon Feb 5 06:42:04 2024","interfaces":[{"interface":"back","average":{"1min":1.859,"5min":1.867,"15min":1.0760000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":6.1719999999999997,"5min":6.7889999999999997,"15min":9.6010000000000009},"last":3.177},{"interface":"front","average":{"1min":2.5870000000000002,"5min":1.9670000000000001,"15min":1.0920000000000001},"min":{"1min":0.248,"5min":0.215,"15min":0.215},"max":{"1min":6.3929999999999998,"5min":8.3040000000000003,"15min":8.3040000000000003},"last":2.2000000000000002}]},{"osd":2,"last update":"Mon Feb 5 06:42:04 2024","interfaces":[{"interface":"back","average":{"1min":2.6099999999999999,"5min":2.3959999999999999,"15min":1.274},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":6.8099999999999996,"5min":41.529000000000003,"15min":41.529000000000003},"last":1.605},{"interface":"front","average":{"1min":2.2160000000000002,"5min":2.3500000000000001,"15min":1.1890000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.248},"max":{"1min":6.5839999999999996,"5min":41.759999999999998,"15min":41.759999999999998},"last":2.9100000000000001}]},{"osd":4,"last update":"Mon Feb 5 06:41:17 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.3819999999999999,"15min":0.88},"min":{"1min":0.32800000000000001,"5min":0.307,"15min":0.307},"max":{"1min":6.3319999999999999,"5min":16.577999999999999,"15min":16.577999999999999},"last":2.343},{"interface":"front","average":{"1min":1.603,"5min":1.512,"15min":0.96999999999999997},"min":{"1min":0.42499999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":4.3490000000000002,"5min":6.6139999999999999,"15min":8.4060000000000006},"last":3.1240000000000001}]},{"osd":5,"last update":"Mon Feb 5 06:41:41 2024","interfaces":[{"interface":"back","average":{"1min":1.921,"5min":1.732,"15min":1.0649999999999999},"min":{"1min":0.41399999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":5.7430000000000003,"5min":13.672000000000001,"15min":13.672000000000001},"last":0.42199999999999999},{"interface":"front","average":{"1min":1.258,"5min":1.4359999999999999,"15min":0.91300000000000003},"min":{"1min":0.36499999999999999,"5min":0.32800000000000001,"15min":0.28399999999999997},"max":{"1min":5.2249999999999996,"5min":15.792999999999999,"15min":15.792999999999999},"last":2.2999999999999998}]},{"osd":6,"last update":"Mon Feb 5 06:41:10 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.603,"15min":0.96099999999999997},"min":{"1min":0.34899999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":4.5439999999999996,"5min":8.3610000000000007,"15min":8.3610000000000007},"last":3.1520000000000001},{"interface":"front","average":{"1min":1.6799999999999999,"5min":1.5600000000000001,"15min":0.97799999999999998},"min":{"1min":0.41699999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":5.1829999999999998,"5min":6.601,"15min":6.601},"last":0.45200000000000001}]},{"osd":7,"last update":"Mon Feb 5 06:41:33 2024","interfaces":[{"interface":"back","average":{"1min":1.9299999999999999,"5min":1.5389999999999999,"15min":1.0580000000000001},"min":{"1min":0.48299999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":6.6429999999999998,"5min":16.524999999999999,"15min":45.039000000000001},"last":2.3679999999999999},{"interface":"front","average":{"1min":2.1099999999999999,"5min":1.708,"15min":1.1599999999999999},"min":{"1min":0.46200000000000002,"5min":0.35899999999999999,"15min":0.34399999999999997},"max":{"1min":5.8159999999999998,"5min":6.6269999999999998,"15min":44.036999999999999},"last":0.52300000000000002}]}]},{"osd":4,"up_from":31,"seq":133143986299,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1578672,"kb_used_data":1240676,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92166480,"statfs":{"total":95995035648,"available":94378475520,"internally_reserved":0,"allocated":1270452224,"data_stored":1269994956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:41:19 2024","interfaces":[{"interface":"back","average":{"1min":1.7609999999999999,"5min":1.9419999999999999,"15min":1.0409999999999999},"min":{"1min":0.22700000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":10.949,"5min":33.372,"15min":33.372},"last":1.337},{"interface":"front","average":{"1min":1.6830000000000001,"5min":2.1179999999999999,"15min":1.109},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":12.106,"5min":33.323999999999998,"15min":33.323999999999998},"last":0.80900000000000005}]},{"osd":1,"last update":"Mon Feb 5 06:41:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.2450000000000001,"15min":0.80100000000000005},"min":{"1min":0.30499999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":4.1879999999999997,"5min":33.747999999999998,"15min":33.747999999999998},"last":1.5840000000000001},{"interface":"front","average":{"1min":0.80100000000000005,"5min":1.323,"15min":0.85199999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.6760000000000002,"5min":33.691000000000003,"15min":33.691000000000003},"last":1.196}]},{"osd":2,"last update":"Mon Feb 5 06:41:14 2024","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.82599999999999996,"15min":0.66400000000000003},"min":{"1min":0.29399999999999998,"5min":0.245,"15min":0.245},"max":{"1min":2.3260000000000001,"5min":4.1239999999999997,"15min":4.1239999999999997},"last":1.8799999999999999},{"interface":"front","average":{"1min":1.018,"5min":1.0780000000000001,"15min":0.749},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.766,"5min":5.7030000000000003,"15min":5.7030000000000003},"last":1.292}]},{"osd":3,"last update":"Mon Feb 5 06:41:19 2024","interfaces":[{"interface":"back","average":{"1min":2.0659999999999998,"5min":2.9390000000000001,"15min":1.698},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":10.816000000000001,"5min":99.546999999999997,"15min":99.546999999999997},"last":7.633},{"interface":"front","average":{"1min":2.3660000000000001,"5min":3.1030000000000002,"15min":1.7130000000000001},"min":{"1min":0.318,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":10.568,"5min":99.245000000000005,"15min":99.245000000000005},"last":2.0790000000000002}]},{"osd":5,"last update":"Mon Feb 5 06:41:42 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.64300000000000002,"15min":0.48699999999999999},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.1100000000000001,"5min":11.098000000000001,"15min":11.098000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.60999999999999999,"15min":0.499},"min":{"1min":0.27000000000000002,"5min":0.25700000000000001,"15min":0.252},"max":{"1min":0.68300000000000005,"5min":10.965999999999999,"15min":10.965999999999999},"last":0.35299999999999998}]},{"osd":6,"last update":"Mon Feb 5 06:41:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.68600000000000005,"15min":0.56000000000000005},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":2.9630000000000001,"5min":11.868,"15min":11.868},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.73799999999999999,"15min":0.56799999999999995},"min":{"1min":0.29899999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":3.0350000000000001,"5min":11.787000000000001,"15min":11.787000000000001},"last":0.55600000000000005}]},{"osd":7,"last update":"Mon Feb 5 06:41:30 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.60199999999999998,"15min":0.54000000000000004},"min":{"1min":0.27800000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":5.1500000000000004,"5min":5.1500000000000004,"15min":5.1500000000000004},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.877,"5min":0.64800000000000002,"15min":0.56999999999999995},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28499999999999998},"max":{"1min":5.5890000000000004,"5min":5.5890000000000004,"15min":5.5890000000000004},"last":0.442}]}]},{"osd":5,"up_from":40,"seq":171798691958,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1058208,"kb_used_data":820184,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":92686944,"statfs":{"total":95995035648,"available":94911430656,"internally_reserved":0,"allocated":839868416,"data_stored":839424213,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":243726790},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:41:40 2024","interfaces":[{"interface":"back","average":{"1min":4.5780000000000003,"5min":2.3140000000000001,"15min":1.1830000000000001},"min":{"1min":0.23799999999999999,"5min":0.223,"15min":0.19700000000000001},"max":{"1min":55.704999999999998,"5min":55.704999999999998,"15min":55.704999999999998},"last":1.7849999999999999},{"interface":"front","average":{"1min":5.4850000000000003,"5min":2.8700000000000001,"15min":1.403},"min":{"1min":0.28399999999999997,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":56.789000000000001,"5min":56.789000000000001,"15min":56.789000000000001},"last":0.46100000000000002}]},{"osd":1,"last update":"Mon Feb 5 06:41:40 2024","interfaces":[{"interface":"back","average":{"1min":4.1580000000000004,"5min":2.2490000000000001,"15min":1.1559999999999999},"min":{"1min":0.28000000000000003,"5min":0.248,"15min":0.248},"max":{"1min":41.191000000000003,"5min":41.191000000000003,"15min":41.191000000000003},"last":0.55600000000000005},{"interface":"front","average":{"1min":4.0540000000000003,"5min":1.855,"15min":1.0009999999999999},"min":{"1min":0.27500000000000002,"5min":0.24399999999999999,"15min":0.23499999999999999},"max":{"1min":41.173000000000002,"5min":41.173000000000002,"15min":41.173000000000002},"last":0.57699999999999996}]},{"osd":2,"last update":"Mon Feb 5 06:41:40 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":2.0179999999999998,"15min":1.069},"min":{"1min":0.34599999999999997,"5min":0.27100000000000002,"15min":0.25800000000000001},"max":{"1min":5.4260000000000002,"5min":36.901000000000003,"15min":36.901000000000003},"last":0.71699999999999997},{"interface":"front","average":{"1min":1.6100000000000001,"5min":1.9099999999999999,"15min":0.98999999999999999},"min":{"1min":0.31900000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":7.008,"5min":37.984000000000002,"15min":37.984000000000002},"last":3.5339999999999998}]},{"osd":3,"last update":"Mon Feb 5 06:41:40 2024","interfaces":[{"interface":"back","average":{"1min":4.1139999999999999,"5min":2.1499999999999999,"15min":1.157},"min":{"1min":0.36799999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":55.417000000000002,"5min":55.417000000000002,"15min":55.417000000000002},"last":0.66200000000000003},{"interface":"front","average":{"1min":4.7140000000000004,"5min":2.1379999999999999,"15min":1.2170000000000001},"min":{"1min":0.35899999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":55.079000000000001,"5min":55.079000000000001,"15min":55.079000000000001},"last":3.1099999999999999}]},{"osd":4,"last update":"Mon Feb 5 06:41:40 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.45300000000000001,"15min":0.41899999999999998},"min":{"1min":0.29499999999999998,"5min":0.249,"15min":0.23200000000000001},"max":{"1min":0.73399999999999999,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.57099999999999995,"15min":0.47199999999999998},"min":{"1min":0.25800000000000001,"5min":0.253,"15min":0.253},"max":{"1min":1.159,"5min":2.8260000000000001,"15min":2.8260000000000001},"last":0.624}]},{"osd":6,"last update":"Mon Feb 5 06:42:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.63300000000000001,"15min":0.51600000000000001},"min":{"1min":0.27600000000000002,"5min":0.27500000000000002,"15min":0.245},"max":{"1min":3.585,"5min":3.585,"15min":3.585},"last":3.585},{"interface":"front","average":{"1min":1.4890000000000001,"5min":0.77000000000000002,"15min":0.57499999999999996},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.27600000000000002},"max":{"1min":6.5179999999999998,"5min":6.5179999999999998,"15min":6.5179999999999998},"last":3.5110000000000001}]},{"osd":7,"last update":"Mon Feb 5 06:41:30 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.57799999999999996,"15min":0.504},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.27800000000000002},"max":{"1min":2.4039999999999999,"5min":2.4039999999999999,"15min":2.4039999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.60199999999999998,"15min":0.53300000000000003},"min":{"1min":0.377,"5min":0.33600000000000002,"15min":0.33500000000000002},"max":{"1min":2.3420000000000001,"5min":2.798,"15min":2.798},"last":3.1469999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1404502016,"data_stored":1404502016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1411055616,"data_stored":1411055616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1390743552,"data_stored":1390739480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1810825216,"data_stored":1810825216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1266286592,"data_stored":1266286592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":856166400,"data_stored":856162328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1184104448,"data_stored":1184104448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":994578432,"data_stored":994574360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T06:42:19.976 INFO:tasks.thrashosds.thrasher:Setting 5.1b to [3, 0, 1] 2024-02-05T06:42:19.976 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '5.1b', '3', '0', '1'] 2024-02-05T06:42:19.976 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd pg-upmap 5.1b 3 0 1 2024-02-05T06:42:19.984 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:20.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:20.070 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:20.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:20.171 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:20.248 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:20.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:20.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:20.350 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:20.383 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:20.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:20.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:20.485 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:42:20.541 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:42:20.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:20.560 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:20.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:20.649 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:42:20.649 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-02-05T06:42:20.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-02-05T06:42:20.650 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-02-05T06:42:20.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.650 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:173001 7.f 7:f6bb3971:::benchmark_data_smithi028_77509_object57881:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-02-05T06:42:20.650 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:15.582635+0000", 2024-02-05T06:42:20.650 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 245.056411408, 2024-02-05T06:42:20.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1376656889999999, 2024-02-05T06:42:20.651 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:42:20.651 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-02-05T06:42:20.651 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:42:20.651 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:42:20.651 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:42:20.652 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 173001 2024-02-05T06:42:20.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.652 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:42:20.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:42:20.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.582635+0000", 2024-02-05T06:42:20.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.653 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:42:20.653 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.582635+0000", 2024-02-05T06:42:20.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:42:20.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.582637+0000", 2024-02-05T06:42:20.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1679999999999998e-06 2024-02-05T06:42:20.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:42:20.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.582645+0000", 2024-02-05T06:42:20.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.452e-06 2024-02-05T06:42:20.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:42:20.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.582659+0000", 2024-02-05T06:42:20.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4156e-05 2024-02-05T06:42:20.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:42:20.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.582670+0000", 2024-02-05T06:42:20.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0922e-05 2024-02-05T06:42:20.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.590472+0000", 2024-02-05T06:42:20.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0078017210000000002 2024-02-05T06:42:20.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:42:20.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.590589+0000", 2024-02-05T06:42:20.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000117376 2024-02-05T06:42:20.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-02-05T06:42:20.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.590706+0000", 2024-02-05T06:42:20.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000117019 2024-02-05T06:42:20.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.660 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.602735+0000", 2024-02-05T06:42:20.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012028687999999999 2024-02-05T06:42:20.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.661 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.661 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:42:20.661 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.602747+0000", 2024-02-05T06:42:20.661 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2431e-05 2024-02-05T06:42:20.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.662 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.662 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.662 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:16.586568+0000", 2024-02-05T06:42:20.662 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.98382046499999998 2024-02-05T06:42:20.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.663 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:42:20.663 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:16.586584+0000", 2024-02-05T06:42:20.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6061000000000001e-05 2024-02-05T06:42:20.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.663 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.664 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.664 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.698088+0000", 2024-02-05T06:42:20.664 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1115044169999999 2024-02-05T06:42:20.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.664 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.664 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for missing object", 2024-02-05T06:42:20.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.698205+0000", 2024-02-05T06:42:20.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000117055 2024-02-05T06:42:20.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.720204+0000", 2024-02-05T06:42:20.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.021999199000000001 2024-02-05T06:42:20.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.666 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:42:20.666 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.720301+0000", 2024-02-05T06:42:20.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6558999999999995e-05 2024-02-05T06:42:20.667 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.667 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:42:20.667 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.667 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:173015 7.6 7:613bc288:::benchmark_data_smithi028_77509_object57895:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-02-05T06:42:20.668 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:15.602395+0000", 2024-02-05T06:42:20.668 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 245.03665167, 2024-02-05T06:42:20.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1434625880000002, 2024-02-05T06:42:20.668 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:42:20.668 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:42:20.668 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:42:20.669 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:42:20.669 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:42:20.669 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 173015 2024-02-05T06:42:20.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.669 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:42:20.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:42:20.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.602395+0000", 2024-02-05T06:42:20.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:42:20.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.602395+0000", 2024-02-05T06:42:20.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:42:20.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.602397+0000", 2024-02-05T06:42:20.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8190000000000001e-06 2024-02-05T06:42:20.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:42:20.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.602401+0000", 2024-02-05T06:42:20.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4959999999999998e-06 2024-02-05T06:42:20.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:42:20.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.602410+0000", 2024-02-05T06:42:20.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2229999999999996e-06 2024-02-05T06:42:20.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:42:20.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.602418+0000", 2024-02-05T06:42:20.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9480000000000008e-06 2024-02-05T06:42:20.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.625628+0000", 2024-02-05T06:42:20.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.023209956 2024-02-05T06:42:20.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:42:20.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.625639+0000", 2024-02-05T06:42:20.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1008e-05 2024-02-05T06:42:20.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.677 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:16.586768+0000", 2024-02-05T06:42:20.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.96112819699999996 2024-02-05T06:42:20.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:42:20.678 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:16.586778+0000", 2024-02-05T06:42:20.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0210000000000001e-05 2024-02-05T06:42:20.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.678 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.699940+0000", 2024-02-05T06:42:20.679 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.113162362 2024-02-05T06:42:20.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.679 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:42:20.679 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.700054+0000", 2024-02-05T06:42:20.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000114168 2024-02-05T06:42:20.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,5,7", 2024-02-05T06:42:20.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.700506+0000", 2024-02-05T06:42:20.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00045122899999999999 2024-02-05T06:42:20.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.720472+0000", 2024-02-05T06:42:20.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.019966085000000001 2024-02-05T06:42:20.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.682 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.682 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.720515+0000", 2024-02-05T06:42:20.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:20.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3850999999999997e-05 2024-02-05T06:42:20.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:42:20.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.735601+0000", 2024-02-05T06:42:20.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015085602 2024-02-05T06:42:20.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.686 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.686 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.745693+0000", 2024-02-05T06:42:20.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010091915999999999 2024-02-05T06:42:20.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.687 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:42:20.687 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.745754+0000", 2024-02-05T06:42:20.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1074999999999994e-05 2024-02-05T06:42:20.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:42:20.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.745858+0000", 2024-02-05T06:42:20.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010344300000000001 2024-02-05T06:42:20.688 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.689 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:42:20.689 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.689 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:173030 7.f 7:f7767a4d:::benchmark_data_smithi028_77509_object57910:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-05T06:42:20.690 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:15.635474+0000", 2024-02-05T06:42:20.690 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 245.00357258099999, 2024-02-05T06:42:20.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.118429119, 2024-02-05T06:42:20.690 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:42:20.690 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:42:20.690 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:42:20.691 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:42:20.691 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:42:20.691 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 173030 2024-02-05T06:42:20.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.691 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:42:20.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:42:20.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.635474+0000", 2024-02-05T06:42:20.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:42:20.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.635474+0000", 2024-02-05T06:42:20.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:42:20.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.635475+0000", 2024-02-05T06:42:20.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3629999999999999e-06 2024-02-05T06:42:20.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:42:20.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.635480+0000", 2024-02-05T06:42:20.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5979999999999999e-06 2024-02-05T06:42:20.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.695 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:42:20.695 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.635489+0000", 2024-02-05T06:42:20.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4599999999999992e-06 2024-02-05T06:42:20.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:42:20.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.635497+0000", 2024-02-05T06:42:20.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7270000000000007e-06 2024-02-05T06:42:20.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.635556+0000", 2024-02-05T06:42:20.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9200000000000002e-05 2024-02-05T06:42:20.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:42:20.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.635567+0000", 2024-02-05T06:42:20.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0159999999999999e-05 2024-02-05T06:42:20.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:16.586613+0000", 2024-02-05T06:42:20.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.95104614300000001 2024-02-05T06:42:20.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:42:20.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:16.586621+0000", 2024-02-05T06:42:20.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2400000000000007e-06 2024-02-05T06:42:20.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.698459+0000", 2024-02-05T06:42:20.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.111838001 2024-02-05T06:42:20.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:42:20.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.698581+0000", 2024-02-05T06:42:20.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121721 2024-02-05T06:42:20.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4,7", 2024-02-05T06:42:20.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.699063+0000", 2024-02-05T06:42:20.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000482337 2024-02-05T06:42:20.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.711963+0000", 2024-02-05T06:42:20.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012899671 2024-02-05T06:42:20.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:42:20.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.711979+0000", 2024-02-05T06:42:20.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6728e-05 2024-02-05T06:42:20.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.719400+0000", 2024-02-05T06:42:20.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0074207259999999999 2024-02-05T06:42:20.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.753700+0000", 2024-02-05T06:42:20.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.034299450000000002 2024-02-05T06:42:20.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:42:20.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.753761+0000", 2024-02-05T06:42:20.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1915999999999995e-05 2024-02-05T06:42:20.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:42:20.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.753903+0000", 2024-02-05T06:42:20.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014167800000000001 2024-02-05T06:42:20.700 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.701 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:42:20.701 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.701 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:173034 7.6 7:691d6e0c:::benchmark_data_smithi028_77509_object57914:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-05T06:42:20.701 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:15.641440+0000", 2024-02-05T06:42:20.701 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 244.99760626299999, 2024-02-05T06:42:20.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1045922149999998, 2024-02-05T06:42:20.701 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:42:20.701 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:42:20.701 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:42:20.701 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:42:20.701 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:42:20.701 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 173034 2024-02-05T06:42:20.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.702 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:42:20.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:42:20.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.641440+0000", 2024-02-05T06:42:20.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:42:20.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.641440+0000", 2024-02-05T06:42:20.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:42:20.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.641442+0000", 2024-02-05T06:42:20.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9889999999999999e-06 2024-02-05T06:42:20.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:42:20.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.641449+0000", 2024-02-05T06:42:20.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5529999999999998e-06 2024-02-05T06:42:20.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:42:20.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.641461+0000", 2024-02-05T06:42:20.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2018000000000001e-05 2024-02-05T06:42:20.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:42:20.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.641469+0000", 2024-02-05T06:42:20.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4349999999999997e-06 2024-02-05T06:42:20.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.641560+0000", 2024-02-05T06:42:20.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0527999999999998e-05 2024-02-05T06:42:20.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:42:20.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.641570+0000", 2024-02-05T06:42:20.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9080000000000007e-06 2024-02-05T06:42:20.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:16.586893+0000", 2024-02-05T06:42:20.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.945323048 2024-02-05T06:42:20.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:42:20.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:16.586903+0000", 2024-02-05T06:42:20.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8870000000000004e-06 2024-02-05T06:42:20.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.713453+0000", 2024-02-05T06:42:20.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1265501040000001 2024-02-05T06:42:20.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:42:20.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.713583+0000", 2024-02-05T06:42:20.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013038899999999999 2024-02-05T06:42:20.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,5,7", 2024-02-05T06:42:20.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.713725+0000", 2024-02-05T06:42:20.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014129799999999999 2024-02-05T06:42:20.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.720555+0000", 2024-02-05T06:42:20.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0068308780000000003 2024-02-05T06:42:20.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.720694+0000", 2024-02-05T06:42:20.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013885500000000001 2024-02-05T06:42:20.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:42:20.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.735608+0000", 2024-02-05T06:42:20.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014914234 2024-02-05T06:42:20.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.745897+0000", 2024-02-05T06:42:20.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010288459999999999 2024-02-05T06:42:20.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:42:20.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.745942+0000", 2024-02-05T06:42:20.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4913000000000002e-05 2024-02-05T06:42:20.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:42:20.708 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.746033+0000", 2024-02-05T06:42:20.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0717999999999994e-05 2024-02-05T06:42:20.708 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.708 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:42:20.708 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.708 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:173046 7.6 7:65434fc4:::benchmark_data_smithi028_77509_object57926:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-05T06:42:20.708 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:15.655354+0000", 2024-02-05T06:42:20.708 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 244.98369304100001, 2024-02-05T06:42:20.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0908921309999999, 2024-02-05T06:42:20.709 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:42:20.709 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:42:20.709 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:42:20.709 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:42:20.709 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:42:20.709 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 173046 2024-02-05T06:42:20.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.709 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:42:20.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.709 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:42:20.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.655354+0000", 2024-02-05T06:42:20.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.709 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:42:20.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.655354+0000", 2024-02-05T06:42:20.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:42:20.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.655356+0000", 2024-02-05T06:42:20.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6309999999999999e-06 2024-02-05T06:42:20.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:42:20.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.655362+0000", 2024-02-05T06:42:20.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6069999999999998e-06 2024-02-05T06:42:20.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:42:20.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.655375+0000", 2024-02-05T06:42:20.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3139000000000001e-05 2024-02-05T06:42:20.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:42:20.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.655383+0000", 2024-02-05T06:42:20.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9920000000000007e-06 2024-02-05T06:42:20.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.655444+0000", 2024-02-05T06:42:20.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0919999999999999e-05 2024-02-05T06:42:20.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:42:20.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.655455+0000", 2024-02-05T06:42:20.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0852e-05 2024-02-05T06:42:20.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.712 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.712 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:16.587003+0000", 2024-02-05T06:42:20.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.93154866000000003 2024-02-05T06:42:20.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.712 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:42:20.712 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:16.587013+0000", 2024-02-05T06:42:20.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4029999999999996e-06 2024-02-05T06:42:20.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.712 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.712 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.716613+0000", 2024-02-05T06:42:20.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1296000939999999 2024-02-05T06:42:20.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:42:20.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.716756+0000", 2024-02-05T06:42:20.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000143516 2024-02-05T06:42:20.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,5,7", 2024-02-05T06:42:20.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.716906+0000", 2024-02-05T06:42:20.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000149853 2024-02-05T06:42:20.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.714 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.720596+0000", 2024-02-05T06:42:20.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0036894929999999999 2024-02-05T06:42:20.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.714 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.720735+0000", 2024-02-05T06:42:20.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013902900000000001 2024-02-05T06:42:20.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:42:20.714 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.735615+0000", 2024-02-05T06:42:20.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014880523 2024-02-05T06:42:20.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.746075+0000", 2024-02-05T06:42:20.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010459954 2024-02-05T06:42:20.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:42:20.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.746134+0000", 2024-02-05T06:42:20.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8508999999999998e-05 2024-02-05T06:42:20.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:42:20.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.746246+0000", 2024-02-05T06:42:20.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111956 2024-02-05T06:42:20.715 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.715 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:42:20.715 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.716 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:173048 7.6 7:6d5ae367:::benchmark_data_smithi028_77509_object57928:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151)", 2024-02-05T06:42:20.716 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:15.658936+0000", 2024-02-05T06:42:20.716 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 244.98011012800001, 2024-02-05T06:42:20.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0875293730000002, 2024-02-05T06:42:20.716 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:42:20.716 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:42:20.716 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:42:20.716 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:42:20.716 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:42:20.716 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 173048 2024-02-05T06:42:20.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.716 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:42:20.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.716 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:42:20.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.658936+0000", 2024-02-05T06:42:20.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:42:20.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.658936+0000", 2024-02-05T06:42:20.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:42:20.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.658938+0000", 2024-02-05T06:42:20.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7460000000000001e-06 2024-02-05T06:42:20.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.718 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:42:20.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.658943+0000", 2024-02-05T06:42:20.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5800000000000002e-06 2024-02-05T06:42:20.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.718 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:42:20.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.658965+0000", 2024-02-05T06:42:20.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1886999999999999e-05 2024-02-05T06:42:20.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.718 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:42:20.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.658973+0000", 2024-02-05T06:42:20.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2670000000000006e-06 2024-02-05T06:42:20.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.719 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.659094+0000", 2024-02-05T06:42:20.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000120641 2024-02-05T06:42:20.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:42:20.719 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.659105+0000", 2024-02-05T06:42:20.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1620999999999999e-05 2024-02-05T06:42:20.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.719 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:16.587041+0000", 2024-02-05T06:42:20.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.92793587799999999 2024-02-05T06:42:20.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.720 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:42:20.720 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:16.587050+0000", 2024-02-05T06:42:20.720 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2380000000000003e-06 2024-02-05T06:42:20.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.720 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.720 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.720909+0000", 2024-02-05T06:42:20.720 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.133858515 2024-02-05T06:42:20.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.720 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:42:20.720 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.721053+0000", 2024-02-05T06:42:20.720 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014435500000000001 2024-02-05T06:42:20.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.721 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,5,7", 2024-02-05T06:42:20.721 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.721201+0000", 2024-02-05T06:42:20.721 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000147942 2024-02-05T06:42:20.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.721 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.721 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.724360+0000", 2024-02-05T06:42:20.721 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003159055 2024-02-05T06:42:20.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.721 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.721 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.724413+0000", 2024-02-05T06:42:20.721 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3059000000000001e-05 2024-02-05T06:42:20.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:42:20.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.735677+0000", 2024-02-05T06:42:20.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011263263000000001 2024-02-05T06:42:20.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.746297+0000", 2024-02-05T06:42:20.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010620661 2024-02-05T06:42:20.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:42:20.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.746354+0000", 2024-02-05T06:42:20.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6715000000000002e-05 2024-02-05T06:42:20.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.723 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:42:20.723 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:17.746466+0000", 2024-02-05T06:42:20.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011195 2024-02-05T06:42:20.723 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.723 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:42:20.723 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.723 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:30036 8.c 8:3ade523f:::benchmark_data_smithi028_105698_object30035:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-02-05T06:42:20.723 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:07.595037+0000", 2024-02-05T06:42:20.723 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 133.04400911799999, 2024-02-05T06:42:20.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0589549119999999, 2024-02-05T06:42:20.723 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:42:20.723 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:42:20.723 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:42:20.724 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:42:20.724 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:42:20.724 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 30036 2024-02-05T06:42:20.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.724 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:42:20.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.724 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:42:20.724 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.595037+0000", 2024-02-05T06:42:20.724 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.724 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:42:20.724 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.595037+0000", 2024-02-05T06:42:20.724 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.725 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:42:20.725 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.595040+0000", 2024-02-05T06:42:20.725 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1229999999999998e-06 2024-02-05T06:42:20.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.725 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:42:20.725 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.595064+0000", 2024-02-05T06:42:20.725 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4287999999999999e-05 2024-02-05T06:42:20.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.725 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:42:20.725 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.595077+0000", 2024-02-05T06:42:20.725 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3159000000000001e-05 2024-02-05T06:42:20.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.726 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:42:20.726 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.595086+0000", 2024-02-05T06:42:20.726 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6780000000000006e-06 2024-02-05T06:42:20.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.726 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.726 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.595321+0000", 2024-02-05T06:42:20.726 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00023525099999999999 2024-02-05T06:42:20.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.726 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:42:20.726 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.595334+0000", 2024-02-05T06:42:20.726 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3071000000000001e-05 2024-02-05T06:42:20.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.727 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.727 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:08.563056+0000", 2024-02-05T06:42:20.727 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.96772240499999995 2024-02-05T06:42:20.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.727 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:42:20.727 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:08.563070+0000", 2024-02-05T06:42:20.727 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3611000000000001e-05 2024-02-05T06:42:20.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.727 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.727 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.646864+0000", 2024-02-05T06:42:20.727 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.083793934 2024-02-05T06:42:20.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.727 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:42:20.728 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.647049+0000", 2024-02-05T06:42:20.728 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018533800000000001 2024-02-05T06:42:20.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.728 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4", 2024-02-05T06:42:20.728 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.647246+0000", 2024-02-05T06:42:20.728 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019643199999999999 2024-02-05T06:42:20.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.728 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:42:20.728 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.650769+0000", 2024-02-05T06:42:20.728 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035230880000000002 2024-02-05T06:42:20.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.729 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.729 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.651480+0000", 2024-02-05T06:42:20.729 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00071135899999999995 2024-02-05T06:42:20.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.729 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.729 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.652111+0000", 2024-02-05T06:42:20.729 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00063044499999999999 2024-02-05T06:42:20.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.729 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:42:20.729 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.652396+0000", 2024-02-05T06:42:20.729 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028486199999999998 2024-02-05T06:42:20.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.730 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:42:20.730 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.653992+0000", 2024-02-05T06:42:20.730 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001596868 2024-02-05T06:42:20.730 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.730 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:42:20.730 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.730 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:30042 8.c 8:3c2527a0:::benchmark_data_smithi028_105698_object30041:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-02-05T06:42:20.730 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:07.601267+0000", 2024-02-05T06:42:20.730 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 133.03777927100001, 2024-02-05T06:42:20.730 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.054788839, 2024-02-05T06:42:20.730 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:42:20.730 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:42:20.730 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:42:20.731 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:42:20.731 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:42:20.731 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 30042 2024-02-05T06:42:20.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.731 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:42:20.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.731 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:42:20.731 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.601267+0000", 2024-02-05T06:42:20.731 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.731 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:42:20.731 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.601267+0000", 2024-02-05T06:42:20.731 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.732 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:42:20.732 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.601270+0000", 2024-02-05T06:42:20.732 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6029999999999999e-06 2024-02-05T06:42:20.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.732 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:42:20.732 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.601292+0000", 2024-02-05T06:42:20.732 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2370000000000001e-05 2024-02-05T06:42:20.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.733 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:42:20.733 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.601308+0000", 2024-02-05T06:42:20.733 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5733e-05 2024-02-05T06:42:20.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.733 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:42:20.733 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.601316+0000", 2024-02-05T06:42:20.733 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.807e-06 2024-02-05T06:42:20.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.733 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.733 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.601368+0000", 2024-02-05T06:42:20.733 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2522999999999998e-05 2024-02-05T06:42:20.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.734 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:42:20.734 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.601378+0000", 2024-02-05T06:42:20.734 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9250000000000007e-06 2024-02-05T06:42:20.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.734 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.734 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:08.563096+0000", 2024-02-05T06:42:20.734 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.96171812499999998 2024-02-05T06:42:20.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.734 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:42:20.734 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:08.563104+0000", 2024-02-05T06:42:20.734 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8709999999999997e-06 2024-02-05T06:42:20.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.735 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.735 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.648708+0000", 2024-02-05T06:42:20.735 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0856032419999999 2024-02-05T06:42:20.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.735 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.735 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:42:20.735 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.648872+0000", 2024-02-05T06:42:20.735 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016487399999999999 2024-02-05T06:42:20.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.735 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.735 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4", 2024-02-05T06:42:20.735 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.649103+0000", 2024-02-05T06:42:20.735 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00023107999999999999 2024-02-05T06:42:20.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.735 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.735 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:42:20.736 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.650988+0000", 2024-02-05T06:42:20.736 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018845820000000001 2024-02-05T06:42:20.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.736 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.736 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.651528+0000", 2024-02-05T06:42:20.736 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00053995900000000001 2024-02-05T06:42:20.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.736 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.736 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.654061+0000", 2024-02-05T06:42:20.736 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025332409999999999 2024-02-05T06:42:20.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.736 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:42:20.737 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.654415+0000", 2024-02-05T06:42:20.737 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00035333199999999998 2024-02-05T06:42:20.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.737 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:42:20.737 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.656056+0000", 2024-02-05T06:42:20.737 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001641572 2024-02-05T06:42:20.737 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.737 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:42:20.737 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.737 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:44130 8.6 8:69a28b8b:::benchmark_data_smithi028_105698_object44129:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-02-05T06:42:20.737 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:26.029868+0000", 2024-02-05T06:42:20.737 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 114.609178417, 2024-02-05T06:42:20.738 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1427735559999999, 2024-02-05T06:42:20.738 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:42:20.738 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-02-05T06:42:20.738 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:42:20.738 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:42:20.738 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:42:20.738 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 44130 2024-02-05T06:42:20.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.738 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:42:20.738 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.738 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:42:20.738 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.029868+0000", 2024-02-05T06:42:20.738 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.738 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.739 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:42:20.739 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.029868+0000", 2024-02-05T06:42:20.739 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.739 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.739 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:42:20.739 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.029870+0000", 2024-02-05T06:42:20.739 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.255e-06 2024-02-05T06:42:20.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.739 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.739 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:42:20.739 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.029894+0000", 2024-02-05T06:42:20.739 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4043999999999999e-05 2024-02-05T06:42:20.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.739 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.739 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:42:20.740 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.029907+0000", 2024-02-05T06:42:20.740 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2666e-05 2024-02-05T06:42:20.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.740 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.740 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:42:20.740 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.029916+0000", 2024-02-05T06:42:20.740 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1579999999999994e-06 2024-02-05T06:42:20.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.740 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.740 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.740 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.031114+0000", 2024-02-05T06:42:20.740 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0011976700000000001 2024-02-05T06:42:20.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.740 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.741 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:42:20.741 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.031300+0000", 2024-02-05T06:42:20.741 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018568600000000001 2024-02-05T06:42:20.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.741 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.741 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,6", 2024-02-05T06:42:20.741 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.031461+0000", 2024-02-05T06:42:20.741 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000161099 2024-02-05T06:42:20.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.741 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.741 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:42:20.741 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.035193+0000", 2024-02-05T06:42:20.741 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0037321699999999999 2024-02-05T06:42:20.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.741 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.741 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.742 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.037176+0000", 2024-02-05T06:42:20.742 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0019833709999999998 2024-02-05T06:42:20.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.742 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.742 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.742 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.070634+0000", 2024-02-05T06:42:20.742 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.033457496000000003 2024-02-05T06:42:20.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.742 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.743 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:42:20.744 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.070646+0000", 2024-02-05T06:42:20.744 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2045000000000001e-05 2024-02-05T06:42:20.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.744 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.744 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.744 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.064553+0000", 2024-02-05T06:42:20.744 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.99390664399999995 2024-02-05T06:42:20.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.744 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.744 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:42:20.744 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.064566+0000", 2024-02-05T06:42:20.744 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3763e-05 2024-02-05T06:42:20.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.745 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.745 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.745 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.172577+0000", 2024-02-05T06:42:20.745 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1080111610000001 2024-02-05T06:42:20.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.745 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.745 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:42:20.745 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.172642+0000", 2024-02-05T06:42:20.745 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4327999999999998e-05 2024-02-05T06:42:20.745 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.745 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:42:20.745 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.745 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.745 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:44147 8.c 8:3925757e:::benchmark_data_smithi028_105698_object44146:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-02-05T06:42:20.746 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:26.048728+0000", 2024-02-05T06:42:20.746 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 114.590318854, 2024-02-05T06:42:20.746 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1723810800000001, 2024-02-05T06:42:20.746 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:42:20.746 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:42:20.746 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:42:20.746 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:42:20.746 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:42:20.746 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 44147 2024-02-05T06:42:20.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.746 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:42:20.746 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.746 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:42:20.746 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.048728+0000", 2024-02-05T06:42:20.746 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.747 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.747 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:42:20.747 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.048728+0000", 2024-02-05T06:42:20.747 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.747 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.747 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:42:20.747 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.048730+0000", 2024-02-05T06:42:20.747 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.199e-06 2024-02-05T06:42:20.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.747 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.747 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:42:20.747 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.048756+0000", 2024-02-05T06:42:20.747 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6001999999999999e-05 2024-02-05T06:42:20.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.748 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.748 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:42:20.748 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.048766+0000", 2024-02-05T06:42:20.748 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0249e-05 2024-02-05T06:42:20.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.748 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.748 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:42:20.748 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.048776+0000", 2024-02-05T06:42:20.748 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7720000000000006e-06 2024-02-05T06:42:20.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.748 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.748 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.748 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.068897+0000", 2024-02-05T06:42:20.748 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020121066999999999 2024-02-05T06:42:20.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.749 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:42:20.749 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.068913+0000", 2024-02-05T06:42:20.749 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5508000000000001e-05 2024-02-05T06:42:20.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.749 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.749 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.210189+0000", 2024-02-05T06:42:20.749 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1412760829999999 2024-02-05T06:42:20.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.749 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:42:20.749 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.210349+0000", 2024-02-05T06:42:20.749 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016018 2024-02-05T06:42:20.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.750 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.750 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-02-05T06:42:20.750 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.210541+0000", 2024-02-05T06:42:20.750 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000192385 2024-02-05T06:42:20.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.750 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.750 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.750 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.217596+0000", 2024-02-05T06:42:20.750 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0070543439999999997 2024-02-05T06:42:20.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.750 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.750 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:42:20.750 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.217718+0000", 2024-02-05T06:42:20.750 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012252500000000001 2024-02-05T06:42:20.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.751 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.751 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.751 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.218772+0000", 2024-02-05T06:42:20.751 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00105378 2024-02-05T06:42:20.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.751 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.751 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:42:20.751 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.218866+0000", 2024-02-05T06:42:20.751 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3962000000000006e-05 2024-02-05T06:42:20.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.751 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.751 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:42:20.751 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.221109+0000", 2024-02-05T06:42:20.751 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022430240000000001 2024-02-05T06:42:20.751 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.751 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:42:20.752 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.752 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.752 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:44148 8.6 8:674244b6:::benchmark_data_smithi028_105698_object44147:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-02-05T06:42:20.752 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:26.065725+0000", 2024-02-05T06:42:20.752 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 114.573322057, 2024-02-05T06:42:20.752 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1232849570000001, 2024-02-05T06:42:20.752 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:42:20.752 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:42:20.752 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:42:20.752 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:42:20.752 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:42:20.752 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 44148 2024-02-05T06:42:20.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.752 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:42:20.753 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.753 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:42:20.753 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.065725+0000", 2024-02-05T06:42:20.753 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.753 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.753 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:42:20.753 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.065725+0000", 2024-02-05T06:42:20.753 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.753 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.753 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:42:20.753 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.065726+0000", 2024-02-05T06:42:20.753 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5260000000000001e-06 2024-02-05T06:42:20.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.754 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.754 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:42:20.754 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.065753+0000", 2024-02-05T06:42:20.754 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6485000000000001e-05 2024-02-05T06:42:20.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.754 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.754 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:42:20.754 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.065760+0000", 2024-02-05T06:42:20.754 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.763e-06 2024-02-05T06:42:20.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.754 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.754 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:42:20.754 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.065768+0000", 2024-02-05T06:42:20.754 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9699999999999999e-06 2024-02-05T06:42:20.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.755 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.755 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.755 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.070676+0000", 2024-02-05T06:42:20.755 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049074990000000001 2024-02-05T06:42:20.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.755 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.755 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:42:20.755 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.070684+0000", 2024-02-05T06:42:20.755 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9100000000000005e-06 2024-02-05T06:42:20.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.755 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.755 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.755 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.064596+0000", 2024-02-05T06:42:20.755 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.99391264999999995 2024-02-05T06:42:20.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.755 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.756 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:42:20.756 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.064605+0000", 2024-02-05T06:42:20.756 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.799e-06 2024-02-05T06:42:20.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.756 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.756 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.756 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.173907+0000", 2024-02-05T06:42:20.756 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1093021279999999 2024-02-05T06:42:20.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.756 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.756 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:42:20.756 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.174032+0000", 2024-02-05T06:42:20.756 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012484200000000001 2024-02-05T06:42:20.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.757 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.757 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 6,7", 2024-02-05T06:42:20.757 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.174188+0000", 2024-02-05T06:42:20.757 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000156121 2024-02-05T06:42:20.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.757 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.757 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:42:20.757 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.179323+0000", 2024-02-05T06:42:20.757 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051351560000000001 2024-02-05T06:42:20.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.757 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.757 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.757 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.179359+0000", 2024-02-05T06:42:20.757 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5886999999999997e-05 2024-02-05T06:42:20.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.758 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.758 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.758 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.187345+0000", 2024-02-05T06:42:20.758 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0079857669999999995 2024-02-05T06:42:20.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.758 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.758 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:42:20.758 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.187383+0000", 2024-02-05T06:42:20.758 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7605000000000003e-05 2024-02-05T06:42:20.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.758 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.758 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:42:20.758 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.189010+0000", 2024-02-05T06:42:20.758 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0016268490000000001 2024-02-05T06:42:20.758 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.759 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:42:20.759 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.759 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.759 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:44151 8.c 8:355b2e83:::benchmark_data_smithi028_105698_object44150:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-02-05T06:42:20.759 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:26.080665+0000", 2024-02-05T06:42:20.759 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 114.558381172, 2024-02-05T06:42:20.759 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.142344483, 2024-02-05T06:42:20.759 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:42:20.759 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:42:20.759 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:42:20.759 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:42:20.759 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:42:20.759 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 44151 2024-02-05T06:42:20.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.760 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:42:20.760 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.760 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:42:20.760 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.080665+0000", 2024-02-05T06:42:20.760 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.760 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.760 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:42:20.760 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.080665+0000", 2024-02-05T06:42:20.760 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.760 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.760 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:42:20.760 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.080667+0000", 2024-02-05T06:42:20.760 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.888e-06 2024-02-05T06:42:20.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.761 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.761 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:42:20.761 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.080692+0000", 2024-02-05T06:42:20.761 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5080999999999999e-05 2024-02-05T06:42:20.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.761 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.761 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:42:20.761 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.080703+0000", 2024-02-05T06:42:20.761 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.045e-05 2024-02-05T06:42:20.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.761 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.761 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:42:20.761 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.080712+0000", 2024-02-05T06:42:20.761 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4180000000000003e-06 2024-02-05T06:42:20.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.762 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.762 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.762 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.080900+0000", 2024-02-05T06:42:20.762 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018818 2024-02-05T06:42:20.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.762 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.762 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:42:20.762 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.080912+0000", 2024-02-05T06:42:20.762 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1569000000000001e-05 2024-02-05T06:42:20.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.762 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.762 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.762 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.212374+0000", 2024-02-05T06:42:20.762 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.131462301 2024-02-05T06:42:20.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.763 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.763 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:42:20.763 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.212564+0000", 2024-02-05T06:42:20.763 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018985900000000001 2024-02-05T06:42:20.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.763 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.763 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-02-05T06:42:20.763 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.212729+0000", 2024-02-05T06:42:20.763 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016445199999999999 2024-02-05T06:42:20.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.763 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.763 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.763 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.217671+0000", 2024-02-05T06:42:20.763 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049428229999999998 2024-02-05T06:42:20.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.764 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.764 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:42:20.764 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.217724+0000", 2024-02-05T06:42:20.764 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2964000000000003e-05 2024-02-05T06:42:20.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.764 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.764 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.764 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.221148+0000", 2024-02-05T06:42:20.764 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034238480000000002 2024-02-05T06:42:20.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.764 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.764 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:42:20.764 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.221184+0000", 2024-02-05T06:42:20.764 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6154e-05 2024-02-05T06:42:20.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.765 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.765 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:42:20.765 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.223010+0000", 2024-02-05T06:42:20.765 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001825496 2024-02-05T06:42:20.765 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.765 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:42:20.765 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.765 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.765 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:44155 8.6 8:67a30c66:::benchmark_data_smithi028_105698_object44154:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-02-05T06:42:20.765 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:26.095757+0000", 2024-02-05T06:42:20.765 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 114.54328998, 2024-02-05T06:42:20.765 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0990290489999999, 2024-02-05T06:42:20.765 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:42:20.765 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:42:20.766 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:42:20.766 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:42:20.766 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:42:20.766 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 44155 2024-02-05T06:42:20.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.766 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:42:20.766 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.766 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:42:20.766 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.095757+0000", 2024-02-05T06:42:20.766 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.766 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.766 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:42:20.766 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.095757+0000", 2024-02-05T06:42:20.766 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.767 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:42:20.767 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.095759+0000", 2024-02-05T06:42:20.767 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.238e-06 2024-02-05T06:42:20.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.767 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:42:20.767 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.095784+0000", 2024-02-05T06:42:20.767 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5406999999999999e-05 2024-02-05T06:42:20.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.767 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:42:20.767 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.095794+0000", 2024-02-05T06:42:20.767 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9429999999999995e-06 2024-02-05T06:42:20.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.768 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:42:20.768 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.095801+0000", 2024-02-05T06:42:20.768 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9079999999999998e-06 2024-02-05T06:42:20.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.768 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.768 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.096034+0000", 2024-02-05T06:42:20.768 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000232921 2024-02-05T06:42:20.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.768 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:42:20.768 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.096044+0000", 2024-02-05T06:42:20.768 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6949999999999995e-06 2024-02-05T06:42:20.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.769 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.769 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.769 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.064631+0000", 2024-02-05T06:42:20.769 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.96858761900000001 2024-02-05T06:42:20.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.769 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.769 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:42:20.769 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.064640+0000", 2024-02-05T06:42:20.769 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5969999999999992e-06 2024-02-05T06:42:20.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.769 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.769 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.769 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.175581+0000", 2024-02-05T06:42:20.769 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1109413770000001 2024-02-05T06:42:20.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.770 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:42:20.770 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.175654+0000", 2024-02-05T06:42:20.770 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2161000000000002e-05 2024-02-05T06:42:20.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.770 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 6,7", 2024-02-05T06:42:20.770 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.175761+0000", 2024-02-05T06:42:20.770 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010715 2024-02-05T06:42:20.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.770 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:42:20.770 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.179379+0000", 2024-02-05T06:42:20.770 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0036188430000000001 2024-02-05T06:42:20.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.771 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.771 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.182192+0000", 2024-02-05T06:42:20.771 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028123649999999998 2024-02-05T06:42:20.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.771 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.771 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.771 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.193550+0000", 2024-02-05T06:42:20.771 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011357812 2024-02-05T06:42:20.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.771 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.771 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:42:20.771 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.193588+0000", 2024-02-05T06:42:20.771 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7843999999999999e-05 2024-02-05T06:42:20.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.772 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.772 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:42:20.772 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.194786+0000", 2024-02-05T06:42:20.772 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001198169 2024-02-05T06:42:20.772 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.772 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:42:20.772 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.772 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.772 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:44158 8.c 8:32d71fe1:::benchmark_data_smithi028_105698_object44157:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-02-05T06:42:20.772 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:26.103850+0000", 2024-02-05T06:42:20.772 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 114.53519700699999, 2024-02-05T06:42:20.772 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1699483719999999, 2024-02-05T06:42:20.772 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:42:20.772 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:42:20.773 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:42:20.773 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:42:20.773 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:42:20.773 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 44158 2024-02-05T06:42:20.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.773 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:42:20.773 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.773 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:42:20.773 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.103850+0000", 2024-02-05T06:42:20.773 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.773 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.773 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:42:20.773 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.103850+0000", 2024-02-05T06:42:20.774 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.774 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.774 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:42:20.774 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.103852+0000", 2024-02-05T06:42:20.774 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2850000000000001e-06 2024-02-05T06:42:20.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.774 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.774 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:42:20.774 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.103878+0000", 2024-02-05T06:42:20.774 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5979999999999999e-05 2024-02-05T06:42:20.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.774 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.774 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:42:20.774 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.103889+0000", 2024-02-05T06:42:20.775 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1473e-05 2024-02-05T06:42:20.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.775 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.775 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:42:20.775 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.103898+0000", 2024-02-05T06:42:20.775 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6559999999999998e-06 2024-02-05T06:42:20.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.775 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.775 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.775 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.103975+0000", 2024-02-05T06:42:20.775 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7322000000000003e-05 2024-02-05T06:42:20.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.775 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.775 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:42:20.775 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.103990+0000", 2024-02-05T06:42:20.776 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4572e-05 2024-02-05T06:42:20.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.776 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.776 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.776 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.215210+0000", 2024-02-05T06:42:20.776 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1112202360000001 2024-02-05T06:42:20.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.776 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.776 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:42:20.776 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.215353+0000", 2024-02-05T06:42:20.776 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014238200000000001 2024-02-05T06:42:20.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.776 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.776 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-02-05T06:42:20.776 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.215466+0000", 2024-02-05T06:42:20.777 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011366699999999999 2024-02-05T06:42:20.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.777 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.777 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:42:20.777 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.218079+0000", 2024-02-05T06:42:20.777 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026129769999999998 2024-02-05T06:42:20.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.777 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.777 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.777 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.223830+0000", 2024-02-05T06:42:20.777 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0057504670000000004 2024-02-05T06:42:20.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.777 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.777 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.778 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.271247+0000", 2024-02-05T06:42:20.778 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.047417775000000002 2024-02-05T06:42:20.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.778 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.778 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:42:20.778 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.271316+0000", 2024-02-05T06:42:20.778 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8305000000000004e-05 2024-02-05T06:42:20.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.778 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.778 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:42:20.778 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.273798+0000", 2024-02-05T06:42:20.778 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024822749999999999 2024-02-05T06:42:20.778 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.778 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:42:20.778 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.779 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.779 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:44164 8.6 8:6695897a:::benchmark_data_smithi028_105698_object44163:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-02-05T06:42:20.779 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:27.204120+0000", 2024-02-05T06:42:20.779 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 113.43492693, 2024-02-05T06:42:20.779 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.005835, 2024-02-05T06:42:20.779 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:42:20.779 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:42:20.779 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:42:20.779 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:42:20.779 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:42:20.779 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 44164 2024-02-05T06:42:20.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.779 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:42:20.779 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.780 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:42:20.780 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.204120+0000", 2024-02-05T06:42:20.780 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.780 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.780 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.780 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:42:20.780 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.204120+0000", 2024-02-05T06:42:20.780 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.780 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.780 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.780 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:42:20.780 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.204121+0000", 2024-02-05T06:42:20.780 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7090000000000001e-06 2024-02-05T06:42:20.780 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.781 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.781 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:42:20.781 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.204148+0000", 2024-02-05T06:42:20.781 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6495e-05 2024-02-05T06:42:20.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.781 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.781 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:42:20.781 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.204163+0000", 2024-02-05T06:42:20.781 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5177000000000001e-05 2024-02-05T06:42:20.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.781 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.781 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:42:20.781 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.204172+0000", 2024-02-05T06:42:20.781 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4800000000000001e-06 2024-02-05T06:42:20.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.782 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.782 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.782 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.209707+0000", 2024-02-05T06:42:20.782 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055354000000000002 2024-02-05T06:42:20.782 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.782 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.782 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:42:20.782 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.209716+0000", 2024-02-05T06:42:20.782 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5179999999999993e-06 2024-02-05T06:42:20.782 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.782 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.782 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.782 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.176460+0000", 2024-02-05T06:42:20.782 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.96674394900000005 2024-02-05T06:42:20.782 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.782 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.783 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:42:20.783 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.176579+0000", 2024-02-05T06:42:20.783 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000118937 2024-02-05T06:42:20.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.783 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.783 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 6,7", 2024-02-05T06:42:20.783 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.176709+0000", 2024-02-05T06:42:20.783 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012960200000000001 2024-02-05T06:42:20.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.783 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.783 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:42:20.783 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.181144+0000", 2024-02-05T06:42:20.783 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0044347759999999996 2024-02-05T06:42:20.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.783 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.784 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.784 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.183642+0000", 2024-02-05T06:42:20.784 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024978779999999998 2024-02-05T06:42:20.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.784 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.784 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.784 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.194820+0000", 2024-02-05T06:42:20.784 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011178641 2024-02-05T06:42:20.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.784 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.784 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:42:20.784 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.207712+0000", 2024-02-05T06:42:20.784 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012892173 2024-02-05T06:42:20.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.784 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.785 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:42:20.785 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.209955+0000", 2024-02-05T06:42:20.785 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022422649999999998 2024-02-05T06:42:20.785 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.785 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:42:20.785 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.785 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.785 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:106739 8.8 8:117d8294:::benchmark_data_smithi028_105698_object17464:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e248)", 2024-02-05T06:42:20.785 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:41:43.397529+0000", 2024-02-05T06:42:20.785 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.241517442000003, 2024-02-05T06:42:20.785 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1348037630000001, 2024-02-05T06:42:20.785 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:42:20.785 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:42:20.785 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:42:20.786 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:42:20.786 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:42:20.786 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 106739 2024-02-05T06:42:20.786 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.786 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:42:20.786 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.786 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:42:20.786 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397529+0000", 2024-02-05T06:42:20.786 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.786 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.786 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.786 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:42:20.786 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397529+0000", 2024-02-05T06:42:20.786 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.786 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.787 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.787 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:42:20.787 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397531+0000", 2024-02-05T06:42:20.787 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.567e-06 2024-02-05T06:42:20.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.787 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.787 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:42:20.787 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397535+0000", 2024-02-05T06:42:20.787 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4399999999999998e-06 2024-02-05T06:42:20.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.787 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.787 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:42:20.787 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397543+0000", 2024-02-05T06:42:20.787 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.61e-06 2024-02-05T06:42:20.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.788 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.788 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:42:20.788 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397551+0000", 2024-02-05T06:42:20.788 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6640000000000004e-06 2024-02-05T06:42:20.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.788 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.788 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.788 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397656+0000", 2024-02-05T06:42:20.788 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000104753 2024-02-05T06:42:20.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.788 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.788 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:42:20.788 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397669+0000", 2024-02-05T06:42:20.788 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3179e-05 2024-02-05T06:42:20.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.789 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.789 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.497144+0000", 2024-02-05T06:42:20.789 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0994749159999999 2024-02-05T06:42:20.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.789 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:42:20.789 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.497289+0000", 2024-02-05T06:42:20.789 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014480499999999999 2024-02-05T06:42:20.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.789 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,5,7", 2024-02-05T06:42:20.789 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.497433+0000", 2024-02-05T06:42:20.789 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000144034 2024-02-05T06:42:20.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.790 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:42:20.790 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.499403+0000", 2024-02-05T06:42:20.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0019702550000000002 2024-02-05T06:42:20.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.790 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.790 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.500873+0000", 2024-02-05T06:42:20.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001469162 2024-02-05T06:42:20.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.790 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.790 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.502297+0000", 2024-02-05T06:42:20.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001424252 2024-02-05T06:42:20.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.532124+0000", 2024-02-05T06:42:20.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029826930000000001 2024-02-05T06:42:20.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:42:20.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.532193+0000", 2024-02-05T06:42:20.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9559000000000003e-05 2024-02-05T06:42:20.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:42:20.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.532333+0000", 2024-02-05T06:42:20.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013963699999999999 2024-02-05T06:42:20.792 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.792 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:42:20.792 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.792 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:106744 8.8 8:11795b7a:::benchmark_data_smithi028_105698_object17469:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e248)", 2024-02-05T06:42:20.792 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:41:43.397570+0000", 2024-02-05T06:42:20.792 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 37.241476626999997, 2024-02-05T06:42:20.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1349640889999999, 2024-02-05T06:42:20.792 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:42:20.792 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:42:20.792 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:42:20.792 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:42:20.792 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:42:20.792 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 106744 2024-02-05T06:42:20.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.793 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:42:20.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.793 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:42:20.793 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397570+0000", 2024-02-05T06:42:20.793 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.793 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:42:20.793 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397570+0000", 2024-02-05T06:42:20.793 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.793 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:42:20.794 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397571+0000", 2024-02-05T06:42:20.794 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4600000000000003e-07 2024-02-05T06:42:20.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.795 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:42:20.795 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397574+0000", 2024-02-05T06:42:20.795 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4199999999999999e-06 2024-02-05T06:42:20.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.795 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.795 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:42:20.795 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397581+0000", 2024-02-05T06:42:20.795 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.455e-06 2024-02-05T06:42:20.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.796 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:42:20.796 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397587+0000", 2024-02-05T06:42:20.796 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9619999999999998e-06 2024-02-05T06:42:20.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.796 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.796 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397712+0000", 2024-02-05T06:42:20.796 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012582399999999999 2024-02-05T06:42:20.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.796 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:42:20.796 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397723+0000", 2024-02-05T06:42:20.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0026e-05 2024-02-05T06:42:20.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.797 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.797 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.499468+0000", 2024-02-05T06:42:20.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1017459270000001 2024-02-05T06:42:20.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.797 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:42:20.797 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.499545+0000", 2024-02-05T06:42:20.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6618000000000004e-05 2024-02-05T06:42:20.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.797 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,5,7", 2024-02-05T06:42:20.797 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.499616+0000", 2024-02-05T06:42:20.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0776000000000001e-05 2024-02-05T06:42:20.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.798 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:42:20.798 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.500824+0000", 2024-02-05T06:42:20.798 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0012077660000000001 2024-02-05T06:42:20.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.798 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.798 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.501234+0000", 2024-02-05T06:42:20.798 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000410651 2024-02-05T06:42:20.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.798 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.798 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.502417+0000", 2024-02-05T06:42:20.798 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0011827910000000001 2024-02-05T06:42:20.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.799 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.799 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.532375+0000", 2024-02-05T06:42:20.799 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029957874999999998 2024-02-05T06:42:20.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.799 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:42:20.799 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.532432+0000", 2024-02-05T06:42:20.799 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7383000000000003e-05 2024-02-05T06:42:20.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.799 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:42:20.799 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.532534+0000", 2024-02-05T06:42:20.799 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010176899999999999 2024-02-05T06:42:20.800 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.800 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:42:20.800 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.800 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.800 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:107326 8.6 8:605b4dc7:::benchmark_data_smithi028_105698_object18051:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-02-05T06:42:20.800 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:41:45.056053+0000", 2024-02-05T06:42:20.800 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 35.582993651000002, 2024-02-05T06:42:20.800 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.012795213, 2024-02-05T06:42:20.800 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:42:20.800 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:42:20.800 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:42:20.800 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:42:20.800 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:42:20.801 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 107326 2024-02-05T06:42:20.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.801 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:42:20.801 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.801 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:42:20.801 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.056053+0000", 2024-02-05T06:42:20.801 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.801 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.801 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:42:20.801 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.056053+0000", 2024-02-05T06:42:20.801 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.801 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.801 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:42:20.802 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.056054+0000", 2024-02-05T06:42:20.802 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7400000000000002e-07 2024-02-05T06:42:20.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.802 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:42:20.802 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.056057+0000", 2024-02-05T06:42:20.802 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4460000000000001e-06 2024-02-05T06:42:20.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.802 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:42:20.802 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.056064+0000", 2024-02-05T06:42:20.802 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3010000000000001e-06 2024-02-05T06:42:20.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.802 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:42:20.803 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.056072+0000", 2024-02-05T06:42:20.803 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.559e-06 2024-02-05T06:42:20.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.803 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.803 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.065347+0000", 2024-02-05T06:42:20.803 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0092747249999999993 2024-02-05T06:42:20.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.803 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:42:20.803 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.065357+0000", 2024-02-05T06:42:20.803 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9410000000000002e-06 2024-02-05T06:42:20.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.803 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.804 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.062162+0000", 2024-02-05T06:42:20.804 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.99680566999999998 2024-02-05T06:42:20.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.804 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.804 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:42:20.804 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.062297+0000", 2024-02-05T06:42:20.804 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000134752 2024-02-05T06:42:20.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.804 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.804 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 6,7", 2024-02-05T06:42:20.804 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.062396+0000", 2024-02-05T06:42:20.804 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8632e-05 2024-02-05T06:42:20.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.804 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.804 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.805 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.068630+0000", 2024-02-05T06:42:20.805 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062346650000000003 2024-02-05T06:42:20.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.805 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.805 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:42:20.805 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.068646+0000", 2024-02-05T06:42:20.805 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5616000000000001e-05 2024-02-05T06:42:20.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.805 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.805 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.805 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.068758+0000", 2024-02-05T06:42:20.805 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011232500000000001 2024-02-05T06:42:20.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.805 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.805 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:42:20.806 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.068811+0000", 2024-02-05T06:42:20.806 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2911000000000002e-05 2024-02-05T06:42:20.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.806 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:42:20.806 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.068848+0000", 2024-02-05T06:42:20.806 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6896e-05 2024-02-05T06:42:20.806 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.806 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:42:20.806 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.806 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:107355 8.6 8:6f7c225d:::benchmark_data_smithi028_105698_object18080:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-02-05T06:42:20.806 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:41:45.094245+0000", 2024-02-05T06:42:20.806 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 35.544802023000003, 2024-02-05T06:42:20.807 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.97918498600000003, 2024-02-05T06:42:20.807 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:42:20.807 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:42:20.807 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:42:20.807 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:42:20.807 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:42:20.807 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 107355 2024-02-05T06:42:20.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.807 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:42:20.807 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.807 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:42:20.807 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.094245+0000", 2024-02-05T06:42:20.807 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.808 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:42:20.808 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.094245+0000", 2024-02-05T06:42:20.808 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.808 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:42:20.808 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.094247+0000", 2024-02-05T06:42:20.808 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9690000000000001e-06 2024-02-05T06:42:20.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.808 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:42:20.808 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.094252+0000", 2024-02-05T06:42:20.808 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7570000000000001e-06 2024-02-05T06:42:20.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.809 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:42:20.809 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.094273+0000", 2024-02-05T06:42:20.809 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0781e-05 2024-02-05T06:42:20.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.809 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:42:20.809 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.094282+0000", 2024-02-05T06:42:20.809 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4889999999999995e-06 2024-02-05T06:42:20.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.809 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.809 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.094346+0000", 2024-02-05T06:42:20.809 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4120000000000003e-05 2024-02-05T06:42:20.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.810 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:42:20.810 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.094356+0000", 2024-02-05T06:42:20.810 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0767999999999999e-05 2024-02-05T06:42:20.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.810 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.810 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.064770+0000", 2024-02-05T06:42:20.810 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.97041343499999999 2024-02-05T06:42:20.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.810 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:42:20.810 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.064863+0000", 2024-02-05T06:42:20.810 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3134000000000006e-05 2024-02-05T06:42:20.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.811 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 6,7", 2024-02-05T06:42:20.811 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.064948+0000", 2024-02-05T06:42:20.811 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5088999999999998e-05 2024-02-05T06:42:20.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.811 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.811 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.068889+0000", 2024-02-05T06:42:20.811 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039410460000000001 2024-02-05T06:42:20.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.811 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:42:20.811 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.069069+0000", 2024-02-05T06:42:20.811 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000179684 2024-02-05T06:42:20.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.812 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.812 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.812 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.073324+0000", 2024-02-05T06:42:20.812 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004254908 2024-02-05T06:42:20.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.812 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.812 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:42:20.812 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.073380+0000", 2024-02-05T06:42:20.812 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6440999999999997e-05 2024-02-05T06:42:20.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.812 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.812 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:42:20.812 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.073430+0000", 2024-02-05T06:42:20.812 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9364999999999999e-05 2024-02-05T06:42:20.812 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.813 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:42:20.813 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.813 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.813 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:107363 8.6 8:679b97e2:::benchmark_data_smithi028_105698_object18088:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-02-05T06:42:20.813 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:41:45.099343+0000", 2024-02-05T06:42:20.813 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 35.539703441, 2024-02-05T06:42:20.813 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.97440879599999997, 2024-02-05T06:42:20.813 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:42:20.813 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:42:20.813 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:42:20.813 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:42:20.813 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:42:20.813 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 107363 2024-02-05T06:42:20.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.814 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:42:20.814 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.814 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:42:20.814 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.099343+0000", 2024-02-05T06:42:20.814 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.814 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.814 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:42:20.814 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.099343+0000", 2024-02-05T06:42:20.814 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:42:20.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.814 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.814 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:42:20.814 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.099344+0000", 2024-02-05T06:42:20.815 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3200000000000004e-07 2024-02-05T06:42:20.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.815 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:42:20.815 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.099348+0000", 2024-02-05T06:42:20.815 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1370000000000004e-06 2024-02-05T06:42:20.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.815 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:42:20.815 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.099355+0000", 2024-02-05T06:42:20.815 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.798e-06 2024-02-05T06:42:20.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.815 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:42:20.815 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.099362+0000", 2024-02-05T06:42:20.816 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6800000000000004e-06 2024-02-05T06:42:20.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.816 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.816 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.816 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.099506+0000", 2024-02-05T06:42:20.816 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014402600000000001 2024-02-05T06:42:20.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.816 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.816 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:42:20.816 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.099513+0000", 2024-02-05T06:42:20.816 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0040000000000002e-06 2024-02-05T06:42:20.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.816 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.816 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:42:20.816 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.069139+0000", 2024-02-05T06:42:20.817 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.96962602499999995 2024-02-05T06:42:20.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.817 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:42:20.817 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.069231+0000", 2024-02-05T06:42:20.817 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2391000000000004e-05 2024-02-05T06:42:20.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.817 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 6,7", 2024-02-05T06:42:20.817 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.069352+0000", 2024-02-05T06:42:20.817 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000120943 2024-02-05T06:42:20.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.817 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:42:20.817 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.073451+0000", 2024-02-05T06:42:20.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040991389999999999 2024-02-05T06:42:20.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.073632+0000", 2024-02-05T06:42:20.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000181044 2024-02-05T06:42:20.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:42:20.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.073677+0000", 2024-02-05T06:42:20.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4539999999999997e-05 2024-02-05T06:42:20.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:42:20.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.073723+0000", 2024-02-05T06:42:20.819 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5890999999999998e-05 2024-02-05T06:42:20.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:42:20.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:42:20.819 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:42:20.819 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.073752+0000", 2024-02-05T06:42:20.819 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9346e-05 2024-02-05T06:42:20.819 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.819 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:42:20.819 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.819 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:42:20.819 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:42:20.819 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:42:20.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:20.820 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:20.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:20.921 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:20.955 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:20.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:20.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:21.057 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:21.099 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:21.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:21.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:21.201 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:21.224 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:21.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:21.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:21.326 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:21.358 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:21.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:21.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:21.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:21 smithi028 ceph-mon[29442]: pgmap v432: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 3 active+remapped+backfilling, 153 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 721 op/s; 8859/133371 objects misplaced (6.642%); 24 MiB/s, 376 objects/s recovering 2024-02-05T06:42:21.431 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:21 smithi028 ceph-mon[37984]: pgmap v432: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 3 active+remapped+backfilling, 153 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 721 op/s; 8859/133371 objects misplaced (6.642%); 24 MiB/s, 376 objects/s recovering 2024-02-05T06:42:21.460 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:21.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:21 smithi167 ceph-mon[32774]: pgmap v432: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 3 active+remapped+backfilling, 153 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 721 op/s; 8859/133371 objects misplaced (6.642%); 24 MiB/s, 376 objects/s recovering 2024-02-05T06:42:21.503 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:21.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:21.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:21.605 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:21.645 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:21.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:21.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:21.747 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:21.788 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:21.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:21.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:21.890 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:21.922 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:21.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:21.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:22.024 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:22.051 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:22.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:22.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:22.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:22.196 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:22.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:22.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:22.298 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:22.332 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:22.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:22.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:22.434 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:22.474 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:22.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:22.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:22.575 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:22.611 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:22.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:22.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:22.713 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:22.748 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:22.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:22.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:22.768 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:42:22 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:42:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:42:22.851 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:22.882 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:22.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:22.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:22.984 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:23.019 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:23.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:23.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:23.121 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:23.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:23.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:23.232 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:42:22 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:42:23.247 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:23.280 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:23.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:23.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:23.382 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:23.405 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:23.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:23.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:23.507 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:23.537 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:23.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:23.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:23.639 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:23.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:23.668 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:23.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:23.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:23 smithi028 ceph-mon[29442]: pgmap v433: 158 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 154 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.21k op/s; 5007/120126 objects misplaced (4.168%); 47 MiB/s, 747 objects/s recovering 2024-02-05T06:42:23.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:23 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:42:23.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:23 smithi028 ceph-mon[37984]: pgmap v433: 158 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 154 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.21k op/s; 5007/120126 objects misplaced (4.168%); 47 MiB/s, 747 objects/s recovering 2024-02-05T06:42:23.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:23 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:42:23.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:23 smithi167 ceph-mon[32774]: pgmap v433: 158 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 154 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.21k op/s; 5007/120126 objects misplaced (4.168%); 47 MiB/s, 747 objects/s recovering 2024-02-05T06:42:23.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:23 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:42:23.769 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:23.798 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:23.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:23.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:23.900 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:23.926 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:23.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:23.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:24.028 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:24.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:24.052 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:24.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:24.153 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:24.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:24.178 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:24.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:24.280 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:24.304 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:24.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:24.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:24.407 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:24.432 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:24.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:24.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:24.534 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:24.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:24.569 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:24.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:24.670 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:24.694 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:24.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:24.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:24.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:24 smithi028 ceph-mon[29442]: osdmap e267: 8 total, 8 up, 8 in 2024-02-05T06:42:24.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:24 smithi028 ceph-mon[37984]: osdmap e267: 8 total, 8 up, 8 in 2024-02-05T06:42:24.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:24 smithi167 ceph-mon[32774]: osdmap e267: 8 total, 8 up, 8 in 2024-02-05T06:42:24.796 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:24.821 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:24.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:24.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:24.923 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:24.947 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:24.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:24.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:25.049 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:25.074 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:25.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:25.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:25.175 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:25.200 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:25.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:25.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:25.301 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:25.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:25.326 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:25.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:25.427 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:25.452 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:25.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:25.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:25.555 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:25.582 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:25.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:25.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:25.684 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:25.713 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:25.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:25.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:25.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:25 smithi167 ceph-mon[32774]: pgmap v435: 158 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 154 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 984 op/s; 5007/120126 objects misplaced (4.168%); 39 MiB/s, 616 objects/s recovering 2024-02-05T06:42:25.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:25 smithi167 ceph-mon[32774]: osdmap e268: 8 total, 8 up, 8 in 2024-02-05T06:42:25.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:25 smithi167 ceph-mon[32774]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:42:25.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:25.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:25.840 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:25.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:25.941 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:25.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:25 smithi028 ceph-mon[37984]: pgmap v435: 158 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 154 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 984 op/s; 5007/120126 objects misplaced (4.168%); 39 MiB/s, 616 objects/s recovering 2024-02-05T06:42:25.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:25 smithi028 ceph-mon[37984]: osdmap e268: 8 total, 8 up, 8 in 2024-02-05T06:42:25.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:25 smithi028 ceph-mon[37984]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:42:25.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:25 smithi028 ceph-mon[29442]: pgmap v435: 158 pgs: 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 154 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 984 op/s; 5007/120126 objects misplaced (4.168%); 39 MiB/s, 616 objects/s recovering 2024-02-05T06:42:25.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:25 smithi028 ceph-mon[29442]: osdmap e268: 8 total, 8 up, 8 in 2024-02-05T06:42:25.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:25 smithi028 ceph-mon[29442]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:42:25.964 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:25.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:25.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:26.066 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:26.089 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:26.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:26.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:26.191 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:26.220 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:26.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:26.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:26.322 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:26.346 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:26.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:26.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:26.448 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:26.475 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:26.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:26.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:26.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:26.621 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:26.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:26.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:26.723 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:26.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:26 smithi167 ceph-mon[32774]: osdmap e269: 8 total, 8 up, 8 in 2024-02-05T06:42:26.753 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:26.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:26.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:26.856 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:26.872 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:26 smithi028 ceph-mon[37984]: osdmap e269: 8 total, 8 up, 8 in 2024-02-05T06:42:26.872 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:26 smithi028 ceph-mon[29442]: osdmap e269: 8 total, 8 up, 8 in 2024-02-05T06:42:26.892 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:26.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:26.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:26.916 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-05T06:42:26.994 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:27.040 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:27.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:27.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:27.073 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:42:27.142 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:27.202 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:27.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:27.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:27.304 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:27.342 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:27.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:27.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:27.444 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:27.487 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:27.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:27.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:27.591 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:27.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:27.632 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:27.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:27.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:27 smithi028 ceph-mon[29442]: pgmap v438: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 155 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 833 op/s; 4459/118392 objects misplaced (3.766%); 39 MiB/s, 617 objects/s recovering 2024-02-05T06:42:27.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:27 smithi028 ceph-mon[29442]: osdmap e270: 8 total, 8 up, 8 in 2024-02-05T06:42:27.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:27 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:42:27.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:27 smithi028 ceph-mon[37984]: pgmap v438: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 155 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 833 op/s; 4459/118392 objects misplaced (3.766%); 39 MiB/s, 617 objects/s recovering 2024-02-05T06:42:27.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:27 smithi028 ceph-mon[37984]: osdmap e270: 8 total, 8 up, 8 in 2024-02-05T06:42:27.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:27 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:42:27.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:27 smithi167 ceph-mon[32774]: pgmap v438: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 155 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 833 op/s; 4459/118392 objects misplaced (3.766%); 39 MiB/s, 617 objects/s recovering 2024-02-05T06:42:27.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:27 smithi167 ceph-mon[32774]: osdmap e270: 8 total, 8 up, 8 in 2024-02-05T06:42:27.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:27 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:42:27.733 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:27.768 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:27.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:27.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:27.870 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:27.924 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:27.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:27.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:28.026 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:28.056 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:28.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:28.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:28.158 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:28.192 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:28.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:28.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:28.294 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:28.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:28.333 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:28.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:28.434 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:28.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:28.475 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:28.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:28.576 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:28.617 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:28.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:28.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:28.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:28.755 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:28.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:28.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:28.857 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:28.890 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:28.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:28.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:28.992 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:29.026 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:29.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:29.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:29.128 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:29.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:29.159 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:29.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:29.260 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:29.299 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:29.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:29.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:29.401 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:29.429 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:29.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:29.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:29.531 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:29.574 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:29.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:29.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:29.676 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:29.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:29 smithi028 ceph-mon[29442]: pgmap v440: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.3 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 592 op/s; 2302/111456 objects misplaced (2.065%); 31 MiB/s, 491 objects/s recovering 2024-02-05T06:42:29.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:29 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:42:29.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:29 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:42:29.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:29 smithi028 ceph-mon[29442]: osdmap e271: 8 total, 8 up, 8 in 2024-02-05T06:42:29.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:29 smithi028 ceph-mon[37984]: pgmap v440: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.3 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 592 op/s; 2302/111456 objects misplaced (2.065%); 31 MiB/s, 491 objects/s recovering 2024-02-05T06:42:29.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:29 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:42:29.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:29 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:42:29.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:29 smithi028 ceph-mon[37984]: osdmap e271: 8 total, 8 up, 8 in 2024-02-05T06:42:29.720 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:29.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:29.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:29.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:29 smithi167 ceph-mon[32774]: pgmap v440: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.3 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 592 op/s; 2302/111456 objects misplaced (2.065%); 31 MiB/s, 491 objects/s recovering 2024-02-05T06:42:29.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:29 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:42:29.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:29 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:42:29.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:29 smithi167 ceph-mon[32774]: osdmap e271: 8 total, 8 up, 8 in 2024-02-05T06:42:29.822 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:29.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:29.860 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:29.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:29.883 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:42:29.961 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:30.009 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:30.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:30.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:30.113 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:30.179 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:30.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:30.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:30.281 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:30.313 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:30.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:30.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:30.415 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:30.456 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:30.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:30.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:30.557 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:30.610 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:30.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:30.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:30.712 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:30.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:30.762 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:30.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:30.863 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:30.900 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:30.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:30.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:31.002 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:31.029 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:31.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:31.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:31.130 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:31.175 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:31.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:31.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:31.277 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:31.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:31.305 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:31.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:31.366 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:31 smithi028 ceph-mon[29442]: osdmap e272: 8 total, 8 up, 8 in 2024-02-05T06:42:31.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:31 smithi028 ceph-mon[29442]: pgmap v443: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.3 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 592 op/s; 2302/111456 objects misplaced (2.065%); 31 MiB/s, 491 objects/s recovering 2024-02-05T06:42:31.367 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:31 smithi028 ceph-mon[37984]: osdmap e272: 8 total, 8 up, 8 in 2024-02-05T06:42:31.367 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:31 smithi028 ceph-mon[37984]: pgmap v443: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.3 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 592 op/s; 2302/111456 objects misplaced (2.065%); 31 MiB/s, 491 objects/s recovering 2024-02-05T06:42:31.405 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:31.447 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:31.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:31.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:31.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:31 smithi167 ceph-mon[32774]: osdmap e272: 8 total, 8 up, 8 in 2024-02-05T06:42:31.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:31 smithi167 ceph-mon[32774]: pgmap v443: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.3 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 592 op/s; 2302/111456 objects misplaced (2.065%); 31 MiB/s, 491 objects/s recovering 2024-02-05T06:42:31.549 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:31.592 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:31.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:31.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:31.693 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:31.735 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:31.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:31.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:31.737 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:42:31.790 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:42:31.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:31.845 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:42:31.879 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:31.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:31.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:31.888 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:42:31.924 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:42:31.966 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:42:31.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:32.006 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:42:32.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:32.020 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:32.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:32.046 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:42:32.049 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:42:32.049 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:42:32.049 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:42:32.049 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:42:32.049 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:42:32.049 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:42:32.049 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:42:32.049 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:42:32.101 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-02-05T06:42:32.109 INFO:teuthology.orchestra.run.smithi028.stderr:set 5.1b pg_upmap mapping to [3,0,1] 2024-02-05T06:42:32.121 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:32.146 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:32.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:32.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:32.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:32.289 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:32.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:32.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:32.391 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:32.421 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:32.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:32.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:32.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:32 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1704835886' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1b", "id": [3, 0, 1]}]: dispatch 2024-02-05T06:42:32.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:32 smithi028 ceph-mon[29442]: from='client.25186 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1b", "id": [3, 0, 1]}]: dispatch 2024-02-05T06:42:32.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:32 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/674578421' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:42:32.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:32 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2396013368' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:42:32.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:32 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1704835886' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1b", "id": [3, 0, 1]}]: dispatch 2024-02-05T06:42:32.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:32 smithi028 ceph-mon[37984]: from='client.25186 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1b", "id": [3, 0, 1]}]: dispatch 2024-02-05T06:42:32.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:32 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/674578421' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:42:32.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:32 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2396013368' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:42:32.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:32 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1704835886' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1b", "id": [3, 0, 1]}]: dispatch 2024-02-05T06:42:32.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:32 smithi167 ceph-mon[32774]: from='client.25186 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.1b", "id": [3, 0, 1]}]: dispatch 2024-02-05T06:42:32.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:32 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/674578421' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:42:32.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:32 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2396013368' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:42:32.523 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:32.555 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:32.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:32.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:32.657 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:32.692 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:32.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:32.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:32.698 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:42:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:42:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:42:32.793 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:32.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:32.831 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:32.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:32.933 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:32.963 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:32.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:32.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:33.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:33.101 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:33.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:33.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:33.114 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:42:32 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:42:33.203 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:33.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:33.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:33.346 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:33.376 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:33.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:33.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:33.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:33 smithi028 ceph-mon[37984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:42:33.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:33 smithi028 ceph-mon[37984]: from='client.25186 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.1b", "id": [3, 0, 1]}]': finished 2024-02-05T06:42:33.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:33 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/674578421' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:42:33.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:33 smithi028 ceph-mon[37984]: osdmap e273: 8 total, 8 up, 8 in 2024-02-05T06:42:33.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:33 smithi028 ceph-mon[37984]: pgmap v445: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.9 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 871 B/s rd, 0 B/s wr, 1.32k op/s; 1923/94083 objects misplaced (2.044%); 20 MiB/s, 317 objects/s recovering 2024-02-05T06:42:33.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:33 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:42:33.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:33 smithi028 ceph-mon[29442]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:42:33.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:33 smithi028 ceph-mon[29442]: from='client.25186 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.1b", "id": [3, 0, 1]}]': finished 2024-02-05T06:42:33.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:33 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/674578421' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:42:33.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:33 smithi028 ceph-mon[29442]: osdmap e273: 8 total, 8 up, 8 in 2024-02-05T06:42:33.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:33 smithi028 ceph-mon[29442]: pgmap v445: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.9 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 871 B/s rd, 0 B/s wr, 1.32k op/s; 1923/94083 objects misplaced (2.044%); 20 MiB/s, 317 objects/s recovering 2024-02-05T06:42:33.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:33 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:42:33.477 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:33.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:33 smithi167 ceph-mon[32774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:42:33.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:33 smithi167 ceph-mon[32774]: from='client.25186 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.1b", "id": [3, 0, 1]}]': finished 2024-02-05T06:42:33.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:33 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/674578421' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:42:33.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:33 smithi167 ceph-mon[32774]: osdmap e273: 8 total, 8 up, 8 in 2024-02-05T06:42:33.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:33 smithi167 ceph-mon[32774]: pgmap v445: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.9 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 871 B/s rd, 0 B/s wr, 1.32k op/s; 1923/94083 objects misplaced (2.044%); 20 MiB/s, 317 objects/s recovering 2024-02-05T06:42:33.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:33 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:42:33.510 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:33.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:33.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:33.612 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:33.645 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:33.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:33.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:33.747 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:33.786 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:33.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:33.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:33.888 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:33.922 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:33.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:33.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:34.023 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:34.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:34.057 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:34.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:34.158 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:34.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:34.195 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:34.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:34.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:34.310 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:34 smithi028 ceph-mon[29442]: 3.c scrub starts 2024-02-05T06:42:34.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:34 smithi028 ceph-mon[29442]: 3.c scrub ok 2024-02-05T06:42:34.311 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:34 smithi028 ceph-mon[29442]: osdmap e274: 8 total, 8 up, 8 in 2024-02-05T06:42:34.311 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:34 smithi028 ceph-mon[37984]: 3.c scrub starts 2024-02-05T06:42:34.311 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:34 smithi028 ceph-mon[37984]: 3.c scrub ok 2024-02-05T06:42:34.311 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:34 smithi028 ceph-mon[37984]: osdmap e274: 8 total, 8 up, 8 in 2024-02-05T06:42:34.336 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:34.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:34.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:34.438 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:34.471 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:34.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:34.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:34.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:34 smithi167 ceph-mon[32774]: 3.c scrub starts 2024-02-05T06:42:34.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:34 smithi167 ceph-mon[32774]: 3.c scrub ok 2024-02-05T06:42:34.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:34 smithi167 ceph-mon[32774]: osdmap e274: 8 total, 8 up, 8 in 2024-02-05T06:42:34.573 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:34.612 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:34.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:34.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:34.714 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:34.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:34.755 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:34.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:34.857 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:34.885 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:42:34.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:34.899 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:34.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:35.000 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:35.057 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:35.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:35.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:35.158 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:35.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:35.203 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:35.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:35.284 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:35 smithi028 ceph-mon[37984]: 3.16 scrub starts 2024-02-05T06:42:35.284 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:35 smithi028 ceph-mon[37984]: 3.16 scrub ok 2024-02-05T06:42:35.284 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:35 smithi028 ceph-mon[37984]: 8.2 scrub starts 2024-02-05T06:42:35.284 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:35 smithi028 ceph-mon[37984]: 8.2 scrub ok 2024-02-05T06:42:35.284 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:35 smithi028 ceph-mon[37984]: pgmap v447: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.9 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 873 B/s rd, 0 B/s wr, 988 op/s; 1923/94083 objects misplaced (2.044%) 2024-02-05T06:42:35.284 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:35 smithi028 ceph-mon[37984]: 5.10 scrub starts 2024-02-05T06:42:35.285 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:35 smithi028 ceph-mon[37984]: 5.10 scrub ok 2024-02-05T06:42:35.285 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:35 smithi028 ceph-mon[37984]: 8.3 scrub starts 2024-02-05T06:42:35.285 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:35 smithi028 ceph-mon[37984]: 6.1a scrub starts 2024-02-05T06:42:35.285 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:35 smithi028 ceph-mon[37984]: 6.1a scrub ok 2024-02-05T06:42:35.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:35 smithi028 ceph-mon[29442]: 3.16 scrub starts 2024-02-05T06:42:35.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:35 smithi028 ceph-mon[29442]: 3.16 scrub ok 2024-02-05T06:42:35.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:35 smithi028 ceph-mon[29442]: 8.2 scrub starts 2024-02-05T06:42:35.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:35 smithi028 ceph-mon[29442]: 8.2 scrub ok 2024-02-05T06:42:35.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:35 smithi028 ceph-mon[29442]: pgmap v447: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.9 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 873 B/s rd, 0 B/s wr, 988 op/s; 1923/94083 objects misplaced (2.044%) 2024-02-05T06:42:35.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:35 smithi028 ceph-mon[29442]: 5.10 scrub starts 2024-02-05T06:42:35.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:35 smithi028 ceph-mon[29442]: 5.10 scrub ok 2024-02-05T06:42:35.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:35 smithi028 ceph-mon[29442]: 8.3 scrub starts 2024-02-05T06:42:35.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:35 smithi028 ceph-mon[29442]: 6.1a scrub starts 2024-02-05T06:42:35.286 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:35 smithi028 ceph-mon[29442]: 6.1a scrub ok 2024-02-05T06:42:35.305 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:35.332 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:35.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:35.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:35.383 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:35 smithi167 ceph-mon[32774]: 3.16 scrub starts 2024-02-05T06:42:35.383 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:35 smithi167 ceph-mon[32774]: 3.16 scrub ok 2024-02-05T06:42:35.383 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:35 smithi167 ceph-mon[32774]: 8.2 scrub starts 2024-02-05T06:42:35.384 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:35 smithi167 ceph-mon[32774]: 8.2 scrub ok 2024-02-05T06:42:35.384 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:35 smithi167 ceph-mon[32774]: pgmap v447: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.9 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 873 B/s rd, 0 B/s wr, 988 op/s; 1923/94083 objects misplaced (2.044%) 2024-02-05T06:42:35.384 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:35 smithi167 ceph-mon[32774]: 5.10 scrub starts 2024-02-05T06:42:35.384 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:35 smithi167 ceph-mon[32774]: 5.10 scrub ok 2024-02-05T06:42:35.384 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:35 smithi167 ceph-mon[32774]: 8.3 scrub starts 2024-02-05T06:42:35.384 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:35 smithi167 ceph-mon[32774]: 6.1a scrub starts 2024-02-05T06:42:35.384 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:35 smithi167 ceph-mon[32774]: 6.1a scrub ok 2024-02-05T06:42:35.395 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:42:35.395 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:42:35.434 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:35.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:35.476 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:35.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:35.577 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:35.609 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:35.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:35.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:35.636 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:42:35.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:35.764 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:35.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:35.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:35.866 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:35.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:35.902 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:35.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:36.003 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:36.035 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:36.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:36.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:36.136 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:36.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:36.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:36.215 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-05T06:42:36.283 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:36.310 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:36.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:36.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:36.412 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:36.442 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:36.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:36.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:36.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:36 smithi028 ceph-mon[29442]: 8.3 scrub ok 2024-02-05T06:42:36.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:36 smithi028 ceph-mon[37984]: 8.3 scrub ok 2024-02-05T06:42:36.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:36 smithi167 ceph-mon[32774]: 8.3 scrub ok 2024-02-05T06:42:36.544 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:36.571 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:36.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:36.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:36.673 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:36.713 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:36.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:36.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:36.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T06:42:36.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:36.859 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:36.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:36.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:36.963 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:37.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:37.015 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:37.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:37.116 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:37.163 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:37.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:37.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:37.266 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:37.306 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:37.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:37.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:37.408 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:37.443 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:37.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:37.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:37.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:37 smithi028 ceph-mon[37984]: pgmap v448: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.8 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.10k op/s; 1923/90891 objects misplaced (2.116%) 2024-02-05T06:42:37.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:37 smithi028 ceph-mon[29442]: pgmap v448: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.8 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.10k op/s; 1923/90891 objects misplaced (2.116%) 2024-02-05T06:42:37.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:37 smithi167 ceph-mon[32774]: pgmap v448: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.8 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.10k op/s; 1923/90891 objects misplaced (2.116%) 2024-02-05T06:42:37.546 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:37.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:37.607 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:37.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:37.681 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:42:37.685 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:42:37.708 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:37.764 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:37.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:37.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:37.866 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:37.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:37.898 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:37.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:37.999 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:38.028 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:38.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:38.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:38.130 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:38.162 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:38.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:38.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:38.263 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:38.299 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:38.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:38.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:38.400 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:38.404 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:42:38.404 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:38 smithi028 ceph-mon[37984]: 6.11 scrub starts 2024-02-05T06:42:38.405 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:38 smithi028 ceph-mon[37984]: 6.11 scrub ok 2024-02-05T06:42:38.405 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:38 smithi028 ceph-mon[37984]: 1.1 scrub starts 2024-02-05T06:42:38.405 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:38 smithi028 ceph-mon[37984]: 1.1 scrub ok 2024-02-05T06:42:38.405 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:38 smithi028 ceph-mon[37984]: 5.1f scrub starts 2024-02-05T06:42:38.405 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:38 smithi028 ceph-mon[37984]: 5.1f scrub ok 2024-02-05T06:42:38.405 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:38 smithi028 ceph-mon[29442]: 6.11 scrub starts 2024-02-05T06:42:38.405 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:38 smithi028 ceph-mon[29442]: 6.11 scrub ok 2024-02-05T06:42:38.405 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:38 smithi028 ceph-mon[29442]: 1.1 scrub starts 2024-02-05T06:42:38.405 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:38 smithi028 ceph-mon[29442]: 1.1 scrub ok 2024-02-05T06:42:38.405 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:38 smithi028 ceph-mon[29442]: 5.1f scrub starts 2024-02-05T06:42:38.405 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:38 smithi028 ceph-mon[29442]: 5.1f scrub ok 2024-02-05T06:42:38.431 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:38.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:38.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:38.443 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:42:38.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:38 smithi167 ceph-mon[32774]: 6.11 scrub starts 2024-02-05T06:42:38.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:38 smithi167 ceph-mon[32774]: 6.11 scrub ok 2024-02-05T06:42:38.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:38 smithi167 ceph-mon[32774]: 1.1 scrub starts 2024-02-05T06:42:38.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:38 smithi167 ceph-mon[32774]: 1.1 scrub ok 2024-02-05T06:42:38.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:38 smithi167 ceph-mon[32774]: 5.1f scrub starts 2024-02-05T06:42:38.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:38 smithi167 ceph-mon[32774]: 5.1f scrub ok 2024-02-05T06:42:38.486 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:42:38.530 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:42:38.532 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:38.575 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:42:38.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:38.576 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:38.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:38.614 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:42:38.635 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:42:38.655 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:42:38.656 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:42:38.656 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:42:38.656 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:42:38.656 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:42:38.657 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:42:38.657 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:42:38.657 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:42:38.657 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:42:38.677 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:38.704 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:38.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:38.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:38.806 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:38.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:38.830 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:38.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:38.931 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:38.971 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:38.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:38.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:39.000 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 4, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T06:42:39.000 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd reweight 0 1 2024-02-05T06:42:39.074 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:39.116 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:39.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:39.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:39.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:39 smithi028 ceph-mon[29442]: 6.7 scrub starts 2024-02-05T06:42:39.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:39 smithi028 ceph-mon[29442]: 6.7 scrub ok 2024-02-05T06:42:39.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:39 smithi028 ceph-mon[29442]: 4.b scrub starts 2024-02-05T06:42:39.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:39 smithi028 ceph-mon[29442]: 4.b scrub ok 2024-02-05T06:42:39.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:39 smithi028 ceph-mon[29442]: pgmap v449: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.5 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 1573/76101 objects misplaced (2.067%) 2024-02-05T06:42:39.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:39 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/4090501887' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:42:39.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:39 smithi028 ceph-mon[29442]: 1.3 scrub starts 2024-02-05T06:42:39.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:39 smithi028 ceph-mon[29442]: 1.3 scrub ok 2024-02-05T06:42:39.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:39 smithi028 ceph-mon[37984]: 6.7 scrub starts 2024-02-05T06:42:39.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:39 smithi028 ceph-mon[37984]: 6.7 scrub ok 2024-02-05T06:42:39.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:39 smithi028 ceph-mon[37984]: 4.b scrub starts 2024-02-05T06:42:39.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:39 smithi028 ceph-mon[37984]: 4.b scrub ok 2024-02-05T06:42:39.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:39 smithi028 ceph-mon[37984]: pgmap v449: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.5 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 1573/76101 objects misplaced (2.067%) 2024-02-05T06:42:39.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:39 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/4090501887' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:42:39.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:39 smithi028 ceph-mon[37984]: 1.3 scrub starts 2024-02-05T06:42:39.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:39 smithi028 ceph-mon[37984]: 1.3 scrub ok 2024-02-05T06:42:39.220 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:39.262 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:39.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:39.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:39.364 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:39.396 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:39.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:39.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:39.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:39 smithi167 ceph-mon[32774]: 6.7 scrub starts 2024-02-05T06:42:39.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:39 smithi167 ceph-mon[32774]: 6.7 scrub ok 2024-02-05T06:42:39.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:39 smithi167 ceph-mon[32774]: 4.b scrub starts 2024-02-05T06:42:39.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:39 smithi167 ceph-mon[32774]: 4.b scrub ok 2024-02-05T06:42:39.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:39 smithi167 ceph-mon[32774]: pgmap v449: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.5 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 1573/76101 objects misplaced (2.067%) 2024-02-05T06:42:39.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:39 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/4090501887' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:42:39.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:39 smithi167 ceph-mon[32774]: 1.3 scrub starts 2024-02-05T06:42:39.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:39 smithi167 ceph-mon[32774]: 1.3 scrub ok 2024-02-05T06:42:39.497 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:39.543 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:39.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:39.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:39.644 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:39.687 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:39.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:39.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:39.789 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:39.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:39.826 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:39.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:39.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:39.973 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:39.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:39.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:40.075 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:40.107 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:40.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:40.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:40.209 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:40.243 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:40.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:40.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:40.344 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:40.383 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:40.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:40.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:40.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:40 smithi028 ceph-mon[29442]: 3.12 scrub starts 2024-02-05T06:42:40.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:40 smithi028 ceph-mon[29442]: 3.12 scrub ok 2024-02-05T06:42:40.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:40 smithi028 ceph-mon[29442]: 3.b scrub starts 2024-02-05T06:42:40.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:40 smithi028 ceph-mon[29442]: 3.b scrub ok 2024-02-05T06:42:40.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:40 smithi028 ceph-mon[29442]: 5.11 scrub starts 2024-02-05T06:42:40.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:40 smithi028 ceph-mon[29442]: 5.11 scrub ok 2024-02-05T06:42:40.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:40 smithi028 ceph-mon[29442]: 6.f scrub starts 2024-02-05T06:42:40.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:40 smithi028 ceph-mon[29442]: 6.f scrub ok 2024-02-05T06:42:40.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:40 smithi028 ceph-mon[37984]: 3.12 scrub starts 2024-02-05T06:42:40.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:40 smithi028 ceph-mon[37984]: 3.12 scrub ok 2024-02-05T06:42:40.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:40 smithi028 ceph-mon[37984]: 3.b scrub starts 2024-02-05T06:42:40.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:40 smithi028 ceph-mon[37984]: 3.b scrub ok 2024-02-05T06:42:40.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:40 smithi028 ceph-mon[37984]: 5.11 scrub starts 2024-02-05T06:42:40.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:40 smithi028 ceph-mon[37984]: 5.11 scrub ok 2024-02-05T06:42:40.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:40 smithi028 ceph-mon[37984]: 6.f scrub starts 2024-02-05T06:42:40.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:40 smithi028 ceph-mon[37984]: 6.f scrub ok 2024-02-05T06:42:40.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:40 smithi167 ceph-mon[32774]: 3.12 scrub starts 2024-02-05T06:42:40.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:40 smithi167 ceph-mon[32774]: 3.12 scrub ok 2024-02-05T06:42:40.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:40 smithi167 ceph-mon[32774]: 3.b scrub starts 2024-02-05T06:42:40.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:40 smithi167 ceph-mon[32774]: 3.b scrub ok 2024-02-05T06:42:40.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:40 smithi167 ceph-mon[32774]: 5.11 scrub starts 2024-02-05T06:42:40.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:40 smithi167 ceph-mon[32774]: 5.11 scrub ok 2024-02-05T06:42:40.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:40 smithi167 ceph-mon[32774]: 6.f scrub starts 2024-02-05T06:42:40.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:40 smithi167 ceph-mon[32774]: 6.f scrub ok 2024-02-05T06:42:40.485 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:40.521 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:40.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:40.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:40.623 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:40.661 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:40.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:40.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:40.763 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:40.799 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:40.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:40.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:40.901 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:40.936 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:40.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:40.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:41.037 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:41.079 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:41.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:41.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:41.181 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:41.216 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:41.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:41.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:41.318 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:41.362 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:41.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:41.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:41.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:41 smithi028 ceph-mon[37984]: 6.1b scrub starts 2024-02-05T06:42:41.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:41 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:42:41.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:41 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:42:41.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:41 smithi028 ceph-mon[37984]: 6.1b scrub ok 2024-02-05T06:42:41.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:41 smithi028 ceph-mon[37984]: 4.5 scrub starts 2024-02-05T06:42:41.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:41 smithi028 ceph-mon[37984]: 4.5 scrub ok 2024-02-05T06:42:41.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:41 smithi028 ceph-mon[37984]: pgmap v450: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.5 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 620 B/s rd, 0 B/s wr, 857 op/s; 1573/76101 objects misplaced (2.067%) 2024-02-05T06:42:41.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:41 smithi028 ceph-mon[29442]: 6.1b scrub starts 2024-02-05T06:42:41.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:41 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:42:41.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:41 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:42:41.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:41 smithi028 ceph-mon[29442]: 6.1b scrub ok 2024-02-05T06:42:41.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:41 smithi028 ceph-mon[29442]: 4.5 scrub starts 2024-02-05T06:42:41.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:41 smithi028 ceph-mon[29442]: 4.5 scrub ok 2024-02-05T06:42:41.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:41 smithi028 ceph-mon[29442]: pgmap v450: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.5 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 620 B/s rd, 0 B/s wr, 857 op/s; 1573/76101 objects misplaced (2.067%) 2024-02-05T06:42:41.465 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:41.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:41 smithi167 ceph-mon[32774]: 6.1b scrub starts 2024-02-05T06:42:41.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:41 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:42:41.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:41 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:42:41.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:41 smithi167 ceph-mon[32774]: 6.1b scrub ok 2024-02-05T06:42:41.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:41 smithi167 ceph-mon[32774]: 4.5 scrub starts 2024-02-05T06:42:41.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:41 smithi167 ceph-mon[32774]: 4.5 scrub ok 2024-02-05T06:42:41.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:41 smithi167 ceph-mon[32774]: pgmap v450: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.5 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 620 B/s rd, 0 B/s wr, 857 op/s; 1573/76101 objects misplaced (2.067%) 2024-02-05T06:42:41.513 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:41.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:41.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:41.610 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:42:41.614 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:41.645 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:41.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:41.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:41.747 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:41.781 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:41.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:41.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:41.883 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:41.918 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:41.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:41.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:42.020 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:42.063 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:42.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:42.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:42.165 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:42.199 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:42.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:42.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:42.301 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:42.337 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:42.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:42.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:42.439 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:42.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:42 smithi028 ceph-mon[29442]: osdmap e275: 8 total, 8 up, 8 in 2024-02-05T06:42:42.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:42 smithi028 ceph-mon[29442]: 1.6 scrub starts 2024-02-05T06:42:42.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:42 smithi028 ceph-mon[29442]: 1.6 scrub ok 2024-02-05T06:42:42.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:42 smithi028 ceph-mon[29442]: 3.13 scrub starts 2024-02-05T06:42:42.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:42 smithi028 ceph-mon[29442]: 3.13 scrub ok 2024-02-05T06:42:42.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:42 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:42:42.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:42 smithi028 ceph-mon[29442]: 6.1 scrub starts 2024-02-05T06:42:42.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:42 smithi028 ceph-mon[29442]: 6.1 scrub ok 2024-02-05T06:42:42.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:42 smithi028 ceph-mon[37984]: osdmap e275: 8 total, 8 up, 8 in 2024-02-05T06:42:42.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:42 smithi028 ceph-mon[37984]: 1.6 scrub starts 2024-02-05T06:42:42.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:42 smithi028 ceph-mon[37984]: 1.6 scrub ok 2024-02-05T06:42:42.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:42 smithi028 ceph-mon[37984]: 3.13 scrub starts 2024-02-05T06:42:42.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:42 smithi028 ceph-mon[37984]: 3.13 scrub ok 2024-02-05T06:42:42.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:42 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:42:42.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:42 smithi028 ceph-mon[37984]: 6.1 scrub starts 2024-02-05T06:42:42.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:42 smithi028 ceph-mon[37984]: 6.1 scrub ok 2024-02-05T06:42:42.466 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:42.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:42.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:42.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:42 smithi167 ceph-mon[32774]: osdmap e275: 8 total, 8 up, 8 in 2024-02-05T06:42:42.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:42 smithi167 ceph-mon[32774]: 1.6 scrub starts 2024-02-05T06:42:42.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:42 smithi167 ceph-mon[32774]: 1.6 scrub ok 2024-02-05T06:42:42.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:42 smithi167 ceph-mon[32774]: 3.13 scrub starts 2024-02-05T06:42:42.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:42 smithi167 ceph-mon[32774]: 3.13 scrub ok 2024-02-05T06:42:42.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:42 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:42:42.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:42 smithi167 ceph-mon[32774]: 6.1 scrub starts 2024-02-05T06:42:42.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:42 smithi167 ceph-mon[32774]: 6.1 scrub ok 2024-02-05T06:42:42.568 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:42.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:42.607 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:42.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:42.708 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:42.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:42.749 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:42.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:42.850 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:42.886 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:42.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:42.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:42.948 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:42:42 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:42:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:42:42.965 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:42:42.989 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:43.064 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:43.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:43.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:43.166 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:43.198 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:42:42 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:42:43.227 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:43.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:43.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:43.280 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:43 smithi028 ceph-mon[29442]: 6.9 scrub starts 2024-02-05T06:42:43.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:43 smithi028 ceph-mon[29442]: 6.9 scrub ok 2024-02-05T06:42:43.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:43 smithi028 ceph-mon[29442]: pgmap v452: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.2 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.31k op/s; 1179/57852 objects misplaced (2.038%) 2024-02-05T06:42:43.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:43 smithi028 ceph-mon[29442]: 3.3 scrub starts 2024-02-05T06:42:43.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:43 smithi028 ceph-mon[29442]: 3.3 scrub ok 2024-02-05T06:42:43.281 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:43 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:42:43.283 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:43 smithi028 ceph-mon[37984]: 6.9 scrub starts 2024-02-05T06:42:43.283 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:43 smithi028 ceph-mon[37984]: 6.9 scrub ok 2024-02-05T06:42:43.283 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:43 smithi028 ceph-mon[37984]: pgmap v452: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.2 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.31k op/s; 1179/57852 objects misplaced (2.038%) 2024-02-05T06:42:43.283 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:43 smithi028 ceph-mon[37984]: 3.3 scrub starts 2024-02-05T06:42:43.283 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:43 smithi028 ceph-mon[37984]: 3.3 scrub ok 2024-02-05T06:42:43.283 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:43 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:42:43.329 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:43.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:43.365 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:43.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:43.466 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:43.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:43 smithi167 ceph-mon[32774]: 6.9 scrub starts 2024-02-05T06:42:43.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:43 smithi167 ceph-mon[32774]: 6.9 scrub ok 2024-02-05T06:42:43.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:43 smithi167 ceph-mon[32774]: pgmap v452: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.2 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.31k op/s; 1179/57852 objects misplaced (2.038%) 2024-02-05T06:42:43.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:43 smithi167 ceph-mon[32774]: 3.3 scrub starts 2024-02-05T06:42:43.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:43 smithi167 ceph-mon[32774]: 3.3 scrub ok 2024-02-05T06:42:43.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:43 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:42:43.493 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:43.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:43.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:43.595 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:43.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:43.629 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:43.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:43.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:43.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:43.771 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:43.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:43.873 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:43.917 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:43.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:43.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:44.019 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:44.034 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:42:44.034 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-02-05T06:42:44.034 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-02-05T06:42:44.034 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-02-05T06:42:44.034 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:42:44.063 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:44.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:44.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:44.165 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:44.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:44.202 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:44.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:44.303 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:44.336 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:44.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:44.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:44.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:44.478 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:44.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:44.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:44.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:44 smithi167 ceph-mon[32774]: 6.d scrub starts 2024-02-05T06:42:44.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:44 smithi167 ceph-mon[32774]: 6.d scrub ok 2024-02-05T06:42:44.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:44 smithi167 ceph-mon[32774]: 5.5 scrub starts 2024-02-05T06:42:44.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:44 smithi167 ceph-mon[32774]: 5.5 scrub ok 2024-02-05T06:42:44.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:44 smithi167 ceph-mon[32774]: 3.e scrub starts 2024-02-05T06:42:44.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:44 smithi167 ceph-mon[32774]: 3.e scrub ok 2024-02-05T06:42:44.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:44 smithi167 ceph-mon[32774]: 6.e scrub starts 2024-02-05T06:42:44.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:44 smithi167 ceph-mon[32774]: 6.e scrub ok 2024-02-05T06:42:44.579 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:44.612 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:44.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:44.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:44.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:44 smithi028 ceph-mon[29442]: 6.d scrub starts 2024-02-05T06:42:44.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:44 smithi028 ceph-mon[29442]: 6.d scrub ok 2024-02-05T06:42:44.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:44 smithi028 ceph-mon[29442]: 5.5 scrub starts 2024-02-05T06:42:44.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:44 smithi028 ceph-mon[29442]: 5.5 scrub ok 2024-02-05T06:42:44.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:44 smithi028 ceph-mon[29442]: 3.e scrub starts 2024-02-05T06:42:44.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:44 smithi028 ceph-mon[29442]: 3.e scrub ok 2024-02-05T06:42:44.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:44 smithi028 ceph-mon[29442]: 6.e scrub starts 2024-02-05T06:42:44.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:44 smithi028 ceph-mon[29442]: 6.e scrub ok 2024-02-05T06:42:44.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:44 smithi028 ceph-mon[37984]: 6.d scrub starts 2024-02-05T06:42:44.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:44 smithi028 ceph-mon[37984]: 6.d scrub ok 2024-02-05T06:42:44.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:44 smithi028 ceph-mon[37984]: 5.5 scrub starts 2024-02-05T06:42:44.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:44 smithi028 ceph-mon[37984]: 5.5 scrub ok 2024-02-05T06:42:44.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:44 smithi028 ceph-mon[37984]: 3.e scrub starts 2024-02-05T06:42:44.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:44 smithi028 ceph-mon[37984]: 3.e scrub ok 2024-02-05T06:42:44.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:44 smithi028 ceph-mon[37984]: 6.e scrub starts 2024-02-05T06:42:44.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:44 smithi028 ceph-mon[37984]: 6.e scrub ok 2024-02-05T06:42:44.715 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:44.753 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:44.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:44.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:44.855 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:44.889 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:44.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:44.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:44.991 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:45.030 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:45.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:45.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:45.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:45.167 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:45.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:45.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:45.269 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:45.292 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:45.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:45.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:45.394 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:45.437 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:45.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:45.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:45.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:45 smithi167 ceph-mon[32774]: 4.1d scrub starts 2024-02-05T06:42:45.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:45 smithi167 ceph-mon[32774]: 4.1d scrub ok 2024-02-05T06:42:45.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:45 smithi167 ceph-mon[32774]: pgmap v453: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.2 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.21k op/s; 1179/57852 objects misplaced (2.038%) 2024-02-05T06:42:45.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:45 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:42:45.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:45 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:42:45.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:45 smithi167 ceph-mon[32774]: osdmap e276: 8 total, 8 up, 8 in 2024-02-05T06:42:45.581 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:45.605 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:45.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:45.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:45.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:45 smithi028 ceph-mon[29442]: 4.1d scrub starts 2024-02-05T06:42:45.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:45 smithi028 ceph-mon[29442]: 4.1d scrub ok 2024-02-05T06:42:45.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:45 smithi028 ceph-mon[29442]: pgmap v453: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.2 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.21k op/s; 1179/57852 objects misplaced (2.038%) 2024-02-05T06:42:45.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:45 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:42:45.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:45 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:42:45.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:45 smithi028 ceph-mon[29442]: osdmap e276: 8 total, 8 up, 8 in 2024-02-05T06:42:45.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:45 smithi028 ceph-mon[37984]: 4.1d scrub starts 2024-02-05T06:42:45.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:45 smithi028 ceph-mon[37984]: 4.1d scrub ok 2024-02-05T06:42:45.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:45 smithi028 ceph-mon[37984]: pgmap v453: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.2 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 1.21k op/s; 1179/57852 objects misplaced (2.038%) 2024-02-05T06:42:45.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:45 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:42:45.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:45 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:42:45.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:45 smithi028 ceph-mon[37984]: osdmap e276: 8 total, 8 up, 8 in 2024-02-05T06:42:45.706 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:45.744 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:45.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:45.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:45.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:45.887 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:45.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:45.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:45.988 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:46.012 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:46.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:46.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:46.114 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:46.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:46.146 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:46.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:46.247 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:46.271 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:46.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:46.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:46.373 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:46.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:46.410 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:46.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:46.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:46 smithi167 ceph-mon[32774]: 4.12 scrub starts 2024-02-05T06:42:46.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:46 smithi167 ceph-mon[32774]: 4.12 scrub ok 2024-02-05T06:42:46.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:46 smithi167 ceph-mon[32774]: 5.1c scrub starts 2024-02-05T06:42:46.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:46 smithi167 ceph-mon[32774]: 5.1c scrub ok 2024-02-05T06:42:46.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:46 smithi167 ceph-mon[32774]: 6.2 scrub starts 2024-02-05T06:42:46.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:46 smithi167 ceph-mon[32774]: 6.2 scrub ok 2024-02-05T06:42:46.511 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:46.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:46.545 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:46.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:46.646 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:46.687 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:46.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:46.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:46.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:46 smithi028 ceph-mon[29442]: 4.12 scrub starts 2024-02-05T06:42:46.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:46 smithi028 ceph-mon[29442]: 4.12 scrub ok 2024-02-05T06:42:46.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:46 smithi028 ceph-mon[29442]: 5.1c scrub starts 2024-02-05T06:42:46.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:46 smithi028 ceph-mon[29442]: 5.1c scrub ok 2024-02-05T06:42:46.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:46 smithi028 ceph-mon[29442]: 6.2 scrub starts 2024-02-05T06:42:46.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:46 smithi028 ceph-mon[29442]: 6.2 scrub ok 2024-02-05T06:42:46.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:46 smithi028 ceph-mon[37984]: 4.12 scrub starts 2024-02-05T06:42:46.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:46 smithi028 ceph-mon[37984]: 4.12 scrub ok 2024-02-05T06:42:46.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:46 smithi028 ceph-mon[37984]: 5.1c scrub starts 2024-02-05T06:42:46.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:46 smithi028 ceph-mon[37984]: 5.1c scrub ok 2024-02-05T06:42:46.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:46 smithi028 ceph-mon[37984]: 6.2 scrub starts 2024-02-05T06:42:46.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:46 smithi028 ceph-mon[37984]: 6.2 scrub ok 2024-02-05T06:42:46.789 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:46.826 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:46.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:46.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:46.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:46.971 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:46.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:46.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:47.073 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:47.115 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:47.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:47.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:47.217 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:47.257 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:47.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:47.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:47.359 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:47.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:47.396 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:47.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:47.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:47 smithi167 ceph-mon[32774]: 8.5 scrub starts 2024-02-05T06:42:47.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:47 smithi167 ceph-mon[32774]: 8.5 scrub ok 2024-02-05T06:42:47.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:47 smithi167 ceph-mon[32774]: pgmap v455: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.1 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 896 op/s; 1179/54522 objects misplaced (2.162%) 2024-02-05T06:42:47.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:47 smithi167 ceph-mon[32774]: 5.1b scrub starts 2024-02-05T06:42:47.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:47 smithi167 ceph-mon[32774]: 5.1b scrub ok 2024-02-05T06:42:47.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:47 smithi167 ceph-mon[32774]: 3.11 scrub starts 2024-02-05T06:42:47.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:47 smithi167 ceph-mon[32774]: 3.11 scrub ok 2024-02-05T06:42:47.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:47 smithi167 ceph-mon[32774]: 5.c scrub starts 2024-02-05T06:42:47.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:47 smithi167 ceph-mon[32774]: 5.c scrub ok 2024-02-05T06:42:47.497 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:47.533 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:47.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:47.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:47.635 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:47.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:47 smithi028 ceph-mon[29442]: 8.5 scrub starts 2024-02-05T06:42:47.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:47 smithi028 ceph-mon[29442]: 8.5 scrub ok 2024-02-05T06:42:47.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:47 smithi028 ceph-mon[29442]: pgmap v455: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.1 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 896 op/s; 1179/54522 objects misplaced (2.162%) 2024-02-05T06:42:47.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:47 smithi028 ceph-mon[29442]: 5.1b scrub starts 2024-02-05T06:42:47.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:47 smithi028 ceph-mon[29442]: 5.1b scrub ok 2024-02-05T06:42:47.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:47 smithi028 ceph-mon[29442]: 3.11 scrub starts 2024-02-05T06:42:47.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:47 smithi028 ceph-mon[29442]: 3.11 scrub ok 2024-02-05T06:42:47.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:47 smithi028 ceph-mon[29442]: 5.c scrub starts 2024-02-05T06:42:47.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:47 smithi028 ceph-mon[29442]: 5.c scrub ok 2024-02-05T06:42:47.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:47 smithi028 ceph-mon[37984]: 8.5 scrub starts 2024-02-05T06:42:47.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:47 smithi028 ceph-mon[37984]: 8.5 scrub ok 2024-02-05T06:42:47.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:47 smithi028 ceph-mon[37984]: pgmap v455: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.1 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 0 B/s wr, 896 op/s; 1179/54522 objects misplaced (2.162%) 2024-02-05T06:42:47.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:47 smithi028 ceph-mon[37984]: 5.1b scrub starts 2024-02-05T06:42:47.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:47 smithi028 ceph-mon[37984]: 5.1b scrub ok 2024-02-05T06:42:47.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:47 smithi028 ceph-mon[37984]: 3.11 scrub starts 2024-02-05T06:42:47.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:47 smithi028 ceph-mon[37984]: 3.11 scrub ok 2024-02-05T06:42:47.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:47 smithi028 ceph-mon[37984]: 5.c scrub starts 2024-02-05T06:42:47.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:47 smithi028 ceph-mon[37984]: 5.c scrub ok 2024-02-05T06:42:47.667 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:47.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:47.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:47.769 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:47.801 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:47.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:47.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:47.902 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:47.938 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:47.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:47.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:48.040 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:48.076 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:48.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:48.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:48.178 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:48.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:48.215 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:48.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:48.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:48 smithi028 ceph-mon[29442]: 4.16 scrub starts 2024-02-05T06:42:48.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:48 smithi028 ceph-mon[29442]: 4.16 scrub ok 2024-02-05T06:42:48.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:48 smithi028 ceph-mon[29442]: 8.9 scrub starts 2024-02-05T06:42:48.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:48 smithi028 ceph-mon[29442]: 4.14 scrub starts 2024-02-05T06:42:48.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:48 smithi028 ceph-mon[29442]: 4.14 scrub ok 2024-02-05T06:42:48.282 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:48 smithi028 ceph-mon[29442]: 8.9 scrub ok 2024-02-05T06:42:48.282 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:48 smithi028 ceph-mon[37984]: 4.16 scrub starts 2024-02-05T06:42:48.282 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:48 smithi028 ceph-mon[37984]: 4.16 scrub ok 2024-02-05T06:42:48.282 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:48 smithi028 ceph-mon[37984]: 8.9 scrub starts 2024-02-05T06:42:48.283 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:48 smithi028 ceph-mon[37984]: 4.14 scrub starts 2024-02-05T06:42:48.283 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:48 smithi028 ceph-mon[37984]: 4.14 scrub ok 2024-02-05T06:42:48.283 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:48 smithi028 ceph-mon[37984]: 8.9 scrub ok 2024-02-05T06:42:48.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:48.363 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:48.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:48.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:48.465 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:48.482 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:48 smithi167 ceph-mon[32774]: 4.16 scrub starts 2024-02-05T06:42:48.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:48 smithi167 ceph-mon[32774]: 4.16 scrub ok 2024-02-05T06:42:48.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:48 smithi167 ceph-mon[32774]: 8.9 scrub starts 2024-02-05T06:42:48.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:48 smithi167 ceph-mon[32774]: 4.14 scrub starts 2024-02-05T06:42:48.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:48 smithi167 ceph-mon[32774]: 4.14 scrub ok 2024-02-05T06:42:48.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:48 smithi167 ceph-mon[32774]: 8.9 scrub ok 2024-02-05T06:42:48.508 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:48.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:48.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:48.610 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:48.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:48.655 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:48.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:48.756 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:48.783 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:48.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:48.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:48.885 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:48.914 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:48.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:48.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:49.016 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:49.047 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:49.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:49.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:49.149 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:49.182 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:49.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:49.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:49.263 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-02-05T06:42:49.283 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:49.312 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:49.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:49.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:49.382 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-05T06:42:49.414 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:49.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:49.448 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:49.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:49.549 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:49.577 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:49.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:49.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:49.632 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:49 smithi028 ceph-mon[37984]: 5.3 scrub starts 2024-02-05T06:42:49.632 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:49 smithi028 ceph-mon[37984]: 5.3 scrub ok 2024-02-05T06:42:49.632 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:49 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:42:49.632 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:49 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:42:49.632 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:49 smithi028 ceph-mon[37984]: 3.10 scrub starts 2024-02-05T06:42:49.632 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:49 smithi028 ceph-mon[37984]: 3.10 scrub ok 2024-02-05T06:42:49.633 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:49 smithi028 ceph-mon[37984]: osdmap e277: 8 total, 8 up, 8 in 2024-02-05T06:42:49.633 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:49 smithi028 ceph-mon[37984]: pgmap v457: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 838 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.64k op/s; 610/40782 objects misplaced (1.496%); 2.0 MiB/s, 31 objects/s recovering 2024-02-05T06:42:49.633 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:49 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1840859249' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:42:49.633 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:49 smithi028 ceph-mon[37984]: from='client.25207 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:42:49.633 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:49 smithi028 ceph-mon[29442]: 5.3 scrub starts 2024-02-05T06:42:49.633 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:49 smithi028 ceph-mon[29442]: 5.3 scrub ok 2024-02-05T06:42:49.633 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:49 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:42:49.633 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:49 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:42:49.633 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:49 smithi028 ceph-mon[29442]: 3.10 scrub starts 2024-02-05T06:42:49.633 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:49 smithi028 ceph-mon[29442]: 3.10 scrub ok 2024-02-05T06:42:49.633 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:49 smithi028 ceph-mon[29442]: osdmap e277: 8 total, 8 up, 8 in 2024-02-05T06:42:49.633 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:49 smithi028 ceph-mon[29442]: pgmap v457: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 838 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.64k op/s; 610/40782 objects misplaced (1.496%); 2.0 MiB/s, 31 objects/s recovering 2024-02-05T06:42:49.634 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:49 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1840859249' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:42:49.634 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:49 smithi028 ceph-mon[29442]: from='client.25207 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:42:49.679 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:49.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:49 smithi167 ceph-mon[32774]: 5.3 scrub starts 2024-02-05T06:42:49.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:49 smithi167 ceph-mon[32774]: 5.3 scrub ok 2024-02-05T06:42:49.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:49 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:42:49.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:49 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:42:49.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:49 smithi167 ceph-mon[32774]: 3.10 scrub starts 2024-02-05T06:42:49.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:49 smithi167 ceph-mon[32774]: 3.10 scrub ok 2024-02-05T06:42:49.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:49 smithi167 ceph-mon[32774]: osdmap e277: 8 total, 8 up, 8 in 2024-02-05T06:42:49.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:49 smithi167 ceph-mon[32774]: pgmap v457: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 838 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.64k op/s; 610/40782 objects misplaced (1.496%); 2.0 MiB/s, 31 objects/s recovering 2024-02-05T06:42:49.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:49 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1840859249' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:42:49.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:49 smithi167 ceph-mon[32774]: from='client.25207 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:42:49.735 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:49.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:49.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:49.837 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:49.868 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:49.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:49.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:49.970 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:50.009 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:50.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:50.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:50.111 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:50.146 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:50.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:50.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:50.247 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:50.256 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:42:50.279 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:50.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:50.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:50.299 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:42:50.380 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:50.406 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:50.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:50.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:50.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:50 smithi028 ceph-mon[29442]: 1.7 scrub starts 2024-02-05T06:42:50.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:50 smithi028 ceph-mon[29442]: 1.7 scrub ok 2024-02-05T06:42:50.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:50 smithi028 ceph-mon[29442]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-05T06:42:50.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:50 smithi028 ceph-mon[29442]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:42:50.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:50 smithi028 ceph-mon[29442]: from='client.25207 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:42:50.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:50 smithi028 ceph-mon[29442]: osdmap e278: 8 total, 8 up, 8 in 2024-02-05T06:42:50.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:50 smithi028 ceph-mon[29442]: 6.15 scrub starts 2024-02-05T06:42:50.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:50 smithi028 ceph-mon[29442]: 6.15 scrub ok 2024-02-05T06:42:50.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:50 smithi028 ceph-mon[29442]: 1.4 scrub starts 2024-02-05T06:42:50.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:50 smithi028 ceph-mon[29442]: 1.4 scrub ok 2024-02-05T06:42:50.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:50 smithi028 ceph-mon[29442]: osdmap e279: 8 total, 8 up, 8 in 2024-02-05T06:42:50.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:50 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/9783606' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T06:42:50.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:50 smithi028 ceph-mon[37984]: 1.7 scrub starts 2024-02-05T06:42:50.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:50 smithi028 ceph-mon[37984]: 1.7 scrub ok 2024-02-05T06:42:50.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:50 smithi028 ceph-mon[37984]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-05T06:42:50.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:50 smithi028 ceph-mon[37984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:42:50.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:50 smithi028 ceph-mon[37984]: from='client.25207 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:42:50.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:50 smithi028 ceph-mon[37984]: osdmap e278: 8 total, 8 up, 8 in 2024-02-05T06:42:50.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:50 smithi028 ceph-mon[37984]: 6.15 scrub starts 2024-02-05T06:42:50.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:50 smithi028 ceph-mon[37984]: 6.15 scrub ok 2024-02-05T06:42:50.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:50 smithi028 ceph-mon[37984]: 1.4 scrub starts 2024-02-05T06:42:50.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:50 smithi028 ceph-mon[37984]: 1.4 scrub ok 2024-02-05T06:42:50.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:50 smithi028 ceph-mon[37984]: osdmap e279: 8 total, 8 up, 8 in 2024-02-05T06:42:50.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:50 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/9783606' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T06:42:50.508 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:50.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:50.533 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:50.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:50.634 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:50.657 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:50.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:50.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:50.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:50 smithi167 ceph-mon[32774]: 1.7 scrub starts 2024-02-05T06:42:50.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:50 smithi167 ceph-mon[32774]: 1.7 scrub ok 2024-02-05T06:42:50.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:50 smithi167 ceph-mon[32774]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-05T06:42:50.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:50 smithi167 ceph-mon[32774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:42:50.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:50 smithi167 ceph-mon[32774]: from='client.25207 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:42:50.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:50 smithi167 ceph-mon[32774]: osdmap e278: 8 total, 8 up, 8 in 2024-02-05T06:42:50.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:50 smithi167 ceph-mon[32774]: 6.15 scrub starts 2024-02-05T06:42:50.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:50 smithi167 ceph-mon[32774]: 6.15 scrub ok 2024-02-05T06:42:50.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:50 smithi167 ceph-mon[32774]: 1.4 scrub starts 2024-02-05T06:42:50.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:50 smithi167 ceph-mon[32774]: 1.4 scrub ok 2024-02-05T06:42:50.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:50 smithi167 ceph-mon[32774]: osdmap e279: 8 total, 8 up, 8 in 2024-02-05T06:42:50.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:50 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/9783606' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T06:42:50.759 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:50.782 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:50.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:50.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:50.888 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:50.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:50.912 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:50.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:51.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:51.037 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:51.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:51.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:51.078 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.0 to 1 (10000) 2024-02-05T06:42:51.139 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:51.162 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:51.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:51.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:51.264 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:51.287 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:51.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:51.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:51.389 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:51.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:51.414 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:51.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:51.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:51 smithi028 ceph-mon[37984]: 3.1 scrub starts 2024-02-05T06:42:51.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:51 smithi028 ceph-mon[37984]: 3.1 scrub ok 2024-02-05T06:42:51.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:51 smithi028 ceph-mon[37984]: 4.a deep-scrub starts 2024-02-05T06:42:51.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:51 smithi028 ceph-mon[37984]: 4.a deep-scrub ok 2024-02-05T06:42:51.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:51 smithi028 ceph-mon[37984]: pgmap v460: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 838 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 972 B/s rd, 0 B/s wr, 1.08k op/s; 610/40782 objects misplaced (1.496%); 2.7 MiB/s, 43 objects/s recovering 2024-02-05T06:42:51.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:51 smithi028 ceph-mon[37984]: 5.b scrub starts 2024-02-05T06:42:51.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:51 smithi028 ceph-mon[37984]: 5.b scrub ok 2024-02-05T06:42:51.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:51 smithi028 ceph-mon[37984]: 3.2 scrub starts 2024-02-05T06:42:51.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:51 smithi028 ceph-mon[37984]: 5.9 scrub starts 2024-02-05T06:42:51.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:51 smithi028 ceph-mon[37984]: 3.2 scrub ok 2024-02-05T06:42:51.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:51 smithi028 ceph-mon[37984]: 5.9 scrub ok 2024-02-05T06:42:51.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:51 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/9783606' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T06:42:51.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:51 smithi028 ceph-mon[37984]: osdmap e280: 8 total, 8 up, 8 in 2024-02-05T06:42:51.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:51 smithi028 ceph-mon[29442]: 3.1 scrub starts 2024-02-05T06:42:51.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:51 smithi028 ceph-mon[29442]: 3.1 scrub ok 2024-02-05T06:42:51.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:51 smithi028 ceph-mon[29442]: 4.a deep-scrub starts 2024-02-05T06:42:51.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:51 smithi028 ceph-mon[29442]: 4.a deep-scrub ok 2024-02-05T06:42:51.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:51 smithi028 ceph-mon[29442]: pgmap v460: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 838 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 972 B/s rd, 0 B/s wr, 1.08k op/s; 610/40782 objects misplaced (1.496%); 2.7 MiB/s, 43 objects/s recovering 2024-02-05T06:42:51.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:51 smithi028 ceph-mon[29442]: 5.b scrub starts 2024-02-05T06:42:51.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:51 smithi028 ceph-mon[29442]: 5.b scrub ok 2024-02-05T06:42:51.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:51 smithi028 ceph-mon[29442]: 3.2 scrub starts 2024-02-05T06:42:51.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:51 smithi028 ceph-mon[29442]: 5.9 scrub starts 2024-02-05T06:42:51.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:51 smithi028 ceph-mon[29442]: 3.2 scrub ok 2024-02-05T06:42:51.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:51 smithi028 ceph-mon[29442]: 5.9 scrub ok 2024-02-05T06:42:51.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:51 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/9783606' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T06:42:51.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:51 smithi028 ceph-mon[29442]: osdmap e280: 8 total, 8 up, 8 in 2024-02-05T06:42:51.516 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:51.541 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:51.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:51.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:51.643 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:51.677 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:51.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:51.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:51.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:51 smithi167 ceph-mon[32774]: 3.1 scrub starts 2024-02-05T06:42:51.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:51 smithi167 ceph-mon[32774]: 3.1 scrub ok 2024-02-05T06:42:51.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:51 smithi167 ceph-mon[32774]: 4.a deep-scrub starts 2024-02-05T06:42:51.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:51 smithi167 ceph-mon[32774]: 4.a deep-scrub ok 2024-02-05T06:42:51.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:51 smithi167 ceph-mon[32774]: pgmap v460: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 838 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 972 B/s rd, 0 B/s wr, 1.08k op/s; 610/40782 objects misplaced (1.496%); 2.7 MiB/s, 43 objects/s recovering 2024-02-05T06:42:51.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:51 smithi167 ceph-mon[32774]: 5.b scrub starts 2024-02-05T06:42:51.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:51 smithi167 ceph-mon[32774]: 5.b scrub ok 2024-02-05T06:42:51.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:51 smithi167 ceph-mon[32774]: 3.2 scrub starts 2024-02-05T06:42:51.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:51 smithi167 ceph-mon[32774]: 5.9 scrub starts 2024-02-05T06:42:51.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:51 smithi167 ceph-mon[32774]: 3.2 scrub ok 2024-02-05T06:42:51.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:51 smithi167 ceph-mon[32774]: 5.9 scrub ok 2024-02-05T06:42:51.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:51 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/9783606' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T06:42:51.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:51 smithi167 ceph-mon[32774]: osdmap e280: 8 total, 8 up, 8 in 2024-02-05T06:42:51.779 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:51.814 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:51.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:51.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:51.919 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:51.959 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:51.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:51.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:52.061 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:52.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:52.086 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:52.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:52.187 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:52.220 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T06:42:52.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:52.224 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:52.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:52.325 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:52.366 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:52.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:52.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:52.468 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:52.501 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:52.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:52.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:52.602 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:52.645 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:52.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:52.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:52.698 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:42:52 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:42:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:42:52.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:52 smithi028 ceph-mon[29442]: 5.15 scrub starts 2024-02-05T06:42:52.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:52 smithi028 ceph-mon[29442]: 5.15 scrub ok 2024-02-05T06:42:52.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:52 smithi028 ceph-mon[29442]: 4.6 scrub starts 2024-02-05T06:42:52.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:52 smithi028 ceph-mon[29442]: 4.6 scrub ok 2024-02-05T06:42:52.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:52 smithi028 ceph-mon[29442]: 4.1c scrub starts 2024-02-05T06:42:52.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:52 smithi028 ceph-mon[29442]: 4.1c scrub ok 2024-02-05T06:42:52.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:52 smithi028 ceph-mon[37984]: 5.15 scrub starts 2024-02-05T06:42:52.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:52 smithi028 ceph-mon[37984]: 5.15 scrub ok 2024-02-05T06:42:52.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:52 smithi028 ceph-mon[37984]: 4.6 scrub starts 2024-02-05T06:42:52.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:52 smithi028 ceph-mon[37984]: 4.6 scrub ok 2024-02-05T06:42:52.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:52 smithi028 ceph-mon[37984]: 4.1c scrub starts 2024-02-05T06:42:52.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:52 smithi028 ceph-mon[37984]: 4.1c scrub ok 2024-02-05T06:42:52.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:52 smithi167 ceph-mon[32774]: 5.15 scrub starts 2024-02-05T06:42:52.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:52 smithi167 ceph-mon[32774]: 5.15 scrub ok 2024-02-05T06:42:52.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:52 smithi167 ceph-mon[32774]: 4.6 scrub starts 2024-02-05T06:42:52.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:52 smithi167 ceph-mon[32774]: 4.6 scrub ok 2024-02-05T06:42:52.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:52 smithi167 ceph-mon[32774]: 4.1c scrub starts 2024-02-05T06:42:52.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:52 smithi167 ceph-mon[32774]: 4.1c scrub ok 2024-02-05T06:42:52.747 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:52.771 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:52.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:52.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:52.873 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:52.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:52.914 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:52.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:53.015 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:53.048 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:53.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:53.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:53.150 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:53.173 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:53.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:53.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:53.232 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:42:52 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:42:53.275 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:53.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:53.303 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:53.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:53.404 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:53.441 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:53.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:53.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:53.543 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:53.574 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:53.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:53.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:53.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:53.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:53 smithi028 ceph-mon[37984]: pgmap v462: 158 pgs: 1 activating+degraded, 157 active+clean; 569 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 2/27870 objects degraded (0.007%); 7.9 MiB/s, 126 objects/s recovering 2024-02-05T06:42:53.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:53 smithi028 ceph-mon[37984]: 5.6 scrub starts 2024-02-05T06:42:53.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:53 smithi028 ceph-mon[37984]: 5.6 scrub ok 2024-02-05T06:42:53.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:53 smithi028 ceph-mon[37984]: 1.0 scrub starts 2024-02-05T06:42:53.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:53 smithi028 ceph-mon[37984]: 1.0 scrub ok 2024-02-05T06:42:53.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:53 smithi028 ceph-mon[37984]: 5.1 scrub starts 2024-02-05T06:42:53.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:53 smithi028 ceph-mon[37984]: 5.1 scrub ok 2024-02-05T06:42:53.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:53 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:42:53.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:53 smithi028 ceph-mon[29442]: pgmap v462: 158 pgs: 1 activating+degraded, 157 active+clean; 569 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 2/27870 objects degraded (0.007%); 7.9 MiB/s, 126 objects/s recovering 2024-02-05T06:42:53.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:53 smithi028 ceph-mon[29442]: 5.6 scrub starts 2024-02-05T06:42:53.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:53 smithi028 ceph-mon[29442]: 5.6 scrub ok 2024-02-05T06:42:53.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:53 smithi028 ceph-mon[29442]: 1.0 scrub starts 2024-02-05T06:42:53.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:53 smithi028 ceph-mon[29442]: 1.0 scrub ok 2024-02-05T06:42:53.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:53 smithi028 ceph-mon[29442]: 5.1 scrub starts 2024-02-05T06:42:53.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:53 smithi028 ceph-mon[29442]: 5.1 scrub ok 2024-02-05T06:42:53.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:53 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:42:53.720 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:53.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:53.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:53.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:53 smithi167 ceph-mon[32774]: pgmap v462: 158 pgs: 1 activating+degraded, 157 active+clean; 569 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.05k op/s; 2/27870 objects degraded (0.007%); 7.9 MiB/s, 126 objects/s recovering 2024-02-05T06:42:53.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:53 smithi167 ceph-mon[32774]: 5.6 scrub starts 2024-02-05T06:42:53.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:53 smithi167 ceph-mon[32774]: 5.6 scrub ok 2024-02-05T06:42:53.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:53 smithi167 ceph-mon[32774]: 1.0 scrub starts 2024-02-05T06:42:53.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:53 smithi167 ceph-mon[32774]: 1.0 scrub ok 2024-02-05T06:42:53.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:53 smithi167 ceph-mon[32774]: 5.1 scrub starts 2024-02-05T06:42:53.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:53 smithi167 ceph-mon[32774]: 5.1 scrub ok 2024-02-05T06:42:53.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:53 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:42:53.823 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:53.863 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:53.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:53.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:53.965 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:54.004 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:54.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:54.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:54.106 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:54.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:54.131 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:54.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:54.232 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:54.262 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:54.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:54.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:54.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:54.399 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:54.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:54.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:54.501 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:54.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:54.533 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:54.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:54.634 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:54.667 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:54.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:54.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:54.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:54 smithi028 ceph-mon[29442]: 4.15 scrub starts 2024-02-05T06:42:54.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:54 smithi028 ceph-mon[29442]: 4.15 scrub ok 2024-02-05T06:42:54.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:54 smithi028 ceph-mon[29442]: 6.18 scrub starts 2024-02-05T06:42:54.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:54 smithi028 ceph-mon[29442]: 6.18 scrub ok 2024-02-05T06:42:54.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:54 smithi028 ceph-mon[29442]: Health check failed: Degraded data redundancy: 2/27870 objects degraded (0.007%), 1 pg degraded (PG_DEGRADED) 2024-02-05T06:42:54.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:54 smithi028 ceph-mon[29442]: 1.2 scrub starts 2024-02-05T06:42:54.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:54 smithi028 ceph-mon[29442]: 1.2 scrub ok 2024-02-05T06:42:54.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:54 smithi028 ceph-mon[37984]: 4.15 scrub starts 2024-02-05T06:42:54.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:54 smithi028 ceph-mon[37984]: 4.15 scrub ok 2024-02-05T06:42:54.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:54 smithi028 ceph-mon[37984]: 6.18 scrub starts 2024-02-05T06:42:54.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:54 smithi028 ceph-mon[37984]: 6.18 scrub ok 2024-02-05T06:42:54.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:54 smithi028 ceph-mon[37984]: Health check failed: Degraded data redundancy: 2/27870 objects degraded (0.007%), 1 pg degraded (PG_DEGRADED) 2024-02-05T06:42:54.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:54 smithi028 ceph-mon[37984]: 1.2 scrub starts 2024-02-05T06:42:54.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:54 smithi028 ceph-mon[37984]: 1.2 scrub ok 2024-02-05T06:42:54.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:54 smithi167 ceph-mon[32774]: 4.15 scrub starts 2024-02-05T06:42:54.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:54 smithi167 ceph-mon[32774]: 4.15 scrub ok 2024-02-05T06:42:54.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:54 smithi167 ceph-mon[32774]: 6.18 scrub starts 2024-02-05T06:42:54.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:54 smithi167 ceph-mon[32774]: 6.18 scrub ok 2024-02-05T06:42:54.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:54 smithi167 ceph-mon[32774]: Health check failed: Degraded data redundancy: 2/27870 objects degraded (0.007%), 1 pg degraded (PG_DEGRADED) 2024-02-05T06:42:54.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:54 smithi167 ceph-mon[32774]: 1.2 scrub starts 2024-02-05T06:42:54.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:54 smithi167 ceph-mon[32774]: 1.2 scrub ok 2024-02-05T06:42:54.769 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:54.801 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:54.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:54.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:54.904 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:54.941 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:54.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:54.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:54.982 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd reweight 1 1 2024-02-05T06:42:55.043 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:55.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:55.088 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:55.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:55.189 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:55.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:55.232 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:55.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:55.333 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:55.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:55.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:55.519 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:55.545 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:55 smithi028 ceph-mon[29442]: pgmap v463: 158 pgs: 1 activating+degraded, 157 active+clean; 569 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 717 op/s; 2/27870 objects degraded (0.007%); 5.4 MiB/s, 86 objects/s recovering 2024-02-05T06:42:55.545 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:55 smithi028 ceph-mon[29442]: 6.5 scrub starts 2024-02-05T06:42:55.545 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:55 smithi028 ceph-mon[29442]: 6.5 scrub ok 2024-02-05T06:42:55.545 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:55 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:42:55.545 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:55 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:42:55.546 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:55 smithi028 ceph-mon[37984]: pgmap v463: 158 pgs: 1 activating+degraded, 157 active+clean; 569 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 717 op/s; 2/27870 objects degraded (0.007%); 5.4 MiB/s, 86 objects/s recovering 2024-02-05T06:42:55.546 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:55 smithi028 ceph-mon[37984]: 6.5 scrub starts 2024-02-05T06:42:55.546 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:55 smithi028 ceph-mon[37984]: 6.5 scrub ok 2024-02-05T06:42:55.546 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:55 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:42:55.546 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:55 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:42:55.586 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:55.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:55.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:55.688 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:55.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:55 smithi167 ceph-mon[32774]: pgmap v463: 158 pgs: 1 activating+degraded, 157 active+clean; 569 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 717 op/s; 2/27870 objects degraded (0.007%); 5.4 MiB/s, 86 objects/s recovering 2024-02-05T06:42:55.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:55 smithi167 ceph-mon[32774]: 6.5 scrub starts 2024-02-05T06:42:55.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:55 smithi167 ceph-mon[32774]: 6.5 scrub ok 2024-02-05T06:42:55.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:55 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:42:55.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:55 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:42:55.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:55.739 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:55.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:55.840 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:55.874 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:55.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:55.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:55.903 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:42:55.951 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:42:55.976 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:55.989 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:42:56.006 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:56.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:56.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:56.031 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:42:56.076 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:42:56.108 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:56.118 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:42:56.147 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:56.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:56.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:56.161 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:42:56.203 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:42:56.213 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:42:56.213 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:42:56.213 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:42:56.213 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:42:56.213 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:42:56.213 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:42:56.213 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:42:56.213 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:42:56.249 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:56.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:56.288 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:56.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:56.389 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:56.422 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:56.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:56.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:56.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:56 smithi028 ceph-mon[29442]: 6.10 scrub starts 2024-02-05T06:42:56.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:56 smithi028 ceph-mon[29442]: 6.10 scrub ok 2024-02-05T06:42:56.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:56 smithi028 ceph-mon[29442]: osdmap e281: 8 total, 8 up, 8 in 2024-02-05T06:42:56.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:56 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1769321903' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:42:56.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:56 smithi028 ceph-mon[37984]: 6.10 scrub starts 2024-02-05T06:42:56.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:56 smithi028 ceph-mon[37984]: 6.10 scrub ok 2024-02-05T06:42:56.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:56 smithi028 ceph-mon[37984]: osdmap e281: 8 total, 8 up, 8 in 2024-02-05T06:42:56.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:56 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1769321903' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:42:56.524 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:56.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:56.558 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:56.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:56.660 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:56.691 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:56.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:56.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:56.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:56 smithi167 ceph-mon[32774]: 6.10 scrub starts 2024-02-05T06:42:56.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:56 smithi167 ceph-mon[32774]: 6.10 scrub ok 2024-02-05T06:42:56.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:56 smithi167 ceph-mon[32774]: osdmap e281: 8 total, 8 up, 8 in 2024-02-05T06:42:56.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:56 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1769321903' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:42:56.792 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:56.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:56.827 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:56.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:56.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:56.967 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:56.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:56.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:57.069 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:42:57.101 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:57.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:57.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:42:57.141 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:42:57.203 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:57.236 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:57.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:57.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:57.338 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:57.373 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:57.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:57.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:57.474 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:42:57.514 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:57.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:57.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:42:57.616 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:57.648 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:57.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:57.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:57.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:57 smithi028 ceph-mon[29442]: 5.f deep-scrub starts 2024-02-05T06:42:57.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:57 smithi028 ceph-mon[29442]: 5.f deep-scrub ok 2024-02-05T06:42:57.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:57 smithi028 ceph-mon[29442]: 5.2 scrub starts 2024-02-05T06:42:57.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:57 smithi028 ceph-mon[29442]: 5.2 scrub ok 2024-02-05T06:42:57.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:57 smithi028 ceph-mon[29442]: pgmap v465: 158 pgs: 158 active+clean; 484 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 905 op/s; 5.2 MiB/s, 83 objects/s recovering 2024-02-05T06:42:57.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:57 smithi028 ceph-mon[29442]: 3.7 scrub starts 2024-02-05T06:42:57.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:57 smithi028 ceph-mon[29442]: 3.7 scrub ok 2024-02-05T06:42:57.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:57 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:42:57.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:57 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:42:57.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:57 smithi028 ceph-mon[29442]: 5.e scrub starts 2024-02-05T06:42:57.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:57 smithi028 ceph-mon[29442]: 5.e scrub ok 2024-02-05T06:42:57.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:57 smithi028 ceph-mon[37984]: 5.f deep-scrub starts 2024-02-05T06:42:57.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:57 smithi028 ceph-mon[37984]: 5.f deep-scrub ok 2024-02-05T06:42:57.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:57 smithi028 ceph-mon[37984]: 5.2 scrub starts 2024-02-05T06:42:57.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:57 smithi028 ceph-mon[37984]: 5.2 scrub ok 2024-02-05T06:42:57.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:57 smithi028 ceph-mon[37984]: pgmap v465: 158 pgs: 158 active+clean; 484 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 905 op/s; 5.2 MiB/s, 83 objects/s recovering 2024-02-05T06:42:57.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:57 smithi028 ceph-mon[37984]: 3.7 scrub starts 2024-02-05T06:42:57.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:57 smithi028 ceph-mon[37984]: 3.7 scrub ok 2024-02-05T06:42:57.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:57 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:42:57.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:57 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:42:57.703 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:57 smithi028 ceph-mon[37984]: 5.e scrub starts 2024-02-05T06:42:57.703 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:57 smithi028 ceph-mon[37984]: 5.e scrub ok 2024-02-05T06:42:57.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:57 smithi167 ceph-mon[32774]: 5.f deep-scrub starts 2024-02-05T06:42:57.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:57 smithi167 ceph-mon[32774]: 5.f deep-scrub ok 2024-02-05T06:42:57.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:57 smithi167 ceph-mon[32774]: 5.2 scrub starts 2024-02-05T06:42:57.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:57 smithi167 ceph-mon[32774]: 5.2 scrub ok 2024-02-05T06:42:57.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:57 smithi167 ceph-mon[32774]: pgmap v465: 158 pgs: 158 active+clean; 484 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 905 op/s; 5.2 MiB/s, 83 objects/s recovering 2024-02-05T06:42:57.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:57 smithi167 ceph-mon[32774]: 3.7 scrub starts 2024-02-05T06:42:57.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:57 smithi167 ceph-mon[32774]: 3.7 scrub ok 2024-02-05T06:42:57.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:57 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:42:57.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:57 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:42:57.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:57 smithi167 ceph-mon[32774]: 5.e scrub starts 2024-02-05T06:42:57.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:57 smithi167 ceph-mon[32774]: 5.e scrub ok 2024-02-05T06:42:57.750 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:57.781 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:57.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:57.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:57.852 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:42:57.883 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:57.936 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:57.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:57.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:58.038 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:58.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:58.083 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:58.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:58.184 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:58.218 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:58.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:58.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:58.321 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:42:58.356 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:58.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:58.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:42:58.458 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:58.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:58.490 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:58.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:58.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:58 smithi028 ceph-mon[37984]: 3.a scrub starts 2024-02-05T06:42:58.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:58 smithi028 ceph-mon[37984]: 3.a scrub ok 2024-02-05T06:42:58.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:58 smithi028 ceph-mon[37984]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/27870 objects degraded (0.007%), 1 pg degraded) 2024-02-05T06:42:58.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:58 smithi028 ceph-mon[37984]: Cluster is now healthy 2024-02-05T06:42:58.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:58 smithi028 ceph-mon[37984]: 3.1a scrub starts 2024-02-05T06:42:58.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:58 smithi028 ceph-mon[37984]: 3.1a scrub ok 2024-02-05T06:42:58.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:58 smithi028 ceph-mon[29442]: 3.a scrub starts 2024-02-05T06:42:58.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:58 smithi028 ceph-mon[29442]: 3.a scrub ok 2024-02-05T06:42:58.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:58 smithi028 ceph-mon[29442]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/27870 objects degraded (0.007%), 1 pg degraded) 2024-02-05T06:42:58.492 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:58 smithi028 ceph-mon[29442]: Cluster is now healthy 2024-02-05T06:42:58.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:58 smithi028 ceph-mon[29442]: 3.1a scrub starts 2024-02-05T06:42:58.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:58 smithi028 ceph-mon[29442]: 3.1a scrub ok 2024-02-05T06:42:58.591 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:42:58.627 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:58.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:58.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:42:58.729 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:58.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:58 smithi167 ceph-mon[32774]: 3.a scrub starts 2024-02-05T06:42:58.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:58 smithi167 ceph-mon[32774]: 3.a scrub ok 2024-02-05T06:42:58.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:58 smithi167 ceph-mon[32774]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/27870 objects degraded (0.007%), 1 pg degraded) 2024-02-05T06:42:58.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:58 smithi167 ceph-mon[32774]: Cluster is now healthy 2024-02-05T06:42:58.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:58 smithi167 ceph-mon[32774]: 3.1a scrub starts 2024-02-05T06:42:58.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:58 smithi167 ceph-mon[32774]: 3.1a scrub ok 2024-02-05T06:42:58.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:58.772 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:58.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:58.873 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:58.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:58.902 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:58.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:59.003 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:59.024 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:42:59.040 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:59.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:59.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:59.074 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:42:59.074 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:42:59.142 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:59.187 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:59.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:59.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:59.289 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:59.319 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:59.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:59.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:59.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:59 smithi028 ceph-mon[29442]: pgmap v466: 158 pgs: 158 active+clean; 214 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 4.1 MiB/s, 65 objects/s recovering 2024-02-05T06:42:59.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:59 smithi028 ceph-mon[29442]: 6.8 scrub starts 2024-02-05T06:42:59.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:59 smithi028 ceph-mon[29442]: 6.8 scrub ok 2024-02-05T06:42:59.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:59 smithi028 ceph-mon[29442]: 5.1e scrub starts 2024-02-05T06:42:59.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:59 smithi028 ceph-mon[29442]: 5.1e scrub ok 2024-02-05T06:42:59.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:59 smithi028 ceph-mon[29442]: 3.14 scrub starts 2024-02-05T06:42:59.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:42:59 smithi028 ceph-mon[29442]: 3.14 scrub ok 2024-02-05T06:42:59.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:59 smithi028 ceph-mon[37984]: pgmap v466: 158 pgs: 158 active+clean; 214 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 4.1 MiB/s, 65 objects/s recovering 2024-02-05T06:42:59.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:59 smithi028 ceph-mon[37984]: 6.8 scrub starts 2024-02-05T06:42:59.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:59 smithi028 ceph-mon[37984]: 6.8 scrub ok 2024-02-05T06:42:59.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:59 smithi028 ceph-mon[37984]: 5.1e scrub starts 2024-02-05T06:42:59.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:59 smithi028 ceph-mon[37984]: 5.1e scrub ok 2024-02-05T06:42:59.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:59 smithi028 ceph-mon[37984]: 3.14 scrub starts 2024-02-05T06:42:59.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:42:59 smithi028 ceph-mon[37984]: 3.14 scrub ok 2024-02-05T06:42:59.415 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Removed 89272 objects 2024-02-05T06:42:59.415 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Clean up completed and total clean up time :94.17 2024-02-05T06:42:59.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:42:59.485 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:59.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:59.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:42:59.587 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:42:59.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:59.611 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:59.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:42:59.712 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:59.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:59 smithi167 ceph-mon[32774]: pgmap v466: 158 pgs: 158 active+clean; 214 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 4.1 MiB/s, 65 objects/s recovering 2024-02-05T06:42:59.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:59 smithi167 ceph-mon[32774]: 6.8 scrub starts 2024-02-05T06:42:59.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:59 smithi167 ceph-mon[32774]: 6.8 scrub ok 2024-02-05T06:42:59.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:59 smithi167 ceph-mon[32774]: 5.1e scrub starts 2024-02-05T06:42:59.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:59 smithi167 ceph-mon[32774]: 5.1e scrub ok 2024-02-05T06:42:59.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:59 smithi167 ceph-mon[32774]: 3.14 scrub starts 2024-02-05T06:42:59.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:42:59 smithi167 ceph-mon[32774]: 3.14 scrub ok 2024-02-05T06:42:59.736 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:59.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:59.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:42:59.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:42:59.861 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:42:59.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:59.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:42:59.963 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:42:59.987 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:42:59.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:42:59.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:00.089 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:00.113 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:00.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:00.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:00.215 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:00.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:00.240 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:00.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:00.284 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-05T06:43:00.340 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:00.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:00.365 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:00.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:00.466 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:00.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:00.490 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:00.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:00.592 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:00.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:00.617 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:00.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:00.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:00 smithi028 ceph-mon[37984]: 5.16 scrub starts 2024-02-05T06:43:00.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:00 smithi028 ceph-mon[37984]: 5.16 scrub ok 2024-02-05T06:43:00.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:00 smithi028 ceph-mon[37984]: 3.1b deep-scrub starts 2024-02-05T06:43:00.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:00 smithi028 ceph-mon[37984]: 3.1b deep-scrub ok 2024-02-05T06:43:00.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:00 smithi028 ceph-mon[37984]: 6.3 scrub starts 2024-02-05T06:43:00.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:00 smithi028 ceph-mon[37984]: 6.3 scrub ok 2024-02-05T06:43:00.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:00 smithi028 ceph-mon[37984]: 2.1 scrub starts 2024-02-05T06:43:00.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:00 smithi028 ceph-mon[37984]: 2.1 scrub ok 2024-02-05T06:43:00.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:00 smithi028 ceph-mon[37984]: 4.1 scrub starts 2024-02-05T06:43:00.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:00 smithi028 ceph-mon[37984]: 4.1 scrub ok 2024-02-05T06:43:00.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:00 smithi028 ceph-mon[37984]: osdmap e282: 8 total, 8 up, 8 in 2024-02-05T06:43:00.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:00 smithi028 ceph-mon[29442]: 5.16 scrub starts 2024-02-05T06:43:00.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:00 smithi028 ceph-mon[29442]: 5.16 scrub ok 2024-02-05T06:43:00.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:00 smithi028 ceph-mon[29442]: 3.1b deep-scrub starts 2024-02-05T06:43:00.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:00 smithi028 ceph-mon[29442]: 3.1b deep-scrub ok 2024-02-05T06:43:00.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:00 smithi028 ceph-mon[29442]: 6.3 scrub starts 2024-02-05T06:43:00.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:00 smithi028 ceph-mon[29442]: 6.3 scrub ok 2024-02-05T06:43:00.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:00 smithi028 ceph-mon[29442]: 2.1 scrub starts 2024-02-05T06:43:00.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:00 smithi028 ceph-mon[29442]: 2.1 scrub ok 2024-02-05T06:43:00.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:00 smithi028 ceph-mon[29442]: 4.1 scrub starts 2024-02-05T06:43:00.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:00 smithi028 ceph-mon[29442]: 4.1 scrub ok 2024-02-05T06:43:00.679 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:00 smithi028 ceph-mon[29442]: osdmap e282: 8 total, 8 up, 8 in 2024-02-05T06:43:00.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:00.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:00 smithi167 ceph-mon[32774]: 5.16 scrub starts 2024-02-05T06:43:00.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:00 smithi167 ceph-mon[32774]: 5.16 scrub ok 2024-02-05T06:43:00.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:00 smithi167 ceph-mon[32774]: 3.1b deep-scrub starts 2024-02-05T06:43:00.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:00 smithi167 ceph-mon[32774]: 3.1b deep-scrub ok 2024-02-05T06:43:00.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:00 smithi167 ceph-mon[32774]: 6.3 scrub starts 2024-02-05T06:43:00.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:00 smithi167 ceph-mon[32774]: 6.3 scrub ok 2024-02-05T06:43:00.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:00 smithi167 ceph-mon[32774]: 2.1 scrub starts 2024-02-05T06:43:00.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:00 smithi167 ceph-mon[32774]: 2.1 scrub ok 2024-02-05T06:43:00.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:00 smithi167 ceph-mon[32774]: 4.1 scrub starts 2024-02-05T06:43:00.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:00 smithi167 ceph-mon[32774]: 4.1 scrub ok 2024-02-05T06:43:00.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:00 smithi167 ceph-mon[32774]: osdmap e282: 8 total, 8 up, 8 in 2024-02-05T06:43:00.741 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:00.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:00.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:00.843 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:00.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:00.882 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:00.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:00.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:01.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:01.007 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:01.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:01.108 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:01.135 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:01.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:01.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:01.237 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:01.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:01.262 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:01.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:01.363 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:01.390 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:01.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:01.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:01.492 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:01.520 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:01.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:01.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:01.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:01 smithi028 ceph-mon[29442]: 6.a scrub starts 2024-02-05T06:43:01.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:01 smithi028 ceph-mon[29442]: 6.a scrub ok 2024-02-05T06:43:01.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:01 smithi028 ceph-mon[29442]: 4.f deep-scrub starts 2024-02-05T06:43:01.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:01 smithi028 ceph-mon[29442]: 4.f deep-scrub ok 2024-02-05T06:43:01.582 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:01 smithi028 ceph-mon[29442]: pgmap v468: 158 pgs: 158 active+clean; 214 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 709 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T06:43:01.583 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:01 smithi028 ceph-mon[37984]: 6.a scrub starts 2024-02-05T06:43:01.583 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:01 smithi028 ceph-mon[37984]: 6.a scrub ok 2024-02-05T06:43:01.583 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:01 smithi028 ceph-mon[37984]: 4.f deep-scrub starts 2024-02-05T06:43:01.583 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:01 smithi028 ceph-mon[37984]: 4.f deep-scrub ok 2024-02-05T06:43:01.583 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:01 smithi028 ceph-mon[37984]: pgmap v468: 158 pgs: 158 active+clean; 214 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 709 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T06:43:01.609 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:43:01.621 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:01.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:01.648 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:01.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:01.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:01 smithi167 ceph-mon[32774]: 6.a scrub starts 2024-02-05T06:43:01.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:01 smithi167 ceph-mon[32774]: 6.a scrub ok 2024-02-05T06:43:01.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:01 smithi167 ceph-mon[32774]: 4.f deep-scrub starts 2024-02-05T06:43:01.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:01 smithi167 ceph-mon[32774]: 4.f deep-scrub ok 2024-02-05T06:43:01.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:01 smithi167 ceph-mon[32774]: pgmap v468: 158 pgs: 158 active+clean; 214 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 709 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T06:43:01.750 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:01.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:01.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:01.875 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:01.899 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:01.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:01.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:02.001 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:02.029 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:02.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:02.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:02.130 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:02.158 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:02.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:02.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:02.260 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:02.286 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:02.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:02.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:02.388 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:02.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:02.414 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:02.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:02.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:02 smithi028 ceph-mon[29442]: 8.d scrub starts 2024-02-05T06:43:02.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:02 smithi028 ceph-mon[29442]: 8.d scrub ok 2024-02-05T06:43:02.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:02 smithi028 ceph-mon[29442]: 5.0 scrub starts 2024-02-05T06:43:02.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:02 smithi028 ceph-mon[29442]: 5.0 scrub ok 2024-02-05T06:43:02.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:02 smithi028 ceph-mon[29442]: 6.14 scrub starts 2024-02-05T06:43:02.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:02 smithi028 ceph-mon[29442]: 6.14 scrub ok 2024-02-05T06:43:02.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:02 smithi028 ceph-mon[29442]: 4.7 scrub starts 2024-02-05T06:43:02.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:02 smithi028 ceph-mon[29442]: 4.7 scrub ok 2024-02-05T06:43:02.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:02 smithi028 ceph-mon[37984]: 8.d scrub starts 2024-02-05T06:43:02.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:02 smithi028 ceph-mon[37984]: 8.d scrub ok 2024-02-05T06:43:02.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:02 smithi028 ceph-mon[37984]: 5.0 scrub starts 2024-02-05T06:43:02.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:02 smithi028 ceph-mon[37984]: 5.0 scrub ok 2024-02-05T06:43:02.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:02 smithi028 ceph-mon[37984]: 6.14 scrub starts 2024-02-05T06:43:02.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:02 smithi028 ceph-mon[37984]: 6.14 scrub ok 2024-02-05T06:43:02.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:02 smithi028 ceph-mon[37984]: 4.7 scrub starts 2024-02-05T06:43:02.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:02 smithi028 ceph-mon[37984]: 4.7 scrub ok 2024-02-05T06:43:02.515 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:02.543 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:02.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:02.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:02.645 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:02.673 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:02.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:02.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:02.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:02 smithi167 ceph-mon[32774]: 8.d scrub starts 2024-02-05T06:43:02.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:02 smithi167 ceph-mon[32774]: 8.d scrub ok 2024-02-05T06:43:02.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:02 smithi167 ceph-mon[32774]: 5.0 scrub starts 2024-02-05T06:43:02.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:02 smithi167 ceph-mon[32774]: 5.0 scrub ok 2024-02-05T06:43:02.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:02 smithi167 ceph-mon[32774]: 6.14 scrub starts 2024-02-05T06:43:02.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:02 smithi167 ceph-mon[32774]: 6.14 scrub ok 2024-02-05T06:43:02.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:02 smithi167 ceph-mon[32774]: 4.7 scrub starts 2024-02-05T06:43:02.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:02 smithi167 ceph-mon[32774]: 4.7 scrub ok 2024-02-05T06:43:02.775 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:02.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:02.801 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:02.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:02.877 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:43:02 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:43:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:43:02.902 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:02.928 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:02.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:02.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:03.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:03.052 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:03.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:03.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:03.154 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:03.177 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:03.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:03.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:03.210 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:43:03.233 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:43:02 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:43:03.278 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:03.306 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:03.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:03.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:03.322 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.1 to 1 (10000) 2024-02-05T06:43:03.324 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-02-05T06:43:03.408 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:03.433 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:03.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:03.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:03.535 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:03.562 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:03.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:03.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:03.567 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:03 smithi028 ceph-mon[29442]: 4.3 scrub starts 2024-02-05T06:43:03.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:03 smithi028 ceph-mon[29442]: 4.3 scrub ok 2024-02-05T06:43:03.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:03 smithi028 ceph-mon[29442]: pgmap v469: 158 pgs: 158 active+clean; 582 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T06:43:03.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:03 smithi028 ceph-mon[29442]: 3.d scrub starts 2024-02-05T06:43:03.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:03 smithi028 ceph-mon[29442]: 3.d scrub ok 2024-02-05T06:43:03.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:03 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1053957747' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T06:43:03.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:03 smithi028 ceph-mon[29442]: 5.d scrub starts 2024-02-05T06:43:03.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:03 smithi028 ceph-mon[29442]: 5.d scrub ok 2024-02-05T06:43:03.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:03 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:43:03.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:03 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1664902521' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:43:03.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:03 smithi028 ceph-mon[29442]: from='client.45188 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:43:03.569 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:03 smithi028 ceph-mon[37984]: 4.3 scrub starts 2024-02-05T06:43:03.569 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:03 smithi028 ceph-mon[37984]: 4.3 scrub ok 2024-02-05T06:43:03.569 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:03 smithi028 ceph-mon[37984]: pgmap v469: 158 pgs: 158 active+clean; 582 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T06:43:03.569 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:03 smithi028 ceph-mon[37984]: 3.d scrub starts 2024-02-05T06:43:03.569 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:03 smithi028 ceph-mon[37984]: 3.d scrub ok 2024-02-05T06:43:03.569 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:03 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1053957747' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T06:43:03.569 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:03 smithi028 ceph-mon[37984]: 5.d scrub starts 2024-02-05T06:43:03.569 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:03 smithi028 ceph-mon[37984]: 5.d scrub ok 2024-02-05T06:43:03.569 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:03 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:43:03.569 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:03 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1664902521' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:43:03.569 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:03 smithi028 ceph-mon[37984]: from='client.45188 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:43:03.664 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:03.691 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:03.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:03.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:03.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:03 smithi167 ceph-mon[32774]: 4.3 scrub starts 2024-02-05T06:43:03.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:03 smithi167 ceph-mon[32774]: 4.3 scrub ok 2024-02-05T06:43:03.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:03 smithi167 ceph-mon[32774]: pgmap v469: 158 pgs: 158 active+clean; 582 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 0 B/s, 0 objects/s recovering 2024-02-05T06:43:03.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:03 smithi167 ceph-mon[32774]: 3.d scrub starts 2024-02-05T06:43:03.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:03 smithi167 ceph-mon[32774]: 3.d scrub ok 2024-02-05T06:43:03.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:03 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1053957747' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T06:43:03.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:03 smithi167 ceph-mon[32774]: 5.d scrub starts 2024-02-05T06:43:03.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:03 smithi167 ceph-mon[32774]: 5.d scrub ok 2024-02-05T06:43:03.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:03 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:43:03.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:03 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1664902521' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:43:03.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:03 smithi167 ceph-mon[32774]: from='client.45188 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:43:03.793 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:03.820 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:03.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:03.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:03.922 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:03.949 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:03.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:03.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:04.051 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:04.077 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:04.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:04.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:04.179 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:04.206 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:04.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:04.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:04.307 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:04.315 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-02-05T06:43:04.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-02-05T06:43:04.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:04.334 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:04.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:04.436 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:04.464 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:04.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:04.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:04.565 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:04.570 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:04 smithi028 ceph-mon[29442]: 4.1f deep-scrub starts 2024-02-05T06:43:04.570 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:04 smithi028 ceph-mon[29442]: 4.1f deep-scrub ok 2024-02-05T06:43:04.570 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:04 smithi028 ceph-mon[29442]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:43:04.570 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:04 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1053957747' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T06:43:04.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:04 smithi028 ceph-mon[29442]: from='client.45188 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:43:04.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:04 smithi028 ceph-mon[29442]: osdmap e283: 8 total, 8 up, 8 in 2024-02-05T06:43:04.571 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:04 smithi028 ceph-mon[29442]: 2.0 scrub starts 2024-02-05T06:43:04.571 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:04 smithi028 ceph-mon[37984]: 4.1f deep-scrub starts 2024-02-05T06:43:04.571 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:04 smithi028 ceph-mon[37984]: 4.1f deep-scrub ok 2024-02-05T06:43:04.571 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:04 smithi028 ceph-mon[37984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:43:04.571 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:04 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1053957747' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T06:43:04.571 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:04 smithi028 ceph-mon[37984]: from='client.45188 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:43:04.571 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:04 smithi028 ceph-mon[37984]: osdmap e283: 8 total, 8 up, 8 in 2024-02-05T06:43:04.571 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:04 smithi028 ceph-mon[37984]: 2.0 scrub starts 2024-02-05T06:43:04.590 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:04.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:04.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:04.692 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:04.720 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:04.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:04.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:04.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:04 smithi167 ceph-mon[32774]: 4.1f deep-scrub starts 2024-02-05T06:43:04.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:04 smithi167 ceph-mon[32774]: 4.1f deep-scrub ok 2024-02-05T06:43:04.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:04 smithi167 ceph-mon[32774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:43:04.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:04 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1053957747' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T06:43:04.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:04 smithi167 ceph-mon[32774]: from='client.45188 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:43:04.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:04 smithi167 ceph-mon[32774]: osdmap e283: 8 total, 8 up, 8 in 2024-02-05T06:43:04.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:04 smithi167 ceph-mon[32774]: 2.0 scrub starts 2024-02-05T06:43:04.822 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:04.848 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:04.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:04.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:04.950 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:04.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:04.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:05.076 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:05.102 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:05.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:05.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:05.204 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:05.228 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:05.229 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd reweight 2 1 2024-02-05T06:43:05.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:05.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:05.331 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:05.355 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:05.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:05.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:05.457 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:05.481 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:05.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:05.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:05.583 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:05.584 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:05 smithi028 ceph-mon[37984]: pgmap v471: 158 pgs: 158 active+clean; 582 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 966 op/s 2024-02-05T06:43:05.584 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:05 smithi028 ceph-mon[37984]: osdmap e284: 8 total, 8 up, 8 in 2024-02-05T06:43:05.584 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:05 smithi028 ceph-mon[29442]: pgmap v471: 158 pgs: 158 active+clean; 582 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 966 op/s 2024-02-05T06:43:05.584 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:05 smithi028 ceph-mon[29442]: osdmap e284: 8 total, 8 up, 8 in 2024-02-05T06:43:05.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:05.612 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:05.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:05.634 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:05 smithi167 ceph-mon[32774]: pgmap v471: 158 pgs: 158 active+clean; 582 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 966 op/s 2024-02-05T06:43:05.634 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:05 smithi167 ceph-mon[32774]: osdmap e284: 8 total, 8 up, 8 in 2024-02-05T06:43:05.714 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:05.741 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:05.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:05.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:05.843 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:05.870 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:05.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:05.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:05.972 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:05.997 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:05.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:05.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:06.099 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:06.123 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:06.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:06.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:06.144 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:43:06.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:06.249 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:06.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:06.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:06.351 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:06.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:06.375 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:06.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:06.476 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:06.504 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:06.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:06.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:06.606 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:06.633 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:06.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:06.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:06.735 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:06.762 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:06.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:06.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:06.817 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:43:06.865 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:06.892 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:06.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:06.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:06.994 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:07.018 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:07.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:07.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:07.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:07.143 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:07.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:07.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:07.244 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:07.272 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:07.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:07.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:07.374 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:07.398 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:07.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:07.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:07.500 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:07.528 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:07.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:07.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:07.538 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:43:07.556 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:43:07.557 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:07 smithi028 ceph-mon[29442]: pgmap v473: 158 pgs: 158 active+clean; 582 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 547 op/s 2024-02-05T06:43:07.557 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:07 smithi028 ceph-mon[37984]: pgmap v473: 158 pgs: 158 active+clean; 582 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 547 op/s 2024-02-05T06:43:07.576 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:43:07.597 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:43:07.618 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:43:07.631 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:07.637 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:43:07.659 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:43:07.659 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:07.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:07.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:07.678 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:43:07.679 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T06:43:07.680 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:43:07.680 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:43:07.680 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:43:07.681 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:43:07.681 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:43:07.681 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:43:07.681 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:43:07.681 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:43:07.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:07 smithi167 ceph-mon[32774]: pgmap v473: 158 pgs: 158 active+clean; 582 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 547 op/s 2024-02-05T06:43:07.762 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:07.785 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:07.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:07.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:07.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:07.910 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:07.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:07.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:08.011 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:08.034 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:08.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:08.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:08.137 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:08.160 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:08.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:08.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:08.262 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:08.285 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:08.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:08.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:08.387 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:08.415 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:08.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:08.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:08.517 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:08.545 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:08.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:08.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:08.647 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:08.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:08.672 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:08.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:08.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:08 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/833911108' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:43:08.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:08 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/833911108' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:43:08.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:08 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/833911108' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:43:08.773 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:08.796 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:08.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:08.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:08.898 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:08.926 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:08.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:08.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:09.028 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:09.056 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:09.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:09.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:09.158 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:09.185 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:09.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:09.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:09.287 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:09.315 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:09.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:09.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:09.414 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.2 to 1 (10000) 2024-02-05T06:43:09.417 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:09.444 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:09.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:09.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:09.546 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:09.571 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:09.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:09.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:09.673 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:09.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:09 smithi028 ceph-mon[29442]: pgmap v474: 158 pgs: 158 active+clean; 582 KiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 428 op/s 2024-02-05T06:43:09.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:09 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:43:09.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:09 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:43:09.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:09 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1954713803' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T06:43:09.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:09 smithi028 ceph-mon[37984]: pgmap v474: 158 pgs: 158 active+clean; 582 KiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 428 op/s 2024-02-05T06:43:09.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:09 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:43:09.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:09 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:43:09.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:09 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1954713803' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T06:43:09.701 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:09.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:09.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:09.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:09 smithi167 ceph-mon[32774]: pgmap v474: 158 pgs: 158 active+clean; 582 KiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 428 op/s 2024-02-05T06:43:09.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:09 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:43:09.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:09 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:43:09.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:09 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1954713803' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T06:43:09.802 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:09.827 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:09.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:09.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:09.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:09.953 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:09.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:09.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:10.055 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:10.080 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:10.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:10.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:10.181 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:10.205 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:10.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:10.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:10.307 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:10.331 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:10.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:10.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:10.433 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:10.461 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:10.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:10.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:10.563 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:10.575 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:43:10.590 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:10.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:10.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:10.668 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:10 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1954713803' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T06:43:10.668 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:10 smithi028 ceph-mon[29442]: osdmap e285: 8 total, 8 up, 8 in 2024-02-05T06:43:10.668 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:10 smithi028 ceph-mon[29442]: osdmap e286: 8 total, 8 up, 8 in 2024-02-05T06:43:10.668 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:10 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1954713803' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T06:43:10.669 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:10 smithi028 ceph-mon[37984]: osdmap e285: 8 total, 8 up, 8 in 2024-02-05T06:43:10.669 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:10 smithi028 ceph-mon[37984]: osdmap e286: 8 total, 8 up, 8 in 2024-02-05T06:43:10.692 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:10.715 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:10.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:10.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:10.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:10 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1954713803' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T06:43:10.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:10 smithi167 ceph-mon[32774]: osdmap e285: 8 total, 8 up, 8 in 2024-02-05T06:43:10.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:10 smithi167 ceph-mon[32774]: osdmap e286: 8 total, 8 up, 8 in 2024-02-05T06:43:10.817 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:10.845 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:10.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:10.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:10.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:10.970 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:10.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:10.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:11.072 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:11.096 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:11.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:11.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:11.198 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:11.222 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:11.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:11.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:11.226 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd reweight 3 1 2024-02-05T06:43:11.324 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:11.353 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:11.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:11.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:11.454 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:11.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:11 smithi028 ceph-mon[29442]: pgmap v477: 158 pgs: 158 active+clean; 582 KiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:43:11.478 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:11 smithi028 ceph-mon[37984]: pgmap v477: 158 pgs: 158 active+clean; 582 KiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:43:11.479 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:11.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:11.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:11.581 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:11.609 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:11.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:11.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:11.684 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:43:11.684 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-02-05T06:43:11.684 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-02-05T06:43:11.684 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:43:11.711 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:11.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:11 smithi167 ceph-mon[32774]: pgmap v477: 158 pgs: 158 active+clean; 582 KiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:43:11.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:11.739 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:11.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:11.840 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:11.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:11.866 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:11.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:11.931 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:43:11.967 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:11.995 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:11.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:11.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:12.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:12.120 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:12.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:12.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:12.222 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:12.246 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:12.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:12.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:12.347 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:12.371 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:12.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:12.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:12.473 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:12.501 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:12.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:12.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:12.603 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:12.626 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:12.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:12.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:12.728 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:12.751 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:12.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:12.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:12.853 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:12.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:12.882 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:12.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:12.891 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:12 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:43:12.891 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:12 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:43:12.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:12 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:43:12.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:12 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:43:12.948 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:43:12 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:43:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:43:12.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:12 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:43:12.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:12 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:43:12.983 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:13.011 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:13.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:13.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:13.113 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:13.141 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:13.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:13.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:13.233 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:43:12 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:43:13.242 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:13.270 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:13.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:13.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:13.284 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-05T06:43:13.371 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:13.399 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:13.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:13.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:13.438 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:43:13.501 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:13.530 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:13.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:13.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:13.632 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:13.658 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:13.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:13.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:13.760 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:13.787 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:13.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:13.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:13.888 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:13.914 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:13.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:13.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:13.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:13 smithi028 ceph-mon[29442]: pgmap v478: 158 pgs: 158 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-05T06:43:13.931 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:13 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:43:13.931 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:13 smithi028 ceph-mon[37984]: pgmap v478: 158 pgs: 158 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-05T06:43:13.931 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:13 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:43:13.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:13 smithi167 ceph-mon[32774]: pgmap v478: 158 pgs: 158 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-05T06:43:13.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:13 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:43:14.016 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:14.040 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:14.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:14.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:14.142 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:14.165 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:14.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:14.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:14.267 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:14.290 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:14.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:14.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:14.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:14.416 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:14.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:14.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:14.518 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:14.545 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:14.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:14.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:14.647 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:14.675 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:14.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:14.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:14.776 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:14.805 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:14.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:14.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:14.867 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:43:14.906 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:14.933 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:14.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:14.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:15.035 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:15.063 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:15.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:15.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:15.165 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:15.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:15.190 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:15.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:15.291 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:15.319 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:15.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:15.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:15.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:15.446 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:15.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:15.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:15.543 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'unique_pool_1' removed 2024-02-05T06:43:15.543 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-02-05T06:43:15.547 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:15.582 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:15.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:15.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:15.684 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:15.708 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:15.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:15.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:15.810 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:15.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:15.834 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:15.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:15.869 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:15 smithi028 ceph-mon[29442]: pgmap v479: 158 pgs: 158 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:43:15.869 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:15 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/625153694' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T06:43:15.869 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:15 smithi028 ceph-mon[29442]: from='client.45206 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T06:43:15.869 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:15 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3810173188' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:43:15.869 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:15 smithi028 ceph-mon[29442]: from='client.45212 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:43:15.869 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:15 smithi028 ceph-mon[37984]: pgmap v479: 158 pgs: 158 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:43:15.870 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:15 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/625153694' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T06:43:15.870 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:15 smithi028 ceph-mon[37984]: from='client.45206 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T06:43:15.870 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:15 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3810173188' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:43:15.870 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:15 smithi028 ceph-mon[37984]: from='client.45212 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:43:15.935 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:15.959 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:15.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:15.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:15.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:15 smithi167 ceph-mon[32774]: pgmap v479: 158 pgs: 158 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:43:15.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:15 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/625153694' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T06:43:15.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:15 smithi167 ceph-mon[32774]: from='client.45206 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T06:43:15.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:15 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3810173188' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:43:15.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:15 smithi167 ceph-mon[32774]: from='client.45212 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:43:16.061 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:16.086 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:16.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:16.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:16.188 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:16.211 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:16.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:16.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:16.313 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:16.337 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:16.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:16.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:16.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:16.462 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:16.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:16.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:16.563 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:16.587 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:16.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:16.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:16.688 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:16.699 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-05T06:43:16.700 INFO:tasks.radosbench:Beginning radosbench... 2024-02-05T06:43:16.700 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-02-05T06:43:16.701 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd pool create unique_pool_2 16 2024-02-05T06:43:16.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:16.713 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:16.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:16.814 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:16.838 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:16.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:16.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:16.890 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:43:16.940 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:16.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:16 smithi028 ceph-mon[29442]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:43:16.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:16 smithi028 ceph-mon[29442]: from='client.45206 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T06:43:16.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:16 smithi028 ceph-mon[29442]: from='client.45212 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:43:16.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:16 smithi028 ceph-mon[29442]: osdmap e287: 8 total, 8 up, 8 in 2024-02-05T06:43:16.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:16 smithi028 ceph-mon[37984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:43:16.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:16 smithi028 ceph-mon[37984]: from='client.45206 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T06:43:16.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:16 smithi028 ceph-mon[37984]: from='client.45212 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:43:16.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:16 smithi028 ceph-mon[37984]: osdmap e287: 8 total, 8 up, 8 in 2024-02-05T06:43:16.964 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:16.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:16.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:16.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:16 smithi167 ceph-mon[32774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:43:16.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:16 smithi167 ceph-mon[32774]: from='client.45206 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T06:43:16.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:16 smithi167 ceph-mon[32774]: from='client.45212 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:43:16.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:16 smithi167 ceph-mon[32774]: osdmap e287: 8 total, 8 up, 8 in 2024-02-05T06:43:17.065 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:17.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:17.091 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:17.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:17.192 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:17.216 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:17.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:17.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:17.318 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:17.342 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:17.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:17.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:17.444 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:17.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:17.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:17.569 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:17.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:17.594 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:17.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:17.695 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:17.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:17.719 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:17.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:17.820 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:17.845 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:17.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:17.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:17.924 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:17 smithi028 ceph-mon[29442]: pgmap v481: 142 pgs: 142 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T06:43:17.924 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:17 smithi028 ceph-mon[37984]: pgmap v481: 142 pgs: 142 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T06:43:17.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:17.970 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:17.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:17.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:17.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:17 smithi167 ceph-mon[32774]: pgmap v481: 142 pgs: 142 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T06:43:18.072 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:18.101 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:18.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:18.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:18.202 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:18.230 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:18.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:18.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:18.332 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:18.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:18.357 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:18.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:18.457 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:18.480 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:18.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:18.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:18.582 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:18.606 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:18.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:18.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:18.708 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:18.736 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:18.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:18.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:18.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:18.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:18.862 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:18.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:18.963 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:18.987 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:18.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:18.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:19.089 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:19.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:19.118 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:19.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:19.219 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:19.229 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T06:43:19.246 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:19.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:19.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:19.348 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:19.374 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:19.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:19.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:19.382 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:43:19.404 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:43:19.427 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:43:19.451 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:43:19.470 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:43:19.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:19.493 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:43:19.502 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:19.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:19.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:19.515 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:43:19.536 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:43:19.537 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:43:19.537 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:43:19.538 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:43:19.538 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:43:19.538 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:43:19.538 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:43:19.538 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:43:19.538 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:43:19.547 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.3 to 1 (10000) 2024-02-05T06:43:19.603 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:19.627 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:19.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:19.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:19.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:19 smithi028 ceph-mon[29442]: pgmap v482: 142 pgs: 142 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:43:19.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:19 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1298688753' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T06:43:19.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:19 smithi028 ceph-mon[29442]: from='client.45218 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T06:43:19.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:19 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:43:19.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:19 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3250658034' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:43:19.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:19 smithi028 ceph-mon[37984]: pgmap v482: 142 pgs: 142 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:43:19.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:19 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1298688753' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T06:43:19.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:19 smithi028 ceph-mon[37984]: from='client.45218 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T06:43:19.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:19 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:43:19.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:19 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3250658034' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:43:19.729 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:19.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:19.753 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:19.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:19.854 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:19.878 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:19.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:19.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:19.980 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:19.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:19 smithi167 ceph-mon[32774]: pgmap v482: 142 pgs: 142 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:43:19.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:19 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1298688753' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T06:43:19.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:19 smithi167 ceph-mon[32774]: from='client.45218 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T06:43:19.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:19 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:43:19.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:19 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3250658034' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:43:20.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:20.006 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:20.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:20.107 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:20.131 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:20.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:20.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:20.232 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:20.256 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:20.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:20.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:20.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:20.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:20.382 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:20.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:20.482 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:20.492 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:43:20.511 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:20.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:20.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:20.613 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:20.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:20.637 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:20.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:20.738 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:20.762 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:20.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:20.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:20.864 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:20.878 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:20 smithi028 ceph-mon[29442]: from='client.45218 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-05T06:43:20.878 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:20 smithi028 ceph-mon[29442]: osdmap e288: 8 total, 8 up, 8 in 2024-02-05T06:43:20.878 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:20 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:43:20.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:20 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:43:20.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:20 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:43:20.879 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:20 smithi028 ceph-mon[37984]: from='client.45218 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-05T06:43:20.879 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:20 smithi028 ceph-mon[37984]: osdmap e288: 8 total, 8 up, 8 in 2024-02-05T06:43:20.879 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:20 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:43:20.879 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:20 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:43:20.879 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:20 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:43:20.892 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:20.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:20.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:20.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:20 smithi167 ceph-mon[32774]: from='client.45218 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-05T06:43:20.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:20 smithi167 ceph-mon[32774]: osdmap e288: 8 total, 8 up, 8 in 2024-02-05T06:43:20.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:20 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:43:20.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:20 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:43:20.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:20 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:43:20.994 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:21.022 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:21.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:21.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:21.124 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:21.149 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:21.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:21.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:21.251 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:21.279 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:21.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:21.439 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd reweight 6 1 2024-02-05T06:43:21.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:21.539 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:21.564 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:21.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:21.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:21.666 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:21.694 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:21.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:21.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:21.741 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:43:21.764 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:43:21.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:21.820 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:21.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:21.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:21.866 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:21 smithi028 ceph-mon[37984]: pgmap v484: 142 pgs: 142 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-05T06:43:21.866 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:21 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:43:21.866 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:21 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:43:21.866 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:21 smithi028 ceph-mon[37984]: osdmap e289: 8 total, 8 up, 8 in 2024-02-05T06:43:21.867 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:21 smithi028 ceph-mon[29442]: pgmap v484: 142 pgs: 142 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-05T06:43:21.867 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:21 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:43:21.867 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:21 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:43:21.867 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:21 smithi028 ceph-mon[29442]: osdmap e289: 8 total, 8 up, 8 in 2024-02-05T06:43:21.922 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:21.945 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:21.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:21.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:21.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:21 smithi167 ceph-mon[32774]: pgmap v484: 142 pgs: 142 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-05T06:43:21.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:21 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:43:21.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:21 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:43:21.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:21 smithi167 ceph-mon[32774]: osdmap e289: 8 total, 8 up, 8 in 2024-02-05T06:43:22.047 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:22.071 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:22.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:22.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:22.173 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:22.201 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:22.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:22.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:22.302 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:22.330 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:22.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:22.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:22.432 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:22.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:22.460 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:22.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:22.561 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:22.589 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:22.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:22.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:22.691 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:22.698 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:43:22 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:43:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:43:22.718 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:22.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:22.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:22.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:22.843 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:22.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:22.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:22.945 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:22.973 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:22.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:22.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:23.075 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:23.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:23.100 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:23.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:23.201 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:23.229 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:23.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:23.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:23.232 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:43:22 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:43:23.331 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:23.359 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:23.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:23.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:23.461 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:23.485 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:23.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:23.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:23.586 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:23.615 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:23.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:23.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:23.706 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:43:23.716 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:23.743 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:23.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:23.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:23.845 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:23.873 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:23.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:23.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:23.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:23 smithi028 ceph-mon[37984]: pgmap v486: 142 pgs: 142 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-05T06:43:23.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:23 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:43:23.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:23 smithi028 ceph-mon[29442]: pgmap v486: 142 pgs: 142 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-05T06:43:23.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:23 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:43:23.974 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:23.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:23 smithi167 ceph-mon[32774]: pgmap v486: 142 pgs: 142 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-05T06:43:23.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:23 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:43:23.998 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:23.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:23.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:24.100 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:24.123 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:24.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:24.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:24.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:24.265 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:24.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:24.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:24.367 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:24.391 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:43:24.394 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:24.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:24.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:24.407 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:43:24.407 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:43:24.495 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:24.519 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:24.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:24.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:24.620 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:24.648 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:24.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:24.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:24.750 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:24.778 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:24.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:24.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:24.880 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:24.908 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:24.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:24.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:25.009 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:25.037 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:25.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:25.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:25.139 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:25.166 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:25.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:25.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:25.268 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:25.294 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:25.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:25.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:25.395 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:25.421 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:25.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:25.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:25.522 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:25.525 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-05T06:43:25.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:25.548 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:25.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:25.649 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:25.676 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:25.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:25.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:25.778 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:25.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:25 smithi028 ceph-mon[29442]: pgmap v487: 142 pgs: 142 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:43:25.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:25 smithi028 ceph-mon[29442]: osdmap e290: 8 total, 8 up, 8 in 2024-02-05T06:43:25.785 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:25 smithi028 ceph-mon[37984]: pgmap v487: 142 pgs: 142 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:43:25.785 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:25 smithi028 ceph-mon[37984]: osdmap e290: 8 total, 8 up, 8 in 2024-02-05T06:43:25.803 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:25.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:25.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:25.905 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:25.933 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:25.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:25.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:25.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:25 smithi167 ceph-mon[32774]: pgmap v487: 142 pgs: 142 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:43:25.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:25 smithi167 ceph-mon[32774]: osdmap e290: 8 total, 8 up, 8 in 2024-02-05T06:43:26.036 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:26.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:26.060 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:26.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:26.161 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:26.189 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:26.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:26.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:26.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:26.315 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:26.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:26.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:26.417 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:26.445 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:26.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:26.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:26.547 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:26.574 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:26.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:26.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:26.676 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:26.703 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:26.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:26.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:26.805 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:26.832 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:26.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:26.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:26.903 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:26 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:43:26.903 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:26 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:43:26.934 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:26.959 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:26.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:26.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:26.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:26 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:43:27.061 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:27.086 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:27.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:27.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:27.189 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:27.217 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:27.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:27.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:27.318 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:27.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:27.346 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:27.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:27.447 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:27.471 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:27.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:27.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:27.573 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:27.599 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:27.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:27.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:27.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:27 smithi028 ceph-mon[29442]: pgmap v489: 142 pgs: 142 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-05T06:43:27.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:27 smithi028 ceph-mon[37984]: pgmap v489: 142 pgs: 142 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-05T06:43:27.701 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:27.724 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:27.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:27.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:27.826 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:27.854 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:27.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:27.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:27.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:27.979 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:27.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:27.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:27.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:27 smithi167 ceph-mon[32774]: pgmap v489: 142 pgs: 142 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-02-05T06:43:28.081 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:28.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:28.105 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:28.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:28.206 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:28.234 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:28.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:28.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:28.335 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:28.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:28.363 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:28.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:28.464 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:28.489 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:28.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:28.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:28.558 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:43:28.558 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:43:28.558 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:43:28.558 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:43:28.558 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:43:28.558 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:43:28.564 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:43:28.585 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:43:28.586 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:43:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:43:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:43:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:43:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:43:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:43:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:43:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:43:28.590 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:28.614 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:28.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:28.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:28.692 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'unique_pool_2' created 2024-02-05T06:43:28.716 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:28.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:28.744 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:28.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:28.825 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:28 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/461450661' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-02-05T06:43:28.826 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:28 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1985277686' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:43:28.826 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:28 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/461450661' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-02-05T06:43:28.826 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:28 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1985277686' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:43:28.845 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:28.868 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:28.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:28.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:28.970 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:28.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:28 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/461450661' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-02-05T06:43:28.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:28 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1985277686' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:43:28.997 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:28.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:28.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:29.099 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:29.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:29.123 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:29.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:29.224 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:29.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:29.249 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:29.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:29.349 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:29.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:29.373 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:29.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:29.474 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:29.497 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:29.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:29.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:29.599 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:29.622 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:29.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:29.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:29.704 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.6 to 1 (10000) 2024-02-05T06:43:29.724 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:29.750 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:29.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:29.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:29.852 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:29.878 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:29.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:29.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:29.979 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:29.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:29 smithi167 ceph-mon[32774]: pgmap v490: 142 pgs: 142 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:43:29.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:29 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/461450661' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-02-05T06:43:29.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:29 smithi167 ceph-mon[32774]: osdmap e291: 8 total, 8 up, 8 in 2024-02-05T06:43:29.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:29 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/681476802' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T06:43:29.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:29 smithi167 ceph-mon[32774]: from='client.45242 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T06:43:30.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:30.003 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:30.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:30.104 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:30.127 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:30.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:30.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:30.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:29 smithi028 ceph-mon[37984]: pgmap v490: 142 pgs: 142 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:43:30.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:29 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/461450661' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-02-05T06:43:30.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:29 smithi028 ceph-mon[37984]: osdmap e291: 8 total, 8 up, 8 in 2024-02-05T06:43:30.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:29 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/681476802' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T06:43:30.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:29 smithi028 ceph-mon[37984]: from='client.45242 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T06:43:30.145 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:29 smithi028 ceph-mon[29442]: pgmap v490: 142 pgs: 142 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:43:30.145 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:29 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/461450661' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-02-05T06:43:30.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:29 smithi028 ceph-mon[29442]: osdmap e291: 8 total, 8 up, 8 in 2024-02-05T06:43:30.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:29 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/681476802' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T06:43:30.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:29 smithi028 ceph-mon[29442]: from='client.45242 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T06:43:30.229 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:30.257 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:30.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:30.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:30.359 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:30.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:30.387 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:30.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:30.488 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:30.512 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:30.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:30.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:30.614 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:30.638 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:30.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:30.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:30.739 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:30.768 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:30.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:30.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:30.843 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd reweight 4 1 2024-02-05T06:43:30.870 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:30.909 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:30.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:30.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:30.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:30 smithi028 ceph-mon[29442]: from='client.45242 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-05T06:43:30.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:30 smithi028 ceph-mon[29442]: osdmap e292: 8 total, 8 up, 8 in 2024-02-05T06:43:30.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:30 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:43:30.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:30 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:43:30.961 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:30 smithi028 ceph-mon[37984]: from='client.45242 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-05T06:43:30.961 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:30 smithi028 ceph-mon[37984]: osdmap e292: 8 total, 8 up, 8 in 2024-02-05T06:43:30.961 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:30 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:43:30.961 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:30 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:43:30.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:30 smithi167 ceph-mon[32774]: from='client.45242 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-05T06:43:30.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:30 smithi167 ceph-mon[32774]: osdmap e292: 8 total, 8 up, 8 in 2024-02-05T06:43:30.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:30 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:43:30.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:30 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:43:31.010 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:31.034 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:31.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:31.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:31.136 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:31.159 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:31.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:31.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:31.261 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:31.289 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:31.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:31.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:31.391 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:31.419 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:31.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:31.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:31.507 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:43:31.521 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:31.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-02-05T06:43:31.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:31.546 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:31.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:31.647 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:31.675 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:31.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:31.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:31.777 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:43:31.777 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:31.802 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:31.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:31.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:31.903 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:31.910 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:31 smithi028 ceph-mon[37984]: pgmap v493: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:43:31.910 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:31 smithi028 ceph-mon[37984]: osdmap e293: 8 total, 8 up, 8 in 2024-02-05T06:43:31.910 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:31 smithi028 ceph-mon[37984]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T06:43:31.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:31 smithi028 ceph-mon[29442]: pgmap v493: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:43:31.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:31 smithi028 ceph-mon[29442]: osdmap e293: 8 total, 8 up, 8 in 2024-02-05T06:43:31.911 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:31 smithi028 ceph-mon[29442]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T06:43:31.928 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:31.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:31.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:31.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:31 smithi167 ceph-mon[32774]: pgmap v493: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:43:31.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:31 smithi167 ceph-mon[32774]: osdmap e293: 8 total, 8 up, 8 in 2024-02-05T06:43:31.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:31 smithi167 ceph-mon[32774]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T06:43:32.029 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:32.057 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:32.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:32.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:32.159 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:32.186 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:32.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:32.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:32.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:32.312 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:32.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:32.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:32.414 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:32.425 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:43:32.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:32.442 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:32.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:32.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:32.567 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:32.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:32.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:32.668 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:32.697 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:32.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:32.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:32.717 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-02-05T06:43:32.799 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:32.826 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:32.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:32.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:32.834 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:32 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2597762817' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:43:32.834 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:32 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2597762817' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:43:32.834 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:43:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:43:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:43:32.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:32.952 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:32.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:32.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:32.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:32 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2597762817' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:43:32.983 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:43:32 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:43:33.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:33.077 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:33.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:33.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:33.179 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:33.208 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:33.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:33.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:33.310 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:33.323 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:43:33.337 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:33.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:33.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:33.439 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:33.467 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:33.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:33.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:33.569 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:33.596 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:33.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:33.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:33.697 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:33.698 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 05 06:43:33 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-0[42047]: 2024-02-05T06:43:33.500+0000 7f6d0fa24700 -1 log_channel(cluster) log [ERR] : osd.0 pg[2.0]: reservation requested while still reserved 2024-02-05T06:43:33.724 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:33.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:33.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:33.825 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:33.850 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:33.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:33.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:33.952 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:33.978 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:33.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:33.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:33.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:33 smithi167 ceph-mon[32774]: pgmap v495: 158 pgs: 158 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:43:33.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:33 smithi167 ceph-mon[32774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:43:33.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:33 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2597762817' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:43:33.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:33 smithi167 ceph-mon[32774]: osdmap e294: 8 total, 8 up, 8 in 2024-02-05T06:43:34.080 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:34.106 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:34.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:34.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:34.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:33 smithi028 ceph-mon[29442]: pgmap v495: 158 pgs: 158 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:43:34.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:33 smithi028 ceph-mon[29442]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:43:34.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:33 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2597762817' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:43:34.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:33 smithi028 ceph-mon[29442]: osdmap e294: 8 total, 8 up, 8 in 2024-02-05T06:43:34.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:33 smithi028 ceph-mon[37984]: pgmap v495: 158 pgs: 158 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:43:34.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:33 smithi028 ceph-mon[37984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:43:34.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:33 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2597762817' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:43:34.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:33 smithi028 ceph-mon[37984]: osdmap e294: 8 total, 8 up, 8 in 2024-02-05T06:43:34.207 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:34.231 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:34.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:34.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:34.333 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:34.361 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:34.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:34.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:34.463 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:34.488 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:34.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:34.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:34.590 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:34.614 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:34.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:34.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:34.715 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:34.739 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:34.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:34.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:34.840 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:34.869 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:34.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:34.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:34.907 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:34 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:43:34.907 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:34 smithi028 ceph-mon[29442]: osd.0 pg[2.0]: reservation requested while still reserved 2024-02-05T06:43:34.907 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:34 smithi028 ceph-mon[29442]: 2.0 scrub starts 2024-02-05T06:43:34.907 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:34 smithi028 ceph-mon[29442]: 2.0 scrub ok 2024-02-05T06:43:34.907 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:34 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:43:34.907 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:34 smithi028 ceph-mon[37984]: osd.0 pg[2.0]: reservation requested while still reserved 2024-02-05T06:43:34.907 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:34 smithi028 ceph-mon[37984]: 2.0 scrub starts 2024-02-05T06:43:34.907 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:34 smithi028 ceph-mon[37984]: 2.0 scrub ok 2024-02-05T06:43:34.971 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:34.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:34 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:43:34.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:34 smithi167 ceph-mon[32774]: osd.0 pg[2.0]: reservation requested while still reserved 2024-02-05T06:43:34.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:34 smithi167 ceph-mon[32774]: 2.0 scrub starts 2024-02-05T06:43:34.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:34 smithi167 ceph-mon[32774]: 2.0 scrub ok 2024-02-05T06:43:34.995 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:34.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:34.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:35.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:35.119 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:35.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:35.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:35.221 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:35.249 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:35.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:35.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:35.351 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:35.380 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:35.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:35.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:35.482 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:35.508 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:35.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:35.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:35.609 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:35.635 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:35.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:35.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:35.737 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:35.761 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:35.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:35.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:35.863 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:35.887 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:35.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:35.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:35.895 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:43:35.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:35 smithi028 ceph-mon[29442]: pgmap v497: 158 pgs: 158 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 903 B/s rd, 0 op/s 2024-02-05T06:43:35.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:35 smithi028 ceph-mon[29442]: 3.1c scrub starts 2024-02-05T06:43:35.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:35 smithi028 ceph-mon[29442]: 3.1c scrub ok 2024-02-05T06:43:35.980 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:35 smithi028 ceph-mon[37984]: pgmap v497: 158 pgs: 158 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 903 B/s rd, 0 op/s 2024-02-05T06:43:35.980 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:35 smithi028 ceph-mon[37984]: 3.1c scrub starts 2024-02-05T06:43:35.980 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:35 smithi028 ceph-mon[37984]: 3.1c scrub ok 2024-02-05T06:43:35.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:35 smithi167 ceph-mon[32774]: pgmap v497: 158 pgs: 158 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 903 B/s rd, 0 op/s 2024-02-05T06:43:35.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:35 smithi167 ceph-mon[32774]: 3.1c scrub starts 2024-02-05T06:43:35.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:35 smithi167 ceph-mon[32774]: 3.1c scrub ok 2024-02-05T06:43:35.989 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:36.017 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:36.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:36.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:36.119 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:36.147 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:36.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:36.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:36.249 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:36.273 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:36.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:36.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:36.375 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:36.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:36.403 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:36.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:36.504 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:36.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:36.534 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:36.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:36.635 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:36.662 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:36.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:36.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:36.764 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:36.780 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:43:36.780 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-02-05T06:43:36.780 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-02-05T06:43:36.780 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-02-05T06:43:36.780 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.780 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:192337 7.8 7:118a156b:::benchmark_data_smithi028_77509_object77217:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-02-05T06:43:36.780 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:38.337115+0000", 2024-02-05T06:43:36.780 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 298.43424530300001, 2024-02-05T06:43:36.780 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0591137100000001, 2024-02-05T06:43:36.780 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:43:36.780 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-02-05T06:43:36.780 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:43:36.781 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:43:36.781 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:43:36.781 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 192337 2024-02-05T06:43:36.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.781 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:43:36.781 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.781 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:43:36.781 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.337115+0000", 2024-02-05T06:43:36.781 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.781 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.781 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:43:36.781 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.337115+0000", 2024-02-05T06:43:36.781 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.781 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.782 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:43:36.782 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.337117+0000", 2024-02-05T06:43:36.782 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.73e-06 2024-02-05T06:43:36.782 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.782 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.782 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:43:36.782 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.337122+0000", 2024-02-05T06:43:36.782 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5310000000000001e-06 2024-02-05T06:43:36.782 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.782 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.782 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:43:36.782 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.337133+0000", 2024-02-05T06:43:36.782 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0570999999999999e-05 2024-02-05T06:43:36.782 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.782 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.783 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:43:36.783 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.337141+0000", 2024-02-05T06:43:36.783 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3979999999999999e-06 2024-02-05T06:43:36.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.783 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.783 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.783 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.337204+0000", 2024-02-05T06:43:36.783 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2786999999999994e-05 2024-02-05T06:43:36.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.783 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.783 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:43:36.783 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.337367+0000", 2024-02-05T06:43:36.783 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016232799999999999 2024-02-05T06:43:36.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.783 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.783 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4,7", 2024-02-05T06:43:36.784 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.337495+0000", 2024-02-05T06:43:36.784 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012881599999999999 2024-02-05T06:43:36.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.784 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.784 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.784 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.346118+0000", 2024-02-05T06:43:36.784 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0086228139999999995 2024-02-05T06:43:36.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.784 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.784 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:43:36.784 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.346131+0000", 2024-02-05T06:43:36.784 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2860999999999999e-05 2024-02-05T06:43:36.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.785 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.785 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.785 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.346165+0000", 2024-02-05T06:43:36.785 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4406999999999998e-05 2024-02-05T06:43:36.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.785 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.785 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.785 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.360213+0000", 2024-02-05T06:43:36.785 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014047484000000001 2024-02-05T06:43:36.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.785 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.785 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:43:36.785 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.360224+0000", 2024-02-05T06:43:36.785 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1392999999999999e-05 2024-02-05T06:43:36.786 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.786 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.786 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.786 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.396146+0000", 2024-02-05T06:43:36.786 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.03592211 2024-02-05T06:43:36.786 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.786 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.786 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:43:36.786 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.396229+0000", 2024-02-05T06:43:36.786 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2479999999999996e-05 2024-02-05T06:43:36.786 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.786 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:43:36.786 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.787 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.787 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:192338 7.8 7:16a7efbe:::benchmark_data_smithi028_77509_object77218:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-02-05T06:43:36.787 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:38.337628+0000", 2024-02-05T06:43:36.787 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 298.43373253200002, 2024-02-05T06:43:36.787 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0591757799999999, 2024-02-05T06:43:36.787 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:43:36.787 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-02-05T06:43:36.787 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:43:36.787 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:43:36.787 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:43:36.787 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 192338 2024-02-05T06:43:36.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.787 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:43:36.787 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.788 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:43:36.788 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.337628+0000", 2024-02-05T06:43:36.788 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.788 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.788 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:43:36.788 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.337628+0000", 2024-02-05T06:43:36.788 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.788 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.788 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:43:36.788 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.337629+0000", 2024-02-05T06:43:36.788 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1880000000000001e-06 2024-02-05T06:43:36.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.789 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:43:36.789 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.337634+0000", 2024-02-05T06:43:36.789 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5689999999999998e-06 2024-02-05T06:43:36.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.789 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:43:36.789 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.337643+0000", 2024-02-05T06:43:36.789 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5529999999999998e-06 2024-02-05T06:43:36.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.789 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:43:36.789 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.337651+0000", 2024-02-05T06:43:36.789 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8280000000000003e-06 2024-02-05T06:43:36.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.790 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.790 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.341287+0000", 2024-02-05T06:43:36.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0036361760000000001 2024-02-05T06:43:36.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.790 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:43:36.790 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.341412+0000", 2024-02-05T06:43:36.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000125112 2024-02-05T06:43:36.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.790 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4,7", 2024-02-05T06:43:36.790 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.341535+0000", 2024-02-05T06:43:36.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000122915 2024-02-05T06:43:36.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.790 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.346203+0000", 2024-02-05T06:43:36.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0046678359999999999 2024-02-05T06:43:36.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.346242+0000", 2024-02-05T06:43:36.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9008999999999999e-05 2024-02-05T06:43:36.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:43:36.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.351060+0000", 2024-02-05T06:43:36.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004818363 2024-02-05T06:43:36.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.360255+0000", 2024-02-05T06:43:36.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0091945059999999999 2024-02-05T06:43:36.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.792 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:43:36.792 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.360270+0000", 2024-02-05T06:43:36.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5324000000000001e-05 2024-02-05T06:43:36.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.792 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.792 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.396734+0000", 2024-02-05T06:43:36.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0364635360000001 2024-02-05T06:43:36.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.792 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:43:36.793 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.396804+0000", 2024-02-05T06:43:36.793 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9864999999999996e-05 2024-02-05T06:43:36.793 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.793 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:43:36.793 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.793 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:192347 7.8 7:11cd75fe:::benchmark_data_smithi028_77509_object77227:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-02-05T06:43:36.793 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:38.339579+0000", 2024-02-05T06:43:36.793 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 298.431781785, 2024-02-05T06:43:36.793 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.057352659, 2024-02-05T06:43:36.793 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:43:36.793 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-02-05T06:43:36.793 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:43:36.794 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:43:36.794 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:43:36.794 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 192347 2024-02-05T06:43:36.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.794 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:43:36.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.794 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:43:36.794 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.339579+0000", 2024-02-05T06:43:36.794 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.794 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:43:36.794 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.339579+0000", 2024-02-05T06:43:36.794 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.795 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.795 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:43:36.795 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.339580+0000", 2024-02-05T06:43:36.795 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.497e-06 2024-02-05T06:43:36.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.795 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.795 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:43:36.795 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.339585+0000", 2024-02-05T06:43:36.795 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0839999999999999e-06 2024-02-05T06:43:36.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.795 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.795 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:43:36.795 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.339596+0000", 2024-02-05T06:43:36.795 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0536e-05 2024-02-05T06:43:36.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.796 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:43:36.796 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.339604+0000", 2024-02-05T06:43:36.796 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2220000000000002e-06 2024-02-05T06:43:36.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.796 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.796 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.346356+0000", 2024-02-05T06:43:36.796 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0067518120000000003 2024-02-05T06:43:36.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.796 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:43:36.796 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.346473+0000", 2024-02-05T06:43:36.796 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011735600000000001 2024-02-05T06:43:36.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.797 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4,7", 2024-02-05T06:43:36.797 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.346587+0000", 2024-02-05T06:43:36.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000114293 2024-02-05T06:43:36.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.797 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.797 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.351674+0000", 2024-02-05T06:43:36.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050869640000000002 2024-02-05T06:43:36.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.797 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.797 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.351712+0000", 2024-02-05T06:43:36.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.786e-05 2024-02-05T06:43:36.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.798 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:43:36.798 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.354090+0000", 2024-02-05T06:43:36.798 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002377601 2024-02-05T06:43:36.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.798 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.798 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.360302+0000", 2024-02-05T06:43:36.798 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062116050000000003 2024-02-05T06:43:36.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.798 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:43:36.798 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.360310+0000", 2024-02-05T06:43:36.798 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9209999999999998e-06 2024-02-05T06:43:36.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.799 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.799 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.396838+0000", 2024-02-05T06:43:36.799 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0365271110000001 2024-02-05T06:43:36.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.799 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:43:36.799 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.396931+0000", 2024-02-05T06:43:36.799 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3796999999999997e-05 2024-02-05T06:43:36.799 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.799 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:43:36.799 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.799 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:192353 7.8 7:150e0f66:::benchmark_data_smithi028_77509_object77233:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e162)", 2024-02-05T06:43:36.800 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:38.356556+0000", 2024-02-05T06:43:36.800 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 298.414804796, 2024-02-05T06:43:36.800 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0493035660000001, 2024-02-05T06:43:36.800 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:43:36.800 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:43:36.800 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:43:36.800 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:43:36.800 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:43:36.800 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 192353 2024-02-05T06:43:36.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.800 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:43:36.800 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.800 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:43:36.800 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.356556+0000", 2024-02-05T06:43:36.801 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.801 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.801 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:43:36.801 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.356556+0000", 2024-02-05T06:43:36.801 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.801 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.801 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:43:36.801 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.356557+0000", 2024-02-05T06:43:36.801 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5820000000000001e-06 2024-02-05T06:43:36.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.801 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.801 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:43:36.801 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.356563+0000", 2024-02-05T06:43:36.802 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2580000000000003e-06 2024-02-05T06:43:36.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.802 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:43:36.802 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.356572+0000", 2024-02-05T06:43:36.802 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7389999999999994e-06 2024-02-05T06:43:36.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.802 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:43:36.802 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.356581+0000", 2024-02-05T06:43:36.802 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7080000000000003e-06 2024-02-05T06:43:36.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.802 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.802 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.360338+0000", 2024-02-05T06:43:36.803 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0037565789999999999 2024-02-05T06:43:36.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.803 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:43:36.803 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.360345+0000", 2024-02-05T06:43:36.803 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7190000000000001e-06 2024-02-05T06:43:36.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.803 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.803 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.396962+0000", 2024-02-05T06:43:36.803 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0366170079999999 2024-02-05T06:43:36.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.803 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:43:36.803 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.397086+0000", 2024-02-05T06:43:36.804 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000123294 2024-02-05T06:43:36.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.804 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.804 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4", 2024-02-05T06:43:36.804 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.397186+0000", 2024-02-05T06:43:36.804 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000100381 2024-02-05T06:43:36.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.804 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.804 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:43:36.804 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.405647+0000", 2024-02-05T06:43:36.804 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0084607150000000006 2024-02-05T06:43:36.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.804 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.804 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.805 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.405705+0000", 2024-02-05T06:43:36.805 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8365000000000001e-05 2024-02-05T06:43:36.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.805 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.805 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.805 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.405747+0000", 2024-02-05T06:43:36.805 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1501000000000001e-05 2024-02-05T06:43:36.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.805 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.805 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:43:36.805 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.405799+0000", 2024-02-05T06:43:36.805 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1999000000000001e-05 2024-02-05T06:43:36.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.805 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.806 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:43:36.807 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.405859+0000", 2024-02-05T06:43:36.807 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0717999999999997e-05 2024-02-05T06:43:36.807 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.807 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:43:36.807 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.807 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.807 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:192358 7.8 7:1a63e476:::benchmark_data_smithi028_77509_object77238:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-02-05T06:43:36.807 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:38.366256+0000", 2024-02-05T06:43:36.807 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 298.405104354, 2024-02-05T06:43:36.807 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0455992860000001, 2024-02-05T06:43:36.807 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:43:36.807 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:43:36.808 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:43:36.808 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:43:36.808 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:43:36.808 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 192358 2024-02-05T06:43:36.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.808 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:43:36.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.808 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:43:36.808 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.366256+0000", 2024-02-05T06:43:36.808 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.808 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:43:36.808 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.366256+0000", 2024-02-05T06:43:36.809 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.809 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:43:36.809 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.366264+0000", 2024-02-05T06:43:36.809 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2290000000000003e-06 2024-02-05T06:43:36.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.809 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:43:36.809 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.366270+0000", 2024-02-05T06:43:36.809 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6119999999999998e-06 2024-02-05T06:43:36.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.809 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:43:36.809 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.366279+0000", 2024-02-05T06:43:36.810 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3999999999999998e-06 2024-02-05T06:43:36.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.810 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:43:36.810 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.366286+0000", 2024-02-05T06:43:36.810 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5799999999999997e-06 2024-02-05T06:43:36.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.810 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.810 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.366360+0000", 2024-02-05T06:43:36.810 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3857999999999997e-05 2024-02-05T06:43:36.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.810 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:43:36.810 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.366370+0000", 2024-02-05T06:43:36.811 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8309999999999996e-06 2024-02-05T06:43:36.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.811 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.811 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.400159+0000", 2024-02-05T06:43:36.811 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.033789549 2024-02-05T06:43:36.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.811 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:43:36.811 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.400280+0000", 2024-02-05T06:43:36.812 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012075500000000001 2024-02-05T06:43:36.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.812 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.812 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4", 2024-02-05T06:43:36.812 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.400401+0000", 2024-02-05T06:43:36.812 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000121271 2024-02-05T06:43:36.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.812 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.812 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:43:36.812 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.405872+0000", 2024-02-05T06:43:36.812 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0054704489999999996 2024-02-05T06:43:36.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.812 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.812 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.812 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.405927+0000", 2024-02-05T06:43:36.813 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5216000000000001e-05 2024-02-05T06:43:36.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.813 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.813 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.813 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.411703+0000", 2024-02-05T06:43:36.813 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0057759270000000001 2024-02-05T06:43:36.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.813 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.813 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:43:36.813 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.411804+0000", 2024-02-05T06:43:36.813 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010108299999999999 2024-02-05T06:43:36.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.813 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.813 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:43:36.813 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.411855+0000", 2024-02-05T06:43:36.814 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1526000000000001e-05 2024-02-05T06:43:36.814 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.814 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:43:36.814 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.814 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.814 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:192376 7.8 7:12aca2d8:::benchmark_data_smithi028_77509_object77256:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-02-05T06:43:36.814 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:38.377362+0000", 2024-02-05T06:43:36.814 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 298.39399849699998, 2024-02-05T06:43:36.814 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.041133079, 2024-02-05T06:43:36.814 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:43:36.814 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:43:36.814 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:43:36.814 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:43:36.814 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:43:36.814 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 192376 2024-02-05T06:43:36.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.815 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:43:36.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.815 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:43:36.815 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.377362+0000", 2024-02-05T06:43:36.815 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.815 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:43:36.815 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.377362+0000", 2024-02-05T06:43:36.815 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.816 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:43:36.816 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.377363+0000", 2024-02-05T06:43:36.816 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.13e-06 2024-02-05T06:43:36.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.816 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.816 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:43:36.816 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.377368+0000", 2024-02-05T06:43:36.816 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7870000000000001e-06 2024-02-05T06:43:36.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.816 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.816 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:43:36.816 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.377378+0000", 2024-02-05T06:43:36.816 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0379e-05 2024-02-05T06:43:36.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.816 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.816 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:43:36.817 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.377386+0000", 2024-02-05T06:43:36.817 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7540000000000006e-06 2024-02-05T06:43:36.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.817 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.817 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.377461+0000", 2024-02-05T06:43:36.817 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5334999999999996e-05 2024-02-05T06:43:36.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.817 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:43:36.817 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.377473+0000", 2024-02-05T06:43:36.817 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1280999999999999e-05 2024-02-05T06:43:36.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.817 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.817 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.402958+0000", 2024-02-05T06:43:36.817 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0254855270000001 2024-02-05T06:43:36.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:43:36.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.403055+0000", 2024-02-05T06:43:36.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7029999999999998e-05 2024-02-05T06:43:36.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4", 2024-02-05T06:43:36.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.403163+0000", 2024-02-05T06:43:36.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108173 2024-02-05T06:43:36.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.405984+0000", 2024-02-05T06:43:36.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028201889999999999 2024-02-05T06:43:36.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.819 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:43:36.819 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.411965+0000", 2024-02-05T06:43:36.819 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0059810710000000001 2024-02-05T06:43:36.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.819 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.819 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.418351+0000", 2024-02-05T06:43:36.819 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0063859299999999997 2024-02-05T06:43:36.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.819 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:43:36.819 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.418452+0000", 2024-02-05T06:43:36.819 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101118 2024-02-05T06:43:36.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.820 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:43:36.820 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.418495+0000", 2024-02-05T06:43:36.820 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3374999999999997e-05 2024-02-05T06:43:36.820 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.820 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:43:36.820 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.820 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:192402 7.8 7:167c0a8c:::benchmark_data_smithi028_77509_object77282:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-02-05T06:43:36.820 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:38.406909+0000", 2024-02-05T06:43:36.820 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 298.36445125099999, 2024-02-05T06:43:36.820 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.011859187, 2024-02-05T06:43:36.820 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:43:36.820 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:43:36.820 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:43:36.821 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:43:36.821 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:43:36.821 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 192402 2024-02-05T06:43:36.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.821 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:43:36.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.821 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:43:36.821 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.406909+0000", 2024-02-05T06:43:36.821 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.821 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:43:36.821 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.406909+0000", 2024-02-05T06:43:36.821 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.822 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:43:36.822 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.406912+0000", 2024-02-05T06:43:36.822 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6699999999999998e-06 2024-02-05T06:43:36.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.822 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.822 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:43:36.822 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.406918+0000", 2024-02-05T06:43:36.822 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2190000000000001e-06 2024-02-05T06:43:36.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.822 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.822 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:43:36.822 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.406934+0000", 2024-02-05T06:43:36.822 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6104999999999999e-05 2024-02-05T06:43:36.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.822 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.823 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:43:36.823 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.406943+0000", 2024-02-05T06:43:36.823 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7099999999999996e-06 2024-02-05T06:43:36.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.823 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.823 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.407092+0000", 2024-02-05T06:43:36.823 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000148796 2024-02-05T06:43:36.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.823 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:43:36.823 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.407102+0000", 2024-02-05T06:43:36.823 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9629999999999993e-06 2024-02-05T06:43:36.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.823 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.824 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.406020+0000", 2024-02-05T06:43:36.824 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.99891851600000003 2024-02-05T06:43:36.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.824 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:43:36.824 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.406242+0000", 2024-02-05T06:43:36.824 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000222184 2024-02-05T06:43:36.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.824 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4", 2024-02-05T06:43:36.824 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.406396+0000", 2024-02-05T06:43:36.824 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015325000000000001 2024-02-05T06:43:36.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.824 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.825 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.411950+0000", 2024-02-05T06:43:36.825 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055542550000000001 2024-02-05T06:43:36.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.825 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.825 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:43:36.825 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.418304+0000", 2024-02-05T06:43:36.825 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0063543740000000003 2024-02-05T06:43:36.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.825 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.825 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.825 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.418627+0000", 2024-02-05T06:43:36.825 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00032294300000000001 2024-02-05T06:43:36.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.825 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.825 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:43:36.826 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.418710+0000", 2024-02-05T06:43:36.826 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3205e-05 2024-02-05T06:43:36.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.826 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:43:36.826 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.418768+0000", 2024-02-05T06:43:36.826 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7997000000000001e-05 2024-02-05T06:43:36.826 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.826 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:43:36.826 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.826 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:192404 7.8 7:12218d7a:::benchmark_data_smithi028_77509_object77284:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-02-05T06:43:36.826 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:38.407199+0000", 2024-02-05T06:43:36.826 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 298.36416168199997, 2024-02-05T06:43:36.827 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0118317029999999, 2024-02-05T06:43:36.827 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:43:36.827 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:43:36.827 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:43:36.827 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:43:36.827 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:43:36.827 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 192404 2024-02-05T06:43:36.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.827 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:43:36.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.827 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:43:36.827 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.407199+0000", 2024-02-05T06:43:36.827 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.828 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:43:36.828 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.407199+0000", 2024-02-05T06:43:36.828 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.828 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:43:36.828 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.407200+0000", 2024-02-05T06:43:36.828 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.319e-06 2024-02-05T06:43:36.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.828 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:43:36.828 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.407205+0000", 2024-02-05T06:43:36.828 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.69e-06 2024-02-05T06:43:36.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.829 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:43:36.829 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.407218+0000", 2024-02-05T06:43:36.829 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3698e-05 2024-02-05T06:43:36.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.829 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:43:36.829 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.407226+0000", 2024-02-05T06:43:36.829 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8420000000000005e-06 2024-02-05T06:43:36.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.829 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.829 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.407287+0000", 2024-02-05T06:43:36.829 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0912000000000002e-05 2024-02-05T06:43:36.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.829 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:43:36.829 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.407296+0000", 2024-02-05T06:43:36.830 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1400000000000006e-06 2024-02-05T06:43:36.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.830 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.830 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.408982+0000", 2024-02-05T06:43:36.830 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.001685299 2024-02-05T06:43:36.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.830 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:43:36.830 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.409096+0000", 2024-02-05T06:43:36.830 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000114498 2024-02-05T06:43:36.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.830 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4", 2024-02-05T06:43:36.830 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.409174+0000", 2024-02-05T06:43:36.830 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7632999999999993e-05 2024-02-05T06:43:36.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.831 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.831 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.412018+0000", 2024-02-05T06:43:36.831 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028445969999999999 2024-02-05T06:43:36.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.831 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:43:36.831 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.418510+0000", 2024-02-05T06:43:36.831 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0064918170000000004 2024-02-05T06:43:36.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.831 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.831 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.418824+0000", 2024-02-05T06:43:36.831 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00031352600000000001 2024-02-05T06:43:36.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.831 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:43:36.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:36.832 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:36 smithi028 ceph-mon[29442]: 3.6 scrub starts 2024-02-05T06:43:36.832 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:36 smithi028 ceph-mon[29442]: 3.6 scrub ok 2024-02-05T06:43:36.832 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:36 smithi028 ceph-mon[29442]: 3.5 scrub starts 2024-02-05T06:43:36.832 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:36 smithi028 ceph-mon[29442]: 3.5 scrub ok 2024-02-05T06:43:36.832 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:36 smithi028 ceph-mon[29442]: 4.2 scrub starts 2024-02-05T06:43:36.832 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:36 smithi028 ceph-mon[29442]: 4.2 scrub ok 2024-02-05T06:43:36.832 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:36 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:43:36.832 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:36 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:43:36.832 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:36 smithi028 ceph-mon[29442]: osdmap e295: 8 total, 8 up, 8 in 2024-02-05T06:43:36.833 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:36 smithi028 ceph-mon[37984]: 3.6 scrub starts 2024-02-05T06:43:36.833 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:36 smithi028 ceph-mon[37984]: 3.6 scrub ok 2024-02-05T06:43:36.833 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:36 smithi028 ceph-mon[37984]: 3.5 scrub starts 2024-02-05T06:43:36.833 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:36 smithi028 ceph-mon[37984]: 3.5 scrub ok 2024-02-05T06:43:36.833 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:36 smithi028 ceph-mon[37984]: 4.2 scrub starts 2024-02-05T06:43:36.833 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:36 smithi028 ceph-mon[37984]: 4.2 scrub ok 2024-02-05T06:43:36.833 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:36 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:43:36.833 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:36 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:43:36.833 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:36 smithi028 ceph-mon[37984]: osdmap e295: 8 total, 8 up, 8 in 2024-02-05T06:43:36.834 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:36.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:36.835 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.418989+0000", 2024-02-05T06:43:36.835 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016478900000000001 2024-02-05T06:43:36.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.835 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:43:36.835 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.419030+0000", 2024-02-05T06:43:36.835 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1943000000000001e-05 2024-02-05T06:43:36.835 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.835 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:43:36.836 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.836 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:192455 7.8 7:145ae87a:::benchmark_data_smithi028_77509_object77335:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-02-05T06:43:36.836 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:38.457684+0000", 2024-02-05T06:43:36.836 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 298.31367665099998, 2024-02-05T06:43:36.836 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.96165713600000002, 2024-02-05T06:43:36.836 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:43:36.836 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:43:36.836 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:43:36.836 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:43:36.836 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:43:36.836 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 192455 2024-02-05T06:43:36.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.836 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:43:36.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.837 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:43:36.837 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.457684+0000", 2024-02-05T06:43:36.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.837 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:43:36.837 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.457684+0000", 2024-02-05T06:43:36.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.837 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:43:36.837 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.457685+0000", 2024-02-05T06:43:36.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6819999999999999e-06 2024-02-05T06:43:36.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.837 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:43:36.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.457690+0000", 2024-02-05T06:43:36.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.357e-06 2024-02-05T06:43:36.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.838 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:43:36.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.457705+0000", 2024-02-05T06:43:36.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5199e-05 2024-02-05T06:43:36.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.838 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:43:36.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.457712+0000", 2024-02-05T06:43:36.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6680000000000004e-06 2024-02-05T06:43:36.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.838 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.457777+0000", 2024-02-05T06:43:36.839 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5244000000000001e-05 2024-02-05T06:43:36.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.839 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:43:36.839 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.457788+0000", 2024-02-05T06:43:36.839 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1168e-05 2024-02-05T06:43:36.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.839 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.839 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.412068+0000", 2024-02-05T06:43:36.839 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.95427959600000001 2024-02-05T06:43:36.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.839 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:43:36.839 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.412185+0000", 2024-02-05T06:43:36.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000117547 2024-02-05T06:43:36.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.840 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4", 2024-02-05T06:43:36.840 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.412852+0000", 2024-02-05T06:43:36.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00066681800000000001 2024-02-05T06:43:36.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.840 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:43:36.840 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.418516+0000", 2024-02-05T06:43:36.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0056640140000000002 2024-02-05T06:43:36.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.840 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.840 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.419112+0000", 2024-02-05T06:43:36.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00059562500000000002 2024-02-05T06:43:36.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.841 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.841 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.419152+0000", 2024-02-05T06:43:36.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0311000000000001e-05 2024-02-05T06:43:36.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.841 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:43:36.841 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.419208+0000", 2024-02-05T06:43:36.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6169999999999999e-05 2024-02-05T06:43:36.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.841 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:43:36.841 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.419341+0000", 2024-02-05T06:43:36.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013273700000000001 2024-02-05T06:43:36.841 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.842 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:43:36.842 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.842 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.842 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:192466 7.8 7:1913b8bb:::benchmark_data_smithi028_77509_object77346:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e163)", 2024-02-05T06:43:36.842 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:38.468224+0000", 2024-02-05T06:43:36.842 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 298.30313647999998, 2024-02-05T06:43:36.842 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.95867438299999996, 2024-02-05T06:43:36.842 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:43:36.842 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:43:36.842 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:43:36.842 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:43:36.842 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:43:36.842 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 192466 2024-02-05T06:43:36.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.842 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:43:36.842 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.843 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:43:36.843 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.468224+0000", 2024-02-05T06:43:36.843 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.843 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:43:36.843 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.468224+0000", 2024-02-05T06:43:36.843 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.843 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:43:36.843 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.468225+0000", 2024-02-05T06:43:36.843 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3349999999999999e-06 2024-02-05T06:43:36.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.844 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:43:36.844 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.468230+0000", 2024-02-05T06:43:36.844 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6609999999999999e-06 2024-02-05T06:43:36.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.844 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.844 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:43:36.844 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.468240+0000", 2024-02-05T06:43:36.844 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5599999999999999e-06 2024-02-05T06:43:36.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.844 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.844 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:43:36.844 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.468247+0000", 2024-02-05T06:43:36.844 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5340000000000002e-06 2024-02-05T06:43:36.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.844 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.845 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.845 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.468315+0000", 2024-02-05T06:43:36.845 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8422000000000003e-05 2024-02-05T06:43:36.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.845 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:43:36.845 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:38.468325+0000", 2024-02-05T06:43:36.845 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5750000000000007e-06 2024-02-05T06:43:36.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.845 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.845 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.415713+0000", 2024-02-05T06:43:36.845 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.94738796999999997 2024-02-05T06:43:36.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.846 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:43:36.846 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.415856+0000", 2024-02-05T06:43:36.846 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014318499999999999 2024-02-05T06:43:36.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.846 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4", 2024-02-05T06:43:36.846 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.415981+0000", 2024-02-05T06:43:36.846 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012492699999999999 2024-02-05T06:43:36.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.846 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:43:36.846 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.418783+0000", 2024-02-05T06:43:36.846 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028021959999999999 2024-02-05T06:43:36.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.846 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.419382+0000", 2024-02-05T06:43:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00059883799999999997 2024-02-05T06:43:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.426789+0000", 2024-02-05T06:43:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0074069119999999999 2024-02-05T06:43:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:43:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.426847+0000", 2024-02-05T06:43:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7590000000000003e-05 2024-02-05T06:43:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:43:36.847 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:39.426898+0000", 2024-02-05T06:43:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1678000000000002e-05 2024-02-05T06:43:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:43:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:192607 7.5 7:ac582dbe:::benchmark_data_smithi028_77509_object77487:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-02-05T06:43:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:40.027506+0000", 2024-02-05T06:43:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 296.743854301, 2024-02-05T06:43:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.97921314100000001, 2024-02-05T06:43:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:43:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:43:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:43:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:43:36.848 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:43:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 192607 2024-02-05T06:43:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:43:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:43:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.027506+0000", 2024-02-05T06:43:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:43:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.027506+0000", 2024-02-05T06:43:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:43:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.027508+0000", 2024-02-05T06:43:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.53e-06 2024-02-05T06:43:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:43:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.027513+0000", 2024-02-05T06:43:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.865e-06 2024-02-05T06:43:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:43:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.027524+0000", 2024-02-05T06:43:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1026000000000001e-05 2024-02-05T06:43:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.850 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:43:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.027532+0000", 2024-02-05T06:43:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9689999999999994e-06 2024-02-05T06:43:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.027591+0000", 2024-02-05T06:43:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9491999999999999e-05 2024-02-05T06:43:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:43:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.027601+0000", 2024-02-05T06:43:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.91e-06 2024-02-05T06:43:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.851 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.998983+0000", 2024-02-05T06:43:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.97138245300000003 2024-02-05T06:43:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:43:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.999074+0000", 2024-02-05T06:43:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0660000000000003e-05 2024-02-05T06:43:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,1", 2024-02-05T06:43:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:40.999397+0000", 2024-02-05T06:43:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00032286800000000003 2024-02-05T06:43:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.852 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:43:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.006547+0000", 2024-02-05T06:43:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0071499659999999998 2024-02-05T06:43:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.006605+0000", 2024-02-05T06:43:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7776999999999999e-05 2024-02-05T06:43:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.006642+0000", 2024-02-05T06:43:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7698999999999999e-05 2024-02-05T06:43:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:43:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.006676+0000", 2024-02-05T06:43:36.853 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3389000000000002e-05 2024-02-05T06:43:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:43:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:41.006719+0000", 2024-02-05T06:43:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3537e-05 2024-02-05T06:43:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:43:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:44121 8.d 8:b9ef805b:::benchmark_data_smithi028_105698_object44120:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-02-05T06:43:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:26.024756+0000", 2024-02-05T06:43:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 190.74660487200001, 2024-02-05T06:43:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.171835365, 2024-02-05T06:43:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:43:36.854 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-02-05T06:43:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:43:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:43:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:43:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 44121 2024-02-05T06:43:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:43:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:43:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.024756+0000", 2024-02-05T06:43:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:43:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.024756+0000", 2024-02-05T06:43:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:43:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.024758+0000", 2024-02-05T06:43:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3309999999999998e-06 2024-02-05T06:43:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:43:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.024783+0000", 2024-02-05T06:43:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5316999999999998e-05 2024-02-05T06:43:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:43:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.024799+0000", 2024-02-05T06:43:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6019999999999999e-05 2024-02-05T06:43:36.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:43:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.024807+0000", 2024-02-05T06:43:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.96e-06 2024-02-05T06:43:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.025654+0000", 2024-02-05T06:43:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00084685999999999995 2024-02-05T06:43:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:43:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.025792+0000", 2024-02-05T06:43:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0001383 2024-02-05T06:43:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,6", 2024-02-05T06:43:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.025948+0000", 2024-02-05T06:43:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015603000000000001 2024-02-05T06:43:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:43:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.031906+0000", 2024-02-05T06:43:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0059571420000000003 2024-02-05T06:43:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.033244+0000", 2024-02-05T06:43:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0013384219999999999 2024-02-05T06:43:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.084787+0000", 2024-02-05T06:43:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.051542880999999999 2024-02-05T06:43:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:43:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.084804+0000", 2024-02-05T06:43:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6735000000000002e-05 2024-02-05T06:43:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.196242+0000", 2024-02-05T06:43:36.859 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1114386940000001 2024-02-05T06:43:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:43:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.196591+0000", 2024-02-05T06:43:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00034867299999999997 2024-02-05T06:43:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:43:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:44128 8.d 8:befb3d76:::benchmark_data_smithi028_105698_object44127:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-02-05T06:43:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:26.027705+0000", 2024-02-05T06:43:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 190.743654972, 2024-02-05T06:43:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1711811700000001, 2024-02-05T06:43:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:43:36.860 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-02-05T06:43:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:43:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:43:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:43:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 44128 2024-02-05T06:43:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:43:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:43:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.027705+0000", 2024-02-05T06:43:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:43:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.027705+0000", 2024-02-05T06:43:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:43:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.027708+0000", 2024-02-05T06:43:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.154e-06 2024-02-05T06:43:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:43:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.027748+0000", 2024-02-05T06:43:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0413000000000001e-05 2024-02-05T06:43:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:43:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.027761+0000", 2024-02-05T06:43:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3002e-05 2024-02-05T06:43:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.862 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:43:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.027771+0000", 2024-02-05T06:43:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0006e-05 2024-02-05T06:43:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.027822+0000", 2024-02-05T06:43:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0810999999999999e-05 2024-02-05T06:43:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:43:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.027967+0000", 2024-02-05T06:43:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014541099999999999 2024-02-05T06:43:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,6", 2024-02-05T06:43:36.863 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.028189+0000", 2024-02-05T06:43:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000221449 2024-02-05T06:43:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:43:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.031913+0000", 2024-02-05T06:43:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0037241900000000001 2024-02-05T06:43:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.034610+0000", 2024-02-05T06:43:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026966730000000001 2024-02-05T06:43:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.087426+0000", 2024-02-05T06:43:36.864 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.052816730999999999 2024-02-05T06:43:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:43:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.087438+0000", 2024-02-05T06:43:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2150000000000001e-05 2024-02-05T06:43:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.198710+0000", 2024-02-05T06:43:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1112717679999999 2024-02-05T06:43:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:43:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.198887+0000", 2024-02-05T06:43:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000176412 2024-02-05T06:43:36.865 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:43:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:44136 8.d 8:bad1cf1f:::benchmark_data_smithi028_105698_object44135:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-02-05T06:43:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:26.036745+0000", 2024-02-05T06:43:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 190.73461530500001, 2024-02-05T06:43:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.162420676, 2024-02-05T06:43:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:43:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-02-05T06:43:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:43:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:43:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:43:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 44136 2024-02-05T06:43:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.866 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:43:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:43:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.036745+0000", 2024-02-05T06:43:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:43:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.036745+0000", 2024-02-05T06:43:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:43:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.036747+0000", 2024-02-05T06:43:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2060000000000001e-06 2024-02-05T06:43:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:43:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.036774+0000", 2024-02-05T06:43:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6584000000000002e-05 2024-02-05T06:43:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:43:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.036785+0000", 2024-02-05T06:43:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1241e-05 2024-02-05T06:43:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:43:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.036795+0000", 2024-02-05T06:43:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0049e-05 2024-02-05T06:43:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.038541+0000", 2024-02-05T06:43:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001745398 2024-02-05T06:43:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:43:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.038688+0000", 2024-02-05T06:43:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014750399999999999 2024-02-05T06:43:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,6", 2024-02-05T06:43:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.038852+0000", 2024-02-05T06:43:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000163862 2024-02-05T06:43:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.053723+0000", 2024-02-05T06:43:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014871025 2024-02-05T06:43:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.087471+0000", 2024-02-05T06:43:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.033747498000000001 2024-02-05T06:43:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:43:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.087479+0000", 2024-02-05T06:43:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2179999999999999e-06 2024-02-05T06:43:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.870 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.198934+0000", 2024-02-05T06:43:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1114548980000001 2024-02-05T06:43:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:43:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.199166+0000", 2024-02-05T06:43:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000232193 2024-02-05T06:43:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:43:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:44140 8.d 8:ba35d532:::benchmark_data_smithi028_105698_object44139:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-02-05T06:43:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:26.041058+0000", 2024-02-05T06:43:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 190.73030248000001, 2024-02-05T06:43:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.15833663, 2024-02-05T06:43:36.871 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:43:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-02-05T06:43:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:43:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:43:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:43:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 44140 2024-02-05T06:43:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:43:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:43:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.041058+0000", 2024-02-05T06:43:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:43:36.872 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.041058+0000", 2024-02-05T06:43:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:43:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.041066+0000", 2024-02-05T06:43:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4009999999999997e-06 2024-02-05T06:43:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:43:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.041102+0000", 2024-02-05T06:43:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5281999999999999e-05 2024-02-05T06:43:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:43:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.041115+0000", 2024-02-05T06:43:36.873 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2978e-05 2024-02-05T06:43:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:43:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.041122+0000", 2024-02-05T06:43:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3679999999999996e-06 2024-02-05T06:43:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.041203+0000", 2024-02-05T06:43:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1396999999999993e-05 2024-02-05T06:43:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:43:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.041365+0000", 2024-02-05T06:43:36.874 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016111700000000001 2024-02-05T06:43:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,6", 2024-02-05T06:43:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.041531+0000", 2024-02-05T06:43:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016638699999999999 2024-02-05T06:43:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.053969+0000", 2024-02-05T06:43:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012437963 2024-02-05T06:43:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.087508+0000", 2024-02-05T06:43:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.033539592 2024-02-05T06:43:36.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:43:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.087516+0000", 2024-02-05T06:43:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6450000000000005e-06 2024-02-05T06:43:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.199211+0000", 2024-02-05T06:43:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1116952549999999 2024-02-05T06:43:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:43:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.199395+0000", 2024-02-05T06:43:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018324500000000001 2024-02-05T06:43:36.876 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:43:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:44144 8.d 8:b5da4aa8:::benchmark_data_smithi028_105698_object44143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-02-05T06:43:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:26.043109+0000", 2024-02-05T06:43:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 190.72825129899999, 2024-02-05T06:43:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1598225790000001, 2024-02-05T06:43:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:43:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-02-05T06:43:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:43:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:43:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:43:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 44144 2024-02-05T06:43:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.877 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:43:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:43:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.043109+0000", 2024-02-05T06:43:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:43:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.043109+0000", 2024-02-05T06:43:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:43:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.043111+0000", 2024-02-05T06:43:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.249e-06 2024-02-05T06:43:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.878 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:43:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.043152+0000", 2024-02-05T06:43:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0367999999999997e-05 2024-02-05T06:43:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:43:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.043165+0000", 2024-02-05T06:43:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2916e-05 2024-02-05T06:43:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:43:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.043174+0000", 2024-02-05T06:43:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8429999999999999e-06 2024-02-05T06:43:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.043558+0000", 2024-02-05T06:43:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000384192 2024-02-05T06:43:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:43:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.044036+0000", 2024-02-05T06:43:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00047839300000000003 2024-02-05T06:43:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,6", 2024-02-05T06:43:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.044200+0000", 2024-02-05T06:43:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000163394 2024-02-05T06:43:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.880 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.054020+0000", 2024-02-05T06:43:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0098206839999999997 2024-02-05T06:43:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.087548+0000", 2024-02-05T06:43:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.033528096 2024-02-05T06:43:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:43:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.087557+0000", 2024-02-05T06:43:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5580000000000001e-06 2024-02-05T06:43:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.202724+0000", 2024-02-05T06:43:36.881 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1151670520000001 2024-02-05T06:43:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:43:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.202932+0000", 2024-02-05T06:43:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000207834 2024-02-05T06:43:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:43:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:106734 8.d 8:bde070b3:::benchmark_data_smithi028_105698_object17459:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e248)", 2024-02-05T06:43:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:41:43.395831+0000", 2024-02-05T06:43:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 113.37552936900001, 2024-02-05T06:43:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.160102811, 2024-02-05T06:43:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:43:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:43:36.882 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:43:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:43:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:43:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 106734 2024-02-05T06:43:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:43:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:43:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.395831+0000", 2024-02-05T06:43:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:43:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.395831+0000", 2024-02-05T06:43:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.883 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:43:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.395832+0000", 2024-02-05T06:43:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3510000000000001e-06 2024-02-05T06:43:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:43:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.395837+0000", 2024-02-05T06:43:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4440000000000001e-06 2024-02-05T06:43:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:43:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.395847+0000", 2024-02-05T06:43:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6639999999999993e-06 2024-02-05T06:43:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.884 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:43:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.395856+0000", 2024-02-05T06:43:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3870000000000001e-06 2024-02-05T06:43:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.408462+0000", 2024-02-05T06:43:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012606334 2024-02-05T06:43:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:43:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.408473+0000", 2024-02-05T06:43:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0638e-05 2024-02-05T06:43:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.885 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.545505+0000", 2024-02-05T06:43:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.137032077 2024-02-05T06:43:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:43:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.545633+0000", 2024-02-05T06:43:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012815 2024-02-05T06:43:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6,7", 2024-02-05T06:43:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.545759+0000", 2024-02-05T06:43:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000126119 2024-02-05T06:43:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:43:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.555623+0000", 2024-02-05T06:43:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0098633609999999993 2024-02-05T06:43:36.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.555667+0000", 2024-02-05T06:43:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4403000000000002e-05 2024-02-05T06:43:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.555733+0000", 2024-02-05T06:43:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5809999999999995e-05 2024-02-05T06:43:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.555798+0000", 2024-02-05T06:43:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4864999999999996e-05 2024-02-05T06:43:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.887 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:43:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.555837+0000", 2024-02-05T06:43:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8858e-05 2024-02-05T06:43:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:43:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.555934+0000", 2024-02-05T06:43:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7349999999999995e-05 2024-02-05T06:43:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:43:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:106747 8.d 8:b5d9bc03:::benchmark_data_smithi028_105698_object17472:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e248)", 2024-02-05T06:43:36.888 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:41:43.395869+0000", 2024-02-05T06:43:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 113.375491259, 2024-02-05T06:43:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1681894660000001, 2024-02-05T06:43:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:43:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:43:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:43:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:43:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:43:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 106747 2024-02-05T06:43:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:43:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:43:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.395869+0000", 2024-02-05T06:43:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.889 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:43:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.395869+0000", 2024-02-05T06:43:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:43:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.395870+0000", 2024-02-05T06:43:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0900000000000002e-07 2024-02-05T06:43:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:43:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.395872+0000", 2024-02-05T06:43:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5689999999999999e-06 2024-02-05T06:43:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.890 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:43:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.395878+0000", 2024-02-05T06:43:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9120000000000003e-06 2024-02-05T06:43:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:43:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.395885+0000", 2024-02-05T06:43:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7440000000000001e-06 2024-02-05T06:43:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.408501+0000", 2024-02-05T06:43:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012616412 2024-02-05T06:43:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:43:36.891 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.408511+0000", 2024-02-05T06:43:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5119999999999997e-06 2024-02-05T06:43:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.547957+0000", 2024-02-05T06:43:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.139445646 2024-02-05T06:43:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:43:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.548030+0000", 2024-02-05T06:43:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3137000000000001e-05 2024-02-05T06:43:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.892 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6,7", 2024-02-05T06:43:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.548098+0000", 2024-02-05T06:43:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8009000000000006e-05 2024-02-05T06:43:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:43:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.555749+0000", 2024-02-05T06:43:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0076516969999999998 2024-02-05T06:43:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.555999+0000", 2024-02-05T06:43:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024915499999999998 2024-02-05T06:43:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.893 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.560970+0000", 2024-02-05T06:43:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049710570000000001 2024-02-05T06:43:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.563813+0000", 2024-02-05T06:43:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028429739999999998 2024-02-05T06:43:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:43:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.563917+0000", 2024-02-05T06:43:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000104664 2024-02-05T06:43:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:43:36.894 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.564059+0000", 2024-02-05T06:43:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000141469 2024-02-05T06:43:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:43:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:106752 8.4 8:24c4f5c6:::benchmark_data_smithi028_105698_object17477:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e248)", 2024-02-05T06:43:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:41:43.395891+0000", 2024-02-05T06:43:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 113.375469395, 2024-02-05T06:43:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1495485460000001, 2024-02-05T06:43:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:43:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:43:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:43:36.895 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:43:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:43:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 106752 2024-02-05T06:43:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:43:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:43:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.395891+0000", 2024-02-05T06:43:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:43:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.395891+0000", 2024-02-05T06:43:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:43:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.395892+0000", 2024-02-05T06:43:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7999999999999996e-07 2024-02-05T06:43:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:43:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.395894+0000", 2024-02-05T06:43:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.046e-06 2024-02-05T06:43:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:43:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.395899+0000", 2024-02-05T06:43:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1699999999999996e-06 2024-02-05T06:43:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:43:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.395904+0000", 2024-02-05T06:43:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1699999999999996e-06 2024-02-05T06:43:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.402967+0000", 2024-02-05T06:43:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0070626220000000002 2024-02-05T06:43:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:43:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.402992+0000", 2024-02-05T06:43:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5732999999999999e-05 2024-02-05T06:43:36.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.899 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.899 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.899 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.536323+0000", 2024-02-05T06:43:36.899 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1333312090000001 2024-02-05T06:43:36.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.899 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.899 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:43:36.899 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.538501+0000", 2024-02-05T06:43:36.899 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021775890000000002 2024-02-05T06:43:36.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.899 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.899 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 4,5,6", 2024-02-05T06:43:36.899 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.538626+0000", 2024-02-05T06:43:36.899 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012469 2024-02-05T06:43:36.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.543594+0000", 2024-02-05T06:43:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049686549999999998 2024-02-05T06:43:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:43:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.543612+0000", 2024-02-05T06:43:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7867999999999999e-05 2024-02-05T06:43:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.543638+0000", 2024-02-05T06:43:36.900 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5477999999999999e-05 2024-02-05T06:43:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.545291+0000", 2024-02-05T06:43:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0016534970000000001 2024-02-05T06:43:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:43:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.545328+0000", 2024-02-05T06:43:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6973e-05 2024-02-05T06:43:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:43:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.545440+0000", 2024-02-05T06:43:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111366 2024-02-05T06:43:36.901 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:43:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:106759 8.4 8:2ba68c35:::benchmark_data_smithi028_105698_object17484:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e248)", 2024-02-05T06:43:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:41:43.395911+0000", 2024-02-05T06:43:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 113.375448977, 2024-02-05T06:43:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1542194889999999, 2024-02-05T06:43:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:43:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:43:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:43:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:43:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:43:36.902 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 106759 2024-02-05T06:43:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:43:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:43:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.395911+0000", 2024-02-05T06:43:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:43:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.395911+0000", 2024-02-05T06:43:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:43:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:43:36.903 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.395912+0000", 2024-02-05T06:43:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.06e-07 2024-02-05T06:43:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:43:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.395914+0000", 2024-02-05T06:43:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1490000000000001e-06 2024-02-05T06:43:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:43:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.395919+0000", 2024-02-05T06:43:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1039999999999998e-06 2024-02-05T06:43:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:43:36.904 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.395925+0000", 2024-02-05T06:43:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5300000000000004e-06 2024-02-05T06:43:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.403024+0000", 2024-02-05T06:43:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0070989590000000002 2024-02-05T06:43:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:43:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.403031+0000", 2024-02-05T06:43:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6739999999999997e-06 2024-02-05T06:43:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:43:36.905 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.541203+0000", 2024-02-05T06:43:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1381717769999999 2024-02-05T06:43:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:43:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.541334+0000", 2024-02-05T06:43:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013034099999999999 2024-02-05T06:43:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 4,5,6", 2024-02-05T06:43:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.541468+0000", 2024-02-05T06:43:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013429300000000001 2024-02-05T06:43:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:43:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.545232+0000", 2024-02-05T06:43:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0037640579999999998 2024-02-05T06:43:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.546808+0000", 2024-02-05T06:43:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0015756870000000001 2024-02-05T06:43:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.547708+0000", 2024-02-05T06:43:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00090085600000000001 2024-02-05T06:43:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:43:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.549770+0000", 2024-02-05T06:43:36.907 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002061107 2024-02-05T06:43:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:43:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.549941+0000", 2024-02-05T06:43:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017191600000000001 2024-02-05T06:43:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:43:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:43:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:43:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.550131+0000", 2024-02-05T06:43:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018953199999999999 2024-02-05T06:43:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:43:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.908 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:43:36.909 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:43:36.909 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:43:36.935 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:36.962 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:36.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:36.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:36.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:36 smithi167 ceph-mon[32774]: 3.6 scrub starts 2024-02-05T06:43:36.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:36 smithi167 ceph-mon[32774]: 3.6 scrub ok 2024-02-05T06:43:36.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:36 smithi167 ceph-mon[32774]: 3.5 scrub starts 2024-02-05T06:43:36.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:36 smithi167 ceph-mon[32774]: 3.5 scrub ok 2024-02-05T06:43:36.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:36 smithi167 ceph-mon[32774]: 4.2 scrub starts 2024-02-05T06:43:36.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:36 smithi167 ceph-mon[32774]: 4.2 scrub ok 2024-02-05T06:43:36.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:36 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:43:36.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:36 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:43:36.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:36 smithi167 ceph-mon[32774]: osdmap e295: 8 total, 8 up, 8 in 2024-02-05T06:43:37.064 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:37.089 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:37.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:37.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:37.191 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:37.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:37.216 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:37.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:37.317 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:37.340 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:37.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:37.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:37.354 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T06:43:37.442 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:37.465 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:37.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:37.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:37.567 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:37.591 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:37.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:37.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:37.693 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:37.717 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:37.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:37.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:37.819 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:37.843 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:37.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:37.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:37.945 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:37.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:37 smithi028 ceph-mon[29442]: 6.16 scrub starts 2024-02-05T06:43:37.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:37 smithi028 ceph-mon[29442]: 6.16 scrub ok 2024-02-05T06:43:37.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:37 smithi028 ceph-mon[29442]: pgmap v499: 158 pgs: 158 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-05T06:43:37.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:37 smithi028 ceph-mon[29442]: 5.19 scrub starts 2024-02-05T06:43:37.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:37 smithi028 ceph-mon[29442]: 5.19 scrub ok 2024-02-05T06:43:37.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:37 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3494416357' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T06:43:37.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:37 smithi028 ceph-mon[29442]: from='client.45254 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T06:43:37.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:37 smithi028 ceph-mon[37984]: 6.16 scrub starts 2024-02-05T06:43:37.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:37 smithi028 ceph-mon[37984]: 6.16 scrub ok 2024-02-05T06:43:37.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:37 smithi028 ceph-mon[37984]: pgmap v499: 158 pgs: 158 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-05T06:43:37.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:37 smithi028 ceph-mon[37984]: 5.19 scrub starts 2024-02-05T06:43:37.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:37 smithi028 ceph-mon[37984]: 5.19 scrub ok 2024-02-05T06:43:37.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:37 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3494416357' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T06:43:37.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:37 smithi028 ceph-mon[37984]: from='client.45254 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T06:43:37.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:37.972 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:37.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:37.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:37 smithi167 ceph-mon[32774]: 6.16 scrub starts 2024-02-05T06:43:37.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:37 smithi167 ceph-mon[32774]: 6.16 scrub ok 2024-02-05T06:43:37.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:37 smithi167 ceph-mon[32774]: pgmap v499: 158 pgs: 158 active+clean; 582 KiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-05T06:43:37.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:37 smithi167 ceph-mon[32774]: 5.19 scrub starts 2024-02-05T06:43:37.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:37 smithi167 ceph-mon[32774]: 5.19 scrub ok 2024-02-05T06:43:37.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:37 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3494416357' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T06:43:37.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:37 smithi167 ceph-mon[32774]: from='client.45254 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T06:43:38.073 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:38.098 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:38.098 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.4 to 1 (10000) 2024-02-05T06:43:38.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:38.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:38.100 INFO:teuthology.orchestra.run.smithi028.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-02-05T06:43:38.201 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:38.225 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:38.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:38.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:38.236 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-05T06:43:38.327 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:38.352 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:38.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:38.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:38.454 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:38.477 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:38.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:38.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:38.578 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:38.603 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:38.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:38.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:38.705 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:38.729 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:38.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:38.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:38.831 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:38.854 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:38.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:38.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:38.956 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:38.981 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:38.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:38.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:38.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:38 smithi167 ceph-mon[32774]: 4.17 scrub starts 2024-02-05T06:43:38.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:38 smithi167 ceph-mon[32774]: 4.17 scrub ok 2024-02-05T06:43:38.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:38 smithi167 ceph-mon[32774]: 5.1d scrub starts 2024-02-05T06:43:38.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:38 smithi167 ceph-mon[32774]: 5.1d scrub ok 2024-02-05T06:43:38.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:38 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3085222362' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T06:43:38.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:38 smithi167 ceph-mon[32774]: from='client.45254 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T06:43:38.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:38 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3085222362' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-05T06:43:38.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:38 smithi167 ceph-mon[32774]: osdmap e296: 8 total, 8 up, 8 in 2024-02-05T06:43:39.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:38 smithi028 ceph-mon[29442]: 4.17 scrub starts 2024-02-05T06:43:39.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:38 smithi028 ceph-mon[29442]: 4.17 scrub ok 2024-02-05T06:43:39.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:38 smithi028 ceph-mon[29442]: 5.1d scrub starts 2024-02-05T06:43:39.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:38 smithi028 ceph-mon[29442]: 5.1d scrub ok 2024-02-05T06:43:39.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:38 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3085222362' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T06:43:39.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:38 smithi028 ceph-mon[29442]: from='client.45254 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T06:43:39.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:38 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3085222362' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-05T06:43:39.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:38 smithi028 ceph-mon[29442]: osdmap e296: 8 total, 8 up, 8 in 2024-02-05T06:43:39.041 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:38 smithi028 ceph-mon[37984]: 4.17 scrub starts 2024-02-05T06:43:39.041 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:38 smithi028 ceph-mon[37984]: 4.17 scrub ok 2024-02-05T06:43:39.041 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:38 smithi028 ceph-mon[37984]: 5.1d scrub starts 2024-02-05T06:43:39.041 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:38 smithi028 ceph-mon[37984]: 5.1d scrub ok 2024-02-05T06:43:39.041 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:38 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3085222362' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T06:43:39.041 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:38 smithi028 ceph-mon[37984]: from='client.45254 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T06:43:39.041 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:38 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3085222362' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-05T06:43:39.041 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:38 smithi028 ceph-mon[37984]: osdmap e296: 8 total, 8 up, 8 in 2024-02-05T06:43:39.083 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:39.108 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:39.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:39.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:39.209 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:39.233 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:39.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:39.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:39.335 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:39.360 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:39.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:39.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:39.461 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:39.484 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:39.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:39.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:39.586 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:39.611 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:39.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:39.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:39.713 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:39.739 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:39.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:39.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:39.841 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:39.853 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd reweight 7 1 2024-02-05T06:43:39.866 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:39.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:39.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:39.968 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:39.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:39 smithi167 ceph-mon[32774]: pgmap v501: 158 pgs: 158 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-02-05T06:43:39.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:39 smithi167 ceph-mon[32774]: 6.1f scrub starts 2024-02-05T06:43:39.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:39 smithi167 ceph-mon[32774]: 6.1f scrub ok 2024-02-05T06:43:39.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:39 smithi167 ceph-mon[32774]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T06:43:39.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:39.993 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:39.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:40.067 DEBUG:teuthology.orchestra.run.smithi028:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-02-05T06:43:40.069 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-05T06:43:40.069 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-05T06:43:40.077 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:39 smithi028 ceph-mon[29442]: pgmap v501: 158 pgs: 158 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-02-05T06:43:40.077 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:39 smithi028 ceph-mon[29442]: 6.1f scrub starts 2024-02-05T06:43:40.077 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:39 smithi028 ceph-mon[29442]: 6.1f scrub ok 2024-02-05T06:43:40.077 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:39 smithi028 ceph-mon[29442]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T06:43:40.077 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:39 smithi028 ceph-mon[37984]: pgmap v501: 158 pgs: 158 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 32 op/s 2024-02-05T06:43:40.078 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:39 smithi028 ceph-mon[37984]: 6.1f scrub starts 2024-02-05T06:43:40.078 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:39 smithi028 ceph-mon[37984]: 6.1f scrub ok 2024-02-05T06:43:40.078 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:39 smithi028 ceph-mon[37984]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T06:43:40.094 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:40.116 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:hints = 1 2024-02-05T06:43:40.116 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-05T06:43:40.117 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Object prefix: benchmark_data_smithi028_129096 2024-02-05T06:43:40.117 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T06:43:40.117 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 0 0 0 0 0 0 - 0 2024-02-05T06:43:40.121 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:40.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:40.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:40.223 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:40.265 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:40.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:40.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:40.309 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:43:40.367 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:40.400 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:40.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:40.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:40.502 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:40.543 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:40.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:40.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:40.644 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:40.680 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:40.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:40.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:40.782 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:40.824 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:40.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:40.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:40.928 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:40.963 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:40.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:40.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:41.064 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:41.097 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:41.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:41.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:41.116 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1 16 3609 3593 224.546 224.562 0.00341572 0.0044384 2024-02-05T06:43:41.199 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:41.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:41.236 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:41.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:41.249 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:41 smithi028 ceph-mon[37984]: 4.11 scrub starts 2024-02-05T06:43:41.249 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:41 smithi028 ceph-mon[37984]: 4.11 scrub ok 2024-02-05T06:43:41.249 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:41 smithi028 ceph-mon[37984]: 3.1f scrub starts 2024-02-05T06:43:41.249 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:41 smithi028 ceph-mon[37984]: 3.1f scrub ok 2024-02-05T06:43:41.250 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:41 smithi028 ceph-mon[37984]: pgmap v502: 158 pgs: 158 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2024-02-05T06:43:41.250 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:41 smithi028 ceph-mon[29442]: 4.11 scrub starts 2024-02-05T06:43:41.250 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:41 smithi028 ceph-mon[29442]: 4.11 scrub ok 2024-02-05T06:43:41.250 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:41 smithi028 ceph-mon[29442]: 3.1f scrub starts 2024-02-05T06:43:41.250 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:41 smithi028 ceph-mon[29442]: 3.1f scrub ok 2024-02-05T06:43:41.250 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:41 smithi028 ceph-mon[29442]: pgmap v502: 158 pgs: 158 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2024-02-05T06:43:41.260 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:43:41.337 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:41.373 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:41.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:41.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:41.474 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:41.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:41 smithi167 ceph-mon[32774]: 4.11 scrub starts 2024-02-05T06:43:41.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:41 smithi167 ceph-mon[32774]: 4.11 scrub ok 2024-02-05T06:43:41.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:41 smithi167 ceph-mon[32774]: 3.1f scrub starts 2024-02-05T06:43:41.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:41 smithi167 ceph-mon[32774]: 3.1f scrub ok 2024-02-05T06:43:41.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:41 smithi167 ceph-mon[32774]: pgmap v502: 158 pgs: 158 active+clean; 582 KiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2024-02-05T06:43:41.516 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:41.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:41.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:41.618 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:41.658 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:41.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:41.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:41.760 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:41.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:41.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:41.906 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:41.948 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:41.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:41.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:42.051 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:42.090 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:42.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:42.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:42.116 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2 16 6936 6920 216.233 207.938 0.00424524 0.00461442 2024-02-05T06:43:42.192 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:42.221 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:42.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:42.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:42.323 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:42.359 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:42.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:42.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:42.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:42 smithi028 ceph-mon[37984]: 2.2 scrub starts 2024-02-05T06:43:42.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:42 smithi028 ceph-mon[37984]: 2.2 scrub ok 2024-02-05T06:43:42.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:42 smithi028 ceph-mon[37984]: 6.6 scrub starts 2024-02-05T06:43:42.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:42 smithi028 ceph-mon[37984]: 6.6 scrub ok 2024-02-05T06:43:42.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:42 smithi028 ceph-mon[37984]: 5.a scrub starts 2024-02-05T06:43:42.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:42 smithi028 ceph-mon[37984]: 5.a scrub ok 2024-02-05T06:43:42.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:42 smithi028 ceph-mon[37984]: 6.1c scrub starts 2024-02-05T06:43:42.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:42 smithi028 ceph-mon[37984]: 6.1c scrub ok 2024-02-05T06:43:42.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:42 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:43:42.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:42 smithi028 ceph-mon[29442]: 2.2 scrub starts 2024-02-05T06:43:42.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:42 smithi028 ceph-mon[29442]: 2.2 scrub ok 2024-02-05T06:43:42.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:42 smithi028 ceph-mon[29442]: 6.6 scrub starts 2024-02-05T06:43:42.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:42 smithi028 ceph-mon[29442]: 6.6 scrub ok 2024-02-05T06:43:42.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:42 smithi028 ceph-mon[29442]: 5.a scrub starts 2024-02-05T06:43:42.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:42 smithi028 ceph-mon[29442]: 5.a scrub ok 2024-02-05T06:43:42.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:42 smithi028 ceph-mon[29442]: 6.1c scrub starts 2024-02-05T06:43:42.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:42 smithi028 ceph-mon[29442]: 6.1c scrub ok 2024-02-05T06:43:42.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:42 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:43:42.461 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:42.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:42 smithi167 ceph-mon[32774]: 2.2 scrub starts 2024-02-05T06:43:42.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:42 smithi167 ceph-mon[32774]: 2.2 scrub ok 2024-02-05T06:43:42.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:42 smithi167 ceph-mon[32774]: 6.6 scrub starts 2024-02-05T06:43:42.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:42 smithi167 ceph-mon[32774]: 6.6 scrub ok 2024-02-05T06:43:42.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:42 smithi167 ceph-mon[32774]: 5.a scrub starts 2024-02-05T06:43:42.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:42 smithi167 ceph-mon[32774]: 5.a scrub ok 2024-02-05T06:43:42.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:42 smithi167 ceph-mon[32774]: 6.1c scrub starts 2024-02-05T06:43:42.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:42 smithi167 ceph-mon[32774]: 6.1c scrub ok 2024-02-05T06:43:42.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:42 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:43:42.499 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:42.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:42.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:42.600 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:42.627 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:42.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:42.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:42.729 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:42.753 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:43:42 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:43:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:43:42.764 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:42.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:42.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:42.867 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:42.902 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:42.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:42.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:43.004 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:43.038 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:43.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:43.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:43.117 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3 16 9922 9906 206.358 186.625 0.00463509 0.00483988 2024-02-05T06:43:43.140 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:43.174 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:43.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:43.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:43.233 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:43:42 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:43:43.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:43 smithi167 ceph-mon[32774]: 6.0 scrub starts 2024-02-05T06:43:43.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:43 smithi167 ceph-mon[32774]: 6.0 scrub ok 2024-02-05T06:43:43.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:43 smithi167 ceph-mon[32774]: pgmap v503: 158 pgs: 158 active+clean; 350 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 55 KiB/s rd, 44 MiB/s wr, 789 op/s 2024-02-05T06:43:43.276 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:43.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:43.315 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:43.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:43.353 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:43 smithi028 ceph-mon[29442]: 6.0 scrub starts 2024-02-05T06:43:43.353 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:43 smithi028 ceph-mon[29442]: 6.0 scrub ok 2024-02-05T06:43:43.353 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:43 smithi028 ceph-mon[29442]: pgmap v503: 158 pgs: 158 active+clean; 350 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 55 KiB/s rd, 44 MiB/s wr, 789 op/s 2024-02-05T06:43:43.353 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:43 smithi028 ceph-mon[37984]: 6.0 scrub starts 2024-02-05T06:43:43.353 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:43 smithi028 ceph-mon[37984]: 6.0 scrub ok 2024-02-05T06:43:43.354 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:43 smithi028 ceph-mon[37984]: pgmap v503: 158 pgs: 158 active+clean; 350 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 55 KiB/s rd, 44 MiB/s wr, 789 op/s 2024-02-05T06:43:43.417 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:43.451 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:43.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:43.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:43.552 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:43.585 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:43.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:43.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:43.687 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:43.720 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:43.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:43.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:43.822 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:43.856 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:43.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:43.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:43.958 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:43.992 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:43.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:43.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:44.095 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:44.117 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4 16 12736 12720 198.733 175.875 0.00794858 0.00502269 2024-02-05T06:43:44.128 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:44.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:44.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:44.229 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:44.241 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:43:44.241 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:43:44.241 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:43:44.241 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:43:44.241 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:43:44.241 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:43:44.241 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:43:44.242 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:43:44.242 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:43:44.242 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:43:44.242 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:43:44.242 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:43:44.242 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:43:44.242 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:43:44.242 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:43:44.242 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:43:44.265 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:44.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:44.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:44.367 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:44.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:44.402 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:44.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:44.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:44 smithi028 ceph-mon[29442]: 6.c scrub starts 2024-02-05T06:43:44.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:44 smithi028 ceph-mon[29442]: 6.c scrub ok 2024-02-05T06:43:44.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:44 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:43:44.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:44 smithi028 ceph-mon[29442]: 5.1f scrub starts 2024-02-05T06:43:44.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:44 smithi028 ceph-mon[29442]: 5.1f scrub ok 2024-02-05T06:43:44.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:44 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3757954707' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:43:44.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:44 smithi028 ceph-mon[29442]: 4.8 scrub starts 2024-02-05T06:43:44.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:44 smithi028 ceph-mon[29442]: 4.8 scrub ok 2024-02-05T06:43:44.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:44 smithi028 ceph-mon[29442]: 5.4 scrub starts 2024-02-05T06:43:44.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:44 smithi028 ceph-mon[29442]: 5.4 scrub ok 2024-02-05T06:43:44.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:44 smithi028 ceph-mon[37984]: 6.c scrub starts 2024-02-05T06:43:44.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:44 smithi028 ceph-mon[37984]: 6.c scrub ok 2024-02-05T06:43:44.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:44 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:43:44.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:44 smithi028 ceph-mon[37984]: 5.1f scrub starts 2024-02-05T06:43:44.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:44 smithi028 ceph-mon[37984]: 5.1f scrub ok 2024-02-05T06:43:44.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:44 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3757954707' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:43:44.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:44 smithi028 ceph-mon[37984]: 4.8 scrub starts 2024-02-05T06:43:44.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:44 smithi028 ceph-mon[37984]: 4.8 scrub ok 2024-02-05T06:43:44.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:44 smithi028 ceph-mon[37984]: 5.4 scrub starts 2024-02-05T06:43:44.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:44 smithi028 ceph-mon[37984]: 5.4 scrub ok 2024-02-05T06:43:44.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:44 smithi167 ceph-mon[32774]: 6.c scrub starts 2024-02-05T06:43:44.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:44 smithi167 ceph-mon[32774]: 6.c scrub ok 2024-02-05T06:43:44.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:44 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:43:44.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:44 smithi167 ceph-mon[32774]: 5.1f scrub starts 2024-02-05T06:43:44.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:44 smithi167 ceph-mon[32774]: 5.1f scrub ok 2024-02-05T06:43:44.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:44 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3757954707' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:43:44.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:44 smithi167 ceph-mon[32774]: 4.8 scrub starts 2024-02-05T06:43:44.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:44 smithi167 ceph-mon[32774]: 4.8 scrub ok 2024-02-05T06:43:44.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:44 smithi167 ceph-mon[32774]: 5.4 scrub starts 2024-02-05T06:43:44.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:44 smithi167 ceph-mon[32774]: 5.4 scrub ok 2024-02-05T06:43:44.503 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:44.538 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:44.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:44.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:44.640 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:44.675 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:44.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:44.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:44.777 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:44.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:44.814 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:44.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:44.915 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:44.948 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:44.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:44.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:45.050 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:45.085 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:45.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:45.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:45.117 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5 16 15723 15707 196.321 186.688 0.00495 0.00508579 2024-02-05T06:43:45.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:45.407 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:45 smithi028 ceph-mon[29442]: 3.8 scrub starts 2024-02-05T06:43:45.407 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:45 smithi028 ceph-mon[29442]: 3.8 scrub ok 2024-02-05T06:43:45.407 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:45 smithi028 ceph-mon[29442]: 3.18 scrub starts 2024-02-05T06:43:45.407 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:45 smithi028 ceph-mon[29442]: 3.18 scrub ok 2024-02-05T06:43:45.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:45 smithi028 ceph-mon[29442]: pgmap v504: 158 pgs: 158 active+clean; 350 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 53 KiB/s rd, 42 MiB/s wr, 763 op/s 2024-02-05T06:43:45.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:45 smithi028 ceph-mon[29442]: 5.8 scrub starts 2024-02-05T06:43:45.408 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:45 smithi028 ceph-mon[29442]: 5.8 scrub ok 2024-02-05T06:43:45.408 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:45 smithi028 ceph-mon[37984]: 3.8 scrub starts 2024-02-05T06:43:45.408 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:45 smithi028 ceph-mon[37984]: 3.8 scrub ok 2024-02-05T06:43:45.408 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:45 smithi028 ceph-mon[37984]: 3.18 scrub starts 2024-02-05T06:43:45.408 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:45 smithi028 ceph-mon[37984]: 3.18 scrub ok 2024-02-05T06:43:45.408 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:45 smithi028 ceph-mon[37984]: pgmap v504: 158 pgs: 158 active+clean; 350 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 53 KiB/s rd, 42 MiB/s wr, 763 op/s 2024-02-05T06:43:45.408 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:45 smithi028 ceph-mon[37984]: 5.8 scrub starts 2024-02-05T06:43:45.408 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:45 smithi028 ceph-mon[37984]: 5.8 scrub ok 2024-02-05T06:43:45.476 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:45.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:45.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:45.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:45 smithi167 ceph-mon[32774]: 3.8 scrub starts 2024-02-05T06:43:45.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:45 smithi167 ceph-mon[32774]: 3.8 scrub ok 2024-02-05T06:43:45.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:45 smithi167 ceph-mon[32774]: 3.18 scrub starts 2024-02-05T06:43:45.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:45 smithi167 ceph-mon[32774]: 3.18 scrub ok 2024-02-05T06:43:45.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:45 smithi167 ceph-mon[32774]: pgmap v504: 158 pgs: 158 active+clean; 350 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 53 KiB/s rd, 42 MiB/s wr, 763 op/s 2024-02-05T06:43:45.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:45 smithi167 ceph-mon[32774]: 5.8 scrub starts 2024-02-05T06:43:45.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:45 smithi167 ceph-mon[32774]: 5.8 scrub ok 2024-02-05T06:43:45.578 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:45.610 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:45.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:45.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:45.712 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:45.751 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:45.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:45.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:45.852 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:45.876 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:45.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:45.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:45.977 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:46.013 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:46.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:46.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:46.115 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:46.118 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 6 16 17935 17919 186.633 138.25 0.00341194 0.00535008 2024-02-05T06:43:46.154 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:46.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:46.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:46.256 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:46.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:46.298 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:46.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:46.399 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:46.429 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:46.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:46.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:46.530 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:46.566 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:46.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:46.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:46.668 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:46.706 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:46.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:46.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:46.807 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:46.842 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:46.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:46.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:46.944 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:46.979 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:46.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:46.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:47.081 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:47.116 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:47.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:47.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:47.117 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 7 16 20470 20454 182.603 158.438 0.0037997 0.00547056 2024-02-05T06:43:47.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:47.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:47.277 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:47.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:47.378 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:47.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:47.420 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:47.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:47.457 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:43:47.521 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:47.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:47.598 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:47.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:47.610 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:47 smithi028 ceph-mon[37984]: 4.1e scrub starts 2024-02-05T06:43:47.610 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:47 smithi028 ceph-mon[37984]: 4.1e scrub ok 2024-02-05T06:43:47.610 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:47 smithi028 ceph-mon[37984]: pgmap v505: 158 pgs: 158 active+clean; 737 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 44 KiB/s rd, 74 MiB/s wr, 1.25k op/s 2024-02-05T06:43:47.610 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:47 smithi028 ceph-mon[37984]: 6.f scrub starts 2024-02-05T06:43:47.610 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:47 smithi028 ceph-mon[37984]: 6.f scrub ok 2024-02-05T06:43:47.610 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:47 smithi028 ceph-mon[37984]: 3.0 scrub starts 2024-02-05T06:43:47.610 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:47 smithi028 ceph-mon[37984]: 3.0 scrub ok 2024-02-05T06:43:47.610 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:47 smithi028 ceph-mon[37984]: 4.0 scrub starts 2024-02-05T06:43:47.611 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:47 smithi028 ceph-mon[37984]: 4.0 scrub ok 2024-02-05T06:43:47.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:47 smithi028 ceph-mon[29442]: 4.1e scrub starts 2024-02-05T06:43:47.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:47 smithi028 ceph-mon[29442]: 4.1e scrub ok 2024-02-05T06:43:47.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:47 smithi028 ceph-mon[29442]: pgmap v505: 158 pgs: 158 active+clean; 737 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 44 KiB/s rd, 74 MiB/s wr, 1.25k op/s 2024-02-05T06:43:47.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:47 smithi028 ceph-mon[29442]: 6.f scrub starts 2024-02-05T06:43:47.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:47 smithi028 ceph-mon[29442]: 6.f scrub ok 2024-02-05T06:43:47.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:47 smithi028 ceph-mon[29442]: 3.0 scrub starts 2024-02-05T06:43:47.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:47 smithi028 ceph-mon[29442]: 3.0 scrub ok 2024-02-05T06:43:47.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:47 smithi028 ceph-mon[29442]: 4.0 scrub starts 2024-02-05T06:43:47.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:47 smithi028 ceph-mon[29442]: 4.0 scrub ok 2024-02-05T06:43:47.699 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:47.728 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:47.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:47.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:47.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:47 smithi167 ceph-mon[32774]: 4.1e scrub starts 2024-02-05T06:43:47.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:47 smithi167 ceph-mon[32774]: 4.1e scrub ok 2024-02-05T06:43:47.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:47 smithi167 ceph-mon[32774]: pgmap v505: 158 pgs: 158 active+clean; 737 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 44 KiB/s rd, 74 MiB/s wr, 1.25k op/s 2024-02-05T06:43:47.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:47 smithi167 ceph-mon[32774]: 6.f scrub starts 2024-02-05T06:43:47.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:47 smithi167 ceph-mon[32774]: 6.f scrub ok 2024-02-05T06:43:47.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:47 smithi167 ceph-mon[32774]: 3.0 scrub starts 2024-02-05T06:43:47.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:47 smithi167 ceph-mon[32774]: 3.0 scrub ok 2024-02-05T06:43:47.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:47 smithi167 ceph-mon[32774]: 4.0 scrub starts 2024-02-05T06:43:47.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:47 smithi167 ceph-mon[32774]: 4.0 scrub ok 2024-02-05T06:43:47.830 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:47.852 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:43:47.852 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:43:47.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:47.880 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:47.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:47.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:48.020 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:48.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:48.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:48.117 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 8 16 22318 22302 174.214 115.5 0.00309608 0.00573299 2024-02-05T06:43:48.122 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:48.159 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:48.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:48.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:48.261 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:48.299 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:48.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:48.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:48.401 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:48.431 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-02-05T06:43:48.435 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:48.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:48.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:48.537 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:48.571 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:48.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:48.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:48.673 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:48.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:48 smithi028 ceph-mon[29442]: 4.13 scrub starts 2024-02-05T06:43:48.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:48 smithi028 ceph-mon[29442]: 4.13 scrub ok 2024-02-05T06:43:48.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:48 smithi028 ceph-mon[29442]: 1.8 scrub starts 2024-02-05T06:43:48.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:48 smithi028 ceph-mon[29442]: 1.8 scrub ok 2024-02-05T06:43:48.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:48 smithi028 ceph-mon[29442]: 1.9 scrub starts 2024-02-05T06:43:48.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:48 smithi028 ceph-mon[29442]: 1.9 scrub ok 2024-02-05T06:43:48.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:48 smithi028 ceph-mon[29442]: 6.b scrub starts 2024-02-05T06:43:48.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:48 smithi028 ceph-mon[29442]: 6.b scrub ok 2024-02-05T06:43:48.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:48 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/234357572' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:43:48.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:48 smithi028 ceph-mon[29442]: from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:43:48.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:48 smithi028 ceph-mon[37984]: 4.13 scrub starts 2024-02-05T06:43:48.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:48 smithi028 ceph-mon[37984]: 4.13 scrub ok 2024-02-05T06:43:48.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:48 smithi028 ceph-mon[37984]: 1.8 scrub starts 2024-02-05T06:43:48.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:48 smithi028 ceph-mon[37984]: 1.8 scrub ok 2024-02-05T06:43:48.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:48 smithi028 ceph-mon[37984]: 1.9 scrub starts 2024-02-05T06:43:48.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:48 smithi028 ceph-mon[37984]: 1.9 scrub ok 2024-02-05T06:43:48.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:48 smithi028 ceph-mon[37984]: 6.b scrub starts 2024-02-05T06:43:48.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:48 smithi028 ceph-mon[37984]: 6.b scrub ok 2024-02-05T06:43:48.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:48 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/234357572' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:43:48.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:48 smithi028 ceph-mon[37984]: from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:43:48.714 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:48.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:48.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:48.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:48 smithi167 ceph-mon[32774]: 4.13 scrub starts 2024-02-05T06:43:48.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:48 smithi167 ceph-mon[32774]: 4.13 scrub ok 2024-02-05T06:43:48.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:48 smithi167 ceph-mon[32774]: 1.8 scrub starts 2024-02-05T06:43:48.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:48 smithi167 ceph-mon[32774]: 1.8 scrub ok 2024-02-05T06:43:48.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:48 smithi167 ceph-mon[32774]: 1.9 scrub starts 2024-02-05T06:43:48.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:48 smithi167 ceph-mon[32774]: 1.9 scrub ok 2024-02-05T06:43:48.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:48 smithi167 ceph-mon[32774]: 6.b scrub starts 2024-02-05T06:43:48.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:48 smithi167 ceph-mon[32774]: 6.b scrub ok 2024-02-05T06:43:48.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:48 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/234357572' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:43:48.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:48 smithi167 ceph-mon[32774]: from='client.25315 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:43:48.816 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:48.851 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:43:48.856 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:48.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:48.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:48.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:49.001 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:49.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:49.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:49.103 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:49.119 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 9 16 24596 24580 170.661 142.375 0.00716052 0.00585247 2024-02-05T06:43:49.161 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:49.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:49.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:49.263 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:49.302 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:49.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:49.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:49.404 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:49.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:49.445 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:49.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:49.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:49 smithi028 ceph-mon[29442]: pgmap v506: 158 pgs: 158 active+clean; 1.2 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 31 KiB/s rd, 118 MiB/s wr, 1.95k op/s 2024-02-05T06:43:49.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:49 smithi028 ceph-mon[29442]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:43:49.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:49 smithi028 ceph-mon[29442]: Cluster is now healthy 2024-02-05T06:43:49.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:49 smithi028 ceph-mon[29442]: from='client.25315 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:43:49.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:49 smithi028 ceph-mon[29442]: osdmap e297: 8 total, 8 up, 8 in 2024-02-05T06:43:49.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:49 smithi028 ceph-mon[29442]: 6.12 deep-scrub starts 2024-02-05T06:43:49.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:49 smithi028 ceph-mon[29442]: 6.12 deep-scrub ok 2024-02-05T06:43:49.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:49 smithi028 ceph-mon[37984]: pgmap v506: 158 pgs: 158 active+clean; 1.2 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 31 KiB/s rd, 118 MiB/s wr, 1.95k op/s 2024-02-05T06:43:49.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:49 smithi028 ceph-mon[37984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:43:49.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:49 smithi028 ceph-mon[37984]: Cluster is now healthy 2024-02-05T06:43:49.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:49 smithi028 ceph-mon[37984]: from='client.25315 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:43:49.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:49 smithi028 ceph-mon[37984]: osdmap e297: 8 total, 8 up, 8 in 2024-02-05T06:43:49.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:49 smithi028 ceph-mon[37984]: 6.12 deep-scrub starts 2024-02-05T06:43:49.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:49 smithi028 ceph-mon[37984]: 6.12 deep-scrub ok 2024-02-05T06:43:49.546 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:49.587 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:49.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:49.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:49.689 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:49.729 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:49.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:49.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:49.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:49 smithi167 ceph-mon[32774]: pgmap v506: 158 pgs: 158 active+clean; 1.2 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 31 KiB/s rd, 118 MiB/s wr, 1.95k op/s 2024-02-05T06:43:49.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:49 smithi167 ceph-mon[32774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:43:49.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:49 smithi167 ceph-mon[32774]: Cluster is now healthy 2024-02-05T06:43:49.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:49 smithi167 ceph-mon[32774]: from='client.25315 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:43:49.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:49 smithi167 ceph-mon[32774]: osdmap e297: 8 total, 8 up, 8 in 2024-02-05T06:43:49.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:49 smithi167 ceph-mon[32774]: 6.12 deep-scrub starts 2024-02-05T06:43:49.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:49 smithi167 ceph-mon[32774]: 6.12 deep-scrub ok 2024-02-05T06:43:49.832 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:49.872 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:49.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:49.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:49.924 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-05T06:43:49.974 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:50.006 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:50.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:50.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:50.107 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:50.119 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 10 16 26874 26858 167.832 142.375 0.0297166 0.00594527 2024-02-05T06:43:50.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:50.133 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:50.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:50.234 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:50.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:50.269 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:50.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:50.370 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:50.405 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:50.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:50.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:50.506 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:50.547 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:50.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:50.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:50.649 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:50.683 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:50.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:50.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:50.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:50 smithi028 ceph-mon[37984]: 3.17 deep-scrub starts 2024-02-05T06:43:50.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:50 smithi028 ceph-mon[37984]: 3.17 deep-scrub ok 2024-02-05T06:43:50.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:50 smithi028 ceph-mon[37984]: 5.7 scrub starts 2024-02-05T06:43:50.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:50 smithi028 ceph-mon[37984]: 5.7 scrub ok 2024-02-05T06:43:50.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:50 smithi028 ceph-mon[37984]: osdmap e298: 8 total, 8 up, 8 in 2024-02-05T06:43:50.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:50 smithi028 ceph-mon[29442]: 3.17 deep-scrub starts 2024-02-05T06:43:50.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:50 smithi028 ceph-mon[29442]: 3.17 deep-scrub ok 2024-02-05T06:43:50.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:50 smithi028 ceph-mon[29442]: 5.7 scrub starts 2024-02-05T06:43:50.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:50 smithi028 ceph-mon[29442]: 5.7 scrub ok 2024-02-05T06:43:50.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:50 smithi028 ceph-mon[29442]: osdmap e298: 8 total, 8 up, 8 in 2024-02-05T06:43:50.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:50 smithi167 ceph-mon[32774]: 3.17 deep-scrub starts 2024-02-05T06:43:50.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:50 smithi167 ceph-mon[32774]: 3.17 deep-scrub ok 2024-02-05T06:43:50.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:50 smithi167 ceph-mon[32774]: 5.7 scrub starts 2024-02-05T06:43:50.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:50 smithi167 ceph-mon[32774]: 5.7 scrub ok 2024-02-05T06:43:50.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:50 smithi167 ceph-mon[32774]: osdmap e298: 8 total, 8 up, 8 in 2024-02-05T06:43:50.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:50.824 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:50.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:50.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:50.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:50.965 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:50.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:50.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:51.067 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:51.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:51.103 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:51.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:51.118 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 11 16 28932 28916 164.267 128.625 0.00746831 0.00608181 2024-02-05T06:43:51.205 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:51.242 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:51.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:51.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:51.344 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:51.378 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:51.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:51.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:51.480 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:51.516 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:51.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:51.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:51.619 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:51.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:51.663 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:51.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:51.680 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:51 smithi028 ceph-mon[37984]: 4.1a scrub starts 2024-02-05T06:43:51.681 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:51 smithi028 ceph-mon[37984]: 4.1a scrub ok 2024-02-05T06:43:51.681 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:51 smithi028 ceph-mon[37984]: pgmap v509: 158 pgs: 158 active+clean; 1.2 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-02-05T06:43:51.681 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:51 smithi028 ceph-mon[37984]: 3.4 scrub starts 2024-02-05T06:43:51.681 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:51 smithi028 ceph-mon[37984]: 3.4 scrub ok 2024-02-05T06:43:51.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:51 smithi028 ceph-mon[29442]: 4.1a scrub starts 2024-02-05T06:43:51.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:51 smithi028 ceph-mon[29442]: 4.1a scrub ok 2024-02-05T06:43:51.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:51 smithi028 ceph-mon[29442]: pgmap v509: 158 pgs: 158 active+clean; 1.2 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-02-05T06:43:51.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:51 smithi028 ceph-mon[29442]: 3.4 scrub starts 2024-02-05T06:43:51.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:51 smithi028 ceph-mon[29442]: 3.4 scrub ok 2024-02-05T06:43:51.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:51 smithi167 ceph-mon[32774]: 4.1a scrub starts 2024-02-05T06:43:51.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:51 smithi167 ceph-mon[32774]: 4.1a scrub ok 2024-02-05T06:43:51.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:51 smithi167 ceph-mon[32774]: pgmap v509: 158 pgs: 158 active+clean; 1.2 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-02-05T06:43:51.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:51 smithi167 ceph-mon[32774]: 3.4 scrub starts 2024-02-05T06:43:51.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:51 smithi167 ceph-mon[32774]: 3.4 scrub ok 2024-02-05T06:43:51.765 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:51.798 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:51.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:51.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:51.900 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:51.935 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:51.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:51.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:52.011 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:43:52.037 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:52.071 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:52.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:52.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:52.119 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 12 15 31163 31148 162.199 139.5 0.00669911 0.00615829 2024-02-05T06:43:52.172 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:52.216 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:52.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:52.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:52.318 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:52.351 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:52.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:52.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:52.453 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:52.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:52.492 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:52.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:52.593 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:52.628 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:52.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:52.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:52.687 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:43:52 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:43:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:43:52.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:52 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:43:52.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:52 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:43:52.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:52 smithi028 ceph-mon[29442]: 4.4 scrub starts 2024-02-05T06:43:52.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:52 smithi028 ceph-mon[29442]: 4.4 scrub ok 2024-02-05T06:43:52.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:52 smithi028 ceph-mon[29442]: 4.10 scrub starts 2024-02-05T06:43:52.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:52 smithi028 ceph-mon[29442]: 4.10 scrub ok 2024-02-05T06:43:52.688 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:52 smithi028 ceph-mon[29442]: osdmap e299: 8 total, 8 up, 8 in 2024-02-05T06:43:52.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:52 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:43:52.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:52 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:43:52.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:52 smithi028 ceph-mon[37984]: 4.4 scrub starts 2024-02-05T06:43:52.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:52 smithi028 ceph-mon[37984]: 4.4 scrub ok 2024-02-05T06:43:52.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:52 smithi028 ceph-mon[37984]: 4.10 scrub starts 2024-02-05T06:43:52.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:52 smithi028 ceph-mon[37984]: 4.10 scrub ok 2024-02-05T06:43:52.688 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:52 smithi028 ceph-mon[37984]: osdmap e299: 8 total, 8 up, 8 in 2024-02-05T06:43:52.731 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:52.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:52 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:43:52.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:52 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:43:52.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:52 smithi167 ceph-mon[32774]: 4.4 scrub starts 2024-02-05T06:43:52.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:52 smithi167 ceph-mon[32774]: 4.4 scrub ok 2024-02-05T06:43:52.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:52 smithi167 ceph-mon[32774]: 4.10 scrub starts 2024-02-05T06:43:52.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:52 smithi167 ceph-mon[32774]: 4.10 scrub ok 2024-02-05T06:43:52.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:52 smithi167 ceph-mon[32774]: osdmap e299: 8 total, 8 up, 8 in 2024-02-05T06:43:52.770 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:52.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:52.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:52.872 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:52.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:52.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:53.015 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:53.054 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:53.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:53.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:53.119 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 13 16 33289 33273 159.934 132.812 0.00707429 0.00624738 2024-02-05T06:43:53.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:53.196 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:53.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:53.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:53.233 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:43:52 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:43:53.298 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:53.333 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:53.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:53.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:53.435 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:53.469 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:53.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:53.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:53.570 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:53.580 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T06:43:53.661 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:53.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:53.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:53.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:53 smithi028 ceph-mon[29442]: 4.b scrub starts 2024-02-05T06:43:53.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:53 smithi028 ceph-mon[29442]: 4.b scrub ok 2024-02-05T06:43:53.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:53 smithi028 ceph-mon[29442]: pgmap v511: 158 pgs: 158 active+clean; 1.8 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 193 MiB/s wr, 3.08k op/s 2024-02-05T06:43:53.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:53 smithi028 ceph-mon[37984]: 4.b scrub starts 2024-02-05T06:43:53.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:53 smithi028 ceph-mon[37984]: 4.b scrub ok 2024-02-05T06:43:53.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:53 smithi028 ceph-mon[37984]: pgmap v511: 158 pgs: 158 active+clean; 1.8 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 193 MiB/s wr, 3.08k op/s 2024-02-05T06:43:53.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:53 smithi167 ceph-mon[32774]: 4.b scrub starts 2024-02-05T06:43:53.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:53 smithi167 ceph-mon[32774]: 4.b scrub ok 2024-02-05T06:43:53.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:53 smithi167 ceph-mon[32774]: pgmap v511: 158 pgs: 158 active+clean; 1.8 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 193 MiB/s wr, 3.08k op/s 2024-02-05T06:43:53.763 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:53.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:53.810 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:53.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:53.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:53.938 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:43:53.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:53.952 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:53.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:54.053 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:54.085 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:54.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:54.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:54.121 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 14 16 35133 35117 156.728 115.25 0.00626703 0.00637372 2024-02-05T06:43:54.187 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:54.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:54.219 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:54.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:54.320 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:54.363 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:54.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:54.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:54.465 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:54.511 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:54.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:54.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:54.612 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:54.661 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:54.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:54.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:54.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:54 smithi028 ceph-mon[29442]: 4.18 scrub starts 2024-02-05T06:43:54.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:54 smithi028 ceph-mon[29442]: 4.18 scrub ok 2024-02-05T06:43:54.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:54 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:43:54.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:54 smithi028 ceph-mon[29442]: 6.19 scrub starts 2024-02-05T06:43:54.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:54 smithi028 ceph-mon[29442]: 6.19 scrub ok 2024-02-05T06:43:54.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:54 smithi028 ceph-mon[29442]: 3.1e scrub starts 2024-02-05T06:43:54.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:54 smithi028 ceph-mon[29442]: 3.1e scrub ok 2024-02-05T06:43:54.683 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:54 smithi028 ceph-mon[37984]: 4.18 scrub starts 2024-02-05T06:43:54.683 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:54 smithi028 ceph-mon[37984]: 4.18 scrub ok 2024-02-05T06:43:54.683 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:54 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:43:54.683 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:54 smithi028 ceph-mon[37984]: 6.19 scrub starts 2024-02-05T06:43:54.683 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:54 smithi028 ceph-mon[37984]: 6.19 scrub ok 2024-02-05T06:43:54.683 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:54 smithi028 ceph-mon[37984]: 3.1e scrub starts 2024-02-05T06:43:54.684 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:54 smithi028 ceph-mon[37984]: 3.1e scrub ok 2024-02-05T06:43:54.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:54 smithi167 ceph-mon[32774]: 4.18 scrub starts 2024-02-05T06:43:54.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:54 smithi167 ceph-mon[32774]: 4.18 scrub ok 2024-02-05T06:43:54.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:54 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:43:54.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:54 smithi167 ceph-mon[32774]: 6.19 scrub starts 2024-02-05T06:43:54.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:54 smithi167 ceph-mon[32774]: 6.19 scrub ok 2024-02-05T06:43:54.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:54 smithi167 ceph-mon[32774]: 3.1e scrub starts 2024-02-05T06:43:54.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:54 smithi167 ceph-mon[32774]: 3.1e scrub ok 2024-02-05T06:43:54.763 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:54.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:54.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:54.910 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:54.950 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:54.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:54.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:55.052 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:55.085 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:55.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:55.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.7 to 1 (10000) 2024-02-05T06:43:55.121 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 15 16 36984 36968 153.989 115.688 0.00469496 0.0064856 2024-02-05T06:43:55.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:55.226 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:55.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:55.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:55.328 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:55.362 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:55.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:55.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:55.465 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:55.501 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:55.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:55.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:55.603 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:55.634 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:55 smithi028 ceph-mon[37984]: 5.1c scrub starts 2024-02-05T06:43:55.634 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:55 smithi028 ceph-mon[37984]: 5.1c scrub ok 2024-02-05T06:43:55.635 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:55 smithi028 ceph-mon[37984]: pgmap v512: 158 pgs: 158 active+clean; 1.8 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 113 MiB/s wr, 1.81k op/s 2024-02-05T06:43:55.635 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:55 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1882914693' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T06:43:55.635 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:55 smithi028 ceph-mon[37984]: 5.1a scrub starts 2024-02-05T06:43:55.635 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:55 smithi028 ceph-mon[37984]: 5.1a scrub ok 2024-02-05T06:43:55.635 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:55 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1882914693' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-05T06:43:55.635 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:55 smithi028 ceph-mon[37984]: osdmap e300: 8 total, 8 up, 8 in 2024-02-05T06:43:55.635 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:55 smithi028 ceph-mon[29442]: 5.1c scrub starts 2024-02-05T06:43:55.635 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:55 smithi028 ceph-mon[29442]: 5.1c scrub ok 2024-02-05T06:43:55.635 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:55 smithi028 ceph-mon[29442]: pgmap v512: 158 pgs: 158 active+clean; 1.8 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 113 MiB/s wr, 1.81k op/s 2024-02-05T06:43:55.635 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:55 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1882914693' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T06:43:55.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:55 smithi028 ceph-mon[29442]: 5.1a scrub starts 2024-02-05T06:43:55.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:55 smithi028 ceph-mon[29442]: 5.1a scrub ok 2024-02-05T06:43:55.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:55 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1882914693' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-05T06:43:55.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:55 smithi028 ceph-mon[29442]: osdmap e300: 8 total, 8 up, 8 in 2024-02-05T06:43:55.637 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:55.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:55.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:55.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:55 smithi167 ceph-mon[32774]: 5.1c scrub starts 2024-02-05T06:43:55.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:55 smithi167 ceph-mon[32774]: 5.1c scrub ok 2024-02-05T06:43:55.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:55 smithi167 ceph-mon[32774]: pgmap v512: 158 pgs: 158 active+clean; 1.8 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 852 B/s rd, 113 MiB/s wr, 1.81k op/s 2024-02-05T06:43:55.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:55 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1882914693' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T06:43:55.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:55 smithi167 ceph-mon[32774]: 5.1a scrub starts 2024-02-05T06:43:55.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:55 smithi167 ceph-mon[32774]: 5.1a scrub ok 2024-02-05T06:43:55.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:55 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1882914693' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-05T06:43:55.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:55 smithi167 ceph-mon[32774]: osdmap e300: 8 total, 8 up, 8 in 2024-02-05T06:43:55.738 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:55.777 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:55.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:55.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:55.855 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:43:55.880 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:55.901 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:43:55.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:55.913 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:55.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:55.942 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:43:55.988 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:43:56.014 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:56.014 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:43:56.034 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:43:56.036 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:56.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:56.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:56.068 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:43:56.111 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:43:56.115 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:43:56.116 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:43:56.116 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:43:56.116 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:43:56.116 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:43:56.116 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:43:56.116 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:43:56.116 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:43:56.121 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 16 16 38826 38810 151.559 115.125 0.015238 0.00659235 2024-02-05T06:43:56.138 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:56.178 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:56.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:56.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:56.280 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:56.333 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:56.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:56.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:56.435 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:56.471 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:56.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:56.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:56.573 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:56.608 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:56.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:56.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:56.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:56 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/799285329' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:43:56.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:56 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/799285329' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:43:56.710 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:56.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:56 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/799285329' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:43:56.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:56.746 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:56.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:56.847 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:56.878 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:56.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:56.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:56.898 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd reweight 5 1 2024-02-05T06:43:56.980 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:57.023 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:57.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:57.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:57.121 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 17 16 40743 40727 149.692 119.812 0.00962671 0.00667394 2024-02-05T06:43:57.125 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:57.165 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:57.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:57.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:57.267 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:57.305 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:57.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:57.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:57.407 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:43:57.445 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:57.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:57.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:43:57.547 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:57.592 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:57.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:57.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:57.696 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:57.739 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:57.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:57.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:57.841 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:57.874 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:57.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:57.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:57.888 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:57 smithi028 ceph-mon[29442]: pgmap v514: 158 pgs: 158 active+clean; 2.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.6 KiB/s rd, 151 MiB/s wr, 2.42k op/s 2024-02-05T06:43:57.888 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:57 smithi028 ceph-mon[29442]: 3.1d scrub starts 2024-02-05T06:43:57.888 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:57 smithi028 ceph-mon[29442]: 3.1d scrub ok 2024-02-05T06:43:57.888 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:57 smithi028 ceph-mon[29442]: 3.f scrub starts 2024-02-05T06:43:57.888 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:57 smithi028 ceph-mon[29442]: 3.f scrub ok 2024-02-05T06:43:57.888 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:57 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:43:57.888 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:57 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:43:57.888 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:57 smithi028 ceph-mon[29442]: 4.19 scrub starts 2024-02-05T06:43:57.888 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:57 smithi028 ceph-mon[29442]: 4.19 scrub ok 2024-02-05T06:43:57.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:57 smithi028 ceph-mon[37984]: pgmap v514: 158 pgs: 158 active+clean; 2.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.6 KiB/s rd, 151 MiB/s wr, 2.42k op/s 2024-02-05T06:43:57.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:57 smithi028 ceph-mon[37984]: 3.1d scrub starts 2024-02-05T06:43:57.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:57 smithi028 ceph-mon[37984]: 3.1d scrub ok 2024-02-05T06:43:57.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:57 smithi028 ceph-mon[37984]: 3.f scrub starts 2024-02-05T06:43:57.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:57 smithi028 ceph-mon[37984]: 3.f scrub ok 2024-02-05T06:43:57.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:57 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:43:57.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:57 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:43:57.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:57 smithi028 ceph-mon[37984]: 4.19 scrub starts 2024-02-05T06:43:57.889 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:57 smithi028 ceph-mon[37984]: 4.19 scrub ok 2024-02-05T06:43:57.976 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:57.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:57 smithi167 ceph-mon[32774]: pgmap v514: 158 pgs: 158 active+clean; 2.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.6 KiB/s rd, 151 MiB/s wr, 2.42k op/s 2024-02-05T06:43:57.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:57 smithi167 ceph-mon[32774]: 3.1d scrub starts 2024-02-05T06:43:57.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:57 smithi167 ceph-mon[32774]: 3.1d scrub ok 2024-02-05T06:43:57.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:57 smithi167 ceph-mon[32774]: 3.f scrub starts 2024-02-05T06:43:57.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:57 smithi167 ceph-mon[32774]: 3.f scrub ok 2024-02-05T06:43:57.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:57 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:43:57.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:57 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:43:57.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:57 smithi167 ceph-mon[32774]: 4.19 scrub starts 2024-02-05T06:43:57.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:57 smithi167 ceph-mon[32774]: 4.19 scrub ok 2024-02-05T06:43:58.013 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:58.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:58.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:58.115 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:58.121 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 18 16 42615 42599 147.876 117 0.00695978 0.00675754 2024-02-05T06:43:58.150 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:58.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:58.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:58.206 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:43:58.252 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:58.283 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:58.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:58.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:58.385 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:43:58.420 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:58.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:58.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:43:58.523 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:43:58.556 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:58.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:58.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:43:58.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:43:58.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:58.700 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:58.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:43:58.801 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:58.841 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:58.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:58.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:58.943 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:58.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:58 smithi028 ceph-mon[29442]: 4.1c scrub starts 2024-02-05T06:43:58.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:58 smithi028 ceph-mon[29442]: 5.12 scrub starts 2024-02-05T06:43:58.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:58 smithi028 ceph-mon[29442]: 4.1c scrub ok 2024-02-05T06:43:58.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:58 smithi028 ceph-mon[29442]: 5.12 scrub ok 2024-02-05T06:43:58.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:58 smithi028 ceph-mon[37984]: 4.1c scrub starts 2024-02-05T06:43:58.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:58 smithi028 ceph-mon[37984]: 5.12 scrub starts 2024-02-05T06:43:58.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:58 smithi028 ceph-mon[37984]: 4.1c scrub ok 2024-02-05T06:43:58.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:58 smithi028 ceph-mon[37984]: 5.12 scrub ok 2024-02-05T06:43:58.970 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:43:58.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:58.979 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:58.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:58.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:58 smithi167 ceph-mon[32774]: 4.1c scrub starts 2024-02-05T06:43:58.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:58 smithi167 ceph-mon[32774]: 5.12 scrub starts 2024-02-05T06:43:58.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:58 smithi167 ceph-mon[32774]: 4.1c scrub ok 2024-02-05T06:43:58.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:58 smithi167 ceph-mon[32774]: 5.12 scrub ok 2024-02-05T06:43:59.080 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:59.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:59.114 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:59.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:59.121 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 19 16 44417 44401 146.02 112.625 0.00338839 0.00683823 2024-02-05T06:43:59.216 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:43:59.253 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:59.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:59.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:43:59.354 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:43:59.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:59.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:43:59.493 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:59.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:59.537 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:59.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:59.638 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:43:59.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:59.680 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:43:59.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:43:59.781 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:59.782 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:43:59.812 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:59.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:59.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:59.914 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:43:59.947 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:43:59.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:43:59.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:43:59.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:59 smithi028 ceph-mon[29442]: 4.e scrub starts 2024-02-05T06:43:59.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:59 smithi028 ceph-mon[29442]: 4.e scrub ok 2024-02-05T06:43:59.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:59 smithi028 ceph-mon[29442]: 4.5 scrub starts 2024-02-05T06:43:59.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:59 smithi028 ceph-mon[29442]: 4.5 scrub ok 2024-02-05T06:43:59.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:59 smithi028 ceph-mon[29442]: pgmap v515: 158 pgs: 158 active+clean; 2.4 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 161 MiB/s wr, 2.57k op/s 2024-02-05T06:43:59.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:59 smithi028 ceph-mon[29442]: 5.13 scrub starts 2024-02-05T06:43:59.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:43:59 smithi028 ceph-mon[29442]: 5.13 scrub ok 2024-02-05T06:43:59.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:59 smithi028 ceph-mon[37984]: 4.e scrub starts 2024-02-05T06:43:59.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:59 smithi028 ceph-mon[37984]: 4.e scrub ok 2024-02-05T06:43:59.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:59 smithi028 ceph-mon[37984]: 4.5 scrub starts 2024-02-05T06:43:59.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:59 smithi028 ceph-mon[37984]: 4.5 scrub ok 2024-02-05T06:43:59.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:59 smithi028 ceph-mon[37984]: pgmap v515: 158 pgs: 158 active+clean; 2.4 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 161 MiB/s wr, 2.57k op/s 2024-02-05T06:43:59.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:59 smithi028 ceph-mon[37984]: 5.13 scrub starts 2024-02-05T06:43:59.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:43:59 smithi028 ceph-mon[37984]: 5.13 scrub ok 2024-02-05T06:43:59.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:59 smithi167 ceph-mon[32774]: 4.e scrub starts 2024-02-05T06:43:59.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:59 smithi167 ceph-mon[32774]: 4.e scrub ok 2024-02-05T06:43:59.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:59 smithi167 ceph-mon[32774]: 4.5 scrub starts 2024-02-05T06:43:59.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:59 smithi167 ceph-mon[32774]: 4.5 scrub ok 2024-02-05T06:43:59.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:59 smithi167 ceph-mon[32774]: pgmap v515: 158 pgs: 158 active+clean; 2.4 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 161 MiB/s wr, 2.57k op/s 2024-02-05T06:43:59.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:59 smithi167 ceph-mon[32774]: 5.13 scrub starts 2024-02-05T06:43:59.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:43:59 smithi167 ceph-mon[32774]: 5.13 scrub ok 2024-02-05T06:44:00.049 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:00.095 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:00.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:00.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:00.121 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-02-05T06:44:00.116184+0000 min lat: 0.0012123 max lat: 0.110475 avg lat: 0.00694376 2024-02-05T06:44:00.121 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T06:44:00.122 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 20 16 46074 46058 143.897 103.562 0.0139385 0.00694376 2024-02-05T06:44:00.197 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:00.228 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:00.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:00.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:00.330 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:00.365 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:00.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:00.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:00.467 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:00.498 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:00.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:00.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:00.600 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:00.635 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:00.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:00.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:00.737 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:00.785 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:00.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:00.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:00.810 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:00 smithi028 ceph-mon[29442]: 1.a scrub starts 2024-02-05T06:44:00.810 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:00 smithi028 ceph-mon[29442]: 1.a scrub ok 2024-02-05T06:44:00.810 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:00 smithi028 ceph-mon[29442]: 6.13 scrub starts 2024-02-05T06:44:00.810 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:00 smithi028 ceph-mon[29442]: 6.13 scrub ok 2024-02-05T06:44:00.810 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:00 smithi028 ceph-mon[37984]: 1.a scrub starts 2024-02-05T06:44:00.810 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:00 smithi028 ceph-mon[37984]: 1.a scrub ok 2024-02-05T06:44:00.810 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:00 smithi028 ceph-mon[37984]: 6.13 scrub starts 2024-02-05T06:44:00.811 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:00 smithi028 ceph-mon[37984]: 6.13 scrub ok 2024-02-05T06:44:00.887 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:00.923 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:00.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:00.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:00.947 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:44:00.947 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-02-05T06:44:00.947 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-02-05T06:44:00.947 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-02-05T06:44:00.947 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:44:00.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:00 smithi167 ceph-mon[32774]: 1.a scrub starts 2024-02-05T06:44:00.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:00 smithi167 ceph-mon[32774]: 1.a scrub ok 2024-02-05T06:44:00.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:00 smithi167 ceph-mon[32774]: 6.13 scrub starts 2024-02-05T06:44:00.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:00 smithi167 ceph-mon[32774]: 6.13 scrub ok 2024-02-05T06:44:01.025 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:01.067 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:01.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:01.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:01.121 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 21 16 47547 47531 141.428 92.0625 0.00600075 0.00706578 2024-02-05T06:44:01.169 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:01.212 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:01.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:01.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:01.313 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:01.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:01.353 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:01.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:01.454 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:01.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:01.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:01.594 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:01.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:01.631 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:01.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:01.732 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:01.769 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:01.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:01.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:01.871 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:01.910 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:01.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:01.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:01.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:01 smithi028 ceph-mon[37984]: 6.9 scrub starts 2024-02-05T06:44:01.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:01 smithi028 ceph-mon[37984]: 6.9 scrub ok 2024-02-05T06:44:01.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:01 smithi028 ceph-mon[37984]: pgmap v516: 158 pgs: 158 active+clean; 2.4 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 620 B/s rd, 109 MiB/s wr, 1.74k op/s 2024-02-05T06:44:01.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:01 smithi028 ceph-mon[37984]: 6.d scrub starts 2024-02-05T06:44:01.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:01 smithi028 ceph-mon[37984]: 3.15 scrub starts 2024-02-05T06:44:01.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:01 smithi028 ceph-mon[37984]: 3.15 scrub ok 2024-02-05T06:44:01.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:01 smithi028 ceph-mon[37984]: 6.d scrub ok 2024-02-05T06:44:01.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:01 smithi028 ceph-mon[29442]: 6.9 scrub starts 2024-02-05T06:44:01.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:01 smithi028 ceph-mon[29442]: 6.9 scrub ok 2024-02-05T06:44:01.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:01 smithi028 ceph-mon[29442]: pgmap v516: 158 pgs: 158 active+clean; 2.4 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 620 B/s rd, 109 MiB/s wr, 1.74k op/s 2024-02-05T06:44:01.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:01 smithi028 ceph-mon[29442]: 6.d scrub starts 2024-02-05T06:44:01.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:01 smithi028 ceph-mon[29442]: 3.15 scrub starts 2024-02-05T06:44:01.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:01 smithi028 ceph-mon[29442]: 3.15 scrub ok 2024-02-05T06:44:01.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:01 smithi028 ceph-mon[29442]: 6.d scrub ok 2024-02-05T06:44:01.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:01 smithi167 ceph-mon[32774]: 6.9 scrub starts 2024-02-05T06:44:01.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:01 smithi167 ceph-mon[32774]: 6.9 scrub ok 2024-02-05T06:44:01.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:01 smithi167 ceph-mon[32774]: pgmap v516: 158 pgs: 158 active+clean; 2.4 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 620 B/s rd, 109 MiB/s wr, 1.74k op/s 2024-02-05T06:44:01.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:01 smithi167 ceph-mon[32774]: 6.d scrub starts 2024-02-05T06:44:01.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:01 smithi167 ceph-mon[32774]: 3.15 scrub starts 2024-02-05T06:44:01.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:01 smithi167 ceph-mon[32774]: 3.15 scrub ok 2024-02-05T06:44:01.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:01 smithi167 ceph-mon[32774]: 6.d scrub ok 2024-02-05T06:44:02.012 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:02.050 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:02.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:02.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:02.122 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 22 16 49297 49281 139.971 109.375 0.0112461 0.00713744 2024-02-05T06:44:02.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:02.202 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:02.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:02.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:02.221 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-05T06:44:02.304 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:02.338 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:02.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:02.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:02.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:02.488 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:02.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:02.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:02.589 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:02.640 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:02.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:02.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:02.742 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:02.776 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:02.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:02.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:02.878 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:02.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:02.934 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:02.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:02.941 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:02 smithi167 ceph-mon[32774]: 3.12 deep-scrub starts 2024-02-05T06:44:02.942 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:02 smithi167 ceph-mon[32774]: 3.12 deep-scrub ok 2024-02-05T06:44:02.942 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:02 smithi167 ceph-mon[32774]: 5.11 scrub starts 2024-02-05T06:44:02.942 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:02 smithi167 ceph-mon[32774]: 5.11 scrub ok 2024-02-05T06:44:02.942 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:02 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:44:02.942 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:02 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:44:02.942 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:02 smithi167 ceph-mon[32774]: 5.b scrub starts 2024-02-05T06:44:02.942 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:02 smithi167 ceph-mon[32774]: 5.b scrub ok 2024-02-05T06:44:02.942 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:02 smithi167 ceph-mon[32774]: 3.9 scrub starts 2024-02-05T06:44:02.942 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:02 smithi167 ceph-mon[32774]: 3.9 scrub ok 2024-02-05T06:44:02.942 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:02 smithi167 ceph-mon[32774]: 5.17 scrub starts 2024-02-05T06:44:02.942 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:02 smithi167 ceph-mon[32774]: 5.17 scrub ok 2024-02-05T06:44:02.942 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:02 smithi167 ceph-mon[32774]: osdmap e301: 8 total, 8 up, 8 in 2024-02-05T06:44:02.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:02 smithi028 ceph-mon[29442]: 3.12 deep-scrub starts 2024-02-05T06:44:02.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:02 smithi028 ceph-mon[29442]: 3.12 deep-scrub ok 2024-02-05T06:44:02.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:02 smithi028 ceph-mon[29442]: 5.11 scrub starts 2024-02-05T06:44:02.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:02 smithi028 ceph-mon[29442]: 5.11 scrub ok 2024-02-05T06:44:02.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:02 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:44:02.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:02 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:44:02.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:02 smithi028 ceph-mon[29442]: 5.b scrub starts 2024-02-05T06:44:02.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:02 smithi028 ceph-mon[29442]: 5.b scrub ok 2024-02-05T06:44:02.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:02 smithi028 ceph-mon[29442]: 3.9 scrub starts 2024-02-05T06:44:02.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:02 smithi028 ceph-mon[29442]: 3.9 scrub ok 2024-02-05T06:44:02.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:02 smithi028 ceph-mon[29442]: 5.17 scrub starts 2024-02-05T06:44:02.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:02 smithi028 ceph-mon[29442]: 5.17 scrub ok 2024-02-05T06:44:02.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:02 smithi028 ceph-mon[29442]: osdmap e301: 8 total, 8 up, 8 in 2024-02-05T06:44:02.950 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:44:02 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:44:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:44:02.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:02 smithi028 ceph-mon[37984]: 3.12 deep-scrub starts 2024-02-05T06:44:02.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:02 smithi028 ceph-mon[37984]: 3.12 deep-scrub ok 2024-02-05T06:44:02.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:02 smithi028 ceph-mon[37984]: 5.11 scrub starts 2024-02-05T06:44:02.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:02 smithi028 ceph-mon[37984]: 5.11 scrub ok 2024-02-05T06:44:02.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:02 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:44:02.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:02 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:44:02.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:02 smithi028 ceph-mon[37984]: 5.b scrub starts 2024-02-05T06:44:02.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:02 smithi028 ceph-mon[37984]: 5.b scrub ok 2024-02-05T06:44:02.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:02 smithi028 ceph-mon[37984]: 3.9 scrub starts 2024-02-05T06:44:02.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:02 smithi028 ceph-mon[37984]: 3.9 scrub ok 2024-02-05T06:44:02.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:02 smithi028 ceph-mon[37984]: 5.17 scrub starts 2024-02-05T06:44:02.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:02 smithi028 ceph-mon[37984]: 5.17 scrub ok 2024-02-05T06:44:02.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:02 smithi028 ceph-mon[37984]: osdmap e301: 8 total, 8 up, 8 in 2024-02-05T06:44:03.035 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:03.075 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:03.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:03.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:03.122 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 23 16 50948 50932 138.372 103.188 0.00466968 0.00722217 2024-02-05T06:44:03.177 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:03.215 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:03.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:03.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:03.233 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:44:02 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:44:03.317 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:03.352 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:03.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:03.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:03.454 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:03.499 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:03.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:03.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:03.601 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:03.659 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:03.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:03.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:03.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:03 smithi028 ceph-mon[29442]: pgmap v518: 158 pgs: 158 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.29k op/s 2024-02-05T06:44:03.730 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:03 smithi028 ceph-mon[37984]: pgmap v518: 158 pgs: 158 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.29k op/s 2024-02-05T06:44:03.761 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:03.814 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:03.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:03.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:03.916 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:03.941 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:03.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:03.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:03.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:03 smithi167 ceph-mon[32774]: pgmap v518: 158 pgs: 158 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.29k op/s 2024-02-05T06:44:04.042 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:04.080 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:04.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:04.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:04.122 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 24 16 52486 52470 136.611 96.125 0.011513 0.00731424 2024-02-05T06:44:04.182 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:04.207 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:04.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:04.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:04.309 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:04.348 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:04.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:04.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:04.450 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:04.494 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:04.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:04.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:04.596 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:04.622 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:44:04.633 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:44:04.641 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:04.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:04.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:04.743 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:04.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:04.808 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:04.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:04.829 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:04 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:44:04.829 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:04 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:44:04.829 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:04 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:44:04.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:04 smithi028 ceph-mon[37984]: 6.4 scrub starts 2024-02-05T06:44:04.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:04 smithi028 ceph-mon[37984]: 6.4 scrub ok 2024-02-05T06:44:04.830 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:04 smithi028 ceph-mon[37984]: osdmap e302: 8 total, 8 up, 8 in 2024-02-05T06:44:04.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:04 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:44:04.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:04 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:44:04.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:04 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:44:04.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:04 smithi028 ceph-mon[29442]: 6.4 scrub starts 2024-02-05T06:44:04.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:04 smithi028 ceph-mon[29442]: 6.4 scrub ok 2024-02-05T06:44:04.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:04 smithi028 ceph-mon[29442]: osdmap e302: 8 total, 8 up, 8 in 2024-02-05T06:44:04.909 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:04.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:04.953 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:04.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:04.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:04 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:44:04.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:04 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:44:04.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:04 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:44:04.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:04 smithi167 ceph-mon[32774]: 6.4 scrub starts 2024-02-05T06:44:04.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:04 smithi167 ceph-mon[32774]: 6.4 scrub ok 2024-02-05T06:44:04.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:04 smithi167 ceph-mon[32774]: osdmap e302: 8 total, 8 up, 8 in 2024-02-05T06:44:05.053 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:05.086 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:05.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:05.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:05.122 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 25 16 53757 53741 134.324 79.4375 0.0105944 0.00743936 2024-02-05T06:44:05.188 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:05.225 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:05.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:05.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:05.327 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:05.366 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:05.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:05.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:05.468 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:05.504 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:05.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:05.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:05.606 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:05.633 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.5 to 1 (10000) 2024-02-05T06:44:05.638 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:05.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:05.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-02-05T06:44:05.740 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:05.781 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:05.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:05.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:05.883 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:05.884 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:05 smithi167 ceph-mon[32774]: 4.16 scrub starts 2024-02-05T06:44:05.884 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:05 smithi167 ceph-mon[32774]: 4.16 scrub ok 2024-02-05T06:44:05.884 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:05 smithi167 ceph-mon[32774]: 1.6 scrub starts 2024-02-05T06:44:05.884 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:05 smithi167 ceph-mon[32774]: 1.6 scrub ok 2024-02-05T06:44:05.884 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:05 smithi167 ceph-mon[32774]: pgmap v520: 158 pgs: 158 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 109 MiB/s wr, 1.75k op/s 2024-02-05T06:44:05.884 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:05 smithi167 ceph-mon[32774]: 1.1 scrub starts 2024-02-05T06:44:05.884 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:05 smithi167 ceph-mon[32774]: 1.1 scrub ok 2024-02-05T06:44:05.885 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:05 smithi167 ceph-mon[32774]: 3.19 scrub starts 2024-02-05T06:44:05.885 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:05 smithi167 ceph-mon[32774]: 3.19 scrub ok 2024-02-05T06:44:05.885 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:05 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/834681048' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T06:44:05.885 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:05 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2159525766' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:44:05.885 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:05 smithi167 ceph-mon[32774]: from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:44:05.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:05.923 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:05.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:05.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:05 smithi028 ceph-mon[37984]: 4.16 scrub starts 2024-02-05T06:44:05.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:05 smithi028 ceph-mon[37984]: 4.16 scrub ok 2024-02-05T06:44:05.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:05 smithi028 ceph-mon[37984]: 1.6 scrub starts 2024-02-05T06:44:05.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:05 smithi028 ceph-mon[37984]: 1.6 scrub ok 2024-02-05T06:44:05.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:05 smithi028 ceph-mon[37984]: pgmap v520: 158 pgs: 158 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 109 MiB/s wr, 1.75k op/s 2024-02-05T06:44:05.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:05 smithi028 ceph-mon[37984]: 1.1 scrub starts 2024-02-05T06:44:05.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:05 smithi028 ceph-mon[37984]: 1.1 scrub ok 2024-02-05T06:44:05.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:05 smithi028 ceph-mon[37984]: 3.19 scrub starts 2024-02-05T06:44:05.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:05 smithi028 ceph-mon[37984]: 3.19 scrub ok 2024-02-05T06:44:05.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:05 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/834681048' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T06:44:05.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:05 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2159525766' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:44:05.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:05 smithi028 ceph-mon[37984]: from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:44:05.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:05 smithi028 ceph-mon[29442]: 4.16 scrub starts 2024-02-05T06:44:05.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:05 smithi028 ceph-mon[29442]: 4.16 scrub ok 2024-02-05T06:44:05.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:05 smithi028 ceph-mon[29442]: 1.6 scrub starts 2024-02-05T06:44:05.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:05 smithi028 ceph-mon[29442]: 1.6 scrub ok 2024-02-05T06:44:05.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:05 smithi028 ceph-mon[29442]: pgmap v520: 158 pgs: 158 active+clean; 3.0 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 109 MiB/s wr, 1.75k op/s 2024-02-05T06:44:05.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:05 smithi028 ceph-mon[29442]: 1.1 scrub starts 2024-02-05T06:44:05.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:05 smithi028 ceph-mon[29442]: 1.1 scrub ok 2024-02-05T06:44:05.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:05 smithi028 ceph-mon[29442]: 3.19 scrub starts 2024-02-05T06:44:05.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:05 smithi028 ceph-mon[29442]: 3.19 scrub ok 2024-02-05T06:44:05.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:05 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/834681048' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T06:44:05.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:05 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2159525766' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:44:05.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:05 smithi028 ceph-mon[29442]: from='client.25333 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:44:06.024 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:06.064 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:06.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:06.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:06.122 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 26 15 55361 55346 133.016 100.312 0.0132738 0.00751229 2024-02-05T06:44:06.166 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:06.202 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:06.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:06.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:06.304 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:06.339 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:06.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:06.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:06.441 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:06.623 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:06.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:06.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:06.725 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:06.899 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:06 smithi028 ceph-mon[37984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:44:06.899 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:06 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/834681048' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-05T06:44:06.899 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:06 smithi028 ceph-mon[37984]: from='client.25333 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:44:06.900 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:06 smithi028 ceph-mon[37984]: osdmap e303: 8 total, 8 up, 8 in 2024-02-05T06:44:06.900 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:06 smithi028 ceph-mon[29442]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:44:06.900 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:06 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/834681048' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-05T06:44:06.900 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:06 smithi028 ceph-mon[29442]: from='client.25333 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:44:06.900 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:06 smithi028 ceph-mon[29442]: osdmap e303: 8 total, 8 up, 8 in 2024-02-05T06:44:06.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:06 smithi167 ceph-mon[32774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:44:06.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:06 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/834681048' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-05T06:44:06.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:06 smithi167 ceph-mon[32774]: from='client.25333 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:44:06.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:06 smithi167 ceph-mon[32774]: osdmap e303: 8 total, 8 up, 8 in 2024-02-05T06:44:07.122 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 27 16 57002 56986 131.886 102.5 0.00660702 0.00757758 2024-02-05T06:44:07.290 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:07.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:07.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:07.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:07.430 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:07.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:07.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:07.532 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:07.566 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:07.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:07.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:07.668 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:07.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:07.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:07.809 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:07.839 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:07 smithi028 ceph-mon[37984]: pgmap v522: 158 pgs: 158 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 125 MiB/s wr, 2.00k op/s 2024-02-05T06:44:07.839 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:07 smithi028 ceph-mon[29442]: pgmap v522: 158 pgs: 158 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 125 MiB/s wr, 2.00k op/s 2024-02-05T06:44:07.843 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:07.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:07.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:07.917 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-05T06:44:07.917 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T06:44:07.945 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:07.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:07 smithi167 ceph-mon[32774]: pgmap v522: 158 pgs: 158 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 125 MiB/s wr, 2.00k op/s 2024-02-05T06:44:08.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:08.010 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:08.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:08.111 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:08.124 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 28 16 58550 58534 130.622 96.75 0.0140183 0.00765029 2024-02-05T06:44:08.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:08.170 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:08.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:08.271 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:08.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:08.312 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:08.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:08.412 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:08.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:08.448 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:08.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:08.549 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:08.586 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:08.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:08.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:08.688 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:08.724 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:08.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:08.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:08.826 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:08.861 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:08.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:08.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:08.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:08 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:44:08.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:08 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:44:08.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:08 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:44:08.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:08 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:44:08.963 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:08.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:08 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:44:08.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:08 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:44:09.004 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:09.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:09.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:09.106 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:09.124 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 29 16 59976 59960 129.191 89.125 0.0106703 0.00773558 2024-02-05T06:44:09.139 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:09.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:09.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:09.240 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:09.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:09.277 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:09.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:09.378 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:09.402 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:09.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:09.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:09.504 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:09.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:09.539 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:09.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:09.640 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:09.679 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:09.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:09.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:09.781 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:09.812 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:09.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:09.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:09.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:09 smithi028 ceph-mon[29442]: pgmap v523: 158 pgs: 158 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 819 B/s rd, 116 MiB/s wr, 1.85k op/s 2024-02-05T06:44:09.910 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:09 smithi028 ceph-mon[29442]: osdmap e304: 8 total, 8 up, 8 in 2024-02-05T06:44:09.911 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:09 smithi028 ceph-mon[37984]: pgmap v523: 158 pgs: 158 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 819 B/s rd, 116 MiB/s wr, 1.85k op/s 2024-02-05T06:44:09.911 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:09 smithi028 ceph-mon[37984]: osdmap e304: 8 total, 8 up, 8 in 2024-02-05T06:44:09.914 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:09.947 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:09.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:09.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:09.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:09 smithi167 ceph-mon[32774]: pgmap v523: 158 pgs: 158 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 819 B/s rd, 116 MiB/s wr, 1.85k op/s 2024-02-05T06:44:09.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:09 smithi167 ceph-mon[32774]: osdmap e304: 8 total, 8 up, 8 in 2024-02-05T06:44:10.050 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:10.084 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:10.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:10.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:10.124 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 30 16 61284 61268 127.61 81.75 0.00983331 0.00783031 2024-02-05T06:44:10.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:10.213 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:10.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:10.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:10.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:10.358 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:10.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:10.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:10.460 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:10.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:10.503 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:10.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:10.604 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:10.638 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:10.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:10.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:10.740 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:10.772 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:10.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:10.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:10.873 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:10.905 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:10.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:10.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:11.007 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:11.042 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:11.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:11.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:11.124 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 31 16 62729 62713 126.406 90.3125 0.00916501 0.00790574 2024-02-05T06:44:11.144 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:11.180 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:11.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:11.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:11.282 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:11.323 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:11.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:11.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:11.425 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:11.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:11 smithi028 ceph-mon[29442]: pgmap v525: 158 pgs: 158 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 819 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-02-05T06:44:11.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:11 smithi028 ceph-mon[37984]: pgmap v525: 158 pgs: 158 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 819 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-02-05T06:44:11.463 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:11.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:11.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:11.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:11 smithi167 ceph-mon[32774]: pgmap v525: 158 pgs: 158 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 819 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-02-05T06:44:11.565 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:11.599 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:11.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:11.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:11.701 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:11.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:11.737 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:11.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:11.839 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:11.875 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:11.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:11.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:11.977 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:12.025 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:12.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:12.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:12.124 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 32 16 63992 63976 124.923 78.9375 0.00820884 0.00800001 2024-02-05T06:44:12.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:12.142 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T06:44:12.169 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:12.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:12.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:12.270 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:12.299 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:12.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:12.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:12.401 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:12.455 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:12.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:12.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:12.557 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:12.592 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:12.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:12.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:12.694 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:12.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:12.726 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:12.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:12.827 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:12.857 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:12.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:12.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:12.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:12 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:44:12.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:12 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:44:12.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:12 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:44:12.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:12 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:44:12.949 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:44:12 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:44:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:44:12.952 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:12 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:44:12.952 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:12 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:44:12.959 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:12.995 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:12.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:12.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:13.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:13.124 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 33 16 65353 65337 123.715 85.0625 0.00475194 0.00807798 2024-02-05T06:44:13.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:13.133 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:13.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:13.233 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:44:12 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:44:13.234 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:13.270 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:13.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:13.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:13.372 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:13.410 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:13.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:13.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:13.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:13.550 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:13.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:13.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:13.652 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:13.683 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:13.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:13.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:13.785 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:13.811 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:13.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:13.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:13.913 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:13.928 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:13 smithi028 ceph-mon[29442]: pgmap v526: 158 pgs: 158 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-02-05T06:44:13.928 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:13 smithi028 ceph-mon[37984]: pgmap v526: 158 pgs: 158 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-02-05T06:44:13.957 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:13.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:13.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:13.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:13 smithi167 ceph-mon[32774]: pgmap v526: 158 pgs: 158 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-02-05T06:44:14.059 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:14.098 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:14.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:14.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:14.124 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 34 16 66695 66679 122.543 83.875 0.0136324 0.00815536 2024-02-05T06:44:14.200 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:14.230 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:14.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:14.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:14.332 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:14.365 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:14.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:14.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:14.466 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:14.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:14.506 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:14.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:14.606 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:14.649 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:14.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:14.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:14.750 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:14.784 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:14.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:14.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:14.886 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:14.920 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:14.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:14.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:14.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:14 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:44:14.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:14 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:44:14.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:14 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:44:15.022 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:15.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:15.055 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:15.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:15.126 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 35 16 67904 67888 121.197 75.5625 0.00909777 0.0082456 2024-02-05T06:44:15.156 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:15.186 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:15.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:15.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:15.209 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:44:15.228 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:44:15.264 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:44:15.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:15.326 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:44:15.331 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:15.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:15.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:15.367 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:44:15.404 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:44:15.433 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:15.456 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:44:15.457 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:15.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:15.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:15.499 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:44:15.509 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:44:15.509 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:44:15.509 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:44:15.509 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:44:15.509 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:44:15.509 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:44:15.509 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:44:15.510 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:44:15.559 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:15.598 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:15.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:15.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:15.699 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:15.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:15.732 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:15.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:15.738 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:15 smithi028 ceph-mon[29442]: pgmap v527: 158 pgs: 158 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 107 MiB/s wr, 1.71k op/s 2024-02-05T06:44:15.739 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:15 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1869764940' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:44:15.739 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:15 smithi028 ceph-mon[37984]: pgmap v527: 158 pgs: 158 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 107 MiB/s wr, 1.71k op/s 2024-02-05T06:44:15.739 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:15 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1869764940' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:44:15.833 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:15.835 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:44:15.835 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:44:15.871 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:15.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:15.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:15.972 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:15.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:15 smithi167 ceph-mon[32774]: pgmap v527: 158 pgs: 158 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 107 MiB/s wr, 1.71k op/s 2024-02-05T06:44:15.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:15 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1869764940' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:44:16.012 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:16.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:16.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:16.114 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:16.126 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 36 16 69039 69023 119.801 70.9375 0.00827736 0.00834185 2024-02-05T06:44:16.149 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:16.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:16.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:16.251 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:16.290 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:16.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:16.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:16.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:16.430 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:16.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:16.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:16.531 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:16.563 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:16.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:16.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:16.665 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:16.701 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:16.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:16.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:16.803 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:16.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:16.838 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:16.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:16.939 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:16.964 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:16.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:16.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:17.066 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:17.111 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:17.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:17.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:17.126 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 37 16 70447 70431 118.942 88 0.00844548 0.00840242 2024-02-05T06:44:17.213 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:17.242 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:17.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:17.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:17.344 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:17.380 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:17.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:17.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:17.482 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:17.515 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:17.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:17.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:17.617 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:17.655 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:17.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:17.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:17.757 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:17.791 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:17.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:17.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:17.893 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:17.930 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:17.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:17.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:17.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:17 smithi028 ceph-mon[29442]: pgmap v528: 158 pgs: 158 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-02-05T06:44:17.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:17 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:44:17.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:17 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:44:17.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:17 smithi028 ceph-mon[37984]: pgmap v528: 158 pgs: 158 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-02-05T06:44:17.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:17 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:44:17.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:17 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:44:17.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:17 smithi167 ceph-mon[32774]: pgmap v528: 158 pgs: 158 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-02-05T06:44:17.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:17 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:44:17.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:17 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:44:18.032 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:18.066 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:18.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:18.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:18.126 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 38 16 71764 71748 117.978 82.3125 0.00752439 0.00847092 2024-02-05T06:44:18.168 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:18.267 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:18.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:18.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:18.369 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:18.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:18.404 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:18.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:18.506 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:18.536 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:18.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:18.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:18.542 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:44:18.553 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-05T06:44:18.571 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:44:18.613 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:44:18.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:18.687 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:18.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:18.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:18.789 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:18.794 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:18 smithi028 ceph-mon[29442]: osdmap e305: 8 total, 8 up, 8 in 2024-02-05T06:44:18.794 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:18 smithi028 ceph-mon[37984]: osdmap e305: 8 total, 8 up, 8 in 2024-02-05T06:44:18.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:18.841 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:18.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:18.942 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:18.954 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:44:18.981 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:18.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:18.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:18.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:18 smithi167 ceph-mon[32774]: osdmap e305: 8 total, 8 up, 8 in 2024-02-05T06:44:19.083 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:19.126 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 39 16 73035 73019 116.988 79.4375 0.0127168 0.00854089 2024-02-05T06:44:19.131 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:19.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:19.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:19.240 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:19.272 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:19.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:19.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:19.374 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:19.415 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:19.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:19.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:19.518 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:19.557 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:19.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:19.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:19.658 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:19.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:19.699 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:19.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:19.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:19.834 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:19 smithi028 ceph-mon[37984]: pgmap v530: 158 pgs: 158 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-02-05T06:44:19.834 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:19 smithi028 ceph-mon[29442]: pgmap v530: 158 pgs: 158 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-02-05T06:44:19.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:19.846 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:19.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:19.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:19.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:19 smithi167 ceph-mon[32774]: pgmap v530: 158 pgs: 158 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-02-05T06:44:19.988 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:19.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:19.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:20.090 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:20.127 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-02-05T06:44:20.122094+0000 min lat: 0.0012123 max lat: 0.145886 avg lat: 0.00860577 2024-02-05T06:44:20.127 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T06:44:20.128 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 40 16 74349 74333 116.114 82.125 0.0172188 0.00860577 2024-02-05T06:44:20.142 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:20.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:20.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:20.244 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:20.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:20.288 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:20.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:20.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:20.444 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:20.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:20.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:20.546 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:20.599 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:20.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:20.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:20.700 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:20.735 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:20.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:20.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:20.837 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:20.873 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:20.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:20.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:20.909 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:20 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:44:20.909 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:20 smithi167 ceph-mon[32774]: osdmap e306: 8 total, 8 up, 8 in 2024-02-05T06:44:20.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:20 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:44:20.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:20 smithi028 ceph-mon[29442]: osdmap e306: 8 total, 8 up, 8 in 2024-02-05T06:44:20.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:20 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:44:20.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:20 smithi028 ceph-mon[37984]: osdmap e306: 8 total, 8 up, 8 in 2024-02-05T06:44:20.975 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:21.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:21.030 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:21.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:21.128 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 41 16 75463 75447 114.98 69.625 0.00574071 0.00869189 2024-02-05T06:44:21.129 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:44:21.129 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:44:21.131 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:21.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:21.172 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:21.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:21.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:21.320 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:21.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:21.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:21.423 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:21.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:21.465 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:21.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:21.565 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:21.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:21.623 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:21.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:21.724 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:21.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:21.772 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:21.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:21.792 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:21 smithi028 ceph-mon[37984]: pgmap v532: 158 pgs: 158 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 50 MiB/s wr, 799 op/s 2024-02-05T06:44:21.792 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:21 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:44:21.792 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:21 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:44:21.792 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:21 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:44:21.793 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:21 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:44:21.793 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:21 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:44:21.793 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:21 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:44:21.793 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:21 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:44:21.793 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:21 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:44:21.793 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:21 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:44:21.793 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:21 smithi028 ceph-mon[29442]: pgmap v532: 158 pgs: 158 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 50 MiB/s wr, 799 op/s 2024-02-05T06:44:21.793 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:21 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:44:21.793 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:21 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:44:21.793 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:21 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:44:21.793 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:21 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:44:21.794 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:21 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:44:21.794 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:21 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:44:21.794 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:21 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:44:21.794 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:21 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:44:21.794 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:21 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:44:21.873 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:21.874 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:21 smithi167 ceph-mon[32774]: pgmap v532: 158 pgs: 158 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 50 MiB/s wr, 799 op/s 2024-02-05T06:44:21.874 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:21 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:44:21.874 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:21 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:44:21.874 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:21 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:44:21.875 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:21 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:44:21.875 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:21 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:44:21.875 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:21 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:44:21.875 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:21 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:44:21.875 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:21 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:44:21.875 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:21 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:44:21.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:21.924 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:21.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:21.927 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-05T06:44:22.025 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:22.070 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:22.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:22.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:22.128 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 42 16 76532 76516 113.834 66.8125 0.00991667 0.00877957 2024-02-05T06:44:22.172 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:22.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:22.226 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:22.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:22.327 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:22.345 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:44:22.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:22.365 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:22.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:22.422 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:44:22.466 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:22.498 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:22.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:22.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:22.600 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:22.650 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:22.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:22.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:22.669 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:44:22 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:44:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:44:22.688 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:44:22.702 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-05T06:44:22.752 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:22.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:22.789 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:22.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:22.803 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:22 smithi167 ceph-mon[32774]: Adjusting osd_memory_target on smithi167 to 3248M 2024-02-05T06:44:22.803 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:22 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:44:22.804 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:22 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:44:22.804 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:22 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:44:22.890 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:22.931 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:22.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:22.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:22.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:22 smithi028 ceph-mon[37984]: Adjusting osd_memory_target on smithi167 to 3248M 2024-02-05T06:44:22.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:22 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:44:22.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:22 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:44:22.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:22 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:44:22.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:22 smithi028 ceph-mon[29442]: Adjusting osd_memory_target on smithi167 to 3248M 2024-02-05T06:44:22.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:22 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:44:22.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:22 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:44:22.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:22 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:44:23.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:23.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:23.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:23.128 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 43 16 77627 77611 112.778 68.4375 0.00547023 0.00886207 2024-02-05T06:44:23.181 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:23.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:23.221 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:23.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:23.233 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:44:22 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:44:23.323 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:23.359 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:23.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:23.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:23.460 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:23.494 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:23.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:23.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:23.595 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:23.639 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:23.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:23.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:23.695 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-02-05T06:44:23.741 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:23.789 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:23.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:23.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:23.892 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:23.930 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:23.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:23.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:23.936 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:23 smithi167 ceph-mon[32774]: pgmap v533: 158 pgs: 158 active+clean; 4.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.58k op/s 2024-02-05T06:44:23.936 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:23 smithi167 ceph-mon[32774]: from='client.15438 v1:172.21.15.28:0/663377666' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:44:23.936 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:23 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3753004378' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:44:23.936 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:23 smithi167 ceph-mon[32774]: from='client.45320 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:44:23.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:23 smithi028 ceph-mon[29442]: pgmap v533: 158 pgs: 158 active+clean; 4.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.58k op/s 2024-02-05T06:44:23.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:23 smithi028 ceph-mon[29442]: from='client.15438 v1:172.21.15.28:0/663377666' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:44:23.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:23 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3753004378' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:44:23.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:23 smithi028 ceph-mon[29442]: from='client.45320 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:44:23.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:23 smithi028 ceph-mon[37984]: pgmap v533: 158 pgs: 158 active+clean; 4.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.58k op/s 2024-02-05T06:44:23.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:23 smithi028 ceph-mon[37984]: from='client.15438 v1:172.21.15.28:0/663377666' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:44:23.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:23 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3753004378' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:44:23.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:23 smithi028 ceph-mon[37984]: from='client.45320 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:44:24.032 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:24.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:24.076 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:24.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:24.115 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":533,"stamp":"2024-02-05T06:44:22.360868+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4990769500,"num_objects":76342,"num_object_clones":0,"num_object_copies":229026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76342,"num_whiteouts":0,"num_read":1368,"num_read_kb":1120,"num_write":76866,"num_write_kb":4873261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":416,"num_bytes_recovered":1586799,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76820,"ondisk_log_size":76820,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":17384916,"kb_used_data":14499132,"kb_used_omap":12,"kb_used_meta":2885683,"kb_avail":732576300,"statfs":{"total":767960285184,"available":750158131200,"internally_reserved":0,"allocated":14847111168,"data_stored":14842222484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":2954939953},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":825884672,"num_objects":12602,"num_object_clones":0,"num_object_copies":37806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12602,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12602,"num_write_kb":806528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001910"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":421,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.119455+0000","last_change":"2024-02-05T06:42:34.762437+0000","last_active":"2024-02-05T06:44:20.119455+0000","last_peered":"2024-02-05T06:44:20.119455+0000","last_clean":"2024-02-05T06:44:20.119455+0000","last_became_active":"2024-02-05T06:41:43.412603+0000","last_became_peered":"2024-02-05T06:41:43.412603+0000","last_unstale":"2024-02-05T06:44:20.119455+0000","last_undegraded":"2024-02-05T06:44:20.119455+0000","last_fullsized":"2024-02-05T06:44:20.119455+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:34.762324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:42:34.762324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:20.891788+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":542,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.108651+0000","last_change":"2024-02-05T06:43:36.504759+0000","last_active":"2024-02-05T06:44:20.108651+0000","last_peered":"2024-02-05T06:44:20.108651+0000","last_clean":"2024-02-05T06:44:20.108651+0000","last_became_active":"2024-02-05T06:41:43.420342+0000","last_became_peered":"2024-02-05T06:41:43.420342+0000","last_unstale":"2024-02-05T06:44:20.108651+0000","last_undegraded":"2024-02-05T06:44:20.108651+0000","last_fullsized":"2024-02-05T06:44:20.108651+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:36.504651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:06.525012+0000","last_clean_scrub_stamp":"2024-02-05T06:43:36.504651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:58.084697+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":532,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.101866+0000","last_change":"2024-02-05T06:43:40.001642+0000","last_active":"2024-02-05T06:44:20.101866+0000","last_peered":"2024-02-05T06:44:20.101866+0000","last_clean":"2024-02-05T06:44:20.101866+0000","last_became_active":"2024-02-05T06:41:43.421315+0000","last_became_peered":"2024-02-05T06:41:43.421315+0000","last_unstale":"2024-02-05T06:44:20.101866+0000","last_undegraded":"2024-02-05T06:44:20.101866+0000","last_fullsized":"2024-02-05T06:44:20.101866+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:40.001568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:43:40.001568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:58.362550+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"295'19","reported_seq":435,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.118592+0000","last_change":"2024-02-05T06:43:52.801066+0000","last_active":"2024-02-05T06:44:20.118592+0000","last_peered":"2024-02-05T06:44:20.118592+0000","last_clean":"2024-02-05T06:44:20.118592+0000","last_became_active":"2024-02-05T06:36:54.959330+0000","last_became_peered":"2024-02-05T06:36:54.959330+0000","last_unstale":"2024-02-05T06:44:20.118592+0000","last_undegraded":"2024-02-05T06:44:20.118592+0000","last_fullsized":"2024-02-05T06:44:20.118592+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'19","last_scrub_stamp":"2024-02-05T06:43:52.800915+0000","last_deep_scrub":"247'13","last_deep_scrub_stamp":"2024-02-05T06:41:42.473424+0000","last_clean_scrub_stamp":"2024-02-05T06:43:52.800915+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:13.160739+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":518,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.111839+0000","last_change":"2024-02-05T06:42:40.263147+0000","last_active":"2024-02-05T06:44:20.111839+0000","last_peered":"2024-02-05T06:44:20.111839+0000","last_clean":"2024-02-05T06:44:20.111839+0000","last_became_active":"2024-02-05T06:40:27.077485+0000","last_became_peered":"2024-02-05T06:40:27.077485+0000","last_unstale":"2024-02-05T06:44:20.111839+0000","last_undegraded":"2024-02-05T06:44:20.111839+0000","last_fullsized":"2024-02-05T06:44:20.111839+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:40.262880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:42:40.262880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:23.769816+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":404,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.118725+0000","last_change":"2024-02-05T06:41:47.546569+0000","last_active":"2024-02-05T06:44:20.118725+0000","last_peered":"2024-02-05T06:44:20.118725+0000","last_clean":"2024-02-05T06:44:20.118725+0000","last_became_active":"2024-02-05T06:36:54.959640+0000","last_became_peered":"2024-02-05T06:36:54.959640+0000","last_unstale":"2024-02-05T06:44:20.118725+0000","last_undegraded":"2024-02-05T06:44:20.118725+0000","last_fullsized":"2024-02-05T06:44:20.118725+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:41:47.546464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:41:47.546464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:20.891690+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":527,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.116987+0000","last_change":"2024-02-05T06:43:53.697132+0000","last_active":"2024-02-05T06:44:20.116987+0000","last_peered":"2024-02-05T06:44:20.116987+0000","last_clean":"2024-02-05T06:44:20.116987+0000","last_became_active":"2024-02-05T06:33:29.553864+0000","last_became_peered":"2024-02-05T06:33:29.553864+0000","last_unstale":"2024-02-05T06:44:20.116987+0000","last_undegraded":"2024-02-05T06:44:20.116987+0000","last_fullsized":"2024-02-05T06:44:20.116987+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:53.696997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:43:53.696997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:58.140971+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"295'34","reported_seq":572,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.112950+0000","last_change":"2024-02-05T06:43:56.716084+0000","last_active":"2024-02-05T06:44:20.112950+0000","last_peered":"2024-02-05T06:44:20.112950+0000","last_clean":"2024-02-05T06:44:20.112950+0000","last_became_active":"2024-02-05T06:33:31.572544+0000","last_became_peered":"2024-02-05T06:33:31.572544+0000","last_unstale":"2024-02-05T06:44:20.112950+0000","last_undegraded":"2024-02-05T06:44:20.112950+0000","last_fullsized":"2024-02-05T06:44:20.112950+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'34","last_scrub_stamp":"2024-02-05T06:43:56.715959+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:39:35.433539+0000","last_clean_scrub_stamp":"2024-02-05T06:43:56.715959+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:08.901916+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":519,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.102003+0000","last_change":"2024-02-05T06:42:52.920370+0000","last_active":"2024-02-05T06:44:20.102003+0000","last_peered":"2024-02-05T06:44:20.102003+0000","last_clean":"2024-02-05T06:44:20.102003+0000","last_became_active":"2024-02-05T06:40:27.068888+0000","last_became_peered":"2024-02-05T06:40:27.068888+0000","last_unstale":"2024-02-05T06:44:20.102003+0000","last_undegraded":"2024-02-05T06:44:20.102003+0000","last_fullsized":"2024-02-05T06:44:20.102003+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:52.920236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:42:52.920236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:20.164709+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":83,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.111927+0000","last_change":"2024-02-05T06:42:46.353608+0000","last_active":"2024-02-05T06:44:20.111927+0000","last_peered":"2024-02-05T06:44:20.111927+0000","last_clean":"2024-02-05T06:44:20.111927+0000","last_became_active":"2024-02-05T06:42:33.129959+0000","last_became_peered":"2024-02-05T06:42:33.129959+0000","last_unstale":"2024-02-05T06:44:20.111927+0000","last_undegraded":"2024-02-05T06:44:20.111927+0000","last_fullsized":"2024-02-05T06:44:20.111927+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:46.351296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:42:46.351296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:23.769841+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":136,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.102975+0000","last_change":"2024-02-05T06:43:56.453639+0000","last_active":"2024-02-05T06:44:20.102975+0000","last_peered":"2024-02-05T06:44:20.102975+0000","last_clean":"2024-02-05T06:44:20.102975+0000","last_became_active":"2024-02-05T06:41:43.403746+0000","last_became_peered":"2024-02-05T06:41:43.403746+0000","last_unstale":"2024-02-05T06:44:20.102975+0000","last_undegraded":"2024-02-05T06:44:20.102975+0000","last_fullsized":"2024-02-05T06:44:20.102975+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:56.453478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:44.081318+0000","last_clean_scrub_stamp":"2024-02-05T06:43:56.453478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:25.000882+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"295'19","reported_seq":430,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.119298+0000","last_change":"2024-02-05T06:43:49.843729+0000","last_active":"2024-02-05T06:44:20.119298+0000","last_peered":"2024-02-05T06:44:20.119298+0000","last_clean":"2024-02-05T06:44:20.119298+0000","last_became_active":"2024-02-05T06:36:54.969454+0000","last_became_peered":"2024-02-05T06:36:54.969454+0000","last_unstale":"2024-02-05T06:44:20.119298+0000","last_undegraded":"2024-02-05T06:44:20.119298+0000","last_fullsized":"2024-02-05T06:44:20.119298+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'19","last_scrub_stamp":"2024-02-05T06:43:49.843581+0000","last_deep_scrub":"92'13","last_deep_scrub_stamp":"2024-02-05T06:34:46.542893+0000","last_clean_scrub_stamp":"2024-02-05T06:43:49.843581+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:52.860608+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":136,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.102206+0000","last_change":"2024-02-05T06:43:53.476980+0000","last_active":"2024-02-05T06:44:20.102206+0000","last_peered":"2024-02-05T06:44:20.102206+0000","last_clean":"2024-02-05T06:44:20.102206+0000","last_became_active":"2024-02-05T06:41:43.407404+0000","last_became_peered":"2024-02-05T06:41:43.407404+0000","last_unstale":"2024-02-05T06:44:20.102206+0000","last_undegraded":"2024-02-05T06:44:20.102206+0000","last_fullsized":"2024-02-05T06:44:20.102206+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:53.476863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:21.998850+0000","last_clean_scrub_stamp":"2024-02-05T06:43:53.476863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:58.879515+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":228,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.106529+0000","last_change":"2024-02-05T06:43:54.540922+0000","last_active":"2024-02-05T06:44:20.106529+0000","last_peered":"2024-02-05T06:44:20.106529+0000","last_clean":"2024-02-05T06:44:20.106529+0000","last_became_active":"2024-02-05T06:40:27.071758+0000","last_became_peered":"2024-02-05T06:40:27.071758+0000","last_unstale":"2024-02-05T06:44:20.106529+0000","last_undegraded":"2024-02-05T06:44:20.106529+0000","last_fullsized":"2024-02-05T06:44:20.106529+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:54.540821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:54.540821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:21.286238+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":136,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.102658+0000","last_change":"2024-02-05T06:43:34.438758+0000","last_active":"2024-02-05T06:44:20.102658+0000","last_peered":"2024-02-05T06:44:20.102658+0000","last_clean":"2024-02-05T06:44:20.102658+0000","last_became_active":"2024-02-05T06:41:43.408428+0000","last_became_peered":"2024-02-05T06:41:43.408428+0000","last_unstale":"2024-02-05T06:44:20.102658+0000","last_undegraded":"2024-02-05T06:44:20.102658+0000","last_fullsized":"2024-02-05T06:44:20.102658+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:34.438631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:50.055245+0000","last_clean_scrub_stamp":"2024-02-05T06:43:34.438631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.429736+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"295'7","reported_seq":412,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.120083+0000","last_change":"2024-02-05T06:41:48.543907+0000","last_active":"2024-02-05T06:44:20.120083+0000","last_peered":"2024-02-05T06:44:20.120083+0000","last_clean":"2024-02-05T06:44:20.120083+0000","last_became_active":"2024-02-05T06:36:54.954385+0000","last_became_peered":"2024-02-05T06:36:54.954385+0000","last_unstale":"2024-02-05T06:44:20.120083+0000","last_undegraded":"2024-02-05T06:44:20.120083+0000","last_fullsized":"2024-02-05T06:44:20.120083+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-02-05T06:41:48.543648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:41:48.543648+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:20.891724+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":416,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.121494+0000","last_change":"2024-02-05T06:41:43.412335+0000","last_active":"2024-02-05T06:44:20.121494+0000","last_peered":"2024-02-05T06:44:20.121494+0000","last_clean":"2024-02-05T06:44:20.121494+0000","last_became_active":"2024-02-05T06:41:43.411344+0000","last_became_peered":"2024-02-05T06:41:43.411344+0000","last_unstale":"2024-02-05T06:44:20.121494+0000","last_undegraded":"2024-02-05T06:44:20.121494+0000","last_fullsized":"2024-02-05T06:44:20.121494+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:41:39.552367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:41:39.552367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:20.891543+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":530,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.111121+0000","last_change":"2024-02-05T06:43:37.552250+0000","last_active":"2024-02-05T06:44:20.111121+0000","last_peered":"2024-02-05T06:44:20.111121+0000","last_clean":"2024-02-05T06:44:20.111121+0000","last_became_active":"2024-02-05T06:36:54.958069+0000","last_became_peered":"2024-02-05T06:36:54.958069+0000","last_unstale":"2024-02-05T06:44:20.111121+0000","last_undegraded":"2024-02-05T06:44:20.111121+0000","last_fullsized":"2024-02-05T06:44:20.111121+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:37.552166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:37.552166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:56.420046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"281'2","reported_seq":566,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.108888+0000","last_change":"2024-02-05T06:42:58.933944+0000","last_active":"2024-02-05T06:44:20.108888+0000","last_peered":"2024-02-05T06:44:20.108888+0000","last_clean":"2024-02-05T06:44:20.108888+0000","last_became_active":"2024-02-05T06:40:27.073360+0000","last_became_peered":"2024-02-05T06:40:27.073360+0000","last_unstale":"2024-02-05T06:44:20.108888+0000","last_undegraded":"2024-02-05T06:44:20.108888+0000","last_fullsized":"2024-02-05T06:44:20.108888+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'2","last_scrub_stamp":"2024-02-05T06:42:58.933820+0000","last_deep_scrub":"281'2","last_deep_scrub_stamp":"2024-02-05T06:42:58.933820+0000","last_clean_scrub_stamp":"2024-02-05T06:42:58.933820+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:23.146502+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"295'34","reported_seq":594,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.114215+0000","last_change":"2024-02-05T06:43:57.470734+0000","last_active":"2024-02-05T06:44:20.114215+0000","last_peered":"2024-02-05T06:44:20.114215+0000","last_clean":"2024-02-05T06:44:20.114215+0000","last_became_active":"2024-02-05T06:33:31.569763+0000","last_became_peered":"2024-02-05T06:33:31.569763+0000","last_unstale":"2024-02-05T06:44:20.114215+0000","last_undegraded":"2024-02-05T06:44:20.114215+0000","last_fullsized":"2024-02-05T06:44:20.114215+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'34","last_scrub_stamp":"2024-02-05T06:43:57.470583+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:34:35.234636+0000","last_clean_scrub_stamp":"2024-02-05T06:43:57.470583+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:14.633396+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":519,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.117957+0000","last_change":"2024-02-05T06:43:38.524343+0000","last_active":"2024-02-05T06:44:20.117957+0000","last_peered":"2024-02-05T06:44:20.117957+0000","last_clean":"2024-02-05T06:44:20.117957+0000","last_became_active":"2024-02-05T06:41:43.411442+0000","last_became_peered":"2024-02-05T06:41:43.411442+0000","last_unstale":"2024-02-05T06:44:20.117957+0000","last_undegraded":"2024-02-05T06:44:20.117957+0000","last_fullsized":"2024-02-05T06:44:20.117957+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:38.524236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:43:38.524236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:51.142093+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":300,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.112290+0000","last_change":"2024-02-05T06:43:54.234456+0000","last_active":"2024-02-05T06:44:20.112290+0000","last_peered":"2024-02-05T06:44:20.112290+0000","last_clean":"2024-02-05T06:44:20.112290+0000","last_became_active":"2024-02-05T06:39:13.248618+0000","last_became_peered":"2024-02-05T06:39:13.248618+0000","last_unstale":"2024-02-05T06:44:20.112290+0000","last_undegraded":"2024-02-05T06:44:20.112290+0000","last_fullsized":"2024-02-05T06:44:20.112290+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:54.234345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:54.234345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:55.938687+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":403,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.119208+0000","last_change":"2024-02-05T06:42:57.721396+0000","last_active":"2024-02-05T06:44:20.119208+0000","last_peered":"2024-02-05T06:44:20.119208+0000","last_clean":"2024-02-05T06:44:20.119208+0000","last_became_active":"2024-02-05T06:36:54.959070+0000","last_became_peered":"2024-02-05T06:36:54.959070+0000","last_unstale":"2024-02-05T06:44:20.119208+0000","last_undegraded":"2024-02-05T06:44:20.119208+0000","last_fullsized":"2024-02-05T06:44:20.119208+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:57.721280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:57.721280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:20.891962+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"295'18","reported_seq":600,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.111925+0000","last_change":"2024-02-05T06:42:44.182931+0000","last_active":"2024-02-05T06:44:20.111925+0000","last_peered":"2024-02-05T06:44:20.111925+0000","last_clean":"2024-02-05T06:44:20.111925+0000","last_became_active":"2024-02-05T06:36:54.954880+0000","last_became_peered":"2024-02-05T06:36:54.954880+0000","last_unstale":"2024-02-05T06:44:20.111925+0000","last_undegraded":"2024-02-05T06:44:20.111925+0000","last_fullsized":"2024-02-05T06:44:20.111925+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:42:44.182785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:42:44.182785+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:23.769652+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"73'1","reported_seq":234,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.106941+0000","last_change":"2024-02-05T06:43:41.472748+0000","last_active":"2024-02-05T06:44:20.106941+0000","last_peered":"2024-02-05T06:44:20.106941+0000","last_clean":"2024-02-05T06:44:20.106941+0000","last_became_active":"2024-02-05T06:41:43.418640+0000","last_became_peered":"2024-02-05T06:41:43.418640+0000","last_unstale":"2024-02-05T06:44:20.106941+0000","last_undegraded":"2024-02-05T06:44:20.106941+0000","last_fullsized":"2024-02-05T06:44:20.106941+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-02-05T06:43:41.472635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:43:41.472635+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:54.907618+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"208'12","reported_seq":713,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.113285+0000","last_change":"2024-02-05T06:43:43.224045+0000","last_active":"2024-02-05T06:44:20.113285+0000","last_peered":"2024-02-05T06:44:20.113285+0000","last_clean":"2024-02-05T06:44:20.113285+0000","last_became_active":"2024-02-05T06:40:27.063399+0000","last_became_peered":"2024-02-05T06:40:27.063399+0000","last_unstale":"2024-02-05T06:44:20.113285+0000","last_undegraded":"2024-02-05T06:44:20.113285+0000","last_fullsized":"2024-02-05T06:44:20.113285+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'12","last_scrub_stamp":"2024-02-05T06:43:43.223933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:43.223933+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:53.049215+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":555,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.108557+0000","last_change":"2024-02-05T06:44:04.592727+0000","last_active":"2024-02-05T06:44:20.108557+0000","last_peered":"2024-02-05T06:44:20.108557+0000","last_clean":"2024-02-05T06:44:20.108557+0000","last_became_active":"2024-02-05T06:36:54.958414+0000","last_became_peered":"2024-02-05T06:36:54.958414+0000","last_unstale":"2024-02-05T06:44:20.108557+0000","last_undegraded":"2024-02-05T06:44:20.108557+0000","last_fullsized":"2024-02-05T06:44:20.108557+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:44:04.592594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:44:04.592594+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:23.618630+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"295'25","reported_seq":595,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.101309+0000","last_change":"2024-02-05T06:43:46.110585+0000","last_active":"2024-02-05T06:44:20.101309+0000","last_peered":"2024-02-05T06:44:20.101309+0000","last_clean":"2024-02-05T06:44:20.101309+0000","last_became_active":"2024-02-05T06:40:27.100610+0000","last_became_peered":"2024-02-05T06:40:27.100610+0000","last_unstale":"2024-02-05T06:44:20.101309+0000","last_undegraded":"2024-02-05T06:44:20.101309+0000","last_fullsized":"2024-02-05T06:44:20.101309+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'25","last_scrub_stamp":"2024-02-05T06:43:46.110443+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-02-05T06:34:50.718768+0000","last_clean_scrub_stamp":"2024-02-05T06:43:46.110443+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:54.008936+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":538,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.111250+0000","last_change":"2024-02-05T06:41:43.414961+0000","last_active":"2024-02-05T06:44:20.111250+0000","last_peered":"2024-02-05T06:44:20.111250+0000","last_clean":"2024-02-05T06:44:20.111250+0000","last_became_active":"2024-02-05T06:41:43.414638+0000","last_became_peered":"2024-02-05T06:41:43.414638+0000","last_unstale":"2024-02-05T06:44:20.111250+0000","last_undegraded":"2024-02-05T06:44:20.111250+0000","last_fullsized":"2024-02-05T06:44:20.111250+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:41:34.298647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:41:34.298647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:22.567985+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":434,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.116839+0000","last_change":"2024-02-05T06:42:58.394440+0000","last_active":"2024-02-05T06:44:20.116839+0000","last_peered":"2024-02-05T06:44:20.116839+0000","last_clean":"2024-02-05T06:44:20.116839+0000","last_became_active":"2024-02-05T06:40:27.097169+0000","last_became_peered":"2024-02-05T06:40:27.097169+0000","last_unstale":"2024-02-05T06:44:20.116839+0000","last_undegraded":"2024-02-05T06:44:20.116839+0000","last_fullsized":"2024-02-05T06:44:20.116839+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:58.394289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:08.713894+0000","last_clean_scrub_stamp":"2024-02-05T06:42:58.394289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:25.870421+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":136,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.103250+0000","last_change":"2024-02-05T06:43:44.345193+0000","last_active":"2024-02-05T06:44:20.103250+0000","last_peered":"2024-02-05T06:44:20.103250+0000","last_clean":"2024-02-05T06:44:20.103250+0000","last_became_active":"2024-02-05T06:41:43.411043+0000","last_became_peered":"2024-02-05T06:41:43.411043+0000","last_unstale":"2024-02-05T06:44:20.103250+0000","last_undegraded":"2024-02-05T06:44:20.103250+0000","last_fullsized":"2024-02-05T06:44:20.103250+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:44.345092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:43:44.345092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:05.600762+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"295'24","reported_seq":605,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.113498+0000","last_change":"2024-02-05T06:43:03.221573+0000","last_active":"2024-02-05T06:44:20.113498+0000","last_peered":"2024-02-05T06:44:20.113498+0000","last_clean":"2024-02-05T06:44:20.113498+0000","last_became_active":"2024-02-05T06:41:43.419769+0000","last_became_peered":"2024-02-05T06:41:43.419769+0000","last_unstale":"2024-02-05T06:44:20.113498+0000","last_undegraded":"2024-02-05T06:44:20.113498+0000","last_fullsized":"2024-02-05T06:44:20.113498+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'16","last_scrub_stamp":"2024-02-05T06:43:03.221466+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:43:03.221466+0000","last_clean_scrub_stamp":"2024-02-05T06:43:03.221466+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":228,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.106649+0000","last_change":"2024-02-05T06:43:48.512794+0000","last_active":"2024-02-05T06:44:20.106649+0000","last_peered":"2024-02-05T06:44:20.106649+0000","last_clean":"2024-02-05T06:44:20.106649+0000","last_became_active":"2024-02-05T06:40:27.098348+0000","last_became_peered":"2024-02-05T06:40:27.098348+0000","last_unstale":"2024-02-05T06:44:20.106649+0000","last_undegraded":"2024-02-05T06:44:20.106649+0000","last_fullsized":"2024-02-05T06:44:20.106649+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:48.512673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:48.512673+0000","last_clean_scrub_stamp":"2024-02-05T06:43:48.512673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:56.039239+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":557,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.112438+0000","last_change":"2024-02-05T06:44:01.297159+0000","last_active":"2024-02-05T06:44:20.112438+0000","last_peered":"2024-02-05T06:44:20.112438+0000","last_clean":"2024-02-05T06:44:20.112438+0000","last_became_active":"2024-02-05T06:40:27.060112+0000","last_became_peered":"2024-02-05T06:40:27.060112+0000","last_unstale":"2024-02-05T06:44:20.112438+0000","last_undegraded":"2024-02-05T06:44:20.112438+0000","last_fullsized":"2024-02-05T06:44:20.112438+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:01.297081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:44:01.297081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:14.265032+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":531,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.102734+0000","last_change":"2024-02-05T06:43:49.118084+0000","last_active":"2024-02-05T06:44:20.102734+0000","last_peered":"2024-02-05T06:44:20.102734+0000","last_clean":"2024-02-05T06:44:20.102734+0000","last_became_active":"2024-02-05T06:41:43.424459+0000","last_became_peered":"2024-02-05T06:41:43.424459+0000","last_unstale":"2024-02-05T06:44:20.102734+0000","last_undegraded":"2024-02-05T06:44:20.102734+0000","last_fullsized":"2024-02-05T06:44:20.102734+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:49.117949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:49.117949+0000","last_clean_scrub_stamp":"2024-02-05T06:43:49.117949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:17.641091+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"295'8","reported_seq":526,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.111702+0000","last_change":"2024-02-05T06:43:51.672673+0000","last_active":"2024-02-05T06:44:20.111702+0000","last_peered":"2024-02-05T06:44:20.111702+0000","last_clean":"2024-02-05T06:44:20.111702+0000","last_became_active":"2024-02-05T06:33:31.573410+0000","last_became_peered":"2024-02-05T06:33:31.573410+0000","last_unstale":"2024-02-05T06:44:20.111702+0000","last_undegraded":"2024-02-05T06:44:20.111702+0000","last_fullsized":"2024-02-05T06:44:20.111702+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'8","last_scrub_stamp":"2024-02-05T06:43:51.672524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:51.672524+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:06.073121+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":293,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.109572+0000","last_change":"2024-02-05T06:43:59.635993+0000","last_active":"2024-02-05T06:44:20.109572+0000","last_peered":"2024-02-05T06:44:20.109572+0000","last_clean":"2024-02-05T06:44:20.109572+0000","last_became_active":"2024-02-05T06:39:13.245684+0000","last_became_peered":"2024-02-05T06:39:13.245684+0000","last_unstale":"2024-02-05T06:44:20.109572+0000","last_undegraded":"2024-02-05T06:44:20.109572+0000","last_fullsized":"2024-02-05T06:44:20.109572+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:59.635733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:43:59.635733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:12.882226+0000","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":220,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.106103+0000","last_change":"2024-02-05T06:42:34.359574+0000","last_active":"2024-02-05T06:44:20.106103+0000","last_peered":"2024-02-05T06:44:20.106103+0000","last_clean":"2024-02-05T06:44:20.106103+0000","last_became_active":"2024-02-05T06:40:27.062975+0000","last_became_peered":"2024-02-05T06:40:27.062975+0000","last_unstale":"2024-02-05T06:44:20.106103+0000","last_undegraded":"2024-02-05T06:44:20.106103+0000","last_fullsized":"2024-02-05T06:44:20.106103+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:34.359494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:42:34.359494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:24.436822+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":136,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.104123+0000","last_change":"2024-02-05T06:42:33.724047+0000","last_active":"2024-02-05T06:44:20.104123+0000","last_peered":"2024-02-05T06:44:20.104123+0000","last_clean":"2024-02-05T06:44:20.104123+0000","last_became_active":"2024-02-05T06:41:43.412654+0000","last_became_peered":"2024-02-05T06:41:43.412654+0000","last_unstale":"2024-02-05T06:44:20.104123+0000","last_undegraded":"2024-02-05T06:44:20.104123+0000","last_fullsized":"2024-02-05T06:44:20.104123+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:33.723941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:33.723941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:23.460792+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"295'24","reported_seq":604,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.112857+0000","last_change":"2024-02-05T06:43:39.456286+0000","last_active":"2024-02-05T06:44:20.112857+0000","last_peered":"2024-02-05T06:44:20.112857+0000","last_clean":"2024-02-05T06:44:20.112857+0000","last_became_active":"2024-02-05T06:40:27.080906+0000","last_became_peered":"2024-02-05T06:40:27.080906+0000","last_unstale":"2024-02-05T06:44:20.112857+0000","last_undegraded":"2024-02-05T06:44:20.112857+0000","last_fullsized":"2024-02-05T06:44:20.112857+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:43:39.456190+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-02-05T06:35:22.938675+0000","last_clean_scrub_stamp":"2024-02-05T06:43:39.456190+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:57.044088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"73'1","reported_seq":528,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.101680+0000","last_change":"2024-02-05T06:42:54.980648+0000","last_active":"2024-02-05T06:44:20.101680+0000","last_peered":"2024-02-05T06:44:20.101680+0000","last_clean":"2024-02-05T06:44:20.101680+0000","last_became_active":"2024-02-05T06:41:43.409761+0000","last_became_peered":"2024-02-05T06:41:43.409761+0000","last_unstale":"2024-02-05T06:44:20.101680+0000","last_undegraded":"2024-02-05T06:44:20.101680+0000","last_fullsized":"2024-02-05T06:44:20.101680+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-02-05T06:42:54.980529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:42:54.980529+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":510,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.111741+0000","last_change":"2024-02-05T06:43:58.672945+0000","last_active":"2024-02-05T06:44:20.111741+0000","last_peered":"2024-02-05T06:44:20.111741+0000","last_clean":"2024-02-05T06:44:20.111741+0000","last_became_active":"2024-02-05T06:33:33.593901+0000","last_became_peered":"2024-02-05T06:33:33.593901+0000","last_unstale":"2024-02-05T06:44:20.111741+0000","last_undegraded":"2024-02-05T06:44:20.111741+0000","last_fullsized":"2024-02-05T06:44:20.111741+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:58.672808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:58.672808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:02.151514+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":228,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.107005+0000","last_change":"2024-02-05T06:44:00.501883+0000","last_active":"2024-02-05T06:44:20.107005+0000","last_peered":"2024-02-05T06:44:20.107005+0000","last_clean":"2024-02-05T06:44:20.107005+0000","last_became_active":"2024-02-05T06:40:27.077701+0000","last_became_peered":"2024-02-05T06:40:27.077701+0000","last_unstale":"2024-02-05T06:44:20.107005+0000","last_undegraded":"2024-02-05T06:44:20.107005+0000","last_fullsized":"2024-02-05T06:44:20.107005+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:00.501644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:44:00.501644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:09.346038+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"295'15","reported_seq":304,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.103971+0000","last_change":"2024-02-05T06:42:45.004868+0000","last_active":"2024-02-05T06:44:20.103971+0000","last_peered":"2024-02-05T06:44:20.103971+0000","last_clean":"2024-02-05T06:44:20.103971+0000","last_became_active":"2024-02-05T06:39:13.244124+0000","last_became_peered":"2024-02-05T06:39:13.244124+0000","last_unstale":"2024-02-05T06:44:20.103971+0000","last_undegraded":"2024-02-05T06:44:20.103971+0000","last_fullsized":"2024-02-05T06:44:20.103971+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:42:45.004744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:42:45.004744+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":518,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.113076+0000","last_change":"2024-02-05T06:42:36.192914+0000","last_active":"2024-02-05T06:44:20.113076+0000","last_peered":"2024-02-05T06:44:20.113076+0000","last_clean":"2024-02-05T06:44:20.113076+0000","last_became_active":"2024-02-05T06:41:43.417155+0000","last_became_peered":"2024-02-05T06:41:43.417155+0000","last_unstale":"2024-02-05T06:44:20.113076+0000","last_undegraded":"2024-02-05T06:44:20.113076+0000","last_fullsized":"2024-02-05T06:44:20.113076+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:36.192791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:42:36.192791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:23.769757+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":544,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.111527+0000","last_change":"2024-02-05T06:43:57.477466+0000","last_active":"2024-02-05T06:44:20.111527+0000","last_peered":"2024-02-05T06:44:20.111527+0000","last_clean":"2024-02-05T06:44:20.111527+0000","last_became_active":"2024-02-05T06:41:43.421433+0000","last_became_peered":"2024-02-05T06:41:43.421433+0000","last_unstale":"2024-02-05T06:44:20.111527+0000","last_undegraded":"2024-02-05T06:44:20.111527+0000","last_fullsized":"2024-02-05T06:44:20.111527+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:57.477296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:57.477296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:57.550396+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":416,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.122003+0000","last_change":"2024-02-05T06:42:58.691714+0000","last_active":"2024-02-05T06:44:20.122003+0000","last_peered":"2024-02-05T06:44:20.122003+0000","last_clean":"2024-02-05T06:44:20.122003+0000","last_became_active":"2024-02-05T06:40:27.065047+0000","last_became_peered":"2024-02-05T06:40:27.065047+0000","last_unstale":"2024-02-05T06:44:20.122003+0000","last_undegraded":"2024-02-05T06:44:20.122003+0000","last_fullsized":"2024-02-05T06:44:20.122003+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:58.691641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:58.691641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:20.891900+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"295'24","reported_seq":428,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.120237+0000","last_change":"2024-02-05T06:43:46.883158+0000","last_active":"2024-02-05T06:44:20.120237+0000","last_peered":"2024-02-05T06:44:20.120237+0000","last_clean":"2024-02-05T06:44:20.120237+0000","last_became_active":"2024-02-05T06:36:54.961610+0000","last_became_peered":"2024-02-05T06:36:54.961610+0000","last_unstale":"2024-02-05T06:44:20.120237+0000","last_undegraded":"2024-02-05T06:44:20.120237+0000","last_fullsized":"2024-02-05T06:44:20.120237+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:43:46.883030+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:35:02.589229+0000","last_clean_scrub_stamp":"2024-02-05T06:43:46.883030+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:49.410851+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":519,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.101039+0000","last_change":"2024-02-05T06:43:35.958292+0000","last_active":"2024-02-05T06:44:20.101039+0000","last_peered":"2024-02-05T06:44:20.101039+0000","last_clean":"2024-02-05T06:44:20.101039+0000","last_became_active":"2024-02-05T06:40:27.096132+0000","last_became_peered":"2024-02-05T06:40:27.096132+0000","last_unstale":"2024-02-05T06:44:20.101039+0000","last_undegraded":"2024-02-05T06:44:20.101039+0000","last_fullsized":"2024-02-05T06:44:20.101039+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:35.958174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:43:35.958174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:36.633481+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"249'10","reported_seq":172,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.102685+0000","last_change":"2024-02-05T06:42:50.652634+0000","last_active":"2024-02-05T06:44:20.102685+0000","last_peered":"2024-02-05T06:44:20.102685+0000","last_clean":"2024-02-05T06:44:20.102685+0000","last_became_active":"2024-02-05T06:41:43.418753+0000","last_became_peered":"2024-02-05T06:41:43.418753+0000","last_unstale":"2024-02-05T06:44:20.102685+0000","last_undegraded":"2024-02-05T06:44:20.102685+0000","last_fullsized":"2024-02-05T06:44:20.102685+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'10","last_scrub_stamp":"2024-02-05T06:42:50.652545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:42:50.652545+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:23.460940+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":220,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.106759+0000","last_change":"2024-02-05T06:42:41.410614+0000","last_active":"2024-02-05T06:44:20.106759+0000","last_peered":"2024-02-05T06:44:20.106759+0000","last_clean":"2024-02-05T06:44:20.106759+0000","last_became_active":"2024-02-05T06:40:27.098418+0000","last_became_peered":"2024-02-05T06:40:27.098418+0000","last_unstale":"2024-02-05T06:44:20.106759+0000","last_undegraded":"2024-02-05T06:44:20.106759+0000","last_fullsized":"2024-02-05T06:44:20.106759+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:41.410495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:41.410495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:24.436894+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"295'20","reported_seq":598,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.113673+0000","last_change":"2024-02-05T06:42:47.381961+0000","last_active":"2024-02-05T06:44:20.113673+0000","last_peered":"2024-02-05T06:44:20.113673+0000","last_clean":"2024-02-05T06:44:20.113673+0000","last_became_active":"2024-02-05T06:40:27.093418+0000","last_became_peered":"2024-02-05T06:40:27.093418+0000","last_unstale":"2024-02-05T06:44:20.113673+0000","last_undegraded":"2024-02-05T06:44:20.113673+0000","last_fullsized":"2024-02-05T06:44:20.113673+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-02-05T06:42:47.381317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:42:47.381317+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:23.769709+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":416,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.121742+0000","last_change":"2024-02-05T06:41:43.413904+0000","last_active":"2024-02-05T06:44:20.121742+0000","last_peered":"2024-02-05T06:44:20.121742+0000","last_clean":"2024-02-05T06:44:20.121742+0000","last_became_active":"2024-02-05T06:41:43.413562+0000","last_became_peered":"2024-02-05T06:41:43.413562+0000","last_unstale":"2024-02-05T06:44:20.121742+0000","last_undegraded":"2024-02-05T06:44:20.121742+0000","last_fullsized":"2024-02-05T06:44:20.121742+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:41:38.559780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:41:38.559780+0000","last_clean_scrub_stamp":"2024-02-05T06:41:38.559780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:20.891511+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"208'10","reported_seq":669,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.114000+0000","last_change":"2024-02-05T06:41:42.168574+0000","last_active":"2024-02-05T06:44:20.114000+0000","last_peered":"2024-02-05T06:44:20.114000+0000","last_clean":"2024-02-05T06:44:20.114000+0000","last_became_active":"2024-02-05T06:40:27.096584+0000","last_became_peered":"2024-02-05T06:40:27.096584+0000","last_unstale":"2024-02-05T06:44:20.114000+0000","last_undegraded":"2024-02-05T06:44:20.114000+0000","last_fullsized":"2024-02-05T06:44:20.114000+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-02-05T06:41:42.168453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:41:42.168453+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:23.769610+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":539,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.104179+0000","last_change":"2024-02-05T06:44:01.185774+0000","last_active":"2024-02-05T06:44:20.104179+0000","last_peered":"2024-02-05T06:44:20.104179+0000","last_clean":"2024-02-05T06:44:20.104179+0000","last_became_active":"2024-02-05T06:40:27.088096+0000","last_became_peered":"2024-02-05T06:40:27.088096+0000","last_unstale":"2024-02-05T06:44:20.104179+0000","last_undegraded":"2024-02-05T06:44:20.104179+0000","last_fullsized":"2024-02-05T06:44:20.104179+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:01.185653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:44:01.185653+0000","last_clean_scrub_stamp":"2024-02-05T06:44:01.185653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:04.676309+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"295'15","reported_seq":155,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.103058+0000","last_change":"2024-02-05T06:42:52.697830+0000","last_active":"2024-02-05T06:44:20.103058+0000","last_peered":"2024-02-05T06:44:20.103058+0000","last_clean":"2024-02-05T06:44:20.103058+0000","last_became_active":"2024-02-05T06:41:43.413708+0000","last_became_peered":"2024-02-05T06:41:43.413708+0000","last_unstale":"2024-02-05T06:44:20.103058+0000","last_undegraded":"2024-02-05T06:44:20.103058+0000","last_fullsized":"2024-02-05T06:44:20.103058+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:42:52.697346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:42:52.697346+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:23.460979+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":552,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.114519+0000","last_change":"2024-02-05T06:43:01.524727+0000","last_active":"2024-02-05T06:44:20.114519+0000","last_peered":"2024-02-05T06:44:20.114519+0000","last_clean":"2024-02-05T06:44:20.114519+0000","last_became_active":"2024-02-05T06:40:27.090632+0000","last_became_peered":"2024-02-05T06:40:27.090632+0000","last_unstale":"2024-02-05T06:44:20.114519+0000","last_undegraded":"2024-02-05T06:44:20.114519+0000","last_fullsized":"2024-02-05T06:44:20.114519+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:01.524512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:43:01.524512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:23.579561+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":536,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.117872+0000","last_change":"2024-02-05T06:44:01.644210+0000","last_active":"2024-02-05T06:44:20.117872+0000","last_peered":"2024-02-05T06:44:20.117872+0000","last_clean":"2024-02-05T06:44:20.117872+0000","last_became_active":"2024-02-05T06:40:27.091544+0000","last_became_peered":"2024-02-05T06:40:27.091544+0000","last_unstale":"2024-02-05T06:44:20.117872+0000","last_undegraded":"2024-02-05T06:44:20.117872+0000","last_fullsized":"2024-02-05T06:44:20.117872+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:01.644095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:44:01.644095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:07.661277+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":220,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.106782+0000","last_change":"2024-02-05T06:42:46.409374+0000","last_active":"2024-02-05T06:44:20.106782+0000","last_peered":"2024-02-05T06:44:20.106782+0000","last_clean":"2024-02-05T06:44:20.106782+0000","last_became_active":"2024-02-05T06:40:27.064388+0000","last_became_peered":"2024-02-05T06:40:27.064388+0000","last_unstale":"2024-02-05T06:44:20.106782+0000","last_undegraded":"2024-02-05T06:44:20.106782+0000","last_fullsized":"2024-02-05T06:44:20.106782+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:46.409259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:46.409259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:24.436920+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"295'15","reported_seq":580,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.101398+0000","last_change":"2024-02-05T06:44:04.244244+0000","last_active":"2024-02-05T06:44:20.101398+0000","last_peered":"2024-02-05T06:44:20.101398+0000","last_clean":"2024-02-05T06:44:20.101398+0000","last_became_active":"2024-02-05T06:40:27.101116+0000","last_became_peered":"2024-02-05T06:40:27.101116+0000","last_unstale":"2024-02-05T06:44:20.101398+0000","last_undegraded":"2024-02-05T06:44:20.101398+0000","last_fullsized":"2024-02-05T06:44:20.101398+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:44:04.244094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:44:04.244094+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:13.039537+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":220,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.107182+0000","last_change":"2024-02-05T06:42:49.428563+0000","last_active":"2024-02-05T06:44:20.107182+0000","last_peered":"2024-02-05T06:44:20.107182+0000","last_clean":"2024-02-05T06:44:20.107182+0000","last_became_active":"2024-02-05T06:40:27.063350+0000","last_became_peered":"2024-02-05T06:40:27.063350+0000","last_unstale":"2024-02-05T06:44:20.107182+0000","last_undegraded":"2024-02-05T06:44:20.107182+0000","last_fullsized":"2024-02-05T06:44:20.107182+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:49.428464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:42:49.428464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:23.430990+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":136,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.104123+0000","last_change":"2024-02-05T06:42:58.680331+0000","last_active":"2024-02-05T06:44:20.104123+0000","last_peered":"2024-02-05T06:44:20.104123+0000","last_clean":"2024-02-05T06:44:20.104123+0000","last_became_active":"2024-02-05T06:41:43.406094+0000","last_became_peered":"2024-02-05T06:41:43.406094+0000","last_unstale":"2024-02-05T06:44:20.104123+0000","last_undegraded":"2024-02-05T06:44:20.104123+0000","last_fullsized":"2024-02-05T06:44:20.104123+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:58.680262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:45.096340+0000","last_clean_scrub_stamp":"2024-02-05T06:42:58.680262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:23.461003+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":545,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.112070+0000","last_change":"2024-02-05T06:42:48.170292+0000","last_active":"2024-02-05T06:44:20.112070+0000","last_peered":"2024-02-05T06:44:20.112070+0000","last_clean":"2024-02-05T06:44:20.112070+0000","last_became_active":"2024-02-05T06:41:43.412130+0000","last_became_peered":"2024-02-05T06:41:43.412130+0000","last_unstale":"2024-02-05T06:44:20.112070+0000","last_undegraded":"2024-02-05T06:44:20.112070+0000","last_fullsized":"2024-02-05T06:44:20.112070+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:48.170173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:48.170173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:22.431601+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"295'12","reported_seq":560,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.118044+0000","last_change":"2024-02-05T06:43:37.426033+0000","last_active":"2024-02-05T06:44:20.118044+0000","last_peered":"2024-02-05T06:44:20.118044+0000","last_clean":"2024-02-05T06:44:20.118044+0000","last_became_active":"2024-02-05T06:40:27.091689+0000","last_became_peered":"2024-02-05T06:40:27.091689+0000","last_unstale":"2024-02-05T06:44:20.118044+0000","last_undegraded":"2024-02-05T06:44:20.118044+0000","last_fullsized":"2024-02-05T06:44:20.118044+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'12","last_scrub_stamp":"2024-02-05T06:43:37.425879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:37.425879+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:38.426084+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"306'4774","reported_seq":4814,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:22.168900+0000","last_change":"2024-02-05T06:43:29.702891+0000","last_active":"2024-02-05T06:44:22.168900+0000","last_peered":"2024-02-05T06:44:22.168900+0000","last_clean":"2024-02-05T06:44:22.168900+0000","last_became_active":"2024-02-05T06:43:29.702579+0000","last_became_peered":"2024-02-05T06:43:29.702579+0000","last_unstale":"2024-02-05T06:44:22.168900+0000","last_undegraded":"2024-02-05T06:44:22.168900+0000","last_fullsized":"2024-02-05T06:44:22.168900+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4774,"log_dups_size":0,"ondisk_log_size":4774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:31.907217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312868864,"num_objects":4774,"num_object_clones":0,"num_object_copies":14322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4774,"num_write_kb":305536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":136,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.103567+0000","last_change":"2024-02-05T06:42:59.672197+0000","last_active":"2024-02-05T06:44:20.103567+0000","last_peered":"2024-02-05T06:44:20.103567+0000","last_clean":"2024-02-05T06:44:20.103567+0000","last_became_active":"2024-02-05T06:41:43.421231+0000","last_became_peered":"2024-02-05T06:41:43.421231+0000","last_unstale":"2024-02-05T06:44:20.103567+0000","last_undegraded":"2024-02-05T06:44:20.103567+0000","last_fullsized":"2024-02-05T06:44:20.103567+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:59.672110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:42:59.672110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:23.460876+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":234,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.107149+0000","last_change":"2024-02-05T06:43:56.531681+0000","last_active":"2024-02-05T06:44:20.107149+0000","last_peered":"2024-02-05T06:44:20.107149+0000","last_clean":"2024-02-05T06:44:20.107149+0000","last_became_active":"2024-02-05T06:40:27.068279+0000","last_became_peered":"2024-02-05T06:40:27.068279+0000","last_unstale":"2024-02-05T06:44:20.107149+0000","last_undegraded":"2024-02-05T06:44:20.107149+0000","last_fullsized":"2024-02-05T06:44:20.107149+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-02-05T06:43:56.531539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:43:56.531539+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:24.413854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"295'27","reported_seq":170,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.103170+0000","last_change":"2024-02-05T06:43:43.345357+0000","last_active":"2024-02-05T06:44:20.103170+0000","last_peered":"2024-02-05T06:44:20.103170+0000","last_clean":"2024-02-05T06:44:20.103170+0000","last_became_active":"2024-02-05T06:41:43.412579+0000","last_became_peered":"2024-02-05T06:41:43.412579+0000","last_unstale":"2024-02-05T06:44:20.103170+0000","last_undegraded":"2024-02-05T06:44:20.103170+0000","last_fullsized":"2024-02-05T06:44:20.103170+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'27","last_scrub_stamp":"2024-02-05T06:43:43.344878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:43.344878+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:55.975676+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"208'12","reported_seq":271,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.106687+0000","last_change":"2024-02-05T06:42:50.411738+0000","last_active":"2024-02-05T06:44:20.106687+0000","last_peered":"2024-02-05T06:44:20.106687+0000","last_clean":"2024-02-05T06:44:20.106687+0000","last_became_active":"2024-02-05T06:40:27.064195+0000","last_became_peered":"2024-02-05T06:40:27.064195+0000","last_unstale":"2024-02-05T06:44:20.106687+0000","last_undegraded":"2024-02-05T06:44:20.106687+0000","last_fullsized":"2024-02-05T06:44:20.106687+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'12","last_scrub_stamp":"2024-02-05T06:42:50.411654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:42:50.411654+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:24.436861+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"306'4812","reported_seq":4848,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:21.850811+0000","last_change":"2024-02-05T06:43:29.697813+0000","last_active":"2024-02-05T06:44:21.850811+0000","last_peered":"2024-02-05T06:44:21.850811+0000","last_clean":"2024-02-05T06:44:21.850811+0000","last_became_active":"2024-02-05T06:43:29.697495+0000","last_became_peered":"2024-02-05T06:43:29.697495+0000","last_unstale":"2024-02-05T06:44:21.850811+0000","last_undegraded":"2024-02-05T06:44:21.850811+0000","last_fullsized":"2024-02-05T06:44:21.850811+0000","mapping_epoch":291,"log_start":"296'100","ondisk_log_start":"296'100","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4712,"log_dups_size":0,"ondisk_log_size":4712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:32.348493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315359232,"num_objects":4812,"num_object_clones":0,"num_object_copies":14436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4812,"num_write_kb":307968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":518,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.112011+0000","last_change":"2024-02-05T06:43:47.662237+0000","last_active":"2024-02-05T06:44:20.112011+0000","last_peered":"2024-02-05T06:44:20.112011+0000","last_clean":"2024-02-05T06:44:20.112011+0000","last_became_active":"2024-02-05T06:40:27.092613+0000","last_became_peered":"2024-02-05T06:40:27.092613+0000","last_unstale":"2024-02-05T06:44:20.112011+0000","last_undegraded":"2024-02-05T06:44:20.112011+0000","last_fullsized":"2024-02-05T06:44:20.112011+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:47.662025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:43:47.662025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:04.272911+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":220,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.106882+0000","last_change":"2024-02-05T06:42:43.390081+0000","last_active":"2024-02-05T06:44:20.106882+0000","last_peered":"2024-02-05T06:44:20.106882+0000","last_clean":"2024-02-05T06:44:20.106882+0000","last_became_active":"2024-02-05T06:40:27.063300+0000","last_became_peered":"2024-02-05T06:40:27.063300+0000","last_unstale":"2024-02-05T06:44:20.106882+0000","last_undegraded":"2024-02-05T06:44:20.106882+0000","last_fullsized":"2024-02-05T06:44:20.106882+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:43.389980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:43.389980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:24.437029+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"295'18","reported_seq":425,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.119646+0000","last_change":"2024-02-05T06:41:52.647443+0000","last_active":"2024-02-05T06:44:20.119646+0000","last_peered":"2024-02-05T06:44:20.119646+0000","last_clean":"2024-02-05T06:44:20.119646+0000","last_became_active":"2024-02-05T06:36:54.954071+0000","last_became_peered":"2024-02-05T06:36:54.954071+0000","last_unstale":"2024-02-05T06:44:20.119646+0000","last_undegraded":"2024-02-05T06:44:20.119646+0000","last_fullsized":"2024-02-05T06:44:20.119646+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:41:52.647335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:41:52.647335+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:20.891759+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":524,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.114229+0000","last_change":"2024-02-05T06:43:44.383974+0000","last_active":"2024-02-05T06:44:20.114229+0000","last_peered":"2024-02-05T06:44:20.114229+0000","last_clean":"2024-02-05T06:44:20.114229+0000","last_became_active":"2024-02-05T06:33:33.592546+0000","last_became_peered":"2024-02-05T06:33:33.592546+0000","last_unstale":"2024-02-05T06:44:20.114229+0000","last_undegraded":"2024-02-05T06:44:20.114229+0000","last_fullsized":"2024-02-05T06:44:20.114229+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:44.382909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:44.382909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:50.463660+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"306'4732","reported_seq":4772,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:21.250502+0000","last_change":"2024-02-05T06:43:29.702980+0000","last_active":"2024-02-05T06:44:21.250502+0000","last_peered":"2024-02-05T06:44:21.250502+0000","last_clean":"2024-02-05T06:44:21.250502+0000","last_became_active":"2024-02-05T06:43:29.702659+0000","last_became_peered":"2024-02-05T06:43:29.702659+0000","last_unstale":"2024-02-05T06:44:21.250502+0000","last_undegraded":"2024-02-05T06:44:21.250502+0000","last_fullsized":"2024-02-05T06:44:21.250502+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4732,"log_dups_size":0,"ondisk_log_size":4732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:50.762568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310116352,"num_objects":4732,"num_object_clones":0,"num_object_copies":14196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4732,"num_write_kb":302848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":220,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.106210+0000","last_change":"2024-02-05T06:42:58.350505+0000","last_active":"2024-02-05T06:44:20.106210+0000","last_peered":"2024-02-05T06:44:20.106210+0000","last_clean":"2024-02-05T06:44:20.106210+0000","last_became_active":"2024-02-05T06:40:27.218081+0000","last_became_peered":"2024-02-05T06:40:27.218081+0000","last_unstale":"2024-02-05T06:44:20.106210+0000","last_undegraded":"2024-02-05T06:44:20.106210+0000","last_fullsized":"2024-02-05T06:44:20.106210+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:58.350239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:32.925000+0000","last_clean_scrub_stamp":"2024-02-05T06:42:58.350239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:24.436975+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":550,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.112649+0000","last_change":"2024-02-05T06:44:01.453193+0000","last_active":"2024-02-05T06:44:20.112649+0000","last_peered":"2024-02-05T06:44:20.112649+0000","last_clean":"2024-02-05T06:44:20.112649+0000","last_became_active":"2024-02-05T06:41:43.413095+0000","last_became_peered":"2024-02-05T06:41:43.413095+0000","last_unstale":"2024-02-05T06:44:20.112649+0000","last_undegraded":"2024-02-05T06:44:20.112649+0000","last_fullsized":"2024-02-05T06:44:20.112649+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:01.452385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:44:01.452385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:30.760517+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":226,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.107102+0000","last_change":"2024-02-05T06:43:02.361335+0000","last_active":"2024-02-05T06:44:20.107102+0000","last_peered":"2024-02-05T06:44:20.107102+0000","last_clean":"2024-02-05T06:44:20.107102+0000","last_became_active":"2024-02-05T06:41:43.418202+0000","last_became_peered":"2024-02-05T06:41:43.418202+0000","last_unstale":"2024-02-05T06:44:20.107102+0000","last_undegraded":"2024-02-05T06:44:20.107102+0000","last_fullsized":"2024-02-05T06:44:20.107102+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:02.361235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:43:02.361235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:24.437054+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"295'44","reported_seq":671,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.112584+0000","last_change":"2024-02-05T06:42:50.149138+0000","last_active":"2024-02-05T06:44:20.112584+0000","last_peered":"2024-02-05T06:44:20.112584+0000","last_clean":"2024-02-05T06:44:20.112584+0000","last_became_active":"2024-02-05T06:40:27.068417+0000","last_became_peered":"2024-02-05T06:40:27.068417+0000","last_unstale":"2024-02-05T06:44:20.112584+0000","last_undegraded":"2024-02-05T06:44:20.112584+0000","last_fullsized":"2024-02-05T06:44:20.112584+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'28","last_scrub_stamp":"2024-02-05T06:42:50.148881+0000","last_deep_scrub":"279'28","last_deep_scrub_stamp":"2024-02-05T06:42:50.148881+0000","last_clean_scrub_stamp":"2024-02-05T06:42:50.148881+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"306'4758","reported_seq":4798,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:21.280242+0000","last_change":"2024-02-05T06:43:29.843229+0000","last_active":"2024-02-05T06:44:21.280242+0000","last_peered":"2024-02-05T06:44:21.280242+0000","last_clean":"2024-02-05T06:44:21.280242+0000","last_became_active":"2024-02-05T06:43:29.842930+0000","last_became_peered":"2024-02-05T06:43:29.842930+0000","last_unstale":"2024-02-05T06:44:21.280242+0000","last_undegraded":"2024-02-05T06:44:21.280242+0000","last_fullsized":"2024-02-05T06:44:21.280242+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4758,"log_dups_size":0,"ondisk_log_size":4758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:53.746948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311820288,"num_objects":4758,"num_object_clones":0,"num_object_copies":14274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4758,"num_write_kb":304512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":519,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.101840+0000","last_change":"2024-02-05T06:44:00.315984+0000","last_active":"2024-02-05T06:44:20.101840+0000","last_peered":"2024-02-05T06:44:20.101840+0000","last_clean":"2024-02-05T06:44:20.101840+0000","last_became_active":"2024-02-05T06:40:27.129625+0000","last_became_peered":"2024-02-05T06:40:27.129625+0000","last_unstale":"2024-02-05T06:44:20.101840+0000","last_undegraded":"2024-02-05T06:44:20.101840+0000","last_fullsized":"2024-02-05T06:44:20.101840+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:00.315851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:44:00.315851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:25.409206+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":544,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.115309+0000","last_change":"2024-02-05T06:43:41.424681+0000","last_active":"2024-02-05T06:44:20.115309+0000","last_peered":"2024-02-05T06:44:20.115309+0000","last_clean":"2024-02-05T06:44:20.115309+0000","last_became_active":"2024-02-05T06:36:54.966016+0000","last_became_peered":"2024-02-05T06:36:54.966016+0000","last_unstale":"2024-02-05T06:44:20.115309+0000","last_undegraded":"2024-02-05T06:44:20.115309+0000","last_fullsized":"2024-02-05T06:44:20.115309+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:41.424561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:41.424561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:54.460356+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":136,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.102635+0000","last_change":"2024-02-05T06:42:32.759801+0000","last_active":"2024-02-05T06:44:20.102635+0000","last_peered":"2024-02-05T06:44:20.102635+0000","last_clean":"2024-02-05T06:44:20.102635+0000","last_became_active":"2024-02-05T06:41:43.412611+0000","last_became_peered":"2024-02-05T06:41:43.412611+0000","last_unstale":"2024-02-05T06:44:20.102635+0000","last_undegraded":"2024-02-05T06:44:20.102635+0000","last_fullsized":"2024-02-05T06:44:20.102635+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:32.759672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:32.759672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:23.460823+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"295'15","reported_seq":628,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.108051+0000","last_change":"2024-02-05T06:43:52.150377+0000","last_active":"2024-02-05T06:44:20.108051+0000","last_peered":"2024-02-05T06:44:20.108051+0000","last_clean":"2024-02-05T06:44:20.108051+0000","last_became_active":"2024-02-05T06:40:08.584204+0000","last_became_peered":"2024-02-05T06:40:08.584204+0000","last_unstale":"2024-02-05T06:44:20.108051+0000","last_undegraded":"2024-02-05T06:44:20.108051+0000","last_fullsized":"2024-02-05T06:44:20.108051+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:43:52.150132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:52.150132+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:05.649207+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"306'4814","reported_seq":4854,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:22.173924+0000","last_change":"2024-02-05T06:43:29.703821+0000","last_active":"2024-02-05T06:44:22.173924+0000","last_peered":"2024-02-05T06:44:22.173924+0000","last_clean":"2024-02-05T06:44:22.173924+0000","last_became_active":"2024-02-05T06:43:29.703452+0000","last_became_peered":"2024-02-05T06:43:29.703452+0000","last_unstale":"2024-02-05T06:44:22.173924+0000","last_undegraded":"2024-02-05T06:44:22.173924+0000","last_fullsized":"2024-02-05T06:44:22.173924+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4814,"log_dups_size":0,"ondisk_log_size":4814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:45.742277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315490304,"num_objects":4814,"num_object_clones":0,"num_object_copies":14442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4814,"num_write_kb":308096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":561,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.102782+0000","last_change":"2024-02-05T06:44:04.452427+0000","last_active":"2024-02-05T06:44:20.102782+0000","last_peered":"2024-02-05T06:44:20.102782+0000","last_clean":"2024-02-05T06:44:20.102782+0000","last_became_active":"2024-02-05T06:41:46.054968+0000","last_became_peered":"2024-02-05T06:41:46.054968+0000","last_unstale":"2024-02-05T06:44:20.102782+0000","last_undegraded":"2024-02-05T06:44:20.102782+0000","last_fullsized":"2024-02-05T06:44:20.102782+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:44:04.452308+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:37.282442+0000","last_clean_scrub_stamp":"2024-02-05T06:44:04.452308+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:32.675613+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":530,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.117179+0000","last_change":"2024-02-05T06:43:40.471046+0000","last_active":"2024-02-05T06:44:20.117179+0000","last_peered":"2024-02-05T06:44:20.117179+0000","last_clean":"2024-02-05T06:44:20.117179+0000","last_became_active":"2024-02-05T06:40:27.082440+0000","last_became_peered":"2024-02-05T06:40:27.082440+0000","last_unstale":"2024-02-05T06:44:20.117179+0000","last_undegraded":"2024-02-05T06:44:20.117179+0000","last_fullsized":"2024-02-05T06:44:20.117179+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:40.470922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:43:40.470922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.918600+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"182'3","reported_seq":140,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.103598+0000","last_change":"2024-02-05T06:43:40.376967+0000","last_active":"2024-02-05T06:44:20.103598+0000","last_peered":"2024-02-05T06:44:20.103598+0000","last_clean":"2024-02-05T06:44:20.103598+0000","last_became_active":"2024-02-05T06:41:43.415806+0000","last_became_peered":"2024-02-05T06:41:43.415806+0000","last_unstale":"2024-02-05T06:44:20.103598+0000","last_undegraded":"2024-02-05T06:44:20.103598+0000","last_fullsized":"2024-02-05T06:44:20.103598+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'3","last_scrub_stamp":"2024-02-05T06:43:40.376865+0000","last_deep_scrub":"182'3","last_deep_scrub_stamp":"2024-02-05T06:39:33.373415+0000","last_clean_scrub_stamp":"2024-02-05T06:43:40.376865+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:50.126421+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001220348,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":404,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.120164+0000","last_change":"2024-02-05T06:42:42.734295+0000","last_active":"2024-02-05T06:44:20.120164+0000","last_peered":"2024-02-05T06:44:20.120164+0000","last_clean":"2024-02-05T06:44:20.120164+0000","last_became_active":"2024-02-05T06:36:54.968205+0000","last_became_peered":"2024-02-05T06:36:54.968205+0000","last_unstale":"2024-02-05T06:44:20.120164+0000","last_undegraded":"2024-02-05T06:44:20.120164+0000","last_fullsized":"2024-02-05T06:44:20.120164+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:42.734048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:42.734048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:20.891870+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"295'42","reported_seq":332,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.108557+0000","last_change":"2024-02-05T06:43:51.551040+0000","last_active":"2024-02-05T06:44:20.108557+0000","last_peered":"2024-02-05T06:44:20.108557+0000","last_clean":"2024-02-05T06:44:20.108557+0000","last_became_active":"2024-02-05T06:40:08.604483+0000","last_became_peered":"2024-02-05T06:40:08.604483+0000","last_unstale":"2024-02-05T06:44:20.108557+0000","last_undegraded":"2024-02-05T06:44:20.108557+0000","last_fullsized":"2024-02-05T06:44:20.108557+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'42","last_scrub_stamp":"2024-02-05T06:43:51.550901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:51.550901+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:01.091650+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":524,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.100802+0000","last_change":"2024-02-05T06:42:42.929540+0000","last_active":"2024-02-05T06:44:20.100802+0000","last_peered":"2024-02-05T06:44:20.100802+0000","last_clean":"2024-02-05T06:44:20.100802+0000","last_became_active":"2024-02-05T06:36:54.973387+0000","last_became_peered":"2024-02-05T06:36:54.973387+0000","last_unstale":"2024-02-05T06:44:20.100802+0000","last_undegraded":"2024-02-05T06:44:20.100802+0000","last_fullsized":"2024-02-05T06:44:20.100802+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:42.929409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:42:42.929409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"306'4705","reported_seq":4745,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:21.268701+0000","last_change":"2024-02-05T06:43:29.702708+0000","last_active":"2024-02-05T06:44:21.268701+0000","last_peered":"2024-02-05T06:44:21.268701+0000","last_clean":"2024-02-05T06:44:21.268701+0000","last_became_active":"2024-02-05T06:43:29.702401+0000","last_became_peered":"2024-02-05T06:43:29.702401+0000","last_unstale":"2024-02-05T06:44:21.268701+0000","last_undegraded":"2024-02-05T06:44:21.268701+0000","last_fullsized":"2024-02-05T06:44:21.268701+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4705,"log_dups_size":0,"ondisk_log_size":4705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:55.338031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308346880,"num_objects":4705,"num_object_clones":0,"num_object_copies":14115,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4705,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4705,"num_write_kb":301120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":641,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.106622+0000","last_change":"2024-02-05T06:42:53.491927+0000","last_active":"2024-02-05T06:44:20.106622+0000","last_peered":"2024-02-05T06:44:20.106622+0000","last_clean":"2024-02-05T06:44:20.106622+0000","last_became_active":"2024-02-05T06:40:30.082467+0000","last_became_peered":"2024-02-05T06:40:30.082467+0000","last_unstale":"2024-02-05T06:44:20.106622+0000","last_undegraded":"2024-02-05T06:44:20.106622+0000","last_fullsized":"2024-02-05T06:44:20.106622+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:42:53.491791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:42:53.491791+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:24.436950+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":220,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.106990+0000","last_change":"2024-02-05T06:42:54.478248+0000","last_active":"2024-02-05T06:44:20.106990+0000","last_peered":"2024-02-05T06:44:20.106990+0000","last_clean":"2024-02-05T06:44:20.106990+0000","last_became_active":"2024-02-05T06:40:27.077966+0000","last_became_peered":"2024-02-05T06:40:27.077966+0000","last_unstale":"2024-02-05T06:44:20.106990+0000","last_undegraded":"2024-02-05T06:44:20.106990+0000","last_fullsized":"2024-02-05T06:44:20.106990+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:54.477914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:42:54.477914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:22.018169+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":527,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.110568+0000","last_change":"2024-02-05T06:43:46.552644+0000","last_active":"2024-02-05T06:44:20.110568+0000","last_peered":"2024-02-05T06:44:20.110568+0000","last_clean":"2024-02-05T06:44:20.110568+0000","last_became_active":"2024-02-05T06:33:29.551549+0000","last_became_peered":"2024-02-05T06:33:29.551549+0000","last_unstale":"2024-02-05T06:44:20.110568+0000","last_undegraded":"2024-02-05T06:44:20.110568+0000","last_fullsized":"2024-02-05T06:44:20.110568+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:46.552530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:43:46.552530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:09.476346+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":542,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.113462+0000","last_change":"2024-02-05T06:42:59.532291+0000","last_active":"2024-02-05T06:44:20.113462+0000","last_peered":"2024-02-05T06:44:20.113462+0000","last_clean":"2024-02-05T06:44:20.113462+0000","last_became_active":"2024-02-05T06:33:28.542977+0000","last_became_peered":"2024-02-05T06:33:28.542977+0000","last_unstale":"2024-02-05T06:44:20.113462+0000","last_undegraded":"2024-02-05T06:44:20.113462+0000","last_fullsized":"2024-02-05T06:44:20.113462+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:59.532192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:42:59.532192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:27.850183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091640799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"295'17","reported_seq":617,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.109175+0000","last_change":"2024-02-05T06:43:01.610636+0000","last_active":"2024-02-05T06:44:20.109175+0000","last_peered":"2024-02-05T06:44:20.109175+0000","last_clean":"2024-02-05T06:44:20.109175+0000","last_became_active":"2024-02-05T06:41:43.420458+0000","last_became_peered":"2024-02-05T06:41:43.420458+0000","last_unstale":"2024-02-05T06:44:20.109175+0000","last_undegraded":"2024-02-05T06:44:20.109175+0000","last_fullsized":"2024-02-05T06:44:20.109175+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-02-05T06:43:01.610491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:01.610491+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:21.520973+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":556,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.114925+0000","last_change":"2024-02-05T06:42:52.383735+0000","last_active":"2024-02-05T06:44:20.114925+0000","last_peered":"2024-02-05T06:44:20.114925+0000","last_clean":"2024-02-05T06:44:20.114925+0000","last_became_active":"2024-02-05T06:41:43.412387+0000","last_became_peered":"2024-02-05T06:41:43.412387+0000","last_unstale":"2024-02-05T06:44:20.114925+0000","last_undegraded":"2024-02-05T06:44:20.114925+0000","last_fullsized":"2024-02-05T06:44:20.114925+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:52.383618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:33.914422+0000","last_clean_scrub_stamp":"2024-02-05T06:42:52.383618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:24.491012+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"306'4872","reported_seq":4912,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:22.175901+0000","last_change":"2024-02-05T06:43:29.701823+0000","last_active":"2024-02-05T06:44:22.175901+0000","last_peered":"2024-02-05T06:44:22.175901+0000","last_clean":"2024-02-05T06:44:22.175901+0000","last_became_active":"2024-02-05T06:43:29.701487+0000","last_became_peered":"2024-02-05T06:43:29.701487+0000","last_unstale":"2024-02-05T06:44:22.175901+0000","last_undegraded":"2024-02-05T06:44:22.175901+0000","last_fullsized":"2024-02-05T06:44:22.175901+0000","mapping_epoch":291,"log_start":"296'400","ondisk_log_start":"296'400","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4472,"log_dups_size":0,"ondisk_log_size":4472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:50.692005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319291392,"num_objects":4872,"num_object_clones":0,"num_object_copies":14616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4872,"num_write_kb":311808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":136,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.105014+0000","last_change":"2024-02-05T06:42:37.728366+0000","last_active":"2024-02-05T06:44:20.105014+0000","last_peered":"2024-02-05T06:44:20.105014+0000","last_clean":"2024-02-05T06:44:20.105014+0000","last_became_active":"2024-02-05T06:41:43.408773+0000","last_became_peered":"2024-02-05T06:41:43.408773+0000","last_unstale":"2024-02-05T06:44:20.105014+0000","last_undegraded":"2024-02-05T06:44:20.105014+0000","last_fullsized":"2024-02-05T06:44:20.105014+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:37.728252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:42:37.728252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:23.460762+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":641,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.106407+0000","last_change":"2024-02-05T06:42:52.491767+0000","last_active":"2024-02-05T06:44:20.106407+0000","last_peered":"2024-02-05T06:44:20.106407+0000","last_clean":"2024-02-05T06:44:20.106407+0000","last_became_active":"2024-02-05T06:40:30.081178+0000","last_became_peered":"2024-02-05T06:40:30.081178+0000","last_unstale":"2024-02-05T06:44:20.106407+0000","last_undegraded":"2024-02-05T06:44:20.106407+0000","last_fullsized":"2024-02-05T06:44:20.106407+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:42:52.491322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:42:52.491322+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:24.437004+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":531,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.112351+0000","last_change":"2024-02-05T06:42:50.411721+0000","last_active":"2024-02-05T06:44:20.112351+0000","last_peered":"2024-02-05T06:44:20.112351+0000","last_clean":"2024-02-05T06:44:20.112351+0000","last_became_active":"2024-02-05T06:41:43.411335+0000","last_became_peered":"2024-02-05T06:41:43.411335+0000","last_unstale":"2024-02-05T06:44:20.112351+0000","last_undegraded":"2024-02-05T06:44:20.112351+0000","last_fullsized":"2024-02-05T06:44:20.112351+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:50.411634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:50.411634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:21.722736+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"295'32","reported_seq":799,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.113241+0000","last_change":"2024-02-05T06:43:58.304760+0000","last_active":"2024-02-05T06:44:20.113241+0000","last_peered":"2024-02-05T06:44:20.113241+0000","last_clean":"2024-02-05T06:44:20.113241+0000","last_became_active":"2024-02-05T06:40:27.061111+0000","last_became_peered":"2024-02-05T06:40:27.061111+0000","last_unstale":"2024-02-05T06:44:20.113241+0000","last_undegraded":"2024-02-05T06:44:20.113241+0000","last_fullsized":"2024-02-05T06:44:20.113241+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'32","last_scrub_stamp":"2024-02-05T06:43:58.304628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:58.304628+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:17.673610+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":234,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.106892+0000","last_change":"2024-02-05T06:43:43.529553+0000","last_active":"2024-02-05T06:44:20.106892+0000","last_peered":"2024-02-05T06:44:20.106892+0000","last_clean":"2024-02-05T06:44:20.106892+0000","last_became_active":"2024-02-05T06:41:43.418198+0000","last_became_peered":"2024-02-05T06:41:43.418198+0000","last_unstale":"2024-02-05T06:44:20.106892+0000","last_undegraded":"2024-02-05T06:44:20.106892+0000","last_fullsized":"2024-02-05T06:44:20.106892+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:43.529437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:43.529437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:00.290531+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"306'4772","reported_seq":4812,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:22.201868+0000","last_change":"2024-02-05T06:43:29.947291+0000","last_active":"2024-02-05T06:44:22.201868+0000","last_peered":"2024-02-05T06:44:22.201868+0000","last_clean":"2024-02-05T06:44:22.201868+0000","last_became_active":"2024-02-05T06:43:29.946961+0000","last_became_peered":"2024-02-05T06:43:29.946961+0000","last_unstale":"2024-02-05T06:44:22.201868+0000","last_undegraded":"2024-02-05T06:44:22.201868+0000","last_fullsized":"2024-02-05T06:44:22.201868+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4772,"log_dups_size":0,"ondisk_log_size":4772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:38.100021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312737792,"num_objects":4772,"num_object_clones":0,"num_object_copies":14316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4772,"num_write_kb":305408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":682,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.109665+0000","last_change":"2024-02-05T06:42:38.598783+0000","last_active":"2024-02-05T06:44:20.109665+0000","last_peered":"2024-02-05T06:44:20.109665+0000","last_clean":"2024-02-05T06:44:20.109665+0000","last_became_active":"2024-02-05T06:37:37.137754+0000","last_became_peered":"2024-02-05T06:37:37.137754+0000","last_unstale":"2024-02-05T06:44:20.109665+0000","last_undegraded":"2024-02-05T06:44:20.109665+0000","last_fullsized":"2024-02-05T06:44:20.109665+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:42:38.598563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:42:38.598563+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:22.568018+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":535,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.109440+0000","last_change":"2024-02-05T06:44:03.559943+0000","last_active":"2024-02-05T06:44:20.109440+0000","last_peered":"2024-02-05T06:44:20.109440+0000","last_clean":"2024-02-05T06:44:20.109440+0000","last_became_active":"2024-02-05T06:41:43.410910+0000","last_became_peered":"2024-02-05T06:41:43.410910+0000","last_unstale":"2024-02-05T06:44:20.109440+0000","last_undegraded":"2024-02-05T06:44:20.109440+0000","last_fullsized":"2024-02-05T06:44:20.109440+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:03.559834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:44:03.559834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:09.328386+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":532,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.101644+0000","last_change":"2024-02-05T06:42:49.943397+0000","last_active":"2024-02-05T06:44:20.101644+0000","last_peered":"2024-02-05T06:44:20.101644+0000","last_clean":"2024-02-05T06:44:20.101644+0000","last_became_active":"2024-02-05T06:36:54.972264+0000","last_became_peered":"2024-02-05T06:36:54.972264+0000","last_unstale":"2024-02-05T06:44:20.101644+0000","last_undegraded":"2024-02-05T06:44:20.101644+0000","last_fullsized":"2024-02-05T06:44:20.101644+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:49.942846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:49.942846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:23.146534+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":461,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.106508+0000","last_change":"2024-02-05T06:43:33.504679+0000","last_active":"2024-02-05T06:44:20.106508+0000","last_peered":"2024-02-05T06:44:20.106508+0000","last_clean":"2024-02-05T06:44:20.106508+0000","last_became_active":"2024-02-05T06:40:27.071509+0000","last_became_peered":"2024-02-05T06:40:27.071509+0000","last_unstale":"2024-02-05T06:44:20.106508+0000","last_undegraded":"2024-02-05T06:44:20.106508+0000","last_fullsized":"2024-02-05T06:44:20.106508+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T06:43:33.504581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:43:33.504581+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:37.402285+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00074151600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":240,"num_read_kb":235,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"295'24","reported_seq":552,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.101166+0000","last_change":"2024-02-05T06:42:50.901662+0000","last_active":"2024-02-05T06:44:20.101166+0000","last_peered":"2024-02-05T06:44:20.101166+0000","last_clean":"2024-02-05T06:44:20.101166+0000","last_became_active":"2024-02-05T06:33:31.564604+0000","last_became_peered":"2024-02-05T06:33:31.564604+0000","last_unstale":"2024-02-05T06:44:20.101166+0000","last_undegraded":"2024-02-05T06:44:20.101166+0000","last_fullsized":"2024-02-05T06:44:20.101166+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:42:50.901560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:42:50.901560+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":136,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.105038+0000","last_change":"2024-02-05T06:43:49.441549+0000","last_active":"2024-02-05T06:44:20.105038+0000","last_peered":"2024-02-05T06:44:20.105038+0000","last_clean":"2024-02-05T06:44:20.105038+0000","last_became_active":"2024-02-05T06:41:43.417751+0000","last_became_peered":"2024-02-05T06:41:43.417751+0000","last_unstale":"2024-02-05T06:44:20.105038+0000","last_undegraded":"2024-02-05T06:44:20.105038+0000","last_fullsized":"2024-02-05T06:44:20.105038+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:49.441328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:49.441328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:50.412164+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"306'4739","reported_seq":4778,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:21.317261+0000","last_change":"2024-02-05T06:43:29.703026+0000","last_active":"2024-02-05T06:44:21.317261+0000","last_peered":"2024-02-05T06:44:21.317261+0000","last_clean":"2024-02-05T06:44:21.317261+0000","last_became_active":"2024-02-05T06:43:29.702549+0000","last_became_peered":"2024-02-05T06:43:29.702549+0000","last_unstale":"2024-02-05T06:44:21.317261+0000","last_undegraded":"2024-02-05T06:44:21.317261+0000","last_fullsized":"2024-02-05T06:44:21.317261+0000","mapping_epoch":291,"log_start":"296'100","ondisk_log_start":"296'100","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4639,"log_dups_size":0,"ondisk_log_size":4639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:47.712668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310575104,"num_objects":4739,"num_object_clones":0,"num_object_copies":14217,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4739,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4739,"num_write_kb":303296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":695,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.113570+0000","last_change":"2024-02-05T06:44:04.313089+0000","last_active":"2024-02-05T06:44:20.113570+0000","last_peered":"2024-02-05T06:44:20.113570+0000","last_clean":"2024-02-05T06:44:20.113570+0000","last_became_active":"2024-02-05T06:37:37.140011+0000","last_became_peered":"2024-02-05T06:37:37.140011+0000","last_unstale":"2024-02-05T06:44:20.113570+0000","last_undegraded":"2024-02-05T06:44:20.113570+0000","last_fullsized":"2024-02-05T06:44:20.113570+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:44:04.312963+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:38.778568+0000","last_clean_scrub_stamp":"2024-02-05T06:44:04.312963+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:32.876919+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":515,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.108732+0000","last_change":"2024-02-05T06:42:41.634187+0000","last_active":"2024-02-05T06:44:20.108732+0000","last_peered":"2024-02-05T06:44:20.108732+0000","last_clean":"2024-02-05T06:44:20.108732+0000","last_became_active":"2024-02-05T06:33:35.664858+0000","last_became_peered":"2024-02-05T06:33:35.664858+0000","last_unstale":"2024-02-05T06:44:20.108732+0000","last_undegraded":"2024-02-05T06:44:20.108732+0000","last_fullsized":"2024-02-05T06:44:20.108732+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:41.634048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:42:41.634048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:22.568068+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"66'1","reported_seq":561,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.109655+0000","last_change":"2024-02-05T06:43:50.530854+0000","last_active":"2024-02-05T06:44:20.109655+0000","last_peered":"2024-02-05T06:44:20.109655+0000","last_clean":"2024-02-05T06:44:20.109655+0000","last_became_active":"2024-02-05T06:41:43.411404+0000","last_became_peered":"2024-02-05T06:41:43.411404+0000","last_unstale":"2024-02-05T06:44:20.109655+0000","last_undegraded":"2024-02-05T06:44:20.109655+0000","last_fullsized":"2024-02-05T06:44:20.109655+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:43:50.530734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:43:50.530734+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:06.449055+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"295'29","reported_seq":663,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.109135+0000","last_change":"2024-02-05T06:43:02.045504+0000","last_active":"2024-02-05T06:44:20.109135+0000","last_peered":"2024-02-05T06:44:20.109135+0000","last_clean":"2024-02-05T06:44:20.109135+0000","last_became_active":"2024-02-05T06:41:43.409664+0000","last_became_peered":"2024-02-05T06:41:43.409664+0000","last_unstale":"2024-02-05T06:44:20.109135+0000","last_undegraded":"2024-02-05T06:44:20.109135+0000","last_fullsized":"2024-02-05T06:44:20.109135+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-02-05T06:43:02.045346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:02.045346+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:27.803252+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":530,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.112666+0000","last_change":"2024-02-05T06:42:56.157261+0000","last_active":"2024-02-05T06:44:20.112666+0000","last_peered":"2024-02-05T06:44:20.112666+0000","last_clean":"2024-02-05T06:44:20.112666+0000","last_became_active":"2024-02-05T06:41:43.411550+0000","last_became_peered":"2024-02-05T06:41:43.411550+0000","last_unstale":"2024-02-05T06:44:20.112666+0000","last_undegraded":"2024-02-05T06:44:20.112666+0000","last_fullsized":"2024-02-05T06:44:20.112666+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:56.157130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:42:56.157130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:25.026365+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"306'4736","reported_seq":4776,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:21.242736+0000","last_change":"2024-02-05T06:43:29.700909+0000","last_active":"2024-02-05T06:44:21.242736+0000","last_peered":"2024-02-05T06:44:21.242736+0000","last_clean":"2024-02-05T06:44:21.242736+0000","last_became_active":"2024-02-05T06:43:29.700331+0000","last_became_peered":"2024-02-05T06:43:29.700331+0000","last_unstale":"2024-02-05T06:44:21.242736+0000","last_undegraded":"2024-02-05T06:44:21.242736+0000","last_fullsized":"2024-02-05T06:44:21.242736+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4736,"log_dups_size":0,"ondisk_log_size":4736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:30.002331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310378496,"num_objects":4736,"num_object_clones":0,"num_object_copies":14208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4736,"num_write_kb":303104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":714,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.113396+0000","last_change":"2024-02-05T06:42:49.138308+0000","last_active":"2024-02-05T06:44:20.113396+0000","last_peered":"2024-02-05T06:44:20.113396+0000","last_clean":"2024-02-05T06:44:20.113396+0000","last_became_active":"2024-02-05T06:40:30.085438+0000","last_became_peered":"2024-02-05T06:40:30.085438+0000","last_unstale":"2024-02-05T06:44:20.113396+0000","last_undegraded":"2024-02-05T06:44:20.113396+0000","last_fullsized":"2024-02-05T06:44:20.113396+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:42:49.138204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:42:49.138204+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":508,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.101783+0000","last_change":"2024-02-05T06:43:42.125866+0000","last_active":"2024-02-05T06:44:20.101783+0000","last_peered":"2024-02-05T06:44:20.101783+0000","last_clean":"2024-02-05T06:44:20.101783+0000","last_became_active":"2024-02-05T06:33:35.600485+0000","last_became_peered":"2024-02-05T06:33:35.600485+0000","last_unstale":"2024-02-05T06:44:20.101783+0000","last_undegraded":"2024-02-05T06:44:20.101783+0000","last_fullsized":"2024-02-05T06:44:20.101783+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:42.125756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:43:42.125756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:04.704617+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":136,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.102889+0000","last_change":"2024-02-05T06:43:35.458031+0000","last_active":"2024-02-05T06:44:20.102889+0000","last_peered":"2024-02-05T06:44:20.102889+0000","last_clean":"2024-02-05T06:44:20.102889+0000","last_became_active":"2024-02-05T06:41:43.411732+0000","last_became_peered":"2024-02-05T06:41:43.411732+0000","last_unstale":"2024-02-05T06:44:20.102889+0000","last_undegraded":"2024-02-05T06:44:20.102889+0000","last_fullsized":"2024-02-05T06:44:20.102889+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:35.457940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:28.353956+0000","last_clean_scrub_stamp":"2024-02-05T06:43:35.457940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:04.715232+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"295'20","reported_seq":625,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.111657+0000","last_change":"2024-02-05T06:43:35.467498+0000","last_active":"2024-02-05T06:44:20.111657+0000","last_peered":"2024-02-05T06:44:20.111657+0000","last_clean":"2024-02-05T06:44:20.111657+0000","last_became_active":"2024-02-05T06:41:43.421030+0000","last_became_peered":"2024-02-05T06:41:43.421030+0000","last_unstale":"2024-02-05T06:44:20.111657+0000","last_undegraded":"2024-02-05T06:44:20.111657+0000","last_fullsized":"2024-02-05T06:44:20.111657+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-02-05T06:43:35.467372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:35.467372+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:00.825350+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"249'10","reported_seq":671,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.106342+0000","last_change":"2024-02-05T06:42:48.013461+0000","last_active":"2024-02-05T06:44:20.106342+0000","last_peered":"2024-02-05T06:44:20.106342+0000","last_clean":"2024-02-05T06:44:20.106342+0000","last_became_active":"2024-02-05T06:41:43.423638+0000","last_became_peered":"2024-02-05T06:41:43.423638+0000","last_unstale":"2024-02-05T06:44:20.106342+0000","last_undegraded":"2024-02-05T06:44:20.106342+0000","last_fullsized":"2024-02-05T06:44:20.106342+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'10","last_scrub_stamp":"2024-02-05T06:42:48.013329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:42:48.013329+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"306'4776","reported_seq":4816,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:22.182495+0000","last_change":"2024-02-05T06:43:29.702358+0000","last_active":"2024-02-05T06:44:22.182495+0000","last_peered":"2024-02-05T06:44:22.182495+0000","last_clean":"2024-02-05T06:44:22.182495+0000","last_became_active":"2024-02-05T06:43:29.701716+0000","last_became_peered":"2024-02-05T06:43:29.701716+0000","last_unstale":"2024-02-05T06:44:22.182495+0000","last_undegraded":"2024-02-05T06:44:22.182495+0000","last_fullsized":"2024-02-05T06:44:22.182495+0000","mapping_epoch":291,"log_start":"296'300","ondisk_log_start":"296'300","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4476,"log_dups_size":0,"ondisk_log_size":4476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:49.602994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312999936,"num_objects":4776,"num_object_clones":0,"num_object_copies":14328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4776,"num_write_kb":305664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":725,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.113449+0000","last_change":"2024-02-05T06:42:49.448963+0000","last_active":"2024-02-05T06:44:20.113449+0000","last_peered":"2024-02-05T06:44:20.113449+0000","last_clean":"2024-02-05T06:44:20.113449+0000","last_became_active":"2024-02-05T06:41:46.053837+0000","last_became_peered":"2024-02-05T06:41:46.053837+0000","last_unstale":"2024-02-05T06:44:20.113449+0000","last_undegraded":"2024-02-05T06:44:20.113449+0000","last_fullsized":"2024-02-05T06:44:20.113449+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:42:49.440873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:42:49.440873+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":220,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.106734+0000","last_change":"2024-02-05T06:42:59.397727+0000","last_active":"2024-02-05T06:44:20.106734+0000","last_peered":"2024-02-05T06:44:20.106734+0000","last_clean":"2024-02-05T06:44:20.106734+0000","last_became_active":"2024-02-05T06:40:27.099168+0000","last_became_peered":"2024-02-05T06:40:27.099168+0000","last_unstale":"2024-02-05T06:44:20.106734+0000","last_undegraded":"2024-02-05T06:44:20.106734+0000","last_fullsized":"2024-02-05T06:44:20.106734+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:59.397620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:42:59.397620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:20.510755+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"95'2","reported_seq":542,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.101216+0000","last_change":"2024-02-05T06:43:34.938744+0000","last_active":"2024-02-05T06:44:20.101216+0000","last_peered":"2024-02-05T06:44:20.101216+0000","last_clean":"2024-02-05T06:44:20.101216+0000","last_became_active":"2024-02-05T06:36:54.975093+0000","last_became_peered":"2024-02-05T06:36:54.975093+0000","last_unstale":"2024-02-05T06:44:20.101216+0000","last_undegraded":"2024-02-05T06:44:20.101216+0000","last_fullsized":"2024-02-05T06:44:20.101216+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-05T06:43:34.938667+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-05T06:35:25.616708+0000","last_clean_scrub_stamp":"2024-02-05T06:43:34.938667+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:02.866138+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"295'28","reported_seq":485,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.119727+0000","last_change":"2024-02-05T06:42:59.726887+0000","last_active":"2024-02-05T06:44:20.119727+0000","last_peered":"2024-02-05T06:44:20.119727+0000","last_clean":"2024-02-05T06:44:20.119727+0000","last_became_active":"2024-02-05T06:41:43.411540+0000","last_became_peered":"2024-02-05T06:41:43.411540+0000","last_unstale":"2024-02-05T06:44:20.119727+0000","last_undegraded":"2024-02-05T06:44:20.119727+0000","last_fullsized":"2024-02-05T06:44:20.119727+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'14","last_scrub_stamp":"2024-02-05T06:42:59.726776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:42:59.726776+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:20.891933+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"119'10","reported_seq":672,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.113172+0000","last_change":"2024-02-05T06:43:01.414791+0000","last_active":"2024-02-05T06:44:20.113172+0000","last_peered":"2024-02-05T06:44:20.113172+0000","last_clean":"2024-02-05T06:44:20.113172+0000","last_became_active":"2024-02-05T06:36:54.954819+0000","last_became_peered":"2024-02-05T06:36:54.954819+0000","last_unstale":"2024-02-05T06:44:20.113172+0000","last_undegraded":"2024-02-05T06:44:20.113172+0000","last_fullsized":"2024-02-05T06:44:20.113172+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-02-05T06:43:01.414710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:01.414710+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:21.722704+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"306'4725","reported_seq":4765,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:21.295526+0000","last_change":"2024-02-05T06:43:29.704287+0000","last_active":"2024-02-05T06:44:21.295526+0000","last_peered":"2024-02-05T06:44:21.295526+0000","last_clean":"2024-02-05T06:44:21.295526+0000","last_became_active":"2024-02-05T06:43:29.703700+0000","last_became_peered":"2024-02-05T06:43:29.703700+0000","last_unstale":"2024-02-05T06:44:21.295526+0000","last_undegraded":"2024-02-05T06:44:21.295526+0000","last_fullsized":"2024-02-05T06:44:21.295526+0000","mapping_epoch":291,"log_start":"296'100","ondisk_log_start":"296'100","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4625,"log_dups_size":0,"ondisk_log_size":4625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:36.320886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309657600,"num_objects":4725,"num_object_clones":0,"num_object_copies":14175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4725,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4725,"num_write_kb":302400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":674,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.120448+0000","last_change":"2024-02-05T06:41:28.466905+0000","last_active":"2024-02-05T06:44:20.120448+0000","last_peered":"2024-02-05T06:44:20.120448+0000","last_clean":"2024-02-05T06:44:20.120448+0000","last_became_active":"2024-02-05T06:37:37.135073+0000","last_became_peered":"2024-02-05T06:37:37.135073+0000","last_unstale":"2024-02-05T06:44:20.120448+0000","last_undegraded":"2024-02-05T06:44:20.120448+0000","last_fullsized":"2024-02-05T06:44:20.120448+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:41:28.466576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:41:28.466576+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:20.891626+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":404,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.119381+0000","last_change":"2024-02-05T06:42:45.696347+0000","last_active":"2024-02-05T06:44:20.119381+0000","last_peered":"2024-02-05T06:44:20.119381+0000","last_clean":"2024-02-05T06:44:20.119381+0000","last_became_active":"2024-02-05T06:36:54.958644+0000","last_became_peered":"2024-02-05T06:36:54.958644+0000","last_unstale":"2024-02-05T06:44:20.119381+0000","last_undegraded":"2024-02-05T06:44:20.119381+0000","last_fullsized":"2024-02-05T06:44:20.119381+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:45.696236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:24.090670+0000","last_clean_scrub_stamp":"2024-02-05T06:42:45.696236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":537,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.117700+0000","last_change":"2024-02-05T06:42:56.396835+0000","last_active":"2024-02-05T06:44:20.117700+0000","last_peered":"2024-02-05T06:44:20.117700+0000","last_clean":"2024-02-05T06:44:20.117700+0000","last_became_active":"2024-02-05T06:40:27.093739+0000","last_became_peered":"2024-02-05T06:40:27.093739+0000","last_unstale":"2024-02-05T06:44:20.117700+0000","last_undegraded":"2024-02-05T06:44:20.117700+0000","last_fullsized":"2024-02-05T06:44:20.117700+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:56.396575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:38:28.150695+0000","last_clean_scrub_stamp":"2024-02-05T06:42:56.396575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:23.769790+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"295'30","reported_seq":613,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.114040+0000","last_change":"2024-02-05T06:43:46.582784+0000","last_active":"2024-02-05T06:44:20.114040+0000","last_peered":"2024-02-05T06:44:20.114040+0000","last_clean":"2024-02-05T06:44:20.114040+0000","last_became_active":"2024-02-05T06:40:27.277747+0000","last_became_peered":"2024-02-05T06:40:27.277747+0000","last_unstale":"2024-02-05T06:44:20.114040+0000","last_undegraded":"2024-02-05T06:44:20.114040+0000","last_fullsized":"2024-02-05T06:44:20.114040+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'30","last_scrub_stamp":"2024-02-05T06:43:46.582265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:46.582265+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:14.477223+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":424,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.119804+0000","last_change":"2024-02-05T06:42:52.703205+0000","last_active":"2024-02-05T06:44:20.119804+0000","last_peered":"2024-02-05T06:44:20.119804+0000","last_clean":"2024-02-05T06:44:20.119804+0000","last_became_active":"2024-02-05T06:40:27.073889+0000","last_became_peered":"2024-02-05T06:40:27.073889+0000","last_unstale":"2024-02-05T06:44:20.119804+0000","last_undegraded":"2024-02-05T06:44:20.119804+0000","last_fullsized":"2024-02-05T06:44:20.119804+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:52.703089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:42:52.703089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"306'4708","reported_seq":4748,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:22.194654+0000","last_change":"2024-02-05T06:43:29.843237+0000","last_active":"2024-02-05T06:44:22.194654+0000","last_peered":"2024-02-05T06:44:22.194654+0000","last_clean":"2024-02-05T06:44:22.194654+0000","last_became_active":"2024-02-05T06:43:29.842722+0000","last_became_peered":"2024-02-05T06:43:29.842722+0000","last_unstale":"2024-02-05T06:44:22.194654+0000","last_undegraded":"2024-02-05T06:44:22.194654+0000","last_fullsized":"2024-02-05T06:44:22.194654+0000","mapping_epoch":291,"log_start":"296'200","ondisk_log_start":"296'200","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4508,"log_dups_size":0,"ondisk_log_size":4508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:33.230195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308543488,"num_objects":4708,"num_object_clones":0,"num_object_copies":14124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4708,"num_write_kb":301312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":555,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.103669+0000","last_change":"2024-02-05T06:43:59.471960+0000","last_active":"2024-02-05T06:44:20.103669+0000","last_peered":"2024-02-05T06:44:20.103669+0000","last_clean":"2024-02-05T06:44:20.103669+0000","last_became_active":"2024-02-05T06:41:46.053993+0000","last_became_peered":"2024-02-05T06:41:46.053993+0000","last_unstale":"2024-02-05T06:44:20.103669+0000","last_undegraded":"2024-02-05T06:44:20.103669+0000","last_fullsized":"2024-02-05T06:44:20.103669+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:43:59.471749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:43:59.471749+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:04.010257+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":144,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.102783+0000","last_change":"2024-02-05T06:44:00.522165+0000","last_active":"2024-02-05T06:44:20.102783+0000","last_peered":"2024-02-05T06:44:20.102783+0000","last_clean":"2024-02-05T06:44:20.102783+0000","last_became_active":"2024-02-05T06:41:43.418066+0000","last_became_peered":"2024-02-05T06:41:43.418066+0000","last_unstale":"2024-02-05T06:44:20.102783+0000","last_undegraded":"2024-02-05T06:44:20.102783+0000","last_fullsized":"2024-02-05T06:44:20.102783+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:00.522003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:44:00.522003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:11.908885+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"249'12","reported_seq":519,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.121505+0000","last_change":"2024-02-05T06:42:56.729086+0000","last_active":"2024-02-05T06:44:20.121505+0000","last_peered":"2024-02-05T06:44:20.121505+0000","last_clean":"2024-02-05T06:44:20.121505+0000","last_became_active":"2024-02-05T06:41:43.411973+0000","last_became_peered":"2024-02-05T06:41:43.411973+0000","last_unstale":"2024-02-05T06:44:20.121505+0000","last_undegraded":"2024-02-05T06:44:20.121505+0000","last_fullsized":"2024-02-05T06:44:20.121505+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'12","last_scrub_stamp":"2024-02-05T06:42:56.728967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:42:56.728967+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":531,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.111834+0000","last_change":"2024-02-05T06:43:43.544887+0000","last_active":"2024-02-05T06:44:20.111834+0000","last_peered":"2024-02-05T06:44:20.111834+0000","last_clean":"2024-02-05T06:44:20.111834+0000","last_became_active":"2024-02-05T06:40:27.213439+0000","last_became_peered":"2024-02-05T06:40:27.213439+0000","last_unstale":"2024-02-05T06:44:20.111834+0000","last_undegraded":"2024-02-05T06:44:20.111834+0000","last_fullsized":"2024-02-05T06:44:20.111834+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:43.544765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:43:43.544765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:50.636394+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"295'34","reported_seq":326,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.101348+0000","last_change":"2024-02-05T06:42:59.970087+0000","last_active":"2024-02-05T06:44:20.101348+0000","last_peered":"2024-02-05T06:44:20.101348+0000","last_clean":"2024-02-05T06:44:20.101348+0000","last_became_active":"2024-02-05T06:39:13.257978+0000","last_became_peered":"2024-02-05T06:39:13.257978+0000","last_unstale":"2024-02-05T06:44:20.101348+0000","last_undegraded":"2024-02-05T06:44:20.101348+0000","last_fullsized":"2024-02-05T06:44:20.101348+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'22","last_scrub_stamp":"2024-02-05T06:42:59.969994+0000","last_deep_scrub":"281'22","last_deep_scrub_stamp":"2024-02-05T06:42:59.969994+0000","last_clean_scrub_stamp":"2024-02-05T06:42:59.969994+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:20.164680+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.3","version":"306'4673","reported_seq":4713,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:21.295126+0000","last_change":"2024-02-05T06:43:29.702128+0000","last_active":"2024-02-05T06:44:21.295126+0000","last_peered":"2024-02-05T06:44:21.295126+0000","last_clean":"2024-02-05T06:44:21.295126+0000","last_became_active":"2024-02-05T06:43:29.701677+0000","last_became_peered":"2024-02-05T06:43:29.701677+0000","last_unstale":"2024-02-05T06:44:21.295126+0000","last_undegraded":"2024-02-05T06:44:21.295126+0000","last_fullsized":"2024-02-05T06:44:21.295126+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4673,"log_dups_size":0,"ondisk_log_size":4673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:32.217143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306249728,"num_objects":4673,"num_object_clones":0,"num_object_copies":14019,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4673,"num_write_kb":299072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":524,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.113593+0000","last_change":"2024-02-05T06:43:42.559000+0000","last_active":"2024-02-05T06:44:20.113593+0000","last_peered":"2024-02-05T06:44:20.113593+0000","last_clean":"2024-02-05T06:44:20.113593+0000","last_became_active":"2024-02-05T06:41:43.410600+0000","last_became_peered":"2024-02-05T06:41:43.410600+0000","last_unstale":"2024-02-05T06:44:20.113593+0000","last_undegraded":"2024-02-05T06:44:20.113593+0000","last_fullsized":"2024-02-05T06:44:20.113593+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:42.558880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:20.042534+0000","last_clean_scrub_stamp":"2024-02-05T06:43:42.558880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:54.376149+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":136,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.104988+0000","last_change":"2024-02-05T06:42:55.610389+0000","last_active":"2024-02-05T06:44:20.104988+0000","last_peered":"2024-02-05T06:44:20.104988+0000","last_clean":"2024-02-05T06:44:20.104988+0000","last_became_active":"2024-02-05T06:41:43.417343+0000","last_became_peered":"2024-02-05T06:41:43.417343+0000","last_unstale":"2024-02-05T06:44:20.104988+0000","last_undegraded":"2024-02-05T06:44:20.104988+0000","last_fullsized":"2024-02-05T06:44:20.104988+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:55.610276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:42:55.610276+0000","last_clean_scrub_stamp":"2024-02-05T06:42:55.610276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:23.460906+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":246,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.108397+0000","last_change":"2024-02-05T06:44:01.550712+0000","last_active":"2024-02-05T06:44:20.108397+0000","last_peered":"2024-02-05T06:44:20.108397+0000","last_clean":"2024-02-05T06:44:20.108397+0000","last_became_active":"2024-02-05T06:40:27.086808+0000","last_became_peered":"2024-02-05T06:40:27.086808+0000","last_unstale":"2024-02-05T06:44:20.108397+0000","last_undegraded":"2024-02-05T06:44:20.108397+0000","last_fullsized":"2024-02-05T06:44:20.108397+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:01.550518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:44:01.550518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:15.652514+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"295'24","reported_seq":428,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.121400+0000","last_change":"2024-02-05T06:43:57.825625+0000","last_active":"2024-02-05T06:44:20.121400+0000","last_peered":"2024-02-05T06:44:20.121400+0000","last_clean":"2024-02-05T06:44:20.121400+0000","last_became_active":"2024-02-05T06:36:54.958582+0000","last_became_peered":"2024-02-05T06:36:54.958582+0000","last_unstale":"2024-02-05T06:44:20.121400+0000","last_undegraded":"2024-02-05T06:44:20.121400+0000","last_fullsized":"2024-02-05T06:44:20.121400+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:43:57.825491+0000","last_deep_scrub":"115'16","last_deep_scrub_stamp":"2024-02-05T06:36:34.793243+0000","last_clean_scrub_stamp":"2024-02-05T06:43:57.825491+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:21.940154+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"306'4772","reported_seq":4812,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:21.827755+0000","last_change":"2024-02-05T06:43:29.947307+0000","last_active":"2024-02-05T06:44:21.827755+0000","last_peered":"2024-02-05T06:44:21.827755+0000","last_clean":"2024-02-05T06:44:21.827755+0000","last_became_active":"2024-02-05T06:43:29.946914+0000","last_became_peered":"2024-02-05T06:43:29.946914+0000","last_unstale":"2024-02-05T06:44:21.827755+0000","last_undegraded":"2024-02-05T06:44:21.827755+0000","last_fullsized":"2024-02-05T06:44:21.827755+0000","mapping_epoch":291,"log_start":"296'100","ondisk_log_start":"296'100","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4672,"log_dups_size":0,"ondisk_log_size":4672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:52.324646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312737792,"num_objects":4772,"num_object_clones":0,"num_object_copies":14316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4772,"num_write_kb":305408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"83'128","reported_seq":559,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.103083+0000","last_change":"2024-02-05T06:43:47.429770+0000","last_active":"2024-02-05T06:44:20.103083+0000","last_peered":"2024-02-05T06:44:20.103083+0000","last_clean":"2024-02-05T06:44:20.103083+0000","last_became_active":"2024-02-05T06:41:46.053324+0000","last_became_peered":"2024-02-05T06:41:46.053324+0000","last_unstale":"2024-02-05T06:44:20.103083+0000","last_undegraded":"2024-02-05T06:44:20.103083+0000","last_fullsized":"2024-02-05T06:44:20.103083+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:43:47.429623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:43:47.429623+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:09.999835+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":540,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.113305+0000","last_change":"2024-02-05T06:43:46.502730+0000","last_active":"2024-02-05T06:44:20.113305+0000","last_peered":"2024-02-05T06:44:20.113305+0000","last_clean":"2024-02-05T06:44:20.113305+0000","last_became_active":"2024-02-05T06:36:54.958805+0000","last_became_peered":"2024-02-05T06:36:54.958805+0000","last_unstale":"2024-02-05T06:44:20.113305+0000","last_undegraded":"2024-02-05T06:44:20.113305+0000","last_fullsized":"2024-02-05T06:44:20.113305+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:46.502588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:43:46.502588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:05.939786+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":530,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.109758+0000","last_change":"2024-02-05T06:42:46.707508+0000","last_active":"2024-02-05T06:44:20.109758+0000","last_peered":"2024-02-05T06:44:20.109758+0000","last_clean":"2024-02-05T06:44:20.109758+0000","last_became_active":"2024-02-05T06:36:54.958082+0000","last_became_peered":"2024-02-05T06:36:54.958082+0000","last_unstale":"2024-02-05T06:44:20.109758+0000","last_undegraded":"2024-02-05T06:44:20.109758+0000","last_fullsized":"2024-02-05T06:44:20.109758+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:46.707382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:42:46.707382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:22.568043+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":544,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.112767+0000","last_change":"2024-02-05T06:42:57.200247+0000","last_active":"2024-02-05T06:44:20.112767+0000","last_peered":"2024-02-05T06:44:20.112767+0000","last_clean":"2024-02-05T06:44:20.112767+0000","last_became_active":"2024-02-05T06:39:13.240038+0000","last_became_peered":"2024-02-05T06:39:13.240038+0000","last_unstale":"2024-02-05T06:44:20.112767+0000","last_undegraded":"2024-02-05T06:44:20.112767+0000","last_fullsized":"2024-02-05T06:44:20.112767+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:57.200123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:57.200123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"295'39","reported_seq":443,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.120984+0000","last_change":"2024-02-05T06:41:54.656081+0000","last_active":"2024-02-05T06:44:20.120984+0000","last_peered":"2024-02-05T06:44:20.120984+0000","last_clean":"2024-02-05T06:44:20.120984+0000","last_became_active":"2024-02-05T06:36:54.961719+0000","last_became_peered":"2024-02-05T06:36:54.961719+0000","last_unstale":"2024-02-05T06:44:20.120984+0000","last_undegraded":"2024-02-05T06:44:20.120984+0000","last_fullsized":"2024-02-05T06:44:20.120984+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'25","last_scrub_stamp":"2024-02-05T06:41:54.655986+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-02-05T06:35:26.274944+0000","last_clean_scrub_stamp":"2024-02-05T06:41:54.655986+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:20.891577+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"306'4776","reported_seq":4816,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:22.186400+0000","last_change":"2024-02-05T06:43:29.947556+0000","last_active":"2024-02-05T06:44:22.186400+0000","last_peered":"2024-02-05T06:44:22.186400+0000","last_clean":"2024-02-05T06:44:22.186400+0000","last_became_active":"2024-02-05T06:43:29.947136+0000","last_became_peered":"2024-02-05T06:43:29.947136+0000","last_unstale":"2024-02-05T06:44:22.186400+0000","last_undegraded":"2024-02-05T06:44:22.186400+0000","last_fullsized":"2024-02-05T06:44:22.186400+0000","mapping_epoch":291,"log_start":"296'300","ondisk_log_start":"296'300","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4476,"log_dups_size":0,"ondisk_log_size":4476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:35.061776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312999936,"num_objects":4776,"num_object_clones":0,"num_object_copies":14328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4776,"num_write_kb":305664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":638,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.106586+0000","last_change":"2024-02-05T06:43:47.527180+0000","last_active":"2024-02-05T06:44:20.106586+0000","last_peered":"2024-02-05T06:44:20.106586+0000","last_clean":"2024-02-05T06:44:20.106586+0000","last_became_active":"2024-02-05T06:40:30.095202+0000","last_became_peered":"2024-02-05T06:40:30.095202+0000","last_unstale":"2024-02-05T06:44:20.106586+0000","last_undegraded":"2024-02-05T06:44:20.106586+0000","last_fullsized":"2024-02-05T06:44:20.106586+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:43:47.527064+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:41:50.706159+0000","last_clean_scrub_stamp":"2024-02-05T06:43:47.527064+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:55.073159+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":404,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.121327+0000","last_change":"2024-02-05T06:42:43.732921+0000","last_active":"2024-02-05T06:44:20.121327+0000","last_peered":"2024-02-05T06:44:20.121327+0000","last_clean":"2024-02-05T06:44:20.121327+0000","last_became_active":"2024-02-05T06:36:54.961560+0000","last_became_peered":"2024-02-05T06:36:54.961560+0000","last_unstale":"2024-02-05T06:44:20.121327+0000","last_undegraded":"2024-02-05T06:44:20.121327+0000","last_fullsized":"2024-02-05T06:44:20.121327+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:43.732781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:30.057099+0000","last_clean_scrub_stamp":"2024-02-05T06:42:43.732781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:20.891818+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"249'13","reported_seq":704,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.112993+0000","last_change":"2024-02-05T06:43:02.565236+0000","last_active":"2024-02-05T06:44:20.112993+0000","last_peered":"2024-02-05T06:44:20.112993+0000","last_clean":"2024-02-05T06:44:20.112993+0000","last_became_active":"2024-02-05T06:41:43.413139+0000","last_became_peered":"2024-02-05T06:41:43.413139+0000","last_unstale":"2024-02-05T06:44:20.112993+0000","last_undegraded":"2024-02-05T06:44:20.112993+0000","last_fullsized":"2024-02-05T06:44:20.112993+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'13","last_scrub_stamp":"2024-02-05T06:43:02.565121+0000","last_deep_scrub":"92'9","last_deep_scrub_stamp":"2024-02-05T06:34:52.283780+0000","last_clean_scrub_stamp":"2024-02-05T06:43:02.565121+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:24.490979+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":530,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.111645+0000","last_change":"2024-02-05T06:42:39.205395+0000","last_active":"2024-02-05T06:44:20.111645+0000","last_peered":"2024-02-05T06:44:20.111645+0000","last_clean":"2024-02-05T06:44:20.111645+0000","last_became_active":"2024-02-05T06:36:54.971304+0000","last_became_peered":"2024-02-05T06:36:54.971304+0000","last_unstale":"2024-02-05T06:44:20.111645+0000","last_undegraded":"2024-02-05T06:44:20.111645+0000","last_fullsized":"2024-02-05T06:44:20.111645+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:39.205140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:39.205140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:23.769679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"295'25","reported_seq":431,"reported_epoch":306,"state":"active+clean","last_fresh":"2024-02-05T06:44:20.122093+0000","last_change":"2024-02-05T06:41:37.498015+0000","last_active":"2024-02-05T06:44:20.122093+0000","last_peered":"2024-02-05T06:44:20.122093+0000","last_clean":"2024-02-05T06:44:20.122093+0000","last_became_active":"2024-02-05T06:36:54.963620+0000","last_became_peered":"2024-02-05T06:36:54.963620+0000","last_unstale":"2024-02-05T06:44:20.122093+0000","last_undegraded":"2024-02-05T06:44:20.122093+0000","last_fullsized":"2024-02-05T06:44:20.122093+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'15","last_scrub_stamp":"2024-02-05T06:41:37.497882+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-02-05T06:35:04.584657+0000","last_clean_scrub_stamp":"2024-02-05T06:41:37.497882+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:20.891660+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":4990173184,"num_objects":76144,"num_object_clones":0,"num_object_copies":228432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":76144,"num_write_kb":4873216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14947319808,"data_stored":14947319808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74544,"ondisk_log_size":74544,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":240,"num_read_kb":235,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1573408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1114,"num_read_kb":871,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":373,"num_bytes_recovered":8234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":761,"ondisk_log_size":761,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103302,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3023880,"kb_used_data":2685884,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":90721272,"statfs":{"total":95995035648,"available":92898582528,"internally_reserved":0,"allocated":2750345216,"data_stored":2749737782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":346094026},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:43:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":1.198,"15min":1.292},"min":{"1min":0.29899999999999999,"5min":0.23699999999999999,"15min":0.219},"max":{"1min":2.2530000000000001,"5min":32.567,"15min":68.921000000000006},"last":2.052},{"interface":"front","average":{"1min":1.946,"5min":1.528,"15min":1.4630000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":18.013000000000002,"5min":32.603000000000002,"15min":69.616},"last":1.1180000000000001}]},{"osd":1,"last update":"Mon Feb 5 06:43:49 2024","interfaces":[{"interface":"back","average":{"1min":2.206,"5min":1.577,"15min":1.2789999999999999},"min":{"1min":0.50900000000000001,"5min":0.25,"15min":0.23599999999999999},"max":{"1min":17.969999999999999,"5min":32.459000000000003,"15min":89.231999999999999},"last":1.8520000000000001},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.724,"15min":1.369},"min":{"1min":0.372,"5min":0.26300000000000001,"15min":0.26100000000000001},"max":{"1min":4.0300000000000002,"5min":32.536000000000001,"15min":87.995999999999995},"last":1.1799999999999999}]},{"osd":2,"last update":"Mon Feb 5 06:43:49 2024","interfaces":[{"interface":"back","average":{"1min":2.1920000000000002,"5min":1.4330000000000001,"15min":1.1279999999999999},"min":{"1min":0.37,"5min":0.28399999999999997,"15min":0.26200000000000001},"max":{"1min":18.071999999999999,"5min":18.071999999999999,"15min":53.947000000000003},"last":1.978},{"interface":"front","average":{"1min":2.0739999999999998,"5min":1.1990000000000001,"15min":1.085},"min":{"1min":0.44700000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":18.045999999999999,"5min":18.045999999999999,"15min":53.299999999999997},"last":1.821}]},{"osd":3,"last update":"Mon Feb 5 06:43:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.593,"15min":1.6060000000000001},"min":{"1min":0.39400000000000002,"5min":0.32700000000000001,"15min":0.316},"max":{"1min":3.4990000000000001,"5min":19.513000000000002,"15min":89.481999999999999},"last":3.8460000000000001},{"interface":"front","average":{"1min":1.167,"5min":1.2490000000000001,"15min":1.5569999999999999},"min":{"1min":0.47799999999999998,"5min":0.30599999999999999,"15min":0.27400000000000002},"max":{"1min":3.681,"5min":12.034000000000001,"15min":92.293999999999997},"last":2.024}]},{"osd":4,"last update":"Mon Feb 5 06:43:49 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.496},"min":{"1min":0.28499999999999998,"5min":0.26200000000000001,"15min":0.24199999999999999},"max":{"1min":0.90700000000000003,"5min":2.2349999999999999,"15min":3.5409999999999999},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.54000000000000004,"15min":0.50900000000000001},"min":{"1min":0.33200000000000002,"5min":0.27100000000000002,"15min":0.23400000000000001},"max":{"1min":2.9039999999999999,"5min":2.9039999999999999,"15min":2.9039999999999999},"last":1.9430000000000001}]},{"osd":5,"last update":"Mon Feb 5 06:43:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.629,"15min":0.55000000000000004},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.25900000000000001},"max":{"1min":3.444,"5min":5.1050000000000004,"15min":5.1050000000000004},"last":1.923},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.59499999999999997,"15min":0.57899999999999996},"min":{"1min":0.36099999999999999,"5min":0.30299999999999999,"15min":0.252},"max":{"1min":2.746,"5min":2.746,"15min":3.5990000000000002},"last":1.2270000000000001}]},{"osd":6,"last update":"Mon Feb 5 06:43:49 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.67300000000000004,"15min":0.61799999999999999},"min":{"1min":0.38800000000000001,"5min":0.32000000000000001,"15min":0.314},"max":{"1min":3.2869999999999999,"5min":3.2869999999999999,"15min":3.5750000000000002},"last":1.21},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.63300000000000001,"15min":0.58499999999999996},"min":{"1min":0.38200000000000001,"5min":0.307,"15min":0.245},"max":{"1min":3.0649999999999999,"5min":3.0649999999999999,"15min":3.2189999999999999},"last":2.0070000000000001}]}]},{"osd":6,"up_from":49,"seq":210453397644,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1201840,"kb_used_data":895780,"kb_used_omap":1,"kb_used_meta":306046,"kb_avail":92543312,"statfs":{"total":95995035648,"available":94764351488,"internally_reserved":0,"allocated":917278720,"data_stored":916694413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":313391558},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:44:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":4.2670000000000003,"15min":2.0510000000000002},"min":{"1min":0.255,"5min":0.224,"15min":0.224},"max":{"1min":1.3560000000000001,"5min":98.918999999999997,"15min":98.918999999999997},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.66600000000000004,"5min":4.3739999999999997,"15min":2.0190000000000001},"min":{"1min":0.245,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.9039999999999999,"5min":98.775000000000006,"15min":98.775000000000006},"last":0.89700000000000002}]},{"osd":1,"last update":"Mon Feb 5 06:44:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":3.8380000000000001,"15min":1.776},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.119,"5min":98.843999999999994,"15min":98.843999999999994},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.80300000000000005,"5min":4.0300000000000002,"15min":1.7969999999999999},"min":{"1min":0.436,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":2.0129999999999999,"5min":101.637,"15min":101.637},"last":0.65900000000000003}]},{"osd":2,"last update":"Mon Feb 5 06:44:11 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":3.5470000000000002,"15min":1.7290000000000001},"min":{"1min":0.33900000000000002,"5min":0.29199999999999998,"15min":0.22500000000000001},"max":{"1min":1.9390000000000001,"5min":85.617000000000004,"15min":85.617000000000004},"last":0.497},{"interface":"front","average":{"1min":0.58699999999999997,"5min":3.383,"15min":1.6259999999999999},"min":{"1min":0.32600000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.365,"5min":85.335999999999999,"15min":85.335999999999999},"last":0.67200000000000004}]},{"osd":3,"last update":"Mon Feb 5 06:44:11 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":3.8809999999999998,"15min":2.21},"min":{"1min":0.32000000000000001,"5min":0.307,"15min":0.27000000000000002},"max":{"1min":4.1150000000000002,"5min":98.888999999999996,"15min":99.269000000000005},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.79900000000000004,"5min":4.0099999999999998,"15min":2.3109999999999999},"min":{"1min":0.40799999999999997,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":2.8410000000000002,"5min":98.969999999999999,"15min":101.75700000000001},"last":0.94999999999999996}]},{"osd":4,"last update":"Mon Feb 5 06:44:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55900000000000005,"15min":0.48799999999999999},"min":{"1min":0.308,"5min":0.26500000000000001,"15min":0.23100000000000001},"max":{"1min":1.391,"5min":3.742,"15min":3.742},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.53500000000000003,"15min":0.47399999999999998},"min":{"1min":0.30099999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.8480000000000001,"5min":2.371,"15min":2.371},"last":0.60999999999999999}]},{"osd":5,"last update":"Mon Feb 5 06:44:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.60899999999999999,"15min":0.51900000000000002},"min":{"1min":0.35299999999999998,"5min":0.29899999999999999,"15min":0.251},"max":{"1min":1.4450000000000001,"5min":4.4740000000000002,"15min":4.4740000000000002},"last":2.6930000000000001},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.628,"15min":0.52500000000000002},"min":{"1min":0.34999999999999998,"5min":0.28699999999999998,"15min":0.24299999999999999},"max":{"1min":1.895,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":1.2110000000000001}]},{"osd":7,"last update":"Mon Feb 5 06:43:29 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.56699999999999995,"15min":0.54700000000000004},"min":{"1min":0.36099999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.621,"5min":2.2330000000000001,"15min":6.0350000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.60299999999999998,"15min":0.60399999999999998},"min":{"1min":0.33000000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.0990000000000002,"5min":3.0990000000000002,"15min":8.5399999999999991},"last":0.70299999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575013,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3103784,"kb_used_data":2729692,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":90641368,"statfs":{"total":95995035648,"available":92816760832,"internally_reserved":0,"allocated":2795204608,"data_stored":2794565189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:44:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":2.0859999999999999,"15min":1.5569999999999999},"min":{"1min":0.247,"5min":0.183,"15min":0.16700000000000001},"max":{"1min":2.4980000000000002,"5min":43.651000000000003,"15min":67.421000000000006},"last":1.4530000000000001},{"interface":"front","average":{"1min":0.80900000000000005,"5min":2.5219999999999998,"15min":1.669},"min":{"1min":0.28499999999999998,"5min":0.20599999999999999,"15min":0.19600000000000001},"max":{"1min":4.3550000000000004,"5min":45.670999999999999,"15min":67.885000000000005},"last":1.254}]},{"osd":2,"last update":"Mon Feb 5 06:43:28 2024","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":2.2989999999999999,"15min":1.304},"min":{"1min":0.249,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.422000000000001,"5min":43.447000000000003,"15min":43.447000000000003},"last":0.53700000000000003},{"interface":"front","average":{"1min":1.9019999999999999,"5min":1.819,"15min":1.2949999999999999},"min":{"1min":0.29399999999999998,"5min":0.249,"15min":0.20899999999999999},"max":{"1min":22.050000000000001,"5min":22.050000000000001,"15min":22.050000000000001},"last":1.833}]},{"osd":3,"last update":"Mon Feb 5 06:44:00 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":2.4249999999999998,"15min":1.8500000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.25600000000000001},"max":{"1min":3.827,"5min":25.236999999999998,"15min":69.798000000000002},"last":1.8979999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":2.1200000000000001,"15min":1.625},"min":{"1min":0.35399999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9059999999999999,"5min":43.354999999999997,"15min":70.765000000000001},"last":1.9930000000000001}]},{"osd":4,"last update":"Mon Feb 5 06:44:18 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.1100000000000001,"15min":1.0189999999999999},"min":{"1min":0.433,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":3.452,"5min":7.0519999999999996,"15min":11.679},"last":1.7250000000000001},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.377,"15min":1.056},"min":{"1min":0.53800000000000003,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":4.9610000000000003,"5min":6.0730000000000004,"15min":6.0730000000000004},"last":0.30599999999999999}]},{"osd":5,"last update":"Mon Feb 5 06:43:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.5469999999999999,"15min":1.1830000000000001},"min":{"1min":0.44900000000000001,"5min":0.29399999999999998,"15min":0.29299999999999998},"max":{"1min":3.8540000000000001,"5min":43.414999999999999,"15min":43.414999999999999},"last":1.5049999999999999},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.2230000000000001,"15min":1.097},"min":{"1min":0.36599999999999999,"5min":0.33500000000000002,"15min":0.33100000000000002},"max":{"1min":4.9459999999999997,"5min":7.0919999999999996,"15min":13.455},"last":1.8009999999999999}]},{"osd":6,"last update":"Mon Feb 5 06:44:10 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":1.2,"15min":1.1519999999999999},"min":{"1min":0.40999999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.282,"5min":7.1260000000000003,"15min":13.391},"last":1.7629999999999999},{"interface":"front","average":{"1min":0.879,"5min":1.5429999999999999,"15min":1.1220000000000001},"min":{"1min":0.46200000000000002,"5min":0.38100000000000001,"15min":0.379},"max":{"1min":1.9870000000000001,"5min":43.475000000000001,"15min":43.475000000000001},"last":0.57199999999999995}]},{"osd":7,"last update":"Mon Feb 5 06:43:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.3149999999999999,"15min":1.0920000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":4.2480000000000002,"5min":7.157,"15min":13.49},"last":1.8620000000000001},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.512,"15min":1.226},"min":{"1min":0.40400000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":4.5039999999999996,"5min":9.3249999999999993,"15min":9.3249999999999993},"last":0.82599999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738539,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1576852,"kb_used_data":1230540,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92168300,"statfs":{"total":95995035648,"available":94380339200,"internally_reserved":0,"allocated":1260072960,"data_stored":1259449118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 06:44:10 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":1.655,"15min":1.1970000000000001},"min":{"1min":0.186,"5min":0.182,"15min":0.182},"max":{"1min":1.9670000000000001,"5min":18.59,"15min":18.59},"last":18.311},{"interface":"front","average":{"1min":0.63100000000000001,"5min":1.637,"15min":1.159},"min":{"1min":0.221,"5min":0.193,"15min":0.193},"max":{"1min":2.2400000000000002,"5min":18.300999999999998,"15min":18.300999999999998},"last":21.088000000000001}]},{"osd":2,"last update":"Mon Feb 5 06:43:25 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":1.8080000000000001,"15min":1.1990000000000001},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.1349999999999998,"5min":16.981999999999999,"15min":16.981999999999999},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.95199999999999996,"5min":1.7669999999999999,"15min":1.1419999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":3.306,"5min":16.998999999999999,"15min":16.998999999999999},"last":0.25900000000000001}]},{"osd":3,"last update":"Mon Feb 5 06:43:54 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":1.998,"15min":1.399},"min":{"1min":0.27300000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.6639999999999999,"5min":43.302,"15min":43.302},"last":18.349},{"interface":"front","average":{"1min":0.56499999999999995,"5min":2.4689999999999999,"15min":1.548},"min":{"1min":0.26500000000000001,"5min":0.26000000000000001,"15min":0.254},"max":{"1min":2.3410000000000002,"5min":43.651000000000003,"15min":43.651000000000003},"last":18.712}]},{"osd":4,"last update":"Mon Feb 5 06:43:21 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":1.762,"15min":1.155},"min":{"1min":0.378,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":2.4470000000000001,"5min":40.789999999999999,"15min":40.789999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.80900000000000005,"5min":1.821,"15min":1.2110000000000001},"min":{"1min":0.375,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":2.2909999999999999,"5min":40.773000000000003,"15min":40.773000000000003},"last":0.71499999999999997}]},{"osd":5,"last update":"Mon Feb 5 06:43:45 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":1.47,"15min":1.0429999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.29299999999999998},"max":{"1min":2.3929999999999998,"5min":18.170999999999999,"15min":18.170999999999999},"last":0.623},{"interface":"front","average":{"1min":0.623,"5min":1.276,"15min":1.048},"min":{"1min":0.40899999999999997,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.145,"5min":17.152999999999999,"15min":17.152999999999999},"last":0.59899999999999998}]},{"osd":6,"last update":"Mon Feb 5 06:44:10 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":1.367,"15min":1.1140000000000001},"min":{"1min":0.42999999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":2.0990000000000002,"5min":18.562999999999999,"15min":18.562999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.5189999999999999,"15min":1.133},"min":{"1min":0.44900000000000001,"5min":0.372,"15min":0.33000000000000002},"max":{"1min":13.102,"5min":18.213999999999999,"15min":18.213999999999999},"last":0.65900000000000003}]},{"osd":7,"last update":"Mon Feb 5 06:43:34 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":1.355,"15min":1.044},"min":{"1min":0.436,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.2430000000000001,"5min":17.210000000000001,"15min":17.210000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.65800000000000003,"5min":1.488,"15min":1.244},"min":{"1min":0.45400000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.161,"5min":18.533999999999999,"15min":18.533999999999999},"last":0.68200000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411488,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1255424,"kb_used_data":909108,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92489728,"statfs":{"total":95995035648,"available":94709481472,"internally_reserved":0,"allocated":930926592,"data_stored":930327763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:43:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":1.5589999999999999,"15min":1.5149999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.5299999999999998,"5min":8.9339999999999993,"15min":50.814},"last":1.0469999999999999},{"interface":"front","average":{"1min":0.85699999999999998,"5min":1.7769999999999999,"15min":1.5},"min":{"1min":0.23899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.919,"5min":10.026,"15min":50.802999999999997},"last":0.52200000000000002}]},{"osd":1,"last update":"Mon Feb 5 06:43:28 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":1.71,"15min":1.2829999999999999},"min":{"1min":0.248,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.3479999999999999,"5min":8.3879999999999999,"15min":31.303000000000001},"last":6.9349999999999996},{"interface":"front","average":{"1min":0.78900000000000003,"5min":1.198,"15min":1.05},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.1389999999999998,"5min":6.7690000000000001,"15min":31.425000000000001},"last":0.876}]},{"osd":3,"last update":"Mon Feb 5 06:44:00 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":1.5820000000000001,"15min":1.621},"min":{"1min":0.30299999999999999,"5min":0.25700000000000001,"15min":0.254},"max":{"1min":4.0739999999999998,"5min":13.504,"15min":53.127000000000002},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":1.631,"15min":1.546},"min":{"1min":0.251,"5min":0.23599999999999999,"15min":0.22500000000000001},"max":{"1min":1.929,"5min":11.388999999999999,"15min":52.607999999999997},"last":1.006}]},{"osd":4,"last update":"Mon Feb 5 06:44:19 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":1.044,"15min":0.93899999999999995},"min":{"1min":0.374,"5min":0.29699999999999999,"15min":0.23999999999999999},"max":{"1min":2.448,"5min":12.041,"15min":12.041},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.71099999999999997,"5min":1.0629999999999999,"15min":0.91900000000000004},"min":{"1min":0.33100000000000002,"5min":0.30199999999999999,"15min":0.29199999999999998},"max":{"1min":2.4609999999999999,"5min":15.689,"15min":15.689},"last":0.75600000000000001}]},{"osd":5,"last update":"Mon Feb 5 06:43:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":2.2010000000000001,"15min":1.2929999999999999},"min":{"1min":0.38200000000000001,"5min":0.316,"15min":0.29899999999999999},"max":{"1min":1.431,"5min":86.614999999999995,"15min":86.614999999999995},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.70599999999999996,"5min":2.5110000000000001,"15min":1.4710000000000001},"min":{"1min":0.372,"5min":0.33600000000000002,"15min":0.33200000000000002},"max":{"1min":1.363,"5min":86.543999999999997,"15min":86.543999999999997},"last":0.61699999999999999}]},{"osd":6,"last update":"Mon Feb 5 06:44:19 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.97899999999999998,"15min":1.264},"min":{"1min":0.40000000000000002,"5min":0.315,"15min":0.315},"max":{"1min":2.4319999999999999,"5min":6.665,"15min":17.143999999999998},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.70299999999999996,"5min":1.1839999999999999,"15min":1.1220000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":2.0489999999999999,"5min":6.7919999999999998,"15min":10.27},"last":0.67800000000000005}]},{"osd":7,"last update":"Mon Feb 5 06:43:38 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":1.4139999999999999,"15min":1.1120000000000001},"min":{"1min":0.39800000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":2.3239999999999998,"5min":7.2949999999999999,"15min":10.206},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.62,"5min":1.141,"15min":1.0049999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.36899999999999999},"max":{"1min":1.0780000000000001,"5min":5.8639999999999999,"15min":6.4749999999999996},"last":0.69599999999999995}]}]},{"osd":3,"up_from":26,"seq":111669149851,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2863700,"kb_used_data":2441032,"kb_used_omap":1,"kb_used_meta":422654,"kb_avail":90881452,"statfs":{"total":95995035648,"available":93062606848,"internally_reserved":0,"allocated":2499616768,"data_stored":2498996922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":432798151},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:44:07 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.4019999999999999,"15min":1.073},"min":{"1min":0.20499999999999999,"5min":0.184,"15min":0.184},"max":{"1min":9.6219999999999999,"5min":9.6219999999999999,"15min":13.391999999999999},"last":1.5900000000000001},{"interface":"front","average":{"1min":0.999,"5min":1.5049999999999999,"15min":1.3160000000000001},"min":{"1min":0.307,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":4.5369999999999999,"5min":7.0659999999999998,"15min":25.087},"last":1.4419999999999999}]},{"osd":1,"last update":"Mon Feb 5 06:44:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7789999999999999,"5min":1.7669999999999999,"15min":1.262},"min":{"1min":0.25800000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.37},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.8069999999999999,"15min":1.23},"min":{"1min":0.27300000000000002,"5min":0.215,"15min":0.215},"max":{"1min":3.5830000000000002,"5min":8.5359999999999996,"15min":8.5359999999999996},"last":1.8680000000000001}]},{"osd":2,"last update":"Mon Feb 5 06:44:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":2.081,"15min":1.3839999999999999},"min":{"1min":0.31900000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":3.3250000000000002,"5min":41.529000000000003,"15min":41.529000000000003},"last":1.409},{"interface":"front","average":{"1min":0.98299999999999998,"5min":2.0049999999999999,"15min":1.294},"min":{"1min":0.27400000000000002,"5min":0.25,"15min":0.248},"max":{"1min":4.8049999999999997,"5min":41.759999999999998,"15min":41.759999999999998},"last":0.39000000000000001}]},{"osd":4,"last update":"Mon Feb 5 06:43:25 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":1.1830000000000001,"15min":0.92200000000000004},"min":{"1min":0.48099999999999998,"5min":0.312,"15min":0.307},"max":{"1min":3.181,"5min":7.8010000000000002,"15min":16.577999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.3220000000000001,"15min":1.091},"min":{"1min":0.30099999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":6.7329999999999997,"5min":6.7329999999999997,"15min":8.4060000000000006},"last":0.57899999999999996}]},{"osd":5,"last update":"Mon Feb 5 06:43:46 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":1.49,"15min":1.163},"min":{"1min":0.52200000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":2.2559999999999998,"5min":5.8970000000000002,"15min":13.672000000000001},"last":1.903},{"interface":"front","average":{"1min":0.78300000000000003,"5min":1.27,"15min":0.96299999999999997},"min":{"1min":0.38700000000000001,"5min":0.32800000000000001,"15min":0.28399999999999997},"max":{"1min":2.2040000000000002,"5min":7.7679999999999998,"15min":15.792999999999999},"last":1.2809999999999999}]},{"osd":6,"last update":"Mon Feb 5 06:44:18 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.407,"15min":1.141},"min":{"1min":0.39500000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":4.5890000000000004,"5min":6.7809999999999997,"15min":8.3610000000000007},"last":0.61599999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.3799999999999999,"15min":1.1479999999999999},"min":{"1min":0.48899999999999999,"5min":0.318,"15min":0.28599999999999998},"max":{"1min":3.5569999999999999,"5min":5.8799999999999999,"15min":6.601},"last":2.1099999999999999}]},{"osd":7,"last update":"Mon Feb 5 06:43:42 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":1.2629999999999999,"15min":1.0860000000000001},"min":{"1min":0.45800000000000002,"5min":0.376,"15min":0.33400000000000002},"max":{"1min":1.6890000000000001,"5min":7.7320000000000002,"15min":45.039000000000001},"last":1.2509999999999999},{"interface":"front","average":{"1min":0.84399999999999997,"5min":1.4890000000000001,"15min":1.256},"min":{"1min":0.58099999999999996,"5min":0.35899999999999999,"15min":0.34399999999999997},"max":{"1min":2.4830000000000001,"5min":5.8159999999999998,"15min":44.036999999999999},"last":2.0800000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986325,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1261712,"kb_used_data":905668,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":92483440,"statfs":{"total":95995035648,"available":94703042560,"internally_reserved":0,"allocated":927404032,"data_stored":926787960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:43:22 2024","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":1.5229999999999999,"15min":1.1060000000000001},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20200000000000001},"max":{"1min":6.0220000000000002,"5min":33.372,"15min":33.372},"last":0.26300000000000001},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.4019999999999999,"15min":1.179},"min":{"1min":0.27100000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":6.0140000000000002,"5min":33.323999999999998,"15min":33.323999999999998},"last":0.38}]},{"osd":1,"last update":"Mon Feb 5 06:43:22 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.49,"15min":0.92600000000000005},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":4.5419999999999998,"5min":33.747999999999998,"15min":33.747999999999998},"last":1.6910000000000001},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.3700000000000001,"15min":0.92100000000000004},"min":{"1min":0.23699999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.8550000000000004,"5min":33.691000000000003,"15min":33.691000000000003},"last":2.1469999999999998}]},{"osd":2,"last update":"Mon Feb 5 06:44:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.93300000000000005,"15min":0.76400000000000001},"min":{"1min":0.39800000000000002,"5min":0.28199999999999997,"15min":0.245},"max":{"1min":1.2549999999999999,"5min":17.379999999999999,"15min":17.379999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.55400000000000005,"5min":1.1739999999999999,"15min":0.89900000000000002},"min":{"1min":0.315,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.3129999999999999,"5min":17.001999999999999,"15min":17.001999999999999},"last":0.32700000000000001}]},{"osd":3,"last update":"Mon Feb 5 06:43:22 2024","interfaces":[{"interface":"back","average":{"1min":3.1160000000000001,"5min":2.278,"15min":1.95},"min":{"1min":0.34699999999999998,"5min":0.252,"15min":0.252},"max":{"1min":44.601999999999997,"5min":44.601999999999997,"15min":99.546999999999997},"last":0.623},{"interface":"front","average":{"1min":3.363,"5min":2.3690000000000002,"15min":1.9650000000000001},"min":{"1min":0.42299999999999999,"5min":0.309,"15min":0.29099999999999998},"max":{"1min":44.511000000000003,"5min":44.511000000000003,"15min":99.245000000000005},"last":0.97199999999999998}]},{"osd":5,"last update":"Mon Feb 5 06:43:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.59499999999999997,"15min":0.54100000000000004},"min":{"1min":0.29599999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.147,"5min":4.6289999999999996,"15min":11.098000000000001},"last":1.3049999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.35299999999999998,"5min":0.25700000000000001,"15min":0.252},"max":{"1min":1.5980000000000001,"5min":3.5880000000000001,"15min":10.965999999999999},"last":0.77000000000000002}]},{"osd":6,"last update":"Mon Feb 5 06:44:20 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.67100000000000004,"15min":0.61499999999999999},"min":{"1min":0.35399999999999998,"5min":0.246,"15min":0.246},"max":{"1min":1.1519999999999999,"5min":4.375,"15min":11.868},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.65500000000000003,"15min":0.61299999999999999},"min":{"1min":0.32700000000000001,"5min":0.29899999999999999,"15min":0.27600000000000002},"max":{"1min":0.73799999999999999,"5min":3.5670000000000002,"15min":11.787000000000001},"last":0.45300000000000001}]},{"osd":7,"last update":"Mon Feb 5 06:43:32 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.66900000000000004,"15min":0.56799999999999995},"min":{"1min":0.36399999999999999,"5min":0.27800000000000002,"15min":0.26000000000000001},"max":{"1min":1.6579999999999999,"5min":5.1500000000000004,"15min":5.1500000000000004},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.71999999999999997,"15min":0.60099999999999998},"min":{"1min":0.32200000000000001,"5min":0.28899999999999998,"15min":0.28499999999999998},"max":{"1min":1.607,"5min":5.5890000000000004,"15min":5.5890000000000004},"last":1.2669999999999999}]}]},{"osd":5,"up_from":40,"seq":171798691984,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3097724,"kb_used_data":2701428,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":90647428,"statfs":{"total":95995035648,"available":92822966272,"internally_reserved":0,"allocated":2766262272,"data_stored":2765663337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":405797318},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":1.9299999999999999,"15min":1.3169999999999999},"min":{"1min":0.316,"5min":0.223,"15min":0.19700000000000001},"max":{"1min":2.8279999999999998,"5min":55.704999999999998,"15min":55.704999999999998},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.59699999999999998,"5min":2.069,"15min":1.516},"min":{"1min":0.33600000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.242,"5min":56.789000000000001,"15min":56.789000000000001},"last":0.80000000000000004}]},{"osd":1,"last update":"Mon Feb 5 06:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":1.859,"15min":1.212},"min":{"1min":0.31,"5min":0.251,"15min":0.248},"max":{"1min":2.2730000000000001,"5min":41.191000000000003,"15min":41.191000000000003},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.70699999999999996,"5min":1.8620000000000001,"15min":1.0609999999999999},"min":{"1min":0.28100000000000003,"5min":0.24399999999999999,"15min":0.23499999999999999},"max":{"1min":3.226,"5min":41.173000000000002,"15min":41.173000000000002},"last":1.2110000000000001}]},{"osd":2,"last update":"Mon Feb 5 06:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":1.379,"15min":1.155},"min":{"1min":0.28999999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.536,"5min":30.052,"15min":36.901000000000003},"last":0.996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3620000000000001,"15min":1.071},"min":{"1min":0.39000000000000001,"5min":0.29499999999999998,"15min":0.28899999999999998},"max":{"1min":2.2970000000000002,"5min":30.472999999999999,"15min":37.984000000000002},"last":1.0960000000000001}]},{"osd":3,"last update":"Mon Feb 5 06:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":1.9970000000000001,"15min":1.2130000000000001},"min":{"1min":0.38,"5min":0.33300000000000002,"15min":0.31900000000000001},"max":{"1min":4.8860000000000001,"5min":55.417000000000002,"15min":55.417000000000002},"last":1.4910000000000001},{"interface":"front","average":{"1min":0.66500000000000004,"5min":2.1680000000000001,"15min":1.3049999999999999},"min":{"1min":0.42599999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.351,"5min":55.079000000000001,"15min":55.079000000000001},"last":0.56999999999999995}]},{"osd":4,"last update":"Mon Feb 5 06:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.58699999999999997,"15min":0.46999999999999997},"min":{"1min":0.26900000000000002,"5min":0.25800000000000001,"15min":0.23200000000000001},"max":{"1min":2.613,"5min":5.4969999999999999,"15min":5.4969999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.57199999999999995,"15min":0.497},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.98099999999999998,"5min":3.0150000000000001,"15min":3.0150000000000001},"last":0.50600000000000001}]},{"osd":6,"last update":"Mon Feb 5 06:44:14 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.70399999999999996,"15min":0.55200000000000005},"min":{"1min":0.28100000000000003,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.393,"5min":3.585,"15min":3.585},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.78000000000000003,"15min":0.59999999999999998},"min":{"1min":0.35299999999999998,"5min":0.31900000000000001,"15min":0.27600000000000002},"max":{"1min":1.5700000000000001,"5min":6.5179999999999998,"15min":6.5179999999999998},"last":0.76100000000000001}]},{"osd":7,"last update":"Mon Feb 5 06:43:38 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.624,"15min":0.52800000000000002},"min":{"1min":0.30199999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.851,"5min":2.6480000000000001,"15min":2.6480000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.70399999999999996,"15min":0.57799999999999996},"min":{"1min":0.28999999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.3759999999999999,"5min":3.3140000000000001,"15min":3.3140000000000001},"last":0.59599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1260650496,"data_stored":1260650496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2807955456,"data_stored":2807955456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":931004416,"data_stored":931004416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2508783616,"data_stored":2508783616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":931266560,"data_stored":931266560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2794979328,"data_stored":2794979328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":928776192,"data_stored":928776192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2783903744,"data_stored":2783903744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T06:44:24.124 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T06:44:24.128 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 44 16 78964 78948 112.114 83.5625 0.0150712 0.00891448 2024-02-05T06:44:24.176 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:24.212 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:24.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:24.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:24.314 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:24.351 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:24.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:24.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:24.453 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:24.492 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:24.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:24.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:24.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:24.644 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:24.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:24.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:24.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:24.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:24.776 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:24.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:24.877 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:24.881 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:24 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:44:24.882 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:24 smithi028 ceph-mon[29442]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:44:24.882 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:24 smithi028 ceph-mon[29442]: from='client.45320 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:44:24.882 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:24 smithi028 ceph-mon[29442]: osdmap e307: 8 total, 8 up, 8 in 2024-02-05T06:44:24.882 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:24 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:44:24.882 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:24 smithi028 ceph-mon[37984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:44:24.882 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:24 smithi028 ceph-mon[37984]: from='client.45320 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:44:24.882 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:24 smithi028 ceph-mon[37984]: osdmap e307: 8 total, 8 up, 8 in 2024-02-05T06:44:24.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:24.911 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:24.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:24.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:24 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:44:24.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:24 smithi167 ceph-mon[32774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:44:24.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:24 smithi167 ceph-mon[32774]: from='client.45320 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:44:24.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:24 smithi167 ceph-mon[32774]: osdmap e307: 8 total, 8 up, 8 in 2024-02-05T06:44:25.012 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:25.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:25.047 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:25.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:25.128 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 45 16 80092 80076 111.189 70.5 0.00598168 0.00898727 2024-02-05T06:44:25.149 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:25.173 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:44:25.174 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:44:25.185 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:25.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:25.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:25.287 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:25.326 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:25.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:25.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:25.428 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:25.467 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:25.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:25.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:25.569 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:25.604 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:25.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:25.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:25.706 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:25.743 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:25.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:25.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:25.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:25.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:25.905 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:25.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:25.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:25 smithi028 ceph-mon[29442]: pgmap v535: 158 pgs: 158 active+clean; 4.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 761 B/s rd, 58 MiB/s wr, 923 op/s 2024-02-05T06:44:25.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:25 smithi028 ceph-mon[29442]: osdmap e308: 8 total, 8 up, 8 in 2024-02-05T06:44:25.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:25 smithi028 ceph-mon[37984]: pgmap v535: 158 pgs: 158 active+clean; 4.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 761 B/s rd, 58 MiB/s wr, 923 op/s 2024-02-05T06:44:25.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:25 smithi028 ceph-mon[37984]: osdmap e308: 8 total, 8 up, 8 in 2024-02-05T06:44:25.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:25 smithi167 ceph-mon[32774]: pgmap v535: 158 pgs: 158 active+clean; 4.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 761 B/s rd, 58 MiB/s wr, 923 op/s 2024-02-05T06:44:25.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:25 smithi167 ceph-mon[32774]: osdmap e308: 8 total, 8 up, 8 in 2024-02-05T06:44:26.006 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:26.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:26.055 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:26.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:26.080 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-05T06:44:26.128 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 46 16 81355 81339 110.488 78.9375 0.00828033 0.00904615 2024-02-05T06:44:26.130 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:44:26.156 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:26.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:26.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:26.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:26.335 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:26.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:26.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:26.437 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:26.481 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:26.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:26.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:26.484 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:44:26.543 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:44:26.583 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:26.588 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:44:26.621 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:26.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:26.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:26.630 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:44:26.675 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:44:26.718 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:44:26.722 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:26.763 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:44:26.785 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:26.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:26.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:26.803 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:44:26.845 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:44:26.845 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:26 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3269932394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:44:26.845 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:26 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:44:26.846 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:26 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3269932394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:44:26.846 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:26 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:44:26.850 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:44:26.850 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:44:26.850 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:44:26.850 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:44:26.850 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:44:26.850 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:44:26.850 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:44:26.850 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:44:26.887 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:26.926 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:26.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:26.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:26.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:26 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3269932394' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:44:26.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:26 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:44:27.028 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:27.067 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:27.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:27.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:27.128 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 47 16 82506 82490 109.668 71.9375 0.00597881 0.00911351 2024-02-05T06:44:27.169 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:27.197 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:27.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:27.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:27.299 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:27.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:27.339 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:27.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:27.440 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:27.476 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:27.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:27.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:27.578 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:27.616 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:27.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:27.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:27.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:27.758 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:27.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:27.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:27.857 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T06:44:27.860 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:27.894 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:27 smithi028 ceph-mon[37984]: pgmap v537: 158 pgs: 158 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.6 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-02-05T06:44:27.894 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:27 smithi028 ceph-mon[29442]: pgmap v537: 158 pgs: 158 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.6 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-02-05T06:44:27.899 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:27.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:27.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:27.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:27 smithi167 ceph-mon[32774]: pgmap v537: 158 pgs: 158 active+clean; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.6 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-02-05T06:44:28.001 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:28.083 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:28.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:28.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:28.128 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 48 16 83662 83646 108.888 72.25 0.00895072 0.0091789 2024-02-05T06:44:28.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:28.229 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:28.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:28.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:28.331 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:28.372 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:28.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:28.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:28.474 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:28.513 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:28.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:28.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:28.615 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:28.657 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:28.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:28.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:28.759 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:28.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:28.802 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:28.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:28.903 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:28.939 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:28.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:28.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:29.041 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:29.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:29.080 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:29.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:29.101 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:44:29.130 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 49 16 84928 84912 108.278 79.125 0.00965754 0.00923025 2024-02-05T06:44:29.180 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:44:29.180 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-02-05T06:44:29.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-02-05T06:44:29.180 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-02-05T06:44:29.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.181 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:30010 8.2 8:4c5236ed:::benchmark_data_smithi028_105698_object30009:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-02-05T06:44:29.181 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:07.547293+0000", 2024-02-05T06:44:29.181 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 261.62287854300001, 2024-02-05T06:44:29.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2328662420000001, 2024-02-05T06:44:29.181 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:29.181 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-02-05T06:44:29.181 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:29.181 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:29.181 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:29.181 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30010 2024-02-05T06:44:29.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.182 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:29.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:29.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.547293+0000", 2024-02-05T06:44:29.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:29.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.547293+0000", 2024-02-05T06:44:29.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:29.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.547295+0000", 2024-02-05T06:44:29.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0810000000000001e-06 2024-02-05T06:44:29.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:29.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.547338+0000", 2024-02-05T06:44:29.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2488999999999999e-05 2024-02-05T06:44:29.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:29.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.547347+0000", 2024-02-05T06:44:29.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2569999999999996e-06 2024-02-05T06:44:29.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:29.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.547355+0000", 2024-02-05T06:44:29.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7540000000000006e-06 2024-02-05T06:44:29.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.547424+0000", 2024-02-05T06:44:29.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9509000000000001e-05 2024-02-05T06:44:29.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:29.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.547552+0000", 2024-02-05T06:44:29.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012786699999999999 2024-02-05T06:44:29.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:44:29.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.547685+0000", 2024-02-05T06:44:29.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013314199999999999 2024-02-05T06:44:29.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.549231+0000", 2024-02-05T06:44:29.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015464529999999999 2024-02-05T06:44:29.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:29.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.549378+0000", 2024-02-05T06:44:29.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014604400000000001 2024-02-05T06:44:29.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.562199+0000", 2024-02-05T06:44:29.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012821572 2024-02-05T06:44:29.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:29.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.562214+0000", 2024-02-05T06:44:29.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4938e-05 2024-02-05T06:44:29.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:08.559178+0000", 2024-02-05T06:44:29.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99696352700000002 2024-02-05T06:44:29.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:29.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:08.559200+0000", 2024-02-05T06:44:29.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2819e-05 2024-02-05T06:44:29.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.780059+0000", 2024-02-05T06:44:29.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2208585730000001 2024-02-05T06:44:29.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:29.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.780159+0000", 2024-02-05T06:44:29.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010021700000000001 2024-02-05T06:44:29.188 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.188 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:29.188 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.188 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:30022 8.2 8:4849d65b:::benchmark_data_smithi028_105698_object30021:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-02-05T06:44:29.188 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:07.561554+0000", 2024-02-05T06:44:29.188 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 261.60861723800002, 2024-02-05T06:44:29.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2397414059999998, 2024-02-05T06:44:29.188 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:29.188 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:29.188 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:29.189 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:29.189 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:29.189 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30022 2024-02-05T06:44:29.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.189 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:29.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:29.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.561554+0000", 2024-02-05T06:44:29.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:29.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.561554+0000", 2024-02-05T06:44:29.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:29.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.561557+0000", 2024-02-05T06:44:29.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5349999999999999e-06 2024-02-05T06:44:29.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:29.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.561599+0000", 2024-02-05T06:44:29.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2616e-05 2024-02-05T06:44:29.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:29.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.561613+0000", 2024-02-05T06:44:29.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3122000000000001e-05 2024-02-05T06:44:29.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:29.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.561624+0000", 2024-02-05T06:44:29.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1398e-05 2024-02-05T06:44:29.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.562244+0000", 2024-02-05T06:44:29.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00061996200000000005 2024-02-05T06:44:29.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:29.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.562252+0000", 2024-02-05T06:44:29.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7659999999999998e-06 2024-02-05T06:44:29.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:08.559259+0000", 2024-02-05T06:44:29.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99700688299999995 2024-02-05T06:44:29.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:29.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:08.559271+0000", 2024-02-05T06:44:29.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2364999999999999e-05 2024-02-05T06:44:29.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.781169+0000", 2024-02-05T06:44:29.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.221897805 2024-02-05T06:44:29.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:29.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.781268+0000", 2024-02-05T06:44:29.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8818000000000002e-05 2024-02-05T06:44:29.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:44:29.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.781446+0000", 2024-02-05T06:44:29.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000178495 2024-02-05T06:44:29.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:29.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.784625+0000", 2024-02-05T06:44:29.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003179472 2024-02-05T06:44:29.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.194 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:29.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.784675+0000", 2024-02-05T06:44:29.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9064999999999999e-05 2024-02-05T06:44:29.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.799907+0000", 2024-02-05T06:44:29.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015232757 2024-02-05T06:44:29.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:29.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.799974+0000", 2024-02-05T06:44:29.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6451000000000005e-05 2024-02-05T06:44:29.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:29.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.801296+0000", 2024-02-05T06:44:29.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0013218959999999999 2024-02-05T06:44:29.197 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.197 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:29.197 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.197 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:30023 8.2 8:4af7df9a:::benchmark_data_smithi028_105698_object30022:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-02-05T06:44:29.197 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:07.569012+0000", 2024-02-05T06:44:29.197 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 261.60115935099998, 2024-02-05T06:44:29.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.233291189, 2024-02-05T06:44:29.198 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:29.198 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:29.198 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:29.198 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:29.198 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:29.198 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30023 2024-02-05T06:44:29.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.198 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:29.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:29.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.569012+0000", 2024-02-05T06:44:29.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:29.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.569012+0000", 2024-02-05T06:44:29.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:29.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.569014+0000", 2024-02-05T06:44:29.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2699999999999999e-06 2024-02-05T06:44:29.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:29.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.569105+0000", 2024-02-05T06:44:29.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0138999999999999e-05 2024-02-05T06:44:29.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:29.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.569122+0000", 2024-02-05T06:44:29.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7382999999999999e-05 2024-02-05T06:44:29.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:29.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.569133+0000", 2024-02-05T06:44:29.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1004e-05 2024-02-05T06:44:29.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.569190+0000", 2024-02-05T06:44:29.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6972999999999998e-05 2024-02-05T06:44:29.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:29.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.569232+0000", 2024-02-05T06:44:29.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2197000000000003e-05 2024-02-05T06:44:29.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:08.559313+0000", 2024-02-05T06:44:29.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99008115900000004 2024-02-05T06:44:29.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:29.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:08.559318+0000", 2024-02-05T06:44:29.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2129999999999999e-06 2024-02-05T06:44:29.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.782618+0000", 2024-02-05T06:44:29.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.223299849 2024-02-05T06:44:29.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:29.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.782747+0000", 2024-02-05T06:44:29.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012904400000000001 2024-02-05T06:44:29.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:44:29.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.782939+0000", 2024-02-05T06:44:29.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019172500000000001 2024-02-05T06:44:29.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:29.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.785396+0000", 2024-02-05T06:44:29.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002457374 2024-02-05T06:44:29.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.791518+0000", 2024-02-05T06:44:29.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0061220119999999996 2024-02-05T06:44:29.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.801346+0000", 2024-02-05T06:44:29.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0098273000000000006 2024-02-05T06:44:29.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:29.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.801400+0000", 2024-02-05T06:44:29.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3910999999999999e-05 2024-02-05T06:44:29.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:29.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.802303+0000", 2024-02-05T06:44:29.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00090363599999999998 2024-02-05T06:44:29.205 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.205 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:29.205 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.205 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:30025 8.2 8:43cd8d1f:::benchmark_data_smithi028_105698_object30024:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-02-05T06:44:29.205 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:07.569160+0000", 2024-02-05T06:44:29.206 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 261.60101122999998, 2024-02-05T06:44:29.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2370548960000001, 2024-02-05T06:44:29.206 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:29.206 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:29.206 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:29.206 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:29.206 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:29.206 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30025 2024-02-05T06:44:29.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.206 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:29.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:29.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.569160+0000", 2024-02-05T06:44:29.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:29.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.569160+0000", 2024-02-05T06:44:29.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:29.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.569161+0000", 2024-02-05T06:44:29.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.207e-06 2024-02-05T06:44:29.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:29.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.569194+0000", 2024-02-05T06:44:29.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2512999999999999e-05 2024-02-05T06:44:29.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:29.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.569208+0000", 2024-02-05T06:44:29.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4239999999999999e-05 2024-02-05T06:44:29.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:29.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.569215+0000", 2024-02-05T06:44:29.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8600000000000004e-06 2024-02-05T06:44:29.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.569268+0000", 2024-02-05T06:44:29.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2688000000000001e-05 2024-02-05T06:44:29.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:29.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.569278+0000", 2024-02-05T06:44:29.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8050000000000001e-06 2024-02-05T06:44:29.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:08.560025+0000", 2024-02-05T06:44:29.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99074740500000003 2024-02-05T06:44:29.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:29.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:08.560035+0000", 2024-02-05T06:44:29.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0155e-05 2024-02-05T06:44:29.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.783783+0000", 2024-02-05T06:44:29.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.223747838 2024-02-05T06:44:29.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:29.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.783909+0000", 2024-02-05T06:44:29.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012643499999999999 2024-02-05T06:44:29.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:44:29.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.784006+0000", 2024-02-05T06:44:29.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7021999999999994e-05 2024-02-05T06:44:29.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:29.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.786832+0000", 2024-02-05T06:44:29.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028256000000000002 2024-02-05T06:44:29.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.792577+0000", 2024-02-05T06:44:29.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0057444920000000003 2024-02-05T06:44:29.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.805021+0000", 2024-02-05T06:44:29.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012444319000000001 2024-02-05T06:44:29.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:29.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.805087+0000", 2024-02-05T06:44:29.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6050000000000006e-05 2024-02-05T06:44:29.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:29.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.806215+0000", 2024-02-05T06:44:29.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0011282670000000001 2024-02-05T06:44:29.212 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.212 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:29.212 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.213 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:47685 8.2 8:43c465ad:::benchmark_data_smithi028_105698_object47684:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-02-05T06:44:29.213 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:33.205036+0000", 2024-02-05T06:44:29.213 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 235.96513572399999, 2024-02-05T06:44:29.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.93568001700000003, 2024-02-05T06:44:29.213 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:29.213 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:29.213 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:29.213 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:29.213 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:29.213 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 47685 2024-02-05T06:44:29.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.213 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:29.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:29.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.205036+0000", 2024-02-05T06:44:29.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:29.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.205036+0000", 2024-02-05T06:44:29.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:29.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.205039+0000", 2024-02-05T06:44:29.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9079999999999999e-06 2024-02-05T06:44:29.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:29.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.205081+0000", 2024-02-05T06:44:29.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2083999999999999e-05 2024-02-05T06:44:29.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:29.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.205098+0000", 2024-02-05T06:44:29.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6843000000000001e-05 2024-02-05T06:44:29.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:29.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.205109+0000", 2024-02-05T06:44:29.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1392e-05 2024-02-05T06:44:29.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.205175+0000", 2024-02-05T06:44:29.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5726000000000001e-05 2024-02-05T06:44:29.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:29.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.205185+0000", 2024-02-05T06:44:29.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9529999999999994e-06 2024-02-05T06:44:29.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.129215+0000", 2024-02-05T06:44:29.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.92403015600000005 2024-02-05T06:44:29.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:29.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.129317+0000", 2024-02-05T06:44:29.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010205900000000001 2024-02-05T06:44:29.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:44:29.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.129418+0000", 2024-02-05T06:44:29.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000100621 2024-02-05T06:44:29.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:29.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.135562+0000", 2024-02-05T06:44:29.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.006144465 2024-02-05T06:44:29.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.138545+0000", 2024-02-05T06:44:29.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002982574 2024-02-05T06:44:29.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.140655+0000", 2024-02-05T06:44:29.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021105450000000001 2024-02-05T06:44:29.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:29.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.140690+0000", 2024-02-05T06:44:29.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4999000000000002e-05 2024-02-05T06:44:29.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:29.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.140716+0000", 2024-02-05T06:44:29.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5692000000000001e-05 2024-02-05T06:44:29.218 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.219 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:29.219 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.219 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:47701 8.2 8:4643231d:::benchmark_data_smithi028_105698_object47700:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-02-05T06:44:29.219 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:33.218535+0000", 2024-02-05T06:44:29.219 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 235.951636689, 2024-02-05T06:44:29.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.92233781100000001, 2024-02-05T06:44:29.219 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:29.219 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:29.219 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:29.219 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:29.219 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:29.219 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 47701 2024-02-05T06:44:29.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.220 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:29.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:29.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.218535+0000", 2024-02-05T06:44:29.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:29.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.218535+0000", 2024-02-05T06:44:29.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:29.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.218537+0000", 2024-02-05T06:44:29.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6589999999999999e-06 2024-02-05T06:44:29.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:29.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.218587+0000", 2024-02-05T06:44:29.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9749000000000001e-05 2024-02-05T06:44:29.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:29.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.218601+0000", 2024-02-05T06:44:29.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3909e-05 2024-02-05T06:44:29.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:29.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.218608+0000", 2024-02-05T06:44:29.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8419999999999999e-06 2024-02-05T06:44:29.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.218680+0000", 2024-02-05T06:44:29.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1952000000000005e-05 2024-02-05T06:44:29.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:29.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.218692+0000", 2024-02-05T06:44:29.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1946999999999999e-05 2024-02-05T06:44:29.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.131026+0000", 2024-02-05T06:44:29.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.91233404299999998 2024-02-05T06:44:29.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:29.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.131112+0000", 2024-02-05T06:44:29.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6299999999999997e-05 2024-02-05T06:44:29.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:44:29.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.131199+0000", 2024-02-05T06:44:29.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7270999999999998e-05 2024-02-05T06:44:29.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:29.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.138459+0000", 2024-02-05T06:44:29.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0072594749999999996 2024-02-05T06:44:29.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.140769+0000", 2024-02-05T06:44:29.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023098010000000002 2024-02-05T06:44:29.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.140814+0000", 2024-02-05T06:44:29.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4956999999999997e-05 2024-02-05T06:44:29.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:29.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.140852+0000", 2024-02-05T06:44:29.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8281999999999997e-05 2024-02-05T06:44:29.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:29.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.140873+0000", 2024-02-05T06:44:29.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0624000000000001e-05 2024-02-05T06:44:29.226 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.226 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:29.226 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.226 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:47705 8.2 8:4e686c31:::benchmark_data_smithi028_105698_object47704:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-02-05T06:44:29.226 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:33.223911+0000", 2024-02-05T06:44:29.226 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 235.946260696, 2024-02-05T06:44:29.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.92633359699999995, 2024-02-05T06:44:29.226 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:29.226 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:29.226 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:29.227 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:29.227 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:29.227 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 47705 2024-02-05T06:44:29.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.227 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:29.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:29.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.223911+0000", 2024-02-05T06:44:29.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:29.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.223911+0000", 2024-02-05T06:44:29.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:29.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.223914+0000", 2024-02-05T06:44:29.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8899999999999999e-06 2024-02-05T06:44:29.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:29.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.223957+0000", 2024-02-05T06:44:29.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2905999999999999e-05 2024-02-05T06:44:29.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:29.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.223967+0000", 2024-02-05T06:44:29.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0798000000000001e-05 2024-02-05T06:44:29.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:29.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.223976+0000", 2024-02-05T06:44:29.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5760000000000006e-06 2024-02-05T06:44:29.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.224043+0000", 2024-02-05T06:44:29.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7403999999999994e-05 2024-02-05T06:44:29.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:29.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:33.224055+0000", 2024-02-05T06:44:29.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1664999999999999e-05 2024-02-05T06:44:29.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.132006+0000", 2024-02-05T06:44:29.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.90795076200000002 2024-02-05T06:44:29.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:29.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.132144+0000", 2024-02-05T06:44:29.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013802700000000001 2024-02-05T06:44:29.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:44:29.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.132295+0000", 2024-02-05T06:44:29.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015124800000000001 2024-02-05T06:44:29.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:29.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.138467+0000", 2024-02-05T06:44:29.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0061714439999999999 2024-02-05T06:44:29.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.141862+0000", 2024-02-05T06:44:29.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0033951900000000002 2024-02-05T06:44:29.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.150139+0000", 2024-02-05T06:44:29.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0082776719999999998 2024-02-05T06:44:29.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:29.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.150202+0000", 2024-02-05T06:44:29.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2496999999999995e-05 2024-02-05T06:44:29.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:29.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:34.150244+0000", 2024-02-05T06:44:29.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2518000000000002e-05 2024-02-05T06:44:29.233 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.233 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:29.233 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.233 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:140508 8.2 8:42800101:::benchmark_data_smithi028_105698_object51233:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-02-05T06:44:29.233 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:23.430618+0000", 2024-02-05T06:44:29.233 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 125.739553517, 2024-02-05T06:44:29.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0314014380000001, 2024-02-05T06:44:29.233 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:29.233 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-02-05T06:44:29.233 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:29.233 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:29.234 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:29.234 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 140508 2024-02-05T06:44:29.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.234 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:29.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:29.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.430618+0000", 2024-02-05T06:44:29.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:29.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.430618+0000", 2024-02-05T06:44:29.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:29.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.430620+0000", 2024-02-05T06:44:29.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.821e-06 2024-02-05T06:44:29.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:29.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.430624+0000", 2024-02-05T06:44:29.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6500000000000004e-06 2024-02-05T06:44:29.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:29.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.430636+0000", 2024-02-05T06:44:29.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1297e-05 2024-02-05T06:44:29.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:29.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.430642+0000", 2024-02-05T06:44:29.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2299999999999996e-06 2024-02-05T06:44:29.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.430696+0000", 2024-02-05T06:44:29.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4397000000000001e-05 2024-02-05T06:44:29.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:29.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.430845+0000", 2024-02-05T06:44:29.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000148636 2024-02-05T06:44:29.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3,5", 2024-02-05T06:44:29.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.431017+0000", 2024-02-05T06:44:29.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017170299999999999 2024-02-05T06:44:29.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:29.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.433418+0000", 2024-02-05T06:44:29.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002401659 2024-02-05T06:44:29.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.434052+0000", 2024-02-05T06:44:29.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00063330400000000001 2024-02-05T06:44:29.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.436375+0000", 2024-02-05T06:44:29.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00232336 2024-02-05T06:44:29.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.446057+0000", 2024-02-05T06:44:29.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0096824479999999997 2024-02-05T06:44:29.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:29.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.446073+0000", 2024-02-05T06:44:29.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.526e-05 2024-02-05T06:44:29.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.461951+0000", 2024-02-05T06:44:29.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.01587871 2024-02-05T06:44:29.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:29.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.462019+0000", 2024-02-05T06:44:29.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7963e-05 2024-02-05T06:44:29.241 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.241 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:29.241 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.241 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:140511 8.2 8:4014bd7d:::benchmark_data_smithi028_105698_object51236:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-02-05T06:44:29.241 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:23.436418+0000", 2024-02-05T06:44:29.241 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 125.73375394999999, 2024-02-05T06:44:29.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.026065132, 2024-02-05T06:44:29.241 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:29.241 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-02-05T06:44:29.241 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:29.241 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:29.242 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:29.242 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 140511 2024-02-05T06:44:29.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.242 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:29.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:29.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.436418+0000", 2024-02-05T06:44:29.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:29.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.436418+0000", 2024-02-05T06:44:29.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:29.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.436419+0000", 2024-02-05T06:44:29.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.649e-06 2024-02-05T06:44:29.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:29.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.436426+0000", 2024-02-05T06:44:29.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0589999999999997e-06 2024-02-05T06:44:29.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:29.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.436460+0000", 2024-02-05T06:44:29.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4032000000000002e-05 2024-02-05T06:44:29.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:29.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.436469+0000", 2024-02-05T06:44:29.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1200000000000008e-06 2024-02-05T06:44:29.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.436592+0000", 2024-02-05T06:44:29.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123046 2024-02-05T06:44:29.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:29.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.436726+0000", 2024-02-05T06:44:29.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013393899999999999 2024-02-05T06:44:29.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3,5", 2024-02-05T06:44:29.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.436861+0000", 2024-02-05T06:44:29.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134368 2024-02-05T06:44:29.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:29.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.441695+0000", 2024-02-05T06:44:29.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.004834634 2024-02-05T06:44:29.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.446106+0000", 2024-02-05T06:44:29.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0044110570000000003 2024-02-05T06:44:29.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:29.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.446117+0000", 2024-02-05T06:44:29.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0428000000000001e-05 2024-02-05T06:44:29.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.462403+0000", 2024-02-05T06:44:29.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0162859929999999 2024-02-05T06:44:29.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:29.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.462483+0000", 2024-02-05T06:44:29.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9807000000000003e-05 2024-02-05T06:44:29.247 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.247 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:29.247 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.247 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:140513 8.2 8:46eb6567:::benchmark_data_smithi028_105698_object51238:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-02-05T06:44:29.247 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:23.436672+0000", 2024-02-05T06:44:29.247 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 125.73349963, 2024-02-05T06:44:29.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.025900399, 2024-02-05T06:44:29.247 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:29.247 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-02-05T06:44:29.247 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:29.247 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:29.248 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:29.248 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 140513 2024-02-05T06:44:29.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.248 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:29.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:29.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.436672+0000", 2024-02-05T06:44:29.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:29.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.436672+0000", 2024-02-05T06:44:29.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:29.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.436673+0000", 2024-02-05T06:44:29.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1370000000000001e-06 2024-02-05T06:44:29.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:29.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.436677+0000", 2024-02-05T06:44:29.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1219999999999997e-06 2024-02-05T06:44:29.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:29.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.436686+0000", 2024-02-05T06:44:29.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7590000000000003e-06 2024-02-05T06:44:29.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:29.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.436693+0000", 2024-02-05T06:44:29.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.951e-06 2024-02-05T06:44:29.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.438808+0000", 2024-02-05T06:44:29.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021152459999999999 2024-02-05T06:44:29.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:29.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.438970+0000", 2024-02-05T06:44:29.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000161435 2024-02-05T06:44:29.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3,5", 2024-02-05T06:44:29.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.439110+0000", 2024-02-05T06:44:29.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014008099999999999 2024-02-05T06:44:29.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.446154+0000", 2024-02-05T06:44:29.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0070444679999999999 2024-02-05T06:44:29.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:29.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.446168+0000", 2024-02-05T06:44:29.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3574000000000001e-05 2024-02-05T06:44:29.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.462516+0000", 2024-02-05T06:44:29.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.01634797 2024-02-05T06:44:29.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:29.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.462572+0000", 2024-02-05T06:44:29.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6656000000000001e-05 2024-02-05T06:44:29.252 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.253 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:29.253 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.253 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:140530 8.2 8:471944ce:::benchmark_data_smithi028_105698_object51255:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-02-05T06:44:29.253 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:23.459861+0000", 2024-02-05T06:44:29.253 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 125.71031018799999, 2024-02-05T06:44:29.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0118483190000001, 2024-02-05T06:44:29.253 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:29.253 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:29.253 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:29.253 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:29.253 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:29.253 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 140530 2024-02-05T06:44:29.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.254 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:29.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:29.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.459861+0000", 2024-02-05T06:44:29.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:29.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.459861+0000", 2024-02-05T06:44:29.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:29.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.459864+0000", 2024-02-05T06:44:29.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3159999999999999e-06 2024-02-05T06:44:29.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:29.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.459870+0000", 2024-02-05T06:44:29.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1279999999999996e-06 2024-02-05T06:44:29.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:29.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.459882+0000", 2024-02-05T06:44:29.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2619000000000001e-05 2024-02-05T06:44:29.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:29.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.459892+0000", 2024-02-05T06:44:29.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.944e-06 2024-02-05T06:44:29.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.459955+0000", 2024-02-05T06:44:29.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2346000000000002e-05 2024-02-05T06:44:29.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:29.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.459966+0000", 2024-02-05T06:44:29.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1378e-05 2024-02-05T06:44:29.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.462603+0000", 2024-02-05T06:44:29.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0026372589999999 2024-02-05T06:44:29.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:29.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.462706+0000", 2024-02-05T06:44:29.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010285000000000001 2024-02-05T06:44:29.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:44:29.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.462827+0000", 2024-02-05T06:44:29.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120946 2024-02-05T06:44:29.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.471480+0000", 2024-02-05T06:44:29.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0086533749999999996 2024-02-05T06:44:29.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:29.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.471499+0000", 2024-02-05T06:44:29.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8510999999999999e-05 2024-02-05T06:44:29.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.471611+0000", 2024-02-05T06:44:29.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111828 2024-02-05T06:44:29.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:29.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.471661+0000", 2024-02-05T06:44:29.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0275000000000002e-05 2024-02-05T06:44:29.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:29.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.471710+0000", 2024-02-05T06:44:29.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8544000000000001e-05 2024-02-05T06:44:29.259 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.259 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:29.259 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.260 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:140538 8.2 8:4be797cf:::benchmark_data_smithi028_105698_object51263:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-02-05T06:44:29.260 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:23.464208+0000", 2024-02-05T06:44:29.260 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 125.70596391700001, 2024-02-05T06:44:29.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.007671134, 2024-02-05T06:44:29.260 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:29.260 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:29.260 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:29.260 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:29.260 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:29.260 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 140538 2024-02-05T06:44:29.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.260 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:29.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:29.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.464208+0000", 2024-02-05T06:44:29.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:29.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.464208+0000", 2024-02-05T06:44:29.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:29.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.464210+0000", 2024-02-05T06:44:29.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2199999999999999e-06 2024-02-05T06:44:29.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:29.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.464218+0000", 2024-02-05T06:44:29.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5750000000000001e-06 2024-02-05T06:44:29.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:29.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.464233+0000", 2024-02-05T06:44:29.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4323e-05 2024-02-05T06:44:29.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:29.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.464241+0000", 2024-02-05T06:44:29.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1300000000000001e-06 2024-02-05T06:44:29.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.464377+0000", 2024-02-05T06:44:29.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000136315 2024-02-05T06:44:29.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:29.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.464385+0000", 2024-02-05T06:44:29.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1189999999999997e-06 2024-02-05T06:44:29.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.465231+0000", 2024-02-05T06:44:29.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0008459249999999 2024-02-05T06:44:29.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:29.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.465412+0000", 2024-02-05T06:44:29.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000181211 2024-02-05T06:44:29.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:44:29.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.465532+0000", 2024-02-05T06:44:29.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119981 2024-02-05T06:44:29.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:29.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.471518+0000", 2024-02-05T06:44:29.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059857690000000002 2024-02-05T06:44:29.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.471560+0000", 2024-02-05T06:44:29.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2061000000000003e-05 2024-02-05T06:44:29.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.471782+0000", 2024-02-05T06:44:29.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022130699999999999 2024-02-05T06:44:29.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:29.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.471830+0000", 2024-02-05T06:44:29.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8029999999999999e-05 2024-02-05T06:44:29.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:29.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.471879+0000", 2024-02-05T06:44:29.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9168000000000001e-05 2024-02-05T06:44:29.266 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.266 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:29.266 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.267 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:140552 8.2 8:4e9a9379:::benchmark_data_smithi028_105698_object51277:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-02-05T06:44:29.267 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:23.477918+0000", 2024-02-05T06:44:29.267 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 125.69225380899999, 2024-02-05T06:44:29.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99409492099999996, 2024-02-05T06:44:29.267 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:29.267 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:29.267 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:29.267 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:29.267 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:29.267 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 140552 2024-02-05T06:44:29.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.267 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:29.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:29.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.477918+0000", 2024-02-05T06:44:29.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:29.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.477918+0000", 2024-02-05T06:44:29.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:29.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.477919+0000", 2024-02-05T06:44:29.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7570000000000001e-06 2024-02-05T06:44:29.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:29.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.477925+0000", 2024-02-05T06:44:29.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3240000000000002e-06 2024-02-05T06:44:29.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:29.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.477936+0000", 2024-02-05T06:44:29.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1384e-05 2024-02-05T06:44:29.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:29.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.477943+0000", 2024-02-05T06:44:29.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.742e-06 2024-02-05T06:44:29.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.477990+0000", 2024-02-05T06:44:29.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7601e-05 2024-02-05T06:44:29.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:29.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.477998+0000", 2024-02-05T06:44:29.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2919999999999998e-06 2024-02-05T06:44:29.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.468484+0000", 2024-02-05T06:44:29.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99048594099999998 2024-02-05T06:44:29.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:29.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.468668+0000", 2024-02-05T06:44:29.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000183936 2024-02-05T06:44:29.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:44:29.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.468783+0000", 2024-02-05T06:44:29.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115238 2024-02-05T06:44:29.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.471747+0000", 2024-02-05T06:44:29.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029645230000000002 2024-02-05T06:44:29.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.471923+0000", 2024-02-05T06:44:29.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000175444 2024-02-05T06:44:29.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:29.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.471933+0000", 2024-02-05T06:44:29.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8959999999999998e-06 2024-02-05T06:44:29.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:29.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.471970+0000", 2024-02-05T06:44:29.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6869000000000002e-05 2024-02-05T06:44:29.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:29.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.472013+0000", 2024-02-05T06:44:29.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2973999999999999e-05 2024-02-05T06:44:29.273 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.273 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:29.273 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.274 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:140582 8.2 8:46601042:::benchmark_data_smithi028_105698_object51307:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-02-05T06:44:29.274 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:23.502792+0000", 2024-02-05T06:44:29.274 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 125.667379728, 2024-02-05T06:44:29.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98169489700000001, 2024-02-05T06:44:29.274 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:29.274 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:29.274 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:29.274 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:29.274 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:29.274 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 140582 2024-02-05T06:44:29.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.274 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:29.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:29.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.502792+0000", 2024-02-05T06:44:29.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:29.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.502792+0000", 2024-02-05T06:44:29.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:29.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.502793+0000", 2024-02-05T06:44:29.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.671e-06 2024-02-05T06:44:29.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:29.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.502798+0000", 2024-02-05T06:44:29.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6539999999999998e-06 2024-02-05T06:44:29.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:29.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.502809+0000", 2024-02-05T06:44:29.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1389999999999999e-05 2024-02-05T06:44:29.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:29.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.502817+0000", 2024-02-05T06:44:29.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8790000000000003e-06 2024-02-05T06:44:29.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.502872+0000", 2024-02-05T06:44:29.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5074000000000002e-05 2024-02-05T06:44:29.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:29.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.502880+0000", 2024-02-05T06:44:29.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.554e-06 2024-02-05T06:44:29.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.472053+0000", 2024-02-05T06:44:29.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96917274899999994 2024-02-05T06:44:29.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:29.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.472160+0000", 2024-02-05T06:44:29.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010762199999999999 2024-02-05T06:44:29.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.279 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:44:29.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.472267+0000", 2024-02-05T06:44:29.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010699799999999999 2024-02-05T06:44:29.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.479955+0000", 2024-02-05T06:44:29.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0076878320000000003 2024-02-05T06:44:29.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.479992+0000", 2024-02-05T06:44:29.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6665999999999998e-05 2024-02-05T06:44:29.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:29.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.484355+0000", 2024-02-05T06:44:29.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0043633669999999999 2024-02-05T06:44:29.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:29.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.484442+0000", 2024-02-05T06:44:29.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7175000000000005e-05 2024-02-05T06:44:29.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:29.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.484487+0000", 2024-02-05T06:44:29.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4265999999999999e-05 2024-02-05T06:44:29.281 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.282 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:29.282 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.282 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:140590 8.2 8:42182456:::benchmark_data_smithi028_105698_object51315:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-02-05T06:44:29.282 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:23.507529+0000", 2024-02-05T06:44:29.282 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 125.662642897, 2024-02-05T06:44:29.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97717015399999996, 2024-02-05T06:44:29.282 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:29.282 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:29.282 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:29.282 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:29.283 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:29.283 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 140590 2024-02-05T06:44:29.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.283 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:29.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:29.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.507529+0000", 2024-02-05T06:44:29.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:29.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.507529+0000", 2024-02-05T06:44:29.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:29.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.507531+0000", 2024-02-05T06:44:29.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4549999999999998e-06 2024-02-05T06:44:29.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:29.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.507539+0000", 2024-02-05T06:44:29.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.763e-06 2024-02-05T06:44:29.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:29.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.507557+0000", 2024-02-05T06:44:29.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7884e-05 2024-02-05T06:44:29.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:29.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.507567+0000", 2024-02-05T06:44:29.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0524e-05 2024-02-05T06:44:29.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.285 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.285 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.507639+0000", 2024-02-05T06:44:29.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1619000000000007e-05 2024-02-05T06:44:29.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:29.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.507649+0000", 2024-02-05T06:44:29.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9729999999999992e-06 2024-02-05T06:44:29.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.474719+0000", 2024-02-05T06:44:29.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96707034000000003 2024-02-05T06:44:29.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:29.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.474844+0000", 2024-02-05T06:44:29.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012465100000000001 2024-02-05T06:44:29.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:44:29.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.474962+0000", 2024-02-05T06:44:29.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118009 2024-02-05T06:44:29.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.480026+0000", 2024-02-05T06:44:29.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0050638699999999998 2024-02-05T06:44:29.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.480060+0000", 2024-02-05T06:44:29.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3849000000000001e-05 2024-02-05T06:44:29.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:29.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.484624+0000", 2024-02-05T06:44:29.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0045649489999999996 2024-02-05T06:44:29.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:29.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.484658+0000", 2024-02-05T06:44:29.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3340000000000003e-05 2024-02-05T06:44:29.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:29.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.484699+0000", 2024-02-05T06:44:29.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0927999999999998e-05 2024-02-05T06:44:29.289 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.289 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:29.289 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.289 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:140609 8.2 8:465732ac:::benchmark_data_smithi028_105698_object51334:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-02-05T06:44:29.289 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:23.527955+0000", 2024-02-05T06:44:29.289 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 125.64221651699999, 2024-02-05T06:44:29.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.95684319500000004, 2024-02-05T06:44:29.289 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:29.289 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:29.289 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:29.289 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:29.290 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:29.290 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 140609 2024-02-05T06:44:29.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.290 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:29.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:29.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.527955+0000", 2024-02-05T06:44:29.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:29.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.527955+0000", 2024-02-05T06:44:29.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:29.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.527957+0000", 2024-02-05T06:44:29.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9120000000000001e-06 2024-02-05T06:44:29.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:29.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.527963+0000", 2024-02-05T06:44:29.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.801e-06 2024-02-05T06:44:29.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.291 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:29.291 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.527976+0000", 2024-02-05T06:44:29.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3478e-05 2024-02-05T06:44:29.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:29.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.527986+0000", 2024-02-05T06:44:29.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5820000000000008e-06 2024-02-05T06:44:29.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.528050+0000", 2024-02-05T06:44:29.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4635999999999996e-05 2024-02-05T06:44:29.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:29.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.528060+0000", 2024-02-05T06:44:29.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0112000000000001e-05 2024-02-05T06:44:29.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.477396+0000", 2024-02-05T06:44:29.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.94933583099999996 2024-02-05T06:44:29.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:29.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.477496+0000", 2024-02-05T06:44:29.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000100043 2024-02-05T06:44:29.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:44:29.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.477616+0000", 2024-02-05T06:44:29.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119469 2024-02-05T06:44:29.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.480095+0000", 2024-02-05T06:44:29.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002478924 2024-02-05T06:44:29.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.484553+0000", 2024-02-05T06:44:29.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0044581530000000003 2024-02-05T06:44:29.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:29.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.484713+0000", 2024-02-05T06:44:29.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000159813 2024-02-05T06:44:29.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:29.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.484764+0000", 2024-02-05T06:44:29.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0933999999999997e-05 2024-02-05T06:44:29.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:29.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.484798+0000", 2024-02-05T06:44:29.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4507e-05 2024-02-05T06:44:29.296 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.296 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:29.296 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.296 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:140616 8.2 8:4eaf74fd:::benchmark_data_smithi028_105698_object51341:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-02-05T06:44:29.296 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:23.533483+0000", 2024-02-05T06:44:29.296 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 125.636688236, 2024-02-05T06:44:29.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.95640482500000001, 2024-02-05T06:44:29.296 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:29.296 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:29.296 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:29.296 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:29.296 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:29.296 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 140616 2024-02-05T06:44:29.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.297 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:29.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:29.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.533483+0000", 2024-02-05T06:44:29.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:29.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.533483+0000", 2024-02-05T06:44:29.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.297 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:29.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.533486+0000", 2024-02-05T06:44:29.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7750000000000001e-06 2024-02-05T06:44:29.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:29.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.533494+0000", 2024-02-05T06:44:29.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2099999999999993e-06 2024-02-05T06:44:29.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:29.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.533513+0000", 2024-02-05T06:44:29.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9057999999999999e-05 2024-02-05T06:44:29.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:29.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.533523+0000", 2024-02-05T06:44:29.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0159999999999999e-05 2024-02-05T06:44:29.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.533601+0000", 2024-02-05T06:44:29.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7757999999999997e-05 2024-02-05T06:44:29.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:29.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.533612+0000", 2024-02-05T06:44:29.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1073000000000001e-05 2024-02-05T06:44:29.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.480122+0000", 2024-02-05T06:44:29.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.94650959300000004 2024-02-05T06:44:29.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:29.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.480230+0000", 2024-02-05T06:44:29.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000107817 2024-02-05T06:44:29.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:44:29.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.480343+0000", 2024-02-05T06:44:29.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113242 2024-02-05T06:44:29.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.484609+0000", 2024-02-05T06:44:29.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0042665589999999996 2024-02-05T06:44:29.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:29.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.484806+0000", 2024-02-05T06:44:29.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000196348 2024-02-05T06:44:29.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.489781+0000", 2024-02-05T06:44:29.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049750039999999999 2024-02-05T06:44:29.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:29.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.489830+0000", 2024-02-05T06:44:29.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8881000000000002e-05 2024-02-05T06:44:29.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.303 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:29.303 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.489888+0000", 2024-02-05T06:44:29.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8347000000000002e-05 2024-02-05T06:44:29.303 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.303 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:29.304 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.304 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:140644 8.2 8:40d608db:::benchmark_data_smithi028_105698_object51369:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-02-05T06:44:29.304 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:23.560295+0000", 2024-02-05T06:44:29.304 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 125.609876216, 2024-02-05T06:44:29.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.92973749400000005, 2024-02-05T06:44:29.304 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:29.304 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:29.304 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:29.304 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:29.304 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:29.304 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 140644 2024-02-05T06:44:29.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.305 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:29.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:29.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.560295+0000", 2024-02-05T06:44:29.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:29.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.560295+0000", 2024-02-05T06:44:29.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:29.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.560297+0000", 2024-02-05T06:44:29.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.751e-06 2024-02-05T06:44:29.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:29.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.560302+0000", 2024-02-05T06:44:29.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3240000000000002e-06 2024-02-05T06:44:29.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:29.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.560313+0000", 2024-02-05T06:44:29.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0715e-05 2024-02-05T06:44:29.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:29.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.560322+0000", 2024-02-05T06:44:29.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2119999999999992e-06 2024-02-05T06:44:29.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.560384+0000", 2024-02-05T06:44:29.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1716000000000003e-05 2024-02-05T06:44:29.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:29.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.560392+0000", 2024-02-05T06:44:29.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7989999999999994e-06 2024-02-05T06:44:29.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.482277+0000", 2024-02-05T06:44:29.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.92188496900000005 2024-02-05T06:44:29.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:29.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.482397+0000", 2024-02-05T06:44:29.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120011 2024-02-05T06:44:29.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:44:29.309 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.482510+0000", 2024-02-05T06:44:29.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113224 2024-02-05T06:44:29.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.309 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:29.309 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.484813+0000", 2024-02-05T06:44:29.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023028620000000001 2024-02-05T06:44:29.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.309 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.309 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.489744+0000", 2024-02-05T06:44:29.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.004931225 2024-02-05T06:44:29.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.489960+0000", 2024-02-05T06:44:29.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000215443 2024-02-05T06:44:29.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:29.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.490001+0000", 2024-02-05T06:44:29.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1946000000000001e-05 2024-02-05T06:44:29.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:29.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.490033+0000", 2024-02-05T06:44:29.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1297000000000002e-05 2024-02-05T06:44:29.310 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.310 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:29.311 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.311 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:140650 8.2 8:4fcc0ff8:::benchmark_data_smithi028_105698_object51375:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-02-05T06:44:29.311 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:23.568610+0000", 2024-02-05T06:44:29.311 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 125.601561426, 2024-02-05T06:44:29.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.92920546400000004, 2024-02-05T06:44:29.311 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:29.311 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:29.311 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:29.311 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:29.311 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:29.311 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 140650 2024-02-05T06:44:29.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.312 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:29.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:29.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.568610+0000", 2024-02-05T06:44:29.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:29.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.568610+0000", 2024-02-05T06:44:29.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:29.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.568612+0000", 2024-02-05T06:44:29.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.857e-06 2024-02-05T06:44:29.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:29.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.568617+0000", 2024-02-05T06:44:29.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7430000000000002e-06 2024-02-05T06:44:29.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:29.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.568635+0000", 2024-02-05T06:44:29.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8253999999999998e-05 2024-02-05T06:44:29.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:29.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.568644+0000", 2024-02-05T06:44:29.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1970000000000002e-06 2024-02-05T06:44:29.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.568707+0000", 2024-02-05T06:44:29.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3250000000000006e-05 2024-02-05T06:44:29.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:29.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.568716+0000", 2024-02-05T06:44:29.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4149999999999999e-06 2024-02-05T06:44:29.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.484867+0000", 2024-02-05T06:44:29.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.91615160900000003 2024-02-05T06:44:29.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:29.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.484986+0000", 2024-02-05T06:44:29.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118678 2024-02-05T06:44:29.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:44:29.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.485112+0000", 2024-02-05T06:44:29.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012614700000000001 2024-02-05T06:44:29.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.489926+0000", 2024-02-05T06:44:29.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048142360000000004 2024-02-05T06:44:29.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.490065+0000", 2024-02-05T06:44:29.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013887599999999999 2024-02-05T06:44:29.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:29.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.497762+0000", 2024-02-05T06:44:29.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0076970340000000002 2024-02-05T06:44:29.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:29.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.497791+0000", 2024-02-05T06:44:29.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8924e-05 2024-02-05T06:44:29.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:29.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.497816+0000", 2024-02-05T06:44:29.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4244e-05 2024-02-05T06:44:29.317 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.317 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:29.317 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.317 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:140674 8.2 8:4359ee71:::benchmark_data_smithi028_105698_object51399:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-02-05T06:44:29.317 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:23.599256+0000", 2024-02-05T06:44:29.317 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 125.570915054, 2024-02-05T06:44:29.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.92447017799999998, 2024-02-05T06:44:29.317 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:29.317 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:29.318 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:29.318 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:29.318 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:29.318 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 140674 2024-02-05T06:44:29.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.318 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:29.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:29.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.599256+0000", 2024-02-05T06:44:29.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:29.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.599256+0000", 2024-02-05T06:44:29.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:29.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:29.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.599259+0000", 2024-02-05T06:44:29.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.627e-06 2024-02-05T06:44:29.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:29.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.599265+0000", 2024-02-05T06:44:29.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1129999999999997e-06 2024-02-05T06:44:29.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:29.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.599279+0000", 2024-02-05T06:44:29.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3539e-05 2024-02-05T06:44:29.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:29.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.599289+0000", 2024-02-05T06:44:29.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8770000000000005e-06 2024-02-05T06:44:29.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.599348+0000", 2024-02-05T06:44:29.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9398000000000003e-05 2024-02-05T06:44:29.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:29.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.599355+0000", 2024-02-05T06:44:29.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.233e-06 2024-02-05T06:44:29.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:29.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.490128+0000", 2024-02-05T06:44:29.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.89077234100000002 2024-02-05T06:44:29.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:29.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.490221+0000", 2024-02-05T06:44:29.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3178999999999997e-05 2024-02-05T06:44:29.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:44:29.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.490320+0000", 2024-02-05T06:44:29.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9621e-05 2024-02-05T06:44:29.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:29.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.497825+0000", 2024-02-05T06:44:29.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0075050489999999997 2024-02-05T06:44:29.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.322 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.322 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.497956+0000", 2024-02-05T06:44:29.322 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013080399999999999 2024-02-05T06:44:29.322 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.322 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:29.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.523657+0000", 2024-02-05T06:44:29.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025700771000000001 2024-02-05T06:44:29.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:29.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.523703+0000", 2024-02-05T06:44:29.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5775999999999997e-05 2024-02-05T06:44:29.323 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:29.323 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:29.323 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:29.323 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.523727+0000", 2024-02-05T06:44:29.323 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.385e-05 2024-02-05T06:44:29.323 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.323 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:29.324 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.324 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:29.324 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:29.324 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:44:29.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:29.325 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:29.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:29.426 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:29.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:29.463 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:29.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:29.564 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:29.598 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:29.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:29.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:29.700 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:29.743 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:29.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:29.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:29.845 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:29.885 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:29.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:29.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:29.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:29 smithi028 ceph-mon[37984]: pgmap v538: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-02-05T06:44:29.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:29 smithi028 ceph-mon[29442]: pgmap v538: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-02-05T06:44:29.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:29 smithi167 ceph-mon[32774]: pgmap v538: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-02-05T06:44:29.991 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:30.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:30.022 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:30.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:30.123 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:30.130 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 50 16 86121 86105 107.603 74.5625 0.0163584 0.00928754 2024-02-05T06:44:30.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:30.156 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:30.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:30.257 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:30.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:30.294 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:30.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:30.373 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:44:30.376 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-05T06:44:30.387 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-05T06:44:30.395 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:30.463 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:30.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:30.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:30.565 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:30.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:30.600 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:30.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:30.701 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:30.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:30.749 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:30.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:30.795 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:44:30.851 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:30.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:30.893 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:30.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:30.993 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:31.036 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:31.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:31.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:31.123 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:44:31.130 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 51 15 87367 87352 107.021 77.9375 0.00923828 0.00933896 2024-02-05T06:44:31.139 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:31.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:31.191 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:31.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:31.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:31 smithi167 ceph-mon[32774]: from='client.45332 v1:172.21.15.28:0/328430573' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:44:31.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:31 smithi167 ceph-mon[32774]: pgmap v539: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 741 op/s 2024-02-05T06:44:31.292 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:31.338 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:31.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:31.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:31.413 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:31 smithi028 ceph-mon[29442]: from='client.45332 v1:172.21.15.28:0/328430573' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:44:31.413 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:31 smithi028 ceph-mon[29442]: pgmap v539: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 741 op/s 2024-02-05T06:44:31.413 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:31 smithi028 ceph-mon[37984]: from='client.45332 v1:172.21.15.28:0/328430573' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:44:31.413 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:31 smithi028 ceph-mon[37984]: pgmap v539: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 46 MiB/s wr, 741 op/s 2024-02-05T06:44:31.440 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:31.472 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:31.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:31.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:31.475 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":538,"stamp":"2024-02-05T06:44:28.366403+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5379332444,"num_objects":82271,"num_object_clones":0,"num_object_copies":246813,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82271,"num_whiteouts":0,"num_read":1373,"num_read_kb":1125,"num_write":82795,"num_write_kb":5252717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":416,"num_bytes_recovered":1586799,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79149,"ondisk_log_size":79149,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":18591504,"kb_used_data":15662652,"kb_used_omap":12,"kb_used_meta":2928755,"kb_avail":731369712,"statfs":{"total":767960285184,"available":748922585088,"internally_reserved":0,"allocated":16038555648,"data_stored":16033583844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":2999045681},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":795410432,"num_objects":12137,"num_object_clones":0,"num_object_copies":36411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12137,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12137,"num_write_kb":776768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005964"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":426,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.102834+0000","last_change":"2024-02-05T06:42:34.762437+0000","last_active":"2024-02-05T06:44:25.102834+0000","last_peered":"2024-02-05T06:44:25.102834+0000","last_clean":"2024-02-05T06:44:25.102834+0000","last_became_active":"2024-02-05T06:41:43.412603+0000","last_became_peered":"2024-02-05T06:41:43.412603+0000","last_unstale":"2024-02-05T06:44:25.102834+0000","last_undegraded":"2024-02-05T06:44:25.102834+0000","last_fullsized":"2024-02-05T06:44:25.102834+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:34.762324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:42:34.762324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:26.837089+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":546,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.107946+0000","last_change":"2024-02-05T06:43:36.504759+0000","last_active":"2024-02-05T06:44:25.107946+0000","last_peered":"2024-02-05T06:44:25.107946+0000","last_clean":"2024-02-05T06:44:25.107946+0000","last_became_active":"2024-02-05T06:41:43.420342+0000","last_became_peered":"2024-02-05T06:41:43.420342+0000","last_unstale":"2024-02-05T06:44:25.107946+0000","last_undegraded":"2024-02-05T06:44:25.107946+0000","last_fullsized":"2024-02-05T06:44:25.107946+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:36.504651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:06.525012+0000","last_clean_scrub_stamp":"2024-02-05T06:43:36.504651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:58.084697+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":536,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.104696+0000","last_change":"2024-02-05T06:43:40.001642+0000","last_active":"2024-02-05T06:44:25.104696+0000","last_peered":"2024-02-05T06:44:25.104696+0000","last_clean":"2024-02-05T06:44:25.104696+0000","last_became_active":"2024-02-05T06:41:43.421315+0000","last_became_peered":"2024-02-05T06:41:43.421315+0000","last_unstale":"2024-02-05T06:44:25.104696+0000","last_undegraded":"2024-02-05T06:44:25.104696+0000","last_fullsized":"2024-02-05T06:44:25.104696+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:40.001568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:43:40.001568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:58.362550+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"295'19","reported_seq":439,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.101495+0000","last_change":"2024-02-05T06:43:52.801066+0000","last_active":"2024-02-05T06:44:25.101495+0000","last_peered":"2024-02-05T06:44:25.101495+0000","last_clean":"2024-02-05T06:44:25.101495+0000","last_became_active":"2024-02-05T06:36:54.959330+0000","last_became_peered":"2024-02-05T06:36:54.959330+0000","last_unstale":"2024-02-05T06:44:25.101495+0000","last_undegraded":"2024-02-05T06:44:25.101495+0000","last_fullsized":"2024-02-05T06:44:25.101495+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'19","last_scrub_stamp":"2024-02-05T06:43:52.800915+0000","last_deep_scrub":"247'13","last_deep_scrub_stamp":"2024-02-05T06:41:42.473424+0000","last_clean_scrub_stamp":"2024-02-05T06:43:52.800915+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:13.160739+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":522,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.116515+0000","last_change":"2024-02-05T06:42:40.263147+0000","last_active":"2024-02-05T06:44:25.116515+0000","last_peered":"2024-02-05T06:44:25.116515+0000","last_clean":"2024-02-05T06:44:25.116515+0000","last_became_active":"2024-02-05T06:40:27.077485+0000","last_became_peered":"2024-02-05T06:40:27.077485+0000","last_unstale":"2024-02-05T06:44:25.116515+0000","last_undegraded":"2024-02-05T06:44:25.116515+0000","last_fullsized":"2024-02-05T06:44:25.116515+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:40.262880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:42:40.262880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":408,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.101628+0000","last_change":"2024-02-05T06:41:47.546569+0000","last_active":"2024-02-05T06:44:25.101628+0000","last_peered":"2024-02-05T06:44:25.101628+0000","last_clean":"2024-02-05T06:44:25.101628+0000","last_became_active":"2024-02-05T06:36:54.959640+0000","last_became_peered":"2024-02-05T06:36:54.959640+0000","last_unstale":"2024-02-05T06:44:25.101628+0000","last_undegraded":"2024-02-05T06:44:25.101628+0000","last_fullsized":"2024-02-05T06:44:25.101628+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:41:47.546464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:41:47.546464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:26.837007+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":531,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.116824+0000","last_change":"2024-02-05T06:43:53.697132+0000","last_active":"2024-02-05T06:44:25.116824+0000","last_peered":"2024-02-05T06:44:25.116824+0000","last_clean":"2024-02-05T06:44:25.116824+0000","last_became_active":"2024-02-05T06:33:29.553864+0000","last_became_peered":"2024-02-05T06:33:29.553864+0000","last_unstale":"2024-02-05T06:44:25.116824+0000","last_undegraded":"2024-02-05T06:44:25.116824+0000","last_fullsized":"2024-02-05T06:44:25.116824+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:53.696997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:43:53.696997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:58.140971+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"295'34","reported_seq":576,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.115099+0000","last_change":"2024-02-05T06:43:56.716084+0000","last_active":"2024-02-05T06:44:25.115099+0000","last_peered":"2024-02-05T06:44:25.115099+0000","last_clean":"2024-02-05T06:44:25.115099+0000","last_became_active":"2024-02-05T06:33:31.572544+0000","last_became_peered":"2024-02-05T06:33:31.572544+0000","last_unstale":"2024-02-05T06:44:25.115099+0000","last_undegraded":"2024-02-05T06:44:25.115099+0000","last_fullsized":"2024-02-05T06:44:25.115099+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'34","last_scrub_stamp":"2024-02-05T06:43:56.715959+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:39:35.433539+0000","last_clean_scrub_stamp":"2024-02-05T06:43:56.715959+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:08.901916+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":523,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.105182+0000","last_change":"2024-02-05T06:42:52.920370+0000","last_active":"2024-02-05T06:44:25.105182+0000","last_peered":"2024-02-05T06:44:25.105182+0000","last_clean":"2024-02-05T06:44:25.105182+0000","last_became_active":"2024-02-05T06:40:27.068888+0000","last_became_peered":"2024-02-05T06:40:27.068888+0000","last_unstale":"2024-02-05T06:44:25.105182+0000","last_undegraded":"2024-02-05T06:44:25.105182+0000","last_fullsized":"2024-02-05T06:44:25.105182+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:52.920236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:42:52.920236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":87,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.116608+0000","last_change":"2024-02-05T06:42:46.353608+0000","last_active":"2024-02-05T06:44:25.116608+0000","last_peered":"2024-02-05T06:44:25.116608+0000","last_clean":"2024-02-05T06:44:25.116608+0000","last_became_active":"2024-02-05T06:42:33.129959+0000","last_became_peered":"2024-02-05T06:42:33.129959+0000","last_unstale":"2024-02-05T06:44:25.116608+0000","last_undegraded":"2024-02-05T06:44:25.116608+0000","last_fullsized":"2024-02-05T06:44:25.116608+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:46.351296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:42:46.351296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":140,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.107491+0000","last_change":"2024-02-05T06:43:56.453639+0000","last_active":"2024-02-05T06:44:25.107491+0000","last_peered":"2024-02-05T06:44:25.107491+0000","last_clean":"2024-02-05T06:44:25.107491+0000","last_became_active":"2024-02-05T06:41:43.403746+0000","last_became_peered":"2024-02-05T06:41:43.403746+0000","last_unstale":"2024-02-05T06:44:25.107491+0000","last_undegraded":"2024-02-05T06:44:25.107491+0000","last_fullsized":"2024-02-05T06:44:25.107491+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:56.453478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:44.081318+0000","last_clean_scrub_stamp":"2024-02-05T06:43:56.453478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:25.000882+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"295'19","reported_seq":435,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.102650+0000","last_change":"2024-02-05T06:43:49.843729+0000","last_active":"2024-02-05T06:44:25.102650+0000","last_peered":"2024-02-05T06:44:25.102650+0000","last_clean":"2024-02-05T06:44:25.102650+0000","last_became_active":"2024-02-05T06:36:54.969454+0000","last_became_peered":"2024-02-05T06:36:54.969454+0000","last_unstale":"2024-02-05T06:44:25.102650+0000","last_undegraded":"2024-02-05T06:44:25.102650+0000","last_fullsized":"2024-02-05T06:44:25.102650+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'19","last_scrub_stamp":"2024-02-05T06:43:49.843581+0000","last_deep_scrub":"92'13","last_deep_scrub_stamp":"2024-02-05T06:34:46.542893+0000","last_clean_scrub_stamp":"2024-02-05T06:43:49.843581+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:52.860608+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":140,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.109036+0000","last_change":"2024-02-05T06:43:53.476980+0000","last_active":"2024-02-05T06:44:25.109036+0000","last_peered":"2024-02-05T06:44:25.109036+0000","last_clean":"2024-02-05T06:44:25.109036+0000","last_became_active":"2024-02-05T06:41:43.407404+0000","last_became_peered":"2024-02-05T06:41:43.407404+0000","last_unstale":"2024-02-05T06:44:25.109036+0000","last_undegraded":"2024-02-05T06:44:25.109036+0000","last_fullsized":"2024-02-05T06:44:25.109036+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:53.476863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:21.998850+0000","last_clean_scrub_stamp":"2024-02-05T06:43:53.476863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:58.879515+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":232,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.110848+0000","last_change":"2024-02-05T06:43:54.540922+0000","last_active":"2024-02-05T06:44:25.110848+0000","last_peered":"2024-02-05T06:44:25.110848+0000","last_clean":"2024-02-05T06:44:25.110848+0000","last_became_active":"2024-02-05T06:40:27.071758+0000","last_became_peered":"2024-02-05T06:40:27.071758+0000","last_unstale":"2024-02-05T06:44:25.110848+0000","last_undegraded":"2024-02-05T06:44:25.110848+0000","last_fullsized":"2024-02-05T06:44:25.110848+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:54.540821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:54.540821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:21.286238+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":140,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.107039+0000","last_change":"2024-02-05T06:43:34.438758+0000","last_active":"2024-02-05T06:44:25.107039+0000","last_peered":"2024-02-05T06:44:25.107039+0000","last_clean":"2024-02-05T06:44:25.107039+0000","last_became_active":"2024-02-05T06:41:43.408428+0000","last_became_peered":"2024-02-05T06:41:43.408428+0000","last_unstale":"2024-02-05T06:44:25.107039+0000","last_undegraded":"2024-02-05T06:44:25.107039+0000","last_fullsized":"2024-02-05T06:44:25.107039+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:34.438631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:50.055245+0000","last_clean_scrub_stamp":"2024-02-05T06:43:34.438631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.429736+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"295'7","reported_seq":417,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.103169+0000","last_change":"2024-02-05T06:41:48.543907+0000","last_active":"2024-02-05T06:44:25.103169+0000","last_peered":"2024-02-05T06:44:25.103169+0000","last_clean":"2024-02-05T06:44:25.103169+0000","last_became_active":"2024-02-05T06:36:54.954385+0000","last_became_peered":"2024-02-05T06:36:54.954385+0000","last_unstale":"2024-02-05T06:44:25.103169+0000","last_undegraded":"2024-02-05T06:44:25.103169+0000","last_fullsized":"2024-02-05T06:44:25.103169+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-02-05T06:41:48.543648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:41:48.543648+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:26.837035+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":420,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.103566+0000","last_change":"2024-02-05T06:41:43.412335+0000","last_active":"2024-02-05T06:44:25.103566+0000","last_peered":"2024-02-05T06:44:25.103566+0000","last_clean":"2024-02-05T06:44:25.103566+0000","last_became_active":"2024-02-05T06:41:43.411344+0000","last_became_peered":"2024-02-05T06:41:43.411344+0000","last_unstale":"2024-02-05T06:44:25.103566+0000","last_undegraded":"2024-02-05T06:44:25.103566+0000","last_fullsized":"2024-02-05T06:44:25.103566+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:41:39.552367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:41:39.552367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:26.836896+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":534,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.112975+0000","last_change":"2024-02-05T06:43:37.552250+0000","last_active":"2024-02-05T06:44:25.112975+0000","last_peered":"2024-02-05T06:44:25.112975+0000","last_clean":"2024-02-05T06:44:25.112975+0000","last_became_active":"2024-02-05T06:36:54.958069+0000","last_became_peered":"2024-02-05T06:36:54.958069+0000","last_unstale":"2024-02-05T06:44:25.112975+0000","last_undegraded":"2024-02-05T06:44:25.112975+0000","last_fullsized":"2024-02-05T06:44:25.112975+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:37.552166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:37.552166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:56.420046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"281'2","reported_seq":570,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.109342+0000","last_change":"2024-02-05T06:42:58.933944+0000","last_active":"2024-02-05T06:44:25.109342+0000","last_peered":"2024-02-05T06:44:25.109342+0000","last_clean":"2024-02-05T06:44:25.109342+0000","last_became_active":"2024-02-05T06:40:27.073360+0000","last_became_peered":"2024-02-05T06:40:27.073360+0000","last_unstale":"2024-02-05T06:44:25.109342+0000","last_undegraded":"2024-02-05T06:44:25.109342+0000","last_fullsized":"2024-02-05T06:44:25.109342+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'2","last_scrub_stamp":"2024-02-05T06:42:58.933820+0000","last_deep_scrub":"281'2","last_deep_scrub_stamp":"2024-02-05T06:42:58.933820+0000","last_clean_scrub_stamp":"2024-02-05T06:42:58.933820+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"295'34","reported_seq":598,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.111868+0000","last_change":"2024-02-05T06:43:57.470734+0000","last_active":"2024-02-05T06:44:25.111868+0000","last_peered":"2024-02-05T06:44:25.111868+0000","last_clean":"2024-02-05T06:44:25.111868+0000","last_became_active":"2024-02-05T06:33:31.569763+0000","last_became_peered":"2024-02-05T06:33:31.569763+0000","last_unstale":"2024-02-05T06:44:25.111868+0000","last_undegraded":"2024-02-05T06:44:25.111868+0000","last_fullsized":"2024-02-05T06:44:25.111868+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'34","last_scrub_stamp":"2024-02-05T06:43:57.470583+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:34:35.234636+0000","last_clean_scrub_stamp":"2024-02-05T06:43:57.470583+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:14.633396+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":523,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.119437+0000","last_change":"2024-02-05T06:43:38.524343+0000","last_active":"2024-02-05T06:44:25.119437+0000","last_peered":"2024-02-05T06:44:25.119437+0000","last_clean":"2024-02-05T06:44:25.119437+0000","last_became_active":"2024-02-05T06:41:43.411442+0000","last_became_peered":"2024-02-05T06:41:43.411442+0000","last_unstale":"2024-02-05T06:44:25.119437+0000","last_undegraded":"2024-02-05T06:44:25.119437+0000","last_fullsized":"2024-02-05T06:44:25.119437+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:38.524236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:43:38.524236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:51.142093+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":304,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.108287+0000","last_change":"2024-02-05T06:43:54.234456+0000","last_active":"2024-02-05T06:44:25.108287+0000","last_peered":"2024-02-05T06:44:25.108287+0000","last_clean":"2024-02-05T06:44:25.108287+0000","last_became_active":"2024-02-05T06:39:13.248618+0000","last_became_peered":"2024-02-05T06:39:13.248618+0000","last_unstale":"2024-02-05T06:44:25.108287+0000","last_undegraded":"2024-02-05T06:44:25.108287+0000","last_fullsized":"2024-02-05T06:44:25.108287+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:54.234345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:54.234345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:55.938687+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":408,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.102570+0000","last_change":"2024-02-05T06:42:57.721396+0000","last_active":"2024-02-05T06:44:25.102570+0000","last_peered":"2024-02-05T06:44:25.102570+0000","last_clean":"2024-02-05T06:44:25.102570+0000","last_became_active":"2024-02-05T06:36:54.959070+0000","last_became_peered":"2024-02-05T06:36:54.959070+0000","last_unstale":"2024-02-05T06:44:25.102570+0000","last_undegraded":"2024-02-05T06:44:25.102570+0000","last_fullsized":"2024-02-05T06:44:25.102570+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:57.721280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:57.721280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:26.837270+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"295'18","reported_seq":604,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.116404+0000","last_change":"2024-02-05T06:42:44.182931+0000","last_active":"2024-02-05T06:44:25.116404+0000","last_peered":"2024-02-05T06:44:25.116404+0000","last_clean":"2024-02-05T06:44:25.116404+0000","last_became_active":"2024-02-05T06:36:54.954880+0000","last_became_peered":"2024-02-05T06:36:54.954880+0000","last_unstale":"2024-02-05T06:44:25.116404+0000","last_undegraded":"2024-02-05T06:44:25.116404+0000","last_fullsized":"2024-02-05T06:44:25.116404+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:42:44.182785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:42:44.182785+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"73'1","reported_seq":238,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.111694+0000","last_change":"2024-02-05T06:43:41.472748+0000","last_active":"2024-02-05T06:44:25.111694+0000","last_peered":"2024-02-05T06:44:25.111694+0000","last_clean":"2024-02-05T06:44:25.111694+0000","last_became_active":"2024-02-05T06:41:43.418640+0000","last_became_peered":"2024-02-05T06:41:43.418640+0000","last_unstale":"2024-02-05T06:44:25.111694+0000","last_undegraded":"2024-02-05T06:44:25.111694+0000","last_fullsized":"2024-02-05T06:44:25.111694+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-02-05T06:43:41.472635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:43:41.472635+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:54.907618+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"208'12","reported_seq":718,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.109195+0000","last_change":"2024-02-05T06:43:43.224045+0000","last_active":"2024-02-05T06:44:25.109195+0000","last_peered":"2024-02-05T06:44:25.109195+0000","last_clean":"2024-02-05T06:44:25.109195+0000","last_became_active":"2024-02-05T06:40:27.063399+0000","last_became_peered":"2024-02-05T06:40:27.063399+0000","last_unstale":"2024-02-05T06:44:25.109195+0000","last_undegraded":"2024-02-05T06:44:25.109195+0000","last_fullsized":"2024-02-05T06:44:25.109195+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'12","last_scrub_stamp":"2024-02-05T06:43:43.223933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:43.223933+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:53.049215+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":559,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.107860+0000","last_change":"2024-02-05T06:44:04.592727+0000","last_active":"2024-02-05T06:44:25.107860+0000","last_peered":"2024-02-05T06:44:25.107860+0000","last_clean":"2024-02-05T06:44:25.107860+0000","last_became_active":"2024-02-05T06:36:54.958414+0000","last_became_peered":"2024-02-05T06:36:54.958414+0000","last_unstale":"2024-02-05T06:44:25.107860+0000","last_undegraded":"2024-02-05T06:44:25.107860+0000","last_fullsized":"2024-02-05T06:44:25.107860+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:44:04.592594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:44:04.592594+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:23.618630+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"295'25","reported_seq":599,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.104397+0000","last_change":"2024-02-05T06:43:46.110585+0000","last_active":"2024-02-05T06:44:25.104397+0000","last_peered":"2024-02-05T06:44:25.104397+0000","last_clean":"2024-02-05T06:44:25.104397+0000","last_became_active":"2024-02-05T06:40:27.100610+0000","last_became_peered":"2024-02-05T06:40:27.100610+0000","last_unstale":"2024-02-05T06:44:25.104397+0000","last_undegraded":"2024-02-05T06:44:25.104397+0000","last_fullsized":"2024-02-05T06:44:25.104397+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'25","last_scrub_stamp":"2024-02-05T06:43:46.110443+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-02-05T06:34:50.718768+0000","last_clean_scrub_stamp":"2024-02-05T06:43:46.110443+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:54.008936+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":542,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.113118+0000","last_change":"2024-02-05T06:41:43.414961+0000","last_active":"2024-02-05T06:44:25.113118+0000","last_peered":"2024-02-05T06:44:25.113118+0000","last_clean":"2024-02-05T06:44:25.113118+0000","last_became_active":"2024-02-05T06:41:43.414638+0000","last_became_peered":"2024-02-05T06:41:43.414638+0000","last_unstale":"2024-02-05T06:44:25.113118+0000","last_undegraded":"2024-02-05T06:44:25.113118+0000","last_fullsized":"2024-02-05T06:44:25.113118+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:41:34.298647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:41:34.298647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":438,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.116665+0000","last_change":"2024-02-05T06:42:58.394440+0000","last_active":"2024-02-05T06:44:25.116665+0000","last_peered":"2024-02-05T06:44:25.116665+0000","last_clean":"2024-02-05T06:44:25.116665+0000","last_became_active":"2024-02-05T06:40:27.097169+0000","last_became_peered":"2024-02-05T06:40:27.097169+0000","last_unstale":"2024-02-05T06:44:25.116665+0000","last_undegraded":"2024-02-05T06:44:25.116665+0000","last_fullsized":"2024-02-05T06:44:25.116665+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:58.394289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:08.713894+0000","last_clean_scrub_stamp":"2024-02-05T06:42:58.394289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:25.870421+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":140,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.106306+0000","last_change":"2024-02-05T06:43:44.345193+0000","last_active":"2024-02-05T06:44:25.106306+0000","last_peered":"2024-02-05T06:44:25.106306+0000","last_clean":"2024-02-05T06:44:25.106306+0000","last_became_active":"2024-02-05T06:41:43.411043+0000","last_became_peered":"2024-02-05T06:41:43.411043+0000","last_unstale":"2024-02-05T06:44:25.106306+0000","last_undegraded":"2024-02-05T06:44:25.106306+0000","last_fullsized":"2024-02-05T06:44:25.106306+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:44.345092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:43:44.345092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:05.600762+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"295'24","reported_seq":609,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.109382+0000","last_change":"2024-02-05T06:43:03.221573+0000","last_active":"2024-02-05T06:44:25.109382+0000","last_peered":"2024-02-05T06:44:25.109382+0000","last_clean":"2024-02-05T06:44:25.109382+0000","last_became_active":"2024-02-05T06:41:43.419769+0000","last_became_peered":"2024-02-05T06:41:43.419769+0000","last_unstale":"2024-02-05T06:44:25.109382+0000","last_undegraded":"2024-02-05T06:44:25.109382+0000","last_fullsized":"2024-02-05T06:44:25.109382+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'16","last_scrub_stamp":"2024-02-05T06:43:03.221466+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:43:03.221466+0000","last_clean_scrub_stamp":"2024-02-05T06:43:03.221466+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:26.443563+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":232,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.111007+0000","last_change":"2024-02-05T06:43:48.512794+0000","last_active":"2024-02-05T06:44:25.111007+0000","last_peered":"2024-02-05T06:44:25.111007+0000","last_clean":"2024-02-05T06:44:25.111007+0000","last_became_active":"2024-02-05T06:40:27.098348+0000","last_became_peered":"2024-02-05T06:40:27.098348+0000","last_unstale":"2024-02-05T06:44:25.111007+0000","last_undegraded":"2024-02-05T06:44:25.111007+0000","last_fullsized":"2024-02-05T06:44:25.111007+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:48.512673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:48.512673+0000","last_clean_scrub_stamp":"2024-02-05T06:43:48.512673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:56.039239+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":561,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.108212+0000","last_change":"2024-02-05T06:44:01.297159+0000","last_active":"2024-02-05T06:44:25.108212+0000","last_peered":"2024-02-05T06:44:25.108212+0000","last_clean":"2024-02-05T06:44:25.108212+0000","last_became_active":"2024-02-05T06:40:27.060112+0000","last_became_peered":"2024-02-05T06:40:27.060112+0000","last_unstale":"2024-02-05T06:44:25.108212+0000","last_undegraded":"2024-02-05T06:44:25.108212+0000","last_fullsized":"2024-02-05T06:44:25.108212+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:01.297081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:44:01.297081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:14.265032+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":535,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.104923+0000","last_change":"2024-02-05T06:43:49.118084+0000","last_active":"2024-02-05T06:44:25.104923+0000","last_peered":"2024-02-05T06:44:25.104923+0000","last_clean":"2024-02-05T06:44:25.104923+0000","last_became_active":"2024-02-05T06:41:43.424459+0000","last_became_peered":"2024-02-05T06:41:43.424459+0000","last_unstale":"2024-02-05T06:44:25.104923+0000","last_undegraded":"2024-02-05T06:44:25.104923+0000","last_fullsized":"2024-02-05T06:44:25.104923+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:49.117949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:49.117949+0000","last_clean_scrub_stamp":"2024-02-05T06:43:49.117949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:17.641091+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"295'8","reported_seq":530,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.115091+0000","last_change":"2024-02-05T06:43:51.672673+0000","last_active":"2024-02-05T06:44:25.115091+0000","last_peered":"2024-02-05T06:44:25.115091+0000","last_clean":"2024-02-05T06:44:25.115091+0000","last_became_active":"2024-02-05T06:33:31.573410+0000","last_became_peered":"2024-02-05T06:33:31.573410+0000","last_unstale":"2024-02-05T06:44:25.115091+0000","last_undegraded":"2024-02-05T06:44:25.115091+0000","last_fullsized":"2024-02-05T06:44:25.115091+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'8","last_scrub_stamp":"2024-02-05T06:43:51.672524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:51.672524+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:06.073121+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":297,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.107845+0000","last_change":"2024-02-05T06:43:59.635993+0000","last_active":"2024-02-05T06:44:25.107845+0000","last_peered":"2024-02-05T06:44:25.107845+0000","last_clean":"2024-02-05T06:44:25.107845+0000","last_became_active":"2024-02-05T06:39:13.245684+0000","last_became_peered":"2024-02-05T06:39:13.245684+0000","last_unstale":"2024-02-05T06:44:25.107845+0000","last_undegraded":"2024-02-05T06:44:25.107845+0000","last_fullsized":"2024-02-05T06:44:25.107845+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:59.635733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:43:59.635733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:12.882226+0000","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":224,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.110444+0000","last_change":"2024-02-05T06:42:34.359574+0000","last_active":"2024-02-05T06:44:25.110444+0000","last_peered":"2024-02-05T06:44:25.110444+0000","last_clean":"2024-02-05T06:44:25.110444+0000","last_became_active":"2024-02-05T06:40:27.062975+0000","last_became_peered":"2024-02-05T06:40:27.062975+0000","last_unstale":"2024-02-05T06:44:25.110444+0000","last_undegraded":"2024-02-05T06:44:25.110444+0000","last_fullsized":"2024-02-05T06:44:25.110444+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:34.359494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:42:34.359494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":140,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.109759+0000","last_change":"2024-02-05T06:42:33.724047+0000","last_active":"2024-02-05T06:44:25.109759+0000","last_peered":"2024-02-05T06:44:25.109759+0000","last_clean":"2024-02-05T06:44:25.109759+0000","last_became_active":"2024-02-05T06:41:43.412654+0000","last_became_peered":"2024-02-05T06:41:43.412654+0000","last_unstale":"2024-02-05T06:44:25.109759+0000","last_undegraded":"2024-02-05T06:44:25.109759+0000","last_fullsized":"2024-02-05T06:44:25.109759+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:33.723941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:33.723941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"295'24","reported_seq":608,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.115061+0000","last_change":"2024-02-05T06:43:39.456286+0000","last_active":"2024-02-05T06:44:25.115061+0000","last_peered":"2024-02-05T06:44:25.115061+0000","last_clean":"2024-02-05T06:44:25.115061+0000","last_became_active":"2024-02-05T06:40:27.080906+0000","last_became_peered":"2024-02-05T06:40:27.080906+0000","last_unstale":"2024-02-05T06:44:25.115061+0000","last_undegraded":"2024-02-05T06:44:25.115061+0000","last_fullsized":"2024-02-05T06:44:25.115061+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:43:39.456190+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-02-05T06:35:22.938675+0000","last_clean_scrub_stamp":"2024-02-05T06:43:39.456190+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:57.044088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"73'1","reported_seq":532,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.103463+0000","last_change":"2024-02-05T06:42:54.980648+0000","last_active":"2024-02-05T06:44:25.103463+0000","last_peered":"2024-02-05T06:44:25.103463+0000","last_clean":"2024-02-05T06:44:25.103463+0000","last_became_active":"2024-02-05T06:41:43.409761+0000","last_became_peered":"2024-02-05T06:41:43.409761+0000","last_unstale":"2024-02-05T06:44:25.103463+0000","last_undegraded":"2024-02-05T06:44:25.103463+0000","last_fullsized":"2024-02-05T06:44:25.103463+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-02-05T06:42:54.980529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:42:54.980529+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":514,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.116429+0000","last_change":"2024-02-05T06:43:58.672945+0000","last_active":"2024-02-05T06:44:25.116429+0000","last_peered":"2024-02-05T06:44:25.116429+0000","last_clean":"2024-02-05T06:44:25.116429+0000","last_became_active":"2024-02-05T06:33:33.593901+0000","last_became_peered":"2024-02-05T06:33:33.593901+0000","last_unstale":"2024-02-05T06:44:25.116429+0000","last_undegraded":"2024-02-05T06:44:25.116429+0000","last_fullsized":"2024-02-05T06:44:25.116429+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:58.672808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:58.672808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:02.151514+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":232,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.112897+0000","last_change":"2024-02-05T06:44:00.501883+0000","last_active":"2024-02-05T06:44:25.112897+0000","last_peered":"2024-02-05T06:44:25.112897+0000","last_clean":"2024-02-05T06:44:25.112897+0000","last_became_active":"2024-02-05T06:40:27.077701+0000","last_became_peered":"2024-02-05T06:40:27.077701+0000","last_unstale":"2024-02-05T06:44:25.112897+0000","last_undegraded":"2024-02-05T06:44:25.112897+0000","last_fullsized":"2024-02-05T06:44:25.112897+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:00.501644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:44:00.501644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:09.346038+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"295'15","reported_seq":308,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.103725+0000","last_change":"2024-02-05T06:42:45.004868+0000","last_active":"2024-02-05T06:44:25.103725+0000","last_peered":"2024-02-05T06:44:25.103725+0000","last_clean":"2024-02-05T06:44:25.103725+0000","last_became_active":"2024-02-05T06:39:13.244124+0000","last_became_peered":"2024-02-05T06:39:13.244124+0000","last_unstale":"2024-02-05T06:44:25.103725+0000","last_undegraded":"2024-02-05T06:44:25.103725+0000","last_fullsized":"2024-02-05T06:44:25.103725+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:42:45.004744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:42:45.004744+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":522,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.114744+0000","last_change":"2024-02-05T06:42:36.192914+0000","last_active":"2024-02-05T06:44:25.114744+0000","last_peered":"2024-02-05T06:44:25.114744+0000","last_clean":"2024-02-05T06:44:25.114744+0000","last_became_active":"2024-02-05T06:41:43.417155+0000","last_became_peered":"2024-02-05T06:41:43.417155+0000","last_unstale":"2024-02-05T06:44:25.114744+0000","last_undegraded":"2024-02-05T06:44:25.114744+0000","last_fullsized":"2024-02-05T06:44:25.114744+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:36.192791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:42:36.192791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":548,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.112239+0000","last_change":"2024-02-05T06:43:57.477466+0000","last_active":"2024-02-05T06:44:25.112239+0000","last_peered":"2024-02-05T06:44:25.112239+0000","last_clean":"2024-02-05T06:44:25.112239+0000","last_became_active":"2024-02-05T06:41:43.421433+0000","last_became_peered":"2024-02-05T06:41:43.421433+0000","last_unstale":"2024-02-05T06:44:25.112239+0000","last_undegraded":"2024-02-05T06:44:25.112239+0000","last_fullsized":"2024-02-05T06:44:25.112239+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:57.477296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:57.477296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:57.550396+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":420,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.102777+0000","last_change":"2024-02-05T06:42:58.691714+0000","last_active":"2024-02-05T06:44:25.102777+0000","last_peered":"2024-02-05T06:44:25.102777+0000","last_clean":"2024-02-05T06:44:25.102777+0000","last_became_active":"2024-02-05T06:40:27.065047+0000","last_became_peered":"2024-02-05T06:40:27.065047+0000","last_unstale":"2024-02-05T06:44:25.102777+0000","last_undegraded":"2024-02-05T06:44:25.102777+0000","last_fullsized":"2024-02-05T06:44:25.102777+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:58.691641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:58.691641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:26.837164+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"295'24","reported_seq":432,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.103220+0000","last_change":"2024-02-05T06:43:46.883158+0000","last_active":"2024-02-05T06:44:25.103220+0000","last_peered":"2024-02-05T06:44:25.103220+0000","last_clean":"2024-02-05T06:44:25.103220+0000","last_became_active":"2024-02-05T06:36:54.961610+0000","last_became_peered":"2024-02-05T06:36:54.961610+0000","last_unstale":"2024-02-05T06:44:25.103220+0000","last_undegraded":"2024-02-05T06:44:25.103220+0000","last_fullsized":"2024-02-05T06:44:25.103220+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:43:46.883030+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:35:02.589229+0000","last_clean_scrub_stamp":"2024-02-05T06:43:46.883030+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:49.410851+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":523,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.104115+0000","last_change":"2024-02-05T06:43:35.958292+0000","last_active":"2024-02-05T06:44:25.104115+0000","last_peered":"2024-02-05T06:44:25.104115+0000","last_clean":"2024-02-05T06:44:25.104115+0000","last_became_active":"2024-02-05T06:40:27.096132+0000","last_became_peered":"2024-02-05T06:40:27.096132+0000","last_unstale":"2024-02-05T06:44:25.104115+0000","last_undegraded":"2024-02-05T06:44:25.104115+0000","last_fullsized":"2024-02-05T06:44:25.104115+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:35.958174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:43:35.958174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:36.633481+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"249'10","reported_seq":177,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.107114+0000","last_change":"2024-02-05T06:42:50.652634+0000","last_active":"2024-02-05T06:44:25.107114+0000","last_peered":"2024-02-05T06:44:25.107114+0000","last_clean":"2024-02-05T06:44:25.107114+0000","last_became_active":"2024-02-05T06:41:43.418753+0000","last_became_peered":"2024-02-05T06:41:43.418753+0000","last_unstale":"2024-02-05T06:44:25.107114+0000","last_undegraded":"2024-02-05T06:44:25.107114+0000","last_fullsized":"2024-02-05T06:44:25.107114+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'10","last_scrub_stamp":"2024-02-05T06:42:50.652545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:42:50.652545+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":224,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.113413+0000","last_change":"2024-02-05T06:42:41.410614+0000","last_active":"2024-02-05T06:44:25.113413+0000","last_peered":"2024-02-05T06:44:25.113413+0000","last_clean":"2024-02-05T06:44:25.113413+0000","last_became_active":"2024-02-05T06:40:27.098418+0000","last_became_peered":"2024-02-05T06:40:27.098418+0000","last_unstale":"2024-02-05T06:44:25.113413+0000","last_undegraded":"2024-02-05T06:44:25.113413+0000","last_fullsized":"2024-02-05T06:44:25.113413+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:41.410495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:41.410495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"295'20","reported_seq":602,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.117225+0000","last_change":"2024-02-05T06:42:47.381961+0000","last_active":"2024-02-05T06:44:25.117225+0000","last_peered":"2024-02-05T06:44:25.117225+0000","last_clean":"2024-02-05T06:44:25.117225+0000","last_became_active":"2024-02-05T06:40:27.093418+0000","last_became_peered":"2024-02-05T06:40:27.093418+0000","last_unstale":"2024-02-05T06:44:25.117225+0000","last_undegraded":"2024-02-05T06:44:25.117225+0000","last_fullsized":"2024-02-05T06:44:25.117225+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-02-05T06:42:47.381317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:42:47.381317+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":420,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.103804+0000","last_change":"2024-02-05T06:41:43.413904+0000","last_active":"2024-02-05T06:44:25.103804+0000","last_peered":"2024-02-05T06:44:25.103804+0000","last_clean":"2024-02-05T06:44:25.103804+0000","last_became_active":"2024-02-05T06:41:43.413562+0000","last_became_peered":"2024-02-05T06:41:43.413562+0000","last_unstale":"2024-02-05T06:44:25.103804+0000","last_undegraded":"2024-02-05T06:44:25.103804+0000","last_fullsized":"2024-02-05T06:44:25.103804+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:41:38.559780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:41:38.559780+0000","last_clean_scrub_stamp":"2024-02-05T06:41:38.559780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:26.836868+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"208'10","reported_seq":674,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.117614+0000","last_change":"2024-02-05T06:41:42.168574+0000","last_active":"2024-02-05T06:44:25.117614+0000","last_peered":"2024-02-05T06:44:25.117614+0000","last_clean":"2024-02-05T06:44:25.117614+0000","last_became_active":"2024-02-05T06:40:27.096584+0000","last_became_peered":"2024-02-05T06:40:27.096584+0000","last_unstale":"2024-02-05T06:44:25.117614+0000","last_undegraded":"2024-02-05T06:44:25.117614+0000","last_fullsized":"2024-02-05T06:44:25.117614+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-02-05T06:41:42.168453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:41:42.168453+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":543,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.104307+0000","last_change":"2024-02-05T06:44:01.185774+0000","last_active":"2024-02-05T06:44:25.104307+0000","last_peered":"2024-02-05T06:44:25.104307+0000","last_clean":"2024-02-05T06:44:25.104307+0000","last_became_active":"2024-02-05T06:40:27.088096+0000","last_became_peered":"2024-02-05T06:40:27.088096+0000","last_unstale":"2024-02-05T06:44:25.104307+0000","last_undegraded":"2024-02-05T06:44:25.104307+0000","last_fullsized":"2024-02-05T06:44:25.104307+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:01.185653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:44:01.185653+0000","last_clean_scrub_stamp":"2024-02-05T06:44:01.185653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:04.676309+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"295'15","reported_seq":159,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.107566+0000","last_change":"2024-02-05T06:42:52.697830+0000","last_active":"2024-02-05T06:44:25.107566+0000","last_peered":"2024-02-05T06:44:25.107566+0000","last_clean":"2024-02-05T06:44:25.107566+0000","last_became_active":"2024-02-05T06:41:43.413708+0000","last_became_peered":"2024-02-05T06:41:43.413708+0000","last_unstale":"2024-02-05T06:44:25.107566+0000","last_undegraded":"2024-02-05T06:44:25.107566+0000","last_fullsized":"2024-02-05T06:44:25.107566+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:42:52.697346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:42:52.697346+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":556,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.112311+0000","last_change":"2024-02-05T06:43:01.524727+0000","last_active":"2024-02-05T06:44:25.112311+0000","last_peered":"2024-02-05T06:44:25.112311+0000","last_clean":"2024-02-05T06:44:25.112311+0000","last_became_active":"2024-02-05T06:40:27.090632+0000","last_became_peered":"2024-02-05T06:40:27.090632+0000","last_unstale":"2024-02-05T06:44:25.112311+0000","last_undegraded":"2024-02-05T06:44:25.112311+0000","last_fullsized":"2024-02-05T06:44:25.112311+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:01.524512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:43:01.524512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":540,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.119357+0000","last_change":"2024-02-05T06:44:01.644210+0000","last_active":"2024-02-05T06:44:25.119357+0000","last_peered":"2024-02-05T06:44:25.119357+0000","last_clean":"2024-02-05T06:44:25.119357+0000","last_became_active":"2024-02-05T06:40:27.091544+0000","last_became_peered":"2024-02-05T06:40:27.091544+0000","last_unstale":"2024-02-05T06:44:25.119357+0000","last_undegraded":"2024-02-05T06:44:25.119357+0000","last_fullsized":"2024-02-05T06:44:25.119357+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:01.644095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:44:01.644095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:07.661277+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":224,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.111051+0000","last_change":"2024-02-05T06:42:46.409374+0000","last_active":"2024-02-05T06:44:25.111051+0000","last_peered":"2024-02-05T06:44:25.111051+0000","last_clean":"2024-02-05T06:44:25.111051+0000","last_became_active":"2024-02-05T06:40:27.064388+0000","last_became_peered":"2024-02-05T06:40:27.064388+0000","last_unstale":"2024-02-05T06:44:25.111051+0000","last_undegraded":"2024-02-05T06:44:25.111051+0000","last_fullsized":"2024-02-05T06:44:25.111051+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:46.409259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:46.409259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"295'15","reported_seq":584,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.104488+0000","last_change":"2024-02-05T06:44:04.244244+0000","last_active":"2024-02-05T06:44:25.104488+0000","last_peered":"2024-02-05T06:44:25.104488+0000","last_clean":"2024-02-05T06:44:25.104488+0000","last_became_active":"2024-02-05T06:40:27.101116+0000","last_became_peered":"2024-02-05T06:40:27.101116+0000","last_unstale":"2024-02-05T06:44:25.104488+0000","last_undegraded":"2024-02-05T06:44:25.104488+0000","last_fullsized":"2024-02-05T06:44:25.104488+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:44:04.244094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:44:04.244094+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:13.039537+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":224,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.113030+0000","last_change":"2024-02-05T06:42:49.428563+0000","last_active":"2024-02-05T06:44:25.113030+0000","last_peered":"2024-02-05T06:44:25.113030+0000","last_clean":"2024-02-05T06:44:25.113030+0000","last_became_active":"2024-02-05T06:40:27.063350+0000","last_became_peered":"2024-02-05T06:40:27.063350+0000","last_unstale":"2024-02-05T06:44:25.113030+0000","last_undegraded":"2024-02-05T06:44:25.113030+0000","last_fullsized":"2024-02-05T06:44:25.113030+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:49.428464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:42:49.428464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":140,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.109661+0000","last_change":"2024-02-05T06:42:58.680331+0000","last_active":"2024-02-05T06:44:25.109661+0000","last_peered":"2024-02-05T06:44:25.109661+0000","last_clean":"2024-02-05T06:44:25.109661+0000","last_became_active":"2024-02-05T06:41:43.406094+0000","last_became_peered":"2024-02-05T06:41:43.406094+0000","last_unstale":"2024-02-05T06:44:25.109661+0000","last_undegraded":"2024-02-05T06:44:25.109661+0000","last_fullsized":"2024-02-05T06:44:25.109661+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:58.680262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:45.096340+0000","last_clean_scrub_stamp":"2024-02-05T06:42:58.680262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":549,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.107922+0000","last_change":"2024-02-05T06:42:48.170292+0000","last_active":"2024-02-05T06:44:25.107922+0000","last_peered":"2024-02-05T06:44:25.107922+0000","last_clean":"2024-02-05T06:44:25.107922+0000","last_became_active":"2024-02-05T06:41:43.412130+0000","last_became_peered":"2024-02-05T06:41:43.412130+0000","last_unstale":"2024-02-05T06:44:25.107922+0000","last_undegraded":"2024-02-05T06:44:25.107922+0000","last_fullsized":"2024-02-05T06:44:25.107922+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:48.170173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:48.170173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"295'12","reported_seq":564,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.119526+0000","last_change":"2024-02-05T06:43:37.426033+0000","last_active":"2024-02-05T06:44:25.119526+0000","last_peered":"2024-02-05T06:44:25.119526+0000","last_clean":"2024-02-05T06:44:25.119526+0000","last_became_active":"2024-02-05T06:40:27.091689+0000","last_became_peered":"2024-02-05T06:40:27.091689+0000","last_unstale":"2024-02-05T06:44:25.119526+0000","last_undegraded":"2024-02-05T06:44:25.119526+0000","last_fullsized":"2024-02-05T06:44:25.119526+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'12","last_scrub_stamp":"2024-02-05T06:43:37.425879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:37.425879+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:38.426084+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"308'5128","reported_seq":5172,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:27.153968+0000","last_change":"2024-02-05T06:43:29.702891+0000","last_active":"2024-02-05T06:44:27.153968+0000","last_peered":"2024-02-05T06:44:27.153968+0000","last_clean":"2024-02-05T06:44:27.153968+0000","last_became_active":"2024-02-05T06:43:29.702579+0000","last_became_peered":"2024-02-05T06:43:29.702579+0000","last_unstale":"2024-02-05T06:44:27.153968+0000","last_undegraded":"2024-02-05T06:44:27.153968+0000","last_fullsized":"2024-02-05T06:44:27.153968+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":5128,"log_dups_size":0,"ondisk_log_size":5128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:31.907217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336068608,"num_objects":5128,"num_object_clones":0,"num_object_copies":15384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5128,"num_write_kb":328192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":140,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.108560+0000","last_change":"2024-02-05T06:42:59.672197+0000","last_active":"2024-02-05T06:44:25.108560+0000","last_peered":"2024-02-05T06:44:25.108560+0000","last_clean":"2024-02-05T06:44:25.108560+0000","last_became_active":"2024-02-05T06:41:43.421231+0000","last_became_peered":"2024-02-05T06:41:43.421231+0000","last_unstale":"2024-02-05T06:44:25.108560+0000","last_undegraded":"2024-02-05T06:44:25.108560+0000","last_fullsized":"2024-02-05T06:44:25.108560+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:59.672110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:42:59.672110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":238,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.111420+0000","last_change":"2024-02-05T06:43:56.531681+0000","last_active":"2024-02-05T06:44:25.111420+0000","last_peered":"2024-02-05T06:44:25.111420+0000","last_clean":"2024-02-05T06:44:25.111420+0000","last_became_active":"2024-02-05T06:40:27.068279+0000","last_became_peered":"2024-02-05T06:40:27.068279+0000","last_unstale":"2024-02-05T06:44:25.111420+0000","last_undegraded":"2024-02-05T06:44:25.111420+0000","last_fullsized":"2024-02-05T06:44:25.111420+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-02-05T06:43:56.531539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:43:56.531539+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:24.413854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"295'27","reported_seq":174,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.106568+0000","last_change":"2024-02-05T06:43:43.345357+0000","last_active":"2024-02-05T06:44:25.106568+0000","last_peered":"2024-02-05T06:44:25.106568+0000","last_clean":"2024-02-05T06:44:25.106568+0000","last_became_active":"2024-02-05T06:41:43.412579+0000","last_became_peered":"2024-02-05T06:41:43.412579+0000","last_unstale":"2024-02-05T06:44:25.106568+0000","last_undegraded":"2024-02-05T06:44:25.106568+0000","last_fullsized":"2024-02-05T06:44:25.106568+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'27","last_scrub_stamp":"2024-02-05T06:43:43.344878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:43.344878+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:55.975676+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"208'12","reported_seq":276,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.110957+0000","last_change":"2024-02-05T06:42:50.411738+0000","last_active":"2024-02-05T06:44:25.110957+0000","last_peered":"2024-02-05T06:44:25.110957+0000","last_clean":"2024-02-05T06:44:25.110957+0000","last_became_active":"2024-02-05T06:40:27.064195+0000","last_became_peered":"2024-02-05T06:40:27.064195+0000","last_unstale":"2024-02-05T06:44:25.110957+0000","last_undegraded":"2024-02-05T06:44:25.110957+0000","last_fullsized":"2024-02-05T06:44:25.110957+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'12","last_scrub_stamp":"2024-02-05T06:42:50.411654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:42:50.411654+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"308'5203","reported_seq":5247,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:26.799445+0000","last_change":"2024-02-05T06:43:29.697813+0000","last_active":"2024-02-05T06:44:26.799445+0000","last_peered":"2024-02-05T06:44:26.799445+0000","last_clean":"2024-02-05T06:44:26.799445+0000","last_became_active":"2024-02-05T06:43:29.697495+0000","last_became_peered":"2024-02-05T06:43:29.697495+0000","last_unstale":"2024-02-05T06:44:26.799445+0000","last_undegraded":"2024-02-05T06:44:26.799445+0000","last_fullsized":"2024-02-05T06:44:26.799445+0000","mapping_epoch":291,"log_start":"296'500","ondisk_log_start":"296'500","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4703,"log_dups_size":0,"ondisk_log_size":4703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:32.348493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340983808,"num_objects":5203,"num_object_clones":0,"num_object_copies":15609,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5203,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5203,"num_write_kb":332992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":522,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.116697+0000","last_change":"2024-02-05T06:43:47.662237+0000","last_active":"2024-02-05T06:44:25.116697+0000","last_peered":"2024-02-05T06:44:25.116697+0000","last_clean":"2024-02-05T06:44:25.116697+0000","last_became_active":"2024-02-05T06:40:27.092613+0000","last_became_peered":"2024-02-05T06:40:27.092613+0000","last_unstale":"2024-02-05T06:44:25.116697+0000","last_undegraded":"2024-02-05T06:44:25.116697+0000","last_fullsized":"2024-02-05T06:44:25.116697+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:47.662025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:43:47.662025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:04.272911+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":224,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.111899+0000","last_change":"2024-02-05T06:42:43.390081+0000","last_active":"2024-02-05T06:44:25.111899+0000","last_peered":"2024-02-05T06:44:25.111899+0000","last_clean":"2024-02-05T06:44:25.111899+0000","last_became_active":"2024-02-05T06:40:27.063300+0000","last_became_peered":"2024-02-05T06:40:27.063300+0000","last_unstale":"2024-02-05T06:44:25.111899+0000","last_undegraded":"2024-02-05T06:44:25.111899+0000","last_fullsized":"2024-02-05T06:44:25.111899+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:43.389980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:43.389980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"295'18","reported_seq":429,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.102018+0000","last_change":"2024-02-05T06:41:52.647443+0000","last_active":"2024-02-05T06:44:25.102018+0000","last_peered":"2024-02-05T06:44:25.102018+0000","last_clean":"2024-02-05T06:44:25.102018+0000","last_became_active":"2024-02-05T06:36:54.954071+0000","last_became_peered":"2024-02-05T06:36:54.954071+0000","last_unstale":"2024-02-05T06:44:25.102018+0000","last_undegraded":"2024-02-05T06:44:25.102018+0000","last_fullsized":"2024-02-05T06:44:25.102018+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:41:52.647335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:41:52.647335+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:26.837064+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":529,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.110934+0000","last_change":"2024-02-05T06:43:44.383974+0000","last_active":"2024-02-05T06:44:25.110934+0000","last_peered":"2024-02-05T06:44:25.110934+0000","last_clean":"2024-02-05T06:44:25.110934+0000","last_became_active":"2024-02-05T06:33:33.592546+0000","last_became_peered":"2024-02-05T06:33:33.592546+0000","last_unstale":"2024-02-05T06:44:25.110934+0000","last_undegraded":"2024-02-05T06:44:25.110934+0000","last_fullsized":"2024-02-05T06:44:25.110934+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:44.382909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:44.382909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:50.463660+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"308'5061","reported_seq":5105,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:26.251837+0000","last_change":"2024-02-05T06:43:29.702980+0000","last_active":"2024-02-05T06:44:26.251837+0000","last_peered":"2024-02-05T06:44:26.251837+0000","last_clean":"2024-02-05T06:44:26.251837+0000","last_became_active":"2024-02-05T06:43:29.702659+0000","last_became_peered":"2024-02-05T06:43:29.702659+0000","last_unstale":"2024-02-05T06:44:26.251837+0000","last_undegraded":"2024-02-05T06:44:26.251837+0000","last_fullsized":"2024-02-05T06:44:26.251837+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":5061,"log_dups_size":0,"ondisk_log_size":5061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:50.762568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331677696,"num_objects":5061,"num_object_clones":0,"num_object_copies":15183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5061,"num_write_kb":323904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":224,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.110529+0000","last_change":"2024-02-05T06:42:58.350505+0000","last_active":"2024-02-05T06:44:25.110529+0000","last_peered":"2024-02-05T06:44:25.110529+0000","last_clean":"2024-02-05T06:44:25.110529+0000","last_became_active":"2024-02-05T06:40:27.218081+0000","last_became_peered":"2024-02-05T06:40:27.218081+0000","last_unstale":"2024-02-05T06:44:25.110529+0000","last_undegraded":"2024-02-05T06:44:25.110529+0000","last_fullsized":"2024-02-05T06:44:25.110529+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:58.350239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:32.925000+0000","last_clean_scrub_stamp":"2024-02-05T06:42:58.350239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":554,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.111641+0000","last_change":"2024-02-05T06:44:01.453193+0000","last_active":"2024-02-05T06:44:25.111641+0000","last_peered":"2024-02-05T06:44:25.111641+0000","last_clean":"2024-02-05T06:44:25.111641+0000","last_became_active":"2024-02-05T06:41:43.413095+0000","last_became_peered":"2024-02-05T06:41:43.413095+0000","last_unstale":"2024-02-05T06:44:25.111641+0000","last_undegraded":"2024-02-05T06:44:25.111641+0000","last_fullsized":"2024-02-05T06:44:25.111641+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:01.452385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:44:01.452385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:30.760517+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":230,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.112969+0000","last_change":"2024-02-05T06:43:02.361335+0000","last_active":"2024-02-05T06:44:25.112969+0000","last_peered":"2024-02-05T06:44:25.112969+0000","last_clean":"2024-02-05T06:44:25.112969+0000","last_became_active":"2024-02-05T06:41:43.418202+0000","last_became_peered":"2024-02-05T06:41:43.418202+0000","last_unstale":"2024-02-05T06:44:25.112969+0000","last_undegraded":"2024-02-05T06:44:25.112969+0000","last_fullsized":"2024-02-05T06:44:25.112969+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:02.361235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:43:02.361235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"295'44","reported_seq":676,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.109703+0000","last_change":"2024-02-05T06:42:50.149138+0000","last_active":"2024-02-05T06:44:25.109703+0000","last_peered":"2024-02-05T06:44:25.109703+0000","last_clean":"2024-02-05T06:44:25.109703+0000","last_became_active":"2024-02-05T06:40:27.068417+0000","last_became_peered":"2024-02-05T06:40:27.068417+0000","last_unstale":"2024-02-05T06:44:25.109703+0000","last_undegraded":"2024-02-05T06:44:25.109703+0000","last_fullsized":"2024-02-05T06:44:25.109703+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'28","last_scrub_stamp":"2024-02-05T06:42:50.148881+0000","last_deep_scrub":"279'28","last_deep_scrub_stamp":"2024-02-05T06:42:50.148881+0000","last_clean_scrub_stamp":"2024-02-05T06:42:50.148881+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:26.443593+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"308'5158","reported_seq":5202,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:26.260088+0000","last_change":"2024-02-05T06:43:29.843229+0000","last_active":"2024-02-05T06:44:26.260088+0000","last_peered":"2024-02-05T06:44:26.260088+0000","last_clean":"2024-02-05T06:44:26.260088+0000","last_became_active":"2024-02-05T06:43:29.842930+0000","last_became_peered":"2024-02-05T06:43:29.842930+0000","last_unstale":"2024-02-05T06:44:26.260088+0000","last_undegraded":"2024-02-05T06:44:26.260088+0000","last_fullsized":"2024-02-05T06:44:26.260088+0000","mapping_epoch":291,"log_start":"296'100","ondisk_log_start":"296'100","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":5058,"log_dups_size":0,"ondisk_log_size":5058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:53.746948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338034688,"num_objects":5158,"num_object_clones":0,"num_object_copies":15474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5158,"num_write_kb":330112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":524,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.104572+0000","last_change":"2024-02-05T06:44:00.315984+0000","last_active":"2024-02-05T06:44:25.104572+0000","last_peered":"2024-02-05T06:44:25.104572+0000","last_clean":"2024-02-05T06:44:25.104572+0000","last_became_active":"2024-02-05T06:40:27.129625+0000","last_became_peered":"2024-02-05T06:40:27.129625+0000","last_unstale":"2024-02-05T06:44:25.104572+0000","last_undegraded":"2024-02-05T06:44:25.104572+0000","last_fullsized":"2024-02-05T06:44:25.104572+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:00.315851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:44:00.315851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:25.409206+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":549,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.113883+0000","last_change":"2024-02-05T06:43:41.424681+0000","last_active":"2024-02-05T06:44:25.113883+0000","last_peered":"2024-02-05T06:44:25.113883+0000","last_clean":"2024-02-05T06:44:25.113883+0000","last_became_active":"2024-02-05T06:36:54.966016+0000","last_became_peered":"2024-02-05T06:36:54.966016+0000","last_unstale":"2024-02-05T06:44:25.113883+0000","last_undegraded":"2024-02-05T06:44:25.113883+0000","last_fullsized":"2024-02-05T06:44:25.113883+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:41.424561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:41.424561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:54.460356+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":140,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.106962+0000","last_change":"2024-02-05T06:42:32.759801+0000","last_active":"2024-02-05T06:44:25.106962+0000","last_peered":"2024-02-05T06:44:25.106962+0000","last_clean":"2024-02-05T06:44:25.106962+0000","last_became_active":"2024-02-05T06:41:43.412611+0000","last_became_peered":"2024-02-05T06:41:43.412611+0000","last_unstale":"2024-02-05T06:44:25.106962+0000","last_undegraded":"2024-02-05T06:44:25.106962+0000","last_fullsized":"2024-02-05T06:44:25.106962+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:32.759672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:32.759672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"295'15","reported_seq":632,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.109201+0000","last_change":"2024-02-05T06:43:52.150377+0000","last_active":"2024-02-05T06:44:25.109201+0000","last_peered":"2024-02-05T06:44:25.109201+0000","last_clean":"2024-02-05T06:44:25.109201+0000","last_became_active":"2024-02-05T06:40:08.584204+0000","last_became_peered":"2024-02-05T06:40:08.584204+0000","last_unstale":"2024-02-05T06:44:25.109201+0000","last_undegraded":"2024-02-05T06:44:25.109201+0000","last_fullsized":"2024-02-05T06:44:25.109201+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:43:52.150132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:52.150132+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:05.649207+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"308'5151","reported_seq":5195,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:27.099968+0000","last_change":"2024-02-05T06:43:29.703821+0000","last_active":"2024-02-05T06:44:27.099968+0000","last_peered":"2024-02-05T06:44:27.099968+0000","last_clean":"2024-02-05T06:44:27.099968+0000","last_became_active":"2024-02-05T06:43:29.703452+0000","last_became_peered":"2024-02-05T06:43:29.703452+0000","last_unstale":"2024-02-05T06:44:27.099968+0000","last_undegraded":"2024-02-05T06:44:27.099968+0000","last_fullsized":"2024-02-05T06:44:27.099968+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":5151,"log_dups_size":0,"ondisk_log_size":5151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:45.742277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337575936,"num_objects":5151,"num_object_clones":0,"num_object_copies":15453,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5151,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5151,"num_write_kb":329664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":565,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.109241+0000","last_change":"2024-02-05T06:44:04.452427+0000","last_active":"2024-02-05T06:44:25.109241+0000","last_peered":"2024-02-05T06:44:25.109241+0000","last_clean":"2024-02-05T06:44:25.109241+0000","last_became_active":"2024-02-05T06:41:46.054968+0000","last_became_peered":"2024-02-05T06:41:46.054968+0000","last_unstale":"2024-02-05T06:44:25.109241+0000","last_undegraded":"2024-02-05T06:44:25.109241+0000","last_fullsized":"2024-02-05T06:44:25.109241+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:44:04.452308+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:37.282442+0000","last_clean_scrub_stamp":"2024-02-05T06:44:04.452308+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:32.675613+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":535,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.117045+0000","last_change":"2024-02-05T06:43:40.471046+0000","last_active":"2024-02-05T06:44:25.117045+0000","last_peered":"2024-02-05T06:44:25.117045+0000","last_clean":"2024-02-05T06:44:25.117045+0000","last_became_active":"2024-02-05T06:40:27.082440+0000","last_became_peered":"2024-02-05T06:40:27.082440+0000","last_unstale":"2024-02-05T06:44:25.117045+0000","last_undegraded":"2024-02-05T06:44:25.117045+0000","last_fullsized":"2024-02-05T06:44:25.117045+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:40.470922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:43:40.470922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.918600+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"182'3","reported_seq":144,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.108654+0000","last_change":"2024-02-05T06:43:40.376967+0000","last_active":"2024-02-05T06:44:25.108654+0000","last_peered":"2024-02-05T06:44:25.108654+0000","last_clean":"2024-02-05T06:44:25.108654+0000","last_became_active":"2024-02-05T06:41:43.415806+0000","last_became_peered":"2024-02-05T06:41:43.415806+0000","last_unstale":"2024-02-05T06:44:25.108654+0000","last_undegraded":"2024-02-05T06:44:25.108654+0000","last_fullsized":"2024-02-05T06:44:25.108654+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'3","last_scrub_stamp":"2024-02-05T06:43:40.376865+0000","last_deep_scrub":"182'3","last_deep_scrub_stamp":"2024-02-05T06:39:33.373415+0000","last_clean_scrub_stamp":"2024-02-05T06:43:40.376865+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:50.126421+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001220348,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":408,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.103192+0000","last_change":"2024-02-05T06:42:42.734295+0000","last_active":"2024-02-05T06:44:25.103192+0000","last_peered":"2024-02-05T06:44:25.103192+0000","last_clean":"2024-02-05T06:44:25.103192+0000","last_became_active":"2024-02-05T06:36:54.968205+0000","last_became_peered":"2024-02-05T06:36:54.968205+0000","last_unstale":"2024-02-05T06:44:25.103192+0000","last_undegraded":"2024-02-05T06:44:25.103192+0000","last_fullsized":"2024-02-05T06:44:25.103192+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:42.734048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:42.734048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:26.837139+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"295'42","reported_seq":337,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.114105+0000","last_change":"2024-02-05T06:43:51.551040+0000","last_active":"2024-02-05T06:44:25.114105+0000","last_peered":"2024-02-05T06:44:25.114105+0000","last_clean":"2024-02-05T06:44:25.114105+0000","last_became_active":"2024-02-05T06:40:08.604483+0000","last_became_peered":"2024-02-05T06:40:08.604483+0000","last_unstale":"2024-02-05T06:44:25.114105+0000","last_undegraded":"2024-02-05T06:44:25.114105+0000","last_fullsized":"2024-02-05T06:44:25.114105+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'42","last_scrub_stamp":"2024-02-05T06:43:51.550901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:51.550901+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:01.091650+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":528,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.104782+0000","last_change":"2024-02-05T06:42:42.929540+0000","last_active":"2024-02-05T06:44:25.104782+0000","last_peered":"2024-02-05T06:44:25.104782+0000","last_clean":"2024-02-05T06:44:25.104782+0000","last_became_active":"2024-02-05T06:36:54.973387+0000","last_became_peered":"2024-02-05T06:36:54.973387+0000","last_unstale":"2024-02-05T06:44:25.104782+0000","last_undegraded":"2024-02-05T06:44:25.104782+0000","last_fullsized":"2024-02-05T06:44:25.104782+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:42.929409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:42:42.929409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"308'5100","reported_seq":5144,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:26.268288+0000","last_change":"2024-02-05T06:43:29.702708+0000","last_active":"2024-02-05T06:44:26.268288+0000","last_peered":"2024-02-05T06:44:26.268288+0000","last_clean":"2024-02-05T06:44:26.268288+0000","last_became_active":"2024-02-05T06:43:29.702401+0000","last_became_peered":"2024-02-05T06:43:29.702401+0000","last_unstale":"2024-02-05T06:44:26.268288+0000","last_undegraded":"2024-02-05T06:44:26.268288+0000","last_fullsized":"2024-02-05T06:44:26.268288+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":5100,"log_dups_size":0,"ondisk_log_size":5100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:55.338031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334233600,"num_objects":5100,"num_object_clones":0,"num_object_copies":15300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5100,"num_write_kb":326400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":645,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.110950+0000","last_change":"2024-02-05T06:42:53.491927+0000","last_active":"2024-02-05T06:44:25.110950+0000","last_peered":"2024-02-05T06:44:25.110950+0000","last_clean":"2024-02-05T06:44:25.110950+0000","last_became_active":"2024-02-05T06:40:30.082467+0000","last_became_peered":"2024-02-05T06:40:30.082467+0000","last_unstale":"2024-02-05T06:44:25.110950+0000","last_undegraded":"2024-02-05T06:44:25.110950+0000","last_fullsized":"2024-02-05T06:44:25.110950+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:42:53.491791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:42:53.491791+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":224,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.112876+0000","last_change":"2024-02-05T06:42:54.478248+0000","last_active":"2024-02-05T06:44:25.112876+0000","last_peered":"2024-02-05T06:44:25.112876+0000","last_clean":"2024-02-05T06:44:25.112876+0000","last_became_active":"2024-02-05T06:40:27.077966+0000","last_became_peered":"2024-02-05T06:40:27.077966+0000","last_unstale":"2024-02-05T06:44:25.112876+0000","last_undegraded":"2024-02-05T06:44:25.112876+0000","last_fullsized":"2024-02-05T06:44:25.112876+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:54.477914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:42:54.477914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:27.360965+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":531,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.113746+0000","last_change":"2024-02-05T06:43:46.552644+0000","last_active":"2024-02-05T06:44:25.113746+0000","last_peered":"2024-02-05T06:44:25.113746+0000","last_clean":"2024-02-05T06:44:25.113746+0000","last_became_active":"2024-02-05T06:33:29.551549+0000","last_became_peered":"2024-02-05T06:33:29.551549+0000","last_unstale":"2024-02-05T06:44:25.113746+0000","last_undegraded":"2024-02-05T06:44:25.113746+0000","last_fullsized":"2024-02-05T06:44:25.113746+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:46.552530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:43:46.552530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:09.476346+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":546,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.111205+0000","last_change":"2024-02-05T06:42:59.532291+0000","last_active":"2024-02-05T06:44:25.111205+0000","last_peered":"2024-02-05T06:44:25.111205+0000","last_clean":"2024-02-05T06:44:25.111205+0000","last_became_active":"2024-02-05T06:33:28.542977+0000","last_became_peered":"2024-02-05T06:33:28.542977+0000","last_unstale":"2024-02-05T06:44:25.111205+0000","last_undegraded":"2024-02-05T06:44:25.111205+0000","last_fullsized":"2024-02-05T06:44:25.111205+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:59.532192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:42:59.532192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:27.850183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091640799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"295'17","reported_seq":622,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.110599+0000","last_change":"2024-02-05T06:43:01.610636+0000","last_active":"2024-02-05T06:44:25.110599+0000","last_peered":"2024-02-05T06:44:25.110599+0000","last_clean":"2024-02-05T06:44:25.110599+0000","last_became_active":"2024-02-05T06:41:43.420458+0000","last_became_peered":"2024-02-05T06:41:43.420458+0000","last_unstale":"2024-02-05T06:44:25.110599+0000","last_undegraded":"2024-02-05T06:44:25.110599+0000","last_fullsized":"2024-02-05T06:44:25.110599+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-02-05T06:43:01.610491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:01.610491+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:30.568814+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":561,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.111279+0000","last_change":"2024-02-05T06:42:52.383735+0000","last_active":"2024-02-05T06:44:25.111279+0000","last_peered":"2024-02-05T06:44:25.111279+0000","last_clean":"2024-02-05T06:44:25.111279+0000","last_became_active":"2024-02-05T06:41:43.412387+0000","last_became_peered":"2024-02-05T06:41:43.412387+0000","last_unstale":"2024-02-05T06:44:25.111279+0000","last_undegraded":"2024-02-05T06:44:25.111279+0000","last_fullsized":"2024-02-05T06:44:25.111279+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:52.383618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:33.914422+0000","last_clean_scrub_stamp":"2024-02-05T06:42:52.383618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:31.493865+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"308'5233","reported_seq":5277,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:27.186338+0000","last_change":"2024-02-05T06:43:29.701823+0000","last_active":"2024-02-05T06:44:27.186338+0000","last_peered":"2024-02-05T06:44:27.186338+0000","last_clean":"2024-02-05T06:44:27.186338+0000","last_became_active":"2024-02-05T06:43:29.701487+0000","last_became_peered":"2024-02-05T06:43:29.701487+0000","last_unstale":"2024-02-05T06:44:27.186338+0000","last_undegraded":"2024-02-05T06:44:27.186338+0000","last_fullsized":"2024-02-05T06:44:27.186338+0000","mapping_epoch":291,"log_start":"296'800","ondisk_log_start":"296'800","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4433,"log_dups_size":0,"ondisk_log_size":4433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:50.692005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342949888,"num_objects":5233,"num_object_clones":0,"num_object_copies":15699,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5233,"num_write_kb":334912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":140,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.108380+0000","last_change":"2024-02-05T06:42:37.728366+0000","last_active":"2024-02-05T06:44:25.108380+0000","last_peered":"2024-02-05T06:44:25.108380+0000","last_clean":"2024-02-05T06:44:25.108380+0000","last_became_active":"2024-02-05T06:41:43.408773+0000","last_became_peered":"2024-02-05T06:41:43.408773+0000","last_unstale":"2024-02-05T06:44:25.108380+0000","last_undegraded":"2024-02-05T06:44:25.108380+0000","last_fullsized":"2024-02-05T06:44:25.108380+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:37.728252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:42:37.728252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":646,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.110756+0000","last_change":"2024-02-05T06:42:52.491767+0000","last_active":"2024-02-05T06:44:25.110756+0000","last_peered":"2024-02-05T06:44:25.110756+0000","last_clean":"2024-02-05T06:44:25.110756+0000","last_became_active":"2024-02-05T06:40:30.081178+0000","last_became_peered":"2024-02-05T06:40:30.081178+0000","last_unstale":"2024-02-05T06:44:25.110756+0000","last_undegraded":"2024-02-05T06:44:25.110756+0000","last_fullsized":"2024-02-05T06:44:25.110756+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:42:52.491322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:42:52.491322+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":535,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.116487+0000","last_change":"2024-02-05T06:42:50.411721+0000","last_active":"2024-02-05T06:44:25.116487+0000","last_peered":"2024-02-05T06:44:25.116487+0000","last_clean":"2024-02-05T06:44:25.116487+0000","last_became_active":"2024-02-05T06:41:43.411335+0000","last_became_peered":"2024-02-05T06:41:43.411335+0000","last_unstale":"2024-02-05T06:44:25.116487+0000","last_undegraded":"2024-02-05T06:44:25.116487+0000","last_fullsized":"2024-02-05T06:44:25.116487+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:50.411634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:50.411634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:27.898141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"295'32","reported_seq":804,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.108694+0000","last_change":"2024-02-05T06:43:58.304760+0000","last_active":"2024-02-05T06:44:25.108694+0000","last_peered":"2024-02-05T06:44:25.108694+0000","last_clean":"2024-02-05T06:44:25.108694+0000","last_became_active":"2024-02-05T06:40:27.061111+0000","last_became_peered":"2024-02-05T06:40:27.061111+0000","last_unstale":"2024-02-05T06:44:25.108694+0000","last_undegraded":"2024-02-05T06:44:25.108694+0000","last_fullsized":"2024-02-05T06:44:25.108694+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'32","last_scrub_stamp":"2024-02-05T06:43:58.304628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:58.304628+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:17.673610+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":238,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.111597+0000","last_change":"2024-02-05T06:43:43.529553+0000","last_active":"2024-02-05T06:44:25.111597+0000","last_peered":"2024-02-05T06:44:25.111597+0000","last_clean":"2024-02-05T06:44:25.111597+0000","last_became_active":"2024-02-05T06:41:43.418198+0000","last_became_peered":"2024-02-05T06:41:43.418198+0000","last_unstale":"2024-02-05T06:44:25.111597+0000","last_undegraded":"2024-02-05T06:44:25.111597+0000","last_fullsized":"2024-02-05T06:44:25.111597+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:43.529437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:43.529437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:00.290531+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"308'5177","reported_seq":5219,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:27.146459+0000","last_change":"2024-02-05T06:43:29.947291+0000","last_active":"2024-02-05T06:44:27.146459+0000","last_peered":"2024-02-05T06:44:27.146459+0000","last_clean":"2024-02-05T06:44:27.146459+0000","last_became_active":"2024-02-05T06:43:29.946961+0000","last_became_peered":"2024-02-05T06:43:29.946961+0000","last_unstale":"2024-02-05T06:44:27.146459+0000","last_undegraded":"2024-02-05T06:44:27.146459+0000","last_fullsized":"2024-02-05T06:44:27.146459+0000","mapping_epoch":291,"log_start":"0'0","ondisk_log_start":"0'0","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":5177,"log_dups_size":0,"ondisk_log_size":5177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:38.100021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339279872,"num_objects":5177,"num_object_clones":0,"num_object_copies":15531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5177,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5177,"num_write_kb":331328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":686,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.107937+0000","last_change":"2024-02-05T06:42:38.598783+0000","last_active":"2024-02-05T06:44:25.107937+0000","last_peered":"2024-02-05T06:44:25.107937+0000","last_clean":"2024-02-05T06:44:25.107937+0000","last_became_active":"2024-02-05T06:37:37.137754+0000","last_became_peered":"2024-02-05T06:37:37.137754+0000","last_unstale":"2024-02-05T06:44:25.107937+0000","last_undegraded":"2024-02-05T06:44:25.107937+0000","last_fullsized":"2024-02-05T06:44:25.107937+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:42:38.598563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:42:38.598563+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":539,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.114208+0000","last_change":"2024-02-05T06:44:03.559943+0000","last_active":"2024-02-05T06:44:25.114208+0000","last_peered":"2024-02-05T06:44:25.114208+0000","last_clean":"2024-02-05T06:44:25.114208+0000","last_became_active":"2024-02-05T06:41:43.410910+0000","last_became_peered":"2024-02-05T06:41:43.410910+0000","last_unstale":"2024-02-05T06:44:25.114208+0000","last_undegraded":"2024-02-05T06:44:25.114208+0000","last_fullsized":"2024-02-05T06:44:25.114208+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:03.559834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:44:03.559834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:09.328386+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":536,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.103656+0000","last_change":"2024-02-05T06:42:49.943397+0000","last_active":"2024-02-05T06:44:25.103656+0000","last_peered":"2024-02-05T06:44:25.103656+0000","last_clean":"2024-02-05T06:44:25.103656+0000","last_became_active":"2024-02-05T06:36:54.972264+0000","last_became_peered":"2024-02-05T06:36:54.972264+0000","last_unstale":"2024-02-05T06:44:25.103656+0000","last_undegraded":"2024-02-05T06:44:25.103656+0000","last_fullsized":"2024-02-05T06:44:25.103656+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:49.942846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:49.942846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":471,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.432284+0000","last_change":"2024-02-05T06:43:33.504679+0000","last_active":"2024-02-05T06:44:25.432284+0000","last_peered":"2024-02-05T06:44:25.432284+0000","last_clean":"2024-02-05T06:44:25.432284+0000","last_became_active":"2024-02-05T06:40:27.071509+0000","last_became_peered":"2024-02-05T06:40:27.071509+0000","last_unstale":"2024-02-05T06:44:25.432284+0000","last_undegraded":"2024-02-05T06:44:25.432284+0000","last_fullsized":"2024-02-05T06:44:25.432284+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T06:43:33.504581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:43:33.504581+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:37.402285+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00074151600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":245,"num_read_kb":240,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"295'24","reported_seq":556,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.104205+0000","last_change":"2024-02-05T06:42:50.901662+0000","last_active":"2024-02-05T06:44:25.104205+0000","last_peered":"2024-02-05T06:44:25.104205+0000","last_clean":"2024-02-05T06:44:25.104205+0000","last_became_active":"2024-02-05T06:33:31.564604+0000","last_became_peered":"2024-02-05T06:33:31.564604+0000","last_unstale":"2024-02-05T06:44:25.104205+0000","last_undegraded":"2024-02-05T06:44:25.104205+0000","last_fullsized":"2024-02-05T06:44:25.104205+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:42:50.901560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:42:50.901560+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":140,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.108463+0000","last_change":"2024-02-05T06:43:49.441549+0000","last_active":"2024-02-05T06:44:25.108463+0000","last_peered":"2024-02-05T06:44:25.108463+0000","last_clean":"2024-02-05T06:44:25.108463+0000","last_became_active":"2024-02-05T06:41:43.417751+0000","last_became_peered":"2024-02-05T06:41:43.417751+0000","last_unstale":"2024-02-05T06:44:25.108463+0000","last_undegraded":"2024-02-05T06:44:25.108463+0000","last_fullsized":"2024-02-05T06:44:25.108463+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:49.441328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:49.441328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:50.412164+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"308'5092","reported_seq":5136,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:26.287951+0000","last_change":"2024-02-05T06:43:29.703026+0000","last_active":"2024-02-05T06:44:26.287951+0000","last_peered":"2024-02-05T06:44:26.287951+0000","last_clean":"2024-02-05T06:44:26.287951+0000","last_became_active":"2024-02-05T06:43:29.702549+0000","last_became_peered":"2024-02-05T06:43:29.702549+0000","last_unstale":"2024-02-05T06:44:26.287951+0000","last_undegraded":"2024-02-05T06:44:26.287951+0000","last_fullsized":"2024-02-05T06:44:26.287951+0000","mapping_epoch":291,"log_start":"296'400","ondisk_log_start":"296'400","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4692,"log_dups_size":0,"ondisk_log_size":4692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:47.712668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333709312,"num_objects":5092,"num_object_clones":0,"num_object_copies":15276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5092,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5092,"num_write_kb":325888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":699,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.109074+0000","last_change":"2024-02-05T06:44:04.313089+0000","last_active":"2024-02-05T06:44:25.109074+0000","last_peered":"2024-02-05T06:44:25.109074+0000","last_clean":"2024-02-05T06:44:25.109074+0000","last_became_active":"2024-02-05T06:37:37.140011+0000","last_became_peered":"2024-02-05T06:37:37.140011+0000","last_unstale":"2024-02-05T06:44:25.109074+0000","last_undegraded":"2024-02-05T06:44:25.109074+0000","last_fullsized":"2024-02-05T06:44:25.109074+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:44:04.312963+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:38.778568+0000","last_clean_scrub_stamp":"2024-02-05T06:44:04.312963+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:32.876919+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":519,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.108023+0000","last_change":"2024-02-05T06:42:41.634187+0000","last_active":"2024-02-05T06:44:25.108023+0000","last_peered":"2024-02-05T06:44:25.108023+0000","last_clean":"2024-02-05T06:44:25.108023+0000","last_became_active":"2024-02-05T06:33:35.664858+0000","last_became_peered":"2024-02-05T06:33:35.664858+0000","last_unstale":"2024-02-05T06:44:25.108023+0000","last_undegraded":"2024-02-05T06:44:25.108023+0000","last_fullsized":"2024-02-05T06:44:25.108023+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:41.634048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:42:41.634048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"66'1","reported_seq":565,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.114487+0000","last_change":"2024-02-05T06:43:50.530854+0000","last_active":"2024-02-05T06:44:25.114487+0000","last_peered":"2024-02-05T06:44:25.114487+0000","last_clean":"2024-02-05T06:44:25.114487+0000","last_became_active":"2024-02-05T06:41:43.411404+0000","last_became_peered":"2024-02-05T06:41:43.411404+0000","last_unstale":"2024-02-05T06:44:25.114487+0000","last_undegraded":"2024-02-05T06:44:25.114487+0000","last_fullsized":"2024-02-05T06:44:25.114487+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:43:50.530734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:43:50.530734+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:06.449055+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"295'29","reported_seq":667,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.109424+0000","last_change":"2024-02-05T06:43:02.045504+0000","last_active":"2024-02-05T06:44:25.109424+0000","last_peered":"2024-02-05T06:44:25.109424+0000","last_clean":"2024-02-05T06:44:25.109424+0000","last_became_active":"2024-02-05T06:41:43.409664+0000","last_became_peered":"2024-02-05T06:41:43.409664+0000","last_unstale":"2024-02-05T06:44:25.109424+0000","last_undegraded":"2024-02-05T06:44:25.109424+0000","last_fullsized":"2024-02-05T06:44:25.109424+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-02-05T06:43:02.045346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:02.045346+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:27.803252+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":534,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.109750+0000","last_change":"2024-02-05T06:42:56.157261+0000","last_active":"2024-02-05T06:44:25.109750+0000","last_peered":"2024-02-05T06:44:25.109750+0000","last_clean":"2024-02-05T06:44:25.109750+0000","last_became_active":"2024-02-05T06:41:43.411550+0000","last_became_peered":"2024-02-05T06:41:43.411550+0000","last_unstale":"2024-02-05T06:44:25.109750+0000","last_undegraded":"2024-02-05T06:44:25.109750+0000","last_fullsized":"2024-02-05T06:44:25.109750+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:56.157130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:42:56.157130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"308'5106","reported_seq":5150,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:26.280945+0000","last_change":"2024-02-05T06:43:29.700909+0000","last_active":"2024-02-05T06:44:26.280945+0000","last_peered":"2024-02-05T06:44:26.280945+0000","last_clean":"2024-02-05T06:44:26.280945+0000","last_became_active":"2024-02-05T06:43:29.700331+0000","last_became_peered":"2024-02-05T06:43:29.700331+0000","last_unstale":"2024-02-05T06:44:26.280945+0000","last_undegraded":"2024-02-05T06:44:26.280945+0000","last_fullsized":"2024-02-05T06:44:26.280945+0000","mapping_epoch":291,"log_start":"296'100","ondisk_log_start":"296'100","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":5006,"log_dups_size":0,"ondisk_log_size":5006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:30.002331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334626816,"num_objects":5106,"num_object_clones":0,"num_object_copies":15318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5106,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5106,"num_write_kb":326784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":718,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.109290+0000","last_change":"2024-02-05T06:42:49.138308+0000","last_active":"2024-02-05T06:44:25.109290+0000","last_peered":"2024-02-05T06:44:25.109290+0000","last_clean":"2024-02-05T06:44:25.109290+0000","last_became_active":"2024-02-05T06:40:30.085438+0000","last_became_peered":"2024-02-05T06:40:30.085438+0000","last_unstale":"2024-02-05T06:44:25.109290+0000","last_undegraded":"2024-02-05T06:44:25.109290+0000","last_fullsized":"2024-02-05T06:44:25.109290+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:42:49.138204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:42:49.138204+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:26.443533+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":512,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.103588+0000","last_change":"2024-02-05T06:43:42.125866+0000","last_active":"2024-02-05T06:44:25.103588+0000","last_peered":"2024-02-05T06:44:25.103588+0000","last_clean":"2024-02-05T06:44:25.103588+0000","last_became_active":"2024-02-05T06:33:35.600485+0000","last_became_peered":"2024-02-05T06:33:35.600485+0000","last_unstale":"2024-02-05T06:44:25.103588+0000","last_undegraded":"2024-02-05T06:44:25.103588+0000","last_fullsized":"2024-02-05T06:44:25.103588+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:42.125756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:43:42.125756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:04.704617+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":140,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.107378+0000","last_change":"2024-02-05T06:43:35.458031+0000","last_active":"2024-02-05T06:44:25.107378+0000","last_peered":"2024-02-05T06:44:25.107378+0000","last_clean":"2024-02-05T06:44:25.107378+0000","last_became_active":"2024-02-05T06:41:43.411732+0000","last_became_peered":"2024-02-05T06:41:43.411732+0000","last_unstale":"2024-02-05T06:44:25.107378+0000","last_undegraded":"2024-02-05T06:44:25.107378+0000","last_fullsized":"2024-02-05T06:44:25.107378+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:35.457940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:28.353956+0000","last_clean_scrub_stamp":"2024-02-05T06:43:35.457940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:04.715232+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"295'20","reported_seq":629,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.112341+0000","last_change":"2024-02-05T06:43:35.467498+0000","last_active":"2024-02-05T06:44:25.112341+0000","last_peered":"2024-02-05T06:44:25.112341+0000","last_clean":"2024-02-05T06:44:25.112341+0000","last_became_active":"2024-02-05T06:41:43.421030+0000","last_became_peered":"2024-02-05T06:41:43.421030+0000","last_unstale":"2024-02-05T06:44:25.112341+0000","last_undegraded":"2024-02-05T06:44:25.112341+0000","last_fullsized":"2024-02-05T06:44:25.112341+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-02-05T06:43:35.467372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:35.467372+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:00.825350+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"249'10","reported_seq":676,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.109044+0000","last_change":"2024-02-05T06:42:48.013461+0000","last_active":"2024-02-05T06:44:25.109044+0000","last_peered":"2024-02-05T06:44:25.109044+0000","last_clean":"2024-02-05T06:44:25.109044+0000","last_became_active":"2024-02-05T06:41:43.423638+0000","last_became_peered":"2024-02-05T06:41:43.423638+0000","last_unstale":"2024-02-05T06:44:25.109044+0000","last_undegraded":"2024-02-05T06:44:25.109044+0000","last_fullsized":"2024-02-05T06:44:25.109044+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'10","last_scrub_stamp":"2024-02-05T06:42:48.013329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:42:48.013329+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"308'5134","reported_seq":5178,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:27.111606+0000","last_change":"2024-02-05T06:43:29.702358+0000","last_active":"2024-02-05T06:44:27.111606+0000","last_peered":"2024-02-05T06:44:27.111606+0000","last_clean":"2024-02-05T06:44:27.111606+0000","last_became_active":"2024-02-05T06:43:29.701716+0000","last_became_peered":"2024-02-05T06:43:29.701716+0000","last_unstale":"2024-02-05T06:44:27.111606+0000","last_undegraded":"2024-02-05T06:44:27.111606+0000","last_fullsized":"2024-02-05T06:44:27.111606+0000","mapping_epoch":291,"log_start":"296'700","ondisk_log_start":"296'700","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4434,"log_dups_size":0,"ondisk_log_size":4434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:49.602994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336461824,"num_objects":5134,"num_object_clones":0,"num_object_copies":15402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5134,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5134,"num_write_kb":328576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":729,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.116977+0000","last_change":"2024-02-05T06:42:49.448963+0000","last_active":"2024-02-05T06:44:25.116977+0000","last_peered":"2024-02-05T06:44:25.116977+0000","last_clean":"2024-02-05T06:44:25.116977+0000","last_became_active":"2024-02-05T06:41:46.053837+0000","last_became_peered":"2024-02-05T06:41:46.053837+0000","last_unstale":"2024-02-05T06:44:25.116977+0000","last_undegraded":"2024-02-05T06:44:25.116977+0000","last_fullsized":"2024-02-05T06:44:25.116977+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:42:49.440873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:42:49.440873+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:27.898083+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":224,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.113321+0000","last_change":"2024-02-05T06:42:59.397727+0000","last_active":"2024-02-05T06:44:25.113321+0000","last_peered":"2024-02-05T06:44:25.113321+0000","last_clean":"2024-02-05T06:44:25.113321+0000","last_became_active":"2024-02-05T06:40:27.099168+0000","last_became_peered":"2024-02-05T06:40:27.099168+0000","last_unstale":"2024-02-05T06:44:25.113321+0000","last_undegraded":"2024-02-05T06:44:25.113321+0000","last_fullsized":"2024-02-05T06:44:25.113321+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:59.397620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:42:59.397620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:26.402944+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"95'2","reported_seq":546,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.104302+0000","last_change":"2024-02-05T06:43:34.938744+0000","last_active":"2024-02-05T06:44:25.104302+0000","last_peered":"2024-02-05T06:44:25.104302+0000","last_clean":"2024-02-05T06:44:25.104302+0000","last_became_active":"2024-02-05T06:36:54.975093+0000","last_became_peered":"2024-02-05T06:36:54.975093+0000","last_unstale":"2024-02-05T06:44:25.104302+0000","last_undegraded":"2024-02-05T06:44:25.104302+0000","last_fullsized":"2024-02-05T06:44:25.104302+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-05T06:43:34.938667+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-05T06:35:25.616708+0000","last_clean_scrub_stamp":"2024-02-05T06:43:34.938667+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:02.866138+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"295'28","reported_seq":489,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.102100+0000","last_change":"2024-02-05T06:42:59.726887+0000","last_active":"2024-02-05T06:44:25.102100+0000","last_peered":"2024-02-05T06:44:25.102100+0000","last_clean":"2024-02-05T06:44:25.102100+0000","last_became_active":"2024-02-05T06:41:43.411540+0000","last_became_peered":"2024-02-05T06:41:43.411540+0000","last_unstale":"2024-02-05T06:44:25.102100+0000","last_undegraded":"2024-02-05T06:44:25.102100+0000","last_fullsized":"2024-02-05T06:44:25.102100+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'14","last_scrub_stamp":"2024-02-05T06:42:59.726776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:42:59.726776+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:26.837192+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"119'10","reported_seq":677,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.115369+0000","last_change":"2024-02-05T06:43:01.414791+0000","last_active":"2024-02-05T06:44:25.115369+0000","last_peered":"2024-02-05T06:44:25.115369+0000","last_clean":"2024-02-05T06:44:25.115369+0000","last_became_active":"2024-02-05T06:36:54.954819+0000","last_became_peered":"2024-02-05T06:36:54.954819+0000","last_unstale":"2024-02-05T06:44:25.115369+0000","last_undegraded":"2024-02-05T06:44:25.115369+0000","last_fullsized":"2024-02-05T06:44:25.115369+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-02-05T06:43:01.414710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:01.414710+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:27.898115+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"308'5085","reported_seq":5129,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:26.313637+0000","last_change":"2024-02-05T06:43:29.704287+0000","last_active":"2024-02-05T06:44:26.313637+0000","last_peered":"2024-02-05T06:44:26.313637+0000","last_clean":"2024-02-05T06:44:26.313637+0000","last_became_active":"2024-02-05T06:43:29.703700+0000","last_became_peered":"2024-02-05T06:43:29.703700+0000","last_unstale":"2024-02-05T06:44:26.313637+0000","last_undegraded":"2024-02-05T06:44:26.313637+0000","last_fullsized":"2024-02-05T06:44:26.313637+0000","mapping_epoch":291,"log_start":"296'400","ondisk_log_start":"296'400","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4685,"log_dups_size":0,"ondisk_log_size":4685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:36.320886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333250560,"num_objects":5085,"num_object_clones":0,"num_object_copies":15255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5085,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5085,"num_write_kb":325440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":678,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.102909+0000","last_change":"2024-02-05T06:41:28.466905+0000","last_active":"2024-02-05T06:44:25.102909+0000","last_peered":"2024-02-05T06:44:25.102909+0000","last_clean":"2024-02-05T06:44:25.102909+0000","last_became_active":"2024-02-05T06:37:37.135073+0000","last_became_peered":"2024-02-05T06:37:37.135073+0000","last_unstale":"2024-02-05T06:44:25.102909+0000","last_undegraded":"2024-02-05T06:44:25.102909+0000","last_fullsized":"2024-02-05T06:44:25.102909+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:41:28.466576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:41:28.466576+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:26.836955+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":408,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.102698+0000","last_change":"2024-02-05T06:42:45.696347+0000","last_active":"2024-02-05T06:44:25.102698+0000","last_peered":"2024-02-05T06:44:25.102698+0000","last_clean":"2024-02-05T06:44:25.102698+0000","last_became_active":"2024-02-05T06:36:54.958644+0000","last_became_peered":"2024-02-05T06:36:54.958644+0000","last_unstale":"2024-02-05T06:44:25.102698+0000","last_undegraded":"2024-02-05T06:44:25.102698+0000","last_fullsized":"2024-02-05T06:44:25.102698+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:45.696236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:24.090670+0000","last_clean_scrub_stamp":"2024-02-05T06:42:45.696236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:26.837246+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":541,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.119144+0000","last_change":"2024-02-05T06:42:56.396835+0000","last_active":"2024-02-05T06:44:25.119144+0000","last_peered":"2024-02-05T06:44:25.119144+0000","last_clean":"2024-02-05T06:44:25.119144+0000","last_became_active":"2024-02-05T06:40:27.093739+0000","last_became_peered":"2024-02-05T06:40:27.093739+0000","last_unstale":"2024-02-05T06:44:25.119144+0000","last_undegraded":"2024-02-05T06:44:25.119144+0000","last_fullsized":"2024-02-05T06:44:25.119144+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:56.396575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:38:28.150695+0000","last_clean_scrub_stamp":"2024-02-05T06:42:56.396575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"295'30","reported_seq":617,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.115280+0000","last_change":"2024-02-05T06:43:46.582784+0000","last_active":"2024-02-05T06:44:25.115280+0000","last_peered":"2024-02-05T06:44:25.115280+0000","last_clean":"2024-02-05T06:44:25.115280+0000","last_became_active":"2024-02-05T06:40:27.277747+0000","last_became_peered":"2024-02-05T06:40:27.277747+0000","last_unstale":"2024-02-05T06:44:25.115280+0000","last_undegraded":"2024-02-05T06:44:25.115280+0000","last_fullsized":"2024-02-05T06:44:25.115280+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'30","last_scrub_stamp":"2024-02-05T06:43:46.582265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:46.582265+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:14.477223+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":428,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.102178+0000","last_change":"2024-02-05T06:42:52.703205+0000","last_active":"2024-02-05T06:44:25.102178+0000","last_peered":"2024-02-05T06:44:25.102178+0000","last_clean":"2024-02-05T06:44:25.102178+0000","last_became_active":"2024-02-05T06:40:27.073889+0000","last_became_peered":"2024-02-05T06:40:27.073889+0000","last_unstale":"2024-02-05T06:44:25.102178+0000","last_undegraded":"2024-02-05T06:44:25.102178+0000","last_fullsized":"2024-02-05T06:44:25.102178+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:52.703089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:42:52.703089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:26.837295+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"308'5086","reported_seq":5130,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:27.162997+0000","last_change":"2024-02-05T06:43:29.843237+0000","last_active":"2024-02-05T06:44:27.162997+0000","last_peered":"2024-02-05T06:44:27.162997+0000","last_clean":"2024-02-05T06:44:27.162997+0000","last_became_active":"2024-02-05T06:43:29.842722+0000","last_became_peered":"2024-02-05T06:43:29.842722+0000","last_unstale":"2024-02-05T06:44:27.162997+0000","last_undegraded":"2024-02-05T06:44:27.162997+0000","last_fullsized":"2024-02-05T06:44:27.162997+0000","mapping_epoch":291,"log_start":"296'600","ondisk_log_start":"296'600","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4486,"log_dups_size":0,"ondisk_log_size":4486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:33.230195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333316096,"num_objects":5086,"num_object_clones":0,"num_object_copies":15258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5086,"num_write_kb":325504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":559,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.108742+0000","last_change":"2024-02-05T06:43:59.471960+0000","last_active":"2024-02-05T06:44:25.108742+0000","last_peered":"2024-02-05T06:44:25.108742+0000","last_clean":"2024-02-05T06:44:25.108742+0000","last_became_active":"2024-02-05T06:41:46.053993+0000","last_became_peered":"2024-02-05T06:41:46.053993+0000","last_unstale":"2024-02-05T06:44:25.108742+0000","last_undegraded":"2024-02-05T06:44:25.108742+0000","last_fullsized":"2024-02-05T06:44:25.108742+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:43:59.471749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:43:59.471749+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:04.010257+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":148,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.107297+0000","last_change":"2024-02-05T06:44:00.522165+0000","last_active":"2024-02-05T06:44:25.107297+0000","last_peered":"2024-02-05T06:44:25.107297+0000","last_clean":"2024-02-05T06:44:25.107297+0000","last_became_active":"2024-02-05T06:41:43.418066+0000","last_became_peered":"2024-02-05T06:41:43.418066+0000","last_unstale":"2024-02-05T06:44:25.107297+0000","last_undegraded":"2024-02-05T06:44:25.107297+0000","last_fullsized":"2024-02-05T06:44:25.107297+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:00.522003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:44:00.522003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:11.908885+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"249'12","reported_seq":524,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.103660+0000","last_change":"2024-02-05T06:42:56.729086+0000","last_active":"2024-02-05T06:44:25.103660+0000","last_peered":"2024-02-05T06:44:25.103660+0000","last_clean":"2024-02-05T06:44:25.103660+0000","last_became_active":"2024-02-05T06:41:43.411973+0000","last_became_peered":"2024-02-05T06:41:43.411973+0000","last_unstale":"2024-02-05T06:44:25.103660+0000","last_undegraded":"2024-02-05T06:44:25.103660+0000","last_fullsized":"2024-02-05T06:44:25.103660+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'12","last_scrub_stamp":"2024-02-05T06:42:56.728967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:42:56.728967+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:26.837221+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":535,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.115186+0000","last_change":"2024-02-05T06:43:43.544887+0000","last_active":"2024-02-05T06:44:25.115186+0000","last_peered":"2024-02-05T06:44:25.115186+0000","last_clean":"2024-02-05T06:44:25.115186+0000","last_became_active":"2024-02-05T06:40:27.213439+0000","last_became_peered":"2024-02-05T06:40:27.213439+0000","last_unstale":"2024-02-05T06:44:25.115186+0000","last_undegraded":"2024-02-05T06:44:25.115186+0000","last_fullsized":"2024-02-05T06:44:25.115186+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:43.544765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:43:43.544765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:50.636394+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"295'34","reported_seq":330,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.104768+0000","last_change":"2024-02-05T06:42:59.970087+0000","last_active":"2024-02-05T06:44:25.104768+0000","last_peered":"2024-02-05T06:44:25.104768+0000","last_clean":"2024-02-05T06:44:25.104768+0000","last_became_active":"2024-02-05T06:39:13.257978+0000","last_became_peered":"2024-02-05T06:39:13.257978+0000","last_unstale":"2024-02-05T06:44:25.104768+0000","last_undegraded":"2024-02-05T06:44:25.104768+0000","last_fullsized":"2024-02-05T06:44:25.104768+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'22","last_scrub_stamp":"2024-02-05T06:42:59.969994+0000","last_deep_scrub":"281'22","last_deep_scrub_stamp":"2024-02-05T06:42:59.969994+0000","last_clean_scrub_stamp":"2024-02-05T06:42:59.969994+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.3","version":"308'5047","reported_seq":5091,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:26.304062+0000","last_change":"2024-02-05T06:43:29.702128+0000","last_active":"2024-02-05T06:44:26.304062+0000","last_peered":"2024-02-05T06:44:26.304062+0000","last_clean":"2024-02-05T06:44:26.304062+0000","last_became_active":"2024-02-05T06:43:29.701677+0000","last_became_peered":"2024-02-05T06:43:29.701677+0000","last_unstale":"2024-02-05T06:44:26.304062+0000","last_undegraded":"2024-02-05T06:44:26.304062+0000","last_fullsized":"2024-02-05T06:44:26.304062+0000","mapping_epoch":291,"log_start":"296'400","ondisk_log_start":"296'400","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4647,"log_dups_size":0,"ondisk_log_size":4647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:32.217143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330760192,"num_objects":5047,"num_object_clones":0,"num_object_copies":15141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5047,"num_write_kb":323008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":528,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.117131+0000","last_change":"2024-02-05T06:43:42.559000+0000","last_active":"2024-02-05T06:44:25.117131+0000","last_peered":"2024-02-05T06:44:25.117131+0000","last_clean":"2024-02-05T06:44:25.117131+0000","last_became_active":"2024-02-05T06:41:43.410600+0000","last_became_peered":"2024-02-05T06:41:43.410600+0000","last_unstale":"2024-02-05T06:44:25.117131+0000","last_undegraded":"2024-02-05T06:44:25.117131+0000","last_fullsized":"2024-02-05T06:44:25.117131+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:42.558880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:20.042534+0000","last_clean_scrub_stamp":"2024-02-05T06:43:42.558880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:54.376149+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":140,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.108301+0000","last_change":"2024-02-05T06:42:55.610389+0000","last_active":"2024-02-05T06:44:25.108301+0000","last_peered":"2024-02-05T06:44:25.108301+0000","last_clean":"2024-02-05T06:44:25.108301+0000","last_became_active":"2024-02-05T06:41:43.417343+0000","last_became_peered":"2024-02-05T06:41:43.417343+0000","last_unstale":"2024-02-05T06:44:25.108301+0000","last_undegraded":"2024-02-05T06:44:25.108301+0000","last_fullsized":"2024-02-05T06:44:25.108301+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:55.610276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:42:55.610276+0000","last_clean_scrub_stamp":"2024-02-05T06:42:55.610276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":250,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.107716+0000","last_change":"2024-02-05T06:44:01.550712+0000","last_active":"2024-02-05T06:44:25.107716+0000","last_peered":"2024-02-05T06:44:25.107716+0000","last_clean":"2024-02-05T06:44:25.107716+0000","last_became_active":"2024-02-05T06:40:27.086808+0000","last_became_peered":"2024-02-05T06:40:27.086808+0000","last_unstale":"2024-02-05T06:44:25.107716+0000","last_undegraded":"2024-02-05T06:44:25.107716+0000","last_fullsized":"2024-02-05T06:44:25.107716+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:01.550518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:44:01.550518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:15.652514+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"295'24","reported_seq":432,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.103469+0000","last_change":"2024-02-05T06:43:57.825625+0000","last_active":"2024-02-05T06:44:25.103469+0000","last_peered":"2024-02-05T06:44:25.103469+0000","last_clean":"2024-02-05T06:44:25.103469+0000","last_became_active":"2024-02-05T06:36:54.958582+0000","last_became_peered":"2024-02-05T06:36:54.958582+0000","last_unstale":"2024-02-05T06:44:25.103469+0000","last_undegraded":"2024-02-05T06:44:25.103469+0000","last_fullsized":"2024-02-05T06:44:25.103469+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:43:57.825491+0000","last_deep_scrub":"115'16","last_deep_scrub_stamp":"2024-02-05T06:36:34.793243+0000","last_clean_scrub_stamp":"2024-02-05T06:43:57.825491+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:21.940154+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"308'5139","reported_seq":5183,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:26.859109+0000","last_change":"2024-02-05T06:43:29.947307+0000","last_active":"2024-02-05T06:44:26.859109+0000","last_peered":"2024-02-05T06:44:26.859109+0000","last_clean":"2024-02-05T06:44:26.859109+0000","last_became_active":"2024-02-05T06:43:29.946914+0000","last_became_peered":"2024-02-05T06:43:29.946914+0000","last_unstale":"2024-02-05T06:44:26.859109+0000","last_undegraded":"2024-02-05T06:44:26.859109+0000","last_fullsized":"2024-02-05T06:44:26.859109+0000","mapping_epoch":291,"log_start":"296'500","ondisk_log_start":"296'500","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4639,"log_dups_size":0,"ondisk_log_size":4639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:52.324646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336789504,"num_objects":5139,"num_object_clones":0,"num_object_copies":15417,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5139,"num_write_kb":328896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"83'128","reported_seq":563,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.106668+0000","last_change":"2024-02-05T06:43:47.429770+0000","last_active":"2024-02-05T06:44:25.106668+0000","last_peered":"2024-02-05T06:44:25.106668+0000","last_clean":"2024-02-05T06:44:25.106668+0000","last_became_active":"2024-02-05T06:41:46.053324+0000","last_became_peered":"2024-02-05T06:41:46.053324+0000","last_unstale":"2024-02-05T06:44:25.106668+0000","last_undegraded":"2024-02-05T06:44:25.106668+0000","last_fullsized":"2024-02-05T06:44:25.106668+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:43:47.429623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:43:47.429623+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:09.999835+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":544,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.112639+0000","last_change":"2024-02-05T06:43:46.502730+0000","last_active":"2024-02-05T06:44:25.112639+0000","last_peered":"2024-02-05T06:44:25.112639+0000","last_clean":"2024-02-05T06:44:25.112639+0000","last_became_active":"2024-02-05T06:36:54.958805+0000","last_became_peered":"2024-02-05T06:36:54.958805+0000","last_unstale":"2024-02-05T06:44:25.112639+0000","last_undegraded":"2024-02-05T06:44:25.112639+0000","last_fullsized":"2024-02-05T06:44:25.112639+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:46.502588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:43:46.502588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:05.939786+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":534,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.114574+0000","last_change":"2024-02-05T06:42:46.707508+0000","last_active":"2024-02-05T06:44:25.114574+0000","last_peered":"2024-02-05T06:44:25.114574+0000","last_clean":"2024-02-05T06:44:25.114574+0000","last_became_active":"2024-02-05T06:36:54.958082+0000","last_became_peered":"2024-02-05T06:36:54.958082+0000","last_unstale":"2024-02-05T06:44:25.114574+0000","last_undegraded":"2024-02-05T06:44:25.114574+0000","last_fullsized":"2024-02-05T06:44:25.114574+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:46.707382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:42:46.707382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":548,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.109828+0000","last_change":"2024-02-05T06:42:57.200247+0000","last_active":"2024-02-05T06:44:25.109828+0000","last_peered":"2024-02-05T06:44:25.109828+0000","last_clean":"2024-02-05T06:44:25.109828+0000","last_became_active":"2024-02-05T06:39:13.240038+0000","last_became_peered":"2024-02-05T06:39:13.240038+0000","last_unstale":"2024-02-05T06:44:25.109828+0000","last_undegraded":"2024-02-05T06:44:25.109828+0000","last_fullsized":"2024-02-05T06:44:25.109828+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:57.200123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:57.200123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:26.443497+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"295'39","reported_seq":447,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.103192+0000","last_change":"2024-02-05T06:41:54.656081+0000","last_active":"2024-02-05T06:44:25.103192+0000","last_peered":"2024-02-05T06:44:25.103192+0000","last_clean":"2024-02-05T06:44:25.103192+0000","last_became_active":"2024-02-05T06:36:54.961719+0000","last_became_peered":"2024-02-05T06:36:54.961719+0000","last_unstale":"2024-02-05T06:44:25.103192+0000","last_undegraded":"2024-02-05T06:44:25.103192+0000","last_fullsized":"2024-02-05T06:44:25.103192+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'25","last_scrub_stamp":"2024-02-05T06:41:54.655986+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-02-05T06:35:26.274944+0000","last_clean_scrub_stamp":"2024-02-05T06:41:54.655986+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:26.836926+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"308'5173","reported_seq":5217,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:27.202750+0000","last_change":"2024-02-05T06:43:29.947556+0000","last_active":"2024-02-05T06:44:27.202750+0000","last_peered":"2024-02-05T06:44:27.202750+0000","last_clean":"2024-02-05T06:44:27.202750+0000","last_became_active":"2024-02-05T06:43:29.947136+0000","last_became_peered":"2024-02-05T06:43:29.947136+0000","last_unstale":"2024-02-05T06:44:27.202750+0000","last_undegraded":"2024-02-05T06:44:27.202750+0000","last_fullsized":"2024-02-05T06:44:27.202750+0000","mapping_epoch":291,"log_start":"296'700","ondisk_log_start":"296'700","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4473,"log_dups_size":0,"ondisk_log_size":4473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:35.061776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339017728,"num_objects":5173,"num_object_clones":0,"num_object_copies":15519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5173,"num_write_kb":331072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":642,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.110861+0000","last_change":"2024-02-05T06:43:47.527180+0000","last_active":"2024-02-05T06:44:25.110861+0000","last_peered":"2024-02-05T06:44:25.110861+0000","last_clean":"2024-02-05T06:44:25.110861+0000","last_became_active":"2024-02-05T06:40:30.095202+0000","last_became_peered":"2024-02-05T06:40:30.095202+0000","last_unstale":"2024-02-05T06:44:25.110861+0000","last_undegraded":"2024-02-05T06:44:25.110861+0000","last_fullsized":"2024-02-05T06:44:25.110861+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:43:47.527064+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:41:50.706159+0000","last_clean_scrub_stamp":"2024-02-05T06:43:47.527064+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:55.073159+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":408,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.103362+0000","last_change":"2024-02-05T06:42:43.732921+0000","last_active":"2024-02-05T06:44:25.103362+0000","last_peered":"2024-02-05T06:44:25.103362+0000","last_clean":"2024-02-05T06:44:25.103362+0000","last_became_active":"2024-02-05T06:36:54.961560+0000","last_became_peered":"2024-02-05T06:36:54.961560+0000","last_unstale":"2024-02-05T06:44:25.103362+0000","last_undegraded":"2024-02-05T06:44:25.103362+0000","last_fullsized":"2024-02-05T06:44:25.103362+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:43.732781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:30.057099+0000","last_clean_scrub_stamp":"2024-02-05T06:42:43.732781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:26.837114+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"249'13","reported_seq":709,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.113003+0000","last_change":"2024-02-05T06:43:02.565236+0000","last_active":"2024-02-05T06:44:25.113003+0000","last_peered":"2024-02-05T06:44:25.113003+0000","last_clean":"2024-02-05T06:44:25.113003+0000","last_became_active":"2024-02-05T06:41:43.413139+0000","last_became_peered":"2024-02-05T06:41:43.413139+0000","last_unstale":"2024-02-05T06:44:25.113003+0000","last_undegraded":"2024-02-05T06:44:25.113003+0000","last_fullsized":"2024-02-05T06:44:25.113003+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'13","last_scrub_stamp":"2024-02-05T06:43:02.565121+0000","last_deep_scrub":"92'9","last_deep_scrub_stamp":"2024-02-05T06:34:52.283780+0000","last_clean_scrub_stamp":"2024-02-05T06:43:02.565121+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":534,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.116342+0000","last_change":"2024-02-05T06:42:39.205395+0000","last_active":"2024-02-05T06:44:25.116342+0000","last_peered":"2024-02-05T06:44:25.116342+0000","last_clean":"2024-02-05T06:44:25.116342+0000","last_became_active":"2024-02-05T06:36:54.971304+0000","last_became_peered":"2024-02-05T06:36:54.971304+0000","last_unstale":"2024-02-05T06:44:25.116342+0000","last_undegraded":"2024-02-05T06:44:25.116342+0000","last_fullsized":"2024-02-05T06:44:25.116342+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:39.205140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:39.205140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"295'25","reported_seq":435,"reported_epoch":308,"state":"active+clean","last_fresh":"2024-02-05T06:44:25.102870+0000","last_change":"2024-02-05T06:41:37.498015+0000","last_active":"2024-02-05T06:44:25.102870+0000","last_peered":"2024-02-05T06:44:25.102870+0000","last_clean":"2024-02-05T06:44:25.102870+0000","last_became_active":"2024-02-05T06:36:54.963620+0000","last_became_peered":"2024-02-05T06:36:54.963620+0000","last_unstale":"2024-02-05T06:44:25.102870+0000","last_undegraded":"2024-02-05T06:44:25.102870+0000","last_fullsized":"2024-02-05T06:44:25.102870+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'15","last_scrub_stamp":"2024-02-05T06:41:37.497882+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-02-05T06:35:04.584657+0000","last_clean_scrub_stamp":"2024-02-05T06:41:37.497882+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:26.836983+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":5378736128,"num_objects":82073,"num_object_clones":0,"num_object_copies":246219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":82073,"num_write_kb":5252672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16110518272,"data_stored":16110518272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76873,"ondisk_log_size":76873,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":245,"num_read_kb":240,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1573408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1114,"num_read_kb":871,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":373,"num_bytes_recovered":8234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":761,"ondisk_log_size":761,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103303,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3280544,"kb_used_data":2928660,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":90464608,"statfs":{"total":95995035648,"available":92635758592,"internally_reserved":0,"allocated":2998947840,"data_stored":2998330016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":360315338},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:43:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":1.198,"15min":1.292},"min":{"1min":0.29899999999999999,"5min":0.23699999999999999,"15min":0.219},"max":{"1min":2.2530000000000001,"5min":32.567,"15min":68.921000000000006},"last":0.623},{"interface":"front","average":{"1min":1.946,"5min":1.528,"15min":1.4630000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":18.013000000000002,"5min":32.603000000000002,"15min":69.616},"last":0.41999999999999998}]},{"osd":1,"last update":"Mon Feb 5 06:43:49 2024","interfaces":[{"interface":"back","average":{"1min":2.206,"5min":1.577,"15min":1.2789999999999999},"min":{"1min":0.50900000000000001,"5min":0.25,"15min":0.23599999999999999},"max":{"1min":17.969999999999999,"5min":32.459000000000003,"15min":89.231999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.724,"15min":1.369},"min":{"1min":0.372,"5min":0.26300000000000001,"15min":0.26100000000000001},"max":{"1min":4.0300000000000002,"5min":32.536000000000001,"15min":87.995999999999995},"last":0.51900000000000002}]},{"osd":2,"last update":"Mon Feb 5 06:43:49 2024","interfaces":[{"interface":"back","average":{"1min":2.1920000000000002,"5min":1.4330000000000001,"15min":1.1279999999999999},"min":{"1min":0.37,"5min":0.28399999999999997,"15min":0.26200000000000001},"max":{"1min":18.071999999999999,"5min":18.071999999999999,"15min":53.947000000000003},"last":0.315},{"interface":"front","average":{"1min":2.0739999999999998,"5min":1.1990000000000001,"15min":1.085},"min":{"1min":0.44700000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":18.045999999999999,"5min":18.045999999999999,"15min":53.299999999999997},"last":0.38700000000000001}]},{"osd":3,"last update":"Mon Feb 5 06:43:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.593,"15min":1.6060000000000001},"min":{"1min":0.39400000000000002,"5min":0.32700000000000001,"15min":0.316},"max":{"1min":3.4990000000000001,"5min":19.513000000000002,"15min":89.481999999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":1.167,"5min":1.2490000000000001,"15min":1.5569999999999999},"min":{"1min":0.47799999999999998,"5min":0.30599999999999999,"15min":0.27400000000000002},"max":{"1min":3.681,"5min":12.034000000000001,"15min":92.293999999999997},"last":0.53300000000000003}]},{"osd":4,"last update":"Mon Feb 5 06:43:49 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.496},"min":{"1min":0.28499999999999998,"5min":0.26200000000000001,"15min":0.24199999999999999},"max":{"1min":0.90700000000000003,"5min":2.2349999999999999,"15min":3.5409999999999999},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.54000000000000004,"15min":0.50900000000000001},"min":{"1min":0.33200000000000002,"5min":0.27100000000000002,"15min":0.23400000000000001},"max":{"1min":2.9039999999999999,"5min":2.9039999999999999,"15min":2.9039999999999999},"last":0.45300000000000001}]},{"osd":5,"last update":"Mon Feb 5 06:43:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.629,"15min":0.55000000000000004},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.25900000000000001},"max":{"1min":3.444,"5min":5.1050000000000004,"15min":5.1050000000000004},"last":0.251},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.59499999999999997,"15min":0.57899999999999996},"min":{"1min":0.36099999999999999,"5min":0.30299999999999999,"15min":0.252},"max":{"1min":2.746,"5min":2.746,"15min":3.5990000000000002},"last":0.58899999999999997}]},{"osd":6,"last update":"Mon Feb 5 06:43:49 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.67300000000000004,"15min":0.61799999999999999},"min":{"1min":0.38800000000000001,"5min":0.32000000000000001,"15min":0.314},"max":{"1min":3.2869999999999999,"5min":3.2869999999999999,"15min":3.5750000000000002},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.63300000000000001,"15min":0.58499999999999996},"min":{"1min":0.38200000000000001,"5min":0.307,"15min":0.245},"max":{"1min":3.0649999999999999,"5min":3.0649999999999999,"15min":3.2189999999999999},"last":0.37}]}]},{"osd":6,"up_from":49,"seq":210453397645,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1278088,"kb_used_data":967868,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":92467064,"statfs":{"total":95995035648,"available":94686273536,"internally_reserved":0,"allocated":991096832,"data_stored":990502135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":317651398},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:44:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":4.2670000000000003,"15min":2.0510000000000002},"min":{"1min":0.255,"5min":0.224,"15min":0.224},"max":{"1min":1.3560000000000001,"5min":98.918999999999997,"15min":98.918999999999997},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.66600000000000004,"5min":4.3739999999999997,"15min":2.0190000000000001},"min":{"1min":0.245,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.9039999999999999,"5min":98.775000000000006,"15min":98.775000000000006},"last":0.312}]},{"osd":1,"last update":"Mon Feb 5 06:44:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":3.8380000000000001,"15min":1.776},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.119,"5min":98.843999999999994,"15min":98.843999999999994},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.80300000000000005,"5min":4.0300000000000002,"15min":1.7969999999999999},"min":{"1min":0.436,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":2.0129999999999999,"5min":101.637,"15min":101.637},"last":0.29999999999999999}]},{"osd":2,"last update":"Mon Feb 5 06:44:11 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":3.5470000000000002,"15min":1.7290000000000001},"min":{"1min":0.33900000000000002,"5min":0.29199999999999998,"15min":0.22500000000000001},"max":{"1min":1.9390000000000001,"5min":85.617000000000004,"15min":85.617000000000004},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":3.383,"15min":1.6259999999999999},"min":{"1min":0.32600000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.365,"5min":85.335999999999999,"15min":85.335999999999999},"last":0.437}]},{"osd":3,"last update":"Mon Feb 5 06:44:11 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":3.8809999999999998,"15min":2.21},"min":{"1min":0.32000000000000001,"5min":0.307,"15min":0.27000000000000002},"max":{"1min":4.1150000000000002,"5min":98.888999999999996,"15min":99.269000000000005},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.79900000000000004,"5min":4.0099999999999998,"15min":2.3109999999999999},"min":{"1min":0.40799999999999997,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":2.8410000000000002,"5min":98.969999999999999,"15min":101.75700000000001},"last":0.42599999999999999}]},{"osd":4,"last update":"Mon Feb 5 06:44:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55900000000000005,"15min":0.48799999999999999},"min":{"1min":0.308,"5min":0.26500000000000001,"15min":0.23100000000000001},"max":{"1min":1.391,"5min":3.742,"15min":3.742},"last":0.249},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.53500000000000003,"15min":0.47399999999999998},"min":{"1min":0.30099999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.8480000000000001,"5min":2.371,"15min":2.371},"last":0.28199999999999997}]},{"osd":5,"last update":"Mon Feb 5 06:44:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.60899999999999999,"15min":0.51900000000000002},"min":{"1min":0.35299999999999998,"5min":0.29899999999999999,"15min":0.251},"max":{"1min":1.4450000000000001,"5min":4.4740000000000002,"15min":4.4740000000000002},"last":36.548999999999999},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.628,"15min":0.52500000000000002},"min":{"1min":0.34999999999999998,"5min":0.28699999999999998,"15min":0.24299999999999999},"max":{"1min":1.895,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":36.433}]},{"osd":7,"last update":"Mon Feb 5 06:43:29 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.56699999999999995,"15min":0.54700000000000004},"min":{"1min":0.36099999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.621,"5min":2.2330000000000001,"15min":6.0350000000000001},"last":38.436999999999998},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.60299999999999998,"15min":0.60399999999999998},"min":{"1min":0.33000000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":3.0990000000000002,"5min":3.0990000000000002,"15min":8.5399999999999991},"last":36.633000000000003}]}]},{"osd":1,"up_from":13,"seq":55834575014,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3320188,"kb_used_data":2950260,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":90424964,"statfs":{"total":95995035648,"available":92595163136,"internally_reserved":0,"allocated":3021066240,"data_stored":3020416431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:44:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":2.0859999999999999,"15min":1.5569999999999999},"min":{"1min":0.247,"5min":0.183,"15min":0.16700000000000001},"max":{"1min":2.4980000000000002,"5min":43.651000000000003,"15min":67.421000000000006},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.80900000000000005,"5min":2.5219999999999998,"15min":1.669},"min":{"1min":0.28499999999999998,"5min":0.20599999999999999,"15min":0.19600000000000001},"max":{"1min":4.3550000000000004,"5min":45.670999999999999,"15min":67.885000000000005},"last":0.76200000000000001}]},{"osd":2,"last update":"Mon Feb 5 06:43:28 2024","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":2.2989999999999999,"15min":1.304},"min":{"1min":0.249,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":17.422000000000001,"5min":43.447000000000003,"15min":43.447000000000003},"last":0.59499999999999997},{"interface":"front","average":{"1min":1.9019999999999999,"5min":1.819,"15min":1.2949999999999999},"min":{"1min":0.29399999999999998,"5min":0.249,"15min":0.20899999999999999},"max":{"1min":22.050000000000001,"5min":22.050000000000001,"15min":22.050000000000001},"last":0.42399999999999999}]},{"osd":3,"last update":"Mon Feb 5 06:44:00 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":2.4249999999999998,"15min":1.8500000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.25600000000000001},"max":{"1min":3.827,"5min":25.236999999999998,"15min":69.798000000000002},"last":1.421},{"interface":"front","average":{"1min":0.61599999999999999,"5min":2.1200000000000001,"15min":1.625},"min":{"1min":0.35399999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9059999999999999,"5min":43.354999999999997,"15min":70.765000000000001},"last":0.875}]},{"osd":4,"last update":"Mon Feb 5 06:44:18 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.1100000000000001,"15min":1.0189999999999999},"min":{"1min":0.433,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":3.452,"5min":7.0519999999999996,"15min":11.679},"last":0.55400000000000005},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.377,"15min":1.056},"min":{"1min":0.53800000000000003,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":4.9610000000000003,"5min":6.0730000000000004,"15min":6.0730000000000004},"last":1.004}]},{"osd":5,"last update":"Mon Feb 5 06:43:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.5469999999999999,"15min":1.1830000000000001},"min":{"1min":0.44900000000000001,"5min":0.29399999999999998,"15min":0.29299999999999998},"max":{"1min":3.8540000000000001,"5min":43.414999999999999,"15min":43.414999999999999},"last":1.224},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.2230000000000001,"15min":1.097},"min":{"1min":0.36599999999999999,"5min":0.33500000000000002,"15min":0.33100000000000002},"max":{"1min":4.9459999999999997,"5min":7.0919999999999996,"15min":13.455},"last":1.2050000000000001}]},{"osd":6,"last update":"Mon Feb 5 06:44:10 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":1.2,"15min":1.1519999999999999},"min":{"1min":0.40999999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.282,"5min":7.1260000000000003,"15min":13.391},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.879,"5min":1.5429999999999999,"15min":1.1220000000000001},"min":{"1min":0.46200000000000002,"5min":0.38100000000000001,"15min":0.379},"max":{"1min":1.9870000000000001,"5min":43.475000000000001,"15min":43.475000000000001},"last":0.84399999999999997}]},{"osd":7,"last update":"Mon Feb 5 06:43:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.3149999999999999,"15min":1.0920000000000001},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":4.2480000000000002,"5min":7.157,"15min":13.49},"last":0.52700000000000002},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.512,"15min":1.226},"min":{"1min":0.40400000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":4.5039999999999996,"5min":9.3249999999999993,"15min":9.3249999999999993},"last":0.96999999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738540,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1678188,"kb_used_data":1322148,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":92066964,"statfs":{"total":95995035648,"available":94276571136,"internally_reserved":0,"allocated":1353879552,"data_stored":1353245320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 06:44:10 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":1.655,"15min":1.1970000000000001},"min":{"1min":0.186,"5min":0.182,"15min":0.182},"max":{"1min":1.9670000000000001,"5min":18.59,"15min":18.59},"last":0.218},{"interface":"front","average":{"1min":0.63100000000000001,"5min":1.637,"15min":1.159},"min":{"1min":0.221,"5min":0.193,"15min":0.193},"max":{"1min":2.2400000000000002,"5min":18.300999999999998,"15min":18.300999999999998},"last":0.441}]},{"osd":2,"last update":"Mon Feb 5 06:44:27 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":1.415,"15min":1.2150000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":2.456,"5min":16.981999999999999,"15min":16.981999999999999},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":1.347,"15min":1.1479999999999999},"min":{"1min":0.255,"5min":0.223,"15min":0.223},"max":{"1min":1.3520000000000001,"5min":16.998999999999999,"15min":16.998999999999999},"last":0.30399999999999999}]},{"osd":3,"last update":"Mon Feb 5 06:43:54 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":1.998,"15min":1.399},"min":{"1min":0.27300000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.6639999999999999,"5min":43.302,"15min":43.302},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":2.4689999999999999,"15min":1.548},"min":{"1min":0.26500000000000001,"5min":0.26000000000000001,"15min":0.254},"max":{"1min":2.3410000000000002,"5min":43.651000000000003,"15min":43.651000000000003},"last":0.36899999999999999}]},{"osd":4,"last update":"Mon Feb 5 06:44:22 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":1.5549999999999999,"15min":1.165},"min":{"1min":0.32900000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.806,"5min":40.789999999999999,"15min":40.789999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":1.006,"5min":1.7430000000000001,"15min":1.2370000000000001},"min":{"1min":0.34599999999999997,"5min":0.32700000000000001,"15min":0.30099999999999999},"max":{"1min":8.4019999999999992,"5min":40.773000000000003,"15min":40.773000000000003},"last":0.33600000000000002}]},{"osd":5,"last update":"Mon Feb 5 06:43:45 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":1.47,"15min":1.0429999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.29299999999999998},"max":{"1min":2.3929999999999998,"5min":18.170999999999999,"15min":18.170999999999999},"last":1.3129999999999999},{"interface":"front","average":{"1min":0.623,"5min":1.276,"15min":1.048},"min":{"1min":0.40899999999999997,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.145,"5min":17.152999999999999,"15min":17.152999999999999},"last":1.8380000000000001}]},{"osd":6,"last update":"Mon Feb 5 06:44:10 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":1.367,"15min":1.1140000000000001},"min":{"1min":0.42999999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":2.0990000000000002,"5min":18.562999999999999,"15min":18.562999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.5189999999999999,"15min":1.133},"min":{"1min":0.44900000000000001,"5min":0.372,"15min":0.33000000000000002},"max":{"1min":13.102,"5min":18.213999999999999,"15min":18.213999999999999},"last":0.40999999999999998}]},{"osd":7,"last update":"Mon Feb 5 06:43:34 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":1.355,"15min":1.044},"min":{"1min":0.436,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.2430000000000001,"5min":17.210000000000001,"15min":17.210000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.65800000000000003,"5min":1.488,"15min":1.244},"min":{"1min":0.45400000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.161,"5min":18.533999999999999,"15min":18.533999999999999},"last":0.48199999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411489,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1319124,"kb_used_data":981132,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92426028,"statfs":{"total":95995035648,"available":94644252672,"internally_reserved":0,"allocated":1004679168,"data_stored":1004069949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:43:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":1.5589999999999999,"15min":1.5149999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.5299999999999998,"5min":8.9339999999999993,"15min":50.814},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.85699999999999998,"5min":1.7769999999999999,"15min":1.5},"min":{"1min":0.23899999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.919,"5min":10.026,"15min":50.802999999999997},"last":0.23499999999999999}]},{"osd":1,"last update":"Mon Feb 5 06:43:28 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":1.71,"15min":1.2829999999999999},"min":{"1min":0.248,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":2.3479999999999999,"5min":8.3879999999999999,"15min":31.303000000000001},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.78900000000000003,"5min":1.198,"15min":1.05},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.1389999999999998,"5min":6.7690000000000001,"15min":31.425000000000001},"last":0.41099999999999998}]},{"osd":3,"last update":"Mon Feb 5 06:44:00 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":1.5820000000000001,"15min":1.621},"min":{"1min":0.30299999999999999,"5min":0.25700000000000001,"15min":0.254},"max":{"1min":4.0739999999999998,"5min":13.504,"15min":53.127000000000002},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":1.631,"15min":1.546},"min":{"1min":0.251,"5min":0.23599999999999999,"15min":0.22500000000000001},"max":{"1min":1.929,"5min":11.388999999999999,"15min":52.607999999999997},"last":0.307}]},{"osd":4,"last update":"Mon Feb 5 06:44:19 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":1.044,"15min":0.93899999999999995},"min":{"1min":0.374,"5min":0.29699999999999999,"15min":0.23999999999999999},"max":{"1min":2.448,"5min":12.041,"15min":12.041},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.71099999999999997,"5min":1.0629999999999999,"15min":0.91900000000000004},"min":{"1min":0.33100000000000002,"5min":0.30199999999999999,"15min":0.29199999999999998},"max":{"1min":2.4609999999999999,"5min":15.689,"15min":15.689},"last":0.36499999999999999}]},{"osd":5,"last update":"Mon Feb 5 06:43:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":2.2010000000000001,"15min":1.2929999999999999},"min":{"1min":0.38200000000000001,"5min":0.316,"15min":0.29899999999999999},"max":{"1min":1.431,"5min":86.614999999999995,"15min":86.614999999999995},"last":1.7330000000000001},{"interface":"front","average":{"1min":0.70599999999999996,"5min":2.5110000000000001,"15min":1.4710000000000001},"min":{"1min":0.372,"5min":0.33600000000000002,"15min":0.33200000000000002},"max":{"1min":1.363,"5min":86.543999999999997,"15min":86.543999999999997},"last":0.69499999999999995}]},{"osd":6,"last update":"Mon Feb 5 06:44:19 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.97899999999999998,"15min":1.264},"min":{"1min":0.40000000000000002,"5min":0.315,"15min":0.315},"max":{"1min":2.4319999999999999,"5min":6.665,"15min":17.143999999999998},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.70299999999999996,"5min":1.1839999999999999,"15min":1.1220000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":2.0489999999999999,"5min":6.7919999999999998,"15min":10.27},"last":0.56899999999999995}]},{"osd":7,"last update":"Mon Feb 5 06:43:38 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":1.4139999999999999,"15min":1.1120000000000001},"min":{"1min":0.39800000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":2.3239999999999998,"5min":7.2949999999999999,"15min":10.206},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.62,"5min":1.141,"15min":1.0049999999999999},"min":{"1min":0.379,"5min":0.379,"15min":0.36899999999999999},"max":{"1min":1.0780000000000001,"5min":5.8639999999999999,"15min":6.4749999999999996},"last":3.3719999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149852,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3063464,"kb_used_data":2631072,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":90681688,"statfs":{"total":95995035648,"available":92858048512,"internally_reserved":0,"allocated":2694217728,"data_stored":2693587492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":442759623},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:44:07 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.4019999999999999,"15min":1.073},"min":{"1min":0.20499999999999999,"5min":0.184,"15min":0.184},"max":{"1min":9.6219999999999999,"5min":9.6219999999999999,"15min":13.391999999999999},"last":0.21199999999999999},{"interface":"front","average":{"1min":0.999,"5min":1.5049999999999999,"15min":1.3160000000000001},"min":{"1min":0.307,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":4.5369999999999999,"5min":7.0659999999999998,"15min":25.087},"last":1.1930000000000001}]},{"osd":1,"last update":"Mon Feb 5 06:44:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7789999999999999,"5min":1.7669999999999999,"15min":1.262},"min":{"1min":0.25800000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":1.1399999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.8069999999999999,"15min":1.23},"min":{"1min":0.27300000000000002,"5min":0.215,"15min":0.215},"max":{"1min":3.5830000000000002,"5min":8.5359999999999996,"15min":8.5359999999999996},"last":6.3010000000000002}]},{"osd":2,"last update":"Mon Feb 5 06:44:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":2.081,"15min":1.3839999999999999},"min":{"1min":0.31900000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":3.3250000000000002,"5min":41.529000000000003,"15min":41.529000000000003},"last":1.383},{"interface":"front","average":{"1min":0.98299999999999998,"5min":2.0049999999999999,"15min":1.294},"min":{"1min":0.27400000000000002,"5min":0.25,"15min":0.248},"max":{"1min":4.8049999999999997,"5min":41.759999999999998,"15min":41.759999999999998},"last":0.27600000000000002}]},{"osd":4,"last update":"Mon Feb 5 06:43:25 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":1.1830000000000001,"15min":0.92200000000000004},"min":{"1min":0.48099999999999998,"5min":0.312,"15min":0.307},"max":{"1min":3.181,"5min":7.8010000000000002,"15min":16.577999999999999},"last":0.85499999999999998},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.3220000000000001,"15min":1.091},"min":{"1min":0.30099999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":6.7329999999999997,"5min":6.7329999999999997,"15min":8.4060000000000006},"last":19.359999999999999}]},{"osd":5,"last update":"Mon Feb 5 06:43:46 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":1.49,"15min":1.163},"min":{"1min":0.52200000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":2.2559999999999998,"5min":5.8970000000000002,"15min":13.672000000000001},"last":20.257999999999999},{"interface":"front","average":{"1min":0.78300000000000003,"5min":1.27,"15min":0.96299999999999997},"min":{"1min":0.38700000000000001,"5min":0.32800000000000001,"15min":0.28399999999999997},"max":{"1min":2.2040000000000002,"5min":7.7679999999999998,"15min":15.792999999999999},"last":0.82899999999999996}]},{"osd":6,"last update":"Mon Feb 5 06:44:18 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.407,"15min":1.141},"min":{"1min":0.39500000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":4.5890000000000004,"5min":6.7809999999999997,"15min":8.3610000000000007},"last":1.171},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.3799999999999999,"15min":1.1479999999999999},"min":{"1min":0.48899999999999999,"5min":0.318,"15min":0.28599999999999998},"max":{"1min":3.5569999999999999,"5min":5.8799999999999999,"15min":6.601},"last":1.413}]},{"osd":7,"last update":"Mon Feb 5 06:43:42 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":1.2629999999999999,"15min":1.0860000000000001},"min":{"1min":0.45800000000000002,"5min":0.376,"15min":0.33400000000000002},"max":{"1min":1.6890000000000001,"5min":7.7320000000000002,"15min":45.039000000000001},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.84399999999999997,"5min":1.4890000000000001,"15min":1.256},"min":{"1min":0.58099999999999996,"5min":0.35899999999999999,"15min":0.34399999999999997},"max":{"1min":2.4830000000000001,"5min":5.8159999999999998,"15min":44.036999999999999},"last":1.4359999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986326,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1336296,"kb_used_data":976092,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":92408856,"statfs":{"total":95995035648,"available":94626668544,"internally_reserved":0,"allocated":999518208,"data_stored":998891746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:44:23 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":1.401,"15min":1.1080000000000001},"min":{"1min":0.26300000000000001,"5min":0.20799999999999999,"15min":0.20200000000000001},"max":{"1min":1.2250000000000001,"5min":33.372,"15min":33.372},"last":1.155},{"interface":"front","average":{"1min":0.47199999999999998,"5min":1.2949999999999999,"15min":1.175},"min":{"1min":0.251,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.1870000000000001,"5min":33.323999999999998,"15min":33.323999999999998},"last":1.1870000000000001}]},{"osd":1,"last update":"Mon Feb 5 06:44:23 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":1.4870000000000001,"15min":0.94399999999999995},"min":{"1min":0.27300000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":5.0599999999999996,"5min":33.747999999999998,"15min":33.747999999999998},"last":0.747},{"interface":"front","average":{"1min":0.98399999999999999,"5min":1.2849999999999999,"15min":0.94699999999999995},"min":{"1min":0.373,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.5840000000000001,"5min":33.691000000000003,"15min":33.691000000000003},"last":1.73}]},{"osd":2,"last update":"Mon Feb 5 06:44:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.93300000000000005,"15min":0.76400000000000001},"min":{"1min":0.39800000000000002,"5min":0.28199999999999997,"15min":0.245},"max":{"1min":1.2549999999999999,"5min":17.379999999999999,"15min":17.379999999999999},"last":0.626},{"interface":"front","average":{"1min":0.55400000000000005,"5min":1.1739999999999999,"15min":0.89900000000000002},"min":{"1min":0.315,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.3129999999999999,"5min":17.001999999999999,"15min":17.001999999999999},"last":1.306}]},{"osd":3,"last update":"Mon Feb 5 06:44:23 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":2.2000000000000002,"15min":1.992},"min":{"1min":0.38100000000000001,"5min":0.252,"15min":0.252},"max":{"1min":3.496,"5min":44.601999999999997,"15min":99.546999999999997},"last":1.895},{"interface":"front","average":{"1min":1.2250000000000001,"5min":2.2040000000000002,"15min":2.0070000000000001},"min":{"1min":0.40000000000000002,"5min":0.309,"15min":0.29099999999999998},"max":{"1min":10.73,"5min":44.511000000000003,"15min":99.245000000000005},"last":0.77500000000000002}]},{"osd":5,"last update":"Mon Feb 5 06:43:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.59499999999999997,"15min":0.54100000000000004},"min":{"1min":0.29599999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.147,"5min":4.6289999999999996,"15min":11.098000000000001},"last":1.2410000000000001},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.35299999999999998,"5min":0.25700000000000001,"15min":0.252},"max":{"1min":1.5980000000000001,"5min":3.5880000000000001,"15min":10.965999999999999},"last":0.66100000000000003}]},{"osd":6,"last update":"Mon Feb 5 06:44:20 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.67100000000000004,"15min":0.61499999999999999},"min":{"1min":0.35399999999999998,"5min":0.246,"15min":0.246},"max":{"1min":1.1519999999999999,"5min":4.375,"15min":11.868},"last":2.0099999999999998},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.65500000000000003,"15min":0.61299999999999999},"min":{"1min":0.32700000000000001,"5min":0.29899999999999999,"15min":0.27600000000000002},"max":{"1min":0.73799999999999999,"5min":3.5670000000000002,"15min":11.787000000000001},"last":1.9770000000000001}]},{"osd":7,"last update":"Mon Feb 5 06:43:32 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.66900000000000004,"15min":0.56799999999999995},"min":{"1min":0.36399999999999999,"5min":0.27800000000000002,"15min":0.26000000000000001},"max":{"1min":1.6579999999999999,"5min":5.1500000000000004,"15min":5.1500000000000004},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.71999999999999997,"15min":0.60099999999999998},"min":{"1min":0.32200000000000001,"5min":0.28899999999999998,"15min":0.28499999999999998},"max":{"1min":1.607,"5min":5.5890000000000004,"15min":5.5890000000000004},"last":1.2769999999999999}]}]},{"osd":5,"up_from":40,"seq":171798691985,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3315612,"kb_used_data":2905420,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":90429540,"statfs":{"total":95995035648,"available":92599848960,"internally_reserved":0,"allocated":2975150080,"data_stored":2974540755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":420018630},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":1.9299999999999999,"15min":1.3169999999999999},"min":{"1min":0.316,"5min":0.223,"15min":0.19700000000000001},"max":{"1min":2.8279999999999998,"5min":55.704999999999998,"15min":55.704999999999998},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":2.069,"15min":1.516},"min":{"1min":0.33600000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.242,"5min":56.789000000000001,"15min":56.789000000000001},"last":0.63200000000000001}]},{"osd":1,"last update":"Mon Feb 5 06:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":1.859,"15min":1.212},"min":{"1min":0.31,"5min":0.251,"15min":0.248},"max":{"1min":2.2730000000000001,"5min":41.191000000000003,"15min":41.191000000000003},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.70699999999999996,"5min":1.8620000000000001,"15min":1.0609999999999999},"min":{"1min":0.28100000000000003,"5min":0.24399999999999999,"15min":0.23499999999999999},"max":{"1min":3.226,"5min":41.173000000000002,"15min":41.173000000000002},"last":2.1720000000000002}]},{"osd":2,"last update":"Mon Feb 5 06:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":1.379,"15min":1.155},"min":{"1min":0.28999999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.536,"5min":30.052,"15min":36.901000000000003},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3620000000000001,"15min":1.071},"min":{"1min":0.39000000000000001,"5min":0.29499999999999998,"15min":0.28899999999999998},"max":{"1min":2.2970000000000002,"5min":30.472999999999999,"15min":37.984000000000002},"last":0.29699999999999999}]},{"osd":3,"last update":"Mon Feb 5 06:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":1.9970000000000001,"15min":1.2130000000000001},"min":{"1min":0.38,"5min":0.33300000000000002,"15min":0.31900000000000001},"max":{"1min":4.8860000000000001,"5min":55.417000000000002,"15min":55.417000000000002},"last":2.6179999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":2.1680000000000001,"15min":1.3049999999999999},"min":{"1min":0.42599999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.351,"5min":55.079000000000001,"15min":55.079000000000001},"last":3.0800000000000001}]},{"osd":4,"last update":"Mon Feb 5 06:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.58699999999999997,"15min":0.46999999999999997},"min":{"1min":0.26900000000000002,"5min":0.25800000000000001,"15min":0.23200000000000001},"max":{"1min":2.613,"5min":5.4969999999999999,"15min":5.4969999999999999},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.57199999999999995,"15min":0.497},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.98099999999999998,"5min":3.0150000000000001,"15min":3.0150000000000001},"last":0.56000000000000005}]},{"osd":6,"last update":"Mon Feb 5 06:44:14 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.70399999999999996,"15min":0.55200000000000005},"min":{"1min":0.28100000000000003,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.393,"5min":3.585,"15min":3.585},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.78000000000000003,"15min":0.59999999999999998},"min":{"1min":0.35299999999999998,"5min":0.31900000000000001,"15min":0.27600000000000002},"max":{"1min":1.5700000000000001,"5min":6.5179999999999998,"15min":6.5179999999999998},"last":0.40200000000000002}]},{"osd":7,"last update":"Mon Feb 5 06:43:38 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.624,"15min":0.52800000000000002},"min":{"1min":0.30199999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.851,"5min":2.6480000000000001,"15min":2.6480000000000001},"last":2.5720000000000001},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.70399999999999996,"15min":0.57799999999999996},"min":{"1min":0.28999999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.3759999999999999,"5min":3.3140000000000001,"15min":3.3140000000000001},"last":0.57699999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1356267520,"data_stored":1356267520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3029663744,"data_stored":3029663744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1005191168,"data_stored":1005191168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2704146432,"data_stored":2704146432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1004011520,"data_stored":1004011520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3010527232,"data_stored":3010527232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1003356160,"data_stored":1003356160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2997354496,"data_stored":2997354496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T06:44:31.482 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-05T06:44:31.482 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T06:44:31.482 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-02-05T06:44:31.482 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-05T06:44:31.574 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:31.612 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:31.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:31.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:31.714 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:31.750 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:31.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:31.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:31.852 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:31.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:31.888 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:31.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:31.989 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:32.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:32.038 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:32.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:32.130 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 52 16 88527 88511 106.356 72.4375 0.0114146 0.00939808 2024-02-05T06:44:32.139 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:32.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:32.180 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:32.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:32.281 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:32.360 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:44:32.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:32.361 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:32.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:32.462 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:32.489 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:32.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:32.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:32.591 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:32.608 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:44:32.624 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:32.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:32.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:32.659 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:44:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:44:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:44:32.726 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:32.764 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:32.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:32.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:32.865 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:32.898 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:32.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:32.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:33.000 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:33.040 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:33.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:33.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:33.130 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 53 16 89645 89629 105.668 69.875 0.0125879 0.00945855 2024-02-05T06:44:33.143 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:33.188 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:33.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:33.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:33.233 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:44:32 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:44:33.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:33.329 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:33.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:33.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:33.431 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:33.478 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:33.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:33.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:33.573 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:33 smithi167 ceph-mon[32774]: pgmap v540: 158 pgs: 158 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-02-05T06:44:33.573 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:33 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:44:33.573 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:33 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:44:33.580 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:33.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:33.625 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:33.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:33.658 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:44:33.658 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-02-05T06:44:33.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.659 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:90232 9.7 9:ed2b3a99:::benchmark_data_smithi028_129096_object90231:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-02-05T06:44:33.659 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:44:33.601115+0000", 2024-02-05T06:44:33.659 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.050400265999999999, 2024-02-05T06:44:33.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050438627, 2024-02-05T06:44:33.659 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:33.659 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-02-05T06:44:33.659 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:33.659 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:44:33.659 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:44:33.659 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 90232 2024-02-05T06:44:33.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.660 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:33.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:33.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.601115+0000", 2024-02-05T06:44:33.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:33.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:33.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.601115+0000", 2024-02-05T06:44:33.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:33.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:33.661 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.601120+0000", 2024-02-05T06:44:33.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4129999999999997e-06 2024-02-05T06:44:33.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.661 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:33.661 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.601192+0000", 2024-02-05T06:44:33.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2229000000000003e-05 2024-02-05T06:44:33.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.661 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:33.661 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.601203+0000", 2024-02-05T06:44:33.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0294e-05 2024-02-05T06:44:33.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.662 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:33.662 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.601210+0000", 2024-02-05T06:44:33.662 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6920000000000002e-06 2024-02-05T06:44:33.662 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.662 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:33.662 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.614151+0000", 2024-02-05T06:44:33.662 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012940568 2024-02-05T06:44:33.662 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.662 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:33.662 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.614258+0000", 2024-02-05T06:44:33.662 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010748500000000001 2024-02-05T06:44:33.662 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-02-05T06:44:33.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.614384+0000", 2024-02-05T06:44:33.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012586800000000001 2024-02-05T06:44:33.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:33.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.621081+0000", 2024-02-05T06:44:33.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0066963589999999998 2024-02-05T06:44:33.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:33.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.623819+0000", 2024-02-05T06:44:33.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0027385819999999998 2024-02-05T06:44:33.664 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:33.664 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:33.664 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:33.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.664 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:90233 9.f 9:fdfb3a9e:::benchmark_data_smithi028_129096_object90232:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-02-05T06:44:33.664 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:44:33.601518+0000", 2024-02-05T06:44:33.664 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.049997034000000003, 2024-02-05T06:44:33.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.050092539999999998, 2024-02-05T06:44:33.664 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:33.664 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-02-05T06:44:33.664 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:33.665 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:44:33.665 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:44:33.665 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 90233 2024-02-05T06:44:33.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.665 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:33.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:33.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.601518+0000", 2024-02-05T06:44:33.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:33.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:33.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.601518+0000", 2024-02-05T06:44:33.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:33.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:33.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.601519+0000", 2024-02-05T06:44:33.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1500000000000003e-07 2024-02-05T06:44:33.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:33.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.601569+0000", 2024-02-05T06:44:33.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0253999999999997e-05 2024-02-05T06:44:33.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:33.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.601577+0000", 2024-02-05T06:44:33.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7069999999999992e-06 2024-02-05T06:44:33.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:33.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.601583+0000", 2024-02-05T06:44:33.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1079999999999998e-06 2024-02-05T06:44:33.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:33.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.613750+0000", 2024-02-05T06:44:33.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012166868000000001 2024-02-05T06:44:33.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:33.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.613899+0000", 2024-02-05T06:44:33.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000149099 2024-02-05T06:44:33.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.668 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:44:33.668 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.614057+0000", 2024-02-05T06:44:33.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015828199999999999 2024-02-05T06:44:33.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.668 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:33.668 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.623824+0000", 2024-02-05T06:44:33.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0097668540000000002 2024-02-05T06:44:33.668 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:33.668 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:33.669 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:33.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.669 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:90239 9.f 9:f814795c:::benchmark_data_smithi028_129096_object90238:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-02-05T06:44:33.669 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:44:33.606346+0000", 2024-02-05T06:44:33.669 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.045168506999999997, 2024-02-05T06:44:33.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.045310988000000003, 2024-02-05T06:44:33.669 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:33.669 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-02-05T06:44:33.669 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:33.669 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:44:33.669 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:44:33.669 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 90239 2024-02-05T06:44:33.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.670 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:33.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:33.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.606346+0000", 2024-02-05T06:44:33.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:33.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:33.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.606346+0000", 2024-02-05T06:44:33.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:33.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:33.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.606348+0000", 2024-02-05T06:44:33.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.421e-06 2024-02-05T06:44:33.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:33.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.606384+0000", 2024-02-05T06:44:33.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5855999999999997e-05 2024-02-05T06:44:33.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:33.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.606398+0000", 2024-02-05T06:44:33.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3861e-05 2024-02-05T06:44:33.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:33.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.606406+0000", 2024-02-05T06:44:33.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6209999999999993e-06 2024-02-05T06:44:33.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:33.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.618502+0000", 2024-02-05T06:44:33.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012095471 2024-02-05T06:44:33.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.673 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:33.673 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.618642+0000", 2024-02-05T06:44:33.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014044099999999999 2024-02-05T06:44:33.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.673 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:44:33.673 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.618795+0000", 2024-02-05T06:44:33.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015270299999999999 2024-02-05T06:44:33.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:33.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.623828+0000", 2024-02-05T06:44:33.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0050337400000000001 2024-02-05T06:44:33.674 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:33.674 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:33.674 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:33.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.674 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:90243 9.a 9:59ab4009:::benchmark_data_smithi028_129096_object90242:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-02-05T06:44:33.674 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:44:33.610132+0000", 2024-02-05T06:44:33.674 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.041383088999999998, 2024-02-05T06:44:33.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041570822, 2024-02-05T06:44:33.674 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:33.675 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-02-05T06:44:33.675 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:33.675 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:44:33.675 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:44:33.675 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 90243 2024-02-05T06:44:33.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.675 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:33.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:33.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.610132+0000", 2024-02-05T06:44:33.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:33.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:33.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.610132+0000", 2024-02-05T06:44:33.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:33.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:33.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.610134+0000", 2024-02-05T06:44:33.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0339999999999999e-06 2024-02-05T06:44:33.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:33.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.610169+0000", 2024-02-05T06:44:33.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4669000000000003e-05 2024-02-05T06:44:33.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:33.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.610181+0000", 2024-02-05T06:44:33.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2566e-05 2024-02-05T06:44:33.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:33.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.610189+0000", 2024-02-05T06:44:33.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3839999999999997e-06 2024-02-05T06:44:33.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:33.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.610240+0000", 2024-02-05T06:44:33.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0368999999999998e-05 2024-02-05T06:44:33.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.678 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:33.678 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.610398+0000", 2024-02-05T06:44:33.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015796199999999999 2024-02-05T06:44:33.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.678 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,7", 2024-02-05T06:44:33.678 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.610553+0000", 2024-02-05T06:44:33.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000154831 2024-02-05T06:44:33.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.678 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:33.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.616518+0000", 2024-02-05T06:44:33.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059652969999999996 2024-02-05T06:44:33.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:33.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.617318+0000", 2024-02-05T06:44:33.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000799715 2024-02-05T06:44:33.679 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:33.679 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:33.679 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:33.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.679 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:90247 9.f 9:f4577c73:::benchmark_data_smithi028_129096_object90246:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-02-05T06:44:33.679 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:44:33.613963+0000", 2024-02-05T06:44:33.679 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.037551735000000003, 2024-02-05T06:44:33.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.037787233000000003, 2024-02-05T06:44:33.680 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:33.680 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-02-05T06:44:33.680 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:33.680 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:44:33.680 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:44:33.680 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 90247 2024-02-05T06:44:33.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.680 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:33.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:33.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.613963+0000", 2024-02-05T06:44:33.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:33.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:33.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.613963+0000", 2024-02-05T06:44:33.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:33.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:33.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.613965+0000", 2024-02-05T06:44:33.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8810000000000001e-06 2024-02-05T06:44:33.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:33.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.613996+0000", 2024-02-05T06:44:33.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0812999999999998e-05 2024-02-05T06:44:33.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:33.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.614007+0000", 2024-02-05T06:44:33.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0954e-05 2024-02-05T06:44:33.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:33.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.614015+0000", 2024-02-05T06:44:33.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3780000000000001e-06 2024-02-05T06:44:33.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:33.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.621072+0000", 2024-02-05T06:44:33.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0070569209999999999 2024-02-05T06:44:33.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:33.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.621191+0000", 2024-02-05T06:44:33.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011854999999999999 2024-02-05T06:44:33.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:44:33.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.621281+0000", 2024-02-05T06:44:33.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0629000000000003e-05 2024-02-05T06:44:33.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.684 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:33.684 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.623940+0000", 2024-02-05T06:44:33.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002658676 2024-02-05T06:44:33.684 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:33.684 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:33.684 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:33.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.684 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:90224 9.7 9:ec46e379:::benchmark_data_smithi028_129096_object90223:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-02-05T06:44:33.684 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:44:33.594473+0000", 2024-02-05T06:44:33.684 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.057041608000000001, 2024-02-05T06:44:33.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057321682999999998, 2024-02-05T06:44:33.684 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:33.685 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-02-05T06:44:33.685 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:33.685 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:44:33.685 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:44:33.685 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 90224 2024-02-05T06:44:33.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.685 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:33.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:33.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.594473+0000", 2024-02-05T06:44:33.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:33.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:33.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.594473+0000", 2024-02-05T06:44:33.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:33.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:33.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.594475+0000", 2024-02-05T06:44:33.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6139999999999999e-06 2024-02-05T06:44:33.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:33.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.594540+0000", 2024-02-05T06:44:33.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5016000000000002e-05 2024-02-05T06:44:33.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.687 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:33.687 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.594550+0000", 2024-02-05T06:44:33.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0127e-05 2024-02-05T06:44:33.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.687 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:33.687 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.594557+0000", 2024-02-05T06:44:33.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.844e-06 2024-02-05T06:44:33.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.687 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:33.687 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.610369+0000", 2024-02-05T06:44:33.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.015811882999999999 2024-02-05T06:44:33.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:33.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.610516+0000", 2024-02-05T06:44:33.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014698000000000001 2024-02-05T06:44:33.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-02-05T06:44:33.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.610656+0000", 2024-02-05T06:44:33.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014030600000000001 2024-02-05T06:44:33.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.689 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:33.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.621059+0000", 2024-02-05T06:44:33.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010403112000000001 2024-02-05T06:44:33.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.689 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:33.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.623810+0000", 2024-02-05T06:44:33.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0027513289999999998 2024-02-05T06:44:33.689 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:33.689 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:33.689 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:33.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.690 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:90248 9.7 9:edeabcd4:::benchmark_data_smithi028_129096_object90247:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-02-05T06:44:33.690 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:44:33.614027+0000", 2024-02-05T06:44:33.690 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.037487805999999999, 2024-02-05T06:44:33.690 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.037839401000000002, 2024-02-05T06:44:33.690 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:33.690 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-02-05T06:44:33.690 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:33.690 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:44:33.690 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:44:33.690 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 90248 2024-02-05T06:44:33.690 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.690 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:33.690 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.691 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:33.691 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.614027+0000", 2024-02-05T06:44:33.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:33.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.691 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:33.691 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.614027+0000", 2024-02-05T06:44:33.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:33.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.691 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.691 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:33.691 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.614028+0000", 2024-02-05T06:44:33.691 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8999999999999996e-07 2024-02-05T06:44:33.691 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.692 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:33.692 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.614055+0000", 2024-02-05T06:44:33.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6778999999999999e-05 2024-02-05T06:44:33.692 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.692 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:33.692 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.614061+0000", 2024-02-05T06:44:33.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4570000000000002e-06 2024-02-05T06:44:33.692 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.692 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.692 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:33.692 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.614066+0000", 2024-02-05T06:44:33.692 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1540000000000001e-06 2024-02-05T06:44:33.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.693 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:33.693 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.621096+0000", 2024-02-05T06:44:33.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0070300409999999999 2024-02-05T06:44:33.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.693 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:33.693 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.621183+0000", 2024-02-05T06:44:33.693 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7015000000000006e-05 2024-02-05T06:44:33.693 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.693 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.693 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,7", 2024-02-05T06:44:33.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.621278+0000", 2024-02-05T06:44:33.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4702000000000002e-05 2024-02-05T06:44:33.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:33.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.623833+0000", 2024-02-05T06:44:33.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0025552550000000002 2024-02-05T06:44:33.694 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:33.694 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:33.694 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:33.694 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:33.624361+0000", 2024-02-05T06:44:33.694 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00052807600000000004 2024-02-05T06:44:33.694 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:33.695 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:33.695 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:33.695 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:33.695 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-02-05T06:44:33.695 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 7 2024-02-05T06:44:33.695 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:44:33.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:33 smithi028 ceph-mon[37984]: pgmap v540: 158 pgs: 158 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-02-05T06:44:33.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:33 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:44:33.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:33 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:44:33.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:33 smithi028 ceph-mon[29442]: pgmap v540: 158 pgs: 158 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-02-05T06:44:33.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:33 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:44:33.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:33 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:44:33.726 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:33.766 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:33.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:33.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:33.868 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:33.904 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:33.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:33.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:34.006 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:34.045 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:34.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:34.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:34.130 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 54 16 90828 90812 105.08 73.9375 0.00760078 0.00951108 2024-02-05T06:44:34.147 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:34.185 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:34.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:34.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:34.286 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:34.312 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:34.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:34.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:34.414 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:34.451 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:34.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:34.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:34.552 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:34.567 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:34 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:44:34.567 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:34 smithi167 ceph-mon[32774]: osdmap e309: 8 total, 8 up, 8 in 2024-02-05T06:44:34.576 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:34.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:34.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:34.591 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-05T06:44:34.678 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:34.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:34 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:44:34.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:34 smithi028 ceph-mon[29442]: osdmap e309: 8 total, 8 up, 8 in 2024-02-05T06:44:34.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:34 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:44:34.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:34 smithi028 ceph-mon[37984]: osdmap e309: 8 total, 8 up, 8 in 2024-02-05T06:44:34.717 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:34.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:34.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:34.819 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:34.871 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:34.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:34.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:34.962 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:44:34.973 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:35.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:35.036 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:35.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:35.130 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 55 16 91947 91931 104.441 69.9375 0.0135906 0.00956978 2024-02-05T06:44:35.138 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:35.176 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:35.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:35.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:35.278 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:35.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:35.314 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:35.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:35.415 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:35.452 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:35.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:35.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:35.554 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:35.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:35.592 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:35.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:35.693 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:35.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:35.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:35.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:35 smithi167 ceph-mon[32774]: pgmap v542: 158 pgs: 158 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-02-05T06:44:35.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:35 smithi167 ceph-mon[32774]: osdmap e310: 8 total, 8 up, 8 in 2024-02-05T06:44:35.830 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:35.869 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:35.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:35.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:35.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:35 smithi028 ceph-mon[37984]: pgmap v542: 158 pgs: 158 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-02-05T06:44:35.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:35 smithi028 ceph-mon[37984]: osdmap e310: 8 total, 8 up, 8 in 2024-02-05T06:44:35.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:35 smithi028 ceph-mon[29442]: pgmap v542: 158 pgs: 158 active+clean; 5.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-02-05T06:44:35.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:35 smithi028 ceph-mon[29442]: osdmap e310: 8 total, 8 up, 8 in 2024-02-05T06:44:35.971 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:36.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:36.006 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:36.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:36.107 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:36.130 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 56 16 92972 92956 103.72 64.0625 0.00863959 0.0096299 2024-02-05T06:44:36.134 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:36.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:36.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:36.236 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:36.266 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:36.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:36.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:36.368 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:36.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:36.420 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:36.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:36.521 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:36.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:36.560 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:36.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:36.661 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:36.691 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:36.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:36.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:36.793 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:36.827 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:36.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:36.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:36.929 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:36.980 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:36.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:36.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:36.991 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:44:36.991 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-02-05T06:44:36.991 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-02-05T06:44:36.991 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-02-05T06:44:36.991 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:44:37.082 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:37.119 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:37.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:37.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:37.130 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 57 16 93999 93983 103.027 64.1875 0.0148827 0.00970135 2024-02-05T06:44:37.221 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:37.262 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:37.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:37.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:37.364 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:37.396 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:37.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:37.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:37.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:37.541 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:37.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:37.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:37.643 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:37.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:37.679 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:37.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:37.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:37 smithi028 ceph-mon[37984]: pgmap v544: 158 pgs: 158 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 67 MiB/s wr, 1.07k op/s 2024-02-05T06:44:37.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:37 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:44:37.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:37 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:44:37.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:37 smithi028 ceph-mon[29442]: pgmap v544: 158 pgs: 158 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 67 MiB/s wr, 1.07k op/s 2024-02-05T06:44:37.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:37 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:44:37.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:37 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:44:37.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:37 smithi167 ceph-mon[32774]: pgmap v544: 158 pgs: 158 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 67 MiB/s wr, 1.07k op/s 2024-02-05T06:44:37.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:37 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:44:37.729 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:37 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:44:37.776 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-05T06:44:37.780 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:37.815 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:37.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:37.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:37.917 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:37.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:37.973 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:37.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:38.073 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:38.111 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:38.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:38.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:38.130 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 58 16 95180 95164 102.523 73.8125 0.00762093 0.00974904 2024-02-05T06:44:38.167 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:44:38.213 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:38.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:38.246 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:38.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:38.347 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:38.382 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:38.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:38.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:38.484 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:38.509 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-02-05T06:44:38.532 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:38.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:38.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:38.634 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:38.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:38.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:38.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:38 smithi167 ceph-mon[32774]: osdmap e311: 8 total, 8 up, 8 in 2024-02-05T06:44:38.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:38 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/449382350' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:44:38.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:38 smithi167 ceph-mon[32774]: from='client.25366 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:44:38.779 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:38.823 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:38.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:38.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:38.832 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:38 smithi028 ceph-mon[29442]: osdmap e311: 8 total, 8 up, 8 in 2024-02-05T06:44:38.832 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:38 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/449382350' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:44:38.832 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:38 smithi028 ceph-mon[29442]: from='client.25366 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:44:38.833 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:38 smithi028 ceph-mon[37984]: osdmap e311: 8 total, 8 up, 8 in 2024-02-05T06:44:38.833 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:38 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/449382350' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:44:38.833 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:38 smithi028 ceph-mon[37984]: from='client.25366 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:44:38.925 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:38.961 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:38.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:38.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:39.063 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:39.131 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 59 16 96377 96361 102.053 74.8125 0.0167629 0.00979391 2024-02-05T06:44:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:39.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:39.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:39.241 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:39.286 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:39.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:39.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:39.388 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:39.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:39.424 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:39.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:39.525 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:39.557 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:39.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:39.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:39.659 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:39.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:39 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:44:39.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:39 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:44:39.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:39 smithi028 ceph-mon[29442]: pgmap v546: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 58 MiB/s wr, 924 op/s 2024-02-05T06:44:39.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:39 smithi028 ceph-mon[29442]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:44:39.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:39 smithi028 ceph-mon[29442]: from='client.25366 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:44:39.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:39 smithi028 ceph-mon[29442]: osdmap e312: 8 total, 8 up, 8 in 2024-02-05T06:44:39.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:39 smithi028 ceph-mon[29442]: 5.d scrub starts 2024-02-05T06:44:39.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:39 smithi028 ceph-mon[29442]: 5.d scrub ok 2024-02-05T06:44:39.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:39 smithi028 ceph-mon[29442]: 9.4 scrub starts 2024-02-05T06:44:39.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:39 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:44:39.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:39 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:44:39.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:39 smithi028 ceph-mon[37984]: pgmap v546: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 58 MiB/s wr, 924 op/s 2024-02-05T06:44:39.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:39 smithi028 ceph-mon[37984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:44:39.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:39 smithi028 ceph-mon[37984]: from='client.25366 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:44:39.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:39 smithi028 ceph-mon[37984]: osdmap e312: 8 total, 8 up, 8 in 2024-02-05T06:44:39.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:39 smithi028 ceph-mon[37984]: 5.d scrub starts 2024-02-05T06:44:39.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:39 smithi028 ceph-mon[37984]: 5.d scrub ok 2024-02-05T06:44:39.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:39 smithi028 ceph-mon[37984]: 9.4 scrub starts 2024-02-05T06:44:39.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:39.701 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:39.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:39.778 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:44:39.779 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:44:39.779 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 312, 2024-02-05T06:44:39.779 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "9801e956-c3ef-11ee-95b6-87774f69a715", 2024-02-05T06:44:39.779 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-02-05T06:27:59.919226+0000", 2024-02-05T06:44:39.779 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-02-05T06:44:38.469111+0000", 2024-02-05T06:44:39.779 INFO:teuthology.orchestra.run.smithi028.stdout: "last_up_change": "2024-02-05T06:33:15.833573+0000", 2024-02-05T06:44:39.779 INFO:teuthology.orchestra.run.smithi028.stdout: "last_in_change": "2024-02-05T06:41:42.383087+0000", 2024-02-05T06:44:39.779 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-05T06:44:39.779 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_num": 5804032, 2024-02-05T06:44:39.779 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_set": [ 2024-02-05T06:44:39.779 INFO:teuthology.orchestra.run.smithi028.stdout: "nodeep-scrub", 2024-02-05T06:44:39.779 INFO:teuthology.orchestra.run.smithi028.stdout: "pglog_hardlimit", 2024-02-05T06:44:39.780 INFO:teuthology.orchestra.run.smithi028.stdout: "purged_snapdirs", 2024-02-05T06:44:39.780 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_deletes", 2024-02-05T06:44:39.780 INFO:teuthology.orchestra.run.smithi028.stdout: "sortbitwise" 2024-02-05T06:44:39.780 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:44:39.780 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_version": 18, 2024-02-05T06:44:39.780 INFO:teuthology.orchestra.run.smithi028.stdout: "full_ratio": 0.94999998807907104, 2024-02-05T06:44:39.780 INFO:teuthology.orchestra.run.smithi028.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-05T06:44:39.780 INFO:teuthology.orchestra.run.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-05T06:44:39.780 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_snapshot": "", 2024-02-05T06:44:39.780 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_max": 9, 2024-02-05T06:44:39.780 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd": 8, 2024-02-05T06:44:39.780 INFO:teuthology.orchestra.run.smithi028.stdout: "require_min_compat_client": "luminous", 2024-02-05T06:44:39.780 INFO:teuthology.orchestra.run.smithi028.stdout: "min_compat_client": "luminous", 2024-02-05T06:44:39.781 INFO:teuthology.orchestra.run.smithi028.stdout: "require_osd_release": "squid", 2024-02-05T06:44:39.781 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_crimson": false, 2024-02-05T06:44:39.781 INFO:teuthology.orchestra.run.smithi028.stdout: "pools": [ 2024-02-05T06:44:39.781 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.781 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 1, 2024-02-05T06:44:39.781 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".mgr", 2024-02-05T06:44:39.781 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:31:10.441424+0000", 2024-02-05T06:44:39.781 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-02-05T06:44:39.781 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-02-05T06:44:39.781 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:44:39.781 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:44:39.781 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:44:39.781 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:44:39.781 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:44:39.781 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:44:39.781 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:44:39.782 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:44:39.782 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:44:39.782 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:44:39.782 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 11, 2024-02-05T06:44:39.782 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 11, 2024-02-05T06:44:39.782 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 11, 2024-02-05T06:44:39.782 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 11, 2024-02-05T06:44:39.782 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 11, 2024-02-05T06:44:39.782 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:44:39.782 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:44:39.782 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:44:39.782 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:44:39.782 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:44:39.782 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:44:39.783 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:44:39.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.783 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "142", 2024-02-05T06:44:39.783 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:44:39.783 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:44:39.783 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "129", 2024-02-05T06:44:39.783 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:44:39.783 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:44:39.783 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-02-05T06:44:39.783 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-02-05T06:44:39.783 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:44:39.783 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:44:39.783 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:44:39.783 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:44:39.783 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:44:39.784 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:44:39.784 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:44:39.784 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:44:39.784 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:44:39.784 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:44:39.784 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:44:39.784 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:44:39.784 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:44:39.784 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:44:39.784 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:44:39.784 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:44:39.784 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:44:39.784 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:44:39.784 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:44:39.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.785 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:44:39.785 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:44:39.785 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:44:39.785 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:44:39.785 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:44:39.785 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:44:39.785 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:44:39.785 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:44:39.785 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:44:39.785 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:44:39.785 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:44:39.785 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-02-05T06:44:39.785 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_max": 32, 2024-02-05T06:44:39.786 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_min": 1 2024-02-05T06:44:39.786 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.786 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:44:39.786 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": {} 2024-02-05T06:44:39.786 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.786 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:44:39.786 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:44:39.786 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.1800000667572021, 2024-02-05T06:44:39.786 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.1800000667572021, 2024-02-05T06:44:39.786 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-02-05T06:44:39.786 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2.1800000667572021, 2024-02-05T06:44:39.786 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2.1800000667572021, 2024-02-05T06:44:39.786 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-02-05T06:44:39.786 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-02-05T06:44:39.787 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-02-05T06:44:39.787 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.787 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.787 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 2, 2024-02-05T06:44:39.787 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "datapool", 2024-02-05T06:44:39.787 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:33:27.397416+0000", 2024-02-05T06:44:39.787 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-02-05T06:44:39.787 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-05T06:44:39.787 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:44:39.787 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:44:39.787 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:44:39.787 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:44:39.788 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:44:39.788 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:44:39.788 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:44:39.788 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:44:39.788 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:44:39.788 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:44:39.788 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 3, 2024-02-05T06:44:39.788 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 3, 2024-02-05T06:44:39.788 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 3, 2024-02-05T06:44:39.788 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 3, 2024-02-05T06:44:39.788 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 3, 2024-02-05T06:44:39.788 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:44:39.788 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:44:39.788 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:44:39.789 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:44:39.789 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:44:39.789 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:44:39.789 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:44:39.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.789 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "69", 2024-02-05T06:44:39.789 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:44:39.789 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:44:39.789 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T06:44:39.789 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:44:39.789 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:44:39.789 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 2, 2024-02-05T06:44:39.789 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 69, 2024-02-05T06:44:39.789 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:44:39.790 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:44:39.790 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:44:39.790 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:44:39.790 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:44:39.790 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:44:39.790 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:44:39.790 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:44:39.790 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:44:39.790 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:44:39.790 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:44:39.790 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:44:39.790 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:44:39.790 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:44:39.790 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:44:39.790 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:44:39.791 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:44:39.791 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:44:39.791 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:44:39.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.791 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:44:39.791 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:44:39.791 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:44:39.791 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:44:39.791 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:44:39.791 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:44:39.791 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:44:39.791 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:44:39.791 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:44:39.791 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:44:39.791 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:44:39.791 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-02-05T06:44:39.792 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:44:39.792 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd": {} 2024-02-05T06:44:39.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.792 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:44:39.792 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:44:39.792 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.6500000953674316, 2024-02-05T06:44:39.792 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.6500000953674316, 2024-02-05T06:44:39.792 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T06:44:39.792 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2.3299999237060547, 2024-02-05T06:44:39.792 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2.3299999237060547, 2024-02-05T06:44:39.792 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-02-05T06:44:39.792 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-02-05T06:44:39.792 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-02-05T06:44:39.792 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.793 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 3, 2024-02-05T06:44:39.793 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".rgw.root", 2024-02-05T06:44:39.793 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:33:28.090506+0000", 2024-02-05T06:44:39.793 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-02-05T06:44:39.793 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-02-05T06:44:39.793 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:44:39.793 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:44:39.793 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:44:39.793 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:44:39.793 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:44:39.793 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:44:39.793 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:44:39.793 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:44:39.794 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:44:39.794 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:44:39.794 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-02-05T06:44:39.794 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-02-05T06:44:39.794 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-02-05T06:44:39.794 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-02-05T06:44:39.794 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-02-05T06:44:39.794 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:44:39.794 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:44:39.794 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:44:39.794 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:44:39.794 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:44:39.794 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:44:39.795 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:44:39.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.795 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "67", 2024-02-05T06:44:39.795 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:44:39.795 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:44:39.795 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T06:44:39.795 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:44:39.796 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:44:39.796 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-02-05T06:44:39.796 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-02-05T06:44:39.796 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:44:39.796 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:44:39.796 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:44:39.796 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:44:39.796 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:44:39.796 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:44:39.796 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:44:39.796 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:44:39.796 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:44:39.796 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:44:39.796 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:44:39.796 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:44:39.797 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:44:39.797 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:44:39.797 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:44:39.797 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:44:39.797 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:44:39.797 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:44:39.797 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:44:39.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.797 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:44:39.797 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:44:39.797 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:44:39.797 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:44:39.797 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:44:39.797 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:44:39.798 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:44:39.798 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:44:39.798 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:44:39.798 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:44:39.798 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:44:39.798 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-02-05T06:44:39.798 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:44:39.798 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-02-05T06:44:39.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.798 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:44:39.798 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:44:39.798 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.5, 2024-02-05T06:44:39.798 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.5, 2024-02-05T06:44:39.798 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-02-05T06:44:39.798 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.5, 2024-02-05T06:44:39.799 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.5, 2024-02-05T06:44:39.799 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-02-05T06:44:39.799 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-02-05T06:44:39.799 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-02-05T06:44:39.799 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.799 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 4, 2024-02-05T06:44:39.799 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.log", 2024-02-05T06:44:39.799 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:33:29.866249+0000", 2024-02-05T06:44:39.799 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-02-05T06:44:39.799 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-02-05T06:44:39.799 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:44:39.799 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:44:39.800 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:44:39.800 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:44:39.800 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:44:39.800 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:44:39.800 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:44:39.800 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:44:39.800 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:44:39.800 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:44:39.800 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-02-05T06:44:39.800 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-02-05T06:44:39.800 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-02-05T06:44:39.800 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-02-05T06:44:39.800 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-02-05T06:44:39.800 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:44:39.801 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:44:39.801 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:44:39.801 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:44:39.801 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:44:39.801 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:44:39.801 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:44:39.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.801 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "69", 2024-02-05T06:44:39.801 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:44:39.801 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:44:39.801 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T06:44:39.801 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:44:39.801 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:44:39.801 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-02-05T06:44:39.801 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-02-05T06:44:39.801 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:44:39.802 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:44:39.802 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:44:39.802 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:44:39.802 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:44:39.802 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:44:39.802 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:44:39.802 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:44:39.802 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:44:39.802 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:44:39.802 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:44:39.802 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:44:39.802 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:44:39.802 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:44:39.802 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:44:39.802 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:44:39.802 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:44:39.803 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:44:39.803 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:44:39.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.803 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:44:39.803 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:44:39.803 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:44:39.803 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:44:39.803 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:44:39.803 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:44:39.803 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:44:39.803 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:44:39.803 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:44:39.803 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:44:39.803 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:44:39.803 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-02-05T06:44:39.803 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:44:39.804 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-02-05T06:44:39.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.804 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:44:39.804 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:44:39.804 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.25, 2024-02-05T06:44:39.804 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.25, 2024-02-05T06:44:39.804 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-02-05T06:44:39.804 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2.25, 2024-02-05T06:44:39.804 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2.25, 2024-02-05T06:44:39.804 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-02-05T06:44:39.804 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-02-05T06:44:39.804 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-02-05T06:44:39.804 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.805 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.805 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 5, 2024-02-05T06:44:39.805 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.control", 2024-02-05T06:44:39.805 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:33:31.826006+0000", 2024-02-05T06:44:39.805 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-02-05T06:44:39.805 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-02-05T06:44:39.805 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:44:39.805 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:44:39.805 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:44:39.805 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:44:39.805 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:44:39.805 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:44:39.805 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:44:39.805 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:44:39.805 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:44:39.806 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:44:39.806 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-02-05T06:44:39.806 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-02-05T06:44:39.806 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-02-05T06:44:39.806 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-02-05T06:44:39.806 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-02-05T06:44:39.806 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:44:39.806 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:44:39.806 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:44:39.806 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:44:39.806 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:44:39.806 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:44:39.806 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:44:39.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.806 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "71", 2024-02-05T06:44:39.806 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:44:39.807 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:44:39.807 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T06:44:39.807 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:44:39.807 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:44:39.807 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-02-05T06:44:39.807 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-02-05T06:44:39.807 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:44:39.807 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:44:39.807 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:44:39.807 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:44:39.807 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:44:39.807 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:44:39.807 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:44:39.807 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:44:39.807 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:44:39.808 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:44:39.808 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:44:39.808 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:44:39.808 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:44:39.808 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:44:39.808 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:44:39.808 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:44:39.808 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:44:39.808 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:44:39.808 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:44:39.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.808 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:44:39.808 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:44:39.808 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:44:39.809 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:44:39.809 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:44:39.809 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:44:39.809 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:44:39.809 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:44:39.809 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:44:39.809 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:44:39.809 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:44:39.809 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-02-05T06:44:39.809 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:44:39.809 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-02-05T06:44:39.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.809 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:44:39.809 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:44:39.809 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.5, 2024-02-05T06:44:39.810 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.5, 2024-02-05T06:44:39.810 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-02-05T06:44:39.810 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.5, 2024-02-05T06:44:39.810 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.5, 2024-02-05T06:44:39.810 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-02-05T06:44:39.810 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-02-05T06:44:39.810 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-02-05T06:44:39.810 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.810 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 6, 2024-02-05T06:44:39.810 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.meta", 2024-02-05T06:44:39.810 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:33:33.890835+0000", 2024-02-05T06:44:39.810 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-02-05T06:44:39.810 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-02-05T06:44:39.810 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:44:39.811 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:44:39.811 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:44:39.811 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:44:39.811 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:44:39.811 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:44:39.811 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:44:39.811 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:44:39.811 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:44:39.811 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:44:39.811 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-02-05T06:44:39.811 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-02-05T06:44:39.811 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-02-05T06:44:39.811 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-02-05T06:44:39.811 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-02-05T06:44:39.811 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:44:39.812 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:44:39.812 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:44:39.812 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:44:39.812 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:44:39.812 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:44:39.812 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:44:39.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.812 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "73", 2024-02-05T06:44:39.812 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:44:39.812 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:44:39.812 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T06:44:39.812 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:44:39.812 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:44:39.813 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-02-05T06:44:39.813 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-02-05T06:44:39.813 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:44:39.813 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:44:39.813 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:44:39.813 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:44:39.813 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:44:39.813 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:44:39.813 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:44:39.813 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:44:39.813 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:44:39.813 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:44:39.813 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:44:39.813 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:44:39.814 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:44:39.814 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:44:39.814 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:44:39.814 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:44:39.814 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:44:39.814 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:44:39.814 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:44:39.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.814 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:44:39.814 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:44:39.814 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:44:39.814 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:44:39.814 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:44:39.814 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:44:39.814 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:44:39.815 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:44:39.815 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:44:39.815 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:44:39.815 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:44:39.815 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-02-05T06:44:39.815 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_bias": 4 2024-02-05T06:44:39.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.815 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:44:39.815 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-02-05T06:44:39.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.815 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:44:39.815 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:44:39.815 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.5, 2024-02-05T06:44:39.815 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.5, 2024-02-05T06:44:39.816 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-02-05T06:44:39.816 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.5, 2024-02-05T06:44:39.816 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.5, 2024-02-05T06:44:39.816 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-02-05T06:44:39.816 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-02-05T06:44:39.816 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-02-05T06:44:39.816 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.816 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.816 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 9, 2024-02-05T06:44:39.816 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "unique_pool_2", 2024-02-05T06:44:39.816 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:43:28.629425+0000", 2024-02-05T06:44:39.816 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-02-05T06:44:39.816 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-02-05T06:44:39.817 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:44:39.817 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:44:39.817 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:44:39.817 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:44:39.817 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:44:39.817 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:44:39.817 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:44:39.817 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:44:39.817 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:44:39.817 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:44:39.817 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 16, 2024-02-05T06:44:39.817 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 16, 2024-02-05T06:44:39.817 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 16, 2024-02-05T06:44:39.817 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 16, 2024-02-05T06:44:39.817 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 16, 2024-02-05T06:44:39.818 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:44:39.818 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:44:39.818 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:44:39.818 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:44:39.818 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:44:39.818 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:44:39.818 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:44:39.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.818 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "296", 2024-02-05T06:44:39.818 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:44:39.818 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:44:39.818 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T06:44:39.818 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:44:39.818 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:44:39.818 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-02-05T06:44:39.819 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-02-05T06:44:39.819 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:44:39.819 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:44:39.819 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:44:39.819 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:44:39.819 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:44:39.819 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:44:39.819 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:44:39.819 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:44:39.819 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:44:39.819 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:44:39.819 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:44:39.819 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:44:39.819 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:44:39.820 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:44:39.820 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:44:39.820 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:44:39.820 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:44:39.820 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:44:39.820 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:44:39.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.820 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:44:39.820 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:44:39.820 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:44:39.820 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:44:39.820 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:44:39.820 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:44:39.820 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:44:39.821 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:44:39.821 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:44:39.821 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:44:39.821 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:44:39.821 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-02-05T06:44:39.821 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:44:39.821 INFO:teuthology.orchestra.run.smithi028.stdout: "rados": {} 2024-02-05T06:44:39.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.821 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:44:39.821 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:44:39.821 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2, 2024-02-05T06:44:39.821 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2, 2024-02-05T06:44:39.821 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-02-05T06:44:39.821 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-02-05T06:44:39.821 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-02-05T06:44:39.821 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 1, 2024-02-05T06:44:39.822 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 1, 2024-02-05T06:44:39.822 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 1 2024-02-05T06:44:39.822 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.822 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.822 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:44:39.822 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-02-05T06:44:39.822 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.822 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-02-05T06:44:39.822 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "c19fdd5b-09f6-45d4-a519-0a938cf1f2a6", 2024-02-05T06:44:39.822 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:44:39.822 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:44:39.822 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:44:39.822 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:44:39.822 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:44:39.822 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:44:39.822 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 8, 2024-02-05T06:44:39.822 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 291, 2024-02-05T06:44:39.823 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:44:39.823 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:44:39.823 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:44:39.823 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:44:39.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.823 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:44:39.823 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-02-05T06:44:39.823 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 301279106 2024-02-05T06:44:39.823 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.823 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.823 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:44:39.823 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:44:39.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.823 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:44:39.823 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6802", 2024-02-05T06:44:39.824 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 301279106 2024-02-05T06:44:39.824 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.824 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.824 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:44:39.824 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:44:39.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.824 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:44:39.824 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6804", 2024-02-05T06:44:39.824 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 301279106 2024-02-05T06:44:39.824 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.824 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.824 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:44:39.824 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:44:39.825 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.825 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:44:39.825 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6803", 2024-02-05T06:44:39.825 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 301279106 2024-02-05T06:44:39.825 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.825 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.825 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6801/301279106", 2024-02-05T06:44:39.825 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6802/301279106", 2024-02-05T06:44:39.825 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6804/301279106", 2024-02-05T06:44:39.825 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6803/301279106", 2024-02-05T06:44:39.825 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:44:39.825 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:44:39.825 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:44:39.825 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.826 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-02-05T06:44:39.826 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "52245f6e-eb96-4d0d-985b-dbb6fd3ce8de", 2024-02-05T06:44:39.826 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:44:39.826 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:44:39.826 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:44:39.826 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:44:39.826 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:44:39.826 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:44:39.826 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-02-05T06:44:39.826 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 291, 2024-02-05T06:44:39.826 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:44:39.826 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:44:39.826 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:44:39.826 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:44:39.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.827 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:39.827 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:39 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:44:39.828 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:39 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:44:39.828 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:39 smithi167 ceph-mon[32774]: pgmap v546: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 58 MiB/s wr, 924 op/s 2024-02-05T06:44:39.828 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:39 smithi167 ceph-mon[32774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:44:39.828 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:39 smithi167 ceph-mon[32774]: from='client.25366 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:44:39.828 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:39 smithi167 ceph-mon[32774]: osdmap e312: 8 total, 8 up, 8 in 2024-02-05T06:44:39.828 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:39 smithi167 ceph-mon[32774]: 5.d scrub starts 2024-02-05T06:44:39.828 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:39 smithi167 ceph-mon[32774]: 5.d scrub ok 2024-02-05T06:44:39.828 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:39 smithi167 ceph-mon[32774]: 9.4 scrub starts 2024-02-05T06:44:39.828 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:44:39.828 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6805", 2024-02-05T06:44:39.829 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1795224649 2024-02-05T06:44:39.829 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.829 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.829 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:44:39.829 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:44:39.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.829 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:44:39.829 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6806", 2024-02-05T06:44:39.829 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1795224649 2024-02-05T06:44:39.829 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.829 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.830 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:44:39.830 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:44:39.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.830 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:44:39.830 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6808", 2024-02-05T06:44:39.830 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1795224649 2024-02-05T06:44:39.830 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.830 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.830 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:44:39.830 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:44:39.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.830 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:44:39.830 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6807", 2024-02-05T06:44:39.830 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1795224649 2024-02-05T06:44:39.831 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.831 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.831 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6805/1795224649", 2024-02-05T06:44:39.831 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6806/1795224649", 2024-02-05T06:44:39.831 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6808/1795224649", 2024-02-05T06:44:39.831 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6807/1795224649", 2024-02-05T06:44:39.831 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:44:39.831 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:44:39.831 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:44:39.831 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.831 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-02-05T06:44:39.832 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "1e18b630-afdf-4de9-a1a2-37bdfdd01c67", 2024-02-05T06:44:39.832 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:44:39.832 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:44:39.832 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:44:39.832 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:44:39.832 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:44:39.832 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:44:39.832 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 18, 2024-02-05T06:44:39.832 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 278, 2024-02-05T06:44:39.832 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:44:39.832 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:44:39.832 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:44:39.832 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:44:39.832 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.833 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:44:39.833 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6809", 2024-02-05T06:44:39.833 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1993280018 2024-02-05T06:44:39.833 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.833 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.833 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:44:39.833 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:44:39.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.833 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:44:39.833 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6810", 2024-02-05T06:44:39.833 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1993280018 2024-02-05T06:44:39.833 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.833 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.834 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:44:39.834 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:44:39.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.834 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:44:39.834 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6812", 2024-02-05T06:44:39.834 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1993280018 2024-02-05T06:44:39.834 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.834 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.834 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:44:39.834 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:44:39.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.834 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:44:39.834 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6811", 2024-02-05T06:44:39.835 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1993280018 2024-02-05T06:44:39.835 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.835 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.835 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6809/1993280018", 2024-02-05T06:44:39.835 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6810/1993280018", 2024-02-05T06:44:39.835 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6812/1993280018", 2024-02-05T06:44:39.835 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6811/1993280018", 2024-02-05T06:44:39.835 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:44:39.835 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:44:39.835 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:44:39.835 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.836 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-02-05T06:44:39.836 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "3ae795af-3d59-41a6-95eb-b8447ff61282", 2024-02-05T06:44:39.836 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:44:39.836 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:44:39.836 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:44:39.836 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:44:39.836 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:44:39.836 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:44:39.836 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 26, 2024-02-05T06:44:39.836 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 291, 2024-02-05T06:44:39.836 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:44:39.836 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:44:39.836 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:44:39.836 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:44:39.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.837 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:44:39.837 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6813", 2024-02-05T06:44:39.837 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1294621398 2024-02-05T06:44:39.837 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.837 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.837 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:44:39.837 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:44:39.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.837 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:44:39.837 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6814", 2024-02-05T06:44:39.837 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1294621398 2024-02-05T06:44:39.837 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.837 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.838 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:44:39.838 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:44:39.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.838 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:44:39.838 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6816", 2024-02-05T06:44:39.838 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1294621398 2024-02-05T06:44:39.838 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.838 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.838 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:44:39.838 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:44:39.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.838 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:44:39.838 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6815", 2024-02-05T06:44:39.839 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1294621398 2024-02-05T06:44:39.839 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.839 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.839 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6813/1294621398", 2024-02-05T06:44:39.839 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6814/1294621398", 2024-02-05T06:44:39.839 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6816/1294621398", 2024-02-05T06:44:39.839 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6815/1294621398", 2024-02-05T06:44:39.839 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:44:39.839 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:44:39.839 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:44:39.839 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.840 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-02-05T06:44:39.840 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "fc6bd83d-570c-48c0-964d-e09b4608bc32", 2024-02-05T06:44:39.840 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:44:39.840 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:44:39.840 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:44:39.840 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:44:39.840 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:44:39.840 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:44:39.840 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 31, 2024-02-05T06:44:39.840 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 267, 2024-02-05T06:44:39.840 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:44:39.840 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:44:39.840 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:44:39.840 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:44:39.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.841 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:44:39.841 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6800", 2024-02-05T06:44:39.841 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4052090595 2024-02-05T06:44:39.841 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.841 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.841 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:44:39.841 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:44:39.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.841 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:44:39.841 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6801", 2024-02-05T06:44:39.841 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4052090595 2024-02-05T06:44:39.841 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.841 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.842 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:44:39.842 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:44:39.842 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.842 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:44:39.842 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6803", 2024-02-05T06:44:39.842 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4052090595 2024-02-05T06:44:39.842 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.842 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.842 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:44:39.842 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:44:39.842 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.842 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:44:39.842 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6802", 2024-02-05T06:44:39.842 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4052090595 2024-02-05T06:44:39.843 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.843 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.843 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.167:6800/4052090595", 2024-02-05T06:44:39.843 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.167:6801/4052090595", 2024-02-05T06:44:39.843 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.167:6803/4052090595", 2024-02-05T06:44:39.843 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.167:6802/4052090595", 2024-02-05T06:44:39.843 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:44:39.843 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:44:39.843 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:44:39.843 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.843 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-02-05T06:44:39.844 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "895bf465-2531-4d9f-bc23-1e75fd969f07", 2024-02-05T06:44:39.844 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:44:39.844 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:44:39.844 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:44:39.844 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:44:39.844 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:44:39.844 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:44:39.844 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 40, 2024-02-05T06:44:39.844 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 291, 2024-02-05T06:44:39.844 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:44:39.844 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:44:39.844 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:44:39.844 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:44:39.844 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.844 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:44:39.845 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6804", 2024-02-05T06:44:39.845 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1631080500 2024-02-05T06:44:39.845 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.845 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.845 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:44:39.845 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:44:39.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.845 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:44:39.845 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6805", 2024-02-05T06:44:39.845 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1631080500 2024-02-05T06:44:39.845 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.845 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.845 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:44:39.846 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:44:39.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.846 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:44:39.846 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6807", 2024-02-05T06:44:39.846 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1631080500 2024-02-05T06:44:39.846 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.846 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.846 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:44:39.846 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:44:39.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.846 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:44:39.847 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6806", 2024-02-05T06:44:39.848 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1631080500 2024-02-05T06:44:39.848 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.848 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.848 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.167:6804/1631080500", 2024-02-05T06:44:39.848 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.167:6805/1631080500", 2024-02-05T06:44:39.848 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.167:6807/1631080500", 2024-02-05T06:44:39.848 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.167:6806/1631080500", 2024-02-05T06:44:39.848 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:44:39.848 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:44:39.848 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:44:39.848 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.849 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.849 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-02-05T06:44:39.849 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "ce0a9300-87a4-4e37-b702-d667f168c0e0", 2024-02-05T06:44:39.849 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:44:39.849 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:44:39.849 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:44:39.849 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:44:39.849 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:44:39.849 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:44:39.849 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 49, 2024-02-05T06:44:39.849 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 254, 2024-02-05T06:44:39.849 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:44:39.849 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:44:39.850 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:44:39.850 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:44:39.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.850 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:44:39.850 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6808", 2024-02-05T06:44:39.850 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1644050617 2024-02-05T06:44:39.850 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.850 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.850 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:44:39.850 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:44:39.850 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.850 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:44:39.851 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6809", 2024-02-05T06:44:39.851 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1644050617 2024-02-05T06:44:39.851 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.851 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.851 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:44:39.851 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:44:39.851 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.851 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:44:39.851 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6811", 2024-02-05T06:44:39.851 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1644050617 2024-02-05T06:44:39.851 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.851 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.852 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:44:39.852 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:44:39.852 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.852 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:44:39.852 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6810", 2024-02-05T06:44:39.852 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1644050617 2024-02-05T06:44:39.852 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.852 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.852 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.167:6808/1644050617", 2024-02-05T06:44:39.852 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.167:6809/1644050617", 2024-02-05T06:44:39.852 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.167:6811/1644050617", 2024-02-05T06:44:39.852 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.167:6810/1644050617", 2024-02-05T06:44:39.852 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:44:39.853 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:44:39.853 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:44:39.853 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.853 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.853 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.853 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-02-05T06:44:39.853 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "0c545cbc-0873-4232-8807-4af9299ef95e", 2024-02-05T06:44:39.853 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:44:39.853 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:44:39.853 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:44:39.853 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:44:39.853 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:44:39.853 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:44:39.853 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 58, 2024-02-05T06:44:39.853 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 291, 2024-02-05T06:44:39.854 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:44:39.854 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:44:39.854 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:44:39.854 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:44:39.854 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.854 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:44:39.854 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6812", 2024-02-05T06:44:39.854 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4031475764 2024-02-05T06:44:39.854 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.854 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.854 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:44:39.854 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:44:39.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.855 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:44:39.855 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6813", 2024-02-05T06:44:39.855 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4031475764 2024-02-05T06:44:39.855 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.855 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.855 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:44:39.855 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:44:39.855 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.855 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:44:39.855 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6815", 2024-02-05T06:44:39.855 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4031475764 2024-02-05T06:44:39.855 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.855 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.856 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:44:39.856 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:44:39.856 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.856 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:44:39.856 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6814", 2024-02-05T06:44:39.856 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4031475764 2024-02-05T06:44:39.856 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.856 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.856 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.167:6812/4031475764", 2024-02-05T06:44:39.856 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.167:6813/4031475764", 2024-02-05T06:44:39.856 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.167:6815/4031475764", 2024-02-05T06:44:39.857 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.167:6814/4031475764", 2024-02-05T06:44:39.857 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:44:39.857 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:44:39.857 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:44:39.857 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.857 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.857 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:44:39.857 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_xinfo": [ 2024-02-05T06:44:39.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.857 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-02-05T06:44:39.857 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:44:39.857 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:44:39.857 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:44:39.857 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:44:39.858 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:44:39.858 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:44:37.880876+0000", 2024-02-05T06:44:39.858 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:44:39.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.858 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-02-05T06:44:39.858 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:44:39.858 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:44:39.858 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:44:39.858 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:44:39.858 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:44:39.858 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:44:01.333974+0000", 2024-02-05T06:44:39.858 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:44:39.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.858 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.859 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-02-05T06:44:39.859 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:44:39.859 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:44:39.859 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:44:39.859 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:44:39.859 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:44:39.859 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:44:36.590674+0000", 2024-02-05T06:44:39.859 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:44:39.859 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.859 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.859 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-02-05T06:44:39.859 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:44:39.859 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:44:39.859 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:44:39.860 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:44:39.860 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:44:39.860 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:44:08.163184+0000", 2024-02-05T06:44:39.860 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:44:39.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.860 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.860 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-02-05T06:44:39.860 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:44:39.860 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:44:39.860 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:44:39.860 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:44:39.860 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:44:39.860 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:44:16.763698+0000", 2024-02-05T06:44:39.860 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:44:39.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.861 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-02-05T06:44:39.861 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:44:39.861 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:44:39.861 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:44:39.861 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:44:39.861 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:44:39.861 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:44:32.888488+0000", 2024-02-05T06:44:39.861 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:44:39.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.861 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.861 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-02-05T06:44:39.861 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:44:39.861 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:44:39.862 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:44:39.862 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:44:39.862 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:44:39.862 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:43:51.376575+0000", 2024-02-05T06:44:39.862 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:44:39.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.862 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.862 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-02-05T06:44:39.862 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:44:39.862 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:44:39.862 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:44:39.862 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:44:39.862 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:44:39.862 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:44:03.223941+0000", 2024-02-05T06:44:39.862 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:44:39.863 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.863 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:44:39.863 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap": [ 2024-02-05T06:44:39.863 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.863 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "3.18", 2024-02-05T06:44:39.863 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-02-05T06:44:39.863 INFO:teuthology.orchestra.run.smithi028.stdout: 5, 2024-02-05T06:44:39.863 INFO:teuthology.orchestra.run.smithi028.stdout: 2, 2024-02-05T06:44:39.863 INFO:teuthology.orchestra.run.smithi028.stdout: 4 2024-02-05T06:44:39.863 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.863 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.863 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.863 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "5.1b", 2024-02-05T06:44:39.863 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-02-05T06:44:39.863 INFO:teuthology.orchestra.run.smithi028.stdout: 3, 2024-02-05T06:44:39.864 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-02-05T06:44:39.864 INFO:teuthology.orchestra.run.smithi028.stdout: 1 2024-02-05T06:44:39.864 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.864 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.864 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:44:39.864 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_items": [ 2024-02-05T06:44:39.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.864 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "3.9", 2024-02-05T06:44:39.864 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:44:39.864 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.864 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 4, 2024-02-05T06:44:39.864 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 1 2024-02-05T06:44:39.864 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.864 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.865 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "3.a", 2024-02-05T06:44:39.865 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:44:39.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.865 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 4, 2024-02-05T06:44:39.865 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-02-05T06:44:39.865 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.865 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.865 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.865 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.f", 2024-02-05T06:44:39.865 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:44:39.865 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.865 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-02-05T06:44:39.866 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 0 2024-02-05T06:44:39.866 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.866 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.866 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.12", 2024-02-05T06:44:39.866 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:44:39.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.866 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-02-05T06:44:39.866 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 0 2024-02-05T06:44:39.866 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.866 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.866 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.14", 2024-02-05T06:44:39.866 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:44:39.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.867 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-02-05T06:44:39.867 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-02-05T06:44:39.867 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.867 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.867 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.15", 2024-02-05T06:44:39.867 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:44:39.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.867 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 3, 2024-02-05T06:44:39.867 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-02-05T06:44:39.867 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.867 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.868 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.1d", 2024-02-05T06:44:39.868 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:44:39.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.868 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-02-05T06:44:39.868 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-02-05T06:44:39.868 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.868 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.868 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "5.1c", 2024-02-05T06:44:39.868 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:44:39.868 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.868 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 4, 2024-02-05T06:44:39.869 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 6 2024-02-05T06:44:39.869 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.869 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.869 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "5.1e", 2024-02-05T06:44:39.869 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:44:39.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.869 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 0, 2024-02-05T06:44:39.869 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 3 2024-02-05T06:44:39.869 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.869 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.869 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.869 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.869 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "6.13", 2024-02-05T06:44:39.870 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:44:39.870 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:44:39.870 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 3, 2024-02-05T06:44:39.870 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 1 2024-02-05T06:44:39.870 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.870 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:44:39.870 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.870 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:44:39.870 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_primaries": [], 2024-02-05T06:44:39.870 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_temp": [], 2024-02-05T06:44:39.870 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_temp": [], 2024-02-05T06:44:39.870 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklist": { 2024-02-05T06:44:39.870 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/1001442173": "2024-02-06T06:34:10.127914+0000", 2024-02-05T06:44:39.871 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3160026720": "2024-02-06T06:34:10.127914+0000", 2024-02-05T06:44:39.871 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/558171292": "2024-02-06T06:34:10.127914+0000", 2024-02-05T06:44:39.871 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2832437766": "2024-02-06T06:34:10.127914+0000", 2024-02-05T06:44:39.871 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/4125764599": "2024-02-06T06:28:54.105170+0000", 2024-02-05T06:44:39.871 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1081543653": "2024-02-06T06:28:54.105170+0000", 2024-02-05T06:44:39.871 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3794420554": "2024-02-06T06:28:27.553034+0000", 2024-02-05T06:44:39.871 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1560471317": "2024-02-06T06:34:10.127914+0000", 2024-02-05T06:44:39.871 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/4266601437": "2024-02-06T06:28:27.553034+0000", 2024-02-05T06:44:39.871 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/388893831": "2024-02-06T06:28:27.553034+0000", 2024-02-05T06:44:39.871 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/460817910": "2024-02-06T06:28:27.553034+0000", 2024-02-05T06:44:39.871 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2728305743": "2024-02-06T06:28:54.105170+0000", 2024-02-05T06:44:39.871 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3149394045": "2024-02-06T06:34:10.127914+0000", 2024-02-05T06:44:39.871 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3605013594": "2024-02-06T06:28:54.105170+0000" 2024-02-05T06:44:39.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.872 INFO:teuthology.orchestra.run.smithi028.stdout: "range_blocklist": {}, 2024-02-05T06:44:39.872 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profiles": { 2024-02-05T06:44:39.872 INFO:teuthology.orchestra.run.smithi028.stdout: "default": { 2024-02-05T06:44:39.872 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-02-05T06:44:39.872 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-02-05T06:44:39.872 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-02-05T06:44:39.872 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-02-05T06:44:39.872 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van" 2024-02-05T06:44:39.872 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:44:39.872 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps_queue": [], 2024-02-05T06:44:39.872 INFO:teuthology.orchestra.run.smithi028.stdout: "new_removed_snaps": [], 2024-02-05T06:44:39.872 INFO:teuthology.orchestra.run.smithi028.stdout: "new_purged_snaps": [], 2024-02-05T06:44:39.872 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_node_flags": {}, 2024-02-05T06:44:39.872 INFO:teuthology.orchestra.run.smithi028.stdout: "device_class_flags": {}, 2024-02-05T06:44:39.873 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": { 2024-02-05T06:44:39.873 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_enabled": false, 2024-02-05T06:44:39.873 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_bucket_count": 0, 2024-02-05T06:44:39.873 INFO:teuthology.orchestra.run.smithi028.stdout: "degraded_stretch_mode": 0, 2024-02-05T06:44:39.873 INFO:teuthology.orchestra.run.smithi028.stdout: "recovering_stretch_mode": 0, 2024-02-05T06:44:39.873 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_bucket": 0 2024-02-05T06:44:39.873 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:44:39.873 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:44:39.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:39.874 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:39.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:39.975 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:40.008 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:40.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:40.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:40.110 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:40.131 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-02-05T06:44:40.125270+0000 min lat: 0.0012123 max lat: 0.145886 avg lat: 0.00985075 2024-02-05T06:44:40.131 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T06:44:40.131 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 60 16 97423 97407 101.442 65.375 0.00938193 0.00985075 2024-02-05T06:44:40.147 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:40.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:40.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:40.249 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:40.315 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:40.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:40.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:40.417 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:40.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:40.454 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:40.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:40.530 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:44:40.531 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-02-05T06:44:40.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-02-05T06:44:40.531 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-02-05T06:44:40.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.531 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.44375.0:1111 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94598155037952 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-02-05T06:44:40.531 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:35:23.909001+0000", 2024-02-05T06:44:40.531 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 556.61141200099996, 2024-02-05T06:44:40.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.038390352, 2024-02-05T06:44:40.531 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:40.531 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-02-05T06:44:40.531 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:40.531 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.44375", 2024-02-05T06:44:40.531 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/1937500867", 2024-02-05T06:44:40.532 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1111 2024-02-05T06:44:40.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.532 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:40.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:40.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909001+0000", 2024-02-05T06:44:40.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:40.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909001+0000", 2024-02-05T06:44:40.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:40.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909003+0000", 2024-02-05T06:44:40.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0389999999999999e-06 2024-02-05T06:44:40.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:40.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909009+0000", 2024-02-05T06:44:40.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.361e-06 2024-02-05T06:44:40.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:40.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909023+0000", 2024-02-05T06:44:40.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4282e-05 2024-02-05T06:44:40.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.534 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:40.534 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909031+0000", 2024-02-05T06:44:40.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8860000000000004e-06 2024-02-05T06:44:40.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.534 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.534 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909101+0000", 2024-02-05T06:44:40.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0049999999999995e-05 2024-02-05T06:44:40.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.534 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-02-05T06:44:40.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909109+0000", 2024-02-05T06:44:40.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0600000000000008e-06 2024-02-05T06:44:40.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.917423+0000", 2024-02-05T06:44:40.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0083142839999999999 2024-02-05T06:44:40.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:40.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.917433+0000", 2024-02-05T06:44:40.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8470000000000008e-06 2024-02-05T06:44:40.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.931694+0000", 2024-02-05T06:44:40.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.014260489 2024-02-05T06:44:40.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for degraded object", 2024-02-05T06:44:40.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.932024+0000", 2024-02-05T06:44:40.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000330565 2024-02-05T06:44:40.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.942678+0000", 2024-02-05T06:44:40.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010653953000000001 2024-02-05T06:44:40.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:40.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.942792+0000", 2024-02-05T06:44:40.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011345000000000001 2024-02-05T06:44:40.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:40.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.947392+0000", 2024-02-05T06:44:40.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0046000859999999998 2024-02-05T06:44:40.537 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.537 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:40.537 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.538 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.538 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:129449 8.d 8:b9130c76:::benchmark_data_smithi028_105698_object40174:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e261)", 2024-02-05T06:44:40.538 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:10.059390+0000", 2024-02-05T06:44:40.538 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 150.461023286, 2024-02-05T06:44:40.538 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0677847279999999, 2024-02-05T06:44:40.538 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:40.538 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-02-05T06:44:40.538 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:40.538 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:40.538 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:40.538 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 129449 2024-02-05T06:44:40.538 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.539 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:40.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:40.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059390+0000", 2024-02-05T06:44:40.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:40.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059390+0000", 2024-02-05T06:44:40.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:40.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059392+0000", 2024-02-05T06:44:40.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.086e-06 2024-02-05T06:44:40.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:40.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059398+0000", 2024-02-05T06:44:40.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5589999999999996e-06 2024-02-05T06:44:40.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:40.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059410+0000", 2024-02-05T06:44:40.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1905e-05 2024-02-05T06:44:40.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:40.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059418+0000", 2024-02-05T06:44:40.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5599999999999994e-06 2024-02-05T06:44:40.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.060054+0000", 2024-02-05T06:44:40.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00063580799999999997 2024-02-05T06:44:40.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:40.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.060063+0000", 2024-02-05T06:44:40.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.346e-06 2024-02-05T06:44:40.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.542 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.542 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.093914+0000", 2024-02-05T06:44:40.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0338508449999999 2024-02-05T06:44:40.542 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.542 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-02-05T06:44:40.542 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.094022+0000", 2024-02-05T06:44:40.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000107959 2024-02-05T06:44:40.542 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.542 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.127068+0000", 2024-02-05T06:44:40.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.033046308000000003 2024-02-05T06:44:40.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:40.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.127175+0000", 2024-02-05T06:44:40.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106352 2024-02-05T06:44:40.543 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.543 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:40.543 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.543 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:129452 8.d 8:b624ec5d:::benchmark_data_smithi028_105698_object40177:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e261)", 2024-02-05T06:44:40.544 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:10.059436+0000", 2024-02-05T06:44:40.544 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 150.46097701100001, 2024-02-05T06:44:40.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.074451045, 2024-02-05T06:44:40.544 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:40.544 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-02-05T06:44:40.544 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:40.544 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:40.544 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:40.544 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 129452 2024-02-05T06:44:40.544 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.544 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:40.544 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.544 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:40.544 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059436+0000", 2024-02-05T06:44:40.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:40.545 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059436+0000", 2024-02-05T06:44:40.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.546 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:40.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059437+0000", 2024-02-05T06:44:40.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7299999999999995e-07 2024-02-05T06:44:40.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.546 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:40.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059440+0000", 2024-02-05T06:44:40.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.897e-06 2024-02-05T06:44:40.546 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.546 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:40.546 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059446+0000", 2024-02-05T06:44:40.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1510000000000001e-06 2024-02-05T06:44:40.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:40.547 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059452+0000", 2024-02-05T06:44:40.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0619999999999997e-06 2024-02-05T06:44:40.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.547 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.060094+0000", 2024-02-05T06:44:40.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00064214899999999995 2024-02-05T06:44:40.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:40.548 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.060101+0000", 2024-02-05T06:44:40.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5180000000000001e-06 2024-02-05T06:44:40.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.548 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.548 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.101188+0000", 2024-02-05T06:44:40.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0410871850000001 2024-02-05T06:44:40.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.548 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-02-05T06:44:40.548 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.101304+0000", 2024-02-05T06:44:40.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000116476 2024-02-05T06:44:40.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.549 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.133814+0000", 2024-02-05T06:44:40.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.032509405999999998 2024-02-05T06:44:40.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:40.549 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.133887+0000", 2024-02-05T06:44:40.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3528000000000005e-05 2024-02-05T06:44:40.549 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.549 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:40.549 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.550 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:129475 8.d 8:becec02b:::benchmark_data_smithi028_105698_object40200:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-02-05T06:44:40.550 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:10.094089+0000", 2024-02-05T06:44:40.550 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 150.42632440099999, 2024-02-05T06:44:40.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0243758060000001, 2024-02-05T06:44:40.550 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:40.550 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:40.550 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:40.550 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:40.550 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:40.550 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 129475 2024-02-05T06:44:40.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.550 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:40.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:40.551 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.094089+0000", 2024-02-05T06:44:40.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.551 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:40.551 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.094089+0000", 2024-02-05T06:44:40.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.551 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:40.551 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.094091+0000", 2024-02-05T06:44:40.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4049999999999998e-06 2024-02-05T06:44:40.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:40.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.094096+0000", 2024-02-05T06:44:40.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.724e-06 2024-02-05T06:44:40.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:40.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.094106+0000", 2024-02-05T06:44:40.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0434e-05 2024-02-05T06:44:40.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:40.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.094114+0000", 2024-02-05T06:44:40.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5950000000000001e-06 2024-02-05T06:44:40.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.553 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.553 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.101344+0000", 2024-02-05T06:44:40.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0072295349999999996 2024-02-05T06:44:40.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.553 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:40.553 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.101353+0000", 2024-02-05T06:44:40.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3430000000000002e-06 2024-02-05T06:44:40.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.555 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.555 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.555 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.111147+0000", 2024-02-05T06:44:40.555 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.009793742 2024-02-05T06:44:40.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:40.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.111256+0000", 2024-02-05T06:44:40.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010896900000000001 2024-02-05T06:44:40.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,7", 2024-02-05T06:44:40.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.111345+0000", 2024-02-05T06:44:40.556 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9696000000000003e-05 2024-02-05T06:44:40.556 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.556 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.556 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:40.556 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.112852+0000", 2024-02-05T06:44:40.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015071500000000001 2024-02-05T06:44:40.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.557 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:40.557 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.118307+0000", 2024-02-05T06:44:40.557 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0054547559999999998 2024-02-05T06:44:40.557 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.557 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.558 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:40.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.118341+0000", 2024-02-05T06:44:40.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4109000000000002e-05 2024-02-05T06:44:40.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.558 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:40.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.118382+0000", 2024-02-05T06:44:40.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0336999999999997e-05 2024-02-05T06:44:40.558 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.558 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.558 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:40.558 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.118465+0000", 2024-02-05T06:44:40.558 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3010999999999995e-05 2024-02-05T06:44:40.558 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.559 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:40.559 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.559 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.559 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.559 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:168946 8.4 8:2a7577b5:::benchmark_data_smithi028_105698_object79671:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:44:40.559 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.072950+0000", 2024-02-05T06:44:40.559 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 110.447463367, 2024-02-05T06:44:40.559 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4796128150000001, 2024-02-05T06:44:40.559 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:40.559 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-02-05T06:44:40.559 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:40.559 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:40.559 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:40.559 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 168946 2024-02-05T06:44:40.560 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.560 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:40.560 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.560 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:40.560 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.072950+0000", 2024-02-05T06:44:40.560 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.560 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.560 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.560 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:40.560 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.072950+0000", 2024-02-05T06:44:40.560 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.560 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.560 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:40.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.072952+0000", 2024-02-05T06:44:40.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9570000000000001e-06 2024-02-05T06:44:40.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:40.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.072956+0000", 2024-02-05T06:44:40.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2440000000000004e-06 2024-02-05T06:44:40.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.561 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:40.561 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.072967+0000", 2024-02-05T06:44:40.561 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0642000000000001e-05 2024-02-05T06:44:40.561 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.561 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.562 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:40.562 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.072975+0000", 2024-02-05T06:44:40.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1850000000000004e-06 2024-02-05T06:44:40.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.562 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.562 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.073035+0000", 2024-02-05T06:44:40.562 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0412999999999999e-05 2024-02-05T06:44:40.562 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.562 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.562 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:40.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.073049+0000", 2024-02-05T06:44:40.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3791000000000001e-05 2024-02-05T06:44:40.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.563 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.537403+0000", 2024-02-05T06:44:40.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.464353424 2024-02-05T06:44:40.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.563 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-02-05T06:44:40.563 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.537533+0000", 2024-02-05T06:44:40.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013027899999999999 2024-02-05T06:44:40.563 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.552442+0000", 2024-02-05T06:44:40.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.014908925999999999 2024-02-05T06:44:40.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:40.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.552563+0000", 2024-02-05T06:44:40.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012095399999999999 2024-02-05T06:44:40.564 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.564 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:40.565 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.565 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.565 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:168953 8.4 8:2441a741:::benchmark_data_smithi028_105698_object79678:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:44:40.565 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.076290+0000", 2024-02-05T06:44:40.565 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 110.444123502, 2024-02-05T06:44:40.565 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4712057160000001, 2024-02-05T06:44:40.565 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:40.565 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:40.565 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:40.565 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:40.565 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:40.566 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 168953 2024-02-05T06:44:40.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.566 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:40.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:40.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.076290+0000", 2024-02-05T06:44:40.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:40.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.076290+0000", 2024-02-05T06:44:40.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.567 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:40.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.076291+0000", 2024-02-05T06:44:40.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0999999999999997e-07 2024-02-05T06:44:40.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.567 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:40.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.076294+0000", 2024-02-05T06:44:40.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6210000000000001e-06 2024-02-05T06:44:40.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.567 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:40.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.076304+0000", 2024-02-05T06:44:40.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0172e-05 2024-02-05T06:44:40.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:40.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.076312+0000", 2024-02-05T06:44:40.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9500000000000001e-06 2024-02-05T06:44:40.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.076362+0000", 2024-02-05T06:44:40.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9302999999999999e-05 2024-02-05T06:44:40.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:40.569 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.076370+0000", 2024-02-05T06:44:40.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0369999999999995e-06 2024-02-05T06:44:40.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.569 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.537753+0000", 2024-02-05T06:44:40.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.461383297 2024-02-05T06:44:40.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:40.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.537848+0000", 2024-02-05T06:44:40.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4944000000000004e-05 2024-02-05T06:44:40.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:44:40.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.538252+0000", 2024-02-05T06:44:40.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000403631 2024-02-05T06:44:40.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:40.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547266+0000", 2024-02-05T06:44:40.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090147909999999994 2024-02-05T06:44:40.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:40.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547282+0000", 2024-02-05T06:44:40.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5639999999999999e-05 2024-02-05T06:44:40.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:40.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547408+0000", 2024-02-05T06:44:40.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012549300000000001 2024-02-05T06:44:40.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:40.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547440+0000", 2024-02-05T06:44:40.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2503999999999999e-05 2024-02-05T06:44:40.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.572 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:40.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547496+0000", 2024-02-05T06:44:40.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5423000000000001e-05 2024-02-05T06:44:40.572 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.572 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:40.572 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.572 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:168969 8.4 8:26e75fb4:::benchmark_data_smithi028_105698_object79694:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:44:40.572 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.095241+0000", 2024-02-05T06:44:40.573 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 110.425172284, 2024-02-05T06:44:40.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.452403653, 2024-02-05T06:44:40.573 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:40.573 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:40.573 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:40.573 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:40.573 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:40.573 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 168969 2024-02-05T06:44:40.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.573 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:40.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:40.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.095241+0000", 2024-02-05T06:44:40.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:40.574 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.095241+0000", 2024-02-05T06:44:40.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:40.574 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.095243+0000", 2024-02-05T06:44:40.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9439999999999999e-06 2024-02-05T06:44:40.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.575 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:40.575 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.095251+0000", 2024-02-05T06:44:40.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0669999999999992e-06 2024-02-05T06:44:40.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.575 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:40.575 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.095262+0000", 2024-02-05T06:44:40.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1071e-05 2024-02-05T06:44:40.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.575 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:40.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.095269+0000", 2024-02-05T06:44:40.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8140000000000003e-06 2024-02-05T06:44:40.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.095313+0000", 2024-02-05T06:44:40.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4387e-05 2024-02-05T06:44:40.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:40.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.095320+0000", 2024-02-05T06:44:40.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6290000000000004e-06 2024-02-05T06:44:40.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.542892+0000", 2024-02-05T06:44:40.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.447572388 2024-02-05T06:44:40.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:40.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.543029+0000", 2024-02-05T06:44:40.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013688 2024-02-05T06:44:40.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:44:40.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.543182+0000", 2024-02-05T06:44:40.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015253200000000001 2024-02-05T06:44:40.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:40.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547287+0000", 2024-02-05T06:44:40.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041050360000000003 2024-02-05T06:44:40.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:40.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547332+0000", 2024-02-05T06:44:40.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4749000000000001e-05 2024-02-05T06:44:40.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:40.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547552+0000", 2024-02-05T06:44:40.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022023999999999999 2024-02-05T06:44:40.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:40.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547579+0000", 2024-02-05T06:44:40.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7698000000000001e-05 2024-02-05T06:44:40.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:40.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547645+0000", 2024-02-05T06:44:40.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5217999999999998e-05 2024-02-05T06:44:40.579 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.580 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:40.580 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.580 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:168990 8.4 8:2816a5c0:::benchmark_data_smithi028_105698_object79715:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:44:40.580 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.116213+0000", 2024-02-05T06:44:40.580 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 110.40420078699999, 2024-02-05T06:44:40.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4316142949999999, 2024-02-05T06:44:40.580 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:40.580 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:40.580 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:40.580 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:40.581 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:40.581 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 168990 2024-02-05T06:44:40.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.581 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:40.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:40.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.116213+0000", 2024-02-05T06:44:40.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:40.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.116213+0000", 2024-02-05T06:44:40.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.582 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:40.582 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.116213+0000", 2024-02-05T06:44:40.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4199999999999996e-07 2024-02-05T06:44:40.582 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.582 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:40.582 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.116216+0000", 2024-02-05T06:44:40.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5890000000000001e-06 2024-02-05T06:44:40.582 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.582 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:40.582 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.116221+0000", 2024-02-05T06:44:40.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3009999999999997e-06 2024-02-05T06:44:40.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.583 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:40.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.116226+0000", 2024-02-05T06:44:40.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8409999999999999e-06 2024-02-05T06:44:40.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.583 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.129047+0000", 2024-02-05T06:44:40.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.01282082 2024-02-05T06:44:40.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.583 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:40.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.129057+0000", 2024-02-05T06:44:40.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0481e-05 2024-02-05T06:44:40.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.545004+0000", 2024-02-05T06:44:40.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.415947125 2024-02-05T06:44:40.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:40.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.545081+0000", 2024-02-05T06:44:40.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6550000000000004e-05 2024-02-05T06:44:40.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:44:40.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.545152+0000", 2024-02-05T06:44:40.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1728000000000002e-05 2024-02-05T06:44:40.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.585 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:40.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547292+0000", 2024-02-05T06:44:40.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002139939 2024-02-05T06:44:40.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.585 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:40.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547373+0000", 2024-02-05T06:44:40.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0328999999999996e-05 2024-02-05T06:44:40.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.586 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:40.586 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547719+0000", 2024-02-05T06:44:40.586 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034667500000000001 2024-02-05T06:44:40.586 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.586 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.586 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:40.586 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547746+0000", 2024-02-05T06:44:40.586 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6506999999999999e-05 2024-02-05T06:44:40.586 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.586 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.586 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:40.586 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547827+0000", 2024-02-05T06:44:40.586 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0867999999999999e-05 2024-02-05T06:44:40.587 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.587 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:40.587 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.587 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.587 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:168997 8.4 8:2fd29a35:::benchmark_data_smithi028_105698_object79722:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:44:40.587 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.123539+0000", 2024-02-05T06:44:40.587 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 110.39687379999999, 2024-02-05T06:44:40.587 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.43626739, 2024-02-05T06:44:40.587 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:40.587 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:40.588 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:40.588 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:40.588 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:40.588 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 168997 2024-02-05T06:44:40.588 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.588 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:40.588 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.588 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:40.588 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.123539+0000", 2024-02-05T06:44:40.588 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.588 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.588 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.588 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:40.588 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.123539+0000", 2024-02-05T06:44:40.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:40.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.123542+0000", 2024-02-05T06:44:40.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1509999999999998e-06 2024-02-05T06:44:40.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:40.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.123548+0000", 2024-02-05T06:44:40.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5780000000000004e-06 2024-02-05T06:44:40.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:40.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.123560+0000", 2024-02-05T06:44:40.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1674999999999999e-05 2024-02-05T06:44:40.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:40.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.123567+0000", 2024-02-05T06:44:40.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5739999999999998e-06 2024-02-05T06:44:40.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.129085+0000", 2024-02-05T06:44:40.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055173410000000003 2024-02-05T06:44:40.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:40.591 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.129091+0000", 2024-02-05T06:44:40.591 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5980000000000002e-06 2024-02-05T06:44:40.591 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.591 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.591 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.591 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:40.592 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547866+0000", 2024-02-05T06:44:40.592 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4187741730000001 2024-02-05T06:44:40.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.593 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:40.593 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547932+0000", 2024-02-05T06:44:40.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6223999999999995e-05 2024-02-05T06:44:40.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.593 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:44:40.593 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547998+0000", 2024-02-05T06:44:40.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5950000000000004e-05 2024-02-05T06:44:40.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:40.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.552551+0000", 2024-02-05T06:44:40.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0045535530000000001 2024-02-05T06:44:40.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:40.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.555220+0000", 2024-02-05T06:44:40.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0026689090000000001 2024-02-05T06:44:40.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:40.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.559657+0000", 2024-02-05T06:44:40.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0044368070000000001 2024-02-05T06:44:40.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:40.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.559714+0000", 2024-02-05T06:44:40.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6481999999999999e-05 2024-02-05T06:44:40.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:40.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.559807+0000", 2024-02-05T06:44:40.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3375000000000006e-05 2024-02-05T06:44:40.595 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.595 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:40.595 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.596 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:168999 8.4 8:2a087a6b:::benchmark_data_smithi028_105698_object79724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:44:40.596 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.124021+0000", 2024-02-05T06:44:40.596 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 110.396392117, 2024-02-05T06:44:40.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.436005618, 2024-02-05T06:44:40.596 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:40.596 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:40.596 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:40.596 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:40.596 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:40.596 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 168999 2024-02-05T06:44:40.596 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.596 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:40.596 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.597 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:40.597 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124021+0000", 2024-02-05T06:44:40.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.597 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.597 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:40.597 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124021+0000", 2024-02-05T06:44:40.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.597 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.597 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:40.597 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124022+0000", 2024-02-05T06:44:40.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.268e-06 2024-02-05T06:44:40.597 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.597 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:40.598 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124027+0000", 2024-02-05T06:44:40.598 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1500000000000001e-06 2024-02-05T06:44:40.598 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.598 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.598 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:40.598 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124037+0000", 2024-02-05T06:44:40.598 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0278999999999999e-05 2024-02-05T06:44:40.598 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.598 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.598 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:40.598 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124045+0000", 2024-02-05T06:44:40.598 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3520000000000007e-06 2024-02-05T06:44:40.598 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.598 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.599 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.129115+0000", 2024-02-05T06:44:40.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0050697160000000002 2024-02-05T06:44:40.599 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.599 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:40.599 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.129122+0000", 2024-02-05T06:44:40.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7549999999999997e-06 2024-02-05T06:44:40.599 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.599 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.599 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.549524+0000", 2024-02-05T06:44:40.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4204025490000001 2024-02-05T06:44:40.600 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.600 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.600 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:40.600 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.549646+0000", 2024-02-05T06:44:40.600 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121548 2024-02-05T06:44:40.600 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.600 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.600 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:44:40.600 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.549777+0000", 2024-02-05T06:44:40.600 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013103499999999999 2024-02-05T06:44:40.600 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.600 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.601 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:40.601 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.552562+0000", 2024-02-05T06:44:40.601 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002784777 2024-02-05T06:44:40.601 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.601 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.601 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:40.601 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.555297+0000", 2024-02-05T06:44:40.601 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002735632 2024-02-05T06:44:40.601 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.601 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.601 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:40.601 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.559878+0000", 2024-02-05T06:44:40.601 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0045809509999999998 2024-02-05T06:44:40.601 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.601 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.602 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:40.602 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.559934+0000", 2024-02-05T06:44:40.602 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5815e-05 2024-02-05T06:44:40.602 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.602 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.602 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:40.602 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.560027+0000", 2024-02-05T06:44:40.602 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2791e-05 2024-02-05T06:44:40.602 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.602 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:40.602 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.602 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.602 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.602 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:169001 8.4 8:2d24d8c7:::benchmark_data_smithi028_105698_object79726:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:44:40.603 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.124640+0000", 2024-02-05T06:44:40.603 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 110.395773639, 2024-02-05T06:44:40.603 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4355753789999999, 2024-02-05T06:44:40.603 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:40.603 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:40.603 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:40.603 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:40.603 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:40.603 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169001 2024-02-05T06:44:40.603 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.603 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:40.603 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.603 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:40.604 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124640+0000", 2024-02-05T06:44:40.604 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.604 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.604 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.604 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:40.604 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124640+0000", 2024-02-05T06:44:40.604 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.604 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.604 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.604 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:40.604 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124641+0000", 2024-02-05T06:44:40.604 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2759999999999999e-06 2024-02-05T06:44:40.605 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.605 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.605 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:40.605 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124646+0000", 2024-02-05T06:44:40.605 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7790000000000003e-06 2024-02-05T06:44:40.605 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.605 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.605 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:40.605 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124657+0000", 2024-02-05T06:44:40.605 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1739000000000001e-05 2024-02-05T06:44:40.605 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.605 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.605 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:40.605 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124666+0000", 2024-02-05T06:44:40.606 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3999999999999992e-06 2024-02-05T06:44:40.606 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.606 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.606 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.606 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.129144+0000", 2024-02-05T06:44:40.606 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0044785270000000004 2024-02-05T06:44:40.606 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.606 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.606 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:40.606 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.129150+0000", 2024-02-05T06:44:40.606 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7849999999999997e-06 2024-02-05T06:44:40.606 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.606 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.607 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.607 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.552592+0000", 2024-02-05T06:44:40.607 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.423442265 2024-02-05T06:44:40.607 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.607 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.607 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:40.607 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.552720+0000", 2024-02-05T06:44:40.607 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012783400000000001 2024-02-05T06:44:40.607 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.607 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.607 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:44:40.607 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.552831+0000", 2024-02-05T06:44:40.608 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000110325 2024-02-05T06:44:40.608 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.608 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.608 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:40.608 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.559592+0000", 2024-02-05T06:44:40.608 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.006761469 2024-02-05T06:44:40.608 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.608 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.608 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:40.608 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.559817+0000", 2024-02-05T06:44:40.608 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022453499999999999 2024-02-05T06:44:40.608 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.608 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.609 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:40.609 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.560092+0000", 2024-02-05T06:44:40.609 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027514500000000002 2024-02-05T06:44:40.609 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.609 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.609 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:40.609 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.560146+0000", 2024-02-05T06:44:40.609 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4534999999999999e-05 2024-02-05T06:44:40.609 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.609 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.609 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:40.609 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.560215+0000", 2024-02-05T06:44:40.610 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8764999999999996e-05 2024-02-05T06:44:40.610 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.610 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:40.610 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.610 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.610 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.610 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:169022 8.4 8:2073961b:::benchmark_data_smithi028_105698_object79747:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:44:40.610 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.143624+0000", 2024-02-05T06:44:40.610 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 110.376789528, 2024-02-05T06:44:40.610 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4328469529999999, 2024-02-05T06:44:40.610 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:40.610 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:40.611 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:40.611 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:40.611 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:40.611 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169022 2024-02-05T06:44:40.611 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.611 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:40.611 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.611 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:40.611 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.143624+0000", 2024-02-05T06:44:40.611 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.611 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.611 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.612 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:40.612 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.143624+0000", 2024-02-05T06:44:40.612 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.612 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.612 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.612 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:40.612 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.143625+0000", 2024-02-05T06:44:40.612 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5120000000000001e-06 2024-02-05T06:44:40.612 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.612 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.612 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:40.612 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.143630+0000", 2024-02-05T06:44:40.613 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8289999999999998e-06 2024-02-05T06:44:40.613 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.613 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.613 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:40.613 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.143642+0000", 2024-02-05T06:44:40.613 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1574e-05 2024-02-05T06:44:40.613 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.613 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.613 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:40.613 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.143648+0000", 2024-02-05T06:44:40.613 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2870000000000001e-06 2024-02-05T06:44:40.613 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.613 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.614 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.614 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.143712+0000", 2024-02-05T06:44:40.614 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3972000000000003e-05 2024-02-05T06:44:40.614 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.614 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.614 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:40.614 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.143724+0000", 2024-02-05T06:44:40.614 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2119999999999999e-05 2024-02-05T06:44:40.614 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.614 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.614 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.614 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.555338+0000", 2024-02-05T06:44:40.614 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4116135439999999 2024-02-05T06:44:40.615 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.615 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.615 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:40.615 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.555474+0000", 2024-02-05T06:44:40.615 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013640399999999999 2024-02-05T06:44:40.615 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.615 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.615 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:44:40.615 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.555604+0000", 2024-02-05T06:44:40.615 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000130102 2024-02-05T06:44:40.615 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.615 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.615 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:40.615 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.559821+0000", 2024-02-05T06:44:40.616 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0042167460000000004 2024-02-05T06:44:40.616 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.616 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.616 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:40.616 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.565344+0000", 2024-02-05T06:44:40.616 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055230440000000004 2024-02-05T06:44:40.616 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.616 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.616 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:40.616 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576302+0000", 2024-02-05T06:44:40.616 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010958239 2024-02-05T06:44:40.616 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.616 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.616 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:40.617 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576356+0000", 2024-02-05T06:44:40.617 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3823999999999998e-05 2024-02-05T06:44:40.617 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.617 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.617 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:40.617 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576471+0000", 2024-02-05T06:44:40.617 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000114756 2024-02-05T06:44:40.617 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.617 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:40.617 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.617 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.617 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.617 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:169057 8.4 8:26322b02:::benchmark_data_smithi028_105698_object79782:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:44:40.617 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.170483+0000", 2024-02-05T06:44:40.618 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 110.34993054100001, 2024-02-05T06:44:40.618 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.409427684, 2024-02-05T06:44:40.618 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:40.618 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:40.618 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:40.618 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:40.618 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:40.618 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169057 2024-02-05T06:44:40.618 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.618 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:40.618 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.618 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:40.619 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.170483+0000", 2024-02-05T06:44:40.619 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.619 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.619 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.619 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:40.619 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.170483+0000", 2024-02-05T06:44:40.619 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.619 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.619 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.619 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:40.619 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.170485+0000", 2024-02-05T06:44:40.619 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1239999999999999e-06 2024-02-05T06:44:40.619 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.619 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.620 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:40.620 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.170490+0000", 2024-02-05T06:44:40.620 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9490000000000004e-06 2024-02-05T06:44:40.620 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.620 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.620 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:40.620 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.170502+0000", 2024-02-05T06:44:40.620 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1851999999999999e-05 2024-02-05T06:44:40.620 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.620 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.620 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:40.620 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.170509+0000", 2024-02-05T06:44:40.620 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2320000000000004e-06 2024-02-05T06:44:40.621 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.621 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.621 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.621 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.170567+0000", 2024-02-05T06:44:40.621 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8152000000000002e-05 2024-02-05T06:44:40.621 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.621 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.621 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:40.621 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.170577+0000", 2024-02-05T06:44:40.621 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9150000000000008e-06 2024-02-05T06:44:40.621 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.621 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.621 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.621 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.557586+0000", 2024-02-05T06:44:40.621 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3870086580000001 2024-02-05T06:44:40.621 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.622 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.622 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:40.622 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.557731+0000", 2024-02-05T06:44:40.622 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014533499999999999 2024-02-05T06:44:40.622 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.622 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.622 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:44:40.622 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.557845+0000", 2024-02-05T06:44:40.622 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000114529 2024-02-05T06:44:40.622 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.622 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.622 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:40.622 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.560223+0000", 2024-02-05T06:44:40.622 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002377503 2024-02-05T06:44:40.623 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.623 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.623 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:40.623 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.565405+0000", 2024-02-05T06:44:40.623 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0051823950000000002 2024-02-05T06:44:40.623 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.623 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.623 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:40.623 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.579808+0000", 2024-02-05T06:44:40.623 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.014402223 2024-02-05T06:44:40.623 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.623 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.623 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:40.623 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.579872+0000", 2024-02-05T06:44:40.624 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4863999999999994e-05 2024-02-05T06:44:40.624 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.624 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.624 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:40.624 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.579910+0000", 2024-02-05T06:44:40.624 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7953000000000001e-05 2024-02-05T06:44:40.624 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.624 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:40.624 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.624 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.624 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.624 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:169060 8.4 8:282dec9e:::benchmark_data_smithi028_105698_object79785:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:44:40.624 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.172664+0000", 2024-02-05T06:44:40.625 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 110.347749509, 2024-02-05T06:44:40.625 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4074563200000001, 2024-02-05T06:44:40.625 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:40.625 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:40.625 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:40.625 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:40.625 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:40.625 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169060 2024-02-05T06:44:40.625 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.625 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:40.625 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.625 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:40.626 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.172664+0000", 2024-02-05T06:44:40.626 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.626 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.626 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.626 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:40.626 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.172664+0000", 2024-02-05T06:44:40.626 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.626 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.626 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.626 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:40.626 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.172665+0000", 2024-02-05T06:44:40.626 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4389999999999999e-06 2024-02-05T06:44:40.626 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.627 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.627 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:40.627 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.172673+0000", 2024-02-05T06:44:40.627 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3529999999999997e-06 2024-02-05T06:44:40.627 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.627 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.627 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:40.627 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.172683+0000", 2024-02-05T06:44:40.627 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0891e-05 2024-02-05T06:44:40.627 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.627 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.627 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:40.628 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.172691+0000", 2024-02-05T06:44:40.628 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4569999999999999e-06 2024-02-05T06:44:40.628 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.628 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.628 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.628 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.172734+0000", 2024-02-05T06:44:40.628 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2962e-05 2024-02-05T06:44:40.628 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.628 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.628 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:40.628 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.172740+0000", 2024-02-05T06:44:40.628 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1149999999999999e-06 2024-02-05T06:44:40.628 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.628 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.629 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.629 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.560263+0000", 2024-02-05T06:44:40.629 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3875225520000001 2024-02-05T06:44:40.629 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.629 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.629 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:40.629 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.560368+0000", 2024-02-05T06:44:40.629 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000105481 2024-02-05T06:44:40.629 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.629 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.629 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:44:40.629 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.560522+0000", 2024-02-05T06:44:40.629 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015420200000000001 2024-02-05T06:44:40.629 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.629 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.630 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:40.630 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.565356+0000", 2024-02-05T06:44:40.630 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048341139999999996 2024-02-05T06:44:40.630 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.630 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.630 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:40.630 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.571129+0000", 2024-02-05T06:44:40.630 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0057722889999999999 2024-02-05T06:44:40.630 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.630 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.630 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:40.630 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580047+0000", 2024-02-05T06:44:40.630 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0089180249999999996 2024-02-05T06:44:40.631 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.631 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.631 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:40.631 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580093+0000", 2024-02-05T06:44:40.631 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6730000000000002e-05 2024-02-05T06:44:40.631 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.631 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.631 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:40.631 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580120+0000", 2024-02-05T06:44:40.631 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.671e-05 2024-02-05T06:44:40.631 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.631 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:40.631 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.631 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.631 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.631 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:169062 8.4 8:2afada59:::benchmark_data_smithi028_105698_object79787:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:44:40.632 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.173201+0000", 2024-02-05T06:44:40.632 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 110.347212227, 2024-02-05T06:44:40.632 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4070665949999999, 2024-02-05T06:44:40.632 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:40.632 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:40.632 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:40.632 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:40.632 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:40.632 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169062 2024-02-05T06:44:40.632 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.632 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:40.632 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.633 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:40.633 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.173201+0000", 2024-02-05T06:44:40.633 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.633 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.633 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.633 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:40.633 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.173201+0000", 2024-02-05T06:44:40.633 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.633 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.633 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.633 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:40.633 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.173202+0000", 2024-02-05T06:44:40.633 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7399999999999991e-07 2024-02-05T06:44:40.633 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.633 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.634 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:40.634 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.173206+0000", 2024-02-05T06:44:40.634 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8999999999999999e-06 2024-02-05T06:44:40.634 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.634 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.634 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:40.634 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.173215+0000", 2024-02-05T06:44:40.634 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7329999999999992e-06 2024-02-05T06:44:40.634 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.634 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.634 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:40.634 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.173221+0000", 2024-02-05T06:44:40.634 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0419999999999999e-06 2024-02-05T06:44:40.634 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.634 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.635 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.635 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.173266+0000", 2024-02-05T06:44:40.635 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5256999999999997e-05 2024-02-05T06:44:40.635 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.635 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.635 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:40.635 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.173275+0000", 2024-02-05T06:44:40.635 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.003e-06 2024-02-05T06:44:40.635 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.635 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.635 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.635 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.562986+0000", 2024-02-05T06:44:40.636 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3897114660000001 2024-02-05T06:44:40.636 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.636 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.636 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:40.636 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.563100+0000", 2024-02-05T06:44:40.636 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113683 2024-02-05T06:44:40.636 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.636 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.636 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:44:40.636 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.563224+0000", 2024-02-05T06:44:40.636 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123738 2024-02-05T06:44:40.637 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.637 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.637 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:40.637 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.571213+0000", 2024-02-05T06:44:40.637 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079889839999999993 2024-02-05T06:44:40.637 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.637 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.637 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:40.637 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576145+0000", 2024-02-05T06:44:40.637 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00493196 2024-02-05T06:44:40.637 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.637 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.637 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:40.638 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580160+0000", 2024-02-05T06:44:40.638 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040155939999999999 2024-02-05T06:44:40.638 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.638 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.638 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:40.638 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580218+0000", 2024-02-05T06:44:40.638 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7779999999999999e-05 2024-02-05T06:44:40.638 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.638 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.638 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:40.638 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580268+0000", 2024-02-05T06:44:40.638 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9481000000000003e-05 2024-02-05T06:44:40.638 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.639 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:40.639 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.639 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.639 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.639 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:169071 8.4 8:2a441816:::benchmark_data_smithi028_105698_object79796:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:44:40.639 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.181972+0000", 2024-02-05T06:44:40.639 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 110.338441223, 2024-02-05T06:44:40.639 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3984312409999999, 2024-02-05T06:44:40.639 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:40.639 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:40.639 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:40.639 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:40.639 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:40.639 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169071 2024-02-05T06:44:40.639 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.639 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:40.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.640 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:40.640 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.181972+0000", 2024-02-05T06:44:40.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.640 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.640 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:40.640 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.181972+0000", 2024-02-05T06:44:40.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.640 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.640 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:40.640 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.181974+0000", 2024-02-05T06:44:40.641 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9980000000000002e-06 2024-02-05T06:44:40.641 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.641 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:40.641 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.181979+0000", 2024-02-05T06:44:40.641 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7310000000000001e-06 2024-02-05T06:44:40.641 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.641 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:40.641 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.181990+0000", 2024-02-05T06:44:40.641 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1381e-05 2024-02-05T06:44:40.641 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.641 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:40.642 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.181998+0000", 2024-02-05T06:44:40.642 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6539999999999999e-06 2024-02-05T06:44:40.642 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.642 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.642 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.642 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.182056+0000", 2024-02-05T06:44:40.642 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8276000000000002e-05 2024-02-05T06:44:40.642 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.642 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.642 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:40.642 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.182066+0000", 2024-02-05T06:44:40.642 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6199999999999994e-06 2024-02-05T06:44:40.642 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.642 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.642 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.643 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.565453+0000", 2024-02-05T06:44:40.643 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.383387401 2024-02-05T06:44:40.643 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.643 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.643 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:40.643 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.565578+0000", 2024-02-05T06:44:40.643 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012461699999999999 2024-02-05T06:44:40.643 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.643 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.643 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:44:40.643 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.565709+0000", 2024-02-05T06:44:40.643 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013105899999999999 2024-02-05T06:44:40.643 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.643 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.644 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:40.644 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576157+0000", 2024-02-05T06:44:40.644 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010448169 2024-02-05T06:44:40.644 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.644 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.644 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:40.644 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576240+0000", 2024-02-05T06:44:40.644 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3319999999999995e-05 2024-02-05T06:44:40.644 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.644 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.644 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:40.644 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580310+0000", 2024-02-05T06:44:40.645 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040695719999999996 2024-02-05T06:44:40.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.645 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:40.645 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580371+0000", 2024-02-05T06:44:40.645 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1490999999999998e-05 2024-02-05T06:44:40.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.645 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:40.645 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580403+0000", 2024-02-05T06:44:40.645 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1952000000000002e-05 2024-02-05T06:44:40.645 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.645 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:40.645 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.646 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:169074 8.4 8:23b7f380:::benchmark_data_smithi028_105698_object79799:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:44:40.646 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.185596+0000", 2024-02-05T06:44:40.646 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 110.334817159, 2024-02-05T06:44:40.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.394963838, 2024-02-05T06:44:40.646 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:40.646 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:40.646 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:40.646 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:40.646 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:40.646 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169074 2024-02-05T06:44:40.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.647 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:40.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:40.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.185596+0000", 2024-02-05T06:44:40.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:40.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.185596+0000", 2024-02-05T06:44:40.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:40.648 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.185597+0000", 2024-02-05T06:44:40.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8199999999999999e-07 2024-02-05T06:44:40.648 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.648 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:40.648 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.185600+0000", 2024-02-05T06:44:40.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6970000000000002e-06 2024-02-05T06:44:40.648 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.648 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:40.648 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.185607+0000", 2024-02-05T06:44:40.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0029999999999997e-06 2024-02-05T06:44:40.648 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.649 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:40.649 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.185611+0000", 2024-02-05T06:44:40.649 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4859999999999999e-06 2024-02-05T06:44:40.649 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.649 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.649 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.649 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.185655+0000", 2024-02-05T06:44:40.650 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4150999999999998e-05 2024-02-05T06:44:40.650 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.650 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.650 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:40.650 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.185664+0000", 2024-02-05T06:44:40.650 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7690000000000002e-06 2024-02-05T06:44:40.650 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.650 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.650 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.650 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.568350+0000", 2024-02-05T06:44:40.650 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.382685704 2024-02-05T06:44:40.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.651 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:40.651 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.568474+0000", 2024-02-05T06:44:40.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012467500000000001 2024-02-05T06:44:40.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.651 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:44:40.651 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.568613+0000", 2024-02-05T06:44:40.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013851099999999999 2024-02-05T06:44:40.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.651 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:40.651 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576163+0000", 2024-02-05T06:44:40.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0075501220000000003 2024-02-05T06:44:40.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.652 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.652 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:40.652 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576519+0000", 2024-02-05T06:44:40.652 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00035637499999999998 2024-02-05T06:44:40.652 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.652 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.652 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:40.652 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580473+0000", 2024-02-05T06:44:40.652 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0039534779999999999 2024-02-05T06:44:40.652 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.652 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.652 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:40.652 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580524+0000", 2024-02-05T06:44:40.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0788999999999998e-05 2024-02-05T06:44:40.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.653 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:40.653 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580560+0000", 2024-02-05T06:44:40.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6396000000000002e-05 2024-02-05T06:44:40.653 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.653 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:40.653 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.653 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:169145 8.4 8:2cf8555b:::benchmark_data_smithi028_105698_object79870:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:44:40.653 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.295920+0000", 2024-02-05T06:44:40.653 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 110.224493743, 2024-02-05T06:44:40.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2886338849999999, 2024-02-05T06:44:40.654 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:40.654 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:40.654 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:40.654 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:40.654 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:40.654 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169145 2024-02-05T06:44:40.654 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.654 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:40.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.654 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:40.654 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.295920+0000", 2024-02-05T06:44:40.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:40.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.295920+0000", 2024-02-05T06:44:40.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:40.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.295920+0000", 2024-02-05T06:44:40.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6700000000000003e-07 2024-02-05T06:44:40.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:40.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.295923+0000", 2024-02-05T06:44:40.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4540000000000001e-06 2024-02-05T06:44:40.656 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.656 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.656 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:40.656 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.295928+0000", 2024-02-05T06:44:40.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3530000000000002e-06 2024-02-05T06:44:40.656 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.656 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.656 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:40.656 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.295932+0000", 2024-02-05T06:44:40.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2030000000000002e-06 2024-02-05T06:44:40.656 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.657 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.657 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.295973+0000", 2024-02-05T06:44:40.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0450000000000001e-05 2024-02-05T06:44:40.657 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.657 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:40.657 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.295978+0000", 2024-02-05T06:44:40.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7620000000000001e-06 2024-02-05T06:44:40.657 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.657 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.657 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.571258+0000", 2024-02-05T06:44:40.658 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.27527938 2024-02-05T06:44:40.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.658 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:40.658 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.571371+0000", 2024-02-05T06:44:40.658 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113448 2024-02-05T06:44:40.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.658 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:44:40.658 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.571485+0000", 2024-02-05T06:44:40.658 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011325 2024-02-05T06:44:40.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:40.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576267+0000", 2024-02-05T06:44:40.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0047829350000000003 2024-02-05T06:44:40.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:40.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576587+0000", 2024-02-05T06:44:40.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00031938600000000001 2024-02-05T06:44:40.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:40.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.584465+0000", 2024-02-05T06:44:40.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078778700000000004 2024-02-05T06:44:40.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:40.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.584516+0000", 2024-02-05T06:44:40.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1570000000000003e-05 2024-02-05T06:44:40.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:40.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.584553+0000", 2024-02-05T06:44:40.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7156999999999997e-05 2024-02-05T06:44:40.660 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.660 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:40.660 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.661 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:169147 8.4 8:27ce7ef9:::benchmark_data_smithi028_105698_object79872:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:44:40.661 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.300286+0000", 2024-02-05T06:44:40.661 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 110.22012699, 2024-02-05T06:44:40.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.284371353, 2024-02-05T06:44:40.661 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:40.661 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:40.661 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:40.661 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:40.661 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:40.661 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169147 2024-02-05T06:44:40.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.662 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:40.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.662 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:40.662 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.300286+0000", 2024-02-05T06:44:40.662 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.662 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.662 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:40.662 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.300286+0000", 2024-02-05T06:44:40.662 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.662 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.662 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:40.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.300287+0000", 2024-02-05T06:44:40.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7900000000000007e-07 2024-02-05T06:44:40.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:40.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.300290+0000", 2024-02-05T06:44:40.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0970000000000002e-06 2024-02-05T06:44:40.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:40.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.300298+0000", 2024-02-05T06:44:40.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9130000000000003e-06 2024-02-05T06:44:40.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:40.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.300305+0000", 2024-02-05T06:44:40.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2069999999999998e-06 2024-02-05T06:44:40.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.300347+0000", 2024-02-05T06:44:40.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1389e-05 2024-02-05T06:44:40.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:40.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.300354+0000", 2024-02-05T06:44:40.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7549999999999997e-06 2024-02-05T06:44:40.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.573495+0000", 2024-02-05T06:44:40.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.273141654 2024-02-05T06:44:40.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:40.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.573626+0000", 2024-02-05T06:44:40.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013036499999999999 2024-02-05T06:44:40.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:44:40.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.573750+0000", 2024-02-05T06:44:40.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012482500000000001 2024-02-05T06:44:40.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:40.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576534+0000", 2024-02-05T06:44:40.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0027837539999999998 2024-02-05T06:44:40.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:40.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576637+0000", 2024-02-05T06:44:40.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010228199999999999 2024-02-05T06:44:40.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:40.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.584587+0000", 2024-02-05T06:44:40.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079500309999999998 2024-02-05T06:44:40.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:40.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.584625+0000", 2024-02-05T06:44:40.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8019000000000003e-05 2024-02-05T06:44:40.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:40.668 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.584658+0000", 2024-02-05T06:44:40.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3083000000000002e-05 2024-02-05T06:44:40.668 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.668 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:40.668 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.668 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:169148 8.4 8:250ae785:::benchmark_data_smithi028_105698_object79873:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:44:40.668 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.301004+0000", 2024-02-05T06:44:40.668 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 110.219409384, 2024-02-05T06:44:40.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.287525246, 2024-02-05T06:44:40.668 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:40.668 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:40.668 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:40.669 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:40.669 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:40.669 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169148 2024-02-05T06:44:40.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.669 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:40.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.669 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:40.669 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.301004+0000", 2024-02-05T06:44:40.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.669 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:40.669 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.301004+0000", 2024-02-05T06:44:40.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:40.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:40.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.301005+0000", 2024-02-05T06:44:40.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8400000000000003e-07 2024-02-05T06:44:40.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:40.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.301008+0000", 2024-02-05T06:44:40.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2969999999999999e-06 2024-02-05T06:44:40.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:40.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.301017+0000", 2024-02-05T06:44:40.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3749999999999992e-06 2024-02-05T06:44:40.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:40.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.301024+0000", 2024-02-05T06:44:40.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2940000000000002e-06 2024-02-05T06:44:40.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.671 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.301066+0000", 2024-02-05T06:44:40.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2660999999999997e-05 2024-02-05T06:44:40.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.671 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:40.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.301074+0000", 2024-02-05T06:44:40.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4139999999999997e-06 2024-02-05T06:44:40.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:40.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576711+0000", 2024-02-05T06:44:40.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2756368330000001 2024-02-05T06:44:40.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:40.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576823+0000", 2024-02-05T06:44:40.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000112048 2024-02-05T06:44:40.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.673 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:44:40.673 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576918+0000", 2024-02-05T06:44:40.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5593000000000004e-05 2024-02-05T06:44:40.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.673 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:40.673 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.583421+0000", 2024-02-05T06:44:40.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0065030330000000001 2024-02-05T06:44:40.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.673 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:40.673 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.583436+0000", 2024-02-05T06:44:40.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4219000000000001e-05 2024-02-05T06:44:40.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:40.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.588452+0000", 2024-02-05T06:44:40.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005016777 2024-02-05T06:44:40.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:40.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.588506+0000", 2024-02-05T06:44:40.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3804999999999997e-05 2024-02-05T06:44:40.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:40.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:40.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:40.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.588529+0000", 2024-02-05T06:44:40.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3013000000000001e-05 2024-02-05T06:44:40.675 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.675 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:40.675 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.675 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:40.675 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:40.675 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:44:40.682 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:40.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:40.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:40.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:40 smithi167 ceph-mon[32774]: 3.a scrub starts 2024-02-05T06:44:40.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:40 smithi167 ceph-mon[32774]: 3.a scrub ok 2024-02-05T06:44:40.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:40 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/785688599' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T06:44:40.784 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:40.818 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:40.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:40.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:40.921 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:40.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:40 smithi028 ceph-mon[37984]: 3.a scrub starts 2024-02-05T06:44:40.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:40 smithi028 ceph-mon[37984]: 3.a scrub ok 2024-02-05T06:44:40.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:40 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/785688599' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T06:44:40.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:40 smithi028 ceph-mon[29442]: 3.a scrub starts 2024-02-05T06:44:40.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:40 smithi028 ceph-mon[29442]: 3.a scrub ok 2024-02-05T06:44:40.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:40 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/785688599' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T06:44:40.961 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:40.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:40.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:41.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:41.099 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:41.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:41.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:41.131 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 61 16 98462 98446 100.844 64.9375 0.00927704 0.00991141 2024-02-05T06:44:41.201 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:41.238 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:41.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:41.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:41.340 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:41.357 INFO:tasks.thrashosds.thrasher:j is {'epoch': 312, 'fsid': '9801e956-c3ef-11ee-95b6-87774f69a715', 'created': '2024-02-05T06:27:59.919226+0000', 'modified': '2024-02-05T06:44:38.469111+0000', 'last_up_change': '2024-02-05T06:33:15.833573+0000', 'last_in_change': '2024-02-05T06:41:42.383087+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-05T06:31:10.441424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '142', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '129', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.180000066757202, 'score_stable': 2.180000066757202, 'optimal_score': 1, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-02-05T06:33:27.397416+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-02-05T06:33:28.090506+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-05T06:33:29.866249+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-05T06:33:31.826006+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-05T06:33:33.890835+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-02-05T06:43:28.629425+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '296', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'c19fdd5b-09f6-45d4-a519-0a938cf1f2a6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 291, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6801', 'nonce': 301279106}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6802', 'nonce': 301279106}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6804', 'nonce': 301279106}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6803', 'nonce': 301279106}]}, 'public_addr': '172.21.15.28:6801/301279106', 'cluster_addr': '172.21.15.28:6802/301279106', 'heartbeat_back_addr': '172.21.15.28:6804/301279106', 'heartbeat_front_addr': '172.21.15.28:6803/301279106', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '52245f6e-eb96-4d0d-985b-dbb6fd3ce8de', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 291, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6805', 'nonce': 1795224649}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6806', 'nonce': 1795224649}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6808', 'nonce': 1795224649}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6807', 'nonce': 1795224649}]}, 'public_addr': '172.21.15.28:6805/1795224649', 'cluster_addr': '172.21.15.28:6806/1795224649', 'heartbeat_back_addr': '172.21.15.28:6808/1795224649', 'heartbeat_front_addr': '172.21.15.28:6807/1795224649', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '1e18b630-afdf-4de9-a1a2-37bdfdd01c67', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 278, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6809', 'nonce': 1993280018}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6810', 'nonce': 1993280018}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6812', 'nonce': 1993280018}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6811', 'nonce': 1993280018}]}, 'public_addr': '172.21.15.28:6809/1993280018', 'cluster_addr': '172.21.15.28:6810/1993280018', 'heartbeat_back_addr': '172.21.15.28:6812/1993280018', 'heartbeat_front_addr': '172.21.15.28:6811/1993280018', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '3ae795af-3d59-41a6-95eb-b8447ff61282', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 291, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6813', 'nonce': 1294621398}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6814', 'nonce': 1294621398}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6816', 'nonce': 1294621398}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6815', 'nonce': 1294621398}]}, 'public_addr': '172.21.15.28:6813/1294621398', 'cluster_addr': '172.21.15.28:6814/1294621398', 'heartbeat_back_addr': '172.21.15.28:6816/1294621398', 'heartbeat_front_addr': '172.21.15.28:6815/1294621398', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'fc6bd83d-570c-48c0-964d-e09b4608bc32', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 267, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6800', 'nonce': 4052090595}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6801', 'nonce': 4052090595}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6803', 'nonce': 4052090595}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6802', 'nonce': 4052090595}]}, 'public_addr': '172.21.15.167:6800/4052090595', 'cluster_addr': '172.21.15.167:6801/4052090595', 'heartbeat_back_addr': '172.21.15.167:6803/4052090595', 'heartbeat_front_addr': '172.21.15.167:6802/4052090595', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '895bf465-2531-4d9f-bc23-1e75fd969f07', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 291, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6804', 'nonce': 1631080500}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6805', 'nonce': 1631080500}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6807', 'nonce': 1631080500}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6806', 'nonce': 1631080500}]}, 'public_addr': '172.21.15.167:6804/1631080500', 'cluster_addr': '172.21.15.167:6805/1631080500', 'heartbeat_back_addr': '172.21.15.167:6807/1631080500', 'heartbeat_front_addr': '172.21.15.167:6806/1631080500', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ce0a9300-87a4-4e37-b702-d667f168c0e0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 254, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6808', 'nonce': 1644050617}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6809', 'nonce': 1644050617}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6811', 'nonce': 1644050617}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6810', 'nonce': 1644050617}]}, 'public_addr': '172.21.15.167:6808/1644050617', 'cluster_addr': '172.21.15.167:6809/1644050617', 'heartbeat_back_addr': '172.21.15.167:6811/1644050617', 'heartbeat_front_addr': '172.21.15.167:6810/1644050617', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '0c545cbc-0873-4232-8807-4af9299ef95e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 291, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6812', 'nonce': 4031475764}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6813', 'nonce': 4031475764}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6815', 'nonce': 4031475764}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6814', 'nonce': 4031475764}]}, 'public_addr': '172.21.15.167:6812/4031475764', 'cluster_addr': '172.21.15.167:6813/4031475764', 'heartbeat_back_addr': '172.21.15.167:6815/4031475764', 'heartbeat_front_addr': '172.21.15.167:6814/4031475764', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:44:37.880876+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:44:01.333974+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:44:36.590674+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:44:08.163184+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:44:16.763698+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:44:32.888488+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:43:51.376575+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:44:03.223941+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.18', 'osds': [5, 2, 4]}, {'pgid': '5.1b', 'osds': [3, 0, 1]}], 'pg_upmap_items': [{'pgid': '3.9', 'mappings': [{'from': 4, 'to': 1}]}, {'pgid': '3.a', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '4.14', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.15', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.1c', 'mappings': [{'from': 4, 'to': 6}]}, {'pgid': '5.1e', 'mappings': [{'from': 0, 'to': 3}]}, {'pgid': '6.13', 'mappings': [{'from': 3, 'to': 1}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.28:6800/1001442173': '2024-02-06T06:34:10.127914+0000', '172.21.15.28:0/3160026720': '2024-02-06T06:34:10.127914+0000', '172.21.15.28:0/558171292': '2024-02-06T06:34:10.127914+0000', '172.21.15.28:0/2832437766': '2024-02-06T06:34:10.127914+0000', '172.21.15.28:6800/4125764599': '2024-02-06T06:28:54.105170+0000', '172.21.15.28:0/1081543653': '2024-02-06T06:28:54.105170+0000', '172.21.15.28:0/3794420554': '2024-02-06T06:28:27.553034+0000', '172.21.15.28:0/1560471317': '2024-02-06T06:34:10.127914+0000', '172.21.15.28:6800/4266601437': '2024-02-06T06:28:27.553034+0000', '172.21.15.28:0/388893831': '2024-02-06T06:28:27.553034+0000', '172.21.15.28:0/460817910': '2024-02-06T06:28:27.553034+0000', '172.21.15.28:0/2728305743': '2024-02-06T06:28:54.105170+0000', '172.21.15.28:0/3149394045': '2024-02-06T06:34:10.127914+0000', '172.21.15.28:0/3605013594': '2024-02-06T06:28:54.105170+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-05T06:44:41.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T06:44:41.373 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:41.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:41.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:41.475 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:41.509 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:41.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:41.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:41.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:41.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:41 smithi028 ceph-mon[29442]: 9.b scrub starts 2024-02-05T06:44:41.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:41 smithi028 ceph-mon[29442]: pgmap v548: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 58 MiB/s wr, 924 op/s 2024-02-05T06:44:41.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:41 smithi028 ceph-mon[29442]: 5.10 scrub starts 2024-02-05T06:44:41.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:41 smithi028 ceph-mon[29442]: 5.10 scrub ok 2024-02-05T06:44:41.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:41 smithi028 ceph-mon[29442]: 6.7 scrub starts 2024-02-05T06:44:41.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:41 smithi028 ceph-mon[29442]: 6.7 scrub ok 2024-02-05T06:44:41.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:41 smithi028 ceph-mon[29442]: 9.4 scrub ok 2024-02-05T06:44:41.656 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:41 smithi028 ceph-mon[37984]: 9.b scrub starts 2024-02-05T06:44:41.656 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:41 smithi028 ceph-mon[37984]: pgmap v548: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 58 MiB/s wr, 924 op/s 2024-02-05T06:44:41.657 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:41 smithi028 ceph-mon[37984]: 5.10 scrub starts 2024-02-05T06:44:41.657 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:41 smithi028 ceph-mon[37984]: 5.10 scrub ok 2024-02-05T06:44:41.657 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:41 smithi028 ceph-mon[37984]: 6.7 scrub starts 2024-02-05T06:44:41.657 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:41 smithi028 ceph-mon[37984]: 6.7 scrub ok 2024-02-05T06:44:41.657 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:41 smithi028 ceph-mon[37984]: 9.4 scrub ok 2024-02-05T06:44:41.662 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:41.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:41.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:41.763 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:41.767 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:44:41.776 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T06:44:41.821 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:44:41.825 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:41.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:41.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:41.865 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:44:41.908 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:44:41.927 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:41.928 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:44:41.947 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:44:41.956 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:41.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:41.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:41.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:41 smithi167 ceph-mon[32774]: 9.b scrub starts 2024-02-05T06:44:41.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:41 smithi167 ceph-mon[32774]: pgmap v548: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 58 MiB/s wr, 924 op/s 2024-02-05T06:44:41.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:41 smithi167 ceph-mon[32774]: 5.10 scrub starts 2024-02-05T06:44:41.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:41 smithi167 ceph-mon[32774]: 5.10 scrub ok 2024-02-05T06:44:41.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:41 smithi167 ceph-mon[32774]: 6.7 scrub starts 2024-02-05T06:44:41.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:41 smithi167 ceph-mon[32774]: 6.7 scrub ok 2024-02-05T06:44:41.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:41 smithi167 ceph-mon[32774]: 9.4 scrub ok 2024-02-05T06:44:41.990 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:44:42.036 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:44:42.049 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-05T06:44:42.059 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:42.085 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:44:42.090 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:44:42.090 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:44:42.090 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:44:42.091 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:44:42.091 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:44:42.091 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:44:42.091 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:44:42.091 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:44:42.110 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:42.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:42.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:42.135 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 62 15 99381 99366 100.144 57.5 0.0162212 0.00998082 2024-02-05T06:44:42.211 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:42.252 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:42.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:42.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:42.354 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:42.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:42.409 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:42.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:42.489 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:44:42.510 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:42.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:42.559 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:42.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:42.660 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:42.698 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:42.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:42.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:42.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:42.837 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:42 smithi167 ceph-mon[32774]: 1.7 scrub starts 2024-02-05T06:44:42.837 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:42 smithi167 ceph-mon[32774]: 1.7 scrub ok 2024-02-05T06:44:42.838 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:42 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:44:42.838 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:42 smithi167 ceph-mon[32774]: 6.1d scrub starts 2024-02-05T06:44:42.838 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:42 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/931215089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:44:42.838 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:42 smithi167 ceph-mon[32774]: 6.1d scrub ok 2024-02-05T06:44:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:42.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:42.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:42.943 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:42.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:42 smithi028 ceph-mon[29442]: 1.7 scrub starts 2024-02-05T06:44:42.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:42 smithi028 ceph-mon[29442]: 1.7 scrub ok 2024-02-05T06:44:42.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:42 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:44:42.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:42 smithi028 ceph-mon[29442]: 6.1d scrub starts 2024-02-05T06:44:42.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:42 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/931215089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:44:42.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:42 smithi028 ceph-mon[29442]: 6.1d scrub ok 2024-02-05T06:44:42.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:42 smithi028 ceph-mon[37984]: 1.7 scrub starts 2024-02-05T06:44:42.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:42 smithi028 ceph-mon[37984]: 1.7 scrub ok 2024-02-05T06:44:42.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:42 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:44:42.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:42 smithi028 ceph-mon[37984]: 6.1d scrub starts 2024-02-05T06:44:42.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:42 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/931215089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:44:42.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:42 smithi028 ceph-mon[37984]: 6.1d scrub ok 2024-02-05T06:44:42.950 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:44:42 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:44:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:44:42.985 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:42.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:42.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:43.087 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:43.126 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:43.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:43.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:43.131 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 63 16 100561 100545 99.724 73.6875 0.0143845 0.0100227 2024-02-05T06:44:43.229 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:43.233 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:44:42 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:44:43.271 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:43.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:43.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:43.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:43.409 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:43.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:43.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:43.511 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:43.549 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:43.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:43.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:43.651 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:43.689 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:43.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:43.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:43.763 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:43 smithi028 ceph-mon[37984]: 9.b scrub ok 2024-02-05T06:44:43.763 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:43 smithi028 ceph-mon[37984]: 6.17 scrub starts 2024-02-05T06:44:43.763 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:43 smithi028 ceph-mon[37984]: 6.17 scrub ok 2024-02-05T06:44:43.763 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:43 smithi028 ceph-mon[37984]: pgmap v549: 158 pgs: 158 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-02-05T06:44:43.763 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:43 smithi028 ceph-mon[37984]: 4.1f scrub starts 2024-02-05T06:44:43.763 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:43 smithi028 ceph-mon[37984]: 4.1f scrub ok 2024-02-05T06:44:43.763 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:43 smithi028 ceph-mon[37984]: 1.3 scrub starts 2024-02-05T06:44:43.763 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:43 smithi028 ceph-mon[37984]: 1.3 scrub ok 2024-02-05T06:44:43.764 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:43 smithi028 ceph-mon[29442]: 9.b scrub ok 2024-02-05T06:44:43.764 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:43 smithi028 ceph-mon[29442]: 6.17 scrub starts 2024-02-05T06:44:43.764 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:43 smithi028 ceph-mon[29442]: 6.17 scrub ok 2024-02-05T06:44:43.764 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:43 smithi028 ceph-mon[29442]: pgmap v549: 158 pgs: 158 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-02-05T06:44:43.764 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:43 smithi028 ceph-mon[29442]: 4.1f scrub starts 2024-02-05T06:44:43.764 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:43 smithi028 ceph-mon[29442]: 4.1f scrub ok 2024-02-05T06:44:43.764 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:43 smithi028 ceph-mon[29442]: 1.3 scrub starts 2024-02-05T06:44:43.764 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:43 smithi028 ceph-mon[29442]: 1.3 scrub ok 2024-02-05T06:44:43.791 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:43.837 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:43.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:43.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:43.939 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:43.950 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:43 smithi167 ceph-mon[32774]: 9.b scrub ok 2024-02-05T06:44:43.950 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:43 smithi167 ceph-mon[32774]: 6.17 scrub starts 2024-02-05T06:44:43.950 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:43 smithi167 ceph-mon[32774]: 6.17 scrub ok 2024-02-05T06:44:43.950 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:43 smithi167 ceph-mon[32774]: pgmap v549: 158 pgs: 158 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.4 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-02-05T06:44:43.950 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:43 smithi167 ceph-mon[32774]: 4.1f scrub starts 2024-02-05T06:44:43.950 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:43 smithi167 ceph-mon[32774]: 4.1f scrub ok 2024-02-05T06:44:43.950 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:43 smithi167 ceph-mon[32774]: 1.3 scrub starts 2024-02-05T06:44:43.950 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:43 smithi167 ceph-mon[32774]: 1.3 scrub ok 2024-02-05T06:44:43.977 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:43.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:43.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:44.079 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:44.114 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:44.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:44.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:44.132 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 64 16 101766 101750 99.3424 75.3125 0.0270207 0.0100588 2024-02-05T06:44:44.216 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:44.257 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:44.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:44.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:44.337 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:44:44.360 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:44.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:44.423 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:44.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:44.524 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:44.564 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:44.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:44.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:44.666 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:44.703 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:44:44.703 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:44:44.710 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:44.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:44.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:44.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:44 smithi028 ceph-mon[29442]: 5.3 scrub starts 2024-02-05T06:44:44.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:44 smithi028 ceph-mon[29442]: 5.3 scrub ok 2024-02-05T06:44:44.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:44 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:44:44.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:44 smithi028 ceph-mon[29442]: 3.10 scrub starts 2024-02-05T06:44:44.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:44 smithi028 ceph-mon[29442]: 3.10 scrub ok 2024-02-05T06:44:44.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:44 smithi028 ceph-mon[29442]: 5.c scrub starts 2024-02-05T06:44:44.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:44 smithi028 ceph-mon[29442]: 5.c scrub ok 2024-02-05T06:44:44.809 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:44 smithi028 ceph-mon[37984]: 5.3 scrub starts 2024-02-05T06:44:44.810 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:44 smithi028 ceph-mon[37984]: 5.3 scrub ok 2024-02-05T06:44:44.810 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:44 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:44:44.810 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:44 smithi028 ceph-mon[37984]: 3.10 scrub starts 2024-02-05T06:44:44.810 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:44 smithi028 ceph-mon[37984]: 3.10 scrub ok 2024-02-05T06:44:44.810 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:44 smithi028 ceph-mon[37984]: 5.c scrub starts 2024-02-05T06:44:44.810 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:44 smithi028 ceph-mon[37984]: 5.c scrub ok 2024-02-05T06:44:44.812 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:44.847 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:44.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:44.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:44.949 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:44.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:44 smithi167 ceph-mon[32774]: 5.3 scrub starts 2024-02-05T06:44:44.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:44 smithi167 ceph-mon[32774]: 5.3 scrub ok 2024-02-05T06:44:44.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:44 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:44:44.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:44 smithi167 ceph-mon[32774]: 3.10 scrub starts 2024-02-05T06:44:44.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:44 smithi167 ceph-mon[32774]: 3.10 scrub ok 2024-02-05T06:44:44.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:44 smithi167 ceph-mon[32774]: 5.c scrub starts 2024-02-05T06:44:44.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:44 smithi167 ceph-mon[32774]: 5.c scrub ok 2024-02-05T06:44:44.984 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:44.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:44.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:45.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:45.131 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 65 16 102925 102909 98.9285 72.4375 0.0131744 0.0101027 2024-02-05T06:44:45.132 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:45.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:45.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:45.234 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:45.272 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:45.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:45.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:45.374 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:45.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:45.423 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:45.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:45.524 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:45.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:45.571 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:45.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:45.572 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-05T06:44:45.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:45.712 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:45.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:45.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:45.714 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:44:45.715 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:45 smithi028 ceph-mon[37984]: 4.12 scrub starts 2024-02-05T06:44:45.715 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:45 smithi028 ceph-mon[37984]: 4.12 scrub ok 2024-02-05T06:44:45.715 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:45 smithi028 ceph-mon[37984]: 6.1e scrub starts 2024-02-05T06:44:45.715 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:45 smithi028 ceph-mon[37984]: 6.1e scrub ok 2024-02-05T06:44:45.715 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:45 smithi028 ceph-mon[37984]: pgmap v550: 158 pgs: 158 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-02-05T06:44:45.715 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:45 smithi028 ceph-mon[37984]: 5.9 scrub starts 2024-02-05T06:44:45.715 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:45 smithi028 ceph-mon[37984]: 5.9 scrub ok 2024-02-05T06:44:45.715 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:45 smithi028 ceph-mon[37984]: 5.6 scrub starts 2024-02-05T06:44:45.715 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:45 smithi028 ceph-mon[37984]: 5.6 scrub ok 2024-02-05T06:44:45.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:45 smithi028 ceph-mon[29442]: 4.12 scrub starts 2024-02-05T06:44:45.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:45 smithi028 ceph-mon[29442]: 4.12 scrub ok 2024-02-05T06:44:45.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:45 smithi028 ceph-mon[29442]: 6.1e scrub starts 2024-02-05T06:44:45.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:45 smithi028 ceph-mon[29442]: 6.1e scrub ok 2024-02-05T06:44:45.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:45 smithi028 ceph-mon[29442]: pgmap v550: 158 pgs: 158 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-02-05T06:44:45.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:45 smithi028 ceph-mon[29442]: 5.9 scrub starts 2024-02-05T06:44:45.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:45 smithi028 ceph-mon[29442]: 5.9 scrub ok 2024-02-05T06:44:45.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:45 smithi028 ceph-mon[29442]: 5.6 scrub starts 2024-02-05T06:44:45.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:45 smithi028 ceph-mon[29442]: 5.6 scrub ok 2024-02-05T06:44:45.723 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-05T06:44:45.808 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:44:45.814 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:45.850 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:45.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:45.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:45.952 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:45.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:45 smithi167 ceph-mon[32774]: 4.12 scrub starts 2024-02-05T06:44:45.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:45 smithi167 ceph-mon[32774]: 4.12 scrub ok 2024-02-05T06:44:45.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:45 smithi167 ceph-mon[32774]: 6.1e scrub starts 2024-02-05T06:44:45.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:45 smithi167 ceph-mon[32774]: 6.1e scrub ok 2024-02-05T06:44:45.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:45 smithi167 ceph-mon[32774]: pgmap v550: 158 pgs: 158 active+clean; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-02-05T06:44:45.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:45 smithi167 ceph-mon[32774]: 5.9 scrub starts 2024-02-05T06:44:45.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:45 smithi167 ceph-mon[32774]: 5.9 scrub ok 2024-02-05T06:44:45.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:45 smithi167 ceph-mon[32774]: 5.6 scrub starts 2024-02-05T06:44:45.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:45 smithi167 ceph-mon[32774]: 5.6 scrub ok 2024-02-05T06:44:45.990 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:44:46.002 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:46.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:46.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:46.104 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:46.132 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 66 16 104057 104041 98.5003 70.75 0.0114191 0.0101474 2024-02-05T06:44:46.151 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:46.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:46.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:46.252 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:46.297 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:46.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:46.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:46.399 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:46.438 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:46.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:46.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:46.540 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:46.604 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":550,"stamp":"2024-02-05T06:44:44.372082+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6494558556,"num_objects":99288,"num_object_clones":0,"num_object_copies":297864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99288,"num_whiteouts":0,"num_read":1388,"num_read_kb":1140,"num_write":99812,"num_write_kb":6341805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":416,"num_bytes_recovered":1586799,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80766,"ondisk_log_size":80766,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":22044224,"kb_used_data":18918268,"kb_used_omap":12,"kb_used_meta":3125875,"kb_avail":727916992,"statfs":{"total":767960285184,"available":745386999808,"internally_reserved":0,"allocated":19372306432,"data_stored":19367169588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":3200896561},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,8,6,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":554827776,"num_objects":8466,"num_object_clones":0,"num_object_copies":25398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8466,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8466,"num_write_kb":541824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002512"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":434,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.487490+0000","last_change":"2024-02-05T06:42:34.762437+0000","last_active":"2024-02-05T06:44:38.487490+0000","last_peered":"2024-02-05T06:44:38.487490+0000","last_clean":"2024-02-05T06:44:38.487490+0000","last_became_active":"2024-02-05T06:41:43.412603+0000","last_became_peered":"2024-02-05T06:41:43.412603+0000","last_unstale":"2024-02-05T06:44:38.487490+0000","last_undegraded":"2024-02-05T06:44:38.487490+0000","last_fullsized":"2024-02-05T06:44:38.487490+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:34.762324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:42:34.762324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.044312+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":554,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.486217+0000","last_change":"2024-02-05T06:43:36.504759+0000","last_active":"2024-02-05T06:44:38.486217+0000","last_peered":"2024-02-05T06:44:38.486217+0000","last_clean":"2024-02-05T06:44:38.486217+0000","last_became_active":"2024-02-05T06:41:43.420342+0000","last_became_peered":"2024-02-05T06:41:43.420342+0000","last_unstale":"2024-02-05T06:44:38.486217+0000","last_undegraded":"2024-02-05T06:44:38.486217+0000","last_fullsized":"2024-02-05T06:44:38.486217+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:36.504651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:06.525012+0000","last_clean_scrub_stamp":"2024-02-05T06:43:36.504651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:58.084697+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":544,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.485545+0000","last_change":"2024-02-05T06:43:40.001642+0000","last_active":"2024-02-05T06:44:38.485545+0000","last_peered":"2024-02-05T06:44:38.485545+0000","last_clean":"2024-02-05T06:44:38.485545+0000","last_became_active":"2024-02-05T06:41:43.421315+0000","last_became_peered":"2024-02-05T06:41:43.421315+0000","last_unstale":"2024-02-05T06:44:38.485545+0000","last_undegraded":"2024-02-05T06:44:38.485545+0000","last_fullsized":"2024-02-05T06:44:38.485545+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:40.001568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:43:40.001568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:58.362550+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"295'19","reported_seq":447,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.487364+0000","last_change":"2024-02-05T06:43:52.801066+0000","last_active":"2024-02-05T06:44:38.487364+0000","last_peered":"2024-02-05T06:44:38.487364+0000","last_clean":"2024-02-05T06:44:38.487364+0000","last_became_active":"2024-02-05T06:36:54.959330+0000","last_became_peered":"2024-02-05T06:36:54.959330+0000","last_unstale":"2024-02-05T06:44:38.487364+0000","last_undegraded":"2024-02-05T06:44:38.487364+0000","last_fullsized":"2024-02-05T06:44:38.487364+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'19","last_scrub_stamp":"2024-02-05T06:43:52.800915+0000","last_deep_scrub":"247'13","last_deep_scrub_stamp":"2024-02-05T06:41:42.473424+0000","last_clean_scrub_stamp":"2024-02-05T06:43:52.800915+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:13.160739+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":530,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.495429+0000","last_change":"2024-02-05T06:42:40.263147+0000","last_active":"2024-02-05T06:44:38.495429+0000","last_peered":"2024-02-05T06:44:38.495429+0000","last_clean":"2024-02-05T06:44:38.495429+0000","last_became_active":"2024-02-05T06:40:27.077485+0000","last_became_peered":"2024-02-05T06:40:27.077485+0000","last_unstale":"2024-02-05T06:44:38.495429+0000","last_undegraded":"2024-02-05T06:44:38.495429+0000","last_fullsized":"2024-02-05T06:44:38.495429+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:40.262880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:42:40.262880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":416,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.487450+0000","last_change":"2024-02-05T06:41:47.546569+0000","last_active":"2024-02-05T06:44:38.487450+0000","last_peered":"2024-02-05T06:44:38.487450+0000","last_clean":"2024-02-05T06:44:38.487450+0000","last_became_active":"2024-02-05T06:36:54.959640+0000","last_became_peered":"2024-02-05T06:36:54.959640+0000","last_unstale":"2024-02-05T06:44:38.487450+0000","last_undegraded":"2024-02-05T06:44:38.487450+0000","last_fullsized":"2024-02-05T06:44:38.487450+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:41:47.546464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:41:47.546464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.044261+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":539,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.495631+0000","last_change":"2024-02-05T06:43:53.697132+0000","last_active":"2024-02-05T06:44:38.495631+0000","last_peered":"2024-02-05T06:44:38.495631+0000","last_clean":"2024-02-05T06:44:38.495631+0000","last_became_active":"2024-02-05T06:33:29.553864+0000","last_became_peered":"2024-02-05T06:33:29.553864+0000","last_unstale":"2024-02-05T06:44:38.495631+0000","last_undegraded":"2024-02-05T06:44:38.495631+0000","last_fullsized":"2024-02-05T06:44:38.495631+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:53.696997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:43:53.696997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:58.140971+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"295'34","reported_seq":584,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.496352+0000","last_change":"2024-02-05T06:43:56.716084+0000","last_active":"2024-02-05T06:44:38.496352+0000","last_peered":"2024-02-05T06:44:38.496352+0000","last_clean":"2024-02-05T06:44:38.496352+0000","last_became_active":"2024-02-05T06:33:31.572544+0000","last_became_peered":"2024-02-05T06:33:31.572544+0000","last_unstale":"2024-02-05T06:44:38.496352+0000","last_undegraded":"2024-02-05T06:44:38.496352+0000","last_fullsized":"2024-02-05T06:44:38.496352+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'34","last_scrub_stamp":"2024-02-05T06:43:56.715959+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:39:35.433539+0000","last_clean_scrub_stamp":"2024-02-05T06:43:56.715959+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:08.901916+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":531,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.486853+0000","last_change":"2024-02-05T06:42:52.920370+0000","last_active":"2024-02-05T06:44:38.486853+0000","last_peered":"2024-02-05T06:44:38.486853+0000","last_clean":"2024-02-05T06:44:38.486853+0000","last_became_active":"2024-02-05T06:40:27.068888+0000","last_became_peered":"2024-02-05T06:40:27.068888+0000","last_unstale":"2024-02-05T06:44:38.486853+0000","last_undegraded":"2024-02-05T06:44:38.486853+0000","last_fullsized":"2024-02-05T06:44:38.486853+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:52.920236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:42:52.920236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:41.954091+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":95,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.495521+0000","last_change":"2024-02-05T06:42:46.353608+0000","last_active":"2024-02-05T06:44:38.495521+0000","last_peered":"2024-02-05T06:44:38.495521+0000","last_clean":"2024-02-05T06:44:38.495521+0000","last_became_active":"2024-02-05T06:42:33.129959+0000","last_became_peered":"2024-02-05T06:42:33.129959+0000","last_unstale":"2024-02-05T06:44:38.495521+0000","last_undegraded":"2024-02-05T06:44:38.495521+0000","last_fullsized":"2024-02-05T06:44:38.495521+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:46.351296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:42:46.351296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":148,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.493372+0000","last_change":"2024-02-05T06:43:56.453639+0000","last_active":"2024-02-05T06:44:38.493372+0000","last_peered":"2024-02-05T06:44:38.493372+0000","last_clean":"2024-02-05T06:44:38.493372+0000","last_became_active":"2024-02-05T06:41:43.403746+0000","last_became_peered":"2024-02-05T06:41:43.403746+0000","last_unstale":"2024-02-05T06:44:38.493372+0000","last_undegraded":"2024-02-05T06:44:38.493372+0000","last_fullsized":"2024-02-05T06:44:38.493372+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:56.453478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:44.081318+0000","last_clean_scrub_stamp":"2024-02-05T06:43:56.453478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:25.000882+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"295'19","reported_seq":443,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.487320+0000","last_change":"2024-02-05T06:43:49.843729+0000","last_active":"2024-02-05T06:44:38.487320+0000","last_peered":"2024-02-05T06:44:38.487320+0000","last_clean":"2024-02-05T06:44:38.487320+0000","last_became_active":"2024-02-05T06:36:54.969454+0000","last_became_peered":"2024-02-05T06:36:54.969454+0000","last_unstale":"2024-02-05T06:44:38.487320+0000","last_undegraded":"2024-02-05T06:44:38.487320+0000","last_fullsized":"2024-02-05T06:44:38.487320+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'19","last_scrub_stamp":"2024-02-05T06:43:49.843581+0000","last_deep_scrub":"92'13","last_deep_scrub_stamp":"2024-02-05T06:34:46.542893+0000","last_clean_scrub_stamp":"2024-02-05T06:43:49.843581+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:52.860608+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":148,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.493713+0000","last_change":"2024-02-05T06:43:53.476980+0000","last_active":"2024-02-05T06:44:38.493713+0000","last_peered":"2024-02-05T06:44:38.493713+0000","last_clean":"2024-02-05T06:44:38.493713+0000","last_became_active":"2024-02-05T06:41:43.407404+0000","last_became_peered":"2024-02-05T06:41:43.407404+0000","last_unstale":"2024-02-05T06:44:38.493713+0000","last_undegraded":"2024-02-05T06:44:38.493713+0000","last_fullsized":"2024-02-05T06:44:38.493713+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:53.476863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:21.998850+0000","last_clean_scrub_stamp":"2024-02-05T06:43:53.476863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:58.879515+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":240,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.494398+0000","last_change":"2024-02-05T06:43:54.540922+0000","last_active":"2024-02-05T06:44:38.494398+0000","last_peered":"2024-02-05T06:44:38.494398+0000","last_clean":"2024-02-05T06:44:38.494398+0000","last_became_active":"2024-02-05T06:40:27.071758+0000","last_became_peered":"2024-02-05T06:40:27.071758+0000","last_unstale":"2024-02-05T06:44:38.494398+0000","last_undegraded":"2024-02-05T06:44:38.494398+0000","last_fullsized":"2024-02-05T06:44:38.494398+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:54.540821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:54.540821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:21.286238+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":148,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.494921+0000","last_change":"2024-02-05T06:43:34.438758+0000","last_active":"2024-02-05T06:44:38.494921+0000","last_peered":"2024-02-05T06:44:38.494921+0000","last_clean":"2024-02-05T06:44:38.494921+0000","last_became_active":"2024-02-05T06:41:43.408428+0000","last_became_peered":"2024-02-05T06:41:43.408428+0000","last_unstale":"2024-02-05T06:44:38.494921+0000","last_undegraded":"2024-02-05T06:44:38.494921+0000","last_fullsized":"2024-02-05T06:44:38.494921+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:34.438631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:50.055245+0000","last_clean_scrub_stamp":"2024-02-05T06:43:34.438631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.429736+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"295'7","reported_seq":425,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.487673+0000","last_change":"2024-02-05T06:41:48.543907+0000","last_active":"2024-02-05T06:44:38.487673+0000","last_peered":"2024-02-05T06:44:38.487673+0000","last_clean":"2024-02-05T06:44:38.487673+0000","last_became_active":"2024-02-05T06:36:54.954385+0000","last_became_peered":"2024-02-05T06:36:54.954385+0000","last_unstale":"2024-02-05T06:44:38.487673+0000","last_undegraded":"2024-02-05T06:44:38.487673+0000","last_fullsized":"2024-02-05T06:44:38.487673+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-02-05T06:41:48.543648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:41:48.543648+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.044278+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":428,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.488247+0000","last_change":"2024-02-05T06:41:43.412335+0000","last_active":"2024-02-05T06:44:38.488247+0000","last_peered":"2024-02-05T06:44:38.488247+0000","last_clean":"2024-02-05T06:44:38.488247+0000","last_became_active":"2024-02-05T06:41:43.411344+0000","last_became_peered":"2024-02-05T06:41:43.411344+0000","last_unstale":"2024-02-05T06:44:38.488247+0000","last_undegraded":"2024-02-05T06:44:38.488247+0000","last_fullsized":"2024-02-05T06:44:38.488247+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:41:39.552367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:41:39.552367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.044190+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":542,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.488619+0000","last_change":"2024-02-05T06:43:37.552250+0000","last_active":"2024-02-05T06:44:38.488619+0000","last_peered":"2024-02-05T06:44:38.488619+0000","last_clean":"2024-02-05T06:44:38.488619+0000","last_became_active":"2024-02-05T06:36:54.958069+0000","last_became_peered":"2024-02-05T06:36:54.958069+0000","last_unstale":"2024-02-05T06:44:38.488619+0000","last_undegraded":"2024-02-05T06:44:38.488619+0000","last_fullsized":"2024-02-05T06:44:38.488619+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:37.552166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:37.552166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:56.420046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"281'2","reported_seq":578,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.486358+0000","last_change":"2024-02-05T06:42:58.933944+0000","last_active":"2024-02-05T06:44:38.486358+0000","last_peered":"2024-02-05T06:44:38.486358+0000","last_clean":"2024-02-05T06:44:38.486358+0000","last_became_active":"2024-02-05T06:40:27.073360+0000","last_became_peered":"2024-02-05T06:40:27.073360+0000","last_unstale":"2024-02-05T06:44:38.486358+0000","last_undegraded":"2024-02-05T06:44:38.486358+0000","last_fullsized":"2024-02-05T06:44:38.486358+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'2","last_scrub_stamp":"2024-02-05T06:42:58.933820+0000","last_deep_scrub":"281'2","last_deep_scrub_stamp":"2024-02-05T06:42:58.933820+0000","last_clean_scrub_stamp":"2024-02-05T06:42:58.933820+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:41.954144+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"295'34","reported_seq":606,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.497231+0000","last_change":"2024-02-05T06:43:57.470734+0000","last_active":"2024-02-05T06:44:38.497231+0000","last_peered":"2024-02-05T06:44:38.497231+0000","last_clean":"2024-02-05T06:44:38.497231+0000","last_became_active":"2024-02-05T06:33:31.569763+0000","last_became_peered":"2024-02-05T06:33:31.569763+0000","last_unstale":"2024-02-05T06:44:38.497231+0000","last_undegraded":"2024-02-05T06:44:38.497231+0000","last_fullsized":"2024-02-05T06:44:38.497231+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'34","last_scrub_stamp":"2024-02-05T06:43:57.470583+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:34:35.234636+0000","last_clean_scrub_stamp":"2024-02-05T06:43:57.470583+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:14.633396+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":531,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.496160+0000","last_change":"2024-02-05T06:43:38.524343+0000","last_active":"2024-02-05T06:44:38.496160+0000","last_peered":"2024-02-05T06:44:38.496160+0000","last_clean":"2024-02-05T06:44:38.496160+0000","last_became_active":"2024-02-05T06:41:43.411442+0000","last_became_peered":"2024-02-05T06:41:43.411442+0000","last_unstale":"2024-02-05T06:44:38.496160+0000","last_undegraded":"2024-02-05T06:44:38.496160+0000","last_fullsized":"2024-02-05T06:44:38.496160+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:38.524236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:43:38.524236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:51.142093+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":312,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.492366+0000","last_change":"2024-02-05T06:43:54.234456+0000","last_active":"2024-02-05T06:44:38.492366+0000","last_peered":"2024-02-05T06:44:38.492366+0000","last_clean":"2024-02-05T06:44:38.492366+0000","last_became_active":"2024-02-05T06:39:13.248618+0000","last_became_peered":"2024-02-05T06:39:13.248618+0000","last_unstale":"2024-02-05T06:44:38.492366+0000","last_undegraded":"2024-02-05T06:44:38.492366+0000","last_fullsized":"2024-02-05T06:44:38.492366+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:54.234345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:54.234345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:55.938687+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":416,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.487236+0000","last_change":"2024-02-05T06:42:57.721396+0000","last_active":"2024-02-05T06:44:38.487236+0000","last_peered":"2024-02-05T06:44:38.487236+0000","last_clean":"2024-02-05T06:44:38.487236+0000","last_became_active":"2024-02-05T06:36:54.959070+0000","last_became_peered":"2024-02-05T06:36:54.959070+0000","last_unstale":"2024-02-05T06:44:38.487236+0000","last_undegraded":"2024-02-05T06:44:38.487236+0000","last_fullsized":"2024-02-05T06:44:38.487236+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:57.721280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:57.721280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.044423+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"295'18","reported_seq":612,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.495985+0000","last_change":"2024-02-05T06:42:44.182931+0000","last_active":"2024-02-05T06:44:38.495985+0000","last_peered":"2024-02-05T06:44:38.495985+0000","last_clean":"2024-02-05T06:44:38.495985+0000","last_became_active":"2024-02-05T06:36:54.954880+0000","last_became_peered":"2024-02-05T06:36:54.954880+0000","last_unstale":"2024-02-05T06:44:38.495985+0000","last_undegraded":"2024-02-05T06:44:38.495985+0000","last_fullsized":"2024-02-05T06:44:38.495985+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:42:44.182785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:42:44.182785+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"73'1","reported_seq":246,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.493360+0000","last_change":"2024-02-05T06:43:41.472748+0000","last_active":"2024-02-05T06:44:38.493360+0000","last_peered":"2024-02-05T06:44:38.493360+0000","last_clean":"2024-02-05T06:44:38.493360+0000","last_became_active":"2024-02-05T06:41:43.418640+0000","last_became_peered":"2024-02-05T06:41:43.418640+0000","last_unstale":"2024-02-05T06:44:38.493360+0000","last_undegraded":"2024-02-05T06:44:38.493360+0000","last_fullsized":"2024-02-05T06:44:38.493360+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-02-05T06:43:41.472635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:43:41.472635+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:54.907618+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"208'12","reported_seq":729,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.493475+0000","last_change":"2024-02-05T06:43:43.224045+0000","last_active":"2024-02-05T06:44:38.493475+0000","last_peered":"2024-02-05T06:44:38.493475+0000","last_clean":"2024-02-05T06:44:38.493475+0000","last_became_active":"2024-02-05T06:40:27.063399+0000","last_became_peered":"2024-02-05T06:40:27.063399+0000","last_unstale":"2024-02-05T06:44:38.493475+0000","last_undegraded":"2024-02-05T06:44:38.493475+0000","last_fullsized":"2024-02-05T06:44:38.493475+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'12","last_scrub_stamp":"2024-02-05T06:43:43.223933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:43.223933+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:53.049215+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":567,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.486126+0000","last_change":"2024-02-05T06:44:04.592727+0000","last_active":"2024-02-05T06:44:38.486126+0000","last_peered":"2024-02-05T06:44:38.486126+0000","last_clean":"2024-02-05T06:44:38.486126+0000","last_became_active":"2024-02-05T06:36:54.958414+0000","last_became_peered":"2024-02-05T06:36:54.958414+0000","last_unstale":"2024-02-05T06:44:38.486126+0000","last_undegraded":"2024-02-05T06:44:38.486126+0000","last_fullsized":"2024-02-05T06:44:38.486126+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:44:04.592594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:44:04.592594+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:23.618630+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"295'25","reported_seq":607,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.485353+0000","last_change":"2024-02-05T06:43:46.110585+0000","last_active":"2024-02-05T06:44:38.485353+0000","last_peered":"2024-02-05T06:44:38.485353+0000","last_clean":"2024-02-05T06:44:38.485353+0000","last_became_active":"2024-02-05T06:40:27.100610+0000","last_became_peered":"2024-02-05T06:40:27.100610+0000","last_unstale":"2024-02-05T06:44:38.485353+0000","last_undegraded":"2024-02-05T06:44:38.485353+0000","last_fullsized":"2024-02-05T06:44:38.485353+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'25","last_scrub_stamp":"2024-02-05T06:43:46.110443+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-02-05T06:34:50.718768+0000","last_clean_scrub_stamp":"2024-02-05T06:43:46.110443+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:54.008936+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":558,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:41.760012+0000","last_change":"2024-02-05T06:44:41.760057+0000","last_active":"2024-02-05T06:44:41.760012+0000","last_peered":"2024-02-05T06:44:41.760012+0000","last_clean":"2024-02-05T06:44:41.760012+0000","last_became_active":"2024-02-05T06:41:43.414638+0000","last_became_peered":"2024-02-05T06:41:43.414638+0000","last_unstale":"2024-02-05T06:44:41.760012+0000","last_undegraded":"2024-02-05T06:44:41.760012+0000","last_fullsized":"2024-02-05T06:44:41.760012+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:41.759930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:44:41.759930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:07.882723+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":446,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.495376+0000","last_change":"2024-02-05T06:42:58.394440+0000","last_active":"2024-02-05T06:44:38.495376+0000","last_peered":"2024-02-05T06:44:38.495376+0000","last_clean":"2024-02-05T06:44:38.495376+0000","last_became_active":"2024-02-05T06:40:27.097169+0000","last_became_peered":"2024-02-05T06:40:27.097169+0000","last_unstale":"2024-02-05T06:44:38.495376+0000","last_undegraded":"2024-02-05T06:44:38.495376+0000","last_fullsized":"2024-02-05T06:44:38.495376+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:58.394289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:08.713894+0000","last_clean_scrub_stamp":"2024-02-05T06:42:58.394289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":148,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.491994+0000","last_change":"2024-02-05T06:43:44.345193+0000","last_active":"2024-02-05T06:44:38.491994+0000","last_peered":"2024-02-05T06:44:38.491994+0000","last_clean":"2024-02-05T06:44:38.491994+0000","last_became_active":"2024-02-05T06:41:43.411043+0000","last_became_peered":"2024-02-05T06:41:43.411043+0000","last_unstale":"2024-02-05T06:44:38.491994+0000","last_undegraded":"2024-02-05T06:44:38.491994+0000","last_fullsized":"2024-02-05T06:44:38.491994+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:44.345092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:43:44.345092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:05.600762+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"295'24","reported_seq":617,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.493629+0000","last_change":"2024-02-05T06:43:03.221573+0000","last_active":"2024-02-05T06:44:38.493629+0000","last_peered":"2024-02-05T06:44:38.493629+0000","last_clean":"2024-02-05T06:44:38.493629+0000","last_became_active":"2024-02-05T06:41:43.419769+0000","last_became_peered":"2024-02-05T06:41:43.419769+0000","last_unstale":"2024-02-05T06:44:38.493629+0000","last_undegraded":"2024-02-05T06:44:38.493629+0000","last_fullsized":"2024-02-05T06:44:38.493629+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'16","last_scrub_stamp":"2024-02-05T06:43:03.221466+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:43:03.221466+0000","last_clean_scrub_stamp":"2024-02-05T06:43:03.221466+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:39.466232+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":240,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.499744+0000","last_change":"2024-02-05T06:43:48.512794+0000","last_active":"2024-02-05T06:44:38.499744+0000","last_peered":"2024-02-05T06:44:38.499744+0000","last_clean":"2024-02-05T06:44:38.499744+0000","last_became_active":"2024-02-05T06:40:27.098348+0000","last_became_peered":"2024-02-05T06:40:27.098348+0000","last_unstale":"2024-02-05T06:44:38.499744+0000","last_undegraded":"2024-02-05T06:44:38.499744+0000","last_fullsized":"2024-02-05T06:44:38.499744+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:48.512673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:48.512673+0000","last_clean_scrub_stamp":"2024-02-05T06:43:48.512673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:56.039239+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":569,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.493828+0000","last_change":"2024-02-05T06:44:01.297159+0000","last_active":"2024-02-05T06:44:38.493828+0000","last_peered":"2024-02-05T06:44:38.493828+0000","last_clean":"2024-02-05T06:44:38.493828+0000","last_became_active":"2024-02-05T06:40:27.060112+0000","last_became_peered":"2024-02-05T06:40:27.060112+0000","last_unstale":"2024-02-05T06:44:38.493828+0000","last_undegraded":"2024-02-05T06:44:38.493828+0000","last_fullsized":"2024-02-05T06:44:38.493828+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:01.297081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:44:01.297081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:14.265032+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":543,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.485736+0000","last_change":"2024-02-05T06:43:49.118084+0000","last_active":"2024-02-05T06:44:38.485736+0000","last_peered":"2024-02-05T06:44:38.485736+0000","last_clean":"2024-02-05T06:44:38.485736+0000","last_became_active":"2024-02-05T06:41:43.424459+0000","last_became_peered":"2024-02-05T06:41:43.424459+0000","last_unstale":"2024-02-05T06:44:38.485736+0000","last_undegraded":"2024-02-05T06:44:38.485736+0000","last_fullsized":"2024-02-05T06:44:38.485736+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:49.117949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:49.117949+0000","last_clean_scrub_stamp":"2024-02-05T06:43:49.117949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:17.641091+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"295'8","reported_seq":538,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.494526+0000","last_change":"2024-02-05T06:43:51.672673+0000","last_active":"2024-02-05T06:44:38.494526+0000","last_peered":"2024-02-05T06:44:38.494526+0000","last_clean":"2024-02-05T06:44:38.494526+0000","last_became_active":"2024-02-05T06:33:31.573410+0000","last_became_peered":"2024-02-05T06:33:31.573410+0000","last_unstale":"2024-02-05T06:44:38.494526+0000","last_undegraded":"2024-02-05T06:44:38.494526+0000","last_fullsized":"2024-02-05T06:44:38.494526+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'8","last_scrub_stamp":"2024-02-05T06:43:51.672524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:51.672524+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:06.073121+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":305,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.488453+0000","last_change":"2024-02-05T06:43:59.635993+0000","last_active":"2024-02-05T06:44:38.488453+0000","last_peered":"2024-02-05T06:44:38.488453+0000","last_clean":"2024-02-05T06:44:38.488453+0000","last_became_active":"2024-02-05T06:39:13.245684+0000","last_became_peered":"2024-02-05T06:39:13.245684+0000","last_unstale":"2024-02-05T06:44:38.488453+0000","last_undegraded":"2024-02-05T06:44:38.488453+0000","last_fullsized":"2024-02-05T06:44:38.488453+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:59.635733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:43:59.635733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:12.882226+0000","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":240,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:40.456560+0000","last_change":"2024-02-05T06:44:40.456593+0000","last_active":"2024-02-05T06:44:40.456560+0000","last_peered":"2024-02-05T06:44:40.456560+0000","last_clean":"2024-02-05T06:44:40.456560+0000","last_became_active":"2024-02-05T06:40:27.062975+0000","last_became_peered":"2024-02-05T06:40:27.062975+0000","last_unstale":"2024-02-05T06:44:40.456560+0000","last_undegraded":"2024-02-05T06:44:40.456560+0000","last_fullsized":"2024-02-05T06:44:40.456560+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:40.456474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:44:40.456474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:04.785266+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":148,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.492688+0000","last_change":"2024-02-05T06:42:33.724047+0000","last_active":"2024-02-05T06:44:38.492688+0000","last_peered":"2024-02-05T06:44:38.492688+0000","last_clean":"2024-02-05T06:44:38.492688+0000","last_became_active":"2024-02-05T06:41:43.412654+0000","last_became_peered":"2024-02-05T06:41:43.412654+0000","last_unstale":"2024-02-05T06:44:38.492688+0000","last_undegraded":"2024-02-05T06:44:38.492688+0000","last_fullsized":"2024-02-05T06:44:38.492688+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:33.723941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:33.723941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"295'24","reported_seq":616,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.496345+0000","last_change":"2024-02-05T06:43:39.456286+0000","last_active":"2024-02-05T06:44:38.496345+0000","last_peered":"2024-02-05T06:44:38.496345+0000","last_clean":"2024-02-05T06:44:38.496345+0000","last_became_active":"2024-02-05T06:40:27.080906+0000","last_became_peered":"2024-02-05T06:40:27.080906+0000","last_unstale":"2024-02-05T06:44:38.496345+0000","last_undegraded":"2024-02-05T06:44:38.496345+0000","last_fullsized":"2024-02-05T06:44:38.496345+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:43:39.456190+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-02-05T06:35:22.938675+0000","last_clean_scrub_stamp":"2024-02-05T06:43:39.456190+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:57.044088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"73'1","reported_seq":540,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.484977+0000","last_change":"2024-02-05T06:42:54.980648+0000","last_active":"2024-02-05T06:44:38.484977+0000","last_peered":"2024-02-05T06:44:38.484977+0000","last_clean":"2024-02-05T06:44:38.484977+0000","last_became_active":"2024-02-05T06:41:43.409761+0000","last_became_peered":"2024-02-05T06:41:43.409761+0000","last_unstale":"2024-02-05T06:44:38.484977+0000","last_undegraded":"2024-02-05T06:44:38.484977+0000","last_fullsized":"2024-02-05T06:44:38.484977+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-02-05T06:42:54.980529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:42:54.980529+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:41.954213+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":522,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.495344+0000","last_change":"2024-02-05T06:43:58.672945+0000","last_active":"2024-02-05T06:44:38.495344+0000","last_peered":"2024-02-05T06:44:38.495344+0000","last_clean":"2024-02-05T06:44:38.495344+0000","last_became_active":"2024-02-05T06:33:33.593901+0000","last_became_peered":"2024-02-05T06:33:33.593901+0000","last_unstale":"2024-02-05T06:44:38.495344+0000","last_undegraded":"2024-02-05T06:44:38.495344+0000","last_fullsized":"2024-02-05T06:44:38.495344+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:58.672808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:58.672808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:02.151514+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":240,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.495459+0000","last_change":"2024-02-05T06:44:00.501883+0000","last_active":"2024-02-05T06:44:38.495459+0000","last_peered":"2024-02-05T06:44:38.495459+0000","last_clean":"2024-02-05T06:44:38.495459+0000","last_became_active":"2024-02-05T06:40:27.077701+0000","last_became_peered":"2024-02-05T06:40:27.077701+0000","last_unstale":"2024-02-05T06:44:38.495459+0000","last_undegraded":"2024-02-05T06:44:38.495459+0000","last_fullsized":"2024-02-05T06:44:38.495459+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:00.501644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:44:00.501644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:09.346038+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"295'15","reported_seq":316,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.484516+0000","last_change":"2024-02-05T06:42:45.004868+0000","last_active":"2024-02-05T06:44:38.484516+0000","last_peered":"2024-02-05T06:44:38.484516+0000","last_clean":"2024-02-05T06:44:38.484516+0000","last_became_active":"2024-02-05T06:39:13.244124+0000","last_became_peered":"2024-02-05T06:39:13.244124+0000","last_unstale":"2024-02-05T06:44:38.484516+0000","last_undegraded":"2024-02-05T06:44:38.484516+0000","last_fullsized":"2024-02-05T06:44:38.484516+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:42:45.004744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:42:45.004744+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:41.953966+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":530,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.495151+0000","last_change":"2024-02-05T06:42:36.192914+0000","last_active":"2024-02-05T06:44:38.495151+0000","last_peered":"2024-02-05T06:44:38.495151+0000","last_clean":"2024-02-05T06:44:38.495151+0000","last_became_active":"2024-02-05T06:41:43.417155+0000","last_became_peered":"2024-02-05T06:41:43.417155+0000","last_unstale":"2024-02-05T06:44:38.495151+0000","last_undegraded":"2024-02-05T06:44:38.495151+0000","last_fullsized":"2024-02-05T06:44:38.495151+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:36.192791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:42:36.192791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":556,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.488121+0000","last_change":"2024-02-05T06:43:57.477466+0000","last_active":"2024-02-05T06:44:38.488121+0000","last_peered":"2024-02-05T06:44:38.488121+0000","last_clean":"2024-02-05T06:44:38.488121+0000","last_became_active":"2024-02-05T06:41:43.421433+0000","last_became_peered":"2024-02-05T06:41:43.421433+0000","last_unstale":"2024-02-05T06:44:38.488121+0000","last_undegraded":"2024-02-05T06:44:38.488121+0000","last_fullsized":"2024-02-05T06:44:38.488121+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:57.477296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:57.477296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:57.550396+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":428,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.488526+0000","last_change":"2024-02-05T06:42:58.691714+0000","last_active":"2024-02-05T06:44:38.488526+0000","last_peered":"2024-02-05T06:44:38.488526+0000","last_clean":"2024-02-05T06:44:38.488526+0000","last_became_active":"2024-02-05T06:40:27.065047+0000","last_became_peered":"2024-02-05T06:40:27.065047+0000","last_unstale":"2024-02-05T06:44:38.488526+0000","last_undegraded":"2024-02-05T06:44:38.488526+0000","last_fullsized":"2024-02-05T06:44:38.488526+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:58.691641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:58.691641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.044358+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"295'24","reported_seq":440,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.487889+0000","last_change":"2024-02-05T06:43:46.883158+0000","last_active":"2024-02-05T06:44:38.487889+0000","last_peered":"2024-02-05T06:44:38.487889+0000","last_clean":"2024-02-05T06:44:38.487889+0000","last_became_active":"2024-02-05T06:36:54.961610+0000","last_became_peered":"2024-02-05T06:36:54.961610+0000","last_unstale":"2024-02-05T06:44:38.487889+0000","last_undegraded":"2024-02-05T06:44:38.487889+0000","last_fullsized":"2024-02-05T06:44:38.487889+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:43:46.883030+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:35:02.589229+0000","last_clean_scrub_stamp":"2024-02-05T06:43:46.883030+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:49.410851+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":531,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.485066+0000","last_change":"2024-02-05T06:43:35.958292+0000","last_active":"2024-02-05T06:44:38.485066+0000","last_peered":"2024-02-05T06:44:38.485066+0000","last_clean":"2024-02-05T06:44:38.485066+0000","last_became_active":"2024-02-05T06:40:27.096132+0000","last_became_peered":"2024-02-05T06:40:27.096132+0000","last_unstale":"2024-02-05T06:44:38.485066+0000","last_undegraded":"2024-02-05T06:44:38.485066+0000","last_fullsized":"2024-02-05T06:44:38.485066+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:35.958174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:43:35.958174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:41.954323+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"249'10","reported_seq":188,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.493114+0000","last_change":"2024-02-05T06:42:50.652634+0000","last_active":"2024-02-05T06:44:38.493114+0000","last_peered":"2024-02-05T06:44:38.493114+0000","last_clean":"2024-02-05T06:44:38.493114+0000","last_became_active":"2024-02-05T06:41:43.418753+0000","last_became_peered":"2024-02-05T06:41:43.418753+0000","last_unstale":"2024-02-05T06:44:38.493114+0000","last_undegraded":"2024-02-05T06:44:38.493114+0000","last_fullsized":"2024-02-05T06:44:38.493114+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'10","last_scrub_stamp":"2024-02-05T06:42:50.652545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:42:50.652545+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":232,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.494477+0000","last_change":"2024-02-05T06:42:41.410614+0000","last_active":"2024-02-05T06:44:38.494477+0000","last_peered":"2024-02-05T06:44:38.494477+0000","last_clean":"2024-02-05T06:44:38.494477+0000","last_became_active":"2024-02-05T06:40:27.098418+0000","last_became_peered":"2024-02-05T06:40:27.098418+0000","last_unstale":"2024-02-05T06:44:38.494477+0000","last_undegraded":"2024-02-05T06:44:38.494477+0000","last_fullsized":"2024-02-05T06:44:38.494477+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:41.410495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:41.410495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.386274+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"295'20","reported_seq":610,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.495451+0000","last_change":"2024-02-05T06:42:47.381961+0000","last_active":"2024-02-05T06:44:38.495451+0000","last_peered":"2024-02-05T06:44:38.495451+0000","last_clean":"2024-02-05T06:44:38.495451+0000","last_became_active":"2024-02-05T06:40:27.093418+0000","last_became_peered":"2024-02-05T06:40:27.093418+0000","last_unstale":"2024-02-05T06:44:38.495451+0000","last_undegraded":"2024-02-05T06:44:38.495451+0000","last_fullsized":"2024-02-05T06:44:38.495451+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-02-05T06:42:47.381317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:42:47.381317+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":428,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.488421+0000","last_change":"2024-02-05T06:41:43.413904+0000","last_active":"2024-02-05T06:44:38.488421+0000","last_peered":"2024-02-05T06:44:38.488421+0000","last_clean":"2024-02-05T06:44:38.488421+0000","last_became_active":"2024-02-05T06:41:43.413562+0000","last_became_peered":"2024-02-05T06:41:43.413562+0000","last_unstale":"2024-02-05T06:44:38.488421+0000","last_undegraded":"2024-02-05T06:44:38.488421+0000","last_fullsized":"2024-02-05T06:44:38.488421+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:41:38.559780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:41:38.559780+0000","last_clean_scrub_stamp":"2024-02-05T06:41:38.559780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.044171+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"208'10","reported_seq":685,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.495830+0000","last_change":"2024-02-05T06:41:42.168574+0000","last_active":"2024-02-05T06:44:38.495830+0000","last_peered":"2024-02-05T06:44:38.495830+0000","last_clean":"2024-02-05T06:44:38.495830+0000","last_became_active":"2024-02-05T06:40:27.096584+0000","last_became_peered":"2024-02-05T06:40:27.096584+0000","last_unstale":"2024-02-05T06:44:38.495830+0000","last_undegraded":"2024-02-05T06:44:38.495830+0000","last_fullsized":"2024-02-05T06:44:38.495830+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-02-05T06:41:42.168453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:41:42.168453+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":551,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.484736+0000","last_change":"2024-02-05T06:44:01.185774+0000","last_active":"2024-02-05T06:44:38.484736+0000","last_peered":"2024-02-05T06:44:38.484736+0000","last_clean":"2024-02-05T06:44:38.484736+0000","last_became_active":"2024-02-05T06:40:27.088096+0000","last_became_peered":"2024-02-05T06:40:27.088096+0000","last_unstale":"2024-02-05T06:44:38.484736+0000","last_undegraded":"2024-02-05T06:44:38.484736+0000","last_fullsized":"2024-02-05T06:44:38.484736+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:01.185653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:44:01.185653+0000","last_clean_scrub_stamp":"2024-02-05T06:44:01.185653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:04.676309+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"295'15","reported_seq":167,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.493455+0000","last_change":"2024-02-05T06:42:52.697830+0000","last_active":"2024-02-05T06:44:38.493455+0000","last_peered":"2024-02-05T06:44:38.493455+0000","last_clean":"2024-02-05T06:44:38.493455+0000","last_became_active":"2024-02-05T06:41:43.413708+0000","last_became_peered":"2024-02-05T06:41:43.413708+0000","last_unstale":"2024-02-05T06:44:38.493455+0000","last_undegraded":"2024-02-05T06:44:38.493455+0000","last_fullsized":"2024-02-05T06:44:38.493455+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-02-05T06:42:52.697346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:42:52.697346+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":564,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.497508+0000","last_change":"2024-02-05T06:43:01.524727+0000","last_active":"2024-02-05T06:44:38.497508+0000","last_peered":"2024-02-05T06:44:38.497508+0000","last_clean":"2024-02-05T06:44:38.497508+0000","last_became_active":"2024-02-05T06:40:27.090632+0000","last_became_peered":"2024-02-05T06:40:27.090632+0000","last_unstale":"2024-02-05T06:44:38.497508+0000","last_undegraded":"2024-02-05T06:44:38.497508+0000","last_fullsized":"2024-02-05T06:44:38.497508+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:01.524512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:43:01.524512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":548,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.496070+0000","last_change":"2024-02-05T06:44:01.644210+0000","last_active":"2024-02-05T06:44:38.496070+0000","last_peered":"2024-02-05T06:44:38.496070+0000","last_clean":"2024-02-05T06:44:38.496070+0000","last_became_active":"2024-02-05T06:40:27.091544+0000","last_became_peered":"2024-02-05T06:40:27.091544+0000","last_unstale":"2024-02-05T06:44:38.496070+0000","last_undegraded":"2024-02-05T06:44:38.496070+0000","last_fullsized":"2024-02-05T06:44:38.496070+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:01.644095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:44:01.644095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:07.661277+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":232,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.493649+0000","last_change":"2024-02-05T06:42:46.409374+0000","last_active":"2024-02-05T06:44:38.493649+0000","last_peered":"2024-02-05T06:44:38.493649+0000","last_clean":"2024-02-05T06:44:38.493649+0000","last_became_active":"2024-02-05T06:40:27.064388+0000","last_became_peered":"2024-02-05T06:40:27.064388+0000","last_unstale":"2024-02-05T06:44:38.493649+0000","last_undegraded":"2024-02-05T06:44:38.493649+0000","last_fullsized":"2024-02-05T06:44:38.493649+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:46.409259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:46.409259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.386311+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"295'15","reported_seq":592,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.485450+0000","last_change":"2024-02-05T06:44:04.244244+0000","last_active":"2024-02-05T06:44:38.485450+0000","last_peered":"2024-02-05T06:44:38.485450+0000","last_clean":"2024-02-05T06:44:38.485450+0000","last_became_active":"2024-02-05T06:40:27.101116+0000","last_became_peered":"2024-02-05T06:40:27.101116+0000","last_unstale":"2024-02-05T06:44:38.485450+0000","last_undegraded":"2024-02-05T06:44:38.485450+0000","last_fullsized":"2024-02-05T06:44:38.485450+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:44:04.244094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:44:04.244094+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:13.039537+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":232,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.495777+0000","last_change":"2024-02-05T06:42:49.428563+0000","last_active":"2024-02-05T06:44:38.495777+0000","last_peered":"2024-02-05T06:44:38.495777+0000","last_clean":"2024-02-05T06:44:38.495777+0000","last_became_active":"2024-02-05T06:40:27.063350+0000","last_became_peered":"2024-02-05T06:40:27.063350+0000","last_unstale":"2024-02-05T06:44:38.495777+0000","last_undegraded":"2024-02-05T06:44:38.495777+0000","last_fullsized":"2024-02-05T06:44:38.495777+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:49.428464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:42:49.428464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.386512+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":148,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.492579+0000","last_change":"2024-02-05T06:42:58.680331+0000","last_active":"2024-02-05T06:44:38.492579+0000","last_peered":"2024-02-05T06:44:38.492579+0000","last_clean":"2024-02-05T06:44:38.492579+0000","last_became_active":"2024-02-05T06:41:43.406094+0000","last_became_peered":"2024-02-05T06:41:43.406094+0000","last_unstale":"2024-02-05T06:44:38.492579+0000","last_undegraded":"2024-02-05T06:44:38.492579+0000","last_fullsized":"2024-02-05T06:44:38.492579+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:58.680262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:45.096340+0000","last_clean_scrub_stamp":"2024-02-05T06:42:58.680262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":557,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.493967+0000","last_change":"2024-02-05T06:42:48.170292+0000","last_active":"2024-02-05T06:44:38.493967+0000","last_peered":"2024-02-05T06:44:38.493967+0000","last_clean":"2024-02-05T06:44:38.493967+0000","last_became_active":"2024-02-05T06:41:43.412130+0000","last_became_peered":"2024-02-05T06:41:43.412130+0000","last_unstale":"2024-02-05T06:44:38.493967+0000","last_undegraded":"2024-02-05T06:44:38.493967+0000","last_fullsized":"2024-02-05T06:44:38.493967+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:48.170173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:48.170173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:39.466258+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"295'12","reported_seq":572,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.501813+0000","last_change":"2024-02-05T06:43:37.426033+0000","last_active":"2024-02-05T06:44:38.501813+0000","last_peered":"2024-02-05T06:44:38.501813+0000","last_clean":"2024-02-05T06:44:38.501813+0000","last_became_active":"2024-02-05T06:40:27.091689+0000","last_became_peered":"2024-02-05T06:40:27.091689+0000","last_unstale":"2024-02-05T06:44:38.501813+0000","last_undegraded":"2024-02-05T06:44:38.501813+0000","last_fullsized":"2024-02-05T06:44:38.501813+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'12","last_scrub_stamp":"2024-02-05T06:43:37.425879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:37.425879+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"312'6181","reported_seq":6233,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:42.214219+0000","last_change":"2024-02-05T06:43:29.702891+0000","last_active":"2024-02-05T06:44:42.214219+0000","last_peered":"2024-02-05T06:44:42.214219+0000","last_clean":"2024-02-05T06:44:42.214219+0000","last_became_active":"2024-02-05T06:43:29.702579+0000","last_became_peered":"2024-02-05T06:43:29.702579+0000","last_unstale":"2024-02-05T06:44:42.214219+0000","last_undegraded":"2024-02-05T06:44:42.214219+0000","last_fullsized":"2024-02-05T06:44:42.214219+0000","mapping_epoch":291,"log_start":"296'500","ondisk_log_start":"296'500","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":5681,"log_dups_size":0,"ondisk_log_size":5681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405078016,"num_objects":6181,"num_object_clones":0,"num_object_copies":18543,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6181,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6181,"num_write_kb":395584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":148,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.495860+0000","last_change":"2024-02-05T06:42:59.672197+0000","last_active":"2024-02-05T06:44:38.495860+0000","last_peered":"2024-02-05T06:44:38.495860+0000","last_clean":"2024-02-05T06:44:38.495860+0000","last_became_active":"2024-02-05T06:41:43.421231+0000","last_became_peered":"2024-02-05T06:41:43.421231+0000","last_unstale":"2024-02-05T06:44:38.495860+0000","last_undegraded":"2024-02-05T06:44:38.495860+0000","last_fullsized":"2024-02-05T06:44:38.495860+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:59.672110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:42:59.672110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":246,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.491768+0000","last_change":"2024-02-05T06:43:56.531681+0000","last_active":"2024-02-05T06:44:38.491768+0000","last_peered":"2024-02-05T06:44:38.491768+0000","last_clean":"2024-02-05T06:44:38.491768+0000","last_became_active":"2024-02-05T06:40:27.068279+0000","last_became_peered":"2024-02-05T06:40:27.068279+0000","last_unstale":"2024-02-05T06:44:38.491768+0000","last_undegraded":"2024-02-05T06:44:38.491768+0000","last_fullsized":"2024-02-05T06:44:38.491768+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-02-05T06:43:56.531539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:43:56.531539+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:24.413854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"295'27","reported_seq":182,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.492242+0000","last_change":"2024-02-05T06:43:43.345357+0000","last_active":"2024-02-05T06:44:38.492242+0000","last_peered":"2024-02-05T06:44:38.492242+0000","last_clean":"2024-02-05T06:44:38.492242+0000","last_became_active":"2024-02-05T06:41:43.412579+0000","last_became_peered":"2024-02-05T06:41:43.412579+0000","last_unstale":"2024-02-05T06:44:38.492242+0000","last_undegraded":"2024-02-05T06:44:38.492242+0000","last_fullsized":"2024-02-05T06:44:38.492242+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'27","last_scrub_stamp":"2024-02-05T06:43:43.344878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:43.344878+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:55.975676+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"208'12","reported_seq":287,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.493602+0000","last_change":"2024-02-05T06:42:50.411738+0000","last_active":"2024-02-05T06:44:38.493602+0000","last_peered":"2024-02-05T06:44:38.493602+0000","last_clean":"2024-02-05T06:44:38.493602+0000","last_became_active":"2024-02-05T06:40:27.064195+0000","last_became_peered":"2024-02-05T06:40:27.064195+0000","last_unstale":"2024-02-05T06:44:38.493602+0000","last_undegraded":"2024-02-05T06:44:38.493602+0000","last_fullsized":"2024-02-05T06:44:38.493602+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'12","last_scrub_stamp":"2024-02-05T06:42:50.411654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:42:50.411654+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.386246+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"312'6294","reported_seq":6416,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:41.847243+0000","last_change":"2024-02-05T06:44:40.820675+0000","last_active":"2024-02-05T06:44:41.847243+0000","last_peered":"2024-02-05T06:44:41.847243+0000","last_clean":"2024-02-05T06:44:41.847243+0000","last_became_active":"2024-02-05T06:43:29.697495+0000","last_became_peered":"2024-02-05T06:43:29.697495+0000","last_unstale":"2024-02-05T06:44:41.847243+0000","last_undegraded":"2024-02-05T06:44:41.847243+0000","last_fullsized":"2024-02-05T06:44:41.847243+0000","mapping_epoch":291,"log_start":"297'1600","ondisk_log_start":"297'1600","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"312'6215","last_scrub_stamp":"2024-02-05T06:44:40.820543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:44:40.820543+0000","objects_scrubbed":6153,"log_size":4694,"log_dups_size":0,"ondisk_log_size":4694,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:55.970797+0000","scrub_duration":2108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412483584,"num_objects":6294,"num_object_clones":0,"num_object_copies":18882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6294,"num_write_kb":402816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":530,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.495604+0000","last_change":"2024-02-05T06:43:47.662237+0000","last_active":"2024-02-05T06:44:38.495604+0000","last_peered":"2024-02-05T06:44:38.495604+0000","last_clean":"2024-02-05T06:44:38.495604+0000","last_became_active":"2024-02-05T06:40:27.092613+0000","last_became_peered":"2024-02-05T06:40:27.092613+0000","last_unstale":"2024-02-05T06:44:38.495604+0000","last_undegraded":"2024-02-05T06:44:38.495604+0000","last_fullsized":"2024-02-05T06:44:38.495604+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:47.662025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:43:47.662025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:04.272911+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":232,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.494920+0000","last_change":"2024-02-05T06:42:43.390081+0000","last_active":"2024-02-05T06:44:38.494920+0000","last_peered":"2024-02-05T06:44:38.494920+0000","last_clean":"2024-02-05T06:44:38.494920+0000","last_became_active":"2024-02-05T06:40:27.063300+0000","last_became_peered":"2024-02-05T06:40:27.063300+0000","last_unstale":"2024-02-05T06:44:38.494920+0000","last_undegraded":"2024-02-05T06:44:38.494920+0000","last_fullsized":"2024-02-05T06:44:38.494920+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:43.389980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:43.389980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.386458+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"295'18","reported_seq":437,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.487785+0000","last_change":"2024-02-05T06:41:52.647443+0000","last_active":"2024-02-05T06:44:38.487785+0000","last_peered":"2024-02-05T06:44:38.487785+0000","last_clean":"2024-02-05T06:44:38.487785+0000","last_became_active":"2024-02-05T06:36:54.954071+0000","last_became_peered":"2024-02-05T06:36:54.954071+0000","last_unstale":"2024-02-05T06:44:38.487785+0000","last_undegraded":"2024-02-05T06:44:38.487785+0000","last_fullsized":"2024-02-05T06:44:38.487785+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:41:52.647335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:41:52.647335+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.044296+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":537,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.496445+0000","last_change":"2024-02-05T06:43:44.383974+0000","last_active":"2024-02-05T06:44:38.496445+0000","last_peered":"2024-02-05T06:44:38.496445+0000","last_clean":"2024-02-05T06:44:38.496445+0000","last_became_active":"2024-02-05T06:33:33.592546+0000","last_became_peered":"2024-02-05T06:33:33.592546+0000","last_unstale":"2024-02-05T06:44:38.496445+0000","last_undegraded":"2024-02-05T06:44:38.496445+0000","last_fullsized":"2024-02-05T06:44:38.496445+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:44.382909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:44.382909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:50.463660+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"312'6054","reported_seq":6106,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:41.277476+0000","last_change":"2024-02-05T06:43:29.702980+0000","last_active":"2024-02-05T06:44:41.277476+0000","last_peered":"2024-02-05T06:44:41.277476+0000","last_clean":"2024-02-05T06:44:41.277476+0000","last_became_active":"2024-02-05T06:43:29.702659+0000","last_became_peered":"2024-02-05T06:43:29.702659+0000","last_unstale":"2024-02-05T06:44:41.277476+0000","last_undegraded":"2024-02-05T06:44:41.277476+0000","last_fullsized":"2024-02-05T06:44:41.277476+0000","mapping_epoch":291,"log_start":"296'1000","ondisk_log_start":"296'1000","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":5054,"log_dups_size":0,"ondisk_log_size":5054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:50.762568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":396754944,"num_objects":6054,"num_object_clones":0,"num_object_copies":18162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6054,"num_write_kb":387456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":232,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.491116+0000","last_change":"2024-02-05T06:42:58.350505+0000","last_active":"2024-02-05T06:44:38.491116+0000","last_peered":"2024-02-05T06:44:38.491116+0000","last_clean":"2024-02-05T06:44:38.491116+0000","last_became_active":"2024-02-05T06:40:27.218081+0000","last_became_peered":"2024-02-05T06:40:27.218081+0000","last_unstale":"2024-02-05T06:44:38.491116+0000","last_undegraded":"2024-02-05T06:44:38.491116+0000","last_fullsized":"2024-02-05T06:44:38.491116+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:58.350239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:32.925000+0000","last_clean_scrub_stamp":"2024-02-05T06:42:58.350239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.386383+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":562,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.497056+0000","last_change":"2024-02-05T06:44:01.453193+0000","last_active":"2024-02-05T06:44:38.497056+0000","last_peered":"2024-02-05T06:44:38.497056+0000","last_clean":"2024-02-05T06:44:38.497056+0000","last_became_active":"2024-02-05T06:41:43.413095+0000","last_became_peered":"2024-02-05T06:41:43.413095+0000","last_unstale":"2024-02-05T06:44:38.497056+0000","last_undegraded":"2024-02-05T06:44:38.497056+0000","last_fullsized":"2024-02-05T06:44:38.497056+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:01.452385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:44:01.452385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:30.760517+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":238,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.495633+0000","last_change":"2024-02-05T06:43:02.361335+0000","last_active":"2024-02-05T06:44:38.495633+0000","last_peered":"2024-02-05T06:44:38.495633+0000","last_clean":"2024-02-05T06:44:38.495633+0000","last_became_active":"2024-02-05T06:41:43.418202+0000","last_became_peered":"2024-02-05T06:41:43.418202+0000","last_unstale":"2024-02-05T06:44:38.495633+0000","last_undegraded":"2024-02-05T06:44:38.495633+0000","last_fullsized":"2024-02-05T06:44:38.495633+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:02.361235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:43:02.361235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.386485+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"295'44","reported_seq":684,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.491704+0000","last_change":"2024-02-05T06:42:50.149138+0000","last_active":"2024-02-05T06:44:38.491704+0000","last_peered":"2024-02-05T06:44:38.491704+0000","last_clean":"2024-02-05T06:44:38.491704+0000","last_became_active":"2024-02-05T06:40:27.068417+0000","last_became_peered":"2024-02-05T06:40:27.068417+0000","last_unstale":"2024-02-05T06:44:38.491704+0000","last_undegraded":"2024-02-05T06:44:38.491704+0000","last_fullsized":"2024-02-05T06:44:38.491704+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"279'28","last_scrub_stamp":"2024-02-05T06:42:50.148881+0000","last_deep_scrub":"279'28","last_deep_scrub_stamp":"2024-02-05T06:42:50.148881+0000","last_clean_scrub_stamp":"2024-02-05T06:42:50.148881+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:39.466287+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"312'6244","reported_seq":6296,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:41.262341+0000","last_change":"2024-02-05T06:43:29.843229+0000","last_active":"2024-02-05T06:44:41.262341+0000","last_peered":"2024-02-05T06:44:41.262341+0000","last_clean":"2024-02-05T06:44:41.262341+0000","last_became_active":"2024-02-05T06:43:29.842930+0000","last_became_peered":"2024-02-05T06:43:29.842930+0000","last_unstale":"2024-02-05T06:44:41.262341+0000","last_undegraded":"2024-02-05T06:44:41.262341+0000","last_fullsized":"2024-02-05T06:44:41.262341+0000","mapping_epoch":291,"log_start":"296'1200","ondisk_log_start":"296'1200","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":5044,"log_dups_size":0,"ondisk_log_size":5044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:53.746948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409206784,"num_objects":6244,"num_object_clones":0,"num_object_copies":18732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6244,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6244,"num_write_kb":399616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":532,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.487363+0000","last_change":"2024-02-05T06:44:00.315984+0000","last_active":"2024-02-05T06:44:38.487363+0000","last_peered":"2024-02-05T06:44:38.487363+0000","last_clean":"2024-02-05T06:44:38.487363+0000","last_became_active":"2024-02-05T06:40:27.129625+0000","last_became_peered":"2024-02-05T06:40:27.129625+0000","last_unstale":"2024-02-05T06:44:38.487363+0000","last_undegraded":"2024-02-05T06:44:38.487363+0000","last_fullsized":"2024-02-05T06:44:38.487363+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:00.315851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:44:00.315851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:25.409206+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":557,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.496883+0000","last_change":"2024-02-05T06:43:41.424681+0000","last_active":"2024-02-05T06:44:38.496883+0000","last_peered":"2024-02-05T06:44:38.496883+0000","last_clean":"2024-02-05T06:44:38.496883+0000","last_became_active":"2024-02-05T06:36:54.966016+0000","last_became_peered":"2024-02-05T06:36:54.966016+0000","last_unstale":"2024-02-05T06:44:38.496883+0000","last_undegraded":"2024-02-05T06:44:38.496883+0000","last_fullsized":"2024-02-05T06:44:38.496883+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:41.424561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:41.424561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:54.460356+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":148,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.494782+0000","last_change":"2024-02-05T06:42:32.759801+0000","last_active":"2024-02-05T06:44:38.494782+0000","last_peered":"2024-02-05T06:44:38.494782+0000","last_clean":"2024-02-05T06:44:38.494782+0000","last_became_active":"2024-02-05T06:41:43.412611+0000","last_became_peered":"2024-02-05T06:41:43.412611+0000","last_unstale":"2024-02-05T06:44:38.494782+0000","last_undegraded":"2024-02-05T06:44:38.494782+0000","last_fullsized":"2024-02-05T06:44:38.494782+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:32.759672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:32.759672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"295'15","reported_seq":640,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.486146+0000","last_change":"2024-02-05T06:43:52.150377+0000","last_active":"2024-02-05T06:44:38.486146+0000","last_peered":"2024-02-05T06:44:38.486146+0000","last_clean":"2024-02-05T06:44:38.486146+0000","last_became_active":"2024-02-05T06:40:08.584204+0000","last_became_peered":"2024-02-05T06:40:08.584204+0000","last_unstale":"2024-02-05T06:44:38.486146+0000","last_undegraded":"2024-02-05T06:44:38.486146+0000","last_fullsized":"2024-02-05T06:44:38.486146+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:43:52.150132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:52.150132+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:05.649207+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"312'6170","reported_seq":6222,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:42.166037+0000","last_change":"2024-02-05T06:43:29.703821+0000","last_active":"2024-02-05T06:44:42.166037+0000","last_peered":"2024-02-05T06:44:42.166037+0000","last_clean":"2024-02-05T06:44:42.166037+0000","last_became_active":"2024-02-05T06:43:29.703452+0000","last_became_peered":"2024-02-05T06:43:29.703452+0000","last_unstale":"2024-02-05T06:44:42.166037+0000","last_undegraded":"2024-02-05T06:44:42.166037+0000","last_fullsized":"2024-02-05T06:44:42.166037+0000","mapping_epoch":291,"log_start":"296'500","ondisk_log_start":"296'500","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":5670,"log_dups_size":0,"ondisk_log_size":5670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:45.742277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404357120,"num_objects":6170,"num_object_clones":0,"num_object_copies":18510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6170,"num_write_kb":394880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":573,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.493933+0000","last_change":"2024-02-05T06:44:04.452427+0000","last_active":"2024-02-05T06:44:38.493933+0000","last_peered":"2024-02-05T06:44:38.493933+0000","last_clean":"2024-02-05T06:44:38.493933+0000","last_became_active":"2024-02-05T06:41:46.054968+0000","last_became_peered":"2024-02-05T06:41:46.054968+0000","last_unstale":"2024-02-05T06:44:38.493933+0000","last_undegraded":"2024-02-05T06:44:38.493933+0000","last_fullsized":"2024-02-05T06:44:38.493933+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:44:04.452308+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:37.282442+0000","last_clean_scrub_stamp":"2024-02-05T06:44:04.452308+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:32.675613+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":543,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.495833+0000","last_change":"2024-02-05T06:43:40.471046+0000","last_active":"2024-02-05T06:44:38.495833+0000","last_peered":"2024-02-05T06:44:38.495833+0000","last_clean":"2024-02-05T06:44:38.495833+0000","last_became_active":"2024-02-05T06:40:27.082440+0000","last_became_peered":"2024-02-05T06:40:27.082440+0000","last_unstale":"2024-02-05T06:44:38.495833+0000","last_undegraded":"2024-02-05T06:44:38.495833+0000","last_fullsized":"2024-02-05T06:44:38.495833+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:40.470922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:43:40.470922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.918600+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"182'3","reported_seq":152,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.495941+0000","last_change":"2024-02-05T06:43:40.376967+0000","last_active":"2024-02-05T06:44:38.495941+0000","last_peered":"2024-02-05T06:44:38.495941+0000","last_clean":"2024-02-05T06:44:38.495941+0000","last_became_active":"2024-02-05T06:41:43.415806+0000","last_became_peered":"2024-02-05T06:41:43.415806+0000","last_unstale":"2024-02-05T06:44:38.495941+0000","last_undegraded":"2024-02-05T06:44:38.495941+0000","last_fullsized":"2024-02-05T06:44:38.495941+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'3","last_scrub_stamp":"2024-02-05T06:43:40.376865+0000","last_deep_scrub":"182'3","last_deep_scrub_stamp":"2024-02-05T06:39:33.373415+0000","last_clean_scrub_stamp":"2024-02-05T06:43:40.376865+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:50.126421+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001220348,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":416,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.487777+0000","last_change":"2024-02-05T06:42:42.734295+0000","last_active":"2024-02-05T06:44:38.487777+0000","last_peered":"2024-02-05T06:44:38.487777+0000","last_clean":"2024-02-05T06:44:38.487777+0000","last_became_active":"2024-02-05T06:36:54.968205+0000","last_became_peered":"2024-02-05T06:36:54.968205+0000","last_unstale":"2024-02-05T06:44:38.487777+0000","last_undegraded":"2024-02-05T06:44:38.487777+0000","last_fullsized":"2024-02-05T06:44:38.487777+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:42.734048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:42.734048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.044342+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"295'42","reported_seq":348,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.485542+0000","last_change":"2024-02-05T06:43:51.551040+0000","last_active":"2024-02-05T06:44:38.485542+0000","last_peered":"2024-02-05T06:44:38.485542+0000","last_clean":"2024-02-05T06:44:38.485542+0000","last_became_active":"2024-02-05T06:40:08.604483+0000","last_became_peered":"2024-02-05T06:40:08.604483+0000","last_unstale":"2024-02-05T06:44:38.485542+0000","last_undegraded":"2024-02-05T06:44:38.485542+0000","last_fullsized":"2024-02-05T06:44:38.485542+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'42","last_scrub_stamp":"2024-02-05T06:43:51.550901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:51.550901+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:01.091650+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":536,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.484599+0000","last_change":"2024-02-05T06:42:42.929540+0000","last_active":"2024-02-05T06:44:38.484599+0000","last_peered":"2024-02-05T06:44:38.484599+0000","last_clean":"2024-02-05T06:44:38.484599+0000","last_became_active":"2024-02-05T06:36:54.973387+0000","last_became_peered":"2024-02-05T06:36:54.973387+0000","last_unstale":"2024-02-05T06:44:38.484599+0000","last_undegraded":"2024-02-05T06:44:38.484599+0000","last_fullsized":"2024-02-05T06:44:38.484599+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:42.929409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:42:42.929409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:41.954061+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"312'6176","reported_seq":6228,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:41.248332+0000","last_change":"2024-02-05T06:43:29.702708+0000","last_active":"2024-02-05T06:44:41.248332+0000","last_peered":"2024-02-05T06:44:41.248332+0000","last_clean":"2024-02-05T06:44:41.248332+0000","last_became_active":"2024-02-05T06:43:29.702401+0000","last_became_peered":"2024-02-05T06:43:29.702401+0000","last_unstale":"2024-02-05T06:44:41.248332+0000","last_undegraded":"2024-02-05T06:44:41.248332+0000","last_fullsized":"2024-02-05T06:44:41.248332+0000","mapping_epoch":291,"log_start":"296'1100","ondisk_log_start":"296'1100","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":5076,"log_dups_size":0,"ondisk_log_size":5076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:55.338031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":404750336,"num_objects":6176,"num_object_clones":0,"num_object_copies":18528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6176,"num_write_kb":395264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":654,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.497606+0000","last_change":"2024-02-05T06:42:53.491927+0000","last_active":"2024-02-05T06:44:38.497606+0000","last_peered":"2024-02-05T06:44:38.497606+0000","last_clean":"2024-02-05T06:44:38.497606+0000","last_became_active":"2024-02-05T06:40:30.082467+0000","last_became_peered":"2024-02-05T06:40:30.082467+0000","last_unstale":"2024-02-05T06:44:38.497606+0000","last_undegraded":"2024-02-05T06:44:38.497606+0000","last_fullsized":"2024-02-05T06:44:38.497606+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:42:53.491791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:42:53.491791+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.386353+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":233,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.495379+0000","last_change":"2024-02-05T06:42:54.478248+0000","last_active":"2024-02-05T06:44:38.495379+0000","last_peered":"2024-02-05T06:44:38.495379+0000","last_clean":"2024-02-05T06:44:38.495379+0000","last_became_active":"2024-02-05T06:40:27.077966+0000","last_became_peered":"2024-02-05T06:40:27.077966+0000","last_unstale":"2024-02-05T06:44:38.495379+0000","last_undegraded":"2024-02-05T06:44:38.495379+0000","last_fullsized":"2024-02-05T06:44:38.495379+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:54.477914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:42:54.477914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.386567+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":539,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.488078+0000","last_change":"2024-02-05T06:43:46.552644+0000","last_active":"2024-02-05T06:44:38.488078+0000","last_peered":"2024-02-05T06:44:38.488078+0000","last_clean":"2024-02-05T06:44:38.488078+0000","last_became_active":"2024-02-05T06:33:29.551549+0000","last_became_peered":"2024-02-05T06:33:29.551549+0000","last_unstale":"2024-02-05T06:44:38.488078+0000","last_undegraded":"2024-02-05T06:44:38.488078+0000","last_fullsized":"2024-02-05T06:44:38.488078+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:46.552530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:43:46.552530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:09.476346+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":554,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.496336+0000","last_change":"2024-02-05T06:42:59.532291+0000","last_active":"2024-02-05T06:44:38.496336+0000","last_peered":"2024-02-05T06:44:38.496336+0000","last_clean":"2024-02-05T06:44:38.496336+0000","last_became_active":"2024-02-05T06:33:28.542977+0000","last_became_peered":"2024-02-05T06:33:28.542977+0000","last_unstale":"2024-02-05T06:44:38.496336+0000","last_undegraded":"2024-02-05T06:44:38.496336+0000","last_fullsized":"2024-02-05T06:44:38.496336+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:59.532192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:42:59.532192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:42.485843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091640799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"295'17","reported_seq":630,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.486593+0000","last_change":"2024-02-05T06:43:01.610636+0000","last_active":"2024-02-05T06:44:38.486593+0000","last_peered":"2024-02-05T06:44:38.486593+0000","last_clean":"2024-02-05T06:44:38.486593+0000","last_became_active":"2024-02-05T06:41:43.420458+0000","last_became_peered":"2024-02-05T06:41:43.420458+0000","last_unstale":"2024-02-05T06:44:38.486593+0000","last_undegraded":"2024-02-05T06:44:38.486593+0000","last_fullsized":"2024-02-05T06:44:38.486593+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-02-05T06:43:01.610491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:01.610491+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.698012+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":569,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.497777+0000","last_change":"2024-02-05T06:42:52.383735+0000","last_active":"2024-02-05T06:44:38.497777+0000","last_peered":"2024-02-05T06:44:38.497777+0000","last_clean":"2024-02-05T06:44:38.497777+0000","last_became_active":"2024-02-05T06:41:43.412387+0000","last_became_peered":"2024-02-05T06:41:43.412387+0000","last_unstale":"2024-02-05T06:44:38.497777+0000","last_undegraded":"2024-02-05T06:44:38.497777+0000","last_fullsized":"2024-02-05T06:44:38.497777+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:52.383618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:33.914422+0000","last_clean_scrub_stamp":"2024-02-05T06:42:52.383618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"312'6250","reported_seq":6302,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:42.228360+0000","last_change":"2024-02-05T06:43:29.701823+0000","last_active":"2024-02-05T06:44:42.228360+0000","last_peered":"2024-02-05T06:44:42.228360+0000","last_clean":"2024-02-05T06:44:42.228360+0000","last_became_active":"2024-02-05T06:43:29.701487+0000","last_became_peered":"2024-02-05T06:43:29.701487+0000","last_unstale":"2024-02-05T06:44:42.228360+0000","last_undegraded":"2024-02-05T06:44:42.228360+0000","last_fullsized":"2024-02-05T06:44:42.228360+0000","mapping_epoch":291,"log_start":"298'1800","ondisk_log_start":"298'1800","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4450,"log_dups_size":0,"ondisk_log_size":4450,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:50.692005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409600000,"num_objects":6250,"num_object_clones":0,"num_object_copies":18750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6250,"num_write_kb":400000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":156,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:40.571736+0000","last_change":"2024-02-05T06:44:40.571834+0000","last_active":"2024-02-05T06:44:40.571736+0000","last_peered":"2024-02-05T06:44:40.571736+0000","last_clean":"2024-02-05T06:44:40.571736+0000","last_became_active":"2024-02-05T06:41:43.408773+0000","last_became_peered":"2024-02-05T06:41:43.408773+0000","last_unstale":"2024-02-05T06:44:40.571736+0000","last_undegraded":"2024-02-05T06:44:40.571736+0000","last_fullsized":"2024-02-05T06:44:40.571736+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:40.571610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:44:40.571610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:57.800737+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":654,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.491221+0000","last_change":"2024-02-05T06:42:52.491767+0000","last_active":"2024-02-05T06:44:38.491221+0000","last_peered":"2024-02-05T06:44:38.491221+0000","last_clean":"2024-02-05T06:44:38.491221+0000","last_became_active":"2024-02-05T06:40:30.081178+0000","last_became_peered":"2024-02-05T06:40:30.081178+0000","last_unstale":"2024-02-05T06:44:38.491221+0000","last_undegraded":"2024-02-05T06:44:38.491221+0000","last_fullsized":"2024-02-05T06:44:38.491221+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:42:52.491322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:42:52.491322+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.386415+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":543,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.496050+0000","last_change":"2024-02-05T06:42:50.411721+0000","last_active":"2024-02-05T06:44:38.496050+0000","last_peered":"2024-02-05T06:44:38.496050+0000","last_clean":"2024-02-05T06:44:38.496050+0000","last_became_active":"2024-02-05T06:41:43.411335+0000","last_became_peered":"2024-02-05T06:41:43.411335+0000","last_unstale":"2024-02-05T06:44:38.496050+0000","last_undegraded":"2024-02-05T06:44:38.496050+0000","last_fullsized":"2024-02-05T06:44:38.496050+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:50.411634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:50.411634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"295'32","reported_seq":815,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.492897+0000","last_change":"2024-02-05T06:43:58.304760+0000","last_active":"2024-02-05T06:44:38.492897+0000","last_peered":"2024-02-05T06:44:38.492897+0000","last_clean":"2024-02-05T06:44:38.492897+0000","last_became_active":"2024-02-05T06:40:27.061111+0000","last_became_peered":"2024-02-05T06:40:27.061111+0000","last_unstale":"2024-02-05T06:44:38.492897+0000","last_undegraded":"2024-02-05T06:44:38.492897+0000","last_fullsized":"2024-02-05T06:44:38.492897+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'32","last_scrub_stamp":"2024-02-05T06:43:58.304628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:58.304628+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:17.673610+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":247,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.493270+0000","last_change":"2024-02-05T06:43:43.529553+0000","last_active":"2024-02-05T06:44:38.493270+0000","last_peered":"2024-02-05T06:44:38.493270+0000","last_clean":"2024-02-05T06:44:38.493270+0000","last_became_active":"2024-02-05T06:41:43.418198+0000","last_became_peered":"2024-02-05T06:41:43.418198+0000","last_unstale":"2024-02-05T06:44:38.493270+0000","last_undegraded":"2024-02-05T06:44:38.493270+0000","last_fullsized":"2024-02-05T06:44:38.493270+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:43.529437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:43.529437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:00.290531+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"312'6253","reported_seq":6375,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:42.146666+0000","last_change":"2024-02-05T06:44:41.914941+0000","last_active":"2024-02-05T06:44:42.146666+0000","last_peered":"2024-02-05T06:44:42.146666+0000","last_clean":"2024-02-05T06:44:42.146666+0000","last_became_active":"2024-02-05T06:43:29.946961+0000","last_became_peered":"2024-02-05T06:43:29.946961+0000","last_unstale":"2024-02-05T06:44:42.146666+0000","last_undegraded":"2024-02-05T06:44:42.146666+0000","last_fullsized":"2024-02-05T06:44:42.146666+0000","mapping_epoch":291,"log_start":"296'500","ondisk_log_start":"296'500","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"312'6240","last_scrub_stamp":"2024-02-05T06:44:41.914681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:44:41.914681+0000","objects_scrubbed":6181,"log_size":5753,"log_dups_size":0,"ondisk_log_size":5753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:50.350956+0000","scrub_duration":2007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":409796608,"num_objects":6253,"num_object_clones":0,"num_object_copies":18759,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6253,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6253,"num_write_kb":400192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":694,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.488553+0000","last_change":"2024-02-05T06:42:38.598783+0000","last_active":"2024-02-05T06:44:38.488553+0000","last_peered":"2024-02-05T06:44:38.488553+0000","last_clean":"2024-02-05T06:44:38.488553+0000","last_became_active":"2024-02-05T06:37:37.137754+0000","last_became_peered":"2024-02-05T06:37:37.137754+0000","last_unstale":"2024-02-05T06:44:38.488553+0000","last_undegraded":"2024-02-05T06:44:38.488553+0000","last_fullsized":"2024-02-05T06:44:38.488553+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:42:38.598563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:42:38.598563+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.697924+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":547,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.485727+0000","last_change":"2024-02-05T06:44:03.559943+0000","last_active":"2024-02-05T06:44:38.485727+0000","last_peered":"2024-02-05T06:44:38.485727+0000","last_clean":"2024-02-05T06:44:38.485727+0000","last_became_active":"2024-02-05T06:41:43.410910+0000","last_became_peered":"2024-02-05T06:41:43.410910+0000","last_unstale":"2024-02-05T06:44:38.485727+0000","last_undegraded":"2024-02-05T06:44:38.485727+0000","last_fullsized":"2024-02-05T06:44:38.485727+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:03.559834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:44:03.559834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:09.328386+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":544,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.487141+0000","last_change":"2024-02-05T06:42:49.943397+0000","last_active":"2024-02-05T06:44:38.487141+0000","last_peered":"2024-02-05T06:44:38.487141+0000","last_clean":"2024-02-05T06:44:38.487141+0000","last_became_active":"2024-02-05T06:36:54.972264+0000","last_became_peered":"2024-02-05T06:36:54.972264+0000","last_unstale":"2024-02-05T06:44:38.487141+0000","last_undegraded":"2024-02-05T06:44:38.487141+0000","last_fullsized":"2024-02-05T06:44:38.487141+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:49.942846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:49.942846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:41.954176+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":494,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:39.456650+0000","last_change":"2024-02-05T06:43:33.504679+0000","last_active":"2024-02-05T06:44:39.456650+0000","last_peered":"2024-02-05T06:44:39.456650+0000","last_clean":"2024-02-05T06:44:39.456650+0000","last_became_active":"2024-02-05T06:40:27.071509+0000","last_became_peered":"2024-02-05T06:40:27.071509+0000","last_unstale":"2024-02-05T06:44:39.456650+0000","last_undegraded":"2024-02-05T06:44:39.456650+0000","last_fullsized":"2024-02-05T06:44:39.456650+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T06:43:33.504581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:43:33.504581+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:42.459019+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00074151600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":260,"num_read_kb":255,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"295'24","reported_seq":564,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.485152+0000","last_change":"2024-02-05T06:42:50.901662+0000","last_active":"2024-02-05T06:44:38.485152+0000","last_peered":"2024-02-05T06:44:38.485152+0000","last_clean":"2024-02-05T06:44:38.485152+0000","last_became_active":"2024-02-05T06:33:31.564604+0000","last_became_peered":"2024-02-05T06:33:31.564604+0000","last_unstale":"2024-02-05T06:44:38.485152+0000","last_undegraded":"2024-02-05T06:44:38.485152+0000","last_fullsized":"2024-02-05T06:44:38.485152+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-02-05T06:42:50.901560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:42:50.901560+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:41.953998+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":148,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.492788+0000","last_change":"2024-02-05T06:43:49.441549+0000","last_active":"2024-02-05T06:44:38.492788+0000","last_peered":"2024-02-05T06:44:38.492788+0000","last_clean":"2024-02-05T06:44:38.492788+0000","last_became_active":"2024-02-05T06:41:43.417751+0000","last_became_peered":"2024-02-05T06:41:43.417751+0000","last_unstale":"2024-02-05T06:44:38.492788+0000","last_undegraded":"2024-02-05T06:44:38.492788+0000","last_fullsized":"2024-02-05T06:44:38.492788+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:49.441328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:49.441328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:50.412164+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"312'6183","reported_seq":6235,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:41.293558+0000","last_change":"2024-02-05T06:43:29.703026+0000","last_active":"2024-02-05T06:44:41.293558+0000","last_peered":"2024-02-05T06:44:41.293558+0000","last_clean":"2024-02-05T06:44:41.293558+0000","last_became_active":"2024-02-05T06:43:29.702549+0000","last_became_peered":"2024-02-05T06:43:29.702549+0000","last_unstale":"2024-02-05T06:44:41.293558+0000","last_undegraded":"2024-02-05T06:44:41.293558+0000","last_fullsized":"2024-02-05T06:44:41.293558+0000","mapping_epoch":291,"log_start":"297'1500","ondisk_log_start":"297'1500","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4683,"log_dups_size":0,"ondisk_log_size":4683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:47.712668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405209088,"num_objects":6183,"num_object_clones":0,"num_object_copies":18549,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6183,"num_write_kb":395712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":707,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.493271+0000","last_change":"2024-02-05T06:44:04.313089+0000","last_active":"2024-02-05T06:44:38.493271+0000","last_peered":"2024-02-05T06:44:38.493271+0000","last_clean":"2024-02-05T06:44:38.493271+0000","last_became_active":"2024-02-05T06:37:37.140011+0000","last_became_peered":"2024-02-05T06:37:37.140011+0000","last_unstale":"2024-02-05T06:44:38.493271+0000","last_undegraded":"2024-02-05T06:44:38.493271+0000","last_fullsized":"2024-02-05T06:44:38.493271+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:44:04.312963+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:38.778568+0000","last_clean_scrub_stamp":"2024-02-05T06:44:04.312963+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:32.876919+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":527,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.486312+0000","last_change":"2024-02-05T06:42:41.634187+0000","last_active":"2024-02-05T06:44:38.486312+0000","last_peered":"2024-02-05T06:44:38.486312+0000","last_clean":"2024-02-05T06:44:38.486312+0000","last_became_active":"2024-02-05T06:33:35.664858+0000","last_became_peered":"2024-02-05T06:33:35.664858+0000","last_unstale":"2024-02-05T06:44:38.486312+0000","last_undegraded":"2024-02-05T06:44:38.486312+0000","last_fullsized":"2024-02-05T06:44:38.486312+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:41.634048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:42:41.634048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.697980+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"66'1","reported_seq":573,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.486623+0000","last_change":"2024-02-05T06:43:50.530854+0000","last_active":"2024-02-05T06:44:38.486623+0000","last_peered":"2024-02-05T06:44:38.486623+0000","last_clean":"2024-02-05T06:44:38.486623+0000","last_became_active":"2024-02-05T06:41:43.411404+0000","last_became_peered":"2024-02-05T06:41:43.411404+0000","last_unstale":"2024-02-05T06:44:38.486623+0000","last_undegraded":"2024-02-05T06:44:38.486623+0000","last_fullsized":"2024-02-05T06:44:38.486623+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:43:50.530734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:43:50.530734+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:06.449055+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"295'29","reported_seq":675,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.486454+0000","last_change":"2024-02-05T06:43:02.045504+0000","last_active":"2024-02-05T06:44:38.486454+0000","last_peered":"2024-02-05T06:44:38.486454+0000","last_clean":"2024-02-05T06:44:38.486454+0000","last_became_active":"2024-02-05T06:41:43.409664+0000","last_became_peered":"2024-02-05T06:41:43.409664+0000","last_unstale":"2024-02-05T06:44:38.486454+0000","last_undegraded":"2024-02-05T06:44:38.486454+0000","last_fullsized":"2024-02-05T06:44:38.486454+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-02-05T06:43:02.045346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:02.045346+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:41.954247+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":542,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.491817+0000","last_change":"2024-02-05T06:42:56.157261+0000","last_active":"2024-02-05T06:44:38.491817+0000","last_peered":"2024-02-05T06:44:38.491817+0000","last_clean":"2024-02-05T06:44:38.491817+0000","last_became_active":"2024-02-05T06:41:43.411550+0000","last_became_peered":"2024-02-05T06:41:43.411550+0000","last_unstale":"2024-02-05T06:44:38.491817+0000","last_undegraded":"2024-02-05T06:44:38.491817+0000","last_fullsized":"2024-02-05T06:44:38.491817+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:56.157130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:42:56.157130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:39.466317+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"312'6205","reported_seq":6256,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:41.296143+0000","last_change":"2024-02-05T06:43:29.700909+0000","last_active":"2024-02-05T06:44:41.296143+0000","last_peered":"2024-02-05T06:44:41.296143+0000","last_clean":"2024-02-05T06:44:41.296143+0000","last_became_active":"2024-02-05T06:43:29.700331+0000","last_became_peered":"2024-02-05T06:43:29.700331+0000","last_unstale":"2024-02-05T06:44:41.296143+0000","last_undegraded":"2024-02-05T06:44:41.296143+0000","last_fullsized":"2024-02-05T06:44:41.296143+0000","mapping_epoch":291,"log_start":"296'1200","ondisk_log_start":"296'1200","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":5005,"log_dups_size":0,"ondisk_log_size":5005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406650880,"num_objects":6205,"num_object_clones":0,"num_object_copies":18615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6205,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6205,"num_write_kb":397120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":726,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.493528+0000","last_change":"2024-02-05T06:42:49.138308+0000","last_active":"2024-02-05T06:44:38.493528+0000","last_peered":"2024-02-05T06:44:38.493528+0000","last_clean":"2024-02-05T06:44:38.493528+0000","last_became_active":"2024-02-05T06:40:30.085438+0000","last_became_peered":"2024-02-05T06:40:30.085438+0000","last_unstale":"2024-02-05T06:44:38.493528+0000","last_undegraded":"2024-02-05T06:44:38.493528+0000","last_fullsized":"2024-02-05T06:44:38.493528+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:42:49.138204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:42:49.138204+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:39.466201+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":520,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.485085+0000","last_change":"2024-02-05T06:43:42.125866+0000","last_active":"2024-02-05T06:44:38.485085+0000","last_peered":"2024-02-05T06:44:38.485085+0000","last_clean":"2024-02-05T06:44:38.485085+0000","last_became_active":"2024-02-05T06:33:35.600485+0000","last_became_peered":"2024-02-05T06:33:35.600485+0000","last_unstale":"2024-02-05T06:44:38.485085+0000","last_undegraded":"2024-02-05T06:44:38.485085+0000","last_fullsized":"2024-02-05T06:44:38.485085+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:42.125756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:43:42.125756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:04.704617+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":148,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.493240+0000","last_change":"2024-02-05T06:43:35.458031+0000","last_active":"2024-02-05T06:44:38.493240+0000","last_peered":"2024-02-05T06:44:38.493240+0000","last_clean":"2024-02-05T06:44:38.493240+0000","last_became_active":"2024-02-05T06:41:43.411732+0000","last_became_peered":"2024-02-05T06:41:43.411732+0000","last_unstale":"2024-02-05T06:44:38.493240+0000","last_undegraded":"2024-02-05T06:44:38.493240+0000","last_fullsized":"2024-02-05T06:44:38.493240+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:35.457940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:28.353956+0000","last_clean_scrub_stamp":"2024-02-05T06:43:35.457940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:04.715232+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"295'20","reported_seq":637,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.489055+0000","last_change":"2024-02-05T06:43:35.467498+0000","last_active":"2024-02-05T06:44:38.489055+0000","last_peered":"2024-02-05T06:44:38.489055+0000","last_clean":"2024-02-05T06:44:38.489055+0000","last_became_active":"2024-02-05T06:41:43.421030+0000","last_became_peered":"2024-02-05T06:41:43.421030+0000","last_unstale":"2024-02-05T06:44:38.489055+0000","last_undegraded":"2024-02-05T06:44:38.489055+0000","last_fullsized":"2024-02-05T06:44:38.489055+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-02-05T06:43:35.467372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:35.467372+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:00.825350+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"249'10","reported_seq":687,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.486046+0000","last_change":"2024-02-05T06:42:48.013461+0000","last_active":"2024-02-05T06:44:38.486046+0000","last_peered":"2024-02-05T06:44:38.486046+0000","last_clean":"2024-02-05T06:44:38.486046+0000","last_became_active":"2024-02-05T06:41:43.423638+0000","last_became_peered":"2024-02-05T06:41:43.423638+0000","last_unstale":"2024-02-05T06:44:38.486046+0000","last_undegraded":"2024-02-05T06:44:38.486046+0000","last_fullsized":"2024-02-05T06:44:38.486046+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'10","last_scrub_stamp":"2024-02-05T06:42:48.013329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:42:48.013329+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:41.953928+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"312'6202","reported_seq":6254,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:42.202312+0000","last_change":"2024-02-05T06:43:29.702358+0000","last_active":"2024-02-05T06:44:42.202312+0000","last_peered":"2024-02-05T06:44:42.202312+0000","last_clean":"2024-02-05T06:44:42.202312+0000","last_became_active":"2024-02-05T06:43:29.701716+0000","last_became_peered":"2024-02-05T06:43:29.701716+0000","last_unstale":"2024-02-05T06:44:42.202312+0000","last_undegraded":"2024-02-05T06:44:42.202312+0000","last_fullsized":"2024-02-05T06:44:42.202312+0000","mapping_epoch":291,"log_start":"297'1700","ondisk_log_start":"297'1700","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4502,"log_dups_size":0,"ondisk_log_size":4502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:49.602994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406454272,"num_objects":6202,"num_object_clones":0,"num_object_copies":18606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6202,"num_write_kb":396928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":737,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.495338+0000","last_change":"2024-02-05T06:42:49.448963+0000","last_active":"2024-02-05T06:44:38.495338+0000","last_peered":"2024-02-05T06:44:38.495338+0000","last_clean":"2024-02-05T06:44:38.495338+0000","last_became_active":"2024-02-05T06:41:46.053837+0000","last_became_peered":"2024-02-05T06:41:46.053837+0000","last_unstale":"2024-02-05T06:44:38.495338+0000","last_undegraded":"2024-02-05T06:44:38.495338+0000","last_fullsized":"2024-02-05T06:44:38.495338+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:42:49.440873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:42:49.440873+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":233,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.494387+0000","last_change":"2024-02-05T06:42:59.397727+0000","last_active":"2024-02-05T06:44:38.494387+0000","last_peered":"2024-02-05T06:44:38.494387+0000","last_clean":"2024-02-05T06:44:38.494387+0000","last_became_active":"2024-02-05T06:40:27.099168+0000","last_became_peered":"2024-02-05T06:40:27.099168+0000","last_unstale":"2024-02-05T06:44:38.494387+0000","last_undegraded":"2024-02-05T06:44:38.494387+0000","last_fullsized":"2024-02-05T06:44:38.494387+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:59.397620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:42:59.397620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.386539+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"95'2","reported_seq":554,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.485244+0000","last_change":"2024-02-05T06:43:34.938744+0000","last_active":"2024-02-05T06:44:38.485244+0000","last_peered":"2024-02-05T06:44:38.485244+0000","last_clean":"2024-02-05T06:44:38.485244+0000","last_became_active":"2024-02-05T06:36:54.975093+0000","last_became_peered":"2024-02-05T06:36:54.975093+0000","last_unstale":"2024-02-05T06:44:38.485244+0000","last_undegraded":"2024-02-05T06:44:38.485244+0000","last_fullsized":"2024-02-05T06:44:38.485244+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-05T06:43:34.938667+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-05T06:35:25.616708+0000","last_clean_scrub_stamp":"2024-02-05T06:43:34.938667+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:02.866138+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"295'28","reported_seq":497,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.487910+0000","last_change":"2024-02-05T06:42:59.726887+0000","last_active":"2024-02-05T06:44:38.487910+0000","last_peered":"2024-02-05T06:44:38.487910+0000","last_clean":"2024-02-05T06:44:38.487910+0000","last_became_active":"2024-02-05T06:41:43.411540+0000","last_became_peered":"2024-02-05T06:41:43.411540+0000","last_unstale":"2024-02-05T06:44:38.487910+0000","last_undegraded":"2024-02-05T06:44:38.487910+0000","last_fullsized":"2024-02-05T06:44:38.487910+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'14","last_scrub_stamp":"2024-02-05T06:42:59.726776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:42:59.726776+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.044375+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"119'10","reported_seq":688,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.494837+0000","last_change":"2024-02-05T06:43:01.414791+0000","last_active":"2024-02-05T06:44:38.494837+0000","last_peered":"2024-02-05T06:44:38.494837+0000","last_clean":"2024-02-05T06:44:38.494837+0000","last_became_active":"2024-02-05T06:36:54.954819+0000","last_became_peered":"2024-02-05T06:36:54.954819+0000","last_unstale":"2024-02-05T06:44:38.494837+0000","last_undegraded":"2024-02-05T06:44:38.494837+0000","last_fullsized":"2024-02-05T06:44:38.494837+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'10","last_scrub_stamp":"2024-02-05T06:43:01.414710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:01.414710+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"312'6196","reported_seq":6248,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:41.311720+0000","last_change":"2024-02-05T06:43:29.704287+0000","last_active":"2024-02-05T06:44:41.311720+0000","last_peered":"2024-02-05T06:44:41.311720+0000","last_clean":"2024-02-05T06:44:41.311720+0000","last_became_active":"2024-02-05T06:43:29.703700+0000","last_became_peered":"2024-02-05T06:43:29.703700+0000","last_unstale":"2024-02-05T06:44:41.311720+0000","last_undegraded":"2024-02-05T06:44:41.311720+0000","last_fullsized":"2024-02-05T06:44:41.311720+0000","mapping_epoch":291,"log_start":"297'1500","ondisk_log_start":"297'1500","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4696,"log_dups_size":0,"ondisk_log_size":4696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:42.458986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406061056,"num_objects":6196,"num_object_clones":0,"num_object_copies":18588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6196,"num_write_kb":396544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":686,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.487351+0000","last_change":"2024-02-05T06:41:28.466905+0000","last_active":"2024-02-05T06:44:38.487351+0000","last_peered":"2024-02-05T06:44:38.487351+0000","last_clean":"2024-02-05T06:44:38.487351+0000","last_became_active":"2024-02-05T06:37:37.135073+0000","last_became_peered":"2024-02-05T06:37:37.135073+0000","last_unstale":"2024-02-05T06:44:38.487351+0000","last_undegraded":"2024-02-05T06:44:38.487351+0000","last_fullsized":"2024-02-05T06:44:38.487351+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:41:28.466576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:41:28.466576+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.044228+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":416,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.487411+0000","last_change":"2024-02-05T06:42:45.696347+0000","last_active":"2024-02-05T06:44:38.487411+0000","last_peered":"2024-02-05T06:44:38.487411+0000","last_clean":"2024-02-05T06:44:38.487411+0000","last_became_active":"2024-02-05T06:36:54.958644+0000","last_became_peered":"2024-02-05T06:36:54.958644+0000","last_unstale":"2024-02-05T06:44:38.487411+0000","last_undegraded":"2024-02-05T06:44:38.487411+0000","last_fullsized":"2024-02-05T06:44:38.487411+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:45.696236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:24.090670+0000","last_clean_scrub_stamp":"2024-02-05T06:42:45.696236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.044408+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":549,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.495830+0000","last_change":"2024-02-05T06:42:56.396835+0000","last_active":"2024-02-05T06:44:38.495830+0000","last_peered":"2024-02-05T06:44:38.495830+0000","last_clean":"2024-02-05T06:44:38.495830+0000","last_became_active":"2024-02-05T06:40:27.093739+0000","last_became_peered":"2024-02-05T06:40:27.093739+0000","last_unstale":"2024-02-05T06:44:38.495830+0000","last_undegraded":"2024-02-05T06:44:38.495830+0000","last_fullsized":"2024-02-05T06:44:38.495830+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:56.396575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:38:28.150695+0000","last_clean_scrub_stamp":"2024-02-05T06:42:56.396575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"295'30","reported_seq":625,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.494702+0000","last_change":"2024-02-05T06:43:46.582784+0000","last_active":"2024-02-05T06:44:38.494702+0000","last_peered":"2024-02-05T06:44:38.494702+0000","last_clean":"2024-02-05T06:44:38.494702+0000","last_became_active":"2024-02-05T06:40:27.277747+0000","last_became_peered":"2024-02-05T06:40:27.277747+0000","last_unstale":"2024-02-05T06:44:38.494702+0000","last_undegraded":"2024-02-05T06:44:38.494702+0000","last_fullsized":"2024-02-05T06:44:38.494702+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'30","last_scrub_stamp":"2024-02-05T06:43:46.582265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:46.582265+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:14.477223+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":436,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.487915+0000","last_change":"2024-02-05T06:42:52.703205+0000","last_active":"2024-02-05T06:44:38.487915+0000","last_peered":"2024-02-05T06:44:38.487915+0000","last_clean":"2024-02-05T06:44:38.487915+0000","last_became_active":"2024-02-05T06:40:27.073889+0000","last_became_peered":"2024-02-05T06:40:27.073889+0000","last_unstale":"2024-02-05T06:44:38.487915+0000","last_undegraded":"2024-02-05T06:44:38.487915+0000","last_fullsized":"2024-02-05T06:44:38.487915+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:52.703089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:42:52.703089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.044437+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"312'6118","reported_seq":6170,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:42.230167+0000","last_change":"2024-02-05T06:43:29.843237+0000","last_active":"2024-02-05T06:44:42.230167+0000","last_peered":"2024-02-05T06:44:42.230167+0000","last_clean":"2024-02-05T06:44:42.230167+0000","last_became_active":"2024-02-05T06:43:29.842722+0000","last_became_peered":"2024-02-05T06:43:29.842722+0000","last_unstale":"2024-02-05T06:44:42.230167+0000","last_undegraded":"2024-02-05T06:44:42.230167+0000","last_fullsized":"2024-02-05T06:44:42.230167+0000","mapping_epoch":291,"log_start":"298'1700","ondisk_log_start":"298'1700","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4418,"log_dups_size":0,"ondisk_log_size":4418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400949248,"num_objects":6118,"num_object_clones":0,"num_object_copies":18354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6118,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6118,"num_write_kb":391552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":567,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.496019+0000","last_change":"2024-02-05T06:43:59.471960+0000","last_active":"2024-02-05T06:44:38.496019+0000","last_peered":"2024-02-05T06:44:38.496019+0000","last_clean":"2024-02-05T06:44:38.496019+0000","last_became_active":"2024-02-05T06:41:46.053993+0000","last_became_peered":"2024-02-05T06:41:46.053993+0000","last_unstale":"2024-02-05T06:44:38.496019+0000","last_undegraded":"2024-02-05T06:44:38.496019+0000","last_fullsized":"2024-02-05T06:44:38.496019+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:43:59.471749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:43:59.471749+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:04.010257+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":156,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.493159+0000","last_change":"2024-02-05T06:44:00.522165+0000","last_active":"2024-02-05T06:44:38.493159+0000","last_peered":"2024-02-05T06:44:38.493159+0000","last_clean":"2024-02-05T06:44:38.493159+0000","last_became_active":"2024-02-05T06:41:43.418066+0000","last_became_peered":"2024-02-05T06:41:43.418066+0000","last_unstale":"2024-02-05T06:44:38.493159+0000","last_undegraded":"2024-02-05T06:44:38.493159+0000","last_fullsized":"2024-02-05T06:44:38.493159+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:00.522003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:44:00.522003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:11.908885+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"249'12","reported_seq":535,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.488322+0000","last_change":"2024-02-05T06:42:56.729086+0000","last_active":"2024-02-05T06:44:38.488322+0000","last_peered":"2024-02-05T06:44:38.488322+0000","last_clean":"2024-02-05T06:44:38.488322+0000","last_became_active":"2024-02-05T06:41:43.411973+0000","last_became_peered":"2024-02-05T06:41:43.411973+0000","last_unstale":"2024-02-05T06:44:38.488322+0000","last_undegraded":"2024-02-05T06:44:38.488322+0000","last_fullsized":"2024-02-05T06:44:38.488322+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'12","last_scrub_stamp":"2024-02-05T06:42:56.728967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:42:56.728967+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.044392+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":543,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.494620+0000","last_change":"2024-02-05T06:43:43.544887+0000","last_active":"2024-02-05T06:44:38.494620+0000","last_peered":"2024-02-05T06:44:38.494620+0000","last_clean":"2024-02-05T06:44:38.494620+0000","last_became_active":"2024-02-05T06:40:27.213439+0000","last_became_peered":"2024-02-05T06:40:27.213439+0000","last_unstale":"2024-02-05T06:44:38.494620+0000","last_undegraded":"2024-02-05T06:44:38.494620+0000","last_fullsized":"2024-02-05T06:44:38.494620+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:43.544765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:43:43.544765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:50.636394+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"295'34","reported_seq":338,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.485455+0000","last_change":"2024-02-05T06:42:59.970087+0000","last_active":"2024-02-05T06:44:38.485455+0000","last_peered":"2024-02-05T06:44:38.485455+0000","last_clean":"2024-02-05T06:44:38.485455+0000","last_became_active":"2024-02-05T06:39:13.257978+0000","last_became_peered":"2024-02-05T06:39:13.257978+0000","last_unstale":"2024-02-05T06:44:38.485455+0000","last_undegraded":"2024-02-05T06:44:38.485455+0000","last_fullsized":"2024-02-05T06:44:38.485455+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'22","last_scrub_stamp":"2024-02-05T06:42:59.969994+0000","last_deep_scrub":"281'22","last_deep_scrub_stamp":"2024-02-05T06:42:59.969994+0000","last_clean_scrub_stamp":"2024-02-05T06:42:59.969994+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:41.954032+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.3","version":"312'6139","reported_seq":6191,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:41.304194+0000","last_change":"2024-02-05T06:43:29.702128+0000","last_active":"2024-02-05T06:44:41.304194+0000","last_peered":"2024-02-05T06:44:41.304194+0000","last_clean":"2024-02-05T06:44:41.304194+0000","last_became_active":"2024-02-05T06:43:29.701677+0000","last_became_peered":"2024-02-05T06:43:29.701677+0000","last_unstale":"2024-02-05T06:44:41.304194+0000","last_undegraded":"2024-02-05T06:44:41.304194+0000","last_fullsized":"2024-02-05T06:44:41.304194+0000","mapping_epoch":291,"log_start":"297'1500","ondisk_log_start":"297'1500","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4639,"log_dups_size":0,"ondisk_log_size":4639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":402325504,"num_objects":6139,"num_object_clones":0,"num_object_copies":18417,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6139,"num_write_kb":392896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":536,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.495641+0000","last_change":"2024-02-05T06:43:42.559000+0000","last_active":"2024-02-05T06:44:38.495641+0000","last_peered":"2024-02-05T06:44:38.495641+0000","last_clean":"2024-02-05T06:44:38.495641+0000","last_became_active":"2024-02-05T06:41:43.410600+0000","last_became_peered":"2024-02-05T06:41:43.410600+0000","last_unstale":"2024-02-05T06:44:38.495641+0000","last_undegraded":"2024-02-05T06:44:38.495641+0000","last_fullsized":"2024-02-05T06:44:38.495641+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:42.558880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:20.042534+0000","last_clean_scrub_stamp":"2024-02-05T06:43:42.558880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:54.376149+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":148,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.492632+0000","last_change":"2024-02-05T06:42:55.610389+0000","last_active":"2024-02-05T06:44:38.492632+0000","last_peered":"2024-02-05T06:44:38.492632+0000","last_clean":"2024-02-05T06:44:38.492632+0000","last_became_active":"2024-02-05T06:41:43.417343+0000","last_became_peered":"2024-02-05T06:41:43.417343+0000","last_unstale":"2024-02-05T06:44:38.492632+0000","last_undegraded":"2024-02-05T06:44:38.492632+0000","last_fullsized":"2024-02-05T06:44:38.492632+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:55.610276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:42:55.610276+0000","last_clean_scrub_stamp":"2024-02-05T06:42:55.610276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":258,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.485973+0000","last_change":"2024-02-05T06:44:01.550712+0000","last_active":"2024-02-05T06:44:38.485973+0000","last_peered":"2024-02-05T06:44:38.485973+0000","last_clean":"2024-02-05T06:44:38.485973+0000","last_became_active":"2024-02-05T06:40:27.086808+0000","last_became_peered":"2024-02-05T06:40:27.086808+0000","last_unstale":"2024-02-05T06:44:38.485973+0000","last_undegraded":"2024-02-05T06:44:38.485973+0000","last_fullsized":"2024-02-05T06:44:38.485973+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:01.550518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:44:01.550518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:15.652514+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"295'24","reported_seq":440,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.488086+0000","last_change":"2024-02-05T06:43:57.825625+0000","last_active":"2024-02-05T06:44:38.488086+0000","last_peered":"2024-02-05T06:44:38.488086+0000","last_clean":"2024-02-05T06:44:38.488086+0000","last_became_active":"2024-02-05T06:36:54.958582+0000","last_became_peered":"2024-02-05T06:36:54.958582+0000","last_unstale":"2024-02-05T06:44:38.488086+0000","last_undegraded":"2024-02-05T06:44:38.488086+0000","last_fullsized":"2024-02-05T06:44:38.488086+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:43:57.825491+0000","last_deep_scrub":"115'16","last_deep_scrub_stamp":"2024-02-05T06:36:34.793243+0000","last_clean_scrub_stamp":"2024-02-05T06:43:57.825491+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:21.940154+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"312'6218","reported_seq":6270,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:41.865917+0000","last_change":"2024-02-05T06:43:29.947307+0000","last_active":"2024-02-05T06:44:41.865917+0000","last_peered":"2024-02-05T06:44:41.865917+0000","last_clean":"2024-02-05T06:44:41.865917+0000","last_became_active":"2024-02-05T06:43:29.946914+0000","last_became_peered":"2024-02-05T06:43:29.946914+0000","last_unstale":"2024-02-05T06:44:41.865917+0000","last_undegraded":"2024-02-05T06:44:41.865917+0000","last_fullsized":"2024-02-05T06:44:41.865917+0000","mapping_epoch":291,"log_start":"297'1600","ondisk_log_start":"297'1600","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:52.324646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407502848,"num_objects":6218,"num_object_clones":0,"num_object_copies":18654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6218,"num_write_kb":397952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"83'128","reported_seq":571,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.492156+0000","last_change":"2024-02-05T06:43:47.429770+0000","last_active":"2024-02-05T06:44:38.492156+0000","last_peered":"2024-02-05T06:44:38.492156+0000","last_clean":"2024-02-05T06:44:38.492156+0000","last_became_active":"2024-02-05T06:41:46.053324+0000","last_became_peered":"2024-02-05T06:41:46.053324+0000","last_unstale":"2024-02-05T06:44:38.492156+0000","last_undegraded":"2024-02-05T06:44:38.492156+0000","last_fullsized":"2024-02-05T06:44:38.492156+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:43:47.429623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:43:47.429623+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:09.999835+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":552,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.497893+0000","last_change":"2024-02-05T06:43:46.502730+0000","last_active":"2024-02-05T06:44:38.497893+0000","last_peered":"2024-02-05T06:44:38.497893+0000","last_clean":"2024-02-05T06:44:38.497893+0000","last_became_active":"2024-02-05T06:36:54.958805+0000","last_became_peered":"2024-02-05T06:36:54.958805+0000","last_unstale":"2024-02-05T06:44:38.497893+0000","last_undegraded":"2024-02-05T06:44:38.497893+0000","last_fullsized":"2024-02-05T06:44:38.497893+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:46.502588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:43:46.502588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:45:05.939786+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":542,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.487765+0000","last_change":"2024-02-05T06:42:46.707508+0000","last_active":"2024-02-05T06:44:38.487765+0000","last_peered":"2024-02-05T06:44:38.487765+0000","last_clean":"2024-02-05T06:44:38.487765+0000","last_became_active":"2024-02-05T06:36:54.958082+0000","last_became_peered":"2024-02-05T06:36:54.958082+0000","last_unstale":"2024-02-05T06:44:38.487765+0000","last_undegraded":"2024-02-05T06:44:38.487765+0000","last_fullsized":"2024-02-05T06:44:38.487765+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:46.707382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:42:46.707382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.697953+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":564,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:39.569879+0000","last_change":"2024-02-05T06:44:39.569920+0000","last_active":"2024-02-05T06:44:39.569879+0000","last_peered":"2024-02-05T06:44:39.569879+0000","last_clean":"2024-02-05T06:44:39.569879+0000","last_became_active":"2024-02-05T06:39:13.240038+0000","last_became_peered":"2024-02-05T06:39:13.240038+0000","last_unstale":"2024-02-05T06:44:39.569879+0000","last_undegraded":"2024-02-05T06:44:39.569879+0000","last_fullsized":"2024-02-05T06:44:39.569879+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:39.569779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:44:39.569779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:03.400020+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"295'39","reported_seq":455,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.487900+0000","last_change":"2024-02-05T06:41:54.656081+0000","last_active":"2024-02-05T06:44:38.487900+0000","last_peered":"2024-02-05T06:44:38.487900+0000","last_clean":"2024-02-05T06:44:38.487900+0000","last_became_active":"2024-02-05T06:36:54.961719+0000","last_became_peered":"2024-02-05T06:36:54.961719+0000","last_unstale":"2024-02-05T06:44:38.487900+0000","last_undegraded":"2024-02-05T06:44:38.487900+0000","last_fullsized":"2024-02-05T06:44:38.487900+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'25","last_scrub_stamp":"2024-02-05T06:41:54.655986+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-02-05T06:35:26.274944+0000","last_clean_scrub_stamp":"2024-02-05T06:41:54.655986+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.044210+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"312'6207","reported_seq":6259,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:42.183378+0000","last_change":"2024-02-05T06:43:29.947556+0000","last_active":"2024-02-05T06:44:42.183378+0000","last_peered":"2024-02-05T06:44:42.183378+0000","last_clean":"2024-02-05T06:44:42.183378+0000","last_became_active":"2024-02-05T06:43:29.947136+0000","last_became_peered":"2024-02-05T06:43:29.947136+0000","last_unstale":"2024-02-05T06:44:42.183378+0000","last_undegraded":"2024-02-05T06:44:42.183378+0000","last_fullsized":"2024-02-05T06:44:42.183378+0000","mapping_epoch":291,"log_start":"298'1700","ondisk_log_start":"298'1700","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4507,"log_dups_size":0,"ondisk_log_size":4507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406781952,"num_objects":6207,"num_object_clones":0,"num_object_copies":18621,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6207,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6207,"num_write_kb":397248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":650,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.493461+0000","last_change":"2024-02-05T06:43:47.527180+0000","last_active":"2024-02-05T06:44:38.493461+0000","last_peered":"2024-02-05T06:44:38.493461+0000","last_clean":"2024-02-05T06:44:38.493461+0000","last_became_active":"2024-02-05T06:40:30.095202+0000","last_became_peered":"2024-02-05T06:40:30.095202+0000","last_unstale":"2024-02-05T06:44:38.493461+0000","last_undegraded":"2024-02-05T06:44:38.493461+0000","last_fullsized":"2024-02-05T06:44:38.493461+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:43:47.527064+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:41:50.706159+0000","last_clean_scrub_stamp":"2024-02-05T06:43:47.527064+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:55.073159+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":416,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.488164+0000","last_change":"2024-02-05T06:42:43.732921+0000","last_active":"2024-02-05T06:44:38.488164+0000","last_peered":"2024-02-05T06:44:38.488164+0000","last_clean":"2024-02-05T06:44:38.488164+0000","last_became_active":"2024-02-05T06:36:54.961560+0000","last_became_peered":"2024-02-05T06:36:54.961560+0000","last_unstale":"2024-02-05T06:44:38.488164+0000","last_undegraded":"2024-02-05T06:44:38.488164+0000","last_fullsized":"2024-02-05T06:44:38.488164+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:43.732781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:30.057099+0000","last_clean_scrub_stamp":"2024-02-05T06:42:43.732781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.044327+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"249'13","reported_seq":728,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.532844+0000","last_change":"2024-02-05T06:44:38.532844+0000","last_active":"2024-02-05T06:44:38.532844+0000","last_peered":"2024-02-05T06:44:38.532844+0000","last_clean":"2024-02-05T06:44:38.532844+0000","last_became_active":"2024-02-05T06:41:43.413139+0000","last_became_peered":"2024-02-05T06:41:43.413139+0000","last_unstale":"2024-02-05T06:44:38.532844+0000","last_undegraded":"2024-02-05T06:44:38.532844+0000","last_fullsized":"2024-02-05T06:44:38.532844+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'13","last_scrub_stamp":"2024-02-05T06:44:38.532716+0000","last_deep_scrub":"92'9","last_deep_scrub_stamp":"2024-02-05T06:34:52.283780+0000","last_clean_scrub_stamp":"2024-02-05T06:44:38.532716+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:04.335894+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":542,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.495238+0000","last_change":"2024-02-05T06:42:39.205395+0000","last_active":"2024-02-05T06:44:38.495238+0000","last_peered":"2024-02-05T06:44:38.495238+0000","last_clean":"2024-02-05T06:44:38.495238+0000","last_became_active":"2024-02-05T06:36:54.971304+0000","last_became_peered":"2024-02-05T06:36:54.971304+0000","last_unstale":"2024-02-05T06:44:38.495238+0000","last_undegraded":"2024-02-05T06:44:38.495238+0000","last_fullsized":"2024-02-05T06:44:38.495238+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:39.205140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:42:39.205140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"295'25","reported_seq":443,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-02-05T06:44:38.488611+0000","last_change":"2024-02-05T06:41:37.498015+0000","last_active":"2024-02-05T06:44:38.488611+0000","last_peered":"2024-02-05T06:44:38.488611+0000","last_clean":"2024-02-05T06:44:38.488611+0000","last_became_active":"2024-02-05T06:36:54.963620+0000","last_became_peered":"2024-02-05T06:36:54.963620+0000","last_unstale":"2024-02-05T06:44:38.488611+0000","last_undegraded":"2024-02-05T06:44:38.488611+0000","last_fullsized":"2024-02-05T06:44:38.488611+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'15","last_scrub_stamp":"2024-02-05T06:41:37.497882+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-02-05T06:35:04.584657+0000","last_clean_scrub_stamp":"2024-02-05T06:41:37.497882+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:44:40.044246+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":6493962240,"num_objects":99090,"num_object_clones":0,"num_object_copies":297270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99090,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":99090,"num_write_kb":6341760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19464781824,"data_stored":19464781824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78490,"ondisk_log_size":78490,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":260,"num_read_kb":255,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1573408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1114,"num_read_kb":871,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":373,"num_bytes_recovered":8234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":761,"ondisk_log_size":761,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103306,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3893260,"kb_used_data":3506692,"kb_used_omap":1,"kb_used_meta":386558,"kb_avail":89851892,"statfs":{"total":95995035648,"available":92008337408,"internally_reserved":0,"allocated":3590852608,"data_stored":3590214154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":395835850},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:43:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":1.198,"15min":1.292},"min":{"1min":0.29899999999999999,"5min":0.23699999999999999,"15min":0.219},"max":{"1min":2.2530000000000001,"5min":32.567,"15min":68.921000000000006},"last":4.1100000000000003},{"interface":"front","average":{"1min":1.946,"5min":1.528,"15min":1.4630000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":18.013000000000002,"5min":32.603000000000002,"15min":69.616},"last":0.41299999999999998}]},{"osd":1,"last update":"Mon Feb 5 06:43:49 2024","interfaces":[{"interface":"back","average":{"1min":2.206,"5min":1.577,"15min":1.2789999999999999},"min":{"1min":0.50900000000000001,"5min":0.25,"15min":0.23599999999999999},"max":{"1min":17.969999999999999,"5min":32.459000000000003,"15min":89.231999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.724,"15min":1.369},"min":{"1min":0.372,"5min":0.26300000000000001,"15min":0.26100000000000001},"max":{"1min":4.0300000000000002,"5min":32.536000000000001,"15min":87.995999999999995},"last":1.0269999999999999}]},{"osd":2,"last update":"Mon Feb 5 06:43:49 2024","interfaces":[{"interface":"back","average":{"1min":2.1920000000000002,"5min":1.4330000000000001,"15min":1.1279999999999999},"min":{"1min":0.37,"5min":0.28399999999999997,"15min":0.26200000000000001},"max":{"1min":18.071999999999999,"5min":18.071999999999999,"15min":53.947000000000003},"last":0.39300000000000002},{"interface":"front","average":{"1min":2.0739999999999998,"5min":1.1990000000000001,"15min":1.085},"min":{"1min":0.44700000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":18.045999999999999,"5min":18.045999999999999,"15min":53.299999999999997},"last":0.48999999999999999}]},{"osd":3,"last update":"Mon Feb 5 06:43:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.593,"15min":1.6060000000000001},"min":{"1min":0.39400000000000002,"5min":0.32700000000000001,"15min":0.316},"max":{"1min":3.4990000000000001,"5min":19.513000000000002,"15min":89.481999999999999},"last":4.5190000000000001},{"interface":"front","average":{"1min":1.167,"5min":1.2490000000000001,"15min":1.5569999999999999},"min":{"1min":0.47799999999999998,"5min":0.30599999999999999,"15min":0.27400000000000002},"max":{"1min":3.681,"5min":12.034000000000001,"15min":92.293999999999997},"last":4.423}]},{"osd":4,"last update":"Mon Feb 5 06:43:49 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.496},"min":{"1min":0.28499999999999998,"5min":0.26200000000000001,"15min":0.24199999999999999},"max":{"1min":0.90700000000000003,"5min":2.2349999999999999,"15min":3.5409999999999999},"last":0.373},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.54000000000000004,"15min":0.50900000000000001},"min":{"1min":0.33200000000000002,"5min":0.27100000000000002,"15min":0.23400000000000001},"max":{"1min":2.9039999999999999,"5min":2.9039999999999999,"15min":2.9039999999999999},"last":0.34599999999999997}]},{"osd":5,"last update":"Mon Feb 5 06:43:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.629,"15min":0.55000000000000004},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.25900000000000001},"max":{"1min":3.444,"5min":5.1050000000000004,"15min":5.1050000000000004},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.59499999999999997,"15min":0.57899999999999996},"min":{"1min":0.36099999999999999,"5min":0.30299999999999999,"15min":0.252},"max":{"1min":2.746,"5min":2.746,"15min":3.5990000000000002},"last":0.47899999999999998}]},{"osd":6,"last update":"Mon Feb 5 06:43:49 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.67300000000000004,"15min":0.61799999999999999},"min":{"1min":0.38800000000000001,"5min":0.32000000000000001,"15min":0.314},"max":{"1min":3.2869999999999999,"5min":3.2869999999999999,"15min":3.5750000000000002},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.63300000000000001,"15min":0.58499999999999996},"min":{"1min":0.38200000000000001,"5min":0.307,"15min":0.245},"max":{"1min":3.0649999999999999,"5min":3.0649999999999999,"15min":3.2189999999999999},"last":0.42999999999999999}]}]},{"osd":6,"up_from":49,"seq":210453397648,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1503092,"kb_used_data":1178988,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":92242060,"statfs":{"total":95995035648,"available":94455869440,"internally_reserved":0,"allocated":1207283712,"data_stored":1206668385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":331872710},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:44:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":4.2670000000000003,"15min":2.0510000000000002},"min":{"1min":0.255,"5min":0.224,"15min":0.224},"max":{"1min":1.3560000000000001,"5min":98.918999999999997,"15min":98.918999999999997},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.66600000000000004,"5min":4.3739999999999997,"15min":2.0190000000000001},"min":{"1min":0.245,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.9039999999999999,"5min":98.775000000000006,"15min":98.775000000000006},"last":0.42899999999999999}]},{"osd":1,"last update":"Mon Feb 5 06:44:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":3.8380000000000001,"15min":1.776},"min":{"1min":0.27200000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.119,"5min":98.843999999999994,"15min":98.843999999999994},"last":3.1379999999999999},{"interface":"front","average":{"1min":0.80300000000000005,"5min":4.0300000000000002,"15min":1.7969999999999999},"min":{"1min":0.436,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":2.0129999999999999,"5min":101.637,"15min":101.637},"last":0.622}]},{"osd":2,"last update":"Mon Feb 5 06:44:11 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":3.5470000000000002,"15min":1.7290000000000001},"min":{"1min":0.33900000000000002,"5min":0.29199999999999998,"15min":0.22500000000000001},"max":{"1min":1.9390000000000001,"5min":85.617000000000004,"15min":85.617000000000004},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":3.383,"15min":1.6259999999999999},"min":{"1min":0.32600000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.365,"5min":85.335999999999999,"15min":85.335999999999999},"last":0.46800000000000003}]},{"osd":3,"last update":"Mon Feb 5 06:44:11 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":3.8809999999999998,"15min":2.21},"min":{"1min":0.32000000000000001,"5min":0.307,"15min":0.27000000000000002},"max":{"1min":4.1150000000000002,"5min":98.888999999999996,"15min":99.269000000000005},"last":3.1789999999999998},{"interface":"front","average":{"1min":0.79900000000000004,"5min":4.0099999999999998,"15min":2.3109999999999999},"min":{"1min":0.40799999999999997,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":2.8410000000000002,"5min":98.969999999999999,"15min":101.75700000000001},"last":0.65100000000000002}]},{"osd":4,"last update":"Mon Feb 5 06:44:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55900000000000005,"15min":0.48799999999999999},"min":{"1min":0.308,"5min":0.26500000000000001,"15min":0.23100000000000001},"max":{"1min":1.391,"5min":3.742,"15min":3.742},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.53500000000000003,"15min":0.47399999999999998},"min":{"1min":0.30099999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.8480000000000001,"5min":2.371,"15min":2.371},"last":3.2149999999999999}]},{"osd":5,"last update":"Mon Feb 5 06:44:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.60899999999999999,"15min":0.51900000000000002},"min":{"1min":0.35299999999999998,"5min":0.29899999999999999,"15min":0.251},"max":{"1min":1.4450000000000001,"5min":4.4740000000000002,"15min":4.4740000000000002},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.628,"15min":0.52500000000000002},"min":{"1min":0.34999999999999998,"5min":0.28699999999999998,"15min":0.24299999999999999},"max":{"1min":1.895,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":3.2450000000000001}]},{"osd":7,"last update":"Mon Feb 5 06:44:32 2024","interfaces":[{"interface":"back","average":{"1min":2.516,"5min":0.96899999999999997,"15min":0.68000000000000005},"min":{"1min":0.39800000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":38.436999999999998,"5min":38.436999999999998,"15min":38.436999999999998},"last":2.3879999999999999},{"interface":"front","average":{"1min":2.5110000000000001,"5min":0.99099999999999999,"15min":0.73399999999999999},"min":{"1min":0.40000000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":36.633000000000003,"5min":36.633000000000003,"15min":36.633000000000003},"last":2.3479999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575017,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3963564,"kb_used_data":3575588,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":89781588,"statfs":{"total":95995035648,"available":91936346112,"internally_reserved":0,"allocated":3661402112,"data_stored":3660731673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":397277637},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:44:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":2.0859999999999999,"15min":1.5569999999999999},"min":{"1min":0.247,"5min":0.183,"15min":0.16700000000000001},"max":{"1min":2.4980000000000002,"5min":43.651000000000003,"15min":67.421000000000006},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.80900000000000005,"5min":2.5219999999999998,"15min":1.669},"min":{"1min":0.28499999999999998,"5min":0.20599999999999999,"15min":0.19600000000000001},"max":{"1min":4.3550000000000004,"5min":45.670999999999999,"15min":67.885000000000005},"last":1.2789999999999999}]},{"osd":2,"last update":"Mon Feb 5 06:44:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.9690000000000001,"15min":1.349},"min":{"1min":0.26600000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":2.9220000000000002,"5min":43.447000000000003,"15min":43.447000000000003},"last":0.76000000000000001},{"interface":"front","average":{"1min":1.23,"5min":1.6379999999999999,"15min":1.3520000000000001},"min":{"1min":0.28699999999999998,"5min":0.249,"15min":0.20899999999999999},"max":{"1min":6.452,"5min":22.050000000000001,"15min":22.050000000000001},"last":0.47899999999999998}]},{"osd":3,"last update":"Mon Feb 5 06:44:00 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":2.4249999999999998,"15min":1.8500000000000001},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.25600000000000001},"max":{"1min":3.827,"5min":25.236999999999998,"15min":69.798000000000002},"last":2.5790000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":2.1200000000000001,"15min":1.625},"min":{"1min":0.35399999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9059999999999999,"5min":43.354999999999997,"15min":70.765000000000001},"last":2.4089999999999998}]},{"osd":4,"last update":"Mon Feb 5 06:44:18 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":1.1100000000000001,"15min":1.0189999999999999},"min":{"1min":0.433,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":3.452,"5min":7.0519999999999996,"15min":11.679},"last":0.54000000000000004},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.377,"15min":1.056},"min":{"1min":0.53800000000000003,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":4.9610000000000003,"5min":6.0730000000000004,"15min":6.0730000000000004},"last":1.2050000000000001}]},{"osd":5,"last update":"Mon Feb 5 06:43:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.5469999999999999,"15min":1.1830000000000001},"min":{"1min":0.44900000000000001,"5min":0.29399999999999998,"15min":0.29299999999999998},"max":{"1min":3.8540000000000001,"5min":43.414999999999999,"15min":43.414999999999999},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.2230000000000001,"15min":1.097},"min":{"1min":0.36599999999999999,"5min":0.33500000000000002,"15min":0.33100000000000002},"max":{"1min":4.9459999999999997,"5min":7.0919999999999996,"15min":13.455},"last":0.63900000000000001}]},{"osd":6,"last update":"Mon Feb 5 06:44:10 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":1.2,"15min":1.1519999999999999},"min":{"1min":0.40999999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.282,"5min":7.1260000000000003,"15min":13.391},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.879,"5min":1.5429999999999999,"15min":1.1220000000000001},"min":{"1min":0.46200000000000002,"5min":0.38100000000000001,"15min":0.379},"max":{"1min":1.9870000000000001,"5min":43.475000000000001,"15min":43.475000000000001},"last":0.97699999999999998}]},{"osd":7,"last update":"Mon Feb 5 06:44:28 2024","interfaces":[{"interface":"back","average":{"1min":1.6850000000000001,"5min":1.335,"15min":1.165},"min":{"1min":0.52700000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":6.4950000000000001,"5min":7.157,"15min":13.49},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.5229999999999999,"15min":1.258},"min":{"1min":0.504,"5min":0.40400000000000003,"15min":0.35499999999999998},"max":{"1min":3.6120000000000001,"5min":9.3249999999999993,"15min":9.3249999999999993},"last":1.2410000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738543,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1939292,"kb_used_data":1587412,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":91805860,"statfs":{"total":95995035648,"available":94009200640,"internally_reserved":0,"allocated":1625509888,"data_stored":1624855026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 06:44:10 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":1.655,"15min":1.1970000000000001},"min":{"1min":0.186,"5min":0.182,"15min":0.182},"max":{"1min":1.9670000000000001,"5min":18.59,"15min":18.59},"last":2.5649999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":1.637,"15min":1.159},"min":{"1min":0.221,"5min":0.193,"15min":0.193},"max":{"1min":2.2400000000000002,"5min":18.300999999999998,"15min":18.300999999999998},"last":2.028}]},{"osd":2,"last update":"Mon Feb 5 06:44:27 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":1.415,"15min":1.2150000000000001},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":2.456,"5min":16.981999999999999,"15min":16.981999999999999},"last":2.1139999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":1.347,"15min":1.1479999999999999},"min":{"1min":0.255,"5min":0.223,"15min":0.223},"max":{"1min":1.3520000000000001,"5min":16.998999999999999,"15min":16.998999999999999},"last":0.45600000000000002}]},{"osd":3,"last update":"Mon Feb 5 06:43:54 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":1.998,"15min":1.399},"min":{"1min":0.27300000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.6639999999999999,"5min":43.302,"15min":43.302},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":2.4689999999999999,"15min":1.548},"min":{"1min":0.26500000000000001,"5min":0.26000000000000001,"15min":0.254},"max":{"1min":2.3410000000000002,"5min":43.651000000000003,"15min":43.651000000000003},"last":2.3759999999999999}]},{"osd":4,"last update":"Mon Feb 5 06:44:22 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":1.5549999999999999,"15min":1.165},"min":{"1min":0.32900000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.806,"5min":40.789999999999999,"15min":40.789999999999999},"last":3.5880000000000001},{"interface":"front","average":{"1min":1.006,"5min":1.7430000000000001,"15min":1.2370000000000001},"min":{"1min":0.34599999999999997,"5min":0.32700000000000001,"15min":0.30099999999999999},"max":{"1min":8.4019999999999992,"5min":40.773000000000003,"15min":40.773000000000003},"last":3.032}]},{"osd":5,"last update":"Mon Feb 5 06:43:45 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":1.47,"15min":1.0429999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.29299999999999998},"max":{"1min":2.3929999999999998,"5min":18.170999999999999,"15min":18.170999999999999},"last":4.2640000000000002},{"interface":"front","average":{"1min":0.623,"5min":1.276,"15min":1.048},"min":{"1min":0.40899999999999997,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.145,"5min":17.152999999999999,"15min":17.152999999999999},"last":2.0600000000000001}]},{"osd":6,"last update":"Mon Feb 5 06:44:10 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":1.367,"15min":1.1140000000000001},"min":{"1min":0.42999999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":2.0990000000000002,"5min":18.562999999999999,"15min":18.562999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":1.5009999999999999,"5min":1.5189999999999999,"15min":1.133},"min":{"1min":0.44900000000000001,"5min":0.372,"15min":0.33000000000000002},"max":{"1min":13.102,"5min":18.213999999999999,"15min":18.213999999999999},"last":0.79300000000000004}]},{"osd":7,"last update":"Mon Feb 5 06:44:34 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":1.2589999999999999,"15min":1.0700000000000001},"min":{"1min":0.40100000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":2.698,"5min":17.210000000000001,"15min":17.210000000000001},"last":2.0859999999999999},{"interface":"front","average":{"1min":0.68899999999999995,"5min":1.3520000000000001,"15min":1.2270000000000001},"min":{"1min":0.47199999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.071,"5min":18.533999999999999,"15min":18.533999999999999},"last":2.004}]}]},{"osd":2,"up_from":18,"seq":77309411492,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1557060,"kb_used_data":1188540,"kb_used_omap":1,"kb_used_meta":368510,"kb_avail":92188092,"statfs":{"total":95995035648,"available":94400606208,"internally_reserved":0,"allocated":1217064960,"data_stored":1216435111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":377354693},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:44:32 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.3160000000000001,"15min":1.5640000000000001},"min":{"1min":0.20899999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":7.1529999999999996,"5min":8.9339999999999993,"15min":50.814},"last":1.8859999999999999},{"interface":"front","average":{"1min":0.88800000000000001,"5min":1.5409999999999999,"15min":1.534},"min":{"1min":0.23100000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":6.5780000000000003,"5min":10.026,"15min":50.802999999999997},"last":0.49099999999999999}]},{"osd":1,"last update":"Mon Feb 5 06:44:32 2024","interfaces":[{"interface":"back","average":{"1min":1.891,"5min":1.5509999999999999,"15min":1.3839999999999999},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":15.382,"5min":15.382,"15min":31.303000000000001},"last":2.4700000000000002},{"interface":"front","average":{"1min":2.2120000000000002,"5min":1.319,"15min":1.169},"min":{"1min":0.25,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":15.742000000000001,"5min":15.742000000000001,"15min":31.425000000000001},"last":0.84399999999999997}]},{"osd":3,"last update":"Mon Feb 5 06:44:00 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":1.5820000000000001,"15min":1.621},"min":{"1min":0.30299999999999999,"5min":0.25700000000000001,"15min":0.254},"max":{"1min":4.0739999999999998,"5min":13.504,"15min":53.127000000000002},"last":4.7359999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":1.631,"15min":1.546},"min":{"1min":0.251,"5min":0.23599999999999999,"15min":0.22500000000000001},"max":{"1min":1.929,"5min":11.388999999999999,"15min":52.607999999999997},"last":1.486}]},{"osd":4,"last update":"Mon Feb 5 06:44:19 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":1.044,"15min":0.93899999999999995},"min":{"1min":0.374,"5min":0.29699999999999999,"15min":0.23999999999999999},"max":{"1min":2.448,"5min":12.041,"15min":12.041},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.71099999999999997,"5min":1.0629999999999999,"15min":0.91900000000000004},"min":{"1min":0.33100000000000002,"5min":0.30199999999999999,"15min":0.29199999999999998},"max":{"1min":2.4609999999999999,"5min":15.689,"15min":15.689},"last":0.72099999999999997}]},{"osd":5,"last update":"Mon Feb 5 06:43:45 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":2.2010000000000001,"15min":1.2929999999999999},"min":{"1min":0.38200000000000001,"5min":0.316,"15min":0.29899999999999999},"max":{"1min":1.431,"5min":86.614999999999995,"15min":86.614999999999995},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.70599999999999996,"5min":2.5110000000000001,"15min":1.4710000000000001},"min":{"1min":0.372,"5min":0.33600000000000002,"15min":0.33200000000000002},"max":{"1min":1.363,"5min":86.543999999999997,"15min":86.543999999999997},"last":2.4430000000000001}]},{"osd":6,"last update":"Mon Feb 5 06:44:19 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.97899999999999998,"15min":1.264},"min":{"1min":0.40000000000000002,"5min":0.315,"15min":0.315},"max":{"1min":2.4319999999999999,"5min":6.665,"15min":17.143999999999998},"last":0.753},{"interface":"front","average":{"1min":0.70299999999999996,"5min":1.1839999999999999,"15min":1.1220000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":2.0489999999999999,"5min":6.7919999999999998,"15min":10.27},"last":2.411}]},{"osd":7,"last update":"Mon Feb 5 06:44:39 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":1.232,"15min":1.1299999999999999},"min":{"1min":0.39400000000000002,"5min":0.377,"15min":0.35699999999999998},"max":{"1min":2.3690000000000002,"5min":6.8609999999999998,"15min":10.206},"last":2.3690000000000002},{"interface":"front","average":{"1min":1.006,"5min":1.0940000000000001,"15min":1.0269999999999999},"min":{"1min":0.438,"5min":0.379,"15min":0.36899999999999999},"max":{"1min":3.3719999999999999,"5min":5.8639999999999999,"15min":6.4749999999999996},"last":1.046}]}]},{"osd":3,"up_from":26,"seq":111669149855,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3651800,"kb_used_data":3169424,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":90093352,"statfs":{"total":95995035648,"available":92255592448,"internally_reserved":0,"allocated":3245490176,"data_stored":3244839310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":493943239},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:44:07 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.4019999999999999,"15min":1.073},"min":{"1min":0.20499999999999999,"5min":0.184,"15min":0.184},"max":{"1min":9.6219999999999999,"5min":9.6219999999999999,"15min":13.391999999999999},"last":1.6639999999999999},{"interface":"front","average":{"1min":0.999,"5min":1.5049999999999999,"15min":1.3160000000000001},"min":{"1min":0.307,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":4.5369999999999999,"5min":7.0659999999999998,"15min":25.087},"last":2.0070000000000001}]},{"osd":1,"last update":"Mon Feb 5 06:44:07 2024","interfaces":[{"interface":"back","average":{"1min":1.7789999999999999,"5min":1.7669999999999999,"15min":1.262},"min":{"1min":0.25800000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":11.412000000000001,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.8069999999999999,"15min":1.23},"min":{"1min":0.27300000000000002,"5min":0.215,"15min":0.215},"max":{"1min":3.5830000000000002,"5min":8.5359999999999996,"15min":8.5359999999999996},"last":1.72}]},{"osd":2,"last update":"Mon Feb 5 06:44:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":2.081,"15min":1.3839999999999999},"min":{"1min":0.31900000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":3.3250000000000002,"5min":41.529000000000003,"15min":41.529000000000003},"last":3.0129999999999999},{"interface":"front","average":{"1min":0.98299999999999998,"5min":2.0049999999999999,"15min":1.294},"min":{"1min":0.27400000000000002,"5min":0.25,"15min":0.248},"max":{"1min":4.8049999999999997,"5min":41.759999999999998,"15min":41.759999999999998},"last":1.244}]},{"osd":4,"last update":"Mon Feb 5 06:44:28 2024","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.1120000000000001,"15min":0.95999999999999996},"min":{"1min":0.436,"5min":0.312,"15min":0.307},"max":{"1min":4.9130000000000003,"5min":6.3319999999999999,"15min":16.577999999999999},"last":1.3029999999999999},{"interface":"front","average":{"1min":2.282,"5min":1.5649999999999999,"15min":1.2030000000000001},"min":{"1min":0.33900000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":19.359999999999999,"5min":19.359999999999999,"15min":19.359999999999999},"last":1.968}]},{"osd":5,"last update":"Mon Feb 5 06:43:46 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":1.49,"15min":1.163},"min":{"1min":0.52200000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":2.2559999999999998,"5min":5.8970000000000002,"15min":13.672000000000001},"last":1.3460000000000001},{"interface":"front","average":{"1min":0.78300000000000003,"5min":1.27,"15min":0.96299999999999997},"min":{"1min":0.38700000000000001,"5min":0.32800000000000001,"15min":0.28399999999999997},"max":{"1min":2.2040000000000002,"5min":7.7679999999999998,"15min":15.792999999999999},"last":1.391}]},{"osd":6,"last update":"Mon Feb 5 06:44:18 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.407,"15min":1.141},"min":{"1min":0.39500000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":4.5890000000000004,"5min":6.7809999999999997,"15min":8.3610000000000007},"last":0.996},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.3799999999999999,"15min":1.1479999999999999},"min":{"1min":0.48899999999999999,"5min":0.318,"15min":0.28599999999999998},"max":{"1min":3.5569999999999999,"5min":5.8799999999999999,"15min":6.601},"last":1.4370000000000001}]},{"osd":7,"last update":"Mon Feb 5 06:43:42 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":1.2629999999999999,"15min":1.0860000000000001},"min":{"1min":0.45800000000000002,"5min":0.376,"15min":0.33400000000000002},"max":{"1min":1.6890000000000001,"5min":7.7320000000000002,"15min":45.039000000000001},"last":1.615},{"interface":"front","average":{"1min":0.84399999999999997,"5min":1.4890000000000001,"15min":1.256},"min":{"1min":0.58099999999999996,"5min":0.35899999999999999,"15min":0.34399999999999997},"max":{"1min":2.4830000000000001,"5min":5.8159999999999998,"15min":44.036999999999999},"last":1.7509999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986329,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1601300,"kb_used_data":1186956,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92143852,"statfs":{"total":95995035648,"available":94355304448,"internally_reserved":0,"allocated":1215442944,"data_stored":1214795852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:44:23 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":1.401,"15min":1.1080000000000001},"min":{"1min":0.26300000000000001,"5min":0.20799999999999999,"15min":0.20200000000000001},"max":{"1min":1.2250000000000001,"5min":33.372,"15min":33.372},"last":3.9249999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":1.2949999999999999,"15min":1.175},"min":{"1min":0.251,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.1870000000000001,"5min":33.323999999999998,"15min":33.323999999999998},"last":3.891}]},{"osd":1,"last update":"Mon Feb 5 06:44:23 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":1.4870000000000001,"15min":0.94399999999999995},"min":{"1min":0.27300000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":5.0599999999999996,"5min":33.747999999999998,"15min":33.747999999999998},"last":4.0490000000000004},{"interface":"front","average":{"1min":0.98399999999999999,"5min":1.2849999999999999,"15min":0.94699999999999995},"min":{"1min":0.373,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.5840000000000001,"5min":33.691000000000003,"15min":33.691000000000003},"last":6.8860000000000001}]},{"osd":2,"last update":"Mon Feb 5 06:44:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.93300000000000005,"15min":0.76400000000000001},"min":{"1min":0.39800000000000002,"5min":0.28199999999999997,"15min":0.245},"max":{"1min":1.2549999999999999,"5min":17.379999999999999,"15min":17.379999999999999},"last":6.6040000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":1.1739999999999999,"15min":0.89900000000000002},"min":{"1min":0.315,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.3129999999999999,"5min":17.001999999999999,"15min":17.001999999999999},"last":3.8439999999999999}]},{"osd":3,"last update":"Mon Feb 5 06:44:23 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":2.2000000000000002,"15min":1.992},"min":{"1min":0.38100000000000001,"5min":0.252,"15min":0.252},"max":{"1min":3.496,"5min":44.601999999999997,"15min":99.546999999999997},"last":4.085},{"interface":"front","average":{"1min":1.2250000000000001,"5min":2.2040000000000002,"15min":2.0070000000000001},"min":{"1min":0.40000000000000002,"5min":0.309,"15min":0.29099999999999998},"max":{"1min":10.73,"5min":44.511000000000003,"15min":99.245000000000005},"last":1.7430000000000001}]},{"osd":5,"last update":"Mon Feb 5 06:43:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.59499999999999997,"15min":0.54100000000000004},"min":{"1min":0.29599999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.147,"5min":4.6289999999999996,"15min":11.098000000000001},"last":4.181},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.35299999999999998,"5min":0.25700000000000001,"15min":0.252},"max":{"1min":1.5980000000000001,"5min":3.5880000000000001,"15min":10.965999999999999},"last":4.1210000000000004}]},{"osd":6,"last update":"Mon Feb 5 06:44:20 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.67100000000000004,"15min":0.61499999999999999},"min":{"1min":0.35399999999999998,"5min":0.246,"15min":0.246},"max":{"1min":1.1519999999999999,"5min":4.375,"15min":11.868},"last":4.2999999999999998},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.65500000000000003,"15min":0.61299999999999999},"min":{"1min":0.32700000000000001,"5min":0.29899999999999999,"15min":0.27600000000000002},"max":{"1min":0.73799999999999999,"5min":3.5670000000000002,"15min":11.787000000000001},"last":1.9610000000000001}]},{"osd":7,"last update":"Mon Feb 5 06:44:35 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.71099999999999997,"15min":0.58599999999999997},"min":{"1min":0.375,"5min":0.27800000000000002,"15min":0.26000000000000001},"max":{"1min":1.986,"5min":5.1500000000000004,"15min":5.1500000000000004},"last":1.788},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.72899999999999998,"15min":0.61299999999999999},"min":{"1min":0.378,"5min":0.28899999999999998,"15min":0.28499999999999998},"max":{"1min":1.2769999999999999,"5min":5.5890000000000004,"15min":5.5890000000000004},"last":4.335}]}]},{"osd":5,"up_from":40,"seq":171798691988,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3934856,"kb_used_data":3524668,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":89810296,"statfs":{"total":95995035648,"available":91965743104,"internally_reserved":0,"allocated":3609260032,"data_stored":3608630077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":420018630},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":1.9299999999999999,"15min":1.3169999999999999},"min":{"1min":0.316,"5min":0.223,"15min":0.19700000000000001},"max":{"1min":2.8279999999999998,"5min":55.704999999999998,"15min":55.704999999999998},"last":1.1180000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":2.069,"15min":1.516},"min":{"1min":0.33600000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.242,"5min":56.789000000000001,"15min":56.789000000000001},"last":0.56599999999999995}]},{"osd":1,"last update":"Mon Feb 5 06:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":1.859,"15min":1.212},"min":{"1min":0.31,"5min":0.251,"15min":0.248},"max":{"1min":2.2730000000000001,"5min":41.191000000000003,"15min":41.191000000000003},"last":0.624},{"interface":"front","average":{"1min":0.70699999999999996,"5min":1.8620000000000001,"15min":1.0609999999999999},"min":{"1min":0.28100000000000003,"5min":0.24399999999999999,"15min":0.23499999999999999},"max":{"1min":3.226,"5min":41.173000000000002,"15min":41.173000000000002},"last":1.0569999999999999}]},{"osd":2,"last update":"Mon Feb 5 06:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":1.379,"15min":1.155},"min":{"1min":0.28999999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.536,"5min":30.052,"15min":36.901000000000003},"last":0.438},{"interface":"front","average":{"1min":0.68600000000000005,"5min":1.3620000000000001,"15min":1.071},"min":{"1min":0.39000000000000001,"5min":0.29499999999999998,"15min":0.28899999999999998},"max":{"1min":2.2970000000000002,"5min":30.472999999999999,"15min":37.984000000000002},"last":0.65000000000000002}]},{"osd":3,"last update":"Mon Feb 5 06:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":1.9970000000000001,"15min":1.2130000000000001},"min":{"1min":0.38,"5min":0.33300000000000002,"15min":0.31900000000000001},"max":{"1min":4.8860000000000001,"5min":55.417000000000002,"15min":55.417000000000002},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.66500000000000004,"5min":2.1680000000000001,"15min":1.3049999999999999},"min":{"1min":0.42599999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.351,"5min":55.079000000000001,"15min":55.079000000000001},"last":0.59199999999999997}]},{"osd":4,"last update":"Mon Feb 5 06:43:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.58699999999999997,"15min":0.46999999999999997},"min":{"1min":0.26900000000000002,"5min":0.25800000000000001,"15min":0.23200000000000001},"max":{"1min":2.613,"5min":5.4969999999999999,"15min":5.4969999999999999},"last":1.091},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.57199999999999995,"15min":0.497},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.98099999999999998,"5min":3.0150000000000001,"15min":3.0150000000000001},"last":0.46600000000000003}]},{"osd":6,"last update":"Mon Feb 5 06:44:14 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.70399999999999996,"15min":0.55200000000000005},"min":{"1min":0.28100000000000003,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.393,"5min":3.585,"15min":3.585},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.78000000000000003,"15min":0.59999999999999998},"min":{"1min":0.35299999999999998,"5min":0.31900000000000001,"15min":0.27600000000000002},"max":{"1min":1.5700000000000001,"5min":6.5179999999999998,"15min":6.5179999999999998},"last":1.145}]},{"osd":7,"last update":"Mon Feb 5 06:44:39 2024","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":0.73199999999999998,"15min":0.57099999999999995},"min":{"1min":0.36799999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":3.952,"5min":3.952,"15min":3.952},"last":0.48899999999999999},{"interface":"front","average":{"1min":1.105,"5min":0.80900000000000005,"15min":0.61799999999999999},"min":{"1min":0.35599999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":4.0869999999999997,"5min":4.0869999999999997,"15min":4.0869999999999997},"last":1.819}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1628962816,"data_stored":1628962816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3665690624,"data_stored":3665690624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1216675840,"data_stored":1216675840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3253272576,"data_stored":3253272576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1217986560,"data_stored":1217986560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3640066048,"data_stored":3640066048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1217200128,"data_stored":1217200128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3624927232,"data_stored":3624927232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T06:44:46.611 INFO:tasks.thrashosds.thrasher:Setting 6.12 to [3, 6, 2, 1, 4, 0] 2024-02-05T06:44:46.611 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '6.12', '3', '6', '2', '1', '4', '0'] 2024-02-05T06:44:46.612 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd pg-upmap-items 6.12 3 6 2 1 4 0 2024-02-05T06:44:46.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:46.613 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:46.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:46.715 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:46.748 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:46.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:46.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:46.850 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:46.882 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:46 smithi167 ceph-mon[32774]: 4.6 scrub starts 2024-02-05T06:44:46.882 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:46 smithi167 ceph-mon[32774]: 4.6 scrub ok 2024-02-05T06:44:46.882 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:46 smithi167 ceph-mon[32774]: 4.a scrub starts 2024-02-05T06:44:46.882 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:46 smithi167 ceph-mon[32774]: 4.a scrub ok 2024-02-05T06:44:46.882 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:46 smithi167 ceph-mon[32774]: 3.13 scrub starts 2024-02-05T06:44:46.882 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:46 smithi167 ceph-mon[32774]: 3.13 scrub ok 2024-02-05T06:44:46.882 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:46 smithi167 ceph-mon[32774]: from='client.15477 v1:172.21.15.28:0/1839603254' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:44:46.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:46.892 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:46.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:46.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:46 smithi028 ceph-mon[29442]: 4.6 scrub starts 2024-02-05T06:44:46.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:46 smithi028 ceph-mon[29442]: 4.6 scrub ok 2024-02-05T06:44:46.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:46 smithi028 ceph-mon[29442]: 4.a scrub starts 2024-02-05T06:44:46.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:46 smithi028 ceph-mon[29442]: 4.a scrub ok 2024-02-05T06:44:46.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:46 smithi028 ceph-mon[29442]: 3.13 scrub starts 2024-02-05T06:44:46.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:46 smithi028 ceph-mon[29442]: 3.13 scrub ok 2024-02-05T06:44:46.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:46 smithi028 ceph-mon[29442]: from='client.15477 v1:172.21.15.28:0/1839603254' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:44:46.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:46 smithi028 ceph-mon[37984]: 4.6 scrub starts 2024-02-05T06:44:46.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:46 smithi028 ceph-mon[37984]: 4.6 scrub ok 2024-02-05T06:44:46.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:46 smithi028 ceph-mon[37984]: 4.a scrub starts 2024-02-05T06:44:46.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:46 smithi028 ceph-mon[37984]: 4.a scrub ok 2024-02-05T06:44:46.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:46 smithi028 ceph-mon[37984]: 3.13 scrub starts 2024-02-05T06:44:46.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:46 smithi028 ceph-mon[37984]: 3.13 scrub ok 2024-02-05T06:44:46.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:46 smithi028 ceph-mon[37984]: from='client.15477 v1:172.21.15.28:0/1839603254' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:44:46.993 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:47.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:47.048 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:47.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:47.097 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:44:47.133 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 67 16 105211 105195 98.1066 72.125 0.0134757 0.0101875 2024-02-05T06:44:47.149 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:47.176 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:47.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:47.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:47.216 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:44:47.278 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:47.316 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:47.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:47.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:47.418 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:47.454 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:47.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:47.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:47.556 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:47.596 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:47.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:47.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:47.698 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:47.733 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:47.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:47 smithi167 ceph-mon[32774]: pgmap v551: 158 pgs: 158 active+clean; 6.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 886 op/s 2024-02-05T06:44:47.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:47 smithi167 ceph-mon[32774]: 3.11 scrub starts 2024-02-05T06:44:47.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:47 smithi167 ceph-mon[32774]: 3.11 scrub ok 2024-02-05T06:44:47.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:47 smithi167 ceph-mon[32774]: 3.16 scrub starts 2024-02-05T06:44:47.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:47 smithi167 ceph-mon[32774]: 3.16 scrub ok 2024-02-05T06:44:47.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:47 smithi167 ceph-mon[32774]: 6.1 scrub starts 2024-02-05T06:44:47.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:47 smithi167 ceph-mon[32774]: 6.1 scrub ok 2024-02-05T06:44:47.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:47.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:47.780 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:47 smithi028 ceph-mon[37984]: pgmap v551: 158 pgs: 158 active+clean; 6.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 886 op/s 2024-02-05T06:44:47.781 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:47 smithi028 ceph-mon[37984]: 3.11 scrub starts 2024-02-05T06:44:47.781 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:47 smithi028 ceph-mon[37984]: 3.11 scrub ok 2024-02-05T06:44:47.781 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:47 smithi028 ceph-mon[37984]: 3.16 scrub starts 2024-02-05T06:44:47.781 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:47 smithi028 ceph-mon[37984]: 3.16 scrub ok 2024-02-05T06:44:47.781 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:47 smithi028 ceph-mon[37984]: 6.1 scrub starts 2024-02-05T06:44:47.781 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:47 smithi028 ceph-mon[37984]: 6.1 scrub ok 2024-02-05T06:44:47.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:47 smithi028 ceph-mon[29442]: pgmap v551: 158 pgs: 158 active+clean; 6.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 886 op/s 2024-02-05T06:44:47.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:47 smithi028 ceph-mon[29442]: 3.11 scrub starts 2024-02-05T06:44:47.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:47 smithi028 ceph-mon[29442]: 3.11 scrub ok 2024-02-05T06:44:47.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:47 smithi028 ceph-mon[29442]: 3.16 scrub starts 2024-02-05T06:44:47.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:47 smithi028 ceph-mon[29442]: 3.16 scrub ok 2024-02-05T06:44:47.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:47 smithi028 ceph-mon[29442]: 6.1 scrub starts 2024-02-05T06:44:47.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:47 smithi028 ceph-mon[29442]: 6.1 scrub ok 2024-02-05T06:44:47.836 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:47.873 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:47.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:47.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:47.975 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:48.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:48.004 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:48.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:48.105 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:48.133 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 68 16 106387 106371 97.7447 73.5 0.0103929 0.0102261 2024-02-05T06:44:48.146 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:48.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:48.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:48.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:48.287 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:48.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:48.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:48.389 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:48.421 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:48.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:48.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:48.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:48.568 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:48.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:48.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:48.670 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:48.706 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:48.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:48.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:48.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:48 smithi028 ceph-mon[29442]: 4.d scrub starts 2024-02-05T06:44:48.757 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:48 smithi028 ceph-mon[29442]: 4.d scrub ok 2024-02-05T06:44:48.758 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:48 smithi028 ceph-mon[29442]: 1.2 scrub starts 2024-02-05T06:44:48.758 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:48 smithi028 ceph-mon[29442]: 1.2 scrub ok 2024-02-05T06:44:48.758 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:48 smithi028 ceph-mon[29442]: 5.2 scrub starts 2024-02-05T06:44:48.758 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:48 smithi028 ceph-mon[29442]: 5.2 scrub ok 2024-02-05T06:44:48.758 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:48 smithi028 ceph-mon[37984]: 4.d scrub starts 2024-02-05T06:44:48.758 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:48 smithi028 ceph-mon[37984]: 4.d scrub ok 2024-02-05T06:44:48.758 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:48 smithi028 ceph-mon[37984]: 1.2 scrub starts 2024-02-05T06:44:48.758 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:48 smithi028 ceph-mon[37984]: 1.2 scrub ok 2024-02-05T06:44:48.758 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:48 smithi028 ceph-mon[37984]: 5.2 scrub starts 2024-02-05T06:44:48.758 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:48 smithi028 ceph-mon[37984]: 5.2 scrub ok 2024-02-05T06:44:48.809 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:48.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:48.854 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:48.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:48.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:48.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:48 smithi167 ceph-mon[32774]: 4.d scrub starts 2024-02-05T06:44:48.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:48 smithi167 ceph-mon[32774]: 4.d scrub ok 2024-02-05T06:44:48.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:48 smithi167 ceph-mon[32774]: 1.2 scrub starts 2024-02-05T06:44:48.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:48 smithi167 ceph-mon[32774]: 1.2 scrub ok 2024-02-05T06:44:48.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:48 smithi167 ceph-mon[32774]: 5.2 scrub starts 2024-02-05T06:44:48.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:48 smithi167 ceph-mon[32774]: 5.2 scrub ok 2024-02-05T06:44:48.993 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:48.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:48.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:49.095 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:49.133 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:49.133 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 69 16 107606 107590 97.4323 76.1875 0.0079316 0.0102585 2024-02-05T06:44:49.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:49.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:49.235 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:49.285 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:49.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:49.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:49.387 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:49.423 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:49.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:49.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:49.525 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:49.560 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-02-05T06:44:49.563 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:49.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:49.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:49.661 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:49 smithi167 ceph-mon[32774]: pgmap v552: 158 pgs: 158 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-02-05T06:44:49.661 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:49 smithi167 ceph-mon[32774]: 3.c scrub starts 2024-02-05T06:44:49.661 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:49 smithi167 ceph-mon[32774]: 3.c scrub ok 2024-02-05T06:44:49.661 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:49 smithi167 ceph-mon[32774]: 4.7 scrub starts 2024-02-05T06:44:49.661 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:49 smithi167 ceph-mon[32774]: 4.7 scrub ok 2024-02-05T06:44:49.661 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:49 smithi167 ceph-mon[32774]: 5.14 scrub starts 2024-02-05T06:44:49.661 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:49 smithi167 ceph-mon[32774]: 5.14 scrub ok 2024-02-05T06:44:49.662 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:49 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2815561372' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:44:49.665 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:49.704 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:49.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:49.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:49.806 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:49.845 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:49.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:49.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:49.848 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:49 smithi028 ceph-mon[29442]: pgmap v552: 158 pgs: 158 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-02-05T06:44:49.849 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:49 smithi028 ceph-mon[29442]: 3.c scrub starts 2024-02-05T06:44:49.849 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:49 smithi028 ceph-mon[29442]: 3.c scrub ok 2024-02-05T06:44:49.849 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:49 smithi028 ceph-mon[29442]: 4.7 scrub starts 2024-02-05T06:44:49.849 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:49 smithi028 ceph-mon[29442]: 4.7 scrub ok 2024-02-05T06:44:49.849 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:49 smithi028 ceph-mon[29442]: 5.14 scrub starts 2024-02-05T06:44:49.849 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:49 smithi028 ceph-mon[29442]: 5.14 scrub ok 2024-02-05T06:44:49.849 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:49 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2815561372' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:44:49.849 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:49 smithi028 ceph-mon[37984]: pgmap v552: 158 pgs: 158 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-02-05T06:44:49.849 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:49 smithi028 ceph-mon[37984]: 3.c scrub starts 2024-02-05T06:44:49.849 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:49 smithi028 ceph-mon[37984]: 3.c scrub ok 2024-02-05T06:44:49.849 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:49 smithi028 ceph-mon[37984]: 4.7 scrub starts 2024-02-05T06:44:49.849 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:49 smithi028 ceph-mon[37984]: 4.7 scrub ok 2024-02-05T06:44:49.849 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:49 smithi028 ceph-mon[37984]: 5.14 scrub starts 2024-02-05T06:44:49.850 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:49 smithi028 ceph-mon[37984]: 5.14 scrub ok 2024-02-05T06:44:49.850 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:49 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2815561372' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:44:49.873 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:44:49.873 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:44:49.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:50.006 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:50.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:50.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:50.108 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:50.133 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 70 16 108854 108838 97.1546 78 0.0179302 0.0102881 2024-02-05T06:44:50.152 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:50.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:50.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:50.253 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:50.285 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:50.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:50.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:50.387 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:50.422 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:50.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:50.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:50.523 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:50.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:50.564 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:50.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:50.664 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:50.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:50 smithi028 ceph-mon[37984]: 1.5 scrub starts 2024-02-05T06:44:50.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:50 smithi028 ceph-mon[37984]: 1.5 scrub ok 2024-02-05T06:44:50.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:50 smithi028 ceph-mon[37984]: 6.8 scrub starts 2024-02-05T06:44:50.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:50 smithi028 ceph-mon[37984]: 6.8 scrub ok 2024-02-05T06:44:50.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:50 smithi028 ceph-mon[37984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:44:50.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:50 smithi028 ceph-mon[37984]: Cluster is now healthy 2024-02-05T06:44:50.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:50 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2815561372' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:44:50.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:50 smithi028 ceph-mon[37984]: osdmap e313: 8 total, 8 up, 8 in 2024-02-05T06:44:50.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:50 smithi028 ceph-mon[29442]: 1.5 scrub starts 2024-02-05T06:44:50.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:50 smithi028 ceph-mon[29442]: 1.5 scrub ok 2024-02-05T06:44:50.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:50 smithi028 ceph-mon[29442]: 6.8 scrub starts 2024-02-05T06:44:50.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:50 smithi028 ceph-mon[29442]: 6.8 scrub ok 2024-02-05T06:44:50.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:50 smithi028 ceph-mon[29442]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:44:50.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:50 smithi028 ceph-mon[29442]: Cluster is now healthy 2024-02-05T06:44:50.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:50 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2815561372' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:44:50.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:50 smithi028 ceph-mon[29442]: osdmap e313: 8 total, 8 up, 8 in 2024-02-05T06:44:50.710 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:50.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:50.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:50.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:50 smithi167 ceph-mon[32774]: 1.5 scrub starts 2024-02-05T06:44:50.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:50 smithi167 ceph-mon[32774]: 1.5 scrub ok 2024-02-05T06:44:50.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:50 smithi167 ceph-mon[32774]: 6.8 scrub starts 2024-02-05T06:44:50.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:50 smithi167 ceph-mon[32774]: 6.8 scrub ok 2024-02-05T06:44:50.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:50 smithi167 ceph-mon[32774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:44:50.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:50 smithi167 ceph-mon[32774]: Cluster is now healthy 2024-02-05T06:44:50.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:50 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2815561372' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:44:50.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:50 smithi167 ceph-mon[32774]: osdmap e313: 8 total, 8 up, 8 in 2024-02-05T06:44:50.812 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:50.856 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:50.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:50.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:50.874 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-05T06:44:50.958 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:50.998 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:50.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:50.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:51.100 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:51.133 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 71 16 110129 110113 96.9085 79.6875 0.00502442 0.0103139 2024-02-05T06:44:51.140 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:51.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:51.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:51.242 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:51.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:51.285 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:51.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:51.319 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:44:51.385 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:51.447 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:51.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:51.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:51.549 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:51.585 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:51.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:51.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:51.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:51 smithi028 ceph-mon[29442]: 4.f scrub starts 2024-02-05T06:44:51.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:51 smithi028 ceph-mon[29442]: 4.f scrub ok 2024-02-05T06:44:51.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:51 smithi028 ceph-mon[29442]: 4.c scrub starts 2024-02-05T06:44:51.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:51 smithi028 ceph-mon[29442]: 4.c scrub ok 2024-02-05T06:44:51.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:51 smithi028 ceph-mon[29442]: pgmap v554: 158 pgs: 158 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-02-05T06:44:51.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:51 smithi028 ceph-mon[29442]: 1.0 scrub starts 2024-02-05T06:44:51.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:51 smithi028 ceph-mon[29442]: 1.0 scrub ok 2024-02-05T06:44:51.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:51 smithi028 ceph-mon[29442]: 6.14 scrub starts 2024-02-05T06:44:51.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:51 smithi028 ceph-mon[29442]: 6.14 scrub ok 2024-02-05T06:44:51.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:51 smithi028 ceph-mon[29442]: 6.a scrub starts 2024-02-05T06:44:51.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:51 smithi028 ceph-mon[29442]: 6.a scrub ok 2024-02-05T06:44:51.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:51 smithi028 ceph-mon[37984]: 4.f scrub starts 2024-02-05T06:44:51.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:51 smithi028 ceph-mon[37984]: 4.f scrub ok 2024-02-05T06:44:51.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:51 smithi028 ceph-mon[37984]: 4.c scrub starts 2024-02-05T06:44:51.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:51 smithi028 ceph-mon[37984]: 4.c scrub ok 2024-02-05T06:44:51.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:51 smithi028 ceph-mon[37984]: pgmap v554: 158 pgs: 158 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-02-05T06:44:51.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:51 smithi028 ceph-mon[37984]: 1.0 scrub starts 2024-02-05T06:44:51.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:51 smithi028 ceph-mon[37984]: 1.0 scrub ok 2024-02-05T06:44:51.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:51 smithi028 ceph-mon[37984]: 6.14 scrub starts 2024-02-05T06:44:51.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:51 smithi028 ceph-mon[37984]: 6.14 scrub ok 2024-02-05T06:44:51.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:51 smithi028 ceph-mon[37984]: 6.a scrub starts 2024-02-05T06:44:51.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:51 smithi028 ceph-mon[37984]: 6.a scrub ok 2024-02-05T06:44:51.687 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:51.724 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:51.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:51.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:51.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:51 smithi167 ceph-mon[32774]: 4.f scrub starts 2024-02-05T06:44:51.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:51 smithi167 ceph-mon[32774]: 4.f scrub ok 2024-02-05T06:44:51.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:51 smithi167 ceph-mon[32774]: 4.c scrub starts 2024-02-05T06:44:51.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:51 smithi167 ceph-mon[32774]: 4.c scrub ok 2024-02-05T06:44:51.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:51 smithi167 ceph-mon[32774]: pgmap v554: 158 pgs: 158 active+clean; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-02-05T06:44:51.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:51 smithi167 ceph-mon[32774]: 1.0 scrub starts 2024-02-05T06:44:51.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:51 smithi167 ceph-mon[32774]: 1.0 scrub ok 2024-02-05T06:44:51.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:51 smithi167 ceph-mon[32774]: 6.14 scrub starts 2024-02-05T06:44:51.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:51 smithi167 ceph-mon[32774]: 6.14 scrub ok 2024-02-05T06:44:51.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:51 smithi167 ceph-mon[32774]: 6.a scrub starts 2024-02-05T06:44:51.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:51 smithi167 ceph-mon[32774]: 6.a scrub ok 2024-02-05T06:44:51.826 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:51.857 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:51.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:51.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:51.959 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:51.993 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:51.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:51.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:52.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:52.133 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 72 16 111182 111166 96.4766 65.8125 0.00809285 0.0103603 2024-02-05T06:44:52.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:52.144 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:52.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:52.176 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:44:52.176 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:44:52.176 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:44:52.176 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:44:52.176 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:44:52.207 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:44:52.245 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:52.252 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:44:52.292 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:52.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:52.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:52.293 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:44:52.302 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:44:52.302 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:44:52.302 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:44:52.302 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:44:52.302 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:44:52.302 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:44:52.302 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:44:52.302 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:44:52.394 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:52.434 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:52.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:52.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:52.536 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:52.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:52.572 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:52.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:52.587 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:44:52 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:44:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:44:52.673 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:52.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:52.723 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:52.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:52.821 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:52 smithi167 ceph-mon[32774]: 5.18 deep-scrub starts 2024-02-05T06:44:52.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:52 smithi167 ceph-mon[32774]: 5.18 deep-scrub ok 2024-02-05T06:44:52.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:52 smithi167 ceph-mon[32774]: 2.1 scrub starts 2024-02-05T06:44:52.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:52 smithi167 ceph-mon[32774]: 2.1 scrub ok 2024-02-05T06:44:52.822 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:52 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3509072850' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:44:52.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:52.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:52 smithi028 ceph-mon[29442]: 5.18 deep-scrub starts 2024-02-05T06:44:52.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:52 smithi028 ceph-mon[29442]: 5.18 deep-scrub ok 2024-02-05T06:44:52.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:52 smithi028 ceph-mon[29442]: 2.1 scrub starts 2024-02-05T06:44:52.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:52 smithi028 ceph-mon[29442]: 2.1 scrub ok 2024-02-05T06:44:52.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:52 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3509072850' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:44:52.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:52 smithi028 ceph-mon[37984]: 5.18 deep-scrub starts 2024-02-05T06:44:52.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:52 smithi028 ceph-mon[37984]: 5.18 deep-scrub ok 2024-02-05T06:44:52.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:52 smithi028 ceph-mon[37984]: 2.1 scrub starts 2024-02-05T06:44:52.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:52 smithi028 ceph-mon[37984]: 2.1 scrub ok 2024-02-05T06:44:52.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:52 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3509072850' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:44:53.050 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:53.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:53.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:53.102 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:44:52 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:44:53.124 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T06:44:53.133 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 73 16 112381 112365 96.1815 74.9375 0.0156823 0.0103922 2024-02-05T06:44:53.152 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:53.190 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:53.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:53.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:53.292 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:53.326 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:44:53.326 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-02-05T06:44:53.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-02-05T06:44:53.326 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-02-05T06:44:53.326 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.326 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.44375.0:1112 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94598155034496 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-02-05T06:44:53.326 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:35:23.909030+0000", 2024-02-05T06:44:53.326 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 569.406208107, 2024-02-05T06:44:53.326 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0337807969999999, 2024-02-05T06:44:53.326 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:53.327 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-02-05T06:44:53.327 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:53.327 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.44375", 2024-02-05T06:44:53.327 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/1937500867", 2024-02-05T06:44:53.327 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1112 2024-02-05T06:44:53.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.327 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:53.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.327 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:53.327 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909030+0000", 2024-02-05T06:44:53.327 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.327 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.327 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:53.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909030+0000", 2024-02-05T06:44:53.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:53.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909033+0000", 2024-02-05T06:44:53.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.458e-06 2024-02-05T06:44:53.328 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.328 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.328 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:53.328 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909039+0000", 2024-02-05T06:44:53.328 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1709999999999999e-06 2024-02-05T06:44:53.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:53.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909055+0000", 2024-02-05T06:44:53.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6042e-05 2024-02-05T06:44:53.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:53.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909067+0000", 2024-02-05T06:44:53.329 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.207e-05 2024-02-05T06:44:53.329 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.329 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.329 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.329 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909147+0000", 2024-02-05T06:44:53.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9798999999999999e-05 2024-02-05T06:44:53.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-02-05T06:44:53.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909154+0000", 2024-02-05T06:44:53.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7329999999999997e-06 2024-02-05T06:44:53.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.330 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.330 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.913608+0000", 2024-02-05T06:44:53.330 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0044539660000000002 2024-02-05T06:44:53.330 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.330 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:53.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.913618+0000", 2024-02-05T06:44:53.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0693000000000001e-05 2024-02-05T06:44:53.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.925995+0000", 2024-02-05T06:44:53.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.012377216 2024-02-05T06:44:53.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.331 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.331 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for degraded object", 2024-02-05T06:44:53.331 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.926372+0000", 2024-02-05T06:44:53.331 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00037610299999999997 2024-02-05T06:44:53.331 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.930243+0000", 2024-02-05T06:44:53.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038717460000000001 2024-02-05T06:44:53.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:53.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.930338+0000", 2024-02-05T06:44:53.332 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4784000000000006e-05 2024-02-05T06:44:53.332 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.332 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.332 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:53.332 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.942811+0000", 2024-02-05T06:44:53.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.012473016 2024-02-05T06:44:53.333 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.333 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:53.333 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.333 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.333 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.333 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.44375.0:1113 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94598159460608 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-02-05T06:44:53.333 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:35:23.909093+0000", 2024-02-05T06:44:53.333 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 569.40614506999998, 2024-02-05T06:44:53.333 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.028018404, 2024-02-05T06:44:53.333 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:53.333 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-02-05T06:44:53.333 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:53.333 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.44375", 2024-02-05T06:44:53.334 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/1937500867", 2024-02-05T06:44:53.334 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1113 2024-02-05T06:44:53.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.334 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:53.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:53.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909093+0000", 2024-02-05T06:44:53.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.334 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.334 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.334 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:53.334 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909093+0000", 2024-02-05T06:44:53.334 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:53.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909094+0000", 2024-02-05T06:44:53.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7499999999999999e-07 2024-02-05T06:44:53.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:53.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909097+0000", 2024-02-05T06:44:53.335 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8779999999999998e-06 2024-02-05T06:44:53.335 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.335 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.335 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:53.335 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909107+0000", 2024-02-05T06:44:53.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8169999999999994e-06 2024-02-05T06:44:53.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:53.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909113+0000", 2024-02-05T06:44:53.336 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5809999999999996e-06 2024-02-05T06:44:53.336 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.336 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.336 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.336 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909185+0000", 2024-02-05T06:44:53.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2577000000000007e-05 2024-02-05T06:44:53.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.337 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.337 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-02-05T06:44:53.337 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909191+0000", 2024-02-05T06:44:53.337 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6640000000000003e-06 2024-02-05T06:44:53.337 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.338 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.915009+0000", 2024-02-05T06:44:53.338 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0058183760000000001 2024-02-05T06:44:53.338 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.338 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.338 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:53.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.915019+0000", 2024-02-05T06:44:53.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7259999999999997e-06 2024-02-05T06:44:53.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.926331+0000", 2024-02-05T06:44:53.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.011312073 2024-02-05T06:44:53.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.339 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.339 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for degraded object", 2024-02-05T06:44:53.339 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.926651+0000", 2024-02-05T06:44:53.339 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00032002799999999998 2024-02-05T06:44:53.339 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.930756+0000", 2024-02-05T06:44:53.340 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.004104707 2024-02-05T06:44:53.340 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.340 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.340 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:53.340 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.930833+0000", 2024-02-05T06:44:53.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7008999999999994e-05 2024-02-05T06:44:53.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.341 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:53.341 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.937112+0000", 2024-02-05T06:44:53.341 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0062790930000000003 2024-02-05T06:44:53.341 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.341 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:53.341 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.341 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.341 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.341 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:30003 8.9 8:9a633465:::benchmark_data_smithi028_105698_object30002:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-02-05T06:44:53.342 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:07.543498+0000", 2024-02-05T06:44:53.342 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 285.77174030899999, 2024-02-05T06:44:53.342 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1580426429999999, 2024-02-05T06:44:53.342 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:53.342 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-02-05T06:44:53.342 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:53.342 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:53.342 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:53.342 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30003 2024-02-05T06:44:53.342 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.342 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:53.342 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.342 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:53.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.543498+0000", 2024-02-05T06:44:53.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:53.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.543498+0000", 2024-02-05T06:44:53.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:53.343 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.543499+0000", 2024-02-05T06:44:53.343 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2900000000000002e-07 2024-02-05T06:44:53.343 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.343 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.343 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:53.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.543538+0000", 2024-02-05T06:44:53.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8693999999999999e-05 2024-02-05T06:44:53.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:53.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.543545+0000", 2024-02-05T06:44:53.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7209999999999994e-06 2024-02-05T06:44:53.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.344 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:53.344 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.543552+0000", 2024-02-05T06:44:53.344 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2330000000000003e-06 2024-02-05T06:44:53.344 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.344 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.543613+0000", 2024-02-05T06:44:53.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1876000000000002e-05 2024-02-05T06:44:53.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:53.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.543688+0000", 2024-02-05T06:44:53.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4887000000000003e-05 2024-02-05T06:44:53.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.345 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.345 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:44:53.345 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.543780+0000", 2024-02-05T06:44:53.345 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1218000000000007e-05 2024-02-05T06:44:53.345 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:53.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.545262+0000", 2024-02-05T06:44:53.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001482803 2024-02-05T06:44:53.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.546695+0000", 2024-02-05T06:44:53.346 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001432317 2024-02-05T06:44:53.346 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.346 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.346 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.346 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.565361+0000", 2024-02-05T06:44:53.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018665870000000001 2024-02-05T06:44:53.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:53.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.565374+0000", 2024-02-05T06:44:53.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3634e-05 2024-02-05T06:44:53.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:08.565071+0000", 2024-02-05T06:44:53.347 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99969719400000001 2024-02-05T06:44:53.347 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.347 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.347 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:53.347 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:08.565090+0000", 2024-02-05T06:44:53.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9026e-05 2024-02-05T06:44:53.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.701428+0000", 2024-02-05T06:44:53.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.136337221 2024-02-05T06:44:53.348 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.348 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.348 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:53.348 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.701541+0000", 2024-02-05T06:44:53.348 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011312 2024-02-05T06:44:53.348 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.348 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:53.349 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.349 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.349 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:30020 8.9 8:93a954f7:::benchmark_data_smithi028_105698_object30019:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-02-05T06:44:53.349 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:07.553736+0000", 2024-02-05T06:44:53.349 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 285.76150246100002, 2024-02-05T06:44:53.349 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.198632489, 2024-02-05T06:44:53.349 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:53.349 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-02-05T06:44:53.349 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:53.349 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:53.349 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:53.349 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30020 2024-02-05T06:44:53.349 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.350 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:53.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:53.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.553736+0000", 2024-02-05T06:44:53.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:53.350 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.553736+0000", 2024-02-05T06:44:53.350 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.350 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.350 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.350 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:53.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.553738+0000", 2024-02-05T06:44:53.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.593e-06 2024-02-05T06:44:53.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:53.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.553798+0000", 2024-02-05T06:44:53.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0925999999999999e-05 2024-02-05T06:44:53.351 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.351 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.351 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:53.351 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.553808+0000", 2024-02-05T06:44:53.351 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8470000000000008e-06 2024-02-05T06:44:53.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:53.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.553815+0000", 2024-02-05T06:44:53.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3269999999999997e-06 2024-02-05T06:44:53.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.352 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.553878+0000", 2024-02-05T06:44:53.352 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2957e-05 2024-02-05T06:44:53.352 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.352 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.352 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:53.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.554018+0000", 2024-02-05T06:44:53.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014072299999999999 2024-02-05T06:44:53.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:44:53.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.554171+0000", 2024-02-05T06:44:53.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015279200000000001 2024-02-05T06:44:53.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:53.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.555457+0000", 2024-02-05T06:44:53.353 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001285795 2024-02-05T06:44:53.353 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.353 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.353 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.353 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.556996+0000", 2024-02-05T06:44:53.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015395560000000001 2024-02-05T06:44:53.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.565509+0000", 2024-02-05T06:44:53.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085128840000000001 2024-02-05T06:44:53.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:53.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.565521+0000", 2024-02-05T06:44:53.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1610000000000001e-05 2024-02-05T06:44:53.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.354 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.354 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.354 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:08.565134+0000", 2024-02-05T06:44:53.354 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99961273399999995 2024-02-05T06:44:53.354 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:53.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:08.565145+0000", 2024-02-05T06:44:53.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1769e-05 2024-02-05T06:44:53.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.355 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.355 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.702593+0000", 2024-02-05T06:44:53.355 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.137447632 2024-02-05T06:44:53.355 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.355 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for degraded object", 2024-02-05T06:44:53.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.703277+0000", 2024-02-05T06:44:53.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00068402600000000001 2024-02-05T06:44:53.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.357 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.752300+0000", 2024-02-05T06:44:53.357 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.049022393999999997 2024-02-05T06:44:53.357 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.357 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.357 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:53.358 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.752368+0000", 2024-02-05T06:44:53.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8924000000000006e-05 2024-02-05T06:44:53.358 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.358 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:53.358 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.358 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.358 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.358 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:30026 8.9 8:946f4619:::benchmark_data_smithi028_105698_object30025:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-02-05T06:44:53.358 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:07.570140+0000", 2024-02-05T06:44:53.358 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 285.74509811299998, 2024-02-05T06:44:53.358 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1827590649999999, 2024-02-05T06:44:53.358 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:53.359 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:53.359 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:53.359 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:53.359 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:53.359 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30026 2024-02-05T06:44:53.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.359 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:53.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.359 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:53.359 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.570140+0000", 2024-02-05T06:44:53.359 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.359 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.359 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:53.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.570140+0000", 2024-02-05T06:44:53.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:53.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.570142+0000", 2024-02-05T06:44:53.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3489999999999999e-06 2024-02-05T06:44:53.360 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.360 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.360 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:53.360 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.570170+0000", 2024-02-05T06:44:53.360 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8802999999999998e-05 2024-02-05T06:44:53.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:53.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.570181+0000", 2024-02-05T06:44:53.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0679999999999999e-05 2024-02-05T06:44:53.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:53.361 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.570189+0000", 2024-02-05T06:44:53.361 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3089999999999996e-06 2024-02-05T06:44:53.361 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.361 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.361 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.570289+0000", 2024-02-05T06:44:53.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9489999999999998e-05 2024-02-05T06:44:53.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:53.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.570299+0000", 2024-02-05T06:44:53.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0013000000000001e-05 2024-02-05T06:44:53.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.362 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.362 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:08.565174+0000", 2024-02-05T06:44:53.362 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99487546000000004 2024-02-05T06:44:53.362 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.362 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:53.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:08.565182+0000", 2024-02-05T06:44:53.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0649999999999999e-06 2024-02-05T06:44:53.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.703322+0000", 2024-02-05T06:44:53.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1381396129999999 2024-02-05T06:44:53.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.363 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:53.363 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.703417+0000", 2024-02-05T06:44:53.363 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4482999999999996e-05 2024-02-05T06:44:53.363 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.363 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:44:53.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.703532+0000", 2024-02-05T06:44:53.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115684 2024-02-05T06:44:53.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:53.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.707152+0000", 2024-02-05T06:44:53.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0036196280000000002 2024-02-05T06:44:53.364 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.364 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.364 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.364 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.710593+0000", 2024-02-05T06:44:53.364 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0034414810000000001 2024-02-05T06:44:53.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.752401+0000", 2024-02-05T06:44:53.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.041807785 2024-02-05T06:44:53.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:53.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.752437+0000", 2024-02-05T06:44:53.365 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5531000000000003e-05 2024-02-05T06:44:53.365 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.365 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.365 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:53.365 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.752899+0000", 2024-02-05T06:44:53.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00046269099999999999 2024-02-05T06:44:53.366 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.366 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:53.366 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.366 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.366 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.366 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:30030 8.9 8:99ce2938:::benchmark_data_smithi028_105698_object30029:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-02-05T06:44:53.366 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:07.581581+0000", 2024-02-05T06:44:53.366 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 285.73365749800001, 2024-02-05T06:44:53.366 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1718570939999999, 2024-02-05T06:44:53.366 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:53.366 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:53.366 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:53.366 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:53.366 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:53.367 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 30030 2024-02-05T06:44:53.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.367 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:53.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:53.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.581581+0000", 2024-02-05T06:44:53.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.367 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:53.367 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.581581+0000", 2024-02-05T06:44:53.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.367 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:53.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.581582+0000", 2024-02-05T06:44:53.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2640000000000001e-06 2024-02-05T06:44:53.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:53.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.581636+0000", 2024-02-05T06:44:53.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3489000000000002e-05 2024-02-05T06:44:53.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:53.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.581646+0000", 2024-02-05T06:44:53.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0628999999999999e-05 2024-02-05T06:44:53.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:53.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.581653+0000", 2024-02-05T06:44:53.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2319999999999998e-06 2024-02-05T06:44:53.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.581732+0000", 2024-02-05T06:44:53.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9882999999999994e-05 2024-02-05T06:44:53.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:53.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:07.581744+0000", 2024-02-05T06:44:53.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1389999999999999e-05 2024-02-05T06:44:53.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:08.565208+0000", 2024-02-05T06:44:53.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98346427800000003 2024-02-05T06:44:53.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:53.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:08.565217+0000", 2024-02-05T06:44:53.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4689999999999997e-06 2024-02-05T06:44:53.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.704462+0000", 2024-02-05T06:44:53.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.139245896 2024-02-05T06:44:53.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:53.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.704610+0000", 2024-02-05T06:44:53.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014754699999999999 2024-02-05T06:44:53.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:44:53.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.704795+0000", 2024-02-05T06:44:53.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018536599999999999 2024-02-05T06:44:53.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:53.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.707159+0000", 2024-02-05T06:44:53.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023637380000000002 2024-02-05T06:44:53.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.710653+0000", 2024-02-05T06:44:53.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0034940010000000001 2024-02-05T06:44:53.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.752937+0000", 2024-02-05T06:44:53.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042284005 2024-02-05T06:44:53.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:53.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.752970+0000", 2024-02-05T06:44:53.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2622e-05 2024-02-05T06:44:53.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:53.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:09.753438+0000", 2024-02-05T06:44:53.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00046828499999999998 2024-02-05T06:44:53.373 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.373 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:53.373 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.373 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:44149 8.5 8:a49895a7:::benchmark_data_smithi028_105698_object44148:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-02-05T06:44:53.374 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:26.078642+0000", 2024-02-05T06:44:53.374 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 267.236596491, 2024-02-05T06:44:53.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1022799999999999, 2024-02-05T06:44:53.374 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:53.374 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:53.374 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:53.374 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:53.374 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:53.374 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44149 2024-02-05T06:44:53.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.374 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:53.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:53.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.078642+0000", 2024-02-05T06:44:53.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:53.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.078642+0000", 2024-02-05T06:44:53.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:53.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.078644+0000", 2024-02-05T06:44:53.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.663e-06 2024-02-05T06:44:53.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:53.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.078720+0000", 2024-02-05T06:44:53.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6079000000000001e-05 2024-02-05T06:44:53.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:53.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.078729+0000", 2024-02-05T06:44:53.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5580000000000006e-06 2024-02-05T06:44:53.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:53.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.078737+0000", 2024-02-05T06:44:53.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1170000000000004e-06 2024-02-05T06:44:53.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.078823+0000", 2024-02-05T06:44:53.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5582999999999997e-05 2024-02-05T06:44:53.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:53.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.078834+0000", 2024-02-05T06:44:53.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0767999999999999e-05 2024-02-05T06:44:53.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.054138+0000", 2024-02-05T06:44:53.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97530467700000001 2024-02-05T06:44:53.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:53.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.054152+0000", 2024-02-05T06:44:53.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3662999999999999e-05 2024-02-05T06:44:53.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.169721+0000", 2024-02-05T06:44:53.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.115569418 2024-02-05T06:44:53.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:53.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.169858+0000", 2024-02-05T06:44:53.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013638600000000001 2024-02-05T06:44:53.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4,7", 2024-02-05T06:44:53.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.169974+0000", 2024-02-05T06:44:53.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011602199999999999 2024-02-05T06:44:53.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.174657+0000", 2024-02-05T06:44:53.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0046829480000000001 2024-02-05T06:44:53.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.174712+0000", 2024-02-05T06:44:53.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5294999999999998e-05 2024-02-05T06:44:53.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:53.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.178773+0000", 2024-02-05T06:44:53.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040608409999999999 2024-02-05T06:44:53.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:53.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.178835+0000", 2024-02-05T06:44:53.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2127999999999999e-05 2024-02-05T06:44:53.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:53.381 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.180922+0000", 2024-02-05T06:44:53.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002086854 2024-02-05T06:44:53.381 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.381 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:53.381 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.381 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:44150 8.5 8:a2178526:::benchmark_data_smithi028_105698_object44149:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-02-05T06:44:53.381 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:26.078763+0000", 2024-02-05T06:44:53.381 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 267.23647498700001, 2024-02-05T06:44:53.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1042297720000001, 2024-02-05T06:44:53.381 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:53.382 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:53.382 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:53.382 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:53.382 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:53.382 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44150 2024-02-05T06:44:53.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.382 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:53.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:53.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.078763+0000", 2024-02-05T06:44:53.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:53.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.078763+0000", 2024-02-05T06:44:53.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:53.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.078764+0000", 2024-02-05T06:44:53.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3200000000000005e-07 2024-02-05T06:44:53.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:53.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.078835+0000", 2024-02-05T06:44:53.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1218999999999997e-05 2024-02-05T06:44:53.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:53.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.078850+0000", 2024-02-05T06:44:53.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4694e-05 2024-02-05T06:44:53.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:53.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.078854+0000", 2024-02-05T06:44:53.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.968e-06 2024-02-05T06:44:53.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.078892+0000", 2024-02-05T06:44:53.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7864999999999997e-05 2024-02-05T06:44:53.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:53.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.078898+0000", 2024-02-05T06:44:53.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2269999999999998e-06 2024-02-05T06:44:53.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.054179+0000", 2024-02-05T06:44:53.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97528090099999998 2024-02-05T06:44:53.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:53.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.054188+0000", 2024-02-05T06:44:53.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1300000000000007e-06 2024-02-05T06:44:53.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.172681+0000", 2024-02-05T06:44:53.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1184931039999999 2024-02-05T06:44:53.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:53.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.172851+0000", 2024-02-05T06:44:53.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017005500000000001 2024-02-05T06:44:53.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4,7", 2024-02-05T06:44:53.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.173030+0000", 2024-02-05T06:44:53.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017874699999999999 2024-02-05T06:44:53.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.387 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.387 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.181024+0000", 2024-02-05T06:44:53.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079938119999999994 2024-02-05T06:44:53.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:53.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.181040+0000", 2024-02-05T06:44:53.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6079e-05 2024-02-05T06:44:53.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.181183+0000", 2024-02-05T06:44:53.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014367199999999999 2024-02-05T06:44:53.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:53.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.181222+0000", 2024-02-05T06:44:53.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8661000000000001e-05 2024-02-05T06:44:53.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:53.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.182993+0000", 2024-02-05T06:44:53.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0017711059999999999 2024-02-05T06:44:53.389 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.389 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:53.389 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.389 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:44153 8.5 8:a3c66f9f:::benchmark_data_smithi028_105698_object44152:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-02-05T06:44:53.389 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:26.080977+0000", 2024-02-05T06:44:53.389 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 267.23426159000002, 2024-02-05T06:44:53.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1036353829999999, 2024-02-05T06:44:53.390 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:53.390 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:53.390 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:53.390 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:53.390 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:53.390 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44153 2024-02-05T06:44:53.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.390 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:53.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:53.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.080977+0000", 2024-02-05T06:44:53.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:53.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.080977+0000", 2024-02-05T06:44:53.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:53.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.080978+0000", 2024-02-05T06:44:53.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4449999999999999e-06 2024-02-05T06:44:53.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:53.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.081040+0000", 2024-02-05T06:44:53.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2063000000000005e-05 2024-02-05T06:44:53.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:53.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.081050+0000", 2024-02-05T06:44:53.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8840000000000006e-06 2024-02-05T06:44:53.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:53.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.081057+0000", 2024-02-05T06:44:53.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0079999999999996e-06 2024-02-05T06:44:53.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.081114+0000", 2024-02-05T06:44:53.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7147e-05 2024-02-05T06:44:53.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:53.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.081125+0000", 2024-02-05T06:44:53.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1026000000000001e-05 2024-02-05T06:44:53.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.054212+0000", 2024-02-05T06:44:53.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97308686099999997 2024-02-05T06:44:53.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:53.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.054220+0000", 2024-02-05T06:44:53.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9070000000000007e-06 2024-02-05T06:44:53.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.174831+0000", 2024-02-05T06:44:53.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.120610968 2024-02-05T06:44:53.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:53.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.175010+0000", 2024-02-05T06:44:53.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017902200000000001 2024-02-05T06:44:53.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.394 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4,7", 2024-02-05T06:44:53.394 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.175176+0000", 2024-02-05T06:44:53.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016625900000000001 2024-02-05T06:44:53.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:53.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.181049+0000", 2024-02-05T06:44:53.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0058720819999999998 2024-02-05T06:44:53.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.181107+0000", 2024-02-05T06:44:53.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8214000000000002e-05 2024-02-05T06:44:53.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.183058+0000", 2024-02-05T06:44:53.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00195094 2024-02-05T06:44:53.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:53.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.183126+0000", 2024-02-05T06:44:53.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8649e-05 2024-02-05T06:44:53.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.396 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.396 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:53.396 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.184612+0000", 2024-02-05T06:44:53.396 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001485908 2024-02-05T06:44:53.396 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.396 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:53.396 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.396 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.397 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:44169 8.5 8:a5d9730b:::benchmark_data_smithi028_105698_object44168:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-02-05T06:44:53.397 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:27.219802+0000", 2024-02-05T06:44:53.397 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 266.09543593799998, 2024-02-05T06:44:53.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96735091399999995, 2024-02-05T06:44:53.397 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:53.397 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:53.397 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:53.397 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:53.397 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:53.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:53.398 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:53.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:53.399 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44169 2024-02-05T06:44:53.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.400 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:53.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:53.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.219802+0000", 2024-02-05T06:44:53.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:53.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.219802+0000", 2024-02-05T06:44:53.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:53.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.219804+0000", 2024-02-05T06:44:53.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9149999999999999e-06 2024-02-05T06:44:53.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:53.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.219868+0000", 2024-02-05T06:44:53.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3923000000000004e-05 2024-02-05T06:44:53.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:53.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.219880+0000", 2024-02-05T06:44:53.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1827999999999999e-05 2024-02-05T06:44:53.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:53.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.219888+0000", 2024-02-05T06:44:53.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2130000000000008e-06 2024-02-05T06:44:53.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.219950+0000", 2024-02-05T06:44:53.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1632999999999998e-05 2024-02-05T06:44:53.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:53.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.219964+0000", 2024-02-05T06:44:53.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3932e-05 2024-02-05T06:44:53.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.176701+0000", 2024-02-05T06:44:53.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.95673720299999998 2024-02-05T06:44:53.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:53.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.176941+0000", 2024-02-05T06:44:53.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00023998499999999999 2024-02-05T06:44:53.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4,7", 2024-02-05T06:44:53.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.177126+0000", 2024-02-05T06:44:53.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018441900000000001 2024-02-05T06:44:53.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:53.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.181057+0000", 2024-02-05T06:44:53.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0039319419999999999 2024-02-05T06:44:53.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.181147+0000", 2024-02-05T06:44:53.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9271000000000006e-05 2024-02-05T06:44:53.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.185949+0000", 2024-02-05T06:44:53.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.004802415 2024-02-05T06:44:53.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:53.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.186011+0000", 2024-02-05T06:44:53.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1540999999999999e-05 2024-02-05T06:44:53.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:53.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.187153+0000", 2024-02-05T06:44:53.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0011426940000000001 2024-02-05T06:44:53.405 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.405 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:53.406 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.406 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:44171 8.5 8:a131905b:::benchmark_data_smithi028_105698_object44170:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-02-05T06:44:53.406 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:27.220613+0000", 2024-02-05T06:44:53.406 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 266.09462507299997, 2024-02-05T06:44:53.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98206730200000003, 2024-02-05T06:44:53.406 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:53.406 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:53.406 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:53.406 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:53.406 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:53.406 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44171 2024-02-05T06:44:53.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.406 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:53.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.407 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:53.407 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.220613+0000", 2024-02-05T06:44:53.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.407 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:53.407 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.220613+0000", 2024-02-05T06:44:53.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.407 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:53.407 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.220614+0000", 2024-02-05T06:44:53.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9900000000000009e-07 2024-02-05T06:44:53.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:53.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.220670+0000", 2024-02-05T06:44:53.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5308e-05 2024-02-05T06:44:53.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:53.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.220679+0000", 2024-02-05T06:44:53.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9809999999999992e-06 2024-02-05T06:44:53.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:53.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.220684+0000", 2024-02-05T06:44:53.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3179999999999997e-06 2024-02-05T06:44:53.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.220733+0000", 2024-02-05T06:44:53.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8961000000000001e-05 2024-02-05T06:44:53.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:53.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.220741+0000", 2024-02-05T06:44:53.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7069999999999992e-06 2024-02-05T06:44:53.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.187203+0000", 2024-02-05T06:44:53.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96646263399999999 2024-02-05T06:44:53.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:53.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.187305+0000", 2024-02-05T06:44:53.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010200900000000001 2024-02-05T06:44:53.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4,7", 2024-02-05T06:44:53.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.187425+0000", 2024-02-05T06:44:53.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119858 2024-02-05T06:44:53.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:53.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.194196+0000", 2024-02-05T06:44:53.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0067711619999999998 2024-02-05T06:44:53.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.194241+0000", 2024-02-05T06:44:53.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5096999999999999e-05 2024-02-05T06:44:53.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.200822+0000", 2024-02-05T06:44:53.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0065805359999999997 2024-02-05T06:44:53.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:53.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.200888+0000", 2024-02-05T06:44:53.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6260000000000006e-05 2024-02-05T06:44:53.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:53.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.202681+0000", 2024-02-05T06:44:53.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001792472 2024-02-05T06:44:53.412 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.412 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:53.412 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.412 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:44174 8.5 8:aaae78c8:::benchmark_data_smithi028_105698_object44173:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-02-05T06:44:53.413 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:27.234186+0000", 2024-02-05T06:44:53.413 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 266.08105244900003, 2024-02-05T06:44:53.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96945963599999996, 2024-02-05T06:44:53.413 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:53.413 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:53.413 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:53.413 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:53.413 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:53.413 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 44174 2024-02-05T06:44:53.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.413 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:53.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:53.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.234186+0000", 2024-02-05T06:44:53.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:53.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.234186+0000", 2024-02-05T06:44:53.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:53.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.234190+0000", 2024-02-05T06:44:53.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6440000000000002e-06 2024-02-05T06:44:53.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:53.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.234251+0000", 2024-02-05T06:44:53.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1204999999999994e-05 2024-02-05T06:44:53.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:53.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.234277+0000", 2024-02-05T06:44:53.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6616000000000001e-05 2024-02-05T06:44:53.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:53.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.234293+0000", 2024-02-05T06:44:53.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5849999999999999e-05 2024-02-05T06:44:53.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.234398+0000", 2024-02-05T06:44:53.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104768 2024-02-05T06:44:53.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:53.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.234406+0000", 2024-02-05T06:44:53.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4060000000000005e-06 2024-02-05T06:44:53.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.188940+0000", 2024-02-05T06:44:53.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.95453351399999997 2024-02-05T06:44:53.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:53.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.189121+0000", 2024-02-05T06:44:53.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018096599999999999 2024-02-05T06:44:53.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4,7", 2024-02-05T06:44:53.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.189246+0000", 2024-02-05T06:44:53.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000125284 2024-02-05T06:44:53.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.194287+0000", 2024-02-05T06:44:53.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0050402800000000003 2024-02-05T06:44:53.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:53.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.194304+0000", 2024-02-05T06:44:53.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7549000000000001e-05 2024-02-05T06:44:53.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.202717+0000", 2024-02-05T06:44:53.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0084132670000000003 2024-02-05T06:44:53.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:53.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.202758+0000", 2024-02-05T06:44:53.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0666e-05 2024-02-05T06:44:53.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:53.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:28.203646+0000", 2024-02-05T06:44:53.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000887621 2024-02-05T06:44:53.419 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.419 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:53.419 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.419 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:46485 8.9 8:9d6e8272:::benchmark_data_smithi028_105698_object46484:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-02-05T06:44:53.419 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:31.124977+0000", 2024-02-05T06:44:53.419 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 262.190261362, 2024-02-05T06:44:53.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99050297600000003, 2024-02-05T06:44:53.420 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:53.420 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:53.420 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:53.420 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:53.420 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:53.420 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46485 2024-02-05T06:44:53.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.420 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:53.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:53.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.124977+0000", 2024-02-05T06:44:53.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:53.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.124977+0000", 2024-02-05T06:44:53.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:53.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.124979+0000", 2024-02-05T06:44:53.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8279999999999999e-06 2024-02-05T06:44:53.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:53.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.125026+0000", 2024-02-05T06:44:53.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7151000000000003e-05 2024-02-05T06:44:53.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:53.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.125039+0000", 2024-02-05T06:44:53.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2863e-05 2024-02-05T06:44:53.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:53.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.125048+0000", 2024-02-05T06:44:53.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3549999999999994e-06 2024-02-05T06:44:53.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.125121+0000", 2024-02-05T06:44:53.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2569000000000003e-05 2024-02-05T06:44:53.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:53.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.125133+0000", 2024-02-05T06:44:53.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2232000000000001e-05 2024-02-05T06:44:53.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.110321+0000", 2024-02-05T06:44:53.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98518825899999996 2024-02-05T06:44:53.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:53.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.110476+0000", 2024-02-05T06:44:53.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015426599999999999 2024-02-05T06:44:53.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:44:53.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.110650+0000", 2024-02-05T06:44:53.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017418699999999999 2024-02-05T06:44:53.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:53.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.115190+0000", 2024-02-05T06:44:53.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0045406680000000003 2024-02-05T06:44:53.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.115229+0000", 2024-02-05T06:44:53.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8279999999999999e-05 2024-02-05T06:44:53.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.115383+0000", 2024-02-05T06:44:53.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015467899999999999 2024-02-05T06:44:53.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:53.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.115440+0000", 2024-02-05T06:44:53.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6660000000000003e-05 2024-02-05T06:44:53.426 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:53.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.115480+0000", 2024-02-05T06:44:53.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9978999999999998e-05 2024-02-05T06:44:53.426 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.426 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:53.426 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.426 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.426 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:46506 8.9 8:9c6d95e5:::benchmark_data_smithi028_105698_object46505:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-02-05T06:44:53.426 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:31.148612+0000", 2024-02-05T06:44:53.426 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 262.16662670800002, 2024-02-05T06:44:53.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97041352599999997, 2024-02-05T06:44:53.427 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:53.427 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:53.427 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:53.427 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:53.427 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:53.427 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46506 2024-02-05T06:44:53.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.427 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:53.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:53.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.148612+0000", 2024-02-05T06:44:53.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.428 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:53.428 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.148612+0000", 2024-02-05T06:44:53.428 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.428 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.428 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:53.428 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.148613+0000", 2024-02-05T06:44:53.428 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6300000000000001e-06 2024-02-05T06:44:53.428 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.428 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:53.428 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.148644+0000", 2024-02-05T06:44:53.428 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1103999999999999e-05 2024-02-05T06:44:53.429 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.429 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:53.429 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.148656+0000", 2024-02-05T06:44:53.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1452e-05 2024-02-05T06:44:53.429 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.429 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:53.429 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.148664+0000", 2024-02-05T06:44:53.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7970000000000001e-06 2024-02-05T06:44:53.429 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.429 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.148719+0000", 2024-02-05T06:44:53.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5123000000000001e-05 2024-02-05T06:44:53.430 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:53.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.148727+0000", 2024-02-05T06:44:53.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9300000000000003e-06 2024-02-05T06:44:53.430 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.112375+0000", 2024-02-05T06:44:53.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96364781300000002 2024-02-05T06:44:53.430 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:53.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.112501+0000", 2024-02-05T06:44:53.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012634899999999999 2024-02-05T06:44:53.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:44:53.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.112654+0000", 2024-02-05T06:44:53.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015268399999999999 2024-02-05T06:44:53.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:53.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.115308+0000", 2024-02-05T06:44:53.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0026542509999999998 2024-02-05T06:44:53.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.432 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.432 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.432 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.118798+0000", 2024-02-05T06:44:53.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0034902459999999998 2024-02-05T06:44:53.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.432 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.432 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.432 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.118917+0000", 2024-02-05T06:44:53.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118804 2024-02-05T06:44:53.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.432 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.432 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:53.433 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.118983+0000", 2024-02-05T06:44:53.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5933999999999996e-05 2024-02-05T06:44:53.433 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.433 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.433 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:53.433 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.119025+0000", 2024-02-05T06:44:53.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2409e-05 2024-02-05T06:44:53.433 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.433 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:53.433 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.433 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.433 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.433 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:46508 8.9 8:91e56426:::benchmark_data_smithi028_105698_object46507:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-02-05T06:44:53.433 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:31.150340+0000", 2024-02-05T06:44:53.434 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 262.16489811700001, 2024-02-05T06:44:53.434 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96883166300000001, 2024-02-05T06:44:53.434 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:53.434 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:53.434 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:53.434 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:53.434 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:53.434 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46508 2024-02-05T06:44:53.434 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.434 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:53.434 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.434 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:53.434 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.150340+0000", 2024-02-05T06:44:53.434 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.435 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.435 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:53.435 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.150340+0000", 2024-02-05T06:44:53.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.435 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.435 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:53.435 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.150341+0000", 2024-02-05T06:44:53.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.147e-06 2024-02-05T06:44:53.435 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.435 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:53.435 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.150397+0000", 2024-02-05T06:44:53.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5393999999999999e-05 2024-02-05T06:44:53.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:53.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.150407+0000", 2024-02-05T06:44:53.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0414999999999999e-05 2024-02-05T06:44:53.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:53.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.150413+0000", 2024-02-05T06:44:53.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1129999999999997e-06 2024-02-05T06:44:53.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.150467+0000", 2024-02-05T06:44:53.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3946000000000002e-05 2024-02-05T06:44:53.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:53.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.150478+0000", 2024-02-05T06:44:53.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0185e-05 2024-02-05T06:44:53.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.113718+0000", 2024-02-05T06:44:53.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96324064700000001 2024-02-05T06:44:53.506 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.506 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.506 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:53.506 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.113809+0000", 2024-02-05T06:44:53.506 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0779000000000006e-05 2024-02-05T06:44:53.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:44:53.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.113932+0000", 2024-02-05T06:44:53.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012355499999999999 2024-02-05T06:44:53.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.507 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:53.507 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.115313+0000", 2024-02-05T06:44:53.507 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0013801919999999999 2024-02-05T06:44:53.507 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.507 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.118853+0000", 2024-02-05T06:44:53.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003540259 2024-02-05T06:44:53.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.508 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.508 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.508 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.119080+0000", 2024-02-05T06:44:53.508 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022730999999999999 2024-02-05T06:44:53.508 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:53.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.119137+0000", 2024-02-05T06:44:53.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7105999999999998e-05 2024-02-05T06:44:53.509 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.509 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.509 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:53.509 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.119172+0000", 2024-02-05T06:44:53.509 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4615e-05 2024-02-05T06:44:53.510 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.510 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:53.510 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.510 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.510 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.510 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:46516 8.9 8:9b8c24b4:::benchmark_data_smithi028_105698_object46515:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-02-05T06:44:53.510 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:31.155494+0000", 2024-02-05T06:44:53.510 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 262.15974468600001, 2024-02-05T06:44:53.510 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96772978700000001, 2024-02-05T06:44:53.510 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:53.511 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:53.511 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:53.511 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:53.511 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:53.511 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46516 2024-02-05T06:44:53.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.511 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:53.511 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.511 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:53.511 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.155494+0000", 2024-02-05T06:44:53.511 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.511 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:53.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.155494+0000", 2024-02-05T06:44:53.512 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.512 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.512 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.512 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:53.512 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.155496+0000", 2024-02-05T06:44:53.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1239999999999999e-06 2024-02-05T06:44:53.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:53.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.155542+0000", 2024-02-05T06:44:53.513 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5765e-05 2024-02-05T06:44:53.513 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.513 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.513 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:53.513 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.155558+0000", 2024-02-05T06:44:53.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6393999999999999e-05 2024-02-05T06:44:53.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:53.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.155569+0000", 2024-02-05T06:44:53.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1443999999999999e-05 2024-02-05T06:44:53.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.514 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.514 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.514 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.155628+0000", 2024-02-05T06:44:53.514 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8758999999999998e-05 2024-02-05T06:44:53.514 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:53.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.155637+0000", 2024-02-05T06:44:53.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3489999999999992e-06 2024-02-05T06:44:53.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.515 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.515 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.115572+0000", 2024-02-05T06:44:53.515 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.95993589199999996 2024-02-05T06:44:53.515 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.515 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:53.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.115691+0000", 2024-02-05T06:44:53.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118769 2024-02-05T06:44:53.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:44:53.516 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.115822+0000", 2024-02-05T06:44:53.516 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000130548 2024-02-05T06:44:53.516 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.516 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.516 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:53.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.119035+0000", 2024-02-05T06:44:53.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032134780000000001 2024-02-05T06:44:53.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.517 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.123125+0000", 2024-02-05T06:44:53.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040896889999999997 2024-02-05T06:44:53.517 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.517 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.517 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.517 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.123148+0000", 2024-02-05T06:44:53.517 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2884e-05 2024-02-05T06:44:53.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:53.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.123188+0000", 2024-02-05T06:44:53.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0499e-05 2024-02-05T06:44:53.518 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.518 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.518 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:53.518 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.123224+0000", 2024-02-05T06:44:53.518 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5193e-05 2024-02-05T06:44:53.518 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.518 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:53.518 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.519 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.519 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.519 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:46524 8.9 8:9b1ff92e:::benchmark_data_smithi028_105698_object46523:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-02-05T06:44:53.519 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:31.164228+0000", 2024-02-05T06:44:53.519 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 262.151010257, 2024-02-05T06:44:53.519 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96416719799999995, 2024-02-05T06:44:53.519 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:53.519 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:53.519 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:53.519 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:53.519 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:53.519 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 46524 2024-02-05T06:44:53.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.520 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:53.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.520 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:53.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.164228+0000", 2024-02-05T06:44:53.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.520 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.520 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.520 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:53.520 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.164228+0000", 2024-02-05T06:44:53.520 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:53.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.164231+0000", 2024-02-05T06:44:53.521 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3860000000000001e-06 2024-02-05T06:44:53.521 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.521 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.521 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:53.521 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.164274+0000", 2024-02-05T06:44:53.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3903000000000003e-05 2024-02-05T06:44:53.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:53.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.164285+0000", 2024-02-05T06:44:53.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0502e-05 2024-02-05T06:44:53.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.522 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:53.522 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.164291+0000", 2024-02-05T06:44:53.522 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2820000000000002e-06 2024-02-05T06:44:53.522 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.522 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.164353+0000", 2024-02-05T06:44:53.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1515999999999999e-05 2024-02-05T06:44:53.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.523 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.523 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:53.523 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:31.164361+0000", 2024-02-05T06:44:53.523 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3119999999999994e-06 2024-02-05T06:44:53.523 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.117177+0000", 2024-02-05T06:44:53.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.95281617799999996 2024-02-05T06:44:53.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.524 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:53.524 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.117287+0000", 2024-02-05T06:44:53.524 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011019299999999999 2024-02-05T06:44:53.524 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.524 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3", 2024-02-05T06:44:53.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.117423+0000", 2024-02-05T06:44:53.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013603700000000001 2024-02-05T06:44:53.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.525 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:53.525 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.120441+0000", 2024-02-05T06:44:53.525 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030176249999999999 2024-02-05T06:44:53.525 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.525 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.526 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.526 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.123285+0000", 2024-02-05T06:44:53.526 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028436529999999998 2024-02-05T06:44:53.526 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.128298+0000", 2024-02-05T06:44:53.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0050136709999999999 2024-02-05T06:44:53.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.527 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.527 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:53.527 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.128357+0000", 2024-02-05T06:44:53.527 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8560000000000002e-05 2024-02-05T06:44:53.527 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.528 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:53.529 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.529 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:53.529 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:32.128395+0000", 2024-02-05T06:44:53.529 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8380000000000002e-05 2024-02-05T06:44:53.529 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.530 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:53.530 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.530 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.530 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.530 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:56711 8.5 8:a2495572:::benchmark_data_smithi028_105698_object56710:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e220)", 2024-02-05T06:44:53.530 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:46.336603+0000", 2024-02-05T06:44:53.530 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 246.97863545800001, 2024-02-05T06:44:53.530 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0099763980000001, 2024-02-05T06:44:53.530 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:53.530 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:53.530 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:53.531 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:53.531 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:53.531 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 56711 2024-02-05T06:44:53.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.531 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:53.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:53.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:46.336603+0000", 2024-02-05T06:44:53.531 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.531 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.531 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.531 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:53.531 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:46.336603+0000", 2024-02-05T06:44:53.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:53.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:46.336605+0000", 2024-02-05T06:44:53.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4480000000000001e-06 2024-02-05T06:44:53.532 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.532 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.532 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:53.532 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:46.336654+0000", 2024-02-05T06:44:53.532 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8424000000000003e-05 2024-02-05T06:44:53.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:53.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:46.336666+0000", 2024-02-05T06:44:53.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2055999999999999e-05 2024-02-05T06:44:53.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:53.533 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:46.336676+0000", 2024-02-05T06:44:53.533 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7909999999999999e-06 2024-02-05T06:44:53.533 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.533 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.533 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.534 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:46.336764+0000", 2024-02-05T06:44:53.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8301999999999996e-05 2024-02-05T06:44:53.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.534 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:53.534 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:46.336779+0000", 2024-02-05T06:44:53.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4817999999999999e-05 2024-02-05T06:44:53.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.534 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.534 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.534 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:47.340172+0000", 2024-02-05T06:44:53.534 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0033936670000001 2024-02-05T06:44:53.534 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:53.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:47.340315+0000", 2024-02-05T06:44:53.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014280400000000001 2024-02-05T06:44:53.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4,7", 2024-02-05T06:44:53.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:47.340489+0000", 2024-02-05T06:44:53.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017340199999999999 2024-02-05T06:44:53.535 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.535 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.535 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.535 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:47.342707+0000", 2024-02-05T06:44:53.535 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022178699999999998 2024-02-05T06:44:53.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:47.342746+0000", 2024-02-05T06:44:53.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9436e-05 2024-02-05T06:44:53.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.536 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:53.536 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:47.346521+0000", 2024-02-05T06:44:53.536 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0037745349999999999 2024-02-05T06:44:53.536 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.536 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:53.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:47.346554+0000", 2024-02-05T06:44:53.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3034000000000003e-05 2024-02-05T06:44:53.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.537 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:53.537 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:47.346579+0000", 2024-02-05T06:44:53.537 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5811000000000001e-05 2024-02-05T06:44:53.537 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.537 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:53.537 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.537 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.537 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.538 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:56716 8.5 8:a9b6369a:::benchmark_data_smithi028_105698_object56715:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e221)", 2024-02-05T06:44:53.538 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:46.387164+0000", 2024-02-05T06:44:53.538 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 246.928074415, 2024-02-05T06:44:53.538 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.95958765499999998, 2024-02-05T06:44:53.538 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:53.538 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:53.538 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:53.538 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:53.538 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:53.538 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 56716 2024-02-05T06:44:53.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.539 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:53.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:53.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:46.387164+0000", 2024-02-05T06:44:53.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.539 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:53.539 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:46.387164+0000", 2024-02-05T06:44:53.539 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.539 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.539 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:53.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:46.387166+0000", 2024-02-05T06:44:53.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.17e-06 2024-02-05T06:44:53.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:53.540 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:46.387220+0000", 2024-02-05T06:44:53.540 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4184999999999997e-05 2024-02-05T06:44:53.540 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.540 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.540 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:53.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:46.387232+0000", 2024-02-05T06:44:53.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1678999999999999e-05 2024-02-05T06:44:53.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:53.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:46.387240+0000", 2024-02-05T06:44:53.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7030000000000006e-06 2024-02-05T06:44:53.541 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.541 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.541 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.541 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:46.387290+0000", 2024-02-05T06:44:53.541 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9985999999999999e-05 2024-02-05T06:44:53.542 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.542 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:53.542 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:46.387298+0000", 2024-02-05T06:44:53.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4649999999999994e-06 2024-02-05T06:44:53.542 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.542 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.542 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:47.342807+0000", 2024-02-05T06:44:53.542 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.95550856399999995 2024-02-05T06:44:53.542 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.542 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:53.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:47.342956+0000", 2024-02-05T06:44:53.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014897000000000001 2024-02-05T06:44:53.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 4,7", 2024-02-05T06:44:53.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:47.343107+0000", 2024-02-05T06:44:53.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000151099 2024-02-05T06:44:53.543 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.543 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.543 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.543 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:47.346510+0000", 2024-02-05T06:44:53.543 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003403016 2024-02-05T06:44:53.544 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.544 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.544 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.544 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:47.346662+0000", 2024-02-05T06:44:53.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015245699999999999 2024-02-05T06:44:53.544 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.544 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.544 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:53.544 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:47.346705+0000", 2024-02-05T06:44:53.544 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2806999999999999e-05 2024-02-05T06:44:53.544 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:53.545 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:47.346732+0000", 2024-02-05T06:44:53.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6846e-05 2024-02-05T06:44:53.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.545 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.545 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:53.545 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:47.346752+0000", 2024-02-05T06:44:53.545 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9707999999999998e-05 2024-02-05T06:44:53.545 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.545 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:53.545 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.545 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.546 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.546 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:106745 8.9 8:91374fd2:::benchmark_data_smithi028_105698_object17470:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-02-05T06:44:53.546 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:41:42.403690+0000", 2024-02-05T06:44:53.546 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 190.91154875000001, 2024-02-05T06:44:53.546 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1376336390000001, 2024-02-05T06:44:53.546 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:53.546 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:44:53.546 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:53.546 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:44:53.546 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:44:53.546 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 106745 2024-02-05T06:44:53.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.547 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:53.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:53.547 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:42.403690+0000", 2024-02-05T06:44:53.547 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.547 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.547 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.547 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:53.547 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:42.403690+0000", 2024-02-05T06:44:53.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:53.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.548 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:53.548 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:42.403692+0000", 2024-02-05T06:44:53.548 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.88e-06 2024-02-05T06:44:53.548 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.548 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.548 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:53.548 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:42.403696+0000", 2024-02-05T06:44:53.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5990000000000004e-06 2024-02-05T06:44:53.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:53.549 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:42.403705+0000", 2024-02-05T06:44:53.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2480000000000002e-06 2024-02-05T06:44:53.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.549 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:53.549 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:42.403717+0000", 2024-02-05T06:44:53.549 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1505000000000001e-05 2024-02-05T06:44:53.549 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.549 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:42.405234+0000", 2024-02-05T06:44:53.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015175830000000001 2024-02-05T06:44:53.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:53.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:42.405246+0000", 2024-02-05T06:44:53.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1528999999999999e-05 2024-02-05T06:44:53.550 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.550 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.550 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.550 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:43.405595+0000", 2024-02-05T06:44:53.550 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0003489270000001 2024-02-05T06:44:53.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.551 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:44:53.551 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:43.405608+0000", 2024-02-05T06:44:53.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3149000000000001e-05 2024-02-05T06:44:53.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.551 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:53.551 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:44.528767+0000", 2024-02-05T06:44:53.551 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.123158745 2024-02-05T06:44:53.551 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.551 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.551 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:53.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:44.528938+0000", 2024-02-05T06:44:53.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000171616 2024-02-05T06:44:53.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3,5", 2024-02-05T06:44:53.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:44.529099+0000", 2024-02-05T06:44:53.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000160948 2024-02-05T06:44:53.552 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.552 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.552 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:44:53.552 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:44.532345+0000", 2024-02-05T06:44:53.552 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032457839999999998 2024-02-05T06:44:53.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.553 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.553 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:44.532390+0000", 2024-02-05T06:44:53.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4975000000000002e-05 2024-02-05T06:44:53.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.553 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.553 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:44.532545+0000", 2024-02-05T06:44:53.553 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015439699999999999 2024-02-05T06:44:53.553 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.553 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.553 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:44:53.553 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:44.541108+0000", 2024-02-05T06:44:53.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0085634679999999994 2024-02-05T06:44:53.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:44:53.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:44.541184+0000", 2024-02-05T06:44:53.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6196e-05 2024-02-05T06:44:53.554 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:53.554 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:53.554 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:44:53.554 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:41:44.541323+0000", 2024-02-05T06:44:53.554 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013909 2024-02-05T06:44:53.554 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.554 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:53.554 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.554 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:53.555 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:53.555 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:44:53.567 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:53.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:53.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:53.600 INFO:teuthology.orchestra.run.smithi028.stderr:set 6.12 pg_upmap_items mapping to [3->6,2->1,4->0] 2024-02-05T06:44:53.604 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:53 smithi167 ceph-mon[32774]: pgmap v555: 158 pgs: 158 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.18k op/s 2024-02-05T06:44:53.604 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:53 smithi167 ceph-mon[32774]: 3.e scrub starts 2024-02-05T06:44:53.605 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:53 smithi167 ceph-mon[32774]: 3.e scrub ok 2024-02-05T06:44:53.605 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:53 smithi167 ceph-mon[32774]: 5.8 scrub starts 2024-02-05T06:44:53.605 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:53 smithi167 ceph-mon[32774]: 5.8 scrub ok 2024-02-05T06:44:53.605 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:53 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/412014001' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.12", "id": [3, 6, 2, 1, 4, 0]}]: dispatch 2024-02-05T06:44:53.605 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:53 smithi167 ceph-mon[32774]: from='client.45371 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.12", "id": [3, 6, 2, 1, 4, 0]}]: dispatch 2024-02-05T06:44:53.669 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:53.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:53.713 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:53.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:53.814 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:53.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:53.856 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:53.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:53.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:53 smithi028 ceph-mon[37984]: pgmap v555: 158 pgs: 158 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.18k op/s 2024-02-05T06:44:53.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:53 smithi028 ceph-mon[37984]: 3.e scrub starts 2024-02-05T06:44:53.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:53 smithi028 ceph-mon[37984]: 3.e scrub ok 2024-02-05T06:44:53.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:53 smithi028 ceph-mon[37984]: 5.8 scrub starts 2024-02-05T06:44:53.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:53 smithi028 ceph-mon[37984]: 5.8 scrub ok 2024-02-05T06:44:53.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:53 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/412014001' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.12", "id": [3, 6, 2, 1, 4, 0]}]: dispatch 2024-02-05T06:44:53.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:53 smithi028 ceph-mon[37984]: from='client.45371 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.12", "id": [3, 6, 2, 1, 4, 0]}]: dispatch 2024-02-05T06:44:53.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:53 smithi028 ceph-mon[29442]: pgmap v555: 158 pgs: 158 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.18k op/s 2024-02-05T06:44:53.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:53 smithi028 ceph-mon[29442]: 3.e scrub starts 2024-02-05T06:44:53.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:53 smithi028 ceph-mon[29442]: 3.e scrub ok 2024-02-05T06:44:53.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:53 smithi028 ceph-mon[29442]: 5.8 scrub starts 2024-02-05T06:44:53.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:53 smithi028 ceph-mon[29442]: 5.8 scrub ok 2024-02-05T06:44:53.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:53 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/412014001' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.12", "id": [3, 6, 2, 1, 4, 0]}]: dispatch 2024-02-05T06:44:53.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:53 smithi028 ceph-mon[29442]: from='client.45371 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.12", "id": [3, 6, 2, 1, 4, 0]}]: dispatch 2024-02-05T06:44:53.957 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:53.997 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:53.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:53.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:54.099 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:54.124 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:54.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:54.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:54.133 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 74 16 113405 113389 95.7466 64 0.00411746 0.0104328 2024-02-05T06:44:54.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:54.251 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:54.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:54.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:54.353 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:54.379 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:54.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:54.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:54.481 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:54.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:54.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:54.619 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:54.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:54.654 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:54.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:54.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:54.807 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:54.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:54.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:54.909 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:44:54.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:54 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:44:54.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:54 smithi028 ceph-mon[37984]: 5.5 scrub starts 2024-02-05T06:44:54.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:54 smithi028 ceph-mon[37984]: 5.5 scrub ok 2024-02-05T06:44:54.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:54 smithi028 ceph-mon[37984]: 3.d scrub starts 2024-02-05T06:44:54.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:54 smithi028 ceph-mon[37984]: 3.d scrub ok 2024-02-05T06:44:54.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:54 smithi028 ceph-mon[37984]: from='client.45371 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.12", "id": [3, 6, 2, 1, 4, 0]}]': finished 2024-02-05T06:44:54.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:54 smithi028 ceph-mon[37984]: osdmap e314: 8 total, 8 up, 8 in 2024-02-05T06:44:54.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:54 smithi028 ceph-mon[37984]: 5.f scrub starts 2024-02-05T06:44:54.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:54 smithi028 ceph-mon[37984]: 5.f scrub ok 2024-02-05T06:44:54.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:54 smithi028 ceph-mon[37984]: 5.1f deep-scrub starts 2024-02-05T06:44:54.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:54 smithi028 ceph-mon[37984]: 5.1f deep-scrub ok 2024-02-05T06:44:54.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:54 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:44:54.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:54 smithi028 ceph-mon[29442]: 5.5 scrub starts 2024-02-05T06:44:54.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:54 smithi028 ceph-mon[29442]: 5.5 scrub ok 2024-02-05T06:44:54.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:54 smithi028 ceph-mon[29442]: 3.d scrub starts 2024-02-05T06:44:54.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:54 smithi028 ceph-mon[29442]: 3.d scrub ok 2024-02-05T06:44:54.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:54 smithi028 ceph-mon[29442]: from='client.45371 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.12", "id": [3, 6, 2, 1, 4, 0]}]': finished 2024-02-05T06:44:54.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:54 smithi028 ceph-mon[29442]: osdmap e314: 8 total, 8 up, 8 in 2024-02-05T06:44:54.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:54 smithi028 ceph-mon[29442]: 5.f scrub starts 2024-02-05T06:44:54.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:54 smithi028 ceph-mon[29442]: 5.f scrub ok 2024-02-05T06:44:54.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:54 smithi028 ceph-mon[29442]: 5.1f deep-scrub starts 2024-02-05T06:44:54.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:54 smithi028 ceph-mon[29442]: 5.1f deep-scrub ok 2024-02-05T06:44:54.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:54.954 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:54.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:44:54.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:54 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:44:54.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:54 smithi167 ceph-mon[32774]: 5.5 scrub starts 2024-02-05T06:44:54.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:54 smithi167 ceph-mon[32774]: 5.5 scrub ok 2024-02-05T06:44:54.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:54 smithi167 ceph-mon[32774]: 3.d scrub starts 2024-02-05T06:44:54.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:54 smithi167 ceph-mon[32774]: 3.d scrub ok 2024-02-05T06:44:54.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:54 smithi167 ceph-mon[32774]: from='client.45371 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.12", "id": [3, 6, 2, 1, 4, 0]}]': finished 2024-02-05T06:44:54.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:54 smithi167 ceph-mon[32774]: osdmap e314: 8 total, 8 up, 8 in 2024-02-05T06:44:54.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:54 smithi167 ceph-mon[32774]: 5.f scrub starts 2024-02-05T06:44:54.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:54 smithi167 ceph-mon[32774]: 5.f scrub ok 2024-02-05T06:44:54.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:54 smithi167 ceph-mon[32774]: 5.1f deep-scrub starts 2024-02-05T06:44:54.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:54 smithi167 ceph-mon[32774]: 5.1f deep-scrub ok 2024-02-05T06:44:55.055 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:55.098 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:55.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:55.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:55.133 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 75 16 114436 114420 95.3291 64.4375 0.00995505 0.0104848 2024-02-05T06:44:55.201 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:55.240 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:55.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:55.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:55.342 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:55.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:55.388 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:55.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:55.489 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:55.527 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:55.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:55.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:55.628 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:55.664 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:55.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:55.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:55.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:55.812 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:55.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:55.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:55.899 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:44:55.915 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:55.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:55 smithi028 ceph-mon[29442]: pgmap v557: 158 pgs: 158 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-02-05T06:44:55.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:55 smithi028 ceph-mon[29442]: osdmap e315: 8 total, 8 up, 8 in 2024-02-05T06:44:55.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:55 smithi028 ceph-mon[37984]: pgmap v557: 158 pgs: 158 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-02-05T06:44:55.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:55 smithi028 ceph-mon[37984]: osdmap e315: 8 total, 8 up, 8 in 2024-02-05T06:44:55.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:55.950 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:55.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:55.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:55 smithi167 ceph-mon[32774]: pgmap v557: 158 pgs: 158 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-02-05T06:44:55.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:55 smithi167 ceph-mon[32774]: osdmap e315: 8 total, 8 up, 8 in 2024-02-05T06:44:56.051 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:56.083 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:56.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:56.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:56.134 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 76 16 115677 115661 95.0953 77.5625 0.00839944 0.010511 2024-02-05T06:44:56.185 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:56.232 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:56.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:56.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:56.334 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:56.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:56.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:56.478 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:56.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:56.519 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:56.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:56.562 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-05T06:44:56.620 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:44:56.660 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:56.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:56.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:44:56.762 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:56.802 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:56.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:56.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:56.904 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:56.945 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:56.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:56.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:56.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:56 smithi028 ceph-mon[29442]: 6.18 deep-scrub starts 2024-02-05T06:44:56.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:56 smithi028 ceph-mon[29442]: 6.18 deep-scrub ok 2024-02-05T06:44:56.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:56 smithi028 ceph-mon[29442]: 5.a scrub starts 2024-02-05T06:44:56.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:56 smithi028 ceph-mon[29442]: 5.a scrub ok 2024-02-05T06:44:56.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:56 smithi028 ceph-mon[29442]: 5.15 scrub starts 2024-02-05T06:44:56.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:56 smithi028 ceph-mon[29442]: 5.15 scrub ok 2024-02-05T06:44:56.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:56 smithi028 ceph-mon[37984]: 6.18 deep-scrub starts 2024-02-05T06:44:56.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:56 smithi028 ceph-mon[37984]: 6.18 deep-scrub ok 2024-02-05T06:44:56.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:56 smithi028 ceph-mon[37984]: 5.a scrub starts 2024-02-05T06:44:56.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:56 smithi028 ceph-mon[37984]: 5.a scrub ok 2024-02-05T06:44:56.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:56 smithi028 ceph-mon[37984]: 5.15 scrub starts 2024-02-05T06:44:56.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:56 smithi028 ceph-mon[37984]: 5.15 scrub ok 2024-02-05T06:44:56.963 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:44:56.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:56 smithi167 ceph-mon[32774]: 6.18 deep-scrub starts 2024-02-05T06:44:56.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:56 smithi167 ceph-mon[32774]: 6.18 deep-scrub ok 2024-02-05T06:44:56.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:56 smithi167 ceph-mon[32774]: 5.a scrub starts 2024-02-05T06:44:56.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:56 smithi167 ceph-mon[32774]: 5.a scrub ok 2024-02-05T06:44:56.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:56 smithi167 ceph-mon[32774]: 5.15 scrub starts 2024-02-05T06:44:56.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:56 smithi167 ceph-mon[32774]: 5.15 scrub ok 2024-02-05T06:44:57.046 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:57.091 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:57.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:57.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:57.133 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 77 16 116914 116898 94.8643 77.3125 0.0062122 0.0105362 2024-02-05T06:44:57.194 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:57.239 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:57.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:57.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:57.267 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:44:57.341 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:57.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:57.373 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:57.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:57.474 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:57.513 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:57.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:57.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:57.615 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:44:57.660 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:57.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:57.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:44:57.763 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:57.801 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:57.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:57.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:57.903 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:57.940 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:57.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:57.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:57.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:57 smithi028 ceph-mon[29442]: 4.1b scrub starts 2024-02-05T06:44:57.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:57 smithi028 ceph-mon[29442]: 4.1b scrub ok 2024-02-05T06:44:57.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:57 smithi028 ceph-mon[29442]: 3.1b deep-scrub starts 2024-02-05T06:44:57.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:57 smithi028 ceph-mon[29442]: 3.1b deep-scrub ok 2024-02-05T06:44:57.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:57 smithi028 ceph-mon[29442]: pgmap v559: 158 pgs: 158 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.5 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-02-05T06:44:57.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:57 smithi028 ceph-mon[29442]: 4.15 scrub starts 2024-02-05T06:44:57.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:57 smithi028 ceph-mon[29442]: 4.15 scrub ok 2024-02-05T06:44:57.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:57 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:44:57.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:57 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:44:57.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:57 smithi028 ceph-mon[37984]: 4.1b scrub starts 2024-02-05T06:44:57.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:57 smithi028 ceph-mon[37984]: 4.1b scrub ok 2024-02-05T06:44:57.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:57 smithi028 ceph-mon[37984]: 3.1b deep-scrub starts 2024-02-05T06:44:57.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:57 smithi028 ceph-mon[37984]: 3.1b deep-scrub ok 2024-02-05T06:44:57.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:57 smithi028 ceph-mon[37984]: pgmap v559: 158 pgs: 158 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.5 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-02-05T06:44:57.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:57 smithi028 ceph-mon[37984]: 4.15 scrub starts 2024-02-05T06:44:57.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:57 smithi028 ceph-mon[37984]: 4.15 scrub ok 2024-02-05T06:44:57.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:57 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:44:57.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:57 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:44:57.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:57 smithi167 ceph-mon[32774]: 4.1b scrub starts 2024-02-05T06:44:57.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:57 smithi167 ceph-mon[32774]: 4.1b scrub ok 2024-02-05T06:44:57.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:57 smithi167 ceph-mon[32774]: 3.1b deep-scrub starts 2024-02-05T06:44:57.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:57 smithi167 ceph-mon[32774]: 3.1b deep-scrub ok 2024-02-05T06:44:57.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:57 smithi167 ceph-mon[32774]: pgmap v559: 158 pgs: 158 active+clean; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.5 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-02-05T06:44:57.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:57 smithi167 ceph-mon[32774]: 4.15 scrub starts 2024-02-05T06:44:57.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:57 smithi167 ceph-mon[32774]: 4.15 scrub ok 2024-02-05T06:44:57.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:57 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:44:57.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:57 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:44:58.042 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:58.078 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:58.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:58.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:58.134 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 78 16 118069 118053 94.5735 72.1875 0.0119038 0.0105694 2024-02-05T06:44:58.180 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:44:58.220 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:58.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:58.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:44:58.322 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:58.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:58.362 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:58.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:58.463 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:58.495 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:58.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:58.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:58.597 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:58.633 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:58.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:58.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:58.735 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:58.758 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:58.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:58.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:58.860 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:58.902 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:58.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:58.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:58.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:58 smithi028 ceph-mon[37984]: 4.9 scrub starts 2024-02-05T06:44:58.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:58 smithi028 ceph-mon[37984]: 4.9 scrub ok 2024-02-05T06:44:58.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:58 smithi028 ceph-mon[37984]: 6.15 scrub starts 2024-02-05T06:44:58.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:58 smithi028 ceph-mon[37984]: 6.15 scrub ok 2024-02-05T06:44:58.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:58 smithi028 ceph-mon[37984]: 5.16 deep-scrub starts 2024-02-05T06:44:58.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:58 smithi028 ceph-mon[37984]: 5.16 deep-scrub ok 2024-02-05T06:44:58.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:58 smithi028 ceph-mon[37984]: 5.1c scrub starts 2024-02-05T06:44:58.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:58 smithi028 ceph-mon[37984]: 5.1c scrub ok 2024-02-05T06:44:58.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:58 smithi028 ceph-mon[29442]: 4.9 scrub starts 2024-02-05T06:44:58.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:58 smithi028 ceph-mon[29442]: 4.9 scrub ok 2024-02-05T06:44:58.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:58 smithi028 ceph-mon[29442]: 6.15 scrub starts 2024-02-05T06:44:58.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:58 smithi028 ceph-mon[29442]: 6.15 scrub ok 2024-02-05T06:44:58.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:58 smithi028 ceph-mon[29442]: 5.16 deep-scrub starts 2024-02-05T06:44:58.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:58 smithi028 ceph-mon[29442]: 5.16 deep-scrub ok 2024-02-05T06:44:58.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:58 smithi028 ceph-mon[29442]: 5.1c scrub starts 2024-02-05T06:44:58.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:58 smithi028 ceph-mon[29442]: 5.1c scrub ok 2024-02-05T06:44:58.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:58 smithi167 ceph-mon[32774]: 4.9 scrub starts 2024-02-05T06:44:58.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:58 smithi167 ceph-mon[32774]: 4.9 scrub ok 2024-02-05T06:44:58.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:58 smithi167 ceph-mon[32774]: 6.15 scrub starts 2024-02-05T06:44:58.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:58 smithi167 ceph-mon[32774]: 6.15 scrub ok 2024-02-05T06:44:58.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:58 smithi167 ceph-mon[32774]: 5.16 deep-scrub starts 2024-02-05T06:44:58.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:58 smithi167 ceph-mon[32774]: 5.16 deep-scrub ok 2024-02-05T06:44:58.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:58 smithi167 ceph-mon[32774]: 5.1c scrub starts 2024-02-05T06:44:58.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:58 smithi167 ceph-mon[32774]: 5.1c scrub ok 2024-02-05T06:44:59.003 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:59.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:59.041 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:59.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:59.134 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 79 16 119193 119177 94.2656 70.25 0.0298147 0.0106039 2024-02-05T06:44:59.141 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:59.200 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:59.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:59.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:59.302 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:44:59.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:59.341 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:59.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:44:59.442 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:59.491 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:59.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:59.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:59.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:44:59.617 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:59.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:59.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:44:59.719 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:44:59.726 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:44:59.726 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-02-05T06:44:59.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:59.727 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.15390.0:119662 9.7 e315/291 9:e4b7335e:::benchmark_data_smithi028_129096_object119661:head v 315'7362, mlcod=315'7360)", 2024-02-05T06:44:59.727 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:44:59.717199+0000", 2024-02-05T06:44:59.727 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0020609539999999998, 2024-02-05T06:44:59.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020925129999999998, 2024-02-05T06:44:59.727 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:59.727 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-02-05T06:44:59.727 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:59.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:59.727 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:59.727 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:59.717199+0000", 2024-02-05T06:44:59.727 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:59.727 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:59.727 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:59.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:59.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:59.717199+0000", 2024-02-05T06:44:59.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:59.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:59.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:59.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:59.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:59.717200+0000", 2024-02-05T06:44:59.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6099999999999999e-07 2024-02-05T06:44:59.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:59.728 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:59.728 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:59.728 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:59.717224+0000", 2024-02-05T06:44:59.728 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3917000000000002e-05 2024-02-05T06:44:59.728 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:59.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:59.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:59.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:59.717233+0000", 2024-02-05T06:44:59.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0620000000000007e-06 2024-02-05T06:44:59.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:59.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:59.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:59.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:59.717239+0000", 2024-02-05T06:44:59.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4160000000000001e-06 2024-02-05T06:44:59.729 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:59.729 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:59.729 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:59.729 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:59.717345+0000", 2024-02-05T06:44:59.729 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010561299999999999 2024-02-05T06:44:59.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:59.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:59.730 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:59.730 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:59.717371+0000", 2024-02-05T06:44:59.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6237e-05 2024-02-05T06:44:59.730 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:59.730 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:59.730 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:59.730 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:59.730 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:59.730 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:119671 9.d 9:b4c81140:::benchmark_data_smithi028_129096_object119670:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-02-05T06:44:59.730 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:44:59.717324+0000", 2024-02-05T06:44:59.730 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0019360060000000001, 2024-02-05T06:44:59.730 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020489950000000001, 2024-02-05T06:44:59.731 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:59.731 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-02-05T06:44:59.731 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:44:59.731 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:44:59.731 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:44:59.731 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 119671 2024-02-05T06:44:59.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:59.731 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:59.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:59.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:59.731 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:59.717324+0000", 2024-02-05T06:44:59.731 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:59.731 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:59.731 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:59.731 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:59.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:59.717324+0000", 2024-02-05T06:44:59.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:59.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:59.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:59.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:59.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:59.717325+0000", 2024-02-05T06:44:59.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4050000000000001e-06 2024-02-05T06:44:59.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:59.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:59.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:59.732 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:59.717364+0000", 2024-02-05T06:44:59.732 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9001999999999997e-05 2024-02-05T06:44:59.732 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:59.732 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:59.732 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:59.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:59.717372+0000", 2024-02-05T06:44:59.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1480000000000006e-06 2024-02-05T06:44:59.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:59.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:59.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:59.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:59.717379+0000", 2024-02-05T06:44:59.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4320000000000004e-06 2024-02-05T06:44:59.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:59.733 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:59.733 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:44:59.733 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:59.717667+0000", 2024-02-05T06:44:59.733 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000288399 2024-02-05T06:44:59.733 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:59.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:59.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:44:59.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:59.717792+0000", 2024-02-05T06:44:59.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012496799999999999 2024-02-05T06:44:59.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:59.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:59.734 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,5", 2024-02-05T06:44:59.734 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:59.717967+0000", 2024-02-05T06:44:59.734 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000174587 2024-02-05T06:44:59.734 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:59.734 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:59.734 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:59.734 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:59.734 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:59.735 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.15390.0:119665 9.c e315/291)", 2024-02-05T06:44:59.735 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:44:59.718523+0000", 2024-02-05T06:44:59.735 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.00073634999999999998, 2024-02-05T06:44:59.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00092837200000000001, 2024-02-05T06:44:59.735 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:44:59.735 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "queued for pg", 2024-02-05T06:44:59.735 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:44:59.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:59.735 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:44:59.735 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:59.718523+0000", 2024-02-05T06:44:59.735 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:59.735 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:59.735 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:59.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:44:59.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:59.718523+0000", 2024-02-05T06:44:59.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:44:59.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:59.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:59.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:44:59.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:59.718525+0000", 2024-02-05T06:44:59.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.296e-06 2024-02-05T06:44:59.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:59.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:59.736 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:44:59.736 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:59.718556+0000", 2024-02-05T06:44:59.736 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0821000000000002e-05 2024-02-05T06:44:59.736 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:59.736 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:59.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:44:59.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:59.718569+0000", 2024-02-05T06:44:59.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.384e-05 2024-02-05T06:44:59.737 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:44:59.737 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:44:59.737 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:44:59.737 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:44:59.718577+0000", 2024-02-05T06:44:59.737 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8890000000000002e-06 2024-02-05T06:44:59.737 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:59.737 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:44:59.737 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:59.737 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:44:59.737 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-02-05T06:44:59.737 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 3 2024-02-05T06:44:59.737 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:44:59.759 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:44:59.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:59.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:44:59.861 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:44:59.898 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:44:59.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:44:59.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:44:59.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:59 smithi028 ceph-mon[37984]: 3.1 scrub starts 2024-02-05T06:44:59.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:59 smithi028 ceph-mon[37984]: 3.1 scrub ok 2024-02-05T06:44:59.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:59 smithi028 ceph-mon[37984]: pgmap v560: 158 pgs: 158 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-02-05T06:44:59.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:59 smithi028 ceph-mon[37984]: 6.3 scrub starts 2024-02-05T06:44:59.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:59 smithi028 ceph-mon[37984]: 6.3 scrub ok 2024-02-05T06:44:59.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:44:59 smithi028 ceph-mon[37984]: 9.f scrub starts 2024-02-05T06:44:59.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:59 smithi028 ceph-mon[29442]: 3.1 scrub starts 2024-02-05T06:44:59.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:59 smithi028 ceph-mon[29442]: 3.1 scrub ok 2024-02-05T06:44:59.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:59 smithi028 ceph-mon[29442]: pgmap v560: 158 pgs: 158 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-02-05T06:44:59.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:59 smithi028 ceph-mon[29442]: 6.3 scrub starts 2024-02-05T06:44:59.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:59 smithi028 ceph-mon[29442]: 6.3 scrub ok 2024-02-05T06:44:59.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:44:59 smithi028 ceph-mon[29442]: 9.f scrub starts 2024-02-05T06:44:59.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:59 smithi167 ceph-mon[32774]: 3.1 scrub starts 2024-02-05T06:44:59.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:59 smithi167 ceph-mon[32774]: 3.1 scrub ok 2024-02-05T06:44:59.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:59 smithi167 ceph-mon[32774]: pgmap v560: 158 pgs: 158 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-02-05T06:44:59.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:59 smithi167 ceph-mon[32774]: 6.3 scrub starts 2024-02-05T06:44:59.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:59 smithi167 ceph-mon[32774]: 6.3 scrub ok 2024-02-05T06:44:59.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:44:59 smithi167 ceph-mon[32774]: 9.f scrub starts 2024-02-05T06:44:59.999 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:00.046 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:00.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:00.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:00.134 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-02-05T06:45:00.128274+0000 min lat: 0.0012123 max lat: 0.149526 avg lat: 0.0106523 2024-02-05T06:45:00.134 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T06:45:00.134 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 80 16 120148 120132 93.8333 59.6875 0.027133 0.0106523 2024-02-05T06:45:00.147 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:00.179 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:00.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:00.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:00.281 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:00.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:00.318 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:00.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:00.419 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:00.456 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:00.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:00.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:00.516 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-05T06:45:00.557 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:00.596 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:00.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:00.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:00.698 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:00.739 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:00.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:00.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:00.789 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 4, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T06:45:00.789 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T06:45:00.789 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-05T06:45:00.789 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.046734 2024-02-05T06:45:00.789 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 4 0.04673354630430937 2024-02-05T06:45:00.829 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:00 smithi028 ceph-mon[29442]: 6.10 scrub starts 2024-02-05T06:45:00.829 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:00 smithi028 ceph-mon[29442]: 6.10 scrub ok 2024-02-05T06:45:00.829 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:00 smithi028 ceph-mon[29442]: 6.5 scrub starts 2024-02-05T06:45:00.829 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:00 smithi028 ceph-mon[29442]: 6.5 scrub ok 2024-02-05T06:45:00.829 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:00 smithi028 ceph-mon[29442]: 9.0 scrub starts 2024-02-05T06:45:00.829 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:00 smithi028 ceph-mon[37984]: 6.10 scrub starts 2024-02-05T06:45:00.829 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:00 smithi028 ceph-mon[37984]: 6.10 scrub ok 2024-02-05T06:45:00.829 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:00 smithi028 ceph-mon[37984]: 6.5 scrub starts 2024-02-05T06:45:00.829 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:00 smithi028 ceph-mon[37984]: 6.5 scrub ok 2024-02-05T06:45:00.829 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:00 smithi028 ceph-mon[37984]: 9.0 scrub starts 2024-02-05T06:45:00.838 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:45:00.841 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:00.882 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:00.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:00.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:00.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:00 smithi167 ceph-mon[32774]: 6.10 scrub starts 2024-02-05T06:45:00.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:00 smithi167 ceph-mon[32774]: 6.10 scrub ok 2024-02-05T06:45:00.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:00 smithi167 ceph-mon[32774]: 6.5 scrub starts 2024-02-05T06:45:00.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:00 smithi167 ceph-mon[32774]: 6.5 scrub ok 2024-02-05T06:45:00.977 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:00 smithi167 ceph-mon[32774]: 9.0 scrub starts 2024-02-05T06:45:00.984 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:01.008 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:01.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:01.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:01.110 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:01.134 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 81 16 120859 120843 93.2235 44.4375 0.0102343 0.0107216 2024-02-05T06:45:01.150 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:01.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:01.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:01.252 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:01.302 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:01.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:01.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:01.404 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:01.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:01.445 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:01.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:01.545 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:01.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:01.587 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:01.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:01.688 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:01.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:01.715 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:01.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:01.816 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:01.849 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:01.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:01.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:01.852 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-02-05T06:45:01.950 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:01.974 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:01.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:01.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:01.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:01 smithi167 ceph-mon[32774]: pgmap v561: 158 pgs: 158 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 44 MiB/s wr, 706 op/s 2024-02-05T06:45:01.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:01 smithi167 ceph-mon[32774]: 9.3 deep-scrub starts 2024-02-05T06:45:01.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:01 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1492063843' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:45:02.076 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:02.114 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:02.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:02.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:02.134 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 82 16 121643 121627 92.6842 49 0.00402557 0.0107827 2024-02-05T06:45:02.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:01 smithi028 ceph-mon[29442]: pgmap v561: 158 pgs: 158 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 44 MiB/s wr, 706 op/s 2024-02-05T06:45:02.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:01 smithi028 ceph-mon[29442]: 9.3 deep-scrub starts 2024-02-05T06:45:02.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:01 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1492063843' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:45:02.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:01 smithi028 ceph-mon[37984]: pgmap v561: 158 pgs: 158 active+clean; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 44 MiB/s wr, 706 op/s 2024-02-05T06:45:02.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:01 smithi028 ceph-mon[37984]: 9.3 deep-scrub starts 2024-02-05T06:45:02.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:01 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1492063843' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:45:02.216 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:02.251 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:02.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:02.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:02.353 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:02.390 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:02.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:02.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:02.492 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:02.536 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:02.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:02.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:02.638 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:02.670 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:02.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:02.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:02.772 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:02.815 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:02.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:02.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:02.916 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:02.948 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:45:02.949 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-02-05T06:45:02.949 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-02-05T06:45:02.949 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-02-05T06:45:02.949 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:45:02.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:02 smithi028 ceph-mon[29442]: 4.3 scrub starts 2024-02-05T06:45:02.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:02 smithi028 ceph-mon[29442]: 4.3 scrub ok 2024-02-05T06:45:02.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:02 smithi028 ceph-mon[29442]: 3.1c scrub starts 2024-02-05T06:45:02.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:02 smithi028 ceph-mon[29442]: 3.1c scrub ok 2024-02-05T06:45:02.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:02 smithi028 ceph-mon[29442]: 9.f scrub ok 2024-02-05T06:45:02.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:02 smithi028 ceph-mon[29442]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:45:02.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:02 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1492063843' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:45:02.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:02 smithi028 ceph-mon[29442]: osdmap e316: 8 total, 8 up, 8 in 2024-02-05T06:45:02.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:02 smithi028 ceph-mon[29442]: 4.1d scrub starts 2024-02-05T06:45:02.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:02 smithi028 ceph-mon[29442]: 4.1d scrub ok 2024-02-05T06:45:02.951 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:45:02 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:45:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:45:02.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:02 smithi028 ceph-mon[37984]: 4.3 scrub starts 2024-02-05T06:45:02.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:02 smithi028 ceph-mon[37984]: 4.3 scrub ok 2024-02-05T06:45:02.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:02 smithi028 ceph-mon[37984]: 3.1c scrub starts 2024-02-05T06:45:02.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:02 smithi028 ceph-mon[37984]: 3.1c scrub ok 2024-02-05T06:45:02.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:02 smithi028 ceph-mon[37984]: 9.f scrub ok 2024-02-05T06:45:02.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:02 smithi028 ceph-mon[37984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:45:02.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:02 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1492063843' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:45:02.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:02 smithi028 ceph-mon[37984]: osdmap e316: 8 total, 8 up, 8 in 2024-02-05T06:45:02.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:02 smithi028 ceph-mon[37984]: 4.1d scrub starts 2024-02-05T06:45:02.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:02 smithi028 ceph-mon[37984]: 4.1d scrub ok 2024-02-05T06:45:02.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:02.953 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:02.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:02.967 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:02 smithi167 ceph-mon[32774]: 4.3 scrub starts 2024-02-05T06:45:02.967 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:02 smithi167 ceph-mon[32774]: 4.3 scrub ok 2024-02-05T06:45:02.967 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:02 smithi167 ceph-mon[32774]: 3.1c scrub starts 2024-02-05T06:45:02.967 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:02 smithi167 ceph-mon[32774]: 3.1c scrub ok 2024-02-05T06:45:02.967 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:02 smithi167 ceph-mon[32774]: 9.f scrub ok 2024-02-05T06:45:02.967 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:02 smithi167 ceph-mon[32774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:45:02.967 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:02 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1492063843' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:45:02.967 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:02 smithi167 ceph-mon[32774]: osdmap e316: 8 total, 8 up, 8 in 2024-02-05T06:45:02.968 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:02 smithi167 ceph-mon[32774]: 4.1d scrub starts 2024-02-05T06:45:02.968 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:02 smithi167 ceph-mon[32774]: 4.1d scrub ok 2024-02-05T06:45:03.055 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:03.087 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:03.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:03.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:03.135 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 83 15 122685 122670 92.3516 65.1875 0.0135107 0.0108234 2024-02-05T06:45:03.168 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:45:03.168 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:45:03.189 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:03.233 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:45:03 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:45:03.248 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:03.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:03.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:03.350 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:03.381 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:03.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:03.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:03.483 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:03.517 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:03.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:03.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:03.619 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:03.655 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:03.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:03.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:03.721 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-05T06:45:03.757 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:03.799 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:03.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:03.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:03.901 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:03.946 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:03.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:03.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:03.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:03 smithi028 ceph-mon[37984]: pgmap v563: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 156 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-02-05T06:45:03.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:03 smithi028 ceph-mon[29442]: pgmap v563: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 156 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-02-05T06:45:03.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:03 smithi167 ceph-mon[32774]: pgmap v563: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 156 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-02-05T06:45:04.048 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:04.088 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:04.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:04.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:04.123 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:45:04.135 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 84 16 123761 123745 92.0519 67.1875 0.00843676 0.0108575 2024-02-05T06:45:04.190 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:04.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:04.226 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:04.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:04.327 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:04.365 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:04.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:04.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:04.466 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:04.506 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:04.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:04.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:04.607 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:04.665 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:04.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:04.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:04.747 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:04 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:45:04.747 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:04 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:45:04.767 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:04.808 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:04.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:04.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:04.910 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:04.943 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:04.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:04.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:04.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:04 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:45:05.035 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T06:45:05.045 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:05.086 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:05.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:05.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:05.136 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 85 16 124824 124808 91.7506 66.4375 0.0110473 0.0108938 2024-02-05T06:45:05.187 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:05.241 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:05.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:05.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:05.343 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:05.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:05.377 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:05.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:05.478 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:05.529 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:05.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:05.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:05.632 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:05.666 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:05.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:05.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:05.768 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:05.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:05.813 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:05.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:05.914 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:05.951 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:05.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:05.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:05.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:05 smithi167 ceph-mon[32774]: pgmap v564: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 156 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.0 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-02-05T06:45:05.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:05 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:45:05.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:05 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:45:06.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:06.089 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:05 smithi028 ceph-mon[29442]: pgmap v564: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 156 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.0 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-02-05T06:45:06.089 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:05 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:45:06.089 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:05 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:45:06.089 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:05 smithi028 ceph-mon[37984]: pgmap v564: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 156 active+clean; 7.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.0 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-02-05T06:45:06.089 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:05 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:45:06.090 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:05 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:45:06.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:06.090 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:06.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:06.136 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 86 16 125810 125794 91.3999 61.625 0.00818473 0.0109359 2024-02-05T06:45:06.148 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:45:06.148 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-02-05T06:45:06.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-02-05T06:45:06.148 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-02-05T06:45:06.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.148 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.44375.0:1114 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94598155039104 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-02-05T06:45:06.148 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:35:23.908978+0000", 2024-02-05T06:45:06.148 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 582.22526574699998, 2024-02-05T06:45:06.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0444529549999999, 2024-02-05T06:45:06.148 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:45:06.148 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-02-05T06:45:06.148 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:45:06.148 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.44375", 2024-02-05T06:45:06.148 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/1937500867", 2024-02-05T06:45:06.149 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1114 2024-02-05T06:45:06.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.149 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:45:06.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:45:06.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.908978+0000", 2024-02-05T06:45:06.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:45:06.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.908978+0000", 2024-02-05T06:45:06.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:45:06.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.908981+0000", 2024-02-05T06:45:06.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0299999999999998e-06 2024-02-05T06:45:06.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:45:06.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.908989+0000", 2024-02-05T06:45:06.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8080000000000005e-06 2024-02-05T06:45:06.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:45:06.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909008+0000", 2024-02-05T06:45:06.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8580000000000002e-05 2024-02-05T06:45:06.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:45:06.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909021+0000", 2024-02-05T06:45:06.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2858e-05 2024-02-05T06:45:06.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:45:06.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909101+0000", 2024-02-05T06:45:06.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0309000000000006e-05 2024-02-05T06:45:06.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-02-05T06:45:06.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.909109+0000", 2024-02-05T06:45:06.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9829999999999996e-06 2024-02-05T06:45:06.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:45:06.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.922952+0000", 2024-02-05T06:45:06.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.013843519 2024-02-05T06:45:06.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:45:06.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:23.922964+0000", 2024-02-05T06:45:06.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1982e-05 2024-02-05T06:45:06.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:45:06.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.946185+0000", 2024-02-05T06:45:06.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0232209109999999 2024-02-05T06:45:06.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for degraded object", 2024-02-05T06:45:06.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.946501+0000", 2024-02-05T06:45:06.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00031523400000000001 2024-02-05T06:45:06.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:45:06.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.951484+0000", 2024-02-05T06:45:06.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049829180000000002 2024-02-05T06:45:06.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:45:06.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.951567+0000", 2024-02-05T06:45:06.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3858999999999998e-05 2024-02-05T06:45:06.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:45:06.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.953431+0000", 2024-02-05T06:45:06.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0018639640000000001 2024-02-05T06:45:06.154 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.154 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:45:06.154 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.154 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.44375.0:1124 5.9 5:93e5b521:::notify.7:head [watch ping cookie 94598155039104 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-02-05T06:45:06.154 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:35:24.614839+0000", 2024-02-05T06:45:06.154 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 581.51940542299997, 2024-02-05T06:45:06.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.33742624700000001, 2024-02-05T06:45:06.154 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:45:06.154 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-02-05T06:45:06.154 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:45:06.154 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.44375", 2024-02-05T06:45:06.155 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/1937500867", 2024-02-05T06:45:06.155 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1124 2024-02-05T06:45:06.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.155 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:45:06.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:45:06.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.614839+0000", 2024-02-05T06:45:06.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:45:06.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.614839+0000", 2024-02-05T06:45:06.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:45:06.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.614847+0000", 2024-02-05T06:45:06.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4840000000000004e-06 2024-02-05T06:45:06.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:45:06.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.614853+0000", 2024-02-05T06:45:06.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9619999999999998e-06 2024-02-05T06:45:06.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:45:06.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.614860+0000", 2024-02-05T06:45:06.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.635e-06 2024-02-05T06:45:06.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:45:06.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.614869+0000", 2024-02-05T06:45:06.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5640000000000002e-06 2024-02-05T06:45:06.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:45:06.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.614962+0000", 2024-02-05T06:45:06.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2708999999999996e-05 2024-02-05T06:45:06.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:45:06.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.614974+0000", 2024-02-05T06:45:06.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2187999999999999e-05 2024-02-05T06:45:06.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.157 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:45:06.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.946534+0000", 2024-02-05T06:45:06.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.33156012000000001 2024-02-05T06:45:06.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for degraded object", 2024-02-05T06:45:06.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.946585+0000", 2024-02-05T06:45:06.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0203e-05 2024-02-05T06:45:06.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:45:06.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.952028+0000", 2024-02-05T06:45:06.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0054438680000000001 2024-02-05T06:45:06.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:45:06.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.952100+0000", 2024-02-05T06:45:06.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1254999999999995e-05 2024-02-05T06:45:06.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:45:06.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:35:24.952265+0000", 2024-02-05T06:45:06.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016525900000000001 2024-02-05T06:45:06.159 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.159 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:45:06.159 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.159 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14847.0:157978 7.5 7:a474196b:::benchmark_data_smithi028_77509_object42858:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-02-05T06:45:06.159 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:38:00.653715+0000", 2024-02-05T06:45:06.159 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 425.48052936599998, 2024-02-05T06:45:06.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.149159445, 2024-02-05T06:45:06.160 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:45:06.160 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:45:06.160 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:45:06.160 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14847", 2024-02-05T06:45:06.160 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:45:06.160 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 157978 2024-02-05T06:45:06.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.160 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:45:06.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:45:06.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.653715+0000", 2024-02-05T06:45:06.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:45:06.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.653715+0000", 2024-02-05T06:45:06.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:45:06.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.653716+0000", 2024-02-05T06:45:06.161 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.269e-06 2024-02-05T06:45:06.161 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.161 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:45:06.161 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.653720+0000", 2024-02-05T06:45:06.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1370000000000004e-06 2024-02-05T06:45:06.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:45:06.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.653735+0000", 2024-02-05T06:45:06.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.452e-05 2024-02-05T06:45:06.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:45:06.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.653743+0000", 2024-02-05T06:45:06.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4710000000000007e-06 2024-02-05T06:45:06.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:45:06.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.675803+0000", 2024-02-05T06:45:06.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022060158 2024-02-05T06:45:06.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:45:06.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.675903+0000", 2024-02-05T06:45:06.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9516000000000001e-05 2024-02-05T06:45:06.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,2", 2024-02-05T06:45:06.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.675987+0000", 2024-02-05T06:45:06.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4429000000000001e-05 2024-02-05T06:45:06.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:45:06.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.677212+0000", 2024-02-05T06:45:06.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0012252020000000001 2024-02-05T06:45:06.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.735136+0000", 2024-02-05T06:45:06.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.057923184000000003 2024-02-05T06:45:06.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.802760+0000", 2024-02-05T06:45:06.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.067624563999999998 2024-02-05T06:45:06.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:45:06.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.802819+0000", 2024-02-05T06:45:06.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8328999999999997e-05 2024-02-05T06:45:06.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.165 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:45:06.165 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:38:00.802874+0000", 2024-02-05T06:45:06.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5665999999999999e-05 2024-02-05T06:45:06.165 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.165 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:45:06.165 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.166 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14568.0:403 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e205)", 2024-02-05T06:45:06.166 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:26.898363+0000", 2024-02-05T06:45:06.166 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 279.23588084699998, 2024-02-05T06:45:06.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.18084518599999999, 2024-02-05T06:45:06.166 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:45:06.166 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-02-05T06:45:06.166 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:45:06.166 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14568", 2024-02-05T06:45:06.166 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.167:0/939366379", 2024-02-05T06:45:06.166 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 403 2024-02-05T06:45:06.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.166 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:45:06.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:45:06.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.898363+0000", 2024-02-05T06:45:06.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:45:06.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.898363+0000", 2024-02-05T06:45:06.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:45:06.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.898366+0000", 2024-02-05T06:45:06.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6520000000000002e-06 2024-02-05T06:45:06.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:45:06.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.898375+0000", 2024-02-05T06:45:06.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7390000000000005e-06 2024-02-05T06:45:06.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:45:06.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.898391+0000", 2024-02-05T06:45:06.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6427999999999999e-05 2024-02-05T06:45:06.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:45:06.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.898400+0000", 2024-02-05T06:45:06.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4610000000000008e-06 2024-02-05T06:45:06.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:45:06.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.898454+0000", 2024-02-05T06:45:06.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4317999999999997e-05 2024-02-05T06:45:06.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:45:06.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.898463+0000", 2024-02-05T06:45:06.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.704e-06 2024-02-05T06:45:06.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:45:06.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.071801+0000", 2024-02-05T06:45:06.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.17333821099999999 2024-02-05T06:45:06.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-02-05T06:45:06.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.071937+0000", 2024-02-05T06:45:06.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013592899999999999 2024-02-05T06:45:06.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:45:06.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.078945+0000", 2024-02-05T06:45:06.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0070086469999999998 2024-02-05T06:45:06.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:45:06.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.079046+0000", 2024-02-05T06:45:06.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010094099999999999 2024-02-05T06:45:06.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:45:06.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.079208+0000", 2024-02-05T06:45:06.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000162156 2024-02-05T06:45:06.171 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.171 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:45:06.171 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.171 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:120503 9.3 9:c1bb4e04:::benchmark_data_smithi028_129096_object120502:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-02-05T06:45:06.171 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:00.517201+0000", 2024-02-05T06:45:06.171 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.6170432080000001, 2024-02-05T06:45:06.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.230580603, 2024-02-05T06:45:06.171 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:45:06.172 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:45:06.172 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:45:06.172 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:45:06.172 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:45:06.172 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 120503 2024-02-05T06:45:06.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.172 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:45:06.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:45:06.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.517201+0000", 2024-02-05T06:45:06.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:45:06.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.517201+0000", 2024-02-05T06:45:06.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:45:06.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.517203+0000", 2024-02-05T06:45:06.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2079999999999999e-06 2024-02-05T06:45:06.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:45:06.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.517260+0000", 2024-02-05T06:45:06.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7302000000000001e-05 2024-02-05T06:45:06.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:45:06.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.517276+0000", 2024-02-05T06:45:06.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5087e-05 2024-02-05T06:45:06.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:45:06.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.517283+0000", 2024-02-05T06:45:06.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.892e-06 2024-02-05T06:45:06.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:45:06.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.546582+0000", 2024-02-05T06:45:06.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029298947999999998 2024-02-05T06:45:06.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:45:06.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.546939+0000", 2024-02-05T06:45:06.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00035615000000000001 2024-02-05T06:45:06.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:45:06.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.547084+0000", 2024-02-05T06:45:06.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014563799999999999 2024-02-05T06:45:06.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:45:06.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.550436+0000", 2024-02-05T06:45:06.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0033520300000000002 2024-02-05T06:45:06.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.568445+0000", 2024-02-05T06:45:06.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018008816 2024-02-05T06:45:06.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.747627+0000", 2024-02-05T06:45:06.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.179181488 2024-02-05T06:45:06.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:45:06.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.747728+0000", 2024-02-05T06:45:06.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101238 2024-02-05T06:45:06.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:45:06.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.747782+0000", 2024-02-05T06:45:06.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3805999999999999e-05 2024-02-05T06:45:06.177 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.177 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:45:06.178 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.178 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:120539 9.3 9:c73bd140:::benchmark_data_smithi028_129096_object120538:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-02-05T06:45:06.178 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:00.564915+0000", 2024-02-05T06:45:06.178 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.5693292320000003, 2024-02-05T06:45:06.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.183043764, 2024-02-05T06:45:06.178 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:45:06.178 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:45:06.178 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:45:06.178 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:45:06.178 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:45:06.178 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 120539 2024-02-05T06:45:06.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.179 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:45:06.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:45:06.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.564915+0000", 2024-02-05T06:45:06.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:45:06.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.564915+0000", 2024-02-05T06:45:06.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:45:06.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.564916+0000", 2024-02-05T06:45:06.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.336e-06 2024-02-05T06:45:06.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:45:06.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.564968+0000", 2024-02-05T06:45:06.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2193e-05 2024-02-05T06:45:06.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:45:06.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.564979+0000", 2024-02-05T06:45:06.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0319000000000001e-05 2024-02-05T06:45:06.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:45:06.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.564987+0000", 2024-02-05T06:45:06.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.011e-06 2024-02-05T06:45:06.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:45:06.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.565037+0000", 2024-02-05T06:45:06.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0166e-05 2024-02-05T06:45:06.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:45:06.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.565142+0000", 2024-02-05T06:45:06.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000105219 2024-02-05T06:45:06.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:45:06.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.565248+0000", 2024-02-05T06:45:06.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010539699999999999 2024-02-05T06:45:06.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:45:06.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.568097+0000", 2024-02-05T06:45:06.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028493239999999999 2024-02-05T06:45:06.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.684628+0000", 2024-02-05T06:45:06.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11653150499999999 2024-02-05T06:45:06.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.747826+0000", 2024-02-05T06:45:06.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.063197958999999998 2024-02-05T06:45:06.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:45:06.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.747913+0000", 2024-02-05T06:45:06.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6248999999999994e-05 2024-02-05T06:45:06.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:45:06.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.747959+0000", 2024-02-05T06:45:06.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6085999999999999e-05 2024-02-05T06:45:06.183 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.183 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:45:06.184 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.184 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:120557 9.3 9:ccd5b569:::benchmark_data_smithi028_129096_object120556:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-02-05T06:45:06.184 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:00.581421+0000", 2024-02-05T06:45:06.184 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.5528230990000003, 2024-02-05T06:45:06.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.166704621, 2024-02-05T06:45:06.184 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:45:06.184 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:45:06.184 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:45:06.184 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:45:06.184 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:45:06.184 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 120557 2024-02-05T06:45:06.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.185 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:45:06.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:45:06.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.581421+0000", 2024-02-05T06:45:06.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:45:06.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.581421+0000", 2024-02-05T06:45:06.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.185 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:45:06.185 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.581423+0000", 2024-02-05T06:45:06.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.142e-06 2024-02-05T06:45:06.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:45:06.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.581477+0000", 2024-02-05T06:45:06.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3495999999999997e-05 2024-02-05T06:45:06.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:45:06.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.581488+0000", 2024-02-05T06:45:06.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1548e-05 2024-02-05T06:45:06.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:45:06.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.581495+0000", 2024-02-05T06:45:06.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7719999999999997e-06 2024-02-05T06:45:06.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:45:06.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.582543+0000", 2024-02-05T06:45:06.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001048156 2024-02-05T06:45:06.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:45:06.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.582705+0000", 2024-02-05T06:45:06.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016137399999999999 2024-02-05T06:45:06.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:45:06.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.582890+0000", 2024-02-05T06:45:06.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018557900000000001 2024-02-05T06:45:06.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:45:06.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.585543+0000", 2024-02-05T06:45:06.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0026533519999999999 2024-02-05T06:45:06.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.747583+0000", 2024-02-05T06:45:06.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.16203905699999999 2024-02-05T06:45:06.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.747997+0000", 2024-02-05T06:45:06.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00041472900000000002 2024-02-05T06:45:06.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:45:06.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.748076+0000", 2024-02-05T06:45:06.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8548999999999996e-05 2024-02-05T06:45:06.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:45:06.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.748126+0000", 2024-02-05T06:45:06.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9867000000000002e-05 2024-02-05T06:45:06.189 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.189 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:45:06.189 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.190 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:120754 9.3 9:ca15cc16:::benchmark_data_smithi028_129096_object120753:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-02-05T06:45:06.190 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:00.930113+0000", 2024-02-05T06:45:06.190 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.2041314349999999, 2024-02-05T06:45:06.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13815897199999999, 2024-02-05T06:45:06.190 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:45:06.190 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:45:06.190 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:45:06.190 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:45:06.190 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:45:06.190 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 120754 2024-02-05T06:45:06.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.190 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:45:06.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:45:06.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.930113+0000", 2024-02-05T06:45:06.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:45:06.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.930113+0000", 2024-02-05T06:45:06.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:45:06.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.930114+0000", 2024-02-05T06:45:06.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5999999999999999e-06 2024-02-05T06:45:06.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:45:06.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.930168+0000", 2024-02-05T06:45:06.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3622000000000003e-05 2024-02-05T06:45:06.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:45:06.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.930179+0000", 2024-02-05T06:45:06.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1321000000000001e-05 2024-02-05T06:45:06.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:45:06.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.930185+0000", 2024-02-05T06:45:06.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0260000000000004e-06 2024-02-05T06:45:06.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:45:06.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.934034+0000", 2024-02-05T06:45:06.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038483530000000001 2024-02-05T06:45:06.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:45:06.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.934161+0000", 2024-02-05T06:45:06.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000127002 2024-02-05T06:45:06.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:45:06.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.934314+0000", 2024-02-05T06:45:06.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015326399999999999 2024-02-05T06:45:06.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:45:06.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.939194+0000", 2024-02-05T06:45:06.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048797060000000001 2024-02-05T06:45:06.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.038164+0000", 2024-02-05T06:45:06.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098970802999999996 2024-02-05T06:45:06.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.068112+0000", 2024-02-05T06:45:06.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029947537999999999 2024-02-05T06:45:06.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:45:06.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.068209+0000", 2024-02-05T06:45:06.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6971000000000004e-05 2024-02-05T06:45:06.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:45:06.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.068272+0000", 2024-02-05T06:45:06.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2766000000000002e-05 2024-02-05T06:45:06.195 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.196 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:45:06.196 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.196 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:120756 9.3 9:ce12c19a:::benchmark_data_smithi028_129096_object120755:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-02-05T06:45:06.196 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:00.933706+0000", 2024-02-05T06:45:06.196 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 5.2005386380000003, 2024-02-05T06:45:06.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.1661552, 2024-02-05T06:45:06.196 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:45:06.196 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:45:06.196 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:45:06.196 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:45:06.196 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:45:06.196 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 120756 2024-02-05T06:45:06.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.197 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:45:06.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:45:06.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.933706+0000", 2024-02-05T06:45:06.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:45:06.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.933706+0000", 2024-02-05T06:45:06.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:45:06.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.933708+0000", 2024-02-05T06:45:06.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.227e-06 2024-02-05T06:45:06.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:45:06.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.933749+0000", 2024-02-05T06:45:06.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1121999999999997e-05 2024-02-05T06:45:06.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:45:06.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.933759+0000", 2024-02-05T06:45:06.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9699999999999994e-06 2024-02-05T06:45:06.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:45:06.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.933767+0000", 2024-02-05T06:45:06.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5269999999999998e-06 2024-02-05T06:45:06.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:45:06.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.936529+0000", 2024-02-05T06:45:06.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00276202 2024-02-05T06:45:06.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:45:06.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.936630+0000", 2024-02-05T06:45:06.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000100756 2024-02-05T06:45:06.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:45:06.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.936730+0000", 2024-02-05T06:45:06.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9611000000000006e-05 2024-02-05T06:45:06.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:45:06.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.939470+0000", 2024-02-05T06:45:06.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00274011 2024-02-05T06:45:06.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.044387+0000", 2024-02-05T06:45:06.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10491689 2024-02-05T06:45:06.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.099752+0000", 2024-02-05T06:45:06.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055365020000000001 2024-02-05T06:45:06.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:45:06.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.099805+0000", 2024-02-05T06:45:06.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3426e-05 2024-02-05T06:45:06.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:45:06.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.099861+0000", 2024-02-05T06:45:06.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5520999999999999e-05 2024-02-05T06:45:06.201 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.201 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:45:06.201 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.201 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:121075 9.3 9:cfccf10c:::benchmark_data_smithi028_129096_object121074:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-02-05T06:45:06.202 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:01.351309+0000", 2024-02-05T06:45:06.202 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.7829353599999997, 2024-02-05T06:45:06.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.147336777, 2024-02-05T06:45:06.202 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:45:06.202 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:45:06.202 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:45:06.202 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:45:06.202 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:45:06.202 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 121075 2024-02-05T06:45:06.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.202 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:45:06.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:45:06.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.351309+0000", 2024-02-05T06:45:06.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:45:06.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.351309+0000", 2024-02-05T06:45:06.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:45:06.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.351311+0000", 2024-02-05T06:45:06.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8500000000000001e-06 2024-02-05T06:45:06.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:45:06.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.351361+0000", 2024-02-05T06:45:06.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.028e-05 2024-02-05T06:45:06.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:45:06.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.351373+0000", 2024-02-05T06:45:06.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1669e-05 2024-02-05T06:45:06.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:45:06.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.351379+0000", 2024-02-05T06:45:06.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2369999999999997e-06 2024-02-05T06:45:06.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:45:06.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.380406+0000", 2024-02-05T06:45:06.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029026700999999999 2024-02-05T06:45:06.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:45:06.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.380557+0000", 2024-02-05T06:45:06.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015128200000000001 2024-02-05T06:45:06.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:45:06.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.380739+0000", 2024-02-05T06:45:06.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018184900000000001 2024-02-05T06:45:06.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:45:06.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.382006+0000", 2024-02-05T06:45:06.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001267476 2024-02-05T06:45:06.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.474802+0000", 2024-02-05T06:45:06.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.092795680000000005 2024-02-05T06:45:06.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.498502+0000", 2024-02-05T06:45:06.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023699989000000001 2024-02-05T06:45:06.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:45:06.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.498596+0000", 2024-02-05T06:45:06.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4557999999999998e-05 2024-02-05T06:45:06.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:45:06.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.498646+0000", 2024-02-05T06:45:06.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9206000000000003e-05 2024-02-05T06:45:06.207 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.207 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:45:06.207 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.207 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:121278 9.3 9:c5b97c0a:::benchmark_data_smithi028_129096_object121277:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-02-05T06:45:06.207 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:01.606264+0000", 2024-02-05T06:45:06.207 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.5279798289999995, 2024-02-05T06:45:06.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.19801748599999999, 2024-02-05T06:45:06.208 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:45:06.208 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:45:06.208 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:45:06.208 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:45:06.208 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:45:06.208 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 121278 2024-02-05T06:45:06.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.208 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:45:06.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:45:06.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.606264+0000", 2024-02-05T06:45:06.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:45:06.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.606264+0000", 2024-02-05T06:45:06.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:45:06.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.606266+0000", 2024-02-05T06:45:06.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.637e-06 2024-02-05T06:45:06.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:45:06.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.606300+0000", 2024-02-05T06:45:06.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3850000000000003e-05 2024-02-05T06:45:06.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:45:06.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.606317+0000", 2024-02-05T06:45:06.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6960999999999999e-05 2024-02-05T06:45:06.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:45:06.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.606324+0000", 2024-02-05T06:45:06.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7939999999999997e-06 2024-02-05T06:45:06.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:45:06.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.627530+0000", 2024-02-05T06:45:06.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021206827000000001 2024-02-05T06:45:06.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:45:06.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.627632+0000", 2024-02-05T06:45:06.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010118 2024-02-05T06:45:06.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:45:06.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.627764+0000", 2024-02-05T06:45:06.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013247399999999999 2024-02-05T06:45:06.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.638046+0000", 2024-02-05T06:45:06.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010282224 2024-02-05T06:45:06.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:45:06.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.672097+0000", 2024-02-05T06:45:06.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.034050992000000002 2024-02-05T06:45:06.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.804158+0000", 2024-02-05T06:45:06.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13206031300000001 2024-02-05T06:45:06.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:45:06.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.804214+0000", 2024-02-05T06:45:06.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6702e-05 2024-02-05T06:45:06.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:45:06.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.804282+0000", 2024-02-05T06:45:06.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7532000000000003e-05 2024-02-05T06:45:06.213 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.213 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:45:06.213 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.213 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:06.214 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:121284 9.3 9:cc770777:::benchmark_data_smithi028_129096_object121283:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-02-05T06:45:06.214 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:01.607527+0000", 2024-02-05T06:45:06.214 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.5267175120000003, 2024-02-05T06:45:06.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.204634289, 2024-02-05T06:45:06.215 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:45:06.215 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:45:06.215 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:45:06.215 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:45:06.215 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:45:06.215 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 121284 2024-02-05T06:45:06.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.215 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:45:06.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:45:06.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.607527+0000", 2024-02-05T06:45:06.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:45:06.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.607527+0000", 2024-02-05T06:45:06.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:45:06.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.607528+0000", 2024-02-05T06:45:06.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2869999999999999e-06 2024-02-05T06:45:06.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:45:06.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.607578+0000", 2024-02-05T06:45:06.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9703999999999997e-05 2024-02-05T06:45:06.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:45:06.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.607588+0000", 2024-02-05T06:45:06.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0623e-05 2024-02-05T06:45:06.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:45:06.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.607594+0000", 2024-02-05T06:45:06.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1990000000000003e-06 2024-02-05T06:45:06.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:45:06.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.630799+0000", 2024-02-05T06:45:06.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023204758999999998 2024-02-05T06:45:06.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:45:06.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.630920+0000", 2024-02-05T06:45:06.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120815 2024-02-05T06:45:06.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:45:06.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.631209+0000", 2024-02-05T06:45:06.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028938500000000002 2024-02-05T06:45:06.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.638095+0000", 2024-02-05T06:45:06.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.006885408 2024-02-05T06:45:06.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:45:06.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.672194+0000", 2024-02-05T06:45:06.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.034099061 2024-02-05T06:45:06.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.811988+0000", 2024-02-05T06:45:06.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13979428599999999 2024-02-05T06:45:06.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:45:06.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.812093+0000", 2024-02-05T06:45:06.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010472299999999999 2024-02-05T06:45:06.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:45:06.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.812161+0000", 2024-02-05T06:45:06.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8039000000000004e-05 2024-02-05T06:45:06.220 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.220 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:45:06.220 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.220 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:121294 9.3 9:c54f2ea6:::benchmark_data_smithi028_129096_object121293:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-02-05T06:45:06.220 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:01.615831+0000", 2024-02-05T06:45:06.220 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.5184126759999996, 2024-02-05T06:45:06.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20027431800000001, 2024-02-05T06:45:06.220 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:45:06.220 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:45:06.220 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:45:06.221 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:45:06.221 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:45:06.221 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 121294 2024-02-05T06:45:06.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.221 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:45:06.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:45:06.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.615831+0000", 2024-02-05T06:45:06.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:45:06.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.615831+0000", 2024-02-05T06:45:06.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:45:06.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.615833+0000", 2024-02-05T06:45:06.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.362e-06 2024-02-05T06:45:06.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:45:06.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.615907+0000", 2024-02-05T06:45:06.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4153000000000006e-05 2024-02-05T06:45:06.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:45:06.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.615917+0000", 2024-02-05T06:45:06.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0443e-05 2024-02-05T06:45:06.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:45:06.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.615925+0000", 2024-02-05T06:45:06.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4930000000000001e-06 2024-02-05T06:45:06.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:45:06.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.635225+0000", 2024-02-05T06:45:06.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019300033000000001 2024-02-05T06:45:06.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:45:06.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.635370+0000", 2024-02-05T06:45:06.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014553100000000001 2024-02-05T06:45:06.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:45:06.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.635524+0000", 2024-02-05T06:45:06.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015344299999999999 2024-02-05T06:45:06.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.672146+0000", 2024-02-05T06:45:06.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036622447000000002 2024-02-05T06:45:06.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:45:06.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.672198+0000", 2024-02-05T06:45:06.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2027000000000002e-05 2024-02-05T06:45:06.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.815828+0000", 2024-02-05T06:45:06.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14362993299999999 2024-02-05T06:45:06.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:45:06.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.816045+0000", 2024-02-05T06:45:06.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000216899 2024-02-05T06:45:06.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:45:06.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.816106+0000", 2024-02-05T06:45:06.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0554000000000003e-05 2024-02-05T06:45:06.226 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.226 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:45:06.226 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.226 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:121296 9.3 9:cf583f78:::benchmark_data_smithi028_129096_object121295:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-02-05T06:45:06.226 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:01.616489+0000", 2024-02-05T06:45:06.226 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.51775489, 2024-02-05T06:45:06.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.19977958800000001, 2024-02-05T06:45:06.226 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:45:06.226 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:45:06.226 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:45:06.226 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:45:06.226 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:45:06.227 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 121296 2024-02-05T06:45:06.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.227 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:45:06.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:45:06.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.616489+0000", 2024-02-05T06:45:06.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:45:06.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.616489+0000", 2024-02-05T06:45:06.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:45:06.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.616490+0000", 2024-02-05T06:45:06.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.19e-06 2024-02-05T06:45:06.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:45:06.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.616543+0000", 2024-02-05T06:45:06.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2602000000000002e-05 2024-02-05T06:45:06.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:45:06.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.616554+0000", 2024-02-05T06:45:06.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0669999999999999e-05 2024-02-05T06:45:06.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:45:06.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.616561+0000", 2024-02-05T06:45:06.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3119999999999996e-06 2024-02-05T06:45:06.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:45:06.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.638128+0000", 2024-02-05T06:45:06.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021566703999999999 2024-02-05T06:45:06.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:45:06.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.638253+0000", 2024-02-05T06:45:06.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012526700000000001 2024-02-05T06:45:06.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:45:06.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.638404+0000", 2024-02-05T06:45:06.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000150989 2024-02-05T06:45:06.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.672186+0000", 2024-02-05T06:45:06.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.033781904000000001 2024-02-05T06:45:06.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:45:06.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.672202+0000", 2024-02-05T06:45:06.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6025e-05 2024-02-05T06:45:06.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.816148+0000", 2024-02-05T06:45:06.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14394652399999999 2024-02-05T06:45:06.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:45:06.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.816223+0000", 2024-02-05T06:45:06.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5007999999999997e-05 2024-02-05T06:45:06.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:45:06.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.816269+0000", 2024-02-05T06:45:06.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5392999999999997e-05 2024-02-05T06:45:06.232 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.232 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:45:06.232 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.232 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:121307 9.e 9:7ff133d8:::benchmark_data_smithi028_129096_object121306:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-02-05T06:45:06.232 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:01.628879+0000", 2024-02-05T06:45:06.232 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.5053651590000001, 2024-02-05T06:45:06.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.175348843, 2024-02-05T06:45:06.232 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:45:06.232 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:45:06.232 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:45:06.232 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:45:06.233 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:45:06.233 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 121307 2024-02-05T06:45:06.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.233 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:45:06.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:45:06.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.628879+0000", 2024-02-05T06:45:06.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:45:06.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.628879+0000", 2024-02-05T06:45:06.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:45:06.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.628880+0000", 2024-02-05T06:45:06.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3990000000000001e-06 2024-02-05T06:45:06.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:45:06.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.628940+0000", 2024-02-05T06:45:06.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9459000000000001e-05 2024-02-05T06:45:06.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:45:06.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.628950+0000", 2024-02-05T06:45:06.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0020999999999999e-05 2024-02-05T06:45:06.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:45:06.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.628958+0000", 2024-02-05T06:45:06.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0989999999999999e-06 2024-02-05T06:45:06.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:45:06.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.629006+0000", 2024-02-05T06:45:06.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7624999999999999e-05 2024-02-05T06:45:06.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:45:06.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.629096+0000", 2024-02-05T06:45:06.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0086000000000005e-05 2024-02-05T06:45:06.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,4", 2024-02-05T06:45:06.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.629183+0000", 2024-02-05T06:45:06.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7297000000000001e-05 2024-02-05T06:45:06.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.634835+0000", 2024-02-05T06:45:06.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005651868 2024-02-05T06:45:06.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:45:06.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.634856+0000", 2024-02-05T06:45:06.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1155e-05 2024-02-05T06:45:06.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.804142+0000", 2024-02-05T06:45:06.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.16928574499999999 2024-02-05T06:45:06.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:45:06.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.804203+0000", 2024-02-05T06:45:06.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1723999999999994e-05 2024-02-05T06:45:06.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:45:06.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.804228+0000", 2024-02-05T06:45:06.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4365000000000001e-05 2024-02-05T06:45:06.238 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.238 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:45:06.238 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.238 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:121308 9.e 9:774caccc:::benchmark_data_smithi028_129096_object121307:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-02-05T06:45:06.238 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:01.630277+0000", 2024-02-05T06:45:06.238 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.5039671190000004, 2024-02-05T06:45:06.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.17403511599999999, 2024-02-05T06:45:06.238 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:45:06.238 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:45:06.238 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:45:06.238 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:45:06.238 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:45:06.238 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 121308 2024-02-05T06:45:06.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.239 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:45:06.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:45:06.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.630277+0000", 2024-02-05T06:45:06.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:45:06.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.630277+0000", 2024-02-05T06:45:06.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:45:06.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.630279+0000", 2024-02-05T06:45:06.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4789999999999999e-06 2024-02-05T06:45:06.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:45:06.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.630352+0000", 2024-02-05T06:45:06.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3459000000000003e-05 2024-02-05T06:45:06.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:45:06.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.630363+0000", 2024-02-05T06:45:06.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1199999999999999e-05 2024-02-05T06:45:06.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:45:06.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.630371+0000", 2024-02-05T06:45:06.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2330000000000006e-06 2024-02-05T06:45:06.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:45:06.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.632159+0000", 2024-02-05T06:45:06.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0017879199999999999 2024-02-05T06:45:06.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:45:06.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.632287+0000", 2024-02-05T06:45:06.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012743800000000001 2024-02-05T06:45:06.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,4", 2024-02-05T06:45:06.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.632431+0000", 2024-02-05T06:45:06.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014446700000000001 2024-02-05T06:45:06.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:45:06.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.635043+0000", 2024-02-05T06:45:06.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0026122419999999999 2024-02-05T06:45:06.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.636549+0000", 2024-02-05T06:45:06.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015052080000000001 2024-02-05T06:45:06.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.804261+0000", 2024-02-05T06:45:06.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.16771262000000001 2024-02-05T06:45:06.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:45:06.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.804292+0000", 2024-02-05T06:45:06.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0821999999999997e-05 2024-02-05T06:45:06.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:45:06.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.804312+0000", 2024-02-05T06:45:06.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0027999999999999e-05 2024-02-05T06:45:06.243 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.243 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:45:06.243 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.244 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:121445 9.3 9:c33e18e6:::benchmark_data_smithi028_129096_object121444:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-02-05T06:45:06.244 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:01.902455+0000", 2024-02-05T06:45:06.244 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.2317887489999997, 2024-02-05T06:45:06.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.15157683999999999, 2024-02-05T06:45:06.244 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:45:06.244 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:45:06.244 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:45:06.244 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:45:06.244 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:45:06.244 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 121445 2024-02-05T06:45:06.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.244 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:45:06.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:45:06.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.902455+0000", 2024-02-05T06:45:06.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:45:06.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.902455+0000", 2024-02-05T06:45:06.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:45:06.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.902456+0000", 2024-02-05T06:45:06.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2900000000000002e-07 2024-02-05T06:45:06.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:45:06.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.902513+0000", 2024-02-05T06:45:06.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6679999999999999e-05 2024-02-05T06:45:06.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:45:06.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.902521+0000", 2024-02-05T06:45:06.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1049999999999995e-06 2024-02-05T06:45:06.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:45:06.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.902528+0000", 2024-02-05T06:45:06.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5030000000000002e-06 2024-02-05T06:45:06.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:45:06.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.943463+0000", 2024-02-05T06:45:06.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.040935188999999997 2024-02-05T06:45:06.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:45:06.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.943593+0000", 2024-02-05T06:45:06.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013027700000000001 2024-02-05T06:45:06.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:45:06.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.943752+0000", 2024-02-05T06:45:06.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015859500000000001 2024-02-05T06:45:06.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:45:06.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.945887+0000", 2024-02-05T06:45:06.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002135227 2024-02-05T06:45:06.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:02.053828+0000", 2024-02-05T06:45:06.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.107940778 2024-02-05T06:45:06.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:02.053920+0000", 2024-02-05T06:45:06.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2571999999999994e-05 2024-02-05T06:45:06.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:45:06.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:02.053974+0000", 2024-02-05T06:45:06.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3718999999999998e-05 2024-02-05T06:45:06.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:45:06.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:02.054032+0000", 2024-02-05T06:45:06.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8366000000000003e-05 2024-02-05T06:45:06.249 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.249 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:45:06.249 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.249 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:121458 9.3 9:c86ca6f3:::benchmark_data_smithi028_129096_object121457:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-02-05T06:45:06.249 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:01.914251+0000", 2024-02-05T06:45:06.249 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.2199934309999998, 2024-02-05T06:45:06.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13998217900000001, 2024-02-05T06:45:06.250 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:45:06.250 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:45:06.250 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:45:06.250 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:45:06.250 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:45:06.250 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 121458 2024-02-05T06:45:06.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.250 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:45:06.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:45:06.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.914251+0000", 2024-02-05T06:45:06.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:45:06.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.914251+0000", 2024-02-05T06:45:06.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:45:06.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.914253+0000", 2024-02-05T06:45:06.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9e-06 2024-02-05T06:45:06.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:45:06.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.914308+0000", 2024-02-05T06:45:06.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5823999999999999e-05 2024-02-05T06:45:06.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:45:06.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.914320+0000", 2024-02-05T06:45:06.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1473e-05 2024-02-05T06:45:06.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:45:06.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.914333+0000", 2024-02-05T06:45:06.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3382e-05 2024-02-05T06:45:06.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:45:06.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.944792+0000", 2024-02-05T06:45:06.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.030458959000000001 2024-02-05T06:45:06.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:45:06.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.944887+0000", 2024-02-05T06:45:06.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4357999999999993e-05 2024-02-05T06:45:06.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:45:06.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.944975+0000", 2024-02-05T06:45:06.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8821999999999998e-05 2024-02-05T06:45:06.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:45:06.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.947252+0000", 2024-02-05T06:45:06.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022763430000000001 2024-02-05T06:45:06.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:02.053876+0000", 2024-02-05T06:45:06.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.106624124 2024-02-05T06:45:06.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:02.054098+0000", 2024-02-05T06:45:06.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022181999999999999 2024-02-05T06:45:06.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:45:06.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:02.054183+0000", 2024-02-05T06:45:06.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4778999999999996e-05 2024-02-05T06:45:06.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:45:06.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:02.054233+0000", 2024-02-05T06:45:06.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0395000000000001e-05 2024-02-05T06:45:06.255 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.255 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:45:06.256 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.256 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:121466 9.3 9:cfd2fce8:::benchmark_data_smithi028_129096_object121465:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-02-05T06:45:06.256 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:01.915221+0000", 2024-02-05T06:45:06.256 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 4.2190228120000004, 2024-02-05T06:45:06.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13948618800000001, 2024-02-05T06:45:06.256 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:45:06.256 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:45:06.256 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:45:06.256 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:45:06.256 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:45:06.256 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 121466 2024-02-05T06:45:06.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.257 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:45:06.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:45:06.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.915221+0000", 2024-02-05T06:45:06.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:45:06.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.915221+0000", 2024-02-05T06:45:06.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:45:06.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.915223+0000", 2024-02-05T06:45:06.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2619999999999999e-06 2024-02-05T06:45:06.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:45:06.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.915297+0000", 2024-02-05T06:45:06.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3987999999999997e-05 2024-02-05T06:45:06.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:45:06.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.915310+0000", 2024-02-05T06:45:06.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3071000000000001e-05 2024-02-05T06:45:06.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:45:06.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.915318+0000", 2024-02-05T06:45:06.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8730000000000007e-06 2024-02-05T06:45:06.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:45:06.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.945908+0000", 2024-02-05T06:45:06.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.030590965000000001 2024-02-05T06:45:06.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:45:06.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.945993+0000", 2024-02-05T06:45:06.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4932000000000006e-05 2024-02-05T06:45:06.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:45:06.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.946130+0000", 2024-02-05T06:45:06.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013608899999999999 2024-02-05T06:45:06.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:45:06.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.947342+0000", 2024-02-05T06:45:06.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001212271 2024-02-05T06:45:06.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:02.054282+0000", 2024-02-05T06:45:06.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.106940691 2024-02-05T06:45:06.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:02.054482+0000", 2024-02-05T06:45:06.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019943599999999999 2024-02-05T06:45:06.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:45:06.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:02.054575+0000", 2024-02-05T06:45:06.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2621999999999995e-05 2024-02-05T06:45:06.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:45:06.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:02.054708+0000", 2024-02-05T06:45:06.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013298800000000001 2024-02-05T06:45:06.261 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.261 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:45:06.262 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.262 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:125072 9.3 9:c7f020ae:::benchmark_data_smithi028_129096_object125071:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-02-05T06:45:06.262 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:05.377918+0000", 2024-02-05T06:45:06.262 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.756326159, 2024-02-05T06:45:06.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14184434800000001, 2024-02-05T06:45:06.262 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:45:06.262 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:45:06.262 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:45:06.262 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:45:06.262 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:45:06.262 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 125072 2024-02-05T06:45:06.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.262 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:45:06.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:45:06.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:05.377918+0000", 2024-02-05T06:45:06.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:45:06.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:05.377918+0000", 2024-02-05T06:45:06.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:45:06.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:45:06.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:05.377921+0000", 2024-02-05T06:45:06.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7049999999999999e-06 2024-02-05T06:45:06.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:45:06.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:05.377984+0000", 2024-02-05T06:45:06.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2971000000000004e-05 2024-02-05T06:45:06.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:45:06.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:05.378004+0000", 2024-02-05T06:45:06.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9935000000000001e-05 2024-02-05T06:45:06.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:45:06.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:05.378012+0000", 2024-02-05T06:45:06.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.371e-06 2024-02-05T06:45:06.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:45:06.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:05.407522+0000", 2024-02-05T06:45:06.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029509937 2024-02-05T06:45:06.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:45:06.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:05.407610+0000", 2024-02-05T06:45:06.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8407999999999998e-05 2024-02-05T06:45:06.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:45:06.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:05.407701+0000", 2024-02-05T06:45:06.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0616000000000001e-05 2024-02-05T06:45:06.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:45:06.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:05.409266+0000", 2024-02-05T06:45:06.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015653889999999999 2024-02-05T06:45:06.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:05.502298+0000", 2024-02-05T06:45:06.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.093031288000000004 2024-02-05T06:45:06.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:45:06.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:05.519660+0000", 2024-02-05T06:45:06.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017361951 2024-02-05T06:45:06.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:45:06.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:05.519715+0000", 2024-02-05T06:45:06.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5090999999999998e-05 2024-02-05T06:45:06.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:45:06.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:45:06.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:45:06.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:05.519762+0000", 2024-02-05T06:45:06.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7685999999999997e-05 2024-02-05T06:45:06.267 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.267 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:45:06.267 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.267 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:45:06.267 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:45:06.268 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:45:06.291 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:06.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:06.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:06.317 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:45:06.357 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:45:06.394 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:06.401 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:45:06.436 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:06.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:06.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:06.445 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:45:06.484 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:45:06.527 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:45:06.537 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:06.570 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:45:06.580 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:06.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:06.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:06.610 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:45:06.616 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:45:06.616 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:45:06.616 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:45:06.616 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:45:06.616 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:45:06.616 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:45:06.617 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:45:06.617 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:45:06.681 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:06.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:06.723 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:06.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:06.817 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:06 smithi167 ceph-mon[32774]: osdmap e317: 8 total, 8 up, 8 in 2024-02-05T06:45:06.817 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:06 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1910190155' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:45:06.824 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:06.863 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:06.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:06.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:06.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:06 smithi028 ceph-mon[29442]: osdmap e317: 8 total, 8 up, 8 in 2024-02-05T06:45:06.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:06 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1910190155' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:45:06.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:06 smithi028 ceph-mon[37984]: osdmap e317: 8 total, 8 up, 8 in 2024-02-05T06:45:06.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:06 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1910190155' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:45:06.965 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:07.003 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:07.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:07.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:07.105 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:07.136 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 87 16 126792 126776 91.0547 61.375 0.00831939 0.0109763 2024-02-05T06:45:07.142 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:07.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:07.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:07.155 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-05T06:45:07.244 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:07.269 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:07.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:07.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:07.371 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:07.414 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:07.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:07.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:07.516 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:07.559 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:07.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:07.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:07.661 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:07.693 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:07.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:07.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:07.733 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:45:07 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=cleanup t=2024-02-05T06:45:07.413492724Z level=info msg="Completed cleanup jobs" duration=2.453083ms 2024-02-05T06:45:07.733 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:45:07 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=sqlstore.transactions t=2024-02-05T06:45:07.421993462Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-02-05T06:45:07.795 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:07.835 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:07.836 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:07 smithi028 ceph-mon[37984]: pgmap v566: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 48 MiB/s wr, 768 op/s 2024-02-05T06:45:07.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:07.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:07.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:07 smithi028 ceph-mon[29442]: pgmap v566: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 48 MiB/s wr, 768 op/s 2024-02-05T06:45:07.937 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:07.981 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:07.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:07.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:08.083 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:08.135 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:45:08.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:08.138 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:08.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:08.139 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 88 16 127717 127701 90.6741 57.8125 0.0129947 0.0110225 2024-02-05T06:45:08.191 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:45:08.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:07 smithi167 ceph-mon[32774]: pgmap v566: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 48 MiB/s wr, 768 op/s 2024-02-05T06:45:08.239 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:08.270 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:08.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:08.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:08.372 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:08.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:08.404 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:08.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:08.505 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:08.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:08.554 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:08.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:08.655 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:08.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:08.690 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:08.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:08.771 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.4 primary-affinity to 0.0467335 (830622) 2024-02-05T06:45:08.791 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:08.833 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:08.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:08.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:08.935 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:08.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:08 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/4209752883' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.04673354630430937}]: dispatch 2024-02-05T06:45:08.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:08 smithi028 ceph-mon[29442]: from='client.25408 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.04673354630430937}]: dispatch 2024-02-05T06:45:08.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:08 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/4209752883' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.04673354630430937}]: dispatch 2024-02-05T06:45:08.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:08 smithi028 ceph-mon[37984]: from='client.25408 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.04673354630430937}]: dispatch 2024-02-05T06:45:08.972 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:08.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:08.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:09.074 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:09.109 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:09.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:09.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:09.139 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 89 16 128738 128722 90.3723 63.8125 0.00765333 0.0110603 2024-02-05T06:45:09.211 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:09.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:08 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/4209752883' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.04673354630430937}]: dispatch 2024-02-05T06:45:09.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:08 smithi167 ceph-mon[32774]: from='client.25408 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.04673354630430937}]: dispatch 2024-02-05T06:45:09.245 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:09.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:09.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:09.347 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:09.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:09.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:09.492 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:09.535 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:09.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:09.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:09.637 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:09.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:09.669 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:09.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:09.770 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:09.803 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:09.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:09.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:09.871 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:09 smithi028 ceph-mon[37984]: pgmap v567: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-02-05T06:45:09.872 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:09 smithi028 ceph-mon[37984]: from='client.25408 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.04673354630430937}]': finished 2024-02-05T06:45:09.872 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:09 smithi028 ceph-mon[37984]: osdmap e318: 8 total, 8 up, 8 in 2024-02-05T06:45:09.872 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:09 smithi028 ceph-mon[29442]: pgmap v567: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-02-05T06:45:09.872 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:09 smithi028 ceph-mon[29442]: from='client.25408 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.04673354630430937}]': finished 2024-02-05T06:45:09.872 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:09 smithi028 ceph-mon[29442]: osdmap e318: 8 total, 8 up, 8 in 2024-02-05T06:45:09.885 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:45:09.905 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:09.937 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:09.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:09.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:10.039 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:10.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:10.083 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:10.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:10.139 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 90 6 129797 129791 90.1105 66.8125 0.0122833 0.0110904 2024-02-05T06:45:10.185 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:10.210 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:10.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:10.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:10.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:09 smithi167 ceph-mon[32774]: pgmap v567: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-02-05T06:45:10.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:09 smithi167 ceph-mon[32774]: from='client.25408 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.04673354630430937}]': finished 2024-02-05T06:45:10.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:09 smithi167 ceph-mon[32774]: osdmap e318: 8 total, 8 up, 8 in 2024-02-05T06:45:10.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:10.337 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:10.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:10.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:10.439 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:10.463 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:10.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:10.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:10.564 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:10.588 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:10.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:10.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:10.689 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:10.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:10.713 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:10.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:10.814 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:10.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:10.839 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:10.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:10.928 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:10 smithi028 ceph-mon[29442]: osdmap e319: 8 total, 8 up, 8 in 2024-02-05T06:45:10.928 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:10 smithi028 ceph-mon[37984]: osdmap e319: 8 total, 8 up, 8 in 2024-02-05T06:45:10.940 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:10.964 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:10.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:10.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:11.066 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:11.090 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:11.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:11.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:11.139 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Total time run: 90.0438 2024-02-05T06:45:11.140 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Total writes made: 129797 2024-02-05T06:45:11.140 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Write size: 65536 2024-02-05T06:45:11.140 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Object size: 65536 2024-02-05T06:45:11.140 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Bandwidth (MB/sec): 90.0929 2024-02-05T06:45:11.140 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Stddev Bandwidth: 34.7581 2024-02-05T06:45:11.140 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Max bandwidth (MB/sec): 224.562 2024-02-05T06:45:11.140 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Min bandwidth (MB/sec): 44.4375 2024-02-05T06:45:11.140 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Average IOPS: 1441 2024-02-05T06:45:11.140 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Stddev IOPS: 556.129 2024-02-05T06:45:11.140 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Max IOPS: 3593 2024-02-05T06:45:11.140 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Min IOPS: 711 2024-02-05T06:45:11.140 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Average Latency(s): 0.0110933 2024-02-05T06:45:11.140 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Stddev Latency(s): 0.00907894 2024-02-05T06:45:11.141 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Max latency(s): 0.230787 2024-02-05T06:45:11.141 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Min latency(s): 0.0012123 2024-02-05T06:45:11.146 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Cleaning up (deleting benchmark objects) 2024-02-05T06:45:11.192 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:11.223 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:11.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:11.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:11.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:10 smithi167 ceph-mon[32774]: osdmap e319: 8 total, 8 up, 8 in 2024-02-05T06:45:11.324 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:11.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:11.362 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:11.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:11.463 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:11.496 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:11.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:11.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:11.597 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:11.631 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:11.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:11.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:11.733 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:11.767 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:11.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:11.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:11.869 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:11.900 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:11.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:11.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:11.904 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:45:12.002 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:12.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:12.034 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:12.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:12.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:11 smithi028 ceph-mon[29442]: pgmap v570: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 53 MiB/s wr, 847 op/s 2024-02-05T06:45:12.092 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:11 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:45:12.093 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:11 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:45:12.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:11 smithi028 ceph-mon[37984]: pgmap v570: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 53 MiB/s wr, 847 op/s 2024-02-05T06:45:12.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:11 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:45:12.093 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:11 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:45:12.135 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:12.171 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:12.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:12.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:12.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:11 smithi167 ceph-mon[32774]: pgmap v570: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 53 MiB/s wr, 847 op/s 2024-02-05T06:45:12.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:11 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:45:12.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:11 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:45:12.273 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:12.304 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:12.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:12.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:12.406 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:12.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:12.452 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:12.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:12.553 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:12.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:12.586 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:12.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:12.686 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:12.731 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:12.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:12.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:12.789 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:45:12 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:45:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:45:12.819 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-02-05T06:45:12.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:12.875 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:12.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:12.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:12.977 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:13.023 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:13.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:13.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:13.124 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:13.159 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:13.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:13.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:13.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:12 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2259273074' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:45:13.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:12 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2259273074' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:45:13.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:12 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2259273074' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:45:13.234 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:45:13 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:45:13.261 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:13.302 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:13.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:13.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:13.404 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:13.444 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:13.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:13.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:13.530 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:45:13.530 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:45:13.545 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:13.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:13.593 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:13.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:13.694 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:13.731 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:13.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:13.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:13.832 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:13.870 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:13.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:13.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:13.906 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:13 smithi028 ceph-mon[29442]: pgmap v571: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 4.5 KiB/s rd, 79 MiB/s wr, 1.40k op/s; 24 KiB/s, 0 objects/s recovering 2024-02-05T06:45:13.906 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:13 smithi028 ceph-mon[29442]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:45:13.906 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:13 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2259273074' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:45:13.906 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:13 smithi028 ceph-mon[29442]: osdmap e320: 8 total, 8 up, 8 in 2024-02-05T06:45:13.907 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:13 smithi028 ceph-mon[37984]: pgmap v571: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 4.5 KiB/s rd, 79 MiB/s wr, 1.40k op/s; 24 KiB/s, 0 objects/s recovering 2024-02-05T06:45:13.907 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:13 smithi028 ceph-mon[37984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:45:13.907 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:13 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2259273074' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:45:13.907 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:13 smithi028 ceph-mon[37984]: osdmap e320: 8 total, 8 up, 8 in 2024-02-05T06:45:13.972 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:14.018 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:14.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:14.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:14.120 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:14.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:14.153 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:14.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:14.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:13 smithi167 ceph-mon[32774]: pgmap v571: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 4.5 KiB/s rd, 79 MiB/s wr, 1.40k op/s; 24 KiB/s, 0 objects/s recovering 2024-02-05T06:45:14.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:13 smithi167 ceph-mon[32774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:45:14.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:13 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2259273074' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:45:14.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:13 smithi167 ceph-mon[32774]: osdmap e320: 8 total, 8 up, 8 in 2024-02-05T06:45:14.254 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:14.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:14.290 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:14.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:14.391 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:14.429 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:14.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:14.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:14.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:14.573 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:14.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:14.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:14.654 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 4, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T06:45:14.655 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T06:45:14.655 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-02-05T06:45:14.655 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd test-reweight-by-utilization 1000 0.05 2024-02-05T06:45:14.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:14.741 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:14.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:14.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:14.843 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:14.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:14.896 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:14.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:14.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:14 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:45:14.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:14 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:45:14.997 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:15.030 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:15.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:15.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:15.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:15.176 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:15.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:15.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:15.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:14 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:45:15.278 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:15.320 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:15.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:15.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:15.421 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:15.455 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:15.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:15.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:15.557 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:15.591 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:15.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:15.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:15.692 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:15.732 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:15.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:15.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:15.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:15.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:15.899 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:15.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:15.999 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:16.039 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:16.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:16.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:16.141 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:16.177 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:16.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:16.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:16.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:15 smithi028 ceph-mon[29442]: pgmap v573: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 4.2 KiB/s rd, 35 MiB/s wr, 699 op/s; 26 KiB/s, 0 objects/s recovering 2024-02-05T06:45:16.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:15 smithi028 ceph-mon[37984]: pgmap v573: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 4.2 KiB/s rd, 35 MiB/s wr, 699 op/s; 26 KiB/s, 0 objects/s recovering 2024-02-05T06:45:16.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:15 smithi167 ceph-mon[32774]: pgmap v573: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.9 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 4.2 KiB/s rd, 35 MiB/s wr, 699 op/s; 26 KiB/s, 0 objects/s recovering 2024-02-05T06:45:16.278 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:16.315 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:16.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:16.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:16.417 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:16.450 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:16.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:16.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:16.552 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:16.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:16.591 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:16.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:16.692 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:16.728 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:16.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:16.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:16.830 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:16.866 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:16.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:16.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:16.969 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:17.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:17.008 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:17.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:17.035 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-05T06:45:17.110 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:17.171 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:17.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:17.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:17.273 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:17.301 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:17.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:17.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:17.313 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:45:17.356 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:45:17.403 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:45:17.403 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:17.431 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:17.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:17.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:17.442 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:45:17.496 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:45:17.533 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:17.533 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:45:17.573 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:45:17.573 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:17.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:17.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:17.616 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:45:17.655 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:45:17.669 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:45:17.669 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:45:17.669 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:45:17.669 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:45:17.669 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:45:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:45:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:45:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:45:17.676 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:17.711 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:17.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:17.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:17.813 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:17.854 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:17.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:17.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:17.890 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T06:45:17.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:17 smithi028 ceph-mon[29442]: pgmap v574: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 3.3 KiB/s rd, 27 MiB/s wr, 752 op/s; 21 KiB/s, 0 objects/s recovering 2024-02-05T06:45:17.932 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:17 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2314351336' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:45:17.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:17 smithi028 ceph-mon[37984]: pgmap v574: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 3.3 KiB/s rd, 27 MiB/s wr, 752 op/s; 21 KiB/s, 0 objects/s recovering 2024-02-05T06:45:17.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:17 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2314351336' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:45:17.956 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:18.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:18.004 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:18.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:18.073 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:17 smithi167 ceph-mon[32774]: pgmap v574: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 7.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 3.3 KiB/s rd, 27 MiB/s wr, 752 op/s; 21 KiB/s, 0 objects/s recovering 2024-02-05T06:45:18.073 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:17 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2314351336' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:45:18.105 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:18.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:18.142 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:18.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:18.243 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:18.312 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:18.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:18.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:18.414 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:18.452 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:18.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:18.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:18.555 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:18.587 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:18.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:18.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:18.689 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:18.727 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:18.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:18.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:18.829 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:18.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:18.862 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:18.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:18.963 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:19.000 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:19.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:19.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:19.102 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:19.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:19.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:19.244 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:19.278 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:19.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:19.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:19.380 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:19.414 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:19.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:19.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:19.516 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:19.550 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:19.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:19.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:19.651 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:19.688 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:19.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:19.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:19.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:19.844 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:19.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:19.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:19.946 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:19.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:19 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:45:19.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:19 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:45:19.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:19 smithi028 ceph-mon[29442]: pgmap v575: 158 pgs: 158 active+clean; 7.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 3.5 KiB/s rd, 24 MiB/s wr, 1.22k op/s; 18 KiB/s, 0 objects/s recovering 2024-02-05T06:45:19.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:19 smithi028 ceph-mon[29442]: osdmap e321: 8 total, 8 up, 8 in 2024-02-05T06:45:19.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:19 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:45:19.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:19 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:45:19.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:19 smithi028 ceph-mon[37984]: pgmap v575: 158 pgs: 158 active+clean; 7.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 3.5 KiB/s rd, 24 MiB/s wr, 1.22k op/s; 18 KiB/s, 0 objects/s recovering 2024-02-05T06:45:19.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:19 smithi028 ceph-mon[37984]: osdmap e321: 8 total, 8 up, 8 in 2024-02-05T06:45:19.979 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:19.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:19.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:20.081 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:20.121 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:20.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:20.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:20.223 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:20.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:19 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:45:20.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:19 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:45:20.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:19 smithi167 ceph-mon[32774]: pgmap v575: 158 pgs: 158 active+clean; 7.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 3.5 KiB/s rd, 24 MiB/s wr, 1.22k op/s; 18 KiB/s, 0 objects/s recovering 2024-02-05T06:45:20.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:19 smithi167 ceph-mon[32774]: osdmap e321: 8 total, 8 up, 8 in 2024-02-05T06:45:20.264 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:20.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:20.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:20.365 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:45:20.367 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:20.420 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:20.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:20.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:20.522 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:20.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:20.558 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:20.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:20.659 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:20.695 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:20.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:20.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:20.797 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:20.830 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:20.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:20.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:20.932 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:20.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:20.967 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:20.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:21.068 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:21.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:21.104 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:21.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:21.205 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:21.240 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:21.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:21.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:21.342 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:21.382 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:21.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:21.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:21.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:21 smithi028 ceph-mon[37984]: pgmap v577: 158 pgs: 158 active+clean; 7.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 785 op/s 2024-02-05T06:45:21.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:21 smithi028 ceph-mon[29442]: pgmap v577: 158 pgs: 158 active+clean; 7.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 785 op/s 2024-02-05T06:45:21.483 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:21 smithi167 ceph-mon[32774]: pgmap v577: 158 pgs: 158 active+clean; 7.5 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 785 op/s 2024-02-05T06:45:21.484 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:21.521 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:21.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:21.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:21.623 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:21.657 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:21.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:21.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:21.759 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:21.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:21.793 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:21.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:21.894 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:21.931 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:21.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:21.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:22.022 INFO:teuthology.orchestra.run.smithi028.stdout:moved 0 / 474 (0%) 2024-02-05T06:45:22.022 INFO:teuthology.orchestra.run.smithi028.stdout:avg 59.25 2024-02-05T06:45:22.022 INFO:teuthology.orchestra.run.smithi028.stdout:stddev 6.75925 -> 6.75925 (expected baseline 7.20026) 2024-02-05T06:45:22.022 INFO:teuthology.orchestra.run.smithi028.stdout:min osd.2 with 48 -> 48 pgs (0.810127 -> 0.810127 * mean) 2024-02-05T06:45:22.022 INFO:teuthology.orchestra.run.smithi028.stdout:max osd.3 with 68 -> 68 pgs (1.14768 -> 1.14768 * mean) 2024-02-05T06:45:22.022 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:45:22.022 INFO:teuthology.orchestra.run.smithi028.stdout:oload 1000 2024-02-05T06:45:22.022 INFO:teuthology.orchestra.run.smithi028.stdout:max_change 0.05 2024-02-05T06:45:22.022 INFO:teuthology.orchestra.run.smithi028.stdout:max_change_osds 4 2024-02-05T06:45:22.022 INFO:teuthology.orchestra.run.smithi028.stdout:average_utilization 0.0358 2024-02-05T06:45:22.023 INFO:teuthology.orchestra.run.smithi028.stdout:overload_utilization 0.3584 2024-02-05T06:45:22.028 INFO:teuthology.orchestra.run.smithi028.stderr:no change 2024-02-05T06:45:22.032 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:22.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:22.061 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:22.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:22.162 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:22.200 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:22.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:22.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:22.282 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:22 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:45:22.283 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:22 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:45:22.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:22.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:22.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:22.356 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:45:22.447 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:22.486 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:22.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:22.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:22.587 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:22.628 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:22.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:22.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:22.699 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:45:22 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:45:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:45:22.730 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:22.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:22 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:45:22.764 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:22.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:22.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:22.866 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:22.911 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:45:22.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:22.914 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:22.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:22.954 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-02-05T06:45:22.954 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd test-reweight-by-utilization 1000 1.0 2024-02-05T06:45:23.015 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:23.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:23.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:23.181 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:23.232 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:23.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:23.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:23.334 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:23.369 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:23.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:23.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:23.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:23.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:23 smithi167 ceph-mon[32774]: from='client.15519 v1:172.21.15.28:0/1253410221' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:45:23.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:23 smithi167 ceph-mon[32774]: pgmap v578: 158 pgs: 158 active+clean; 7.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-02-05T06:45:23.484 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:45:23 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:45:23.495 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:23.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:23.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:23.597 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:23.632 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:23.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:23.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:23.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:23 smithi028 ceph-mon[29442]: from='client.15519 v1:172.21.15.28:0/1253410221' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:45:23.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:23 smithi028 ceph-mon[29442]: pgmap v578: 158 pgs: 158 active+clean; 7.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-02-05T06:45:23.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:23 smithi028 ceph-mon[37984]: from='client.15519 v1:172.21.15.28:0/1253410221' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:45:23.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:23 smithi028 ceph-mon[37984]: pgmap v578: 158 pgs: 158 active+clean; 7.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-02-05T06:45:23.734 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:23.771 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:23.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:23.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:23.873 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:23.908 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:23.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:23.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:24.010 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:24.048 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:24.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:24.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:24.150 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:24.184 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:24.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:24.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:24.329 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:24.369 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:24.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:24.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:24.470 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:24.505 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:24.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:24.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:24.607 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:24.647 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:24.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:24.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:24.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:24 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:45:24.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:24 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:45:24.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:24 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:45:24.750 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:24.790 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:24.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:24.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:24.892 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:24.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:24.923 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:24.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:25.024 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:25.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:25.078 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:25.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:25.180 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:25.231 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:25.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:25.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:25.333 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:25.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:25.379 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:25.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:25.480 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:25.520 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:25.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:25.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:25.621 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:25.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:25.663 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:25.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:25.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:25 smithi028 ceph-mon[37984]: pgmap v579: 158 pgs: 158 active+clean; 7.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s 2024-02-05T06:45:25.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:25 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:45:25.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:25 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:45:25.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:25 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:45:25.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:25 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:45:25.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:25 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:45:25.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:25 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:45:25.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:25 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:45:25.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:25 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:45:25.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:25 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:45:25.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:25 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:45:25.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:25 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:45:25.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:25 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:45:25.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:25 smithi028 ceph-mon[29442]: pgmap v579: 158 pgs: 158 active+clean; 7.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s 2024-02-05T06:45:25.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:25 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:45:25.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:25 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:45:25.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:25 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:45:25.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:25 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:45:25.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:25 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:45:25.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:25 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:45:25.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:25 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:45:25.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:25 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:45:25.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:25 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:45:25.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:25 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:45:25.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:25 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:45:25.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:25 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:45:25.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:25 smithi167 ceph-mon[32774]: pgmap v579: 158 pgs: 158 active+clean; 7.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s 2024-02-05T06:45:25.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:25 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:45:25.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:25 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:45:25.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:25 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:45:25.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:25 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:45:25.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:25 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:45:25.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:25 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:45:25.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:25 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:45:25.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:25 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-05T06:45:25.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:25 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:45:25.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:25 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:45:25.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:25 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:45:25.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:25 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:45:25.764 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:25.801 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:25.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:25.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:25.903 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:25.942 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:25.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:25.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:26.044 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:26.077 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:26.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:26.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:26.178 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:26.215 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:26.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:26.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:26.317 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:26.351 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:26.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:26.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:26.453 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:26.487 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:26.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:26.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:26.589 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:26.595 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:26 smithi028 ceph-mon[29442]: Adjusting osd_memory_target on smithi028 to 3249M 2024-02-05T06:45:26.595 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:26 smithi028 ceph-mon[37984]: Adjusting osd_memory_target on smithi028 to 3249M 2024-02-05T06:45:26.621 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:26.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:26.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:26.723 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:26.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:26 smithi167 ceph-mon[32774]: Adjusting osd_memory_target on smithi028 to 3249M 2024-02-05T06:45:26.758 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:26.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:26.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:26.860 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:26.884 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:26.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:26.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:26.985 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:27.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:27.020 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:27.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:27.121 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:27.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:27.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:27.261 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:27.296 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:27.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:27.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:27.398 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:27.426 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:27.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:27.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:27.528 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:27.552 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:27 smithi028 ceph-mon[37984]: pgmap v580: 158 pgs: 158 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s 2024-02-05T06:45:27.552 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:27 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:45:27.552 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:27 smithi028 ceph-mon[29442]: pgmap v580: 158 pgs: 158 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s 2024-02-05T06:45:27.552 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:27 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:45:27.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:27.559 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:27.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:27.660 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:27.695 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:27.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:27.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:27.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:27 smithi167 ceph-mon[32774]: pgmap v580: 158 pgs: 158 active+clean; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s 2024-02-05T06:45:27.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:27 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:45:27.797 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:27.839 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:27.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:27.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:27.941 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:27.982 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:27.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:27.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:28.084 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:28.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:28.123 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:28.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:28.224 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:28.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:28.267 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:28.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:28.368 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:28.383 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:45:28.383 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:45:28.439 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:28.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:28.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:28.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:28 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:45:28.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:28 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:45:28.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:28 smithi028 ceph-mon[37984]: osdmap e322: 8 total, 8 up, 8 in 2024-02-05T06:45:28.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:28 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:45:28.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:28 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:45:28.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:28 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:45:28.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:28 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:45:28.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:28 smithi028 ceph-mon[29442]: osdmap e322: 8 total, 8 up, 8 in 2024-02-05T06:45:28.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:28 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:45:28.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:28 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:45:28.492 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:45:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-02-05T06:45:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-02-05T06:45:28.492 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-02-05T06:45:28.493 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:45:28.541 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:28.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:28.579 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:28.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:28.680 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:28.709 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:28.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:28.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:28.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:28 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:45:28.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:28 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:45:28.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:28 smithi167 ceph-mon[32774]: osdmap e322: 8 total, 8 up, 8 in 2024-02-05T06:45:28.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:28 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:45:28.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:28 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:45:28.811 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:28.851 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:28.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:28.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:28.953 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:28.989 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:28.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:28.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:29.091 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:29.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:29.130 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:29.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:29.230 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:29.265 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:29.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:29.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:29.334 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-02-05T06:45:29.367 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:29.401 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:29.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:29.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:29.503 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:29.545 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:29.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:29.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:29.647 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:29.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:29 smithi028 ceph-mon[29442]: osdmap e323: 8 total, 8 up, 8 in 2024-02-05T06:45:29.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:29 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3393171258' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:45:29.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:29 smithi028 ceph-mon[29442]: pgmap v583: 158 pgs: 158 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s 2024-02-05T06:45:29.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:29 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:45:29.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:29 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:45:29.661 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:29 smithi028 ceph-mon[37984]: osdmap e323: 8 total, 8 up, 8 in 2024-02-05T06:45:29.661 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:29 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3393171258' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:45:29.661 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:29 smithi028 ceph-mon[37984]: pgmap v583: 158 pgs: 158 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s 2024-02-05T06:45:29.661 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:29 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:45:29.661 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:29 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:45:29.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:29.679 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:29.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:29.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:29 smithi167 ceph-mon[32774]: osdmap e323: 8 total, 8 up, 8 in 2024-02-05T06:45:29.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:29 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3393171258' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:45:29.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:29 smithi167 ceph-mon[32774]: pgmap v583: 158 pgs: 158 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.60k op/s 2024-02-05T06:45:29.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:29 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:45:29.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:29 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:45:29.781 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:29.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:29.817 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:29.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:29.851 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-05T06:45:29.919 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:29.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:29.977 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:29.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:30.078 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:30.112 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:30.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:30.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:30.214 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:30.256 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:30.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:30.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:30.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:30.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:30.397 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:30.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:30.498 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:30.534 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:30.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:30.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:30.590 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:30 smithi028 ceph-mon[37984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:45:30.590 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:30 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3393171258' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:45:30.590 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:30 smithi028 ceph-mon[37984]: osdmap e324: 8 total, 8 up, 8 in 2024-02-05T06:45:30.590 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:30 smithi028 ceph-mon[37984]: 4.13 scrub starts 2024-02-05T06:45:30.590 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:30 smithi028 ceph-mon[37984]: 4.13 scrub ok 2024-02-05T06:45:30.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:30 smithi028 ceph-mon[29442]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:45:30.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:30 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3393171258' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:45:30.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:30 smithi028 ceph-mon[29442]: osdmap e324: 8 total, 8 up, 8 in 2024-02-05T06:45:30.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:30 smithi028 ceph-mon[29442]: 4.13 scrub starts 2024-02-05T06:45:30.590 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:30 smithi028 ceph-mon[29442]: 4.13 scrub ok 2024-02-05T06:45:30.636 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:30.672 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:30.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:30.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:30.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:30 smithi167 ceph-mon[32774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:45:30.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:30 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3393171258' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:45:30.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:30 smithi167 ceph-mon[32774]: osdmap e324: 8 total, 8 up, 8 in 2024-02-05T06:45:30.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:30 smithi167 ceph-mon[32774]: 4.13 scrub starts 2024-02-05T06:45:30.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:30 smithi167 ceph-mon[32774]: 4.13 scrub ok 2024-02-05T06:45:30.773 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:30.817 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:30.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:30.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:30.919 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:30.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:30.956 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:30.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:31.057 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:31.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:31.092 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:31.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:31.193 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:31.231 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:31.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:31.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:31.334 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:31.374 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:31.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:31.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:31.476 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:31.513 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:31.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:31.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:31.538 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:31 smithi028 ceph-mon[37984]: 9.3 scrub starts 2024-02-05T06:45:31.538 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:31 smithi028 ceph-mon[37984]: pgmap v585: 158 pgs: 158 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-05T06:45:31.538 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:31 smithi028 ceph-mon[37984]: 3.19 scrub starts 2024-02-05T06:45:31.538 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:31 smithi028 ceph-mon[37984]: 3.19 scrub ok 2024-02-05T06:45:31.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:31 smithi028 ceph-mon[29442]: 9.3 scrub starts 2024-02-05T06:45:31.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:31 smithi028 ceph-mon[29442]: pgmap v585: 158 pgs: 158 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-05T06:45:31.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:31 smithi028 ceph-mon[29442]: 3.19 scrub starts 2024-02-05T06:45:31.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:31 smithi028 ceph-mon[29442]: 3.19 scrub ok 2024-02-05T06:45:31.615 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:31.644 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:31.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:31.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:31.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:31 smithi167 ceph-mon[32774]: 9.3 scrub starts 2024-02-05T06:45:31.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:31 smithi167 ceph-mon[32774]: pgmap v585: 158 pgs: 158 active+clean; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-05T06:45:31.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:31 smithi167 ceph-mon[32774]: 3.19 scrub starts 2024-02-05T06:45:31.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:31 smithi167 ceph-mon[32774]: 3.19 scrub ok 2024-02-05T06:45:31.746 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:31.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:31.778 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:31.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:31.879 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:31.915 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:31.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:31.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:32.017 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:32.018 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:45:32.054 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:32.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:32.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:32.156 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:32.197 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:32.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:32.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:32.299 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:32.323 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T06:45:32.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:32.347 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:32.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:32.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:32 smithi028 ceph-mon[29442]: 3.b scrub starts 2024-02-05T06:45:32.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:32 smithi028 ceph-mon[29442]: 3.b scrub ok 2024-02-05T06:45:32.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:32 smithi028 ceph-mon[29442]: 5.1a scrub starts 2024-02-05T06:45:32.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:32 smithi028 ceph-mon[29442]: 5.1a scrub ok 2024-02-05T06:45:32.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:32 smithi028 ceph-mon[29442]: 6.f scrub starts 2024-02-05T06:45:32.382 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:32 smithi028 ceph-mon[29442]: 6.f scrub ok 2024-02-05T06:45:32.382 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:32 smithi028 ceph-mon[37984]: 3.b scrub starts 2024-02-05T06:45:32.382 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:32 smithi028 ceph-mon[37984]: 3.b scrub ok 2024-02-05T06:45:32.382 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:32 smithi028 ceph-mon[37984]: 5.1a scrub starts 2024-02-05T06:45:32.382 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:32 smithi028 ceph-mon[37984]: 5.1a scrub ok 2024-02-05T06:45:32.382 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:32 smithi028 ceph-mon[37984]: 6.f scrub starts 2024-02-05T06:45:32.383 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:32 smithi028 ceph-mon[37984]: 6.f scrub ok 2024-02-05T06:45:32.448 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:32.487 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:32.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:32.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:32.588 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:32.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:32.633 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:32.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:32.699 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:45:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:45:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:45:32.733 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:32.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:32 smithi167 ceph-mon[32774]: 3.b scrub starts 2024-02-05T06:45:32.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:32 smithi167 ceph-mon[32774]: 3.b scrub ok 2024-02-05T06:45:32.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:32 smithi167 ceph-mon[32774]: 5.1a scrub starts 2024-02-05T06:45:32.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:32 smithi167 ceph-mon[32774]: 5.1a scrub ok 2024-02-05T06:45:32.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:32 smithi167 ceph-mon[32774]: 6.f scrub starts 2024-02-05T06:45:32.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:32 smithi167 ceph-mon[32774]: 6.f scrub ok 2024-02-05T06:45:32.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:32.793 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:32.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:32.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:32.942 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:32.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:32.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:33.044 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:33.078 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:33.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:33.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:33.180 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:33.216 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:33.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:33.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:33.318 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:33.340 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:45:33 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:45:33.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:33.368 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:33.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:33.470 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:33.504 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:33.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:33.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:33.577 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:33 smithi028 ceph-mon[37984]: 4.14 scrub starts 2024-02-05T06:45:33.578 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:33 smithi028 ceph-mon[37984]: 4.14 scrub ok 2024-02-05T06:45:33.578 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:33 smithi028 ceph-mon[37984]: 9.3 scrub ok 2024-02-05T06:45:33.578 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:33 smithi028 ceph-mon[37984]: 3.14 scrub starts 2024-02-05T06:45:33.578 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:33 smithi028 ceph-mon[37984]: 3.14 scrub ok 2024-02-05T06:45:33.578 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:33 smithi028 ceph-mon[37984]: pgmap v586: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.86k op/s 2024-02-05T06:45:33.578 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:33 smithi028 ceph-mon[37984]: 1.5 scrub starts 2024-02-05T06:45:33.578 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:33 smithi028 ceph-mon[37984]: 1.5 scrub ok 2024-02-05T06:45:33.578 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:33 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2404713374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:45:33.578 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:33 smithi028 ceph-mon[29442]: 4.14 scrub starts 2024-02-05T06:45:33.578 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:33 smithi028 ceph-mon[29442]: 4.14 scrub ok 2024-02-05T06:45:33.578 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:33 smithi028 ceph-mon[29442]: 9.3 scrub ok 2024-02-05T06:45:33.579 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:33 smithi028 ceph-mon[29442]: 3.14 scrub starts 2024-02-05T06:45:33.579 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:33 smithi028 ceph-mon[29442]: 3.14 scrub ok 2024-02-05T06:45:33.579 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:33 smithi028 ceph-mon[29442]: pgmap v586: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.86k op/s 2024-02-05T06:45:33.579 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:33 smithi028 ceph-mon[29442]: 1.5 scrub starts 2024-02-05T06:45:33.579 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:33 smithi028 ceph-mon[29442]: 1.5 scrub ok 2024-02-05T06:45:33.579 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:33 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2404713374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:45:33.605 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:33.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:33.649 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:33.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:33.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:33 smithi167 ceph-mon[32774]: 4.14 scrub starts 2024-02-05T06:45:33.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:33 smithi167 ceph-mon[32774]: 4.14 scrub ok 2024-02-05T06:45:33.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:33 smithi167 ceph-mon[32774]: 9.3 scrub ok 2024-02-05T06:45:33.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:33 smithi167 ceph-mon[32774]: 3.14 scrub starts 2024-02-05T06:45:33.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:33 smithi167 ceph-mon[32774]: 3.14 scrub ok 2024-02-05T06:45:33.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:33 smithi167 ceph-mon[32774]: pgmap v586: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.86k op/s 2024-02-05T06:45:33.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:33 smithi167 ceph-mon[32774]: 1.5 scrub starts 2024-02-05T06:45:33.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:33 smithi167 ceph-mon[32774]: 1.5 scrub ok 2024-02-05T06:45:33.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:33 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2404713374' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:45:33.751 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:33.770 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:45:33.770 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:45:33.770 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:45:33.770 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:45:33.770 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:45:33.770 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:45:33.770 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:45:33.770 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:45:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:45:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:45:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:45:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:45:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:45:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:45:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:45:33.771 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:45:33.790 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:33.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:33.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:33.892 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:33.928 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:33.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:33.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:34.030 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:34.062 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:34.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:34.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:34.163 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:34.199 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:34.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:34.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:34.300 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:34.336 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:34.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:34.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:34.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:34.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:34.482 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:34.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:34.582 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:34.596 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:34 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:45:34.596 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:34 smithi028 ceph-mon[37984]: 6.1a scrub starts 2024-02-05T06:45:34.596 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:34 smithi028 ceph-mon[37984]: 6.1a scrub ok 2024-02-05T06:45:34.596 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:34 smithi028 ceph-mon[37984]: 9.d scrub starts 2024-02-05T06:45:34.596 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:34 smithi028 ceph-mon[37984]: 4.1c scrub starts 2024-02-05T06:45:34.596 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:34 smithi028 ceph-mon[37984]: 4.1c scrub ok 2024-02-05T06:45:34.596 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:34 smithi028 ceph-mon[37984]: 5.11 scrub starts 2024-02-05T06:45:34.596 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:34 smithi028 ceph-mon[37984]: 5.11 scrub ok 2024-02-05T06:45:34.596 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:34 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:45:34.597 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:34 smithi028 ceph-mon[29442]: 6.1a scrub starts 2024-02-05T06:45:34.597 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:34 smithi028 ceph-mon[29442]: 6.1a scrub ok 2024-02-05T06:45:34.597 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:34 smithi028 ceph-mon[29442]: 9.d scrub starts 2024-02-05T06:45:34.597 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:34 smithi028 ceph-mon[29442]: 4.1c scrub starts 2024-02-05T06:45:34.597 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:34 smithi028 ceph-mon[29442]: 4.1c scrub ok 2024-02-05T06:45:34.597 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:34 smithi028 ceph-mon[29442]: 5.11 scrub starts 2024-02-05T06:45:34.597 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:34 smithi028 ceph-mon[29442]: 5.11 scrub ok 2024-02-05T06:45:34.624 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:34.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:34.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:34.726 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:34.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:34 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:45:34.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:34 smithi167 ceph-mon[32774]: 6.1a scrub starts 2024-02-05T06:45:34.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:34 smithi167 ceph-mon[32774]: 6.1a scrub ok 2024-02-05T06:45:34.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:34 smithi167 ceph-mon[32774]: 9.d scrub starts 2024-02-05T06:45:34.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:34 smithi167 ceph-mon[32774]: 4.1c scrub starts 2024-02-05T06:45:34.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:34 smithi167 ceph-mon[32774]: 4.1c scrub ok 2024-02-05T06:45:34.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:34 smithi167 ceph-mon[32774]: 5.11 scrub starts 2024-02-05T06:45:34.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:34 smithi167 ceph-mon[32774]: 5.11 scrub ok 2024-02-05T06:45:34.766 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:34.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:34.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:34.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:34.906 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:34.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:34.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:35.008 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:35.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:35.043 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:35.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:35.144 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:35.179 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:35.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:35.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:35.281 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:35.317 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:35.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:35.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:35.419 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:35.454 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:35.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:35.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:35.556 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:35.593 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:35.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:35.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:35.634 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:35 smithi167 ceph-mon[32774]: pgmap v587: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 719 B/s rd, 0 B/s wr, 1.29k op/s 2024-02-05T06:45:35.634 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:35 smithi167 ceph-mon[32774]: 5.b scrub starts 2024-02-05T06:45:35.634 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:35 smithi167 ceph-mon[32774]: 5.b scrub ok 2024-02-05T06:45:35.634 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:35 smithi167 ceph-mon[32774]: 6.e scrub starts 2024-02-05T06:45:35.634 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:35 smithi167 ceph-mon[32774]: 6.e scrub ok 2024-02-05T06:45:35.695 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:35.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:35 smithi028 ceph-mon[37984]: pgmap v587: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 719 B/s rd, 0 B/s wr, 1.29k op/s 2024-02-05T06:45:35.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:35 smithi028 ceph-mon[37984]: 5.b scrub starts 2024-02-05T06:45:35.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:35 smithi028 ceph-mon[37984]: 5.b scrub ok 2024-02-05T06:45:35.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:35 smithi028 ceph-mon[37984]: 6.e scrub starts 2024-02-05T06:45:35.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:35 smithi028 ceph-mon[37984]: 6.e scrub ok 2024-02-05T06:45:35.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:35 smithi028 ceph-mon[29442]: pgmap v587: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 719 B/s rd, 0 B/s wr, 1.29k op/s 2024-02-05T06:45:35.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:35 smithi028 ceph-mon[29442]: 5.b scrub starts 2024-02-05T06:45:35.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:35 smithi028 ceph-mon[29442]: 5.b scrub ok 2024-02-05T06:45:35.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:35 smithi028 ceph-mon[29442]: 6.e scrub starts 2024-02-05T06:45:35.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:35 smithi028 ceph-mon[29442]: 6.e scrub ok 2024-02-05T06:45:35.735 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:35.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:35.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:35.837 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:35.874 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:35.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:35.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:35.976 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:36.012 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:36.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:36.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:36.114 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:36.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:36.150 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:36.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:36.251 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:36.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:36.275 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:36.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:36.376 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:36.411 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:36.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:36.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:36.513 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:36.548 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:36.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:36.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:36.650 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:36.693 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:36.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:36.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:36.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:36 smithi028 ceph-mon[29442]: 3.3 scrub starts 2024-02-05T06:45:36.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:36 smithi028 ceph-mon[29442]: 3.3 scrub ok 2024-02-05T06:45:36.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:36 smithi028 ceph-mon[29442]: 6.11 scrub starts 2024-02-05T06:45:36.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:36 smithi028 ceph-mon[29442]: 6.11 scrub ok 2024-02-05T06:45:36.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:36 smithi028 ceph-mon[29442]: 9.d scrub ok 2024-02-05T06:45:36.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:36 smithi028 ceph-mon[29442]: 3.1a scrub starts 2024-02-05T06:45:36.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:36 smithi028 ceph-mon[29442]: 3.1a scrub ok 2024-02-05T06:45:36.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:36 smithi028 ceph-mon[37984]: 3.3 scrub starts 2024-02-05T06:45:36.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:36 smithi028 ceph-mon[37984]: 3.3 scrub ok 2024-02-05T06:45:36.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:36 smithi028 ceph-mon[37984]: 6.11 scrub starts 2024-02-05T06:45:36.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:36 smithi028 ceph-mon[37984]: 6.11 scrub ok 2024-02-05T06:45:36.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:36 smithi028 ceph-mon[37984]: 9.d scrub ok 2024-02-05T06:45:36.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:36 smithi028 ceph-mon[37984]: 3.1a scrub starts 2024-02-05T06:45:36.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:36 smithi028 ceph-mon[37984]: 3.1a scrub ok 2024-02-05T06:45:36.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:36 smithi167 ceph-mon[32774]: 3.3 scrub starts 2024-02-05T06:45:36.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:36 smithi167 ceph-mon[32774]: 3.3 scrub ok 2024-02-05T06:45:36.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:36 smithi167 ceph-mon[32774]: 6.11 scrub starts 2024-02-05T06:45:36.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:36 smithi167 ceph-mon[32774]: 6.11 scrub ok 2024-02-05T06:45:36.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:36 smithi167 ceph-mon[32774]: 9.d scrub ok 2024-02-05T06:45:36.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:36 smithi167 ceph-mon[32774]: 3.1a scrub starts 2024-02-05T06:45:36.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:36 smithi167 ceph-mon[32774]: 3.1a scrub ok 2024-02-05T06:45:36.794 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:36.826 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:36.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:36.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:36.928 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:36.960 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:36.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:36.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:37.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:37.100 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:37.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:37.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:37.202 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:37.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:37.238 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:37.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:37.339 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:37.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:37.376 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:37.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:37.451 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:45:37.477 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:37.538 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:37.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:37.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:37.600 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:37 smithi028 ceph-mon[37984]: pgmap v588: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 632 B/s rd, 0 B/s wr, 991 op/s 2024-02-05T06:45:37.600 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:37 smithi028 ceph-mon[37984]: 4.5 scrub starts 2024-02-05T06:45:37.600 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:37 smithi028 ceph-mon[37984]: 4.5 scrub ok 2024-02-05T06:45:37.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:37 smithi028 ceph-mon[29442]: pgmap v588: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 632 B/s rd, 0 B/s wr, 991 op/s 2024-02-05T06:45:37.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:37 smithi028 ceph-mon[29442]: 4.5 scrub starts 2024-02-05T06:45:37.600 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:37 smithi028 ceph-mon[29442]: 4.5 scrub ok 2024-02-05T06:45:37.640 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:37.673 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:37.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:37.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:37.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:37 smithi167 ceph-mon[32774]: pgmap v588: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 632 B/s rd, 0 B/s wr, 991 op/s 2024-02-05T06:45:37.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:37 smithi167 ceph-mon[32774]: 4.5 scrub starts 2024-02-05T06:45:37.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:37 smithi167 ceph-mon[32774]: 4.5 scrub ok 2024-02-05T06:45:37.775 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:37.803 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:37.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:37.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:37.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:37.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:37.941 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:37.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:38.043 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:38.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:38.082 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:38.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:38.183 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:38.224 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:38.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:38.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:38.326 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:38.359 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:38.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:38.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:38.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:38 smithi028 ceph-mon[29442]: 9.5 scrub starts 2024-02-05T06:45:38.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:38 smithi028 ceph-mon[37984]: 9.5 scrub starts 2024-02-05T06:45:38.461 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:38.497 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:38.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:38.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:38.599 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:38.639 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:38.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:38.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:38.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:38 smithi167 ceph-mon[32774]: 9.5 scrub starts 2024-02-05T06:45:38.740 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:38.773 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:38.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:38.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:38.875 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:38.913 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:38.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:38.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:39.015 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:39.050 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:39.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:39.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:39.151 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:39.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:39.185 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:39.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:39.286 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:39.328 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:39.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:39.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:39.430 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:39.459 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:39 smithi028 ceph-mon[29442]: 5.e scrub starts 2024-02-05T06:45:39.459 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:39 smithi028 ceph-mon[29442]: 5.e scrub ok 2024-02-05T06:45:39.459 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:39 smithi028 ceph-mon[29442]: pgmap v589: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.26k op/s 2024-02-05T06:45:39.459 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:39 smithi028 ceph-mon[29442]: 4.e scrub starts 2024-02-05T06:45:39.459 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:39 smithi028 ceph-mon[29442]: 4.e scrub ok 2024-02-05T06:45:39.459 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:39 smithi028 ceph-mon[29442]: 9.0 scrub starts 2024-02-05T06:45:39.459 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:39 smithi028 ceph-mon[37984]: 5.e scrub starts 2024-02-05T06:45:39.460 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:39 smithi028 ceph-mon[37984]: 5.e scrub ok 2024-02-05T06:45:39.460 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:39 smithi028 ceph-mon[37984]: pgmap v589: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.26k op/s 2024-02-05T06:45:39.460 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:39 smithi028 ceph-mon[37984]: 4.e scrub starts 2024-02-05T06:45:39.460 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:39 smithi028 ceph-mon[37984]: 4.e scrub ok 2024-02-05T06:45:39.460 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:39 smithi028 ceph-mon[37984]: 9.0 scrub starts 2024-02-05T06:45:39.465 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:39.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:39.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:39.506 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:45:39.567 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:39.601 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:39.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:39.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:39.703 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:39.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:39 smithi167 ceph-mon[32774]: 5.e scrub starts 2024-02-05T06:45:39.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:39 smithi167 ceph-mon[32774]: 5.e scrub ok 2024-02-05T06:45:39.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:39 smithi167 ceph-mon[32774]: pgmap v589: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.26k op/s 2024-02-05T06:45:39.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:39 smithi167 ceph-mon[32774]: 4.e scrub starts 2024-02-05T06:45:39.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:39 smithi167 ceph-mon[32774]: 4.e scrub ok 2024-02-05T06:45:39.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:39 smithi167 ceph-mon[32774]: 9.0 scrub starts 2024-02-05T06:45:39.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:39.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:39.854 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:39.884 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:39.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:39.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:39.986 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:40.016 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:40.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:40.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:40.058 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:45:40.058 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:45:40.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:40.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:40.197 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:40.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:40.299 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:40.329 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:40.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:40.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:40.431 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:40.469 INFO:teuthology.orchestra.run.smithi028.stdout:moved 0 / 474 (0%) 2024-02-05T06:45:40.469 INFO:teuthology.orchestra.run.smithi028.stdout:avg 59.25 2024-02-05T06:45:40.470 INFO:teuthology.orchestra.run.smithi028.stdout:stddev 6.75925 -> 6.75925 (expected baseline 7.20026) 2024-02-05T06:45:40.470 INFO:teuthology.orchestra.run.smithi028.stdout:min osd.2 with 48 -> 48 pgs (0.810127 -> 0.810127 * mean) 2024-02-05T06:45:40.470 INFO:teuthology.orchestra.run.smithi028.stdout:max osd.3 with 68 -> 68 pgs (1.14768 -> 1.14768 * mean) 2024-02-05T06:45:40.470 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:45:40.470 INFO:teuthology.orchestra.run.smithi028.stdout:oload 1000 2024-02-05T06:45:40.470 INFO:teuthology.orchestra.run.smithi028.stdout:max_change 1 2024-02-05T06:45:40.470 INFO:teuthology.orchestra.run.smithi028.stdout:max_change_osds 4 2024-02-05T06:45:40.470 INFO:teuthology.orchestra.run.smithi028.stdout:average_utilization 0.0296 2024-02-05T06:45:40.470 INFO:teuthology.orchestra.run.smithi028.stdout:overload_utilization 0.2958 2024-02-05T06:45:40.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:40.471 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:40.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:40.477 INFO:teuthology.orchestra.run.smithi028.stderr:no change 2024-02-05T06:45:40.572 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:40.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:40.607 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:40.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:40.675 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:40 smithi028 ceph-mon[37984]: 6.16 scrub starts 2024-02-05T06:45:40.676 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:40 smithi028 ceph-mon[37984]: 9.5 scrub ok 2024-02-05T06:45:40.676 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:40 smithi028 ceph-mon[37984]: 6.16 scrub ok 2024-02-05T06:45:40.676 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:40 smithi028 ceph-mon[37984]: 4.1 scrub starts 2024-02-05T06:45:40.676 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:40 smithi028 ceph-mon[37984]: 4.1 scrub ok 2024-02-05T06:45:40.676 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:40 smithi028 ceph-mon[37984]: 2.0 scrub starts 2024-02-05T06:45:40.676 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:40 smithi028 ceph-mon[37984]: 2.0 scrub ok 2024-02-05T06:45:40.676 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:40 smithi028 ceph-mon[37984]: 4.18 scrub starts 2024-02-05T06:45:40.676 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:40 smithi028 ceph-mon[37984]: 4.18 scrub ok 2024-02-05T06:45:40.676 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:40 smithi028 ceph-mon[37984]: 5.1d scrub starts 2024-02-05T06:45:40.676 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:40 smithi028 ceph-mon[37984]: 5.1d scrub ok 2024-02-05T06:45:40.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:40 smithi028 ceph-mon[29442]: 6.16 scrub starts 2024-02-05T06:45:40.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:40 smithi028 ceph-mon[29442]: 9.5 scrub ok 2024-02-05T06:45:40.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:40 smithi028 ceph-mon[29442]: 6.16 scrub ok 2024-02-05T06:45:40.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:40 smithi028 ceph-mon[29442]: 4.1 scrub starts 2024-02-05T06:45:40.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:40 smithi028 ceph-mon[29442]: 4.1 scrub ok 2024-02-05T06:45:40.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:40 smithi028 ceph-mon[29442]: 2.0 scrub starts 2024-02-05T06:45:40.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:40 smithi028 ceph-mon[29442]: 2.0 scrub ok 2024-02-05T06:45:40.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:40 smithi028 ceph-mon[29442]: 4.18 scrub starts 2024-02-05T06:45:40.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:40 smithi028 ceph-mon[29442]: 4.18 scrub ok 2024-02-05T06:45:40.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:40 smithi028 ceph-mon[29442]: 5.1d scrub starts 2024-02-05T06:45:40.678 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:40 smithi028 ceph-mon[29442]: 5.1d scrub ok 2024-02-05T06:45:40.708 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:40.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:40 smithi167 ceph-mon[32774]: 6.16 scrub starts 2024-02-05T06:45:40.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:40 smithi167 ceph-mon[32774]: 9.5 scrub ok 2024-02-05T06:45:40.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:40 smithi167 ceph-mon[32774]: 6.16 scrub ok 2024-02-05T06:45:40.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:40 smithi167 ceph-mon[32774]: 4.1 scrub starts 2024-02-05T06:45:40.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:40 smithi167 ceph-mon[32774]: 4.1 scrub ok 2024-02-05T06:45:40.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:40 smithi167 ceph-mon[32774]: 2.0 scrub starts 2024-02-05T06:45:40.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:40 smithi167 ceph-mon[32774]: 2.0 scrub ok 2024-02-05T06:45:40.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:40 smithi167 ceph-mon[32774]: 4.18 scrub starts 2024-02-05T06:45:40.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:40 smithi167 ceph-mon[32774]: 4.18 scrub ok 2024-02-05T06:45:40.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:40 smithi167 ceph-mon[32774]: 5.1d scrub starts 2024-02-05T06:45:40.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:40 smithi167 ceph-mon[32774]: 5.1d scrub ok 2024-02-05T06:45:40.765 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:40.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:40.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:40.867 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:40.902 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:40.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:40.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:41.004 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:41.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:41.039 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:41.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:41.140 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:41.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:41.180 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:41.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:41.281 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:41.314 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:41.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:41.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:41.416 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:41.441 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-02-05T06:45:41.441 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd test-reweight-by-utilization 110 1.0 2024-02-05T06:45:41.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:41.452 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:41.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:41.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:41 smithi028 ceph-mon[29442]: 9.9 scrub starts 2024-02-05T06:45:41.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:41 smithi028 ceph-mon[29442]: pgmap v590: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-02-05T06:45:41.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:41 smithi028 ceph-mon[29442]: from='client.25438 v1:172.21.15.28:0/2231314552' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:45:41.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:41 smithi028 ceph-mon[29442]: 5.12 scrub starts 2024-02-05T06:45:41.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:41 smithi028 ceph-mon[29442]: 5.12 scrub ok 2024-02-05T06:45:41.478 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:41 smithi028 ceph-mon[29442]: 9.0 scrub ok 2024-02-05T06:45:41.479 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:41 smithi028 ceph-mon[37984]: 9.9 scrub starts 2024-02-05T06:45:41.479 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:41 smithi028 ceph-mon[37984]: pgmap v590: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-02-05T06:45:41.479 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:41 smithi028 ceph-mon[37984]: from='client.25438 v1:172.21.15.28:0/2231314552' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:45:41.479 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:41 smithi028 ceph-mon[37984]: 5.12 scrub starts 2024-02-05T06:45:41.479 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:41 smithi028 ceph-mon[37984]: 5.12 scrub ok 2024-02-05T06:45:41.479 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:41 smithi028 ceph-mon[37984]: 9.0 scrub ok 2024-02-05T06:45:41.554 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:41.583 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:41.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:41.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:41.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:41.732 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:41.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:41 smithi167 ceph-mon[32774]: 9.9 scrub starts 2024-02-05T06:45:41.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:41 smithi167 ceph-mon[32774]: pgmap v590: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 0 B/s wr, 1.14k op/s 2024-02-05T06:45:41.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:41 smithi167 ceph-mon[32774]: from='client.25438 v1:172.21.15.28:0/2231314552' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:45:41.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:41 smithi167 ceph-mon[32774]: 5.12 scrub starts 2024-02-05T06:45:41.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:41 smithi167 ceph-mon[32774]: 5.12 scrub ok 2024-02-05T06:45:41.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:41 smithi167 ceph-mon[32774]: 9.0 scrub ok 2024-02-05T06:45:41.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:41.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:41.836 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:41.879 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:41.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:41.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:41.982 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:42.020 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:42.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:42.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:42.122 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:42.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:42.160 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:42.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:42.261 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:42.303 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:42.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:42.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:42.405 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:42.451 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:42.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:42.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:42.553 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:42.595 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:42.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:42.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:42.697 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:42.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:42 smithi028 ceph-mon[29442]: 4.1e scrub starts 2024-02-05T06:45:42.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:42 smithi028 ceph-mon[29442]: 4.1e scrub ok 2024-02-05T06:45:42.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:42 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:45:42.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:42 smithi028 ceph-mon[29442]: 3.7 scrub starts 2024-02-05T06:45:42.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:42 smithi028 ceph-mon[29442]: 3.7 scrub ok 2024-02-05T06:45:42.701 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:45:42 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:45:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:45:42.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:42 smithi028 ceph-mon[37984]: 4.1e scrub starts 2024-02-05T06:45:42.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:42 smithi028 ceph-mon[37984]: 4.1e scrub ok 2024-02-05T06:45:42.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:42 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:45:42.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:42 smithi028 ceph-mon[37984]: 3.7 scrub starts 2024-02-05T06:45:42.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:42 smithi028 ceph-mon[37984]: 3.7 scrub ok 2024-02-05T06:45:42.731 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:42.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:42.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:42.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:42 smithi167 ceph-mon[32774]: 4.1e scrub starts 2024-02-05T06:45:42.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:42 smithi167 ceph-mon[32774]: 4.1e scrub ok 2024-02-05T06:45:42.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:42 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:45:42.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:42 smithi167 ceph-mon[32774]: 3.7 scrub starts 2024-02-05T06:45:42.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:42 smithi167 ceph-mon[32774]: 3.7 scrub ok 2024-02-05T06:45:42.833 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:42.862 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:42.863 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:45:42.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:42.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:42.881 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-05T06:45:42.943 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:45:42.964 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:42.992 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:42.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:42.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:43.094 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:43.122 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:43.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:43.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:43.224 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:43.284 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:43.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:43.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:43.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:43.424 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:45:43 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:45:43.427 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-02-05T06:45:43.437 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:43 smithi028 ceph-mon[29442]: 9.9 scrub ok 2024-02-05T06:45:43.437 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:43 smithi028 ceph-mon[37984]: 9.9 scrub ok 2024-02-05T06:45:43.439 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:43.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:43.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:43.541 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:43.577 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:43.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:43.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:43.679 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:43.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:43 smithi028 ceph-mon[29442]: pgmap v591: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.42k op/s 2024-02-05T06:45:43.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:43 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/919738484' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:45:43.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:43 smithi028 ceph-mon[29442]: from='client.45425 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:45:43.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:43 smithi028 ceph-mon[29442]: 1.6 scrub starts 2024-02-05T06:45:43.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:43 smithi028 ceph-mon[29442]: 1.6 scrub ok 2024-02-05T06:45:43.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:43 smithi028 ceph-mon[37984]: pgmap v591: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.42k op/s 2024-02-05T06:45:43.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:43 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/919738484' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:45:43.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:43 smithi028 ceph-mon[37984]: from='client.45425 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:45:43.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:43 smithi028 ceph-mon[37984]: 1.6 scrub starts 2024-02-05T06:45:43.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:43 smithi028 ceph-mon[37984]: 1.6 scrub ok 2024-02-05T06:45:43.715 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:43.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:43.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:43.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:43 smithi167 ceph-mon[32774]: 9.9 scrub ok 2024-02-05T06:45:43.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:43 smithi167 ceph-mon[32774]: pgmap v591: 158 pgs: 158 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.42k op/s 2024-02-05T06:45:43.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:43 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/919738484' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:45:43.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:43 smithi167 ceph-mon[32774]: from='client.45425 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:45:43.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:43 smithi167 ceph-mon[32774]: 1.6 scrub starts 2024-02-05T06:45:43.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:43 smithi167 ceph-mon[32774]: 1.6 scrub ok 2024-02-05T06:45:43.817 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:43.845 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:43.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:43.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:43.947 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:43.979 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:43.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:43.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:44.080 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:44.119 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:44.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:44.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:44.221 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:44.258 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:44.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:44.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:44.360 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:44.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:44.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:44.495 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:44.534 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:44.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:44.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:44.636 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:44.672 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:44.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:44.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:44.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:44 smithi028 ceph-mon[29442]: 3.1f scrub starts 2024-02-05T06:45:44.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:44 smithi028 ceph-mon[29442]: 3.1f scrub ok 2024-02-05T06:45:44.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:44 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:45:44.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:44 smithi028 ceph-mon[29442]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:45:44.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:44 smithi028 ceph-mon[29442]: Cluster is now healthy 2024-02-05T06:45:44.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:44 smithi028 ceph-mon[29442]: from='client.45425 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:45:44.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:44 smithi028 ceph-mon[29442]: osdmap e325: 8 total, 8 up, 8 in 2024-02-05T06:45:44.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:44 smithi028 ceph-mon[37984]: 3.1f scrub starts 2024-02-05T06:45:44.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:44 smithi028 ceph-mon[37984]: 3.1f scrub ok 2024-02-05T06:45:44.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:44 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:45:44.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:44 smithi028 ceph-mon[37984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:45:44.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:44 smithi028 ceph-mon[37984]: Cluster is now healthy 2024-02-05T06:45:44.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:44 smithi028 ceph-mon[37984]: from='client.45425 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:45:44.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:44 smithi028 ceph-mon[37984]: osdmap e325: 8 total, 8 up, 8 in 2024-02-05T06:45:44.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:44 smithi167 ceph-mon[32774]: 3.1f scrub starts 2024-02-05T06:45:44.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:44 smithi167 ceph-mon[32774]: 3.1f scrub ok 2024-02-05T06:45:44.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:44 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:45:44.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:44 smithi167 ceph-mon[32774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:45:44.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:44 smithi167 ceph-mon[32774]: Cluster is now healthy 2024-02-05T06:45:44.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:44 smithi167 ceph-mon[32774]: from='client.45425 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:45:44.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:44 smithi167 ceph-mon[32774]: osdmap e325: 8 total, 8 up, 8 in 2024-02-05T06:45:44.774 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:44.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:44.812 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:44.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:44.913 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:44.952 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:44.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:44.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:45.053 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:45.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:45.086 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:45.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:45.187 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:45.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:45.224 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:45.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:45.325 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:45.357 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:45.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:45.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:45.459 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:45.497 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:45.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:45.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:45.599 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:45.638 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:45.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:45.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:45.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:45 smithi028 ceph-mon[29442]: 2.2 scrub starts 2024-02-05T06:45:45.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:45 smithi028 ceph-mon[29442]: 2.2 scrub ok 2024-02-05T06:45:45.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:45 smithi028 ceph-mon[29442]: 9.e scrub starts 2024-02-05T06:45:45.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:45 smithi028 ceph-mon[29442]: pgmap v593: 158 pgs: 158 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s 2024-02-05T06:45:45.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:45 smithi028 ceph-mon[29442]: 5.19 scrub starts 2024-02-05T06:45:45.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:45 smithi028 ceph-mon[29442]: 5.19 scrub ok 2024-02-05T06:45:45.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:45 smithi028 ceph-mon[29442]: 6.1b scrub starts 2024-02-05T06:45:45.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:45 smithi028 ceph-mon[29442]: 6.1b scrub ok 2024-02-05T06:45:45.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:45 smithi028 ceph-mon[37984]: 2.2 scrub starts 2024-02-05T06:45:45.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:45 smithi028 ceph-mon[37984]: 2.2 scrub ok 2024-02-05T06:45:45.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:45 smithi028 ceph-mon[37984]: 9.e scrub starts 2024-02-05T06:45:45.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:45 smithi028 ceph-mon[37984]: pgmap v593: 158 pgs: 158 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s 2024-02-05T06:45:45.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:45 smithi028 ceph-mon[37984]: 5.19 scrub starts 2024-02-05T06:45:45.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:45 smithi028 ceph-mon[37984]: 5.19 scrub ok 2024-02-05T06:45:45.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:45 smithi028 ceph-mon[37984]: 6.1b scrub starts 2024-02-05T06:45:45.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:45 smithi028 ceph-mon[37984]: 6.1b scrub ok 2024-02-05T06:45:45.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:45 smithi167 ceph-mon[32774]: 2.2 scrub starts 2024-02-05T06:45:45.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:45 smithi167 ceph-mon[32774]: 2.2 scrub ok 2024-02-05T06:45:45.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:45 smithi167 ceph-mon[32774]: 9.e scrub starts 2024-02-05T06:45:45.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:45 smithi167 ceph-mon[32774]: pgmap v593: 158 pgs: 158 active+clean; 5.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.20k op/s 2024-02-05T06:45:45.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:45 smithi167 ceph-mon[32774]: 5.19 scrub starts 2024-02-05T06:45:45.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:45 smithi167 ceph-mon[32774]: 5.19 scrub ok 2024-02-05T06:45:45.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:45 smithi167 ceph-mon[32774]: 6.1b scrub starts 2024-02-05T06:45:45.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:45 smithi167 ceph-mon[32774]: 6.1b scrub ok 2024-02-05T06:45:45.740 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:45.781 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:45.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:45.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:45.883 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:45.916 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:45.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:45.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:46.018 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:46.052 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:46.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:46.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:46.153 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:46.187 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:46.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:46.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:46.288 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:46.324 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:46.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:46.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:46.426 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:46.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:46.470 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:46.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:46.571 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:46.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:46.608 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:46.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:46.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:46 smithi028 ceph-mon[29442]: 3.6 scrub starts 2024-02-05T06:45:46.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:46 smithi028 ceph-mon[29442]: 3.6 scrub ok 2024-02-05T06:45:46.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:46 smithi028 ceph-mon[29442]: 9.e scrub ok 2024-02-05T06:45:46.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:46 smithi028 ceph-mon[29442]: 5.7 scrub starts 2024-02-05T06:45:46.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:46 smithi028 ceph-mon[29442]: 5.7 scrub ok 2024-02-05T06:45:46.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:46 smithi028 ceph-mon[37984]: 3.6 scrub starts 2024-02-05T06:45:46.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:46 smithi028 ceph-mon[37984]: 3.6 scrub ok 2024-02-05T06:45:46.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:46 smithi028 ceph-mon[37984]: 9.e scrub ok 2024-02-05T06:45:46.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:46 smithi028 ceph-mon[37984]: 5.7 scrub starts 2024-02-05T06:45:46.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:46 smithi028 ceph-mon[37984]: 5.7 scrub ok 2024-02-05T06:45:46.709 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:46.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:46 smithi167 ceph-mon[32774]: 3.6 scrub starts 2024-02-05T06:45:46.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:46 smithi167 ceph-mon[32774]: 3.6 scrub ok 2024-02-05T06:45:46.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:46 smithi167 ceph-mon[32774]: 9.e scrub ok 2024-02-05T06:45:46.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:46 smithi167 ceph-mon[32774]: 5.7 scrub starts 2024-02-05T06:45:46.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:46 smithi167 ceph-mon[32774]: 5.7 scrub ok 2024-02-05T06:45:46.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:46.748 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:46.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:46.849 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:46.886 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:46.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:46.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:46.988 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:47.024 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:47.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:47.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:47.126 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:47.158 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:47.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:47.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:47.260 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:47.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:47.299 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:47.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:47.400 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:47.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:47.434 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:47.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:47.535 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:47.567 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:47.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:47.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:47.635 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:47 smithi028 ceph-mon[37984]: 3.12 scrub starts 2024-02-05T06:45:47.635 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:47 smithi028 ceph-mon[37984]: 3.12 scrub ok 2024-02-05T06:45:47.635 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:47 smithi028 ceph-mon[37984]: 6.1c deep-scrub starts 2024-02-05T06:45:47.636 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:47 smithi028 ceph-mon[37984]: 6.1c deep-scrub ok 2024-02-05T06:45:47.636 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:47 smithi028 ceph-mon[37984]: pgmap v594: 158 pgs: 158 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.19k op/s 2024-02-05T06:45:47.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:47 smithi028 ceph-mon[29442]: 3.12 scrub starts 2024-02-05T06:45:47.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:47 smithi028 ceph-mon[29442]: 3.12 scrub ok 2024-02-05T06:45:47.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:47 smithi028 ceph-mon[29442]: 6.1c deep-scrub starts 2024-02-05T06:45:47.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:47 smithi028 ceph-mon[29442]: 6.1c deep-scrub ok 2024-02-05T06:45:47.636 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:47 smithi028 ceph-mon[29442]: pgmap v594: 158 pgs: 158 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.19k op/s 2024-02-05T06:45:47.669 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:47.709 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:47.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:47.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:47.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:47 smithi167 ceph-mon[32774]: 3.12 scrub starts 2024-02-05T06:45:47.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:47 smithi167 ceph-mon[32774]: 3.12 scrub ok 2024-02-05T06:45:47.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:47 smithi167 ceph-mon[32774]: 6.1c deep-scrub starts 2024-02-05T06:45:47.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:47 smithi167 ceph-mon[32774]: 6.1c deep-scrub ok 2024-02-05T06:45:47.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:47 smithi167 ceph-mon[32774]: pgmap v594: 158 pgs: 158 active+clean; 5.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 0 B/s wr, 1.19k op/s 2024-02-05T06:45:47.811 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:47.845 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:47.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:47.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:47.946 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:47.975 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:47.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:47.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:48.077 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:48.112 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:48.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:48.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:48.214 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:48.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:48.250 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:48.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:48.255 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:45:48.255 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:45:48.285 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:45:48.332 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:45:48.333 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:45:48.352 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:48.374 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:45:48.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:48.406 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:48.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:48.413 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:45:48.451 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:45:48.495 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:45:48.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:48 smithi028 ceph-mon[29442]: 1.9 scrub starts 2024-02-05T06:45:48.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:48 smithi028 ceph-mon[29442]: 1.9 scrub ok 2024-02-05T06:45:48.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:48 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1199414888' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:45:48.496 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:48 smithi028 ceph-mon[37984]: 1.9 scrub starts 2024-02-05T06:45:48.496 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:48 smithi028 ceph-mon[37984]: 1.9 scrub ok 2024-02-05T06:45:48.497 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:48 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1199414888' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:45:48.503 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:45:48.503 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:45:48.503 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:45:48.503 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:45:48.503 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:45:48.503 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:45:48.503 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:45:48.503 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:45:48.508 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:48.549 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:48.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:48.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:48.651 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:48.686 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:48.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:48.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:48.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:48 smithi167 ceph-mon[32774]: 1.9 scrub starts 2024-02-05T06:45:48.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:48 smithi167 ceph-mon[32774]: 1.9 scrub ok 2024-02-05T06:45:48.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:48 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1199414888' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:45:48.788 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:48.824 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:48.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:48.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:48.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:48.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:48.982 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:48.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:49.083 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:49.113 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T06:45:49.124 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:49.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:49.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:49.226 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:49.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:49.260 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:49.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:49.361 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:49.400 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:49.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:49.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:49.502 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:49.537 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:49.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:49.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:49.639 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:49.667 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:49.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:49.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:49.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:49 smithi167 ceph-mon[32774]: pgmap v595: 158 pgs: 158 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.19k op/s 2024-02-05T06:45:49.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:49 smithi167 ceph-mon[32774]: 6.2 deep-scrub starts 2024-02-05T06:45:49.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:49 smithi167 ceph-mon[32774]: 6.2 deep-scrub ok 2024-02-05T06:45:49.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:49 smithi167 ceph-mon[32774]: 9.7 deep-scrub starts 2024-02-05T06:45:49.769 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:49.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:49.827 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:49.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:49.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:49 smithi028 ceph-mon[29442]: pgmap v595: 158 pgs: 158 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.19k op/s 2024-02-05T06:45:49.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:49 smithi028 ceph-mon[29442]: 6.2 deep-scrub starts 2024-02-05T06:45:49.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:49 smithi028 ceph-mon[29442]: 6.2 deep-scrub ok 2024-02-05T06:45:49.890 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:49 smithi028 ceph-mon[29442]: 9.7 deep-scrub starts 2024-02-05T06:45:49.890 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:49 smithi028 ceph-mon[37984]: pgmap v595: 158 pgs: 158 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.19k op/s 2024-02-05T06:45:49.890 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:49 smithi028 ceph-mon[37984]: 6.2 deep-scrub starts 2024-02-05T06:45:49.890 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:49 smithi028 ceph-mon[37984]: 6.2 deep-scrub ok 2024-02-05T06:45:49.890 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:49 smithi028 ceph-mon[37984]: 9.7 deep-scrub starts 2024-02-05T06:45:49.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:49.962 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:49.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:49.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:50.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:50.097 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:50.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:50.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:50.199 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:50.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:50.231 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:50.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:50.332 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:50.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:50.363 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:50.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:50.464 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:50.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:50.497 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:50.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:50.598 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:50.632 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:50.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:50.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:50.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:50 smithi028 ceph-mon[37984]: 6.0 scrub starts 2024-02-05T06:45:50.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:50 smithi028 ceph-mon[37984]: 6.0 scrub ok 2024-02-05T06:45:50.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:50 smithi028 ceph-mon[37984]: 5.4 scrub starts 2024-02-05T06:45:50.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:50 smithi028 ceph-mon[37984]: 5.4 scrub ok 2024-02-05T06:45:50.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:50 smithi028 ceph-mon[29442]: 6.0 scrub starts 2024-02-05T06:45:50.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:50 smithi028 ceph-mon[29442]: 6.0 scrub ok 2024-02-05T06:45:50.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:50 smithi028 ceph-mon[29442]: 5.4 scrub starts 2024-02-05T06:45:50.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:50 smithi028 ceph-mon[29442]: 5.4 scrub ok 2024-02-05T06:45:50.733 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:50.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:50 smithi167 ceph-mon[32774]: 6.0 scrub starts 2024-02-05T06:45:50.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:50 smithi167 ceph-mon[32774]: 6.0 scrub ok 2024-02-05T06:45:50.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:50 smithi167 ceph-mon[32774]: 5.4 scrub starts 2024-02-05T06:45:50.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:50 smithi167 ceph-mon[32774]: 5.4 scrub ok 2024-02-05T06:45:50.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:50.777 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:50.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:50.878 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:50.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:50.916 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:50.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:51.017 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:51.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:51.056 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:51.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:51.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:51.200 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:51.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:51.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:51.301 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:51.340 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:51.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:51.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:51.442 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:51.488 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:51.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:51.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:51.589 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:51.623 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:51.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:51.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:51.725 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:51.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:51 smithi167 ceph-mon[32774]: 4.b scrub starts 2024-02-05T06:45:51.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:51 smithi167 ceph-mon[32774]: 4.b scrub ok 2024-02-05T06:45:51.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:51 smithi167 ceph-mon[32774]: pgmap v596: 158 pgs: 158 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 920 B/s rd, 0 B/s wr, 1.19k op/s 2024-02-05T06:45:51.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:51 smithi167 ceph-mon[32774]: 5.1b deep-scrub starts 2024-02-05T06:45:51.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:51 smithi167 ceph-mon[32774]: 5.1b deep-scrub ok 2024-02-05T06:45:51.737 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:51 smithi028 ceph-mon[29442]: 4.b scrub starts 2024-02-05T06:45:51.737 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:51 smithi028 ceph-mon[29442]: 4.b scrub ok 2024-02-05T06:45:51.737 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:51 smithi028 ceph-mon[29442]: pgmap v596: 158 pgs: 158 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 920 B/s rd, 0 B/s wr, 1.19k op/s 2024-02-05T06:45:51.737 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:51 smithi028 ceph-mon[29442]: 5.1b deep-scrub starts 2024-02-05T06:45:51.738 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:51 smithi028 ceph-mon[29442]: 5.1b deep-scrub ok 2024-02-05T06:45:51.738 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:51 smithi028 ceph-mon[37984]: 4.b scrub starts 2024-02-05T06:45:51.738 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:51 smithi028 ceph-mon[37984]: 4.b scrub ok 2024-02-05T06:45:51.738 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:51 smithi028 ceph-mon[37984]: pgmap v596: 158 pgs: 158 active+clean; 5.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 920 B/s rd, 0 B/s wr, 1.19k op/s 2024-02-05T06:45:51.738 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:51 smithi028 ceph-mon[37984]: 5.1b deep-scrub starts 2024-02-05T06:45:51.738 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:51 smithi028 ceph-mon[37984]: 5.1b deep-scrub ok 2024-02-05T06:45:51.778 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:51.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:51.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:51.851 INFO:teuthology.orchestra.run.smithi028.stdout:moved 69 / 474 (14.557%) 2024-02-05T06:45:51.851 INFO:teuthology.orchestra.run.smithi028.stdout:avg 59.25 2024-02-05T06:45:51.851 INFO:teuthology.orchestra.run.smithi028.stdout:stddev 6.75925 -> 6.51441 (expected baseline 7.20026) 2024-02-05T06:45:51.851 INFO:teuthology.orchestra.run.smithi028.stdout:min osd.2 with 48 -> 62 pgs (0.810127 -> 1.04641 * mean) 2024-02-05T06:45:51.851 INFO:teuthology.orchestra.run.smithi028.stdout:max osd.3 with 68 -> 62 pgs (1.14768 -> 1.04641 * mean) 2024-02-05T06:45:51.851 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:45:51.851 INFO:teuthology.orchestra.run.smithi028.stdout:oload 110 2024-02-05T06:45:51.851 INFO:teuthology.orchestra.run.smithi028.stdout:max_change 1 2024-02-05T06:45:51.851 INFO:teuthology.orchestra.run.smithi028.stdout:max_change_osds 4 2024-02-05T06:45:51.851 INFO:teuthology.orchestra.run.smithi028.stdout:average_utilization 0.0266 2024-02-05T06:45:51.851 INFO:teuthology.orchestra.run.smithi028.stdout:overload_utilization 0.0292 2024-02-05T06:45:51.851 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 weight 1.0000 -> 0.6939 2024-02-05T06:45:51.851 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 weight 1.0000 -> 0.6971 2024-02-05T06:45:51.852 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 weight 1.0000 -> 0.7084 2024-02-05T06:45:51.852 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 weight 1.0000 -> 0.7686 2024-02-05T06:45:51.858 INFO:teuthology.orchestra.run.smithi028.stderr:no change 2024-02-05T06:45:51.880 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:51.921 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:51.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:51.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:52.022 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:52.055 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:52.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:52.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:52.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:52.191 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:52.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:52.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:52.199 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:45:51 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:45:51.842+0000 7f652db03700 -1 mgr.server reply reply (4) Interrupted system call no change 2024-02-05T06:45:52.293 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:52.329 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:52.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:52.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:52.431 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:52.463 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:52.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:52.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:52.483 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:45:52.564 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:52.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:52.602 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:52.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:52.703 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:52.737 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:52.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:52.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:52.839 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:52.872 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:52.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:52.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:52.922 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:45:52 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:45:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:45:52.922 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:52 smithi028 ceph-mon[37984]: 6.12 scrub starts 2024-02-05T06:45:52.922 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:52 smithi028 ceph-mon[37984]: 6.12 scrub ok 2024-02-05T06:45:52.922 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:52 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:45:52.922 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:52 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:45:52.922 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:52 smithi028 ceph-mon[37984]: 4.2 scrub starts 2024-02-05T06:45:52.922 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:52 smithi028 ceph-mon[37984]: 4.2 scrub ok 2024-02-05T06:45:52.922 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:52 smithi028 ceph-mon[37984]: from='client.45437 v1:172.21.15.28:0/981800848' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:45:52.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:52 smithi028 ceph-mon[29442]: 6.12 scrub starts 2024-02-05T06:45:52.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:52 smithi028 ceph-mon[29442]: 6.12 scrub ok 2024-02-05T06:45:52.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:52 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:45:52.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:52 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:45:52.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:52 smithi028 ceph-mon[29442]: 4.2 scrub starts 2024-02-05T06:45:52.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:52 smithi028 ceph-mon[29442]: 4.2 scrub ok 2024-02-05T06:45:52.923 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:52 smithi028 ceph-mon[29442]: from='client.45437 v1:172.21.15.28:0/981800848' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:45:52.974 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:52.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:52 smithi167 ceph-mon[32774]: 6.12 scrub starts 2024-02-05T06:45:52.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:52 smithi167 ceph-mon[32774]: 6.12 scrub ok 2024-02-05T06:45:52.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:52 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:45:52.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:52 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:45:52.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:52 smithi167 ceph-mon[32774]: 4.2 scrub starts 2024-02-05T06:45:52.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:52 smithi167 ceph-mon[32774]: 4.2 scrub ok 2024-02-05T06:45:52.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:52 smithi167 ceph-mon[32774]: from='client.45437 v1:172.21.15.28:0/981800848' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:45:53.003 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:53.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:53.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:53.105 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:53.137 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:53.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:53.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:53.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:53.277 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:53.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:53.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:53.379 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:53.397 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:45:53.397 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-02-05T06:45:53.397 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-02-05T06:45:53.397 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:45:53.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:53.418 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:53.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:53.484 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:45:53 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:45:53.519 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:53.562 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:53.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:53.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:53.663 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:53.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:53.701 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:53.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:53.802 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:53.837 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:53.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:53.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:53.925 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:53 smithi028 ceph-mon[29442]: pgmap v597: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-02-05T06:45:53.926 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:53 smithi028 ceph-mon[29442]: osdmap e326: 8 total, 8 up, 8 in 2024-02-05T06:45:53.926 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:53 smithi028 ceph-mon[29442]: 9.6 scrub starts 2024-02-05T06:45:53.926 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:53 smithi028 ceph-mon[37984]: pgmap v597: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-02-05T06:45:53.926 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:53 smithi028 ceph-mon[37984]: osdmap e326: 8 total, 8 up, 8 in 2024-02-05T06:45:53.926 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:53 smithi028 ceph-mon[37984]: 9.6 scrub starts 2024-02-05T06:45:53.938 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:53.974 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:53.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:53.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:53.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:53 smithi167 ceph-mon[32774]: pgmap v597: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-02-05T06:45:53.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:53 smithi167 ceph-mon[32774]: osdmap e326: 8 total, 8 up, 8 in 2024-02-05T06:45:53.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:53 smithi167 ceph-mon[32774]: 9.6 scrub starts 2024-02-05T06:45:54.075 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:54.115 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:54.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:54.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:54.217 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:54.251 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:54.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:54.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:54.326 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-05T06:45:54.352 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:54.407 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:54.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:54.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:54.509 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:54.516 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:45:54.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:54.573 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:54.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:54.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:54 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:45:54.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:54 smithi028 ceph-mon[29442]: 4.1f scrub starts 2024-02-05T06:45:54.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:54 smithi028 ceph-mon[29442]: 4.1f scrub ok 2024-02-05T06:45:54.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:54 smithi028 ceph-mon[29442]: 4.4 deep-scrub starts 2024-02-05T06:45:54.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:54 smithi028 ceph-mon[29442]: 4.4 deep-scrub ok 2024-02-05T06:45:54.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:54 smithi028 ceph-mon[29442]: 1.4 scrub starts 2024-02-05T06:45:54.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:54 smithi028 ceph-mon[29442]: 1.4 scrub ok 2024-02-05T06:45:54.630 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:54 smithi028 ceph-mon[29442]: 9.6 scrub ok 2024-02-05T06:45:54.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:54 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:45:54.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:54 smithi028 ceph-mon[37984]: 4.1f scrub starts 2024-02-05T06:45:54.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:54 smithi028 ceph-mon[37984]: 4.1f scrub ok 2024-02-05T06:45:54.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:54 smithi028 ceph-mon[37984]: 4.4 deep-scrub starts 2024-02-05T06:45:54.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:54 smithi028 ceph-mon[37984]: 4.4 deep-scrub ok 2024-02-05T06:45:54.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:54 smithi028 ceph-mon[37984]: 1.4 scrub starts 2024-02-05T06:45:54.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:54 smithi028 ceph-mon[37984]: 1.4 scrub ok 2024-02-05T06:45:54.630 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:54 smithi028 ceph-mon[37984]: 9.6 scrub ok 2024-02-05T06:45:54.675 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:54.713 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:54.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:54.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:54.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:54.851 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:54.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:54.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:54.953 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:54.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:54 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:45:54.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:54 smithi167 ceph-mon[32774]: 4.1f scrub starts 2024-02-05T06:45:54.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:54 smithi167 ceph-mon[32774]: 4.1f scrub ok 2024-02-05T06:45:54.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:54 smithi167 ceph-mon[32774]: 4.4 deep-scrub starts 2024-02-05T06:45:54.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:54 smithi167 ceph-mon[32774]: 4.4 deep-scrub ok 2024-02-05T06:45:54.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:54 smithi167 ceph-mon[32774]: 1.4 scrub starts 2024-02-05T06:45:54.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:54 smithi167 ceph-mon[32774]: 1.4 scrub ok 2024-02-05T06:45:54.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:54 smithi167 ceph-mon[32774]: 9.6 scrub ok 2024-02-05T06:45:54.996 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:54.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:54.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:55.020 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-02-05T06:45:55.020 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd test-reweight-by-utilization 1000 0.05 2024-02-05T06:45:55.098 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:55.134 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:55.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:55.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:55.235 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:55.268 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:55.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:55.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:55.370 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:55.403 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:55.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:55.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:55.505 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:55.540 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:55.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:55.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:55.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:55 smithi028 ceph-mon[29442]: pgmap v599: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-02-05T06:45:55.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:55 smithi028 ceph-mon[29442]: 9.a scrub starts 2024-02-05T06:45:55.541 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:55 smithi028 ceph-mon[37984]: pgmap v599: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-02-05T06:45:55.542 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:55 smithi028 ceph-mon[37984]: 9.a scrub starts 2024-02-05T06:45:55.642 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:55.680 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:55.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:55.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:55.759 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:45:55.782 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:55.815 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:55.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:55.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:55.917 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:55.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:55.961 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:55.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:55.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:55 smithi167 ceph-mon[32774]: pgmap v599: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-02-05T06:45:55.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:55 smithi167 ceph-mon[32774]: 9.a scrub starts 2024-02-05T06:45:56.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:56.099 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:56.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:56.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:56.200 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:56.234 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:56.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:56.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:56.336 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:56.364 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:45:56.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:56.372 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:56.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:56.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:56.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:56.517 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:56.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:56.618 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:56.650 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:56.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:56.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:56.692 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:56 smithi028 ceph-mon[29442]: 4.8 scrub starts 2024-02-05T06:45:56.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:56 smithi028 ceph-mon[29442]: 5.0 deep-scrub starts 2024-02-05T06:45:56.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:56 smithi028 ceph-mon[29442]: 5.0 deep-scrub ok 2024-02-05T06:45:56.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:56 smithi028 ceph-mon[29442]: 4.8 scrub ok 2024-02-05T06:45:56.693 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:56 smithi028 ceph-mon[29442]: 9.a scrub ok 2024-02-05T06:45:56.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:56 smithi028 ceph-mon[37984]: 4.8 scrub starts 2024-02-05T06:45:56.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:56 smithi028 ceph-mon[37984]: 5.0 deep-scrub starts 2024-02-05T06:45:56.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:56 smithi028 ceph-mon[37984]: 5.0 deep-scrub ok 2024-02-05T06:45:56.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:56 smithi028 ceph-mon[37984]: 4.8 scrub ok 2024-02-05T06:45:56.693 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:56 smithi028 ceph-mon[37984]: 9.a scrub ok 2024-02-05T06:45:56.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:56.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:56.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:56.893 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:56.931 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:56.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:56.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:56.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:56 smithi167 ceph-mon[32774]: 4.8 scrub starts 2024-02-05T06:45:56.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:56 smithi167 ceph-mon[32774]: 5.0 deep-scrub starts 2024-02-05T06:45:56.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:56 smithi167 ceph-mon[32774]: 5.0 deep-scrub ok 2024-02-05T06:45:56.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:56 smithi167 ceph-mon[32774]: 4.8 scrub ok 2024-02-05T06:45:56.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:56 smithi167 ceph-mon[32774]: 9.a scrub ok 2024-02-05T06:45:57.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:57.076 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:57.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:57.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:57.178 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:57.233 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:57.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:57.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:57.335 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:57.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:57.372 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:57.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:45:57.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:57.509 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:57.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:57.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:57.610 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:57.642 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:57.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:57.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:57.744 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:57.779 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:57.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:57.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:57.797 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-02-05T06:45:57.880 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:45:57.911 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:57.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:57.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:45:58.012 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:58.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:58.044 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:58.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:58.145 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:58.183 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:58.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:58.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:58.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:57 smithi028 ceph-mon[29442]: 4.16 scrub starts 2024-02-05T06:45:58.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:57 smithi028 ceph-mon[29442]: 4.16 scrub ok 2024-02-05T06:45:58.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:57 smithi028 ceph-mon[29442]: pgmap v600: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s 2024-02-05T06:45:58.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:57 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:45:58.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:57 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:45:58.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:57 smithi028 ceph-mon[29442]: 3.2 scrub starts 2024-02-05T06:45:58.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:57 smithi028 ceph-mon[29442]: 3.2 scrub ok 2024-02-05T06:45:58.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:57 smithi028 ceph-mon[29442]: 6.19 scrub starts 2024-02-05T06:45:58.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:57 smithi028 ceph-mon[29442]: 6.19 scrub ok 2024-02-05T06:45:58.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:57 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3447421566' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:45:58.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:57 smithi028 ceph-mon[37984]: 4.16 scrub starts 2024-02-05T06:45:58.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:57 smithi028 ceph-mon[37984]: 4.16 scrub ok 2024-02-05T06:45:58.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:57 smithi028 ceph-mon[37984]: pgmap v600: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s 2024-02-05T06:45:58.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:57 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:45:58.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:57 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:45:58.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:57 smithi028 ceph-mon[37984]: 3.2 scrub starts 2024-02-05T06:45:58.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:57 smithi028 ceph-mon[37984]: 3.2 scrub ok 2024-02-05T06:45:58.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:57 smithi028 ceph-mon[37984]: 6.19 scrub starts 2024-02-05T06:45:58.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:57 smithi028 ceph-mon[37984]: 6.19 scrub ok 2024-02-05T06:45:58.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:57 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3447421566' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:45:58.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:57 smithi167 ceph-mon[32774]: 4.16 scrub starts 2024-02-05T06:45:58.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:57 smithi167 ceph-mon[32774]: 4.16 scrub ok 2024-02-05T06:45:58.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:57 smithi167 ceph-mon[32774]: pgmap v600: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 4.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s 2024-02-05T06:45:58.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:57 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:45:58.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:57 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:45:58.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:57 smithi167 ceph-mon[32774]: 3.2 scrub starts 2024-02-05T06:45:58.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:57 smithi167 ceph-mon[32774]: 3.2 scrub ok 2024-02-05T06:45:58.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:57 smithi167 ceph-mon[32774]: 6.19 scrub starts 2024-02-05T06:45:58.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:57 smithi167 ceph-mon[32774]: 6.19 scrub ok 2024-02-05T06:45:58.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:57 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3447421566' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:45:58.286 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:58.322 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:58.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:58.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:58.425 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:58.457 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:58.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:58.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:58.559 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:45:58.592 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:58.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:58.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:45:58.693 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:45:58.726 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:58.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:58.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:45:58.828 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:58.858 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:58.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:58.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:58.960 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:58.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:58.991 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:58.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:59.092 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:59.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:59.130 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:59.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:59.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:58 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:45:59.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:58 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:45:59.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:58 smithi028 ceph-mon[37984]: 5.1f scrub starts 2024-02-05T06:45:59.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:58 smithi028 ceph-mon[37984]: 5.1f scrub ok 2024-02-05T06:45:59.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:58 smithi028 ceph-mon[37984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:45:59.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:58 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3447421566' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:45:59.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:58 smithi028 ceph-mon[37984]: osdmap e327: 8 total, 8 up, 8 in 2024-02-05T06:45:59.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:58 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:45:59.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:58 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:45:59.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:58 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:45:59.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:58 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:45:59.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:58 smithi028 ceph-mon[29442]: 5.1f scrub starts 2024-02-05T06:45:59.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:58 smithi028 ceph-mon[29442]: 5.1f scrub ok 2024-02-05T06:45:59.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:58 smithi028 ceph-mon[29442]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:45:59.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:58 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3447421566' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:45:59.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:58 smithi028 ceph-mon[29442]: osdmap e327: 8 total, 8 up, 8 in 2024-02-05T06:45:59.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:58 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:45:59.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:58 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:45:59.231 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:45:59.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:58 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:45:59.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:58 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:45:59.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:58 smithi167 ceph-mon[32774]: 5.1f scrub starts 2024-02-05T06:45:59.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:58 smithi167 ceph-mon[32774]: 5.1f scrub ok 2024-02-05T06:45:59.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:58 smithi167 ceph-mon[32774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:45:59.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:58 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3447421566' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:45:59.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:58 smithi167 ceph-mon[32774]: osdmap e327: 8 total, 8 up, 8 in 2024-02-05T06:45:59.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:58 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:45:59.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:58 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:45:59.254 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:59.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:59.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:45:59.356 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:45:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:59.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:59.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:45:59.492 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:59.521 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:59.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:59.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:59.624 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:45:59.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:59.657 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:59.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:45:59.758 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:45:59.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:59.791 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:45:59.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:45:59.892 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:45:59.929 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:45:59.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:45:59.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:00.031 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:00.065 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:00.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:00.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:00.166 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:00.189 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:00.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:00.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:00.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:59 smithi028 ceph-mon[29442]: pgmap v602: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-02-05T06:46:00.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:45:59 smithi028 ceph-mon[29442]: osdmap e328: 8 total, 8 up, 8 in 2024-02-05T06:46:00.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:59 smithi028 ceph-mon[37984]: pgmap v602: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-02-05T06:46:00.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:45:59 smithi028 ceph-mon[37984]: osdmap e328: 8 total, 8 up, 8 in 2024-02-05T06:46:00.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:59 smithi167 ceph-mon[32774]: pgmap v602: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-02-05T06:46:00.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:45:59 smithi167 ceph-mon[32774]: osdmap e328: 8 total, 8 up, 8 in 2024-02-05T06:46:00.291 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:00.320 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:00.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:00.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:00.421 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:00.458 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:00.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:00.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:00.560 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:00.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:00.592 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:00.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:00.693 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:00.728 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:00.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:00.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:00.830 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:00.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:00.865 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:00.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:00.966 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:01.001 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:01.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:01.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:01.102 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:01.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:01.136 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:01.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:01.174 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:46:01.237 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:01.276 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:01.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:01.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:01.378 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:01.410 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:01.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:01.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:01.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:01 smithi028 ceph-mon[29442]: pgmap v604: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 642 op/s 2024-02-05T06:46:01.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:01 smithi028 ceph-mon[37984]: pgmap v604: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 642 op/s 2024-02-05T06:46:01.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:01 smithi167 ceph-mon[32774]: pgmap v604: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 4.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 642 op/s 2024-02-05T06:46:01.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:01.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:01.546 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:01.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:01.647 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:01.682 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:01.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:01.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:01.784 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:01.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:01.816 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:01.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:01.917 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:01.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:01.963 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:01.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:02.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:02.099 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:02.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:02.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:02.201 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:02.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:02.233 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:02.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:02.334 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:02.367 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:02.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:02.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:02.469 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:02.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:02.503 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:02.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:02.604 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:02.639 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:02.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:02.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:02.699 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:46:02 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:46:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:46:02.741 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:02.774 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:02.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:02.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:02.876 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:02.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:02.909 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:02.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:03.010 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:03.041 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:03.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:03.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:03.109 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:46:03.131 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:46:03.142 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:03.163 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T06:46:03.170 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:46:03.183 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:03.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:03.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:03.209 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:46:03.242 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:46:03.282 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:46:03.286 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:03.327 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:46:03.341 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:03.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:03.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:03.375 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:46:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:46:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:46:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:46:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:46:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:46:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:46:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:46:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:46:03.443 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:03.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:03.474 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:03.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:03.477 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:46:03 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:46:03.575 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:03.616 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:03.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:03.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:03.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:03 smithi028 ceph-mon[29442]: pgmap v605: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-02-05T06:46:03.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:03 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3567076831' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:46:03.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:03 smithi028 ceph-mon[37984]: pgmap v605: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-02-05T06:46:03.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:03 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3567076831' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:46:03.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:03.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:03 smithi167 ceph-mon[32774]: pgmap v605: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-02-05T06:46:03.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:03 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3567076831' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:46:03.754 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:03.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:03.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:03.856 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:03.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:03.889 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:03.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:03.941 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:46:03.942 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:46:03.989 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:04.016 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:04.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:04.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:04.118 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:04.151 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:04.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:04.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:04.253 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:04.289 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:04.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:04.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:04.391 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:04.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:04.430 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:04.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:04.531 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:04.569 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:04.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:04.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:04.670 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:04.674 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:04 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:46:04.674 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:04 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:46:04.704 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:04.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:04.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:04.806 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:04.842 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:04.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:04.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:04.944 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:04.978 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:04.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:04.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:04.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:04 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:46:05.080 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:05.111 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:05.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:05.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:05.213 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:05.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:05.248 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:05.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:05.349 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:05.390 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:05.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:05.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:05.491 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:05.522 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:05.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:05.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:05.580 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:05 smithi028 ceph-mon[29442]: pgmap v606: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-02-05T06:46:05.580 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:05 smithi028 ceph-mon[37984]: pgmap v606: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-02-05T06:46:05.624 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:05.655 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:05.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:05.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:05.757 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:05.787 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:05.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:05.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:05.884 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:05 smithi167 ceph-mon[32774]: pgmap v606: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-02-05T06:46:05.889 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:05.926 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:05.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:05.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:06.027 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:06.028 INFO:teuthology.orchestra.run.smithi028.stdout:moved 0 / 474 (0%) 2024-02-05T06:46:06.028 INFO:teuthology.orchestra.run.smithi028.stdout:avg 59.25 2024-02-05T06:46:06.028 INFO:teuthology.orchestra.run.smithi028.stdout:stddev 6.75925 -> 6.75925 (expected baseline 7.20026) 2024-02-05T06:46:06.028 INFO:teuthology.orchestra.run.smithi028.stdout:min osd.2 with 48 -> 48 pgs (0.810127 -> 0.810127 * mean) 2024-02-05T06:46:06.028 INFO:teuthology.orchestra.run.smithi028.stdout:max osd.3 with 68 -> 68 pgs (1.14768 -> 1.14768 * mean) 2024-02-05T06:46:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:46:06.029 INFO:teuthology.orchestra.run.smithi028.stdout:oload 1000 2024-02-05T06:46:06.029 INFO:teuthology.orchestra.run.smithi028.stdout:max_change 0.05 2024-02-05T06:46:06.029 INFO:teuthology.orchestra.run.smithi028.stdout:max_change_osds 4 2024-02-05T06:46:06.029 INFO:teuthology.orchestra.run.smithi028.stdout:average_utilization 0.0225 2024-02-05T06:46:06.029 INFO:teuthology.orchestra.run.smithi028.stdout:overload_utilization 0.2246 2024-02-05T06:46:06.038 INFO:teuthology.orchestra.run.smithi028.stderr:no change 2024-02-05T06:46:06.070 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:06.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:06.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:06.171 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:06.209 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:06.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:06.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:06.311 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:06.351 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:06.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:06.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:06.452 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:06.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:06.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:06.592 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:06.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:06.629 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:06.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:06.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:06.760 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:06.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:06.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:06.862 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:06.898 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:06.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:06.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:06.999 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:07.037 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:07.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:07.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:07.139 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:07.170 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:07.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:07.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:07.206 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-05T06:46:07.272 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:07.310 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:07.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:07.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:07.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:46:07.412 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:07.453 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:07.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:07.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:07.555 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:07.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:07.597 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:07.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:07.698 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:07.734 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:07.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:07.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:07.786 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-02-05T06:46:07.786 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd test-reweight-by-utilization 110 0.05 2024-02-05T06:46:07.787 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:07 smithi028 ceph-mon[29442]: from='client.15555 v1:172.21.15.28:0/2473057150' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:46:07.787 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:07 smithi028 ceph-mon[29442]: pgmap v607: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 593 B/s rd, 0 B/s wr, 605 op/s 2024-02-05T06:46:07.788 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:07 smithi028 ceph-mon[37984]: from='client.15555 v1:172.21.15.28:0/2473057150' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:46:07.788 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:07 smithi028 ceph-mon[37984]: pgmap v607: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 593 B/s rd, 0 B/s wr, 605 op/s 2024-02-05T06:46:07.836 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:07.865 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:46:07.869 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:07.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:07.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:07.887 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:07 smithi167 ceph-mon[32774]: from='client.15555 v1:172.21.15.28:0/2473057150' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:46:07.887 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:07 smithi167 ceph-mon[32774]: pgmap v607: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 593 B/s rd, 0 B/s wr, 605 op/s 2024-02-05T06:46:07.942 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:46:07.971 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:08.015 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:08.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:08.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:08.117 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:08.150 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:08.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:08.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:08.252 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:08.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:08.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:08.386 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:08.418 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:08.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:08.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:08.519 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:08.552 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:08.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:08.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:08.654 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:08.687 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:08.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:08.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:08.789 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:08.824 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:08.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:08.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:08.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:08.965 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:08.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:08.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:09.067 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:09.099 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:09.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:09.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:09.201 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:09.237 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:09.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:09.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:09.338 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:09.372 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:09.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:09.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:09.473 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:09.508 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:09.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:09.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:09.610 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:09.644 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:09.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:09.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:09.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:09.773 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:09.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:09.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:09.875 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:09.912 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:09.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:09.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:09.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:09 smithi028 ceph-mon[29442]: pgmap v608: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-05T06:46:09.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:09 smithi028 ceph-mon[37984]: pgmap v608: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-05T06:46:09.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:09 smithi167 ceph-mon[32774]: pgmap v608: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-05T06:46:09.987 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:46:10.014 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:10.050 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:10.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:10.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:10.152 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:10.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:10.188 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:10.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:10.289 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:10.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:10.329 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:10.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:10.430 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:10.470 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:10.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:10.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:10.571 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:10.604 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:10.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:10.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:10.706 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:10.739 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:10.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:10.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:10.841 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:10.877 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:10.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:10.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:10.979 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:11.033 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:11.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:11.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:11.134 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:11.165 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:11.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:11.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:11.266 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:11.297 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:11.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:11.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:11.399 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:11.433 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:11.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:11.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:11.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:11 smithi028 ceph-mon[37984]: pgmap v609: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 880 B/s rd, 0 B/s wr, 908 op/s 2024-02-05T06:46:11.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:11 smithi028 ceph-mon[29442]: pgmap v609: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 880 B/s rd, 0 B/s wr, 908 op/s 2024-02-05T06:46:11.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:11 smithi167 ceph-mon[32774]: pgmap v609: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 880 B/s rd, 0 B/s wr, 908 op/s 2024-02-05T06:46:11.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:11.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:11.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:11.681 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:11.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:11.714 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:11.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:11.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:11.851 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:11.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:11.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:11.953 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:11.981 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:11.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:11.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:12.083 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:12.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:12.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:12.216 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:12.249 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:12.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:12.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:12.351 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:12.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:12.385 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:12.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:12.486 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:12.529 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:12.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:12.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:12.631 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:12.658 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:12.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:12.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:12.759 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:12.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:12.797 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:12.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:12.823 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:46:12 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:46:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:46:12.841 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-02-05T06:46:12.898 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:12.930 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:12.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:12.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:12.987 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:46:13.031 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:46:13.031 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:13.063 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:13.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:13.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:13.071 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:46:13.087 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:46:13.092 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:12 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:46:13.092 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:12 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:46:13.092 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:12 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3217439989' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:46:13.092 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:12 smithi167 ceph-mon[32774]: from='client.25474 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:46:13.121 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:12 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:46:13.121 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:12 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:46:13.121 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:12 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3217439989' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:46:13.121 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:12 smithi028 ceph-mon[29442]: from='client.25474 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:46:13.121 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:12 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:46:13.121 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:12 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:46:13.121 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:12 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3217439989' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:46:13.122 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:12 smithi028 ceph-mon[37984]: from='client.25474 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:46:13.131 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:46:13.166 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:13.172 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:46:13.198 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:13.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:13.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:13.213 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:46:13.251 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:46:13.288 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:46:13.295 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:46:13.295 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:46:13.295 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:46:13.295 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:46:13.296 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:46:13.296 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:46:13.296 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:46:13.296 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:46:13.300 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:13.345 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:13.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:13.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:13.449 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:13.484 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:46:13 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:46:13.488 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:13.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:13.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:13.590 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:13.632 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:13.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:13.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:13.734 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:13.769 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:13.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:13.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:13.871 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:13.909 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:13.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:13.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:14.011 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:14.047 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:14.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:14.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:14.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:13 smithi028 ceph-mon[37984]: pgmap v610: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-02-05T06:46:14.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:13 smithi028 ceph-mon[37984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:46:14.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:13 smithi028 ceph-mon[37984]: from='client.25474 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:46:14.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:13 smithi028 ceph-mon[37984]: osdmap e329: 8 total, 8 up, 8 in 2024-02-05T06:46:14.145 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:13 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/4020090363' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:46:14.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:13 smithi028 ceph-mon[29442]: pgmap v610: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-02-05T06:46:14.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:13 smithi028 ceph-mon[29442]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:46:14.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:13 smithi028 ceph-mon[29442]: from='client.25474 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:46:14.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:13 smithi028 ceph-mon[29442]: osdmap e329: 8 total, 8 up, 8 in 2024-02-05T06:46:14.146 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:13 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/4020090363' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:46:14.149 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:14.185 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:14.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:14.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:14.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:13 smithi167 ceph-mon[32774]: pgmap v610: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-02-05T06:46:14.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:13 smithi167 ceph-mon[32774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:46:14.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:13 smithi167 ceph-mon[32774]: from='client.25474 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:46:14.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:13 smithi167 ceph-mon[32774]: osdmap e329: 8 total, 8 up, 8 in 2024-02-05T06:46:14.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:13 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/4020090363' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:46:14.287 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:14.323 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:14.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:14.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:14.425 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:14.465 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:14.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:14.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:14.567 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:14.604 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:14.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:14.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:14.706 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:14.745 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:14.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:14.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:14.847 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:14.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:14.880 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:14.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:14.980 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:15.019 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:15.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:15.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:15.033 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:14 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:46:15.033 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:14 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:46:15.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:15.159 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:15.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:15.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:15.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:14 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:46:15.261 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:15.286 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:15.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:15.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:15.388 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:15.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:15.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:15.530 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:15.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:15.571 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:15.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:15.608 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T06:46:15.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:15.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:15.718 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:15.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:15.818 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:15.865 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:15.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:15.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:15.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:15 smithi028 ceph-mon[29442]: pgmap v612: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-05T06:46:15.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:15 smithi028 ceph-mon[37984]: pgmap v612: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-05T06:46:15.967 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:16.002 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:16.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:16.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:16.103 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:16.140 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:16.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:16.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:16.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:15 smithi167 ceph-mon[32774]: pgmap v612: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-05T06:46:16.241 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:16.278 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:16.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:16.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:16.380 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:16.415 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:16.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:16.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:16.517 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:16.566 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:16.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:16.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:16.668 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:16.703 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:16.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:16.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:16.795 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:46:16.805 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:16.839 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:16.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:16.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:16.941 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:17.010 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:17.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:17.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:17.112 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:17.149 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:17.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:17.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:17.250 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:17.283 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:17.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:17.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:17.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:17.425 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:17.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:17.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:17.527 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:17.579 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:17.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:17.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:17.681 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:17.714 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:17.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:17.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:17.816 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:17.848 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:17.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:17.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:17.889 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:46:17.889 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-02-05T06:46:17.889 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-02-05T06:46:17.889 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-02-05T06:46:17.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.890 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14568.0:273 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e151)", 2024-02-05T06:46:17.890 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:38:15.680881+0000", 2024-02-05T06:46:17.890 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 482.19770540100001, 2024-02-05T06:46:17.890 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.92211636600000002, 2024-02-05T06:46:17.890 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:46:17.890 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-02-05T06:46:17.890 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:46:17.890 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14568", 2024-02-05T06:46:17.890 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.167:0/939366379", 2024-02-05T06:46:17.890 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 273 2024-02-05T06:46:17.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.890 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:46:17.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.890 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:46:17.891 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.680881+0000", 2024-02-05T06:46:17.891 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:17.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.891 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.891 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:46:17.891 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.680881+0000", 2024-02-05T06:46:17.891 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:17.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.891 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.891 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:46:17.891 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.680883+0000", 2024-02-05T06:46:17.891 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3130000000000001e-06 2024-02-05T06:46:17.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.891 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.892 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:46:17.892 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.680891+0000", 2024-02-05T06:46:17.892 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1389999999999995e-06 2024-02-05T06:46:17.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.892 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:46:17.892 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.680906+0000", 2024-02-05T06:46:17.892 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5218000000000001e-05 2024-02-05T06:46:17.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.892 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:46:17.892 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.680918+0000", 2024-02-05T06:46:17.892 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1949e-05 2024-02-05T06:46:17.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.893 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.893 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.680990+0000", 2024-02-05T06:46:17.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1134e-05 2024-02-05T06:46:17.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.893 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:46:17.893 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:15.681002+0000", 2024-02-05T06:46:17.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2743e-05 2024-02-05T06:46:17.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.893 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.893 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:16.602683+0000", 2024-02-05T06:46:17.893 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.92168123499999999 2024-02-05T06:46:17.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.894 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:46:17.894 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:16.602846+0000", 2024-02-05T06:46:17.894 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016218800000000001 2024-02-05T06:46:17.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.894 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:46:17.894 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:38:16.602997+0000", 2024-02-05T06:46:17.894 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000151447 2024-02-05T06:46:17.894 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:17.894 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:46:17.894 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:17.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.894 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:30029 8.3 8:c5f2c81c:::benchmark_data_smithi028_105698_object30028:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-02-05T06:46:17.895 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:07.592571+0000", 2024-02-05T06:46:17.895 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 370.28601491500001, 2024-02-05T06:46:17.895 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1152291089999999, 2024-02-05T06:46:17.895 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:46:17.895 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:46:17.895 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:46:17.895 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:46:17.895 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:46:17.895 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 30029 2024-02-05T06:46:17.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.895 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:46:17.895 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.895 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:46:17.896 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.592571+0000", 2024-02-05T06:46:17.896 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:17.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.896 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.896 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:46:17.896 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.592571+0000", 2024-02-05T06:46:17.896 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:17.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.896 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.896 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:46:17.896 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.592573+0000", 2024-02-05T06:46:17.896 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.821e-06 2024-02-05T06:46:17.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.897 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:46:17.897 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.592599+0000", 2024-02-05T06:46:17.897 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6058999999999999e-05 2024-02-05T06:46:17.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.897 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:46:17.897 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.592609+0000", 2024-02-05T06:46:17.897 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.023e-05 2024-02-05T06:46:17.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.897 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.897 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:46:17.897 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.592618+0000", 2024-02-05T06:46:17.897 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6459999999999999e-06 2024-02-05T06:46:17.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.898 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.898 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.898 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.592730+0000", 2024-02-05T06:46:17.898 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011203 2024-02-05T06:46:17.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.898 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.898 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:46:17.898 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.592742+0000", 2024-02-05T06:46:17.898 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1506e-05 2024-02-05T06:46:17.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.898 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.898 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.899 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:08.579853+0000", 2024-02-05T06:46:17.899 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.987111344 2024-02-05T06:46:17.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.899 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.899 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:46:17.899 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:08.579866+0000", 2024-02-05T06:46:17.899 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3498e-05 2024-02-05T06:46:17.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.899 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.899 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.899 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.692983+0000", 2024-02-05T06:46:17.899 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.113116824 2024-02-05T06:46:17.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.899 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.899 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:46:17.900 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.693165+0000", 2024-02-05T06:46:17.900 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018180799999999999 2024-02-05T06:46:17.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.900 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,6", 2024-02-05T06:46:17.900 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.693355+0000", 2024-02-05T06:46:17.900 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019034600000000001 2024-02-05T06:46:17.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.900 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:46:17.900 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.699405+0000", 2024-02-05T06:46:17.900 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0060496889999999996 2024-02-05T06:46:17.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.901 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:46:17.901 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.703216+0000", 2024-02-05T06:46:17.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0038114249999999998 2024-02-05T06:46:17.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.901 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:46:17.901 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.706785+0000", 2024-02-05T06:46:17.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035685700000000001 2024-02-05T06:46:17.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.901 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.901 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:46:17.901 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.706829+0000", 2024-02-05T06:46:17.901 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3662000000000003e-05 2024-02-05T06:46:17.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.902 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:46:17.902 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.707800+0000", 2024-02-05T06:46:17.902 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00097165100000000003 2024-02-05T06:46:17.902 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:17.902 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:46:17.902 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:17.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.902 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.902 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:30035 8.3 8:c1ac7661:::benchmark_data_smithi028_105698_object30034:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-02-05T06:46:17.902 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:07.594660+0000", 2024-02-05T06:46:17.902 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 370.283926071, 2024-02-05T06:46:17.902 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1138501129999998, 2024-02-05T06:46:17.902 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:46:17.903 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:46:17.903 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:46:17.903 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:46:17.903 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:46:17.903 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 30035 2024-02-05T06:46:17.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.903 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:46:17.904 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.904 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:46:17.904 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.594660+0000", 2024-02-05T06:46:17.904 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:17.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.904 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.904 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:46:17.904 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.594660+0000", 2024-02-05T06:46:17.904 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:17.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.904 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.904 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:46:17.904 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.594662+0000", 2024-02-05T06:46:17.904 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.66e-06 2024-02-05T06:46:17.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.905 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:46:17.905 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.594694+0000", 2024-02-05T06:46:17.905 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2314999999999998e-05 2024-02-05T06:46:17.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.905 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:46:17.905 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.594710+0000", 2024-02-05T06:46:17.905 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5560999999999999e-05 2024-02-05T06:46:17.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.905 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.905 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:46:17.905 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.594718+0000", 2024-02-05T06:46:17.906 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8779999999999998e-06 2024-02-05T06:46:17.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.906 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.906 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.594904+0000", 2024-02-05T06:46:17.906 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000186291 2024-02-05T06:46:17.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.906 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:46:17.906 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.594914+0000", 2024-02-05T06:46:17.906 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0645000000000001e-05 2024-02-05T06:46:17.906 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.906 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.906 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.906 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:08.579896+0000", 2024-02-05T06:46:17.907 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.98498187599999998 2024-02-05T06:46:17.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.907 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.907 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:46:17.907 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:08.579906+0000", 2024-02-05T06:46:17.907 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1819999999999995e-06 2024-02-05T06:46:17.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.907 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.907 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.907 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.697271+0000", 2024-02-05T06:46:17.907 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.117365127 2024-02-05T06:46:17.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.908 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:46:17.908 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.697455+0000", 2024-02-05T06:46:17.908 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018462200000000001 2024-02-05T06:46:17.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.908 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,6", 2024-02-05T06:46:17.908 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.697650+0000", 2024-02-05T06:46:17.908 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000195052 2024-02-05T06:46:17.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.908 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:46:17.908 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.702913+0000", 2024-02-05T06:46:17.909 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0052624050000000004 2024-02-05T06:46:17.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.909 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.909 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:46:17.909 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.703222+0000", 2024-02-05T06:46:17.909 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00030888699999999999 2024-02-05T06:46:17.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.909 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.909 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:46:17.909 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.707910+0000", 2024-02-05T06:46:17.909 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004688289 2024-02-05T06:46:17.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.909 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.909 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:46:17.909 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.707948+0000", 2024-02-05T06:46:17.909 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8109999999999999e-05 2024-02-05T06:46:17.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.910 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:46:17.910 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.708510+0000", 2024-02-05T06:46:17.910 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00056221299999999999 2024-02-05T06:46:17.910 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:17.910 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:46:17.910 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:17.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.910 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:30040 8.a 8:58e944f2:::benchmark_data_smithi028_105698_object30039:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-02-05T06:46:17.910 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:07.599231+0000", 2024-02-05T06:46:17.910 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 370.27935473100001, 2024-02-05T06:46:17.910 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2105997569999998, 2024-02-05T06:46:17.910 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:46:17.910 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:46:17.911 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:46:17.911 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:46:17.911 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:46:17.911 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 30040 2024-02-05T06:46:17.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.911 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:46:17.911 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.911 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:46:17.911 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.599231+0000", 2024-02-05T06:46:17.911 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:17.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.911 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.911 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:46:17.911 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.599231+0000", 2024-02-05T06:46:17.911 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:17.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.911 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.912 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:46:17.912 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.599234+0000", 2024-02-05T06:46:17.912 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1179999999999999e-06 2024-02-05T06:46:17.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.912 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.912 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:46:17.912 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.599271+0000", 2024-02-05T06:46:17.912 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7178999999999998e-05 2024-02-05T06:46:17.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.912 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.912 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:46:17.912 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.599282+0000", 2024-02-05T06:46:17.912 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.164e-05 2024-02-05T06:46:17.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.912 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.913 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:46:17.913 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.599291+0000", 2024-02-05T06:46:17.913 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1620000000000008e-06 2024-02-05T06:46:17.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.913 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.913 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.913 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.599364+0000", 2024-02-05T06:46:17.913 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3365e-05 2024-02-05T06:46:17.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.913 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.913 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:46:17.913 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.599379+0000", 2024-02-05T06:46:17.913 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4748e-05 2024-02-05T06:46:17.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.913 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.914 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.914 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:08.574924+0000", 2024-02-05T06:46:17.914 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.975544894 2024-02-05T06:46:17.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.914 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.914 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:46:17.914 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:08.574939+0000", 2024-02-05T06:46:17.914 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5614999999999999e-05 2024-02-05T06:46:17.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.914 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.914 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.914 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.784422+0000", 2024-02-05T06:46:17.914 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2094823859999999 2024-02-05T06:46:17.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.914 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.914 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:46:17.915 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.784569+0000", 2024-02-05T06:46:17.915 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000147264 2024-02-05T06:46:17.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.915 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.915 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,3", 2024-02-05T06:46:17.915 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.784735+0000", 2024-02-05T06:46:17.915 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016588400000000001 2024-02-05T06:46:17.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.915 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.915 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:46:17.915 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.789891+0000", 2024-02-05T06:46:17.915 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051558660000000003 2024-02-05T06:46:17.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.915 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.915 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:46:17.915 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.799372+0000", 2024-02-05T06:46:17.915 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0094815720000000006 2024-02-05T06:46:17.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.916 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.916 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:46:17.916 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.807079+0000", 2024-02-05T06:46:17.916 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0077065839999999998 2024-02-05T06:46:17.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.916 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.916 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:46:17.916 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.807132+0000", 2024-02-05T06:46:17.916 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3000000000000001e-05 2024-02-05T06:46:17.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.916 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.916 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:46:17.916 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.809831+0000", 2024-02-05T06:46:17.917 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026994800000000002 2024-02-05T06:46:17.917 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:17.917 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:46:17.917 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:17.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.917 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.917 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:30041 8.3 8:c6bad42b:::benchmark_data_smithi028_105698_object30040:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-02-05T06:46:17.917 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:07.600796+0000", 2024-02-05T06:46:17.917 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 370.277790584, 2024-02-05T06:46:17.917 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1085740049999999, 2024-02-05T06:46:17.917 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:46:17.917 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:46:17.917 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:46:17.917 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:46:17.918 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:46:17.918 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 30041 2024-02-05T06:46:17.918 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.918 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:46:17.918 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.918 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:46:17.918 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.600796+0000", 2024-02-05T06:46:17.918 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:17.918 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.918 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.918 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:46:17.919 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.600796+0000", 2024-02-05T06:46:17.919 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:17.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.919 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.919 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:46:17.919 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.600797+0000", 2024-02-05T06:46:17.919 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.206e-06 2024-02-05T06:46:17.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.919 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.919 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:46:17.919 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.600817+0000", 2024-02-05T06:46:17.919 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0333999999999999e-05 2024-02-05T06:46:17.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.920 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.920 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:46:17.920 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.600827+0000", 2024-02-05T06:46:17.920 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3759999999999997e-06 2024-02-05T06:46:17.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.920 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.920 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:46:17.920 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.600833+0000", 2024-02-05T06:46:17.920 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3829999999999997e-06 2024-02-05T06:46:17.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.920 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.921 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.921 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.600894+0000", 2024-02-05T06:46:17.921 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0829999999999999e-05 2024-02-05T06:46:17.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.921 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.921 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:46:17.921 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.600906+0000", 2024-02-05T06:46:17.921 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1997000000000001e-05 2024-02-05T06:46:17.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.921 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.921 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.922 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:08.579931+0000", 2024-02-05T06:46:17.922 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.97902493999999995 2024-02-05T06:46:17.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.922 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.922 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:46:17.922 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:08.579939+0000", 2024-02-05T06:46:17.922 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2009999999999999e-06 2024-02-05T06:46:17.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.922 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.922 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.922 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.699490+0000", 2024-02-05T06:46:17.922 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1195510200000001 2024-02-05T06:46:17.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.922 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.923 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:46:17.923 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.699607+0000", 2024-02-05T06:46:17.923 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000116642 2024-02-05T06:46:17.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.923 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.923 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,6", 2024-02-05T06:46:17.923 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.699714+0000", 2024-02-05T06:46:17.923 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000106994 2024-02-05T06:46:17.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.923 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.923 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:46:17.923 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.702980+0000", 2024-02-05T06:46:17.924 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032668300000000001 2024-02-05T06:46:17.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.924 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.924 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:46:17.924 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.703225+0000", 2024-02-05T06:46:17.924 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024474099999999998 2024-02-05T06:46:17.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.924 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.924 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:46:17.924 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.708569+0000", 2024-02-05T06:46:17.924 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0053439530000000002 2024-02-05T06:46:17.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.924 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.924 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:46:17.925 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.708625+0000", 2024-02-05T06:46:17.925 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5634000000000003e-05 2024-02-05T06:46:17.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.925 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.925 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:46:17.925 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.709370+0000", 2024-02-05T06:46:17.925 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000744924 2024-02-05T06:46:17.925 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:17.925 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:46:17.925 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:17.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.925 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.925 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:49687 8.3 8:cc75f722:::benchmark_data_smithi028_105698_object49686:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-02-05T06:46:17.925 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:36.282700+0000", 2024-02-05T06:46:17.925 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 341.59588587600001, 2024-02-05T06:46:17.925 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1319326570000001, 2024-02-05T06:46:17.926 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:46:17.926 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-02-05T06:46:17.926 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:46:17.926 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:46:17.926 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:46:17.926 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49687 2024-02-05T06:46:17.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.926 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:46:17.926 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.926 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:46:17.926 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.282700+0000", 2024-02-05T06:46:17.926 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:17.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.926 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.926 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:46:17.926 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.282700+0000", 2024-02-05T06:46:17.927 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:17.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.927 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.927 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:46:17.927 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.282701+0000", 2024-02-05T06:46:17.927 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5899999999999995e-07 2024-02-05T06:46:17.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.927 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.927 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:46:17.927 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.282725+0000", 2024-02-05T06:46:17.927 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3589000000000001e-05 2024-02-05T06:46:17.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.927 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.927 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:46:17.927 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.282733+0000", 2024-02-05T06:46:17.927 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9410000000000007e-06 2024-02-05T06:46:17.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.928 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.928 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:46:17.928 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.282740+0000", 2024-02-05T06:46:17.928 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0570000000000004e-06 2024-02-05T06:46:17.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.928 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.928 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.928 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.284394+0000", 2024-02-05T06:46:17.928 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0016543350000000001 2024-02-05T06:46:17.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.928 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.928 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:46:17.928 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.284539+0000", 2024-02-05T06:46:17.928 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000144623 2024-02-05T06:46:17.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.929 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.929 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,6", 2024-02-05T06:46:17.929 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.284693+0000", 2024-02-05T06:46:17.929 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000154728 2024-02-05T06:46:17.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.929 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.929 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:46:17.929 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.293625+0000", 2024-02-05T06:46:17.929 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0089314140000000004 2024-02-05T06:46:17.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.929 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.929 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.929 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.295373+0000", 2024-02-05T06:46:17.929 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017478400000000001 2024-02-05T06:46:17.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.930 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.930 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:46:17.930 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.295385+0000", 2024-02-05T06:46:17.930 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2675e-05 2024-02-05T06:46:17.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.930 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.930 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.408242+0000", 2024-02-05T06:46:17.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1128566529999999 2024-02-05T06:46:17.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.931 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for degraded object", 2024-02-05T06:46:17.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.408817+0000", 2024-02-05T06:46:17.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000575345 2024-02-05T06:46:17.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.931 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.931 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.414557+0000", 2024-02-05T06:46:17.931 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0057392299999999997 2024-02-05T06:46:17.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.931 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:46:17.932 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.414633+0000", 2024-02-05T06:46:17.932 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6367999999999998e-05 2024-02-05T06:46:17.932 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:17.932 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:46:17.932 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:17.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.932 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.932 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:49713 8.3 8:ced18c56:::benchmark_data_smithi028_105698_object49712:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T06:46:17.932 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:36.312789+0000", 2024-02-05T06:46:17.932 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 341.56579672800001, 2024-02-05T06:46:17.932 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1023306589999999, 2024-02-05T06:46:17.932 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:46:17.932 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:46:17.933 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:46:17.933 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:46:17.933 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:46:17.933 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49713 2024-02-05T06:46:17.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.933 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:46:17.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.933 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:46:17.933 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.312789+0000", 2024-02-05T06:46:17.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:17.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.933 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:46:17.933 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.312789+0000", 2024-02-05T06:46:17.933 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:17.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.934 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:46:17.934 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.312791+0000", 2024-02-05T06:46:17.934 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3769999999999999e-06 2024-02-05T06:46:17.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.934 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:46:17.934 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.312812+0000", 2024-02-05T06:46:17.934 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1488000000000002e-05 2024-02-05T06:46:17.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.934 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:46:17.934 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.312822+0000", 2024-02-05T06:46:17.934 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.55e-06 2024-02-05T06:46:17.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.935 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:46:17.935 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.312830+0000", 2024-02-05T06:46:17.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6539999999999999e-06 2024-02-05T06:46:17.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.935 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.935 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.312943+0000", 2024-02-05T06:46:17.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000113674 2024-02-05T06:46:17.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.935 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:46:17.935 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.312953+0000", 2024-02-05T06:46:17.935 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0131e-05 2024-02-05T06:46:17.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.936 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.936 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.409203+0000", 2024-02-05T06:46:17.936 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0962497149999999 2024-02-05T06:46:17.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.936 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:46:17.936 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.409352+0000", 2024-02-05T06:46:17.936 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014935999999999999 2024-02-05T06:46:17.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.936 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,6", 2024-02-05T06:46:17.936 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.409538+0000", 2024-02-05T06:46:17.937 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018521400000000001 2024-02-05T06:46:17.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.937 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:46:17.937 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.414863+0000", 2024-02-05T06:46:17.937 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0053253859999999997 2024-02-05T06:46:17.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.937 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:46:17.937 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.414911+0000", 2024-02-05T06:46:17.937 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8269999999999997e-05 2024-02-05T06:46:17.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.937 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.937 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:46:17.937 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.414953+0000", 2024-02-05T06:46:17.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1390000000000002e-05 2024-02-05T06:46:17.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.938 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:46:17.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.415014+0000", 2024-02-05T06:46:17.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1610000000000001e-05 2024-02-05T06:46:17.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.938 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:46:17.938 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.415120+0000", 2024-02-05T06:46:17.938 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010584 2024-02-05T06:46:17.938 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:17.938 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:46:17.938 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:17.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.938 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.939 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:49724 8.3 8:c5be311a:::benchmark_data_smithi028_105698_object49723:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T06:46:17.939 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:36.330146+0000", 2024-02-05T06:46:17.939 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 341.54844044399999, 2024-02-05T06:46:17.939 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0852702599999999, 2024-02-05T06:46:17.939 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:46:17.939 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:46:17.939 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:46:17.939 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:46:17.939 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:46:17.939 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49724 2024-02-05T06:46:17.939 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.939 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:46:17.939 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.939 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:46:17.940 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.330146+0000", 2024-02-05T06:46:17.940 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:17.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.940 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.940 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:46:17.940 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.330146+0000", 2024-02-05T06:46:17.940 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:17.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.940 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.940 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:46:17.940 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.330146+0000", 2024-02-05T06:46:17.940 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3600000000000003e-07 2024-02-05T06:46:17.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.940 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.940 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:46:17.941 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.330166+0000", 2024-02-05T06:46:17.941 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9395999999999999e-05 2024-02-05T06:46:17.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.941 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.941 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:46:17.941 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.330172+0000", 2024-02-05T06:46:17.941 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4330000000000001e-06 2024-02-05T06:46:17.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.941 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.941 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:46:17.941 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.330178+0000", 2024-02-05T06:46:17.941 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7239999999999997e-06 2024-02-05T06:46:17.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.941 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.942 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.942 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.343280+0000", 2024-02-05T06:46:17.942 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013101880999999999 2024-02-05T06:46:17.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.942 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:46:17.942 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.343289+0000", 2024-02-05T06:46:17.942 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1630000000000002e-06 2024-02-05T06:46:17.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.942 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.942 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.411443+0000", 2024-02-05T06:46:17.942 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0681536309999999 2024-02-05T06:46:17.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.942 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.942 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:46:17.942 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.412020+0000", 2024-02-05T06:46:17.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00057775299999999997 2024-02-05T06:46:17.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.943 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,6", 2024-02-05T06:46:17.943 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.412206+0000", 2024-02-05T06:46:17.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018605 2024-02-05T06:46:17.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.943 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:46:17.943 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.414869+0000", 2024-02-05T06:46:17.943 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026630299999999998 2024-02-05T06:46:17.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.943 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.943 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:46:17.943 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.415202+0000", 2024-02-05T06:46:17.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00033210299999999999 2024-02-05T06:46:17.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.944 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:46:17.944 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.415272+0000", 2024-02-05T06:46:17.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0333000000000006e-05 2024-02-05T06:46:17.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.944 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:46:17.944 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.415331+0000", 2024-02-05T06:46:17.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8961e-05 2024-02-05T06:46:17.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.944 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:46:17.945 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.415416+0000", 2024-02-05T06:46:17.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5166000000000004e-05 2024-02-05T06:46:17.945 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:17.945 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:46:17.945 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:17.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.945 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:49733 8.3 8:c9dc6c6c:::benchmark_data_smithi028_105698_object49732:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T06:46:17.945 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:36.343139+0000", 2024-02-05T06:46:17.945 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 341.53544682900002, 2024-02-05T06:46:17.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0826276269999999, 2024-02-05T06:46:17.945 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:46:17.945 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:46:17.945 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:46:17.945 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:46:17.946 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:46:17.946 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49733 2024-02-05T06:46:17.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.946 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:46:17.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:46:17.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.343139+0000", 2024-02-05T06:46:17.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:17.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:46:17.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.343139+0000", 2024-02-05T06:46:17.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:17.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:46:17.947 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.343142+0000", 2024-02-05T06:46:17.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6010000000000002e-06 2024-02-05T06:46:17.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:46:17.947 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.343172+0000", 2024-02-05T06:46:17.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0071000000000001e-05 2024-02-05T06:46:17.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:46:17.947 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.343186+0000", 2024-02-05T06:46:17.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3716e-05 2024-02-05T06:46:17.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.948 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:46:17.948 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.343195+0000", 2024-02-05T06:46:17.948 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4800000000000007e-06 2024-02-05T06:46:17.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.948 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.948 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.343312+0000", 2024-02-05T06:46:17.948 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011652800000000001 2024-02-05T06:46:17.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.948 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:46:17.948 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.343318+0000", 2024-02-05T06:46:17.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.1929999999999998e-06 2024-02-05T06:46:17.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.415593+0000", 2024-02-05T06:46:17.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0722749810000001 2024-02-05T06:46:17.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:46:17.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.415729+0000", 2024-02-05T06:46:17.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013639699999999999 2024-02-05T06:46:17.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,6", 2024-02-05T06:46:17.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.415857+0000", 2024-02-05T06:46:17.950 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012768600000000001 2024-02-05T06:46:17.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.950 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:46:17.950 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.423145+0000", 2024-02-05T06:46:17.950 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0072879090000000004 2024-02-05T06:46:17.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.950 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:46:17.950 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.425359+0000", 2024-02-05T06:46:17.950 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022138290000000001 2024-02-05T06:46:17.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.950 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:46:17.950 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.425476+0000", 2024-02-05T06:46:17.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011764699999999999 2024-02-05T06:46:17.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:46:17.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.425663+0000", 2024-02-05T06:46:17.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018675699999999999 2024-02-05T06:46:17.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:46:17.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.425767+0000", 2024-02-05T06:46:17.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103832 2024-02-05T06:46:17.951 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:17.951 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:46:17.952 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:17.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.952 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:49757 8.3 8:c0e382f1:::benchmark_data_smithi028_105698_object49756:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T06:46:17.952 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:36.366231+0000", 2024-02-05T06:46:17.952 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 341.51235541199998, 2024-02-05T06:46:17.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.059756564, 2024-02-05T06:46:17.952 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:46:17.952 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:46:17.952 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:46:17.952 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:46:17.952 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:46:17.952 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49757 2024-02-05T06:46:17.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.953 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:46:17.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.953 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:46:17.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.366231+0000", 2024-02-05T06:46:17.953 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:17.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.953 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:46:17.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.366231+0000", 2024-02-05T06:46:17.953 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:17.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.953 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:46:17.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.366232+0000", 2024-02-05T06:46:17.953 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.57e-06 2024-02-05T06:46:17.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.954 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:46:17.954 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.366254+0000", 2024-02-05T06:46:17.954 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1361000000000001e-05 2024-02-05T06:46:17.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.954 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:46:17.954 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.366274+0000", 2024-02-05T06:46:17.954 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0615000000000002e-05 2024-02-05T06:46:17.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.954 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:46:17.954 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.366282+0000", 2024-02-05T06:46:17.954 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8509999999999999e-06 2024-02-05T06:46:17.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.955 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.955 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.955 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.366330+0000", 2024-02-05T06:46:17.955 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8111000000000001e-05 2024-02-05T06:46:17.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.955 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.955 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:46:17.955 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.366337+0000", 2024-02-05T06:46:17.955 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6830000000000002e-06 2024-02-05T06:46:17.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.955 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.955 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.955 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.417831+0000", 2024-02-05T06:46:17.956 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0514945309999999 2024-02-05T06:46:17.956 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:17.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.957 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:46:17.957 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.418001+0000", 2024-02-05T06:46:17.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000169118 2024-02-05T06:46:17.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.958 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,6", 2024-02-05T06:46:17.958 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.418267+0000", 2024-02-05T06:46:17.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000265961 2024-02-05T06:46:17.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.958 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:46:17.958 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.425317+0000", 2024-02-05T06:46:17.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0070499940000000004 2024-02-05T06:46:17.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.958 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:46:17.958 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.425816+0000", 2024-02-05T06:46:17.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00049923399999999998 2024-02-05T06:46:17.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.959 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:46:17.959 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.425853+0000", 2024-02-05T06:46:17.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7292000000000001e-05 2024-02-05T06:46:17.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.959 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:46:17.959 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.425906+0000", 2024-02-05T06:46:17.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2920000000000002e-05 2024-02-05T06:46:17.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.960 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:46:17.960 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.425987+0000", 2024-02-05T06:46:17.960 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1322999999999994e-05 2024-02-05T06:46:17.960 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:17.960 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:46:17.960 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:17.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.960 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:49823 8.3 8:ced86690:::benchmark_data_smithi028_105698_object49822:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T06:46:17.960 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:36.503768+0000", 2024-02-05T06:46:17.960 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 341.37481772199999, 2024-02-05T06:46:17.960 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.92251770899999996, 2024-02-05T06:46:17.960 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:46:17.961 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:46:17.961 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:46:17.961 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:46:17.961 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:46:17.961 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49823 2024-02-05T06:46:17.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.961 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:46:17.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.961 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:46:17.961 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.503768+0000", 2024-02-05T06:46:17.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:17.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.961 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:46:17.961 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.503768+0000", 2024-02-05T06:46:17.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:17.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:46:17.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.503771+0000", 2024-02-05T06:46:17.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.08e-06 2024-02-05T06:46:17.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:46:17.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.503805+0000", 2024-02-05T06:46:17.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4252999999999999e-05 2024-02-05T06:46:17.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:46:17.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.503822+0000", 2024-02-05T06:46:17.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6937000000000001e-05 2024-02-05T06:46:17.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:46:17.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.503837+0000", 2024-02-05T06:46:17.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5699999999999999e-05 2024-02-05T06:46:17.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.503909+0000", 2024-02-05T06:46:17.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1506000000000003e-05 2024-02-05T06:46:17.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:46:17.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.503919+0000", 2024-02-05T06:46:17.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0224000000000001e-05 2024-02-05T06:46:17.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.420610+0000", 2024-02-05T06:46:17.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.91669115000000001 2024-02-05T06:46:17.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:46:17.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.420741+0000", 2024-02-05T06:46:17.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013119999999999999 2024-02-05T06:46:17.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.964 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.964 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,6", 2024-02-05T06:46:17.964 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.420886+0000", 2024-02-05T06:46:17.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000144334 2024-02-05T06:46:17.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.965 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:46:17.965 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.425374+0000", 2024-02-05T06:46:17.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0044879689999999996 2024-02-05T06:46:17.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.965 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:46:17.965 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.426035+0000", 2024-02-05T06:46:17.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00066087299999999997 2024-02-05T06:46:17.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.965 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:46:17.966 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.426092+0000", 2024-02-05T06:46:17.966 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7082e-05 2024-02-05T06:46:17.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.966 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:46:17.966 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.426149+0000", 2024-02-05T06:46:17.966 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6952e-05 2024-02-05T06:46:17.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.966 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:46:17.966 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.426286+0000", 2024-02-05T06:46:17.966 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013744899999999999 2024-02-05T06:46:17.966 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:17.966 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:46:17.967 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:17.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.967 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.967 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:49829 8.3 8:cb27f3a5:::benchmark_data_smithi028_105698_object49828:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-02-05T06:46:17.967 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:36.507979+0000", 2024-02-05T06:46:17.967 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 341.37060669200002, 2024-02-05T06:46:17.967 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.925142778, 2024-02-05T06:46:17.967 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:46:17.967 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:46:17.967 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:46:17.967 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:46:17.967 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:46:17.967 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 49829 2024-02-05T06:46:17.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.967 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:46:17.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.968 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:46:17.968 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.507979+0000", 2024-02-05T06:46:17.968 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:17.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.968 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:46:17.968 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.507979+0000", 2024-02-05T06:46:17.968 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:17.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.968 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:46:17.969 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.507982+0000", 2024-02-05T06:46:17.969 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5220000000000002e-06 2024-02-05T06:46:17.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.969 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:46:17.969 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.508005+0000", 2024-02-05T06:46:17.969 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3258999999999999e-05 2024-02-05T06:46:17.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.969 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:46:17.969 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.508017+0000", 2024-02-05T06:46:17.969 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1950000000000001e-05 2024-02-05T06:46:17.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.969 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:46:17.969 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.508025+0000", 2024-02-05T06:46:17.970 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9670000000000001e-06 2024-02-05T06:46:17.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.970 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.970 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.508081+0000", 2024-02-05T06:46:17.970 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6161e-05 2024-02-05T06:46:17.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.970 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:46:17.970 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:36.508092+0000", 2024-02-05T06:46:17.970 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0543999999999999e-05 2024-02-05T06:46:17.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.971 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.971 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.423176+0000", 2024-02-05T06:46:17.971 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.91508398999999996 2024-02-05T06:46:17.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.971 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:46:17.971 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.423308+0000", 2024-02-05T06:46:17.971 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000132309 2024-02-05T06:46:17.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.971 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,6", 2024-02-05T06:46:17.971 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.423451+0000", 2024-02-05T06:46:17.971 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014281100000000001 2024-02-05T06:46:17.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.971 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:46:17.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.426341+0000", 2024-02-05T06:46:17.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002890007 2024-02-05T06:46:17.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.972 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:46:17.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.432812+0000", 2024-02-05T06:46:17.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0064713119999999999 2024-02-05T06:46:17.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.972 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:46:17.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.432967+0000", 2024-02-05T06:46:17.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000154382 2024-02-05T06:46:17.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.973 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:46:17.973 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.433027+0000", 2024-02-05T06:46:17.973 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0563000000000003e-05 2024-02-05T06:46:17.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.973 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:46:17.973 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:37.433122+0000", 2024-02-05T06:46:17.973 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5000999999999994e-05 2024-02-05T06:46:17.973 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:17.973 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:46:17.973 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:17.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.974 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:106730 8.a 8:5db71e1e:::benchmark_data_smithi028_105698_object17455:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-02-05T06:46:17.974 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:41:42.382713+0000", 2024-02-05T06:46:17.974 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 275.49587285400003, 2024-02-05T06:46:17.974 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1951173160000002, 2024-02-05T06:46:17.974 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:46:17.974 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-02-05T06:46:17.974 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:46:17.974 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:46:17.974 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:46:17.974 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 106730 2024-02-05T06:46:17.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.974 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:46:17.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.974 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:46:17.974 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.382713+0000", 2024-02-05T06:46:17.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:17.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:46:17.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.382713+0000", 2024-02-05T06:46:17.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:17.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:46:17.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.382714+0000", 2024-02-05T06:46:17.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0529999999999999e-06 2024-02-05T06:46:17.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:46:17.976 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.382719+0000", 2024-02-05T06:46:17.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2420000000000002e-06 2024-02-05T06:46:17.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:46:17.976 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.382728+0000", 2024-02-05T06:46:17.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8689999999999999e-06 2024-02-05T06:46:17.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:46:17.976 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.382735+0000", 2024-02-05T06:46:17.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9539999999999998e-06 2024-02-05T06:46:17.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.977 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.977 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.387071+0000", 2024-02-05T06:46:17.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0043357619999999999 2024-02-05T06:46:17.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.977 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:46:17.977 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.387175+0000", 2024-02-05T06:46:17.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103968 2024-02-05T06:46:17.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.977 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,3", 2024-02-05T06:46:17.977 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.387287+0000", 2024-02-05T06:46:17.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000111748 2024-02-05T06:46:17.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:46:17.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.394405+0000", 2024-02-05T06:46:17.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0071176089999999996 2024-02-05T06:46:17.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.419810+0000", 2024-02-05T06:46:17.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.025405345999999999 2024-02-05T06:46:17.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:46:17.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.419824+0000", 2024-02-05T06:46:17.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4107999999999999e-05 2024-02-05T06:46:17.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.979 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.979 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.391199+0000", 2024-02-05T06:46:17.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.97137471799999997 2024-02-05T06:46:17.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.979 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:46:17.979 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.391213+0000", 2024-02-05T06:46:17.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4256e-05 2024-02-05T06:46:17.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.979 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.979 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.577761+0000", 2024-02-05T06:46:17.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1865480289999999 2024-02-05T06:46:17.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:46:17.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.577831+0000", 2024-02-05T06:46:17.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9653999999999994e-05 2024-02-05T06:46:17.980 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:17.980 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:46:17.980 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:17.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.980 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:106732 8.a 8:56d03e12:::benchmark_data_smithi028_105698_object17457:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-02-05T06:46:17.980 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:41:42.385105+0000", 2024-02-05T06:46:17.981 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 275.49348071899999, 2024-02-05T06:46:17.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1930773729999999, 2024-02-05T06:46:17.981 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:46:17.981 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-02-05T06:46:17.981 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:46:17.981 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:46:17.981 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:46:17.981 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 106732 2024-02-05T06:46:17.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.981 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:46:17.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.981 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:46:17.981 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.385105+0000", 2024-02-05T06:46:17.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:17.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.982 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:46:17.982 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.385105+0000", 2024-02-05T06:46:17.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:17.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.982 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:46:17.982 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.385107+0000", 2024-02-05T06:46:17.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9190000000000002e-06 2024-02-05T06:46:17.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.982 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:46:17.983 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.385112+0000", 2024-02-05T06:46:17.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.921e-06 2024-02-05T06:46:17.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:46:17.983 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.385124+0000", 2024-02-05T06:46:17.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1905999999999999e-05 2024-02-05T06:46:17.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:46:17.983 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.385132+0000", 2024-02-05T06:46:17.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2029999999999992e-06 2024-02-05T06:46:17.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.984 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.389919+0000", 2024-02-05T06:46:17.984 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004786312 2024-02-05T06:46:17.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.984 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:46:17.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.390050+0000", 2024-02-05T06:46:17.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000131335 2024-02-05T06:46:17.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,3", 2024-02-05T06:46:17.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.390173+0000", 2024-02-05T06:46:17.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012250999999999999 2024-02-05T06:46:17.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:46:17.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.394410+0000", 2024-02-05T06:46:17.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042371209999999999 2024-02-05T06:46:17.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.425761+0000", 2024-02-05T06:46:17.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.031351338999999999 2024-02-05T06:46:17.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:46:17.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.425780+0000", 2024-02-05T06:46:17.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9089999999999998e-05 2024-02-05T06:46:17.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.391247+0000", 2024-02-05T06:46:17.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.96546644400000003 2024-02-05T06:46:17.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:46:17.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.391268+0000", 2024-02-05T06:46:17.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1591e-05 2024-02-05T06:46:17.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.988 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.988 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.578110+0000", 2024-02-05T06:46:17.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1868418679999999 2024-02-05T06:46:17.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.988 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:46:17.988 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.578183+0000", 2024-02-05T06:46:17.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2813999999999998e-05 2024-02-05T06:46:17.988 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:17.988 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:46:17.988 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:17.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.989 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:106735 8.a 8:5d2d892e:::benchmark_data_smithi028_105698_object17460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-02-05T06:46:17.989 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:41:42.388915+0000", 2024-02-05T06:46:17.989 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 275.48967159400001, 2024-02-05T06:46:17.989 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2029950270000001, 2024-02-05T06:46:17.989 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:46:17.989 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-02-05T06:46:17.989 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:46:17.989 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:46:17.989 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:46:17.989 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 106735 2024-02-05T06:46:17.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.989 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:46:17.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.989 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:46:17.990 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.388915+0000", 2024-02-05T06:46:17.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:17.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.990 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:46:17.990 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.388915+0000", 2024-02-05T06:46:17.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:17.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.990 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:46:17.990 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.388916+0000", 2024-02-05T06:46:17.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.745e-06 2024-02-05T06:46:17.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.990 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:46:17.990 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.388922+0000", 2024-02-05T06:46:17.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2360000000000004e-06 2024-02-05T06:46:17.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:46:17.991 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.388936+0000", 2024-02-05T06:46:17.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4083000000000001e-05 2024-02-05T06:46:17.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:46:17.991 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.388943+0000", 2024-02-05T06:46:17.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.79e-06 2024-02-05T06:46:17.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.992 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.394437+0000", 2024-02-05T06:46:17.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0054937889999999998 2024-02-05T06:46:17.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.992 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:46:17.992 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.394619+0000", 2024-02-05T06:46:17.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018215700000000001 2024-02-05T06:46:17.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.992 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,3", 2024-02-05T06:46:17.992 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.394728+0000", 2024-02-05T06:46:17.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000108829 2024-02-05T06:46:17.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.425808+0000", 2024-02-05T06:46:17.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.031079828 2024-02-05T06:46:17.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:46:17.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.425816+0000", 2024-02-05T06:46:17.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8150000000000006e-06 2024-02-05T06:46:17.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.391302+0000", 2024-02-05T06:46:17.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.96548610800000001 2024-02-05T06:46:17.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:46:17.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.391312+0000", 2024-02-05T06:46:17.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8970000000000003e-06 2024-02-05T06:46:17.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.578218+0000", 2024-02-05T06:46:17.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.186906182 2024-02-05T06:46:17.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for missing object", 2024-02-05T06:46:17.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.578335+0000", 2024-02-05T06:46:17.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000116867 2024-02-05T06:46:17.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.591819+0000", 2024-02-05T06:46:17.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013484542 2024-02-05T06:46:17.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:46:17.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.591910+0000", 2024-02-05T06:46:17.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0159000000000002e-05 2024-02-05T06:46:17.995 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:17.995 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:46:17.996 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:17.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.996 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:106736 8.3 8:c62d8118:::benchmark_data_smithi028_105698_object17461:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-02-05T06:46:17.996 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:41:42.388954+0000", 2024-02-05T06:46:17.996 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 275.48963169500001, 2024-02-05T06:46:17.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.157134895, 2024-02-05T06:46:17.996 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:46:17.996 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-02-05T06:46:17.996 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:46:17.996 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:46:17.996 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:46:17.996 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 106736 2024-02-05T06:46:17.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.997 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:46:17.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.997 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:46:17.997 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.388954+0000", 2024-02-05T06:46:17.997 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:17.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.997 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:46:17.997 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.388954+0000", 2024-02-05T06:46:17.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:17.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:46:17.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.388955+0000", 2024-02-05T06:46:17.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.13e-07 2024-02-05T06:46:17.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:46:17.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.388958+0000", 2024-02-05T06:46:17.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1379999999999999e-06 2024-02-05T06:46:17.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.999 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:46:17.999 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.388964+0000", 2024-02-05T06:46:17.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.558e-06 2024-02-05T06:46:17.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.999 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:46:17.999 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.388969+0000", 2024-02-05T06:46:17.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.259e-06 2024-02-05T06:46:17.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:17.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:17.999 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:17.999 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.389977+0000", 2024-02-05T06:46:17.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001008 2024-02-05T06:46:17.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:46:18.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.390087+0000", 2024-02-05T06:46:18.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011037200000000001 2024-02-05T06:46:18.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,6", 2024-02-05T06:46:18.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.390196+0000", 2024-02-05T06:46:18.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010816599999999999 2024-02-05T06:46:18.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:46:18.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.394221+0000", 2024-02-05T06:46:18.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040258289999999999 2024-02-05T06:46:18.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:46:18.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.396962+0000", 2024-02-05T06:46:18.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027404560000000001 2024-02-05T06:46:18.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:18.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.413658+0000", 2024-02-05T06:46:18.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016696447 2024-02-05T06:46:18.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:46:18.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.413672+0000", 2024-02-05T06:46:18.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3555e-05 2024-02-05T06:46:18.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:18.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397218+0000", 2024-02-05T06:46:18.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.98354662699999995 2024-02-05T06:46:18.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:46:18.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397232+0000", 2024-02-05T06:46:18.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3247e-05 2024-02-05T06:46:18.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:18.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.546022+0000", 2024-02-05T06:46:18.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1487898000000001 2024-02-05T06:46:18.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:46:18.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.546089+0000", 2024-02-05T06:46:18.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7828000000000002e-05 2024-02-05T06:46:18.003 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:18.003 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:46:18.003 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:18.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.003 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:106738 8.a 8:5baa6da2:::benchmark_data_smithi028_105698_object17463:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-02-05T06:46:18.003 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:41:42.394882+0000", 2024-02-05T06:46:18.003 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 275.48370430199998, 2024-02-05T06:46:18.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1963950419999998, 2024-02-05T06:46:18.003 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:46:18.003 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:46:18.004 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:46:18.004 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:46:18.004 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:46:18.004 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 106738 2024-02-05T06:46:18.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.004 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:46:18.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:46:18.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.394882+0000", 2024-02-05T06:46:18.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:18.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:46:18.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.394882+0000", 2024-02-05T06:46:18.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:18.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.005 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:46:18.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.394884+0000", 2024-02-05T06:46:18.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.871e-06 2024-02-05T06:46:18.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.005 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:46:18.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.394888+0000", 2024-02-05T06:46:18.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5310000000000003e-06 2024-02-05T06:46:18.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:46:18.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.394899+0000", 2024-02-05T06:46:18.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0621e-05 2024-02-05T06:46:18.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:46:18.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.394906+0000", 2024-02-05T06:46:18.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5279999999999998e-06 2024-02-05T06:46:18.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:18.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.425852+0000", 2024-02-05T06:46:18.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.030946009 2024-02-05T06:46:18.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:46:18.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.425872+0000", 2024-02-05T06:46:18.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9262999999999998e-05 2024-02-05T06:46:18.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:18.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.391338+0000", 2024-02-05T06:46:18.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.96546637599999996 2024-02-05T06:46:18.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:46:18.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.391347+0000", 2024-02-05T06:46:18.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5089999999999993e-06 2024-02-05T06:46:18.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:18.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.578369+0000", 2024-02-05T06:46:18.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1870220869999999 2024-02-05T06:46:18.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:46:18.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.578519+0000", 2024-02-05T06:46:18.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015045999999999999 2024-02-05T06:46:18.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,3,5", 2024-02-05T06:46:18.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.578635+0000", 2024-02-05T06:46:18.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000116319 2024-02-05T06:46:18.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:46:18.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.587578+0000", 2024-02-05T06:46:18.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0089424740000000006 2024-02-05T06:46:18.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:46:18.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.587638+0000", 2024-02-05T06:46:18.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0151000000000001e-05 2024-02-05T06:46:18.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:46:18.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.587653+0000", 2024-02-05T06:46:18.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5104e-05 2024-02-05T06:46:18.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:46:18.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.590960+0000", 2024-02-05T06:46:18.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033067510000000001 2024-02-05T06:46:18.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:46:18.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.591136+0000", 2024-02-05T06:46:18.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017596800000000001 2024-02-05T06:46:18.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:46:18.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.591277+0000", 2024-02-05T06:46:18.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014102 2024-02-05T06:46:18.011 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:18.011 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:46:18.011 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:18.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.011 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:140713 8.a 8:58c58541:::benchmark_data_smithi028_105698_object51438:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-02-05T06:46:18.012 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:42:24.477844+0000", 2024-02-05T06:46:18.012 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 233.40074224599999, 2024-02-05T06:46:18.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.99289517599999999, 2024-02-05T06:46:18.012 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:46:18.012 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:46:18.012 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:46:18.012 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:46:18.012 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:46:18.012 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 140713 2024-02-05T06:46:18.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.012 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:46:18.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:46:18.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:24.477844+0000", 2024-02-05T06:46:18.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:18.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.013 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:46:18.013 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:24.477844+0000", 2024-02-05T06:46:18.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:18.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.013 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:46:18.014 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:24.477845+0000", 2024-02-05T06:46:18.014 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3230000000000001e-06 2024-02-05T06:46:18.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.014 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:46:18.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:24.477850+0000", 2024-02-05T06:46:18.015 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.865e-06 2024-02-05T06:46:18.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.015 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:46:18.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:24.477860+0000", 2024-02-05T06:46:18.015 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0010000000000001e-05 2024-02-05T06:46:18.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.015 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:46:18.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:24.477868+0000", 2024-02-05T06:46:18.015 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5730000000000002e-06 2024-02-05T06:46:18.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.016 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:18.016 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:24.477931+0000", 2024-02-05T06:46:18.016 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3156000000000003e-05 2024-02-05T06:46:18.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.016 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:46:18.016 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:24.477939+0000", 2024-02-05T06:46:18.016 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1030000000000002e-06 2024-02-05T06:46:18.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:18.017 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:25.465254+0000", 2024-02-05T06:46:18.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.98731529299999998 2024-02-05T06:46:18.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:46:18.017 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:25.465404+0000", 2024-02-05T06:46:18.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014962999999999999 2024-02-05T06:46:18.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,5", 2024-02-05T06:46:18.017 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:25.465539+0000", 2024-02-05T06:46:18.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013499700000000001 2024-02-05T06:46:18.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.018 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:46:18.018 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:25.470640+0000", 2024-02-05T06:46:18.018 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051007530000000004 2024-02-05T06:46:18.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.018 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:46:18.018 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:25.470654+0000", 2024-02-05T06:46:18.018 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4647999999999999e-05 2024-02-05T06:46:18.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.018 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:46:18.019 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:25.470676+0000", 2024-02-05T06:46:18.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1790999999999999e-05 2024-02-05T06:46:18.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.019 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:46:18.019 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:25.470705+0000", 2024-02-05T06:46:18.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9295e-05 2024-02-05T06:46:18.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.019 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:46:18.019 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:25.470739+0000", 2024-02-05T06:46:18.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3738999999999997e-05 2024-02-05T06:46:18.019 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:18.020 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:46:18.020 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:18.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.020 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:140714 8.a 8:529663f4:::benchmark_data_smithi028_105698_object51439:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-02-05T06:46:18.020 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:42:24.480177+0000", 2024-02-05T06:46:18.020 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 233.398409265, 2024-02-05T06:46:18.020 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.99080147900000004, 2024-02-05T06:46:18.020 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:46:18.020 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:46:18.020 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:46:18.020 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:46:18.021 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:46:18.021 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 140714 2024-02-05T06:46:18.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.021 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:46:18.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.021 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:46:18.021 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:24.480177+0000", 2024-02-05T06:46:18.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:18.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.021 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:46:18.021 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:24.480177+0000", 2024-02-05T06:46:18.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:18.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.022 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:46:18.022 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:24.480178+0000", 2024-02-05T06:46:18.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.491e-06 2024-02-05T06:46:18.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.022 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:46:18.022 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:24.480183+0000", 2024-02-05T06:46:18.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9189999999999998e-06 2024-02-05T06:46:18.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.022 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:46:18.022 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:24.480194+0000", 2024-02-05T06:46:18.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0475000000000001e-05 2024-02-05T06:46:18.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:46:18.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:24.480202+0000", 2024-02-05T06:46:18.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8660000000000006e-06 2024-02-05T06:46:18.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:18.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:24.480266+0000", 2024-02-05T06:46:18.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4072999999999995e-05 2024-02-05T06:46:18.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:46:18.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:24.480275+0000", 2024-02-05T06:46:18.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5119999999999997e-06 2024-02-05T06:46:18.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.024 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:18.024 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:25.467725+0000", 2024-02-05T06:46:18.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.98744968899999996 2024-02-05T06:46:18.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.024 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:46:18.024 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:25.467829+0000", 2024-02-05T06:46:18.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010427799999999999 2024-02-05T06:46:18.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.024 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,5", 2024-02-05T06:46:18.024 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:25.467932+0000", 2024-02-05T06:46:18.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000102618 2024-02-05T06:46:18.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.025 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:46:18.025 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:25.470861+0000", 2024-02-05T06:46:18.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00292873 2024-02-05T06:46:18.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.025 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:46:18.025 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:25.470904+0000", 2024-02-05T06:46:18.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3032999999999999e-05 2024-02-05T06:46:18.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.025 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:46:18.025 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:25.470912+0000", 2024-02-05T06:46:18.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1100000000000003e-06 2024-02-05T06:46:18.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.026 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:46:18.026 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:25.470949+0000", 2024-02-05T06:46:18.026 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7741999999999999e-05 2024-02-05T06:46:18.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.026 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:46:18.026 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:25.470978+0000", 2024-02-05T06:46:18.026 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8943000000000001e-05 2024-02-05T06:46:18.026 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:18.026 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:46:18.026 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:18.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.026 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:140737 8.a 8:526b9de6:::benchmark_data_smithi028_105698_object51462:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e268)", 2024-02-05T06:46:18.026 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:42:24.526230+0000", 2024-02-05T06:46:18.027 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 233.352356558, 2024-02-05T06:46:18.027 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.94493433199999999, 2024-02-05T06:46:18.027 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:46:18.027 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:46:18.027 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:46:18.027 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:46:18.027 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:46:18.027 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 140737 2024-02-05T06:46:18.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.027 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:46:18.027 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.027 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:46:18.027 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:24.526230+0000", 2024-02-05T06:46:18.027 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:18.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.027 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.028 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:46:18.028 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:24.526230+0000", 2024-02-05T06:46:18.028 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:46:18.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.028 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.028 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:46:18.028 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:24.526231+0000", 2024-02-05T06:46:18.028 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.508e-06 2024-02-05T06:46:18.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.028 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.028 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:46:18.028 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:24.526236+0000", 2024-02-05T06:46:18.028 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4460000000000003e-06 2024-02-05T06:46:18.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.028 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.029 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:46:18.029 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:24.526246+0000", 2024-02-05T06:46:18.029 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0682e-05 2024-02-05T06:46:18.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.029 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:46:18.029 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:24.526254+0000", 2024-02-05T06:46:18.029 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5320000000000001e-06 2024-02-05T06:46:18.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.029 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:18.029 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:24.526477+0000", 2024-02-05T06:46:18.029 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00022293599999999999 2024-02-05T06:46:18.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.029 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:46:18.030 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:24.526485+0000", 2024-02-05T06:46:18.030 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9659999999999996e-06 2024-02-05T06:46:18.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.030 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:46:18.030 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:25.469356+0000", 2024-02-05T06:46:18.030 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.94287162000000002 2024-02-05T06:46:18.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.030 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:46:18.030 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:25.469445+0000", 2024-02-05T06:46:18.030 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8307999999999995e-05 2024-02-05T06:46:18.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.030 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,5", 2024-02-05T06:46:18.030 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:25.469505+0000", 2024-02-05T06:46:18.030 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0362999999999998e-05 2024-02-05T06:46:18.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.031 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:46:18.031 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:25.470985+0000", 2024-02-05T06:46:18.031 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0014799870000000001 2024-02-05T06:46:18.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.031 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:46:18.031 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:25.471035+0000", 2024-02-05T06:46:18.031 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9682000000000003e-05 2024-02-05T06:46:18.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.031 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:46:18.031 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:25.471087+0000", 2024-02-05T06:46:18.031 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2787999999999997e-05 2024-02-05T06:46:18.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.032 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:46:18.032 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:25.471133+0000", 2024-02-05T06:46:18.032 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5572999999999999e-05 2024-02-05T06:46:18.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:46:18.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:46:18.032 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:46:18.032 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:25.471164+0000", 2024-02-05T06:46:18.032 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0941000000000001e-05 2024-02-05T06:46:18.032 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:18.032 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:46:18.032 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:18.032 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:46:18.032 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:46:18.032 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:46:18.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:18.033 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:18.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:18.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:17 smithi028 ceph-mon[29442]: pgmap v613: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-05T06:46:18.034 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:17 smithi028 ceph-mon[37984]: pgmap v613: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-05T06:46:18.134 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:18.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:18.170 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:18.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:18.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:17 smithi167 ceph-mon[32774]: pgmap v613: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-05T06:46:18.271 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:18.311 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:18.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:18.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:18.414 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:18.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:18.452 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:18.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:18.553 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:18.594 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:18.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:18.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:18.696 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:18.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:18.728 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:18.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:18.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:18.892 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:18.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:18.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:18.994 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:19.029 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:19.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:19.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:19.131 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:19.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:19.166 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:19.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:19.267 INFO:teuthology.orchestra.run.smithi028.stdout:moved 11 / 474 (2.32068%) 2024-02-05T06:46:19.267 INFO:teuthology.orchestra.run.smithi028.stdout:avg 59.25 2024-02-05T06:46:19.267 INFO:teuthology.orchestra.run.smithi028.stdout:stddev 6.75925 -> 6.70354 (expected baseline 7.20026) 2024-02-05T06:46:19.267 INFO:teuthology.orchestra.run.smithi028.stdout:min osd.2 with 48 -> 49 pgs (0.810127 -> 0.827004 * mean) 2024-02-05T06:46:19.267 INFO:teuthology.orchestra.run.smithi028.stdout:max osd.3 with 68 -> 68 pgs (1.14768 -> 1.14768 * mean) 2024-02-05T06:46:19.267 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:46:19.267 INFO:teuthology.orchestra.run.smithi028.stdout:oload 110 2024-02-05T06:46:19.267 INFO:teuthology.orchestra.run.smithi028.stdout:max_change 0.05 2024-02-05T06:46:19.268 INFO:teuthology.orchestra.run.smithi028.stdout:max_change_osds 4 2024-02-05T06:46:19.268 INFO:teuthology.orchestra.run.smithi028.stdout:average_utilization 0.0183 2024-02-05T06:46:19.268 INFO:teuthology.orchestra.run.smithi028.stdout:overload_utilization 0.0202 2024-02-05T06:46:19.268 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 weight 1.0000 -> 0.9500 2024-02-05T06:46:19.268 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 weight 1.0000 -> 0.9500 2024-02-05T06:46:19.268 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 weight 1.0000 -> 0.9500 2024-02-05T06:46:19.268 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 weight 1.0000 -> 0.9500 2024-02-05T06:46:19.268 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:19.274 INFO:teuthology.orchestra.run.smithi028.stderr:no change 2024-02-05T06:46:19.305 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:19.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:19.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:19.406 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:19.429 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:19.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:19.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:19.449 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:46:19 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:46:19.258+0000 7f652db03700 -1 mgr.server reply reply (4) Interrupted system call no change 2024-02-05T06:46:19.531 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:19.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:19.565 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:19.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:19.666 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:19.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:19.712 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:19.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:19.813 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:19.849 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:19.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:19.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:19.866 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-05T06:46:19.951 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:19.984 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:19.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:19.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:20.015 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:46:20.086 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:20.092 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:46:20.121 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:20.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:20.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:20.166 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:19 smithi028 ceph-mon[29442]: pgmap v614: 158 pgs: 158 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-05T06:46:20.168 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:19 smithi028 ceph-mon[37984]: pgmap v614: 158 pgs: 158 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-05T06:46:20.223 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:20.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:19 smithi167 ceph-mon[32774]: pgmap v614: 158 pgs: 158 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-05T06:46:20.260 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:20.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:20.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:20.362 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:20.399 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:20.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:20.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:20.501 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:20.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:20.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:20.659 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:20.700 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:20.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:20.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:20.802 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:20.837 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:20.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:20.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:20.940 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:20.980 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:20.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:20.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:21.082 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:21.115 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:21.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:21.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:21.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:20 smithi028 ceph-mon[37984]: from='client.15573 v1:172.21.15.28:0/4241700313' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:46:21.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:20 smithi028 ceph-mon[29442]: from='client.15573 v1:172.21.15.28:0/4241700313' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:46:21.217 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:21.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:20 smithi167 ceph-mon[32774]: from='client.15573 v1:172.21.15.28:0/4241700313' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-02-05T06:46:21.255 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:21.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:21.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:21.357 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:21.392 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:21.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:21.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:21.494 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:21.529 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:21.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:21.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:21.631 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:21.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:21.667 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:21.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:21.768 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:21.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:21.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:21.909 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:21.938 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:21.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:21.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:22.040 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:22.076 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:22.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:22.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:22.178 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:22.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:21 smithi028 ceph-mon[37984]: pgmap v615: 158 pgs: 158 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-05T06:46:22.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:21 smithi028 ceph-mon[29442]: pgmap v615: 158 pgs: 158 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-05T06:46:22.212 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:22.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:22.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:22.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:21 smithi167 ceph-mon[32774]: pgmap v615: 158 pgs: 158 active+clean; 3.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-05T06:46:22.314 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:22.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:22.349 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:22.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:22.450 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:22.476 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:22.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:22.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:22.578 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:22.616 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:22.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:22.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:22.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:22.756 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:22.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:22.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:22.858 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:22.897 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:22.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:22.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:22.949 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:46:22 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:46:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:46:22.999 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:23.034 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:23.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:23.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:23.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:23.170 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:23.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:23.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:23.272 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:23.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:23.311 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:23.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:23.412 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:23.450 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:23.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:23.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:23.484 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:46:23 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:46:23.551 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:23.588 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:23.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:23.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:23.690 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:23.713 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:23.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:23.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:23.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:23.855 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:23.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:23.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:23.957 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:23.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:23.994 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:23.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:24.095 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:24.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:24.130 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:24.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:24.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:23 smithi028 ceph-mon[29442]: pgmap v616: 158 pgs: 158 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-02-05T06:46:24.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:23 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:46:24.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:23 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:46:24.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:23 smithi028 ceph-mon[29442]: osdmap e330: 8 total, 8 up, 8 in 2024-02-05T06:46:24.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:23 smithi028 ceph-mon[37984]: pgmap v616: 158 pgs: 158 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-02-05T06:46:24.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:23 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:46:24.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:23 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:46:24.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:23 smithi028 ceph-mon[37984]: osdmap e330: 8 total, 8 up, 8 in 2024-02-05T06:46:24.231 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:24.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:23 smithi167 ceph-mon[32774]: pgmap v616: 158 pgs: 158 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-02-05T06:46:24.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:23 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:46:24.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:23 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:46:24.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:23 smithi167 ceph-mon[32774]: osdmap e330: 8 total, 8 up, 8 in 2024-02-05T06:46:24.272 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:24.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:24.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:24.374 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:24.408 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:24.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:24.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:24.510 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:24.545 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:24.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:24.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:24.647 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:24.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:24.691 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:24.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:24.792 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:24.824 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:24.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:24.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:24.926 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:24.961 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:24.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:24.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:25.063 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:25.098 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:25.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:25.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:25.199 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:25.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:24 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:46:25.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:24 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:46:25.233 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:46:25.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:24 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:46:25.249 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:25.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:25.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:25.351 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:25.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:25.406 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:25.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:25.507 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:25.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:25.547 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:25.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:25.648 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:25.678 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:25.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:25.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:25.780 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:25.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:25.827 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:25.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:25.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:25.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:25 smithi028 ceph-mon[37984]: pgmap v618: 158 pgs: 158 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-02-05T06:46:25.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:25 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:46:25.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:25 smithi028 ceph-mon[29442]: pgmap v618: 158 pgs: 158 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-02-05T06:46:25.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:25 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:46:25.971 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:25.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:25.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:26.073 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:26.119 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:26.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:26.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:26.221 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:26.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:25 smithi167 ceph-mon[32774]: pgmap v618: 158 pgs: 158 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-02-05T06:46:26.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:25 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:46:26.260 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:26.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:26.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:26.362 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:26.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:26.398 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:26.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:26.499 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:26.544 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:26.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:26.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:26.645 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:26.677 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:26.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:26.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:26.779 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:26.812 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:26.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:26.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:26.852 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:46:26.892 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 4, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T06:46:26.893 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T06:46:26.893 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T06:46:26.903 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:46:26.914 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:26.959 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:46:26.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:26.960 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:26.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:27.009 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:46:27.055 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:46:27.061 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:27.101 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:46:27.102 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:26 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:46:27.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:26 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:46:27.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:26 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:46:27.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:26 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2613703611' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:46:27.103 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:26 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:46:27.103 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:26 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:46:27.103 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:26 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:46:27.103 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:26 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:46:27.103 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:26 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2613703611' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:46:27.103 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:26 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:46:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:27.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:27.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:27.144 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:46:27.185 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:46:27.204 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:46:27.204 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:46:27.204 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:46:27.204 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:46:27.204 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:46:27.204 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:46:27.204 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:46:27.204 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:46:27.223 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:27.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:26 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:46:27.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:26 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:46:27.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:26 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:46:27.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:26 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2613703611' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:46:27.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:26 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:46:27.288 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:27.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:27.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:27.390 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:27.428 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:27.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:27.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:27.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:27.571 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:27.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:27.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:27.672 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:27.709 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:27.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:27.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:27.810 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:27.843 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:27.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:27.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:27.914 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-02-05T06:46:27.945 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:27.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:27.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:28.089 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:28.124 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:28.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:28.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:28.127 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:27 smithi028 ceph-mon[29442]: pgmap v619: 158 pgs: 158 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-02-05T06:46:28.127 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:27 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/476945599' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:46:28.127 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:27 smithi028 ceph-mon[29442]: from='client.25489 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:46:28.127 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:27 smithi028 ceph-mon[37984]: pgmap v619: 158 pgs: 158 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-02-05T06:46:28.127 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:27 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/476945599' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:46:28.127 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:27 smithi028 ceph-mon[37984]: from='client.25489 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:46:28.227 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:28.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:27 smithi167 ceph-mon[32774]: pgmap v619: 158 pgs: 158 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s 2024-02-05T06:46:28.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:27 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/476945599' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:46:28.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:27 smithi167 ceph-mon[32774]: from='client.25489 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:46:28.259 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:28.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:28.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:28.361 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:28.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:28.402 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:28.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:28.503 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:28.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:28.542 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:28.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:28.643 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:28.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:28.677 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:28.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:28.778 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:28.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:28.815 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:28.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:28.916 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:28.951 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:28.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:28.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:29.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:29.089 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:29.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:29.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:29.191 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:29.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:28 smithi028 ceph-mon[29442]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:46:29.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:28 smithi028 ceph-mon[29442]: from='client.25489 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:46:29.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:28 smithi028 ceph-mon[29442]: osdmap e331: 8 total, 8 up, 8 in 2024-02-05T06:46:29.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:28 smithi028 ceph-mon[37984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:46:29.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:28 smithi028 ceph-mon[37984]: from='client.25489 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:46:29.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:28 smithi028 ceph-mon[37984]: osdmap e331: 8 total, 8 up, 8 in 2024-02-05T06:46:29.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:29.228 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:29.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:29.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:28 smithi167 ceph-mon[32774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:46:29.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:28 smithi167 ceph-mon[32774]: from='client.25489 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:46:29.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:28 smithi167 ceph-mon[32774]: osdmap e331: 8 total, 8 up, 8 in 2024-02-05T06:46:29.330 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:29.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:29.369 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:29.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:29.469 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:29.505 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:29.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:29.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:29.607 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:29.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:29.645 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:29.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:29.746 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:29.779 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:29.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:29.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:29.880 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:29.917 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:29.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:29.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:30.019 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:30.056 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:30.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:30.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:30.158 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:30.175 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:46:30.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:30.196 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:30.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:30.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:29 smithi028 ceph-mon[37984]: pgmap v621: 158 pgs: 158 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s 2024-02-05T06:46:30.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:29 smithi028 ceph-mon[29442]: pgmap v621: 158 pgs: 158 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s 2024-02-05T06:46:30.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:29 smithi167 ceph-mon[32774]: pgmap v621: 158 pgs: 158 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s 2024-02-05T06:46:30.272 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:46:30.298 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:30.327 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:30.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:30.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:30.429 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:30.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:30.488 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:30.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:30.589 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:30.621 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:30.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:30.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:30.723 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:30.755 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:30.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:30.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:30.857 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:30.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:30.898 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:30.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:30.967 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:30 smithi028 ceph-mon[29442]: 4.a scrub starts 2024-02-05T06:46:30.967 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:30 smithi028 ceph-mon[29442]: 4.a scrub ok 2024-02-05T06:46:30.967 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:30 smithi028 ceph-mon[29442]: 3.15 scrub starts 2024-02-05T06:46:30.967 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:30 smithi028 ceph-mon[29442]: 3.15 scrub ok 2024-02-05T06:46:30.967 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:30 smithi028 ceph-mon[29442]: 5.1 scrub starts 2024-02-05T06:46:30.967 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:30 smithi028 ceph-mon[29442]: 5.1 scrub ok 2024-02-05T06:46:30.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:30 smithi028 ceph-mon[37984]: 4.a scrub starts 2024-02-05T06:46:30.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:30 smithi028 ceph-mon[37984]: 4.a scrub ok 2024-02-05T06:46:30.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:30 smithi028 ceph-mon[37984]: 3.15 scrub starts 2024-02-05T06:46:30.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:30 smithi028 ceph-mon[37984]: 3.15 scrub ok 2024-02-05T06:46:30.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:30 smithi028 ceph-mon[37984]: 5.1 scrub starts 2024-02-05T06:46:30.968 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:30 smithi028 ceph-mon[37984]: 5.1 scrub ok 2024-02-05T06:46:30.999 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:31.038 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:31.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:31.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:31.141 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:31.173 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:31.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:31.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:31.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:30 smithi167 ceph-mon[32774]: 4.a scrub starts 2024-02-05T06:46:31.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:30 smithi167 ceph-mon[32774]: 4.a scrub ok 2024-02-05T06:46:31.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:30 smithi167 ceph-mon[32774]: 3.15 scrub starts 2024-02-05T06:46:31.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:30 smithi167 ceph-mon[32774]: 3.15 scrub ok 2024-02-05T06:46:31.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:30 smithi167 ceph-mon[32774]: 5.1 scrub starts 2024-02-05T06:46:31.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:30 smithi167 ceph-mon[32774]: 5.1 scrub ok 2024-02-05T06:46:31.275 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:31.310 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:31.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:31.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:31.412 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:31.445 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:31.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:31.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:31.546 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:31.588 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:31.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:31.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:31.633 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:46:31.633 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:46:31.690 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:31.723 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:31.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:31.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:31.757 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T06:46:31.825 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:31.851 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:31.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:31.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:31.952 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:31.988 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:31.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:31.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:32.091 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:32.124 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:32.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:32.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:32.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:31 smithi028 ceph-mon[29442]: pgmap v622: 158 pgs: 158 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 967 op/s 2024-02-05T06:46:32.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:31 smithi028 ceph-mon[29442]: 5.2 scrub starts 2024-02-05T06:46:32.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:31 smithi028 ceph-mon[29442]: 5.2 scrub ok 2024-02-05T06:46:32.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:31 smithi028 ceph-mon[29442]: 3.4 scrub starts 2024-02-05T06:46:32.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:31 smithi028 ceph-mon[29442]: 3.4 scrub ok 2024-02-05T06:46:32.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:31 smithi028 ceph-mon[29442]: 3.17 scrub starts 2024-02-05T06:46:32.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:31 smithi028 ceph-mon[29442]: 3.17 scrub ok 2024-02-05T06:46:32.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:31 smithi028 ceph-mon[29442]: 9.7 scrub starts 2024-02-05T06:46:32.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:31 smithi028 ceph-mon[37984]: pgmap v622: 158 pgs: 158 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 967 op/s 2024-02-05T06:46:32.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:31 smithi028 ceph-mon[37984]: 5.2 scrub starts 2024-02-05T06:46:32.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:31 smithi028 ceph-mon[37984]: 5.2 scrub ok 2024-02-05T06:46:32.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:31 smithi028 ceph-mon[37984]: 3.4 scrub starts 2024-02-05T06:46:32.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:31 smithi028 ceph-mon[37984]: 3.4 scrub ok 2024-02-05T06:46:32.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:31 smithi028 ceph-mon[37984]: 3.17 scrub starts 2024-02-05T06:46:32.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:31 smithi028 ceph-mon[37984]: 3.17 scrub ok 2024-02-05T06:46:32.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:31 smithi028 ceph-mon[37984]: 9.7 scrub starts 2024-02-05T06:46:32.226 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:32.236 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:31 smithi167 ceph-mon[32774]: pgmap v622: 158 pgs: 158 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 967 op/s 2024-02-05T06:46:32.236 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:31 smithi167 ceph-mon[32774]: 5.2 scrub starts 2024-02-05T06:46:32.236 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:31 smithi167 ceph-mon[32774]: 5.2 scrub ok 2024-02-05T06:46:32.236 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:31 smithi167 ceph-mon[32774]: 3.4 scrub starts 2024-02-05T06:46:32.236 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:31 smithi167 ceph-mon[32774]: 3.4 scrub ok 2024-02-05T06:46:32.236 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:31 smithi167 ceph-mon[32774]: 3.17 scrub starts 2024-02-05T06:46:32.236 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:31 smithi167 ceph-mon[32774]: 3.17 scrub ok 2024-02-05T06:46:32.236 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:31 smithi167 ceph-mon[32774]: 9.7 scrub starts 2024-02-05T06:46:32.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:32.261 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:32.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:32.362 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:32.401 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:32.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:32.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:32.503 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:32.546 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:32.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:32.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:32.647 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:32.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:32.684 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:32.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:32.702 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:46:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:46:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:46:32.784 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:32.817 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:32.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:32.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:32.919 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:32.949 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:32.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:32.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:33.051 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:33.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:32 smithi028 ceph-mon[29442]: 6.4 scrub starts 2024-02-05T06:46:33.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:32 smithi028 ceph-mon[29442]: 6.4 scrub ok 2024-02-05T06:46:33.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:32 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:46:33.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:32 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:46:33.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:32 smithi028 ceph-mon[29442]: 9.7 scrub ok 2024-02-05T06:46:33.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:32 smithi028 ceph-mon[29442]: osdmap e332: 8 total, 8 up, 8 in 2024-02-05T06:46:33.063 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:32 smithi028 ceph-mon[37984]: 6.4 scrub starts 2024-02-05T06:46:33.063 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:32 smithi028 ceph-mon[37984]: 6.4 scrub ok 2024-02-05T06:46:33.063 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:32 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:46:33.063 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:32 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:46:33.064 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:32 smithi028 ceph-mon[37984]: 9.7 scrub ok 2024-02-05T06:46:33.064 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:32 smithi028 ceph-mon[37984]: osdmap e332: 8 total, 8 up, 8 in 2024-02-05T06:46:33.094 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:33.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:33.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:33.196 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:33.229 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:33.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:33.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:33.234 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:46:33 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:46:33.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:32 smithi167 ceph-mon[32774]: 6.4 scrub starts 2024-02-05T06:46:33.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:32 smithi167 ceph-mon[32774]: 6.4 scrub ok 2024-02-05T06:46:33.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:32 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:46:33.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:32 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:46:33.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:32 smithi167 ceph-mon[32774]: 9.7 scrub ok 2024-02-05T06:46:33.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:32 smithi167 ceph-mon[32774]: osdmap e332: 8 total, 8 up, 8 in 2024-02-05T06:46:33.331 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:33.365 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:33.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:33.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:33.467 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:33.496 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:33.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:33.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:33.502 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-05T06:46:33.598 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:33.650 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:33.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:33.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:33.752 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:33.782 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:33.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:33.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:33.884 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:33.913 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:33.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:33.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:34.015 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:34.035 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:46:34.042 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:46:34.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:34.094 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:34.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:34.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:33 smithi028 ceph-mon[29442]: pgmap v624: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.1 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-02-05T06:46:34.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:33 smithi028 ceph-mon[29442]: 3.0 scrub starts 2024-02-05T06:46:34.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:33 smithi028 ceph-mon[29442]: 3.0 scrub ok 2024-02-05T06:46:34.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:33 smithi028 ceph-mon[29442]: 3.f scrub starts 2024-02-05T06:46:34.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:33 smithi028 ceph-mon[29442]: 3.f scrub ok 2024-02-05T06:46:34.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:33 smithi028 ceph-mon[29442]: 6.9 scrub starts 2024-02-05T06:46:34.188 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:33 smithi028 ceph-mon[29442]: 6.9 scrub ok 2024-02-05T06:46:34.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:33 smithi028 ceph-mon[37984]: pgmap v624: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.1 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-02-05T06:46:34.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:33 smithi028 ceph-mon[37984]: 3.0 scrub starts 2024-02-05T06:46:34.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:33 smithi028 ceph-mon[37984]: 3.0 scrub ok 2024-02-05T06:46:34.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:33 smithi028 ceph-mon[37984]: 3.f scrub starts 2024-02-05T06:46:34.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:33 smithi028 ceph-mon[37984]: 3.f scrub ok 2024-02-05T06:46:34.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:33 smithi028 ceph-mon[37984]: 6.9 scrub starts 2024-02-05T06:46:34.189 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:33 smithi028 ceph-mon[37984]: 6.9 scrub ok 2024-02-05T06:46:34.195 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:34.231 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:34.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:34.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:34.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:33 smithi167 ceph-mon[32774]: pgmap v624: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.1 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-02-05T06:46:34.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:33 smithi167 ceph-mon[32774]: 3.0 scrub starts 2024-02-05T06:46:34.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:33 smithi167 ceph-mon[32774]: 3.0 scrub ok 2024-02-05T06:46:34.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:33 smithi167 ceph-mon[32774]: 3.f scrub starts 2024-02-05T06:46:34.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:33 smithi167 ceph-mon[32774]: 3.f scrub ok 2024-02-05T06:46:34.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:33 smithi167 ceph-mon[32774]: 6.9 scrub starts 2024-02-05T06:46:34.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:33 smithi167 ceph-mon[32774]: 6.9 scrub ok 2024-02-05T06:46:34.331 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:46:34.336 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:34.344 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-05T06:46:34.362 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:34.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:34.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:34.464 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:34.502 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:34.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:34.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:34.604 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:34.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:34.635 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:34.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:34.736 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:34.771 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:34.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:34.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:34.873 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:34.909 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:34.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:34.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:35.011 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:35.052 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:35.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:35.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:35.154 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:35.194 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:35.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:35.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:35.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:34 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:46:35.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:34 smithi028 ceph-mon[29442]: 5.a scrub starts 2024-02-05T06:46:35.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:34 smithi028 ceph-mon[29442]: 5.a scrub ok 2024-02-05T06:46:35.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:34 smithi028 ceph-mon[29442]: 1.0 scrub starts 2024-02-05T06:46:35.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:34 smithi028 ceph-mon[29442]: 1.0 scrub ok 2024-02-05T06:46:35.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:34 smithi028 ceph-mon[29442]: 4.12 scrub starts 2024-02-05T06:46:35.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:34 smithi028 ceph-mon[29442]: 4.12 scrub ok 2024-02-05T06:46:35.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:34 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:46:35.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:34 smithi028 ceph-mon[37984]: 5.a scrub starts 2024-02-05T06:46:35.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:34 smithi028 ceph-mon[37984]: 5.a scrub ok 2024-02-05T06:46:35.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:34 smithi028 ceph-mon[37984]: 1.0 scrub starts 2024-02-05T06:46:35.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:34 smithi028 ceph-mon[37984]: 1.0 scrub ok 2024-02-05T06:46:35.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:34 smithi028 ceph-mon[37984]: 4.12 scrub starts 2024-02-05T06:46:35.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:34 smithi028 ceph-mon[37984]: 4.12 scrub ok 2024-02-05T06:46:35.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:34 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:46:35.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:34 smithi167 ceph-mon[32774]: 5.a scrub starts 2024-02-05T06:46:35.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:34 smithi167 ceph-mon[32774]: 5.a scrub ok 2024-02-05T06:46:35.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:34 smithi167 ceph-mon[32774]: 1.0 scrub starts 2024-02-05T06:46:35.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:34 smithi167 ceph-mon[32774]: 1.0 scrub ok 2024-02-05T06:46:35.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:34 smithi167 ceph-mon[32774]: 4.12 scrub starts 2024-02-05T06:46:35.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:34 smithi167 ceph-mon[32774]: 4.12 scrub ok 2024-02-05T06:46:35.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:35.332 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:35.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:35.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:35.434 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:35.472 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:35.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:35.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:35.574 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:35.609 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:35.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:35.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:35.711 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:35.748 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:35.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:35.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:35.809 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:46:35.812 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":624,"stamp":"2024-02-05T06:46:32.423422+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2293569908,"num_objects":35187,"num_object_clones":0,"num_object_copies":105561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35187,"num_whiteouts":0,"num_read":1491,"num_read_kb":1243,"num_write":225329,"num_write_kb":8307054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":415,"num_bytes_recovered":1586799,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80249,"ondisk_log_size":80249,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":10326684,"kb_used_data":6789596,"kb_used_omap":12,"kb_used_meta":3536819,"kb_avail":739634532,"statfs":{"total":767960285184,"available":757385760768,"internally_reserved":0,"allocated":6952546304,"data_stored":6949494506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":3621703217},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-725680128,"num_objects":-11073,"num_object_clones":0,"num_object_copies":-33219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11073,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11073,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001879"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":331,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.903155+0000","last_change":"2024-02-05T06:45:33.131808+0000","last_active":"2024-02-05T06:46:27.903155+0000","last_peered":"2024-02-05T06:46:27.903155+0000","last_clean":"2024-02-05T06:46:27.903155+0000","last_became_active":"2024-02-05T06:45:09.787134+0000","last_became_peered":"2024-02-05T06:45:09.787134+0000","last_unstale":"2024-02-05T06:46:27.903155+0000","last_undegraded":"2024-02-05T06:46:27.903155+0000","last_fullsized":"2024-02-05T06:46:27.903155+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:33.131697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:45:33.131697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:36.740059+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":601,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.900699+0000","last_change":"2024-02-05T06:45:44.761855+0000","last_active":"2024-02-05T06:46:27.900699+0000","last_peered":"2024-02-05T06:46:27.900699+0000","last_clean":"2024-02-05T06:46:27.900699+0000","last_became_active":"2024-02-05T06:41:43.420342+0000","last_became_peered":"2024-02-05T06:41:43.420342+0000","last_unstale":"2024-02-05T06:46:27.900699+0000","last_undegraded":"2024-02-05T06:46:27.900699+0000","last_fullsized":"2024-02-05T06:46:27.900699+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:44.761725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:06.525012+0000","last_clean_scrub_stamp":"2024-02-05T06:45:44.761725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:47:03.089622+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":593,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-05T06:46:32.168562+0000","last_change":"2024-02-05T06:45:42.923556+0000","last_active":"2024-02-05T06:46:32.168562+0000","last_peered":"2024-02-05T06:46:32.168562+0000","last_clean":"2024-02-05T06:46:32.168562+0000","last_became_active":"2024-02-05T06:41:43.421315+0000","last_became_peered":"2024-02-05T06:41:43.421315+0000","last_unstale":"2024-02-05T06:46:32.168562+0000","last_undegraded":"2024-02-05T06:46:32.168562+0000","last_fullsized":"2024-02-05T06:46:32.168562+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:42.923435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:45:42.923435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:51.403604+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"295'19","reported_seq":493,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.904861+0000","last_change":"2024-02-05T06:45:39.258979+0000","last_active":"2024-02-05T06:46:27.904861+0000","last_peered":"2024-02-05T06:46:27.904861+0000","last_clean":"2024-02-05T06:46:27.904861+0000","last_became_active":"2024-02-05T06:36:54.959330+0000","last_became_peered":"2024-02-05T06:36:54.959330+0000","last_unstale":"2024-02-05T06:46:27.904861+0000","last_undegraded":"2024-02-05T06:46:27.904861+0000","last_fullsized":"2024-02-05T06:46:27.904861+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'19","last_scrub_stamp":"2024-02-05T06:45:39.258744+0000","last_deep_scrub":"247'13","last_deep_scrub_stamp":"2024-02-05T06:41:42.473424+0000","last_clean_scrub_stamp":"2024-02-05T06:45:39.258744+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:48.502551+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":575,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:22.880886+0000","last_change":"2024-02-05T06:45:45.252029+0000","last_active":"2024-02-05T06:46:22.880886+0000","last_peered":"2024-02-05T06:46:22.880886+0000","last_clean":"2024-02-05T06:46:22.880886+0000","last_became_active":"2024-02-05T06:40:27.077485+0000","last_became_peered":"2024-02-05T06:40:27.077485+0000","last_unstale":"2024-02-05T06:46:22.880886+0000","last_undegraded":"2024-02-05T06:46:22.880886+0000","last_fullsized":"2024-02-05T06:46:22.880886+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:45.251895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:45:45.251895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:50.044182+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":91,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.897490+0000","last_change":"2024-02-05T06:45:09.775379+0000","last_active":"2024-02-05T06:46:27.897490+0000","last_peered":"2024-02-05T06:46:27.897490+0000","last_clean":"2024-02-05T06:46:27.897490+0000","last_became_active":"2024-02-05T06:45:09.775037+0000","last_became_peered":"2024-02-05T06:45:09.775037+0000","last_unstale":"2024-02-05T06:46:27.897490+0000","last_undegraded":"2024-02-05T06:46:27.897490+0000","last_fullsized":"2024-02-05T06:46:27.897490+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:51.070907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:44:51.070907+0000","last_clean_scrub_stamp":"2024-02-05T06:44:51.070907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":576,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:22.881314+0000","last_change":"2024-02-05T06:43:53.697132+0000","last_active":"2024-02-05T06:46:22.881314+0000","last_peered":"2024-02-05T06:46:22.881314+0000","last_clean":"2024-02-05T06:46:22.881314+0000","last_became_active":"2024-02-05T06:33:29.553864+0000","last_became_peered":"2024-02-05T06:33:29.553864+0000","last_unstale":"2024-02-05T06:46:22.881314+0000","last_undegraded":"2024-02-05T06:46:22.881314+0000","last_fullsized":"2024-02-05T06:46:22.881314+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:53.696997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:43:53.696997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:24.289040+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"295'34","reported_seq":621,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:22.883149+0000","last_change":"2024-02-05T06:43:56.716084+0000","last_active":"2024-02-05T06:46:22.883149+0000","last_peered":"2024-02-05T06:46:22.883149+0000","last_clean":"2024-02-05T06:46:22.883149+0000","last_became_active":"2024-02-05T06:33:31.572544+0000","last_became_peered":"2024-02-05T06:33:31.572544+0000","last_unstale":"2024-02-05T06:46:22.883149+0000","last_undegraded":"2024-02-05T06:46:22.883149+0000","last_fullsized":"2024-02-05T06:46:22.883149+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'34","last_scrub_stamp":"2024-02-05T06:43:56.715959+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:39:35.433539+0000","last_clean_scrub_stamp":"2024-02-05T06:43:56.715959+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:24.289177+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":580,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-05T06:46:32.174007+0000","last_change":"2024-02-05T06:44:55.126040+0000","last_active":"2024-02-05T06:46:32.174007+0000","last_peered":"2024-02-05T06:46:32.174007+0000","last_clean":"2024-02-05T06:46:32.174007+0000","last_became_active":"2024-02-05T06:40:27.068888+0000","last_became_peered":"2024-02-05T06:40:27.068888+0000","last_unstale":"2024-02-05T06:46:32.174007+0000","last_undegraded":"2024-02-05T06:46:32.174007+0000","last_fullsized":"2024-02-05T06:46:32.174007+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:55.125921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:44:55.125921+0000","last_clean_scrub_stamp":"2024-02-05T06:44:55.125921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":140,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:22.880947+0000","last_change":"2024-02-05T06:45:51.218029+0000","last_active":"2024-02-05T06:46:22.880947+0000","last_peered":"2024-02-05T06:46:22.880947+0000","last_clean":"2024-02-05T06:46:22.880947+0000","last_became_active":"2024-02-05T06:42:33.129959+0000","last_became_peered":"2024-02-05T06:42:33.129959+0000","last_unstale":"2024-02-05T06:46:22.880947+0000","last_undegraded":"2024-02-05T06:46:22.880947+0000","last_fullsized":"2024-02-05T06:46:22.880947+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:51.217900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:45:51.217900+0000","last_clean_scrub_stamp":"2024-02-05T06:45:51.217900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:47:00.138599+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":186,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.909171+0000","last_change":"2024-02-05T06:43:56.453639+0000","last_active":"2024-02-05T06:46:27.909171+0000","last_peered":"2024-02-05T06:46:27.909171+0000","last_clean":"2024-02-05T06:46:27.909171+0000","last_became_active":"2024-02-05T06:41:43.403746+0000","last_became_peered":"2024-02-05T06:41:43.403746+0000","last_unstale":"2024-02-05T06:46:27.909171+0000","last_undegraded":"2024-02-05T06:46:27.909171+0000","last_fullsized":"2024-02-05T06:46:27.909171+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:56.453478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:44.081318+0000","last_clean_scrub_stamp":"2024-02-05T06:43:56.453478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:30.354647+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"295'19","reported_seq":432,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:22.881899+0000","last_change":"2024-02-05T06:45:09.788862+0000","last_active":"2024-02-05T06:46:22.881899+0000","last_peered":"2024-02-05T06:46:22.881899+0000","last_clean":"2024-02-05T06:46:22.881899+0000","last_became_active":"2024-02-05T06:45:09.787626+0000","last_became_peered":"2024-02-05T06:45:09.787626+0000","last_unstale":"2024-02-05T06:46:22.881899+0000","last_undegraded":"2024-02-05T06:46:22.881899+0000","last_fullsized":"2024-02-05T06:46:22.881899+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'19","last_scrub_stamp":"2024-02-05T06:43:49.843581+0000","last_deep_scrub":"92'13","last_deep_scrub_stamp":"2024-02-05T06:34:46.542893+0000","last_clean_scrub_stamp":"2024-02-05T06:43:49.843581+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:24.289237+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":194,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.902737+0000","last_change":"2024-02-05T06:45:57.292808+0000","last_active":"2024-02-05T06:46:27.902737+0000","last_peered":"2024-02-05T06:46:27.902737+0000","last_clean":"2024-02-05T06:46:27.902737+0000","last_became_active":"2024-02-05T06:41:43.407404+0000","last_became_peered":"2024-02-05T06:41:43.407404+0000","last_unstale":"2024-02-05T06:46:27.902737+0000","last_undegraded":"2024-02-05T06:46:27.902737+0000","last_fullsized":"2024-02-05T06:46:27.902737+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:57.292608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:21.998850+0000","last_clean_scrub_stamp":"2024-02-05T06:45:57.292608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:47:19.133240+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":287,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.908892+0000","last_change":"2024-02-05T06:45:31.314145+0000","last_active":"2024-02-05T06:46:27.908892+0000","last_peered":"2024-02-05T06:46:27.908892+0000","last_clean":"2024-02-05T06:46:27.908892+0000","last_became_active":"2024-02-05T06:40:27.071758+0000","last_became_peered":"2024-02-05T06:40:27.071758+0000","last_unstale":"2024-02-05T06:46:27.908892+0000","last_undegraded":"2024-02-05T06:46:27.908892+0000","last_fullsized":"2024-02-05T06:46:27.908892+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:31.314014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:45:31.314014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:57.109204+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":194,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.903207+0000","last_change":"2024-02-05T06:45:01.461298+0000","last_active":"2024-02-05T06:46:27.903207+0000","last_peered":"2024-02-05T06:46:27.903207+0000","last_clean":"2024-02-05T06:46:27.903207+0000","last_became_active":"2024-02-05T06:41:43.408428+0000","last_became_peered":"2024-02-05T06:41:43.408428+0000","last_unstale":"2024-02-05T06:46:27.903207+0000","last_undegraded":"2024-02-05T06:46:27.903207+0000","last_fullsized":"2024-02-05T06:46:27.903207+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:01.461184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:50.055245+0000","last_clean_scrub_stamp":"2024-02-05T06:45:01.461184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:30.354918+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"295'7","reported_seq":422,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:22.880746+0000","last_change":"2024-02-05T06:45:09.809100+0000","last_active":"2024-02-05T06:46:22.880746+0000","last_peered":"2024-02-05T06:46:22.880746+0000","last_clean":"2024-02-05T06:46:22.880746+0000","last_became_active":"2024-02-05T06:45:09.808663+0000","last_became_peered":"2024-02-05T06:45:09.808663+0000","last_unstale":"2024-02-05T06:46:22.880746+0000","last_undegraded":"2024-02-05T06:46:22.880746+0000","last_fullsized":"2024-02-05T06:46:22.880746+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'7","last_scrub_stamp":"2024-02-05T06:44:55.988817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:44:55.988817+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:24.676066+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":8,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":326,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.898131+0000","last_change":"2024-02-05T06:45:09.771205+0000","last_active":"2024-02-05T06:46:27.898131+0000","last_peered":"2024-02-05T06:46:27.898131+0000","last_clean":"2024-02-05T06:46:27.898131+0000","last_became_active":"2024-02-05T06:45:09.770902+0000","last_became_peered":"2024-02-05T06:45:09.770902+0000","last_unstale":"2024-02-05T06:46:27.898131+0000","last_undegraded":"2024-02-05T06:46:27.898131+0000","last_fullsized":"2024-02-05T06:46:27.898131+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:44.092904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:44:44.092904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":589,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.906376+0000","last_change":"2024-02-05T06:45:39.848063+0000","last_active":"2024-02-05T06:46:27.906376+0000","last_peered":"2024-02-05T06:46:27.906376+0000","last_clean":"2024-02-05T06:46:27.906376+0000","last_became_active":"2024-02-05T06:36:54.958069+0000","last_became_peered":"2024-02-05T06:36:54.958069+0000","last_unstale":"2024-02-05T06:46:27.906376+0000","last_undegraded":"2024-02-05T06:46:27.906376+0000","last_fullsized":"2024-02-05T06:46:27.906376+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:39.847950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:45:39.847950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:47:07.253205+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"281'2","reported_seq":627,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-05T06:46:32.168386+0000","last_change":"2024-02-05T06:44:56.076850+0000","last_active":"2024-02-05T06:46:32.168386+0000","last_peered":"2024-02-05T06:46:32.168386+0000","last_clean":"2024-02-05T06:46:32.168386+0000","last_became_active":"2024-02-05T06:40:27.073360+0000","last_became_peered":"2024-02-05T06:40:27.073360+0000","last_unstale":"2024-02-05T06:46:32.168386+0000","last_undegraded":"2024-02-05T06:46:32.168386+0000","last_fullsized":"2024-02-05T06:46:32.168386+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'2","last_scrub_stamp":"2024-02-05T06:44:56.076717+0000","last_deep_scrub":"281'2","last_deep_scrub_stamp":"2024-02-05T06:44:56.076717+0000","last_clean_scrub_stamp":"2024-02-05T06:44:56.076717+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"295'34","reported_seq":653,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.912195+0000","last_change":"2024-02-05T06:45:33.659911+0000","last_active":"2024-02-05T06:46:27.912195+0000","last_peered":"2024-02-05T06:46:27.912195+0000","last_clean":"2024-02-05T06:46:27.912195+0000","last_became_active":"2024-02-05T06:33:31.569763+0000","last_became_peered":"2024-02-05T06:33:31.569763+0000","last_unstale":"2024-02-05T06:46:27.912195+0000","last_undegraded":"2024-02-05T06:46:27.912195+0000","last_fullsized":"2024-02-05T06:46:27.912195+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'34","last_scrub_stamp":"2024-02-05T06:45:33.659771+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:34:35.234636+0000","last_clean_scrub_stamp":"2024-02-05T06:45:33.659771+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:44.863648+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":568,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:22.883787+0000","last_change":"2024-02-05T06:43:38.524343+0000","last_active":"2024-02-05T06:46:22.883787+0000","last_peered":"2024-02-05T06:46:22.883787+0000","last_clean":"2024-02-05T06:46:22.883787+0000","last_became_active":"2024-02-05T06:41:43.411442+0000","last_became_peered":"2024-02-05T06:41:43.411442+0000","last_unstale":"2024-02-05T06:46:22.883787+0000","last_undegraded":"2024-02-05T06:46:22.883787+0000","last_fullsized":"2024-02-05T06:46:22.883787+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:38.524236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:43:38.524236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:24.288958+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":358,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.897887+0000","last_change":"2024-02-05T06:44:57.658960+0000","last_active":"2024-02-05T06:46:27.897887+0000","last_peered":"2024-02-05T06:46:27.897887+0000","last_clean":"2024-02-05T06:46:27.897887+0000","last_became_active":"2024-02-05T06:39:13.248618+0000","last_became_peered":"2024-02-05T06:39:13.248618+0000","last_unstale":"2024-02-05T06:46:27.897887+0000","last_undegraded":"2024-02-05T06:46:27.897887+0000","last_fullsized":"2024-02-05T06:46:27.897887+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:57.658852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:44:57.658852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":107,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.906392+0000","last_change":"2024-02-05T06:45:35.741350+0000","last_active":"2024-02-05T06:46:27.906392+0000","last_peered":"2024-02-05T06:46:27.906392+0000","last_clean":"2024-02-05T06:46:27.906392+0000","last_became_active":"2024-02-05T06:45:09.786579+0000","last_became_peered":"2024-02-05T06:45:09.786579+0000","last_unstale":"2024-02-05T06:46:27.906392+0000","last_undegraded":"2024-02-05T06:46:27.906392+0000","last_fullsized":"2024-02-05T06:46:27.906392+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:35.741029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:45:35.741029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:58.979180+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"295'18","reported_seq":657,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:22.881086+0000","last_change":"2024-02-05T06:45:01.790024+0000","last_active":"2024-02-05T06:46:22.881086+0000","last_peered":"2024-02-05T06:46:22.881086+0000","last_clean":"2024-02-05T06:46:22.881086+0000","last_became_active":"2024-02-05T06:36:54.954880+0000","last_became_peered":"2024-02-05T06:36:54.954880+0000","last_unstale":"2024-02-05T06:46:22.881086+0000","last_undegraded":"2024-02-05T06:46:22.881086+0000","last_fullsized":"2024-02-05T06:46:22.881086+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'18","last_scrub_stamp":"2024-02-05T06:45:01.789938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:45:01.789938+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:24.289305+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"325'2","reported_seq":294,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.910308+0000","last_change":"2024-02-05T06:45:46.242637+0000","last_active":"2024-02-05T06:46:27.910308+0000","last_peered":"2024-02-05T06:46:27.910308+0000","last_clean":"2024-02-05T06:46:27.910308+0000","last_became_active":"2024-02-05T06:41:43.418640+0000","last_became_peered":"2024-02-05T06:41:43.418640+0000","last_unstale":"2024-02-05T06:46:27.910308+0000","last_undegraded":"2024-02-05T06:46:27.910308+0000","last_fullsized":"2024-02-05T06:46:27.910308+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'2","last_scrub_stamp":"2024-02-05T06:45:46.242415+0000","last_deep_scrub":"325'2","last_deep_scrub_stamp":"2024-02-05T06:45:46.242415+0000","last_clean_scrub_stamp":"2024-02-05T06:45:46.242415+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:47:00.636082+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"314'13","reported_seq":807,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.897983+0000","last_change":"2024-02-05T06:45:57.611451+0000","last_active":"2024-02-05T06:46:27.897983+0000","last_peered":"2024-02-05T06:46:27.897983+0000","last_clean":"2024-02-05T06:46:27.897983+0000","last_became_active":"2024-02-05T06:40:27.063399+0000","last_became_peered":"2024-02-05T06:40:27.063399+0000","last_unstale":"2024-02-05T06:46:27.897983+0000","last_undegraded":"2024-02-05T06:46:27.897983+0000","last_fullsized":"2024-02-05T06:46:27.897983+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"314'13","last_scrub_stamp":"2024-02-05T06:45:57.611330+0000","last_deep_scrub":"314'13","last_deep_scrub_stamp":"2024-02-05T06:44:53.670923+0000","last_clean_scrub_stamp":"2024-02-05T06:45:57.611330+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:47:02.547216+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":614,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.900611+0000","last_change":"2024-02-05T06:45:30.780092+0000","last_active":"2024-02-05T06:46:27.900611+0000","last_peered":"2024-02-05T06:46:27.900611+0000","last_clean":"2024-02-05T06:46:27.900611+0000","last_became_active":"2024-02-05T06:36:54.958414+0000","last_became_peered":"2024-02-05T06:36:54.958414+0000","last_unstale":"2024-02-05T06:46:27.900611+0000","last_undegraded":"2024-02-05T06:46:27.900611+0000","last_fullsized":"2024-02-05T06:46:27.900611+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:45:30.779959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:45:30.779959+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:53.194323+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"295'25","reported_seq":656,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-05T06:46:32.169273+0000","last_change":"2024-02-05T06:45:40.897349+0000","last_active":"2024-02-05T06:46:32.169273+0000","last_peered":"2024-02-05T06:46:32.169273+0000","last_clean":"2024-02-05T06:46:32.169273+0000","last_became_active":"2024-02-05T06:40:27.100610+0000","last_became_peered":"2024-02-05T06:40:27.100610+0000","last_unstale":"2024-02-05T06:46:32.169273+0000","last_undegraded":"2024-02-05T06:46:32.169273+0000","last_fullsized":"2024-02-05T06:46:32.169273+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'25","last_scrub_stamp":"2024-02-05T06:45:40.897208+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-02-05T06:34:50.718768+0000","last_clean_scrub_stamp":"2024-02-05T06:45:40.897208+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:49.855517+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":597,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.906516+0000","last_change":"2024-02-05T06:44:41.760057+0000","last_active":"2024-02-05T06:46:27.906516+0000","last_peered":"2024-02-05T06:46:27.906516+0000","last_clean":"2024-02-05T06:46:27.906516+0000","last_became_active":"2024-02-05T06:41:43.414638+0000","last_became_peered":"2024-02-05T06:41:43.414638+0000","last_unstale":"2024-02-05T06:46:27.906516+0000","last_undegraded":"2024-02-05T06:46:27.906516+0000","last_fullsized":"2024-02-05T06:46:27.906516+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:41.759930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:44:41.759930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":483,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:22.881141+0000","last_change":"2024-02-05T06:42:58.394440+0000","last_active":"2024-02-05T06:46:22.881141+0000","last_peered":"2024-02-05T06:46:22.881141+0000","last_clean":"2024-02-05T06:46:22.881141+0000","last_became_active":"2024-02-05T06:40:27.097169+0000","last_became_peered":"2024-02-05T06:40:27.097169+0000","last_unstale":"2024-02-05T06:46:22.881141+0000","last_undegraded":"2024-02-05T06:46:22.881141+0000","last_fullsized":"2024-02-05T06:46:22.881141+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:58.394289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:08.713894+0000","last_clean_scrub_stamp":"2024-02-05T06:42:58.394289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:24.284059+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":186,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.902651+0000","last_change":"2024-02-05T06:43:44.345193+0000","last_active":"2024-02-05T06:46:27.902651+0000","last_peered":"2024-02-05T06:46:27.902651+0000","last_clean":"2024-02-05T06:46:27.902651+0000","last_became_active":"2024-02-05T06:41:43.411043+0000","last_became_peered":"2024-02-05T06:41:43.411043+0000","last_unstale":"2024-02-05T06:46:27.902651+0000","last_undegraded":"2024-02-05T06:46:27.902651+0000","last_fullsized":"2024-02-05T06:46:27.902651+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:44.345092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:43:44.345092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:30.354569+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"295'24","reported_seq":672,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.898197+0000","last_change":"2024-02-05T06:45:53.499400+0000","last_active":"2024-02-05T06:46:27.898197+0000","last_peered":"2024-02-05T06:46:27.898197+0000","last_clean":"2024-02-05T06:46:27.898197+0000","last_became_active":"2024-02-05T06:41:43.419769+0000","last_became_peered":"2024-02-05T06:41:43.419769+0000","last_unstale":"2024-02-05T06:46:27.898197+0000","last_undegraded":"2024-02-05T06:46:27.898197+0000","last_fullsized":"2024-02-05T06:46:27.898197+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:45:53.499260+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:43:03.221466+0000","last_clean_scrub_stamp":"2024-02-05T06:45:53.499260+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:58.133845+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":293,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.909130+0000","last_change":"2024-02-05T06:45:51.212570+0000","last_active":"2024-02-05T06:46:27.909130+0000","last_peered":"2024-02-05T06:46:27.909130+0000","last_clean":"2024-02-05T06:46:27.909130+0000","last_became_active":"2024-02-05T06:44:54.625067+0000","last_became_peered":"2024-02-05T06:44:54.625067+0000","last_unstale":"2024-02-05T06:46:27.909130+0000","last_undegraded":"2024-02-05T06:46:27.909130+0000","last_fullsized":"2024-02-05T06:46:27.909130+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:51.212462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:48.512673+0000","last_clean_scrub_stamp":"2024-02-05T06:45:51.212462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:47:19.879788+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":616,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.897300+0000","last_change":"2024-02-05T06:45:33.691776+0000","last_active":"2024-02-05T06:46:27.897300+0000","last_peered":"2024-02-05T06:46:27.897300+0000","last_clean":"2024-02-05T06:46:27.897300+0000","last_became_active":"2024-02-05T06:40:27.060112+0000","last_became_peered":"2024-02-05T06:40:27.060112+0000","last_unstale":"2024-02-05T06:46:27.897300+0000","last_undegraded":"2024-02-05T06:46:27.897300+0000","last_fullsized":"2024-02-05T06:46:27.897300+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:33.691525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:45:33.691525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:48.873744+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":592,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-05T06:46:32.168754+0000","last_change":"2024-02-05T06:46:31.409743+0000","last_active":"2024-02-05T06:46:32.168754+0000","last_peered":"2024-02-05T06:46:32.168754+0000","last_clean":"2024-02-05T06:46:32.168754+0000","last_became_active":"2024-02-05T06:41:43.424459+0000","last_became_peered":"2024-02-05T06:41:43.424459+0000","last_unstale":"2024-02-05T06:46:32.168754+0000","last_undegraded":"2024-02-05T06:46:32.168754+0000","last_fullsized":"2024-02-05T06:46:32.168754+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:31.409628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:49.117949+0000","last_clean_scrub_stamp":"2024-02-05T06:46:31.409628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:47:31.490646+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"295'8","reported_seq":575,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:22.880865+0000","last_change":"2024-02-05T06:43:51.672673+0000","last_active":"2024-02-05T06:46:22.880865+0000","last_peered":"2024-02-05T06:46:22.880865+0000","last_clean":"2024-02-05T06:46:22.880865+0000","last_became_active":"2024-02-05T06:33:31.573410+0000","last_became_peered":"2024-02-05T06:33:31.573410+0000","last_unstale":"2024-02-05T06:46:22.880865+0000","last_undegraded":"2024-02-05T06:46:22.880865+0000","last_fullsized":"2024-02-05T06:46:22.880865+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'8","last_scrub_stamp":"2024-02-05T06:43:51.672524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:51.672524+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:24.289121+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":344,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.901063+0000","last_change":"2024-02-05T06:43:59.635993+0000","last_active":"2024-02-05T06:46:27.901063+0000","last_peered":"2024-02-05T06:46:27.901063+0000","last_clean":"2024-02-05T06:46:27.901063+0000","last_became_active":"2024-02-05T06:39:13.245684+0000","last_became_peered":"2024-02-05T06:39:13.245684+0000","last_unstale":"2024-02-05T06:46:27.901063+0000","last_undegraded":"2024-02-05T06:46:27.901063+0000","last_fullsized":"2024-02-05T06:46:27.901063+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:59.635733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:43:59.635733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":279,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.907807+0000","last_change":"2024-02-05T06:44:40.456593+0000","last_active":"2024-02-05T06:46:27.907807+0000","last_peered":"2024-02-05T06:46:27.907807+0000","last_clean":"2024-02-05T06:46:27.907807+0000","last_became_active":"2024-02-05T06:40:27.062975+0000","last_became_peered":"2024-02-05T06:40:27.062975+0000","last_unstale":"2024-02-05T06:46:27.907807+0000","last_undegraded":"2024-02-05T06:46:27.907807+0000","last_fullsized":"2024-02-05T06:46:27.907807+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:40.456474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:44:40.456474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":194,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.904189+0000","last_change":"2024-02-05T06:44:46.523296+0000","last_active":"2024-02-05T06:46:27.904189+0000","last_peered":"2024-02-05T06:46:27.904189+0000","last_clean":"2024-02-05T06:46:27.904189+0000","last_became_active":"2024-02-05T06:41:43.412654+0000","last_became_peered":"2024-02-05T06:41:43.412654+0000","last_unstale":"2024-02-05T06:46:27.904189+0000","last_undegraded":"2024-02-05T06:46:27.904189+0000","last_fullsized":"2024-02-05T06:46:27.904189+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:46.523202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:44:46.523202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:30.354755+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"295'24","reported_seq":653,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:22.883056+0000","last_change":"2024-02-05T06:43:39.456286+0000","last_active":"2024-02-05T06:46:22.883056+0000","last_peered":"2024-02-05T06:46:22.883056+0000","last_clean":"2024-02-05T06:46:22.883056+0000","last_became_active":"2024-02-05T06:40:27.080906+0000","last_became_peered":"2024-02-05T06:40:27.080906+0000","last_unstale":"2024-02-05T06:46:22.883056+0000","last_undegraded":"2024-02-05T06:46:22.883056+0000","last_fullsized":"2024-02-05T06:46:22.883056+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:43:39.456190+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-02-05T06:35:22.938675+0000","last_clean_scrub_stamp":"2024-02-05T06:43:39.456190+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:24.289014+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"73'1","reported_seq":589,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-05T06:46:32.173017+0000","last_change":"2024-02-05T06:44:59.110461+0000","last_active":"2024-02-05T06:46:32.173017+0000","last_peered":"2024-02-05T06:46:32.173017+0000","last_clean":"2024-02-05T06:46:32.173017+0000","last_became_active":"2024-02-05T06:41:43.409761+0000","last_became_peered":"2024-02-05T06:41:43.409761+0000","last_unstale":"2024-02-05T06:46:32.173017+0000","last_undegraded":"2024-02-05T06:46:32.173017+0000","last_fullsized":"2024-02-05T06:46:32.173017+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-02-05T06:44:59.110330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:44:59.110330+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":559,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:22.880859+0000","last_change":"2024-02-05T06:43:58.672945+0000","last_active":"2024-02-05T06:46:22.880859+0000","last_peered":"2024-02-05T06:46:22.880859+0000","last_clean":"2024-02-05T06:46:22.880859+0000","last_became_active":"2024-02-05T06:33:33.593901+0000","last_became_peered":"2024-02-05T06:33:33.593901+0000","last_unstale":"2024-02-05T06:46:22.880859+0000","last_undegraded":"2024-02-05T06:46:22.880859+0000","last_fullsized":"2024-02-05T06:46:22.880859+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:58.672808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:43:58.672808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:24.289066+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":287,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:29.865858+0000","last_change":"2024-02-05T06:46:29.865891+0000","last_active":"2024-02-05T06:46:29.865858+0000","last_peered":"2024-02-05T06:46:29.865858+0000","last_clean":"2024-02-05T06:46:29.865858+0000","last_became_active":"2024-02-05T06:40:27.077701+0000","last_became_peered":"2024-02-05T06:40:27.077701+0000","last_unstale":"2024-02-05T06:46:29.865858+0000","last_undegraded":"2024-02-05T06:46:29.865858+0000","last_fullsized":"2024-02-05T06:46:29.865858+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:29.865777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:46:29.865777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:47:54.658113+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"295'15","reported_seq":365,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-05T06:46:32.167776+0000","last_change":"2024-02-05T06:44:43.912018+0000","last_active":"2024-02-05T06:46:32.167776+0000","last_peered":"2024-02-05T06:46:32.167776+0000","last_clean":"2024-02-05T06:46:32.167776+0000","last_became_active":"2024-02-05T06:39:13.244124+0000","last_became_peered":"2024-02-05T06:39:13.244124+0000","last_unstale":"2024-02-05T06:46:32.167776+0000","last_undegraded":"2024-02-05T06:46:32.167776+0000","last_fullsized":"2024-02-05T06:46:32.167776+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:44:43.911889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:44:43.911889+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":575,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:22.881615+0000","last_change":"2024-02-05T06:45:35.052699+0000","last_active":"2024-02-05T06:46:22.881615+0000","last_peered":"2024-02-05T06:46:22.881615+0000","last_clean":"2024-02-05T06:46:22.881615+0000","last_became_active":"2024-02-05T06:41:43.417155+0000","last_became_peered":"2024-02-05T06:41:43.417155+0000","last_unstale":"2024-02-05T06:46:22.881615+0000","last_undegraded":"2024-02-05T06:46:22.881615+0000","last_fullsized":"2024-02-05T06:46:22.881615+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:35.052451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:45:35.052451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:50.794956+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":603,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.908447+0000","last_change":"2024-02-05T06:45:40.802539+0000","last_active":"2024-02-05T06:46:27.908447+0000","last_peered":"2024-02-05T06:46:27.908447+0000","last_clean":"2024-02-05T06:46:27.908447+0000","last_became_active":"2024-02-05T06:41:43.421433+0000","last_became_peered":"2024-02-05T06:41:43.421433+0000","last_unstale":"2024-02-05T06:46:27.908447+0000","last_undegraded":"2024-02-05T06:46:27.908447+0000","last_fullsized":"2024-02-05T06:46:27.908447+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:40.802433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:45:40.802433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:47:07.971018+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":233,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.910208+0000","last_change":"2024-02-05T06:45:32.269031+0000","last_active":"2024-02-05T06:46:27.910208+0000","last_peered":"2024-02-05T06:46:27.910208+0000","last_clean":"2024-02-05T06:46:27.910208+0000","last_became_active":"2024-02-05T06:45:09.773490+0000","last_became_peered":"2024-02-05T06:45:09.773490+0000","last_unstale":"2024-02-05T06:46:27.910208+0000","last_undegraded":"2024-02-05T06:46:27.910208+0000","last_fullsized":"2024-02-05T06:46:27.910208+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:32.268929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:45:32.268929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:51.730123+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"295'24","reported_seq":439,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.897639+0000","last_change":"2024-02-05T06:45:29.680009+0000","last_active":"2024-02-05T06:46:27.897639+0000","last_peered":"2024-02-05T06:46:27.897639+0000","last_clean":"2024-02-05T06:46:27.897639+0000","last_became_active":"2024-02-05T06:45:09.784411+0000","last_became_peered":"2024-02-05T06:45:09.784411+0000","last_unstale":"2024-02-05T06:46:27.897639+0000","last_undegraded":"2024-02-05T06:46:27.897639+0000","last_fullsized":"2024-02-05T06:46:27.897639+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:45:29.679614+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:35:02.589229+0000","last_clean_scrub_stamp":"2024-02-05T06:45:29.679614+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:40.001407+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":580,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-05T06:46:32.169016+0000","last_change":"2024-02-05T06:45:39.000945+0000","last_active":"2024-02-05T06:46:32.169016+0000","last_peered":"2024-02-05T06:46:32.169016+0000","last_clean":"2024-02-05T06:46:32.169016+0000","last_became_active":"2024-02-05T06:40:27.096132+0000","last_became_peered":"2024-02-05T06:40:27.096132+0000","last_unstale":"2024-02-05T06:46:32.169016+0000","last_undegraded":"2024-02-05T06:46:32.169016+0000","last_fullsized":"2024-02-05T06:46:32.169016+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:39.000627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:45:39.000627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:59.403199+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"249'10","reported_seq":256,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.908937+0000","last_change":"2024-02-05T06:44:55.577666+0000","last_active":"2024-02-05T06:46:27.908937+0000","last_peered":"2024-02-05T06:46:27.908937+0000","last_clean":"2024-02-05T06:46:27.908937+0000","last_became_active":"2024-02-05T06:41:43.418753+0000","last_became_peered":"2024-02-05T06:41:43.418753+0000","last_unstale":"2024-02-05T06:46:27.908937+0000","last_undegraded":"2024-02-05T06:46:27.908937+0000","last_fullsized":"2024-02-05T06:46:27.908937+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'10","last_scrub_stamp":"2024-02-05T06:44:55.577535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:44:55.577535+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:30.354783+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":279,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.909399+0000","last_change":"2024-02-05T06:44:45.493197+0000","last_active":"2024-02-05T06:46:27.909399+0000","last_peered":"2024-02-05T06:46:27.909399+0000","last_clean":"2024-02-05T06:46:27.909399+0000","last_became_active":"2024-02-05T06:40:27.098418+0000","last_became_peered":"2024-02-05T06:40:27.098418+0000","last_unstale":"2024-02-05T06:46:27.909399+0000","last_undegraded":"2024-02-05T06:46:27.909399+0000","last_fullsized":"2024-02-05T06:46:27.909399+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:45.493068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:44:45.493068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"295'20","reported_seq":655,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:22.884909+0000","last_change":"2024-02-05T06:45:31.996335+0000","last_active":"2024-02-05T06:46:22.884909+0000","last_peered":"2024-02-05T06:46:22.884909+0000","last_clean":"2024-02-05T06:46:22.884909+0000","last_became_active":"2024-02-05T06:40:27.093418+0000","last_became_peered":"2024-02-05T06:40:27.093418+0000","last_unstale":"2024-02-05T06:46:22.884909+0000","last_undegraded":"2024-02-05T06:46:22.884909+0000","last_fullsized":"2024-02-05T06:46:22.884909+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'20","last_scrub_stamp":"2024-02-05T06:45:31.996183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:45:31.996183+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:43.441041+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":325,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.913152+0000","last_change":"2024-02-05T06:45:09.775451+0000","last_active":"2024-02-05T06:46:27.913152+0000","last_peered":"2024-02-05T06:46:27.913152+0000","last_clean":"2024-02-05T06:46:27.913152+0000","last_became_active":"2024-02-05T06:45:09.775152+0000","last_became_peered":"2024-02-05T06:45:09.775152+0000","last_unstale":"2024-02-05T06:46:27.913152+0000","last_undegraded":"2024-02-05T06:46:27.913152+0000","last_fullsized":"2024-02-05T06:46:27.913152+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:42.166062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:41:38.559780+0000","last_clean_scrub_stamp":"2024-02-05T06:44:42.166062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:31.745886+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"208'10","reported_seq":751,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:22.885333+0000","last_change":"2024-02-05T06:44:49.013016+0000","last_active":"2024-02-05T06:46:22.885333+0000","last_peered":"2024-02-05T06:46:22.885333+0000","last_clean":"2024-02-05T06:46:22.885333+0000","last_became_active":"2024-02-05T06:40:27.096584+0000","last_became_peered":"2024-02-05T06:40:27.096584+0000","last_unstale":"2024-02-05T06:46:22.885333+0000","last_undegraded":"2024-02-05T06:46:22.885333+0000","last_fullsized":"2024-02-05T06:46:22.885333+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-02-05T06:44:49.012783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:44:49.012783+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":600,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-05T06:46:32.167962+0000","last_change":"2024-02-05T06:45:45.945300+0000","last_active":"2024-02-05T06:46:32.167962+0000","last_peered":"2024-02-05T06:46:32.167962+0000","last_clean":"2024-02-05T06:46:32.167962+0000","last_became_active":"2024-02-05T06:40:27.088096+0000","last_became_peered":"2024-02-05T06:40:27.088096+0000","last_unstale":"2024-02-05T06:46:32.167962+0000","last_undegraded":"2024-02-05T06:46:32.167962+0000","last_fullsized":"2024-02-05T06:46:32.167962+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:45.945168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:44:01.185653+0000","last_clean_scrub_stamp":"2024-02-05T06:45:45.945168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:47:06.078406+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"295'15","reported_seq":213,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.909363+0000","last_change":"2024-02-05T06:44:56.555609+0000","last_active":"2024-02-05T06:46:27.909363+0000","last_peered":"2024-02-05T06:46:27.909363+0000","last_clean":"2024-02-05T06:46:27.909363+0000","last_became_active":"2024-02-05T06:41:43.413708+0000","last_became_peered":"2024-02-05T06:41:43.413708+0000","last_unstale":"2024-02-05T06:46:27.909363+0000","last_undegraded":"2024-02-05T06:46:27.909363+0000","last_fullsized":"2024-02-05T06:46:27.909363+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:44:56.553738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:44:56.553738+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:30.354730+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":611,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.912478+0000","last_change":"2024-02-05T06:44:50.560814+0000","last_active":"2024-02-05T06:46:27.912478+0000","last_peered":"2024-02-05T06:46:27.912478+0000","last_clean":"2024-02-05T06:46:27.912478+0000","last_became_active":"2024-02-05T06:40:27.090632+0000","last_became_peered":"2024-02-05T06:40:27.090632+0000","last_unstale":"2024-02-05T06:46:27.912478+0000","last_undegraded":"2024-02-05T06:46:27.912478+0000","last_fullsized":"2024-02-05T06:46:27.912478+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:50.560688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:44:50.560688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:31.745914+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":585,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:22.883697+0000","last_change":"2024-02-05T06:44:01.644210+0000","last_active":"2024-02-05T06:46:22.883697+0000","last_peered":"2024-02-05T06:46:22.883697+0000","last_clean":"2024-02-05T06:46:22.883697+0000","last_became_active":"2024-02-05T06:40:27.091544+0000","last_became_peered":"2024-02-05T06:40:27.091544+0000","last_unstale":"2024-02-05T06:46:22.883697+0000","last_undegraded":"2024-02-05T06:46:22.883697+0000","last_fullsized":"2024-02-05T06:46:22.883697+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:01.644095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:44:01.644095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:24.289147+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":279,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.908330+0000","last_change":"2024-02-05T06:44:46.519345+0000","last_active":"2024-02-05T06:46:27.908330+0000","last_peered":"2024-02-05T06:46:27.908330+0000","last_clean":"2024-02-05T06:46:27.908330+0000","last_became_active":"2024-02-05T06:40:27.064388+0000","last_became_peered":"2024-02-05T06:40:27.064388+0000","last_unstale":"2024-02-05T06:46:27.908330+0000","last_undegraded":"2024-02-05T06:46:27.908330+0000","last_fullsized":"2024-02-05T06:46:27.908330+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:46.519235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:44:46.519235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"295'15","reported_seq":641,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-05T06:46:32.169421+0000","last_change":"2024-02-05T06:45:55.988708+0000","last_active":"2024-02-05T06:46:32.169421+0000","last_peered":"2024-02-05T06:46:32.169421+0000","last_clean":"2024-02-05T06:46:32.169421+0000","last_became_active":"2024-02-05T06:40:27.101116+0000","last_became_peered":"2024-02-05T06:40:27.101116+0000","last_unstale":"2024-02-05T06:46:32.169421+0000","last_undegraded":"2024-02-05T06:46:32.169421+0000","last_fullsized":"2024-02-05T06:46:32.169421+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:45:55.988575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:45:55.988575+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:47:14.265458+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":279,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.910055+0000","last_change":"2024-02-05T06:44:57.486102+0000","last_active":"2024-02-05T06:46:27.910055+0000","last_peered":"2024-02-05T06:46:27.910055+0000","last_clean":"2024-02-05T06:46:27.910055+0000","last_became_active":"2024-02-05T06:40:27.063350+0000","last_became_peered":"2024-02-05T06:40:27.063350+0000","last_unstale":"2024-02-05T06:46:27.910055+0000","last_undegraded":"2024-02-05T06:46:27.910055+0000","last_fullsized":"2024-02-05T06:46:27.910055+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:57.485931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:44:57.485931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:30.838097+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":194,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.904120+0000","last_change":"2024-02-05T06:44:57.507114+0000","last_active":"2024-02-05T06:46:27.904120+0000","last_peered":"2024-02-05T06:46:27.904120+0000","last_clean":"2024-02-05T06:46:27.904120+0000","last_became_active":"2024-02-05T06:41:43.406094+0000","last_became_peered":"2024-02-05T06:41:43.406094+0000","last_unstale":"2024-02-05T06:46:27.904120+0000","last_undegraded":"2024-02-05T06:46:27.904120+0000","last_fullsized":"2024-02-05T06:46:27.904120+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:57.506909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:44:57.506909+0000","last_clean_scrub_stamp":"2024-02-05T06:44:57.506909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:30.354809+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":604,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.896834+0000","last_change":"2024-02-05T06:44:43.506379+0000","last_active":"2024-02-05T06:46:27.896834+0000","last_peered":"2024-02-05T06:46:27.896834+0000","last_clean":"2024-02-05T06:46:27.896834+0000","last_became_active":"2024-02-05T06:41:43.412130+0000","last_became_peered":"2024-02-05T06:41:43.412130+0000","last_unstale":"2024-02-05T06:46:27.896834+0000","last_undegraded":"2024-02-05T06:46:27.896834+0000","last_fullsized":"2024-02-05T06:46:27.896834+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:43.505934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:44:43.505934+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"295'12","reported_seq":609,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:22.883873+0000","last_change":"2024-02-05T06:43:37.426033+0000","last_active":"2024-02-05T06:46:22.883873+0000","last_peered":"2024-02-05T06:46:22.883873+0000","last_clean":"2024-02-05T06:46:22.883873+0000","last_became_active":"2024-02-05T06:40:27.091689+0000","last_became_peered":"2024-02-05T06:40:27.091689+0000","last_unstale":"2024-02-05T06:46:22.883873+0000","last_undegraded":"2024-02-05T06:46:22.883873+0000","last_fullsized":"2024-02-05T06:46:22.883873+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'12","last_scrub_stamp":"2024-02-05T06:43:37.425879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:37.425879+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:24.288821+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"332'14103","reported_seq":14262,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-05T06:46:32.362646+0000","last_change":"2024-02-05T06:45:38.991351+0000","last_active":"2024-02-05T06:46:32.362646+0000","last_peered":"2024-02-05T06:46:32.362646+0000","last_clean":"2024-02-05T06:46:32.362646+0000","last_became_active":"2024-02-05T06:43:29.702579+0000","last_became_peered":"2024-02-05T06:43:29.702579+0000","last_unstale":"2024-02-05T06:46:32.362646+0000","last_undegraded":"2024-02-05T06:46:32.362646+0000","last_fullsized":"2024-02-05T06:46:32.362646+0000","mapping_epoch":291,"log_start":"320'8545","ondisk_log_start":"320'8545","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"324'10275","last_scrub_stamp":"2024-02-05T06:45:38.990030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:45:38.990030+0000","objects_scrubbed":5957,"log_size":5558,"log_dups_size":0,"ondisk_log_size":5558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:58.426092+0000","scrub_duration":2033,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134545408,"num_objects":2053,"num_object_clones":0,"num_object_copies":6159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14103,"num_write_kb":516992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":195,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.902956+0000","last_change":"2024-02-05T06:44:50.593901+0000","last_active":"2024-02-05T06:46:27.902956+0000","last_peered":"2024-02-05T06:46:27.902956+0000","last_clean":"2024-02-05T06:46:27.902956+0000","last_became_active":"2024-02-05T06:41:43.421231+0000","last_became_peered":"2024-02-05T06:41:43.421231+0000","last_unstale":"2024-02-05T06:46:27.902956+0000","last_undegraded":"2024-02-05T06:46:27.902956+0000","last_fullsized":"2024-02-05T06:46:27.902956+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:50.593756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:44:50.593756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:30.354893+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":285,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.909704+0000","last_change":"2024-02-05T06:43:56.531681+0000","last_active":"2024-02-05T06:46:27.909704+0000","last_peered":"2024-02-05T06:46:27.909704+0000","last_clean":"2024-02-05T06:46:27.909704+0000","last_became_active":"2024-02-05T06:40:27.068279+0000","last_became_peered":"2024-02-05T06:40:27.068279+0000","last_unstale":"2024-02-05T06:46:27.909704+0000","last_undegraded":"2024-02-05T06:46:27.909704+0000","last_fullsized":"2024-02-05T06:46:27.909704+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-02-05T06:43:56.531539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:43:56.531539+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"295'27","reported_seq":229,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.903001+0000","last_change":"2024-02-05T06:45:56.217323+0000","last_active":"2024-02-05T06:46:27.903001+0000","last_peered":"2024-02-05T06:46:27.903001+0000","last_clean":"2024-02-05T06:46:27.903001+0000","last_became_active":"2024-02-05T06:41:43.412579+0000","last_became_peered":"2024-02-05T06:41:43.412579+0000","last_unstale":"2024-02-05T06:46:27.903001+0000","last_undegraded":"2024-02-05T06:46:27.903001+0000","last_fullsized":"2024-02-05T06:46:27.903001+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'27","last_scrub_stamp":"2024-02-05T06:45:56.217180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:45:56.217180+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:47:16.735123+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"208'12","reported_seq":356,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.908323+0000","last_change":"2024-02-05T06:44:44.448466+0000","last_active":"2024-02-05T06:46:27.908323+0000","last_peered":"2024-02-05T06:46:27.908323+0000","last_clean":"2024-02-05T06:46:27.908323+0000","last_became_active":"2024-02-05T06:40:27.064195+0000","last_became_peered":"2024-02-05T06:40:27.064195+0000","last_unstale":"2024-02-05T06:46:27.908323+0000","last_undegraded":"2024-02-05T06:46:27.908323+0000","last_fullsized":"2024-02-05T06:46:27.908323+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'12","last_scrub_stamp":"2024-02-05T06:44:44.448215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:44:44.448215+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"331'14287","reported_seq":14448,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:32.025112+0000","last_change":"2024-02-05T06:44:40.820675+0000","last_active":"2024-02-05T06:46:32.025112+0000","last_peered":"2024-02-05T06:46:32.025112+0000","last_clean":"2024-02-05T06:46:32.025112+0000","last_became_active":"2024-02-05T06:43:29.697495+0000","last_became_peered":"2024-02-05T06:43:29.697495+0000","last_unstale":"2024-02-05T06:46:32.025112+0000","last_undegraded":"2024-02-05T06:46:32.025112+0000","last_fullsized":"2024-02-05T06:46:32.025112+0000","mapping_epoch":291,"log_start":"324'9723","ondisk_log_start":"324'9723","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"312'6215","last_scrub_stamp":"2024-02-05T06:44:40.820543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:44:40.820543+0000","objects_scrubbed":6153,"log_size":4564,"log_dups_size":0,"ondisk_log_size":4564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135725056,"num_objects":2071,"num_object_clones":0,"num_object_copies":6213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2071,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14287,"num_write_kb":523456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":567,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:22.881378+0000","last_change":"2024-02-05T06:43:47.662237+0000","last_active":"2024-02-05T06:46:22.881378+0000","last_peered":"2024-02-05T06:46:22.881378+0000","last_clean":"2024-02-05T06:46:22.881378+0000","last_became_active":"2024-02-05T06:40:27.092613+0000","last_became_peered":"2024-02-05T06:40:27.092613+0000","last_unstale":"2024-02-05T06:46:22.881378+0000","last_undegraded":"2024-02-05T06:46:22.881378+0000","last_fullsized":"2024-02-05T06:46:22.881378+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:47.662025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:43:47.662025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:24.289091+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":279,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.909923+0000","last_change":"2024-02-05T06:44:52.547893+0000","last_active":"2024-02-05T06:46:27.909923+0000","last_peered":"2024-02-05T06:46:27.909923+0000","last_clean":"2024-02-05T06:46:27.909923+0000","last_became_active":"2024-02-05T06:40:27.063300+0000","last_became_peered":"2024-02-05T06:40:27.063300+0000","last_unstale":"2024-02-05T06:46:27.909923+0000","last_undegraded":"2024-02-05T06:46:27.909923+0000","last_fullsized":"2024-02-05T06:46:27.909923+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:52.547420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:44:52.547420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"295'18","reported_seq":437,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.900519+0000","last_change":"2024-02-05T06:45:09.783400+0000","last_active":"2024-02-05T06:46:27.900519+0000","last_peered":"2024-02-05T06:46:27.900519+0000","last_clean":"2024-02-05T06:46:27.900519+0000","last_became_active":"2024-02-05T06:45:09.783036+0000","last_became_peered":"2024-02-05T06:45:09.783036+0000","last_unstale":"2024-02-05T06:46:27.900519+0000","last_undegraded":"2024-02-05T06:46:27.900519+0000","last_fullsized":"2024-02-05T06:46:27.900519+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'18","last_scrub_stamp":"2024-02-05T06:44:57.025034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:44:57.025034+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:28.733372+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":20,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":583,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.911223+0000","last_change":"2024-02-05T06:44:52.595693+0000","last_active":"2024-02-05T06:46:27.911223+0000","last_peered":"2024-02-05T06:46:27.911223+0000","last_clean":"2024-02-05T06:46:27.911223+0000","last_became_active":"2024-02-05T06:33:33.592546+0000","last_became_peered":"2024-02-05T06:33:33.592546+0000","last_unstale":"2024-02-05T06:46:27.911223+0000","last_undegraded":"2024-02-05T06:46:27.911223+0000","last_fullsized":"2024-02-05T06:46:27.911223+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:52.595426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:44:52.595426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:31.745792+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"331'18089","reported_seq":18361,"reported_epoch":331,"state":"active+clean+scrubbing","last_fresh":"2024-02-05T06:46:31.539382+0000","last_change":"2024-02-05T06:46:31.413214+0000","last_active":"2024-02-05T06:46:31.539382+0000","last_peered":"2024-02-05T06:46:31.539382+0000","last_clean":"2024-02-05T06:46:31.539382+0000","last_became_active":"2024-02-05T06:43:29.702659+0000","last_became_peered":"2024-02-05T06:43:29.702659+0000","last_unstale":"2024-02-05T06:46:31.539382+0000","last_undegraded":"2024-02-05T06:46:31.539382+0000","last_fullsized":"2024-02-05T06:46:31.539382+0000","mapping_epoch":291,"log_start":"326'13000","ondisk_log_start":"326'13000","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":399,"log_size":5089,"log_dups_size":0,"ondisk_log_size":5089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":137887744,"num_objects":2104,"num_object_clones":0,"num_object_copies":6312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13872,"num_write_kb":511232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":279,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.907899+0000","last_change":"2024-02-05T06:44:49.443745+0000","last_active":"2024-02-05T06:46:27.907899+0000","last_peered":"2024-02-05T06:46:27.907899+0000","last_clean":"2024-02-05T06:46:27.907899+0000","last_became_active":"2024-02-05T06:40:27.218081+0000","last_became_peered":"2024-02-05T06:40:27.218081+0000","last_unstale":"2024-02-05T06:46:27.907899+0000","last_undegraded":"2024-02-05T06:46:27.907899+0000","last_fullsized":"2024-02-05T06:46:27.907899+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:49.443636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:32.925000+0000","last_clean_scrub_stamp":"2024-02-05T06:44:49.443636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":609,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.911954+0000","last_change":"2024-02-05T06:45:34.691217+0000","last_active":"2024-02-05T06:46:27.911954+0000","last_peered":"2024-02-05T06:46:27.911954+0000","last_clean":"2024-02-05T06:46:27.911954+0000","last_became_active":"2024-02-05T06:41:43.413095+0000","last_became_peered":"2024-02-05T06:41:43.413095+0000","last_unstale":"2024-02-05T06:46:27.911954+0000","last_undegraded":"2024-02-05T06:46:27.911954+0000","last_fullsized":"2024-02-05T06:46:27.911954+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:34.691084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:45:34.691084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:47.092357+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":285,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.910006+0000","last_change":"2024-02-05T06:44:53.524657+0000","last_active":"2024-02-05T06:46:27.910006+0000","last_peered":"2024-02-05T06:46:27.910006+0000","last_clean":"2024-02-05T06:46:27.910006+0000","last_became_active":"2024-02-05T06:41:43.418202+0000","last_became_peered":"2024-02-05T06:41:43.418202+0000","last_unstale":"2024-02-05T06:46:27.910006+0000","last_undegraded":"2024-02-05T06:46:27.910006+0000","last_fullsized":"2024-02-05T06:46:27.910006+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:53.524418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:44:53.524418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:30.838073+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"295'44","reported_seq":738,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:29.593347+0000","last_change":"2024-02-05T06:46:29.593413+0000","last_active":"2024-02-05T06:46:29.593347+0000","last_peered":"2024-02-05T06:46:29.593347+0000","last_clean":"2024-02-05T06:46:29.593347+0000","last_became_active":"2024-02-05T06:40:27.068417+0000","last_became_peered":"2024-02-05T06:40:27.068417+0000","last_unstale":"2024-02-05T06:46:29.593347+0000","last_undegraded":"2024-02-05T06:46:29.593347+0000","last_fullsized":"2024-02-05T06:46:29.593347+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'44","last_scrub_stamp":"2024-02-05T06:46:29.593256+0000","last_deep_scrub":"279'28","last_deep_scrub_stamp":"2024-02-05T06:42:50.148881+0000","last_clean_scrub_stamp":"2024-02-05T06:46:29.593256+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:47:47.491869+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"331'14204","reported_seq":14352,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:31.543026+0000","last_change":"2024-02-05T06:45:54.422133+0000","last_active":"2024-02-05T06:46:31.543026+0000","last_peered":"2024-02-05T06:46:31.543026+0000","last_clean":"2024-02-05T06:46:31.543026+0000","last_became_active":"2024-02-05T06:43:29.842930+0000","last_became_peered":"2024-02-05T06:43:29.842930+0000","last_unstale":"2024-02-05T06:46:31.543026+0000","last_undegraded":"2024-02-05T06:46:31.543026+0000","last_fullsized":"2024-02-05T06:46:31.543026+0000","mapping_epoch":291,"log_start":"321'9200","ondisk_log_start":"321'9200","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'11399","last_scrub_stamp":"2024-02-05T06:45:54.421570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:45:54.421570+0000","objects_scrubbed":4905,"log_size":5004,"log_dups_size":0,"ondisk_log_size":5004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:58.090326+0000","scrub_duration":1259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134873088,"num_objects":2058,"num_object_clones":0,"num_object_copies":6174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14204,"num_write_kb":520384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":572,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-05T06:46:32.169804+0000","last_change":"2024-02-05T06:44:00.315984+0000","last_active":"2024-02-05T06:46:32.169804+0000","last_peered":"2024-02-05T06:46:32.169804+0000","last_clean":"2024-02-05T06:46:32.169804+0000","last_became_active":"2024-02-05T06:40:27.129625+0000","last_became_peered":"2024-02-05T06:40:27.129625+0000","last_unstale":"2024-02-05T06:46:32.169804+0000","last_undegraded":"2024-02-05T06:46:32.169804+0000","last_fullsized":"2024-02-05T06:46:32.169804+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:00.315851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:44:00.315851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":603,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.911560+0000","last_change":"2024-02-05T06:44:55.529116+0000","last_active":"2024-02-05T06:46:27.911560+0000","last_peered":"2024-02-05T06:46:27.911560+0000","last_clean":"2024-02-05T06:46:27.911560+0000","last_became_active":"2024-02-05T06:36:54.966016+0000","last_became_peered":"2024-02-05T06:36:54.966016+0000","last_unstale":"2024-02-05T06:46:27.911560+0000","last_undegraded":"2024-02-05T06:46:27.911560+0000","last_fullsized":"2024-02-05T06:46:27.911560+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:55.528983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:44:55.528983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:31.745733+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":195,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.903122+0000","last_change":"2024-02-05T06:44:48.524505+0000","last_active":"2024-02-05T06:46:27.903122+0000","last_peered":"2024-02-05T06:46:27.903122+0000","last_clean":"2024-02-05T06:46:27.903122+0000","last_became_active":"2024-02-05T06:41:43.412611+0000","last_became_peered":"2024-02-05T06:41:43.412611+0000","last_unstale":"2024-02-05T06:46:27.903122+0000","last_undegraded":"2024-02-05T06:46:27.903122+0000","last_fullsized":"2024-02-05T06:46:27.903122+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:48.524392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:44:48.524392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:30.354946+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"295'15","reported_seq":689,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-05T06:46:32.168281+0000","last_change":"2024-02-05T06:45:49.873565+0000","last_active":"2024-02-05T06:46:32.168281+0000","last_peered":"2024-02-05T06:46:32.168281+0000","last_clean":"2024-02-05T06:46:32.168281+0000","last_became_active":"2024-02-05T06:40:08.584204+0000","last_became_peered":"2024-02-05T06:40:08.584204+0000","last_unstale":"2024-02-05T06:46:32.168281+0000","last_undegraded":"2024-02-05T06:46:32.168281+0000","last_fullsized":"2024-02-05T06:46:32.168281+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:45:49.873251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:45:49.873251+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:55.653031+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"332'14036","reported_seq":14193,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-05T06:46:32.361128+0000","last_change":"2024-02-05T06:45:41.858831+0000","last_active":"2024-02-05T06:46:32.361128+0000","last_peered":"2024-02-05T06:46:32.361128+0000","last_clean":"2024-02-05T06:46:32.361128+0000","last_became_active":"2024-02-05T06:43:29.703452+0000","last_became_peered":"2024-02-05T06:43:29.703452+0000","last_unstale":"2024-02-05T06:46:32.361128+0000","last_undegraded":"2024-02-05T06:46:32.361128+0000","last_fullsized":"2024-02-05T06:46:32.361128+0000","mapping_epoch":291,"log_start":"320'8471","ondisk_log_start":"320'8471","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"324'10381","last_scrub_stamp":"2024-02-05T06:45:41.857376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:45:41.857376+0000","objects_scrubbed":5755,"log_size":5565,"log_dups_size":0,"ondisk_log_size":5565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:59.505416+0000","scrub_duration":1933,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132775936,"num_objects":2026,"num_object_clones":0,"num_object_copies":6078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14036,"num_write_kb":513984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":612,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.903035+0000","last_change":"2024-02-05T06:44:04.452427+0000","last_active":"2024-02-05T06:46:27.903035+0000","last_peered":"2024-02-05T06:46:27.903035+0000","last_clean":"2024-02-05T06:46:27.903035+0000","last_became_active":"2024-02-05T06:41:46.054968+0000","last_became_peered":"2024-02-05T06:41:46.054968+0000","last_unstale":"2024-02-05T06:46:27.903035+0000","last_undegraded":"2024-02-05T06:46:27.903035+0000","last_fullsized":"2024-02-05T06:46:27.903035+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:44:04.452308+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:37.282442+0000","last_clean_scrub_stamp":"2024-02-05T06:44:04.452308+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:30.354676+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":579,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:22.881528+0000","last_change":"2024-02-05T06:43:40.471046+0000","last_active":"2024-02-05T06:46:22.881528+0000","last_peered":"2024-02-05T06:46:22.881528+0000","last_clean":"2024-02-05T06:46:22.881528+0000","last_became_active":"2024-02-05T06:40:27.082440+0000","last_became_peered":"2024-02-05T06:40:27.082440+0000","last_unstale":"2024-02-05T06:46:22.881528+0000","last_undegraded":"2024-02-05T06:46:22.881528+0000","last_fullsized":"2024-02-05T06:46:22.881528+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:40.470922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:43:40.470922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:24.288848+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"182'3","reported_seq":199,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.903141+0000","last_change":"2024-02-05T06:45:44.164026+0000","last_active":"2024-02-05T06:46:27.903141+0000","last_peered":"2024-02-05T06:46:27.903141+0000","last_clean":"2024-02-05T06:46:27.903141+0000","last_became_active":"2024-02-05T06:41:43.415806+0000","last_became_peered":"2024-02-05T06:41:43.415806+0000","last_unstale":"2024-02-05T06:46:27.903141+0000","last_undegraded":"2024-02-05T06:46:27.903141+0000","last_fullsized":"2024-02-05T06:46:27.903141+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'3","last_scrub_stamp":"2024-02-05T06:45:44.163910+0000","last_deep_scrub":"182'3","last_deep_scrub_stamp":"2024-02-05T06:39:33.373415+0000","last_clean_scrub_stamp":"2024-02-05T06:45:44.163910+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:47.117860+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0.001220348,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":109,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-05T06:46:32.168306+0000","last_change":"2024-02-05T06:45:34.964039+0000","last_active":"2024-02-05T06:46:32.168306+0000","last_peered":"2024-02-05T06:46:32.168306+0000","last_clean":"2024-02-05T06:46:32.168306+0000","last_became_active":"2024-02-05T06:45:09.776169+0000","last_became_peered":"2024-02-05T06:45:09.776169+0000","last_unstale":"2024-02-05T06:46:32.168306+0000","last_undegraded":"2024-02-05T06:46:32.168306+0000","last_fullsized":"2024-02-05T06:46:32.168306+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:34.963923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:45:34.963923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:41.620101+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"326'52","reported_seq":427,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.900697+0000","last_change":"2024-02-05T06:45:53.715638+0000","last_active":"2024-02-05T06:46:27.900697+0000","last_peered":"2024-02-05T06:46:27.900697+0000","last_clean":"2024-02-05T06:46:27.900697+0000","last_became_active":"2024-02-05T06:40:08.604483+0000","last_became_peered":"2024-02-05T06:40:08.604483+0000","last_unstale":"2024-02-05T06:46:27.900697+0000","last_undegraded":"2024-02-05T06:46:27.900697+0000","last_fullsized":"2024-02-05T06:46:27.900697+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'52","last_scrub_stamp":"2024-02-05T06:45:53.715482+0000","last_deep_scrub":"326'52","last_deep_scrub_stamp":"2024-02-05T06:45:53.715482+0000","last_clean_scrub_stamp":"2024-02-05T06:45:53.715482+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:47:03.040655+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":585,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-05T06:46:32.167930+0000","last_change":"2024-02-05T06:44:53.034324+0000","last_active":"2024-02-05T06:46:32.167930+0000","last_peered":"2024-02-05T06:46:32.167930+0000","last_clean":"2024-02-05T06:46:32.167930+0000","last_became_active":"2024-02-05T06:36:54.973387+0000","last_became_peered":"2024-02-05T06:36:54.973387+0000","last_unstale":"2024-02-05T06:46:32.167930+0000","last_undegraded":"2024-02-05T06:46:32.167930+0000","last_fullsized":"2024-02-05T06:46:32.167930+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:53.034202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:44:53.034202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"331'14115","reported_seq":14261,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:31.548370+0000","last_change":"2024-02-05T06:45:56.357229+0000","last_active":"2024-02-05T06:46:31.548370+0000","last_peered":"2024-02-05T06:46:31.548370+0000","last_clean":"2024-02-05T06:46:31.548370+0000","last_became_active":"2024-02-05T06:43:29.702401+0000","last_became_peered":"2024-02-05T06:43:29.702401+0000","last_unstale":"2024-02-05T06:46:31.548370+0000","last_undegraded":"2024-02-05T06:46:31.548370+0000","last_fullsized":"2024-02-05T06:46:31.548370+0000","mapping_epoch":291,"log_start":"321'9100","ondisk_log_start":"321'9100","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'11553","last_scrub_stamp":"2024-02-05T06:45:56.357067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:45:56.357067+0000","objects_scrubbed":4736,"log_size":5015,"log_dups_size":0,"ondisk_log_size":5015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:47:04.016758+0000","scrub_duration":1198,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139657216,"num_objects":2131,"num_object_clones":0,"num_object_copies":6393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14115,"num_write_kb":519872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":700,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.909039+0000","last_change":"2024-02-05T06:44:47.520919+0000","last_active":"2024-02-05T06:46:27.909039+0000","last_peered":"2024-02-05T06:46:27.909039+0000","last_clean":"2024-02-05T06:46:27.909039+0000","last_became_active":"2024-02-05T06:40:30.082467+0000","last_became_peered":"2024-02-05T06:40:30.082467+0000","last_unstale":"2024-02-05T06:46:27.909039+0000","last_undegraded":"2024-02-05T06:46:27.909039+0000","last_fullsized":"2024-02-05T06:46:27.909039+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:44:47.520760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:44:47.520760+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:30.838045+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":279,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.909837+0000","last_change":"2024-02-05T06:44:59.464189+0000","last_active":"2024-02-05T06:46:27.909837+0000","last_peered":"2024-02-05T06:46:27.909837+0000","last_clean":"2024-02-05T06:46:27.909837+0000","last_became_active":"2024-02-05T06:40:27.077966+0000","last_became_peered":"2024-02-05T06:40:27.077966+0000","last_unstale":"2024-02-05T06:46:27.909837+0000","last_undegraded":"2024-02-05T06:46:27.909837+0000","last_fullsized":"2024-02-05T06:46:27.909837+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:59.463573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:44:59.463573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":578,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.905396+0000","last_change":"2024-02-05T06:43:46.552644+0000","last_active":"2024-02-05T06:46:27.905396+0000","last_peered":"2024-02-05T06:46:27.905396+0000","last_clean":"2024-02-05T06:46:27.905396+0000","last_became_active":"2024-02-05T06:33:29.551549+0000","last_became_peered":"2024-02-05T06:33:29.551549+0000","last_unstale":"2024-02-05T06:46:27.905396+0000","last_undegraded":"2024-02-05T06:46:27.905396+0000","last_fullsized":"2024-02-05T06:46:27.905396+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:46.552530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:43:46.552530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":601,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.911507+0000","last_change":"2024-02-05T06:44:51.548376+0000","last_active":"2024-02-05T06:46:27.911507+0000","last_peered":"2024-02-05T06:46:27.911507+0000","last_clean":"2024-02-05T06:46:27.911507+0000","last_became_active":"2024-02-05T06:33:28.542977+0000","last_became_peered":"2024-02-05T06:33:28.542977+0000","last_unstale":"2024-02-05T06:46:27.911507+0000","last_undegraded":"2024-02-05T06:46:27.911507+0000","last_fullsized":"2024-02-05T06:46:27.911507+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:51.548204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:44:51.548204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:31.745763+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.00091640799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"295'17","reported_seq":676,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.901141+0000","last_change":"2024-02-05T06:44:48.709669+0000","last_active":"2024-02-05T06:46:27.901141+0000","last_peered":"2024-02-05T06:46:27.901141+0000","last_clean":"2024-02-05T06:46:27.901141+0000","last_became_active":"2024-02-05T06:41:43.420458+0000","last_became_peered":"2024-02-05T06:41:43.420458+0000","last_unstale":"2024-02-05T06:46:27.901141+0000","last_undegraded":"2024-02-05T06:46:27.901141+0000","last_fullsized":"2024-02-05T06:46:27.901141+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'17","last_scrub_stamp":"2024-02-05T06:44:48.709541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:44:48.709541+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":615,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.913790+0000","last_change":"2024-02-05T06:44:44.519966+0000","last_active":"2024-02-05T06:46:27.913790+0000","last_peered":"2024-02-05T06:46:27.913790+0000","last_clean":"2024-02-05T06:46:27.913790+0000","last_became_active":"2024-02-05T06:41:43.412387+0000","last_became_peered":"2024-02-05T06:41:43.412387+0000","last_unstale":"2024-02-05T06:46:27.913790+0000","last_undegraded":"2024-02-05T06:46:27.913790+0000","last_fullsized":"2024-02-05T06:46:27.913790+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:44.519682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:33.914422+0000","last_clean_scrub_stamp":"2024-02-05T06:44:44.519682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:31.745819+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"330'13862","reported_seq":13951,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.502936+0000","last_change":"2024-02-05T06:43:29.701823+0000","last_active":"2024-02-05T06:46:27.502936+0000","last_peered":"2024-02-05T06:46:27.502936+0000","last_clean":"2024-02-05T06:46:27.502936+0000","last_became_active":"2024-02-05T06:43:29.701487+0000","last_became_peered":"2024-02-05T06:43:29.701487+0000","last_unstale":"2024-02-05T06:46:27.502936+0000","last_undegraded":"2024-02-05T06:46:27.502936+0000","last_fullsized":"2024-02-05T06:46:27.502936+0000","mapping_epoch":291,"log_start":"321'9400","ondisk_log_start":"321'9400","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4462,"log_dups_size":0,"ondisk_log_size":4462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158072832,"num_objects":2412,"num_object_clones":0,"num_object_copies":7236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13862,"num_write_kb":520768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":195,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.903713+0000","last_change":"2024-02-05T06:44:40.571834+0000","last_active":"2024-02-05T06:46:27.903713+0000","last_peered":"2024-02-05T06:46:27.903713+0000","last_clean":"2024-02-05T06:46:27.903713+0000","last_became_active":"2024-02-05T06:41:43.408773+0000","last_became_peered":"2024-02-05T06:41:43.408773+0000","last_unstale":"2024-02-05T06:46:27.903713+0000","last_undegraded":"2024-02-05T06:46:27.903713+0000","last_fullsized":"2024-02-05T06:46:27.903713+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:40.571610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:44:40.571610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:30.354701+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":701,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.907985+0000","last_change":"2024-02-05T06:44:50.489745+0000","last_active":"2024-02-05T06:46:27.907985+0000","last_peered":"2024-02-05T06:46:27.907985+0000","last_clean":"2024-02-05T06:46:27.907985+0000","last_became_active":"2024-02-05T06:40:30.081178+0000","last_became_peered":"2024-02-05T06:40:30.081178+0000","last_unstale":"2024-02-05T06:46:27.907985+0000","last_undegraded":"2024-02-05T06:46:27.907985+0000","last_fullsized":"2024-02-05T06:46:27.907985+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:44:50.489566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:44:50.489566+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":588,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:22.881835+0000","last_change":"2024-02-05T06:45:57.195429+0000","last_active":"2024-02-05T06:46:22.881835+0000","last_peered":"2024-02-05T06:46:22.881835+0000","last_clean":"2024-02-05T06:46:22.881835+0000","last_became_active":"2024-02-05T06:41:43.411335+0000","last_became_peered":"2024-02-05T06:41:43.411335+0000","last_unstale":"2024-02-05T06:46:22.881835+0000","last_undegraded":"2024-02-05T06:46:22.881835+0000","last_fullsized":"2024-02-05T06:46:22.881835+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:57.195321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:45:57.195321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:47:11.529838+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"295'32","reported_seq":884,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.898108+0000","last_change":"2024-02-05T06:45:36.749656+0000","last_active":"2024-02-05T06:46:27.898108+0000","last_peered":"2024-02-05T06:46:27.898108+0000","last_clean":"2024-02-05T06:46:27.898108+0000","last_became_active":"2024-02-05T06:40:27.061111+0000","last_became_peered":"2024-02-05T06:40:27.061111+0000","last_unstale":"2024-02-05T06:46:27.898108+0000","last_undegraded":"2024-02-05T06:46:27.898108+0000","last_fullsized":"2024-02-05T06:46:27.898108+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'32","last_scrub_stamp":"2024-02-05T06:45:36.749508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:45:36.749508+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:56.362245+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":293,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.910278+0000","last_change":"2024-02-05T06:45:49.152771+0000","last_active":"2024-02-05T06:46:27.910278+0000","last_peered":"2024-02-05T06:46:27.910278+0000","last_clean":"2024-02-05T06:46:27.910278+0000","last_became_active":"2024-02-05T06:41:43.418198+0000","last_became_peered":"2024-02-05T06:41:43.418198+0000","last_unstale":"2024-02-05T06:46:27.910278+0000","last_undegraded":"2024-02-05T06:46:27.910278+0000","last_fullsized":"2024-02-05T06:46:27.910278+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:49.152682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:45:49.152682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:47:02.809579+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"332'14259","reported_seq":14421,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-05T06:46:32.352036+0000","last_change":"2024-02-05T06:44:41.914941+0000","last_active":"2024-02-05T06:46:32.352036+0000","last_peered":"2024-02-05T06:46:32.352036+0000","last_clean":"2024-02-05T06:46:32.352036+0000","last_became_active":"2024-02-05T06:43:29.946961+0000","last_became_peered":"2024-02-05T06:43:29.946961+0000","last_unstale":"2024-02-05T06:46:32.352036+0000","last_undegraded":"2024-02-05T06:46:32.352036+0000","last_fullsized":"2024-02-05T06:46:32.352036+0000","mapping_epoch":291,"log_start":"320'8627","ondisk_log_start":"320'8627","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"312'6240","last_scrub_stamp":"2024-02-05T06:44:41.914681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:44:41.914681+0000","objects_scrubbed":6181,"log_size":5632,"log_dups_size":0,"ondisk_log_size":5632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135331840,"num_objects":2065,"num_object_clones":0,"num_object_copies":6195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2065,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14259,"num_write_kb":522368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":741,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.903302+0000","last_change":"2024-02-05T06:44:42.664214+0000","last_active":"2024-02-05T06:46:27.903302+0000","last_peered":"2024-02-05T06:46:27.903302+0000","last_clean":"2024-02-05T06:46:27.903302+0000","last_became_active":"2024-02-05T06:37:37.137754+0000","last_became_peered":"2024-02-05T06:37:37.137754+0000","last_unstale":"2024-02-05T06:46:27.903302+0000","last_undegraded":"2024-02-05T06:46:27.903302+0000","last_fullsized":"2024-02-05T06:46:27.903302+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:44:42.664095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:44:42.664095+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":594,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:31.716290+0000","last_change":"2024-02-05T06:46:31.716329+0000","last_active":"2024-02-05T06:46:31.716290+0000","last_peered":"2024-02-05T06:46:31.716290+0000","last_clean":"2024-02-05T06:46:31.716290+0000","last_became_active":"2024-02-05T06:41:43.410910+0000","last_became_peered":"2024-02-05T06:41:43.410910+0000","last_unstale":"2024-02-05T06:46:31.716290+0000","last_undegraded":"2024-02-05T06:46:31.716290+0000","last_fullsized":"2024-02-05T06:46:31.716290+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:31.716099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:46:31.716099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:47:39.508366+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":593,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-05T06:46:32.169507+0000","last_change":"2024-02-05T06:44:58.042869+0000","last_active":"2024-02-05T06:46:32.169507+0000","last_peered":"2024-02-05T06:46:32.169507+0000","last_clean":"2024-02-05T06:46:32.169507+0000","last_became_active":"2024-02-05T06:36:54.972264+0000","last_became_peered":"2024-02-05T06:36:54.972264+0000","last_unstale":"2024-02-05T06:46:32.169507+0000","last_undegraded":"2024-02-05T06:46:32.169507+0000","last_fullsized":"2024-02-05T06:46:32.169507+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:58.042462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:44:58.042462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":651,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:29.625211+0000","last_change":"2024-02-05T06:45:39.224123+0000","last_active":"2024-02-05T06:46:29.625211+0000","last_peered":"2024-02-05T06:46:29.625211+0000","last_clean":"2024-02-05T06:46:29.625211+0000","last_became_active":"2024-02-05T06:40:27.071509+0000","last_became_peered":"2024-02-05T06:40:27.071509+0000","last_unstale":"2024-02-05T06:46:29.625211+0000","last_undegraded":"2024-02-05T06:46:29.625211+0000","last_fullsized":"2024-02-05T06:46:29.625211+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-02-05T06:45:39.224013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:45:39.224013+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:47.009162+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.00074151600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":370,"num_read_kb":365,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"295'24","reported_seq":613,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-05T06:46:32.169153+0000","last_change":"2024-02-05T06:44:44.938340+0000","last_active":"2024-02-05T06:46:32.169153+0000","last_peered":"2024-02-05T06:46:32.169153+0000","last_clean":"2024-02-05T06:46:32.169153+0000","last_became_active":"2024-02-05T06:33:31.564604+0000","last_became_peered":"2024-02-05T06:33:31.564604+0000","last_unstale":"2024-02-05T06:46:32.169153+0000","last_undegraded":"2024-02-05T06:46:32.169153+0000","last_fullsized":"2024-02-05T06:46:32.169153+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:44:44.938187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:44:44.938187+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":195,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.903755+0000","last_change":"2024-02-05T06:45:46.133321+0000","last_active":"2024-02-05T06:46:27.903755+0000","last_peered":"2024-02-05T06:46:27.903755+0000","last_clean":"2024-02-05T06:46:27.903755+0000","last_became_active":"2024-02-05T06:41:43.417751+0000","last_became_peered":"2024-02-05T06:41:43.417751+0000","last_unstale":"2024-02-05T06:46:27.903755+0000","last_undegraded":"2024-02-05T06:46:27.903755+0000","last_fullsized":"2024-02-05T06:46:27.903755+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:46.133190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:45:46.133190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:50.104920+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"331'14129","reported_seq":14283,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:31.549037+0000","last_change":"2024-02-05T06:45:45.731638+0000","last_active":"2024-02-05T06:46:31.549037+0000","last_peered":"2024-02-05T06:46:31.549037+0000","last_clean":"2024-02-05T06:46:31.549037+0000","last_became_active":"2024-02-05T06:43:29.702549+0000","last_became_peered":"2024-02-05T06:43:29.702549+0000","last_unstale":"2024-02-05T06:46:31.549037+0000","last_undegraded":"2024-02-05T06:46:31.549037+0000","last_fullsized":"2024-02-05T06:46:31.549037+0000","mapping_epoch":291,"log_start":"322'9500","ondisk_log_start":"322'9500","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'10849","last_scrub_stamp":"2024-02-05T06:45:45.731520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:45:45.731520+0000","objects_scrubbed":5452,"log_size":4629,"log_dups_size":0,"ondisk_log_size":4629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:47:10.241303+0000","scrub_duration":1526,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138870784,"num_objects":2119,"num_object_clones":0,"num_object_copies":6357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14129,"num_write_kb":519936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":754,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.898185+0000","last_change":"2024-02-05T06:45:42.619837+0000","last_active":"2024-02-05T06:46:27.898185+0000","last_peered":"2024-02-05T06:46:27.898185+0000","last_clean":"2024-02-05T06:46:27.898185+0000","last_became_active":"2024-02-05T06:37:37.140011+0000","last_became_peered":"2024-02-05T06:37:37.140011+0000","last_unstale":"2024-02-05T06:46:27.898185+0000","last_undegraded":"2024-02-05T06:46:27.898185+0000","last_fullsized":"2024-02-05T06:46:27.898185+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:45:42.619751+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:38.778568+0000","last_clean_scrub_stamp":"2024-02-05T06:45:42.619751+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:44.358539+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":574,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.900781+0000","last_change":"2024-02-05T06:44:46.776072+0000","last_active":"2024-02-05T06:46:27.900781+0000","last_peered":"2024-02-05T06:46:27.900781+0000","last_clean":"2024-02-05T06:46:27.900781+0000","last_became_active":"2024-02-05T06:33:35.664858+0000","last_became_peered":"2024-02-05T06:33:35.664858+0000","last_unstale":"2024-02-05T06:46:27.900781+0000","last_undegraded":"2024-02-05T06:46:27.900781+0000","last_fullsized":"2024-02-05T06:46:27.900781+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:46.775962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:44:46.775962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:28.733401+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"66'1","reported_seq":620,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:30.749159+0000","last_change":"2024-02-05T06:46:30.749245+0000","last_active":"2024-02-05T06:46:30.749159+0000","last_peered":"2024-02-05T06:46:30.749159+0000","last_clean":"2024-02-05T06:46:30.749159+0000","last_became_active":"2024-02-05T06:41:43.411404+0000","last_became_peered":"2024-02-05T06:41:43.411404+0000","last_unstale":"2024-02-05T06:46:30.749159+0000","last_undegraded":"2024-02-05T06:46:30.749159+0000","last_fullsized":"2024-02-05T06:46:30.749159+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:46:30.748577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:46:30.748577+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:47:58.022569+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"295'29","reported_seq":724,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-05T06:46:32.168489+0000","last_change":"2024-02-05T06:45:01.037760+0000","last_active":"2024-02-05T06:46:32.168489+0000","last_peered":"2024-02-05T06:46:32.168489+0000","last_clean":"2024-02-05T06:46:32.168489+0000","last_became_active":"2024-02-05T06:41:43.409664+0000","last_became_peered":"2024-02-05T06:41:43.409664+0000","last_unstale":"2024-02-05T06:46:32.168489+0000","last_undegraded":"2024-02-05T06:46:32.168489+0000","last_fullsized":"2024-02-05T06:46:32.168489+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'29","last_scrub_stamp":"2024-02-05T06:45:01.036847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:45:01.036847+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":597,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:30.561575+0000","last_change":"2024-02-05T06:46:30.561622+0000","last_active":"2024-02-05T06:46:30.561575+0000","last_peered":"2024-02-05T06:46:30.561575+0000","last_clean":"2024-02-05T06:46:30.561575+0000","last_became_active":"2024-02-05T06:41:43.411550+0000","last_became_peered":"2024-02-05T06:41:43.411550+0000","last_unstale":"2024-02-05T06:46:30.561575+0000","last_undegraded":"2024-02-05T06:46:30.561575+0000","last_fullsized":"2024-02-05T06:46:30.561575+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:30.561494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:46:30.561494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:47:36.317107+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"331'14159","reported_seq":14333,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:31.543464+0000","last_change":"2024-02-05T06:45:01.647851+0000","last_active":"2024-02-05T06:46:31.543464+0000","last_peered":"2024-02-05T06:46:31.543464+0000","last_clean":"2024-02-05T06:46:31.543464+0000","last_became_active":"2024-02-05T06:43:29.700331+0000","last_became_peered":"2024-02-05T06:43:29.700331+0000","last_unstale":"2024-02-05T06:46:31.543464+0000","last_undegraded":"2024-02-05T06:46:31.543464+0000","last_fullsized":"2024-02-05T06:46:31.543464+0000","mapping_epoch":291,"log_start":"321'9100","ondisk_log_start":"321'9100","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'7624","last_scrub_stamp":"2024-02-05T06:45:01.647471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:45:01.647471+0000","objects_scrubbed":7549,"log_size":5059,"log_dups_size":0,"ondisk_log_size":5059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3192,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138608640,"num_objects":2115,"num_object_clones":0,"num_object_copies":6345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14159,"num_write_kb":520768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":773,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.898104+0000","last_change":"2024-02-05T06:44:41.558027+0000","last_active":"2024-02-05T06:46:27.898104+0000","last_peered":"2024-02-05T06:46:27.898104+0000","last_clean":"2024-02-05T06:46:27.898104+0000","last_became_active":"2024-02-05T06:40:30.085438+0000","last_became_peered":"2024-02-05T06:40:30.085438+0000","last_unstale":"2024-02-05T06:46:27.898104+0000","last_undegraded":"2024-02-05T06:46:27.898104+0000","last_fullsized":"2024-02-05T06:46:27.898104+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:44:41.557909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:44:41.557909+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":569,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-05T06:46:32.173164+0000","last_change":"2024-02-05T06:45:48.925337+0000","last_active":"2024-02-05T06:46:32.173164+0000","last_peered":"2024-02-05T06:46:32.173164+0000","last_clean":"2024-02-05T06:46:32.173164+0000","last_became_active":"2024-02-05T06:33:35.600485+0000","last_became_peered":"2024-02-05T06:33:35.600485+0000","last_unstale":"2024-02-05T06:46:32.173164+0000","last_undegraded":"2024-02-05T06:46:32.173164+0000","last_fullsized":"2024-02-05T06:46:32.173164+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:48.925218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:45:48.925218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:47:16.692434+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":187,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.909079+0000","last_change":"2024-02-05T06:43:35.458031+0000","last_active":"2024-02-05T06:46:27.909079+0000","last_peered":"2024-02-05T06:46:27.909079+0000","last_clean":"2024-02-05T06:46:27.909079+0000","last_became_active":"2024-02-05T06:41:43.411732+0000","last_became_peered":"2024-02-05T06:41:43.411732+0000","last_unstale":"2024-02-05T06:46:27.909079+0000","last_undegraded":"2024-02-05T06:46:27.909079+0000","last_fullsized":"2024-02-05T06:46:27.909079+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:35.457940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:28.353956+0000","last_clean_scrub_stamp":"2024-02-05T06:43:35.457940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:30.354542+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"295'20","reported_seq":684,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.907372+0000","last_change":"2024-02-05T06:45:51.645898+0000","last_active":"2024-02-05T06:46:27.907372+0000","last_peered":"2024-02-05T06:46:27.907372+0000","last_clean":"2024-02-05T06:46:27.907372+0000","last_became_active":"2024-02-05T06:41:43.421030+0000","last_became_peered":"2024-02-05T06:41:43.421030+0000","last_unstale":"2024-02-05T06:46:27.907372+0000","last_undegraded":"2024-02-05T06:46:27.907372+0000","last_fullsized":"2024-02-05T06:46:27.907372+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'20","last_scrub_stamp":"2024-02-05T06:45:51.645768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:45:51.645768+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:47:06.901859+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"249'10","reported_seq":758,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-05T06:46:32.168197+0000","last_change":"2024-02-05T06:44:42.892873+0000","last_active":"2024-02-05T06:46:32.168197+0000","last_peered":"2024-02-05T06:46:32.168197+0000","last_clean":"2024-02-05T06:46:32.168197+0000","last_became_active":"2024-02-05T06:41:43.423638+0000","last_became_peered":"2024-02-05T06:41:43.423638+0000","last_unstale":"2024-02-05T06:46:32.168197+0000","last_undegraded":"2024-02-05T06:46:32.168197+0000","last_fullsized":"2024-02-05T06:46:32.168197+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'10","last_scrub_stamp":"2024-02-05T06:44:42.892604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:44:42.892604+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"330'13790","reported_seq":13879,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.517919+0000","last_change":"2024-02-05T06:43:29.702358+0000","last_active":"2024-02-05T06:46:27.517919+0000","last_peered":"2024-02-05T06:46:27.517919+0000","last_clean":"2024-02-05T06:46:27.517919+0000","last_became_active":"2024-02-05T06:43:29.701716+0000","last_became_peered":"2024-02-05T06:43:29.701716+0000","last_unstale":"2024-02-05T06:46:27.517919+0000","last_undegraded":"2024-02-05T06:46:27.517919+0000","last_fullsized":"2024-02-05T06:46:27.517919+0000","mapping_epoch":291,"log_start":"321'9300","ondisk_log_start":"321'9300","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163315712,"num_objects":2492,"num_object_clones":0,"num_object_copies":7476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2492,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13790,"num_write_kb":521024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":782,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:22.884694+0000","last_change":"2024-02-05T06:45:54.266666+0000","last_active":"2024-02-05T06:46:22.884694+0000","last_peered":"2024-02-05T06:46:22.884694+0000","last_clean":"2024-02-05T06:46:22.884694+0000","last_became_active":"2024-02-05T06:41:46.053837+0000","last_became_peered":"2024-02-05T06:41:46.053837+0000","last_unstale":"2024-02-05T06:46:22.884694+0000","last_undegraded":"2024-02-05T06:46:22.884694+0000","last_fullsized":"2024-02-05T06:46:22.884694+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:45:54.266536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:45:54.266536+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:55.018242+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":279,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.909312+0000","last_change":"2024-02-05T06:44:58.469801+0000","last_active":"2024-02-05T06:46:27.909312+0000","last_peered":"2024-02-05T06:46:27.909312+0000","last_clean":"2024-02-05T06:46:27.909312+0000","last_became_active":"2024-02-05T06:40:27.099168+0000","last_became_peered":"2024-02-05T06:40:27.099168+0000","last_unstale":"2024-02-05T06:46:27.909312+0000","last_undegraded":"2024-02-05T06:46:27.909312+0000","last_fullsized":"2024-02-05T06:46:27.909312+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:58.469560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:44:58.469560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:30.838122+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"95'2","reported_seq":603,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-05T06:46:32.169203+0000","last_change":"2024-02-05T06:45:44.911946+0000","last_active":"2024-02-05T06:46:32.169203+0000","last_peered":"2024-02-05T06:46:32.169203+0000","last_clean":"2024-02-05T06:46:32.169203+0000","last_became_active":"2024-02-05T06:36:54.975093+0000","last_became_peered":"2024-02-05T06:36:54.975093+0000","last_unstale":"2024-02-05T06:46:32.169203+0000","last_undegraded":"2024-02-05T06:46:32.169203+0000","last_fullsized":"2024-02-05T06:46:32.169203+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-05T06:45:44.911816+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-05T06:35:25.616708+0000","last_clean_scrub_stamp":"2024-02-05T06:45:44.911816+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:47:06.054059+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"295'28","reported_seq":505,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.902847+0000","last_change":"2024-02-05T06:45:39.141292+0000","last_active":"2024-02-05T06:46:27.902847+0000","last_peered":"2024-02-05T06:46:27.902847+0000","last_clean":"2024-02-05T06:46:27.902847+0000","last_became_active":"2024-02-05T06:45:09.764696+0000","last_became_peered":"2024-02-05T06:45:09.764696+0000","last_unstale":"2024-02-05T06:46:27.902847+0000","last_undegraded":"2024-02-05T06:46:27.902847+0000","last_fullsized":"2024-02-05T06:46:27.902847+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'28","last_scrub_stamp":"2024-02-05T06:45:39.141209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:45:39.141209+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:47.828056+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":40,"num_read_kb":33,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"326'11","reported_seq":755,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:22.881238+0000","last_change":"2024-02-05T06:45:56.212330+0000","last_active":"2024-02-05T06:46:22.881238+0000","last_peered":"2024-02-05T06:46:22.881238+0000","last_clean":"2024-02-05T06:46:22.881238+0000","last_became_active":"2024-02-05T06:36:54.954819+0000","last_became_peered":"2024-02-05T06:36:54.954819+0000","last_unstale":"2024-02-05T06:46:22.881238+0000","last_undegraded":"2024-02-05T06:46:22.881238+0000","last_fullsized":"2024-02-05T06:46:22.881238+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'11","last_scrub_stamp":"2024-02-05T06:45:56.212177+0000","last_deep_scrub":"326'11","last_deep_scrub_stamp":"2024-02-05T06:45:56.212177+0000","last_clean_scrub_stamp":"2024-02-05T06:45:56.212177+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:47:22.165080+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"331'14134","reported_seq":14297,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:31.514680+0000","last_change":"2024-02-05T06:45:35.115745+0000","last_active":"2024-02-05T06:46:31.514680+0000","last_peered":"2024-02-05T06:46:31.514680+0000","last_clean":"2024-02-05T06:46:31.514680+0000","last_became_active":"2024-02-05T06:43:29.703700+0000","last_became_peered":"2024-02-05T06:43:29.703700+0000","last_unstale":"2024-02-05T06:46:31.514680+0000","last_undegraded":"2024-02-05T06:46:31.514680+0000","last_fullsized":"2024-02-05T06:46:31.514680+0000","mapping_epoch":291,"log_start":"323'9500","ondisk_log_start":"323'9500","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"324'10059","last_scrub_stamp":"2024-02-05T06:45:35.115134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:45:35.115134+0000","objects_scrubbed":6315,"log_size":4634,"log_dups_size":0,"ondisk_log_size":4634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:54.901316+0000","scrub_duration":1878,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142016536,"num_objects":2168,"num_object_clones":0,"num_object_copies":6504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2168,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14134,"num_write_kb":521601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":343,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.912627+0000","last_change":"2024-02-05T06:45:32.666834+0000","last_active":"2024-02-05T06:46:27.912627+0000","last_peered":"2024-02-05T06:46:27.912627+0000","last_clean":"2024-02-05T06:46:27.912627+0000","last_became_active":"2024-02-05T06:45:09.776631+0000","last_became_peered":"2024-02-05T06:45:09.776631+0000","last_unstale":"2024-02-05T06:46:27.912627+0000","last_undegraded":"2024-02-05T06:46:27.912627+0000","last_fullsized":"2024-02-05T06:46:27.912627+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:45:32.666711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:45:32.666711+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:49.264806+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":106,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:22.881975+0000","last_change":"2024-02-05T06:45:49.145234+0000","last_active":"2024-02-05T06:46:22.881975+0000","last_peered":"2024-02-05T06:46:22.881975+0000","last_clean":"2024-02-05T06:46:22.881975+0000","last_became_active":"2024-02-05T06:45:09.780390+0000","last_became_peered":"2024-02-05T06:45:09.780390+0000","last_unstale":"2024-02-05T06:46:22.881975+0000","last_undegraded":"2024-02-05T06:46:22.881975+0000","last_fullsized":"2024-02-05T06:46:22.881975+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:49.145117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:45:49.145117+0000","last_clean_scrub_stamp":"2024-02-05T06:45:49.145117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:47:12.441915+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":593,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:22.883490+0000","last_change":"2024-02-05T06:45:42.080964+0000","last_active":"2024-02-05T06:46:22.883490+0000","last_peered":"2024-02-05T06:46:22.883490+0000","last_clean":"2024-02-05T06:46:22.883490+0000","last_became_active":"2024-02-05T06:40:27.093739+0000","last_became_peered":"2024-02-05T06:40:27.093739+0000","last_unstale":"2024-02-05T06:46:22.883490+0000","last_undegraded":"2024-02-05T06:46:22.883490+0000","last_fullsized":"2024-02-05T06:46:22.883490+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:42.080862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:38:28.150695+0000","last_clean_scrub_stamp":"2024-02-05T06:45:42.080862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:48.342066+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"295'30","reported_seq":662,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:22.881103+0000","last_change":"2024-02-05T06:43:46.582784+0000","last_active":"2024-02-05T06:46:22.881103+0000","last_peered":"2024-02-05T06:46:22.881103+0000","last_clean":"2024-02-05T06:46:22.881103+0000","last_became_active":"2024-02-05T06:40:27.277747+0000","last_became_peered":"2024-02-05T06:40:27.277747+0000","last_unstale":"2024-02-05T06:46:22.881103+0000","last_undegraded":"2024-02-05T06:46:22.881103+0000","last_fullsized":"2024-02-05T06:46:22.881103+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'30","last_scrub_stamp":"2024-02-05T06:43:46.582265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:43:46.582265+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:24.289207+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":239,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:22.882971+0000","last_change":"2024-02-05T06:45:09.778936+0000","last_active":"2024-02-05T06:46:22.882971+0000","last_peered":"2024-02-05T06:46:22.882971+0000","last_clean":"2024-02-05T06:46:22.882971+0000","last_became_active":"2024-02-05T06:45:09.778609+0000","last_became_peered":"2024-02-05T06:45:09.778609+0000","last_unstale":"2024-02-05T06:46:22.882971+0000","last_undegraded":"2024-02-05T06:46:22.882971+0000","last_fullsized":"2024-02-05T06:46:22.882971+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:42:52.703089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:42:52.703089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:24.284029+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2","version":"330'13589","reported_seq":13677,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.528555+0000","last_change":"2024-02-05T06:43:29.843237+0000","last_active":"2024-02-05T06:46:27.528555+0000","last_peered":"2024-02-05T06:46:27.528555+0000","last_clean":"2024-02-05T06:46:27.528555+0000","last_became_active":"2024-02-05T06:43:29.842722+0000","last_became_peered":"2024-02-05T06:43:29.842722+0000","last_unstale":"2024-02-05T06:46:27.528555+0000","last_undegraded":"2024-02-05T06:46:27.528555+0000","last_fullsized":"2024-02-05T06:46:27.528555+0000","mapping_epoch":291,"log_start":"321'9100","ondisk_log_start":"321'9100","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4489,"log_dups_size":0,"ondisk_log_size":4489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159318016,"num_objects":2431,"num_object_clones":0,"num_object_copies":7293,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13589,"num_write_kb":512640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":606,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.903067+0000","last_change":"2024-02-05T06:43:59.471960+0000","last_active":"2024-02-05T06:46:27.903067+0000","last_peered":"2024-02-05T06:46:27.903067+0000","last_clean":"2024-02-05T06:46:27.903067+0000","last_became_active":"2024-02-05T06:41:46.053993+0000","last_became_peered":"2024-02-05T06:41:46.053993+0000","last_unstale":"2024-02-05T06:46:27.903067+0000","last_undegraded":"2024-02-05T06:46:27.903067+0000","last_fullsized":"2024-02-05T06:46:27.903067+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:43:59.471749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:43:59.471749+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:30.354517+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":195,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.908990+0000","last_change":"2024-02-05T06:44:00.522165+0000","last_active":"2024-02-05T06:46:27.908990+0000","last_peered":"2024-02-05T06:46:27.908990+0000","last_clean":"2024-02-05T06:46:27.908990+0000","last_became_active":"2024-02-05T06:41:43.418066+0000","last_became_peered":"2024-02-05T06:41:43.418066+0000","last_unstale":"2024-02-05T06:46:27.908990+0000","last_undegraded":"2024-02-05T06:46:27.908990+0000","last_fullsized":"2024-02-05T06:46:27.908990+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:00.522003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:44:00.522003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:30.354622+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"319'13","reported_seq":410,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.904031+0000","last_change":"2024-02-05T06:45:38.107375+0000","last_active":"2024-02-05T06:46:27.904031+0000","last_peered":"2024-02-05T06:46:27.904031+0000","last_clean":"2024-02-05T06:46:27.904031+0000","last_became_active":"2024-02-05T06:45:09.766220+0000","last_became_peered":"2024-02-05T06:45:09.766220+0000","last_unstale":"2024-02-05T06:46:27.904031+0000","last_undegraded":"2024-02-05T06:46:27.904031+0000","last_fullsized":"2024-02-05T06:46:27.904031+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'13","last_scrub_stamp":"2024-02-05T06:45:38.107271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:45:38.107271+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:45.773376+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":579,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:22.881020+0000","last_change":"2024-02-05T06:43:43.544887+0000","last_active":"2024-02-05T06:46:22.881020+0000","last_peered":"2024-02-05T06:46:22.881020+0000","last_clean":"2024-02-05T06:46:22.881020+0000","last_became_active":"2024-02-05T06:40:27.213439+0000","last_became_peered":"2024-02-05T06:40:27.213439+0000","last_unstale":"2024-02-05T06:46:22.881020+0000","last_undegraded":"2024-02-05T06:46:22.881020+0000","last_fullsized":"2024-02-05T06:46:22.881020+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:43.544765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:43:43.544765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:24.288902+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"295'34","reported_seq":387,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-02-05T06:46:32.168815+0000","last_change":"2024-02-05T06:44:50.029401+0000","last_active":"2024-02-05T06:46:32.168815+0000","last_peered":"2024-02-05T06:46:32.168815+0000","last_clean":"2024-02-05T06:46:32.168815+0000","last_became_active":"2024-02-05T06:39:13.257978+0000","last_became_peered":"2024-02-05T06:39:13.257978+0000","last_unstale":"2024-02-05T06:46:32.168815+0000","last_undegraded":"2024-02-05T06:46:32.168815+0000","last_fullsized":"2024-02-05T06:46:32.168815+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'34","last_scrub_stamp":"2024-02-05T06:44:50.029270+0000","last_deep_scrub":"281'22","last_deep_scrub_stamp":"2024-02-05T06:42:59.969994+0000","last_clean_scrub_stamp":"2024-02-05T06:44:50.029270+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.3","version":"331'16391","reported_seq":16683,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:31.570205+0000","last_change":"2024-02-05T06:45:32.263366+0000","last_active":"2024-02-05T06:46:31.570205+0000","last_peered":"2024-02-05T06:46:31.570205+0000","last_clean":"2024-02-05T06:46:31.570205+0000","last_became_active":"2024-02-05T06:43:29.701677+0000","last_became_peered":"2024-02-05T06:43:29.701677+0000","last_unstale":"2024-02-05T06:46:31.570205+0000","last_undegraded":"2024-02-05T06:46:31.570205+0000","last_fullsized":"2024-02-05T06:46:31.570205+0000","mapping_epoch":291,"log_start":"321'11701","ondisk_log_start":"321'11701","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"324'12094","last_scrub_stamp":"2024-02-05T06:45:32.263146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:45:32.263146+0000","objects_scrubbed":6511,"log_size":4690,"log_dups_size":0,"ondisk_log_size":4690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:36.111812+0000","scrub_duration":1952,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140115968,"num_objects":2138,"num_object_clones":0,"num_object_copies":6414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14052,"num_write_kb":518080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":573,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:22.884831+0000","last_change":"2024-02-05T06:43:42.559000+0000","last_active":"2024-02-05T06:46:22.884831+0000","last_peered":"2024-02-05T06:46:22.884831+0000","last_clean":"2024-02-05T06:46:22.884831+0000","last_became_active":"2024-02-05T06:41:43.410600+0000","last_became_peered":"2024-02-05T06:41:43.410600+0000","last_unstale":"2024-02-05T06:46:22.884831+0000","last_undegraded":"2024-02-05T06:46:22.884831+0000","last_fullsized":"2024-02-05T06:46:22.884831+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:42.558880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:20.042534+0000","last_clean_scrub_stamp":"2024-02-05T06:43:42.558880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:24.288984+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":195,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.903671+0000","last_change":"2024-02-05T06:44:53.595624+0000","last_active":"2024-02-05T06:46:27.903671+0000","last_peered":"2024-02-05T06:46:27.903671+0000","last_clean":"2024-02-05T06:46:27.903671+0000","last_became_active":"2024-02-05T06:41:43.417343+0000","last_became_peered":"2024-02-05T06:41:43.417343+0000","last_unstale":"2024-02-05T06:46:27.903671+0000","last_undegraded":"2024-02-05T06:46:27.903671+0000","last_fullsized":"2024-02-05T06:46:27.903671+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:53.595517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:42:55.610276+0000","last_clean_scrub_stamp":"2024-02-05T06:44:53.595517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:30.354863+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":297,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.900427+0000","last_change":"2024-02-05T06:44:01.550712+0000","last_active":"2024-02-05T06:46:27.900427+0000","last_peered":"2024-02-05T06:46:27.900427+0000","last_clean":"2024-02-05T06:46:27.900427+0000","last_became_active":"2024-02-05T06:40:27.086808+0000","last_became_peered":"2024-02-05T06:40:27.086808+0000","last_unstale":"2024-02-05T06:46:27.900427+0000","last_undegraded":"2024-02-05T06:46:27.900427+0000","last_fullsized":"2024-02-05T06:46:27.900427+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:01.550518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:44:01.550518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"295'24","reported_seq":439,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.897986+0000","last_change":"2024-02-05T06:45:38.738448+0000","last_active":"2024-02-05T06:46:27.897986+0000","last_peered":"2024-02-05T06:46:27.897986+0000","last_clean":"2024-02-05T06:46:27.897986+0000","last_became_active":"2024-02-05T06:45:09.774868+0000","last_became_peered":"2024-02-05T06:45:09.774868+0000","last_unstale":"2024-02-05T06:46:27.897986+0000","last_undegraded":"2024-02-05T06:46:27.897986+0000","last_fullsized":"2024-02-05T06:46:27.897986+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:45:38.738302+0000","last_deep_scrub":"115'16","last_deep_scrub_stamp":"2024-02-05T06:36:34.793243+0000","last_clean_scrub_stamp":"2024-02-05T06:45:38.738302+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:53.532700+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.0","version":"331'14239","reported_seq":14450,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:32.026653+0000","last_change":"2024-02-05T06:45:40.871153+0000","last_active":"2024-02-05T06:46:32.026653+0000","last_peered":"2024-02-05T06:46:32.026653+0000","last_clean":"2024-02-05T06:46:32.026653+0000","last_became_active":"2024-02-05T06:43:29.946914+0000","last_became_peered":"2024-02-05T06:43:29.946914+0000","last_unstale":"2024-02-05T06:46:32.026653+0000","last_undegraded":"2024-02-05T06:46:32.026653+0000","last_fullsized":"2024-02-05T06:46:32.026653+0000","mapping_epoch":291,"log_start":"323'9637","ondisk_log_start":"323'9637","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"324'10554","last_scrub_stamp":"2024-02-05T06:45:40.870869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:45:40.870869+0000","objects_scrubbed":5893,"log_size":4602,"log_dups_size":0,"ondisk_log_size":4602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:47:05.436853+0000","scrub_duration":2075,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140836864,"num_objects":2149,"num_object_clones":0,"num_object_copies":6447,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2149,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14239,"num_write_kb":524416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.8","version":"83'128","reported_seq":610,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.902906+0000","last_change":"2024-02-05T06:43:47.429770+0000","last_active":"2024-02-05T06:46:27.902906+0000","last_peered":"2024-02-05T06:46:27.902906+0000","last_clean":"2024-02-05T06:46:27.902906+0000","last_became_active":"2024-02-05T06:41:46.053324+0000","last_became_peered":"2024-02-05T06:41:46.053324+0000","last_unstale":"2024-02-05T06:46:27.902906+0000","last_undegraded":"2024-02-05T06:46:27.902906+0000","last_fullsized":"2024-02-05T06:46:27.902906+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:43:47.429623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:43:47.429623+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:30.354596+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":600,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.913362+0000","last_change":"2024-02-05T06:45:31.765060+0000","last_active":"2024-02-05T06:46:27.913362+0000","last_peered":"2024-02-05T06:46:27.913362+0000","last_clean":"2024-02-05T06:46:27.913362+0000","last_became_active":"2024-02-05T06:36:54.958805+0000","last_became_peered":"2024-02-05T06:36:54.958805+0000","last_unstale":"2024-02-05T06:46:27.913362+0000","last_undegraded":"2024-02-05T06:46:27.913362+0000","last_fullsized":"2024-02-05T06:46:27.913362+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:31.764712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:45:31.764712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:35.647839+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":589,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.901026+0000","last_change":"2024-02-05T06:44:43.706515+0000","last_active":"2024-02-05T06:46:27.901026+0000","last_peered":"2024-02-05T06:46:27.901026+0000","last_clean":"2024-02-05T06:46:27.901026+0000","last_became_active":"2024-02-05T06:36:54.958082+0000","last_became_peered":"2024-02-05T06:36:54.958082+0000","last_unstale":"2024-02-05T06:46:27.901026+0000","last_undegraded":"2024-02-05T06:46:27.901026+0000","last_fullsized":"2024-02-05T06:46:27.901026+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:43.706297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:44:43.706297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:30.695540+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":603,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.897356+0000","last_change":"2024-02-05T06:44:39.569920+0000","last_active":"2024-02-05T06:46:27.897356+0000","last_peered":"2024-02-05T06:46:27.897356+0000","last_clean":"2024-02-05T06:46:27.897356+0000","last_became_active":"2024-02-05T06:39:13.240038+0000","last_became_peered":"2024-02-05T06:39:13.240038+0000","last_unstale":"2024-02-05T06:46:27.897356+0000","last_undegraded":"2024-02-05T06:46:27.897356+0000","last_fullsized":"2024-02-05T06:46:27.897356+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:44:39.569779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:44:39.569779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"295'39","reported_seq":454,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.912767+0000","last_change":"2024-02-05T06:45:09.783575+0000","last_active":"2024-02-05T06:46:27.912767+0000","last_peered":"2024-02-05T06:46:27.912767+0000","last_clean":"2024-02-05T06:46:27.912767+0000","last_became_active":"2024-02-05T06:45:09.782645+0000","last_became_peered":"2024-02-05T06:45:09.782645+0000","last_unstale":"2024-02-05T06:46:27.912767+0000","last_undegraded":"2024-02-05T06:46:27.912767+0000","last_fullsized":"2024-02-05T06:46:27.912767+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'39","last_scrub_stamp":"2024-02-05T06:44:47.082558+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-02-05T06:35:26.274944+0000","last_clean_scrub_stamp":"2024-02-05T06:44:47.082558+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:31.745946+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"330'13777","reported_seq":13866,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.512999+0000","last_change":"2024-02-05T06:43:29.947556+0000","last_active":"2024-02-05T06:46:27.512999+0000","last_peered":"2024-02-05T06:46:27.512999+0000","last_clean":"2024-02-05T06:46:27.512999+0000","last_became_active":"2024-02-05T06:43:29.947136+0000","last_became_peered":"2024-02-05T06:43:29.947136+0000","last_unstale":"2024-02-05T06:46:27.512999+0000","last_undegraded":"2024-02-05T06:46:27.512999+0000","last_fullsized":"2024-02-05T06:46:27.512999+0000","mapping_epoch":291,"log_start":"321'9300","ondisk_log_start":"321'9300","created":291,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:28.679568+0000","last_clean_scrub_stamp":"2024-02-05T06:43:28.679568+0000","objects_scrubbed":0,"log_size":4477,"log_dups_size":0,"ondisk_log_size":4477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161021952,"num_objects":2457,"num_object_clones":0,"num_object_copies":7371,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13777,"num_write_kb":519488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":697,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.908175+0000","last_change":"2024-02-05T06:45:47.191297+0000","last_active":"2024-02-05T06:46:27.908175+0000","last_peered":"2024-02-05T06:46:27.908175+0000","last_clean":"2024-02-05T06:46:27.908175+0000","last_became_active":"2024-02-05T06:40:30.095202+0000","last_became_peered":"2024-02-05T06:40:30.095202+0000","last_unstale":"2024-02-05T06:46:27.908175+0000","last_undegraded":"2024-02-05T06:46:27.908175+0000","last_fullsized":"2024-02-05T06:46:27.908175+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:45:47.191180+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:41:50.706159+0000","last_clean_scrub_stamp":"2024-02-05T06:45:47.191180+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:58.066455+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":108,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.907990+0000","last_change":"2024-02-05T06:45:34.707123+0000","last_active":"2024-02-05T06:46:27.907990+0000","last_peered":"2024-02-05T06:46:27.907990+0000","last_clean":"2024-02-05T06:46:27.907990+0000","last_became_active":"2024-02-05T06:45:09.784229+0000","last_became_peered":"2024-02-05T06:45:09.784229+0000","last_unstale":"2024-02-05T06:46:27.907990+0000","last_undegraded":"2024-02-05T06:46:27.907990+0000","last_fullsized":"2024-02-05T06:46:27.907990+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:34.706775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:30.057099+0000","last_clean_scrub_stamp":"2024-02-05T06:45:34.706775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:46.223244+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"249'13","reported_seq":789,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-02-05T06:46:27.912926+0000","last_change":"2024-02-05T06:44:38.532844+0000","last_active":"2024-02-05T06:46:27.912926+0000","last_peered":"2024-02-05T06:46:27.912926+0000","last_clean":"2024-02-05T06:46:27.912926+0000","last_became_active":"2024-02-05T06:41:43.413139+0000","last_became_peered":"2024-02-05T06:41:43.413139+0000","last_unstale":"2024-02-05T06:46:27.912926+0000","last_undegraded":"2024-02-05T06:46:27.912926+0000","last_fullsized":"2024-02-05T06:46:27.912926+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'13","last_scrub_stamp":"2024-02-05T06:44:38.532716+0000","last_deep_scrub":"92'9","last_deep_scrub_stamp":"2024-02-05T06:34:52.283780+0000","last_clean_scrub_stamp":"2024-02-05T06:44:38.532716+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:31.745857+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":587,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:22.880785+0000","last_change":"2024-02-05T06:45:30.989035+0000","last_active":"2024-02-05T06:46:22.880785+0000","last_peered":"2024-02-05T06:46:22.880785+0000","last_clean":"2024-02-05T06:46:22.880785+0000","last_became_active":"2024-02-05T06:36:54.971304+0000","last_became_peered":"2024-02-05T06:36:54.971304+0000","last_unstale":"2024-02-05T06:46:22.880785+0000","last_undegraded":"2024-02-05T06:46:22.880785+0000","last_fullsized":"2024-02-05T06:46:22.880785+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:30.987881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:45:30.987881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:33.237741+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"295'25","reported_seq":440,"reported_epoch":330,"state":"active+clean","last_fresh":"2024-02-05T06:46:22.885082+0000","last_change":"2024-02-05T06:45:09.799601+0000","last_active":"2024-02-05T06:46:22.885082+0000","last_peered":"2024-02-05T06:46:22.885082+0000","last_clean":"2024-02-05T06:46:22.885082+0000","last_became_active":"2024-02-05T06:45:09.796902+0000","last_became_peered":"2024-02-05T06:45:09.796902+0000","last_unstale":"2024-02-05T06:46:22.885082+0000","last_undegraded":"2024-02-05T06:46:22.885082+0000","last_fullsized":"2024-02-05T06:46:22.885082+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'25","last_scrub_stamp":"2024-02-05T06:44:50.081252+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-02-05T06:35:04.584657+0000","last_clean_scrub_stamp":"2024-02-05T06:44:50.081252+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:46:24.289335+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":2292973592,"num_objects":34989,"num_object_clones":0,"num_object_copies":104967,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34989,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":224607,"num_write_kb":8307009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6817853440,"data_stored":6817841224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77959,"ondisk_log_size":77959,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":370,"num_read_kb":365,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1573408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1106,"num_read_kb":863,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":373,"num_bytes_recovered":8234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":771,"ondisk_log_size":771,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103328,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1689044,"kb_used_data":1260812,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":92056108,"statfs":{"total":95995035648,"available":94265454592,"internally_reserved":0,"allocated":1291071488,"data_stored":1290690479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":438499786},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:45:51 2024","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.4590000000000001,"15min":1.413},"min":{"1min":0.26200000000000001,"5min":0.23699999999999999,"15min":0.219},"max":{"1min":3.7930000000000001,"5min":32.567,"15min":68.921000000000006},"last":1.75},{"interface":"front","average":{"1min":1.5469999999999999,"5min":1.663,"15min":1.5620000000000001},"min":{"1min":0.252,"5min":0.222,"15min":0.222},"max":{"1min":4.1619999999999999,"5min":32.603000000000002,"15min":69.616},"last":0.997}]},{"osd":1,"last update":"Mon Feb 5 06:45:51 2024","interfaces":[{"interface":"back","average":{"1min":6.4240000000000004,"5min":2.6669999999999998,"15min":1.7190000000000001},"min":{"1min":0.26500000000000001,"5min":0.25,"15min":0.23599999999999999},"max":{"1min":84.216999999999999,"5min":84.216999999999999,"15min":89.231999999999999},"last":2.9870000000000001},{"interface":"front","average":{"1min":6.3550000000000004,"5min":2.5699999999999998,"15min":1.7949999999999999},"min":{"1min":0.36499999999999999,"5min":0.32600000000000001,"15min":0.26100000000000001},"max":{"1min":84.126999999999995,"5min":84.126999999999995,"15min":87.995999999999995},"last":3.085}]},{"osd":2,"last update":"Mon Feb 5 06:45:51 2024","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":1.3560000000000001,"15min":1.2010000000000001},"min":{"1min":0.30299999999999999,"5min":0.28599999999999998,"15min":0.26200000000000001},"max":{"1min":4.3479999999999999,"5min":18.071999999999999,"15min":53.947000000000003},"last":0.69599999999999995},{"interface":"front","average":{"1min":1.242,"5min":1.3089999999999999,"15min":1.165},"min":{"1min":0.34399999999999997,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":3.758,"5min":18.045999999999999,"15min":53.299999999999997},"last":0.40300000000000002}]},{"osd":3,"last update":"Mon Feb 5 06:45:51 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.4350000000000001,"15min":1.748},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":4.2409999999999997,"5min":8.5120000000000005,"15min":89.481999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":1.6539999999999999,"5min":1.4430000000000001,"15min":1.7090000000000001},"min":{"1min":0.33000000000000002,"5min":0.30599999999999999,"15min":0.27400000000000002},"max":{"1min":5.9370000000000003,"5min":7.9199999999999999,"15min":92.293999999999997},"last":0.36799999999999999}]},{"osd":4,"last update":"Mon Feb 5 06:45:51 2024","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":0.71899999999999997,"15min":0.56699999999999995},"min":{"1min":0.379,"5min":0.251,"15min":0.24199999999999999},"max":{"1min":4.1059999999999999,"5min":4.1059999999999999,"15min":4.1059999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.72999999999999998,"15min":0.57099999999999995},"min":{"1min":0.36599999999999999,"5min":0.27100000000000002,"15min":0.23400000000000001},"max":{"1min":3.0819999999999999,"5min":3.0819999999999999,"15min":3.0819999999999999},"last":0.67000000000000004}]},{"osd":5,"last update":"Mon Feb 5 06:45:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":0.94499999999999995,"15min":0.65900000000000003},"min":{"1min":0.36499999999999999,"5min":0.251,"15min":0.251},"max":{"1min":4.008,"5min":6.2190000000000003,"15min":6.2190000000000003},"last":0.434},{"interface":"front","average":{"1min":1.95,"5min":1.0580000000000001,"15min":0.73299999999999998},"min":{"1min":0.313,"5min":0.30299999999999999,"15min":0.252},"max":{"1min":4.6130000000000004,"5min":7.8360000000000003,"15min":7.8360000000000003},"last":0.58899999999999997}]},{"osd":6,"last update":"Mon Feb 5 06:45:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":0.874,"15min":0.68999999999999995},"min":{"1min":0.33900000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":4.2009999999999996,"5min":4.2009999999999996,"15min":4.2009999999999996},"last":0.52400000000000002},{"interface":"front","average":{"1min":1.0549999999999999,"5min":0.87,"15min":0.66400000000000003},"min":{"1min":0.34300000000000003,"5min":0.307,"15min":0.245},"max":{"1min":3.3769999999999998,"5min":3.3769999999999998,"15min":3.3769999999999998},"last":0.71999999999999997}]}]},{"osd":6,"up_from":49,"seq":210453397670,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":768696,"kb_used_data":430704,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92976456,"statfs":{"total":95995035648,"available":95207890944,"internally_reserved":0,"allocated":441040896,"data_stored":440687086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":346094022},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":3.125,"15min":2.0840000000000001},"min":{"1min":0.23200000000000001,"5min":0.22900000000000001,"15min":0.224},"max":{"1min":3.8690000000000002,"5min":98.918999999999997,"15min":98.918999999999997},"last":0.33500000000000002},{"interface":"front","average":{"1min":1.0349999999999999,"5min":3.2749999999999999,"15min":2.0649999999999999},"min":{"1min":0.246,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.681,"5min":98.775000000000006,"15min":98.775000000000006},"last":0.26900000000000002}]},{"osd":1,"last update":"Mon Feb 5 06:46:15 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":3.2360000000000002,"15min":1.8660000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.24099999999999999},"max":{"1min":5.2750000000000004,"5min":98.843999999999994,"15min":98.843999999999994},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.83299999999999996,"5min":3.3769999999999998,"15min":1.847},"min":{"1min":0.36299999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":3.0920000000000001,"5min":101.637,"15min":101.637},"last":0.29199999999999998}]},{"osd":2,"last update":"Mon Feb 5 06:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":1.9730000000000001,"15min":1.7549999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.22500000000000001},"max":{"1min":2.6179999999999999,"5min":74.454999999999998,"15min":85.617000000000004},"last":0.49099999999999999},{"interface":"front","average":{"1min":1.069,"5min":1.804,"15min":1.6679999999999999},"min":{"1min":0.29599999999999999,"5min":0.28499999999999998,"15min":0.27000000000000002},"max":{"1min":5.6029999999999998,"5min":74.546000000000006,"15min":85.335999999999999},"last":0.40999999999999998}]},{"osd":3,"last update":"Mon Feb 5 06:46:15 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":3.0099999999999998,"15min":2.3050000000000002},"min":{"1min":0.313,"5min":0.307,"15min":0.27000000000000002},"max":{"1min":3.1749999999999998,"5min":98.888999999999996,"15min":99.269000000000005},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.94199999999999995,"5min":2.9620000000000002,"15min":2.4060000000000001},"min":{"1min":0.28799999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":2.698,"5min":98.969999999999999,"15min":101.75700000000001},"last":0.53900000000000003}]},{"osd":4,"last update":"Mon Feb 5 06:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.64000000000000001,"15min":0.52300000000000002},"min":{"1min":0.28899999999999998,"5min":0.249,"15min":0.23100000000000001},"max":{"1min":2.641,"5min":3.742,"15min":3.742},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.64000000000000001,"15min":0.52000000000000002},"min":{"1min":0.29199999999999998,"5min":0.24399999999999999,"15min":0.24299999999999999},"max":{"1min":3.0659999999999998,"5min":4.4379999999999997,"15min":4.4379999999999997},"last":0.66300000000000003}]},{"osd":5,"last update":"Mon Feb 5 06:46:15 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":1.234,"15min":0.73599999999999999},"min":{"1min":0.34100000000000003,"5min":0.314,"15min":0.251},"max":{"1min":2.8029999999999999,"5min":36.548999999999999,"15min":36.548999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.3360000000000001,"15min":0.76900000000000002},"min":{"1min":0.34999999999999998,"5min":0.28699999999999998,"15min":0.24299999999999999},"max":{"1min":4.5970000000000004,"5min":36.433,"15min":36.433},"last":0.89000000000000001}]},{"osd":7,"last update":"Mon Feb 5 06:45:36 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.115,"15min":0.72899999999999998},"min":{"1min":0.36699999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":7.157,"5min":38.436999999999998,"15min":38.436999999999998},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.92100000000000004,"5min":1.0609999999999999,"15min":0.75800000000000001},"min":{"1min":0.45700000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.843,"5min":36.633000000000003,"15min":36.633000000000003},"last":0.61499999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575039,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1746420,"kb_used_data":1219628,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":91998732,"statfs":{"total":95995035648,"available":94206701568,"internally_reserved":0,"allocated":1248899072,"data_stored":1248486078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":539425221},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:46:17 2024","interfaces":[{"interface":"back","average":{"1min":1.865,"5min":2.2109999999999999,"15min":1.712},"min":{"1min":0.218,"5min":0.21199999999999999,"15min":0.16700000000000001},"max":{"1min":7.4370000000000003,"5min":43.651000000000003,"15min":67.421000000000006},"last":0.36299999999999999},{"interface":"front","average":{"1min":2.1909999999999998,"5min":2.3399999999999999,"15min":1.861},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.19600000000000001},"max":{"1min":10.333,"5min":45.670999999999999,"15min":67.885000000000005},"last":2.9239999999999999}]},{"osd":2,"last update":"Mon Feb 5 06:45:33 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":2.0070000000000001,"15min":1.3779999999999999},"min":{"1min":0.26200000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":4.0039999999999996,"5min":43.447000000000003,"15min":43.447000000000003},"last":1.1739999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":1.5649999999999999,"15min":1.3660000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.20899999999999999},"max":{"1min":3.117,"5min":22.050000000000001,"15min":22.050000000000001},"last":1.141}]},{"osd":3,"last update":"Mon Feb 5 06:46:02 2024","interfaces":[{"interface":"back","average":{"1min":2.2400000000000002,"5min":2.1680000000000001,"15min":2.0790000000000002},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":10.456,"5min":25.236999999999998,"15min":69.798000000000002},"last":2.8879999999999999},{"interface":"front","average":{"1min":1.706,"5min":2.2080000000000002,"15min":1.778},"min":{"1min":0.35099999999999998,"5min":0.317,"15min":0.23699999999999999},"max":{"1min":5.8520000000000003,"5min":43.354999999999997,"15min":70.765000000000001},"last":1.508}]},{"osd":4,"last update":"Mon Feb 5 06:46:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.177,"15min":1.0940000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":5.6369999999999996,"5min":7.0519999999999996,"15min":11.679},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.5720000000000001,"5min":1.476,"15min":1.2030000000000001},"min":{"1min":0.317,"5min":0.30599999999999999,"15min":0.26300000000000001},"max":{"1min":10.419,"5min":10.419,"15min":10.419},"last":0.73699999999999999}]},{"osd":5,"last update":"Mon Feb 5 06:45:55 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.7250000000000001,"15min":1.2370000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.29299999999999998},"max":{"1min":5.335,"5min":43.414999999999999,"15min":43.414999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":1.6539999999999999,"5min":1.448,"15min":1.2290000000000001},"min":{"1min":0.39000000000000001,"5min":0.36599999999999999,"15min":0.33100000000000002},"max":{"1min":5.6909999999999998,"5min":7.0919999999999996,"15min":13.455},"last":0.49399999999999999}]},{"osd":6,"last update":"Mon Feb 5 06:46:17 2024","interfaces":[{"interface":"back","average":{"1min":2.0019999999999998,"5min":1.45,"15min":1.29},"min":{"1min":0.376,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":23.902999999999999,"5min":23.902999999999999,"15min":23.902999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.694,"15min":1.2070000000000001},"min":{"1min":0.47999999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":5.4160000000000004,"5min":43.475000000000001,"15min":43.475000000000001},"last":0.438}]},{"osd":7,"last update":"Mon Feb 5 06:45:33 2024","interfaces":[{"interface":"back","average":{"1min":1.3240000000000001,"5min":1.387,"15min":1.214},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":5.9740000000000002,"5min":7.157,"15min":13.49},"last":0.51900000000000002},{"interface":"front","average":{"1min":1.819,"5min":1.548,"15min":1.3100000000000001},"min":{"1min":0.40699999999999997,"5min":0.40400000000000003,"15min":0.35499999999999998},"max":{"1min":9.0730000000000004,"5min":9.3249999999999993,"15min":9.3249999999999993},"last":0.76600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738565,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":940640,"kb_used_data":544344,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":92804512,"statfs":{"total":95995035648,"available":95031820288,"internally_reserved":0,"allocated":557408256,"data_stored":557014911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 06:46:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.583,"15min":1.397},"min":{"1min":0.23400000000000001,"5min":0.182,"15min":0.182},"max":{"1min":3.798,"5min":18.311,"15min":18.59},"last":1.4810000000000001},{"interface":"front","average":{"1min":1.361,"5min":1.6160000000000001,"15min":1.3759999999999999},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":4.1760000000000002,"5min":21.088000000000001,"15min":21.088000000000001},"last":4.4080000000000004}]},{"osd":2,"last update":"Mon Feb 5 06:46:32 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":1.0349999999999999,"15min":1.3},"min":{"1min":0.26200000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.1110000000000002,"5min":5.3689999999999998,"15min":16.981999999999999},"last":1.5349999999999999},{"interface":"front","average":{"1min":0.79200000000000004,"5min":1.073,"15min":1.22},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":2.2029999999999998,"5min":5.3490000000000002,"15min":16.998999999999999},"last":1.5740000000000001}]},{"osd":3,"last update":"Mon Feb 5 06:45:58 2024","interfaces":[{"interface":"back","average":{"1min":1.5960000000000001,"5min":1.968,"15min":1.5740000000000001},"min":{"1min":0.30399999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":3.9420000000000002,"5min":43.302,"15min":43.302},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.284,"5min":2.1269999999999998,"15min":1.7270000000000001},"min":{"1min":0.28699999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.476,"5min":43.651000000000003,"15min":43.651000000000003},"last":0.65400000000000003}]},{"osd":4,"last update":"Mon Feb 5 06:46:23 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":0.92200000000000004,"15min":1.232},"min":{"1min":0.33600000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":3.246,"5min":4.5449999999999999,"15min":40.789999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":1.002,"5min":1.121,"15min":1.2989999999999999},"min":{"1min":0.35099999999999998,"5min":0.33600000000000002,"15min":0.30099999999999999},"max":{"1min":3.7309999999999999,"5min":8.4019999999999992,"15min":40.773000000000003},"last":0.63100000000000001}]},{"osd":5,"last update":"Mon Feb 5 06:45:47 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.887,"15min":1.337},"min":{"1min":0.32700000000000001,"5min":0.32500000000000001,"15min":0.29299999999999998},"max":{"1min":3.8450000000000002,"5min":48.841999999999999,"15min":48.841999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.7090000000000001,"15min":1.304},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":3.355,"5min":48.450000000000003,"15min":48.450000000000003},"last":0.72299999999999998}]},{"osd":6,"last update":"Mon Feb 5 06:46:11 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":1.093,"15min":1.155},"min":{"1min":0.376,"5min":0.36799999999999999,"15min":0.33700000000000002},"max":{"1min":2.7120000000000002,"5min":5.992,"15min":18.562999999999999},"last":0.878},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.369,"15min":1.1930000000000001},"min":{"1min":0.39200000000000002,"5min":0.372,"15min":0.33000000000000002},"max":{"1min":3.1389999999999998,"5min":13.102,"15min":18.213999999999999},"last":0.70899999999999996}]},{"osd":7,"last update":"Mon Feb 5 06:45:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.153,"15min":1.1160000000000001},"min":{"1min":0.52300000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":3.21,"5min":4.5789999999999997,"15min":17.210000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":1.141,"5min":1.2290000000000001,"15min":1.24},"min":{"1min":0.378,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":2.8039999999999998,"5min":6.8470000000000004,"15min":18.533999999999999},"last":0.57499999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411514,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":798600,"kb_used_data":420352,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92946552,"statfs":{"total":95995035648,"available":95177269248,"internally_reserved":0,"allocated":430440448,"data_stored":430072116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:45:33 2024","interfaces":[{"interface":"back","average":{"1min":1.8959999999999999,"5min":1.5509999999999999,"15min":1.6659999999999999},"min":{"1min":0.22600000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":14.007,"5min":14.007,"15min":50.814},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.923,"5min":1.548,"15min":1.6379999999999999},"min":{"1min":0.22600000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":13.968999999999999,"5min":13.968999999999999,"15min":50.802999999999997},"last":2.254}]},{"osd":1,"last update":"Mon Feb 5 06:45:33 2024","interfaces":[{"interface":"back","average":{"1min":2.891,"5min":1.835,"15min":1.5509999999999999},"min":{"1min":0.21199999999999999,"5min":0.189,"15min":0.189},"max":{"1min":14.526999999999999,"5min":15.382,"15min":31.303000000000001},"last":1.0329999999999999},{"interface":"front","average":{"1min":2.2200000000000002,"5min":1.5600000000000001,"15min":1.2889999999999999},"min":{"1min":0.32800000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":14.179,"5min":15.742000000000001,"15min":31.425000000000001},"last":0.30599999999999999}]},{"osd":3,"last update":"Mon Feb 5 06:46:01 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.5389999999999999,"15min":1.7769999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":7.0469999999999997,"5min":15.166,"15min":53.127000000000002},"last":0.35399999999999998},{"interface":"front","average":{"1min":1.681,"5min":1.4590000000000001,"15min":1.712},"min":{"1min":0.29899999999999999,"5min":0.23599999999999999,"15min":0.22500000000000001},"max":{"1min":8.4039999999999999,"5min":15.541,"15min":52.607999999999997},"last":0.41699999999999998}]},{"osd":4,"last update":"Mon Feb 5 06:46:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.1080000000000001,"15min":0.995},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.23999999999999999},"max":{"1min":4.5759999999999996,"5min":12.041,"15min":12.041},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":1.0660000000000001,"15min":0.95499999999999996},"min":{"1min":0.35799999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":4.1139999999999999,"5min":15.689,"15min":15.689},"last":0.59499999999999997}]},{"osd":5,"last update":"Mon Feb 5 06:45:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.3879999999999999,"15min":1.46},"min":{"1min":0.315,"5min":0.315,"15min":0.29899999999999999},"max":{"1min":4.984,"5min":30.265000000000001,"15min":86.614999999999995},"last":0.38},{"interface":"front","average":{"1min":1.853,"5min":1.571,"15min":1.6539999999999999},"min":{"1min":0.40699999999999997,"5min":0.312,"15min":0.312},"max":{"1min":9.7550000000000008,"5min":30.02,"15min":86.543999999999997},"last":0.84599999999999997}]},{"osd":6,"last update":"Mon Feb 5 06:46:23 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.99399999999999999,"15min":1.1699999999999999},"min":{"1min":0.36599999999999999,"5min":0.34200000000000003,"15min":0.315},"max":{"1min":4.173,"5min":6.665,"15min":17.143999999999998},"last":0.47899999999999998},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.325,"15min":1.254},"min":{"1min":0.35999999999999999,"5min":0.316,"15min":0.316},"max":{"1min":3.649,"5min":14.416,"15min":14.416},"last":0.80500000000000005}]},{"osd":7,"last update":"Mon Feb 5 06:45:40 2024","interfaces":[{"interface":"back","average":{"1min":4.718,"5min":1.8899999999999999,"15min":1.401},"min":{"1min":0.38900000000000001,"5min":0.377,"15min":0.35699999999999998},"max":{"1min":45.243000000000002,"5min":45.243000000000002,"15min":45.243000000000002},"last":1.839},{"interface":"front","average":{"1min":3.968,"5min":1.663,"15min":1.2470000000000001},"min":{"1min":0.47599999999999998,"5min":0.379,"15min":0.36899999999999999},"max":{"1min":45.171999999999997,"5min":45.171999999999997,"15min":45.171999999999997},"last":0.76400000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149876,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1807568,"kb_used_data":1262600,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":91937584,"statfs":{"total":95995035648,"available":94144086016,"internally_reserved":0,"allocated":1292902400,"data_stored":1292518177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":557906375},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:46:12 2024","interfaces":[{"interface":"back","average":{"1min":1.857,"5min":1.3129999999999999,"15min":1.23},"min":{"1min":0.35899999999999999,"5min":0.184,"15min":0.184},"max":{"1min":8.9809999999999999,"5min":9.6219999999999999,"15min":13.391999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.5209999999999999,"15min":1.452},"min":{"1min":0.26100000000000001,"5min":0.21099999999999999,"15min":0.20399999999999999},"max":{"1min":5.4950000000000001,"5min":6.4939999999999998,"15min":25.087},"last":3.5979999999999999}]},{"osd":1,"last update":"Mon Feb 5 06:46:12 2024","interfaces":[{"interface":"back","average":{"1min":2.081,"5min":1.873,"15min":1.4490000000000001},"min":{"1min":0.309,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":7.2939999999999996,"5min":11.412000000000001,"15min":11.412000000000001},"last":2.0659999999999998},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.911,"15min":1.4339999999999999},"min":{"1min":0.38700000000000001,"5min":0.248,"15min":0.215},"max":{"1min":3.7869999999999999,"5min":9.8659999999999997,"15min":9.8659999999999997},"last":8.6590000000000007}]},{"osd":2,"last update":"Mon Feb 5 06:46:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4410000000000001,"5min":1.6719999999999999,"15min":1.524},"min":{"1min":0.34999999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":4.7050000000000001,"5min":7.7080000000000002,"15min":41.529000000000003},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.486,"15min":1.411},"min":{"1min":0.28899999999999998,"5min":0.249,"15min":0.248},"max":{"1min":7.3559999999999999,"5min":7.6029999999999998,"15min":41.759999999999998},"last":1.198}]},{"osd":4,"last update":"Mon Feb 5 06:45:30 2024","interfaces":[{"interface":"back","average":{"1min":1.772,"5min":1.3160000000000001,"15min":1.0369999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":7.7329999999999997,"5min":7.7329999999999997,"15min":16.577999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":2.1949999999999998,"5min":1.825,"15min":1.3080000000000001},"min":{"1min":0.41799999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":7.6029999999999998,"5min":19.359999999999999,"15min":19.359999999999999},"last":3.5219999999999998}]},{"osd":5,"last update":"Mon Feb 5 06:45:46 2024","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.8149999999999999,"15min":1.3799999999999999},"min":{"1min":0.39800000000000002,"5min":0.32000000000000001,"15min":0.29299999999999998},"max":{"1min":3.4870000000000001,"5min":20.257999999999999,"15min":20.257999999999999},"last":1.631},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.3129999999999999,"15min":1.099},"min":{"1min":0.46999999999999997,"5min":0.36499999999999999,"15min":0.28399999999999997},"max":{"1min":4.2670000000000003,"5min":7.859,"15min":15.792999999999999},"last":0.82999999999999996}]},{"osd":6,"last update":"Mon Feb 5 06:46:25 2024","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.5720000000000001,"15min":1.296},"min":{"1min":0.44500000000000001,"5min":0.33300000000000002,"15min":0.28100000000000003},"max":{"1min":5.5750000000000002,"5min":7.556,"15min":8.3610000000000007},"last":3.5630000000000002},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.456,"15min":1.25},"min":{"1min":0.502,"5min":0.41199999999999998,"15min":0.28599999999999998},"max":{"1min":3.0920000000000001,"5min":7.2439999999999998,"15min":7.2439999999999998},"last":1.669}]},{"osd":7,"last update":"Mon Feb 5 06:45:46 2024","interfaces":[{"interface":"back","average":{"1min":1.581,"5min":1.431,"15min":1.25},"min":{"1min":0.47399999999999998,"5min":0.376,"15min":0.33400000000000002},"max":{"1min":4.2320000000000002,"5min":7.8029999999999999,"15min":45.039000000000001},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.266,"5min":1.6499999999999999,"15min":1.4059999999999999},"min":{"1min":0.46200000000000002,"5min":0.45700000000000002,"15min":0.34399999999999997},"max":{"1min":2.363,"5min":7.2720000000000002,"15min":44.036999999999999},"last":1.7}]}]},{"osd":4,"up_from":31,"seq":133143986351,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":853592,"kb_used_data":425360,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":92891560,"statfs":{"total":95995035648,"available":95120957440,"internally_reserved":0,"allocated":435568640,"data_stored":435183065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.91700000000000004,"15min":1.149},"min":{"1min":0.23000000000000001,"5min":0.20799999999999999,"15min":0.20200000000000001},"max":{"1min":3.4159999999999999,"5min":6.0220000000000002,"15min":33.372},"last":0.40500000000000003},{"interface":"front","average":{"1min":1.032,"5min":1.0129999999999999,"15min":1.2669999999999999},"min":{"1min":0.247,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":10.913,"5min":10.913,"15min":33.323999999999998},"last":0.504}]},{"osd":1,"last update":"Mon Feb 5 06:46:28 2024","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":1.2390000000000001,"15min":1.026},"min":{"1min":0.26900000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.165,"5min":5.0599999999999996,"15min":33.747999999999998},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.75800000000000001,"5min":1.0649999999999999,"15min":1.0049999999999999},"min":{"1min":0.26300000000000001,"5min":0.23699999999999999,"15min":0.23000000000000001},"max":{"1min":3.21,"5min":6.8860000000000001,"15min":33.691000000000003},"last":0.36899999999999999}]},{"osd":2,"last update":"Mon Feb 5 06:46:25 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.98499999999999999,"15min":0.80300000000000005},"min":{"1min":0.30299999999999999,"5min":0.27500000000000002,"15min":0.245},"max":{"1min":3.1469999999999998,"5min":17.379999999999999,"15min":17.379999999999999},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":1.4359999999999999,"15min":1.0309999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.26000000000000001},"max":{"1min":2.133,"5min":18.077000000000002,"15min":18.077000000000002},"last":0.48199999999999998}]},{"osd":3,"last update":"Mon Feb 5 06:46:28 2024","interfaces":[{"interface":"back","average":{"1min":2.484,"5min":2.3730000000000002,"15min":2.2879999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.252},"max":{"1min":33.923999999999999,"5min":44.601999999999997,"15min":99.546999999999997},"last":33.923999999999999},{"interface":"front","average":{"1min":2.6509999999999998,"5min":2.3460000000000001,"15min":2.2989999999999999},"min":{"1min":0.36599999999999999,"5min":0.29799999999999999,"15min":0.29099999999999998},"max":{"1min":33.804000000000002,"5min":44.511000000000003,"15min":99.245000000000005},"last":33.804000000000002}]},{"osd":5,"last update":"Mon Feb 5 06:45:50 2024","interfaces":[{"interface":"back","average":{"1min":1.349,"5min":0.86299999999999999,"15min":0.64000000000000001},"min":{"1min":0.309,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":3.5579999999999998,"5min":4.6289999999999996,"15min":11.098000000000001},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.78100000000000003,"15min":0.61399999999999999},"min":{"1min":0.45500000000000002,"5min":0.27000000000000002,"15min":0.252},"max":{"1min":2.032,"5min":4.1210000000000004,"15min":10.965999999999999},"last":0.28499999999999998}]},{"osd":6,"last update":"Mon Feb 5 06:46:25 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.90000000000000002,"15min":0.69799999999999995},"min":{"1min":0.36899999999999999,"5min":0.29999999999999999,"15min":0.246},"max":{"1min":3.1850000000000001,"5min":10.039999999999999,"15min":11.868},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85199999999999998,"15min":0.69499999999999995},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.27600000000000002},"max":{"1min":5.0789999999999997,"5min":10.074,"15min":11.787000000000001},"last":0.35199999999999998}]},{"osd":7,"last update":"Mon Feb 5 06:45:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":0.83499999999999996,"15min":0.63300000000000001},"min":{"1min":0.316,"5min":0.27800000000000002,"15min":0.26000000000000001},"max":{"1min":3.4129999999999998,"5min":5.1500000000000004,"15min":5.1500000000000004},"last":0.38900000000000001},{"interface":"front","average":{"1min":1.0980000000000001,"5min":0.84299999999999997,"15min":0.65000000000000002},"min":{"1min":0.374,"5min":0.28899999999999998,"15min":0.28499999999999998},"max":{"1min":4.335,"5min":5.5890000000000004,"15min":5.5890000000000004},"last":0.307}]}]},{"osd":5,"up_from":40,"seq":171798692010,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1722124,"kb_used_data":1225796,"kb_used_omap":1,"kb_used_meta":496254,"kb_avail":92023028,"statfs":{"total":95995035648,"available":94231580672,"internally_reserved":0,"allocated":1255215104,"data_stored":1254842594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":508164550},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:45:44 2024","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":2.0619999999999998,"15min":1.4219999999999999},"min":{"1min":0.28799999999999998,"5min":0.23799999999999999,"15min":0.19700000000000001},"max":{"1min":4.0629999999999997,"5min":55.704999999999998,"15min":55.704999999999998},"last":0.67100000000000004},{"interface":"front","average":{"1min":1.5600000000000001,"5min":2.1970000000000001,"15min":1.623},"min":{"1min":0.34000000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":6.532,"5min":56.789000000000001,"15min":56.789000000000001},"last":0.96399999999999997}]},{"osd":1,"last update":"Mon Feb 5 06:45:44 2024","interfaces":[{"interface":"back","average":{"1min":1.7709999999999999,"5min":1.7889999999999999,"15min":1.325},"min":{"1min":0.27800000000000002,"5min":0.251,"15min":0.248},"max":{"1min":6.5049999999999999,"5min":41.191000000000003,"15min":41.191000000000003},"last":3.1040000000000001},{"interface":"front","average":{"1min":1.536,"5min":1.7749999999999999,"15min":1.177},"min":{"1min":0.32600000000000001,"5min":0.27500000000000002,"15min":0.23499999999999999},"max":{"1min":4.0129999999999999,"5min":41.173000000000002,"15min":41.173000000000002},"last":0.54400000000000004}]},{"osd":2,"last update":"Mon Feb 5 06:45:44 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":1.115,"15min":1.206},"min":{"1min":0.26800000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.2029999999999998,"5min":17.806000000000001,"15min":36.901000000000003},"last":0.47199999999999998},{"interface":"front","average":{"1min":1.556,"5min":1.3360000000000001,"15min":1.1850000000000001},"min":{"1min":0.379,"5min":0.29499999999999998,"15min":0.28899999999999998},"max":{"1min":6.4619999999999997,"5min":7.008,"15min":37.984000000000002},"last":0.503}]},{"osd":3,"last update":"Mon Feb 5 06:45:44 2024","interfaces":[{"interface":"back","average":{"1min":4.7480000000000002,"5min":2.4140000000000001,"15min":1.538},"min":{"1min":0.33800000000000002,"5min":0.33200000000000002,"15min":0.31900000000000001},"max":{"1min":62.508000000000003,"5min":62.508000000000003,"15min":62.508000000000003},"last":1.6519999999999999},{"interface":"front","average":{"1min":5.383,"5min":2.8340000000000001,"15min":1.6919999999999999},"min":{"1min":0.36799999999999999,"5min":0.35899999999999999,"15min":0.27400000000000002},"max":{"1min":62.460000000000001,"5min":62.460000000000001,"15min":62.460000000000001},"last":0.48299999999999998}]},{"osd":4,"last update":"Mon Feb 5 06:45:44 2024","interfaces":[{"interface":"back","average":{"1min":1.2390000000000001,"5min":0.82099999999999995,"15min":0.55400000000000005},"min":{"1min":0.26800000000000002,"5min":0.25800000000000001,"15min":0.23200000000000001},"max":{"1min":4.3799999999999999,"5min":5.4969999999999999,"15min":5.4969999999999999},"last":0.436},{"interface":"front","average":{"1min":1.0469999999999999,"5min":0.72099999999999997,"15min":0.56599999999999995},"min":{"1min":0.308,"5min":0.252,"15min":0.252},"max":{"1min":4.0380000000000003,"5min":4.0380000000000003,"15min":4.0380000000000003},"last":0.36199999999999999}]},{"osd":6,"last update":"Mon Feb 5 06:46:19 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.0109999999999999,"15min":0.66900000000000004},"min":{"1min":0.34100000000000003,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":7.6710000000000003,"5min":7.6710000000000003,"15min":7.6710000000000003},"last":0.45100000000000001},{"interface":"front","average":{"1min":1.9510000000000001,"5min":1.127,"15min":0.72699999999999998},"min":{"1min":0.378,"5min":0.31900000000000001,"15min":0.27600000000000002},"max":{"1min":7.21,"5min":7.21,"15min":7.21},"last":0.41899999999999998}]},{"osd":7,"last update":"Mon Feb 5 06:45:42 2024","interfaces":[{"interface":"back","average":{"1min":1.593,"5min":0.93700000000000006,"15min":0.64600000000000002},"min":{"1min":0.36499999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":6.9870000000000001,"5min":6.9870000000000001,"15min":6.9870000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":1.659,"5min":1.0329999999999999,"15min":0.69499999999999995},"min":{"1min":0.41799999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":6.5839999999999996,"5min":6.5839999999999996,"15min":6.5839999999999996},"last":0.52100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":537067520,"data_stored":537067520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1233719296,"data_stored":1233715224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":414646272,"data_stored":414646272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1282801664,"data_stored":1282801664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":422510592,"data_stored":422510592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1247088640,"data_stored":1247084568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":425656320,"data_stored":425656320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1254363136,"data_stored":1254359064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T06:46:35.861 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:35.894 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:35.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:35.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:35.996 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:36.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:36.037 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:36.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:36.138 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:36.171 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:36.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:36.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:36.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:35 smithi028 ceph-mon[29442]: from='client.15588 v1:172.21.15.28:0/3379595239' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:46:36.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:35 smithi028 ceph-mon[29442]: pgmap v625: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.0 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s 2024-02-05T06:46:36.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:35 smithi028 ceph-mon[29442]: 1.a scrub starts 2024-02-05T06:46:36.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:35 smithi028 ceph-mon[29442]: 1.a scrub ok 2024-02-05T06:46:36.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:35 smithi028 ceph-mon[37984]: from='client.15588 v1:172.21.15.28:0/3379595239' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:46:36.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:35 smithi028 ceph-mon[37984]: pgmap v625: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.0 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s 2024-02-05T06:46:36.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:35 smithi028 ceph-mon[37984]: 1.a scrub starts 2024-02-05T06:46:36.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:35 smithi028 ceph-mon[37984]: 1.a scrub ok 2024-02-05T06:46:36.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:35 smithi167 ceph-mon[32774]: from='client.15588 v1:172.21.15.28:0/3379595239' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:46:36.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:35 smithi167 ceph-mon[32774]: pgmap v625: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.0 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.58k op/s 2024-02-05T06:46:36.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:35 smithi167 ceph-mon[32774]: 1.a scrub starts 2024-02-05T06:46:36.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:35 smithi167 ceph-mon[32774]: 1.a scrub ok 2024-02-05T06:46:36.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:36.311 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:36.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:36.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:36.413 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:36.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:36.478 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:36.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:36.579 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:36.617 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:36.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:36.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:36.719 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:36.754 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:36.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:36.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:36.856 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:36.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:36.913 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:36.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:37.014 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:37.051 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:37.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:37.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:37.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:37.191 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:37.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:37.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:37.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:36 smithi028 ceph-mon[29442]: 3.a scrub starts 2024-02-05T06:46:37.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:36 smithi028 ceph-mon[29442]: 3.a scrub ok 2024-02-05T06:46:37.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:36 smithi028 ceph-mon[29442]: 6.13 scrub starts 2024-02-05T06:46:37.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:36 smithi028 ceph-mon[29442]: 6.13 scrub ok 2024-02-05T06:46:37.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:36 smithi028 ceph-mon[29442]: 5.1e scrub starts 2024-02-05T06:46:37.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:36 smithi028 ceph-mon[29442]: 5.1e scrub ok 2024-02-05T06:46:37.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:36 smithi028 ceph-mon[29442]: 9.b scrub starts 2024-02-05T06:46:37.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:36 smithi028 ceph-mon[37984]: 3.a scrub starts 2024-02-05T06:46:37.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:36 smithi028 ceph-mon[37984]: 3.a scrub ok 2024-02-05T06:46:37.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:36 smithi028 ceph-mon[37984]: 6.13 scrub starts 2024-02-05T06:46:37.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:36 smithi028 ceph-mon[37984]: 6.13 scrub ok 2024-02-05T06:46:37.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:36 smithi028 ceph-mon[37984]: 5.1e scrub starts 2024-02-05T06:46:37.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:36 smithi028 ceph-mon[37984]: 5.1e scrub ok 2024-02-05T06:46:37.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:36 smithi028 ceph-mon[37984]: 9.b scrub starts 2024-02-05T06:46:37.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:36 smithi167 ceph-mon[32774]: 3.a scrub starts 2024-02-05T06:46:37.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:36 smithi167 ceph-mon[32774]: 3.a scrub ok 2024-02-05T06:46:37.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:36 smithi167 ceph-mon[32774]: 6.13 scrub starts 2024-02-05T06:46:37.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:36 smithi167 ceph-mon[32774]: 6.13 scrub ok 2024-02-05T06:46:37.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:36 smithi167 ceph-mon[32774]: 5.1e scrub starts 2024-02-05T06:46:37.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:36 smithi167 ceph-mon[32774]: 5.1e scrub ok 2024-02-05T06:46:37.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:36 smithi167 ceph-mon[32774]: 9.b scrub starts 2024-02-05T06:46:37.293 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:37.331 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:37.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:37.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:37.433 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:37.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:37.468 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:37.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:37.569 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:37.604 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:37.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:37.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:37.706 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:37.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:37.744 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:37.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:37.845 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:37.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:37.885 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:37.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:37.986 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:38.021 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:38.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:38.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:38.122 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:38.158 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:38.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:38.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:38.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:37 smithi028 ceph-mon[29442]: pgmap v626: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.0 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 600 B/s rd, 0 B/s wr, 758 op/s 2024-02-05T06:46:38.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:37 smithi028 ceph-mon[29442]: 2.1 scrub starts 2024-02-05T06:46:38.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:37 smithi028 ceph-mon[29442]: 2.1 scrub ok 2024-02-05T06:46:38.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:37 smithi028 ceph-mon[29442]: 9.b scrub ok 2024-02-05T06:46:38.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:37 smithi028 ceph-mon[37984]: pgmap v626: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.0 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 600 B/s rd, 0 B/s wr, 758 op/s 2024-02-05T06:46:38.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:37 smithi028 ceph-mon[37984]: 2.1 scrub starts 2024-02-05T06:46:38.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:37 smithi028 ceph-mon[37984]: 2.1 scrub ok 2024-02-05T06:46:38.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:37 smithi028 ceph-mon[37984]: 9.b scrub ok 2024-02-05T06:46:38.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:37 smithi167 ceph-mon[32774]: pgmap v626: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.0 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 600 B/s rd, 0 B/s wr, 758 op/s 2024-02-05T06:46:38.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:37 smithi167 ceph-mon[32774]: 2.1 scrub starts 2024-02-05T06:46:38.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:37 smithi167 ceph-mon[32774]: 2.1 scrub ok 2024-02-05T06:46:38.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:37 smithi167 ceph-mon[32774]: 9.b scrub ok 2024-02-05T06:46:38.260 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:38.299 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:38.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:38.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:38.401 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:38.439 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:38.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:38.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:38.540 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:38.556 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:46:38.578 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:38.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:38.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:38.610 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:46:38.654 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:46:38.680 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:38.697 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:46:38.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:38.717 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:38.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:38.739 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:46:38.779 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:46:38.818 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:38.823 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:46:38.854 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:38.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:38.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:38.861 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:46:38.864 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:46:38.864 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:46:38.864 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:46:38.864 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:46:38.864 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:46:38.864 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:46:38.864 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:46:38.865 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:46:38.956 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:38.995 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:38.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:38.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:39.097 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:39.132 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:39.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:39.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:39.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:38 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:46:39.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:38 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:46:39.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:38 smithi028 ceph-mon[29442]: 9.2 scrub starts 2024-02-05T06:46:39.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:38 smithi028 ceph-mon[29442]: 5.9 scrub starts 2024-02-05T06:46:39.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:38 smithi028 ceph-mon[29442]: 5.9 scrub ok 2024-02-05T06:46:39.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:38 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:46:39.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:38 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:46:39.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:38 smithi028 ceph-mon[29442]: osdmap e333: 8 total, 8 up, 8 in 2024-02-05T06:46:39.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:38 smithi028 ceph-mon[29442]: 9.2 scrub ok 2024-02-05T06:46:39.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:38 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1580841017' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:46:39.150 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:38 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:46:39.151 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:38 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:46:39.151 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:38 smithi028 ceph-mon[37984]: 9.2 scrub starts 2024-02-05T06:46:39.151 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:38 smithi028 ceph-mon[37984]: 5.9 scrub starts 2024-02-05T06:46:39.151 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:38 smithi028 ceph-mon[37984]: 5.9 scrub ok 2024-02-05T06:46:39.151 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:38 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:46:39.151 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:38 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:46:39.151 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:38 smithi028 ceph-mon[37984]: osdmap e333: 8 total, 8 up, 8 in 2024-02-05T06:46:39.151 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:38 smithi028 ceph-mon[37984]: 9.2 scrub ok 2024-02-05T06:46:39.151 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:38 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1580841017' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:46:39.234 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:39.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:38 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:46:39.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:38 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:46:39.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:38 smithi167 ceph-mon[32774]: 9.2 scrub starts 2024-02-05T06:46:39.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:38 smithi167 ceph-mon[32774]: 5.9 scrub starts 2024-02-05T06:46:39.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:38 smithi167 ceph-mon[32774]: 5.9 scrub ok 2024-02-05T06:46:39.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:38 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:46:39.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:38 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:46:39.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:38 smithi167 ceph-mon[32774]: osdmap e333: 8 total, 8 up, 8 in 2024-02-05T06:46:39.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:38 smithi167 ceph-mon[32774]: 9.2 scrub ok 2024-02-05T06:46:39.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:38 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1580841017' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:46:39.270 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:39.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:39.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:39.371 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:39.408 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:39.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:39.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:39.510 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:39.536 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:39.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:39.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:39.638 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:39.670 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:39.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:39.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:39.772 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:39.811 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:39.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:39.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:39.912 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:39.945 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:39.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:39.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:40.047 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:40.087 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:40.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:40.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:40.189 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:40.220 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:40.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:40.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:40.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:39 smithi167 ceph-mon[32774]: pgmap v628: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.6 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.62k op/s 2024-02-05T06:46:40.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:39 smithi167 ceph-mon[32774]: 3.5 scrub starts 2024-02-05T06:46:40.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:39 smithi167 ceph-mon[32774]: 3.5 scrub ok 2024-02-05T06:46:40.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:39 smithi167 ceph-mon[32774]: 3.13 scrub starts 2024-02-05T06:46:40.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:39 smithi167 ceph-mon[32774]: 3.13 scrub ok 2024-02-05T06:46:40.272 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:39 smithi028 ceph-mon[29442]: pgmap v628: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.6 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.62k op/s 2024-02-05T06:46:40.272 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:39 smithi028 ceph-mon[29442]: 3.5 scrub starts 2024-02-05T06:46:40.272 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:39 smithi028 ceph-mon[29442]: 3.5 scrub ok 2024-02-05T06:46:40.272 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:39 smithi028 ceph-mon[29442]: 3.13 scrub starts 2024-02-05T06:46:40.272 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:39 smithi028 ceph-mon[29442]: 3.13 scrub ok 2024-02-05T06:46:40.272 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:39 smithi028 ceph-mon[37984]: pgmap v628: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.6 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.62k op/s 2024-02-05T06:46:40.273 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:39 smithi028 ceph-mon[37984]: 3.5 scrub starts 2024-02-05T06:46:40.273 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:39 smithi028 ceph-mon[37984]: 3.5 scrub ok 2024-02-05T06:46:40.273 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:39 smithi028 ceph-mon[37984]: 3.13 scrub starts 2024-02-05T06:46:40.273 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:39 smithi028 ceph-mon[37984]: 3.13 scrub ok 2024-02-05T06:46:40.322 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:40.363 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:40.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:40.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:40.465 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:40.500 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:40.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:40.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:40.601 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:40.637 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:40.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:40.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:40.739 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:40.771 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:40.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:40.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:40.861 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 4, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T06:46:40.862 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd reweight 0 1 2024-02-05T06:46:40.873 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:40.950 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:40.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:40.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:41.052 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:41.084 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:41.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:41.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:41.186 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:41.219 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:41.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:41.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:41.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:40 smithi167 ceph-mon[32774]: 1.7 scrub starts 2024-02-05T06:46:41.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:40 smithi167 ceph-mon[32774]: 1.7 scrub ok 2024-02-05T06:46:41.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:40 smithi167 ceph-mon[32774]: 3.18 scrub starts 2024-02-05T06:46:41.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:40 smithi167 ceph-mon[32774]: 3.18 scrub ok 2024-02-05T06:46:41.272 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:40 smithi028 ceph-mon[29442]: 1.7 scrub starts 2024-02-05T06:46:41.272 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:40 smithi028 ceph-mon[29442]: 1.7 scrub ok 2024-02-05T06:46:41.272 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:40 smithi028 ceph-mon[29442]: 3.18 scrub starts 2024-02-05T06:46:41.272 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:40 smithi028 ceph-mon[29442]: 3.18 scrub ok 2024-02-05T06:46:41.272 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:40 smithi028 ceph-mon[37984]: 1.7 scrub starts 2024-02-05T06:46:41.272 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:40 smithi028 ceph-mon[37984]: 1.7 scrub ok 2024-02-05T06:46:41.272 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:40 smithi028 ceph-mon[37984]: 3.18 scrub starts 2024-02-05T06:46:41.272 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:40 smithi028 ceph-mon[37984]: 3.18 scrub ok 2024-02-05T06:46:41.321 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:41.375 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:41.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:41.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:41.477 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:41.509 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:41.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:41.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:41.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:41.651 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:41.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:41.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:41.753 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:41.791 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:41.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:41.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:41.894 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:41.926 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:41.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:41.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:42.027 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:42.090 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:46:42.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:42.117 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:42.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:42.169 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-02-05T06:46:42.219 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:42.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:41 smithi167 ceph-mon[32774]: pgmap v629: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.6 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 619 B/s rd, 0 B/s wr, 1.12k op/s 2024-02-05T06:46:42.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:41 smithi167 ceph-mon[32774]: 3.9 scrub starts 2024-02-05T06:46:42.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:41 smithi167 ceph-mon[32774]: 3.9 scrub ok 2024-02-05T06:46:42.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:41 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:46:42.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:41 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2581895937' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:46:42.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:41 smithi167 ceph-mon[32774]: from='client.25507 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:46:42.239 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:41 smithi028 ceph-mon[37984]: pgmap v629: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.6 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 619 B/s rd, 0 B/s wr, 1.12k op/s 2024-02-05T06:46:42.239 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:41 smithi028 ceph-mon[37984]: 3.9 scrub starts 2024-02-05T06:46:42.239 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:41 smithi028 ceph-mon[37984]: 3.9 scrub ok 2024-02-05T06:46:42.239 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:41 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:46:42.239 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:41 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2581895937' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:46:42.239 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:41 smithi028 ceph-mon[37984]: from='client.25507 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:46:42.239 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:41 smithi028 ceph-mon[29442]: pgmap v629: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.6 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 619 B/s rd, 0 B/s wr, 1.12k op/s 2024-02-05T06:46:42.239 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:41 smithi028 ceph-mon[29442]: 3.9 scrub starts 2024-02-05T06:46:42.239 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:41 smithi028 ceph-mon[29442]: 3.9 scrub ok 2024-02-05T06:46:42.240 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:41 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:46:42.240 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:41 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2581895937' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:46:42.240 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:41 smithi028 ceph-mon[29442]: from='client.25507 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:46:42.269 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:42.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:42.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:42.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:42.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:42.409 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:42.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:42.510 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:42.541 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:42.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:42.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:42.643 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:42.681 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:42.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:42.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:42.783 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:42.815 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:42.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:42.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:42.917 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:42.949 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:46:42 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:46:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:46:42.955 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:42.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:42.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:43.057 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:43.098 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:43.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:43.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:43.183 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:46:43.199 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:43.232 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:42 smithi028 ceph-mon[29442]: 5.10 scrub starts 2024-02-05T06:46:43.232 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:42 smithi028 ceph-mon[29442]: 5.10 scrub ok 2024-02-05T06:46:43.232 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:42 smithi028 ceph-mon[29442]: 9.4 scrub starts 2024-02-05T06:46:43.232 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:42 smithi028 ceph-mon[29442]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:46:43.232 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:42 smithi028 ceph-mon[29442]: Cluster is now healthy 2024-02-05T06:46:43.232 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:42 smithi028 ceph-mon[29442]: from='client.25507 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:46:43.232 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:42 smithi028 ceph-mon[29442]: osdmap e334: 8 total, 8 up, 8 in 2024-02-05T06:46:43.233 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:42 smithi028 ceph-mon[37984]: 5.10 scrub starts 2024-02-05T06:46:43.233 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:42 smithi028 ceph-mon[37984]: 5.10 scrub ok 2024-02-05T06:46:43.233 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:42 smithi028 ceph-mon[37984]: 9.4 scrub starts 2024-02-05T06:46:43.233 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:42 smithi028 ceph-mon[37984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:46:43.233 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:42 smithi028 ceph-mon[37984]: Cluster is now healthy 2024-02-05T06:46:43.233 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:42 smithi028 ceph-mon[37984]: from='client.25507 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:46:43.233 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:42 smithi028 ceph-mon[37984]: osdmap e334: 8 total, 8 up, 8 in 2024-02-05T06:46:43.234 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:46:43 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:46:43.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:42 smithi167 ceph-mon[32774]: 5.10 scrub starts 2024-02-05T06:46:43.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:42 smithi167 ceph-mon[32774]: 5.10 scrub ok 2024-02-05T06:46:43.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:42 smithi167 ceph-mon[32774]: 9.4 scrub starts 2024-02-05T06:46:43.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:42 smithi167 ceph-mon[32774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:46:43.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:42 smithi167 ceph-mon[32774]: Cluster is now healthy 2024-02-05T06:46:43.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:42 smithi167 ceph-mon[32774]: from='client.25507 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:46:43.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:42 smithi167 ceph-mon[32774]: osdmap e334: 8 total, 8 up, 8 in 2024-02-05T06:46:43.276 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:43.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:43.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:43.380 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:43.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:43.415 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:43.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:43.516 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:43.558 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:43.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:43.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:43.661 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:43.695 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:43.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:43.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:43.797 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:43.829 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:43.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:43.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:43.931 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:43.983 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:43.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:43.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:44.085 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:44.115 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:44.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:44.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:44.155 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:46:44.155 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-02-05T06:46:44.155 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-02-05T06:46:44.155 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-02-05T06:46:44.155 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:46:44.156 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:43 smithi028 ceph-mon[37984]: 5.1c deep-scrub starts 2024-02-05T06:46:44.156 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:43 smithi028 ceph-mon[37984]: 5.1c deep-scrub ok 2024-02-05T06:46:44.156 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:43 smithi028 ceph-mon[37984]: pgmap v631: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 1.4 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-02-05T06:46:44.156 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:43 smithi028 ceph-mon[37984]: 9.4 scrub ok 2024-02-05T06:46:44.156 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:43 smithi028 ceph-mon[37984]: 5.8 deep-scrub starts 2024-02-05T06:46:44.156 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:43 smithi028 ceph-mon[37984]: 5.8 deep-scrub ok 2024-02-05T06:46:44.156 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:43 smithi028 ceph-mon[37984]: 1.3 scrub starts 2024-02-05T06:46:44.156 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:43 smithi028 ceph-mon[37984]: 1.3 scrub ok 2024-02-05T06:46:44.156 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:43 smithi028 ceph-mon[29442]: 5.1c deep-scrub starts 2024-02-05T06:46:44.156 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:43 smithi028 ceph-mon[29442]: 5.1c deep-scrub ok 2024-02-05T06:46:44.157 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:43 smithi028 ceph-mon[29442]: pgmap v631: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 1.4 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-02-05T06:46:44.157 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:43 smithi028 ceph-mon[29442]: 9.4 scrub ok 2024-02-05T06:46:44.157 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:43 smithi028 ceph-mon[29442]: 5.8 deep-scrub starts 2024-02-05T06:46:44.157 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:43 smithi028 ceph-mon[29442]: 5.8 deep-scrub ok 2024-02-05T06:46:44.157 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:43 smithi028 ceph-mon[29442]: 1.3 scrub starts 2024-02-05T06:46:44.157 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:43 smithi028 ceph-mon[29442]: 1.3 scrub ok 2024-02-05T06:46:44.216 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:44.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:43 smithi167 ceph-mon[32774]: 5.1c deep-scrub starts 2024-02-05T06:46:44.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:43 smithi167 ceph-mon[32774]: 5.1c deep-scrub ok 2024-02-05T06:46:44.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:43 smithi167 ceph-mon[32774]: pgmap v631: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 1.4 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-02-05T06:46:44.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:43 smithi167 ceph-mon[32774]: 9.4 scrub ok 2024-02-05T06:46:44.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:43 smithi167 ceph-mon[32774]: 5.8 deep-scrub starts 2024-02-05T06:46:44.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:43 smithi167 ceph-mon[32774]: 5.8 deep-scrub ok 2024-02-05T06:46:44.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:43 smithi167 ceph-mon[32774]: 1.3 scrub starts 2024-02-05T06:46:44.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:43 smithi167 ceph-mon[32774]: 1.3 scrub ok 2024-02-05T06:46:44.247 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:44.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:44.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:44.348 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:44.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:44.385 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:44.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:44.486 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:44.526 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:44.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:44.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:44.627 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:44.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:44.662 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:44.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:44.763 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:44.799 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:44.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:44.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:44.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:44.936 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:44.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:44.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:45.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:45.074 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:45.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:45.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:45.176 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:45.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:44 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:46:45.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:44 smithi028 ceph-mon[29442]: 3.10 scrub starts 2024-02-05T06:46:45.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:44 smithi028 ceph-mon[29442]: 3.10 scrub ok 2024-02-05T06:46:45.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:44 smithi028 ceph-mon[29442]: 3.11 deep-scrub starts 2024-02-05T06:46:45.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:44 smithi028 ceph-mon[29442]: 3.11 deep-scrub ok 2024-02-05T06:46:45.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:44 smithi028 ceph-mon[29442]: 4.7 scrub starts 2024-02-05T06:46:45.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:44 smithi028 ceph-mon[29442]: 4.7 scrub ok 2024-02-05T06:46:45.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:44 smithi028 ceph-mon[29442]: 4.f deep-scrub starts 2024-02-05T06:46:45.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:44 smithi028 ceph-mon[29442]: 4.f deep-scrub ok 2024-02-05T06:46:45.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:44 smithi028 ceph-mon[29442]: 9.1 scrub starts 2024-02-05T06:46:45.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:44 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:46:45.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:44 smithi028 ceph-mon[37984]: 3.10 scrub starts 2024-02-05T06:46:45.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:44 smithi028 ceph-mon[37984]: 3.10 scrub ok 2024-02-05T06:46:45.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:44 smithi028 ceph-mon[37984]: 3.11 deep-scrub starts 2024-02-05T06:46:45.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:44 smithi028 ceph-mon[37984]: 3.11 deep-scrub ok 2024-02-05T06:46:45.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:44 smithi028 ceph-mon[37984]: 4.7 scrub starts 2024-02-05T06:46:45.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:44 smithi028 ceph-mon[37984]: 4.7 scrub ok 2024-02-05T06:46:45.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:44 smithi028 ceph-mon[37984]: 4.f deep-scrub starts 2024-02-05T06:46:45.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:44 smithi028 ceph-mon[37984]: 4.f deep-scrub ok 2024-02-05T06:46:45.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:44 smithi028 ceph-mon[37984]: 9.1 scrub starts 2024-02-05T06:46:45.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:44 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:46:45.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:44 smithi167 ceph-mon[32774]: 3.10 scrub starts 2024-02-05T06:46:45.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:44 smithi167 ceph-mon[32774]: 3.10 scrub ok 2024-02-05T06:46:45.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:44 smithi167 ceph-mon[32774]: 3.11 deep-scrub starts 2024-02-05T06:46:45.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:44 smithi167 ceph-mon[32774]: 3.11 deep-scrub ok 2024-02-05T06:46:45.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:44 smithi167 ceph-mon[32774]: 4.7 scrub starts 2024-02-05T06:46:45.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:44 smithi167 ceph-mon[32774]: 4.7 scrub ok 2024-02-05T06:46:45.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:44 smithi167 ceph-mon[32774]: 4.f deep-scrub starts 2024-02-05T06:46:45.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:44 smithi167 ceph-mon[32774]: 4.f deep-scrub ok 2024-02-05T06:46:45.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:44 smithi167 ceph-mon[32774]: 9.1 scrub starts 2024-02-05T06:46:45.279 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:45.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:45.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:45.381 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:45.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:45.417 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:45.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:45.518 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:45.555 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:45.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:45.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:45.560 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T06:46:45.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:45.708 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:45.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:45.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:45.810 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:45.857 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:45.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:45.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:45.958 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:46.000 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:46.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:46.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:46.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:46.128 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:46.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:46.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:46.230 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:46.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:45 smithi167 ceph-mon[32774]: pgmap v632: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.3 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-02-05T06:46:46.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:45 smithi167 ceph-mon[32774]: 1.8 scrub starts 2024-02-05T06:46:46.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:45 smithi167 ceph-mon[32774]: 1.8 scrub ok 2024-02-05T06:46:46.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:45 smithi167 ceph-mon[32774]: 1.2 scrub starts 2024-02-05T06:46:46.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:45 smithi167 ceph-mon[32774]: 1.2 scrub ok 2024-02-05T06:46:46.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:45 smithi167 ceph-mon[32774]: 9.1 scrub ok 2024-02-05T06:46:46.269 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:46.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:46.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:46.371 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:46.409 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:46.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:46.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:46.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:45 smithi028 ceph-mon[29442]: pgmap v632: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.3 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-02-05T06:46:46.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:45 smithi028 ceph-mon[29442]: 1.8 scrub starts 2024-02-05T06:46:46.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:45 smithi028 ceph-mon[29442]: 1.8 scrub ok 2024-02-05T06:46:46.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:45 smithi028 ceph-mon[29442]: 1.2 scrub starts 2024-02-05T06:46:46.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:45 smithi028 ceph-mon[29442]: 1.2 scrub ok 2024-02-05T06:46:46.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:45 smithi028 ceph-mon[29442]: 9.1 scrub ok 2024-02-05T06:46:46.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:45 smithi028 ceph-mon[37984]: pgmap v632: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.3 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s 2024-02-05T06:46:46.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:45 smithi028 ceph-mon[37984]: 1.8 scrub starts 2024-02-05T06:46:46.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:45 smithi028 ceph-mon[37984]: 1.8 scrub ok 2024-02-05T06:46:46.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:45 smithi028 ceph-mon[37984]: 1.2 scrub starts 2024-02-05T06:46:46.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:45 smithi028 ceph-mon[37984]: 1.2 scrub ok 2024-02-05T06:46:46.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:45 smithi028 ceph-mon[37984]: 9.1 scrub ok 2024-02-05T06:46:46.511 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:46.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:46.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:46.650 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:46.685 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:46.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:46.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:46.787 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:46.822 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:46.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:46.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:46.923 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:46.959 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:46.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:46.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:47.061 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:47.099 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:47.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:47.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:47.201 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:47.226 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:47.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:47.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:47.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:46 smithi167 ceph-mon[32774]: 6.d deep-scrub starts 2024-02-05T06:46:47.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:46 smithi167 ceph-mon[32774]: 6.5 deep-scrub starts 2024-02-05T06:46:47.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:46 smithi167 ceph-mon[32774]: 6.1d scrub starts 2024-02-05T06:46:47.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:46 smithi167 ceph-mon[32774]: 6.d deep-scrub ok 2024-02-05T06:46:47.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:46 smithi167 ceph-mon[32774]: 6.5 deep-scrub ok 2024-02-05T06:46:47.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:46 smithi167 ceph-mon[32774]: 6.1d scrub ok 2024-02-05T06:46:47.328 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:47.369 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:47.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:47.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:47.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:46 smithi028 ceph-mon[29442]: 6.d deep-scrub starts 2024-02-05T06:46:47.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:46 smithi028 ceph-mon[29442]: 6.5 deep-scrub starts 2024-02-05T06:46:47.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:46 smithi028 ceph-mon[29442]: 6.1d scrub starts 2024-02-05T06:46:47.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:46 smithi028 ceph-mon[29442]: 6.d deep-scrub ok 2024-02-05T06:46:47.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:46 smithi028 ceph-mon[29442]: 6.5 deep-scrub ok 2024-02-05T06:46:47.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:46 smithi028 ceph-mon[29442]: 6.1d scrub ok 2024-02-05T06:46:47.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:46 smithi028 ceph-mon[37984]: 6.d deep-scrub starts 2024-02-05T06:46:47.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:46 smithi028 ceph-mon[37984]: 6.5 deep-scrub starts 2024-02-05T06:46:47.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:46 smithi028 ceph-mon[37984]: 6.1d scrub starts 2024-02-05T06:46:47.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:46 smithi028 ceph-mon[37984]: 6.d deep-scrub ok 2024-02-05T06:46:47.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:46 smithi028 ceph-mon[37984]: 6.5 deep-scrub ok 2024-02-05T06:46:47.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:46 smithi028 ceph-mon[37984]: 6.1d scrub ok 2024-02-05T06:46:47.471 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:47.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:47.509 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:47.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:47.610 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:47.645 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:47.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:47.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:47.747 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:47.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:47.785 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:47.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:47.836 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-05T06:46:47.886 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:47.893 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:46:47.917 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:47.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:47.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:48.019 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:48.075 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:48.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:48.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:48.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:47 smithi028 ceph-mon[29442]: pgmap v633: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.3 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 604 B/s rd, 0 B/s wr, 713 op/s 2024-02-05T06:46:48.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:47 smithi028 ceph-mon[29442]: 3.1d scrub starts 2024-02-05T06:46:48.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:47 smithi028 ceph-mon[29442]: 3.1d scrub ok 2024-02-05T06:46:48.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:47 smithi028 ceph-mon[29442]: 5.6 scrub starts 2024-02-05T06:46:48.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:47 smithi028 ceph-mon[29442]: 5.6 scrub ok 2024-02-05T06:46:48.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:47 smithi028 ceph-mon[29442]: 6.8 scrub starts 2024-02-05T06:46:48.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:47 smithi028 ceph-mon[29442]: 6.8 scrub ok 2024-02-05T06:46:48.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:47 smithi028 ceph-mon[29442]: 4.17 scrub starts 2024-02-05T06:46:48.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:47 smithi028 ceph-mon[29442]: 4.17 scrub ok 2024-02-05T06:46:48.116 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:47 smithi028 ceph-mon[37984]: pgmap v633: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.3 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 604 B/s rd, 0 B/s wr, 713 op/s 2024-02-05T06:46:48.116 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:47 smithi028 ceph-mon[37984]: 3.1d scrub starts 2024-02-05T06:46:48.116 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:47 smithi028 ceph-mon[37984]: 3.1d scrub ok 2024-02-05T06:46:48.116 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:47 smithi028 ceph-mon[37984]: 5.6 scrub starts 2024-02-05T06:46:48.116 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:47 smithi028 ceph-mon[37984]: 5.6 scrub ok 2024-02-05T06:46:48.116 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:47 smithi028 ceph-mon[37984]: 6.8 scrub starts 2024-02-05T06:46:48.117 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:47 smithi028 ceph-mon[37984]: 6.8 scrub ok 2024-02-05T06:46:48.117 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:47 smithi028 ceph-mon[37984]: 4.17 scrub starts 2024-02-05T06:46:48.117 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:47 smithi028 ceph-mon[37984]: 4.17 scrub ok 2024-02-05T06:46:48.177 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:48.216 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:48.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:48.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:48.317 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:48.358 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:48.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:48.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:48.460 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:48.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:47 smithi167 ceph-mon[32774]: pgmap v633: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.3 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 604 B/s rd, 0 B/s wr, 713 op/s 2024-02-05T06:46:48.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:47 smithi167 ceph-mon[32774]: 3.1d scrub starts 2024-02-05T06:46:48.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:47 smithi167 ceph-mon[32774]: 3.1d scrub ok 2024-02-05T06:46:48.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:47 smithi167 ceph-mon[32774]: 5.6 scrub starts 2024-02-05T06:46:48.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:47 smithi167 ceph-mon[32774]: 5.6 scrub ok 2024-02-05T06:46:48.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:47 smithi167 ceph-mon[32774]: 6.8 scrub starts 2024-02-05T06:46:48.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:47 smithi167 ceph-mon[32774]: 6.8 scrub ok 2024-02-05T06:46:48.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:47 smithi167 ceph-mon[32774]: 4.17 scrub starts 2024-02-05T06:46:48.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:47 smithi167 ceph-mon[32774]: 4.17 scrub ok 2024-02-05T06:46:48.500 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:48.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:48.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:48.601 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:48.638 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:48.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:48.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:48.740 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:48.772 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:48.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:48.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:48.874 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:48.910 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:48.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:48.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:49.012 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:49.047 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:49.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:49.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:49.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:48 smithi028 ceph-mon[37984]: 1.1 scrub starts 2024-02-05T06:46:49.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:48 smithi028 ceph-mon[37984]: 1.1 scrub ok 2024-02-05T06:46:49.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:48 smithi028 ceph-mon[37984]: 5.d scrub starts 2024-02-05T06:46:49.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:48 smithi028 ceph-mon[37984]: 5.d scrub ok 2024-02-05T06:46:49.135 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:48 smithi028 ceph-mon[37984]: 6.6 scrub starts 2024-02-05T06:46:49.135 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:48 smithi028 ceph-mon[29442]: 1.1 scrub starts 2024-02-05T06:46:49.136 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:48 smithi028 ceph-mon[29442]: 1.1 scrub ok 2024-02-05T06:46:49.136 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:48 smithi028 ceph-mon[29442]: 5.d scrub starts 2024-02-05T06:46:49.136 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:48 smithi028 ceph-mon[29442]: 5.d scrub ok 2024-02-05T06:46:49.136 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:48 smithi028 ceph-mon[29442]: 6.6 scrub starts 2024-02-05T06:46:49.149 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:49.182 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:49.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:49.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:49.284 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:49.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:49.320 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:49.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:49.422 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:49.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:49.466 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:49.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:49.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:48 smithi167 ceph-mon[32774]: 1.1 scrub starts 2024-02-05T06:46:49.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:48 smithi167 ceph-mon[32774]: 1.1 scrub ok 2024-02-05T06:46:49.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:48 smithi167 ceph-mon[32774]: 5.d scrub starts 2024-02-05T06:46:49.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:48 smithi167 ceph-mon[32774]: 5.d scrub ok 2024-02-05T06:46:49.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:48 smithi167 ceph-mon[32774]: 6.6 scrub starts 2024-02-05T06:46:49.567 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:49.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:49.610 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:49.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:49.711 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:49.741 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:49.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:49.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:49.843 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:49.874 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:49.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:49.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:49.947 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:46:49.976 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:50.026 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:50.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:50.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:50.128 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:50.175 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:50.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:50.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:50.277 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:50.315 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:50.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:50.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:50.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:49 smithi028 ceph-mon[29442]: pgmap v634: 158 pgs: 158 active+clean; 922 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-02-05T06:46:50.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:49 smithi028 ceph-mon[29442]: 6.6 scrub ok 2024-02-05T06:46:50.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:49 smithi028 ceph-mon[29442]: 6.7 scrub starts 2024-02-05T06:46:50.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:49 smithi028 ceph-mon[29442]: 6.7 scrub ok 2024-02-05T06:46:50.363 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:49 smithi028 ceph-mon[37984]: pgmap v634: 158 pgs: 158 active+clean; 922 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-02-05T06:46:50.363 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:49 smithi028 ceph-mon[37984]: 6.6 scrub ok 2024-02-05T06:46:50.363 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:49 smithi028 ceph-mon[37984]: 6.7 scrub starts 2024-02-05T06:46:50.363 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:49 smithi028 ceph-mon[37984]: 6.7 scrub ok 2024-02-05T06:46:50.417 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:50.446 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:50.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:50.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:50.447 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:46:50.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:49 smithi167 ceph-mon[32774]: pgmap v634: 158 pgs: 158 active+clean; 922 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-02-05T06:46:50.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:49 smithi167 ceph-mon[32774]: 6.6 scrub ok 2024-02-05T06:46:50.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:49 smithi167 ceph-mon[32774]: 6.7 scrub starts 2024-02-05T06:46:50.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:49 smithi167 ceph-mon[32774]: 6.7 scrub ok 2024-02-05T06:46:50.548 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:50.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:50.585 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:50.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:50.686 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:50.723 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:50.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:50.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:50.825 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:50.857 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:50.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:50.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:50.959 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:50.995 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:50.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:50.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:51.018 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.0 to 1 (10000) 2024-02-05T06:46:51.097 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:51.132 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:51.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:51.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:51.234 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:51.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:51.278 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:51.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:51.379 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:51.405 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:51.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:51.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:51.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:50 smithi028 ceph-mon[29442]: 6.1e scrub starts 2024-02-05T06:46:51.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:50 smithi028 ceph-mon[29442]: 6.1e scrub ok 2024-02-05T06:46:51.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:50 smithi028 ceph-mon[29442]: 4.15 scrub starts 2024-02-05T06:46:51.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:50 smithi028 ceph-mon[29442]: 4.15 scrub ok 2024-02-05T06:46:51.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:50 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1027431303' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T06:46:51.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:50 smithi028 ceph-mon[37984]: 6.1e scrub starts 2024-02-05T06:46:51.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:50 smithi028 ceph-mon[37984]: 6.1e scrub ok 2024-02-05T06:46:51.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:50 smithi028 ceph-mon[37984]: 4.15 scrub starts 2024-02-05T06:46:51.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:50 smithi028 ceph-mon[37984]: 4.15 scrub ok 2024-02-05T06:46:51.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:50 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1027431303' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T06:46:51.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:50 smithi167 ceph-mon[32774]: 6.1e scrub starts 2024-02-05T06:46:51.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:50 smithi167 ceph-mon[32774]: 6.1e scrub ok 2024-02-05T06:46:51.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:50 smithi167 ceph-mon[32774]: 4.15 scrub starts 2024-02-05T06:46:51.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:50 smithi167 ceph-mon[32774]: 4.15 scrub ok 2024-02-05T06:46:51.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:50 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1027431303' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T06:46:51.507 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:51.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:51.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:51.652 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:51.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:51.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:51.791 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:51.828 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:51.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:51.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:51.929 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:51.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:51.961 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:51.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:52.062 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:52.096 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:52.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:52.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:52.198 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:52.234 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:52.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:52.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:52.336 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:52.369 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:52.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:52.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:52.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:52 smithi028 ceph-mon[37984]: pgmap v635: 158 pgs: 158 active+clean; 922 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-02-05T06:46:52.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:52 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1027431303' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T06:46:52.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:52 smithi028 ceph-mon[37984]: osdmap e335: 8 total, 8 up, 8 in 2024-02-05T06:46:52.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:52 smithi028 ceph-mon[29442]: pgmap v635: 158 pgs: 158 active+clean; 922 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-02-05T06:46:52.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:52 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1027431303' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T06:46:52.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:52 smithi028 ceph-mon[29442]: osdmap e335: 8 total, 8 up, 8 in 2024-02-05T06:46:52.470 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:52.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:52 smithi167 ceph-mon[32774]: pgmap v635: 158 pgs: 158 active+clean; 922 MiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-02-05T06:46:52.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:52 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1027431303' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T06:46:52.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:52 smithi167 ceph-mon[32774]: osdmap e335: 8 total, 8 up, 8 in 2024-02-05T06:46:52.515 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:52.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:52.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:52.617 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:52.650 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:52.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:52.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:52.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:52.753 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:46:52 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:46:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:46:52.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:52.788 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:52.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:52.889 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:52.922 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:52.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:52.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:53.025 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:53.067 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:53.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:53.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:53.169 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:53.207 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:53.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:53.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:53.254 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd reweight 1 1 2024-02-05T06:46:53.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:53 smithi028 ceph-mon[29442]: 5.18 deep-scrub starts 2024-02-05T06:46:53.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:53 smithi028 ceph-mon[29442]: 5.18 deep-scrub ok 2024-02-05T06:46:53.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:53 smithi028 ceph-mon[29442]: 9.c scrub starts 2024-02-05T06:46:53.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:53 smithi028 ceph-mon[29442]: 3.16 scrub starts 2024-02-05T06:46:53.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:53 smithi028 ceph-mon[29442]: 3.16 scrub ok 2024-02-05T06:46:53.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:53 smithi028 ceph-mon[29442]: 6.17 scrub starts 2024-02-05T06:46:53.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:53 smithi028 ceph-mon[29442]: 6.17 scrub ok 2024-02-05T06:46:53.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:53 smithi028 ceph-mon[29442]: 9.c scrub ok 2024-02-05T06:46:53.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:53 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:46:53.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:53 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:46:53.296 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:53 smithi028 ceph-mon[29442]: 5.3 deep-scrub starts 2024-02-05T06:46:53.296 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:53 smithi028 ceph-mon[29442]: 5.3 deep-scrub ok 2024-02-05T06:46:53.296 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:53 smithi028 ceph-mon[29442]: 3.8 deep-scrub starts 2024-02-05T06:46:53.296 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:53 smithi028 ceph-mon[29442]: 3.8 deep-scrub ok 2024-02-05T06:46:53.296 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:53 smithi028 ceph-mon[37984]: 5.18 deep-scrub starts 2024-02-05T06:46:53.296 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:53 smithi028 ceph-mon[37984]: 5.18 deep-scrub ok 2024-02-05T06:46:53.296 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:53 smithi028 ceph-mon[37984]: 9.c scrub starts 2024-02-05T06:46:53.296 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:53 smithi028 ceph-mon[37984]: 3.16 scrub starts 2024-02-05T06:46:53.296 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:53 smithi028 ceph-mon[37984]: 3.16 scrub ok 2024-02-05T06:46:53.296 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:53 smithi028 ceph-mon[37984]: 6.17 scrub starts 2024-02-05T06:46:53.296 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:53 smithi028 ceph-mon[37984]: 6.17 scrub ok 2024-02-05T06:46:53.296 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:53 smithi028 ceph-mon[37984]: 9.c scrub ok 2024-02-05T06:46:53.297 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:53 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:46:53.297 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:53 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:46:53.297 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:53 smithi028 ceph-mon[37984]: 5.3 deep-scrub starts 2024-02-05T06:46:53.297 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:53 smithi028 ceph-mon[37984]: 5.3 deep-scrub ok 2024-02-05T06:46:53.297 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:53 smithi028 ceph-mon[37984]: 3.8 deep-scrub starts 2024-02-05T06:46:53.297 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:53 smithi028 ceph-mon[37984]: 3.8 deep-scrub ok 2024-02-05T06:46:53.309 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:53.342 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:53.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:53.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:53.444 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:53.484 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:46:53 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:46:53.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:53 smithi167 ceph-mon[32774]: 5.18 deep-scrub starts 2024-02-05T06:46:53.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:53 smithi167 ceph-mon[32774]: 5.18 deep-scrub ok 2024-02-05T06:46:53.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:53 smithi167 ceph-mon[32774]: 9.c scrub starts 2024-02-05T06:46:53.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:53 smithi167 ceph-mon[32774]: 3.16 scrub starts 2024-02-05T06:46:53.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:53 smithi167 ceph-mon[32774]: 3.16 scrub ok 2024-02-05T06:46:53.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:53 smithi167 ceph-mon[32774]: 6.17 scrub starts 2024-02-05T06:46:53.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:53 smithi167 ceph-mon[32774]: 6.17 scrub ok 2024-02-05T06:46:53.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:53 smithi167 ceph-mon[32774]: 9.c scrub ok 2024-02-05T06:46:53.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:53 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:46:53.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:53 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:46:53.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:53 smithi167 ceph-mon[32774]: 5.3 deep-scrub starts 2024-02-05T06:46:53.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:53 smithi167 ceph-mon[32774]: 5.3 deep-scrub ok 2024-02-05T06:46:53.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:53 smithi167 ceph-mon[32774]: 3.8 deep-scrub starts 2024-02-05T06:46:53.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:53 smithi167 ceph-mon[32774]: 3.8 deep-scrub ok 2024-02-05T06:46:53.504 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:53.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:53.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:53.606 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:53.646 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:53.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:53.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:53.748 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:53.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:53.794 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:53.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:53.895 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:53.927 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:53.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:53.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:54.029 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:54.061 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:54.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:54.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:54.163 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:54.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:54.205 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:54.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:54.305 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:54.340 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:54.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:54.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:54.442 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:54.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:54 smithi028 ceph-mon[37984]: 4.13 scrub starts 2024-02-05T06:46:54.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:54 smithi028 ceph-mon[37984]: 4.13 scrub ok 2024-02-05T06:46:54.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:54 smithi028 ceph-mon[37984]: pgmap v637: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 641 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-02-05T06:46:54.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:54 smithi028 ceph-mon[37984]: osdmap e336: 8 total, 8 up, 8 in 2024-02-05T06:46:54.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:54 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:46:54.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:54 smithi028 ceph-mon[37984]: 6.10 deep-scrub starts 2024-02-05T06:46:54.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:54 smithi028 ceph-mon[37984]: 9.8 deep-scrub starts 2024-02-05T06:46:54.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:54 smithi028 ceph-mon[37984]: 6.10 deep-scrub ok 2024-02-05T06:46:54.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:54 smithi028 ceph-mon[37984]: 4.18 scrub starts 2024-02-05T06:46:54.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:54 smithi028 ceph-mon[37984]: 4.18 scrub ok 2024-02-05T06:46:54.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:54 smithi028 ceph-mon[29442]: 4.13 scrub starts 2024-02-05T06:46:54.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:54 smithi028 ceph-mon[29442]: 4.13 scrub ok 2024-02-05T06:46:54.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:54 smithi028 ceph-mon[29442]: pgmap v637: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 641 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-02-05T06:46:54.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:54 smithi028 ceph-mon[29442]: osdmap e336: 8 total, 8 up, 8 in 2024-02-05T06:46:54.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:54 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:46:54.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:54 smithi028 ceph-mon[29442]: 6.10 deep-scrub starts 2024-02-05T06:46:54.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:54 smithi028 ceph-mon[29442]: 9.8 deep-scrub starts 2024-02-05T06:46:54.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:54 smithi028 ceph-mon[29442]: 6.10 deep-scrub ok 2024-02-05T06:46:54.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:54 smithi028 ceph-mon[29442]: 4.18 scrub starts 2024-02-05T06:46:54.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:54 smithi028 ceph-mon[29442]: 4.18 scrub ok 2024-02-05T06:46:54.479 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:54.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:54.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:54.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:54 smithi167 ceph-mon[32774]: 4.13 scrub starts 2024-02-05T06:46:54.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:54 smithi167 ceph-mon[32774]: 4.13 scrub ok 2024-02-05T06:46:54.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:54 smithi167 ceph-mon[32774]: pgmap v637: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 641 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.22k op/s 2024-02-05T06:46:54.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:54 smithi167 ceph-mon[32774]: osdmap e336: 8 total, 8 up, 8 in 2024-02-05T06:46:54.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:54 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:46:54.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:54 smithi167 ceph-mon[32774]: 6.10 deep-scrub starts 2024-02-05T06:46:54.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:54 smithi167 ceph-mon[32774]: 9.8 deep-scrub starts 2024-02-05T06:46:54.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:54 smithi167 ceph-mon[32774]: 6.10 deep-scrub ok 2024-02-05T06:46:54.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:54 smithi167 ceph-mon[32774]: 4.18 scrub starts 2024-02-05T06:46:54.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:54 smithi167 ceph-mon[32774]: 4.18 scrub ok 2024-02-05T06:46:54.581 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:54.610 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:54.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:54.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:54.711 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:54.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:54.744 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:54.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:54.845 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:54.880 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:54.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:54.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:54.982 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:55.022 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:55.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:55.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:55.124 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:55.155 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:55.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:55.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:55.199 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:46:55.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:55 smithi028 ceph-mon[29442]: 1.6 scrub starts 2024-02-05T06:46:55.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:55 smithi028 ceph-mon[29442]: 1.6 scrub ok 2024-02-05T06:46:55.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:55 smithi028 ceph-mon[29442]: 3.e scrub starts 2024-02-05T06:46:55.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:55 smithi028 ceph-mon[29442]: 3.e scrub ok 2024-02-05T06:46:55.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:55 smithi028 ceph-mon[29442]: 5.c scrub starts 2024-02-05T06:46:55.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:55 smithi028 ceph-mon[29442]: 5.c scrub ok 2024-02-05T06:46:55.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:55 smithi028 ceph-mon[29442]: pgmap v639: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 547 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s 2024-02-05T06:46:55.208 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:55 smithi028 ceph-mon[29442]: 6.1f scrub starts 2024-02-05T06:46:55.209 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:55 smithi028 ceph-mon[37984]: 1.6 scrub starts 2024-02-05T06:46:55.209 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:55 smithi028 ceph-mon[37984]: 1.6 scrub ok 2024-02-05T06:46:55.209 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:55 smithi028 ceph-mon[37984]: 3.e scrub starts 2024-02-05T06:46:55.209 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:55 smithi028 ceph-mon[37984]: 3.e scrub ok 2024-02-05T06:46:55.209 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:55 smithi028 ceph-mon[37984]: 5.c scrub starts 2024-02-05T06:46:55.209 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:55 smithi028 ceph-mon[37984]: 5.c scrub ok 2024-02-05T06:46:55.209 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:55 smithi028 ceph-mon[37984]: pgmap v639: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 547 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s 2024-02-05T06:46:55.209 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:55 smithi028 ceph-mon[37984]: 6.1f scrub starts 2024-02-05T06:46:55.246 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:46:55.257 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:55.288 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:46:55.290 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:55.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:55.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:55.327 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:46:55.365 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:46:55.392 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:55.407 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:46:55.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:55.424 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:55.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:55.446 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:46:55.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:55 smithi167 ceph-mon[32774]: 1.6 scrub starts 2024-02-05T06:46:55.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:55 smithi167 ceph-mon[32774]: 1.6 scrub ok 2024-02-05T06:46:55.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:55 smithi167 ceph-mon[32774]: 3.e scrub starts 2024-02-05T06:46:55.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:55 smithi167 ceph-mon[32774]: 3.e scrub ok 2024-02-05T06:46:55.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:55 smithi167 ceph-mon[32774]: 5.c scrub starts 2024-02-05T06:46:55.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:55 smithi167 ceph-mon[32774]: 5.c scrub ok 2024-02-05T06:46:55.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:55 smithi167 ceph-mon[32774]: pgmap v639: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 547 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.52k op/s 2024-02-05T06:46:55.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:55 smithi167 ceph-mon[32774]: 6.1f scrub starts 2024-02-05T06:46:55.487 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:46:55.499 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:46:55.499 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:46:55.499 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:46:55.499 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:46:55.500 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:46:55.500 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:46:55.500 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:46:55.500 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:46:55.525 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:55.555 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:55.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:55.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:55.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:55.702 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:55.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:55.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:55.803 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:55.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:55.837 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:55.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:55.937 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:55.977 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:55.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:55.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:56.079 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:56.110 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:56.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:56.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:56.212 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:56.247 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:56.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:56.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:56.326 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:56 smithi028 ceph-mon[37984]: 5.11 scrub starts 2024-02-05T06:46:56.327 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:56 smithi028 ceph-mon[37984]: 5.11 scrub ok 2024-02-05T06:46:56.327 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:56 smithi028 ceph-mon[37984]: 6.1f scrub ok 2024-02-05T06:46:56.327 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:56 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/334594892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:46:56.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:56 smithi028 ceph-mon[29442]: 5.11 scrub starts 2024-02-05T06:46:56.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:56 smithi028 ceph-mon[29442]: 5.11 scrub ok 2024-02-05T06:46:56.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:56 smithi028 ceph-mon[29442]: 6.1f scrub ok 2024-02-05T06:46:56.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:56 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/334594892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:46:56.349 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:56.384 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:56.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:56.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:56.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:56 smithi167 ceph-mon[32774]: 5.11 scrub starts 2024-02-05T06:46:56.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:56 smithi167 ceph-mon[32774]: 5.11 scrub ok 2024-02-05T06:46:56.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:56 smithi167 ceph-mon[32774]: 6.1f scrub ok 2024-02-05T06:46:56.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:56 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/334594892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:46:56.487 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:56.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:56.526 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:56.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:56.626 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:56.661 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:56.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:56.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:56.763 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:56.805 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:56.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:56.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:56.906 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:56.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:56.943 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:56.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:57.044 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:57.077 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:57.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:57.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:57.179 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:57.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:57.217 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:57.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:57.318 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:57.351 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:57.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:57.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:57.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:57 smithi028 ceph-mon[29442]: 9.8 deep-scrub ok 2024-02-05T06:46:57.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:57 smithi028 ceph-mon[29442]: 5.15 deep-scrub starts 2024-02-05T06:46:57.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:57 smithi028 ceph-mon[29442]: 5.15 deep-scrub ok 2024-02-05T06:46:57.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:57 smithi028 ceph-mon[29442]: pgmap v640: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 547 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 749 op/s 2024-02-05T06:46:57.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:57 smithi028 ceph-mon[29442]: 3.1 scrub starts 2024-02-05T06:46:57.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:57 smithi028 ceph-mon[29442]: 3.1 scrub ok 2024-02-05T06:46:57.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:57 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:46:57.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:57 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:46:57.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:57 smithi028 ceph-mon[37984]: 9.8 deep-scrub ok 2024-02-05T06:46:57.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:57 smithi028 ceph-mon[37984]: 5.15 deep-scrub starts 2024-02-05T06:46:57.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:57 smithi028 ceph-mon[37984]: 5.15 deep-scrub ok 2024-02-05T06:46:57.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:57 smithi028 ceph-mon[37984]: pgmap v640: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 547 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 749 op/s 2024-02-05T06:46:57.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:57 smithi028 ceph-mon[37984]: 3.1 scrub starts 2024-02-05T06:46:57.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:57 smithi028 ceph-mon[37984]: 3.1 scrub ok 2024-02-05T06:46:57.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:57 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:46:57.386 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:57 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:46:57.453 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:57.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:57 smithi167 ceph-mon[32774]: 9.8 deep-scrub ok 2024-02-05T06:46:57.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:57 smithi167 ceph-mon[32774]: 5.15 deep-scrub starts 2024-02-05T06:46:57.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:57 smithi167 ceph-mon[32774]: 5.15 deep-scrub ok 2024-02-05T06:46:57.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:57 smithi167 ceph-mon[32774]: pgmap v640: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 547 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 749 op/s 2024-02-05T06:46:57.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:57 smithi167 ceph-mon[32774]: 3.1 scrub starts 2024-02-05T06:46:57.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:57 smithi167 ceph-mon[32774]: 3.1 scrub ok 2024-02-05T06:46:57.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:57 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:46:57.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:57 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:46:57.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:57.504 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:57.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:46:57.605 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:57.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:57.641 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:57.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:57.741 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:57.790 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:57.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:57.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:57.891 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:57.922 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:57.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:57.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:58.024 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:58.058 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:58.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:58.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:58.159 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:58.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:58 smithi028 ceph-mon[37984]: 5.16 scrub starts 2024-02-05T06:46:58.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:58 smithi028 ceph-mon[37984]: 5.16 scrub ok 2024-02-05T06:46:58.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:58 smithi028 ceph-mon[37984]: 6.14 scrub starts 2024-02-05T06:46:58.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:58 smithi028 ceph-mon[37984]: 6.14 scrub ok 2024-02-05T06:46:58.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:58 smithi028 ceph-mon[29442]: 5.16 scrub starts 2024-02-05T06:46:58.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:58 smithi028 ceph-mon[29442]: 5.16 scrub ok 2024-02-05T06:46:58.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:58 smithi028 ceph-mon[29442]: 6.14 scrub starts 2024-02-05T06:46:58.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:58 smithi028 ceph-mon[29442]: 6.14 scrub ok 2024-02-05T06:46:58.202 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:58.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:58.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:58.304 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:58.335 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:58.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:58.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:58.356 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:46:58.357 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:46:58.369 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:46:58.437 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:58.467 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:58.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:58.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:58.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:58 smithi167 ceph-mon[32774]: 5.16 scrub starts 2024-02-05T06:46:58.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:58 smithi167 ceph-mon[32774]: 5.16 scrub ok 2024-02-05T06:46:58.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:58 smithi167 ceph-mon[32774]: 6.14 scrub starts 2024-02-05T06:46:58.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:58 smithi167 ceph-mon[32774]: 6.14 scrub ok 2024-02-05T06:46:58.569 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:46:58.629 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:46:58.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:58.629 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:58.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:46:58.730 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:58.763 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:58.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:58.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:58.864 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:46:58.904 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:58.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:58.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:46:59.006 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:46:59.040 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:59.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:59.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:46:59.070 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-02-05T06:46:59.106 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-05T06:46:59.142 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:59.173 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:59.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:59.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:59.274 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:46:59.282 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Removed 129797 objects 2024-02-05T06:46:59.282 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Clean up completed and total clean up time :108.135 2024-02-05T06:46:59.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:59.301 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:59.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:46:59.402 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:59.425 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:59.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:59.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:59.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:59 smithi028 ceph-mon[37984]: 5.f scrub starts 2024-02-05T06:46:59.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:59 smithi028 ceph-mon[37984]: 5.f scrub ok 2024-02-05T06:46:59.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:59 smithi028 ceph-mon[37984]: pgmap v641: 158 pgs: 158 active+clean; 169 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s 2024-02-05T06:46:59.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:59 smithi028 ceph-mon[37984]: 6.c deep-scrub starts 2024-02-05T06:46:59.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:59 smithi028 ceph-mon[37984]: 6.18 scrub starts 2024-02-05T06:46:59.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:59 smithi028 ceph-mon[37984]: 6.c deep-scrub ok 2024-02-05T06:46:59.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:59 smithi028 ceph-mon[37984]: 6.18 scrub ok 2024-02-05T06:46:59.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:46:59 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2832559475' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:46:59.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:59 smithi028 ceph-mon[29442]: 5.f scrub starts 2024-02-05T06:46:59.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:59 smithi028 ceph-mon[29442]: 5.f scrub ok 2024-02-05T06:46:59.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:59 smithi028 ceph-mon[29442]: pgmap v641: 158 pgs: 158 active+clean; 169 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s 2024-02-05T06:46:59.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:59 smithi028 ceph-mon[29442]: 6.c deep-scrub starts 2024-02-05T06:46:59.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:59 smithi028 ceph-mon[29442]: 6.18 scrub starts 2024-02-05T06:46:59.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:59 smithi028 ceph-mon[29442]: 6.c deep-scrub ok 2024-02-05T06:46:59.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:59 smithi028 ceph-mon[29442]: 6.18 scrub ok 2024-02-05T06:46:59.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:46:59 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2832559475' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:46:59.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:59 smithi167 ceph-mon[32774]: 5.f scrub starts 2024-02-05T06:46:59.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:59 smithi167 ceph-mon[32774]: 5.f scrub ok 2024-02-05T06:46:59.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:59 smithi167 ceph-mon[32774]: pgmap v641: 158 pgs: 158 active+clean; 169 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.50k op/s 2024-02-05T06:46:59.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:59 smithi167 ceph-mon[32774]: 6.c deep-scrub starts 2024-02-05T06:46:59.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:59 smithi167 ceph-mon[32774]: 6.18 scrub starts 2024-02-05T06:46:59.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:59 smithi167 ceph-mon[32774]: 6.c deep-scrub ok 2024-02-05T06:46:59.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:59 smithi167 ceph-mon[32774]: 6.18 scrub ok 2024-02-05T06:46:59.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:46:59 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2832559475' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:46:59.527 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:46:59.550 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:59.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:59.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:46:59.652 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:46:59.675 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:59.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:59.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:46:59.777 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:46:59.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:59.801 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:46:59.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:46:59.902 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:46:59.926 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:46:59.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:46:59.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:00.031 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:00.055 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:00.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:00.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:00.157 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:00.182 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:00.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:00.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:00.283 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:00.307 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:00.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:00.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:00.409 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:00.434 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:00.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:00.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:00.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:00 smithi028 ceph-mon[37984]: 4.e scrub starts 2024-02-05T06:47:00.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:00 smithi028 ceph-mon[37984]: 4.e scrub ok 2024-02-05T06:47:00.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:00 smithi028 ceph-mon[37984]: 4.9 scrub starts 2024-02-05T06:47:00.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:00 smithi028 ceph-mon[37984]: 4.9 scrub ok 2024-02-05T06:47:00.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:00 smithi028 ceph-mon[37984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:47:00.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:00 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2832559475' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:47:00.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:00 smithi028 ceph-mon[37984]: osdmap e337: 8 total, 8 up, 8 in 2024-02-05T06:47:00.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:00 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:47:00.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:00 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:47:00.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:00 smithi028 ceph-mon[29442]: 4.e scrub starts 2024-02-05T06:47:00.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:00 smithi028 ceph-mon[29442]: 4.e scrub ok 2024-02-05T06:47:00.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:00 smithi028 ceph-mon[29442]: 4.9 scrub starts 2024-02-05T06:47:00.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:00 smithi028 ceph-mon[29442]: 4.9 scrub ok 2024-02-05T06:47:00.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:00 smithi028 ceph-mon[29442]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:47:00.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:00 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2832559475' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:47:00.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:00 smithi028 ceph-mon[29442]: osdmap e337: 8 total, 8 up, 8 in 2024-02-05T06:47:00.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:00 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:47:00.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:00 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:47:00.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:00 smithi167 ceph-mon[32774]: 4.e scrub starts 2024-02-05T06:47:00.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:00 smithi167 ceph-mon[32774]: 4.e scrub ok 2024-02-05T06:47:00.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:00 smithi167 ceph-mon[32774]: 4.9 scrub starts 2024-02-05T06:47:00.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:00 smithi167 ceph-mon[32774]: 4.9 scrub ok 2024-02-05T06:47:00.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:00 smithi167 ceph-mon[32774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:47:00.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:00 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2832559475' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:47:00.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:00 smithi167 ceph-mon[32774]: osdmap e337: 8 total, 8 up, 8 in 2024-02-05T06:47:00.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:00 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:47:00.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:00 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:47:00.536 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:00.561 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:00.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:00.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:00.663 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:00.688 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:00.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:00.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:00.790 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:00.816 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:00.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:00.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:00.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:00.946 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:00.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:00.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:01.048 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:01.075 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:01.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:01.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:01.177 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:01.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:01 smithi028 ceph-mon[29442]: osdmap e338: 8 total, 8 up, 8 in 2024-02-05T06:47:01.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:01 smithi028 ceph-mon[29442]: pgmap v644: 158 pgs: 158 active+clean; 169 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 690 B/s rd, 0 B/s wr, 815 op/s 2024-02-05T06:47:01.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:01 smithi028 ceph-mon[37984]: osdmap e338: 8 total, 8 up, 8 in 2024-02-05T06:47:01.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:01 smithi028 ceph-mon[37984]: pgmap v644: 158 pgs: 158 active+clean; 169 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 690 B/s rd, 0 B/s wr, 815 op/s 2024-02-05T06:47:01.204 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:01.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:01.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:01.306 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:01.331 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:01.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:01.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:01.356 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:47:01.396 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:47:01.425 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-02-05T06:47:01.425 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-02-05T06:47:01.432 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:01.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:01 smithi167 ceph-mon[32774]: osdmap e338: 8 total, 8 up, 8 in 2024-02-05T06:47:01.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:01 smithi167 ceph-mon[32774]: pgmap v644: 158 pgs: 158 active+clean; 169 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 690 B/s rd, 0 B/s wr, 815 op/s 2024-02-05T06:47:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:01.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:01.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:01.592 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:01.616 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:01.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:01.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:01.718 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:01.744 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:01.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:01.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:01.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T06:47:01.846 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:01.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:01.872 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:01.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:01.973 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:01.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:01.998 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:01.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:02.099 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:02.122 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:02.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:02.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:02.224 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:02.249 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:02.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:02.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:02.351 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:02.375 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:02.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:02.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:02.477 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:02.501 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:02.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:02.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:02.603 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:02.628 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:02.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:02.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:02.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:02.754 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:02.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:02.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:02.856 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:02.879 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:02.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:02.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:02.949 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:47:02 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:47:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:47:02.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:03.004 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:03.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:03.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:03.106 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:03.129 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:03.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:03.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:03.231 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:03.259 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:03.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:03.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:03.361 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:03.386 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:03.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:03.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:03.484 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:47:03 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:47:03.488 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:03.508 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.1 to 1 (10000) 2024-02-05T06:47:03.518 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:03.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:03.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:03.619 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:03.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:03 smithi028 ceph-mon[29442]: pgmap v645: 158 pgs: 158 active+clean; 33 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-02-05T06:47:03.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:03 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1270640654' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T06:47:03.640 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:03 smithi028 ceph-mon[29442]: from='client.45518 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T06:47:03.640 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:03 smithi028 ceph-mon[37984]: pgmap v645: 158 pgs: 158 active+clean; 33 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-02-05T06:47:03.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:03 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1270640654' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T06:47:03.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:03 smithi028 ceph-mon[37984]: from='client.45518 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T06:47:03.647 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:03.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:03.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:03.749 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:03.777 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:03.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:03.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:03.879 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:03.903 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:03.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:03.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:03.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:03 smithi167 ceph-mon[32774]: pgmap v645: 158 pgs: 158 active+clean; 33 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-02-05T06:47:03.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:03 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1270640654' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T06:47:03.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:03 smithi167 ceph-mon[32774]: from='client.45518 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T06:47:04.005 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:04.033 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:04.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:04.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:04.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:04.159 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:04.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:04.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:04.260 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:04.288 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:04.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:04.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:04.390 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:04.418 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:04.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:04.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:04.520 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:04.544 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:04.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:04.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:04.611 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:47:04.611 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:47:04.611 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:47:04.611 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:47:04.611 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:47:04.628 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:47:04.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:04.649 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:47:04.650 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:04 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:47:04.650 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:04 smithi028 ceph-mon[37984]: from='client.45518 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T06:47:04.650 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:04 smithi028 ceph-mon[37984]: osdmap e339: 8 total, 8 up, 8 in 2024-02-05T06:47:04.650 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:04 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:47:04.650 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:04 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:47:04.650 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:04 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/287539062' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:47:04.650 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:04 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:47:04.650 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:04 smithi028 ceph-mon[29442]: from='client.45518 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T06:47:04.650 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:04 smithi028 ceph-mon[29442]: osdmap e339: 8 total, 8 up, 8 in 2024-02-05T06:47:04.651 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:04 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:47:04.651 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:04 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:47:04.651 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:04 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/287539062' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:47:04.669 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:04.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:04.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:04.671 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:47:04.673 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:47:04.673 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:47:04.673 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:47:04.673 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:47:04.673 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:47:04.673 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:47:04.673 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:47:04.673 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:47:04.675 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:47:04.771 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:04.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:04.797 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:04.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:04.897 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:04.920 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:04.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:04.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:04.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:04 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:47:04.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:04 smithi167 ceph-mon[32774]: from='client.45518 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T06:47:04.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:04 smithi167 ceph-mon[32774]: osdmap e339: 8 total, 8 up, 8 in 2024-02-05T06:47:04.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:04 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:47:04.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:04 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:47:04.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:04 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/287539062' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:47:05.022 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:05.045 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:05.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:05.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:05.147 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:05.175 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:05.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:05.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:05.267 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd reweight 2 1 2024-02-05T06:47:05.277 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:05.304 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:05.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:05.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:05.406 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:05.433 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:05.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:05.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:05.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:05.558 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:05.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:05.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:05.659 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:05.682 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:05.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:05.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:05.784 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:05.795 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:05 smithi028 ceph-mon[37984]: pgmap v647: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 450 op/s 2024-02-05T06:47:05.795 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:05 smithi028 ceph-mon[37984]: osdmap e340: 8 total, 8 up, 8 in 2024-02-05T06:47:05.795 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:05 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:47:05.795 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:05 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:47:05.795 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:05 smithi028 ceph-mon[29442]: pgmap v647: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 450 op/s 2024-02-05T06:47:05.795 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:05 smithi028 ceph-mon[29442]: osdmap e340: 8 total, 8 up, 8 in 2024-02-05T06:47:05.795 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:05 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:47:05.795 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:05 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:47:05.809 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:05.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:05.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:05.886 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:05 smithi167 ceph-mon[32774]: pgmap v647: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 450 op/s 2024-02-05T06:47:05.886 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:05 smithi167 ceph-mon[32774]: osdmap e340: 8 total, 8 up, 8 in 2024-02-05T06:47:05.886 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:05 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:47:05.886 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:05 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:47:05.910 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:05.938 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:05.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:05.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:06.040 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:06.064 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:06.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:06.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:06.166 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:06.190 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:06.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:06.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:06.292 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:06.320 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:06.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:06.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:06.422 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:06.446 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:47:06.449 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:06.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:06.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:06.551 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:06.574 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:06.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:06.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:06.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:06 smithi028 ceph-mon[29442]: osdmap e341: 8 total, 8 up, 8 in 2024-02-05T06:47:06.661 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:06 smithi028 ceph-mon[37984]: osdmap e341: 8 total, 8 up, 8 in 2024-02-05T06:47:06.676 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:06.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:06.706 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:06.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:06.807 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:06.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:06.833 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:06.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:06.934 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:06.962 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:06.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:06.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:06.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:06 smithi167 ceph-mon[32774]: osdmap e341: 8 total, 8 up, 8 in 2024-02-05T06:47:07.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:07.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:07.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:07.190 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:07.213 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:07.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:07.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:07.315 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:07.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:07.343 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:07.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:07.444 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:07.471 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:07.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:07.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:07.573 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:07.600 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:07.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:07.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:07.702 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:07.728 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:07.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:07.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:07.830 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:07.856 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:07.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:07.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:07.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:07 smithi028 ceph-mon[29442]: pgmap v650: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 450 op/s 2024-02-05T06:47:07.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:07 smithi028 ceph-mon[37984]: pgmap v650: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 450 op/s 2024-02-05T06:47:07.958 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:07.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:07 smithi167 ceph-mon[32774]: pgmap v650: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 450 op/s 2024-02-05T06:47:07.986 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:07.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:07.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:08.088 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:08.115 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:08.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:08.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:08.217 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:08.245 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:08.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:08.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:08.346 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:08.374 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:08.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:08.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:08.426 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:47:08.475 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:08.499 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:08.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:08.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:08.600 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:08.629 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:08.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:08.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:08.730 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:08.757 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:08.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:08.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:08.858 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:08.885 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:08.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:08.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:08.986 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:09.012 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:09.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:09.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:09.072 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:47:09.072 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-02-05T06:47:09.072 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 0 2024-02-05T06:47:09.072 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:47:09.114 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:09.140 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:09.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:09.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:09.242 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:09.268 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:09.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:09.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:09.370 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:09.396 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:09.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:09.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:09.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:09.524 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:09.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:09.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:09.625 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:09.631 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:09 smithi028 ceph-mon[37984]: pgmap v651: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 87 op/s 2024-02-05T06:47:09.632 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:09 smithi028 ceph-mon[29442]: pgmap v651: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 87 op/s 2024-02-05T06:47:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:09.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:09.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:09.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-05T06:47:09.753 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:09.776 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:09.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:09.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:09.878 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:09.894 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:47:09.909 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:09.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:09.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:09.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:09 smithi167 ceph-mon[32774]: pgmap v651: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 87 op/s 2024-02-05T06:47:10.011 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:10.039 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:10.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:10.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:10.141 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:10.169 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:10.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:10.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:10.271 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:10.297 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:10.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:10.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:10.399 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:10.425 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:10.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:10.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:10.527 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:10.549 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'unique_pool_2' removed 2024-02-05T06:47:10.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:10.561 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:10.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:10.601 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:10 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/218559430' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T06:47:10.602 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:10 smithi028 ceph-mon[29442]: from='client.45530 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T06:47:10.602 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:10 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/218559430' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T06:47:10.602 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:10 smithi028 ceph-mon[37984]: from='client.45530 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T06:47:10.662 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:10.686 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:10.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:10.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:10.788 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:10.812 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:10.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:10.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:10.914 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:10.937 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:10.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:10.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:10.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:10 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/218559430' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T06:47:10.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:10 smithi167 ceph-mon[32774]: from='client.45530 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T06:47:11.040 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:11.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:11.069 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:11.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:11.170 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:11.193 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:11.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:11.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:11.295 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:11.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:11.319 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:11.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:11.420 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:11.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:11.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:11.546 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:11.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:11.570 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:11.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:11.598 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-05T06:47:11.599 INFO:tasks.radosbench:Beginning radosbench... 2024-02-05T06:47:11.599 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-02-05T06:47:11.599 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd pool create unique_pool_3 16 2024-02-05T06:47:11.671 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:11.695 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:11.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:11.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:11.797 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:11.827 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:11.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:11.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:11.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:11 smithi028 ceph-mon[29442]: pgmap v652: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 737 B/s rd, 0 op/s 2024-02-05T06:47:11.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:11 smithi028 ceph-mon[29442]: from='client.45530 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T06:47:11.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:11 smithi028 ceph-mon[29442]: osdmap e342: 8 total, 8 up, 8 in 2024-02-05T06:47:11.832 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:11 smithi028 ceph-mon[37984]: pgmap v652: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 737 B/s rd, 0 op/s 2024-02-05T06:47:11.832 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:11 smithi028 ceph-mon[37984]: from='client.45530 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T06:47:11.832 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:11 smithi028 ceph-mon[37984]: osdmap e342: 8 total, 8 up, 8 in 2024-02-05T06:47:11.930 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:11.954 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:11.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:11.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:11.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:11 smithi167 ceph-mon[32774]: pgmap v652: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 737 B/s rd, 0 op/s 2024-02-05T06:47:11.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:11 smithi167 ceph-mon[32774]: from='client.45530 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T06:47:11.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:11 smithi167 ceph-mon[32774]: osdmap e342: 8 total, 8 up, 8 in 2024-02-05T06:47:12.056 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:12.080 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:12.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:12.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:12.182 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:12.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:12.208 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:12.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:12.248 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:47:12.265 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:47:12.308 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:12.331 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:12.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:12.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:12.433 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:12.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:12.459 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:12.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:12.560 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:12.584 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:12.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:12.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:12.686 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:12.710 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:12.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:12.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:12.733 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:47:12 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:47:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:47:12.811 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:12.835 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:12.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:12.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:12.937 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:12.960 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:12.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:12.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:13.053 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:12 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:47:13.053 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:12 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:47:13.053 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:12 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:47:13.054 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:12 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:47:13.061 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:13.087 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:13.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:13.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:13.188 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:13.213 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:13.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:13.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:13.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:12 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:47:13.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:12 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:47:13.234 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:47:13 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:47:13.315 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:13.344 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:13.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:13.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:13.446 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:13.469 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:13.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:13.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:13.571 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:13.594 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:13.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:13.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:13.696 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:13.722 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:13.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:13.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:13.824 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:13.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:13.850 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:13.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:13.951 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:13.976 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:13.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:13.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:14.078 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:14.102 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:14.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:14.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:14.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:13 smithi028 ceph-mon[37984]: pgmap v654: 142 pgs: 142 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:47:14.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:13 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3841889914' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:47:14.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:13 smithi028 ceph-mon[37984]: from='client.45536 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:47:14.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:13 smithi028 ceph-mon[29442]: pgmap v654: 142 pgs: 142 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:47:14.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:13 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3841889914' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:47:14.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:13 smithi028 ceph-mon[29442]: from='client.45536 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:47:14.204 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:14.227 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:14.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:14.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:14.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:13 smithi167 ceph-mon[32774]: pgmap v654: 142 pgs: 142 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:47:14.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:13 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3841889914' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:47:14.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:13 smithi167 ceph-mon[32774]: from='client.45536 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:47:14.330 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:14.353 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:14.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:14.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:14.354 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-02-05T06:47:14.455 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:14.479 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:14.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:14.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:14.581 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:14.609 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:14.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:14.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:14.711 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:14.739 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:14.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:14.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:14.841 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:14.868 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:14.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:14.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:14.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:14 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:47:14.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:14 smithi028 ceph-mon[37984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:47:14.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:14 smithi028 ceph-mon[37984]: from='client.45536 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:47:14.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:14 smithi028 ceph-mon[37984]: osdmap e343: 8 total, 8 up, 8 in 2024-02-05T06:47:14.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:14 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:47:14.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:14 smithi028 ceph-mon[29442]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:47:14.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:14 smithi028 ceph-mon[29442]: from='client.45536 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:47:14.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:14 smithi028 ceph-mon[29442]: osdmap e343: 8 total, 8 up, 8 in 2024-02-05T06:47:14.969 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:14.994 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:14.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:14.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:15.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:15.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:15.121 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:15.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:15.222 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:15.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:14 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:47:15.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:14 smithi167 ceph-mon[32774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:47:15.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:14 smithi167 ceph-mon[32774]: from='client.45536 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:47:15.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:14 smithi167 ceph-mon[32774]: osdmap e343: 8 total, 8 up, 8 in 2024-02-05T06:47:15.250 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:15.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:15.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:15.352 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:15.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:15.379 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:15.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:15.480 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:15.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:15.505 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:15.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:15.606 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:15.629 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:15.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:15.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:15.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:15.753 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:15.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:15.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:15.855 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:15.882 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:15.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:15.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:15.984 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:16.007 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:16.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:16.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:16.109 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:16.133 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:16.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:16.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:16.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:15 smithi028 ceph-mon[29442]: pgmap v656: 142 pgs: 142 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:47:16.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:15 smithi028 ceph-mon[37984]: pgmap v656: 142 pgs: 142 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:47:16.234 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:16.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:15 smithi167 ceph-mon[32774]: pgmap v656: 142 pgs: 142 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:47:16.262 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:16.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:16.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:16.364 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:16.391 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:16.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:16.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:16.493 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:16.516 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:16.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:16.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:16.618 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:16.646 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:16.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:16.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:16.748 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:16.774 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:16.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:16.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:16.876 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:16.903 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:16.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:16.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:17.006 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:17.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:17.033 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:17.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:17.064 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:47:17.117 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:47:17.133 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:17.136 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:47:17.155 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:47:17.157 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:17.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:17.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:17.175 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:47:17.196 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:47:17.217 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:47:17.237 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:47:17.257 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:47:17.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:17.259 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:47:17.259 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:47:17.259 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:47:17.259 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:47:17.259 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:47:17.259 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:47:17.260 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:47:17.260 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:47:17.283 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:17.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:17.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:17.385 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:17.408 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:17.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:17.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:17.510 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:17.539 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:17.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:17.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:17.640 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:17.664 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:17.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:17.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:17.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:17.790 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:17.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:17.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:17.891 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:17.915 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:17.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:17.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:17.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:17 smithi028 ceph-mon[29442]: pgmap v657: 142 pgs: 142 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-05T06:47:17.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:17 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3097291222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:47:17.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:17 smithi028 ceph-mon[37984]: pgmap v657: 142 pgs: 142 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-05T06:47:17.942 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:17 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3097291222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:47:17.988 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T06:47:18.018 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:18.057 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:18.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:18.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:18.159 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:18.183 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:18.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:18.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:18.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:17 smithi167 ceph-mon[32774]: pgmap v657: 142 pgs: 142 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-05T06:47:18.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:17 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3097291222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:47:18.285 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:18.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:18.314 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:18.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:18.415 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:18.442 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:18.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:18.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:18.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:18.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:18.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:18.670 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:18.698 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:18.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:18.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:18.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:18.826 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:18.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:18.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:18.928 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:18.955 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:18.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:18.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:19.057 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:19.085 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:19.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:19.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:19.102 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:47:19.102 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:47:19.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:19.210 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:19.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:19.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:19.311 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:19.339 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:19.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:19.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:19.441 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:19.467 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:19.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:19.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:19.568 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:19.595 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:19.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:19.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:19.696 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:19.721 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:19.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:19.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:19.823 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:19.849 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:19.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:19.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:19.944 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:47:19.951 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:20.011 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:20.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:20.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:20.021 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:19 smithi028 ceph-mon[29442]: pgmap v658: 142 pgs: 142 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:47:20.021 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:19 smithi028 ceph-mon[37984]: pgmap v658: 142 pgs: 142 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:47:20.114 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:20.138 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:20.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:20.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:20.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:19 smithi167 ceph-mon[32774]: pgmap v658: 142 pgs: 142 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:47:20.240 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:20.264 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:20.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:20.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:20.365 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:20.393 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:20.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:20.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:20.495 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:20.522 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:20.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:20.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:20.624 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:20.650 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:20.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:20.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:20.752 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:20.779 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:20.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:20.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:20.881 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:20.905 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:20.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:20.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:21.006 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:21.029 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:21.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:21.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:21.131 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:21.154 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:21.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:21.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:21.169 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.2 to 1 (10000) 2024-02-05T06:47:21.234 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:21 smithi028 ceph-mon[37984]: pgmap v659: 142 pgs: 142 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T06:47:21.235 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:21 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2076127242' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T06:47:21.235 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:21 smithi028 ceph-mon[29442]: pgmap v659: 142 pgs: 142 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T06:47:21.235 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:21 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2076127242' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T06:47:21.256 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:21.260 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-05T06:47:21.285 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:21.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:21.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:21.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:21.409 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:21.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:21.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:21.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:21 smithi167 ceph-mon[32774]: pgmap v659: 142 pgs: 142 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T06:47:21.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:21 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2076127242' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T06:47:21.511 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:21.534 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:21.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:21.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:21.635 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:21.663 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:21.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:21.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:21.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:21.789 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:21.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:21.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:21.891 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:21.898 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd reweight 3 1 2024-02-05T06:47:21.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:21.915 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:21.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:22.017 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:22.056 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:22.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:22.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:22.159 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:22.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:22.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:22.285 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:22.313 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:22.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:22.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:22.343 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:47:22.343 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:47:22.392 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:47:22.415 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:22.442 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:22.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:22.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:22.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:22 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2076127242' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T06:47:22.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:22 smithi028 ceph-mon[29442]: osdmap e344: 8 total, 8 up, 8 in 2024-02-05T06:47:22.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:22 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2076127242' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T06:47:22.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:22 smithi028 ceph-mon[37984]: osdmap e344: 8 total, 8 up, 8 in 2024-02-05T06:47:22.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:22 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2076127242' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T06:47:22.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:22 smithi167 ceph-mon[32774]: osdmap e344: 8 total, 8 up, 8 in 2024-02-05T06:47:22.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:22.567 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:22.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:22.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:22.669 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:22.697 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:22.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:22.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:22.737 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:47:22 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:47:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:47:22.799 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:22.822 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:22.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:22.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:22.924 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:22.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:22.952 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:22.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:23.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:23.077 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:23.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:23.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:23.175 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'unique_pool_3' created 2024-02-05T06:47:23.179 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:23.205 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:23.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:23.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:23.307 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:23.331 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:23.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:23.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:23.432 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:23.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:23 smithi028 ceph-mon[29442]: pgmap v661: 142 pgs: 142 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:47:23.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:23 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2218853467' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-02-05T06:47:23.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:23 smithi028 ceph-mon[37984]: pgmap v661: 142 pgs: 142 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:47:23.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:23 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2218853467' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-02-05T06:47:23.456 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:23.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:23.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:23.484 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:47:23 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:47:23.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:23 smithi167 ceph-mon[32774]: pgmap v661: 142 pgs: 142 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:47:23.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:23 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2218853467' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-02-05T06:47:23.558 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:23.585 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:23.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:23.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:23.686 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:23.713 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:23.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:23.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:23.815 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:23.842 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:23.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:23.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:23.944 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:23.968 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:23.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:23.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:24.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-02-05T06:47:24.071 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:24.099 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:24.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:24.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:24.200 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:24.226 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:24.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:24.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:24.328 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:24.354 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:24.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:24.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:24.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:24 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:47:24.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:24 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2218853467' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-02-05T06:47:24.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:24 smithi028 ceph-mon[29442]: osdmap e345: 8 total, 8 up, 8 in 2024-02-05T06:47:24.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:24 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:47:24.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:24 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2218853467' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-02-05T06:47:24.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:24 smithi028 ceph-mon[37984]: osdmap e345: 8 total, 8 up, 8 in 2024-02-05T06:47:24.456 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:24.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:24.481 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:24.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:24.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:24 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:47:24.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:24 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2218853467' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-02-05T06:47:24.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:24 smithi167 ceph-mon[32774]: osdmap e345: 8 total, 8 up, 8 in 2024-02-05T06:47:24.582 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:24.607 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:24.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:24.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:24.709 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:24.737 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:24.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:24.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:24.839 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:24.867 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:24.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:24.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:24.968 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:24.996 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:24.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:24.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:25.098 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:25.125 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:25.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:25.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:25.227 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:25.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:25.255 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:25.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:25.356 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:25.383 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:25.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:25.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:25.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:25 smithi028 ceph-mon[29442]: osdmap e346: 8 total, 8 up, 8 in 2024-02-05T06:47:25.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:25 smithi028 ceph-mon[29442]: pgmap v664: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:47:25.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:25 smithi028 ceph-mon[37984]: osdmap e346: 8 total, 8 up, 8 in 2024-02-05T06:47:25.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:25 smithi028 ceph-mon[37984]: pgmap v664: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:47:25.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:25 smithi167 ceph-mon[32774]: osdmap e346: 8 total, 8 up, 8 in 2024-02-05T06:47:25.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:25 smithi167 ceph-mon[32774]: pgmap v664: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:47:25.485 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:25.513 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:25.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:25.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:25.615 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:25.643 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:25.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:25.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:25.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:25.769 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:25.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:25.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:25.870 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:25.894 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:25.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:25.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:25.996 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:26.019 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:26.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:26.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:26.120 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:26.149 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:26.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:26.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:26.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:26.274 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:26.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:26.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:26.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:26.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:26.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:26.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:26 smithi028 ceph-mon[37984]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T06:47:26.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:26 smithi028 ceph-mon[37984]: osdmap e347: 8 total, 8 up, 8 in 2024-02-05T06:47:26.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:26 smithi028 ceph-mon[29442]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T06:47:26.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:26 smithi028 ceph-mon[29442]: osdmap e347: 8 total, 8 up, 8 in 2024-02-05T06:47:26.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:26 smithi167 ceph-mon[32774]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T06:47:26.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:26 smithi167 ceph-mon[32774]: osdmap e347: 8 total, 8 up, 8 in 2024-02-05T06:47:26.501 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:26.529 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:26.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:26.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:26.630 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:26.657 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:26.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:26.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:26.759 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:26.784 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:26.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:26.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:26.886 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:26.909 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:26.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:26.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:27.011 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:27.035 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:27.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:27.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:27.137 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:27.162 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:27.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:27.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:27.183 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:47:27.237 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:47:27.257 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:47:27.263 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:27.277 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:47:27.287 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:27.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:27.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:27.297 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:47:27.316 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:47:27.336 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:47:27.357 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:47:27.376 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:47:27.377 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:47:27.378 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:47:27.378 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:47:27.378 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:47:27.378 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:47:27.378 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:47:27.378 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:47:27.378 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:47:27.389 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:27.413 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:27.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:27.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:27.433 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:27 smithi028 ceph-mon[37984]: pgmap v666: 158 pgs: 9 unknown, 149 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 968 B/s rd, 0 op/s 2024-02-05T06:47:27.433 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:27 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:47:27.433 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:27 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:47:27.433 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:27 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:47:27.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:27 smithi028 ceph-mon[29442]: pgmap v666: 158 pgs: 9 unknown, 149 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 968 B/s rd, 0 op/s 2024-02-05T06:47:27.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:27 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:47:27.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:27 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:47:27.434 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:27 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:47:27.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:27 smithi167 ceph-mon[32774]: pgmap v666: 158 pgs: 9 unknown, 149 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 968 B/s rd, 0 op/s 2024-02-05T06:47:27.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:27 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:47:27.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:27 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:47:27.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:27 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:47:27.515 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:27.539 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:27.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:27.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:27.641 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:27.665 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:27.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:27.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:27.767 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:27.790 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:27.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:27.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:27.892 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:27.915 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:27.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:27.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:28.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:28.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:28.044 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:28.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:28.145 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:28.168 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:28.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:28.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:28.270 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:28.293 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:28.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:28.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:28.395 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:28.418 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:28.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:28.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:28.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:28 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/917411583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:47:28.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:28 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/917411583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:47:28.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:28 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/917411583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:47:28.519 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:28.542 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:28.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:28.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:28.644 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:28.667 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:28.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:28.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:28.769 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:28.792 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:28.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:28.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:28.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:28.918 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:28.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:28.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:29.020 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:29.043 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:29.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:29.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:29.145 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:29.169 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:29.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:29.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:29.203 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.3 to 1 (10000) 2024-02-05T06:47:29.271 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:29.296 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:29.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:29.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:29.392 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:29 smithi028 ceph-mon[29442]: pgmap v667: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:47:29.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:29 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/4063028128' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T06:47:29.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:29 smithi028 ceph-mon[29442]: from='client.45566 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T06:47:29.393 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:29 smithi028 ceph-mon[37984]: pgmap v667: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:47:29.393 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:29 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/4063028128' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T06:47:29.393 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:29 smithi028 ceph-mon[37984]: from='client.45566 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T06:47:29.397 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:29.421 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:29.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:29.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:29.433 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:29 smithi167 ceph-mon[32774]: pgmap v667: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:47:29.433 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:29 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/4063028128' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T06:47:29.433 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:29 smithi167 ceph-mon[32774]: from='client.45566 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T06:47:29.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:29.546 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:29.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:29.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:29.629 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:47:29.648 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:29.672 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:29.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:29.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:29.773 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:29.797 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:29.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:29.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:29.899 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:29.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:29.924 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:29.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:30.025 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:30.054 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:30.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:30.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:30.156 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:30.175 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:47:30.176 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd reweight 6 1 2024-02-05T06:47:30.186 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:30.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:30.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:30.211 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-02-05T06:47:30.289 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:30.314 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:30.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:30.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:30.402 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:30 smithi028 ceph-mon[29442]: from='client.45566 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-05T06:47:30.403 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:30 smithi028 ceph-mon[29442]: osdmap e348: 8 total, 8 up, 8 in 2024-02-05T06:47:30.403 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:30 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3400218280' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:47:30.403 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:30 smithi028 ceph-mon[37984]: from='client.45566 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-05T06:47:30.403 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:30 smithi028 ceph-mon[37984]: osdmap e348: 8 total, 8 up, 8 in 2024-02-05T06:47:30.403 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:30 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3400218280' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:47:30.415 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:30.439 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:30.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:30.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:30.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:30 smithi167 ceph-mon[32774]: from='client.45566 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-05T06:47:30.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:30 smithi167 ceph-mon[32774]: osdmap e348: 8 total, 8 up, 8 in 2024-02-05T06:47:30.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:30 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3400218280' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:47:30.540 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:30.564 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:30.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:30.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:30.665 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:30.690 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:30.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:30.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:30.792 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:30.815 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:30.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:30.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:30.917 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:30.941 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:30.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:30.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:31.042 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:31.066 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:31.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:31.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:31.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:31.191 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:31.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:31.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:31.293 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:31.316 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:31.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:31.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:31.418 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:31.443 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:31.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:31.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:31.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:31 smithi028 ceph-mon[29442]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:47:31.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:31 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3400218280' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:47:31.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:31 smithi028 ceph-mon[29442]: osdmap e349: 8 total, 8 up, 8 in 2024-02-05T06:47:31.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:31 smithi028 ceph-mon[29442]: 4.11 scrub starts 2024-02-05T06:47:31.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:31 smithi028 ceph-mon[29442]: 4.11 scrub ok 2024-02-05T06:47:31.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:31 smithi028 ceph-mon[29442]: pgmap v670: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:47:31.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:31 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:47:31.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:31 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:47:31.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:31 smithi028 ceph-mon[37984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:47:31.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:31 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3400218280' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:47:31.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:31 smithi028 ceph-mon[37984]: osdmap e349: 8 total, 8 up, 8 in 2024-02-05T06:47:31.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:31 smithi028 ceph-mon[37984]: 4.11 scrub starts 2024-02-05T06:47:31.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:31 smithi028 ceph-mon[37984]: 4.11 scrub ok 2024-02-05T06:47:31.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:31 smithi028 ceph-mon[37984]: pgmap v670: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:47:31.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:31 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:47:31.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:31 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:47:31.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:31 smithi167 ceph-mon[32774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:47:31.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:31 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3400218280' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:47:31.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:31 smithi167 ceph-mon[32774]: osdmap e349: 8 total, 8 up, 8 in 2024-02-05T06:47:31.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:31 smithi167 ceph-mon[32774]: 4.11 scrub starts 2024-02-05T06:47:31.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:31 smithi167 ceph-mon[32774]: 4.11 scrub ok 2024-02-05T06:47:31.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:31 smithi167 ceph-mon[32774]: pgmap v670: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:47:31.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:31 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:47:31.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:31 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:47:31.545 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:31.568 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:31.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:31.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:31.670 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:31.693 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:31.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:31.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:31.794 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:31.818 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:31.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:31.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:31.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:31.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:31.944 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:31.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:32.045 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:32.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:32.070 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:32.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:32.170 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:32.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:32.195 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:32.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:32.296 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:32.319 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:32.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:32.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:32.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:32.445 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:32.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:32.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:32.547 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:32.572 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:32.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:32.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:32.674 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:32.698 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:32.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:32.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:32.795 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:47:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:47:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:47:32.801 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:32.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:32.827 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:32.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:32.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:32.952 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:32.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:32.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:33.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:33.077 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:33.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:33.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:33.178 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:33.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:33.208 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:33.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:33.309 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:33.337 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:33.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:33.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:33.439 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:33.466 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:33.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:33.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:33.484 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:47:33 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:47:33.569 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:33.595 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:33.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:33.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:33.697 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:33.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:33 smithi028 ceph-mon[29442]: pgmap v671: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-05T06:47:33.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:33 smithi028 ceph-mon[37984]: pgmap v671: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-05T06:47:33.723 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:33.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:33.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:33.825 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:33.851 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:33.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:33.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:33.944 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:47:33.944 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-02-05T06:47:33.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-02-05T06:47:33.944 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-02-05T06:47:33.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.944 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.14847.0:156932 7.3 7:c8f4fb15:::benchmark_data_smithi028_77509_object41812:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-02-05T06:47:33.944 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:37:58.523516+0000", 2024-02-05T06:47:33.944 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 575.04408447699996, 2024-02-05T06:47:33.944 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.035096529, 2024-02-05T06:47:33.945 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:47:33.945 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-02-05T06:47:33.945 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:47:33.945 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.14847", 2024-02-05T06:47:33.945 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/776803457", 2024-02-05T06:47:33.945 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 156932 2024-02-05T06:47:33.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.945 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:47:33.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.945 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:47:33.945 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:58.523516+0000", 2024-02-05T06:47:33.945 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:33.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.945 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:47:33.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:58.523516+0000", 2024-02-05T06:47:33.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:33.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:47:33.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:58.523516+0000", 2024-02-05T06:47:33.946 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9999999999999997e-07 2024-02-05T06:47:33.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.946 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.946 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:47:33.946 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:58.523519+0000", 2024-02-05T06:47:33.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0290000000000001e-06 2024-02-05T06:47:33.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:47:33.947 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:58.523525+0000", 2024-02-05T06:47:33.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6520000000000003e-06 2024-02-05T06:47:33.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:47:33.947 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:58.523531+0000", 2024-02-05T06:47:33.947 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.789e-06 2024-02-05T06:47:33.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.947 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.947 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:33.948 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:58.523598+0000", 2024-02-05T06:47:33.948 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7563000000000003e-05 2024-02-05T06:47:33.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.948 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:47:33.948 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:58.523721+0000", 2024-02-05T06:47:33.948 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012293499999999999 2024-02-05T06:47:33.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.948 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.948 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,4,6", 2024-02-05T06:47:33.948 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:58.523874+0000", 2024-02-05T06:47:33.948 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000152363 2024-02-05T06:47:33.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:47:33.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:58.526532+0000", 2024-02-05T06:47:33.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026586629999999999 2024-02-05T06:47:33.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:33.949 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:58.531323+0000", 2024-02-05T06:47:33.949 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047907929999999998 2024-02-05T06:47:33.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.949 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.949 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:33.950 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:58.542070+0000", 2024-02-05T06:47:33.950 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010746479 2024-02-05T06:47:33.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.950 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:47:33.950 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:58.542081+0000", 2024-02-05T06:47:33.950 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1423000000000001e-05 2024-02-05T06:47:33.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.950 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.950 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:33.950 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:59.558526+0000", 2024-02-05T06:47:33.950 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.016444965 2024-02-05T06:47:33.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.951 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:47:33.951 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:37:59.558612+0000", 2024-02-05T06:47:33.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6174999999999994e-05 2024-02-05T06:47:33.951 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:33.951 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:47:33.951 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:33.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.951 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.951 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:30015 8.0 8:01b50849:::benchmark_data_smithi028_105698_object30014:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194)", 2024-02-05T06:47:33.951 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:07.557315+0000", 2024-02-05T06:47:33.951 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 446.010284931, 2024-02-05T06:47:33.951 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.122448779, 2024-02-05T06:47:33.952 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:47:33.952 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-02-05T06:47:33.952 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:47:33.952 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:47:33.952 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:33.952 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 30015 2024-02-05T06:47:33.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.952 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:47:33.952 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.952 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:47:33.952 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.557315+0000", 2024-02-05T06:47:33.952 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:33.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.953 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:47:33.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.557315+0000", 2024-02-05T06:47:33.953 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:33.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.953 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:47:33.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.557317+0000", 2024-02-05T06:47:33.953 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.542e-06 2024-02-05T06:47:33.953 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.953 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.953 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:47:33.953 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.557421+0000", 2024-02-05T06:47:33.954 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000104709 2024-02-05T06:47:33.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.954 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:47:33.954 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.557433+0000", 2024-02-05T06:47:33.954 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2034e-05 2024-02-05T06:47:33.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.954 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:47:33.954 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.557442+0000", 2024-02-05T06:47:33.954 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3289999999999994e-06 2024-02-05T06:47:33.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.955 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:33.955 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.557643+0000", 2024-02-05T06:47:33.955 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000201032 2024-02-05T06:47:33.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.955 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.955 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:47:33.955 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.557781+0000", 2024-02-05T06:47:33.955 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013791199999999999 2024-02-05T06:47:33.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.955 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.955 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,5", 2024-02-05T06:47:33.955 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.558065+0000", 2024-02-05T06:47:33.955 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028466100000000002 2024-02-05T06:47:33.955 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.956 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:33.956 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.573870+0000", 2024-02-05T06:47:33.956 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.015804874999999999 2024-02-05T06:47:33.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.956 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:47:33.956 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.573886+0000", 2024-02-05T06:47:33.956 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5447999999999998e-05 2024-02-05T06:47:33.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.956 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:33.956 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:08.574109+0000", 2024-02-05T06:47:33.956 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.00022286 2024-02-05T06:47:33.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.957 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:47:33.957 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:08.574127+0000", 2024-02-05T06:47:33.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7995999999999998e-05 2024-02-05T06:47:33.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.957 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:33.957 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.679672+0000", 2024-02-05T06:47:33.957 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1055452020000001 2024-02-05T06:47:33.957 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.957 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.957 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:47:33.958 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.679764+0000", 2024-02-05T06:47:33.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2179e-05 2024-02-05T06:47:33.958 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:33.958 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:47:33.958 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:33.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.958 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.958 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:30012 8.8 8:150b8b2f:::benchmark_data_smithi028_105698_object30011:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e195)", 2024-02-05T06:47:33.958 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:07.564290+0000", 2024-02-05T06:47:33.958 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 446.003310113, 2024-02-05T06:47:33.958 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1817648730000001, 2024-02-05T06:47:33.958 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:47:33.958 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-02-05T06:47:33.958 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:47:33.959 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:47:33.959 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:33.959 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 30012 2024-02-05T06:47:33.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.959 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:47:33.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.959 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:47:33.959 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.564290+0000", 2024-02-05T06:47:33.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:33.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.959 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.959 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:47:33.959 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.564290+0000", 2024-02-05T06:47:33.959 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:33.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.960 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:47:33.960 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.564292+0000", 2024-02-05T06:47:33.960 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.99e-06 2024-02-05T06:47:33.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.960 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:47:33.960 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.564319+0000", 2024-02-05T06:47:33.960 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6927000000000002e-05 2024-02-05T06:47:33.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.960 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.960 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:47:33.961 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.566985+0000", 2024-02-05T06:47:33.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002665906 2024-02-05T06:47:33.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.961 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:47:33.961 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.567002+0000", 2024-02-05T06:47:33.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7164999999999999e-05 2024-02-05T06:47:33.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.961 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:33.961 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.573203+0000", 2024-02-05T06:47:33.961 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062012320000000001 2024-02-05T06:47:33.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.961 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.961 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:47:33.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.573214+0000", 2024-02-05T06:47:33.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1083999999999999e-05 2024-02-05T06:47:33.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:33.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:08.574161+0000", 2024-02-05T06:47:33.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.000946299 2024-02-05T06:47:33.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.962 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.962 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:47:33.962 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:08.574170+0000", 2024-02-05T06:47:33.962 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9700000000000005e-06 2024-02-05T06:47:33.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:33.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.745283+0000", 2024-02-05T06:47:33.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1711130729999999 2024-02-05T06:47:33.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.963 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:47:33.963 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.746055+0000", 2024-02-05T06:47:33.963 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000772227 2024-02-05T06:47:33.963 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:33.963 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:47:33.963 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:33.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.963 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.964 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:44143 8.8 8:1a9500f5:::benchmark_data_smithi028_105698_object44142:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-02-05T06:47:33.964 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:26.042979+0000", 2024-02-05T06:47:33.964 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 427.52462155500001, 2024-02-05T06:47:33.964 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3135507470000001, 2024-02-05T06:47:33.964 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:47:33.964 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-02-05T06:47:33.964 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:47:33.964 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:47:33.964 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:33.964 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 44143 2024-02-05T06:47:33.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.964 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:47:33.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.965 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:47:33.965 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.042979+0000", 2024-02-05T06:47:33.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:33.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.965 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:47:33.965 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.042979+0000", 2024-02-05T06:47:33.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:33.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.965 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:47:33.965 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.042981+0000", 2024-02-05T06:47:33.965 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0430000000000002e-06 2024-02-05T06:47:33.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.966 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:47:33.966 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.043007+0000", 2024-02-05T06:47:33.966 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6285999999999999e-05 2024-02-05T06:47:33.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.966 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:47:33.966 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.043019+0000", 2024-02-05T06:47:33.966 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2398000000000001e-05 2024-02-05T06:47:33.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.966 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:47:33.967 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.043028+0000", 2024-02-05T06:47:33.967 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1339999999999993e-06 2024-02-05T06:47:33.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.967 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.967 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:33.967 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.043115+0000", 2024-02-05T06:47:33.967 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6144999999999996e-05 2024-02-05T06:47:33.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.967 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.967 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:47:33.967 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.043372+0000", 2024-02-05T06:47:33.967 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00025751700000000002 2024-02-05T06:47:33.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.968 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,2", 2024-02-05T06:47:33.968 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.043572+0000", 2024-02-05T06:47:33.968 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019952199999999999 2024-02-05T06:47:33.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.968 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:33.968 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.086854+0000", 2024-02-05T06:47:33.968 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.043281950999999999 2024-02-05T06:47:33.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.968 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:47:33.969 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.086871+0000", 2024-02-05T06:47:33.969 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6979000000000001e-05 2024-02-05T06:47:33.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.969 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:33.969 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.276656+0000", 2024-02-05T06:47:33.969 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1897851070000001 2024-02-05T06:47:33.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.969 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for degraded object", 2024-02-05T06:47:33.969 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.277207+0000", 2024-02-05T06:47:33.969 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00055184299999999999 2024-02-05T06:47:33.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.970 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:33.970 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.356387+0000", 2024-02-05T06:47:33.970 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.079179709000000001 2024-02-05T06:47:33.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.970 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:47:33.970 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.356529+0000", 2024-02-05T06:47:33.970 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000142113 2024-02-05T06:47:33.970 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:33.970 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:47:33.970 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:33.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.970 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:44159 8.8 8:1960bcdc:::benchmark_data_smithi028_105698_object44158:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-02-05T06:47:33.971 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:26.104840+0000", 2024-02-05T06:47:33.971 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 427.462760038, 2024-02-05T06:47:33.971 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2646835759999999, 2024-02-05T06:47:33.971 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:47:33.971 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:47:33.971 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:47:33.971 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:47:33.971 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:33.971 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 44159 2024-02-05T06:47:33.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.971 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:47:33.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.971 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:47:33.971 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.104840+0000", 2024-02-05T06:47:33.971 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:33.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.972 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:47:33.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.104840+0000", 2024-02-05T06:47:33.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:33.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.972 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:47:33.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.104842+0000", 2024-02-05T06:47:33.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8679999999999999e-06 2024-02-05T06:47:33.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.972 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:47:33.972 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.104865+0000", 2024-02-05T06:47:33.972 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3221e-05 2024-02-05T06:47:33.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.973 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:47:33.973 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.104879+0000", 2024-02-05T06:47:33.973 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3658999999999999e-05 2024-02-05T06:47:33.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.973 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:47:33.973 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.104887+0000", 2024-02-05T06:47:33.973 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2390000000000002e-06 2024-02-05T06:47:33.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.973 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:33.973 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.104959+0000", 2024-02-05T06:47:33.973 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2438e-05 2024-02-05T06:47:33.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.974 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:47:33.974 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.104972+0000", 2024-02-05T06:47:33.974 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2081e-05 2024-02-05T06:47:33.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:33.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.359125+0000", 2024-02-05T06:47:33.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2541535160000001 2024-02-05T06:47:33.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.975 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:47:33.975 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.359284+0000", 2024-02-05T06:47:33.975 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015925100000000001 2024-02-05T06:47:33.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,7", 2024-02-05T06:47:33.976 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.359439+0000", 2024-02-05T06:47:33.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015446000000000001 2024-02-05T06:47:33.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:33.976 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.363542+0000", 2024-02-05T06:47:33.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041027190000000003 2024-02-05T06:47:33.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.976 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:33.976 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.363641+0000", 2024-02-05T06:47:33.976 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9438999999999994e-05 2024-02-05T06:47:33.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.977 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:47:33.977 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.365375+0000", 2024-02-05T06:47:33.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001734225 2024-02-05T06:47:33.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.977 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:47:33.977 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.365453+0000", 2024-02-05T06:47:33.977 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8304000000000001e-05 2024-02-05T06:47:33.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.978 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:47:33.978 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.369524+0000", 2024-02-05T06:47:33.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040701560000000001 2024-02-05T06:47:33.978 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:33.978 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:47:33.978 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:33.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.978 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:84974 8.8 8:1f44453d:::benchmark_data_smithi028_105698_object84973:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-02-05T06:47:33.978 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:41:20.030183+0000", 2024-02-05T06:47:33.978 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 373.53741691200003, 2024-02-05T06:47:33.978 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.035111442, 2024-02-05T06:47:33.978 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:47:33.979 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-02-05T06:47:33.979 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:47:33.979 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:47:33.979 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:33.979 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 84974 2024-02-05T06:47:33.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.979 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:47:33.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.979 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:47:33.979 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.030183+0000", 2024-02-05T06:47:33.979 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:33.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.979 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.979 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:47:33.979 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.030183+0000", 2024-02-05T06:47:33.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:33.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:47:33.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.030185+0000", 2024-02-05T06:47:33.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.311e-06 2024-02-05T06:47:33.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.980 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:47:33.980 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.030209+0000", 2024-02-05T06:47:33.980 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3499e-05 2024-02-05T06:47:33.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.980 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.981 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:47:33.981 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.030223+0000", 2024-02-05T06:47:33.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3672000000000001e-05 2024-02-05T06:47:33.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.981 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:47:33.981 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.030295+0000", 2024-02-05T06:47:33.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2236000000000004e-05 2024-02-05T06:47:33.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.981 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.981 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:33.981 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.030358+0000", 2024-02-05T06:47:33.981 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2645999999999996e-05 2024-02-05T06:47:33.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.982 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:47:33.982 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.030493+0000", 2024-02-05T06:47:33.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013525000000000001 2024-02-05T06:47:33.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.982 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,7", 2024-02-05T06:47:33.982 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.030648+0000", 2024-02-05T06:47:33.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015509599999999999 2024-02-05T06:47:33.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.982 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.982 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:33.982 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.033405+0000", 2024-02-05T06:47:33.982 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027567239999999999 2024-02-05T06:47:33.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:47:33.983 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.033420+0000", 2024-02-05T06:47:33.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5658000000000002e-05 2024-02-05T06:47:33.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:33.983 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.054220+0000", 2024-02-05T06:47:33.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020799957000000001 2024-02-05T06:47:33.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.983 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.983 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:47:33.983 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.054244+0000", 2024-02-05T06:47:33.983 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3609e-05 2024-02-05T06:47:33.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.984 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:33.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:21.065229+0000", 2024-02-05T06:47:33.984 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.010984857 2024-02-05T06:47:33.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.984 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.984 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:47:33.984 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:21.065295+0000", 2024-02-05T06:47:33.984 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5926999999999994e-05 2024-02-05T06:47:33.984 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:33.984 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:47:33.984 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:33.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.985 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:84981 8.8 8:1663915c:::benchmark_data_smithi028_105698_object84980:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-02-05T06:47:33.985 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:41:20.037896+0000", 2024-02-05T06:47:33.985 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 373.52970373900001, 2024-02-05T06:47:33.985 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0520791410000001, 2024-02-05T06:47:33.985 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:47:33.985 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-02-05T06:47:33.985 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:47:33.985 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:47:33.985 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:33.985 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 84981 2024-02-05T06:47:33.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.985 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:47:33.985 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.985 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:47:33.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.037896+0000", 2024-02-05T06:47:33.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:33.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:47:33.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.037896+0000", 2024-02-05T06:47:33.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:33.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.986 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:47:33.986 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.037898+0000", 2024-02-05T06:47:33.986 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9290000000000001e-06 2024-02-05T06:47:33.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:47:33.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.037919+0000", 2024-02-05T06:47:33.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0800000000000001e-05 2024-02-05T06:47:33.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:47:33.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.037930+0000", 2024-02-05T06:47:33.987 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1106e-05 2024-02-05T06:47:33.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.987 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.987 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:47:33.987 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.037938+0000", 2024-02-05T06:47:33.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6480000000000003e-06 2024-02-05T06:47:33.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.988 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:33.988 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.039576+0000", 2024-02-05T06:47:33.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0016384870000000001 2024-02-05T06:47:33.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.988 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.988 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:47:33.988 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.039745+0000", 2024-02-05T06:47:33.988 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016862100000000001 2024-02-05T06:47:33.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.989 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,7", 2024-02-05T06:47:33.989 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.039944+0000", 2024-02-05T06:47:33.989 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000199366 2024-02-05T06:47:33.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.989 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:33.989 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.055997+0000", 2024-02-05T06:47:33.989 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016053082 2024-02-05T06:47:33.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.989 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.989 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:47:33.989 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.056010+0000", 2024-02-05T06:47:33.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2578999999999999e-05 2024-02-05T06:47:33.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.990 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:33.990 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:21.078029+0000", 2024-02-05T06:47:33.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.022018935 2024-02-05T06:47:33.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.990 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for degraded object", 2024-02-05T06:47:33.990 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:21.078392+0000", 2024-02-05T06:47:33.990 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00036348700000000001 2024-02-05T06:47:33.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:33.991 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:21.089887+0000", 2024-02-05T06:47:33.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011494924 2024-02-05T06:47:33.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:47:33.991 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:21.089975+0000", 2024-02-05T06:47:33.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8177000000000006e-05 2024-02-05T06:47:33.991 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:33.991 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:47:33.991 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:33.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.992 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:84984 8.8 8:1a81b593:::benchmark_data_smithi028_105698_object84983:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-02-05T06:47:33.992 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:41:20.045463+0000", 2024-02-05T06:47:33.992 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 373.52213720600002, 2024-02-05T06:47:33.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0446798479999999, 2024-02-05T06:47:33.992 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:47:33.992 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:47:33.992 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:47:33.992 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:47:33.992 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:33.992 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 84984 2024-02-05T06:47:33.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.993 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:47:33.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:47:33.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.045463+0000", 2024-02-05T06:47:33.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:33.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:47:33.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.045463+0000", 2024-02-05T06:47:33.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:33.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:47:33.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.045464+0000", 2024-02-05T06:47:33.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.575e-06 2024-02-05T06:47:33.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:47:33.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.045490+0000", 2024-02-05T06:47:33.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.546e-05 2024-02-05T06:47:33.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:47:33.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.045500+0000", 2024-02-05T06:47:33.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0393e-05 2024-02-05T06:47:33.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:47:33.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.045508+0000", 2024-02-05T06:47:33.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2509999999999997e-06 2024-02-05T06:47:33.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:33.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.056063+0000", 2024-02-05T06:47:33.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010555619 2024-02-05T06:47:33.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:47:33.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.056072+0000", 2024-02-05T06:47:33.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6489999999999997e-06 2024-02-05T06:47:33.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.996 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:33.996 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:21.078452+0000", 2024-02-05T06:47:33.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0223801699999999 2024-02-05T06:47:33.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.996 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:47:33.996 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:21.078545+0000", 2024-02-05T06:47:33.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2649e-05 2024-02-05T06:47:33.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.996 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,7", 2024-02-05T06:47:33.997 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:21.078703+0000", 2024-02-05T06:47:33.997 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000158066 2024-02-05T06:47:33.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.997 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:33.997 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:21.084427+0000", 2024-02-05T06:47:33.997 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0057246570000000002 2024-02-05T06:47:33.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.997 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:47:33.997 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:21.084441+0000", 2024-02-05T06:47:33.997 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.364e-05 2024-02-05T06:47:33.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:33.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:21.090010+0000", 2024-02-05T06:47:33.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055693000000000001 2024-02-05T06:47:33.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:47:33.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:21.090058+0000", 2024-02-05T06:47:33.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7342000000000002e-05 2024-02-05T06:47:33.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:47:33.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:21.090143+0000", 2024-02-05T06:47:33.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5076999999999999e-05 2024-02-05T06:47:33.998 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:33.999 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:47:33.999 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:33.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:33.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:33.999 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:84994 8.8 8:1778b1de:::benchmark_data_smithi028_105698_object84993:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e235)", 2024-02-05T06:47:33.999 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:41:20.059303+0000", 2024-02-05T06:47:33.999 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 373.50829737499998, 2024-02-05T06:47:33.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.031525265, 2024-02-05T06:47:33.999 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:47:33.999 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:47:33.999 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:47:33.999 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:47:33.999 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:34.000 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 84994 2024-02-05T06:47:34.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.000 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:47:34.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:47:34.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.059303+0000", 2024-02-05T06:47:34.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:34.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:47:34.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.059303+0000", 2024-02-05T06:47:34.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:34.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:47:34.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.059304+0000", 2024-02-05T06:47:34.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7090000000000001e-06 2024-02-05T06:47:34.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:47:34.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.059328+0000", 2024-02-05T06:47:34.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3587999999999998e-05 2024-02-05T06:47:34.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:47:34.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.059337+0000", 2024-02-05T06:47:34.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2730000000000008e-06 2024-02-05T06:47:34.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:47:34.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.059344+0000", 2024-02-05T06:47:34.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0369999999999997e-06 2024-02-05T06:47:34.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:34.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.059402+0000", 2024-02-05T06:47:34.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7679000000000001e-05 2024-02-05T06:47:34.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:47:34.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:20.059415+0000", 2024-02-05T06:47:34.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3023999999999999e-05 2024-02-05T06:47:34.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:34.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:21.081801+0000", 2024-02-05T06:47:34.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.022385984 2024-02-05T06:47:34.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:47:34.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:21.081951+0000", 2024-02-05T06:47:34.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014963500000000001 2024-02-05T06:47:34.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,7", 2024-02-05T06:47:34.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:21.082138+0000", 2024-02-05T06:47:34.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000187725 2024-02-05T06:47:34.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:34.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:21.084476+0000", 2024-02-05T06:47:34.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023379249999999998 2024-02-05T06:47:34.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.005 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:47:34.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:21.089528+0000", 2024-02-05T06:47:34.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0050520850000000004 2024-02-05T06:47:34.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.005 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:34.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:21.090320+0000", 2024-02-05T06:47:34.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00079138500000000003 2024-02-05T06:47:34.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.005 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:47:34.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:21.090371+0000", 2024-02-05T06:47:34.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1060000000000002e-05 2024-02-05T06:47:34.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:47:34.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:21.090828+0000", 2024-02-05T06:47:34.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00045715600000000001 2024-02-05T06:47:34.006 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:34.006 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:47:34.006 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:34.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.006 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:106718 8.0 8:0e30f0e6:::benchmark_data_smithi028_105698_object17443:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e246)", 2024-02-05T06:47:34.006 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:41:42.373779+0000", 2024-02-05T06:47:34.006 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 351.19382133400001, 2024-02-05T06:47:34.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1886464299999999, 2024-02-05T06:47:34.006 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:47:34.007 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-02-05T06:47:34.007 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:47:34.007 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:47:34.007 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:34.007 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 106718 2024-02-05T06:47:34.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.007 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:47:34.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:47:34.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.373779+0000", 2024-02-05T06:47:34.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:34.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:47:34.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.373779+0000", 2024-02-05T06:47:34.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:34.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:47:34.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.373780+0000", 2024-02-05T06:47:34.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.195e-06 2024-02-05T06:47:34.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:47:34.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.373784+0000", 2024-02-05T06:47:34.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2549999999999999e-06 2024-02-05T06:47:34.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:47:34.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.373794+0000", 2024-02-05T06:47:34.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8460000000000003e-06 2024-02-05T06:47:34.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:47:34.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.373801+0000", 2024-02-05T06:47:34.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3189999999999997e-06 2024-02-05T06:47:34.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:34.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.378370+0000", 2024-02-05T06:47:34.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0045686859999999998 2024-02-05T06:47:34.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:47:34.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.378487+0000", 2024-02-05T06:47:34.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000117185 2024-02-05T06:47:34.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,4", 2024-02-05T06:47:34.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.378611+0000", 2024-02-05T06:47:34.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012412799999999999 2024-02-05T06:47:34.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:47:34.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.380921+0000", 2024-02-05T06:47:34.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023095009999999998 2024-02-05T06:47:34.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:34.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.384834+0000", 2024-02-05T06:47:34.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039135239999999998 2024-02-05T06:47:34.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:34.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.422857+0000", 2024-02-05T06:47:34.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.038022367000000001 2024-02-05T06:47:34.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:47:34.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.422870+0000", 2024-02-05T06:47:34.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3122000000000001e-05 2024-02-05T06:47:34.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:34.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.400541+0000", 2024-02-05T06:47:34.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.97767121999999995 2024-02-05T06:47:34.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:47:34.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.400555+0000", 2024-02-05T06:47:34.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4134999999999999e-05 2024-02-05T06:47:34.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.013 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:34.013 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.562332+0000", 2024-02-05T06:47:34.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.161776806 2024-02-05T06:47:34.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.013 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:47:34.013 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.562425+0000", 2024-02-05T06:47:34.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3140999999999995e-05 2024-02-05T06:47:34.013 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:34.013 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:47:34.013 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:34.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.014 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:106737 8.0 8:083ba507:::benchmark_data_smithi028_105698_object17462:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-02-05T06:47:34.014 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:41:42.394852+0000", 2024-02-05T06:47:34.014 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 351.17274829500002, 2024-02-05T06:47:34.014 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1766352879999999, 2024-02-05T06:47:34.014 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:47:34.014 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:47:34.014 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:47:34.014 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:47:34.014 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:34.014 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 106737 2024-02-05T06:47:34.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.014 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:47:34.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.015 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:47:34.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.394852+0000", 2024-02-05T06:47:34.015 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:34.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.015 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:47:34.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.394852+0000", 2024-02-05T06:47:34.015 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:34.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.015 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:47:34.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.394854+0000", 2024-02-05T06:47:34.016 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8649999999999999e-06 2024-02-05T06:47:34.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.016 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:47:34.016 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.394859+0000", 2024-02-05T06:47:34.016 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6620000000000002e-06 2024-02-05T06:47:34.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.016 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:47:34.016 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.394869+0000", 2024-02-05T06:47:34.016 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5230000000000001e-06 2024-02-05T06:47:34.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:47:34.017 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.394876+0000", 2024-02-05T06:47:34.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9809999999999997e-06 2024-02-05T06:47:34.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:34.017 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.394929+0000", 2024-02-05T06:47:34.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3106999999999998e-05 2024-02-05T06:47:34.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op must wait for map", 2024-02-05T06:47:34.017 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.394935+0000", 2024-02-05T06:47:34.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9109999999999998e-06 2024-02-05T06:47:34.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.018 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:34.018 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.426299+0000", 2024-02-05T06:47:34.018 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.031364106000000003 2024-02-05T06:47:34.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.018 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:47:34.018 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.426308+0000", 2024-02-05T06:47:34.018 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8650000000000007e-06 2024-02-05T06:47:34.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.018 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:34.018 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.400585+0000", 2024-02-05T06:47:34.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.97427749699999999 2024-02-05T06:47:34.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.019 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:47:34.019 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.400594+0000", 2024-02-05T06:47:34.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8769999999999999e-06 2024-02-05T06:47:34.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.019 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:34.019 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.562766+0000", 2024-02-05T06:47:34.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1621715530000001 2024-02-05T06:47:34.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.019 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:47:34.020 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.562890+0000", 2024-02-05T06:47:34.020 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012439800000000001 2024-02-05T06:47:34.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.020 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,4,5", 2024-02-05T06:47:34.020 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.563009+0000", 2024-02-05T06:47:34.020 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000119175 2024-02-05T06:47:34.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.020 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:34.020 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.571128+0000", 2024-02-05T06:47:34.020 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0081191149999999997 2024-02-05T06:47:34.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.021 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:34.021 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.571169+0000", 2024-02-05T06:47:34.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0327000000000002e-05 2024-02-05T06:47:34.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.021 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:34.021 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.571207+0000", 2024-02-05T06:47:34.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7944999999999997e-05 2024-02-05T06:47:34.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.021 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:47:34.021 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.571338+0000", 2024-02-05T06:47:34.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013164700000000001 2024-02-05T06:47:34.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.022 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:47:34.022 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.571386+0000", 2024-02-05T06:47:34.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7414999999999999e-05 2024-02-05T06:47:34.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.022 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:47:34.022 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.571487+0000", 2024-02-05T06:47:34.022 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101319 2024-02-05T06:47:34.022 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:34.022 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:47:34.022 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:34.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.022 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:106741 8.0 8:0fa5dafb:::benchmark_data_smithi028_105698_object17466:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-02-05T06:47:34.023 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:41:42.398246+0000", 2024-02-05T06:47:34.023 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 351.16935362200002, 2024-02-05T06:47:34.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.176239072, 2024-02-05T06:47:34.023 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:47:34.023 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:47:34.023 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:47:34.023 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:47:34.023 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:34.023 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 106741 2024-02-05T06:47:34.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.023 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:47:34.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:47:34.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.398246+0000", 2024-02-05T06:47:34.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:34.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.024 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:47:34.024 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.398246+0000", 2024-02-05T06:47:34.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:34.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.024 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:47:34.024 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.398247+0000", 2024-02-05T06:47:34.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.029e-06 2024-02-05T06:47:34.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.024 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:47:34.025 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.398251+0000", 2024-02-05T06:47:34.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.348e-06 2024-02-05T06:47:34.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.025 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:47:34.025 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.398268+0000", 2024-02-05T06:47:34.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7586000000000001e-05 2024-02-05T06:47:34.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.025 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:47:34.025 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.398275+0000", 2024-02-05T06:47:34.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8950000000000001e-06 2024-02-05T06:47:34.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.026 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:34.026 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.398821+0000", 2024-02-05T06:47:34.026 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00054611600000000003 2024-02-05T06:47:34.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.026 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting_for_map not empty", 2024-02-05T06:47:34.026 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.398828+0000", 2024-02-05T06:47:34.026 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4790000000000001e-06 2024-02-05T06:47:34.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.026 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:34.026 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.426331+0000", 2024-02-05T06:47:34.026 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.027503217 2024-02-05T06:47:34.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.027 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.027 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:47:34.027 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.426338+0000", 2024-02-05T06:47:34.027 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1709999999999996e-06 2024-02-05T06:47:34.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.027 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.027 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:34.027 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.400620+0000", 2024-02-05T06:47:34.027 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.97428194000000001 2024-02-05T06:47:34.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.027 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.027 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:47:34.027 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.400630+0000", 2024-02-05T06:47:34.027 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6609999999999995e-06 2024-02-05T06:47:34.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.028 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.028 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:34.028 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.565387+0000", 2024-02-05T06:47:34.028 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.164757542 2024-02-05T06:47:34.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.028 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.028 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:47:34.028 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.565500+0000", 2024-02-05T06:47:34.028 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011222600000000001 2024-02-05T06:47:34.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.028 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.028 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,4,5", 2024-02-05T06:47:34.028 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.565629+0000", 2024-02-05T06:47:34.029 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000129429 2024-02-05T06:47:34.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.029 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:34.029 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.571245+0000", 2024-02-05T06:47:34.029 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0056158190000000002 2024-02-05T06:47:34.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.029 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:34.029 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.571290+0000", 2024-02-05T06:47:34.029 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5287999999999997e-05 2024-02-05T06:47:34.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.030 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:47:34.030 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.571496+0000", 2024-02-05T06:47:34.030 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00020537600000000001 2024-02-05T06:47:34.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.030 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:34.030 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.574317+0000", 2024-02-05T06:47:34.030 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028216890000000001 2024-02-05T06:47:34.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.030 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:47:34.030 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.574380+0000", 2024-02-05T06:47:34.030 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2892999999999996e-05 2024-02-05T06:47:34.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.031 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:47:34.031 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.574486+0000", 2024-02-05T06:47:34.031 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000105368 2024-02-05T06:47:34.031 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:34.031 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:47:34.031 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:34.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.031 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:106757 8.0 8:06f5095c:::benchmark_data_smithi028_105698_object17482:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247)", 2024-02-05T06:47:34.031 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:41:42.445127+0000", 2024-02-05T06:47:34.031 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 351.12247296999999, 2024-02-05T06:47:34.031 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.129532465, 2024-02-05T06:47:34.031 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:47:34.032 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:47:34.032 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:47:34.032 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:47:34.032 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:34.032 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 106757 2024-02-05T06:47:34.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.032 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:47:34.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.032 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:47:34.032 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.445127+0000", 2024-02-05T06:47:34.032 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:34.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.032 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:47:34.033 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.445127+0000", 2024-02-05T06:47:34.033 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:34.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.033 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:47:34.033 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.445129+0000", 2024-02-05T06:47:34.033 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.761e-06 2024-02-05T06:47:34.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.033 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:47:34.033 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.445134+0000", 2024-02-05T06:47:34.033 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0830000000000003e-06 2024-02-05T06:47:34.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.034 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:47:34.034 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.445145+0000", 2024-02-05T06:47:34.034 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1276e-05 2024-02-05T06:47:34.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.034 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:47:34.034 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.445152+0000", 2024-02-05T06:47:34.034 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3180000000000003e-06 2024-02-05T06:47:34.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.034 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:34.034 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.445220+0000", 2024-02-05T06:47:34.035 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8936999999999994e-05 2024-02-05T06:47:34.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.035 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.035 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:47:34.035 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:42.445232+0000", 2024-02-05T06:47:34.035 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1545999999999999e-05 2024-02-05T06:47:34.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.035 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.035 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:34.035 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.400656+0000", 2024-02-05T06:47:34.035 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.95542370200000004 2024-02-05T06:47:34.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.035 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.035 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:47:34.036 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.400664+0000", 2024-02-05T06:47:34.036 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4300000000000006e-06 2024-02-05T06:47:34.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.036 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:34.036 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.568136+0000", 2024-02-05T06:47:34.036 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1674722399999999 2024-02-05T06:47:34.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.036 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:47:34.036 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.568246+0000", 2024-02-05T06:47:34.036 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000109368 2024-02-05T06:47:34.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.037 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,4,5", 2024-02-05T06:47:34.037 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.568365+0000", 2024-02-05T06:47:34.037 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000119164 2024-02-05T06:47:34.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.037 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:47:34.037 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.571501+0000", 2024-02-05T06:47:34.037 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031363839999999999 2024-02-05T06:47:34.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.037 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:34.037 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.573847+0000", 2024-02-05T06:47:34.038 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002345654 2024-02-05T06:47:34.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.038 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:34.038 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.573898+0000", 2024-02-05T06:47:34.038 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1044000000000001e-05 2024-02-05T06:47:34.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.038 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:34.038 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.574525+0000", 2024-02-05T06:47:34.038 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00062694200000000002 2024-02-05T06:47:34.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.038 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:47:34.039 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.574570+0000", 2024-02-05T06:47:34.039 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5077999999999998e-05 2024-02-05T06:47:34.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.039 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.039 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:47:34.039 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.574660+0000", 2024-02-05T06:47:34.039 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9537999999999996e-05 2024-02-05T06:47:34.039 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:34.039 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:47:34.039 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:34.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.039 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.039 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:134086 8.0 8:0a9dc049:::benchmark_data_smithi028_105698_object44811:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-02-05T06:47:34.039 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:42:16.081514+0000", 2024-02-05T06:47:34.040 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 317.48608623400003, 2024-02-05T06:47:34.040 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0412309829999999, 2024-02-05T06:47:34.040 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:47:34.040 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:47:34.040 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:47:34.040 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:47:34.040 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:34.040 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 134086 2024-02-05T06:47:34.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.040 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:47:34.040 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.040 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:47:34.040 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.081514+0000", 2024-02-05T06:47:34.040 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:34.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.041 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.041 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:47:34.041 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.081514+0000", 2024-02-05T06:47:34.041 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:34.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.041 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.041 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:47:34.041 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.081516+0000", 2024-02-05T06:47:34.041 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6980000000000001e-06 2024-02-05T06:47:34.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.041 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.041 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:47:34.041 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.081521+0000", 2024-02-05T06:47:34.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1739999999999999e-06 2024-02-05T06:47:34.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.042 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:47:34.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.081531+0000", 2024-02-05T06:47:34.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0456e-05 2024-02-05T06:47:34.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.042 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:47:34.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.081538+0000", 2024-02-05T06:47:34.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.199e-06 2024-02-05T06:47:34.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.042 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:34.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.085804+0000", 2024-02-05T06:47:34.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0042652979999999998 2024-02-05T06:47:34.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:47:34.043 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.085813+0000", 2024-02-05T06:47:34.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0089999999999996e-06 2024-02-05T06:47:34.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:34.043 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.119600+0000", 2024-02-05T06:47:34.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0337872400000001 2024-02-05T06:47:34.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:47:34.044 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.119744+0000", 2024-02-05T06:47:34.044 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014427 2024-02-05T06:47:34.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.044 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,5", 2024-02-05T06:47:34.044 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.119864+0000", 2024-02-05T06:47:34.044 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000120035 2024-02-05T06:47:34.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.044 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:34.044 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.122576+0000", 2024-02-05T06:47:34.044 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027114859999999999 2024-02-05T06:47:34.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.045 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:47:34.045 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.122585+0000", 2024-02-05T06:47:34.045 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9730000000000003e-06 2024-02-05T06:47:34.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.045 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:34.045 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.122636+0000", 2024-02-05T06:47:34.045 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1795999999999997e-05 2024-02-05T06:47:34.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.045 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:47:34.045 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.122695+0000", 2024-02-05T06:47:34.045 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8156999999999999e-05 2024-02-05T06:47:34.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.046 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:47:34.046 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.122745+0000", 2024-02-05T06:47:34.046 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0192000000000003e-05 2024-02-05T06:47:34.046 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:34.046 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:47:34.046 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:34.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.046 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.046 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:134094 8.0 8:02346960:::benchmark_data_smithi028_105698_object44819:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-02-05T06:47:34.046 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:42:16.091253+0000", 2024-02-05T06:47:34.046 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 317.476347481, 2024-02-05T06:47:34.046 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0422653049999999, 2024-02-05T06:47:34.046 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:47:34.046 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:47:34.046 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:47:34.046 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:47:34.047 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:34.047 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 134094 2024-02-05T06:47:34.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.047 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:47:34.047 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.047 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:47:34.047 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.091253+0000", 2024-02-05T06:47:34.047 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:34.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.047 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.047 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:47:34.047 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.091253+0000", 2024-02-05T06:47:34.047 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:34.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.048 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.048 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:47:34.048 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.091254+0000", 2024-02-05T06:47:34.048 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8390000000000001e-06 2024-02-05T06:47:34.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.048 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.048 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:47:34.048 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.091264+0000", 2024-02-05T06:47:34.048 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8099999999999992e-06 2024-02-05T06:47:34.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.048 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.048 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:47:34.048 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.091280+0000", 2024-02-05T06:47:34.049 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5325e-05 2024-02-05T06:47:34.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.049 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.049 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:47:34.049 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.091288+0000", 2024-02-05T06:47:34.049 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8750000000000006e-06 2024-02-05T06:47:34.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.049 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.049 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:34.049 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.091372+0000", 2024-02-05T06:47:34.049 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3906000000000006e-05 2024-02-05T06:47:34.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.049 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.049 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:47:34.049 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.091381+0000", 2024-02-05T06:47:34.050 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1169999999999993e-06 2024-02-05T06:47:34.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.050 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.050 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:34.050 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.122790+0000", 2024-02-05T06:47:34.050 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0314082849999999 2024-02-05T06:47:34.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.050 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.050 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:47:34.050 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.122885+0000", 2024-02-05T06:47:34.050 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4804999999999998e-05 2024-02-05T06:47:34.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.050 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.050 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,5", 2024-02-05T06:47:34.050 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.123008+0000", 2024-02-05T06:47:34.051 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000123194 2024-02-05T06:47:34.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.051 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.051 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:34.051 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.131253+0000", 2024-02-05T06:47:34.051 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0082448799999999996 2024-02-05T06:47:34.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.051 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.051 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:34.051 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.131308+0000", 2024-02-05T06:47:34.051 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5096000000000002e-05 2024-02-05T06:47:34.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.051 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.051 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:47:34.051 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.133423+0000", 2024-02-05T06:47:34.052 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021154429999999998 2024-02-05T06:47:34.052 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.052 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.052 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:47:34.052 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.133478+0000", 2024-02-05T06:47:34.052 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4491999999999999e-05 2024-02-05T06:47:34.052 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.052 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.052 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:47:34.052 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.133518+0000", 2024-02-05T06:47:34.052 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0238000000000003e-05 2024-02-05T06:47:34.052 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:34.053 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:47:34.053 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:34.053 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.053 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.053 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:134100 8.0 8:07c2f585:::benchmark_data_smithi028_105698_object44825:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-02-05T06:47:34.053 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:42:16.093262+0000", 2024-02-05T06:47:34.053 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 317.47433854299999, 2024-02-05T06:47:34.053 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.040349175, 2024-02-05T06:47:34.053 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:47:34.053 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:47:34.053 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:47:34.053 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:47:34.053 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:34.054 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 134100 2024-02-05T06:47:34.054 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.054 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:47:34.054 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.054 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:47:34.054 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.093262+0000", 2024-02-05T06:47:34.054 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:34.054 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.054 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.054 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:47:34.054 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.093262+0000", 2024-02-05T06:47:34.054 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:34.054 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.055 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.055 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:47:34.055 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.093263+0000", 2024-02-05T06:47:34.055 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6139999999999999e-06 2024-02-05T06:47:34.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.055 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.055 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:47:34.055 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.093268+0000", 2024-02-05T06:47:34.055 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7010000000000004e-06 2024-02-05T06:47:34.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.055 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.055 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:47:34.055 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.093279+0000", 2024-02-05T06:47:34.055 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1045e-05 2024-02-05T06:47:34.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.056 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.056 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:47:34.056 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.093287+0000", 2024-02-05T06:47:34.056 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2490000000000001e-06 2024-02-05T06:47:34.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.056 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.056 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:34.056 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.093524+0000", 2024-02-05T06:47:34.056 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00023680899999999999 2024-02-05T06:47:34.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.056 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.056 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:47:34.056 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.093530+0000", 2024-02-05T06:47:34.057 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9689999999999999e-06 2024-02-05T06:47:34.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.057 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:34.057 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.124994+0000", 2024-02-05T06:47:34.057 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0314644719999999 2024-02-05T06:47:34.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.057 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:47:34.057 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.125100+0000", 2024-02-05T06:47:34.057 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000105832 2024-02-05T06:47:34.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.058 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,5", 2024-02-05T06:47:34.058 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.125215+0000", 2024-02-05T06:47:34.058 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000114842 2024-02-05T06:47:34.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.058 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:34.058 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.131348+0000", 2024-02-05T06:47:34.058 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.006132991 2024-02-05T06:47:34.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.058 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:34.058 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.131388+0000", 2024-02-05T06:47:34.058 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9765999999999998e-05 2024-02-05T06:47:34.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.059 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:47:34.059 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.133531+0000", 2024-02-05T06:47:34.059 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021427579999999998 2024-02-05T06:47:34.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.059 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:47:34.059 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.133577+0000", 2024-02-05T06:47:34.059 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6606000000000001e-05 2024-02-05T06:47:34.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.059 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:47:34.060 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.133611+0000", 2024-02-05T06:47:34.060 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3521e-05 2024-02-05T06:47:34.060 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:34.060 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:47:34.060 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:34.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.060 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.060 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:134117 8.0 8:0db19117:::benchmark_data_smithi028_105698_object44842:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-02-05T06:47:34.060 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:42:16.108461+0000", 2024-02-05T06:47:34.060 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 317.45913880900002, 2024-02-05T06:47:34.060 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0273589009999999, 2024-02-05T06:47:34.060 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:47:34.060 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:47:34.061 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:47:34.061 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:47:34.061 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:34.061 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 134117 2024-02-05T06:47:34.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.061 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:47:34.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.061 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:47:34.061 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.108461+0000", 2024-02-05T06:47:34.061 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:34.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.061 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:47:34.062 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.108461+0000", 2024-02-05T06:47:34.062 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:34.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.062 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:47:34.062 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.108463+0000", 2024-02-05T06:47:34.062 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.911e-06 2024-02-05T06:47:34.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.062 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:47:34.062 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.108468+0000", 2024-02-05T06:47:34.062 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1880000000000001e-06 2024-02-05T06:47:34.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.063 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:47:34.063 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.108481+0000", 2024-02-05T06:47:34.063 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3001e-05 2024-02-05T06:47:34.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.063 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:47:34.063 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.108489+0000", 2024-02-05T06:47:34.063 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8650000000000001e-06 2024-02-05T06:47:34.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.063 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:34.063 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.108548+0000", 2024-02-05T06:47:34.064 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8328999999999997e-05 2024-02-05T06:47:34.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.064 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:47:34.064 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.108557+0000", 2024-02-05T06:47:34.064 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9260000000000006e-06 2024-02-05T06:47:34.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.064 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:34.064 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.127355+0000", 2024-02-05T06:47:34.064 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0187987549999999 2024-02-05T06:47:34.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.065 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:47:34.065 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.127554+0000", 2024-02-05T06:47:34.065 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019846800000000001 2024-02-05T06:47:34.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.065 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,5", 2024-02-05T06:47:34.065 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.127682+0000", 2024-02-05T06:47:34.065 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012822300000000001 2024-02-05T06:47:34.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.065 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:34.065 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.131429+0000", 2024-02-05T06:47:34.065 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003747167 2024-02-05T06:47:34.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.066 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:47:34.066 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.133620+0000", 2024-02-05T06:47:34.066 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021913119999999999 2024-02-05T06:47:34.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.066 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:34.066 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.135685+0000", 2024-02-05T06:47:34.066 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020647769999999998 2024-02-05T06:47:34.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.066 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:47:34.067 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.135780+0000", 2024-02-05T06:47:34.067 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4320999999999994e-05 2024-02-05T06:47:34.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.067 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:47:34.067 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.135820+0000", 2024-02-05T06:47:34.067 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0658000000000003e-05 2024-02-05T06:47:34.067 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:34.067 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:47:34.067 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:34.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.067 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:134138 8.0 8:00450567:::benchmark_data_smithi028_105698_object44863:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-02-05T06:47:34.068 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:42:16.126253+0000", 2024-02-05T06:47:34.068 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 317.44134725100002, 2024-02-05T06:47:34.068 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.017566019, 2024-02-05T06:47:34.068 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:47:34.068 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:47:34.068 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:47:34.068 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:47:34.068 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:34.068 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 134138 2024-02-05T06:47:34.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.068 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:47:34.068 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.068 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:47:34.069 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.126253+0000", 2024-02-05T06:47:34.069 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:34.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.069 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.069 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:47:34.069 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.126253+0000", 2024-02-05T06:47:34.069 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:34.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.069 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.069 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:47:34.069 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.126254+0000", 2024-02-05T06:47:34.069 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2240000000000001e-06 2024-02-05T06:47:34.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.069 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.070 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:47:34.070 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.126266+0000", 2024-02-05T06:47:34.070 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2231e-05 2024-02-05T06:47:34.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.070 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.070 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:47:34.070 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.126276+0000", 2024-02-05T06:47:34.070 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3540000000000006e-06 2024-02-05T06:47:34.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.070 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.070 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:47:34.070 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.126284+0000", 2024-02-05T06:47:34.070 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4069999999999993e-06 2024-02-05T06:47:34.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.071 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.071 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:34.071 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.126516+0000", 2024-02-05T06:47:34.071 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00023155699999999999 2024-02-05T06:47:34.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.071 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.071 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:47:34.071 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.126525+0000", 2024-02-05T06:47:34.071 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4769999999999992e-06 2024-02-05T06:47:34.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.071 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.071 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:34.071 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.129491+0000", 2024-02-05T06:47:34.071 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0029661409999999 2024-02-05T06:47:34.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.072 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.072 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:47:34.072 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.129587+0000", 2024-02-05T06:47:34.072 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5390000000000007e-05 2024-02-05T06:47:34.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.072 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.072 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,5", 2024-02-05T06:47:34.072 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.129654+0000", 2024-02-05T06:47:34.072 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7045999999999995e-05 2024-02-05T06:47:34.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.072 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.072 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:34.073 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.135880+0000", 2024-02-05T06:47:34.073 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062267549999999996 2024-02-05T06:47:34.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.073 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.073 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:34.073 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.135920+0000", 2024-02-05T06:47:34.073 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9684000000000002e-05 2024-02-05T06:47:34.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.073 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.073 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:47:34.073 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.143738+0000", 2024-02-05T06:47:34.073 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0078178840000000006 2024-02-05T06:47:34.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.073 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.074 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:47:34.074 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.143773+0000", 2024-02-05T06:47:34.074 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5398999999999998e-05 2024-02-05T06:47:34.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.074 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.074 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:47:34.074 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.143819+0000", 2024-02-05T06:47:34.074 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5469999999999997e-05 2024-02-05T06:47:34.074 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:34.074 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:47:34.074 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:34.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.074 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.074 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:134150 8.0 8:05f3f520:::benchmark_data_smithi028_105698_object44875:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e265)", 2024-02-05T06:47:34.074 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:42:16.136169+0000", 2024-02-05T06:47:34.075 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 317.43143139, 2024-02-05T06:47:34.075 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.010314267, 2024-02-05T06:47:34.075 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:47:34.075 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:47:34.075 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:47:34.075 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:47:34.075 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:34.075 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 134150 2024-02-05T06:47:34.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.075 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:47:34.075 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.075 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:47:34.075 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.136169+0000", 2024-02-05T06:47:34.075 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:34.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.076 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.076 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:47:34.076 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.136169+0000", 2024-02-05T06:47:34.076 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:34.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.076 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.076 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:47:34.076 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.136170+0000", 2024-02-05T06:47:34.076 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.406e-06 2024-02-05T06:47:34.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.076 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.076 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:47:34.077 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.136174+0000", 2024-02-05T06:47:34.077 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3800000000000004e-06 2024-02-05T06:47:34.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.077 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:47:34.077 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.136187+0000", 2024-02-05T06:47:34.077 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2294e-05 2024-02-05T06:47:34.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.077 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:47:34.077 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.136195+0000", 2024-02-05T06:47:34.077 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8909999999999995e-06 2024-02-05T06:47:34.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.078 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:34.078 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.136307+0000", 2024-02-05T06:47:34.078 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0001121 2024-02-05T06:47:34.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.078 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:47:34.078 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:16.136315+0000", 2024-02-05T06:47:34.078 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7819999999999994e-06 2024-02-05T06:47:34.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.078 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:34.078 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.133670+0000", 2024-02-05T06:47:34.078 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.99735495399999996 2024-02-05T06:47:34.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.079 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:47:34.079 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.133776+0000", 2024-02-05T06:47:34.079 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010623 2024-02-05T06:47:34.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.079 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,5", 2024-02-05T06:47:34.079 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.133841+0000", 2024-02-05T06:47:34.079 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5715000000000004e-05 2024-02-05T06:47:34.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.079 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:34.079 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.136037+0000", 2024-02-05T06:47:34.080 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021954520000000001 2024-02-05T06:47:34.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.080 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:34.080 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.136081+0000", 2024-02-05T06:47:34.080 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3761999999999999e-05 2024-02-05T06:47:34.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.080 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:47:34.080 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.146408+0000", 2024-02-05T06:47:34.080 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010327375 2024-02-05T06:47:34.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.080 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:47:34.081 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.146449+0000", 2024-02-05T06:47:34.081 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0564e-05 2024-02-05T06:47:34.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.081 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.081 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:47:34.081 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:42:17.146483+0000", 2024-02-05T06:47:34.081 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4362000000000001e-05 2024-02-05T06:47:34.081 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:34.081 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:47:34.081 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:34.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.081 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.081 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15390.0:254201 9.8 9:1e6b1338:::benchmark_data_smithi028_129096_object124401:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-02-05T06:47:34.082 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:46:54.936125+0000", 2024-02-05T06:47:34.082 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 38.631474838000003, 2024-02-05T06:47:34.082 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.14893442800000001, 2024-02-05T06:47:34.082 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:47:34.082 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:47:34.082 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:47:34.082 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15390", 2024-02-05T06:47:34.082 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:47:34.082 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 254201 2024-02-05T06:47:34.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.082 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:47:34.082 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.082 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:47:34.083 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:46:54.936125+0000", 2024-02-05T06:47:34.083 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:34.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.083 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.083 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:47:34.083 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:46:54.936125+0000", 2024-02-05T06:47:34.083 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:47:34.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.083 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.083 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:47:34.083 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:46:54.936126+0000", 2024-02-05T06:47:34.083 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2900000000000002e-07 2024-02-05T06:47:34.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.083 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.084 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:47:34.084 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:46:54.936129+0000", 2024-02-05T06:47:34.084 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3720000000000001e-06 2024-02-05T06:47:34.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.084 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:47:34.084 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:46:54.936136+0000", 2024-02-05T06:47:34.084 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9990000000000002e-06 2024-02-05T06:47:34.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.084 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:47:34.084 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:46:54.936143+0000", 2024-02-05T06:47:34.084 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6649999999999997e-06 2024-02-05T06:47:34.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.085 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:47:34.085 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:46:54.953011+0000", 2024-02-05T06:47:34.085 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.016867545000000001 2024-02-05T06:47:34.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.085 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:47:34.085 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:46:54.953123+0000", 2024-02-05T06:47:34.085 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011202 2024-02-05T06:47:34.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.085 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:47:34.086 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:46:54.953274+0000", 2024-02-05T06:47:34.086 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015105399999999999 2024-02-05T06:47:34.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.086 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:47:34.086 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:46:54.956972+0000", 2024-02-05T06:47:34.086 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0036987130000000002 2024-02-05T06:47:34.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.086 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:34.086 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:46:54.957009+0000", 2024-02-05T06:47:34.086 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6758000000000003e-05 2024-02-05T06:47:34.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.087 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:34.087 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:46:55.084954+0000", 2024-02-05T06:47:34.087 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.12794449099999999 2024-02-05T06:47:34.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.087 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:47:34.087 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:46:55.085007+0000", 2024-02-05T06:47:34.087 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2939000000000003e-05 2024-02-05T06:47:34.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:47:34.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:47:34.087 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:47:34.088 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:46:55.085060+0000", 2024-02-05T06:47:34.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3043e-05 2024-02-05T06:47:34.088 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:34.088 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:47:34.088 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:34.088 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:47:34.088 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:47:34.088 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:47:34.088 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:34.089 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:33 smithi167 ceph-mon[32774]: pgmap v671: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-05T06:47:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:34.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:34.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:34.214 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:34.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:34.239 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:34.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:34.301 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:47:34.302 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:47:34.340 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:34.367 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:34.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:34.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:34.469 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:34.496 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:34.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:34.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:34.527 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T06:47:34.598 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:34.624 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:34.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:34.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:34.725 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:34.751 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:34.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:34.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:34.853 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:34.874 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:34 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:47:34.874 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:34 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:47:34.876 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:34.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:34.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:34.903 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-05T06:47:34.930 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:34 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:47:34.979 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:35.004 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:35.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:35.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:35.106 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:35.130 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:35.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:35.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:35.154 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:47:35.232 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:35.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:35.258 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:35.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:35.319 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:47:35.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:35.382 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:35.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:35.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:35.484 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:35.508 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:35.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:35.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:35.598 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:35 smithi028 ceph-mon[37984]: 3.1e scrub starts 2024-02-05T06:47:35.598 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:35 smithi028 ceph-mon[37984]: 3.1e scrub ok 2024-02-05T06:47:35.598 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:35 smithi028 ceph-mon[37984]: pgmap v672: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:47:35.598 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:35 smithi028 ceph-mon[37984]: osdmap e350: 8 total, 8 up, 8 in 2024-02-05T06:47:35.598 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:35 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3626071451' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T06:47:35.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:35 smithi028 ceph-mon[29442]: 3.1e scrub starts 2024-02-05T06:47:35.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:35 smithi028 ceph-mon[29442]: 3.1e scrub ok 2024-02-05T06:47:35.598 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:35 smithi028 ceph-mon[29442]: pgmap v672: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:47:35.599 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:35 smithi028 ceph-mon[29442]: osdmap e350: 8 total, 8 up, 8 in 2024-02-05T06:47:35.599 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:35 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3626071451' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T06:47:35.609 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:35.632 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:35.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:35.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:35.734 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:35.758 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:35.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:35.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:35.859 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:35.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:35.884 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:35.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:35.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:35 smithi167 ceph-mon[32774]: 3.1e scrub starts 2024-02-05T06:47:35.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:35 smithi167 ceph-mon[32774]: 3.1e scrub ok 2024-02-05T06:47:35.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:35 smithi167 ceph-mon[32774]: pgmap v672: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:47:35.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:35 smithi167 ceph-mon[32774]: osdmap e350: 8 total, 8 up, 8 in 2024-02-05T06:47:35.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:35 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3626071451' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T06:47:35.985 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:36.010 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:36.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:36.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:36.112 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:36.135 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:36.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:36.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-02-05T06:47:36.237 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:36.261 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:36.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:36.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:36.363 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:36.387 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:36.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:36.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:36.489 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:36.512 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:36.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:36.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:36.614 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:36.633 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:36 smithi167 ceph-mon[32774]: 4.5 scrub starts 2024-02-05T06:47:36.633 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:36 smithi167 ceph-mon[32774]: 4.5 scrub ok 2024-02-05T06:47:36.633 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:36 smithi167 ceph-mon[32774]: 5.13 scrub starts 2024-02-05T06:47:36.633 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:36 smithi167 ceph-mon[32774]: 5.13 scrub ok 2024-02-05T06:47:36.633 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:36 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:47:36.633 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:36 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:47:36.633 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:36 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3626071451' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T06:47:36.633 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:36 smithi167 ceph-mon[32774]: osdmap e351: 8 total, 8 up, 8 in 2024-02-05T06:47:36.633 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:36 smithi167 ceph-mon[32774]: 6.b scrub starts 2024-02-05T06:47:36.637 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:36.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:36.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:36.739 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:36.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:36.766 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:36.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:36.867 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:36.892 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:36.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:36.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:36.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:36 smithi028 ceph-mon[37984]: 4.5 scrub starts 2024-02-05T06:47:36.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:36 smithi028 ceph-mon[37984]: 4.5 scrub ok 2024-02-05T06:47:36.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:36 smithi028 ceph-mon[37984]: 5.13 scrub starts 2024-02-05T06:47:36.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:36 smithi028 ceph-mon[37984]: 5.13 scrub ok 2024-02-05T06:47:36.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:36 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:47:36.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:36 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:47:36.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:36 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3626071451' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T06:47:36.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:36 smithi028 ceph-mon[37984]: osdmap e351: 8 total, 8 up, 8 in 2024-02-05T06:47:36.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:36 smithi028 ceph-mon[37984]: 6.b scrub starts 2024-02-05T06:47:36.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:36 smithi028 ceph-mon[29442]: 4.5 scrub starts 2024-02-05T06:47:36.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:36 smithi028 ceph-mon[29442]: 4.5 scrub ok 2024-02-05T06:47:36.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:36 smithi028 ceph-mon[29442]: 5.13 scrub starts 2024-02-05T06:47:36.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:36 smithi028 ceph-mon[29442]: 5.13 scrub ok 2024-02-05T06:47:36.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:36 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:47:36.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:36 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:47:36.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:36 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3626071451' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T06:47:36.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:36 smithi028 ceph-mon[29442]: osdmap e351: 8 total, 8 up, 8 in 2024-02-05T06:47:36.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:36 smithi028 ceph-mon[29442]: 6.b scrub starts 2024-02-05T06:47:36.994 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:37.001 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:47:37.001 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-02-05T06:47:37.001 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-02-05T06:47:37.001 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:47:37.020 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:37.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:37.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:37.122 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:37.145 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:37.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:37.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:37.247 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:37.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:37.272 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:37.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:37.373 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:37.397 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:37.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:37.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:37.499 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:37.523 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:37.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:37.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:37.620 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:37 smithi167 ceph-mon[32774]: pgmap v675: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 819 B/s rd, 0 op/s 2024-02-05T06:47:37.621 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:37 smithi167 ceph-mon[32774]: 6.b scrub ok 2024-02-05T06:47:37.621 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:37 smithi167 ceph-mon[32774]: 3.d scrub starts 2024-02-05T06:47:37.621 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:37 smithi167 ceph-mon[32774]: 3.d scrub ok 2024-02-05T06:47:37.621 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:37 smithi167 ceph-mon[32774]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T06:47:37.625 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:37.648 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-05T06:47:37.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:37.650 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:37.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:37.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:37 smithi028 ceph-mon[29442]: pgmap v675: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 819 B/s rd, 0 op/s 2024-02-05T06:47:37.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:37 smithi028 ceph-mon[29442]: 6.b scrub ok 2024-02-05T06:47:37.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:37 smithi028 ceph-mon[29442]: 3.d scrub starts 2024-02-05T06:47:37.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:37 smithi028 ceph-mon[29442]: 3.d scrub ok 2024-02-05T06:47:37.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:37 smithi028 ceph-mon[29442]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T06:47:37.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:37 smithi028 ceph-mon[37984]: pgmap v675: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 819 B/s rd, 0 op/s 2024-02-05T06:47:37.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:37 smithi028 ceph-mon[37984]: 6.b scrub ok 2024-02-05T06:47:37.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:37 smithi028 ceph-mon[37984]: 3.d scrub starts 2024-02-05T06:47:37.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:37 smithi028 ceph-mon[37984]: 3.d scrub ok 2024-02-05T06:47:37.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:37 smithi028 ceph-mon[37984]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T06:47:37.750 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:37.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:37.775 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:37.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:37.876 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:37.891 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:47:37.899 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:37.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:37.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:38.001 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:38.024 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:38.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:38.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:38.126 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:38.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:38.152 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:38.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:38.253 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:38.277 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:38.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:38.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:38.378 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:38.402 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:38.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:38.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:38.504 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:38.529 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:38.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:38.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:38.630 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:38.654 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:38.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:38.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:38.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:38 smithi167 ceph-mon[32774]: 4.1f scrub starts 2024-02-05T06:47:38.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:38 smithi167 ceph-mon[32774]: 4.1f scrub ok 2024-02-05T06:47:38.756 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:38.770 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:38 smithi028 ceph-mon[29442]: 4.1f scrub starts 2024-02-05T06:47:38.770 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:38 smithi028 ceph-mon[29442]: 4.1f scrub ok 2024-02-05T06:47:38.770 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:38 smithi028 ceph-mon[37984]: 4.1f scrub starts 2024-02-05T06:47:38.771 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:38 smithi028 ceph-mon[37984]: 4.1f scrub ok 2024-02-05T06:47:38.779 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:38.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:38.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:38.881 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:38.905 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:38.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:38.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:39.007 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:39.031 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:39.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:39.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:39.133 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:39.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:39.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:39.261 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:39.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:39.285 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:39.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:39.386 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:39.411 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:39.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:39.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:39.513 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:39.537 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:39.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:39.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:39.579 DEBUG:teuthology.orchestra.run.smithi028:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-02-05T06:47:39.582 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-05T06:47:39.582 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-05T06:47:39.614 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:47:39.614 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-02-05T06:47:39.614 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-02-05T06:47:39.614 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-02-05T06:47:39.614 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:47:39.615 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:39 smithi167 ceph-mon[32774]: pgmap v676: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:47:39.615 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:39 smithi167 ceph-mon[32774]: 6.a scrub starts 2024-02-05T06:47:39.615 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:39 smithi167 ceph-mon[32774]: 6.a scrub ok 2024-02-05T06:47:39.615 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:39 smithi167 ceph-mon[32774]: 6.15 scrub starts 2024-02-05T06:47:39.615 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:39 smithi167 ceph-mon[32774]: 6.15 scrub ok 2024-02-05T06:47:39.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:39 smithi028 ceph-mon[29442]: pgmap v676: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:47:39.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:39 smithi028 ceph-mon[29442]: 6.a scrub starts 2024-02-05T06:47:39.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:39 smithi028 ceph-mon[29442]: 6.a scrub ok 2024-02-05T06:47:39.623 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:39 smithi028 ceph-mon[29442]: 6.15 scrub starts 2024-02-05T06:47:39.624 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:39 smithi028 ceph-mon[29442]: 6.15 scrub ok 2024-02-05T06:47:39.624 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:39 smithi028 ceph-mon[37984]: pgmap v676: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:47:39.624 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:39 smithi028 ceph-mon[37984]: 6.a scrub starts 2024-02-05T06:47:39.624 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:39 smithi028 ceph-mon[37984]: 6.a scrub ok 2024-02-05T06:47:39.624 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:39 smithi028 ceph-mon[37984]: 6.15 scrub starts 2024-02-05T06:47:39.624 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:39 smithi028 ceph-mon[37984]: 6.15 scrub ok 2024-02-05T06:47:39.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:39.662 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:39.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:39.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:39.764 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:39.789 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:39.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:39.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:39.891 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:39.914 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:39.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:39.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:39.933 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:hints = 1 2024-02-05T06:47:39.933 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-05T06:47:39.933 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Object prefix: benchmark_data_smithi028_154709 2024-02-05T06:47:39.934 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T06:47:39.934 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 0 0 0 0 0 0 - 0 2024-02-05T06:47:40.016 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:40.051 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:40.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:40.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:40.153 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:40.187 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:40.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:40.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:40.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:40.335 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:40.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:40.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:40.344 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-05T06:47:40.437 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:40.478 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:40.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:40.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:40.580 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:40.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:40.623 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:40.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:40.701 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:47:40.724 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:40.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:40 smithi167 ceph-mon[32774]: 4.3 scrub starts 2024-02-05T06:47:40.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:40 smithi167 ceph-mon[32774]: 4.3 scrub ok 2024-02-05T06:47:40.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:40 smithi167 ceph-mon[32774]: 3.1c scrub starts 2024-02-05T06:47:40.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:40 smithi167 ceph-mon[32774]: 3.1c scrub ok 2024-02-05T06:47:40.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:40 smithi167 ceph-mon[32774]: osdmap e352: 8 total, 8 up, 8 in 2024-02-05T06:47:40.770 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:40.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:40.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:40.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:40 smithi028 ceph-mon[29442]: 4.3 scrub starts 2024-02-05T06:47:40.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:40 smithi028 ceph-mon[29442]: 4.3 scrub ok 2024-02-05T06:47:40.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:40 smithi028 ceph-mon[29442]: 3.1c scrub starts 2024-02-05T06:47:40.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:40 smithi028 ceph-mon[29442]: 3.1c scrub ok 2024-02-05T06:47:40.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:40 smithi028 ceph-mon[29442]: osdmap e352: 8 total, 8 up, 8 in 2024-02-05T06:47:40.831 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:40 smithi028 ceph-mon[37984]: 4.3 scrub starts 2024-02-05T06:47:40.832 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:40 smithi028 ceph-mon[37984]: 4.3 scrub ok 2024-02-05T06:47:40.832 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:40 smithi028 ceph-mon[37984]: 3.1c scrub starts 2024-02-05T06:47:40.832 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:40 smithi028 ceph-mon[37984]: 3.1c scrub ok 2024-02-05T06:47:40.832 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:40 smithi028 ceph-mon[37984]: osdmap e352: 8 total, 8 up, 8 in 2024-02-05T06:47:40.872 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:40.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:40.915 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:40.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:40.935 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1 15 2681 2666 166.455 166.625 0.0052478 0.0059864 2024-02-05T06:47:41.017 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:41.052 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:41.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:41.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:41.064 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:47:41.105 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:47:41.148 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:47:41.154 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:41.191 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:47:41.196 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:41.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:41.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:41.234 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:47:41.276 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:47:41.298 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:41.318 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:47:41.333 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:41.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:41.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:41.356 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:47:41.362 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:47:41.362 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:47:41.362 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:47:41.362 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:47:41.362 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:47:41.362 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:47:41.362 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:47:41.362 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:47:41.435 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:41.472 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:41.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:41.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:41.550 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.6 to 1 (10000) 2024-02-05T06:47:41.554 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-02-05T06:47:41.574 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:41.615 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:41.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:41.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:41.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:41 smithi028 ceph-mon[37984]: pgmap v678: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:47:41.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:41 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2513469525' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T06:47:41.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:41 smithi028 ceph-mon[37984]: from='client.45593 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T06:47:41.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:41 smithi028 ceph-mon[37984]: 4.6 scrub starts 2024-02-05T06:47:41.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:41 smithi028 ceph-mon[37984]: 4.6 scrub ok 2024-02-05T06:47:41.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:41 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/356695813' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:47:41.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:41 smithi028 ceph-mon[37984]: 4.d scrub starts 2024-02-05T06:47:41.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:41 smithi028 ceph-mon[37984]: 4.d scrub ok 2024-02-05T06:47:41.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:41 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3985327689' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:47:41.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:41 smithi028 ceph-mon[37984]: from='client.45599 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:47:41.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:41 smithi028 ceph-mon[37984]: 4.10 scrub starts 2024-02-05T06:47:41.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:41 smithi028 ceph-mon[37984]: 4.10 scrub ok 2024-02-05T06:47:41.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:41 smithi028 ceph-mon[29442]: pgmap v678: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:47:41.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:41 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2513469525' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T06:47:41.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:41 smithi028 ceph-mon[29442]: from='client.45593 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T06:47:41.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:41 smithi028 ceph-mon[29442]: 4.6 scrub starts 2024-02-05T06:47:41.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:41 smithi028 ceph-mon[29442]: 4.6 scrub ok 2024-02-05T06:47:41.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:41 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/356695813' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:47:41.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:41 smithi028 ceph-mon[29442]: 4.d scrub starts 2024-02-05T06:47:41.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:41 smithi028 ceph-mon[29442]: 4.d scrub ok 2024-02-05T06:47:41.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:41 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3985327689' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:47:41.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:41 smithi028 ceph-mon[29442]: from='client.45599 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:47:41.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:41 smithi028 ceph-mon[29442]: 4.10 scrub starts 2024-02-05T06:47:41.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:41 smithi028 ceph-mon[29442]: 4.10 scrub ok 2024-02-05T06:47:41.717 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:41.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:41.752 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:41.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:41.853 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:41.890 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:41.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:41.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:41.898 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:41 smithi167 ceph-mon[32774]: pgmap v678: 158 pgs: 158 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:47:41.898 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:41 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2513469525' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T06:47:41.898 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:41 smithi167 ceph-mon[32774]: from='client.45593 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T06:47:41.898 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:41 smithi167 ceph-mon[32774]: 4.6 scrub starts 2024-02-05T06:47:41.898 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:41 smithi167 ceph-mon[32774]: 4.6 scrub ok 2024-02-05T06:47:41.898 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:41 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/356695813' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:47:41.898 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:41 smithi167 ceph-mon[32774]: 4.d scrub starts 2024-02-05T06:47:41.898 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:41 smithi167 ceph-mon[32774]: 4.d scrub ok 2024-02-05T06:47:41.898 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:41 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3985327689' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:47:41.898 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:41 smithi167 ceph-mon[32774]: from='client.45599 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:47:41.898 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:41 smithi167 ceph-mon[32774]: 4.10 scrub starts 2024-02-05T06:47:41.898 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:41 smithi167 ceph-mon[32774]: 4.10 scrub ok 2024-02-05T06:47:41.935 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2 16 5471 5455 170.374 174.312 0.00469294 0.00585331 2024-02-05T06:47:41.993 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:42.030 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:42.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:42.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:42.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:42.155 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:42.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:42.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:42.257 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:42.293 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:42.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:42.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:42.394 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:42.430 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:42.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:42.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:42.532 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:42.567 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:42 smithi167 ceph-mon[32774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:47:42.567 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:42 smithi167 ceph-mon[32774]: Cluster is now healthy 2024-02-05T06:47:42.567 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:42 smithi167 ceph-mon[32774]: from='client.45593 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-05T06:47:42.567 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:42 smithi167 ceph-mon[32774]: from='client.45599 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:47:42.567 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:42 smithi167 ceph-mon[32774]: osdmap e353: 8 total, 8 up, 8 in 2024-02-05T06:47:42.568 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:42 smithi167 ceph-mon[32774]: 3.c deep-scrub starts 2024-02-05T06:47:42.568 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:42 smithi167 ceph-mon[32774]: 3.c deep-scrub ok 2024-02-05T06:47:42.568 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:42 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:47:42.568 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:42 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:47:42.568 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:42 smithi167 ceph-mon[32774]: 6.f scrub starts 2024-02-05T06:47:42.568 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:42 smithi167 ceph-mon[32774]: 6.f scrub ok 2024-02-05T06:47:42.570 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:42.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:42.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:42.673 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:42.708 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:42.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:42.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:42.810 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:42.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:42.850 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:42.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:42.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:42 smithi028 ceph-mon[37984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:47:42.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:42 smithi028 ceph-mon[37984]: Cluster is now healthy 2024-02-05T06:47:42.932 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:42 smithi028 ceph-mon[37984]: from='client.45593 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-05T06:47:42.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:42 smithi028 ceph-mon[37984]: from='client.45599 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:47:42.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:42 smithi028 ceph-mon[37984]: osdmap e353: 8 total, 8 up, 8 in 2024-02-05T06:47:42.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:42 smithi028 ceph-mon[37984]: 3.c deep-scrub starts 2024-02-05T06:47:42.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:42 smithi028 ceph-mon[37984]: 3.c deep-scrub ok 2024-02-05T06:47:42.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:42 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:47:42.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:42 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:47:42.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:42 smithi028 ceph-mon[37984]: 6.f scrub starts 2024-02-05T06:47:42.933 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:42 smithi028 ceph-mon[37984]: 6.f scrub ok 2024-02-05T06:47:42.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:42 smithi028 ceph-mon[29442]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:47:42.933 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:42 smithi028 ceph-mon[29442]: Cluster is now healthy 2024-02-05T06:47:42.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:42 smithi028 ceph-mon[29442]: from='client.45593 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-05T06:47:42.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:42 smithi028 ceph-mon[29442]: from='client.45599 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:47:42.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:42 smithi028 ceph-mon[29442]: osdmap e353: 8 total, 8 up, 8 in 2024-02-05T06:47:42.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:42 smithi028 ceph-mon[29442]: 3.c deep-scrub starts 2024-02-05T06:47:42.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:42 smithi028 ceph-mon[29442]: 3.c deep-scrub ok 2024-02-05T06:47:42.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:42 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:47:42.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:42 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:47:42.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:42 smithi028 ceph-mon[29442]: 6.f scrub starts 2024-02-05T06:47:42.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:42 smithi028 ceph-mon[29442]: 6.f scrub ok 2024-02-05T06:47:42.934 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:47:42 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:47:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:47:42.935 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3 15 8453 8438 175.721 186.438 0.00265556 0.00568108 2024-02-05T06:47:42.951 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:42.988 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:42.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:42.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:43.043 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:47:43.043 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:47:43.090 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:43.126 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:43.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:43.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:43.228 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:43.268 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:43.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:43.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:43.370 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:43.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:43.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:43.484 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:47:43 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:47:43.509 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:43.546 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:43.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:43.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:43.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd reweight 4 1 2024-02-05T06:47:43.647 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:43.683 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:43.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:43.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:43.753 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-05T06:47:43.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:43.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:43.824 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:43.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:43.924 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:43.935 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4 16 11426 11410 178.223 185.75 0.00482042 0.00560013 2024-02-05T06:47:43.966 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:43.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:43.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:43.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:43 smithi167 ceph-mon[32774]: 5.1f scrub starts 2024-02-05T06:47:43.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:43 smithi167 ceph-mon[32774]: 5.1f scrub ok 2024-02-05T06:47:43.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:43 smithi167 ceph-mon[32774]: pgmap v680: 158 pgs: 158 active+clean; 304 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.6 KiB/s rd, 48 MiB/s wr, 774 op/s 2024-02-05T06:47:43.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:43 smithi167 ceph-mon[32774]: 4.1c scrub starts 2024-02-05T06:47:43.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:43 smithi167 ceph-mon[32774]: 4.1c scrub ok 2024-02-05T06:47:43.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:43 smithi167 ceph-mon[32774]: 5.17 scrub starts 2024-02-05T06:47:43.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:43 smithi167 ceph-mon[32774]: 5.17 scrub ok 2024-02-05T06:47:44.069 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:44.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:44.115 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:44.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:44.137 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:43 smithi028 ceph-mon[29442]: 5.1f scrub starts 2024-02-05T06:47:44.137 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:43 smithi028 ceph-mon[29442]: 5.1f scrub ok 2024-02-05T06:47:44.137 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:43 smithi028 ceph-mon[29442]: pgmap v680: 158 pgs: 158 active+clean; 304 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.6 KiB/s rd, 48 MiB/s wr, 774 op/s 2024-02-05T06:47:44.137 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:43 smithi028 ceph-mon[29442]: 4.1c scrub starts 2024-02-05T06:47:44.137 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:43 smithi028 ceph-mon[29442]: 4.1c scrub ok 2024-02-05T06:47:44.137 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:43 smithi028 ceph-mon[29442]: 5.17 scrub starts 2024-02-05T06:47:44.138 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:43 smithi028 ceph-mon[29442]: 5.17 scrub ok 2024-02-05T06:47:44.138 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:43 smithi028 ceph-mon[37984]: 5.1f scrub starts 2024-02-05T06:47:44.138 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:43 smithi028 ceph-mon[37984]: 5.1f scrub ok 2024-02-05T06:47:44.138 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:43 smithi028 ceph-mon[37984]: pgmap v680: 158 pgs: 158 active+clean; 304 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.6 KiB/s rd, 48 MiB/s wr, 774 op/s 2024-02-05T06:47:44.138 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:43 smithi028 ceph-mon[37984]: 4.1c scrub starts 2024-02-05T06:47:44.138 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:43 smithi028 ceph-mon[37984]: 4.1c scrub ok 2024-02-05T06:47:44.138 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:43 smithi028 ceph-mon[37984]: 5.17 scrub starts 2024-02-05T06:47:44.138 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:43 smithi028 ceph-mon[37984]: 5.17 scrub ok 2024-02-05T06:47:44.167 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:47:44.216 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:44.252 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:44.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:44.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:44.354 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:44.394 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:44.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:44.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:44.496 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:44.534 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:44.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:44.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:44.630 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T06:47:44.636 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:44.711 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:44.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:44.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:44.813 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:44.867 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:44.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:44.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:44.935 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5 16 14209 14193 177.362 173.938 0.0031927 0.00563203 2024-02-05T06:47:44.969 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:45.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:45.006 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:45.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:45.055 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:47:45.107 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:45.110 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:47:45.140 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:45.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:45.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:45.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:44 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:47:45.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:44 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:47:45.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:44 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:47:45.242 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:45.277 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:45.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:45.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:45.379 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:45.429 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:45.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:45.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:45.531 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:45.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:45.571 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:45.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:45.672 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:45.711 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:45.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:45.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:45.813 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:45.851 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:45.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:45.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:45.935 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 6 15 16805 16790 174.852 162.312 0.00343523 0.00571294 2024-02-05T06:47:45.953 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:45.990 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:45.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:45.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:46.092 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:46.127 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:46.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:46.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:46.185 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:47:46.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:45 smithi028 ceph-mon[29442]: pgmap v681: 158 pgs: 158 active+clean; 304 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 38 MiB/s wr, 608 op/s 2024-02-05T06:47:46.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:45 smithi028 ceph-mon[29442]: 6.1a scrub starts 2024-02-05T06:47:46.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:45 smithi028 ceph-mon[29442]: 6.1a scrub ok 2024-02-05T06:47:46.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:45 smithi028 ceph-mon[29442]: 6.1 scrub starts 2024-02-05T06:47:46.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:45 smithi028 ceph-mon[29442]: 6.1 scrub ok 2024-02-05T06:47:46.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:45 smithi028 ceph-mon[29442]: 5.b deep-scrub starts 2024-02-05T06:47:46.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:45 smithi028 ceph-mon[29442]: 5.b deep-scrub ok 2024-02-05T06:47:46.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:45 smithi028 ceph-mon[29442]: osdmap e354: 8 total, 8 up, 8 in 2024-02-05T06:47:46.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:45 smithi028 ceph-mon[37984]: pgmap v681: 158 pgs: 158 active+clean; 304 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 38 MiB/s wr, 608 op/s 2024-02-05T06:47:46.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:45 smithi028 ceph-mon[37984]: 6.1a scrub starts 2024-02-05T06:47:46.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:45 smithi028 ceph-mon[37984]: 6.1a scrub ok 2024-02-05T06:47:46.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:45 smithi028 ceph-mon[37984]: 6.1 scrub starts 2024-02-05T06:47:46.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:45 smithi028 ceph-mon[37984]: 6.1 scrub ok 2024-02-05T06:47:46.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:45 smithi028 ceph-mon[37984]: 5.b deep-scrub starts 2024-02-05T06:47:46.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:45 smithi028 ceph-mon[37984]: 5.b deep-scrub ok 2024-02-05T06:47:46.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:45 smithi028 ceph-mon[37984]: osdmap e354: 8 total, 8 up, 8 in 2024-02-05T06:47:46.229 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:46.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:45 smithi167 ceph-mon[32774]: pgmap v681: 158 pgs: 158 active+clean; 304 MiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 38 MiB/s wr, 608 op/s 2024-02-05T06:47:46.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:45 smithi167 ceph-mon[32774]: 6.1a scrub starts 2024-02-05T06:47:46.236 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:45 smithi167 ceph-mon[32774]: 6.1a scrub ok 2024-02-05T06:47:46.236 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:45 smithi167 ceph-mon[32774]: 6.1 scrub starts 2024-02-05T06:47:46.236 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:45 smithi167 ceph-mon[32774]: 6.1 scrub ok 2024-02-05T06:47:46.236 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:45 smithi167 ceph-mon[32774]: 5.b deep-scrub starts 2024-02-05T06:47:46.236 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:45 smithi167 ceph-mon[32774]: 5.b deep-scrub ok 2024-02-05T06:47:46.236 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:45 smithi167 ceph-mon[32774]: osdmap e354: 8 total, 8 up, 8 in 2024-02-05T06:47:46.266 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:46.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:46.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:46.368 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:46.404 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:46.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:46.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:46.506 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:46.520 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:47:46.520 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:47:46.543 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:46.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:46.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:46.644 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:46.680 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:46.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:46.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:46.782 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:46.822 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:46.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:46.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:46.924 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:46.936 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 7 16 19312 19296 172.245 156.625 0.00474118 0.00579924 2024-02-05T06:47:46.963 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:46.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:46.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:47.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:47.105 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:47.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:47.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:47.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:46 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:47:47.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:46 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:47:47.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:46 smithi028 ceph-mon[29442]: 6.3 scrub starts 2024-02-05T06:47:47.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:46 smithi028 ceph-mon[29442]: 6.3 scrub ok 2024-02-05T06:47:47.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:46 smithi028 ceph-mon[29442]: 1.5 scrub starts 2024-02-05T06:47:47.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:46 smithi028 ceph-mon[29442]: 1.5 scrub ok 2024-02-05T06:47:47.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:46 smithi028 ceph-mon[29442]: osdmap e355: 8 total, 8 up, 8 in 2024-02-05T06:47:47.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:46 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:47:47.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:46 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:47:47.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:46 smithi028 ceph-mon[37984]: 6.3 scrub starts 2024-02-05T06:47:47.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:46 smithi028 ceph-mon[37984]: 6.3 scrub ok 2024-02-05T06:47:47.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:46 smithi028 ceph-mon[37984]: 1.5 scrub starts 2024-02-05T06:47:47.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:46 smithi028 ceph-mon[37984]: 1.5 scrub ok 2024-02-05T06:47:47.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:46 smithi028 ceph-mon[37984]: osdmap e355: 8 total, 8 up, 8 in 2024-02-05T06:47:47.207 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:47.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:46 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:47:47.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:46 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:47:47.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:46 smithi167 ceph-mon[32774]: 6.3 scrub starts 2024-02-05T06:47:47.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:46 smithi167 ceph-mon[32774]: 6.3 scrub ok 2024-02-05T06:47:47.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:46 smithi167 ceph-mon[32774]: 1.5 scrub starts 2024-02-05T06:47:47.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:46 smithi167 ceph-mon[32774]: 1.5 scrub ok 2024-02-05T06:47:47.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:46 smithi167 ceph-mon[32774]: osdmap e355: 8 total, 8 up, 8 in 2024-02-05T06:47:47.242 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:47.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:47.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:47.344 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:47.382 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:47.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:47.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:47.484 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:47.518 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-05T06:47:47.522 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:47.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:47.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:47.623 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:47.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:47.665 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:47.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:47.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:47.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:47.803 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:47.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:47.904 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:47.934 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:47:47.936 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 8 15 21742 21727 169.705 151.938 0.0144268 0.00588728 2024-02-05T06:47:47.946 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:47.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:47.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:48.049 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:48.081 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:48.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:48.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:48.183 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:48.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:47 smithi028 ceph-mon[29442]: pgmap v684: 158 pgs: 158 active+clean; 684 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 114 MiB/s wr, 1.82k op/s 2024-02-05T06:47:48.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:47 smithi028 ceph-mon[29442]: 5.e scrub starts 2024-02-05T06:47:48.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:47 smithi028 ceph-mon[29442]: 5.e scrub ok 2024-02-05T06:47:48.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:47 smithi028 ceph-mon[29442]: 2.0 deep-scrub starts 2024-02-05T06:47:48.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:47 smithi028 ceph-mon[29442]: 2.0 deep-scrub ok 2024-02-05T06:47:48.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:47 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:47:48.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:47 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:47:48.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:47 smithi028 ceph-mon[37984]: pgmap v684: 158 pgs: 158 active+clean; 684 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 114 MiB/s wr, 1.82k op/s 2024-02-05T06:47:48.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:47 smithi028 ceph-mon[37984]: 5.e scrub starts 2024-02-05T06:47:48.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:47 smithi028 ceph-mon[37984]: 5.e scrub ok 2024-02-05T06:47:48.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:47 smithi028 ceph-mon[37984]: 2.0 deep-scrub starts 2024-02-05T06:47:48.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:47 smithi028 ceph-mon[37984]: 2.0 deep-scrub ok 2024-02-05T06:47:48.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:47 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:47:48.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:47 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:47:48.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:48.220 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:48.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:48.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:47 smithi167 ceph-mon[32774]: pgmap v684: 158 pgs: 158 active+clean; 684 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 114 MiB/s wr, 1.82k op/s 2024-02-05T06:47:48.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:47 smithi167 ceph-mon[32774]: 5.e scrub starts 2024-02-05T06:47:48.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:47 smithi167 ceph-mon[32774]: 5.e scrub ok 2024-02-05T06:47:48.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:47 smithi167 ceph-mon[32774]: 2.0 deep-scrub starts 2024-02-05T06:47:48.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:47 smithi167 ceph-mon[32774]: 2.0 deep-scrub ok 2024-02-05T06:47:48.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:47 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:47:48.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:47 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:47:48.322 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:48.383 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:48.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:48.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:48.485 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:48.537 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:48.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:48.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:48.638 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:48.679 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:48.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:48.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:48.780 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:48.820 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:48.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:48.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:48.921 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:48.936 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 9 16 24040 24024 166.798 143.562 0.00471823 0.005988 2024-02-05T06:47:48.946 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:47:48.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:48.969 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:48.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:49.070 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:49.104 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:49.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:49.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:49.178 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:48 smithi028 ceph-mon[37984]: 3.14 scrub starts 2024-02-05T06:47:49.178 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:48 smithi028 ceph-mon[37984]: 3.14 scrub ok 2024-02-05T06:47:49.178 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:48 smithi028 ceph-mon[37984]: 4.19 scrub starts 2024-02-05T06:47:49.178 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:48 smithi028 ceph-mon[37984]: 4.19 scrub ok 2024-02-05T06:47:49.178 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:48 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:47:49.178 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:48 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:47:49.178 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:48 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:47:49.178 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:48 smithi028 ceph-mon[29442]: 3.14 scrub starts 2024-02-05T06:47:49.178 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:48 smithi028 ceph-mon[29442]: 3.14 scrub ok 2024-02-05T06:47:49.178 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:48 smithi028 ceph-mon[29442]: 4.19 scrub starts 2024-02-05T06:47:49.179 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:48 smithi028 ceph-mon[29442]: 4.19 scrub ok 2024-02-05T06:47:49.179 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:48 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:47:49.179 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:48 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:47:49.179 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:48 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:47:49.206 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:49.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:48 smithi167 ceph-mon[32774]: 3.14 scrub starts 2024-02-05T06:47:49.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:48 smithi167 ceph-mon[32774]: 3.14 scrub ok 2024-02-05T06:47:49.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:48 smithi167 ceph-mon[32774]: 4.19 scrub starts 2024-02-05T06:47:49.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:48 smithi167 ceph-mon[32774]: 4.19 scrub ok 2024-02-05T06:47:49.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:48 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:47:49.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:48 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:47:49.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:48 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:47:49.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:49.244 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:49.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:49.345 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:49.369 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:49.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:49.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:49.470 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:49.507 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:49.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:49.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:49.609 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:49.637 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:49.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:49.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:49.739 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:49.780 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:49.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:49.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:49.882 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:49.914 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:49.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:49.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:49.936 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 10 16 25983 25967 162.261 121.438 0.00485329 0.0061571 2024-02-05T06:47:50.016 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:50.050 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:50.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:50.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:50.083 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:49 smithi028 ceph-mon[37984]: 5.2 scrub starts 2024-02-05T06:47:50.084 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:49 smithi028 ceph-mon[37984]: 5.2 scrub ok 2024-02-05T06:47:50.084 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:49 smithi028 ceph-mon[37984]: pgmap v685: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 1.1 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 152 MiB/s wr, 2.43k op/s 2024-02-05T06:47:50.084 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:49 smithi028 ceph-mon[37984]: 6.e scrub starts 2024-02-05T06:47:50.084 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:49 smithi028 ceph-mon[37984]: 6.e scrub ok 2024-02-05T06:47:50.084 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:49 smithi028 ceph-mon[37984]: 2.1 scrub starts 2024-02-05T06:47:50.084 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:49 smithi028 ceph-mon[37984]: 2.1 scrub ok 2024-02-05T06:47:50.084 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:49 smithi028 ceph-mon[37984]: 5.5 scrub starts 2024-02-05T06:47:50.084 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:49 smithi028 ceph-mon[37984]: 5.5 scrub ok 2024-02-05T06:47:50.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:49 smithi028 ceph-mon[29442]: 5.2 scrub starts 2024-02-05T06:47:50.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:49 smithi028 ceph-mon[29442]: 5.2 scrub ok 2024-02-05T06:47:50.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:49 smithi028 ceph-mon[29442]: pgmap v685: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 1.1 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 152 MiB/s wr, 2.43k op/s 2024-02-05T06:47:50.085 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:49 smithi028 ceph-mon[29442]: 6.e scrub starts 2024-02-05T06:47:50.085 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:49 smithi028 ceph-mon[29442]: 6.e scrub ok 2024-02-05T06:47:50.085 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:49 smithi028 ceph-mon[29442]: 2.1 scrub starts 2024-02-05T06:47:50.085 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:49 smithi028 ceph-mon[29442]: 2.1 scrub ok 2024-02-05T06:47:50.085 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:49 smithi028 ceph-mon[29442]: 5.5 scrub starts 2024-02-05T06:47:50.085 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:49 smithi028 ceph-mon[29442]: 5.5 scrub ok 2024-02-05T06:47:50.151 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:50.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:50.192 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:50.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:50.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:49 smithi167 ceph-mon[32774]: 5.2 scrub starts 2024-02-05T06:47:50.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:49 smithi167 ceph-mon[32774]: 5.2 scrub ok 2024-02-05T06:47:50.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:49 smithi167 ceph-mon[32774]: pgmap v685: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 1.1 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.4 KiB/s rd, 152 MiB/s wr, 2.43k op/s 2024-02-05T06:47:50.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:49 smithi167 ceph-mon[32774]: 6.e scrub starts 2024-02-05T06:47:50.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:49 smithi167 ceph-mon[32774]: 6.e scrub ok 2024-02-05T06:47:50.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:49 smithi167 ceph-mon[32774]: 2.1 scrub starts 2024-02-05T06:47:50.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:49 smithi167 ceph-mon[32774]: 2.1 scrub ok 2024-02-05T06:47:50.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:49 smithi167 ceph-mon[32774]: 5.5 scrub starts 2024-02-05T06:47:50.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:49 smithi167 ceph-mon[32774]: 5.5 scrub ok 2024-02-05T06:47:50.294 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:50.329 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:50.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:50.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:50.431 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:50.570 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:50.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:50.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:50.671 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:50.704 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:50.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:50.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:50.806 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:50.835 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:50.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:50.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:50.936 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 11 16 27858 27842 158.163 117.188 0.00565344 0.00631709 2024-02-05T06:47:50.936 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:50.975 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:50.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:50.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:51.077 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:51.106 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:51.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:51.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:51.208 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:51.222 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.4 to 1 (10000) 2024-02-05T06:47:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:51.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:51.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:51.350 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:51.386 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:51.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:51.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:51.455 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:47:51.455 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:47:51.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:51 smithi167 ceph-mon[32774]: 3.19 scrub starts 2024-02-05T06:47:51.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:51 smithi167 ceph-mon[32774]: 3.19 scrub ok 2024-02-05T06:47:51.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:51 smithi167 ceph-mon[32774]: 4.a scrub starts 2024-02-05T06:47:51.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:51 smithi167 ceph-mon[32774]: 4.a scrub ok 2024-02-05T06:47:51.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:51 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/362909571' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T06:47:51.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:51 smithi167 ceph-mon[32774]: from='client.45611 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T06:47:51.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:51 smithi167 ceph-mon[32774]: 4.0 scrub starts 2024-02-05T06:47:51.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:51 smithi167 ceph-mon[32774]: 4.0 scrub ok 2024-02-05T06:47:51.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:51 smithi167 ceph-mon[32774]: 3.1b deep-scrub starts 2024-02-05T06:47:51.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:51 smithi167 ceph-mon[32774]: 3.1b deep-scrub ok 2024-02-05T06:47:51.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:51 smithi167 ceph-mon[32774]: pgmap v686: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 1.1 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-02-05T06:47:51.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:51 smithi167 ceph-mon[32774]: 5.6 scrub starts 2024-02-05T06:47:51.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:51 smithi167 ceph-mon[32774]: 5.6 scrub ok 2024-02-05T06:47:51.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:51.516 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:51.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:51.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:51.619 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:51.632 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:51 smithi028 ceph-mon[37984]: 3.19 scrub starts 2024-02-05T06:47:51.632 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:51 smithi028 ceph-mon[37984]: 3.19 scrub ok 2024-02-05T06:47:51.632 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:51 smithi028 ceph-mon[37984]: 4.a scrub starts 2024-02-05T06:47:51.632 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:51 smithi028 ceph-mon[37984]: 4.a scrub ok 2024-02-05T06:47:51.632 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:51 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/362909571' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T06:47:51.632 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:51 smithi028 ceph-mon[37984]: from='client.45611 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T06:47:51.632 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:51 smithi028 ceph-mon[37984]: 4.0 scrub starts 2024-02-05T06:47:51.632 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:51 smithi028 ceph-mon[37984]: 4.0 scrub ok 2024-02-05T06:47:51.632 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:51 smithi028 ceph-mon[37984]: 3.1b deep-scrub starts 2024-02-05T06:47:51.633 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:51 smithi028 ceph-mon[37984]: 3.1b deep-scrub ok 2024-02-05T06:47:51.633 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:51 smithi028 ceph-mon[37984]: pgmap v686: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 1.1 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-02-05T06:47:51.633 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:51 smithi028 ceph-mon[37984]: 5.6 scrub starts 2024-02-05T06:47:51.633 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:51 smithi028 ceph-mon[37984]: 5.6 scrub ok 2024-02-05T06:47:51.633 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:51 smithi028 ceph-mon[29442]: 3.19 scrub starts 2024-02-05T06:47:51.634 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:51 smithi028 ceph-mon[29442]: 3.19 scrub ok 2024-02-05T06:47:51.634 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:51 smithi028 ceph-mon[29442]: 4.a scrub starts 2024-02-05T06:47:51.634 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:51 smithi028 ceph-mon[29442]: 4.a scrub ok 2024-02-05T06:47:51.634 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:51 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/362909571' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T06:47:51.634 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:51 smithi028 ceph-mon[29442]: from='client.45611 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T06:47:51.634 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:51 smithi028 ceph-mon[29442]: 4.0 scrub starts 2024-02-05T06:47:51.634 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:51 smithi028 ceph-mon[29442]: 4.0 scrub ok 2024-02-05T06:47:51.634 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:51 smithi028 ceph-mon[29442]: 3.1b deep-scrub starts 2024-02-05T06:47:51.634 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:51 smithi028 ceph-mon[29442]: 3.1b deep-scrub ok 2024-02-05T06:47:51.634 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:51 smithi028 ceph-mon[29442]: pgmap v686: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 1.1 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-02-05T06:47:51.634 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:51 smithi028 ceph-mon[29442]: 5.6 scrub starts 2024-02-05T06:47:51.634 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:51 smithi028 ceph-mon[29442]: 5.6 scrub ok 2024-02-05T06:47:51.664 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:51.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:51.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:51.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:51.808 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:51.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:51.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:51.910 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:51.936 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 12 16 29672 29656 154.43 113.375 0.00736169 0.00646924 2024-02-05T06:47:51.952 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:51.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:51.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:52.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:52.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:52.084 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:52.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:52.185 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:52.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:52.223 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:52.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:52.324 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:52.347 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:52.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:52.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:52.449 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:52.474 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:52.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:52.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:52.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:52 smithi167 ceph-mon[32774]: from='client.45611 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-05T06:47:52.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:52 smithi167 ceph-mon[32774]: osdmap e356: 8 total, 8 up, 8 in 2024-02-05T06:47:52.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:52 smithi167 ceph-mon[32774]: 5.1a scrub starts 2024-02-05T06:47:52.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:52 smithi167 ceph-mon[32774]: 5.1a scrub ok 2024-02-05T06:47:52.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:52 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3657489094' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:47:52.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:52 smithi167 ceph-mon[32774]: 5.d scrub starts 2024-02-05T06:47:52.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:52 smithi167 ceph-mon[32774]: 5.d scrub ok 2024-02-05T06:47:52.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:52 smithi028 ceph-mon[37984]: from='client.45611 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-05T06:47:52.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:52 smithi028 ceph-mon[37984]: osdmap e356: 8 total, 8 up, 8 in 2024-02-05T06:47:52.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:52 smithi028 ceph-mon[37984]: 5.1a scrub starts 2024-02-05T06:47:52.503 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:52 smithi028 ceph-mon[37984]: 5.1a scrub ok 2024-02-05T06:47:52.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:52 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3657489094' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:47:52.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:52 smithi028 ceph-mon[37984]: 5.d scrub starts 2024-02-05T06:47:52.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:52 smithi028 ceph-mon[37984]: 5.d scrub ok 2024-02-05T06:47:52.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:52 smithi028 ceph-mon[29442]: from='client.45611 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-05T06:47:52.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:52 smithi028 ceph-mon[29442]: osdmap e356: 8 total, 8 up, 8 in 2024-02-05T06:47:52.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:52 smithi028 ceph-mon[29442]: 5.1a scrub starts 2024-02-05T06:47:52.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:52 smithi028 ceph-mon[29442]: 5.1a scrub ok 2024-02-05T06:47:52.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:52 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3657489094' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:47:52.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:52 smithi028 ceph-mon[29442]: 5.d scrub starts 2024-02-05T06:47:52.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:52 smithi028 ceph-mon[29442]: 5.d scrub ok 2024-02-05T06:47:52.576 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:52.599 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:52.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:52.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:52.670 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-02-05T06:47:52.701 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:52.739 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:52.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:52.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:52.841 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:52.865 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:52.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:52.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:52.936 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 13 16 30708 30692 147.532 64.75 0.00200587 0.00672182 2024-02-05T06:47:52.950 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:47:52 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:47:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:47:52.967 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:53.006 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:53.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:53.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:53.108 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:53.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:53.147 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:53.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:53.248 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:53.271 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:53.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:53.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:53.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:53.414 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:53.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:53.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:53.484 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:47:53 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:47:53.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:53 smithi167 ceph-mon[32774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:47:53.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:53 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3657489094' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:47:53.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:53 smithi167 ceph-mon[32774]: osdmap e357: 8 total, 8 up, 8 in 2024-02-05T06:47:53.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:53 smithi167 ceph-mon[32774]: pgmap v689: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.8 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.6 KiB/s rd, 243 MiB/s wr, 3.89k op/s 2024-02-05T06:47:53.515 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:53.551 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:53.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:53.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:53.653 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:53.690 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:53.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:53.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:53.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:53 smithi028 ceph-mon[29442]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:47:53.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:53 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3657489094' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:47:53.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:53 smithi028 ceph-mon[29442]: osdmap e357: 8 total, 8 up, 8 in 2024-02-05T06:47:53.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:53 smithi028 ceph-mon[29442]: pgmap v689: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.8 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.6 KiB/s rd, 243 MiB/s wr, 3.89k op/s 2024-02-05T06:47:53.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:53 smithi028 ceph-mon[37984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:47:53.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:53 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3657489094' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:47:53.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:53 smithi028 ceph-mon[37984]: osdmap e357: 8 total, 8 up, 8 in 2024-02-05T06:47:53.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:53 smithi028 ceph-mon[37984]: pgmap v689: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.8 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.6 KiB/s rd, 243 MiB/s wr, 3.89k op/s 2024-02-05T06:47:53.792 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:53.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:53.820 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:53.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:53.921 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:53.936 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 14 16 32178 32162 143.556 91.875 0.0018618 0.00693111 2024-02-05T06:47:53.944 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:53.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:53.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:54.046 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:54.082 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:54.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:54.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:54.183 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:54.217 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:54.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:54.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:54.319 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:54.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:54.363 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:54.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:54.464 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:54.499 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:54.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:54.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:54.601 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:54.624 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:54.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:54.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:54.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:54 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:47:54.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:54 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:47:54.725 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:54.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:54 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:47:54.749 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:54.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:54.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:54.850 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:54.887 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:54.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:54.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:54.936 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 15 16 33588 33572 139.86 88.125 0.00417248 0.00714461 2024-02-05T06:47:54.989 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:55.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:55.029 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:55.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:55.121 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd reweight 7 1 2024-02-05T06:47:55.130 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:55.188 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:55.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:55.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:55.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:55.329 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:55.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:55.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:55.431 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:55.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:55 smithi028 ceph-mon[29442]: pgmap v690: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.8 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.29k op/s 2024-02-05T06:47:55.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:55 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:47:55.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:55 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:47:55.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:55 smithi028 ceph-mon[37984]: pgmap v690: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.8 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.29k op/s 2024-02-05T06:47:55.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:55 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:47:55.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:55 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:47:55.454 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:55.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:55.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:55.555 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:55.594 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:55.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:55.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:55.655 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:55 smithi167 ceph-mon[32774]: pgmap v690: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.8 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 143 MiB/s wr, 2.29k op/s 2024-02-05T06:47:55.656 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:55 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:47:55.656 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:55 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:47:55.696 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:55.732 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:55.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:55.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:55.833 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:55.867 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:55.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:55.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:55.937 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 16 16 35179 35163 137.325 99.4375 0.00536517 0.00727615 2024-02-05T06:47:55.969 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:56.017 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:56.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:56.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:56.035 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:47:56.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:56.154 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:56.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:56.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:56.177 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-05T06:47:56.238 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T06:47:56.256 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:56.291 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:56.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:56.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:56.393 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:56.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:56.433 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:56.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:56.442 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:56 smithi028 ceph-mon[37984]: osdmap e358: 8 total, 8 up, 8 in 2024-02-05T06:47:56.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:56 smithi028 ceph-mon[29442]: osdmap e358: 8 total, 8 up, 8 in 2024-02-05T06:47:56.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:56 smithi167 ceph-mon[32774]: osdmap e358: 8 total, 8 up, 8 in 2024-02-05T06:47:56.534 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:56.571 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:47:56.581 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:56.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:56.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:56.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:56.715 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:47:56.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:56.730 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:56.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:56.761 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:47:56.795 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:47:56.832 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:56.835 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:47:56.867 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:56.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:56.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:56.877 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:47:56.917 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:47:56.938 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 17 16 36963 36947 135.805 111.5 0.0096964 0.00735786 2024-02-05T06:47:56.958 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:47:56.969 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:56.998 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:47:57.000 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:47:57.001 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:47:57.001 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:47:57.001 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:47:57.001 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:47:57.001 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:47:57.001 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:47:57.001 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:47:57.004 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:57.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:57.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:57.107 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:57.142 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:57.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:57.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:57.244 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:57.249 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:57 smithi028 ceph-mon[37984]: pgmap v692: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.0 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 144 MiB/s wr, 2.30k op/s 2024-02-05T06:47:57.250 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:57 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3435345215' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:47:57.287 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:57.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:57.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:57.390 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:57.431 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:57.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:57.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:57.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:57 smithi167 ceph-mon[32774]: pgmap v692: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.0 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 144 MiB/s wr, 2.30k op/s 2024-02-05T06:47:57.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:57 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3435345215' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:47:57.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:57 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:47:57.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:57 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:47:57.533 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:57.570 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:57.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:57.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:57.671 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:57.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:57 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:47:57.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:57 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:47:57.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:57 smithi028 ceph-mon[29442]: pgmap v692: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.0 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 144 MiB/s wr, 2.30k op/s 2024-02-05T06:47:57.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:57 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3435345215' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:47:57.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:57 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:47:57.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:57 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:47:57.710 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:57.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:57.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:57.812 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:57.855 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:57.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:57.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:57.938 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 18 16 38930 38914 135.089 122.938 0.00687006 0.0073965 2024-02-05T06:47:57.957 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:58.005 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:58.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:58.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:58.106 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:47:58.136 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:58.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:58.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:47:58.238 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:47:58.271 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:58.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:58.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:47:58.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:47:58.410 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:58.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:58.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:47:58.480 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:47:58.512 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:58.550 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:58.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:58.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:58.563 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:47:58.563 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-02-05T06:47:58.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-02-05T06:47:58.563 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-02-05T06:47:58.563 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.563 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:129449 8.d 8:b9130c76:::benchmark_data_smithi028_105698_object40174:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e261)", 2024-02-05T06:47:58.563 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:10.059390+0000", 2024-02-05T06:47:58.563 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 348.49230191800001, 2024-02-05T06:47:58.563 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0677847279999999, 2024-02-05T06:47:58.564 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:47:58.564 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-02-05T06:47:58.564 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:47:58.564 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:47:58.564 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:58.564 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 129449 2024-02-05T06:47:58.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.564 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:47:58.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.564 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:47:58.564 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059390+0000", 2024-02-05T06:47:58.564 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.564 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.564 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:47:58.565 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059390+0000", 2024-02-05T06:47:58.565 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.565 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:47:58.565 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059392+0000", 2024-02-05T06:47:58.565 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.086e-06 2024-02-05T06:47:58.565 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.565 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:47:58.565 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059398+0000", 2024-02-05T06:47:58.565 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5589999999999996e-06 2024-02-05T06:47:58.565 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.565 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:47:58.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059410+0000", 2024-02-05T06:47:58.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1905e-05 2024-02-05T06:47:58.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:47:58.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059418+0000", 2024-02-05T06:47:58.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5599999999999994e-06 2024-02-05T06:47:58.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.566 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.566 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.060054+0000", 2024-02-05T06:47:58.566 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00063580799999999997 2024-02-05T06:47:58.566 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.566 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.567 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:47:58.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.060063+0000", 2024-02-05T06:47:58.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.346e-06 2024-02-05T06:47:58.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.567 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.093914+0000", 2024-02-05T06:47:58.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0338508449999999 2024-02-05T06:47:58.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.567 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-02-05T06:47:58.567 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.094022+0000", 2024-02-05T06:47:58.567 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000107959 2024-02-05T06:47:58.567 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.567 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.127068+0000", 2024-02-05T06:47:58.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.033046308000000003 2024-02-05T06:47:58.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.568 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:47:58.568 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.127175+0000", 2024-02-05T06:47:58.568 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106352 2024-02-05T06:47:58.568 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.568 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:47:58.568 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.568 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.568 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.568 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:129452 8.d 8:b624ec5d:::benchmark_data_smithi028_105698_object40177:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e261)", 2024-02-05T06:47:58.569 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:10.059436+0000", 2024-02-05T06:47:58.569 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 348.49225564300002, 2024-02-05T06:47:58.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.074451045, 2024-02-05T06:47:58.569 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:47:58.569 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-02-05T06:47:58.569 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:47:58.569 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:47:58.569 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:58.569 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 129452 2024-02-05T06:47:58.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.569 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:47:58.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.569 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:47:58.569 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059436+0000", 2024-02-05T06:47:58.569 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.569 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.569 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:47:58.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059436+0000", 2024-02-05T06:47:58.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:47:58.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059437+0000", 2024-02-05T06:47:58.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7299999999999995e-07 2024-02-05T06:47:58.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.570 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:47:58.570 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059440+0000", 2024-02-05T06:47:58.570 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.897e-06 2024-02-05T06:47:58.570 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.570 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:47:58.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059446+0000", 2024-02-05T06:47:58.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1510000000000001e-06 2024-02-05T06:47:58.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:47:58.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059452+0000", 2024-02-05T06:47:58.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0619999999999997e-06 2024-02-05T06:47:58.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.571 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.060094+0000", 2024-02-05T06:47:58.571 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00064214899999999995 2024-02-05T06:47:58.571 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.571 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.571 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:47:58.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.060101+0000", 2024-02-05T06:47:58.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5180000000000001e-06 2024-02-05T06:47:58.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.572 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.101188+0000", 2024-02-05T06:47:58.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0410871850000001 2024-02-05T06:47:58.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.572 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.572 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-02-05T06:47:58.572 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.101304+0000", 2024-02-05T06:47:58.572 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000116476 2024-02-05T06:47:58.572 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.133814+0000", 2024-02-05T06:47:58.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.032509405999999998 2024-02-05T06:47:58.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.573 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:47:58.573 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.133887+0000", 2024-02-05T06:47:58.573 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3528000000000005e-05 2024-02-05T06:47:58.573 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.573 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:47:58.573 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.573 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.573 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.573 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:129475 8.d 8:becec02b:::benchmark_data_smithi028_105698_object40200:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e261)", 2024-02-05T06:47:58.573 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:10.094089+0000", 2024-02-05T06:47:58.573 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 348.457603033, 2024-02-05T06:47:58.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0243758060000001, 2024-02-05T06:47:58.574 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:47:58.574 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:47:58.574 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:47:58.574 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:47:58.574 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:58.574 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 129475 2024-02-05T06:47:58.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.574 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:47:58.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.574 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:47:58.574 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.094089+0000", 2024-02-05T06:47:58.574 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.574 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.574 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.575 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:47:58.575 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.094089+0000", 2024-02-05T06:47:58.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.575 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:47:58.575 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.094091+0000", 2024-02-05T06:47:58.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4049999999999998e-06 2024-02-05T06:47:58.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.575 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.575 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:47:58.575 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.094096+0000", 2024-02-05T06:47:58.575 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.724e-06 2024-02-05T06:47:58.575 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.576 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:47:58.576 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.094106+0000", 2024-02-05T06:47:58.576 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0434e-05 2024-02-05T06:47:58.576 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.576 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:47:58.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.094114+0000", 2024-02-05T06:47:58.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5950000000000001e-06 2024-02-05T06:47:58.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.101344+0000", 2024-02-05T06:47:58.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0072295349999999996 2024-02-05T06:47:58.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.577 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.577 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:47:58.577 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.101353+0000", 2024-02-05T06:47:58.577 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3430000000000002e-06 2024-02-05T06:47:58.577 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.111147+0000", 2024-02-05T06:47:58.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.009793742 2024-02-05T06:47:58.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:47:58.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.111256+0000", 2024-02-05T06:47:58.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010896900000000001 2024-02-05T06:47:58.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.578 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.578 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,7", 2024-02-05T06:47:58.578 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.111345+0000", 2024-02-05T06:47:58.578 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9696000000000003e-05 2024-02-05T06:47:58.578 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:47:58.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.112852+0000", 2024-02-05T06:47:58.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015071500000000001 2024-02-05T06:47:58.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.118307+0000", 2024-02-05T06:47:58.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0054547559999999998 2024-02-05T06:47:58.579 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.579 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.579 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.579 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.118341+0000", 2024-02-05T06:47:58.579 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4109000000000002e-05 2024-02-05T06:47:58.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.580 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:47:58.580 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.118382+0000", 2024-02-05T06:47:58.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0336999999999997e-05 2024-02-05T06:47:58.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.580 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:47:58.580 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.118465+0000", 2024-02-05T06:47:58.580 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3010999999999995e-05 2024-02-05T06:47:58.580 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.580 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:47:58.580 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.580 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.580 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.581 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:168946 8.4 8:2a7577b5:::benchmark_data_smithi028_105698_object79671:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:47:58.581 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.072950+0000", 2024-02-05T06:47:58.581 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 308.47874199900002, 2024-02-05T06:47:58.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4796128150000001, 2024-02-05T06:47:58.581 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:47:58.581 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-02-05T06:47:58.581 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:47:58.581 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:47:58.581 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:58.581 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 168946 2024-02-05T06:47:58.581 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.581 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:47:58.581 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.581 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:47:58.581 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.072950+0000", 2024-02-05T06:47:58.581 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.582 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.582 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:47:58.582 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.072950+0000", 2024-02-05T06:47:58.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.582 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.582 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:47:58.582 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.072952+0000", 2024-02-05T06:47:58.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9570000000000001e-06 2024-02-05T06:47:58.582 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.582 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.582 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:47:58.582 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.072956+0000", 2024-02-05T06:47:58.582 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2440000000000004e-06 2024-02-05T06:47:58.582 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.583 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:47:58.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.072967+0000", 2024-02-05T06:47:58.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0642000000000001e-05 2024-02-05T06:47:58.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.583 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:47:58.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.072975+0000", 2024-02-05T06:47:58.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1850000000000004e-06 2024-02-05T06:47:58.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.583 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.583 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.073035+0000", 2024-02-05T06:47:58.583 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0412999999999999e-05 2024-02-05T06:47:58.583 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.583 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:47:58.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.073049+0000", 2024-02-05T06:47:58.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3791000000000001e-05 2024-02-05T06:47:58.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.537403+0000", 2024-02-05T06:47:58.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.464353424 2024-02-05T06:47:58.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.584 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-02-05T06:47:58.584 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.537533+0000", 2024-02-05T06:47:58.584 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013027899999999999 2024-02-05T06:47:58.584 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.584 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.585 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.552442+0000", 2024-02-05T06:47:58.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.014908925999999999 2024-02-05T06:47:58.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.585 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:47:58.585 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.552563+0000", 2024-02-05T06:47:58.585 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012095399999999999 2024-02-05T06:47:58.585 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.585 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:47:58.585 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.585 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.585 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.585 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:168953 8.4 8:2441a741:::benchmark_data_smithi028_105698_object79678:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:47:58.585 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.076290+0000", 2024-02-05T06:47:58.585 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 308.47540213399998, 2024-02-05T06:47:58.586 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4712057160000001, 2024-02-05T06:47:58.586 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:47:58.586 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:47:58.586 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:47:58.586 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:47:58.586 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:58.586 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 168953 2024-02-05T06:47:58.586 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.586 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:47:58.586 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.586 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:47:58.586 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.076290+0000", 2024-02-05T06:47:58.586 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.586 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.586 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.587 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:47:58.587 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.076290+0000", 2024-02-05T06:47:58.587 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.587 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.587 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:47:58.587 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.076291+0000", 2024-02-05T06:47:58.587 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0999999999999997e-07 2024-02-05T06:47:58.587 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.587 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:47:58.587 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.076294+0000", 2024-02-05T06:47:58.587 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6210000000000001e-06 2024-02-05T06:47:58.587 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.587 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.588 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:47:58.588 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.076304+0000", 2024-02-05T06:47:58.588 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0172e-05 2024-02-05T06:47:58.588 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.588 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.588 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:47:58.588 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.076312+0000", 2024-02-05T06:47:58.588 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9500000000000001e-06 2024-02-05T06:47:58.588 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.588 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.588 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.588 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.076362+0000", 2024-02-05T06:47:58.588 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9302999999999999e-05 2024-02-05T06:47:58.588 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:47:58.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.076370+0000", 2024-02-05T06:47:58.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0369999999999995e-06 2024-02-05T06:47:58.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.537753+0000", 2024-02-05T06:47:58.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.461383297 2024-02-05T06:47:58.589 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.589 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.589 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:47:58.589 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.537848+0000", 2024-02-05T06:47:58.589 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4944000000000004e-05 2024-02-05T06:47:58.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:47:58.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.538252+0000", 2024-02-05T06:47:58.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000403631 2024-02-05T06:47:58.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547266+0000", 2024-02-05T06:47:58.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090147909999999994 2024-02-05T06:47:58.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.590 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.590 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:47:58.590 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547282+0000", 2024-02-05T06:47:58.590 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5639999999999999e-05 2024-02-05T06:47:58.590 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.591 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.591 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.591 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547408+0000", 2024-02-05T06:47:58.591 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012549300000000001 2024-02-05T06:47:58.591 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.591 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.591 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:47:58.591 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547440+0000", 2024-02-05T06:47:58.591 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2503999999999999e-05 2024-02-05T06:47:58.591 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.591 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.591 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:47:58.591 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547496+0000", 2024-02-05T06:47:58.591 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5423000000000001e-05 2024-02-05T06:47:58.591 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.591 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:47:58.592 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.592 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.592 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.592 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:168969 8.4 8:26e75fb4:::benchmark_data_smithi028_105698_object79694:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:47:58.592 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.095241+0000", 2024-02-05T06:47:58.592 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 308.45645091599999, 2024-02-05T06:47:58.592 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.452403653, 2024-02-05T06:47:58.592 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:47:58.592 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:47:58.592 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:47:58.592 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:47:58.592 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:58.592 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 168969 2024-02-05T06:47:58.592 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.592 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:47:58.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.593 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:47:58.593 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.095241+0000", 2024-02-05T06:47:58.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.593 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:47:58.593 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.095241+0000", 2024-02-05T06:47:58.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.593 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:47:58.593 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.095243+0000", 2024-02-05T06:47:58.593 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9439999999999999e-06 2024-02-05T06:47:58.593 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.593 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:47:58.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.095251+0000", 2024-02-05T06:47:58.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0669999999999992e-06 2024-02-05T06:47:58.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:47:58.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.095262+0000", 2024-02-05T06:47:58.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1071e-05 2024-02-05T06:47:58.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:47:58.594 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.095269+0000", 2024-02-05T06:47:58.594 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8140000000000003e-06 2024-02-05T06:47:58.594 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.594 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.594 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.095313+0000", 2024-02-05T06:47:58.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4387e-05 2024-02-05T06:47:58.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:47:58.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.095320+0000", 2024-02-05T06:47:58.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6290000000000004e-06 2024-02-05T06:47:58.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.542892+0000", 2024-02-05T06:47:58.595 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.447572388 2024-02-05T06:47:58.595 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.595 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.595 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:47:58.595 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.543029+0000", 2024-02-05T06:47:58.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013688 2024-02-05T06:47:58.596 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.596 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.596 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:47:58.596 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.543182+0000", 2024-02-05T06:47:58.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015253200000000001 2024-02-05T06:47:58.596 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.596 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.596 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:47:58.596 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547287+0000", 2024-02-05T06:47:58.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041050360000000003 2024-02-05T06:47:58.596 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.596 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.596 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.596 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547332+0000", 2024-02-05T06:47:58.596 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4749000000000001e-05 2024-02-05T06:47:58.597 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.597 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.597 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547552+0000", 2024-02-05T06:47:58.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022023999999999999 2024-02-05T06:47:58.597 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.597 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:47:58.597 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547579+0000", 2024-02-05T06:47:58.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7698000000000001e-05 2024-02-05T06:47:58.597 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.597 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.597 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:47:58.597 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547645+0000", 2024-02-05T06:47:58.597 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5217999999999998e-05 2024-02-05T06:47:58.598 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.598 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:47:58.598 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.598 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.598 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.598 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:168990 8.4 8:2816a5c0:::benchmark_data_smithi028_105698_object79715:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:47:58.598 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.116213+0000", 2024-02-05T06:47:58.598 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 308.43547941899999, 2024-02-05T06:47:58.598 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4316142949999999, 2024-02-05T06:47:58.598 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:47:58.598 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:47:58.598 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:47:58.598 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:47:58.598 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:58.599 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 168990 2024-02-05T06:47:58.599 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.599 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:47:58.599 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:47:58.599 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.116213+0000", 2024-02-05T06:47:58.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.599 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.599 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:47:58.599 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.116213+0000", 2024-02-05T06:47:58.599 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.599 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.599 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.599 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:47:58.599 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.116213+0000", 2024-02-05T06:47:58.600 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4199999999999996e-07 2024-02-05T06:47:58.600 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.600 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.600 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:47:58.600 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.116216+0000", 2024-02-05T06:47:58.600 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5890000000000001e-06 2024-02-05T06:47:58.600 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.600 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.600 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:47:58.600 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.116221+0000", 2024-02-05T06:47:58.600 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3009999999999997e-06 2024-02-05T06:47:58.600 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.600 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.600 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:47:58.601 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.116226+0000", 2024-02-05T06:47:58.601 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8409999999999999e-06 2024-02-05T06:47:58.601 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.601 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.601 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.601 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.129047+0000", 2024-02-05T06:47:58.601 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.01282082 2024-02-05T06:47:58.601 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.601 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.601 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:47:58.602 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.129057+0000", 2024-02-05T06:47:58.602 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0481e-05 2024-02-05T06:47:58.602 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.602 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.602 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.602 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.545004+0000", 2024-02-05T06:47:58.602 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.415947125 2024-02-05T06:47:58.602 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.602 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.602 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:47:58.603 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.545081+0000", 2024-02-05T06:47:58.603 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6550000000000004e-05 2024-02-05T06:47:58.603 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.603 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.603 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:47:58.603 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.545152+0000", 2024-02-05T06:47:58.603 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1728000000000002e-05 2024-02-05T06:47:58.603 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.603 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.603 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:47:58.603 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547292+0000", 2024-02-05T06:47:58.603 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002139939 2024-02-05T06:47:58.603 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.603 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.604 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.604 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547373+0000", 2024-02-05T06:47:58.604 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0328999999999996e-05 2024-02-05T06:47:58.604 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.604 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.604 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.604 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547719+0000", 2024-02-05T06:47:58.604 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034667500000000001 2024-02-05T06:47:58.604 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.604 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.604 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:47:58.604 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547746+0000", 2024-02-05T06:47:58.604 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6506999999999999e-05 2024-02-05T06:47:58.604 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.605 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.605 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:47:58.605 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547827+0000", 2024-02-05T06:47:58.605 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0867999999999999e-05 2024-02-05T06:47:58.605 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.605 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:47:58.605 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.605 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.605 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.605 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:168997 8.4 8:2fd29a35:::benchmark_data_smithi028_105698_object79722:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:47:58.605 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.123539+0000", 2024-02-05T06:47:58.605 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 308.42815243199999, 2024-02-05T06:47:58.605 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.43626739, 2024-02-05T06:47:58.605 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:47:58.605 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:47:58.606 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:47:58.606 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:47:58.606 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:58.606 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 168997 2024-02-05T06:47:58.606 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.606 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:47:58.606 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.606 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:47:58.606 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.123539+0000", 2024-02-05T06:47:58.606 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.606 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.606 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.606 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:47:58.606 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.123539+0000", 2024-02-05T06:47:58.607 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.607 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.607 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.607 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:47:58.607 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.123542+0000", 2024-02-05T06:47:58.607 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1509999999999998e-06 2024-02-05T06:47:58.607 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.607 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.607 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:47:58.607 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.123548+0000", 2024-02-05T06:47:58.607 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5780000000000004e-06 2024-02-05T06:47:58.607 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.607 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.607 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:47:58.607 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.123560+0000", 2024-02-05T06:47:58.608 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1674999999999999e-05 2024-02-05T06:47:58.608 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.608 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.608 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:47:58.608 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.123567+0000", 2024-02-05T06:47:58.608 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5739999999999998e-06 2024-02-05T06:47:58.608 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.608 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.608 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.608 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.129085+0000", 2024-02-05T06:47:58.608 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055173410000000003 2024-02-05T06:47:58.608 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.608 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.608 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:47:58.609 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.129091+0000", 2024-02-05T06:47:58.609 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5980000000000002e-06 2024-02-05T06:47:58.609 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.609 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.609 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.609 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547866+0000", 2024-02-05T06:47:58.609 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4187741730000001 2024-02-05T06:47:58.609 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.609 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.609 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:47:58.609 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547932+0000", 2024-02-05T06:47:58.609 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6223999999999995e-05 2024-02-05T06:47:58.609 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.609 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.609 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:47:58.610 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547998+0000", 2024-02-05T06:47:58.610 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5950000000000004e-05 2024-02-05T06:47:58.610 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.610 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.610 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:47:58.610 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.552551+0000", 2024-02-05T06:47:58.610 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0045535530000000001 2024-02-05T06:47:58.610 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.610 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.610 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.610 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.555220+0000", 2024-02-05T06:47:58.610 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0026689090000000001 2024-02-05T06:47:58.610 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.610 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.610 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.611 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.559657+0000", 2024-02-05T06:47:58.611 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0044368070000000001 2024-02-05T06:47:58.611 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.611 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.611 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:47:58.611 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.559714+0000", 2024-02-05T06:47:58.611 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6481999999999999e-05 2024-02-05T06:47:58.611 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.611 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.611 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:47:58.611 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.559807+0000", 2024-02-05T06:47:58.611 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3375000000000006e-05 2024-02-05T06:47:58.611 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.611 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:47:58.611 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.612 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.612 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.612 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:168999 8.4 8:2a087a6b:::benchmark_data_smithi028_105698_object79724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:47:58.612 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.124021+0000", 2024-02-05T06:47:58.612 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 308.42767074900001, 2024-02-05T06:47:58.612 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.436005618, 2024-02-05T06:47:58.612 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:47:58.612 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:47:58.612 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:47:58.612 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:47:58.612 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:58.612 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 168999 2024-02-05T06:47:58.612 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.612 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:47:58.612 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.612 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:47:58.613 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124021+0000", 2024-02-05T06:47:58.613 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.613 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.613 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.613 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:47:58.613 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124021+0000", 2024-02-05T06:47:58.613 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.613 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.613 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.613 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:47:58.613 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124022+0000", 2024-02-05T06:47:58.613 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.268e-06 2024-02-05T06:47:58.613 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.613 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.614 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:47:58.614 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124027+0000", 2024-02-05T06:47:58.614 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1500000000000001e-06 2024-02-05T06:47:58.614 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.614 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.614 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:47:58.614 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124037+0000", 2024-02-05T06:47:58.614 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0278999999999999e-05 2024-02-05T06:47:58.614 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.614 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.614 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:47:58.614 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124045+0000", 2024-02-05T06:47:58.614 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3520000000000007e-06 2024-02-05T06:47:58.614 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.615 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.615 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.615 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.129115+0000", 2024-02-05T06:47:58.615 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0050697160000000002 2024-02-05T06:47:58.615 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.615 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.615 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:47:58.615 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.129122+0000", 2024-02-05T06:47:58.615 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7549999999999997e-06 2024-02-05T06:47:58.615 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.615 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.615 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.615 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.549524+0000", 2024-02-05T06:47:58.616 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4204025490000001 2024-02-05T06:47:58.616 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.616 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.616 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:47:58.616 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.549646+0000", 2024-02-05T06:47:58.616 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121548 2024-02-05T06:47:58.616 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.616 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.616 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:47:58.616 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.549777+0000", 2024-02-05T06:47:58.616 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013103499999999999 2024-02-05T06:47:58.616 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.616 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.616 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:47:58.616 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.552562+0000", 2024-02-05T06:47:58.616 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002784777 2024-02-05T06:47:58.617 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.617 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.617 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.617 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.555297+0000", 2024-02-05T06:47:58.617 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002735632 2024-02-05T06:47:58.617 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.617 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.617 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.617 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.559878+0000", 2024-02-05T06:47:58.617 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0045809509999999998 2024-02-05T06:47:58.617 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.617 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.617 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:47:58.617 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.559934+0000", 2024-02-05T06:47:58.618 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5815e-05 2024-02-05T06:47:58.618 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.618 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.618 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:47:58.618 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.560027+0000", 2024-02-05T06:47:58.618 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2791e-05 2024-02-05T06:47:58.618 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.618 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:47:58.618 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.618 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.618 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.618 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:169001 8.4 8:2d24d8c7:::benchmark_data_smithi028_105698_object79726:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:47:58.618 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.124640+0000", 2024-02-05T06:47:58.618 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 308.42705227099998, 2024-02-05T06:47:58.618 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4355753789999999, 2024-02-05T06:47:58.619 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:47:58.619 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:47:58.619 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:47:58.619 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:47:58.619 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:58.619 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169001 2024-02-05T06:47:58.619 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.619 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:47:58.619 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.619 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:47:58.619 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124640+0000", 2024-02-05T06:47:58.619 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.619 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.619 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.619 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:47:58.620 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124640+0000", 2024-02-05T06:47:58.620 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.620 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.620 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.620 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:47:58.620 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124641+0000", 2024-02-05T06:47:58.620 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2759999999999999e-06 2024-02-05T06:47:58.620 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.620 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.620 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:47:58.620 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124646+0000", 2024-02-05T06:47:58.620 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7790000000000003e-06 2024-02-05T06:47:58.620 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.620 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.621 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:47:58.621 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124657+0000", 2024-02-05T06:47:58.621 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1739000000000001e-05 2024-02-05T06:47:58.621 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.621 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.621 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:47:58.621 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124666+0000", 2024-02-05T06:47:58.621 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3999999999999992e-06 2024-02-05T06:47:58.621 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.621 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.621 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.621 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.129144+0000", 2024-02-05T06:47:58.621 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0044785270000000004 2024-02-05T06:47:58.621 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.621 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.621 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:47:58.622 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.129150+0000", 2024-02-05T06:47:58.622 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7849999999999997e-06 2024-02-05T06:47:58.622 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.622 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.622 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.622 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.552592+0000", 2024-02-05T06:47:58.622 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.423442265 2024-02-05T06:47:58.622 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.622 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.622 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:47:58.622 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.552720+0000", 2024-02-05T06:47:58.622 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012783400000000001 2024-02-05T06:47:58.622 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.622 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.622 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:47:58.623 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.552831+0000", 2024-02-05T06:47:58.623 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000110325 2024-02-05T06:47:58.623 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.623 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.623 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.623 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.559592+0000", 2024-02-05T06:47:58.623 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.006761469 2024-02-05T06:47:58.623 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.623 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.623 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:47:58.623 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.559817+0000", 2024-02-05T06:47:58.623 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022453499999999999 2024-02-05T06:47:58.623 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.623 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.623 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.624 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.560092+0000", 2024-02-05T06:47:58.624 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027514500000000002 2024-02-05T06:47:58.624 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.624 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.624 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:47:58.624 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.560146+0000", 2024-02-05T06:47:58.624 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4534999999999999e-05 2024-02-05T06:47:58.624 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.624 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.624 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:47:58.624 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.560215+0000", 2024-02-05T06:47:58.624 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8764999999999996e-05 2024-02-05T06:47:58.624 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.624 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:47:58.624 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.625 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.625 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.625 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:169022 8.4 8:2073961b:::benchmark_data_smithi028_105698_object79747:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:47:58.625 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.143624+0000", 2024-02-05T06:47:58.625 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 308.40806816000003, 2024-02-05T06:47:58.625 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4328469529999999, 2024-02-05T06:47:58.625 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:47:58.625 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:47:58.625 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:47:58.625 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:47:58.625 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:58.625 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169022 2024-02-05T06:47:58.625 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.625 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:47:58.626 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.626 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:47:58.626 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.143624+0000", 2024-02-05T06:47:58.626 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.626 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.626 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.626 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:47:58.626 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.143624+0000", 2024-02-05T06:47:58.626 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.626 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.626 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.626 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:47:58.626 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.143625+0000", 2024-02-05T06:47:58.626 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5120000000000001e-06 2024-02-05T06:47:58.626 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.627 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.627 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:47:58.627 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.143630+0000", 2024-02-05T06:47:58.627 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8289999999999998e-06 2024-02-05T06:47:58.627 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.627 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.627 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:47:58.627 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.143642+0000", 2024-02-05T06:47:58.627 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1574e-05 2024-02-05T06:47:58.627 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.627 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.627 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:47:58.627 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.143648+0000", 2024-02-05T06:47:58.627 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2870000000000001e-06 2024-02-05T06:47:58.627 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.628 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.628 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.628 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.143712+0000", 2024-02-05T06:47:58.628 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3972000000000003e-05 2024-02-05T06:47:58.628 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.628 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.628 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:47:58.628 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.143724+0000", 2024-02-05T06:47:58.628 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2119999999999999e-05 2024-02-05T06:47:58.628 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.628 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.628 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.628 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.555338+0000", 2024-02-05T06:47:58.628 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4116135439999999 2024-02-05T06:47:58.629 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.629 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.629 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:47:58.629 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.555474+0000", 2024-02-05T06:47:58.629 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013640399999999999 2024-02-05T06:47:58.629 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.629 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.629 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:47:58.629 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.555604+0000", 2024-02-05T06:47:58.629 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000130102 2024-02-05T06:47:58.629 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.629 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.629 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:47:58.630 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.559821+0000", 2024-02-05T06:47:58.630 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0042167460000000004 2024-02-05T06:47:58.630 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.630 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.630 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.630 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.565344+0000", 2024-02-05T06:47:58.630 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055230440000000004 2024-02-05T06:47:58.630 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.630 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.630 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.630 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576302+0000", 2024-02-05T06:47:58.630 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010958239 2024-02-05T06:47:58.630 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.630 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.630 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:47:58.630 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576356+0000", 2024-02-05T06:47:58.631 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3823999999999998e-05 2024-02-05T06:47:58.631 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.631 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.631 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:47:58.631 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576471+0000", 2024-02-05T06:47:58.631 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000114756 2024-02-05T06:47:58.631 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.631 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:47:58.631 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.631 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.631 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.631 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:169057 8.4 8:26322b02:::benchmark_data_smithi028_105698_object79782:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:47:58.631 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.170483+0000", 2024-02-05T06:47:58.631 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 308.381209173, 2024-02-05T06:47:58.632 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.409427684, 2024-02-05T06:47:58.632 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:47:58.632 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:47:58.632 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:47:58.632 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:47:58.632 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:58.632 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169057 2024-02-05T06:47:58.632 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.632 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:47:58.632 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.632 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:47:58.632 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.170483+0000", 2024-02-05T06:47:58.632 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.632 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.632 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.633 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:47:58.633 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.170483+0000", 2024-02-05T06:47:58.633 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.633 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.633 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.633 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:47:58.633 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.170485+0000", 2024-02-05T06:47:58.633 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1239999999999999e-06 2024-02-05T06:47:58.633 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.633 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.633 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:47:58.633 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.170490+0000", 2024-02-05T06:47:58.633 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9490000000000004e-06 2024-02-05T06:47:58.633 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.633 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.634 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:47:58.634 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.170502+0000", 2024-02-05T06:47:58.634 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1851999999999999e-05 2024-02-05T06:47:58.634 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.634 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.634 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:47:58.634 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.170509+0000", 2024-02-05T06:47:58.634 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2320000000000004e-06 2024-02-05T06:47:58.634 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.634 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.634 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.634 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.170567+0000", 2024-02-05T06:47:58.634 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8152000000000002e-05 2024-02-05T06:47:58.634 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.634 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.635 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:47:58.635 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.170577+0000", 2024-02-05T06:47:58.635 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9150000000000008e-06 2024-02-05T06:47:58.635 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.635 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.635 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.635 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.557586+0000", 2024-02-05T06:47:58.635 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3870086580000001 2024-02-05T06:47:58.635 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.635 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.635 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:47:58.635 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.557731+0000", 2024-02-05T06:47:58.635 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014533499999999999 2024-02-05T06:47:58.635 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.635 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.635 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:47:58.635 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.557845+0000", 2024-02-05T06:47:58.636 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000114529 2024-02-05T06:47:58.636 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.636 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.636 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:47:58.636 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.560223+0000", 2024-02-05T06:47:58.636 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002377503 2024-02-05T06:47:58.636 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.636 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.636 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.636 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.565405+0000", 2024-02-05T06:47:58.636 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0051823950000000002 2024-02-05T06:47:58.636 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.636 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.636 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.636 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.579808+0000", 2024-02-05T06:47:58.637 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.014402223 2024-02-05T06:47:58.637 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.637 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.637 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:47:58.637 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.579872+0000", 2024-02-05T06:47:58.637 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4863999999999994e-05 2024-02-05T06:47:58.637 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.637 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.637 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:47:58.637 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.579910+0000", 2024-02-05T06:47:58.637 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7953000000000001e-05 2024-02-05T06:47:58.637 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.637 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:47:58.637 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.637 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.638 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.638 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:169060 8.4 8:282dec9e:::benchmark_data_smithi028_105698_object79785:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:47:58.638 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.172664+0000", 2024-02-05T06:47:58.638 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 308.37902814099999, 2024-02-05T06:47:58.638 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4074563200000001, 2024-02-05T06:47:58.638 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:47:58.638 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:47:58.638 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:47:58.638 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:47:58.638 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:58.638 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169060 2024-02-05T06:47:58.638 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.638 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:47:58.638 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.638 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:47:58.639 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.172664+0000", 2024-02-05T06:47:58.639 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.639 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.639 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.639 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:47:58.639 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.172664+0000", 2024-02-05T06:47:58.639 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.639 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.639 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.639 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:47:58.639 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.172665+0000", 2024-02-05T06:47:58.639 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4389999999999999e-06 2024-02-05T06:47:58.639 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.639 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.639 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:47:58.640 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.172673+0000", 2024-02-05T06:47:58.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3529999999999997e-06 2024-02-05T06:47:58.640 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.640 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:47:58.640 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.172683+0000", 2024-02-05T06:47:58.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0891e-05 2024-02-05T06:47:58.640 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.640 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:47:58.640 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.172691+0000", 2024-02-05T06:47:58.640 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4569999999999999e-06 2024-02-05T06:47:58.640 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.640 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.641 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.641 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.172734+0000", 2024-02-05T06:47:58.641 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2962e-05 2024-02-05T06:47:58.641 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.641 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.641 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:47:58.642 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.172740+0000", 2024-02-05T06:47:58.642 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1149999999999999e-06 2024-02-05T06:47:58.642 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.642 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.642 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.642 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.560263+0000", 2024-02-05T06:47:58.642 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3875225520000001 2024-02-05T06:47:58.642 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.642 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.642 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:47:58.642 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.560368+0000", 2024-02-05T06:47:58.642 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000105481 2024-02-05T06:47:58.642 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.642 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.643 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:47:58.643 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.560522+0000", 2024-02-05T06:47:58.643 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015420200000000001 2024-02-05T06:47:58.643 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.643 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.643 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:47:58.643 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.565356+0000", 2024-02-05T06:47:58.643 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048341139999999996 2024-02-05T06:47:58.643 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.643 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.643 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.643 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.571129+0000", 2024-02-05T06:47:58.643 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0057722889999999999 2024-02-05T06:47:58.643 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.643 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.644 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.644 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580047+0000", 2024-02-05T06:47:58.644 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0089180249999999996 2024-02-05T06:47:58.644 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.644 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.644 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:47:58.644 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580093+0000", 2024-02-05T06:47:58.644 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6730000000000002e-05 2024-02-05T06:47:58.644 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.644 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.644 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:47:58.644 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580120+0000", 2024-02-05T06:47:58.644 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.671e-05 2024-02-05T06:47:58.645 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.645 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:47:58.645 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.645 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.645 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.645 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:169062 8.4 8:2afada59:::benchmark_data_smithi028_105698_object79787:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:47:58.645 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.173201+0000", 2024-02-05T06:47:58.645 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 308.37849085900001, 2024-02-05T06:47:58.645 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4070665949999999, 2024-02-05T06:47:58.645 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:47:58.645 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:47:58.645 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:47:58.645 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:47:58.645 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:58.645 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169062 2024-02-05T06:47:58.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.646 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:47:58.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.646 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:47:58.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.173201+0000", 2024-02-05T06:47:58.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.646 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:47:58.646 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.173201+0000", 2024-02-05T06:47:58.646 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.646 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.646 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.646 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:47:58.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.173202+0000", 2024-02-05T06:47:58.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7399999999999991e-07 2024-02-05T06:47:58.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:47:58.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.173206+0000", 2024-02-05T06:47:58.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8999999999999999e-06 2024-02-05T06:47:58.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.647 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:47:58.647 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.173215+0000", 2024-02-05T06:47:58.647 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7329999999999992e-06 2024-02-05T06:47:58.647 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.647 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.648 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:47:58.648 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.173221+0000", 2024-02-05T06:47:58.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0419999999999999e-06 2024-02-05T06:47:58.648 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.648 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.648 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.173266+0000", 2024-02-05T06:47:58.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5256999999999997e-05 2024-02-05T06:47:58.648 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.648 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.648 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:47:58.648 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.173275+0000", 2024-02-05T06:47:58.648 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.003e-06 2024-02-05T06:47:58.649 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.649 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.649 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.649 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.562986+0000", 2024-02-05T06:47:58.649 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3897114660000001 2024-02-05T06:47:58.649 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.649 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.649 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:47:58.649 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.563100+0000", 2024-02-05T06:47:58.649 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113683 2024-02-05T06:47:58.649 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.649 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.649 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:47:58.649 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.563224+0000", 2024-02-05T06:47:58.649 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123738 2024-02-05T06:47:58.650 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.650 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.650 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.650 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.571213+0000", 2024-02-05T06:47:58.650 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079889839999999993 2024-02-05T06:47:58.650 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.650 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.650 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:47:58.650 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576145+0000", 2024-02-05T06:47:58.650 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00493196 2024-02-05T06:47:58.650 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.650 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.650 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.650 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580160+0000", 2024-02-05T06:47:58.650 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040155939999999999 2024-02-05T06:47:58.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.651 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:47:58.651 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580218+0000", 2024-02-05T06:47:58.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7779999999999999e-05 2024-02-05T06:47:58.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.651 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:47:58.651 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580268+0000", 2024-02-05T06:47:58.651 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9481000000000003e-05 2024-02-05T06:47:58.651 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.651 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:47:58.651 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.651 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.651 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.652 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:169071 8.4 8:2a441816:::benchmark_data_smithi028_105698_object79796:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:47:58.652 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.181972+0000", 2024-02-05T06:47:58.652 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 308.36971985500003, 2024-02-05T06:47:58.652 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3984312409999999, 2024-02-05T06:47:58.652 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:47:58.652 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:47:58.652 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:47:58.652 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:47:58.652 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:58.652 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169071 2024-02-05T06:47:58.652 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.652 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:47:58.652 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.652 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:47:58.652 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.181972+0000", 2024-02-05T06:47:58.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.653 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:47:58.653 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.181972+0000", 2024-02-05T06:47:58.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.653 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:47:58.653 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.181974+0000", 2024-02-05T06:47:58.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9980000000000002e-06 2024-02-05T06:47:58.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.653 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.653 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:47:58.653 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.181979+0000", 2024-02-05T06:47:58.653 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7310000000000001e-06 2024-02-05T06:47:58.653 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.654 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:47:58.654 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.181990+0000", 2024-02-05T06:47:58.654 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1381e-05 2024-02-05T06:47:58.654 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.654 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:47:58.654 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.181998+0000", 2024-02-05T06:47:58.654 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6539999999999999e-06 2024-02-05T06:47:58.654 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.654 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.654 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.654 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.182056+0000", 2024-02-05T06:47:58.654 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8276000000000002e-05 2024-02-05T06:47:58.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:47:58.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.182066+0000", 2024-02-05T06:47:58.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6199999999999994e-06 2024-02-05T06:47:58.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.565453+0000", 2024-02-05T06:47:58.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.383387401 2024-02-05T06:47:58.655 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.655 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.655 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:47:58.655 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.565578+0000", 2024-02-05T06:47:58.655 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012461699999999999 2024-02-05T06:47:58.656 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.656 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.656 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:47:58.656 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.565709+0000", 2024-02-05T06:47:58.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013105899999999999 2024-02-05T06:47:58.656 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.656 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.656 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:47:58.656 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576157+0000", 2024-02-05T06:47:58.656 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010448169 2024-02-05T06:47:58.656 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.656 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.657 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.657 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576240+0000", 2024-02-05T06:47:58.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3319999999999995e-05 2024-02-05T06:47:58.657 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.657 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.657 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580310+0000", 2024-02-05T06:47:58.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040695719999999996 2024-02-05T06:47:58.657 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.657 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:47:58.657 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580371+0000", 2024-02-05T06:47:58.657 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1490999999999998e-05 2024-02-05T06:47:58.657 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.657 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.658 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:47:58.658 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580403+0000", 2024-02-05T06:47:58.658 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1952000000000002e-05 2024-02-05T06:47:58.658 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.658 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:47:58.658 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.658 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.658 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.658 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:169074 8.4 8:23b7f380:::benchmark_data_smithi028_105698_object79799:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:47:58.658 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.185596+0000", 2024-02-05T06:47:58.658 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 308.36609579100002, 2024-02-05T06:47:58.658 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.394963838, 2024-02-05T06:47:58.658 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:47:58.658 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:47:58.659 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:47:58.659 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:47:58.659 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:58.659 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169074 2024-02-05T06:47:58.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.659 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:47:58.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:47:58.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.185596+0000", 2024-02-05T06:47:58.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.659 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.659 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.659 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:47:58.659 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.185596+0000", 2024-02-05T06:47:58.659 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:47:58.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.185597+0000", 2024-02-05T06:47:58.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8199999999999999e-07 2024-02-05T06:47:58.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:47:58.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.185600+0000", 2024-02-05T06:47:58.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6970000000000002e-06 2024-02-05T06:47:58.660 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.660 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.660 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:47:58.660 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.185607+0000", 2024-02-05T06:47:58.660 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0029999999999997e-06 2024-02-05T06:47:58.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.661 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:47:58.661 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.185611+0000", 2024-02-05T06:47:58.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4859999999999999e-06 2024-02-05T06:47:58.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.661 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.661 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.185655+0000", 2024-02-05T06:47:58.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4150999999999998e-05 2024-02-05T06:47:58.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.661 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.661 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:47:58.661 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.185664+0000", 2024-02-05T06:47:58.661 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7690000000000002e-06 2024-02-05T06:47:58.661 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.662 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.662 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.568350+0000", 2024-02-05T06:47:58.662 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.382685704 2024-02-05T06:47:58.662 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.662 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:47:58.662 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.568474+0000", 2024-02-05T06:47:58.662 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012467500000000001 2024-02-05T06:47:58.662 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.662 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:47:58.662 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.568613+0000", 2024-02-05T06:47:58.662 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013851099999999999 2024-02-05T06:47:58.662 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.662 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:47:58.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576163+0000", 2024-02-05T06:47:58.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0075501220000000003 2024-02-05T06:47:58.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576519+0000", 2024-02-05T06:47:58.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00035637499999999998 2024-02-05T06:47:58.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.663 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.663 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580473+0000", 2024-02-05T06:47:58.663 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0039534779999999999 2024-02-05T06:47:58.663 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.663 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:47:58.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580524+0000", 2024-02-05T06:47:58.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0788999999999998e-05 2024-02-05T06:47:58.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.664 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:47:58.664 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580560+0000", 2024-02-05T06:47:58.664 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6396000000000002e-05 2024-02-05T06:47:58.664 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.664 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:47:58.664 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.664 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.664 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.664 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:169145 8.4 8:2cf8555b:::benchmark_data_smithi028_105698_object79870:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:47:58.664 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.295920+0000", 2024-02-05T06:47:58.665 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 308.25577237499999, 2024-02-05T06:47:58.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2886338849999999, 2024-02-05T06:47:58.665 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:47:58.665 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:47:58.665 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:47:58.665 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:47:58.665 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:58.665 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169145 2024-02-05T06:47:58.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.665 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:47:58.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.665 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:47:58.665 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.295920+0000", 2024-02-05T06:47:58.665 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.665 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.665 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:47:58.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.295920+0000", 2024-02-05T06:47:58.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:47:58.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.295920+0000", 2024-02-05T06:47:58.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6700000000000003e-07 2024-02-05T06:47:58.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.666 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:47:58.666 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.295923+0000", 2024-02-05T06:47:58.666 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4540000000000001e-06 2024-02-05T06:47:58.666 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.666 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:47:58.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.295928+0000", 2024-02-05T06:47:58.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3530000000000002e-06 2024-02-05T06:47:58.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:47:58.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.295932+0000", 2024-02-05T06:47:58.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2030000000000002e-06 2024-02-05T06:47:58.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.667 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.295973+0000", 2024-02-05T06:47:58.667 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0450000000000001e-05 2024-02-05T06:47:58.667 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.667 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.667 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:47:58.668 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.295978+0000", 2024-02-05T06:47:58.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7620000000000001e-06 2024-02-05T06:47:58.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.668 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.668 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.571258+0000", 2024-02-05T06:47:58.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.27527938 2024-02-05T06:47:58.668 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.668 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.668 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:47:58.668 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.571371+0000", 2024-02-05T06:47:58.668 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113448 2024-02-05T06:47:58.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.669 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:47:58.669 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.571485+0000", 2024-02-05T06:47:58.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011325 2024-02-05T06:47:58.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.669 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:47:58.669 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576267+0000", 2024-02-05T06:47:58.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0047829350000000003 2024-02-05T06:47:58.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.669 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.669 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.669 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576587+0000", 2024-02-05T06:47:58.669 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00031938600000000001 2024-02-05T06:47:58.669 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.584465+0000", 2024-02-05T06:47:58.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078778700000000004 2024-02-05T06:47:58.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:47:58.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.584516+0000", 2024-02-05T06:47:58.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1570000000000003e-05 2024-02-05T06:47:58.670 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.670 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.670 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:47:58.670 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.584553+0000", 2024-02-05T06:47:58.670 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7156999999999997e-05 2024-02-05T06:47:58.670 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.670 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:47:58.671 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.671 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.671 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:169147 8.4 8:27ce7ef9:::benchmark_data_smithi028_105698_object79872:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:47:58.671 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.300286+0000", 2024-02-05T06:47:58.671 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 308.25140562199999, 2024-02-05T06:47:58.671 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.284371353, 2024-02-05T06:47:58.671 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:47:58.671 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:47:58.671 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:47:58.671 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:47:58.671 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:58.671 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169147 2024-02-05T06:47:58.671 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.672 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:47:58.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:47:58.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.300286+0000", 2024-02-05T06:47:58.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:47:58.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.300286+0000", 2024-02-05T06:47:58.672 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.672 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.672 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.672 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:47:58.672 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.300287+0000", 2024-02-05T06:47:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7900000000000007e-07 2024-02-05T06:47:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:47:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.300290+0000", 2024-02-05T06:47:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0970000000000002e-06 2024-02-05T06:47:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:47:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.300298+0000", 2024-02-05T06:47:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9130000000000003e-06 2024-02-05T06:47:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.673 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:47:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.300305+0000", 2024-02-05T06:47:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2069999999999998e-06 2024-02-05T06:47:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.300347+0000", 2024-02-05T06:47:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1389e-05 2024-02-05T06:47:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:47:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.300354+0000", 2024-02-05T06:47:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7549999999999997e-06 2024-02-05T06:47:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.674 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.573495+0000", 2024-02-05T06:47:58.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.273141654 2024-02-05T06:47:58.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:47:58.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.573626+0000", 2024-02-05T06:47:58.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013036499999999999 2024-02-05T06:47:58.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:47:58.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.573750+0000", 2024-02-05T06:47:58.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012482500000000001 2024-02-05T06:47:58.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.675 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.675 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:47:58.675 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576534+0000", 2024-02-05T06:47:58.675 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0027837539999999998 2024-02-05T06:47:58.675 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576637+0000", 2024-02-05T06:47:58.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010228199999999999 2024-02-05T06:47:58.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.584587+0000", 2024-02-05T06:47:58.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079500309999999998 2024-02-05T06:47:58.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.676 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.676 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:47:58.676 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.584625+0000", 2024-02-05T06:47:58.676 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8019000000000003e-05 2024-02-05T06:47:58.676 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.677 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:47:58.677 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.584658+0000", 2024-02-05T06:47:58.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3083000000000002e-05 2024-02-05T06:47:58.677 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.677 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:47:58.677 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.677 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.677 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.677 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:169148 8.4 8:250ae785:::benchmark_data_smithi028_105698_object79873:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:47:58.677 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.301004+0000", 2024-02-05T06:47:58.677 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 308.25068801600003, 2024-02-05T06:47:58.677 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.287525246, 2024-02-05T06:47:58.677 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:47:58.677 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:47:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:47:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:47:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:47:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169148 2024-02-05T06:47:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:47:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:47:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.301004+0000", 2024-02-05T06:47:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:47:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.301004+0000", 2024-02-05T06:47:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.678 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:47:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.301005+0000", 2024-02-05T06:47:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8400000000000003e-07 2024-02-05T06:47:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:47:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.301008+0000", 2024-02-05T06:47:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2969999999999999e-06 2024-02-05T06:47:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:47:58.679 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.301017+0000", 2024-02-05T06:47:58.679 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:58.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3749999999999992e-06 2024-02-05T06:47:58.680 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.680 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.680 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:47:58.680 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.301024+0000", 2024-02-05T06:47:58.680 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2940000000000002e-06 2024-02-05T06:47:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.301066+0000", 2024-02-05T06:47:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2660999999999997e-05 2024-02-05T06:47:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:47:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.301074+0000", 2024-02-05T06:47:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4139999999999997e-06 2024-02-05T06:47:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576711+0000", 2024-02-05T06:47:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2756368330000001 2024-02-05T06:47:58.681 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:47:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576823+0000", 2024-02-05T06:47:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000112048 2024-02-05T06:47:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:47:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576918+0000", 2024-02-05T06:47:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5593000000000004e-05 2024-02-05T06:47:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.583421+0000", 2024-02-05T06:47:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0065030330000000001 2024-02-05T06:47:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.682 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:47:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.583436+0000", 2024-02-05T06:47:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4219000000000001e-05 2024-02-05T06:47:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.588452+0000", 2024-02-05T06:47:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005016777 2024-02-05T06:47:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:47:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.588506+0000", 2024-02-05T06:47:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3804999999999997e-05 2024-02-05T06:47:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:47:58.683 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.588529+0000", 2024-02-05T06:47:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3013000000000001e-05 2024-02-05T06:47:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:47:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:187356 9.f 9:fb240c97:::benchmark_data_smithi028_129096_object57556:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-02-05T06:47:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:58.930290+0000", 2024-02-05T06:47:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 119.621402342, 2024-02-05T06:47:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20743583800000001, 2024-02-05T06:47:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:47:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:47:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:47:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:47:58.684 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:47:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 187356 2024-02-05T06:47:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:47:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:47:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:58.930290+0000", 2024-02-05T06:47:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:47:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:58.930290+0000", 2024-02-05T06:47:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:47:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:47:58.685 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:58.930292+0000", 2024-02-05T06:47:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.939e-06 2024-02-05T06:47:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:47:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:58.930299+0000", 2024-02-05T06:47:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9370000000000004e-06 2024-02-05T06:47:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:47:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:58.930311+0000", 2024-02-05T06:47:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1309e-05 2024-02-05T06:47:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:47:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:58.930320+0000", 2024-02-05T06:47:58.686 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7940000000000008e-06 2024-02-05T06:47:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:47:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:58.989201+0000", 2024-02-05T06:47:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.058881082000000001 2024-02-05T06:47:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:47:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:58.989296+0000", 2024-02-05T06:47:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5229000000000006e-05 2024-02-05T06:47:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:47:58.687 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:58.989384+0000", 2024-02-05T06:47:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7812999999999994e-05 2024-02-05T06:47:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:58.991255+0000", 2024-02-05T06:47:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0018709429999999999 2024-02-05T06:47:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:47:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:58.994117+0000", 2024-02-05T06:47:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028622959999999999 2024-02-05T06:47:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.688 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:47:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:59.137668+0000", 2024-02-05T06:47:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.143550663 2024-02-05T06:47:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:47:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:59.137695+0000", 2024-02-05T06:47:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7393999999999999e-05 2024-02-05T06:47:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:47:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:47:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:47:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:59.137725+0000", 2024-02-05T06:47:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0439000000000001e-05 2024-02-05T06:47:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:47:58.689 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.690 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:47:58.690 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:47:58.690 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:47:58.728 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:58.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:58.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:58.830 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:58.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:58.867 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:58.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:58.938 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 19 16 40795 40779 134.114 116.562 0.00325663 0.00745104 2024-02-05T06:47:58.968 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:47:59.006 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:59.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:59.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:47:59.108 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:59.144 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:59.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:59.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:59.246 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:59.286 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:59.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:59.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:59.389 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:47:59.426 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:59.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:59.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:47:59.528 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:59.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:47:59 smithi028 ceph-mon[29442]: pgmap v693: 158 pgs: 158 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 154 MiB/s wr, 2.46k op/s 2024-02-05T06:47:59.541 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:47:59 smithi028 ceph-mon[37984]: pgmap v693: 158 pgs: 158 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 154 MiB/s wr, 2.46k op/s 2024-02-05T06:47:59.578 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:59.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:59.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:47:59.680 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:47:59.714 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:59.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:59.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:47:59.815 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:47:59.853 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:47:59.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:47:59.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:47:59.898 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:47:59.938 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-02-05T06:47:59.931251+0000 min lat: 0.00116262 max lat: 0.237422 avg lat: 0.00751194 2024-02-05T06:47:59.938 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T06:47:59.938 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 20 15 42590 42575 133.02 112.25 0.00814085 0.00751194 2024-02-05T06:47:59.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:47:59.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:47:59 smithi167 ceph-mon[32774]: pgmap v693: 158 pgs: 158 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 154 MiB/s wr, 2.46k op/s 2024-02-05T06:47:59.999 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:47:59.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:00.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:00.100 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:00.142 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:00.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:00.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:00.245 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:00.290 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:00.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:00.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:00.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:00.418 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:00.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:00.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:00.433 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-05T06:48:00.519 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:00.562 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:00.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:00.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:00.664 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:00.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:00.708 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:00.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:00.809 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:00.865 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:00.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:00.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:00.868 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:48:00.938 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 21 16 44216 44200 131.522 101.562 0.00632811 0.00759814 2024-02-05T06:48:00.968 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:01.006 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:01.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:01.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:01.108 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:01.145 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:01.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:01.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:01.214 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.7 to 1 (10000) 2024-02-05T06:48:01.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:01.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:01.289 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:01.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:01.303 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:01 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:48:01.303 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:01 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:48:01.303 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:01 smithi167 ceph-mon[32774]: osdmap e359: 8 total, 8 up, 8 in 2024-02-05T06:48:01.304 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:01 smithi167 ceph-mon[32774]: pgmap v695: 158 pgs: 158 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 906 op/s 2024-02-05T06:48:01.304 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:01 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2541006013' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T06:48:01.304 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:01 smithi167 ceph-mon[32774]: from='client.25636 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T06:48:01.390 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:01.432 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:01.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:01.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:01.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:01 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:48:01.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:01 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:48:01.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:01 smithi028 ceph-mon[37984]: osdmap e359: 8 total, 8 up, 8 in 2024-02-05T06:48:01.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:01 smithi028 ceph-mon[37984]: pgmap v695: 158 pgs: 158 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 906 op/s 2024-02-05T06:48:01.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:01 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2541006013' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T06:48:01.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:01 smithi028 ceph-mon[37984]: from='client.25636 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T06:48:01.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:01 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:48:01.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:01 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:48:01.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:01 smithi028 ceph-mon[29442]: osdmap e359: 8 total, 8 up, 8 in 2024-02-05T06:48:01.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:01 smithi028 ceph-mon[29442]: pgmap v695: 158 pgs: 158 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 906 op/s 2024-02-05T06:48:01.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:01 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2541006013' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T06:48:01.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:01 smithi028 ceph-mon[29442]: from='client.25636 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T06:48:01.534 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:01.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:01.569 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:01.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:01.670 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:01.709 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:01.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:01.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:01.811 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:01.845 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:01.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:01.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:01.938 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 22 16 45998 45982 130.606 111.375 0.00919786 0.00765098 2024-02-05T06:48:01.948 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:01.991 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:01.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:01.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:02.093 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:02.135 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:02.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:02.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:02.237 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:02.276 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:02.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:02.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:02.342 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:02 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:48:02.342 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:02 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:48:02.342 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:02 smithi167 ceph-mon[32774]: from='client.25636 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-05T06:48:02.342 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:02 smithi167 ceph-mon[32774]: osdmap e360: 8 total, 8 up, 8 in 2024-02-05T06:48:02.377 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:02.418 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:02.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:02.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:02.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:02 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:48:02.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:02 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:48:02.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:02 smithi028 ceph-mon[29442]: from='client.25636 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-05T06:48:02.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:02 smithi028 ceph-mon[29442]: osdmap e360: 8 total, 8 up, 8 in 2024-02-05T06:48:02.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:02 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:48:02.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:02 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:48:02.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:02 smithi028 ceph-mon[37984]: from='client.25636 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-05T06:48:02.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:02 smithi028 ceph-mon[37984]: osdmap e360: 8 total, 8 up, 8 in 2024-02-05T06:48:02.520 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:02.559 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:02.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:02.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:02.661 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:02.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:02.705 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:02.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:02.759 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:48:02.800 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:48:02 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:48:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:48:02.801 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd reweight 5 1 2024-02-05T06:48:02.805 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:02.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:02.843 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:02.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:02.877 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:48:02.877 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-02-05T06:48:02.878 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.878 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.45587.0:47514 10.7 10:eb29c026:::benchmark_data_smithi028_154709_object47513:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-02-05T06:48:02.878 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:48:02.864002+0000", 2024-02-05T06:48:02.878 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0044354299999999998, 2024-02-05T06:48:02.878 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0044725750000000003, 2024-02-05T06:48:02.878 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:48:02.878 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-02-05T06:48:02.878 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:48:02.878 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.45587", 2024-02-05T06:48:02.878 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:48:02.878 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 47514 2024-02-05T06:48:02.878 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.879 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:48:02.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:48:02.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.864002+0000", 2024-02-05T06:48:02.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:02.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:48:02.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.864002+0000", 2024-02-05T06:48:02.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:02.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.879 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.879 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:48:02.879 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.864003+0000", 2024-02-05T06:48:02.879 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.341e-06 2024-02-05T06:48:02.879 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.880 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:48:02.880 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.864115+0000", 2024-02-05T06:48:02.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000112079 2024-02-05T06:48:02.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.880 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:48:02.880 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.864125+0000", 2024-02-05T06:48:02.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.798e-06 2024-02-05T06:48:02.880 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.880 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.880 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:48:02.880 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.864132+0000", 2024-02-05T06:48:02.880 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6790000000000005e-06 2024-02-05T06:48:02.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:48:02.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.864220+0000", 2024-02-05T06:48:02.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7081000000000002e-05 2024-02-05T06:48:02.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:48:02.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.864339+0000", 2024-02-05T06:48:02.881 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119874 2024-02-05T06:48:02.881 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.881 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.881 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 3,5", 2024-02-05T06:48:02.881 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.864465+0000", 2024-02-05T06:48:02.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012583000000000001 2024-02-05T06:48:02.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.882 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:48:02.882 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.868410+0000", 2024-02-05T06:48:02.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0039444670000000001 2024-02-05T06:48:02.882 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:02.882 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:48:02.882 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:02.882 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.882 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.882 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.45587.0:47517 10.0 10.934d4550 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-02-05T06:48:02.882 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:48:02.866934+0000", 2024-02-05T06:48:02.882 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.001502714, 2024-02-05T06:48:02.882 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001635678, 2024-02-05T06:48:02.883 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:48:02.883 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "queued for pg", 2024-02-05T06:48:02.883 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:48:02.883 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.45587", 2024-02-05T06:48:02.883 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:48:02.883 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 47517 2024-02-05T06:48:02.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.883 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:48:02.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.883 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:48:02.883 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.866934+0000", 2024-02-05T06:48:02.883 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:02.883 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.883 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:48:02.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.866934+0000", 2024-02-05T06:48:02.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:02.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:48:02.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.866936+0000", 2024-02-05T06:48:02.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2759999999999999e-06 2024-02-05T06:48:02.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.884 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.884 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:48:02.884 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.866968+0000", 2024-02-05T06:48:02.884 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2614000000000003e-05 2024-02-05T06:48:02.884 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:48:02.885 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.866977+0000", 2024-02-05T06:48:02.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7299999999999994e-06 2024-02-05T06:48:02.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.885 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:48:02.885 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.866984+0000", 2024-02-05T06:48:02.885 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6980000000000001e-06 2024-02-05T06:48:02.885 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:02.885 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:48:02.885 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:02.885 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.885 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.885 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.45587.0:47493 10.0 10:04f9d585:::benchmark_data_smithi028_154709_object47492:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-02-05T06:48:02.886 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:48:02.854529+0000", 2024-02-05T06:48:02.886 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.013907780999999999, 2024-02-05T06:48:02.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.014112858000000001, 2024-02-05T06:48:02.886 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:48:02.886 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-02-05T06:48:02.886 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:48:02.886 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.45587", 2024-02-05T06:48:02.886 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:48:02.886 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 47493 2024-02-05T06:48:02.886 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.886 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:48:02.886 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.886 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:48:02.886 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.854529+0000", 2024-02-05T06:48:02.886 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:02.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:48:02.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.854529+0000", 2024-02-05T06:48:02.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:02.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:48:02.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.854531+0000", 2024-02-05T06:48:02.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.697e-06 2024-02-05T06:48:02.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.887 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.887 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:48:02.887 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.854570+0000", 2024-02-05T06:48:02.887 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9538e-05 2024-02-05T06:48:02.887 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:48:02.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.854584+0000", 2024-02-05T06:48:02.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3353000000000001e-05 2024-02-05T06:48:02.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:48:02.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.854593+0000", 2024-02-05T06:48:02.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8170000000000005e-06 2024-02-05T06:48:02.888 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.888 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.888 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:48:02.888 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.857651+0000", 2024-02-05T06:48:02.888 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0030580379999999999 2024-02-05T06:48:02.889 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:48:02.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.857766+0000", 2024-02-05T06:48:02.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115462 2024-02-05T06:48:02.889 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,4", 2024-02-05T06:48:02.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.857903+0000", 2024-02-05T06:48:02.889 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013653 2024-02-05T06:48:02.889 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.889 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.889 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:02.889 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.864704+0000", 2024-02-05T06:48:02.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0068015649999999999 2024-02-05T06:48:02.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.890 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:02.890 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.864734+0000", 2024-02-05T06:48:02.890 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.974e-05 2024-02-05T06:48:02.890 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:02.890 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:48:02.890 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:02.890 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.890 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.890 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.45587.0:47495 10.a 10:5cf0c334:::benchmark_data_smithi028_154709_object47494:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-02-05T06:48:02.891 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:48:02.854632+0000", 2024-02-05T06:48:02.891 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.013805412, 2024-02-05T06:48:02.891 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.014118830000000001, 2024-02-05T06:48:02.891 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:48:02.891 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-02-05T06:48:02.891 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:48:02.891 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.45587", 2024-02-05T06:48:02.891 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:48:02.891 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 47495 2024-02-05T06:48:02.891 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.891 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:48:02.891 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:48:02.892 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.854632+0000", 2024-02-05T06:48:02.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:02.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:48:02.892 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.854632+0000", 2024-02-05T06:48:02.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:02.892 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.892 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.892 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:48:02.892 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.854633+0000", 2024-02-05T06:48:02.892 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.879999999999999e-07 2024-02-05T06:48:02.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:48:02.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.854659+0000", 2024-02-05T06:48:02.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6497000000000001e-05 2024-02-05T06:48:02.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:48:02.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.854667+0000", 2024-02-05T06:48:02.893 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6839999999999996e-06 2024-02-05T06:48:02.893 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.893 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.893 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:48:02.893 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.854673+0000", 2024-02-05T06:48:02.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4489999999999996e-06 2024-02-05T06:48:02.894 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.894 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.894 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:48:02.894 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.858166+0000", 2024-02-05T06:48:02.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003492537 2024-02-05T06:48:02.894 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.894 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.894 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:48:02.894 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.858300+0000", 2024-02-05T06:48:02.894 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013376800000000001 2024-02-05T06:48:02.894 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.894 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.894 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:48:02.895 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.858441+0000", 2024-02-05T06:48:02.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014110899999999999 2024-02-05T06:48:02.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.895 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.895 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:48:02.895 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.860609+0000", 2024-02-05T06:48:02.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021678560000000001 2024-02-05T06:48:02.895 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.895 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.895 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:02.895 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.863293+0000", 2024-02-05T06:48:02.895 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002684826 2024-02-05T06:48:02.895 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:02.895 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:48:02.896 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:02.896 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.896 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.896 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.45587.0:47506 10.0 10:077ff0d3:::benchmark_data_smithi028_154709_object47505:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-02-05T06:48:02.896 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:48:02.861393+0000", 2024-02-05T06:48:02.896 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0070441610000000002, 2024-02-05T06:48:02.896 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0074643349999999999, 2024-02-05T06:48:02.896 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:48:02.896 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-02-05T06:48:02.896 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:48:02.896 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.45587", 2024-02-05T06:48:02.896 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:48:02.896 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 47506 2024-02-05T06:48:02.897 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.897 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:48:02.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.897 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:48:02.897 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.861393+0000", 2024-02-05T06:48:02.897 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:02.897 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.897 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:48:02.897 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.861393+0000", 2024-02-05T06:48:02.897 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:02.897 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.897 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.897 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:48:02.897 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.861395+0000", 2024-02-05T06:48:02.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8789999999999999e-06 2024-02-05T06:48:02.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:48:02.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.861426+0000", 2024-02-05T06:48:02.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1476000000000002e-05 2024-02-05T06:48:02.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:48:02.898 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.861468+0000", 2024-02-05T06:48:02.898 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1641999999999999e-05 2024-02-05T06:48:02.898 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.898 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.898 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:48:02.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.861476+0000", 2024-02-05T06:48:02.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8529999999999992e-06 2024-02-05T06:48:02.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:48:02.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.864759+0000", 2024-02-05T06:48:02.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003283149 2024-02-05T06:48:02.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.899 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:48:02.899 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.864873+0000", 2024-02-05T06:48:02.899 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011417900000000001 2024-02-05T06:48:02.899 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.899 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.900 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,4", 2024-02-05T06:48:02.900 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.865031+0000", 2024-02-05T06:48:02.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015753499999999999 2024-02-05T06:48:02.900 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:02.900 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:48:02.900 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:02.900 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.900 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.900 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.45587.0:47511 10.0 10:07add4a5:::benchmark_data_smithi028_154709_object47510:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-02-05T06:48:02.900 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:48:02.862657+0000", 2024-02-05T06:48:02.900 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0057799399999999999, 2024-02-05T06:48:02.900 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0066621980000000003, 2024-02-05T06:48:02.901 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:48:02.901 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-02-05T06:48:02.901 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:48:02.901 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.45587", 2024-02-05T06:48:02.901 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:48:02.901 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 47511 2024-02-05T06:48:02.901 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.901 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:48:02.901 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.901 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:48:02.901 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.862657+0000", 2024-02-05T06:48:02.901 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:02.901 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.901 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.902 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:48:02.902 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.862657+0000", 2024-02-05T06:48:02.902 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:02.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.902 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.902 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:48:02.902 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.862658+0000", 2024-02-05T06:48:02.902 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2950000000000001e-06 2024-02-05T06:48:02.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.902 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.902 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:48:02.902 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.862698+0000", 2024-02-05T06:48:02.902 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9345999999999999e-05 2024-02-05T06:48:02.902 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.903 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:48:02.903 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.862706+0000", 2024-02-05T06:48:02.903 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2009999999999999e-06 2024-02-05T06:48:02.903 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.903 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:48:02.903 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.862712+0000", 2024-02-05T06:48:02.903 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4389999999999997e-06 2024-02-05T06:48:02.903 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.903 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.903 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:48:02.903 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.866626+0000", 2024-02-05T06:48:02.903 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.003913438 2024-02-05T06:48:02.904 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.904 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.904 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:48:02.904 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.866767+0000", 2024-02-05T06:48:02.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000140722 2024-02-05T06:48:02.904 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.904 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.904 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,4", 2024-02-05T06:48:02.904 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.866924+0000", 2024-02-05T06:48:02.904 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015755500000000001 2024-02-05T06:48:02.904 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.904 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.904 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:48:02.904 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.868943+0000", 2024-02-05T06:48:02.905 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020187590000000002 2024-02-05T06:48:02.905 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:02.905 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:48:02.905 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:02.905 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.905 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.905 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.45587.0:47512 10.d 10:b236d231:::benchmark_data_smithi028_154709_object47511:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-02-05T06:48:02.905 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:48:02.863585+0000", 2024-02-05T06:48:02.905 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0048524650000000003, 2024-02-05T06:48:02.905 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0058251120000000003, 2024-02-05T06:48:02.905 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:48:02.905 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-02-05T06:48:02.905 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:48:02.906 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.45587", 2024-02-05T06:48:02.906 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:48:02.906 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 47512 2024-02-05T06:48:02.906 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.906 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:48:02.906 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.906 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:48:02.906 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.863585+0000", 2024-02-05T06:48:02.906 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:02.906 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.906 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.906 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:48:02.906 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.863585+0000", 2024-02-05T06:48:02.906 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:02.906 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.907 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:48:02.907 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.863586+0000", 2024-02-05T06:48:02.907 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2500000000000001e-06 2024-02-05T06:48:02.907 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.907 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:48:02.907 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.863618+0000", 2024-02-05T06:48:02.907 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1924999999999997e-05 2024-02-05T06:48:02.907 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.907 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.907 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:48:02.907 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.863626+0000", 2024-02-05T06:48:02.907 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0269999999999995e-06 2024-02-05T06:48:02.907 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.908 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:48:02.908 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.863632+0000", 2024-02-05T06:48:02.908 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0769999999999996e-06 2024-02-05T06:48:02.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.908 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:48:02.908 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.863701+0000", 2024-02-05T06:48:02.908 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8799000000000003e-05 2024-02-05T06:48:02.908 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.908 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.908 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:48:02.908 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.863830+0000", 2024-02-05T06:48:02.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012914299999999999 2024-02-05T06:48:02.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,4", 2024-02-05T06:48:02.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.863977+0000", 2024-02-05T06:48:02.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014677200000000001 2024-02-05T06:48:02.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:02.909 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.866907+0000", 2024-02-05T06:48:02.909 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029307370000000001 2024-02-05T06:48:02.909 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:02.909 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:02.909 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:48:02.910 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:02.868385+0000", 2024-02-05T06:48:02.910 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0014772559999999999 2024-02-05T06:48:02.910 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:02.910 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:48:02.910 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:02.910 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:02.910 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-02-05T06:48:02.910 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 7 2024-02-05T06:48:02.910 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:48:02.939 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 23 16 47617 47601 129.323 101.188 0.0106298 0.00772737 2024-02-05T06:48:02.943 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:02.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:02.996 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:02.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:03.097 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:03.141 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:03.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:03.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:03.230 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-02-05T06:48:03.243 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:03.286 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:03.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:03.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:03.387 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:03.425 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:03.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:03.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:03.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:03 smithi028 ceph-mon[29442]: pgmap v697: 158 pgs: 158 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 141 MiB/s wr, 2.26k op/s 2024-02-05T06:48:03.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:03 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3703190358' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:48:03.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:03 smithi028 ceph-mon[37984]: pgmap v697: 158 pgs: 158 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 141 MiB/s wr, 2.26k op/s 2024-02-05T06:48:03.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:03 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3703190358' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:48:03.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:03 smithi167 ceph-mon[32774]: pgmap v697: 158 pgs: 158 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 141 MiB/s wr, 2.26k op/s 2024-02-05T06:48:03.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:03 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3703190358' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:48:03.485 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:48:03 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:48:03.527 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:03.566 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:03.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:03.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:03.667 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:03.702 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:03.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:03.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:03.770 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-05T06:48:03.804 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:03.836 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:03.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:03.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:03.938 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:03.939 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 24 16 49168 49152 127.974 96.9375 0.00457946 0.00780845 2024-02-05T06:48:03.990 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:03.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:03.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:04.092 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:04.141 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:04.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:04.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:04.146 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:48:04.243 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:04.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:04.287 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:04.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:04.388 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:04.410 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:04 smithi028 ceph-mon[29442]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:48:04.410 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:04 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:48:04.411 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:04 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3703190358' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:48:04.411 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:04 smithi028 ceph-mon[29442]: osdmap e361: 8 total, 8 up, 8 in 2024-02-05T06:48:04.411 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:04 smithi028 ceph-mon[37984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:48:04.411 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:04 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:48:04.411 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:04 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3703190358' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:48:04.411 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:04 smithi028 ceph-mon[37984]: osdmap e361: 8 total, 8 up, 8 in 2024-02-05T06:48:04.428 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:04.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:04.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:04.473 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:04 smithi167 ceph-mon[32774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:48:04.473 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:04 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:48:04.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:04 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3703190358' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:48:04.474 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:04 smithi167 ceph-mon[32774]: osdmap e361: 8 total, 8 up, 8 in 2024-02-05T06:48:04.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:04.577 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:04.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:04.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:04.679 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:04.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:04.721 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:04.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:04.822 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:04.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:04.865 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:04.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:04.939 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 25 15 50815 50800 126.974 103 0.00688178 0.00786996 2024-02-05T06:48:04.966 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:05.000 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:05.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:05.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:05.008 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:48:05.102 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:05.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:05.145 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:05.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:05.246 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:05.280 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:05.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:05.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:05.382 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:05.421 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:05.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:05.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:05.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:05 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:48:05.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:05 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:48:05.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:05 smithi028 ceph-mon[29442]: osdmap e362: 8 total, 8 up, 8 in 2024-02-05T06:48:05.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:05 smithi028 ceph-mon[29442]: pgmap v700: 158 pgs: 158 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 132 MiB/s wr, 2.12k op/s 2024-02-05T06:48:05.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:05 smithi028 ceph-mon[29442]: osdmap e363: 8 total, 8 up, 8 in 2024-02-05T06:48:05.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:05 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:48:05.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:05 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:48:05.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:05 smithi028 ceph-mon[37984]: osdmap e362: 8 total, 8 up, 8 in 2024-02-05T06:48:05.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:05 smithi028 ceph-mon[37984]: pgmap v700: 158 pgs: 158 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 132 MiB/s wr, 2.12k op/s 2024-02-05T06:48:05.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:05 smithi028 ceph-mon[37984]: osdmap e363: 8 total, 8 up, 8 in 2024-02-05T06:48:05.484 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:05 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:48:05.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:05 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:48:05.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:05 smithi167 ceph-mon[32774]: osdmap e362: 8 total, 8 up, 8 in 2024-02-05T06:48:05.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:05 smithi167 ceph-mon[32774]: pgmap v700: 158 pgs: 158 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 132 MiB/s wr, 2.12k op/s 2024-02-05T06:48:05.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:05 smithi167 ceph-mon[32774]: osdmap e363: 8 total, 8 up, 8 in 2024-02-05T06:48:05.524 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:05.562 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:05.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:05.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:05.664 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:05.704 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:05.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:05.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:05.806 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:05.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:05.840 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:05.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:05.939 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 26 16 52388 52372 125.869 98.25 0.00758509 0.00793909 2024-02-05T06:48:05.941 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:05.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:05.979 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:05.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:05.988 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:48:05.988 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:48:05.988 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:48:06.012 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:48:06.053 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:48:06.081 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:06.089 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:48:06.117 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:06.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:06.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:06.129 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:48:06.161 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:48:06.161 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:48:06.168 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:48:06.175 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:48:06.175 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:48:06.175 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:48:06.175 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:48:06.175 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:48:06.175 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:48:06.175 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:48:06.176 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:48:06.219 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:06.253 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:06.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:06.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:06.355 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:06.355 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:06 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3895432936' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:48:06.394 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:06.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:06.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:06.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:06 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3895432936' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:48:06.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:06 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3895432936' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:48:06.465 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T06:48:06.496 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:06.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:06.551 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:06.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:06.652 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:06.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:06.689 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:06.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:06.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:06.825 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:06.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:06.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:06.901 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-05T06:48:06.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:06.940 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 27 16 53857 53841 124.605 91.8125 0.00621538 0.0080204 2024-02-05T06:48:06.962 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:06.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:06.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:07.064 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:07.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:07.109 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:07.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:07.209 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:07.253 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:07.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:07.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:07.326 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:48:07.355 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:07.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:07.427 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:07.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:07.528 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:07.537 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:07 smithi167 ceph-mon[32774]: pgmap v702: 158 pgs: 158 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 973 B/s rd, 108 MiB/s wr, 1.72k op/s 2024-02-05T06:48:07.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:07.576 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:07.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:07.677 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:07.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:07 smithi028 ceph-mon[29442]: pgmap v702: 158 pgs: 158 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 973 B/s rd, 108 MiB/s wr, 1.72k op/s 2024-02-05T06:48:07.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:07 smithi028 ceph-mon[37984]: pgmap v702: 158 pgs: 158 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 973 B/s rd, 108 MiB/s wr, 1.72k op/s 2024-02-05T06:48:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:07.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:07.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:07.812 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:07.853 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:07.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:07.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:07.940 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 28 16 55246 55230 123.255 86.8125 0.0085502 0.00810843 2024-02-05T06:48:07.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:07.994 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:07.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:07.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:08.096 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:08.130 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:08.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:08.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:08.232 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:08.273 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:08.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:08.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:08.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:08.420 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:08.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:08.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:08.522 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:08.566 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:08.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:08.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:08.668 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:08.712 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:08.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:08.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:08.814 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:08.855 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:08.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:08.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:08.940 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 29 16 56801 56785 122.356 97.1875 0.0116626 0.008166 2024-02-05T06:48:08.957 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:08.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:08.995 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:08.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:09.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:09.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:09.134 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:09.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:09.150 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:48:09.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:09.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:09.277 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:09.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:09.370 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:48:09.370 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:48:09.378 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:09.421 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:09.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:09.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:09.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:09.564 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:09.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:09.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:09.620 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:09 smithi028 ceph-mon[29442]: pgmap v703: 158 pgs: 158 active+clean; 3.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 83 MiB/s wr, 1.32k op/s 2024-02-05T06:48:09.620 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:09 smithi028 ceph-mon[37984]: pgmap v703: 158 pgs: 158 active+clean; 3.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 83 MiB/s wr, 1.32k op/s 2024-02-05T06:48:09.666 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:09.703 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:09.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:09.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:09.804 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:09.841 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:09.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:09.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:09.940 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 30 15 58286 58271 121.373 92.875 0.0129355 0.00823384 2024-02-05T06:48:09.943 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:09.980 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:09.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:09.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:09.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:09 smithi167 ceph-mon[32774]: pgmap v703: 158 pgs: 158 active+clean; 3.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 83 MiB/s wr, 1.32k op/s 2024-02-05T06:48:10.082 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:10.119 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:10.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:10.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:10.150 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-05T06:48:10.221 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:10.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:48:10.260 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:10.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:10.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:10.363 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:10.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:10.407 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:10.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:10.508 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:10.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:10.553 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:10.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:10.564 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.5 to 1 (10000) 2024-02-05T06:48:10.590 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:48:10.654 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:10.689 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:10.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:10.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:10.792 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:10.833 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:10.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:10.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:10.937 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:10.943 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 31 15 59720 59705 120.349 89.625 0.0197191 0.00830346 2024-02-05T06:48:10.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:10 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/226470919' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T06:48:10.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:10 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/226470919' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T06:48:10.976 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:10.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:10.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:10.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:10 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/226470919' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T06:48:11.078 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:11.116 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:11.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:11.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:11.218 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:11.254 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:11.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:11.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:11.356 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:11.381 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:11.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:11.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:11.466 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-05T06:48:11.466 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T06:48:11.483 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:11.525 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:48:11.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:11.575 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:11.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:11.676 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:11.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:11 smithi028 ceph-mon[29442]: pgmap v704: 158 pgs: 158 active+clean; 3.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 706 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-02-05T06:48:11.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:11 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/226470919' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-05T06:48:11.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:11 smithi028 ceph-mon[29442]: osdmap e364: 8 total, 8 up, 8 in 2024-02-05T06:48:11.677 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:11 smithi028 ceph-mon[37984]: pgmap v704: 158 pgs: 158 active+clean; 3.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 706 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-02-05T06:48:11.677 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:11 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/226470919' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-05T06:48:11.677 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:11 smithi028 ceph-mon[37984]: osdmap e364: 8 total, 8 up, 8 in 2024-02-05T06:48:11.701 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:11.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:11.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:11.803 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:11.833 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:11.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:11.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:11.853 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:11 smithi167 ceph-mon[32774]: pgmap v704: 158 pgs: 158 active+clean; 3.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 706 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-02-05T06:48:11.853 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:11 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/226470919' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-05T06:48:11.853 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:11 smithi167 ceph-mon[32774]: osdmap e364: 8 total, 8 up, 8 in 2024-02-05T06:48:11.935 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:11.941 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 32 16 60943 60927 118.974 76.375 0.00669078 0.0084001 2024-02-05T06:48:11.972 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:11.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:11.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:12.074 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:12.113 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:12.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:12.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:12.214 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:12.254 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:12.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:12.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:12.356 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:12.391 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:12.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:12.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:12.493 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:12.535 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:12.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:12.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:12.639 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:12.648 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:12 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:48:12.678 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:12.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:12.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:12.780 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:12.824 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:12.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:12.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:12.846 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:48:12.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:12.942 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 33 15 62354 62339 118.037 88.25 0.00753767 0.00846703 2024-02-05T06:48:12.950 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:48:12 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:48:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:48:12.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:12 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:48:12.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:12 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:48:12.974 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:12.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:12.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:13.076 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:13.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:13.122 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:13.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:13.223 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:13.264 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:13.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:13.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:13.366 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:13.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:13.408 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:13.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:13.485 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:48:13 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:48:13.509 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:13.554 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:13.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:13.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:13.555 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:48:13.555 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:48:13.579 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-02-05T06:48:13.656 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:13.686 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:13.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:13.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:13.788 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:13.821 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:13.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:13.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:13.910 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:13 smithi167 ceph-mon[32774]: pgmap v706: 158 pgs: 158 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 118 MiB/s wr, 1.89k op/s 2024-02-05T06:48:13.910 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:13 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/462735475' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:48:13.922 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:13.942 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 34 16 63721 63705 117.076 85.375 0.0103524 0.00853642 2024-02-05T06:48:13.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:13 smithi028 ceph-mon[29442]: pgmap v706: 158 pgs: 158 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 118 MiB/s wr, 1.89k op/s 2024-02-05T06:48:13.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:13 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/462735475' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:48:13.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:13 smithi028 ceph-mon[37984]: pgmap v706: 158 pgs: 158 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 118 MiB/s wr, 1.89k op/s 2024-02-05T06:48:13.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:13 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/462735475' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:48:13.959 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:13.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:13.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:14.060 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:14.098 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:14.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:14.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:14.200 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:14.224 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:14.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:14.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:14.244 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-05T06:48:14.326 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:14.359 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:14.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:14.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:14.461 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:14.497 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:14.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:14.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:14.599 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:14.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:14.664 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:14.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:14.679 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:48:14.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:14.803 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:14.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:14.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:14.818 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:14 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:48:14.818 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:14 smithi167 ceph-mon[32774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:48:14.818 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:14 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/462735475' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:48:14.818 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:14 smithi167 ceph-mon[32774]: osdmap e365: 8 total, 8 up, 8 in 2024-02-05T06:48:14.818 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:14 smithi167 ceph-mon[32774]: 5.1c scrub starts 2024-02-05T06:48:14.818 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:14 smithi167 ceph-mon[32774]: 5.1c scrub ok 2024-02-05T06:48:14.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:14.943 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 35 16 65001 64985 116.015 80 0.0159766 0.00861453 2024-02-05T06:48:14.946 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:14.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:14.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:14.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:14 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:48:14.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:14 smithi028 ceph-mon[29442]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:48:14.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:14 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/462735475' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:48:14.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:14 smithi028 ceph-mon[29442]: osdmap e365: 8 total, 8 up, 8 in 2024-02-05T06:48:14.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:14 smithi028 ceph-mon[29442]: 5.1c scrub starts 2024-02-05T06:48:14.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:14 smithi028 ceph-mon[29442]: 5.1c scrub ok 2024-02-05T06:48:14.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:14 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:48:14.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:14 smithi028 ceph-mon[37984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:48:14.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:14 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/462735475' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:48:14.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:14 smithi028 ceph-mon[37984]: osdmap e365: 8 total, 8 up, 8 in 2024-02-05T06:48:14.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:14 smithi028 ceph-mon[37984]: 5.1c scrub starts 2024-02-05T06:48:14.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:14 smithi028 ceph-mon[37984]: 5.1c scrub ok 2024-02-05T06:48:15.049 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:15.086 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:15.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:15.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:15.188 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:15.218 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:15.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:15.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:15.320 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:15.356 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:15.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:15.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:15.458 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:15.491 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:15.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:15.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:15.593 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:15.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:15.634 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:15.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:15.736 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:15.775 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:15.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:15.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:15.833 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:15 smithi028 ceph-mon[29442]: pgmap v708: 158 pgs: 158 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 118 MiB/s wr, 1.89k op/s 2024-02-05T06:48:15.833 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:15 smithi028 ceph-mon[29442]: 2.2 scrub starts 2024-02-05T06:48:15.833 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:15 smithi028 ceph-mon[29442]: 2.2 scrub ok 2024-02-05T06:48:15.833 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:15 smithi028 ceph-mon[29442]: 3.8 scrub starts 2024-02-05T06:48:15.833 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:15 smithi028 ceph-mon[29442]: 3.8 scrub ok 2024-02-05T06:48:15.833 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:15 smithi028 ceph-mon[29442]: 3.1a scrub starts 2024-02-05T06:48:15.834 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:15 smithi028 ceph-mon[29442]: 3.1a scrub ok 2024-02-05T06:48:15.834 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:15 smithi028 ceph-mon[37984]: pgmap v708: 158 pgs: 158 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 118 MiB/s wr, 1.89k op/s 2024-02-05T06:48:15.834 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:15 smithi028 ceph-mon[37984]: 2.2 scrub starts 2024-02-05T06:48:15.834 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:15 smithi028 ceph-mon[37984]: 2.2 scrub ok 2024-02-05T06:48:15.834 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:15 smithi028 ceph-mon[37984]: 3.8 scrub starts 2024-02-05T06:48:15.834 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:15 smithi028 ceph-mon[37984]: 3.8 scrub ok 2024-02-05T06:48:15.834 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:15 smithi028 ceph-mon[37984]: 3.1a scrub starts 2024-02-05T06:48:15.834 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:15 smithi028 ceph-mon[37984]: 3.1a scrub ok 2024-02-05T06:48:15.877 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:15.916 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:15.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:15.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:15.943 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 36 16 66236 66220 114.936 77.1875 0.0142571 0.00869504 2024-02-05T06:48:15.948 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:15 smithi167 ceph-mon[32774]: pgmap v708: 158 pgs: 158 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 118 MiB/s wr, 1.89k op/s 2024-02-05T06:48:15.949 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:15 smithi167 ceph-mon[32774]: 2.2 scrub starts 2024-02-05T06:48:15.949 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:15 smithi167 ceph-mon[32774]: 2.2 scrub ok 2024-02-05T06:48:15.949 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:15 smithi167 ceph-mon[32774]: 3.8 scrub starts 2024-02-05T06:48:15.949 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:15 smithi167 ceph-mon[32774]: 3.8 scrub ok 2024-02-05T06:48:15.949 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:15 smithi167 ceph-mon[32774]: 3.1a scrub starts 2024-02-05T06:48:15.949 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:15 smithi167 ceph-mon[32774]: 3.1a scrub ok 2024-02-05T06:48:16.017 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:16.053 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:16.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:16.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:16.154 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:16.193 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:16.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:16.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:16.295 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:16.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:16.332 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:16.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:16.433 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:16.468 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:16.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:16.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:16.569 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:16.609 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:16.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:16.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:16.711 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:16.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:16.749 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:16.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:16.850 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:16.851 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:16 smithi028 ceph-mon[29442]: 6.1e scrub starts 2024-02-05T06:48:16.851 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:16 smithi028 ceph-mon[29442]: 6.1e scrub ok 2024-02-05T06:48:16.851 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:16 smithi028 ceph-mon[29442]: 4.4 scrub starts 2024-02-05T06:48:16.851 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:16 smithi028 ceph-mon[29442]: 4.4 scrub ok 2024-02-05T06:48:16.851 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:16 smithi028 ceph-mon[37984]: 6.1e scrub starts 2024-02-05T06:48:16.852 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:16 smithi028 ceph-mon[37984]: 6.1e scrub ok 2024-02-05T06:48:16.852 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:16 smithi028 ceph-mon[37984]: 4.4 scrub starts 2024-02-05T06:48:16.852 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:16 smithi028 ceph-mon[37984]: 4.4 scrub ok 2024-02-05T06:48:16.892 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:16.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:16.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:16.943 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 37 16 67576 67560 114.093 83.75 0.0053287 0.00875985 2024-02-05T06:48:16.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:16 smithi167 ceph-mon[32774]: 6.1e scrub starts 2024-02-05T06:48:16.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:16 smithi167 ceph-mon[32774]: 6.1e scrub ok 2024-02-05T06:48:16.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:16 smithi167 ceph-mon[32774]: 4.4 scrub starts 2024-02-05T06:48:16.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:16 smithi167 ceph-mon[32774]: 4.4 scrub ok 2024-02-05T06:48:16.994 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:17.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:17.036 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:17.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:17.137 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:17.177 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:17.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:17.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:17.279 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:17.318 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:17.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:17.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:17.420 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:17.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:17.449 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:17.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:17.550 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:17.583 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:17.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:17.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:17.685 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:17.720 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:17.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:17.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:17.822 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:17.844 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:17.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:17.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:17.943 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 38 16 68814 68798 113.127 77.375 0.00980798 0.00883441 2024-02-05T06:48:17.946 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:17.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:17 smithi028 ceph-mon[29442]: pgmap v709: 158 pgs: 158 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 56 MiB/s wr, 894 op/s 2024-02-05T06:48:17.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:17 smithi028 ceph-mon[29442]: 3.3 scrub starts 2024-02-05T06:48:17.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:17 smithi028 ceph-mon[29442]: 3.3 scrub ok 2024-02-05T06:48:17.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:17 smithi028 ceph-mon[37984]: pgmap v709: 158 pgs: 158 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 56 MiB/s wr, 894 op/s 2024-02-05T06:48:17.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:17 smithi028 ceph-mon[37984]: 3.3 scrub starts 2024-02-05T06:48:17.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:17 smithi028 ceph-mon[37984]: 3.3 scrub ok 2024-02-05T06:48:17.984 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:17.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:17 smithi167 ceph-mon[32774]: pgmap v709: 158 pgs: 158 active+clean; 3.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 56 MiB/s wr, 894 op/s 2024-02-05T06:48:17.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:17 smithi167 ceph-mon[32774]: 3.3 scrub starts 2024-02-05T06:48:17.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:17 smithi167 ceph-mon[32774]: 3.3 scrub ok 2024-02-05T06:48:17.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:17.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:18.086 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:18.109 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:18.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:18.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:18.211 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:18.248 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:18.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:18.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:18.350 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:18.376 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:18.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:18.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:18.478 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:18.516 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:18.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:18.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:18.618 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:18.662 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:18.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:18.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:18.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:18 smithi028 ceph-mon[37984]: 4.1e scrub starts 2024-02-05T06:48:18.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:18 smithi028 ceph-mon[37984]: 4.1e scrub ok 2024-02-05T06:48:18.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:18 smithi028 ceph-mon[37984]: 4.1 scrub starts 2024-02-05T06:48:18.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:18 smithi028 ceph-mon[37984]: 4.1 scrub ok 2024-02-05T06:48:18.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:18 smithi028 ceph-mon[37984]: 4.1a scrub starts 2024-02-05T06:48:18.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:18 smithi028 ceph-mon[37984]: 4.1a scrub ok 2024-02-05T06:48:18.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:18 smithi028 ceph-mon[37984]: 1.9 scrub starts 2024-02-05T06:48:18.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:18 smithi028 ceph-mon[37984]: 1.9 scrub ok 2024-02-05T06:48:18.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:18 smithi028 ceph-mon[37984]: 1.6 scrub starts 2024-02-05T06:48:18.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:18 smithi028 ceph-mon[37984]: 1.6 scrub ok 2024-02-05T06:48:18.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:18 smithi028 ceph-mon[29442]: 4.1e scrub starts 2024-02-05T06:48:18.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:18 smithi028 ceph-mon[29442]: 4.1e scrub ok 2024-02-05T06:48:18.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:18 smithi028 ceph-mon[29442]: 4.1 scrub starts 2024-02-05T06:48:18.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:18 smithi028 ceph-mon[29442]: 4.1 scrub ok 2024-02-05T06:48:18.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:18 smithi028 ceph-mon[29442]: 4.1a scrub starts 2024-02-05T06:48:18.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:18 smithi028 ceph-mon[29442]: 4.1a scrub ok 2024-02-05T06:48:18.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:18 smithi028 ceph-mon[29442]: 1.9 scrub starts 2024-02-05T06:48:18.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:18 smithi028 ceph-mon[29442]: 1.9 scrub ok 2024-02-05T06:48:18.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:18 smithi028 ceph-mon[29442]: 1.6 scrub starts 2024-02-05T06:48:18.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:18 smithi028 ceph-mon[29442]: 1.6 scrub ok 2024-02-05T06:48:18.722 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:48:18.756 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:48:18.763 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:18.795 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:48:18.808 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:18.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:18.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:18.841 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:48:18.853 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T06:48:18.881 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:48:18.910 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:18.926 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:48:18.944 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 39 16 69777 69761 111.768 60.1875 0.0159739 0.00894187 2024-02-05T06:48:18.955 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:18.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:18.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:18.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:18 smithi167 ceph-mon[32774]: 4.1e scrub starts 2024-02-05T06:48:18.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:18 smithi167 ceph-mon[32774]: 4.1e scrub ok 2024-02-05T06:48:18.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:18 smithi167 ceph-mon[32774]: 4.1 scrub starts 2024-02-05T06:48:18.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:18 smithi167 ceph-mon[32774]: 4.1 scrub ok 2024-02-05T06:48:18.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:18 smithi167 ceph-mon[32774]: 4.1a scrub starts 2024-02-05T06:48:18.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:18 smithi167 ceph-mon[32774]: 4.1a scrub ok 2024-02-05T06:48:18.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:18 smithi167 ceph-mon[32774]: 1.9 scrub starts 2024-02-05T06:48:18.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:18 smithi167 ceph-mon[32774]: 1.9 scrub ok 2024-02-05T06:48:18.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:18 smithi167 ceph-mon[32774]: 1.6 scrub starts 2024-02-05T06:48:18.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:18 smithi167 ceph-mon[32774]: 1.6 scrub ok 2024-02-05T06:48:18.995 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:48:19.041 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:48:19.044 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:48:19.044 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:48:19.044 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:48:19.044 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:48:19.044 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:48:19.044 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:48:19.044 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:48:19.044 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:48:19.058 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:19.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:19.081 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:19.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:19.182 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:19.205 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:19.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:19.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:19.307 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:19.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:19.336 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:19.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:19.437 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:19.471 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:19.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:19.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:19.572 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:19.604 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:48:19.608 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-05T06:48:19.616 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:19.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:19.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:19.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:19.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:19.757 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:19.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:19.858 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:19.883 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:19.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:19.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:19.944 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-02-05T06:48:19.937351+0000 min lat: 0.00116262 max lat: 0.237422 avg lat: 0.00903485 2024-02-05T06:48:19.944 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T06:48:19.945 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 40 16 70833 70817 110.624 66 0.00775047 0.00903485 2024-02-05T06:48:19.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:19 smithi028 ceph-mon[29442]: pgmap v710: 158 pgs: 158 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.72k op/s 2024-02-05T06:48:19.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:19 smithi028 ceph-mon[29442]: 5.14 scrub starts 2024-02-05T06:48:19.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:19 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/331730711' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:48:19.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:19 smithi028 ceph-mon[29442]: 5.14 scrub ok 2024-02-05T06:48:19.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:19 smithi028 ceph-mon[29442]: 5.18 scrub starts 2024-02-05T06:48:19.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:19 smithi028 ceph-mon[29442]: 5.18 scrub ok 2024-02-05T06:48:19.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:19 smithi028 ceph-mon[29442]: 5.19 scrub starts 2024-02-05T06:48:19.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:19 smithi028 ceph-mon[29442]: 5.19 scrub ok 2024-02-05T06:48:19.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:19 smithi028 ceph-mon[37984]: pgmap v710: 158 pgs: 158 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.72k op/s 2024-02-05T06:48:19.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:19 smithi028 ceph-mon[37984]: 5.14 scrub starts 2024-02-05T06:48:19.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:19 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/331730711' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:48:19.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:19 smithi028 ceph-mon[37984]: 5.14 scrub ok 2024-02-05T06:48:19.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:19 smithi028 ceph-mon[37984]: 5.18 scrub starts 2024-02-05T06:48:19.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:19 smithi028 ceph-mon[37984]: 5.18 scrub ok 2024-02-05T06:48:19.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:19 smithi028 ceph-mon[37984]: 5.19 scrub starts 2024-02-05T06:48:19.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:19 smithi028 ceph-mon[37984]: 5.19 scrub ok 2024-02-05T06:48:19.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:19 smithi167 ceph-mon[32774]: pgmap v710: 158 pgs: 158 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.72k op/s 2024-02-05T06:48:19.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:19 smithi167 ceph-mon[32774]: 5.14 scrub starts 2024-02-05T06:48:19.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:19 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/331730711' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:48:19.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:19 smithi167 ceph-mon[32774]: 5.14 scrub ok 2024-02-05T06:48:19.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:19 smithi167 ceph-mon[32774]: 5.18 scrub starts 2024-02-05T06:48:19.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:19 smithi167 ceph-mon[32774]: 5.18 scrub ok 2024-02-05T06:48:19.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:19 smithi167 ceph-mon[32774]: 5.19 scrub starts 2024-02-05T06:48:19.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:19 smithi167 ceph-mon[32774]: 5.19 scrub ok 2024-02-05T06:48:19.985 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:20.018 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:20.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:20.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:20.120 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:20.159 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:20.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:20.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:20.260 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:20.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:20.310 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:20.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:20.410 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:20.437 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:20.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:20.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:20.538 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:20.574 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:20.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:20.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:20.676 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:20.709 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:20.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:20.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:20.811 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:20.834 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:20.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:20.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:20.914 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:20 smithi028 ceph-mon[37984]: 4.18 scrub starts 2024-02-05T06:48:20.914 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:20 smithi028 ceph-mon[37984]: 4.18 scrub ok 2024-02-05T06:48:20.914 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:20 smithi028 ceph-mon[37984]: from='client.15771 v1:172.21.15.28:0/1132243188' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:48:20.915 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:20 smithi028 ceph-mon[37984]: 5.8 scrub starts 2024-02-05T06:48:20.915 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:20 smithi028 ceph-mon[37984]: 5.8 scrub ok 2024-02-05T06:48:20.915 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:20 smithi028 ceph-mon[29442]: 4.18 scrub starts 2024-02-05T06:48:20.915 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:20 smithi028 ceph-mon[29442]: 4.18 scrub ok 2024-02-05T06:48:20.915 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:20 smithi028 ceph-mon[29442]: from='client.15771 v1:172.21.15.28:0/1132243188' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:48:20.915 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:20 smithi028 ceph-mon[29442]: 5.8 scrub starts 2024-02-05T06:48:20.915 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:20 smithi028 ceph-mon[29442]: 5.8 scrub ok 2024-02-05T06:48:20.935 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:20.945 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 41 16 71947 71931 109.623 69.625 0.0126772 0.00911623 2024-02-05T06:48:20.963 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:48:20.965 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":710,"stamp":"2024-02-05T06:48:18.461435+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4410513756,"num_objects":67488,"num_object_clones":0,"num_object_copies":202464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67488,"num_whiteouts":0,"num_read":1595,"num_read_kb":1347,"num_write":68012,"num_write_kb":4306605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":415,"num_bytes_recovered":1586799,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69584,"ondisk_log_size":69584,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":16996480,"kb_used_data":12839036,"kb_used_omap":12,"kb_used_meta":4157363,"kb_avail":732964736,"statfs":{"total":767960285184,"available":750555889664,"internally_reserved":0,"allocated":13147172864,"data_stored":13143121316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":4257140273},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":901251072,"num_objects":13752,"num_object_clones":0,"num_object_copies":41256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13752,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":13752,"num_write_kb":880128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002114"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":407,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.571410+0000","last_change":"2024-02-05T06:47:44.624372+0000","last_active":"2024-02-05T06:48:13.571410+0000","last_peered":"2024-02-05T06:48:13.571410+0000","last_clean":"2024-02-05T06:48:13.571410+0000","last_became_active":"2024-02-05T06:45:09.787134+0000","last_became_peered":"2024-02-05T06:45:09.787134+0000","last_unstale":"2024-02-05T06:48:13.571410+0000","last_undegraded":"2024-02-05T06:48:13.571410+0000","last_fullsized":"2024-02-05T06:48:13.571410+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:44.624254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:47:44.624254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:58.304608+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":670,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.568901+0000","last_change":"2024-02-05T06:45:44.761855+0000","last_active":"2024-02-05T06:48:13.568901+0000","last_peered":"2024-02-05T06:48:13.568901+0000","last_clean":"2024-02-05T06:48:13.568901+0000","last_became_active":"2024-02-05T06:41:43.420342+0000","last_became_peered":"2024-02-05T06:41:43.420342+0000","last_unstale":"2024-02-05T06:48:13.568901+0000","last_undegraded":"2024-02-05T06:48:13.568901+0000","last_fullsized":"2024-02-05T06:48:13.568901+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:44.761725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:06.525012+0000","last_clean_scrub_stamp":"2024-02-05T06:45:44.761725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:15.075031+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":660,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.576953+0000","last_change":"2024-02-05T06:45:42.923556+0000","last_active":"2024-02-05T06:48:13.576953+0000","last_peered":"2024-02-05T06:48:13.576953+0000","last_clean":"2024-02-05T06:48:13.576953+0000","last_became_active":"2024-02-05T06:41:43.421315+0000","last_became_peered":"2024-02-05T06:41:43.421315+0000","last_unstale":"2024-02-05T06:48:13.576953+0000","last_undegraded":"2024-02-05T06:48:13.576953+0000","last_fullsized":"2024-02-05T06:48:13.576953+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:42.923435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:45:42.923435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.496465+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"295'19","reported_seq":570,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.573278+0000","last_change":"2024-02-05T06:46:53.701820+0000","last_active":"2024-02-05T06:48:13.573278+0000","last_peered":"2024-02-05T06:48:13.573278+0000","last_clean":"2024-02-05T06:48:13.573278+0000","last_became_active":"2024-02-05T06:36:54.959330+0000","last_became_peered":"2024-02-05T06:36:54.959330+0000","last_unstale":"2024-02-05T06:48:13.573278+0000","last_undegraded":"2024-02-05T06:48:13.573278+0000","last_fullsized":"2024-02-05T06:48:13.573278+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'19","last_scrub_stamp":"2024-02-05T06:46:53.701692+0000","last_deep_scrub":"247'13","last_deep_scrub_stamp":"2024-02-05T06:41:42.473424+0000","last_clean_scrub_stamp":"2024-02-05T06:46:53.701692+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":645,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.566510+0000","last_change":"2024-02-05T06:45:45.252029+0000","last_active":"2024-02-05T06:48:13.566510+0000","last_peered":"2024-02-05T06:48:13.566510+0000","last_clean":"2024-02-05T06:48:13.566510+0000","last_became_active":"2024-02-05T06:40:27.077485+0000","last_became_peered":"2024-02-05T06:40:27.077485+0000","last_unstale":"2024-02-05T06:48:13.566510+0000","last_undegraded":"2024-02-05T06:48:13.566510+0000","last_fullsized":"2024-02-05T06:48:13.566510+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:45.251895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:45:45.251895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.572678+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":168,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.564257+0000","last_change":"2024-02-05T06:46:51.388272+0000","last_active":"2024-02-05T06:48:13.564257+0000","last_peered":"2024-02-05T06:48:13.564257+0000","last_clean":"2024-02-05T06:48:13.564257+0000","last_became_active":"2024-02-05T06:45:09.775037+0000","last_became_peered":"2024-02-05T06:45:09.775037+0000","last_unstale":"2024-02-05T06:48:13.564257+0000","last_undegraded":"2024-02-05T06:48:13.564257+0000","last_fullsized":"2024-02-05T06:48:13.564257+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:51.388156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:51.388156+0000","last_clean_scrub_stamp":"2024-02-05T06:46:51.388156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:13.711795+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":655,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.565530+0000","last_change":"2024-02-05T06:47:34.440851+0000","last_active":"2024-02-05T06:48:13.565530+0000","last_peered":"2024-02-05T06:48:13.565530+0000","last_clean":"2024-02-05T06:48:13.565530+0000","last_became_active":"2024-02-05T06:33:29.553864+0000","last_became_peered":"2024-02-05T06:33:29.553864+0000","last_unstale":"2024-02-05T06:48:13.565530+0000","last_undegraded":"2024-02-05T06:48:13.565530+0000","last_fullsized":"2024-02-05T06:48:13.565530+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:34.440773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:47:34.440773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:40.674347+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"295'34","reported_seq":700,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.566042+0000","last_change":"2024-02-05T06:47:48.407121+0000","last_active":"2024-02-05T06:48:13.566042+0000","last_peered":"2024-02-05T06:48:13.566042+0000","last_clean":"2024-02-05T06:48:13.566042+0000","last_became_active":"2024-02-05T06:33:31.572544+0000","last_became_peered":"2024-02-05T06:33:31.572544+0000","last_unstale":"2024-02-05T06:48:13.566042+0000","last_undegraded":"2024-02-05T06:48:13.566042+0000","last_fullsized":"2024-02-05T06:48:13.566042+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'34","last_scrub_stamp":"2024-02-05T06:47:48.406976+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:39:35.433539+0000","last_clean_scrub_stamp":"2024-02-05T06:47:48.406976+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:12.788812+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":655,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.574379+0000","last_change":"2024-02-05T06:46:58.507850+0000","last_active":"2024-02-05T06:48:13.574379+0000","last_peered":"2024-02-05T06:48:13.574379+0000","last_clean":"2024-02-05T06:48:13.574379+0000","last_became_active":"2024-02-05T06:40:27.068888+0000","last_became_peered":"2024-02-05T06:40:27.068888+0000","last_unstale":"2024-02-05T06:48:13.574379+0000","last_undegraded":"2024-02-05T06:48:13.574379+0000","last_fullsized":"2024-02-05T06:48:13.574379+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:58.507407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:44:55.125921+0000","last_clean_scrub_stamp":"2024-02-05T06:46:58.507407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.221600+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":211,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.566601+0000","last_change":"2024-02-05T06:45:51.218029+0000","last_active":"2024-02-05T06:48:13.566601+0000","last_peered":"2024-02-05T06:48:13.566601+0000","last_clean":"2024-02-05T06:48:13.566601+0000","last_became_active":"2024-02-05T06:42:33.129959+0000","last_became_peered":"2024-02-05T06:42:33.129959+0000","last_unstale":"2024-02-05T06:48:13.566601+0000","last_undegraded":"2024-02-05T06:48:13.566601+0000","last_fullsized":"2024-02-05T06:48:13.566601+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:51.217900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:45:51.217900+0000","last_clean_scrub_stamp":"2024-02-05T06:45:51.217900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.572756+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":262,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.570442+0000","last_change":"2024-02-05T06:46:46.643447+0000","last_active":"2024-02-05T06:48:13.570442+0000","last_peered":"2024-02-05T06:48:13.570442+0000","last_clean":"2024-02-05T06:48:13.570442+0000","last_became_active":"2024-02-05T06:41:43.403746+0000","last_became_peered":"2024-02-05T06:41:43.403746+0000","last_unstale":"2024-02-05T06:48:13.570442+0000","last_undegraded":"2024-02-05T06:48:13.570442+0000","last_fullsized":"2024-02-05T06:48:13.570442+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:46.643336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:44.081318+0000","last_clean_scrub_stamp":"2024-02-05T06:46:46.643336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"295'19","reported_seq":503,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.565311+0000","last_change":"2024-02-05T06:45:09.788862+0000","last_active":"2024-02-05T06:48:13.565311+0000","last_peered":"2024-02-05T06:48:13.565311+0000","last_clean":"2024-02-05T06:48:13.565311+0000","last_became_active":"2024-02-05T06:45:09.787626+0000","last_became_peered":"2024-02-05T06:45:09.787626+0000","last_unstale":"2024-02-05T06:48:13.565311+0000","last_undegraded":"2024-02-05T06:48:13.565311+0000","last_fullsized":"2024-02-05T06:48:13.565311+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'19","last_scrub_stamp":"2024-02-05T06:43:49.843581+0000","last_deep_scrub":"92'13","last_deep_scrub_stamp":"2024-02-05T06:34:46.542893+0000","last_clean_scrub_stamp":"2024-02-05T06:43:49.843581+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.572453+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":262,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.574174+0000","last_change":"2024-02-05T06:45:57.292808+0000","last_active":"2024-02-05T06:48:13.574174+0000","last_peered":"2024-02-05T06:48:13.574174+0000","last_clean":"2024-02-05T06:48:13.574174+0000","last_became_active":"2024-02-05T06:41:43.407404+0000","last_became_peered":"2024-02-05T06:41:43.407404+0000","last_unstale":"2024-02-05T06:48:13.574174+0000","last_undegraded":"2024-02-05T06:48:13.574174+0000","last_fullsized":"2024-02-05T06:48:13.574174+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:57.292608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:21.998850+0000","last_clean_scrub_stamp":"2024-02-05T06:45:57.292608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":363,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.577692+0000","last_change":"2024-02-05T06:47:51.640890+0000","last_active":"2024-02-05T06:48:13.577692+0000","last_peered":"2024-02-05T06:48:13.577692+0000","last_clean":"2024-02-05T06:48:13.577692+0000","last_became_active":"2024-02-05T06:40:27.071758+0000","last_became_peered":"2024-02-05T06:40:27.071758+0000","last_unstale":"2024-02-05T06:48:13.577692+0000","last_undegraded":"2024-02-05T06:48:13.577692+0000","last_fullsized":"2024-02-05T06:48:13.577692+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:51.640785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:47:51.640785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:06.140529+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":270,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.570193+0000","last_change":"2024-02-05T06:47:39.612949+0000","last_active":"2024-02-05T06:48:13.570193+0000","last_peered":"2024-02-05T06:48:13.570193+0000","last_clean":"2024-02-05T06:48:13.570193+0000","last_became_active":"2024-02-05T06:41:43.408428+0000","last_became_peered":"2024-02-05T06:41:43.408428+0000","last_unstale":"2024-02-05T06:48:13.570193+0000","last_undegraded":"2024-02-05T06:48:13.570193+0000","last_fullsized":"2024-02-05T06:48:13.570193+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:39.612865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:50.055245+0000","last_clean_scrub_stamp":"2024-02-05T06:47:39.612865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:46.400154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"295'7","reported_seq":493,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.565001+0000","last_change":"2024-02-05T06:45:09.809100+0000","last_active":"2024-02-05T06:48:13.565001+0000","last_peered":"2024-02-05T06:48:13.565001+0000","last_clean":"2024-02-05T06:48:13.565001+0000","last_became_active":"2024-02-05T06:45:09.808663+0000","last_became_peered":"2024-02-05T06:45:09.808663+0000","last_unstale":"2024-02-05T06:48:13.565001+0000","last_undegraded":"2024-02-05T06:48:13.565001+0000","last_fullsized":"2024-02-05T06:48:13.565001+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'7","last_scrub_stamp":"2024-02-05T06:44:55.988817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:44:55.988817+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.572574+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":8,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":411,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:15.743559+0000","last_change":"2024-02-05T06:48:15.743595+0000","last_active":"2024-02-05T06:48:15.743559+0000","last_peered":"2024-02-05T06:48:15.743559+0000","last_clean":"2024-02-05T06:48:15.743559+0000","last_became_active":"2024-02-05T06:45:09.770902+0000","last_became_peered":"2024-02-05T06:45:09.770902+0000","last_unstale":"2024-02-05T06:48:15.743559+0000","last_undegraded":"2024-02-05T06:48:15.743559+0000","last_fullsized":"2024-02-05T06:48:15.743559+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:15.743307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:48:15.743307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:32.352762+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":657,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.569559+0000","last_change":"2024-02-05T06:45:39.848063+0000","last_active":"2024-02-05T06:48:13.569559+0000","last_peered":"2024-02-05T06:48:13.569559+0000","last_clean":"2024-02-05T06:48:13.569559+0000","last_became_active":"2024-02-05T06:36:54.958069+0000","last_became_peered":"2024-02-05T06:36:54.958069+0000","last_unstale":"2024-02-05T06:48:13.569559+0000","last_undegraded":"2024-02-05T06:48:13.569559+0000","last_fullsized":"2024-02-05T06:48:13.569559+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:39.847950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:45:39.847950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:15.075089+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"281'2","reported_seq":701,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.578621+0000","last_change":"2024-02-05T06:47:50.442268+0000","last_active":"2024-02-05T06:48:13.578621+0000","last_peered":"2024-02-05T06:48:13.578621+0000","last_clean":"2024-02-05T06:48:13.578621+0000","last_became_active":"2024-02-05T06:40:27.073360+0000","last_became_peered":"2024-02-05T06:40:27.073360+0000","last_unstale":"2024-02-05T06:48:13.578621+0000","last_undegraded":"2024-02-05T06:48:13.578621+0000","last_fullsized":"2024-02-05T06:48:13.578621+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'2","last_scrub_stamp":"2024-02-05T06:47:50.442159+0000","last_deep_scrub":"281'2","last_deep_scrub_stamp":"2024-02-05T06:47:50.442159+0000","last_clean_scrub_stamp":"2024-02-05T06:47:50.442159+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:16.471755+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"295'34","reported_seq":729,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.574820+0000","last_change":"2024-02-05T06:47:43.040717+0000","last_active":"2024-02-05T06:48:13.574820+0000","last_peered":"2024-02-05T06:48:13.574820+0000","last_clean":"2024-02-05T06:48:13.574820+0000","last_became_active":"2024-02-05T06:33:31.569763+0000","last_became_peered":"2024-02-05T06:33:31.569763+0000","last_unstale":"2024-02-05T06:48:13.574820+0000","last_undegraded":"2024-02-05T06:48:13.574820+0000","last_fullsized":"2024-02-05T06:48:13.574820+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'34","last_scrub_stamp":"2024-02-05T06:47:43.040574+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:34:35.234636+0000","last_clean_scrub_stamp":"2024-02-05T06:47:43.040574+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:49.320938+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":647,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.566573+0000","last_change":"2024-02-05T06:46:54.584153+0000","last_active":"2024-02-05T06:48:13.566573+0000","last_peered":"2024-02-05T06:48:13.566573+0000","last_clean":"2024-02-05T06:48:13.566573+0000","last_became_active":"2024-02-05T06:41:43.411442+0000","last_became_peered":"2024-02-05T06:41:43.411442+0000","last_unstale":"2024-02-05T06:48:13.566573+0000","last_undegraded":"2024-02-05T06:48:13.566573+0000","last_fullsized":"2024-02-05T06:48:13.566573+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:54.584073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:46:54.584073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.572989+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":443,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.677903+0000","last_change":"2024-02-05T06:48:13.677951+0000","last_active":"2024-02-05T06:48:13.677903+0000","last_peered":"2024-02-05T06:48:13.677903+0000","last_clean":"2024-02-05T06:48:13.677903+0000","last_became_active":"2024-02-05T06:39:13.248618+0000","last_became_peered":"2024-02-05T06:39:13.248618+0000","last_unstale":"2024-02-05T06:48:13.677903+0000","last_undegraded":"2024-02-05T06:48:13.677903+0000","last_fullsized":"2024-02-05T06:48:13.677903+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:13.677795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:42.381636+0000","last_clean_scrub_stamp":"2024-02-05T06:48:13.677795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:41.284546+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":184,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:15.138970+0000","last_change":"2024-02-05T06:48:15.138990+0000","last_active":"2024-02-05T06:48:15.138970+0000","last_peered":"2024-02-05T06:48:15.138970+0000","last_clean":"2024-02-05T06:48:15.138970+0000","last_became_active":"2024-02-05T06:45:09.786579+0000","last_became_peered":"2024-02-05T06:45:09.786579+0000","last_unstale":"2024-02-05T06:48:15.138970+0000","last_undegraded":"2024-02-05T06:48:15.138970+0000","last_fullsized":"2024-02-05T06:48:15.138970+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:15.138923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:48:15.138923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:40.202990+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"295'18","reported_seq":727,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.567108+0000","last_change":"2024-02-05T06:45:01.790024+0000","last_active":"2024-02-05T06:48:13.567108+0000","last_peered":"2024-02-05T06:48:13.567108+0000","last_clean":"2024-02-05T06:48:13.567108+0000","last_became_active":"2024-02-05T06:36:54.954880+0000","last_became_peered":"2024-02-05T06:36:54.954880+0000","last_unstale":"2024-02-05T06:48:13.567108+0000","last_undegraded":"2024-02-05T06:48:13.567108+0000","last_fullsized":"2024-02-05T06:48:13.567108+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'18","last_scrub_stamp":"2024-02-05T06:45:01.789938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:45:01.789938+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.572515+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"325'2","reported_seq":362,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.578700+0000","last_change":"2024-02-05T06:45:46.242637+0000","last_active":"2024-02-05T06:48:13.578700+0000","last_peered":"2024-02-05T06:48:13.578700+0000","last_clean":"2024-02-05T06:48:13.578700+0000","last_became_active":"2024-02-05T06:41:43.418640+0000","last_became_peered":"2024-02-05T06:41:43.418640+0000","last_unstale":"2024-02-05T06:48:13.578700+0000","last_undegraded":"2024-02-05T06:48:13.578700+0000","last_fullsized":"2024-02-05T06:48:13.578700+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'2","last_scrub_stamp":"2024-02-05T06:45:46.242415+0000","last_deep_scrub":"325'2","last_deep_scrub_stamp":"2024-02-05T06:45:46.242415+0000","last_clean_scrub_stamp":"2024-02-05T06:45:46.242415+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:13.594331+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"314'13","reported_seq":905,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.571484+0000","last_change":"2024-02-05T06:47:42.202376+0000","last_active":"2024-02-05T06:48:13.571484+0000","last_peered":"2024-02-05T06:48:13.571484+0000","last_clean":"2024-02-05T06:48:13.571484+0000","last_became_active":"2024-02-05T06:40:27.063399+0000","last_became_peered":"2024-02-05T06:40:27.063399+0000","last_unstale":"2024-02-05T06:48:13.571484+0000","last_undegraded":"2024-02-05T06:48:13.571484+0000","last_fullsized":"2024-02-05T06:48:13.571484+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"314'13","last_scrub_stamp":"2024-02-05T06:47:42.202237+0000","last_deep_scrub":"314'13","last_deep_scrub_stamp":"2024-02-05T06:44:53.670923+0000","last_clean_scrub_stamp":"2024-02-05T06:47:42.202237+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:46.335289+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":691,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.568802+0000","last_change":"2024-02-05T06:47:49.989990+0000","last_active":"2024-02-05T06:48:13.568802+0000","last_peered":"2024-02-05T06:48:13.568802+0000","last_clean":"2024-02-05T06:48:13.568802+0000","last_became_active":"2024-02-05T06:36:54.958414+0000","last_became_peered":"2024-02-05T06:36:54.958414+0000","last_unstale":"2024-02-05T06:48:13.568802+0000","last_undegraded":"2024-02-05T06:48:13.568802+0000","last_fullsized":"2024-02-05T06:48:13.568802+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:47:49.989863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:47:49.989863+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:14.685287+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"295'25","reported_seq":723,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.578111+0000","last_change":"2024-02-05T06:45:40.897349+0000","last_active":"2024-02-05T06:48:13.578111+0000","last_peered":"2024-02-05T06:48:13.578111+0000","last_clean":"2024-02-05T06:48:13.578111+0000","last_became_active":"2024-02-05T06:40:27.100610+0000","last_became_peered":"2024-02-05T06:40:27.100610+0000","last_unstale":"2024-02-05T06:48:13.578111+0000","last_undegraded":"2024-02-05T06:48:13.578111+0000","last_fullsized":"2024-02-05T06:48:13.578111+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'25","last_scrub_stamp":"2024-02-05T06:45:40.897208+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-02-05T06:34:50.718768+0000","last_clean_scrub_stamp":"2024-02-05T06:45:40.897208+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.496439+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":673,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.569731+0000","last_change":"2024-02-05T06:46:45.866461+0000","last_active":"2024-02-05T06:48:13.569731+0000","last_peered":"2024-02-05T06:48:13.569731+0000","last_clean":"2024-02-05T06:48:13.569731+0000","last_became_active":"2024-02-05T06:41:43.414638+0000","last_became_peered":"2024-02-05T06:41:43.414638+0000","last_unstale":"2024-02-05T06:48:13.569731+0000","last_undegraded":"2024-02-05T06:48:13.569731+0000","last_fullsized":"2024-02-05T06:48:13.569731+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:45.866348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:46:45.866348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:15.075391+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":562,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.565288+0000","last_change":"2024-02-05T06:46:36.444754+0000","last_active":"2024-02-05T06:48:13.565288+0000","last_peered":"2024-02-05T06:48:13.565288+0000","last_clean":"2024-02-05T06:48:13.565288+0000","last_became_active":"2024-02-05T06:40:27.097169+0000","last_became_peered":"2024-02-05T06:40:27.097169+0000","last_unstale":"2024-02-05T06:48:13.565288+0000","last_undegraded":"2024-02-05T06:48:13.565288+0000","last_fullsized":"2024-02-05T06:48:13.565288+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:36.444642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:08.713894+0000","last_clean_scrub_stamp":"2024-02-05T06:46:36.444642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.572885+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":262,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.567792+0000","last_change":"2024-02-05T06:46:39.579976+0000","last_active":"2024-02-05T06:48:13.567792+0000","last_peered":"2024-02-05T06:48:13.567792+0000","last_clean":"2024-02-05T06:48:13.567792+0000","last_became_active":"2024-02-05T06:41:43.411043+0000","last_became_peered":"2024-02-05T06:41:43.411043+0000","last_unstale":"2024-02-05T06:48:13.567792+0000","last_undegraded":"2024-02-05T06:48:13.567792+0000","last_fullsized":"2024-02-05T06:48:13.567792+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:39.579866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:46:39.579866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"295'24","reported_seq":749,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.571532+0000","last_change":"2024-02-05T06:47:37.226174+0000","last_active":"2024-02-05T06:48:13.571532+0000","last_peered":"2024-02-05T06:48:13.571532+0000","last_clean":"2024-02-05T06:48:13.571532+0000","last_became_active":"2024-02-05T06:41:43.419769+0000","last_became_peered":"2024-02-05T06:41:43.419769+0000","last_unstale":"2024-02-05T06:48:13.571532+0000","last_undegraded":"2024-02-05T06:48:13.571532+0000","last_fullsized":"2024-02-05T06:48:13.571532+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:47:37.226071+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:43:03.221466+0000","last_clean_scrub_stamp":"2024-02-05T06:47:37.226071+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:40.560472+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":361,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.577896+0000","last_change":"2024-02-05T06:45:51.212570+0000","last_active":"2024-02-05T06:48:13.577896+0000","last_peered":"2024-02-05T06:48:13.577896+0000","last_clean":"2024-02-05T06:48:13.577896+0000","last_became_active":"2024-02-05T06:44:54.625067+0000","last_became_peered":"2024-02-05T06:44:54.625067+0000","last_unstale":"2024-02-05T06:48:13.577896+0000","last_undegraded":"2024-02-05T06:48:13.577896+0000","last_fullsized":"2024-02-05T06:48:13.577896+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:51.212462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:48.512673+0000","last_clean_scrub_stamp":"2024-02-05T06:45:51.212462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:13.594387+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":693,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.563958+0000","last_change":"2024-02-05T06:46:54.260053+0000","last_active":"2024-02-05T06:48:13.563958+0000","last_peered":"2024-02-05T06:48:13.563958+0000","last_clean":"2024-02-05T06:48:13.563958+0000","last_became_active":"2024-02-05T06:40:27.060112+0000","last_became_peered":"2024-02-05T06:40:27.060112+0000","last_unstale":"2024-02-05T06:48:13.563958+0000","last_undegraded":"2024-02-05T06:48:13.563958+0000","last_fullsized":"2024-02-05T06:48:13.563958+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:54.259661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:46:54.259661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:24.236615+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":659,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.577220+0000","last_change":"2024-02-05T06:46:31.409743+0000","last_active":"2024-02-05T06:48:13.577220+0000","last_peered":"2024-02-05T06:48:13.577220+0000","last_clean":"2024-02-05T06:48:13.577220+0000","last_became_active":"2024-02-05T06:41:43.424459+0000","last_became_peered":"2024-02-05T06:41:43.424459+0000","last_unstale":"2024-02-05T06:48:13.577220+0000","last_undegraded":"2024-02-05T06:48:13.577220+0000","last_fullsized":"2024-02-05T06:48:13.577220+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:31.409628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:49.117949+0000","last_clean_scrub_stamp":"2024-02-05T06:46:31.409628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.496700+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"295'8","reported_seq":654,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.564692+0000","last_change":"2024-02-05T06:47:41.444191+0000","last_active":"2024-02-05T06:48:13.564692+0000","last_peered":"2024-02-05T06:48:13.564692+0000","last_clean":"2024-02-05T06:48:13.564692+0000","last_became_active":"2024-02-05T06:33:31.573410+0000","last_became_peered":"2024-02-05T06:33:31.573410+0000","last_unstale":"2024-02-05T06:48:13.564692+0000","last_undegraded":"2024-02-05T06:48:13.564692+0000","last_fullsized":"2024-02-05T06:48:13.564692+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'8","last_scrub_stamp":"2024-02-05T06:47:41.444069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:47:41.444069+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:54.176721+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":420,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.570360+0000","last_change":"2024-02-05T06:46:35.933464+0000","last_active":"2024-02-05T06:48:13.570360+0000","last_peered":"2024-02-05T06:48:13.570360+0000","last_clean":"2024-02-05T06:48:13.570360+0000","last_became_active":"2024-02-05T06:39:13.245684+0000","last_became_peered":"2024-02-05T06:39:13.245684+0000","last_unstale":"2024-02-05T06:48:13.570360+0000","last_undegraded":"2024-02-05T06:48:13.570360+0000","last_fullsized":"2024-02-05T06:48:13.570360+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:35.933354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:46:35.933354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:15.075248+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":355,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.578822+0000","last_change":"2024-02-05T06:46:41.612056+0000","last_active":"2024-02-05T06:48:13.578822+0000","last_peered":"2024-02-05T06:48:13.578822+0000","last_clean":"2024-02-05T06:48:13.578822+0000","last_became_active":"2024-02-05T06:40:27.062975+0000","last_became_peered":"2024-02-05T06:40:27.062975+0000","last_unstale":"2024-02-05T06:48:13.578822+0000","last_undegraded":"2024-02-05T06:48:13.578822+0000","last_fullsized":"2024-02-05T06:48:13.578822+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:41.611955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:46:41.611955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:13.594575+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":270,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.568463+0000","last_change":"2024-02-05T06:46:51.634330+0000","last_active":"2024-02-05T06:48:13.568463+0000","last_peered":"2024-02-05T06:48:13.568463+0000","last_clean":"2024-02-05T06:48:13.568463+0000","last_became_active":"2024-02-05T06:41:43.412654+0000","last_became_peered":"2024-02-05T06:41:43.412654+0000","last_unstale":"2024-02-05T06:48:13.568463+0000","last_undegraded":"2024-02-05T06:48:13.568463+0000","last_fullsized":"2024-02-05T06:48:13.568463+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:51.634250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:46:51.634250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:14.278559+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"295'24","reported_seq":732,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.565950+0000","last_change":"2024-02-05T06:47:30.341586+0000","last_active":"2024-02-05T06:48:13.565950+0000","last_peered":"2024-02-05T06:48:13.565950+0000","last_clean":"2024-02-05T06:48:13.565950+0000","last_became_active":"2024-02-05T06:40:27.080906+0000","last_became_peered":"2024-02-05T06:40:27.080906+0000","last_unstale":"2024-02-05T06:48:13.565950+0000","last_undegraded":"2024-02-05T06:48:13.565950+0000","last_fullsized":"2024-02-05T06:48:13.565950+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:47:30.341433+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-02-05T06:35:22.938675+0000","last_clean_scrub_stamp":"2024-02-05T06:47:30.341433+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:51.439941+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"336'2","reported_seq":665,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.574090+0000","last_change":"2024-02-05T06:46:53.500514+0000","last_active":"2024-02-05T06:48:13.574090+0000","last_peered":"2024-02-05T06:48:13.574090+0000","last_clean":"2024-02-05T06:48:13.574090+0000","last_became_active":"2024-02-05T06:41:43.409761+0000","last_became_peered":"2024-02-05T06:41:43.409761+0000","last_unstale":"2024-02-05T06:48:13.574090+0000","last_undegraded":"2024-02-05T06:48:13.574090+0000","last_fullsized":"2024-02-05T06:48:13.574090+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'2","last_scrub_stamp":"2024-02-05T06:46:53.500155+0000","last_deep_scrub":"336'2","last_deep_scrub_stamp":"2024-02-05T06:46:53.500155+0000","last_clean_scrub_stamp":"2024-02-05T06:46:53.500155+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":638,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.567299+0000","last_change":"2024-02-05T06:47:35.411302+0000","last_active":"2024-02-05T06:48:13.567299+0000","last_peered":"2024-02-05T06:48:13.567299+0000","last_clean":"2024-02-05T06:48:13.567299+0000","last_became_active":"2024-02-05T06:33:33.593901+0000","last_became_peered":"2024-02-05T06:33:33.593901+0000","last_unstale":"2024-02-05T06:48:13.567299+0000","last_undegraded":"2024-02-05T06:48:13.567299+0000","last_fullsized":"2024-02-05T06:48:13.567299+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:35.411219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:47:35.411219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:41.160858+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":355,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.579208+0000","last_change":"2024-02-05T06:46:29.865891+0000","last_active":"2024-02-05T06:48:13.579208+0000","last_peered":"2024-02-05T06:48:13.579208+0000","last_clean":"2024-02-05T06:48:13.579208+0000","last_became_active":"2024-02-05T06:40:27.077701+0000","last_became_peered":"2024-02-05T06:40:27.077701+0000","last_unstale":"2024-02-05T06:48:13.579208+0000","last_undegraded":"2024-02-05T06:48:13.579208+0000","last_fullsized":"2024-02-05T06:48:13.579208+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:29.865777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:46:29.865777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:13.594633+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"295'15","reported_seq":439,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.576569+0000","last_change":"2024-02-05T06:46:34.466054+0000","last_active":"2024-02-05T06:48:13.576569+0000","last_peered":"2024-02-05T06:48:13.576569+0000","last_clean":"2024-02-05T06:48:13.576569+0000","last_became_active":"2024-02-05T06:39:13.244124+0000","last_became_peered":"2024-02-05T06:39:13.244124+0000","last_unstale":"2024-02-05T06:48:13.576569+0000","last_undegraded":"2024-02-05T06:48:13.576569+0000","last_fullsized":"2024-02-05T06:48:13.576569+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:46:34.465921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:46:34.465921+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.496797+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":646,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.565588+0000","last_change":"2024-02-05T06:45:35.052699+0000","last_active":"2024-02-05T06:48:13.565588+0000","last_peered":"2024-02-05T06:48:13.565588+0000","last_clean":"2024-02-05T06:48:13.565588+0000","last_became_active":"2024-02-05T06:41:43.417155+0000","last_became_peered":"2024-02-05T06:41:43.417155+0000","last_unstale":"2024-02-05T06:48:13.565588+0000","last_undegraded":"2024-02-05T06:48:13.565588+0000","last_fullsized":"2024-02-05T06:48:13.565588+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:35.052451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:45:35.052451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.572702+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":672,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.569605+0000","last_change":"2024-02-05T06:45:40.802539+0000","last_active":"2024-02-05T06:48:13.569605+0000","last_peered":"2024-02-05T06:48:13.569605+0000","last_clean":"2024-02-05T06:48:13.569605+0000","last_became_active":"2024-02-05T06:41:43.421433+0000","last_became_peered":"2024-02-05T06:41:43.421433+0000","last_unstale":"2024-02-05T06:48:13.569605+0000","last_undegraded":"2024-02-05T06:48:13.569605+0000","last_fullsized":"2024-02-05T06:48:13.569605+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:40.802433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:45:40.802433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:15.075115+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":309,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.578385+0000","last_change":"2024-02-05T06:47:47.669670+0000","last_active":"2024-02-05T06:48:13.578385+0000","last_peered":"2024-02-05T06:48:13.578385+0000","last_clean":"2024-02-05T06:48:13.578385+0000","last_became_active":"2024-02-05T06:45:09.773490+0000","last_became_peered":"2024-02-05T06:45:09.773490+0000","last_unstale":"2024-02-05T06:48:13.578385+0000","last_undegraded":"2024-02-05T06:48:13.578385+0000","last_fullsized":"2024-02-05T06:48:13.578385+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:47.669594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:47:47.669594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:04.867910+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"295'24","reported_seq":516,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.566866+0000","last_change":"2024-02-05T06:46:52.290692+0000","last_active":"2024-02-05T06:48:13.566866+0000","last_peered":"2024-02-05T06:48:13.566866+0000","last_clean":"2024-02-05T06:48:13.566866+0000","last_became_active":"2024-02-05T06:45:09.784411+0000","last_became_peered":"2024-02-05T06:45:09.784411+0000","last_unstale":"2024-02-05T06:48:13.566866+0000","last_undegraded":"2024-02-05T06:48:13.566866+0000","last_fullsized":"2024-02-05T06:48:13.566866+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:46:52.290420+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:35:02.589229+0000","last_clean_scrub_stamp":"2024-02-05T06:46:52.290420+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.697998+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":647,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.577682+0000","last_change":"2024-02-05T06:45:39.000945+0000","last_active":"2024-02-05T06:48:13.577682+0000","last_peered":"2024-02-05T06:48:13.577682+0000","last_clean":"2024-02-05T06:48:13.577682+0000","last_became_active":"2024-02-05T06:40:27.096132+0000","last_became_peered":"2024-02-05T06:40:27.096132+0000","last_unstale":"2024-02-05T06:48:13.577682+0000","last_undegraded":"2024-02-05T06:48:13.577682+0000","last_fullsized":"2024-02-05T06:48:13.577682+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:39.000627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:45:39.000627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.496520+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"336'11","reported_seq":354,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.570079+0000","last_change":"2024-02-05T06:46:55.722262+0000","last_active":"2024-02-05T06:48:13.570079+0000","last_peered":"2024-02-05T06:48:13.570079+0000","last_clean":"2024-02-05T06:48:13.570079+0000","last_became_active":"2024-02-05T06:41:43.418753+0000","last_became_peered":"2024-02-05T06:41:43.418753+0000","last_unstale":"2024-02-05T06:48:13.570079+0000","last_undegraded":"2024-02-05T06:48:13.570079+0000","last_fullsized":"2024-02-05T06:48:13.570079+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'11","last_scrub_stamp":"2024-02-05T06:46:55.721412+0000","last_deep_scrub":"336'11","last_deep_scrub_stamp":"2024-02-05T06:46:55.721412+0000","last_clean_scrub_stamp":"2024-02-05T06:46:55.721412+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":355,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.579600+0000","last_change":"2024-02-05T06:46:38.694358+0000","last_active":"2024-02-05T06:48:13.579600+0000","last_peered":"2024-02-05T06:48:13.579600+0000","last_clean":"2024-02-05T06:48:13.579600+0000","last_became_active":"2024-02-05T06:40:27.098418+0000","last_became_peered":"2024-02-05T06:40:27.098418+0000","last_unstale":"2024-02-05T06:48:13.579600+0000","last_undegraded":"2024-02-05T06:48:13.579600+0000","last_fullsized":"2024-02-05T06:48:13.579600+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:38.694245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:46:38.694245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:13.594449+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"295'20","reported_seq":726,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.565884+0000","last_change":"2024-02-05T06:45:31.996335+0000","last_active":"2024-02-05T06:48:13.565884+0000","last_peered":"2024-02-05T06:48:13.565884+0000","last_clean":"2024-02-05T06:48:13.565884+0000","last_became_active":"2024-02-05T06:40:27.093418+0000","last_became_peered":"2024-02-05T06:40:27.093418+0000","last_unstale":"2024-02-05T06:48:13.565884+0000","last_undegraded":"2024-02-05T06:48:13.565884+0000","last_fullsized":"2024-02-05T06:48:13.565884+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'20","last_scrub_stamp":"2024-02-05T06:45:31.996183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:45:31.996183+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.572628+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":402,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.573801+0000","last_change":"2024-02-05T06:46:51.823639+0000","last_active":"2024-02-05T06:48:13.573801+0000","last_peered":"2024-02-05T06:48:13.573801+0000","last_clean":"2024-02-05T06:48:13.573801+0000","last_became_active":"2024-02-05T06:45:09.775152+0000","last_became_peered":"2024-02-05T06:45:09.775152+0000","last_unstale":"2024-02-05T06:48:13.573801+0000","last_undegraded":"2024-02-05T06:48:13.573801+0000","last_fullsized":"2024-02-05T06:48:13.573801+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:51.823530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:41:38.559780+0000","last_clean_scrub_stamp":"2024-02-05T06:46:51.823530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:18.081463+0000","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"208'10","reported_seq":844,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.567457+0000","last_change":"2024-02-05T06:44:49.013016+0000","last_active":"2024-02-05T06:48:13.567457+0000","last_peered":"2024-02-05T06:48:13.567457+0000","last_clean":"2024-02-05T06:48:13.567457+0000","last_became_active":"2024-02-05T06:40:27.096584+0000","last_became_peered":"2024-02-05T06:40:27.096584+0000","last_unstale":"2024-02-05T06:48:13.567457+0000","last_undegraded":"2024-02-05T06:48:13.567457+0000","last_fullsized":"2024-02-05T06:48:13.567457+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-02-05T06:44:49.012783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:44:49.012783+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.572486+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":667,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.577135+0000","last_change":"2024-02-05T06:45:45.945300+0000","last_active":"2024-02-05T06:48:13.577135+0000","last_peered":"2024-02-05T06:48:13.577135+0000","last_clean":"2024-02-05T06:48:13.577135+0000","last_became_active":"2024-02-05T06:40:27.088096+0000","last_became_peered":"2024-02-05T06:40:27.088096+0000","last_unstale":"2024-02-05T06:48:13.577135+0000","last_undegraded":"2024-02-05T06:48:13.577135+0000","last_fullsized":"2024-02-05T06:48:13.577135+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:45.945168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:44:01.185653+0000","last_clean_scrub_stamp":"2024-02-05T06:45:45.945168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.496575+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"295'15","reported_seq":289,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.570342+0000","last_change":"2024-02-05T06:46:49.603908+0000","last_active":"2024-02-05T06:48:13.570342+0000","last_peered":"2024-02-05T06:48:13.570342+0000","last_clean":"2024-02-05T06:48:13.570342+0000","last_became_active":"2024-02-05T06:41:43.413708+0000","last_became_peered":"2024-02-05T06:41:43.413708+0000","last_unstale":"2024-02-05T06:48:13.570342+0000","last_undegraded":"2024-02-05T06:48:13.570342+0000","last_fullsized":"2024-02-05T06:48:13.570342+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:46:49.603788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:46:49.603788+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.864227+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":688,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.575166+0000","last_change":"2024-02-05T06:46:56.761919+0000","last_active":"2024-02-05T06:48:13.575166+0000","last_peered":"2024-02-05T06:48:13.575166+0000","last_clean":"2024-02-05T06:48:13.575166+0000","last_became_active":"2024-02-05T06:40:27.090632+0000","last_became_peered":"2024-02-05T06:40:27.090632+0000","last_unstale":"2024-02-05T06:48:13.575166+0000","last_undegraded":"2024-02-05T06:48:13.575166+0000","last_fullsized":"2024-02-05T06:48:13.575166+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:56.761768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:46:56.761768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:18.081518+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":664,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.566721+0000","last_change":"2024-02-05T06:47:43.427431+0000","last_active":"2024-02-05T06:48:13.566721+0000","last_peered":"2024-02-05T06:48:13.566721+0000","last_clean":"2024-02-05T06:48:13.566721+0000","last_became_active":"2024-02-05T06:40:27.091544+0000","last_became_peered":"2024-02-05T06:40:27.091544+0000","last_unstale":"2024-02-05T06:48:13.566721+0000","last_undegraded":"2024-02-05T06:48:13.566721+0000","last_fullsized":"2024-02-05T06:48:13.566721+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:43.427301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:47:43.427301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:08.540744+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":355,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.579629+0000","last_change":"2024-02-05T06:46:43.606599+0000","last_active":"2024-02-05T06:48:13.579629+0000","last_peered":"2024-02-05T06:48:13.579629+0000","last_clean":"2024-02-05T06:48:13.579629+0000","last_became_active":"2024-02-05T06:40:27.064388+0000","last_became_peered":"2024-02-05T06:40:27.064388+0000","last_unstale":"2024-02-05T06:48:13.579629+0000","last_undegraded":"2024-02-05T06:48:13.579629+0000","last_fullsized":"2024-02-05T06:48:13.579629+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:43.606488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:43.606488+0000","last_clean_scrub_stamp":"2024-02-05T06:46:43.606488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"295'15","reported_seq":708,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.578040+0000","last_change":"2024-02-05T06:45:55.988708+0000","last_active":"2024-02-05T06:48:13.578040+0000","last_peered":"2024-02-05T06:48:13.578040+0000","last_clean":"2024-02-05T06:48:13.578040+0000","last_became_active":"2024-02-05T06:40:27.101116+0000","last_became_peered":"2024-02-05T06:40:27.101116+0000","last_unstale":"2024-02-05T06:48:13.578040+0000","last_undegraded":"2024-02-05T06:48:13.578040+0000","last_fullsized":"2024-02-05T06:48:13.578040+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:45:55.988575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:45:55.988575+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.496646+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":355,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.579531+0000","last_change":"2024-02-05T06:47:38.782082+0000","last_active":"2024-02-05T06:48:13.579531+0000","last_peered":"2024-02-05T06:48:13.579531+0000","last_clean":"2024-02-05T06:48:13.579531+0000","last_became_active":"2024-02-05T06:40:27.063350+0000","last_became_peered":"2024-02-05T06:40:27.063350+0000","last_unstale":"2024-02-05T06:48:13.579531+0000","last_undegraded":"2024-02-05T06:48:13.579531+0000","last_fullsized":"2024-02-05T06:48:13.579531+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:38.781966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:47:38.781966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:57.815067+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":270,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.568393+0000","last_change":"2024-02-05T06:46:56.707237+0000","last_active":"2024-02-05T06:48:13.568393+0000","last_peered":"2024-02-05T06:48:13.568393+0000","last_clean":"2024-02-05T06:48:13.568393+0000","last_became_active":"2024-02-05T06:41:43.406094+0000","last_became_peered":"2024-02-05T06:41:43.406094+0000","last_unstale":"2024-02-05T06:48:13.568393+0000","last_undegraded":"2024-02-05T06:48:13.568393+0000","last_fullsized":"2024-02-05T06:48:13.568393+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:56.707131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:44:57.506909+0000","last_clean_scrub_stamp":"2024-02-05T06:46:56.707131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:17.685381+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":681,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.564185+0000","last_change":"2024-02-05T06:46:43.354213+0000","last_active":"2024-02-05T06:48:13.564185+0000","last_peered":"2024-02-05T06:48:13.564185+0000","last_clean":"2024-02-05T06:48:13.564185+0000","last_became_active":"2024-02-05T06:41:43.412130+0000","last_became_peered":"2024-02-05T06:41:43.412130+0000","last_unstale":"2024-02-05T06:48:13.564185+0000","last_undegraded":"2024-02-05T06:48:13.564185+0000","last_fullsized":"2024-02-05T06:48:13.564185+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:43.354098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:46:43.354098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:14.727834+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"295'12","reported_seq":688,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.566658+0000","last_change":"2024-02-05T06:46:47.455766+0000","last_active":"2024-02-05T06:48:13.566658+0000","last_peered":"2024-02-05T06:48:13.566658+0000","last_clean":"2024-02-05T06:48:13.566658+0000","last_became_active":"2024-02-05T06:40:27.091689+0000","last_became_peered":"2024-02-05T06:40:27.091689+0000","last_unstale":"2024-02-05T06:48:13.566658+0000","last_undegraded":"2024-02-05T06:48:13.566658+0000","last_fullsized":"2024-02-05T06:48:13.566658+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'12","last_scrub_stamp":"2024-02-05T06:46:47.455632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:46:47.455632+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.572963+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"365'4218","reported_seq":4267,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:16.588964+0000","last_change":"2024-02-05T06:47:24.184995+0000","last_active":"2024-02-05T06:48:16.588964+0000","last_peered":"2024-02-05T06:48:16.588964+0000","last_clean":"2024-02-05T06:48:16.588964+0000","last_became_active":"2024-02-05T06:47:24.184581+0000","last_became_peered":"2024-02-05T06:47:24.184581+0000","last_unstale":"2024-02-05T06:48:16.588964+0000","last_undegraded":"2024-02-05T06:48:16.588964+0000","last_fullsized":"2024-02-05T06:48:16.588964+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":4218,"log_dups_size":0,"ondisk_log_size":4218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:50.103785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276430848,"num_objects":4218,"num_object_clones":0,"num_object_copies":12654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4218,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4218,"num_write_kb":269952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":271,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.569604+0000","last_change":"2024-02-05T06:47:38.659232+0000","last_active":"2024-02-05T06:48:13.569604+0000","last_peered":"2024-02-05T06:48:13.569604+0000","last_clean":"2024-02-05T06:48:13.569604+0000","last_became_active":"2024-02-05T06:41:43.421231+0000","last_became_peered":"2024-02-05T06:41:43.421231+0000","last_unstale":"2024-02-05T06:48:13.569604+0000","last_undegraded":"2024-02-05T06:48:13.569604+0000","last_fullsized":"2024-02-05T06:48:13.569604+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:38.659123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:47:38.659123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:53.928020+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":361,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.578636+0000","last_change":"2024-02-05T06:46:32.836286+0000","last_active":"2024-02-05T06:48:13.578636+0000","last_peered":"2024-02-05T06:48:13.578636+0000","last_clean":"2024-02-05T06:48:13.578636+0000","last_became_active":"2024-02-05T06:40:27.068279+0000","last_became_peered":"2024-02-05T06:40:27.068279+0000","last_unstale":"2024-02-05T06:48:13.578636+0000","last_undegraded":"2024-02-05T06:48:13.578636+0000","last_fullsized":"2024-02-05T06:48:13.578636+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-02-05T06:46:32.835758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:46:32.835758+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:13.594419+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"295'27","reported_seq":297,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.568068+0000","last_change":"2024-02-05T06:45:56.217323+0000","last_active":"2024-02-05T06:48:13.568068+0000","last_peered":"2024-02-05T06:48:13.568068+0000","last_clean":"2024-02-05T06:48:13.568068+0000","last_became_active":"2024-02-05T06:41:43.412579+0000","last_became_peered":"2024-02-05T06:41:43.412579+0000","last_unstale":"2024-02-05T06:48:13.568068+0000","last_undegraded":"2024-02-05T06:48:13.568068+0000","last_fullsized":"2024-02-05T06:48:13.568068+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'27","last_scrub_stamp":"2024-02-05T06:45:56.217180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:45:56.217180+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"208'12","reported_seq":453,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.579530+0000","last_change":"2024-02-05T06:46:37.741787+0000","last_active":"2024-02-05T06:48:13.579530+0000","last_peered":"2024-02-05T06:48:13.579530+0000","last_clean":"2024-02-05T06:48:13.579530+0000","last_became_active":"2024-02-05T06:40:27.064195+0000","last_became_peered":"2024-02-05T06:40:27.064195+0000","last_unstale":"2024-02-05T06:48:13.579530+0000","last_undegraded":"2024-02-05T06:48:13.579530+0000","last_fullsized":"2024-02-05T06:48:13.579530+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'12","last_scrub_stamp":"2024-02-05T06:46:37.741678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:46:37.741678+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"365'4270","reported_seq":4320,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:16.458285+0000","last_change":"2024-02-05T06:47:24.176419+0000","last_active":"2024-02-05T06:48:16.458285+0000","last_peered":"2024-02-05T06:48:16.458285+0000","last_clean":"2024-02-05T06:48:16.458285+0000","last_became_active":"2024-02-05T06:47:24.176055+0000","last_became_peered":"2024-02-05T06:47:24.176055+0000","last_unstale":"2024-02-05T06:48:16.458285+0000","last_undegraded":"2024-02-05T06:48:16.458285+0000","last_fullsized":"2024-02-05T06:48:16.458285+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":4270,"log_dups_size":0,"ondisk_log_size":4270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:37.445734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279838720,"num_objects":4270,"num_object_clones":0,"num_object_copies":12810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4270,"num_write_kb":273280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":646,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.566683+0000","last_change":"2024-02-05T06:47:36.512402+0000","last_active":"2024-02-05T06:48:13.566683+0000","last_peered":"2024-02-05T06:48:13.566683+0000","last_clean":"2024-02-05T06:48:13.566683+0000","last_became_active":"2024-02-05T06:40:27.092613+0000","last_became_peered":"2024-02-05T06:40:27.092613+0000","last_unstale":"2024-02-05T06:48:13.566683+0000","last_undegraded":"2024-02-05T06:48:13.566683+0000","last_fullsized":"2024-02-05T06:48:13.566683+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:36.512306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:47:36.512306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:40.368980+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":355,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.578183+0000","last_change":"2024-02-05T06:46:53.740645+0000","last_active":"2024-02-05T06:48:13.578183+0000","last_peered":"2024-02-05T06:48:13.578183+0000","last_clean":"2024-02-05T06:48:13.578183+0000","last_became_active":"2024-02-05T06:40:27.063300+0000","last_became_peered":"2024-02-05T06:40:27.063300+0000","last_unstale":"2024-02-05T06:48:13.578183+0000","last_undegraded":"2024-02-05T06:48:13.578183+0000","last_fullsized":"2024-02-05T06:48:13.578183+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:53.740542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:46:53.740542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:13.594666+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"295'18","reported_seq":514,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.568787+0000","last_change":"2024-02-05T06:46:58.712252+0000","last_active":"2024-02-05T06:48:13.568787+0000","last_peered":"2024-02-05T06:48:13.568787+0000","last_clean":"2024-02-05T06:48:13.568787+0000","last_became_active":"2024-02-05T06:45:09.783036+0000","last_became_peered":"2024-02-05T06:45:09.783036+0000","last_unstale":"2024-02-05T06:48:13.568787+0000","last_undegraded":"2024-02-05T06:48:13.568787+0000","last_fullsized":"2024-02-05T06:48:13.568787+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'18","last_scrub_stamp":"2024-02-05T06:46:58.712125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:46:58.712125+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:24.407018+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":20,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":660,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.572815+0000","last_change":"2024-02-05T06:46:42.685554+0000","last_active":"2024-02-05T06:48:13.572815+0000","last_peered":"2024-02-05T06:48:13.572815+0000","last_clean":"2024-02-05T06:48:13.572815+0000","last_became_active":"2024-02-05T06:33:33.592546+0000","last_became_peered":"2024-02-05T06:33:33.592546+0000","last_unstale":"2024-02-05T06:48:13.572815+0000","last_undegraded":"2024-02-05T06:48:13.572815+0000","last_fullsized":"2024-02-05T06:48:13.572815+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:42.685240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:42.685240+0000","last_clean_scrub_stamp":"2024-02-05T06:46:42.685240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:18.081433+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.4","version":"365'4172","reported_seq":4222,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:17.397451+0000","last_change":"2024-02-05T06:47:24.185199+0000","last_active":"2024-02-05T06:48:17.397451+0000","last_peered":"2024-02-05T06:48:17.397451+0000","last_clean":"2024-02-05T06:48:17.397451+0000","last_became_active":"2024-02-05T06:47:24.184891+0000","last_became_peered":"2024-02-05T06:47:24.184891+0000","last_unstale":"2024-02-05T06:48:17.397451+0000","last_undegraded":"2024-02-05T06:48:17.397451+0000","last_fullsized":"2024-02-05T06:48:17.397451+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":4172,"log_dups_size":0,"ondisk_log_size":4172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:26.662797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273416192,"num_objects":4172,"num_object_clones":0,"num_object_copies":12516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4172,"num_write_kb":267008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":355,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.578902+0000","last_change":"2024-02-05T06:46:46.694131+0000","last_active":"2024-02-05T06:48:13.578902+0000","last_peered":"2024-02-05T06:48:13.578902+0000","last_clean":"2024-02-05T06:48:13.578902+0000","last_became_active":"2024-02-05T06:40:27.218081+0000","last_became_peered":"2024-02-05T06:40:27.218081+0000","last_unstale":"2024-02-05T06:48:13.578902+0000","last_undegraded":"2024-02-05T06:48:13.578902+0000","last_fullsized":"2024-02-05T06:48:13.578902+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:46.694027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:32.925000+0000","last_clean_scrub_stamp":"2024-02-05T06:46:46.694027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:13.594605+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":686,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.573189+0000","last_change":"2024-02-05T06:47:45.073824+0000","last_active":"2024-02-05T06:48:13.573189+0000","last_peered":"2024-02-05T06:48:13.573189+0000","last_clean":"2024-02-05T06:48:13.573189+0000","last_became_active":"2024-02-05T06:41:43.413095+0000","last_became_peered":"2024-02-05T06:41:43.413095+0000","last_unstale":"2024-02-05T06:48:13.573189+0000","last_undegraded":"2024-02-05T06:48:13.573189+0000","last_fullsized":"2024-02-05T06:48:13.573189+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:45.073681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:45.073681+0000","last_clean_scrub_stamp":"2024-02-05T06:47:45.073681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:57.251457+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":361,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.579290+0000","last_change":"2024-02-05T06:47:36.783062+0000","last_active":"2024-02-05T06:48:13.579290+0000","last_peered":"2024-02-05T06:48:13.579290+0000","last_clean":"2024-02-05T06:48:13.579290+0000","last_became_active":"2024-02-05T06:41:43.418202+0000","last_became_peered":"2024-02-05T06:41:43.418202+0000","last_unstale":"2024-02-05T06:48:13.579290+0000","last_undegraded":"2024-02-05T06:48:13.579290+0000","last_fullsized":"2024-02-05T06:48:13.579290+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:36.782960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:47:36.782960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:48.010223+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"295'44","reported_seq":815,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.567502+0000","last_change":"2024-02-05T06:47:50.338639+0000","last_active":"2024-02-05T06:48:13.567502+0000","last_peered":"2024-02-05T06:48:13.567502+0000","last_clean":"2024-02-05T06:48:13.567502+0000","last_became_active":"2024-02-05T06:40:27.068417+0000","last_became_peered":"2024-02-05T06:40:27.068417+0000","last_unstale":"2024-02-05T06:48:13.567502+0000","last_undegraded":"2024-02-05T06:48:13.567502+0000","last_fullsized":"2024-02-05T06:48:13.567502+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'44","last_scrub_stamp":"2024-02-05T06:47:50.338505+0000","last_deep_scrub":"279'28","last_deep_scrub_stamp":"2024-02-05T06:42:50.148881+0000","last_clean_scrub_stamp":"2024-02-05T06:47:50.338505+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:03.563604+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"365'4257","reported_seq":4307,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:16.805184+0000","last_change":"2024-02-05T06:47:24.178230+0000","last_active":"2024-02-05T06:48:16.805184+0000","last_peered":"2024-02-05T06:48:16.805184+0000","last_clean":"2024-02-05T06:48:16.805184+0000","last_became_active":"2024-02-05T06:47:24.177943+0000","last_became_peered":"2024-02-05T06:47:24.177943+0000","last_unstale":"2024-02-05T06:48:16.805184+0000","last_undegraded":"2024-02-05T06:48:16.805184+0000","last_fullsized":"2024-02-05T06:48:16.805184+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":4257,"log_dups_size":0,"ondisk_log_size":4257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:45.429907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278986752,"num_objects":4257,"num_object_clones":0,"num_object_copies":12771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4257,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4257,"num_write_kb":272448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":647,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.575209+0000","last_change":"2024-02-05T06:46:33.542381+0000","last_active":"2024-02-05T06:48:13.575209+0000","last_peered":"2024-02-05T06:48:13.575209+0000","last_clean":"2024-02-05T06:48:13.575209+0000","last_became_active":"2024-02-05T06:40:27.129625+0000","last_became_peered":"2024-02-05T06:40:27.129625+0000","last_unstale":"2024-02-05T06:48:13.575209+0000","last_undegraded":"2024-02-05T06:48:13.575209+0000","last_fullsized":"2024-02-05T06:48:13.575209+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:33.542276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:46:33.542276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.496759+0000","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":680,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.573144+0000","last_change":"2024-02-05T06:46:33.712118+0000","last_active":"2024-02-05T06:48:13.573144+0000","last_peered":"2024-02-05T06:48:13.573144+0000","last_clean":"2024-02-05T06:48:13.573144+0000","last_became_active":"2024-02-05T06:36:54.966016+0000","last_became_peered":"2024-02-05T06:36:54.966016+0000","last_unstale":"2024-02-05T06:48:13.573144+0000","last_undegraded":"2024-02-05T06:48:13.573144+0000","last_fullsized":"2024-02-05T06:48:13.573144+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:33.711989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:46:33.711989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:18.081490+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":270,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.570278+0000","last_change":"2024-02-05T06:47:41.617654+0000","last_active":"2024-02-05T06:48:13.570278+0000","last_peered":"2024-02-05T06:48:13.570278+0000","last_clean":"2024-02-05T06:48:13.570278+0000","last_became_active":"2024-02-05T06:41:43.412611+0000","last_became_peered":"2024-02-05T06:41:43.412611+0000","last_unstale":"2024-02-05T06:48:13.570278+0000","last_undegraded":"2024-02-05T06:48:13.570278+0000","last_fullsized":"2024-02-05T06:48:13.570278+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:41.617526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:41.617526+0000","last_clean_scrub_stamp":"2024-02-05T06:47:41.617526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:51.274032+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"295'15","reported_seq":756,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.578535+0000","last_change":"2024-02-05T06:45:49.873565+0000","last_active":"2024-02-05T06:48:13.578535+0000","last_peered":"2024-02-05T06:48:13.578535+0000","last_clean":"2024-02-05T06:48:13.578535+0000","last_became_active":"2024-02-05T06:40:08.584204+0000","last_became_peered":"2024-02-05T06:40:08.584204+0000","last_unstale":"2024-02-05T06:48:13.578535+0000","last_undegraded":"2024-02-05T06:48:13.578535+0000","last_fullsized":"2024-02-05T06:48:13.578535+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:45:49.873251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:45:49.873251+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.496495+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"365'4276","reported_seq":4326,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:16.443180+0000","last_change":"2024-02-05T06:47:24.178003+0000","last_active":"2024-02-05T06:48:16.443180+0000","last_peered":"2024-02-05T06:48:16.443180+0000","last_clean":"2024-02-05T06:48:16.443180+0000","last_became_active":"2024-02-05T06:47:24.177714+0000","last_became_peered":"2024-02-05T06:47:24.177714+0000","last_unstale":"2024-02-05T06:48:16.443180+0000","last_undegraded":"2024-02-05T06:48:16.443180+0000","last_fullsized":"2024-02-05T06:48:16.443180+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":4276,"log_dups_size":0,"ondisk_log_size":4276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:28.973293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280231936,"num_objects":4276,"num_object_clones":0,"num_object_copies":12828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4276,"num_write_kb":273664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":688,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.574363+0000","last_change":"2024-02-05T06:46:47.669870+0000","last_active":"2024-02-05T06:48:13.574363+0000","last_peered":"2024-02-05T06:48:13.574363+0000","last_clean":"2024-02-05T06:48:13.574363+0000","last_became_active":"2024-02-05T06:41:46.054968+0000","last_became_peered":"2024-02-05T06:41:46.054968+0000","last_unstale":"2024-02-05T06:48:13.574363+0000","last_undegraded":"2024-02-05T06:48:13.574363+0000","last_fullsized":"2024-02-05T06:48:13.574363+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:46:47.669634+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:37.282442+0000","last_clean_scrub_stamp":"2024-02-05T06:46:47.669634+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":658,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.565784+0000","last_change":"2024-02-05T06:46:48.502814+0000","last_active":"2024-02-05T06:48:13.565784+0000","last_peered":"2024-02-05T06:48:13.565784+0000","last_clean":"2024-02-05T06:48:13.565784+0000","last_became_active":"2024-02-05T06:40:27.082440+0000","last_became_peered":"2024-02-05T06:40:27.082440+0000","last_unstale":"2024-02-05T06:48:13.565784+0000","last_undegraded":"2024-02-05T06:48:13.565784+0000","last_fullsized":"2024-02-05T06:48:13.565784+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:48.502695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:46:48.502695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.572909+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"182'3","reported_seq":275,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:14.571526+0000","last_change":"2024-02-05T06:48:14.571567+0000","last_active":"2024-02-05T06:48:14.571526+0000","last_peered":"2024-02-05T06:48:14.571526+0000","last_clean":"2024-02-05T06:48:14.571526+0000","last_became_active":"2024-02-05T06:41:43.415806+0000","last_became_peered":"2024-02-05T06:41:43.415806+0000","last_unstale":"2024-02-05T06:48:14.571526+0000","last_undegraded":"2024-02-05T06:48:14.571526+0000","last_fullsized":"2024-02-05T06:48:14.571526+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'3","last_scrub_stamp":"2024-02-05T06:48:14.571441+0000","last_deep_scrub":"182'3","last_deep_scrub_stamp":"2024-02-05T06:39:33.373415+0000","last_clean_scrub_stamp":"2024-02-05T06:48:14.571441+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:26.936434+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.001220348,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":184,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:16.597917+0000","last_change":"2024-02-05T06:48:16.597953+0000","last_active":"2024-02-05T06:48:16.597917+0000","last_peered":"2024-02-05T06:48:16.597917+0000","last_clean":"2024-02-05T06:48:16.597917+0000","last_became_active":"2024-02-05T06:45:09.776169+0000","last_became_peered":"2024-02-05T06:45:09.776169+0000","last_unstale":"2024-02-05T06:48:16.597917+0000","last_undegraded":"2024-02-05T06:48:16.597917+0000","last_fullsized":"2024-02-05T06:48:16.597917+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:16.597853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:48:16.597853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:26.531928+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"326'52","reported_seq":524,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:16.104808+0000","last_change":"2024-02-05T06:48:16.104872+0000","last_active":"2024-02-05T06:48:16.104808+0000","last_peered":"2024-02-05T06:48:16.104808+0000","last_clean":"2024-02-05T06:48:16.104808+0000","last_became_active":"2024-02-05T06:40:08.604483+0000","last_became_peered":"2024-02-05T06:40:08.604483+0000","last_unstale":"2024-02-05T06:48:16.104808+0000","last_undegraded":"2024-02-05T06:48:16.104808+0000","last_fullsized":"2024-02-05T06:48:16.104808+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'52","last_scrub_stamp":"2024-02-05T06:48:16.104709+0000","last_deep_scrub":"326'52","last_deep_scrub_stamp":"2024-02-05T06:45:53.715482+0000","last_clean_scrub_stamp":"2024-02-05T06:48:16.104709+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:21.284859+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":660,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.579619+0000","last_change":"2024-02-05T06:47:49.412086+0000","last_active":"2024-02-05T06:48:13.579619+0000","last_peered":"2024-02-05T06:48:13.579619+0000","last_clean":"2024-02-05T06:48:13.579619+0000","last_became_active":"2024-02-05T06:36:54.973387+0000","last_became_peered":"2024-02-05T06:36:54.973387+0000","last_unstale":"2024-02-05T06:48:13.579619+0000","last_undegraded":"2024-02-05T06:48:13.579619+0000","last_fullsized":"2024-02-05T06:48:13.579619+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:49.411958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:47:49.411958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:08.497020+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"365'4112","reported_seq":4161,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:16.612575+0000","last_change":"2024-02-05T06:47:24.177476+0000","last_active":"2024-02-05T06:48:16.612575+0000","last_peered":"2024-02-05T06:48:16.612575+0000","last_clean":"2024-02-05T06:48:16.612575+0000","last_became_active":"2024-02-05T06:47:24.177145+0000","last_became_peered":"2024-02-05T06:47:24.177145+0000","last_unstale":"2024-02-05T06:48:16.612575+0000","last_undegraded":"2024-02-05T06:48:16.612575+0000","last_fullsized":"2024-02-05T06:48:16.612575+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":4112,"log_dups_size":0,"ondisk_log_size":4112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:45.854097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269484032,"num_objects":4112,"num_object_clones":0,"num_object_copies":12336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4112,"num_write_kb":263168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":776,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.577785+0000","last_change":"2024-02-05T06:46:44.660384+0000","last_active":"2024-02-05T06:48:13.577785+0000","last_peered":"2024-02-05T06:48:13.577785+0000","last_clean":"2024-02-05T06:48:13.577785+0000","last_became_active":"2024-02-05T06:40:30.082467+0000","last_became_peered":"2024-02-05T06:40:30.082467+0000","last_unstale":"2024-02-05T06:48:13.577785+0000","last_undegraded":"2024-02-05T06:48:13.577785+0000","last_fullsized":"2024-02-05T06:48:13.577785+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:46:44.660271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:46:44.660271+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:13.594517+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":355,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.579035+0000","last_change":"2024-02-05T06:46:45.774572+0000","last_active":"2024-02-05T06:48:13.579035+0000","last_peered":"2024-02-05T06:48:13.579035+0000","last_clean":"2024-02-05T06:48:13.579035+0000","last_became_active":"2024-02-05T06:40:27.077966+0000","last_became_peered":"2024-02-05T06:40:27.077966+0000","last_unstale":"2024-02-05T06:48:13.579035+0000","last_undegraded":"2024-02-05T06:48:13.579035+0000","last_fullsized":"2024-02-05T06:48:13.579035+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:45.774441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:45.774441+0000","last_clean_scrub_stamp":"2024-02-05T06:46:45.774441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:13.594547+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":655,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.572463+0000","last_change":"2024-02-05T06:46:32.768890+0000","last_active":"2024-02-05T06:48:13.572463+0000","last_peered":"2024-02-05T06:48:13.572463+0000","last_clean":"2024-02-05T06:48:13.572463+0000","last_became_active":"2024-02-05T06:33:29.551549+0000","last_became_peered":"2024-02-05T06:33:29.551549+0000","last_unstale":"2024-02-05T06:48:13.572463+0000","last_undegraded":"2024-02-05T06:48:13.572463+0000","last_fullsized":"2024-02-05T06:48:13.572463+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:32.768777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:46:32.768777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:15.075193+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":686,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.572962+0000","last_change":"2024-02-05T06:47:49.065297+0000","last_active":"2024-02-05T06:48:13.572962+0000","last_peered":"2024-02-05T06:48:13.572962+0000","last_clean":"2024-02-05T06:48:13.572962+0000","last_became_active":"2024-02-05T06:33:28.542977+0000","last_became_peered":"2024-02-05T06:33:28.542977+0000","last_unstale":"2024-02-05T06:48:13.572962+0000","last_undegraded":"2024-02-05T06:48:13.572962+0000","last_fullsized":"2024-02-05T06:48:13.572962+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:49.065028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:47:49.065028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:57.562644+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00091640799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"295'17","reported_seq":753,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.570314+0000","last_change":"2024-02-05T06:46:43.816340+0000","last_active":"2024-02-05T06:48:13.570314+0000","last_peered":"2024-02-05T06:48:13.570314+0000","last_clean":"2024-02-05T06:48:13.570314+0000","last_became_active":"2024-02-05T06:41:43.420458+0000","last_became_peered":"2024-02-05T06:41:43.420458+0000","last_unstale":"2024-02-05T06:48:13.570314+0000","last_undegraded":"2024-02-05T06:48:13.570314+0000","last_fullsized":"2024-02-05T06:48:13.570314+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'17","last_scrub_stamp":"2024-02-05T06:46:43.816187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:46:43.816187+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:15.075364+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":700,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.573640+0000","last_change":"2024-02-05T06:47:51.064183+0000","last_active":"2024-02-05T06:48:13.573640+0000","last_peered":"2024-02-05T06:48:13.573640+0000","last_clean":"2024-02-05T06:48:13.573640+0000","last_became_active":"2024-02-05T06:41:43.412387+0000","last_became_peered":"2024-02-05T06:41:43.412387+0000","last_unstale":"2024-02-05T06:48:13.573640+0000","last_undegraded":"2024-02-05T06:48:13.573640+0000","last_fullsized":"2024-02-05T06:48:13.573640+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:51.063926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:33.914422+0000","last_clean_scrub_stamp":"2024-02-05T06:47:51.063926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:55.492615+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"365'4241","reported_seq":4291,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:16.448961+0000","last_change":"2024-02-05T06:47:24.180383+0000","last_active":"2024-02-05T06:48:16.448961+0000","last_peered":"2024-02-05T06:48:16.448961+0000","last_clean":"2024-02-05T06:48:16.448961+0000","last_became_active":"2024-02-05T06:47:24.179124+0000","last_became_peered":"2024-02-05T06:47:24.179124+0000","last_unstale":"2024-02-05T06:48:16.448961+0000","last_undegraded":"2024-02-05T06:48:16.448961+0000","last_fullsized":"2024-02-05T06:48:16.448961+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":4241,"log_dups_size":0,"ondisk_log_size":4241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:49.727806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277938176,"num_objects":4241,"num_object_clones":0,"num_object_copies":12723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4241,"num_write_kb":271424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":271,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.570997+0000","last_change":"2024-02-05T06:46:48.716378+0000","last_active":"2024-02-05T06:48:13.570997+0000","last_peered":"2024-02-05T06:48:13.570997+0000","last_clean":"2024-02-05T06:48:13.570997+0000","last_became_active":"2024-02-05T06:41:43.408773+0000","last_became_peered":"2024-02-05T06:41:43.408773+0000","last_unstale":"2024-02-05T06:48:13.570997+0000","last_undegraded":"2024-02-05T06:48:13.570997+0000","last_fullsized":"2024-02-05T06:48:13.570997+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:48.716273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:46:48.716273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":776,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.579009+0000","last_change":"2024-02-05T06:46:33.834624+0000","last_active":"2024-02-05T06:48:13.579009+0000","last_peered":"2024-02-05T06:48:13.579009+0000","last_clean":"2024-02-05T06:48:13.579009+0000","last_became_active":"2024-02-05T06:40:30.081178+0000","last_became_peered":"2024-02-05T06:40:30.081178+0000","last_unstale":"2024-02-05T06:48:13.579009+0000","last_undegraded":"2024-02-05T06:48:13.579009+0000","last_fullsized":"2024-02-05T06:48:13.579009+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:46:33.834510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:46:33.834510+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:13.594711+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":659,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.565077+0000","last_change":"2024-02-05T06:45:57.195429+0000","last_active":"2024-02-05T06:48:13.565077+0000","last_peered":"2024-02-05T06:48:13.565077+0000","last_clean":"2024-02-05T06:48:13.565077+0000","last_became_active":"2024-02-05T06:41:43.411335+0000","last_became_peered":"2024-02-05T06:41:43.411335+0000","last_unstale":"2024-02-05T06:48:13.565077+0000","last_undegraded":"2024-02-05T06:48:13.565077+0000","last_fullsized":"2024-02-05T06:48:13.565077+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:57.195321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:45:57.195321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.572780+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"295'32","reported_seq":982,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.570119+0000","last_change":"2024-02-05T06:47:35.173300+0000","last_active":"2024-02-05T06:48:13.570119+0000","last_peered":"2024-02-05T06:48:13.570119+0000","last_clean":"2024-02-05T06:48:13.570119+0000","last_became_active":"2024-02-05T06:40:27.061111+0000","last_became_peered":"2024-02-05T06:40:27.061111+0000","last_unstale":"2024-02-05T06:48:13.570119+0000","last_undegraded":"2024-02-05T06:48:13.570119+0000","last_fullsized":"2024-02-05T06:48:13.570119+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'32","last_scrub_stamp":"2024-02-05T06:47:35.173150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:47:35.173150+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:58.867739+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":361,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.578609+0000","last_change":"2024-02-05T06:45:49.152771+0000","last_active":"2024-02-05T06:48:13.578609+0000","last_peered":"2024-02-05T06:48:13.578609+0000","last_clean":"2024-02-05T06:48:13.578609+0000","last_became_active":"2024-02-05T06:41:43.418198+0000","last_became_peered":"2024-02-05T06:41:43.418198+0000","last_unstale":"2024-02-05T06:48:13.578609+0000","last_undegraded":"2024-02-05T06:48:13.578609+0000","last_fullsized":"2024-02-05T06:48:13.578609+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:49.152682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:45:49.152682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:13.594359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"365'4250","reported_seq":4300,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:17.369849+0000","last_change":"2024-02-05T06:47:24.177582+0000","last_active":"2024-02-05T06:48:17.369849+0000","last_peered":"2024-02-05T06:48:17.369849+0000","last_clean":"2024-02-05T06:48:17.369849+0000","last_became_active":"2024-02-05T06:47:24.177037+0000","last_became_peered":"2024-02-05T06:47:24.177037+0000","last_unstale":"2024-02-05T06:48:17.369849+0000","last_undegraded":"2024-02-05T06:48:17.369849+0000","last_fullsized":"2024-02-05T06:48:17.369849+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":4250,"log_dups_size":0,"ondisk_log_size":4250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:32.432264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278528000,"num_objects":4250,"num_object_clones":0,"num_object_copies":12750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4250,"num_write_kb":272000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":818,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.570455+0000","last_change":"2024-02-05T06:46:42.818851+0000","last_active":"2024-02-05T06:48:13.570455+0000","last_peered":"2024-02-05T06:48:13.570455+0000","last_clean":"2024-02-05T06:48:13.570455+0000","last_became_active":"2024-02-05T06:37:37.137754+0000","last_became_peered":"2024-02-05T06:37:37.137754+0000","last_unstale":"2024-02-05T06:48:13.570455+0000","last_undegraded":"2024-02-05T06:48:13.570455+0000","last_fullsized":"2024-02-05T06:48:13.570455+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:46:42.818725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:46:42.818725+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:15.075223+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":663,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.569013+0000","last_change":"2024-02-05T06:46:31.716329+0000","last_active":"2024-02-05T06:48:13.569013+0000","last_peered":"2024-02-05T06:48:13.569013+0000","last_clean":"2024-02-05T06:48:13.569013+0000","last_became_active":"2024-02-05T06:41:43.410910+0000","last_became_peered":"2024-02-05T06:41:43.410910+0000","last_unstale":"2024-02-05T06:48:13.569013+0000","last_undegraded":"2024-02-05T06:48:13.569013+0000","last_fullsized":"2024-02-05T06:48:13.569013+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:31.716099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:46:31.716099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:15.075141+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":668,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.574782+0000","last_change":"2024-02-05T06:46:56.459465+0000","last_active":"2024-02-05T06:48:13.574782+0000","last_peered":"2024-02-05T06:48:13.574782+0000","last_clean":"2024-02-05T06:48:13.574782+0000","last_became_active":"2024-02-05T06:36:54.972264+0000","last_became_peered":"2024-02-05T06:36:54.972264+0000","last_unstale":"2024-02-05T06:48:13.574782+0000","last_undegraded":"2024-02-05T06:48:13.574782+0000","last_fullsized":"2024-02-05T06:48:13.574782+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:56.459164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:46:56.459164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"355'6","reported_seq":833,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:14.787848+0000","last_change":"2024-02-05T06:47:46.642736+0000","last_active":"2024-02-05T06:48:14.787848+0000","last_peered":"2024-02-05T06:48:14.787848+0000","last_clean":"2024-02-05T06:48:14.787848+0000","last_became_active":"2024-02-05T06:40:27.071509+0000","last_became_peered":"2024-02-05T06:40:27.071509+0000","last_unstale":"2024-02-05T06:48:14.787848+0000","last_undegraded":"2024-02-05T06:48:14.787848+0000","last_fullsized":"2024-02-05T06:48:14.787848+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'6","last_scrub_stamp":"2024-02-05T06:47:46.642454+0000","last_deep_scrub":"355'6","last_deep_scrub_stamp":"2024-02-05T06:47:46.642454+0000","last_clean_scrub_stamp":"2024-02-05T06:47:46.642454+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:59.661377+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00074151600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":475,"num_read_kb":470,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"295'24","reported_seq":687,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.577800+0000","last_change":"2024-02-05T06:47:40.582521+0000","last_active":"2024-02-05T06:48:13.577800+0000","last_peered":"2024-02-05T06:48:13.577800+0000","last_clean":"2024-02-05T06:48:13.577800+0000","last_became_active":"2024-02-05T06:33:31.564604+0000","last_became_peered":"2024-02-05T06:33:31.564604+0000","last_unstale":"2024-02-05T06:48:13.577800+0000","last_undegraded":"2024-02-05T06:48:13.577800+0000","last_fullsized":"2024-02-05T06:48:13.577800+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:47:40.582236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:47:40.582236+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:54.033478+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":263,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.571080+0000","last_change":"2024-02-05T06:45:46.133321+0000","last_active":"2024-02-05T06:48:13.571080+0000","last_peered":"2024-02-05T06:48:13.571080+0000","last_clean":"2024-02-05T06:48:13.571080+0000","last_became_active":"2024-02-05T06:41:43.417751+0000","last_became_peered":"2024-02-05T06:41:43.417751+0000","last_unstale":"2024-02-05T06:48:13.571080+0000","last_undegraded":"2024-02-05T06:48:13.571080+0000","last_fullsized":"2024-02-05T06:48:13.571080+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:46.133190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:45:46.133190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"365'4261","reported_seq":4309,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:16.450461+0000","last_change":"2024-02-05T06:47:24.182432+0000","last_active":"2024-02-05T06:48:16.450461+0000","last_peered":"2024-02-05T06:48:16.450461+0000","last_clean":"2024-02-05T06:48:16.450461+0000","last_became_active":"2024-02-05T06:47:24.182110+0000","last_became_peered":"2024-02-05T06:47:24.182110+0000","last_unstale":"2024-02-05T06:48:16.450461+0000","last_undegraded":"2024-02-05T06:48:16.450461+0000","last_fullsized":"2024-02-05T06:48:16.450461+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":4261,"log_dups_size":0,"ondisk_log_size":4261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:46.830260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279248896,"num_objects":4261,"num_object_clones":0,"num_object_copies":12783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4261,"num_write_kb":272704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":830,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.570760+0000","last_change":"2024-02-05T06:46:53.293283+0000","last_active":"2024-02-05T06:48:13.570760+0000","last_peered":"2024-02-05T06:48:13.570760+0000","last_clean":"2024-02-05T06:48:13.570760+0000","last_became_active":"2024-02-05T06:37:37.140011+0000","last_became_peered":"2024-02-05T06:37:37.140011+0000","last_unstale":"2024-02-05T06:48:13.570760+0000","last_undegraded":"2024-02-05T06:48:13.570760+0000","last_fullsized":"2024-02-05T06:48:13.570760+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:46:53.293189+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:38.778568+0000","last_clean_scrub_stamp":"2024-02-05T06:46:53.293189+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:13.711710+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":651,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.568908+0000","last_change":"2024-02-05T06:47:45.022360+0000","last_active":"2024-02-05T06:48:13.568908+0000","last_peered":"2024-02-05T06:48:13.568908+0000","last_clean":"2024-02-05T06:48:13.568908+0000","last_became_active":"2024-02-05T06:33:35.664858+0000","last_became_peered":"2024-02-05T06:33:35.664858+0000","last_unstale":"2024-02-05T06:48:13.568908+0000","last_undegraded":"2024-02-05T06:48:13.568908+0000","last_fullsized":"2024-02-05T06:48:13.568908+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:45.022240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:47:45.022240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:09.456304+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"66'1","reported_seq":689,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.570696+0000","last_change":"2024-02-05T06:46:30.749245+0000","last_active":"2024-02-05T06:48:13.570696+0000","last_peered":"2024-02-05T06:48:13.570696+0000","last_clean":"2024-02-05T06:48:13.570696+0000","last_became_active":"2024-02-05T06:41:43.411404+0000","last_became_peered":"2024-02-05T06:41:43.411404+0000","last_unstale":"2024-02-05T06:48:13.570696+0000","last_undegraded":"2024-02-05T06:48:13.570696+0000","last_fullsized":"2024-02-05T06:48:13.570696+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:46:30.748577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:46:30.748577+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:15.075310+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"295'29","reported_seq":799,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.578708+0000","last_change":"2024-02-05T06:47:39.575482+0000","last_active":"2024-02-05T06:48:13.578708+0000","last_peered":"2024-02-05T06:48:13.578708+0000","last_clean":"2024-02-05T06:48:13.578708+0000","last_became_active":"2024-02-05T06:41:43.409664+0000","last_became_peered":"2024-02-05T06:41:43.409664+0000","last_unstale":"2024-02-05T06:48:13.578708+0000","last_undegraded":"2024-02-05T06:48:13.578708+0000","last_fullsized":"2024-02-05T06:48:13.578708+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'29","last_scrub_stamp":"2024-02-05T06:47:39.575331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:47:39.575331+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:55.789881+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":674,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.568017+0000","last_change":"2024-02-05T06:47:48.255664+0000","last_active":"2024-02-05T06:48:13.568017+0000","last_peered":"2024-02-05T06:48:13.568017+0000","last_clean":"2024-02-05T06:48:13.568017+0000","last_became_active":"2024-02-05T06:41:43.411550+0000","last_became_peered":"2024-02-05T06:41:43.411550+0000","last_unstale":"2024-02-05T06:48:13.568017+0000","last_undegraded":"2024-02-05T06:48:13.568017+0000","last_fullsized":"2024-02-05T06:48:13.568017+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:48.255364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:47:48.255364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:56.793069+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"365'4091","reported_seq":4140,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:16.657553+0000","last_change":"2024-02-05T06:47:24.185798+0000","last_active":"2024-02-05T06:48:16.657553+0000","last_peered":"2024-02-05T06:48:16.657553+0000","last_clean":"2024-02-05T06:48:16.657553+0000","last_became_active":"2024-02-05T06:47:24.185452+0000","last_became_peered":"2024-02-05T06:47:24.185452+0000","last_unstale":"2024-02-05T06:48:16.657553+0000","last_undegraded":"2024-02-05T06:48:16.657553+0000","last_fullsized":"2024-02-05T06:48:16.657553+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":4091,"log_dups_size":0,"ondisk_log_size":4091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:43.544292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268107776,"num_objects":4091,"num_object_clones":0,"num_object_copies":12273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4091,"num_write_kb":261824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":849,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.571532+0000","last_change":"2024-02-05T06:46:39.414808+0000","last_active":"2024-02-05T06:48:13.571532+0000","last_peered":"2024-02-05T06:48:13.571532+0000","last_clean":"2024-02-05T06:48:13.571532+0000","last_became_active":"2024-02-05T06:40:30.085438+0000","last_became_peered":"2024-02-05T06:40:30.085438+0000","last_unstale":"2024-02-05T06:48:13.571532+0000","last_undegraded":"2024-02-05T06:48:13.571532+0000","last_fullsized":"2024-02-05T06:48:13.571532+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:46:39.414680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:46:39.414680+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:13.711825+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":636,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.574205+0000","last_change":"2024-02-05T06:45:48.925337+0000","last_active":"2024-02-05T06:48:13.574205+0000","last_peered":"2024-02-05T06:48:13.574205+0000","last_clean":"2024-02-05T06:48:13.574205+0000","last_became_active":"2024-02-05T06:33:35.600485+0000","last_became_peered":"2024-02-05T06:33:35.600485+0000","last_unstale":"2024-02-05T06:48:13.574205+0000","last_undegraded":"2024-02-05T06:48:13.574205+0000","last_fullsized":"2024-02-05T06:48:13.574205+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:48.925218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:45:48.925218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.496673+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":263,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.570225+0000","last_change":"2024-02-05T06:46:38.483037+0000","last_active":"2024-02-05T06:48:13.570225+0000","last_peered":"2024-02-05T06:48:13.570225+0000","last_clean":"2024-02-05T06:48:13.570225+0000","last_became_active":"2024-02-05T06:41:43.411732+0000","last_became_peered":"2024-02-05T06:41:43.411732+0000","last_unstale":"2024-02-05T06:48:13.570225+0000","last_undegraded":"2024-02-05T06:48:13.570225+0000","last_fullsized":"2024-02-05T06:48:13.570225+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:38.482939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:28.353956+0000","last_clean_scrub_stamp":"2024-02-05T06:46:38.482939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"295'20","reported_seq":753,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.569715+0000","last_change":"2024-02-05T06:45:51.645898+0000","last_active":"2024-02-05T06:48:13.569715+0000","last_peered":"2024-02-05T06:48:13.569715+0000","last_clean":"2024-02-05T06:48:13.569715+0000","last_became_active":"2024-02-05T06:41:43.421030+0000","last_became_peered":"2024-02-05T06:41:43.421030+0000","last_unstale":"2024-02-05T06:48:13.569715+0000","last_undegraded":"2024-02-05T06:48:13.569715+0000","last_fullsized":"2024-02-05T06:48:13.569715+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'20","last_scrub_stamp":"2024-02-05T06:45:51.645768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:45:51.645768+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:15.075062+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"335'11","reported_seq":855,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.578483+0000","last_change":"2024-02-05T06:46:52.427460+0000","last_active":"2024-02-05T06:48:13.578483+0000","last_peered":"2024-02-05T06:48:13.578483+0000","last_clean":"2024-02-05T06:48:13.578483+0000","last_became_active":"2024-02-05T06:41:43.423638+0000","last_became_peered":"2024-02-05T06:41:43.423638+0000","last_unstale":"2024-02-05T06:48:13.578483+0000","last_undegraded":"2024-02-05T06:48:13.578483+0000","last_fullsized":"2024-02-05T06:48:13.578483+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"335'11","last_scrub_stamp":"2024-02-05T06:46:52.423878+0000","last_deep_scrub":"335'11","last_deep_scrub_stamp":"2024-02-05T06:46:52.423878+0000","last_clean_scrub_stamp":"2024-02-05T06:46:52.423878+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"365'4223","reported_seq":4273,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:16.467780+0000","last_change":"2024-02-05T06:47:24.175068+0000","last_active":"2024-02-05T06:48:16.467780+0000","last_peered":"2024-02-05T06:48:16.467780+0000","last_clean":"2024-02-05T06:48:16.467780+0000","last_became_active":"2024-02-05T06:47:24.174574+0000","last_became_peered":"2024-02-05T06:47:24.174574+0000","last_unstale":"2024-02-05T06:48:16.467780+0000","last_undegraded":"2024-02-05T06:48:16.467780+0000","last_fullsized":"2024-02-05T06:48:16.467780+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":4223,"log_dups_size":0,"ondisk_log_size":4223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:30.861106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276758528,"num_objects":4223,"num_object_clones":0,"num_object_copies":12669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4223,"num_write_kb":270272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":853,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.565207+0000","last_change":"2024-02-05T06:45:54.266666+0000","last_active":"2024-02-05T06:48:13.565207+0000","last_peered":"2024-02-05T06:48:13.565207+0000","last_clean":"2024-02-05T06:48:13.565207+0000","last_became_active":"2024-02-05T06:41:46.053837+0000","last_became_peered":"2024-02-05T06:41:46.053837+0000","last_unstale":"2024-02-05T06:48:13.565207+0000","last_undegraded":"2024-02-05T06:48:13.565207+0000","last_fullsized":"2024-02-05T06:48:13.565207+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:45:54.266536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:45:54.266536+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.572731+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":355,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.579518+0000","last_change":"2024-02-05T06:47:45.759776+0000","last_active":"2024-02-05T06:48:13.579518+0000","last_peered":"2024-02-05T06:48:13.579518+0000","last_clean":"2024-02-05T06:48:13.579518+0000","last_became_active":"2024-02-05T06:40:27.099168+0000","last_became_peered":"2024-02-05T06:40:27.099168+0000","last_unstale":"2024-02-05T06:48:13.579518+0000","last_undegraded":"2024-02-05T06:48:13.579518+0000","last_fullsized":"2024-02-05T06:48:13.579518+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:45.759651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:47:45.759651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:48.053819+0000","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"95'2","reported_seq":669,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.577887+0000","last_change":"2024-02-05T06:45:44.911946+0000","last_active":"2024-02-05T06:48:13.577887+0000","last_peered":"2024-02-05T06:48:13.577887+0000","last_clean":"2024-02-05T06:48:13.577887+0000","last_became_active":"2024-02-05T06:36:54.975093+0000","last_became_peered":"2024-02-05T06:36:54.975093+0000","last_unstale":"2024-02-05T06:48:13.577887+0000","last_undegraded":"2024-02-05T06:48:13.577887+0000","last_fullsized":"2024-02-05T06:48:13.577887+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-05T06:45:44.911816+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-05T06:35:25.616708+0000","last_clean_scrub_stamp":"2024-02-05T06:45:44.911816+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.496550+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"295'28","reported_seq":573,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.574255+0000","last_change":"2024-02-05T06:45:39.141292+0000","last_active":"2024-02-05T06:48:13.574255+0000","last_peered":"2024-02-05T06:48:13.574255+0000","last_clean":"2024-02-05T06:48:13.574255+0000","last_became_active":"2024-02-05T06:45:09.764696+0000","last_became_peered":"2024-02-05T06:45:09.764696+0000","last_unstale":"2024-02-05T06:48:13.574255+0000","last_undegraded":"2024-02-05T06:48:13.574255+0000","last_fullsized":"2024-02-05T06:48:13.574255+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'28","last_scrub_stamp":"2024-02-05T06:45:39.141209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:45:39.141209+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":40,"num_read_kb":33,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"326'11","reported_seq":848,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.564950+0000","last_change":"2024-02-05T06:45:56.212330+0000","last_active":"2024-02-05T06:48:13.564950+0000","last_peered":"2024-02-05T06:48:13.564950+0000","last_clean":"2024-02-05T06:48:13.564950+0000","last_became_active":"2024-02-05T06:36:54.954819+0000","last_became_peered":"2024-02-05T06:36:54.954819+0000","last_unstale":"2024-02-05T06:48:13.564950+0000","last_undegraded":"2024-02-05T06:48:13.564950+0000","last_fullsized":"2024-02-05T06:48:13.564950+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'11","last_scrub_stamp":"2024-02-05T06:45:56.212177+0000","last_deep_scrub":"326'11","last_deep_scrub_stamp":"2024-02-05T06:45:56.212177+0000","last_clean_scrub_stamp":"2024-02-05T06:45:56.212177+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.572835+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"365'4195","reported_seq":4245,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:16.462508+0000","last_change":"2024-02-05T06:47:24.184945+0000","last_active":"2024-02-05T06:48:16.462508+0000","last_peered":"2024-02-05T06:48:16.462508+0000","last_clean":"2024-02-05T06:48:16.462508+0000","last_became_active":"2024-02-05T06:47:24.184494+0000","last_became_peered":"2024-02-05T06:47:24.184494+0000","last_unstale":"2024-02-05T06:48:16.462508+0000","last_undegraded":"2024-02-05T06:48:16.462508+0000","last_fullsized":"2024-02-05T06:48:16.462508+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":4195,"log_dups_size":0,"ondisk_log_size":4195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:51.789612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274923520,"num_objects":4195,"num_object_clones":0,"num_object_copies":12585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4195,"num_write_kb":268480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":420,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.574186+0000","last_change":"2024-02-05T06:47:46.034108+0000","last_active":"2024-02-05T06:48:13.574186+0000","last_peered":"2024-02-05T06:48:13.574186+0000","last_clean":"2024-02-05T06:48:13.574186+0000","last_became_active":"2024-02-05T06:45:09.776631+0000","last_became_peered":"2024-02-05T06:45:09.776631+0000","last_unstale":"2024-02-05T06:48:13.574186+0000","last_undegraded":"2024-02-05T06:48:13.574186+0000","last_fullsized":"2024-02-05T06:48:13.574186+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:47:46.033702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:47:46.033702+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:00.978347+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":177,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.565435+0000","last_change":"2024-02-05T06:45:49.145234+0000","last_active":"2024-02-05T06:48:13.565435+0000","last_peered":"2024-02-05T06:48:13.565435+0000","last_clean":"2024-02-05T06:48:13.565435+0000","last_became_active":"2024-02-05T06:45:09.780390+0000","last_became_peered":"2024-02-05T06:45:09.780390+0000","last_unstale":"2024-02-05T06:48:13.565435+0000","last_undegraded":"2024-02-05T06:48:13.565435+0000","last_fullsized":"2024-02-05T06:48:13.565435+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:49.145117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:45:49.145117+0000","last_clean_scrub_stamp":"2024-02-05T06:45:49.145117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.572805+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":664,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.566254+0000","last_change":"2024-02-05T06:45:42.080964+0000","last_active":"2024-02-05T06:48:13.566254+0000","last_peered":"2024-02-05T06:48:13.566254+0000","last_clean":"2024-02-05T06:48:13.566254+0000","last_became_active":"2024-02-05T06:40:27.093739+0000","last_became_peered":"2024-02-05T06:40:27.093739+0000","last_unstale":"2024-02-05T06:48:13.566254+0000","last_undegraded":"2024-02-05T06:48:13.566254+0000","last_fullsized":"2024-02-05T06:48:13.566254+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:42.080862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:38:28.150695+0000","last_clean_scrub_stamp":"2024-02-05T06:45:42.080862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.572653+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"295'30","reported_seq":741,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.564852+0000","last_change":"2024-02-05T06:47:50.402454+0000","last_active":"2024-02-05T06:48:13.564852+0000","last_peered":"2024-02-05T06:48:13.564852+0000","last_clean":"2024-02-05T06:48:13.564852+0000","last_became_active":"2024-02-05T06:40:27.277747+0000","last_became_peered":"2024-02-05T06:40:27.277747+0000","last_unstale":"2024-02-05T06:48:13.564852+0000","last_undegraded":"2024-02-05T06:48:13.564852+0000","last_fullsized":"2024-02-05T06:48:13.564852+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'30","last_scrub_stamp":"2024-02-05T06:47:50.402305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:47:50.402305+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:08.001287+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":318,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.565864+0000","last_change":"2024-02-05T06:46:30.377076+0000","last_active":"2024-02-05T06:48:13.565864+0000","last_peered":"2024-02-05T06:48:13.565864+0000","last_clean":"2024-02-05T06:48:13.565864+0000","last_became_active":"2024-02-05T06:45:09.778609+0000","last_became_peered":"2024-02-05T06:45:09.778609+0000","last_unstale":"2024-02-05T06:48:13.565864+0000","last_undegraded":"2024-02-05T06:48:13.565864+0000","last_fullsized":"2024-02-05T06:48:13.565864+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:30.376861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:46:30.376861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.572859+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"365'4172","reported_seq":4220,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:16.645435+0000","last_change":"2024-02-05T06:47:24.182652+0000","last_active":"2024-02-05T06:48:16.645435+0000","last_peered":"2024-02-05T06:48:16.645435+0000","last_clean":"2024-02-05T06:48:16.645435+0000","last_became_active":"2024-02-05T06:47:24.182190+0000","last_became_peered":"2024-02-05T06:47:24.182190+0000","last_unstale":"2024-02-05T06:48:16.645435+0000","last_undegraded":"2024-02-05T06:48:16.645435+0000","last_fullsized":"2024-02-05T06:48:16.645435+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":4172,"log_dups_size":0,"ondisk_log_size":4172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:50.966086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273416192,"num_objects":4172,"num_object_clones":0,"num_object_copies":12516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4172,"num_write_kb":267008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":682,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.571343+0000","last_change":"2024-02-05T06:46:35.425172+0000","last_active":"2024-02-05T06:48:13.571343+0000","last_peered":"2024-02-05T06:48:13.571343+0000","last_clean":"2024-02-05T06:48:13.571343+0000","last_became_active":"2024-02-05T06:41:46.053993+0000","last_became_peered":"2024-02-05T06:41:46.053993+0000","last_unstale":"2024-02-05T06:48:13.571343+0000","last_undegraded":"2024-02-05T06:48:13.571343+0000","last_fullsized":"2024-02-05T06:48:13.571343+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:46:35.424973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:46:35.424973+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":270,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.570133+0000","last_change":"2024-02-05T06:46:45.769966+0000","last_active":"2024-02-05T06:48:13.570133+0000","last_peered":"2024-02-05T06:48:13.570133+0000","last_clean":"2024-02-05T06:48:13.570133+0000","last_became_active":"2024-02-05T06:41:43.418066+0000","last_became_peered":"2024-02-05T06:41:43.418066+0000","last_unstale":"2024-02-05T06:48:13.570133+0000","last_undegraded":"2024-02-05T06:48:13.570133+0000","last_fullsized":"2024-02-05T06:48:13.570133+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:45.769853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:45.769853+0000","last_clean_scrub_stamp":"2024-02-05T06:46:45.769853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"319'13","reported_seq":507,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.568306+0000","last_change":"2024-02-05T06:47:46.597798+0000","last_active":"2024-02-05T06:48:13.568306+0000","last_peered":"2024-02-05T06:48:13.568306+0000","last_clean":"2024-02-05T06:48:13.568306+0000","last_became_active":"2024-02-05T06:45:09.766220+0000","last_became_peered":"2024-02-05T06:45:09.766220+0000","last_unstale":"2024-02-05T06:48:13.568306+0000","last_undegraded":"2024-02-05T06:48:13.568306+0000","last_fullsized":"2024-02-05T06:48:13.568306+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'13","last_scrub_stamp":"2024-02-05T06:47:46.597456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:47:46.597456+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:57.094687+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":666,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:14.631460+0000","last_change":"2024-02-05T06:48:14.631495+0000","last_active":"2024-02-05T06:48:14.631460+0000","last_peered":"2024-02-05T06:48:14.631460+0000","last_clean":"2024-02-05T06:48:14.631460+0000","last_became_active":"2024-02-05T06:40:27.213439+0000","last_became_peered":"2024-02-05T06:40:27.213439+0000","last_unstale":"2024-02-05T06:48:14.631460+0000","last_undegraded":"2024-02-05T06:48:14.631460+0000","last_fullsized":"2024-02-05T06:48:14.631460+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:14.631382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:52.482566+0000","last_clean_scrub_stamp":"2024-02-05T06:48:14.631382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:29.843880+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"295'34","reported_seq":462,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.576897+0000","last_change":"2024-02-05T06:46:44.355699+0000","last_active":"2024-02-05T06:48:13.576897+0000","last_peered":"2024-02-05T06:48:13.576897+0000","last_clean":"2024-02-05T06:48:13.576897+0000","last_became_active":"2024-02-05T06:39:13.257978+0000","last_became_peered":"2024-02-05T06:39:13.257978+0000","last_unstale":"2024-02-05T06:48:13.576897+0000","last_undegraded":"2024-02-05T06:48:13.576897+0000","last_fullsized":"2024-02-05T06:48:13.576897+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'34","last_scrub_stamp":"2024-02-05T06:46:44.355563+0000","last_deep_scrub":"295'34","last_deep_scrub_stamp":"2024-02-05T06:46:44.355563+0000","last_clean_scrub_stamp":"2024-02-05T06:46:44.355563+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.0","version":"365'4067","reported_seq":4117,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:16.447315+0000","last_change":"2024-02-05T06:47:24.183477+0000","last_active":"2024-02-05T06:48:16.447315+0000","last_peered":"2024-02-05T06:48:16.447315+0000","last_clean":"2024-02-05T06:48:16.447315+0000","last_became_active":"2024-02-05T06:47:24.183098+0000","last_became_peered":"2024-02-05T06:47:24.183098+0000","last_unstale":"2024-02-05T06:48:16.447315+0000","last_undegraded":"2024-02-05T06:48:16.447315+0000","last_fullsized":"2024-02-05T06:48:16.447315+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":4067,"log_dups_size":0,"ondisk_log_size":4067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:34.968232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":266534912,"num_objects":4067,"num_object_clones":0,"num_object_copies":12201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4067,"num_write_kb":260288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":652,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.565771+0000","last_change":"2024-02-05T06:46:58.479329+0000","last_active":"2024-02-05T06:48:13.565771+0000","last_peered":"2024-02-05T06:48:13.565771+0000","last_clean":"2024-02-05T06:48:13.565771+0000","last_became_active":"2024-02-05T06:41:43.410600+0000","last_became_peered":"2024-02-05T06:41:43.410600+0000","last_unstale":"2024-02-05T06:48:13.565771+0000","last_undegraded":"2024-02-05T06:48:13.565771+0000","last_fullsized":"2024-02-05T06:48:13.565771+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:58.479208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:58.479208+0000","last_clean_scrub_stamp":"2024-02-05T06:46:58.479208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.572934+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":270,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.570873+0000","last_change":"2024-02-05T06:46:57.669791+0000","last_active":"2024-02-05T06:48:13.570873+0000","last_peered":"2024-02-05T06:48:13.570873+0000","last_clean":"2024-02-05T06:48:13.570873+0000","last_became_active":"2024-02-05T06:41:43.417343+0000","last_became_peered":"2024-02-05T06:41:43.417343+0000","last_unstale":"2024-02-05T06:48:13.570873+0000","last_undegraded":"2024-02-05T06:48:13.570873+0000","last_fullsized":"2024-02-05T06:48:13.570873+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:57.669673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:42:55.610276+0000","last_clean_scrub_stamp":"2024-02-05T06:46:57.669673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":374,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.568710+0000","last_change":"2024-02-05T06:46:40.785712+0000","last_active":"2024-02-05T06:48:13.568710+0000","last_peered":"2024-02-05T06:48:13.568710+0000","last_clean":"2024-02-05T06:48:13.568710+0000","last_became_active":"2024-02-05T06:40:27.086808+0000","last_became_peered":"2024-02-05T06:40:27.086808+0000","last_unstale":"2024-02-05T06:48:13.568710+0000","last_undegraded":"2024-02-05T06:48:13.568710+0000","last_fullsized":"2024-02-05T06:48:13.568710+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:40.785606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:46:40.785606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:15.075166+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"295'24","reported_seq":516,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.570530+0000","last_change":"2024-02-05T06:46:58.206860+0000","last_active":"2024-02-05T06:48:13.570530+0000","last_peered":"2024-02-05T06:48:13.570530+0000","last_clean":"2024-02-05T06:48:13.570530+0000","last_became_active":"2024-02-05T06:45:09.774868+0000","last_became_peered":"2024-02-05T06:45:09.774868+0000","last_unstale":"2024-02-05T06:48:13.570530+0000","last_undegraded":"2024-02-05T06:48:13.570530+0000","last_fullsized":"2024-02-05T06:48:13.570530+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:46:58.206720+0000","last_deep_scrub":"115'16","last_deep_scrub_stamp":"2024-02-05T06:36:34.793243+0000","last_clean_scrub_stamp":"2024-02-05T06:46:58.206720+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.344692+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3","version":"365'4289","reported_seq":4338,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:16.844266+0000","last_change":"2024-02-05T06:47:24.177225+0000","last_active":"2024-02-05T06:48:16.844266+0000","last_peered":"2024-02-05T06:48:16.844266+0000","last_clean":"2024-02-05T06:48:16.844266+0000","last_became_active":"2024-02-05T06:47:24.176139+0000","last_became_peered":"2024-02-05T06:47:24.176139+0000","last_unstale":"2024-02-05T06:48:16.844266+0000","last_undegraded":"2024-02-05T06:48:16.844266+0000","last_fullsized":"2024-02-05T06:48:16.844266+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":4289,"log_dups_size":0,"ondisk_log_size":4289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:43.822725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281083904,"num_objects":4289,"num_object_clones":0,"num_object_copies":12867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4289,"num_write_kb":274496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"83'128","reported_seq":686,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.567987+0000","last_change":"2024-02-05T06:46:44.625857+0000","last_active":"2024-02-05T06:48:13.567987+0000","last_peered":"2024-02-05T06:48:13.567987+0000","last_clean":"2024-02-05T06:48:13.567987+0000","last_became_active":"2024-02-05T06:41:46.053324+0000","last_became_peered":"2024-02-05T06:41:46.053324+0000","last_unstale":"2024-02-05T06:48:13.567987+0000","last_undegraded":"2024-02-05T06:48:13.567987+0000","last_fullsized":"2024-02-05T06:48:13.567987+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:46:44.625739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:46:44.625739+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":677,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.573852+0000","last_change":"2024-02-05T06:47:42.155963+0000","last_active":"2024-02-05T06:48:13.573852+0000","last_peered":"2024-02-05T06:48:13.573852+0000","last_clean":"2024-02-05T06:48:13.573852+0000","last_became_active":"2024-02-05T06:36:54.958805+0000","last_became_peered":"2024-02-05T06:36:54.958805+0000","last_unstale":"2024-02-05T06:48:13.573852+0000","last_undegraded":"2024-02-05T06:48:13.573852+0000","last_fullsized":"2024-02-05T06:48:13.573852+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:42.155801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:47:42.155801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:53.870607+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":666,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.570780+0000","last_change":"2024-02-05T06:46:53.791434+0000","last_active":"2024-02-05T06:48:13.570780+0000","last_peered":"2024-02-05T06:48:13.570780+0000","last_clean":"2024-02-05T06:48:13.570780+0000","last_became_active":"2024-02-05T06:36:54.958082+0000","last_became_peered":"2024-02-05T06:36:54.958082+0000","last_unstale":"2024-02-05T06:48:13.570780+0000","last_undegraded":"2024-02-05T06:48:13.570780+0000","last_fullsized":"2024-02-05T06:48:13.570780+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:53.791322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:46:53.791322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:15.075335+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":679,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.568096+0000","last_change":"2024-02-05T06:46:35.495500+0000","last_active":"2024-02-05T06:48:13.568096+0000","last_peered":"2024-02-05T06:48:13.568096+0000","last_clean":"2024-02-05T06:48:13.568096+0000","last_became_active":"2024-02-05T06:39:13.240038+0000","last_became_peered":"2024-02-05T06:39:13.240038+0000","last_unstale":"2024-02-05T06:48:13.568096+0000","last_undegraded":"2024-02-05T06:48:13.568096+0000","last_fullsized":"2024-02-05T06:48:13.568096+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:35.495370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:46:35.495370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:13.711859+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"295'39","reported_seq":531,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.574354+0000","last_change":"2024-02-05T06:47:41.020322+0000","last_active":"2024-02-05T06:48:13.574354+0000","last_peered":"2024-02-05T06:48:13.574354+0000","last_clean":"2024-02-05T06:48:13.574354+0000","last_became_active":"2024-02-05T06:45:09.782645+0000","last_became_peered":"2024-02-05T06:45:09.782645+0000","last_unstale":"2024-02-05T06:48:13.574354+0000","last_undegraded":"2024-02-05T06:48:13.574354+0000","last_fullsized":"2024-02-05T06:48:13.574354+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'39","last_scrub_stamp":"2024-02-05T06:47:41.020104+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-02-05T06:35:26.274944+0000","last_clean_scrub_stamp":"2024-02-05T06:47:41.020104+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:06.072004+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"365'4196","reported_seq":4246,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:17.065363+0000","last_change":"2024-02-05T06:47:24.182784+0000","last_active":"2024-02-05T06:48:17.065363+0000","last_peered":"2024-02-05T06:48:17.065363+0000","last_clean":"2024-02-05T06:48:17.065363+0000","last_became_active":"2024-02-05T06:47:24.182360+0000","last_became_peered":"2024-02-05T06:47:24.182360+0000","last_unstale":"2024-02-05T06:48:17.065363+0000","last_undegraded":"2024-02-05T06:48:17.065363+0000","last_fullsized":"2024-02-05T06:48:17.065363+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":4196,"log_dups_size":0,"ondisk_log_size":4196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:30.397375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274989056,"num_objects":4196,"num_object_clones":0,"num_object_copies":12588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4196,"num_write_kb":268544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":765,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.579398+0000","last_change":"2024-02-05T06:45:47.191297+0000","last_active":"2024-02-05T06:48:13.579398+0000","last_peered":"2024-02-05T06:48:13.579398+0000","last_clean":"2024-02-05T06:48:13.579398+0000","last_became_active":"2024-02-05T06:40:30.095202+0000","last_became_peered":"2024-02-05T06:40:30.095202+0000","last_unstale":"2024-02-05T06:48:13.579398+0000","last_undegraded":"2024-02-05T06:48:13.579398+0000","last_fullsized":"2024-02-05T06:48:13.579398+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:45:47.191180+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:41:50.706159+0000","last_clean_scrub_stamp":"2024-02-05T06:45:47.191180+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:13.594295+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":185,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.569890+0000","last_change":"2024-02-05T06:47:49.056694+0000","last_active":"2024-02-05T06:48:13.569890+0000","last_peered":"2024-02-05T06:48:13.569890+0000","last_clean":"2024-02-05T06:48:13.569890+0000","last_became_active":"2024-02-05T06:45:09.784229+0000","last_became_peered":"2024-02-05T06:45:09.784229+0000","last_unstale":"2024-02-05T06:48:13.569890+0000","last_undegraded":"2024-02-05T06:48:13.569890+0000","last_fullsized":"2024-02-05T06:48:13.569890+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:49.056561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:30.057099+0000","last_clean_scrub_stamp":"2024-02-05T06:47:49.056561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:05.788674+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"249'13","reported_seq":895,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.574537+0000","last_change":"2024-02-05T06:47:52.101405+0000","last_active":"2024-02-05T06:48:13.574537+0000","last_peered":"2024-02-05T06:48:13.574537+0000","last_clean":"2024-02-05T06:48:13.574537+0000","last_became_active":"2024-02-05T06:41:43.413139+0000","last_became_peered":"2024-02-05T06:41:43.413139+0000","last_unstale":"2024-02-05T06:48:13.574537+0000","last_undegraded":"2024-02-05T06:48:13.574537+0000","last_fullsized":"2024-02-05T06:48:13.574537+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'13","last_scrub_stamp":"2024-02-05T06:47:52.101279+0000","last_deep_scrub":"92'9","last_deep_scrub_stamp":"2024-02-05T06:34:52.283780+0000","last_clean_scrub_stamp":"2024-02-05T06:47:52.101279+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:56.214584+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":658,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.565070+0000","last_change":"2024-02-05T06:45:30.989035+0000","last_active":"2024-02-05T06:48:13.565070+0000","last_peered":"2024-02-05T06:48:13.565070+0000","last_clean":"2024-02-05T06:48:13.565070+0000","last_became_active":"2024-02-05T06:36:54.971304+0000","last_became_peered":"2024-02-05T06:36:54.971304+0000","last_unstale":"2024-02-05T06:48:13.565070+0000","last_undegraded":"2024-02-05T06:48:13.565070+0000","last_fullsized":"2024-02-05T06:48:13.565070+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:30.987881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:45:30.987881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.572599+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"295'25","reported_seq":511,"reported_epoch":365,"state":"active+clean","last_fresh":"2024-02-05T06:48:13.566213+0000","last_change":"2024-02-05T06:45:09.799601+0000","last_active":"2024-02-05T06:48:13.566213+0000","last_peered":"2024-02-05T06:48:13.566213+0000","last_clean":"2024-02-05T06:48:13.566213+0000","last_became_active":"2024-02-05T06:45:09.796902+0000","last_became_peered":"2024-02-05T06:45:09.796902+0000","last_unstale":"2024-02-05T06:48:13.566213+0000","last_undegraded":"2024-02-05T06:48:13.566213+0000","last_fullsized":"2024-02-05T06:48:13.566213+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'25","last_scrub_stamp":"2024-02-05T06:44:50.081252+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-02-05T06:35:04.584657+0000","last_clean_scrub_stamp":"2024-02-05T06:44:50.081252+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:16.572545+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":4409917440,"num_objects":67290,"num_object_clones":0,"num_object_copies":201870,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67290,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67290,"num_write_kb":4306560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13274054656,"data_stored":13274054656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67290,"ondisk_log_size":67290,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":475,"num_read_kb":470,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1573408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1106,"num_read_kb":863,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":373,"num_bytes_recovered":8234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":771,"ondisk_log_size":771,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103349,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1601844,"kb_used_data":1061164,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":92143308,"statfs":{"total":95995035648,"available":94354747392,"internally_reserved":0,"allocated":1086631936,"data_stored":1086129144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":553646538},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":1.135,"15min":1.476},"min":{"1min":0.40000000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.2909999999999999,"5min":4.1100000000000003,"15min":68.921000000000006},"last":3.1320000000000001},{"interface":"front","average":{"1min":0.63700000000000001,"5min":1.2569999999999999,"15min":1.593},"min":{"1min":0.27300000000000002,"5min":0.222,"15min":0.222},"max":{"1min":1.403,"5min":18.013000000000002,"15min":69.616},"last":0.68500000000000005}]},{"osd":1,"last update":"Mon Feb 5 06:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":2.3919999999999999,"15min":1.7789999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.23599999999999999},"max":{"1min":1.504,"5min":84.216999999999999,"15min":89.231999999999999},"last":2.6440000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":2.2480000000000002,"15min":1.879},"min":{"1min":0.34699999999999998,"5min":0.30399999999999999,"15min":0.26100000000000001},"max":{"1min":1.3540000000000001,"5min":84.126999999999995,"15min":87.995999999999995},"last":0.628}]},{"osd":2,"last update":"Mon Feb 5 06:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":1.214,"15min":1.2330000000000001},"min":{"1min":0.32300000000000001,"5min":0.30299999999999999,"15min":0.26200000000000001},"max":{"1min":1.994,"5min":18.071999999999999,"15min":53.947000000000003},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":1.2310000000000001,"15min":1.206},"min":{"1min":0.39500000000000002,"5min":0.29999999999999999,"15min":0.26800000000000002},"max":{"1min":0.92800000000000005,"5min":18.045999999999999,"15min":53.299999999999997},"last":2.9039999999999999}]},{"osd":3,"last update":"Mon Feb 5 06:47:57 2024","interfaces":[{"interface":"back","average":{"1min":4.6980000000000004,"5min":2.371,"15min":2.165},"min":{"1min":0.35699999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":73.283000000000001,"5min":73.283000000000001,"15min":89.481999999999999},"last":0.373},{"interface":"front","average":{"1min":4.7990000000000004,"5min":2.4220000000000002,"15min":2.1259999999999999},"min":{"1min":0.34100000000000003,"5min":0.32900000000000001,"15min":0.27400000000000002},"max":{"1min":73.323999999999998,"5min":73.323999999999998,"15min":92.293999999999997},"last":2.694}]},{"osd":4,"last update":"Mon Feb 5 06:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.81299999999999994,"15min":0.61499999999999999},"min":{"1min":0.29899999999999999,"5min":0.251,"15min":0.24199999999999999},"max":{"1min":1.329,"5min":4.1059999999999999,"15min":4.1059999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.751,"5min":0.82299999999999995,"15min":0.60499999999999998},"min":{"1min":0.35599999999999998,"5min":0.29299999999999998,"15min":0.23400000000000001},"max":{"1min":2.7429999999999999,"5min":3.0819999999999999,"15min":3.0819999999999999},"last":0.312}]},{"osd":5,"last update":"Mon Feb 5 06:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":1.083,"15min":0.72699999999999998},"min":{"1min":0.33700000000000002,"5min":0.251,"15min":0.251},"max":{"1min":1.276,"5min":6.2190000000000003,"15min":6.2190000000000003},"last":2.548},{"interface":"front","average":{"1min":0.71099999999999997,"5min":1.2769999999999999,"15min":0.80700000000000005},"min":{"1min":0.38,"5min":0.313,"15min":0.252},"max":{"1min":1.292,"5min":7.8360000000000003,"15min":7.8360000000000003},"last":0.55700000000000005}]},{"osd":6,"last update":"Mon Feb 5 06:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":1.01,"15min":0.76300000000000001},"min":{"1min":0.43099999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.379,"5min":4.524,"15min":4.524},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.996,"15min":0.71999999999999997},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.245},"max":{"1min":2.7869999999999999,"5min":3.3769999999999998,"15min":3.3769999999999998},"last":1.6799999999999999}]}]},{"osd":6,"up_from":49,"seq":210453397691,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2248288,"kb_used_data":1860308,"kb_used_omap":1,"kb_used_meta":387966,"kb_avail":91496864,"statfs":{"total":95995035648,"available":93692788736,"internally_reserved":0,"allocated":1904955392,"data_stored":1904475727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":397277638},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:47:18 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":2.1400000000000001,"15min":2.1349999999999998},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.224},"max":{"1min":7.6600000000000001,"5min":98.918999999999997,"15min":98.918999999999997},"last":2.851},{"interface":"front","average":{"1min":0.72499999999999998,"5min":2.1269999999999998,"15min":2.077},"min":{"1min":0.26300000000000001,"5min":0.245,"15min":0.23000000000000001},"max":{"1min":1.899,"5min":98.775000000000006,"15min":98.775000000000006},"last":0.95499999999999996}]},{"osd":1,"last update":"Mon Feb 5 06:47:18 2024","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":2.355,"15min":1.8999999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.24099999999999999},"max":{"1min":4.2850000000000001,"5min":98.843999999999994,"15min":98.843999999999994},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.99299999999999999,"5min":2.379,"15min":1.8740000000000001},"min":{"1min":0.29199999999999998,"5min":0.26500000000000001,"15min":0.23100000000000001},"max":{"1min":4.4779999999999998,"5min":101.637,"15min":101.637},"last":2.8780000000000001}]},{"osd":2,"last update":"Mon Feb 5 06:47:18 2024","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.7050000000000001,"15min":1.8160000000000001},"min":{"1min":0.28499999999999998,"5min":0.26900000000000002,"15min":0.22500000000000001},"max":{"1min":7.726,"5min":74.454999999999998,"15min":85.617000000000004},"last":2.9239999999999999},{"interface":"front","average":{"1min":1.367,"5min":1.5149999999999999,"15min":1.722},"min":{"1min":0.29899999999999999,"5min":0.28499999999999998,"15min":0.27000000000000002},"max":{"1min":7.6970000000000001,"5min":74.546000000000006,"15min":85.335999999999999},"last":0.98099999999999998}]},{"osd":3,"last update":"Mon Feb 5 06:47:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":2.673,"15min":2.3439999999999999},"min":{"1min":0.32900000000000001,"5min":0.307,"15min":0.27000000000000002},"max":{"1min":3.1640000000000001,"5min":98.888999999999996,"15min":99.269000000000005},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":2.5099999999999998,"15min":2.4079999999999999},"min":{"1min":0.38600000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.409,"5min":98.969999999999999,"15min":101.75700000000001},"last":2.9020000000000001}]},{"osd":4,"last update":"Mon Feb 5 06:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.65700000000000003,"15min":0.53000000000000003},"min":{"1min":0.314,"5min":0.249,"15min":0.23100000000000001},"max":{"1min":0.98799999999999999,"5min":3.742,"15min":3.742},"last":3.3159999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.65000000000000002,"15min":0.53100000000000003},"min":{"1min":0.249,"5min":0.249,"15min":0.24299999999999999},"max":{"1min":3.117,"5min":4.4379999999999997,"15min":4.4379999999999997},"last":0.72399999999999998}]},{"osd":5,"last update":"Mon Feb 5 06:47:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":1.248,"15min":0.74399999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.251},"max":{"1min":1.0269999999999999,"5min":36.548999999999999,"15min":36.548999999999999},"last":1.2450000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":1.3320000000000001,"15min":0.78300000000000003},"min":{"1min":0.309,"5min":0.308,"15min":0.24299999999999999},"max":{"1min":3.2010000000000001,"5min":36.433,"15min":36.433},"last":0.80200000000000005}]},{"osd":7,"last update":"Mon Feb 5 06:47:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":1.1870000000000001,"15min":0.76000000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.30599999999999999},"max":{"1min":1.278,"5min":38.436999999999998,"15min":38.436999999999998},"last":2.8119999999999998},{"interface":"front","average":{"1min":0.70999999999999996,"5min":1.117,"15min":0.77800000000000002},"min":{"1min":0.378,"5min":0.33000000000000002,"15min":0.29699999999999999},"max":{"1min":2.2850000000000001,"5min":36.633000000000003,"15min":36.633000000000003},"last":0.91900000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575060,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1374100,"kb_used_data":807052,"kb_used_omap":1,"kb_used_meta":567038,"kb_avail":92371052,"statfs":{"total":95995035648,"available":94587957248,"internally_reserved":0,"allocated":826421248,"data_stored":825886471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":580647365},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:47:22 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":1.1739999999999999,"15min":1.752},"min":{"1min":0.22500000000000001,"5min":0.218,"15min":0.16700000000000001},"max":{"1min":5.5529999999999999,"5min":7.4370000000000003,"15min":67.421000000000006},"last":3.5049999999999999},{"interface":"front","average":{"1min":0.68500000000000005,"5min":1.264,"15min":1.885},"min":{"1min":0.27000000000000002,"5min":0.20300000000000001,"15min":0.19600000000000001},"max":{"1min":2.9239999999999999,"5min":10.333,"15min":67.885000000000005},"last":3.8479999999999999}]},{"osd":2,"last update":"Mon Feb 5 06:47:38 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":1.0569999999999999,"15min":1.452},"min":{"1min":0.255,"5min":0.249,"15min":0.21099999999999999},"max":{"1min":5.4969999999999999,"5min":17.422000000000001,"15min":43.447000000000003},"last":3.298},{"interface":"front","average":{"1min":0.442,"5min":1.081,"15min":1.429},"min":{"1min":0.22700000000000001,"5min":0.22600000000000001,"15min":0.20899999999999999},"max":{"1min":1.0489999999999999,"5min":22.050000000000001,"15min":22.050000000000001},"last":3.4350000000000001}]},{"osd":3,"last update":"Mon Feb 5 06:48:09 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":1.417,"15min":2.157},"min":{"1min":0.27400000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":2.3359999999999999,"5min":10.456,"15min":69.798000000000002},"last":3.7639999999999998},{"interface":"front","average":{"1min":0.69699999999999995,"5min":1.2310000000000001,"15min":1.8680000000000001},"min":{"1min":0.31,"5min":0.27400000000000002,"15min":0.23699999999999999},"max":{"1min":2.056,"5min":5.8520000000000003,"15min":70.765000000000001},"last":3.4580000000000002}]},{"osd":4,"last update":"Mon Feb 5 06:47:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.97199999999999998,"15min":1.095},"min":{"1min":0.40100000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.99399999999999999,"5min":6.3360000000000003,"15min":11.679},"last":3.5209999999999999},{"interface":"front","average":{"1min":0.70199999999999996,"5min":1.3420000000000001,"15min":1.212},"min":{"1min":0.35899999999999999,"5min":0.30599999999999999,"15min":0.26300000000000001},"max":{"1min":2.3490000000000002,"5min":10.419,"15min":10.419},"last":3.363}]},{"osd":5,"last update":"Mon Feb 5 06:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":1.163,"15min":1.232},"min":{"1min":0.44900000000000001,"5min":0.32900000000000001,"15min":0.29299999999999998},"max":{"1min":1.4570000000000001,"5min":5.5830000000000002,"15min":43.414999999999999},"last":3.343},{"interface":"front","average":{"1min":0.80000000000000004,"5min":1.1919999999999999,"15min":1.254},"min":{"1min":0.36199999999999999,"5min":0.35699999999999998,"15min":0.33100000000000002},"max":{"1min":2.7669999999999999,"5min":6.3849999999999998,"15min":13.455},"last":3.5790000000000002}]},{"osd":6,"last update":"Mon Feb 5 06:47:22 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":1.1879999999999999,"15min":1.2869999999999999},"min":{"1min":0.39000000000000001,"5min":0.376,"15min":0.34799999999999998},"max":{"1min":1.0409999999999999,"5min":23.902999999999999,"15min":23.902999999999999},"last":3.5550000000000002},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.149,"15min":1.232},"min":{"1min":0.438,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":5.5250000000000004,"5min":5.5250000000000004,"15min":43.475000000000001},"last":3.3780000000000001}]},{"osd":7,"last update":"Mon Feb 5 06:47:38 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":1.2889999999999999,"15min":1.2929999999999999},"min":{"1min":0.45900000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.8779999999999999,"5min":8.016,"15min":13.49},"last":3.4740000000000002},{"interface":"front","average":{"1min":0.628,"5min":1.425,"15min":1.3400000000000001},"min":{"1min":0.41699999999999998,"5min":0.40400000000000003,"15min":0.35499999999999998},"max":{"1min":1.6259999999999999,"5min":10.481,"15min":10.481},"last":3.415}]}]},{"osd":0,"up_from":8,"seq":34359738586,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3193924,"kb_used_data":2685180,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":90551228,"statfs":{"total":95995035648,"available":92724457472,"internally_reserved":0,"allocated":2749624320,"data_stored":2749105120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 06:48:13 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":1.1619999999999999,"15min":1.464},"min":{"1min":0.22600000000000001,"5min":0.186,"15min":0.182},"max":{"1min":2.3439999999999999,"5min":18.311,"15min":18.59},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.381,"15min":1.4850000000000001},"min":{"1min":0.23799999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.4020000000000001,"5min":21.088000000000001,"15min":21.088000000000001},"last":0.79900000000000004}]},{"osd":2,"last update":"Mon Feb 5 06:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.86299999999999999,"15min":1.333},"min":{"1min":0.24199999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":3.661,"5min":3.661,"15min":16.981999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.88400000000000001,"15min":1.262},"min":{"1min":0.27900000000000003,"5min":0.221,"15min":0.221},"max":{"1min":7.093,"5min":7.093,"15min":16.998999999999999},"last":1.1160000000000001}]},{"osd":3,"last update":"Mon Feb 5 06:48:05 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":1.175,"15min":1.6359999999999999},"min":{"1min":0.314,"5min":0.27300000000000002,"15min":0.25700000000000001},"max":{"1min":2.032,"5min":18.349,"15min":43.302},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.86099999999999999,"5min":1.3300000000000001,"15min":1.8340000000000001},"min":{"1min":0.33500000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.1390000000000002,"5min":18.712,"15min":43.651000000000003},"last":1.079}]},{"osd":4,"last update":"Mon Feb 5 06:47:24 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":0.96099999999999997,"15min":1.2649999999999999},"min":{"1min":0.41999999999999998,"5min":0.312,"15min":0.27200000000000002},"max":{"1min":3.6179999999999999,"5min":3.6179999999999999,"15min":40.789999999999999},"last":3.0249999999999999},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.97699999999999998,"15min":1.3220000000000001},"min":{"1min":0.45400000000000001,"5min":0.33600000000000002,"15min":0.30099999999999999},"max":{"1min":2.9489999999999998,"5min":8.4019999999999992,"15min":40.773000000000003},"last":3.0409999999999999}]},{"osd":5,"last update":"Mon Feb 5 06:47:52 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":1.728,"15min":1.401},"min":{"1min":0.38900000000000001,"5min":0.32500000000000001,"15min":0.29299999999999998},"max":{"1min":1.252,"5min":48.841999999999999,"15min":48.841999999999999},"last":1.0289999999999999},{"interface":"front","average":{"1min":0.93899999999999995,"5min":1.619,"15min":1.373},"min":{"1min":0.47199999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":4.3449999999999998,"5min":48.450000000000003,"15min":48.450000000000003},"last":0.83499999999999996}]},{"osd":6,"last update":"Mon Feb 5 06:48:13 2024","interfaces":[{"interface":"back","average":{"1min":9.0020000000000007,"5min":2.54,"15min":1.746},"min":{"1min":0.42099999999999999,"5min":0.376,"15min":0.33700000000000002},"max":{"1min":136.76300000000001,"5min":136.76300000000001,"15min":136.76300000000001},"last":7.0049999999999999},{"interface":"front","average":{"1min":9.1989999999999998,"5min":2.8260000000000001,"15min":1.8},"min":{"1min":0.51300000000000001,"5min":0.39200000000000002,"15min":0.33000000000000002},"max":{"1min":135.92400000000001,"5min":135.92400000000001,"15min":135.92400000000001},"last":2.0590000000000002}]},{"osd":7,"last update":"Mon Feb 5 06:47:41 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":1.081,"15min":1.204},"min":{"1min":0.434,"5min":0.40100000000000002,"15min":0.31900000000000001},"max":{"1min":2.391,"5min":6.1420000000000003,"15min":17.210000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.98199999999999998,"15min":1.276},"min":{"1min":0.435,"5min":0.378,"15min":0.34899999999999998},"max":{"1min":1.496,"5min":6.7220000000000004,"15min":18.533999999999999},"last":0.97199999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411535,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2017260,"kb_used_data":1598756,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":91727892,"statfs":{"total":95995035648,"available":93929361408,"internally_reserved":0,"allocated":1637126144,"data_stored":1636631957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:47:39 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":1.0600000000000001,"15min":1.72},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.0259999999999998,"5min":14.007,"15min":50.814},"last":0.218},{"interface":"front","average":{"1min":0.77600000000000002,"5min":1.1379999999999999,"15min":1.7230000000000001},"min":{"1min":0.23100000000000001,"5min":0.22600000000000001,"15min":0.17699999999999999},"max":{"1min":4.7480000000000002,"5min":13.968999999999999,"15min":50.802999999999997},"last":0.51100000000000001}]},{"osd":1,"last update":"Mon Feb 5 06:47:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":1.482,"15min":1.6259999999999999},"min":{"1min":0.19700000000000001,"5min":0.189,"15min":0.189},"max":{"1min":3.1429999999999998,"5min":15.382,"15min":31.303000000000001},"last":0.251},{"interface":"front","average":{"1min":1.028,"5min":1.54,"15min":1.3979999999999999},"min":{"1min":0.27700000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.3019999999999996,"5min":15.742000000000001,"15min":31.425000000000001},"last":0.46100000000000002}]},{"osd":3,"last update":"Mon Feb 5 06:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":1.153,"15min":1.8320000000000001},"min":{"1min":0.316,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.591,"5min":15.166,"15min":53.127000000000002},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.1299999999999999,"15min":1.7529999999999999},"min":{"1min":0.30199999999999999,"5min":0.23400000000000001,"15min":0.22500000000000001},"max":{"1min":2.2930000000000001,"5min":15.541,"15min":52.607999999999997},"last":1.194}]},{"osd":4,"last update":"Mon Feb 5 06:47:24 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.87,"15min":1.004},"min":{"1min":0.34200000000000003,"5min":0.29999999999999999,"15min":0.23999999999999999},"max":{"1min":2.8610000000000002,"5min":7.0110000000000001,"15min":12.041},"last":0.432},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.78600000000000003,"15min":0.95699999999999996},"min":{"1min":0.36099999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.625,"5min":5.0350000000000001,"15min":15.689},"last":1.2330000000000001}]},{"osd":5,"last update":"Mon Feb 5 06:47:54 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":1.2330000000000001,"15min":1.4750000000000001},"min":{"1min":0.33800000000000002,"5min":0.315,"15min":0.29899999999999999},"max":{"1min":1.2729999999999999,"5min":30.265000000000001,"15min":86.614999999999995},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.89400000000000002,"5min":1.373,"15min":1.6990000000000001},"min":{"1min":0.36599999999999999,"5min":0.312,"15min":0.312},"max":{"1min":3.222,"5min":30.02,"15min":86.543999999999997},"last":0.35599999999999998}]},{"osd":6,"last update":"Mon Feb 5 06:47:24 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.83199999999999996,"15min":1.1020000000000001},"min":{"1min":0.42799999999999999,"5min":0.35499999999999998,"15min":0.315},"max":{"1min":1.6599999999999999,"5min":4.9420000000000002,"15min":17.143999999999998},"last":1.2689999999999999},{"interface":"front","average":{"1min":0.97899999999999998,"5min":1.1799999999999999,"15min":1.2729999999999999},"min":{"1min":0.38400000000000001,"5min":0.316,"15min":0.316},"max":{"1min":3.1819999999999999,"5min":14.416,"15min":14.416},"last":1.2090000000000001}]},{"osd":7,"last update":"Mon Feb 5 06:47:43 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":1.6859999999999999,"15min":1.4510000000000001},"min":{"1min":0.495,"5min":0.373,"15min":0.35699999999999998},"max":{"1min":3.2530000000000001,"5min":45.243000000000002,"15min":45.243000000000002},"last":0.48599999999999999},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.702,"15min":1.3520000000000001},"min":{"1min":0.40799999999999997,"5min":0.379,"15min":0.36899999999999999},"max":{"1min":17.611000000000001,"5min":45.171999999999997,"15min":45.171999999999997},"last":0.95499999999999996}]}]},{"osd":3,"up_from":26,"seq":111669149898,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1971520,"kb_used_data":1354488,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":91773632,"statfs":{"total":95995035648,"available":93976199168,"internally_reserved":0,"allocated":1386995712,"data_stored":1386480508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":631830983},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.3970000000000002,"5min":1.9510000000000001,"15min":1.5349999999999999},"min":{"1min":0.21099999999999999,"5min":0.189,"15min":0.184},"max":{"1min":53.656999999999996,"5min":53.656999999999996,"15min":53.656999999999996},"last":53.656999999999996},{"interface":"front","average":{"1min":4.1260000000000003,"5min":1.8280000000000001,"15min":1.736},"min":{"1min":0.217,"5min":0.217,"15min":0.20399999999999999},"max":{"1min":53.338000000000001,"5min":53.338000000000001,"15min":53.338000000000001},"last":53.338000000000001}]},{"osd":1,"last update":"Mon Feb 5 06:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":1.4370000000000001,"15min":1.4950000000000001},"min":{"1min":0.29199999999999998,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":2.157,"5min":11.412000000000001,"15min":11.412000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":1.419,"15min":1.518},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.215},"max":{"1min":1.5680000000000001,"5min":9.8659999999999997,"15min":9.8659999999999997},"last":0.315}]},{"osd":2,"last update":"Mon Feb 5 06:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":1.157,"15min":1.5649999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.3540000000000001,"5min":7.7080000000000002,"15min":41.529000000000003},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.623,"5min":1.0640000000000001,"15min":1.4570000000000001},"min":{"1min":0.28499999999999998,"5min":0.249,"15min":0.248},"max":{"1min":2.1949999999999998,"5min":7.3559999999999999,"15min":41.759999999999998},"last":0.28499999999999998}]},{"osd":4,"last update":"Mon Feb 5 06:47:34 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":1.24,"15min":1.109},"min":{"1min":0.41999999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.8360000000000001,"5min":7.7329999999999997,"15min":16.577999999999999},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":1.607,"15min":1.369},"min":{"1min":0.371,"5min":0.30099999999999999,"15min":0.28599999999999998},"max":{"1min":1.831,"5min":19.359999999999999,"15min":19.359999999999999},"last":0.36299999999999999}]},{"osd":5,"last update":"Mon Feb 5 06:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":1.464,"15min":1.4339999999999999},"min":{"1min":0.46600000000000003,"5min":0.32000000000000001,"15min":0.29299999999999998},"max":{"1min":1.5589999999999999,"5min":20.257999999999999,"15min":20.257999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.626,"5min":1.3140000000000001,"15min":1.181},"min":{"1min":0.441,"5min":0.38700000000000001,"15min":0.28399999999999997},"max":{"1min":0.95999999999999996,"5min":7.859,"15min":15.792999999999999},"last":0.35099999999999998}]},{"osd":6,"last update":"Mon Feb 5 06:47:27 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":1.429,"15min":1.319},"min":{"1min":0.46100000000000002,"5min":0.34999999999999998,"15min":0.28100000000000003},"max":{"1min":3.5630000000000002,"5min":7.556,"15min":8.3610000000000007},"last":0.38700000000000001},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.284,"15min":1.276},"min":{"1min":0.46300000000000002,"5min":0.44900000000000001,"15min":0.28599999999999998},"max":{"1min":2.4550000000000001,"5min":7.2439999999999998,"15min":7.2439999999999998},"last":0.52400000000000002}]},{"osd":7,"last update":"Mon Feb 5 06:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":1.3600000000000001,"15min":1.335},"min":{"1min":0.376,"5min":0.376,"15min":0.33400000000000002},"max":{"1min":1.1950000000000001,"5min":7.8029999999999999,"15min":45.039000000000001},"last":0.374},{"interface":"front","average":{"1min":0.75800000000000001,"5min":1.3220000000000001,"15min":1.4770000000000001},"min":{"1min":0.51500000000000001,"5min":0.377,"15min":0.34399999999999997},"max":{"1min":1.349,"5min":7.2720000000000002,"15min":44.036999999999999},"last":0.44800000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986372,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2608380,"kb_used_data":2117684,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":91136772,"statfs":{"total":95995035648,"available":93324054528,"internally_reserved":0,"allocated":2168508416,"data_stored":2167996986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:47:28 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.83499999999999996,"15min":1.151},"min":{"1min":0.28499999999999998,"5min":0.20799999999999999,"15min":0.20200000000000001},"max":{"1min":3.0910000000000002,"5min":6.0220000000000002,"15min":33.372},"last":2.2109999999999999},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.95899999999999996,"15min":1.2829999999999999},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.22700000000000001},"max":{"1min":2.9830000000000001,"5min":10.913,"15min":33.323999999999998},"last":0.28199999999999997}]},{"osd":1,"last update":"Mon Feb 5 06:47:28 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.177,"15min":1.0900000000000001},"min":{"1min":0.28399999999999997,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":17.283000000000001,"5min":17.283000000000001,"15min":33.747999999999998},"last":0.42999999999999999},{"interface":"front","average":{"1min":1.454,"5min":1.1040000000000001,"15min":1.0620000000000001},"min":{"1min":0.255,"5min":0.23699999999999999,"15min":0.23000000000000001},"max":{"1min":17.460999999999999,"5min":17.460999999999999,"15min":33.691000000000003},"last":0.58399999999999996}]},{"osd":2,"last update":"Mon Feb 5 06:47:27 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.88500000000000001,"15min":0.80200000000000005},"min":{"1min":0.29199999999999998,"5min":0.27500000000000002,"15min":0.245},"max":{"1min":1.8160000000000001,"5min":17.379999999999999,"15min":17.379999999999999},"last":1.9490000000000001},{"interface":"front","average":{"1min":0.72799999999999998,"5min":1.2,"15min":1.04},"min":{"1min":0.35499999999999998,"5min":0.29799999999999999,"15min":0.26000000000000001},"max":{"1min":2.6099999999999999,"5min":18.077000000000002,"15min":18.077000000000002},"last":3.1440000000000001}]},{"osd":3,"last update":"Mon Feb 5 06:47:28 2024","interfaces":[{"interface":"back","average":{"1min":1.893,"5min":2.3780000000000001,"15min":2.3740000000000001},"min":{"1min":0.36799999999999999,"5min":0.32000000000000001,"15min":0.252},"max":{"1min":20.562999999999999,"5min":44.601999999999997,"15min":99.546999999999997},"last":0.38600000000000001},{"interface":"front","average":{"1min":1.5840000000000001,"5min":2.3450000000000002,"15min":2.3650000000000002},"min":{"1min":0.33200000000000002,"5min":0.29799999999999999,"15min":0.29099999999999998},"max":{"1min":17.332000000000001,"5min":44.511000000000003,"15min":99.245000000000005},"last":0.46000000000000002}]},{"osd":5,"last update":"Mon Feb 5 06:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.87,"15min":0.69099999999999995},"min":{"1min":0.249,"5min":0.249,"15min":0.24399999999999999},"max":{"1min":1.4930000000000001,"5min":5.6059999999999999,"15min":11.098000000000001},"last":3.1800000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.82599999999999996,"15min":0.66000000000000003},"min":{"1min":0.28899999999999998,"5min":0.28499999999999998,"15min":0.252},"max":{"1min":2.5339999999999998,"5min":4.1210000000000004,"15min":10.965999999999999},"last":0.55600000000000005}]},{"osd":6,"last update":"Mon Feb 5 06:47:27 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.87,"15min":0.71299999999999997},"min":{"1min":0.33900000000000002,"5min":0.31,"15min":0.246},"max":{"1min":3.2829999999999999,"5min":10.039999999999999,"15min":11.868},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.751,"5min":0.83799999999999997,"15min":0.71399999999999997},"min":{"1min":0.35199999999999998,"5min":0.32700000000000001,"15min":0.27600000000000002},"max":{"1min":3.25,"5min":10.074,"15min":11.787000000000001},"last":0.52800000000000002}]},{"osd":7,"last update":"Mon Feb 5 06:47:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.80600000000000005,"15min":0.65400000000000003},"min":{"1min":0.32900000000000001,"5min":0.316,"15min":0.26000000000000001},"max":{"1min":1.7749999999999999,"5min":3.4129999999999998,"15min":5.1500000000000004},"last":0.497},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.81299999999999994,"15min":0.68899999999999995},"min":{"1min":0.314,"5min":0.307,"15min":0.28499999999999998},"max":{"1min":2.9009999999999998,"5min":4.5350000000000001,"15min":5.5890000000000004},"last":3.2090000000000001}]}]},{"osd":5,"up_from":40,"seq":171798692031,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1981164,"kb_used_data":1354404,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":91763988,"statfs":{"total":95995035648,"available":93966323712,"internally_reserved":0,"allocated":1386909696,"data_stored":1386415403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":641792454},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.079,"15min":1.492},"min":{"1min":0.33900000000000002,"5min":0.24299999999999999,"15min":0.19700000000000001},"max":{"1min":4.2869999999999999,"5min":4.6479999999999997,"15min":55.704999999999998},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.02,"5min":1.052,"15min":1.681},"min":{"1min":0.379,"5min":0.255,"15min":0.22600000000000001},"max":{"1min":5.2030000000000003,"5min":6.532,"15min":56.789000000000001},"last":6.1379999999999999}]},{"osd":1,"last update":"Mon Feb 5 06:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":1.1160000000000001,"15min":1.389},"min":{"1min":0.378,"5min":0.26900000000000002,"15min":0.248},"max":{"1min":4.6340000000000003,"5min":6.5049999999999999,"15min":41.191000000000003},"last":6.0739999999999998},{"interface":"front","average":{"1min":0.89000000000000001,"5min":1.153,"15min":1.252},"min":{"1min":0.38200000000000001,"5min":0.28100000000000003,"15min":0.23499999999999999},"max":{"1min":4.8609999999999998,"5min":4.8609999999999998,"15min":41.173000000000002},"last":1.204}]},{"osd":2,"last update":"Mon Feb 5 06:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.84399999999999997,"15min":1.25},"min":{"1min":0.33900000000000002,"5min":0.26800000000000002,"15min":0.23000000000000001},"max":{"1min":0.75700000000000001,"5min":4.0599999999999996,"15min":36.901000000000003},"last":3.2309999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.95999999999999996,"15min":1.2030000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.94599999999999995,"5min":6.4619999999999997,"15min":37.984000000000002},"last":3.0990000000000002}]},{"osd":3,"last update":"Mon Feb 5 06:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":1.716,"15min":1.585},"min":{"1min":0.34399999999999997,"5min":0.33200000000000002,"15min":0.31900000000000001},"max":{"1min":1.0620000000000001,"5min":62.508000000000003,"15min":62.508000000000003},"last":3.1680000000000001},{"interface":"front","average":{"1min":0.79300000000000004,"5min":1.929,"15min":1.748},"min":{"1min":0.36499999999999999,"5min":0.28100000000000003,"15min":0.27400000000000002},"max":{"1min":4.4820000000000002,"5min":62.460000000000001,"15min":62.460000000000001},"last":3.069}]},{"osd":4,"last update":"Mon Feb 5 06:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.83699999999999997,"15min":0.60399999999999998},"min":{"1min":0.33400000000000002,"5min":0.26200000000000001,"15min":0.23200000000000001},"max":{"1min":1.099,"5min":4.7370000000000001,"15min":5.4969999999999999},"last":1.23},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.79900000000000004,"15min":0.61699999999999999},"min":{"1min":0.35099999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.61199999999999999,"5min":4.0380000000000003,"15min":4.0380000000000003},"last":3.206}]},{"osd":6,"last update":"Mon Feb 5 06:47:23 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.98099999999999998,"15min":0.69499999999999995},"min":{"1min":0.32500000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.8109999999999999,"5min":7.6710000000000003,"15min":7.6710000000000003},"last":3.1400000000000001},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.95699999999999996,"15min":0.73899999999999999},"min":{"1min":0.35499999999999998,"5min":0.31900000000000001,"15min":0.27600000000000002},"max":{"1min":1.7549999999999999,"5min":7.21,"15min":7.21},"last":1.3520000000000001}]},{"osd":7,"last update":"Mon Feb 5 06:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":1.073,"15min":0.72299999999999998},"min":{"1min":0.38500000000000001,"5min":0.30199999999999999,"15min":0.25600000000000001},"max":{"1min":0.88200000000000001,"5min":6.9870000000000001,"15min":6.9870000000000001},"last":3.2949999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":1.0680000000000001,"15min":0.75600000000000001},"min":{"1min":0.36599999999999999,"5min":0.28999999999999998,"15min":0.27100000000000002},"max":{"1min":0.68999999999999995,"5min":6.5839999999999996,"15min":6.5839999999999996},"last":3.032}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2793799680,"data_stored":2793799680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":825819136,"data_stored":825819136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1661599744,"data_stored":1661599744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1411579904,"data_stored":1411579904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2172452864,"data_stored":2172452864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1386676224,"data_stored":1386676224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1935802368,"data_stored":1935802368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1086324736,"data_stored":1086324736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T06:48:20.978 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T06:48:20.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:20.983 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:20.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:20.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:20 smithi167 ceph-mon[32774]: 4.18 scrub starts 2024-02-05T06:48:20.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:20 smithi167 ceph-mon[32774]: 4.18 scrub ok 2024-02-05T06:48:20.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:20 smithi167 ceph-mon[32774]: from='client.15771 v1:172.21.15.28:0/1132243188' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:48:20.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:20 smithi167 ceph-mon[32774]: 5.8 scrub starts 2024-02-05T06:48:20.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:20 smithi167 ceph-mon[32774]: 5.8 scrub ok 2024-02-05T06:48:21.084 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:21.123 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:21.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:21.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:21.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:21.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:21.266 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:21.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:21.367 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:21.407 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:21.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:21.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:21.509 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:21.532 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:21.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:21.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:21.634 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:21.668 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:21.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:21.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:21.770 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:21.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:21.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:21.908 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:21.915 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:21 smithi028 ceph-mon[37984]: pgmap v711: 158 pgs: 158 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-02-05T06:48:21.915 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:21 smithi028 ceph-mon[37984]: 3.1f scrub starts 2024-02-05T06:48:21.915 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:21 smithi028 ceph-mon[37984]: 3.1f scrub ok 2024-02-05T06:48:21.915 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:21 smithi028 ceph-mon[37984]: 6.1c scrub starts 2024-02-05T06:48:21.915 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:21 smithi028 ceph-mon[37984]: 6.1c scrub ok 2024-02-05T06:48:21.916 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:21 smithi028 ceph-mon[37984]: 5.7 scrub starts 2024-02-05T06:48:21.916 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:21 smithi028 ceph-mon[37984]: 5.7 scrub ok 2024-02-05T06:48:21.916 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:21 smithi028 ceph-mon[37984]: 4.2 scrub starts 2024-02-05T06:48:21.916 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:21 smithi028 ceph-mon[37984]: 4.2 scrub ok 2024-02-05T06:48:21.916 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:21 smithi028 ceph-mon[29442]: pgmap v711: 158 pgs: 158 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-02-05T06:48:21.916 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:21 smithi028 ceph-mon[29442]: 3.1f scrub starts 2024-02-05T06:48:21.916 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:21 smithi028 ceph-mon[29442]: 3.1f scrub ok 2024-02-05T06:48:21.916 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:21 smithi028 ceph-mon[29442]: 6.1c scrub starts 2024-02-05T06:48:21.916 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:21 smithi028 ceph-mon[29442]: 6.1c scrub ok 2024-02-05T06:48:21.916 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:21 smithi028 ceph-mon[29442]: 5.7 scrub starts 2024-02-05T06:48:21.916 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:21 smithi028 ceph-mon[29442]: 5.7 scrub ok 2024-02-05T06:48:21.916 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:21 smithi028 ceph-mon[29442]: 4.2 scrub starts 2024-02-05T06:48:21.916 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:21 smithi028 ceph-mon[29442]: 4.2 scrub ok 2024-02-05T06:48:21.945 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 42 16 73123 73107 108.763 73.5 0.0104636 0.00918927 2024-02-05T06:48:21.946 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:21.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:21.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:21.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:21 smithi167 ceph-mon[32774]: pgmap v711: 158 pgs: 158 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-02-05T06:48:21.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:21 smithi167 ceph-mon[32774]: 3.1f scrub starts 2024-02-05T06:48:21.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:21 smithi167 ceph-mon[32774]: 3.1f scrub ok 2024-02-05T06:48:21.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:21 smithi167 ceph-mon[32774]: 6.1c scrub starts 2024-02-05T06:48:21.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:21 smithi167 ceph-mon[32774]: 6.1c scrub ok 2024-02-05T06:48:21.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:21 smithi167 ceph-mon[32774]: 5.7 scrub starts 2024-02-05T06:48:21.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:21 smithi167 ceph-mon[32774]: 5.7 scrub ok 2024-02-05T06:48:21.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:21 smithi167 ceph-mon[32774]: 4.2 scrub starts 2024-02-05T06:48:21.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:21 smithi167 ceph-mon[32774]: 4.2 scrub ok 2024-02-05T06:48:22.011 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:48:22.048 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:22.076 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:22.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:22.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:22.178 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:22.215 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:22.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:22.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:22.317 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:22.348 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:22.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:22.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:22.450 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:22.493 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:22.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:22.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:22.595 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:22.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:22.630 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:22.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:22.731 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:22.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:22 smithi167 ceph-mon[32774]: 5.4 scrub starts 2024-02-05T06:48:22.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:22 smithi167 ceph-mon[32774]: 5.4 scrub ok 2024-02-05T06:48:22.771 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:22.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:22.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:22.873 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:22.903 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:22.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:22.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:22.945 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 43 16 74360 74344 108.031 77.3125 0.076304 0.00924953 2024-02-05T06:48:22.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:22 smithi028 ceph-mon[37984]: 5.4 scrub starts 2024-02-05T06:48:22.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:22 smithi028 ceph-mon[37984]: 5.4 scrub ok 2024-02-05T06:48:22.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:22 smithi028 ceph-mon[29442]: 5.4 scrub starts 2024-02-05T06:48:22.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:22 smithi028 ceph-mon[29442]: 5.4 scrub ok 2024-02-05T06:48:22.951 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:48:22 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:48:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:48:22.955 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:48:22.955 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-02-05T06:48:22.955 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-02-05T06:48:22.955 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-02-05T06:48:22.956 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:48:23.005 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:23.044 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:23.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:23.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:23.146 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:23.176 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:23.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:23.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:23.205 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:48:23.277 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:23.318 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:23.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:23.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:23.420 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:23.457 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:23.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:23.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:23.485 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:48:23 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:48:23.559 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:23.601 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:23.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:23.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:23.685 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-05T06:48:23.702 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:23.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:23.780 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:23.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:23.880 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:23.883 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:23 smithi028 ceph-mon[29442]: pgmap v712: 158 pgs: 158 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-02-05T06:48:23.883 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:23 smithi028 ceph-mon[29442]: 4.8 scrub starts 2024-02-05T06:48:23.883 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:23 smithi028 ceph-mon[29442]: 4.8 scrub ok 2024-02-05T06:48:23.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:23 smithi028 ceph-mon[29442]: 6.17 scrub starts 2024-02-05T06:48:23.884 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:23 smithi028 ceph-mon[37984]: pgmap v712: 158 pgs: 158 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-02-05T06:48:23.884 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:23 smithi028 ceph-mon[37984]: 4.8 scrub starts 2024-02-05T06:48:23.884 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:23 smithi028 ceph-mon[37984]: 4.8 scrub ok 2024-02-05T06:48:23.884 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:23 smithi028 ceph-mon[37984]: 6.17 scrub starts 2024-02-05T06:48:23.921 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:23.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:23.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:23.945 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 44 16 75548 75532 107.263 74.25 0.00442707 0.00931755 2024-02-05T06:48:23.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:23 smithi167 ceph-mon[32774]: pgmap v712: 158 pgs: 158 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-02-05T06:48:23.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:23 smithi167 ceph-mon[32774]: 4.8 scrub starts 2024-02-05T06:48:23.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:23 smithi167 ceph-mon[32774]: 4.8 scrub ok 2024-02-05T06:48:23.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:23 smithi167 ceph-mon[32774]: 6.17 scrub starts 2024-02-05T06:48:24.022 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:24.078 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:24.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:24.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:24.145 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:48:24.180 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:24.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:24.220 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:24.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:24.321 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:24.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:24.363 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:24.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:24.455 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:48:24.464 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:24.509 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:24.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:24.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:24.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:24.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:24.660 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:24.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:24.760 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:24.794 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:24.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:24.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:24.896 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:24.934 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:24.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:24.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:24.945 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 45 16 76766 76750 106.571 76.125 0.0100265 0.00937801 2024-02-05T06:48:24.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:24 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:48:24.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:24 smithi028 ceph-mon[37984]: 6.17 scrub ok 2024-02-05T06:48:24.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:24 smithi028 ceph-mon[37984]: 6.19 scrub starts 2024-02-05T06:48:24.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:24 smithi028 ceph-mon[37984]: 4.b scrub starts 2024-02-05T06:48:24.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:24 smithi028 ceph-mon[37984]: 4.b scrub ok 2024-02-05T06:48:24.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:24 smithi028 ceph-mon[37984]: 6.19 scrub ok 2024-02-05T06:48:24.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:24 smithi028 ceph-mon[37984]: 6.12 scrub starts 2024-02-05T06:48:24.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:24 smithi028 ceph-mon[37984]: 6.12 scrub ok 2024-02-05T06:48:24.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:24 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:48:24.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:24 smithi028 ceph-mon[29442]: 6.17 scrub ok 2024-02-05T06:48:24.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:24 smithi028 ceph-mon[29442]: 6.19 scrub starts 2024-02-05T06:48:24.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:24 smithi028 ceph-mon[29442]: 4.b scrub starts 2024-02-05T06:48:24.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:24 smithi028 ceph-mon[29442]: 4.b scrub ok 2024-02-05T06:48:24.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:24 smithi028 ceph-mon[29442]: 6.19 scrub ok 2024-02-05T06:48:24.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:24 smithi028 ceph-mon[29442]: 6.12 scrub starts 2024-02-05T06:48:24.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:24 smithi028 ceph-mon[29442]: 6.12 scrub ok 2024-02-05T06:48:24.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:24 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:48:24.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:24 smithi167 ceph-mon[32774]: 6.17 scrub ok 2024-02-05T06:48:24.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:24 smithi167 ceph-mon[32774]: 6.19 scrub starts 2024-02-05T06:48:24.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:24 smithi167 ceph-mon[32774]: 4.b scrub starts 2024-02-05T06:48:24.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:24 smithi167 ceph-mon[32774]: 4.b scrub ok 2024-02-05T06:48:24.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:24 smithi167 ceph-mon[32774]: 6.19 scrub ok 2024-02-05T06:48:24.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:24 smithi167 ceph-mon[32774]: 6.12 scrub starts 2024-02-05T06:48:24.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:24 smithi167 ceph-mon[32774]: 6.12 scrub ok 2024-02-05T06:48:25.036 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:25.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:25.090 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:25.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:25.191 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:25.204 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-02-05T06:48:25.228 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:25.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:25.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:25.329 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:25.366 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:25.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:25.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:25.468 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:25.508 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:25.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:25.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:25.610 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:25.645 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:25.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:25.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:25.690 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:25 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:48:25.690 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:25 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:48:25.690 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:25 smithi167 ceph-mon[32774]: pgmap v713: 158 pgs: 158 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 939 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-02-05T06:48:25.691 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:25 smithi167 ceph-mon[32774]: 6.16 scrub starts 2024-02-05T06:48:25.691 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:25 smithi167 ceph-mon[32774]: 4.1d scrub starts 2024-02-05T06:48:25.691 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:25 smithi167 ceph-mon[32774]: 6.16 scrub ok 2024-02-05T06:48:25.691 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:25 smithi167 ceph-mon[32774]: 4.1d scrub ok 2024-02-05T06:48:25.691 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:25 smithi167 ceph-mon[32774]: 1.a scrub starts 2024-02-05T06:48:25.691 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:25 smithi167 ceph-mon[32774]: 1.a scrub ok 2024-02-05T06:48:25.691 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:25 smithi167 ceph-mon[32774]: osdmap e366: 8 total, 8 up, 8 in 2024-02-05T06:48:25.691 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:25 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2737284712' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:48:25.691 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:25 smithi167 ceph-mon[32774]: from='client.45668 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:48:25.691 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:25 smithi167 ceph-mon[32774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:48:25.691 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:25 smithi167 ceph-mon[32774]: Cluster is now healthy 2024-02-05T06:48:25.691 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:25 smithi167 ceph-mon[32774]: from='client.45668 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:48:25.691 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:25 smithi167 ceph-mon[32774]: osdmap e367: 8 total, 8 up, 8 in 2024-02-05T06:48:25.747 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:25.790 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:25.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:25.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:25.892 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:25.945 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 46 16 78028 78012 105.969 78.875 0.00837633 0.0094318 2024-02-05T06:48:25.950 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:25.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:25 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:48:25.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:25 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:48:25.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:25 smithi028 ceph-mon[37984]: pgmap v713: 158 pgs: 158 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 939 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-02-05T06:48:25.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:25 smithi028 ceph-mon[37984]: 6.16 scrub starts 2024-02-05T06:48:25.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:25 smithi028 ceph-mon[37984]: 4.1d scrub starts 2024-02-05T06:48:25.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:25 smithi028 ceph-mon[37984]: 6.16 scrub ok 2024-02-05T06:48:25.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:25 smithi028 ceph-mon[37984]: 4.1d scrub ok 2024-02-05T06:48:25.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:25 smithi028 ceph-mon[37984]: 1.a scrub starts 2024-02-05T06:48:25.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:25 smithi028 ceph-mon[37984]: 1.a scrub ok 2024-02-05T06:48:25.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:25 smithi028 ceph-mon[37984]: osdmap e366: 8 total, 8 up, 8 in 2024-02-05T06:48:25.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:25 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2737284712' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:48:25.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:25 smithi028 ceph-mon[37984]: from='client.45668 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:48:25.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:25 smithi028 ceph-mon[37984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:48:25.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:25 smithi028 ceph-mon[37984]: Cluster is now healthy 2024-02-05T06:48:25.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:25 smithi028 ceph-mon[37984]: from='client.45668 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:48:25.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:25 smithi028 ceph-mon[37984]: osdmap e367: 8 total, 8 up, 8 in 2024-02-05T06:48:25.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:25 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:48:25.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:25 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:48:25.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:25 smithi028 ceph-mon[29442]: pgmap v713: 158 pgs: 158 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 939 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-02-05T06:48:25.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:25 smithi028 ceph-mon[29442]: 6.16 scrub starts 2024-02-05T06:48:25.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:25 smithi028 ceph-mon[29442]: 4.1d scrub starts 2024-02-05T06:48:25.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:25 smithi028 ceph-mon[29442]: 6.16 scrub ok 2024-02-05T06:48:25.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:25 smithi028 ceph-mon[29442]: 4.1d scrub ok 2024-02-05T06:48:25.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:25 smithi028 ceph-mon[29442]: 1.a scrub starts 2024-02-05T06:48:25.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:25 smithi028 ceph-mon[29442]: 1.a scrub ok 2024-02-05T06:48:25.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:25 smithi028 ceph-mon[29442]: osdmap e366: 8 total, 8 up, 8 in 2024-02-05T06:48:25.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:25 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2737284712' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:48:25.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:25 smithi028 ceph-mon[29442]: from='client.45668 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:48:25.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:25 smithi028 ceph-mon[29442]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:48:25.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:25 smithi028 ceph-mon[29442]: Cluster is now healthy 2024-02-05T06:48:25.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:25 smithi028 ceph-mon[29442]: from='client.45668 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:48:25.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:25 smithi028 ceph-mon[29442]: osdmap e367: 8 total, 8 up, 8 in 2024-02-05T06:48:25.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:25.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:26.054 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:26.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:26.093 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:26.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:26.194 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:26.200 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:48:26.200 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-02-05T06:48:26.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-02-05T06:48:26.200 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-02-05T06:48:26.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.200 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14568.0:403 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e205)", 2024-02-05T06:48:26.200 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:40:26.898363+0000", 2024-02-05T06:48:26.200 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 479.28808772999997, 2024-02-05T06:48:26.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.18084518599999999, 2024-02-05T06:48:26.200 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:48:26.200 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-02-05T06:48:26.201 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:48:26.201 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14568", 2024-02-05T06:48:26.201 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.167:0/939366379", 2024-02-05T06:48:26.201 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 403 2024-02-05T06:48:26.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.201 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:48:26.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:48:26.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.898363+0000", 2024-02-05T06:48:26.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:48:26.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.898363+0000", 2024-02-05T06:48:26.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:48:26.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.898366+0000", 2024-02-05T06:48:26.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6520000000000002e-06 2024-02-05T06:48:26.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:48:26.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.898375+0000", 2024-02-05T06:48:26.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7390000000000005e-06 2024-02-05T06:48:26.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:48:26.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.898391+0000", 2024-02-05T06:48:26.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6427999999999999e-05 2024-02-05T06:48:26.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:48:26.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.898400+0000", 2024-02-05T06:48:26.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4610000000000008e-06 2024-02-05T06:48:26.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:48:26.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.898454+0000", 2024-02-05T06:48:26.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4317999999999997e-05 2024-02-05T06:48:26.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:48:26.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:26.898463+0000", 2024-02-05T06:48:26.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.704e-06 2024-02-05T06:48:26.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:48:26.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.071801+0000", 2024-02-05T06:48:26.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.17333821099999999 2024-02-05T06:48:26.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-02-05T06:48:26.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.071937+0000", 2024-02-05T06:48:26.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013592899999999999 2024-02-05T06:48:26.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:48:26.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.078945+0000", 2024-02-05T06:48:26.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0070086469999999998 2024-02-05T06:48:26.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:48:26.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.079046+0000", 2024-02-05T06:48:26.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010094099999999999 2024-02-05T06:48:26.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:48:26.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:40:27.079208+0000", 2024-02-05T06:48:26.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000162156 2024-02-05T06:48:26.205 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.206 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:48:26.206 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.206 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:120503 9.3 9:c1bb4e04:::benchmark_data_smithi028_129096_object120502:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-02-05T06:48:26.206 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:00.517201+0000", 2024-02-05T06:48:26.206 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 205.66925009100001, 2024-02-05T06:48:26.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.230580603, 2024-02-05T06:48:26.206 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:48:26.206 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:26.206 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:48:26.206 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:48:26.206 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:48:26.206 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 120503 2024-02-05T06:48:26.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.207 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:48:26.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:48:26.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.517201+0000", 2024-02-05T06:48:26.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:48:26.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.517201+0000", 2024-02-05T06:48:26.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:48:26.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.517203+0000", 2024-02-05T06:48:26.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2079999999999999e-06 2024-02-05T06:48:26.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:48:26.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.517260+0000", 2024-02-05T06:48:26.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7302000000000001e-05 2024-02-05T06:48:26.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:48:26.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.517276+0000", 2024-02-05T06:48:26.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5087e-05 2024-02-05T06:48:26.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:48:26.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.517283+0000", 2024-02-05T06:48:26.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.892e-06 2024-02-05T06:48:26.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:48:26.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.546582+0000", 2024-02-05T06:48:26.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029298947999999998 2024-02-05T06:48:26.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:48:26.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.546939+0000", 2024-02-05T06:48:26.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00035615000000000001 2024-02-05T06:48:26.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:48:26.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.547084+0000", 2024-02-05T06:48:26.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014563799999999999 2024-02-05T06:48:26.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:48:26.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.550436+0000", 2024-02-05T06:48:26.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0033520300000000002 2024-02-05T06:48:26.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.568445+0000", 2024-02-05T06:48:26.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.018008816 2024-02-05T06:48:26.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.747627+0000", 2024-02-05T06:48:26.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.179181488 2024-02-05T06:48:26.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:48:26.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.747728+0000", 2024-02-05T06:48:26.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000101238 2024-02-05T06:48:26.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:48:26.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.747782+0000", 2024-02-05T06:48:26.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3805999999999999e-05 2024-02-05T06:48:26.211 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.211 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:48:26.212 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.212 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:120539 9.3 9:c73bd140:::benchmark_data_smithi028_129096_object120538:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-02-05T06:48:26.212 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:00.564915+0000", 2024-02-05T06:48:26.212 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 205.621536115, 2024-02-05T06:48:26.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.183043764, 2024-02-05T06:48:26.212 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:48:26.212 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:26.212 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:48:26.212 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:48:26.212 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:48:26.212 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 120539 2024-02-05T06:48:26.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.213 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:48:26.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:48:26.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.564915+0000", 2024-02-05T06:48:26.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:48:26.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.564915+0000", 2024-02-05T06:48:26.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:48:26.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.564916+0000", 2024-02-05T06:48:26.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.336e-06 2024-02-05T06:48:26.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:48:26.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.564968+0000", 2024-02-05T06:48:26.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2193e-05 2024-02-05T06:48:26.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:48:26.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.564979+0000", 2024-02-05T06:48:26.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0319000000000001e-05 2024-02-05T06:48:26.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:48:26.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.564987+0000", 2024-02-05T06:48:26.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.011e-06 2024-02-05T06:48:26.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:48:26.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.565037+0000", 2024-02-05T06:48:26.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0166e-05 2024-02-05T06:48:26.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:48:26.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.565142+0000", 2024-02-05T06:48:26.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000105219 2024-02-05T06:48:26.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:48:26.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.565248+0000", 2024-02-05T06:48:26.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010539699999999999 2024-02-05T06:48:26.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:48:26.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.568097+0000", 2024-02-05T06:48:26.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028493239999999999 2024-02-05T06:48:26.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.684628+0000", 2024-02-05T06:48:26.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11653150499999999 2024-02-05T06:48:26.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.747826+0000", 2024-02-05T06:48:26.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.063197958999999998 2024-02-05T06:48:26.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:48:26.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.747913+0000", 2024-02-05T06:48:26.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.6248999999999994e-05 2024-02-05T06:48:26.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:48:26.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.747959+0000", 2024-02-05T06:48:26.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6085999999999999e-05 2024-02-05T06:48:26.217 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.217 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:48:26.217 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.218 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:120557 9.3 9:ccd5b569:::benchmark_data_smithi028_129096_object120556:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-02-05T06:48:26.218 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:00.581421+0000", 2024-02-05T06:48:26.218 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 205.60502998199999, 2024-02-05T06:48:26.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.166704621, 2024-02-05T06:48:26.218 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:48:26.218 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:26.218 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:48:26.218 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:48:26.218 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:48:26.218 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 120557 2024-02-05T06:48:26.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.218 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:48:26.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:48:26.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.581421+0000", 2024-02-05T06:48:26.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:48:26.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.581421+0000", 2024-02-05T06:48:26.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:48:26.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.581423+0000", 2024-02-05T06:48:26.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.142e-06 2024-02-05T06:48:26.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:48:26.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.581477+0000", 2024-02-05T06:48:26.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3495999999999997e-05 2024-02-05T06:48:26.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:48:26.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.581488+0000", 2024-02-05T06:48:26.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1548e-05 2024-02-05T06:48:26.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:48:26.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.581495+0000", 2024-02-05T06:48:26.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7719999999999997e-06 2024-02-05T06:48:26.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:48:26.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.582543+0000", 2024-02-05T06:48:26.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001048156 2024-02-05T06:48:26.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:48:26.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.582705+0000", 2024-02-05T06:48:26.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00016137399999999999 2024-02-05T06:48:26.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:48:26.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.582890+0000", 2024-02-05T06:48:26.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018557900000000001 2024-02-05T06:48:26.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:48:26.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.585543+0000", 2024-02-05T06:48:26.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0026533519999999999 2024-02-05T06:48:26.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.747583+0000", 2024-02-05T06:48:26.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.16203905699999999 2024-02-05T06:48:26.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.747997+0000", 2024-02-05T06:48:26.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00041472900000000002 2024-02-05T06:48:26.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:48:26.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.748076+0000", 2024-02-05T06:48:26.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8548999999999996e-05 2024-02-05T06:48:26.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:48:26.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.748126+0000", 2024-02-05T06:48:26.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9867000000000002e-05 2024-02-05T06:48:26.223 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.223 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:48:26.223 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.224 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:120754 9.3 9:ca15cc16:::benchmark_data_smithi028_129096_object120753:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-02-05T06:48:26.224 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:00.930113+0000", 2024-02-05T06:48:26.224 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 205.25633831799999, 2024-02-05T06:48:26.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13815897199999999, 2024-02-05T06:48:26.224 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:48:26.224 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:26.224 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:48:26.224 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:48:26.224 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:48:26.224 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 120754 2024-02-05T06:48:26.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.224 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:48:26.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:48:26.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.930113+0000", 2024-02-05T06:48:26.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:48:26.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.930113+0000", 2024-02-05T06:48:26.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:48:26.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.930114+0000", 2024-02-05T06:48:26.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5999999999999999e-06 2024-02-05T06:48:26.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:48:26.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.930168+0000", 2024-02-05T06:48:26.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3622000000000003e-05 2024-02-05T06:48:26.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:48:26.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.930179+0000", 2024-02-05T06:48:26.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1321000000000001e-05 2024-02-05T06:48:26.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:48:26.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.930185+0000", 2024-02-05T06:48:26.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0260000000000004e-06 2024-02-05T06:48:26.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:48:26.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.934034+0000", 2024-02-05T06:48:26.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0038483530000000001 2024-02-05T06:48:26.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:48:26.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.934161+0000", 2024-02-05T06:48:26.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000127002 2024-02-05T06:48:26.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:48:26.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.934314+0000", 2024-02-05T06:48:26.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015326399999999999 2024-02-05T06:48:26.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:48:26.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.939194+0000", 2024-02-05T06:48:26.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048797060000000001 2024-02-05T06:48:26.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.038164+0000", 2024-02-05T06:48:26.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.098970802999999996 2024-02-05T06:48:26.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.068112+0000", 2024-02-05T06:48:26.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029947537999999999 2024-02-05T06:48:26.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:48:26.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.068209+0000", 2024-02-05T06:48:26.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6971000000000004e-05 2024-02-05T06:48:26.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:48:26.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.068272+0000", 2024-02-05T06:48:26.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2766000000000002e-05 2024-02-05T06:48:26.230 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.230 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:48:26.230 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.230 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:120756 9.3 9:ce12c19a:::benchmark_data_smithi028_129096_object120755:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-02-05T06:48:26.230 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:00.933706+0000", 2024-02-05T06:48:26.230 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 205.25274552100001, 2024-02-05T06:48:26.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.1661552, 2024-02-05T06:48:26.230 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:48:26.230 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:26.230 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:48:26.230 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:48:26.231 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:48:26.231 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 120756 2024-02-05T06:48:26.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.231 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:48:26.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:48:26.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.933706+0000", 2024-02-05T06:48:26.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:48:26.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.933706+0000", 2024-02-05T06:48:26.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:48:26.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.933708+0000", 2024-02-05T06:48:26.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.227e-06 2024-02-05T06:48:26.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:48:26.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.933749+0000", 2024-02-05T06:48:26.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1121999999999997e-05 2024-02-05T06:48:26.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:48:26.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.933759+0000", 2024-02-05T06:48:26.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9699999999999994e-06 2024-02-05T06:48:26.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:48:26.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.933767+0000", 2024-02-05T06:48:26.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5269999999999998e-06 2024-02-05T06:48:26.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:48:26.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.936529+0000", 2024-02-05T06:48:26.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00276202 2024-02-05T06:48:26.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:48:26.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.936630+0000", 2024-02-05T06:48:26.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000100756 2024-02-05T06:48:26.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:48:26.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.936730+0000", 2024-02-05T06:48:26.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9611000000000006e-05 2024-02-05T06:48:26.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:48:26.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:00.939470+0000", 2024-02-05T06:48:26.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00274011 2024-02-05T06:48:26.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.044387+0000", 2024-02-05T06:48:26.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.10491689 2024-02-05T06:48:26.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.099752+0000", 2024-02-05T06:48:26.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.055365020000000001 2024-02-05T06:48:26.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:48:26.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.099805+0000", 2024-02-05T06:48:26.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3426e-05 2024-02-05T06:48:26.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:48:26.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.099861+0000", 2024-02-05T06:48:26.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5520999999999999e-05 2024-02-05T06:48:26.235 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.235 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:48:26.236 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.236 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:121075 9.3 9:cfccf10c:::benchmark_data_smithi028_129096_object121074:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-02-05T06:48:26.236 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:01.351309+0000", 2024-02-05T06:48:26.236 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 204.83514224300001, 2024-02-05T06:48:26.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.147336777, 2024-02-05T06:48:26.236 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:48:26.236 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:26.236 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:48:26.236 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:48:26.236 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:48:26.236 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 121075 2024-02-05T06:48:26.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.237 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:48:26.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:48:26.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.351309+0000", 2024-02-05T06:48:26.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:48:26.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.351309+0000", 2024-02-05T06:48:26.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:48:26.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.351311+0000", 2024-02-05T06:48:26.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8500000000000001e-06 2024-02-05T06:48:26.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:48:26.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.351361+0000", 2024-02-05T06:48:26.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.028e-05 2024-02-05T06:48:26.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:48:26.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.351373+0000", 2024-02-05T06:48:26.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1669e-05 2024-02-05T06:48:26.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:48:26.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.351379+0000", 2024-02-05T06:48:26.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2369999999999997e-06 2024-02-05T06:48:26.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:48:26.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.380406+0000", 2024-02-05T06:48:26.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029026700999999999 2024-02-05T06:48:26.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.239 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:48:26.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.380557+0000", 2024-02-05T06:48:26.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015128200000000001 2024-02-05T06:48:26.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:48:26.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.380739+0000", 2024-02-05T06:48:26.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00018184900000000001 2024-02-05T06:48:26.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:48:26.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.382006+0000", 2024-02-05T06:48:26.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001267476 2024-02-05T06:48:26.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.474802+0000", 2024-02-05T06:48:26.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.092795680000000005 2024-02-05T06:48:26.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.498502+0000", 2024-02-05T06:48:26.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023699989000000001 2024-02-05T06:48:26.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:48:26.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.498596+0000", 2024-02-05T06:48:26.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4557999999999998e-05 2024-02-05T06:48:26.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:48:26.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.498646+0000", 2024-02-05T06:48:26.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9206000000000003e-05 2024-02-05T06:48:26.242 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.242 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:48:26.242 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.242 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:121278 9.3 9:c5b97c0a:::benchmark_data_smithi028_129096_object121277:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-02-05T06:48:26.242 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:01.606264+0000", 2024-02-05T06:48:26.242 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 204.580186712, 2024-02-05T06:48:26.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.19801748599999999, 2024-02-05T06:48:26.242 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:48:26.242 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:26.242 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:48:26.242 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:48:26.242 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:48:26.242 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 121278 2024-02-05T06:48:26.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.243 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:48:26.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:48:26.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.606264+0000", 2024-02-05T06:48:26.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:48:26.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.606264+0000", 2024-02-05T06:48:26.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:48:26.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.606266+0000", 2024-02-05T06:48:26.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.637e-06 2024-02-05T06:48:26.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:48:26.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.606300+0000", 2024-02-05T06:48:26.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3850000000000003e-05 2024-02-05T06:48:26.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:48:26.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.606317+0000", 2024-02-05T06:48:26.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6960999999999999e-05 2024-02-05T06:48:26.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:48:26.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.606324+0000", 2024-02-05T06:48:26.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7939999999999997e-06 2024-02-05T06:48:26.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:48:26.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.627530+0000", 2024-02-05T06:48:26.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021206827000000001 2024-02-05T06:48:26.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:48:26.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.627632+0000", 2024-02-05T06:48:26.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010118 2024-02-05T06:48:26.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:48:26.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.627764+0000", 2024-02-05T06:48:26.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013247399999999999 2024-02-05T06:48:26.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.638046+0000", 2024-02-05T06:48:26.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010282224 2024-02-05T06:48:26.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:48:26.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.672097+0000", 2024-02-05T06:48:26.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.034050992000000002 2024-02-05T06:48:26.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.804158+0000", 2024-02-05T06:48:26.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13206031300000001 2024-02-05T06:48:26.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:48:26.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.804214+0000", 2024-02-05T06:48:26.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6702e-05 2024-02-05T06:48:26.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:48:26.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.804282+0000", 2024-02-05T06:48:26.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7532000000000003e-05 2024-02-05T06:48:26.247 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.247 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:48:26.247 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.247 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:121284 9.3 9:cc770777:::benchmark_data_smithi028_129096_object121283:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-02-05T06:48:26.247 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:01.607527+0000", 2024-02-05T06:48:26.247 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 204.578924395, 2024-02-05T06:48:26.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.204634289, 2024-02-05T06:48:26.248 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:48:26.248 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:26.248 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:48:26.248 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:48:26.248 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:48:26.248 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 121284 2024-02-05T06:48:26.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.248 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:48:26.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:48:26.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.607527+0000", 2024-02-05T06:48:26.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:48:26.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.607527+0000", 2024-02-05T06:48:26.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:48:26.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.607528+0000", 2024-02-05T06:48:26.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2869999999999999e-06 2024-02-05T06:48:26.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:48:26.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.607578+0000", 2024-02-05T06:48:26.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9703999999999997e-05 2024-02-05T06:48:26.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:48:26.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.607588+0000", 2024-02-05T06:48:26.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0623e-05 2024-02-05T06:48:26.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:48:26.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.607594+0000", 2024-02-05T06:48:26.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1990000000000003e-06 2024-02-05T06:48:26.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:48:26.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.630799+0000", 2024-02-05T06:48:26.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023204758999999998 2024-02-05T06:48:26.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:48:26.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.630920+0000", 2024-02-05T06:48:26.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120815 2024-02-05T06:48:26.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:48:26.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.631209+0000", 2024-02-05T06:48:26.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028938500000000002 2024-02-05T06:48:26.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.638095+0000", 2024-02-05T06:48:26.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.006885408 2024-02-05T06:48:26.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:48:26.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.672194+0000", 2024-02-05T06:48:26.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.034099061 2024-02-05T06:48:26.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.811988+0000", 2024-02-05T06:48:26.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13979428599999999 2024-02-05T06:48:26.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:48:26.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.812093+0000", 2024-02-05T06:48:26.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010472299999999999 2024-02-05T06:48:26.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:48:26.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.812161+0000", 2024-02-05T06:48:26.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8039000000000004e-05 2024-02-05T06:48:26.253 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.253 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:48:26.253 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.253 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:121294 9.3 9:c54f2ea6:::benchmark_data_smithi028_129096_object121293:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-02-05T06:48:26.253 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:01.615831+0000", 2024-02-05T06:48:26.253 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 204.57061955899999, 2024-02-05T06:48:26.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20027431800000001, 2024-02-05T06:48:26.253 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:48:26.254 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:26.254 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:48:26.254 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:48:26.254 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:48:26.254 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 121294 2024-02-05T06:48:26.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.254 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:48:26.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:48:26.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.615831+0000", 2024-02-05T06:48:26.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:48:26.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.615831+0000", 2024-02-05T06:48:26.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:48:26.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.615833+0000", 2024-02-05T06:48:26.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.362e-06 2024-02-05T06:48:26.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:48:26.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.615907+0000", 2024-02-05T06:48:26.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4153000000000006e-05 2024-02-05T06:48:26.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:48:26.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.615917+0000", 2024-02-05T06:48:26.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0443e-05 2024-02-05T06:48:26.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:48:26.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.615925+0000", 2024-02-05T06:48:26.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4930000000000001e-06 2024-02-05T06:48:26.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:48:26.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.635225+0000", 2024-02-05T06:48:26.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019300033000000001 2024-02-05T06:48:26.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:48:26.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.635370+0000", 2024-02-05T06:48:26.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014553100000000001 2024-02-05T06:48:26.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:48:26.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.635524+0000", 2024-02-05T06:48:26.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015344299999999999 2024-02-05T06:48:26.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.672146+0000", 2024-02-05T06:48:26.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036622447000000002 2024-02-05T06:48:26.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:48:26.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.672198+0000", 2024-02-05T06:48:26.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2027000000000002e-05 2024-02-05T06:48:26.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.815828+0000", 2024-02-05T06:48:26.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14362993299999999 2024-02-05T06:48:26.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:48:26.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.816045+0000", 2024-02-05T06:48:26.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000216899 2024-02-05T06:48:26.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:48:26.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.816106+0000", 2024-02-05T06:48:26.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0554000000000003e-05 2024-02-05T06:48:26.259 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.259 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:48:26.259 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.259 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:121296 9.3 9:cf583f78:::benchmark_data_smithi028_129096_object121295:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-02-05T06:48:26.259 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:01.616489+0000", 2024-02-05T06:48:26.259 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 204.56996177299999, 2024-02-05T06:48:26.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.19977958800000001, 2024-02-05T06:48:26.259 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:48:26.260 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:26.260 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:48:26.260 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:48:26.260 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:48:26.260 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 121296 2024-02-05T06:48:26.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.260 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:48:26.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:48:26.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.616489+0000", 2024-02-05T06:48:26.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:48:26.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.616489+0000", 2024-02-05T06:48:26.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:48:26.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.616490+0000", 2024-02-05T06:48:26.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.19e-06 2024-02-05T06:48:26.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:48:26.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.616543+0000", 2024-02-05T06:48:26.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2602000000000002e-05 2024-02-05T06:48:26.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:48:26.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.616554+0000", 2024-02-05T06:48:26.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0669999999999999e-05 2024-02-05T06:48:26.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:48:26.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.616561+0000", 2024-02-05T06:48:26.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3119999999999996e-06 2024-02-05T06:48:26.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:48:26.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.638128+0000", 2024-02-05T06:48:26.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021566703999999999 2024-02-05T06:48:26.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:48:26.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.638253+0000", 2024-02-05T06:48:26.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012526700000000001 2024-02-05T06:48:26.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:48:26.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.638404+0000", 2024-02-05T06:48:26.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000150989 2024-02-05T06:48:26.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.672186+0000", 2024-02-05T06:48:26.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.033781904000000001 2024-02-05T06:48:26.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:48:26.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.672202+0000", 2024-02-05T06:48:26.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6025e-05 2024-02-05T06:48:26.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.816148+0000", 2024-02-05T06:48:26.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14394652399999999 2024-02-05T06:48:26.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:48:26.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.816223+0000", 2024-02-05T06:48:26.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5007999999999997e-05 2024-02-05T06:48:26.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:48:26.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.816269+0000", 2024-02-05T06:48:26.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:26.266 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:26.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:26.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5392999999999997e-05 2024-02-05T06:48:26.266 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.267 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:48:26.267 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.267 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:121307 9.e 9:7ff133d8:::benchmark_data_smithi028_129096_object121306:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-02-05T06:48:26.267 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:01.628879+0000", 2024-02-05T06:48:26.267 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 204.557572042, 2024-02-05T06:48:26.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.175348843, 2024-02-05T06:48:26.267 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:48:26.267 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:26.267 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:48:26.267 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:48:26.267 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:48:26.267 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 121307 2024-02-05T06:48:26.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.268 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:48:26.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:48:26.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.628879+0000", 2024-02-05T06:48:26.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:48:26.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.628879+0000", 2024-02-05T06:48:26.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:48:26.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.628880+0000", 2024-02-05T06:48:26.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3990000000000001e-06 2024-02-05T06:48:26.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:48:26.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.628940+0000", 2024-02-05T06:48:26.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9459000000000001e-05 2024-02-05T06:48:26.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:48:26.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.628950+0000", 2024-02-05T06:48:26.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0020999999999999e-05 2024-02-05T06:48:26.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:48:26.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.628958+0000", 2024-02-05T06:48:26.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0989999999999999e-06 2024-02-05T06:48:26.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:48:26.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.629006+0000", 2024-02-05T06:48:26.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7624999999999999e-05 2024-02-05T06:48:26.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:48:26.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.629096+0000", 2024-02-05T06:48:26.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0086000000000005e-05 2024-02-05T06:48:26.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,4", 2024-02-05T06:48:26.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.629183+0000", 2024-02-05T06:48:26.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7297000000000001e-05 2024-02-05T06:48:26.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.634835+0000", 2024-02-05T06:48:26.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005651868 2024-02-05T06:48:26.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:48:26.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.634856+0000", 2024-02-05T06:48:26.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1155e-05 2024-02-05T06:48:26.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.804142+0000", 2024-02-05T06:48:26.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.16928574499999999 2024-02-05T06:48:26.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:48:26.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.804203+0000", 2024-02-05T06:48:26.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1723999999999994e-05 2024-02-05T06:48:26.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:48:26.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.804228+0000", 2024-02-05T06:48:26.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4365000000000001e-05 2024-02-05T06:48:26.272 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.273 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:48:26.273 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.273 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:121308 9.e 9:774caccc:::benchmark_data_smithi028_129096_object121307:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-02-05T06:48:26.273 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:01.630277+0000", 2024-02-05T06:48:26.273 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 204.55617400200001, 2024-02-05T06:48:26.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.17403511599999999, 2024-02-05T06:48:26.273 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:48:26.273 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:26.273 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:48:26.273 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:48:26.273 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:48:26.273 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 121308 2024-02-05T06:48:26.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.274 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:48:26.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:48:26.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.630277+0000", 2024-02-05T06:48:26.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:48:26.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.630277+0000", 2024-02-05T06:48:26.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:48:26.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.630279+0000", 2024-02-05T06:48:26.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4789999999999999e-06 2024-02-05T06:48:26.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:48:26.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.630352+0000", 2024-02-05T06:48:26.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3459000000000003e-05 2024-02-05T06:48:26.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:48:26.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.630363+0000", 2024-02-05T06:48:26.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1199999999999999e-05 2024-02-05T06:48:26.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:48:26.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.630371+0000", 2024-02-05T06:48:26.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2330000000000006e-06 2024-02-05T06:48:26.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:48:26.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.632159+0000", 2024-02-05T06:48:26.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0017879199999999999 2024-02-05T06:48:26.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:48:26.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.632287+0000", 2024-02-05T06:48:26.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012743800000000001 2024-02-05T06:48:26.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,4", 2024-02-05T06:48:26.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.632431+0000", 2024-02-05T06:48:26.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014446700000000001 2024-02-05T06:48:26.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:48:26.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.635043+0000", 2024-02-05T06:48:26.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0026122419999999999 2024-02-05T06:48:26.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.636549+0000", 2024-02-05T06:48:26.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015052080000000001 2024-02-05T06:48:26.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.804261+0000", 2024-02-05T06:48:26.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.16771262000000001 2024-02-05T06:48:26.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:48:26.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.804292+0000", 2024-02-05T06:48:26.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0821999999999997e-05 2024-02-05T06:48:26.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:48:26.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.804312+0000", 2024-02-05T06:48:26.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0027999999999999e-05 2024-02-05T06:48:26.278 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.279 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:48:26.279 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.279 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.279 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.279 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:121445 9.3 9:c33e18e6:::benchmark_data_smithi028_129096_object121444:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-02-05T06:48:26.279 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:01.902455+0000", 2024-02-05T06:48:26.279 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 204.283995632, 2024-02-05T06:48:26.279 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.15157683999999999, 2024-02-05T06:48:26.279 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:48:26.279 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:26.279 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:48:26.279 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:48:26.279 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:48:26.279 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 121445 2024-02-05T06:48:26.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.280 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:48:26.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:48:26.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.902455+0000", 2024-02-05T06:48:26.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.280 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:48:26.280 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.902455+0000", 2024-02-05T06:48:26.280 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.280 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.280 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:48:26.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.902456+0000", 2024-02-05T06:48:26.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2900000000000002e-07 2024-02-05T06:48:26.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:48:26.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.902513+0000", 2024-02-05T06:48:26.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6679999999999999e-05 2024-02-05T06:48:26.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.281 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:48:26.281 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.902521+0000", 2024-02-05T06:48:26.281 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1049999999999995e-06 2024-02-05T06:48:26.281 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.281 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:48:26.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.902528+0000", 2024-02-05T06:48:26.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5030000000000002e-06 2024-02-05T06:48:26.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:48:26.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.943463+0000", 2024-02-05T06:48:26.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.040935188999999997 2024-02-05T06:48:26.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.282 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:48:26.282 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.943593+0000", 2024-02-05T06:48:26.282 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013027700000000001 2024-02-05T06:48:26.282 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.282 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:48:26.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.943752+0000", 2024-02-05T06:48:26.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015859500000000001 2024-02-05T06:48:26.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:48:26.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.945887+0000", 2024-02-05T06:48:26.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002135227 2024-02-05T06:48:26.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.283 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.283 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:02.053828+0000", 2024-02-05T06:48:26.283 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.107940778 2024-02-05T06:48:26.283 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.283 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:02.053920+0000", 2024-02-05T06:48:26.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2571999999999994e-05 2024-02-05T06:48:26.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:48:26.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:02.053974+0000", 2024-02-05T06:48:26.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3718999999999998e-05 2024-02-05T06:48:26.284 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.284 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.284 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:48:26.284 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:02.054032+0000", 2024-02-05T06:48:26.284 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8366000000000003e-05 2024-02-05T06:48:26.285 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.285 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:48:26.285 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.285 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.285 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.285 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:121458 9.3 9:c86ca6f3:::benchmark_data_smithi028_129096_object121457:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-02-05T06:48:26.285 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:01.914251+0000", 2024-02-05T06:48:26.285 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 204.272200314, 2024-02-05T06:48:26.285 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13998217900000001, 2024-02-05T06:48:26.285 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:48:26.285 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:26.285 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:48:26.285 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:48:26.285 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:48:26.285 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 121458 2024-02-05T06:48:26.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.286 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:48:26.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:48:26.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.914251+0000", 2024-02-05T06:48:26.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:48:26.286 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.914251+0000", 2024-02-05T06:48:26.286 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.286 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.286 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.286 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:48:26.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.914253+0000", 2024-02-05T06:48:26.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9e-06 2024-02-05T06:48:26.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:48:26.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.914308+0000", 2024-02-05T06:48:26.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5823999999999999e-05 2024-02-05T06:48:26.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:48:26.287 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.914320+0000", 2024-02-05T06:48:26.287 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1473e-05 2024-02-05T06:48:26.287 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.287 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.287 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:48:26.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.914333+0000", 2024-02-05T06:48:26.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3382e-05 2024-02-05T06:48:26.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:48:26.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.944792+0000", 2024-02-05T06:48:26.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.030458959000000001 2024-02-05T06:48:26.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.288 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.288 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:48:26.288 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.944887+0000", 2024-02-05T06:48:26.288 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4357999999999993e-05 2024-02-05T06:48:26.288 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:48:26.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.944975+0000", 2024-02-05T06:48:26.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8821999999999998e-05 2024-02-05T06:48:26.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:48:26.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.947252+0000", 2024-02-05T06:48:26.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0022763430000000001 2024-02-05T06:48:26.289 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.289 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.289 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.289 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:02.053876+0000", 2024-02-05T06:48:26.289 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.106624124 2024-02-05T06:48:26.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:02.054098+0000", 2024-02-05T06:48:26.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022181999999999999 2024-02-05T06:48:26.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:48:26.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:02.054183+0000", 2024-02-05T06:48:26.290 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4778999999999996e-05 2024-02-05T06:48:26.290 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.290 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.290 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:48:26.290 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:02.054233+0000", 2024-02-05T06:48:26.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0395000000000001e-05 2024-02-05T06:48:26.291 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.291 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:48:26.291 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.291 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.291 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.291 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:121466 9.3 9:cfd2fce8:::benchmark_data_smithi028_129096_object121465:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-02-05T06:48:26.291 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:01.915221+0000", 2024-02-05T06:48:26.291 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 204.27122969499999, 2024-02-05T06:48:26.291 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13948618800000001, 2024-02-05T06:48:26.291 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:48:26.291 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:26.291 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:48:26.291 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:48:26.292 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:48:26.292 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 121466 2024-02-05T06:48:26.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.292 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:48:26.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:48:26.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.915221+0000", 2024-02-05T06:48:26.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.292 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.292 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.292 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:48:26.292 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.915221+0000", 2024-02-05T06:48:26.292 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:48:26.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.915223+0000", 2024-02-05T06:48:26.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2619999999999999e-06 2024-02-05T06:48:26.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:48:26.293 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.915297+0000", 2024-02-05T06:48:26.293 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3987999999999997e-05 2024-02-05T06:48:26.293 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.293 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.293 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:48:26.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.915310+0000", 2024-02-05T06:48:26.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3071000000000001e-05 2024-02-05T06:48:26.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:48:26.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.915318+0000", 2024-02-05T06:48:26.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8730000000000007e-06 2024-02-05T06:48:26.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:48:26.294 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.945908+0000", 2024-02-05T06:48:26.294 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.030590965000000001 2024-02-05T06:48:26.294 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.294 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.294 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:48:26.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.945993+0000", 2024-02-05T06:48:26.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4932000000000006e-05 2024-02-05T06:48:26.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:48:26.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.946130+0000", 2024-02-05T06:48:26.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013608899999999999 2024-02-05T06:48:26.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:48:26.295 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.947342+0000", 2024-02-05T06:48:26.295 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001212271 2024-02-05T06:48:26.295 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.295 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.295 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:02.054282+0000", 2024-02-05T06:48:26.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.106940691 2024-02-05T06:48:26.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:02.054482+0000", 2024-02-05T06:48:26.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00019943599999999999 2024-02-05T06:48:26.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:48:26.296 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:02.054575+0000", 2024-02-05T06:48:26.296 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2621999999999995e-05 2024-02-05T06:48:26.296 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.296 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.296 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:48:26.297 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:02.054708+0000", 2024-02-05T06:48:26.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013298800000000001 2024-02-05T06:48:26.297 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.297 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:48:26.297 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.297 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.297 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.297 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:125072 9.3 9:c7f020ae:::benchmark_data_smithi028_129096_object125071:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-02-05T06:48:26.297 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:05.377918+0000", 2024-02-05T06:48:26.297 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 200.80853304199999, 2024-02-05T06:48:26.297 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14184434800000001, 2024-02-05T06:48:26.297 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:48:26.297 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:26.298 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:48:26.298 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:48:26.298 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:48:26.298 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 125072 2024-02-05T06:48:26.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.298 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:48:26.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:48:26.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:05.377918+0000", 2024-02-05T06:48:26.298 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.298 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.298 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.298 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:48:26.298 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:05.377918+0000", 2024-02-05T06:48:26.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:48:26.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:05.377921+0000", 2024-02-05T06:48:26.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7049999999999999e-06 2024-02-05T06:48:26.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:48:26.299 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:05.377984+0000", 2024-02-05T06:48:26.299 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2971000000000004e-05 2024-02-05T06:48:26.299 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.299 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.299 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:48:26.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:05.378004+0000", 2024-02-05T06:48:26.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9935000000000001e-05 2024-02-05T06:48:26.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:48:26.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:05.378012+0000", 2024-02-05T06:48:26.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.371e-06 2024-02-05T06:48:26.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.300 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:48:26.300 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:05.407522+0000", 2024-02-05T06:48:26.300 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029509937 2024-02-05T06:48:26.300 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.300 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:48:26.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:05.407610+0000", 2024-02-05T06:48:26.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8407999999999998e-05 2024-02-05T06:48:26.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:48:26.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:05.407701+0000", 2024-02-05T06:48:26.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0616000000000001e-05 2024-02-05T06:48:26.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:48:26.301 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:05.409266+0000", 2024-02-05T06:48:26.301 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0015653889999999999 2024-02-05T06:48:26.301 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.301 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.301 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:05.502298+0000", 2024-02-05T06:48:26.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.093031288000000004 2024-02-05T06:48:26.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:05.519660+0000", 2024-02-05T06:48:26.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017361951 2024-02-05T06:48:26.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:48:26.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:05.519715+0000", 2024-02-05T06:48:26.302 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5090999999999998e-05 2024-02-05T06:48:26.302 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.302 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.302 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:48:26.302 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:05.519762+0000", 2024-02-05T06:48:26.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7685999999999997e-05 2024-02-05T06:48:26.303 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.303 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:48:26.303 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.303 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.303 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.303 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:188490 9.3 9:c2eb4e86:::benchmark_data_smithi028_129096_object58690:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-02-05T06:48:26.303 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:46:00.102219+0000", 2024-02-05T06:48:26.303 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 146.08423161900001, 2024-02-05T06:48:26.303 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.107880008, 2024-02-05T06:48:26.303 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:48:26.303 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:26.303 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:48:26.303 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:48:26.303 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:48:26.304 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 188490 2024-02-05T06:48:26.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.304 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:48:26.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:48:26.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:46:00.102219+0000", 2024-02-05T06:48:26.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.304 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:48:26.304 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:46:00.102219+0000", 2024-02-05T06:48:26.304 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.304 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.304 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:48:26.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:46:00.102220+0000", 2024-02-05T06:48:26.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9100000000000002e-07 2024-02-05T06:48:26.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:48:26.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:46:00.102224+0000", 2024-02-05T06:48:26.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.5109999999999999e-06 2024-02-05T06:48:26.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:48:26.305 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:46:00.102234+0000", 2024-02-05T06:48:26.305 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.045e-05 2024-02-05T06:48:26.305 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.305 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.305 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:48:26.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:46:00.102240+0000", 2024-02-05T06:48:26.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9370000000000001e-06 2024-02-05T06:48:26.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.306 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:48:26.306 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:46:00.105903+0000", 2024-02-05T06:48:26.306 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0036623839999999999 2024-02-05T06:48:26.306 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.306 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:48:26.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:46:00.106007+0000", 2024-02-05T06:48:26.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000104041 2024-02-05T06:48:26.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:48:26.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:46:00.106117+0000", 2024-02-05T06:48:26.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011070599999999999 2024-02-05T06:48:26.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.307 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.307 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:48:26.307 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:46:00.108517+0000", 2024-02-05T06:48:26.307 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023995650000000002 2024-02-05T06:48:26.307 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:46:00.108607+0000", 2024-02-05T06:48:26.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9891999999999999e-05 2024-02-05T06:48:26.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:46:00.209966+0000", 2024-02-05T06:48:26.308 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.101359412 2024-02-05T06:48:26.308 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.308 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.308 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:48:26.308 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:46:00.210059+0000", 2024-02-05T06:48:26.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2998999999999995e-05 2024-02-05T06:48:26.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.309 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:48:26.309 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:46:00.210099+0000", 2024-02-05T06:48:26.309 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0219999999999998e-05 2024-02-05T06:48:26.309 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.309 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:48:26.309 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.309 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.309 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.309 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:250097 9.3 9:c6b248c2:::benchmark_data_smithi028_129096_object120297:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-02-05T06:48:26.309 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:46:51.282931+0000", 2024-02-05T06:48:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 94.903519756999998, 2024-02-05T06:48:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.11473765499999999, 2024-02-05T06:48:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:48:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:48:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:48:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:48:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 250097 2024-02-05T06:48:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:48:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:48:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:46:51.282931+0000", 2024-02-05T06:48:26.310 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:48:26.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:46:51.282931+0000", 2024-02-05T06:48:26.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:48:26.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:46:51.282932+0000", 2024-02-05T06:48:26.311 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1829999999999999e-06 2024-02-05T06:48:26.311 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.311 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.311 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:48:26.311 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:46:51.282940+0000", 2024-02-05T06:48:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7700000000000001e-06 2024-02-05T06:48:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:48:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:46:51.282949+0000", 2024-02-05T06:48:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0780000000000002e-06 2024-02-05T06:48:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:48:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:46:51.282956+0000", 2024-02-05T06:48:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1770000000000001e-06 2024-02-05T06:48:26.312 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:48:26.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:46:51.285743+0000", 2024-02-05T06:48:26.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0027870109999999998 2024-02-05T06:48:26.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:48:26.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:46:51.285826+0000", 2024-02-05T06:48:26.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2306999999999996e-05 2024-02-05T06:48:26.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.313 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.313 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:48:26.313 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:46:51.285901+0000", 2024-02-05T06:48:26.313 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4966e-05 2024-02-05T06:48:26.313 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:48:26.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:46:51.288716+0000", 2024-02-05T06:48:26.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028147839999999999 2024-02-05T06:48:26.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:46:51.290810+0000", 2024-02-05T06:48:26.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020945069999999998 2024-02-05T06:48:26.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.314 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.314 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:46:51.397564+0000", 2024-02-05T06:48:26.314 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.106754074 2024-02-05T06:48:26.314 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.314 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:48:26.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:46:51.397625+0000", 2024-02-05T06:48:26.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0801999999999998e-05 2024-02-05T06:48:26.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.315 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:48:26.315 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:46:51.397669+0000", 2024-02-05T06:48:26.315 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3995999999999997e-05 2024-02-05T06:48:26.315 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.315 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:48:26.315 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.315 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.315 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.315 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.45587.0:69916 10.1 10:8bb9b02d:::benchmark_data_smithi028_154709_object69915:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-02-05T06:48:26.315 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:48:19.108226+0000", 2024-02-05T06:48:26.315 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 7.0782252139999997, 2024-02-05T06:48:26.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.091842751, 2024-02-05T06:48:26.316 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:48:26.316 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:26.316 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:48:26.316 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.45587", 2024-02-05T06:48:26.316 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:48:26.316 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 69916 2024-02-05T06:48:26.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.316 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:48:26.316 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.316 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:48:26.316 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:19.108226+0000", 2024-02-05T06:48:26.316 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.316 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:48:26.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:19.108226+0000", 2024-02-05T06:48:26.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:48:26.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:48:26.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:19.108228+0000", 2024-02-05T06:48:26.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2579999999999998e-06 2024-02-05T06:48:26.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.317 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.317 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:48:26.317 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:19.108285+0000", 2024-02-05T06:48:26.317 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6842000000000002e-05 2024-02-05T06:48:26.317 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:48:26.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:19.108298+0000", 2024-02-05T06:48:26.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2806000000000001e-05 2024-02-05T06:48:26.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:48:26.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:19.108307+0000", 2024-02-05T06:48:26.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2709999999999998e-06 2024-02-05T06:48:26.318 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.318 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.318 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:48:26.318 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:19.108367+0000", 2024-02-05T06:48:26.318 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0069999999999999e-05 2024-02-05T06:48:26.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:48:26.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:19.108509+0000", 2024-02-05T06:48:26.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000141669 2024-02-05T06:48:26.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,4", 2024-02-05T06:48:26.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:19.108655+0000", 2024-02-05T06:48:26.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014674299999999999 2024-02-05T06:48:26.319 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.319 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.319 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:48:26.319 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:19.111411+0000", 2024-02-05T06:48:26.319 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0027551490000000001 2024-02-05T06:48:26.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:19.113505+0000", 2024-02-05T06:48:26.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002094149 2024-02-05T06:48:26.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:26.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:19.199974+0000", 2024-02-05T06:48:26.320 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.086468751999999996 2024-02-05T06:48:26.320 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.320 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.320 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:48:26.320 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:19.200025+0000", 2024-02-05T06:48:26.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1254000000000001e-05 2024-02-05T06:48:26.321 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:48:26.321 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:48:26.321 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:48:26.321 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:19.200069+0000", 2024-02-05T06:48:26.321 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3788000000000002e-05 2024-02-05T06:48:26.321 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.321 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:48:26.321 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.321 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:48:26.321 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:48:26.321 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:48:26.367 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:26.410 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:26.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:26.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:26.512 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:26.558 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:26.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:26.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:26.659 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:26.700 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:26.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:26.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:26.802 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:26.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:26.838 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:26.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:26.939 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:26.946 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 47 15 79252 79237 105.342 76.5625 0.0110265 0.0094876 2024-02-05T06:48:26.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:26 smithi028 ceph-mon[29442]: 3.6 scrub starts 2024-02-05T06:48:26.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:26 smithi028 ceph-mon[29442]: 3.6 scrub ok 2024-02-05T06:48:26.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:26 smithi028 ceph-mon[29442]: 3.f scrub starts 2024-02-05T06:48:26.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:26 smithi028 ceph-mon[29442]: 3.f scrub ok 2024-02-05T06:48:26.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:26 smithi028 ceph-mon[29442]: 1.7 deep-scrub starts 2024-02-05T06:48:26.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:26 smithi028 ceph-mon[29442]: 1.7 deep-scrub ok 2024-02-05T06:48:26.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:26 smithi028 ceph-mon[29442]: 5.a scrub starts 2024-02-05T06:48:26.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:26 smithi028 ceph-mon[29442]: 5.a scrub ok 2024-02-05T06:48:26.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:26 smithi028 ceph-mon[37984]: 3.6 scrub starts 2024-02-05T06:48:26.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:26 smithi028 ceph-mon[37984]: 3.6 scrub ok 2024-02-05T06:48:26.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:26 smithi028 ceph-mon[37984]: 3.f scrub starts 2024-02-05T06:48:26.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:26 smithi028 ceph-mon[37984]: 3.f scrub ok 2024-02-05T06:48:26.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:26 smithi028 ceph-mon[37984]: 1.7 deep-scrub starts 2024-02-05T06:48:26.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:26 smithi028 ceph-mon[37984]: 1.7 deep-scrub ok 2024-02-05T06:48:26.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:26 smithi028 ceph-mon[37984]: 5.a scrub starts 2024-02-05T06:48:26.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:26 smithi028 ceph-mon[37984]: 5.a scrub ok 2024-02-05T06:48:26.977 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:26.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:26.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:26.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:26 smithi167 ceph-mon[32774]: 3.6 scrub starts 2024-02-05T06:48:26.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:26 smithi167 ceph-mon[32774]: 3.6 scrub ok 2024-02-05T06:48:26.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:26 smithi167 ceph-mon[32774]: 3.f scrub starts 2024-02-05T06:48:26.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:26 smithi167 ceph-mon[32774]: 3.f scrub ok 2024-02-05T06:48:26.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:26 smithi167 ceph-mon[32774]: 1.7 deep-scrub starts 2024-02-05T06:48:26.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:26 smithi167 ceph-mon[32774]: 1.7 deep-scrub ok 2024-02-05T06:48:26.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:26 smithi167 ceph-mon[32774]: 5.a scrub starts 2024-02-05T06:48:26.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:26 smithi167 ceph-mon[32774]: 5.a scrub ok 2024-02-05T06:48:27.079 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:27.112 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:27.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:27.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:27.179 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-05T06:48:27.214 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:27.282 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:27.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:27.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:27.384 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:27.417 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:27.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:27.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:27.518 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:27.553 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:27.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:27.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:27.655 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:27.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:27.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:27.806 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:27.841 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:27.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:27.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:27.943 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:27.946 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 48 16 80436 80420 104.688 73.9375 0.00576353 0.00954668 2024-02-05T06:48:27.977 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:27.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:27.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:28.079 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:28.116 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:28.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:28.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:28.218 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:28.257 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:28.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:28.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:28.358 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:28.392 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:28.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:28.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:28.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:28 smithi028 ceph-mon[29442]: pgmap v716: 158 pgs: 158 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 44 MiB/s wr, 700 op/s 2024-02-05T06:48:28.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:28 smithi028 ceph-mon[29442]: 3.13 scrub starts 2024-02-05T06:48:28.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:28 smithi028 ceph-mon[29442]: 3.13 scrub ok 2024-02-05T06:48:28.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:28 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:48:28.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:28 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:48:28.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:28 smithi028 ceph-mon[37984]: pgmap v716: 158 pgs: 158 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 44 MiB/s wr, 700 op/s 2024-02-05T06:48:28.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:28 smithi028 ceph-mon[37984]: 3.13 scrub starts 2024-02-05T06:48:28.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:28 smithi028 ceph-mon[37984]: 3.13 scrub ok 2024-02-05T06:48:28.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:28 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:48:28.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:28 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:48:28.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:28 smithi167 ceph-mon[32774]: pgmap v716: 158 pgs: 158 active+clean; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 44 MiB/s wr, 700 op/s 2024-02-05T06:48:28.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:28 smithi167 ceph-mon[32774]: 3.13 scrub starts 2024-02-05T06:48:28.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:28 smithi167 ceph-mon[32774]: 3.13 scrub ok 2024-02-05T06:48:28.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:28 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:48:28.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:28 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:48:28.494 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:28.531 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:28.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:28.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:28.554 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:48:28.564 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-05T06:48:28.632 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:28.671 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:28.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:28.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:28.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:28.816 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:28.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:28.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:28.918 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:28.946 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 49 16 81656 81640 104.107 76.25 0.0104863 0.00960038 2024-02-05T06:48:28.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:28.954 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:28.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:29.055 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:29.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:29.112 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:29.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:29.166 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:48:29.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:29 smithi028 ceph-mon[37984]: 5.9 scrub starts 2024-02-05T06:48:29.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:29 smithi028 ceph-mon[37984]: 5.9 scrub ok 2024-02-05T06:48:29.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:29 smithi028 ceph-mon[37984]: 3.18 scrub starts 2024-02-05T06:48:29.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:29 smithi028 ceph-mon[37984]: 3.18 scrub ok 2024-02-05T06:48:29.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:29 smithi028 ceph-mon[37984]: 3.a scrub starts 2024-02-05T06:48:29.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:29 smithi028 ceph-mon[37984]: 3.a scrub ok 2024-02-05T06:48:29.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:29 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:48:29.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:29 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:48:29.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:29 smithi028 ceph-mon[37984]: osdmap e368: 8 total, 8 up, 8 in 2024-02-05T06:48:29.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:29 smithi028 ceph-mon[37984]: 6.14 scrub starts 2024-02-05T06:48:29.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:29 smithi028 ceph-mon[37984]: 6.14 scrub ok 2024-02-05T06:48:29.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:29 smithi028 ceph-mon[37984]: 3.12 scrub starts 2024-02-05T06:48:29.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:29 smithi028 ceph-mon[37984]: 3.12 scrub ok 2024-02-05T06:48:29.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:29 smithi028 ceph-mon[29442]: 5.9 scrub starts 2024-02-05T06:48:29.202 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:29 smithi028 ceph-mon[29442]: 5.9 scrub ok 2024-02-05T06:48:29.202 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:29 smithi028 ceph-mon[29442]: 3.18 scrub starts 2024-02-05T06:48:29.202 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:29 smithi028 ceph-mon[29442]: 3.18 scrub ok 2024-02-05T06:48:29.202 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:29 smithi028 ceph-mon[29442]: 3.a scrub starts 2024-02-05T06:48:29.202 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:29 smithi028 ceph-mon[29442]: 3.a scrub ok 2024-02-05T06:48:29.202 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:29 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:48:29.202 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:29 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:48:29.202 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:29 smithi028 ceph-mon[29442]: osdmap e368: 8 total, 8 up, 8 in 2024-02-05T06:48:29.202 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:29 smithi028 ceph-mon[29442]: 6.14 scrub starts 2024-02-05T06:48:29.202 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:29 smithi028 ceph-mon[29442]: 6.14 scrub ok 2024-02-05T06:48:29.202 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:29 smithi028 ceph-mon[29442]: 3.12 scrub starts 2024-02-05T06:48:29.202 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:29 smithi028 ceph-mon[29442]: 3.12 scrub ok 2024-02-05T06:48:29.211 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:48:29.213 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:29.255 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:29.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:29.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:29.257 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:48:29.300 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:48:29.338 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:48:29.357 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:29.379 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:48:29.395 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:29.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:29.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:29.415 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:48:29.456 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:48:29.458 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:48:29.458 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:48:29.458 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:48:29.458 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:48:29.459 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:48:29.459 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:48:29.459 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:48:29.459 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:48:29.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:29 smithi167 ceph-mon[32774]: 5.9 scrub starts 2024-02-05T06:48:29.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:29 smithi167 ceph-mon[32774]: 5.9 scrub ok 2024-02-05T06:48:29.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:29 smithi167 ceph-mon[32774]: 3.18 scrub starts 2024-02-05T06:48:29.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:29 smithi167 ceph-mon[32774]: 3.18 scrub ok 2024-02-05T06:48:29.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:29 smithi167 ceph-mon[32774]: 3.a scrub starts 2024-02-05T06:48:29.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:29 smithi167 ceph-mon[32774]: 3.a scrub ok 2024-02-05T06:48:29.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:29 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:48:29.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:29 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:48:29.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:29 smithi167 ceph-mon[32774]: osdmap e368: 8 total, 8 up, 8 in 2024-02-05T06:48:29.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:29 smithi167 ceph-mon[32774]: 6.14 scrub starts 2024-02-05T06:48:29.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:29 smithi167 ceph-mon[32774]: 6.14 scrub ok 2024-02-05T06:48:29.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:29 smithi167 ceph-mon[32774]: 3.12 scrub starts 2024-02-05T06:48:29.486 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:29 smithi167 ceph-mon[32774]: 3.12 scrub ok 2024-02-05T06:48:29.497 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:29.530 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:29.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:29.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:29.631 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:29.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:29.669 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:29.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:29.685 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":718,"stamp":"2024-02-05T06:48:28.464938+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5178726748,"num_objects":79210,"num_object_clones":0,"num_object_copies":237630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79210,"num_whiteouts":0,"num_read":1605,"num_read_kb":1357,"num_write":79734,"num_write_kb":5056813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":415,"num_bytes_recovered":1586799,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81306,"ondisk_log_size":81306,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":19275396,"kb_used_data":15115132,"kb_used_omap":12,"kb_used_meta":4160179,"kb_avail":730685820,"statfs":{"total":767960285184,"available":748222279680,"internally_reserved":0,"allocated":15477895168,"data_stored":15473762004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":4260023857},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,3,11,10,8,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":446496768,"num_objects":6813,"num_object_clones":0,"num_object_copies":20439,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6813,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6813,"num_write_kb":436032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001976"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":411,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.188035+0000","last_change":"2024-02-05T06:47:44.624372+0000","last_active":"2024-02-05T06:48:25.188035+0000","last_peered":"2024-02-05T06:48:25.188035+0000","last_clean":"2024-02-05T06:48:25.188035+0000","last_became_active":"2024-02-05T06:45:09.787134+0000","last_became_peered":"2024-02-05T06:45:09.787134+0000","last_unstale":"2024-02-05T06:48:25.188035+0000","last_undegraded":"2024-02-05T06:48:25.188035+0000","last_fullsized":"2024-02-05T06:48:25.188035+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:44.624254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:47:44.624254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:58.304608+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":682,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.186927+0000","last_change":"2024-02-05T06:48:19.036192+0000","last_active":"2024-02-05T06:48:25.186927+0000","last_peered":"2024-02-05T06:48:25.186927+0000","last_clean":"2024-02-05T06:48:25.186927+0000","last_became_active":"2024-02-05T06:41:43.420342+0000","last_became_peered":"2024-02-05T06:41:43.420342+0000","last_unstale":"2024-02-05T06:48:25.186927+0000","last_undegraded":"2024-02-05T06:48:25.186927+0000","last_fullsized":"2024-02-05T06:48:25.186927+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:19.036048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:06.525012+0000","last_clean_scrub_stamp":"2024-02-05T06:48:19.036048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:27.993992+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":672,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.194343+0000","last_change":"2024-02-05T06:48:20.602707+0000","last_active":"2024-02-05T06:48:25.194343+0000","last_peered":"2024-02-05T06:48:25.194343+0000","last_clean":"2024-02-05T06:48:25.194343+0000","last_became_active":"2024-02-05T06:41:43.421315+0000","last_became_peered":"2024-02-05T06:41:43.421315+0000","last_unstale":"2024-02-05T06:48:25.194343+0000","last_undegraded":"2024-02-05T06:48:25.194343+0000","last_fullsized":"2024-02-05T06:48:25.194343+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:20.602591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:48:20.602591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:41.700957+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"295'19","reported_seq":582,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.182862+0000","last_change":"2024-02-05T06:48:19.282787+0000","last_active":"2024-02-05T06:48:25.182862+0000","last_peered":"2024-02-05T06:48:25.182862+0000","last_clean":"2024-02-05T06:48:25.182862+0000","last_became_active":"2024-02-05T06:36:54.959330+0000","last_became_peered":"2024-02-05T06:36:54.959330+0000","last_unstale":"2024-02-05T06:48:25.182862+0000","last_undegraded":"2024-02-05T06:48:25.182862+0000","last_fullsized":"2024-02-05T06:48:25.182862+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'19","last_scrub_stamp":"2024-02-05T06:48:19.282672+0000","last_deep_scrub":"247'13","last_deep_scrub_stamp":"2024-02-05T06:41:42.473424+0000","last_clean_scrub_stamp":"2024-02-05T06:48:19.282672+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:41.837304+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":649,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.197573+0000","last_change":"2024-02-05T06:45:45.252029+0000","last_active":"2024-02-05T06:48:25.197573+0000","last_peered":"2024-02-05T06:48:25.197573+0000","last_clean":"2024-02-05T06:48:25.197573+0000","last_became_active":"2024-02-05T06:40:27.077485+0000","last_became_peered":"2024-02-05T06:40:27.077485+0000","last_unstale":"2024-02-05T06:48:25.197573+0000","last_undegraded":"2024-02-05T06:48:25.197573+0000","last_fullsized":"2024-02-05T06:48:25.197573+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:45.251895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:45:45.251895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":180,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.195150+0000","last_change":"2024-02-05T06:48:18.702091+0000","last_active":"2024-02-05T06:48:25.195150+0000","last_peered":"2024-02-05T06:48:25.195150+0000","last_clean":"2024-02-05T06:48:25.195150+0000","last_became_active":"2024-02-05T06:45:09.775037+0000","last_became_peered":"2024-02-05T06:45:09.775037+0000","last_unstale":"2024-02-05T06:48:25.195150+0000","last_undegraded":"2024-02-05T06:48:25.195150+0000","last_fullsized":"2024-02-05T06:48:25.195150+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:18.701981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:51.388156+0000","last_clean_scrub_stamp":"2024-02-05T06:48:18.701981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:33.942331+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":659,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.194229+0000","last_change":"2024-02-05T06:47:34.440851+0000","last_active":"2024-02-05T06:48:25.194229+0000","last_peered":"2024-02-05T06:48:25.194229+0000","last_clean":"2024-02-05T06:48:25.194229+0000","last_became_active":"2024-02-05T06:33:29.553864+0000","last_became_peered":"2024-02-05T06:33:29.553864+0000","last_unstale":"2024-02-05T06:48:25.194229+0000","last_undegraded":"2024-02-05T06:48:25.194229+0000","last_fullsized":"2024-02-05T06:48:25.194229+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:34.440773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:47:34.440773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:40.674347+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"295'34","reported_seq":704,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.194015+0000","last_change":"2024-02-05T06:47:48.407121+0000","last_active":"2024-02-05T06:48:25.194015+0000","last_peered":"2024-02-05T06:48:25.194015+0000","last_clean":"2024-02-05T06:48:25.194015+0000","last_became_active":"2024-02-05T06:33:31.572544+0000","last_became_peered":"2024-02-05T06:33:31.572544+0000","last_unstale":"2024-02-05T06:48:25.194015+0000","last_undegraded":"2024-02-05T06:48:25.194015+0000","last_fullsized":"2024-02-05T06:48:25.194015+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'34","last_scrub_stamp":"2024-02-05T06:47:48.406976+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:39:35.433539+0000","last_clean_scrub_stamp":"2024-02-05T06:47:48.406976+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:12.788812+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":659,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.191331+0000","last_change":"2024-02-05T06:46:58.507850+0000","last_active":"2024-02-05T06:48:25.191331+0000","last_peered":"2024-02-05T06:48:25.191331+0000","last_clean":"2024-02-05T06:48:25.191331+0000","last_became_active":"2024-02-05T06:40:27.068888+0000","last_became_peered":"2024-02-05T06:40:27.068888+0000","last_unstale":"2024-02-05T06:48:25.191331+0000","last_undegraded":"2024-02-05T06:48:25.191331+0000","last_fullsized":"2024-02-05T06:48:25.191331+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:58.507407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:44:55.125921+0000","last_clean_scrub_stamp":"2024-02-05T06:46:58.507407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":215,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.197662+0000","last_change":"2024-02-05T06:45:51.218029+0000","last_active":"2024-02-05T06:48:25.197662+0000","last_peered":"2024-02-05T06:48:25.197662+0000","last_clean":"2024-02-05T06:48:25.197662+0000","last_became_active":"2024-02-05T06:42:33.129959+0000","last_became_peered":"2024-02-05T06:42:33.129959+0000","last_unstale":"2024-02-05T06:48:25.197662+0000","last_undegraded":"2024-02-05T06:48:25.197662+0000","last_fullsized":"2024-02-05T06:48:25.197662+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:51.217900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:45:51.217900+0000","last_clean_scrub_stamp":"2024-02-05T06:45:51.217900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":266,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.188379+0000","last_change":"2024-02-05T06:46:46.643447+0000","last_active":"2024-02-05T06:48:25.188379+0000","last_peered":"2024-02-05T06:48:25.188379+0000","last_clean":"2024-02-05T06:48:25.188379+0000","last_became_active":"2024-02-05T06:41:43.403746+0000","last_became_peered":"2024-02-05T06:41:43.403746+0000","last_unstale":"2024-02-05T06:48:25.188379+0000","last_undegraded":"2024-02-05T06:48:25.188379+0000","last_fullsized":"2024-02-05T06:48:25.188379+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:46.643336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:44.081318+0000","last_clean_scrub_stamp":"2024-02-05T06:46:46.643336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"295'19","reported_seq":515,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.193723+0000","last_change":"2024-02-05T06:48:17.662641+0000","last_active":"2024-02-05T06:48:25.193723+0000","last_peered":"2024-02-05T06:48:25.193723+0000","last_clean":"2024-02-05T06:48:25.193723+0000","last_became_active":"2024-02-05T06:45:09.787626+0000","last_became_peered":"2024-02-05T06:45:09.787626+0000","last_unstale":"2024-02-05T06:48:25.193723+0000","last_undegraded":"2024-02-05T06:48:25.193723+0000","last_fullsized":"2024-02-05T06:48:25.193723+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'19","last_scrub_stamp":"2024-02-05T06:48:17.662510+0000","last_deep_scrub":"92'13","last_deep_scrub_stamp":"2024-02-05T06:34:46.542893+0000","last_clean_scrub_stamp":"2024-02-05T06:48:17.662510+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:32.649577+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":274,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.189921+0000","last_change":"2024-02-05T06:48:23.678385+0000","last_active":"2024-02-05T06:48:25.189921+0000","last_peered":"2024-02-05T06:48:25.189921+0000","last_clean":"2024-02-05T06:48:25.189921+0000","last_became_active":"2024-02-05T06:41:43.407404+0000","last_became_peered":"2024-02-05T06:41:43.407404+0000","last_unstale":"2024-02-05T06:48:25.189921+0000","last_undegraded":"2024-02-05T06:48:25.189921+0000","last_fullsized":"2024-02-05T06:48:25.189921+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:23.678255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:21.998850+0000","last_clean_scrub_stamp":"2024-02-05T06:48:23.678255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:35.610704+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":367,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.185966+0000","last_change":"2024-02-05T06:47:51.640890+0000","last_active":"2024-02-05T06:48:25.185966+0000","last_peered":"2024-02-05T06:48:25.185966+0000","last_clean":"2024-02-05T06:48:25.185966+0000","last_became_active":"2024-02-05T06:40:27.071758+0000","last_became_peered":"2024-02-05T06:40:27.071758+0000","last_unstale":"2024-02-05T06:48:25.185966+0000","last_undegraded":"2024-02-05T06:48:25.185966+0000","last_fullsized":"2024-02-05T06:48:25.185966+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:51.640785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:47:51.640785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:06.140529+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":274,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.187822+0000","last_change":"2024-02-05T06:47:39.612949+0000","last_active":"2024-02-05T06:48:25.187822+0000","last_peered":"2024-02-05T06:48:25.187822+0000","last_clean":"2024-02-05T06:48:25.187822+0000","last_became_active":"2024-02-05T06:41:43.408428+0000","last_became_peered":"2024-02-05T06:41:43.408428+0000","last_unstale":"2024-02-05T06:48:25.187822+0000","last_undegraded":"2024-02-05T06:48:25.187822+0000","last_fullsized":"2024-02-05T06:48:25.187822+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:39.612865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:50.055245+0000","last_clean_scrub_stamp":"2024-02-05T06:47:39.612865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:46.400154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"295'7","reported_seq":497,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.197297+0000","last_change":"2024-02-05T06:45:09.809100+0000","last_active":"2024-02-05T06:48:25.197297+0000","last_peered":"2024-02-05T06:48:25.197297+0000","last_clean":"2024-02-05T06:48:25.197297+0000","last_became_active":"2024-02-05T06:45:09.808663+0000","last_became_peered":"2024-02-05T06:45:09.808663+0000","last_unstale":"2024-02-05T06:48:25.197297+0000","last_undegraded":"2024-02-05T06:48:25.197297+0000","last_fullsized":"2024-02-05T06:48:25.197297+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'7","last_scrub_stamp":"2024-02-05T06:44:55.988817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:44:55.988817+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":8,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":415,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.194811+0000","last_change":"2024-02-05T06:48:15.743595+0000","last_active":"2024-02-05T06:48:25.194811+0000","last_peered":"2024-02-05T06:48:25.194811+0000","last_clean":"2024-02-05T06:48:25.194811+0000","last_became_active":"2024-02-05T06:45:09.770902+0000","last_became_peered":"2024-02-05T06:45:09.770902+0000","last_unstale":"2024-02-05T06:48:25.194811+0000","last_undegraded":"2024-02-05T06:48:25.194811+0000","last_fullsized":"2024-02-05T06:48:25.194811+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:15.743307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:48:15.743307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:32.352762+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":661,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.200278+0000","last_change":"2024-02-05T06:45:39.848063+0000","last_active":"2024-02-05T06:48:25.200278+0000","last_peered":"2024-02-05T06:48:25.200278+0000","last_clean":"2024-02-05T06:48:25.200278+0000","last_became_active":"2024-02-05T06:36:54.958069+0000","last_became_peered":"2024-02-05T06:36:54.958069+0000","last_unstale":"2024-02-05T06:48:25.200278+0000","last_undegraded":"2024-02-05T06:48:25.200278+0000","last_fullsized":"2024-02-05T06:48:25.200278+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:39.847950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:45:39.847950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"281'2","reported_seq":705,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.195210+0000","last_change":"2024-02-05T06:47:50.442268+0000","last_active":"2024-02-05T06:48:25.195210+0000","last_peered":"2024-02-05T06:48:25.195210+0000","last_clean":"2024-02-05T06:48:25.195210+0000","last_became_active":"2024-02-05T06:40:27.073360+0000","last_became_peered":"2024-02-05T06:40:27.073360+0000","last_unstale":"2024-02-05T06:48:25.195210+0000","last_undegraded":"2024-02-05T06:48:25.195210+0000","last_fullsized":"2024-02-05T06:48:25.195210+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'2","last_scrub_stamp":"2024-02-05T06:47:50.442159+0000","last_deep_scrub":"281'2","last_deep_scrub_stamp":"2024-02-05T06:47:50.442159+0000","last_clean_scrub_stamp":"2024-02-05T06:47:50.442159+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:16.471755+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"295'34","reported_seq":733,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.189289+0000","last_change":"2024-02-05T06:47:43.040717+0000","last_active":"2024-02-05T06:48:25.189289+0000","last_peered":"2024-02-05T06:48:25.189289+0000","last_clean":"2024-02-05T06:48:25.189289+0000","last_became_active":"2024-02-05T06:33:31.569763+0000","last_became_peered":"2024-02-05T06:33:31.569763+0000","last_unstale":"2024-02-05T06:48:25.189289+0000","last_undegraded":"2024-02-05T06:48:25.189289+0000","last_fullsized":"2024-02-05T06:48:25.189289+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'34","last_scrub_stamp":"2024-02-05T06:47:43.040574+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:34:35.234636+0000","last_clean_scrub_stamp":"2024-02-05T06:47:43.040574+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:49.320938+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":651,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.192670+0000","last_change":"2024-02-05T06:46:54.584153+0000","last_active":"2024-02-05T06:48:25.192670+0000","last_peered":"2024-02-05T06:48:25.192670+0000","last_clean":"2024-02-05T06:48:25.192670+0000","last_became_active":"2024-02-05T06:41:43.411442+0000","last_became_peered":"2024-02-05T06:41:43.411442+0000","last_unstale":"2024-02-05T06:48:25.192670+0000","last_undegraded":"2024-02-05T06:48:25.192670+0000","last_fullsized":"2024-02-05T06:48:25.192670+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:54.584073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:46:54.584073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":447,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.193961+0000","last_change":"2024-02-05T06:48:13.677951+0000","last_active":"2024-02-05T06:48:25.193961+0000","last_peered":"2024-02-05T06:48:25.193961+0000","last_clean":"2024-02-05T06:48:25.193961+0000","last_became_active":"2024-02-05T06:39:13.248618+0000","last_became_peered":"2024-02-05T06:39:13.248618+0000","last_unstale":"2024-02-05T06:48:25.193961+0000","last_undegraded":"2024-02-05T06:48:25.193961+0000","last_fullsized":"2024-02-05T06:48:25.193961+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:13.677795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:42.381636+0000","last_clean_scrub_stamp":"2024-02-05T06:48:13.677795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:41.284546+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":188,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.187604+0000","last_change":"2024-02-05T06:48:15.138990+0000","last_active":"2024-02-05T06:48:25.187604+0000","last_peered":"2024-02-05T06:48:25.187604+0000","last_clean":"2024-02-05T06:48:25.187604+0000","last_became_active":"2024-02-05T06:45:09.786579+0000","last_became_peered":"2024-02-05T06:45:09.786579+0000","last_unstale":"2024-02-05T06:48:25.187604+0000","last_undegraded":"2024-02-05T06:48:25.187604+0000","last_fullsized":"2024-02-05T06:48:25.187604+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:15.138923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:48:15.138923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:40.202990+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"295'18","reported_seq":739,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.191988+0000","last_change":"2024-02-05T06:48:24.628198+0000","last_active":"2024-02-05T06:48:25.191988+0000","last_peered":"2024-02-05T06:48:25.191988+0000","last_clean":"2024-02-05T06:48:25.191988+0000","last_became_active":"2024-02-05T06:36:54.954880+0000","last_became_peered":"2024-02-05T06:36:54.954880+0000","last_unstale":"2024-02-05T06:48:25.191988+0000","last_undegraded":"2024-02-05T06:48:25.191988+0000","last_fullsized":"2024-02-05T06:48:25.191988+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'18","last_scrub_stamp":"2024-02-05T06:48:24.627686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:48:24.627686+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:35.142648+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"325'2","reported_seq":374,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.186837+0000","last_change":"2024-02-05T06:48:20.648253+0000","last_active":"2024-02-05T06:48:25.186837+0000","last_peered":"2024-02-05T06:48:25.186837+0000","last_clean":"2024-02-05T06:48:25.186837+0000","last_became_active":"2024-02-05T06:41:43.418640+0000","last_became_peered":"2024-02-05T06:41:43.418640+0000","last_unstale":"2024-02-05T06:48:25.186837+0000","last_undegraded":"2024-02-05T06:48:25.186837+0000","last_fullsized":"2024-02-05T06:48:25.186837+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'2","last_scrub_stamp":"2024-02-05T06:48:20.648179+0000","last_deep_scrub":"325'2","last_deep_scrub_stamp":"2024-02-05T06:45:46.242415+0000","last_clean_scrub_stamp":"2024-02-05T06:48:20.648179+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:36.679419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"314'13","reported_seq":911,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.195818+0000","last_change":"2024-02-05T06:47:42.202376+0000","last_active":"2024-02-05T06:48:25.195818+0000","last_peered":"2024-02-05T06:48:25.195818+0000","last_clean":"2024-02-05T06:48:25.195818+0000","last_became_active":"2024-02-05T06:40:27.063399+0000","last_became_peered":"2024-02-05T06:40:27.063399+0000","last_unstale":"2024-02-05T06:48:25.195818+0000","last_undegraded":"2024-02-05T06:48:25.195818+0000","last_fullsized":"2024-02-05T06:48:25.195818+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"314'13","last_scrub_stamp":"2024-02-05T06:47:42.202237+0000","last_deep_scrub":"314'13","last_deep_scrub_stamp":"2024-02-05T06:44:53.670923+0000","last_clean_scrub_stamp":"2024-02-05T06:47:42.202237+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:46.335289+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":695,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.186834+0000","last_change":"2024-02-05T06:47:49.989990+0000","last_active":"2024-02-05T06:48:25.186834+0000","last_peered":"2024-02-05T06:48:25.186834+0000","last_clean":"2024-02-05T06:48:25.186834+0000","last_became_active":"2024-02-05T06:36:54.958414+0000","last_became_peered":"2024-02-05T06:36:54.958414+0000","last_unstale":"2024-02-05T06:48:25.186834+0000","last_undegraded":"2024-02-05T06:48:25.186834+0000","last_fullsized":"2024-02-05T06:48:25.186834+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:47:49.989863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:47:49.989863+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:14.685287+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"295'25","reported_seq":735,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.195192+0000","last_change":"2024-02-05T06:48:17.606222+0000","last_active":"2024-02-05T06:48:25.195192+0000","last_peered":"2024-02-05T06:48:25.195192+0000","last_clean":"2024-02-05T06:48:25.195192+0000","last_became_active":"2024-02-05T06:40:27.100610+0000","last_became_peered":"2024-02-05T06:40:27.100610+0000","last_unstale":"2024-02-05T06:48:25.195192+0000","last_undegraded":"2024-02-05T06:48:25.195192+0000","last_fullsized":"2024-02-05T06:48:25.195192+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'25","last_scrub_stamp":"2024-02-05T06:48:17.605995+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-02-05T06:34:50.718768+0000","last_clean_scrub_stamp":"2024-02-05T06:48:17.605995+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:38.852335+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":677,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.200398+0000","last_change":"2024-02-05T06:46:45.866461+0000","last_active":"2024-02-05T06:48:25.200398+0000","last_peered":"2024-02-05T06:48:25.200398+0000","last_clean":"2024-02-05T06:48:25.200398+0000","last_became_active":"2024-02-05T06:41:43.414638+0000","last_became_peered":"2024-02-05T06:41:43.414638+0000","last_unstale":"2024-02-05T06:48:25.200398+0000","last_undegraded":"2024-02-05T06:48:25.200398+0000","last_fullsized":"2024-02-05T06:48:25.200398+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:45.866348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:46:45.866348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":566,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.194048+0000","last_change":"2024-02-05T06:46:36.444754+0000","last_active":"2024-02-05T06:48:25.194048+0000","last_peered":"2024-02-05T06:48:25.194048+0000","last_clean":"2024-02-05T06:48:25.194048+0000","last_became_active":"2024-02-05T06:40:27.097169+0000","last_became_peered":"2024-02-05T06:40:27.097169+0000","last_unstale":"2024-02-05T06:48:25.194048+0000","last_undegraded":"2024-02-05T06:48:25.194048+0000","last_fullsized":"2024-02-05T06:48:25.194048+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:36.444642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:08.713894+0000","last_clean_scrub_stamp":"2024-02-05T06:46:36.444642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":266,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.186988+0000","last_change":"2024-02-05T06:46:39.579976+0000","last_active":"2024-02-05T06:48:25.186988+0000","last_peered":"2024-02-05T06:48:25.186988+0000","last_clean":"2024-02-05T06:48:25.186988+0000","last_became_active":"2024-02-05T06:41:43.411043+0000","last_became_peered":"2024-02-05T06:41:43.411043+0000","last_unstale":"2024-02-05T06:48:25.186988+0000","last_undegraded":"2024-02-05T06:48:25.186988+0000","last_fullsized":"2024-02-05T06:48:25.186988+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:39.579866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:46:39.579866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"295'24","reported_seq":753,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.195920+0000","last_change":"2024-02-05T06:47:37.226174+0000","last_active":"2024-02-05T06:48:25.195920+0000","last_peered":"2024-02-05T06:48:25.195920+0000","last_clean":"2024-02-05T06:48:25.195920+0000","last_became_active":"2024-02-05T06:41:43.419769+0000","last_became_peered":"2024-02-05T06:41:43.419769+0000","last_unstale":"2024-02-05T06:48:25.195920+0000","last_undegraded":"2024-02-05T06:48:25.195920+0000","last_fullsized":"2024-02-05T06:48:25.195920+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:47:37.226071+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:43:03.221466+0000","last_clean_scrub_stamp":"2024-02-05T06:47:37.226071+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:40.560472+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":373,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.186215+0000","last_change":"2024-02-05T06:48:23.748347+0000","last_active":"2024-02-05T06:48:25.186215+0000","last_peered":"2024-02-05T06:48:25.186215+0000","last_clean":"2024-02-05T06:48:25.186215+0000","last_became_active":"2024-02-05T06:44:54.625067+0000","last_became_peered":"2024-02-05T06:44:54.625067+0000","last_unstale":"2024-02-05T06:48:25.186215+0000","last_undegraded":"2024-02-05T06:48:25.186215+0000","last_fullsized":"2024-02-05T06:48:25.186215+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:23.748226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:48.512673+0000","last_clean_scrub_stamp":"2024-02-05T06:48:23.748226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:44.753039+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":697,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.194288+0000","last_change":"2024-02-05T06:46:54.260053+0000","last_active":"2024-02-05T06:48:25.194288+0000","last_peered":"2024-02-05T06:48:25.194288+0000","last_clean":"2024-02-05T06:48:25.194288+0000","last_became_active":"2024-02-05T06:40:27.060112+0000","last_became_peered":"2024-02-05T06:40:27.060112+0000","last_unstale":"2024-02-05T06:48:25.194288+0000","last_undegraded":"2024-02-05T06:48:25.194288+0000","last_fullsized":"2024-02-05T06:48:25.194288+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:54.259661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:46:54.259661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":663,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.194533+0000","last_change":"2024-02-05T06:46:31.409743+0000","last_active":"2024-02-05T06:48:25.194533+0000","last_peered":"2024-02-05T06:48:25.194533+0000","last_clean":"2024-02-05T06:48:25.194533+0000","last_became_active":"2024-02-05T06:41:43.424459+0000","last_became_peered":"2024-02-05T06:41:43.424459+0000","last_unstale":"2024-02-05T06:48:25.194533+0000","last_undegraded":"2024-02-05T06:48:25.194533+0000","last_fullsized":"2024-02-05T06:48:25.194533+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:31.409628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:49.117949+0000","last_clean_scrub_stamp":"2024-02-05T06:46:31.409628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"295'8","reported_seq":658,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.192796+0000","last_change":"2024-02-05T06:47:41.444191+0000","last_active":"2024-02-05T06:48:25.192796+0000","last_peered":"2024-02-05T06:48:25.192796+0000","last_clean":"2024-02-05T06:48:25.192796+0000","last_became_active":"2024-02-05T06:33:31.573410+0000","last_became_peered":"2024-02-05T06:33:31.573410+0000","last_unstale":"2024-02-05T06:48:25.192796+0000","last_undegraded":"2024-02-05T06:48:25.192796+0000","last_fullsized":"2024-02-05T06:48:25.192796+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'8","last_scrub_stamp":"2024-02-05T06:47:41.444069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:47:41.444069+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:54.176721+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":424,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.186703+0000","last_change":"2024-02-05T06:46:35.933464+0000","last_active":"2024-02-05T06:48:25.186703+0000","last_peered":"2024-02-05T06:48:25.186703+0000","last_clean":"2024-02-05T06:48:25.186703+0000","last_became_active":"2024-02-05T06:39:13.245684+0000","last_became_peered":"2024-02-05T06:39:13.245684+0000","last_unstale":"2024-02-05T06:48:25.186703+0000","last_undegraded":"2024-02-05T06:48:25.186703+0000","last_fullsized":"2024-02-05T06:48:25.186703+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:35.933354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:46:35.933354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":359,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.186046+0000","last_change":"2024-02-05T06:46:41.612056+0000","last_active":"2024-02-05T06:48:25.186046+0000","last_peered":"2024-02-05T06:48:25.186046+0000","last_clean":"2024-02-05T06:48:25.186046+0000","last_became_active":"2024-02-05T06:40:27.062975+0000","last_became_peered":"2024-02-05T06:40:27.062975+0000","last_unstale":"2024-02-05T06:48:25.186046+0000","last_undegraded":"2024-02-05T06:48:25.186046+0000","last_fullsized":"2024-02-05T06:48:25.186046+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:41.611955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:46:41.611955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":274,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.188154+0000","last_change":"2024-02-05T06:46:51.634330+0000","last_active":"2024-02-05T06:48:25.188154+0000","last_peered":"2024-02-05T06:48:25.188154+0000","last_clean":"2024-02-05T06:48:25.188154+0000","last_became_active":"2024-02-05T06:41:43.412654+0000","last_became_peered":"2024-02-05T06:41:43.412654+0000","last_unstale":"2024-02-05T06:48:25.188154+0000","last_undegraded":"2024-02-05T06:48:25.188154+0000","last_fullsized":"2024-02-05T06:48:25.188154+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:51.634250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:46:51.634250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"295'24","reported_seq":736,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.193940+0000","last_change":"2024-02-05T06:47:30.341586+0000","last_active":"2024-02-05T06:48:25.193940+0000","last_peered":"2024-02-05T06:48:25.193940+0000","last_clean":"2024-02-05T06:48:25.193940+0000","last_became_active":"2024-02-05T06:40:27.080906+0000","last_became_peered":"2024-02-05T06:40:27.080906+0000","last_unstale":"2024-02-05T06:48:25.193940+0000","last_undegraded":"2024-02-05T06:48:25.193940+0000","last_fullsized":"2024-02-05T06:48:25.193940+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:47:30.341433+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-02-05T06:35:22.938675+0000","last_clean_scrub_stamp":"2024-02-05T06:47:30.341433+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:51.439941+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"336'2","reported_seq":669,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.190722+0000","last_change":"2024-02-05T06:46:53.500514+0000","last_active":"2024-02-05T06:48:25.190722+0000","last_peered":"2024-02-05T06:48:25.190722+0000","last_clean":"2024-02-05T06:48:25.190722+0000","last_became_active":"2024-02-05T06:41:43.409761+0000","last_became_peered":"2024-02-05T06:41:43.409761+0000","last_unstale":"2024-02-05T06:48:25.190722+0000","last_undegraded":"2024-02-05T06:48:25.190722+0000","last_fullsized":"2024-02-05T06:48:25.190722+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'2","last_scrub_stamp":"2024-02-05T06:46:53.500155+0000","last_deep_scrub":"336'2","last_deep_scrub_stamp":"2024-02-05T06:46:53.500155+0000","last_clean_scrub_stamp":"2024-02-05T06:46:53.500155+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":642,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.197490+0000","last_change":"2024-02-05T06:47:35.411302+0000","last_active":"2024-02-05T06:48:25.197490+0000","last_peered":"2024-02-05T06:48:25.197490+0000","last_clean":"2024-02-05T06:48:25.197490+0000","last_became_active":"2024-02-05T06:33:33.593901+0000","last_became_peered":"2024-02-05T06:33:33.593901+0000","last_unstale":"2024-02-05T06:48:25.197490+0000","last_undegraded":"2024-02-05T06:48:25.197490+0000","last_fullsized":"2024-02-05T06:48:25.197490+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:35.411219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:47:35.411219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:41.160858+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":359,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.186921+0000","last_change":"2024-02-05T06:46:29.865891+0000","last_active":"2024-02-05T06:48:25.186921+0000","last_peered":"2024-02-05T06:48:25.186921+0000","last_clean":"2024-02-05T06:48:25.186921+0000","last_became_active":"2024-02-05T06:40:27.077701+0000","last_became_peered":"2024-02-05T06:40:27.077701+0000","last_unstale":"2024-02-05T06:48:25.186921+0000","last_undegraded":"2024-02-05T06:48:25.186921+0000","last_fullsized":"2024-02-05T06:48:25.186921+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:29.865777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:46:29.865777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"295'15","reported_seq":443,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.197396+0000","last_change":"2024-02-05T06:46:34.466054+0000","last_active":"2024-02-05T06:48:25.197396+0000","last_peered":"2024-02-05T06:48:25.197396+0000","last_clean":"2024-02-05T06:48:25.197396+0000","last_became_active":"2024-02-05T06:39:13.244124+0000","last_became_peered":"2024-02-05T06:39:13.244124+0000","last_unstale":"2024-02-05T06:48:25.197396+0000","last_undegraded":"2024-02-05T06:48:25.197396+0000","last_fullsized":"2024-02-05T06:48:25.197396+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:46:34.465921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:46:34.465921+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":650,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.193609+0000","last_change":"2024-02-05T06:45:35.052699+0000","last_active":"2024-02-05T06:48:25.193609+0000","last_peered":"2024-02-05T06:48:25.193609+0000","last_clean":"2024-02-05T06:48:25.193609+0000","last_became_active":"2024-02-05T06:41:43.417155+0000","last_became_peered":"2024-02-05T06:41:43.417155+0000","last_unstale":"2024-02-05T06:48:25.193609+0000","last_undegraded":"2024-02-05T06:48:25.193609+0000","last_fullsized":"2024-02-05T06:48:25.193609+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:35.052451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:45:35.052451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":676,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.187749+0000","last_change":"2024-02-05T06:45:40.802539+0000","last_active":"2024-02-05T06:48:25.187749+0000","last_peered":"2024-02-05T06:48:25.187749+0000","last_clean":"2024-02-05T06:48:25.187749+0000","last_became_active":"2024-02-05T06:41:43.421433+0000","last_became_peered":"2024-02-05T06:41:43.421433+0000","last_unstale":"2024-02-05T06:48:25.187749+0000","last_undegraded":"2024-02-05T06:48:25.187749+0000","last_fullsized":"2024-02-05T06:48:25.187749+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:40.802433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:45:40.802433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":313,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.186526+0000","last_change":"2024-02-05T06:47:47.669670+0000","last_active":"2024-02-05T06:48:25.186526+0000","last_peered":"2024-02-05T06:48:25.186526+0000","last_clean":"2024-02-05T06:48:25.186526+0000","last_became_active":"2024-02-05T06:45:09.773490+0000","last_became_peered":"2024-02-05T06:45:09.773490+0000","last_unstale":"2024-02-05T06:48:25.186526+0000","last_undegraded":"2024-02-05T06:48:25.186526+0000","last_fullsized":"2024-02-05T06:48:25.186526+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:47.669594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:47:47.669594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:04.867910+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"295'24","reported_seq":520,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.195182+0000","last_change":"2024-02-05T06:46:52.290692+0000","last_active":"2024-02-05T06:48:25.195182+0000","last_peered":"2024-02-05T06:48:25.195182+0000","last_clean":"2024-02-05T06:48:25.195182+0000","last_became_active":"2024-02-05T06:45:09.784411+0000","last_became_peered":"2024-02-05T06:45:09.784411+0000","last_unstale":"2024-02-05T06:48:25.195182+0000","last_undegraded":"2024-02-05T06:48:25.195182+0000","last_fullsized":"2024-02-05T06:48:25.195182+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:46:52.290420+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:35:02.589229+0000","last_clean_scrub_stamp":"2024-02-05T06:46:52.290420+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":659,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.194925+0000","last_change":"2024-02-05T06:48:24.623914+0000","last_active":"2024-02-05T06:48:25.194925+0000","last_peered":"2024-02-05T06:48:25.194925+0000","last_clean":"2024-02-05T06:48:25.194925+0000","last_became_active":"2024-02-05T06:40:27.096132+0000","last_became_peered":"2024-02-05T06:40:27.096132+0000","last_unstale":"2024-02-05T06:48:25.194925+0000","last_undegraded":"2024-02-05T06:48:25.194925+0000","last_fullsized":"2024-02-05T06:48:25.194925+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:24.623815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:48:24.623815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:51.647757+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"336'11","reported_seq":360,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.188125+0000","last_change":"2024-02-05T06:46:55.722262+0000","last_active":"2024-02-05T06:48:25.188125+0000","last_peered":"2024-02-05T06:48:25.188125+0000","last_clean":"2024-02-05T06:48:25.188125+0000","last_became_active":"2024-02-05T06:41:43.418753+0000","last_became_peered":"2024-02-05T06:41:43.418753+0000","last_unstale":"2024-02-05T06:48:25.188125+0000","last_undegraded":"2024-02-05T06:48:25.188125+0000","last_fullsized":"2024-02-05T06:48:25.188125+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'11","last_scrub_stamp":"2024-02-05T06:46:55.721412+0000","last_deep_scrub":"336'11","last_deep_scrub_stamp":"2024-02-05T06:46:55.721412+0000","last_clean_scrub_stamp":"2024-02-05T06:46:55.721412+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":363,"reported_epoch":367,"state":"active+clean+scrubbing","last_fresh":"2024-02-05T06:48:26.662315+0000","last_change":"2024-02-05T06:48:26.647768+0000","last_active":"2024-02-05T06:48:26.662315+0000","last_peered":"2024-02-05T06:48:26.662315+0000","last_clean":"2024-02-05T06:48:26.662315+0000","last_became_active":"2024-02-05T06:40:27.098418+0000","last_became_peered":"2024-02-05T06:40:27.098418+0000","last_unstale":"2024-02-05T06:48:26.662315+0000","last_undegraded":"2024-02-05T06:48:26.662315+0000","last_fullsized":"2024-02-05T06:48:26.662315+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:38.694245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:46:38.694245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"295'20","reported_seq":730,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.194493+0000","last_change":"2024-02-05T06:45:31.996335+0000","last_active":"2024-02-05T06:48:25.194493+0000","last_peered":"2024-02-05T06:48:25.194493+0000","last_clean":"2024-02-05T06:48:25.194493+0000","last_became_active":"2024-02-05T06:40:27.093418+0000","last_became_peered":"2024-02-05T06:40:27.093418+0000","last_unstale":"2024-02-05T06:48:25.194493+0000","last_undegraded":"2024-02-05T06:48:25.194493+0000","last_fullsized":"2024-02-05T06:48:25.194493+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'20","last_scrub_stamp":"2024-02-05T06:45:31.996183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:45:31.996183+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":414,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.188073+0000","last_change":"2024-02-05T06:48:23.249326+0000","last_active":"2024-02-05T06:48:25.188073+0000","last_peered":"2024-02-05T06:48:25.188073+0000","last_clean":"2024-02-05T06:48:25.188073+0000","last_became_active":"2024-02-05T06:45:09.775152+0000","last_became_peered":"2024-02-05T06:45:09.775152+0000","last_unstale":"2024-02-05T06:48:25.188073+0000","last_undegraded":"2024-02-05T06:48:25.188073+0000","last_fullsized":"2024-02-05T06:48:25.188073+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:23.249061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:41:38.559780+0000","last_clean_scrub_stamp":"2024-02-05T06:48:23.249061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:43.366513+0000","scrub_duration":164,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"208'10","reported_seq":858,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.194863+0000","last_change":"2024-02-05T06:48:18.678733+0000","last_active":"2024-02-05T06:48:25.194863+0000","last_peered":"2024-02-05T06:48:25.194863+0000","last_clean":"2024-02-05T06:48:25.194863+0000","last_became_active":"2024-02-05T06:40:27.096584+0000","last_became_peered":"2024-02-05T06:40:27.096584+0000","last_unstale":"2024-02-05T06:48:25.194863+0000","last_undegraded":"2024-02-05T06:48:25.194863+0000","last_fullsized":"2024-02-05T06:48:25.194863+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-02-05T06:48:18.677391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:48:18.677391+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:43.044904+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":671,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.197717+0000","last_change":"2024-02-05T06:45:45.945300+0000","last_active":"2024-02-05T06:48:25.197717+0000","last_peered":"2024-02-05T06:48:25.197717+0000","last_clean":"2024-02-05T06:48:25.197717+0000","last_became_active":"2024-02-05T06:40:27.088096+0000","last_became_peered":"2024-02-05T06:40:27.088096+0000","last_unstale":"2024-02-05T06:48:25.197717+0000","last_undegraded":"2024-02-05T06:48:25.197717+0000","last_fullsized":"2024-02-05T06:48:25.197717+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:45.945168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:44:01.185653+0000","last_clean_scrub_stamp":"2024-02-05T06:45:45.945168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"295'15","reported_seq":293,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.188457+0000","last_change":"2024-02-05T06:46:49.603908+0000","last_active":"2024-02-05T06:48:25.188457+0000","last_peered":"2024-02-05T06:48:25.188457+0000","last_clean":"2024-02-05T06:48:25.188457+0000","last_became_active":"2024-02-05T06:41:43.413708+0000","last_became_peered":"2024-02-05T06:41:43.413708+0000","last_unstale":"2024-02-05T06:48:25.188457+0000","last_undegraded":"2024-02-05T06:48:25.188457+0000","last_fullsized":"2024-02-05T06:48:25.188457+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:46:49.603788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:46:49.603788+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":692,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.189582+0000","last_change":"2024-02-05T06:46:56.761919+0000","last_active":"2024-02-05T06:48:25.189582+0000","last_peered":"2024-02-05T06:48:25.189582+0000","last_clean":"2024-02-05T06:48:25.189582+0000","last_became_active":"2024-02-05T06:40:27.090632+0000","last_became_peered":"2024-02-05T06:40:27.090632+0000","last_unstale":"2024-02-05T06:48:25.189582+0000","last_undegraded":"2024-02-05T06:48:25.189582+0000","last_fullsized":"2024-02-05T06:48:25.189582+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:56.761768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:46:56.761768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":668,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.192594+0000","last_change":"2024-02-05T06:47:43.427431+0000","last_active":"2024-02-05T06:48:25.192594+0000","last_peered":"2024-02-05T06:48:25.192594+0000","last_clean":"2024-02-05T06:48:25.192594+0000","last_became_active":"2024-02-05T06:40:27.091544+0000","last_became_peered":"2024-02-05T06:40:27.091544+0000","last_unstale":"2024-02-05T06:48:25.192594+0000","last_undegraded":"2024-02-05T06:48:25.192594+0000","last_fullsized":"2024-02-05T06:48:25.192594+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:43.427301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:47:43.427301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:08.540744+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":359,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.186067+0000","last_change":"2024-02-05T06:46:43.606599+0000","last_active":"2024-02-05T06:48:25.186067+0000","last_peered":"2024-02-05T06:48:25.186067+0000","last_clean":"2024-02-05T06:48:25.186067+0000","last_became_active":"2024-02-05T06:40:27.064388+0000","last_became_peered":"2024-02-05T06:40:27.064388+0000","last_unstale":"2024-02-05T06:48:25.186067+0000","last_undegraded":"2024-02-05T06:48:25.186067+0000","last_fullsized":"2024-02-05T06:48:25.186067+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:43.606488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:43.606488+0000","last_clean_scrub_stamp":"2024-02-05T06:46:43.606488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"295'15","reported_seq":712,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.195291+0000","last_change":"2024-02-05T06:45:55.988708+0000","last_active":"2024-02-05T06:48:25.195291+0000","last_peered":"2024-02-05T06:48:25.195291+0000","last_clean":"2024-02-05T06:48:25.195291+0000","last_became_active":"2024-02-05T06:40:27.101116+0000","last_became_peered":"2024-02-05T06:40:27.101116+0000","last_unstale":"2024-02-05T06:48:25.195291+0000","last_undegraded":"2024-02-05T06:48:25.195291+0000","last_fullsized":"2024-02-05T06:48:25.195291+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:45:55.988575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:45:55.988575+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":359,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.187148+0000","last_change":"2024-02-05T06:47:38.782082+0000","last_active":"2024-02-05T06:48:25.187148+0000","last_peered":"2024-02-05T06:48:25.187148+0000","last_clean":"2024-02-05T06:48:25.187148+0000","last_became_active":"2024-02-05T06:40:27.063350+0000","last_became_peered":"2024-02-05T06:40:27.063350+0000","last_unstale":"2024-02-05T06:48:25.187148+0000","last_undegraded":"2024-02-05T06:48:25.187148+0000","last_fullsized":"2024-02-05T06:48:25.187148+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:38.781966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:47:38.781966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:57.815067+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":274,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.188109+0000","last_change":"2024-02-05T06:46:56.707237+0000","last_active":"2024-02-05T06:48:25.188109+0000","last_peered":"2024-02-05T06:48:25.188109+0000","last_clean":"2024-02-05T06:48:25.188109+0000","last_became_active":"2024-02-05T06:41:43.406094+0000","last_became_peered":"2024-02-05T06:41:43.406094+0000","last_unstale":"2024-02-05T06:48:25.188109+0000","last_undegraded":"2024-02-05T06:48:25.188109+0000","last_fullsized":"2024-02-05T06:48:25.188109+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:56.707131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:44:57.506909+0000","last_clean_scrub_stamp":"2024-02-05T06:46:56.707131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":685,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.195071+0000","last_change":"2024-02-05T06:46:43.354213+0000","last_active":"2024-02-05T06:48:25.195071+0000","last_peered":"2024-02-05T06:48:25.195071+0000","last_clean":"2024-02-05T06:48:25.195071+0000","last_became_active":"2024-02-05T06:41:43.412130+0000","last_became_peered":"2024-02-05T06:41:43.412130+0000","last_unstale":"2024-02-05T06:48:25.195071+0000","last_undegraded":"2024-02-05T06:48:25.195071+0000","last_fullsized":"2024-02-05T06:48:25.195071+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:43.354098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:46:43.354098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"295'12","reported_seq":692,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.192758+0000","last_change":"2024-02-05T06:46:47.455766+0000","last_active":"2024-02-05T06:48:25.192758+0000","last_peered":"2024-02-05T06:48:25.192758+0000","last_clean":"2024-02-05T06:48:25.192758+0000","last_became_active":"2024-02-05T06:40:27.091689+0000","last_became_peered":"2024-02-05T06:40:27.091689+0000","last_unstale":"2024-02-05T06:48:25.192758+0000","last_undegraded":"2024-02-05T06:48:25.192758+0000","last_fullsized":"2024-02-05T06:48:25.192758+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'12","last_scrub_stamp":"2024-02-05T06:46:47.455632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:46:47.455632+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"367'4943","reported_seq":4996,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:26.634719+0000","last_change":"2024-02-05T06:47:24.184995+0000","last_active":"2024-02-05T06:48:26.634719+0000","last_peered":"2024-02-05T06:48:26.634719+0000","last_clean":"2024-02-05T06:48:26.634719+0000","last_became_active":"2024-02-05T06:47:24.184581+0000","last_became_peered":"2024-02-05T06:47:24.184581+0000","last_unstale":"2024-02-05T06:48:26.634719+0000","last_undegraded":"2024-02-05T06:48:26.634719+0000","last_fullsized":"2024-02-05T06:48:26.634719+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":4943,"log_dups_size":0,"ondisk_log_size":4943,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:50.103785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323944448,"num_objects":4943,"num_object_clones":0,"num_object_copies":14829,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4943,"num_write_kb":316352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":275,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.187399+0000","last_change":"2024-02-05T06:47:38.659232+0000","last_active":"2024-02-05T06:48:25.187399+0000","last_peered":"2024-02-05T06:48:25.187399+0000","last_clean":"2024-02-05T06:48:25.187399+0000","last_became_active":"2024-02-05T06:41:43.421231+0000","last_became_peered":"2024-02-05T06:41:43.421231+0000","last_unstale":"2024-02-05T06:48:25.187399+0000","last_undegraded":"2024-02-05T06:48:25.187399+0000","last_fullsized":"2024-02-05T06:48:25.187399+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:38.659123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:47:38.659123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:53.928020+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":373,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.672898+0000","last_change":"2024-02-05T06:48:25.672942+0000","last_active":"2024-02-05T06:48:25.672898+0000","last_peered":"2024-02-05T06:48:25.672898+0000","last_clean":"2024-02-05T06:48:25.672898+0000","last_became_active":"2024-02-05T06:40:27.068279+0000","last_became_peered":"2024-02-05T06:40:27.068279+0000","last_unstale":"2024-02-05T06:48:25.672898+0000","last_undegraded":"2024-02-05T06:48:25.672898+0000","last_fullsized":"2024-02-05T06:48:25.672898+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-02-05T06:48:25.672806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:48:25.672806+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:44.484988+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"295'27","reported_seq":309,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.187078+0000","last_change":"2024-02-05T06:48:22.632196+0000","last_active":"2024-02-05T06:48:25.187078+0000","last_peered":"2024-02-05T06:48:25.187078+0000","last_clean":"2024-02-05T06:48:25.187078+0000","last_became_active":"2024-02-05T06:41:43.412579+0000","last_became_peered":"2024-02-05T06:41:43.412579+0000","last_unstale":"2024-02-05T06:48:25.187078+0000","last_undegraded":"2024-02-05T06:48:25.187078+0000","last_fullsized":"2024-02-05T06:48:25.187078+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'27","last_scrub_stamp":"2024-02-05T06:48:22.632056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:48:22.632056+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:28.762241+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"208'12","reported_seq":459,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.185910+0000","last_change":"2024-02-05T06:46:37.741787+0000","last_active":"2024-02-05T06:48:25.185910+0000","last_peered":"2024-02-05T06:48:25.185910+0000","last_clean":"2024-02-05T06:48:25.185910+0000","last_became_active":"2024-02-05T06:40:27.064195+0000","last_became_peered":"2024-02-05T06:40:27.064195+0000","last_unstale":"2024-02-05T06:48:25.185910+0000","last_undegraded":"2024-02-05T06:48:25.185910+0000","last_fullsized":"2024-02-05T06:48:25.185910+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'12","last_scrub_stamp":"2024-02-05T06:46:37.741678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:46:37.741678+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"367'5028","reported_seq":5082,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:26.538174+0000","last_change":"2024-02-05T06:47:24.176419+0000","last_active":"2024-02-05T06:48:26.538174+0000","last_peered":"2024-02-05T06:48:26.538174+0000","last_clean":"2024-02-05T06:48:26.538174+0000","last_became_active":"2024-02-05T06:47:24.176055+0000","last_became_peered":"2024-02-05T06:47:24.176055+0000","last_unstale":"2024-02-05T06:48:26.538174+0000","last_undegraded":"2024-02-05T06:48:26.538174+0000","last_fullsized":"2024-02-05T06:48:26.538174+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":5028,"log_dups_size":0,"ondisk_log_size":5028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:37.445734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329515008,"num_objects":5028,"num_object_clones":0,"num_object_copies":15084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5028,"num_write_kb":321792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":650,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.197756+0000","last_change":"2024-02-05T06:47:36.512402+0000","last_active":"2024-02-05T06:48:25.197756+0000","last_peered":"2024-02-05T06:48:25.197756+0000","last_clean":"2024-02-05T06:48:25.197756+0000","last_became_active":"2024-02-05T06:40:27.092613+0000","last_became_peered":"2024-02-05T06:40:27.092613+0000","last_unstale":"2024-02-05T06:48:25.197756+0000","last_undegraded":"2024-02-05T06:48:25.197756+0000","last_fullsized":"2024-02-05T06:48:25.197756+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:36.512306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:47:36.512306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:40.368980+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":359,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.188146+0000","last_change":"2024-02-05T06:46:53.740645+0000","last_active":"2024-02-05T06:48:25.188146+0000","last_peered":"2024-02-05T06:48:25.188146+0000","last_clean":"2024-02-05T06:48:25.188146+0000","last_became_active":"2024-02-05T06:40:27.063300+0000","last_became_peered":"2024-02-05T06:40:27.063300+0000","last_unstale":"2024-02-05T06:48:25.188146+0000","last_undegraded":"2024-02-05T06:48:25.188146+0000","last_fullsized":"2024-02-05T06:48:25.188146+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:53.740542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:46:53.740542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"295'18","reported_seq":518,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.191000+0000","last_change":"2024-02-05T06:46:58.712252+0000","last_active":"2024-02-05T06:48:25.191000+0000","last_peered":"2024-02-05T06:48:25.191000+0000","last_clean":"2024-02-05T06:48:25.191000+0000","last_became_active":"2024-02-05T06:45:09.783036+0000","last_became_peered":"2024-02-05T06:45:09.783036+0000","last_unstale":"2024-02-05T06:48:25.191000+0000","last_undegraded":"2024-02-05T06:48:25.191000+0000","last_fullsized":"2024-02-05T06:48:25.191000+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'18","last_scrub_stamp":"2024-02-05T06:46:58.712125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:46:58.712125+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":20,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":672,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.187905+0000","last_change":"2024-02-05T06:48:20.050901+0000","last_active":"2024-02-05T06:48:25.187905+0000","last_peered":"2024-02-05T06:48:25.187905+0000","last_clean":"2024-02-05T06:48:25.187905+0000","last_became_active":"2024-02-05T06:33:33.592546+0000","last_became_peered":"2024-02-05T06:33:33.592546+0000","last_unstale":"2024-02-05T06:48:25.187905+0000","last_undegraded":"2024-02-05T06:48:25.187905+0000","last_fullsized":"2024-02-05T06:48:25.187905+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:20.050623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:42.685240+0000","last_clean_scrub_stamp":"2024-02-05T06:48:20.050623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:39.992367+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.4","version":"367'4885","reported_seq":4939,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:27.440844+0000","last_change":"2024-02-05T06:47:24.185199+0000","last_active":"2024-02-05T06:48:27.440844+0000","last_peered":"2024-02-05T06:48:27.440844+0000","last_clean":"2024-02-05T06:48:27.440844+0000","last_became_active":"2024-02-05T06:47:24.184891+0000","last_became_peered":"2024-02-05T06:47:24.184891+0000","last_unstale":"2024-02-05T06:48:27.440844+0000","last_undegraded":"2024-02-05T06:48:27.440844+0000","last_fullsized":"2024-02-05T06:48:27.440844+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":4885,"log_dups_size":0,"ondisk_log_size":4885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320143360,"num_objects":4885,"num_object_clones":0,"num_object_copies":14655,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4885,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4885,"num_write_kb":312640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":359,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.186214+0000","last_change":"2024-02-05T06:46:46.694131+0000","last_active":"2024-02-05T06:48:25.186214+0000","last_peered":"2024-02-05T06:48:25.186214+0000","last_clean":"2024-02-05T06:48:25.186214+0000","last_became_active":"2024-02-05T06:40:27.218081+0000","last_became_peered":"2024-02-05T06:40:27.218081+0000","last_unstale":"2024-02-05T06:48:25.186214+0000","last_undegraded":"2024-02-05T06:48:25.186214+0000","last_fullsized":"2024-02-05T06:48:25.186214+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:46.694027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:32.925000+0000","last_clean_scrub_stamp":"2024-02-05T06:46:46.694027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":690,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.192003+0000","last_change":"2024-02-05T06:47:45.073824+0000","last_active":"2024-02-05T06:48:25.192003+0000","last_peered":"2024-02-05T06:48:25.192003+0000","last_clean":"2024-02-05T06:48:25.192003+0000","last_became_active":"2024-02-05T06:41:43.413095+0000","last_became_peered":"2024-02-05T06:41:43.413095+0000","last_unstale":"2024-02-05T06:48:25.192003+0000","last_undegraded":"2024-02-05T06:48:25.192003+0000","last_fullsized":"2024-02-05T06:48:25.192003+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:45.073681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:45.073681+0000","last_clean_scrub_stamp":"2024-02-05T06:47:45.073681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:57.251457+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":365,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.187000+0000","last_change":"2024-02-05T06:47:36.783062+0000","last_active":"2024-02-05T06:48:25.187000+0000","last_peered":"2024-02-05T06:48:25.187000+0000","last_clean":"2024-02-05T06:48:25.187000+0000","last_became_active":"2024-02-05T06:41:43.418202+0000","last_became_peered":"2024-02-05T06:41:43.418202+0000","last_unstale":"2024-02-05T06:48:25.187000+0000","last_undegraded":"2024-02-05T06:48:25.187000+0000","last_fullsized":"2024-02-05T06:48:25.187000+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:36.782960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:47:36.782960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:48.010223+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"295'44","reported_seq":819,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.195018+0000","last_change":"2024-02-05T06:47:50.338639+0000","last_active":"2024-02-05T06:48:25.195018+0000","last_peered":"2024-02-05T06:48:25.195018+0000","last_clean":"2024-02-05T06:48:25.195018+0000","last_became_active":"2024-02-05T06:40:27.068417+0000","last_became_peered":"2024-02-05T06:40:27.068417+0000","last_unstale":"2024-02-05T06:48:25.195018+0000","last_undegraded":"2024-02-05T06:48:25.195018+0000","last_fullsized":"2024-02-05T06:48:25.195018+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'44","last_scrub_stamp":"2024-02-05T06:47:50.338505+0000","last_deep_scrub":"279'28","last_deep_scrub_stamp":"2024-02-05T06:42:50.148881+0000","last_clean_scrub_stamp":"2024-02-05T06:47:50.338505+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:03.563604+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"367'5034","reported_seq":5087,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:26.856279+0000","last_change":"2024-02-05T06:47:24.178230+0000","last_active":"2024-02-05T06:48:26.856279+0000","last_peered":"2024-02-05T06:48:26.856279+0000","last_clean":"2024-02-05T06:48:26.856279+0000","last_became_active":"2024-02-05T06:47:24.177943+0000","last_became_peered":"2024-02-05T06:47:24.177943+0000","last_unstale":"2024-02-05T06:48:26.856279+0000","last_undegraded":"2024-02-05T06:48:26.856279+0000","last_fullsized":"2024-02-05T06:48:26.856279+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":5034,"log_dups_size":0,"ondisk_log_size":5034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:45.429907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329908224,"num_objects":5034,"num_object_clones":0,"num_object_copies":15102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5034,"num_write_kb":322176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":651,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.193863+0000","last_change":"2024-02-05T06:46:33.542381+0000","last_active":"2024-02-05T06:48:25.193863+0000","last_peered":"2024-02-05T06:48:25.193863+0000","last_clean":"2024-02-05T06:48:25.193863+0000","last_became_active":"2024-02-05T06:40:27.129625+0000","last_became_peered":"2024-02-05T06:40:27.129625+0000","last_unstale":"2024-02-05T06:48:25.193863+0000","last_undegraded":"2024-02-05T06:48:25.193863+0000","last_fullsized":"2024-02-05T06:48:25.193863+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:33.542276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:46:33.542276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":692,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:26.126887+0000","last_change":"2024-02-05T06:48:26.126925+0000","last_active":"2024-02-05T06:48:26.126887+0000","last_peered":"2024-02-05T06:48:26.126887+0000","last_clean":"2024-02-05T06:48:26.126887+0000","last_became_active":"2024-02-05T06:36:54.966016+0000","last_became_peered":"2024-02-05T06:36:54.966016+0000","last_unstale":"2024-02-05T06:48:26.126887+0000","last_undegraded":"2024-02-05T06:48:26.126887+0000","last_fullsized":"2024-02-05T06:48:26.126887+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:26.126804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:48:26.126804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:55.439596+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":274,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.187741+0000","last_change":"2024-02-05T06:47:41.617654+0000","last_active":"2024-02-05T06:48:25.187741+0000","last_peered":"2024-02-05T06:48:25.187741+0000","last_clean":"2024-02-05T06:48:25.187741+0000","last_became_active":"2024-02-05T06:41:43.412611+0000","last_became_peered":"2024-02-05T06:41:43.412611+0000","last_unstale":"2024-02-05T06:48:25.187741+0000","last_undegraded":"2024-02-05T06:48:25.187741+0000","last_fullsized":"2024-02-05T06:48:25.187741+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:41.617526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:41.617526+0000","last_clean_scrub_stamp":"2024-02-05T06:47:41.617526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:51.274032+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"295'15","reported_seq":768,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.195041+0000","last_change":"2024-02-05T06:48:23.607645+0000","last_active":"2024-02-05T06:48:25.195041+0000","last_peered":"2024-02-05T06:48:25.195041+0000","last_clean":"2024-02-05T06:48:25.195041+0000","last_became_active":"2024-02-05T06:40:08.584204+0000","last_became_peered":"2024-02-05T06:40:08.584204+0000","last_unstale":"2024-02-05T06:48:25.195041+0000","last_undegraded":"2024-02-05T06:48:25.195041+0000","last_fullsized":"2024-02-05T06:48:25.195041+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:48:23.607540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:48:23.607540+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:43.691182+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"367'4972","reported_seq":5026,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:26.546542+0000","last_change":"2024-02-05T06:47:24.178003+0000","last_active":"2024-02-05T06:48:26.546542+0000","last_peered":"2024-02-05T06:48:26.546542+0000","last_clean":"2024-02-05T06:48:26.546542+0000","last_became_active":"2024-02-05T06:47:24.177714+0000","last_became_peered":"2024-02-05T06:47:24.177714+0000","last_unstale":"2024-02-05T06:48:26.546542+0000","last_undegraded":"2024-02-05T06:48:26.546542+0000","last_fullsized":"2024-02-05T06:48:26.546542+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":4972,"log_dups_size":0,"ondisk_log_size":4972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:28.973293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325844992,"num_objects":4972,"num_object_clones":0,"num_object_copies":14916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4972,"num_write_kb":318208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":692,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.190281+0000","last_change":"2024-02-05T06:46:47.669870+0000","last_active":"2024-02-05T06:48:25.190281+0000","last_peered":"2024-02-05T06:48:25.190281+0000","last_clean":"2024-02-05T06:48:25.190281+0000","last_became_active":"2024-02-05T06:41:46.054968+0000","last_became_peered":"2024-02-05T06:41:46.054968+0000","last_unstale":"2024-02-05T06:48:25.190281+0000","last_undegraded":"2024-02-05T06:48:25.190281+0000","last_fullsized":"2024-02-05T06:48:25.190281+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:46:47.669634+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:37.282442+0000","last_clean_scrub_stamp":"2024-02-05T06:46:47.669634+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":662,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.194439+0000","last_change":"2024-02-05T06:46:48.502814+0000","last_active":"2024-02-05T06:48:25.194439+0000","last_peered":"2024-02-05T06:48:25.194439+0000","last_clean":"2024-02-05T06:48:25.194439+0000","last_became_active":"2024-02-05T06:40:27.082440+0000","last_became_peered":"2024-02-05T06:40:27.082440+0000","last_unstale":"2024-02-05T06:48:25.194439+0000","last_undegraded":"2024-02-05T06:48:25.194439+0000","last_fullsized":"2024-02-05T06:48:25.194439+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:48.502695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:46:48.502695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"182'3","reported_seq":279,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.187824+0000","last_change":"2024-02-05T06:48:14.571567+0000","last_active":"2024-02-05T06:48:25.187824+0000","last_peered":"2024-02-05T06:48:25.187824+0000","last_clean":"2024-02-05T06:48:25.187824+0000","last_became_active":"2024-02-05T06:41:43.415806+0000","last_became_peered":"2024-02-05T06:41:43.415806+0000","last_unstale":"2024-02-05T06:48:25.187824+0000","last_undegraded":"2024-02-05T06:48:25.187824+0000","last_fullsized":"2024-02-05T06:48:25.187824+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'3","last_scrub_stamp":"2024-02-05T06:48:14.571441+0000","last_deep_scrub":"182'3","last_deep_scrub_stamp":"2024-02-05T06:39:33.373415+0000","last_clean_scrub_stamp":"2024-02-05T06:48:14.571441+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:26.936434+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.001220348,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":188,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.195131+0000","last_change":"2024-02-05T06:48:16.597953+0000","last_active":"2024-02-05T06:48:25.195131+0000","last_peered":"2024-02-05T06:48:25.195131+0000","last_clean":"2024-02-05T06:48:25.195131+0000","last_became_active":"2024-02-05T06:45:09.776169+0000","last_became_peered":"2024-02-05T06:45:09.776169+0000","last_unstale":"2024-02-05T06:48:25.195131+0000","last_undegraded":"2024-02-05T06:48:25.195131+0000","last_fullsized":"2024-02-05T06:48:25.195131+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:16.597853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:48:16.597853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:26.531928+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"326'52","reported_seq":530,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.187156+0000","last_change":"2024-02-05T06:48:16.104872+0000","last_active":"2024-02-05T06:48:25.187156+0000","last_peered":"2024-02-05T06:48:25.187156+0000","last_clean":"2024-02-05T06:48:25.187156+0000","last_became_active":"2024-02-05T06:40:08.604483+0000","last_became_peered":"2024-02-05T06:40:08.604483+0000","last_unstale":"2024-02-05T06:48:25.187156+0000","last_undegraded":"2024-02-05T06:48:25.187156+0000","last_fullsized":"2024-02-05T06:48:25.187156+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'52","last_scrub_stamp":"2024-02-05T06:48:16.104709+0000","last_deep_scrub":"326'52","last_deep_scrub_stamp":"2024-02-05T06:45:53.715482+0000","last_clean_scrub_stamp":"2024-02-05T06:48:16.104709+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:21.284859+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":664,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.195625+0000","last_change":"2024-02-05T06:47:49.412086+0000","last_active":"2024-02-05T06:48:25.195625+0000","last_peered":"2024-02-05T06:48:25.195625+0000","last_clean":"2024-02-05T06:48:25.195625+0000","last_became_active":"2024-02-05T06:36:54.973387+0000","last_became_peered":"2024-02-05T06:36:54.973387+0000","last_unstale":"2024-02-05T06:48:25.195625+0000","last_undegraded":"2024-02-05T06:48:25.195625+0000","last_fullsized":"2024-02-05T06:48:25.195625+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:49.411958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:47:49.411958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:08.497020+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"367'4816","reported_seq":4869,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:26.618912+0000","last_change":"2024-02-05T06:47:24.177476+0000","last_active":"2024-02-05T06:48:26.618912+0000","last_peered":"2024-02-05T06:48:26.618912+0000","last_clean":"2024-02-05T06:48:26.618912+0000","last_became_active":"2024-02-05T06:47:24.177145+0000","last_became_peered":"2024-02-05T06:47:24.177145+0000","last_unstale":"2024-02-05T06:48:26.618912+0000","last_undegraded":"2024-02-05T06:48:26.618912+0000","last_fullsized":"2024-02-05T06:48:26.618912+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":4816,"log_dups_size":0,"ondisk_log_size":4816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:45.854097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315621376,"num_objects":4816,"num_object_clones":0,"num_object_copies":14448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4816,"num_write_kb":308224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":780,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.186118+0000","last_change":"2024-02-05T06:46:44.660384+0000","last_active":"2024-02-05T06:48:25.186118+0000","last_peered":"2024-02-05T06:48:25.186118+0000","last_clean":"2024-02-05T06:48:25.186118+0000","last_became_active":"2024-02-05T06:40:30.082467+0000","last_became_peered":"2024-02-05T06:40:30.082467+0000","last_unstale":"2024-02-05T06:48:25.186118+0000","last_undegraded":"2024-02-05T06:48:25.186118+0000","last_fullsized":"2024-02-05T06:48:25.186118+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:46:44.660271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:46:44.660271+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":359,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.186827+0000","last_change":"2024-02-05T06:46:45.774572+0000","last_active":"2024-02-05T06:48:25.186827+0000","last_peered":"2024-02-05T06:48:25.186827+0000","last_clean":"2024-02-05T06:48:25.186827+0000","last_became_active":"2024-02-05T06:40:27.077966+0000","last_became_peered":"2024-02-05T06:40:27.077966+0000","last_unstale":"2024-02-05T06:48:25.186827+0000","last_undegraded":"2024-02-05T06:48:25.186827+0000","last_fullsized":"2024-02-05T06:48:25.186827+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:45.774441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:45.774441+0000","last_clean_scrub_stamp":"2024-02-05T06:46:45.774441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":659,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.189119+0000","last_change":"2024-02-05T06:46:32.768890+0000","last_active":"2024-02-05T06:48:25.189119+0000","last_peered":"2024-02-05T06:48:25.189119+0000","last_clean":"2024-02-05T06:48:25.189119+0000","last_became_active":"2024-02-05T06:33:29.551549+0000","last_became_peered":"2024-02-05T06:33:29.551549+0000","last_unstale":"2024-02-05T06:48:25.189119+0000","last_undegraded":"2024-02-05T06:48:25.189119+0000","last_fullsized":"2024-02-05T06:48:25.189119+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:32.768777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:46:32.768777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":690,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.188727+0000","last_change":"2024-02-05T06:47:49.065297+0000","last_active":"2024-02-05T06:48:25.188727+0000","last_peered":"2024-02-05T06:48:25.188727+0000","last_clean":"2024-02-05T06:48:25.188727+0000","last_became_active":"2024-02-05T06:33:28.542977+0000","last_became_peered":"2024-02-05T06:33:28.542977+0000","last_unstale":"2024-02-05T06:48:25.188727+0000","last_undegraded":"2024-02-05T06:48:25.188727+0000","last_fullsized":"2024-02-05T06:48:25.188727+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:49.065028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:47:49.065028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:57.562644+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00091640799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"295'17","reported_seq":757,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.200029+0000","last_change":"2024-02-05T06:46:43.816340+0000","last_active":"2024-02-05T06:48:25.200029+0000","last_peered":"2024-02-05T06:48:25.200029+0000","last_clean":"2024-02-05T06:48:25.200029+0000","last_became_active":"2024-02-05T06:41:43.420458+0000","last_became_peered":"2024-02-05T06:41:43.420458+0000","last_unstale":"2024-02-05T06:48:25.200029+0000","last_undegraded":"2024-02-05T06:48:25.200029+0000","last_fullsized":"2024-02-05T06:48:25.200029+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'17","last_scrub_stamp":"2024-02-05T06:46:43.816187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:46:43.816187+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":704,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.187763+0000","last_change":"2024-02-05T06:47:51.064183+0000","last_active":"2024-02-05T06:48:25.187763+0000","last_peered":"2024-02-05T06:48:25.187763+0000","last_clean":"2024-02-05T06:48:25.187763+0000","last_became_active":"2024-02-05T06:41:43.412387+0000","last_became_peered":"2024-02-05T06:41:43.412387+0000","last_unstale":"2024-02-05T06:48:25.187763+0000","last_undegraded":"2024-02-05T06:48:25.187763+0000","last_fullsized":"2024-02-05T06:48:25.187763+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:51.063926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:33.914422+0000","last_clean_scrub_stamp":"2024-02-05T06:47:51.063926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:55.492615+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"367'4948","reported_seq":5001,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:26.528687+0000","last_change":"2024-02-05T06:47:24.180383+0000","last_active":"2024-02-05T06:48:26.528687+0000","last_peered":"2024-02-05T06:48:26.528687+0000","last_clean":"2024-02-05T06:48:26.528687+0000","last_became_active":"2024-02-05T06:47:24.179124+0000","last_became_peered":"2024-02-05T06:47:24.179124+0000","last_unstale":"2024-02-05T06:48:26.528687+0000","last_undegraded":"2024-02-05T06:48:26.528687+0000","last_fullsized":"2024-02-05T06:48:26.528687+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":4948,"log_dups_size":0,"ondisk_log_size":4948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:49.727806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324272128,"num_objects":4948,"num_object_clones":0,"num_object_copies":14844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4948,"num_write_kb":316672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":275,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.189067+0000","last_change":"2024-02-05T06:46:48.716378+0000","last_active":"2024-02-05T06:48:25.189067+0000","last_peered":"2024-02-05T06:48:25.189067+0000","last_clean":"2024-02-05T06:48:25.189067+0000","last_became_active":"2024-02-05T06:41:43.408773+0000","last_became_peered":"2024-02-05T06:41:43.408773+0000","last_unstale":"2024-02-05T06:48:25.189067+0000","last_undegraded":"2024-02-05T06:48:25.189067+0000","last_fullsized":"2024-02-05T06:48:25.189067+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:48.716273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:46:48.716273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":781,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.186359+0000","last_change":"2024-02-05T06:46:33.834624+0000","last_active":"2024-02-05T06:48:25.186359+0000","last_peered":"2024-02-05T06:48:25.186359+0000","last_clean":"2024-02-05T06:48:25.186359+0000","last_became_active":"2024-02-05T06:40:30.081178+0000","last_became_peered":"2024-02-05T06:40:30.081178+0000","last_unstale":"2024-02-05T06:48:25.186359+0000","last_undegraded":"2024-02-05T06:48:25.186359+0000","last_fullsized":"2024-02-05T06:48:25.186359+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:46:33.834510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:46:33.834510+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":663,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.193539+0000","last_change":"2024-02-05T06:45:57.195429+0000","last_active":"2024-02-05T06:48:25.193539+0000","last_peered":"2024-02-05T06:48:25.193539+0000","last_clean":"2024-02-05T06:48:25.193539+0000","last_became_active":"2024-02-05T06:41:43.411335+0000","last_became_peered":"2024-02-05T06:41:43.411335+0000","last_unstale":"2024-02-05T06:48:25.193539+0000","last_undegraded":"2024-02-05T06:48:25.193539+0000","last_fullsized":"2024-02-05T06:48:25.193539+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:57.195321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:45:57.195321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"295'32","reported_seq":988,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.194482+0000","last_change":"2024-02-05T06:47:35.173300+0000","last_active":"2024-02-05T06:48:25.194482+0000","last_peered":"2024-02-05T06:48:25.194482+0000","last_clean":"2024-02-05T06:48:25.194482+0000","last_became_active":"2024-02-05T06:40:27.061111+0000","last_became_peered":"2024-02-05T06:40:27.061111+0000","last_unstale":"2024-02-05T06:48:25.194482+0000","last_undegraded":"2024-02-05T06:48:25.194482+0000","last_fullsized":"2024-02-05T06:48:25.194482+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'32","last_scrub_stamp":"2024-02-05T06:47:35.173150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:47:35.173150+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:58.867739+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":373,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.186758+0000","last_change":"2024-02-05T06:48:21.697741+0000","last_active":"2024-02-05T06:48:25.186758+0000","last_peered":"2024-02-05T06:48:25.186758+0000","last_clean":"2024-02-05T06:48:25.186758+0000","last_became_active":"2024-02-05T06:41:43.418198+0000","last_became_peered":"2024-02-05T06:41:43.418198+0000","last_unstale":"2024-02-05T06:48:25.186758+0000","last_undegraded":"2024-02-05T06:48:25.186758+0000","last_fullsized":"2024-02-05T06:48:25.186758+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:21.697458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:48:21.697458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:35.467036+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"367'4985","reported_seq":5039,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:27.441917+0000","last_change":"2024-02-05T06:47:24.177582+0000","last_active":"2024-02-05T06:48:27.441917+0000","last_peered":"2024-02-05T06:48:27.441917+0000","last_clean":"2024-02-05T06:48:27.441917+0000","last_became_active":"2024-02-05T06:47:24.177037+0000","last_became_peered":"2024-02-05T06:47:24.177037+0000","last_unstale":"2024-02-05T06:48:27.441917+0000","last_undegraded":"2024-02-05T06:48:27.441917+0000","last_fullsized":"2024-02-05T06:48:27.441917+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":4985,"log_dups_size":0,"ondisk_log_size":4985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:32.432264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326696960,"num_objects":4985,"num_object_clones":0,"num_object_copies":14955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4985,"num_write_kb":319040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":822,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.186798+0000","last_change":"2024-02-05T06:46:42.818851+0000","last_active":"2024-02-05T06:48:25.186798+0000","last_peered":"2024-02-05T06:48:25.186798+0000","last_clean":"2024-02-05T06:48:25.186798+0000","last_became_active":"2024-02-05T06:37:37.137754+0000","last_became_peered":"2024-02-05T06:37:37.137754+0000","last_unstale":"2024-02-05T06:48:25.186798+0000","last_undegraded":"2024-02-05T06:48:25.186798+0000","last_fullsized":"2024-02-05T06:48:25.186798+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:46:42.818725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:46:42.818725+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":667,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.187244+0000","last_change":"2024-02-05T06:46:31.716329+0000","last_active":"2024-02-05T06:48:25.187244+0000","last_peered":"2024-02-05T06:48:25.187244+0000","last_clean":"2024-02-05T06:48:25.187244+0000","last_became_active":"2024-02-05T06:41:43.410910+0000","last_became_peered":"2024-02-05T06:41:43.410910+0000","last_unstale":"2024-02-05T06:48:25.187244+0000","last_undegraded":"2024-02-05T06:48:25.187244+0000","last_fullsized":"2024-02-05T06:48:25.187244+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:31.716099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:46:31.716099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":672,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.193723+0000","last_change":"2024-02-05T06:46:56.459465+0000","last_active":"2024-02-05T06:48:25.193723+0000","last_peered":"2024-02-05T06:48:25.193723+0000","last_clean":"2024-02-05T06:48:25.193723+0000","last_became_active":"2024-02-05T06:36:54.972264+0000","last_became_peered":"2024-02-05T06:36:54.972264+0000","last_unstale":"2024-02-05T06:48:25.193723+0000","last_undegraded":"2024-02-05T06:48:25.193723+0000","last_fullsized":"2024-02-05T06:48:25.193723+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:56.459164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:46:56.459164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"355'6","reported_seq":847,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.186260+0000","last_change":"2024-02-05T06:47:46.642736+0000","last_active":"2024-02-05T06:48:25.186260+0000","last_peered":"2024-02-05T06:48:25.186260+0000","last_clean":"2024-02-05T06:48:25.186260+0000","last_became_active":"2024-02-05T06:40:27.071509+0000","last_became_peered":"2024-02-05T06:40:27.071509+0000","last_unstale":"2024-02-05T06:48:25.186260+0000","last_undegraded":"2024-02-05T06:48:25.186260+0000","last_fullsized":"2024-02-05T06:48:25.186260+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'6","last_scrub_stamp":"2024-02-05T06:47:46.642454+0000","last_deep_scrub":"355'6","last_deep_scrub_stamp":"2024-02-05T06:47:46.642454+0000","last_clean_scrub_stamp":"2024-02-05T06:47:46.642454+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:59.661377+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00074151600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":485,"num_read_kb":480,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"295'24","reported_seq":691,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.195011+0000","last_change":"2024-02-05T06:47:40.582521+0000","last_active":"2024-02-05T06:48:25.195011+0000","last_peered":"2024-02-05T06:48:25.195011+0000","last_clean":"2024-02-05T06:48:25.195011+0000","last_became_active":"2024-02-05T06:33:31.564604+0000","last_became_peered":"2024-02-05T06:33:31.564604+0000","last_unstale":"2024-02-05T06:48:25.195011+0000","last_undegraded":"2024-02-05T06:48:25.195011+0000","last_fullsized":"2024-02-05T06:48:25.195011+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:47:40.582236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:47:40.582236+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:54.033478+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":275,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.189166+0000","last_change":"2024-02-05T06:48:20.716170+0000","last_active":"2024-02-05T06:48:25.189166+0000","last_peered":"2024-02-05T06:48:25.189166+0000","last_clean":"2024-02-05T06:48:25.189166+0000","last_became_active":"2024-02-05T06:41:43.417751+0000","last_became_peered":"2024-02-05T06:41:43.417751+0000","last_unstale":"2024-02-05T06:48:25.189166+0000","last_undegraded":"2024-02-05T06:48:25.189166+0000","last_fullsized":"2024-02-05T06:48:25.189166+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:20.716046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:48:20.716046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:30.332921+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"367'5052","reported_seq":5105,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:26.558350+0000","last_change":"2024-02-05T06:47:24.182432+0000","last_active":"2024-02-05T06:48:26.558350+0000","last_peered":"2024-02-05T06:48:26.558350+0000","last_clean":"2024-02-05T06:48:26.558350+0000","last_became_active":"2024-02-05T06:47:24.182110+0000","last_became_peered":"2024-02-05T06:47:24.182110+0000","last_unstale":"2024-02-05T06:48:26.558350+0000","last_undegraded":"2024-02-05T06:48:26.558350+0000","last_fullsized":"2024-02-05T06:48:26.558350+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":5052,"log_dups_size":0,"ondisk_log_size":5052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:46.830260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331087872,"num_objects":5052,"num_object_clones":0,"num_object_copies":15156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5052,"num_write_kb":323328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":843,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.194863+0000","last_change":"2024-02-05T06:48:17.729708+0000","last_active":"2024-02-05T06:48:25.194863+0000","last_peered":"2024-02-05T06:48:25.194863+0000","last_clean":"2024-02-05T06:48:25.194863+0000","last_became_active":"2024-02-05T06:37:37.140011+0000","last_became_peered":"2024-02-05T06:37:37.140011+0000","last_unstale":"2024-02-05T06:48:25.194863+0000","last_undegraded":"2024-02-05T06:48:25.194863+0000","last_fullsized":"2024-02-05T06:48:25.194863+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:48:17.729582+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:38.778568+0000","last_clean_scrub_stamp":"2024-02-05T06:48:17.729582+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:39.030922+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":655,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.187007+0000","last_change":"2024-02-05T06:47:45.022360+0000","last_active":"2024-02-05T06:48:25.187007+0000","last_peered":"2024-02-05T06:48:25.187007+0000","last_clean":"2024-02-05T06:48:25.187007+0000","last_became_active":"2024-02-05T06:33:35.664858+0000","last_became_peered":"2024-02-05T06:33:35.664858+0000","last_unstale":"2024-02-05T06:48:25.187007+0000","last_undegraded":"2024-02-05T06:48:25.187007+0000","last_fullsized":"2024-02-05T06:48:25.187007+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:45.022240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:47:45.022240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:09.456304+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"66'1","reported_seq":693,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.187437+0000","last_change":"2024-02-05T06:46:30.749245+0000","last_active":"2024-02-05T06:48:25.187437+0000","last_peered":"2024-02-05T06:48:25.187437+0000","last_clean":"2024-02-05T06:48:25.187437+0000","last_became_active":"2024-02-05T06:41:43.411404+0000","last_became_peered":"2024-02-05T06:41:43.411404+0000","last_unstale":"2024-02-05T06:48:25.187437+0000","last_undegraded":"2024-02-05T06:48:25.187437+0000","last_fullsized":"2024-02-05T06:48:25.187437+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:46:30.748577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:46:30.748577+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"295'29","reported_seq":803,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.195307+0000","last_change":"2024-02-05T06:47:39.575482+0000","last_active":"2024-02-05T06:48:25.195307+0000","last_peered":"2024-02-05T06:48:25.195307+0000","last_clean":"2024-02-05T06:48:25.195307+0000","last_became_active":"2024-02-05T06:41:43.409664+0000","last_became_peered":"2024-02-05T06:41:43.409664+0000","last_unstale":"2024-02-05T06:48:25.195307+0000","last_undegraded":"2024-02-05T06:48:25.195307+0000","last_fullsized":"2024-02-05T06:48:25.195307+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'29","last_scrub_stamp":"2024-02-05T06:47:39.575331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:47:39.575331+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:55.789881+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":678,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.195105+0000","last_change":"2024-02-05T06:47:48.255664+0000","last_active":"2024-02-05T06:48:25.195105+0000","last_peered":"2024-02-05T06:48:25.195105+0000","last_clean":"2024-02-05T06:48:25.195105+0000","last_became_active":"2024-02-05T06:41:43.411550+0000","last_became_peered":"2024-02-05T06:41:43.411550+0000","last_unstale":"2024-02-05T06:48:25.195105+0000","last_undegraded":"2024-02-05T06:48:25.195105+0000","last_fullsized":"2024-02-05T06:48:25.195105+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:48.255364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:47:48.255364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:56.793069+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"367'4794","reported_seq":4847,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:26.656196+0000","last_change":"2024-02-05T06:47:24.185798+0000","last_active":"2024-02-05T06:48:26.656196+0000","last_peered":"2024-02-05T06:48:26.656196+0000","last_clean":"2024-02-05T06:48:26.656196+0000","last_became_active":"2024-02-05T06:47:24.185452+0000","last_became_peered":"2024-02-05T06:47:24.185452+0000","last_unstale":"2024-02-05T06:48:26.656196+0000","last_undegraded":"2024-02-05T06:48:26.656196+0000","last_fullsized":"2024-02-05T06:48:26.656196+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":4794,"log_dups_size":0,"ondisk_log_size":4794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:43.544292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314179584,"num_objects":4794,"num_object_clones":0,"num_object_copies":14382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4794,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4794,"num_write_kb":306816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":862,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.726145+0000","last_change":"2024-02-05T06:48:25.726145+0000","last_active":"2024-02-05T06:48:25.726145+0000","last_peered":"2024-02-05T06:48:25.726145+0000","last_clean":"2024-02-05T06:48:25.726145+0000","last_became_active":"2024-02-05T06:40:30.085438+0000","last_became_peered":"2024-02-05T06:40:30.085438+0000","last_unstale":"2024-02-05T06:48:25.726145+0000","last_undegraded":"2024-02-05T06:48:25.726145+0000","last_fullsized":"2024-02-05T06:48:25.726145+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:48:25.725892+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:48:25.725892+0000","last_clean_scrub_stamp":"2024-02-05T06:48:25.725892+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:51.750841+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":640,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.191172+0000","last_change":"2024-02-05T06:45:48.925337+0000","last_active":"2024-02-05T06:48:25.191172+0000","last_peered":"2024-02-05T06:48:25.191172+0000","last_clean":"2024-02-05T06:48:25.191172+0000","last_became_active":"2024-02-05T06:33:35.600485+0000","last_became_peered":"2024-02-05T06:33:35.600485+0000","last_unstale":"2024-02-05T06:48:25.191172+0000","last_undegraded":"2024-02-05T06:48:25.191172+0000","last_fullsized":"2024-02-05T06:48:25.191172+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:48.925218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:45:48.925218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":267,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.188302+0000","last_change":"2024-02-05T06:46:38.483037+0000","last_active":"2024-02-05T06:48:25.188302+0000","last_peered":"2024-02-05T06:48:25.188302+0000","last_clean":"2024-02-05T06:48:25.188302+0000","last_became_active":"2024-02-05T06:41:43.411732+0000","last_became_peered":"2024-02-05T06:41:43.411732+0000","last_unstale":"2024-02-05T06:48:25.188302+0000","last_undegraded":"2024-02-05T06:48:25.188302+0000","last_fullsized":"2024-02-05T06:48:25.188302+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:38.482939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:28.353956+0000","last_clean_scrub_stamp":"2024-02-05T06:46:38.482939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"295'20","reported_seq":765,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.190123+0000","last_change":"2024-02-05T06:48:21.064564+0000","last_active":"2024-02-05T06:48:25.190123+0000","last_peered":"2024-02-05T06:48:25.190123+0000","last_clean":"2024-02-05T06:48:25.190123+0000","last_became_active":"2024-02-05T06:41:43.421030+0000","last_became_peered":"2024-02-05T06:41:43.421030+0000","last_unstale":"2024-02-05T06:48:25.190123+0000","last_undegraded":"2024-02-05T06:48:25.190123+0000","last_fullsized":"2024-02-05T06:48:25.190123+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'20","last_scrub_stamp":"2024-02-05T06:48:21.064424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:48:21.064424+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:50.547913+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"335'11","reported_seq":861,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.194896+0000","last_change":"2024-02-05T06:46:52.427460+0000","last_active":"2024-02-05T06:48:25.194896+0000","last_peered":"2024-02-05T06:48:25.194896+0000","last_clean":"2024-02-05T06:48:25.194896+0000","last_became_active":"2024-02-05T06:41:43.423638+0000","last_became_peered":"2024-02-05T06:41:43.423638+0000","last_unstale":"2024-02-05T06:48:25.194896+0000","last_undegraded":"2024-02-05T06:48:25.194896+0000","last_fullsized":"2024-02-05T06:48:25.194896+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"335'11","last_scrub_stamp":"2024-02-05T06:46:52.423878+0000","last_deep_scrub":"335'11","last_deep_scrub_stamp":"2024-02-05T06:46:52.423878+0000","last_clean_scrub_stamp":"2024-02-05T06:46:52.423878+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"367'4960","reported_seq":5013,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:26.544641+0000","last_change":"2024-02-05T06:47:24.175068+0000","last_active":"2024-02-05T06:48:26.544641+0000","last_peered":"2024-02-05T06:48:26.544641+0000","last_clean":"2024-02-05T06:48:26.544641+0000","last_became_active":"2024-02-05T06:47:24.174574+0000","last_became_peered":"2024-02-05T06:47:24.174574+0000","last_unstale":"2024-02-05T06:48:26.544641+0000","last_undegraded":"2024-02-05T06:48:26.544641+0000","last_fullsized":"2024-02-05T06:48:26.544641+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":4960,"log_dups_size":0,"ondisk_log_size":4960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:30.861106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325058560,"num_objects":4960,"num_object_clones":0,"num_object_copies":14880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4960,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4960,"num_write_kb":317440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":857,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.194249+0000","last_change":"2024-02-05T06:45:54.266666+0000","last_active":"2024-02-05T06:48:25.194249+0000","last_peered":"2024-02-05T06:48:25.194249+0000","last_clean":"2024-02-05T06:48:25.194249+0000","last_became_active":"2024-02-05T06:41:46.053837+0000","last_became_peered":"2024-02-05T06:41:46.053837+0000","last_unstale":"2024-02-05T06:48:25.194249+0000","last_undegraded":"2024-02-05T06:48:25.194249+0000","last_fullsized":"2024-02-05T06:48:25.194249+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:45:54.266536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:45:54.266536+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":359,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.186537+0000","last_change":"2024-02-05T06:47:45.759776+0000","last_active":"2024-02-05T06:48:25.186537+0000","last_peered":"2024-02-05T06:48:25.186537+0000","last_clean":"2024-02-05T06:48:25.186537+0000","last_became_active":"2024-02-05T06:40:27.099168+0000","last_became_peered":"2024-02-05T06:40:27.099168+0000","last_unstale":"2024-02-05T06:48:25.186537+0000","last_undegraded":"2024-02-05T06:48:25.186537+0000","last_fullsized":"2024-02-05T06:48:25.186537+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:45.759651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:47:45.759651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:48.053819+0000","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"95'2","reported_seq":681,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.529770+0000","last_change":"2024-02-05T06:48:25.529812+0000","last_active":"2024-02-05T06:48:25.529770+0000","last_peered":"2024-02-05T06:48:25.529770+0000","last_clean":"2024-02-05T06:48:25.529770+0000","last_became_active":"2024-02-05T06:36:54.975093+0000","last_became_peered":"2024-02-05T06:36:54.975093+0000","last_unstale":"2024-02-05T06:48:25.529770+0000","last_undegraded":"2024-02-05T06:48:25.529770+0000","last_fullsized":"2024-02-05T06:48:25.529770+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-05T06:48:25.529692+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-05T06:35:25.616708+0000","last_clean_scrub_stamp":"2024-02-05T06:48:25.529692+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:35.785150+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"295'28","reported_seq":585,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.190030+0000","last_change":"2024-02-05T06:48:17.616455+0000","last_active":"2024-02-05T06:48:25.190030+0000","last_peered":"2024-02-05T06:48:25.190030+0000","last_clean":"2024-02-05T06:48:25.190030+0000","last_became_active":"2024-02-05T06:45:09.764696+0000","last_became_peered":"2024-02-05T06:45:09.764696+0000","last_unstale":"2024-02-05T06:48:25.190030+0000","last_undegraded":"2024-02-05T06:48:25.190030+0000","last_fullsized":"2024-02-05T06:48:25.190030+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'28","last_scrub_stamp":"2024-02-05T06:48:17.615783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:48:17.615783+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:46.705649+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":40,"num_read_kb":33,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"326'11","reported_seq":854,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.193072+0000","last_change":"2024-02-05T06:45:56.212330+0000","last_active":"2024-02-05T06:48:25.193072+0000","last_peered":"2024-02-05T06:48:25.193072+0000","last_clean":"2024-02-05T06:48:25.193072+0000","last_became_active":"2024-02-05T06:36:54.954819+0000","last_became_peered":"2024-02-05T06:36:54.954819+0000","last_unstale":"2024-02-05T06:48:25.193072+0000","last_undegraded":"2024-02-05T06:48:25.193072+0000","last_fullsized":"2024-02-05T06:48:25.193072+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'11","last_scrub_stamp":"2024-02-05T06:45:56.212177+0000","last_deep_scrub":"326'11","last_deep_scrub_stamp":"2024-02-05T06:45:56.212177+0000","last_clean_scrub_stamp":"2024-02-05T06:45:56.212177+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"367'4935","reported_seq":4989,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:26.552461+0000","last_change":"2024-02-05T06:47:24.184945+0000","last_active":"2024-02-05T06:48:26.552461+0000","last_peered":"2024-02-05T06:48:26.552461+0000","last_clean":"2024-02-05T06:48:26.552461+0000","last_became_active":"2024-02-05T06:47:24.184494+0000","last_became_peered":"2024-02-05T06:47:24.184494+0000","last_unstale":"2024-02-05T06:48:26.552461+0000","last_undegraded":"2024-02-05T06:48:26.552461+0000","last_fullsized":"2024-02-05T06:48:26.552461+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":4935,"log_dups_size":0,"ondisk_log_size":4935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:51.789612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323420160,"num_objects":4935,"num_object_clones":0,"num_object_copies":14805,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4935,"num_write_kb":315840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":424,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.188206+0000","last_change":"2024-02-05T06:47:46.034108+0000","last_active":"2024-02-05T06:48:25.188206+0000","last_peered":"2024-02-05T06:48:25.188206+0000","last_clean":"2024-02-05T06:48:25.188206+0000","last_became_active":"2024-02-05T06:45:09.776631+0000","last_became_peered":"2024-02-05T06:45:09.776631+0000","last_unstale":"2024-02-05T06:48:25.188206+0000","last_undegraded":"2024-02-05T06:48:25.188206+0000","last_fullsized":"2024-02-05T06:48:25.188206+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:47:46.033702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:47:46.033702+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:00.978347+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":181,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.193868+0000","last_change":"2024-02-05T06:45:49.145234+0000","last_active":"2024-02-05T06:48:25.193868+0000","last_peered":"2024-02-05T06:48:25.193868+0000","last_clean":"2024-02-05T06:48:25.193868+0000","last_became_active":"2024-02-05T06:45:09.780390+0000","last_became_peered":"2024-02-05T06:45:09.780390+0000","last_unstale":"2024-02-05T06:48:25.193868+0000","last_undegraded":"2024-02-05T06:48:25.193868+0000","last_fullsized":"2024-02-05T06:48:25.193868+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:49.145117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:45:49.145117+0000","last_clean_scrub_stamp":"2024-02-05T06:45:49.145117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":668,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.192426+0000","last_change":"2024-02-05T06:45:42.080964+0000","last_active":"2024-02-05T06:48:25.192426+0000","last_peered":"2024-02-05T06:48:25.192426+0000","last_clean":"2024-02-05T06:48:25.192426+0000","last_became_active":"2024-02-05T06:40:27.093739+0000","last_became_peered":"2024-02-05T06:40:27.093739+0000","last_unstale":"2024-02-05T06:48:25.192426+0000","last_undegraded":"2024-02-05T06:48:25.192426+0000","last_fullsized":"2024-02-05T06:48:25.192426+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:42.080862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:38:28.150695+0000","last_clean_scrub_stamp":"2024-02-05T06:45:42.080862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"295'30","reported_seq":745,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.192985+0000","last_change":"2024-02-05T06:47:50.402454+0000","last_active":"2024-02-05T06:48:25.192985+0000","last_peered":"2024-02-05T06:48:25.192985+0000","last_clean":"2024-02-05T06:48:25.192985+0000","last_became_active":"2024-02-05T06:40:27.277747+0000","last_became_peered":"2024-02-05T06:40:27.277747+0000","last_unstale":"2024-02-05T06:48:25.192985+0000","last_undegraded":"2024-02-05T06:48:25.192985+0000","last_fullsized":"2024-02-05T06:48:25.192985+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'30","last_scrub_stamp":"2024-02-05T06:47:50.402305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:47:50.402305+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:08.001287+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":322,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.193867+0000","last_change":"2024-02-05T06:46:30.377076+0000","last_active":"2024-02-05T06:48:25.193867+0000","last_peered":"2024-02-05T06:48:25.193867+0000","last_clean":"2024-02-05T06:48:25.193867+0000","last_became_active":"2024-02-05T06:45:09.778609+0000","last_became_peered":"2024-02-05T06:45:09.778609+0000","last_unstale":"2024-02-05T06:48:25.193867+0000","last_undegraded":"2024-02-05T06:48:25.193867+0000","last_fullsized":"2024-02-05T06:48:25.193867+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:30.376861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:46:30.376861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"367'4917","reported_seq":4970,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:26.663925+0000","last_change":"2024-02-05T06:47:24.182652+0000","last_active":"2024-02-05T06:48:26.663925+0000","last_peered":"2024-02-05T06:48:26.663925+0000","last_clean":"2024-02-05T06:48:26.663925+0000","last_became_active":"2024-02-05T06:47:24.182190+0000","last_became_peered":"2024-02-05T06:47:24.182190+0000","last_unstale":"2024-02-05T06:48:26.663925+0000","last_undegraded":"2024-02-05T06:48:26.663925+0000","last_fullsized":"2024-02-05T06:48:26.663925+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":4917,"log_dups_size":0,"ondisk_log_size":4917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:50.966086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322240512,"num_objects":4917,"num_object_clones":0,"num_object_copies":14751,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4917,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4917,"num_write_kb":314688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":694,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.187939+0000","last_change":"2024-02-05T06:48:24.641478+0000","last_active":"2024-02-05T06:48:25.187939+0000","last_peered":"2024-02-05T06:48:25.187939+0000","last_clean":"2024-02-05T06:48:25.187939+0000","last_became_active":"2024-02-05T06:41:46.053993+0000","last_became_peered":"2024-02-05T06:41:46.053993+0000","last_unstale":"2024-02-05T06:48:25.187939+0000","last_undegraded":"2024-02-05T06:48:25.187939+0000","last_fullsized":"2024-02-05T06:48:25.187939+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:48:24.641369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:48:24.641369+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:32.985701+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":274,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.188217+0000","last_change":"2024-02-05T06:46:45.769966+0000","last_active":"2024-02-05T06:48:25.188217+0000","last_peered":"2024-02-05T06:48:25.188217+0000","last_clean":"2024-02-05T06:48:25.188217+0000","last_became_active":"2024-02-05T06:41:43.418066+0000","last_became_peered":"2024-02-05T06:41:43.418066+0000","last_unstale":"2024-02-05T06:48:25.188217+0000","last_undegraded":"2024-02-05T06:48:25.188217+0000","last_fullsized":"2024-02-05T06:48:25.188217+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:45.769853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:45.769853+0000","last_clean_scrub_stamp":"2024-02-05T06:46:45.769853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"319'13","reported_seq":513,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.187994+0000","last_change":"2024-02-05T06:47:46.597798+0000","last_active":"2024-02-05T06:48:25.187994+0000","last_peered":"2024-02-05T06:48:25.187994+0000","last_clean":"2024-02-05T06:48:25.187994+0000","last_became_active":"2024-02-05T06:45:09.766220+0000","last_became_peered":"2024-02-05T06:45:09.766220+0000","last_unstale":"2024-02-05T06:48:25.187994+0000","last_undegraded":"2024-02-05T06:48:25.187994+0000","last_fullsized":"2024-02-05T06:48:25.187994+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'13","last_scrub_stamp":"2024-02-05T06:47:46.597456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:47:46.597456+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:57.094687+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":670,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.192892+0000","last_change":"2024-02-05T06:48:14.631495+0000","last_active":"2024-02-05T06:48:25.192892+0000","last_peered":"2024-02-05T06:48:25.192892+0000","last_clean":"2024-02-05T06:48:25.192892+0000","last_became_active":"2024-02-05T06:40:27.213439+0000","last_became_peered":"2024-02-05T06:40:27.213439+0000","last_unstale":"2024-02-05T06:48:25.192892+0000","last_undegraded":"2024-02-05T06:48:25.192892+0000","last_fullsized":"2024-02-05T06:48:25.192892+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:14.631382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:52.482566+0000","last_clean_scrub_stamp":"2024-02-05T06:48:14.631382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:29.843880+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"295'34","reported_seq":466,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.194240+0000","last_change":"2024-02-05T06:46:44.355699+0000","last_active":"2024-02-05T06:48:25.194240+0000","last_peered":"2024-02-05T06:48:25.194240+0000","last_clean":"2024-02-05T06:48:25.194240+0000","last_became_active":"2024-02-05T06:39:13.257978+0000","last_became_peered":"2024-02-05T06:39:13.257978+0000","last_unstale":"2024-02-05T06:48:25.194240+0000","last_undegraded":"2024-02-05T06:48:25.194240+0000","last_fullsized":"2024-02-05T06:48:25.194240+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'34","last_scrub_stamp":"2024-02-05T06:46:44.355563+0000","last_deep_scrub":"295'34","last_deep_scrub_stamp":"2024-02-05T06:46:44.355563+0000","last_clean_scrub_stamp":"2024-02-05T06:46:44.355563+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.0","version":"367'4783","reported_seq":4837,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:26.521196+0000","last_change":"2024-02-05T06:47:24.183477+0000","last_active":"2024-02-05T06:48:26.521196+0000","last_peered":"2024-02-05T06:48:26.521196+0000","last_clean":"2024-02-05T06:48:26.521196+0000","last_became_active":"2024-02-05T06:47:24.183098+0000","last_became_peered":"2024-02-05T06:47:24.183098+0000","last_unstale":"2024-02-05T06:48:26.521196+0000","last_undegraded":"2024-02-05T06:48:26.521196+0000","last_fullsized":"2024-02-05T06:48:26.521196+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":4783,"log_dups_size":0,"ondisk_log_size":4783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:34.968232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313458688,"num_objects":4783,"num_object_clones":0,"num_object_copies":14349,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4783,"num_write_kb":306112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":656,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.194422+0000","last_change":"2024-02-05T06:46:58.479329+0000","last_active":"2024-02-05T06:48:25.194422+0000","last_peered":"2024-02-05T06:48:25.194422+0000","last_clean":"2024-02-05T06:48:25.194422+0000","last_became_active":"2024-02-05T06:41:43.410600+0000","last_became_peered":"2024-02-05T06:41:43.410600+0000","last_unstale":"2024-02-05T06:48:25.194422+0000","last_undegraded":"2024-02-05T06:48:25.194422+0000","last_fullsized":"2024-02-05T06:48:25.194422+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:58.479208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:58.479208+0000","last_clean_scrub_stamp":"2024-02-05T06:46:58.479208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":274,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.188984+0000","last_change":"2024-02-05T06:46:57.669791+0000","last_active":"2024-02-05T06:48:25.188984+0000","last_peered":"2024-02-05T06:48:25.188984+0000","last_clean":"2024-02-05T06:48:25.188984+0000","last_became_active":"2024-02-05T06:41:43.417343+0000","last_became_peered":"2024-02-05T06:41:43.417343+0000","last_unstale":"2024-02-05T06:48:25.188984+0000","last_undegraded":"2024-02-05T06:48:25.188984+0000","last_fullsized":"2024-02-05T06:48:25.188984+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:57.669673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:42:55.610276+0000","last_clean_scrub_stamp":"2024-02-05T06:46:57.669673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":378,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.186703+0000","last_change":"2024-02-05T06:46:40.785712+0000","last_active":"2024-02-05T06:48:25.186703+0000","last_peered":"2024-02-05T06:48:25.186703+0000","last_clean":"2024-02-05T06:48:25.186703+0000","last_became_active":"2024-02-05T06:40:27.086808+0000","last_became_peered":"2024-02-05T06:40:27.086808+0000","last_unstale":"2024-02-05T06:48:25.186703+0000","last_undegraded":"2024-02-05T06:48:25.186703+0000","last_fullsized":"2024-02-05T06:48:25.186703+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:40.785606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:46:40.785606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"295'24","reported_seq":520,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.194664+0000","last_change":"2024-02-05T06:46:58.206860+0000","last_active":"2024-02-05T06:48:25.194664+0000","last_peered":"2024-02-05T06:48:25.194664+0000","last_clean":"2024-02-05T06:48:25.194664+0000","last_became_active":"2024-02-05T06:45:09.774868+0000","last_became_peered":"2024-02-05T06:45:09.774868+0000","last_unstale":"2024-02-05T06:48:25.194664+0000","last_undegraded":"2024-02-05T06:48:25.194664+0000","last_fullsized":"2024-02-05T06:48:25.194664+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:46:58.206720+0000","last_deep_scrub":"115'16","last_deep_scrub_stamp":"2024-02-05T06:36:34.793243+0000","last_clean_scrub_stamp":"2024-02-05T06:46:58.206720+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3","version":"367'5030","reported_seq":5084,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:26.846584+0000","last_change":"2024-02-05T06:47:24.177225+0000","last_active":"2024-02-05T06:48:26.846584+0000","last_peered":"2024-02-05T06:48:26.846584+0000","last_clean":"2024-02-05T06:48:26.846584+0000","last_became_active":"2024-02-05T06:47:24.176139+0000","last_became_peered":"2024-02-05T06:47:24.176139+0000","last_unstale":"2024-02-05T06:48:26.846584+0000","last_undegraded":"2024-02-05T06:48:26.846584+0000","last_fullsized":"2024-02-05T06:48:26.846584+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":5030,"log_dups_size":0,"ondisk_log_size":5030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:43.822725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329646080,"num_objects":5030,"num_object_clones":0,"num_object_copies":15090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5030,"num_write_kb":321920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"83'128","reported_seq":690,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.187071+0000","last_change":"2024-02-05T06:46:44.625857+0000","last_active":"2024-02-05T06:48:25.187071+0000","last_peered":"2024-02-05T06:48:25.187071+0000","last_clean":"2024-02-05T06:48:25.187071+0000","last_became_active":"2024-02-05T06:41:46.053324+0000","last_became_peered":"2024-02-05T06:41:46.053324+0000","last_unstale":"2024-02-05T06:48:25.187071+0000","last_undegraded":"2024-02-05T06:48:25.187071+0000","last_fullsized":"2024-02-05T06:48:25.187071+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:46:44.625739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:46:44.625739+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":681,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.188165+0000","last_change":"2024-02-05T06:47:42.155963+0000","last_active":"2024-02-05T06:48:25.188165+0000","last_peered":"2024-02-05T06:48:25.188165+0000","last_clean":"2024-02-05T06:48:25.188165+0000","last_became_active":"2024-02-05T06:36:54.958805+0000","last_became_peered":"2024-02-05T06:36:54.958805+0000","last_unstale":"2024-02-05T06:48:25.188165+0000","last_undegraded":"2024-02-05T06:48:25.188165+0000","last_fullsized":"2024-02-05T06:48:25.188165+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:42.155801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:47:42.155801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:53.870607+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":670,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.187524+0000","last_change":"2024-02-05T06:46:53.791434+0000","last_active":"2024-02-05T06:48:25.187524+0000","last_peered":"2024-02-05T06:48:25.187524+0000","last_clean":"2024-02-05T06:48:25.187524+0000","last_became_active":"2024-02-05T06:36:54.958082+0000","last_became_peered":"2024-02-05T06:36:54.958082+0000","last_unstale":"2024-02-05T06:48:25.187524+0000","last_undegraded":"2024-02-05T06:48:25.187524+0000","last_fullsized":"2024-02-05T06:48:25.187524+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:53.791322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:46:53.791322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":683,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.195185+0000","last_change":"2024-02-05T06:46:35.495500+0000","last_active":"2024-02-05T06:48:25.195185+0000","last_peered":"2024-02-05T06:48:25.195185+0000","last_clean":"2024-02-05T06:48:25.195185+0000","last_became_active":"2024-02-05T06:39:13.240038+0000","last_became_peered":"2024-02-05T06:39:13.240038+0000","last_unstale":"2024-02-05T06:48:25.195185+0000","last_undegraded":"2024-02-05T06:48:25.195185+0000","last_fullsized":"2024-02-05T06:48:25.195185+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:35.495370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:46:35.495370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"295'39","reported_seq":535,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.188380+0000","last_change":"2024-02-05T06:47:41.020322+0000","last_active":"2024-02-05T06:48:25.188380+0000","last_peered":"2024-02-05T06:48:25.188380+0000","last_clean":"2024-02-05T06:48:25.188380+0000","last_became_active":"2024-02-05T06:45:09.782645+0000","last_became_peered":"2024-02-05T06:45:09.782645+0000","last_unstale":"2024-02-05T06:48:25.188380+0000","last_undegraded":"2024-02-05T06:48:25.188380+0000","last_fullsized":"2024-02-05T06:48:25.188380+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'39","last_scrub_stamp":"2024-02-05T06:47:41.020104+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-02-05T06:35:26.274944+0000","last_clean_scrub_stamp":"2024-02-05T06:47:41.020104+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:06.072004+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"367'4930","reported_seq":4984,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:27.063433+0000","last_change":"2024-02-05T06:47:24.182784+0000","last_active":"2024-02-05T06:48:27.063433+0000","last_peered":"2024-02-05T06:48:27.063433+0000","last_clean":"2024-02-05T06:48:27.063433+0000","last_became_active":"2024-02-05T06:47:24.182360+0000","last_became_peered":"2024-02-05T06:47:24.182360+0000","last_unstale":"2024-02-05T06:48:27.063433+0000","last_undegraded":"2024-02-05T06:48:27.063433+0000","last_fullsized":"2024-02-05T06:48:27.063433+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":4930,"log_dups_size":0,"ondisk_log_size":4930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:30.397375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323092480,"num_objects":4930,"num_object_clones":0,"num_object_copies":14790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4930,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4930,"num_write_kb":315520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":777,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.185971+0000","last_change":"2024-02-05T06:48:17.676758+0000","last_active":"2024-02-05T06:48:25.185971+0000","last_peered":"2024-02-05T06:48:25.185971+0000","last_clean":"2024-02-05T06:48:25.185971+0000","last_became_active":"2024-02-05T06:40:30.095202+0000","last_became_peered":"2024-02-05T06:40:30.095202+0000","last_unstale":"2024-02-05T06:48:25.185971+0000","last_undegraded":"2024-02-05T06:48:25.185971+0000","last_fullsized":"2024-02-05T06:48:25.185971+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:48:17.676587+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:41:50.706159+0000","last_clean_scrub_stamp":"2024-02-05T06:48:17.676587+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:29.131120+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":189,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.199641+0000","last_change":"2024-02-05T06:47:49.056694+0000","last_active":"2024-02-05T06:48:25.199641+0000","last_peered":"2024-02-05T06:48:25.199641+0000","last_clean":"2024-02-05T06:48:25.199641+0000","last_became_active":"2024-02-05T06:45:09.784229+0000","last_became_peered":"2024-02-05T06:45:09.784229+0000","last_unstale":"2024-02-05T06:48:25.199641+0000","last_undegraded":"2024-02-05T06:48:25.199641+0000","last_fullsized":"2024-02-05T06:48:25.199641+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:49.056561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:30.057099+0000","last_clean_scrub_stamp":"2024-02-05T06:47:49.056561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:05.788674+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"249'13","reported_seq":901,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.188524+0000","last_change":"2024-02-05T06:47:52.101405+0000","last_active":"2024-02-05T06:48:25.188524+0000","last_peered":"2024-02-05T06:48:25.188524+0000","last_clean":"2024-02-05T06:48:25.188524+0000","last_became_active":"2024-02-05T06:41:43.413139+0000","last_became_peered":"2024-02-05T06:41:43.413139+0000","last_unstale":"2024-02-05T06:48:25.188524+0000","last_undegraded":"2024-02-05T06:48:25.188524+0000","last_fullsized":"2024-02-05T06:48:25.188524+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'13","last_scrub_stamp":"2024-02-05T06:47:52.101279+0000","last_deep_scrub":"92'9","last_deep_scrub_stamp":"2024-02-05T06:34:52.283780+0000","last_clean_scrub_stamp":"2024-02-05T06:47:52.101279+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:56.214584+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":662,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.197406+0000","last_change":"2024-02-05T06:45:30.989035+0000","last_active":"2024-02-05T06:48:25.197406+0000","last_peered":"2024-02-05T06:48:25.197406+0000","last_clean":"2024-02-05T06:48:25.197406+0000","last_became_active":"2024-02-05T06:36:54.971304+0000","last_became_peered":"2024-02-05T06:36:54.971304+0000","last_unstale":"2024-02-05T06:48:25.197406+0000","last_undegraded":"2024-02-05T06:48:25.197406+0000","last_fullsized":"2024-02-05T06:48:25.197406+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:30.987881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:45:30.987881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"295'25","reported_seq":515,"reported_epoch":367,"state":"active+clean","last_fresh":"2024-02-05T06:48:25.194658+0000","last_change":"2024-02-05T06:45:09.799601+0000","last_active":"2024-02-05T06:48:25.194658+0000","last_peered":"2024-02-05T06:48:25.194658+0000","last_clean":"2024-02-05T06:48:25.194658+0000","last_became_active":"2024-02-05T06:45:09.796902+0000","last_became_peered":"2024-02-05T06:45:09.796902+0000","last_unstale":"2024-02-05T06:48:25.194658+0000","last_undegraded":"2024-02-05T06:48:25.194658+0000","last_fullsized":"2024-02-05T06:48:25.194658+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'25","last_scrub_stamp":"2024-02-05T06:44:50.081252+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-02-05T06:35:04.584657+0000","last_clean_scrub_stamp":"2024-02-05T06:44:50.081252+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":5178130432,"num_objects":79012,"num_object_clones":0,"num_object_copies":237036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79012,"num_write_kb":5056768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15574499328,"data_stored":15574499328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79012,"ondisk_log_size":79012,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":485,"num_read_kb":480,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1573408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1106,"num_read_kb":863,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":373,"num_bytes_recovered":8234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":771,"ondisk_log_size":771,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103351,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1778060,"kb_used_data":1245700,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":91967092,"statfs":{"total":95995035648,"available":94174302208,"internally_reserved":0,"allocated":1275596800,"data_stored":1275083806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":545126858},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":1.135,"15min":1.476},"min":{"1min":0.40000000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.2909999999999999,"5min":4.1100000000000003,"15min":68.921000000000006},"last":0.433},{"interface":"front","average":{"1min":0.63700000000000001,"5min":1.2569999999999999,"15min":1.593},"min":{"1min":0.27300000000000002,"5min":0.222,"15min":0.222},"max":{"1min":1.403,"5min":18.013000000000002,"15min":69.616},"last":0.45200000000000001}]},{"osd":1,"last update":"Mon Feb 5 06:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":2.3919999999999999,"15min":1.7789999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.23599999999999999},"max":{"1min":1.504,"5min":84.216999999999999,"15min":89.231999999999999},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.66000000000000003,"5min":2.2480000000000002,"15min":1.879},"min":{"1min":0.34699999999999998,"5min":0.30399999999999999,"15min":0.26100000000000001},"max":{"1min":1.3540000000000001,"5min":84.126999999999995,"15min":87.995999999999995},"last":0.54700000000000004}]},{"osd":2,"last update":"Mon Feb 5 06:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":1.214,"15min":1.2330000000000001},"min":{"1min":0.32300000000000001,"5min":0.30299999999999999,"15min":0.26200000000000001},"max":{"1min":1.994,"5min":18.071999999999999,"15min":53.947000000000003},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":1.2310000000000001,"15min":1.206},"min":{"1min":0.39500000000000002,"5min":0.29999999999999999,"15min":0.26800000000000002},"max":{"1min":0.92800000000000005,"5min":18.045999999999999,"15min":53.299999999999997},"last":0.28899999999999998}]},{"osd":3,"last update":"Mon Feb 5 06:47:57 2024","interfaces":[{"interface":"back","average":{"1min":4.6980000000000004,"5min":2.371,"15min":2.165},"min":{"1min":0.35699999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":73.283000000000001,"5min":73.283000000000001,"15min":89.481999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":4.7990000000000004,"5min":2.4220000000000002,"15min":2.1259999999999999},"min":{"1min":0.34100000000000003,"5min":0.32900000000000001,"15min":0.27400000000000002},"max":{"1min":73.323999999999998,"5min":73.323999999999998,"15min":92.293999999999997},"last":0.39700000000000002}]},{"osd":4,"last update":"Mon Feb 5 06:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.81299999999999994,"15min":0.61499999999999999},"min":{"1min":0.29899999999999999,"5min":0.251,"15min":0.24199999999999999},"max":{"1min":1.329,"5min":4.1059999999999999,"15min":4.1059999999999999},"last":6.1980000000000004},{"interface":"front","average":{"1min":0.751,"5min":0.82299999999999995,"15min":0.60499999999999998},"min":{"1min":0.35599999999999998,"5min":0.29299999999999998,"15min":0.23400000000000001},"max":{"1min":2.7429999999999999,"5min":3.0819999999999999,"15min":3.0819999999999999},"last":0.66900000000000004}]},{"osd":5,"last update":"Mon Feb 5 06:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":1.083,"15min":0.72699999999999998},"min":{"1min":0.33700000000000002,"5min":0.251,"15min":0.251},"max":{"1min":1.276,"5min":6.2190000000000003,"15min":6.2190000000000003},"last":0.372},{"interface":"front","average":{"1min":0.71099999999999997,"5min":1.2769999999999999,"15min":0.80700000000000005},"min":{"1min":0.38,"5min":0.313,"15min":0.252},"max":{"1min":1.292,"5min":7.8360000000000003,"15min":7.8360000000000003},"last":0.48799999999999999}]},{"osd":6,"last update":"Mon Feb 5 06:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":1.01,"15min":0.76300000000000001},"min":{"1min":0.43099999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.379,"5min":4.524,"15min":4.524},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.996,"15min":0.71999999999999997},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.245},"max":{"1min":2.7869999999999999,"5min":3.3769999999999998,"15min":3.3769999999999998},"last":0.71899999999999997}]}]},{"osd":6,"up_from":49,"seq":210453397693,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2582964,"kb_used_data":2190828,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":91162188,"statfs":{"total":95995035648,"available":93350080512,"internally_reserved":0,"allocated":2243407872,"data_stored":2242918005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":401537478},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:48:20 2024","interfaces":[{"interface":"back","average":{"1min":1.855,"5min":1.0640000000000001,"15min":2.2210000000000001},"min":{"1min":0.33900000000000002,"5min":0.22900000000000001,"15min":0.224},"max":{"1min":18.393000000000001,"5min":18.393000000000001,"15min":98.918999999999997},"last":0.23699999999999999},{"interface":"front","average":{"1min":2.0899999999999999,"5min":1.052,"15min":2.1800000000000002},"min":{"1min":0.34599999999999997,"5min":0.245,"15min":0.23000000000000001},"max":{"1min":17.995999999999999,"5min":17.995999999999999,"15min":98.775000000000006},"last":0.80200000000000005}]},{"osd":1,"last update":"Mon Feb 5 06:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.96399999999999997,"15min":1.9199999999999999},"min":{"1min":0.27400000000000002,"5min":0.249,"15min":0.24099999999999999},"max":{"1min":2.363,"5min":5.2750000000000004,"15min":98.843999999999994},"last":0.75},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.93500000000000005,"15min":1.899},"min":{"1min":0.32800000000000001,"5min":0.26500000000000001,"15min":0.23100000000000001},"max":{"1min":4.516,"5min":4.516,"15min":101.637},"last":0.63}]},{"osd":2,"last update":"Mon Feb 5 06:48:20 2024","interfaces":[{"interface":"back","average":{"1min":1.919,"5min":1.131,"15min":1.905},"min":{"1min":0.377,"5min":0.26900000000000002,"15min":0.22500000000000001},"max":{"1min":16.131,"5min":16.131,"15min":85.617000000000004},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.0620000000000001,"15min":1.794},"min":{"1min":0.42099999999999999,"5min":0.29599999999999999,"15min":0.27000000000000002},"max":{"1min":16.210000000000001,"5min":16.210000000000001,"15min":85.335999999999999},"last":2.8530000000000002}]},{"osd":3,"last update":"Mon Feb 5 06:48:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.1160000000000001,"15min":2.379},"min":{"1min":0.315,"5min":0.313,"15min":0.27000000000000002},"max":{"1min":3.6360000000000001,"5min":6.2720000000000002,"15min":99.269000000000005},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.0009999999999999,"5min":0.997,"15min":2.4369999999999998},"min":{"1min":0.29599999999999999,"5min":0.28799999999999998,"15min":0.27800000000000002},"max":{"1min":3.8940000000000001,"5min":7.4210000000000003,"15min":101.75700000000001},"last":1.056}]},{"osd":4,"last update":"Mon Feb 5 06:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.69299999999999995,"15min":0.56499999999999995},"min":{"1min":0.33400000000000002,"5min":0.249,"15min":0.23100000000000001},"max":{"1min":3.444,"5min":3.444,"15min":3.742},"last":2.3639999999999999},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.69999999999999996,"15min":0.55900000000000005},"min":{"1min":0.34899999999999998,"5min":0.249,"15min":0.24299999999999999},"max":{"1min":2.319,"5min":4.4379999999999997,"15min":4.4379999999999997},"last":1.421}]},{"osd":5,"last update":"Mon Feb 5 06:48:20 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.298,"15min":0.79000000000000004},"min":{"1min":0.373,"5min":0.313,"15min":0.251},"max":{"1min":3.4079999999999999,"5min":36.548999999999999,"15min":36.548999999999999},"last":1.24},{"interface":"front","average":{"1min":1.077,"5min":1.3979999999999999,"15min":0.82499999999999996},"min":{"1min":0.34599999999999997,"5min":0.309,"15min":0.24299999999999999},"max":{"1min":4.6269999999999998,"5min":36.433,"15min":36.433},"last":0.69799999999999995}]},{"osd":7,"last update":"Mon Feb 5 06:47:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":1.1870000000000001,"15min":0.76000000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.30599999999999999},"max":{"1min":1.278,"5min":38.436999999999998,"15min":38.436999999999998},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.70999999999999996,"5min":1.117,"15min":0.77800000000000002},"min":{"1min":0.378,"5min":0.33000000000000002,"15min":0.29699999999999999},"max":{"1min":2.2850000000000001,"5min":36.633000000000003,"15min":36.633000000000003},"last":0.35799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575062,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1504172,"kb_used_data":945444,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":92240980,"statfs":{"total":95995035648,"available":94454763520,"internally_reserved":0,"allocated":968134656,"data_stored":967589677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:48:23 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.153,"15min":1.8029999999999999},"min":{"1min":0.26200000000000001,"5min":0.218,"15min":0.16700000000000001},"max":{"1min":3.5049999999999999,"5min":7.4370000000000003,"15min":67.421000000000006},"last":1.167},{"interface":"front","average":{"1min":1.2,"5min":1.2470000000000001,"15min":1.9370000000000001},"min":{"1min":0.224,"5min":0.20300000000000001,"15min":0.19600000000000001},"max":{"1min":3.8479999999999999,"5min":10.333,"15min":67.885000000000005},"last":7.9859999999999998}]},{"osd":2,"last update":"Mon Feb 5 06:47:38 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":1.0569999999999999,"15min":1.452},"min":{"1min":0.255,"5min":0.249,"15min":0.21099999999999999},"max":{"1min":5.4969999999999999,"5min":17.422000000000001,"15min":43.447000000000003},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.442,"5min":1.081,"15min":1.429},"min":{"1min":0.22700000000000001,"5min":0.22600000000000001,"15min":0.20899999999999999},"max":{"1min":1.0489999999999999,"5min":22.050000000000001,"15min":22.050000000000001},"last":0.56200000000000006}]},{"osd":3,"last update":"Mon Feb 5 06:48:09 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":1.417,"15min":2.157},"min":{"1min":0.27400000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":2.3359999999999999,"5min":10.456,"15min":69.798000000000002},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.69699999999999995,"5min":1.2310000000000001,"15min":1.8680000000000001},"min":{"1min":0.31,"5min":0.27400000000000002,"15min":0.23699999999999999},"max":{"1min":2.056,"5min":5.8520000000000003,"15min":70.765000000000001},"last":2.9630000000000001}]},{"osd":4,"last update":"Mon Feb 5 06:47:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.97199999999999998,"15min":1.095},"min":{"1min":0.40100000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.99399999999999999,"5min":6.3360000000000003,"15min":11.679},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.70199999999999996,"5min":1.3420000000000001,"15min":1.212},"min":{"1min":0.35899999999999999,"5min":0.30599999999999999,"15min":0.26300000000000001},"max":{"1min":2.3490000000000002,"5min":10.419,"15min":10.419},"last":0.51000000000000001}]},{"osd":5,"last update":"Mon Feb 5 06:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":1.163,"15min":1.232},"min":{"1min":0.44900000000000001,"5min":0.32900000000000001,"15min":0.29299999999999998},"max":{"1min":1.4570000000000001,"5min":5.5830000000000002,"15min":43.414999999999999},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.80000000000000004,"5min":1.1919999999999999,"15min":1.254},"min":{"1min":0.36199999999999999,"5min":0.35699999999999998,"15min":0.33100000000000002},"max":{"1min":2.7669999999999999,"5min":6.3849999999999998,"15min":13.455},"last":0.49399999999999999}]},{"osd":6,"last update":"Mon Feb 5 06:48:24 2024","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.1479999999999999,"15min":1.3160000000000001},"min":{"1min":0.434,"5min":0.376,"15min":0.34799999999999998},"max":{"1min":3.5550000000000002,"5min":23.902999999999999,"15min":23.902999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.155,"15min":1.266},"min":{"1min":0.49099999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":3.3780000000000001,"5min":5.5250000000000004,"15min":43.475000000000001},"last":0.39300000000000002}]},{"osd":7,"last update":"Mon Feb 5 06:47:38 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":1.2889999999999999,"15min":1.2929999999999999},"min":{"1min":0.45900000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.8779999999999999,"5min":8.016,"15min":13.49},"last":0.434},{"interface":"front","average":{"1min":0.628,"5min":1.425,"15min":1.3400000000000001},"min":{"1min":0.41699999999999998,"5min":0.40400000000000003,"15min":0.35499999999999998},"max":{"1min":1.6259999999999999,"5min":10.481,"15min":10.481},"last":0.47999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738588,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3650460,"kb_used_data":3150036,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":90094692,"statfs":{"total":95995035648,"available":92256964608,"internally_reserved":0,"allocated":3225636864,"data_stored":3225107462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 06:48:13 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":1.1619999999999999,"15min":1.464},"min":{"1min":0.22600000000000001,"5min":0.186,"15min":0.182},"max":{"1min":2.3439999999999999,"5min":18.311,"15min":18.59},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.381,"15min":1.4850000000000001},"min":{"1min":0.23799999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.4020000000000001,"5min":21.088000000000001,"15min":21.088000000000001},"last":2.7810000000000001}]},{"osd":2,"last update":"Mon Feb 5 06:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.86299999999999999,"15min":1.333},"min":{"1min":0.24199999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":3.661,"5min":3.661,"15min":16.981999999999999},"last":2.976},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.88400000000000001,"15min":1.262},"min":{"1min":0.27900000000000003,"5min":0.221,"15min":0.221},"max":{"1min":7.093,"5min":7.093,"15min":16.998999999999999},"last":4.984}]},{"osd":3,"last update":"Mon Feb 5 06:48:05 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":1.175,"15min":1.6359999999999999},"min":{"1min":0.314,"5min":0.27300000000000002,"15min":0.25700000000000001},"max":{"1min":2.032,"5min":18.349,"15min":43.302},"last":1.3620000000000001},{"interface":"front","average":{"1min":0.86099999999999999,"5min":1.3300000000000001,"15min":1.8340000000000001},"min":{"1min":0.33500000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.1390000000000002,"5min":18.712,"15min":43.651000000000003},"last":1.623}]},{"osd":4,"last update":"Mon Feb 5 06:48:26 2024","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":1.1020000000000001,"15min":1.3240000000000001},"min":{"1min":0.41299999999999998,"5min":0.312,"15min":0.27200000000000002},"max":{"1min":4.4989999999999997,"5min":4.4989999999999997,"15min":40.789999999999999},"last":2.5099999999999998},{"interface":"front","average":{"1min":1.224,"5min":1.0600000000000001,"15min":1.363},"min":{"1min":0.41499999999999998,"5min":0.33600000000000002,"15min":0.30099999999999999},"max":{"1min":4.1130000000000004,"5min":8.4019999999999992,"15min":40.773000000000003},"last":1.3200000000000001}]},{"osd":5,"last update":"Mon Feb 5 06:47:52 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":1.728,"15min":1.401},"min":{"1min":0.38900000000000001,"5min":0.32500000000000001,"15min":0.29299999999999998},"max":{"1min":1.252,"5min":48.841999999999999,"15min":48.841999999999999},"last":1.663},{"interface":"front","average":{"1min":0.93899999999999995,"5min":1.619,"15min":1.373},"min":{"1min":0.47199999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":4.3449999999999998,"5min":48.450000000000003,"15min":48.450000000000003},"last":2.863}]},{"osd":6,"last update":"Mon Feb 5 06:48:13 2024","interfaces":[{"interface":"back","average":{"1min":9.0020000000000007,"5min":2.54,"15min":1.746},"min":{"1min":0.42099999999999999,"5min":0.376,"15min":0.33700000000000002},"max":{"1min":136.76300000000001,"5min":136.76300000000001,"15min":136.76300000000001},"last":1.387},{"interface":"front","average":{"1min":9.1989999999999998,"5min":2.8260000000000001,"15min":1.8},"min":{"1min":0.51300000000000001,"5min":0.39200000000000002,"15min":0.33000000000000002},"max":{"1min":135.92400000000001,"5min":135.92400000000001,"15min":135.92400000000001},"last":1.6950000000000001}]},{"osd":7,"last update":"Mon Feb 5 06:47:41 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":1.081,"15min":1.204},"min":{"1min":0.434,"5min":0.40100000000000002,"15min":0.31900000000000001},"max":{"1min":2.391,"5min":6.1420000000000003,"15min":17.210000000000001},"last":2.7429999999999999},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.98199999999999998,"15min":1.276},"min":{"1min":0.435,"5min":0.378,"15min":0.34899999999999998},"max":{"1min":1.496,"5min":6.7220000000000004,"15min":18.533999999999999},"last":1.4199999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411537,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2322252,"kb_used_data":1885692,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":91422900,"statfs":{"total":95995035648,"available":93617049600,"internally_reserved":0,"allocated":1930948608,"data_stored":1930444219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:47:39 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":1.0600000000000001,"15min":1.72},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.0259999999999998,"5min":14.007,"15min":50.814},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.77600000000000002,"5min":1.1379999999999999,"15min":1.7230000000000001},"min":{"1min":0.23100000000000001,"5min":0.22600000000000001,"15min":0.17699999999999999},"max":{"1min":4.7480000000000002,"5min":13.968999999999999,"15min":50.802999999999997},"last":0.58499999999999996}]},{"osd":1,"last update":"Mon Feb 5 06:47:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":1.482,"15min":1.6259999999999999},"min":{"1min":0.19700000000000001,"5min":0.189,"15min":0.189},"max":{"1min":3.1429999999999998,"5min":15.382,"15min":31.303000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":1.028,"5min":1.54,"15min":1.3979999999999999},"min":{"1min":0.27700000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.3019999999999996,"5min":15.742000000000001,"15min":31.425000000000001},"last":0.78800000000000003}]},{"osd":3,"last update":"Mon Feb 5 06:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":1.153,"15min":1.8320000000000001},"min":{"1min":0.316,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.591,"5min":15.166,"15min":53.127000000000002},"last":1.042},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.1299999999999999,"15min":1.7529999999999999},"min":{"1min":0.30199999999999999,"5min":0.23400000000000001,"15min":0.22500000000000001},"max":{"1min":2.2930000000000001,"5min":15.541,"15min":52.607999999999997},"last":0.62}]},{"osd":4,"last update":"Mon Feb 5 06:48:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":0.95199999999999996,"15min":1.03},"min":{"1min":0.38,"5min":0.29999999999999999,"15min":0.23999999999999999},"max":{"1min":3.573,"5min":7.0110000000000001,"15min":12.041},"last":1.45},{"interface":"front","average":{"1min":1.075,"5min":0.871,"15min":0.98399999999999999},"min":{"1min":0.40100000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":4.907,"5min":5.0350000000000001,"15min":15.689},"last":1.2370000000000001}]},{"osd":5,"last update":"Mon Feb 5 06:47:54 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":1.2330000000000001,"15min":1.4750000000000001},"min":{"1min":0.33800000000000002,"5min":0.315,"15min":0.29899999999999999},"max":{"1min":1.2729999999999999,"5min":30.265000000000001,"15min":86.614999999999995},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.89400000000000002,"5min":1.373,"15min":1.6990000000000001},"min":{"1min":0.36599999999999999,"5min":0.312,"15min":0.312},"max":{"1min":3.222,"5min":30.02,"15min":86.543999999999997},"last":0.68500000000000005}]},{"osd":6,"last update":"Mon Feb 5 06:48:25 2024","interfaces":[{"interface":"back","average":{"1min":2.6030000000000002,"5min":1.2190000000000001,"15min":1.161},"min":{"1min":0.437,"5min":0.35499999999999998,"15min":0.315},"max":{"1min":29.780999999999999,"5min":29.780999999999999,"15min":29.780999999999999},"last":2.3450000000000002},{"interface":"front","average":{"1min":3.2999999999999998,"5min":1.6739999999999999,"15min":1.446},"min":{"1min":0.46300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":29.763000000000002,"5min":29.763000000000002,"15min":29.763000000000002},"last":2.2170000000000001}]},{"osd":7,"last update":"Mon Feb 5 06:47:43 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":1.6859999999999999,"15min":1.4510000000000001},"min":{"1min":0.495,"5min":0.373,"15min":0.35699999999999998},"max":{"1min":3.2530000000000001,"5min":45.243000000000002,"15min":45.243000000000002},"last":1.5109999999999999},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.702,"15min":1.3520000000000001},"min":{"1min":0.40799999999999997,"5min":0.379,"15min":0.36899999999999999},"max":{"1min":17.611000000000001,"5min":45.171999999999997,"15min":45.171999999999997},"last":1.0029999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149900,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2233496,"kb_used_data":1616464,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":91511656,"statfs":{"total":95995035648,"available":93707935744,"internally_reserved":0,"allocated":1655259136,"data_stored":1654733730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":631830983},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.3970000000000002,"5min":1.9510000000000001,"15min":1.5349999999999999},"min":{"1min":0.21099999999999999,"5min":0.189,"15min":0.184},"max":{"1min":53.656999999999996,"5min":53.656999999999996,"15min":53.656999999999996},"last":1.3799999999999999},{"interface":"front","average":{"1min":4.1260000000000003,"5min":1.8280000000000001,"15min":1.736},"min":{"1min":0.217,"5min":0.217,"15min":0.20399999999999999},"max":{"1min":53.338000000000001,"5min":53.338000000000001,"15min":53.338000000000001},"last":2.9060000000000001}]},{"osd":1,"last update":"Mon Feb 5 06:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":1.4370000000000001,"15min":1.4950000000000001},"min":{"1min":0.29199999999999998,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":2.157,"5min":11.412000000000001,"15min":11.412000000000001},"last":2.766},{"interface":"front","average":{"1min":0.56699999999999995,"5min":1.419,"15min":1.518},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.215},"max":{"1min":1.5680000000000001,"5min":9.8659999999999997,"15min":9.8659999999999997},"last":0.80600000000000005}]},{"osd":2,"last update":"Mon Feb 5 06:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":1.157,"15min":1.5649999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.3540000000000001,"5min":7.7080000000000002,"15min":41.529000000000003},"last":2.238},{"interface":"front","average":{"1min":0.623,"5min":1.0640000000000001,"15min":1.4570000000000001},"min":{"1min":0.28499999999999998,"5min":0.249,"15min":0.248},"max":{"1min":2.1949999999999998,"5min":7.3559999999999999,"15min":41.759999999999998},"last":1.4259999999999999}]},{"osd":4,"last update":"Mon Feb 5 06:47:34 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":1.24,"15min":1.109},"min":{"1min":0.41999999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.8360000000000001,"5min":7.7329999999999997,"15min":16.577999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":1.607,"15min":1.369},"min":{"1min":0.371,"5min":0.30099999999999999,"15min":0.28599999999999998},"max":{"1min":1.831,"5min":19.359999999999999,"15min":19.359999999999999},"last":0.36199999999999999}]},{"osd":5,"last update":"Mon Feb 5 06:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":1.464,"15min":1.4339999999999999},"min":{"1min":0.46600000000000003,"5min":0.32000000000000001,"15min":0.29299999999999998},"max":{"1min":1.5589999999999999,"5min":20.257999999999999,"15min":20.257999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.626,"5min":1.3140000000000001,"15min":1.181},"min":{"1min":0.441,"5min":0.38700000000000001,"15min":0.28399999999999997},"max":{"1min":0.95999999999999996,"5min":7.859,"15min":15.792999999999999},"last":0.81999999999999995}]},{"osd":6,"last update":"Mon Feb 5 06:47:27 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":1.429,"15min":1.319},"min":{"1min":0.46100000000000002,"5min":0.34999999999999998,"15min":0.28100000000000003},"max":{"1min":3.5630000000000002,"5min":7.556,"15min":8.3610000000000007},"last":0.40500000000000003},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.284,"15min":1.276},"min":{"1min":0.46300000000000002,"5min":0.44900000000000001,"15min":0.28599999999999998},"max":{"1min":2.4550000000000001,"5min":7.2439999999999998,"15min":7.2439999999999998},"last":0.501}]},{"osd":7,"last update":"Mon Feb 5 06:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":1.3600000000000001,"15min":1.335},"min":{"1min":0.376,"5min":0.376,"15min":0.33400000000000002},"max":{"1min":1.1950000000000001,"5min":7.8029999999999999,"15min":45.039000000000001},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.75800000000000001,"5min":1.3220000000000001,"15min":1.4770000000000001},"min":{"1min":0.51500000000000001,"5min":0.377,"15min":0.34399999999999997},"max":{"1min":1.349,"5min":7.2720000000000002,"15min":44.036999999999999},"last":0.53700000000000003}]}]},{"osd":4,"up_from":31,"seq":133143986374,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2994004,"kb_used_data":2493580,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":90751148,"statfs":{"total":95995035648,"available":92929175552,"internally_reserved":0,"allocated":2553425920,"data_stored":2552904288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:47:28 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.83499999999999996,"15min":1.151},"min":{"1min":0.28499999999999998,"5min":0.20799999999999999,"15min":0.20200000000000001},"max":{"1min":3.0910000000000002,"5min":6.0220000000000002,"15min":33.372},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.95899999999999996,"15min":1.2829999999999999},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.22700000000000001},"max":{"1min":2.9830000000000001,"5min":10.913,"15min":33.323999999999998},"last":4.1559999999999997}]},{"osd":1,"last update":"Mon Feb 5 06:47:28 2024","interfaces":[{"interface":"back","average":{"1min":1.5269999999999999,"5min":1.177,"15min":1.0900000000000001},"min":{"1min":0.28399999999999997,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":17.283000000000001,"5min":17.283000000000001,"15min":33.747999999999998},"last":0.46200000000000002},{"interface":"front","average":{"1min":1.454,"5min":1.1040000000000001,"15min":1.0620000000000001},"min":{"1min":0.255,"5min":0.23699999999999999,"15min":0.23000000000000001},"max":{"1min":17.460999999999999,"5min":17.460999999999999,"15min":33.691000000000003},"last":0.747}]},{"osd":2,"last update":"Mon Feb 5 06:47:27 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.88500000000000001,"15min":0.80200000000000005},"min":{"1min":0.29199999999999998,"5min":0.27500000000000002,"15min":0.245},"max":{"1min":1.8160000000000001,"5min":17.379999999999999,"15min":17.379999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.72799999999999998,"5min":1.2,"15min":1.04},"min":{"1min":0.35499999999999998,"5min":0.29799999999999999,"15min":0.26000000000000001},"max":{"1min":2.6099999999999999,"5min":18.077000000000002,"15min":18.077000000000002},"last":2.012}]},{"osd":3,"last update":"Mon Feb 5 06:47:28 2024","interfaces":[{"interface":"back","average":{"1min":1.893,"5min":2.3780000000000001,"15min":2.3740000000000001},"min":{"1min":0.36799999999999999,"5min":0.32000000000000001,"15min":0.252},"max":{"1min":20.562999999999999,"5min":44.601999999999997,"15min":99.546999999999997},"last":2.4950000000000001},{"interface":"front","average":{"1min":1.5840000000000001,"5min":2.3450000000000002,"15min":2.3650000000000002},"min":{"1min":0.33200000000000002,"5min":0.29799999999999999,"15min":0.29099999999999998},"max":{"1min":17.332000000000001,"5min":44.511000000000003,"15min":99.245000000000005},"last":2.359}]},{"osd":5,"last update":"Mon Feb 5 06:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.87,"15min":0.69099999999999995},"min":{"1min":0.249,"5min":0.249,"15min":0.24399999999999999},"max":{"1min":1.4930000000000001,"5min":5.6059999999999999,"15min":11.098000000000001},"last":1.696},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.82599999999999996,"15min":0.66000000000000003},"min":{"1min":0.28899999999999998,"5min":0.28499999999999998,"15min":0.252},"max":{"1min":2.5339999999999998,"5min":4.1210000000000004,"15min":10.965999999999999},"last":0.93100000000000005}]},{"osd":6,"last update":"Mon Feb 5 06:47:27 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.87,"15min":0.71299999999999997},"min":{"1min":0.33900000000000002,"5min":0.31,"15min":0.246},"max":{"1min":3.2829999999999999,"5min":10.039999999999999,"15min":11.868},"last":5.9119999999999999},{"interface":"front","average":{"1min":0.751,"5min":0.83799999999999997,"15min":0.71399999999999997},"min":{"1min":0.35199999999999998,"5min":0.32700000000000001,"15min":0.27600000000000002},"max":{"1min":3.25,"5min":10.074,"15min":11.787000000000001},"last":5.8970000000000002}]},{"osd":7,"last update":"Mon Feb 5 06:47:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.80600000000000005,"15min":0.65400000000000003},"min":{"1min":0.32900000000000001,"5min":0.316,"15min":0.26000000000000001},"max":{"1min":1.7749999999999999,"5min":3.4129999999999998,"15min":5.1500000000000004},"last":0.432},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.81299999999999994,"15min":0.68899999999999995},"min":{"1min":0.314,"5min":0.307,"15min":0.28499999999999998},"max":{"1min":2.9009999999999998,"5min":4.5350000000000001,"15min":5.5890000000000004},"last":3.3769999999999998}]}]},{"osd":5,"up_from":40,"seq":171798692033,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2209988,"kb_used_data":1587388,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":91535164,"statfs":{"total":95995035648,"available":93732007936,"internally_reserved":0,"allocated":1625485312,"data_stored":1624980817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":637532614},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,1,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.079,"15min":1.492},"min":{"1min":0.33900000000000002,"5min":0.24299999999999999,"15min":0.19700000000000001},"max":{"1min":4.2869999999999999,"5min":4.6479999999999997,"15min":55.704999999999998},"last":0.60499999999999998},{"interface":"front","average":{"1min":1.02,"5min":1.052,"15min":1.681},"min":{"1min":0.379,"5min":0.255,"15min":0.22600000000000001},"max":{"1min":5.2030000000000003,"5min":6.532,"15min":56.789000000000001},"last":3.0640000000000001}]},{"osd":1,"last update":"Mon Feb 5 06:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":1.1160000000000001,"15min":1.389},"min":{"1min":0.378,"5min":0.26900000000000002,"15min":0.248},"max":{"1min":4.6340000000000003,"5min":6.5049999999999999,"15min":41.191000000000003},"last":2.964},{"interface":"front","average":{"1min":0.89000000000000001,"5min":1.153,"15min":1.252},"min":{"1min":0.38200000000000001,"5min":0.28100000000000003,"15min":0.23499999999999999},"max":{"1min":4.8609999999999998,"5min":4.8609999999999998,"15min":41.173000000000002},"last":0.69799999999999995}]},{"osd":2,"last update":"Mon Feb 5 06:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.84399999999999997,"15min":1.25},"min":{"1min":0.33900000000000002,"5min":0.26800000000000002,"15min":0.23000000000000001},"max":{"1min":0.75700000000000001,"5min":4.0599999999999996,"15min":36.901000000000003},"last":1.5569999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.95999999999999996,"15min":1.2030000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.94599999999999995,"5min":6.4619999999999997,"15min":37.984000000000002},"last":3.0910000000000002}]},{"osd":3,"last update":"Mon Feb 5 06:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":1.716,"15min":1.585},"min":{"1min":0.34399999999999997,"5min":0.33200000000000002,"15min":0.31900000000000001},"max":{"1min":1.0620000000000001,"5min":62.508000000000003,"15min":62.508000000000003},"last":1.6240000000000001},{"interface":"front","average":{"1min":0.79300000000000004,"5min":1.929,"15min":1.748},"min":{"1min":0.36499999999999999,"5min":0.28100000000000003,"15min":0.27400000000000002},"max":{"1min":4.4820000000000002,"5min":62.460000000000001,"15min":62.460000000000001},"last":3.0049999999999999}]},{"osd":4,"last update":"Mon Feb 5 06:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.83699999999999997,"15min":0.60399999999999998},"min":{"1min":0.33400000000000002,"5min":0.26200000000000001,"15min":0.23200000000000001},"max":{"1min":1.099,"5min":4.7370000000000001,"15min":5.4969999999999999},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.79900000000000004,"15min":0.61699999999999999},"min":{"1min":0.35099999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.61199999999999999,"5min":4.0380000000000003,"15min":4.0380000000000003},"last":1.5880000000000001}]},{"osd":6,"last update":"Mon Feb 5 06:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":1.024,"15min":0.72799999999999998},"min":{"1min":0.308,"5min":0.28000000000000003,"15min":0.23799999999999999},"max":{"1min":3.2269999999999999,"5min":7.6710000000000003,"15min":7.6710000000000003},"last":3.1179999999999999},{"interface":"front","average":{"1min":0.91900000000000004,"5min":1.002,"15min":0.76900000000000002},"min":{"1min":0.38800000000000001,"5min":0.33900000000000002,"15min":0.27600000000000002},"max":{"1min":3.266,"5min":7.21,"15min":7.21},"last":1.4730000000000001}]},{"osd":7,"last update":"Mon Feb 5 06:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":1.073,"15min":0.72299999999999998},"min":{"1min":0.38500000000000001,"5min":0.30199999999999999,"15min":0.25600000000000001},"max":{"1min":0.88200000000000001,"5min":6.9870000000000001,"15min":6.9870000000000001},"last":4.25},{"interface":"front","average":{"1min":0.55400000000000005,"5min":1.0680000000000001,"15min":0.75600000000000001},"min":{"1min":0.36599999999999999,"5min":0.28999999999999998,"15min":0.27100000000000002},"max":{"1min":0.68999999999999995,"5min":6.5839999999999996,"15min":6.5839999999999996},"last":3.036}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3273523200,"data_stored":3273523200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":967114752,"data_stored":967114752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1952186368,"data_stored":1952186368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1654063104,"data_stored":1654063104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2555445248,"data_stored":2555445248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1624637440,"data_stored":1624637440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2272788480,"data_stored":2272788480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1274740736,"data_stored":1274740736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T06:48:29.692 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-05T06:48:29.692 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T06:48:29.693 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_3 2024-02-05T06:48:29.693 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T06:48:29.770 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:29.810 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:29.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:29.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:29.912 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:29.947 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 50 16 82842 82826 103.506 74.125 0.0226595 0.00965601 2024-02-05T06:48:29.960 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:29.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:29.961 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:30.062 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:30.097 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:30.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:30.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:30.199 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:30.241 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:30.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:30.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:30.342 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:30.381 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:30.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:30.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:30.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:30 smithi028 ceph-mon[29442]: pgmap v718: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-02-05T06:48:30.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:30 smithi028 ceph-mon[29442]: from='client.15783 v1:172.21.15.28:0/1395856957' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:48:30.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:30 smithi028 ceph-mon[29442]: 1.2 deep-scrub starts 2024-02-05T06:48:30.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:30 smithi028 ceph-mon[29442]: 1.2 deep-scrub ok 2024-02-05T06:48:30.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:30 smithi028 ceph-mon[29442]: 3.10 scrub starts 2024-02-05T06:48:30.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:30 smithi028 ceph-mon[29442]: 3.10 scrub ok 2024-02-05T06:48:30.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:30 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3587513508' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:48:30.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:30 smithi028 ceph-mon[29442]: 5.1d scrub starts 2024-02-05T06:48:30.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:30 smithi028 ceph-mon[29442]: 5.1d scrub ok 2024-02-05T06:48:30.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:30 smithi028 ceph-mon[29442]: 4.16 scrub starts 2024-02-05T06:48:30.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:30 smithi028 ceph-mon[29442]: 4.16 scrub ok 2024-02-05T06:48:30.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:30 smithi028 ceph-mon[37984]: pgmap v718: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-02-05T06:48:30.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:30 smithi028 ceph-mon[37984]: from='client.15783 v1:172.21.15.28:0/1395856957' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:48:30.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:30 smithi028 ceph-mon[37984]: 1.2 deep-scrub starts 2024-02-05T06:48:30.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:30 smithi028 ceph-mon[37984]: 1.2 deep-scrub ok 2024-02-05T06:48:30.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:30 smithi028 ceph-mon[37984]: 3.10 scrub starts 2024-02-05T06:48:30.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:30 smithi028 ceph-mon[37984]: 3.10 scrub ok 2024-02-05T06:48:30.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:30 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3587513508' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:48:30.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:30 smithi028 ceph-mon[37984]: 5.1d scrub starts 2024-02-05T06:48:30.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:30 smithi028 ceph-mon[37984]: 5.1d scrub ok 2024-02-05T06:48:30.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:30 smithi028 ceph-mon[37984]: 4.16 scrub starts 2024-02-05T06:48:30.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:30 smithi028 ceph-mon[37984]: 4.16 scrub ok 2024-02-05T06:48:30.483 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:30.486 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:30 smithi167 ceph-mon[32774]: pgmap v718: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-02-05T06:48:30.486 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:30 smithi167 ceph-mon[32774]: from='client.15783 v1:172.21.15.28:0/1395856957' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:48:30.486 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:30 smithi167 ceph-mon[32774]: 1.2 deep-scrub starts 2024-02-05T06:48:30.486 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:30 smithi167 ceph-mon[32774]: 1.2 deep-scrub ok 2024-02-05T06:48:30.486 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:30 smithi167 ceph-mon[32774]: 3.10 scrub starts 2024-02-05T06:48:30.486 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:30 smithi167 ceph-mon[32774]: 3.10 scrub ok 2024-02-05T06:48:30.486 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:30 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3587513508' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:48:30.487 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:30 smithi167 ceph-mon[32774]: 5.1d scrub starts 2024-02-05T06:48:30.487 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:30 smithi167 ceph-mon[32774]: 5.1d scrub ok 2024-02-05T06:48:30.487 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:30 smithi167 ceph-mon[32774]: 4.16 scrub starts 2024-02-05T06:48:30.487 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:30 smithi167 ceph-mon[32774]: 4.16 scrub ok 2024-02-05T06:48:30.520 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:30.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:30.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:30.622 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:30.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T06:48:30.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:30.658 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:30.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:30.760 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:30.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:30.795 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:30.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:30.896 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:30.919 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:48:30.950 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 51 16 84051 84035 102.952 75.5625 0.0230047 0.00970681 2024-02-05T06:48:30.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:30.952 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:30.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:31.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:31.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:31.098 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:31.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:31.199 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:31.245 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:31.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:31.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:31.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:31 smithi028 ceph-mon[29442]: 6.5 scrub starts 2024-02-05T06:48:31.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:31 smithi028 ceph-mon[29442]: 6.5 scrub ok 2024-02-05T06:48:31.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:31 smithi028 ceph-mon[29442]: osdmap e369: 8 total, 8 up, 8 in 2024-02-05T06:48:31.256 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:31 smithi028 ceph-mon[29442]: pgmap v720: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 881 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-02-05T06:48:31.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:31 smithi028 ceph-mon[37984]: 6.5 scrub starts 2024-02-05T06:48:31.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:31 smithi028 ceph-mon[37984]: 6.5 scrub ok 2024-02-05T06:48:31.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:31 smithi028 ceph-mon[37984]: osdmap e369: 8 total, 8 up, 8 in 2024-02-05T06:48:31.257 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:31 smithi028 ceph-mon[37984]: pgmap v720: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 881 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-02-05T06:48:31.347 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:31.385 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:31.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:31.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:31.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:31 smithi167 ceph-mon[32774]: 6.5 scrub starts 2024-02-05T06:48:31.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:31 smithi167 ceph-mon[32774]: 6.5 scrub ok 2024-02-05T06:48:31.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:31 smithi167 ceph-mon[32774]: osdmap e369: 8 total, 8 up, 8 in 2024-02-05T06:48:31.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:31 smithi167 ceph-mon[32774]: pgmap v720: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 881 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-02-05T06:48:31.488 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:31.524 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:31.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:31.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:31.626 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:31.659 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:31.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:31.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:31.683 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:48:31.761 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:31.771 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:48:31.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:31.793 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:31.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:31.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:31.932 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:31.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:31.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:31.950 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 52 16 85280 85264 102.449 76.8125 0.010115 0.00975645 2024-02-05T06:48:32.034 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:32.069 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:32.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:32.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:32.170 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:32.212 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:32.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:32.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:32.313 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:32.356 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:32.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:32.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:32.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:32 smithi028 ceph-mon[29442]: 3.5 scrub starts 2024-02-05T06:48:32.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:32 smithi028 ceph-mon[29442]: 3.5 scrub ok 2024-02-05T06:48:32.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:32 smithi028 ceph-mon[29442]: 4.13 scrub starts 2024-02-05T06:48:32.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:32 smithi028 ceph-mon[29442]: 4.13 scrub ok 2024-02-05T06:48:32.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:32 smithi028 ceph-mon[37984]: 3.5 scrub starts 2024-02-05T06:48:32.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:32 smithi028 ceph-mon[37984]: 3.5 scrub ok 2024-02-05T06:48:32.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:32 smithi028 ceph-mon[37984]: 4.13 scrub starts 2024-02-05T06:48:32.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:32 smithi028 ceph-mon[37984]: 4.13 scrub ok 2024-02-05T06:48:32.458 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:32.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:32 smithi167 ceph-mon[32774]: 3.5 scrub starts 2024-02-05T06:48:32.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:32 smithi167 ceph-mon[32774]: 3.5 scrub ok 2024-02-05T06:48:32.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:32 smithi167 ceph-mon[32774]: 4.13 scrub starts 2024-02-05T06:48:32.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:32 smithi167 ceph-mon[32774]: 4.13 scrub ok 2024-02-05T06:48:32.498 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:32.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:32.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:32.599 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:32.633 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:32.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:32.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:32.735 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:32.760 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:48:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:48:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:48:32.781 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:32.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:32.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:32.832 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:48:32.883 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:32.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:32.933 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:32.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:32.951 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 53 16 86493 86477 101.946 75.8125 0.0133064 0.00980391 2024-02-05T06:48:33.034 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:33.068 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:33.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:33.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:33.170 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:33.232 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:33.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:33.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:33.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:33 smithi028 ceph-mon[29442]: 3.1d scrub starts 2024-02-05T06:48:33.274 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:33 smithi028 ceph-mon[29442]: 3.1d scrub ok 2024-02-05T06:48:33.275 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:33 smithi028 ceph-mon[29442]: 4.e scrub starts 2024-02-05T06:48:33.275 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:33 smithi028 ceph-mon[29442]: 4.e scrub ok 2024-02-05T06:48:33.275 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:33 smithi028 ceph-mon[29442]: pgmap v721: 158 pgs: 158 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.4 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-02-05T06:48:33.275 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:33 smithi028 ceph-mon[37984]: 3.1d scrub starts 2024-02-05T06:48:33.275 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:33 smithi028 ceph-mon[37984]: 3.1d scrub ok 2024-02-05T06:48:33.275 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:33 smithi028 ceph-mon[37984]: 4.e scrub starts 2024-02-05T06:48:33.275 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:33 smithi028 ceph-mon[37984]: 4.e scrub ok 2024-02-05T06:48:33.275 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:33 smithi028 ceph-mon[37984]: pgmap v721: 158 pgs: 158 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.4 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-02-05T06:48:33.335 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:33.369 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:33.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:33.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:33.470 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:33.485 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:48:33 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:48:33.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:33 smithi167 ceph-mon[32774]: 3.1d scrub starts 2024-02-05T06:48:33.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:33 smithi167 ceph-mon[32774]: 3.1d scrub ok 2024-02-05T06:48:33.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:33 smithi167 ceph-mon[32774]: 4.e scrub starts 2024-02-05T06:48:33.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:33 smithi167 ceph-mon[32774]: 4.e scrub ok 2024-02-05T06:48:33.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:33 smithi167 ceph-mon[32774]: pgmap v721: 158 pgs: 158 active+clean; 5.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.4 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-02-05T06:48:33.507 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:33.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:33.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:33.609 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:33.645 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:33.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:33.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:33.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:33.784 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:33.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:33.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:33.886 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:33.936 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:33.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:33.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:33.951 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 54 16 87687 87671 101.44 74.625 0.0104965 0.00985341 2024-02-05T06:48:34.037 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:34.078 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:34.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:34.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:34.179 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:34.212 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:34.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:34.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:34.314 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:34.352 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:34.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:34.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:34.427 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:34 smithi028 ceph-mon[29442]: 5.12 scrub starts 2024-02-05T06:48:34.427 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:34 smithi028 ceph-mon[29442]: 5.12 scrub ok 2024-02-05T06:48:34.427 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:34 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:48:34.428 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:34 smithi028 ceph-mon[29442]: 6.0 scrub starts 2024-02-05T06:48:34.428 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:34 smithi028 ceph-mon[29442]: 6.0 scrub ok 2024-02-05T06:48:34.428 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:34 smithi028 ceph-mon[37984]: 5.12 scrub starts 2024-02-05T06:48:34.428 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:34 smithi028 ceph-mon[37984]: 5.12 scrub ok 2024-02-05T06:48:34.428 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:34 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:48:34.428 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:34 smithi028 ceph-mon[37984]: 6.0 scrub starts 2024-02-05T06:48:34.428 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:34 smithi028 ceph-mon[37984]: 6.0 scrub ok 2024-02-05T06:48:34.454 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:34.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:34 smithi167 ceph-mon[32774]: 5.12 scrub starts 2024-02-05T06:48:34.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:34 smithi167 ceph-mon[32774]: 5.12 scrub ok 2024-02-05T06:48:34.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:34 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:48:34.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:34 smithi167 ceph-mon[32774]: 6.0 scrub starts 2024-02-05T06:48:34.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:34 smithi167 ceph-mon[32774]: 6.0 scrub ok 2024-02-05T06:48:34.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:34.491 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:34.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:34.592 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:34.643 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:34.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:34.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:34.744 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:34.777 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:34.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:34.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:34.856 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:48:34.856 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:48:34.879 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:34.913 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:34.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:34.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:34.951 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 55 16 88871 88855 100.941 74 0.00977301 0.00990145 2024-02-05T06:48:35.014 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:35.048 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:35.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:35.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:35.150 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:35.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:35.186 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:35.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:35.287 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:35.364 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:35.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:35.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:35.465 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:35.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:35 smithi167 ceph-mon[32774]: 5.11 deep-scrub starts 2024-02-05T06:48:35.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:35 smithi167 ceph-mon[32774]: 5.11 deep-scrub ok 2024-02-05T06:48:35.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:35 smithi167 ceph-mon[32774]: 6.4 scrub starts 2024-02-05T06:48:35.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:35 smithi167 ceph-mon[32774]: 6.4 scrub ok 2024-02-05T06:48:35.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:35 smithi167 ceph-mon[32774]: pgmap v722: 158 pgs: 158 active+clean; 5.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-02-05T06:48:35.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:35 smithi167 ceph-mon[32774]: 6.d deep-scrub starts 2024-02-05T06:48:35.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:35 smithi167 ceph-mon[32774]: 5.10 deep-scrub starts 2024-02-05T06:48:35.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:35 smithi167 ceph-mon[32774]: 5.10 deep-scrub ok 2024-02-05T06:48:35.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:35 smithi028 ceph-mon[37984]: 5.11 deep-scrub starts 2024-02-05T06:48:35.492 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:35 smithi028 ceph-mon[37984]: 5.11 deep-scrub ok 2024-02-05T06:48:35.493 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:35 smithi028 ceph-mon[37984]: 6.4 scrub starts 2024-02-05T06:48:35.493 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:35 smithi028 ceph-mon[37984]: 6.4 scrub ok 2024-02-05T06:48:35.493 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:35 smithi028 ceph-mon[37984]: pgmap v722: 158 pgs: 158 active+clean; 5.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-02-05T06:48:35.493 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:35 smithi028 ceph-mon[37984]: 6.d deep-scrub starts 2024-02-05T06:48:35.493 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:35 smithi028 ceph-mon[37984]: 5.10 deep-scrub starts 2024-02-05T06:48:35.493 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:35 smithi028 ceph-mon[37984]: 5.10 deep-scrub ok 2024-02-05T06:48:35.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:35 smithi028 ceph-mon[29442]: 5.11 deep-scrub starts 2024-02-05T06:48:35.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:35 smithi028 ceph-mon[29442]: 5.11 deep-scrub ok 2024-02-05T06:48:35.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:35 smithi028 ceph-mon[29442]: 6.4 scrub starts 2024-02-05T06:48:35.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:35 smithi028 ceph-mon[29442]: 6.4 scrub ok 2024-02-05T06:48:35.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:35 smithi028 ceph-mon[29442]: pgmap v722: 158 pgs: 158 active+clean; 5.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-02-05T06:48:35.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:35 smithi028 ceph-mon[29442]: 6.d deep-scrub starts 2024-02-05T06:48:35.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:35 smithi028 ceph-mon[29442]: 5.10 deep-scrub starts 2024-02-05T06:48:35.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:35 smithi028 ceph-mon[29442]: 5.10 deep-scrub ok 2024-02-05T06:48:35.508 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:35.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:35.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:35.610 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:35.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:35.646 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:35.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:35.747 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:35.783 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:35.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:35.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:35.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-05T06:48:35.885 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:35.919 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:35.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:35.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:35.951 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 56 16 90035 90019 100.438 72.75 0.00752593 0.00995028 2024-02-05T06:48:36.020 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:36.055 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:36.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:36.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:36.110 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:48:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-05T06:48:36.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:36.208 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:36.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:36.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:36.310 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:36.348 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:36.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:36.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:36.450 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:36.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:36 smithi028 ceph-mon[29442]: 4.c scrub starts 2024-02-05T06:48:36.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:36 smithi028 ceph-mon[29442]: 4.c scrub ok 2024-02-05T06:48:36.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:36 smithi028 ceph-mon[29442]: 6.d deep-scrub ok 2024-02-05T06:48:36.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:36 smithi028 ceph-mon[29442]: 3.17 scrub starts 2024-02-05T06:48:36.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:36 smithi028 ceph-mon[29442]: 3.17 scrub ok 2024-02-05T06:48:36.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:36 smithi028 ceph-mon[37984]: 4.c scrub starts 2024-02-05T06:48:36.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:36 smithi028 ceph-mon[37984]: 4.c scrub ok 2024-02-05T06:48:36.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:36 smithi028 ceph-mon[37984]: 6.d deep-scrub ok 2024-02-05T06:48:36.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:36 smithi028 ceph-mon[37984]: 3.17 scrub starts 2024-02-05T06:48:36.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:36 smithi028 ceph-mon[37984]: 3.17 scrub ok 2024-02-05T06:48:36.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:36 smithi167 ceph-mon[32774]: 4.c scrub starts 2024-02-05T06:48:36.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:36 smithi167 ceph-mon[32774]: 4.c scrub ok 2024-02-05T06:48:36.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:36 smithi167 ceph-mon[32774]: 6.d deep-scrub ok 2024-02-05T06:48:36.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:36 smithi167 ceph-mon[32774]: 3.17 scrub starts 2024-02-05T06:48:36.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:36 smithi167 ceph-mon[32774]: 3.17 scrub ok 2024-02-05T06:48:36.496 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:36.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:36.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:36.598 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:36.633 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:36.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:36.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:36.735 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:36.772 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:36.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:36.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:36.874 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:36.906 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:36.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:36.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:36.953 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 57 16 91172 91156 99.9189 71.0625 0.0265768 0.0100022 2024-02-05T06:48:37.008 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:37.055 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:37.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:37.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:37.157 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:37.193 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:37.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:37.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:37.295 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:37.335 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:37.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:37.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:37.437 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:37.475 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:37.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:37.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:37.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:37.610 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:37.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:37.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:37.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:37 smithi028 ceph-mon[29442]: 5.15 scrub starts 2024-02-05T06:48:37.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:37 smithi028 ceph-mon[29442]: 5.15 scrub ok 2024-02-05T06:48:37.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:37 smithi028 ceph-mon[29442]: from='client.45686 v1:172.21.15.28:0/1217521066' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:48:37.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:37 smithi028 ceph-mon[29442]: pgmap v723: 158 pgs: 158 active+clean; 5.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 605 B/s rd, 45 MiB/s wr, 714 op/s 2024-02-05T06:48:37.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:37 smithi028 ceph-mon[37984]: 5.15 scrub starts 2024-02-05T06:48:37.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:37 smithi028 ceph-mon[37984]: 5.15 scrub ok 2024-02-05T06:48:37.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:37 smithi028 ceph-mon[37984]: from='client.45686 v1:172.21.15.28:0/1217521066' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:48:37.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:37 smithi028 ceph-mon[37984]: pgmap v723: 158 pgs: 158 active+clean; 5.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 605 B/s rd, 45 MiB/s wr, 714 op/s 2024-02-05T06:48:37.712 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:37.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:37 smithi167 ceph-mon[32774]: 5.15 scrub starts 2024-02-05T06:48:37.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:37 smithi167 ceph-mon[32774]: 5.15 scrub ok 2024-02-05T06:48:37.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:37 smithi167 ceph-mon[32774]: from='client.45686 v1:172.21.15.28:0/1217521066' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:48:37.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:37 smithi167 ceph-mon[32774]: pgmap v723: 158 pgs: 158 active+clean; 5.2 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 605 B/s rd, 45 MiB/s wr, 714 op/s 2024-02-05T06:48:37.740 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:48:37.742 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:37.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:37.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:37.765 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":722,"stamp":"2024-02-05T06:48:34.468156+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5573974364,"num_objects":85241,"num_object_clones":0,"num_object_copies":255723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85241,"num_whiteouts":0,"num_read":1610,"num_read_kb":1362,"num_write":85765,"num_write_kb":5442797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":415,"num_bytes_recovered":1586799,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86237,"ondisk_log_size":86237,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":20468492,"kb_used_data":16273532,"kb_used_omap":12,"kb_used_meta":4194867,"kb_avail":729492724,"statfs":{"total":767960285184,"available":747000549376,"internally_reserved":0,"allocated":16664096768,"data_stored":16659881988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":4295544369},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":796393472,"num_objects":12152,"num_object_clones":0,"num_object_copies":36456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12152,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12152,"num_write_kb":777728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003594"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":415,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.203643+0000","last_change":"2024-02-05T06:47:44.624372+0000","last_active":"2024-02-05T06:48:30.203643+0000","last_peered":"2024-02-05T06:48:30.203643+0000","last_clean":"2024-02-05T06:48:30.203643+0000","last_became_active":"2024-02-05T06:45:09.787134+0000","last_became_peered":"2024-02-05T06:45:09.787134+0000","last_unstale":"2024-02-05T06:48:30.203643+0000","last_undegraded":"2024-02-05T06:48:30.203643+0000","last_fullsized":"2024-02-05T06:48:30.203643+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:44.624254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:47:44.624254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:58.304608+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":686,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.186439+0000","last_change":"2024-02-05T06:48:19.036192+0000","last_active":"2024-02-05T06:48:30.186439+0000","last_peered":"2024-02-05T06:48:30.186439+0000","last_clean":"2024-02-05T06:48:30.186439+0000","last_became_active":"2024-02-05T06:41:43.420342+0000","last_became_peered":"2024-02-05T06:41:43.420342+0000","last_unstale":"2024-02-05T06:48:30.186439+0000","last_undegraded":"2024-02-05T06:48:30.186439+0000","last_fullsized":"2024-02-05T06:48:30.186439+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:19.036048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:06.525012+0000","last_clean_scrub_stamp":"2024-02-05T06:48:19.036048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:27.993992+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":676,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.188329+0000","last_change":"2024-02-05T06:48:20.602707+0000","last_active":"2024-02-05T06:48:30.188329+0000","last_peered":"2024-02-05T06:48:30.188329+0000","last_clean":"2024-02-05T06:48:30.188329+0000","last_became_active":"2024-02-05T06:41:43.421315+0000","last_became_peered":"2024-02-05T06:41:43.421315+0000","last_unstale":"2024-02-05T06:48:30.188329+0000","last_undegraded":"2024-02-05T06:48:30.188329+0000","last_fullsized":"2024-02-05T06:48:30.188329+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:20.602591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:48:20.602591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:41.700957+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"295'19","reported_seq":586,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.188672+0000","last_change":"2024-02-05T06:48:19.282787+0000","last_active":"2024-02-05T06:48:30.188672+0000","last_peered":"2024-02-05T06:48:30.188672+0000","last_clean":"2024-02-05T06:48:30.188672+0000","last_became_active":"2024-02-05T06:36:54.959330+0000","last_became_peered":"2024-02-05T06:36:54.959330+0000","last_unstale":"2024-02-05T06:48:30.188672+0000","last_undegraded":"2024-02-05T06:48:30.188672+0000","last_fullsized":"2024-02-05T06:48:30.188672+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'19","last_scrub_stamp":"2024-02-05T06:48:19.282672+0000","last_deep_scrub":"247'13","last_deep_scrub_stamp":"2024-02-05T06:41:42.473424+0000","last_clean_scrub_stamp":"2024-02-05T06:48:19.282672+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:41.837304+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":653,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.193032+0000","last_change":"2024-02-05T06:45:45.252029+0000","last_active":"2024-02-05T06:48:30.193032+0000","last_peered":"2024-02-05T06:48:30.193032+0000","last_clean":"2024-02-05T06:48:30.193032+0000","last_became_active":"2024-02-05T06:40:27.077485+0000","last_became_peered":"2024-02-05T06:40:27.077485+0000","last_unstale":"2024-02-05T06:48:30.193032+0000","last_undegraded":"2024-02-05T06:48:30.193032+0000","last_fullsized":"2024-02-05T06:48:30.193032+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:45.251895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:45:45.251895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":184,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.195191+0000","last_change":"2024-02-05T06:48:18.702091+0000","last_active":"2024-02-05T06:48:30.195191+0000","last_peered":"2024-02-05T06:48:30.195191+0000","last_clean":"2024-02-05T06:48:30.195191+0000","last_became_active":"2024-02-05T06:45:09.775037+0000","last_became_peered":"2024-02-05T06:45:09.775037+0000","last_unstale":"2024-02-05T06:48:30.195191+0000","last_undegraded":"2024-02-05T06:48:30.195191+0000","last_fullsized":"2024-02-05T06:48:30.195191+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:18.701981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:51.388156+0000","last_clean_scrub_stamp":"2024-02-05T06:48:18.701981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:33.942331+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":663,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.193160+0000","last_change":"2024-02-05T06:47:34.440851+0000","last_active":"2024-02-05T06:48:30.193160+0000","last_peered":"2024-02-05T06:48:30.193160+0000","last_clean":"2024-02-05T06:48:30.193160+0000","last_became_active":"2024-02-05T06:33:29.553864+0000","last_became_peered":"2024-02-05T06:33:29.553864+0000","last_unstale":"2024-02-05T06:48:30.193160+0000","last_undegraded":"2024-02-05T06:48:30.193160+0000","last_fullsized":"2024-02-05T06:48:30.193160+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:34.440773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:47:34.440773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:40.674347+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"295'34","reported_seq":708,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.193928+0000","last_change":"2024-02-05T06:47:48.407121+0000","last_active":"2024-02-05T06:48:30.193928+0000","last_peered":"2024-02-05T06:48:30.193928+0000","last_clean":"2024-02-05T06:48:30.193928+0000","last_became_active":"2024-02-05T06:33:31.572544+0000","last_became_peered":"2024-02-05T06:33:31.572544+0000","last_unstale":"2024-02-05T06:48:30.193928+0000","last_undegraded":"2024-02-05T06:48:30.193928+0000","last_fullsized":"2024-02-05T06:48:30.193928+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'34","last_scrub_stamp":"2024-02-05T06:47:48.406976+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:39:35.433539+0000","last_clean_scrub_stamp":"2024-02-05T06:47:48.406976+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:12.788812+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":663,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.188353+0000","last_change":"2024-02-05T06:46:58.507850+0000","last_active":"2024-02-05T06:48:30.188353+0000","last_peered":"2024-02-05T06:48:30.188353+0000","last_clean":"2024-02-05T06:48:30.188353+0000","last_became_active":"2024-02-05T06:40:27.068888+0000","last_became_peered":"2024-02-05T06:40:27.068888+0000","last_unstale":"2024-02-05T06:48:30.188353+0000","last_undegraded":"2024-02-05T06:48:30.188353+0000","last_fullsized":"2024-02-05T06:48:30.188353+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:58.507407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:44:55.125921+0000","last_clean_scrub_stamp":"2024-02-05T06:46:58.507407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":219,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.193119+0000","last_change":"2024-02-05T06:45:51.218029+0000","last_active":"2024-02-05T06:48:30.193119+0000","last_peered":"2024-02-05T06:48:30.193119+0000","last_clean":"2024-02-05T06:48:30.193119+0000","last_became_active":"2024-02-05T06:42:33.129959+0000","last_became_peered":"2024-02-05T06:42:33.129959+0000","last_unstale":"2024-02-05T06:48:30.193119+0000","last_undegraded":"2024-02-05T06:48:30.193119+0000","last_fullsized":"2024-02-05T06:48:30.193119+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:51.217900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:45:51.217900+0000","last_clean_scrub_stamp":"2024-02-05T06:45:51.217900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":278,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:31.562342+0000","last_change":"2024-02-05T06:48:31.562380+0000","last_active":"2024-02-05T06:48:31.562342+0000","last_peered":"2024-02-05T06:48:31.562342+0000","last_clean":"2024-02-05T06:48:31.562342+0000","last_became_active":"2024-02-05T06:41:43.403746+0000","last_became_peered":"2024-02-05T06:41:43.403746+0000","last_unstale":"2024-02-05T06:48:31.562342+0000","last_undegraded":"2024-02-05T06:48:31.562342+0000","last_fullsized":"2024-02-05T06:48:31.562342+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:31.562268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:44.081318+0000","last_clean_scrub_stamp":"2024-02-05T06:48:31.562268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:46.410622+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"295'19","reported_seq":519,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.192693+0000","last_change":"2024-02-05T06:48:17.662641+0000","last_active":"2024-02-05T06:48:30.192693+0000","last_peered":"2024-02-05T06:48:30.192693+0000","last_clean":"2024-02-05T06:48:30.192693+0000","last_became_active":"2024-02-05T06:45:09.787626+0000","last_became_peered":"2024-02-05T06:45:09.787626+0000","last_unstale":"2024-02-05T06:48:30.192693+0000","last_undegraded":"2024-02-05T06:48:30.192693+0000","last_fullsized":"2024-02-05T06:48:30.192693+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'19","last_scrub_stamp":"2024-02-05T06:48:17.662510+0000","last_deep_scrub":"92'13","last_deep_scrub_stamp":"2024-02-05T06:34:46.542893+0000","last_clean_scrub_stamp":"2024-02-05T06:48:17.662510+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:32.649577+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":278,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.202925+0000","last_change":"2024-02-05T06:48:23.678385+0000","last_active":"2024-02-05T06:48:30.202925+0000","last_peered":"2024-02-05T06:48:30.202925+0000","last_clean":"2024-02-05T06:48:30.202925+0000","last_became_active":"2024-02-05T06:41:43.407404+0000","last_became_peered":"2024-02-05T06:41:43.407404+0000","last_unstale":"2024-02-05T06:48:30.202925+0000","last_undegraded":"2024-02-05T06:48:30.202925+0000","last_fullsized":"2024-02-05T06:48:30.202925+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:23.678255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:21.998850+0000","last_clean_scrub_stamp":"2024-02-05T06:48:23.678255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:35.610704+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":371,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.189868+0000","last_change":"2024-02-05T06:47:51.640890+0000","last_active":"2024-02-05T06:48:30.189868+0000","last_peered":"2024-02-05T06:48:30.189868+0000","last_clean":"2024-02-05T06:48:30.189868+0000","last_became_active":"2024-02-05T06:40:27.071758+0000","last_became_peered":"2024-02-05T06:40:27.071758+0000","last_unstale":"2024-02-05T06:48:30.189868+0000","last_undegraded":"2024-02-05T06:48:30.189868+0000","last_fullsized":"2024-02-05T06:48:30.189868+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:51.640785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:47:51.640785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:06.140529+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":278,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.204120+0000","last_change":"2024-02-05T06:47:39.612949+0000","last_active":"2024-02-05T06:48:30.204120+0000","last_peered":"2024-02-05T06:48:30.204120+0000","last_clean":"2024-02-05T06:48:30.204120+0000","last_became_active":"2024-02-05T06:41:43.408428+0000","last_became_peered":"2024-02-05T06:41:43.408428+0000","last_unstale":"2024-02-05T06:48:30.204120+0000","last_undegraded":"2024-02-05T06:48:30.204120+0000","last_fullsized":"2024-02-05T06:48:30.204120+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:39.612865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:50.055245+0000","last_clean_scrub_stamp":"2024-02-05T06:47:39.612865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:46.400154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"295'7","reported_seq":501,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.192785+0000","last_change":"2024-02-05T06:45:09.809100+0000","last_active":"2024-02-05T06:48:30.192785+0000","last_peered":"2024-02-05T06:48:30.192785+0000","last_clean":"2024-02-05T06:48:30.192785+0000","last_became_active":"2024-02-05T06:45:09.808663+0000","last_became_peered":"2024-02-05T06:45:09.808663+0000","last_unstale":"2024-02-05T06:48:30.192785+0000","last_undegraded":"2024-02-05T06:48:30.192785+0000","last_fullsized":"2024-02-05T06:48:30.192785+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'7","last_scrub_stamp":"2024-02-05T06:44:55.988817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:44:55.988817+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":8,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":419,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.194993+0000","last_change":"2024-02-05T06:48:15.743595+0000","last_active":"2024-02-05T06:48:30.194993+0000","last_peered":"2024-02-05T06:48:30.194993+0000","last_clean":"2024-02-05T06:48:30.194993+0000","last_became_active":"2024-02-05T06:45:09.770902+0000","last_became_peered":"2024-02-05T06:45:09.770902+0000","last_unstale":"2024-02-05T06:48:30.194993+0000","last_undegraded":"2024-02-05T06:48:30.194993+0000","last_fullsized":"2024-02-05T06:48:30.194993+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:15.743307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:48:15.743307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:32.352762+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":673,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.191360+0000","last_change":"2024-02-05T06:48:29.124890+0000","last_active":"2024-02-05T06:48:30.191360+0000","last_peered":"2024-02-05T06:48:30.191360+0000","last_clean":"2024-02-05T06:48:30.191360+0000","last_became_active":"2024-02-05T06:36:54.958069+0000","last_became_peered":"2024-02-05T06:36:54.958069+0000","last_unstale":"2024-02-05T06:48:30.191360+0000","last_undegraded":"2024-02-05T06:48:30.191360+0000","last_fullsized":"2024-02-05T06:48:30.191360+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:29.124778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:48:29.124778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:48.986025+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"281'2","reported_seq":710,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.189631+0000","last_change":"2024-02-05T06:47:50.442268+0000","last_active":"2024-02-05T06:48:30.189631+0000","last_peered":"2024-02-05T06:48:30.189631+0000","last_clean":"2024-02-05T06:48:30.189631+0000","last_became_active":"2024-02-05T06:40:27.073360+0000","last_became_peered":"2024-02-05T06:40:27.073360+0000","last_unstale":"2024-02-05T06:48:30.189631+0000","last_undegraded":"2024-02-05T06:48:30.189631+0000","last_fullsized":"2024-02-05T06:48:30.189631+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'2","last_scrub_stamp":"2024-02-05T06:47:50.442159+0000","last_deep_scrub":"281'2","last_deep_scrub_stamp":"2024-02-05T06:47:50.442159+0000","last_clean_scrub_stamp":"2024-02-05T06:47:50.442159+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:16.471755+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"295'34","reported_seq":737,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.196195+0000","last_change":"2024-02-05T06:47:43.040717+0000","last_active":"2024-02-05T06:48:30.196195+0000","last_peered":"2024-02-05T06:48:30.196195+0000","last_clean":"2024-02-05T06:48:30.196195+0000","last_became_active":"2024-02-05T06:33:31.569763+0000","last_became_peered":"2024-02-05T06:33:31.569763+0000","last_unstale":"2024-02-05T06:48:30.196195+0000","last_undegraded":"2024-02-05T06:48:30.196195+0000","last_fullsized":"2024-02-05T06:48:30.196195+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'34","last_scrub_stamp":"2024-02-05T06:47:43.040574+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:34:35.234636+0000","last_clean_scrub_stamp":"2024-02-05T06:47:43.040574+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:49.320938+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":655,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.203305+0000","last_change":"2024-02-05T06:46:54.584153+0000","last_active":"2024-02-05T06:48:30.203305+0000","last_peered":"2024-02-05T06:48:30.203305+0000","last_clean":"2024-02-05T06:48:30.203305+0000","last_became_active":"2024-02-05T06:41:43.411442+0000","last_became_peered":"2024-02-05T06:41:43.411442+0000","last_unstale":"2024-02-05T06:48:30.203305+0000","last_undegraded":"2024-02-05T06:48:30.203305+0000","last_fullsized":"2024-02-05T06:48:30.203305+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:54.584073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:46:54.584073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":451,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.193372+0000","last_change":"2024-02-05T06:48:13.677951+0000","last_active":"2024-02-05T06:48:30.193372+0000","last_peered":"2024-02-05T06:48:30.193372+0000","last_clean":"2024-02-05T06:48:30.193372+0000","last_became_active":"2024-02-05T06:39:13.248618+0000","last_became_peered":"2024-02-05T06:39:13.248618+0000","last_unstale":"2024-02-05T06:48:30.193372+0000","last_undegraded":"2024-02-05T06:48:30.193372+0000","last_fullsized":"2024-02-05T06:48:30.193372+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:13.677795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:42.381636+0000","last_clean_scrub_stamp":"2024-02-05T06:48:13.677795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:41.284546+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":192,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.190167+0000","last_change":"2024-02-05T06:48:15.138990+0000","last_active":"2024-02-05T06:48:30.190167+0000","last_peered":"2024-02-05T06:48:30.190167+0000","last_clean":"2024-02-05T06:48:30.190167+0000","last_became_active":"2024-02-05T06:45:09.786579+0000","last_became_peered":"2024-02-05T06:45:09.786579+0000","last_unstale":"2024-02-05T06:48:30.190167+0000","last_undegraded":"2024-02-05T06:48:30.190167+0000","last_fullsized":"2024-02-05T06:48:30.190167+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:15.138923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:48:15.138923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:40.202990+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"295'18","reported_seq":744,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.200680+0000","last_change":"2024-02-05T06:48:24.628198+0000","last_active":"2024-02-05T06:48:30.200680+0000","last_peered":"2024-02-05T06:48:30.200680+0000","last_clean":"2024-02-05T06:48:30.200680+0000","last_became_active":"2024-02-05T06:36:54.954880+0000","last_became_peered":"2024-02-05T06:36:54.954880+0000","last_unstale":"2024-02-05T06:48:30.200680+0000","last_undegraded":"2024-02-05T06:48:30.200680+0000","last_fullsized":"2024-02-05T06:48:30.200680+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'18","last_scrub_stamp":"2024-02-05T06:48:24.627686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:48:24.627686+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:35.142648+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"325'2","reported_seq":378,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.190594+0000","last_change":"2024-02-05T06:48:20.648253+0000","last_active":"2024-02-05T06:48:30.190594+0000","last_peered":"2024-02-05T06:48:30.190594+0000","last_clean":"2024-02-05T06:48:30.190594+0000","last_became_active":"2024-02-05T06:41:43.418640+0000","last_became_peered":"2024-02-05T06:41:43.418640+0000","last_unstale":"2024-02-05T06:48:30.190594+0000","last_undegraded":"2024-02-05T06:48:30.190594+0000","last_fullsized":"2024-02-05T06:48:30.190594+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'2","last_scrub_stamp":"2024-02-05T06:48:20.648179+0000","last_deep_scrub":"325'2","last_deep_scrub_stamp":"2024-02-05T06:45:46.242415+0000","last_clean_scrub_stamp":"2024-02-05T06:48:20.648179+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:36.679419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"314'13","reported_seq":916,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.198194+0000","last_change":"2024-02-05T06:47:42.202376+0000","last_active":"2024-02-05T06:48:30.198194+0000","last_peered":"2024-02-05T06:48:30.198194+0000","last_clean":"2024-02-05T06:48:30.198194+0000","last_became_active":"2024-02-05T06:40:27.063399+0000","last_became_peered":"2024-02-05T06:40:27.063399+0000","last_unstale":"2024-02-05T06:48:30.198194+0000","last_undegraded":"2024-02-05T06:48:30.198194+0000","last_fullsized":"2024-02-05T06:48:30.198194+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"314'13","last_scrub_stamp":"2024-02-05T06:47:42.202237+0000","last_deep_scrub":"314'13","last_deep_scrub_stamp":"2024-02-05T06:44:53.670923+0000","last_clean_scrub_stamp":"2024-02-05T06:47:42.202237+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:46.335289+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":699,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.186340+0000","last_change":"2024-02-05T06:47:49.989990+0000","last_active":"2024-02-05T06:48:30.186340+0000","last_peered":"2024-02-05T06:48:30.186340+0000","last_clean":"2024-02-05T06:48:30.186340+0000","last_became_active":"2024-02-05T06:36:54.958414+0000","last_became_peered":"2024-02-05T06:36:54.958414+0000","last_unstale":"2024-02-05T06:48:30.186340+0000","last_undegraded":"2024-02-05T06:48:30.186340+0000","last_fullsized":"2024-02-05T06:48:30.186340+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:47:49.989863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:47:49.989863+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:14.685287+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"295'25","reported_seq":739,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.189119+0000","last_change":"2024-02-05T06:48:17.606222+0000","last_active":"2024-02-05T06:48:30.189119+0000","last_peered":"2024-02-05T06:48:30.189119+0000","last_clean":"2024-02-05T06:48:30.189119+0000","last_became_active":"2024-02-05T06:40:27.100610+0000","last_became_peered":"2024-02-05T06:40:27.100610+0000","last_unstale":"2024-02-05T06:48:30.189119+0000","last_undegraded":"2024-02-05T06:48:30.189119+0000","last_fullsized":"2024-02-05T06:48:30.189119+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'25","last_scrub_stamp":"2024-02-05T06:48:17.605995+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-02-05T06:34:50.718768+0000","last_clean_scrub_stamp":"2024-02-05T06:48:17.605995+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:38.852335+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":681,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.191491+0000","last_change":"2024-02-05T06:46:45.866461+0000","last_active":"2024-02-05T06:48:30.191491+0000","last_peered":"2024-02-05T06:48:30.191491+0000","last_clean":"2024-02-05T06:48:30.191491+0000","last_became_active":"2024-02-05T06:41:43.414638+0000","last_became_peered":"2024-02-05T06:41:43.414638+0000","last_unstale":"2024-02-05T06:48:30.191491+0000","last_undegraded":"2024-02-05T06:48:30.191491+0000","last_fullsized":"2024-02-05T06:48:30.191491+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:45.866348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:46:45.866348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":570,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.193011+0000","last_change":"2024-02-05T06:46:36.444754+0000","last_active":"2024-02-05T06:48:30.193011+0000","last_peered":"2024-02-05T06:48:30.193011+0000","last_clean":"2024-02-05T06:48:30.193011+0000","last_became_active":"2024-02-05T06:40:27.097169+0000","last_became_peered":"2024-02-05T06:40:27.097169+0000","last_unstale":"2024-02-05T06:48:30.193011+0000","last_undegraded":"2024-02-05T06:48:30.193011+0000","last_fullsized":"2024-02-05T06:48:30.193011+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:36.444642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:08.713894+0000","last_clean_scrub_stamp":"2024-02-05T06:46:36.444642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":278,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.202650+0000","last_change":"2024-02-05T06:48:27.651205+0000","last_active":"2024-02-05T06:48:30.202650+0000","last_peered":"2024-02-05T06:48:30.202650+0000","last_clean":"2024-02-05T06:48:30.202650+0000","last_became_active":"2024-02-05T06:41:43.411043+0000","last_became_peered":"2024-02-05T06:41:43.411043+0000","last_unstale":"2024-02-05T06:48:30.202650+0000","last_undegraded":"2024-02-05T06:48:30.202650+0000","last_fullsized":"2024-02-05T06:48:30.202650+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:27.651102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:48:27.651102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:35.027113+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"295'24","reported_seq":757,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.198311+0000","last_change":"2024-02-05T06:47:37.226174+0000","last_active":"2024-02-05T06:48:30.198311+0000","last_peered":"2024-02-05T06:48:30.198311+0000","last_clean":"2024-02-05T06:48:30.198311+0000","last_became_active":"2024-02-05T06:41:43.419769+0000","last_became_peered":"2024-02-05T06:41:43.419769+0000","last_unstale":"2024-02-05T06:48:30.198311+0000","last_undegraded":"2024-02-05T06:48:30.198311+0000","last_fullsized":"2024-02-05T06:48:30.198311+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:47:37.226071+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:43:03.221466+0000","last_clean_scrub_stamp":"2024-02-05T06:47:37.226071+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:40.560472+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":377,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.190048+0000","last_change":"2024-02-05T06:48:23.748347+0000","last_active":"2024-02-05T06:48:30.190048+0000","last_peered":"2024-02-05T06:48:30.190048+0000","last_clean":"2024-02-05T06:48:30.190048+0000","last_became_active":"2024-02-05T06:44:54.625067+0000","last_became_peered":"2024-02-05T06:44:54.625067+0000","last_unstale":"2024-02-05T06:48:30.190048+0000","last_undegraded":"2024-02-05T06:48:30.190048+0000","last_fullsized":"2024-02-05T06:48:30.190048+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:23.748226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:48.512673+0000","last_clean_scrub_stamp":"2024-02-05T06:48:23.748226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:44.753039+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":701,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.193201+0000","last_change":"2024-02-05T06:46:54.260053+0000","last_active":"2024-02-05T06:48:30.193201+0000","last_peered":"2024-02-05T06:48:30.193201+0000","last_clean":"2024-02-05T06:48:30.193201+0000","last_became_active":"2024-02-05T06:40:27.060112+0000","last_became_peered":"2024-02-05T06:40:27.060112+0000","last_unstale":"2024-02-05T06:48:30.193201+0000","last_undegraded":"2024-02-05T06:48:30.193201+0000","last_fullsized":"2024-02-05T06:48:30.193201+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:54.259661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:46:54.259661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":667,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.188511+0000","last_change":"2024-02-05T06:46:31.409743+0000","last_active":"2024-02-05T06:48:30.188511+0000","last_peered":"2024-02-05T06:48:30.188511+0000","last_clean":"2024-02-05T06:48:30.188511+0000","last_became_active":"2024-02-05T06:41:43.424459+0000","last_became_peered":"2024-02-05T06:41:43.424459+0000","last_unstale":"2024-02-05T06:48:30.188511+0000","last_undegraded":"2024-02-05T06:48:30.188511+0000","last_fullsized":"2024-02-05T06:48:30.188511+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:31.409628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:49.117949+0000","last_clean_scrub_stamp":"2024-02-05T06:46:31.409628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"295'8","reported_seq":662,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.192294+0000","last_change":"2024-02-05T06:47:41.444191+0000","last_active":"2024-02-05T06:48:30.192294+0000","last_peered":"2024-02-05T06:48:30.192294+0000","last_clean":"2024-02-05T06:48:30.192294+0000","last_became_active":"2024-02-05T06:33:31.573410+0000","last_became_peered":"2024-02-05T06:33:31.573410+0000","last_unstale":"2024-02-05T06:48:30.192294+0000","last_undegraded":"2024-02-05T06:48:30.192294+0000","last_fullsized":"2024-02-05T06:48:30.192294+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'8","last_scrub_stamp":"2024-02-05T06:47:41.444069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:47:41.444069+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:54.176721+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":428,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.188398+0000","last_change":"2024-02-05T06:46:35.933464+0000","last_active":"2024-02-05T06:48:30.188398+0000","last_peered":"2024-02-05T06:48:30.188398+0000","last_clean":"2024-02-05T06:48:30.188398+0000","last_became_active":"2024-02-05T06:39:13.245684+0000","last_became_peered":"2024-02-05T06:39:13.245684+0000","last_unstale":"2024-02-05T06:48:30.188398+0000","last_undegraded":"2024-02-05T06:48:30.188398+0000","last_fullsized":"2024-02-05T06:48:30.188398+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:35.933354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:46:35.933354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":363,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.189404+0000","last_change":"2024-02-05T06:46:41.612056+0000","last_active":"2024-02-05T06:48:30.189404+0000","last_peered":"2024-02-05T06:48:30.189404+0000","last_clean":"2024-02-05T06:48:30.189404+0000","last_became_active":"2024-02-05T06:40:27.062975+0000","last_became_peered":"2024-02-05T06:40:27.062975+0000","last_unstale":"2024-02-05T06:48:30.189404+0000","last_undegraded":"2024-02-05T06:48:30.189404+0000","last_fullsized":"2024-02-05T06:48:30.189404+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:41.611955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:46:41.611955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":278,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.205360+0000","last_change":"2024-02-05T06:46:51.634330+0000","last_active":"2024-02-05T06:48:30.205360+0000","last_peered":"2024-02-05T06:48:30.205360+0000","last_clean":"2024-02-05T06:48:30.205360+0000","last_became_active":"2024-02-05T06:41:43.412654+0000","last_became_peered":"2024-02-05T06:41:43.412654+0000","last_unstale":"2024-02-05T06:48:30.205360+0000","last_undegraded":"2024-02-05T06:48:30.205360+0000","last_fullsized":"2024-02-05T06:48:30.205360+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:51.634250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:46:51.634250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"295'24","reported_seq":740,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.193838+0000","last_change":"2024-02-05T06:47:30.341586+0000","last_active":"2024-02-05T06:48:30.193838+0000","last_peered":"2024-02-05T06:48:30.193838+0000","last_clean":"2024-02-05T06:48:30.193838+0000","last_became_active":"2024-02-05T06:40:27.080906+0000","last_became_peered":"2024-02-05T06:40:27.080906+0000","last_unstale":"2024-02-05T06:48:30.193838+0000","last_undegraded":"2024-02-05T06:48:30.193838+0000","last_fullsized":"2024-02-05T06:48:30.193838+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:47:30.341433+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-02-05T06:35:22.938675+0000","last_clean_scrub_stamp":"2024-02-05T06:47:30.341433+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:51.439941+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"336'2","reported_seq":673,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.187971+0000","last_change":"2024-02-05T06:46:53.500514+0000","last_active":"2024-02-05T06:48:30.187971+0000","last_peered":"2024-02-05T06:48:30.187971+0000","last_clean":"2024-02-05T06:48:30.187971+0000","last_became_active":"2024-02-05T06:41:43.409761+0000","last_became_peered":"2024-02-05T06:41:43.409761+0000","last_unstale":"2024-02-05T06:48:30.187971+0000","last_undegraded":"2024-02-05T06:48:30.187971+0000","last_fullsized":"2024-02-05T06:48:30.187971+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'2","last_scrub_stamp":"2024-02-05T06:46:53.500155+0000","last_deep_scrub":"336'2","last_deep_scrub_stamp":"2024-02-05T06:46:53.500155+0000","last_clean_scrub_stamp":"2024-02-05T06:46:53.500155+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":646,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.192955+0000","last_change":"2024-02-05T06:47:35.411302+0000","last_active":"2024-02-05T06:48:30.192955+0000","last_peered":"2024-02-05T06:48:30.192955+0000","last_clean":"2024-02-05T06:48:30.192955+0000","last_became_active":"2024-02-05T06:33:33.593901+0000","last_became_peered":"2024-02-05T06:33:33.593901+0000","last_unstale":"2024-02-05T06:48:30.192955+0000","last_undegraded":"2024-02-05T06:48:30.192955+0000","last_fullsized":"2024-02-05T06:48:30.192955+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:35.411219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:47:35.411219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:41.160858+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":363,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.189895+0000","last_change":"2024-02-05T06:46:29.865891+0000","last_active":"2024-02-05T06:48:30.189895+0000","last_peered":"2024-02-05T06:48:30.189895+0000","last_clean":"2024-02-05T06:48:30.189895+0000","last_became_active":"2024-02-05T06:40:27.077701+0000","last_became_peered":"2024-02-05T06:40:27.077701+0000","last_unstale":"2024-02-05T06:48:30.189895+0000","last_undegraded":"2024-02-05T06:48:30.189895+0000","last_fullsized":"2024-02-05T06:48:30.189895+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:29.865777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:46:29.865777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"295'15","reported_seq":447,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.190323+0000","last_change":"2024-02-05T06:46:34.466054+0000","last_active":"2024-02-05T06:48:30.190323+0000","last_peered":"2024-02-05T06:48:30.190323+0000","last_clean":"2024-02-05T06:48:30.190323+0000","last_became_active":"2024-02-05T06:39:13.244124+0000","last_became_peered":"2024-02-05T06:39:13.244124+0000","last_unstale":"2024-02-05T06:48:30.190323+0000","last_undegraded":"2024-02-05T06:48:30.190323+0000","last_fullsized":"2024-02-05T06:48:30.190323+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:46:34.465921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:46:34.465921+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":654,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.193517+0000","last_change":"2024-02-05T06:45:35.052699+0000","last_active":"2024-02-05T06:48:30.193517+0000","last_peered":"2024-02-05T06:48:30.193517+0000","last_clean":"2024-02-05T06:48:30.193517+0000","last_became_active":"2024-02-05T06:41:43.417155+0000","last_became_peered":"2024-02-05T06:41:43.417155+0000","last_unstale":"2024-02-05T06:48:30.193517+0000","last_undegraded":"2024-02-05T06:48:30.193517+0000","last_fullsized":"2024-02-05T06:48:30.193517+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:35.052451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:45:35.052451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":680,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.190331+0000","last_change":"2024-02-05T06:45:40.802539+0000","last_active":"2024-02-05T06:48:30.190331+0000","last_peered":"2024-02-05T06:48:30.190331+0000","last_clean":"2024-02-05T06:48:30.190331+0000","last_became_active":"2024-02-05T06:41:43.421433+0000","last_became_peered":"2024-02-05T06:41:43.421433+0000","last_unstale":"2024-02-05T06:48:30.190331+0000","last_undegraded":"2024-02-05T06:48:30.190331+0000","last_fullsized":"2024-02-05T06:48:30.190331+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:40.802433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:45:40.802433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":317,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.190344+0000","last_change":"2024-02-05T06:47:47.669670+0000","last_active":"2024-02-05T06:48:30.190344+0000","last_peered":"2024-02-05T06:48:30.190344+0000","last_clean":"2024-02-05T06:48:30.190344+0000","last_became_active":"2024-02-05T06:45:09.773490+0000","last_became_peered":"2024-02-05T06:45:09.773490+0000","last_unstale":"2024-02-05T06:48:30.190344+0000","last_undegraded":"2024-02-05T06:48:30.190344+0000","last_fullsized":"2024-02-05T06:48:30.190344+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:47.669594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:47:47.669594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:04.867910+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"295'24","reported_seq":532,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.861273+0000","last_change":"2024-02-05T06:48:30.861303+0000","last_active":"2024-02-05T06:48:30.861273+0000","last_peered":"2024-02-05T06:48:30.861273+0000","last_clean":"2024-02-05T06:48:30.861273+0000","last_became_active":"2024-02-05T06:45:09.784411+0000","last_became_peered":"2024-02-05T06:45:09.784411+0000","last_unstale":"2024-02-05T06:48:30.861273+0000","last_undegraded":"2024-02-05T06:48:30.861273+0000","last_fullsized":"2024-02-05T06:48:30.861273+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:48:30.861217+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:35:02.589229+0000","last_clean_scrub_stamp":"2024-02-05T06:48:30.861217+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:37.479764+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":663,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.188832+0000","last_change":"2024-02-05T06:48:24.623914+0000","last_active":"2024-02-05T06:48:30.188832+0000","last_peered":"2024-02-05T06:48:30.188832+0000","last_clean":"2024-02-05T06:48:30.188832+0000","last_became_active":"2024-02-05T06:40:27.096132+0000","last_became_peered":"2024-02-05T06:40:27.096132+0000","last_unstale":"2024-02-05T06:48:30.188832+0000","last_undegraded":"2024-02-05T06:48:30.188832+0000","last_fullsized":"2024-02-05T06:48:30.188832+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:24.623815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:48:24.623815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:51.647757+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"336'11","reported_seq":365,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.203389+0000","last_change":"2024-02-05T06:46:55.722262+0000","last_active":"2024-02-05T06:48:30.203389+0000","last_peered":"2024-02-05T06:48:30.203389+0000","last_clean":"2024-02-05T06:48:30.203389+0000","last_became_active":"2024-02-05T06:41:43.418753+0000","last_became_peered":"2024-02-05T06:41:43.418753+0000","last_unstale":"2024-02-05T06:48:30.203389+0000","last_undegraded":"2024-02-05T06:48:30.203389+0000","last_fullsized":"2024-02-05T06:48:30.203389+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'11","last_scrub_stamp":"2024-02-05T06:46:55.721412+0000","last_deep_scrub":"336'11","last_deep_scrub_stamp":"2024-02-05T06:46:55.721412+0000","last_clean_scrub_stamp":"2024-02-05T06:46:55.721412+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":372,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.190926+0000","last_change":"2024-02-05T06:48:26.670085+0000","last_active":"2024-02-05T06:48:30.190926+0000","last_peered":"2024-02-05T06:48:30.190926+0000","last_clean":"2024-02-05T06:48:30.190926+0000","last_became_active":"2024-02-05T06:40:27.098418+0000","last_became_peered":"2024-02-05T06:40:27.098418+0000","last_unstale":"2024-02-05T06:48:30.190926+0000","last_undegraded":"2024-02-05T06:48:30.190926+0000","last_fullsized":"2024-02-05T06:48:30.190926+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:26.669933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:48:26.669933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:53.011503+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"295'20","reported_seq":734,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.198723+0000","last_change":"2024-02-05T06:45:31.996335+0000","last_active":"2024-02-05T06:48:30.198723+0000","last_peered":"2024-02-05T06:48:30.198723+0000","last_clean":"2024-02-05T06:48:30.198723+0000","last_became_active":"2024-02-05T06:40:27.093418+0000","last_became_peered":"2024-02-05T06:40:27.093418+0000","last_unstale":"2024-02-05T06:48:30.198723+0000","last_undegraded":"2024-02-05T06:48:30.198723+0000","last_fullsized":"2024-02-05T06:48:30.198723+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'20","last_scrub_stamp":"2024-02-05T06:45:31.996183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:45:31.996183+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":418,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.200836+0000","last_change":"2024-02-05T06:48:23.249326+0000","last_active":"2024-02-05T06:48:30.200836+0000","last_peered":"2024-02-05T06:48:30.200836+0000","last_clean":"2024-02-05T06:48:30.200836+0000","last_became_active":"2024-02-05T06:45:09.775152+0000","last_became_peered":"2024-02-05T06:45:09.775152+0000","last_unstale":"2024-02-05T06:48:30.200836+0000","last_undegraded":"2024-02-05T06:48:30.200836+0000","last_fullsized":"2024-02-05T06:48:30.200836+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:23.249061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:41:38.559780+0000","last_clean_scrub_stamp":"2024-02-05T06:48:23.249061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:43.366513+0000","scrub_duration":164,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"208'10","reported_seq":863,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.199095+0000","last_change":"2024-02-05T06:48:18.678733+0000","last_active":"2024-02-05T06:48:30.199095+0000","last_peered":"2024-02-05T06:48:30.199095+0000","last_clean":"2024-02-05T06:48:30.199095+0000","last_became_active":"2024-02-05T06:40:27.096584+0000","last_became_peered":"2024-02-05T06:40:27.096584+0000","last_unstale":"2024-02-05T06:48:30.199095+0000","last_undegraded":"2024-02-05T06:48:30.199095+0000","last_fullsized":"2024-02-05T06:48:30.199095+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-02-05T06:48:18.677391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:48:18.677391+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:43.044904+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":683,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.190725+0000","last_change":"2024-02-05T06:48:28.523288+0000","last_active":"2024-02-05T06:48:30.190725+0000","last_peered":"2024-02-05T06:48:30.190725+0000","last_clean":"2024-02-05T06:48:30.190725+0000","last_became_active":"2024-02-05T06:40:27.088096+0000","last_became_peered":"2024-02-05T06:40:27.088096+0000","last_unstale":"2024-02-05T06:48:30.190725+0000","last_undegraded":"2024-02-05T06:48:30.190725+0000","last_fullsized":"2024-02-05T06:48:30.190725+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:28.522344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:44:01.185653+0000","last_clean_scrub_stamp":"2024-02-05T06:48:28.522344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:31.793069+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"295'15","reported_seq":297,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.203697+0000","last_change":"2024-02-05T06:46:49.603908+0000","last_active":"2024-02-05T06:48:30.203697+0000","last_peered":"2024-02-05T06:48:30.203697+0000","last_clean":"2024-02-05T06:48:30.203697+0000","last_became_active":"2024-02-05T06:41:43.413708+0000","last_became_peered":"2024-02-05T06:41:43.413708+0000","last_unstale":"2024-02-05T06:48:30.203697+0000","last_undegraded":"2024-02-05T06:48:30.203697+0000","last_fullsized":"2024-02-05T06:48:30.203697+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:46:49.603788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:46:49.603788+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":704,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.196516+0000","last_change":"2024-02-05T06:48:28.169618+0000","last_active":"2024-02-05T06:48:30.196516+0000","last_peered":"2024-02-05T06:48:30.196516+0000","last_clean":"2024-02-05T06:48:30.196516+0000","last_became_active":"2024-02-05T06:40:27.090632+0000","last_became_peered":"2024-02-05T06:40:27.090632+0000","last_unstale":"2024-02-05T06:48:30.196516+0000","last_undegraded":"2024-02-05T06:48:30.196516+0000","last_fullsized":"2024-02-05T06:48:30.196516+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:28.169488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:48:28.169488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:37.234174+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":672,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.203284+0000","last_change":"2024-02-05T06:47:43.427431+0000","last_active":"2024-02-05T06:48:30.203284+0000","last_peered":"2024-02-05T06:48:30.203284+0000","last_clean":"2024-02-05T06:48:30.203284+0000","last_became_active":"2024-02-05T06:40:27.091544+0000","last_became_peered":"2024-02-05T06:40:27.091544+0000","last_unstale":"2024-02-05T06:48:30.203284+0000","last_undegraded":"2024-02-05T06:48:30.203284+0000","last_fullsized":"2024-02-05T06:48:30.203284+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:43.427301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:47:43.427301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:08.540744+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":363,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.190027+0000","last_change":"2024-02-05T06:46:43.606599+0000","last_active":"2024-02-05T06:48:30.190027+0000","last_peered":"2024-02-05T06:48:30.190027+0000","last_clean":"2024-02-05T06:48:30.190027+0000","last_became_active":"2024-02-05T06:40:27.064388+0000","last_became_peered":"2024-02-05T06:40:27.064388+0000","last_unstale":"2024-02-05T06:48:30.190027+0000","last_undegraded":"2024-02-05T06:48:30.190027+0000","last_fullsized":"2024-02-05T06:48:30.190027+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:43.606488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:43.606488+0000","last_clean_scrub_stamp":"2024-02-05T06:46:43.606488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"295'15","reported_seq":724,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.189213+0000","last_change":"2024-02-05T06:48:29.493348+0000","last_active":"2024-02-05T06:48:30.189213+0000","last_peered":"2024-02-05T06:48:30.189213+0000","last_clean":"2024-02-05T06:48:30.189213+0000","last_became_active":"2024-02-05T06:40:27.101116+0000","last_became_peered":"2024-02-05T06:40:27.101116+0000","last_unstale":"2024-02-05T06:48:30.189213+0000","last_undegraded":"2024-02-05T06:48:30.189213+0000","last_fullsized":"2024-02-05T06:48:30.189213+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:48:29.493077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:48:29.493077+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:35.236632+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":363,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.191060+0000","last_change":"2024-02-05T06:47:38.782082+0000","last_active":"2024-02-05T06:48:30.191060+0000","last_peered":"2024-02-05T06:48:30.191060+0000","last_clean":"2024-02-05T06:48:30.191060+0000","last_became_active":"2024-02-05T06:40:27.063350+0000","last_became_peered":"2024-02-05T06:40:27.063350+0000","last_unstale":"2024-02-05T06:48:30.191060+0000","last_undegraded":"2024-02-05T06:48:30.191060+0000","last_fullsized":"2024-02-05T06:48:30.191060+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:38.781966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:47:38.781966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:57.815067+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":278,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.205273+0000","last_change":"2024-02-05T06:46:56.707237+0000","last_active":"2024-02-05T06:48:30.205273+0000","last_peered":"2024-02-05T06:48:30.205273+0000","last_clean":"2024-02-05T06:48:30.205273+0000","last_became_active":"2024-02-05T06:41:43.406094+0000","last_became_peered":"2024-02-05T06:41:43.406094+0000","last_unstale":"2024-02-05T06:48:30.205273+0000","last_undegraded":"2024-02-05T06:48:30.205273+0000","last_fullsized":"2024-02-05T06:48:30.205273+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:56.707131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:44:57.506909+0000","last_clean_scrub_stamp":"2024-02-05T06:46:56.707131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":697,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.195109+0000","last_change":"2024-02-05T06:48:28.815276+0000","last_active":"2024-02-05T06:48:30.195109+0000","last_peered":"2024-02-05T06:48:30.195109+0000","last_clean":"2024-02-05T06:48:30.195109+0000","last_became_active":"2024-02-05T06:41:43.412130+0000","last_became_peered":"2024-02-05T06:41:43.412130+0000","last_unstale":"2024-02-05T06:48:30.195109+0000","last_undegraded":"2024-02-05T06:48:30.195109+0000","last_fullsized":"2024-02-05T06:48:30.195109+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:28.815050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:48:28.815050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:55.835746+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"295'12","reported_seq":696,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.203374+0000","last_change":"2024-02-05T06:46:47.455766+0000","last_active":"2024-02-05T06:48:30.203374+0000","last_peered":"2024-02-05T06:48:30.203374+0000","last_clean":"2024-02-05T06:48:30.203374+0000","last_became_active":"2024-02-05T06:40:27.091689+0000","last_became_peered":"2024-02-05T06:40:27.091689+0000","last_unstale":"2024-02-05T06:48:30.203374+0000","last_undegraded":"2024-02-05T06:48:30.203374+0000","last_fullsized":"2024-02-05T06:48:30.203374+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'12","last_scrub_stamp":"2024-02-05T06:46:47.455632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:46:47.455632+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"369'5323","reported_seq":5380,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:31.629328+0000","last_change":"2024-02-05T06:47:24.184995+0000","last_active":"2024-02-05T06:48:31.629328+0000","last_peered":"2024-02-05T06:48:31.629328+0000","last_clean":"2024-02-05T06:48:31.629328+0000","last_became_active":"2024-02-05T06:47:24.184581+0000","last_became_peered":"2024-02-05T06:47:24.184581+0000","last_unstale":"2024-02-05T06:48:31.629328+0000","last_undegraded":"2024-02-05T06:48:31.629328+0000","last_fullsized":"2024-02-05T06:48:31.629328+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":5323,"log_dups_size":0,"ondisk_log_size":5323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:50.103785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348848128,"num_objects":5323,"num_object_clones":0,"num_object_copies":15969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5323,"num_write_kb":340672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":279,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.203336+0000","last_change":"2024-02-05T06:47:38.659232+0000","last_active":"2024-02-05T06:48:30.203336+0000","last_peered":"2024-02-05T06:48:30.203336+0000","last_clean":"2024-02-05T06:48:30.203336+0000","last_became_active":"2024-02-05T06:41:43.421231+0000","last_became_peered":"2024-02-05T06:41:43.421231+0000","last_unstale":"2024-02-05T06:48:30.203336+0000","last_undegraded":"2024-02-05T06:48:30.203336+0000","last_fullsized":"2024-02-05T06:48:30.203336+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:38.659123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:47:38.659123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:53.928020+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":377,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.190644+0000","last_change":"2024-02-05T06:48:25.672942+0000","last_active":"2024-02-05T06:48:30.190644+0000","last_peered":"2024-02-05T06:48:30.190644+0000","last_clean":"2024-02-05T06:48:30.190644+0000","last_became_active":"2024-02-05T06:40:27.068279+0000","last_became_peered":"2024-02-05T06:40:27.068279+0000","last_unstale":"2024-02-05T06:48:30.190644+0000","last_undegraded":"2024-02-05T06:48:30.190644+0000","last_fullsized":"2024-02-05T06:48:30.190644+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-02-05T06:48:25.672806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:48:25.672806+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:44.484988+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"295'27","reported_seq":313,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.200937+0000","last_change":"2024-02-05T06:48:22.632196+0000","last_active":"2024-02-05T06:48:30.200937+0000","last_peered":"2024-02-05T06:48:30.200937+0000","last_clean":"2024-02-05T06:48:30.200937+0000","last_became_active":"2024-02-05T06:41:43.412579+0000","last_became_peered":"2024-02-05T06:41:43.412579+0000","last_unstale":"2024-02-05T06:48:30.200937+0000","last_undegraded":"2024-02-05T06:48:30.200937+0000","last_fullsized":"2024-02-05T06:48:30.200937+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'27","last_scrub_stamp":"2024-02-05T06:48:22.632056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:48:22.632056+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:28.762241+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"208'12","reported_seq":472,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.189937+0000","last_change":"2024-02-05T06:48:27.609263+0000","last_active":"2024-02-05T06:48:30.189937+0000","last_peered":"2024-02-05T06:48:30.189937+0000","last_clean":"2024-02-05T06:48:30.189937+0000","last_became_active":"2024-02-05T06:40:27.064195+0000","last_became_peered":"2024-02-05T06:40:27.064195+0000","last_unstale":"2024-02-05T06:48:30.189937+0000","last_undegraded":"2024-02-05T06:48:30.189937+0000","last_fullsized":"2024-02-05T06:48:30.189937+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'12","last_scrub_stamp":"2024-02-05T06:48:27.609150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:48:27.609150+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:31.872261+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"369'5384","reported_seq":5442,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:31.531924+0000","last_change":"2024-02-05T06:47:24.176419+0000","last_active":"2024-02-05T06:48:31.531924+0000","last_peered":"2024-02-05T06:48:31.531924+0000","last_clean":"2024-02-05T06:48:31.531924+0000","last_became_active":"2024-02-05T06:47:24.176055+0000","last_became_peered":"2024-02-05T06:47:24.176055+0000","last_unstale":"2024-02-05T06:48:31.531924+0000","last_undegraded":"2024-02-05T06:48:31.531924+0000","last_fullsized":"2024-02-05T06:48:31.531924+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":5384,"log_dups_size":0,"ondisk_log_size":5384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:37.445734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":352845824,"num_objects":5384,"num_object_clones":0,"num_object_copies":16152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5384,"num_write_kb":344576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":654,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.193199+0000","last_change":"2024-02-05T06:47:36.512402+0000","last_active":"2024-02-05T06:48:30.193199+0000","last_peered":"2024-02-05T06:48:30.193199+0000","last_clean":"2024-02-05T06:48:30.193199+0000","last_became_active":"2024-02-05T06:40:27.092613+0000","last_became_peered":"2024-02-05T06:40:27.092613+0000","last_unstale":"2024-02-05T06:48:30.193199+0000","last_undegraded":"2024-02-05T06:48:30.193199+0000","last_fullsized":"2024-02-05T06:48:30.193199+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:36.512306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:47:36.512306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:40.368980+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":363,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.190249+0000","last_change":"2024-02-05T06:46:53.740645+0000","last_active":"2024-02-05T06:48:30.190249+0000","last_peered":"2024-02-05T06:48:30.190249+0000","last_clean":"2024-02-05T06:48:30.190249+0000","last_became_active":"2024-02-05T06:40:27.063300+0000","last_became_peered":"2024-02-05T06:40:27.063300+0000","last_unstale":"2024-02-05T06:48:30.190249+0000","last_undegraded":"2024-02-05T06:48:30.190249+0000","last_fullsized":"2024-02-05T06:48:30.190249+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:53.740542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:46:53.740542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"295'18","reported_seq":522,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.186299+0000","last_change":"2024-02-05T06:46:58.712252+0000","last_active":"2024-02-05T06:48:30.186299+0000","last_peered":"2024-02-05T06:48:30.186299+0000","last_clean":"2024-02-05T06:48:30.186299+0000","last_became_active":"2024-02-05T06:45:09.783036+0000","last_became_peered":"2024-02-05T06:45:09.783036+0000","last_unstale":"2024-02-05T06:48:30.186299+0000","last_undegraded":"2024-02-05T06:48:30.186299+0000","last_fullsized":"2024-02-05T06:48:30.186299+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'18","last_scrub_stamp":"2024-02-05T06:46:58.712125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:46:58.712125+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":20,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":676,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.195470+0000","last_change":"2024-02-05T06:48:20.050901+0000","last_active":"2024-02-05T06:48:30.195470+0000","last_peered":"2024-02-05T06:48:30.195470+0000","last_clean":"2024-02-05T06:48:30.195470+0000","last_became_active":"2024-02-05T06:33:33.592546+0000","last_became_peered":"2024-02-05T06:33:33.592546+0000","last_unstale":"2024-02-05T06:48:30.195470+0000","last_undegraded":"2024-02-05T06:48:30.195470+0000","last_fullsized":"2024-02-05T06:48:30.195470+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:20.050623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:42.685240+0000","last_clean_scrub_stamp":"2024-02-05T06:48:20.050623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:39.992367+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.4","version":"369'5261","reported_seq":5318,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:32.473286+0000","last_change":"2024-02-05T06:47:24.185199+0000","last_active":"2024-02-05T06:48:32.473286+0000","last_peered":"2024-02-05T06:48:32.473286+0000","last_clean":"2024-02-05T06:48:32.473286+0000","last_became_active":"2024-02-05T06:47:24.184891+0000","last_became_peered":"2024-02-05T06:47:24.184891+0000","last_unstale":"2024-02-05T06:48:32.473286+0000","last_undegraded":"2024-02-05T06:48:32.473286+0000","last_fullsized":"2024-02-05T06:48:32.473286+0000","mapping_epoch":345,"log_start":"352'200","ondisk_log_start":"352'200","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":5061,"log_dups_size":0,"ondisk_log_size":5061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":344784896,"num_objects":5261,"num_object_clones":0,"num_object_copies":15783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5261,"num_write_kb":336704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":363,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.189495+0000","last_change":"2024-02-05T06:46:46.694131+0000","last_active":"2024-02-05T06:48:30.189495+0000","last_peered":"2024-02-05T06:48:30.189495+0000","last_clean":"2024-02-05T06:48:30.189495+0000","last_became_active":"2024-02-05T06:40:27.218081+0000","last_became_peered":"2024-02-05T06:40:27.218081+0000","last_unstale":"2024-02-05T06:48:30.189495+0000","last_undegraded":"2024-02-05T06:48:30.189495+0000","last_fullsized":"2024-02-05T06:48:30.189495+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:46.694027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:32.925000+0000","last_clean_scrub_stamp":"2024-02-05T06:46:46.694027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":694,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.196618+0000","last_change":"2024-02-05T06:47:45.073824+0000","last_active":"2024-02-05T06:48:30.196618+0000","last_peered":"2024-02-05T06:48:30.196618+0000","last_clean":"2024-02-05T06:48:30.196618+0000","last_became_active":"2024-02-05T06:41:43.413095+0000","last_became_peered":"2024-02-05T06:41:43.413095+0000","last_unstale":"2024-02-05T06:48:30.196618+0000","last_undegraded":"2024-02-05T06:48:30.196618+0000","last_fullsized":"2024-02-05T06:48:30.196618+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:45.073681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:45.073681+0000","last_clean_scrub_stamp":"2024-02-05T06:47:45.073681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:57.251457+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":369,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.189985+0000","last_change":"2024-02-05T06:47:36.783062+0000","last_active":"2024-02-05T06:48:30.189985+0000","last_peered":"2024-02-05T06:48:30.189985+0000","last_clean":"2024-02-05T06:48:30.189985+0000","last_became_active":"2024-02-05T06:41:43.418202+0000","last_became_peered":"2024-02-05T06:41:43.418202+0000","last_unstale":"2024-02-05T06:48:30.189985+0000","last_undegraded":"2024-02-05T06:48:30.189985+0000","last_fullsized":"2024-02-05T06:48:30.189985+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:36.782960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:47:36.782960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:48.010223+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"295'44","reported_seq":823,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.193634+0000","last_change":"2024-02-05T06:47:50.338639+0000","last_active":"2024-02-05T06:48:30.193634+0000","last_peered":"2024-02-05T06:48:30.193634+0000","last_clean":"2024-02-05T06:48:30.193634+0000","last_became_active":"2024-02-05T06:40:27.068417+0000","last_became_peered":"2024-02-05T06:40:27.068417+0000","last_unstale":"2024-02-05T06:48:30.193634+0000","last_undegraded":"2024-02-05T06:48:30.193634+0000","last_fullsized":"2024-02-05T06:48:30.193634+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'44","last_scrub_stamp":"2024-02-05T06:47:50.338505+0000","last_deep_scrub":"279'28","last_deep_scrub_stamp":"2024-02-05T06:42:50.148881+0000","last_clean_scrub_stamp":"2024-02-05T06:47:50.338505+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:03.563604+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"369'5406","reported_seq":5464,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:31.860885+0000","last_change":"2024-02-05T06:47:24.178230+0000","last_active":"2024-02-05T06:48:31.860885+0000","last_peered":"2024-02-05T06:48:31.860885+0000","last_clean":"2024-02-05T06:48:31.860885+0000","last_became_active":"2024-02-05T06:47:24.177943+0000","last_became_peered":"2024-02-05T06:47:24.177943+0000","last_unstale":"2024-02-05T06:48:31.860885+0000","last_undegraded":"2024-02-05T06:48:31.860885+0000","last_fullsized":"2024-02-05T06:48:31.860885+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":5406,"log_dups_size":0,"ondisk_log_size":5406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:45.429907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":354287616,"num_objects":5406,"num_object_clones":0,"num_object_copies":16218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5406,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5406,"num_write_kb":345984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":655,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.190537+0000","last_change":"2024-02-05T06:46:33.542381+0000","last_active":"2024-02-05T06:48:30.190537+0000","last_peered":"2024-02-05T06:48:30.190537+0000","last_clean":"2024-02-05T06:48:30.190537+0000","last_became_active":"2024-02-05T06:40:27.129625+0000","last_became_peered":"2024-02-05T06:40:27.129625+0000","last_unstale":"2024-02-05T06:48:30.190537+0000","last_undegraded":"2024-02-05T06:48:30.190537+0000","last_fullsized":"2024-02-05T06:48:30.190537+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:33.542276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:46:33.542276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":130,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":696,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.195844+0000","last_change":"2024-02-05T06:48:26.126925+0000","last_active":"2024-02-05T06:48:30.195844+0000","last_peered":"2024-02-05T06:48:30.195844+0000","last_clean":"2024-02-05T06:48:30.195844+0000","last_became_active":"2024-02-05T06:36:54.966016+0000","last_became_peered":"2024-02-05T06:36:54.966016+0000","last_unstale":"2024-02-05T06:48:30.195844+0000","last_undegraded":"2024-02-05T06:48:30.195844+0000","last_fullsized":"2024-02-05T06:48:30.195844+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:26.126804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:48:26.126804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:55.439596+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":279,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.204081+0000","last_change":"2024-02-05T06:47:41.617654+0000","last_active":"2024-02-05T06:48:30.204081+0000","last_peered":"2024-02-05T06:48:30.204081+0000","last_clean":"2024-02-05T06:48:30.204081+0000","last_became_active":"2024-02-05T06:41:43.412611+0000","last_became_peered":"2024-02-05T06:41:43.412611+0000","last_unstale":"2024-02-05T06:48:30.204081+0000","last_undegraded":"2024-02-05T06:48:30.204081+0000","last_fullsized":"2024-02-05T06:48:30.204081+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:41.617526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:41.617526+0000","last_clean_scrub_stamp":"2024-02-05T06:47:41.617526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:51.274032+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"295'15","reported_seq":772,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.189495+0000","last_change":"2024-02-05T06:48:23.607645+0000","last_active":"2024-02-05T06:48:30.189495+0000","last_peered":"2024-02-05T06:48:30.189495+0000","last_clean":"2024-02-05T06:48:30.189495+0000","last_became_active":"2024-02-05T06:40:08.584204+0000","last_became_peered":"2024-02-05T06:40:08.584204+0000","last_unstale":"2024-02-05T06:48:30.189495+0000","last_undegraded":"2024-02-05T06:48:30.189495+0000","last_fullsized":"2024-02-05T06:48:30.189495+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:48:23.607540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:48:23.607540+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:43.691182+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"369'5348","reported_seq":5405,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:31.536303+0000","last_change":"2024-02-05T06:47:24.178003+0000","last_active":"2024-02-05T06:48:31.536303+0000","last_peered":"2024-02-05T06:48:31.536303+0000","last_clean":"2024-02-05T06:48:31.536303+0000","last_became_active":"2024-02-05T06:47:24.177714+0000","last_became_peered":"2024-02-05T06:47:24.177714+0000","last_unstale":"2024-02-05T06:48:31.536303+0000","last_undegraded":"2024-02-05T06:48:31.536303+0000","last_fullsized":"2024-02-05T06:48:31.536303+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":5348,"log_dups_size":0,"ondisk_log_size":5348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350486528,"num_objects":5348,"num_object_clones":0,"num_object_copies":16044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5348,"num_write_kb":342272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":696,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.203017+0000","last_change":"2024-02-05T06:46:47.669870+0000","last_active":"2024-02-05T06:48:30.203017+0000","last_peered":"2024-02-05T06:48:30.203017+0000","last_clean":"2024-02-05T06:48:30.203017+0000","last_became_active":"2024-02-05T06:41:46.054968+0000","last_became_peered":"2024-02-05T06:41:46.054968+0000","last_unstale":"2024-02-05T06:48:30.203017+0000","last_undegraded":"2024-02-05T06:48:30.203017+0000","last_fullsized":"2024-02-05T06:48:30.203017+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:46:47.669634+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:37.282442+0000","last_clean_scrub_stamp":"2024-02-05T06:46:47.669634+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":666,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.193356+0000","last_change":"2024-02-05T06:46:48.502814+0000","last_active":"2024-02-05T06:48:30.193356+0000","last_peered":"2024-02-05T06:48:30.193356+0000","last_clean":"2024-02-05T06:48:30.193356+0000","last_became_active":"2024-02-05T06:40:27.082440+0000","last_became_peered":"2024-02-05T06:40:27.082440+0000","last_unstale":"2024-02-05T06:48:30.193356+0000","last_undegraded":"2024-02-05T06:48:30.193356+0000","last_fullsized":"2024-02-05T06:48:30.193356+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:48.502695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:46:48.502695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"182'3","reported_seq":283,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.203474+0000","last_change":"2024-02-05T06:48:14.571567+0000","last_active":"2024-02-05T06:48:30.203474+0000","last_peered":"2024-02-05T06:48:30.203474+0000","last_clean":"2024-02-05T06:48:30.203474+0000","last_became_active":"2024-02-05T06:41:43.415806+0000","last_became_peered":"2024-02-05T06:41:43.415806+0000","last_unstale":"2024-02-05T06:48:30.203474+0000","last_undegraded":"2024-02-05T06:48:30.203474+0000","last_fullsized":"2024-02-05T06:48:30.203474+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'3","last_scrub_stamp":"2024-02-05T06:48:14.571441+0000","last_deep_scrub":"182'3","last_deep_scrub_stamp":"2024-02-05T06:39:33.373415+0000","last_clean_scrub_stamp":"2024-02-05T06:48:14.571441+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:26.936434+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.001220348,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":192,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.189532+0000","last_change":"2024-02-05T06:48:16.597953+0000","last_active":"2024-02-05T06:48:30.189532+0000","last_peered":"2024-02-05T06:48:30.189532+0000","last_clean":"2024-02-05T06:48:30.189532+0000","last_became_active":"2024-02-05T06:45:09.776169+0000","last_became_peered":"2024-02-05T06:45:09.776169+0000","last_unstale":"2024-02-05T06:48:30.189532+0000","last_undegraded":"2024-02-05T06:48:30.189532+0000","last_fullsized":"2024-02-05T06:48:30.189532+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:16.597853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:48:16.597853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:26.531928+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"326'52","reported_seq":536,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.188028+0000","last_change":"2024-02-05T06:48:16.104872+0000","last_active":"2024-02-05T06:48:30.188028+0000","last_peered":"2024-02-05T06:48:30.188028+0000","last_clean":"2024-02-05T06:48:30.188028+0000","last_became_active":"2024-02-05T06:40:08.604483+0000","last_became_peered":"2024-02-05T06:40:08.604483+0000","last_unstale":"2024-02-05T06:48:30.188028+0000","last_undegraded":"2024-02-05T06:48:30.188028+0000","last_fullsized":"2024-02-05T06:48:30.188028+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'52","last_scrub_stamp":"2024-02-05T06:48:16.104709+0000","last_deep_scrub":"326'52","last_deep_scrub_stamp":"2024-02-05T06:45:53.715482+0000","last_clean_scrub_stamp":"2024-02-05T06:48:16.104709+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:21.284859+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":668,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.188323+0000","last_change":"2024-02-05T06:47:49.412086+0000","last_active":"2024-02-05T06:48:30.188323+0000","last_peered":"2024-02-05T06:48:30.188323+0000","last_clean":"2024-02-05T06:48:30.188323+0000","last_became_active":"2024-02-05T06:36:54.973387+0000","last_became_peered":"2024-02-05T06:36:54.973387+0000","last_unstale":"2024-02-05T06:48:30.188323+0000","last_undegraded":"2024-02-05T06:48:30.188323+0000","last_fullsized":"2024-02-05T06:48:30.188323+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:49.411958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:47:49.411958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:08.497020+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"369'5206","reported_seq":5263,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:31.627225+0000","last_change":"2024-02-05T06:47:24.177476+0000","last_active":"2024-02-05T06:48:31.627225+0000","last_peered":"2024-02-05T06:48:31.627225+0000","last_clean":"2024-02-05T06:48:31.627225+0000","last_became_active":"2024-02-05T06:47:24.177145+0000","last_became_peered":"2024-02-05T06:47:24.177145+0000","last_unstale":"2024-02-05T06:48:31.627225+0000","last_undegraded":"2024-02-05T06:48:31.627225+0000","last_fullsized":"2024-02-05T06:48:31.627225+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":5206,"log_dups_size":0,"ondisk_log_size":5206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:45.854097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341180416,"num_objects":5206,"num_object_clones":0,"num_object_copies":15618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5206,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5206,"num_write_kb":333184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":792,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.189954+0000","last_change":"2024-02-05T06:48:28.657825+0000","last_active":"2024-02-05T06:48:30.189954+0000","last_peered":"2024-02-05T06:48:30.189954+0000","last_clean":"2024-02-05T06:48:30.189954+0000","last_became_active":"2024-02-05T06:40:30.082467+0000","last_became_peered":"2024-02-05T06:40:30.082467+0000","last_unstale":"2024-02-05T06:48:30.189954+0000","last_undegraded":"2024-02-05T06:48:30.189954+0000","last_fullsized":"2024-02-05T06:48:30.189954+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:48:28.657679+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:48:28.657679+0000","last_clean_scrub_stamp":"2024-02-05T06:48:28.657679+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:36.972014+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":371,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.191132+0000","last_change":"2024-02-05T06:48:29.707093+0000","last_active":"2024-02-05T06:48:30.191132+0000","last_peered":"2024-02-05T06:48:30.191132+0000","last_clean":"2024-02-05T06:48:30.191132+0000","last_became_active":"2024-02-05T06:40:27.077966+0000","last_became_peered":"2024-02-05T06:40:27.077966+0000","last_unstale":"2024-02-05T06:48:30.191132+0000","last_undegraded":"2024-02-05T06:48:30.191132+0000","last_fullsized":"2024-02-05T06:48:30.191132+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:29.706963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:45.774441+0000","last_clean_scrub_stamp":"2024-02-05T06:48:29.706963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:54.993746+0000","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":663,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.187078+0000","last_change":"2024-02-05T06:46:32.768890+0000","last_active":"2024-02-05T06:48:30.187078+0000","last_peered":"2024-02-05T06:48:30.187078+0000","last_clean":"2024-02-05T06:48:30.187078+0000","last_became_active":"2024-02-05T06:33:29.551549+0000","last_became_peered":"2024-02-05T06:33:29.551549+0000","last_unstale":"2024-02-05T06:48:30.187078+0000","last_undegraded":"2024-02-05T06:48:30.187078+0000","last_fullsized":"2024-02-05T06:48:30.187078+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:32.768777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:46:32.768777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":694,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.195692+0000","last_change":"2024-02-05T06:47:49.065297+0000","last_active":"2024-02-05T06:48:30.195692+0000","last_peered":"2024-02-05T06:48:30.195692+0000","last_clean":"2024-02-05T06:48:30.195692+0000","last_became_active":"2024-02-05T06:33:28.542977+0000","last_became_peered":"2024-02-05T06:33:28.542977+0000","last_unstale":"2024-02-05T06:48:30.195692+0000","last_undegraded":"2024-02-05T06:48:30.195692+0000","last_fullsized":"2024-02-05T06:48:30.195692+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:49.065028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:47:49.065028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:57.562644+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00091640799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"295'17","reported_seq":761,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.197372+0000","last_change":"2024-02-05T06:46:43.816340+0000","last_active":"2024-02-05T06:48:30.197372+0000","last_peered":"2024-02-05T06:48:30.197372+0000","last_clean":"2024-02-05T06:48:30.197372+0000","last_became_active":"2024-02-05T06:41:43.420458+0000","last_became_peered":"2024-02-05T06:41:43.420458+0000","last_unstale":"2024-02-05T06:48:30.197372+0000","last_undegraded":"2024-02-05T06:48:30.197372+0000","last_fullsized":"2024-02-05T06:48:30.197372+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'17","last_scrub_stamp":"2024-02-05T06:46:43.816187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:46:43.816187+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":708,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.200063+0000","last_change":"2024-02-05T06:47:51.064183+0000","last_active":"2024-02-05T06:48:30.200063+0000","last_peered":"2024-02-05T06:48:30.200063+0000","last_clean":"2024-02-05T06:48:30.200063+0000","last_became_active":"2024-02-05T06:41:43.412387+0000","last_became_peered":"2024-02-05T06:41:43.412387+0000","last_unstale":"2024-02-05T06:48:30.200063+0000","last_undegraded":"2024-02-05T06:48:30.200063+0000","last_fullsized":"2024-02-05T06:48:30.200063+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:51.063926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:33.914422+0000","last_clean_scrub_stamp":"2024-02-05T06:47:51.063926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:55.492615+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"369'5295","reported_seq":5353,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:31.543234+0000","last_change":"2024-02-05T06:47:24.180383+0000","last_active":"2024-02-05T06:48:31.543234+0000","last_peered":"2024-02-05T06:48:31.543234+0000","last_clean":"2024-02-05T06:48:31.543234+0000","last_became_active":"2024-02-05T06:47:24.179124+0000","last_became_peered":"2024-02-05T06:47:24.179124+0000","last_unstale":"2024-02-05T06:48:31.543234+0000","last_undegraded":"2024-02-05T06:48:31.543234+0000","last_fullsized":"2024-02-05T06:48:31.543234+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":5295,"log_dups_size":0,"ondisk_log_size":5295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:49.727806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347013120,"num_objects":5295,"num_object_clones":0,"num_object_copies":15885,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5295,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5295,"num_write_kb":338880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":279,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.209676+0000","last_change":"2024-02-05T06:46:48.716378+0000","last_active":"2024-02-05T06:48:30.209676+0000","last_peered":"2024-02-05T06:48:30.209676+0000","last_clean":"2024-02-05T06:48:30.209676+0000","last_became_active":"2024-02-05T06:41:43.408773+0000","last_became_peered":"2024-02-05T06:41:43.408773+0000","last_unstale":"2024-02-05T06:48:30.209676+0000","last_undegraded":"2024-02-05T06:48:30.209676+0000","last_fullsized":"2024-02-05T06:48:30.209676+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:48.716273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:46:48.716273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":785,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.189760+0000","last_change":"2024-02-05T06:46:33.834624+0000","last_active":"2024-02-05T06:48:30.189760+0000","last_peered":"2024-02-05T06:48:30.189760+0000","last_clean":"2024-02-05T06:48:30.189760+0000","last_became_active":"2024-02-05T06:40:30.081178+0000","last_became_peered":"2024-02-05T06:40:30.081178+0000","last_unstale":"2024-02-05T06:48:30.189760+0000","last_undegraded":"2024-02-05T06:48:30.189760+0000","last_fullsized":"2024-02-05T06:48:30.189760+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:46:33.834510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:46:33.834510+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":667,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.192546+0000","last_change":"2024-02-05T06:45:57.195429+0000","last_active":"2024-02-05T06:48:30.192546+0000","last_peered":"2024-02-05T06:48:30.192546+0000","last_clean":"2024-02-05T06:48:30.192546+0000","last_became_active":"2024-02-05T06:41:43.411335+0000","last_became_peered":"2024-02-05T06:41:43.411335+0000","last_unstale":"2024-02-05T06:48:30.192546+0000","last_undegraded":"2024-02-05T06:48:30.192546+0000","last_fullsized":"2024-02-05T06:48:30.192546+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:57.195321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:45:57.195321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"295'32","reported_seq":993,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.194639+0000","last_change":"2024-02-05T06:47:35.173300+0000","last_active":"2024-02-05T06:48:30.194639+0000","last_peered":"2024-02-05T06:48:30.194639+0000","last_clean":"2024-02-05T06:48:30.194639+0000","last_became_active":"2024-02-05T06:40:27.061111+0000","last_became_peered":"2024-02-05T06:40:27.061111+0000","last_unstale":"2024-02-05T06:48:30.194639+0000","last_undegraded":"2024-02-05T06:48:30.194639+0000","last_fullsized":"2024-02-05T06:48:30.194639+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'32","last_scrub_stamp":"2024-02-05T06:47:35.173150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:47:35.173150+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:58.867739+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":377,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.190517+0000","last_change":"2024-02-05T06:48:21.697741+0000","last_active":"2024-02-05T06:48:30.190517+0000","last_peered":"2024-02-05T06:48:30.190517+0000","last_clean":"2024-02-05T06:48:30.190517+0000","last_became_active":"2024-02-05T06:41:43.418198+0000","last_became_peered":"2024-02-05T06:41:43.418198+0000","last_unstale":"2024-02-05T06:48:30.190517+0000","last_undegraded":"2024-02-05T06:48:30.190517+0000","last_fullsized":"2024-02-05T06:48:30.190517+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:21.697458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:48:21.697458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:35.467036+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"369'5350","reported_seq":5408,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:32.420832+0000","last_change":"2024-02-05T06:47:24.177582+0000","last_active":"2024-02-05T06:48:32.420832+0000","last_peered":"2024-02-05T06:48:32.420832+0000","last_clean":"2024-02-05T06:48:32.420832+0000","last_became_active":"2024-02-05T06:47:24.177037+0000","last_became_peered":"2024-02-05T06:47:24.177037+0000","last_unstale":"2024-02-05T06:48:32.420832+0000","last_undegraded":"2024-02-05T06:48:32.420832+0000","last_fullsized":"2024-02-05T06:48:32.420832+0000","mapping_epoch":345,"log_start":"352'300","ondisk_log_start":"352'300","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":5050,"log_dups_size":0,"ondisk_log_size":5050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350617600,"num_objects":5350,"num_object_clones":0,"num_object_copies":16050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5350,"num_write_kb":342400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":826,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.188493+0000","last_change":"2024-02-05T06:46:42.818851+0000","last_active":"2024-02-05T06:48:30.188493+0000","last_peered":"2024-02-05T06:48:30.188493+0000","last_clean":"2024-02-05T06:48:30.188493+0000","last_became_active":"2024-02-05T06:37:37.137754+0000","last_became_peered":"2024-02-05T06:37:37.137754+0000","last_unstale":"2024-02-05T06:48:30.188493+0000","last_undegraded":"2024-02-05T06:48:30.188493+0000","last_fullsized":"2024-02-05T06:48:30.188493+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:46:42.818725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:46:42.818725+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":671,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.190290+0000","last_change":"2024-02-05T06:46:31.716329+0000","last_active":"2024-02-05T06:48:30.190290+0000","last_peered":"2024-02-05T06:48:30.190290+0000","last_clean":"2024-02-05T06:48:30.190290+0000","last_became_active":"2024-02-05T06:41:43.410910+0000","last_became_peered":"2024-02-05T06:41:43.410910+0000","last_unstale":"2024-02-05T06:48:30.190290+0000","last_undegraded":"2024-02-05T06:48:30.190290+0000","last_fullsized":"2024-02-05T06:48:30.190290+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:31.716099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:46:31.716099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":676,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.190323+0000","last_change":"2024-02-05T06:46:56.459465+0000","last_active":"2024-02-05T06:48:30.190323+0000","last_peered":"2024-02-05T06:48:30.190323+0000","last_clean":"2024-02-05T06:48:30.190323+0000","last_became_active":"2024-02-05T06:36:54.972264+0000","last_became_peered":"2024-02-05T06:36:54.972264+0000","last_unstale":"2024-02-05T06:48:30.190323+0000","last_undegraded":"2024-02-05T06:48:30.190323+0000","last_fullsized":"2024-02-05T06:48:30.190323+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:56.459164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:46:56.459164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"355'6","reported_seq":856,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.189707+0000","last_change":"2024-02-05T06:47:46.642736+0000","last_active":"2024-02-05T06:48:30.189707+0000","last_peered":"2024-02-05T06:48:30.189707+0000","last_clean":"2024-02-05T06:48:30.189707+0000","last_became_active":"2024-02-05T06:40:27.071509+0000","last_became_peered":"2024-02-05T06:40:27.071509+0000","last_unstale":"2024-02-05T06:48:30.189707+0000","last_undegraded":"2024-02-05T06:48:30.189707+0000","last_fullsized":"2024-02-05T06:48:30.189707+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'6","last_scrub_stamp":"2024-02-05T06:47:46.642454+0000","last_deep_scrub":"355'6","last_deep_scrub_stamp":"2024-02-05T06:47:46.642454+0000","last_clean_scrub_stamp":"2024-02-05T06:47:46.642454+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:59.661377+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00074151600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":490,"num_read_kb":485,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"295'24","reported_seq":695,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.188927+0000","last_change":"2024-02-05T06:47:40.582521+0000","last_active":"2024-02-05T06:48:30.188927+0000","last_peered":"2024-02-05T06:48:30.188927+0000","last_clean":"2024-02-05T06:48:30.188927+0000","last_became_active":"2024-02-05T06:33:31.564604+0000","last_became_peered":"2024-02-05T06:33:31.564604+0000","last_unstale":"2024-02-05T06:48:30.188927+0000","last_undegraded":"2024-02-05T06:48:30.188927+0000","last_fullsized":"2024-02-05T06:48:30.188927+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:47:40.582236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:47:40.582236+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:54.033478+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":279,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.209735+0000","last_change":"2024-02-05T06:48:20.716170+0000","last_active":"2024-02-05T06:48:30.209735+0000","last_peered":"2024-02-05T06:48:30.209735+0000","last_clean":"2024-02-05T06:48:30.209735+0000","last_became_active":"2024-02-05T06:41:43.417751+0000","last_became_peered":"2024-02-05T06:41:43.417751+0000","last_unstale":"2024-02-05T06:48:30.209735+0000","last_undegraded":"2024-02-05T06:48:30.209735+0000","last_fullsized":"2024-02-05T06:48:30.209735+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:20.716046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:48:20.716046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:30.332921+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"369'5430","reported_seq":5486,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:31.554673+0000","last_change":"2024-02-05T06:47:24.182432+0000","last_active":"2024-02-05T06:48:31.554673+0000","last_peered":"2024-02-05T06:48:31.554673+0000","last_clean":"2024-02-05T06:48:31.554673+0000","last_became_active":"2024-02-05T06:47:24.182110+0000","last_became_peered":"2024-02-05T06:47:24.182110+0000","last_unstale":"2024-02-05T06:48:31.554673+0000","last_undegraded":"2024-02-05T06:48:31.554673+0000","last_fullsized":"2024-02-05T06:48:31.554673+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":5430,"log_dups_size":0,"ondisk_log_size":5430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:46.830260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":355860480,"num_objects":5430,"num_object_clones":0,"num_object_copies":16290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5430,"num_write_kb":347520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":847,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.195086+0000","last_change":"2024-02-05T06:48:17.729708+0000","last_active":"2024-02-05T06:48:30.195086+0000","last_peered":"2024-02-05T06:48:30.195086+0000","last_clean":"2024-02-05T06:48:30.195086+0000","last_became_active":"2024-02-05T06:37:37.140011+0000","last_became_peered":"2024-02-05T06:37:37.140011+0000","last_unstale":"2024-02-05T06:48:30.195086+0000","last_undegraded":"2024-02-05T06:48:30.195086+0000","last_fullsized":"2024-02-05T06:48:30.195086+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:48:17.729582+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:38.778568+0000","last_clean_scrub_stamp":"2024-02-05T06:48:17.729582+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:39.030922+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":659,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.186558+0000","last_change":"2024-02-05T06:47:45.022360+0000","last_active":"2024-02-05T06:48:30.186558+0000","last_peered":"2024-02-05T06:48:30.186558+0000","last_clean":"2024-02-05T06:48:30.186558+0000","last_became_active":"2024-02-05T06:33:35.664858+0000","last_became_peered":"2024-02-05T06:33:35.664858+0000","last_unstale":"2024-02-05T06:48:30.186558+0000","last_undegraded":"2024-02-05T06:48:30.186558+0000","last_fullsized":"2024-02-05T06:48:30.186558+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:45.022240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:47:45.022240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:09.456304+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"66'1","reported_seq":697,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.190361+0000","last_change":"2024-02-05T06:46:30.749245+0000","last_active":"2024-02-05T06:48:30.190361+0000","last_peered":"2024-02-05T06:48:30.190361+0000","last_clean":"2024-02-05T06:48:30.190361+0000","last_became_active":"2024-02-05T06:41:43.411404+0000","last_became_peered":"2024-02-05T06:41:43.411404+0000","last_unstale":"2024-02-05T06:48:30.190361+0000","last_undegraded":"2024-02-05T06:48:30.190361+0000","last_fullsized":"2024-02-05T06:48:30.190361+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:46:30.748577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:46:30.748577+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"295'29","reported_seq":807,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.189612+0000","last_change":"2024-02-05T06:47:39.575482+0000","last_active":"2024-02-05T06:48:30.189612+0000","last_peered":"2024-02-05T06:48:30.189612+0000","last_clean":"2024-02-05T06:48:30.189612+0000","last_became_active":"2024-02-05T06:41:43.409664+0000","last_became_peered":"2024-02-05T06:41:43.409664+0000","last_unstale":"2024-02-05T06:48:30.189612+0000","last_undegraded":"2024-02-05T06:48:30.189612+0000","last_fullsized":"2024-02-05T06:48:30.189612+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'29","last_scrub_stamp":"2024-02-05T06:47:39.575331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:47:39.575331+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:55.789881+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":682,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.193668+0000","last_change":"2024-02-05T06:47:48.255664+0000","last_active":"2024-02-05T06:48:30.193668+0000","last_peered":"2024-02-05T06:48:30.193668+0000","last_clean":"2024-02-05T06:48:30.193668+0000","last_became_active":"2024-02-05T06:41:43.411550+0000","last_became_peered":"2024-02-05T06:41:43.411550+0000","last_unstale":"2024-02-05T06:48:30.193668+0000","last_undegraded":"2024-02-05T06:48:30.193668+0000","last_fullsized":"2024-02-05T06:48:30.193668+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:48.255364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:47:48.255364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:56.793069+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"369'5171","reported_seq":5228,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:31.630248+0000","last_change":"2024-02-05T06:47:24.185798+0000","last_active":"2024-02-05T06:48:31.630248+0000","last_peered":"2024-02-05T06:48:31.630248+0000","last_clean":"2024-02-05T06:48:31.630248+0000","last_became_active":"2024-02-05T06:47:24.185452+0000","last_became_peered":"2024-02-05T06:47:24.185452+0000","last_unstale":"2024-02-05T06:48:31.630248+0000","last_undegraded":"2024-02-05T06:48:31.630248+0000","last_fullsized":"2024-02-05T06:48:31.630248+0000","mapping_epoch":345,"log_start":"352'100","ondisk_log_start":"352'100","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":5071,"log_dups_size":0,"ondisk_log_size":5071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:43.544292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338886656,"num_objects":5171,"num_object_clones":0,"num_object_copies":15513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5171,"num_write_kb":330944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":866,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.198287+0000","last_change":"2024-02-05T06:48:25.726145+0000","last_active":"2024-02-05T06:48:30.198287+0000","last_peered":"2024-02-05T06:48:30.198287+0000","last_clean":"2024-02-05T06:48:30.198287+0000","last_became_active":"2024-02-05T06:40:30.085438+0000","last_became_peered":"2024-02-05T06:40:30.085438+0000","last_unstale":"2024-02-05T06:48:30.198287+0000","last_undegraded":"2024-02-05T06:48:30.198287+0000","last_fullsized":"2024-02-05T06:48:30.198287+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:48:25.725892+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:48:25.725892+0000","last_clean_scrub_stamp":"2024-02-05T06:48:25.725892+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:51.750841+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":644,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.188084+0000","last_change":"2024-02-05T06:45:48.925337+0000","last_active":"2024-02-05T06:48:30.188084+0000","last_peered":"2024-02-05T06:48:30.188084+0000","last_clean":"2024-02-05T06:48:30.188084+0000","last_became_active":"2024-02-05T06:33:35.600485+0000","last_became_peered":"2024-02-05T06:33:35.600485+0000","last_unstale":"2024-02-05T06:48:30.188084+0000","last_undegraded":"2024-02-05T06:48:30.188084+0000","last_fullsized":"2024-02-05T06:48:30.188084+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:48.925218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:45:48.925218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":279,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.554802+0000","last_change":"2024-02-05T06:48:30.554838+0000","last_active":"2024-02-05T06:48:30.554802+0000","last_peered":"2024-02-05T06:48:30.554802+0000","last_clean":"2024-02-05T06:48:30.554802+0000","last_became_active":"2024-02-05T06:41:43.411732+0000","last_became_peered":"2024-02-05T06:41:43.411732+0000","last_unstale":"2024-02-05T06:48:30.554802+0000","last_undegraded":"2024-02-05T06:48:30.554802+0000","last_fullsized":"2024-02-05T06:48:30.554802+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:30.554713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:28.353956+0000","last_clean_scrub_stamp":"2024-02-05T06:48:30.554713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:32.556146+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"295'20","reported_seq":769,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.190738+0000","last_change":"2024-02-05T06:48:21.064564+0000","last_active":"2024-02-05T06:48:30.190738+0000","last_peered":"2024-02-05T06:48:30.190738+0000","last_clean":"2024-02-05T06:48:30.190738+0000","last_became_active":"2024-02-05T06:41:43.421030+0000","last_became_peered":"2024-02-05T06:41:43.421030+0000","last_unstale":"2024-02-05T06:48:30.190738+0000","last_undegraded":"2024-02-05T06:48:30.190738+0000","last_fullsized":"2024-02-05T06:48:30.190738+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'20","last_scrub_stamp":"2024-02-05T06:48:21.064424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:48:21.064424+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:50.547913+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"335'11","reported_seq":866,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.189156+0000","last_change":"2024-02-05T06:46:52.427460+0000","last_active":"2024-02-05T06:48:30.189156+0000","last_peered":"2024-02-05T06:48:30.189156+0000","last_clean":"2024-02-05T06:48:30.189156+0000","last_became_active":"2024-02-05T06:41:43.423638+0000","last_became_peered":"2024-02-05T06:41:43.423638+0000","last_unstale":"2024-02-05T06:48:30.189156+0000","last_undegraded":"2024-02-05T06:48:30.189156+0000","last_fullsized":"2024-02-05T06:48:30.189156+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"335'11","last_scrub_stamp":"2024-02-05T06:46:52.423878+0000","last_deep_scrub":"335'11","last_deep_scrub_stamp":"2024-02-05T06:46:52.423878+0000","last_clean_scrub_stamp":"2024-02-05T06:46:52.423878+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"369'5329","reported_seq":5387,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:31.553728+0000","last_change":"2024-02-05T06:47:24.175068+0000","last_active":"2024-02-05T06:48:31.553728+0000","last_peered":"2024-02-05T06:48:31.553728+0000","last_clean":"2024-02-05T06:48:31.553728+0000","last_became_active":"2024-02-05T06:47:24.174574+0000","last_became_peered":"2024-02-05T06:47:24.174574+0000","last_unstale":"2024-02-05T06:48:31.553728+0000","last_undegraded":"2024-02-05T06:48:31.553728+0000","last_fullsized":"2024-02-05T06:48:31.553728+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":5329,"log_dups_size":0,"ondisk_log_size":5329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349241344,"num_objects":5329,"num_object_clones":0,"num_object_copies":15987,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5329,"num_write_kb":341056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":861,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.198478+0000","last_change":"2024-02-05T06:45:54.266666+0000","last_active":"2024-02-05T06:48:30.198478+0000","last_peered":"2024-02-05T06:48:30.198478+0000","last_clean":"2024-02-05T06:48:30.198478+0000","last_became_active":"2024-02-05T06:41:46.053837+0000","last_became_peered":"2024-02-05T06:41:46.053837+0000","last_unstale":"2024-02-05T06:48:30.198478+0000","last_undegraded":"2024-02-05T06:48:30.198478+0000","last_fullsized":"2024-02-05T06:48:30.198478+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:45:54.266536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:45:54.266536+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":363,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.190846+0000","last_change":"2024-02-05T06:47:45.759776+0000","last_active":"2024-02-05T06:48:30.190846+0000","last_peered":"2024-02-05T06:48:30.190846+0000","last_clean":"2024-02-05T06:48:30.190846+0000","last_became_active":"2024-02-05T06:40:27.099168+0000","last_became_peered":"2024-02-05T06:40:27.099168+0000","last_unstale":"2024-02-05T06:48:30.190846+0000","last_undegraded":"2024-02-05T06:48:30.190846+0000","last_fullsized":"2024-02-05T06:48:30.190846+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:45.759651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:47:45.759651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:48.053819+0000","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"95'2","reported_seq":685,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.189027+0000","last_change":"2024-02-05T06:48:25.529812+0000","last_active":"2024-02-05T06:48:30.189027+0000","last_peered":"2024-02-05T06:48:30.189027+0000","last_clean":"2024-02-05T06:48:30.189027+0000","last_became_active":"2024-02-05T06:36:54.975093+0000","last_became_peered":"2024-02-05T06:36:54.975093+0000","last_unstale":"2024-02-05T06:48:30.189027+0000","last_undegraded":"2024-02-05T06:48:30.189027+0000","last_fullsized":"2024-02-05T06:48:30.189027+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-05T06:48:25.529692+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-05T06:35:25.616708+0000","last_clean_scrub_stamp":"2024-02-05T06:48:25.529692+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:35.785150+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"295'28","reported_seq":589,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.202977+0000","last_change":"2024-02-05T06:48:17.616455+0000","last_active":"2024-02-05T06:48:30.202977+0000","last_peered":"2024-02-05T06:48:30.202977+0000","last_clean":"2024-02-05T06:48:30.202977+0000","last_became_active":"2024-02-05T06:45:09.764696+0000","last_became_peered":"2024-02-05T06:45:09.764696+0000","last_unstale":"2024-02-05T06:48:30.202977+0000","last_undegraded":"2024-02-05T06:48:30.202977+0000","last_fullsized":"2024-02-05T06:48:30.202977+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'28","last_scrub_stamp":"2024-02-05T06:48:17.615783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:48:17.615783+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:46.705649+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":40,"num_read_kb":33,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"326'11","reported_seq":859,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.192587+0000","last_change":"2024-02-05T06:45:56.212330+0000","last_active":"2024-02-05T06:48:30.192587+0000","last_peered":"2024-02-05T06:48:30.192587+0000","last_clean":"2024-02-05T06:48:30.192587+0000","last_became_active":"2024-02-05T06:36:54.954819+0000","last_became_peered":"2024-02-05T06:36:54.954819+0000","last_unstale":"2024-02-05T06:48:30.192587+0000","last_undegraded":"2024-02-05T06:48:30.192587+0000","last_fullsized":"2024-02-05T06:48:30.192587+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'11","last_scrub_stamp":"2024-02-05T06:45:56.212177+0000","last_deep_scrub":"326'11","last_deep_scrub_stamp":"2024-02-05T06:45:56.212177+0000","last_clean_scrub_stamp":"2024-02-05T06:45:56.212177+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"369'5342","reported_seq":5400,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:31.561925+0000","last_change":"2024-02-05T06:47:24.184945+0000","last_active":"2024-02-05T06:48:31.561925+0000","last_peered":"2024-02-05T06:48:31.561925+0000","last_clean":"2024-02-05T06:48:31.561925+0000","last_became_active":"2024-02-05T06:47:24.184494+0000","last_became_peered":"2024-02-05T06:47:24.184494+0000","last_unstale":"2024-02-05T06:48:31.561925+0000","last_undegraded":"2024-02-05T06:48:31.561925+0000","last_fullsized":"2024-02-05T06:48:31.561925+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":5342,"log_dups_size":0,"ondisk_log_size":5342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:51.789612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350093312,"num_objects":5342,"num_object_clones":0,"num_object_copies":16026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5342,"num_write_kb":341888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":428,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.200349+0000","last_change":"2024-02-05T06:47:46.034108+0000","last_active":"2024-02-05T06:48:30.200349+0000","last_peered":"2024-02-05T06:48:30.200349+0000","last_clean":"2024-02-05T06:48:30.200349+0000","last_became_active":"2024-02-05T06:45:09.776631+0000","last_became_peered":"2024-02-05T06:45:09.776631+0000","last_unstale":"2024-02-05T06:48:30.200349+0000","last_undegraded":"2024-02-05T06:48:30.200349+0000","last_fullsized":"2024-02-05T06:48:30.200349+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:47:46.033702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:47:46.033702+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:00.978347+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":185,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.192835+0000","last_change":"2024-02-05T06:45:49.145234+0000","last_active":"2024-02-05T06:48:30.192835+0000","last_peered":"2024-02-05T06:48:30.192835+0000","last_clean":"2024-02-05T06:48:30.192835+0000","last_became_active":"2024-02-05T06:45:09.780390+0000","last_became_peered":"2024-02-05T06:45:09.780390+0000","last_unstale":"2024-02-05T06:48:30.192835+0000","last_undegraded":"2024-02-05T06:48:30.192835+0000","last_fullsized":"2024-02-05T06:48:30.192835+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:49.145117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:45:49.145117+0000","last_clean_scrub_stamp":"2024-02-05T06:45:49.145117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":672,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.203139+0000","last_change":"2024-02-05T06:45:42.080964+0000","last_active":"2024-02-05T06:48:30.203139+0000","last_peered":"2024-02-05T06:48:30.203139+0000","last_clean":"2024-02-05T06:48:30.203139+0000","last_became_active":"2024-02-05T06:40:27.093739+0000","last_became_peered":"2024-02-05T06:40:27.093739+0000","last_unstale":"2024-02-05T06:48:30.203139+0000","last_undegraded":"2024-02-05T06:48:30.203139+0000","last_fullsized":"2024-02-05T06:48:30.203139+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:42.080862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:38:28.150695+0000","last_clean_scrub_stamp":"2024-02-05T06:45:42.080862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"295'30","reported_seq":749,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.192490+0000","last_change":"2024-02-05T06:47:50.402454+0000","last_active":"2024-02-05T06:48:30.192490+0000","last_peered":"2024-02-05T06:48:30.192490+0000","last_clean":"2024-02-05T06:48:30.192490+0000","last_became_active":"2024-02-05T06:40:27.277747+0000","last_became_peered":"2024-02-05T06:40:27.277747+0000","last_unstale":"2024-02-05T06:48:30.192490+0000","last_undegraded":"2024-02-05T06:48:30.192490+0000","last_fullsized":"2024-02-05T06:48:30.192490+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'30","last_scrub_stamp":"2024-02-05T06:47:50.402305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:47:50.402305+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:08.001287+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":326,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.193753+0000","last_change":"2024-02-05T06:46:30.377076+0000","last_active":"2024-02-05T06:48:30.193753+0000","last_peered":"2024-02-05T06:48:30.193753+0000","last_clean":"2024-02-05T06:48:30.193753+0000","last_became_active":"2024-02-05T06:45:09.778609+0000","last_became_peered":"2024-02-05T06:45:09.778609+0000","last_unstale":"2024-02-05T06:48:30.193753+0000","last_undegraded":"2024-02-05T06:48:30.193753+0000","last_fullsized":"2024-02-05T06:48:30.193753+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:30.376861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:46:30.376861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1","version":"369'5302","reported_seq":5359,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:31.664363+0000","last_change":"2024-02-05T06:47:24.182652+0000","last_active":"2024-02-05T06:48:31.664363+0000","last_peered":"2024-02-05T06:48:31.664363+0000","last_clean":"2024-02-05T06:48:31.664363+0000","last_became_active":"2024-02-05T06:47:24.182190+0000","last_became_peered":"2024-02-05T06:47:24.182190+0000","last_unstale":"2024-02-05T06:48:31.664363+0000","last_undegraded":"2024-02-05T06:48:31.664363+0000","last_fullsized":"2024-02-05T06:48:31.664363+0000","mapping_epoch":345,"log_start":"353'300","ondisk_log_start":"353'300","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":5002,"log_dups_size":0,"ondisk_log_size":5002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:50.966086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347471872,"num_objects":5302,"num_object_clones":0,"num_object_copies":15906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5302,"num_write_kb":339328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":698,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.203559+0000","last_change":"2024-02-05T06:48:24.641478+0000","last_active":"2024-02-05T06:48:30.203559+0000","last_peered":"2024-02-05T06:48:30.203559+0000","last_clean":"2024-02-05T06:48:30.203559+0000","last_became_active":"2024-02-05T06:41:46.053993+0000","last_became_peered":"2024-02-05T06:41:46.053993+0000","last_unstale":"2024-02-05T06:48:30.203559+0000","last_undegraded":"2024-02-05T06:48:30.203559+0000","last_fullsized":"2024-02-05T06:48:30.203559+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:48:24.641369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:48:24.641369+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:32.985701+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":279,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.203513+0000","last_change":"2024-02-05T06:46:45.769966+0000","last_active":"2024-02-05T06:48:30.203513+0000","last_peered":"2024-02-05T06:48:30.203513+0000","last_clean":"2024-02-05T06:48:30.203513+0000","last_became_active":"2024-02-05T06:41:43.418066+0000","last_became_peered":"2024-02-05T06:41:43.418066+0000","last_unstale":"2024-02-05T06:48:30.203513+0000","last_undegraded":"2024-02-05T06:48:30.203513+0000","last_fullsized":"2024-02-05T06:48:30.203513+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:45.769853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:45.769853+0000","last_clean_scrub_stamp":"2024-02-05T06:46:45.769853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"319'13","reported_seq":518,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.205113+0000","last_change":"2024-02-05T06:47:46.597798+0000","last_active":"2024-02-05T06:48:30.205113+0000","last_peered":"2024-02-05T06:48:30.205113+0000","last_clean":"2024-02-05T06:48:30.205113+0000","last_became_active":"2024-02-05T06:45:09.766220+0000","last_became_peered":"2024-02-05T06:45:09.766220+0000","last_unstale":"2024-02-05T06:48:30.205113+0000","last_undegraded":"2024-02-05T06:48:30.205113+0000","last_fullsized":"2024-02-05T06:48:30.205113+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'13","last_scrub_stamp":"2024-02-05T06:47:46.597456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:47:46.597456+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:57.094687+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":674,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.192406+0000","last_change":"2024-02-05T06:48:14.631495+0000","last_active":"2024-02-05T06:48:30.192406+0000","last_peered":"2024-02-05T06:48:30.192406+0000","last_clean":"2024-02-05T06:48:30.192406+0000","last_became_active":"2024-02-05T06:40:27.213439+0000","last_became_peered":"2024-02-05T06:40:27.213439+0000","last_unstale":"2024-02-05T06:48:30.192406+0000","last_undegraded":"2024-02-05T06:48:30.192406+0000","last_fullsized":"2024-02-05T06:48:30.192406+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:14.631382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:52.482566+0000","last_clean_scrub_stamp":"2024-02-05T06:48:14.631382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:29.843880+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"295'34","reported_seq":470,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.188228+0000","last_change":"2024-02-05T06:46:44.355699+0000","last_active":"2024-02-05T06:48:30.188228+0000","last_peered":"2024-02-05T06:48:30.188228+0000","last_clean":"2024-02-05T06:48:30.188228+0000","last_became_active":"2024-02-05T06:39:13.257978+0000","last_became_peered":"2024-02-05T06:39:13.257978+0000","last_unstale":"2024-02-05T06:48:30.188228+0000","last_undegraded":"2024-02-05T06:48:30.188228+0000","last_fullsized":"2024-02-05T06:48:30.188228+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'34","last_scrub_stamp":"2024-02-05T06:46:44.355563+0000","last_deep_scrub":"295'34","last_deep_scrub_stamp":"2024-02-05T06:46:44.355563+0000","last_clean_scrub_stamp":"2024-02-05T06:46:44.355563+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.0","version":"369'5162","reported_seq":5220,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:31.522865+0000","last_change":"2024-02-05T06:47:24.183477+0000","last_active":"2024-02-05T06:48:31.522865+0000","last_peered":"2024-02-05T06:48:31.522865+0000","last_clean":"2024-02-05T06:48:31.522865+0000","last_became_active":"2024-02-05T06:47:24.183098+0000","last_became_peered":"2024-02-05T06:47:24.183098+0000","last_unstale":"2024-02-05T06:48:31.522865+0000","last_undegraded":"2024-02-05T06:48:31.522865+0000","last_fullsized":"2024-02-05T06:48:31.522865+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":5162,"log_dups_size":0,"ondisk_log_size":5162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:34.968232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338296832,"num_objects":5162,"num_object_clones":0,"num_object_copies":15486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5162,"num_write_kb":330368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":660,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.198640+0000","last_change":"2024-02-05T06:46:58.479329+0000","last_active":"2024-02-05T06:48:30.198640+0000","last_peered":"2024-02-05T06:48:30.198640+0000","last_clean":"2024-02-05T06:48:30.198640+0000","last_became_active":"2024-02-05T06:41:43.410600+0000","last_became_peered":"2024-02-05T06:41:43.410600+0000","last_unstale":"2024-02-05T06:48:30.198640+0000","last_undegraded":"2024-02-05T06:48:30.198640+0000","last_fullsized":"2024-02-05T06:48:30.198640+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:58.479208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:58.479208+0000","last_clean_scrub_stamp":"2024-02-05T06:46:58.479208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":279,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.209687+0000","last_change":"2024-02-05T06:46:57.669791+0000","last_active":"2024-02-05T06:48:30.209687+0000","last_peered":"2024-02-05T06:48:30.209687+0000","last_clean":"2024-02-05T06:48:30.209687+0000","last_became_active":"2024-02-05T06:41:43.417343+0000","last_became_peered":"2024-02-05T06:41:43.417343+0000","last_unstale":"2024-02-05T06:48:30.209687+0000","last_undegraded":"2024-02-05T06:48:30.209687+0000","last_fullsized":"2024-02-05T06:48:30.209687+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:57.669673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:42:55.610276+0000","last_clean_scrub_stamp":"2024-02-05T06:46:57.669673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":382,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.186229+0000","last_change":"2024-02-05T06:46:40.785712+0000","last_active":"2024-02-05T06:48:30.186229+0000","last_peered":"2024-02-05T06:48:30.186229+0000","last_clean":"2024-02-05T06:48:30.186229+0000","last_became_active":"2024-02-05T06:40:27.086808+0000","last_became_peered":"2024-02-05T06:40:27.086808+0000","last_unstale":"2024-02-05T06:48:30.186229+0000","last_undegraded":"2024-02-05T06:48:30.186229+0000","last_fullsized":"2024-02-05T06:48:30.186229+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:40.785606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:46:40.785606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"295'24","reported_seq":524,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.194483+0000","last_change":"2024-02-05T06:46:58.206860+0000","last_active":"2024-02-05T06:48:30.194483+0000","last_peered":"2024-02-05T06:48:30.194483+0000","last_clean":"2024-02-05T06:48:30.194483+0000","last_became_active":"2024-02-05T06:45:09.774868+0000","last_became_peered":"2024-02-05T06:45:09.774868+0000","last_unstale":"2024-02-05T06:48:30.194483+0000","last_undegraded":"2024-02-05T06:48:30.194483+0000","last_fullsized":"2024-02-05T06:48:30.194483+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:46:58.206720+0000","last_deep_scrub":"115'16","last_deep_scrub_stamp":"2024-02-05T06:36:34.793243+0000","last_clean_scrub_stamp":"2024-02-05T06:46:58.206720+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3","version":"369'5445","reported_seq":5503,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:31.839177+0000","last_change":"2024-02-05T06:47:24.177225+0000","last_active":"2024-02-05T06:48:31.839177+0000","last_peered":"2024-02-05T06:48:31.839177+0000","last_clean":"2024-02-05T06:48:31.839177+0000","last_became_active":"2024-02-05T06:47:24.176139+0000","last_became_peered":"2024-02-05T06:47:24.176139+0000","last_unstale":"2024-02-05T06:48:31.839177+0000","last_undegraded":"2024-02-05T06:48:31.839177+0000","last_fullsized":"2024-02-05T06:48:31.839177+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":5445,"log_dups_size":0,"ondisk_log_size":5445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:43.822725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":356843520,"num_objects":5445,"num_object_clones":0,"num_object_copies":16335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5445,"num_write_kb":348480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"83'128","reported_seq":694,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.200802+0000","last_change":"2024-02-05T06:46:44.625857+0000","last_active":"2024-02-05T06:48:30.200802+0000","last_peered":"2024-02-05T06:48:30.200802+0000","last_clean":"2024-02-05T06:48:30.200802+0000","last_became_active":"2024-02-05T06:41:46.053324+0000","last_became_peered":"2024-02-05T06:41:46.053324+0000","last_unstale":"2024-02-05T06:48:30.200802+0000","last_undegraded":"2024-02-05T06:48:30.200802+0000","last_fullsized":"2024-02-05T06:48:30.200802+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:46:44.625739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:46:44.625739+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":685,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.201053+0000","last_change":"2024-02-05T06:47:42.155963+0000","last_active":"2024-02-05T06:48:30.201053+0000","last_peered":"2024-02-05T06:48:30.201053+0000","last_clean":"2024-02-05T06:48:30.201053+0000","last_became_active":"2024-02-05T06:36:54.958805+0000","last_became_peered":"2024-02-05T06:36:54.958805+0000","last_unstale":"2024-02-05T06:48:30.201053+0000","last_undegraded":"2024-02-05T06:48:30.201053+0000","last_fullsized":"2024-02-05T06:48:30.201053+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:42.155801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:47:42.155801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:53.870607+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":674,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.190512+0000","last_change":"2024-02-05T06:46:53.791434+0000","last_active":"2024-02-05T06:48:30.190512+0000","last_peered":"2024-02-05T06:48:30.190512+0000","last_clean":"2024-02-05T06:48:30.190512+0000","last_became_active":"2024-02-05T06:36:54.958082+0000","last_became_peered":"2024-02-05T06:36:54.958082+0000","last_unstale":"2024-02-05T06:48:30.190512+0000","last_undegraded":"2024-02-05T06:48:30.190512+0000","last_fullsized":"2024-02-05T06:48:30.190512+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:46:53.791322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:46:53.791322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":695,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.193718+0000","last_change":"2024-02-05T06:48:27.784796+0000","last_active":"2024-02-05T06:48:30.193718+0000","last_peered":"2024-02-05T06:48:30.193718+0000","last_clean":"2024-02-05T06:48:30.193718+0000","last_became_active":"2024-02-05T06:39:13.240038+0000","last_became_peered":"2024-02-05T06:39:13.240038+0000","last_unstale":"2024-02-05T06:48:30.193718+0000","last_undegraded":"2024-02-05T06:48:30.193718+0000","last_fullsized":"2024-02-05T06:48:30.193718+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:27.784677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:48:27.784677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:57.144027+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"295'39","reported_seq":539,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.200506+0000","last_change":"2024-02-05T06:47:41.020322+0000","last_active":"2024-02-05T06:48:30.200506+0000","last_peered":"2024-02-05T06:48:30.200506+0000","last_clean":"2024-02-05T06:48:30.200506+0000","last_became_active":"2024-02-05T06:45:09.782645+0000","last_became_peered":"2024-02-05T06:45:09.782645+0000","last_unstale":"2024-02-05T06:48:30.200506+0000","last_undegraded":"2024-02-05T06:48:30.200506+0000","last_fullsized":"2024-02-05T06:48:30.200506+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'39","last_scrub_stamp":"2024-02-05T06:47:41.020104+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-02-05T06:35:26.274944+0000","last_clean_scrub_stamp":"2024-02-05T06:47:41.020104+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:06.072004+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"369'5289","reported_seq":5347,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:32.077500+0000","last_change":"2024-02-05T06:47:24.182784+0000","last_active":"2024-02-05T06:48:32.077500+0000","last_peered":"2024-02-05T06:48:32.077500+0000","last_clean":"2024-02-05T06:48:32.077500+0000","last_became_active":"2024-02-05T06:47:24.182360+0000","last_became_peered":"2024-02-05T06:47:24.182360+0000","last_unstale":"2024-02-05T06:48:32.077500+0000","last_undegraded":"2024-02-05T06:48:32.077500+0000","last_fullsized":"2024-02-05T06:48:32.077500+0000","mapping_epoch":345,"log_start":"352'200","ondisk_log_start":"352'200","created":345,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:23.158718+0000","last_clean_scrub_stamp":"2024-02-05T06:47:23.158718+0000","objects_scrubbed":0,"log_size":5089,"log_dups_size":0,"ondisk_log_size":5089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346619904,"num_objects":5289,"num_object_clones":0,"num_object_copies":15867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5289,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5289,"num_write_kb":338496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":781,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.189832+0000","last_change":"2024-02-05T06:48:17.676758+0000","last_active":"2024-02-05T06:48:30.189832+0000","last_peered":"2024-02-05T06:48:30.189832+0000","last_clean":"2024-02-05T06:48:30.189832+0000","last_became_active":"2024-02-05T06:40:30.095202+0000","last_became_peered":"2024-02-05T06:40:30.095202+0000","last_unstale":"2024-02-05T06:48:30.189832+0000","last_undegraded":"2024-02-05T06:48:30.189832+0000","last_fullsized":"2024-02-05T06:48:30.189832+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:48:17.676587+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:41:50.706159+0000","last_clean_scrub_stamp":"2024-02-05T06:48:17.676587+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:29.131120+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":193,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.197040+0000","last_change":"2024-02-05T06:47:49.056694+0000","last_active":"2024-02-05T06:48:30.197040+0000","last_peered":"2024-02-05T06:48:30.197040+0000","last_clean":"2024-02-05T06:48:30.197040+0000","last_became_active":"2024-02-05T06:45:09.784229+0000","last_became_peered":"2024-02-05T06:45:09.784229+0000","last_unstale":"2024-02-05T06:48:30.197040+0000","last_undegraded":"2024-02-05T06:48:30.197040+0000","last_fullsized":"2024-02-05T06:48:30.197040+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:49.056561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:30.057099+0000","last_clean_scrub_stamp":"2024-02-05T06:47:49.056561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:49:05.788674+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"249'13","reported_seq":906,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.200649+0000","last_change":"2024-02-05T06:47:52.101405+0000","last_active":"2024-02-05T06:48:30.200649+0000","last_peered":"2024-02-05T06:48:30.200649+0000","last_clean":"2024-02-05T06:48:30.200649+0000","last_became_active":"2024-02-05T06:41:43.413139+0000","last_became_peered":"2024-02-05T06:41:43.413139+0000","last_unstale":"2024-02-05T06:48:30.200649+0000","last_undegraded":"2024-02-05T06:48:30.200649+0000","last_fullsized":"2024-02-05T06:48:30.200649+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'13","last_scrub_stamp":"2024-02-05T06:47:52.101279+0000","last_deep_scrub":"92'9","last_deep_scrub_stamp":"2024-02-05T06:34:52.283780+0000","last_clean_scrub_stamp":"2024-02-05T06:47:52.101279+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:48:56.214584+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":666,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.192877+0000","last_change":"2024-02-05T06:45:30.989035+0000","last_active":"2024-02-05T06:48:30.192877+0000","last_peered":"2024-02-05T06:48:30.192877+0000","last_clean":"2024-02-05T06:48:30.192877+0000","last_became_active":"2024-02-05T06:36:54.971304+0000","last_became_peered":"2024-02-05T06:36:54.971304+0000","last_unstale":"2024-02-05T06:48:30.192877+0000","last_undegraded":"2024-02-05T06:48:30.192877+0000","last_fullsized":"2024-02-05T06:48:30.192877+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:45:30.987881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:45:30.987881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"295'25","reported_seq":519,"reported_epoch":369,"state":"active+clean","last_fresh":"2024-02-05T06:48:30.198902+0000","last_change":"2024-02-05T06:45:09.799601+0000","last_active":"2024-02-05T06:48:30.198902+0000","last_peered":"2024-02-05T06:48:30.198902+0000","last_clean":"2024-02-05T06:48:30.198902+0000","last_became_active":"2024-02-05T06:45:09.796902+0000","last_became_peered":"2024-02-05T06:45:09.796902+0000","last_unstale":"2024-02-05T06:48:30.198902+0000","last_undegraded":"2024-02-05T06:48:30.198902+0000","last_fullsized":"2024-02-05T06:48:30.198902+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'25","last_scrub_stamp":"2024-02-05T06:44:50.081252+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-02-05T06:35:04.584657+0000","last_clean_scrub_stamp":"2024-02-05T06:44:50.081252+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":5573378048,"num_objects":85043,"num_object_clones":0,"num_object_copies":255129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":85043,"num_write_kb":5442752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16762667008,"data_stored":16762667008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83943,"ondisk_log_size":83943,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":490,"num_read_kb":485,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1573408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1106,"num_read_kb":863,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":373,"num_bytes_recovered":8234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":771,"ondisk_log_size":771,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103352,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1883496,"kb_used_data":1342812,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":91861656,"statfs":{"total":95995035648,"available":94066335744,"internally_reserved":0,"allocated":1375039488,"data_stored":1374516292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":553646538},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":1.135,"15min":1.476},"min":{"1min":0.40000000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":1.2909999999999999,"5min":4.1100000000000003,"15min":68.921000000000006},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.63700000000000001,"5min":1.2569999999999999,"15min":1.593},"min":{"1min":0.27300000000000002,"5min":0.222,"15min":0.222},"max":{"1min":1.403,"5min":18.013000000000002,"15min":69.616},"last":0.44700000000000001}]},{"osd":1,"last update":"Mon Feb 5 06:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":2.3919999999999999,"15min":1.7789999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.23599999999999999},"max":{"1min":1.504,"5min":84.216999999999999,"15min":89.231999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.66000000000000003,"5min":2.2480000000000002,"15min":1.879},"min":{"1min":0.34699999999999998,"5min":0.30399999999999999,"15min":0.26100000000000001},"max":{"1min":1.3540000000000001,"5min":84.126999999999995,"15min":87.995999999999995},"last":0.47799999999999998}]},{"osd":2,"last update":"Mon Feb 5 06:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":1.214,"15min":1.2330000000000001},"min":{"1min":0.32300000000000001,"5min":0.30299999999999999,"15min":0.26200000000000001},"max":{"1min":1.994,"5min":18.071999999999999,"15min":53.947000000000003},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":1.2310000000000001,"15min":1.206},"min":{"1min":0.39500000000000002,"5min":0.29999999999999999,"15min":0.26800000000000002},"max":{"1min":0.92800000000000005,"5min":18.045999999999999,"15min":53.299999999999997},"last":1.0069999999999999}]},{"osd":3,"last update":"Mon Feb 5 06:47:57 2024","interfaces":[{"interface":"back","average":{"1min":4.6980000000000004,"5min":2.371,"15min":2.165},"min":{"1min":0.35699999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":73.283000000000001,"5min":73.283000000000001,"15min":89.481999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":4.7990000000000004,"5min":2.4220000000000002,"15min":2.1259999999999999},"min":{"1min":0.34100000000000003,"5min":0.32900000000000001,"15min":0.27400000000000002},"max":{"1min":73.323999999999998,"5min":73.323999999999998,"15min":92.293999999999997},"last":1.387}]},{"osd":4,"last update":"Mon Feb 5 06:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.81299999999999994,"15min":0.61499999999999999},"min":{"1min":0.29899999999999999,"5min":0.251,"15min":0.24199999999999999},"max":{"1min":1.329,"5min":4.1059999999999999,"15min":4.1059999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.751,"5min":0.82299999999999995,"15min":0.60499999999999998},"min":{"1min":0.35599999999999998,"5min":0.29299999999999998,"15min":0.23400000000000001},"max":{"1min":2.7429999999999999,"5min":3.0819999999999999,"15min":3.0819999999999999},"last":0.34799999999999998}]},{"osd":5,"last update":"Mon Feb 5 06:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":1.083,"15min":0.72699999999999998},"min":{"1min":0.33700000000000002,"5min":0.251,"15min":0.251},"max":{"1min":1.276,"5min":6.2190000000000003,"15min":6.2190000000000003},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.71099999999999997,"5min":1.2769999999999999,"15min":0.80700000000000005},"min":{"1min":0.38,"5min":0.313,"15min":0.252},"max":{"1min":1.292,"5min":7.8360000000000003,"15min":7.8360000000000003},"last":0.58099999999999996}]},{"osd":6,"last update":"Mon Feb 5 06:47:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":1.01,"15min":0.76300000000000001},"min":{"1min":0.43099999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.379,"5min":4.524,"15min":4.524},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.996,"15min":0.71999999999999997},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.245},"max":{"1min":2.7869999999999999,"5min":3.3769999999999998,"15min":3.3769999999999998},"last":0.68600000000000005}]}]},{"osd":6,"up_from":49,"seq":210453397694,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2765904,"kb_used_data":2365444,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":90979248,"statfs":{"total":95995035648,"available":93162749952,"internally_reserved":0,"allocated":2422214656,"data_stored":2421714587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":410057158},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:48:20 2024","interfaces":[{"interface":"back","average":{"1min":1.855,"5min":1.0640000000000001,"15min":2.2210000000000001},"min":{"1min":0.33900000000000002,"5min":0.22900000000000001,"15min":0.224},"max":{"1min":18.393000000000001,"5min":18.393000000000001,"15min":98.918999999999997},"last":0.72899999999999998},{"interface":"front","average":{"1min":2.0899999999999999,"5min":1.052,"15min":2.1800000000000002},"min":{"1min":0.34599999999999997,"5min":0.245,"15min":0.23000000000000001},"max":{"1min":17.995999999999999,"5min":17.995999999999999,"15min":98.775000000000006},"last":1.6200000000000001}]},{"osd":1,"last update":"Mon Feb 5 06:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.96399999999999997,"15min":1.9199999999999999},"min":{"1min":0.27400000000000002,"5min":0.249,"15min":0.24099999999999999},"max":{"1min":2.363,"5min":5.2750000000000004,"15min":98.843999999999994},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.93500000000000005,"15min":1.899},"min":{"1min":0.32800000000000001,"5min":0.26500000000000001,"15min":0.23100000000000001},"max":{"1min":4.516,"5min":4.516,"15min":101.637},"last":0.40799999999999997}]},{"osd":2,"last update":"Mon Feb 5 06:48:20 2024","interfaces":[{"interface":"back","average":{"1min":1.919,"5min":1.131,"15min":1.905},"min":{"1min":0.377,"5min":0.26900000000000002,"15min":0.22500000000000001},"max":{"1min":16.131,"5min":16.131,"15min":85.617000000000004},"last":1.575},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.0620000000000001,"15min":1.794},"min":{"1min":0.42099999999999999,"5min":0.29599999999999999,"15min":0.27000000000000002},"max":{"1min":16.210000000000001,"5min":16.210000000000001,"15min":85.335999999999999},"last":0.92700000000000005}]},{"osd":3,"last update":"Mon Feb 5 06:48:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.1160000000000001,"15min":2.379},"min":{"1min":0.315,"5min":0.313,"15min":0.27000000000000002},"max":{"1min":3.6360000000000001,"5min":6.2720000000000002,"15min":99.269000000000005},"last":0.44400000000000001},{"interface":"front","average":{"1min":1.0009999999999999,"5min":0.997,"15min":2.4369999999999998},"min":{"1min":0.29599999999999999,"5min":0.28799999999999998,"15min":0.27800000000000002},"max":{"1min":3.8940000000000001,"5min":7.4210000000000003,"15min":101.75700000000001},"last":0.876}]},{"osd":4,"last update":"Mon Feb 5 06:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.69299999999999995,"15min":0.56499999999999995},"min":{"1min":0.33400000000000002,"5min":0.249,"15min":0.23100000000000001},"max":{"1min":3.444,"5min":3.444,"15min":3.742},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.69999999999999996,"15min":0.55900000000000005},"min":{"1min":0.34899999999999998,"5min":0.249,"15min":0.24299999999999999},"max":{"1min":2.319,"5min":4.4379999999999997,"15min":4.4379999999999997},"last":2.028}]},{"osd":5,"last update":"Mon Feb 5 06:48:20 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.298,"15min":0.79000000000000004},"min":{"1min":0.373,"5min":0.313,"15min":0.251},"max":{"1min":3.4079999999999999,"5min":36.548999999999999,"15min":36.548999999999999},"last":1.5249999999999999},{"interface":"front","average":{"1min":1.077,"5min":1.3979999999999999,"15min":0.82499999999999996},"min":{"1min":0.34599999999999997,"5min":0.309,"15min":0.24299999999999999},"max":{"1min":4.6269999999999998,"5min":36.433,"15min":36.433},"last":0.495}]},{"osd":7,"last update":"Mon Feb 5 06:47:38 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":1.1870000000000001,"15min":0.76000000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.30599999999999999},"max":{"1min":1.278,"5min":38.436999999999998,"15min":38.436999999999998},"last":1.2030000000000001},{"interface":"front","average":{"1min":0.70999999999999996,"5min":1.117,"15min":0.77800000000000002},"min":{"1min":0.378,"5min":0.33000000000000002,"15min":0.29699999999999999},"max":{"1min":2.2850000000000001,"5min":36.633000000000003,"15min":36.633000000000003},"last":0.80500000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575063,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1559944,"kb_used_data":1001212,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":92185208,"statfs":{"total":95995035648,"available":94397652992,"internally_reserved":0,"allocated":1025241088,"data_stored":1024685907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:48:23 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.153,"15min":1.8029999999999999},"min":{"1min":0.26200000000000001,"5min":0.218,"15min":0.16700000000000001},"max":{"1min":3.5049999999999999,"5min":7.4370000000000003,"15min":67.421000000000006},"last":1.464},{"interface":"front","average":{"1min":1.2,"5min":1.2470000000000001,"15min":1.9370000000000001},"min":{"1min":0.224,"5min":0.20300000000000001,"15min":0.19600000000000001},"max":{"1min":3.8479999999999999,"5min":10.333,"15min":67.885000000000005},"last":1.4359999999999999}]},{"osd":2,"last update":"Mon Feb 5 06:47:38 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":1.0569999999999999,"15min":1.452},"min":{"1min":0.255,"5min":0.249,"15min":0.21099999999999999},"max":{"1min":5.4969999999999999,"5min":17.422000000000001,"15min":43.447000000000003},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.442,"5min":1.081,"15min":1.429},"min":{"1min":0.22700000000000001,"5min":0.22600000000000001,"15min":0.20899999999999999},"max":{"1min":1.0489999999999999,"5min":22.050000000000001,"15min":22.050000000000001},"last":1.8600000000000001}]},{"osd":3,"last update":"Mon Feb 5 06:48:09 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":1.417,"15min":2.157},"min":{"1min":0.27400000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":2.3359999999999999,"5min":10.456,"15min":69.798000000000002},"last":1.252},{"interface":"front","average":{"1min":0.69699999999999995,"5min":1.2310000000000001,"15min":1.8680000000000001},"min":{"1min":0.31,"5min":0.27400000000000002,"15min":0.23699999999999999},"max":{"1min":2.056,"5min":5.8520000000000003,"15min":70.765000000000001},"last":0.89200000000000002}]},{"osd":4,"last update":"Mon Feb 5 06:48:28 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.95299999999999996,"15min":1.1200000000000001},"min":{"1min":0.374,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":3.5209999999999999,"5min":6.3360000000000003,"15min":11.679},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.353,"15min":1.2470000000000001},"min":{"1min":0.33800000000000002,"5min":0.30599999999999999,"15min":0.26300000000000001},"max":{"1min":3.363,"5min":10.419,"15min":10.419},"last":1.1870000000000001}]},{"osd":5,"last update":"Mon Feb 5 06:48:04 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":1.163,"15min":1.232},"min":{"1min":0.44900000000000001,"5min":0.32900000000000001,"15min":0.29299999999999998},"max":{"1min":1.4570000000000001,"5min":5.5830000000000002,"15min":43.414999999999999},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.80000000000000004,"5min":1.1919999999999999,"15min":1.254},"min":{"1min":0.36199999999999999,"5min":0.35699999999999998,"15min":0.33100000000000002},"max":{"1min":2.7669999999999999,"5min":6.3849999999999998,"15min":13.455},"last":1.079}]},{"osd":6,"last update":"Mon Feb 5 06:48:24 2024","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.1479999999999999,"15min":1.3160000000000001},"min":{"1min":0.434,"5min":0.376,"15min":0.34799999999999998},"max":{"1min":3.5550000000000002,"5min":23.902999999999999,"15min":23.902999999999999},"last":1.04},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.155,"15min":1.266},"min":{"1min":0.49099999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":3.3780000000000001,"5min":5.5250000000000004,"15min":43.475000000000001},"last":0.97799999999999998}]},{"osd":7,"last update":"Mon Feb 5 06:47:38 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":1.2889999999999999,"15min":1.2929999999999999},"min":{"1min":0.45900000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.8779999999999999,"5min":8.016,"15min":13.49},"last":1.3340000000000001},{"interface":"front","average":{"1min":0.628,"5min":1.425,"15min":1.3400000000000001},"min":{"1min":0.41699999999999998,"5min":0.40400000000000003,"15min":0.35499999999999998},"max":{"1min":1.6259999999999999,"5min":10.481,"15min":10.481},"last":1.4079999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738589,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3931700,"kb_used_data":3431276,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":89813452,"statfs":{"total":95995035648,"available":91968974848,"internally_reserved":0,"allocated":3513626624,"data_stored":3513087020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 06:48:13 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":1.1619999999999999,"15min":1.464},"min":{"1min":0.22600000000000001,"5min":0.186,"15min":0.182},"max":{"1min":2.3439999999999999,"5min":18.311,"15min":18.59},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.381,"15min":1.4850000000000001},"min":{"1min":0.23799999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":3.4020000000000001,"5min":21.088000000000001,"15min":21.088000000000001},"last":3.9990000000000001}]},{"osd":2,"last update":"Mon Feb 5 06:47:33 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.86299999999999999,"15min":1.333},"min":{"1min":0.24199999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":3.661,"5min":3.661,"15min":16.981999999999999},"last":4.3570000000000002},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.88400000000000001,"15min":1.262},"min":{"1min":0.27900000000000003,"5min":0.221,"15min":0.221},"max":{"1min":7.093,"5min":7.093,"15min":16.998999999999999},"last":1.1799999999999999}]},{"osd":3,"last update":"Mon Feb 5 06:48:05 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":1.175,"15min":1.6359999999999999},"min":{"1min":0.314,"5min":0.27300000000000002,"15min":0.25700000000000001},"max":{"1min":2.032,"5min":18.349,"15min":43.302},"last":0.875},{"interface":"front","average":{"1min":0.86099999999999999,"5min":1.3300000000000001,"15min":1.8340000000000001},"min":{"1min":0.33500000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.1390000000000002,"5min":18.712,"15min":43.651000000000003},"last":1.0960000000000001}]},{"osd":4,"last update":"Mon Feb 5 06:48:26 2024","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":1.1020000000000001,"15min":1.3240000000000001},"min":{"1min":0.41299999999999998,"5min":0.312,"15min":0.27200000000000002},"max":{"1min":4.4989999999999997,"5min":4.4989999999999997,"15min":40.789999999999999},"last":4.1280000000000001},{"interface":"front","average":{"1min":1.224,"5min":1.0600000000000001,"15min":1.363},"min":{"1min":0.41499999999999998,"5min":0.33600000000000002,"15min":0.30099999999999999},"max":{"1min":4.1130000000000004,"5min":8.4019999999999992,"15min":40.773000000000003},"last":4.7729999999999997}]},{"osd":5,"last update":"Mon Feb 5 06:47:52 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":1.728,"15min":1.401},"min":{"1min":0.38900000000000001,"5min":0.32500000000000001,"15min":0.29299999999999998},"max":{"1min":1.252,"5min":48.841999999999999,"15min":48.841999999999999},"last":1.208},{"interface":"front","average":{"1min":0.93899999999999995,"5min":1.619,"15min":1.373},"min":{"1min":0.47199999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":4.3449999999999998,"5min":48.450000000000003,"15min":48.450000000000003},"last":4.1559999999999997}]},{"osd":6,"last update":"Mon Feb 5 06:48:13 2024","interfaces":[{"interface":"back","average":{"1min":9.0020000000000007,"5min":2.54,"15min":1.746},"min":{"1min":0.42099999999999999,"5min":0.376,"15min":0.33700000000000002},"max":{"1min":136.76300000000001,"5min":136.76300000000001,"15min":136.76300000000001},"last":0.98199999999999998},{"interface":"front","average":{"1min":9.1989999999999998,"5min":2.8260000000000001,"15min":1.8},"min":{"1min":0.51300000000000001,"5min":0.39200000000000002,"15min":0.33000000000000002},"max":{"1min":135.92400000000001,"5min":135.92400000000001,"15min":135.92400000000001},"last":1.1439999999999999}]},{"osd":7,"last update":"Mon Feb 5 06:47:41 2024","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":1.081,"15min":1.204},"min":{"1min":0.434,"5min":0.40100000000000002,"15min":0.31900000000000001},"max":{"1min":2.391,"5min":6.1420000000000003,"15min":17.210000000000001},"last":4.1849999999999996},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.98199999999999998,"15min":1.276},"min":{"1min":0.435,"5min":0.378,"15min":0.34899999999999998},"max":{"1min":1.496,"5min":6.7220000000000004,"15min":18.533999999999999},"last":1.014}]}]},{"osd":2,"up_from":18,"seq":77309411538,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2464540,"kb_used_data":2036308,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":91280612,"statfs":{"total":95995035648,"available":93471346688,"internally_reserved":0,"allocated":2085179392,"data_stored":2084664801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:47:39 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":1.0600000000000001,"15min":1.72},"min":{"1min":0.22900000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":4.0259999999999998,"5min":14.007,"15min":50.814},"last":5.7839999999999998},{"interface":"front","average":{"1min":0.77600000000000002,"5min":1.1379999999999999,"15min":1.7230000000000001},"min":{"1min":0.23100000000000001,"5min":0.22600000000000001,"15min":0.17699999999999999},"max":{"1min":4.7480000000000002,"5min":13.968999999999999,"15min":50.802999999999997},"last":2.5270000000000001}]},{"osd":1,"last update":"Mon Feb 5 06:47:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":1.482,"15min":1.6259999999999999},"min":{"1min":0.19700000000000001,"5min":0.189,"15min":0.189},"max":{"1min":3.1429999999999998,"5min":15.382,"15min":31.303000000000001},"last":0.95299999999999996},{"interface":"front","average":{"1min":1.028,"5min":1.54,"15min":1.3979999999999999},"min":{"1min":0.27700000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":9.3019999999999996,"5min":15.742000000000001,"15min":31.425000000000001},"last":1.2110000000000001}]},{"osd":3,"last update":"Mon Feb 5 06:48:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":1.153,"15min":1.8320000000000001},"min":{"1min":0.316,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.591,"5min":15.166,"15min":53.127000000000002},"last":1.698},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.1299999999999999,"15min":1.7529999999999999},"min":{"1min":0.30199999999999999,"5min":0.23400000000000001,"15min":0.22500000000000001},"max":{"1min":2.2930000000000001,"5min":15.541,"15min":52.607999999999997},"last":1.573}]},{"osd":4,"last update":"Mon Feb 5 06:48:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":0.95199999999999996,"15min":1.03},"min":{"1min":0.38,"5min":0.29999999999999999,"15min":0.23999999999999999},"max":{"1min":3.573,"5min":7.0110000000000001,"15min":12.041},"last":1.2869999999999999},{"interface":"front","average":{"1min":1.075,"5min":0.871,"15min":0.98399999999999999},"min":{"1min":0.40100000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":4.907,"5min":5.0350000000000001,"15min":15.689},"last":3.4980000000000002}]},{"osd":5,"last update":"Mon Feb 5 06:47:54 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":1.2330000000000001,"15min":1.4750000000000001},"min":{"1min":0.33800000000000002,"5min":0.315,"15min":0.29899999999999999},"max":{"1min":1.2729999999999999,"5min":30.265000000000001,"15min":86.614999999999995},"last":1.161},{"interface":"front","average":{"1min":0.89400000000000002,"5min":1.373,"15min":1.6990000000000001},"min":{"1min":0.36599999999999999,"5min":0.312,"15min":0.312},"max":{"1min":3.222,"5min":30.02,"15min":86.543999999999997},"last":1.0089999999999999}]},{"osd":6,"last update":"Mon Feb 5 06:48:25 2024","interfaces":[{"interface":"back","average":{"1min":2.6030000000000002,"5min":1.2190000000000001,"15min":1.161},"min":{"1min":0.437,"5min":0.35499999999999998,"15min":0.315},"max":{"1min":29.780999999999999,"5min":29.780999999999999,"15min":29.780999999999999},"last":3.641},{"interface":"front","average":{"1min":3.2999999999999998,"5min":1.6739999999999999,"15min":1.446},"min":{"1min":0.46300000000000002,"5min":0.316,"15min":0.316},"max":{"1min":29.763000000000002,"5min":29.763000000000002,"15min":29.763000000000002},"last":3.5670000000000002}]},{"osd":7,"last update":"Mon Feb 5 06:47:43 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":1.6859999999999999,"15min":1.4510000000000001},"min":{"1min":0.495,"5min":0.373,"15min":0.35699999999999998},"max":{"1min":3.2530000000000001,"5min":45.243000000000002,"15min":45.243000000000002},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.702,"15min":1.3520000000000001},"min":{"1min":0.40799999999999997,"5min":0.379,"15min":0.36899999999999999},"max":{"1min":17.611000000000001,"5min":45.171999999999997,"15min":45.171999999999997},"last":1.258}]}]},{"osd":3,"up_from":26,"seq":111669149901,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2364272,"kb_used_data":1737512,"kb_used_omap":1,"kb_used_meta":626750,"kb_avail":91380880,"statfs":{"total":95995035648,"available":93574021120,"internally_reserved":0,"allocated":1779212288,"data_stored":1778676680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":641792455},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:48:14 2024","interfaces":[{"interface":"back","average":{"1min":4.3970000000000002,"5min":1.9510000000000001,"15min":1.5349999999999999},"min":{"1min":0.21099999999999999,"5min":0.189,"15min":0.184},"max":{"1min":53.656999999999996,"5min":53.656999999999996,"15min":53.656999999999996},"last":0.52900000000000003},{"interface":"front","average":{"1min":4.1260000000000003,"5min":1.8280000000000001,"15min":1.736},"min":{"1min":0.217,"5min":0.217,"15min":0.20399999999999999},"max":{"1min":53.338000000000001,"5min":53.338000000000001,"15min":53.338000000000001},"last":5.9489999999999998}]},{"osd":1,"last update":"Mon Feb 5 06:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":1.4370000000000001,"15min":1.4950000000000001},"min":{"1min":0.29199999999999998,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":2.157,"5min":11.412000000000001,"15min":11.412000000000001},"last":5.8860000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":1.419,"15min":1.518},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.215},"max":{"1min":1.5680000000000001,"5min":9.8659999999999997,"15min":9.8659999999999997},"last":1.244}]},{"osd":2,"last update":"Mon Feb 5 06:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":1.157,"15min":1.5649999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":2.3540000000000001,"5min":7.7080000000000002,"15min":41.529000000000003},"last":5.7450000000000001},{"interface":"front","average":{"1min":0.623,"5min":1.0640000000000001,"15min":1.4570000000000001},"min":{"1min":0.28499999999999998,"5min":0.249,"15min":0.248},"max":{"1min":2.1949999999999998,"5min":7.3559999999999999,"15min":41.759999999999998},"last":0.79400000000000004}]},{"osd":4,"last update":"Mon Feb 5 06:47:34 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":1.24,"15min":1.109},"min":{"1min":0.41999999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.8360000000000001,"5min":7.7329999999999997,"15min":16.577999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.64300000000000002,"5min":1.607,"15min":1.369},"min":{"1min":0.371,"5min":0.30099999999999999,"15min":0.28599999999999998},"max":{"1min":1.831,"5min":19.359999999999999,"15min":19.359999999999999},"last":1.367}]},{"osd":5,"last update":"Mon Feb 5 06:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":1.464,"15min":1.4339999999999999},"min":{"1min":0.46600000000000003,"5min":0.32000000000000001,"15min":0.29299999999999998},"max":{"1min":1.5589999999999999,"5min":20.257999999999999,"15min":20.257999999999999},"last":1.202},{"interface":"front","average":{"1min":0.626,"5min":1.3140000000000001,"15min":1.181},"min":{"1min":0.441,"5min":0.38700000000000001,"15min":0.28399999999999997},"max":{"1min":0.95999999999999996,"5min":7.859,"15min":15.792999999999999},"last":0.48699999999999999}]},{"osd":6,"last update":"Mon Feb 5 06:48:30 2024","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.345,"15min":1.3540000000000001},"min":{"1min":0.38700000000000001,"5min":0.35499999999999998,"15min":0.28100000000000003},"max":{"1min":3.6469999999999998,"5min":7.556,"15min":8.3610000000000007},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.276,"5min":1.302,"15min":1.319},"min":{"1min":0.46600000000000003,"5min":0.44900000000000001,"15min":0.28599999999999998},"max":{"1min":3.0190000000000001,"5min":7.2439999999999998,"15min":7.2439999999999998},"last":1.3240000000000001}]},{"osd":7,"last update":"Mon Feb 5 06:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":1.3600000000000001,"15min":1.335},"min":{"1min":0.376,"5min":0.376,"15min":0.33400000000000002},"max":{"1min":1.1950000000000001,"5min":7.8029999999999999,"15min":45.039000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.75800000000000001,"5min":1.3220000000000001,"15min":1.4770000000000001},"min":{"1min":0.51500000000000001,"5min":0.377,"15min":0.34399999999999997},"max":{"1min":1.349,"5min":7.2720000000000002,"15min":44.036999999999999},"last":1.282}]}]},{"osd":4,"up_from":31,"seq":133143986375,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3159280,"kb_used_data":2650532,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":90585872,"statfs":{"total":95995035648,"available":92759932928,"internally_reserved":0,"allocated":2714144768,"data_stored":2713612934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:48:29 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":0.91200000000000003,"15min":1.212},"min":{"1min":0.27000000000000002,"5min":0.217,"15min":0.20200000000000001},"max":{"1min":5.0170000000000003,"5min":5.0170000000000003,"15min":33.372},"last":0.629},{"interface":"front","average":{"1min":1.0629999999999999,"5min":0.95499999999999996,"15min":1.3180000000000001},"min":{"1min":0.26000000000000001,"5min":0.247,"15min":0.22700000000000001},"max":{"1min":4.1559999999999997,"5min":10.913,"15min":33.323999999999998},"last":3.7519999999999998}]},{"osd":1,"last update":"Mon Feb 5 06:48:29 2024","interfaces":[{"interface":"back","average":{"1min":3.8039999999999998,"5min":1.704,"15min":1.306},"min":{"1min":0.26400000000000001,"5min":0.247,"15min":0.21299999999999999},"max":{"1min":53.414000000000001,"5min":53.414000000000001,"15min":53.414000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":3.7759999999999998,"5min":1.661,"15min":1.2729999999999999},"min":{"1min":0.36899999999999999,"5min":0.254,"15min":0.23000000000000001},"max":{"1min":53.719999999999999,"5min":53.719999999999999,"15min":53.719999999999999},"last":0.66900000000000004}]},{"osd":2,"last update":"Mon Feb 5 06:48:29 2024","interfaces":[{"interface":"back","average":{"1min":1.01,"5min":0.77100000000000002,"15min":0.83199999999999996},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.245},"max":{"1min":2.879,"5min":6.6040000000000001,"15min":17.379999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.97599999999999998,"5min":1.083,"15min":1.0660000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.26000000000000001},"max":{"1min":3.1440000000000001,"5min":18.077000000000002,"15min":18.077000000000002},"last":0.52700000000000002}]},{"osd":3,"last update":"Mon Feb 5 06:48:29 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.9630000000000001,"15min":2.403},"min":{"1min":0.376,"5min":0.32000000000000001,"15min":0.252},"max":{"1min":3.3999999999999999,"5min":33.923999999999999,"15min":99.546999999999997},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.84799999999999998,"5min":1.8420000000000001,"15min":2.3820000000000001},"min":{"1min":0.32000000000000001,"5min":0.29799999999999999,"15min":0.29099999999999998},"max":{"1min":2.544,"5min":33.804000000000002,"15min":99.245000000000005},"last":0.39700000000000002}]},{"osd":5,"last update":"Mon Feb 5 06:47:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.87,"15min":0.69099999999999995},"min":{"1min":0.249,"5min":0.249,"15min":0.24399999999999999},"max":{"1min":1.4930000000000001,"5min":5.6059999999999999,"15min":11.098000000000001},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.82599999999999996,"15min":0.66000000000000003},"min":{"1min":0.28899999999999998,"5min":0.28499999999999998,"15min":0.252},"max":{"1min":2.5339999999999998,"5min":4.1210000000000004,"15min":10.965999999999999},"last":0.65200000000000002}]},{"osd":6,"last update":"Mon Feb 5 06:48:29 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.0009999999999999,"15min":0.77800000000000002},"min":{"1min":0.35599999999999998,"5min":0.32300000000000001,"15min":0.246},"max":{"1min":5.9119999999999999,"5min":10.039999999999999,"15min":11.868},"last":0.68400000000000005},{"interface":"front","average":{"1min":1.147,"5min":0.92000000000000004,"15min":0.75900000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.27600000000000002},"max":{"1min":5.8970000000000002,"5min":10.074,"15min":11.787000000000001},"last":0.46400000000000002}]},{"osd":7,"last update":"Mon Feb 5 06:47:36 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.80600000000000005,"15min":0.65400000000000003},"min":{"1min":0.32900000000000001,"5min":0.316,"15min":0.26000000000000001},"max":{"1min":1.7749999999999999,"5min":3.4129999999999998,"15min":5.1500000000000004},"last":0.432},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.81299999999999994,"15min":0.68899999999999995},"min":{"1min":0.314,"5min":0.307,"15min":0.28499999999999998},"max":{"1min":2.9009999999999998,"5min":4.5350000000000001,"15min":5.5890000000000004},"last":0.48799999999999999}]}]},{"osd":5,"up_from":40,"seq":171798692034,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2339356,"kb_used_data":1708436,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":91405796,"statfs":{"total":95995035648,"available":93599535104,"internally_reserved":0,"allocated":1749438464,"data_stored":1748923767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":646052294},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.079,"15min":1.492},"min":{"1min":0.33900000000000002,"5min":0.24299999999999999,"15min":0.19700000000000001},"max":{"1min":4.2869999999999999,"5min":4.6479999999999997,"15min":55.704999999999998},"last":0.505},{"interface":"front","average":{"1min":1.02,"5min":1.052,"15min":1.681},"min":{"1min":0.379,"5min":0.255,"15min":0.22600000000000001},"max":{"1min":5.2030000000000003,"5min":6.532,"15min":56.789000000000001},"last":1.6060000000000001}]},{"osd":1,"last update":"Mon Feb 5 06:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":1.1160000000000001,"15min":1.389},"min":{"1min":0.378,"5min":0.26900000000000002,"15min":0.248},"max":{"1min":4.6340000000000003,"5min":6.5049999999999999,"15min":41.191000000000003},"last":1.2290000000000001},{"interface":"front","average":{"1min":0.89000000000000001,"5min":1.153,"15min":1.252},"min":{"1min":0.38200000000000001,"5min":0.28100000000000003,"15min":0.23499999999999999},"max":{"1min":4.8609999999999998,"5min":4.8609999999999998,"15min":41.173000000000002},"last":1.034}]},{"osd":2,"last update":"Mon Feb 5 06:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.84399999999999997,"15min":1.25},"min":{"1min":0.33900000000000002,"5min":0.26800000000000002,"15min":0.23000000000000001},"max":{"1min":0.75700000000000001,"5min":4.0599999999999996,"15min":36.901000000000003},"last":2.0600000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.95999999999999996,"15min":1.2030000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.94599999999999995,"5min":6.4619999999999997,"15min":37.984000000000002},"last":1.696}]},{"osd":3,"last update":"Mon Feb 5 06:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":1.716,"15min":1.585},"min":{"1min":0.34399999999999997,"5min":0.33200000000000002,"15min":0.31900000000000001},"max":{"1min":1.0620000000000001,"5min":62.508000000000003,"15min":62.508000000000003},"last":1.212},{"interface":"front","average":{"1min":0.79300000000000004,"5min":1.929,"15min":1.748},"min":{"1min":0.36499999999999999,"5min":0.28100000000000003,"15min":0.27400000000000002},"max":{"1min":4.4820000000000002,"5min":62.460000000000001,"15min":62.460000000000001},"last":1.272}]},{"osd":4,"last update":"Mon Feb 5 06:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.83699999999999997,"15min":0.60399999999999998},"min":{"1min":0.33400000000000002,"5min":0.26200000000000001,"15min":0.23200000000000001},"max":{"1min":1.099,"5min":4.7370000000000001,"15min":5.4969999999999999},"last":0.75},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.79900000000000004,"15min":0.61699999999999999},"min":{"1min":0.35099999999999998,"5min":0.252,"15min":0.252},"max":{"1min":0.61199999999999999,"5min":4.0380000000000003,"15min":4.0380000000000003},"last":1.115}]},{"osd":6,"last update":"Mon Feb 5 06:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":1.024,"15min":0.72799999999999998},"min":{"1min":0.308,"5min":0.28000000000000003,"15min":0.23799999999999999},"max":{"1min":3.2269999999999999,"5min":7.6710000000000003,"15min":7.6710000000000003},"last":1.337},{"interface":"front","average":{"1min":0.91900000000000004,"5min":1.002,"15min":0.76900000000000002},"min":{"1min":0.38800000000000001,"5min":0.33900000000000002,"15min":0.27600000000000002},"max":{"1min":3.266,"5min":7.21,"15min":7.21},"last":0.99099999999999999}]},{"osd":7,"last update":"Mon Feb 5 06:47:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":1.073,"15min":0.72299999999999998},"min":{"1min":0.38500000000000001,"5min":0.30199999999999999,"15min":0.25600000000000001},"max":{"1min":0.88200000000000001,"5min":6.9870000000000001,"15min":6.9870000000000001},"last":1.1539999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":1.0680000000000001,"15min":0.75600000000000001},"min":{"1min":0.36599999999999999,"5min":0.28999999999999998,"15min":0.27100000000000002},"max":{"1min":0.68999999999999995,"5min":6.5839999999999996,"15min":6.5839999999999996},"last":1.3080000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3517317120,"data_stored":3517317120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1036320768,"data_stored":1036320768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2104819712,"data_stored":2104819712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1779564544,"data_stored":1779564544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2754871296,"data_stored":2754871296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1750204416,"data_stored":1750204416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2444492800,"data_stored":2444492800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1375076352,"data_stored":1375076352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T06:48:37.772 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-05T06:48:37.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd pool set unique_pool_3 pg_num 26 2024-02-05T06:48:37.844 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:37.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:37.878 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:37.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:37.953 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 58 16 92193 92177 99.2964 63.8125 0.00507537 0.0100611 2024-02-05T06:48:37.979 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:38.013 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:38.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:38.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:38.115 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:38.158 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:38.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:38.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:38.259 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:38.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:38 smithi028 ceph-mon[29442]: 6.7 scrub starts 2024-02-05T06:48:38.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:38 smithi028 ceph-mon[29442]: 6.7 scrub ok 2024-02-05T06:48:38.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:38 smithi028 ceph-mon[29442]: 10.a deep-scrub starts 2024-02-05T06:48:38.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:38 smithi028 ceph-mon[29442]: 4.f scrub starts 2024-02-05T06:48:38.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:38 smithi028 ceph-mon[29442]: 4.f scrub ok 2024-02-05T06:48:38.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:38 smithi028 ceph-mon[29442]: 6.8 scrub starts 2024-02-05T06:48:38.293 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:38 smithi028 ceph-mon[29442]: 6.8 scrub ok 2024-02-05T06:48:38.293 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:38 smithi028 ceph-mon[37984]: 6.7 scrub starts 2024-02-05T06:48:38.293 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:38 smithi028 ceph-mon[37984]: 6.7 scrub ok 2024-02-05T06:48:38.293 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:38 smithi028 ceph-mon[37984]: 10.a deep-scrub starts 2024-02-05T06:48:38.293 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:38 smithi028 ceph-mon[37984]: 4.f scrub starts 2024-02-05T06:48:38.293 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:38 smithi028 ceph-mon[37984]: 4.f scrub ok 2024-02-05T06:48:38.293 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:38 smithi028 ceph-mon[37984]: 6.8 scrub starts 2024-02-05T06:48:38.293 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:38 smithi028 ceph-mon[37984]: 6.8 scrub ok 2024-02-05T06:48:38.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:38.295 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:38.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:38.396 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:38.453 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:38.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:38.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:38.555 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:38.588 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:38.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:38.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:38.632 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:48:38.690 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:38.720 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:38.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:38.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:38.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:38 smithi167 ceph-mon[32774]: 6.7 scrub starts 2024-02-05T06:48:38.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:38 smithi167 ceph-mon[32774]: 6.7 scrub ok 2024-02-05T06:48:38.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:38 smithi167 ceph-mon[32774]: 10.a deep-scrub starts 2024-02-05T06:48:38.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:38 smithi167 ceph-mon[32774]: 4.f scrub starts 2024-02-05T06:48:38.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:38 smithi167 ceph-mon[32774]: 4.f scrub ok 2024-02-05T06:48:38.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:38 smithi167 ceph-mon[32774]: 6.8 scrub starts 2024-02-05T06:48:38.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:38 smithi167 ceph-mon[32774]: 6.8 scrub ok 2024-02-05T06:48:38.822 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:38.855 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:38.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:38.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:38.953 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 59 16 93025 93009 98.4949 52 0.0141691 0.0101471 2024-02-05T06:48:38.956 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:38.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:38.995 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:38.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:39.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:39.149 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:39.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:39.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:39.249 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-02-05T06:48:39.251 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:39.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:39.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:39.397 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:39.429 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:39.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:39.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:39.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:39 smithi028 ceph-mon[29442]: 1.8 scrub starts 2024-02-05T06:48:39.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:39 smithi028 ceph-mon[29442]: 1.8 scrub ok 2024-02-05T06:48:39.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:39 smithi028 ceph-mon[29442]: 10.f deep-scrub starts 2024-02-05T06:48:39.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:39 smithi028 ceph-mon[29442]: 3.9 scrub starts 2024-02-05T06:48:39.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:39 smithi028 ceph-mon[29442]: 3.9 scrub ok 2024-02-05T06:48:39.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:39 smithi028 ceph-mon[29442]: 6.9 scrub starts 2024-02-05T06:48:39.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:39 smithi028 ceph-mon[29442]: pgmap v724: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-02-05T06:48:39.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:39 smithi028 ceph-mon[29442]: 6.9 scrub ok 2024-02-05T06:48:39.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:39 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3246039849' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:48:39.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:39 smithi028 ceph-mon[37984]: 1.8 scrub starts 2024-02-05T06:48:39.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:39 smithi028 ceph-mon[37984]: 1.8 scrub ok 2024-02-05T06:48:39.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:39 smithi028 ceph-mon[37984]: 10.f deep-scrub starts 2024-02-05T06:48:39.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:39 smithi028 ceph-mon[37984]: 3.9 scrub starts 2024-02-05T06:48:39.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:39 smithi028 ceph-mon[37984]: 3.9 scrub ok 2024-02-05T06:48:39.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:39 smithi028 ceph-mon[37984]: 6.9 scrub starts 2024-02-05T06:48:39.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:39 smithi028 ceph-mon[37984]: pgmap v724: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-02-05T06:48:39.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:39 smithi028 ceph-mon[37984]: 6.9 scrub ok 2024-02-05T06:48:39.500 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:39 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3246039849' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:48:39.531 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:39.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:39.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:39.673 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:39.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:39.706 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:39.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:39.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:39 smithi167 ceph-mon[32774]: 1.8 scrub starts 2024-02-05T06:48:39.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:39 smithi167 ceph-mon[32774]: 1.8 scrub ok 2024-02-05T06:48:39.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:39 smithi167 ceph-mon[32774]: 10.f deep-scrub starts 2024-02-05T06:48:39.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:39 smithi167 ceph-mon[32774]: 3.9 scrub starts 2024-02-05T06:48:39.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:39 smithi167 ceph-mon[32774]: 3.9 scrub ok 2024-02-05T06:48:39.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:39 smithi167 ceph-mon[32774]: 6.9 scrub starts 2024-02-05T06:48:39.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:39 smithi167 ceph-mon[32774]: pgmap v724: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-02-05T06:48:39.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:39 smithi167 ceph-mon[32774]: 6.9 scrub ok 2024-02-05T06:48:39.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:39 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3246039849' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:48:39.807 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:39.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:39.846 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:39.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:39.947 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:39.953 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-02-05T06:48:39.946248+0000 min lat: 0.00116262 max lat: 0.237422 avg lat: 0.010225 2024-02-05T06:48:39.953 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T06:48:39.953 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 60 16 93856 93840 97.7191 51.9375 0.00566663 0.010225 2024-02-05T06:48:39.986 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:39.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:39.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:40.088 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:40.122 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:40.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:40.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:40.224 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:40.256 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:40.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:40.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:40.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:40.396 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:40.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:40.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:40.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:40.530 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:40.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:40.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:40.633 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:40.667 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:40.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:40.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:40.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:40 smithi028 ceph-mon[29442]: 5.f scrub starts 2024-02-05T06:48:40.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:40 smithi028 ceph-mon[29442]: 5.f scrub ok 2024-02-05T06:48:40.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:40 smithi028 ceph-mon[29442]: 3.0 scrub starts 2024-02-05T06:48:40.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:40 smithi028 ceph-mon[29442]: 3.0 scrub ok 2024-02-05T06:48:40.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:40 smithi028 ceph-mon[29442]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:48:40.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:40 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3246039849' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:48:40.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:40 smithi028 ceph-mon[29442]: osdmap e370: 8 total, 8 up, 8 in 2024-02-05T06:48:40.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:40 smithi028 ceph-mon[37984]: 5.f scrub starts 2024-02-05T06:48:40.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:40 smithi028 ceph-mon[37984]: 5.f scrub ok 2024-02-05T06:48:40.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:40 smithi028 ceph-mon[37984]: 3.0 scrub starts 2024-02-05T06:48:40.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:40 smithi028 ceph-mon[37984]: 3.0 scrub ok 2024-02-05T06:48:40.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:40 smithi028 ceph-mon[37984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:48:40.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:40 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3246039849' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:48:40.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:40 smithi028 ceph-mon[37984]: osdmap e370: 8 total, 8 up, 8 in 2024-02-05T06:48:40.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:40 smithi167 ceph-mon[32774]: 5.f scrub starts 2024-02-05T06:48:40.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:40 smithi167 ceph-mon[32774]: 5.f scrub ok 2024-02-05T06:48:40.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:40 smithi167 ceph-mon[32774]: 3.0 scrub starts 2024-02-05T06:48:40.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:40 smithi167 ceph-mon[32774]: 3.0 scrub ok 2024-02-05T06:48:40.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:40 smithi167 ceph-mon[32774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:48:40.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:40 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3246039849' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:48:40.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:40 smithi167 ceph-mon[32774]: osdmap e370: 8 total, 8 up, 8 in 2024-02-05T06:48:40.769 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:40.803 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:40.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:40.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:40.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:40.939 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:40.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:40.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:40.953 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 61 16 94801 94785 97.0854 59.0625 0.00702093 0.010294 2024-02-05T06:48:41.041 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:41.081 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:41.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:41.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:41.182 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:41.222 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:41.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:41.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:41.324 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:41.362 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:41.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:41.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:41.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:41 smithi028 ceph-mon[29442]: pgmap v726: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-02-05T06:48:41.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:41 smithi028 ceph-mon[37984]: pgmap v726: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-02-05T06:48:41.463 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:41.503 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:41.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:41.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:41.604 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:41.641 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:41.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:41.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:41.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:41 smithi167 ceph-mon[32774]: pgmap v726: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-02-05T06:48:41.742 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:41.776 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:41.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:41.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:41.878 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:41.915 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:41.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:41.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:41.953 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 62 16 95683 95667 96.4087 55.125 0.00548951 0.0103643 2024-02-05T06:48:42.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:42.056 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:42.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:42.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:42.158 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:42.194 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:42.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:42.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:42.296 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:42.320 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:42.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:42.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:42.421 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:42.462 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:42.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:42.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:42.564 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:42.600 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:42.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:42.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:42.702 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:42.731 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:42.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:42.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:42.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:42.867 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:42.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:42.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:42.950 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:48:42 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:48:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:48:42.953 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 63 16 96517 96501 95.7058 52.125 0.0126212 0.0104433 2024-02-05T06:48:42.969 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:43.007 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:43.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:43.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:43.108 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:43.147 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:43.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:43.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:43.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:43.263 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:48:43.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:43.312 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:43.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:43.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:43 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:48:43.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:43 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:48:43.384 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:43 smithi028 ceph-mon[29442]: pgmap v727: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-02-05T06:48:43.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:43 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:48:43.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:43 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:48:43.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:43 smithi028 ceph-mon[37984]: pgmap v727: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-02-05T06:48:43.414 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:43.449 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:43.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:43.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:43.485 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:48:43 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:48:43.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:43 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:48:43.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:43 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:48:43.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:43 smithi167 ceph-mon[32774]: pgmap v727: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-02-05T06:48:43.550 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:43.585 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:43.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:43.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:43.687 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:43.721 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:43.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:43.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:43.822 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:43.857 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:43.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:43.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:43.951 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:48:43.951 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:48:43.954 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 64 16 97379 97363 95.0523 53.875 0.0631284 0.0105135 2024-02-05T06:48:43.958 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:44.001 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:44.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:44.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:44.024 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T06:48:44.104 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:44.147 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:44.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:44.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:44.249 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:44.283 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:44.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:44.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:44.385 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:44.388 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:44 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:48:44.389 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:44 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:48:44.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:44.422 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:44.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:44.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:44 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:48:44.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:44.555 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:44.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:44.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:44.657 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:44.693 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:44.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:44.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:44.795 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:44.825 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-05T06:48:44.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:44.829 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:44.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:44.930 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:44.954 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 65 16 98249 98233 94.4265 54.375 0.0483253 0.0105844 2024-02-05T06:48:44.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:44.971 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:44.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:45.072 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:45.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:45.109 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:45.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:45.210 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:45.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:45.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:45.360 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:45.378 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:45 smithi028 ceph-mon[29442]: pgmap v728: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 5.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-02-05T06:48:45.378 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:45 smithi028 ceph-mon[37984]: pgmap v728: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 5.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-02-05T06:48:45.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:45.388 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:45.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:45.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:45 smithi167 ceph-mon[32774]: pgmap v728: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 5.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-02-05T06:48:45.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:45.491 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:48:45.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:45.535 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:45.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:45.535 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:48:45.575 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:48:45.613 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:48:45.636 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:45.657 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:48:45.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:45.684 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:45.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:45.708 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:48:45.750 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:48:45.786 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:45.793 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:48:45.799 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:48:45.799 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:48:45.799 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:48:45.799 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:48:45.800 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:48:45.800 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:48:45.800 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:48:45.800 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:48:45.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:45.823 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:45.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:45.924 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:45.954 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 66 16 99034 99018 93.7393 49.0625 0.00716007 0.0106626 2024-02-05T06:48:45.965 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:45.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:45.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:46.067 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:46.107 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:46.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:46.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:46.209 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:46.247 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:46.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:46.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:46.348 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:46.381 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:46.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:46.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:46.393 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:46 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/4209518561' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:48:46.394 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:46 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/4209518561' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:48:46.483 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:46.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:46 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/4209518561' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:48:46.521 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:46.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:46.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:46.623 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:46.692 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:46.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:46.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:46.794 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:46.828 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:46.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:46.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:46.929 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:46.954 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 67 16 99901 99885 93.149 54.1875 0.0664664 0.0107296 2024-02-05T06:48:46.965 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:46.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:46.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:47.067 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:47.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:47.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:47.203 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:47.242 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:48:47.247 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:48:47.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:47.268 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:47.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:47.369 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:47.403 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:47.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:47.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:47.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:47 smithi028 ceph-mon[29442]: pgmap v729: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 5.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-02-05T06:48:47.422 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:47 smithi028 ceph-mon[37984]: pgmap v729: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 5.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-02-05T06:48:47.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:47 smithi167 ceph-mon[32774]: pgmap v729: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 5.8 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-02-05T06:48:47.505 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:47.542 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:47.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:47.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:47.644 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:47.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:47.680 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:47.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:47.781 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:47.830 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:47.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:47.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:47.933 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:47.954 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 68 16 100763 100747 92.5715 53.875 0.0145292 0.010797 2024-02-05T06:48:47.975 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:47.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:47.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:48.076 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:48.112 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:48.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:48.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:48.214 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:48.251 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:48.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:48.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:48.353 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:48.394 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:48.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:48.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:48.495 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:48.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:48.530 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:48.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:48.564 INFO:teuthology.orchestra.run.smithi028.stderr:set pool 10 pg_num to 26 2024-02-05T06:48:48.631 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:48.640 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:48:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:48.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:48.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:48.768 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:48.800 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:48.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:48.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:48.902 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:48.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:48 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1653845522' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-02-05T06:48:48.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:48 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1653845522' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-02-05T06:48:48.957 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 69 16 101526 101510 91.9207 47.6875 0.0423665 0.0108709 2024-02-05T06:48:48.966 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:48.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:48.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:48.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:48 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1653845522' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-02-05T06:48:49.068 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:49.097 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:49.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:49.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:49.199 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:49.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:49.242 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:49.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:49.343 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:49.379 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:49.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:49.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:49.480 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:49.513 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:49.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:49.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:49.615 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:49.654 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:49.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:49.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:49.756 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:49.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:49.804 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:49.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:49.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:49.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:49 smithi028 ceph-mon[29442]: pgmap v730: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 55 MiB/s wr, 884 op/s 2024-02-05T06:48:49.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:49 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1653845522' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-02-05T06:48:49.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:49 smithi028 ceph-mon[29442]: osdmap e371: 8 total, 8 up, 8 in 2024-02-05T06:48:49.944 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:49 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:48:49.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:49 smithi028 ceph-mon[37984]: pgmap v730: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 55 MiB/s wr, 884 op/s 2024-02-05T06:48:49.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:49 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1653845522' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-02-05T06:48:49.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:49 smithi028 ceph-mon[37984]: osdmap e371: 8 total, 8 up, 8 in 2024-02-05T06:48:49.944 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:49 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:48:49.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:49.945 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:49.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:49.954 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 70 16 102327 102311 91.3228 50.0625 0.00785252 0.0109411 2024-02-05T06:48:49.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:49 smithi167 ceph-mon[32774]: pgmap v730: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 55 MiB/s wr, 884 op/s 2024-02-05T06:48:49.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:49 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1653845522' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-02-05T06:48:49.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:49 smithi167 ceph-mon[32774]: osdmap e371: 8 total, 8 up, 8 in 2024-02-05T06:48:49.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:49 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:48:50.046 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:50.083 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:50.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:50.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:50.185 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:50.219 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:50.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:50.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:50.321 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:50.356 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:50.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:50.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:50.458 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:50.496 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:50.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:50.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:50.598 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:50.653 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:50.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:50.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:50.755 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:50.786 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:50.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:50.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:50.887 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:50.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:50.922 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:50.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:50.954 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 71 16 103167 103151 90.776 52.5 0.00628955 0.0110104 2024-02-05T06:48:51.023 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:51.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:51.066 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:51.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:51.167 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:51.200 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:51.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:51.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:51.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:51.376 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:51.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:51.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:51.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:51 smithi028 ceph-mon[37984]: pgmap v732: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 55 MiB/s wr, 884 op/s 2024-02-05T06:48:51.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:51 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T06:48:51.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:51 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T06:48:51.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:51 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:48:51.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:51 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:48:51.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:51 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:48:51.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:51 smithi028 ceph-mon[29442]: pgmap v732: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 55 MiB/s wr, 884 op/s 2024-02-05T06:48:51.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:51 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T06:48:51.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:51 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T06:48:51.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:51 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:48:51.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:51 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:48:51.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:51 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:48:51.478 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:51.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:51 smithi167 ceph-mon[32774]: pgmap v732: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 55 MiB/s wr, 884 op/s 2024-02-05T06:48:51.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:51 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T06:48:51.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:51 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-02-05T06:48:51.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:51 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:48:51.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:51 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:48:51.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:51 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:48:51.955 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 72 16 103411 103395 89.7272 15.25 0.0664074 0.0110341 2024-02-05T06:48:52.312 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:52.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:52.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:52.414 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:52.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:52.450 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:52.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:52.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:52 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-05T06:48:52.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:52 smithi167 ceph-mon[32774]: osdmap e372: 8 total, 8 up, 8 in 2024-02-05T06:48:52.495 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:52 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-05T06:48:52.495 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:52 smithi028 ceph-mon[37984]: osdmap e372: 8 total, 8 up, 8 in 2024-02-05T06:48:52.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:52 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-02-05T06:48:52.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:52 smithi028 ceph-mon[29442]: osdmap e372: 8 total, 8 up, 8 in 2024-02-05T06:48:52.551 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:52.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:52.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:52.686 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:52.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:52.724 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:52.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:52.825 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:52.857 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:52.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:52.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:52.950 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:48:52 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:48:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:48:52.955 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 73 16 104098 104082 89.0863 42.9375 0.00536034 0.0112193 2024-02-05T06:48:52.959 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:52.991 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:52.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:52.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:53.092 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:53.105 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:48:53.121 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:53.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:53.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:53.223 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:53.231 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-02-05T06:48:53.258 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:53.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:53.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:53.360 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:53.384 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:53.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:53.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:53.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:53 smithi028 ceph-mon[29442]: osdmap e373: 8 total, 8 up, 8 in 2024-02-05T06:48:53.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:53 smithi028 ceph-mon[29442]: pgmap v735: 168 pgs: 8 peering, 10 unknown, 2 active+clean+scrubbing+deep, 148 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.7 KiB/s rd, 78 MiB/s wr, 1.24k op/s 2024-02-05T06:48:53.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:53 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-05T06:48:53.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:53 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-05T06:48:53.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:53 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/720459233' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:48:53.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:53 smithi028 ceph-mon[37984]: osdmap e373: 8 total, 8 up, 8 in 2024-02-05T06:48:53.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:53 smithi028 ceph-mon[37984]: pgmap v735: 168 pgs: 8 peering, 10 unknown, 2 active+clean+scrubbing+deep, 148 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.7 KiB/s rd, 78 MiB/s wr, 1.24k op/s 2024-02-05T06:48:53.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:53 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-05T06:48:53.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:53 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-05T06:48:53.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:53 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/720459233' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:48:53.486 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:53.510 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:53.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:53.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:53.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:53.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:53.640 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:53.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:53.735 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:48:53 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:48:53.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:53 smithi167 ceph-mon[32774]: osdmap e373: 8 total, 8 up, 8 in 2024-02-05T06:48:53.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:53 smithi167 ceph-mon[32774]: pgmap v735: 168 pgs: 8 peering, 10 unknown, 2 active+clean+scrubbing+deep, 148 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.7 KiB/s rd, 78 MiB/s wr, 1.24k op/s 2024-02-05T06:48:53.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:53 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-05T06:48:53.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:53 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-02-05T06:48:53.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:53 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/720459233' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:48:53.741 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:53.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:53.766 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:53.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:53.867 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:53.892 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:53.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:53.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:53.955 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 74 16 104406 104390 88.1427 19.25 0.00233091 0.0112354 2024-02-05T06:48:53.994 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:54.018 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:54.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:54.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:54.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:54.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:54.158 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:54.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:54.259 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:54.284 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:54.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:54.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:54.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:54.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:54.410 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:54.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:54.511 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:54.525 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:54 smithi028 ceph-mon[37984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:48:54.525 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:54 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-02-05T06:48:54.525 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:54 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/720459233' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:48:54.525 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:54 smithi028 ceph-mon[37984]: osdmap e374: 8 total, 8 up, 8 in 2024-02-05T06:48:54.525 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:54 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:48:54.525 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:54 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:48:54.525 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:54 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:48:54.525 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:54 smithi028 ceph-mon[37984]: osdmap e375: 8 total, 8 up, 8 in 2024-02-05T06:48:54.525 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:54 smithi028 ceph-mon[29442]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:48:54.526 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:54 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-02-05T06:48:54.526 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:54 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/720459233' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:48:54.526 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:54 smithi028 ceph-mon[29442]: osdmap e374: 8 total, 8 up, 8 in 2024-02-05T06:48:54.526 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:54 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:48:54.526 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:54 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:48:54.526 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:54 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:48:54.526 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:54 smithi028 ceph-mon[29442]: osdmap e375: 8 total, 8 up, 8 in 2024-02-05T06:48:54.537 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:54.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:54.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:54.577 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:48:54.577 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-02-05T06:48:54.577 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-02-05T06:48:54.578 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-02-05T06:48:54.578 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.578 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:30036 8.c 8:3ade523f:::benchmark_data_smithi028_105698_object30035:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-02-05T06:48:54.578 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:07.595037+0000", 2024-02-05T06:48:54.578 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 526.97215498000003, 2024-02-05T06:48:54.578 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0589549119999999, 2024-02-05T06:48:54.578 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:48:54.578 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:54.578 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:48:54.578 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:48:54.578 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:48:54.578 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 30036 2024-02-05T06:48:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:48:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:48:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.595037+0000", 2024-02-05T06:48:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:48:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.595037+0000", 2024-02-05T06:48:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.579 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:48:54.580 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.595040+0000", 2024-02-05T06:48:54.580 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1229999999999998e-06 2024-02-05T06:48:54.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.580 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.580 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:48:54.580 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.595064+0000", 2024-02-05T06:48:54.580 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4287999999999999e-05 2024-02-05T06:48:54.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.580 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.580 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:48:54.580 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.595077+0000", 2024-02-05T06:48:54.580 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3159000000000001e-05 2024-02-05T06:48:54.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.580 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.581 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:48:54.581 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.595086+0000", 2024-02-05T06:48:54.581 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6780000000000006e-06 2024-02-05T06:48:54.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.581 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.581 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.581 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.595321+0000", 2024-02-05T06:48:54.581 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00023525099999999999 2024-02-05T06:48:54.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.581 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.581 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:48:54.581 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.595334+0000", 2024-02-05T06:48:54.581 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3071000000000001e-05 2024-02-05T06:48:54.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.581 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.582 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.582 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:08.563056+0000", 2024-02-05T06:48:54.582 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.96772240499999995 2024-02-05T06:48:54.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.582 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.582 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:48:54.582 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:08.563070+0000", 2024-02-05T06:48:54.582 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3611000000000001e-05 2024-02-05T06:48:54.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.582 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.582 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.582 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.646864+0000", 2024-02-05T06:48:54.582 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.083793934 2024-02-05T06:48:54.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.583 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.583 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:48:54.583 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.647049+0000", 2024-02-05T06:48:54.583 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018533800000000001 2024-02-05T06:48:54.583 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.583 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.583 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4", 2024-02-05T06:48:54.583 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.647246+0000", 2024-02-05T06:48:54.583 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019643199999999999 2024-02-05T06:48:54.583 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.583 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.583 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:48:54.583 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.650769+0000", 2024-02-05T06:48:54.583 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0035230880000000002 2024-02-05T06:48:54.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.584 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.584 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.584 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.651480+0000", 2024-02-05T06:48:54.584 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00071135899999999995 2024-02-05T06:48:54.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.584 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.584 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.584 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.652111+0000", 2024-02-05T06:48:54.584 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00063044499999999999 2024-02-05T06:48:54.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.584 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.584 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:48:54.585 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.652396+0000", 2024-02-05T06:48:54.585 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028486199999999998 2024-02-05T06:48:54.585 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.585 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.585 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:48:54.585 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.653992+0000", 2024-02-05T06:48:54.585 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001596868 2024-02-05T06:48:54.585 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.585 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:48:54.585 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.585 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.585 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.585 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:30042 8.c 8:3c2527a0:::benchmark_data_smithi028_105698_object30041:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-02-05T06:48:54.585 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:07.601267+0000", 2024-02-05T06:48:54.586 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 526.96592513300004, 2024-02-05T06:48:54.586 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.054788839, 2024-02-05T06:48:54.586 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:48:54.586 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:54.586 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:48:54.586 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:48:54.586 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:48:54.586 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 30042 2024-02-05T06:48:54.586 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.586 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:48:54.586 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.586 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:48:54.586 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.601267+0000", 2024-02-05T06:48:54.586 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.587 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.587 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:48:54.587 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.601267+0000", 2024-02-05T06:48:54.587 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.587 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.587 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:48:54.587 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.601270+0000", 2024-02-05T06:48:54.587 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6029999999999999e-06 2024-02-05T06:48:54.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.587 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.587 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:48:54.588 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.601292+0000", 2024-02-05T06:48:54.588 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2370000000000001e-05 2024-02-05T06:48:54.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.588 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.588 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:48:54.588 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.601308+0000", 2024-02-05T06:48:54.588 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5733e-05 2024-02-05T06:48:54.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.588 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.588 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:48:54.588 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.601316+0000", 2024-02-05T06:48:54.588 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.807e-06 2024-02-05T06:48:54.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.589 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.589 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.589 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.601368+0000", 2024-02-05T06:48:54.589 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2522999999999998e-05 2024-02-05T06:48:54.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.589 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.589 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:48:54.589 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:07.601378+0000", 2024-02-05T06:48:54.589 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9250000000000007e-06 2024-02-05T06:48:54.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.589 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.589 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.589 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:08.563096+0000", 2024-02-05T06:48:54.589 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.96171812499999998 2024-02-05T06:48:54.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.590 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.590 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:48:54.590 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:08.563104+0000", 2024-02-05T06:48:54.590 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8709999999999997e-06 2024-02-05T06:48:54.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.590 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.590 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.590 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.648708+0000", 2024-02-05T06:48:54.590 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0856032419999999 2024-02-05T06:48:54.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.590 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.590 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:48:54.591 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.648872+0000", 2024-02-05T06:48:54.591 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016487399999999999 2024-02-05T06:48:54.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.591 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.591 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4", 2024-02-05T06:48:54.591 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.649103+0000", 2024-02-05T06:48:54.591 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00023107999999999999 2024-02-05T06:48:54.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.591 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.591 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:48:54.591 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.650988+0000", 2024-02-05T06:48:54.591 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018845820000000001 2024-02-05T06:48:54.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.592 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.592 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.592 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.651528+0000", 2024-02-05T06:48:54.592 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00053995900000000001 2024-02-05T06:48:54.592 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.592 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.592 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.592 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.654061+0000", 2024-02-05T06:48:54.592 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025332409999999999 2024-02-05T06:48:54.592 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.592 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.592 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:48:54.592 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.654415+0000", 2024-02-05T06:48:54.593 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00035333199999999998 2024-02-05T06:48:54.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.593 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.593 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:48:54.593 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:09.656056+0000", 2024-02-05T06:48:54.593 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001641572 2024-02-05T06:48:54.593 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.593 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:48:54.593 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.593 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.593 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:44130 8.6 8:69a28b8b:::benchmark_data_smithi028_105698_object44129:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-02-05T06:48:54.594 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:26.029868+0000", 2024-02-05T06:48:54.594 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 508.53732427900002, 2024-02-05T06:48:54.594 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1427735559999999, 2024-02-05T06:48:54.594 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:48:54.594 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-02-05T06:48:54.594 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:48:54.594 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:48:54.594 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:48:54.594 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 44130 2024-02-05T06:48:54.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.594 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:48:54.594 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.594 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:48:54.594 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.029868+0000", 2024-02-05T06:48:54.594 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.595 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.595 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:48:54.595 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.029868+0000", 2024-02-05T06:48:54.595 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.595 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.595 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:48:54.595 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.029870+0000", 2024-02-05T06:48:54.595 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.255e-06 2024-02-05T06:48:54.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.595 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.595 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:48:54.595 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.029894+0000", 2024-02-05T06:48:54.595 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4043999999999999e-05 2024-02-05T06:48:54.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.596 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.596 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:48:54.596 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.029907+0000", 2024-02-05T06:48:54.596 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2666e-05 2024-02-05T06:48:54.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.596 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.596 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:48:54.596 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.029916+0000", 2024-02-05T06:48:54.596 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1579999999999994e-06 2024-02-05T06:48:54.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.596 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.596 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.596 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.031114+0000", 2024-02-05T06:48:54.597 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0011976700000000001 2024-02-05T06:48:54.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.597 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.597 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:48:54.597 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.031300+0000", 2024-02-05T06:48:54.597 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018568600000000001 2024-02-05T06:48:54.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.597 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.597 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,6", 2024-02-05T06:48:54.597 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.031461+0000", 2024-02-05T06:48:54.597 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000161099 2024-02-05T06:48:54.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.597 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.598 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:48:54.598 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.035193+0000", 2024-02-05T06:48:54.598 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0037321699999999999 2024-02-05T06:48:54.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.598 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.598 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.598 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.037176+0000", 2024-02-05T06:48:54.598 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0019833709999999998 2024-02-05T06:48:54.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.598 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.598 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.598 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.070634+0000", 2024-02-05T06:48:54.598 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.033457496000000003 2024-02-05T06:48:54.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.599 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.599 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:48:54.599 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.070646+0000", 2024-02-05T06:48:54.599 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2045000000000001e-05 2024-02-05T06:48:54.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.599 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.599 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.599 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.064553+0000", 2024-02-05T06:48:54.599 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.99390664399999995 2024-02-05T06:48:54.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.599 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.599 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:48:54.599 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.064566+0000", 2024-02-05T06:48:54.600 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3763e-05 2024-02-05T06:48:54.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.600 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.600 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.600 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.172577+0000", 2024-02-05T06:48:54.600 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1080111610000001 2024-02-05T06:48:54.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.600 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.600 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:48:54.600 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.172642+0000", 2024-02-05T06:48:54.600 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4327999999999998e-05 2024-02-05T06:48:54.600 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.600 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:48:54.600 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.601 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.601 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:44147 8.c 8:3925757e:::benchmark_data_smithi028_105698_object44146:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-02-05T06:48:54.601 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:26.048728+0000", 2024-02-05T06:48:54.601 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 508.51846471599998, 2024-02-05T06:48:54.601 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1723810800000001, 2024-02-05T06:48:54.601 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:48:54.601 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:54.601 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:48:54.601 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:48:54.601 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:48:54.601 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 44147 2024-02-05T06:48:54.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.601 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:48:54.601 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.601 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:48:54.601 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.048728+0000", 2024-02-05T06:48:54.602 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.602 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.602 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:48:54.602 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.048728+0000", 2024-02-05T06:48:54.602 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.602 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.602 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:48:54.602 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.048730+0000", 2024-02-05T06:48:54.602 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.199e-06 2024-02-05T06:48:54.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.602 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.602 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:48:54.603 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.048756+0000", 2024-02-05T06:48:54.603 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6001999999999999e-05 2024-02-05T06:48:54.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.603 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.603 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:48:54.603 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.048766+0000", 2024-02-05T06:48:54.603 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0249e-05 2024-02-05T06:48:54.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.603 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.603 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:48:54.604 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.048776+0000", 2024-02-05T06:48:54.604 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7720000000000006e-06 2024-02-05T06:48:54.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.604 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.604 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.604 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.068897+0000", 2024-02-05T06:48:54.604 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.020121066999999999 2024-02-05T06:48:54.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.604 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.604 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:48:54.604 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.068913+0000", 2024-02-05T06:48:54.604 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5508000000000001e-05 2024-02-05T06:48:54.604 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.605 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.605 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.605 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.210189+0000", 2024-02-05T06:48:54.605 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1412760829999999 2024-02-05T06:48:54.605 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.606 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.606 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:48:54.606 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.210349+0000", 2024-02-05T06:48:54.606 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016018 2024-02-05T06:48:54.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.606 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.606 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-02-05T06:48:54.607 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.210541+0000", 2024-02-05T06:48:54.607 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000192385 2024-02-05T06:48:54.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.607 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.607 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.217596+0000", 2024-02-05T06:48:54.607 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0070543439999999997 2024-02-05T06:48:54.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.607 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.607 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:48:54.607 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.217718+0000", 2024-02-05T06:48:54.607 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012252500000000001 2024-02-05T06:48:54.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.608 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.608 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.608 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.218772+0000", 2024-02-05T06:48:54.608 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00105378 2024-02-05T06:48:54.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.608 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.608 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:48:54.608 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.218866+0000", 2024-02-05T06:48:54.608 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3962000000000006e-05 2024-02-05T06:48:54.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.608 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.608 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:48:54.608 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.221109+0000", 2024-02-05T06:48:54.609 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022430240000000001 2024-02-05T06:48:54.609 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.609 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:48:54.609 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.609 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.609 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:44148 8.6 8:674244b6:::benchmark_data_smithi028_105698_object44147:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-02-05T06:48:54.609 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:26.065725+0000", 2024-02-05T06:48:54.609 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 508.50146791899999, 2024-02-05T06:48:54.609 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1232849570000001, 2024-02-05T06:48:54.609 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:48:54.609 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:54.610 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:48:54.610 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:48:54.610 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:48:54.610 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 44148 2024-02-05T06:48:54.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.610 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:48:54.610 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.610 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:48:54.610 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.065725+0000", 2024-02-05T06:48:54.610 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.610 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.610 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:48:54.610 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.065725+0000", 2024-02-05T06:48:54.610 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.611 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:48:54.611 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.065726+0000", 2024-02-05T06:48:54.611 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5260000000000001e-06 2024-02-05T06:48:54.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.611 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:48:54.611 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.065753+0000", 2024-02-05T06:48:54.611 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6485000000000001e-05 2024-02-05T06:48:54.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.611 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.611 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:48:54.612 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.065760+0000", 2024-02-05T06:48:54.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.763e-06 2024-02-05T06:48:54.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.612 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:48:54.612 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.065768+0000", 2024-02-05T06:48:54.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9699999999999999e-06 2024-02-05T06:48:54.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.612 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.612 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.612 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.070676+0000", 2024-02-05T06:48:54.612 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049074990000000001 2024-02-05T06:48:54.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:48:54.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.070684+0000", 2024-02-05T06:48:54.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9100000000000005e-06 2024-02-05T06:48:54.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.064596+0000", 2024-02-05T06:48:54.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.99391264999999995 2024-02-05T06:48:54.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.613 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:48:54.613 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.064605+0000", 2024-02-05T06:48:54.613 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.799e-06 2024-02-05T06:48:54.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.173907+0000", 2024-02-05T06:48:54.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1093021279999999 2024-02-05T06:48:54.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:48:54.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.174032+0000", 2024-02-05T06:48:54.614 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012484200000000001 2024-02-05T06:48:54.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.614 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.614 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 6,7", 2024-02-05T06:48:54.614 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.174188+0000", 2024-02-05T06:48:54.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000156121 2024-02-05T06:48:54.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.615 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:48:54.615 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.179323+0000", 2024-02-05T06:48:54.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0051351560000000001 2024-02-05T06:48:54.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.615 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.615 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.179359+0000", 2024-02-05T06:48:54.615 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5886999999999997e-05 2024-02-05T06:48:54.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.615 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.615 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.615 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.187345+0000", 2024-02-05T06:48:54.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0079857669999999995 2024-02-05T06:48:54.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.616 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:48:54.616 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.187383+0000", 2024-02-05T06:48:54.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7605000000000003e-05 2024-02-05T06:48:54.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.616 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.616 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:48:54.616 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.189010+0000", 2024-02-05T06:48:54.616 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0016268490000000001 2024-02-05T06:48:54.616 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.617 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:48:54.617 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.617 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.617 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:44151 8.c 8:355b2e83:::benchmark_data_smithi028_105698_object44150:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-02-05T06:48:54.617 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:26.080665+0000", 2024-02-05T06:48:54.617 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 508.48652703400001, 2024-02-05T06:48:54.617 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.142344483, 2024-02-05T06:48:54.617 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:48:54.617 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:54.617 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:48:54.617 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:48:54.617 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:48:54.618 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 44151 2024-02-05T06:48:54.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.618 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:48:54.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.618 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:48:54.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.080665+0000", 2024-02-05T06:48:54.618 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.618 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.618 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:48:54.618 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.080665+0000", 2024-02-05T06:48:54.618 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:48:54.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.080667+0000", 2024-02-05T06:48:54.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.888e-06 2024-02-05T06:48:54.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.619 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:48:54.619 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.080692+0000", 2024-02-05T06:48:54.619 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5080999999999999e-05 2024-02-05T06:48:54.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.619 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:48:54.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.080703+0000", 2024-02-05T06:48:54.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.045e-05 2024-02-05T06:48:54.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:48:54.620 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.080712+0000", 2024-02-05T06:48:54.620 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4180000000000003e-06 2024-02-05T06:48:54.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.620 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.620 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.080900+0000", 2024-02-05T06:48:54.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018818 2024-02-05T06:48:54.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.621 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:48:54.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.080912+0000", 2024-02-05T06:48:54.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1569000000000001e-05 2024-02-05T06:48:54.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.621 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.621 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.621 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.212374+0000", 2024-02-05T06:48:54.621 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.131462301 2024-02-05T06:48:54.621 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.622 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:48:54.622 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.212564+0000", 2024-02-05T06:48:54.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018985900000000001 2024-02-05T06:48:54.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.622 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-02-05T06:48:54.622 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.212729+0000", 2024-02-05T06:48:54.622 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016445199999999999 2024-02-05T06:48:54.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.622 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.622 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.217671+0000", 2024-02-05T06:48:54.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049428229999999998 2024-02-05T06:48:54.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:48:54.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.217724+0000", 2024-02-05T06:48:54.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2964000000000003e-05 2024-02-05T06:48:54.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.623 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.623 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.623 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.221148+0000", 2024-02-05T06:48:54.623 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034238480000000002 2024-02-05T06:48:54.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.624 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:48:54.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.221184+0000", 2024-02-05T06:48:54.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6154e-05 2024-02-05T06:48:54.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.624 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.624 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:48:54.624 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.223010+0000", 2024-02-05T06:48:54.624 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001825496 2024-02-05T06:48:54.624 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.624 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:48:54.624 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.625 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.625 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:44155 8.6 8:67a30c66:::benchmark_data_smithi028_105698_object44154:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-02-05T06:48:54.625 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:26.095757+0000", 2024-02-05T06:48:54.625 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 508.47143584200001, 2024-02-05T06:48:54.625 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0990290489999999, 2024-02-05T06:48:54.625 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:48:54.625 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:54.625 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:48:54.625 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:48:54.625 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:48:54.625 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 44155 2024-02-05T06:48:54.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.626 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:48:54.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.626 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:48:54.626 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.095757+0000", 2024-02-05T06:48:54.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.626 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:48:54.626 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.095757+0000", 2024-02-05T06:48:54.626 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.626 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.626 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.626 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:48:54.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.095759+0000", 2024-02-05T06:48:54.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.238e-06 2024-02-05T06:48:54.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:48:54.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.095784+0000", 2024-02-05T06:48:54.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5406999999999999e-05 2024-02-05T06:48:54.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.627 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:48:54.627 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.095794+0000", 2024-02-05T06:48:54.627 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9429999999999995e-06 2024-02-05T06:48:54.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.627 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:48:54.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.095801+0000", 2024-02-05T06:48:54.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9079999999999998e-06 2024-02-05T06:48:54.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.096034+0000", 2024-02-05T06:48:54.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000232921 2024-02-05T06:48:54.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.628 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:48:54.628 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.096044+0000", 2024-02-05T06:48:54.628 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6949999999999995e-06 2024-02-05T06:48:54.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.064631+0000", 2024-02-05T06:48:54.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.96858761900000001 2024-02-05T06:48:54.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:48:54.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.064640+0000", 2024-02-05T06:48:54.629 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5969999999999992e-06 2024-02-05T06:48:54.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.629 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.629 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.629 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.175581+0000", 2024-02-05T06:48:54.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1109413770000001 2024-02-05T06:48:54.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:48:54.630 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.175654+0000", 2024-02-05T06:48:54.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2161000000000002e-05 2024-02-05T06:48:54.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 6,7", 2024-02-05T06:48:54.630 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.175761+0000", 2024-02-05T06:48:54.630 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010715 2024-02-05T06:48:54.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.630 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.630 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:48:54.630 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.179379+0000", 2024-02-05T06:48:54.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0036188430000000001 2024-02-05T06:48:54.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.182192+0000", 2024-02-05T06:48:54.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028123649999999998 2024-02-05T06:48:54.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.193550+0000", 2024-02-05T06:48:54.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011357812 2024-02-05T06:48:54.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.631 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.631 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:48:54.631 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.193588+0000", 2024-02-05T06:48:54.631 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7843999999999999e-05 2024-02-05T06:48:54.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.632 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:48:54.632 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.194786+0000", 2024-02-05T06:48:54.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001198169 2024-02-05T06:48:54.632 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.632 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:48:54.632 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.632 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.632 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:44158 8.c 8:32d71fe1:::benchmark_data_smithi028_105698_object44157:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-02-05T06:48:54.632 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:26.103850+0000", 2024-02-05T06:48:54.632 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 508.46334286899997, 2024-02-05T06:48:54.632 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1699483719999999, 2024-02-05T06:48:54.633 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:48:54.633 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:54.633 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:48:54.633 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:48:54.633 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:48:54.633 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 44158 2024-02-05T06:48:54.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.633 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:48:54.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.633 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:48:54.633 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.103850+0000", 2024-02-05T06:48:54.633 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.633 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:48:54.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.103850+0000", 2024-02-05T06:48:54.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:48:54.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.103852+0000", 2024-02-05T06:48:54.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2850000000000001e-06 2024-02-05T06:48:54.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.634 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:48:54.634 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.103878+0000", 2024-02-05T06:48:54.634 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5979999999999999e-05 2024-02-05T06:48:54.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.634 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:48:54.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.103889+0000", 2024-02-05T06:48:54.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1473e-05 2024-02-05T06:48:54.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:48:54.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.103898+0000", 2024-02-05T06:48:54.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6559999999999998e-06 2024-02-05T06:48:54.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.635 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.635 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.103975+0000", 2024-02-05T06:48:54.635 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7322000000000003e-05 2024-02-05T06:48:54.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.635 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.636 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:48:54.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:26.103990+0000", 2024-02-05T06:48:54.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4572e-05 2024-02-05T06:48:54.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.636 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.215210+0000", 2024-02-05T06:48:54.636 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1112202360000001 2024-02-05T06:48:54.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.636 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.636 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:48:54.636 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.215353+0000", 2024-02-05T06:48:54.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014238200000000001 2024-02-05T06:48:54.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-02-05T06:48:54.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.215466+0000", 2024-02-05T06:48:54.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011366699999999999 2024-02-05T06:48:54.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:48:54.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.218079+0000", 2024-02-05T06:48:54.637 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026129769999999998 2024-02-05T06:48:54.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.637 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.637 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.637 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.223830+0000", 2024-02-05T06:48:54.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0057504670000000004 2024-02-05T06:48:54.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.271247+0000", 2024-02-05T06:48:54.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.047417775000000002 2024-02-05T06:48:54.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:48:54.638 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.271316+0000", 2024-02-05T06:48:54.638 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8305000000000004e-05 2024-02-05T06:48:54.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.638 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:48:54.639 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.273798+0000", 2024-02-05T06:48:54.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024822749999999999 2024-02-05T06:48:54.639 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.639 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:48:54.639 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.639 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.639 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:44164 8.6 8:6695897a:::benchmark_data_smithi028_105698_object44163:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-02-05T06:48:54.639 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:40:27.204120+0000", 2024-02-05T06:48:54.639 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 507.36307279200003, 2024-02-05T06:48:54.639 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.005835, 2024-02-05T06:48:54.639 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:48:54.639 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:54.639 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:48:54.640 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:48:54.640 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:48:54.640 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 44164 2024-02-05T06:48:54.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.640 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:48:54.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:48:54.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.204120+0000", 2024-02-05T06:48:54.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.640 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.640 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:48:54.640 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.204120+0000", 2024-02-05T06:48:54.640 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:48:54.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.204121+0000", 2024-02-05T06:48:54.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7090000000000001e-06 2024-02-05T06:48:54.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:48:54.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.204148+0000", 2024-02-05T06:48:54.641 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6495e-05 2024-02-05T06:48:54.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.641 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.641 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:48:54.641 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.204163+0000", 2024-02-05T06:48:54.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5177000000000001e-05 2024-02-05T06:48:54.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:48:54.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.204172+0000", 2024-02-05T06:48:54.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4800000000000001e-06 2024-02-05T06:48:54.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.642 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.209707+0000", 2024-02-05T06:48:54.642 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0055354000000000002 2024-02-05T06:48:54.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.642 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.642 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:48:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:27.209716+0000", 2024-02-05T06:48:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.5179999999999993e-06 2024-02-05T06:48:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.176460+0000", 2024-02-05T06:48:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.96674394900000005 2024-02-05T06:48:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:48:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.176579+0000", 2024-02-05T06:48:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000118937 2024-02-05T06:48:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.643 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 6,7", 2024-02-05T06:48:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.176709+0000", 2024-02-05T06:48:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012960200000000001 2024-02-05T06:48:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:48:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.181144+0000", 2024-02-05T06:48:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0044347759999999996 2024-02-05T06:48:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.183642+0000", 2024-02-05T06:48:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024978779999999998 2024-02-05T06:48:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.644 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.194820+0000", 2024-02-05T06:48:54.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011178641 2024-02-05T06:48:54.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:48:54.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.207712+0000", 2024-02-05T06:48:54.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012892173 2024-02-05T06:48:54.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.645 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.645 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:48:54.645 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:40:28.209955+0000", 2024-02-05T06:48:54.645 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022422649999999998 2024-02-05T06:48:54.645 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.645 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:48:54.645 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.646 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:106739 8.8 8:117d8294:::benchmark_data_smithi028_105698_object17464:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e248)", 2024-02-05T06:48:54.646 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:41:43.397529+0000", 2024-02-05T06:48:54.646 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 431.16966330399998, 2024-02-05T06:48:54.646 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1348037630000001, 2024-02-05T06:48:54.646 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:48:54.646 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:54.646 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:48:54.646 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:48:54.646 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:48:54.646 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 106739 2024-02-05T06:48:54.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.646 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:48:54.646 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:48:54.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397529+0000", 2024-02-05T06:48:54.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:48:54.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397529+0000", 2024-02-05T06:48:54.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.647 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.647 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:48:54.647 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397531+0000", 2024-02-05T06:48:54.647 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.567e-06 2024-02-05T06:48:54.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:48:54.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397535+0000", 2024-02-05T06:48:54.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4399999999999998e-06 2024-02-05T06:48:54.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:48:54.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397543+0000", 2024-02-05T06:48:54.648 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.61e-06 2024-02-05T06:48:54.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.648 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.648 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:48:54.648 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397551+0000", 2024-02-05T06:48:54.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6640000000000004e-06 2024-02-05T06:48:54.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397656+0000", 2024-02-05T06:48:54.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000104753 2024-02-05T06:48:54.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:48:54.649 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397669+0000", 2024-02-05T06:48:54.649 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3179e-05 2024-02-05T06:48:54.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.649 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.649 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.497144+0000", 2024-02-05T06:48:54.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0994749159999999 2024-02-05T06:48:54.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:48:54.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.497289+0000", 2024-02-05T06:48:54.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014480499999999999 2024-02-05T06:48:54.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.650 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,5,7", 2024-02-05T06:48:54.650 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.497433+0000", 2024-02-05T06:48:54.650 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000144034 2024-02-05T06:48:54.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.650 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:48:54.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.499403+0000", 2024-02-05T06:48:54.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0019702550000000002 2024-02-05T06:48:54.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.500873+0000", 2024-02-05T06:48:54.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001469162 2024-02-05T06:48:54.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.651 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.651 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.502297+0000", 2024-02-05T06:48:54.651 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001424252 2024-02-05T06:48:54.651 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.651 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.532124+0000", 2024-02-05T06:48:54.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029826930000000001 2024-02-05T06:48:54.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:48:54.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.532193+0000", 2024-02-05T06:48:54.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9559000000000003e-05 2024-02-05T06:48:54.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.652 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.652 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:48:54.652 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.532333+0000", 2024-02-05T06:48:54.652 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013963699999999999 2024-02-05T06:48:54.653 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.653 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:48:54.653 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.653 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.653 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.653 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:106744 8.8 8:11795b7a:::benchmark_data_smithi028_105698_object17469:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e248)", 2024-02-05T06:48:54.653 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:41:43.397570+0000", 2024-02-05T06:48:54.653 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 431.16962248900001, 2024-02-05T06:48:54.653 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1349640889999999, 2024-02-05T06:48:54.653 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:48:54.653 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:54.653 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:48:54.653 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:48:54.653 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:48:54.654 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 106744 2024-02-05T06:48:54.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.654 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:48:54.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:48:54.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397570+0000", 2024-02-05T06:48:54.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.654 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:48:54.654 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397570+0000", 2024-02-05T06:48:54.654 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.654 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:48:54.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397571+0000", 2024-02-05T06:48:54.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4600000000000003e-07 2024-02-05T06:48:54.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:48:54.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397574+0000", 2024-02-05T06:48:54.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4199999999999999e-06 2024-02-05T06:48:54.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.655 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.655 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:48:54.655 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397581+0000", 2024-02-05T06:48:54.655 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.455e-06 2024-02-05T06:48:54.655 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:48:54.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397587+0000", 2024-02-05T06:48:54.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9619999999999998e-06 2024-02-05T06:48:54.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397712+0000", 2024-02-05T06:48:54.656 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012582399999999999 2024-02-05T06:48:54.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.656 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:48:54.656 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:43.397723+0000", 2024-02-05T06:48:54.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0026e-05 2024-02-05T06:48:54.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.499468+0000", 2024-02-05T06:48:54.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1017459270000001 2024-02-05T06:48:54.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:48:54.657 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.499545+0000", 2024-02-05T06:48:54.657 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6618000000000004e-05 2024-02-05T06:48:54.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.657 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.657 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,5,7", 2024-02-05T06:48:54.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.499616+0000", 2024-02-05T06:48:54.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0776000000000001e-05 2024-02-05T06:48:54.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:48:54.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.500824+0000", 2024-02-05T06:48:54.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0012077660000000001 2024-02-05T06:48:54.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.658 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.501234+0000", 2024-02-05T06:48:54.658 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000410651 2024-02-05T06:48:54.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.658 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.658 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.502417+0000", 2024-02-05T06:48:54.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0011827910000000001 2024-02-05T06:48:54.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.532375+0000", 2024-02-05T06:48:54.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.029957874999999998 2024-02-05T06:48:54.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:48:54.659 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.532432+0000", 2024-02-05T06:48:54.659 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7383000000000003e-05 2024-02-05T06:48:54.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.659 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.659 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:48:54.660 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:44.532534+0000", 2024-02-05T06:48:54.660 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010176899999999999 2024-02-05T06:48:54.660 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.660 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:48:54.660 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.660 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.660 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.660 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:107326 8.6 8:605b4dc7:::benchmark_data_smithi028_105698_object18051:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-02-05T06:48:54.660 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:41:45.056053+0000", 2024-02-05T06:48:54.660 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 429.51113951299999, 2024-02-05T06:48:54.663 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.012795213, 2024-02-05T06:48:54.664 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:48:54.664 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:54.664 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:48:54.665 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:48:54.665 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:48:54.665 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 107326 2024-02-05T06:48:54.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.665 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:48:54.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:48:54.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.056053+0000", 2024-02-05T06:48:54.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.665 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.665 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.665 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:48:54.665 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.056053+0000", 2024-02-05T06:48:54.665 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.666 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:48:54.666 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.056054+0000", 2024-02-05T06:48:54.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.7400000000000002e-07 2024-02-05T06:48:54.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.666 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:48:54.666 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.056057+0000", 2024-02-05T06:48:54.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4460000000000001e-06 2024-02-05T06:48:54.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.666 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.666 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:48:54.666 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.056064+0000", 2024-02-05T06:48:54.666 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3010000000000001e-06 2024-02-05T06:48:54.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:48:54.667 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.056072+0000", 2024-02-05T06:48:54.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.559e-06 2024-02-05T06:48:54.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.667 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.065347+0000", 2024-02-05T06:48:54.667 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0092747249999999993 2024-02-05T06:48:54.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.667 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.667 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:48:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.065357+0000", 2024-02-05T06:48:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9410000000000002e-06 2024-02-05T06:48:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.062162+0000", 2024-02-05T06:48:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.99680566999999998 2024-02-05T06:48:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:48:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.062297+0000", 2024-02-05T06:48:54.668 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000134752 2024-02-05T06:48:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 6,7", 2024-02-05T06:48:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.062396+0000", 2024-02-05T06:48:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8632e-05 2024-02-05T06:48:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.068630+0000", 2024-02-05T06:48:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062346650000000003 2024-02-05T06:48:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.669 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:48:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.068646+0000", 2024-02-05T06:48:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5616000000000001e-05 2024-02-05T06:48:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.068758+0000", 2024-02-05T06:48:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011232500000000001 2024-02-05T06:48:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:48:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.068811+0000", 2024-02-05T06:48:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2911000000000002e-05 2024-02-05T06:48:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:48:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.068848+0000", 2024-02-05T06:48:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6896e-05 2024-02-05T06:48:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:48:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:107355 8.6 8:6f7c225d:::benchmark_data_smithi028_105698_object18080:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-02-05T06:48:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:41:45.094245+0000", 2024-02-05T06:48:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 429.472947885, 2024-02-05T06:48:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.97918498600000003, 2024-02-05T06:48:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:48:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:48:54.671 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:48:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:48:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 107355 2024-02-05T06:48:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:48:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:48:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.094245+0000", 2024-02-05T06:48:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:48:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.094245+0000", 2024-02-05T06:48:54.672 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:48:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.094247+0000", 2024-02-05T06:48:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9690000000000001e-06 2024-02-05T06:48:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:48:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.094252+0000", 2024-02-05T06:48:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7570000000000001e-06 2024-02-05T06:48:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:48:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.094273+0000", 2024-02-05T06:48:54.673 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0781e-05 2024-02-05T06:48:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:48:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.094282+0000", 2024-02-05T06:48:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4889999999999995e-06 2024-02-05T06:48:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.094346+0000", 2024-02-05T06:48:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4120000000000003e-05 2024-02-05T06:48:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:48:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.094356+0000", 2024-02-05T06:48:54.674 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0767999999999999e-05 2024-02-05T06:48:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.064770+0000", 2024-02-05T06:48:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.97041343499999999 2024-02-05T06:48:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:48:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.064863+0000", 2024-02-05T06:48:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3134000000000006e-05 2024-02-05T06:48:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 6,7", 2024-02-05T06:48:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.064948+0000", 2024-02-05T06:48:54.675 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5088999999999998e-05 2024-02-05T06:48:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.068889+0000", 2024-02-05T06:48:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039410460000000001 2024-02-05T06:48:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:48:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.069069+0000", 2024-02-05T06:48:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000179684 2024-02-05T06:48:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.676 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.073324+0000", 2024-02-05T06:48:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.004254908 2024-02-05T06:48:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:48:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.073380+0000", 2024-02-05T06:48:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6440999999999997e-05 2024-02-05T06:48:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:48:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.073430+0000", 2024-02-05T06:48:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9364999999999999e-05 2024-02-05T06:48:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:48:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.678 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.15144.0:107363 8.6 8:679b97e2:::benchmark_data_smithi028_105698_object18088:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e250)", 2024-02-05T06:48:54.678 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:41:45.099343+0000", 2024-02-05T06:48:54.678 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 429.46784930299998, 2024-02-05T06:48:54.678 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.97440879599999997, 2024-02-05T06:48:54.678 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:48:54.678 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:54.678 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:48:54.678 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.15144", 2024-02-05T06:48:54.678 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:48:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 107363 2024-02-05T06:48:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:48:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:48:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.099343+0000", 2024-02-05T06:48:54.680 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:48:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.099343+0000", 2024-02-05T06:48:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:48:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.099344+0000", 2024-02-05T06:48:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3200000000000004e-07 2024-02-05T06:48:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:48:54.681 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.099348+0000", 2024-02-05T06:48:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1370000000000004e-06 2024-02-05T06:48:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:48:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.099355+0000", 2024-02-05T06:48:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.798e-06 2024-02-05T06:48:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:48:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.099362+0000", 2024-02-05T06:48:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6800000000000004e-06 2024-02-05T06:48:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.682 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.099506+0000", 2024-02-05T06:48:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014402600000000001 2024-02-05T06:48:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:48:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:45.099513+0000", 2024-02-05T06:48:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0040000000000002e-06 2024-02-05T06:48:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.683 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.069139+0000", 2024-02-05T06:48:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.96962602499999995 2024-02-05T06:48:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:48:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.069231+0000", 2024-02-05T06:48:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2391000000000004e-05 2024-02-05T06:48:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 6,7", 2024-02-05T06:48:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.069352+0000", 2024-02-05T06:48:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000120943 2024-02-05T06:48:54.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:48:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.073451+0000", 2024-02-05T06:48:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0040991389999999999 2024-02-05T06:48:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.073632+0000", 2024-02-05T06:48:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000181044 2024-02-05T06:48:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.685 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.073677+0000", 2024-02-05T06:48:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4539999999999997e-05 2024-02-05T06:48:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:48:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.073723+0000", 2024-02-05T06:48:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5890999999999998e-05 2024-02-05T06:48:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:48:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:41:46.073752+0000", 2024-02-05T06:48:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9346e-05 2024-02-05T06:48:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.686 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:48:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.45587.0:103389 10.6 e371/345 10:6a1ea5a8:::benchmark_data_smithi028_154709_object103388:head v 371'6465, mlcod=371'6462)", 2024-02-05T06:48:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:48:51.200936+0000", 2024-02-05T06:48:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 3.3662563479999998, 2024-02-05T06:48:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.27161048500000001, 2024-02-05T06:48:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:48:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-02-05T06:48:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:48:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:48:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.200936+0000", 2024-02-05T06:48:54.687 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:48:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.200936+0000", 2024-02-05T06:48:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:48:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.200938+0000", 2024-02-05T06:48:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.483e-06 2024-02-05T06:48:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.688 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:48:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.201052+0000", 2024-02-05T06:48:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000114824 2024-02-05T06:48:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:48:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.201064+0000", 2024-02-05T06:48:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1712999999999999e-05 2024-02-05T06:48:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:48:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.201072+0000", 2024-02-05T06:48:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8679999999999999e-06 2024-02-05T06:48:54.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.201436+0000", 2024-02-05T06:48:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000363927 2024-02-05T06:48:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:48:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.201454+0000", 2024-02-05T06:48:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8012999999999998e-05 2024-02-05T06:48:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:48:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.472547+0000", 2024-02-05T06:48:54.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.27109265700000001 2024-02-05T06:48:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:48:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.45587.0:103385 10.1 e371/345 10:887dabf2:::benchmark_data_smithi028_154709_object103384:head v 371'6460, mlcod=371'6456)", 2024-02-05T06:48:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:48:51.201198+0000", 2024-02-05T06:48:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 3.365994267, 2024-02-05T06:48:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.242855927, 2024-02-05T06:48:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:48:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-02-05T06:48:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:48:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:48:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.201198+0000", 2024-02-05T06:48:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:48:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.201198+0000", 2024-02-05T06:48:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:48:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.201199+0000", 2024-02-05T06:48:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2079999999999999e-06 2024-02-05T06:48:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:48:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.201240+0000", 2024-02-05T06:48:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0222000000000002e-05 2024-02-05T06:48:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:48:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.201250+0000", 2024-02-05T06:48:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0519e-05 2024-02-05T06:48:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:48:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.201263+0000", 2024-02-05T06:48:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3328e-05 2024-02-05T06:48:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.201335+0000", 2024-02-05T06:48:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1188999999999999e-05 2024-02-05T06:48:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:48:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.201355+0000", 2024-02-05T06:48:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9967999999999999e-05 2024-02-05T06:48:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:48:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.444054+0000", 2024-02-05T06:48:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.24269949299999999 2024-02-05T06:48:54.694 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:48:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.45587.0:103394 10.1 e371/345 10:85e12e54:::benchmark_data_smithi028_154709_object103393:head v 371'6461, mlcod=371'6458)", 2024-02-05T06:48:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:48:51.208062+0000", 2024-02-05T06:48:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 3.359130634, 2024-02-05T06:48:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.23599954200000001, 2024-02-05T06:48:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:48:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-02-05T06:48:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:48:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.695 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:48:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.208062+0000", 2024-02-05T06:48:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:48:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.208062+0000", 2024-02-05T06:48:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:48:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.208063+0000", 2024-02-05T06:48:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0400000000000005e-07 2024-02-05T06:48:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:48:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.208131+0000", 2024-02-05T06:48:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8509000000000004e-05 2024-02-05T06:48:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:48:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.208138+0000", 2024-02-05T06:48:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.109e-06 2024-02-05T06:48:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:48:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.208145+0000", 2024-02-05T06:48:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3899999999999998e-06 2024-02-05T06:48:54.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.210304+0000", 2024-02-05T06:48:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0021594969999999998 2024-02-05T06:48:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:48:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.210323+0000", 2024-02-05T06:48:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9049999999999999e-05 2024-02-05T06:48:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:48:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.444061+0000", 2024-02-05T06:48:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.23373818299999999 2024-02-05T06:48:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:48:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(osd.6.0:114793 10.a e371/345 10:58d945df:::benchmark_data_smithi028_154709_object52047:head v 371'8748, mlcod=371'8744)", 2024-02-05T06:48:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:48:51.208341+0000", 2024-02-05T06:48:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 3.3588511670000001, 2024-02-05T06:48:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.182475836, 2024-02-05T06:48:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:48:54.699 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:48:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:48:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.208341+0000", 2024-02-05T06:48:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:48:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.208341+0000", 2024-02-05T06:48:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:48:54.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.208343+0000", 2024-02-05T06:48:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3540000000000001e-06 2024-02-05T06:48:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:48:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.208350+0000", 2024-02-05T06:48:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9149999999999999e-06 2024-02-05T06:48:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:48:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.208359+0000", 2024-02-05T06:48:54.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0650000000000005e-06 2024-02-05T06:48:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:48:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.208365+0000", 2024-02-05T06:48:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6640000000000001e-06 2024-02-05T06:48:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.208509+0000", 2024-02-05T06:48:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014384599999999999 2024-02-05T06:48:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:48:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.208528+0000", 2024-02-05T06:48:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8515000000000001e-05 2024-02-05T06:48:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:48:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.390782+0000", 2024-02-05T06:48:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.182254207 2024-02-05T06:48:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:48:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.390817+0000", 2024-02-05T06:48:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5269999999999999e-05 2024-02-05T06:48:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.704 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:48:54.704 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.704 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45587.0:103404 10.18 10:1859de67:::benchmark_data_smithi028_154709_object103403:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-02-05T06:48:54.704 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:48:51.238167+0000", 2024-02-05T06:48:54.704 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 3.329025321, 2024-02-05T06:48:54.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0233144270000001, 2024-02-05T06:48:54.704 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:48:54.704 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:54.704 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:48:54.704 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45587", 2024-02-05T06:48:54.704 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:48:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 103404 2024-02-05T06:48:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:48:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:48:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.238167+0000", 2024-02-05T06:48:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:48:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.238167+0000", 2024-02-05T06:48:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:48:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.238171+0000", 2024-02-05T06:48:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5140000000000002e-06 2024-02-05T06:48:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:48:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.238205+0000", 2024-02-05T06:48:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4638000000000003e-05 2024-02-05T06:48:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:48:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.238224+0000", 2024-02-05T06:48:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8281e-05 2024-02-05T06:48:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:48:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.238236+0000", 2024-02-05T06:48:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2145e-05 2024-02-05T06:48:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.465383+0000", 2024-02-05T06:48:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.22714728300000001 2024-02-05T06:48:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:48:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.465394+0000", 2024-02-05T06:48:54.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.095e-05 2024-02-05T06:48:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:52.252859+0000", 2024-02-05T06:48:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.78746483899999997 2024-02-05T06:48:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:48:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:52.253004+0000", 2024-02-05T06:48:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014537400000000001 2024-02-05T06:48:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,7", 2024-02-05T06:48:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:52.253310+0000", 2024-02-05T06:48:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0003056 2024-02-05T06:48:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:48:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:52.254727+0000", 2024-02-05T06:48:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0014170809999999999 2024-02-05T06:48:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:52.255917+0000", 2024-02-05T06:48:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0011899739999999999 2024-02-05T06:48:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:52.261386+0000", 2024-02-05T06:48:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0054691009999999997 2024-02-05T06:48:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:48:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:52.261444+0000", 2024-02-05T06:48:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8294999999999997e-05 2024-02-05T06:48:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:48:54.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:52.261482+0000", 2024-02-05T06:48:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7351999999999997e-05 2024-02-05T06:48:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:48:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45587.0:103406 10.4 10:265740a8:::benchmark_data_smithi028_154709_object103405:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-02-05T06:48:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:48:51.246525+0000", 2024-02-05T06:48:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 3.3206670379999998, 2024-02-05T06:48:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.013782808, 2024-02-05T06:48:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:48:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:48:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:48:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45587", 2024-02-05T06:48:54.711 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:48:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 103406 2024-02-05T06:48:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:48:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:48:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.246525+0000", 2024-02-05T06:48:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:48:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.246525+0000", 2024-02-05T06:48:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:48:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:48:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.246528+0000", 2024-02-05T06:48:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5840000000000002e-06 2024-02-05T06:48:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:48:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.246561+0000", 2024-02-05T06:48:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2866e-05 2024-02-05T06:48:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:48:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.246580+0000", 2024-02-05T06:48:54.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8980000000000001e-05 2024-02-05T06:48:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:48:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.246593+0000", 2024-02-05T06:48:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2668e-05 2024-02-05T06:48:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.486572+0000", 2024-02-05T06:48:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.23997907199999999 2024-02-05T06:48:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:48:54.714 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.486587+0000", 2024-02-05T06:48:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5880999999999999e-05 2024-02-05T06:48:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:48:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:52.250119+0000", 2024-02-05T06:48:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.76353134700000003 2024-02-05T06:48:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:48:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:52.250266+0000", 2024-02-05T06:48:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014745399999999999 2024-02-05T06:48:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4", 2024-02-05T06:48:54.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:52.250606+0000", 2024-02-05T06:48:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00034009399999999999 2024-02-05T06:48:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:48:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:52.254555+0000", 2024-02-05T06:48:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039482240000000002 2024-02-05T06:48:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:52.254937+0000", 2024-02-05T06:48:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00038206900000000002 2024-02-05T06:48:54.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:48:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:52.260195+0000", 2024-02-05T06:48:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0052587420000000003 2024-02-05T06:48:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:48:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:52.260266+0000", 2024-02-05T06:48:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0658e-05 2024-02-05T06:48:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:48:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:48:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:48:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:52.260308+0000", 2024-02-05T06:48:54.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2169000000000002e-05 2024-02-05T06:48:54.718 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.718 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:48:54.718 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.718 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:48:54.718 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:48:54.718 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:48:54.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:54.720 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:54.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:54.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:54 smithi167 ceph-mon[32774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:48:54.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:54 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-02-05T06:48:54.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:54 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/720459233' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:48:54.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:54 smithi167 ceph-mon[32774]: osdmap e374: 8 total, 8 up, 8 in 2024-02-05T06:48:54.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:54 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:48:54.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:54 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:48:54.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:54 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:48:54.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:54 smithi167 ceph-mon[32774]: osdmap e375: 8 total, 8 up, 8 in 2024-02-05T06:48:54.820 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:54.847 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:54.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:54.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:54.949 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:54.955 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 75 16 104406 104390 86.9677 0 - 0.0112354 2024-02-05T06:48:54.974 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:54.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:54.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:55.076 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:55.103 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:55.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:55.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:55.204 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:55.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:55.234 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:55.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:55.335 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:55.362 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:55.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:55.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:55.464 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:55.494 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:55.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:55.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:55.596 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:55.633 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:55.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:55.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:55.651 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:55 smithi028 ceph-mon[29442]: pgmap v738: 168 pgs: 8 peering, 8 unknown, 2 active+clean+scrubbing+deep, 150 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 56 MiB/s wr, 897 op/s 2024-02-05T06:48:55.652 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:55 smithi028 ceph-mon[29442]: osdmap e376: 8 total, 8 up, 8 in 2024-02-05T06:48:55.652 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:55 smithi028 ceph-mon[37984]: pgmap v738: 168 pgs: 8 peering, 8 unknown, 2 active+clean+scrubbing+deep, 150 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 56 MiB/s wr, 897 op/s 2024-02-05T06:48:55.652 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:55 smithi028 ceph-mon[37984]: osdmap e376: 8 total, 8 up, 8 in 2024-02-05T06:48:55.734 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:55.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:55 smithi167 ceph-mon[32774]: pgmap v738: 168 pgs: 8 peering, 8 unknown, 2 active+clean+scrubbing+deep, 150 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 56 MiB/s wr, 897 op/s 2024-02-05T06:48:55.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:55 smithi167 ceph-mon[32774]: osdmap e376: 8 total, 8 up, 8 in 2024-02-05T06:48:55.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:55.781 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:55.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:55.882 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:55.920 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:55.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:55.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:55.955 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 76 16 105176 105160 86.4566 24.0625 0.0192793 0.0115618 2024-02-05T06:48:56.023 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:56.065 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:56.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:56.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:56.167 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:56.200 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:56.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:56.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:56.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:56.332 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T06:48:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:56.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:56.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:56.444 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:56.477 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:56.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:56.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:56.579 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:56.627 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:56.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:56.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:56.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[29442]: 10.11 starting backfill to osd.5 from (0'0,0'0] MAX to 373'6490 2024-02-05T06:48:56.658 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[29442]: 10.17 starting backfill to osd.4 from (0'0,0'0] MAX to 373'6595 2024-02-05T06:48:56.659 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[29442]: 10.10 starting backfill to osd.7 from (0'0,0'0] MAX to 373'6321 2024-02-05T06:48:56.659 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[29442]: 10.18 starting backfill to osd.1 from (0'0,0'0] MAX to 373'6463 2024-02-05T06:48:56.659 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[29442]: 10.19 starting backfill to osd.1 from (0'0,0'0] MAX to 373'6357 2024-02-05T06:48:56.659 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[29442]: 10.13 starting backfill to osd.1 from (0'0,0'0] MAX to 373'6584 2024-02-05T06:48:56.659 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[29442]: 10.15 starting backfill to osd.1 from (0'0,0'0] MAX to 373'6614 2024-02-05T06:48:56.659 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[29442]: 10.14 starting backfill to osd.1 from (0'0,0'0] MAX to 373'6410 2024-02-05T06:48:56.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[29442]: 10.12 starting backfill to osd.2 from (0'0,0'0] MAX to 373'6468 2024-02-05T06:48:56.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[29442]: 10.16 starting backfill to osd.1 from (0'0,0'0] MAX to 373'6507 2024-02-05T06:48:56.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[29442]: 10.15 starting backfill to osd.5 from (0'0,0'0] MAX to 373'6614 2024-02-05T06:48:56.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[29442]: 10.14 starting backfill to osd.2 from (0'0,0'0] MAX to 373'6410 2024-02-05T06:48:56.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[29442]: 10.18 starting backfill to osd.3 from (0'0,0'0] MAX to 373'6463 2024-02-05T06:48:56.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[29442]: 10.19 starting backfill to osd.3 from (0'0,0'0] MAX to 373'6357 2024-02-05T06:48:56.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[29442]: 10.16 starting backfill to osd.6 from (0'0,0'0] MAX to 373'6507 2024-02-05T06:48:56.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[29442]: 10.13 starting backfill to osd.7 from (0'0,0'0] MAX to 373'6584 2024-02-05T06:48:56.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[29442]: 10.19 starting backfill to osd.4 from (0'0,0'0] MAX to 373'6357 2024-02-05T06:48:56.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[29442]: 10.12 starting backfill to osd.5 from (0'0,0'0] MAX to 373'6468 2024-02-05T06:48:56.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[29442]: 10.12 starting backfill to osd.6 from (0'0,0'0] MAX to 373'6468 2024-02-05T06:48:56.662 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[37984]: 10.11 starting backfill to osd.5 from (0'0,0'0] MAX to 373'6490 2024-02-05T06:48:56.662 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[37984]: 10.17 starting backfill to osd.4 from (0'0,0'0] MAX to 373'6595 2024-02-05T06:48:56.662 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[37984]: 10.10 starting backfill to osd.7 from (0'0,0'0] MAX to 373'6321 2024-02-05T06:48:56.662 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[37984]: 10.18 starting backfill to osd.1 from (0'0,0'0] MAX to 373'6463 2024-02-05T06:48:56.662 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[37984]: 10.19 starting backfill to osd.1 from (0'0,0'0] MAX to 373'6357 2024-02-05T06:48:56.663 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[37984]: 10.13 starting backfill to osd.1 from (0'0,0'0] MAX to 373'6584 2024-02-05T06:48:56.663 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[37984]: 10.15 starting backfill to osd.1 from (0'0,0'0] MAX to 373'6614 2024-02-05T06:48:56.663 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[37984]: 10.14 starting backfill to osd.1 from (0'0,0'0] MAX to 373'6410 2024-02-05T06:48:56.663 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[37984]: 10.12 starting backfill to osd.2 from (0'0,0'0] MAX to 373'6468 2024-02-05T06:48:56.663 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[37984]: 10.16 starting backfill to osd.1 from (0'0,0'0] MAX to 373'6507 2024-02-05T06:48:56.663 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[37984]: 10.15 starting backfill to osd.5 from (0'0,0'0] MAX to 373'6614 2024-02-05T06:48:56.663 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[37984]: 10.14 starting backfill to osd.2 from (0'0,0'0] MAX to 373'6410 2024-02-05T06:48:56.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[37984]: 10.18 starting backfill to osd.3 from (0'0,0'0] MAX to 373'6463 2024-02-05T06:48:56.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[37984]: 10.19 starting backfill to osd.3 from (0'0,0'0] MAX to 373'6357 2024-02-05T06:48:56.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[37984]: 10.16 starting backfill to osd.6 from (0'0,0'0] MAX to 373'6507 2024-02-05T06:48:56.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[37984]: 10.13 starting backfill to osd.7 from (0'0,0'0] MAX to 373'6584 2024-02-05T06:48:56.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[37984]: 10.19 starting backfill to osd.4 from (0'0,0'0] MAX to 373'6357 2024-02-05T06:48:56.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[37984]: 10.12 starting backfill to osd.5 from (0'0,0'0] MAX to 373'6468 2024-02-05T06:48:56.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:56 smithi028 ceph-mon[37984]: 10.12 starting backfill to osd.6 from (0'0,0'0] MAX to 373'6468 2024-02-05T06:48:56.729 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:56.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:56 smithi167 ceph-mon[32774]: 10.11 starting backfill to osd.5 from (0'0,0'0] MAX to 373'6490 2024-02-05T06:48:56.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:56 smithi167 ceph-mon[32774]: 10.17 starting backfill to osd.4 from (0'0,0'0] MAX to 373'6595 2024-02-05T06:48:56.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:56 smithi167 ceph-mon[32774]: 10.10 starting backfill to osd.7 from (0'0,0'0] MAX to 373'6321 2024-02-05T06:48:56.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:56 smithi167 ceph-mon[32774]: 10.18 starting backfill to osd.1 from (0'0,0'0] MAX to 373'6463 2024-02-05T06:48:56.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:56 smithi167 ceph-mon[32774]: 10.19 starting backfill to osd.1 from (0'0,0'0] MAX to 373'6357 2024-02-05T06:48:56.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:56 smithi167 ceph-mon[32774]: 10.13 starting backfill to osd.1 from (0'0,0'0] MAX to 373'6584 2024-02-05T06:48:56.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:56 smithi167 ceph-mon[32774]: 10.15 starting backfill to osd.1 from (0'0,0'0] MAX to 373'6614 2024-02-05T06:48:56.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:56 smithi167 ceph-mon[32774]: 10.14 starting backfill to osd.1 from (0'0,0'0] MAX to 373'6410 2024-02-05T06:48:56.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:56 smithi167 ceph-mon[32774]: 10.12 starting backfill to osd.2 from (0'0,0'0] MAX to 373'6468 2024-02-05T06:48:56.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:56 smithi167 ceph-mon[32774]: 10.16 starting backfill to osd.1 from (0'0,0'0] MAX to 373'6507 2024-02-05T06:48:56.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:56 smithi167 ceph-mon[32774]: 10.15 starting backfill to osd.5 from (0'0,0'0] MAX to 373'6614 2024-02-05T06:48:56.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:56 smithi167 ceph-mon[32774]: 10.14 starting backfill to osd.2 from (0'0,0'0] MAX to 373'6410 2024-02-05T06:48:56.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:56 smithi167 ceph-mon[32774]: 10.18 starting backfill to osd.3 from (0'0,0'0] MAX to 373'6463 2024-02-05T06:48:56.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:56 smithi167 ceph-mon[32774]: 10.19 starting backfill to osd.3 from (0'0,0'0] MAX to 373'6357 2024-02-05T06:48:56.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:56 smithi167 ceph-mon[32774]: 10.16 starting backfill to osd.6 from (0'0,0'0] MAX to 373'6507 2024-02-05T06:48:56.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:56 smithi167 ceph-mon[32774]: 10.13 starting backfill to osd.7 from (0'0,0'0] MAX to 373'6584 2024-02-05T06:48:56.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:56 smithi167 ceph-mon[32774]: 10.19 starting backfill to osd.4 from (0'0,0'0] MAX to 373'6357 2024-02-05T06:48:56.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:56 smithi167 ceph-mon[32774]: 10.12 starting backfill to osd.5 from (0'0,0'0] MAX to 373'6468 2024-02-05T06:48:56.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:56 smithi167 ceph-mon[32774]: 10.12 starting backfill to osd.6 from (0'0,0'0] MAX to 373'6468 2024-02-05T06:48:56.764 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:56.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:56.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:56.806 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-05T06:48:56.866 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:56.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:56.902 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:56.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:56.956 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 77 16 106394 106378 86.3211 76.125 0.0071745 0.01158 2024-02-05T06:48:57.003 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:57.026 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:57.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:57.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:57.128 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:57.176 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:57.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:57.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:57.278 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:57.319 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:57.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:57.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:57.421 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:57.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:57 smithi028 ceph-mon[37984]: pgmap v740: 168 pgs: 8 peering, 8 unknown, 2 active+clean+scrubbing+deep, 150 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 6.0 MiB/s wr, 96 op/s 2024-02-05T06:48:57.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:57 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:48:57.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:57 smithi028 ceph-mon[29442]: pgmap v740: 168 pgs: 8 peering, 8 unknown, 2 active+clean+scrubbing+deep, 150 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 6.0 MiB/s wr, 96 op/s 2024-02-05T06:48:57.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:57 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:48:57.453 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:57.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:57.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:57.555 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:57.594 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:57.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:57.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:57.696 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:57.732 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:57.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:57.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:57.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:57 smithi167 ceph-mon[32774]: pgmap v740: 168 pgs: 8 peering, 8 unknown, 2 active+clean+scrubbing+deep, 150 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 6.0 MiB/s wr, 96 op/s 2024-02-05T06:48:57.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:57 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:48:57.834 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:57.895 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:48:57.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:57.931 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:57.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:57.960 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 78 16 107454 107438 86.0605 66.25 0.0130423 0.0116143 2024-02-05T06:48:58.032 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:58.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:58.072 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:58.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:58.129 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 4, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T06:48:58.129 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T06:48:58.129 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-02-05T06:48:58.129 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-02-05T06:48:58.129 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-02-05T06:48:58.130 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-02-05T06:48:58.174 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:58.223 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:58.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:58.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:58.325 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:48:58.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:58.364 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:58.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:48:58.465 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:58.506 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:58.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:58.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:58.607 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:48:58.648 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:58.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:58.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:48:58.717 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:48:58.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:58.793 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:58.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:58.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:58.874 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:48:58.895 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:48:58.929 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:58.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:58.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:48:58.959 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 79 16 108647 108631 85.9149 74.5625 0.0201456 0.0116347 2024-02-05T06:48:59.031 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:59.063 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:59.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:59.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:59.165 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:48:59.217 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:59.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:59.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:48:59.272 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:48:59.319 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:48:59.321 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:48:59.353 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:59.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:59.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:48:59.368 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:48:59.409 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:48:59.452 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:48:59.455 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:48:59.492 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:48:59.501 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:59 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:48:59.501 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:59 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:48:59.501 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:59 smithi167 ceph-mon[32774]: osdmap e377: 8 total, 8 up, 8 in 2024-02-05T06:48:59.501 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:59 smithi167 ceph-mon[32774]: pgmap v742: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 972 B/s rd, 35 MiB/s wr, 566 op/s; 62077/319818 objects misplaced (19.410%); 8.0 MiB/s, 128 objects/s recovering 2024-02-05T06:48:59.501 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:48:59 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1713865185' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:48:59.503 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:59.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:59.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:48:59.532 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:48:59.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:59 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:48:59.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:59 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:48:59.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:59 smithi028 ceph-mon[29442]: osdmap e377: 8 total, 8 up, 8 in 2024-02-05T06:48:59.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:59 smithi028 ceph-mon[29442]: pgmap v742: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 972 B/s rd, 35 MiB/s wr, 566 op/s; 62077/319818 objects misplaced (19.410%); 8.0 MiB/s, 128 objects/s recovering 2024-02-05T06:48:59.577 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:48:59 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1713865185' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:48:59.577 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:59 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:48:59.577 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:59 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:48:59.577 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:59 smithi028 ceph-mon[37984]: osdmap e377: 8 total, 8 up, 8 in 2024-02-05T06:48:59.577 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:59 smithi028 ceph-mon[37984]: pgmap v742: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 972 B/s rd, 35 MiB/s wr, 566 op/s; 62077/319818 objects misplaced (19.410%); 8.0 MiB/s, 128 objects/s recovering 2024-02-05T06:48:59.578 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:48:59 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1713865185' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:48:59.578 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:48:59.580 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:48:59.580 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:48:59.580 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:48:59.580 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:48:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:48:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:48:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:48:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:48:59.604 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:59.636 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:59.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:59.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:59.738 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:48:59.774 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:48:59.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:59.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:48:59.876 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:48:59.927 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:48:59.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:48:59.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:48:59.959 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-02-05T06:48:59.952468+0000 min lat: 0.00116262 max lat: 2.12057 avg lat: 0.0116556 2024-02-05T06:48:59.960 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T06:48:59.960 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 80 16 109824 109808 85.7605 73.5625 0.0117657 0.0116556 2024-02-05T06:49:00.028 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:00.075 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:00.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:00.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:00.176 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:00.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:00.220 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:00.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:00.321 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:00.358 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:00.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:00.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:00.449 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:00 smithi167 ceph-mon[32774]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:49:00.460 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:00.501 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:00.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:00.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:00.603 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:00.646 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:00.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:00.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:00.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:00 smithi028 ceph-mon[29442]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:49:00.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:00 smithi028 ceph-mon[37984]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:49:00.747 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:00.771 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:00.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:00.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:00.872 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:00.911 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:00.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:00.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:00.960 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 81 16 110953 110937 85.5728 70.5625 0.00370408 0.0116791 2024-02-05T06:49:00.961 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:49:00.961 INFO:teuthology.orchestra.run.smithi167.stdout: "success": "" 2024-02-05T06:49:00.961 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:49:01.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:01.052 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:01.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:01.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:01.154 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:01.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:01.205 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:01.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:01.306 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:01.343 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:01.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:01.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:01.445 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:01.476 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:01.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:01.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:01.577 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:01.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:01.615 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:01.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:01.660 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:01 smithi167 ceph-mon[32774]: pgmap v743: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 816 B/s rd, 30 MiB/s wr, 475 op/s; 62077/319818 objects misplaced (19.410%); 6.7 MiB/s, 107 objects/s recovering 2024-02-05T06:49:01.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:01 smithi028 ceph-mon[37984]: pgmap v743: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 816 B/s rd, 30 MiB/s wr, 475 op/s; 62077/319818 objects misplaced (19.410%); 6.7 MiB/s, 107 objects/s recovering 2024-02-05T06:49:01.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:01 smithi028 ceph-mon[29442]: pgmap v743: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.5 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 816 B/s rd, 30 MiB/s wr, 475 op/s; 62077/319818 objects misplaced (19.410%); 6.7 MiB/s, 107 objects/s recovering 2024-02-05T06:49:01.716 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:01.740 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:01.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:01.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:01.842 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:01.875 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:01.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:01.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:01.960 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 82 16 112019 112003 85.3416 66.625 0.00925477 0.0117129 2024-02-05T06:49:01.977 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:02.013 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:02.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:02.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:02.115 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:02.149 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:02.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:02.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:02.251 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:02.287 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:02.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:02.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:02.389 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:02.427 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:02.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:02.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:02.533 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:02.578 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:02.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:02.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:02.680 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:02.715 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:02.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:02.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:02.817 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:02.829 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:49:02 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:49:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:49:02.850 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:02.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:02.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:02.952 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:02.960 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 83 16 113264 113248 85.2509 77.8125 0.00841518 0.0117254 2024-02-05T06:49:02.985 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:02.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:02.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:03.087 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:03.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:03.120 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:03.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:03.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:49:03.221 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:03.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:03.268 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:03.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:03.369 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:03.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:03.422 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:03.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:03.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:03.549 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:49:03 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:49:03.552 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:03 smithi028 ceph-mon[29442]: pgmap v744: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.8 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 905 op/s; 62258/332610 objects misplaced (18.718%); 9.8 MiB/s, 156 objects/s recovering 2024-02-05T06:49:03.552 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:03 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1924740910' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:49:03.552 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:03 smithi028 ceph-mon[29442]: from='client.45716 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:49:03.552 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:03 smithi028 ceph-mon[37984]: pgmap v744: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.8 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 905 op/s; 62258/332610 objects misplaced (18.718%); 9.8 MiB/s, 156 objects/s recovering 2024-02-05T06:49:03.552 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:03 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1924740910' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:49:03.552 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:03 smithi028 ceph-mon[37984]: from='client.45716 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:49:03.587 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:03.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:03.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:03.590 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-02-05T06:49:03.689 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:03.721 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:03.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:03.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:03.823 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:03.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:03.859 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:03.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:03.960 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:03.961 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 84 16 114341 114325 85.0369 67.3125 0.0144277 0.0117547 2024-02-05T06:49:03.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:03 smithi167 ceph-mon[32774]: pgmap v744: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.8 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 905 op/s; 62258/332610 objects misplaced (18.718%); 9.8 MiB/s, 156 objects/s recovering 2024-02-05T06:49:03.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:03 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1924740910' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:49:03.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:03 smithi167 ceph-mon[32774]: from='client.45716 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:49:03.995 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:03.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:03.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:04.097 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:04.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:04.153 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:04.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:04.254 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:04.291 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:04.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:04.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:04.393 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:04.429 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:04.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:04.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:04.531 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:04.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:04.568 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:04.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:04.669 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:04.708 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:04.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:04.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:04.810 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:04.846 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:04.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:04.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:04.851 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:04 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:49:04.851 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:04 smithi028 ceph-mon[29442]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:49:04.852 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:04 smithi028 ceph-mon[29442]: from='client.45716 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:49:04.852 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:04 smithi028 ceph-mon[29442]: osdmap e378: 8 total, 8 up, 8 in 2024-02-05T06:49:04.852 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:04 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:49:04.852 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:04 smithi028 ceph-mon[37984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:49:04.852 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:04 smithi028 ceph-mon[37984]: from='client.45716 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:49:04.852 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:04 smithi028 ceph-mon[37984]: osdmap e378: 8 total, 8 up, 8 in 2024-02-05T06:49:04.948 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:04.960 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 85 16 115522 115506 84.9049 73.8125 0.00615874 0.0117735 2024-02-05T06:49:04.984 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:04.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:04.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:04.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:04 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:49:04.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:04 smithi167 ceph-mon[32774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:49:04.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:04 smithi167 ceph-mon[32774]: from='client.45716 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:49:04.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:04 smithi167 ceph-mon[32774]: osdmap e378: 8 total, 8 up, 8 in 2024-02-05T06:49:05.085 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:05.126 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:05.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:05.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:05.188 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:49:05.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:05.290 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:49:05.290 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:49:05.292 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:05.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:05.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:05.393 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:05.441 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:05.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:05.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:05.543 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:05.580 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:05.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:05.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:05.681 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:05.715 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:05.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:05.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:05.817 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:05.831 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:05 smithi028 ceph-mon[37984]: pgmap v746: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s; 62483/336954 objects misplaced (18.543%); 19 MiB/s, 305 objects/s recovering 2024-02-05T06:49:05.831 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:05 smithi028 ceph-mon[29442]: pgmap v746: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s; 62483/336954 objects misplaced (18.543%); 19 MiB/s, 305 objects/s recovering 2024-02-05T06:49:05.862 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:05.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:05.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:05.960 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 86 16 116692 116676 84.7679 73.125 0.0154028 0.0117926 2024-02-05T06:49:05.965 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:05.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:05 smithi167 ceph-mon[32774]: pgmap v746: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s; 62483/336954 objects misplaced (18.543%); 19 MiB/s, 305 objects/s recovering 2024-02-05T06:49:06.001 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:06.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:06.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:06.103 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:06.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:06.137 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:06.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:06.238 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:06.275 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:06.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:06.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:06.377 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:06.416 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:06.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:06.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:06.518 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:06.554 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:06.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:06.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:06.656 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:06.663 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 4, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T06:49:06.663 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T06:49:06.663 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-02-05T06:49:06.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph tell osd.0 'trim stale osdmaps' 2024-02-05T06:49:06.696 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:06.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:06.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:06.798 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:06.831 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:06.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:06.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:06.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T06:49:06.933 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:06.961 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 87 16 117882 117866 84.6481 74.375 0.0310511 0.0118082 2024-02-05T06:49:07.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:07.054 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:07.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:07.155 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:07.186 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:07.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:07.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:07.288 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:07.322 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:07.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:07.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:07.424 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:07.462 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:07.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:07.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:07.519 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-05T06:49:07.564 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:07.616 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:07.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:07.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:07.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:07.730 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:07 smithi028 ceph-mon[29442]: pgmap v747: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 630 B/s rd, 44 MiB/s wr, 704 op/s; 62483/336954 objects misplaced (18.543%); 14 MiB/s, 218 objects/s recovering 2024-02-05T06:49:07.730 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:07 smithi028 ceph-mon[37984]: pgmap v747: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 630 B/s rd, 44 MiB/s wr, 704 op/s; 62483/336954 objects misplaced (18.543%); 14 MiB/s, 218 objects/s recovering 2024-02-05T06:49:07.777 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:07.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:07.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:07.879 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:07.902 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:07.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:07.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:07.961 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 88 16 118883 118867 84.3971 62.5625 0.00979628 0.0118442 2024-02-05T06:49:07.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:07 smithi167 ceph-mon[32774]: pgmap v747: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 630 B/s rd, 44 MiB/s wr, 704 op/s; 62483/336954 objects misplaced (18.543%); 14 MiB/s, 218 objects/s recovering 2024-02-05T06:49:08.004 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:08.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:08.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:08.145 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:08.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:08.181 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:08.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:08.282 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:08.314 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:08.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:08.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:08.416 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:08.469 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:08.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:08.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:08.521 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:49:08.540 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:49:08.571 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:08.585 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:49:08.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:08.643 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:08.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:08.644 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:49:08.694 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:49:08.737 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:49:08.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:08.778 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:49:08.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:08.798 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:08.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:08.817 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:49:08.861 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:49:08.862 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:08 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:49:08.862 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:08 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:49:08.862 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:08 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/51505783' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:49:08.863 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:08 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:49:08.863 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:08 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:49:08.863 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:08 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/51505783' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:49:08.899 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:08.902 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:49:08.912 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:49:08.912 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:49:08.912 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:49:08.912 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:49:08.913 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:49:08.913 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:49:08.913 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:49:08.913 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:49:08.937 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:08.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:08.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:08.961 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 89 16 119924 119908 84.1799 65.0625 0.0192374 0.0118744 2024-02-05T06:49:08.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:08 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:49:08.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:08 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:49:08.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:08 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/51505783' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:49:09.038 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:09.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:09.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:09.179 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:09.221 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:09.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:09.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:09.324 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:09.363 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:09.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:09.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:09.465 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:09.507 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:09.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:09.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:09.609 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:09.644 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:09.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:09.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:09.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:09.770 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:09.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:09.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:09.871 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:09.921 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:09.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:09.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:09.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:09 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:49:09.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:09 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:49:09.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:09 smithi028 ceph-mon[37984]: pgmap v748: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 72 MiB/s wr, 1.15k op/s; 62781/354423 objects misplaced (17.714%); 22 MiB/s, 352 objects/s recovering 2024-02-05T06:49:09.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:09 smithi028 ceph-mon[37984]: osdmap e379: 8 total, 8 up, 8 in 2024-02-05T06:49:09.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:09 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:49:09.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:09 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:49:09.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:09 smithi028 ceph-mon[29442]: pgmap v748: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 72 MiB/s wr, 1.15k op/s; 62781/354423 objects misplaced (17.714%); 22 MiB/s, 352 objects/s recovering 2024-02-05T06:49:09.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:09 smithi028 ceph-mon[29442]: osdmap e379: 8 total, 8 up, 8 in 2024-02-05T06:49:09.961 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Total time run: 90.006 2024-02-05T06:49:09.961 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Total writes made: 120998 2024-02-05T06:49:09.961 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Write size: 65536 2024-02-05T06:49:09.961 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Object size: 65536 2024-02-05T06:49:09.961 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Bandwidth (MB/sec): 84.0208 2024-02-05T06:49:09.962 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Stddev Bandwidth: 35.941 2024-02-05T06:49:09.962 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Max bandwidth (MB/sec): 186.438 2024-02-05T06:49:09.962 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Min bandwidth (MB/sec): 0 2024-02-05T06:49:09.962 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Average IOPS: 1344 2024-02-05T06:49:09.962 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Stddev IOPS: 575.055 2024-02-05T06:49:09.962 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Max IOPS: 2983 2024-02-05T06:49:09.962 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Min IOPS: 0 2024-02-05T06:49:09.962 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Average Latency(s): 0.0118977 2024-02-05T06:49:09.962 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Stddev Latency(s): 0.0287976 2024-02-05T06:49:09.962 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Max latency(s): 2.12057 2024-02-05T06:49:09.962 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Min latency(s): 0.00116262 2024-02-05T06:49:09.967 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Cleaning up (deleting benchmark objects) 2024-02-05T06:49:09.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:09 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:49:09.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:09 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:49:09.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:09 smithi167 ceph-mon[32774]: pgmap v748: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 72 MiB/s wr, 1.15k op/s; 62781/354423 objects misplaced (17.714%); 22 MiB/s, 352 objects/s recovering 2024-02-05T06:49:09.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:09 smithi167 ceph-mon[32774]: osdmap e379: 8 total, 8 up, 8 in 2024-02-05T06:49:10.023 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:10.047 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:10.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:10.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:10.149 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:10.174 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:10.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:10.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:10.276 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:10.314 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:10.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:10.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:10.416 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:10.453 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:10.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:10.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:10.555 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:10.587 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:10.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:10.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:10.689 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:10.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:10.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:10.832 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:10.861 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:10.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:10.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:10.963 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:10.996 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:10.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:10.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:11.098 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:11.140 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:11.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:11.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:11.241 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:11.276 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:11.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:11.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:11.378 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:11.416 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:11.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:11.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:11.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:11 smithi167 ceph-mon[32774]: 4.1b scrub starts 2024-02-05T06:49:11.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:11 smithi167 ceph-mon[32774]: 4.1b scrub ok 2024-02-05T06:49:11.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:11 smithi167 ceph-mon[32774]: pgmap v750: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 908 op/s; 62781/354423 objects misplaced (17.714%); 23 MiB/s, 368 objects/s recovering 2024-02-05T06:49:11.517 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:11.555 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:11.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:11.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:11.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:11.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:11 smithi028 ceph-mon[29442]: 4.1b scrub starts 2024-02-05T06:49:11.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:11 smithi028 ceph-mon[29442]: 4.1b scrub ok 2024-02-05T06:49:11.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:11 smithi028 ceph-mon[29442]: pgmap v750: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 908 op/s; 62781/354423 objects misplaced (17.714%); 23 MiB/s, 368 objects/s recovering 2024-02-05T06:49:11.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:11 smithi028 ceph-mon[37984]: 4.1b scrub starts 2024-02-05T06:49:11.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:11 smithi028 ceph-mon[37984]: 4.1b scrub ok 2024-02-05T06:49:11.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:11 smithi028 ceph-mon[37984]: pgmap v750: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 908 op/s; 62781/354423 objects misplaced (17.714%); 23 MiB/s, 368 objects/s recovering 2024-02-05T06:49:11.713 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:11.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:11.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:11.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:11.863 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:11.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:11.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:11.965 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:11.996 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:11.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:11.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:12.098 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:12.142 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:12.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:12.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:12.244 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:12.281 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:12.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:12.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:12.382 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:12.417 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:12.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:12.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:12.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:12 smithi167 ceph-mon[32774]: 3.b scrub starts 2024-02-05T06:49:12.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:12 smithi167 ceph-mon[32774]: 3.b scrub ok 2024-02-05T06:49:12.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:12 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:49:12.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:12 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:49:12.503 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:12 smithi028 ceph-mon[29442]: 3.b scrub starts 2024-02-05T06:49:12.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:12 smithi028 ceph-mon[29442]: 3.b scrub ok 2024-02-05T06:49:12.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:12 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:49:12.504 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:12 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:49:12.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:12 smithi028 ceph-mon[37984]: 3.b scrub starts 2024-02-05T06:49:12.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:12 smithi028 ceph-mon[37984]: 3.b scrub ok 2024-02-05T06:49:12.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:12 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:49:12.504 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:12 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:49:12.520 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:12.552 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:12.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:12.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:12.654 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:12.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:12.705 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:12.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:12.806 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:12.830 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:12.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:12.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:12.931 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:12.950 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:49:12 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:49:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:49:12.966 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:12.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:12.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:13.067 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:13.105 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:13.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:13.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:13.207 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:13.243 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:13.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:13.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:13.345 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:13.384 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:13.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:13.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:13.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:13 smithi167 ceph-mon[32774]: pgmap v751: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 7.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 1.19k op/s; 61038/357510 objects misplaced (17.073%); 22 MiB/s, 351 objects/s recovering 2024-02-05T06:49:13.485 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:49:13 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:49:13.486 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:13.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:13.530 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:13.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:13.631 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:13.656 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:13.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:13.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:13.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:13 smithi028 ceph-mon[29442]: pgmap v751: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 7.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 1.19k op/s; 61038/357510 objects misplaced (17.073%); 22 MiB/s, 351 objects/s recovering 2024-02-05T06:49:13.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:13 smithi028 ceph-mon[37984]: pgmap v751: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 7.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 1.19k op/s; 61038/357510 objects misplaced (17.073%); 22 MiB/s, 351 objects/s recovering 2024-02-05T06:49:13.758 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:13.791 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:13.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:13.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:13.893 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:13.931 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:13.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:13.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:14.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:14.075 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:14.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:14.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:14.177 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:14.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:14.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:14.317 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:14.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:14.353 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:14.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:14.454 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:14.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:14 smithi167 ceph-mon[32774]: 4.14 scrub starts 2024-02-05T06:49:14.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:14 smithi167 ceph-mon[32774]: 4.14 scrub ok 2024-02-05T06:49:14.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:14 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:49:14.490 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:14.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:14.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:14.592 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:14.633 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:14.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:14.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:14.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:14 smithi028 ceph-mon[29442]: 4.14 scrub starts 2024-02-05T06:49:14.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:14 smithi028 ceph-mon[29442]: 4.14 scrub ok 2024-02-05T06:49:14.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:14 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:49:14.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:14 smithi028 ceph-mon[37984]: 4.14 scrub starts 2024-02-05T06:49:14.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:14 smithi028 ceph-mon[37984]: 4.14 scrub ok 2024-02-05T06:49:14.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:14 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:49:14.734 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:14.769 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:14.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:14.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:14.871 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:14.894 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:14.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:14.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:14.996 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:15.031 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:15.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:15.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:15.133 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:15.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:15.177 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:15.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:15.278 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:15.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:15.319 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:15.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:15.420 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:15.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:15.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:15.558 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:15.598 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:15.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:15.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:15.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:15.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:15 smithi028 ceph-mon[37984]: 3.7 scrub starts 2024-02-05T06:49:15.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:15 smithi028 ceph-mon[37984]: 3.7 scrub ok 2024-02-05T06:49:15.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:15 smithi028 ceph-mon[37984]: pgmap v752: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 7.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 1.09k op/s; 60985/357390 objects misplaced (17.064%); 20 MiB/s, 313 objects/s recovering 2024-02-05T06:49:15.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:15 smithi028 ceph-mon[29442]: 3.7 scrub starts 2024-02-05T06:49:15.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:15 smithi028 ceph-mon[29442]: 3.7 scrub ok 2024-02-05T06:49:15.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:15 smithi028 ceph-mon[29442]: pgmap v752: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 7.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 1.09k op/s; 60985/357390 objects misplaced (17.064%); 20 MiB/s, 313 objects/s recovering 2024-02-05T06:49:15.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:15 smithi167 ceph-mon[32774]: 3.7 scrub starts 2024-02-05T06:49:15.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:15 smithi167 ceph-mon[32774]: 3.7 scrub ok 2024-02-05T06:49:15.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:15 smithi167 ceph-mon[32774]: pgmap v752: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 7.3 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.1 KiB/s rd, 55 MiB/s wr, 1.09k op/s; 60985/357390 objects misplaced (17.064%); 20 MiB/s, 313 objects/s recovering 2024-02-05T06:49:15.741 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:15.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:15.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:15.843 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:15.879 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:15.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:15.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:15.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:16.021 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:16.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:16.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:16.123 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:16.181 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:16.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:16.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:16.283 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:16.315 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:16.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:16.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:16.417 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:16.454 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:16.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:16.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:16.557 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:16.588 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:16.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:16.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:16.690 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:16.731 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:16.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:16.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:16.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:16.874 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:16.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:16.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:16.975 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:17.016 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:17.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:17.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:17.097 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:49:17.119 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:17.148 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:17.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:17.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:17.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:17.273 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-02-05T06:49:17.295 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:17.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:17.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:17.398 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:17.431 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:17.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:17.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:17.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:17 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:49:17.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:17 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:49:17.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:17 smithi028 ceph-mon[37984]: osdmap e380: 8 total, 8 up, 8 in 2024-02-05T06:49:17.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:17 smithi028 ceph-mon[37984]: pgmap v754: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 767 B/s rd, 24 MiB/s wr, 772 op/s; 58378/354231 objects misplaced (16.480%); 19 MiB/s, 303 objects/s recovering 2024-02-05T06:49:17.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:17 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/4258433077' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:49:17.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:17 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:49:17.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:17 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:49:17.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:17 smithi028 ceph-mon[29442]: osdmap e380: 8 total, 8 up, 8 in 2024-02-05T06:49:17.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:17 smithi028 ceph-mon[29442]: pgmap v754: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 767 B/s rd, 24 MiB/s wr, 772 op/s; 58378/354231 objects misplaced (16.480%); 19 MiB/s, 303 objects/s recovering 2024-02-05T06:49:17.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:17 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/4258433077' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:49:17.533 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:17.565 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:17.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:17.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:17.669 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:17.711 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:17.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:17.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:17.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:17 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:49:17.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:17 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:49:17.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:17 smithi167 ceph-mon[32774]: osdmap e380: 8 total, 8 up, 8 in 2024-02-05T06:49:17.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:17 smithi167 ceph-mon[32774]: pgmap v754: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 7.2 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 767 B/s rd, 24 MiB/s wr, 772 op/s; 58378/354231 objects misplaced (16.480%); 19 MiB/s, 303 objects/s recovering 2024-02-05T06:49:17.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:17 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/4258433077' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:49:17.813 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:17.846 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:17.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:17.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:17.948 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:17.989 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:17.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:17.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:18.091 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:18.126 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:18.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:18.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:18.227 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:18.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:18.255 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:18.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:18.356 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:18.397 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:18.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:18.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:18.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:18.538 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:18.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:18.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:18.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:18.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:18 smithi028 ceph-mon[37984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:49:18.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:18 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/4258433077' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:49:18.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:18 smithi028 ceph-mon[37984]: osdmap e381: 8 total, 8 up, 8 in 2024-02-05T06:49:18.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:18 smithi028 ceph-mon[29442]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:49:18.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:18 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/4258433077' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:49:18.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:18 smithi028 ceph-mon[29442]: osdmap e381: 8 total, 8 up, 8 in 2024-02-05T06:49:18.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:18.674 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:18.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:18.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:18 smithi167 ceph-mon[32774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:49:18.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:18 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/4258433077' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:49:18.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:18 smithi167 ceph-mon[32774]: osdmap e381: 8 total, 8 up, 8 in 2024-02-05T06:49:18.775 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:18.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:18.813 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:18.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:18.914 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:18.944 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:18.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:18.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:19.048 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:19.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:19.081 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:19.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:19.138 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:49:19.182 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:19.241 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:19.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:19.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:19.343 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:19.366 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:19.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:19.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:19.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:19 smithi028 ceph-mon[29442]: pgmap v756: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.4 KiB/s rd, 24 MiB/s wr, 1.32k op/s; 55366/340995 objects misplaced (16.237%); 24 MiB/s, 382 objects/s recovering 2024-02-05T06:49:19.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:19 smithi028 ceph-mon[37984]: pgmap v756: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.4 KiB/s rd, 24 MiB/s wr, 1.32k op/s; 55366/340995 objects misplaced (16.237%); 24 MiB/s, 382 objects/s recovering 2024-02-05T06:49:19.468 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:19.501 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:19.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:19.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:19.603 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:19.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:19.639 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:19.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:19.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:19 smithi167 ceph-mon[32774]: pgmap v756: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.4 KiB/s rd, 24 MiB/s wr, 1.32k op/s; 55366/340995 objects misplaced (16.237%); 24 MiB/s, 382 objects/s recovering 2024-02-05T06:49:19.740 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:19.772 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:19.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:19.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:19.874 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:19.923 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:19.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:19.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:20.025 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:20.066 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:20.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:20.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:20.168 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:20.204 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:20.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:20.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:20.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:20.347 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:20.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:20.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:20.448 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:20.486 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:20.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:20.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:20.588 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:20.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:20.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:20.737 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:20.779 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:20.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:20.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:20.880 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:20.909 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:20.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:20.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:21.010 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:21.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:21.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:21.151 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:21.190 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:21.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:21.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:21.291 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:21.327 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:21.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:21.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:21.429 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:21.464 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:21.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:21.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:21.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:21 smithi167 ceph-mon[32774]: pgmap v757: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 1.1 MiB/s wr, 724 op/s; 55366/340995 objects misplaced (16.237%); 13 MiB/s, 209 objects/s recovering 2024-02-05T06:49:21.565 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:21.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:21.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:21.696 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:21 smithi028 ceph-mon[37984]: pgmap v757: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 1.1 MiB/s wr, 724 op/s; 55366/340995 objects misplaced (16.237%); 13 MiB/s, 209 objects/s recovering 2024-02-05T06:49:21.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:21 smithi028 ceph-mon[29442]: pgmap v757: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.9 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 639 B/s rd, 1.1 MiB/s wr, 724 op/s; 55366/340995 objects misplaced (16.237%); 13 MiB/s, 209 objects/s recovering 2024-02-05T06:49:21.709 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:21.773 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:21.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:21.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:21.875 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:21.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:21.916 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:21.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:22.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:22.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:22.059 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:22.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:22.160 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:22.209 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:22.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:22.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:22.310 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:22.339 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:22.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:22.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:22.441 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:22.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:22 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:49:22.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:22 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:49:22.444 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:22 smithi028 ceph-mon[29442]: osdmap e382: 8 total, 8 up, 8 in 2024-02-05T06:49:22.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:22 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:49:22.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:22 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:49:22.445 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:22 smithi028 ceph-mon[37984]: osdmap e382: 8 total, 8 up, 8 in 2024-02-05T06:49:22.483 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:22.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:22.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:22.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:22 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:49:22.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:22 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:49:22.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:22 smithi167 ceph-mon[32774]: osdmap e382: 8 total, 8 up, 8 in 2024-02-05T06:49:22.586 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:22.617 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:22.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:22.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:22.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:49:22 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:49:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:49:22.719 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:22.744 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:22.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:22.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:22.834 INFO:teuthology.orchestra.run.smithi028.stderr:no valid command found; 10 closest matches: 2024-02-05T06:49:22.834 INFO:teuthology.orchestra.run.smithi028.stderr:0 2024-02-05T06:49:22.834 INFO:teuthology.orchestra.run.smithi028.stderr:1 2024-02-05T06:49:22.834 INFO:teuthology.orchestra.run.smithi028.stderr:2 2024-02-05T06:49:22.834 INFO:teuthology.orchestra.run.smithi028.stderr:abort 2024-02-05T06:49:22.835 INFO:teuthology.orchestra.run.smithi028.stderr:assert 2024-02-05T06:49:22.835 INFO:teuthology.orchestra.run.smithi028.stderr:bench [] [] [] [] 2024-02-05T06:49:22.835 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs debug_inject_read_zeros 2024-02-05T06:49:22.835 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs files list 2024-02-05T06:49:22.835 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs stats 2024-02-05T06:49:22.835 INFO:teuthology.orchestra.run.smithi028.stderr:bluestore allocator dump block 2024-02-05T06:49:22.835 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: invalid command 2024-02-05T06:49:22.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:22.880 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:22.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:22.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:22.981 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:23.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:23.007 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:23.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:23.107 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:23.133 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:23.133 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:49:23.133 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-02-05T06:49:23.133 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-02-05T06:49:23.134 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-02-05T06:49:23.134 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:49:23.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:23.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:23.235 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:23.266 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:23.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:23.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:23.368 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:23.406 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:23.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:23.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:23.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:23 smithi028 ceph-mon[37984]: osdmap e383: 8 total, 8 up, 8 in 2024-02-05T06:49:23.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:23 smithi028 ceph-mon[37984]: pgmap v760: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.63k op/s; 50401/324942 objects misplaced (15.511%); 25 MiB/s, 393 objects/s recovering 2024-02-05T06:49:23.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:23 smithi028 ceph-mon[29442]: osdmap e383: 8 total, 8 up, 8 in 2024-02-05T06:49:23.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:23 smithi028 ceph-mon[29442]: pgmap v760: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.63k op/s; 50401/324942 objects misplaced (15.511%); 25 MiB/s, 393 objects/s recovering 2024-02-05T06:49:23.508 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:23.543 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:23.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:23.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:23.644 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:23.680 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:23.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:23.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:23.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:23 smithi167 ceph-mon[32774]: osdmap e383: 8 total, 8 up, 8 in 2024-02-05T06:49:23.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:23 smithi167 ceph-mon[32774]: pgmap v760: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.63k op/s; 50401/324942 objects misplaced (15.511%); 25 MiB/s, 393 objects/s recovering 2024-02-05T06:49:23.735 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:49:23 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:49:23.753 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T06:49:23.782 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:23.812 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:23.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:23.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:23.851 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-05T06:49:23.851 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-05T06:49:23.851 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph tell osd.1 'trim stale osdmaps' 2024-02-05T06:49:23.914 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:23.949 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:23.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:23.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:24.051 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:24.085 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:49:24.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:24.128 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:24.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:24.229 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:24.282 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:24.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:24.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:24.384 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:24.419 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:24.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:24.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:24.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:24 smithi028 ceph-mon[37984]: osdmap e384: 8 total, 8 up, 8 in 2024-02-05T06:49:24.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:24 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:49:24.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:24 smithi028 ceph-mon[29442]: osdmap e384: 8 total, 8 up, 8 in 2024-02-05T06:49:24.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:24 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:49:24.520 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:24.557 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:24.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:24.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:24.618 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-05T06:49:24.659 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:24.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:24.721 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:24.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:24.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:24 smithi167 ceph-mon[32774]: osdmap e384: 8 total, 8 up, 8 in 2024-02-05T06:49:24.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:24 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:49:24.822 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:24.899 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:24.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:24.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:25.001 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:25.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:25.044 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:25.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:25.145 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:25.180 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:25.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:25.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:25.282 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:25.307 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:25.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:25.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:25.399 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:25 smithi028 ceph-mon[37984]: 10.11 scrub starts 2024-02-05T06:49:25.399 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:25 smithi028 ceph-mon[37984]: pgmap v762: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 954 op/s; 49847/323817 objects misplaced (15.394%); 18 MiB/s, 288 objects/s recovering 2024-02-05T06:49:25.399 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:25 smithi028 ceph-mon[37984]: osdmap e385: 8 total, 8 up, 8 in 2024-02-05T06:49:25.400 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:25 smithi028 ceph-mon[29442]: 10.11 scrub starts 2024-02-05T06:49:25.400 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:25 smithi028 ceph-mon[29442]: pgmap v762: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 954 op/s; 49847/323817 objects misplaced (15.394%); 18 MiB/s, 288 objects/s recovering 2024-02-05T06:49:25.400 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:25 smithi028 ceph-mon[29442]: osdmap e385: 8 total, 8 up, 8 in 2024-02-05T06:49:25.409 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:25.441 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:25.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:25.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:25.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:25.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:25.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:25.681 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:25.712 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:25.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:25.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:25.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:25 smithi167 ceph-mon[32774]: 10.11 scrub starts 2024-02-05T06:49:25.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:25 smithi167 ceph-mon[32774]: pgmap v762: 168 pgs: 3 active+remapped+backfilling, 7 active+remapped+backfill_toofull, 158 active+clean; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 954 op/s; 49847/323817 objects misplaced (15.394%); 18 MiB/s, 288 objects/s recovering 2024-02-05T06:49:25.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:25 smithi167 ceph-mon[32774]: osdmap e385: 8 total, 8 up, 8 in 2024-02-05T06:49:25.814 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:25.849 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:25.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:25.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:25.951 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:25.980 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:25.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:25.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:26.081 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:26.115 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:26.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:26.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:26.217 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:26.236 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:49:26.243 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:26.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:26.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:26.289 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:49:26.346 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:26.401 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:26.401 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:26 smithi028 ceph-mon[37984]: 10.11 scrub ok 2024-02-05T06:49:26.402 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:26 smithi028 ceph-mon[37984]: 10.1 scrub starts 2024-02-05T06:49:26.402 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:26 smithi028 ceph-mon[37984]: osdmap e386: 8 total, 8 up, 8 in 2024-02-05T06:49:26.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:26.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:26.403 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:26 smithi028 ceph-mon[29442]: 10.11 scrub ok 2024-02-05T06:49:26.403 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:26 smithi028 ceph-mon[29442]: 10.1 scrub starts 2024-02-05T06:49:26.403 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:26 smithi028 ceph-mon[29442]: osdmap e386: 8 total, 8 up, 8 in 2024-02-05T06:49:26.504 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:26.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:26.539 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:26.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:26.613 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:49:26.640 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:26.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:26.694 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:26.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:26.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:26 smithi167 ceph-mon[32774]: 10.11 scrub ok 2024-02-05T06:49:26.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:26 smithi167 ceph-mon[32774]: 10.1 scrub starts 2024-02-05T06:49:26.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:26 smithi167 ceph-mon[32774]: osdmap e386: 8 total, 8 up, 8 in 2024-02-05T06:49:26.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:26.837 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:26.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:26.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:26.939 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:26.962 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:26.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:26.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:27.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:27.104 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:27.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:27.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:27.206 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:27.242 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:27.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:27.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:27.343 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:27.380 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:27.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:27.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:27.482 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:27.512 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:27.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:27.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:27.614 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:27.638 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:27.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:27.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:27.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:27 smithi028 ceph-mon[29442]: 10.1 scrub ok 2024-02-05T06:49:27.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:27 smithi028 ceph-mon[29442]: pgmap v765: 168 pgs: 4 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 158 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 257 op/s; 46810/321645 objects misplaced (14.553%); 16 MiB/s, 261 objects/s recovering 2024-02-05T06:49:27.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:27 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:49:27.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:27 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:49:27.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:27 smithi028 ceph-mon[37984]: 10.1 scrub ok 2024-02-05T06:49:27.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:27 smithi028 ceph-mon[37984]: pgmap v765: 168 pgs: 4 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 158 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 257 op/s; 46810/321645 objects misplaced (14.553%); 16 MiB/s, 261 objects/s recovering 2024-02-05T06:49:27.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:27 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:49:27.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:27 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:49:27.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:27 smithi167 ceph-mon[32774]: 10.1 scrub ok 2024-02-05T06:49:27.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:27 smithi167 ceph-mon[32774]: pgmap v765: 168 pgs: 4 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 158 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s wr, 257 op/s; 46810/321645 objects misplaced (14.553%); 16 MiB/s, 261 objects/s recovering 2024-02-05T06:49:27.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:27 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:49:27.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:27 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:49:27.740 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:27.774 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:27.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:27.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:27.876 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:27.909 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:27.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:27.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:28.011 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:28.060 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:28.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:28.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:28.162 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:28.202 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:28.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:28.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:28.304 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:28.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:28.340 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:28.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:28.427 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:28 smithi028 ceph-mon[37984]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:49:28.427 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:28 smithi028 ceph-mon[29442]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:49:28.441 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:28.481 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:28.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:28.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:28.583 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:28.624 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:28.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:28.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:28.726 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:28.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:28 smithi167 ceph-mon[32774]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:49:28.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:28.761 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:28.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:28.862 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:28.905 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:28.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:28.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:29.008 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:29.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:29.047 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:29.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:29.148 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:29.194 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:29.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:29.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:29.296 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:29.332 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:29.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:29.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:29.435 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:29.476 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:29.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:29.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:29.554 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:29 smithi028 ceph-mon[37984]: pgmap v766: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.3 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 43072/310635 objects misplaced (13.866%) 2024-02-05T06:49:29.554 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:29 smithi028 ceph-mon[29442]: pgmap v766: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.3 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 43072/310635 objects misplaced (13.866%) 2024-02-05T06:49:29.579 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:29.624 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:29.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:29.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:29.726 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:29.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:29 smithi167 ceph-mon[32774]: pgmap v766: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.3 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 43072/310635 objects misplaced (13.866%) 2024-02-05T06:49:29.758 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:29.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:29.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:29.860 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:29.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:29.905 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:29.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:29.970 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:49:29.971 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:49:29.971 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:49:29.971 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:49:29.971 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:49:29.971 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:49:29.971 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:49:29.971 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:49:29.971 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:49:29.971 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:49:29.972 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:49:29.972 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:49:29.972 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:49:29.972 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:49:29.972 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:49:29.972 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:49:30.006 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:30.034 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:30.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:30.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:30.136 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:30.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:30.170 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:30.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:30.271 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:30.309 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:30.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:30.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:30.410 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:30.444 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:30.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:30.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:30.546 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:30.554 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:30 smithi028 ceph-mon[37984]: osdmap e387: 8 total, 8 up, 8 in 2024-02-05T06:49:30.554 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:30 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2482680297' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:49:30.555 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:30 smithi028 ceph-mon[37984]: osdmap e388: 8 total, 8 up, 8 in 2024-02-05T06:49:30.555 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:30 smithi028 ceph-mon[29442]: osdmap e387: 8 total, 8 up, 8 in 2024-02-05T06:49:30.555 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:30 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2482680297' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:49:30.555 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:30 smithi028 ceph-mon[29442]: osdmap e388: 8 total, 8 up, 8 in 2024-02-05T06:49:30.584 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:30.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:30.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:30.686 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:30.723 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:30.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:30.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:30.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:30 smithi167 ceph-mon[32774]: osdmap e387: 8 total, 8 up, 8 in 2024-02-05T06:49:30.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:30 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2482680297' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:49:30.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:30 smithi167 ceph-mon[32774]: osdmap e388: 8 total, 8 up, 8 in 2024-02-05T06:49:30.825 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:30.852 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:30.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:30.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:30.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:30.996 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:30.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:30.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:31.098 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:31.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:31.133 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:31.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:31.234 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:31.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:31.272 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:31.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:31.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:31.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:31.416 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:31.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:31.517 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:31.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:31.700 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:31.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:31.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:31 smithi028 ceph-mon[37984]: 10.0 scrub starts 2024-02-05T06:49:31.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:31 smithi028 ceph-mon[37984]: pgmap v769: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.3 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 966 B/s rd, 0 op/s; 43072/310635 objects misplaced (13.866%) 2024-02-05T06:49:31.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:31 smithi028 ceph-mon[29442]: 10.0 scrub starts 2024-02-05T06:49:31.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:31 smithi028 ceph-mon[29442]: pgmap v769: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.3 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 966 B/s rd, 0 op/s; 43072/310635 objects misplaced (13.866%) 2024-02-05T06:49:31.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:31 smithi167 ceph-mon[32774]: 10.0 scrub starts 2024-02-05T06:49:31.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:31 smithi167 ceph-mon[32774]: pgmap v769: 168 pgs: 4 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.3 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 966 B/s rd, 0 op/s; 43072/310635 objects misplaced (13.866%) 2024-02-05T06:49:31.801 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:31.836 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:31.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:31.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:31.938 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:31.979 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:31.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:31.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:32.081 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:32.114 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:32.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:32.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:32.216 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:32.275 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:32.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:32.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:32.377 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:32.427 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:32.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:32.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:32.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:32 smithi028 ceph-mon[29442]: 10.0 scrub ok 2024-02-05T06:49:32.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:32 smithi028 ceph-mon[29442]: 10.7 scrub starts 2024-02-05T06:49:32.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:32 smithi028 ceph-mon[37984]: 10.0 scrub ok 2024-02-05T06:49:32.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:32 smithi028 ceph-mon[37984]: 10.7 scrub starts 2024-02-05T06:49:32.529 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:32.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:32.563 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:32.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:32.664 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:32.700 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:32.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:32.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:32.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:32 smithi167 ceph-mon[32774]: 10.0 scrub ok 2024-02-05T06:49:32.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:32 smithi167 ceph-mon[32774]: 10.7 scrub starts 2024-02-05T06:49:32.802 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:32.819 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:49:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:49:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:49:32.838 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:32.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:32.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:32.940 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:32.984 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:32.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:32.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:33.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:33.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:33.123 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:33.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:33.224 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:33.272 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:33.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:33.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:33.276 INFO:teuthology.orchestra.run.smithi028.stderr:no valid command found; 10 closest matches: 2024-02-05T06:49:33.276 INFO:teuthology.orchestra.run.smithi028.stderr:0 2024-02-05T06:49:33.276 INFO:teuthology.orchestra.run.smithi028.stderr:1 2024-02-05T06:49:33.276 INFO:teuthology.orchestra.run.smithi028.stderr:2 2024-02-05T06:49:33.276 INFO:teuthology.orchestra.run.smithi028.stderr:abort 2024-02-05T06:49:33.276 INFO:teuthology.orchestra.run.smithi028.stderr:assert 2024-02-05T06:49:33.276 INFO:teuthology.orchestra.run.smithi028.stderr:bench [] [] [] [] 2024-02-05T06:49:33.277 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs debug_inject_read_zeros 2024-02-05T06:49:33.277 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs files list 2024-02-05T06:49:33.277 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs stats 2024-02-05T06:49:33.277 INFO:teuthology.orchestra.run.smithi028.stderr:bluestore allocator dump block 2024-02-05T06:49:33.277 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: invalid command 2024-02-05T06:49:33.329 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:49:33.339 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:33 smithi028 ceph-mon[37984]: 10.7 scrub ok 2024-02-05T06:49:33.340 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:33 smithi028 ceph-mon[37984]: 10.16 scrub starts 2024-02-05T06:49:33.340 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:33 smithi028 ceph-mon[37984]: pgmap v770: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s; 39983/296025 objects misplaced (13.507%) 2024-02-05T06:49:33.340 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:33 smithi028 ceph-mon[37984]: 3.15 scrub starts 2024-02-05T06:49:33.374 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:33.421 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:33.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:33.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:33.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:33.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:33.603 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:33.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:33.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:33 smithi028 ceph-mon[29442]: 10.7 scrub ok 2024-02-05T06:49:33.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:33 smithi028 ceph-mon[29442]: 10.16 scrub starts 2024-02-05T06:49:33.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:33 smithi028 ceph-mon[29442]: pgmap v770: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s; 39983/296025 objects misplaced (13.507%) 2024-02-05T06:49:33.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:33 smithi028 ceph-mon[29442]: 3.15 scrub starts 2024-02-05T06:49:33.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:33 smithi028 ceph-mon[29442]: 3.15 scrub ok 2024-02-05T06:49:33.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:33 smithi028 ceph-mon[37984]: 3.15 scrub ok 2024-02-05T06:49:33.704 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:33.735 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:49:33 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:49:33.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:33 smithi167 ceph-mon[32774]: 10.7 scrub ok 2024-02-05T06:49:33.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:33 smithi167 ceph-mon[32774]: 10.16 scrub starts 2024-02-05T06:49:33.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:33 smithi167 ceph-mon[32774]: pgmap v770: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s; 39983/296025 objects misplaced (13.507%) 2024-02-05T06:49:33.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:33 smithi167 ceph-mon[32774]: 3.15 scrub starts 2024-02-05T06:49:33.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:33 smithi167 ceph-mon[32774]: 3.15 scrub ok 2024-02-05T06:49:33.758 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:33.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:33.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:33.861 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:33.898 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:33.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:33.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:34.000 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:34.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:34.045 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:34.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:34.146 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:34.237 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:34.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:34.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:34.339 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:34.385 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:34.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:34.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:34.486 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:34.517 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:34.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:34.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:34.602 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:34 smithi028 ceph-mon[37984]: 10.16 scrub ok 2024-02-05T06:49:34.602 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:34 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:49:34.603 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:34 smithi028 ceph-mon[37984]: 3.e scrub starts 2024-02-05T06:49:34.603 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:34 smithi028 ceph-mon[37984]: 3.e scrub ok 2024-02-05T06:49:34.603 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:34 smithi028 ceph-mon[29442]: 10.16 scrub ok 2024-02-05T06:49:34.603 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:34 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:49:34.603 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:34 smithi028 ceph-mon[29442]: 3.e scrub starts 2024-02-05T06:49:34.603 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:34 smithi028 ceph-mon[29442]: 3.e scrub ok 2024-02-05T06:49:34.619 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:34.648 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-05T06:49:34.648 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-05T06:49:34.649 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph tell osd.2 'trim stale osdmaps' 2024-02-05T06:49:34.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:34.650 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:34.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:34.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:34 smithi167 ceph-mon[32774]: 10.16 scrub ok 2024-02-05T06:49:34.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:34 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:49:34.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:34 smithi167 ceph-mon[32774]: 3.e scrub starts 2024-02-05T06:49:34.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:34 smithi167 ceph-mon[32774]: 3.e scrub ok 2024-02-05T06:49:34.751 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:34.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:34.779 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:34.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:34.879 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:49:34.880 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:34.929 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:49:34.929 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:49:34.955 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:34.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:34.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:35.060 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:35.116 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:35.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:35.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:35.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:35.228 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-02-05T06:49:35.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:35.271 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:35.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:35.372 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:35.414 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:35.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:35.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:35.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:35 smithi028 ceph-mon[37984]: pgmap v771: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 37379/295014 objects misplaced (12.670%); 2.8 MiB/s, 44 objects/s recovering 2024-02-05T06:49:35.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:35 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2665377013' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:49:35.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:35 smithi028 ceph-mon[37984]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:49:35.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:35 smithi028 ceph-mon[37984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:49:35.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:35 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2665377013' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:49:35.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:35 smithi028 ceph-mon[37984]: osdmap e389: 8 total, 8 up, 8 in 2024-02-05T06:49:35.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:35 smithi028 ceph-mon[29442]: pgmap v771: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 37379/295014 objects misplaced (12.670%); 2.8 MiB/s, 44 objects/s recovering 2024-02-05T06:49:35.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:35 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2665377013' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:49:35.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:35 smithi028 ceph-mon[29442]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:49:35.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:35 smithi028 ceph-mon[29442]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:49:35.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:35 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2665377013' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:49:35.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:35 smithi028 ceph-mon[29442]: osdmap e389: 8 total, 8 up, 8 in 2024-02-05T06:49:35.516 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:35.550 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:35.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:35.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:35.652 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:35.699 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:35.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:35.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:35.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:35 smithi167 ceph-mon[32774]: pgmap v771: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 37379/295014 objects misplaced (12.670%); 2.8 MiB/s, 44 objects/s recovering 2024-02-05T06:49:35.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:35 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2665377013' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:49:35.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:35 smithi167 ceph-mon[32774]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:49:35.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:35 smithi167 ceph-mon[32774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:49:35.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:35 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2665377013' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:49:35.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:35 smithi167 ceph-mon[32774]: osdmap e389: 8 total, 8 up, 8 in 2024-02-05T06:49:35.801 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:35.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:35.840 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:35.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:35.941 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:35.987 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:35.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:35.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:36.089 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:36.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:36.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:36.230 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:36.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:36.273 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:36.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:36.317 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-05T06:49:36.374 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:36.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:36.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:36.542 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:36.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:36.597 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:36.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:36.698 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:36.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:36 smithi028 ceph-mon[29442]: 10.a scrub starts 2024-02-05T06:49:36.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:36 smithi028 ceph-mon[37984]: 10.a scrub starts 2024-02-05T06:49:36.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:36 smithi167 ceph-mon[32774]: 10.a scrub starts 2024-02-05T06:49:36.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:36.743 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:36.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:36.845 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:36.885 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:36.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:36.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:36.987 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:37.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:37.026 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:37.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:37.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:37.170 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:37.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:37.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:37.272 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:37.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:37.305 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:37.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:37.406 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:37.441 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:37.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:37.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:37.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:37.579 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:37.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:37.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:37.681 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:37.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:37 smithi028 ceph-mon[29442]: pgmap v773: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 710 B/s rd, 0 B/s wr, 203 op/s; 35722/293049 objects misplaced (12.190%); 24 MiB/s, 381 objects/s recovering 2024-02-05T06:49:37.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:37 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:49:37.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:37 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:49:37.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:37 smithi028 ceph-mon[37984]: pgmap v773: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 710 B/s rd, 0 B/s wr, 203 op/s; 35722/293049 objects misplaced (12.190%); 24 MiB/s, 381 objects/s recovering 2024-02-05T06:49:37.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:37 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:49:37.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:37 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:49:37.721 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:37.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:37.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:37.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:37 smithi167 ceph-mon[32774]: pgmap v773: 168 pgs: 1 active+clean+scrubbing, 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 160 active+clean; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 710 B/s rd, 0 B/s wr, 203 op/s; 35722/293049 objects misplaced (12.190%); 24 MiB/s, 381 objects/s recovering 2024-02-05T06:49:37.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:37 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:49:37.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:37 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:49:37.823 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:37.870 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:37.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:37.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:37.972 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:38.004 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:38.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:38.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:38.106 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:38.139 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:38.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:38.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:38.241 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:38.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:38.296 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:38.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:38.397 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:38.438 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:38.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:38.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:38.540 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:38.573 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:38.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:38.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:38.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:38.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:38 smithi028 ceph-mon[37984]: osdmap e390: 8 total, 8 up, 8 in 2024-02-05T06:49:38.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:38 smithi028 ceph-mon[29442]: osdmap e390: 8 total, 8 up, 8 in 2024-02-05T06:49:38.714 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:38.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:38.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:38.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:38 smithi167 ceph-mon[32774]: osdmap e390: 8 total, 8 up, 8 in 2024-02-05T06:49:38.816 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:38.861 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:38.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:38.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:38.964 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:38.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:38.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:39.097 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:39.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:39.125 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:39.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:39.226 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:39.267 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:39.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:39.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:39.369 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:39.411 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:39.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:39.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:39.513 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:39.551 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:39.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:39.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:39.653 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:39.695 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:39.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:39.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:39.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:39 smithi028 ceph-mon[29442]: pgmap v775: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 32641/278904 objects misplaced (11.703%); 3.7 MiB/s, 58 objects/s recovering 2024-02-05T06:49:39.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:39 smithi028 ceph-mon[37984]: pgmap v775: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 32641/278904 objects misplaced (11.703%); 3.7 MiB/s, 58 objects/s recovering 2024-02-05T06:49:39.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:39 smithi167 ceph-mon[32774]: pgmap v775: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 32641/278904 objects misplaced (11.703%); 3.7 MiB/s, 58 objects/s recovering 2024-02-05T06:49:39.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:39.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:39.841 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:39.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:39.942 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:39.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:39.978 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:39.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:40.079 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:40.114 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:40.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:40.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:40.216 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:40.246 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:40.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:40.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:40.348 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:40.389 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:40.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:40.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:40.491 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:40.539 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:40.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:40.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:40.641 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:40.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:40.676 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:40.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:40.777 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:40.819 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:40.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:40.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:40.920 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:40.953 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:40.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:40.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:41.055 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:41.086 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:41.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:41.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:41.187 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:41.230 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:41.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:41.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:41.332 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:41.367 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:41.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:41.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:41.468 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:41.495 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:41.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:41.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:41.597 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:41.630 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:41.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:41.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:41.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:41 smithi028 ceph-mon[29442]: pgmap v776: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 32641/278904 objects misplaced (11.703%) 2024-02-05T06:49:41.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:41 smithi028 ceph-mon[37984]: pgmap v776: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 32641/278904 objects misplaced (11.703%) 2024-02-05T06:49:41.732 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:41.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:41 smithi167 ceph-mon[32774]: pgmap v776: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 32641/278904 objects misplaced (11.703%) 2024-02-05T06:49:41.775 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:41.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:41.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:41.877 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:41.916 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:41.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:41.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:42.018 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:42.057 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:42.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:42.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:42.159 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:42.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:42.195 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:42.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:42.296 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:42.331 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:42.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:42.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:42.433 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:42.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:42.477 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:42.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:42.578 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:42.601 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:42.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:42.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:42.702 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:42.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:42.737 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:42.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:42.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:42.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:42.862 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:42.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:42.951 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:49:42 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:49:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:49:42.963 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:43.009 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:43.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:43.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:43.111 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:43.149 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:43.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:43.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:43.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:43.288 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:43.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:43.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:43.390 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:43.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:43.427 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:43.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:43.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:43 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:49:43.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:43 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:49:43.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:43 smithi028 ceph-mon[29442]: pgmap v777: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 181 op/s; 30182/263730 objects misplaced (11.444%) 2024-02-05T06:49:43.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:43 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:49:43.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:43 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:49:43.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:43 smithi028 ceph-mon[37984]: pgmap v777: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 181 op/s; 30182/263730 objects misplaced (11.444%) 2024-02-05T06:49:43.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:43 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:49:43.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:43 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:49:43.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:43 smithi167 ceph-mon[32774]: pgmap v777: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 181 op/s; 30182/263730 objects misplaced (11.444%) 2024-02-05T06:49:43.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:49:43 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:49:43.528 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:43.568 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:43.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:43.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:43.670 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:43.709 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:43.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:43.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:43.811 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:43.847 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:43.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:43.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:43.949 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:43.982 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:43.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:43.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:44.084 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:44.159 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:44.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:44.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:44.261 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:44.285 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:44.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:44.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:44.386 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:44.415 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:44.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:44.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:44.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:44 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:49:44.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:44 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:49:44.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:44 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:49:44.517 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:44.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:44.549 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:44.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:44.650 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:44.680 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:44.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:44.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:44.782 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:44.806 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:44.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:44.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:44.907 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:44.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:44.932 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:44.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:45.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:45.064 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:45.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:45.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:45.165 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:45.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:45.196 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:45.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:45.298 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:45.338 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:45.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:45.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:45.440 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:45.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:45 smithi028 ceph-mon[37984]: osdmap e391: 8 total, 8 up, 8 in 2024-02-05T06:49:45.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:45 smithi028 ceph-mon[37984]: pgmap v779: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 144 op/s; 28196/262653 objects misplaced (10.735%) 2024-02-05T06:49:45.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:45 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:49:45.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:45 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:49:45.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:45 smithi028 ceph-mon[29442]: osdmap e391: 8 total, 8 up, 8 in 2024-02-05T06:49:45.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:45 smithi028 ceph-mon[29442]: pgmap v779: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 144 op/s; 28196/262653 objects misplaced (10.735%) 2024-02-05T06:49:45.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:45 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:49:45.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:45 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:49:45.473 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:45.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:45.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:45.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:45 smithi167 ceph-mon[32774]: osdmap e391: 8 total, 8 up, 8 in 2024-02-05T06:49:45.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:45 smithi167 ceph-mon[32774]: pgmap v779: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 144 op/s; 28196/262653 objects misplaced (10.735%) 2024-02-05T06:49:45.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:45 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:49:45.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:45 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:49:45.575 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:45.609 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:45.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:45.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:45.710 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:45.742 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:45.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:45.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:45.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:45.885 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:45.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:45.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:45.988 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:46.023 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:46.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:46.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:46.125 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:46.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:46.165 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:46.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:46.266 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:46.298 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:46.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:46.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:46.400 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:46.433 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:46.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:46.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:46.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:46 smithi028 ceph-mon[29442]: osdmap e392: 8 total, 8 up, 8 in 2024-02-05T06:49:46.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:46 smithi028 ceph-mon[37984]: osdmap e392: 8 total, 8 up, 8 in 2024-02-05T06:49:46.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:46 smithi167 ceph-mon[32774]: osdmap e392: 8 total, 8 up, 8 in 2024-02-05T06:49:46.536 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:46.572 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:46.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:46.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:46.673 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:46.713 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:46.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:46.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:46.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:46.857 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:46.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:46.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:46.959 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:46.993 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:46.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:46.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:47.095 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:47.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:47.136 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:47.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:47.237 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:47.261 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:47.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:47.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:47.363 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:47.395 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:47.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:47.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:47.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:47 smithi028 ceph-mon[37984]: pgmap v781: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 748 op/s; 26413/260955 objects misplaced (10.122%); 18 MiB/s, 283 objects/s recovering 2024-02-05T06:49:47.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:47 smithi028 ceph-mon[29442]: pgmap v781: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 748 op/s; 26413/260955 objects misplaced (10.122%); 18 MiB/s, 283 objects/s recovering 2024-02-05T06:49:47.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:47 smithi167 ceph-mon[32774]: pgmap v781: 168 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 161 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 748 op/s; 26413/260955 objects misplaced (10.122%); 18 MiB/s, 283 objects/s recovering 2024-02-05T06:49:47.496 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:47.537 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:47.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:47.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:47.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:47.678 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:47.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:47.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:47.779 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:47.803 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:47.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:47.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:47.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:47.944 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:47.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:47.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:48.046 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:48.086 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:48.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:48.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:48.187 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:48.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:48.213 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:48.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:48.314 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:48.346 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:48.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:48.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:48.436 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T06:49:48.448 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:48.476 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:48.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:48.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:48.578 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:48.601 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:48.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:48.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:48.704 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:48.730 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:48.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:48.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:48.831 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:48.859 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:48.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:48.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:48.961 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:48.985 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:48.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:48.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:49.087 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:49.111 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:49.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:49.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:49.213 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:49.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:49.246 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:49.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:49.347 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:49.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:49.380 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:49.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:49.481 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:49.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:49 smithi167 ceph-mon[32774]: osdmap e393: 8 total, 8 up, 8 in 2024-02-05T06:49:49.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:49 smithi167 ceph-mon[32774]: pgmap v783: 168 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 162 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 831 op/s; 23840/248772 objects misplaced (9.583%); 28 MiB/s, 452 objects/s recovering 2024-02-05T06:49:49.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:49.523 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:49.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:49.624 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:49.657 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:49.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:49.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:49.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:49 smithi028 ceph-mon[29442]: osdmap e393: 8 total, 8 up, 8 in 2024-02-05T06:49:49.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:49 smithi028 ceph-mon[29442]: pgmap v783: 168 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 162 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 831 op/s; 23840/248772 objects misplaced (9.583%); 28 MiB/s, 452 objects/s recovering 2024-02-05T06:49:49.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:49 smithi028 ceph-mon[37984]: osdmap e393: 8 total, 8 up, 8 in 2024-02-05T06:49:49.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:49 smithi028 ceph-mon[37984]: pgmap v783: 168 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 162 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 831 op/s; 23840/248772 objects misplaced (9.583%); 28 MiB/s, 452 objects/s recovering 2024-02-05T06:49:49.759 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:49.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:49.785 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:49.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:49.886 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:49.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:49.911 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:49.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:50.012 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:50.036 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:50.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:50.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:50.138 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:50.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:50.180 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:50.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:50.281 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:50.324 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:50.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:50.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:50.426 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:50.460 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:50.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:50.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:50.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:50 smithi167 ceph-mon[32774]: osdmap e394: 8 total, 8 up, 8 in 2024-02-05T06:49:50.563 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:50.598 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:50.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:50.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:50.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:50.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:50 smithi028 ceph-mon[29442]: osdmap e394: 8 total, 8 up, 8 in 2024-02-05T06:49:50.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:50 smithi028 ceph-mon[37984]: osdmap e394: 8 total, 8 up, 8 in 2024-02-05T06:49:50.738 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:50.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:50.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:50.840 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:50.873 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:50.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:50.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:50.975 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:51.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:51.009 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:51.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:51.110 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:51.134 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:51.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:51.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:51.236 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:51.275 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:51.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:51.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:51.377 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:51.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:51.414 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:51.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:51.515 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:51.556 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:51.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:51.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:51.658 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:51.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:51.695 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:51.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:51.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:51 smithi028 ceph-mon[29442]: pgmap v785: 168 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 162 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 771 op/s; 23840/248772 objects misplaced (9.583%); 15 MiB/s, 240 objects/s recovering 2024-02-05T06:49:51.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:51 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:49:51.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:51 smithi028 ceph-mon[37984]: pgmap v785: 168 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 162 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 771 op/s; 23840/248772 objects misplaced (9.583%); 15 MiB/s, 240 objects/s recovering 2024-02-05T06:49:51.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:51 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:49:51.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:51 smithi167 ceph-mon[32774]: pgmap v785: 168 pgs: 1 active+clean+remapped, 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 162 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 771 op/s; 23840/248772 objects misplaced (9.583%); 15 MiB/s, 240 objects/s recovering 2024-02-05T06:49:51.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:51 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:49:51.797 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:51.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:51.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:51.932 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:51.967 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:51.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:51.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:52.069 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:52.101 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:52.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:52.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:52.146 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:49:52.146 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:49:52.203 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:52.236 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:52.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:52.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:52.337 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:52.368 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:52.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:52.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:52.470 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:52.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:52.508 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:52.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:52.609 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:52.627 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:49:52 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:49:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:49:52.643 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:52.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:52.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:52.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:52.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:52.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:52.880 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:52.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:52.904 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:52.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:53.005 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:53.030 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:53.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:53.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:53.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:53.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:53.156 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:53.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:53.257 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:53.282 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:53.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:53.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:53.384 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:53.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:53 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:49:53.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:53 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:49:53.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:53 smithi028 ceph-mon[29442]: osdmap e395: 8 total, 8 up, 8 in 2024-02-05T06:49:53.397 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:53 smithi028 ceph-mon[29442]: pgmap v787: 168 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 4 active+remapped+backfill_toofull, 162 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.31k op/s; 21482/237396 objects misplaced (9.049%); 15 MiB/s, 239 objects/s recovering 2024-02-05T06:49:53.398 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:53 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:49:53.398 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:53 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:49:53.398 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:53 smithi028 ceph-mon[37984]: osdmap e395: 8 total, 8 up, 8 in 2024-02-05T06:49:53.398 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:53 smithi028 ceph-mon[37984]: pgmap v787: 168 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 4 active+remapped+backfill_toofull, 162 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.31k op/s; 21482/237396 objects misplaced (9.049%); 15 MiB/s, 239 objects/s recovering 2024-02-05T06:49:53.426 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:53.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:53.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:53.485 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:49:53 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:49:53.486 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:53 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:49:53.486 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:53 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:49:53.486 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:53 smithi167 ceph-mon[32774]: osdmap e395: 8 total, 8 up, 8 in 2024-02-05T06:49:53.486 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:53 smithi167 ceph-mon[32774]: pgmap v787: 168 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 4 active+remapped+backfill_toofull, 162 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.31k op/s; 21482/237396 objects misplaced (9.049%); 15 MiB/s, 239 objects/s recovering 2024-02-05T06:49:53.527 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:53.567 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:53.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:53.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:53.669 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:53.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:53.709 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:53.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:53.787 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:49:53.809 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:53.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:53.865 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:53.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:53.966 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:54.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:54.007 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:54.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:54.109 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:54.151 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:54.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:54.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:54.253 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:54.290 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:54.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:54.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:54.392 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:54.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:54.427 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:54.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:54.528 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:54.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:54.569 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:54.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:54.670 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:54.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:54 smithi028 ceph-mon[37984]: osdmap e396: 8 total, 8 up, 8 in 2024-02-05T06:49:54.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:54 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:49:54.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:54 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:49:54.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:54 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:49:54.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:54 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:49:54.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:54 smithi028 ceph-mon[29442]: osdmap e396: 8 total, 8 up, 8 in 2024-02-05T06:49:54.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:54 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:49:54.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:54 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:49:54.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:54 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:49:54.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:54 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:49:54.704 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:54.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:54.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:54.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:54 smithi167 ceph-mon[32774]: osdmap e396: 8 total, 8 up, 8 in 2024-02-05T06:49:54.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:54 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:49:54.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:54 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:49:54.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:54 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:49:54.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:54 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:49:54.806 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:54.845 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:54.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:54.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:54.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:55.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:55.003 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:55.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:55.104 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:55.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:55.141 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:55.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:55.242 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:55.279 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:55.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:55.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:55.374 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:55 smithi028 ceph-mon[29442]: pgmap v789: 168 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 163 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 678 op/s; 21482/236577 objects misplaced (9.080%); 6.2 MiB/s, 98 objects/s recovering 2024-02-05T06:49:55.374 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:55 smithi028 ceph-mon[37984]: pgmap v789: 168 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 163 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 678 op/s; 21482/236577 objects misplaced (9.080%); 6.2 MiB/s, 98 objects/s recovering 2024-02-05T06:49:55.380 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:55.413 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:55.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:55.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:55.515 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:55.551 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:55.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:55.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:55.681 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:55.719 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:55.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:55.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:55.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:55 smithi167 ceph-mon[32774]: pgmap v789: 168 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 163 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 678 op/s; 21482/236577 objects misplaced (9.080%); 6.2 MiB/s, 98 objects/s recovering 2024-02-05T06:49:55.821 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:55.866 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:55.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:55.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:55.968 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:55.995 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:49:55.995 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:49:56.017 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:56.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:56.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:56.119 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:56.152 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:56.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:56.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:56.226 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:49:56.254 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:56.273 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:49:56.294 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:56.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:56.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:56.317 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:49:56.360 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:49:56.396 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:56.401 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:49:56.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:56.436 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:56.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:56.443 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:49:56.480 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:49:56.521 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:49:56.521 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:56 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/4047561327' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:49:56.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:56 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/4047561327' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:49:56.530 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:49:56.530 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:49:56.530 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:49:56.530 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:49:56.530 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:49:56.530 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:49:56.530 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:49:56.530 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:49:56.537 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:56.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:56.580 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:56.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:56.680 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:56.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:56.715 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:56.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:56.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:56 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/4047561327' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:49:56.817 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:56.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:56.854 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:56.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:56.955 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:56.990 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:56.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:56.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:57.092 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:57.126 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:57.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:57.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:57.228 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:57.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:57.260 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:57.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:57.361 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:57.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:57.394 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:57.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:57.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:57 smithi028 ceph-mon[29442]: pgmap v790: 168 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 163 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 700 B/s rd, 0 B/s wr, 622 op/s; 20849/235134 objects misplaced (8.867%); 5.6 MiB/s, 89 objects/s recovering 2024-02-05T06:49:57.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:57 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:49:57.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:57 smithi028 ceph-mon[37984]: pgmap v790: 168 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 163 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 700 B/s rd, 0 B/s wr, 622 op/s; 20849/235134 objects misplaced (8.867%); 5.6 MiB/s, 89 objects/s recovering 2024-02-05T06:49:57.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:57 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:49:57.452 INFO:teuthology.orchestra.run.smithi028.stderr:no valid command found; 10 closest matches: 2024-02-05T06:49:57.452 INFO:teuthology.orchestra.run.smithi028.stderr:0 2024-02-05T06:49:57.452 INFO:teuthology.orchestra.run.smithi028.stderr:1 2024-02-05T06:49:57.452 INFO:teuthology.orchestra.run.smithi028.stderr:2 2024-02-05T06:49:57.452 INFO:teuthology.orchestra.run.smithi028.stderr:abort 2024-02-05T06:49:57.452 INFO:teuthology.orchestra.run.smithi028.stderr:assert 2024-02-05T06:49:57.452 INFO:teuthology.orchestra.run.smithi028.stderr:bench [] [] [] [] 2024-02-05T06:49:57.452 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs debug_inject_read_zeros 2024-02-05T06:49:57.452 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs files list 2024-02-05T06:49:57.452 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs stats 2024-02-05T06:49:57.453 INFO:teuthology.orchestra.run.smithi028.stderr:bluestore allocator dump block 2024-02-05T06:49:57.453 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: invalid command 2024-02-05T06:49:57.495 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:57.534 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:57.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:57.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:57.635 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:57.681 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:57.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:57.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:57.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:57 smithi167 ceph-mon[32774]: pgmap v790: 168 pgs: 2 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 163 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 700 B/s rd, 0 B/s wr, 622 op/s; 20849/235134 objects misplaced (8.867%); 5.6 MiB/s, 89 objects/s recovering 2024-02-05T06:49:57.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:57 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:49:57.783 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:49:57.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:57.821 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:57.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:49:57.922 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:57.959 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:57.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:57.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:58.061 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:58.102 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:58.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:58.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:58.204 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:58.239 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:58.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:58.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:58.341 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:58.381 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:58.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:58.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:58.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:58 smithi028 ceph-mon[37984]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:49:58.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:58 smithi028 ceph-mon[29442]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:49:58.483 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:58.524 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:58.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:58.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:58.626 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:58.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:58.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:58.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:58 smithi167 ceph-mon[32774]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:49:58.767 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:58.799 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:58.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:58.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:58.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:49:58.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-05T06:49:58.941 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:58.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:58.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:49:59.044 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:49:59.084 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:59.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:59.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:49:59.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:49:59.234 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:59.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:59.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:49:59.336 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:59.360 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-05T06:49:59.361 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-05T06:49:59.361 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph tell osd.3 'trim stale osdmaps' 2024-02-05T06:49:59.367 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:59.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:59.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:49:59.468 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:49:59.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:59.502 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:59.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:49:59.603 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:49:59.637 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:59.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:59.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:49:59.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:49:59 smithi028 ceph-mon[29442]: pgmap v791: 168 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 164 active+clean; 4.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 19560/220359 objects misplaced (8.876%); 7.3 MiB/s, 117 objects/s recovering 2024-02-05T06:49:59.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:49:59 smithi028 ceph-mon[37984]: pgmap v791: 168 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 164 active+clean; 4.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 19560/220359 objects misplaced (8.876%); 7.3 MiB/s, 117 objects/s recovering 2024-02-05T06:49:59.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:49:59 smithi167 ceph-mon[32774]: pgmap v791: 168 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 164 active+clean; 4.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 19560/220359 objects misplaced (8.876%); 7.3 MiB/s, 117 objects/s recovering 2024-02-05T06:49:59.740 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:49:59.784 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:49:59.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:59.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:49:59.885 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:49:59.920 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:49:59.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:49:59.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:00.022 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:00.068 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:00.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:00.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:00.170 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:00.212 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:00.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:00.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:00.314 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:00.344 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:00.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:00.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:00.352 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:50:00.380 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:50:00.445 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:00.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:00.483 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:00.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:00.528 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:00 smithi028 ceph-mon[29442]: Health detail: HEALTH_WARN noscrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull 2024-02-05T06:50:00.528 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:00 smithi028 ceph-mon[29442]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-02-05T06:50:00.528 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:00 smithi028 ceph-mon[29442]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull 2024-02-05T06:50:00.528 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:00 smithi028 ceph-mon[29442]: pg 10.18 is active+remapped+backfill_toofull, acting [2,0,7] 2024-02-05T06:50:00.528 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:00 smithi028 ceph-mon[29442]: pg 10.19 is active+remapped+backfill_toofull, acting [2,5,7] 2024-02-05T06:50:00.528 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:00 smithi028 ceph-mon[37984]: Health detail: HEALTH_WARN noscrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull 2024-02-05T06:50:00.528 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:00 smithi028 ceph-mon[37984]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-02-05T06:50:00.529 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:00 smithi028 ceph-mon[37984]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull 2024-02-05T06:50:00.529 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:00 smithi028 ceph-mon[37984]: pg 10.18 is active+remapped+backfill_toofull, acting [2,0,7] 2024-02-05T06:50:00.529 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:00 smithi028 ceph-mon[37984]: pg 10.19 is active+remapped+backfill_toofull, acting [2,5,7] 2024-02-05T06:50:00.583 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:00.621 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:00.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:00.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:00.722 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:00.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:00 smithi167 ceph-mon[32774]: Health detail: HEALTH_WARN noscrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull 2024-02-05T06:50:00.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:00 smithi167 ceph-mon[32774]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-02-05T06:50:00.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:00 smithi167 ceph-mon[32774]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull 2024-02-05T06:50:00.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:00 smithi167 ceph-mon[32774]: pg 10.18 is active+remapped+backfill_toofull, acting [2,0,7] 2024-02-05T06:50:00.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:00 smithi167 ceph-mon[32774]: pg 10.19 is active+remapped+backfill_toofull, acting [2,5,7] 2024-02-05T06:50:00.757 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:00.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:00.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:00.858 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:00.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:00.903 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:00.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:01.004 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:01.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:01.048 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:01.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:01.149 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:01.186 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:01.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:01.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:01.287 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:01.331 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:01.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:01.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:01.433 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:01.465 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:01.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:01.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:01.567 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:01.599 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:01.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:01.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:01.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:01 smithi028 ceph-mon[29442]: pgmap v792: 168 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 164 active+clean; 4.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 619 B/s rd, 0 B/s wr, 687 op/s; 19560/220359 objects misplaced (8.876%); 2.6 MiB/s, 42 objects/s recovering 2024-02-05T06:50:01.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:01 smithi028 ceph-mon[37984]: pgmap v792: 168 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 164 active+clean; 4.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 619 B/s rd, 0 B/s wr, 687 op/s; 19560/220359 objects misplaced (8.876%); 2.6 MiB/s, 42 objects/s recovering 2024-02-05T06:50:01.701 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:01.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:01 smithi167 ceph-mon[32774]: pgmap v792: 168 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 164 active+clean; 4.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 619 B/s rd, 0 B/s wr, 687 op/s; 19560/220359 objects misplaced (8.876%); 2.6 MiB/s, 42 objects/s recovering 2024-02-05T06:50:01.739 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:01.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:01.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:01.841 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:01.857 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:50:01.881 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:01.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:01.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:01.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:02.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:02.061 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:02.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:02.161 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:02.191 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:02.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:02.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:02.294 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:02.331 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:02.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:02.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:02.341 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-02-05T06:50:02.434 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:02.483 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:02.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:02.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:02.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:02 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1801976102' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:50:02.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:02 smithi028 ceph-mon[29442]: from='client.45761 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:50:02.505 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:02 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1801976102' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:50:02.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:02 smithi028 ceph-mon[37984]: from='client.45761 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:50:02.585 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:02.622 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:02.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:02.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:02.723 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:02.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:02 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1801976102' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:50:02.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:02 smithi167 ceph-mon[32774]: from='client.45761 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:50:02.761 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:02.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:02.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:02.763 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:50:02 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:50:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:50:02.863 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:02.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:02.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:03.010 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:03.047 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:03.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:03.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:03.149 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:03.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:03.193 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:03.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:03.294 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:03.333 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:03.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:03.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:03.435 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:03.473 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:03.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:03.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:03.575 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:03.615 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:03.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:03.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:03.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:03 smithi028 ceph-mon[29442]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:50:03.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:03 smithi028 ceph-mon[29442]: from='client.45761 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:50:03.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:03 smithi028 ceph-mon[29442]: osdmap e397: 8 total, 8 up, 8 in 2024-02-05T06:50:03.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:03 smithi028 ceph-mon[29442]: pgmap v794: 168 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 164 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.17k op/s; 16139/204102 objects misplaced (7.907%); 10 MiB/s, 163 objects/s recovering 2024-02-05T06:50:03.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:03 smithi028 ceph-mon[37984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:50:03.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:03 smithi028 ceph-mon[37984]: from='client.45761 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:50:03.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:03 smithi028 ceph-mon[37984]: osdmap e397: 8 total, 8 up, 8 in 2024-02-05T06:50:03.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:03 smithi028 ceph-mon[37984]: pgmap v794: 168 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 164 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.17k op/s; 16139/204102 objects misplaced (7.907%); 10 MiB/s, 163 objects/s recovering 2024-02-05T06:50:03.717 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:03.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:03 smithi167 ceph-mon[32774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:50:03.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:03 smithi167 ceph-mon[32774]: from='client.45761 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:50:03.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:03 smithi167 ceph-mon[32774]: osdmap e397: 8 total, 8 up, 8 in 2024-02-05T06:50:03.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:03 smithi167 ceph-mon[32774]: pgmap v794: 168 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 164 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.17k op/s; 16139/204102 objects misplaced (7.907%); 10 MiB/s, 163 objects/s recovering 2024-02-05T06:50:03.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:50:03 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:50:03.762 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:03.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:03.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:03.864 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:03.899 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:03.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:03.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:04.001 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:04.040 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:04.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:04.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:04.142 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:04.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:04.179 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:04.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:04.280 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:04.315 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:04.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:04.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:04.417 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:04.427 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:04 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:50:04.427 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:04 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:50:04.466 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:04.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:04.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:04.569 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:04.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:04.651 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:04.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:04.651 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:50:04.651 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-02-05T06:50:04.651 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-02-05T06:50:04.652 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-02-05T06:50:04.652 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:50:04.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:04 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:50:04.752 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:04.787 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:04.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:04.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:04.889 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:04.939 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:04.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:04.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:05.037 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:50:05.041 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:05.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:05.101 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:05.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:05.202 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:05.239 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:05.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:05.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:05.341 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:05.387 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:05.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:05.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:05.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:05.532 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:05.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:05.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:05.634 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:05.669 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:05.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:05.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:05.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:05 smithi028 ceph-mon[29442]: pgmap v795: 168 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 164 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 16139/203052 objects misplaced (7.948%); 9.4 MiB/s, 151 objects/s recovering 2024-02-05T06:50:05.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:05 smithi028 ceph-mon[29442]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:50:05.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:05 smithi028 ceph-mon[29442]: osdmap e398: 8 total, 8 up, 8 in 2024-02-05T06:50:05.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:05 smithi028 ceph-mon[37984]: pgmap v795: 168 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 164 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 16139/203052 objects misplaced (7.948%); 9.4 MiB/s, 151 objects/s recovering 2024-02-05T06:50:05.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:05 smithi028 ceph-mon[37984]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:50:05.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:05 smithi028 ceph-mon[37984]: osdmap e398: 8 total, 8 up, 8 in 2024-02-05T06:50:05.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:05 smithi167 ceph-mon[32774]: pgmap v795: 168 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 164 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 16139/203052 objects misplaced (7.948%); 9.4 MiB/s, 151 objects/s recovering 2024-02-05T06:50:05.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:05 smithi167 ceph-mon[32774]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:50:05.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:05 smithi167 ceph-mon[32774]: osdmap e398: 8 total, 8 up, 8 in 2024-02-05T06:50:05.771 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:05.807 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:05.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:05.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:05.813 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T06:50:05.875 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-05T06:50:05.908 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:05.968 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:05.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:05.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:06.071 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:06.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:06.115 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:06.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:06.216 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:06.232 INFO:teuthology.orchestra.run.smithi028.stderr:no valid command found; 10 closest matches: 2024-02-05T06:50:06.232 INFO:teuthology.orchestra.run.smithi028.stderr:0 2024-02-05T06:50:06.232 INFO:teuthology.orchestra.run.smithi028.stderr:1 2024-02-05T06:50:06.232 INFO:teuthology.orchestra.run.smithi028.stderr:2 2024-02-05T06:50:06.232 INFO:teuthology.orchestra.run.smithi028.stderr:abort 2024-02-05T06:50:06.233 INFO:teuthology.orchestra.run.smithi028.stderr:assert 2024-02-05T06:50:06.233 INFO:teuthology.orchestra.run.smithi028.stderr:bench [] [] [] [] 2024-02-05T06:50:06.233 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs debug_inject_read_zeros 2024-02-05T06:50:06.233 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs files list 2024-02-05T06:50:06.233 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs stats 2024-02-05T06:50:06.233 INFO:teuthology.orchestra.run.smithi028.stderr:bluestore allocator dump block 2024-02-05T06:50:06.233 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: invalid command 2024-02-05T06:50:06.245 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:06.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:06.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:06.347 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:06.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:06.382 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:06.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:06.483 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:06.515 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:06.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:06.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:06.617 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:06.659 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:06.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:06.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:06.761 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:06.807 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:06.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:06.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:06.909 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:06.953 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:06.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:06.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:07.055 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:07.098 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:07.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:07.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:07.200 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:07.234 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:07.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:07.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:07.235 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:50:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=sqlstore.transactions t=2024-02-05T06:50:06.740327918Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-02-05T06:50:07.336 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:07.378 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:07.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:07.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:07.480 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:07.525 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:07.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:07.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:07.628 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:07.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:07.667 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:07.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:07.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:07 smithi028 ceph-mon[29442]: pgmap v797: 168 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 164 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 787 op/s; 14621/201477 objects misplaced (7.257%); 12 MiB/s, 193 objects/s recovering 2024-02-05T06:50:07.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:07 smithi028 ceph-mon[37984]: pgmap v797: 168 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 164 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 787 op/s; 14621/201477 objects misplaced (7.257%); 12 MiB/s, 193 objects/s recovering 2024-02-05T06:50:07.769 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:07.808 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:07.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:07.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:07.845 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-05T06:50:07.845 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-05T06:50:07.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph tell osd.6 'trim stale osdmaps' 2024-02-05T06:50:07.910 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:07.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:07.948 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:07.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:07.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:07 smithi167 ceph-mon[32774]: pgmap v797: 168 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 164 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 787 op/s; 14621/201477 objects misplaced (7.257%); 12 MiB/s, 193 objects/s recovering 2024-02-05T06:50:08.049 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:08.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:08.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:08.196 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:08.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:08.231 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:08.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:08.332 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:08.368 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:08.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:08.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:08.470 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:08.500 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:08.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:08.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:08.528 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:50:08.549 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:50:08.601 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:08.643 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:08.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:08.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:08.744 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:08.779 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:08.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:08.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:08.881 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:08.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:08.917 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:08.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:09.018 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:09.055 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:09.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:09.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:09.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:09.204 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:09.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:09.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:09.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:09.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:09.353 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:09.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:09.454 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:09.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:09.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:09.599 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:09.631 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:09.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:09.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:09.734 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:09.766 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:09.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:09.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:09.864 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:09 smithi028 ceph-mon[29442]: pgmap v798: 168 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 164 active+clean; 3.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s; 12907/186030 objects misplaced (6.938%); 18 MiB/s, 285 objects/s recovering 2024-02-05T06:50:09.864 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:09 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/254545248' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:50:09.864 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:09 smithi028 ceph-mon[37984]: pgmap v798: 168 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 164 active+clean; 3.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s; 12907/186030 objects misplaced (6.938%); 18 MiB/s, 285 objects/s recovering 2024-02-05T06:50:09.864 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:09 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/254545248' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:50:09.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:09.900 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:09.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:09.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:09.966 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:50:09.966 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:50:09.966 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:50:09.966 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:50:09.966 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:50:09.966 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:50:09.966 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:50:09.966 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:50:09.967 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:50:09.967 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:50:09.967 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:50:09.967 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:50:09.967 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:50:09.967 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:50:09.967 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:50:09.967 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:50:09.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:09 smithi167 ceph-mon[32774]: pgmap v798: 168 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 164 active+clean; 3.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.43k op/s; 12907/186030 objects misplaced (6.938%); 18 MiB/s, 285 objects/s recovering 2024-02-05T06:50:09.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:09 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/254545248' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:50:10.003 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:10.030 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:10.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:10.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:10.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:10.159 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:10.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:10.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:10.261 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:10.299 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:10.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:10.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:10.401 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:10.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:10.439 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:10.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:10.540 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:10.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:10.575 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:10.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:10.676 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:10.709 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:10.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:10.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:10.811 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:10.857 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:10.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:10.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:10.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:10 smithi028 ceph-mon[29442]: osdmap e399: 8 total, 8 up, 8 in 2024-02-05T06:50:10.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:10 smithi028 ceph-mon[29442]: osdmap e400: 8 total, 8 up, 8 in 2024-02-05T06:50:10.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:10 smithi028 ceph-mon[37984]: osdmap e399: 8 total, 8 up, 8 in 2024-02-05T06:50:10.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:10 smithi028 ceph-mon[37984]: osdmap e400: 8 total, 8 up, 8 in 2024-02-05T06:50:10.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:10.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:10 smithi167 ceph-mon[32774]: osdmap e399: 8 total, 8 up, 8 in 2024-02-05T06:50:10.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:10 smithi167 ceph-mon[32774]: osdmap e400: 8 total, 8 up, 8 in 2024-02-05T06:50:11.001 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:11.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:11.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:11.103 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:11.136 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:11.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:11.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:11.238 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:11.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:11.282 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:11.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:11.383 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:11.434 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:11.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:11.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:11.536 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:11.570 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:11.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:11.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:11.672 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:11.700 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:11.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:11.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:11.802 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:11.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:11.845 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:11.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:11.946 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:11.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:11 smithi028 ceph-mon[29442]: pgmap v801: 168 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 164 active+clean; 3.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 946 op/s; 12907/186030 objects misplaced (6.938%); 12 MiB/s, 187 objects/s recovering 2024-02-05T06:50:11.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:11 smithi028 ceph-mon[37984]: pgmap v801: 168 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 164 active+clean; 3.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 946 op/s; 12907/186030 objects misplaced (6.938%); 12 MiB/s, 187 objects/s recovering 2024-02-05T06:50:11.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:11.981 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:11.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:11.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:11 smithi167 ceph-mon[32774]: pgmap v801: 168 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 164 active+clean; 3.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 946 op/s; 12907/186030 objects misplaced (6.938%); 12 MiB/s, 187 objects/s recovering 2024-02-05T06:50:12.082 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:12.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:12.109 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:12.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:12.210 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:12.254 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:12.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:12.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:12.355 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:12.379 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:12.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:12.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:12.481 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:12.520 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:12.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:12.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:12.622 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:12.663 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:12.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:12.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:12.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:12.799 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:12.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:12.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:12.826 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:50:12.868 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:12 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:50:12.868 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:50:12 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:50:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:50:12.868 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:12 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:50:12.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:12.943 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:12.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:12.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:12.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:12 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:50:13.045 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:13.084 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:13.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:13.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:13.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:13.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:13.231 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:13.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:13.332 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:13.370 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:13.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:13.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:13.472 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:13.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:13.501 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:13.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:13.602 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:13.603 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:50:13 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:50:13.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:13.640 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:13.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:13.741 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:13.780 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:13.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:13.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:13.882 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:13.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:13.919 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:13.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:13.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:13 smithi028 ceph-mon[37984]: pgmap v802: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.48k op/s; 9814/170706 objects misplaced (5.749%); 17 MiB/s, 272 objects/s recovering 2024-02-05T06:50:13.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:13 smithi028 ceph-mon[29442]: pgmap v802: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.48k op/s; 9814/170706 objects misplaced (5.749%); 17 MiB/s, 272 objects/s recovering 2024-02-05T06:50:13.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:13 smithi167 ceph-mon[32774]: pgmap v802: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.48k op/s; 9814/170706 objects misplaced (5.749%); 17 MiB/s, 272 objects/s recovering 2024-02-05T06:50:14.020 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:14.053 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:14.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:14.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:14.155 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:14.197 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:14.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:14.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:14.299 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:14.333 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:14.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:14.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:14.435 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:14.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:14.474 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:14.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:14.575 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:14.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:14.639 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:14.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:14.740 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:14.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:14.780 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:14.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:14.881 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:14.922 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:14.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:14.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:14.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:14 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:50:14.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:14 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:50:14.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:14 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:50:15.024 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:15.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:15.060 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:15.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:15.161 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:15.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:15.211 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:15.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:15.214 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:50:15.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:15.362 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:15.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:15.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:15.468 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:15.502 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:15.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:15.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:15.604 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:15.647 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:15.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:15.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:15.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:15 smithi028 ceph-mon[37984]: pgmap v803: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 9814/169773 objects misplaced (5.781%); 12 MiB/s, 199 objects/s recovering 2024-02-05T06:50:15.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:15 smithi028 ceph-mon[29442]: pgmap v803: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 9814/169773 objects misplaced (5.781%); 12 MiB/s, 199 objects/s recovering 2024-02-05T06:50:15.743 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:50:15.749 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:15.802 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:15.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:15.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:15.904 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:15.940 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:15.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:15.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:15.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:15 smithi167 ceph-mon[32774]: pgmap v803: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 9814/169773 objects misplaced (5.781%); 12 MiB/s, 199 objects/s recovering 2024-02-05T06:50:16.041 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:16.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:16.073 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:16.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:16.174 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:16.227 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:16.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:16.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:16.329 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:16.363 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:16.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:16.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:16.465 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:16.500 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:16.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:16.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:16.602 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:16.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:16.641 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:16.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:16.742 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:16.774 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:16.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:16.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:16.876 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:16.910 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:16.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:16.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:16.916 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:50:16.916 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:50:17.012 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:17.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:17.046 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:17.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:17.147 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:17.180 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:17.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:17.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:17.282 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:17.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:17.319 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:17.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:17.420 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:17.461 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:17.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:17.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:17.563 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:17.598 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:17.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:17.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:17.700 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:17.733 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:17.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:17.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:17.834 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:17.867 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:17.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:17.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:17.896 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:17 smithi028 ceph-mon[29442]: pgmap v804: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 743 op/s; 8620/168198 objects misplaced (5.125%); 9.8 MiB/s, 156 objects/s recovering 2024-02-05T06:50:17.896 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:17 smithi028 ceph-mon[37984]: pgmap v804: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 743 op/s; 8620/168198 objects misplaced (5.125%); 9.8 MiB/s, 156 objects/s recovering 2024-02-05T06:50:17.937 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-05T06:50:17.969 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:17.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:17 smithi167 ceph-mon[32774]: pgmap v804: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 743 op/s; 8620/168198 objects misplaced (5.125%); 9.8 MiB/s, 156 objects/s recovering 2024-02-05T06:50:17.998 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:17.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:17.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:18.100 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:18.123 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:18.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:18.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:18.225 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:18.249 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:18.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:18.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:18.351 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:18.374 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:18.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:18.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:18.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:18.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:18.501 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:18.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:18.602 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:18.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:18.627 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:18.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:18.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:18 smithi028 ceph-mon[29442]: osdmap e401: 8 total, 8 up, 8 in 2024-02-05T06:50:18.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:18 smithi028 ceph-mon[37984]: osdmap e401: 8 total, 8 up, 8 in 2024-02-05T06:50:18.728 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:18.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:18.768 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:18.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:18.869 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:18.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:18.903 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:18.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:18.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:18 smithi167 ceph-mon[32774]: osdmap e401: 8 total, 8 up, 8 in 2024-02-05T06:50:19.004 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:19.031 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:19.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:19.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:19.132 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:19.166 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:19.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:19.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:19.268 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:19.291 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:19.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:19.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:19.393 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:19.429 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:19.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:19.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:19.531 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:19.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:19.568 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:19.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:19.669 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:19.709 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:19.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:19.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:19.811 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:19.851 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:19.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:19.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:19.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:19 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:50:19.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:19 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:50:19.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:19 smithi028 ceph-mon[29442]: pgmap v806: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s; 7738/152988 objects misplaced (5.058%); 9.5 MiB/s, 151 objects/s recovering 2024-02-05T06:50:19.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:19 smithi028 ceph-mon[29442]: osdmap e402: 8 total, 8 up, 8 in 2024-02-05T06:50:19.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:19 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1595776720' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:50:19.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:19 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:50:19.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:19 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:50:19.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:19 smithi028 ceph-mon[37984]: pgmap v806: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s; 7738/152988 objects misplaced (5.058%); 9.5 MiB/s, 151 objects/s recovering 2024-02-05T06:50:19.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:19 smithi028 ceph-mon[37984]: osdmap e402: 8 total, 8 up, 8 in 2024-02-05T06:50:19.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:19 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1595776720' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:50:19.953 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:19.985 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:19.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:19 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:50:19.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:19 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:50:19.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:19 smithi167 ceph-mon[32774]: pgmap v806: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s; 7738/152988 objects misplaced (5.058%); 9.5 MiB/s, 151 objects/s recovering 2024-02-05T06:50:19.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:19 smithi167 ceph-mon[32774]: osdmap e402: 8 total, 8 up, 8 in 2024-02-05T06:50:19.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:19 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1595776720' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:50:19.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:19.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:20.088 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:20.118 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:20.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:20.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:20.220 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:20.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:20.261 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:20.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:20.362 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:20.404 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:20.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:20.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:20.505 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:20.542 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:20.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:20.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:20.644 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:20.677 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:20.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:20.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:20.779 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:20.818 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:20.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:20.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:20.920 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:20.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:20 smithi028 ceph-mon[29442]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:50:20.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:20 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1595776720' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:50:20.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:20 smithi028 ceph-mon[29442]: osdmap e403: 8 total, 8 up, 8 in 2024-02-05T06:50:20.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:20 smithi028 ceph-mon[29442]: 10.2 scrub starts 2024-02-05T06:50:20.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:20 smithi028 ceph-mon[29442]: osdmap e404: 8 total, 8 up, 8 in 2024-02-05T06:50:20.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:20 smithi028 ceph-mon[37984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:50:20.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:20 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1595776720' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:50:20.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:20 smithi028 ceph-mon[37984]: osdmap e403: 8 total, 8 up, 8 in 2024-02-05T06:50:20.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:20 smithi028 ceph-mon[37984]: 10.2 scrub starts 2024-02-05T06:50:20.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:20 smithi028 ceph-mon[37984]: osdmap e404: 8 total, 8 up, 8 in 2024-02-05T06:50:20.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:20.964 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:20.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:20.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:20 smithi167 ceph-mon[32774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:50:20.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:20 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1595776720' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:50:20.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:20 smithi167 ceph-mon[32774]: osdmap e403: 8 total, 8 up, 8 in 2024-02-05T06:50:20.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:20 smithi167 ceph-mon[32774]: 10.2 scrub starts 2024-02-05T06:50:20.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:20 smithi167 ceph-mon[32774]: osdmap e404: 8 total, 8 up, 8 in 2024-02-05T06:50:21.064 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:21.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:21.103 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:21.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:21.204 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:21.251 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:21.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:21.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:21.354 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:21.397 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:21.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:21.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:21.499 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:21.534 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:21.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:21.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:21.636 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:21.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:21.677 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:21.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:21.778 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:21.820 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:21.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:21.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:21.921 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:21.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:21 smithi028 ceph-mon[29442]: pgmap v810: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s; 7738/152988 objects misplaced (5.058%) 2024-02-05T06:50:21.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:21 smithi028 ceph-mon[29442]: 10.2 scrub ok 2024-02-05T06:50:21.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:21 smithi028 ceph-mon[29442]: 10.10 scrub starts 2024-02-05T06:50:21.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:21 smithi028 ceph-mon[29442]: 10.3 scrub starts 2024-02-05T06:50:21.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:21 smithi028 ceph-mon[29442]: 10.14 scrub starts 2024-02-05T06:50:21.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:21 smithi028 ceph-mon[29442]: 10.3 scrub ok 2024-02-05T06:50:21.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:21 smithi028 ceph-mon[37984]: pgmap v810: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s; 7738/152988 objects misplaced (5.058%) 2024-02-05T06:50:21.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:21 smithi028 ceph-mon[37984]: 10.2 scrub ok 2024-02-05T06:50:21.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:21 smithi028 ceph-mon[37984]: 10.10 scrub starts 2024-02-05T06:50:21.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:21 smithi028 ceph-mon[37984]: 10.3 scrub starts 2024-02-05T06:50:21.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:21 smithi028 ceph-mon[37984]: 10.14 scrub starts 2024-02-05T06:50:21.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:21 smithi028 ceph-mon[37984]: 10.3 scrub ok 2024-02-05T06:50:21.956 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:21.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:21.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:21.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:21 smithi167 ceph-mon[32774]: pgmap v810: 168 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 165 active+clean; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s; 7738/152988 objects misplaced (5.058%) 2024-02-05T06:50:21.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:21 smithi167 ceph-mon[32774]: 10.2 scrub ok 2024-02-05T06:50:21.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:21 smithi167 ceph-mon[32774]: 10.10 scrub starts 2024-02-05T06:50:21.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:21 smithi167 ceph-mon[32774]: 10.3 scrub starts 2024-02-05T06:50:21.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:21 smithi167 ceph-mon[32774]: 10.14 scrub starts 2024-02-05T06:50:21.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:21 smithi167 ceph-mon[32774]: 10.3 scrub ok 2024-02-05T06:50:22.057 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:22.095 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:22.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:22.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:22.197 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:22.225 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:22.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:22.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:22.327 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:22.368 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:22.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:22.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:22.470 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:22.507 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:22.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:22.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:22.609 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:22.642 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:22.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:22.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:22.732 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-02-05T06:50:22.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:22.784 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:22.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:22.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:22.886 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:22.928 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:22.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:22.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:22.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:22 smithi028 ceph-mon[37984]: 10.15 scrub starts 2024-02-05T06:50:22.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:22 smithi028 ceph-mon[37984]: 10.10 scrub ok 2024-02-05T06:50:22.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:22 smithi028 ceph-mon[37984]: 10.15 scrub ok 2024-02-05T06:50:22.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:22 smithi028 ceph-mon[37984]: 4.18 scrub starts 2024-02-05T06:50:22.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:22 smithi028 ceph-mon[37984]: 4.18 scrub ok 2024-02-05T06:50:22.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:22 smithi028 ceph-mon[37984]: 10.14 scrub ok 2024-02-05T06:50:22.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:22 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:50:22.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:22 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:50:22.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:22 smithi028 ceph-mon[37984]: 10.9 scrub starts 2024-02-05T06:50:22.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:22 smithi028 ceph-mon[37984]: 10.5 scrub starts 2024-02-05T06:50:22.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:22 smithi028 ceph-mon[37984]: 6.10 scrub starts 2024-02-05T06:50:22.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:22 smithi028 ceph-mon[37984]: 6.10 scrub ok 2024-02-05T06:50:22.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:22 smithi028 ceph-mon[37984]: 10.9 scrub ok 2024-02-05T06:50:22.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:22 smithi028 ceph-mon[37984]: 10.13 scrub starts 2024-02-05T06:50:22.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:22 smithi028 ceph-mon[37984]: 10.5 scrub ok 2024-02-05T06:50:22.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:22 smithi028 ceph-mon[29442]: 10.15 scrub starts 2024-02-05T06:50:22.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:22 smithi028 ceph-mon[29442]: 10.10 scrub ok 2024-02-05T06:50:22.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:22 smithi028 ceph-mon[29442]: 10.15 scrub ok 2024-02-05T06:50:22.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:22 smithi028 ceph-mon[29442]: 4.18 scrub starts 2024-02-05T06:50:22.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:22 smithi028 ceph-mon[29442]: 4.18 scrub ok 2024-02-05T06:50:22.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:22 smithi028 ceph-mon[29442]: 10.14 scrub ok 2024-02-05T06:50:22.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:22 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:50:22.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:22 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:50:22.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:22 smithi028 ceph-mon[29442]: 10.9 scrub starts 2024-02-05T06:50:22.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:22 smithi028 ceph-mon[29442]: 10.5 scrub starts 2024-02-05T06:50:22.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:22 smithi028 ceph-mon[29442]: 6.10 scrub starts 2024-02-05T06:50:22.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:22 smithi028 ceph-mon[29442]: 6.10 scrub ok 2024-02-05T06:50:22.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:22 smithi028 ceph-mon[29442]: 10.9 scrub ok 2024-02-05T06:50:22.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:22 smithi028 ceph-mon[29442]: 10.13 scrub starts 2024-02-05T06:50:22.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:22 smithi028 ceph-mon[29442]: 10.5 scrub ok 2024-02-05T06:50:22.953 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:50:22 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:50:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:50:22.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:22 smithi167 ceph-mon[32774]: 10.15 scrub starts 2024-02-05T06:50:22.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:22 smithi167 ceph-mon[32774]: 10.10 scrub ok 2024-02-05T06:50:22.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:22 smithi167 ceph-mon[32774]: 10.15 scrub ok 2024-02-05T06:50:22.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:22 smithi167 ceph-mon[32774]: 4.18 scrub starts 2024-02-05T06:50:22.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:22 smithi167 ceph-mon[32774]: 4.18 scrub ok 2024-02-05T06:50:22.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:22 smithi167 ceph-mon[32774]: 10.14 scrub ok 2024-02-05T06:50:22.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:22 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:50:22.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:22 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:50:22.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:22 smithi167 ceph-mon[32774]: 10.9 scrub starts 2024-02-05T06:50:22.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:22 smithi167 ceph-mon[32774]: 10.5 scrub starts 2024-02-05T06:50:22.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:22 smithi167 ceph-mon[32774]: 6.10 scrub starts 2024-02-05T06:50:22.987 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:22 smithi167 ceph-mon[32774]: 6.10 scrub ok 2024-02-05T06:50:22.987 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:22 smithi167 ceph-mon[32774]: 10.9 scrub ok 2024-02-05T06:50:22.987 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:22 smithi167 ceph-mon[32774]: 10.13 scrub starts 2024-02-05T06:50:22.987 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:22 smithi167 ceph-mon[32774]: 10.5 scrub ok 2024-02-05T06:50:23.030 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:23.065 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:23.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:23.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:23.167 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:23.201 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:23.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:23.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:23.303 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:23.343 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:23.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:23.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:23.445 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:23.479 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:23.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:23.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:23.581 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:23.621 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:23.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:23.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:23.663 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:50:23 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:50:23.724 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:23.780 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:23.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:23.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:23.882 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:23.914 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:23.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:23.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:23.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:23 smithi028 ceph-mon[37984]: pgmap v811: 168 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 164 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 6997/137976 objects misplaced (5.071%) 2024-02-05T06:50:23.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:23 smithi028 ceph-mon[37984]: osdmap e405: 8 total, 8 up, 8 in 2024-02-05T06:50:23.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:23 smithi028 ceph-mon[37984]: 10.6 scrub starts 2024-02-05T06:50:23.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:23 smithi028 ceph-mon[37984]: 10.13 scrub ok 2024-02-05T06:50:23.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:23 smithi028 ceph-mon[37984]: 1.0 scrub starts 2024-02-05T06:50:23.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:23 smithi028 ceph-mon[37984]: 1.0 scrub ok 2024-02-05T06:50:23.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:23 smithi028 ceph-mon[37984]: 4.1c scrub starts 2024-02-05T06:50:23.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:23 smithi028 ceph-mon[37984]: 4.1c scrub ok 2024-02-05T06:50:23.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:23 smithi028 ceph-mon[37984]: 10.6 scrub ok 2024-02-05T06:50:23.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:23 smithi028 ceph-mon[29442]: pgmap v811: 168 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 164 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 6997/137976 objects misplaced (5.071%) 2024-02-05T06:50:23.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:23 smithi028 ceph-mon[29442]: osdmap e405: 8 total, 8 up, 8 in 2024-02-05T06:50:23.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:23 smithi028 ceph-mon[29442]: 10.6 scrub starts 2024-02-05T06:50:23.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:23 smithi028 ceph-mon[29442]: 10.13 scrub ok 2024-02-05T06:50:23.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:23 smithi028 ceph-mon[29442]: 1.0 scrub starts 2024-02-05T06:50:23.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:23 smithi028 ceph-mon[29442]: 1.0 scrub ok 2024-02-05T06:50:23.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:23 smithi028 ceph-mon[29442]: 4.1c scrub starts 2024-02-05T06:50:23.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:23 smithi028 ceph-mon[29442]: 4.1c scrub ok 2024-02-05T06:50:23.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:23 smithi028 ceph-mon[29442]: 10.6 scrub ok 2024-02-05T06:50:23.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:23 smithi167 ceph-mon[32774]: pgmap v811: 168 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 164 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 6997/137976 objects misplaced (5.071%) 2024-02-05T06:50:23.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:23 smithi167 ceph-mon[32774]: osdmap e405: 8 total, 8 up, 8 in 2024-02-05T06:50:23.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:23 smithi167 ceph-mon[32774]: 10.6 scrub starts 2024-02-05T06:50:23.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:23 smithi167 ceph-mon[32774]: 10.13 scrub ok 2024-02-05T06:50:23.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:23 smithi167 ceph-mon[32774]: 1.0 scrub starts 2024-02-05T06:50:23.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:23 smithi167 ceph-mon[32774]: 1.0 scrub ok 2024-02-05T06:50:23.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:23 smithi167 ceph-mon[32774]: 4.1c scrub starts 2024-02-05T06:50:23.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:23 smithi167 ceph-mon[32774]: 4.1c scrub ok 2024-02-05T06:50:23.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:23 smithi167 ceph-mon[32774]: 10.6 scrub ok 2024-02-05T06:50:24.017 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:24.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:24.051 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:24.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:24.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:24.197 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:24.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:24.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:24.298 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:24.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:24.335 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:24.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:24.434 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:50:24.436 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:24.475 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:50:24.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:24.483 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:24.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:24.514 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:50:24.561 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:50:24.585 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:24.602 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:50:24.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:24.623 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:24.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:24.645 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:50:24.697 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:50:24.725 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:24.745 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:50:24.754 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:50:24.754 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:50:24.754 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:50:24.754 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:50:24.755 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:50:24.755 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:50:24.755 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:50:24.755 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:50:24.759 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:24.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:24.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:24.795 INFO:teuthology.orchestra.run.smithi028.stderr:no valid command found; 10 closest matches: 2024-02-05T06:50:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:0 2024-02-05T06:50:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:1 2024-02-05T06:50:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:2 2024-02-05T06:50:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:abort 2024-02-05T06:50:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:assert 2024-02-05T06:50:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:bench [] [] [] [] 2024-02-05T06:50:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs debug_inject_read_zeros 2024-02-05T06:50:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs files list 2024-02-05T06:50:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs stats 2024-02-05T06:50:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:bluestore allocator dump block 2024-02-05T06:50:24.796 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: invalid command 2024-02-05T06:50:24.861 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:24.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:24.897 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:24.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:24.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:24 smithi028 ceph-mon[29442]: 4.1b scrub starts 2024-02-05T06:50:24.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:24 smithi028 ceph-mon[29442]: 4.1b scrub ok 2024-02-05T06:50:24.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:24 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:50:24.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:24 smithi028 ceph-mon[29442]: 1.1 scrub starts 2024-02-05T06:50:24.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:24 smithi028 ceph-mon[29442]: 1.1 scrub ok 2024-02-05T06:50:24.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:24 smithi028 ceph-mon[29442]: 10.12 scrub starts 2024-02-05T06:50:24.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:24 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3807985593' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:50:24.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:24 smithi028 ceph-mon[37984]: 4.1b scrub starts 2024-02-05T06:50:24.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:24 smithi028 ceph-mon[37984]: 4.1b scrub ok 2024-02-05T06:50:24.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:24 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:50:24.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:24 smithi028 ceph-mon[37984]: 1.1 scrub starts 2024-02-05T06:50:24.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:24 smithi028 ceph-mon[37984]: 1.1 scrub ok 2024-02-05T06:50:24.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:24 smithi028 ceph-mon[37984]: 10.12 scrub starts 2024-02-05T06:50:24.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:24 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3807985593' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:50:24.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:24 smithi167 ceph-mon[32774]: 4.1b scrub starts 2024-02-05T06:50:24.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:24 smithi167 ceph-mon[32774]: 4.1b scrub ok 2024-02-05T06:50:24.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:24 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:50:24.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:24 smithi167 ceph-mon[32774]: 1.1 scrub starts 2024-02-05T06:50:24.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:24 smithi167 ceph-mon[32774]: 1.1 scrub ok 2024-02-05T06:50:24.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:24 smithi167 ceph-mon[32774]: 10.12 scrub starts 2024-02-05T06:50:24.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:24 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3807985593' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:50:24.998 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:25.035 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:25.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:25.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:25.137 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:25.182 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:25.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:25.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:25.284 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:25.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:25.327 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:25.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:25.428 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:25.474 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:25.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:25.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:25.576 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:25.614 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:25.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:25.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:25.661 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:50:25.717 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:25.754 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:25.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:25.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:25.856 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:25.891 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:25.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:25.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:25.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:25 smithi028 ceph-mon[37984]: pgmap v813: 168 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 164 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 868 B/s rd, 0 op/s; 6997/136935 objects misplaced (5.110%) 2024-02-05T06:50:25.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:25 smithi028 ceph-mon[37984]: 10.12 scrub ok 2024-02-05T06:50:25.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:25 smithi028 ceph-mon[37984]: 3.16 scrub starts 2024-02-05T06:50:25.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:25 smithi028 ceph-mon[37984]: 3.16 scrub ok 2024-02-05T06:50:25.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:25 smithi028 ceph-mon[37984]: 3.11 scrub starts 2024-02-05T06:50:25.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:25 smithi028 ceph-mon[37984]: 3.11 scrub ok 2024-02-05T06:50:25.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:25 smithi028 ceph-mon[37984]: 10.4 scrub starts 2024-02-05T06:50:25.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:25 smithi028 ceph-mon[37984]: osdmap e406: 8 total, 8 up, 8 in 2024-02-05T06:50:25.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:25 smithi028 ceph-mon[37984]: 10.4 scrub ok 2024-02-05T06:50:25.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:25 smithi028 ceph-mon[29442]: pgmap v813: 168 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 164 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 868 B/s rd, 0 op/s; 6997/136935 objects misplaced (5.110%) 2024-02-05T06:50:25.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:25 smithi028 ceph-mon[29442]: 10.12 scrub ok 2024-02-05T06:50:25.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:25 smithi028 ceph-mon[29442]: 3.16 scrub starts 2024-02-05T06:50:25.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:25 smithi028 ceph-mon[29442]: 3.16 scrub ok 2024-02-05T06:50:25.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:25 smithi028 ceph-mon[29442]: 3.11 scrub starts 2024-02-05T06:50:25.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:25 smithi028 ceph-mon[29442]: 3.11 scrub ok 2024-02-05T06:50:25.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:25 smithi028 ceph-mon[29442]: 10.4 scrub starts 2024-02-05T06:50:25.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:25 smithi028 ceph-mon[29442]: osdmap e406: 8 total, 8 up, 8 in 2024-02-05T06:50:25.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:25 smithi028 ceph-mon[29442]: 10.4 scrub ok 2024-02-05T06:50:25.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:25 smithi167 ceph-mon[32774]: pgmap v813: 168 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 164 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 868 B/s rd, 0 op/s; 6997/136935 objects misplaced (5.110%) 2024-02-05T06:50:25.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:25 smithi167 ceph-mon[32774]: 10.12 scrub ok 2024-02-05T06:50:25.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:25 smithi167 ceph-mon[32774]: 3.16 scrub starts 2024-02-05T06:50:25.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:25 smithi167 ceph-mon[32774]: 3.16 scrub ok 2024-02-05T06:50:25.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:25 smithi167 ceph-mon[32774]: 3.11 scrub starts 2024-02-05T06:50:25.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:25 smithi167 ceph-mon[32774]: 3.11 scrub ok 2024-02-05T06:50:25.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:25 smithi167 ceph-mon[32774]: 10.4 scrub starts 2024-02-05T06:50:25.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:25 smithi167 ceph-mon[32774]: osdmap e406: 8 total, 8 up, 8 in 2024-02-05T06:50:25.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:25 smithi167 ceph-mon[32774]: 10.4 scrub ok 2024-02-05T06:50:25.993 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:26.026 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:26.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:26.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:26.128 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:26.157 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:26.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:26.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:26.168 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-05T06:50:26.169 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-05T06:50:26.169 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph tell osd.4 'trim stale osdmaps' 2024-02-05T06:50:26.259 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:26.292 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:26.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:26.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:26.394 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:26.465 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:26.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:26.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:26.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:26.600 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:26.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:26.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:26.702 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:26.736 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:26.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:26.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:26.838 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:26.882 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:26.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:26.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:26.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:26 smithi028 ceph-mon[29442]: 10.17 scrub starts 2024-02-05T06:50:26.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:26 smithi028 ceph-mon[29442]: 10.17 scrub ok 2024-02-05T06:50:26.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:26 smithi028 ceph-mon[29442]: 6.f scrub starts 2024-02-05T06:50:26.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:26 smithi028 ceph-mon[29442]: 10.8 scrub starts 2024-02-05T06:50:26.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:26 smithi028 ceph-mon[29442]: 6.f scrub ok 2024-02-05T06:50:26.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:26 smithi028 ceph-mon[29442]: 10.a scrub starts 2024-02-05T06:50:26.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:26 smithi028 ceph-mon[29442]: 10.8 scrub ok 2024-02-05T06:50:26.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:26 smithi028 ceph-mon[37984]: 10.17 scrub starts 2024-02-05T06:50:26.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:26 smithi028 ceph-mon[37984]: 10.17 scrub ok 2024-02-05T06:50:26.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:26 smithi028 ceph-mon[37984]: 6.f scrub starts 2024-02-05T06:50:26.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:26 smithi028 ceph-mon[37984]: 10.8 scrub starts 2024-02-05T06:50:26.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:26 smithi028 ceph-mon[37984]: 6.f scrub ok 2024-02-05T06:50:26.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:26 smithi028 ceph-mon[37984]: 10.a scrub starts 2024-02-05T06:50:26.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:26 smithi028 ceph-mon[37984]: 10.8 scrub ok 2024-02-05T06:50:26.984 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:26.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:26 smithi167 ceph-mon[32774]: 10.17 scrub starts 2024-02-05T06:50:26.987 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:26 smithi167 ceph-mon[32774]: 10.17 scrub ok 2024-02-05T06:50:26.987 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:26 smithi167 ceph-mon[32774]: 6.f scrub starts 2024-02-05T06:50:26.987 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:26 smithi167 ceph-mon[32774]: 10.8 scrub starts 2024-02-05T06:50:26.987 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:26 smithi167 ceph-mon[32774]: 6.f scrub ok 2024-02-05T06:50:26.987 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:26 smithi167 ceph-mon[32774]: 10.a scrub starts 2024-02-05T06:50:26.987 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:26 smithi167 ceph-mon[32774]: 10.8 scrub ok 2024-02-05T06:50:27.020 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:27.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:27.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:27.122 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:27.159 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:27.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:27.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:27.261 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:27.293 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:27.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:27.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:27.396 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:27.430 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:27.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:27.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:27.466 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T06:50:27.531 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:27.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:27.568 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:27.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:27.669 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:27.706 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:27.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:27.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:27.796 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:27 smithi028 ceph-mon[37984]: 6.1b scrub starts 2024-02-05T06:50:27.796 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:27 smithi028 ceph-mon[37984]: 6.1b scrub ok 2024-02-05T06:50:27.796 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:27 smithi028 ceph-mon[37984]: pgmap v815: 168 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 164 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 813 B/s rd, 0 op/s; 6997/135726 objects misplaced (5.155%) 2024-02-05T06:50:27.796 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:27 smithi028 ceph-mon[37984]: 5.6 scrub starts 2024-02-05T06:50:27.796 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:27 smithi028 ceph-mon[37984]: 5.6 scrub ok 2024-02-05T06:50:27.796 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:27 smithi028 ceph-mon[37984]: 10.a scrub ok 2024-02-05T06:50:27.796 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:27 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:50:27.796 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:27 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:50:27.796 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:27 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:50:27.797 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:27 smithi028 ceph-mon[37984]: 10.d scrub starts 2024-02-05T06:50:27.797 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:27 smithi028 ceph-mon[37984]: 1.3 scrub starts 2024-02-05T06:50:27.797 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:27 smithi028 ceph-mon[37984]: 1.3 scrub ok 2024-02-05T06:50:27.798 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:27 smithi028 ceph-mon[29442]: 6.1b scrub starts 2024-02-05T06:50:27.798 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:27 smithi028 ceph-mon[29442]: 6.1b scrub ok 2024-02-05T06:50:27.798 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:27 smithi028 ceph-mon[29442]: pgmap v815: 168 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 164 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 813 B/s rd, 0 op/s; 6997/135726 objects misplaced (5.155%) 2024-02-05T06:50:27.798 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:27 smithi028 ceph-mon[29442]: 5.6 scrub starts 2024-02-05T06:50:27.798 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:27 smithi028 ceph-mon[29442]: 5.6 scrub ok 2024-02-05T06:50:27.798 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:27 smithi028 ceph-mon[29442]: 10.a scrub ok 2024-02-05T06:50:27.798 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:27 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:50:27.799 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:27 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:50:27.799 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:27 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:50:27.799 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:27 smithi028 ceph-mon[29442]: 10.d scrub starts 2024-02-05T06:50:27.799 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:27 smithi028 ceph-mon[29442]: 1.3 scrub starts 2024-02-05T06:50:27.799 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:27 smithi028 ceph-mon[29442]: 1.3 scrub ok 2024-02-05T06:50:27.807 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:27.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:27.861 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:27.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:27.963 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:27.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:27 smithi167 ceph-mon[32774]: 6.1b scrub starts 2024-02-05T06:50:27.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:27 smithi167 ceph-mon[32774]: 6.1b scrub ok 2024-02-05T06:50:27.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:27 smithi167 ceph-mon[32774]: pgmap v815: 168 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 164 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 813 B/s rd, 0 op/s; 6997/135726 objects misplaced (5.155%) 2024-02-05T06:50:27.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:27 smithi167 ceph-mon[32774]: 5.6 scrub starts 2024-02-05T06:50:27.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:27 smithi167 ceph-mon[32774]: 5.6 scrub ok 2024-02-05T06:50:27.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:27 smithi167 ceph-mon[32774]: 10.a scrub ok 2024-02-05T06:50:27.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:27 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:50:27.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:27 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:50:27.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:27 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:50:27.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:27 smithi167 ceph-mon[32774]: 10.d scrub starts 2024-02-05T06:50:27.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:27 smithi167 ceph-mon[32774]: 1.3 scrub starts 2024-02-05T06:50:27.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:27 smithi167 ceph-mon[32774]: 1.3 scrub ok 2024-02-05T06:50:28.001 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:28.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:28.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:28.101 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:50:28.103 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:28.155 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:28.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:28.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:28.257 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:28.292 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:28.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:28.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:28.393 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:28.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:28.425 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:28.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:28.526 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:28.563 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:28.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:28.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:28.665 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:28.700 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:28.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:28.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:28.802 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:28.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:28.838 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:28.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:28.891 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:50:28.939 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:28.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:28 smithi028 ceph-mon[37984]: 6.11 scrub starts 2024-02-05T06:50:28.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:28 smithi028 ceph-mon[37984]: 6.11 scrub ok 2024-02-05T06:50:28.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:28 smithi028 ceph-mon[37984]: 4.15 scrub starts 2024-02-05T06:50:28.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:28 smithi028 ceph-mon[37984]: 4.15 scrub ok 2024-02-05T06:50:28.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:28 smithi028 ceph-mon[37984]: osdmap e407: 8 total, 8 up, 8 in 2024-02-05T06:50:28.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:28 smithi028 ceph-mon[37984]: 10.c scrub starts 2024-02-05T06:50:28.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:28 smithi028 ceph-mon[37984]: 10.d scrub ok 2024-02-05T06:50:28.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:28 smithi028 ceph-mon[29442]: 6.11 scrub starts 2024-02-05T06:50:28.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:28 smithi028 ceph-mon[29442]: 6.11 scrub ok 2024-02-05T06:50:28.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:28 smithi028 ceph-mon[29442]: 4.15 scrub starts 2024-02-05T06:50:28.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:28 smithi028 ceph-mon[29442]: 4.15 scrub ok 2024-02-05T06:50:28.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:28 smithi028 ceph-mon[29442]: osdmap e407: 8 total, 8 up, 8 in 2024-02-05T06:50:28.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:28 smithi028 ceph-mon[29442]: 10.c scrub starts 2024-02-05T06:50:28.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:28 smithi028 ceph-mon[29442]: 10.d scrub ok 2024-02-05T06:50:28.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:28 smithi167 ceph-mon[32774]: 6.11 scrub starts 2024-02-05T06:50:28.987 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:28 smithi167 ceph-mon[32774]: 6.11 scrub ok 2024-02-05T06:50:28.987 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:28 smithi167 ceph-mon[32774]: 4.15 scrub starts 2024-02-05T06:50:28.987 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:28 smithi167 ceph-mon[32774]: 4.15 scrub ok 2024-02-05T06:50:28.987 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:28 smithi167 ceph-mon[32774]: osdmap e407: 8 total, 8 up, 8 in 2024-02-05T06:50:28.987 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:28 smithi167 ceph-mon[32774]: 10.c scrub starts 2024-02-05T06:50:28.987 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:28 smithi167 ceph-mon[32774]: 10.d scrub ok 2024-02-05T06:50:29.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:29.003 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:29.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:29.104 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:29.141 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:29.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:29.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:29.243 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:29.281 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:29.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:29.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:29.383 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:29.429 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:29.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:29.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:29.531 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:29.571 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:29.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:29.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:29.673 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:29.687 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:50:29.687 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-02-05T06:50:29.688 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.688 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45587.0:205652 10.17 10:ee780e2a:::benchmark_data_smithi028_154709_object84651:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e407)", 2024-02-05T06:50:29.688 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:50:29.559176+0000", 2024-02-05T06:50:29.688 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.011933127, 2024-02-05T06:50:29.688 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011966862, 2024-02-05T06:50:29.688 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:50:29.688 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "waiting for sub ops", 2024-02-05T06:50:29.688 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:50:29.688 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45587", 2024-02-05T06:50:29.688 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:50:29.688 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 205652 2024-02-05T06:50:29.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.689 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:50:29.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:50:29.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.559176+0000", 2024-02-05T06:50:29.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:50:29.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:50:29.689 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.559176+0000", 2024-02-05T06:50:29.689 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:50:29.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.689 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.689 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:50:29.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.559178+0000", 2024-02-05T06:50:29.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1009999999999999e-06 2024-02-05T06:50:29.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:50:29.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.559184+0000", 2024-02-05T06:50:29.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9939999999999997e-06 2024-02-05T06:50:29.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.690 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.690 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:50:29.690 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.559200+0000", 2024-02-05T06:50:29.690 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6569999999999999e-05 2024-02-05T06:50:29.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:50:29.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.559209+0000", 2024-02-05T06:50:29.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0560000000000005e-06 2024-02-05T06:50:29.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:50:29.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.559823+0000", 2024-02-05T06:50:29.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00061399600000000001 2024-02-05T06:50:29.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.691 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.691 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:50:29.691 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.559936+0000", 2024-02-05T06:50:29.691 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000113044 2024-02-05T06:50:29.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 4,6", 2024-02-05T06:50:29.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.560405+0000", 2024-02-05T06:50:29.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00046909600000000001 2024-02-05T06:50:29.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.692 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.692 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:50:29.692 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.566159+0000", 2024-02-05T06:50:29.692 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0057547459999999998 2024-02-05T06:50:29.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.693 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:50:29.693 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.566173+0000", 2024-02-05T06:50:29.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3603e-05 2024-02-05T06:50:29.693 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:50:29.693 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:50:29.693 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:50:29.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.693 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.693 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45587.0:205655 10.15 10:ae48f5d0:::benchmark_data_smithi028_154709_object84654:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e407)", 2024-02-05T06:50:29.693 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:50:29.559224+0000", 2024-02-05T06:50:29.693 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.011885224, 2024-02-05T06:50:29.693 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012054441000000001, 2024-02-05T06:50:29.694 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:50:29.694 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "waiting for sub ops", 2024-02-05T06:50:29.694 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:50:29.694 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45587", 2024-02-05T06:50:29.694 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:50:29.694 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 205655 2024-02-05T06:50:29.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.694 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:50:29.694 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.694 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:50:29.694 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.559224+0000", 2024-02-05T06:50:29.694 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:50:29.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.695 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:50:29.695 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.559224+0000", 2024-02-05T06:50:29.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:50:29.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.695 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:50:29.695 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.559224+0000", 2024-02-05T06:50:29.695 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0999999999999998e-07 2024-02-05T06:50:29.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.695 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.695 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:50:29.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.559228+0000", 2024-02-05T06:50:29.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4350000000000001e-06 2024-02-05T06:50:29.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:50:29.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.559234+0000", 2024-02-05T06:50:29.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2589999999999997e-06 2024-02-05T06:50:29.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.696 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:50:29.696 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.559240+0000", 2024-02-05T06:50:29.696 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.524e-06 2024-02-05T06:50:29.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.696 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:50:29.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.559335+0000", 2024-02-05T06:50:29.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4869000000000003e-05 2024-02-05T06:50:29.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:50:29.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.559459+0000", 2024-02-05T06:50:29.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012425700000000001 2024-02-05T06:50:29.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.697 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,5", 2024-02-05T06:50:29.697 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.559922+0000", 2024-02-05T06:50:29.697 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00046361500000000001 2024-02-05T06:50:29.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.697 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:50:29.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.569427+0000", 2024-02-05T06:50:29.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0095048690000000009 2024-02-05T06:50:29.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.698 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:50:29.698 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.569441+0000", 2024-02-05T06:50:29.698 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3478e-05 2024-02-05T06:50:29.698 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:50:29.698 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:50:29.698 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:50:29.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.698 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.698 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.45587.0:205653 10.f e407/345 10:f12f6b79:::benchmark_data_smithi028_154709_object84652:head v 407'15308, mlcod=407'15305)", 2024-02-05T06:50:29.698 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:50:29.563329+0000", 2024-02-05T06:50:29.699 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0077799180000000002, 2024-02-05T06:50:29.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0080390710000000001, 2024-02-05T06:50:29.699 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:50:29.699 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-02-05T06:50:29.699 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:50:29.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:50:29.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.563329+0000", 2024-02-05T06:50:29.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:50:29.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.699 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.699 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:50:29.699 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.563329+0000", 2024-02-05T06:50:29.699 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:50:29.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:50:29.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.563330+0000", 2024-02-05T06:50:29.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4339999999999999e-06 2024-02-05T06:50:29.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:50:29.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.563341+0000", 2024-02-05T06:50:29.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0737000000000001e-05 2024-02-05T06:50:29.700 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.700 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.700 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:50:29.700 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.563351+0000", 2024-02-05T06:50:29.700 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7720000000000006e-06 2024-02-05T06:50:29.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:50:29.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.563359+0000", 2024-02-05T06:50:29.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.892e-06 2024-02-05T06:50:29.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:50:29.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.565865+0000", 2024-02-05T06:50:29.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002506369 2024-02-05T06:50:29.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.701 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.701 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:50:29.701 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.565884+0000", 2024-02-05T06:50:29.701 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9003000000000001e-05 2024-02-05T06:50:29.702 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:50:29.702 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:50:29.702 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:50:29.702 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.702 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.45587.0:205656 10.f e407/345 10:fe7c22b6:::benchmark_data_smithi028_154709_object84655:head v 407'15309, mlcod=407'15305)", 2024-02-05T06:50:29.702 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:50:29.563409+0000", 2024-02-05T06:50:29.702 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0076999850000000003, 2024-02-05T06:50:29.702 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0080231940000000009, 2024-02-05T06:50:29.702 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:50:29.702 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-02-05T06:50:29.702 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:50:29.702 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.702 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:50:29.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.563409+0000", 2024-02-05T06:50:29.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:50:29.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:50:29.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.563409+0000", 2024-02-05T06:50:29.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:50:29.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:50:29.703 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.563410+0000", 2024-02-05T06:50:29.703 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7700000000000004e-07 2024-02-05T06:50:29.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.703 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.703 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:50:29.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.563418+0000", 2024-02-05T06:50:29.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3189999999999995e-06 2024-02-05T06:50:29.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:50:29.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.563424+0000", 2024-02-05T06:50:29.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.0680000000000002e-06 2024-02-05T06:50:29.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.704 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:50:29.704 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.563429+0000", 2024-02-05T06:50:29.704 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.2700000000000004e-06 2024-02-05T06:50:29.704 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:50:29.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.568898+0000", 2024-02-05T06:50:29.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0054691849999999997 2024-02-05T06:50:29.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.705 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:50:29.705 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.568918+0000", 2024-02-05T06:50:29.705 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9318999999999999e-05 2024-02-05T06:50:29.705 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:50:29.705 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:50:29.705 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:50:29.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.705 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.45587.0:205651 10.3 e407/372 10:c275e3e5:::benchmark_data_smithi028_154709_object84650:head v 407'9801, mlcod=407'9796)", 2024-02-05T06:50:29.705 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:50:29.566578+0000", 2024-02-05T06:50:29.706 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0045306529999999999, 2024-02-05T06:50:29.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049135409999999996, 2024-02-05T06:50:29.706 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:50:29.706 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-02-05T06:50:29.706 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:50:29.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:50:29.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.566578+0000", 2024-02-05T06:50:29.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:50:29.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.706 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:50:29.706 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.566578+0000", 2024-02-05T06:50:29.706 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:50:29.706 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.706 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:50:29.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.566580+0000", 2024-02-05T06:50:29.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.313e-06 2024-02-05T06:50:29.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:50:29.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.566587+0000", 2024-02-05T06:50:29.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1069999999999999e-06 2024-02-05T06:50:29.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:50:29.707 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.566597+0000", 2024-02-05T06:50:29.707 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9720000000000004e-06 2024-02-05T06:50:29.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.707 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.707 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:50:29.708 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.566604+0000", 2024-02-05T06:50:29.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1520000000000003e-06 2024-02-05T06:50:29.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:50:29.708 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.570765+0000", 2024-02-05T06:50:29.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0041607429999999997 2024-02-05T06:50:29.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.708 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.708 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:50:29.708 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.570785+0000", 2024-02-05T06:50:29.708 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0186e-05 2024-02-05T06:50:29.708 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:50:29.709 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:50:29.709 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:50:29.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.709 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.45587.0:205660 10.f e407/345)", 2024-02-05T06:50:29.709 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:50:29.566624+0000", 2024-02-05T06:50:29.709 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0044844819999999997, 2024-02-05T06:50:29.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0049233580000000001, 2024-02-05T06:50:29.709 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:50:29.709 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "queued for pg", 2024-02-05T06:50:29.709 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:50:29.709 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.709 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:50:29.709 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.566624+0000", 2024-02-05T06:50:29.709 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:50:29.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:50:29.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.566624+0000", 2024-02-05T06:50:29.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:50:29.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:50:29.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.566625+0000", 2024-02-05T06:50:29.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7800000000000001e-07 2024-02-05T06:50:29.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.710 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.710 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:50:29.710 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.566632+0000", 2024-02-05T06:50:29.710 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3520000000000001e-06 2024-02-05T06:50:29.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:50:29.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.566638+0000", 2024-02-05T06:50:29.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9889999999999997e-06 2024-02-05T06:50:29.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.711 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.711 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:50:29.711 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.566644+0000", 2024-02-05T06:50:29.711 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0379999999999999e-06 2024-02-05T06:50:29.711 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:50:29.711 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:50:29.711 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:50:29.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.712 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(client.45587.0:205652 10.17 e407/391)", 2024-02-05T06:50:29.712 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:50:29.570511+0000", 2024-02-05T06:50:29.712 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.00059826300000000003, 2024-02-05T06:50:29.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0010822480000000001, 2024-02-05T06:50:29.712 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:50:29.712 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "queued for pg", 2024-02-05T06:50:29.712 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:50:29.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.712 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:50:29.712 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.570511+0000", 2024-02-05T06:50:29.712 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:50:29.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.712 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:50:29.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.570511+0000", 2024-02-05T06:50:29.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:50:29.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:50:29.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.570512+0000", 2024-02-05T06:50:29.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.488e-06 2024-02-05T06:50:29.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.713 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.713 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:50:29.713 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.570518+0000", 2024-02-05T06:50:29.713 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8320000000000002e-06 2024-02-05T06:50:29.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:50:29.714 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.570532+0000", 2024-02-05T06:50:29.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3719e-05 2024-02-05T06:50:29.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.714 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:50:29.714 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.570540+0000", 2024-02-05T06:50:29.714 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4100000000000008e-06 2024-02-05T06:50:29.714 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:50:29.714 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:50:29.714 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:50:29.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.714 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.715 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.45587.0:205662 10.18 e407/375)", 2024-02-05T06:50:29.715 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:50:29.571289+0000", 2024-02-05T06:50:29.715 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 4294967295.9998202, 2024-02-05T06:50:29.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00035039499999999998, 2024-02-05T06:50:29.715 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:50:29.715 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "queued for pg", 2024-02-05T06:50:29.715 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:50:29.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:50:29.715 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.571289+0000", 2024-02-05T06:50:29.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:50:29.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.715 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:50:29.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.571289+0000", 2024-02-05T06:50:29.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:50:29.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.716 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:50:29.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.571290+0000", 2024-02-05T06:50:29.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1710000000000001e-06 2024-02-05T06:50:29.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.716 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:50:29.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.571304+0000", 2024-02-05T06:50:29.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4372000000000001e-05 2024-02-05T06:50:29.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:50:29.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.571317+0000", 2024-02-05T06:50:29.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2850000000000001e-05 2024-02-05T06:50:29.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:50:29.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:50:29.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:50:29.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:29.571324+0000", 2024-02-05T06:50:29.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9870000000000002e-06 2024-02-05T06:50:29.717 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:50:29.717 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:50:29.718 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:50:29.718 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:50:29.718 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:50:29.718 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 8 2024-02-05T06:50:29.718 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:50:29.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:29.719 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:29.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:29.821 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:29.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:29.860 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:29.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:29.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:29 smithi028 ceph-mon[37984]: 1.4 scrub starts 2024-02-05T06:50:29.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:29 smithi028 ceph-mon[37984]: 1.4 scrub ok 2024-02-05T06:50:29.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:29 smithi028 ceph-mon[37984]: 10.c scrub ok 2024-02-05T06:50:29.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:29 smithi028 ceph-mon[37984]: pgmap v817: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 6093/119007 objects misplaced (5.120%) 2024-02-05T06:50:29.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:29 smithi028 ceph-mon[37984]: 5.d scrub starts 2024-02-05T06:50:29.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:29 smithi028 ceph-mon[37984]: 5.d scrub ok 2024-02-05T06:50:29.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:29 smithi028 ceph-mon[37984]: 10.e scrub starts 2024-02-05T06:50:29.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:29 smithi028 ceph-mon[29442]: 1.4 scrub starts 2024-02-05T06:50:29.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:29 smithi028 ceph-mon[29442]: 1.4 scrub ok 2024-02-05T06:50:29.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:29 smithi028 ceph-mon[29442]: 10.c scrub ok 2024-02-05T06:50:29.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:29 smithi028 ceph-mon[29442]: pgmap v817: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 6093/119007 objects misplaced (5.120%) 2024-02-05T06:50:29.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:29 smithi028 ceph-mon[29442]: 5.d scrub starts 2024-02-05T06:50:29.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:29 smithi028 ceph-mon[29442]: 5.d scrub ok 2024-02-05T06:50:29.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:29 smithi028 ceph-mon[29442]: 10.e scrub starts 2024-02-05T06:50:29.961 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:29.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:29 smithi167 ceph-mon[32774]: 1.4 scrub starts 2024-02-05T06:50:29.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:29 smithi167 ceph-mon[32774]: 1.4 scrub ok 2024-02-05T06:50:29.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:29 smithi167 ceph-mon[32774]: 10.c scrub ok 2024-02-05T06:50:29.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:29 smithi167 ceph-mon[32774]: pgmap v817: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 6093/119007 objects misplaced (5.120%) 2024-02-05T06:50:29.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:29 smithi167 ceph-mon[32774]: 5.d scrub starts 2024-02-05T06:50:29.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:29 smithi167 ceph-mon[32774]: 5.d scrub ok 2024-02-05T06:50:29.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:29 smithi167 ceph-mon[32774]: 10.e scrub starts 2024-02-05T06:50:29.996 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:29.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:29.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:30.029 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:50:30.098 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:30.132 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:30.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:30.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:30.234 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:30.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:30.282 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:30.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:30.383 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:30.780 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:30.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:30.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:30.882 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:30.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:30.919 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:30.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:31.019 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:31.053 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:31.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:31.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:31.155 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:31.195 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:31.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:31.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:31.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:31.337 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:31.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:31.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:31.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:31.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:31.474 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:31.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:31.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:31 smithi028 ceph-mon[29442]: 5.16 scrub starts 2024-02-05T06:50:31.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:31 smithi028 ceph-mon[29442]: 5.16 scrub ok 2024-02-05T06:50:31.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:31 smithi028 ceph-mon[29442]: 5.b scrub starts 2024-02-05T06:50:31.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:31 smithi028 ceph-mon[29442]: 5.b scrub ok 2024-02-05T06:50:31.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:31 smithi028 ceph-mon[29442]: 10.e scrub ok 2024-02-05T06:50:31.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:31 smithi028 ceph-mon[29442]: osdmap e408: 8 total, 8 up, 8 in 2024-02-05T06:50:31.505 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:31 smithi028 ceph-mon[29442]: pgmap v819: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 6093/119007 objects misplaced (5.120%) 2024-02-05T06:50:31.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:31 smithi028 ceph-mon[37984]: 5.16 scrub starts 2024-02-05T06:50:31.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:31 smithi028 ceph-mon[37984]: 5.16 scrub ok 2024-02-05T06:50:31.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:31 smithi028 ceph-mon[37984]: 5.b scrub starts 2024-02-05T06:50:31.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:31 smithi028 ceph-mon[37984]: 5.b scrub ok 2024-02-05T06:50:31.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:31 smithi028 ceph-mon[37984]: 10.e scrub ok 2024-02-05T06:50:31.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:31 smithi028 ceph-mon[37984]: osdmap e408: 8 total, 8 up, 8 in 2024-02-05T06:50:31.506 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:31 smithi028 ceph-mon[37984]: pgmap v819: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 6093/119007 objects misplaced (5.120%) 2024-02-05T06:50:31.549 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-05T06:50:31.575 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:31.611 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:31.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:31.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:31.713 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:31.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:31 smithi167 ceph-mon[32774]: 5.16 scrub starts 2024-02-05T06:50:31.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:31 smithi167 ceph-mon[32774]: 5.16 scrub ok 2024-02-05T06:50:31.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:31 smithi167 ceph-mon[32774]: 5.b scrub starts 2024-02-05T06:50:31.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:31 smithi167 ceph-mon[32774]: 5.b scrub ok 2024-02-05T06:50:31.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:31 smithi167 ceph-mon[32774]: 10.e scrub ok 2024-02-05T06:50:31.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:31 smithi167 ceph-mon[32774]: osdmap e408: 8 total, 8 up, 8 in 2024-02-05T06:50:31.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:31 smithi167 ceph-mon[32774]: pgmap v819: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 6093/119007 objects misplaced (5.120%) 2024-02-05T06:50:31.751 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:31.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:31.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:31.853 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:31.887 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:31.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:31.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:31.989 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:32.042 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:32.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:32.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:32.144 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:32.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:32.180 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:32.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:32.281 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:32.324 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:32.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:32.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:32.426 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:32.462 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:32.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:32.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:32.564 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:32.596 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:32.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:32.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:32.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:32 smithi028 ceph-mon[29442]: 3.1c scrub starts 2024-02-05T06:50:32.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:32 smithi028 ceph-mon[29442]: 3.1c scrub ok 2024-02-05T06:50:32.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:32 smithi028 ceph-mon[29442]: 2.1 scrub starts 2024-02-05T06:50:32.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:32 smithi028 ceph-mon[29442]: 2.1 scrub ok 2024-02-05T06:50:32.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:32 smithi028 ceph-mon[29442]: 5.1b scrub starts 2024-02-05T06:50:32.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:32 smithi028 ceph-mon[29442]: 5.1b scrub ok 2024-02-05T06:50:32.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:32 smithi028 ceph-mon[29442]: 10.b scrub starts 2024-02-05T06:50:32.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:32 smithi028 ceph-mon[29442]: 3.d scrub starts 2024-02-05T06:50:32.696 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:32 smithi028 ceph-mon[29442]: 3.d scrub ok 2024-02-05T06:50:32.697 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:50:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:50:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:50:32.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:32 smithi028 ceph-mon[37984]: 3.1c scrub starts 2024-02-05T06:50:32.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:32 smithi028 ceph-mon[37984]: 3.1c scrub ok 2024-02-05T06:50:32.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:32 smithi028 ceph-mon[37984]: 2.1 scrub starts 2024-02-05T06:50:32.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:32 smithi028 ceph-mon[37984]: 2.1 scrub ok 2024-02-05T06:50:32.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:32 smithi028 ceph-mon[37984]: 5.1b scrub starts 2024-02-05T06:50:32.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:32 smithi028 ceph-mon[37984]: 5.1b scrub ok 2024-02-05T06:50:32.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:32 smithi028 ceph-mon[37984]: 10.b scrub starts 2024-02-05T06:50:32.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:32 smithi028 ceph-mon[37984]: 3.d scrub starts 2024-02-05T06:50:32.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:32 smithi028 ceph-mon[37984]: 3.d scrub ok 2024-02-05T06:50:32.698 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:32.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:32 smithi167 ceph-mon[32774]: 3.1c scrub starts 2024-02-05T06:50:32.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:32 smithi167 ceph-mon[32774]: 3.1c scrub ok 2024-02-05T06:50:32.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:32 smithi167 ceph-mon[32774]: 2.1 scrub starts 2024-02-05T06:50:32.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:32 smithi167 ceph-mon[32774]: 2.1 scrub ok 2024-02-05T06:50:32.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:32 smithi167 ceph-mon[32774]: 5.1b scrub starts 2024-02-05T06:50:32.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:32 smithi167 ceph-mon[32774]: 5.1b scrub ok 2024-02-05T06:50:32.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:32 smithi167 ceph-mon[32774]: 10.b scrub starts 2024-02-05T06:50:32.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:32 smithi167 ceph-mon[32774]: 3.d scrub starts 2024-02-05T06:50:32.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:32 smithi167 ceph-mon[32774]: 3.d scrub ok 2024-02-05T06:50:32.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:32.740 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:32.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:32.838 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:50:32.838 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:50:32.842 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:32.901 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:32.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:32.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:33.003 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:33.047 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:33.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:33.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:33.149 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:33.177 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:33.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:33.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:33.240 INFO:teuthology.orchestra.run.smithi028.stderr:no valid command found; 10 closest matches: 2024-02-05T06:50:33.240 INFO:teuthology.orchestra.run.smithi028.stderr:0 2024-02-05T06:50:33.240 INFO:teuthology.orchestra.run.smithi028.stderr:1 2024-02-05T06:50:33.240 INFO:teuthology.orchestra.run.smithi028.stderr:2 2024-02-05T06:50:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:abort 2024-02-05T06:50:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:assert 2024-02-05T06:50:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:bench [] [] [] [] 2024-02-05T06:50:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs debug_inject_read_zeros 2024-02-05T06:50:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs files list 2024-02-05T06:50:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs stats 2024-02-05T06:50:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:bluestore allocator dump block 2024-02-05T06:50:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: invalid command 2024-02-05T06:50:33.278 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:33.319 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:33.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:33.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:33.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:33.459 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:33.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:33.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:33.561 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:33.600 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:33.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:33.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:33.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:33 smithi028 ceph-mon[29442]: 10.b scrub ok 2024-02-05T06:50:33.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:33 smithi028 ceph-mon[29442]: 3.2 scrub starts 2024-02-05T06:50:33.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:33 smithi028 ceph-mon[29442]: 3.2 scrub ok 2024-02-05T06:50:33.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:33 smithi028 ceph-mon[29442]: 6.13 scrub starts 2024-02-05T06:50:33.628 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:33 smithi028 ceph-mon[29442]: pgmap v820: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 2.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 5163/102003 objects misplaced (5.062%) 2024-02-05T06:50:33.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:33 smithi028 ceph-mon[29442]: 6.3 scrub starts 2024-02-05T06:50:33.629 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:33 smithi028 ceph-mon[29442]: 6.3 scrub ok 2024-02-05T06:50:33.629 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:33 smithi028 ceph-mon[37984]: 10.b scrub ok 2024-02-05T06:50:33.629 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:33 smithi028 ceph-mon[37984]: 3.2 scrub starts 2024-02-05T06:50:33.629 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:33 smithi028 ceph-mon[37984]: 3.2 scrub ok 2024-02-05T06:50:33.629 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:33 smithi028 ceph-mon[37984]: 6.13 scrub starts 2024-02-05T06:50:33.629 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:33 smithi028 ceph-mon[37984]: pgmap v820: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 2.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 5163/102003 objects misplaced (5.062%) 2024-02-05T06:50:33.629 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:33 smithi028 ceph-mon[37984]: 6.3 scrub starts 2024-02-05T06:50:33.629 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:33 smithi028 ceph-mon[37984]: 6.3 scrub ok 2024-02-05T06:50:33.702 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:33.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:33 smithi167 ceph-mon[32774]: 10.b scrub ok 2024-02-05T06:50:33.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:33 smithi167 ceph-mon[32774]: 3.2 scrub starts 2024-02-05T06:50:33.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:33 smithi167 ceph-mon[32774]: 3.2 scrub ok 2024-02-05T06:50:33.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:33 smithi167 ceph-mon[32774]: 6.13 scrub starts 2024-02-05T06:50:33.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:33 smithi167 ceph-mon[32774]: pgmap v820: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 2.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 5163/102003 objects misplaced (5.062%) 2024-02-05T06:50:33.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:33 smithi167 ceph-mon[32774]: 6.3 scrub starts 2024-02-05T06:50:33.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:33 smithi167 ceph-mon[32774]: 6.3 scrub ok 2024-02-05T06:50:33.737 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:50:33 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:50:33.738 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:33.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:33.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:33.840 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:33.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:33.878 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:33.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:33.979 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:34.024 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:34.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:34.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:34.121 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-05T06:50:34.121 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-05T06:50:34.122 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph tell osd.7 'trim stale osdmaps' 2024-02-05T06:50:34.126 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:34.210 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:34.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:34.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:34.312 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:34.345 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:34.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:34.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:34.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:34 smithi028 ceph-mon[29442]: 6.13 scrub ok 2024-02-05T06:50:34.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:34 smithi028 ceph-mon[29442]: 3.c scrub starts 2024-02-05T06:50:34.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:34 smithi028 ceph-mon[29442]: 3.c scrub ok 2024-02-05T06:50:34.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:34 smithi028 ceph-mon[29442]: 4.12 scrub starts 2024-02-05T06:50:34.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:34 smithi028 ceph-mon[29442]: 4.12 scrub ok 2024-02-05T06:50:34.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:34 smithi028 ceph-mon[29442]: 6.2 scrub starts 2024-02-05T06:50:34.438 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:34 smithi028 ceph-mon[29442]: 6.2 scrub ok 2024-02-05T06:50:34.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:34 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:50:34.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:34 smithi028 ceph-mon[29442]: 6.15 scrub starts 2024-02-05T06:50:34.439 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:34 smithi028 ceph-mon[29442]: 6.15 scrub ok 2024-02-05T06:50:34.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:34 smithi028 ceph-mon[37984]: 6.13 scrub ok 2024-02-05T06:50:34.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:34 smithi028 ceph-mon[37984]: 3.c scrub starts 2024-02-05T06:50:34.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:34 smithi028 ceph-mon[37984]: 3.c scrub ok 2024-02-05T06:50:34.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:34 smithi028 ceph-mon[37984]: 4.12 scrub starts 2024-02-05T06:50:34.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:34 smithi028 ceph-mon[37984]: 4.12 scrub ok 2024-02-05T06:50:34.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:34 smithi028 ceph-mon[37984]: 6.2 scrub starts 2024-02-05T06:50:34.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:34 smithi028 ceph-mon[37984]: 6.2 scrub ok 2024-02-05T06:50:34.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:34 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:50:34.439 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:34 smithi028 ceph-mon[37984]: 6.15 scrub starts 2024-02-05T06:50:34.440 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:34 smithi028 ceph-mon[37984]: 6.15 scrub ok 2024-02-05T06:50:34.447 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:34.480 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:34.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:34.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:34.581 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:34.627 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:34.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:34.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:34.729 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:34.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:34 smithi167 ceph-mon[32774]: 6.13 scrub ok 2024-02-05T06:50:34.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:34 smithi167 ceph-mon[32774]: 3.c scrub starts 2024-02-05T06:50:34.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:34 smithi167 ceph-mon[32774]: 3.c scrub ok 2024-02-05T06:50:34.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:34 smithi167 ceph-mon[32774]: 4.12 scrub starts 2024-02-05T06:50:34.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:34 smithi167 ceph-mon[32774]: 4.12 scrub ok 2024-02-05T06:50:34.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:34 smithi167 ceph-mon[32774]: 6.2 scrub starts 2024-02-05T06:50:34.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:34 smithi167 ceph-mon[32774]: 6.2 scrub ok 2024-02-05T06:50:34.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:34 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:50:34.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:34 smithi167 ceph-mon[32774]: 6.15 scrub starts 2024-02-05T06:50:34.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:34 smithi167 ceph-mon[32774]: 6.15 scrub ok 2024-02-05T06:50:34.761 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:34.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:34.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:34.863 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:34.895 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:34.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:34.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:34.997 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:35.031 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:35.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:35.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:35.133 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:35.168 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:35.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:35.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:35.270 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:35.304 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:35.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:35.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:35.406 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:35.431 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:35.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:35.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:35.532 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:35.571 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:35.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:35.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:35.672 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:35.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:35 smithi028 ceph-mon[37984]: 6.a scrub starts 2024-02-05T06:50:35.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:35 smithi028 ceph-mon[37984]: 6.a scrub ok 2024-02-05T06:50:35.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:35 smithi028 ceph-mon[37984]: pgmap v821: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 5163/100872 objects misplaced (5.118%) 2024-02-05T06:50:35.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:35 smithi028 ceph-mon[37984]: 2.0 scrub starts 2024-02-05T06:50:35.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:35 smithi028 ceph-mon[37984]: 2.0 scrub ok 2024-02-05T06:50:35.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:35 smithi028 ceph-mon[29442]: 6.a scrub starts 2024-02-05T06:50:35.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:35 smithi028 ceph-mon[29442]: 6.a scrub ok 2024-02-05T06:50:35.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:35 smithi028 ceph-mon[29442]: pgmap v821: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 5163/100872 objects misplaced (5.118%) 2024-02-05T06:50:35.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:35 smithi028 ceph-mon[29442]: 2.0 scrub starts 2024-02-05T06:50:35.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:35 smithi028 ceph-mon[29442]: 2.0 scrub ok 2024-02-05T06:50:35.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:35.707 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:35.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:35.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:35 smithi167 ceph-mon[32774]: 6.a scrub starts 2024-02-05T06:50:35.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:35 smithi167 ceph-mon[32774]: 6.a scrub ok 2024-02-05T06:50:35.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:35 smithi167 ceph-mon[32774]: pgmap v821: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 5163/100872 objects misplaced (5.118%) 2024-02-05T06:50:35.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:35 smithi167 ceph-mon[32774]: 2.0 scrub starts 2024-02-05T06:50:35.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:35 smithi167 ceph-mon[32774]: 2.0 scrub ok 2024-02-05T06:50:35.809 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:35.833 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:35.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:35.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:35.934 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:35.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:35.977 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:35.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:36.078 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:36.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:36.103 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:36.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:36.204 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:36.240 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:36.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:36.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:36.343 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:36.513 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:36.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:36.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:36.615 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:36.652 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:36.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:36.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:36.670 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:36 smithi028 ceph-mon[37984]: 5.3 scrub starts 2024-02-05T06:50:36.670 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:36 smithi028 ceph-mon[37984]: 5.3 scrub ok 2024-02-05T06:50:36.670 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:36 smithi028 ceph-mon[37984]: 3.4 scrub starts 2024-02-05T06:50:36.671 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:36 smithi028 ceph-mon[37984]: 3.4 scrub ok 2024-02-05T06:50:36.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:36 smithi028 ceph-mon[29442]: 5.3 scrub starts 2024-02-05T06:50:36.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:36 smithi028 ceph-mon[29442]: 5.3 scrub ok 2024-02-05T06:50:36.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:36 smithi028 ceph-mon[29442]: 3.4 scrub starts 2024-02-05T06:50:36.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:36 smithi028 ceph-mon[29442]: 3.4 scrub ok 2024-02-05T06:50:36.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:36 smithi167 ceph-mon[32774]: 5.3 scrub starts 2024-02-05T06:50:36.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:36 smithi167 ceph-mon[32774]: 5.3 scrub ok 2024-02-05T06:50:36.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:36 smithi167 ceph-mon[32774]: 3.4 scrub starts 2024-02-05T06:50:36.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:36 smithi167 ceph-mon[32774]: 3.4 scrub ok 2024-02-05T06:50:36.753 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:36.796 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:36.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:36.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:36.898 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:36.936 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:36.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:36.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:37.038 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:37.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:37.091 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:37.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:37.191 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:37.232 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:37.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:37.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:37.333 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:37.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:37.370 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:37.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:37.471 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:37.513 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:37.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:37.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:37.615 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:37.649 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:37.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:37.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:37.681 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:37 smithi028 ceph-mon[37984]: 5.c scrub starts 2024-02-05T06:50:37.681 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:37 smithi028 ceph-mon[37984]: 5.c scrub ok 2024-02-05T06:50:37.681 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:37 smithi028 ceph-mon[37984]: pgmap v822: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 580 B/s rd, 0 B/s wr, 54 op/s; 5163/99702 objects misplaced (5.178%) 2024-02-05T06:50:37.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:37 smithi028 ceph-mon[29442]: 5.c scrub starts 2024-02-05T06:50:37.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:37 smithi028 ceph-mon[29442]: 5.c scrub ok 2024-02-05T06:50:37.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:37 smithi028 ceph-mon[29442]: pgmap v822: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 580 B/s rd, 0 B/s wr, 54 op/s; 5163/99702 objects misplaced (5.178%) 2024-02-05T06:50:37.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:37 smithi167 ceph-mon[32774]: 5.c scrub starts 2024-02-05T06:50:37.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:37 smithi167 ceph-mon[32774]: 5.c scrub ok 2024-02-05T06:50:37.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:37 smithi167 ceph-mon[32774]: pgmap v822: 168 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 165 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 580 B/s rd, 0 B/s wr, 54 op/s; 5163/99702 objects misplaced (5.178%) 2024-02-05T06:50:37.751 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:37.789 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:37.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:37.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:37.892 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:37.924 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:37.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:37.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:38.026 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:38.063 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:38.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:38.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:38.165 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:38.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:38.212 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:38.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:38.313 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:38.346 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:38.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:38.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:38.448 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:38.482 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:38.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:38.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:38.584 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:38.618 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:38.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:38.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:38.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:38 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:50:38.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:38 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:50:38.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:38 smithi028 ceph-mon[37984]: 3.1 scrub starts 2024-02-05T06:50:38.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:38 smithi028 ceph-mon[37984]: 3.1 scrub ok 2024-02-05T06:50:38.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:38 smithi028 ceph-mon[37984]: 5.0 scrub starts 2024-02-05T06:50:38.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:38 smithi028 ceph-mon[37984]: osdmap e409: 8 total, 8 up, 8 in 2024-02-05T06:50:38.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:38 smithi028 ceph-mon[37984]: 5.0 scrub ok 2024-02-05T06:50:38.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:38 smithi028 ceph-mon[37984]: 3.14 scrub starts 2024-02-05T06:50:38.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:38 smithi028 ceph-mon[37984]: 3.14 scrub ok 2024-02-05T06:50:38.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:38 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:50:38.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:38 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:50:38.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:38 smithi028 ceph-mon[29442]: 3.1 scrub starts 2024-02-05T06:50:38.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:38 smithi028 ceph-mon[29442]: 3.1 scrub ok 2024-02-05T06:50:38.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:38 smithi028 ceph-mon[29442]: 5.0 scrub starts 2024-02-05T06:50:38.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:38 smithi028 ceph-mon[29442]: osdmap e409: 8 total, 8 up, 8 in 2024-02-05T06:50:38.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:38 smithi028 ceph-mon[29442]: 5.0 scrub ok 2024-02-05T06:50:38.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:38 smithi028 ceph-mon[29442]: 3.14 scrub starts 2024-02-05T06:50:38.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:38 smithi028 ceph-mon[29442]: 3.14 scrub ok 2024-02-05T06:50:38.720 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:38.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:38 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:50:38.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:38 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:50:38.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:38 smithi167 ceph-mon[32774]: 3.1 scrub starts 2024-02-05T06:50:38.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:38 smithi167 ceph-mon[32774]: 3.1 scrub ok 2024-02-05T06:50:38.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:38 smithi167 ceph-mon[32774]: 5.0 scrub starts 2024-02-05T06:50:38.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:38 smithi167 ceph-mon[32774]: osdmap e409: 8 total, 8 up, 8 in 2024-02-05T06:50:38.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:38 smithi167 ceph-mon[32774]: 5.0 scrub ok 2024-02-05T06:50:38.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:38 smithi167 ceph-mon[32774]: 3.14 scrub starts 2024-02-05T06:50:38.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:38 smithi167 ceph-mon[32774]: 3.14 scrub ok 2024-02-05T06:50:38.755 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:38.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:38.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:38.857 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:38.889 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:38.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:38.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:38.991 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:39.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:39.032 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:39.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:39.133 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:39.165 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:39.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:39.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:39.267 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:39.300 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:39.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:39.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:39.402 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:39.439 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:39.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:39.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:39.541 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:39.574 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:39 smithi028 ceph-mon[37984]: pgmap v824: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 1.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 4222/82473 objects misplaced (5.119%) 2024-02-05T06:50:39.574 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:39 smithi028 ceph-mon[29442]: pgmap v824: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 1.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 4222/82473 objects misplaced (5.119%) 2024-02-05T06:50:39.576 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:39.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:39.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:39.678 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:39.722 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:39.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:39.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:39.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:39 smithi167 ceph-mon[32774]: pgmap v824: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 1.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.47k op/s; 4222/82473 objects misplaced (5.119%) 2024-02-05T06:50:39.796 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:50:39.825 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:39.838 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:50:39.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:39.869 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:39.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:39.899 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:50:39.924 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:50:39.951 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:50:39.970 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:39.995 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:50:39.999 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:40.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:40.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:40.031 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:50:40.031 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:50:40.038 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:50:40.079 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:50:40.102 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:40.126 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:50:40.132 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:40.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:40.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:40.135 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:50:40.135 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:50:40.135 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:50:40.135 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:50:40.135 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:50:40.135 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:50:40.135 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:50:40.135 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:50:40.234 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:40.264 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-02-05T06:50:40.282 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:40.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:40.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:40.384 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:40.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:40.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:40.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:40 smithi028 ceph-mon[29442]: 5.e scrub starts 2024-02-05T06:50:40.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:40 smithi028 ceph-mon[29442]: 5.e scrub ok 2024-02-05T06:50:40.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:40 smithi028 ceph-mon[29442]: 6.18 scrub starts 2024-02-05T06:50:40.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:40 smithi028 ceph-mon[29442]: 6.18 scrub ok 2024-02-05T06:50:40.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:40 smithi028 ceph-mon[29442]: 4.7 scrub starts 2024-02-05T06:50:40.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:40 smithi028 ceph-mon[29442]: 4.7 scrub ok 2024-02-05T06:50:40.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:40 smithi028 ceph-mon[29442]: 10.f scrub starts 2024-02-05T06:50:40.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:40 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1557610310' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:50:40.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:40 smithi028 ceph-mon[29442]: from='client.25783 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:50:40.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:40 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1064842644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:50:40.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:40 smithi028 ceph-mon[29442]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:50:40.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:40 smithi028 ceph-mon[29442]: from='client.25783 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:50:40.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:40 smithi028 ceph-mon[29442]: osdmap e410: 8 total, 8 up, 8 in 2024-02-05T06:50:40.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:40 smithi028 ceph-mon[37984]: 5.e scrub starts 2024-02-05T06:50:40.453 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:40 smithi028 ceph-mon[37984]: 5.e scrub ok 2024-02-05T06:50:40.453 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:40 smithi028 ceph-mon[37984]: 6.18 scrub starts 2024-02-05T06:50:40.453 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:40 smithi028 ceph-mon[37984]: 6.18 scrub ok 2024-02-05T06:50:40.453 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:40 smithi028 ceph-mon[37984]: 4.7 scrub starts 2024-02-05T06:50:40.453 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:40 smithi028 ceph-mon[37984]: 4.7 scrub ok 2024-02-05T06:50:40.453 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:40 smithi028 ceph-mon[37984]: 10.f scrub starts 2024-02-05T06:50:40.453 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:40 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1557610310' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:50:40.453 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:40 smithi028 ceph-mon[37984]: from='client.25783 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:50:40.453 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:40 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1064842644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:50:40.453 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:40 smithi028 ceph-mon[37984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:50:40.453 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:40 smithi028 ceph-mon[37984]: from='client.25783 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:50:40.453 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:40 smithi028 ceph-mon[37984]: osdmap e410: 8 total, 8 up, 8 in 2024-02-05T06:50:40.522 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:40.558 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:40.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:40.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:40.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:40.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:40.696 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:40.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:40.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:40 smithi167 ceph-mon[32774]: 5.e scrub starts 2024-02-05T06:50:40.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:40 smithi167 ceph-mon[32774]: 5.e scrub ok 2024-02-05T06:50:40.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:40 smithi167 ceph-mon[32774]: 6.18 scrub starts 2024-02-05T06:50:40.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:40 smithi167 ceph-mon[32774]: 6.18 scrub ok 2024-02-05T06:50:40.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:40 smithi167 ceph-mon[32774]: 4.7 scrub starts 2024-02-05T06:50:40.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:40 smithi167 ceph-mon[32774]: 4.7 scrub ok 2024-02-05T06:50:40.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:40 smithi167 ceph-mon[32774]: 10.f scrub starts 2024-02-05T06:50:40.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:40 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1557610310' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:50:40.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:40 smithi167 ceph-mon[32774]: from='client.25783 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:50:40.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:40 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1064842644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:50:40.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:40 smithi167 ceph-mon[32774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:50:40.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:40 smithi167 ceph-mon[32774]: from='client.25783 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:50:40.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:40 smithi167 ceph-mon[32774]: osdmap e410: 8 total, 8 up, 8 in 2024-02-05T06:50:40.797 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:40.833 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:40.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:40.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:40.935 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:40.971 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:40.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:40.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:40.975 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-05T06:50:41.072 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:41.108 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:41.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:41.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:41.210 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:41.246 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:41.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:41.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:41.348 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:41.401 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:41.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:41.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:41.503 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:41.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:41.539 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:41.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:41.637 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:41 smithi167 ceph-mon[32774]: 6.1a scrub starts 2024-02-05T06:50:41.640 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:41 smithi167 ceph-mon[32774]: 6.1a scrub ok 2024-02-05T06:50:41.640 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:41 smithi167 ceph-mon[32774]: 10.f scrub ok 2024-02-05T06:50:41.640 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:41 smithi167 ceph-mon[32774]: 4.6 scrub starts 2024-02-05T06:50:41.640 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:41 smithi167 ceph-mon[32774]: 4.6 scrub ok 2024-02-05T06:50:41.640 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:41 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:50:41.640 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:41 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:50:41.640 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:41 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:50:41.640 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:41 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:50:41.640 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:41 smithi167 ceph-mon[32774]: pgmap v826: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 1.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 814 op/s; 4222/82473 objects misplaced (5.119%) 2024-02-05T06:50:41.641 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:41.668 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:41.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:41.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:41.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:41 smithi028 ceph-mon[29442]: 6.1a scrub starts 2024-02-05T06:50:41.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:41 smithi028 ceph-mon[29442]: 6.1a scrub ok 2024-02-05T06:50:41.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:41 smithi028 ceph-mon[29442]: 10.f scrub ok 2024-02-05T06:50:41.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:41 smithi028 ceph-mon[29442]: 4.6 scrub starts 2024-02-05T06:50:41.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:41 smithi028 ceph-mon[29442]: 4.6 scrub ok 2024-02-05T06:50:41.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:41 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:50:41.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:41 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:50:41.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:41 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:50:41.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:41 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:50:41.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:41 smithi028 ceph-mon[29442]: pgmap v826: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 1.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 814 op/s; 4222/82473 objects misplaced (5.119%) 2024-02-05T06:50:41.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:41 smithi028 ceph-mon[37984]: 6.1a scrub starts 2024-02-05T06:50:41.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:41 smithi028 ceph-mon[37984]: 6.1a scrub ok 2024-02-05T06:50:41.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:41 smithi028 ceph-mon[37984]: 10.f scrub ok 2024-02-05T06:50:41.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:41 smithi028 ceph-mon[37984]: 4.6 scrub starts 2024-02-05T06:50:41.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:41 smithi028 ceph-mon[37984]: 4.6 scrub ok 2024-02-05T06:50:41.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:41 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:50:41.703 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:41 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:50:41.703 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:41 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:50:41.703 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:41 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:50:41.703 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:41 smithi028 ceph-mon[37984]: pgmap v826: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 1.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 814 op/s; 4222/82473 objects misplaced (5.119%) 2024-02-05T06:50:41.770 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:41.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:41.802 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:41.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:41.903 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:41.941 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:41.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:41.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:42.043 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:42.082 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:42.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:42.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:42.183 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:42.227 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:42.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:42.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:42.329 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:42.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:42.371 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:42.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:42.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:42 smithi028 ceph-mon[29442]: 1.5 scrub starts 2024-02-05T06:50:42.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:42 smithi028 ceph-mon[29442]: 1.5 scrub ok 2024-02-05T06:50:42.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:42 smithi028 ceph-mon[29442]: 4.3 scrub starts 2024-02-05T06:50:42.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:42 smithi028 ceph-mon[29442]: 4.3 scrub ok 2024-02-05T06:50:42.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:42 smithi028 ceph-mon[29442]: 5.1 scrub starts 2024-02-05T06:50:42.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:42 smithi028 ceph-mon[29442]: 5.1 scrub ok 2024-02-05T06:50:42.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:42 smithi028 ceph-mon[29442]: osdmap e411: 8 total, 8 up, 8 in 2024-02-05T06:50:42.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:42 smithi028 ceph-mon[29442]: 5.1a scrub starts 2024-02-05T06:50:42.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:42 smithi028 ceph-mon[29442]: 5.1a scrub ok 2024-02-05T06:50:42.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:42 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:50:42.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:42 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:50:42.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:42 smithi028 ceph-mon[37984]: 1.5 scrub starts 2024-02-05T06:50:42.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:42 smithi028 ceph-mon[37984]: 1.5 scrub ok 2024-02-05T06:50:42.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:42 smithi028 ceph-mon[37984]: 4.3 scrub starts 2024-02-05T06:50:42.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:42 smithi028 ceph-mon[37984]: 4.3 scrub ok 2024-02-05T06:50:42.453 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:42 smithi028 ceph-mon[37984]: 5.1 scrub starts 2024-02-05T06:50:42.453 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:42 smithi028 ceph-mon[37984]: 5.1 scrub ok 2024-02-05T06:50:42.453 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:42 smithi028 ceph-mon[37984]: osdmap e411: 8 total, 8 up, 8 in 2024-02-05T06:50:42.453 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:42 smithi028 ceph-mon[37984]: 5.1a scrub starts 2024-02-05T06:50:42.453 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:42 smithi028 ceph-mon[37984]: 5.1a scrub ok 2024-02-05T06:50:42.453 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:42 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:50:42.453 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:42 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:50:42.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:42.509 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:42.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:42.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:42.608 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:42 smithi167 ceph-mon[32774]: 1.5 scrub starts 2024-02-05T06:50:42.609 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:42 smithi167 ceph-mon[32774]: 1.5 scrub ok 2024-02-05T06:50:42.609 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:42 smithi167 ceph-mon[32774]: 4.3 scrub starts 2024-02-05T06:50:42.609 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:42 smithi167 ceph-mon[32774]: 4.3 scrub ok 2024-02-05T06:50:42.609 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:42 smithi167 ceph-mon[32774]: 5.1 scrub starts 2024-02-05T06:50:42.609 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:42 smithi167 ceph-mon[32774]: 5.1 scrub ok 2024-02-05T06:50:42.609 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:42 smithi167 ceph-mon[32774]: osdmap e411: 8 total, 8 up, 8 in 2024-02-05T06:50:42.609 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:42 smithi167 ceph-mon[32774]: 5.1a scrub starts 2024-02-05T06:50:42.609 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:42 smithi167 ceph-mon[32774]: 5.1a scrub ok 2024-02-05T06:50:42.609 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:42 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:50:42.609 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:42 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:50:42.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:42.643 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:42.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:42.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:42.745 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:42.782 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:42.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:42.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:42.884 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:42.920 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:42.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:42.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:42.951 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:50:42 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:50:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:50:43.021 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:43.057 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:43.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:43.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:43.159 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:43.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:43.197 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:43.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:43.298 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:43.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:43.335 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:43.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:43.436 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:43.480 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:43.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:43.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:43.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:50:43 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:50:43.486 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:43 smithi167 ceph-mon[32774]: 5.5 scrub starts 2024-02-05T06:50:43.486 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:43 smithi167 ceph-mon[32774]: 5.5 scrub ok 2024-02-05T06:50:43.486 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:43 smithi167 ceph-mon[32774]: 6.1d deep-scrub starts 2024-02-05T06:50:43.486 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:43 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:50:43.486 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:43 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:50:43.486 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:43 smithi167 ceph-mon[32774]: 6.1d deep-scrub ok 2024-02-05T06:50:43.486 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:43 smithi167 ceph-mon[32774]: osdmap e412: 8 total, 8 up, 8 in 2024-02-05T06:50:43.486 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:43 smithi167 ceph-mon[32774]: pgmap v829: 168 pgs: 1 active+clean+scrubbing+deep, 2 active+remapped+backfill_toofull, 165 active+clean; 1.3 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 978 B/s rd, 0 B/s wr, 1.19k op/s; 3274/65073 objects misplaced (5.031%) 2024-02-05T06:50:43.582 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:43.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:43 smithi028 ceph-mon[29442]: 5.5 scrub starts 2024-02-05T06:50:43.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:43 smithi028 ceph-mon[29442]: 5.5 scrub ok 2024-02-05T06:50:43.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:43 smithi028 ceph-mon[29442]: 6.1d deep-scrub starts 2024-02-05T06:50:43.611 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:43 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:50:43.612 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:43 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:50:43.612 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:43 smithi028 ceph-mon[29442]: 6.1d deep-scrub ok 2024-02-05T06:50:43.612 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:43 smithi028 ceph-mon[29442]: osdmap e412: 8 total, 8 up, 8 in 2024-02-05T06:50:43.612 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:43 smithi028 ceph-mon[29442]: pgmap v829: 168 pgs: 1 active+clean+scrubbing+deep, 2 active+remapped+backfill_toofull, 165 active+clean; 1.3 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 978 B/s rd, 0 B/s wr, 1.19k op/s; 3274/65073 objects misplaced (5.031%) 2024-02-05T06:50:43.612 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:43 smithi028 ceph-mon[37984]: 5.5 scrub starts 2024-02-05T06:50:43.612 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:43 smithi028 ceph-mon[37984]: 5.5 scrub ok 2024-02-05T06:50:43.612 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:43 smithi028 ceph-mon[37984]: 6.1d deep-scrub starts 2024-02-05T06:50:43.612 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:43 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:50:43.612 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:43 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:50:43.612 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:43 smithi028 ceph-mon[37984]: 6.1d deep-scrub ok 2024-02-05T06:50:43.612 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:43 smithi028 ceph-mon[37984]: osdmap e412: 8 total, 8 up, 8 in 2024-02-05T06:50:43.613 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:43 smithi028 ceph-mon[37984]: pgmap v829: 168 pgs: 1 active+clean+scrubbing+deep, 2 active+remapped+backfill_toofull, 165 active+clean; 1.3 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 978 B/s rd, 0 B/s wr, 1.19k op/s; 3274/65073 objects misplaced (5.031%) 2024-02-05T06:50:43.618 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:43.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:43.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:43.720 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:43.757 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:43.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:43.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:43.859 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:43.892 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:43.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:43.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:43.993 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:44.033 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:44.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:44.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:44.135 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:44.175 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:44.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:44.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:44.277 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:44.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:44.312 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:44.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:44.413 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:44.449 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:44.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:44.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:44.551 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:44.584 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:44.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:44.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:44.686 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:44.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:44 smithi028 ceph-mon[37984]: 3.1b scrub starts 2024-02-05T06:50:44.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:44 smithi028 ceph-mon[37984]: 3.1b scrub ok 2024-02-05T06:50:44.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:44 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:50:44.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:44 smithi028 ceph-mon[37984]: 1.9 scrub starts 2024-02-05T06:50:44.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:44 smithi028 ceph-mon[37984]: 1.9 scrub ok 2024-02-05T06:50:44.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:44 smithi028 ceph-mon[37984]: 4.9 scrub starts 2024-02-05T06:50:44.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:44 smithi028 ceph-mon[37984]: 4.9 scrub ok 2024-02-05T06:50:44.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:44 smithi028 ceph-mon[29442]: 3.1b scrub starts 2024-02-05T06:50:44.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:44 smithi028 ceph-mon[29442]: 3.1b scrub ok 2024-02-05T06:50:44.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:44 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:50:44.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:44 smithi028 ceph-mon[29442]: 1.9 scrub starts 2024-02-05T06:50:44.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:44 smithi028 ceph-mon[29442]: 1.9 scrub ok 2024-02-05T06:50:44.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:44 smithi028 ceph-mon[29442]: 4.9 scrub starts 2024-02-05T06:50:44.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:44 smithi028 ceph-mon[29442]: 4.9 scrub ok 2024-02-05T06:50:44.717 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:44.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:44.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:44.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:44 smithi167 ceph-mon[32774]: 3.1b scrub starts 2024-02-05T06:50:44.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:44 smithi167 ceph-mon[32774]: 3.1b scrub ok 2024-02-05T06:50:44.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:44 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:50:44.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:44 smithi167 ceph-mon[32774]: 1.9 scrub starts 2024-02-05T06:50:44.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:44 smithi167 ceph-mon[32774]: 1.9 scrub ok 2024-02-05T06:50:44.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:44 smithi167 ceph-mon[32774]: 4.9 scrub starts 2024-02-05T06:50:44.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:44 smithi167 ceph-mon[32774]: 4.9 scrub ok 2024-02-05T06:50:44.818 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:44.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:44.854 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:44.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:44.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:44.992 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:44.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:44.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:45.094 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:45.133 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:45.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:45.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:45.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:45.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:45.274 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:45.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:45.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:45.409 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:45.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:45.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:45.455 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:50:45.511 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:45.542 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:45.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:45.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:45.644 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:45.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:45.693 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:45.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:45.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:45 smithi028 ceph-mon[29442]: 2.2 scrub starts 2024-02-05T06:50:45.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:45 smithi028 ceph-mon[29442]: 2.2 scrub ok 2024-02-05T06:50:45.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:45 smithi028 ceph-mon[29442]: pgmap v830: 168 pgs: 1 active+clean+scrubbing+deep, 2 active+remapped+backfill_toofull, 165 active+clean; 1.3 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.03k op/s; 3274/63987 objects misplaced (5.117%) 2024-02-05T06:50:45.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:45 smithi028 ceph-mon[37984]: 2.2 scrub starts 2024-02-05T06:50:45.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:45 smithi028 ceph-mon[37984]: 2.2 scrub ok 2024-02-05T06:50:45.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:45 smithi028 ceph-mon[37984]: pgmap v830: 168 pgs: 1 active+clean+scrubbing+deep, 2 active+remapped+backfill_toofull, 165 active+clean; 1.3 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.03k op/s; 3274/63987 objects misplaced (5.117%) 2024-02-05T06:50:45.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:45 smithi167 ceph-mon[32774]: 2.2 scrub starts 2024-02-05T06:50:45.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:45 smithi167 ceph-mon[32774]: 2.2 scrub ok 2024-02-05T06:50:45.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:45 smithi167 ceph-mon[32774]: pgmap v830: 168 pgs: 1 active+clean+scrubbing+deep, 2 active+remapped+backfill_toofull, 165 active+clean; 1.3 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.03k op/s; 3274/63987 objects misplaced (5.117%) 2024-02-05T06:50:45.794 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:45.835 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:45.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:45.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:45.937 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:45.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:45.971 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:45.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:46.072 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:46.110 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:46.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:46.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:46.212 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:46.244 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:46.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:46.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:46.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T06:50:46.346 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:46.378 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:46.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:46.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:46.480 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:46.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:46.543 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:46.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:46.644 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:46.673 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:46.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:46.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:46.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:46 smithi028 ceph-mon[37984]: 4.d scrub starts 2024-02-05T06:50:46.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:46 smithi028 ceph-mon[37984]: 4.d scrub ok 2024-02-05T06:50:46.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:46 smithi028 ceph-mon[37984]: 4.8 scrub starts 2024-02-05T06:50:46.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:46 smithi028 ceph-mon[37984]: 4.8 scrub ok 2024-02-05T06:50:46.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:46 smithi028 ceph-mon[37984]: 3.3 scrub starts 2024-02-05T06:50:46.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:46 smithi028 ceph-mon[37984]: 3.3 scrub ok 2024-02-05T06:50:46.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:46 smithi028 ceph-mon[37984]: 5.9 scrub starts 2024-02-05T06:50:46.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:46 smithi028 ceph-mon[37984]: 5.9 scrub ok 2024-02-05T06:50:46.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:46 smithi028 ceph-mon[29442]: 4.d scrub starts 2024-02-05T06:50:46.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:46 smithi028 ceph-mon[29442]: 4.d scrub ok 2024-02-05T06:50:46.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:46 smithi028 ceph-mon[29442]: 4.8 scrub starts 2024-02-05T06:50:46.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:46 smithi028 ceph-mon[29442]: 4.8 scrub ok 2024-02-05T06:50:46.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:46 smithi028 ceph-mon[29442]: 3.3 scrub starts 2024-02-05T06:50:46.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:46 smithi028 ceph-mon[29442]: 3.3 scrub ok 2024-02-05T06:50:46.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:46 smithi028 ceph-mon[29442]: 5.9 scrub starts 2024-02-05T06:50:46.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:46 smithi028 ceph-mon[29442]: 5.9 scrub ok 2024-02-05T06:50:46.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:46 smithi167 ceph-mon[32774]: 4.d scrub starts 2024-02-05T06:50:46.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:46 smithi167 ceph-mon[32774]: 4.d scrub ok 2024-02-05T06:50:46.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:46 smithi167 ceph-mon[32774]: 4.8 scrub starts 2024-02-05T06:50:46.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:46 smithi167 ceph-mon[32774]: 4.8 scrub ok 2024-02-05T06:50:46.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:46 smithi167 ceph-mon[32774]: 3.3 scrub starts 2024-02-05T06:50:46.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:46 smithi167 ceph-mon[32774]: 3.3 scrub ok 2024-02-05T06:50:46.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:46 smithi167 ceph-mon[32774]: 5.9 scrub starts 2024-02-05T06:50:46.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:46 smithi167 ceph-mon[32774]: 5.9 scrub ok 2024-02-05T06:50:46.748 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:50:46.775 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:46.817 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:46.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:46.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:46.919 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:46.960 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:46.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:46.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:47.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:47.102 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:47.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:47.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:47.204 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:47.245 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:47.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:47.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:47.347 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:47.380 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:47.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:47.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:47.419 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:47 smithi028 ceph-mon[29442]: 6.14 deep-scrub starts 2024-02-05T06:50:47.420 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:47 smithi028 ceph-mon[29442]: 6.14 deep-scrub ok 2024-02-05T06:50:47.420 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:47 smithi028 ceph-mon[29442]: pgmap v831: 168 pgs: 1 active+clean+scrubbing+deep, 2 active+remapped+backfill_toofull, 165 active+clean; 1.3 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 814 B/s rd, 0 B/s wr, 1.04k op/s; 3274/62817 objects misplaced (5.212%) 2024-02-05T06:50:47.420 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:47 smithi028 ceph-mon[37984]: 6.14 deep-scrub starts 2024-02-05T06:50:47.420 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:47 smithi028 ceph-mon[37984]: 6.14 deep-scrub ok 2024-02-05T06:50:47.420 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:47 smithi028 ceph-mon[37984]: pgmap v831: 168 pgs: 1 active+clean+scrubbing+deep, 2 active+remapped+backfill_toofull, 165 active+clean; 1.3 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 814 B/s rd, 0 B/s wr, 1.04k op/s; 3274/62817 objects misplaced (5.212%) 2024-02-05T06:50:47.482 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:47.508 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:47.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:47.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:47.610 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:47.653 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:47.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:47.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:47.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:47 smithi167 ceph-mon[32774]: 6.14 deep-scrub starts 2024-02-05T06:50:47.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:47 smithi167 ceph-mon[32774]: 6.14 deep-scrub ok 2024-02-05T06:50:47.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:47 smithi167 ceph-mon[32774]: pgmap v831: 168 pgs: 1 active+clean+scrubbing+deep, 2 active+remapped+backfill_toofull, 165 active+clean; 1.3 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 814 B/s rd, 0 B/s wr, 1.04k op/s; 3274/62817 objects misplaced (5.212%) 2024-02-05T06:50:47.755 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:47.792 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:47.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:47.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:47.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:47.934 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:47.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:47.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:48.035 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:48.075 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:48.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:48.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:48.177 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:48.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:48.212 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:48.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:48.260 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:50:48.281 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:50:48.313 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:48.365 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:48.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:48.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:48.405 INFO:teuthology.orchestra.run.smithi028.stderr:no valid command found; 10 closest matches: 2024-02-05T06:50:48.405 INFO:teuthology.orchestra.run.smithi028.stderr:0 2024-02-05T06:50:48.405 INFO:teuthology.orchestra.run.smithi028.stderr:1 2024-02-05T06:50:48.405 INFO:teuthology.orchestra.run.smithi028.stderr:2 2024-02-05T06:50:48.405 INFO:teuthology.orchestra.run.smithi028.stderr:abort 2024-02-05T06:50:48.405 INFO:teuthology.orchestra.run.smithi028.stderr:assert 2024-02-05T06:50:48.405 INFO:teuthology.orchestra.run.smithi028.stderr:bench [] [] [] [] 2024-02-05T06:50:48.406 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs debug_inject_read_zeros 2024-02-05T06:50:48.406 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs files list 2024-02-05T06:50:48.406 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs stats 2024-02-05T06:50:48.406 INFO:teuthology.orchestra.run.smithi028.stderr:bluestore allocator dump block 2024-02-05T06:50:48.406 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: invalid command 2024-02-05T06:50:48.406 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:48 smithi028 ceph-mon[29442]: 3.12 deep-scrub starts 2024-02-05T06:50:48.406 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:48 smithi028 ceph-mon[29442]: 3.12 deep-scrub ok 2024-02-05T06:50:48.406 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:48 smithi028 ceph-mon[29442]: 4.1f scrub starts 2024-02-05T06:50:48.406 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:48 smithi028 ceph-mon[29442]: 4.1f scrub ok 2024-02-05T06:50:48.406 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:48 smithi028 ceph-mon[29442]: 5.4 scrub starts 2024-02-05T06:50:48.406 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:48 smithi028 ceph-mon[29442]: 5.4 scrub ok 2024-02-05T06:50:48.407 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:48 smithi028 ceph-mon[37984]: 3.12 deep-scrub starts 2024-02-05T06:50:48.407 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:48 smithi028 ceph-mon[37984]: 3.12 deep-scrub ok 2024-02-05T06:50:48.407 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:48 smithi028 ceph-mon[37984]: 4.1f scrub starts 2024-02-05T06:50:48.407 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:48 smithi028 ceph-mon[37984]: 4.1f scrub ok 2024-02-05T06:50:48.407 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:48 smithi028 ceph-mon[37984]: 5.4 scrub starts 2024-02-05T06:50:48.407 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:48 smithi028 ceph-mon[37984]: 5.4 scrub ok 2024-02-05T06:50:48.468 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:48.506 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:48.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:48.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:48.608 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:48.643 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:48.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:48.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:48.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:48 smithi167 ceph-mon[32774]: 3.12 deep-scrub starts 2024-02-05T06:50:48.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:48 smithi167 ceph-mon[32774]: 3.12 deep-scrub ok 2024-02-05T06:50:48.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:48 smithi167 ceph-mon[32774]: 4.1f scrub starts 2024-02-05T06:50:48.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:48 smithi167 ceph-mon[32774]: 4.1f scrub ok 2024-02-05T06:50:48.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:48 smithi167 ceph-mon[32774]: 5.4 scrub starts 2024-02-05T06:50:48.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:48 smithi167 ceph-mon[32774]: 5.4 scrub ok 2024-02-05T06:50:48.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:48.786 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:48.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:48.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:48.888 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:48.925 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:48.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:48.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:49.027 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:49.062 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:49.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:49.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:49.164 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:49.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:49.200 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:49.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:49.301 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:49.342 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:49.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:49.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:49.444 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:49.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:49.481 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:49.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:49.497 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:49 smithi028 ceph-mon[37984]: 5.1e scrub starts 2024-02-05T06:50:49.497 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:49 smithi028 ceph-mon[37984]: 5.1e scrub ok 2024-02-05T06:50:49.497 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:49 smithi028 ceph-mon[37984]: pgmap v832: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 953 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s; 2479/46308 objects misplaced (5.353%) 2024-02-05T06:50:49.497 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:49 smithi028 ceph-mon[37984]: 6.1c scrub starts 2024-02-05T06:50:49.497 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:49 smithi028 ceph-mon[37984]: 6.1c scrub ok 2024-02-05T06:50:49.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:49 smithi028 ceph-mon[29442]: 5.1e scrub starts 2024-02-05T06:50:49.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:49 smithi028 ceph-mon[29442]: 5.1e scrub ok 2024-02-05T06:50:49.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:49 smithi028 ceph-mon[29442]: pgmap v832: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 953 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s; 2479/46308 objects misplaced (5.353%) 2024-02-05T06:50:49.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:49 smithi028 ceph-mon[29442]: 6.1c scrub starts 2024-02-05T06:50:49.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:49 smithi028 ceph-mon[29442]: 6.1c scrub ok 2024-02-05T06:50:49.582 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:49.620 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:49.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:49.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:49.722 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:49.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:49 smithi167 ceph-mon[32774]: 5.1e scrub starts 2024-02-05T06:50:49.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:49 smithi167 ceph-mon[32774]: 5.1e scrub ok 2024-02-05T06:50:49.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:49 smithi167 ceph-mon[32774]: pgmap v832: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 953 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s; 2479/46308 objects misplaced (5.353%) 2024-02-05T06:50:49.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:49 smithi167 ceph-mon[32774]: 6.1c scrub starts 2024-02-05T06:50:49.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:49 smithi167 ceph-mon[32774]: 6.1c scrub ok 2024-02-05T06:50:49.758 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:49.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:49.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:49.861 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:49.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:49.901 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:49.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:50.002 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:50.047 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:50.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:50.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:50.149 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:50.187 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:50.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:50.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:50.289 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:50.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:50.328 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:50.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:50.429 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:50.468 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:50.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:50.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:50.570 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:50.611 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:50.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:50.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:50.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:50 smithi028 ceph-mon[37984]: 5.7 scrub starts 2024-02-05T06:50:50.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:50 smithi028 ceph-mon[37984]: 5.7 scrub ok 2024-02-05T06:50:50.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:50 smithi028 ceph-mon[37984]: 1.2 scrub starts 2024-02-05T06:50:50.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:50 smithi028 ceph-mon[37984]: 1.2 scrub ok 2024-02-05T06:50:50.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:50 smithi028 ceph-mon[29442]: 5.7 scrub starts 2024-02-05T06:50:50.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:50 smithi028 ceph-mon[29442]: 5.7 scrub ok 2024-02-05T06:50:50.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:50 smithi028 ceph-mon[29442]: 1.2 scrub starts 2024-02-05T06:50:50.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:50 smithi028 ceph-mon[29442]: 1.2 scrub ok 2024-02-05T06:50:50.716 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:50.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:50 smithi167 ceph-mon[32774]: 5.7 scrub starts 2024-02-05T06:50:50.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:50 smithi167 ceph-mon[32774]: 5.7 scrub ok 2024-02-05T06:50:50.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:50 smithi167 ceph-mon[32774]: 1.2 scrub starts 2024-02-05T06:50:50.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:50 smithi167 ceph-mon[32774]: 1.2 scrub ok 2024-02-05T06:50:50.753 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:50.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:50.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:50.855 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:50.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:50.890 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:50.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:50.991 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:51.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:51.031 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:51.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:51.132 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:51.167 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:51.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:51.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:51.269 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:51.303 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:51.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:51.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:51.400 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:51 smithi028 ceph-mon[29442]: 5.8 scrub starts 2024-02-05T06:50:51.400 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:51 smithi028 ceph-mon[29442]: 5.8 scrub ok 2024-02-05T06:50:51.400 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:51 smithi028 ceph-mon[29442]: 3.5 scrub starts 2024-02-05T06:50:51.404 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:51.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:51.442 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:51.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:51.543 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:51.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:51.577 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:51.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:51.678 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:51.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:51 smithi028 ceph-mon[37984]: 5.8 scrub starts 2024-02-05T06:50:51.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:51 smithi028 ceph-mon[37984]: 5.8 scrub ok 2024-02-05T06:50:51.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:51 smithi028 ceph-mon[37984]: 3.5 scrub starts 2024-02-05T06:50:51.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:51 smithi028 ceph-mon[37984]: 3.5 scrub ok 2024-02-05T06:50:51.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:51 smithi028 ceph-mon[37984]: pgmap v833: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 953 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.31k op/s; 2479/46308 objects misplaced (5.353%) 2024-02-05T06:50:51.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:51 smithi028 ceph-mon[37984]: 6.8 scrub starts 2024-02-05T06:50:51.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:51 smithi028 ceph-mon[37984]: 6.8 scrub ok 2024-02-05T06:50:51.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:51 smithi028 ceph-mon[29442]: 3.5 scrub ok 2024-02-05T06:50:51.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:51 smithi028 ceph-mon[29442]: pgmap v833: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 953 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.31k op/s; 2479/46308 objects misplaced (5.353%) 2024-02-05T06:50:51.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:51 smithi028 ceph-mon[29442]: 6.8 scrub starts 2024-02-05T06:50:51.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:51 smithi028 ceph-mon[29442]: 6.8 scrub ok 2024-02-05T06:50:51.720 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:51.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:51.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:51.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:51 smithi167 ceph-mon[32774]: 5.8 scrub starts 2024-02-05T06:50:51.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:51 smithi167 ceph-mon[32774]: 5.8 scrub ok 2024-02-05T06:50:51.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:51 smithi167 ceph-mon[32774]: 3.5 scrub starts 2024-02-05T06:50:51.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:51 smithi167 ceph-mon[32774]: 3.5 scrub ok 2024-02-05T06:50:51.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:51 smithi167 ceph-mon[32774]: pgmap v833: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 953 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.31k op/s; 2479/46308 objects misplaced (5.353%) 2024-02-05T06:50:51.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:51 smithi167 ceph-mon[32774]: 6.8 scrub starts 2024-02-05T06:50:51.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:51 smithi167 ceph-mon[32774]: 6.8 scrub ok 2024-02-05T06:50:51.823 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:51.862 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:51.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:51.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:51.964 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:52.010 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:52.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:52.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:52.113 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:52.146 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:52.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:52.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:52.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:52.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:52.290 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:52.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:52.391 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:52.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:52.431 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:52.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:52.532 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:52.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:52.574 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:52.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:52.676 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:52.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:52 smithi028 ceph-mon[29442]: 1.a scrub starts 2024-02-05T06:50:52.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:52 smithi028 ceph-mon[29442]: 1.a scrub ok 2024-02-05T06:50:52.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:52 smithi028 ceph-mon[29442]: 4.16 scrub starts 2024-02-05T06:50:52.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:52 smithi028 ceph-mon[29442]: 4.16 scrub ok 2024-02-05T06:50:52.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:52 smithi028 ceph-mon[29442]: 5.1f scrub starts 2024-02-05T06:50:52.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:52 smithi028 ceph-mon[29442]: 5.1f scrub ok 2024-02-05T06:50:52.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:52 smithi028 ceph-mon[29442]: 3.f deep-scrub starts 2024-02-05T06:50:52.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:52 smithi028 ceph-mon[29442]: 3.f deep-scrub ok 2024-02-05T06:50:52.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:52 smithi028 ceph-mon[37984]: 1.a scrub starts 2024-02-05T06:50:52.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:52 smithi028 ceph-mon[37984]: 1.a scrub ok 2024-02-05T06:50:52.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:52 smithi028 ceph-mon[37984]: 4.16 scrub starts 2024-02-05T06:50:52.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:52 smithi028 ceph-mon[37984]: 4.16 scrub ok 2024-02-05T06:50:52.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:52 smithi028 ceph-mon[37984]: 5.1f scrub starts 2024-02-05T06:50:52.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:52 smithi028 ceph-mon[37984]: 5.1f scrub ok 2024-02-05T06:50:52.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:52 smithi028 ceph-mon[37984]: 3.f deep-scrub starts 2024-02-05T06:50:52.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:52 smithi028 ceph-mon[37984]: 3.f deep-scrub ok 2024-02-05T06:50:52.703 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:50:52 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:50:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:50:52.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:52.715 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:52.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:52.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:52 smithi167 ceph-mon[32774]: 1.a scrub starts 2024-02-05T06:50:52.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:52 smithi167 ceph-mon[32774]: 1.a scrub ok 2024-02-05T06:50:52.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:52 smithi167 ceph-mon[32774]: 4.16 scrub starts 2024-02-05T06:50:52.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:52 smithi167 ceph-mon[32774]: 4.16 scrub ok 2024-02-05T06:50:52.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:52 smithi167 ceph-mon[32774]: 5.1f scrub starts 2024-02-05T06:50:52.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:52 smithi167 ceph-mon[32774]: 5.1f scrub ok 2024-02-05T06:50:52.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:52 smithi167 ceph-mon[32774]: 3.f deep-scrub starts 2024-02-05T06:50:52.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:52 smithi167 ceph-mon[32774]: 3.f deep-scrub ok 2024-02-05T06:50:52.816 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:52.850 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:52.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:52.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:52.952 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:52.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:52.988 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:52.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:53.089 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:53.125 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:53.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:53.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:53.227 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:53.265 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:53.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:53.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:53.315 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-05T06:50:53.316 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-05T06:50:53.316 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph tell osd.5 'trim stale osdmaps' 2024-02-05T06:50:53.367 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:53.397 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:53.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:53.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:53.499 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:53.520 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:50:53.520 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:50:53.533 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:53.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:53.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:53.635 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:53.656 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:50:53 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:50:53.657 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:53 smithi167 ceph-mon[32774]: 6.6 deep-scrub starts 2024-02-05T06:50:53.657 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:53 smithi167 ceph-mon[32774]: 6.6 deep-scrub ok 2024-02-05T06:50:53.657 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:53 smithi167 ceph-mon[32774]: pgmap v834: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 588 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1004 B/s rd, 0 B/s wr, 1.19k op/s; 1521/28791 objects misplaced (5.283%) 2024-02-05T06:50:53.672 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:53.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:53.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:53.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:53 smithi028 ceph-mon[29442]: 6.6 deep-scrub starts 2024-02-05T06:50:53.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:53 smithi028 ceph-mon[29442]: 6.6 deep-scrub ok 2024-02-05T06:50:53.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:53 smithi028 ceph-mon[29442]: pgmap v834: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 588 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1004 B/s rd, 0 B/s wr, 1.19k op/s; 1521/28791 objects misplaced (5.283%) 2024-02-05T06:50:53.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:53 smithi028 ceph-mon[37984]: 6.6 deep-scrub starts 2024-02-05T06:50:53.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:53 smithi028 ceph-mon[37984]: 6.6 deep-scrub ok 2024-02-05T06:50:53.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:53 smithi028 ceph-mon[37984]: pgmap v834: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 588 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1004 B/s rd, 0 B/s wr, 1.19k op/s; 1521/28791 objects misplaced (5.283%) 2024-02-05T06:50:53.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:53.812 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:53.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:53.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:53.914 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:53.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:53.951 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:53.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:54.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:54.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:54.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:54.191 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:54.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:54.225 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:54.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:54.326 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:54.365 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:54.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:54.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:54.467 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:54.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:54.534 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:54.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:54.636 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:54.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:54.684 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:54.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:54.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:54 smithi028 ceph-mon[37984]: 6.17 scrub starts 2024-02-05T06:50:54.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:54 smithi028 ceph-mon[37984]: 6.17 scrub ok 2024-02-05T06:50:54.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:54 smithi028 ceph-mon[37984]: 3.6 scrub starts 2024-02-05T06:50:54.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:54 smithi028 ceph-mon[37984]: 3.6 scrub ok 2024-02-05T06:50:54.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:54 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:50:54.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:54 smithi028 ceph-mon[37984]: 6.c scrub starts 2024-02-05T06:50:54.703 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:54 smithi028 ceph-mon[37984]: 6.c scrub ok 2024-02-05T06:50:54.703 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:54 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:50:54.703 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:54 smithi028 ceph-mon[37984]: 6.e scrub starts 2024-02-05T06:50:54.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:54 smithi028 ceph-mon[29442]: 6.17 scrub starts 2024-02-05T06:50:54.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:54 smithi028 ceph-mon[29442]: 6.17 scrub ok 2024-02-05T06:50:54.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:54 smithi028 ceph-mon[29442]: 3.6 scrub starts 2024-02-05T06:50:54.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:54 smithi028 ceph-mon[29442]: 3.6 scrub ok 2024-02-05T06:50:54.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:54 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:50:54.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:54 smithi028 ceph-mon[29442]: 6.c scrub starts 2024-02-05T06:50:54.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:54 smithi028 ceph-mon[29442]: 6.c scrub ok 2024-02-05T06:50:54.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:54 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:50:54.705 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:54 smithi028 ceph-mon[29442]: 6.e scrub starts 2024-02-05T06:50:54.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:54 smithi167 ceph-mon[32774]: 6.17 scrub starts 2024-02-05T06:50:54.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:54 smithi167 ceph-mon[32774]: 6.17 scrub ok 2024-02-05T06:50:54.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:54 smithi167 ceph-mon[32774]: 3.6 scrub starts 2024-02-05T06:50:54.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:54 smithi167 ceph-mon[32774]: 3.6 scrub ok 2024-02-05T06:50:54.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:54 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:50:54.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:54 smithi167 ceph-mon[32774]: 6.c scrub starts 2024-02-05T06:50:54.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:54 smithi167 ceph-mon[32774]: 6.c scrub ok 2024-02-05T06:50:54.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:54 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:50:54.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:54 smithi167 ceph-mon[32774]: 6.e scrub starts 2024-02-05T06:50:54.786 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:54.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:54.824 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:54.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:54.869 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-05T06:50:54.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:54.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:54.963 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:54.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:55.065 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:55.094 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:55.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:55.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:55.196 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:55.255 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:55.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:55.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:55.357 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:55.385 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:50:55.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:55.389 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:55.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:55.491 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:55.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:55.537 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:55.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:55.638 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:55.685 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:55.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:55.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:55.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:55 smithi028 ceph-mon[29442]: 6.e scrub ok 2024-02-05T06:50:55.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:55 smithi028 ceph-mon[29442]: 5.2 scrub starts 2024-02-05T06:50:55.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:55 smithi028 ceph-mon[29442]: 5.2 scrub ok 2024-02-05T06:50:55.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:55 smithi028 ceph-mon[29442]: pgmap v835: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 562 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.04k op/s; 1521/27525 objects misplaced (5.526%) 2024-02-05T06:50:55.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:55 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:50:55.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:55 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:50:55.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:55 smithi028 ceph-mon[37984]: 6.e scrub ok 2024-02-05T06:50:55.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:55 smithi028 ceph-mon[37984]: 5.2 scrub starts 2024-02-05T06:50:55.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:55 smithi028 ceph-mon[37984]: 5.2 scrub ok 2024-02-05T06:50:55.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:55 smithi028 ceph-mon[37984]: pgmap v835: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 562 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.04k op/s; 1521/27525 objects misplaced (5.526%) 2024-02-05T06:50:55.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:55 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:50:55.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:55 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:50:55.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:55 smithi167 ceph-mon[32774]: 6.e scrub ok 2024-02-05T06:50:55.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:55 smithi167 ceph-mon[32774]: 5.2 scrub starts 2024-02-05T06:50:55.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:55 smithi167 ceph-mon[32774]: 5.2 scrub ok 2024-02-05T06:50:55.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:55 smithi167 ceph-mon[32774]: pgmap v835: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 562 MiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.04k op/s; 1521/27525 objects misplaced (5.526%) 2024-02-05T06:50:55.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:55 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:50:55.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:55 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:50:55.787 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:55.844 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:55.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:55.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:55.946 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:50:55.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:55.987 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:55.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:50:56.089 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:56.118 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:56.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:56.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:56.220 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:56.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:56.264 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:56.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:56.365 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:56.399 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:56.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:56.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:56.501 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:56.550 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:56.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:56.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:56.652 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:56.681 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:56.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:56.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:56.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:56 smithi028 ceph-mon[29442]: 4.17 deep-scrub starts 2024-02-05T06:50:56.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:56 smithi028 ceph-mon[29442]: 4.17 deep-scrub ok 2024-02-05T06:50:56.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:56 smithi028 ceph-mon[37984]: 4.17 deep-scrub starts 2024-02-05T06:50:56.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:56 smithi028 ceph-mon[37984]: 4.17 deep-scrub ok 2024-02-05T06:50:56.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:56 smithi167 ceph-mon[32774]: 4.17 deep-scrub starts 2024-02-05T06:50:56.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:56 smithi167 ceph-mon[32774]: 4.17 deep-scrub ok 2024-02-05T06:50:56.783 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:56.817 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:56.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:56.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:56.920 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:56.945 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:56.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:56.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:57.047 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:57.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:57.072 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:57.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:57.173 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:57.197 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:57.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:57.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:57.250 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:50:57.272 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:50:57.292 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:50:57.299 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:57.313 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:50:57.323 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:57.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:57.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:57.334 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:50:57.354 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:50:57.374 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:50:57.394 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:50:57.395 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:50:57.395 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:50:57.395 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:50:57.395 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:50:57.396 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:50:57.396 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:50:57.396 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:50:57.396 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:50:57.425 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:57.450 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:57.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:57.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:57.552 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:50:57.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:57.576 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:57.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:50:57.679 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:57.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:57 smithi028 ceph-mon[29442]: osdmap e413: 8 total, 8 up, 8 in 2024-02-05T06:50:57.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:57 smithi028 ceph-mon[29442]: 4.5 deep-scrub starts 2024-02-05T06:50:57.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:57 smithi028 ceph-mon[29442]: 4.5 deep-scrub ok 2024-02-05T06:50:57.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:57 smithi028 ceph-mon[29442]: pgmap v837: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 538 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.21k op/s; 1521/26415 objects misplaced (5.758%) 2024-02-05T06:50:57.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:57 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:50:57.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:57 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:50:57.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:57 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:50:57.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:57 smithi028 ceph-mon[29442]: 3.17 scrub starts 2024-02-05T06:50:57.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:57 smithi028 ceph-mon[29442]: 3.17 scrub ok 2024-02-05T06:50:57.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:57 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/653379452' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:50:57.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:57 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:50:57.685 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:57 smithi028 ceph-mon[37984]: osdmap e413: 8 total, 8 up, 8 in 2024-02-05T06:50:57.685 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:57 smithi028 ceph-mon[37984]: 4.5 deep-scrub starts 2024-02-05T06:50:57.685 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:57 smithi028 ceph-mon[37984]: 4.5 deep-scrub ok 2024-02-05T06:50:57.685 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:57 smithi028 ceph-mon[37984]: pgmap v837: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 538 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.21k op/s; 1521/26415 objects misplaced (5.758%) 2024-02-05T06:50:57.685 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:57 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:50:57.685 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:57 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:50:57.685 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:57 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:50:57.685 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:57 smithi028 ceph-mon[37984]: 3.17 scrub starts 2024-02-05T06:50:57.685 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:57 smithi028 ceph-mon[37984]: 3.17 scrub ok 2024-02-05T06:50:57.685 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:57 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/653379452' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:50:57.685 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:57 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:50:57.693 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:50:57.699 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:57 smithi167 ceph-mon[32774]: osdmap e413: 8 total, 8 up, 8 in 2024-02-05T06:50:57.700 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:57 smithi167 ceph-mon[32774]: 4.5 deep-scrub starts 2024-02-05T06:50:57.700 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:57 smithi167 ceph-mon[32774]: 4.5 deep-scrub ok 2024-02-05T06:50:57.700 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:57 smithi167 ceph-mon[32774]: pgmap v837: 168 pgs: 2 active+remapped+backfill_toofull, 166 active+clean; 538 MiB data, 7.1 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.21k op/s; 1521/26415 objects misplaced (5.758%) 2024-02-05T06:50:57.700 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:57 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:50:57.700 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:57 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:50:57.700 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:57 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:50:57.700 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:57 smithi167 ceph-mon[32774]: 3.17 scrub starts 2024-02-05T06:50:57.700 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:57 smithi167 ceph-mon[32774]: 3.17 scrub ok 2024-02-05T06:50:57.700 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:57 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/653379452' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:50:57.700 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:57 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:50:57.711 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:57.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:57.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:57.813 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:57.850 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:57.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:57.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:57.952 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:57.985 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:57.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:57.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:58.047 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:50:58.047 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-02-05T06:50:58.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:50:58.047 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.45587.0:238729 10.14 e414/395 10:29fe9bb3:::benchmark_data_smithi028_154709_object117728:head v 414'10595, mlcod=414'10593)", 2024-02-05T06:50:58.047 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:50:58.033348+0000", 2024-02-05T06:50:58.047 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0042013789999999999, 2024-02-05T06:50:58.047 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0042328629999999999, 2024-02-05T06:50:58.047 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:50:58.047 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-02-05T06:50:58.047 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:50:58.047 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:50:58.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:50:58.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:58.033348+0000", 2024-02-05T06:50:58.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:50:58.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:50:58.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:50:58.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:50:58.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:58.033348+0000", 2024-02-05T06:50:58.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:50:58.048 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:50:58.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:50:58.048 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:50:58.048 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:58.033349+0000", 2024-02-05T06:50:58.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4470000000000001e-06 2024-02-05T06:50:58.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:50:58.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:50:58.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:50:58.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:58.033360+0000", 2024-02-05T06:50:58.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0212e-05 2024-02-05T06:50:58.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:50:58.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:50:58.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:50:58.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:58.033374+0000", 2024-02-05T06:50:58.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4015e-05 2024-02-05T06:50:58.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:50:58.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:50:58.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:50:58.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:58.033381+0000", 2024-02-05T06:50:58.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3939999999999998e-06 2024-02-05T06:50:58.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:50:58.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:50:58.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:50:58.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:58.033540+0000", 2024-02-05T06:50:58.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015929299999999999 2024-02-05T06:50:58.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:50:58.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:50:58.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:50:58.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:58.033597+0000", 2024-02-05T06:50:58.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6475999999999999e-05 2024-02-05T06:50:58.050 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:50:58.050 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:50:58.050 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:50:58.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:50:58.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:50:58.050 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.45587.0:238730 10.e e414/345 10:7d037097:::benchmark_data_smithi028_154709_object117729:head v 414'15189, mlcod=414'15187)", 2024-02-05T06:50:58.051 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:50:58.034827+0000", 2024-02-05T06:50:58.051 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0027224760000000001, 2024-02-05T06:50:58.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0028208349999999998, 2024-02-05T06:50:58.051 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:50:58.051 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-02-05T06:50:58.051 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:50:58.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:50:58.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:50:58.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:58.034827+0000", 2024-02-05T06:50:58.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:50:58.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:50:58.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:50:58.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:50:58.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:58.034827+0000", 2024-02-05T06:50:58.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:50:58.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:50:58.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:50:58.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:50:58.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:58.034828+0000", 2024-02-05T06:50:58.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.19e-06 2024-02-05T06:50:58.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:50:58.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:50:58.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:50:58.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:58.034836+0000", 2024-02-05T06:50:58.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5789999999999998e-06 2024-02-05T06:50:58.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:50:58.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:50:58.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:50:58.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:58.034859+0000", 2024-02-05T06:50:58.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3442999999999999e-05 2024-02-05T06:50:58.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:50:58.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:50:58.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:50:58.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:58.034867+0000", 2024-02-05T06:50:58.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5100000000000001e-06 2024-02-05T06:50:58.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:50:58.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:50:58.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:50:58.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:58.035023+0000", 2024-02-05T06:50:58.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000156625 2024-02-05T06:50:58.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:50:58.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:50:58.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:50:58.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:58.035077+0000", 2024-02-05T06:50:58.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3817000000000003e-05 2024-02-05T06:50:58.054 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:50:58.054 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:50:58.054 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:50:58.054 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:50:58.054 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-02-05T06:50:58.054 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 2 2024-02-05T06:50:58.054 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:50:58.087 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:58.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:58.123 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:58.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:58.224 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:58.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:58.264 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:58.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:58.366 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:58.396 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:58.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:58.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:58.477 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-02-05T06:50:58.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:58 smithi028 ceph-mon[29442]: 3.18 scrub starts 2024-02-05T06:50:58.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:58 smithi028 ceph-mon[29442]: 3.18 scrub ok 2024-02-05T06:50:58.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:58 smithi028 ceph-mon[29442]: osdmap e414: 8 total, 8 up, 8 in 2024-02-05T06:50:58.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:58 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/311933084' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:50:58.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:58 smithi028 ceph-mon[29442]: from='client.45821 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:50:58.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:58 smithi028 ceph-mon[29442]: 6.12 scrub starts 2024-02-05T06:50:58.496 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:58 smithi028 ceph-mon[29442]: 6.12 scrub ok 2024-02-05T06:50:58.497 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:58 smithi028 ceph-mon[37984]: 3.18 scrub starts 2024-02-05T06:50:58.497 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:58 smithi028 ceph-mon[37984]: 3.18 scrub ok 2024-02-05T06:50:58.497 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:58 smithi028 ceph-mon[37984]: osdmap e414: 8 total, 8 up, 8 in 2024-02-05T06:50:58.497 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:58 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/311933084' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:50:58.497 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:58 smithi028 ceph-mon[37984]: from='client.45821 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:50:58.497 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:58 smithi028 ceph-mon[37984]: 6.12 scrub starts 2024-02-05T06:50:58.497 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:58 smithi028 ceph-mon[37984]: 6.12 scrub ok 2024-02-05T06:50:58.497 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:58.543 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:58.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:58.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:58.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:58.675 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:58.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:58.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:58.724 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:58 smithi167 ceph-mon[32774]: 3.18 scrub starts 2024-02-05T06:50:58.724 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:58 smithi167 ceph-mon[32774]: 3.18 scrub ok 2024-02-05T06:50:58.724 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:58 smithi167 ceph-mon[32774]: osdmap e414: 8 total, 8 up, 8 in 2024-02-05T06:50:58.724 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:58 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/311933084' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:50:58.724 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:58 smithi167 ceph-mon[32774]: from='client.45821 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:50:58.724 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:58 smithi167 ceph-mon[32774]: 6.12 scrub starts 2024-02-05T06:50:58.724 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:58 smithi167 ceph-mon[32774]: 6.12 scrub ok 2024-02-05T06:50:58.774 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-05T06:50:58.777 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:58.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:58.809 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:58.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:58.910 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:50:58.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:58.962 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:58.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:50:59.063 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:50:59.125 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:59.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:59.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:50:59.218 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:50:59.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:50:59.256 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:59.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:59.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:50:59.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:59.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:59.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:59.493 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:59.533 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:59.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:59.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:50:59.635 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:50:59.675 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:50:59.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:59.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:50:59.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:59 smithi028 ceph-mon[37984]: 6.19 scrub starts 2024-02-05T06:50:59.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:59 smithi028 ceph-mon[37984]: 6.19 scrub ok 2024-02-05T06:50:59.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:59 smithi028 ceph-mon[37984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:50:59.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:59 smithi028 ceph-mon[37984]: from='client.45821 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:50:59.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:59 smithi028 ceph-mon[37984]: osdmap e415: 8 total, 8 up, 8 in 2024-02-05T06:50:59.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:50:59 smithi028 ceph-mon[37984]: pgmap v840: 168 pgs: 1 active+remapped+backfill_toofull, 1 peering, 166 active+clean; 231 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 951 op/s; 438/11670 objects misplaced (3.753%); 1.7 MiB/s, 26 objects/s recovering 2024-02-05T06:50:59.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:59 smithi028 ceph-mon[29442]: 6.19 scrub starts 2024-02-05T06:50:59.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:59 smithi028 ceph-mon[29442]: 6.19 scrub ok 2024-02-05T06:50:59.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:59 smithi028 ceph-mon[29442]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:50:59.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:59 smithi028 ceph-mon[29442]: from='client.45821 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:50:59.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:59 smithi028 ceph-mon[29442]: osdmap e415: 8 total, 8 up, 8 in 2024-02-05T06:50:59.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:50:59 smithi028 ceph-mon[29442]: pgmap v840: 168 pgs: 1 active+remapped+backfill_toofull, 1 peering, 166 active+clean; 231 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 951 op/s; 438/11670 objects misplaced (3.753%); 1.7 MiB/s, 26 objects/s recovering 2024-02-05T06:50:59.723 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:59 smithi167 ceph-mon[32774]: 6.19 scrub starts 2024-02-05T06:50:59.723 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:59 smithi167 ceph-mon[32774]: 6.19 scrub ok 2024-02-05T06:50:59.723 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:59 smithi167 ceph-mon[32774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:50:59.723 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:59 smithi167 ceph-mon[32774]: from='client.45821 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:50:59.724 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:59 smithi167 ceph-mon[32774]: osdmap e415: 8 total, 8 up, 8 in 2024-02-05T06:50:59.724 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:50:59 smithi167 ceph-mon[32774]: pgmap v840: 168 pgs: 1 active+remapped+backfill_toofull, 1 peering, 166 active+clean; 231 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 951 op/s; 438/11670 objects misplaced (3.753%); 1.7 MiB/s, 26 objects/s recovering 2024-02-05T06:50:59.776 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:50:59.809 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:59.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:59.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:50:59.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:50:59.949 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:50:59.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:50:59.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:00.051 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:00.088 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:00.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:00.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:00.190 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:00.226 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:00.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:00.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:00.328 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:00.367 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:00.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:00.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:00.469 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:00.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:00.504 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:00.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:00.605 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:00.635 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:00.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:00.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:00.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:00 smithi028 ceph-mon[37984]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:51:00.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:00 smithi028 ceph-mon[29442]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:51:00.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:00 smithi167 ceph-mon[32774]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-02-05T06:51:00.737 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:00.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:00.776 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:00.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:00.877 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:00.901 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Removed 120998 objects 2024-02-05T06:51:00.901 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Clean up completed and total clean up time :110.933 2024-02-05T06:51:00.909 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:00.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:00.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:00.923 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-02-05T06:51:00.924 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-02-05T06:51:01.011 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:01.037 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:01.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:01.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:01.139 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:01.145 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:51:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-02-05T06:51:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-02-05T06:51:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-02-05T06:51:01.145 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:51:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:01.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:01.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:01.269 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:01.293 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:01.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:01.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:01.395 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:01.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:01.420 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:01.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:01.521 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:01.545 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:01.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:01.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:01.647 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:01.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:01 smithi028 ceph-mon[29442]: pgmap v841: 168 pgs: 1 active+remapped+backfill_toofull, 1 peering, 166 active+clean; 231 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 881 op/s; 438/11670 objects misplaced (3.753%); 1.7 MiB/s, 26 objects/s recovering 2024-02-05T06:51:01.661 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:01 smithi028 ceph-mon[37984]: pgmap v841: 168 pgs: 1 active+remapped+backfill_toofull, 1 peering, 166 active+clean; 231 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 881 op/s; 438/11670 objects misplaced (3.753%); 1.7 MiB/s, 26 objects/s recovering 2024-02-05T06:51:01.671 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:01.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:01.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:01.703 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:51:01.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:01 smithi167 ceph-mon[32774]: pgmap v841: 168 pgs: 1 active+remapped+backfill_toofull, 1 peering, 166 active+clean; 231 MiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 881 op/s; 438/11670 objects misplaced (3.753%); 1.7 MiB/s, 26 objects/s recovering 2024-02-05T06:51:01.743 INFO:teuthology.orchestra.run.smithi028.stderr:no valid command found; 10 closest matches: 2024-02-05T06:51:01.743 INFO:teuthology.orchestra.run.smithi028.stderr:0 2024-02-05T06:51:01.743 INFO:teuthology.orchestra.run.smithi028.stderr:1 2024-02-05T06:51:01.743 INFO:teuthology.orchestra.run.smithi028.stderr:2 2024-02-05T06:51:01.743 INFO:teuthology.orchestra.run.smithi028.stderr:abort 2024-02-05T06:51:01.743 INFO:teuthology.orchestra.run.smithi028.stderr:assert 2024-02-05T06:51:01.744 INFO:teuthology.orchestra.run.smithi028.stderr:bench [] [] [] [] 2024-02-05T06:51:01.744 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs debug_inject_read_zeros 2024-02-05T06:51:01.744 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs files list 2024-02-05T06:51:01.744 INFO:teuthology.orchestra.run.smithi028.stderr:bluefs stats 2024-02-05T06:51:01.744 INFO:teuthology.orchestra.run.smithi028.stderr:bluestore allocator dump block 2024-02-05T06:51:01.744 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: invalid command 2024-02-05T06:51:01.762 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T06:51:01.773 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:01.797 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:01.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:01.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:01.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:01.923 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:01.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:01.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:02.024 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:02.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:02.051 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:02.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:02.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:02.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:02.176 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:02.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:02.277 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:02.301 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:02.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:02.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:02.403 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:02.431 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:02.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:02.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:02.533 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:02.561 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:02.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:02.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:02.663 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:02.689 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:51:02 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:51:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:51:02.691 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:02.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:02.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:02.693 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:51:02.794 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:02.817 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:02.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:02.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:02.919 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:02.947 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:02.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:02.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:03.049 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:03.057 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-05T06:51:03.074 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:03.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:03.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:03.090 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-02-05T06:51:03.091 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-02-05T06:51:03.176 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:03.199 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:03.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:03.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:03.301 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:03.322 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:51:03.325 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:03.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:03.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:03.427 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:03.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:03.451 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:03.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:03.552 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:03.575 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:03.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:03.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:03.677 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:03.704 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:03.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:03.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:03.717 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:51:03.731 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:51:03.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:03 smithi167 ceph-mon[32774]: pgmap v842: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 15 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.37k op/s; 24/1272 objects misplaced (1.887%); 1.7 MiB/s, 26 objects/s recovering 2024-02-05T06:51:03.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:51:03 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:51:03.806 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:03.829 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:03.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:03.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:03.845 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:03 smithi028 ceph-mon[29442]: pgmap v842: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 15 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.37k op/s; 24/1272 objects misplaced (1.887%); 1.7 MiB/s, 26 objects/s recovering 2024-02-05T06:51:03.845 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:03 smithi028 ceph-mon[37984]: pgmap v842: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 15 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.37k op/s; 24/1272 objects misplaced (1.887%); 1.7 MiB/s, 26 objects/s recovering 2024-02-05T06:51:03.931 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:03.957 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:03.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:03.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:04.059 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:04.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:04.087 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:04.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:04.188 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:04.213 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:04.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:04.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:04.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:04.340 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:04.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:04.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:04.442 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:04.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:04.467 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:04.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:04.569 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:04.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:04.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:04.612 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'unique_pool_3' removed 2024-02-05T06:51:04.688 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:04 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:51:04.688 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:04 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/11484387' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T06:51:04.695 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:04.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:04.720 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:04.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:04.821 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:04.845 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:04.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:04.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:04.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:04.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:04 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:51:04.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:04 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/11484387' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T06:51:04.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:04 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:51:04.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:04 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/11484387' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T06:51:04.972 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:04.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:04.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:05.074 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:05.097 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:05.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:05.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:05.199 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:05.223 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:05.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:05.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:05.325 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:05.342 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-02-05T06:51:05.343 INFO:tasks.radosbench:Beginning radosbench... 2024-02-05T06:51:05.343 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-02-05T06:51:05.343 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd pool create unique_pool_4 16 2024-02-05T06:51:05.349 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:05.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:05.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:05.451 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:05.476 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:05.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:05.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:05.577 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:05.602 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:05.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:05.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:05.704 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:05.729 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:05.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:05.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:05.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:05 smithi167 ceph-mon[32774]: pgmap v843: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.1 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 24/618 objects misplaced (3.883%); 1.3 MiB/s, 20 objects/s recovering 2024-02-05T06:51:05.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:05 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/11484387' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T06:51:05.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:05 smithi167 ceph-mon[32774]: osdmap e416: 8 total, 8 up, 8 in 2024-02-05T06:51:05.831 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:05.855 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:05.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:05.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:05.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:05 smithi028 ceph-mon[29442]: pgmap v843: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.1 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 24/618 objects misplaced (3.883%); 1.3 MiB/s, 20 objects/s recovering 2024-02-05T06:51:05.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:05 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/11484387' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T06:51:05.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:05 smithi028 ceph-mon[29442]: osdmap e416: 8 total, 8 up, 8 in 2024-02-05T06:51:05.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:05 smithi028 ceph-mon[37984]: pgmap v843: 168 pgs: 1 active+remapped+backfill_toofull, 167 active+clean; 1.1 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 24/618 objects misplaced (3.883%); 1.3 MiB/s, 20 objects/s recovering 2024-02-05T06:51:05.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:05 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/11484387' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T06:51:05.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:05 smithi028 ceph-mon[37984]: osdmap e416: 8 total, 8 up, 8 in 2024-02-05T06:51:05.957 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:05.980 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:05.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:05.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:06.081 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:06.107 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:06.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:06.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:06.208 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:06.232 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:06.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:06.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:06.334 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:06.358 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:06.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:06.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:06.366 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:51:06.366 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-02-05T06:51:06.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-02-05T06:51:06.367 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-02-05T06:51:06.367 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.367 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:140508 8.2 8:42800101:::benchmark_data_smithi028_105698_object51233:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-02-05T06:51:06.367 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:23.430618+0000", 2024-02-05T06:51:06.367 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 522.92525516600006, 2024-02-05T06:51:06.367 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0314014380000001, 2024-02-05T06:51:06.367 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:06.367 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-02-05T06:51:06.367 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:06.367 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:51:06.367 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:51:06.367 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 140508 2024-02-05T06:51:06.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.368 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:06.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:06.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.430618+0000", 2024-02-05T06:51:06.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.368 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.368 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:06.368 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.430618+0000", 2024-02-05T06:51:06.368 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.368 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:06.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.430620+0000", 2024-02-05T06:51:06.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.821e-06 2024-02-05T06:51:06.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:06.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.430624+0000", 2024-02-05T06:51:06.369 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6500000000000004e-06 2024-02-05T06:51:06.369 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.369 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.369 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:06.369 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.430636+0000", 2024-02-05T06:51:06.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1297e-05 2024-02-05T06:51:06.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:06.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.430642+0000", 2024-02-05T06:51:06.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2299999999999996e-06 2024-02-05T06:51:06.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.370 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.430696+0000", 2024-02-05T06:51:06.370 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4397000000000001e-05 2024-02-05T06:51:06.370 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.370 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.370 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:06.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.430845+0000", 2024-02-05T06:51:06.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000148636 2024-02-05T06:51:06.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3,5", 2024-02-05T06:51:06.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.431017+0000", 2024-02-05T06:51:06.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017170299999999999 2024-02-05T06:51:06.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:51:06.371 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.433418+0000", 2024-02-05T06:51:06.371 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002401659 2024-02-05T06:51:06.371 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.371 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.371 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:06.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.434052+0000", 2024-02-05T06:51:06.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00063330400000000001 2024-02-05T06:51:06.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:06.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.436375+0000", 2024-02-05T06:51:06.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00232336 2024-02-05T06:51:06.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.372 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.372 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.372 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.446057+0000", 2024-02-05T06:51:06.372 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0096824479999999997 2024-02-05T06:51:06.372 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:06.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.446073+0000", 2024-02-05T06:51:06.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.526e-05 2024-02-05T06:51:06.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.373 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.373 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.461951+0000", 2024-02-05T06:51:06.373 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.01587871 2024-02-05T06:51:06.373 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.373 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.374 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:06.374 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.462019+0000", 2024-02-05T06:51:06.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7963e-05 2024-02-05T06:51:06.374 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.374 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:06.374 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.374 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.374 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.374 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:140511 8.2 8:4014bd7d:::benchmark_data_smithi028_105698_object51236:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-02-05T06:51:06.374 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:23.436418+0000", 2024-02-05T06:51:06.374 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 522.919455599, 2024-02-05T06:51:06.374 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.026065132, 2024-02-05T06:51:06.374 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:06.375 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-02-05T06:51:06.375 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:06.375 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:51:06.375 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:51:06.375 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 140511 2024-02-05T06:51:06.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.375 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:06.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:06.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.436418+0000", 2024-02-05T06:51:06.375 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.375 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.375 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.375 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:06.375 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.436418+0000", 2024-02-05T06:51:06.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:06.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.436419+0000", 2024-02-05T06:51:06.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.649e-06 2024-02-05T06:51:06.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.376 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.376 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:06.376 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.436426+0000", 2024-02-05T06:51:06.376 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0589999999999997e-06 2024-02-05T06:51:06.376 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:06.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.436460+0000", 2024-02-05T06:51:06.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4032000000000002e-05 2024-02-05T06:51:06.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:06.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.436469+0000", 2024-02-05T06:51:06.377 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1200000000000008e-06 2024-02-05T06:51:06.377 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.377 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.377 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.377 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.436592+0000", 2024-02-05T06:51:06.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123046 2024-02-05T06:51:06.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:06.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.436726+0000", 2024-02-05T06:51:06.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013393899999999999 2024-02-05T06:51:06.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3,5", 2024-02-05T06:51:06.378 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.436861+0000", 2024-02-05T06:51:06.378 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000134368 2024-02-05T06:51:06.378 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.378 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.378 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:51:06.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.441695+0000", 2024-02-05T06:51:06.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.004834634 2024-02-05T06:51:06.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.446106+0000", 2024-02-05T06:51:06.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0044110570000000003 2024-02-05T06:51:06.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.379 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:06.379 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.446117+0000", 2024-02-05T06:51:06.379 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0428000000000001e-05 2024-02-05T06:51:06.379 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.379 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.462403+0000", 2024-02-05T06:51:06.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0162859929999999 2024-02-05T06:51:06.380 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.380 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.380 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:06.380 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.462483+0000", 2024-02-05T06:51:06.380 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9807000000000003e-05 2024-02-05T06:51:06.380 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.380 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:06.381 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.381 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.381 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.381 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:140513 8.2 8:46eb6567:::benchmark_data_smithi028_105698_object51238:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-02-05T06:51:06.381 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:23.436672+0000", 2024-02-05T06:51:06.381 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 522.91920127900005, 2024-02-05T06:51:06.381 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.025900399, 2024-02-05T06:51:06.381 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:06.381 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-02-05T06:51:06.381 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:06.381 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:51:06.381 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:51:06.381 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 140513 2024-02-05T06:51:06.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.382 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:06.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:06.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.436672+0000", 2024-02-05T06:51:06.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.382 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.382 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:06.382 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.436672+0000", 2024-02-05T06:51:06.382 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.382 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:06.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.436673+0000", 2024-02-05T06:51:06.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1370000000000001e-06 2024-02-05T06:51:06.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:06.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.436677+0000", 2024-02-05T06:51:06.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1219999999999997e-06 2024-02-05T06:51:06.383 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.383 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.383 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:06.383 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.436686+0000", 2024-02-05T06:51:06.383 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7590000000000003e-06 2024-02-05T06:51:06.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:06.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.436693+0000", 2024-02-05T06:51:06.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.951e-06 2024-02-05T06:51:06.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.384 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.384 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.438808+0000", 2024-02-05T06:51:06.384 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021152459999999999 2024-02-05T06:51:06.384 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.384 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:06.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.438970+0000", 2024-02-05T06:51:06.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000161435 2024-02-05T06:51:06.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,3,5", 2024-02-05T06:51:06.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.439110+0000", 2024-02-05T06:51:06.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014008099999999999 2024-02-05T06:51:06.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.385 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.385 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.385 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.446154+0000", 2024-02-05T06:51:06.385 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0070444679999999999 2024-02-05T06:51:06.385 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:06.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.446168+0000", 2024-02-05T06:51:06.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3574000000000001e-05 2024-02-05T06:51:06.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.462516+0000", 2024-02-05T06:51:06.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.01634797 2024-02-05T06:51:06.386 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.386 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.386 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:06.386 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.462572+0000", 2024-02-05T06:51:06.386 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6656000000000001e-05 2024-02-05T06:51:06.387 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.387 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:06.387 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.387 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.387 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.387 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:140530 8.2 8:471944ce:::benchmark_data_smithi028_105698_object51255:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-02-05T06:51:06.387 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:23.459861+0000", 2024-02-05T06:51:06.387 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 522.89601183699995, 2024-02-05T06:51:06.387 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0118483190000001, 2024-02-05T06:51:06.387 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:06.387 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:51:06.387 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:06.387 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:51:06.387 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:51:06.388 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 140530 2024-02-05T06:51:06.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.388 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:06.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:06.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.459861+0000", 2024-02-05T06:51:06.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.388 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.388 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:06.388 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.459861+0000", 2024-02-05T06:51:06.388 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.388 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:06.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.459864+0000", 2024-02-05T06:51:06.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3159999999999999e-06 2024-02-05T06:51:06.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:06.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.459870+0000", 2024-02-05T06:51:06.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1279999999999996e-06 2024-02-05T06:51:06.389 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.389 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.389 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:06.389 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.459882+0000", 2024-02-05T06:51:06.389 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2619000000000001e-05 2024-02-05T06:51:06.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:06.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.459892+0000", 2024-02-05T06:51:06.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.944e-06 2024-02-05T06:51:06.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.459955+0000", 2024-02-05T06:51:06.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2346000000000002e-05 2024-02-05T06:51:06.390 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.390 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.390 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:06.390 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.459966+0000", 2024-02-05T06:51:06.390 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1378e-05 2024-02-05T06:51:06.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.462603+0000", 2024-02-05T06:51:06.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0026372589999999 2024-02-05T06:51:06.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:06.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.462706+0000", 2024-02-05T06:51:06.391 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010285000000000001 2024-02-05T06:51:06.391 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.391 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.391 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:51:06.391 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.462827+0000", 2024-02-05T06:51:06.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120946 2024-02-05T06:51:06.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:06.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.471480+0000", 2024-02-05T06:51:06.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0086533749999999996 2024-02-05T06:51:06.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:51:06.392 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.471499+0000", 2024-02-05T06:51:06.392 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8510999999999999e-05 2024-02-05T06:51:06.392 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.392 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.392 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:06.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.471611+0000", 2024-02-05T06:51:06.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000111828 2024-02-05T06:51:06.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:51:06.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.471661+0000", 2024-02-05T06:51:06.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0275000000000002e-05 2024-02-05T06:51:06.393 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.393 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.393 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:06.393 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.471710+0000", 2024-02-05T06:51:06.393 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8544000000000001e-05 2024-02-05T06:51:06.393 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.393 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:06.394 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.394 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.394 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:140538 8.2 8:4be797cf:::benchmark_data_smithi028_105698_object51263:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-02-05T06:51:06.394 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:23.464208+0000", 2024-02-05T06:51:06.394 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 522.89166556600003, 2024-02-05T06:51:06.394 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.007671134, 2024-02-05T06:51:06.394 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:06.394 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:51:06.394 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:06.394 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:51:06.394 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:51:06.394 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 140538 2024-02-05T06:51:06.394 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.395 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:06.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:06.395 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.464208+0000", 2024-02-05T06:51:06.395 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.395 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.395 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.395 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:06.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.464208+0000", 2024-02-05T06:51:06.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.397 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.397 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.397 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:06.397 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.464210+0000", 2024-02-05T06:51:06.397 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2199999999999999e-06 2024-02-05T06:51:06.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.398 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:06.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.464218+0000", 2024-02-05T06:51:06.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5750000000000001e-06 2024-02-05T06:51:06.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.398 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:06.398 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.464233+0000", 2024-02-05T06:51:06.398 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4323e-05 2024-02-05T06:51:06.398 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.398 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:06.399 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.464241+0000", 2024-02-05T06:51:06.399 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1300000000000001e-06 2024-02-05T06:51:06.399 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.399 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.399 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.400 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.464377+0000", 2024-02-05T06:51:06.400 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000136315 2024-02-05T06:51:06.400 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.400 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.400 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:06.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.464385+0000", 2024-02-05T06:51:06.401 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1189999999999997e-06 2024-02-05T06:51:06.401 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.401 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.401 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.401 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.465231+0000", 2024-02-05T06:51:06.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0008459249999999 2024-02-05T06:51:06.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:06.402 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.465412+0000", 2024-02-05T06:51:06.402 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000181211 2024-02-05T06:51:06.402 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.402 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.402 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:51:06.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.465532+0000", 2024-02-05T06:51:06.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119981 2024-02-05T06:51:06.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.403 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:51:06.403 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.471518+0000", 2024-02-05T06:51:06.403 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059857690000000002 2024-02-05T06:51:06.403 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.403 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:06.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.471560+0000", 2024-02-05T06:51:06.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2061000000000003e-05 2024-02-05T06:51:06.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:06.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.471782+0000", 2024-02-05T06:51:06.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022130699999999999 2024-02-05T06:51:06.404 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.404 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.404 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:51:06.404 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.471830+0000", 2024-02-05T06:51:06.404 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8029999999999999e-05 2024-02-05T06:51:06.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.405 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:06.405 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.471879+0000", 2024-02-05T06:51:06.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9168000000000001e-05 2024-02-05T06:51:06.405 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.405 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:06.405 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.405 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.405 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.405 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:140552 8.2 8:4e9a9379:::benchmark_data_smithi028_105698_object51277:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-02-05T06:51:06.405 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:23.477918+0000", 2024-02-05T06:51:06.405 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 522.87795545799997, 2024-02-05T06:51:06.405 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99409492099999996, 2024-02-05T06:51:06.406 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:06.406 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:51:06.406 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:06.406 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:51:06.406 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:51:06.406 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 140552 2024-02-05T06:51:06.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.406 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:06.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.406 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:06.406 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.477918+0000", 2024-02-05T06:51:06.406 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.406 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.406 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.406 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:06.407 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.477918+0000", 2024-02-05T06:51:06.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.407 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:06.407 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.477919+0000", 2024-02-05T06:51:06.407 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7570000000000001e-06 2024-02-05T06:51:06.407 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.407 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.407 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:06.407 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.477925+0000", 2024-02-05T06:51:06.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3240000000000002e-06 2024-02-05T06:51:06.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:06.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.477936+0000", 2024-02-05T06:51:06.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1384e-05 2024-02-05T06:51:06.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.408 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:06.408 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.477943+0000", 2024-02-05T06:51:06.408 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.742e-06 2024-02-05T06:51:06.408 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.408 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.477990+0000", 2024-02-05T06:51:06.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7601e-05 2024-02-05T06:51:06.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:06.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.477998+0000", 2024-02-05T06:51:06.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2919999999999998e-06 2024-02-05T06:51:06.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.409 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.409 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.409 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.468484+0000", 2024-02-05T06:51:06.409 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.99048594099999998 2024-02-05T06:51:06.409 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:06.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.468668+0000", 2024-02-05T06:51:06.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000183936 2024-02-05T06:51:06.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:51:06.410 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.468783+0000", 2024-02-05T06:51:06.410 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000115238 2024-02-05T06:51:06.410 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.410 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.410 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:06.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.471747+0000", 2024-02-05T06:51:06.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0029645230000000002 2024-02-05T06:51:06.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:06.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.471923+0000", 2024-02-05T06:51:06.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000175444 2024-02-05T06:51:06.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.411 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:51:06.411 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.471933+0000", 2024-02-05T06:51:06.411 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8959999999999998e-06 2024-02-05T06:51:06.411 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.411 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:51:06.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.471970+0000", 2024-02-05T06:51:06.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6869000000000002e-05 2024-02-05T06:51:06.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.412 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:06.412 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.472013+0000", 2024-02-05T06:51:06.412 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2973999999999999e-05 2024-02-05T06:51:06.412 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.412 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:06.412 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.412 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.412 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.412 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:140582 8.2 8:46601042:::benchmark_data_smithi028_105698_object51307:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-02-05T06:51:06.413 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:23.502792+0000", 2024-02-05T06:51:06.413 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 522.85308137699997, 2024-02-05T06:51:06.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.98169489700000001, 2024-02-05T06:51:06.413 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:06.413 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:51:06.413 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:06.413 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:51:06.413 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:51:06.413 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 140582 2024-02-05T06:51:06.413 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.413 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:06.413 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.413 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:06.413 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.502792+0000", 2024-02-05T06:51:06.413 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:06.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.502792+0000", 2024-02-05T06:51:06.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:06.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.502793+0000", 2024-02-05T06:51:06.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.671e-06 2024-02-05T06:51:06.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.414 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.414 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:06.414 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.502798+0000", 2024-02-05T06:51:06.414 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6539999999999998e-06 2024-02-05T06:51:06.414 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:06.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.502809+0000", 2024-02-05T06:51:06.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1389999999999999e-05 2024-02-05T06:51:06.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:06.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.502817+0000", 2024-02-05T06:51:06.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8790000000000003e-06 2024-02-05T06:51:06.415 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.415 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.415 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.415 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.502872+0000", 2024-02-05T06:51:06.415 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5074000000000002e-05 2024-02-05T06:51:06.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:06.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.502880+0000", 2024-02-05T06:51:06.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.554e-06 2024-02-05T06:51:06.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.416 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.472053+0000", 2024-02-05T06:51:06.416 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96917274899999994 2024-02-05T06:51:06.416 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.416 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.416 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:06.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.472160+0000", 2024-02-05T06:51:06.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010762199999999999 2024-02-05T06:51:06.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:51:06.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.472267+0000", 2024-02-05T06:51:06.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010699799999999999 2024-02-05T06:51:06.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.417 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:06.417 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.479955+0000", 2024-02-05T06:51:06.417 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0076878320000000003 2024-02-05T06:51:06.417 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.417 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:06.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.479992+0000", 2024-02-05T06:51:06.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6665999999999998e-05 2024-02-05T06:51:06.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:51:06.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.484355+0000", 2024-02-05T06:51:06.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0043633669999999999 2024-02-05T06:51:06.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.418 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.418 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:51:06.418 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.484442+0000", 2024-02-05T06:51:06.418 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7175000000000005e-05 2024-02-05T06:51:06.418 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.419 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:06.419 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.484487+0000", 2024-02-05T06:51:06.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4265999999999999e-05 2024-02-05T06:51:06.419 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.419 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:06.419 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.419 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.419 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.419 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:140590 8.2 8:42182456:::benchmark_data_smithi028_105698_object51315:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-02-05T06:51:06.419 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:23.507529+0000", 2024-02-05T06:51:06.419 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 522.84834454600002, 2024-02-05T06:51:06.419 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.97717015399999996, 2024-02-05T06:51:06.419 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:06.420 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:51:06.420 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:06.420 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:51:06.420 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:51:06.420 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 140590 2024-02-05T06:51:06.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.420 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:06.420 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.420 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:06.420 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.507529+0000", 2024-02-05T06:51:06.420 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.420 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:06.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.507529+0000", 2024-02-05T06:51:06.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:06.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.507531+0000", 2024-02-05T06:51:06.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4549999999999998e-06 2024-02-05T06:51:06.421 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.421 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.421 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:06.421 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.507539+0000", 2024-02-05T06:51:06.421 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.763e-06 2024-02-05T06:51:06.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:06.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.507557+0000", 2024-02-05T06:51:06.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7884e-05 2024-02-05T06:51:06.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:06.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.507567+0000", 2024-02-05T06:51:06.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0524e-05 2024-02-05T06:51:06.422 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.422 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.422 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.422 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.507639+0000", 2024-02-05T06:51:06.422 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1619000000000007e-05 2024-02-05T06:51:06.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:06.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.507649+0000", 2024-02-05T06:51:06.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9729999999999992e-06 2024-02-05T06:51:06.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.474719+0000", 2024-02-05T06:51:06.423 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96707034000000003 2024-02-05T06:51:06.423 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.423 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.423 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:06.423 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.474844+0000", 2024-02-05T06:51:06.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012465100000000001 2024-02-05T06:51:06.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:51:06.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.474962+0000", 2024-02-05T06:51:06.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118009 2024-02-05T06:51:06.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.424 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:06.424 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.480026+0000", 2024-02-05T06:51:06.424 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0050638699999999998 2024-02-05T06:51:06.424 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.424 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:06.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.480060+0000", 2024-02-05T06:51:06.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3849000000000001e-05 2024-02-05T06:51:06.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:51:06.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.484624+0000", 2024-02-05T06:51:06.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0045649489999999996 2024-02-05T06:51:06.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.425 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.425 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:51:06.425 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.484658+0000", 2024-02-05T06:51:06.425 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3340000000000003e-05 2024-02-05T06:51:06.425 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.426 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:06.426 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.484699+0000", 2024-02-05T06:51:06.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0927999999999998e-05 2024-02-05T06:51:06.426 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.426 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:06.426 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.426 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.426 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.426 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:140609 8.2 8:465732ac:::benchmark_data_smithi028_105698_object51334:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-02-05T06:51:06.426 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:23.527955+0000", 2024-02-05T06:51:06.426 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 522.82791816600002, 2024-02-05T06:51:06.426 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.95684319500000004, 2024-02-05T06:51:06.426 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:06.427 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:51:06.427 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:06.427 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:51:06.427 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:51:06.427 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 140609 2024-02-05T06:51:06.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.427 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:06.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:06.427 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.527955+0000", 2024-02-05T06:51:06.427 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.427 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.427 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.427 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:06.428 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.527955+0000", 2024-02-05T06:51:06.428 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.428 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.428 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:06.428 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.527957+0000", 2024-02-05T06:51:06.428 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9120000000000001e-06 2024-02-05T06:51:06.428 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.428 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:06.428 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.527963+0000", 2024-02-05T06:51:06.428 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.801e-06 2024-02-05T06:51:06.428 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.428 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.429 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:06.429 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.527976+0000", 2024-02-05T06:51:06.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3478e-05 2024-02-05T06:51:06.429 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.429 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:06.429 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.527986+0000", 2024-02-05T06:51:06.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5820000000000008e-06 2024-02-05T06:51:06.429 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.429 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.429 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.429 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.528050+0000", 2024-02-05T06:51:06.429 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4635999999999996e-05 2024-02-05T06:51:06.429 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:06.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.528060+0000", 2024-02-05T06:51:06.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0112000000000001e-05 2024-02-05T06:51:06.430 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.477396+0000", 2024-02-05T06:51:06.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.94933583099999996 2024-02-05T06:51:06.430 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.430 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.430 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:06.430 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.477496+0000", 2024-02-05T06:51:06.430 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000100043 2024-02-05T06:51:06.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:51:06.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.477616+0000", 2024-02-05T06:51:06.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000119469 2024-02-05T06:51:06.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:06.431 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.480095+0000", 2024-02-05T06:51:06.431 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002478924 2024-02-05T06:51:06.431 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.431 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.431 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:06.432 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.484553+0000", 2024-02-05T06:51:06.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0044581530000000003 2024-02-05T06:51:06.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.432 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.432 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:51:06.432 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.484713+0000", 2024-02-05T06:51:06.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000159813 2024-02-05T06:51:06.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.432 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.432 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:51:06.432 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.484764+0000", 2024-02-05T06:51:06.432 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0933999999999997e-05 2024-02-05T06:51:06.432 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.433 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.433 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:06.433 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.484798+0000", 2024-02-05T06:51:06.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4507e-05 2024-02-05T06:51:06.433 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.433 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:06.433 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.433 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.433 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.433 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:140616 8.2 8:4eaf74fd:::benchmark_data_smithi028_105698_object51341:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-02-05T06:51:06.433 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:23.533483+0000", 2024-02-05T06:51:06.433 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 522.82238988500001, 2024-02-05T06:51:06.433 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.95640482500000001, 2024-02-05T06:51:06.433 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:06.434 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:51:06.434 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:06.434 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:51:06.434 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:51:06.434 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 140616 2024-02-05T06:51:06.434 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.434 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:06.434 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.434 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:06.434 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.533483+0000", 2024-02-05T06:51:06.434 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.434 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.434 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.434 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:06.434 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.533483+0000", 2024-02-05T06:51:06.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.435 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.435 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:06.435 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.533486+0000", 2024-02-05T06:51:06.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7750000000000001e-06 2024-02-05T06:51:06.435 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.435 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:06.435 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.533494+0000", 2024-02-05T06:51:06.435 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2099999999999993e-06 2024-02-05T06:51:06.435 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.435 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.435 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:06.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.533513+0000", 2024-02-05T06:51:06.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9057999999999999e-05 2024-02-05T06:51:06.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:06.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.533523+0000", 2024-02-05T06:51:06.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0159999999999999e-05 2024-02-05T06:51:06.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.436 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.436 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.533601+0000", 2024-02-05T06:51:06.436 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7757999999999997e-05 2024-02-05T06:51:06.436 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.436 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:06.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.533612+0000", 2024-02-05T06:51:06.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1073000000000001e-05 2024-02-05T06:51:06.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.480122+0000", 2024-02-05T06:51:06.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.94650959300000004 2024-02-05T06:51:06.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.437 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.437 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:06.437 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.480230+0000", 2024-02-05T06:51:06.437 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000107817 2024-02-05T06:51:06.437 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:51:06.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.480343+0000", 2024-02-05T06:51:06.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113242 2024-02-05T06:51:06.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:06.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.484609+0000", 2024-02-05T06:51:06.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0042665589999999996 2024-02-05T06:51:06.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.438 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.438 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:51:06.438 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.484806+0000", 2024-02-05T06:51:06.438 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000196348 2024-02-05T06:51:06.438 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:06.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.489781+0000", 2024-02-05T06:51:06.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049750039999999999 2024-02-05T06:51:06.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:51:06.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.489830+0000", 2024-02-05T06:51:06.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8881000000000002e-05 2024-02-05T06:51:06.439 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.439 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.439 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:06.439 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.489888+0000", 2024-02-05T06:51:06.439 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8347000000000002e-05 2024-02-05T06:51:06.439 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.440 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:06.440 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.440 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.440 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.440 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:140644 8.2 8:40d608db:::benchmark_data_smithi028_105698_object51369:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-02-05T06:51:06.440 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:23.560295+0000", 2024-02-05T06:51:06.440 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 522.79557786500004, 2024-02-05T06:51:06.440 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.92973749400000005, 2024-02-05T06:51:06.440 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:06.440 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:51:06.440 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:06.440 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:51:06.440 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:51:06.441 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 140644 2024-02-05T06:51:06.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.441 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:06.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.441 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:06.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.560295+0000", 2024-02-05T06:51:06.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.441 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:06.441 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.560295+0000", 2024-02-05T06:51:06.441 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.441 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.441 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.441 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:06.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.560297+0000", 2024-02-05T06:51:06.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.751e-06 2024-02-05T06:51:06.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:06.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.560302+0000", 2024-02-05T06:51:06.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3240000000000002e-06 2024-02-05T06:51:06.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.442 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:06.442 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.560313+0000", 2024-02-05T06:51:06.442 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0715e-05 2024-02-05T06:51:06.442 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.442 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:06.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.560322+0000", 2024-02-05T06:51:06.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2119999999999992e-06 2024-02-05T06:51:06.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.560384+0000", 2024-02-05T06:51:06.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1716000000000003e-05 2024-02-05T06:51:06.443 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.443 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.443 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:06.443 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.560392+0000", 2024-02-05T06:51:06.443 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7989999999999994e-06 2024-02-05T06:51:06.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.482277+0000", 2024-02-05T06:51:06.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.92188496900000005 2024-02-05T06:51:06.444 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.444 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.444 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:06.444 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.482397+0000", 2024-02-05T06:51:06.444 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120011 2024-02-05T06:51:06.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:51:06.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.482510+0000", 2024-02-05T06:51:06.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113224 2024-02-05T06:51:06.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:51:06.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.484813+0000", 2024-02-05T06:51:06.445 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0023028620000000001 2024-02-05T06:51:06.445 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.445 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.445 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:06.445 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.489744+0000", 2024-02-05T06:51:06.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.004931225 2024-02-05T06:51:06.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:06.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.489960+0000", 2024-02-05T06:51:06.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000215443 2024-02-05T06:51:06.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.446 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.446 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:51:06.446 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.490001+0000", 2024-02-05T06:51:06.446 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1946000000000001e-05 2024-02-05T06:51:06.446 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.447 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:06.447 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.490033+0000", 2024-02-05T06:51:06.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1297000000000002e-05 2024-02-05T06:51:06.447 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.447 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:06.447 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.447 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.447 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.447 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:140650 8.2 8:4fcc0ff8:::benchmark_data_smithi028_105698_object51375:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-02-05T06:51:06.447 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:23.568610+0000", 2024-02-05T06:51:06.447 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 522.78726307500006, 2024-02-05T06:51:06.447 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.92920546400000004, 2024-02-05T06:51:06.447 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:06.447 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:51:06.448 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:06.448 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:51:06.448 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:51:06.448 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 140650 2024-02-05T06:51:06.448 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.448 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:06.448 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.448 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:06.448 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.568610+0000", 2024-02-05T06:51:06.448 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.449 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:06.449 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.568610+0000", 2024-02-05T06:51:06.449 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.449 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.449 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.449 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:06.449 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.568612+0000", 2024-02-05T06:51:06.449 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.857e-06 2024-02-05T06:51:06.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:06.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.568617+0000", 2024-02-05T06:51:06.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7430000000000002e-06 2024-02-05T06:51:06.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.450 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:06.450 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.568635+0000", 2024-02-05T06:51:06.450 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8253999999999998e-05 2024-02-05T06:51:06.450 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.450 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:06.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.568644+0000", 2024-02-05T06:51:06.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1970000000000002e-06 2024-02-05T06:51:06.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.568707+0000", 2024-02-05T06:51:06.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3250000000000006e-05 2024-02-05T06:51:06.451 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.451 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.451 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:06.451 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.568716+0000", 2024-02-05T06:51:06.451 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4149999999999999e-06 2024-02-05T06:51:06.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.484867+0000", 2024-02-05T06:51:06.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.91615160900000003 2024-02-05T06:51:06.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:06.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.484986+0000", 2024-02-05T06:51:06.452 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000118678 2024-02-05T06:51:06.452 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.452 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.452 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:51:06.452 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.485112+0000", 2024-02-05T06:51:06.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012614700000000001 2024-02-05T06:51:06.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:06.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.489926+0000", 2024-02-05T06:51:06.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048142360000000004 2024-02-05T06:51:06.453 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.453 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.453 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:06.453 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.490065+0000", 2024-02-05T06:51:06.453 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013887599999999999 2024-02-05T06:51:06.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:51:06.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.497762+0000", 2024-02-05T06:51:06.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0076970340000000002 2024-02-05T06:51:06.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:51:06.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.497791+0000", 2024-02-05T06:51:06.454 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8924e-05 2024-02-05T06:51:06.454 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.454 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.454 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:06.454 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.497816+0000", 2024-02-05T06:51:06.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4244e-05 2024-02-05T06:51:06.455 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.455 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:06.455 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.455 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.455 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.455 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:140674 8.2 8:4359ee71:::benchmark_data_smithi028_105698_object51399:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e267)", 2024-02-05T06:51:06.455 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:23.599256+0000", 2024-02-05T06:51:06.455 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 522.75661670299996, 2024-02-05T06:51:06.455 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.92447017799999998, 2024-02-05T06:51:06.455 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:06.455 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:51:06.455 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:06.455 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:51:06.455 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:51:06.456 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 140674 2024-02-05T06:51:06.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.456 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:06.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:06.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.599256+0000", 2024-02-05T06:51:06.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.456 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.456 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.456 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:06.456 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.599256+0000", 2024-02-05T06:51:06.456 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:06.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.599259+0000", 2024-02-05T06:51:06.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.627e-06 2024-02-05T06:51:06.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:06.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.599265+0000", 2024-02-05T06:51:06.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1129999999999997e-06 2024-02-05T06:51:06.457 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.457 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.457 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:06.457 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.599279+0000", 2024-02-05T06:51:06.457 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3539e-05 2024-02-05T06:51:06.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:06.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.599289+0000", 2024-02-05T06:51:06.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.8770000000000005e-06 2024-02-05T06:51:06.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.458 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.458 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.458 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.599348+0000", 2024-02-05T06:51:06.458 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9398000000000003e-05 2024-02-05T06:51:06.458 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:06.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:23.599355+0000", 2024-02-05T06:51:06.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.233e-06 2024-02-05T06:51:06.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.459 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.459 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.459 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.490128+0000", 2024-02-05T06:51:06.459 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.89077234100000002 2024-02-05T06:51:06.459 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:06.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.490221+0000", 2024-02-05T06:51:06.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3178999999999997e-05 2024-02-05T06:51:06.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,5", 2024-02-05T06:51:06.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.490320+0000", 2024-02-05T06:51:06.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9621e-05 2024-02-05T06:51:06.460 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.460 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.460 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:51:06.460 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.497825+0000", 2024-02-05T06:51:06.460 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0075050489999999997 2024-02-05T06:51:06.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:06.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.497956+0000", 2024-02-05T06:51:06.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013080399999999999 2024-02-05T06:51:06.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:06.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.523657+0000", 2024-02-05T06:51:06.461 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025700771000000001 2024-02-05T06:51:06.461 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.461 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.461 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:51:06.461 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.523703+0000", 2024-02-05T06:51:06.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5775999999999997e-05 2024-02-05T06:51:06.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.462 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:06.462 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:24.523727+0000", 2024-02-05T06:51:06.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.385e-05 2024-02-05T06:51:06.462 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.462 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:06.462 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.462 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.462 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.462 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.45587.0:209921 10.c e408/345 10:329d43f6:::benchmark_data_smithi028_154709_object88920:head v 408'12862, mlcod=408'12860)", 2024-02-05T06:51:06.462 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:50:33.136656+0000", 2024-02-05T06:51:06.462 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 33.219217432999997, 2024-02-05T06:51:06.462 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035684118000000001, 2024-02-05T06:51:06.463 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:06.463 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:51:06.463 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:06.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.463 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:06.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.136656+0000", 2024-02-05T06:51:06.463 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.463 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.463 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.463 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:06.463 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.136656+0000", 2024-02-05T06:51:06.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:06.464 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.136657+0000", 2024-02-05T06:51:06.464 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.367e-06 2024-02-05T06:51:06.464 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.464 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.464 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:06.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.136664+0000", 2024-02-05T06:51:06.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9859999999999997e-06 2024-02-05T06:51:06.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:06.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.136675+0000", 2024-02-05T06:51:06.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0749e-05 2024-02-05T06:51:06.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:06.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.136682+0000", 2024-02-05T06:51:06.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5619999999999998e-06 2024-02-05T06:51:06.465 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.465 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.465 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.465 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.136747+0000", 2024-02-05T06:51:06.465 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5055000000000006e-05 2024-02-05T06:51:06.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:06.466 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.137416+0000", 2024-02-05T06:51:06.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00066860599999999997 2024-02-05T06:51:06.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:51:06.466 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.172293+0000", 2024-02-05T06:51:06.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.034877189000000003 2024-02-05T06:51:06.466 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.466 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.466 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:06.466 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.172340+0000", 2024-02-05T06:51:06.466 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6604000000000003e-05 2024-02-05T06:51:06.467 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.467 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:06.467 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.467 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.45587.0:209929 10.e e408/345 10:77891d7c:::benchmark_data_smithi028_154709_object88928:head v 408'13330, mlcod=408'13328)", 2024-02-05T06:51:06.467 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:50:33.144148+0000", 2024-02-05T06:51:06.467 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 33.211725205, 2024-02-05T06:51:06.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.034022338999999999, 2024-02-05T06:51:06.467 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:06.467 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:51:06.467 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:06.467 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.467 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:06.467 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.144148+0000", 2024-02-05T06:51:06.467 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.467 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.468 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:06.468 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.144148+0000", 2024-02-05T06:51:06.468 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.468 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.468 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:06.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.144149+0000", 2024-02-05T06:51:06.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.392e-06 2024-02-05T06:51:06.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:06.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.144158+0000", 2024-02-05T06:51:06.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5080000000000005e-06 2024-02-05T06:51:06.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.472 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.472 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:06.472 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.144167+0000", 2024-02-05T06:51:06.472 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1870000000000003e-06 2024-02-05T06:51:06.472 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:06.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.144174+0000", 2024-02-05T06:51:06.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.182e-06 2024-02-05T06:51:06.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.144258+0000", 2024-02-05T06:51:06.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4027e-05 2024-02-05T06:51:06.473 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.473 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.473 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:06.473 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.144279+0000", 2024-02-05T06:51:06.473 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0925999999999999e-05 2024-02-05T06:51:06.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:51:06.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.177334+0000", 2024-02-05T06:51:06.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.033054596999999998 2024-02-05T06:51:06.474 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.474 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.474 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:06.474 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.178170+0000", 2024-02-05T06:51:06.474 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00083651999999999999 2024-02-05T06:51:06.474 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.474 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:06.475 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.475 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.45587.0:209933 10.e e408/345 10:7ff443b1:::benchmark_data_smithi028_154709_object88932:head v 408'13331, mlcod=408'13328)", 2024-02-05T06:51:06.475 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:50:33.145780+0000", 2024-02-05T06:51:06.475 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 33.210093092000001, 2024-02-05T06:51:06.475 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.034673046999999999, 2024-02-05T06:51:06.475 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:06.475 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:51:06.475 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:06.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.475 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:06.475 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.145780+0000", 2024-02-05T06:51:06.475 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.475 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.475 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:06.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.145780+0000", 2024-02-05T06:51:06.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:06.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.145781+0000", 2024-02-05T06:51:06.476 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.15e-06 2024-02-05T06:51:06.476 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.476 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.476 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:06.476 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.145788+0000", 2024-02-05T06:51:06.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2099999999999996e-06 2024-02-05T06:51:06.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:06.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.172299+0000", 2024-02-05T06:51:06.477 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.026510617 2024-02-05T06:51:06.477 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.477 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.477 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:06.477 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.172311+0000", 2024-02-05T06:51:06.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1578e-05 2024-02-05T06:51:06.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.478 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.478 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.174930+0000", 2024-02-05T06:51:06.478 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00261911 2024-02-05T06:51:06.478 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.478 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.478 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:06.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.174949+0000", 2024-02-05T06:51:06.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9046e-05 2024-02-05T06:51:06.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.479 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:51:06.479 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.180415+0000", 2024-02-05T06:51:06.479 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0054665069999999998 2024-02-05T06:51:06.479 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.479 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.480 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:06.480 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.180453+0000", 2024-02-05T06:51:06.480 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7829e-05 2024-02-05T06:51:06.480 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.480 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:06.480 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.480 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.481 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.481 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.45587.0:209938 10.c e408/345 10:379d7ec8:::benchmark_data_smithi028_154709_object88937:head v 408'12864, mlcod=408'12860)", 2024-02-05T06:51:06.481 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:50:33.147866+0000", 2024-02-05T06:51:06.481 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 33.208006975000004, 2024-02-05T06:51:06.481 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.030309638999999999, 2024-02-05T06:51:06.481 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:06.481 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:51:06.481 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:06.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:06.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.147866+0000", 2024-02-05T06:51:06.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.482 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.482 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.482 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:06.482 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.147866+0000", 2024-02-05T06:51:06.482 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:06.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.147868+0000", 2024-02-05T06:51:06.483 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.595e-06 2024-02-05T06:51:06.483 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.483 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.483 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:06.483 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.147876+0000", 2024-02-05T06:51:06.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5059999999999995e-06 2024-02-05T06:51:06.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.484 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:06.484 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.172293+0000", 2024-02-05T06:51:06.484 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024416473000000001 2024-02-05T06:51:06.484 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.484 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:06.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.172306+0000", 2024-02-05T06:51:06.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.363e-05 2024-02-05T06:51:06.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.485 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.485 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.485 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.172484+0000", 2024-02-05T06:51:06.485 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017794800000000001 2024-02-05T06:51:06.485 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:06.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.173493+0000", 2024-02-05T06:51:06.486 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010082489999999999 2024-02-05T06:51:06.486 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.486 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.486 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:51:06.486 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.178147+0000", 2024-02-05T06:51:06.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0046546130000000002 2024-02-05T06:51:06.487 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.487 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.487 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:06.487 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.178176+0000", 2024-02-05T06:51:06.487 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8625000000000001e-05 2024-02-05T06:51:06.487 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.487 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:06.487 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.488 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.488 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.488 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.45587.0:210825 10.c e408/345 10:3727982e:::benchmark_data_smithi028_154709_object89824:head v 408'12922, mlcod=408'12917)", 2024-02-05T06:51:06.488 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:50:33.853962+0000", 2024-02-05T06:51:06.488 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 32.501911544000002, 2024-02-05T06:51:06.488 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.034219205000000003, 2024-02-05T06:51:06.488 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:06.488 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:51:06.489 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:06.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.489 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:06.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.853962+0000", 2024-02-05T06:51:06.489 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.489 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.489 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.489 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:06.489 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.853962+0000", 2024-02-05T06:51:06.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.490 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:06.490 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.853963+0000", 2024-02-05T06:51:06.490 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.051e-06 2024-02-05T06:51:06.490 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.490 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:06.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.853976+0000", 2024-02-05T06:51:06.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3429e-05 2024-02-05T06:51:06.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:06.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.853990+0000", 2024-02-05T06:51:06.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4197e-05 2024-02-05T06:51:06.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.491 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.491 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:06.491 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.853998+0000", 2024-02-05T06:51:06.491 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2690000000000002e-06 2024-02-05T06:51:06.491 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.861481+0000", 2024-02-05T06:51:06.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0074829839999999998 2024-02-05T06:51:06.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:06.492 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.861498+0000", 2024-02-05T06:51:06.492 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.783e-05 2024-02-05T06:51:06.492 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.492 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.492 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:51:06.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.888090+0000", 2024-02-05T06:51:06.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.02659166 2024-02-05T06:51:06.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.493 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:06.493 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.888181+0000", 2024-02-05T06:51:06.493 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0785000000000006e-05 2024-02-05T06:51:06.493 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.493 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:06.493 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.493 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.493 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.494 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:06.495 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.45587.0:210829 10.c e408/345 10:3c1f6ab4:::benchmark_data_smithi028_154709_object89828:head v 408'12923, mlcod=408'12917)", 2024-02-05T06:51:06.495 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:50:33.860507+0000", 2024-02-05T06:51:06.495 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 32.495366054999998, 2024-02-05T06:51:06.495 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.027722588999999999, 2024-02-05T06:51:06.495 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:06.495 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:51:06.495 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:06.495 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.495 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:06.495 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.860507+0000", 2024-02-05T06:51:06.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:06.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.860507+0000", 2024-02-05T06:51:06.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.496 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:06.496 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.860509+0000", 2024-02-05T06:51:06.496 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.435e-06 2024-02-05T06:51:06.496 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.496 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:06.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.860517+0000", 2024-02-05T06:51:06.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1119999999999996e-06 2024-02-05T06:51:06.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:06.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.860531+0000", 2024-02-05T06:51:06.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4411e-05 2024-02-05T06:51:06.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.497 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:06.497 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.860539+0000", 2024-02-05T06:51:06.497 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9379999999999992e-06 2024-02-05T06:51:06.497 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.497 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.863982+0000", 2024-02-05T06:51:06.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0034430179999999999 2024-02-05T06:51:06.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:06.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.864027+0000", 2024-02-05T06:51:06.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5321999999999997e-05 2024-02-05T06:51:06.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.498 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:51:06.498 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.888186+0000", 2024-02-05T06:51:06.498 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024158204999999999 2024-02-05T06:51:06.498 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.498 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.499 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:06.499 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.888230+0000", 2024-02-05T06:51:06.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4147999999999998e-05 2024-02-05T06:51:06.499 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.499 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:06.499 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.499 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.499 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.499 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.45587.0:239891 10.c e415/345 10:3a20df36:::benchmark_data_smithi028_154709_object118890:head v 415'14753, mlcod=415'14751)", 2024-02-05T06:51:06.499 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:50:59.122882+0000", 2024-02-05T06:51:06.499 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 7.2329913929999998, 2024-02-05T06:51:06.499 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.026260540999999998, 2024-02-05T06:51:06.499 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:06.499 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:51:06.499 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:06.499 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:06.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:59.122882+0000", 2024-02-05T06:51:06.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:06.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:59.122882+0000", 2024-02-05T06:51:06.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:06.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.500 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:06.500 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:59.122882+0000", 2024-02-05T06:51:06.500 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7000000000000005e-07 2024-02-05T06:51:06.500 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.500 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:06.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:59.122890+0000", 2024-02-05T06:51:06.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3590000000000002e-06 2024-02-05T06:51:06.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:06.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:59.122896+0000", 2024-02-05T06:51:06.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3319999999999997e-06 2024-02-05T06:51:06.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.501 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.501 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:06.501 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:59.122904+0000", 2024-02-05T06:51:06.501 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0839999999999992e-06 2024-02-05T06:51:06.501 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:06.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:59.126198+0000", 2024-02-05T06:51:06.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0032941789999999999 2024-02-05T06:51:06.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:06.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:59.126218+0000", 2024-02-05T06:51:06.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9984e-05 2024-02-05T06:51:06.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.502 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:51:06.502 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:59.148928+0000", 2024-02-05T06:51:06.502 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.022709507 2024-02-05T06:51:06.502 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:06.502 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:06.503 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:06.503 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:59.149142+0000", 2024-02-05T06:51:06.503 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00021452600000000001 2024-02-05T06:51:06.503 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.503 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:06.503 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.503 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:06.503 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:06.503 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:51:06.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:06.518 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:06.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:06.619 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:06.644 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:06.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:06.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:06.746 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:06.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:06.772 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:06.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:06.873 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:06.897 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:06.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:06.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:06.916 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:06 smithi167 ceph-mon[32774]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-05T06:51:06.936 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:06 smithi028 ceph-mon[29442]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-05T06:51:06.936 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:06 smithi028 ceph-mon[37984]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-02-05T06:51:06.999 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:07.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:07.024 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:07.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:07.125 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:07.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:07.150 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:07.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:07.250 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-05T06:51:07.252 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:51:07.274 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:07.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:07.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:07.376 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:07.399 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:07.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:07.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:07.492 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:51:07.501 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:07.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:07.527 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:07.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:07.628 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:07.653 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:07.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:07.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:07.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:07.769 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:07 smithi028 ceph-mon[29442]: pgmap v845: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 507 B/s rd, 0 op/s 2024-02-05T06:51:07.769 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:07 smithi028 ceph-mon[37984]: pgmap v845: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 507 B/s rd, 0 op/s 2024-02-05T06:51:07.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:07.780 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:07.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:07.865 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:07 smithi167 ceph-mon[32774]: pgmap v845: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 507 B/s rd, 0 op/s 2024-02-05T06:51:07.881 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:07.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:07.906 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:07.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:08.007 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:08.017 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:51:08.031 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:08.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:08.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:08.037 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:51:08.056 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:51:08.075 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:51:08.091 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 4, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T06:51:08.092 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T06:51:08.095 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:51:08.115 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:51:08.134 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:08.135 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:51:08.156 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:51:08.158 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:08.158 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:51:08.158 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:51:08.158 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:51:08.158 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:51:08.159 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:51:08.159 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:51:08.159 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:51:08.159 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:51:08.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:08.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:08.260 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:08.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:08.285 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:08.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:08.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:08.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:08.410 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:08.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:08.511 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:08.535 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:08.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:08.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:08.637 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:08.662 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:08.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:08.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:08.666 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-02-05T06:51:08.765 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:08.788 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:08.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:08.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:08.841 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:08 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2142982919' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:51:08.841 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:08 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1558409787' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:51:08.842 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:08 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2142982919' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:51:08.842 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:08 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1558409787' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:51:08.890 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:08.910 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:08 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2142982919' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:51:08.910 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:08 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1558409787' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:51:08.914 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:08.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:08.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:09.016 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:09.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:09.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:09.142 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:09.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:09.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:09.181 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:51:09.181 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:51:09.267 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:09.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:09.292 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:09.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:09.393 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:09.416 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:09.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:09.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:09.518 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:09.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:09.545 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:09.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:09.646 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:09.671 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:09.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:09.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:09.773 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:09.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:09.798 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:09.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:09.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:09.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:09.923 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:09.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:09.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:09 smithi028 ceph-mon[37984]: pgmap v846: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-05T06:51:09.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:09 smithi028 ceph-mon[37984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:51:09.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:09 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1558409787' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:51:09.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:09 smithi028 ceph-mon[37984]: osdmap e417: 8 total, 8 up, 8 in 2024-02-05T06:51:09.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:09 smithi028 ceph-mon[29442]: pgmap v846: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-05T06:51:09.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:09 smithi028 ceph-mon[29442]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:51:09.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:09 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1558409787' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:51:09.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:09 smithi028 ceph-mon[29442]: osdmap e417: 8 total, 8 up, 8 in 2024-02-05T06:51:09.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:09 smithi167 ceph-mon[32774]: pgmap v846: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-02-05T06:51:09.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:09 smithi167 ceph-mon[32774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:51:09.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:09 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1558409787' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:51:09.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:09 smithi167 ceph-mon[32774]: osdmap e417: 8 total, 8 up, 8 in 2024-02-05T06:51:10.025 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:10.048 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:10.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:10.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:10.150 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:10.175 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:10.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:10.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:10.217 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-05T06:51:10.277 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:10.323 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:10.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:10.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:10.425 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:10.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:10.450 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:10.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:10.502 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:51:10.550 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:10.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:10.575 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:10.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:10.676 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:10.700 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:10.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:10.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:10.801 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:10.825 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:10.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:10.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:10.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:51:10.927 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:10.952 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:10.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:10.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:11.054 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:11.078 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:11.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:11.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:11.180 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:11.206 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:11.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:11.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:11.308 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:11.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:11.333 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:11.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:11.434 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:11.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:11.460 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:11.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:11.464 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:51:11.562 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:11.585 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:11.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:11.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:11.587 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:11 smithi028 ceph-mon[29442]: pgmap v848: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-05T06:51:11.587 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:11 smithi028 ceph-mon[37984]: pgmap v848: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-05T06:51:11.687 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:11.711 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:11.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:11.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:11.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:11 smithi167 ceph-mon[32774]: pgmap v848: 142 pgs: 142 active+clean; 582 KiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-05T06:51:11.736 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T06:51:11.814 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:11.838 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:11.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:11.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:11.940 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:11.964 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:11.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:11.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:12.066 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:12.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:12.090 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:12.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:12.191 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:12.214 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:12.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:12.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:12.274 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'unique_pool_4' created 2024-02-05T06:51:12.316 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:12.339 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:12 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/429968088' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-02-05T06:51:12.340 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:12.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:12.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:12.442 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:12.464 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:51:12.465 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:51:12.465 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:12.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:12.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:12.537 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:12 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/429968088' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-02-05T06:51:12.537 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:12 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/429968088' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-02-05T06:51:12.537 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:51:12 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:51:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:51:12.567 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:12.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:12.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:12.693 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:12.717 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:12.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:12.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:12.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:12.847 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:12.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:12.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:12.949 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:12.972 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:12.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:12.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:13.074 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:13.101 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:13.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:13.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:13.148 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-05T06:51:13.203 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:13.230 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:13.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:13.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:13.332 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:13.353 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:13 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/429968088' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-02-05T06:51:13.353 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:13 smithi028 ceph-mon[29442]: osdmap e418: 8 total, 8 up, 8 in 2024-02-05T06:51:13.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:13 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:51:13.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:13 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:51:13.354 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:13 smithi028 ceph-mon[29442]: pgmap v850: 158 pgs: 3 creating+peering, 13 unknown, 142 active+clean; 582 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T06:51:13.354 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:13 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/429968088' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-02-05T06:51:13.354 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:13 smithi028 ceph-mon[37984]: osdmap e418: 8 total, 8 up, 8 in 2024-02-05T06:51:13.354 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:13 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:51:13.354 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:13 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:51:13.354 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:13 smithi028 ceph-mon[37984]: pgmap v850: 158 pgs: 3 creating+peering, 13 unknown, 142 active+clean; 582 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T06:51:13.356 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:13.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:13.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:13.379 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:13 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/429968088' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-02-05T06:51:13.379 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:13 smithi167 ceph-mon[32774]: osdmap e418: 8 total, 8 up, 8 in 2024-02-05T06:51:13.379 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:13 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:51:13.379 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:13 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:51:13.379 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:13 smithi167 ceph-mon[32774]: pgmap v850: 158 pgs: 3 creating+peering, 13 unknown, 142 active+clean; 582 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T06:51:13.409 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:51:13.458 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:13.471 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:51:13.480 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:51:13.483 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:13.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:13.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:13.504 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:51:13.523 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:51:13.543 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:51:13.563 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:51:13.584 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:51:13.585 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:13.606 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:51:13.608 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:13.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:13.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:13.626 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:51:13.627 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:51:13.627 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:51:13.627 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:51:13.628 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:51:13.628 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:51:13.628 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:51:13.628 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:51:13.628 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:51:13.710 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:13.717 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:51:13 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:51:13.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:13.737 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:13.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:13.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:13.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:13.867 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:13.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:13.968 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:13.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:13.993 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:13.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:14.080 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-02-05T06:51:14.094 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:14.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:14.119 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:14.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:14.220 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:14.244 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:14.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:14.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:14.348 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:14.371 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:14.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:14.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:14.430 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:14 smithi028 ceph-mon[37984]: osdmap e419: 8 total, 8 up, 8 in 2024-02-05T06:51:14.430 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:14 smithi028 ceph-mon[37984]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T06:51:14.430 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:14 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:51:14.431 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:14 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:51:14.431 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:14 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:51:14.431 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:14 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2852290441' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:51:14.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:14 smithi028 ceph-mon[29442]: osdmap e419: 8 total, 8 up, 8 in 2024-02-05T06:51:14.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:14 smithi028 ceph-mon[29442]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T06:51:14.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:14 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:51:14.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:14 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:51:14.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:14 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:51:14.431 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:14 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2852290441' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:51:14.473 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:14.486 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:14 smithi167 ceph-mon[32774]: osdmap e419: 8 total, 8 up, 8 in 2024-02-05T06:51:14.486 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:14 smithi167 ceph-mon[32774]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-05T06:51:14.486 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:14 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:51:14.486 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:14 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:51:14.486 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:14 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:51:14.486 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:14 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2852290441' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:51:14.496 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:14.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:14.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:14.598 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:14.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:14.623 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:14.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:14.724 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:14.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:14.751 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:14.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:14.852 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:14.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:14.878 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:14.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:14.979 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:15.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:15.003 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:15.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:15.104 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:15.127 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:15.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:15.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:15.133 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:51:15.134 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-02-05T06:51:15.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-02-05T06:51:15.134 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-02-05T06:51:15.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.134 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:129449 8.d 8:b9130c76:::benchmark_data_smithi028_105698_object40174:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e261)", 2024-02-05T06:51:15.134 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:10.059390+0000", 2024-02-05T06:51:15.134 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 545.06346950499994, 2024-02-05T06:51:15.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0677847279999999, 2024-02-05T06:51:15.134 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:15.134 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-02-05T06:51:15.134 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:15.134 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:51:15.134 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:51:15.135 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 129449 2024-02-05T06:51:15.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.135 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:15.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:15.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059390+0000", 2024-02-05T06:51:15.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:15.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059390+0000", 2024-02-05T06:51:15.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:15.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059392+0000", 2024-02-05T06:51:15.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.086e-06 2024-02-05T06:51:15.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:15.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059398+0000", 2024-02-05T06:51:15.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5589999999999996e-06 2024-02-05T06:51:15.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:15.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059410+0000", 2024-02-05T06:51:15.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1905e-05 2024-02-05T06:51:15.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:15.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059418+0000", 2024-02-05T06:51:15.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5599999999999994e-06 2024-02-05T06:51:15.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.060054+0000", 2024-02-05T06:51:15.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00063580799999999997 2024-02-05T06:51:15.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:15.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.060063+0000", 2024-02-05T06:51:15.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.346e-06 2024-02-05T06:51:15.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.093914+0000", 2024-02-05T06:51:15.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0338508449999999 2024-02-05T06:51:15.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-02-05T06:51:15.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.094022+0000", 2024-02-05T06:51:15.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000107959 2024-02-05T06:51:15.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.127068+0000", 2024-02-05T06:51:15.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.033046308000000003 2024-02-05T06:51:15.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.139 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:15.139 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.127175+0000", 2024-02-05T06:51:15.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000106352 2024-02-05T06:51:15.139 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.139 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:15.139 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.140 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:129452 8.d 8:b624ec5d:::benchmark_data_smithi028_105698_object40177:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e261)", 2024-02-05T06:51:15.140 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:10.059436+0000", 2024-02-05T06:51:15.140 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 545.06342323000001, 2024-02-05T06:51:15.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.074451045, 2024-02-05T06:51:15.140 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:15.140 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-02-05T06:51:15.140 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:15.140 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:51:15.140 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:51:15.140 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 129452 2024-02-05T06:51:15.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.140 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:15.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:15.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059436+0000", 2024-02-05T06:51:15.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:15.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059436+0000", 2024-02-05T06:51:15.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:15.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059437+0000", 2024-02-05T06:51:15.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7299999999999995e-07 2024-02-05T06:51:15.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:15.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059440+0000", 2024-02-05T06:51:15.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.897e-06 2024-02-05T06:51:15.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:15.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059446+0000", 2024-02-05T06:51:15.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1510000000000001e-06 2024-02-05T06:51:15.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:15.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.059452+0000", 2024-02-05T06:51:15.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0619999999999997e-06 2024-02-05T06:51:15.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.060094+0000", 2024-02-05T06:51:15.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00064214899999999995 2024-02-05T06:51:15.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:15.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:10.060101+0000", 2024-02-05T06:51:15.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5180000000000001e-06 2024-02-05T06:51:15.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.101188+0000", 2024-02-05T06:51:15.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0410871850000001 2024-02-05T06:51:15.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-02-05T06:51:15.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.101304+0000", 2024-02-05T06:51:15.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000116476 2024-02-05T06:51:15.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.133814+0000", 2024-02-05T06:51:15.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.032509405999999998 2024-02-05T06:51:15.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:15.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:11.133887+0000", 2024-02-05T06:51:15.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3528000000000005e-05 2024-02-05T06:51:15.145 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.145 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:15.145 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.145 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:168946 8.4 8:2a7577b5:::benchmark_data_smithi028_105698_object79671:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:51:15.145 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.072950+0000", 2024-02-05T06:51:15.145 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 505.04990958600001, 2024-02-05T06:51:15.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4796128150000001, 2024-02-05T06:51:15.145 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:15.145 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-02-05T06:51:15.145 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:15.145 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:51:15.146 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:51:15.146 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 168946 2024-02-05T06:51:15.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.146 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:15.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:15.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.072950+0000", 2024-02-05T06:51:15.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:15.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.072950+0000", 2024-02-05T06:51:15.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:15.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.072952+0000", 2024-02-05T06:51:15.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9570000000000001e-06 2024-02-05T06:51:15.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:15.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.072956+0000", 2024-02-05T06:51:15.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2440000000000004e-06 2024-02-05T06:51:15.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:15.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.072967+0000", 2024-02-05T06:51:15.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0642000000000001e-05 2024-02-05T06:51:15.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:15.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.072975+0000", 2024-02-05T06:51:15.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1850000000000004e-06 2024-02-05T06:51:15.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.073035+0000", 2024-02-05T06:51:15.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0412999999999999e-05 2024-02-05T06:51:15.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:15.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.073049+0000", 2024-02-05T06:51:15.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3791000000000001e-05 2024-02-05T06:51:15.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.537403+0000", 2024-02-05T06:51:15.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.464353424 2024-02-05T06:51:15.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for missing object", 2024-02-05T06:51:15.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.537533+0000", 2024-02-05T06:51:15.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013027899999999999 2024-02-05T06:51:15.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.552442+0000", 2024-02-05T06:51:15.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.014908925999999999 2024-02-05T06:51:15.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:15.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.552563+0000", 2024-02-05T06:51:15.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012095399999999999 2024-02-05T06:51:15.150 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.150 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:15.150 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.150 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:168953 8.4 8:2441a741:::benchmark_data_smithi028_105698_object79678:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:51:15.150 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.076290+0000", 2024-02-05T06:51:15.150 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 505.04656972100003, 2024-02-05T06:51:15.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4712057160000001, 2024-02-05T06:51:15.151 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:15.151 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:51:15.151 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:15.151 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:51:15.151 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:51:15.151 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 168953 2024-02-05T06:51:15.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.151 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:15.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:15.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.076290+0000", 2024-02-05T06:51:15.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:15.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.076290+0000", 2024-02-05T06:51:15.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:15.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.076291+0000", 2024-02-05T06:51:15.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0999999999999997e-07 2024-02-05T06:51:15.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:15.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.076294+0000", 2024-02-05T06:51:15.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6210000000000001e-06 2024-02-05T06:51:15.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:15.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.076304+0000", 2024-02-05T06:51:15.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0172e-05 2024-02-05T06:51:15.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:15.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.076312+0000", 2024-02-05T06:51:15.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9500000000000001e-06 2024-02-05T06:51:15.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.076362+0000", 2024-02-05T06:51:15.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9302999999999999e-05 2024-02-05T06:51:15.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:15.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.076370+0000", 2024-02-05T06:51:15.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0369999999999995e-06 2024-02-05T06:51:15.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.537753+0000", 2024-02-05T06:51:15.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.461383297 2024-02-05T06:51:15.154 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:15.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.537848+0000", 2024-02-05T06:51:15.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4944000000000004e-05 2024-02-05T06:51:15.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:51:15.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.538252+0000", 2024-02-05T06:51:15.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000403631 2024-02-05T06:51:15.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:15.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547266+0000", 2024-02-05T06:51:15.155 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0090147909999999994 2024-02-05T06:51:15.155 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.155 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.155 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:51:15.155 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547282+0000", 2024-02-05T06:51:15.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5639999999999999e-05 2024-02-05T06:51:15.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:15.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547408+0000", 2024-02-05T06:51:15.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012549300000000001 2024-02-05T06:51:15.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:51:15.156 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547440+0000", 2024-02-05T06:51:15.156 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2503999999999999e-05 2024-02-05T06:51:15.156 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.156 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.156 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:15.157 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547496+0000", 2024-02-05T06:51:15.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5423000000000001e-05 2024-02-05T06:51:15.157 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.157 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:15.157 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.157 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.157 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.157 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:168969 8.4 8:26e75fb4:::benchmark_data_smithi028_105698_object79694:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:51:15.157 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.095241+0000", 2024-02-05T06:51:15.157 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 505.02761850299999, 2024-02-05T06:51:15.157 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.452403653, 2024-02-05T06:51:15.157 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:15.157 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:51:15.157 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:15.157 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:51:15.157 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:51:15.158 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 168969 2024-02-05T06:51:15.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.158 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:15.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:15.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.095241+0000", 2024-02-05T06:51:15.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:15.158 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.095241+0000", 2024-02-05T06:51:15.158 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.158 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.158 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.158 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:15.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.095243+0000", 2024-02-05T06:51:15.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9439999999999999e-06 2024-02-05T06:51:15.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:15.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.095251+0000", 2024-02-05T06:51:15.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0669999999999992e-06 2024-02-05T06:51:15.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:15.159 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.095262+0000", 2024-02-05T06:51:15.159 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1071e-05 2024-02-05T06:51:15.159 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.159 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.159 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:15.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.095269+0000", 2024-02-05T06:51:15.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8140000000000003e-06 2024-02-05T06:51:15.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.095313+0000", 2024-02-05T06:51:15.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4387e-05 2024-02-05T06:51:15.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.160 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.160 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:15.160 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.095320+0000", 2024-02-05T06:51:15.160 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6290000000000004e-06 2024-02-05T06:51:15.160 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.161 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.542892+0000", 2024-02-05T06:51:15.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.447572388 2024-02-05T06:51:15.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:15.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.543029+0000", 2024-02-05T06:51:15.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013688 2024-02-05T06:51:15.162 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.162 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.162 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:51:15.162 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.543182+0000", 2024-02-05T06:51:15.162 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015253200000000001 2024-02-05T06:51:15.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:51:15.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547287+0000", 2024-02-05T06:51:15.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0041050360000000003 2024-02-05T06:51:15.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:15.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547332+0000", 2024-02-05T06:51:15.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4749000000000001e-05 2024-02-05T06:51:15.163 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.163 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.163 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:15.163 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547552+0000", 2024-02-05T06:51:15.163 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022023999999999999 2024-02-05T06:51:15.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:51:15.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547579+0000", 2024-02-05T06:51:15.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7698000000000001e-05 2024-02-05T06:51:15.164 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.164 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.164 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:15.164 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547645+0000", 2024-02-05T06:51:15.164 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5217999999999998e-05 2024-02-05T06:51:15.164 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.164 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:15.164 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.165 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.165 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:168990 8.4 8:2816a5c0:::benchmark_data_smithi028_105698_object79715:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:51:15.165 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.116213+0000", 2024-02-05T06:51:15.165 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 505.00664700599998, 2024-02-05T06:51:15.165 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4316142949999999, 2024-02-05T06:51:15.165 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:15.165 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:51:15.165 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:15.165 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:51:15.165 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:51:15.165 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 168990 2024-02-05T06:51:15.165 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.165 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:15.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:15.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.116213+0000", 2024-02-05T06:51:15.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:15.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.116213+0000", 2024-02-05T06:51:15.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.166 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.166 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.166 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:15.166 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.116213+0000", 2024-02-05T06:51:15.166 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4199999999999996e-07 2024-02-05T06:51:15.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:15.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.116216+0000", 2024-02-05T06:51:15.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5890000000000001e-06 2024-02-05T06:51:15.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:15.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.116221+0000", 2024-02-05T06:51:15.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3009999999999997e-06 2024-02-05T06:51:15.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.167 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.167 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:15.167 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.116226+0000", 2024-02-05T06:51:15.167 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8409999999999999e-06 2024-02-05T06:51:15.167 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.129047+0000", 2024-02-05T06:51:15.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.01282082 2024-02-05T06:51:15.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:15.168 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.129057+0000", 2024-02-05T06:51:15.168 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0481e-05 2024-02-05T06:51:15.168 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.168 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.168 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.545004+0000", 2024-02-05T06:51:15.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.415947125 2024-02-05T06:51:15.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:15.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.545081+0000", 2024-02-05T06:51:15.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6550000000000004e-05 2024-02-05T06:51:15.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:51:15.169 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.545152+0000", 2024-02-05T06:51:15.169 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1728000000000002e-05 2024-02-05T06:51:15.169 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.169 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.169 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:51:15.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547292+0000", 2024-02-05T06:51:15.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002139939 2024-02-05T06:51:15.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:15.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547373+0000", 2024-02-05T06:51:15.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0328999999999996e-05 2024-02-05T06:51:15.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:15.170 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547719+0000", 2024-02-05T06:51:15.170 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00034667500000000001 2024-02-05T06:51:15.170 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.170 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.170 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:51:15.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547746+0000", 2024-02-05T06:51:15.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6506999999999999e-05 2024-02-05T06:51:15.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.171 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:15.171 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547827+0000", 2024-02-05T06:51:15.171 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0867999999999999e-05 2024-02-05T06:51:15.171 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.171 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:15.171 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.171 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.171 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.171 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:168997 8.4 8:2fd29a35:::benchmark_data_smithi028_105698_object79722:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:51:15.171 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.123539+0000", 2024-02-05T06:51:15.171 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 504.99932001899998, 2024-02-05T06:51:15.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.43626739, 2024-02-05T06:51:15.172 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:15.172 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:51:15.172 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:15.172 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:51:15.172 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:51:15.172 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 168997 2024-02-05T06:51:15.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.172 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:15.172 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.172 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:15.172 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.123539+0000", 2024-02-05T06:51:15.172 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.172 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:15.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.123539+0000", 2024-02-05T06:51:15.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:15.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.123542+0000", 2024-02-05T06:51:15.173 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1509999999999998e-06 2024-02-05T06:51:15.173 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.173 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.173 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:15.173 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.123548+0000", 2024-02-05T06:51:15.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5780000000000004e-06 2024-02-05T06:51:15.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:15.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.123560+0000", 2024-02-05T06:51:15.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1674999999999999e-05 2024-02-05T06:51:15.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:15.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.123567+0000", 2024-02-05T06:51:15.174 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5739999999999998e-06 2024-02-05T06:51:15.174 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.174 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.174 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.174 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.129085+0000", 2024-02-05T06:51:15.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055173410000000003 2024-02-05T06:51:15.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:15.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.129091+0000", 2024-02-05T06:51:15.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5980000000000002e-06 2024-02-05T06:51:15.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547866+0000", 2024-02-05T06:51:15.175 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4187741730000001 2024-02-05T06:51:15.175 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.175 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.175 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:15.175 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547932+0000", 2024-02-05T06:51:15.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6223999999999995e-05 2024-02-05T06:51:15.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:51:15.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.547998+0000", 2024-02-05T06:51:15.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5950000000000004e-05 2024-02-05T06:51:15.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.176 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:51:15.176 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.552551+0000", 2024-02-05T06:51:15.176 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0045535530000000001 2024-02-05T06:51:15.176 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.176 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:15.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.555220+0000", 2024-02-05T06:51:15.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0026689090000000001 2024-02-05T06:51:15.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:15.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.559657+0000", 2024-02-05T06:51:15.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0044368070000000001 2024-02-05T06:51:15.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.177 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.177 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:51:15.177 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.559714+0000", 2024-02-05T06:51:15.177 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6481999999999999e-05 2024-02-05T06:51:15.177 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.178 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:15.178 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.559807+0000", 2024-02-05T06:51:15.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3375000000000006e-05 2024-02-05T06:51:15.178 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.178 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:15.178 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.178 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.178 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.178 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:168999 8.4 8:2a087a6b:::benchmark_data_smithi028_105698_object79724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:51:15.178 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.124021+0000", 2024-02-05T06:51:15.178 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 504.99883833600001, 2024-02-05T06:51:15.178 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.436005618, 2024-02-05T06:51:15.178 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:15.178 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:51:15.179 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:15.179 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:51:15.179 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:51:15.179 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 168999 2024-02-05T06:51:15.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.179 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:15.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:15.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124021+0000", 2024-02-05T06:51:15.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.179 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.179 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.179 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:15.179 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124021+0000", 2024-02-05T06:51:15.179 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:15.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124022+0000", 2024-02-05T06:51:15.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.268e-06 2024-02-05T06:51:15.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:15.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124027+0000", 2024-02-05T06:51:15.180 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1500000000000001e-06 2024-02-05T06:51:15.180 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.180 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.180 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:15.180 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124037+0000", 2024-02-05T06:51:15.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0278999999999999e-05 2024-02-05T06:51:15.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:15.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124045+0000", 2024-02-05T06:51:15.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3520000000000007e-06 2024-02-05T06:51:15.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.181 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.129115+0000", 2024-02-05T06:51:15.181 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0050697160000000002 2024-02-05T06:51:15.181 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.181 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.181 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:15.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.129122+0000", 2024-02-05T06:51:15.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7549999999999997e-06 2024-02-05T06:51:15.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.549524+0000", 2024-02-05T06:51:15.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4204025490000001 2024-02-05T06:51:15.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.182 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:15.182 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.549646+0000", 2024-02-05T06:51:15.182 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121548 2024-02-05T06:51:15.182 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.182 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:51:15.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.549777+0000", 2024-02-05T06:51:15.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013103499999999999 2024-02-05T06:51:15.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:51:15.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.552562+0000", 2024-02-05T06:51:15.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002784777 2024-02-05T06:51:15.183 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.183 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.183 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:15.183 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.555297+0000", 2024-02-05T06:51:15.183 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002735632 2024-02-05T06:51:15.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:15.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.559878+0000", 2024-02-05T06:51:15.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0045809509999999998 2024-02-05T06:51:15.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:51:15.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.559934+0000", 2024-02-05T06:51:15.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5815e-05 2024-02-05T06:51:15.184 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.184 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.184 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:15.184 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.560027+0000", 2024-02-05T06:51:15.184 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2791e-05 2024-02-05T06:51:15.185 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.185 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:15.185 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.185 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.185 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.185 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:169001 8.4 8:2d24d8c7:::benchmark_data_smithi028_105698_object79726:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:51:15.185 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.124640+0000", 2024-02-05T06:51:15.185 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 504.99821985800003, 2024-02-05T06:51:15.185 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4355753789999999, 2024-02-05T06:51:15.185 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:15.185 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:51:15.185 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:15.185 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:51:15.185 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:51:15.185 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169001 2024-02-05T06:51:15.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.186 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:15.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:15.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124640+0000", 2024-02-05T06:51:15.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:15.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124640+0000", 2024-02-05T06:51:15.186 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.186 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.186 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.186 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:15.186 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124641+0000", 2024-02-05T06:51:15.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2759999999999999e-06 2024-02-05T06:51:15.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:15.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124646+0000", 2024-02-05T06:51:15.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7790000000000003e-06 2024-02-05T06:51:15.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:15.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124657+0000", 2024-02-05T06:51:15.187 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1739000000000001e-05 2024-02-05T06:51:15.187 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.187 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.187 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:15.187 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.124666+0000", 2024-02-05T06:51:15.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3999999999999992e-06 2024-02-05T06:51:15.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.129144+0000", 2024-02-05T06:51:15.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0044785270000000004 2024-02-05T06:51:15.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:15.188 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.129150+0000", 2024-02-05T06:51:15.188 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7849999999999997e-06 2024-02-05T06:51:15.188 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.188 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.188 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.552592+0000", 2024-02-05T06:51:15.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.423442265 2024-02-05T06:51:15.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:15.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.552720+0000", 2024-02-05T06:51:15.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012783400000000001 2024-02-05T06:51:15.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:51:15.189 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.552831+0000", 2024-02-05T06:51:15.189 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000110325 2024-02-05T06:51:15.189 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.189 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.189 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:15.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.559592+0000", 2024-02-05T06:51:15.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.006761469 2024-02-05T06:51:15.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:51:15.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.559817+0000", 2024-02-05T06:51:15.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00022453499999999999 2024-02-05T06:51:15.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.190 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:15.190 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.560092+0000", 2024-02-05T06:51:15.190 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027514500000000002 2024-02-05T06:51:15.190 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.190 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:51:15.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.560146+0000", 2024-02-05T06:51:15.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4534999999999999e-05 2024-02-05T06:51:15.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.191 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:15.191 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.560215+0000", 2024-02-05T06:51:15.191 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8764999999999996e-05 2024-02-05T06:51:15.191 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.191 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:15.191 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.191 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.191 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.191 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:169022 8.4 8:2073961b:::benchmark_data_smithi028_105698_object79747:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:51:15.191 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.143624+0000", 2024-02-05T06:51:15.192 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 504.97923574700002, 2024-02-05T06:51:15.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4328469529999999, 2024-02-05T06:51:15.192 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:15.192 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:51:15.192 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:15.192 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:51:15.192 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:51:15.192 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169022 2024-02-05T06:51:15.192 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.192 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:15.192 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.192 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:15.192 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.143624+0000", 2024-02-05T06:51:15.192 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:15.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.143624+0000", 2024-02-05T06:51:15.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:15.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.143625+0000", 2024-02-05T06:51:15.193 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5120000000000001e-06 2024-02-05T06:51:15.193 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.193 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.193 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:15.193 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.143630+0000", 2024-02-05T06:51:15.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8289999999999998e-06 2024-02-05T06:51:15.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:15.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.143642+0000", 2024-02-05T06:51:15.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1574e-05 2024-02-05T06:51:15.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:15.194 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.143648+0000", 2024-02-05T06:51:15.194 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2870000000000001e-06 2024-02-05T06:51:15.194 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.194 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.194 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.143712+0000", 2024-02-05T06:51:15.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3972000000000003e-05 2024-02-05T06:51:15.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:15.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.143724+0000", 2024-02-05T06:51:15.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2119999999999999e-05 2024-02-05T06:51:15.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.195 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.555338+0000", 2024-02-05T06:51:15.195 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4116135439999999 2024-02-05T06:51:15.195 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.195 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.195 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:15.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.555474+0000", 2024-02-05T06:51:15.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013640399999999999 2024-02-05T06:51:15.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:51:15.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.555604+0000", 2024-02-05T06:51:15.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000130102 2024-02-05T06:51:15.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.196 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:51:15.196 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.559821+0000", 2024-02-05T06:51:15.196 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0042167460000000004 2024-02-05T06:51:15.196 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.196 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:15.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.565344+0000", 2024-02-05T06:51:15.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0055230440000000004 2024-02-05T06:51:15.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:15.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576302+0000", 2024-02-05T06:51:15.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010958239 2024-02-05T06:51:15.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.197 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:51:15.197 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576356+0000", 2024-02-05T06:51:15.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3823999999999998e-05 2024-02-05T06:51:15.197 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:15.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576471+0000", 2024-02-05T06:51:15.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000114756 2024-02-05T06:51:15.198 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.198 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:15.198 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.198 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:169057 8.4 8:26322b02:::benchmark_data_smithi028_105698_object79782:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:51:15.198 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.170483+0000", 2024-02-05T06:51:15.198 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 504.95237675999999, 2024-02-05T06:51:15.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.409427684, 2024-02-05T06:51:15.198 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:15.198 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:51:15.198 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:15.199 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:51:15.199 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:51:15.199 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169057 2024-02-05T06:51:15.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.199 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:15.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:15.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.170483+0000", 2024-02-05T06:51:15.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:15.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.170483+0000", 2024-02-05T06:51:15.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:15.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.170485+0000", 2024-02-05T06:51:15.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1239999999999999e-06 2024-02-05T06:51:15.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:15.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.170490+0000", 2024-02-05T06:51:15.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9490000000000004e-06 2024-02-05T06:51:15.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:15.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.170502+0000", 2024-02-05T06:51:15.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1851999999999999e-05 2024-02-05T06:51:15.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:15.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.170509+0000", 2024-02-05T06:51:15.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2320000000000004e-06 2024-02-05T06:51:15.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.170567+0000", 2024-02-05T06:51:15.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8152000000000002e-05 2024-02-05T06:51:15.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:15.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.170577+0000", 2024-02-05T06:51:15.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9150000000000008e-06 2024-02-05T06:51:15.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.557586+0000", 2024-02-05T06:51:15.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3870086580000001 2024-02-05T06:51:15.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:15.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.557731+0000", 2024-02-05T06:51:15.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014533499999999999 2024-02-05T06:51:15.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:51:15.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.557845+0000", 2024-02-05T06:51:15.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000114529 2024-02-05T06:51:15.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:51:15.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.560223+0000", 2024-02-05T06:51:15.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002377503 2024-02-05T06:51:15.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:15.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.565405+0000", 2024-02-05T06:51:15.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0051823950000000002 2024-02-05T06:51:15.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:15.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.579808+0000", 2024-02-05T06:51:15.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.014402223 2024-02-05T06:51:15.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:51:15.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.579872+0000", 2024-02-05T06:51:15.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4863999999999994e-05 2024-02-05T06:51:15.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:15.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.579910+0000", 2024-02-05T06:51:15.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7953000000000001e-05 2024-02-05T06:51:15.204 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.204 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:15.205 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.205 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:169060 8.4 8:282dec9e:::benchmark_data_smithi028_105698_object79785:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:51:15.205 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.172664+0000", 2024-02-05T06:51:15.205 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 504.95019572799998, 2024-02-05T06:51:15.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4074563200000001, 2024-02-05T06:51:15.205 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:15.205 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:51:15.205 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:15.205 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:51:15.205 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:51:15.205 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169060 2024-02-05T06:51:15.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.205 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:15.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:15.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.172664+0000", 2024-02-05T06:51:15.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:15.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.172664+0000", 2024-02-05T06:51:15.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:15.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.172665+0000", 2024-02-05T06:51:15.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4389999999999999e-06 2024-02-05T06:51:15.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:15.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.172673+0000", 2024-02-05T06:51:15.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3529999999999997e-06 2024-02-05T06:51:15.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:15.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.172683+0000", 2024-02-05T06:51:15.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0891e-05 2024-02-05T06:51:15.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:15.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.172691+0000", 2024-02-05T06:51:15.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4569999999999999e-06 2024-02-05T06:51:15.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.172734+0000", 2024-02-05T06:51:15.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2962e-05 2024-02-05T06:51:15.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:15.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.172740+0000", 2024-02-05T06:51:15.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1149999999999999e-06 2024-02-05T06:51:15.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.560263+0000", 2024-02-05T06:51:15.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3875225520000001 2024-02-05T06:51:15.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:15.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.560368+0000", 2024-02-05T06:51:15.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000105481 2024-02-05T06:51:15.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:51:15.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.560522+0000", 2024-02-05T06:51:15.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015420200000000001 2024-02-05T06:51:15.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:51:15.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.565356+0000", 2024-02-05T06:51:15.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0048341139999999996 2024-02-05T06:51:15.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:15.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.571129+0000", 2024-02-05T06:51:15.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0057722889999999999 2024-02-05T06:51:15.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.210 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.210 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:15.210 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580047+0000", 2024-02-05T06:51:15.210 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0089180249999999996 2024-02-05T06:51:15.210 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:51:15.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580093+0000", 2024-02-05T06:51:15.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6730000000000002e-05 2024-02-05T06:51:15.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:15.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580120+0000", 2024-02-05T06:51:15.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.671e-05 2024-02-05T06:51:15.211 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.211 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:15.211 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.212 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:169062 8.4 8:2afada59:::benchmark_data_smithi028_105698_object79787:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:51:15.212 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.173201+0000", 2024-02-05T06:51:15.212 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 504.949658446, 2024-02-05T06:51:15.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4070665949999999, 2024-02-05T06:51:15.212 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:15.212 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:51:15.212 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:15.212 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:51:15.212 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:51:15.212 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169062 2024-02-05T06:51:15.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.212 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:15.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:15.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.173201+0000", 2024-02-05T06:51:15.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:15.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.173201+0000", 2024-02-05T06:51:15.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:15.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.173202+0000", 2024-02-05T06:51:15.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7399999999999991e-07 2024-02-05T06:51:15.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:15.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.173206+0000", 2024-02-05T06:51:15.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8999999999999999e-06 2024-02-05T06:51:15.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:15.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.173215+0000", 2024-02-05T06:51:15.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7329999999999992e-06 2024-02-05T06:51:15.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:15.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.173221+0000", 2024-02-05T06:51:15.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0419999999999999e-06 2024-02-05T06:51:15.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.173266+0000", 2024-02-05T06:51:15.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5256999999999997e-05 2024-02-05T06:51:15.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:15.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.173275+0000", 2024-02-05T06:51:15.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.003e-06 2024-02-05T06:51:15.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.562986+0000", 2024-02-05T06:51:15.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3897114660000001 2024-02-05T06:51:15.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:15.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.563100+0000", 2024-02-05T06:51:15.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113683 2024-02-05T06:51:15.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:51:15.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.563224+0000", 2024-02-05T06:51:15.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000123738 2024-02-05T06:51:15.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:15.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.571213+0000", 2024-02-05T06:51:15.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079889839999999993 2024-02-05T06:51:15.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:51:15.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576145+0000", 2024-02-05T06:51:15.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00493196 2024-02-05T06:51:15.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:15.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580160+0000", 2024-02-05T06:51:15.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040155939999999999 2024-02-05T06:51:15.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:51:15.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580218+0000", 2024-02-05T06:51:15.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7779999999999999e-05 2024-02-05T06:51:15.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:15.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580268+0000", 2024-02-05T06:51:15.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9481000000000003e-05 2024-02-05T06:51:15.218 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.218 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:15.218 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.218 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:169071 8.4 8:2a441816:::benchmark_data_smithi028_105698_object79796:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:51:15.218 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.181972+0000", 2024-02-05T06:51:15.218 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 504.94088744200002, 2024-02-05T06:51:15.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3984312409999999, 2024-02-05T06:51:15.218 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:15.218 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:51:15.219 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:15.219 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:51:15.219 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:51:15.219 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169071 2024-02-05T06:51:15.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.219 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:15.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:15.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.181972+0000", 2024-02-05T06:51:15.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:15.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.181972+0000", 2024-02-05T06:51:15.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:15.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.181974+0000", 2024-02-05T06:51:15.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9980000000000002e-06 2024-02-05T06:51:15.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:15.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.181979+0000", 2024-02-05T06:51:15.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.7310000000000001e-06 2024-02-05T06:51:15.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:15.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.181990+0000", 2024-02-05T06:51:15.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1381e-05 2024-02-05T06:51:15.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:15.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.181998+0000", 2024-02-05T06:51:15.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.6539999999999999e-06 2024-02-05T06:51:15.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.182056+0000", 2024-02-05T06:51:15.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8276000000000002e-05 2024-02-05T06:51:15.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:15.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.182066+0000", 2024-02-05T06:51:15.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.6199999999999994e-06 2024-02-05T06:51:15.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.565453+0000", 2024-02-05T06:51:15.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.383387401 2024-02-05T06:51:15.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:15.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.565578+0000", 2024-02-05T06:51:15.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012461699999999999 2024-02-05T06:51:15.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:51:15.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.565709+0000", 2024-02-05T06:51:15.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013105899999999999 2024-02-05T06:51:15.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:51:15.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576157+0000", 2024-02-05T06:51:15.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010448169 2024-02-05T06:51:15.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:15.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576240+0000", 2024-02-05T06:51:15.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3319999999999995e-05 2024-02-05T06:51:15.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:15.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580310+0000", 2024-02-05T06:51:15.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0040695719999999996 2024-02-05T06:51:15.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:51:15.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580371+0000", 2024-02-05T06:51:15.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1490999999999998e-05 2024-02-05T06:51:15.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:15.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580403+0000", 2024-02-05T06:51:15.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.1952000000000002e-05 2024-02-05T06:51:15.224 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.224 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:15.224 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.224 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:169074 8.4 8:23b7f380:::benchmark_data_smithi028_105698_object79799:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:51:15.224 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.185596+0000", 2024-02-05T06:51:15.224 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 504.93726337800001, 2024-02-05T06:51:15.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.394963838, 2024-02-05T06:51:15.225 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:15.225 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:51:15.225 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:15.225 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:51:15.225 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:51:15.225 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169074 2024-02-05T06:51:15.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.225 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:15.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:15.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.185596+0000", 2024-02-05T06:51:15.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:15.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.185596+0000", 2024-02-05T06:51:15.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:15.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.185597+0000", 2024-02-05T06:51:15.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8199999999999999e-07 2024-02-05T06:51:15.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:15.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.185600+0000", 2024-02-05T06:51:15.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6970000000000002e-06 2024-02-05T06:51:15.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:15.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.185607+0000", 2024-02-05T06:51:15.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0029999999999997e-06 2024-02-05T06:51:15.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:15.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.185611+0000", 2024-02-05T06:51:15.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4859999999999999e-06 2024-02-05T06:51:15.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.185655+0000", 2024-02-05T06:51:15.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4150999999999998e-05 2024-02-05T06:51:15.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:15.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.185664+0000", 2024-02-05T06:51:15.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7690000000000002e-06 2024-02-05T06:51:15.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.568350+0000", 2024-02-05T06:51:15.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.382685704 2024-02-05T06:51:15.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:15.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.568474+0000", 2024-02-05T06:51:15.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012467500000000001 2024-02-05T06:51:15.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:51:15.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.568613+0000", 2024-02-05T06:51:15.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013851099999999999 2024-02-05T06:51:15.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:51:15.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576163+0000", 2024-02-05T06:51:15.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0075501220000000003 2024-02-05T06:51:15.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:15.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576519+0000", 2024-02-05T06:51:15.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00035637499999999998 2024-02-05T06:51:15.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:15.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580473+0000", 2024-02-05T06:51:15.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0039534779999999999 2024-02-05T06:51:15.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:51:15.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580524+0000", 2024-02-05T06:51:15.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0788999999999998e-05 2024-02-05T06:51:15.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:15.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.580560+0000", 2024-02-05T06:51:15.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6396000000000002e-05 2024-02-05T06:51:15.231 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.232 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:15.232 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.232 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:169145 8.4 8:2cf8555b:::benchmark_data_smithi028_105698_object79870:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:51:15.232 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.295920+0000", 2024-02-05T06:51:15.232 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 504.82693996199998, 2024-02-05T06:51:15.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2886338849999999, 2024-02-05T06:51:15.232 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:15.232 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:51:15.232 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:15.232 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:51:15.232 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:51:15.233 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169145 2024-02-05T06:51:15.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.233 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:15.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:15.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.295920+0000", 2024-02-05T06:51:15.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:15.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.295920+0000", 2024-02-05T06:51:15.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:15.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.295920+0000", 2024-02-05T06:51:15.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6700000000000003e-07 2024-02-05T06:51:15.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:15.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.295923+0000", 2024-02-05T06:51:15.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4540000000000001e-06 2024-02-05T06:51:15.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:15.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.295928+0000", 2024-02-05T06:51:15.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3530000000000002e-06 2024-02-05T06:51:15.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:15.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.295932+0000", 2024-02-05T06:51:15.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2030000000000002e-06 2024-02-05T06:51:15.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.295973+0000", 2024-02-05T06:51:15.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0450000000000001e-05 2024-02-05T06:51:15.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:15.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.295978+0000", 2024-02-05T06:51:15.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7620000000000001e-06 2024-02-05T06:51:15.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.571258+0000", 2024-02-05T06:51:15.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.27527938 2024-02-05T06:51:15.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:15.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.571371+0000", 2024-02-05T06:51:15.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113448 2024-02-05T06:51:15.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:51:15.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.571485+0000", 2024-02-05T06:51:15.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00011325 2024-02-05T06:51:15.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:51:15.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576267+0000", 2024-02-05T06:51:15.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0047829350000000003 2024-02-05T06:51:15.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:15.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576587+0000", 2024-02-05T06:51:15.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00031938600000000001 2024-02-05T06:51:15.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:15.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.584465+0000", 2024-02-05T06:51:15.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078778700000000004 2024-02-05T06:51:15.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:51:15.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.584516+0000", 2024-02-05T06:51:15.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1570000000000003e-05 2024-02-05T06:51:15.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:15.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.584553+0000", 2024-02-05T06:51:15.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7156999999999997e-05 2024-02-05T06:51:15.238 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.238 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:15.238 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.239 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:169147 8.4 8:27ce7ef9:::benchmark_data_smithi028_105698_object79872:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:51:15.239 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.300286+0000", 2024-02-05T06:51:15.239 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 504.82257320899998, 2024-02-05T06:51:15.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.284371353, 2024-02-05T06:51:15.239 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:15.239 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:51:15.239 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:15.239 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:51:15.239 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:51:15.239 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169147 2024-02-05T06:51:15.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.239 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:15.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:15.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.300286+0000", 2024-02-05T06:51:15.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:15.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.300286+0000", 2024-02-05T06:51:15.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:15.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.300287+0000", 2024-02-05T06:51:15.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7900000000000007e-07 2024-02-05T06:51:15.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:15.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.300290+0000", 2024-02-05T06:51:15.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.0970000000000002e-06 2024-02-05T06:51:15.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:15.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.300298+0000", 2024-02-05T06:51:15.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9130000000000003e-06 2024-02-05T06:51:15.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:15.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.300305+0000", 2024-02-05T06:51:15.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2069999999999998e-06 2024-02-05T06:51:15.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.300347+0000", 2024-02-05T06:51:15.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1389e-05 2024-02-05T06:51:15.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:15.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.300354+0000", 2024-02-05T06:51:15.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7549999999999997e-06 2024-02-05T06:51:15.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.573495+0000", 2024-02-05T06:51:15.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.273141654 2024-02-05T06:51:15.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:15.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.573626+0000", 2024-02-05T06:51:15.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013036499999999999 2024-02-05T06:51:15.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:51:15.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.573750+0000", 2024-02-05T06:51:15.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012482500000000001 2024-02-05T06:51:15.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:51:15.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576534+0000", 2024-02-05T06:51:15.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0027837539999999998 2024-02-05T06:51:15.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:15.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576637+0000", 2024-02-05T06:51:15.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010228199999999999 2024-02-05T06:51:15.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:15.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.584587+0000", 2024-02-05T06:51:15.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0079500309999999998 2024-02-05T06:51:15.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:51:15.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.584625+0000", 2024-02-05T06:51:15.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8019000000000003e-05 2024-02-05T06:51:15.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:15.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.584658+0000", 2024-02-05T06:51:15.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3083000000000002e-05 2024-02-05T06:51:15.245 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.245 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:15.245 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.246 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15144.0:169148 8.4 8:250ae785:::benchmark_data_smithi028_105698_object79873:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e279)", 2024-02-05T06:51:15.246 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:42:50.301004+0000", 2024-02-05T06:51:15.246 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 504.82185560300002, 2024-02-05T06:51:15.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.287525246, 2024-02-05T06:51:15.246 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:15.246 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:51:15.246 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:15.246 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15144", 2024-02-05T06:51:15.246 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2347420402", 2024-02-05T06:51:15.246 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 169148 2024-02-05T06:51:15.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.247 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:15.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:15.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.301004+0000", 2024-02-05T06:51:15.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:15.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.301004+0000", 2024-02-05T06:51:15.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:15.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.301005+0000", 2024-02-05T06:51:15.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8400000000000003e-07 2024-02-05T06:51:15.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:15.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.301008+0000", 2024-02-05T06:51:15.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2969999999999999e-06 2024-02-05T06:51:15.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:15.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.301017+0000", 2024-02-05T06:51:15.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.3749999999999992e-06 2024-02-05T06:51:15.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:15.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.301024+0000", 2024-02-05T06:51:15.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2940000000000002e-06 2024-02-05T06:51:15.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.301066+0000", 2024-02-05T06:51:15.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2660999999999997e-05 2024-02-05T06:51:15.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:15.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:50.301074+0000", 2024-02-05T06:51:15.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4139999999999997e-06 2024-02-05T06:51:15.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576711+0000", 2024-02-05T06:51:15.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2756368330000001 2024-02-05T06:51:15.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:51:15.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576823+0000", 2024-02-05T06:51:15.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000112048 2024-02-05T06:51:15.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,4", 2024-02-05T06:51:15.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.576918+0000", 2024-02-05T06:51:15.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5593000000000004e-05 2024-02-05T06:51:15.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:15.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.583421+0000", 2024-02-05T06:51:15.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0065030330000000001 2024-02-05T06:51:15.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:51:15.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.583436+0000", 2024-02-05T06:51:15.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4219000000000001e-05 2024-02-05T06:51:15.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:51:15.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.588452+0000", 2024-02-05T06:51:15.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.005016777 2024-02-05T06:51:15.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:51:15.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.588506+0000", 2024-02-05T06:51:15.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3804999999999997e-05 2024-02-05T06:51:15.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:15.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:42:51.588529+0000", 2024-02-05T06:51:15.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3013000000000001e-05 2024-02-05T06:51:15.252 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.252 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:15.252 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.253 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.45587.0:103397 10.19 10:98d8228c:::benchmark_data_smithi028_154709_object103396:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-02-05T06:51:15.253 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:48:51.224080+0000", 2024-02-05T06:51:15.253 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 143.898779297, 2024-02-05T06:51:15.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.038693122, 2024-02-05T06:51:15.253 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:15.253 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-02-05T06:51:15.253 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:15.253 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.45587", 2024-02-05T06:51:15.253 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:51:15.253 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 103397 2024-02-05T06:51:15.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.253 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:15.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:15.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.224080+0000", 2024-02-05T06:51:15.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:15.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.224080+0000", 2024-02-05T06:51:15.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:15.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.224081+0000", 2024-02-05T06:51:15.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2799999999999996e-07 2024-02-05T06:51:15.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:15.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.224122+0000", 2024-02-05T06:51:15.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0972e-05 2024-02-05T06:51:15.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:15.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.224128+0000", 2024-02-05T06:51:15.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2840000000000003e-06 2024-02-05T06:51:15.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:15.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.224133+0000", 2024-02-05T06:51:15.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0690000000000001e-06 2024-02-05T06:51:15.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.469567+0000", 2024-02-05T06:51:15.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.24543419499999999 2024-02-05T06:51:15.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:15.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.469572+0000", 2024-02-05T06:51:15.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0069999999999997e-06 2024-02-05T06:51:15.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:52.262706+0000", 2024-02-05T06:51:15.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.79313390500000003 2024-02-05T06:51:15.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:15.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:52.262773+0000", 2024-02-05T06:51:15.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7062000000000003e-05 2024-02-05T06:51:15.257 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.257 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:15.257 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.257 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.45587.0:103399 10.19 10:9a33e9a1:::benchmark_data_smithi028_154709_object103398:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-02-05T06:51:15.257 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:48:51.224139+0000", 2024-02-05T06:51:15.257 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 143.89872010799999, 2024-02-05T06:51:15.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.038963115, 2024-02-05T06:51:15.258 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:51:15.258 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-02-05T06:51:15.258 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:51:15.258 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.45587", 2024-02-05T06:51:15.258 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:51:15.258 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 103399 2024-02-05T06:51:15.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.258 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:51:15.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:51:15.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.224139+0000", 2024-02-05T06:51:15.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:51:15.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.224139+0000", 2024-02-05T06:51:15.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:51:15.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:51:15.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.224140+0000", 2024-02-05T06:51:15.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9599999999999999e-07 2024-02-05T06:51:15.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:51:15.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.224172+0000", 2024-02-05T06:51:15.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2469999999999999e-05 2024-02-05T06:51:15.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:51:15.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.224179+0000", 2024-02-05T06:51:15.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.348e-06 2024-02-05T06:51:15.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:51:15.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.224184+0000", 2024-02-05T06:51:15.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1250000000000001e-06 2024-02-05T06:51:15.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.469590+0000", 2024-02-05T06:51:15.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.24540587999999999 2024-02-05T06:51:15.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:51:15.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.469595+0000", 2024-02-05T06:51:15.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6890000000000003e-06 2024-02-05T06:51:15.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:51:15.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:52.262965+0000", 2024-02-05T06:51:15.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.79337095300000005 2024-02-05T06:51:15.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:51:15.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:51:15.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:51:15.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:52.263103+0000", 2024-02-05T06:51:15.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013705400000000001 2024-02-05T06:51:15.261 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.261 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:15.262 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.262 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:51:15.262 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:51:15.262 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:51:15.262 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:15.286 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:15.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:15.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:15.388 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:15.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:15.414 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:15.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:15.476 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:15 smithi167 ceph-mon[32774]: osdmap e420: 8 total, 8 up, 8 in 2024-02-05T06:51:15.476 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:15 smithi167 ceph-mon[32774]: pgmap v853: 158 pgs: 7 creating+peering, 9 unknown, 142 active+clean; 582 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 870 B/s rd, 0 op/s 2024-02-05T06:51:15.515 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:15.519 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:51:15.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:15.540 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:15.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:15.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:15 smithi028 ceph-mon[29442]: osdmap e420: 8 total, 8 up, 8 in 2024-02-05T06:51:15.641 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:15 smithi028 ceph-mon[29442]: pgmap v853: 158 pgs: 7 creating+peering, 9 unknown, 142 active+clean; 582 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 870 B/s rd, 0 op/s 2024-02-05T06:51:15.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:15 smithi028 ceph-mon[37984]: osdmap e420: 8 total, 8 up, 8 in 2024-02-05T06:51:15.641 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:15 smithi028 ceph-mon[37984]: pgmap v853: 158 pgs: 7 creating+peering, 9 unknown, 142 active+clean; 582 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 870 B/s rd, 0 op/s 2024-02-05T06:51:15.642 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:15.665 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:15.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:15.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:15.767 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:15.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:15.792 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:15.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:15.895 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:15.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:15.919 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:15.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:15.998 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-05T06:51:16.021 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:16.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:16.045 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:16.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:16.146 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:16.171 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:16.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:16.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:16.241 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:51:16.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:16.297 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:16.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:16.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:16.399 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:16.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:16.424 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:16.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:16.526 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:16.550 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:16.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:16.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:16.652 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:16.676 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:16.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:16.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:16.778 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:16.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:16.802 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:16.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:16.903 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:16.927 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:16.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:16.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:17.028 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:17.052 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:17.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:17.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:17.154 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:17.177 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:17.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:17.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:17.279 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:17.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:17.303 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:17.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:17.404 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:17.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:17.429 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:17.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:17.530 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:17.554 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:17.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:17.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:17.608 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-02-05T06:51:17.655 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:17.679 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:17.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:17.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:17.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:17 smithi028 ceph-mon[29442]: pgmap v854: 158 pgs: 7 creating+peering, 8 unknown, 143 active+clean; 582 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:51:17.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:17 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2761810582' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:51:17.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:17 smithi028 ceph-mon[37984]: pgmap v854: 158 pgs: 7 creating+peering, 8 unknown, 143 active+clean; 582 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:51:17.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:17 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2761810582' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:51:17.781 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:17.806 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:17.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:17.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:17.891 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:17 smithi167 ceph-mon[32774]: pgmap v854: 158 pgs: 7 creating+peering, 8 unknown, 143 active+clean; 582 KiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:51:17.891 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:17 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2761810582' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:51:17.908 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:17.926 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:51:17.926 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-02-05T06:51:17.926 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-02-05T06:51:17.926 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:51:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:17.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:17.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:18.034 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:18.058 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:18.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:18.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:18.160 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:18.184 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:18.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:18.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:18.285 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:18.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:18.309 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:18.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:18.410 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:18.434 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:18.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:18.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:18.536 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:18.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:18.561 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:18.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:18.617 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-05T06:51:18.642 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:18 smithi167 ceph-mon[32774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:51:18.642 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:18 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2761810582' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:51:18.642 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:18 smithi167 ceph-mon[32774]: osdmap e421: 8 total, 8 up, 8 in 2024-02-05T06:51:18.662 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:18.687 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:18.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:18.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:18.789 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:18.813 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:18.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:18.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:18.858 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:51:18.893 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:18 smithi028 ceph-mon[37984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:51:18.893 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:18 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2761810582' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:51:18.893 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:18 smithi028 ceph-mon[37984]: osdmap e421: 8 total, 8 up, 8 in 2024-02-05T06:51:18.894 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:18 smithi028 ceph-mon[29442]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:51:18.894 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:18 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2761810582' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:51:18.894 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:18 smithi028 ceph-mon[29442]: osdmap e421: 8 total, 8 up, 8 in 2024-02-05T06:51:18.915 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:18.939 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:18.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:18.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:19.041 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:19.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:19.065 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:19.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:19.166 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:19.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:19.192 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:19.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:19.293 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:19.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:19.319 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:19.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:19.420 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:19.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:19.445 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:19.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:19.546 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:19.569 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:19.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:19.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:19.614 INFO:teuthology.orchestra.run.smithi028.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-02-05T06:51:19.673 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:19.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:19.699 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:19.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:19.800 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:19.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:19.826 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:19.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:19.907 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:19 smithi028 ceph-mon[37984]: pgmap v856: 158 pgs: 158 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:51:19.907 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:19 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1908074450' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T06:51:19.907 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:19 smithi028 ceph-mon[37984]: from='client.45869 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T06:51:19.907 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:19 smithi028 ceph-mon[29442]: pgmap v856: 158 pgs: 158 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:51:19.907 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:19 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1908074450' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T06:51:19.908 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:19 smithi028 ceph-mon[29442]: from='client.45869 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T06:51:19.927 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:19.952 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:19.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:19.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:19.971 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:51:19.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:19 smithi167 ceph-mon[32774]: pgmap v856: 158 pgs: 158 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:51:19.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:19 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1908074450' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T06:51:19.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:19 smithi167 ceph-mon[32774]: from='client.45869 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-02-05T06:51:20.053 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:20.077 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:20.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:20.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:20.179 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:20.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:20.204 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:20.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:20.305 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:20.329 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:20.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:20.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:20.346 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:51:20.432 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:20.456 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:20.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:20.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:20.558 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:20.583 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:20.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:20.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:20.684 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:20.708 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:20.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:20.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:20.743 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T06:51:20.760 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:20 smithi028 ceph-mon[37984]: from='client.45869 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T06:51:20.760 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:20 smithi028 ceph-mon[37984]: osdmap e422: 8 total, 8 up, 8 in 2024-02-05T06:51:20.760 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:20 smithi028 ceph-mon[29442]: from='client.45869 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T06:51:20.760 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:20 smithi028 ceph-mon[29442]: osdmap e422: 8 total, 8 up, 8 in 2024-02-05T06:51:20.810 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:20.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:20.835 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:20.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:20.881 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:20 smithi167 ceph-mon[32774]: from='client.45869 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-02-05T06:51:20.881 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:20 smithi167 ceph-mon[32774]: osdmap e422: 8 total, 8 up, 8 in 2024-02-05T06:51:20.881 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:51:20.882 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-02-05T06:51:20.882 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-02-05T06:51:20.882 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-02-05T06:51:20.882 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:51:20.936 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:20.959 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:20.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:20.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:21.061 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:21.086 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:21.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:21.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:21.188 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:21.212 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:21.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:21.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:21.285 DEBUG:teuthology.orchestra.run.smithi028:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-02-05T06:51:21.285 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-02-05T06:51:21.285 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T06:51:21.287 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-02-05T06:51:21.288 INFO:teuthology.orchestra.run:waiting for 3000 2024-02-05T06:51:21.314 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:21.335 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:hints = 1 2024-02-05T06:51:21.335 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-02-05T06:51:21.335 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Object prefix: benchmark_data_smithi028_177697 2024-02-05T06:51:21.335 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T06:51:21.335 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 0 0 0 0 0 0 - 0 2024-02-05T06:51:21.373 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:21.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:21.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:21.475 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:21.509 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:21.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:21.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:21.520 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:51:21.611 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:21.645 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:21.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:21.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:21.669 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-05T06:51:21.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:21.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:21.785 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:21.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:21.810 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:21 smithi028 ceph-mon[37984]: pgmap v858: 158 pgs: 158 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 816 B/s rd, 0 op/s 2024-02-05T06:51:21.810 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:21 smithi028 ceph-mon[37984]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T06:51:21.810 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:21 smithi028 ceph-mon[29442]: pgmap v858: 158 pgs: 158 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 816 B/s rd, 0 op/s 2024-02-05T06:51:21.810 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:21 smithi028 ceph-mon[29442]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T06:51:21.886 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:21.925 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:21.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:21.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:21.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:21 smithi167 ceph-mon[32774]: pgmap v858: 158 pgs: 158 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 816 B/s rd, 0 op/s 2024-02-05T06:51:21.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:21 smithi167 ceph-mon[32774]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-02-05T06:51:22.027 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:22.065 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:22.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:22.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:22.096 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:51:22.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:22.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:22.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:22.307 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:22.336 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 1 16 3132 3116 194.739 194.75 0.00649111 0.00511664 2024-02-05T06:51:22.342 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:22.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:22.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:22.444 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:22.486 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:22.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:22.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:22.588 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:22.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:22.665 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:22.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:22.766 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:22.794 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:22.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:22.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:22.865 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:51:22.866 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:22 smithi028 ceph-mon[29442]: 6.1f scrub starts 2024-02-05T06:51:22.866 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:22 smithi028 ceph-mon[29442]: 5.10 scrub starts 2024-02-05T06:51:22.866 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:22 smithi028 ceph-mon[29442]: 5.10 scrub ok 2024-02-05T06:51:22.866 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:22 smithi028 ceph-mon[29442]: 4.a scrub starts 2024-02-05T06:51:22.866 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:22 smithi028 ceph-mon[29442]: 4.a scrub ok 2024-02-05T06:51:22.866 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:51:22 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:51:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:51:22.867 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:22 smithi028 ceph-mon[37984]: 6.1f scrub starts 2024-02-05T06:51:22.867 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:22 smithi028 ceph-mon[37984]: 5.10 scrub starts 2024-02-05T06:51:22.867 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:22 smithi028 ceph-mon[37984]: 5.10 scrub ok 2024-02-05T06:51:22.867 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:22 smithi028 ceph-mon[37984]: 4.a scrub starts 2024-02-05T06:51:22.867 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:22 smithi028 ceph-mon[37984]: 4.a scrub ok 2024-02-05T06:51:22.896 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:22.910 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:51:22.941 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:22.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:22.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:22.951 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:51:22.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:22 smithi167 ceph-mon[32774]: 6.1f scrub starts 2024-02-05T06:51:22.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:22 smithi167 ceph-mon[32774]: 5.10 scrub starts 2024-02-05T06:51:22.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:22 smithi167 ceph-mon[32774]: 5.10 scrub ok 2024-02-05T06:51:22.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:22 smithi167 ceph-mon[32774]: 4.a scrub starts 2024-02-05T06:51:22.987 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:22 smithi167 ceph-mon[32774]: 4.a scrub ok 2024-02-05T06:51:22.993 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:51:23.032 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:51:23.043 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:23.074 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:51:23.076 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:23.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:23.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:23.113 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:51:23.144 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:51:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:51:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:51:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:51:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:51:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:51:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:51:23.147 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:51:23.147 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:51:23.178 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:23.218 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:23.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:23.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:23.320 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:23.335 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 2 16 5877 5861 183.143 171.562 0.00408717 0.00544933 2024-02-05T06:51:23.355 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:23.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:23.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:23.459 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:23.499 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:23.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:23.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:23.601 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:23.630 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:51:23 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:51:23.637 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:23.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:23.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:23.739 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:23.781 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:23.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:23.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:23.883 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:23.889 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:23 smithi167 ceph-mon[32774]: 6.1f scrub ok 2024-02-05T06:51:23.889 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:23 smithi167 ceph-mon[32774]: 6.b scrub starts 2024-02-05T06:51:23.889 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:23 smithi167 ceph-mon[32774]: 5.a scrub starts 2024-02-05T06:51:23.889 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:23 smithi167 ceph-mon[32774]: pgmap v859: 158 pgs: 158 active+clean; 95 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 12 MiB/s wr, 190 op/s 2024-02-05T06:51:23.889 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:23 smithi167 ceph-mon[32774]: 5.a scrub ok 2024-02-05T06:51:23.889 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:23 smithi167 ceph-mon[32774]: 6.b scrub ok 2024-02-05T06:51:23.889 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:23 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2354543836' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:51:23.890 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:23 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:51:23.890 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:23 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:51:23.890 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:23 smithi167 ceph-mon[32774]: 4.1e scrub starts 2024-02-05T06:51:23.890 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:23 smithi167 ceph-mon[32774]: 4.1e scrub ok 2024-02-05T06:51:23.890 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:23 smithi167 ceph-mon[32774]: 6.1e scrub starts 2024-02-05T06:51:23.922 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:23.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:23.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:23.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:23 smithi028 ceph-mon[37984]: 6.1f scrub ok 2024-02-05T06:51:23.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:23 smithi028 ceph-mon[37984]: 6.b scrub starts 2024-02-05T06:51:23.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:23 smithi028 ceph-mon[37984]: 5.a scrub starts 2024-02-05T06:51:23.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:23 smithi028 ceph-mon[37984]: pgmap v859: 158 pgs: 158 active+clean; 95 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 12 MiB/s wr, 190 op/s 2024-02-05T06:51:23.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:23 smithi028 ceph-mon[37984]: 5.a scrub ok 2024-02-05T06:51:23.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:23 smithi028 ceph-mon[37984]: 6.b scrub ok 2024-02-05T06:51:23.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:23 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2354543836' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:51:23.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:23 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:51:23.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:23 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:51:23.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:23 smithi028 ceph-mon[37984]: 4.1e scrub starts 2024-02-05T06:51:23.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:23 smithi028 ceph-mon[37984]: 4.1e scrub ok 2024-02-05T06:51:23.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:23 smithi028 ceph-mon[37984]: 6.1e scrub starts 2024-02-05T06:51:23.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:23 smithi028 ceph-mon[29442]: 6.1f scrub ok 2024-02-05T06:51:23.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:23 smithi028 ceph-mon[29442]: 6.b scrub starts 2024-02-05T06:51:23.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:23 smithi028 ceph-mon[29442]: 5.a scrub starts 2024-02-05T06:51:23.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:23 smithi028 ceph-mon[29442]: pgmap v859: 158 pgs: 158 active+clean; 95 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 12 MiB/s wr, 190 op/s 2024-02-05T06:51:23.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:23 smithi028 ceph-mon[29442]: 5.a scrub ok 2024-02-05T06:51:23.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:23 smithi028 ceph-mon[29442]: 6.b scrub ok 2024-02-05T06:51:23.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:23 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2354543836' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:51:23.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:23 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:51:23.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:23 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:51:23.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:23 smithi028 ceph-mon[29442]: 4.1e scrub starts 2024-02-05T06:51:23.954 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:23 smithi028 ceph-mon[29442]: 4.1e scrub ok 2024-02-05T06:51:23.955 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:23 smithi028 ceph-mon[29442]: 6.1e scrub starts 2024-02-05T06:51:24.024 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:24.062 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:24.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:24.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:24.084 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:51:24.084 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:51:24.138 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:51:24.164 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:24.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:24.188 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:24.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:24.289 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:24.330 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:24.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:24.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:24.335 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 3 16 8520 8504 177.153 165.188 0.00417188 0.00563376 2024-02-05T06:51:24.432 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:24.467 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:24.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:24.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:24.569 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:24.608 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:24.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:24.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:24.710 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:24.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:24.736 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:24.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:24.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:24 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:51:24.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:24 smithi167 ceph-mon[32774]: 3.1e scrub starts 2024-02-05T06:51:24.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:24 smithi167 ceph-mon[32774]: 3.1e scrub ok 2024-02-05T06:51:24.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:24 smithi167 ceph-mon[32774]: 6.1e scrub ok 2024-02-05T06:51:24.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:24 smithi167 ceph-mon[32774]: 6.1 scrub starts 2024-02-05T06:51:24.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:24 smithi167 ceph-mon[32774]: osdmap e423: 8 total, 8 up, 8 in 2024-02-05T06:51:24.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:24 smithi167 ceph-mon[32774]: 6.1 scrub ok 2024-02-05T06:51:24.837 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:24.870 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:24.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:24.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:24.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:24 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:51:24.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:24 smithi028 ceph-mon[37984]: 3.1e scrub starts 2024-02-05T06:51:24.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:24 smithi028 ceph-mon[37984]: 3.1e scrub ok 2024-02-05T06:51:24.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:24 smithi028 ceph-mon[37984]: 6.1e scrub ok 2024-02-05T06:51:24.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:24 smithi028 ceph-mon[37984]: 6.1 scrub starts 2024-02-05T06:51:24.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:24 smithi028 ceph-mon[37984]: osdmap e423: 8 total, 8 up, 8 in 2024-02-05T06:51:24.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:24 smithi028 ceph-mon[37984]: 6.1 scrub ok 2024-02-05T06:51:24.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:24 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:51:24.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:24 smithi028 ceph-mon[29442]: 3.1e scrub starts 2024-02-05T06:51:24.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:24 smithi028 ceph-mon[29442]: 3.1e scrub ok 2024-02-05T06:51:24.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:24 smithi028 ceph-mon[29442]: 6.1e scrub ok 2024-02-05T06:51:24.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:24 smithi028 ceph-mon[29442]: 6.1 scrub starts 2024-02-05T06:51:24.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:24 smithi028 ceph-mon[29442]: osdmap e423: 8 total, 8 up, 8 in 2024-02-05T06:51:24.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:24 smithi028 ceph-mon[29442]: 6.1 scrub ok 2024-02-05T06:51:24.972 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:24.974 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-05T06:51:25.010 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:25.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:25.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:25.112 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:25.151 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:25.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:25.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:25.253 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:25.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:25.289 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:25.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:25.309 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:51:25.336 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 4 16 11038 11022 172.204 157.375 0.00744121 0.00580022 2024-02-05T06:51:25.390 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:25.432 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:25.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:25.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:25.534 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:25.558 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:25.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:25.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:25.660 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:25.692 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:25.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:25.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:25.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:25 smithi167 ceph-mon[32774]: pgmap v861: 158 pgs: 158 active+clean; 162 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 737 B/s rd, 23 MiB/s wr, 372 op/s 2024-02-05T06:51:25.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:25 smithi167 ceph-mon[32774]: 5.18 scrub starts 2024-02-05T06:51:25.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:25 smithi167 ceph-mon[32774]: 5.18 scrub ok 2024-02-05T06:51:25.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:25 smithi167 ceph-mon[32774]: 3.19 scrub starts 2024-02-05T06:51:25.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:25 smithi167 ceph-mon[32774]: 3.19 scrub ok 2024-02-05T06:51:25.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:25 smithi167 ceph-mon[32774]: 4.e scrub starts 2024-02-05T06:51:25.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:25 smithi167 ceph-mon[32774]: 4.e scrub ok 2024-02-05T06:51:25.794 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:25.835 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:25.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:25.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:25.937 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:25.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:25 smithi028 ceph-mon[29442]: pgmap v861: 158 pgs: 158 active+clean; 162 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 737 B/s rd, 23 MiB/s wr, 372 op/s 2024-02-05T06:51:25.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:25 smithi028 ceph-mon[29442]: 5.18 scrub starts 2024-02-05T06:51:25.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:25 smithi028 ceph-mon[29442]: 5.18 scrub ok 2024-02-05T06:51:25.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:25 smithi028 ceph-mon[29442]: 3.19 scrub starts 2024-02-05T06:51:25.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:25 smithi028 ceph-mon[29442]: 3.19 scrub ok 2024-02-05T06:51:25.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:25 smithi028 ceph-mon[29442]: 4.e scrub starts 2024-02-05T06:51:25.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:25 smithi028 ceph-mon[29442]: 4.e scrub ok 2024-02-05T06:51:25.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:25 smithi028 ceph-mon[37984]: pgmap v861: 158 pgs: 158 active+clean; 162 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 737 B/s rd, 23 MiB/s wr, 372 op/s 2024-02-05T06:51:25.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:25 smithi028 ceph-mon[37984]: 5.18 scrub starts 2024-02-05T06:51:25.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:25 smithi028 ceph-mon[37984]: 5.18 scrub ok 2024-02-05T06:51:25.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:25 smithi028 ceph-mon[37984]: 3.19 scrub starts 2024-02-05T06:51:25.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:25 smithi028 ceph-mon[37984]: 3.19 scrub ok 2024-02-05T06:51:25.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:25 smithi028 ceph-mon[37984]: 4.e scrub starts 2024-02-05T06:51:25.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:25 smithi028 ceph-mon[37984]: 4.e scrub ok 2024-02-05T06:51:25.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:25.993 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:25.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:26.094 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:26.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:26.138 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:26.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:26.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:26.284 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:26.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:26.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:26.336 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 5 16 13469 13453 168.148 151.938 0.008703 0.00593965 2024-02-05T06:51:26.387 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:26.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:26.422 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:26.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:26.523 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:26.554 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:26.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:26.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:26.656 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:26.687 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:26.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:26.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:26.789 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:26.826 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:26.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:26.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:26.928 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:26.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:26 smithi028 ceph-mon[37984]: 4.4 scrub starts 2024-02-05T06:51:26.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:26 smithi028 ceph-mon[37984]: 4.4 scrub ok 2024-02-05T06:51:26.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:26 smithi028 ceph-mon[37984]: 6.d scrub starts 2024-02-05T06:51:26.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:26 smithi028 ceph-mon[37984]: 6.d scrub ok 2024-02-05T06:51:26.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:26 smithi028 ceph-mon[29442]: 4.4 scrub starts 2024-02-05T06:51:26.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:26 smithi028 ceph-mon[29442]: 4.4 scrub ok 2024-02-05T06:51:26.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:26 smithi028 ceph-mon[29442]: 6.d scrub starts 2024-02-05T06:51:26.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:26 smithi028 ceph-mon[29442]: 6.d scrub ok 2024-02-05T06:51:26.973 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:26.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:26.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:26.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:26 smithi167 ceph-mon[32774]: 4.4 scrub starts 2024-02-05T06:51:26.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:26 smithi167 ceph-mon[32774]: 4.4 scrub ok 2024-02-05T06:51:26.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:26 smithi167 ceph-mon[32774]: 6.d scrub starts 2024-02-05T06:51:26.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:26 smithi167 ceph-mon[32774]: 6.d scrub ok 2024-02-05T06:51:27.075 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:27.110 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:27.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:27.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:27.213 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:27.336 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 6 16 15970 15954 166.174 156.312 0.00195185 0.00600112 2024-02-05T06:51:27.434 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:27.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:27.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:27.537 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:27.579 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:27.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:27.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:27.598 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:51:27.598 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:51:27.677 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-02-05T06:51:27.680 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:27.693 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:51:27.710 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:27.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:27.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:27.812 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:27.847 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:27.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:27.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:27.946 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:27 smithi167 ceph-mon[32774]: pgmap v862: 158 pgs: 158 active+clean; 217 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 27 MiB/s wr, 432 op/s 2024-02-05T06:51:27.946 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:27 smithi167 ceph-mon[32774]: 5.19 scrub starts 2024-02-05T06:51:27.946 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:27 smithi167 ceph-mon[32774]: 5.19 scrub ok 2024-02-05T06:51:27.946 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:27 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1264990109' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:51:27.946 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:27 smithi167 ceph-mon[32774]: from='client.25873 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:51:27.946 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:27 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:51:27.946 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:27 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:51:27.946 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:27 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:51:27.949 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:27.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:27 smithi028 ceph-mon[29442]: pgmap v862: 158 pgs: 158 active+clean; 217 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 27 MiB/s wr, 432 op/s 2024-02-05T06:51:27.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:27 smithi028 ceph-mon[29442]: 5.19 scrub starts 2024-02-05T06:51:27.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:27 smithi028 ceph-mon[29442]: 5.19 scrub ok 2024-02-05T06:51:27.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:27 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1264990109' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:51:27.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:27 smithi028 ceph-mon[29442]: from='client.25873 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:51:27.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:27 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:51:27.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:27 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:51:27.953 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:27 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:51:27.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:27 smithi028 ceph-mon[37984]: pgmap v862: 158 pgs: 158 active+clean; 217 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 27 MiB/s wr, 432 op/s 2024-02-05T06:51:27.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:27 smithi028 ceph-mon[37984]: 5.19 scrub starts 2024-02-05T06:51:27.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:27 smithi028 ceph-mon[37984]: 5.19 scrub ok 2024-02-05T06:51:27.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:27 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1264990109' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:51:27.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:27 smithi028 ceph-mon[37984]: from='client.25873 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:51:27.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:27 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:51:27.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:27 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:51:27.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:27 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:51:27.990 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:27.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:27.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:28.092 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:28.143 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:28.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:28.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:28.245 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:28.284 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:28.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:28.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:28.307 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-05T06:51:28.336 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 7 16 18428 18412 164.379 153.625 0.00528209 0.00607724 2024-02-05T06:51:28.387 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:28.425 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:28.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:28.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:28.527 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:28.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:28.564 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:28.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:28.665 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:28.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:28.705 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:28.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:28.718 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:51:28.806 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:28.846 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:28.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:28.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:28.948 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:28.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:28 smithi028 ceph-mon[37984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:51:28.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:28 smithi028 ceph-mon[37984]: Cluster is now healthy 2024-02-05T06:51:28.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:28 smithi028 ceph-mon[37984]: from='client.25873 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:51:28.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:28 smithi028 ceph-mon[37984]: osdmap e424: 8 total, 8 up, 8 in 2024-02-05T06:51:28.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:28 smithi028 ceph-mon[37984]: 4.13 scrub starts 2024-02-05T06:51:28.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:28 smithi028 ceph-mon[37984]: 4.13 scrub ok 2024-02-05T06:51:28.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:28 smithi028 ceph-mon[29442]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:51:28.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:28 smithi028 ceph-mon[29442]: Cluster is now healthy 2024-02-05T06:51:28.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:28 smithi028 ceph-mon[29442]: from='client.25873 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:51:28.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:28 smithi028 ceph-mon[29442]: osdmap e424: 8 total, 8 up, 8 in 2024-02-05T06:51:28.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:28 smithi028 ceph-mon[29442]: 4.13 scrub starts 2024-02-05T06:51:28.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:28 smithi028 ceph-mon[29442]: 4.13 scrub ok 2024-02-05T06:51:28.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:28 smithi167 ceph-mon[32774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:51:28.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:28 smithi167 ceph-mon[32774]: Cluster is now healthy 2024-02-05T06:51:28.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:28 smithi167 ceph-mon[32774]: from='client.25873 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:51:28.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:28 smithi167 ceph-mon[32774]: osdmap e424: 8 total, 8 up, 8 in 2024-02-05T06:51:28.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:28 smithi167 ceph-mon[32774]: 4.13 scrub starts 2024-02-05T06:51:28.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:28 smithi167 ceph-mon[32774]: 4.13 scrub ok 2024-02-05T06:51:28.987 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:28.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:28.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:29.091 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:29.131 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:29.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:29.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:29.233 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:29.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:29.295 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:29.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:29.336 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 8 16 20699 20683 161.572 141.938 0.00988948 0.0061816 2024-02-05T06:51:29.396 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:29.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:29.436 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:29.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:29.537 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:29.586 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:29.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:29.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:29.687 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:29.728 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:29.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:29.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:29.830 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:29.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:29.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:29.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:29 smithi028 ceph-mon[37984]: pgmap v864: 158 pgs: 158 active+clean; 974 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 122 MiB/s wr, 1.95k op/s 2024-02-05T06:51:29.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:29 smithi028 ceph-mon[37984]: 3.13 scrub starts 2024-02-05T06:51:29.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:29 smithi028 ceph-mon[37984]: 3.13 scrub ok 2024-02-05T06:51:29.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:29 smithi028 ceph-mon[29442]: pgmap v864: 158 pgs: 158 active+clean; 974 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 122 MiB/s wr, 1.95k op/s 2024-02-05T06:51:29.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:29 smithi028 ceph-mon[29442]: 3.13 scrub starts 2024-02-05T06:51:29.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:29 smithi028 ceph-mon[29442]: 3.13 scrub ok 2024-02-05T06:51:29.957 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:29 smithi167 ceph-mon[32774]: pgmap v864: 158 pgs: 158 active+clean; 974 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 122 MiB/s wr, 1.95k op/s 2024-02-05T06:51:29.957 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:29 smithi167 ceph-mon[32774]: 3.13 scrub starts 2024-02-05T06:51:29.957 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:29 smithi167 ceph-mon[32774]: 3.13 scrub ok 2024-02-05T06:51:29.961 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:51:29.973 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:30.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:30.028 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:30.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:30.129 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:30.165 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:30.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:30.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:30.267 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:30.299 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:30.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:30.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:30.338 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 9 16 22805 22789 158.221 131.625 0.00836984 0.00631506 2024-02-05T06:51:30.401 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:30.432 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:30.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:30.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:30.439 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:51:30.443 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-05T06:51:30.534 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:30.570 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:30.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:30.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:30.672 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:30.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:30.704 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:30.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:30.805 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:30.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:30.845 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:30.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:30.946 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:30.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:30.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:31.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:31.125 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:31.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:31.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:31.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:31.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:31.263 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:31.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:31.277 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T06:51:31.338 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 10 16 24447 24431 152.661 102.625 0.00416081 0.00654339 2024-02-05T06:51:31.365 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:31.407 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:31.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:31.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:31.454 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:51:31.455 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:51:31.455 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:31 smithi167 ceph-mon[32774]: 6.9 scrub starts 2024-02-05T06:51:31.455 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:31 smithi167 ceph-mon[32774]: from='client.16002 v1:172.21.15.28:0/2709044650' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:51:31.455 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:31 smithi167 ceph-mon[32774]: pgmap v865: 158 pgs: 158 active+clean; 974 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 110 MiB/s wr, 1.76k op/s 2024-02-05T06:51:31.455 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:31 smithi167 ceph-mon[32774]: 5.13 deep-scrub starts 2024-02-05T06:51:31.455 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:31 smithi167 ceph-mon[32774]: 5.13 deep-scrub ok 2024-02-05T06:51:31.510 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:31.549 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:31.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:31.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:31.561 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:31 smithi028 ceph-mon[37984]: 6.9 scrub starts 2024-02-05T06:51:31.561 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:31 smithi028 ceph-mon[37984]: from='client.16002 v1:172.21.15.28:0/2709044650' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:51:31.561 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:31 smithi028 ceph-mon[37984]: pgmap v865: 158 pgs: 158 active+clean; 974 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 110 MiB/s wr, 1.76k op/s 2024-02-05T06:51:31.561 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:31 smithi028 ceph-mon[37984]: 5.13 deep-scrub starts 2024-02-05T06:51:31.562 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:31 smithi028 ceph-mon[37984]: 5.13 deep-scrub ok 2024-02-05T06:51:31.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:31 smithi028 ceph-mon[29442]: 6.9 scrub starts 2024-02-05T06:51:31.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:31 smithi028 ceph-mon[29442]: from='client.16002 v1:172.21.15.28:0/2709044650' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:51:31.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:31 smithi028 ceph-mon[29442]: pgmap v865: 158 pgs: 158 active+clean; 974 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 110 MiB/s wr, 1.76k op/s 2024-02-05T06:51:31.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:31 smithi028 ceph-mon[29442]: 5.13 deep-scrub starts 2024-02-05T06:51:31.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:31 smithi028 ceph-mon[29442]: 5.13 deep-scrub ok 2024-02-05T06:51:31.651 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:31.685 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:31.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:31.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:31.787 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:31.826 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:31.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:31.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:31.928 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:31.964 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:31.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:31.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:32.066 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:32.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:32.100 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:32.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:32.201 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:32.237 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:32.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:32.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:32.338 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 11 16 26532 26516 150.628 130.312 0.00454393 0.00662378 2024-02-05T06:51:32.338 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:32.361 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:32.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:32.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:32.463 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:32.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:32 smithi167 ceph-mon[32774]: 6.9 scrub ok 2024-02-05T06:51:32.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:32 smithi167 ceph-mon[32774]: 5.11 scrub starts 2024-02-05T06:51:32.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:32 smithi167 ceph-mon[32774]: 5.11 scrub ok 2024-02-05T06:51:32.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:32 smithi167 ceph-mon[32774]: 3.1d scrub starts 2024-02-05T06:51:32.478 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:32 smithi167 ceph-mon[32774]: 3.1d scrub ok 2024-02-05T06:51:32.504 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:32.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:32.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:32.533 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-05T06:51:32.606 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:32.643 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:32.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:32.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:32.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:32 smithi028 ceph-mon[29442]: 6.9 scrub ok 2024-02-05T06:51:32.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:32 smithi028 ceph-mon[29442]: 5.11 scrub starts 2024-02-05T06:51:32.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:32 smithi028 ceph-mon[29442]: 5.11 scrub ok 2024-02-05T06:51:32.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:32 smithi028 ceph-mon[29442]: 3.1d scrub starts 2024-02-05T06:51:32.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:32 smithi028 ceph-mon[29442]: 3.1d scrub ok 2024-02-05T06:51:32.702 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:51:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:51:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:51:32.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:32 smithi028 ceph-mon[37984]: 6.9 scrub ok 2024-02-05T06:51:32.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:32 smithi028 ceph-mon[37984]: 5.11 scrub starts 2024-02-05T06:51:32.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:32 smithi028 ceph-mon[37984]: 5.11 scrub ok 2024-02-05T06:51:32.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:32 smithi028 ceph-mon[37984]: 3.1d scrub starts 2024-02-05T06:51:32.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:32 smithi028 ceph-mon[37984]: 3.1d scrub ok 2024-02-05T06:51:32.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:32.783 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:32.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:32.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:32.885 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:32.920 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:32.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:32.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:32.955 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:51:33.023 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:33.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:33.075 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:33.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:33.176 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:33.205 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:33.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:33.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:33.307 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:33.338 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 12 16 28183 28167 146.669 103.188 0.00685164 0.00681167 2024-02-05T06:51:33.343 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:33.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:33.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:33.424 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:33 smithi167 ceph-mon[32774]: 4.11 scrub starts 2024-02-05T06:51:33.424 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:33 smithi167 ceph-mon[32774]: 4.11 scrub ok 2024-02-05T06:51:33.424 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:33 smithi167 ceph-mon[32774]: pgmap v866: 158 pgs: 158 active+clean; 1.4 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 147 MiB/s wr, 2.36k op/s 2024-02-05T06:51:33.424 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:33 smithi167 ceph-mon[32774]: 6.5 scrub starts 2024-02-05T06:51:33.424 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:33 smithi167 ceph-mon[32774]: 6.5 scrub ok 2024-02-05T06:51:33.446 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:33.482 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:33.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:33.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:33.584 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:33.616 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:33.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:33.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:33.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:33 smithi028 ceph-mon[29442]: 4.11 scrub starts 2024-02-05T06:51:33.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:33 smithi028 ceph-mon[29442]: 4.11 scrub ok 2024-02-05T06:51:33.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:33 smithi028 ceph-mon[29442]: pgmap v866: 158 pgs: 158 active+clean; 1.4 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 147 MiB/s wr, 2.36k op/s 2024-02-05T06:51:33.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:33 smithi028 ceph-mon[29442]: 6.5 scrub starts 2024-02-05T06:51:33.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:33 smithi028 ceph-mon[29442]: 6.5 scrub ok 2024-02-05T06:51:33.703 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:33 smithi028 ceph-mon[37984]: 4.11 scrub starts 2024-02-05T06:51:33.703 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:33 smithi028 ceph-mon[37984]: 4.11 scrub ok 2024-02-05T06:51:33.703 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:33 smithi028 ceph-mon[37984]: pgmap v866: 158 pgs: 158 active+clean; 1.4 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 147 MiB/s wr, 2.36k op/s 2024-02-05T06:51:33.703 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:33 smithi028 ceph-mon[37984]: 6.5 scrub starts 2024-02-05T06:51:33.703 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:33 smithi028 ceph-mon[37984]: 6.5 scrub ok 2024-02-05T06:51:33.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:33.735 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:51:33 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:51:33.756 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:33.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:33.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:33.858 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:33.896 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:33.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:33.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:33.998 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:34.028 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:34.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:34.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:34.130 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:34.170 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:34.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:34.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:34.272 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:34.321 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:34.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:34.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:34.338 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 13 16 30092 30076 144.564 119.312 0.00413556 0.0069119 2024-02-05T06:51:34.414 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:34 smithi167 ceph-mon[32774]: 1.6 scrub starts 2024-02-05T06:51:34.414 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:34 smithi167 ceph-mon[32774]: 1.6 scrub ok 2024-02-05T06:51:34.414 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:34 smithi167 ceph-mon[32774]: 3.1a scrub starts 2024-02-05T06:51:34.415 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:34 smithi167 ceph-mon[32774]: 3.1a scrub ok 2024-02-05T06:51:34.415 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:34 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:51:34.415 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:34 smithi167 ceph-mon[32774]: 4.10 scrub starts 2024-02-05T06:51:34.415 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:34 smithi167 ceph-mon[32774]: 4.10 scrub ok 2024-02-05T06:51:34.415 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:34 smithi167 ceph-mon[32774]: 3.15 deep-scrub starts 2024-02-05T06:51:34.415 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:34 smithi167 ceph-mon[32774]: 3.15 deep-scrub ok 2024-02-05T06:51:34.423 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:34.462 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:34.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:34.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:34.564 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:34.605 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:34.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:34.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:34.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:34 smithi028 ceph-mon[37984]: 1.6 scrub starts 2024-02-05T06:51:34.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:34 smithi028 ceph-mon[37984]: 1.6 scrub ok 2024-02-05T06:51:34.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:34 smithi028 ceph-mon[37984]: 3.1a scrub starts 2024-02-05T06:51:34.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:34 smithi028 ceph-mon[37984]: 3.1a scrub ok 2024-02-05T06:51:34.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:34 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:51:34.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:34 smithi028 ceph-mon[37984]: 4.10 scrub starts 2024-02-05T06:51:34.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:34 smithi028 ceph-mon[37984]: 4.10 scrub ok 2024-02-05T06:51:34.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:34 smithi028 ceph-mon[37984]: 3.15 deep-scrub starts 2024-02-05T06:51:34.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:34 smithi028 ceph-mon[37984]: 3.15 deep-scrub ok 2024-02-05T06:51:34.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:34 smithi028 ceph-mon[29442]: 1.6 scrub starts 2024-02-05T06:51:34.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:34 smithi028 ceph-mon[29442]: 1.6 scrub ok 2024-02-05T06:51:34.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:34 smithi028 ceph-mon[29442]: 3.1a scrub starts 2024-02-05T06:51:34.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:34 smithi028 ceph-mon[29442]: 3.1a scrub ok 2024-02-05T06:51:34.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:34 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:51:34.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:34 smithi028 ceph-mon[29442]: 4.10 scrub starts 2024-02-05T06:51:34.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:34 smithi028 ceph-mon[29442]: 4.10 scrub ok 2024-02-05T06:51:34.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:34 smithi028 ceph-mon[29442]: 3.15 deep-scrub starts 2024-02-05T06:51:34.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:34 smithi028 ceph-mon[29442]: 3.15 deep-scrub ok 2024-02-05T06:51:34.707 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:34.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:34.739 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:34.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:34.840 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:34.874 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:34.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:34.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:34.930 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:51:34.931 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:51:34.976 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:35.017 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:35.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:35.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:35.119 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:35.152 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:35.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:35.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:35.254 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:35.288 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:35.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:35.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:35.338 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 14 16 32048 32032 142.97 122.25 0.00439552 0.00698871 2024-02-05T06:51:35.390 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:35.423 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:35.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:35.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:35.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:35 smithi028 ceph-mon[29442]: 4.1 scrub starts 2024-02-05T06:51:35.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:35 smithi028 ceph-mon[29442]: 4.1 scrub ok 2024-02-05T06:51:35.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:35 smithi028 ceph-mon[29442]: pgmap v867: 158 pgs: 158 active+clean; 1.6 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 148 MiB/s wr, 2.36k op/s 2024-02-05T06:51:35.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:35 smithi028 ceph-mon[29442]: 5.1c deep-scrub starts 2024-02-05T06:51:35.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:35 smithi028 ceph-mon[29442]: 3.e deep-scrub starts 2024-02-05T06:51:35.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:35 smithi028 ceph-mon[29442]: 3.e deep-scrub ok 2024-02-05T06:51:35.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:35 smithi028 ceph-mon[37984]: 4.1 scrub starts 2024-02-05T06:51:35.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:35 smithi028 ceph-mon[37984]: 4.1 scrub ok 2024-02-05T06:51:35.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:35 smithi028 ceph-mon[37984]: pgmap v867: 158 pgs: 158 active+clean; 1.6 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 148 MiB/s wr, 2.36k op/s 2024-02-05T06:51:35.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:35 smithi028 ceph-mon[37984]: 5.1c deep-scrub starts 2024-02-05T06:51:35.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:35 smithi028 ceph-mon[37984]: 3.e deep-scrub starts 2024-02-05T06:51:35.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:35 smithi028 ceph-mon[37984]: 3.e deep-scrub ok 2024-02-05T06:51:35.525 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:35.560 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:35.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:35.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:35.620 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:35 smithi167 ceph-mon[32774]: 4.1 scrub starts 2024-02-05T06:51:35.620 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:35 smithi167 ceph-mon[32774]: 4.1 scrub ok 2024-02-05T06:51:35.621 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:35 smithi167 ceph-mon[32774]: pgmap v867: 158 pgs: 158 active+clean; 1.6 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 148 MiB/s wr, 2.36k op/s 2024-02-05T06:51:35.621 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:35 smithi167 ceph-mon[32774]: 5.1c deep-scrub starts 2024-02-05T06:51:35.621 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:35 smithi167 ceph-mon[32774]: 3.e deep-scrub starts 2024-02-05T06:51:35.621 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:35 smithi167 ceph-mon[32774]: 3.e deep-scrub ok 2024-02-05T06:51:35.662 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:35.668 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-02-05T06:51:35.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:35.702 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:35.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:35.803 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:35.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:35.850 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:35.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:35.951 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:35.999 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:36.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:36.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:36.102 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:36.133 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:36.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:36.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:36.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:36.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:36.277 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:36.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:36.338 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 15 16 33994 33978 141.546 121.625 0.00853482 0.00705826 2024-02-05T06:51:36.378 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:36.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:36.423 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:36.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:36.433 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:36 smithi028 ceph-mon[29442]: 5.1c deep-scrub ok 2024-02-05T06:51:36.434 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:36 smithi028 ceph-mon[37984]: 5.1c deep-scrub ok 2024-02-05T06:51:36.487 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":864,"stamp":"2024-02-05T06:51:28.540984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1021253980,"num_objects":15772,"num_object_clones":0,"num_object_copies":47316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15772,"num_whiteouts":0,"num_read":1785,"num_read_kb":1537,"num_write":16296,"num_write_kb":996781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":415,"num_bytes_recovered":1586799,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17881,"ondisk_log_size":17881,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8125916,"kb_used_data":2699548,"kb_used_omap":12,"kb_used_meta":5426291,"kb_avail":741835300,"statfs":{"total":767960285184,"available":759639347200,"internally_reserved":0,"allocated":2764337152,"data_stored":2761163652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":5556522545},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1020657664,"num_objects":15574,"num_object_clones":0,"num_object_copies":46722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15574,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":15574,"num_write_kb":996736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003456"},"pg_stats":[{"pgid":"11.3","version":"423'956","reported_seq":976,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.066019+0000","last_change":"2024-02-05T06:51:13.270132+0000","last_active":"2024-02-05T06:51:27.066019+0000","last_peered":"2024-02-05T06:51:27.066019+0000","last_clean":"2024-02-05T06:51:27.066019+0000","last_became_active":"2024-02-05T06:51:13.269833+0000","last_became_peered":"2024-02-05T06:51:13.269833+0000","last_unstale":"2024-02-05T06:51:27.066019+0000","last_undegraded":"2024-02-05T06:51:27.066019+0000","last_fullsized":"2024-02-05T06:51:27.066019+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:51:12.249449+0000","objects_scrubbed":0,"log_size":956,"log_dups_size":0,"ondisk_log_size":956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:20.214232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62652416,"num_objects":956,"num_object_clones":0,"num_object_copies":2868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":956,"num_write_kb":61184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"295'25","reported_seq":638,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.660926+0000","last_change":"2024-02-05T06:48:34.587509+0000","last_active":"2024-02-05T06:51:27.660926+0000","last_peered":"2024-02-05T06:51:27.660926+0000","last_clean":"2024-02-05T06:51:27.660926+0000","last_became_active":"2024-02-05T06:45:09.796902+0000","last_became_peered":"2024-02-05T06:45:09.796902+0000","last_unstale":"2024-02-05T06:51:27.660926+0000","last_undegraded":"2024-02-05T06:51:27.660926+0000","last_fullsized":"2024-02-05T06:51:27.660926+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'25","last_scrub_stamp":"2024-02-05T06:48:34.587070+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-02-05T06:35:04.584657+0000","last_clean_scrub_stamp":"2024-02-05T06:48:34.587070+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":785,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.663103+0000","last_change":"2024-02-05T06:49:10.852783+0000","last_active":"2024-02-05T06:51:27.663103+0000","last_peered":"2024-02-05T06:51:27.663103+0000","last_clean":"2024-02-05T06:51:27.663103+0000","last_became_active":"2024-02-05T06:36:54.971304+0000","last_became_peered":"2024-02-05T06:36:54.971304+0000","last_unstale":"2024-02-05T06:51:27.663103+0000","last_undegraded":"2024-02-05T06:51:27.663103+0000","last_fullsized":"2024-02-05T06:51:27.663103+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:49:10.851971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:49:10.851971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"249'13","reported_seq":1058,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.618897+0000","last_change":"2024-02-05T06:50:28.852441+0000","last_active":"2024-02-05T06:51:23.618897+0000","last_peered":"2024-02-05T06:51:23.618897+0000","last_clean":"2024-02-05T06:51:23.618897+0000","last_became_active":"2024-02-05T06:41:43.413139+0000","last_became_peered":"2024-02-05T06:41:43.413139+0000","last_unstale":"2024-02-05T06:51:23.618897+0000","last_undegraded":"2024-02-05T06:51:23.618897+0000","last_fullsized":"2024-02-05T06:51:23.618897+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'13","last_scrub_stamp":"2024-02-05T06:50:28.852324+0000","last_deep_scrub":"92'9","last_deep_scrub_stamp":"2024-02-05T06:34:52.283780+0000","last_clean_scrub_stamp":"2024-02-05T06:50:28.852324+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:47.334666+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":310,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.628892+0000","last_change":"2024-02-05T06:50:54.364593+0000","last_active":"2024-02-05T06:51:23.628892+0000","last_peered":"2024-02-05T06:51:23.628892+0000","last_clean":"2024-02-05T06:51:23.628892+0000","last_became_active":"2024-02-05T06:45:09.784229+0000","last_became_peered":"2024-02-05T06:45:09.784229+0000","last_unstale":"2024-02-05T06:51:23.628892+0000","last_undegraded":"2024-02-05T06:51:23.628892+0000","last_fullsized":"2024-02-05T06:51:23.628892+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:54.364481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:30.057099+0000","last_clean_scrub_stamp":"2024-02-05T06:50:54.364481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:08.797872+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":899,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.619613+0000","last_change":"2024-02-05T06:50:43.809383+0000","last_active":"2024-02-05T06:51:23.619613+0000","last_peered":"2024-02-05T06:51:23.619613+0000","last_clean":"2024-02-05T06:51:23.619613+0000","last_became_active":"2024-02-05T06:40:30.095202+0000","last_became_peered":"2024-02-05T06:40:30.095202+0000","last_unstale":"2024-02-05T06:51:23.619613+0000","last_undegraded":"2024-02-05T06:51:23.619613+0000","last_fullsized":"2024-02-05T06:51:23.619613+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:50:43.809271+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:41:50.706159+0000","last_clean_scrub_stamp":"2024-02-05T06:50:43.809271+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:10.746536+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.2","version":"423'1031","reported_seq":1051,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.116014+0000","last_change":"2024-02-05T06:51:13.275881+0000","last_active":"2024-02-05T06:51:27.116014+0000","last_peered":"2024-02-05T06:51:27.116014+0000","last_clean":"2024-02-05T06:51:27.116014+0000","last_became_active":"2024-02-05T06:51:13.275602+0000","last_became_peered":"2024-02-05T06:51:13.275602+0000","last_unstale":"2024-02-05T06:51:27.116014+0000","last_undegraded":"2024-02-05T06:51:27.116014+0000","last_fullsized":"2024-02-05T06:51:27.116014+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:51:12.249449+0000","objects_scrubbed":0,"log_size":1031,"log_dups_size":0,"ondisk_log_size":1031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:34.899689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67567616,"num_objects":1031,"num_object_clones":0,"num_object_copies":3093,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1031,"num_write_kb":65984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"295'39","reported_seq":656,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.618604+0000","last_change":"2024-02-05T06:50:44.705599+0000","last_active":"2024-02-05T06:51:23.618604+0000","last_peered":"2024-02-05T06:51:23.618604+0000","last_clean":"2024-02-05T06:51:23.618604+0000","last_became_active":"2024-02-05T06:45:09.782645+0000","last_became_peered":"2024-02-05T06:45:09.782645+0000","last_unstale":"2024-02-05T06:51:23.618604+0000","last_undegraded":"2024-02-05T06:51:23.618604+0000","last_fullsized":"2024-02-05T06:51:23.618604+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'39","last_scrub_stamp":"2024-02-05T06:50:44.705369+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-02-05T06:35:26.274944+0000","last_clean_scrub_stamp":"2024-02-05T06:50:44.705369+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:04.996962+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":805,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.617777+0000","last_change":"2024-02-05T06:48:27.784796+0000","last_active":"2024-02-05T06:51:23.617777+0000","last_peered":"2024-02-05T06:51:23.617777+0000","last_clean":"2024-02-05T06:51:23.617777+0000","last_became_active":"2024-02-05T06:39:13.240038+0000","last_became_peered":"2024-02-05T06:39:13.240038+0000","last_unstale":"2024-02-05T06:51:23.617777+0000","last_undegraded":"2024-02-05T06:51:23.617777+0000","last_fullsized":"2024-02-05T06:51:23.617777+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:27.784677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:48:27.784677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":791,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.628880+0000","last_change":"2024-02-05T06:50:36.200638+0000","last_active":"2024-02-05T06:51:23.628880+0000","last_peered":"2024-02-05T06:51:23.628880+0000","last_clean":"2024-02-05T06:51:23.628880+0000","last_became_active":"2024-02-05T06:36:54.958082+0000","last_became_peered":"2024-02-05T06:36:54.958082+0000","last_unstale":"2024-02-05T06:51:23.628880+0000","last_undegraded":"2024-02-05T06:51:23.628880+0000","last_fullsized":"2024-02-05T06:51:23.628880+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:36.200485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:50:36.200485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:56.140955+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":802,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.619298+0000","last_change":"2024-02-05T06:50:26.004078+0000","last_active":"2024-02-05T06:51:23.619298+0000","last_peered":"2024-02-05T06:51:23.619298+0000","last_clean":"2024-02-05T06:51:23.619298+0000","last_became_active":"2024-02-05T06:36:54.958805+0000","last_became_peered":"2024-02-05T06:36:54.958805+0000","last_unstale":"2024-02-05T06:51:23.619298+0000","last_undegraded":"2024-02-05T06:51:23.619298+0000","last_fullsized":"2024-02-05T06:51:23.619298+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:26.003863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:50:26.003863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:40.957033+0000","scrub_duration":182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"83'128","reported_seq":811,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.619741+0000","last_change":"2024-02-05T06:48:37.548716+0000","last_active":"2024-02-05T06:51:23.619741+0000","last_peered":"2024-02-05T06:51:23.619741+0000","last_clean":"2024-02-05T06:51:23.619741+0000","last_became_active":"2024-02-05T06:41:46.053324+0000","last_became_peered":"2024-02-05T06:41:46.053324+0000","last_unstale":"2024-02-05T06:51:23.619741+0000","last_undegraded":"2024-02-05T06:51:23.619741+0000","last_fullsized":"2024-02-05T06:51:23.619741+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:48:37.548595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:48:37.548595+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"423'906","reported_seq":926,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.060989+0000","last_change":"2024-02-05T06:51:13.273503+0000","last_active":"2024-02-05T06:51:27.060989+0000","last_peered":"2024-02-05T06:51:27.060989+0000","last_clean":"2024-02-05T06:51:27.060989+0000","last_became_active":"2024-02-05T06:51:13.273200+0000","last_became_peered":"2024-02-05T06:51:13.273200+0000","last_unstale":"2024-02-05T06:51:27.060989+0000","last_undegraded":"2024-02-05T06:51:27.060989+0000","last_fullsized":"2024-02-05T06:51:27.060989+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:51:12.249449+0000","objects_scrubbed":0,"log_size":906,"log_dups_size":0,"ondisk_log_size":906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:28.743396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59375616,"num_objects":906,"num_object_clones":0,"num_object_copies":2718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":906,"num_write_kb":57984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"295'24","reported_seq":649,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:25.545219+0000","last_change":"2024-02-05T06:51:25.545249+0000","last_active":"2024-02-05T06:51:25.545219+0000","last_peered":"2024-02-05T06:51:25.545219+0000","last_clean":"2024-02-05T06:51:25.545219+0000","last_became_active":"2024-02-05T06:45:09.774868+0000","last_became_peered":"2024-02-05T06:45:09.774868+0000","last_unstale":"2024-02-05T06:51:25.545219+0000","last_undegraded":"2024-02-05T06:51:25.545219+0000","last_fullsized":"2024-02-05T06:51:25.545219+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:51:25.545165+0000","last_deep_scrub":"115'16","last_deep_scrub_stamp":"2024-02-05T06:36:34.793243+0000","last_clean_scrub_stamp":"2024-02-05T06:51:25.545165+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:29.148376+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":500,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.626684+0000","last_change":"2024-02-05T06:48:38.056060+0000","last_active":"2024-02-05T06:51:23.626684+0000","last_peered":"2024-02-05T06:51:23.626684+0000","last_clean":"2024-02-05T06:51:23.626684+0000","last_became_active":"2024-02-05T06:40:27.086808+0000","last_became_peered":"2024-02-05T06:40:27.086808+0000","last_unstale":"2024-02-05T06:51:23.626684+0000","last_undegraded":"2024-02-05T06:51:23.626684+0000","last_fullsized":"2024-02-05T06:51:23.626684+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:38.055933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:48:38.055933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":396,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.621284+0000","last_change":"2024-02-05T06:48:38.583638+0000","last_active":"2024-02-05T06:51:23.621284+0000","last_peered":"2024-02-05T06:51:23.621284+0000","last_clean":"2024-02-05T06:51:23.621284+0000","last_became_active":"2024-02-05T06:41:43.417343+0000","last_became_peered":"2024-02-05T06:41:43.417343+0000","last_unstale":"2024-02-05T06:51:23.621284+0000","last_undegraded":"2024-02-05T06:51:23.621284+0000","last_fullsized":"2024-02-05T06:51:23.621284+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:38.583376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:42:55.610276+0000","last_clean_scrub_stamp":"2024-02-05T06:48:38.583376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":779,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.660639+0000","last_change":"2024-02-05T06:50:53.397399+0000","last_active":"2024-02-05T06:51:27.660639+0000","last_peered":"2024-02-05T06:51:27.660639+0000","last_clean":"2024-02-05T06:51:27.660639+0000","last_became_active":"2024-02-05T06:41:43.410600+0000","last_became_peered":"2024-02-05T06:41:43.410600+0000","last_unstale":"2024-02-05T06:51:27.660639+0000","last_undegraded":"2024-02-05T06:51:27.660639+0000","last_fullsized":"2024-02-05T06:51:27.660639+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:53.395911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:58.479208+0000","last_clean_scrub_stamp":"2024-02-05T06:50:53.395911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:05.372056+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"423'975","reported_seq":995,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.293776+0000","last_change":"2024-02-05T06:51:13.273741+0000","last_active":"2024-02-05T06:51:27.293776+0000","last_peered":"2024-02-05T06:51:27.293776+0000","last_clean":"2024-02-05T06:51:27.293776+0000","last_became_active":"2024-02-05T06:51:13.273376+0000","last_became_peered":"2024-02-05T06:51:13.273376+0000","last_unstale":"2024-02-05T06:51:27.293776+0000","last_undegraded":"2024-02-05T06:51:27.293776+0000","last_fullsized":"2024-02-05T06:51:27.293776+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:51:12.249449+0000","objects_scrubbed":0,"log_size":975,"log_dups_size":0,"ondisk_log_size":975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:28.994991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63897600,"num_objects":975,"num_object_clones":0,"num_object_copies":2925,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":975,"num_write_kb":62400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"295'34","reported_seq":587,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.612990+0000","last_change":"2024-02-05T06:48:37.385567+0000","last_active":"2024-02-05T06:51:23.612990+0000","last_peered":"2024-02-05T06:51:23.612990+0000","last_clean":"2024-02-05T06:51:23.612990+0000","last_became_active":"2024-02-05T06:39:13.257978+0000","last_became_peered":"2024-02-05T06:39:13.257978+0000","last_unstale":"2024-02-05T06:51:23.612990+0000","last_undegraded":"2024-02-05T06:51:23.612990+0000","last_fullsized":"2024-02-05T06:51:23.612990+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'34","last_scrub_stamp":"2024-02-05T06:48:37.385422+0000","last_deep_scrub":"295'34","last_deep_scrub_stamp":"2024-02-05T06:46:44.355563+0000","last_clean_scrub_stamp":"2024-02-05T06:48:37.385422+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":786,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.661063+0000","last_change":"2024-02-05T06:48:14.631495+0000","last_active":"2024-02-05T06:51:27.661063+0000","last_peered":"2024-02-05T06:51:27.661063+0000","last_clean":"2024-02-05T06:51:27.661063+0000","last_became_active":"2024-02-05T06:40:27.213439+0000","last_became_peered":"2024-02-05T06:40:27.213439+0000","last_unstale":"2024-02-05T06:51:27.661063+0000","last_undegraded":"2024-02-05T06:51:27.661063+0000","last_fullsized":"2024-02-05T06:51:27.661063+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:14.631382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:52.482566+0000","last_clean_scrub_stamp":"2024-02-05T06:48:14.631382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"319'13","reported_seq":670,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.620719+0000","last_change":"2024-02-05T06:50:38.776072+0000","last_active":"2024-02-05T06:51:23.620719+0000","last_peered":"2024-02-05T06:51:23.620719+0000","last_clean":"2024-02-05T06:51:23.620719+0000","last_became_active":"2024-02-05T06:45:09.766220+0000","last_became_peered":"2024-02-05T06:45:09.766220+0000","last_unstale":"2024-02-05T06:51:23.620719+0000","last_undegraded":"2024-02-05T06:51:23.620719+0000","last_fullsized":"2024-02-05T06:51:23.620719+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'13","last_scrub_stamp":"2024-02-05T06:50:38.775962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:50:38.775962+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:51.510274+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":404,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:26.027758+0000","last_change":"2024-02-05T06:51:26.027783+0000","last_active":"2024-02-05T06:51:26.027758+0000","last_peered":"2024-02-05T06:51:26.027758+0000","last_clean":"2024-02-05T06:51:26.027758+0000","last_became_active":"2024-02-05T06:41:43.418066+0000","last_became_peered":"2024-02-05T06:41:43.418066+0000","last_unstale":"2024-02-05T06:51:26.027758+0000","last_undegraded":"2024-02-05T06:51:26.027758+0000","last_fullsized":"2024-02-05T06:51:26.027758+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:26.027679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:48:34.623889+0000","last_clean_scrub_stamp":"2024-02-05T06:51:26.027679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:26.586347+0000","scrub_duration":205,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":815,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.620179+0000","last_change":"2024-02-05T06:50:51.036149+0000","last_active":"2024-02-05T06:51:23.620179+0000","last_peered":"2024-02-05T06:51:23.620179+0000","last_clean":"2024-02-05T06:51:23.620179+0000","last_became_active":"2024-02-05T06:41:46.053993+0000","last_became_peered":"2024-02-05T06:41:46.053993+0000","last_unstale":"2024-02-05T06:51:23.620179+0000","last_undegraded":"2024-02-05T06:51:23.620179+0000","last_fullsized":"2024-02-05T06:51:23.620179+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:50:51.035953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:50:51.035953+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:02.122313+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"423'920","reported_seq":940,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:26.731054+0000","last_change":"2024-02-05T06:51:13.274549+0000","last_active":"2024-02-05T06:51:26.731054+0000","last_peered":"2024-02-05T06:51:26.731054+0000","last_clean":"2024-02-05T06:51:26.731054+0000","last_became_active":"2024-02-05T06:51:13.274262+0000","last_became_peered":"2024-02-05T06:51:13.274262+0000","last_unstale":"2024-02-05T06:51:26.731054+0000","last_undegraded":"2024-02-05T06:51:26.731054+0000","last_fullsized":"2024-02-05T06:51:26.731054+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:51:12.249449+0000","objects_scrubbed":0,"log_size":920,"log_dups_size":0,"ondisk_log_size":920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:42.186667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60293120,"num_objects":920,"num_object_clones":0,"num_object_copies":2760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":920,"num_write_kb":58880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":446,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.661516+0000","last_change":"2024-02-05T06:50:41.271891+0000","last_active":"2024-02-05T06:51:27.661516+0000","last_peered":"2024-02-05T06:51:27.661516+0000","last_clean":"2024-02-05T06:51:27.661516+0000","last_became_active":"2024-02-05T06:45:09.778609+0000","last_became_peered":"2024-02-05T06:45:09.778609+0000","last_unstale":"2024-02-05T06:51:27.661516+0000","last_undegraded":"2024-02-05T06:51:27.661516+0000","last_fullsized":"2024-02-05T06:51:27.661516+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:41.271775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:50:41.271775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:01.517878+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"295'30","reported_seq":860,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.661148+0000","last_change":"2024-02-05T06:47:50.402454+0000","last_active":"2024-02-05T06:51:27.661148+0000","last_peered":"2024-02-05T06:51:27.661148+0000","last_clean":"2024-02-05T06:51:27.661148+0000","last_became_active":"2024-02-05T06:40:27.277747+0000","last_became_peered":"2024-02-05T06:40:27.277747+0000","last_unstale":"2024-02-05T06:51:27.661148+0000","last_undegraded":"2024-02-05T06:51:27.661148+0000","last_fullsized":"2024-02-05T06:51:27.661148+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'30","last_scrub_stamp":"2024-02-05T06:47:50.402305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:47:50.402305+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":792,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.662427+0000","last_change":"2024-02-05T06:49:13.868761+0000","last_active":"2024-02-05T06:51:27.662427+0000","last_peered":"2024-02-05T06:51:27.662427+0000","last_clean":"2024-02-05T06:51:27.662427+0000","last_became_active":"2024-02-05T06:40:27.093739+0000","last_became_peered":"2024-02-05T06:40:27.093739+0000","last_unstale":"2024-02-05T06:51:27.662427+0000","last_undegraded":"2024-02-05T06:51:27.662427+0000","last_fullsized":"2024-02-05T06:51:27.662427+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:49:13.868664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:38:28.150695+0000","last_clean_scrub_stamp":"2024-02-05T06:49:13.868664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":305,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.661566+0000","last_change":"2024-02-05T06:50:33.252884+0000","last_active":"2024-02-05T06:51:27.661566+0000","last_peered":"2024-02-05T06:51:27.661566+0000","last_clean":"2024-02-05T06:51:27.661566+0000","last_became_active":"2024-02-05T06:45:09.780390+0000","last_became_peered":"2024-02-05T06:45:09.780390+0000","last_unstale":"2024-02-05T06:51:27.661566+0000","last_undegraded":"2024-02-05T06:51:27.661566+0000","last_fullsized":"2024-02-05T06:51:27.661566+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:33.252762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:45:49.145117+0000","last_clean_scrub_stamp":"2024-02-05T06:50:33.252762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:33.730321+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":546,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.618478+0000","last_change":"2024-02-05T06:50:40.671543+0000","last_active":"2024-02-05T06:51:23.618478+0000","last_peered":"2024-02-05T06:51:23.618478+0000","last_clean":"2024-02-05T06:51:23.618478+0000","last_became_active":"2024-02-05T06:45:09.776631+0000","last_became_peered":"2024-02-05T06:45:09.776631+0000","last_unstale":"2024-02-05T06:51:23.618478+0000","last_undegraded":"2024-02-05T06:51:23.618478+0000","last_fullsized":"2024-02-05T06:51:23.618478+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:50:40.671434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:50:40.671434+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:06.630089+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"423'1024","reported_seq":1044,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.332642+0000","last_change":"2024-02-05T06:51:13.267996+0000","last_active":"2024-02-05T06:51:27.332642+0000","last_peered":"2024-02-05T06:51:27.332642+0000","last_clean":"2024-02-05T06:51:27.332642+0000","last_became_active":"2024-02-05T06:51:13.267644+0000","last_became_peered":"2024-02-05T06:51:13.267644+0000","last_unstale":"2024-02-05T06:51:27.332642+0000","last_undegraded":"2024-02-05T06:51:27.332642+0000","last_fullsized":"2024-02-05T06:51:27.332642+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:51:12.249449+0000","objects_scrubbed":0,"log_size":1024,"log_dups_size":0,"ondisk_log_size":1024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:36.283170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":1024,"num_object_clones":0,"num_object_copies":3072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1024,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"326'11","reported_seq":1013,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.661244+0000","last_change":"2024-02-05T06:50:37.301724+0000","last_active":"2024-02-05T06:51:27.661244+0000","last_peered":"2024-02-05T06:51:27.661244+0000","last_clean":"2024-02-05T06:51:27.661244+0000","last_became_active":"2024-02-05T06:36:54.954819+0000","last_became_peered":"2024-02-05T06:36:54.954819+0000","last_unstale":"2024-02-05T06:51:27.661244+0000","last_undegraded":"2024-02-05T06:51:27.661244+0000","last_fullsized":"2024-02-05T06:51:27.661244+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'11","last_scrub_stamp":"2024-02-05T06:50:37.301610+0000","last_deep_scrub":"326'11","last_deep_scrub_stamp":"2024-02-05T06:45:56.212177+0000","last_clean_scrub_stamp":"2024-02-05T06:50:37.301610+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:02.978159+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"295'28","reported_seq":699,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.619475+0000","last_change":"2024-02-05T06:48:17.616455+0000","last_active":"2024-02-05T06:51:23.619475+0000","last_peered":"2024-02-05T06:51:23.619475+0000","last_clean":"2024-02-05T06:51:23.619475+0000","last_became_active":"2024-02-05T06:45:09.764696+0000","last_became_peered":"2024-02-05T06:45:09.764696+0000","last_unstale":"2024-02-05T06:51:23.619475+0000","last_undegraded":"2024-02-05T06:51:23.619475+0000","last_fullsized":"2024-02-05T06:51:23.619475+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'28","last_scrub_stamp":"2024-02-05T06:48:17.615783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:48:17.615783+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":40,"num_read_kb":33,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"95'2","reported_seq":803,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.612983+0000","last_change":"2024-02-05T06:50:53.112300+0000","last_active":"2024-02-05T06:51:23.612983+0000","last_peered":"2024-02-05T06:51:23.612983+0000","last_clean":"2024-02-05T06:51:23.612983+0000","last_became_active":"2024-02-05T06:36:54.975093+0000","last_became_peered":"2024-02-05T06:36:54.975093+0000","last_unstale":"2024-02-05T06:51:23.612983+0000","last_undegraded":"2024-02-05T06:51:23.612983+0000","last_fullsized":"2024-02-05T06:51:23.612983+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-05T06:50:53.112087+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-05T06:35:25.616708+0000","last_clean_scrub_stamp":"2024-02-05T06:50:53.112087+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:10.610915+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":481,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.621197+0000","last_change":"2024-02-05T06:50:32.894236+0000","last_active":"2024-02-05T06:51:23.621197+0000","last_peered":"2024-02-05T06:51:23.621197+0000","last_clean":"2024-02-05T06:51:23.621197+0000","last_became_active":"2024-02-05T06:40:27.099168+0000","last_became_peered":"2024-02-05T06:40:27.099168+0000","last_unstale":"2024-02-05T06:51:23.621197+0000","last_undegraded":"2024-02-05T06:51:23.621197+0000","last_fullsized":"2024-02-05T06:51:23.621197+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:32.894114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:50:32.894114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:40.890366+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":980,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.660471+0000","last_change":"2024-02-05T06:50:28.114612+0000","last_active":"2024-02-05T06:51:27.660471+0000","last_peered":"2024-02-05T06:51:27.660471+0000","last_clean":"2024-02-05T06:51:27.660471+0000","last_became_active":"2024-02-05T06:41:46.053837+0000","last_became_peered":"2024-02-05T06:41:46.053837+0000","last_unstale":"2024-02-05T06:51:27.660471+0000","last_undegraded":"2024-02-05T06:51:27.660471+0000","last_fullsized":"2024-02-05T06:51:27.660471+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:50:28.114084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:50:28.114084+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:30.315333+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1310720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"423'936","reported_seq":956,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.124188+0000","last_change":"2024-02-05T06:51:13.272943+0000","last_active":"2024-02-05T06:51:27.124188+0000","last_peered":"2024-02-05T06:51:27.124188+0000","last_clean":"2024-02-05T06:51:27.124188+0000","last_became_active":"2024-02-05T06:51:13.272662+0000","last_became_peered":"2024-02-05T06:51:13.272662+0000","last_unstale":"2024-02-05T06:51:27.124188+0000","last_undegraded":"2024-02-05T06:51:27.124188+0000","last_fullsized":"2024-02-05T06:51:27.124188+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:51:12.249449+0000","objects_scrubbed":0,"log_size":936,"log_dups_size":0,"ondisk_log_size":936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:38.414728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61341696,"num_objects":936,"num_object_clones":0,"num_object_copies":2808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":936,"num_write_kb":59904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"335'11","reported_seq":1018,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.613715+0000","last_change":"2024-02-05T06:50:34.918746+0000","last_active":"2024-02-05T06:51:23.613715+0000","last_peered":"2024-02-05T06:51:23.613715+0000","last_clean":"2024-02-05T06:51:23.613715+0000","last_became_active":"2024-02-05T06:41:43.423638+0000","last_became_peered":"2024-02-05T06:41:43.423638+0000","last_unstale":"2024-02-05T06:51:23.613715+0000","last_undegraded":"2024-02-05T06:51:23.613715+0000","last_fullsized":"2024-02-05T06:51:23.613715+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"335'11","last_scrub_stamp":"2024-02-05T06:50:34.918636+0000","last_deep_scrub":"335'11","last_deep_scrub_stamp":"2024-02-05T06:46:52.423878+0000","last_clean_scrub_stamp":"2024-02-05T06:50:34.918636+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:38.089289+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"295'20","reported_seq":878,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.628001+0000","last_change":"2024-02-05T06:48:21.064564+0000","last_active":"2024-02-05T06:51:23.628001+0000","last_peered":"2024-02-05T06:51:23.628001+0000","last_clean":"2024-02-05T06:51:23.628001+0000","last_became_active":"2024-02-05T06:41:43.421030+0000","last_became_peered":"2024-02-05T06:41:43.421030+0000","last_unstale":"2024-02-05T06:51:23.628001+0000","last_undegraded":"2024-02-05T06:51:23.628001+0000","last_fullsized":"2024-02-05T06:51:23.628001+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'20","last_scrub_stamp":"2024-02-05T06:48:21.064424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:48:21.064424+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":396,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.620507+0000","last_change":"2024-02-05T06:50:49.997331+0000","last_active":"2024-02-05T06:51:23.620507+0000","last_peered":"2024-02-05T06:51:23.620507+0000","last_clean":"2024-02-05T06:51:23.620507+0000","last_became_active":"2024-02-05T06:41:43.411732+0000","last_became_peered":"2024-02-05T06:41:43.411732+0000","last_unstale":"2024-02-05T06:51:23.620507+0000","last_undegraded":"2024-02-05T06:51:23.620507+0000","last_fullsized":"2024-02-05T06:51:23.620507+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:49.997245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:28.353956+0000","last_clean_scrub_stamp":"2024-02-05T06:50:49.997245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:54.089895+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":761,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.611791+0000","last_change":"2024-02-05T06:48:33.451506+0000","last_active":"2024-02-05T06:51:23.611791+0000","last_peered":"2024-02-05T06:51:23.611791+0000","last_clean":"2024-02-05T06:51:23.611791+0000","last_became_active":"2024-02-05T06:33:35.600485+0000","last_became_peered":"2024-02-05T06:33:35.600485+0000","last_unstale":"2024-02-05T06:51:23.611791+0000","last_undegraded":"2024-02-05T06:51:23.611791+0000","last_fullsized":"2024-02-05T06:51:23.611791+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:33.451272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:48:33.451272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":975,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.614731+0000","last_change":"2024-02-05T06:48:25.726145+0000","last_active":"2024-02-05T06:51:23.614731+0000","last_peered":"2024-02-05T06:51:23.614731+0000","last_clean":"2024-02-05T06:51:23.614731+0000","last_became_active":"2024-02-05T06:40:30.085438+0000","last_became_peered":"2024-02-05T06:40:30.085438+0000","last_unstale":"2024-02-05T06:51:23.614731+0000","last_undegraded":"2024-02-05T06:51:23.614731+0000","last_fullsized":"2024-02-05T06:51:23.614731+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:48:25.725892+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:48:25.725892+0000","last_clean_scrub_stamp":"2024-02-05T06:48:25.725892+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"423'1002","reported_seq":1022,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.316754+0000","last_change":"2024-02-05T06:51:13.269564+0000","last_active":"2024-02-05T06:51:27.316754+0000","last_peered":"2024-02-05T06:51:27.316754+0000","last_clean":"2024-02-05T06:51:27.316754+0000","last_became_active":"2024-02-05T06:51:13.269283+0000","last_became_peered":"2024-02-05T06:51:13.269283+0000","last_unstale":"2024-02-05T06:51:27.316754+0000","last_undegraded":"2024-02-05T06:51:27.316754+0000","last_fullsized":"2024-02-05T06:51:27.316754+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:51:12.249449+0000","objects_scrubbed":0,"log_size":1002,"log_dups_size":0,"ondisk_log_size":1002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:34.978457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65667072,"num_objects":1002,"num_object_clones":0,"num_object_copies":3006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1002,"num_write_kb":64128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":799,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.616303+0000","last_change":"2024-02-05T06:50:54.391997+0000","last_active":"2024-02-05T06:51:23.616303+0000","last_peered":"2024-02-05T06:51:23.616303+0000","last_clean":"2024-02-05T06:51:23.616303+0000","last_became_active":"2024-02-05T06:41:43.411550+0000","last_became_peered":"2024-02-05T06:41:43.411550+0000","last_unstale":"2024-02-05T06:51:23.616303+0000","last_undegraded":"2024-02-05T06:51:23.616303+0000","last_fullsized":"2024-02-05T06:51:23.616303+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:54.391720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:50:54.391720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:56.924327+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"295'29","reported_seq":924,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.614116+0000","last_change":"2024-02-05T06:50:40.988959+0000","last_active":"2024-02-05T06:51:23.614116+0000","last_peered":"2024-02-05T06:51:23.614116+0000","last_clean":"2024-02-05T06:51:23.614116+0000","last_became_active":"2024-02-05T06:41:43.409664+0000","last_became_peered":"2024-02-05T06:41:43.409664+0000","last_unstale":"2024-02-05T06:51:23.614116+0000","last_undegraded":"2024-02-05T06:51:23.614116+0000","last_fullsized":"2024-02-05T06:51:23.614116+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'29","last_scrub_stamp":"2024-02-05T06:50:40.988813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:50:40.988813+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:51.349525+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"66'1","reported_seq":814,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.628799+0000","last_change":"2024-02-05T06:50:35.211701+0000","last_active":"2024-02-05T06:51:23.628799+0000","last_peered":"2024-02-05T06:51:23.628799+0000","last_clean":"2024-02-05T06:51:23.628799+0000","last_became_active":"2024-02-05T06:41:43.411404+0000","last_became_peered":"2024-02-05T06:41:43.411404+0000","last_unstale":"2024-02-05T06:51:23.628799+0000","last_undegraded":"2024-02-05T06:51:23.628799+0000","last_fullsized":"2024-02-05T06:51:23.628799+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:50:35.211584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:50:35.211584+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:48.221813+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":776,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.651224+0000","last_change":"2024-02-05T06:51:23.651271+0000","last_active":"2024-02-05T06:51:23.651224+0000","last_peered":"2024-02-05T06:51:23.651224+0000","last_clean":"2024-02-05T06:51:23.651224+0000","last_became_active":"2024-02-05T06:33:35.664858+0000","last_became_peered":"2024-02-05T06:33:35.664858+0000","last_unstale":"2024-02-05T06:51:23.651224+0000","last_undegraded":"2024-02-05T06:51:23.651224+0000","last_fullsized":"2024-02-05T06:51:23.651224+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:23.651135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:51:23.651135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:42.470104+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":956,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.614873+0000","last_change":"2024-02-05T06:48:17.729708+0000","last_active":"2024-02-05T06:51:23.614873+0000","last_peered":"2024-02-05T06:51:23.614873+0000","last_clean":"2024-02-05T06:51:23.614873+0000","last_became_active":"2024-02-05T06:37:37.140011+0000","last_became_peered":"2024-02-05T06:37:37.140011+0000","last_unstale":"2024-02-05T06:51:23.614873+0000","last_undegraded":"2024-02-05T06:51:23.614873+0000","last_fullsized":"2024-02-05T06:51:23.614873+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:48:17.729582+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:38.778568+0000","last_clean_scrub_stamp":"2024-02-05T06:48:17.729582+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"424'1057","reported_seq":1079,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.752169+0000","last_change":"2024-02-05T06:51:13.271458+0000","last_active":"2024-02-05T06:51:27.752169+0000","last_peered":"2024-02-05T06:51:27.752169+0000","last_clean":"2024-02-05T06:51:27.752169+0000","last_became_active":"2024-02-05T06:51:13.270994+0000","last_became_peered":"2024-02-05T06:51:13.270994+0000","last_unstale":"2024-02-05T06:51:27.752169+0000","last_undegraded":"2024-02-05T06:51:27.752169+0000","last_fullsized":"2024-02-05T06:51:27.752169+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:51:12.249449+0000","objects_scrubbed":0,"log_size":1057,"log_dups_size":0,"ondisk_log_size":1057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:42.221745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69271552,"num_objects":1057,"num_object_clones":0,"num_object_copies":3171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1057,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1057,"num_write_kb":67648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":396,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.621449+0000","last_change":"2024-02-05T06:50:49.033515+0000","last_active":"2024-02-05T06:51:23.621449+0000","last_peered":"2024-02-05T06:51:23.621449+0000","last_clean":"2024-02-05T06:51:23.621449+0000","last_became_active":"2024-02-05T06:41:43.417751+0000","last_became_peered":"2024-02-05T06:41:43.417751+0000","last_unstale":"2024-02-05T06:51:23.621449+0000","last_undegraded":"2024-02-05T06:51:23.621449+0000","last_fullsized":"2024-02-05T06:51:23.621449+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:49.033397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:50:49.033397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:53.482070+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"295'24","reported_seq":813,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.612801+0000","last_change":"2024-02-05T06:50:39.970085+0000","last_active":"2024-02-05T06:51:23.612801+0000","last_peered":"2024-02-05T06:51:23.612801+0000","last_clean":"2024-02-05T06:51:23.612801+0000","last_became_active":"2024-02-05T06:33:31.564604+0000","last_became_peered":"2024-02-05T06:33:31.564604+0000","last_unstale":"2024-02-05T06:51:23.612801+0000","last_undegraded":"2024-02-05T06:51:23.612801+0000","last_fullsized":"2024-02-05T06:51:23.612801+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:50:39.969954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:50:39.969954+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:54.041197+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"355'6","reported_seq":1148,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:25.103216+0000","last_change":"2024-02-05T06:50:34.763443+0000","last_active":"2024-02-05T06:51:25.103216+0000","last_peered":"2024-02-05T06:51:25.103216+0000","last_clean":"2024-02-05T06:51:25.103216+0000","last_became_active":"2024-02-05T06:40:27.071509+0000","last_became_peered":"2024-02-05T06:40:27.071509+0000","last_unstale":"2024-02-05T06:51:25.103216+0000","last_undegraded":"2024-02-05T06:51:25.103216+0000","last_fullsized":"2024-02-05T06:51:25.103216+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'6","last_scrub_stamp":"2024-02-05T06:50:34.763187+0000","last_deep_scrub":"355'6","last_deep_scrub_stamp":"2024-02-05T06:47:46.642454+0000","last_clean_scrub_stamp":"2024-02-05T06:50:34.763187+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:53.178598+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0.00074151600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":665,"num_read_kb":660,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":793,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.612206+0000","last_change":"2024-02-05T06:50:36.946693+0000","last_active":"2024-02-05T06:51:23.612206+0000","last_peered":"2024-02-05T06:51:23.612206+0000","last_clean":"2024-02-05T06:51:23.612206+0000","last_became_active":"2024-02-05T06:36:54.972264+0000","last_became_peered":"2024-02-05T06:36:54.972264+0000","last_unstale":"2024-02-05T06:51:23.612206+0000","last_undegraded":"2024-02-05T06:51:23.612206+0000","last_fullsized":"2024-02-05T06:51:23.612206+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:36.946429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:50:36.946429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:00.471758+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":788,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.628539+0000","last_change":"2024-02-05T06:48:34.143372+0000","last_active":"2024-02-05T06:51:23.628539+0000","last_peered":"2024-02-05T06:51:23.628539+0000","last_clean":"2024-02-05T06:51:23.628539+0000","last_became_active":"2024-02-05T06:41:43.410910+0000","last_became_peered":"2024-02-05T06:41:43.410910+0000","last_unstale":"2024-02-05T06:51:23.628539+0000","last_undegraded":"2024-02-05T06:51:23.628539+0000","last_fullsized":"2024-02-05T06:51:23.628539+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:34.143222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:48:34.143222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":943,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.628229+0000","last_change":"2024-02-05T06:50:27.315358+0000","last_active":"2024-02-05T06:51:23.628229+0000","last_peered":"2024-02-05T06:51:23.628229+0000","last_clean":"2024-02-05T06:51:23.628229+0000","last_became_active":"2024-02-05T06:37:37.137754+0000","last_became_peered":"2024-02-05T06:37:37.137754+0000","last_unstale":"2024-02-05T06:51:23.628229+0000","last_undegraded":"2024-02-05T06:51:23.628229+0000","last_fullsized":"2024-02-05T06:51:23.628229+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:50:27.315164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:50:27.315164+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:57.156322+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"424'1049","reported_seq":1071,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.767846+0000","last_change":"2024-02-05T06:51:13.274707+0000","last_active":"2024-02-05T06:51:27.767846+0000","last_peered":"2024-02-05T06:51:27.767846+0000","last_clean":"2024-02-05T06:51:27.767846+0000","last_became_active":"2024-02-05T06:51:13.274381+0000","last_became_peered":"2024-02-05T06:51:13.274381+0000","last_unstale":"2024-02-05T06:51:27.767846+0000","last_undegraded":"2024-02-05T06:51:27.767846+0000","last_fullsized":"2024-02-05T06:51:27.767846+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:51:12.249449+0000","objects_scrubbed":0,"log_size":1049,"log_dups_size":0,"ondisk_log_size":1049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:20.501958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68747264,"num_objects":1049,"num_object_clones":0,"num_object_copies":3147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1049,"num_write_kb":67136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":495,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.622318+0000","last_change":"2024-02-05T06:50:47.835337+0000","last_active":"2024-02-05T06:51:23.622318+0000","last_peered":"2024-02-05T06:51:23.622318+0000","last_clean":"2024-02-05T06:51:23.622318+0000","last_became_active":"2024-02-05T06:41:43.418198+0000","last_became_peered":"2024-02-05T06:41:43.418198+0000","last_unstale":"2024-02-05T06:51:23.622318+0000","last_undegraded":"2024-02-05T06:51:23.622318+0000","last_fullsized":"2024-02-05T06:51:23.622318+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:47.835116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:50:47.835116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:48.791212+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"413'40","reported_seq":1153,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.615031+0000","last_change":"2024-02-05T06:50:56.447138+0000","last_active":"2024-02-05T06:51:23.615031+0000","last_peered":"2024-02-05T06:51:23.615031+0000","last_clean":"2024-02-05T06:51:23.615031+0000","last_became_active":"2024-02-05T06:40:27.061111+0000","last_became_peered":"2024-02-05T06:40:27.061111+0000","last_unstale":"2024-02-05T06:51:23.615031+0000","last_undegraded":"2024-02-05T06:51:23.615031+0000","last_fullsized":"2024-02-05T06:51:23.615031+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'40","last_scrub_stamp":"2024-02-05T06:50:56.446985+0000","last_deep_scrub":"413'40","last_deep_scrub_stamp":"2024-02-05T06:50:56.446985+0000","last_clean_scrub_stamp":"2024-02-05T06:50:56.446985+0000","objects_scrubbed":8,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:18.155849+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":786,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.661164+0000","last_change":"2024-02-05T06:50:32.218378+0000","last_active":"2024-02-05T06:51:27.661164+0000","last_peered":"2024-02-05T06:51:27.661164+0000","last_clean":"2024-02-05T06:51:27.661164+0000","last_became_active":"2024-02-05T06:41:43.411335+0000","last_became_peered":"2024-02-05T06:41:43.411335+0000","last_unstale":"2024-02-05T06:51:27.661164+0000","last_undegraded":"2024-02-05T06:51:27.661164+0000","last_fullsized":"2024-02-05T06:51:27.661164+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:32.218254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:50:32.218254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:56.216113+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":902,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.619662+0000","last_change":"2024-02-05T06:50:22.799059+0000","last_active":"2024-02-05T06:51:23.619662+0000","last_peered":"2024-02-05T06:51:23.619662+0000","last_clean":"2024-02-05T06:51:23.619662+0000","last_became_active":"2024-02-05T06:40:30.081178+0000","last_became_peered":"2024-02-05T06:40:30.081178+0000","last_unstale":"2024-02-05T06:51:23.619662+0000","last_undegraded":"2024-02-05T06:51:23.619662+0000","last_fullsized":"2024-02-05T06:51:23.619662+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:50:22.798931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:50:22.798931+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:30.937717+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":397,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.621367+0000","last_change":"2024-02-05T06:48:36.692980+0000","last_active":"2024-02-05T06:51:23.621367+0000","last_peered":"2024-02-05T06:51:23.621367+0000","last_clean":"2024-02-05T06:51:23.621367+0000","last_became_active":"2024-02-05T06:41:43.408773+0000","last_became_peered":"2024-02-05T06:41:43.408773+0000","last_unstale":"2024-02-05T06:51:23.621367+0000","last_undegraded":"2024-02-05T06:51:23.621367+0000","last_fullsized":"2024-02-05T06:51:23.621367+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:36.692858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:48:36.692858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":146,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"424'1044","reported_seq":1066,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.767563+0000","last_change":"2024-02-05T06:51:13.274836+0000","last_active":"2024-02-05T06:51:27.767563+0000","last_peered":"2024-02-05T06:51:27.767563+0000","last_clean":"2024-02-05T06:51:27.767563+0000","last_became_active":"2024-02-05T06:51:13.272948+0000","last_became_peered":"2024-02-05T06:51:13.272948+0000","last_unstale":"2024-02-05T06:51:27.767563+0000","last_undegraded":"2024-02-05T06:51:27.767563+0000","last_fullsized":"2024-02-05T06:51:27.767563+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:51:12.249449+0000","objects_scrubbed":0,"log_size":1044,"log_dups_size":0,"ondisk_log_size":1044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:31.809387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68419584,"num_objects":1044,"num_object_clones":0,"num_object_copies":3132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1044,"num_write_kb":66816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":826,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.619634+0000","last_change":"2024-02-05T06:50:26.849680+0000","last_active":"2024-02-05T06:51:23.619634+0000","last_peered":"2024-02-05T06:51:23.619634+0000","last_clean":"2024-02-05T06:51:23.619634+0000","last_became_active":"2024-02-05T06:41:43.412387+0000","last_became_peered":"2024-02-05T06:41:43.412387+0000","last_unstale":"2024-02-05T06:51:23.619634+0000","last_undegraded":"2024-02-05T06:51:23.619634+0000","last_fullsized":"2024-02-05T06:51:23.619634+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:26.849564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:33.914422+0000","last_clean_scrub_stamp":"2024-02-05T06:50:26.849564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:50.275575+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"295'17","reported_seq":879,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.629354+0000","last_change":"2024-02-05T06:50:39.155165+0000","last_active":"2024-02-05T06:51:23.629354+0000","last_peered":"2024-02-05T06:51:23.629354+0000","last_clean":"2024-02-05T06:51:23.629354+0000","last_became_active":"2024-02-05T06:41:43.420458+0000","last_became_peered":"2024-02-05T06:41:43.420458+0000","last_unstale":"2024-02-05T06:51:23.629354+0000","last_undegraded":"2024-02-05T06:51:23.629354+0000","last_fullsized":"2024-02-05T06:51:23.629354+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'17","last_scrub_stamp":"2024-02-05T06:50:39.155040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:50:39.155040+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:03.210509+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":811,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.618448+0000","last_change":"2024-02-05T06:50:30.893584+0000","last_active":"2024-02-05T06:51:23.618448+0000","last_peered":"2024-02-05T06:51:23.618448+0000","last_clean":"2024-02-05T06:51:23.618448+0000","last_became_active":"2024-02-05T06:33:28.542977+0000","last_became_peered":"2024-02-05T06:33:28.542977+0000","last_unstale":"2024-02-05T06:51:23.618448+0000","last_undegraded":"2024-02-05T06:51:23.618448+0000","last_fullsized":"2024-02-05T06:51:23.618448+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:30.893472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:50:30.893472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:48.685353+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00091640799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":780,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.626873+0000","last_change":"2024-02-05T06:48:39.024116+0000","last_active":"2024-02-05T06:51:23.626873+0000","last_peered":"2024-02-05T06:51:23.626873+0000","last_clean":"2024-02-05T06:51:23.626873+0000","last_became_active":"2024-02-05T06:33:29.551549+0000","last_became_peered":"2024-02-05T06:33:29.551549+0000","last_unstale":"2024-02-05T06:51:23.626873+0000","last_undegraded":"2024-02-05T06:51:23.626873+0000","last_fullsized":"2024-02-05T06:51:23.626873+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:39.023856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:48:39.023856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":481,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.621386+0000","last_change":"2024-02-05T06:48:29.707093+0000","last_active":"2024-02-05T06:51:23.621386+0000","last_peered":"2024-02-05T06:51:23.621386+0000","last_clean":"2024-02-05T06:51:23.621386+0000","last_became_active":"2024-02-05T06:40:27.077966+0000","last_became_peered":"2024-02-05T06:40:27.077966+0000","last_unstale":"2024-02-05T06:51:23.621386+0000","last_undegraded":"2024-02-05T06:51:23.621386+0000","last_fullsized":"2024-02-05T06:51:23.621386+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:29.706963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:45.774441+0000","last_clean_scrub_stamp":"2024-02-05T06:48:29.706963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":910,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.620918+0000","last_change":"2024-02-05T06:50:49.806829+0000","last_active":"2024-02-05T06:51:23.620918+0000","last_peered":"2024-02-05T06:51:23.620918+0000","last_clean":"2024-02-05T06:51:23.620918+0000","last_became_active":"2024-02-05T06:40:30.082467+0000","last_became_peered":"2024-02-05T06:40:30.082467+0000","last_unstale":"2024-02-05T06:51:23.620918+0000","last_undegraded":"2024-02-05T06:51:23.620918+0000","last_fullsized":"2024-02-05T06:51:23.620918+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:50:49.806754+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:48:28.657679+0000","last_clean_scrub_stamp":"2024-02-05T06:50:49.806754+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:07.278615+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"424'1097","reported_seq":1119,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.732963+0000","last_change":"2024-02-05T06:51:13.274482+0000","last_active":"2024-02-05T06:51:27.732963+0000","last_peered":"2024-02-05T06:51:27.732963+0000","last_clean":"2024-02-05T06:51:27.732963+0000","last_became_active":"2024-02-05T06:51:13.274182+0000","last_became_peered":"2024-02-05T06:51:13.274182+0000","last_unstale":"2024-02-05T06:51:27.732963+0000","last_undegraded":"2024-02-05T06:51:27.732963+0000","last_fullsized":"2024-02-05T06:51:27.732963+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:51:12.249449+0000","objects_scrubbed":0,"log_size":1097,"log_dups_size":0,"ondisk_log_size":1097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:17.942693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71892992,"num_objects":1097,"num_object_clones":0,"num_object_copies":3291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1097,"num_write_kb":70208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":785,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.612616+0000","last_change":"2024-02-05T06:50:41.994796+0000","last_active":"2024-02-05T06:51:23.612616+0000","last_peered":"2024-02-05T06:51:23.612616+0000","last_clean":"2024-02-05T06:51:23.612616+0000","last_became_active":"2024-02-05T06:36:54.973387+0000","last_became_peered":"2024-02-05T06:36:54.973387+0000","last_unstale":"2024-02-05T06:51:23.612616+0000","last_undegraded":"2024-02-05T06:51:23.612616+0000","last_fullsized":"2024-02-05T06:51:23.612616+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:41.994700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:50:41.994700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:08.306710+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"326'52","reported_seq":688,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:25.658194+0000","last_change":"2024-02-05T06:51:25.658272+0000","last_active":"2024-02-05T06:51:25.658194+0000","last_peered":"2024-02-05T06:51:25.658194+0000","last_clean":"2024-02-05T06:51:25.658194+0000","last_became_active":"2024-02-05T06:40:08.604483+0000","last_became_peered":"2024-02-05T06:40:08.604483+0000","last_unstale":"2024-02-05T06:51:25.658194+0000","last_undegraded":"2024-02-05T06:51:25.658194+0000","last_fullsized":"2024-02-05T06:51:25.658194+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'52","last_scrub_stamp":"2024-02-05T06:51:25.658118+0000","last_deep_scrub":"326'52","last_deep_scrub_stamp":"2024-02-05T06:45:53.715482+0000","last_clean_scrub_stamp":"2024-02-05T06:51:25.658118+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:40.188141+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":310,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.613986+0000","last_change":"2024-02-05T06:50:44.957516+0000","last_active":"2024-02-05T06:51:23.613986+0000","last_peered":"2024-02-05T06:51:23.613986+0000","last_clean":"2024-02-05T06:51:23.613986+0000","last_became_active":"2024-02-05T06:45:09.776169+0000","last_became_peered":"2024-02-05T06:45:09.776169+0000","last_unstale":"2024-02-05T06:51:23.613986+0000","last_undegraded":"2024-02-05T06:51:23.613986+0000","last_fullsized":"2024-02-05T06:51:23.613986+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:44.956750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:50:44.956750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:50.265748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"182'3","reported_seq":400,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.620110+0000","last_change":"2024-02-05T06:50:43.941448+0000","last_active":"2024-02-05T06:51:23.620110+0000","last_peered":"2024-02-05T06:51:23.620110+0000","last_clean":"2024-02-05T06:51:23.620110+0000","last_became_active":"2024-02-05T06:41:43.415806+0000","last_became_peered":"2024-02-05T06:41:43.415806+0000","last_unstale":"2024-02-05T06:51:23.620110+0000","last_undegraded":"2024-02-05T06:51:23.620110+0000","last_fullsized":"2024-02-05T06:51:23.620110+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'3","last_scrub_stamp":"2024-02-05T06:50:43.941322+0000","last_deep_scrub":"182'3","last_deep_scrub_stamp":"2024-02-05T06:39:33.373415+0000","last_clean_scrub_stamp":"2024-02-05T06:50:43.941322+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:49.767545+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0.001220348,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":786,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.661908+0000","last_change":"2024-02-05T06:50:52.403321+0000","last_active":"2024-02-05T06:51:27.661908+0000","last_peered":"2024-02-05T06:51:27.661908+0000","last_clean":"2024-02-05T06:51:27.661908+0000","last_became_active":"2024-02-05T06:40:27.082440+0000","last_became_peered":"2024-02-05T06:40:27.082440+0000","last_unstale":"2024-02-05T06:51:27.661908+0000","last_undegraded":"2024-02-05T06:51:27.661908+0000","last_fullsized":"2024-02-05T06:51:27.661908+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:52.403179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:50:52.403179+0000","last_clean_scrub_stamp":"2024-02-05T06:50:52.403179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:22.075122+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":813,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.619525+0000","last_change":"2024-02-05T06:50:23.712414+0000","last_active":"2024-02-05T06:51:23.619525+0000","last_peered":"2024-02-05T06:51:23.619525+0000","last_clean":"2024-02-05T06:51:23.619525+0000","last_became_active":"2024-02-05T06:41:46.054968+0000","last_became_peered":"2024-02-05T06:41:46.054968+0000","last_unstale":"2024-02-05T06:51:23.619525+0000","last_undegraded":"2024-02-05T06:51:23.619525+0000","last_fullsized":"2024-02-05T06:51:23.619525+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:50:23.712320+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:37.282442+0000","last_clean_scrub_stamp":"2024-02-05T06:50:23.712320+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:49.166612+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"423'889","reported_seq":909,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:26.736990+0000","last_change":"2024-02-05T06:51:13.273074+0000","last_active":"2024-02-05T06:51:26.736990+0000","last_peered":"2024-02-05T06:51:26.736990+0000","last_clean":"2024-02-05T06:51:26.736990+0000","last_became_active":"2024-02-05T06:51:13.272748+0000","last_became_peered":"2024-02-05T06:51:13.272748+0000","last_unstale":"2024-02-05T06:51:26.736990+0000","last_undegraded":"2024-02-05T06:51:26.736990+0000","last_fullsized":"2024-02-05T06:51:26.736990+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:51:12.249449+0000","objects_scrubbed":0,"log_size":889,"log_dups_size":0,"ondisk_log_size":889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:20.985370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58261504,"num_objects":889,"num_object_clones":0,"num_object_copies":2667,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":889,"num_write_kb":56896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"295'15","reported_seq":881,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.613826+0000","last_change":"2024-02-05T06:48:23.607645+0000","last_active":"2024-02-05T06:51:23.613826+0000","last_peered":"2024-02-05T06:51:23.613826+0000","last_clean":"2024-02-05T06:51:23.613826+0000","last_became_active":"2024-02-05T06:40:08.584204+0000","last_became_peered":"2024-02-05T06:40:08.584204+0000","last_unstale":"2024-02-05T06:51:23.613826+0000","last_undegraded":"2024-02-05T06:51:23.613826+0000","last_fullsized":"2024-02-05T06:51:23.613826+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:48:23.607540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:48:23.607540+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":396,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.620126+0000","last_change":"2024-02-05T06:50:32.708358+0000","last_active":"2024-02-05T06:51:23.620126+0000","last_peered":"2024-02-05T06:51:23.620126+0000","last_clean":"2024-02-05T06:51:23.620126+0000","last_became_active":"2024-02-05T06:41:43.412611+0000","last_became_peered":"2024-02-05T06:41:43.412611+0000","last_unstale":"2024-02-05T06:51:23.620126+0000","last_undegraded":"2024-02-05T06:51:23.620126+0000","last_fullsized":"2024-02-05T06:51:23.620126+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:32.708281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:41.617526+0000","last_clean_scrub_stamp":"2024-02-05T06:50:32.708281+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:49.222309+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":814,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.618702+0000","last_change":"2024-02-05T06:51:22.540723+0000","last_active":"2024-02-05T06:51:23.618702+0000","last_peered":"2024-02-05T06:51:23.618702+0000","last_clean":"2024-02-05T06:51:23.618702+0000","last_became_active":"2024-02-05T06:36:54.966016+0000","last_became_peered":"2024-02-05T06:36:54.966016+0000","last_unstale":"2024-02-05T06:51:23.618702+0000","last_undegraded":"2024-02-05T06:51:23.618702+0000","last_fullsized":"2024-02-05T06:51:23.618702+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:22.540545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:51:22.540545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:28.657036+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":773,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.612940+0000","last_change":"2024-02-05T06:48:38.548888+0000","last_active":"2024-02-05T06:51:23.612940+0000","last_peered":"2024-02-05T06:51:23.612940+0000","last_clean":"2024-02-05T06:51:23.612940+0000","last_became_active":"2024-02-05T06:40:27.129625+0000","last_became_peered":"2024-02-05T06:40:27.129625+0000","last_unstale":"2024-02-05T06:51:23.612940+0000","last_undegraded":"2024-02-05T06:51:23.612940+0000","last_fullsized":"2024-02-05T06:51:23.612940+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:38.548777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:48:38.548777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"423'879","reported_seq":899,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:26.742538+0000","last_change":"2024-02-05T06:51:13.271334+0000","last_active":"2024-02-05T06:51:26.742538+0000","last_peered":"2024-02-05T06:51:26.742538+0000","last_clean":"2024-02-05T06:51:26.742538+0000","last_became_active":"2024-02-05T06:51:13.270923+0000","last_became_peered":"2024-02-05T06:51:13.270923+0000","last_unstale":"2024-02-05T06:51:26.742538+0000","last_undegraded":"2024-02-05T06:51:26.742538+0000","last_fullsized":"2024-02-05T06:51:26.742538+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:51:12.249449+0000","objects_scrubbed":0,"log_size":879,"log_dups_size":0,"ondisk_log_size":879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:16.435504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57606144,"num_objects":879,"num_object_clones":0,"num_object_copies":2637,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":879,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":879,"num_write_kb":56256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"295'44","reported_seq":941,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.616217+0000","last_change":"2024-02-05T06:51:22.499732+0000","last_active":"2024-02-05T06:51:23.616217+0000","last_peered":"2024-02-05T06:51:23.616217+0000","last_clean":"2024-02-05T06:51:23.616217+0000","last_became_active":"2024-02-05T06:40:27.068417+0000","last_became_peered":"2024-02-05T06:40:27.068417+0000","last_unstale":"2024-02-05T06:51:23.616217+0000","last_undegraded":"2024-02-05T06:51:23.616217+0000","last_fullsized":"2024-02-05T06:51:23.616217+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'44","last_scrub_stamp":"2024-02-05T06:51:22.499591+0000","last_deep_scrub":"279'28","last_deep_scrub_stamp":"2024-02-05T06:42:50.148881+0000","last_clean_scrub_stamp":"2024-02-05T06:51:22.499591+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:27.347842+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":486,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.621634+0000","last_change":"2024-02-05T06:50:31.765109+0000","last_active":"2024-02-05T06:51:23.621634+0000","last_peered":"2024-02-05T06:51:23.621634+0000","last_clean":"2024-02-05T06:51:23.621634+0000","last_became_active":"2024-02-05T06:41:43.418202+0000","last_became_peered":"2024-02-05T06:41:43.418202+0000","last_unstale":"2024-02-05T06:51:23.621634+0000","last_undegraded":"2024-02-05T06:51:23.621634+0000","last_fullsized":"2024-02-05T06:51:23.621634+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:31.764984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:50:31.764984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:01.654479+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":811,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.618837+0000","last_change":"2024-02-05T06:50:29.885236+0000","last_active":"2024-02-05T06:51:23.618837+0000","last_peered":"2024-02-05T06:51:23.618837+0000","last_clean":"2024-02-05T06:51:23.618837+0000","last_became_active":"2024-02-05T06:41:43.413095+0000","last_became_peered":"2024-02-05T06:41:43.413095+0000","last_unstale":"2024-02-05T06:51:23.618837+0000","last_undegraded":"2024-02-05T06:51:23.618837+0000","last_fullsized":"2024-02-05T06:51:23.618837+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:29.885126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:45.073681+0000","last_clean_scrub_stamp":"2024-02-05T06:50:29.885126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:55.049273+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":489,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.619474+0000","last_change":"2024-02-05T06:50:50.786030+0000","last_active":"2024-02-05T06:51:23.619474+0000","last_peered":"2024-02-05T06:51:23.619474+0000","last_clean":"2024-02-05T06:51:23.619474+0000","last_became_active":"2024-02-05T06:40:27.218081+0000","last_became_peered":"2024-02-05T06:40:27.218081+0000","last_unstale":"2024-02-05T06:51:23.619474+0000","last_undegraded":"2024-02-05T06:51:23.619474+0000","last_fullsized":"2024-02-05T06:51:23.619474+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:50.785918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:32.925000+0000","last_clean_scrub_stamp":"2024-02-05T06:50:50.785918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:13.062057+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"423'889","reported_seq":909,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:26.364105+0000","last_change":"2024-02-05T06:51:13.267811+0000","last_active":"2024-02-05T06:51:26.364105+0000","last_peered":"2024-02-05T06:51:26.364105+0000","last_clean":"2024-02-05T06:51:26.364105+0000","last_became_active":"2024-02-05T06:51:13.267525+0000","last_became_peered":"2024-02-05T06:51:13.267525+0000","last_unstale":"2024-02-05T06:51:26.364105+0000","last_undegraded":"2024-02-05T06:51:26.364105+0000","last_fullsized":"2024-02-05T06:51:26.364105+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:51:12.249449+0000","objects_scrubbed":0,"log_size":889,"log_dups_size":0,"ondisk_log_size":889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:14.651133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58261504,"num_objects":889,"num_object_clones":0,"num_object_copies":2667,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":889,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":889,"num_write_kb":56896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":794,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.618058+0000","last_change":"2024-02-05T06:50:49.698381+0000","last_active":"2024-02-05T06:51:23.618058+0000","last_peered":"2024-02-05T06:51:23.618058+0000","last_clean":"2024-02-05T06:51:23.618058+0000","last_became_active":"2024-02-05T06:33:33.592546+0000","last_became_peered":"2024-02-05T06:33:33.592546+0000","last_unstale":"2024-02-05T06:51:23.618058+0000","last_undegraded":"2024-02-05T06:51:23.618058+0000","last_fullsized":"2024-02-05T06:51:23.618058+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:49.698240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:42.685240+0000","last_clean_scrub_stamp":"2024-02-05T06:50:49.698240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:15.371145+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"295'18","reported_seq":639,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.626776+0000","last_change":"2024-02-05T06:50:44.126051+0000","last_active":"2024-02-05T06:51:23.626776+0000","last_peered":"2024-02-05T06:51:23.626776+0000","last_clean":"2024-02-05T06:51:23.626776+0000","last_became_active":"2024-02-05T06:45:09.783036+0000","last_became_peered":"2024-02-05T06:45:09.783036+0000","last_unstale":"2024-02-05T06:51:23.626776+0000","last_undegraded":"2024-02-05T06:51:23.626776+0000","last_fullsized":"2024-02-05T06:51:23.626776+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'18","last_scrub_stamp":"2024-02-05T06:50:44.125828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:50:44.125828+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:45.063073+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":20,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":480,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.622019+0000","last_change":"2024-02-05T06:49:33.623425+0000","last_active":"2024-02-05T06:51:23.622019+0000","last_peered":"2024-02-05T06:51:23.622019+0000","last_clean":"2024-02-05T06:51:23.622019+0000","last_became_active":"2024-02-05T06:40:27.063300+0000","last_became_peered":"2024-02-05T06:40:27.063300+0000","last_unstale":"2024-02-05T06:51:23.622019+0000","last_undegraded":"2024-02-05T06:51:23.622019+0000","last_fullsized":"2024-02-05T06:51:23.622019+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:49:33.623344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:49:33.623344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":773,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.663429+0000","last_change":"2024-02-05T06:51:22.703991+0000","last_active":"2024-02-05T06:51:27.663429+0000","last_peered":"2024-02-05T06:51:27.663429+0000","last_clean":"2024-02-05T06:51:27.663429+0000","last_became_active":"2024-02-05T06:40:27.092613+0000","last_became_peered":"2024-02-05T06:40:27.092613+0000","last_unstale":"2024-02-05T06:51:27.663429+0000","last_undegraded":"2024-02-05T06:51:27.663429+0000","last_fullsized":"2024-02-05T06:51:27.663429+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:22.703865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:51:22.703865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:38.023268+0000","scrub_duration":190,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"423'920","reported_seq":940,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:26.787109+0000","last_change":"2024-02-05T06:51:13.272382+0000","last_active":"2024-02-05T06:51:26.787109+0000","last_peered":"2024-02-05T06:51:26.787109+0000","last_clean":"2024-02-05T06:51:26.787109+0000","last_became_active":"2024-02-05T06:51:13.272066+0000","last_became_peered":"2024-02-05T06:51:13.272066+0000","last_unstale":"2024-02-05T06:51:26.787109+0000","last_undegraded":"2024-02-05T06:51:26.787109+0000","last_fullsized":"2024-02-05T06:51:26.787109+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:51:12.249449+0000","objects_scrubbed":0,"log_size":920,"log_dups_size":0,"ondisk_log_size":920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:35.875814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":60293120,"num_objects":920,"num_object_clones":0,"num_object_copies":2760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":920,"num_write_kb":58880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"208'12","reported_seq":625,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.620717+0000","last_change":"2024-02-05T06:50:45.860042+0000","last_active":"2024-02-05T06:51:23.620717+0000","last_peered":"2024-02-05T06:51:23.620717+0000","last_clean":"2024-02-05T06:51:23.620717+0000","last_became_active":"2024-02-05T06:40:27.064195+0000","last_became_peered":"2024-02-05T06:40:27.064195+0000","last_unstale":"2024-02-05T06:51:23.620717+0000","last_undegraded":"2024-02-05T06:51:23.620717+0000","last_fullsized":"2024-02-05T06:51:23.620717+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'12","last_scrub_stamp":"2024-02-05T06:50:45.859928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:50:45.859928+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:50.250914+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"295'27","reported_seq":430,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.619779+0000","last_change":"2024-02-05T06:50:44.924804+0000","last_active":"2024-02-05T06:51:23.619779+0000","last_peered":"2024-02-05T06:51:23.619779+0000","last_clean":"2024-02-05T06:51:23.619779+0000","last_became_active":"2024-02-05T06:41:43.412579+0000","last_became_peered":"2024-02-05T06:41:43.412579+0000","last_unstale":"2024-02-05T06:51:23.619779+0000","last_undegraded":"2024-02-05T06:51:23.619779+0000","last_fullsized":"2024-02-05T06:51:23.619779+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'27","last_scrub_stamp":"2024-02-05T06:50:44.924678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:50:44.924678+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:05.213176+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"412'4","reported_seq":496,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.622876+0000","last_change":"2024-02-05T06:50:51.842000+0000","last_active":"2024-02-05T06:51:23.622876+0000","last_peered":"2024-02-05T06:51:23.622876+0000","last_clean":"2024-02-05T06:51:23.622876+0000","last_became_active":"2024-02-05T06:40:27.068279+0000","last_became_peered":"2024-02-05T06:40:27.068279+0000","last_unstale":"2024-02-05T06:51:23.622876+0000","last_undegraded":"2024-02-05T06:51:23.622876+0000","last_fullsized":"2024-02-05T06:51:23.622876+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'4","last_scrub_stamp":"2024-02-05T06:50:51.841825+0000","last_deep_scrub":"412'4","last_deep_scrub_stamp":"2024-02-05T06:50:51.841825+0000","last_clean_scrub_stamp":"2024-02-05T06:50:51.841825+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:58.170151+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":396,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.620017+0000","last_change":"2024-02-05T06:50:33.743900+0000","last_active":"2024-02-05T06:51:23.620017+0000","last_peered":"2024-02-05T06:51:23.620017+0000","last_clean":"2024-02-05T06:51:23.620017+0000","last_became_active":"2024-02-05T06:41:43.421231+0000","last_became_peered":"2024-02-05T06:41:43.421231+0000","last_unstale":"2024-02-05T06:51:23.620017+0000","last_undegraded":"2024-02-05T06:51:23.620017+0000","last_fullsized":"2024-02-05T06:51:23.620017+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:33.743650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:50:33.743650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:37.777397+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"412'15","reported_seq":818,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.662800+0000","last_change":"2024-02-05T06:50:55.397833+0000","last_active":"2024-02-05T06:51:27.662800+0000","last_peered":"2024-02-05T06:51:27.662800+0000","last_clean":"2024-02-05T06:51:27.662800+0000","last_became_active":"2024-02-05T06:40:27.091689+0000","last_became_peered":"2024-02-05T06:40:27.091689+0000","last_unstale":"2024-02-05T06:51:27.662800+0000","last_undegraded":"2024-02-05T06:51:27.662800+0000","last_fullsized":"2024-02-05T06:51:27.662800+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'15","last_scrub_stamp":"2024-02-05T06:50:55.397689+0000","last_deep_scrub":"412'15","last_deep_scrub_stamp":"2024-02-05T06:50:55.397689+0000","last_clean_scrub_stamp":"2024-02-05T06:50:55.397689+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:02.398185+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":806,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.615831+0000","last_change":"2024-02-05T06:48:28.815276+0000","last_active":"2024-02-05T06:51:23.615831+0000","last_peered":"2024-02-05T06:51:23.615831+0000","last_clean":"2024-02-05T06:51:23.615831+0000","last_became_active":"2024-02-05T06:41:43.412130+0000","last_became_peered":"2024-02-05T06:41:43.412130+0000","last_unstale":"2024-02-05T06:51:23.615831+0000","last_undegraded":"2024-02-05T06:51:23.615831+0000","last_fullsized":"2024-02-05T06:51:23.615831+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:28.815050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:48:28.815050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":396,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.620893+0000","last_change":"2024-02-05T06:50:29.738203+0000","last_active":"2024-02-05T06:51:23.620893+0000","last_peered":"2024-02-05T06:51:23.620893+0000","last_clean":"2024-02-05T06:51:23.620893+0000","last_became_active":"2024-02-05T06:41:43.406094+0000","last_became_peered":"2024-02-05T06:41:43.406094+0000","last_unstale":"2024-02-05T06:51:23.620893+0000","last_undegraded":"2024-02-05T06:51:23.620893+0000","last_fullsized":"2024-02-05T06:51:23.620893+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:29.737989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:44:57.506909+0000","last_clean_scrub_stamp":"2024-02-05T06:50:29.737989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:34.296425+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":480,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.621775+0000","last_change":"2024-02-05T06:50:33.846940+0000","last_active":"2024-02-05T06:51:23.621775+0000","last_peered":"2024-02-05T06:51:23.621775+0000","last_clean":"2024-02-05T06:51:23.621775+0000","last_became_active":"2024-02-05T06:40:27.063350+0000","last_became_peered":"2024-02-05T06:40:27.063350+0000","last_unstale":"2024-02-05T06:51:23.621775+0000","last_undegraded":"2024-02-05T06:51:23.621775+0000","last_fullsized":"2024-02-05T06:51:23.621775+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:33.846812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:50:33.846812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:44.822563+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"295'15","reported_seq":841,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.613736+0000","last_change":"2024-02-05T06:50:51.071678+0000","last_active":"2024-02-05T06:51:23.613736+0000","last_peered":"2024-02-05T06:51:23.613736+0000","last_clean":"2024-02-05T06:51:23.613736+0000","last_became_active":"2024-02-05T06:40:27.101116+0000","last_became_peered":"2024-02-05T06:40:27.101116+0000","last_unstale":"2024-02-05T06:51:23.613736+0000","last_undegraded":"2024-02-05T06:51:23.613736+0000","last_fullsized":"2024-02-05T06:51:23.613736+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:50:51.071549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:50:51.071549+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:09.003471+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":480,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.620784+0000","last_change":"2024-02-05T06:50:24.699857+0000","last_active":"2024-02-05T06:51:23.620784+0000","last_peered":"2024-02-05T06:51:23.620784+0000","last_clean":"2024-02-05T06:51:23.620784+0000","last_became_active":"2024-02-05T06:40:27.064388+0000","last_became_peered":"2024-02-05T06:40:27.064388+0000","last_unstale":"2024-02-05T06:51:23.620784+0000","last_undegraded":"2024-02-05T06:51:23.620784+0000","last_fullsized":"2024-02-05T06:51:23.620784+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:24.699732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:43.606488+0000","last_clean_scrub_stamp":"2024-02-05T06:50:24.699732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:28.533627+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":783,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.662629+0000","last_change":"2024-02-05T06:47:43.427431+0000","last_active":"2024-02-05T06:51:27.662629+0000","last_peered":"2024-02-05T06:51:27.662629+0000","last_clean":"2024-02-05T06:51:27.662629+0000","last_became_active":"2024-02-05T06:40:27.091544+0000","last_became_peered":"2024-02-05T06:40:27.091544+0000","last_unstale":"2024-02-05T06:51:27.662629+0000","last_undegraded":"2024-02-05T06:51:27.662629+0000","last_fullsized":"2024-02-05T06:51:27.662629+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:43.427301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:47:43.427301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":821,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.619482+0000","last_change":"2024-02-05T06:50:45.676425+0000","last_active":"2024-02-05T06:51:23.619482+0000","last_peered":"2024-02-05T06:51:23.619482+0000","last_clean":"2024-02-05T06:51:23.619482+0000","last_became_active":"2024-02-05T06:40:27.090632+0000","last_became_peered":"2024-02-05T06:40:27.090632+0000","last_unstale":"2024-02-05T06:51:23.619482+0000","last_undegraded":"2024-02-05T06:51:23.619482+0000","last_fullsized":"2024-02-05T06:51:23.619482+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:45.676320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:50:45.676320+0000","last_clean_scrub_stamp":"2024-02-05T06:50:45.676320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:13.665886+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"295'15","reported_seq":415,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.620632+0000","last_change":"2024-02-05T06:50:27.668626+0000","last_active":"2024-02-05T06:51:23.620632+0000","last_peered":"2024-02-05T06:51:23.620632+0000","last_clean":"2024-02-05T06:51:23.620632+0000","last_became_active":"2024-02-05T06:41:43.413708+0000","last_became_peered":"2024-02-05T06:41:43.413708+0000","last_unstale":"2024-02-05T06:51:23.620632+0000","last_undegraded":"2024-02-05T06:51:23.620632+0000","last_fullsized":"2024-02-05T06:51:23.620632+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:50:27.668502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:50:27.668502+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:42.632566+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":800,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.612497+0000","last_change":"2024-02-05T06:50:47.080623+0000","last_active":"2024-02-05T06:51:23.612497+0000","last_peered":"2024-02-05T06:51:23.612497+0000","last_clean":"2024-02-05T06:51:23.612497+0000","last_became_active":"2024-02-05T06:40:27.088096+0000","last_became_peered":"2024-02-05T06:40:27.088096+0000","last_unstale":"2024-02-05T06:51:23.612497+0000","last_undegraded":"2024-02-05T06:51:23.612497+0000","last_fullsized":"2024-02-05T06:51:23.612497+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:47.080511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:50:47.080511+0000","last_clean_scrub_stamp":"2024-02-05T06:50:47.080511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:13.723612+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"208'10","reported_seq":1009,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.661142+0000","last_change":"2024-02-05T06:48:18.678733+0000","last_active":"2024-02-05T06:51:27.661142+0000","last_peered":"2024-02-05T06:51:27.661142+0000","last_clean":"2024-02-05T06:51:27.661142+0000","last_became_active":"2024-02-05T06:40:27.096584+0000","last_became_peered":"2024-02-05T06:40:27.096584+0000","last_unstale":"2024-02-05T06:51:27.661142+0000","last_undegraded":"2024-02-05T06:51:27.661142+0000","last_fullsized":"2024-02-05T06:51:27.661142+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-02-05T06:48:18.677391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:48:18.677391+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":535,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.619202+0000","last_change":"2024-02-05T06:50:52.908134+0000","last_active":"2024-02-05T06:51:23.619202+0000","last_peered":"2024-02-05T06:51:23.619202+0000","last_clean":"2024-02-05T06:51:23.619202+0000","last_became_active":"2024-02-05T06:45:09.775152+0000","last_became_peered":"2024-02-05T06:45:09.775152+0000","last_unstale":"2024-02-05T06:51:23.619202+0000","last_undegraded":"2024-02-05T06:51:23.619202+0000","last_fullsized":"2024-02-05T06:51:23.619202+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:52.908047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:41:38.559780+0000","last_clean_scrub_stamp":"2024-02-05T06:50:52.908047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:11.423810+0000","scrub_duration":176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"295'20","reported_seq":853,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.660731+0000","last_change":"2024-02-05T06:49:12.884710+0000","last_active":"2024-02-05T06:51:27.660731+0000","last_peered":"2024-02-05T06:51:27.660731+0000","last_clean":"2024-02-05T06:51:27.660731+0000","last_became_active":"2024-02-05T06:40:27.093418+0000","last_became_peered":"2024-02-05T06:40:27.093418+0000","last_unstale":"2024-02-05T06:51:27.660731+0000","last_undegraded":"2024-02-05T06:51:27.660731+0000","last_fullsized":"2024-02-05T06:51:27.660731+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'20","last_scrub_stamp":"2024-02-05T06:49:12.884128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:49:12.884128+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":481,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.621284+0000","last_change":"2024-02-05T06:48:26.670085+0000","last_active":"2024-02-05T06:51:23.621284+0000","last_peered":"2024-02-05T06:51:23.621284+0000","last_clean":"2024-02-05T06:51:23.621284+0000","last_became_active":"2024-02-05T06:40:27.098418+0000","last_became_peered":"2024-02-05T06:40:27.098418+0000","last_unstale":"2024-02-05T06:51:23.621284+0000","last_undegraded":"2024-02-05T06:51:23.621284+0000","last_fullsized":"2024-02-05T06:51:23.621284+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:26.669933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:48:26.669933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"336'11","reported_seq":518,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:24.807779+0000","last_change":"2024-02-05T06:48:35.564079+0000","last_active":"2024-02-05T06:51:24.807779+0000","last_peered":"2024-02-05T06:51:24.807779+0000","last_clean":"2024-02-05T06:51:24.807779+0000","last_became_active":"2024-02-05T06:41:43.418753+0000","last_became_peered":"2024-02-05T06:41:43.418753+0000","last_unstale":"2024-02-05T06:51:24.807779+0000","last_undegraded":"2024-02-05T06:51:24.807779+0000","last_fullsized":"2024-02-05T06:51:24.807779+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'11","last_scrub_stamp":"2024-02-05T06:48:35.563959+0000","last_deep_scrub":"336'11","last_deep_scrub_stamp":"2024-02-05T06:46:55.721412+0000","last_clean_scrub_stamp":"2024-02-05T06:48:35.563959+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":772,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.612700+0000","last_change":"2024-02-05T06:48:24.623914+0000","last_active":"2024-02-05T06:51:23.612700+0000","last_peered":"2024-02-05T06:51:23.612700+0000","last_clean":"2024-02-05T06:51:23.612700+0000","last_became_active":"2024-02-05T06:40:27.096132+0000","last_became_peered":"2024-02-05T06:40:27.096132+0000","last_unstale":"2024-02-05T06:51:23.612700+0000","last_undegraded":"2024-02-05T06:51:23.612700+0000","last_fullsized":"2024-02-05T06:51:23.612700+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:24.623815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:48:24.623815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"295'24","reported_seq":641,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.615365+0000","last_change":"2024-02-05T06:48:30.861303+0000","last_active":"2024-02-05T06:51:23.615365+0000","last_peered":"2024-02-05T06:51:23.615365+0000","last_clean":"2024-02-05T06:51:23.615365+0000","last_became_active":"2024-02-05T06:45:09.784411+0000","last_became_peered":"2024-02-05T06:45:09.784411+0000","last_unstale":"2024-02-05T06:51:23.615365+0000","last_undegraded":"2024-02-05T06:51:23.615365+0000","last_fullsized":"2024-02-05T06:51:23.615365+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:48:30.861217+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:35:02.589229+0000","last_clean_scrub_stamp":"2024-02-05T06:48:30.861217+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":434,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.622119+0000","last_change":"2024-02-05T06:50:37.793063+0000","last_active":"2024-02-05T06:51:23.622119+0000","last_peered":"2024-02-05T06:51:23.622119+0000","last_clean":"2024-02-05T06:51:23.622119+0000","last_became_active":"2024-02-05T06:45:09.773490+0000","last_became_peered":"2024-02-05T06:45:09.773490+0000","last_unstale":"2024-02-05T06:51:23.622119+0000","last_undegraded":"2024-02-05T06:51:23.622119+0000","last_fullsized":"2024-02-05T06:51:23.622119+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:37.792619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:50:37.792619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:06.288935+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":797,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.627912+0000","last_change":"2024-02-05T06:48:33.228199+0000","last_active":"2024-02-05T06:51:23.627912+0000","last_peered":"2024-02-05T06:51:23.627912+0000","last_clean":"2024-02-05T06:51:23.627912+0000","last_became_active":"2024-02-05T06:41:43.421433+0000","last_became_peered":"2024-02-05T06:41:43.421433+0000","last_unstale":"2024-02-05T06:51:23.627912+0000","last_undegraded":"2024-02-05T06:51:23.627912+0000","last_fullsized":"2024-02-05T06:51:23.627912+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:33.228080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:48:33.228080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":773,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.660655+0000","last_change":"2024-02-05T06:50:27.146146+0000","last_active":"2024-02-05T06:51:27.660655+0000","last_peered":"2024-02-05T06:51:27.660655+0000","last_clean":"2024-02-05T06:51:27.660655+0000","last_became_active":"2024-02-05T06:41:43.417155+0000","last_became_peered":"2024-02-05T06:41:43.417155+0000","last_unstale":"2024-02-05T06:51:27.660655+0000","last_undegraded":"2024-02-05T06:51:27.660655+0000","last_fullsized":"2024-02-05T06:51:27.660655+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:27.145766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:50:27.145766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:51.843640+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"295'15","reported_seq":565,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.611748+0000","last_change":"2024-02-05T06:50:32.899366+0000","last_active":"2024-02-05T06:51:23.611748+0000","last_peered":"2024-02-05T06:51:23.611748+0000","last_clean":"2024-02-05T06:51:23.611748+0000","last_became_active":"2024-02-05T06:39:13.244124+0000","last_became_peered":"2024-02-05T06:39:13.244124+0000","last_unstale":"2024-02-05T06:51:23.611748+0000","last_undegraded":"2024-02-05T06:51:23.611748+0000","last_fullsized":"2024-02-05T06:51:23.611748+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:50:32.899213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:50:32.899213+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:39.603583+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":480,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.621462+0000","last_change":"2024-02-05T06:49:32.587538+0000","last_active":"2024-02-05T06:51:23.621462+0000","last_peered":"2024-02-05T06:51:23.621462+0000","last_clean":"2024-02-05T06:51:23.621462+0000","last_became_active":"2024-02-05T06:40:27.077701+0000","last_became_peered":"2024-02-05T06:40:27.077701+0000","last_unstale":"2024-02-05T06:51:23.621462+0000","last_undegraded":"2024-02-05T06:51:23.621462+0000","last_fullsized":"2024-02-05T06:51:23.621462+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:49:32.587414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:49:32.587414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":757,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.663175+0000","last_change":"2024-02-05T06:47:35.411302+0000","last_active":"2024-02-05T06:51:27.663175+0000","last_peered":"2024-02-05T06:51:27.663175+0000","last_clean":"2024-02-05T06:51:27.663175+0000","last_became_active":"2024-02-05T06:33:33.593901+0000","last_became_peered":"2024-02-05T06:33:33.593901+0000","last_unstale":"2024-02-05T06:51:27.663175+0000","last_undegraded":"2024-02-05T06:51:27.663175+0000","last_fullsized":"2024-02-05T06:51:27.663175+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:47:35.411219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:47:35.411219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"336'2","reported_seq":790,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.611673+0000","last_change":"2024-02-05T06:50:21.995924+0000","last_active":"2024-02-05T06:51:23.611673+0000","last_peered":"2024-02-05T06:51:23.611673+0000","last_clean":"2024-02-05T06:51:23.611673+0000","last_became_active":"2024-02-05T06:41:43.409761+0000","last_became_peered":"2024-02-05T06:41:43.409761+0000","last_unstale":"2024-02-05T06:51:23.611673+0000","last_undegraded":"2024-02-05T06:51:23.611673+0000","last_fullsized":"2024-02-05T06:51:23.611673+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'2","last_scrub_stamp":"2024-02-05T06:50:21.995789+0000","last_deep_scrub":"336'2","last_deep_scrub_stamp":"2024-02-05T06:46:53.500155+0000","last_clean_scrub_stamp":"2024-02-05T06:50:21.995789+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:43.284610+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"295'24","reported_seq":851,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.661564+0000","last_change":"2024-02-05T06:47:30.341586+0000","last_active":"2024-02-05T06:51:27.661564+0000","last_peered":"2024-02-05T06:51:27.661564+0000","last_clean":"2024-02-05T06:51:27.661564+0000","last_became_active":"2024-02-05T06:40:27.080906+0000","last_became_peered":"2024-02-05T06:40:27.080906+0000","last_unstale":"2024-02-05T06:51:27.661564+0000","last_undegraded":"2024-02-05T06:51:27.661564+0000","last_fullsized":"2024-02-05T06:51:27.661564+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:47:30.341433+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-02-05T06:35:22.938675+0000","last_clean_scrub_stamp":"2024-02-05T06:47:30.341433+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":396,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.620920+0000","last_change":"2024-02-05T06:50:24.688710+0000","last_active":"2024-02-05T06:51:23.620920+0000","last_peered":"2024-02-05T06:51:23.620920+0000","last_clean":"2024-02-05T06:51:23.620920+0000","last_became_active":"2024-02-05T06:41:43.412654+0000","last_became_peered":"2024-02-05T06:41:43.412654+0000","last_unstale":"2024-02-05T06:51:23.620920+0000","last_undegraded":"2024-02-05T06:51:23.620920+0000","last_fullsized":"2024-02-05T06:51:23.620920+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:24.688587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:50:24.688587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":488,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.619386+0000","last_change":"2024-02-05T06:51:21.888952+0000","last_active":"2024-02-05T06:51:23.619386+0000","last_peered":"2024-02-05T06:51:23.619386+0000","last_clean":"2024-02-05T06:51:23.619386+0000","last_became_active":"2024-02-05T06:40:27.062975+0000","last_became_peered":"2024-02-05T06:40:27.062975+0000","last_unstale":"2024-02-05T06:51:23.619386+0000","last_undegraded":"2024-02-05T06:51:23.619386+0000","last_fullsized":"2024-02-05T06:51:23.619386+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:21.888827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:48:34.628182+0000","last_clean_scrub_stamp":"2024-02-05T06:51:21.888827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:38.376074+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":546,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.628216+0000","last_change":"2024-02-05T06:50:32.488291+0000","last_active":"2024-02-05T06:51:23.628216+0000","last_peered":"2024-02-05T06:51:23.628216+0000","last_clean":"2024-02-05T06:51:23.628216+0000","last_became_active":"2024-02-05T06:39:13.245684+0000","last_became_peered":"2024-02-05T06:39:13.245684+0000","last_unstale":"2024-02-05T06:51:23.628216+0000","last_undegraded":"2024-02-05T06:51:23.628216+0000","last_fullsized":"2024-02-05T06:51:23.628216+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:32.488165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:50:32.488165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:01.682087+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"295'8","reported_seq":773,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.660890+0000","last_change":"2024-02-05T06:47:41.444191+0000","last_active":"2024-02-05T06:51:27.660890+0000","last_peered":"2024-02-05T06:51:27.660890+0000","last_clean":"2024-02-05T06:51:27.660890+0000","last_became_active":"2024-02-05T06:33:31.573410+0000","last_became_peered":"2024-02-05T06:33:31.573410+0000","last_unstale":"2024-02-05T06:51:27.660890+0000","last_undegraded":"2024-02-05T06:51:27.660890+0000","last_fullsized":"2024-02-05T06:51:27.660890+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'8","last_scrub_stamp":"2024-02-05T06:47:41.444069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:47:41.444069+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":792,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.613370+0000","last_change":"2024-02-05T06:50:57.085436+0000","last_active":"2024-02-05T06:51:23.613370+0000","last_peered":"2024-02-05T06:51:23.613370+0000","last_clean":"2024-02-05T06:51:23.613370+0000","last_became_active":"2024-02-05T06:41:43.424459+0000","last_became_peered":"2024-02-05T06:41:43.424459+0000","last_unstale":"2024-02-05T06:51:23.613370+0000","last_undegraded":"2024-02-05T06:51:23.613370+0000","last_fullsized":"2024-02-05T06:51:23.613370+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:57.085297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:49.117949+0000","last_clean_scrub_stamp":"2024-02-05T06:50:57.085297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:16.331096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":818,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.645571+0000","last_change":"2024-02-05T06:48:33.848832+0000","last_active":"2024-02-05T06:51:23.645571+0000","last_peered":"2024-02-05T06:51:23.645571+0000","last_clean":"2024-02-05T06:51:23.645571+0000","last_became_active":"2024-02-05T06:40:27.060112+0000","last_became_peered":"2024-02-05T06:40:27.060112+0000","last_unstale":"2024-02-05T06:51:23.645571+0000","last_undegraded":"2024-02-05T06:51:23.645571+0000","last_fullsized":"2024-02-05T06:51:23.645571+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:33.848718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:48:33.848718+0000","last_clean_scrub_stamp":"2024-02-05T06:48:33.848718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":494,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.621018+0000","last_change":"2024-02-05T06:50:57.883175+0000","last_active":"2024-02-05T06:51:23.621018+0000","last_peered":"2024-02-05T06:51:23.621018+0000","last_clean":"2024-02-05T06:51:23.621018+0000","last_became_active":"2024-02-05T06:44:54.625067+0000","last_became_peered":"2024-02-05T06:44:54.625067+0000","last_unstale":"2024-02-05T06:51:23.621018+0000","last_undegraded":"2024-02-05T06:51:23.621018+0000","last_fullsized":"2024-02-05T06:51:23.621018+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:57.883016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:48.512673+0000","last_clean_scrub_stamp":"2024-02-05T06:50:57.883016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:17.065757+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"295'24","reported_seq":874,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.614824+0000","last_change":"2024-02-05T06:50:47.441890+0000","last_active":"2024-02-05T06:51:23.614824+0000","last_peered":"2024-02-05T06:51:23.614824+0000","last_clean":"2024-02-05T06:51:23.614824+0000","last_became_active":"2024-02-05T06:41:43.419769+0000","last_became_peered":"2024-02-05T06:41:43.419769+0000","last_unstale":"2024-02-05T06:51:23.614824+0000","last_undegraded":"2024-02-05T06:51:23.614824+0000","last_fullsized":"2024-02-05T06:51:23.614824+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:50:47.441758+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:43:03.221466+0000","last_clean_scrub_stamp":"2024-02-05T06:50:47.441758+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:16.240607+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":396,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.619620+0000","last_change":"2024-02-05T06:50:57.061723+0000","last_active":"2024-02-05T06:51:23.619620+0000","last_peered":"2024-02-05T06:51:23.619620+0000","last_clean":"2024-02-05T06:51:23.619620+0000","last_became_active":"2024-02-05T06:41:43.411043+0000","last_became_peered":"2024-02-05T06:41:43.411043+0000","last_unstale":"2024-02-05T06:51:23.619620+0000","last_undegraded":"2024-02-05T06:51:23.619620+0000","last_fullsized":"2024-02-05T06:51:23.619620+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:57.061648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:50:57.061648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:07.405148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":689,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.661596+0000","last_change":"2024-02-05T06:50:48.360755+0000","last_active":"2024-02-05T06:51:27.661596+0000","last_peered":"2024-02-05T06:51:27.661596+0000","last_clean":"2024-02-05T06:51:27.661596+0000","last_became_active":"2024-02-05T06:40:27.097169+0000","last_became_peered":"2024-02-05T06:40:27.097169+0000","last_unstale":"2024-02-05T06:51:27.661596+0000","last_undegraded":"2024-02-05T06:51:27.661596+0000","last_fullsized":"2024-02-05T06:51:27.661596+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:48.360650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:08.713894+0000","last_clean_scrub_stamp":"2024-02-05T06:50:48.360650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:56.352693+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":799,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.628667+0000","last_change":"2024-02-05T06:50:42.316849+0000","last_active":"2024-02-05T06:51:23.628667+0000","last_peered":"2024-02-05T06:51:23.628667+0000","last_clean":"2024-02-05T06:51:23.628667+0000","last_became_active":"2024-02-05T06:41:43.414638+0000","last_became_peered":"2024-02-05T06:41:43.414638+0000","last_unstale":"2024-02-05T06:51:23.628667+0000","last_undegraded":"2024-02-05T06:51:23.628667+0000","last_fullsized":"2024-02-05T06:51:23.628667+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:42.316726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:50:42.316726+0000","last_clean_scrub_stamp":"2024-02-05T06:50:42.316726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:47.184638+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"295'25","reported_seq":856,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.613638+0000","last_change":"2024-02-05T06:51:23.298564+0000","last_active":"2024-02-05T06:51:23.613638+0000","last_peered":"2024-02-05T06:51:23.613638+0000","last_clean":"2024-02-05T06:51:23.613638+0000","last_became_active":"2024-02-05T06:40:27.100610+0000","last_became_peered":"2024-02-05T06:40:27.100610+0000","last_unstale":"2024-02-05T06:51:23.613638+0000","last_undegraded":"2024-02-05T06:51:23.613638+0000","last_fullsized":"2024-02-05T06:51:23.613638+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'25","last_scrub_stamp":"2024-02-05T06:51:23.298421+0000","last_deep_scrub":"92'15","last_deep_scrub_stamp":"2024-02-05T06:34:50.718768+0000","last_clean_scrub_stamp":"2024-02-05T06:51:23.298421+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:53.009924+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":816,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:24.654849+0000","last_change":"2024-02-05T06:51:24.654897+0000","last_active":"2024-02-05T06:51:24.654849+0000","last_peered":"2024-02-05T06:51:24.654849+0000","last_clean":"2024-02-05T06:51:24.654849+0000","last_became_active":"2024-02-05T06:36:54.958414+0000","last_became_peered":"2024-02-05T06:36:54.958414+0000","last_unstale":"2024-02-05T06:51:24.654849+0000","last_undegraded":"2024-02-05T06:51:24.654849+0000","last_fullsized":"2024-02-05T06:51:24.654849+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:51:24.654770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:51:24.654770+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:34.152746+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"314'13","reported_seq":1068,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.614679+0000","last_change":"2024-02-05T06:50:51.445774+0000","last_active":"2024-02-05T06:51:23.614679+0000","last_peered":"2024-02-05T06:51:23.614679+0000","last_clean":"2024-02-05T06:51:23.614679+0000","last_became_active":"2024-02-05T06:40:27.063399+0000","last_became_peered":"2024-02-05T06:40:27.063399+0000","last_unstale":"2024-02-05T06:51:23.614679+0000","last_undegraded":"2024-02-05T06:51:23.614679+0000","last_fullsized":"2024-02-05T06:51:23.614679+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"314'13","last_scrub_stamp":"2024-02-05T06:50:51.445651+0000","last_deep_scrub":"314'13","last_deep_scrub_stamp":"2024-02-05T06:44:53.670923+0000","last_clean_scrub_stamp":"2024-02-05T06:50:51.445651+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:01.685095+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"325'2","reported_seq":495,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.622403+0000","last_change":"2024-02-05T06:50:48.850525+0000","last_active":"2024-02-05T06:51:23.622403+0000","last_peered":"2024-02-05T06:51:23.622403+0000","last_clean":"2024-02-05T06:51:23.622403+0000","last_became_active":"2024-02-05T06:41:43.418640+0000","last_became_peered":"2024-02-05T06:41:43.418640+0000","last_unstale":"2024-02-05T06:51:23.622403+0000","last_undegraded":"2024-02-05T06:51:23.622403+0000","last_fullsized":"2024-02-05T06:51:23.622403+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'2","last_scrub_stamp":"2024-02-05T06:50:48.850401+0000","last_deep_scrub":"325'2","last_deep_scrub_stamp":"2024-02-05T06:45:46.242415+0000","last_clean_scrub_stamp":"2024-02-05T06:50:48.850401+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:13.517543+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"295'18","reported_seq":855,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.661365+0000","last_change":"2024-02-05T06:48:24.628198+0000","last_active":"2024-02-05T06:51:27.661365+0000","last_peered":"2024-02-05T06:51:27.661365+0000","last_clean":"2024-02-05T06:51:27.661365+0000","last_became_active":"2024-02-05T06:36:54.954880+0000","last_became_peered":"2024-02-05T06:36:54.954880+0000","last_unstale":"2024-02-05T06:51:27.661365+0000","last_undegraded":"2024-02-05T06:51:27.661365+0000","last_fullsized":"2024-02-05T06:51:27.661365+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'18","last_scrub_stamp":"2024-02-05T06:48:24.627686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:48:24.627686+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":301,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.627527+0000","last_change":"2024-02-05T06:48:15.138990+0000","last_active":"2024-02-05T06:51:23.627527+0000","last_peered":"2024-02-05T06:51:23.627527+0000","last_clean":"2024-02-05T06:51:23.627527+0000","last_became_active":"2024-02-05T06:45:09.786579+0000","last_became_peered":"2024-02-05T06:45:09.786579+0000","last_unstale":"2024-02-05T06:51:23.627527+0000","last_undegraded":"2024-02-05T06:51:23.627527+0000","last_fullsized":"2024-02-05T06:51:23.627527+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:15.138923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:48:15.138923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":560,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.614608+0000","last_change":"2024-02-05T06:48:13.677951+0000","last_active":"2024-02-05T06:51:23.614608+0000","last_peered":"2024-02-05T06:51:23.614608+0000","last_clean":"2024-02-05T06:51:23.614608+0000","last_became_active":"2024-02-05T06:39:13.248618+0000","last_became_peered":"2024-02-05T06:39:13.248618+0000","last_unstale":"2024-02-05T06:51:23.614608+0000","last_undegraded":"2024-02-05T06:51:23.614608+0000","last_fullsized":"2024-02-05T06:51:23.614608+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:13.677795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:42.381636+0000","last_clean_scrub_stamp":"2024-02-05T06:48:13.677795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":774,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.662712+0000","last_change":"2024-02-05T06:51:21.832566+0000","last_active":"2024-02-05T06:51:27.662712+0000","last_peered":"2024-02-05T06:51:27.662712+0000","last_clean":"2024-02-05T06:51:27.662712+0000","last_became_active":"2024-02-05T06:41:43.411442+0000","last_became_peered":"2024-02-05T06:41:43.411442+0000","last_unstale":"2024-02-05T06:51:27.662712+0000","last_undegraded":"2024-02-05T06:51:27.662712+0000","last_fullsized":"2024-02-05T06:51:27.662712+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:21.832441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:51:21.832441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:31.141277+0000","scrub_duration":295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"295'34","reported_seq":855,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.619104+0000","last_change":"2024-02-05T06:50:22.878438+0000","last_active":"2024-02-05T06:51:23.619104+0000","last_peered":"2024-02-05T06:51:23.619104+0000","last_clean":"2024-02-05T06:51:23.619104+0000","last_became_active":"2024-02-05T06:33:31.569763+0000","last_became_peered":"2024-02-05T06:33:31.569763+0000","last_unstale":"2024-02-05T06:51:23.619104+0000","last_undegraded":"2024-02-05T06:51:23.619104+0000","last_fullsized":"2024-02-05T06:51:23.619104+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'34","last_scrub_stamp":"2024-02-05T06:50:22.878292+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:34:35.234636+0000","last_clean_scrub_stamp":"2024-02-05T06:50:22.878292+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:38.858946+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"281'2","reported_seq":827,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.614039+0000","last_change":"2024-02-05T06:50:42.976166+0000","last_active":"2024-02-05T06:51:23.614039+0000","last_peered":"2024-02-05T06:51:23.614039+0000","last_clean":"2024-02-05T06:51:23.614039+0000","last_became_active":"2024-02-05T06:40:27.073360+0000","last_became_peered":"2024-02-05T06:40:27.073360+0000","last_unstale":"2024-02-05T06:51:23.614039+0000","last_undegraded":"2024-02-05T06:51:23.614039+0000","last_fullsized":"2024-02-05T06:51:23.614039+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'2","last_scrub_stamp":"2024-02-05T06:50:42.975975+0000","last_deep_scrub":"281'2","last_deep_scrub_stamp":"2024-02-05T06:47:50.442159+0000","last_clean_scrub_stamp":"2024-02-05T06:50:42.975975+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:00.027519+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":783,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.628518+0000","last_change":"2024-02-05T06:48:29.124890+0000","last_active":"2024-02-05T06:51:23.628518+0000","last_peered":"2024-02-05T06:51:23.628518+0000","last_clean":"2024-02-05T06:51:23.628518+0000","last_became_active":"2024-02-05T06:36:54.958069+0000","last_became_peered":"2024-02-05T06:36:54.958069+0000","last_unstale":"2024-02-05T06:51:23.628518+0000","last_undegraded":"2024-02-05T06:51:23.628518+0000","last_fullsized":"2024-02-05T06:51:23.628518+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:29.124778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:48:29.124778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":537,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.614767+0000","last_change":"2024-02-05T06:51:23.558117+0000","last_active":"2024-02-05T06:51:23.614767+0000","last_peered":"2024-02-05T06:51:23.614767+0000","last_clean":"2024-02-05T06:51:23.614767+0000","last_became_active":"2024-02-05T06:45:09.770902+0000","last_became_peered":"2024-02-05T06:45:09.770902+0000","last_unstale":"2024-02-05T06:51:23.614767+0000","last_undegraded":"2024-02-05T06:51:23.614767+0000","last_fullsized":"2024-02-05T06:51:23.614767+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:23.557998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:51:23.557998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:39.093127+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"295'7","reported_seq":628,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.663019+0000","last_change":"2024-02-05T06:50:22.976117+0000","last_active":"2024-02-05T06:51:27.663019+0000","last_peered":"2024-02-05T06:51:27.663019+0000","last_clean":"2024-02-05T06:51:27.663019+0000","last_became_active":"2024-02-05T06:45:09.808663+0000","last_became_peered":"2024-02-05T06:45:09.808663+0000","last_unstale":"2024-02-05T06:51:27.663019+0000","last_undegraded":"2024-02-05T06:51:27.663019+0000","last_fullsized":"2024-02-05T06:51:27.663019+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'7","last_scrub_stamp":"2024-02-05T06:50:22.975997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:50:22.975997+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:31.378151+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":8,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":395,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.620221+0000","last_change":"2024-02-05T06:50:30.750966+0000","last_active":"2024-02-05T06:51:23.620221+0000","last_peered":"2024-02-05T06:51:23.620221+0000","last_clean":"2024-02-05T06:51:23.620221+0000","last_became_active":"2024-02-05T06:41:43.408428+0000","last_became_peered":"2024-02-05T06:41:43.408428+0000","last_unstale":"2024-02-05T06:51:23.620221+0000","last_undegraded":"2024-02-05T06:51:23.620221+0000","last_fullsized":"2024-02-05T06:51:23.620221+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:30.750889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:50.055245+0000","last_clean_scrub_stamp":"2024-02-05T06:50:30.750889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:51.105102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":488,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.620811+0000","last_change":"2024-02-05T06:50:41.787708+0000","last_active":"2024-02-05T06:51:23.620811+0000","last_peered":"2024-02-05T06:51:23.620811+0000","last_clean":"2024-02-05T06:51:23.620811+0000","last_became_active":"2024-02-05T06:40:27.071758+0000","last_became_peered":"2024-02-05T06:40:27.071758+0000","last_unstale":"2024-02-05T06:51:23.620811+0000","last_undegraded":"2024-02-05T06:51:23.620811+0000","last_fullsized":"2024-02-05T06:51:23.620811+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:41.787606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:50:41.787606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:57.814726+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":396,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.619393+0000","last_change":"2024-02-05T06:50:58.110289+0000","last_active":"2024-02-05T06:51:23.619393+0000","last_peered":"2024-02-05T06:51:23.619393+0000","last_clean":"2024-02-05T06:51:23.619393+0000","last_became_active":"2024-02-05T06:41:43.407404+0000","last_became_peered":"2024-02-05T06:41:43.407404+0000","last_unstale":"2024-02-05T06:51:23.619393+0000","last_undegraded":"2024-02-05T06:51:23.619393+0000","last_fullsized":"2024-02-05T06:51:23.619393+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:58.109957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:21.998850+0000","last_clean_scrub_stamp":"2024-02-05T06:50:58.109957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:04.582740+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"295'19","reported_seq":630,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.661321+0000","last_change":"2024-02-05T06:48:17.662641+0000","last_active":"2024-02-05T06:51:27.661321+0000","last_peered":"2024-02-05T06:51:27.661321+0000","last_clean":"2024-02-05T06:51:27.661321+0000","last_became_active":"2024-02-05T06:45:09.787626+0000","last_became_peered":"2024-02-05T06:45:09.787626+0000","last_unstale":"2024-02-05T06:51:27.661321+0000","last_undegraded":"2024-02-05T06:51:27.661321+0000","last_fullsized":"2024-02-05T06:51:27.661321+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'19","last_scrub_stamp":"2024-02-05T06:48:17.662510+0000","last_deep_scrub":"92'13","last_deep_scrub_stamp":"2024-02-05T06:34:46.542893+0000","last_clean_scrub_stamp":"2024-02-05T06:48:17.662510+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":387,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.620541+0000","last_change":"2024-02-05T06:48:31.562380+0000","last_active":"2024-02-05T06:51:23.620541+0000","last_peered":"2024-02-05T06:51:23.620541+0000","last_clean":"2024-02-05T06:51:23.620541+0000","last_became_active":"2024-02-05T06:41:43.403746+0000","last_became_peered":"2024-02-05T06:41:43.403746+0000","last_unstale":"2024-02-05T06:51:23.620541+0000","last_undegraded":"2024-02-05T06:51:23.620541+0000","last_fullsized":"2024-02-05T06:51:23.620541+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:31.562268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:44.081318+0000","last_clean_scrub_stamp":"2024-02-05T06:48:31.562268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":338,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.663358+0000","last_change":"2024-02-05T06:50:31.175967+0000","last_active":"2024-02-05T06:51:27.663358+0000","last_peered":"2024-02-05T06:51:27.663358+0000","last_clean":"2024-02-05T06:51:27.663358+0000","last_became_active":"2024-02-05T06:42:33.129959+0000","last_became_peered":"2024-02-05T06:42:33.129959+0000","last_unstale":"2024-02-05T06:51:27.663358+0000","last_undegraded":"2024-02-05T06:51:27.663358+0000","last_fullsized":"2024-02-05T06:51:27.663358+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:31.175896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:45:51.217900+0000","last_clean_scrub_stamp":"2024-02-05T06:50:31.175896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:53.496486+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":780,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.611928+0000","last_change":"2024-02-05T06:50:39.007211+0000","last_active":"2024-02-05T06:51:23.611928+0000","last_peered":"2024-02-05T06:51:23.611928+0000","last_clean":"2024-02-05T06:51:23.611928+0000","last_became_active":"2024-02-05T06:40:27.068888+0000","last_became_peered":"2024-02-05T06:40:27.068888+0000","last_unstale":"2024-02-05T06:51:23.611928+0000","last_undegraded":"2024-02-05T06:51:23.611928+0000","last_fullsized":"2024-02-05T06:51:23.611928+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:39.006521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:44:55.125921+0000","last_clean_scrub_stamp":"2024-02-05T06:50:39.006521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:53.142393+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"295'34","reported_seq":819,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.661673+0000","last_change":"2024-02-05T06:47:48.407121+0000","last_active":"2024-02-05T06:51:27.661673+0000","last_peered":"2024-02-05T06:51:27.661673+0000","last_clean":"2024-02-05T06:51:27.661673+0000","last_became_active":"2024-02-05T06:33:31.572544+0000","last_became_peered":"2024-02-05T06:33:31.572544+0000","last_unstale":"2024-02-05T06:51:27.661673+0000","last_undegraded":"2024-02-05T06:51:27.661673+0000","last_fullsized":"2024-02-05T06:51:27.661673+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'34","last_scrub_stamp":"2024-02-05T06:47:48.406976+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:39:35.433539+0000","last_clean_scrub_stamp":"2024-02-05T06:47:48.406976+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":782,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.661738+0000","last_change":"2024-02-05T06:51:23.547319+0000","last_active":"2024-02-05T06:51:27.661738+0000","last_peered":"2024-02-05T06:51:27.661738+0000","last_clean":"2024-02-05T06:51:27.661738+0000","last_became_active":"2024-02-05T06:33:29.553864+0000","last_became_peered":"2024-02-05T06:33:29.553864+0000","last_unstale":"2024-02-05T06:51:27.661738+0000","last_undegraded":"2024-02-05T06:51:27.661738+0000","last_fullsized":"2024-02-05T06:51:27.661738+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:23.547188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:51:23.547188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:40.738667+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":301,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:24.579487+0000","last_change":"2024-02-05T06:51:24.579523+0000","last_active":"2024-02-05T06:51:24.579487+0000","last_peered":"2024-02-05T06:51:24.579487+0000","last_clean":"2024-02-05T06:51:24.579487+0000","last_became_active":"2024-02-05T06:45:09.775037+0000","last_became_peered":"2024-02-05T06:45:09.775037+0000","last_unstale":"2024-02-05T06:51:24.579487+0000","last_undegraded":"2024-02-05T06:51:24.579487+0000","last_fullsized":"2024-02-05T06:51:24.579487+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:24.579415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:51.388156+0000","last_clean_scrub_stamp":"2024-02-05T06:51:24.579415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:41.558209+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":773,"reported_epoch":424,"state":"active+clean","last_fresh":"2024-02-05T06:51:27.663246+0000","last_change":"2024-02-05T06:50:26.260667+0000","last_active":"2024-02-05T06:51:27.663246+0000","last_peered":"2024-02-05T06:51:27.663246+0000","last_clean":"2024-02-05T06:51:27.663246+0000","last_became_active":"2024-02-05T06:40:27.077485+0000","last_became_peered":"2024-02-05T06:40:27.077485+0000","last_unstale":"2024-02-05T06:51:27.663246+0000","last_undegraded":"2024-02-05T06:51:27.663246+0000","last_fullsized":"2024-02-05T06:51:27.663246+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:26.260536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:50:26.260536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:43.819004+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"295'19","reported_seq":704,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.613590+0000","last_change":"2024-02-05T06:50:21.626059+0000","last_active":"2024-02-05T06:51:23.613590+0000","last_peered":"2024-02-05T06:51:23.613590+0000","last_clean":"2024-02-05T06:51:23.613590+0000","last_became_active":"2024-02-05T06:36:54.959330+0000","last_became_peered":"2024-02-05T06:36:54.959330+0000","last_unstale":"2024-02-05T06:51:23.613590+0000","last_undegraded":"2024-02-05T06:51:23.613590+0000","last_fullsized":"2024-02-05T06:51:23.613590+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'19","last_scrub_stamp":"2024-02-05T06:50:21.625812+0000","last_deep_scrub":"247'13","last_deep_scrub_stamp":"2024-02-05T06:41:42.473424+0000","last_clean_scrub_stamp":"2024-02-05T06:50:21.625812+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:48.872122+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":22,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":785,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.613157+0000","last_change":"2024-02-05T06:48:20.602707+0000","last_active":"2024-02-05T06:51:23.613157+0000","last_peered":"2024-02-05T06:51:23.613157+0000","last_clean":"2024-02-05T06:51:23.613157+0000","last_became_active":"2024-02-05T06:41:43.421315+0000","last_became_peered":"2024-02-05T06:41:43.421315+0000","last_unstale":"2024-02-05T06:51:23.613157+0000","last_undegraded":"2024-02-05T06:51:23.613157+0000","last_fullsized":"2024-02-05T06:51:23.613157+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:48:20.602591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:48:20.602591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":803,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:26.669164+0000","last_change":"2024-02-05T06:51:26.669201+0000","last_active":"2024-02-05T06:51:26.669164+0000","last_peered":"2024-02-05T06:51:26.669164+0000","last_clean":"2024-02-05T06:51:26.669164+0000","last_became_active":"2024-02-05T06:41:43.420342+0000","last_became_peered":"2024-02-05T06:41:43.420342+0000","last_unstale":"2024-02-05T06:51:26.669164+0000","last_undegraded":"2024-02-05T06:51:26.669164+0000","last_fullsized":"2024-02-05T06:51:26.669164+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:26.669087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:06.525012+0000","last_clean_scrub_stamp":"2024-02-05T06:51:26.669087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:52:50.420709+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":532,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-02-05T06:51:23.620220+0000","last_change":"2024-02-05T06:50:39.817837+0000","last_active":"2024-02-05T06:51:23.620220+0000","last_peered":"2024-02-05T06:51:23.620220+0000","last_clean":"2024-02-05T06:51:23.620220+0000","last_became_active":"2024-02-05T06:45:09.787134+0000","last_became_peered":"2024-02-05T06:45:09.787134+0000","last_unstale":"2024-02-05T06:51:23.620220+0000","last_undegraded":"2024-02-05T06:51:23.620220+0000","last_fullsized":"2024-02-05T06:51:23.620220+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:39.817583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:50:39.817583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:51:41.621800+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":1020657664,"num_objects":15574,"num_object_clones":0,"num_object_copies":46722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15574,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15574,"num_write_kb":996736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2996568064,"data_stored":2996568064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15574,"ondisk_log_size":15574,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":665,"num_read_kb":660,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1573408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1106,"num_read_kb":863,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":373,"num_bytes_recovered":8234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":782,"ondisk_log_size":782,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103387,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":957000,"kb_used_data":299712,"kb_used_omap":1,"kb_used_meta":657278,"kb_avail":92788152,"statfs":{"total":95995035648,"available":95015067648,"internally_reserved":0,"allocated":306905088,"data_stored":306512052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":673053130},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:51:03 2024","interfaces":[{"interface":"back","average":{"1min":1.6160000000000001,"5min":1.147,"15min":1.607},"min":{"1min":0.23400000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":5.165,"5min":5.2329999999999997,"15min":68.921000000000006},"last":0.879},{"interface":"front","average":{"1min":3.27,"5min":1.4610000000000001,"15min":1.849},"min":{"1min":0.26400000000000001,"5min":0.247,"15min":0.222},"max":{"1min":41.109999999999999,"5min":41.109999999999999,"15min":69.616},"last":0.91200000000000003}]},{"osd":1,"last update":"Mon Feb 5 06:51:03 2024","interfaces":[{"interface":"back","average":{"1min":2.2730000000000001,"5min":1.4039999999999999,"15min":2.0009999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.25},"max":{"1min":23.716999999999999,"5min":23.716999999999999,"15min":89.231999999999999},"last":2.105},{"interface":"front","average":{"1min":4.2619999999999996,"5min":1.6859999999999999,"15min":2.157},"min":{"1min":0.30299999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":41.167999999999999,"5min":41.167999999999999,"15min":87.995999999999995},"last":0.377}]},{"osd":2,"last update":"Mon Feb 5 06:51:03 2024","interfaces":[{"interface":"back","average":{"1min":2.653,"5min":1.3899999999999999,"15min":1.474},"min":{"1min":0.35099999999999998,"5min":0.30199999999999999,"15min":0.26200000000000001},"max":{"1min":28.704999999999998,"5min":28.704999999999998,"15min":53.947000000000003},"last":1.429},{"interface":"front","average":{"1min":2.456,"5min":1.4219999999999999,"15min":1.448},"min":{"1min":0.30499999999999999,"5min":0.28899999999999998,"15min":0.26800000000000002},"max":{"1min":23.678999999999998,"5min":23.678999999999998,"15min":53.299999999999997},"last":1.054}]},{"osd":3,"last update":"Mon Feb 5 06:51:03 2024","interfaces":[{"interface":"back","average":{"1min":2.4510000000000001,"5min":2.3980000000000001,"15min":2.3490000000000002},"min":{"1min":0.41899999999999998,"5min":0.29099999999999998,"15min":0.28899999999999998},"max":{"1min":23.199000000000002,"5min":73.283000000000001,"15min":89.481999999999999},"last":2},{"interface":"front","average":{"1min":4.2409999999999997,"5min":2.7709999999999999,"15min":2.431},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.27400000000000002},"max":{"1min":41.219000000000001,"5min":73.323999999999998,"15min":92.293999999999997},"last":0.99299999999999999}]},{"osd":4,"last update":"Mon Feb 5 06:51:03 2024","interfaces":[{"interface":"back","average":{"1min":1.534,"5min":1.034,"15min":0.76300000000000001},"min":{"1min":0.28599999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":5.4470000000000001,"5min":6.1980000000000004,"15min":6.1980000000000004},"last":0.95099999999999996},{"interface":"front","average":{"1min":3.7010000000000001,"5min":1.403,"15min":0.872},"min":{"1min":0.28299999999999997,"5min":0.25600000000000001,"15min":0.23400000000000001},"max":{"1min":41.270000000000003,"5min":41.270000000000003,"15min":41.270000000000003},"last":0.45400000000000001}]},{"osd":5,"last update":"Mon Feb 5 06:51:03 2024","interfaces":[{"interface":"back","average":{"1min":2.927,"5min":2.4609999999999999,"15min":1.3089999999999999},"min":{"1min":0.32600000000000001,"5min":0.27200000000000002,"15min":0.251},"max":{"1min":41.255000000000003,"5min":109.482,"15min":109.482},"last":2.0640000000000001},{"interface":"front","average":{"1min":3.262,"5min":2.6339999999999999,"15min":1.4279999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.252},"max":{"1min":41.203000000000003,"5min":109.56399999999999,"15min":109.56399999999999},"last":0.41199999999999998}]},{"osd":6,"last update":"Mon Feb 5 06:51:03 2024","interfaces":[{"interface":"back","average":{"1min":3.7559999999999998,"5min":2.8410000000000002,"15min":1.458},"min":{"1min":0.376,"5min":0.34899999999999998,"15min":0.28499999999999998},"max":{"1min":41.326999999999998,"5min":113.914,"15min":113.914},"last":0.48799999999999999},{"interface":"front","average":{"1min":3.6299999999999999,"5min":2.7360000000000002,"15min":1.3939999999999999},"min":{"1min":0.39400000000000002,"5min":0.36099999999999999,"15min":0.245},"max":{"1min":41.384,"5min":113.212,"15min":113.212},"last":0.53300000000000003}]}]},{"osd":6,"up_from":49,"seq":210453397729,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1016240,"kb_used_data":435304,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":92728912,"statfs":{"total":95995035648,"available":94954405888,"internally_reserved":0,"allocated":445751296,"data_stored":445381387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":594868678},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:50:29 2024","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.292,"15min":2.3170000000000002},"min":{"1min":0.33800000000000002,"5min":0.22900000000000001,"15min":0.224},"max":{"1min":5.7240000000000002,"5min":18.393000000000001,"15min":98.918999999999997},"last":0.34499999999999997},{"interface":"front","average":{"1min":1.7649999999999999,"5min":1.498,"15min":2.351},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":9.3940000000000001,"5min":17.995999999999999,"15min":98.775000000000006},"last":0.84799999999999998}]},{"osd":1,"last update":"Mon Feb 5 06:50:29 2024","interfaces":[{"interface":"back","average":{"1min":3.637,"5min":1.4950000000000001,"15min":2.1459999999999999},"min":{"1min":0.254,"5min":0.249,"15min":0.24099999999999999},"max":{"1min":47.348999999999997,"5min":47.348999999999997,"15min":98.843999999999994},"last":0.67800000000000005},{"interface":"front","average":{"1min":3.2679999999999998,"5min":1.357,"15min":2.089},"min":{"1min":0.32600000000000001,"5min":0.29099999999999998,"15min":0.23100000000000001},"max":{"1min":47.777999999999999,"5min":47.777999999999999,"15min":101.637},"last":0.94499999999999995}]},{"osd":2,"last update":"Mon Feb 5 06:50:29 2024","interfaces":[{"interface":"back","average":{"1min":1.7350000000000001,"5min":1.357,"15min":2.0099999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.22500000000000001},"max":{"1min":4.548,"5min":16.131,"15min":85.617000000000004},"last":0.38600000000000001},{"interface":"front","average":{"1min":1.5800000000000001,"5min":1.484,"15min":1.944},"min":{"1min":0.318,"5min":0.29599999999999999,"15min":0.27000000000000002},"max":{"1min":5.0990000000000002,"5min":16.210000000000001,"15min":85.335999999999999},"last":0.89100000000000001}]},{"osd":3,"last update":"Mon Feb 5 06:50:29 2024","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.119,"15min":2.4580000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27000000000000002},"max":{"1min":4.2309999999999999,"5min":4.2309999999999999,"15min":99.269000000000005},"last":1.591},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.0189999999999999,"15min":2.5299999999999998},"min":{"1min":0.33200000000000002,"5min":0.28799999999999998,"15min":0.27800000000000002},"max":{"1min":7.327,"5min":7.327,"15min":101.75700000000001},"last":0.73399999999999999}]},{"osd":4,"last update":"Mon Feb 5 06:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.85499999999999998,"15min":0.63600000000000001},"min":{"1min":0.29899999999999999,"5min":0.28899999999999998,"15min":0.23100000000000001},"max":{"1min":2.9489999999999998,"5min":4.9580000000000002,"15min":4.9580000000000002},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.0920000000000001,"5min":0.95499999999999996,"15min":0.67000000000000004},"min":{"1min":0.312,"5min":0.249,"15min":0.24299999999999999},"max":{"1min":4.024,"5min":5.0069999999999997,"15min":5.0069999999999997},"last":0.64500000000000002}]},{"osd":5,"last update":"Mon Feb 5 06:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.96499999999999997,"15min":0.878},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.251},"max":{"1min":4.9610000000000003,"5min":5.5570000000000004,"15min":36.548999999999999},"last":1.1060000000000001},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.92500000000000004,"15min":0.88700000000000001},"min":{"1min":0.33600000000000002,"5min":0.30299999999999999,"15min":0.24299999999999999},"max":{"1min":3.5760000000000001,"5min":4.6269999999999998,"15min":36.433},"last":0.60699999999999998}]},{"osd":7,"last update":"Mon Feb 5 06:50:42 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.79700000000000004,"15min":0.81999999999999995},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":2.5070000000000001,"5min":4.0439999999999996,"15min":38.436999999999998},"last":0.871},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.0289999999999999,"15min":0.91700000000000004},"min":{"1min":0.39000000000000001,"5min":0.34699999999999998,"15min":0.29699999999999999},"max":{"1min":5.3280000000000003,"5min":5.3280000000000003,"15min":36.633000000000003},"last":1.4950000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575098,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1194792,"kb_used_data":463904,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":92550360,"statfs":{"total":95995035648,"available":94771568640,"internally_reserved":0,"allocated":475037696,"data_stored":474612675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:51:27 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.637,"15min":1.867},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.16700000000000001},"max":{"1min":3.3359999999999999,"5min":11.276999999999999,"15min":43.651000000000003},"last":0.66000000000000003},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.4770000000000001,"15min":1.976},"min":{"1min":0.245,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":3.7949999999999999,"5min":9.1829999999999998,"15min":45.670999999999999},"last":1.5800000000000001}]},{"osd":2,"last update":"Mon Feb 5 06:50:47 2024","interfaces":[{"interface":"back","average":{"1min":2.5790000000000002,"5min":1.5009999999999999,"15min":1.7450000000000001},"min":{"1min":0.222,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":11.747999999999999,"5min":11.747999999999999,"15min":43.447000000000003},"last":1.044},{"interface":"front","average":{"1min":1.99,"5min":1.3979999999999999,"15min":1.704},"min":{"1min":0.30099999999999999,"5min":0.22700000000000001,"15min":0.20899999999999999},"max":{"1min":10.856999999999999,"5min":10.856999999999999,"15min":22.050000000000001},"last":0.68200000000000005}]},{"osd":3,"last update":"Mon Feb 5 06:51:11 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":1.2430000000000001,"15min":2.3500000000000001},"min":{"1min":0.28100000000000003,"5min":0.26000000000000001,"15min":0.23499999999999999},"max":{"1min":3.0310000000000001,"5min":7.1109999999999998,"15min":69.798000000000002},"last":1.609},{"interface":"front","average":{"1min":1.7749999999999999,"5min":1.5680000000000001,"15min":2.1230000000000002},"min":{"1min":0.35199999999999998,"5min":0.25600000000000001,"15min":0.23699999999999999},"max":{"1min":4.7960000000000003,"5min":11.018000000000001,"15min":70.765000000000001},"last":1.0780000000000001}]},{"osd":4,"last update":"Mon Feb 5 06:50:32 2024","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.0920000000000001,"15min":1.23},"min":{"1min":0.34899999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":7.4059999999999997,"5min":7.4059999999999997,"15min":11.679},"last":0.83399999999999996},{"interface":"front","average":{"1min":2.0630000000000002,"5min":1.353,"15min":1.3899999999999999},"min":{"1min":0.437,"5min":0.317,"15min":0.26300000000000001},"max":{"1min":6.9870000000000001,"5min":10.419,"15min":10.419},"last":1.478}]},{"osd":5,"last update":"Mon Feb 5 06:51:09 2024","interfaces":[{"interface":"back","average":{"1min":1.587,"5min":1.478,"15min":1.4490000000000001},"min":{"1min":0.435,"5min":0.35299999999999998,"15min":0.29299999999999998},"max":{"1min":6.3280000000000003,"5min":11.058,"15min":43.414999999999999},"last":1.008},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0469999999999999,"15min":1.3720000000000001},"min":{"1min":0.41599999999999998,"5min":0.35699999999999998,"15min":0.33100000000000002},"max":{"1min":2.895,"5min":8.1669999999999998,"15min":13.455},"last":0.87}]},{"osd":6,"last update":"Mon Feb 5 06:51:27 2024","interfaces":[{"interface":"back","average":{"1min":2.0339999999999998,"5min":1.3400000000000001,"15min":1.5069999999999999},"min":{"1min":0.44900000000000001,"5min":0.39000000000000001,"15min":0.34799999999999998},"max":{"1min":19.827999999999999,"5min":19.827999999999999,"15min":23.902999999999999},"last":0.92200000000000004},{"interface":"front","average":{"1min":2.3769999999999998,"5min":1.7689999999999999,"15min":1.5669999999999999},"min":{"1min":0.40999999999999998,"5min":0.39300000000000002,"15min":0.35599999999999998},"max":{"1min":19.526,"5min":19.526,"15min":43.475000000000001},"last":0.95199999999999996}]},{"osd":7,"last update":"Mon Feb 5 06:50:47 2024","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.26,"15min":1.4370000000000001},"min":{"1min":0.45000000000000001,"5min":0.34100000000000003,"15min":0.32900000000000001},"max":{"1min":7.4480000000000004,"5min":8.016,"15min":13.49},"last":0.89700000000000002},{"interface":"front","average":{"1min":1.796,"5min":1.371,"15min":1.4470000000000001},"min":{"1min":0.51000000000000001,"5min":0.36599999999999999,"15min":0.35499999999999998},"max":{"1min":6.9569999999999999,"5min":10.481,"15min":10.481},"last":1.5109999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738624,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":930776,"kb_used_data":272080,"kb_used_omap":1,"kb_used_meta":658686,"kb_avail":92814376,"statfs":{"total":95995035648,"available":95041921024,"internally_reserved":0,"allocated":278609920,"data_stored":278200476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":674494917},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 06:51:18 2024","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.2769999999999999,"15min":1.675},"min":{"1min":0.22900000000000001,"5min":0.19700000000000001,"15min":0.182},"max":{"1min":5.2160000000000002,"5min":12.199,"15min":18.59},"last":2.0139999999999998},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.667,"15min":1.79},"min":{"1min":0.26500000000000001,"5min":0.23799999999999999,"15min":0.18099999999999999},"max":{"1min":2.8109999999999999,"5min":8.7859999999999996,"15min":21.088000000000001},"last":2.2869999999999999}]},{"osd":2,"last update":"Mon Feb 5 06:50:38 2024","interfaces":[{"interface":"back","average":{"1min":2.0019999999999998,"5min":1.8560000000000001,"15min":1.7350000000000001},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":6.4969999999999999,"5min":10.901999999999999,"15min":16.981999999999999},"last":1.631},{"interface":"front","average":{"1min":1.9099999999999999,"5min":1.5549999999999999,"15min":1.5640000000000001},"min":{"1min":0.374,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.7750000000000004,"5min":11.943,"15min":16.998999999999999},"last":1.1890000000000001}]},{"osd":3,"last update":"Mon Feb 5 06:51:09 2024","interfaces":[{"interface":"back","average":{"1min":1.6859999999999999,"5min":1.5449999999999999,"15min":1.9330000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":5.4249999999999998,"5min":12.167,"15min":43.302},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.704,"5min":1.4690000000000001,"15min":2.0569999999999999},"min":{"1min":0.33200000000000002,"5min":0.27900000000000003,"15min":0.23799999999999999},"max":{"1min":5.2400000000000002,"5min":6.7759999999999998,"15min":43.651000000000003},"last":1.05}]},{"osd":4,"last update":"Mon Feb 5 06:50:28 2024","interfaces":[{"interface":"back","average":{"1min":1.6910000000000001,"5min":1.6639999999999999,"15min":1.5580000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.27200000000000002},"max":{"1min":6.4020000000000001,"5min":10.952999999999999,"15min":40.789999999999999},"last":1.544},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.343,"15min":1.5009999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.30099999999999999},"max":{"1min":5.8289999999999997,"5min":5.9459999999999997,"15min":40.773000000000003},"last":0.54500000000000004}]},{"osd":5,"last update":"Mon Feb 5 06:51:03 2024","interfaces":[{"interface":"back","average":{"1min":1.853,"5min":1.3839999999999999,"15min":1.593},"min":{"1min":0.495,"5min":0.32400000000000001,"15min":0.29299999999999998},"max":{"1min":5.1859999999999999,"5min":6.8250000000000002,"15min":48.841999999999999},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.792,"15min":1.6899999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.28599999999999998},"max":{"1min":4.4429999999999996,"5min":11.156000000000001,"15min":48.450000000000003},"last":1.5880000000000001}]},{"osd":6,"last update":"Mon Feb 5 06:51:18 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":2.952,"15min":1.899},"min":{"1min":0.50700000000000001,"5min":0.39400000000000002,"15min":0.33700000000000002},"max":{"1min":4.2619999999999996,"5min":136.76300000000001,"15min":136.76300000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":1.8169999999999999,"5min":3.1669999999999998,"15min":2.02},"min":{"1min":0.44,"5min":0.33400000000000002,"15min":0.33000000000000002},"max":{"1min":5.0999999999999996,"5min":135.92400000000001,"15min":135.92400000000001},"last":1.3109999999999999}]},{"osd":7,"last update":"Mon Feb 5 06:50:43 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.7889999999999999,"15min":1.5129999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.31900000000000001},"max":{"1min":6.3730000000000002,"5min":10.989000000000001,"15min":17.210000000000001},"last":1.218},{"interface":"front","average":{"1min":1.133,"5min":1.369,"15min":1.3999999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.34899999999999998},"max":{"1min":4.2919999999999998,"5min":6.7220000000000004,"15min":18.533999999999999},"last":0.66300000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411573,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":952320,"kb_used_data":339448,"kb_used_omap":1,"kb_used_meta":612862,"kb_avail":92792832,"statfs":{"total":95995035648,"available":95019859968,"internally_reserved":0,"allocated":347594752,"data_stored":347210321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":627571141},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:50:48 2024","interfaces":[{"interface":"back","average":{"1min":6.2110000000000003,"5min":3.8170000000000002,"15min":2.8100000000000001},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":86.593000000000004,"5min":101.67700000000001,"15min":101.67700000000001},"last":0.28899999999999998},{"interface":"front","average":{"1min":5.6079999999999997,"5min":3.5579999999999998,"15min":2.6850000000000001},"min":{"1min":0.23599999999999999,"5min":0.23100000000000001,"15min":0.17699999999999999},"max":{"1min":86.274000000000001,"5min":101.27800000000001,"15min":101.27800000000001},"last":1.6990000000000001}]},{"osd":1,"last update":"Mon Feb 5 06:50:48 2024","interfaces":[{"interface":"back","average":{"1min":3.1659999999999999,"5min":1.6359999999999999,"15min":1.9630000000000001},"min":{"1min":0.23999999999999999,"5min":0.185,"15min":0.185},"max":{"1min":24.093,"5min":24.093,"15min":31.303000000000001},"last":0.248},{"interface":"front","average":{"1min":3.0449999999999999,"5min":1.5449999999999999,"15min":1.653},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":27.114000000000001,"5min":27.114000000000001,"15min":31.425000000000001},"last":0.5}]},{"osd":3,"last update":"Mon Feb 5 06:51:16 2024","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":2.3879999999999999,"15min":2.411},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.20899999999999999},"max":{"1min":3.6379999999999999,"5min":84.129000000000005,"15min":84.129000000000005},"last":0.53400000000000003},{"interface":"front","average":{"1min":1.3300000000000001,"5min":2.2400000000000002,"15min":2.2869999999999999},"min":{"1min":0.29999999999999999,"5min":0.214,"15min":0.214},"max":{"1min":5.4889999999999999,"5min":84.090999999999994,"15min":84.090999999999994},"last":0.76400000000000001}]},{"osd":4,"last update":"Mon Feb 5 06:50:29 2024","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.1100000000000001,"15min":1.1240000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.23999999999999999},"max":{"1min":4.1509999999999998,"5min":4.5759999999999996,"15min":12.041},"last":0.32300000000000001},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.0589999999999999,"15min":1.079},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.26300000000000001},"max":{"1min":4.335,"5min":4.907,"15min":15.689},"last":1.754}]},{"osd":5,"last update":"Mon Feb 5 06:51:00 2024","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.625,"15min":1.7789999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.29899999999999999},"max":{"1min":5.4029999999999996,"5min":27.109000000000002,"15min":86.614999999999995},"last":0.88},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.522,"15min":1.944},"min":{"1min":0.32300000000000001,"5min":0.32000000000000001,"15min":0.312},"max":{"1min":3.9089999999999998,"5min":27.042000000000002,"15min":86.543999999999997},"last":0.36699999999999999}]},{"osd":6,"last update":"Mon Feb 5 06:50:29 2024","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.446,"15min":1.198},"min":{"1min":0.40100000000000002,"5min":0.36599999999999999,"15min":0.315},"max":{"1min":2.7050000000000001,"5min":29.780999999999999,"15min":29.780999999999999},"last":1.8460000000000001},{"interface":"front","average":{"1min":1.327,"5min":1.726,"15min":1.573},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.316},"max":{"1min":4.8079999999999998,"5min":29.763000000000002,"15min":29.763000000000002},"last":0.55700000000000005}]},{"osd":7,"last update":"Mon Feb 5 06:50:48 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.3280000000000001,"15min":1.6279999999999999},"min":{"1min":0.377,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":4.8499999999999996,"5min":8.5050000000000008,"15min":45.243000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.6679999999999999,"15min":1.579},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":4.3280000000000003,"5min":17.611000000000001,"15min":45.171999999999997},"last":0.70499999999999996}]}]},{"osd":3,"up_from":26,"seq":111669149936,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1190548,"kb_used_data":461068,"kb_used_omap":1,"kb_used_meta":729470,"kb_avail":92554604,"statfs":{"total":95995035648,"available":94775914496,"internally_reserved":0,"allocated":472133632,"data_stored":471728184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":746977735},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:51:23 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":2.399,"15min":1.8919999999999999},"min":{"1min":0.23799999999999999,"5min":0.21099999999999999,"15min":0.184},"max":{"1min":3.121,"5min":53.656999999999996,"15min":53.656999999999996},"last":0.20200000000000001},{"interface":"front","average":{"1min":1.292,"5min":2.653,"15min":2.1829999999999998},"min":{"1min":0.29099999999999998,"5min":0.217,"15min":0.20399999999999999},"max":{"1min":6.7300000000000004,"5min":53.338000000000001,"15min":53.338000000000001},"last":1.538}]},{"osd":1,"last update":"Mon Feb 5 06:51:23 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.373,"15min":1.7509999999999999},"min":{"1min":0.25900000000000001,"5min":0.24299999999999999,"15min":0.20499999999999999},"max":{"1min":7.0890000000000004,"5min":9.9420000000000002,"15min":11.412000000000001},"last":4.024},{"interface":"front","average":{"1min":1.494,"5min":1.3089999999999999,"15min":1.7190000000000001},"min":{"1min":0.27100000000000002,"5min":0.23300000000000001,"15min":0.215},"max":{"1min":3.2200000000000002,"5min":8.6590000000000007,"15min":9.8659999999999997},"last":0.70999999999999996}]},{"osd":2,"last update":"Mon Feb 5 06:51:23 2024","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.659,"15min":1.901},"min":{"1min":0.38400000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.5369999999999999,"5min":14.746,"15min":41.529000000000003},"last":0.747},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.3959999999999999,"15min":1.726},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":5.4909999999999997,"5min":14.734,"15min":41.759999999999998},"last":0.92500000000000004}]},{"osd":4,"last update":"Mon Feb 5 06:50:38 2024","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.1850000000000001,"15min":1.238},"min":{"1min":0.32800000000000001,"5min":0.32200000000000001,"15min":0.29799999999999999},"max":{"1min":5.3929999999999998,"5min":7.3979999999999997,"15min":16.577999999999999},"last":0.35299999999999998},{"interface":"front","average":{"1min":2.2080000000000002,"5min":1.4199999999999999,"15min":1.587},"min":{"1min":0.36499999999999999,"5min":0.36199999999999999,"15min":0.28599999999999998},"max":{"1min":7.6769999999999996,"5min":8.7260000000000009,"15min":19.359999999999999},"last":1.5760000000000001}]},{"osd":5,"last update":"Mon Feb 5 06:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.1160000000000001,"15min":1.554},"min":{"1min":0.38100000000000001,"5min":0.34799999999999998,"15min":0.29299999999999998},"max":{"1min":3.5249999999999999,"5min":3.5249999999999999,"15min":20.257999999999999},"last":0.78300000000000003},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.254,"15min":1.3140000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.28399999999999997},"max":{"1min":5.4640000000000004,"5min":7.4530000000000003,"15min":15.792999999999999},"last":0.373}]},{"osd":6,"last update":"Mon Feb 5 06:50:34 2024","interfaces":[{"interface":"back","average":{"1min":2.3050000000000002,"5min":1.472,"15min":1.5309999999999999},"min":{"1min":0.373,"5min":0.34100000000000003,"15min":0.28100000000000003},"max":{"1min":7.8579999999999997,"5min":8.6660000000000004,"15min":8.6660000000000004},"last":1.7589999999999999},{"interface":"front","average":{"1min":2.2170000000000001,"5min":1.5429999999999999,"15min":1.518},"min":{"1min":0.56000000000000005,"5min":0.46300000000000002,"15min":0.28599999999999998},"max":{"1min":12.75,"5min":12.75,"15min":12.75},"last":1.002}]},{"osd":7,"last update":"Mon Feb 5 06:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.3320000000000001,"15min":1.498},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.33400000000000002},"max":{"1min":5.4329999999999998,"5min":7.4279999999999999,"15min":45.039000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":1.577,"5min":1.292,"15min":1.631},"min":{"1min":0.47999999999999998,"5min":0.377,"15min":0.34399999999999997},"max":{"1min":4.4000000000000004,"5min":4.4000000000000004,"15min":44.036999999999999},"last":0.83699999999999997}]}]},{"osd":4,"up_from":31,"seq":133143986410,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":846672,"kb_used_data":197704,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92898480,"statfs":{"total":95995035648,"available":95128043520,"internally_reserved":0,"allocated":202448896,"data_stored":202047222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:50:32 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.0720000000000001,"15min":1.2989999999999999},"min":{"1min":0.23899999999999999,"5min":0.216,"15min":0.20200000000000001},"max":{"1min":5.0460000000000003,"5min":5.6559999999999997,"15min":33.372},"last":0.92300000000000004},{"interface":"front","average":{"1min":1.341,"5min":1.238,"15min":1.4630000000000001},"min":{"1min":0.26000000000000001,"5min":0.247,"15min":0.22700000000000001},"max":{"1min":5.952,"5min":10.913,"15min":33.323999999999998},"last":1.6160000000000001}]},{"osd":1,"last update":"Mon Feb 5 06:50:32 2024","interfaces":[{"interface":"back","average":{"1min":1.8140000000000001,"5min":1.8300000000000001,"15min":1.423},"min":{"1min":0.32500000000000001,"5min":0.254,"15min":0.21299999999999999},"max":{"1min":8.0600000000000005,"5min":53.414000000000001,"15min":53.414000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":1.339,"5min":1.6200000000000001,"15min":1.3400000000000001},"min":{"1min":0.27300000000000002,"5min":0.255,"15min":0.23000000000000001},"max":{"1min":5.3440000000000003,"5min":53.719999999999999,"15min":53.719999999999999},"last":1.7490000000000001}]},{"osd":2,"last update":"Mon Feb 5 06:50:32 2024","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":0.98299999999999998,"15min":0.92500000000000004},"min":{"1min":0.27700000000000002,"5min":0.26000000000000001,"15min":0.245},"max":{"1min":7.3630000000000004,"5min":7.3630000000000004,"15min":17.379999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.91000000000000003,"15min":1.1319999999999999},"min":{"1min":0.28399999999999997,"5min":0.28000000000000003,"15min":0.26000000000000001},"max":{"1min":2.7200000000000002,"5min":3.1440000000000001,"15min":18.077000000000002},"last":0.41699999999999998}]},{"osd":3,"last update":"Mon Feb 5 06:50:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.643,"15min":2.5129999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.252},"max":{"1min":5.468,"5min":33.923999999999999,"15min":99.546999999999997},"last":0.45600000000000002},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.4850000000000001,"15min":2.464},"min":{"1min":0.41399999999999998,"5min":0.32000000000000001,"15min":0.29099999999999998},"max":{"1min":3.0529999999999999,"5min":33.804000000000002,"15min":99.245000000000005},"last":0.39900000000000002}]},{"osd":5,"last update":"Mon Feb 5 06:51:00 2024","interfaces":[{"interface":"back","average":{"1min":1.5309999999999999,"5min":0.96399999999999997,"15min":0.81899999999999995},"min":{"1min":0.32600000000000001,"5min":0.249,"15min":0.24399999999999999},"max":{"1min":4.649,"5min":5.6059999999999999,"15min":11.098000000000001},"last":0.34200000000000003},{"interface":"front","average":{"1min":1.3999999999999999,"5min":0.93899999999999995,"15min":0.77800000000000002},"min":{"1min":0.30499999999999999,"5min":0.28499999999999998,"15min":0.252},"max":{"1min":7.7290000000000001,"5min":7.7290000000000001,"15min":10.965999999999999},"last":0.435}]},{"osd":6,"last update":"Mon Feb 5 06:50:32 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.96399999999999997,"15min":0.83299999999999996},"min":{"1min":0.40699999999999997,"5min":0.33900000000000002,"15min":0.246},"max":{"1min":2.9460000000000002,"5min":5.9119999999999999,"15min":11.868},"last":0.70699999999999996},{"interface":"front","average":{"1min":1.0649999999999999,"5min":0.97199999999999998,"15min":0.82999999999999996},"min":{"1min":0.44600000000000001,"5min":0.29499999999999998,"15min":0.27600000000000002},"max":{"1min":2.907,"5min":5.8970000000000002,"15min":11.787000000000001},"last":0.67800000000000005}]},{"osd":7,"last update":"Mon Feb 5 06:50:43 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.77000000000000002,"15min":0.71799999999999997},"min":{"1min":0.34999999999999998,"5min":0.27400000000000002,"15min":0.26000000000000001},"max":{"1min":2.7370000000000001,"5min":2.7370000000000001,"15min":5.1500000000000004},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.92000000000000004,"15min":0.77600000000000002},"min":{"1min":0.36199999999999999,"5min":0.307,"15min":0.28499999999999998},"max":{"1min":4.5970000000000004,"5min":4.5970000000000004,"15min":5.5890000000000004},"last":0.51700000000000002}]}]},{"osd":5,"up_from":40,"seq":171798692069,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1037568,"kb_used_data":230328,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":92707584,"statfs":{"total":95995035648,"available":94932566016,"internally_reserved":0,"allocated":235855872,"data_stored":235471335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":826603974},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:50:51 2024","interfaces":[{"interface":"back","average":{"1min":2.528,"5min":1.7569999999999999,"15min":1.8340000000000001},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.19700000000000001},"max":{"1min":9.4309999999999992,"5min":22.326000000000001,"15min":55.704999999999998},"last":1.7090000000000001},{"interface":"front","average":{"1min":1.8380000000000001,"5min":1.6459999999999999,"15min":1.931},"min":{"1min":0.28399999999999997,"5min":0.26700000000000002,"15min":0.22600000000000001},"max":{"1min":5.306,"5min":22.247,"15min":56.789000000000001},"last":0.77700000000000002}]},{"osd":1,"last update":"Mon Feb 5 06:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":4.6470000000000002,"15min":2.6890000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.248},"max":{"1min":4.3170000000000002,"5min":317.53399999999999,"15min":317.53399999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":1.909,"5min":4.7560000000000002,"15min":2.5840000000000001},"min":{"1min":0.23599999999999999,"5min":0.22,"15min":0.22},"max":{"1min":9.4030000000000005,"5min":317.80500000000001,"15min":317.80500000000001},"last":3.1230000000000002}]},{"osd":2,"last update":"Mon Feb 5 06:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.9870000000000001,"5min":1.4750000000000001,"15min":1.518},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.23000000000000001},"max":{"1min":7.9489999999999998,"5min":7.9489999999999998,"15min":36.901000000000003},"last":1.583},{"interface":"front","average":{"1min":1.8100000000000001,"5min":1.002,"15min":1.3400000000000001},"min":{"1min":0.33400000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":7.0540000000000003,"5min":7.0540000000000003,"15min":37.984000000000002},"last":0.39400000000000002}]},{"osd":3,"last update":"Mon Feb 5 06:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":1.464,"15min":1.8460000000000001},"min":{"1min":0.317,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":8.0030000000000001,"5min":22.105,"15min":62.508000000000003},"last":0.47799999999999998},{"interface":"front","average":{"1min":1.6739999999999999,"5min":1.387,"15min":1.9630000000000001},"min":{"1min":0.30299999999999999,"5min":0.28100000000000003,"15min":0.27400000000000002},"max":{"1min":4.9379999999999997,"5min":22.032,"15min":62.460000000000001},"last":0.88200000000000001}]},{"osd":4,"last update":"Mon Feb 5 06:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":0.90800000000000003,"15min":0.72199999999999998},"min":{"1min":0.315,"5min":0.27100000000000002,"15min":0.249},"max":{"1min":9.3629999999999995,"5min":9.3629999999999995,"15min":9.3629999999999995},"last":0.376},{"interface":"front","average":{"1min":1.427,"5min":1.01,"15min":0.76000000000000001},"min":{"1min":0.29199999999999998,"5min":0.25600000000000001,"15min":0.252},"max":{"1min":9.1839999999999993,"5min":9.1839999999999993,"15min":9.1839999999999993},"last":0.50800000000000001}]},{"osd":6,"last update":"Mon Feb 5 06:50:29 2024","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.1599999999999999,"15min":0.84299999999999997},"min":{"1min":0.316,"5min":0.27700000000000002,"15min":0.23799999999999999},"max":{"1min":7.0179999999999998,"5min":7.7430000000000003,"15min":7.7430000000000003},"last":0.53100000000000003},{"interface":"front","average":{"1min":1.6950000000000001,"5min":1.274,"15min":0.89300000000000002},"min":{"1min":0.39600000000000002,"5min":0.35499999999999998,"15min":0.27600000000000002},"max":{"1min":10.85,"5min":10.85,"15min":10.85},"last":0.51900000000000002}]},{"osd":7,"last update":"Mon Feb 5 06:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.2190000000000001,"15min":0.89600000000000002},"min":{"1min":0.33600000000000002,"5min":0.31900000000000001,"15min":0.25600000000000001},"max":{"1min":11.691000000000001,"5min":11.691000000000001,"15min":11.691000000000001},"last":0.432},{"interface":"front","average":{"1min":1.381,"5min":1.038,"15min":0.873},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.27100000000000002},"max":{"1min":4.9939999999999998,"5min":4.9939999999999998,"15min":6.5839999999999996},"last":0.35699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282787840,"data_stored":282787840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":531169280,"data_stored":531169280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":384434176,"data_stored":384434176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":487849984,"data_stored":487849984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":228524032,"data_stored":228524032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":236126208,"data_stored":236126208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":491454464,"data_stored":491454464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":354222080,"data_stored":354222080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T06:51:36.496 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-02-05T06:51:36.496 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd pool set .mgr pg_num 21 2024-02-05T06:51:36.524 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:36.559 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:36.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:36.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:36.664 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:36.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:36.718 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:36.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:36.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:36 smithi167 ceph-mon[32774]: 5.1c deep-scrub ok 2024-02-05T06:51:36.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:36.865 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:36.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:36.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:36.967 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:37.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:37.104 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:37.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:37.193 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:51:37.205 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:37.208 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:51:37.212 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:51:37.234 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:51:37.277 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:37.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:37.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:37.283 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:51:37.299 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:51:37.328 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:51:37.338 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 16 16 35638 35622 139.12 102.75 0.0111261 0.00718117 2024-02-05T06:51:37.365 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:51:37.379 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:37.408 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:51:37.430 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:37.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:37.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:37.449 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:51:37.489 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:51:37.494 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:51:37.494 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:51:37.494 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:51:37.494 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:51:37.494 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:51:37.494 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:51:37.494 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:51:37.495 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:51:37.532 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:37.561 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:37 smithi028 ceph-mon[37984]: 1.7 scrub starts 2024-02-05T06:51:37.561 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:37 smithi028 ceph-mon[37984]: 1.7 scrub ok 2024-02-05T06:51:37.562 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:37 smithi028 ceph-mon[37984]: 5.12 scrub starts 2024-02-05T06:51:37.562 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:37 smithi028 ceph-mon[37984]: 5.12 scrub ok 2024-02-05T06:51:37.562 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:37 smithi028 ceph-mon[37984]: pgmap v868: 158 pgs: 158 active+clean; 1.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 146 MiB/s wr, 2.33k op/s 2024-02-05T06:51:37.562 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:37 smithi028 ceph-mon[37984]: 3.11 scrub starts 2024-02-05T06:51:37.562 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:37 smithi028 ceph-mon[37984]: 3.11 scrub ok 2024-02-05T06:51:37.562 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:37 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1289039524' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:51:37.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:37 smithi028 ceph-mon[29442]: 1.7 scrub starts 2024-02-05T06:51:37.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:37 smithi028 ceph-mon[29442]: 1.7 scrub ok 2024-02-05T06:51:37.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:37 smithi028 ceph-mon[29442]: 5.12 scrub starts 2024-02-05T06:51:37.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:37 smithi028 ceph-mon[29442]: 5.12 scrub ok 2024-02-05T06:51:37.562 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:37 smithi028 ceph-mon[29442]: pgmap v868: 158 pgs: 158 active+clean; 1.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 146 MiB/s wr, 2.33k op/s 2024-02-05T06:51:37.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:37 smithi028 ceph-mon[29442]: 3.11 scrub starts 2024-02-05T06:51:37.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:37 smithi028 ceph-mon[29442]: 3.11 scrub ok 2024-02-05T06:51:37.563 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:37 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1289039524' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:51:37.568 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:37.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:37.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:37.670 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:37.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:37.702 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:37.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:37.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:37 smithi167 ceph-mon[32774]: 1.7 scrub starts 2024-02-05T06:51:37.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:37 smithi167 ceph-mon[32774]: 1.7 scrub ok 2024-02-05T06:51:37.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:37 smithi167 ceph-mon[32774]: 5.12 scrub starts 2024-02-05T06:51:37.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:37 smithi167 ceph-mon[32774]: 5.12 scrub ok 2024-02-05T06:51:37.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:37 smithi167 ceph-mon[32774]: pgmap v868: 158 pgs: 158 active+clean; 1.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 146 MiB/s wr, 2.33k op/s 2024-02-05T06:51:37.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:37 smithi167 ceph-mon[32774]: 3.11 scrub starts 2024-02-05T06:51:37.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:37 smithi167 ceph-mon[32774]: 3.11 scrub ok 2024-02-05T06:51:37.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:37 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1289039524' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:51:37.803 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:37.837 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:37.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:37.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:37.939 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:37.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:37.984 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:37.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:38.085 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:38.119 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:38.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:38.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:38.222 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:38.264 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:38.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:38.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:38.339 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 17 16 37445 37429 137.579 112.938 0.00491218 0.00726252 2024-02-05T06:51:38.366 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:38.399 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:38.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:38.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:38.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:38 smithi028 ceph-mon[37984]: 3.10 scrub starts 2024-02-05T06:51:38.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:38 smithi028 ceph-mon[37984]: 3.10 scrub ok 2024-02-05T06:51:38.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:38 smithi028 ceph-mon[29442]: 3.10 scrub starts 2024-02-05T06:51:38.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:38 smithi028 ceph-mon[29442]: 3.10 scrub ok 2024-02-05T06:51:38.501 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:38.538 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:38.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:38.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:38.639 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:38.681 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:38.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:38.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:38.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:38 smithi167 ceph-mon[32774]: 3.10 scrub starts 2024-02-05T06:51:38.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:38 smithi167 ceph-mon[32774]: 3.10 scrub ok 2024-02-05T06:51:38.783 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:38.816 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:38.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:38.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:38.918 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:38.953 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:38.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:38.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:39.054 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:39.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:39.093 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:39.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:39.194 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:39.243 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:39.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:39.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:39.339 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 18 15 39339 39324 136.514 118.438 0.00829322 0.00731902 2024-02-05T06:51:39.345 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:39.379 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:39.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:39.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:39.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:39 smithi028 ceph-mon[29442]: 6.4 scrub starts 2024-02-05T06:51:39.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:39 smithi028 ceph-mon[29442]: 6.4 scrub ok 2024-02-05T06:51:39.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:39 smithi028 ceph-mon[29442]: pgmap v869: 158 pgs: 158 active+clean; 2.2 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 940 B/s rd, 132 MiB/s wr, 2.12k op/s 2024-02-05T06:51:39.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:39 smithi028 ceph-mon[37984]: 6.4 scrub starts 2024-02-05T06:51:39.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:39 smithi028 ceph-mon[37984]: 6.4 scrub ok 2024-02-05T06:51:39.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:39 smithi028 ceph-mon[37984]: pgmap v869: 158 pgs: 158 active+clean; 2.2 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 940 B/s rd, 132 MiB/s wr, 2.12k op/s 2024-02-05T06:51:39.481 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:39.514 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:39.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:39.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:39.616 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:39.650 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:39.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:39.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:39.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:39 smithi167 ceph-mon[32774]: 6.4 scrub starts 2024-02-05T06:51:39.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:39 smithi167 ceph-mon[32774]: 6.4 scrub ok 2024-02-05T06:51:39.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:39 smithi167 ceph-mon[32774]: pgmap v869: 158 pgs: 158 active+clean; 2.2 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 940 B/s rd, 132 MiB/s wr, 2.12k op/s 2024-02-05T06:51:39.752 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:39.782 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:39.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:39.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:39.884 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:39.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:39.928 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:39.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:40.029 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:40.060 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:40.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:40.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:40.162 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:40.194 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:40.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:40.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:40.296 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:40.339 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 19 16 41113 41097 135.161 110.812 0.00524492 0.00739387 2024-02-05T06:51:40.363 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:40.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:40.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:40.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:40 smithi028 ceph-mon[37984]: 3.a scrub starts 2024-02-05T06:51:40.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:40 smithi028 ceph-mon[37984]: 3.a scrub ok 2024-02-05T06:51:40.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:40 smithi028 ceph-mon[37984]: 5.1d deep-scrub starts 2024-02-05T06:51:40.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:40 smithi028 ceph-mon[37984]: 5.1d deep-scrub ok 2024-02-05T06:51:40.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:40 smithi028 ceph-mon[29442]: 3.a scrub starts 2024-02-05T06:51:40.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:40 smithi028 ceph-mon[29442]: 3.a scrub ok 2024-02-05T06:51:40.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:40 smithi028 ceph-mon[29442]: 5.1d deep-scrub starts 2024-02-05T06:51:40.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:40 smithi028 ceph-mon[29442]: 5.1d deep-scrub ok 2024-02-05T06:51:40.465 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:40.505 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:40.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:40.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:40.606 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:40.640 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:40.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:40.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:40.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:40 smithi167 ceph-mon[32774]: 3.a scrub starts 2024-02-05T06:51:40.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:40 smithi167 ceph-mon[32774]: 3.a scrub ok 2024-02-05T06:51:40.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:40 smithi167 ceph-mon[32774]: 5.1d deep-scrub starts 2024-02-05T06:51:40.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:40 smithi167 ceph-mon[32774]: 5.1d deep-scrub ok 2024-02-05T06:51:40.743 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:40.770 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:40.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:40.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:40.872 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:40.914 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:40.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:40.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:41.015 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:41.057 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:41.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:41.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:41.159 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:41.198 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:41.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:41.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:41.300 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:41.333 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:41.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:41.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:41.339 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-02-05T06:51:41.331268+0000 min lat: 0.0012181 max lat: 0.138627 avg lat: 0.00750554 2024-02-05T06:51:41.339 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T06:51:41.339 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 20 16 42626 42610 133.131 94.5625 0.0083986 0.00750554 2024-02-05T06:51:41.435 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:41.468 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:41.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:41.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:41.570 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:41.601 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:41.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:41.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:41.636 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:41 smithi167 ceph-mon[32774]: 4.1c scrub starts 2024-02-05T06:51:41.636 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:41 smithi167 ceph-mon[32774]: 4.1c scrub ok 2024-02-05T06:51:41.636 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:41 smithi167 ceph-mon[32774]: 4.2 deep-scrub starts 2024-02-05T06:51:41.637 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:41 smithi167 ceph-mon[32774]: 4.2 deep-scrub ok 2024-02-05T06:51:41.637 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:41 smithi167 ceph-mon[32774]: 3.1f scrub starts 2024-02-05T06:51:41.637 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:41 smithi167 ceph-mon[32774]: 3.1f scrub ok 2024-02-05T06:51:41.637 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:41 smithi167 ceph-mon[32774]: 4.0 deep-scrub starts 2024-02-05T06:51:41.637 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:41 smithi167 ceph-mon[32774]: 4.0 deep-scrub ok 2024-02-05T06:51:41.637 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:41 smithi167 ceph-mon[32774]: pgmap v870: 158 pgs: 158 active+clean; 2.2 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-02-05T06:51:41.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:41 smithi028 ceph-mon[37984]: 4.1c scrub starts 2024-02-05T06:51:41.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:41 smithi028 ceph-mon[37984]: 4.1c scrub ok 2024-02-05T06:51:41.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:41 smithi028 ceph-mon[37984]: 4.2 deep-scrub starts 2024-02-05T06:51:41.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:41 smithi028 ceph-mon[37984]: 4.2 deep-scrub ok 2024-02-05T06:51:41.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:41 smithi028 ceph-mon[37984]: 3.1f scrub starts 2024-02-05T06:51:41.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:41 smithi028 ceph-mon[37984]: 3.1f scrub ok 2024-02-05T06:51:41.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:41 smithi028 ceph-mon[37984]: 4.0 deep-scrub starts 2024-02-05T06:51:41.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:41 smithi028 ceph-mon[37984]: 4.0 deep-scrub ok 2024-02-05T06:51:41.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:41 smithi028 ceph-mon[37984]: pgmap v870: 158 pgs: 158 active+clean; 2.2 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-02-05T06:51:41.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:41 smithi028 ceph-mon[29442]: 4.1c scrub starts 2024-02-05T06:51:41.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:41 smithi028 ceph-mon[29442]: 4.1c scrub ok 2024-02-05T06:51:41.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:41 smithi028 ceph-mon[29442]: 4.2 deep-scrub starts 2024-02-05T06:51:41.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:41 smithi028 ceph-mon[29442]: 4.2 deep-scrub ok 2024-02-05T06:51:41.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:41 smithi028 ceph-mon[29442]: 3.1f scrub starts 2024-02-05T06:51:41.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:41 smithi028 ceph-mon[29442]: 3.1f scrub ok 2024-02-05T06:51:41.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:41 smithi028 ceph-mon[29442]: 4.0 deep-scrub starts 2024-02-05T06:51:41.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:41 smithi028 ceph-mon[29442]: 4.0 deep-scrub ok 2024-02-05T06:51:41.703 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:41 smithi028 ceph-mon[29442]: pgmap v870: 158 pgs: 158 active+clean; 2.2 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-02-05T06:51:41.703 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:41.736 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:41.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:41.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:41.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:41.876 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:41.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:41.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:41.978 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:42.020 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:42.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:42.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:42.122 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:42.154 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:42.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:42.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:42.255 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:42.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:42.294 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:42.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:42.339 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 21 15 44092 44077 131.157 91.6875 0.00525157 0.00761896 2024-02-05T06:51:42.395 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:42.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:42.432 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:42.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:42.533 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:42.570 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:42.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:42.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:42.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:42.696 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:42.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:42.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:42.701 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:51:42 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:51:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:51:42.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:42 smithi028 ceph-mon[29442]: 6.7 scrub starts 2024-02-05T06:51:42.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:42 smithi028 ceph-mon[29442]: 3.0 scrub starts 2024-02-05T06:51:42.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:42 smithi028 ceph-mon[29442]: 3.0 scrub ok 2024-02-05T06:51:42.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:42 smithi028 ceph-mon[29442]: 6.7 scrub ok 2024-02-05T06:51:42.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:42 smithi028 ceph-mon[29442]: 1.0 scrub starts 2024-02-05T06:51:42.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:42 smithi028 ceph-mon[29442]: 1.0 scrub ok 2024-02-05T06:51:42.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:42 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:51:42.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:42 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:51:42.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:42 smithi028 ceph-mon[37984]: 6.7 scrub starts 2024-02-05T06:51:42.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:42 smithi028 ceph-mon[37984]: 3.0 scrub starts 2024-02-05T06:51:42.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:42 smithi028 ceph-mon[37984]: 3.0 scrub ok 2024-02-05T06:51:42.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:42 smithi028 ceph-mon[37984]: 6.7 scrub ok 2024-02-05T06:51:42.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:42 smithi028 ceph-mon[37984]: 1.0 scrub starts 2024-02-05T06:51:42.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:42 smithi028 ceph-mon[37984]: 1.0 scrub ok 2024-02-05T06:51:42.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:42 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:51:42.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:42 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:51:42.708 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:42 smithi167 ceph-mon[32774]: 6.7 scrub starts 2024-02-05T06:51:42.708 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:42 smithi167 ceph-mon[32774]: 3.0 scrub starts 2024-02-05T06:51:42.708 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:42 smithi167 ceph-mon[32774]: 3.0 scrub ok 2024-02-05T06:51:42.708 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:42 smithi167 ceph-mon[32774]: 6.7 scrub ok 2024-02-05T06:51:42.708 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:42 smithi167 ceph-mon[32774]: 1.0 scrub starts 2024-02-05T06:51:42.708 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:42 smithi167 ceph-mon[32774]: 1.0 scrub ok 2024-02-05T06:51:42.708 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:42 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:51:42.708 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:42 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:51:42.798 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:42.837 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:42.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:42.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:42.939 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:42.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:42.983 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:42.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:43.084 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:43.116 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:43.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:43.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:43.218 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:43.251 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:43.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:43.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:43.339 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 22 16 45627 45611 129.553 95.875 0.00624995 0.00771176 2024-02-05T06:51:43.353 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:43.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:43.399 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:43.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:43.500 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:43.537 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:43.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:43.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:43.638 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:43.675 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:43.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:43.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:43.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:43 smithi028 ceph-mon[37984]: 6.f deep-scrub starts 2024-02-05T06:51:43.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:43 smithi028 ceph-mon[37984]: 6.f deep-scrub ok 2024-02-05T06:51:43.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:43 smithi028 ceph-mon[37984]: 3.9 deep-scrub starts 2024-02-05T06:51:43.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:43 smithi028 ceph-mon[37984]: 3.9 deep-scrub ok 2024-02-05T06:51:43.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:43 smithi028 ceph-mon[37984]: 4.b scrub starts 2024-02-05T06:51:43.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:43 smithi028 ceph-mon[37984]: 4.b scrub ok 2024-02-05T06:51:43.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:43 smithi028 ceph-mon[37984]: pgmap v871: 158 pgs: 158 active+clean; 2.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 137 MiB/s wr, 2.19k op/s 2024-02-05T06:51:43.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:43 smithi028 ceph-mon[37984]: 6.3 scrub starts 2024-02-05T06:51:43.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:43 smithi028 ceph-mon[37984]: 6.3 scrub ok 2024-02-05T06:51:43.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:43 smithi028 ceph-mon[29442]: 6.f deep-scrub starts 2024-02-05T06:51:43.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:43 smithi028 ceph-mon[29442]: 6.f deep-scrub ok 2024-02-05T06:51:43.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:43 smithi028 ceph-mon[29442]: 3.9 deep-scrub starts 2024-02-05T06:51:43.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:43 smithi028 ceph-mon[29442]: 3.9 deep-scrub ok 2024-02-05T06:51:43.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:43 smithi028 ceph-mon[29442]: 4.b scrub starts 2024-02-05T06:51:43.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:43 smithi028 ceph-mon[29442]: 4.b scrub ok 2024-02-05T06:51:43.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:43 smithi028 ceph-mon[29442]: pgmap v871: 158 pgs: 158 active+clean; 2.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 137 MiB/s wr, 2.19k op/s 2024-02-05T06:51:43.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:43 smithi028 ceph-mon[29442]: 6.3 scrub starts 2024-02-05T06:51:43.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:43 smithi028 ceph-mon[29442]: 6.3 scrub ok 2024-02-05T06:51:43.736 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:51:43 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:51:43.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:43 smithi167 ceph-mon[32774]: 6.f deep-scrub starts 2024-02-05T06:51:43.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:43 smithi167 ceph-mon[32774]: 6.f deep-scrub ok 2024-02-05T06:51:43.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:43 smithi167 ceph-mon[32774]: 3.9 deep-scrub starts 2024-02-05T06:51:43.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:43 smithi167 ceph-mon[32774]: 3.9 deep-scrub ok 2024-02-05T06:51:43.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:43 smithi167 ceph-mon[32774]: 4.b scrub starts 2024-02-05T06:51:43.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:43 smithi167 ceph-mon[32774]: 4.b scrub ok 2024-02-05T06:51:43.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:43 smithi167 ceph-mon[32774]: pgmap v871: 158 pgs: 158 active+clean; 2.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 137 MiB/s wr, 2.19k op/s 2024-02-05T06:51:43.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:43 smithi167 ceph-mon[32774]: 6.3 scrub starts 2024-02-05T06:51:43.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:43 smithi167 ceph-mon[32774]: 6.3 scrub ok 2024-02-05T06:51:43.777 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:43.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:43.808 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:43.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:43.909 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:43.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:43.998 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:43.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:44.099 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:44.137 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:44.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:44.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:44.239 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:44.272 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:44.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:44.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:44.340 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 23 16 47111 47095 127.953 92.75 0.00720637 0.00780999 2024-02-05T06:51:44.374 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-02-05T06:51:44.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:44.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:44.519 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:44.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:44.556 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:44.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:44.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:44.682 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:44 smithi028 ceph-mon[37984]: 1.8 scrub starts 2024-02-05T06:51:44.682 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:44 smithi028 ceph-mon[37984]: 1.8 scrub ok 2024-02-05T06:51:44.682 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:44 smithi028 ceph-mon[37984]: 6.0 scrub starts 2024-02-05T06:51:44.682 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:44 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:51:44.682 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:44 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/859366538' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:51:44.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:44 smithi028 ceph-mon[29442]: 1.8 scrub starts 2024-02-05T06:51:44.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:44 smithi028 ceph-mon[29442]: 1.8 scrub ok 2024-02-05T06:51:44.682 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:44 smithi028 ceph-mon[29442]: 6.0 scrub starts 2024-02-05T06:51:44.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:44 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:51:44.683 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:44 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/859366538' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:51:44.696 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:44.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:44.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:44.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:44 smithi167 ceph-mon[32774]: 1.8 scrub starts 2024-02-05T06:51:44.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:44 smithi167 ceph-mon[32774]: 1.8 scrub ok 2024-02-05T06:51:44.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:44 smithi167 ceph-mon[32774]: 6.0 scrub starts 2024-02-05T06:51:44.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:44 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:51:44.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:44 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/859366538' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:51:44.798 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:44.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:44.832 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:44.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:44.933 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:44.966 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:44.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:44.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:45.068 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:45.105 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:45.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:45.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:45.207 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:45.248 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:45.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:45.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:45.339 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 24 15 48612 48597 126.532 93.875 0.00681489 0.0078981 2024-02-05T06:51:45.350 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:45.381 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:45.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:45.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:45.483 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:45.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:45.517 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:45.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:45.618 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:45.648 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:45.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:45.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:45.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:45 smithi028 ceph-mon[37984]: 6.0 scrub ok 2024-02-05T06:51:45.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:45 smithi028 ceph-mon[37984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:51:45.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:45 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/859366538' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:51:45.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:45 smithi028 ceph-mon[37984]: osdmap e425: 8 total, 8 up, 8 in 2024-02-05T06:51:45.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:45 smithi028 ceph-mon[37984]: pgmap v873: 158 pgs: 158 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 110 MiB/s wr, 1.77k op/s 2024-02-05T06:51:45.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:45 smithi028 ceph-mon[29442]: 6.0 scrub ok 2024-02-05T06:51:45.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:45 smithi028 ceph-mon[29442]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:51:45.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:45 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/859366538' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:51:45.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:45 smithi028 ceph-mon[29442]: osdmap e425: 8 total, 8 up, 8 in 2024-02-05T06:51:45.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:45 smithi028 ceph-mon[29442]: pgmap v873: 158 pgs: 158 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 110 MiB/s wr, 1.77k op/s 2024-02-05T06:51:45.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:45 smithi167 ceph-mon[32774]: 6.0 scrub ok 2024-02-05T06:51:45.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:45 smithi167 ceph-mon[32774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:51:45.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:45 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/859366538' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:51:45.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:45 smithi167 ceph-mon[32774]: osdmap e425: 8 total, 8 up, 8 in 2024-02-05T06:51:45.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:45 smithi167 ceph-mon[32774]: pgmap v873: 158 pgs: 158 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 110 MiB/s wr, 1.77k op/s 2024-02-05T06:51:45.750 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:45.786 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:45.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:45.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:45.888 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:45.929 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:45.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:45.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:46.031 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:46.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:51:46.067 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:46.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:46.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:46.169 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:46.199 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:46.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:46.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:46.303 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:46.333 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:46.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:46.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:46.340 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 25 16 50096 50080 125.178 92.6875 0.016786 0.00798332 2024-02-05T06:51:46.435 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:46.464 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:46.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:46.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:46.565 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:46.609 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:46.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:46.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:46.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:46.754 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:46.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:46.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:46.856 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:46.885 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:46.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:46.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:46.987 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:47.023 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:47.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:47.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:47.060 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:51:47.060 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-02-05T06:51:47.060 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:51:47.060 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.45878.0:50676 11.9 e426/418 11:95846b65:::benchmark_data_smithi028_177697_object50675:head v 426'3174, mlcod=426'3171)", 2024-02-05T06:51:47.060 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:51:46.724616+0000", 2024-02-05T06:51:47.060 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0012017, 2024-02-05T06:51:47.060 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0012344470000000001, 2024-02-05T06:51:47.060 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:51:47.061 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-02-05T06:51:47.061 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:51:47.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:51:47.061 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:51:47.061 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:46.724616+0000", 2024-02-05T06:51:47.061 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:51:47.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:51:47.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:51:47.061 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:51:47.061 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:46.724616+0000", 2024-02-05T06:51:47.061 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:51:47.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:51:47.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:51:47.061 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:51:47.061 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:46.724617+0000", 2024-02-05T06:51:47.062 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2130000000000001e-06 2024-02-05T06:51:47.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:51:47.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:51:47.062 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:51:47.062 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:46.724642+0000", 2024-02-05T06:51:47.062 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4649000000000001e-05 2024-02-05T06:51:47.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:51:47.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:51:47.062 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:51:47.062 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:46.724657+0000", 2024-02-05T06:51:47.062 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4976e-05 2024-02-05T06:51:47.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:51:47.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:51:47.062 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:51:47.062 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:46.724665+0000", 2024-02-05T06:51:47.063 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4829999999999999e-06 2024-02-05T06:51:47.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:51:47.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:51:47.063 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:51:47.063 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:46.724734+0000", 2024-02-05T06:51:47.063 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8906999999999996e-05 2024-02-05T06:51:47.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:51:47.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:51:47.063 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:51:47.063 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:46.724752+0000", 2024-02-05T06:51:47.063 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8286000000000001e-05 2024-02-05T06:51:47.063 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:51:47.063 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:51:47.063 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:51:47.063 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:51:47.063 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:51:47.064 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 1 2024-02-05T06:51:47.064 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:51:47.125 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:47.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:47.167 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:47.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:47.268 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:47.300 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:47.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:47.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:47.340 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 26 16 51532 51516 123.815 89.75 0.0110705 0.00807145 2024-02-05T06:51:47.402 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:47.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:47.441 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:47.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:47.542 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:47.578 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:47.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:47.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:47.680 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:47.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:47 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:51:47.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:47 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:51:47.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:47 smithi028 ceph-mon[37984]: osdmap e426: 8 total, 8 up, 8 in 2024-02-05T06:51:47.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:47 smithi028 ceph-mon[37984]: pgmap v875: 158 pgs: 158 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-02-05T06:51:47.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:47 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:51:47.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:47 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:51:47.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:47 smithi028 ceph-mon[29442]: osdmap e426: 8 total, 8 up, 8 in 2024-02-05T06:51:47.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:47 smithi028 ceph-mon[29442]: pgmap v875: 158 pgs: 158 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-02-05T06:51:47.713 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:47.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:47.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:47.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:47 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:51:47.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:47 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:51:47.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:47 smithi167 ceph-mon[32774]: osdmap e426: 8 total, 8 up, 8 in 2024-02-05T06:51:47.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:47 smithi167 ceph-mon[32774]: pgmap v875: 158 pgs: 158 active+clean; 2.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-02-05T06:51:47.816 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:47.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:47.849 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:47.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:47.950 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:47.990 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:47.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:47.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:48.092 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:48.128 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:48.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:48.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:48.232 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:48.284 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:48.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:48.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:48.340 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 27 16 53110 53094 122.882 98.625 0.00220988 0.00812946 2024-02-05T06:51:48.385 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:48.413 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:48.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:48.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:48.515 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:48.538 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:48.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:48.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:48.643 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:48.683 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:48.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:48.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:48.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:48.823 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:48.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:48.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:48.925 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:48.955 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:48.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:48.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:49.057 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:49.099 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:49.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:49.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:49.201 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:49.240 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:49.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:49.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:49.340 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 28 16 54355 54339 121.272 77.8125 0.00722709 0.0082403 2024-02-05T06:51:49.342 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:49.377 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:49.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:49.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:49.479 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:49.519 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:49.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:49.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:49.620 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:49.652 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:49.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:49.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:49.754 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:49.785 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:49.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:49.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:49.886 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:49.915 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:49.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:49.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:49.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:49 smithi028 ceph-mon[29442]: pgmap v876: 158 pgs: 158 active+clean; 3.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.98k op/s 2024-02-05T06:51:49.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:49 smithi028 ceph-mon[37984]: pgmap v876: 158 pgs: 158 active+clean; 3.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.98k op/s 2024-02-05T06:51:49.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:49 smithi167 ceph-mon[32774]: pgmap v876: 158 pgs: 158 active+clean; 3.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 124 MiB/s wr, 1.98k op/s 2024-02-05T06:51:50.017 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:50.020 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-02-05T06:51:50.046 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:50.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:50.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:50.054 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:51:50.148 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:50.176 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:50.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:50.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:50.278 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:50.304 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:50.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:50.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:50.340 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 29 16 55587 55571 119.746 77 0.013154 0.00834213 2024-02-05T06:51:50.406 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:50.410 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T06:51:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:50.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:50.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:50.541 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:50.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:50.576 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:50.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:50.677 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:50.704 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:50.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:50.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:50.806 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:50.842 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:50.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:50.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:50.944 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:50.990 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:50.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:50.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:51.092 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:51.123 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:51.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:51.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:51.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:51.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:51.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:51.298 INFO:teuthology.orchestra.run.smithi028.stderr:set pool 1 pg_num to 21 2024-02-05T06:51:51.340 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 30 16 56839 56823 118.362 78.25 0.00523758 0.0084422 2024-02-05T06:51:51.366 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:51.409 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:51.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:51.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:51.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:51 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/4137075720' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-02-05T06:51:51.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:51 smithi028 ceph-mon[29442]: from='client.25885 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-02-05T06:51:51.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:51 smithi028 ceph-mon[29442]: pgmap v877: 158 pgs: 158 active+clean; 3.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-02-05T06:51:51.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:51 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/4137075720' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-02-05T06:51:51.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:51 smithi028 ceph-mon[37984]: from='client.25885 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-02-05T06:51:51.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:51 smithi028 ceph-mon[37984]: pgmap v877: 158 pgs: 158 active+clean; 3.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-02-05T06:51:51.511 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:51.554 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:51.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:51.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:51.656 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:51.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:51.690 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:51.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:51.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:51 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/4137075720' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-02-05T06:51:51.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:51 smithi167 ceph-mon[32774]: from='client.25885 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-02-05T06:51:51.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:51 smithi167 ceph-mon[32774]: pgmap v877: 158 pgs: 158 active+clean; 3.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 74 MiB/s wr, 1.19k op/s 2024-02-05T06:51:51.791 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:51.819 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:51.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:51.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:51.921 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:51.955 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:51.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:51.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:52.056 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:52.096 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:52.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:52.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:52.198 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:52.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:52.243 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:52.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:52.340 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 31 16 58264 58248 117.417 89.0625 0.0151594 0.00851068 2024-02-05T06:51:52.343 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:52.376 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:52.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:52.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:52.478 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:52.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:52.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:52.619 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:52.652 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:52.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:52.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:52.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:52 smithi028 ceph-mon[37984]: from='client.25885 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-02-05T06:51:52.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:52 smithi028 ceph-mon[37984]: osdmap e427: 8 total, 8 up, 8 in 2024-02-05T06:51:52.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:52 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:51:52.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:52 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:51:52.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:52 smithi028 ceph-mon[29442]: from='client.25885 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-02-05T06:51:52.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:52 smithi028 ceph-mon[29442]: osdmap e427: 8 total, 8 up, 8 in 2024-02-05T06:51:52.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:52 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:51:52.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:52 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:51:52.702 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:51:52 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:51:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:51:52.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:52 smithi167 ceph-mon[32774]: from='client.25885 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-02-05T06:51:52.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:52 smithi167 ceph-mon[32774]: osdmap e427: 8 total, 8 up, 8 in 2024-02-05T06:51:52.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:52 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:51:52.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:52 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:51:52.754 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:52.794 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:52.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:52.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:52.895 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:52.928 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:52.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:52.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:52.942 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:51:53.003 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:51:53.029 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:53.059 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:53.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:53.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:53.161 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:53.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:53.209 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:53.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:53.310 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:53.340 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 32 16 59623 59607 116.402 84.9375 0.0129569 0.00858411 2024-02-05T06:51:53.343 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:53.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:53.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:53.445 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:53.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:53 smithi028 ceph-mon[37984]: osdmap e428: 8 total, 8 up, 8 in 2024-02-05T06:51:53.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:53 smithi028 ceph-mon[37984]: pgmap v880: 158 pgs: 158 active+clean; 3.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.6 KiB/s rd, 127 MiB/s wr, 2.03k op/s 2024-02-05T06:51:53.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:53 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-05T06:51:53.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:53 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-05T06:51:53.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:53 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:51:53.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:53 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:51:53.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:53 smithi028 ceph-mon[29442]: osdmap e428: 8 total, 8 up, 8 in 2024-02-05T06:51:53.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:53 smithi028 ceph-mon[29442]: pgmap v880: 158 pgs: 158 active+clean; 3.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.6 KiB/s rd, 127 MiB/s wr, 2.03k op/s 2024-02-05T06:51:53.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:53 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-05T06:51:53.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:53 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-05T06:51:53.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:53 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:51:53.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:53 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:51:53.480 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:53.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:53.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:53.582 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:53.619 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:53.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:53.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:53.721 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:53.735 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:51:53 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:51:53.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:53 smithi167 ceph-mon[32774]: osdmap e428: 8 total, 8 up, 8 in 2024-02-05T06:51:53.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:53 smithi167 ceph-mon[32774]: pgmap v880: 158 pgs: 158 active+clean; 3.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.6 KiB/s rd, 127 MiB/s wr, 2.03k op/s 2024-02-05T06:51:53.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:53 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-05T06:51:53.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:53 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-02-05T06:51:53.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:53 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:51:53.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:53 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:51:53.753 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:53.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:53.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:53.855 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:53.886 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:53.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:53.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:53.988 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:54.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:54.021 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:54.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:54.122 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:54.153 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:54.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:54.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:54.255 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:54.293 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:54.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:54.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:54.340 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 33 16 61016 61000 115.512 87.0625 0.00387624 0.00864972 2024-02-05T06:51:54.396 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:54.432 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:54.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:54.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:54.534 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:54.571 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:54.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:54.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:54.580 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:54 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-05T06:51:54.580 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:54 smithi028 ceph-mon[37984]: osdmap e429: 8 total, 8 up, 8 in 2024-02-05T06:51:54.580 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:54 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:51:54.580 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:54 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-05T06:51:54.580 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:54 smithi028 ceph-mon[29442]: osdmap e429: 8 total, 8 up, 8 in 2024-02-05T06:51:54.581 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:54 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:51:54.673 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:54.710 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:54.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:54.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:54.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:54 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-02-05T06:51:54.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:54 smithi167 ceph-mon[32774]: osdmap e429: 8 total, 8 up, 8 in 2024-02-05T06:51:54.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:54 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:51:54.812 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:54.844 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:54.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:54.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:54.946 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:54.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:54.998 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:54.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:55.100 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:55.142 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:55.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:55.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:55.243 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:55.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:55.284 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:55.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:55.340 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 34 16 62344 62328 114.556 83 0.0103162 0.00872244 2024-02-05T06:51:55.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:55 smithi028 ceph-mon[37984]: osdmap e430: 8 total, 8 up, 8 in 2024-02-05T06:51:55.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:55 smithi028 ceph-mon[37984]: pgmap v883: 168 pgs: 10 unknown, 158 active+clean; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.69k op/s 2024-02-05T06:51:55.384 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:55 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-05T06:51:55.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:55 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-05T06:51:55.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:55 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-02-05T06:51:55.385 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:55 smithi028 ceph-mon[37984]: osdmap e431: 8 total, 8 up, 8 in 2024-02-05T06:51:55.385 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:55 smithi028 ceph-mon[29442]: osdmap e430: 8 total, 8 up, 8 in 2024-02-05T06:51:55.385 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:55 smithi028 ceph-mon[29442]: pgmap v883: 168 pgs: 10 unknown, 158 active+clean; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.69k op/s 2024-02-05T06:51:55.385 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:55 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-05T06:51:55.385 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:55 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-05T06:51:55.385 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:55 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-02-05T06:51:55.385 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:55 smithi028 ceph-mon[29442]: osdmap e431: 8 total, 8 up, 8 in 2024-02-05T06:51:55.385 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:55.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:55.416 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:55.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:55.485 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:55 smithi167 ceph-mon[32774]: osdmap e430: 8 total, 8 up, 8 in 2024-02-05T06:51:55.486 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:55 smithi167 ceph-mon[32774]: pgmap v883: 168 pgs: 10 unknown, 158 active+clean; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.69k op/s 2024-02-05T06:51:55.486 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:55 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-05T06:51:55.486 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:55 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-02-05T06:51:55.486 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:55 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-02-05T06:51:55.486 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:55 smithi167 ceph-mon[32774]: osdmap e431: 8 total, 8 up, 8 in 2024-02-05T06:51:55.517 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:55.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:55.553 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:55.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:55.654 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:55.700 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:55.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:55.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:55.803 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:55.830 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:55.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:55.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:55.932 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:55.964 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:55.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:55.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:56.066 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:56.094 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:56.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:56.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:56.196 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:56.229 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:56.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:56.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:56.248 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:51:56.248 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:51:56.249 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:51:56.249 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:51:56.249 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:51:56.249 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:51:56.249 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:51:56.249 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:51:56.249 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:51:56.249 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:51:56.249 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:51:56.249 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:51:56.250 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:51:56.250 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:51:56.250 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:51:56.250 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:51:56.285 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-02-05T06:51:56.331 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:56.343 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 35 16 63535 63519 113.409 74.4375 0.0113066 0.00881201 2024-02-05T06:51:56.364 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:56.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:56.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:56.466 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:56.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:56.507 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:56.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:56.608 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:56.643 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:56.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:56.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:56.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:56 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3123961883' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:51:56.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:56 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1534936827' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:51:56.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:56 smithi028 ceph-mon[37984]: from='client.25894 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:51:56.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:56 smithi028 ceph-mon[37984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:51:56.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:56 smithi028 ceph-mon[37984]: from='client.25894 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:51:56.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:56 smithi028 ceph-mon[37984]: osdmap e432: 8 total, 8 up, 8 in 2024-02-05T06:51:56.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:56 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3123961883' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:51:56.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:56 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1534936827' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:51:56.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:56 smithi028 ceph-mon[29442]: from='client.25894 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:51:56.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:56 smithi028 ceph-mon[29442]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:51:56.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:56 smithi028 ceph-mon[29442]: from='client.25894 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:51:56.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:56 smithi028 ceph-mon[29442]: osdmap e432: 8 total, 8 up, 8 in 2024-02-05T06:51:56.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:56 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3123961883' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:51:56.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:56 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1534936827' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:51:56.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:56 smithi167 ceph-mon[32774]: from='client.25894 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:51:56.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:56 smithi167 ceph-mon[32774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:51:56.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:56 smithi167 ceph-mon[32774]: from='client.25894 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:51:56.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:56 smithi167 ceph-mon[32774]: osdmap e432: 8 total, 8 up, 8 in 2024-02-05T06:51:56.745 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:56.782 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:56.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:56.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:56.884 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:56.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:56.923 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:56.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:57.024 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:57.058 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:57.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:57.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:57.160 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:57.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:57.204 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:57.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:57.305 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:57.340 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 36 16 64911 64895 112.648 86 0.0105691 0.00887241 2024-02-05T06:51:57.341 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:57.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:57.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:57.370 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 4, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T06:51:57.370 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd reweight 0 1 2024-02-05T06:51:57.443 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:57.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:57.494 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:57.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:57.596 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:57.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:57.636 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:57.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:57.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:57 smithi028 ceph-mon[29442]: pgmap v886: 168 pgs: 4 remapped+peering, 6 unknown, 158 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 7.0 KiB/s rd, 33 MiB/s wr, 531 op/s; 39 KiB/s, 0 objects/s recovering 2024-02-05T06:51:57.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:57 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:51:57.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:57 smithi028 ceph-mon[29442]: osdmap e433: 8 total, 8 up, 8 in 2024-02-05T06:51:57.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:57 smithi028 ceph-mon[37984]: pgmap v886: 168 pgs: 4 remapped+peering, 6 unknown, 158 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 7.0 KiB/s rd, 33 MiB/s wr, 531 op/s; 39 KiB/s, 0 objects/s recovering 2024-02-05T06:51:57.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:57 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:51:57.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:57 smithi028 ceph-mon[37984]: osdmap e433: 8 total, 8 up, 8 in 2024-02-05T06:51:57.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:57 smithi167 ceph-mon[32774]: pgmap v886: 168 pgs: 4 remapped+peering, 6 unknown, 158 active+clean; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 7.0 KiB/s rd, 33 MiB/s wr, 531 op/s; 39 KiB/s, 0 objects/s recovering 2024-02-05T06:51:57.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:57 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:51:57.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:57 smithi167 ceph-mon[32774]: osdmap e433: 8 total, 8 up, 8 in 2024-02-05T06:51:57.737 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:57.776 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:57.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:57.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:57.877 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:57.908 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:57.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:57.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:58.012 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:51:58.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:58.057 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:58.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:51:58.158 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:58.195 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:58.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:58.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:58.297 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:51:58.328 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:58.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:58.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:51:58.340 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 37 16 66063 66047 111.549 72 0.0109072 0.00895951 2024-02-05T06:51:58.430 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:51:58.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:58 smithi028 ceph-mon[29442]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-02-05T06:51:58.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:58 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:51:58.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:58 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:51:58.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:58 smithi028 ceph-mon[29442]: osdmap e434: 8 total, 8 up, 8 in 2024-02-05T06:51:58.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:58 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:51:58.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:58 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:51:58.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:58 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:51:58.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:58 smithi028 ceph-mon[37984]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-02-05T06:51:58.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:58 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:51:58.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:58 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:51:58.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:58 smithi028 ceph-mon[37984]: osdmap e434: 8 total, 8 up, 8 in 2024-02-05T06:51:58.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:58 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:51:58.453 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:58 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:51:58.453 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:58 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:51:58.471 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:58.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:58.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:51:58.573 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:58.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:58.611 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:58.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:58.713 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:58.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:58 smithi167 ceph-mon[32774]: Health check failed: Reduced data availability: 4 pgs peering (PG_AVAILABILITY) 2024-02-05T06:51:58.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:58 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:51:58.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:58 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:51:58.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:58 smithi167 ceph-mon[32774]: osdmap e434: 8 total, 8 up, 8 in 2024-02-05T06:51:58.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:58 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:51:58.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:58 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:51:58.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:58 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:51:58.746 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:58.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:58.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:58.848 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:51:58.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:58.887 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:58.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:51:58.988 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:59.022 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:59.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:59.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:59.093 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:51:59.124 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:51:59.156 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:59.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:59.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:51:59.258 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:59.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:59.290 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:59.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:59.341 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 38 16 67329 67313 110.696 79.125 0.0141154 0.00902794 2024-02-05T06:51:59.392 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:59.424 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:59.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:59.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:59.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:59 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:51:59.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:59 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:51:59.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:59 smithi028 ceph-mon[29442]: pgmap v889: 168 pgs: 4 active+undersized+remapped, 6 remapped+peering, 158 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 8.2 KiB/s rd, 103 MiB/s wr, 1.66k op/s; 119 KiB/s, 0 objects/s recovering 2024-02-05T06:51:59.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:51:59 smithi028 ceph-mon[29442]: osdmap e435: 8 total, 8 up, 8 in 2024-02-05T06:51:59.447 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:59 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:51:59.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:59 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:51:59.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:59 smithi028 ceph-mon[37984]: pgmap v889: 168 pgs: 4 active+undersized+remapped, 6 remapped+peering, 158 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 8.2 KiB/s rd, 103 MiB/s wr, 1.66k op/s; 119 KiB/s, 0 objects/s recovering 2024-02-05T06:51:59.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:51:59 smithi028 ceph-mon[37984]: osdmap e435: 8 total, 8 up, 8 in 2024-02-05T06:51:59.455 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:51:59.526 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:51:59.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:59.583 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:51:59.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:51:59.592 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:51:59.592 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:51:59.685 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:51:59.716 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:59.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:59.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:51:59.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:59 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:51:59.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:59 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:51:59.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:59 smithi167 ceph-mon[32774]: pgmap v889: 168 pgs: 4 active+undersized+remapped, 6 remapped+peering, 158 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 8.2 KiB/s rd, 103 MiB/s wr, 1.66k op/s; 119 KiB/s, 0 objects/s recovering 2024-02-05T06:51:59.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:51:59 smithi167 ceph-mon[32774]: osdmap e435: 8 total, 8 up, 8 in 2024-02-05T06:51:59.818 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:59.858 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:59.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:59.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:51:59.960 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:51:59.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:51:59.995 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:51:59.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:00.096 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:00.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:00.134 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:00.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:00.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:00.276 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:00.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:00.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:00.341 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 39 16 68592 68576 109.881 78.9375 0.00998428 0.00909561 2024-02-05T06:52:00.377 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:00.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:00.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:00.518 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:00.558 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:00.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:00.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:00.560 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-02-05T06:52:00.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:00.703 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:00.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:00.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:00.807 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:00.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:00.845 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:00.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:00.882 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T06:52:00.928 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:52:00.945 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:00.975 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:00.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:00.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:01.077 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:01.120 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:01.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:01.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:01.222 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:01.260 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:01.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:01.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:01.341 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-02-05T06:52:01.333275+0000 min lat: 0.0012181 max lat: 0.204312 avg lat: 0.00917147 2024-02-05T06:52:01.341 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T06:52:01.341 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 40 16 69771 69755 108.976 73.6875 0.00775993 0.00917147 2024-02-05T06:52:01.361 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:01.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:01.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:01.504 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:01.543 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:01.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:01.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:01.645 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:01.685 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:01.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:01.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:01.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:01 smithi028 ceph-mon[29442]: pgmap v891: 168 pgs: 4 active+undersized+remapped, 6 remapped+peering, 158 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 7.7 KiB/s rd, 96 MiB/s wr, 1.55k op/s; 111 KiB/s, 0 objects/s recovering 2024-02-05T06:52:01.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:01 smithi028 ceph-mon[37984]: pgmap v891: 168 pgs: 4 active+undersized+remapped, 6 remapped+peering, 158 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 7.7 KiB/s rd, 96 MiB/s wr, 1.55k op/s; 111 KiB/s, 0 objects/s recovering 2024-02-05T06:52:01.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:01 smithi167 ceph-mon[32774]: pgmap v891: 168 pgs: 4 active+undersized+remapped, 6 remapped+peering, 158 active+clean; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 7.7 KiB/s rd, 96 MiB/s wr, 1.55k op/s; 111 KiB/s, 0 objects/s recovering 2024-02-05T06:52:01.787 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:01.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:01.820 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:01.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:01.921 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:01.956 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:01.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:01.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:02.058 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:02.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:02.102 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:02.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:02.203 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:02.242 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:02.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:02.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:02.341 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 41 16 71110 71094 108.359 83.6875 0.0224056 0.00922262 2024-02-05T06:52:02.343 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:02.373 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:02.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:02.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:02.475 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:02.519 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:02.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:02.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:02.621 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:02.663 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:02.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:02.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:02.667 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:52:02 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:52:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:52:02.767 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:02.798 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:02.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:02.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:02.900 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:02.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:02.933 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:02.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:02.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:02 smithi028 ceph-mon[29442]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2024-02-05T06:52:02.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:02 smithi028 ceph-mon[37984]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2024-02-05T06:52:02.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:02 smithi167 ceph-mon[32774]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs peering) 2024-02-05T06:52:03.034 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:03.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:03.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:03.163 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:03.199 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:03.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:03.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:03.301 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:03.337 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:03.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:03.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:03.341 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 42 16 72282 72266 107.523 73.25 0.0145825 0.00929544 2024-02-05T06:52:03.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:03.479 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:03.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:03.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:03.581 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:03.623 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:03.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:03.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:03.725 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:03.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:03 smithi167 ceph-mon[32774]: pgmap v892: 168 pgs: 1 active+undersized+remapped, 167 active+clean; 4.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 113 MiB/s wr, 1.81k op/s; 53 KiB/s, 0 objects/s recovering 2024-02-05T06:52:03.735 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:52:03 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:52:03.756 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:03.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:03.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:03.858 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:03.896 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:03.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:03.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:03.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:03 smithi028 ceph-mon[37984]: pgmap v892: 168 pgs: 1 active+undersized+remapped, 167 active+clean; 4.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 113 MiB/s wr, 1.81k op/s; 53 KiB/s, 0 objects/s recovering 2024-02-05T06:52:03.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:03 smithi028 ceph-mon[29442]: pgmap v892: 168 pgs: 1 active+undersized+remapped, 167 active+clean; 4.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 113 MiB/s wr, 1.81k op/s; 53 KiB/s, 0 objects/s recovering 2024-02-05T06:52:03.998 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:04.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:04.035 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:04.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:04.136 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:04.171 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:04.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:04.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:04.273 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:04.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:04.312 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:04.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:04.341 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 43 16 73535 73519 106.844 78.3125 0.00472514 0.00935392 2024-02-05T06:52:04.413 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:04.445 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:04.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:04.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:04.547 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:04.582 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:04.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:04.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:04.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:04.708 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:04.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:04.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:04.810 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:04.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:04.843 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:04.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:04.944 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:04.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:04 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:52:04.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:04 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:52:04.968 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:04.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:04.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:04.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:04 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:52:05.071 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:05.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:05.110 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:05.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:05.211 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:05.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:05.236 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:05.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:05.337 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:05.341 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 44 16 74565 74549 105.878 64.375 0.0137293 0.00943952 2024-02-05T06:52:05.376 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:05.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:05.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:05.477 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:05.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:05.510 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:05.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:05.611 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:05.643 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:05.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:05.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:05.745 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:05.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:05.784 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:05.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:05.886 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:05.915 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:05.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:05.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:05.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:05 smithi028 ceph-mon[29442]: pgmap v893: 168 pgs: 168 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 702 B/s rd, 78 MiB/s wr, 1.24k op/s; 44 KiB/s, 0 objects/s recovering 2024-02-05T06:52:05.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:05 smithi028 ceph-mon[37984]: pgmap v893: 168 pgs: 168 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 702 B/s rd, 78 MiB/s wr, 1.24k op/s; 44 KiB/s, 0 objects/s recovering 2024-02-05T06:52:05.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:05 smithi167 ceph-mon[32774]: pgmap v893: 168 pgs: 168 active+clean; 4.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 702 B/s rd, 78 MiB/s wr, 1.24k op/s; 44 KiB/s, 0 objects/s recovering 2024-02-05T06:52:06.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:06.048 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:06.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:06.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:06.149 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:06.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:06.181 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:06.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:06.282 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:06.311 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:06.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:06.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:06.341 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 45 16 75598 75582 104.96 64.5625 0.0137433 0.00952261 2024-02-05T06:52:06.414 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:06.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:06.452 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:06.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:06.553 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:06.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:06.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:06.696 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:06.701 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.0 to 1 (10000) 2024-02-05T06:52:06.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:06.745 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:06.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:06.781 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:52:06.846 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:06.885 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:06.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:06.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:06.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:06 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1712922633' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T06:52:06.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:06 smithi028 ceph-mon[29442]: from='client.25900 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T06:52:06.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:06 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1712922633' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T06:52:06.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:06 smithi028 ceph-mon[37984]: from='client.25900 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T06:52:06.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:06 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1712922633' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T06:52:06.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:06 smithi167 ceph-mon[32774]: from='client.25900 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T06:52:06.987 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:07.018 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:07.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:07.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:07.120 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:07.155 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:07.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:07.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:07.256 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:07.296 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:07.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:07.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:07.341 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 46 16 76813 76797 104.329 75.9375 0.00367079 0.00957934 2024-02-05T06:52:07.398 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:07.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:07.435 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:07.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:07.536 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:07.577 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:07.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:07.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:07.679 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:07.715 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:07.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:07.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:07.817 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:07.859 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:07.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:07.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:07.865 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:07 smithi028 ceph-mon[29442]: pgmap v894: 168 pgs: 168 active+clean; 4.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 618 B/s rd, 50 MiB/s wr, 796 op/s 2024-02-05T06:52:07.865 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:07 smithi028 ceph-mon[29442]: from='client.25900 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T06:52:07.865 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:07 smithi028 ceph-mon[29442]: osdmap e436: 8 total, 8 up, 8 in 2024-02-05T06:52:07.865 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:07 smithi028 ceph-mon[37984]: pgmap v894: 168 pgs: 168 active+clean; 4.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 618 B/s rd, 50 MiB/s wr, 796 op/s 2024-02-05T06:52:07.865 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:07 smithi028 ceph-mon[37984]: from='client.25900 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T06:52:07.866 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:07 smithi028 ceph-mon[37984]: osdmap e436: 8 total, 8 up, 8 in 2024-02-05T06:52:07.910 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:52:07.917 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd reweight 1 1 2024-02-05T06:52:07.961 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:07.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:07 smithi167 ceph-mon[32774]: pgmap v894: 168 pgs: 168 active+clean; 4.4 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 618 B/s rd, 50 MiB/s wr, 796 op/s 2024-02-05T06:52:07.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:07 smithi167 ceph-mon[32774]: from='client.25900 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T06:52:07.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:07 smithi167 ceph-mon[32774]: osdmap e436: 8 total, 8 up, 8 in 2024-02-05T06:52:07.990 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:07.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:07.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:08.092 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:08.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:08.121 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:08.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:08.222 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:08.296 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:08.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:08.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:08.343 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 47 16 78029 78013 103.723 76 0.0175307 0.00963576 2024-02-05T06:52:08.397 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:08.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:08.429 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:08.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:08.530 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:08.567 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:08.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:08.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:08.669 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:08.717 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:08.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:08.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:08.819 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:08.852 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:08.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:08.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:08.954 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:08.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:08.994 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:08.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:09.095 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:09.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:09.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:09.234 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:09.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:09.267 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:09.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:09.343 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 48 16 79310 79294 103.23 80.0625 0.0168121 0.00968135 2024-02-05T06:52:09.368 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:09.396 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:09.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:09.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:09.498 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:09.527 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:09.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:09.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:09.629 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:09.663 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:09.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:09.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:09.765 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:09.800 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:09.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:09.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:09.902 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:09.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:09.936 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:09.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:09.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:09 smithi028 ceph-mon[37984]: pgmap v896: 168 pgs: 168 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-02-05T06:52:09.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:09 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/4000013501' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:52:09.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:09 smithi028 ceph-mon[29442]: pgmap v896: 168 pgs: 168 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-02-05T06:52:09.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:09 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/4000013501' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:52:09.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:09 smithi167 ceph-mon[32774]: pgmap v896: 168 pgs: 168 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-02-05T06:52:09.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:09 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/4000013501' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:52:10.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:10.080 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:10.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:10.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:10.182 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:10.207 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:52:10.207 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:52:10.207 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:52:10.207 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:52:10.207 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:52:10.207 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:52:10.207 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:52:10.207 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:52:10.208 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:52:10.208 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:52:10.208 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:52:10.208 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:52:10.208 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:52:10.208 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:52:10.208 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:52:10.208 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:52:10.213 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:10.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:10.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:10.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:10.343 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 49 16 80445 80429 102.57 70.9375 0.0121161 0.00974407 2024-02-05T06:52:10.363 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:10.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:10.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:10.465 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:10.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:10.506 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:10.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:10.607 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:10.641 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:10.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:10.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:10.743 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:10.774 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:10.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:10.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:10.870 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:10 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:52:10.871 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:10 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:52:10.871 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:10 smithi028 ceph-mon[37984]: osdmap e437: 8 total, 8 up, 8 in 2024-02-05T06:52:10.871 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:10 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:52:10.871 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:10 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:52:10.871 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:10 smithi028 ceph-mon[29442]: osdmap e437: 8 total, 8 up, 8 in 2024-02-05T06:52:10.875 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:10.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:10.904 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:10.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:10.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:10 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:52:10.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:10 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:52:10.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:10 smithi167 ceph-mon[32774]: osdmap e437: 8 total, 8 up, 8 in 2024-02-05T06:52:11.005 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:11.043 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:11.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:11.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:11.144 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:11.186 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:11.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:11.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:11.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:11.330 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:11.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:11.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:11.343 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 50 16 81689 81673 102.074 77.75 0.00977169 0.00979045 2024-02-05T06:52:11.433 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:11.478 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:11.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:11.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:11.580 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:11.612 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:11.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:11.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:11.714 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:11.746 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:11.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:11.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:11.848 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:11.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:11.894 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:11.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:11.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:11 smithi167 ceph-mon[32774]: pgmap v898: 168 pgs: 168 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 911 op/s 2024-02-05T06:52:11.995 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:12.038 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:12.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:12.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:12.141 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:12.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:12.180 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:12.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:12.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:11 smithi028 ceph-mon[37984]: pgmap v898: 168 pgs: 168 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 911 op/s 2024-02-05T06:52:12.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:11 smithi028 ceph-mon[29442]: pgmap v898: 168 pgs: 168 active+clean; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 911 op/s 2024-02-05T06:52:12.281 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:12.312 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:12.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:12.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:12.343 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 51 16 82797 82781 101.43 69.25 0.0124324 0.00985458 2024-02-05T06:52:12.414 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:12.452 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:12.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:12.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:12.554 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:12.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:12.589 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:12.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:12.690 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:12.723 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:12.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:12.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:12.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:12.863 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:12.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:12.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:12.951 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:52:12 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:52:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:52:12.965 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:13.002 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:13.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:13.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:13.104 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:13.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:13.141 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:13.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:13.242 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:13.272 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:13.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:13.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:13.343 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 52 16 84080 84064 101.022 80.1875 0.00493594 0.00989333 2024-02-05T06:52:13.374 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:13.413 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:13.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:13.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:13.515 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:13.555 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:13.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:13.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:13.657 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:13.687 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:13.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:13.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:13.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:13 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:52:13.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:13 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:52:13.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:13 smithi028 ceph-mon[29442]: pgmap v899: 168 pgs: 168 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-02-05T06:52:13.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:13 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:52:13.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:13 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:52:13.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:13 smithi028 ceph-mon[37984]: pgmap v899: 168 pgs: 168 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-02-05T06:52:13.735 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:52:13 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:52:13.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:13 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:52:13.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:13 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:52:13.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:13 smithi167 ceph-mon[32774]: pgmap v899: 168 pgs: 168 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-02-05T06:52:13.790 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:13.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:13.824 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:13.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:13.925 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:13.955 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:13.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:13.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:14.056 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:14.090 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:14.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:14.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:14.191 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:14.242 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:14.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:14.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:14.343 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:14.344 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 53 16 85304 85288 100.559 76.5 0.0184468 0.00993924 2024-02-05T06:52:14.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:14.392 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:14.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:14.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:14 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:52:14.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:14 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:52:14.493 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:14.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:14.530 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:14.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:14.631 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:14.669 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:14.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:14.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:14.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:14 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:52:14.770 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:14.800 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:14.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:14.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:14.899 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:52:14.902 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:14.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:14.963 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:14.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:15.064 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:15.092 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:15.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:15.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:15.194 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:15.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:15.266 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:15.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:15.343 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 54 16 86454 86438 100.028 71.875 0.0149521 0.00999214 2024-02-05T06:52:15.368 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:15.404 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:15.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:15.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:15.506 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:15.550 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:15.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:15.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:15.584 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:15 smithi028 ceph-mon[29442]: pgmap v900: 168 pgs: 168 active+clean; 5.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-02-05T06:52:15.584 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:15 smithi028 ceph-mon[37984]: pgmap v900: 168 pgs: 168 active+clean; 5.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-02-05T06:52:15.652 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:15.690 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:15.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:15.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:15.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:15 smithi167 ceph-mon[32774]: pgmap v900: 168 pgs: 168 active+clean; 5.0 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-02-05T06:52:15.791 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:15.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:15.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:15.938 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:15.980 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:15.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:15.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:16.082 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:16.120 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:16.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:16.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:16.222 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:16.250 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:52:16.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:16.263 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:16.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:16.344 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 55 16 87606 87590 99.5167 72 0.00785616 0.010043 2024-02-05T06:52:16.365 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:16.392 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:16.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:16.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:16.494 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:16.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:16.562 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:16.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:16.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:16.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:16.711 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:16.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:16.714 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:52:16.714 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-02-05T06:52:16.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-02-05T06:52:16.715 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-02-05T06:52:16.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.715 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45587.0:103404 10.18 10:1859de67:::benchmark_data_smithi028_154709_object103403:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-02-05T06:52:16.715 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:48:51.238167+0000", 2024-02-05T06:52:16.715 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 205.46461140400001, 2024-02-05T06:52:16.715 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0233144270000001, 2024-02-05T06:52:16.715 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:16.715 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:16.715 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:52:16.715 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45587", 2024-02-05T06:52:16.715 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:52:16.715 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 103404 2024-02-05T06:52:16.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.715 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:16.715 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.716 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:16.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.238167+0000", 2024-02-05T06:52:16.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.716 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:16.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.238167+0000", 2024-02-05T06:52:16.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.716 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.716 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:16.716 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.238171+0000", 2024-02-05T06:52:16.716 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5140000000000002e-06 2024-02-05T06:52:16.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:16.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.238205+0000", 2024-02-05T06:52:16.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4638000000000003e-05 2024-02-05T06:52:16.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:16.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.238224+0000", 2024-02-05T06:52:16.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8281e-05 2024-02-05T06:52:16.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.717 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.717 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:16.717 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.238236+0000", 2024-02-05T06:52:16.717 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2145e-05 2024-02-05T06:52:16.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.718 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:16.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.465383+0000", 2024-02-05T06:52:16.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.22714728300000001 2024-02-05T06:52:16.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.718 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:52:16.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.465394+0000", 2024-02-05T06:52:16.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.095e-05 2024-02-05T06:52:16.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.718 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:16.718 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:52.252859+0000", 2024-02-05T06:52:16.718 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.78746483899999997 2024-02-05T06:52:16.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.718 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:16.719 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:52.253004+0000", 2024-02-05T06:52:16.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014537400000000001 2024-02-05T06:52:16.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,7", 2024-02-05T06:52:16.719 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:52.253310+0000", 2024-02-05T06:52:16.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0003056 2024-02-05T06:52:16.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.719 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:52:16.719 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:52.254727+0000", 2024-02-05T06:52:16.719 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0014170809999999999 2024-02-05T06:52:16.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.719 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.720 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.720 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:52.255917+0000", 2024-02-05T06:52:16.720 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0011899739999999999 2024-02-05T06:52:16.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.720 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.720 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:52.261386+0000", 2024-02-05T06:52:16.720 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0054691009999999997 2024-02-05T06:52:16.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.720 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.720 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:16.720 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:52.261444+0000", 2024-02-05T06:52:16.720 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8294999999999997e-05 2024-02-05T06:52:16.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.721 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:16.721 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:52.261482+0000", 2024-02-05T06:52:16.721 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7351999999999997e-05 2024-02-05T06:52:16.721 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.721 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:16.721 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.721 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.721 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45587.0:103406 10.4 10:265740a8:::benchmark_data_smithi028_154709_object103405:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-02-05T06:52:16.721 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:48:51.246525+0000", 2024-02-05T06:52:16.721 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 205.456253121, 2024-02-05T06:52:16.721 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.013782808, 2024-02-05T06:52:16.721 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:16.722 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:16.722 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:52:16.722 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45587", 2024-02-05T06:52:16.722 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:52:16.722 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 103406 2024-02-05T06:52:16.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.722 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:16.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:16.722 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.246525+0000", 2024-02-05T06:52:16.722 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.722 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.722 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:16.723 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.246525+0000", 2024-02-05T06:52:16.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.723 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:16.723 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.246528+0000", 2024-02-05T06:52:16.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5840000000000002e-06 2024-02-05T06:52:16.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.723 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:16.723 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.246561+0000", 2024-02-05T06:52:16.723 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.2866e-05 2024-02-05T06:52:16.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.723 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.723 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:16.723 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.246580+0000", 2024-02-05T06:52:16.724 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8980000000000001e-05 2024-02-05T06:52:16.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.724 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:16.724 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.246593+0000", 2024-02-05T06:52:16.724 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2668e-05 2024-02-05T06:52:16.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.724 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:16.724 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.486572+0000", 2024-02-05T06:52:16.724 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.23997907199999999 2024-02-05T06:52:16.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.724 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.724 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:52:16.724 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:51.486587+0000", 2024-02-05T06:52:16.725 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5880999999999999e-05 2024-02-05T06:52:16.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.725 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:16.725 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:52.250119+0000", 2024-02-05T06:52:16.725 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.76353134700000003 2024-02-05T06:52:16.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.725 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:16.725 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:52.250266+0000", 2024-02-05T06:52:16.725 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014745399999999999 2024-02-05T06:52:16.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.726 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4", 2024-02-05T06:52:16.726 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:52.250606+0000", 2024-02-05T06:52:16.726 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00034009399999999999 2024-02-05T06:52:16.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.726 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:52:16.726 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:52.254555+0000", 2024-02-05T06:52:16.726 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0039482240000000002 2024-02-05T06:52:16.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.726 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.726 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:52.254937+0000", 2024-02-05T06:52:16.726 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00038206900000000002 2024-02-05T06:52:16.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.726 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.726 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.726 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:52.260195+0000", 2024-02-05T06:52:16.726 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0052587420000000003 2024-02-05T06:52:16.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.727 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:16.727 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:52.260266+0000", 2024-02-05T06:52:16.727 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0658e-05 2024-02-05T06:52:16.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.727 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:16.727 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:52.260308+0000", 2024-02-05T06:52:16.727 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2169000000000002e-05 2024-02-05T06:52:16.727 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.727 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:16.727 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.727 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.728 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45587.0:104360 10.11 10:8f50f9be:::benchmark_data_smithi028_154709_object104359:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e375)", 2024-02-05T06:52:16.728 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:48:54.217921+0000", 2024-02-05T06:52:16.728 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 202.48485708300001, 2024-02-05T06:52:16.728 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0760841249999999, 2024-02-05T06:52:16.728 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:16.728 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "reached pg", 2024-02-05T06:52:16.728 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:52:16.728 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45587", 2024-02-05T06:52:16.728 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:52:16.728 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 104360 2024-02-05T06:52:16.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.728 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:16.728 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.728 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:16.728 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.217921+0000", 2024-02-05T06:52:16.729 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.729 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:16.729 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.217921+0000", 2024-02-05T06:52:16.729 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.729 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:16.729 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.217924+0000", 2024-02-05T06:52:16.729 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1090000000000001e-06 2024-02-05T06:52:16.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.729 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.729 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:16.730 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.217949+0000", 2024-02-05T06:52:16.730 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4966000000000001e-05 2024-02-05T06:52:16.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.730 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:16.730 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.217960+0000", 2024-02-05T06:52:16.730 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1623999999999999e-05 2024-02-05T06:52:16.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.730 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:16.730 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.217969+0000", 2024-02-05T06:52:16.730 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2399999999999996e-06 2024-02-05T06:52:16.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.730 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.730 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:16.730 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.219885+0000", 2024-02-05T06:52:16.731 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001915569 2024-02-05T06:52:16.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.731 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:52:16.731 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.219895+0000", 2024-02-05T06:52:16.731 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0234999999999999e-05 2024-02-05T06:52:16.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.731 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:16.731 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.293911+0000", 2024-02-05T06:52:16.731 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0740159250000001 2024-02-05T06:52:16.731 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.731 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.731 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:16.731 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.294006+0000", 2024-02-05T06:52:16.732 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.4456999999999994e-05 2024-02-05T06:52:16.732 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.732 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:16.732 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.732 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.732 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45587.0:104367 10.11 10:899ed155:::benchmark_data_smithi028_154709_object104366:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e375)", 2024-02-05T06:52:16.732 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:48:54.219789+0000", 2024-02-05T06:52:16.732 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 202.482989592, 2024-02-05T06:52:16.732 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.12406389, 2024-02-05T06:52:16.732 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:16.732 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:16.732 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:52:16.733 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45587", 2024-02-05T06:52:16.733 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:52:16.733 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 104367 2024-02-05T06:52:16.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.733 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:16.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.733 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:16.733 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.219789+0000", 2024-02-05T06:52:16.733 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.733 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.733 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:16.733 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.219789+0000", 2024-02-05T06:52:16.733 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.734 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:16.734 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.219790+0000", 2024-02-05T06:52:16.734 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.342e-06 2024-02-05T06:52:16.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.734 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:16.734 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.219828+0000", 2024-02-05T06:52:16.734 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7248e-05 2024-02-05T06:52:16.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.734 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:16.734 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.219836+0000", 2024-02-05T06:52:16.734 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8030000000000003e-06 2024-02-05T06:52:16.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.734 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.734 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:16.735 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.219845+0000", 2024-02-05T06:52:16.735 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0659999999999993e-06 2024-02-05T06:52:16.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.735 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.735 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:16.735 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220011+0000", 2024-02-05T06:52:16.735 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000165957 2024-02-05T06:52:16.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.735 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.735 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:52:16.735 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220020+0000", 2024-02-05T06:52:16.735 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7790000000000001e-06 2024-02-05T06:52:16.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.735 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.735 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:16.735 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.294256+0000", 2024-02-05T06:52:16.735 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.074235396 2024-02-05T06:52:16.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.736 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:16.736 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.294596+0000", 2024-02-05T06:52:16.736 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00034046599999999999 2024-02-05T06:52:16.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.736 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 4,5,7", 2024-02-05T06:52:16.736 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.294967+0000", 2024-02-05T06:52:16.736 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00037079399999999997 2024-02-05T06:52:16.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.736 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.736 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:52:16.736 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.298237+0000", 2024-02-05T06:52:16.736 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0032700810000000002 2024-02-05T06:52:16.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.737 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.737 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.301079+0000", 2024-02-05T06:52:16.737 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028423760000000002 2024-02-05T06:52:16.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.737 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.737 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.303489+0000", 2024-02-05T06:52:16.737 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024097630000000001 2024-02-05T06:52:16.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.737 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.737 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.737 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.343669+0000", 2024-02-05T06:52:16.737 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.040179497000000002 2024-02-05T06:52:16.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.738 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.738 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:16.738 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.343736+0000", 2024-02-05T06:52:16.738 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7384000000000004e-05 2024-02-05T06:52:16.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.738 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.738 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:16.738 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.343853+0000", 2024-02-05T06:52:16.738 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000116938 2024-02-05T06:52:16.738 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.738 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:16.738 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.738 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.738 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.739 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45587.0:104369 10.12 10:494edbbb:::benchmark_data_smithi028_154709_object104368:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e375)", 2024-02-05T06:52:16.739 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:48:54.219947+0000", 2024-02-05T06:52:16.739 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 202.482831506, 2024-02-05T06:52:16.739 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1380725229999999, 2024-02-05T06:52:16.739 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:16.739 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:16.739 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:52:16.739 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45587", 2024-02-05T06:52:16.740 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:52:16.740 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 104369 2024-02-05T06:52:16.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.740 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:16.740 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.740 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:16.740 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.219947+0000", 2024-02-05T06:52:16.740 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.740 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.741 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:16.741 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.219947+0000", 2024-02-05T06:52:16.741 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.741 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.741 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:16.741 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.219948+0000", 2024-02-05T06:52:16.741 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7599999999999996e-07 2024-02-05T06:52:16.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.741 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.741 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:16.741 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.219972+0000", 2024-02-05T06:52:16.741 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4145999999999999e-05 2024-02-05T06:52:16.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.741 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.742 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:16.742 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.219981+0000", 2024-02-05T06:52:16.742 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2690000000000005e-06 2024-02-05T06:52:16.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.742 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.742 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:16.742 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.219990+0000", 2024-02-05T06:52:16.742 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7730000000000005e-06 2024-02-05T06:52:16.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.742 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.742 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:16.742 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220044+0000", 2024-02-05T06:52:16.742 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.4156000000000001e-05 2024-02-05T06:52:16.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.743 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:52:16.743 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220053+0000", 2024-02-05T06:52:16.743 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7809999999999994e-06 2024-02-05T06:52:16.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.743 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:16.743 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.336780+0000", 2024-02-05T06:52:16.743 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.116727231 2024-02-05T06:52:16.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.743 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:16.743 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.336941+0000", 2024-02-05T06:52:16.744 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016117799999999999 2024-02-05T06:52:16.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.744 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.744 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2,4,5,6", 2024-02-05T06:52:16.744 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.337573+0000", 2024-02-05T06:52:16.744 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00063131099999999996 2024-02-05T06:52:16.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.744 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.744 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:52:16.744 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.340232+0000", 2024-02-05T06:52:16.744 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002658728 2024-02-05T06:52:16.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.744 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.744 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.744 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.340282+0000", 2024-02-05T06:52:16.745 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0696999999999999e-05 2024-02-05T06:52:16.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.745 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.745 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.745 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.341117+0000", 2024-02-05T06:52:16.745 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000834537 2024-02-05T06:52:16.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.745 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.745 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.745 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.341161+0000", 2024-02-05T06:52:16.745 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4085000000000002e-05 2024-02-05T06:52:16.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.746 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.746 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.746 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.345798+0000", 2024-02-05T06:52:16.746 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0046371870000000001 2024-02-05T06:52:16.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.746 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.746 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.746 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.357804+0000", 2024-02-05T06:52:16.746 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.012005775999999999 2024-02-05T06:52:16.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.746 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.746 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:16.746 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.357873+0000", 2024-02-05T06:52:16.746 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8750000000000004e-05 2024-02-05T06:52:16.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.747 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.747 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:16.747 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.358020+0000", 2024-02-05T06:52:16.747 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014704200000000001 2024-02-05T06:52:16.747 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.747 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:16.747 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.747 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.747 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45587.0:104370 10.14 10:2d8044f1:::benchmark_data_smithi028_154709_object104369:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e375)", 2024-02-05T06:52:16.747 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:48:54.220010+0000", 2024-02-05T06:52:16.747 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 202.48276885300001, 2024-02-05T06:52:16.747 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.118127831, 2024-02-05T06:52:16.747 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:16.747 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:16.748 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:52:16.748 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45587", 2024-02-05T06:52:16.748 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:52:16.748 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 104370 2024-02-05T06:52:16.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.748 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:16.748 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.748 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:16.748 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220010+0000", 2024-02-05T06:52:16.748 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.748 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.748 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:16.748 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220010+0000", 2024-02-05T06:52:16.748 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.749 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:16.749 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220010+0000", 2024-02-05T06:52:16.749 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6700000000000003e-07 2024-02-05T06:52:16.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.749 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:16.749 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220035+0000", 2024-02-05T06:52:16.749 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5015999999999999e-05 2024-02-05T06:52:16.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.749 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.749 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:16.749 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220055+0000", 2024-02-05T06:52:16.750 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.912e-05 2024-02-05T06:52:16.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.750 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.750 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:16.750 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220063+0000", 2024-02-05T06:52:16.750 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7870000000000007e-06 2024-02-05T06:52:16.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.750 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.750 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:16.750 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220109+0000", 2024-02-05T06:52:16.750 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5257999999999999e-05 2024-02-05T06:52:16.750 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.750 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.750 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:52:16.750 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220116+0000", 2024-02-05T06:52:16.750 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6260000000000003e-06 2024-02-05T06:52:16.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.751 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.751 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:16.751 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.331376+0000", 2024-02-05T06:52:16.751 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.111259958 2024-02-05T06:52:16.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.751 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.751 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:16.751 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.331544+0000", 2024-02-05T06:52:16.751 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016788600000000001 2024-02-05T06:52:16.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.751 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.751 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2,3,4", 2024-02-05T06:52:16.751 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.331851+0000", 2024-02-05T06:52:16.751 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00030651000000000002 2024-02-05T06:52:16.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.752 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.752 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.752 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.333786+0000", 2024-02-05T06:52:16.752 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001935876 2024-02-05T06:52:16.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.752 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.752 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:52:16.752 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.333815+0000", 2024-02-05T06:52:16.752 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.8135000000000001e-05 2024-02-05T06:52:16.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.752 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.752 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.752 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.334950+0000", 2024-02-05T06:52:16.752 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0011358690000000001 2024-02-05T06:52:16.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.753 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.753 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.753 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.337811+0000", 2024-02-05T06:52:16.753 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0028608819999999999 2024-02-05T06:52:16.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.753 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.753 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.753 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.337935+0000", 2024-02-05T06:52:16.753 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012391999999999999 2024-02-05T06:52:16.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.753 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.753 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:16.753 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.338004+0000", 2024-02-05T06:52:16.753 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8287999999999994e-05 2024-02-05T06:52:16.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.754 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.754 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:16.754 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.338138+0000", 2024-02-05T06:52:16.754 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013403300000000001 2024-02-05T06:52:16.754 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.754 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:16.754 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.754 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.754 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45587.0:104380 10.16 10:6f5a1ea4:::benchmark_data_smithi028_154709_object104379:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e375)", 2024-02-05T06:52:16.754 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:48:54.220080+0000", 2024-02-05T06:52:16.754 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 202.482698175, 2024-02-05T06:52:16.754 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.153608239, 2024-02-05T06:52:16.754 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:16.754 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:16.755 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:52:16.755 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45587", 2024-02-05T06:52:16.755 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:52:16.755 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 104380 2024-02-05T06:52:16.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.755 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:16.755 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.755 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:16.755 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220080+0000", 2024-02-05T06:52:16.755 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.755 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.755 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:16.755 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220080+0000", 2024-02-05T06:52:16.755 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.755 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.755 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.756 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:16.756 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220081+0000", 2024-02-05T06:52:16.756 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3799999999999996e-07 2024-02-05T06:52:16.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.756 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.756 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:16.756 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220107+0000", 2024-02-05T06:52:16.756 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6037000000000002e-05 2024-02-05T06:52:16.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.756 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.756 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:16.756 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220114+0000", 2024-02-05T06:52:16.756 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8839999999999997e-06 2024-02-05T06:52:16.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.756 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.756 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:16.757 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220120+0000", 2024-02-05T06:52:16.757 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4250000000000003e-06 2024-02-05T06:52:16.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.757 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.757 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:16.757 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220183+0000", 2024-02-05T06:52:16.757 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2460999999999997e-05 2024-02-05T06:52:16.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.757 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.757 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:52:16.757 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220191+0000", 2024-02-05T06:52:16.757 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1810000000000001e-06 2024-02-05T06:52:16.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.757 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.757 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:16.757 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.361008+0000", 2024-02-05T06:52:16.758 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1408166399999999 2024-02-05T06:52:16.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.758 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.758 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:16.758 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.361163+0000", 2024-02-05T06:52:16.758 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015500400000000001 2024-02-05T06:52:16.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.758 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.758 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,4,5,6", 2024-02-05T06:52:16.758 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.361464+0000", 2024-02-05T06:52:16.758 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00030082299999999998 2024-02-05T06:52:16.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.758 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.759 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.759 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.370351+0000", 2024-02-05T06:52:16.759 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0088871969999999995 2024-02-05T06:52:16.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.759 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.759 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.759 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.370395+0000", 2024-02-05T06:52:16.759 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4577999999999999e-05 2024-02-05T06:52:16.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.759 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.759 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.759 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.370435+0000", 2024-02-05T06:52:16.759 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0024000000000002e-05 2024-02-05T06:52:16.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.759 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.759 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:52:16.760 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.371544+0000", 2024-02-05T06:52:16.760 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0011082830000000001 2024-02-05T06:52:16.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.760 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.760 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.760 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.373513+0000", 2024-02-05T06:52:16.760 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0019691769999999999 2024-02-05T06:52:16.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.760 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.760 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:16.760 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.373570+0000", 2024-02-05T06:52:16.760 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6833000000000003e-05 2024-02-05T06:52:16.760 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.760 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.761 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:16.761 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.373689+0000", 2024-02-05T06:52:16.761 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000118954 2024-02-05T06:52:16.761 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.761 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:16.761 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.761 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.761 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45587.0:104395 10.16 10:6a47d5ab:::benchmark_data_smithi028_154709_object104394:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e375)", 2024-02-05T06:52:16.761 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:48:54.220129+0000", 2024-02-05T06:52:16.761 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 202.48264957699999, 2024-02-05T06:52:16.761 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.154474088, 2024-02-05T06:52:16.761 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:16.761 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:16.761 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:52:16.761 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45587", 2024-02-05T06:52:16.761 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:52:16.762 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 104395 2024-02-05T06:52:16.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.762 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:16.762 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.762 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:16.762 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220129+0000", 2024-02-05T06:52:16.762 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.762 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.762 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:16.762 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220129+0000", 2024-02-05T06:52:16.762 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.762 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.762 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:16.762 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220130+0000", 2024-02-05T06:52:16.763 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8500000000000001e-07 2024-02-05T06:52:16.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.763 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.763 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:16.763 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220154+0000", 2024-02-05T06:52:16.763 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4148999999999999e-05 2024-02-05T06:52:16.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.763 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.763 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:16.763 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220162+0000", 2024-02-05T06:52:16.763 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.824e-06 2024-02-05T06:52:16.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.763 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.763 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:16.763 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220168+0000", 2024-02-05T06:52:16.763 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.3280000000000002e-06 2024-02-05T06:52:16.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.764 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.764 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:16.764 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220217+0000", 2024-02-05T06:52:16.764 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.9113000000000002e-05 2024-02-05T06:52:16.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.764 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.764 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:52:16.764 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220224+0000", 2024-02-05T06:52:16.764 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.1359999999999999e-06 2024-02-05T06:52:16.764 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.764 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.764 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:16.764 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.365481+0000", 2024-02-05T06:52:16.765 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.145257325 2024-02-05T06:52:16.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.765 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.765 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:16.765 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.365630+0000", 2024-02-05T06:52:16.765 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014853699999999999 2024-02-05T06:52:16.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.765 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.765 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,4,5,6", 2024-02-05T06:52:16.765 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.365939+0000", 2024-02-05T06:52:16.765 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000309401 2024-02-05T06:52:16.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.765 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.765 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.765 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.370567+0000", 2024-02-05T06:52:16.765 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0046277649999999998 2024-02-05T06:52:16.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.766 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.766 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.766 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.371240+0000", 2024-02-05T06:52:16.766 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00067323999999999997 2024-02-05T06:52:16.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.766 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.766 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.766 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.371528+0000", 2024-02-05T06:52:16.766 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028775399999999998 2024-02-05T06:52:16.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.766 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.766 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:52:16.766 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.371548+0000", 2024-02-05T06:52:16.766 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0254e-05 2024-02-05T06:52:16.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.767 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.767 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.374409+0000", 2024-02-05T06:52:16.767 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00286105 2024-02-05T06:52:16.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.767 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:16.767 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.374475+0000", 2024-02-05T06:52:16.767 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.5717999999999997e-05 2024-02-05T06:52:16.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.767 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.767 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:16.767 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.374603+0000", 2024-02-05T06:52:16.767 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000127909 2024-02-05T06:52:16.767 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.767 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:16.768 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.768 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45587.0:104400 10.16 10:6ec29b03:::benchmark_data_smithi028_154709_object104399:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e375)", 2024-02-05T06:52:16.768 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:48:54.220177+0000", 2024-02-05T06:52:16.768 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 202.48260167199999, 2024-02-05T06:52:16.768 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1560564360000001, 2024-02-05T06:52:16.768 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:16.768 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:16.768 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:52:16.768 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45587", 2024-02-05T06:52:16.768 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:52:16.768 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 104400 2024-02-05T06:52:16.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.768 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:16.768 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.769 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:16.769 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220177+0000", 2024-02-05T06:52:16.769 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.769 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.769 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:16.769 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220177+0000", 2024-02-05T06:52:16.769 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.769 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.769 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.769 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:16.769 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220178+0000", 2024-02-05T06:52:16.770 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4499999999999996e-07 2024-02-05T06:52:16.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.770 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:16.770 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220202+0000", 2024-02-05T06:52:16.770 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4136999999999999e-05 2024-02-05T06:52:16.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.770 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.771 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:16.771 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220208+0000", 2024-02-05T06:52:16.771 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6170000000000003e-06 2024-02-05T06:52:16.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.771 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.771 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:16.771 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220215+0000", 2024-02-05T06:52:16.772 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2609999999999999e-06 2024-02-05T06:52:16.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.772 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.772 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:16.772 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220249+0000", 2024-02-05T06:52:16.772 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.4428999999999999e-05 2024-02-05T06:52:16.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.773 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.773 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:52:16.773 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.220256+0000", 2024-02-05T06:52:16.773 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8580000000000002e-06 2024-02-05T06:52:16.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.773 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.774 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:16.774 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.368713+0000", 2024-02-05T06:52:16.774 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1484565739999999 2024-02-05T06:52:16.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.774 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.774 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:16.774 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.368853+0000", 2024-02-05T06:52:16.775 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000140271 2024-02-05T06:52:16.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.775 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.775 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,4,5,6", 2024-02-05T06:52:16.775 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.369153+0000", 2024-02-05T06:52:16.775 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00030054700000000002 2024-02-05T06:52:16.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.776 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.776 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.776 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.370632+0000", 2024-02-05T06:52:16.776 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001478446 2024-02-05T06:52:16.776 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.776 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.776 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:52:16.776 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.371553+0000", 2024-02-05T06:52:16.776 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00092075600000000001 2024-02-05T06:52:16.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.777 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.777 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.777 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.374059+0000", 2024-02-05T06:52:16.777 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025063949999999998 2024-02-05T06:52:16.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.777 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.777 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.777 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.374876+0000", 2024-02-05T06:52:16.778 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00081689700000000002 2024-02-05T06:52:16.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.778 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.778 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.778 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.375520+0000", 2024-02-05T06:52:16.778 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00064422399999999999 2024-02-05T06:52:16.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.778 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.779 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:16.779 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.375808+0000", 2024-02-05T06:52:16.779 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000287693 2024-02-05T06:52:16.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.779 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.779 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:16.779 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.376233+0000", 2024-02-05T06:52:16.779 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00042558599999999999 2024-02-05T06:52:16.779 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.779 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:16.780 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.780 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.780 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.780 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45587.0:213316 10.8 10:14e7d146:::benchmark_data_smithi028_154709_object92315:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-02-05T06:52:16.780 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:50:35.803795+0000", 2024-02-05T06:52:16.780 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 100.898983692, 2024-02-05T06:52:16.780 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.078076377000000002, 2024-02-05T06:52:16.781 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:16.781 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:16.781 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:52:16.781 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45587", 2024-02-05T06:52:16.781 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:52:16.781 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 213316 2024-02-05T06:52:16.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.781 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:16.781 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.781 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:16.781 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:35.803795+0000", 2024-02-05T06:52:16.781 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.781 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.782 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:16.782 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:35.803795+0000", 2024-02-05T06:52:16.782 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.782 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.782 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.782 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:16.782 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:35.803797+0000", 2024-02-05T06:52:16.782 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.17e-06 2024-02-05T06:52:16.782 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.782 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.782 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:16.782 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:35.803802+0000", 2024-02-05T06:52:16.782 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3380000000000004e-06 2024-02-05T06:52:16.782 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.782 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.782 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:16.783 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:35.803814+0000", 2024-02-05T06:52:16.783 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1481000000000001e-05 2024-02-05T06:52:16.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.783 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.783 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:16.783 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:35.803822+0000", 2024-02-05T06:52:16.783 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5890000000000003e-06 2024-02-05T06:52:16.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.783 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.783 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:16.783 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:35.803878+0000", 2024-02-05T06:52:16.783 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5201000000000002e-05 2024-02-05T06:52:16.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.783 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.783 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:16.784 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:35.803962+0000", 2024-02-05T06:52:16.784 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3974999999999995e-05 2024-02-05T06:52:16.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.784 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.784 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,7", 2024-02-05T06:52:16.784 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:35.804359+0000", 2024-02-05T06:52:16.784 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00039707300000000001 2024-02-05T06:52:16.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.784 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.784 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:52:16.784 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:35.806154+0000", 2024-02-05T06:52:16.784 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001795474 2024-02-05T06:52:16.784 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.784 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.785 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.785 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:35.806839+0000", 2024-02-05T06:52:16.785 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00068532900000000004 2024-02-05T06:52:16.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.785 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.785 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.785 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:35.881712+0000", 2024-02-05T06:52:16.785 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.074872528999999993 2024-02-05T06:52:16.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.785 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.785 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:16.785 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:35.881764+0000", 2024-02-05T06:52:16.785 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1703000000000003e-05 2024-02-05T06:52:16.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.785 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.785 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:16.786 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:35.881871+0000", 2024-02-05T06:52:16.786 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000107515 2024-02-05T06:52:16.786 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.786 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:16.786 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.786 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.786 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.786 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45587.0:213555 10.4 10:2343f6b7:::benchmark_data_smithi028_154709_object92554:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-02-05T06:52:16.786 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:50:36.045196+0000", 2024-02-05T06:52:16.786 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 100.657582336, 2024-02-05T06:52:16.786 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.085254327000000005, 2024-02-05T06:52:16.786 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:16.786 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:16.786 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:52:16.786 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45587", 2024-02-05T06:52:16.786 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:52:16.787 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 213555 2024-02-05T06:52:16.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.787 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:16.787 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.787 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:16.787 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:36.045196+0000", 2024-02-05T06:52:16.787 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.787 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.787 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:16.787 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:36.045196+0000", 2024-02-05T06:52:16.787 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.788 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.788 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:16.788 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:36.045198+0000", 2024-02-05T06:52:16.788 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7719999999999999e-06 2024-02-05T06:52:16.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.788 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.788 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:16.788 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:36.045203+0000", 2024-02-05T06:52:16.788 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3310000000000003e-06 2024-02-05T06:52:16.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.788 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.789 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:16.789 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:36.045215+0000", 2024-02-05T06:52:16.789 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1189000000000001e-05 2024-02-05T06:52:16.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.789 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:16.789 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:36.045223+0000", 2024-02-05T06:52:16.789 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7399999999999993e-06 2024-02-05T06:52:16.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.789 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:16.789 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:36.045306+0000", 2024-02-05T06:52:16.789 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3159999999999997e-05 2024-02-05T06:52:16.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.790 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:16.790 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:36.045416+0000", 2024-02-05T06:52:16.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00010955600000000001 2024-02-05T06:52:16.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.790 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4", 2024-02-05T06:52:16.790 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:36.045886+0000", 2024-02-05T06:52:16.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00046954400000000002 2024-02-05T06:52:16.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.790 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:52:16.790 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:36.049757+0000", 2024-02-05T06:52:16.790 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003871228 2024-02-05T06:52:16.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.790 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.790 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:36.052058+0000", 2024-02-05T06:52:16.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0023014910000000001 2024-02-05T06:52:16.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:36.130312+0000", 2024-02-05T06:52:16.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.078253952000000002 2024-02-05T06:52:16.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:16.791 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:36.130409+0000", 2024-02-05T06:52:16.791 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7198e-05 2024-02-05T06:52:16.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.791 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.791 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:16.792 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:36.130451+0000", 2024-02-05T06:52:16.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1165999999999999e-05 2024-02-05T06:52:16.792 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.792 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:16.792 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.792 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.792 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45587.0:213572 10.4 10:26059152:::benchmark_data_smithi028_154709_object92571:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-02-05T06:52:16.792 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:50:36.057845+0000", 2024-02-05T06:52:16.792 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 100.644933483, 2024-02-05T06:52:16.792 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.087135957, 2024-02-05T06:52:16.792 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:16.792 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:16.792 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:52:16.792 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45587", 2024-02-05T06:52:16.793 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:52:16.793 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 213572 2024-02-05T06:52:16.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.793 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:16.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.793 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:16.793 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:36.057845+0000", 2024-02-05T06:52:16.793 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.793 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:16.793 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:36.057845+0000", 2024-02-05T06:52:16.793 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.793 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.794 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:16.794 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:36.057847+0000", 2024-02-05T06:52:16.794 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8309999999999999e-06 2024-02-05T06:52:16.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.794 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:16.794 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:36.057851+0000", 2024-02-05T06:52:16.794 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1439999999999996e-06 2024-02-05T06:52:16.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.794 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:16.794 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:36.121076+0000", 2024-02-05T06:52:16.794 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.063225206000000006 2024-02-05T06:52:16.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.794 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.794 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:16.795 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:36.121097+0000", 2024-02-05T06:52:16.795 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1041000000000001e-05 2024-02-05T06:52:16.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.795 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.795 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:16.795 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:36.132219+0000", 2024-02-05T06:52:16.795 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011121376000000001 2024-02-05T06:52:16.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.795 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.795 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:16.795 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:36.132360+0000", 2024-02-05T06:52:16.795 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014103600000000001 2024-02-05T06:52:16.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.796 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.796 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,4", 2024-02-05T06:52:16.796 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:36.132847+0000", 2024-02-05T06:52:16.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000486806 2024-02-05T06:52:16.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.797 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:52:16.797 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:36.135135+0000", 2024-02-05T06:52:16.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002288389 2024-02-05T06:52:16.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.797 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.797 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:36.135765+0000", 2024-02-05T06:52:16.797 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00063015300000000005 2024-02-05T06:52:16.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.797 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.797 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:36.144655+0000", 2024-02-05T06:52:16.798 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0088899510000000001 2024-02-05T06:52:16.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.798 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:16.798 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:36.144894+0000", 2024-02-05T06:52:16.798 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00023885399999999999 2024-02-05T06:52:16.798 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.798 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.798 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:16.798 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:36.144981+0000", 2024-02-05T06:52:16.799 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.7169999999999994e-05 2024-02-05T06:52:16.799 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.799 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:16.799 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.799 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.799 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45878.0:42493 11.6 11:6b5cc266:::benchmark_data_smithi028_177697_object42492:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-02-05T06:52:16.799 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:51:41.181034+0000", 2024-02-05T06:52:16.799 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 35.521744145, 2024-02-05T06:52:16.799 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.080653806999999994, 2024-02-05T06:52:16.799 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:16.799 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:16.799 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:52:16.800 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45878", 2024-02-05T06:52:16.800 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3059201968", 2024-02-05T06:52:16.800 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 42493 2024-02-05T06:52:16.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.800 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:16.800 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.800 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:16.800 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:41.181034+0000", 2024-02-05T06:52:16.800 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.800 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.800 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:16.800 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:41.181034+0000", 2024-02-05T06:52:16.800 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.801 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.801 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:16.801 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:41.181036+0000", 2024-02-05T06:52:16.801 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2780000000000001e-06 2024-02-05T06:52:16.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.801 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.801 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:16.801 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:41.181058+0000", 2024-02-05T06:52:16.801 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2687000000000001e-05 2024-02-05T06:52:16.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.801 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.801 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:16.801 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:41.181070+0000", 2024-02-05T06:52:16.801 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1317e-05 2024-02-05T06:52:16.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.802 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:16.802 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:41.181079+0000", 2024-02-05T06:52:16.802 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0499999999999997e-06 2024-02-05T06:52:16.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.802 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:16.802 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:41.182434+0000", 2024-02-05T06:52:16.802 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0013556250000000001 2024-02-05T06:52:16.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.802 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:16.802 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:41.182566+0000", 2024-02-05T06:52:16.802 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013212 2024-02-05T06:52:16.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.802 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.803 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-02-05T06:52:16.803 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:41.182751+0000", 2024-02-05T06:52:16.803 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018432400000000001 2024-02-05T06:52:16.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.803 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:52:16.803 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:41.185706+0000", 2024-02-05T06:52:16.803 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002955624 2024-02-05T06:52:16.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.803 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.803 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:41.186761+0000", 2024-02-05T06:52:16.803 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.001054422 2024-02-05T06:52:16.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.803 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.804 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.804 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:41.261548+0000", 2024-02-05T06:52:16.804 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.074787093999999998 2024-02-05T06:52:16.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.804 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.804 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:16.804 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:41.261630+0000", 2024-02-05T06:52:16.804 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.2045000000000005e-05 2024-02-05T06:52:16.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.804 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.804 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:16.804 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:41.261688+0000", 2024-02-05T06:52:16.804 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8220999999999997e-05 2024-02-05T06:52:16.804 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.805 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:16.805 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.805 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.805 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45878.0:44608 11.6 11:65e3973c:::benchmark_data_smithi028_177697_object44607:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-02-05T06:52:16.805 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:51:42.632127+0000", 2024-02-05T06:52:16.805 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 34.070651491, 2024-02-05T06:52:16.805 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.097412076, 2024-02-05T06:52:16.805 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:16.805 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:16.805 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:52:16.805 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45878", 2024-02-05T06:52:16.805 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3059201968", 2024-02-05T06:52:16.805 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 44608 2024-02-05T06:52:16.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.805 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:16.805 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.806 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:16.806 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.632127+0000", 2024-02-05T06:52:16.806 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.806 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:16.806 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.632127+0000", 2024-02-05T06:52:16.806 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.806 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:16.806 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.632129+0000", 2024-02-05T06:52:16.806 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.8369999999999999e-06 2024-02-05T06:52:16.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.806 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.807 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:16.807 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.632151+0000", 2024-02-05T06:52:16.807 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2325000000000001e-05 2024-02-05T06:52:16.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.807 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.807 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:16.807 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.632161+0000", 2024-02-05T06:52:16.807 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8640000000000008e-06 2024-02-05T06:52:16.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.807 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.807 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:16.807 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.632406+0000", 2024-02-05T06:52:16.807 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024460200000000001 2024-02-05T06:52:16.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.807 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.808 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:16.808 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.632605+0000", 2024-02-05T06:52:16.808 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00019923 2024-02-05T06:52:16.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.808 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:16.808 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.632850+0000", 2024-02-05T06:52:16.808 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024500800000000001 2024-02-05T06:52:16.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.808 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.808 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-02-05T06:52:16.808 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.633001+0000", 2024-02-05T06:52:16.808 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000151216 2024-02-05T06:52:16.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.809 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:52:16.809 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.635576+0000", 2024-02-05T06:52:16.809 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0025749169999999999 2024-02-05T06:52:16.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.809 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.809 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.635870+0000", 2024-02-05T06:52:16.809 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029367800000000002 2024-02-05T06:52:16.809 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.809 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.809 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.809 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.729443+0000", 2024-02-05T06:52:16.809 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.093573325999999998 2024-02-05T06:52:16.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.810 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:16.810 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.729496+0000", 2024-02-05T06:52:16.810 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3176e-05 2024-02-05T06:52:16.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.810 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:16.810 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.729539+0000", 2024-02-05T06:52:16.810 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2896999999999999e-05 2024-02-05T06:52:16.810 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.810 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:16.810 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.810 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.811 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45878.0:44626 11.6 11:6be00445:::benchmark_data_smithi028_177697_object44625:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-02-05T06:52:16.811 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:51:42.642217+0000", 2024-02-05T06:52:16.811 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 34.060561067000002, 2024-02-05T06:52:16.811 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.087468888999999994, 2024-02-05T06:52:16.811 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:16.811 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:16.811 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:52:16.811 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45878", 2024-02-05T06:52:16.811 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3059201968", 2024-02-05T06:52:16.811 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 44626 2024-02-05T06:52:16.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.811 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:16.811 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.811 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:16.811 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.642217+0000", 2024-02-05T06:52:16.811 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.812 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.812 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:16.812 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.642217+0000", 2024-02-05T06:52:16.812 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.812 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.812 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:16.812 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.642219+0000", 2024-02-05T06:52:16.812 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9319999999999999e-06 2024-02-05T06:52:16.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.812 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.812 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:16.812 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.642242+0000", 2024-02-05T06:52:16.812 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.23e-05 2024-02-05T06:52:16.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.813 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.813 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:16.813 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.642266+0000", 2024-02-05T06:52:16.813 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3869000000000001e-05 2024-02-05T06:52:16.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.813 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.813 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:16.813 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.642275+0000", 2024-02-05T06:52:16.813 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.9309999999999997e-06 2024-02-05T06:52:16.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.813 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.813 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:16.813 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.642339+0000", 2024-02-05T06:52:16.813 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4045000000000001e-05 2024-02-05T06:52:16.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.814 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.814 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:16.814 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.642466+0000", 2024-02-05T06:52:16.814 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00012707699999999999 2024-02-05T06:52:16.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.814 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.814 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-02-05T06:52:16.814 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.642598+0000", 2024-02-05T06:52:16.814 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013264099999999999 2024-02-05T06:52:16.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.814 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.814 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:52:16.814 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.644770+0000", 2024-02-05T06:52:16.815 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002171769 2024-02-05T06:52:16.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.815 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.815 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.645124+0000", 2024-02-05T06:52:16.815 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00035430900000000003 2024-02-05T06:52:16.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.815 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.815 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.729600+0000", 2024-02-05T06:52:16.815 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.084475250000000002 2024-02-05T06:52:16.815 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.815 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.815 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:16.816 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.729650+0000", 2024-02-05T06:52:16.816 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0560999999999999e-05 2024-02-05T06:52:16.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.816 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.816 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:16.816 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.729686+0000", 2024-02-05T06:52:16.816 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.6205000000000003e-05 2024-02-05T06:52:16.816 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.816 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:16.816 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.816 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.816 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45878.0:44632 11.6 11:6fc1aaf2:::benchmark_data_smithi028_177697_object44631:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-02-05T06:52:16.817 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:51:42.645995+0000", 2024-02-05T06:52:16.817 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 34.056783893999999, 2024-02-05T06:52:16.817 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.083844451, 2024-02-05T06:52:16.817 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:16.817 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:16.817 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:52:16.817 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45878", 2024-02-05T06:52:16.817 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3059201968", 2024-02-05T06:52:16.817 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 44632 2024-02-05T06:52:16.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.817 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:16.817 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.817 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:16.817 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.645995+0000", 2024-02-05T06:52:16.817 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:16.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.645995+0000", 2024-02-05T06:52:16.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:16.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.645997+0000", 2024-02-05T06:52:16.818 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2089999999999999e-06 2024-02-05T06:52:16.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.818 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.818 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:16.818 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.646023+0000", 2024-02-05T06:52:16.819 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.588e-05 2024-02-05T06:52:16.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.819 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:16.819 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.646039+0000", 2024-02-05T06:52:16.819 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6507e-05 2024-02-05T06:52:16.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.819 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:16.819 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.646049+0000", 2024-02-05T06:52:16.819 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2499999999999995e-06 2024-02-05T06:52:16.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.819 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:16.819 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.646129+0000", 2024-02-05T06:52:16.820 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.0238e-05 2024-02-05T06:52:16.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.820 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:16.820 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.646289+0000", 2024-02-05T06:52:16.820 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000160364 2024-02-05T06:52:16.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.820 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-02-05T06:52:16.820 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.646434+0000", 2024-02-05T06:52:16.820 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000144747 2024-02-05T06:52:16.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.820 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.820 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:52:16.821 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.648451+0000", 2024-02-05T06:52:16.821 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020175459999999998 2024-02-05T06:52:16.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.821 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.821 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.650347+0000", 2024-02-05T06:52:16.821 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018956120000000001 2024-02-05T06:52:16.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.821 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.821 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.729741+0000", 2024-02-05T06:52:16.821 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.079393620999999998 2024-02-05T06:52:16.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.821 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.821 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:16.822 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.729798+0000", 2024-02-05T06:52:16.822 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7008e-05 2024-02-05T06:52:16.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.822 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.822 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:16.822 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.729839+0000", 2024-02-05T06:52:16.822 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1468999999999999e-05 2024-02-05T06:52:16.822 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.822 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:16.822 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.822 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.822 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.822 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45878.0:44635 11.6 11:676f35bd:::benchmark_data_smithi028_177697_object44634:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e424)", 2024-02-05T06:52:16.822 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:51:42.648710+0000", 2024-02-05T06:52:16.822 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 34.054068233999999, 2024-02-05T06:52:16.822 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.085479095000000005, 2024-02-05T06:52:16.823 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:16.823 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:16.823 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:52:16.823 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45878", 2024-02-05T06:52:16.823 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3059201968", 2024-02-05T06:52:16.823 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 44635 2024-02-05T06:52:16.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.823 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:16.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.823 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:16.823 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.648710+0000", 2024-02-05T06:52:16.823 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.823 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.823 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:16.824 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.648710+0000", 2024-02-05T06:52:16.824 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.824 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:16.824 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.648713+0000", 2024-02-05T06:52:16.824 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4880000000000001e-06 2024-02-05T06:52:16.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.824 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:16.824 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.648736+0000", 2024-02-05T06:52:16.824 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3683999999999999e-05 2024-02-05T06:52:16.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.824 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.825 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:16.825 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.648749+0000", 2024-02-05T06:52:16.825 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2884000000000001e-05 2024-02-05T06:52:16.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.825 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.825 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:16.825 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.648759+0000", 2024-02-05T06:52:16.825 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7459999999999995e-06 2024-02-05T06:52:16.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.825 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.825 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:16.825 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.648835+0000", 2024-02-05T06:52:16.826 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5982000000000005e-05 2024-02-05T06:52:16.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.826 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:16.826 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.648980+0000", 2024-02-05T06:52:16.826 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014538499999999999 2024-02-05T06:52:16.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.826 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-02-05T06:52:16.826 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.649131+0000", 2024-02-05T06:52:16.826 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015049700000000001 2024-02-05T06:52:16.826 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.826 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.826 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:52:16.827 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.650437+0000", 2024-02-05T06:52:16.827 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0013057769999999999 2024-02-05T06:52:16.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.827 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.827 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.650618+0000", 2024-02-05T06:52:16.827 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00018158000000000001 2024-02-05T06:52:16.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.827 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.827 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.734075+0000", 2024-02-05T06:52:16.827 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.083456354999999996 2024-02-05T06:52:16.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.827 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.828 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:16.828 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.734151+0000", 2024-02-05T06:52:16.828 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5821000000000005e-05 2024-02-05T06:52:16.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.828 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:16.828 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:42.734189+0000", 2024-02-05T06:52:16.828 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8896000000000002e-05 2024-02-05T06:52:16.828 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.828 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:16.828 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.828 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.829 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "rep_scrubmap(6.0 e71 from shard 3)", 2024-02-05T06:52:16.829 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:51:43.440414+0000", 2024-02-05T06:52:16.829 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 33.262364374999997, 2024-02-05T06:52:16.829 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.058327498999999998, 2024-02-05T06:52:16.829 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:16.829 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-02-05T06:52:16.829 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:16.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.829 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:16.829 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:43.440414+0000", 2024-02-05T06:52:16.829 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.829 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.829 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.829 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:16.829 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:43.440414+0000", 2024-02-05T06:52:16.830 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.830 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:16.830 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:43.440416+0000", 2024-02-05T06:52:16.830 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.319e-06 2024-02-05T06:52:16.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.830 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:16.830 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:43.440421+0000", 2024-02-05T06:52:16.830 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.558e-06 2024-02-05T06:52:16.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.830 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.830 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:16.831 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:43.440431+0000", 2024-02-05T06:52:16.831 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0360000000000001e-05 2024-02-05T06:52:16.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.831 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:16.831 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:43.440439+0000", 2024-02-05T06:52:16.831 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2860000000000002e-06 2024-02-05T06:52:16.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.831 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:16.831 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:43.498716+0000", 2024-02-05T06:52:16.831 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.058277743999999999 2024-02-05T06:52:16.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.831 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.831 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:16.831 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:43.498726+0000", 2024-02-05T06:52:16.832 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9529999999999994e-06 2024-02-05T06:52:16.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.832 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.832 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:16.832 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:43.498742+0000", 2024-02-05T06:52:16.832 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5279000000000001e-05 2024-02-05T06:52:16.832 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.832 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:16.832 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.832 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.832 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "rep_scrubmap(6.0 e71 from shard 2)", 2024-02-05T06:52:16.832 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:51:43.441450+0000", 2024-02-05T06:52:16.832 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 33.261328863000003, 2024-02-05T06:52:16.832 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.057349417999999999, 2024-02-05T06:52:16.833 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:16.833 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-02-05T06:52:16.833 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:16.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.833 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:16.833 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:43.441450+0000", 2024-02-05T06:52:16.833 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.833 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:16.833 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:43.441450+0000", 2024-02-05T06:52:16.833 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.833 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:16.833 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:43.441450+0000", 2024-02-05T06:52:16.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6700000000000003e-07 2024-02-05T06:52:16.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.834 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:16.834 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:43.441454+0000", 2024-02-05T06:52:16.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8079999999999998e-06 2024-02-05T06:52:16.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.834 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:16.834 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:43.441459+0000", 2024-02-05T06:52:16.834 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0889999999999999e-06 2024-02-05T06:52:16.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.834 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:16.834 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:43.441463+0000", 2024-02-05T06:52:16.835 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2200000000000003e-06 2024-02-05T06:52:16.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.835 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:16.835 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:43.498763+0000", 2024-02-05T06:52:16.835 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.057299783999999999 2024-02-05T06:52:16.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.835 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:16.835 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:43.498771+0000", 2024-02-05T06:52:16.835 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9680000000000006e-06 2024-02-05T06:52:16.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.835 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:16.835 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:43.498799+0000", 2024-02-05T06:52:16.835 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7982000000000001e-05 2024-02-05T06:52:16.836 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.836 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:16.836 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.836 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45878.0:55583 11.6 11:65d76fa7:::benchmark_data_smithi028_177697_object55582:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-02-05T06:52:16.836 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:51:50.318395+0000", 2024-02-05T06:52:16.836 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 26.384383452000002, 2024-02-05T06:52:16.836 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.097895789999999996, 2024-02-05T06:52:16.836 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:16.836 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:16.836 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:52:16.836 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45878", 2024-02-05T06:52:16.836 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3059201968", 2024-02-05T06:52:16.837 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 55583 2024-02-05T06:52:16.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.837 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:16.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.837 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:16.837 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.318395+0000", 2024-02-05T06:52:16.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.837 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:16.837 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.318395+0000", 2024-02-05T06:52:16.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:16.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.837 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:16.837 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.318397+0000", 2024-02-05T06:52:16.837 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.778e-06 2024-02-05T06:52:16.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.838 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:16.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.318420+0000", 2024-02-05T06:52:16.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3289999999999999e-05 2024-02-05T06:52:16.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.838 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:16.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.318430+0000", 2024-02-05T06:52:16.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.8519999999999999e-06 2024-02-05T06:52:16.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.838 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:16.838 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.318438+0000", 2024-02-05T06:52:16.838 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.79e-06 2024-02-05T06:52:16.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.839 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:16.839 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.318745+0000", 2024-02-05T06:52:16.839 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00030676899999999998 2024-02-05T06:52:16.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.839 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:16.839 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.318894+0000", 2024-02-05T06:52:16.839 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014895800000000001 2024-02-05T06:52:16.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.839 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 1,2", 2024-02-05T06:52:16.839 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.319044+0000", 2024-02-05T06:52:16.839 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014997200000000001 2024-02-05T06:52:16.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.840 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:52:16.840 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.321453+0000", 2024-02-05T06:52:16.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024091999999999998 2024-02-05T06:52:16.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.840 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.840 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.322219+0000", 2024-02-05T06:52:16.840 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00076581999999999996 2024-02-05T06:52:16.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.840 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:16.841 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.416117+0000", 2024-02-05T06:52:16.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.093898809 2024-02-05T06:52:16.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.841 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:16.841 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.416234+0000", 2024-02-05T06:52:16.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000116253 2024-02-05T06:52:16.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:16.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:16.841 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:16.841 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.416291+0000", 2024-02-05T06:52:16.841 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7299000000000001e-05 2024-02-05T06:52:16.841 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.841 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:16.841 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.841 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:16.841 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:16.842 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:52:16.842 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:16.881 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:16.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:16.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:16.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:17.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:17.027 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:17.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:17.128 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:17.163 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:17.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:17.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:17.265 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:17.310 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:17.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:17.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:17.344 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 56 16 88584 88568 98.8311 61.125 0.0167237 0.0101128 2024-02-05T06:52:17.412 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:17.450 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:17.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:17.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:17.552 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:17.587 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:17.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:17.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:17.651 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-02-05T06:52:17.688 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:17.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:17.721 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:17.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:17.822 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:17.856 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:17.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:17.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:17.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:17 smithi028 ceph-mon[29442]: pgmap v901: 168 pgs: 168 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-02-05T06:52:17.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:17 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3442049935' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:52:17.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:17 smithi028 ceph-mon[37984]: pgmap v901: 168 pgs: 168 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-02-05T06:52:17.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:17 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3442049935' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:52:17.957 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:17.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:17 smithi167 ceph-mon[32774]: pgmap v901: 168 pgs: 168 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-02-05T06:52:17.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:17 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3442049935' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:52:17.999 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:17.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:18.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:18.100 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:18.143 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:18.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:18.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:18.244 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:18.277 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:18.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:18.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:18.344 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 57 16 89753 89737 98.3789 73.0625 0.0088036 0.01016 2024-02-05T06:52:18.378 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:18.410 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:18.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:18.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:18.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:18.555 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:18.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:18.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:18.657 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:18.689 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:18.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:18.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:18.791 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:18.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:18.834 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:18.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:18.935 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:18.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:18 smithi028 ceph-mon[29442]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:52:18.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:18 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3442049935' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:52:18.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:18 smithi028 ceph-mon[29442]: osdmap e438: 8 total, 8 up, 8 in 2024-02-05T06:52:18.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:18 smithi028 ceph-mon[29442]: 1.1 scrub starts 2024-02-05T06:52:18.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:18 smithi028 ceph-mon[29442]: 1.1 scrub ok 2024-02-05T06:52:18.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:18 smithi028 ceph-mon[37984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:52:18.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:18 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3442049935' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:52:18.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:18 smithi028 ceph-mon[37984]: osdmap e438: 8 total, 8 up, 8 in 2024-02-05T06:52:18.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:18 smithi028 ceph-mon[37984]: 1.1 scrub starts 2024-02-05T06:52:18.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:18 smithi028 ceph-mon[37984]: 1.1 scrub ok 2024-02-05T06:52:18.967 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:18.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:18.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:18.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:18 smithi167 ceph-mon[32774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:52:18.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:18 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3442049935' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:52:18.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:18 smithi167 ceph-mon[32774]: osdmap e438: 8 total, 8 up, 8 in 2024-02-05T06:52:18.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:18 smithi167 ceph-mon[32774]: 1.1 scrub starts 2024-02-05T06:52:18.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:18 smithi167 ceph-mon[32774]: 1.1 scrub ok 2024-02-05T06:52:19.070 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:19.104 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:19.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:19.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:19.206 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:19.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:19.238 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:19.238 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:19.339 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:19.345 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 58 16 90997 90981 98.0232 77.75 0.0261108 0.0101969 2024-02-05T06:52:19.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:19.390 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:19.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:19.405 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-02-05T06:52:19.491 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:19.541 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:19.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:19.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:19.643 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:19.672 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:19.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:19.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:19.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:19.816 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:19.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:19.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:19.918 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:19.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:19 smithi028 ceph-mon[29442]: 4.18 scrub starts 2024-02-05T06:52:19.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:19 smithi028 ceph-mon[29442]: 4.18 scrub ok 2024-02-05T06:52:19.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:19 smithi028 ceph-mon[29442]: 1.4 scrub starts 2024-02-05T06:52:19.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:19 smithi028 ceph-mon[29442]: 1.4 scrub ok 2024-02-05T06:52:19.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:19 smithi028 ceph-mon[29442]: pgmap v903: 168 pgs: 168 active+clean; 5.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-02-05T06:52:19.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:19 smithi028 ceph-mon[29442]: 1.6 scrub starts 2024-02-05T06:52:19.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:19 smithi028 ceph-mon[29442]: 1.6 scrub ok 2024-02-05T06:52:19.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:19 smithi028 ceph-mon[29442]: 5.15 scrub starts 2024-02-05T06:52:19.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:19 smithi028 ceph-mon[29442]: 5.15 scrub ok 2024-02-05T06:52:19.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:19 smithi028 ceph-mon[37984]: 4.18 scrub starts 2024-02-05T06:52:19.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:19 smithi028 ceph-mon[37984]: 4.18 scrub ok 2024-02-05T06:52:19.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:19 smithi028 ceph-mon[37984]: 1.4 scrub starts 2024-02-05T06:52:19.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:19 smithi028 ceph-mon[37984]: 1.4 scrub ok 2024-02-05T06:52:19.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:19 smithi028 ceph-mon[37984]: pgmap v903: 168 pgs: 168 active+clean; 5.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-02-05T06:52:19.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:19 smithi028 ceph-mon[37984]: 1.6 scrub starts 2024-02-05T06:52:19.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:19 smithi028 ceph-mon[37984]: 1.6 scrub ok 2024-02-05T06:52:19.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:19 smithi028 ceph-mon[37984]: 5.15 scrub starts 2024-02-05T06:52:19.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:19 smithi028 ceph-mon[37984]: 5.15 scrub ok 2024-02-05T06:52:19.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:19.953 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:19.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:19.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:19 smithi167 ceph-mon[32774]: 4.18 scrub starts 2024-02-05T06:52:19.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:19 smithi167 ceph-mon[32774]: 4.18 scrub ok 2024-02-05T06:52:19.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:19 smithi167 ceph-mon[32774]: 1.4 scrub starts 2024-02-05T06:52:19.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:19 smithi167 ceph-mon[32774]: 1.4 scrub ok 2024-02-05T06:52:19.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:19 smithi167 ceph-mon[32774]: pgmap v903: 168 pgs: 168 active+clean; 5.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-02-05T06:52:19.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:19 smithi167 ceph-mon[32774]: 1.6 scrub starts 2024-02-05T06:52:19.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:19 smithi167 ceph-mon[32774]: 1.6 scrub ok 2024-02-05T06:52:19.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:19 smithi167 ceph-mon[32774]: 5.15 scrub starts 2024-02-05T06:52:19.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:19 smithi167 ceph-mon[32774]: 5.15 scrub ok 2024-02-05T06:52:20.054 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:20.092 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:20.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:20.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:20.185 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:52:20.194 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:20.222 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:20.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:20.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:20.324 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:20.345 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 59 16 92115 92099 97.5453 69.875 0.00835016 0.0102468 2024-02-05T06:52:20.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:20.378 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:20.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:20.478 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:20.517 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:20.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:20.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:20.619 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:20.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:20.653 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:20.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:20.754 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:20.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:20.785 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:20.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:20.878 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:20 smithi028 ceph-mon[29442]: 1.7 scrub starts 2024-02-05T06:52:20.878 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:20 smithi028 ceph-mon[29442]: 1.7 scrub ok 2024-02-05T06:52:20.878 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:20 smithi028 ceph-mon[29442]: 1.14 scrub starts 2024-02-05T06:52:20.878 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:20 smithi028 ceph-mon[29442]: 1.14 scrub ok 2024-02-05T06:52:20.878 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:20 smithi028 ceph-mon[29442]: 5.f scrub starts 2024-02-05T06:52:20.878 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:20 smithi028 ceph-mon[29442]: 5.f scrub ok 2024-02-05T06:52:20.878 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:20 smithi028 ceph-mon[29442]: 6.15 scrub starts 2024-02-05T06:52:20.878 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:20 smithi028 ceph-mon[29442]: 6.15 scrub ok 2024-02-05T06:52:20.879 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:20 smithi028 ceph-mon[29442]: osdmap e439: 8 total, 8 up, 8 in 2024-02-05T06:52:20.879 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:20 smithi028 ceph-mon[37984]: 1.7 scrub starts 2024-02-05T06:52:20.879 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:20 smithi028 ceph-mon[37984]: 1.7 scrub ok 2024-02-05T06:52:20.879 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:20 smithi028 ceph-mon[37984]: 1.14 scrub starts 2024-02-05T06:52:20.879 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:20 smithi028 ceph-mon[37984]: 1.14 scrub ok 2024-02-05T06:52:20.879 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:20 smithi028 ceph-mon[37984]: 5.f scrub starts 2024-02-05T06:52:20.879 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:20 smithi028 ceph-mon[37984]: 5.f scrub ok 2024-02-05T06:52:20.879 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:20 smithi028 ceph-mon[37984]: 6.15 scrub starts 2024-02-05T06:52:20.879 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:20 smithi028 ceph-mon[37984]: 6.15 scrub ok 2024-02-05T06:52:20.879 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:20 smithi028 ceph-mon[37984]: osdmap e439: 8 total, 8 up, 8 in 2024-02-05T06:52:20.886 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:20.919 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:20.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:20.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:20.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:20 smithi167 ceph-mon[32774]: 1.7 scrub starts 2024-02-05T06:52:20.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:20 smithi167 ceph-mon[32774]: 1.7 scrub ok 2024-02-05T06:52:20.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:20 smithi167 ceph-mon[32774]: 1.14 scrub starts 2024-02-05T06:52:20.987 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:20 smithi167 ceph-mon[32774]: 1.14 scrub ok 2024-02-05T06:52:20.987 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:20 smithi167 ceph-mon[32774]: 5.f scrub starts 2024-02-05T06:52:20.987 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:20 smithi167 ceph-mon[32774]: 5.f scrub ok 2024-02-05T06:52:20.987 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:20 smithi167 ceph-mon[32774]: 6.15 scrub starts 2024-02-05T06:52:20.987 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:20 smithi167 ceph-mon[32774]: 6.15 scrub ok 2024-02-05T06:52:20.988 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:20 smithi167 ceph-mon[32774]: osdmap e439: 8 total, 8 up, 8 in 2024-02-05T06:52:21.021 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:21.054 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:21.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:21.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:21.155 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:21.170 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:52:21.206 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:21.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:21.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:21.308 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:21.348 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-02-05T06:52:21.340787+0000 min lat: 0.0012181 max lat: 0.204312 avg lat: 0.0102892 2024-02-05T06:52:21.348 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T06:52:21.348 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 60 16 93298 93282 97.1471 73.9375 0.0101794 0.0102892 2024-02-05T06:52:21.355 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:21.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:21.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:21.457 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:21.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:21.502 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:21.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:21.603 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:21.643 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:21.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:21.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:21.745 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:21.752 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:21 smithi028 ceph-mon[37984]: pgmap v905: 168 pgs: 168 active+clean; 5.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 910 op/s 2024-02-05T06:52:21.752 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:21 smithi028 ceph-mon[37984]: 5.2 scrub starts 2024-02-05T06:52:21.752 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:21 smithi028 ceph-mon[37984]: 5.2 scrub ok 2024-02-05T06:52:21.752 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:21 smithi028 ceph-mon[37984]: 1.10 scrub starts 2024-02-05T06:52:21.752 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:21 smithi028 ceph-mon[37984]: 1.10 scrub ok 2024-02-05T06:52:21.752 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:21 smithi028 ceph-mon[37984]: 1.13 scrub starts 2024-02-05T06:52:21.752 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:21 smithi028 ceph-mon[37984]: 1.13 scrub ok 2024-02-05T06:52:21.753 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:21 smithi028 ceph-mon[37984]: 1.e scrub starts 2024-02-05T06:52:21.753 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:21 smithi028 ceph-mon[37984]: 1.e scrub ok 2024-02-05T06:52:21.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:21 smithi028 ceph-mon[29442]: pgmap v905: 168 pgs: 168 active+clean; 5.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 910 op/s 2024-02-05T06:52:21.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:21 smithi028 ceph-mon[29442]: 5.2 scrub starts 2024-02-05T06:52:21.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:21 smithi028 ceph-mon[29442]: 5.2 scrub ok 2024-02-05T06:52:21.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:21 smithi028 ceph-mon[29442]: 1.10 scrub starts 2024-02-05T06:52:21.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:21 smithi028 ceph-mon[29442]: 1.10 scrub ok 2024-02-05T06:52:21.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:21 smithi028 ceph-mon[29442]: 1.13 scrub starts 2024-02-05T06:52:21.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:21 smithi028 ceph-mon[29442]: 1.13 scrub ok 2024-02-05T06:52:21.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:21 smithi028 ceph-mon[29442]: 1.e scrub starts 2024-02-05T06:52:21.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:21 smithi028 ceph-mon[29442]: 1.e scrub ok 2024-02-05T06:52:21.790 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:21.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:21.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:21.893 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:21.931 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:21.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:21.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:21.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:21 smithi167 ceph-mon[32774]: pgmap v905: 168 pgs: 168 active+clean; 5.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 910 op/s 2024-02-05T06:52:21.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:21 smithi167 ceph-mon[32774]: 5.2 scrub starts 2024-02-05T06:52:21.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:21 smithi167 ceph-mon[32774]: 5.2 scrub ok 2024-02-05T06:52:21.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:21 smithi167 ceph-mon[32774]: 1.10 scrub starts 2024-02-05T06:52:21.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:21 smithi167 ceph-mon[32774]: 1.10 scrub ok 2024-02-05T06:52:21.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:21 smithi167 ceph-mon[32774]: 1.13 scrub starts 2024-02-05T06:52:21.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:21 smithi167 ceph-mon[32774]: 1.13 scrub ok 2024-02-05T06:52:21.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:21 smithi167 ceph-mon[32774]: 1.e scrub starts 2024-02-05T06:52:21.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:21 smithi167 ceph-mon[32774]: 1.e scrub ok 2024-02-05T06:52:22.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:22.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:22.075 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:22.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:22.177 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:22.217 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:22.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:22.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:22.319 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:22.348 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 61 16 94512 94496 96.7983 75.875 0.0106829 0.0103259 2024-02-05T06:52:22.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:22.354 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:22.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:22.455 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:22.491 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:22.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:22.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:22.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:22.625 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:52:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:52:22.634 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:22.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:22.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:22.736 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:22.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T06:52:22.777 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:22.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:22.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:22.879 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:22.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:22.913 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:22.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:22.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[29442]: 1.c scrub starts 2024-02-05T06:52:22.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[29442]: 1.c scrub ok 2024-02-05T06:52:22.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:52:22.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:52:22.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[29442]: 1.f scrub starts 2024-02-05T06:52:22.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[29442]: 1.f scrub ok 2024-02-05T06:52:22.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[29442]: 1.5 scrub starts 2024-02-05T06:52:22.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[29442]: 1.5 scrub ok 2024-02-05T06:52:22.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[29442]: 1.12 scrub starts 2024-02-05T06:52:22.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[29442]: 1.12 scrub ok 2024-02-05T06:52:22.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[29442]: 1.b scrub starts 2024-02-05T06:52:22.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[29442]: 1.b scrub ok 2024-02-05T06:52:22.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[29442]: 1.2 scrub starts 2024-02-05T06:52:22.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[29442]: 1.2 scrub ok 2024-02-05T06:52:22.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[29442]: 1.11 scrub starts 2024-02-05T06:52:22.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[29442]: 1.11 scrub ok 2024-02-05T06:52:22.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[29442]: osdmap e440: 8 total, 8 up, 8 in 2024-02-05T06:52:22.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[37984]: 1.c scrub starts 2024-02-05T06:52:22.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[37984]: 1.c scrub ok 2024-02-05T06:52:22.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:52:22.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:52:22.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[37984]: 1.f scrub starts 2024-02-05T06:52:22.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[37984]: 1.f scrub ok 2024-02-05T06:52:22.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[37984]: 1.5 scrub starts 2024-02-05T06:52:22.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[37984]: 1.5 scrub ok 2024-02-05T06:52:22.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[37984]: 1.12 scrub starts 2024-02-05T06:52:22.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[37984]: 1.12 scrub ok 2024-02-05T06:52:22.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[37984]: 1.b scrub starts 2024-02-05T06:52:22.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[37984]: 1.b scrub ok 2024-02-05T06:52:22.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[37984]: 1.2 scrub starts 2024-02-05T06:52:22.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[37984]: 1.2 scrub ok 2024-02-05T06:52:22.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[37984]: 1.11 scrub starts 2024-02-05T06:52:22.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[37984]: 1.11 scrub ok 2024-02-05T06:52:22.954 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:22 smithi028 ceph-mon[37984]: osdmap e440: 8 total, 8 up, 8 in 2024-02-05T06:52:22.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:22 smithi167 ceph-mon[32774]: 1.c scrub starts 2024-02-05T06:52:22.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:22 smithi167 ceph-mon[32774]: 1.c scrub ok 2024-02-05T06:52:22.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:22 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:52:22.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:22 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:52:22.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:22 smithi167 ceph-mon[32774]: 1.f scrub starts 2024-02-05T06:52:22.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:22 smithi167 ceph-mon[32774]: 1.f scrub ok 2024-02-05T06:52:22.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:22 smithi167 ceph-mon[32774]: 1.5 scrub starts 2024-02-05T06:52:22.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:22 smithi167 ceph-mon[32774]: 1.5 scrub ok 2024-02-05T06:52:22.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:22 smithi167 ceph-mon[32774]: 1.12 scrub starts 2024-02-05T06:52:22.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:22 smithi167 ceph-mon[32774]: 1.12 scrub ok 2024-02-05T06:52:22.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:22 smithi167 ceph-mon[32774]: 1.b scrub starts 2024-02-05T06:52:22.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:22 smithi167 ceph-mon[32774]: 1.b scrub ok 2024-02-05T06:52:22.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:22 smithi167 ceph-mon[32774]: 1.2 scrub starts 2024-02-05T06:52:22.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:22 smithi167 ceph-mon[32774]: 1.2 scrub ok 2024-02-05T06:52:22.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:22 smithi167 ceph-mon[32774]: 1.11 scrub starts 2024-02-05T06:52:22.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:22 smithi167 ceph-mon[32774]: 1.11 scrub ok 2024-02-05T06:52:22.987 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:22 smithi167 ceph-mon[32774]: osdmap e440: 8 total, 8 up, 8 in 2024-02-05T06:52:23.015 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:23.045 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:23.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:23.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:23.148 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:23.177 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:23.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:23.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:23.279 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:23.337 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:23.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:23.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:23.348 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 62 16 95663 95647 96.3972 71.9375 0.0194682 0.0103692 2024-02-05T06:52:23.442 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:23.477 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:23.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:23.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:23.579 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:23.608 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:23.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:23.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:23.710 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:23.739 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:23.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:23.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:23.841 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:23.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:23.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:23.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:23 smithi028 ceph-mon[29442]: pgmap v907: 168 pgs: 168 active+clean; 5.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s rd, 103 MiB/s wr, 1.64k op/s 2024-02-05T06:52:23.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:23 smithi028 ceph-mon[29442]: 5.1f scrub starts 2024-02-05T06:52:23.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:23 smithi028 ceph-mon[29442]: 5.1f scrub ok 2024-02-05T06:52:23.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:23 smithi028 ceph-mon[29442]: 3.16 scrub starts 2024-02-05T06:52:23.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:23 smithi028 ceph-mon[29442]: 3.16 scrub ok 2024-02-05T06:52:23.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:23 smithi028 ceph-mon[29442]: 1.3 scrub starts 2024-02-05T06:52:23.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:23 smithi028 ceph-mon[29442]: 1.3 scrub ok 2024-02-05T06:52:23.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:23 smithi028 ceph-mon[29442]: 1.0 scrub starts 2024-02-05T06:52:23.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:23 smithi028 ceph-mon[29442]: 1.0 scrub ok 2024-02-05T06:52:23.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:23 smithi028 ceph-mon[37984]: pgmap v907: 168 pgs: 168 active+clean; 5.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s rd, 103 MiB/s wr, 1.64k op/s 2024-02-05T06:52:23.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:23 smithi028 ceph-mon[37984]: 5.1f scrub starts 2024-02-05T06:52:23.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:23 smithi028 ceph-mon[37984]: 5.1f scrub ok 2024-02-05T06:52:23.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:23 smithi028 ceph-mon[37984]: 3.16 scrub starts 2024-02-05T06:52:23.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:23 smithi028 ceph-mon[37984]: 3.16 scrub ok 2024-02-05T06:52:23.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:23 smithi028 ceph-mon[37984]: 1.3 scrub starts 2024-02-05T06:52:23.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:23 smithi028 ceph-mon[37984]: 1.3 scrub ok 2024-02-05T06:52:23.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:23 smithi028 ceph-mon[37984]: 1.0 scrub starts 2024-02-05T06:52:23.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:23 smithi028 ceph-mon[37984]: 1.0 scrub ok 2024-02-05T06:52:23.982 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:23.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:23 smithi167 ceph-mon[32774]: pgmap v907: 168 pgs: 168 active+clean; 5.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 0 B/s rd, 103 MiB/s wr, 1.64k op/s 2024-02-05T06:52:23.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:23 smithi167 ceph-mon[32774]: 5.1f scrub starts 2024-02-05T06:52:23.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:23 smithi167 ceph-mon[32774]: 5.1f scrub ok 2024-02-05T06:52:23.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:23 smithi167 ceph-mon[32774]: 3.16 scrub starts 2024-02-05T06:52:23.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:23 smithi167 ceph-mon[32774]: 3.16 scrub ok 2024-02-05T06:52:23.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:23 smithi167 ceph-mon[32774]: 1.3 scrub starts 2024-02-05T06:52:23.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:23 smithi167 ceph-mon[32774]: 1.3 scrub ok 2024-02-05T06:52:23.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:23 smithi167 ceph-mon[32774]: 1.0 scrub starts 2024-02-05T06:52:23.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:23 smithi167 ceph-mon[32774]: 1.0 scrub ok 2024-02-05T06:52:23.986 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:52:23 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:52:24.025 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:24.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:24.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:24.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:24.167 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:24.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:24.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:24.268 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:24.295 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:24.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:24.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:24.348 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 63 16 96703 96687 95.8988 65 0.0128409 0.0104224 2024-02-05T06:52:24.395 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:52:24.396 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:52:24.396 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:52:24.396 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:52:24.397 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:24.422 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:52:24.433 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:24.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:24.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:24.461 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:52:24.512 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:52:24.534 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:24.550 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:52:24.554 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:52:24.554 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:52:24.554 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:52:24.554 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:52:24.554 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:52:24.555 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:52:24.555 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:52:24.555 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:52:24.585 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:24.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:24.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:24.676 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.1 to 1 (10000) 2024-02-05T06:52:24.688 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:24.722 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:24.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:24.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:24.825 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:24.827 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:24 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:52:24.827 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:24 smithi028 ceph-mon[37984]: 4.1f scrub starts 2024-02-05T06:52:24.827 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:24 smithi028 ceph-mon[37984]: 4.1f scrub ok 2024-02-05T06:52:24.827 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:24 smithi028 ceph-mon[37984]: 1.d scrub starts 2024-02-05T06:52:24.827 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:24 smithi028 ceph-mon[37984]: 1.d scrub ok 2024-02-05T06:52:24.827 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:24 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3050602408' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T06:52:24.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:24 smithi028 ceph-mon[37984]: from='client.45932 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T06:52:24.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:24 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3556564022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:52:24.828 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:24 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:52:24.829 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:24 smithi028 ceph-mon[29442]: 4.1f scrub starts 2024-02-05T06:52:24.829 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:24 smithi028 ceph-mon[29442]: 4.1f scrub ok 2024-02-05T06:52:24.829 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:24 smithi028 ceph-mon[29442]: 1.d scrub starts 2024-02-05T06:52:24.829 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:24 smithi028 ceph-mon[29442]: 1.d scrub ok 2024-02-05T06:52:24.829 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:24 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3050602408' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T06:52:24.829 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:24 smithi028 ceph-mon[29442]: from='client.45932 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T06:52:24.829 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:24 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3556564022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:52:24.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:24.875 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:24.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:24.976 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:24.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:24 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:52:24.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:24 smithi167 ceph-mon[32774]: 4.1f scrub starts 2024-02-05T06:52:24.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:24 smithi167 ceph-mon[32774]: 4.1f scrub ok 2024-02-05T06:52:24.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:24 smithi167 ceph-mon[32774]: 1.d scrub starts 2024-02-05T06:52:24.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:24 smithi167 ceph-mon[32774]: 1.d scrub ok 2024-02-05T06:52:24.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:24 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3050602408' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T06:52:24.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:24 smithi167 ceph-mon[32774]: from='client.45932 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T06:52:24.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:24 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3556564022' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:52:25.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:25.017 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:25.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:25.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:25.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:25.159 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:25.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:25.260 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:25.293 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:25.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:25.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:25.348 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 64 16 97855 97839 95.5254 72 0.0168413 0.0104637 2024-02-05T06:52:25.395 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:25.435 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:25.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:25.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:25.538 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:25.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:25.573 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:25.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:25.674 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:25.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:25.705 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:25.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:25.806 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:25.843 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:25.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:25.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:25.945 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:25.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:25 smithi028 ceph-mon[29442]: pgmap v908: 168 pgs: 168 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-02-05T06:52:25.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:25 smithi028 ceph-mon[29442]: from='client.45932 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T06:52:25.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:25 smithi028 ceph-mon[29442]: osdmap e441: 8 total, 8 up, 8 in 2024-02-05T06:52:25.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:25 smithi028 ceph-mon[29442]: 5.d scrub starts 2024-02-05T06:52:25.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:25 smithi028 ceph-mon[29442]: 5.d scrub ok 2024-02-05T06:52:25.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:25 smithi028 ceph-mon[29442]: 5.16 scrub starts 2024-02-05T06:52:25.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:25 smithi028 ceph-mon[29442]: 5.16 scrub ok 2024-02-05T06:52:25.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:25 smithi028 ceph-mon[29442]: 1.9 scrub starts 2024-02-05T06:52:25.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:25 smithi028 ceph-mon[29442]: 1.9 scrub ok 2024-02-05T06:52:25.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:25 smithi028 ceph-mon[37984]: pgmap v908: 168 pgs: 168 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-02-05T06:52:25.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:25 smithi028 ceph-mon[37984]: from='client.45932 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T06:52:25.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:25 smithi028 ceph-mon[37984]: osdmap e441: 8 total, 8 up, 8 in 2024-02-05T06:52:25.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:25 smithi028 ceph-mon[37984]: 5.d scrub starts 2024-02-05T06:52:25.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:25 smithi028 ceph-mon[37984]: 5.d scrub ok 2024-02-05T06:52:25.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:25 smithi028 ceph-mon[37984]: 5.16 scrub starts 2024-02-05T06:52:25.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:25 smithi028 ceph-mon[37984]: 5.16 scrub ok 2024-02-05T06:52:25.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:25 smithi028 ceph-mon[37984]: 1.9 scrub starts 2024-02-05T06:52:25.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:25 smithi028 ceph-mon[37984]: 1.9 scrub ok 2024-02-05T06:52:25.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:25 smithi167 ceph-mon[32774]: pgmap v908: 168 pgs: 168 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 67 MiB/s wr, 1.07k op/s 2024-02-05T06:52:25.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:25 smithi167 ceph-mon[32774]: from='client.45932 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T06:52:25.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:25 smithi167 ceph-mon[32774]: osdmap e441: 8 total, 8 up, 8 in 2024-02-05T06:52:25.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:25 smithi167 ceph-mon[32774]: 5.d scrub starts 2024-02-05T06:52:25.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:25 smithi167 ceph-mon[32774]: 5.d scrub ok 2024-02-05T06:52:25.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:25 smithi167 ceph-mon[32774]: 5.16 scrub starts 2024-02-05T06:52:25.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:25 smithi167 ceph-mon[32774]: 5.16 scrub ok 2024-02-05T06:52:25.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:25 smithi167 ceph-mon[32774]: 1.9 scrub starts 2024-02-05T06:52:25.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:25 smithi167 ceph-mon[32774]: 1.9 scrub ok 2024-02-05T06:52:25.988 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:25.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:25.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:26.089 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:26.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:26.126 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:26.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:26.204 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd reweight 2 1 2024-02-05T06:52:26.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:26.323 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:26.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:26.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:26.348 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 65 16 99046 99030 95.2009 74.4375 0.01784 0.0104986 2024-02-05T06:52:26.426 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:26.477 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:26.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:26.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:26.579 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:26.627 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:26.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:26.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:26.648 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:52:26.729 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:26.794 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:26.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:26.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:26.896 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:26.928 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:26.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:26.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:27.030 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:27.067 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:27.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:27.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:27.169 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:27.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:27.210 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:27.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:27.311 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:27.349 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 66 16 100186 100170 94.8366 71.25 0.00586335 0.0105392 2024-02-05T06:52:27.352 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:27.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:27.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:27.454 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:27.483 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:27.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:27.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:27.585 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:27.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:27.620 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:27.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:27.721 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:27.752 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:52:27.752 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:52:27.774 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:27.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:27.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:27.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:27 smithi028 ceph-mon[29442]: 5.17 scrub starts 2024-02-05T06:52:27.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:27 smithi028 ceph-mon[29442]: 5.17 scrub ok 2024-02-05T06:52:27.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:27 smithi028 ceph-mon[29442]: pgmap v910: 168 pgs: 168 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 62 MiB/s wr, 995 op/s 2024-02-05T06:52:27.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:27 smithi028 ceph-mon[29442]: 6.a scrub starts 2024-02-05T06:52:27.827 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:27 smithi028 ceph-mon[29442]: 6.a scrub ok 2024-02-05T06:52:27.828 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:27 smithi028 ceph-mon[29442]: 4.9 scrub starts 2024-02-05T06:52:27.828 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:27 smithi028 ceph-mon[29442]: 4.9 scrub ok 2024-02-05T06:52:27.828 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:27 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:52:27.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:27 smithi028 ceph-mon[37984]: 5.17 scrub starts 2024-02-05T06:52:27.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:27 smithi028 ceph-mon[37984]: 5.17 scrub ok 2024-02-05T06:52:27.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:27 smithi028 ceph-mon[37984]: pgmap v910: 168 pgs: 168 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 62 MiB/s wr, 995 op/s 2024-02-05T06:52:27.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:27 smithi028 ceph-mon[37984]: 6.a scrub starts 2024-02-05T06:52:27.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:27 smithi028 ceph-mon[37984]: 6.a scrub ok 2024-02-05T06:52:27.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:27 smithi028 ceph-mon[37984]: 4.9 scrub starts 2024-02-05T06:52:27.828 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:27 smithi028 ceph-mon[37984]: 4.9 scrub ok 2024-02-05T06:52:27.829 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:27 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:52:27.876 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:27.908 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:27.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:27.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:27.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:27 smithi167 ceph-mon[32774]: 5.17 scrub starts 2024-02-05T06:52:27.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:27 smithi167 ceph-mon[32774]: 5.17 scrub ok 2024-02-05T06:52:27.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:27 smithi167 ceph-mon[32774]: pgmap v910: 168 pgs: 168 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 62 MiB/s wr, 995 op/s 2024-02-05T06:52:27.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:27 smithi167 ceph-mon[32774]: 6.a scrub starts 2024-02-05T06:52:27.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:27 smithi167 ceph-mon[32774]: 6.a scrub ok 2024-02-05T06:52:27.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:27 smithi167 ceph-mon[32774]: 4.9 scrub starts 2024-02-05T06:52:27.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:27 smithi167 ceph-mon[32774]: 4.9 scrub ok 2024-02-05T06:52:27.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:27 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:52:28.010 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:28.048 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:28.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:28.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:28.151 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:28.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:28.193 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:28.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:28.294 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:28.295 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:52:28.341 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:28.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:28.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:28.350 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 67 16 101323 101307 94.4804 71.0625 0.00652278 0.010579 2024-02-05T06:52:28.443 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:28.480 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:28.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:28.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:28.583 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:28.622 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:28.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:28.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:28.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:52:28.724 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:28.752 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:28.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:28.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:28.854 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:28.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:28.884 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:28.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:28.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-02-05T06:52:28.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:28 smithi028 ceph-mon[29442]: 4.19 scrub starts 2024-02-05T06:52:28.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:28 smithi028 ceph-mon[29442]: 4.19 scrub ok 2024-02-05T06:52:28.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:28 smithi028 ceph-mon[37984]: 4.19 scrub starts 2024-02-05T06:52:28.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:28 smithi028 ceph-mon[37984]: 4.19 scrub ok 2024-02-05T06:52:28.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:28 smithi167 ceph-mon[32774]: 4.19 scrub starts 2024-02-05T06:52:28.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:28 smithi167 ceph-mon[32774]: 4.19 scrub ok 2024-02-05T06:52:28.985 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:29.011 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:29.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:29.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:29.113 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:29.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:29.194 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:29.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:29.295 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:29.327 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:29.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:29.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:29.351 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 68 16 102399 102383 94.0799 67.25 0.0219987 0.010625 2024-02-05T06:52:29.429 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:29.462 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:29.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:29.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:29.563 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:29.602 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:29.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:29.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:29.703 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:29.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:29.752 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:29.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:29.853 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:29.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:29.889 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:29.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:29.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:29 smithi028 ceph-mon[37984]: pgmap v911: 168 pgs: 168 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-02-05T06:52:29.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:29 smithi028 ceph-mon[37984]: 6.1a scrub starts 2024-02-05T06:52:29.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:29 smithi028 ceph-mon[37984]: 6.1a scrub ok 2024-02-05T06:52:29.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:29 smithi028 ceph-mon[37984]: 6.1d scrub starts 2024-02-05T06:52:29.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:29 smithi028 ceph-mon[37984]: 6.1d scrub ok 2024-02-05T06:52:29.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:29 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:52:29.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:29 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:52:29.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:29 smithi028 ceph-mon[29442]: pgmap v911: 168 pgs: 168 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-02-05T06:52:29.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:29 smithi028 ceph-mon[29442]: 6.1a scrub starts 2024-02-05T06:52:29.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:29 smithi028 ceph-mon[29442]: 6.1a scrub ok 2024-02-05T06:52:29.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:29 smithi028 ceph-mon[29442]: 6.1d scrub starts 2024-02-05T06:52:29.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:29 smithi028 ceph-mon[29442]: 6.1d scrub ok 2024-02-05T06:52:29.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:29 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:52:29.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:29 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:52:29.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:29 smithi167 ceph-mon[32774]: pgmap v911: 168 pgs: 168 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 90 MiB/s wr, 1.44k op/s 2024-02-05T06:52:29.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:29 smithi167 ceph-mon[32774]: 6.1a scrub starts 2024-02-05T06:52:29.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:29 smithi167 ceph-mon[32774]: 6.1a scrub ok 2024-02-05T06:52:29.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:29 smithi167 ceph-mon[32774]: 6.1d scrub starts 2024-02-05T06:52:29.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:29 smithi167 ceph-mon[32774]: 6.1d scrub ok 2024-02-05T06:52:29.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:29 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:52:29.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:29 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:52:29.991 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:30.024 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:30.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:30.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:30.126 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:30.168 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:30.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:30.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:30.270 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:30.302 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:30.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:30.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:30.354 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 69 16 103596 103580 93.7956 74.8125 0.0231456 0.0106546 2024-02-05T06:52:30.404 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:30.441 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:30.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:30.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:30.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:30.589 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:30.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:30.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:30.691 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:30.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:30.722 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:30.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:30.823 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:30.834 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:30 smithi028 ceph-mon[37984]: osdmap e442: 8 total, 8 up, 8 in 2024-02-05T06:52:30.834 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:30 smithi028 ceph-mon[37984]: 5.4 scrub starts 2024-02-05T06:52:30.834 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:30 smithi028 ceph-mon[37984]: 5.4 scrub ok 2024-02-05T06:52:30.834 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:30 smithi028 ceph-mon[37984]: 6.16 scrub starts 2024-02-05T06:52:30.834 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:30 smithi028 ceph-mon[37984]: osdmap e443: 8 total, 8 up, 8 in 2024-02-05T06:52:30.834 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:30 smithi028 ceph-mon[37984]: 6.16 scrub ok 2024-02-05T06:52:30.834 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:30 smithi028 ceph-mon[29442]: osdmap e442: 8 total, 8 up, 8 in 2024-02-05T06:52:30.834 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:30 smithi028 ceph-mon[29442]: 5.4 scrub starts 2024-02-05T06:52:30.834 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:30 smithi028 ceph-mon[29442]: 5.4 scrub ok 2024-02-05T06:52:30.835 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:30 smithi028 ceph-mon[29442]: 6.16 scrub starts 2024-02-05T06:52:30.835 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:30 smithi028 ceph-mon[29442]: osdmap e443: 8 total, 8 up, 8 in 2024-02-05T06:52:30.835 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:30 smithi028 ceph-mon[29442]: 6.16 scrub ok 2024-02-05T06:52:30.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:30.855 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:30.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:30.956 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:30.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:30 smithi167 ceph-mon[32774]: osdmap e442: 8 total, 8 up, 8 in 2024-02-05T06:52:30.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:30 smithi167 ceph-mon[32774]: 5.4 scrub starts 2024-02-05T06:52:30.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:30 smithi167 ceph-mon[32774]: 5.4 scrub ok 2024-02-05T06:52:30.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:30 smithi167 ceph-mon[32774]: 6.16 scrub starts 2024-02-05T06:52:30.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:30 smithi167 ceph-mon[32774]: osdmap e443: 8 total, 8 up, 8 in 2024-02-05T06:52:30.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:30 smithi167 ceph-mon[32774]: 6.16 scrub ok 2024-02-05T06:52:30.991 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:30.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:30.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:31.093 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:31.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:31.162 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:31.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:31.263 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:31.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:31.301 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:31.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:31.354 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 70 15 104753 104738 93.4896 72.375 0.0194046 0.0106914 2024-02-05T06:52:31.402 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:31.446 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:31.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:31.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:31.548 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:31.576 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:31.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:31.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:31.678 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:31.717 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:31.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:31.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:31.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:31.857 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:31.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:31.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:31.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:31 smithi028 ceph-mon[37984]: 3.8 scrub starts 2024-02-05T06:52:31.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:31 smithi028 ceph-mon[37984]: 3.8 scrub ok 2024-02-05T06:52:31.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:31 smithi028 ceph-mon[37984]: pgmap v914: 168 pgs: 168 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 59 MiB/s wr, 938 op/s 2024-02-05T06:52:31.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:31 smithi028 ceph-mon[29442]: 3.8 scrub starts 2024-02-05T06:52:31.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:31 smithi028 ceph-mon[29442]: 3.8 scrub ok 2024-02-05T06:52:31.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:31 smithi028 ceph-mon[29442]: pgmap v914: 168 pgs: 168 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 59 MiB/s wr, 938 op/s 2024-02-05T06:52:31.960 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:31.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:31 smithi167 ceph-mon[32774]: 3.8 scrub starts 2024-02-05T06:52:31.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:31 smithi167 ceph-mon[32774]: 3.8 scrub ok 2024-02-05T06:52:31.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:31 smithi167 ceph-mon[32774]: pgmap v914: 168 pgs: 168 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 59 MiB/s wr, 938 op/s 2024-02-05T06:52:31.992 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:31.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:31.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:32.094 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:32.136 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:32.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:32.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:32.238 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:32.272 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:32.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:32.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:32.354 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 71 16 105975 105959 93.2477 76.3125 0.00679562 0.0107192 2024-02-05T06:52:32.374 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:32.380 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:52:32.380 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:52:32.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:32.446 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:32.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:32.547 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:32.576 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:32.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:32.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:32.678 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:32.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:52:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:52:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:52:32.711 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:32.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:32.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:32.813 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:32.845 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:32.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:32.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:32.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:32.979 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:32.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:32.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:32.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:32 smithi167 ceph-mon[32774]: 4.1a scrub starts 2024-02-05T06:52:32.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:32 smithi167 ceph-mon[32774]: 4.1a scrub ok 2024-02-05T06:52:32.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:32 smithi167 ceph-mon[32774]: 2.1 scrub starts 2024-02-05T06:52:32.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:32 smithi167 ceph-mon[32774]: 2.1 scrub ok 2024-02-05T06:52:32.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:32 smithi167 ceph-mon[32774]: 4.15 scrub starts 2024-02-05T06:52:32.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:32 smithi167 ceph-mon[32774]: 4.15 scrub ok 2024-02-05T06:52:33.081 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:33.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:33.117 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:33.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:33.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:32 smithi028 ceph-mon[37984]: 4.1a scrub starts 2024-02-05T06:52:33.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:32 smithi028 ceph-mon[37984]: 4.1a scrub ok 2024-02-05T06:52:33.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:32 smithi028 ceph-mon[37984]: 2.1 scrub starts 2024-02-05T06:52:33.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:32 smithi028 ceph-mon[37984]: 2.1 scrub ok 2024-02-05T06:52:33.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:32 smithi028 ceph-mon[37984]: 4.15 scrub starts 2024-02-05T06:52:33.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:32 smithi028 ceph-mon[37984]: 4.15 scrub ok 2024-02-05T06:52:33.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:32 smithi028 ceph-mon[29442]: 4.1a scrub starts 2024-02-05T06:52:33.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:32 smithi028 ceph-mon[29442]: 4.1a scrub ok 2024-02-05T06:52:33.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:32 smithi028 ceph-mon[29442]: 2.1 scrub starts 2024-02-05T06:52:33.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:32 smithi028 ceph-mon[29442]: 2.1 scrub ok 2024-02-05T06:52:33.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:32 smithi028 ceph-mon[29442]: 4.15 scrub starts 2024-02-05T06:52:33.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:32 smithi028 ceph-mon[29442]: 4.15 scrub ok 2024-02-05T06:52:33.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:33.243 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:33.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:33.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:33.345 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:33.355 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 72 16 107059 107043 92.8928 67.75 0.0122975 0.01076 2024-02-05T06:52:33.390 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:33.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:33.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:33.492 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:33.522 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:33.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:33.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:33.623 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:33.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:33.665 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:33.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:33.729 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.2 to 1 (10000) 2024-02-05T06:52:33.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:33.808 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:33.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:33.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:33.910 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:33.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:33.945 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:33.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:33.985 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:52:33 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:52:33.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:33 smithi167 ceph-mon[32774]: pgmap v915: 168 pgs: 168 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-02-05T06:52:33.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:33 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1841479075' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T06:52:33.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:33 smithi167 ceph-mon[32774]: from='client.25924 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T06:52:33.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:33 smithi167 ceph-mon[32774]: 3.c scrub starts 2024-02-05T06:52:33.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:33 smithi167 ceph-mon[32774]: 3.c scrub ok 2024-02-05T06:52:33.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:33 smithi167 ceph-mon[32774]: 5.9 scrub starts 2024-02-05T06:52:33.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:33 smithi167 ceph-mon[32774]: 5.9 scrub ok 2024-02-05T06:52:34.046 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:34.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:34.088 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:34.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:34.189 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:34.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:33 smithi028 ceph-mon[29442]: pgmap v915: 168 pgs: 168 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-02-05T06:52:34.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:33 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1841479075' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T06:52:34.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:33 smithi028 ceph-mon[29442]: from='client.25924 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T06:52:34.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:33 smithi028 ceph-mon[29442]: 3.c scrub starts 2024-02-05T06:52:34.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:33 smithi028 ceph-mon[29442]: 3.c scrub ok 2024-02-05T06:52:34.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:33 smithi028 ceph-mon[29442]: 5.9 scrub starts 2024-02-05T06:52:34.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:33 smithi028 ceph-mon[29442]: 5.9 scrub ok 2024-02-05T06:52:34.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:33 smithi028 ceph-mon[37984]: pgmap v915: 168 pgs: 168 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 0 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-02-05T06:52:34.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:33 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1841479075' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T06:52:34.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:33 smithi028 ceph-mon[37984]: from='client.25924 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T06:52:34.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:33 smithi028 ceph-mon[37984]: 3.c scrub starts 2024-02-05T06:52:34.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:33 smithi028 ceph-mon[37984]: 3.c scrub ok 2024-02-05T06:52:34.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:33 smithi028 ceph-mon[37984]: 5.9 scrub starts 2024-02-05T06:52:34.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:33 smithi028 ceph-mon[37984]: 5.9 scrub ok 2024-02-05T06:52:34.221 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:34.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:34.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:34.323 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:34.355 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 73 16 108177 108161 92.5775 69.875 0.0142347 0.0107974 2024-02-05T06:52:34.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:34.356 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:34.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:34.457 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:34.497 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:34.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:34.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:34.599 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:34.640 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:34.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:34.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:34.742 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:34.761 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-02-05T06:52:34.780 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:34.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:34.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:34.882 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:34.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:34.914 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:34.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:35.015 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:35.055 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:35.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:35.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:35.157 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:35.191 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:35.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:35.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:35.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:34 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:52:35.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:34 smithi028 ceph-mon[37984]: from='client.25924 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T06:52:35.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:34 smithi028 ceph-mon[37984]: osdmap e444: 8 total, 8 up, 8 in 2024-02-05T06:52:35.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:34 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2537914747' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:52:35.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:34 smithi028 ceph-mon[37984]: from='client.45947 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:52:35.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:34 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:52:35.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:34 smithi028 ceph-mon[29442]: from='client.25924 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T06:52:35.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:34 smithi028 ceph-mon[29442]: osdmap e444: 8 total, 8 up, 8 in 2024-02-05T06:52:35.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:34 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2537914747' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:52:35.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:34 smithi028 ceph-mon[29442]: from='client.45947 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:52:35.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:34 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:52:35.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:34 smithi167 ceph-mon[32774]: from='client.25924 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T06:52:35.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:34 smithi167 ceph-mon[32774]: osdmap e444: 8 total, 8 up, 8 in 2024-02-05T06:52:35.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:34 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2537914747' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:52:35.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:34 smithi167 ceph-mon[32774]: from='client.45947 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:52:35.287 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd reweight 3 1 2024-02-05T06:52:35.292 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:35.355 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 74 16 109366 109350 92.3306 74.3125 0.00645004 0.0108255 2024-02-05T06:52:35.396 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:35.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:35.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:35.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:35.556 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:35.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:35.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:35.658 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:35.708 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:35.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:35.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:35.810 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:35.839 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:35.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:35.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:35.941 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:35.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:35 smithi028 ceph-mon[37984]: pgmap v917: 168 pgs: 168 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 61 MiB/s wr, 969 op/s 2024-02-05T06:52:35.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:35 smithi028 ceph-mon[37984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:52:35.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:35 smithi028 ceph-mon[37984]: Cluster is now healthy 2024-02-05T06:52:35.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:35 smithi028 ceph-mon[37984]: from='client.45947 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:52:35.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:35 smithi028 ceph-mon[37984]: osdmap e445: 8 total, 8 up, 8 in 2024-02-05T06:52:35.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:35 smithi028 ceph-mon[37984]: 5.6 deep-scrub starts 2024-02-05T06:52:35.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:35 smithi028 ceph-mon[37984]: 5.6 deep-scrub ok 2024-02-05T06:52:35.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:35 smithi028 ceph-mon[37984]: 2.0 scrub starts 2024-02-05T06:52:35.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:35 smithi028 ceph-mon[37984]: 2.0 scrub ok 2024-02-05T06:52:35.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:35 smithi028 ceph-mon[29442]: pgmap v917: 168 pgs: 168 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 61 MiB/s wr, 969 op/s 2024-02-05T06:52:35.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:35 smithi028 ceph-mon[29442]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:52:35.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:35 smithi028 ceph-mon[29442]: Cluster is now healthy 2024-02-05T06:52:35.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:35 smithi028 ceph-mon[29442]: from='client.45947 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:52:35.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:35 smithi028 ceph-mon[29442]: osdmap e445: 8 total, 8 up, 8 in 2024-02-05T06:52:35.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:35 smithi028 ceph-mon[29442]: 5.6 deep-scrub starts 2024-02-05T06:52:35.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:35 smithi028 ceph-mon[29442]: 5.6 deep-scrub ok 2024-02-05T06:52:35.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:35 smithi028 ceph-mon[29442]: 2.0 scrub starts 2024-02-05T06:52:35.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:35 smithi028 ceph-mon[29442]: 2.0 scrub ok 2024-02-05T06:52:35.971 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:35.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:35.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:36.073 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:36.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:36.105 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:36.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:36.206 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:36.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:35 smithi167 ceph-mon[32774]: pgmap v917: 168 pgs: 168 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 61 MiB/s wr, 969 op/s 2024-02-05T06:52:36.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:35 smithi167 ceph-mon[32774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:52:36.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:35 smithi167 ceph-mon[32774]: Cluster is now healthy 2024-02-05T06:52:36.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:35 smithi167 ceph-mon[32774]: from='client.45947 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:52:36.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:35 smithi167 ceph-mon[32774]: osdmap e445: 8 total, 8 up, 8 in 2024-02-05T06:52:36.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:35 smithi167 ceph-mon[32774]: 5.6 deep-scrub starts 2024-02-05T06:52:36.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:35 smithi167 ceph-mon[32774]: 5.6 deep-scrub ok 2024-02-05T06:52:36.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:35 smithi167 ceph-mon[32774]: 2.0 scrub starts 2024-02-05T06:52:36.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:35 smithi167 ceph-mon[32774]: 2.0 scrub ok 2024-02-05T06:52:36.237 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:36.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:36.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:36.339 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:36.355 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 75 16 110568 110552 92.1012 75.125 0.00691975 0.0108532 2024-02-05T06:52:36.369 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:36.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:36.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:36.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:36.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:36.508 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:36.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:36.609 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:36.666 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:36.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:36.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:36.768 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:36.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:36.807 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:36.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:36.908 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:36.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:36 smithi028 ceph-mon[29442]: 5.b scrub starts 2024-02-05T06:52:36.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:36 smithi028 ceph-mon[29442]: 5.b scrub ok 2024-02-05T06:52:36.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:36 smithi028 ceph-mon[37984]: 5.b scrub starts 2024-02-05T06:52:36.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:36 smithi028 ceph-mon[37984]: 5.b scrub ok 2024-02-05T06:52:36.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:36.954 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:36.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:37.056 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:37.088 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:37.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:37.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:37.190 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:37.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:36 smithi167 ceph-mon[32774]: 5.b scrub starts 2024-02-05T06:52:37.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:36 smithi167 ceph-mon[32774]: 5.b scrub ok 2024-02-05T06:52:37.235 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:37.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:37.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:37.338 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:37.355 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 76 16 111797 111781 91.8999 76.8125 0.00871651 0.0108771 2024-02-05T06:52:37.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:37.376 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:37.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:37.477 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:37.511 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:37.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:37.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:37.612 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:37.656 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:37.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:37.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:37.758 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:37.791 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:37.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:37.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:37.892 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:37.931 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:37.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:37.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:37.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:37 smithi028 ceph-mon[29442]: pgmap v919: 168 pgs: 168 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 813 B/s rd, 61 MiB/s wr, 979 op/s 2024-02-05T06:52:37.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:37 smithi028 ceph-mon[29442]: 3.4 deep-scrub starts 2024-02-05T06:52:37.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:37 smithi028 ceph-mon[29442]: 3.4 deep-scrub ok 2024-02-05T06:52:37.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:37 smithi028 ceph-mon[29442]: 5.1a deep-scrub starts 2024-02-05T06:52:37.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:37 smithi028 ceph-mon[29442]: 5.1a deep-scrub ok 2024-02-05T06:52:37.971 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:37 smithi028 ceph-mon[37984]: pgmap v919: 168 pgs: 168 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 813 B/s rd, 61 MiB/s wr, 979 op/s 2024-02-05T06:52:37.971 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:37 smithi028 ceph-mon[37984]: 3.4 deep-scrub starts 2024-02-05T06:52:37.971 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:37 smithi028 ceph-mon[37984]: 3.4 deep-scrub ok 2024-02-05T06:52:37.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:37 smithi028 ceph-mon[37984]: 5.1a deep-scrub starts 2024-02-05T06:52:37.972 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:37 smithi028 ceph-mon[37984]: 5.1a deep-scrub ok 2024-02-05T06:52:38.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:38.093 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:38.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:38.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:38.195 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:38.234 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:38.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:37 smithi167 ceph-mon[32774]: pgmap v919: 168 pgs: 168 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 813 B/s rd, 61 MiB/s wr, 979 op/s 2024-02-05T06:52:38.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:37 smithi167 ceph-mon[32774]: 3.4 deep-scrub starts 2024-02-05T06:52:38.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:37 smithi167 ceph-mon[32774]: 3.4 deep-scrub ok 2024-02-05T06:52:38.236 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:37 smithi167 ceph-mon[32774]: 5.1a deep-scrub starts 2024-02-05T06:52:38.236 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:37 smithi167 ceph-mon[32774]: 5.1a deep-scrub ok 2024-02-05T06:52:38.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:38.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:38.337 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:38.355 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 77 16 112995 112979 91.6787 74.875 0.0107109 0.0109033 2024-02-05T06:52:38.378 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:38.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:38.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:38.480 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:38.492 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:52:38.510 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:38.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:38.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:38.612 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:38.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:38.659 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:38.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:38.761 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:38.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:38.813 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:38.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:38.914 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:38.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:38.952 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:38.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:39.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:39.092 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:39.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:39.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:39.194 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:39.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:38 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:52:39.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:38 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:52:39.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:38 smithi028 ceph-mon[29442]: 4.1d deep-scrub starts 2024-02-05T06:52:39.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:38 smithi028 ceph-mon[29442]: 4.1d deep-scrub ok 2024-02-05T06:52:39.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:38 smithi028 ceph-mon[29442]: osdmap e446: 8 total, 8 up, 8 in 2024-02-05T06:52:39.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:38 smithi028 ceph-mon[29442]: 5.c deep-scrub starts 2024-02-05T06:52:39.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:38 smithi028 ceph-mon[29442]: 5.c deep-scrub ok 2024-02-05T06:52:39.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:38 smithi028 ceph-mon[29442]: 4.f scrub starts 2024-02-05T06:52:39.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:38 smithi028 ceph-mon[29442]: 4.f scrub ok 2024-02-05T06:52:39.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:38 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:52:39.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:38 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:52:39.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:38 smithi028 ceph-mon[37984]: 4.1d deep-scrub starts 2024-02-05T06:52:39.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:38 smithi028 ceph-mon[37984]: 4.1d deep-scrub ok 2024-02-05T06:52:39.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:38 smithi028 ceph-mon[37984]: osdmap e446: 8 total, 8 up, 8 in 2024-02-05T06:52:39.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:38 smithi028 ceph-mon[37984]: 5.c deep-scrub starts 2024-02-05T06:52:39.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:38 smithi028 ceph-mon[37984]: 5.c deep-scrub ok 2024-02-05T06:52:39.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:38 smithi028 ceph-mon[37984]: 4.f scrub starts 2024-02-05T06:52:39.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:38 smithi028 ceph-mon[37984]: 4.f scrub ok 2024-02-05T06:52:39.231 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:39.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:39.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:39.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:38 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:52:39.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:38 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:52:39.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:38 smithi167 ceph-mon[32774]: 4.1d deep-scrub starts 2024-02-05T06:52:39.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:38 smithi167 ceph-mon[32774]: 4.1d deep-scrub ok 2024-02-05T06:52:39.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:38 smithi167 ceph-mon[32774]: osdmap e446: 8 total, 8 up, 8 in 2024-02-05T06:52:39.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:38 smithi167 ceph-mon[32774]: 5.c deep-scrub starts 2024-02-05T06:52:39.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:38 smithi167 ceph-mon[32774]: 5.c deep-scrub ok 2024-02-05T06:52:39.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:38 smithi167 ceph-mon[32774]: 4.f scrub starts 2024-02-05T06:52:39.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:38 smithi167 ceph-mon[32774]: 4.f scrub ok 2024-02-05T06:52:39.333 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:39.355 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 78 16 114190 114174 91.4609 74.6875 0.0106391 0.010929 2024-02-05T06:52:39.371 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:39.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:39.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:39.435 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T06:52:39.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:39.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:39.534 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:39.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:39.635 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:39.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:39.678 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:39.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:39.779 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:39.831 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:39.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:39.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:39.933 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:39.964 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:39.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:39.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:40.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:39 smithi028 ceph-mon[29442]: pgmap v921: 168 pgs: 168 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-02-05T06:52:40.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:39 smithi028 ceph-mon[29442]: 3.f scrub starts 2024-02-05T06:52:40.061 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:39 smithi028 ceph-mon[29442]: 3.f scrub ok 2024-02-05T06:52:40.061 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:39 smithi028 ceph-mon[37984]: pgmap v921: 168 pgs: 168 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-02-05T06:52:40.062 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:39 smithi028 ceph-mon[37984]: 3.f scrub starts 2024-02-05T06:52:40.062 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:39 smithi028 ceph-mon[37984]: 3.f scrub ok 2024-02-05T06:52:40.066 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:40.099 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:40.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:40.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:40.201 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:40.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:39 smithi167 ceph-mon[32774]: pgmap v921: 168 pgs: 168 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-02-05T06:52:40.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:39 smithi167 ceph-mon[32774]: 3.f scrub starts 2024-02-05T06:52:40.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:39 smithi167 ceph-mon[32774]: 3.f scrub ok 2024-02-05T06:52:40.235 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:40.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:40.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:40.337 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:40.355 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 79 16 115228 115212 91.1243 64.875 0.00948797 0.0109648 2024-02-05T06:52:40.366 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:40.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:40.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:40.468 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:40.511 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:40.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:40.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:40.613 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:40.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:40.656 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:40.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:40.758 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:40.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:40.782 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:40.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:40.883 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:40.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:40.915 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:40.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:41.016 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:41.057 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:41.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:41.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:41.159 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:41.190 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:41.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:41.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:41.292 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:41.329 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:41.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:41.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:41.356 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:2024-02-05T06:52:41.348654+0000 min lat: 0.0012181 max lat: 0.204312 avg lat: 0.0110043 2024-02-05T06:52:41.356 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-02-05T06:52:41.356 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 80 15 116315 116300 90.8353 68 0.00906098 0.0110043 2024-02-05T06:52:41.432 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:41.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:41.466 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:41.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:41.567 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:41.596 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:52:41.596 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:52:41.602 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:41 smithi028 ceph-mon[29442]: 4.5 scrub starts 2024-02-05T06:52:41.602 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:41 smithi028 ceph-mon[29442]: 4.5 scrub ok 2024-02-05T06:52:41.602 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:41 smithi028 ceph-mon[29442]: 6.13 scrub starts 2024-02-05T06:52:41.603 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:41 smithi028 ceph-mon[29442]: 5.3 deep-scrub starts 2024-02-05T06:52:41.603 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:41 smithi028 ceph-mon[29442]: 5.3 deep-scrub ok 2024-02-05T06:52:41.603 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:41 smithi028 ceph-mon[29442]: osdmap e447: 8 total, 8 up, 8 in 2024-02-05T06:52:41.603 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:41 smithi028 ceph-mon[29442]: 6.13 scrub ok 2024-02-05T06:52:41.603 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:41 smithi028 ceph-mon[29442]: pgmap v923: 168 pgs: 168 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 61 MiB/s wr, 972 op/s 2024-02-05T06:52:41.603 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:41 smithi028 ceph-mon[29442]: 3.d scrub starts 2024-02-05T06:52:41.603 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:41 smithi028 ceph-mon[29442]: 3.d scrub ok 2024-02-05T06:52:41.603 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:41 smithi028 ceph-mon[37984]: 4.5 scrub starts 2024-02-05T06:52:41.603 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:41 smithi028 ceph-mon[37984]: 4.5 scrub ok 2024-02-05T06:52:41.603 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:41 smithi028 ceph-mon[37984]: 6.13 scrub starts 2024-02-05T06:52:41.603 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:41 smithi028 ceph-mon[37984]: 5.3 deep-scrub starts 2024-02-05T06:52:41.603 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:41 smithi028 ceph-mon[37984]: 5.3 deep-scrub ok 2024-02-05T06:52:41.604 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:41 smithi028 ceph-mon[37984]: osdmap e447: 8 total, 8 up, 8 in 2024-02-05T06:52:41.604 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:41 smithi028 ceph-mon[37984]: 6.13 scrub ok 2024-02-05T06:52:41.604 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:41 smithi028 ceph-mon[37984]: pgmap v923: 168 pgs: 168 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 61 MiB/s wr, 972 op/s 2024-02-05T06:52:41.604 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:41 smithi028 ceph-mon[37984]: 3.d scrub starts 2024-02-05T06:52:41.604 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:41 smithi028 ceph-mon[37984]: 3.d scrub ok 2024-02-05T06:52:41.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:41.612 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:41.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:41.713 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:41.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:41 smithi167 ceph-mon[32774]: 4.5 scrub starts 2024-02-05T06:52:41.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:41 smithi167 ceph-mon[32774]: 4.5 scrub ok 2024-02-05T06:52:41.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:41 smithi167 ceph-mon[32774]: 6.13 scrub starts 2024-02-05T06:52:41.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:41 smithi167 ceph-mon[32774]: 5.3 deep-scrub starts 2024-02-05T06:52:41.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:41 smithi167 ceph-mon[32774]: 5.3 deep-scrub ok 2024-02-05T06:52:41.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:41 smithi167 ceph-mon[32774]: osdmap e447: 8 total, 8 up, 8 in 2024-02-05T06:52:41.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:41 smithi167 ceph-mon[32774]: 6.13 scrub ok 2024-02-05T06:52:41.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:41 smithi167 ceph-mon[32774]: pgmap v923: 168 pgs: 168 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 853 B/s rd, 61 MiB/s wr, 972 op/s 2024-02-05T06:52:41.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:41 smithi167 ceph-mon[32774]: 3.d scrub starts 2024-02-05T06:52:41.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:41 smithi167 ceph-mon[32774]: 3.d scrub ok 2024-02-05T06:52:41.746 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:41.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:41.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:41.848 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:41.887 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:41.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:41.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:41.989 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:42.027 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:42.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:42.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:42.129 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:42.163 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:42.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:42.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:42.265 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:42.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:42.301 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:42.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:42.356 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 81 16 117471 117455 90.605 72.1875 0.00426446 0.0110321 2024-02-05T06:52:42.402 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:42.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:42.441 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:42.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:42.542 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:42.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:42.568 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:42.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:42.669 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:42.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:42 smithi028 ceph-mon[29442]: 2.2 scrub starts 2024-02-05T06:52:42.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:42 smithi028 ceph-mon[29442]: 2.2 scrub ok 2024-02-05T06:52:42.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:42 smithi028 ceph-mon[29442]: 4.d scrub starts 2024-02-05T06:52:42.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:42 smithi028 ceph-mon[29442]: 4.d scrub ok 2024-02-05T06:52:42.701 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:52:42 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:52:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:52:42.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:42 smithi028 ceph-mon[37984]: 2.2 scrub starts 2024-02-05T06:52:42.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:42 smithi028 ceph-mon[37984]: 2.2 scrub ok 2024-02-05T06:52:42.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:42 smithi028 ceph-mon[37984]: 4.d scrub starts 2024-02-05T06:52:42.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:42 smithi028 ceph-mon[37984]: 4.d scrub ok 2024-02-05T06:52:42.705 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:42.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:42.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:42.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:42 smithi167 ceph-mon[32774]: 2.2 scrub starts 2024-02-05T06:52:42.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:42 smithi167 ceph-mon[32774]: 2.2 scrub ok 2024-02-05T06:52:42.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:42 smithi167 ceph-mon[32774]: 4.d scrub starts 2024-02-05T06:52:42.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:42 smithi167 ceph-mon[32774]: 4.d scrub ok 2024-02-05T06:52:42.807 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:42.841 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:42.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:42.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:42.942 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:42.976 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:42.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:42.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:43.078 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:43.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:43.117 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:43.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:43.219 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:43.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:43.261 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:43.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:43.356 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 82 16 118732 118716 90.4611 78.8125 0.0107047 0.0110494 2024-02-05T06:52:43.362 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:43.397 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:43.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:43.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:43.499 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:43.531 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:43.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:43.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:43.580 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-02-05T06:52:43.633 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:43.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:43.686 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:43.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:43.735 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:52:43 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:52:43.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:43 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:52:43.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:43 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:52:43.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:43 smithi167 ceph-mon[32774]: pgmap v924: 168 pgs: 168 active+clean; 7.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.3 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-02-05T06:52:43.787 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:43.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:43.849 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:43.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:43.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:43 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:52:43.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:43 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:52:43.875 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:43 smithi028 ceph-mon[29442]: pgmap v924: 168 pgs: 168 active+clean; 7.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.3 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-02-05T06:52:43.876 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:43 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:52:43.876 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:43 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:52:43.876 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:43 smithi028 ceph-mon[37984]: pgmap v924: 168 pgs: 168 active+clean; 7.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.3 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-02-05T06:52:43.950 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:43.983 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:43.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:43.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:44.085 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:44.113 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:44.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:44.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:44.215 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:44.248 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:44.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:44.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:44.350 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:44.356 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 83 15 119856 119841 90.2183 70.3125 0.0192886 0.0110797 2024-02-05T06:52:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:44.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:44.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:44.492 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:44.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:44.530 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:44.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:44.631 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:44.664 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:44.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:44.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:44.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:44 smithi167 ceph-mon[32774]: 5.14 scrub starts 2024-02-05T06:52:44.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:44 smithi167 ceph-mon[32774]: 5.14 scrub ok 2024-02-05T06:52:44.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:44 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:52:44.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:44 smithi167 ceph-mon[32774]: 3.14 scrub starts 2024-02-05T06:52:44.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:44 smithi167 ceph-mon[32774]: 3.14 scrub ok 2024-02-05T06:52:44.757 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:44 smithi028 ceph-mon[37984]: 5.14 scrub starts 2024-02-05T06:52:44.757 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:44 smithi028 ceph-mon[37984]: 5.14 scrub ok 2024-02-05T06:52:44.757 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:44 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:52:44.757 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:44 smithi028 ceph-mon[37984]: 3.14 scrub starts 2024-02-05T06:52:44.758 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:44 smithi028 ceph-mon[37984]: 3.14 scrub ok 2024-02-05T06:52:44.758 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:44 smithi028 ceph-mon[29442]: 5.14 scrub starts 2024-02-05T06:52:44.758 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:44 smithi028 ceph-mon[29442]: 5.14 scrub ok 2024-02-05T06:52:44.758 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:44 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:52:44.758 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:44 smithi028 ceph-mon[29442]: 3.14 scrub starts 2024-02-05T06:52:44.758 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:44 smithi028 ceph-mon[29442]: 3.14 scrub ok 2024-02-05T06:52:44.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:44.802 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:44.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:44.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:44.904 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:44.934 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:44.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:44.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:45.035 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:45.063 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:45.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:45.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:45.165 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:45.199 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:45.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:45.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:45.232 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:52:45.301 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:45.356 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 84 16 121030 121014 90.017 73.3125 0.00841101 0.0111037 2024-02-05T06:52:45.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:45.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:45.459 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:45.486 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:45.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:45.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:45.588 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:45.630 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:45.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:45.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:45.687 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:52:45.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:45 smithi028 ceph-mon[37984]: 3.1c deep-scrub starts 2024-02-05T06:52:45.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:45 smithi028 ceph-mon[37984]: 3.1c deep-scrub ok 2024-02-05T06:52:45.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:45 smithi028 ceph-mon[37984]: 6.17 scrub starts 2024-02-05T06:52:45.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:45 smithi028 ceph-mon[37984]: 4.7 scrub starts 2024-02-05T06:52:45.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:45 smithi028 ceph-mon[37984]: 4.7 scrub ok 2024-02-05T06:52:45.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:45 smithi028 ceph-mon[37984]: 6.17 scrub ok 2024-02-05T06:52:45.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:45 smithi028 ceph-mon[37984]: pgmap v925: 168 pgs: 168 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-02-05T06:52:45.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:45 smithi028 ceph-mon[29442]: 3.1c deep-scrub starts 2024-02-05T06:52:45.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:45 smithi028 ceph-mon[29442]: 3.1c deep-scrub ok 2024-02-05T06:52:45.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:45 smithi028 ceph-mon[29442]: 6.17 scrub starts 2024-02-05T06:52:45.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:45 smithi028 ceph-mon[29442]: 4.7 scrub starts 2024-02-05T06:52:45.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:45 smithi028 ceph-mon[29442]: 4.7 scrub ok 2024-02-05T06:52:45.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:45 smithi028 ceph-mon[29442]: 6.17 scrub ok 2024-02-05T06:52:45.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:45 smithi028 ceph-mon[29442]: pgmap v925: 168 pgs: 168 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-02-05T06:52:45.730 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:52:45.732 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:45.767 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:45.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:45.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:45.773 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:52:45.818 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:52:45.859 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:52:45.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:45.905 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:52:45.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:45.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:45.957 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:52:45.968 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:52:45.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:45 smithi167 ceph-mon[32774]: 3.1c deep-scrub starts 2024-02-05T06:52:45.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:45 smithi167 ceph-mon[32774]: 3.1c deep-scrub ok 2024-02-05T06:52:45.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:45 smithi167 ceph-mon[32774]: 6.17 scrub starts 2024-02-05T06:52:45.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:45 smithi167 ceph-mon[32774]: 4.7 scrub starts 2024-02-05T06:52:45.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:45 smithi167 ceph-mon[32774]: 4.7 scrub ok 2024-02-05T06:52:45.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:45 smithi167 ceph-mon[32774]: 6.17 scrub ok 2024-02-05T06:52:45.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:45 smithi167 ceph-mon[32774]: pgmap v925: 168 pgs: 168 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-02-05T06:52:45.993 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:52:46.005 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:52:46.005 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:52:46.005 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:52:46.005 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:52:46.005 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:52:46.006 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:52:46.006 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:52:46.006 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:52:46.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:46.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:46.080 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:46.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:46.181 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:46.230 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:46.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:46.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:46.332 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:46.356 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 85 16 122044 122028 89.7036 63.375 0.0217708 0.0111427 2024-02-05T06:52:46.369 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:46.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:46.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:46.470 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:46.501 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:46.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:46.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:46.596 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:46 smithi028 ceph-mon[37984]: 6.14 deep-scrub starts 2024-02-05T06:52:46.596 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:46 smithi028 ceph-mon[37984]: 6.14 deep-scrub ok 2024-02-05T06:52:46.596 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:46 smithi028 ceph-mon[37984]: 4.12 scrub starts 2024-02-05T06:52:46.597 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:46 smithi028 ceph-mon[37984]: 4.12 scrub ok 2024-02-05T06:52:46.597 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:46 smithi028 ceph-mon[37984]: 4.c scrub starts 2024-02-05T06:52:46.597 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:46 smithi028 ceph-mon[37984]: 4.c scrub ok 2024-02-05T06:52:46.597 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:46 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3423436123' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:52:46.597 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:46 smithi028 ceph-mon[29442]: 6.14 deep-scrub starts 2024-02-05T06:52:46.597 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:46 smithi028 ceph-mon[29442]: 6.14 deep-scrub ok 2024-02-05T06:52:46.597 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:46 smithi028 ceph-mon[29442]: 4.12 scrub starts 2024-02-05T06:52:46.597 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:46 smithi028 ceph-mon[29442]: 4.12 scrub ok 2024-02-05T06:52:46.597 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:46 smithi028 ceph-mon[29442]: 4.c scrub starts 2024-02-05T06:52:46.597 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:46 smithi028 ceph-mon[29442]: 4.c scrub ok 2024-02-05T06:52:46.597 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:46 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3423436123' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:52:46.603 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:46.654 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:46.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:46.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:46.756 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:46.801 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:46.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:46.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:46.904 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:46.940 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:46.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:46.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:46.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:46 smithi167 ceph-mon[32774]: 6.14 deep-scrub starts 2024-02-05T06:52:46.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:46 smithi167 ceph-mon[32774]: 6.14 deep-scrub ok 2024-02-05T06:52:46.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:46 smithi167 ceph-mon[32774]: 4.12 scrub starts 2024-02-05T06:52:46.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:46 smithi167 ceph-mon[32774]: 4.12 scrub ok 2024-02-05T06:52:46.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:46 smithi167 ceph-mon[32774]: 4.c scrub starts 2024-02-05T06:52:46.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:46 smithi167 ceph-mon[32774]: 4.c scrub ok 2024-02-05T06:52:46.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:46 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3423436123' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:52:47.042 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:47.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:47.078 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:47.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:47.179 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:47.206 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:47.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:47.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:47.308 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:47.354 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:47.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:47.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:47.356 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 86 16 123118 123102 89.441 67.125 0.0162754 0.0111751 2024-02-05T06:52:47.456 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:47.500 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:47.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:47.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:47.520 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.3 to 1 (10000) 2024-02-05T06:52:47.602 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:47.632 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:47.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:47.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:47.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:47 smithi028 ceph-mon[29442]: 5.8 scrub starts 2024-02-05T06:52:47.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:47 smithi028 ceph-mon[29442]: 5.8 scrub ok 2024-02-05T06:52:47.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:47 smithi028 ceph-mon[29442]: 6.e scrub starts 2024-02-05T06:52:47.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:47 smithi028 ceph-mon[29442]: 6.e scrub ok 2024-02-05T06:52:47.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:47 smithi028 ceph-mon[29442]: 6.10 scrub starts 2024-02-05T06:52:47.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:47 smithi028 ceph-mon[29442]: 6.10 scrub ok 2024-02-05T06:52:47.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:47 smithi028 ceph-mon[29442]: pgmap v926: 168 pgs: 168 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 580 B/s rd, 53 MiB/s wr, 853 op/s 2024-02-05T06:52:47.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:47 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3649518536' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T06:52:47.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:47 smithi028 ceph-mon[29442]: from='client.25936 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T06:52:47.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:47 smithi028 ceph-mon[29442]: 6.8 scrub starts 2024-02-05T06:52:47.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:47 smithi028 ceph-mon[29442]: 6.8 scrub ok 2024-02-05T06:52:47.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:47 smithi028 ceph-mon[29442]: 3.3 deep-scrub starts 2024-02-05T06:52:47.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:47 smithi028 ceph-mon[29442]: 3.3 deep-scrub ok 2024-02-05T06:52:47.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:47 smithi028 ceph-mon[37984]: 5.8 scrub starts 2024-02-05T06:52:47.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:47 smithi028 ceph-mon[37984]: 5.8 scrub ok 2024-02-05T06:52:47.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:47 smithi028 ceph-mon[37984]: 6.e scrub starts 2024-02-05T06:52:47.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:47 smithi028 ceph-mon[37984]: 6.e scrub ok 2024-02-05T06:52:47.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:47 smithi028 ceph-mon[37984]: 6.10 scrub starts 2024-02-05T06:52:47.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:47 smithi028 ceph-mon[37984]: 6.10 scrub ok 2024-02-05T06:52:47.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:47 smithi028 ceph-mon[37984]: pgmap v926: 168 pgs: 168 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 580 B/s rd, 53 MiB/s wr, 853 op/s 2024-02-05T06:52:47.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:47 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3649518536' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T06:52:47.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:47 smithi028 ceph-mon[37984]: from='client.25936 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T06:52:47.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:47 smithi028 ceph-mon[37984]: 6.8 scrub starts 2024-02-05T06:52:47.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:47 smithi028 ceph-mon[37984]: 6.8 scrub ok 2024-02-05T06:52:47.703 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:47 smithi028 ceph-mon[37984]: 3.3 deep-scrub starts 2024-02-05T06:52:47.703 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:47 smithi028 ceph-mon[37984]: 3.3 deep-scrub ok 2024-02-05T06:52:47.733 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:47.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:47.777 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:47.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:47.878 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:47.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:47.909 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:47.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:47.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:47 smithi167 ceph-mon[32774]: 5.8 scrub starts 2024-02-05T06:52:47.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:47 smithi167 ceph-mon[32774]: 5.8 scrub ok 2024-02-05T06:52:47.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:47 smithi167 ceph-mon[32774]: 6.e scrub starts 2024-02-05T06:52:47.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:47 smithi167 ceph-mon[32774]: 6.e scrub ok 2024-02-05T06:52:47.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:47 smithi167 ceph-mon[32774]: 6.10 scrub starts 2024-02-05T06:52:47.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:47 smithi167 ceph-mon[32774]: 6.10 scrub ok 2024-02-05T06:52:47.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:47 smithi167 ceph-mon[32774]: pgmap v926: 168 pgs: 168 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 580 B/s rd, 53 MiB/s wr, 853 op/s 2024-02-05T06:52:47.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:47 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3649518536' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T06:52:47.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:47 smithi167 ceph-mon[32774]: from='client.25936 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T06:52:47.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:47 smithi167 ceph-mon[32774]: 6.8 scrub starts 2024-02-05T06:52:47.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:47 smithi167 ceph-mon[32774]: 6.8 scrub ok 2024-02-05T06:52:47.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:47 smithi167 ceph-mon[32774]: 3.3 deep-scrub starts 2024-02-05T06:52:47.986 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:47 smithi167 ceph-mon[32774]: 3.3 deep-scrub ok 2024-02-05T06:52:48.010 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:48.056 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:48.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:48.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:48.158 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:48.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:48.201 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:48.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:48.302 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:48.339 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:48.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:48.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:48.356 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 87 16 124198 124182 89.1888 67.5 0.0106169 0.0112067 2024-02-05T06:52:48.441 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:48.474 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:48.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:48.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:48.576 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:48.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:48.604 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:48.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:48.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:48 smithi028 ceph-mon[37984]: from='client.25936 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-05T06:52:48.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:48 smithi028 ceph-mon[37984]: osdmap e448: 8 total, 8 up, 8 in 2024-02-05T06:52:48.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:48 smithi028 ceph-mon[37984]: 6.1c scrub starts 2024-02-05T06:52:48.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:48 smithi028 ceph-mon[37984]: 6.1c scrub ok 2024-02-05T06:52:48.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:48 smithi028 ceph-mon[29442]: from='client.25936 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-05T06:52:48.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:48 smithi028 ceph-mon[29442]: osdmap e448: 8 total, 8 up, 8 in 2024-02-05T06:52:48.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:48 smithi028 ceph-mon[29442]: 6.1c scrub starts 2024-02-05T06:52:48.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:48 smithi028 ceph-mon[29442]: 6.1c scrub ok 2024-02-05T06:52:48.706 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:48.736 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:48.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:48.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:48.838 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:48.868 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:48.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:48.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:48.970 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:48.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:48 smithi167 ceph-mon[32774]: from='client.25936 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-05T06:52:48.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:48 smithi167 ceph-mon[32774]: osdmap e448: 8 total, 8 up, 8 in 2024-02-05T06:52:48.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:48 smithi167 ceph-mon[32774]: 6.1c scrub starts 2024-02-05T06:52:48.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:48 smithi167 ceph-mon[32774]: 6.1c scrub ok 2024-02-05T06:52:48.994 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:48.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:48.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:49.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:49.142 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:49.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:49.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:49.243 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:49.282 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:49.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:49.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:49.356 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 88 16 125193 125177 88.882 62.1875 0.0101069 0.0112431 2024-02-05T06:52:49.384 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:49.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:49.419 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:49.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:49.520 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:49.555 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:49.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:49.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:49.657 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:49.689 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:49.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:49.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:49.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:49 smithi028 ceph-mon[37984]: 5.a scrub starts 2024-02-05T06:52:49.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:49 smithi028 ceph-mon[37984]: 5.a scrub ok 2024-02-05T06:52:49.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:49 smithi028 ceph-mon[37984]: 11.0 deep-scrub starts 2024-02-05T06:52:49.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:49 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:52:49.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:49 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:52:49.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:49 smithi028 ceph-mon[37984]: 3.b scrub starts 2024-02-05T06:52:49.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:49 smithi028 ceph-mon[37984]: 3.b scrub ok 2024-02-05T06:52:49.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:49 smithi028 ceph-mon[37984]: osdmap e449: 8 total, 8 up, 8 in 2024-02-05T06:52:49.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:49 smithi028 ceph-mon[37984]: pgmap v929: 168 pgs: 168 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-02-05T06:52:49.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:49 smithi028 ceph-mon[37984]: 11.5 scrub starts 2024-02-05T06:52:49.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:49 smithi028 ceph-mon[29442]: 5.a scrub starts 2024-02-05T06:52:49.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:49 smithi028 ceph-mon[29442]: 5.a scrub ok 2024-02-05T06:52:49.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:49 smithi028 ceph-mon[29442]: 11.0 deep-scrub starts 2024-02-05T06:52:49.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:49 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:52:49.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:49 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:52:49.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:49 smithi028 ceph-mon[29442]: 3.b scrub starts 2024-02-05T06:52:49.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:49 smithi028 ceph-mon[29442]: 3.b scrub ok 2024-02-05T06:52:49.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:49 smithi028 ceph-mon[29442]: osdmap e449: 8 total, 8 up, 8 in 2024-02-05T06:52:49.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:49 smithi028 ceph-mon[29442]: pgmap v929: 168 pgs: 168 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-02-05T06:52:49.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:49 smithi028 ceph-mon[29442]: 11.5 scrub starts 2024-02-05T06:52:49.791 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:49.825 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:49.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:49.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:49.926 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:49.939 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:52:49.965 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:49.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:49.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:49.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:49 smithi167 ceph-mon[32774]: 5.a scrub starts 2024-02-05T06:52:49.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:49 smithi167 ceph-mon[32774]: 5.a scrub ok 2024-02-05T06:52:49.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:49 smithi167 ceph-mon[32774]: 11.0 deep-scrub starts 2024-02-05T06:52:49.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:49 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:52:49.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:49 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:52:49.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:49 smithi167 ceph-mon[32774]: 3.b scrub starts 2024-02-05T06:52:49.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:49 smithi167 ceph-mon[32774]: 3.b scrub ok 2024-02-05T06:52:49.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:49 smithi167 ceph-mon[32774]: osdmap e449: 8 total, 8 up, 8 in 2024-02-05T06:52:49.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:49 smithi167 ceph-mon[32774]: pgmap v929: 168 pgs: 168 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-02-05T06:52:49.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:49 smithi167 ceph-mon[32774]: 11.5 scrub starts 2024-02-05T06:52:50.067 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:50.103 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:50.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:50.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:50.205 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:50.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:50.258 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:50.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:50.356 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 89 16 126162 126146 88.5638 60.5625 0.0167018 0.0112867 2024-02-05T06:52:50.360 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:50.391 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:50.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:50.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:50.402 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd reweight 6 1 2024-02-05T06:52:50.493 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:50.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:50.530 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:50.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:50.631 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:50.665 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:50.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:50.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:50.766 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:50.797 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:50.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:50.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:50.899 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:50.931 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:50.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:50.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:51.032 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:51.066 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:51.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:51.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:51.169 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:51.199 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:51.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:51.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:51.301 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:51.353 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:51.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:51.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:51.356 INFO:tasks.radosbench.radosbench.0.smithi028.stdout: 90 8 126963 126955 88.1416 50.5625 0.0178344 0.0113385 2024-02-05T06:52:51.420 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:51 smithi028 ceph-mon[37984]: 5.e scrub starts 2024-02-05T06:52:51.420 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:51 smithi028 ceph-mon[37984]: 5.e scrub ok 2024-02-05T06:52:51.420 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:51 smithi028 ceph-mon[37984]: 4.3 scrub starts 2024-02-05T06:52:51.421 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:51 smithi028 ceph-mon[37984]: 4.3 scrub ok 2024-02-05T06:52:51.421 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:51 smithi028 ceph-mon[37984]: 3.19 scrub starts 2024-02-05T06:52:51.421 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:51 smithi028 ceph-mon[37984]: 3.19 scrub ok 2024-02-05T06:52:51.421 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:51 smithi028 ceph-mon[37984]: pgmap v930: 168 pgs: 168 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 883 op/s 2024-02-05T06:52:51.421 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:51 smithi028 ceph-mon[37984]: 11.5 scrub ok 2024-02-05T06:52:51.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:51 smithi028 ceph-mon[29442]: 5.e scrub starts 2024-02-05T06:52:51.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:51 smithi028 ceph-mon[29442]: 5.e scrub ok 2024-02-05T06:52:51.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:51 smithi028 ceph-mon[29442]: 4.3 scrub starts 2024-02-05T06:52:51.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:51 smithi028 ceph-mon[29442]: 4.3 scrub ok 2024-02-05T06:52:51.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:51 smithi028 ceph-mon[29442]: 3.19 scrub starts 2024-02-05T06:52:51.421 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:51 smithi028 ceph-mon[29442]: 3.19 scrub ok 2024-02-05T06:52:51.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:51 smithi028 ceph-mon[29442]: pgmap v930: 168 pgs: 168 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 883 op/s 2024-02-05T06:52:51.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:51 smithi028 ceph-mon[29442]: 11.5 scrub ok 2024-02-05T06:52:51.455 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:51.480 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:51.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:51.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:51.585 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:51.612 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:51.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:51.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:51.713 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:51.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:51 smithi167 ceph-mon[32774]: 5.e scrub starts 2024-02-05T06:52:51.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:51 smithi167 ceph-mon[32774]: 5.e scrub ok 2024-02-05T06:52:51.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:51 smithi167 ceph-mon[32774]: 4.3 scrub starts 2024-02-05T06:52:51.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:51 smithi167 ceph-mon[32774]: 4.3 scrub ok 2024-02-05T06:52:51.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:51 smithi167 ceph-mon[32774]: 3.19 scrub starts 2024-02-05T06:52:51.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:51 smithi167 ceph-mon[32774]: 3.19 scrub ok 2024-02-05T06:52:51.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:51 smithi167 ceph-mon[32774]: pgmap v930: 168 pgs: 168 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 883 op/s 2024-02-05T06:52:51.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:51 smithi167 ceph-mon[32774]: 11.5 scrub ok 2024-02-05T06:52:51.741 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:51.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:51.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:51.843 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:51.882 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:51.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:51.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:51.984 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:51.991 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:52:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 20, 2024-02-05T06:52:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 600, 2024-02-05T06:52:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-02-05T06:52:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45587.0:104378 10.10 10:080d0673:::benchmark_data_smithi028_154709_object104377:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e375)", 2024-02-05T06:52:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:48:54.218171+0000", 2024-02-05T06:52:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 237.76149141900001, 2024-02-05T06:52:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1396632680000001, 2024-02-05T06:52:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:52:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45587", 2024-02-05T06:52:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:52:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 104378 2024-02-05T06:52:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.218171+0000", 2024-02-05T06:52:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.218171+0000", 2024-02-05T06:52:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.218173+0000", 2024-02-05T06:52:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1400000000000001e-06 2024-02-05T06:52:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.218195+0000", 2024-02-05T06:52:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1977999999999999e-05 2024-02-05T06:52:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.218206+0000", 2024-02-05T06:52:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1805e-05 2024-02-05T06:52:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.218216+0000", 2024-02-05T06:52:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7529999999999996e-06 2024-02-05T06:52:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.221523+0000", 2024-02-05T06:52:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033071429999999998 2024-02-05T06:52:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:52:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.221532+0000", 2024-02-05T06:52:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.1700000000000003e-06 2024-02-05T06:52:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.342401+0000", 2024-02-05T06:52:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.120868891 2024-02-05T06:52:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.342647+0000", 2024-02-05T06:52:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00024530100000000002 2024-02-05T06:52:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 4,6,7", 2024-02-05T06:52:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.342988+0000", 2024-02-05T06:52:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00034145399999999999 2024-02-05T06:52:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.348745+0000", 2024-02-05T06:52:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0057571380000000002 2024-02-05T06:52:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.353582+0000", 2024-02-05T06:52:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0048372329999999998 2024-02-05T06:52:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.354920+0000", 2024-02-05T06:52:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0013377910000000001 2024-02-05T06:52:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:52:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.357666+0000", 2024-02-05T06:52:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0027462379999999998 2024-02-05T06:52:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.357723+0000", 2024-02-05T06:52:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6134999999999997e-05 2024-02-05T06:52:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.357835+0000", 2024-02-05T06:52:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000112098 2024-02-05T06:52:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45587.0:104383 10.10 10:0bd2e656:::benchmark_data_smithi028_154709_object104382:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e375)", 2024-02-05T06:52:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:48:54.218233+0000", 2024-02-05T06:52:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 237.76142942000001, 2024-02-05T06:52:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1397879959999999, 2024-02-05T06:52:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:52:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45587", 2024-02-05T06:52:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:52:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 104383 2024-02-05T06:52:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.218233+0000", 2024-02-05T06:52:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.218233+0000", 2024-02-05T06:52:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.218234+0000", 2024-02-05T06:52:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5700000000000002e-07 2024-02-05T06:52:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.218254+0000", 2024-02-05T06:52:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9911999999999998e-05 2024-02-05T06:52:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.218267+0000", 2024-02-05T06:52:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2683999999999999e-05 2024-02-05T06:52:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.218274+0000", 2024-02-05T06:52:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7399999999999998e-06 2024-02-05T06:52:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.221614+0000", 2024-02-05T06:52:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0033409160000000002 2024-02-05T06:52:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:52:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.221622+0000", 2024-02-05T06:52:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4189999999999996e-06 2024-02-05T06:52:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.345581+0000", 2024-02-05T06:52:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.123958888 2024-02-05T06:52:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.345750+0000", 2024-02-05T06:52:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000169278 2024-02-05T06:52:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 4,6,7", 2024-02-05T06:52:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.346094+0000", 2024-02-05T06:52:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00034360499999999997 2024-02-05T06:52:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.350680+0000", 2024-02-05T06:52:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0045867809999999998 2024-02-05T06:52:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.355143+0000", 2024-02-05T06:52:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0044621030000000002 2024-02-05T06:52:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.355233+0000", 2024-02-05T06:52:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0019000000000007e-05 2024-02-05T06:52:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:52:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.357878+0000", 2024-02-05T06:52:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0026450900000000001 2024-02-05T06:52:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.357924+0000", 2024-02-05T06:52:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5998000000000002e-05 2024-02-05T06:52:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.358021+0000", 2024-02-05T06:52:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7806000000000005e-05 2024-02-05T06:52:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45587.0:104402 10.10 10:09eb41b3:::benchmark_data_smithi028_154709_object104401:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e375)", 2024-02-05T06:52:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:48:54.218282+0000", 2024-02-05T06:52:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 237.76138133200001, 2024-02-05T06:52:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1400402810000001, 2024-02-05T06:52:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:52:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45587", 2024-02-05T06:52:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:52:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 104402 2024-02-05T06:52:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.218282+0000", 2024-02-05T06:52:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.218282+0000", 2024-02-05T06:52:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.218282+0000", 2024-02-05T06:52:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.9699999999999995e-07 2024-02-05T06:52:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.218305+0000", 2024-02-05T06:52:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2441000000000001e-05 2024-02-05T06:52:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.218312+0000", 2024-02-05T06:52:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3810000000000001e-06 2024-02-05T06:52:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.218318+0000", 2024-02-05T06:52:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2469999999999997e-06 2024-02-05T06:52:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.221788+0000", 2024-02-05T06:52:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0034700080000000001 2024-02-05T06:52:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:52:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:54.221796+0000", 2024-02-05T06:52:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1089999999999998e-06 2024-02-05T06:52:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.348782+0000", 2024-02-05T06:52:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1269857320000001 2024-02-05T06:52:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.348945+0000", 2024-02-05T06:52:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00016271699999999999 2024-02-05T06:52:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 4,6,7", 2024-02-05T06:52:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.349300+0000", 2024-02-05T06:52:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00035472999999999999 2024-02-05T06:52:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.350962+0000", 2024-02-05T06:52:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0016620529999999999 2024-02-05T06:52:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.355741+0000", 2024-02-05T06:52:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0047792700000000004 2024-02-05T06:52:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.355774+0000", 2024-02-05T06:52:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3062999999999998e-05 2024-02-05T06:52:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:52:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.358175+0000", 2024-02-05T06:52:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024009420000000001 2024-02-05T06:52:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.358221+0000", 2024-02-05T06:52:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5569999999999999e-05 2024-02-05T06:52:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:48:55.358322+0000", 2024-02-05T06:52:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000101321 2024-02-05T06:52:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45587.0:137579 10.13 10:cd4162a7:::benchmark_data_smithi028_154709_object16578:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-02-05T06:52:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:49:25.212124+0000", 2024-02-05T06:52:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 206.76753851699999, 2024-02-05T06:52:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0479841780000001, 2024-02-05T06:52:52.014 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:52.014 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:52.014 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:52:52.014 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45587", 2024-02-05T06:52:52.014 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:52:52.014 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 137579 2024-02-05T06:52:52.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.014 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:52.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.014 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:52.014 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:25.212124+0000", 2024-02-05T06:52:52.014 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.015 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:52.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:25.212124+0000", 2024-02-05T06:52:52.015 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.015 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:52.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:25.212126+0000", 2024-02-05T06:52:52.015 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4750000000000001e-06 2024-02-05T06:52:52.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.015 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:52.015 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:25.212131+0000", 2024-02-05T06:52:52.015 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.8400000000000002e-06 2024-02-05T06:52:52.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:52.017 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:25.212141+0000", 2024-02-05T06:52:52.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0390999999999999e-05 2024-02-05T06:52:52.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:52.017 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:25.212153+0000", 2024-02-05T06:52:52.017 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1739000000000001e-05 2024-02-05T06:52:52.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.017 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:52.018 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:25.212518+0000", 2024-02-05T06:52:52.018 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00036544300000000002 2024-02-05T06:52:52.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.018 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:52:52.018 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:25.212528+0000", 2024-02-05T06:52:52.018 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.2869999999999993e-06 2024-02-05T06:52:52.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.018 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:52.018 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:26.239516+0000", 2024-02-05T06:52:52.018 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0269884410000001 2024-02-05T06:52:52.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.018 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:52.019 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:26.239648+0000", 2024-02-05T06:52:52.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00013184399999999999 2024-02-05T06:52:52.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.019 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,7", 2024-02-05T06:52:52.019 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:26.240110+0000", 2024-02-05T06:52:52.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00046219099999999998 2024-02-05T06:52:52.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.019 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:52.019 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:26.253881+0000", 2024-02-05T06:52:52.019 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.013770762000000001 2024-02-05T06:52:52.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.020 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:52.020 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:26.253922+0000", 2024-02-05T06:52:52.020 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1226999999999997e-05 2024-02-05T06:52:52.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.020 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:52:52.020 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:26.259969+0000", 2024-02-05T06:52:52.020 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0060465830000000003 2024-02-05T06:52:52.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.020 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:52.020 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:26.260066+0000", 2024-02-05T06:52:52.020 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.7198e-05 2024-02-05T06:52:52.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.021 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:52.021 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:26.260109+0000", 2024-02-05T06:52:52.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.2756999999999997e-05 2024-02-05T06:52:52.021 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.021 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:52.021 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.021 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45587.0:137588 10.13 10:ce5cf5d4:::benchmark_data_smithi028_154709_object16587:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-02-05T06:52:52.021 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:49:25.221768+0000", 2024-02-05T06:52:52.021 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 206.75789523200001, 2024-02-05T06:52:52.021 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0385810980000001, 2024-02-05T06:52:52.022 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:52.022 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:52.022 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:52:52.022 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45587", 2024-02-05T06:52:52.022 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:52:52.022 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 137588 2024-02-05T06:52:52.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.022 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:52.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:52.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:25.221768+0000", 2024-02-05T06:52:52.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:52.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:25.221768+0000", 2024-02-05T06:52:52.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.023 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:52.023 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:25.221770+0000", 2024-02-05T06:52:52.023 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.136e-06 2024-02-05T06:52:52.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.024 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:52.024 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:25.221775+0000", 2024-02-05T06:52:52.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.3430000000000003e-06 2024-02-05T06:52:52.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.024 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:52.024 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:25.221786+0000", 2024-02-05T06:52:52.024 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.1234e-05 2024-02-05T06:52:52.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.024 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:52.025 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:25.221795+0000", 2024-02-05T06:52:52.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.3529999999999995e-06 2024-02-05T06:52:52.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.025 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:52.025 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:25.221841+0000", 2024-02-05T06:52:52.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5908000000000002e-05 2024-02-05T06:52:52.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.025 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:52:52.025 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:25.221849+0000", 2024-02-05T06:52:52.025 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1990000000000006e-06 2024-02-05T06:52:52.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.026 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:52.026 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:26.242746+0000", 2024-02-05T06:52:52.026 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0208967330000001 2024-02-05T06:52:52.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.026 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:52.026 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:26.242860+0000", 2024-02-05T06:52:52.026 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00011419199999999999 2024-02-05T06:52:52.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.026 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,7", 2024-02-05T06:52:52.026 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:26.243244+0000", 2024-02-05T06:52:52.026 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00038460000000000002 2024-02-05T06:52:52.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.027 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.027 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:52.027 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:26.253961+0000", 2024-02-05T06:52:52.027 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010716249000000001 2024-02-05T06:52:52.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.027 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.027 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:52.027 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:26.254001+0000", 2024-02-05T06:52:52.027 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0017e-05 2024-02-05T06:52:52.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.027 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.027 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:52:52.027 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:26.260234+0000", 2024-02-05T06:52:52.027 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062330240000000002 2024-02-05T06:52:52.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.027 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.028 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:52.028 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:26.260313+0000", 2024-02-05T06:52:52.028 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.9857000000000005e-05 2024-02-05T06:52:52.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.028 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.028 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:52.028 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:26.260349+0000", 2024-02-05T06:52:52.028 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.5253000000000003e-05 2024-02-05T06:52:52.028 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.028 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:52.028 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.028 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.028 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45587.0:137605 10.13 10:cc369462:::benchmark_data_smithi028_154709_object16604:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e385)", 2024-02-05T06:52:52.029 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:49:25.235352+0000", 2024-02-05T06:52:52.029 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 206.74431078699999, 2024-02-05T06:52:52.029 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.025074107, 2024-02-05T06:52:52.029 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:52.029 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:52.029 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:52:52.029 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45587", 2024-02-05T06:52:52.029 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:52:52.029 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 137605 2024-02-05T06:52:52.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.029 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:52.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.029 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:52.029 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:25.235352+0000", 2024-02-05T06:52:52.029 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.030 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:52.030 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:25.235352+0000", 2024-02-05T06:52:52.030 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.030 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:52.030 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:25.235354+0000", 2024-02-05T06:52:52.030 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.7990000000000001e-06 2024-02-05T06:52:52.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.030 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:52.030 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:25.235359+0000", 2024-02-05T06:52:52.030 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.899e-06 2024-02-05T06:52:52.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.031 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:52.031 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:25.235428+0000", 2024-02-05T06:52:52.031 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.8959000000000002e-05 2024-02-05T06:52:52.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.031 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:52.031 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:25.235437+0000", 2024-02-05T06:52:52.031 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0229999999999998e-06 2024-02-05T06:52:52.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.031 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:52.031 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:25.235496+0000", 2024-02-05T06:52:52.031 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9695999999999999e-05 2024-02-05T06:52:52.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.032 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for peered", 2024-02-05T06:52:52.032 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:25.235506+0000", 2024-02-05T06:52:52.032 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3589999999999997e-06 2024-02-05T06:52:52.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.032 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:52.032 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:26.245148+0000", 2024-02-05T06:52:52.032 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.009642433 2024-02-05T06:52:52.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.032 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:52.032 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:26.246918+0000", 2024-02-05T06:52:52.032 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0017698099999999999 2024-02-05T06:52:52.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.033 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,7", 2024-02-05T06:52:52.033 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:26.247403+0000", 2024-02-05T06:52:52.033 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00048498299999999997 2024-02-05T06:52:52.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.033 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:52.033 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:26.254039+0000", 2024-02-05T06:52:52.033 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0066355060000000002 2024-02-05T06:52:52.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.034 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:52.034 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:26.254076+0000", 2024-02-05T06:52:52.034 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7589000000000002e-05 2024-02-05T06:52:52.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.034 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:52:52.034 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:26.260355+0000", 2024-02-05T06:52:52.034 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0062788139999999997 2024-02-05T06:52:52.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.034 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:52.034 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:26.260395+0000", 2024-02-05T06:52:52.034 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.0407999999999997e-05 2024-02-05T06:52:52.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.035 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:52.035 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:49:26.260426+0000", 2024-02-05T06:52:52.035 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.0828999999999999e-05 2024-02-05T06:52:52.035 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.035 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:52.035 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.035 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.035 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.45587.0:208959 10.b e408/345 10:d384102e:::benchmark_data_smithi028_154709_object87958:head v 408'12988, mlcod=408'12985)", 2024-02-05T06:52:52.035 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:50:32.290892+0000", 2024-02-05T06:52:52.035 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 139.68877105600001, 2024-02-05T06:52:52.035 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.182915619, 2024-02-05T06:52:52.035 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:52.035 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:52.035 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:52.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.036 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:52.036 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.290892+0000", 2024-02-05T06:52:52.036 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.036 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:52.036 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.290892+0000", 2024-02-05T06:52:52.036 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.036 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:52.036 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.290893+0000", 2024-02-05T06:52:52.037 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5099999999999999e-06 2024-02-05T06:52:52.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.037 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:52.037 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.290900+0000", 2024-02-05T06:52:52.037 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0879999999999997e-06 2024-02-05T06:52:52.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.037 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:52.037 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.290916+0000", 2024-02-05T06:52:52.037 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5201999999999999e-05 2024-02-05T06:52:52.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.037 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:52.038 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.290923+0000", 2024-02-05T06:52:52.038 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8450000000000003e-06 2024-02-05T06:52:52.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.038 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:52.038 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.292964+0000", 2024-02-05T06:52:52.038 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0020402380000000002 2024-02-05T06:52:52.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.038 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:52.038 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.292983+0000", 2024-02-05T06:52:52.038 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.944e-05 2024-02-05T06:52:52.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.038 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:52.039 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.473776+0000", 2024-02-05T06:52:52.039 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.180792915 2024-02-05T06:52:52.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.039 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.039 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:52.039 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.473807+0000", 2024-02-05T06:52:52.039 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.1381000000000003e-05 2024-02-05T06:52:52.039 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.039 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:52.039 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.039 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.039 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45587.0:208974 10.13 10:c8b3e3c7:::benchmark_data_smithi028_154709_object87973:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-02-05T06:52:52.039 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:50:32.303733+0000", 2024-02-05T06:52:52.039 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 139.675929782, 2024-02-05T06:52:52.039 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.170137863, 2024-02-05T06:52:52.040 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:52.040 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:52.040 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:52:52.040 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45587", 2024-02-05T06:52:52.040 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:52:52.040 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 208974 2024-02-05T06:52:52.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.040 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:52.040 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.040 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:52.040 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.303733+0000", 2024-02-05T06:52:52.040 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.041 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.041 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:52.041 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.303733+0000", 2024-02-05T06:52:52.041 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.041 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.041 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:52.041 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.303736+0000", 2024-02-05T06:52:52.041 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6299999999999998e-06 2024-02-05T06:52:52.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.041 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.041 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:52.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.303746+0000", 2024-02-05T06:52:52.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0094e-05 2024-02-05T06:52:52.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.042 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:52.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.303766+0000", 2024-02-05T06:52:52.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0417000000000001e-05 2024-02-05T06:52:52.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.042 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:52.042 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.303777+0000", 2024-02-05T06:52:52.042 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0579e-05 2024-02-05T06:52:52.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.042 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:52.043 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.303837+0000", 2024-02-05T06:52:52.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.012e-05 2024-02-05T06:52:52.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:52.043 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.304008+0000", 2024-02-05T06:52:52.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000171347 2024-02-05T06:52:52.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.043 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,7", 2024-02-05T06:52:52.043 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.304498+0000", 2024-02-05T06:52:52.043 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00049010899999999997 2024-02-05T06:52:52.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.043 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.044 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:52.044 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.312067+0000", 2024-02-05T06:52:52.044 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0075686160000000002 2024-02-05T06:52:52.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.044 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:52.044 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.321921+0000", 2024-02-05T06:52:52.044 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0098538670000000005 2024-02-05T06:52:52.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.044 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.044 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:52:52.044 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.473815+0000", 2024-02-05T06:52:52.044 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.15189397499999999 2024-02-05T06:52:52.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.045 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:52.045 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.473845+0000", 2024-02-05T06:52:52.045 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.9941999999999999e-05 2024-02-05T06:52:52.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.045 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:52.045 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.473871+0000", 2024-02-05T06:52:52.045 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6166999999999999e-05 2024-02-05T06:52:52.045 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.045 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:52.045 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.045 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.45587.0:208982 10.b e408/345 10:d81930cb:::benchmark_data_smithi028_154709_object87981:head v 408'12989, mlcod=408'12985)", 2024-02-05T06:52:52.045 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:50:32.315384+0000", 2024-02-05T06:52:52.046 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 139.66427937, 2024-02-05T06:52:52.046 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.16064932600000001, 2024-02-05T06:52:52.046 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:52.046 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:52.046 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:52.046 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.046 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:52.046 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:52:52.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:52.047 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:52.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:52.048 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.315384+0000", 2024-02-05T06:52:52.048 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.048 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.048 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:52.048 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.315384+0000", 2024-02-05T06:52:52.048 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.048 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.048 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:52.048 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.315385+0000", 2024-02-05T06:52:52.048 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9920000000000002e-06 2024-02-05T06:52:52.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.049 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.049 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:52.049 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.315397+0000", 2024-02-05T06:52:52.049 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.199e-05 2024-02-05T06:52:52.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.049 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.049 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:52.049 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.315413+0000", 2024-02-05T06:52:52.049 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.5909e-05 2024-02-05T06:52:52.049 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.049 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.049 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:52.049 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.315422+0000", 2024-02-05T06:52:52.049 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.4130000000000006e-06 2024-02-05T06:52:52.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.050 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.050 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:52.050 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.315969+0000", 2024-02-05T06:52:52.050 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00054710800000000003 2024-02-05T06:52:52.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.050 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.050 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:52.050 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.315990+0000", 2024-02-05T06:52:52.050 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1064e-05 2024-02-05T06:52:52.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.050 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.050 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:52.050 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.475990+0000", 2024-02-05T06:52:52.051 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.159999528 2024-02-05T06:52:52.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.051 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.051 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:52.051 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.476033+0000", 2024-02-05T06:52:52.051 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3322000000000003e-05 2024-02-05T06:52:52.051 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.051 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:52.051 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.051 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.051 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45587.0:209002 10.13 10:cd612952:::benchmark_data_smithi028_154709_object88001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e408)", 2024-02-05T06:52:52.051 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:50:32.330084+0000", 2024-02-05T06:52:52.051 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 139.649578746, 2024-02-05T06:52:52.052 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.146121425, 2024-02-05T06:52:52.052 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:52.052 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:52.052 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:52:52.052 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45587", 2024-02-05T06:52:52.052 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:52:52.052 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 209002 2024-02-05T06:52:52.052 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.052 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:52.052 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.052 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:52.052 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.330084+0000", 2024-02-05T06:52:52.052 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.053 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.053 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.053 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:52.053 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.330084+0000", 2024-02-05T06:52:52.053 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.053 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.053 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.053 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:52.053 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.330087+0000", 2024-02-05T06:52:52.053 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.5239999999999999e-06 2024-02-05T06:52:52.053 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.053 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.053 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:52.053 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.330094+0000", 2024-02-05T06:52:52.053 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4599999999999997e-06 2024-02-05T06:52:52.054 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.054 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.054 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:52.054 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.330115+0000", 2024-02-05T06:52:52.054 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0460000000000001e-05 2024-02-05T06:52:52.054 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.054 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.054 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:52.054 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.330125+0000", 2024-02-05T06:52:52.054 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.077e-05 2024-02-05T06:52:52.054 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.054 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.054 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:52.054 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.330792+0000", 2024-02-05T06:52:52.055 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00066642100000000005 2024-02-05T06:52:52.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.055 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.055 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:52.055 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.330934+0000", 2024-02-05T06:52:52.055 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014244799999999999 2024-02-05T06:52:52.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.055 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.055 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 2,7", 2024-02-05T06:52:52.055 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.331462+0000", 2024-02-05T06:52:52.055 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00052763899999999999 2024-02-05T06:52:52.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.055 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.055 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:52.055 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.334560+0000", 2024-02-05T06:52:52.056 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003097893 2024-02-05T06:52:52.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.056 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.056 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:52.056 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.334599+0000", 2024-02-05T06:52:52.056 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9644999999999997e-05 2024-02-05T06:52:52.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.056 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.056 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:52:52.056 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.476043+0000", 2024-02-05T06:52:52.056 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.141443241 2024-02-05T06:52:52.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.056 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.056 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:52.056 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.476134+0000", 2024-02-05T06:52:52.057 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.0896999999999994e-05 2024-02-05T06:52:52.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.057 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:52.057 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:50:32.476206+0000", 2024-02-05T06:52:52.057 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2026999999999993e-05 2024-02-05T06:52:52.057 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.057 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:52.057 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.057 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.45878.0:55570 11.f e426/418 11:fcfcc82d:::benchmark_data_smithi028_177697_object55569:head v 426'3515, mlcod=426'3513)", 2024-02-05T06:52:52.057 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:51:50.310180+0000", 2024-02-05T06:52:52.057 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 61.669482737999999, 2024-02-05T06:52:52.057 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.105644114, 2024-02-05T06:52:52.058 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:52.058 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:52.058 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:52.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.058 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:52.058 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.310180+0000", 2024-02-05T06:52:52.058 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.058 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:52.058 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.310180+0000", 2024-02-05T06:52:52.059 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.059 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:52.059 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.310181+0000", 2024-02-05T06:52:52.059 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.066e-06 2024-02-05T06:52:52.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.059 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:52.059 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.310215+0000", 2024-02-05T06:52:52.059 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3300000000000003e-05 2024-02-05T06:52:52.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.059 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:52.060 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.310222+0000", 2024-02-05T06:52:52.060 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.2200000000000003e-06 2024-02-05T06:52:52.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.060 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.060 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:52.060 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.310228+0000", 2024-02-05T06:52:52.060 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8379999999999998e-06 2024-02-05T06:52:52.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.060 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.060 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:52.060 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.312639+0000", 2024-02-05T06:52:52.060 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0024111570000000001 2024-02-05T06:52:52.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.060 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.060 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:52.061 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.312658+0000", 2024-02-05T06:52:52.061 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9519999999999999e-05 2024-02-05T06:52:52.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.061 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:52.061 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.320009+0000", 2024-02-05T06:52:52.061 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073509980000000001 2024-02-05T06:52:52.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.061 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:52.061 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.415824+0000", 2024-02-05T06:52:52.062 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.095815015000000003 2024-02-05T06:52:52.062 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.062 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:52.062 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.062 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.45878.0:55574 11.f e426/418 11:f0570c00:::benchmark_data_smithi028_177697_object55573:head v 426'3516, mlcod=426'3513)", 2024-02-05T06:52:52.062 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:51:50.312683+0000", 2024-02-05T06:52:52.062 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 61.666980264999999, 2024-02-05T06:52:52.062 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.103188111, 2024-02-05T06:52:52.062 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:52.062 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:52.062 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:52.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.063 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:52.063 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.312683+0000", 2024-02-05T06:52:52.063 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.063 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:52.063 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.312683+0000", 2024-02-05T06:52:52.063 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.063 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:52.063 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.312683+0000", 2024-02-05T06:52:52.063 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3799999999999996e-07 2024-02-05T06:52:52.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.064 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:52.064 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.312710+0000", 2024-02-05T06:52:52.064 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.7042999999999998e-05 2024-02-05T06:52:52.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.064 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:52.064 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.312717+0000", 2024-02-05T06:52:52.064 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.2210000000000002e-06 2024-02-05T06:52:52.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.064 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:52.064 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.312722+0000", 2024-02-05T06:52:52.065 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1240000000000004e-06 2024-02-05T06:52:52.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.065 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:52.065 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.316614+0000", 2024-02-05T06:52:52.065 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0038926529999999998 2024-02-05T06:52:52.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.065 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:52.065 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.316634+0000", 2024-02-05T06:52:52.065 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9164000000000002e-05 2024-02-05T06:52:52.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.065 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:52.066 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.415833+0000", 2024-02-05T06:52:52.066 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.099199490000000001 2024-02-05T06:52:52.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.066 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:52.066 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:51:50.415871+0000", 2024-02-05T06:52:52.066 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.7678000000000001e-05 2024-02-05T06:52:52.066 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.066 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:52.066 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.066 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45878.0:124776 11.0 11:0e7f18c9:::benchmark_data_smithi028_177697_object124775:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-02-05T06:52:52.066 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:52:48.913400+0000", 2024-02-05T06:52:52.066 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 3.0662628199999999, 2024-02-05T06:52:52.067 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.123938773, 2024-02-05T06:52:52.067 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:52.067 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:52.067 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:52:52.067 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45878", 2024-02-05T06:52:52.067 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3059201968", 2024-02-05T06:52:52.067 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 124776 2024-02-05T06:52:52.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.067 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:52.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.067 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:52.067 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.913400+0000", 2024-02-05T06:52:52.067 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.068 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:52.068 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.913400+0000", 2024-02-05T06:52:52.068 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.068 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.068 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:52.068 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.913402+0000", 2024-02-05T06:52:52.068 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.485e-06 2024-02-05T06:52:52.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.068 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.068 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:52.068 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.913423+0000", 2024-02-05T06:52:52.068 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1219000000000001e-05 2024-02-05T06:52:52.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.068 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.069 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:52.069 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.913431+0000", 2024-02-05T06:52:52.069 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.6179999999999995e-06 2024-02-05T06:52:52.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.069 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.069 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:52.069 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.913438+0000", 2024-02-05T06:52:52.069 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0509999999999999e-06 2024-02-05T06:52:52.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.069 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.069 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:52.069 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.914081+0000", 2024-02-05T06:52:52.069 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00064211100000000001 2024-02-05T06:52:52.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.070 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.070 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:52.070 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.914303+0000", 2024-02-05T06:52:52.070 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00022268400000000001 2024-02-05T06:52:52.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.070 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.070 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-02-05T06:52:52.070 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.914591+0000", 2024-02-05T06:52:52.070 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00028742999999999998 2024-02-05T06:52:52.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.070 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.070 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:52.070 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.919126+0000", 2024-02-05T06:52:52.070 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0045356329999999999 2024-02-05T06:52:52.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.070 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.071 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:52.071 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.036989+0000", 2024-02-05T06:52:52.071 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.11786248100000001 2024-02-05T06:52:52.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.071 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.071 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:52:52.071 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.037002+0000", 2024-02-05T06:52:52.071 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.3117e-05 2024-02-05T06:52:52.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.071 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.071 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:52.071 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.037102+0000", 2024-02-05T06:52:52.071 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.9782000000000001e-05 2024-02-05T06:52:52.071 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.071 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.072 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:52.072 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.037339+0000", 2024-02-05T06:52:52.072 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00023716199999999999 2024-02-05T06:52:52.072 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.072 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:52.072 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.072 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.072 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45878.0:124786 11.0 11:020fdc99:::benchmark_data_smithi028_177697_object124785:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-02-05T06:52:52.072 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:52:48.921384+0000", 2024-02-05T06:52:52.072 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 3.0582791899999999, 2024-02-05T06:52:52.072 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.120739475, 2024-02-05T06:52:52.072 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:52.072 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:52.073 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:52:52.073 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45878", 2024-02-05T06:52:52.073 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3059201968", 2024-02-05T06:52:52.073 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 124786 2024-02-05T06:52:52.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.073 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:52.073 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.073 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:52.073 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.921384+0000", 2024-02-05T06:52:52.073 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.073 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.073 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:52.073 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.921384+0000", 2024-02-05T06:52:52.073 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.074 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.074 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:52.074 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.921385+0000", 2024-02-05T06:52:52.074 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.8199999999999998e-07 2024-02-05T06:52:52.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.074 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.074 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:52.074 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.921406+0000", 2024-02-05T06:52:52.074 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.1520000000000001e-05 2024-02-05T06:52:52.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.074 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.074 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:52.074 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.921414+0000", 2024-02-05T06:52:52.075 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.6799999999999993e-06 2024-02-05T06:52:52.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.075 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.075 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:52.075 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.921420+0000", 2024-02-05T06:52:52.075 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.6610000000000003e-06 2024-02-05T06:52:52.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.075 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.075 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:52.075 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.034294+0000", 2024-02-05T06:52:52.076 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.112873826 2024-02-05T06:52:52.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.076 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.076 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:52.076 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.034615+0000", 2024-02-05T06:52:52.076 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00032054799999999998 2024-02-05T06:52:52.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.076 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.076 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-02-05T06:52:52.076 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.034841+0000", 2024-02-05T06:52:52.076 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00022667400000000001 2024-02-05T06:52:52.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.076 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.076 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:52:52.077 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.040712+0000", 2024-02-05T06:52:52.077 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0058708809999999997 2024-02-05T06:52:52.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.077 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:52.077 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.041984+0000", 2024-02-05T06:52:52.077 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0012715770000000001 2024-02-05T06:52:52.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.077 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:52.077 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.042023+0000", 2024-02-05T06:52:52.077 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9444999999999999e-05 2024-02-05T06:52:52.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.077 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.078 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:52.078 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.042079+0000", 2024-02-05T06:52:52.078 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.5581999999999997e-05 2024-02-05T06:52:52.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.078 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:52.078 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.042123+0000", 2024-02-05T06:52:52.078 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4199000000000002e-05 2024-02-05T06:52:52.078 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.078 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:52.078 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.078 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(client.45878.0:124776 11.0 e449/418 ondisk, result = 0)", 2024-02-05T06:52:52.078 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:52:48.924048+0000", 2024-02-05T06:52:52.079 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 3.0556149779999999, 2024-02-05T06:52:52.079 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.11340863299999999, 2024-02-05T06:52:52.079 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:52.079 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-02-05T06:52:52.079 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:52.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.079 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:52.079 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.924048+0000", 2024-02-05T06:52:52.079 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.079 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:52.079 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.924048+0000", 2024-02-05T06:52:52.079 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.080 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:52.080 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.924049+0000", 2024-02-05T06:52:52.080 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2389999999999999e-06 2024-02-05T06:52:52.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.080 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:52.080 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.924054+0000", 2024-02-05T06:52:52.080 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.6550000000000003e-06 2024-02-05T06:52:52.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.080 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:52.080 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.924062+0000", 2024-02-05T06:52:52.080 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.704e-06 2024-02-05T06:52:52.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.081 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.081 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:52.081 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.924070+0000", 2024-02-05T06:52:52.081 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8990000000000001e-06 2024-02-05T06:52:52.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.081 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.081 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:52.081 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.036974+0000", 2024-02-05T06:52:52.081 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.11290357500000001 2024-02-05T06:52:52.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.081 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.081 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:52.081 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.036982+0000", 2024-02-05T06:52:52.081 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5169999999999999e-06 2024-02-05T06:52:52.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.082 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.082 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:52.082 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.037457+0000", 2024-02-05T06:52:52.082 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00047404400000000002 2024-02-05T06:52:52.082 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.082 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:52.082 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.082 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.082 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(osd.1.0:136468 11.0 e449/418 ondisk, result = 0)", 2024-02-05T06:52:52.082 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:52:48.930444+0000", 2024-02-05T06:52:52.082 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 3.049218953, 2024-02-05T06:52:52.083 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.107162225, 2024-02-05T06:52:52.083 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:52.083 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-02-05T06:52:52.083 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:52.083 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.083 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:52.083 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.930444+0000", 2024-02-05T06:52:52.083 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.083 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.083 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:52.083 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.930444+0000", 2024-02-05T06:52:52.083 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.083 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.083 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:52.084 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.930445+0000", 2024-02-05T06:52:52.084 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2330000000000001e-06 2024-02-05T06:52:52.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.084 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:52.084 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.930450+0000", 2024-02-05T06:52:52.084 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4020000000000004e-06 2024-02-05T06:52:52.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.084 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:52.084 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.930459+0000", 2024-02-05T06:52:52.084 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6180000000000001e-06 2024-02-05T06:52:52.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.085 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:52.085 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.930466+0000", 2024-02-05T06:52:52.085 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.4269999999999996e-06 2024-02-05T06:52:52.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.085 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:52.085 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.037583+0000", 2024-02-05T06:52:52.085 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.107117111 2024-02-05T06:52:52.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.085 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:52.085 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.037590+0000", 2024-02-05T06:52:52.085 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3379999999999999e-06 2024-02-05T06:52:52.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.086 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:52.086 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.037606+0000", 2024-02-05T06:52:52.086 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.6096e-05 2024-02-05T06:52:52.086 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.086 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:52.086 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.086 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45878.0:124815 11.0 11:0b353aa7:::benchmark_data_smithi028_177697_object124814:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-02-05T06:52:52.086 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:52:48.944484+0000", 2024-02-05T06:52:52.086 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 3.035179093, 2024-02-05T06:52:52.086 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.11944555, 2024-02-05T06:52:52.086 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:52.086 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:52.087 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:52:52.087 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45878", 2024-02-05T06:52:52.087 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3059201968", 2024-02-05T06:52:52.087 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 124815 2024-02-05T06:52:52.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.087 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:52.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.087 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:52.087 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.944484+0000", 2024-02-05T06:52:52.087 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.087 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:52.087 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.944484+0000", 2024-02-05T06:52:52.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.088 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:52.088 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.944486+0000", 2024-02-05T06:52:52.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.03e-06 2024-02-05T06:52:52.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.088 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:52.088 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.944510+0000", 2024-02-05T06:52:52.088 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.4289000000000001e-05 2024-02-05T06:52:52.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.088 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:52.089 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.944522+0000", 2024-02-05T06:52:52.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2268e-05 2024-02-05T06:52:52.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.089 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:52.089 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.944530+0000", 2024-02-05T06:52:52.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.1189999999999997e-06 2024-02-05T06:52:52.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.089 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:52.089 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.042155+0000", 2024-02-05T06:52:52.089 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.097624162 2024-02-05T06:52:52.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.090 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:52.090 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.042306+0000", 2024-02-05T06:52:52.090 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015108100000000001 2024-02-05T06:52:52.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.090 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-02-05T06:52:52.090 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.042446+0000", 2024-02-05T06:52:52.090 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00014075599999999999 2024-02-05T06:52:52.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.090 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:52.091 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.050511+0000", 2024-02-05T06:52:52.091 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0080645170000000002 2024-02-05T06:52:52.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.091 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:52:52.091 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.053237+0000", 2024-02-05T06:52:52.091 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.002726287 2024-02-05T06:52:52.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.091 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:52.091 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.063777+0000", 2024-02-05T06:52:52.091 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010539464 2024-02-05T06:52:52.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.092 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:52.092 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.063873+0000", 2024-02-05T06:52:52.092 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.6044000000000004e-05 2024-02-05T06:52:52.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.092 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:52.092 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.063929+0000", 2024-02-05T06:52:52.092 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6533000000000002e-05 2024-02-05T06:52:52.092 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.092 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:52.092 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.092 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45878.0:124824 11.0 11:024bae04:::benchmark_data_smithi028_177697_object124823:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-02-05T06:52:52.093 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:52:48.949510+0000", 2024-02-05T06:52:52.093 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 3.0301528169999998, 2024-02-05T06:52:52.093 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.11472816, 2024-02-05T06:52:52.093 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:52.093 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:52.093 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:52:52.093 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45878", 2024-02-05T06:52:52.093 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3059201968", 2024-02-05T06:52:52.093 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 124824 2024-02-05T06:52:52.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.093 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:52.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.093 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:52.094 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.949510+0000", 2024-02-05T06:52:52.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.094 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:52.094 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.949510+0000", 2024-02-05T06:52:52.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.094 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:52.094 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.949512+0000", 2024-02-05T06:52:52.094 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.592e-06 2024-02-05T06:52:52.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.094 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:52.095 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.949534+0000", 2024-02-05T06:52:52.095 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.2779e-05 2024-02-05T06:52:52.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.095 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:52.095 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.949545+0000", 2024-02-05T06:52:52.095 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0692e-05 2024-02-05T06:52:52.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.095 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:52.095 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:48.949553+0000", 2024-02-05T06:52:52.095 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8140000000000001e-06 2024-02-05T06:52:52.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.095 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:52.096 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.045264+0000", 2024-02-05T06:52:52.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.095711059000000001 2024-02-05T06:52:52.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.096 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:52.096 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.045416+0000", 2024-02-05T06:52:52.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015159 2024-02-05T06:52:52.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.096 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-02-05T06:52:52.096 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.045570+0000", 2024-02-05T06:52:52.096 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015490600000000001 2024-02-05T06:52:52.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.097 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:52:52.097 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.053253+0000", 2024-02-05T06:52:52.097 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0076825340000000004 2024-02-05T06:52:52.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.097 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:52.097 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.056361+0000", 2024-02-05T06:52:52.097 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.003107837 2024-02-05T06:52:52.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.097 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:52.099 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.064016+0000", 2024-02-05T06:52:52.099 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0076551400000000004 2024-02-05T06:52:52.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.099 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:52.099 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.064187+0000", 2024-02-05T06:52:52.099 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00017098999999999999 2024-02-05T06:52:52.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.100 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:52.100 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.064238+0000", 2024-02-05T06:52:52.100 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1227000000000003e-05 2024-02-05T06:52:52.100 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.100 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:52.100 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.100 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45878.0:125108 11.0 11:003ae563:::benchmark_data_smithi028_177697_object125107:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-02-05T06:52:52.100 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:52:49.247210+0000", 2024-02-05T06:52:52.100 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 2.7324530550000001, 2024-02-05T06:52:52.100 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.122199079, 2024-02-05T06:52:52.100 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:52.100 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:52.101 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:52:52.101 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45878", 2024-02-05T06:52:52.101 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3059201968", 2024-02-05T06:52:52.101 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 125108 2024-02-05T06:52:52.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.101 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:52.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.101 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:52.101 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.247210+0000", 2024-02-05T06:52:52.101 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.101 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:52.101 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.247210+0000", 2024-02-05T06:52:52.101 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.102 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:52.102 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.247212+0000", 2024-02-05T06:52:52.102 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.154e-06 2024-02-05T06:52:52.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.102 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:52.102 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.247239+0000", 2024-02-05T06:52:52.102 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6716e-05 2024-02-05T06:52:52.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.102 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:52.102 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.247251+0000", 2024-02-05T06:52:52.103 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2483999999999999e-05 2024-02-05T06:52:52.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.103 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:52.103 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.247265+0000", 2024-02-05T06:52:52.103 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.4316e-05 2024-02-05T06:52:52.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.103 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:52.103 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.280377+0000", 2024-02-05T06:52:52.103 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.033111069999999999 2024-02-05T06:52:52.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.103 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:52.104 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.280678+0000", 2024-02-05T06:52:52.104 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00030190099999999998 2024-02-05T06:52:52.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.104 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-02-05T06:52:52.104 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.280830+0000", 2024-02-05T06:52:52.104 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000151118 2024-02-05T06:52:52.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.105 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:52:52.105 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.292822+0000", 2024-02-05T06:52:52.105 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.011992788000000001 2024-02-05T06:52:52.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.105 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:52.105 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.300153+0000", 2024-02-05T06:52:52.105 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0073301410000000001 2024-02-05T06:52:52.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.106 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:52.106 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.369279+0000", 2024-02-05T06:52:52.106 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.069126339999999994 2024-02-05T06:52:52.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.106 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:52.106 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.369349+0000", 2024-02-05T06:52:52.106 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0349e-05 2024-02-05T06:52:52.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.106 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:52.106 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:49.369409+0000", 2024-02-05T06:52:52.106 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.9701999999999999e-05 2024-02-05T06:52:52.106 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.106 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:52.107 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.107 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45878.0:126761 11.0 11:099ffd6c:::benchmark_data_smithi028_177697_object126760:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e449)", 2024-02-05T06:52:52.107 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:52:51.050603+0000", 2024-02-05T06:52:52.107 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.92905980799999999, 2024-02-05T06:52:52.107 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.10237437200000001, 2024-02-05T06:52:52.107 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:52:52.107 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:52:52.107 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:52:52.107 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45878", 2024-02-05T06:52:52.107 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3059201968", 2024-02-05T06:52:52.107 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 126761 2024-02-05T06:52:52.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.107 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:52:52.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.108 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:52:52.108 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:51.050603+0000", 2024-02-05T06:52:52.108 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.108 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:52:52.108 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:51.050603+0000", 2024-02-05T06:52:52.108 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:52:52.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.108 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.108 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:52:52.108 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:51.050605+0000", 2024-02-05T06:52:52.108 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.809e-06 2024-02-05T06:52:52.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.109 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:52:52.109 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:51.050661+0000", 2024-02-05T06:52:52.109 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6045999999999998e-05 2024-02-05T06:52:52.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.109 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:52:52.109 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:51.050671+0000", 2024-02-05T06:52:52.109 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.0339999999999999e-05 2024-02-05T06:52:52.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.109 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:52:52.109 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:51.050679+0000", 2024-02-05T06:52:52.109 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.4019999999999996e-06 2024-02-05T06:52:52.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:52:52.110 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:51.127547+0000", 2024-02-05T06:52:52.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.076868150999999996 2024-02-05T06:52:52.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:52:52.110 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:51.127704+0000", 2024-02-05T06:52:52.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00015761200000000001 2024-02-05T06:52:52.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 3,7", 2024-02-05T06:52:52.110 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:51.128001+0000", 2024-02-05T06:52:52.110 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.00029701499999999997 2024-02-05T06:52:52.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.110 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:52.111 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:51.134643+0000", 2024-02-05T06:52:52.111 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0066414940000000004 2024-02-05T06:52:52.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.111 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:52:52.111 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:51.145137+0000", 2024-02-05T06:52:52.111 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.010493926000000001 2024-02-05T06:52:52.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.111 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:52:52.111 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:51.152810+0000", 2024-02-05T06:52:52.111 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0076734589999999997 2024-02-05T06:52:52.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.111 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "commit_sent", 2024-02-05T06:52:52.111 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:51.152920+0000", 2024-02-05T06:52:52.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000109249 2024-02-05T06:52:52.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:52:52.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:52:52.112 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "done", 2024-02-05T06:52:52.112 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:52:51.152977+0000", 2024-02-05T06:52:52.112 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.7868999999999998e-05 2024-02-05T06:52:52.112 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.112 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:52.112 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.112 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:52:52.112 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:52:52.112 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:52:52.149 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:52.211 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:52.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:52.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:52.295 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:52:52.313 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:52.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:52.339 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:52.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:52.358 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Total time run: 90.0367 2024-02-05T06:52:52.358 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Total writes made: 126963 2024-02-05T06:52:52.358 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Write size: 65536 2024-02-05T06:52:52.358 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Object size: 65536 2024-02-05T06:52:52.358 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Bandwidth (MB/sec): 88.1328 2024-02-05T06:52:52.358 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Stddev Bandwidth: 28.384 2024-02-05T06:52:52.358 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Max bandwidth (MB/sec): 194.75 2024-02-05T06:52:52.358 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Min bandwidth (MB/sec): 50.5625 2024-02-05T06:52:52.358 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Average IOPS: 1410 2024-02-05T06:52:52.359 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Stddev IOPS: 454.144 2024-02-05T06:52:52.359 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Max IOPS: 3116 2024-02-05T06:52:52.359 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Min IOPS: 809 2024-02-05T06:52:52.359 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Average Latency(s): 0.0113406 2024-02-05T06:52:52.359 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Stddev Latency(s): 0.00822784 2024-02-05T06:52:52.359 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Max latency(s): 0.204312 2024-02-05T06:52:52.359 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Min latency(s): 0.0012181 2024-02-05T06:52:52.395 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Cleaning up (deleting benchmark objects) 2024-02-05T06:52:52.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:52.489 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:52.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:52.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:52.591 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:52.601 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:52 smithi028 ceph-mon[37984]: 11.3 deep-scrub starts 2024-02-05T06:52:52.601 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:52 smithi028 ceph-mon[37984]: 5.7 scrub starts 2024-02-05T06:52:52.601 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:52 smithi028 ceph-mon[37984]: 5.7 scrub ok 2024-02-05T06:52:52.601 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:52 smithi028 ceph-mon[37984]: 6.18 scrub starts 2024-02-05T06:52:52.601 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:52 smithi028 ceph-mon[37984]: 11.2 scrub starts 2024-02-05T06:52:52.601 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:52 smithi028 ceph-mon[37984]: 11.c scrub starts 2024-02-05T06:52:52.601 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:52 smithi028 ceph-mon[37984]: 6.18 scrub ok 2024-02-05T06:52:52.602 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:52 smithi028 ceph-mon[29442]: 11.3 deep-scrub starts 2024-02-05T06:52:52.602 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:52 smithi028 ceph-mon[29442]: 5.7 scrub starts 2024-02-05T06:52:52.602 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:52 smithi028 ceph-mon[29442]: 5.7 scrub ok 2024-02-05T06:52:52.602 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:52 smithi028 ceph-mon[29442]: 6.18 scrub starts 2024-02-05T06:52:52.602 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:52 smithi028 ceph-mon[29442]: 11.2 scrub starts 2024-02-05T06:52:52.602 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:52 smithi028 ceph-mon[29442]: 11.c scrub starts 2024-02-05T06:52:52.602 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:52 smithi028 ceph-mon[29442]: 6.18 scrub ok 2024-02-05T06:52:52.602 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:52:52 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:52:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:52:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:52.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:52.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:52.733 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:52.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:52 smithi167 ceph-mon[32774]: 11.3 deep-scrub starts 2024-02-05T06:52:52.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:52 smithi167 ceph-mon[32774]: 5.7 scrub starts 2024-02-05T06:52:52.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:52 smithi167 ceph-mon[32774]: 5.7 scrub ok 2024-02-05T06:52:52.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:52 smithi167 ceph-mon[32774]: 6.18 scrub starts 2024-02-05T06:52:52.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:52 smithi167 ceph-mon[32774]: 11.2 scrub starts 2024-02-05T06:52:52.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:52 smithi167 ceph-mon[32774]: 11.c scrub starts 2024-02-05T06:52:52.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:52 smithi167 ceph-mon[32774]: 6.18 scrub ok 2024-02-05T06:52:52.766 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:52.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:52.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:52.868 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:52.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:52.897 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:52.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:52.999 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:53.048 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:53.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:53.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:53.150 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:53.185 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:53.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:53.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:53.287 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:53.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:53.320 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:53.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:53.346 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-02-05T06:52:53.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:53.459 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:53.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:53.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:53.561 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:53.599 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:53.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:53.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:53.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:53 smithi028 ceph-mon[37984]: 4.a scrub starts 2024-02-05T06:52:53.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:53 smithi028 ceph-mon[37984]: 4.a scrub ok 2024-02-05T06:52:53.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:53 smithi028 ceph-mon[37984]: 11.d scrub starts 2024-02-05T06:52:53.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:53 smithi028 ceph-mon[37984]: 4.6 scrub starts 2024-02-05T06:52:53.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:53 smithi028 ceph-mon[37984]: 4.6 scrub ok 2024-02-05T06:52:53.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:53 smithi028 ceph-mon[37984]: 11.e scrub starts 2024-02-05T06:52:53.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:53 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2148341388' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:52:53.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:53 smithi028 ceph-mon[37984]: from='client.25942 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:52:53.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:53 smithi028 ceph-mon[37984]: pgmap v931: 168 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 165 active+clean; 7.7 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-02-05T06:52:53.701 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:53.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:53 smithi028 ceph-mon[29442]: 4.a scrub starts 2024-02-05T06:52:53.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:53 smithi028 ceph-mon[29442]: 4.a scrub ok 2024-02-05T06:52:53.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:53 smithi028 ceph-mon[29442]: 11.d scrub starts 2024-02-05T06:52:53.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:53 smithi028 ceph-mon[29442]: 4.6 scrub starts 2024-02-05T06:52:53.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:53 smithi028 ceph-mon[29442]: 4.6 scrub ok 2024-02-05T06:52:53.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:53 smithi028 ceph-mon[29442]: 11.e scrub starts 2024-02-05T06:52:53.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:53 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2148341388' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:52:53.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:53 smithi028 ceph-mon[29442]: from='client.25942 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:52:53.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:53 smithi028 ceph-mon[29442]: pgmap v931: 168 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 165 active+clean; 7.7 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-02-05T06:52:53.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:53.736 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:53.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:53.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:53 smithi167 ceph-mon[32774]: 4.a scrub starts 2024-02-05T06:52:53.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:53 smithi167 ceph-mon[32774]: 4.a scrub ok 2024-02-05T06:52:53.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:53 smithi167 ceph-mon[32774]: 11.d scrub starts 2024-02-05T06:52:53.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:53 smithi167 ceph-mon[32774]: 4.6 scrub starts 2024-02-05T06:52:53.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:53 smithi167 ceph-mon[32774]: 4.6 scrub ok 2024-02-05T06:52:53.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:53 smithi167 ceph-mon[32774]: 11.e scrub starts 2024-02-05T06:52:53.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:53 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2148341388' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:52:53.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:53 smithi167 ceph-mon[32774]: from='client.25942 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:52:53.737 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:53 smithi167 ceph-mon[32774]: pgmap v931: 168 pgs: 1 active+clean+scrubbing+deep, 2 active+clean+scrubbing, 165 active+clean; 7.7 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-02-05T06:52:53.737 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:52:53 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:52:53.837 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:53.866 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:53.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:53.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:53.968 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:54.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:54.020 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:54.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:54.121 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:54.150 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:54.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:54.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:54.252 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:54.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:54.292 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:54.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:54.347 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-02-05T06:52:54.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:54.437 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:54.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:54.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:54.539 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:54.567 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:54.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:54.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:54.669 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:54.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:54 smithi028 ceph-mon[37984]: 11.1 scrub starts 2024-02-05T06:52:54.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:54 smithi028 ceph-mon[37984]: 11.c scrub ok 2024-02-05T06:52:54.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:54 smithi028 ceph-mon[37984]: 11.2 scrub ok 2024-02-05T06:52:54.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:54 smithi028 ceph-mon[37984]: 6.f scrub starts 2024-02-05T06:52:54.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:54 smithi028 ceph-mon[37984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:52:54.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:54 smithi028 ceph-mon[37984]: from='client.25942 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:52:54.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:54 smithi028 ceph-mon[37984]: osdmap e450: 8 total, 8 up, 8 in 2024-02-05T06:52:54.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:54 smithi028 ceph-mon[37984]: 3.7 deep-scrub starts 2024-02-05T06:52:54.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:54 smithi028 ceph-mon[37984]: 3.7 deep-scrub ok 2024-02-05T06:52:54.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:54 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:52:54.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:54 smithi028 ceph-mon[29442]: 11.1 scrub starts 2024-02-05T06:52:54.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:54 smithi028 ceph-mon[29442]: 11.c scrub ok 2024-02-05T06:52:54.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:54 smithi028 ceph-mon[29442]: 11.2 scrub ok 2024-02-05T06:52:54.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:54 smithi028 ceph-mon[29442]: 6.f scrub starts 2024-02-05T06:52:54.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:54 smithi028 ceph-mon[29442]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:52:54.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:54 smithi028 ceph-mon[29442]: from='client.25942 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:52:54.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:54 smithi028 ceph-mon[29442]: osdmap e450: 8 total, 8 up, 8 in 2024-02-05T06:52:54.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:54 smithi028 ceph-mon[29442]: 3.7 deep-scrub starts 2024-02-05T06:52:54.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:54 smithi028 ceph-mon[29442]: 3.7 deep-scrub ok 2024-02-05T06:52:54.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:54 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:52:54.704 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:54.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:54.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:54.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:54 smithi167 ceph-mon[32774]: 11.1 scrub starts 2024-02-05T06:52:54.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:54 smithi167 ceph-mon[32774]: 11.c scrub ok 2024-02-05T06:52:54.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:54 smithi167 ceph-mon[32774]: 11.2 scrub ok 2024-02-05T06:52:54.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:54 smithi167 ceph-mon[32774]: 6.f scrub starts 2024-02-05T06:52:54.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:54 smithi167 ceph-mon[32774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:52:54.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:54 smithi167 ceph-mon[32774]: from='client.25942 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:52:54.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:54 smithi167 ceph-mon[32774]: osdmap e450: 8 total, 8 up, 8 in 2024-02-05T06:52:54.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:54 smithi167 ceph-mon[32774]: 3.7 deep-scrub starts 2024-02-05T06:52:54.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:54 smithi167 ceph-mon[32774]: 3.7 deep-scrub ok 2024-02-05T06:52:54.736 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:54 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:52:54.806 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:54.842 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:54.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:54.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:54.944 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:54.987 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:54.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:54.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:55.089 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:55.117 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:55.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:55.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:55.219 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:55.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:55.257 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:55.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:55.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:55.395 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:55.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:55.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:55.497 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:55.529 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:55.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:55.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:55.631 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:55.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:55.665 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:55.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:55.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:55 smithi028 ceph-mon[29442]: pgmap v933: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+scrubbing, 162 active+clean; 7.7 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 723 B/s rd, 56 MiB/s wr, 906 op/s 2024-02-05T06:52:55.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:55 smithi028 ceph-mon[29442]: osdmap e451: 8 total, 8 up, 8 in 2024-02-05T06:52:55.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:55 smithi028 ceph-mon[37984]: pgmap v933: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+scrubbing, 162 active+clean; 7.7 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 723 B/s rd, 56 MiB/s wr, 906 op/s 2024-02-05T06:52:55.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:55 smithi028 ceph-mon[37984]: osdmap e451: 8 total, 8 up, 8 in 2024-02-05T06:52:55.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:55 smithi167 ceph-mon[32774]: pgmap v933: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+scrubbing, 162 active+clean; 7.7 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 723 B/s rd, 56 MiB/s wr, 906 op/s 2024-02-05T06:52:55.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:55 smithi167 ceph-mon[32774]: osdmap e451: 8 total, 8 up, 8 in 2024-02-05T06:52:55.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:55.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:55.793 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:55.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:55.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:55.927 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:55.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:55.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:56.029 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:56.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:56.065 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:56.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:56.166 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:56.203 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:56.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:56.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:56.305 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:56.338 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:56.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:56.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:56.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:56.481 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:56.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:56.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:56.583 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:56.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:56.617 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:56.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:56.718 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:56.754 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:56.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:56.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:56.856 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:56.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:56.900 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:56.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:57.001 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:57.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:57.025 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:57.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:57.126 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:57.164 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:57.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:57.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:57.266 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:52:57.300 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:57.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:57.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:52:57.402 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:57.436 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:57.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:57.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:57.538 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:57.568 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:57.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:57.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:57.670 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:57.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:57.706 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:57.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:57.807 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:52:57.835 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:57.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:57.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:52:57.937 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:52:57.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:57 smithi028 ceph-mon[37984]: pgmap v935: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+scrubbing, 162 active+clean; 7.7 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 639 B/s rd, 31 MiB/s wr, 526 op/s 2024-02-05T06:52:57.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:57 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:52:57.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:57 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:52:57.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:57 smithi028 ceph-mon[29442]: pgmap v935: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+scrubbing, 162 active+clean; 7.7 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 639 B/s rd, 31 MiB/s wr, 526 op/s 2024-02-05T06:52:57.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:57 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:52:57.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:57 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:52:57.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:57.975 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:57.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:52:57.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:57 smithi167 ceph-mon[32774]: pgmap v935: 168 pgs: 2 active+clean+scrubbing+deep, 4 active+clean+scrubbing, 162 active+clean; 7.7 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 639 B/s rd, 31 MiB/s wr, 526 op/s 2024-02-05T06:52:57.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:57 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:52:57.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:57 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:52:58.076 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:58.115 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:58.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:58.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:58.217 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:58.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:58.241 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:58.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:58.342 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:52:58.374 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:58.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:58.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:52:58.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:58.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:58.514 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:58.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:58.615 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:58.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:58.652 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:58.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:58.753 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:58.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:58.806 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:58.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:58.907 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:58.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:58 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:52:58.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:58 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:52:58.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:58 smithi028 ceph-mon[29442]: osdmap e452: 8 total, 8 up, 8 in 2024-02-05T06:52:58.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:58 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:52:58.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:58 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:52:58.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:58 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:52:58.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:58 smithi028 ceph-mon[37984]: osdmap e452: 8 total, 8 up, 8 in 2024-02-05T06:52:58.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:58 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:52:58.952 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:58.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:58.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:58.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:58 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:52:58.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:58 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:52:58.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:58 smithi167 ceph-mon[32774]: osdmap e452: 8 total, 8 up, 8 in 2024-02-05T06:52:58.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:58 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:52:59.054 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:59.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:59.093 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:59.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:59.194 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:59.230 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:59.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:59.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:59.331 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:52:59.382 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:59.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:59.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:52:59.484 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:59.519 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:59.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:59.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:59.621 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:52:59.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:59.664 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:59.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:52:59.765 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:52:59.796 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:52:59.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:59.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:52:59.898 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:52:59.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:52:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:52:59.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:52:59.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:52:59 smithi028 ceph-mon[29442]: pgmap v937: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1023 B/s rd, 16 MiB/s wr, 941 op/s 2024-02-05T06:52:59.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:52:59 smithi028 ceph-mon[37984]: pgmap v937: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1023 B/s rd, 16 MiB/s wr, 941 op/s 2024-02-05T06:52:59.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:52:59 smithi167 ceph-mon[32774]: pgmap v937: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1023 B/s rd, 16 MiB/s wr, 941 op/s 2024-02-05T06:53:00.044 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:00.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:00.090 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:00.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:00.192 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:00.222 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:00.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:00.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:00.324 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:00.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:00.363 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:00.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:00.464 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:00.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:00.496 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:00.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:00.551 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:53:00.596 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:00.598 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:53:00.626 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:00.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:00.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:00.642 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:53:00.686 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:53:00.723 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:53:00.732 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:00.767 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:53:00.770 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:00.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:00.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:00.803 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:53:00.804 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:00 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:53:00.804 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:00 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:53:00.804 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:00 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:53:00.804 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:00 smithi028 ceph-mon[29442]: osdmap e453: 8 total, 8 up, 8 in 2024-02-05T06:53:00.804 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:00 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/130995537' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:53:00.804 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:00 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:53:00.804 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:00 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:53:00.804 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:00 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:53:00.804 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:00 smithi028 ceph-mon[37984]: osdmap e453: 8 total, 8 up, 8 in 2024-02-05T06:53:00.804 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:00 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/130995537' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:53:00.845 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:53:00.849 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:53:00.849 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:53:00.849 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:53:00.850 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:53:00.850 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:53:00.850 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:53:00.850 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:53:00.850 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:53:00.873 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:00.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:00.902 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:00.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:00.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:00 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:53:00.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:00 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:53:00.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:00 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:53:00.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:00 smithi167 ceph-mon[32774]: osdmap e453: 8 total, 8 up, 8 in 2024-02-05T06:53:00.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:00 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/130995537' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:53:01.003 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:01.041 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:01.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:01.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:01.142 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:01.175 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:01.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:01.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:01.277 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:01.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:01.307 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:01.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:01.408 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:01.438 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:01.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:01.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:01.540 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:01.591 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:01.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:01.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:01.693 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:01.733 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:01.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:01.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:01.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:01.873 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:01.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:01.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:01.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:01 smithi028 ceph-mon[29442]: pgmap v939: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1023 B/s rd, 170 B/s wr, 672 op/s 2024-02-05T06:53:01.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:01 smithi028 ceph-mon[37984]: pgmap v939: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1023 B/s rd, 170 B/s wr, 672 op/s 2024-02-05T06:53:01.975 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:01.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:01 smithi167 ceph-mon[32774]: pgmap v939: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.5 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1023 B/s rd, 170 B/s wr, 672 op/s 2024-02-05T06:53:02.006 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:02.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:02.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:02.108 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:02.140 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:02.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:02.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:02.241 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:02.276 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:02.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:02.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:02.378 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:02.409 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:02.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:02.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:02.511 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:02.547 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:02.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:02.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:02.649 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:02.685 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:02.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:02.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:02.787 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:02.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:02.821 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:02.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:02.849 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T06:53:02.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:02 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:53:02.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:02 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:53:02.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:02 smithi028 ceph-mon[29442]: osdmap e454: 8 total, 8 up, 8 in 2024-02-05T06:53:02.886 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:02 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:53:02.886 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:02 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:53:02.886 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:02 smithi028 ceph-mon[37984]: osdmap e454: 8 total, 8 up, 8 in 2024-02-05T06:53:02.887 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:53:02 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:53:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:53:02.922 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:02.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:02.970 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:02.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:02.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:02 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:53:02.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:02 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:53:02.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:02 smithi167 ceph-mon[32774]: osdmap e454: 8 total, 8 up, 8 in 2024-02-05T06:53:03.071 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:03.104 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:03.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:03.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:03.206 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:03.242 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:03.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:03.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:03.344 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:03.377 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:03.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:03.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:03.479 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:03.515 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:03.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:03.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:03.616 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:03.649 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:03.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:03.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:03.750 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:03.781 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:03.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:03.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:03.791 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:53:03.882 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:03.911 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:03.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:03.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:03.926 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:03 smithi028 ceph-mon[29442]: pgmap v941: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.4 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1.8 KiB/s rd, 170 B/s wr, 1.03k op/s 2024-02-05T06:53:03.926 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:03 smithi028 ceph-mon[37984]: pgmap v941: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.4 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1.8 KiB/s rd, 170 B/s wr, 1.03k op/s 2024-02-05T06:53:03.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:03 smithi167 ceph-mon[32774]: pgmap v941: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.4 GiB data, 28 GiB used, 687 GiB / 715 GiB avail; 1.8 KiB/s rd, 170 B/s wr, 1.03k op/s 2024-02-05T06:53:03.985 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:53:03 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:53:04.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:04.052 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:04.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:04.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:04.154 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:04.193 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:04.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:04.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:04.295 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:04.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:04.324 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:04.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:04.425 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:04.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:04.465 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:04.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:04.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:04.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:04.591 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:04.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:04.693 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:04.733 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:04.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:04.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:04.834 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:04.869 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:04.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:04.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:04.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:04 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:53:04.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:04 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:53:04.970 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:04.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:04 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:53:05.001 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:05.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:05.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:05.103 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:05.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:05.139 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:05.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:05.240 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:05.261 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:53:05.271 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:05.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:05.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:05.296 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.6 to 1 (10000) 2024-02-05T06:53:05.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:05.398 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:05.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:05.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:05.500 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:05.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:05.537 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:05.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:05.638 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:05.666 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:05.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:05.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:05.767 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:05.778 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:53:05.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:05.803 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:05.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:05.905 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:05.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:05.949 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:05.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:05.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:05 smithi028 ceph-mon[37984]: pgmap v942: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.2 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 737 B/s rd, 0 B/s wr, 771 op/s 2024-02-05T06:53:05.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:05 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/781496743' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T06:53:05.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:05 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/781496743' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-05T06:53:05.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:05 smithi028 ceph-mon[37984]: osdmap e455: 8 total, 8 up, 8 in 2024-02-05T06:53:05.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:05 smithi028 ceph-mon[29442]: pgmap v942: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.2 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 737 B/s rd, 0 B/s wr, 771 op/s 2024-02-05T06:53:05.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:05 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/781496743' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T06:53:05.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:05 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/781496743' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-05T06:53:05.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:05 smithi028 ceph-mon[29442]: osdmap e455: 8 total, 8 up, 8 in 2024-02-05T06:53:05.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:05 smithi167 ceph-mon[32774]: pgmap v942: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.2 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 737 B/s rd, 0 B/s wr, 771 op/s 2024-02-05T06:53:05.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:05 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/781496743' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T06:53:05.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:05 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/781496743' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-05T06:53:05.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:05 smithi167 ceph-mon[32774]: osdmap e455: 8 total, 8 up, 8 in 2024-02-05T06:53:06.050 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:06.082 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:06.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:06.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:06.184 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:06.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:06.217 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:06.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:06.317 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:06.353 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:06.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:06.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:06.455 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:06.488 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:06.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:06.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:06.590 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:06.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:06.624 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:06.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:06.725 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:06.757 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:06.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:06.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:06.859 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:06.894 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:06.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:06.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:06.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:06 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:53:06.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:06 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:53:06.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:06 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:53:06.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:06 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:53:06.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:06 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:53:06.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:06 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:53:06.996 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:07.030 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:07.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:07.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:07.133 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:07.173 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:07.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:07.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:07.275 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:07.306 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:07.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:07.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:07.408 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:07.451 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:07.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:07.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:07.553 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:07.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:07.585 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:07.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:07.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:07.745 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:07.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:07.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:07.847 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:07.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:07.883 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:07.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:07.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:07 smithi028 ceph-mon[37984]: pgmap v944: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 813 B/s rd, 0 B/s wr, 725 op/s 2024-02-05T06:53:07.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:07 smithi028 ceph-mon[37984]: osdmap e456: 8 total, 8 up, 8 in 2024-02-05T06:53:07.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:07 smithi028 ceph-mon[29442]: pgmap v944: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 813 B/s rd, 0 B/s wr, 725 op/s 2024-02-05T06:53:07.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:07 smithi028 ceph-mon[29442]: osdmap e456: 8 total, 8 up, 8 in 2024-02-05T06:53:07.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:07 smithi167 ceph-mon[32774]: pgmap v944: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.2 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 813 B/s rd, 0 B/s wr, 725 op/s 2024-02-05T06:53:07.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:07 smithi167 ceph-mon[32774]: osdmap e456: 8 total, 8 up, 8 in 2024-02-05T06:53:07.984 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:08.020 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:08.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:08.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:08.122 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:08.151 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:08.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:08.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:08.253 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:08.284 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:08.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:08.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:08.385 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:08.416 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:08.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:08.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:08.518 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:08.551 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:08.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:08.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:08.653 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:08.688 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:08.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:08.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:08.726 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd reweight 4 1 2024-02-05T06:53:08.790 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:08.825 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:08.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:08.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:08.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:09.002 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:09.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:09.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:09.104 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:09.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:09.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:09.259 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:09.295 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:09.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:09.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:09.397 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:09.432 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:09.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:09.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:09.534 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:09.573 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:09.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:09.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:09.675 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:09.707 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:09.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:09.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:09.809 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:09.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:09.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:09.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:09.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:09 smithi028 ceph-mon[29442]: pgmap v946: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 815 B/s rd, 0 B/s wr, 964 op/s 2024-02-05T06:53:09.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:09 smithi028 ceph-mon[37984]: pgmap v946: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 815 B/s rd, 0 B/s wr, 964 op/s 2024-02-05T06:53:09.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:09 smithi167 ceph-mon[32774]: pgmap v946: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 815 B/s rd, 0 B/s wr, 964 op/s 2024-02-05T06:53:09.985 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:09.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:09.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:10.087 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:10.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:10.128 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:10.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:10.229 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:10.262 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:10.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:10.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:10.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:10.397 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:10.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:10.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:10.499 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:10.538 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:10.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:10.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:10.640 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:10.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:10.669 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:10.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:10.770 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:10.820 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:10.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:10.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:10.921 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:10.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:10.956 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:10.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:11.057 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:11.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:11.113 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:11.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:11.214 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:11.247 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:11.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:11.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:11.349 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:11.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:11.384 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:11.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:11.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:11 smithi028 ceph-mon[29442]: osdmap e457: 8 total, 8 up, 8 in 2024-02-05T06:53:11.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:11 smithi028 ceph-mon[29442]: pgmap v948: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 691 op/s 2024-02-05T06:53:11.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:11 smithi028 ceph-mon[37984]: osdmap e457: 8 total, 8 up, 8 in 2024-02-05T06:53:11.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:11 smithi028 ceph-mon[37984]: pgmap v948: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 691 op/s 2024-02-05T06:53:11.486 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:11.528 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:11.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:11.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:11.629 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:11.640 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:53:11.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:11.715 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:11.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:11.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:11 smithi167 ceph-mon[32774]: osdmap e457: 8 total, 8 up, 8 in 2024-02-05T06:53:11.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:11 smithi167 ceph-mon[32774]: pgmap v948: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 7.0 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 691 op/s 2024-02-05T06:53:11.816 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:11.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:11.852 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:11.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:11.953 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:12.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:12.001 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:12.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:12.050 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:53:12.102 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:12.132 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:12.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:12.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:12.234 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:12.261 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:12.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:12.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:12.326 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-02-05T06:53:12.327 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:12 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/260253928' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:53:12.327 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:12 smithi028 ceph-mon[37984]: from='client.45977 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:53:12.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:12 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/260253928' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:53:12.327 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:12 smithi028 ceph-mon[29442]: from='client.45977 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:53:12.363 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:12.400 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:12.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:12.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:12.501 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:12.531 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:12.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:12.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:12.633 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:12.665 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:12.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:12.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:12.673 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:53:12 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:53:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:53:12.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:12 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/260253928' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:53:12.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:12 smithi167 ceph-mon[32774]: from='client.45977 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:53:12.768 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:12.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:12.806 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:12.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:12.907 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:12.940 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:12.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:12.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:13.042 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:13.080 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:13.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:13.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:13.182 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:13.217 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:13.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:13.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:13.319 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:13.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:13.362 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:13.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:13.463 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:13.506 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:13.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:13.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:13.608 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:13.643 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:13.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:13.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:13.646 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:13 smithi028 ceph-mon[37984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:53:13.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:13 smithi028 ceph-mon[37984]: from='client.45977 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:53:13.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:13 smithi028 ceph-mon[37984]: osdmap e458: 8 total, 8 up, 8 in 2024-02-05T06:53:13.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:13 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:53:13.647 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:13 smithi028 ceph-mon[37984]: pgmap v950: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-02-05T06:53:13.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:13 smithi028 ceph-mon[29442]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:53:13.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:13 smithi028 ceph-mon[29442]: from='client.45977 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:53:13.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:13 smithi028 ceph-mon[29442]: osdmap e458: 8 total, 8 up, 8 in 2024-02-05T06:53:13.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:13 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:53:13.647 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:13 smithi028 ceph-mon[29442]: pgmap v950: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-02-05T06:53:13.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:13 smithi167 ceph-mon[32774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:53:13.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:13 smithi167 ceph-mon[32774]: from='client.45977 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:53:13.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:13 smithi167 ceph-mon[32774]: osdmap e458: 8 total, 8 up, 8 in 2024-02-05T06:53:13.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:13 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:53:13.735 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:13 smithi167 ceph-mon[32774]: pgmap v950: 168 pgs: 2 active+clean+scrubbing+deep, 166 active+clean; 6.8 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-02-05T06:53:13.735 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:53:13 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:53:13.744 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:13.777 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:13.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:13.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:13.879 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:13.906 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:13.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:13.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:13.986 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:53:13.986 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:53:14.011 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:14.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:14.059 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:14.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:14.160 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:14.195 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:14.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:14.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:14.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:14.344 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:14.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:14.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:14.446 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:14.476 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:14.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:14.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:14.577 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:14.607 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:14.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:14.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:14.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:14 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:53:14.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:14 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:53:14.709 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:14.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:14 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:53:14.745 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:14.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:14.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:14.847 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:14.884 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:14.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:14.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:14.986 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:15.028 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:15.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:15.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:15.130 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:15.156 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:15.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:15.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:15.258 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:15.291 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:15.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:15.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:15.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:15.430 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:15.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:15.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:15.532 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:15.570 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:15.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:15.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:15.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:15.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:15 smithi028 ceph-mon[37984]: pgmap v951: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-02-05T06:53:15.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:15 smithi028 ceph-mon[37984]: osdmap e459: 8 total, 8 up, 8 in 2024-02-05T06:53:15.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:15 smithi028 ceph-mon[29442]: pgmap v951: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-02-05T06:53:15.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:15 smithi028 ceph-mon[29442]: osdmap e459: 8 total, 8 up, 8 in 2024-02-05T06:53:15.714 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:15.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:15.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:15.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:15 smithi167 ceph-mon[32774]: pgmap v951: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-02-05T06:53:15.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:15 smithi167 ceph-mon[32774]: osdmap e459: 8 total, 8 up, 8 in 2024-02-05T06:53:15.815 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:15.846 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:15.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:15.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:15.948 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:15.983 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:15.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:15.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:16.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:16.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:16.141 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:16.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:16.242 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:16.282 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:16.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:16.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:16.384 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:16.418 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:16.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:16.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:16.520 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:16.554 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:16.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:16.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:16.656 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:16.688 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:16.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:16.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:16.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:16.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:16.835 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:16.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:16.936 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:16.975 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:16.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:16.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:17.076 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:17.095 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-02-05T06:53:17.104 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:17.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:17.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:17.206 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:17.262 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:17.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:17.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:17.309 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T06:53:17.364 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:17.437 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:17.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:17.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:17.539 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:17.568 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:17.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:17.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:17.670 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:17.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:17.703 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:17.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:17.804 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:17.844 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:17.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:17.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:17.941 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:17 smithi028 ceph-mon[37984]: pgmap v953: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 812 B/s rd, 0 B/s wr, 775 op/s 2024-02-05T06:53:17.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:17 smithi028 ceph-mon[29442]: pgmap v953: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 812 B/s rd, 0 B/s wr, 775 op/s 2024-02-05T06:53:17.946 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:17.975 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:17.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:17.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:17.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:17 smithi167 ceph-mon[32774]: pgmap v953: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 812 B/s rd, 0 B/s wr, 775 op/s 2024-02-05T06:53:18.077 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:18.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:18.105 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:18.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:18.206 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:18.250 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:18.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:18.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:18.353 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:18.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:18.399 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:18.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:18.439 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:53:18.500 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:18.518 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:53:18.550 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:53:18.573 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:18.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:18.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:18.593 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:53:18.634 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:53:18.671 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.4 to 1 (10000) 2024-02-05T06:53:18.674 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:18.686 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:53:18.724 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:18.724 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:53:18.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:18.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:18.766 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:53:18.808 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:53:18.826 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:18.848 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:53:18.848 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:18 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/134461873' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T06:53:18.848 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:18 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/52185453' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:53:18.848 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:18 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/134461873' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T06:53:18.848 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:18 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/52185453' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:53:18.855 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:53:18.855 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:53:18.855 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:53:18.855 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:53:18.855 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:53:18.856 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:53:18.856 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:53:18.856 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:53:18.866 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:18.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:18.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:18.968 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:18.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:18 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/134461873' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T06:53:18.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:18 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/52185453' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:53:18.999 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:19.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:19.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:19.101 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:19.135 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:19.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:19.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:19.237 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:19.265 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:19.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:19.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:19.367 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:19.405 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:19.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:19.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:19.507 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:19.549 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:19.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:19.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:19.597 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd reweight 7 1 2024-02-05T06:53:19.651 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:19.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:19.678 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:19.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:19.780 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:19.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:19.813 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:19.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:19.914 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:19.946 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:19.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:19.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:19.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:19 smithi028 ceph-mon[29442]: pgmap v954: 168 pgs: 168 active+clean; 6.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-02-05T06:53:19.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:19 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/134461873' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-05T06:53:19.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:19 smithi028 ceph-mon[29442]: osdmap e460: 8 total, 8 up, 8 in 2024-02-05T06:53:19.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:19 smithi028 ceph-mon[37984]: pgmap v954: 168 pgs: 168 active+clean; 6.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-02-05T06:53:19.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:19 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/134461873' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-05T06:53:19.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:19 smithi028 ceph-mon[37984]: osdmap e460: 8 total, 8 up, 8 in 2024-02-05T06:53:19.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:19 smithi167 ceph-mon[32774]: pgmap v954: 168 pgs: 168 active+clean; 6.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-02-05T06:53:19.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:19 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/134461873' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-05T06:53:19.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:19 smithi167 ceph-mon[32774]: osdmap e460: 8 total, 8 up, 8 in 2024-02-05T06:53:20.048 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:20.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:20.097 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:20.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:20.197 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:20.229 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:20.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:20.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:20.331 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:20.428 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:20.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:20.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:20.530 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:20.564 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:20.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:20.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:20.666 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:20.696 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:20.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:20.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:20.798 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:20.835 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:20.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:20.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:20.937 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:20.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:20.971 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:20.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:21.072 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:21.111 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:21.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:21.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:21.212 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:21.243 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:21.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:21.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:21.345 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:21.391 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:21.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:21.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:21.493 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:21.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:21.530 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:21.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:21.631 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:21.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:21.660 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:21.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:21.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:21 smithi028 ceph-mon[29442]: osdmap e461: 8 total, 8 up, 8 in 2024-02-05T06:53:21.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:21 smithi028 ceph-mon[29442]: pgmap v957: 168 pgs: 168 active+clean; 6.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.01k op/s 2024-02-05T06:53:21.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:21 smithi028 ceph-mon[37984]: osdmap e461: 8 total, 8 up, 8 in 2024-02-05T06:53:21.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:21 smithi028 ceph-mon[37984]: pgmap v957: 168 pgs: 168 active+clean; 6.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.01k op/s 2024-02-05T06:53:21.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:21 smithi167 ceph-mon[32774]: osdmap e461: 8 total, 8 up, 8 in 2024-02-05T06:53:21.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:21 smithi167 ceph-mon[32774]: pgmap v957: 168 pgs: 168 active+clean; 6.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.01k op/s 2024-02-05T06:53:21.762 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:21.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:21.799 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:21.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:21.900 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:21.931 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:21.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:21.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:22.033 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:22.065 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:22.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:22.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:22.167 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:22.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:22.200 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:22.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:22.301 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:22.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:22.344 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:22.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:22.445 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:22.474 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:22.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:22.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:22.575 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:22.603 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:22.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:22.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:22.709 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:22.754 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:22.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:53:22 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:53:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:53:22.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:22.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:22.775 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:53:22.856 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:22.892 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:22.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:22.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:22.994 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:23.045 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:23.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:23.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:23.148 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:23.176 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:23.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:23.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:23.278 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:23.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:23.332 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:23.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:23.434 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:23.470 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:23.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:23.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:23.572 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:23.612 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:23.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:23.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:23.713 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:23.743 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:23.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:23.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:23.773 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:53:23.845 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:23.876 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:23.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:23.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:23.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:23 smithi028 ceph-mon[37984]: pgmap v958: 168 pgs: 168 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-02-05T06:53:23.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:23 smithi028 ceph-mon[29442]: pgmap v958: 168 pgs: 168 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-02-05T06:53:23.978 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:23.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:23 smithi167 ceph-mon[32774]: pgmap v958: 168 pgs: 168 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-02-05T06:53:23.984 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:53:23 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:53:24.010 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:24.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:24.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:24.111 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:24.139 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:24.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:24.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:24.241 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:24.305 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:24.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:24.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:24.406 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:24.438 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:24.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:24.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:24.539 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:24.578 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:24.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:24.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:24.680 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:24.718 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:24.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:24.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:24.820 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:24.841 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:24 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:53:24.842 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:24 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:53:24.863 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:24.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:24.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:24.965 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:24.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:24 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:53:25.005 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:25.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:25.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:25.107 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:25.137 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:25.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:25.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:25.152 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:53:25.152 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-02-05T06:53:25.152 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-02-05T06:53:25.152 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-02-05T06:53:25.152 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:53:25.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:25.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:25.281 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:25.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:25.382 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:25.419 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:25.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:25.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:25.521 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:25.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:25.562 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:25.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:25.663 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:25.697 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:25.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:25.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:25.700 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-02-05T06:53:25.799 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:25.830 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:25.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:25.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:25.839 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:25 smithi028 ceph-mon[37984]: pgmap v959: 168 pgs: 168 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-02-05T06:53:25.839 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:25 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/263466440' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:53:25.840 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:25 smithi028 ceph-mon[29442]: pgmap v959: 168 pgs: 168 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-02-05T06:53:25.840 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:25 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/263466440' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:53:25.932 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:25.960 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:25.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:25.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:25.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:25 smithi167 ceph-mon[32774]: pgmap v959: 168 pgs: 168 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-02-05T06:53:25.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:25 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/263466440' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:53:26.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:26.099 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:26.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:26.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:26.201 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:26.238 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:26.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:26.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:26.339 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:26.344 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-02-05T06:53:26.376 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:26.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:26.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:26.478 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:26.521 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:26.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:26.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:26.623 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:26.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:26.657 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:26.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:26.758 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:26.801 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:26.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:26.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:26.903 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:26.937 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:26.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:26.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:26.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:26 smithi028 ceph-mon[37984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:53:26.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:26 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/263466440' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:53:26.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:26 smithi028 ceph-mon[37984]: osdmap e462: 8 total, 8 up, 8 in 2024-02-05T06:53:26.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:26 smithi028 ceph-mon[29442]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:53:26.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:26 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/263466440' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:53:26.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:26 smithi028 ceph-mon[29442]: osdmap e462: 8 total, 8 up, 8 in 2024-02-05T06:53:26.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:26 smithi167 ceph-mon[32774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:53:26.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:26 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/263466440' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:53:26.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:26 smithi167 ceph-mon[32774]: osdmap e462: 8 total, 8 up, 8 in 2024-02-05T06:53:27.039 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:27.074 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:27.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:27.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:27.176 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:27.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:27.210 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:27.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:27.311 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:27.351 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:27.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:27.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:27.453 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:27.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:27.499 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:27.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:27.601 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:27.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:27.640 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:27.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:27.741 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:27.773 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:27.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:27.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:27.875 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:27.910 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:27.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:27.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:27.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:27 smithi028 ceph-mon[37984]: 4.18 scrub starts 2024-02-05T06:53:27.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:27 smithi028 ceph-mon[37984]: 4.18 scrub ok 2024-02-05T06:53:27.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:27 smithi028 ceph-mon[37984]: 4.14 scrub starts 2024-02-05T06:53:27.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:27 smithi028 ceph-mon[37984]: 4.14 scrub ok 2024-02-05T06:53:27.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:27 smithi028 ceph-mon[37984]: pgmap v961: 168 pgs: 168 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 644 B/s rd, 0 B/s wr, 792 op/s 2024-02-05T06:53:27.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:27 smithi028 ceph-mon[37984]: 11.3 scrub starts 2024-02-05T06:53:27.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:27 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:53:27.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:27 smithi028 ceph-mon[29442]: 4.18 scrub starts 2024-02-05T06:53:27.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:27 smithi028 ceph-mon[29442]: 4.18 scrub ok 2024-02-05T06:53:27.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:27 smithi028 ceph-mon[29442]: 4.14 scrub starts 2024-02-05T06:53:27.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:27 smithi028 ceph-mon[29442]: 4.14 scrub ok 2024-02-05T06:53:27.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:27 smithi028 ceph-mon[29442]: pgmap v961: 168 pgs: 168 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 644 B/s rd, 0 B/s wr, 792 op/s 2024-02-05T06:53:27.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:27 smithi028 ceph-mon[29442]: 11.3 scrub starts 2024-02-05T06:53:27.951 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:27 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:53:27.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:27 smithi167 ceph-mon[32774]: 4.18 scrub starts 2024-02-05T06:53:27.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:27 smithi167 ceph-mon[32774]: 4.18 scrub ok 2024-02-05T06:53:27.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:27 smithi167 ceph-mon[32774]: 4.14 scrub starts 2024-02-05T06:53:27.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:27 smithi167 ceph-mon[32774]: 4.14 scrub ok 2024-02-05T06:53:27.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:27 smithi167 ceph-mon[32774]: pgmap v961: 168 pgs: 168 active+clean; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 644 B/s rd, 0 B/s wr, 792 op/s 2024-02-05T06:53:27.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:27 smithi167 ceph-mon[32774]: 11.3 scrub starts 2024-02-05T06:53:27.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:27 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:53:28.012 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:28.051 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:28.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:28.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:28.153 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:28.186 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:28.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:28.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:28.287 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:28.325 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:28.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:28.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:28.426 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:28.468 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:28.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:28.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:28.569 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:28.609 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:28.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:28.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:28.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:28.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:28.752 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:28.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:28.853 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:28.877 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:28.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:28.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:28.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:28 smithi028 ceph-mon[37984]: 1.2 scrub starts 2024-02-05T06:53:28.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:28 smithi028 ceph-mon[37984]: 1.2 scrub ok 2024-02-05T06:53:28.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:28 smithi028 ceph-mon[37984]: 4.1b scrub starts 2024-02-05T06:53:28.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:28 smithi028 ceph-mon[37984]: 4.1b scrub ok 2024-02-05T06:53:28.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:28 smithi028 ceph-mon[37984]: 11.d scrub starts 2024-02-05T06:53:28.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:28 smithi028 ceph-mon[37984]: 6.12 scrub starts 2024-02-05T06:53:28.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:28 smithi028 ceph-mon[29442]: 1.2 scrub starts 2024-02-05T06:53:28.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:28 smithi028 ceph-mon[29442]: 1.2 scrub ok 2024-02-05T06:53:28.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:28 smithi028 ceph-mon[29442]: 4.1b scrub starts 2024-02-05T06:53:28.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:28 smithi028 ceph-mon[29442]: 4.1b scrub ok 2024-02-05T06:53:28.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:28 smithi028 ceph-mon[29442]: 11.d scrub starts 2024-02-05T06:53:28.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:28 smithi028 ceph-mon[29442]: 6.12 scrub starts 2024-02-05T06:53:28.978 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:28.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:28 smithi167 ceph-mon[32774]: 1.2 scrub starts 2024-02-05T06:53:28.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:28 smithi167 ceph-mon[32774]: 1.2 scrub ok 2024-02-05T06:53:28.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:28 smithi167 ceph-mon[32774]: 4.1b scrub starts 2024-02-05T06:53:28.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:28 smithi167 ceph-mon[32774]: 4.1b scrub ok 2024-02-05T06:53:28.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:28 smithi167 ceph-mon[32774]: 11.d scrub starts 2024-02-05T06:53:28.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:28 smithi167 ceph-mon[32774]: 6.12 scrub starts 2024-02-05T06:53:29.007 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:29.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:29.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:29.109 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:29.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:29.146 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:29.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:29.156 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T06:53:29.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:29.297 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:29.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:29.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:29.398 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:53:29.399 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:29.454 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:29.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:29.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:29.555 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:29.609 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:29.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:29.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:29.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:29.753 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:29.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:29.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:29.854 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:29.888 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:29.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:29.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:29.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:29 smithi167 ceph-mon[32774]: 6.12 scrub ok 2024-02-05T06:53:29.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:29 smithi167 ceph-mon[32774]: 6.2 scrub starts 2024-02-05T06:53:29.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:29 smithi167 ceph-mon[32774]: 6.2 scrub ok 2024-02-05T06:53:29.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:29 smithi167 ceph-mon[32774]: pgmap v962: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-02-05T06:53:29.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:29 smithi167 ceph-mon[32774]: 11.3 scrub ok 2024-02-05T06:53:29.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:29 smithi167 ceph-mon[32774]: 6.f scrub starts 2024-02-05T06:53:29.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:29 smithi167 ceph-mon[32774]: 6.f scrub ok 2024-02-05T06:53:29.991 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:30.027 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:30.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:30.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:30.129 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:30.153 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:30.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:30.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:30.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:29 smithi028 ceph-mon[29442]: 6.12 scrub ok 2024-02-05T06:53:30.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:29 smithi028 ceph-mon[29442]: 6.2 scrub starts 2024-02-05T06:53:30.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:29 smithi028 ceph-mon[29442]: 6.2 scrub ok 2024-02-05T06:53:30.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:29 smithi028 ceph-mon[29442]: pgmap v962: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-02-05T06:53:30.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:29 smithi028 ceph-mon[29442]: 11.3 scrub ok 2024-02-05T06:53:30.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:29 smithi028 ceph-mon[29442]: 6.f scrub starts 2024-02-05T06:53:30.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:29 smithi028 ceph-mon[29442]: 6.f scrub ok 2024-02-05T06:53:30.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:29 smithi028 ceph-mon[37984]: 6.12 scrub ok 2024-02-05T06:53:30.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:29 smithi028 ceph-mon[37984]: 6.2 scrub starts 2024-02-05T06:53:30.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:29 smithi028 ceph-mon[37984]: 6.2 scrub ok 2024-02-05T06:53:30.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:29 smithi028 ceph-mon[37984]: pgmap v962: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-02-05T06:53:30.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:29 smithi028 ceph-mon[37984]: 11.3 scrub ok 2024-02-05T06:53:30.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:29 smithi028 ceph-mon[37984]: 6.f scrub starts 2024-02-05T06:53:30.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:29 smithi028 ceph-mon[37984]: 6.f scrub ok 2024-02-05T06:53:30.255 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:30.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:30.295 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:30.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:30.396 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:30.438 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:30.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:30.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:30.540 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:30.571 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:30.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:30.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:30.673 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:30.709 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:30.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:30.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:30.811 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:30.847 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:30.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:30.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:30.949 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:30.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:30 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:53:30.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:30 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:53:30.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:30 smithi167 ceph-mon[32774]: osdmap e463: 8 total, 8 up, 8 in 2024-02-05T06:53:30.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:30 smithi167 ceph-mon[32774]: 11.1 scrub starts 2024-02-05T06:53:30.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:30 smithi167 ceph-mon[32774]: 11.d scrub ok 2024-02-05T06:53:30.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:30 smithi167 ceph-mon[32774]: 3.1b scrub starts 2024-02-05T06:53:30.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:30 smithi167 ceph-mon[32774]: 3.1b scrub ok 2024-02-05T06:53:30.993 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:30.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:30.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:31.094 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:31.118 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:31.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:31.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:31.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:30 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:53:31.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:30 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:53:31.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:30 smithi028 ceph-mon[29442]: osdmap e463: 8 total, 8 up, 8 in 2024-02-05T06:53:31.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:30 smithi028 ceph-mon[29442]: 11.1 scrub starts 2024-02-05T06:53:31.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:30 smithi028 ceph-mon[29442]: 11.d scrub ok 2024-02-05T06:53:31.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:30 smithi028 ceph-mon[29442]: 3.1b scrub starts 2024-02-05T06:53:31.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:30 smithi028 ceph-mon[29442]: 3.1b scrub ok 2024-02-05T06:53:31.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:30 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:53:31.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:30 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:53:31.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:30 smithi028 ceph-mon[37984]: osdmap e463: 8 total, 8 up, 8 in 2024-02-05T06:53:31.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:30 smithi028 ceph-mon[37984]: 11.1 scrub starts 2024-02-05T06:53:31.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:30 smithi028 ceph-mon[37984]: 11.d scrub ok 2024-02-05T06:53:31.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:30 smithi028 ceph-mon[37984]: 3.1b scrub starts 2024-02-05T06:53:31.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:30 smithi028 ceph-mon[37984]: 3.1b scrub ok 2024-02-05T06:53:31.220 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:31.257 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:31.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:31.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:31.359 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:31.393 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:31.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:31.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:31.495 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:31.527 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:31.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:31.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:31.628 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:31.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:31.673 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:31.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:31.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:31.797 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:31.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:31.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:31.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:31.936 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:31.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:31.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:31.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:31 smithi167 ceph-mon[32774]: 6.1b scrub starts 2024-02-05T06:53:31.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:31 smithi167 ceph-mon[32774]: pgmap v964: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.09k op/s 2024-02-05T06:53:31.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:31 smithi167 ceph-mon[32774]: 6.1b scrub ok 2024-02-05T06:53:32.037 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:32.077 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:32.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:32.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:32.179 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:32.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:31 smithi028 ceph-mon[29442]: 6.1b scrub starts 2024-02-05T06:53:32.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:31 smithi028 ceph-mon[29442]: pgmap v964: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.09k op/s 2024-02-05T06:53:32.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:31 smithi028 ceph-mon[29442]: 6.1b scrub ok 2024-02-05T06:53:32.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:31 smithi028 ceph-mon[37984]: 6.1b scrub starts 2024-02-05T06:53:32.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:31 smithi028 ceph-mon[37984]: pgmap v964: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.09k op/s 2024-02-05T06:53:32.201 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:31 smithi028 ceph-mon[37984]: 6.1b scrub ok 2024-02-05T06:53:32.216 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:32.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:32.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:32.318 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:32.342 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:32.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:32.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:32.444 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:32.480 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:32.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:32.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:32.582 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:32.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:32.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:32.724 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:32.754 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:53:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:53:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:53:32.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:32 smithi028 ceph-mon[29442]: 6.11 scrub starts 2024-02-05T06:53:32.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:32 smithi028 ceph-mon[29442]: 6.11 scrub ok 2024-02-05T06:53:32.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:32 smithi028 ceph-mon[29442]: 4.e scrub starts 2024-02-05T06:53:32.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:32 smithi028 ceph-mon[29442]: 4.e scrub ok 2024-02-05T06:53:32.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:32 smithi028 ceph-mon[29442]: 11.1 scrub ok 2024-02-05T06:53:32.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:32 smithi028 ceph-mon[29442]: 11.4 scrub starts 2024-02-05T06:53:32.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:32 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:53:32.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:32 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:53:32.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:32 smithi028 ceph-mon[29442]: 11.0 scrub starts 2024-02-05T06:53:32.773 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:32.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:32.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:32.875 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:32.906 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:32.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:32.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:33.008 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:33.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:33.056 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:33.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:33.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:33.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:33.182 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:33.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:33.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:32 smithi028 ceph-mon[37984]: 6.11 scrub starts 2024-02-05T06:53:33.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:32 smithi028 ceph-mon[37984]: 6.11 scrub ok 2024-02-05T06:53:33.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:32 smithi028 ceph-mon[37984]: 4.e scrub starts 2024-02-05T06:53:33.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:32 smithi028 ceph-mon[37984]: 4.e scrub ok 2024-02-05T06:53:33.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:32 smithi028 ceph-mon[37984]: 11.1 scrub ok 2024-02-05T06:53:33.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:32 smithi028 ceph-mon[37984]: 11.4 scrub starts 2024-02-05T06:53:33.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:32 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:53:33.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:32 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:53:33.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:32 smithi028 ceph-mon[37984]: 11.0 scrub starts 2024-02-05T06:53:33.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:32 smithi167 ceph-mon[32774]: 6.11 scrub starts 2024-02-05T06:53:33.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:32 smithi167 ceph-mon[32774]: 6.11 scrub ok 2024-02-05T06:53:33.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:32 smithi167 ceph-mon[32774]: 4.e scrub starts 2024-02-05T06:53:33.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:32 smithi167 ceph-mon[32774]: 4.e scrub ok 2024-02-05T06:53:33.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:32 smithi167 ceph-mon[32774]: 11.1 scrub ok 2024-02-05T06:53:33.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:32 smithi167 ceph-mon[32774]: 11.4 scrub starts 2024-02-05T06:53:33.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:32 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:53:33.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:32 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:53:33.236 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:32 smithi167 ceph-mon[32774]: 11.0 scrub starts 2024-02-05T06:53:33.283 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:33.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:33.323 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:33.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:33.424 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:33.456 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:33.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:33.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:33.558 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:33.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:33.582 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:33.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:33.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:33.706 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:33.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:33.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:33.808 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:33.845 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:33.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:33.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:33.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:33.983 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:33.984 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:53:33 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:53:33.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:33 smithi167 ceph-mon[32774]: pgmap v965: 168 pgs: 168 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-02-05T06:53:33.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:33 smithi167 ceph-mon[32774]: osdmap e464: 8 total, 8 up, 8 in 2024-02-05T06:53:33.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:33 smithi167 ceph-mon[32774]: 3.5 scrub starts 2024-02-05T06:53:33.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:33 smithi167 ceph-mon[32774]: 3.5 scrub ok 2024-02-05T06:53:33.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:33 smithi167 ceph-mon[32774]: 4.13 scrub starts 2024-02-05T06:53:33.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:33 smithi167 ceph-mon[32774]: 4.13 scrub ok 2024-02-05T06:53:33.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:33 smithi167 ceph-mon[32774]: 4.1c scrub starts 2024-02-05T06:53:33.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:33 smithi167 ceph-mon[32774]: 4.1c scrub ok 2024-02-05T06:53:33.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:33 smithi167 ceph-mon[32774]: 3.1 scrub starts 2024-02-05T06:53:33.985 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:33 smithi167 ceph-mon[32774]: 3.1 scrub ok 2024-02-05T06:53:33.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:33.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:34.021 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:33 smithi028 ceph-mon[29442]: pgmap v965: 168 pgs: 168 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-02-05T06:53:34.021 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:33 smithi028 ceph-mon[29442]: osdmap e464: 8 total, 8 up, 8 in 2024-02-05T06:53:34.021 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:33 smithi028 ceph-mon[29442]: 3.5 scrub starts 2024-02-05T06:53:34.021 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:33 smithi028 ceph-mon[29442]: 3.5 scrub ok 2024-02-05T06:53:34.021 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:33 smithi028 ceph-mon[29442]: 4.13 scrub starts 2024-02-05T06:53:34.021 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:33 smithi028 ceph-mon[29442]: 4.13 scrub ok 2024-02-05T06:53:34.021 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:33 smithi028 ceph-mon[29442]: 4.1c scrub starts 2024-02-05T06:53:34.021 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:33 smithi028 ceph-mon[29442]: 4.1c scrub ok 2024-02-05T06:53:34.021 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:33 smithi028 ceph-mon[29442]: 3.1 scrub starts 2024-02-05T06:53:34.021 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:33 smithi028 ceph-mon[29442]: 3.1 scrub ok 2024-02-05T06:53:34.022 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:33 smithi028 ceph-mon[37984]: pgmap v965: 168 pgs: 168 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-02-05T06:53:34.022 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:33 smithi028 ceph-mon[37984]: osdmap e464: 8 total, 8 up, 8 in 2024-02-05T06:53:34.022 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:33 smithi028 ceph-mon[37984]: 3.5 scrub starts 2024-02-05T06:53:34.022 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:33 smithi028 ceph-mon[37984]: 3.5 scrub ok 2024-02-05T06:53:34.022 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:33 smithi028 ceph-mon[37984]: 4.13 scrub starts 2024-02-05T06:53:34.022 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:33 smithi028 ceph-mon[37984]: 4.13 scrub ok 2024-02-05T06:53:34.022 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:33 smithi028 ceph-mon[37984]: 4.1c scrub starts 2024-02-05T06:53:34.022 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:33 smithi028 ceph-mon[37984]: 4.1c scrub ok 2024-02-05T06:53:34.022 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:33 smithi028 ceph-mon[37984]: 3.1 scrub starts 2024-02-05T06:53:34.022 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:33 smithi028 ceph-mon[37984]: 3.1 scrub ok 2024-02-05T06:53:34.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:34.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:34.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:34.218 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:34.242 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:34.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:34.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:34.344 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:34.368 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:34.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:34.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:34.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:34.507 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:34.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:34.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:34.609 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:34.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:34.646 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:34.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:34.747 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:34.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:34.789 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:34.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:34.890 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:34.917 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:34.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:34.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:35.017 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:34 smithi028 ceph-mon[37984]: 11.4 scrub ok 2024-02-05T06:53:35.017 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:34 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:53:35.017 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:34 smithi028 ceph-mon[37984]: 5.10 scrub starts 2024-02-05T06:53:35.017 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:34 smithi028 ceph-mon[37984]: 5.10 scrub ok 2024-02-05T06:53:35.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:34 smithi028 ceph-mon[29442]: 11.4 scrub ok 2024-02-05T06:53:35.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:34 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:53:35.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:34 smithi028 ceph-mon[29442]: 5.10 scrub starts 2024-02-05T06:53:35.018 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:34 smithi028 ceph-mon[29442]: 5.10 scrub ok 2024-02-05T06:53:35.019 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:35.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:35.053 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:35.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:35.154 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:35.186 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:35.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:35.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:35.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:34 smithi167 ceph-mon[32774]: 11.4 scrub ok 2024-02-05T06:53:35.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:34 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:53:35.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:34 smithi167 ceph-mon[32774]: 5.10 scrub starts 2024-02-05T06:53:35.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:34 smithi167 ceph-mon[32774]: 5.10 scrub ok 2024-02-05T06:53:35.289 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:35.338 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:35.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:35.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:35.440 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:35.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:35.493 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:35.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:35.586 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:53:35.594 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:35.667 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:35.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:35.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:35.768 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:35.798 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:35.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:35.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:35.900 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:35.932 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:35.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:35.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:36.034 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:36.072 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:36.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:36.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:36.174 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:36.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:35 smithi028 ceph-mon[29442]: 11.0 scrub ok 2024-02-05T06:53:36.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:35 smithi028 ceph-mon[29442]: 5.1b scrub starts 2024-02-05T06:53:36.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:35 smithi028 ceph-mon[29442]: 5.1b scrub ok 2024-02-05T06:53:36.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:35 smithi028 ceph-mon[29442]: pgmap v967: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-02-05T06:53:36.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:35 smithi028 ceph-mon[29442]: 11.e scrub starts 2024-02-05T06:53:36.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:35 smithi028 ceph-mon[29442]: 6.19 scrub starts 2024-02-05T06:53:36.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:35 smithi028 ceph-mon[29442]: 6.19 scrub ok 2024-02-05T06:53:36.201 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:35 smithi028 ceph-mon[29442]: 11.f scrub starts 2024-02-05T06:53:36.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:35 smithi028 ceph-mon[37984]: 11.0 scrub ok 2024-02-05T06:53:36.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:35 smithi028 ceph-mon[37984]: 5.1b scrub starts 2024-02-05T06:53:36.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:35 smithi028 ceph-mon[37984]: 5.1b scrub ok 2024-02-05T06:53:36.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:35 smithi028 ceph-mon[37984]: pgmap v967: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-02-05T06:53:36.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:35 smithi028 ceph-mon[37984]: 11.e scrub starts 2024-02-05T06:53:36.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:35 smithi028 ceph-mon[37984]: 6.19 scrub starts 2024-02-05T06:53:36.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:35 smithi028 ceph-mon[37984]: 6.19 scrub ok 2024-02-05T06:53:36.202 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:35 smithi028 ceph-mon[37984]: 11.f scrub starts 2024-02-05T06:53:36.215 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:36.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:36.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:36.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:35 smithi167 ceph-mon[32774]: 11.0 scrub ok 2024-02-05T06:53:36.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:35 smithi167 ceph-mon[32774]: 5.1b scrub starts 2024-02-05T06:53:36.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:35 smithi167 ceph-mon[32774]: 5.1b scrub ok 2024-02-05T06:53:36.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:35 smithi167 ceph-mon[32774]: pgmap v967: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-02-05T06:53:36.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:35 smithi167 ceph-mon[32774]: 11.e scrub starts 2024-02-05T06:53:36.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:35 smithi167 ceph-mon[32774]: 6.19 scrub starts 2024-02-05T06:53:36.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:35 smithi167 ceph-mon[32774]: 6.19 scrub ok 2024-02-05T06:53:36.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:35 smithi167 ceph-mon[32774]: 11.f scrub starts 2024-02-05T06:53:36.317 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:36.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:36.355 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:36.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:53:36.455 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:36.482 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:36.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:36.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:36.584 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:36.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:36.626 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:36.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:36.727 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:36.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:36.756 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:36.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.7 to 1 (10000) 2024-02-05T06:53:36.857 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:36.905 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:36.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:36.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:36.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:36 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2912128227' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T06:53:36.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:36 smithi028 ceph-mon[29442]: from='client.25969 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T06:53:36.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:36 smithi028 ceph-mon[29442]: 6.1e scrub starts 2024-02-05T06:53:36.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:36 smithi028 ceph-mon[29442]: 6.1e scrub ok 2024-02-05T06:53:36.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:36 smithi028 ceph-mon[29442]: 3.e scrub starts 2024-02-05T06:53:36.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:36 smithi028 ceph-mon[29442]: 3.e scrub ok 2024-02-05T06:53:36.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:36 smithi028 ceph-mon[29442]: 5.5 scrub starts 2024-02-05T06:53:36.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:36 smithi028 ceph-mon[29442]: 5.5 scrub ok 2024-02-05T06:53:36.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:36 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2912128227' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T06:53:36.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:36 smithi028 ceph-mon[37984]: from='client.25969 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T06:53:36.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:36 smithi028 ceph-mon[37984]: 6.1e scrub starts 2024-02-05T06:53:36.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:36 smithi028 ceph-mon[37984]: 6.1e scrub ok 2024-02-05T06:53:36.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:36 smithi028 ceph-mon[37984]: 3.e scrub starts 2024-02-05T06:53:36.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:36 smithi028 ceph-mon[37984]: 3.e scrub ok 2024-02-05T06:53:36.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:36 smithi028 ceph-mon[37984]: 5.5 scrub starts 2024-02-05T06:53:36.951 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:36 smithi028 ceph-mon[37984]: 5.5 scrub ok 2024-02-05T06:53:37.007 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:37.036 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:37.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:37.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:37.138 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:37.173 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:37.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:37.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:37.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:36 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2912128227' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T06:53:37.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:36 smithi167 ceph-mon[32774]: from='client.25969 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T06:53:37.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:36 smithi167 ceph-mon[32774]: 6.1e scrub starts 2024-02-05T06:53:37.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:36 smithi167 ceph-mon[32774]: 6.1e scrub ok 2024-02-05T06:53:37.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:36 smithi167 ceph-mon[32774]: 3.e scrub starts 2024-02-05T06:53:37.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:36 smithi167 ceph-mon[32774]: 3.e scrub ok 2024-02-05T06:53:37.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:36 smithi167 ceph-mon[32774]: 5.5 scrub starts 2024-02-05T06:53:37.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:36 smithi167 ceph-mon[32774]: 5.5 scrub ok 2024-02-05T06:53:37.275 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:37.321 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:37.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:37.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:37.423 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:37.464 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:37.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:37.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:37.520 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:53:37.559 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:53:37.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:37.600 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:53:37.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:37.609 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:37.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:37.646 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:53:37.687 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:53:37.710 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:37.738 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:53:37.768 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:37.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:37.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:37.780 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:53:37.823 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:53:37.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:37 smithi028 ceph-mon[37984]: pgmap v968: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 679 op/s 2024-02-05T06:53:37.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:37 smithi028 ceph-mon[37984]: 11.e scrub ok 2024-02-05T06:53:37.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:37 smithi028 ceph-mon[37984]: 11.f scrub ok 2024-02-05T06:53:37.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:37 smithi028 ceph-mon[37984]: from='client.25969 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-05T06:53:37.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:37 smithi028 ceph-mon[37984]: osdmap e465: 8 total, 8 up, 8 in 2024-02-05T06:53:37.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:37 smithi028 ceph-mon[37984]: 4.8 scrub starts 2024-02-05T06:53:37.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:37 smithi028 ceph-mon[37984]: 4.8 scrub ok 2024-02-05T06:53:37.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:37 smithi028 ceph-mon[37984]: 5.18 scrub starts 2024-02-05T06:53:37.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:37 smithi028 ceph-mon[37984]: 5.18 scrub ok 2024-02-05T06:53:37.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:37 smithi028 ceph-mon[37984]: 4.16 scrub starts 2024-02-05T06:53:37.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:37 smithi028 ceph-mon[37984]: 4.16 scrub ok 2024-02-05T06:53:37.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:37 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2412136041' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:53:37.824 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:37 smithi028 ceph-mon[29442]: pgmap v968: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 679 op/s 2024-02-05T06:53:37.825 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:37 smithi028 ceph-mon[29442]: 11.e scrub ok 2024-02-05T06:53:37.825 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:37 smithi028 ceph-mon[29442]: 11.f scrub ok 2024-02-05T06:53:37.825 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:37 smithi028 ceph-mon[29442]: from='client.25969 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-05T06:53:37.825 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:37 smithi028 ceph-mon[29442]: osdmap e465: 8 total, 8 up, 8 in 2024-02-05T06:53:37.825 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:37 smithi028 ceph-mon[29442]: 4.8 scrub starts 2024-02-05T06:53:37.825 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:37 smithi028 ceph-mon[29442]: 4.8 scrub ok 2024-02-05T06:53:37.825 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:37 smithi028 ceph-mon[29442]: 5.18 scrub starts 2024-02-05T06:53:37.825 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:37 smithi028 ceph-mon[29442]: 5.18 scrub ok 2024-02-05T06:53:37.825 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:37 smithi028 ceph-mon[29442]: 4.16 scrub starts 2024-02-05T06:53:37.825 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:37 smithi028 ceph-mon[29442]: 4.16 scrub ok 2024-02-05T06:53:37.825 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:37 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2412136041' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:53:37.832 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:53:37.832 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:53:37.832 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:53:37.832 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:53:37.832 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:53:37.832 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:53:37.832 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:53:37.832 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:53:37.871 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:37.899 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:37.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:37.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:38.001 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:38.031 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:38.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:38.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:38.133 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:38.166 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:38.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:38.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:38.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:37 smithi167 ceph-mon[32774]: pgmap v968: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 679 op/s 2024-02-05T06:53:38.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:37 smithi167 ceph-mon[32774]: 11.e scrub ok 2024-02-05T06:53:38.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:37 smithi167 ceph-mon[32774]: 11.f scrub ok 2024-02-05T06:53:38.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:37 smithi167 ceph-mon[32774]: from='client.25969 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-05T06:53:38.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:37 smithi167 ceph-mon[32774]: osdmap e465: 8 total, 8 up, 8 in 2024-02-05T06:53:38.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:37 smithi167 ceph-mon[32774]: 4.8 scrub starts 2024-02-05T06:53:38.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:37 smithi167 ceph-mon[32774]: 4.8 scrub ok 2024-02-05T06:53:38.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:37 smithi167 ceph-mon[32774]: 5.18 scrub starts 2024-02-05T06:53:38.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:37 smithi167 ceph-mon[32774]: 5.18 scrub ok 2024-02-05T06:53:38.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:37 smithi167 ceph-mon[32774]: 4.16 scrub starts 2024-02-05T06:53:38.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:37 smithi167 ceph-mon[32774]: 4.16 scrub ok 2024-02-05T06:53:38.235 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:37 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2412136041' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:53:38.268 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:38.302 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:38.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:38.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:38.404 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:38.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:38.437 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:38.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:38.538 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:38.575 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:38.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:38.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:38.677 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:38.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:38.704 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:38.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:38.805 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:38.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:38.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:38.948 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:38.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:38 smithi028 ceph-mon[29442]: 4.4 scrub starts 2024-02-05T06:53:38.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:38 smithi028 ceph-mon[29442]: 4.4 scrub ok 2024-02-05T06:53:38.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:38 smithi028 ceph-mon[37984]: 4.4 scrub starts 2024-02-05T06:53:38.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:38 smithi028 ceph-mon[37984]: 4.4 scrub ok 2024-02-05T06:53:38.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:38.980 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:38.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:39.081 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:39.113 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:39.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:39.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:39.215 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:39.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:38 smithi167 ceph-mon[32774]: 4.4 scrub starts 2024-02-05T06:53:39.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:38 smithi167 ceph-mon[32774]: 4.4 scrub ok 2024-02-05T06:53:39.249 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:39.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:39.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:39.351 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:39.383 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:39.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:39.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:39.488 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:39.520 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:39.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:39.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:39.622 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:39.663 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:39.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:39.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:39.765 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:39.796 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:39.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:39.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:39.864 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd reweight 5 1 2024-02-05T06:53:39.898 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:39.952 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:39.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:39.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:40.054 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:40.083 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:40.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:40.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:40.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:39 smithi028 ceph-mon[29442]: 3.2 scrub starts 2024-02-05T06:53:40.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:39 smithi028 ceph-mon[29442]: 3.2 scrub ok 2024-02-05T06:53:40.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:39 smithi028 ceph-mon[29442]: pgmap v970: 168 pgs: 168 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-02-05T06:53:40.132 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:39 smithi028 ceph-mon[29442]: 6.1 scrub starts 2024-02-05T06:53:40.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:39 smithi028 ceph-mon[29442]: 6.1 scrub ok 2024-02-05T06:53:40.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:39 smithi028 ceph-mon[29442]: 3.18 scrub starts 2024-02-05T06:53:40.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:39 smithi028 ceph-mon[29442]: 3.18 scrub ok 2024-02-05T06:53:40.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:39 smithi028 ceph-mon[29442]: 3.0 scrub starts 2024-02-05T06:53:40.133 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:39 smithi028 ceph-mon[29442]: 3.0 scrub ok 2024-02-05T06:53:40.133 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:39 smithi028 ceph-mon[37984]: 3.2 scrub starts 2024-02-05T06:53:40.133 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:39 smithi028 ceph-mon[37984]: 3.2 scrub ok 2024-02-05T06:53:40.133 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:39 smithi028 ceph-mon[37984]: pgmap v970: 168 pgs: 168 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-02-05T06:53:40.133 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:39 smithi028 ceph-mon[37984]: 6.1 scrub starts 2024-02-05T06:53:40.133 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:39 smithi028 ceph-mon[37984]: 6.1 scrub ok 2024-02-05T06:53:40.133 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:39 smithi028 ceph-mon[37984]: 3.18 scrub starts 2024-02-05T06:53:40.133 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:39 smithi028 ceph-mon[37984]: 3.18 scrub ok 2024-02-05T06:53:40.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:39 smithi028 ceph-mon[37984]: 3.0 scrub starts 2024-02-05T06:53:40.134 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:39 smithi028 ceph-mon[37984]: 3.0 scrub ok 2024-02-05T06:53:40.185 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:40.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:40.228 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:40.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:40.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:39 smithi167 ceph-mon[32774]: 3.2 scrub starts 2024-02-05T06:53:40.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:39 smithi167 ceph-mon[32774]: 3.2 scrub ok 2024-02-05T06:53:40.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:39 smithi167 ceph-mon[32774]: pgmap v970: 168 pgs: 168 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-02-05T06:53:40.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:39 smithi167 ceph-mon[32774]: 6.1 scrub starts 2024-02-05T06:53:40.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:39 smithi167 ceph-mon[32774]: 6.1 scrub ok 2024-02-05T06:53:40.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:39 smithi167 ceph-mon[32774]: 3.18 scrub starts 2024-02-05T06:53:40.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:39 smithi167 ceph-mon[32774]: 3.18 scrub ok 2024-02-05T06:53:40.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:39 smithi167 ceph-mon[32774]: 3.0 scrub starts 2024-02-05T06:53:40.234 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:39 smithi167 ceph-mon[32774]: 3.0 scrub ok 2024-02-05T06:53:40.329 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:40.358 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:40.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:40.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:40.459 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:40.500 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:40.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:40.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:40.602 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:40.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:40.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:40.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:40.783 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:40.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:40.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:40.885 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:40.922 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:40.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:40.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:41.024 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:41.057 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:41.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:41.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:41.159 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:41.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:41.187 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:41.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:41.288 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:41.315 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:41.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:41.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:41.417 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:41.461 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:53:41.465 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:41.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:41.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:41.571 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:41.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:41 smithi028 ceph-mon[29442]: 1.5 scrub starts 2024-02-05T06:53:41.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:41 smithi028 ceph-mon[29442]: 1.5 scrub ok 2024-02-05T06:53:41.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:41 smithi028 ceph-mon[29442]: pgmap v971: 168 pgs: 168 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 885 op/s 2024-02-05T06:53:41.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:41 smithi028 ceph-mon[29442]: 6.5 scrub starts 2024-02-05T06:53:41.572 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:41 smithi028 ceph-mon[29442]: 6.5 scrub ok 2024-02-05T06:53:41.573 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:41 smithi028 ceph-mon[37984]: 1.5 scrub starts 2024-02-05T06:53:41.573 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:41 smithi028 ceph-mon[37984]: 1.5 scrub ok 2024-02-05T06:53:41.573 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:41 smithi028 ceph-mon[37984]: pgmap v971: 168 pgs: 168 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 885 op/s 2024-02-05T06:53:41.573 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:41 smithi028 ceph-mon[37984]: 6.5 scrub starts 2024-02-05T06:53:41.573 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:41 smithi028 ceph-mon[37984]: 6.5 scrub ok 2024-02-05T06:53:41.627 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:41.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:41.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:41.729 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:41.733 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:53:41.733 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:53:41.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:41 smithi167 ceph-mon[32774]: 1.5 scrub starts 2024-02-05T06:53:41.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:41 smithi167 ceph-mon[32774]: 1.5 scrub ok 2024-02-05T06:53:41.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:41 smithi167 ceph-mon[32774]: pgmap v971: 168 pgs: 168 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 885 op/s 2024-02-05T06:53:41.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:41 smithi167 ceph-mon[32774]: 6.5 scrub starts 2024-02-05T06:53:41.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:41 smithi167 ceph-mon[32774]: 6.5 scrub ok 2024-02-05T06:53:41.757 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:41.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:41.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:41.858 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:41.894 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:41.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:41.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:41.995 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:42.030 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:42.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:42.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:42.132 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:42.159 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:42.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:42.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:42.260 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:42.304 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:42.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:42.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:42.406 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:42.445 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:42.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:42.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:42.547 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:42.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:42.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:42.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:42 smithi028 ceph-mon[29442]: 5.11 scrub starts 2024-02-05T06:53:42.656 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:42 smithi028 ceph-mon[29442]: 5.11 scrub ok 2024-02-05T06:53:42.656 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:42 smithi028 ceph-mon[37984]: 5.11 scrub starts 2024-02-05T06:53:42.657 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:42 smithi028 ceph-mon[37984]: 5.11 scrub ok 2024-02-05T06:53:42.657 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:53:42 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:53:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:53:42.692 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:42.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:42 smithi167 ceph-mon[32774]: 5.11 scrub starts 2024-02-05T06:53:42.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:42 smithi167 ceph-mon[32774]: 5.11 scrub ok 2024-02-05T06:53:42.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:42.743 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:42.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:42.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:42.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:42.882 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:42.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:42.983 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:43.013 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:43.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:43.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:43.115 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:43.151 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:43.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:43.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:43.253 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:43.280 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:43.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:43.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:43.283 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-02-05T06:53:43.325 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:53:43.330 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:53:43.358 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-02-05T06:53:43.382 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:43.411 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:43.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:43.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:43.513 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:43.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:43.560 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:43.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:43.661 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:43.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:43 smithi028 ceph-mon[29442]: 3.6 scrub starts 2024-02-05T06:53:43.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:43 smithi028 ceph-mon[29442]: 3.6 scrub ok 2024-02-05T06:53:43.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:43 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:53:43.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:43 smithi028 ceph-mon[29442]: pgmap v972: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-02-05T06:53:43.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:43 smithi028 ceph-mon[29442]: 3.13 scrub starts 2024-02-05T06:53:43.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:43 smithi028 ceph-mon[29442]: 3.13 scrub ok 2024-02-05T06:53:43.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:43 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3959639101' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:53:43.702 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:43 smithi028 ceph-mon[29442]: from='client.25978 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:53:43.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:43 smithi028 ceph-mon[37984]: 3.6 scrub starts 2024-02-05T06:53:43.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:43 smithi028 ceph-mon[37984]: 3.6 scrub ok 2024-02-05T06:53:43.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:43 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:53:43.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:43 smithi028 ceph-mon[37984]: pgmap v972: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-02-05T06:53:43.702 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:43 smithi028 ceph-mon[37984]: 3.13 scrub starts 2024-02-05T06:53:43.703 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:43 smithi028 ceph-mon[37984]: 3.13 scrub ok 2024-02-05T06:53:43.703 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:43 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3959639101' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:53:43.703 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:43 smithi028 ceph-mon[37984]: from='client.25978 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:53:43.719 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:43.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:43.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:43.733 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:53:43 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:53:43.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:43 smithi167 ceph-mon[32774]: 3.6 scrub starts 2024-02-05T06:53:43.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:43 smithi167 ceph-mon[32774]: 3.6 scrub ok 2024-02-05T06:53:43.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:43 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:53:43.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:43 smithi167 ceph-mon[32774]: pgmap v972: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-02-05T06:53:43.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:43 smithi167 ceph-mon[32774]: 3.13 scrub starts 2024-02-05T06:53:43.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:43 smithi167 ceph-mon[32774]: 3.13 scrub ok 2024-02-05T06:53:43.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:43 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3959639101' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:53:43.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:43 smithi167 ceph-mon[32774]: from='client.25978 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:53:43.821 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:43.863 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:43.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:43.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:43.965 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:44.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:44.002 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:44.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:44.103 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:44.134 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:44.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:44.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:44.236 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:44.274 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:44.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:44.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:44.376 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:44.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:44.411 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:44.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:44.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:44.548 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:44.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:44.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:44.650 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:44.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:44.692 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:44.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:44.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:44 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:53:44.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:44 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:53:44.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:44 smithi028 ceph-mon[29442]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:53:44.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:44 smithi028 ceph-mon[29442]: Cluster is now healthy 2024-02-05T06:53:44.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:44 smithi028 ceph-mon[29442]: from='client.25978 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:53:44.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:44 smithi028 ceph-mon[29442]: osdmap e466: 8 total, 8 up, 8 in 2024-02-05T06:53:44.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:44 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:53:44.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:44 smithi028 ceph-mon[29442]: 3.15 scrub starts 2024-02-05T06:53:44.704 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:44 smithi028 ceph-mon[29442]: 3.15 scrub ok 2024-02-05T06:53:44.705 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:44 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:53:44.705 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:44 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:53:44.705 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:44 smithi028 ceph-mon[37984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:53:44.705 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:44 smithi028 ceph-mon[37984]: Cluster is now healthy 2024-02-05T06:53:44.705 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:44 smithi028 ceph-mon[37984]: from='client.25978 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:53:44.705 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:44 smithi028 ceph-mon[37984]: osdmap e466: 8 total, 8 up, 8 in 2024-02-05T06:53:44.705 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:44 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:53:44.705 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:44 smithi028 ceph-mon[37984]: 3.15 scrub starts 2024-02-05T06:53:44.705 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:44 smithi028 ceph-mon[37984]: 3.15 scrub ok 2024-02-05T06:53:44.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:44 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:53:44.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:44 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:53:44.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:44 smithi167 ceph-mon[32774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:53:44.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:44 smithi167 ceph-mon[32774]: Cluster is now healthy 2024-02-05T06:53:44.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:44 smithi167 ceph-mon[32774]: from='client.25978 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:53:44.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:44 smithi167 ceph-mon[32774]: osdmap e466: 8 total, 8 up, 8 in 2024-02-05T06:53:44.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:44 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:53:44.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:44 smithi167 ceph-mon[32774]: 3.15 scrub starts 2024-02-05T06:53:44.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:44 smithi167 ceph-mon[32774]: 3.15 scrub ok 2024-02-05T06:53:44.794 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:44.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:44.824 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:44.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:44.925 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:44.975 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:44.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:44.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:45.076 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:45.108 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:45.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:45.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:45.210 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:45.250 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:45.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:45.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:45.352 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:45.400 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:45.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:45.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:45.502 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:45.538 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:45.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:45.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:45.639 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:45.668 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:45.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:45.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:45.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:45 smithi028 ceph-mon[37984]: 3.12 deep-scrub starts 2024-02-05T06:53:45.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:45 smithi028 ceph-mon[37984]: 3.12 deep-scrub ok 2024-02-05T06:53:45.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:45 smithi028 ceph-mon[37984]: 2.1 scrub starts 2024-02-05T06:53:45.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:45 smithi028 ceph-mon[37984]: 2.1 scrub ok 2024-02-05T06:53:45.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:45 smithi028 ceph-mon[37984]: pgmap v974: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-02-05T06:53:45.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:45 smithi028 ceph-mon[37984]: 3.11 deep-scrub starts 2024-02-05T06:53:45.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:45 smithi028 ceph-mon[37984]: 3.11 deep-scrub ok 2024-02-05T06:53:45.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:45 smithi028 ceph-mon[29442]: 3.12 deep-scrub starts 2024-02-05T06:53:45.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:45 smithi028 ceph-mon[29442]: 3.12 deep-scrub ok 2024-02-05T06:53:45.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:45 smithi028 ceph-mon[29442]: 2.1 scrub starts 2024-02-05T06:53:45.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:45 smithi028 ceph-mon[29442]: 2.1 scrub ok 2024-02-05T06:53:45.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:45 smithi028 ceph-mon[29442]: pgmap v974: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-02-05T06:53:45.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:45 smithi028 ceph-mon[29442]: 3.11 deep-scrub starts 2024-02-05T06:53:45.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:45 smithi028 ceph-mon[29442]: 3.11 deep-scrub ok 2024-02-05T06:53:45.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:45 smithi167 ceph-mon[32774]: 3.12 deep-scrub starts 2024-02-05T06:53:45.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:45 smithi167 ceph-mon[32774]: 3.12 deep-scrub ok 2024-02-05T06:53:45.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:45 smithi167 ceph-mon[32774]: 2.1 scrub starts 2024-02-05T06:53:45.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:45 smithi167 ceph-mon[32774]: 2.1 scrub ok 2024-02-05T06:53:45.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:45 smithi167 ceph-mon[32774]: pgmap v974: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-02-05T06:53:45.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:45 smithi167 ceph-mon[32774]: 3.11 deep-scrub starts 2024-02-05T06:53:45.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:45 smithi167 ceph-mon[32774]: 3.11 deep-scrub ok 2024-02-05T06:53:45.770 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:45.813 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:45.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:45.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:45.914 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:45.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:45.962 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:45.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:46.063 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:46.100 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:46.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:46.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:46.116 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:53:46.201 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:46.230 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:46.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:46.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:46.332 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:46.373 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:46.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:46.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:46.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:46 smithi028 ceph-mon[29442]: 11.6 scrub starts 2024-02-05T06:53:46.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:46 smithi028 ceph-mon[29442]: 1.a scrub starts 2024-02-05T06:53:46.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:46 smithi028 ceph-mon[29442]: 1.a scrub ok 2024-02-05T06:53:46.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:46 smithi028 ceph-mon[29442]: 3.a scrub starts 2024-02-05T06:53:46.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:46 smithi028 ceph-mon[29442]: 3.a scrub ok 2024-02-05T06:53:46.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:46 smithi028 ceph-mon[37984]: 11.6 scrub starts 2024-02-05T06:53:46.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:46 smithi028 ceph-mon[37984]: 1.a scrub starts 2024-02-05T06:53:46.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:46 smithi028 ceph-mon[37984]: 1.a scrub ok 2024-02-05T06:53:46.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:46 smithi028 ceph-mon[37984]: 3.a scrub starts 2024-02-05T06:53:46.450 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:46 smithi028 ceph-mon[37984]: 3.a scrub ok 2024-02-05T06:53:46.474 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:46.511 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:46.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:46.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:46.612 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:46.647 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:46.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:46.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:46.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:46 smithi167 ceph-mon[32774]: 11.6 scrub starts 2024-02-05T06:53:46.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:46 smithi167 ceph-mon[32774]: 1.a scrub starts 2024-02-05T06:53:46.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:46 smithi167 ceph-mon[32774]: 1.a scrub ok 2024-02-05T06:53:46.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:46 smithi167 ceph-mon[32774]: 3.a scrub starts 2024-02-05T06:53:46.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:46 smithi167 ceph-mon[32774]: 3.a scrub ok 2024-02-05T06:53:46.749 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:46.781 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:46.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:46.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:46.883 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:46.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:46.925 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:46.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:47.026 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:47.066 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:47.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:47.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:47.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:47.209 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:47.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:47.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:47.311 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:47.357 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:47.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:47.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:47.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:47 smithi028 ceph-mon[29442]: 11.6 scrub ok 2024-02-05T06:53:47.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:47 smithi028 ceph-mon[29442]: pgmap v975: 168 pgs: 168 active+clean; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-02-05T06:53:47.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:47 smithi028 ceph-mon[29442]: 5.19 scrub starts 2024-02-05T06:53:47.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:47 smithi028 ceph-mon[29442]: 5.19 scrub ok 2024-02-05T06:53:47.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:47 smithi028 ceph-mon[37984]: 11.6 scrub ok 2024-02-05T06:53:47.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:47 smithi028 ceph-mon[37984]: pgmap v975: 168 pgs: 168 active+clean; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-02-05T06:53:47.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:47 smithi028 ceph-mon[37984]: 5.19 scrub starts 2024-02-05T06:53:47.443 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:47 smithi028 ceph-mon[37984]: 5.19 scrub ok 2024-02-05T06:53:47.459 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:47.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:47.489 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:47.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:47.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T06:53:47.590 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:47.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:47.641 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:47.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:47.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:47 smithi167 ceph-mon[32774]: 11.6 scrub ok 2024-02-05T06:53:47.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:47 smithi167 ceph-mon[32774]: pgmap v975: 168 pgs: 168 active+clean; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-02-05T06:53:47.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:47 smithi167 ceph-mon[32774]: 5.19 scrub starts 2024-02-05T06:53:47.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:47 smithi167 ceph-mon[32774]: 5.19 scrub ok 2024-02-05T06:53:47.743 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:47.761 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:53:47.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:47.785 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:47.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:47.810 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:53:47.858 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:53:47.886 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:47.899 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:53:47.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:47.932 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:47.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:47.936 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:53:47.971 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:53:48.010 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:53:48.034 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:48.059 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:53:48.063 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:53:48.063 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:53:48.063 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:53:48.063 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:53:48.063 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:53:48.064 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:53:48.064 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:53:48.064 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:53:48.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:48.088 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:48.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:48.189 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:48.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:48.237 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:48.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:48.338 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:48.369 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:48.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:48.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:48.470 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:48.498 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:48.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:48.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:48.599 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:48.638 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:48.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:48.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:48.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:48 smithi028 ceph-mon[29442]: 6.d scrub starts 2024-02-05T06:53:48.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:48 smithi028 ceph-mon[29442]: 6.d scrub ok 2024-02-05T06:53:48.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:48 smithi028 ceph-mon[29442]: 1.14 scrub starts 2024-02-05T06:53:48.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:48 smithi028 ceph-mon[29442]: 1.14 scrub ok 2024-02-05T06:53:48.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:48 smithi028 ceph-mon[29442]: 3.17 scrub starts 2024-02-05T06:53:48.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:48 smithi028 ceph-mon[29442]: 3.17 scrub ok 2024-02-05T06:53:48.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:48 smithi028 ceph-mon[29442]: 6.4 deep-scrub starts 2024-02-05T06:53:48.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:48 smithi028 ceph-mon[29442]: 6.4 deep-scrub ok 2024-02-05T06:53:48.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:48 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/635629668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:53:48.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:48 smithi028 ceph-mon[37984]: 6.d scrub starts 2024-02-05T06:53:48.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:48 smithi028 ceph-mon[37984]: 6.d scrub ok 2024-02-05T06:53:48.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:48 smithi028 ceph-mon[37984]: 1.14 scrub starts 2024-02-05T06:53:48.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:48 smithi028 ceph-mon[37984]: 1.14 scrub ok 2024-02-05T06:53:48.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:48 smithi028 ceph-mon[37984]: 3.17 scrub starts 2024-02-05T06:53:48.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:48 smithi028 ceph-mon[37984]: 3.17 scrub ok 2024-02-05T06:53:48.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:48 smithi028 ceph-mon[37984]: 6.4 deep-scrub starts 2024-02-05T06:53:48.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:48 smithi028 ceph-mon[37984]: 6.4 deep-scrub ok 2024-02-05T06:53:48.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:48 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/635629668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:53:48.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:48 smithi167 ceph-mon[32774]: 6.d scrub starts 2024-02-05T06:53:48.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:48 smithi167 ceph-mon[32774]: 6.d scrub ok 2024-02-05T06:53:48.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:48 smithi167 ceph-mon[32774]: 1.14 scrub starts 2024-02-05T06:53:48.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:48 smithi167 ceph-mon[32774]: 1.14 scrub ok 2024-02-05T06:53:48.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:48 smithi167 ceph-mon[32774]: 3.17 scrub starts 2024-02-05T06:53:48.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:48 smithi167 ceph-mon[32774]: 3.17 scrub ok 2024-02-05T06:53:48.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:48 smithi167 ceph-mon[32774]: 6.4 deep-scrub starts 2024-02-05T06:53:48.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:48 smithi167 ceph-mon[32774]: 6.4 deep-scrub ok 2024-02-05T06:53:48.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:48 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/635629668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:53:48.740 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:48.776 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:48.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:48.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:48.878 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:48.911 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:48.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:48.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:49.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:49.048 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:49.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:49.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:49.150 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:49.185 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:49.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:49.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:49.287 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:49.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:49.329 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:49.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:49.430 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:49.466 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:49.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:49.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:49.568 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:49.604 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:49.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:49.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:49.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:49 smithi028 ceph-mon[37984]: 3.1f scrub starts 2024-02-05T06:53:49.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:49 smithi028 ceph-mon[37984]: 3.1f scrub ok 2024-02-05T06:53:49.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:49 smithi028 ceph-mon[37984]: 5.1e scrub starts 2024-02-05T06:53:49.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:49 smithi028 ceph-mon[37984]: 5.1e scrub ok 2024-02-05T06:53:49.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:49 smithi028 ceph-mon[37984]: pgmap v976: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-02-05T06:53:49.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:49 smithi028 ceph-mon[29442]: 3.1f scrub starts 2024-02-05T06:53:49.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:49 smithi028 ceph-mon[29442]: 3.1f scrub ok 2024-02-05T06:53:49.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:49 smithi028 ceph-mon[29442]: 5.1e scrub starts 2024-02-05T06:53:49.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:49 smithi028 ceph-mon[29442]: 5.1e scrub ok 2024-02-05T06:53:49.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:49 smithi028 ceph-mon[29442]: pgmap v976: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-02-05T06:53:49.706 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:49.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:49 smithi167 ceph-mon[32774]: 3.1f scrub starts 2024-02-05T06:53:49.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:49 smithi167 ceph-mon[32774]: 3.1f scrub ok 2024-02-05T06:53:49.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:49 smithi167 ceph-mon[32774]: 5.1e scrub starts 2024-02-05T06:53:49.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:49 smithi167 ceph-mon[32774]: 5.1e scrub ok 2024-02-05T06:53:49.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:49 smithi167 ceph-mon[32774]: pgmap v976: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 35 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-02-05T06:53:49.745 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:49.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:49.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:49.847 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:49.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:49.880 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:49.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:49.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:50.018 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:50.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:50.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:50.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:50.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:50.163 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:50.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:50.264 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:50.295 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:50.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:50.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:50.397 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:50.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:50.433 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:50.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:50.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:50 smithi028 ceph-mon[37984]: 11.7 deep-scrub starts 2024-02-05T06:53:50.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:50 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:53:50.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:50 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:53:50.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:50 smithi028 ceph-mon[37984]: osdmap e467: 8 total, 8 up, 8 in 2024-02-05T06:53:50.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:50 smithi028 ceph-mon[29442]: 11.7 deep-scrub starts 2024-02-05T06:53:50.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:50 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:53:50.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:50 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:53:50.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:50 smithi028 ceph-mon[29442]: osdmap e467: 8 total, 8 up, 8 in 2024-02-05T06:53:50.534 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:50.562 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:50.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:50.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:50.665 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:50.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:50.700 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:50.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:50.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:50 smithi167 ceph-mon[32774]: 11.7 deep-scrub starts 2024-02-05T06:53:50.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:50 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:53:50.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:50 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:53:50.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:50 smithi167 ceph-mon[32774]: osdmap e467: 8 total, 8 up, 8 in 2024-02-05T06:53:50.801 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:50.832 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:50.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:50.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:50.934 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:50.968 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:50.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:50.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:51.070 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:51.108 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:51.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:51.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:51.210 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:51.252 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:51.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:51.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:51.354 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:51.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:51.387 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:51.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:51.484 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:51 smithi028 ceph-mon[37984]: 6.9 scrub starts 2024-02-05T06:53:51.485 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:51 smithi028 ceph-mon[37984]: 5.1 scrub starts 2024-02-05T06:53:51.485 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:51 smithi028 ceph-mon[37984]: 5.1 scrub ok 2024-02-05T06:53:51.485 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:51 smithi028 ceph-mon[37984]: pgmap v978: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-02-05T06:53:51.485 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:51 smithi028 ceph-mon[29442]: 6.9 scrub starts 2024-02-05T06:53:51.485 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:51 smithi028 ceph-mon[29442]: 5.1 scrub starts 2024-02-05T06:53:51.485 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:51 smithi028 ceph-mon[29442]: 5.1 scrub ok 2024-02-05T06:53:51.485 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:51 smithi028 ceph-mon[29442]: pgmap v978: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-02-05T06:53:51.488 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:51.520 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:51.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:51.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:51.622 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:51.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:51.664 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:51.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:51.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:51 smithi167 ceph-mon[32774]: 6.9 scrub starts 2024-02-05T06:53:51.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:51 smithi167 ceph-mon[32774]: 5.1 scrub starts 2024-02-05T06:53:51.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:51 smithi167 ceph-mon[32774]: 5.1 scrub ok 2024-02-05T06:53:51.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:51 smithi167 ceph-mon[32774]: pgmap v978: 168 pgs: 1 active+clean+scrubbing, 167 active+clean; 4.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 12 KiB/s rd, 0 B/s wr, 1.05k op/s 2024-02-05T06:53:51.765 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:51.791 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:51.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:51.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:51.893 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:51.930 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:51.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:51.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:51.990 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:53:51.990 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [ 2024-02-05T06:53:51.990 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:51.990 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.45878.0:188408 11.0 e467/418 11:0d83bf2c:::benchmark_data_smithi028_177697_object61442:head v 467'15991, mlcod=467'15989)", 2024-02-05T06:53:51.990 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:53:51.809447+0000", 2024-02-05T06:53:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0031079839999999998, 2024-02-05T06:53:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031392350000000002, 2024-02-05T06:53:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:53:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "started", 2024-02-05T06:53:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:53:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:53:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.809447+0000", 2024-02-05T06:53:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:53:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:53:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.809447+0000", 2024-02-05T06:53:51.991 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:53:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:53:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.809448+0000", 2024-02-05T06:53:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.6600000000000003e-07 2024-02-05T06:53:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:53:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.809452+0000", 2024-02-05T06:53:51.992 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.5229999999999997e-06 2024-02-05T06:53:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:53:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.809456+0000", 2024-02-05T06:53:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.3209999999999998e-06 2024-02-05T06:53:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:53:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.809461+0000", 2024-02-05T06:53:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.1459999999999998e-06 2024-02-05T06:53:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:51.993 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:53:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.811327+0000", 2024-02-05T06:53:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0018669240000000001 2024-02-05T06:53:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:53:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.811348+0000", 2024-02-05T06:53:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.0988999999999998e-05 2024-02-05T06:53:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:53:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:53:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:53:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.45878.0:188400 11.2 e467/418)", 2024-02-05T06:53:51.994 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:53:51.809547+0000", 2024-02-05T06:53:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0030083710000000001, 2024-02-05T06:53:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031035609999999999, 2024-02-05T06:53:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:53:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "queued for pg", 2024-02-05T06:53:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:53:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:53:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.809547+0000", 2024-02-05T06:53:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:53:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:53:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.809547+0000", 2024-02-05T06:53:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:53:51.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:53:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.809548+0000", 2024-02-05T06:53:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.8299999999999997e-07 2024-02-05T06:53:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:53:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.809554+0000", 2024-02-05T06:53:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 6.7170000000000002e-06 2024-02-05T06:53:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:53:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.809562+0000", 2024-02-05T06:53:51.996 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.8450000000000003e-06 2024-02-05T06:53:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:53:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.809568+0000", 2024-02-05T06:53:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.6910000000000002e-06 2024-02-05T06:53:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:53:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:53:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:53:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.45878.0:188407 11.2 e467/418)", 2024-02-05T06:53:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:53:51.809577+0000", 2024-02-05T06:53:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0029785219999999999, 2024-02-05T06:53:51.997 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0031244490000000001, 2024-02-05T06:53:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:53:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "queued for pg", 2024-02-05T06:53:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:53:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:53:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.809577+0000", 2024-02-05T06:53:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:53:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:53:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.809577+0000", 2024-02-05T06:53:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:53:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:51.998 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:53:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.809577+0000", 2024-02-05T06:53:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.03e-07 2024-02-05T06:53:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:53:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.809581+0000", 2024-02-05T06:53:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.8589999999999998e-06 2024-02-05T06:53:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:53:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.809585+0000", 2024-02-05T06:53:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.4819999999999996e-06 2024-02-05T06:53:51.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:53:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.809589+0000", 2024-02-05T06:53:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.985e-06 2024-02-05T06:53:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:53:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:53:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:53:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop(client.45878.0:188409 11.0 e467/418)", 2024-02-05T06:53:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:53:51.810523+0000", 2024-02-05T06:53:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0020326060000000002, 2024-02-05T06:53:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0022292319999999998, 2024-02-05T06:53:52.000 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:53:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "queued for pg", 2024-02-05T06:53:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:53:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:53:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.810523+0000", 2024-02-05T06:53:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:53:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:53:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.810523+0000", 2024-02-05T06:53:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:53:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.001 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:53:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.810524+0000", 2024-02-05T06:53:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.2619999999999999e-06 2024-02-05T06:53:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:53:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.810531+0000", 2024-02-05T06:53:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.3420000000000002e-06 2024-02-05T06:53:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:53:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.810551+0000", 2024-02-05T06:53:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.9806999999999999e-05 2024-02-05T06:53:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:53:52.002 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.810559+0000", 2024-02-05T06:53:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.5889999999999997e-06 2024-02-05T06:53:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:53:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:53:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:53:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_op(client.45878.0:188405 11.a 11:526952a1:::benchmark_data_smithi028_177697_object61439:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e467)", 2024-02-05T06:53:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:53:51.798571+0000", 2024-02-05T06:53:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.013984084000000001, 2024-02-05T06:53:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.014239122999999999, 2024-02-05T06:53:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:53:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "waiting for sub ops", 2024-02-05T06:53:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: "client_info": { 2024-02-05T06:53:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: "client": "client.45878", 2024-02-05T06:53:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: "client_addr": "v1:172.21.15.28:0/3059201968", 2024-02-05T06:53:52.003 INFO:teuthology.orchestra.run.smithi028.stdout: "tid": 188405 2024-02-05T06:53:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:53:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:53:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.798571+0000", 2024-02-05T06:53:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:53:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:53:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.798571+0000", 2024-02-05T06:53:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:53:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.004 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:53:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.798572+0000", 2024-02-05T06:53:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 1.248e-06 2024-02-05T06:53:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:53:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.798577+0000", 2024-02-05T06:53:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.0200000000000002e-06 2024-02-05T06:53:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:53:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.798587+0000", 2024-02-05T06:53:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 9.3910000000000004e-06 2024-02-05T06:53:52.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:53:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.798594+0000", 2024-02-05T06:53:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 7.0949999999999998e-06 2024-02-05T06:53:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "reached_pg", 2024-02-05T06:53:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.799694+0000", 2024-02-05T06:53:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0011002119999999999 2024-02-05T06:53:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "started", 2024-02-05T06:53:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.799812+0000", 2024-02-05T06:53:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000118262 2024-02-05T06:53:52.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "waiting for subops from 5,6", 2024-02-05T06:53:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.799916+0000", 2024-02-05T06:53:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.000103243 2024-02-05T06:53:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "op_commit", 2024-02-05T06:53:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.805726+0000", 2024-02-05T06:53:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0058106490000000002 2024-02-05T06:53:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:53:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:53:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:53:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.007 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(client.45878.0:188405 11.a e467/418)", 2024-02-05T06:53:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:53:51.805965+0000", 2024-02-05T06:53:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.0065899799999999996, 2024-02-05T06:53:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0069275059999999999, 2024-02-05T06:53:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:53:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "queued for pg", 2024-02-05T06:53:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:53:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:53:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.805965+0000", 2024-02-05T06:53:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:53:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:53:52.008 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.805965+0000", 2024-02-05T06:53:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:53:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:53:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.805966+0000", 2024-02-05T06:53:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1500000000000005e-07 2024-02-05T06:53:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:53:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.805969+0000", 2024-02-05T06:53:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.3220000000000001e-06 2024-02-05T06:53:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:52.009 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:53:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.805978+0000", 2024-02-05T06:53:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 8.5920000000000001e-06 2024-02-05T06:53:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:53:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.805983+0000", 2024-02-05T06:53:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 5.1150000000000002e-06 2024-02-05T06:53:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:53:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:53:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:53:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.010 INFO:teuthology.orchestra.run.smithi028.stdout: "description": "osd_repop_reply(client.45878.0:188405 11.a e467/418)", 2024-02-05T06:53:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: "initiated_at": "2024-02-05T06:53:51.805993+0000", 2024-02-05T06:53:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: "age": 0.006561852, 2024-02-05T06:53:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0.0069496920000000004, 2024-02-05T06:53:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: "type_data": { 2024-02-05T06:53:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: "flag_point": "queued for pg", 2024-02-05T06:53:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: "events": [ 2024-02-05T06:53:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "initiated", 2024-02-05T06:53:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.805993+0000", 2024-02-05T06:53:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:53:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "header_read", 2024-02-05T06:53:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.805993+0000", 2024-02-05T06:53:52.011 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 0 2024-02-05T06:53:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "throttled", 2024-02-05T06:53:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.805994+0000", 2024-02-05T06:53:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.96e-07 2024-02-05T06:53:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "all_read", 2024-02-05T06:53:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.805996+0000", 2024-02-05T06:53:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 2.3889999999999999e-06 2024-02-05T06:53:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "dispatched", 2024-02-05T06:53:52.012 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.806001+0000", 2024-02-05T06:53:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 4.7010000000000004e-06 2024-02-05T06:53:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:53:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:53:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: "event": "queued_for_pg", 2024-02-05T06:53:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: "time": "2024-02-05T06:53:51.806005+0000", 2024-02-05T06:53:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: "duration": 3.9519999999999996e-06 2024-02-05T06:53:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:53:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:53:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:53:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:53:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:53:52.013 INFO:teuthology.orchestra.run.smithi028.stdout: "num_ops": 7 2024-02-05T06:53:52.013 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:53:52.032 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:52.064 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:52.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:52.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:52.166 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:52.200 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:52.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:52.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:52.301 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:52.336 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:52.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:52.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:52.407 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.5 to 1 (10000) 2024-02-05T06:53:52.438 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:52.471 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:52.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:52.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:52.572 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:52.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:52.610 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:52.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:52.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:52 smithi028 ceph-mon[37984]: 6.9 scrub ok 2024-02-05T06:53:52.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:52 smithi028 ceph-mon[37984]: 5.1c scrub starts 2024-02-05T06:53:52.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:52 smithi028 ceph-mon[37984]: 5.1c scrub ok 2024-02-05T06:53:52.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:52 smithi028 ceph-mon[37984]: 4.17 scrub starts 2024-02-05T06:53:52.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:52 smithi028 ceph-mon[37984]: 4.17 scrub ok 2024-02-05T06:53:52.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:52 smithi028 ceph-mon[37984]: 3.1a scrub starts 2024-02-05T06:53:52.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:52 smithi028 ceph-mon[37984]: 3.1a scrub ok 2024-02-05T06:53:52.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:52 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1439753081' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T06:53:52.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:52 smithi028 ceph-mon[29442]: 6.9 scrub ok 2024-02-05T06:53:52.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:52 smithi028 ceph-mon[29442]: 5.1c scrub starts 2024-02-05T06:53:52.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:52 smithi028 ceph-mon[29442]: 5.1c scrub ok 2024-02-05T06:53:52.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:52 smithi028 ceph-mon[29442]: 4.17 scrub starts 2024-02-05T06:53:52.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:52 smithi028 ceph-mon[29442]: 4.17 scrub ok 2024-02-05T06:53:52.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:52 smithi028 ceph-mon[29442]: 3.1a scrub starts 2024-02-05T06:53:52.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:52 smithi028 ceph-mon[29442]: 3.1a scrub ok 2024-02-05T06:53:52.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:52 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1439753081' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T06:53:52.701 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:53:52 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:53:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:53:52.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:52.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:52 smithi167 ceph-mon[32774]: 6.9 scrub ok 2024-02-05T06:53:52.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:52 smithi167 ceph-mon[32774]: 5.1c scrub starts 2024-02-05T06:53:52.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:52 smithi167 ceph-mon[32774]: 5.1c scrub ok 2024-02-05T06:53:52.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:52 smithi167 ceph-mon[32774]: 4.17 scrub starts 2024-02-05T06:53:52.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:52 smithi167 ceph-mon[32774]: 4.17 scrub ok 2024-02-05T06:53:52.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:52 smithi167 ceph-mon[32774]: 3.1a scrub starts 2024-02-05T06:53:52.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:52 smithi167 ceph-mon[32774]: 3.1a scrub ok 2024-02-05T06:53:52.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:52 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1439753081' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T06:53:52.754 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:52.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:52.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:52.856 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:52.867 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-02-05T06:53:52.901 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:52.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:52.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:53.003 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:53.030 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:53.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:53.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:53.092 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:53:53.132 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:53.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:53.165 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:53.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:53.265 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:53.301 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:53.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:53.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:53.403 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:53.438 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:53.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:53.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:53.540 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:53.577 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:53.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:53.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:53.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:53 smithi028 ceph-mon[29442]: 5.b scrub starts 2024-02-05T06:53:53.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:53 smithi028 ceph-mon[29442]: 5.b scrub ok 2024-02-05T06:53:53.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:53 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1439753081' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-05T06:53:53.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:53 smithi028 ceph-mon[29442]: osdmap e468: 8 total, 8 up, 8 in 2024-02-05T06:53:53.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:53 smithi028 ceph-mon[29442]: 6.0 scrub starts 2024-02-05T06:53:53.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:53 smithi028 ceph-mon[29442]: pgmap v980: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-02-05T06:53:53.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:53 smithi028 ceph-mon[29442]: 5.1d scrub starts 2024-02-05T06:53:53.660 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:53 smithi028 ceph-mon[29442]: 5.1d scrub ok 2024-02-05T06:53:53.661 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:53 smithi028 ceph-mon[37984]: 5.b scrub starts 2024-02-05T06:53:53.661 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:53 smithi028 ceph-mon[37984]: 5.b scrub ok 2024-02-05T06:53:53.661 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:53 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1439753081' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-05T06:53:53.661 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:53 smithi028 ceph-mon[37984]: osdmap e468: 8 total, 8 up, 8 in 2024-02-05T06:53:53.661 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:53 smithi028 ceph-mon[37984]: 6.0 scrub starts 2024-02-05T06:53:53.661 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:53 smithi028 ceph-mon[37984]: pgmap v980: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-02-05T06:53:53.661 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:53 smithi028 ceph-mon[37984]: 5.1d scrub starts 2024-02-05T06:53:53.661 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:53 smithi028 ceph-mon[37984]: 5.1d scrub ok 2024-02-05T06:53:53.679 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:53.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:53.710 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:53.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:53.711 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-05T06:53:53.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T06:53:53.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:53 smithi167 ceph-mon[32774]: 5.b scrub starts 2024-02-05T06:53:53.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:53 smithi167 ceph-mon[32774]: 5.b scrub ok 2024-02-05T06:53:53.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:53 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1439753081' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-05T06:53:53.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:53 smithi167 ceph-mon[32774]: osdmap e468: 8 total, 8 up, 8 in 2024-02-05T06:53:53.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:53 smithi167 ceph-mon[32774]: 6.0 scrub starts 2024-02-05T06:53:53.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:53 smithi167 ceph-mon[32774]: pgmap v980: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-02-05T06:53:53.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:53 smithi167 ceph-mon[32774]: 5.1d scrub starts 2024-02-05T06:53:53.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:53 smithi167 ceph-mon[32774]: 5.1d scrub ok 2024-02-05T06:53:53.734 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:53:53 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:53:53.811 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:53.847 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:53.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:53.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:53.865 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:53:53.949 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:53.981 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:53.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:53.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:54.083 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:54.128 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:54.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:54.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:54.230 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:54.265 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:54.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:54.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:54.367 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:54.403 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:54.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:54.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:54.505 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:54.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:54.542 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:54.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:54.643 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:54.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:54.677 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:54.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:54.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:54 smithi028 ceph-mon[29442]: 6.0 scrub ok 2024-02-05T06:53:54.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:54 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:53:54.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:54 smithi028 ceph-mon[29442]: 5.12 scrub starts 2024-02-05T06:53:54.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:54 smithi028 ceph-mon[29442]: 5.12 scrub ok 2024-02-05T06:53:54.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:54 smithi028 ceph-mon[29442]: 6.3 scrub starts 2024-02-05T06:53:54.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:54 smithi028 ceph-mon[29442]: 6.3 scrub ok 2024-02-05T06:53:54.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:54 smithi028 ceph-mon[37984]: 6.0 scrub ok 2024-02-05T06:53:54.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:54 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:53:54.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:54 smithi028 ceph-mon[37984]: 5.12 scrub starts 2024-02-05T06:53:54.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:54 smithi028 ceph-mon[37984]: 5.12 scrub ok 2024-02-05T06:53:54.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:54 smithi028 ceph-mon[37984]: 6.3 scrub starts 2024-02-05T06:53:54.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:54 smithi028 ceph-mon[37984]: 6.3 scrub ok 2024-02-05T06:53:54.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:54 smithi167 ceph-mon[32774]: 6.0 scrub ok 2024-02-05T06:53:54.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:54 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:53:54.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:54 smithi167 ceph-mon[32774]: 5.12 scrub starts 2024-02-05T06:53:54.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:54 smithi167 ceph-mon[32774]: 5.12 scrub ok 2024-02-05T06:53:54.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:54 smithi167 ceph-mon[32774]: 6.3 scrub starts 2024-02-05T06:53:54.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:54 smithi167 ceph-mon[32774]: 6.3 scrub ok 2024-02-05T06:53:54.779 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:54.817 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:54.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:54.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:54.919 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:54.956 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:54.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:54.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:55.058 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:55.097 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:55.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:55.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:55.199 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:55.232 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:55.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:55.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:55.334 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:55.364 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:55.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:55.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:55.466 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:55.499 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:55.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:55.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:55.601 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:55.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:55.638 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:55.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:55.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:55 smithi028 ceph-mon[29442]: 5.6 scrub starts 2024-02-05T06:53:55.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:55 smithi028 ceph-mon[29442]: 5.6 scrub ok 2024-02-05T06:53:55.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:55 smithi028 ceph-mon[29442]: 4.1e deep-scrub starts 2024-02-05T06:53:55.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:55 smithi028 ceph-mon[29442]: 4.1e deep-scrub ok 2024-02-05T06:53:55.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:55 smithi028 ceph-mon[29442]: pgmap v981: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-02-05T06:53:55.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:55 smithi028 ceph-mon[29442]: 6.15 scrub starts 2024-02-05T06:53:55.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:55 smithi028 ceph-mon[29442]: 6.15 scrub ok 2024-02-05T06:53:55.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:55 smithi028 ceph-mon[37984]: 5.6 scrub starts 2024-02-05T06:53:55.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:55 smithi028 ceph-mon[37984]: 5.6 scrub ok 2024-02-05T06:53:55.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:55 smithi028 ceph-mon[37984]: 4.1e deep-scrub starts 2024-02-05T06:53:55.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:55 smithi028 ceph-mon[37984]: 4.1e deep-scrub ok 2024-02-05T06:53:55.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:55 smithi028 ceph-mon[37984]: pgmap v981: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-02-05T06:53:55.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:55 smithi028 ceph-mon[37984]: 6.15 scrub starts 2024-02-05T06:53:55.701 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:55 smithi028 ceph-mon[37984]: 6.15 scrub ok 2024-02-05T06:53:55.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:55 smithi167 ceph-mon[32774]: 5.6 scrub starts 2024-02-05T06:53:55.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:55 smithi167 ceph-mon[32774]: 5.6 scrub ok 2024-02-05T06:53:55.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:55 smithi167 ceph-mon[32774]: 4.1e deep-scrub starts 2024-02-05T06:53:55.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:55 smithi167 ceph-mon[32774]: 4.1e deep-scrub ok 2024-02-05T06:53:55.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:55 smithi167 ceph-mon[32774]: pgmap v981: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-02-05T06:53:55.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:55 smithi167 ceph-mon[32774]: 6.15 scrub starts 2024-02-05T06:53:55.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:55 smithi167 ceph-mon[32774]: 6.15 scrub ok 2024-02-05T06:53:55.739 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:55.783 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:55.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:55.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:55.885 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:55.921 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:55.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:55.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:56.023 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:56.058 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:56.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:56.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:56.159 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:56.197 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:56.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:56.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:56.299 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:56.333 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:56.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:56.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:56.435 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:56.459 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:56.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:56.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:56.563 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:56.607 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:56.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:56.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:56.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:56 smithi028 ceph-mon[29442]: 3.10 scrub starts 2024-02-05T06:53:56.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:56 smithi028 ceph-mon[29442]: 3.10 scrub ok 2024-02-05T06:53:56.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:56 smithi028 ceph-mon[37984]: 3.10 scrub starts 2024-02-05T06:53:56.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:56 smithi028 ceph-mon[37984]: 3.10 scrub ok 2024-02-05T06:53:56.709 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:53:56.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:56 smithi167 ceph-mon[32774]: 3.10 scrub starts 2024-02-05T06:53:56.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:56 smithi167 ceph-mon[32774]: 3.10 scrub ok 2024-02-05T06:53:56.743 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:56.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:56.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:53:56.845 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:53:56.892 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:56.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:56.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:53:56.993 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:57.017 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:57.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:57.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:57.119 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:57.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:57.154 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:57.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:57.255 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:57.285 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:57.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:57.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:57.387 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:57.418 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:57.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:57.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:57.520 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:57.559 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:57.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:57.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:57.661 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:57.692 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:57.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:57.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:57.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:57 smithi028 ceph-mon[29442]: 1.8 scrub starts 2024-02-05T06:53:57.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:57 smithi028 ceph-mon[29442]: 1.8 scrub ok 2024-02-05T06:53:57.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:57 smithi028 ceph-mon[29442]: 5.d scrub starts 2024-02-05T06:53:57.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:57 smithi028 ceph-mon[29442]: 5.d scrub ok 2024-02-05T06:53:57.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:57 smithi028 ceph-mon[29442]: pgmap v982: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 715 op/s 2024-02-05T06:53:57.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:57 smithi028 ceph-mon[29442]: 3.9 scrub starts 2024-02-05T06:53:57.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:57 smithi028 ceph-mon[29442]: 3.9 scrub ok 2024-02-05T06:53:57.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:57 smithi028 ceph-mon[37984]: 1.8 scrub starts 2024-02-05T06:53:57.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:57 smithi028 ceph-mon[37984]: 1.8 scrub ok 2024-02-05T06:53:57.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:57 smithi028 ceph-mon[37984]: 5.d scrub starts 2024-02-05T06:53:57.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:57 smithi028 ceph-mon[37984]: 5.d scrub ok 2024-02-05T06:53:57.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:57 smithi028 ceph-mon[37984]: pgmap v982: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 715 op/s 2024-02-05T06:53:57.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:57 smithi028 ceph-mon[37984]: 3.9 scrub starts 2024-02-05T06:53:57.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:57 smithi028 ceph-mon[37984]: 3.9 scrub ok 2024-02-05T06:53:57.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:57 smithi167 ceph-mon[32774]: 1.8 scrub starts 2024-02-05T06:53:57.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:57 smithi167 ceph-mon[32774]: 1.8 scrub ok 2024-02-05T06:53:57.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:57 smithi167 ceph-mon[32774]: 5.d scrub starts 2024-02-05T06:53:57.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:57 smithi167 ceph-mon[32774]: 5.d scrub ok 2024-02-05T06:53:57.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:57 smithi167 ceph-mon[32774]: pgmap v982: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 715 op/s 2024-02-05T06:53:57.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:57 smithi167 ceph-mon[32774]: 3.9 scrub starts 2024-02-05T06:53:57.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:57 smithi167 ceph-mon[32774]: 3.9 scrub ok 2024-02-05T06:53:57.795 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:57.828 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:57.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:57.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:57.930 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:57.963 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:57.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:57.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:58.065 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:58.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:58.106 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:58.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:58.208 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:58.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:58.241 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:58.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:58.341 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:58.370 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:58.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:58.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:58.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:58.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:58.512 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:58.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:58.613 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:58.652 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:58.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:58.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:58.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:58 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:53:58.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:58 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:53:58.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:58 smithi028 ceph-mon[37984]: osdmap e469: 8 total, 8 up, 8 in 2024-02-05T06:53:58.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:58 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:53:58.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:58 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:53:58.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:58 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:53:58.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:58 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:53:58.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:58 smithi028 ceph-mon[29442]: osdmap e469: 8 total, 8 up, 8 in 2024-02-05T06:53:58.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:58 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:53:58.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:58 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:53:58.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:58 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:53:58.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:58 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:53:58.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:58 smithi167 ceph-mon[32774]: osdmap e469: 8 total, 8 up, 8 in 2024-02-05T06:53:58.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:58 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:53:58.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:58 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:53:58.753 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:58.786 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:58.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:58.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:58.888 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:53:58.918 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:58.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:58.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:53:59.020 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:59.045 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:59.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:59.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:59.147 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:59.181 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:59.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:59.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:59.283 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:53:59.321 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:59.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:59.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:53:59.422 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:53:59.460 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:59.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:59.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:53:59.562 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:53:59.597 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:53:59.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:59.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:53:59.699 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:53:59.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:59 smithi167 ceph-mon[32774]: pgmap v984: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-02-05T06:53:59.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:59 smithi167 ceph-mon[32774]: 5.1a scrub starts 2024-02-05T06:53:59.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:53:59 smithi167 ceph-mon[32774]: 5.1a scrub ok 2024-02-05T06:53:59.737 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:59.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:59.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:53:59.839 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:53:59.874 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:53:59.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:53:59.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:53:59.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:59 smithi028 ceph-mon[37984]: pgmap v984: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-02-05T06:53:59.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:59 smithi028 ceph-mon[37984]: 5.1a scrub starts 2024-02-05T06:53:59.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:53:59 smithi028 ceph-mon[37984]: 5.1a scrub ok 2024-02-05T06:53:59.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:59 smithi028 ceph-mon[29442]: pgmap v984: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-02-05T06:53:59.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:59 smithi028 ceph-mon[29442]: 5.1a scrub starts 2024-02-05T06:53:59.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:53:59 smithi028 ceph-mon[29442]: 5.1a scrub ok 2024-02-05T06:53:59.976 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:00.008 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:00.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:00.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:00.109 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:00.141 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:00.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:00.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:00.243 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:00.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:00.270 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:00.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:00.371 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:00.416 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:00.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:00.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:00.518 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:00.554 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:00.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:00.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:00.656 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:00.685 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:00.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:00.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:00.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:00 smithi167 ceph-mon[32774]: 5.0 scrub starts 2024-02-05T06:54:00.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:00 smithi167 ceph-mon[32774]: 5.0 scrub ok 2024-02-05T06:54:00.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:00 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:54:00.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:00 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:54:00.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:00 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:54:00.786 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:00.813 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:00.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:00.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:00.915 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:00.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:00.949 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:00.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:00.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:00 smithi028 ceph-mon[29442]: 5.0 scrub starts 2024-02-05T06:54:00.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:00 smithi028 ceph-mon[29442]: 5.0 scrub ok 2024-02-05T06:54:00.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:00 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:54:00.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:00 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:54:00.952 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:00 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:54:00.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:00 smithi028 ceph-mon[37984]: 5.0 scrub starts 2024-02-05T06:54:00.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:00 smithi028 ceph-mon[37984]: 5.0 scrub ok 2024-02-05T06:54:00.952 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:00 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:54:00.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:00 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:54:00.953 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:00 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:54:01.050 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:01.087 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:01.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:01.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:01.189 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:01.212 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:01.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:01.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:01.313 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:01.353 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:01.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:01.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:01.455 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:01.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:01.487 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:01.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:01.589 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:01.621 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:01.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:01.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:01.723 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:01.765 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:01.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:01.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:01.792 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:01 smithi028 ceph-mon[29442]: osdmap e470: 8 total, 8 up, 8 in 2024-02-05T06:54:01.792 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:01 smithi028 ceph-mon[29442]: pgmap v986: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 957 op/s 2024-02-05T06:54:01.792 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:01 smithi028 ceph-mon[29442]: 4.2 scrub starts 2024-02-05T06:54:01.792 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:01 smithi028 ceph-mon[29442]: 4.2 scrub ok 2024-02-05T06:54:01.792 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:01 smithi028 ceph-mon[37984]: osdmap e470: 8 total, 8 up, 8 in 2024-02-05T06:54:01.793 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:01 smithi028 ceph-mon[37984]: pgmap v986: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 957 op/s 2024-02-05T06:54:01.793 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:01 smithi028 ceph-mon[37984]: 4.2 scrub starts 2024-02-05T06:54:01.793 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:01 smithi028 ceph-mon[37984]: 4.2 scrub ok 2024-02-05T06:54:01.867 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:01.896 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:01.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:01.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:01.928 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:54:01.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:01 smithi167 ceph-mon[32774]: osdmap e470: 8 total, 8 up, 8 in 2024-02-05T06:54:01.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:01 smithi167 ceph-mon[32774]: pgmap v986: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 957 op/s 2024-02-05T06:54:01.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:01 smithi167 ceph-mon[32774]: 4.2 scrub starts 2024-02-05T06:54:01.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:01 smithi167 ceph-mon[32774]: 4.2 scrub ok 2024-02-05T06:54:01.998 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:02.011 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:54:02.059 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:02.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:02.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:02.161 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:02.194 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:02.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:02.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:02.296 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:02.343 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:02.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:02.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:02.445 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:02.453 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:54:02.523 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:02.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:02.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:02.624 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:02.657 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:02.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:02.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:02.759 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:02.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:02.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:02.815 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:02 smithi028 ceph-mon[37984]: 3.d scrub starts 2024-02-05T06:54:02.815 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:02 smithi028 ceph-mon[37984]: 3.d scrub ok 2024-02-05T06:54:02.815 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:02 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:54:02.815 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:02 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:54:02.815 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:02 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:54:02.816 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:54:02 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:54:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:54:02.816 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:02 smithi028 ceph-mon[29442]: 3.d scrub starts 2024-02-05T06:54:02.816 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:02 smithi028 ceph-mon[29442]: 3.d scrub ok 2024-02-05T06:54:02.816 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:02 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:54:02.816 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:02 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:54:02.816 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:02 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:54:02.895 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:02.931 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:02.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:02.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:02.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:02 smithi167 ceph-mon[32774]: 3.d scrub starts 2024-02-05T06:54:02.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:02 smithi167 ceph-mon[32774]: 3.d scrub ok 2024-02-05T06:54:02.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:02 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:54:02.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:02 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:54:02.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:02 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:54:03.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:03.073 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:03.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:03.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:03.175 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:03.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:03.235 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:03.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:03.336 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:03.360 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:03.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:03.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:03.462 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:03.492 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:03.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:03.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:03.594 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:03.632 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:03.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:03.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:03.735 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:03.766 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:03 smithi028 ceph-mon[29442]: 6.c scrub starts 2024-02-05T06:54:03.766 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:03 smithi028 ceph-mon[29442]: 6.c scrub ok 2024-02-05T06:54:03.767 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:03 smithi028 ceph-mon[29442]: 5.4 scrub starts 2024-02-05T06:54:03.767 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:03 smithi028 ceph-mon[29442]: 5.4 scrub ok 2024-02-05T06:54:03.767 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:03 smithi028 ceph-mon[29442]: pgmap v987: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-02-05T06:54:03.767 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:03 smithi028 ceph-mon[29442]: 1.13 scrub starts 2024-02-05T06:54:03.767 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:03 smithi028 ceph-mon[29442]: 1.13 scrub ok 2024-02-05T06:54:03.767 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:03 smithi028 ceph-mon[37984]: 6.c scrub starts 2024-02-05T06:54:03.767 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:03 smithi028 ceph-mon[37984]: 6.c scrub ok 2024-02-05T06:54:03.767 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:03 smithi028 ceph-mon[37984]: 5.4 scrub starts 2024-02-05T06:54:03.767 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:03 smithi028 ceph-mon[37984]: 5.4 scrub ok 2024-02-05T06:54:03.767 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:03 smithi028 ceph-mon[37984]: pgmap v987: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-02-05T06:54:03.767 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:03 smithi028 ceph-mon[37984]: 1.13 scrub starts 2024-02-05T06:54:03.767 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:03 smithi028 ceph-mon[37984]: 1.13 scrub ok 2024-02-05T06:54:03.768 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:03.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:03.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:03.870 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:03.904 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:03.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:03.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:03.983 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:54:03 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:54:03.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:03 smithi167 ceph-mon[32774]: 6.c scrub starts 2024-02-05T06:54:03.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:03 smithi167 ceph-mon[32774]: 6.c scrub ok 2024-02-05T06:54:03.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:03 smithi167 ceph-mon[32774]: 5.4 scrub starts 2024-02-05T06:54:03.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:03 smithi167 ceph-mon[32774]: 5.4 scrub ok 2024-02-05T06:54:03.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:03 smithi167 ceph-mon[32774]: pgmap v987: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-02-05T06:54:03.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:03 smithi167 ceph-mon[32774]: 1.13 scrub starts 2024-02-05T06:54:03.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:03 smithi167 ceph-mon[32774]: 1.13 scrub ok 2024-02-05T06:54:04.005 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:04.039 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:04.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:04.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:04.141 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:04.169 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:04.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:04.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:04.271 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:04.299 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:04.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:04.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:04.401 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:04.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:04.444 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:04.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:04.518 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-02-05T06:54:04.545 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:04.584 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:04.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:04.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:04.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:04.724 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:04.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:04.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:04.826 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:04.863 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:04.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:04.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:04.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:04 smithi028 ceph-mon[37984]: 4.1 scrub starts 2024-02-05T06:54:04.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:04 smithi028 ceph-mon[37984]: 4.1 scrub ok 2024-02-05T06:54:04.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:04 smithi028 ceph-mon[37984]: 1.0 scrub starts 2024-02-05T06:54:04.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:04 smithi028 ceph-mon[37984]: 1.0 scrub ok 2024-02-05T06:54:04.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:04 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3513288841' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:54:04.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:04 smithi028 ceph-mon[37984]: from='client.25990 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:54:04.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:04 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:54:04.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:04 smithi028 ceph-mon[29442]: 4.1 scrub starts 2024-02-05T06:54:04.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:04 smithi028 ceph-mon[29442]: 4.1 scrub ok 2024-02-05T06:54:04.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:04 smithi028 ceph-mon[29442]: 1.0 scrub starts 2024-02-05T06:54:04.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:04 smithi028 ceph-mon[29442]: 1.0 scrub ok 2024-02-05T06:54:04.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:04 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3513288841' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:54:04.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:04 smithi028 ceph-mon[29442]: from='client.25990 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:54:04.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:04 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:54:04.965 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:04.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:04 smithi167 ceph-mon[32774]: 4.1 scrub starts 2024-02-05T06:54:04.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:04 smithi167 ceph-mon[32774]: 4.1 scrub ok 2024-02-05T06:54:04.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:04 smithi167 ceph-mon[32774]: 1.0 scrub starts 2024-02-05T06:54:04.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:04 smithi167 ceph-mon[32774]: 1.0 scrub ok 2024-02-05T06:54:04.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:04 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3513288841' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:54:04.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:04 smithi167 ceph-mon[32774]: from='client.25990 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:54:04.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:04 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:54:04.995 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:04.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:04.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:05.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:05.135 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:05.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:05.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:05.237 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:05.272 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:05.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:05.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:05.373 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:05.408 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:05.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:05.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:05.510 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:05.551 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:05.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:05.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:05.653 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:05.691 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:05.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:05.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:05.793 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:05.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:05 smithi028 ceph-mon[37984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:54:05.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:05 smithi028 ceph-mon[37984]: from='client.25990 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:54:05.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:05 smithi028 ceph-mon[37984]: osdmap e471: 8 total, 8 up, 8 in 2024-02-05T06:54:05.824 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:05 smithi028 ceph-mon[37984]: pgmap v989: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 714 B/s rd, 0 B/s wr, 1.02k op/s 2024-02-05T06:54:05.825 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:05 smithi028 ceph-mon[29442]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:54:05.825 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:05 smithi028 ceph-mon[29442]: from='client.25990 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:54:05.825 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:05 smithi028 ceph-mon[29442]: osdmap e471: 8 total, 8 up, 8 in 2024-02-05T06:54:05.825 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:05 smithi028 ceph-mon[29442]: pgmap v989: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 714 B/s rd, 0 B/s wr, 1.02k op/s 2024-02-05T06:54:05.832 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:05.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:05.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:05.934 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:05.973 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:05.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:05.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:05.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:05 smithi167 ceph-mon[32774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:54:05.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:05 smithi167 ceph-mon[32774]: from='client.25990 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:54:05.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:05 smithi167 ceph-mon[32774]: osdmap e471: 8 total, 8 up, 8 in 2024-02-05T06:54:05.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:05 smithi167 ceph-mon[32774]: pgmap v989: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 714 B/s rd, 0 B/s wr, 1.02k op/s 2024-02-05T06:54:06.076 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:06.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:06.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:06.213 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:06.256 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:06.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:06.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:06.358 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:06.416 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:06.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:06.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:06.518 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:06.561 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:06.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:06.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:06.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:06.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:06.703 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:06.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:06.804 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:06.843 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:06.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:06.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:06.945 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:06.983 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:06.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:06.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:07.085 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:07.120 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:07.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:07.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:07.222 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:07.252 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:07.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:07.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:07.353 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:07.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:07.399 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:07.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:07.500 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:07.536 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:07.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:07.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:07.638 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:07.674 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:07.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:07.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:07.728 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set nodeep-scrub 2024-02-05T06:54:07.776 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:07.831 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:07.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:07.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:07.933 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:07.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:07 smithi028 ceph-mon[29442]: pgmap v990: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 669 op/s 2024-02-05T06:54:07.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:07 smithi028 ceph-mon[37984]: pgmap v990: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 669 op/s 2024-02-05T06:54:07.963 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:07.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:07.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:07.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:07 smithi167 ceph-mon[32774]: pgmap v990: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 669 op/s 2024-02-05T06:54:08.065 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:08.095 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:08.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:08.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:08.197 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:08.228 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:08.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:08.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:08.329 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:08.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:08.365 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:08.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:08.466 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:08.507 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:08.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:08.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:08.608 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:08.646 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:08.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:08.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:08.747 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:08.780 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:08.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:08.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:08.836 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:54:08.836 INFO:teuthology.orchestra.run.smithi028.stdout: "ops": [], 2024-02-05T06:54:08.836 INFO:teuthology.orchestra.run.smithi028.stdout: "complaint_time": 30, 2024-02-05T06:54:08.836 INFO:teuthology.orchestra.run.smithi028.stdout: "num_blocked_ops": 0 2024-02-05T06:54:08.836 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:54:08.882 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:08.883 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:08 smithi028 ceph-mon[29442]: 11.7 deep-scrub ok 2024-02-05T06:54:08.883 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:08 smithi028 ceph-mon[37984]: 11.7 deep-scrub ok 2024-02-05T06:54:08.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:08.916 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:08.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:08.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:08 smithi167 ceph-mon[32774]: 11.7 deep-scrub ok 2024-02-05T06:54:09.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:09.061 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:09.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:09.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:09.163 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:09.191 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:09.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:09.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:09.293 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:09.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:09.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:09.443 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:09.474 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:09.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:09.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:09.496 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:54:09.509 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-05T06:54:09.576 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:09.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:09.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:09.730 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:09.761 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:09.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:09.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:09.787 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:09 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:54:09.787 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:09 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:54:09.787 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:09 smithi028 ceph-mon[37984]: pgmap v991: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-02-05T06:54:09.787 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:09 smithi028 ceph-mon[37984]: osdmap e472: 8 total, 8 up, 8 in 2024-02-05T06:54:09.787 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:09 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:54:09.787 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:09 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:54:09.787 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:09 smithi028 ceph-mon[29442]: pgmap v991: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-02-05T06:54:09.787 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:09 smithi028 ceph-mon[29442]: osdmap e472: 8 total, 8 up, 8 in 2024-02-05T06:54:09.863 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:09.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:09.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:09.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:09 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:54:09.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:09 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:54:09.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:09 smithi167 ceph-mon[32774]: pgmap v991: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-02-05T06:54:09.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:09 smithi167 ceph-mon[32774]: osdmap e472: 8 total, 8 up, 8 in 2024-02-05T06:54:10.001 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:10.044 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:10.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:10.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:10.146 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:10.158 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:54:10.158 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:54:10.158 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:54:10.158 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:54:10.158 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:54:10.158 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:54:10.158 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:54:10.158 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:54:10.159 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:54:10.159 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:54:10.159 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:54:10.159 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:54:10.159 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:54:10.159 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:54:10.159 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:54:10.159 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:54:10.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:10.180 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:10.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:10.281 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:10.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:10.311 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:10.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:10.412 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:10.435 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:10.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:10.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:10.536 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:10.574 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:10.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:10.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:10.651 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:54:10.676 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:10.704 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:10.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:10.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:10.806 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:10.848 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:10.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:10.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:10.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:10 smithi028 ceph-mon[29442]: from='client.25996 v1:172.21.15.28:0/3059095088' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:54:10.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:10 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3326119258' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:54:10.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:10 smithi028 ceph-mon[37984]: from='client.25996 v1:172.21.15.28:0/3059095088' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:54:10.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:10 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3326119258' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:54:10.950 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:10.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:10 smithi167 ceph-mon[32774]: from='client.25996 v1:172.21.15.28:0/3059095088' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:54:10.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:10 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3326119258' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:54:10.990 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:10.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:10.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:11.092 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:11.126 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:11.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:11.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:11.227 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:11.261 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:11.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:11.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:11.363 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:11.396 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:11.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:11.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:11.498 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:11.532 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:11.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:11.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:11.634 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:11.675 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:11.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:11.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:11.778 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:11.807 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:11.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:11.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:11.833 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":992,"stamp":"2024-02-05T06:54:08.671363+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3236698484,"num_objects":49578,"num_object_clones":0,"num_object_copies":148734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49578,"num_whiteouts":0,"num_read":2375,"num_read_kb":2127,"num_write":205556,"num_write_kb":8125678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":411,"num_bytes_recovered":276079,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79763,"ondisk_log_size":79763,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":15990032,"kb_used_data":9583048,"kb_used_omap":12,"kb_used_meta":6406451,"kb_avail":733971184,"statfs":{"total":767960285184,"available":751586492416,"internally_reserved":0,"allocated":9813041152,"data_stored":9809849052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":6560206385},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[9,4,8,11,11,7,1,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-668925952,"num_objects":-10207,"num_object_clones":0,"num_object_copies":-30621,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10207,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10207,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.069649"},"pg_stats":[{"pgid":"11.3","version":"471'16816","reported_seq":17200,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:07.262994+0000","last_change":"2024-02-05T06:53:28.707185+0000","last_active":"2024-02-05T06:54:07.262994+0000","last_peered":"2024-02-05T06:54:07.262994+0000","last_clean":"2024-02-05T06:54:07.262994+0000","last_became_active":"2024-02-05T06:51:13.269833+0000","last_became_peered":"2024-02-05T06:51:13.269833+0000","last_unstale":"2024-02-05T06:54:07.262994+0000","last_undegraded":"2024-02-05T06:54:07.262994+0000","last_fullsized":"2024-02-05T06:54:07.262994+0000","mapping_epoch":418,"log_start":"456'11803","ondisk_log_start":"456'11803","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"462'14268","last_scrub_stamp":"2024-02-05T06:53:28.706913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:53:28.706913+0000","objects_scrubbed":5685,"log_size":5013,"log_dups_size":0,"ondisk_log_size":5013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:45.571701+0000","scrub_duration":1678,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201523200,"num_objects":3075,"num_object_clones":0,"num_object_copies":9225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12789,"num_write_kb":507648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"465'35","reported_seq":756,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.524386+0000","last_change":"2024-02-05T06:52:45.303488+0000","last_active":"2024-02-05T06:54:04.524386+0000","last_peered":"2024-02-05T06:54:04.524386+0000","last_clean":"2024-02-05T06:54:04.524386+0000","last_became_active":"2024-02-05T06:45:09.796902+0000","last_became_peered":"2024-02-05T06:45:09.796902+0000","last_unstale":"2024-02-05T06:54:04.524386+0000","last_undegraded":"2024-02-05T06:54:04.524386+0000","last_fullsized":"2024-02-05T06:54:04.524386+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'25","last_scrub_stamp":"2024-02-05T06:52:45.303356+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-02-05T06:35:04.584657+0000","last_clean_scrub_stamp":"2024-02-05T06:52:45.303356+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":889,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.527060+0000","last_change":"2024-02-05T06:52:48.393899+0000","last_active":"2024-02-05T06:54:04.527060+0000","last_peered":"2024-02-05T06:54:04.527060+0000","last_clean":"2024-02-05T06:54:04.527060+0000","last_became_active":"2024-02-05T06:36:54.971304+0000","last_became_peered":"2024-02-05T06:36:54.971304+0000","last_unstale":"2024-02-05T06:54:04.527060+0000","last_undegraded":"2024-02-05T06:54:04.527060+0000","last_fullsized":"2024-02-05T06:54:04.527060+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:48.393628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:52:48.393628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"249'13","reported_seq":1204,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.507711+0000","last_change":"2024-02-05T06:53:56.010563+0000","last_active":"2024-02-05T06:54:04.507711+0000","last_peered":"2024-02-05T06:54:04.507711+0000","last_clean":"2024-02-05T06:54:04.507711+0000","last_became_active":"2024-02-05T06:41:43.413139+0000","last_became_peered":"2024-02-05T06:41:43.413139+0000","last_unstale":"2024-02-05T06:54:04.507711+0000","last_undegraded":"2024-02-05T06:54:04.507711+0000","last_fullsized":"2024-02-05T06:54:04.507711+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'13","last_scrub_stamp":"2024-02-05T06:53:56.009983+0000","last_deep_scrub":"92'9","last_deep_scrub_stamp":"2024-02-05T06:34:52.283780+0000","last_clean_scrub_stamp":"2024-02-05T06:53:56.009983+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:08.067138+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":415,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.505171+0000","last_change":"2024-02-05T06:52:46.097175+0000","last_active":"2024-02-05T06:54:04.505171+0000","last_peered":"2024-02-05T06:54:04.505171+0000","last_clean":"2024-02-05T06:54:04.505171+0000","last_became_active":"2024-02-05T06:45:09.784229+0000","last_became_peered":"2024-02-05T06:45:09.784229+0000","last_unstale":"2024-02-05T06:54:04.505171+0000","last_undegraded":"2024-02-05T06:54:04.505171+0000","last_fullsized":"2024-02-05T06:54:04.505171+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:46.096965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:30.057099+0000","last_clean_scrub_stamp":"2024-02-05T06:52:46.096965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":1007,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.516761+0000","last_change":"2024-02-05T06:52:25.102060+0000","last_active":"2024-02-05T06:54:04.516761+0000","last_peered":"2024-02-05T06:54:04.516761+0000","last_clean":"2024-02-05T06:54:04.516761+0000","last_became_active":"2024-02-05T06:40:30.095202+0000","last_became_peered":"2024-02-05T06:40:30.095202+0000","last_unstale":"2024-02-05T06:54:04.516761+0000","last_undegraded":"2024-02-05T06:54:04.516761+0000","last_fullsized":"2024-02-05T06:54:04.516761+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:52:25.101936+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:41:50.706159+0000","last_clean_scrub_stamp":"2024-02-05T06:52:25.101936+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.2","version":"471'12987","reported_seq":13192,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:07.323445+0000","last_change":"2024-02-05T06:52:53.157729+0000","last_active":"2024-02-05T06:54:07.323445+0000","last_peered":"2024-02-05T06:54:07.323445+0000","last_clean":"2024-02-05T06:54:07.323445+0000","last_became_active":"2024-02-05T06:51:13.275602+0000","last_became_peered":"2024-02-05T06:51:13.275602+0000","last_unstale":"2024-02-05T06:54:07.323445+0000","last_undegraded":"2024-02-05T06:54:07.323445+0000","last_fullsized":"2024-02-05T06:54:07.323445+0000","mapping_epoch":418,"log_start":"446'7200","ondisk_log_start":"446'7200","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'8067","last_scrub_stamp":"2024-02-05T06:52:53.155060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:52:53.155060+0000","objects_scrubbed":8022,"log_size":5787,"log_dups_size":0,"ondisk_log_size":5787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:10.006447+0000","scrub_duration":2122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201129984,"num_objects":3069,"num_object_clones":0,"num_object_copies":9207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12987,"num_write_kb":513792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"465'53","reported_seq":782,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.507410+0000","last_change":"2024-02-05T06:52:40.987741+0000","last_active":"2024-02-05T06:54:04.507410+0000","last_peered":"2024-02-05T06:54:04.507410+0000","last_clean":"2024-02-05T06:54:04.507410+0000","last_became_active":"2024-02-05T06:45:09.782645+0000","last_became_peered":"2024-02-05T06:45:09.782645+0000","last_unstale":"2024-02-05T06:54:04.507410+0000","last_undegraded":"2024-02-05T06:54:04.507410+0000","last_fullsized":"2024-02-05T06:54:04.507410+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'39","last_scrub_stamp":"2024-02-05T06:52:40.987572+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-02-05T06:35:26.274944+0000","last_clean_scrub_stamp":"2024-02-05T06:52:40.987572+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:04.805989+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":918,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.511914+0000","last_change":"2024-02-05T06:53:45.072923+0000","last_active":"2024-02-05T06:54:04.511914+0000","last_peered":"2024-02-05T06:54:04.511914+0000","last_clean":"2024-02-05T06:54:04.511914+0000","last_became_active":"2024-02-05T06:39:13.240038+0000","last_became_peered":"2024-02-05T06:39:13.240038+0000","last_unstale":"2024-02-05T06:54:04.511914+0000","last_undegraded":"2024-02-05T06:54:04.511914+0000","last_fullsized":"2024-02-05T06:54:04.511914+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:45.072812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:53:45.072812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:02.736389+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":897,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.505311+0000","last_change":"2024-02-05T06:52:38.073950+0000","last_active":"2024-02-05T06:54:04.505311+0000","last_peered":"2024-02-05T06:54:04.505311+0000","last_clean":"2024-02-05T06:54:04.505311+0000","last_became_active":"2024-02-05T06:36:54.958082+0000","last_became_peered":"2024-02-05T06:36:54.958082+0000","last_unstale":"2024-02-05T06:54:04.505311+0000","last_undegraded":"2024-02-05T06:54:04.505311+0000","last_fullsized":"2024-02-05T06:54:04.505311+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:38.072995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:38.072995+0000","last_clean_scrub_stamp":"2024-02-05T06:52:38.072995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":918,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.508502+0000","last_change":"2024-02-05T06:53:29.258337+0000","last_active":"2024-02-05T06:54:04.508502+0000","last_peered":"2024-02-05T06:54:04.508502+0000","last_clean":"2024-02-05T06:54:04.508502+0000","last_became_active":"2024-02-05T06:36:54.958805+0000","last_became_peered":"2024-02-05T06:36:54.958805+0000","last_unstale":"2024-02-05T06:54:04.508502+0000","last_undegraded":"2024-02-05T06:54:04.508502+0000","last_fullsized":"2024-02-05T06:54:04.508502+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:29.258153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:41.833719+0000","last_clean_scrub_stamp":"2024-02-05T06:53:29.258153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:38.792895+0000","scrub_duration":273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"83'128","reported_seq":927,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.503267+0000","last_change":"2024-02-05T06:53:55.916525+0000","last_active":"2024-02-05T06:54:04.503267+0000","last_peered":"2024-02-05T06:54:04.503267+0000","last_clean":"2024-02-05T06:54:04.503267+0000","last_became_active":"2024-02-05T06:41:46.053324+0000","last_became_peered":"2024-02-05T06:41:46.053324+0000","last_unstale":"2024-02-05T06:54:04.503267+0000","last_undegraded":"2024-02-05T06:54:04.503267+0000","last_fullsized":"2024-02-05T06:54:04.503267+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:53:55.916390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:53:55.916390+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:19.660693+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"83'128","reported_seq":1063,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.503711+0000","last_change":"2024-02-05T06:52:21.969321+0000","last_active":"2024-02-05T06:54:04.503711+0000","last_peered":"2024-02-05T06:54:04.503711+0000","last_clean":"2024-02-05T06:54:04.503711+0000","last_became_active":"2024-02-05T06:51:59.303575+0000","last_became_peered":"2024-02-05T06:51:59.303575+0000","last_unstale":"2024-02-05T06:54:04.503711+0000","last_undegraded":"2024-02-05T06:54:04.503711+0000","last_fullsized":"2024-02-05T06:54:04.503711+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:52:21.969067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:52:21.969067+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"471'12704","reported_seq":12914,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:07.288825+0000","last_change":"2024-02-05T06:53:32.058096+0000","last_active":"2024-02-05T06:54:07.288825+0000","last_peered":"2024-02-05T06:54:07.288825+0000","last_clean":"2024-02-05T06:54:07.288825+0000","last_became_active":"2024-02-05T06:51:13.273200+0000","last_became_peered":"2024-02-05T06:51:13.273200+0000","last_unstale":"2024-02-05T06:54:07.288825+0000","last_undegraded":"2024-02-05T06:54:07.288825+0000","last_fullsized":"2024-02-05T06:54:07.288825+0000","mapping_epoch":418,"log_start":"448'7700","ondisk_log_start":"448'7700","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"463'10360","last_scrub_stamp":"2024-02-05T06:53:32.057849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:53:32.057849+0000","objects_scrubbed":5494,"log_size":5004,"log_dups_size":0,"ondisk_log_size":5004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:01.054911+0000","scrub_duration":2042,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202244096,"num_objects":3086,"num_object_clones":0,"num_object_copies":9258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12704,"num_write_kb":505280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"465'32","reported_seq":766,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.510455+0000","last_change":"2024-02-05T06:53:31.987710+0000","last_active":"2024-02-05T06:54:04.510455+0000","last_peered":"2024-02-05T06:54:04.510455+0000","last_clean":"2024-02-05T06:54:04.510455+0000","last_became_active":"2024-02-05T06:45:09.774868+0000","last_became_peered":"2024-02-05T06:45:09.774868+0000","last_unstale":"2024-02-05T06:54:04.510455+0000","last_undegraded":"2024-02-05T06:54:04.510455+0000","last_fullsized":"2024-02-05T06:54:04.510455+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:53:31.987585+0000","last_deep_scrub":"115'16","last_deep_scrub_stamp":"2024-02-05T06:36:34.793243+0000","last_clean_scrub_stamp":"2024-02-05T06:53:31.987585+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:40.316102+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":613,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.503489+0000","last_change":"2024-02-05T06:53:56.618075+0000","last_active":"2024-02-05T06:54:04.503489+0000","last_peered":"2024-02-05T06:54:04.503489+0000","last_clean":"2024-02-05T06:54:04.503489+0000","last_became_active":"2024-02-05T06:40:27.086808+0000","last_became_peered":"2024-02-05T06:40:27.086808+0000","last_unstale":"2024-02-05T06:54:04.503489+0000","last_undegraded":"2024-02-05T06:54:04.503489+0000","last_fullsized":"2024-02-05T06:54:04.503489+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:56.617959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:41.860956+0000","last_clean_scrub_stamp":"2024-02-05T06:53:56.617959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:03.244501+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":501,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.504412+0000","last_change":"2024-02-05T06:52:19.870040+0000","last_active":"2024-02-05T06:54:04.504412+0000","last_peered":"2024-02-05T06:54:04.504412+0000","last_clean":"2024-02-05T06:54:04.504412+0000","last_became_active":"2024-02-05T06:41:43.417343+0000","last_became_peered":"2024-02-05T06:41:43.417343+0000","last_unstale":"2024-02-05T06:54:04.504412+0000","last_undegraded":"2024-02-05T06:54:04.504412+0000","last_fullsized":"2024-02-05T06:54:04.504412+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:19.869888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:42:55.610276+0000","last_clean_scrub_stamp":"2024-02-05T06:52:19.869888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":883,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.524096+0000","last_change":"2024-02-05T06:54:02.502678+0000","last_active":"2024-02-05T06:54:04.524096+0000","last_peered":"2024-02-05T06:54:04.524096+0000","last_clean":"2024-02-05T06:54:04.524096+0000","last_became_active":"2024-02-05T06:41:43.410600+0000","last_became_peered":"2024-02-05T06:41:43.410600+0000","last_unstale":"2024-02-05T06:54:04.524096+0000","last_undegraded":"2024-02-05T06:54:04.524096+0000","last_fullsized":"2024-02-05T06:54:04.524096+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:54:02.502543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:58.479208+0000","last_clean_scrub_stamp":"2024-02-05T06:54:02.502543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:20.782865+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"471'16998","reported_seq":17392,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:07.928611+0000","last_change":"2024-02-05T06:53:34.387508+0000","last_active":"2024-02-05T06:54:07.928611+0000","last_peered":"2024-02-05T06:54:07.928611+0000","last_clean":"2024-02-05T06:54:07.928611+0000","last_became_active":"2024-02-05T06:51:13.273376+0000","last_became_peered":"2024-02-05T06:51:13.273376+0000","last_unstale":"2024-02-05T06:54:07.928611+0000","last_undegraded":"2024-02-05T06:54:07.928611+0000","last_fullsized":"2024-02-05T06:54:07.928611+0000","mapping_epoch":418,"log_start":"456'12603","ondisk_log_start":"456'12603","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'14766","last_scrub_stamp":"2024-02-05T06:53:34.387362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:53:34.387362+0000","objects_scrubbed":5240,"log_size":4395,"log_dups_size":0,"ondisk_log_size":4395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:38.451839+0000","scrub_duration":1825,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":195493888,"num_objects":2983,"num_object_clones":0,"num_object_copies":8949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12769,"num_write_kb":504064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"465'46","reported_seq":710,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.509624+0000","last_change":"2024-02-05T06:52:38.220941+0000","last_active":"2024-02-05T06:54:04.509624+0000","last_peered":"2024-02-05T06:54:04.509624+0000","last_clean":"2024-02-05T06:54:04.509624+0000","last_became_active":"2024-02-05T06:39:13.257978+0000","last_became_peered":"2024-02-05T06:39:13.257978+0000","last_unstale":"2024-02-05T06:54:04.509624+0000","last_undegraded":"2024-02-05T06:54:04.509624+0000","last_fullsized":"2024-02-05T06:54:04.509624+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'34","last_scrub_stamp":"2024-02-05T06:52:38.220772+0000","last_deep_scrub":"295'34","last_deep_scrub_stamp":"2024-02-05T06:46:44.355563+0000","last_clean_scrub_stamp":"2024-02-05T06:52:38.220772+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":889,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.520208+0000","last_change":"2024-02-05T06:52:30.417398+0000","last_active":"2024-02-05T06:54:04.520208+0000","last_peered":"2024-02-05T06:54:04.520208+0000","last_clean":"2024-02-05T06:54:04.520208+0000","last_became_active":"2024-02-05T06:40:27.213439+0000","last_became_peered":"2024-02-05T06:40:27.213439+0000","last_unstale":"2024-02-05T06:54:04.520208+0000","last_undegraded":"2024-02-05T06:54:04.520208+0000","last_fullsized":"2024-02-05T06:54:04.520208+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:30.417237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:52.482566+0000","last_clean_scrub_stamp":"2024-02-05T06:52:30.417237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"319'13","reported_seq":807,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.503929+0000","last_change":"2024-02-05T06:52:49.830770+0000","last_active":"2024-02-05T06:54:04.503929+0000","last_peered":"2024-02-05T06:54:04.503929+0000","last_clean":"2024-02-05T06:54:04.503929+0000","last_became_active":"2024-02-05T06:45:09.766220+0000","last_became_peered":"2024-02-05T06:45:09.766220+0000","last_unstale":"2024-02-05T06:54:04.503929+0000","last_undegraded":"2024-02-05T06:54:04.503929+0000","last_fullsized":"2024-02-05T06:54:04.503929+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'13","last_scrub_stamp":"2024-02-05T06:52:49.830615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:52:49.830615+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:13.660513+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":510,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.504569+0000","last_change":"2024-02-05T06:53:47.006909+0000","last_active":"2024-02-05T06:54:04.504569+0000","last_peered":"2024-02-05T06:54:04.504569+0000","last_clean":"2024-02-05T06:54:04.504569+0000","last_became_active":"2024-02-05T06:41:43.418066+0000","last_became_peered":"2024-02-05T06:41:43.418066+0000","last_unstale":"2024-02-05T06:54:04.504569+0000","last_undegraded":"2024-02-05T06:54:04.504569+0000","last_fullsized":"2024-02-05T06:54:04.504569+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:47.006807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:48:34.623889+0000","last_clean_scrub_stamp":"2024-02-05T06:53:47.006807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:54.416716+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":923,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.503900+0000","last_change":"2024-02-05T06:53:44.829783+0000","last_active":"2024-02-05T06:54:04.503900+0000","last_peered":"2024-02-05T06:54:04.503900+0000","last_clean":"2024-02-05T06:54:04.503900+0000","last_became_active":"2024-02-05T06:41:46.053993+0000","last_became_peered":"2024-02-05T06:41:46.053993+0000","last_unstale":"2024-02-05T06:54:04.503900+0000","last_undegraded":"2024-02-05T06:54:04.503900+0000","last_fullsized":"2024-02-05T06:54:04.503900+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:53:44.829662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:53:44.829662+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:49.174258+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"471'13073","reported_seq":13255,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:06.840794+0000","last_change":"2024-02-05T06:53:36.664242+0000","last_active":"2024-02-05T06:54:06.840794+0000","last_peered":"2024-02-05T06:54:06.840794+0000","last_clean":"2024-02-05T06:54:06.840794+0000","last_became_active":"2024-02-05T06:51:13.274262+0000","last_became_peered":"2024-02-05T06:51:13.274262+0000","last_unstale":"2024-02-05T06:54:06.840794+0000","last_undegraded":"2024-02-05T06:54:06.840794+0000","last_fullsized":"2024-02-05T06:54:06.840794+0000","mapping_epoch":418,"log_start":"452'8500","ondisk_log_start":"452'8500","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'10930","last_scrub_stamp":"2024-02-05T06:53:36.664069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:53:36.664069+0000","objects_scrubbed":5464,"log_size":4573,"log_dups_size":0,"ondisk_log_size":4573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:43.016542+0000","scrub_duration":1595,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213319680,"num_objects":3255,"num_object_clones":0,"num_object_copies":9765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13073,"num_write_kb":522496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":549,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.518695+0000","last_change":"2024-02-05T06:53:50.578263+0000","last_active":"2024-02-05T06:54:04.518695+0000","last_peered":"2024-02-05T06:54:04.518695+0000","last_clean":"2024-02-05T06:54:04.518695+0000","last_became_active":"2024-02-05T06:45:09.778609+0000","last_became_peered":"2024-02-05T06:45:09.778609+0000","last_unstale":"2024-02-05T06:54:04.518695+0000","last_undegraded":"2024-02-05T06:54:04.518695+0000","last_fullsized":"2024-02-05T06:54:04.518695+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:50.578152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:53:50.578152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:55.388149+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"465'50","reported_seq":990,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.520299+0000","last_change":"2024-02-05T06:51:40.451314+0000","last_active":"2024-02-05T06:54:04.520299+0000","last_peered":"2024-02-05T06:54:04.520299+0000","last_clean":"2024-02-05T06:54:04.520299+0000","last_became_active":"2024-02-05T06:40:27.277747+0000","last_became_peered":"2024-02-05T06:40:27.277747+0000","last_unstale":"2024-02-05T06:54:04.520299+0000","last_undegraded":"2024-02-05T06:54:04.520299+0000","last_fullsized":"2024-02-05T06:54:04.520299+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"424'38","last_scrub_stamp":"2024-02-05T06:51:40.451087+0000","last_deep_scrub":"424'38","last_deep_scrub_stamp":"2024-02-05T06:51:40.451087+0000","last_clean_scrub_stamp":"2024-02-05T06:51:40.451087+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":895,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.526708+0000","last_change":"2024-02-05T06:52:53.475658+0000","last_active":"2024-02-05T06:54:04.526708+0000","last_peered":"2024-02-05T06:54:04.526708+0000","last_clean":"2024-02-05T06:54:04.526708+0000","last_became_active":"2024-02-05T06:40:27.093739+0000","last_became_peered":"2024-02-05T06:40:27.093739+0000","last_unstale":"2024-02-05T06:54:04.526708+0000","last_undegraded":"2024-02-05T06:54:04.526708+0000","last_fullsized":"2024-02-05T06:54:04.526708+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:53.475540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:53.475540+0000","last_clean_scrub_stamp":"2024-02-05T06:52:53.475540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:23.045094+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":408,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.523894+0000","last_change":"2024-02-05T06:53:28.389799+0000","last_active":"2024-02-05T06:54:04.523894+0000","last_peered":"2024-02-05T06:54:04.523894+0000","last_clean":"2024-02-05T06:54:04.523894+0000","last_became_active":"2024-02-05T06:45:09.780390+0000","last_became_peered":"2024-02-05T06:45:09.780390+0000","last_unstale":"2024-02-05T06:54:04.523894+0000","last_undegraded":"2024-02-05T06:54:04.523894+0000","last_fullsized":"2024-02-05T06:54:04.523894+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:28.389689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:45:49.145117+0000","last_clean_scrub_stamp":"2024-02-05T06:53:28.389689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:51.789095+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":667,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.507238+0000","last_change":"2024-02-05T06:53:40.036388+0000","last_active":"2024-02-05T06:54:04.507238+0000","last_peered":"2024-02-05T06:54:04.507238+0000","last_clean":"2024-02-05T06:54:04.507238+0000","last_became_active":"2024-02-05T06:51:54.322200+0000","last_became_peered":"2024-02-05T06:51:54.322200+0000","last_unstale":"2024-02-05T06:54:04.507238+0000","last_undegraded":"2024-02-05T06:54:04.507238+0000","last_fullsized":"2024-02-05T06:54:04.507238+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:53:40.036268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:53:40.036268+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:59.840080+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"471'12993","reported_seq":13211,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:07.956390+0000","last_change":"2024-02-05T06:53:36.641829+0000","last_active":"2024-02-05T06:54:07.956390+0000","last_peered":"2024-02-05T06:54:07.956390+0000","last_clean":"2024-02-05T06:54:07.956390+0000","last_became_active":"2024-02-05T06:51:13.267644+0000","last_became_peered":"2024-02-05T06:51:13.267644+0000","last_unstale":"2024-02-05T06:54:07.956390+0000","last_undegraded":"2024-02-05T06:54:07.956390+0000","last_fullsized":"2024-02-05T06:54:07.956390+0000","mapping_epoch":418,"log_start":"454'8600","ondisk_log_start":"454'8600","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'10815","last_scrub_stamp":"2024-02-05T06:53:36.641665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:53:36.641665+0000","objects_scrubbed":5282,"log_size":4393,"log_dups_size":0,"ondisk_log_size":4393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:40.621164+0000","scrub_duration":1905,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199557120,"num_objects":3045,"num_object_clones":0,"num_object_copies":9135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3045,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12993,"num_write_kb":513216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"326'11","reported_seq":1149,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.520439+0000","last_change":"2024-02-05T06:53:59.536292+0000","last_active":"2024-02-05T06:54:04.520439+0000","last_peered":"2024-02-05T06:54:04.520439+0000","last_clean":"2024-02-05T06:54:04.520439+0000","last_became_active":"2024-02-05T06:36:54.954819+0000","last_became_peered":"2024-02-05T06:36:54.954819+0000","last_unstale":"2024-02-05T06:54:04.520439+0000","last_undegraded":"2024-02-05T06:54:04.520439+0000","last_fullsized":"2024-02-05T06:54:04.520439+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'11","last_scrub_stamp":"2024-02-05T06:53:59.535905+0000","last_deep_scrub":"326'11","last_deep_scrub_stamp":"2024-02-05T06:45:56.212177+0000","last_clean_scrub_stamp":"2024-02-05T06:53:59.535905+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:19.671764+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"465'42","reported_seq":833,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.503394+0000","last_change":"2024-02-05T06:54:02.913296+0000","last_active":"2024-02-05T06:54:04.503394+0000","last_peered":"2024-02-05T06:54:04.503394+0000","last_clean":"2024-02-05T06:54:04.503394+0000","last_became_active":"2024-02-05T06:45:09.764696+0000","last_became_peered":"2024-02-05T06:45:09.764696+0000","last_unstale":"2024-02-05T06:54:04.503394+0000","last_undegraded":"2024-02-05T06:54:04.503394+0000","last_fullsized":"2024-02-05T06:54:04.503394+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'42","last_scrub_stamp":"2024-02-05T06:54:02.913160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:54:02.913160+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:26.597570+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"95'2","reported_seq":908,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.509625+0000","last_change":"2024-02-05T06:53:41.525189+0000","last_active":"2024-02-05T06:54:04.509625+0000","last_peered":"2024-02-05T06:54:04.509625+0000","last_clean":"2024-02-05T06:54:04.509625+0000","last_became_active":"2024-02-05T06:36:54.975093+0000","last_became_peered":"2024-02-05T06:36:54.975093+0000","last_unstale":"2024-02-05T06:54:04.509625+0000","last_undegraded":"2024-02-05T06:54:04.509625+0000","last_fullsized":"2024-02-05T06:54:04.509625+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-05T06:53:41.524836+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-05T06:35:25.616708+0000","last_clean_scrub_stamp":"2024-02-05T06:53:41.524836+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:59.895714+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":594,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.515974+0000","last_change":"2024-02-05T06:53:53.881450+0000","last_active":"2024-02-05T06:54:04.515974+0000","last_peered":"2024-02-05T06:54:04.515974+0000","last_clean":"2024-02-05T06:54:04.515974+0000","last_became_active":"2024-02-05T06:40:27.099168+0000","last_became_peered":"2024-02-05T06:40:27.099168+0000","last_unstale":"2024-02-05T06:54:04.515974+0000","last_undegraded":"2024-02-05T06:54:04.515974+0000","last_fullsized":"2024-02-05T06:54:04.515974+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:53.881337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:53:53.881337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:10.513794+0000","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":1091,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.523954+0000","last_change":"2024-02-05T06:52:18.376581+0000","last_active":"2024-02-05T06:54:04.523954+0000","last_peered":"2024-02-05T06:54:04.523954+0000","last_clean":"2024-02-05T06:54:04.523954+0000","last_became_active":"2024-02-05T06:51:54.332037+0000","last_became_peered":"2024-02-05T06:51:54.332037+0000","last_unstale":"2024-02-05T06:54:04.523954+0000","last_undegraded":"2024-02-05T06:54:04.523954+0000","last_fullsized":"2024-02-05T06:54:04.523954+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:52:18.376463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:52:18.376463+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"471'12676","reported_seq":12904,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:07.318453+0000","last_change":"2024-02-05T06:53:30.282702+0000","last_active":"2024-02-05T06:54:07.318453+0000","last_peered":"2024-02-05T06:54:07.318453+0000","last_clean":"2024-02-05T06:54:07.318453+0000","last_became_active":"2024-02-05T06:51:13.272662+0000","last_became_peered":"2024-02-05T06:51:13.272662+0000","last_unstale":"2024-02-05T06:54:07.318453+0000","last_undegraded":"2024-02-05T06:54:07.318453+0000","last_fullsized":"2024-02-05T06:54:07.318453+0000","mapping_epoch":418,"log_start":"445'6900","ondisk_log_start":"445'6900","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"463'10228","last_scrub_stamp":"2024-02-05T06:53:30.282379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:53:30.282379+0000","objects_scrubbed":5646,"log_size":5776,"log_dups_size":0,"ondisk_log_size":5776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:40.107663+0000","scrub_duration":2319,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204668952,"num_objects":3124,"num_object_clones":0,"num_object_copies":9372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3124,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12676,"num_write_kb":505537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"335'11","reported_seq":1155,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.509627+0000","last_change":"2024-02-05T06:52:40.188002+0000","last_active":"2024-02-05T06:54:04.509627+0000","last_peered":"2024-02-05T06:54:04.509627+0000","last_clean":"2024-02-05T06:54:04.509627+0000","last_became_active":"2024-02-05T06:41:43.423638+0000","last_became_peered":"2024-02-05T06:41:43.423638+0000","last_unstale":"2024-02-05T06:54:04.509627+0000","last_undegraded":"2024-02-05T06:54:04.509627+0000","last_fullsized":"2024-02-05T06:54:04.509627+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"335'11","last_scrub_stamp":"2024-02-05T06:52:40.187864+0000","last_deep_scrub":"335'11","last_deep_scrub_stamp":"2024-02-05T06:52:40.187864+0000","last_clean_scrub_stamp":"2024-02-05T06:52:40.187864+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"465'35","reported_seq":1011,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.504822+0000","last_change":"2024-02-05T06:54:00.633400+0000","last_active":"2024-02-05T06:54:04.504822+0000","last_peered":"2024-02-05T06:54:04.504822+0000","last_clean":"2024-02-05T06:54:04.504822+0000","last_became_active":"2024-02-05T06:41:43.421030+0000","last_became_peered":"2024-02-05T06:41:43.421030+0000","last_unstale":"2024-02-05T06:54:04.504822+0000","last_undegraded":"2024-02-05T06:54:04.504822+0000","last_fullsized":"2024-02-05T06:54:04.504822+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'35","last_scrub_stamp":"2024-02-05T06:54:00.633175+0000","last_deep_scrub":"424'25","last_deep_scrub_stamp":"2024-02-05T06:51:39.813235+0000","last_clean_scrub_stamp":"2024-02-05T06:54:00.633175+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:18.151735+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":501,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.504602+0000","last_change":"2024-02-05T06:53:32.828121+0000","last_active":"2024-02-05T06:54:04.504602+0000","last_peered":"2024-02-05T06:54:04.504602+0000","last_clean":"2024-02-05T06:54:04.504602+0000","last_became_active":"2024-02-05T06:41:43.411732+0000","last_became_peered":"2024-02-05T06:41:43.411732+0000","last_unstale":"2024-02-05T06:54:04.504602+0000","last_undegraded":"2024-02-05T06:54:04.504602+0000","last_fullsized":"2024-02-05T06:54:04.504602+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:32.828031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:28.353956+0000","last_clean_scrub_stamp":"2024-02-05T06:53:32.828031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:39.494277+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":875,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.508960+0000","last_change":"2024-02-05T06:53:52.550097+0000","last_active":"2024-02-05T06:54:04.508960+0000","last_peered":"2024-02-05T06:54:04.508960+0000","last_clean":"2024-02-05T06:54:04.508960+0000","last_became_active":"2024-02-05T06:33:35.600485+0000","last_became_peered":"2024-02-05T06:33:35.600485+0000","last_unstale":"2024-02-05T06:54:04.508960+0000","last_undegraded":"2024-02-05T06:54:04.508960+0000","last_fullsized":"2024-02-05T06:54:04.508960+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:52.549976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:53:52.549976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:17.433853+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":1096,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.511038+0000","last_change":"2024-02-05T06:52:19.674161+0000","last_active":"2024-02-05T06:54:04.511038+0000","last_peered":"2024-02-05T06:54:04.511038+0000","last_clean":"2024-02-05T06:54:04.511038+0000","last_became_active":"2024-02-05T06:51:54.332371+0000","last_became_peered":"2024-02-05T06:51:54.332371+0000","last_unstale":"2024-02-05T06:54:04.511038+0000","last_undegraded":"2024-02-05T06:54:04.511038+0000","last_fullsized":"2024-02-05T06:54:04.511038+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:52:19.674004+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:48:25.725892+0000","last_clean_scrub_stamp":"2024-02-05T06:52:19.674004+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"471'12513","reported_seq":12715,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:07.904219+0000","last_change":"2024-02-05T06:52:53.084302+0000","last_active":"2024-02-05T06:54:07.904219+0000","last_peered":"2024-02-05T06:54:07.904219+0000","last_clean":"2024-02-05T06:54:07.904219+0000","last_became_active":"2024-02-05T06:51:13.269283+0000","last_became_peered":"2024-02-05T06:51:13.269283+0000","last_unstale":"2024-02-05T06:54:07.904219+0000","last_undegraded":"2024-02-05T06:54:07.904219+0000","last_fullsized":"2024-02-05T06:54:07.904219+0000","mapping_epoch":418,"log_start":"452'8100","ondisk_log_start":"452'8100","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'7755","last_scrub_stamp":"2024-02-05T06:52:53.084023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:52:53.084023+0000","objects_scrubbed":7733,"log_size":4413,"log_dups_size":0,"ondisk_log_size":4413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:15.548872+0000","scrub_duration":1955,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193921024,"num_objects":2959,"num_object_clones":0,"num_object_copies":8877,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12513,"num_write_kb":495104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":904,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.511896+0000","last_change":"2024-02-05T06:52:20.642047+0000","last_active":"2024-02-05T06:54:04.511896+0000","last_peered":"2024-02-05T06:54:04.511896+0000","last_clean":"2024-02-05T06:54:04.511896+0000","last_became_active":"2024-02-05T06:41:43.411550+0000","last_became_peered":"2024-02-05T06:41:43.411550+0000","last_unstale":"2024-02-05T06:54:04.511896+0000","last_undegraded":"2024-02-05T06:54:04.511896+0000","last_fullsized":"2024-02-05T06:54:04.511896+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:20.641923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:52:20.641923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"465'39","reported_seq":1044,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.510086+0000","last_change":"2024-02-05T06:52:50.055893+0000","last_active":"2024-02-05T06:54:04.510086+0000","last_peered":"2024-02-05T06:54:04.510086+0000","last_clean":"2024-02-05T06:54:04.510086+0000","last_became_active":"2024-02-05T06:41:43.409664+0000","last_became_peered":"2024-02-05T06:41:43.409664+0000","last_unstale":"2024-02-05T06:54:04.510086+0000","last_undegraded":"2024-02-05T06:54:04.510086+0000","last_fullsized":"2024-02-05T06:54:04.510086+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'29","last_scrub_stamp":"2024-02-05T06:52:50.055600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:52:50.055600+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"445'2","reported_seq":920,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.505217+0000","last_change":"2024-02-05T06:52:37.034737+0000","last_active":"2024-02-05T06:54:04.505217+0000","last_peered":"2024-02-05T06:54:04.505217+0000","last_clean":"2024-02-05T06:54:04.505217+0000","last_became_active":"2024-02-05T06:41:43.411404+0000","last_became_peered":"2024-02-05T06:41:43.411404+0000","last_unstale":"2024-02-05T06:54:04.505217+0000","last_undegraded":"2024-02-05T06:54:04.505217+0000","last_fullsized":"2024-02-05T06:54:04.505217+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'2","last_scrub_stamp":"2024-02-05T06:52:37.034607+0000","last_deep_scrub":"445'2","last_deep_scrub_stamp":"2024-02-05T06:52:37.034607+0000","last_clean_scrub_stamp":"2024-02-05T06:52:37.034607+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":881,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.503870+0000","last_change":"2024-02-05T06:53:38.739502+0000","last_active":"2024-02-05T06:54:04.503870+0000","last_peered":"2024-02-05T06:54:04.503870+0000","last_clean":"2024-02-05T06:54:04.503870+0000","last_became_active":"2024-02-05T06:33:35.664858+0000","last_became_peered":"2024-02-05T06:33:35.664858+0000","last_unstale":"2024-02-05T06:54:04.503870+0000","last_undegraded":"2024-02-05T06:54:04.503870+0000","last_fullsized":"2024-02-05T06:54:04.503870+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:38.739377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:53:38.739377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:50.477820+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":1077,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.510701+0000","last_change":"2024-02-05T06:52:18.642202+0000","last_active":"2024-02-05T06:54:04.510701+0000","last_peered":"2024-02-05T06:54:04.510701+0000","last_clean":"2024-02-05T06:54:04.510701+0000","last_became_active":"2024-02-05T06:51:54.330806+0000","last_became_peered":"2024-02-05T06:51:54.330806+0000","last_unstale":"2024-02-05T06:54:04.510701+0000","last_undegraded":"2024-02-05T06:54:04.510701+0000","last_fullsized":"2024-02-05T06:54:04.510701+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:52:18.642076+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:38.778568+0000","last_clean_scrub_stamp":"2024-02-05T06:52:18.642076+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"471'12827","reported_seq":12944,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:07.952356+0000","last_change":"2024-02-05T06:51:13.271458+0000","last_active":"2024-02-05T06:54:07.952356+0000","last_peered":"2024-02-05T06:54:07.952356+0000","last_clean":"2024-02-05T06:54:07.952356+0000","last_became_active":"2024-02-05T06:51:13.270994+0000","last_became_peered":"2024-02-05T06:51:13.270994+0000","last_unstale":"2024-02-05T06:54:07.952356+0000","last_undegraded":"2024-02-05T06:54:07.952356+0000","last_fullsized":"2024-02-05T06:54:07.952356+0000","mapping_epoch":418,"log_start":"454'8600","ondisk_log_start":"454'8600","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:51:12.249449+0000","objects_scrubbed":0,"log_size":4227,"log_dups_size":0,"ondisk_log_size":4227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:12.546608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204013568,"num_objects":3113,"num_object_clones":0,"num_object_copies":9339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12827,"num_write_kb":510080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":501,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.504503+0000","last_change":"2024-02-05T06:52:50.894323+0000","last_active":"2024-02-05T06:54:04.504503+0000","last_peered":"2024-02-05T06:54:04.504503+0000","last_clean":"2024-02-05T06:54:04.504503+0000","last_became_active":"2024-02-05T06:41:43.417751+0000","last_became_peered":"2024-02-05T06:41:43.417751+0000","last_unstale":"2024-02-05T06:54:04.504503+0000","last_undegraded":"2024-02-05T06:54:04.504503+0000","last_fullsized":"2024-02-05T06:54:04.504503+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:50.894195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:52:50.894195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:05.470356+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"465'36","reported_seq":936,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.509518+0000","last_change":"2024-02-05T06:52:52.080789+0000","last_active":"2024-02-05T06:54:04.509518+0000","last_peered":"2024-02-05T06:54:04.509518+0000","last_clean":"2024-02-05T06:54:04.509518+0000","last_became_active":"2024-02-05T06:33:31.564604+0000","last_became_peered":"2024-02-05T06:33:31.564604+0000","last_unstale":"2024-02-05T06:54:04.509518+0000","last_undegraded":"2024-02-05T06:54:04.509518+0000","last_fullsized":"2024-02-05T06:54:04.509518+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:52:52.080633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:52:52.080633+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"355'6","reported_seq":1414,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:05.487197+0000","last_change":"2024-02-05T06:52:35.156214+0000","last_active":"2024-02-05T06:54:05.487197+0000","last_peered":"2024-02-05T06:54:05.487197+0000","last_clean":"2024-02-05T06:54:05.487197+0000","last_became_active":"2024-02-05T06:40:27.071509+0000","last_became_peered":"2024-02-05T06:40:27.071509+0000","last_unstale":"2024-02-05T06:54:05.487197+0000","last_undegraded":"2024-02-05T06:54:05.487197+0000","last_fullsized":"2024-02-05T06:54:05.487197+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'6","last_scrub_stamp":"2024-02-05T06:52:35.156085+0000","last_deep_scrub":"355'6","last_deep_scrub_stamp":"2024-02-05T06:47:46.642454+0000","last_clean_scrub_stamp":"2024-02-05T06:52:35.156085+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00074151600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":825,"num_read_kb":820,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":898,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.510895+0000","last_change":"2024-02-05T06:53:33.469434+0000","last_active":"2024-02-05T06:54:04.510895+0000","last_peered":"2024-02-05T06:54:04.510895+0000","last_clean":"2024-02-05T06:54:04.510895+0000","last_became_active":"2024-02-05T06:36:54.972264+0000","last_became_peered":"2024-02-05T06:36:54.972264+0000","last_unstale":"2024-02-05T06:54:04.510895+0000","last_undegraded":"2024-02-05T06:54:04.510895+0000","last_fullsized":"2024-02-05T06:54:04.510895+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:33.469051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:53:33.469051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:45.519225+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":901,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.504834+0000","last_change":"2024-02-05T06:53:47.688370+0000","last_active":"2024-02-05T06:54:04.504834+0000","last_peered":"2024-02-05T06:54:04.504834+0000","last_clean":"2024-02-05T06:54:04.504834+0000","last_became_active":"2024-02-05T06:41:43.410910+0000","last_became_peered":"2024-02-05T06:41:43.410910+0000","last_unstale":"2024-02-05T06:54:04.504834+0000","last_undegraded":"2024-02-05T06:54:04.504834+0000","last_fullsized":"2024-02-05T06:54:04.504834+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:47.688112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:53:47.688112+0000","last_clean_scrub_stamp":"2024-02-05T06:53:47.688112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:00.832816+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":1056,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.504074+0000","last_change":"2024-02-05T06:52:22.946733+0000","last_active":"2024-02-05T06:54:04.504074+0000","last_peered":"2024-02-05T06:54:04.504074+0000","last_clean":"2024-02-05T06:54:04.504074+0000","last_became_active":"2024-02-05T06:51:54.323320+0000","last_became_peered":"2024-02-05T06:51:54.323320+0000","last_unstale":"2024-02-05T06:54:04.504074+0000","last_undegraded":"2024-02-05T06:54:04.504074+0000","last_fullsized":"2024-02-05T06:54:04.504074+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:52:22.946483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:52:22.946483+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"471'12927","reported_seq":13044,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:07.958482+0000","last_change":"2024-02-05T06:51:13.274707+0000","last_active":"2024-02-05T06:54:07.958482+0000","last_peered":"2024-02-05T06:54:07.958482+0000","last_clean":"2024-02-05T06:54:07.958482+0000","last_became_active":"2024-02-05T06:51:13.274381+0000","last_became_peered":"2024-02-05T06:51:13.274381+0000","last_unstale":"2024-02-05T06:54:07.958482+0000","last_undegraded":"2024-02-05T06:54:07.958482+0000","last_fullsized":"2024-02-05T06:54:07.958482+0000","mapping_epoch":418,"log_start":"455'8700","ondisk_log_start":"455'8700","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:51:12.249449+0000","objects_scrubbed":0,"log_size":4227,"log_dups_size":0,"ondisk_log_size":4227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:12.545582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201523200,"num_objects":3075,"num_object_clones":0,"num_object_copies":9225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12927,"num_write_kb":512064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":608,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.517291+0000","last_change":"2024-02-05T06:54:02.593564+0000","last_active":"2024-02-05T06:54:04.517291+0000","last_peered":"2024-02-05T06:54:04.517291+0000","last_clean":"2024-02-05T06:54:04.517291+0000","last_became_active":"2024-02-05T06:41:43.418198+0000","last_became_peered":"2024-02-05T06:41:43.418198+0000","last_unstale":"2024-02-05T06:54:04.517291+0000","last_undegraded":"2024-02-05T06:54:04.517291+0000","last_fullsized":"2024-02-05T06:54:04.517291+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:54:02.593328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:54:02.593328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:12.160360+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"465'52","reported_seq":1308,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.511686+0000","last_change":"2024-02-05T06:52:39.574360+0000","last_active":"2024-02-05T06:54:04.511686+0000","last_peered":"2024-02-05T06:54:04.511686+0000","last_clean":"2024-02-05T06:54:04.511686+0000","last_became_active":"2024-02-05T06:40:27.061111+0000","last_became_peered":"2024-02-05T06:40:27.061111+0000","last_unstale":"2024-02-05T06:54:04.511686+0000","last_undegraded":"2024-02-05T06:54:04.511686+0000","last_fullsized":"2024-02-05T06:54:04.511686+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'40","last_scrub_stamp":"2024-02-05T06:52:39.574272+0000","last_deep_scrub":"413'40","last_deep_scrub_stamp":"2024-02-05T06:50:56.446985+0000","last_clean_scrub_stamp":"2024-02-05T06:52:39.574272+0000","objects_scrubbed":8,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":889,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.513666+0000","last_change":"2024-02-05T06:53:38.592980+0000","last_active":"2024-02-05T06:54:04.513666+0000","last_peered":"2024-02-05T06:54:04.513666+0000","last_clean":"2024-02-05T06:54:04.513666+0000","last_became_active":"2024-02-05T06:41:43.411335+0000","last_became_peered":"2024-02-05T06:41:43.411335+0000","last_unstale":"2024-02-05T06:54:04.513666+0000","last_undegraded":"2024-02-05T06:54:04.513666+0000","last_fullsized":"2024-02-05T06:54:04.513666+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:38.592868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:53:38.592868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:54.703288+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1031,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.515294+0000","last_change":"2024-02-05T06:54:03.548187+0000","last_active":"2024-02-05T06:54:04.515294+0000","last_peered":"2024-02-05T06:54:04.515294+0000","last_clean":"2024-02-05T06:54:04.515294+0000","last_became_active":"2024-02-05T06:51:54.326585+0000","last_became_peered":"2024-02-05T06:51:54.326585+0000","last_unstale":"2024-02-05T06:54:04.515294+0000","last_undegraded":"2024-02-05T06:54:04.515294+0000","last_fullsized":"2024-02-05T06:54:04.515294+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:54:03.548068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:54:03.548068+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:18.630524+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":502,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.504473+0000","last_change":"2024-02-05T06:51:40.969197+0000","last_active":"2024-02-05T06:54:04.504473+0000","last_peered":"2024-02-05T06:54:04.504473+0000","last_clean":"2024-02-05T06:54:04.504473+0000","last_became_active":"2024-02-05T06:41:43.408773+0000","last_became_peered":"2024-02-05T06:41:43.408773+0000","last_unstale":"2024-02-05T06:54:04.504473+0000","last_undegraded":"2024-02-05T06:54:04.504473+0000","last_fullsized":"2024-02-05T06:54:04.504473+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:40.969057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:51:40.969057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"471'12807","reported_seq":12924,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:07.968424+0000","last_change":"2024-02-05T06:51:13.274836+0000","last_active":"2024-02-05T06:54:07.968424+0000","last_peered":"2024-02-05T06:54:07.968424+0000","last_clean":"2024-02-05T06:54:07.968424+0000","last_became_active":"2024-02-05T06:51:13.272948+0000","last_became_peered":"2024-02-05T06:51:13.272948+0000","last_unstale":"2024-02-05T06:54:07.968424+0000","last_undegraded":"2024-02-05T06:54:07.968424+0000","last_fullsized":"2024-02-05T06:54:07.968424+0000","mapping_epoch":418,"log_start":"454'8500","ondisk_log_start":"454'8500","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:51:12.249449+0000","objects_scrubbed":0,"log_size":4307,"log_dups_size":0,"ondisk_log_size":4307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:12.545669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199426048,"num_objects":3043,"num_object_clones":0,"num_object_copies":9129,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3043,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12807,"num_write_kb":507200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":939,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.508117+0000","last_change":"2024-02-05T06:53:54.096200+0000","last_active":"2024-02-05T06:54:04.508117+0000","last_peered":"2024-02-05T06:54:04.508117+0000","last_clean":"2024-02-05T06:54:04.508117+0000","last_became_active":"2024-02-05T06:41:43.412387+0000","last_became_peered":"2024-02-05T06:41:43.412387+0000","last_unstale":"2024-02-05T06:54:04.508117+0000","last_undegraded":"2024-02-05T06:54:04.508117+0000","last_fullsized":"2024-02-05T06:54:04.508117+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:54.095702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:34.871805+0000","last_clean_scrub_stamp":"2024-02-05T06:53:54.095702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:14.492332+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"465'21","reported_seq":990,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.510820+0000","last_change":"2024-02-05T06:52:44.087732+0000","last_active":"2024-02-05T06:54:04.510820+0000","last_peered":"2024-02-05T06:54:04.510820+0000","last_clean":"2024-02-05T06:54:04.510820+0000","last_became_active":"2024-02-05T06:41:43.420458+0000","last_became_peered":"2024-02-05T06:41:43.420458+0000","last_unstale":"2024-02-05T06:54:04.510820+0000","last_undegraded":"2024-02-05T06:54:04.510820+0000","last_fullsized":"2024-02-05T06:54:04.510820+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'17","last_scrub_stamp":"2024-02-05T06:52:44.087598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:52:44.087598+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":924,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.506801+0000","last_change":"2024-02-05T06:53:44.055170+0000","last_active":"2024-02-05T06:54:04.506801+0000","last_peered":"2024-02-05T06:54:04.506801+0000","last_clean":"2024-02-05T06:54:04.506801+0000","last_became_active":"2024-02-05T06:33:28.542977+0000","last_became_peered":"2024-02-05T06:33:28.542977+0000","last_unstale":"2024-02-05T06:54:04.506801+0000","last_undegraded":"2024-02-05T06:54:04.506801+0000","last_fullsized":"2024-02-05T06:54:04.506801+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:44.054895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:53:44.054895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:11.925676+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.00091640799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":894,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.506125+0000","last_change":"2024-02-05T06:53:39.709665+0000","last_active":"2024-02-05T06:54:04.506125+0000","last_peered":"2024-02-05T06:54:04.506125+0000","last_clean":"2024-02-05T06:54:04.506125+0000","last_became_active":"2024-02-05T06:33:29.551549+0000","last_became_peered":"2024-02-05T06:33:29.551549+0000","last_unstale":"2024-02-05T06:54:04.506125+0000","last_undegraded":"2024-02-05T06:54:04.506125+0000","last_fullsized":"2024-02-05T06:54:04.506125+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:39.709556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:53:39.709556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:06.007029+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":594,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.516115+0000","last_change":"2024-02-05T06:53:41.146247+0000","last_active":"2024-02-05T06:54:04.516115+0000","last_peered":"2024-02-05T06:54:04.516115+0000","last_clean":"2024-02-05T06:54:04.516115+0000","last_became_active":"2024-02-05T06:40:27.077966+0000","last_became_peered":"2024-02-05T06:40:27.077966+0000","last_unstale":"2024-02-05T06:54:04.516115+0000","last_undegraded":"2024-02-05T06:54:04.516115+0000","last_fullsized":"2024-02-05T06:54:04.516115+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:41.146141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:45.774441+0000","last_clean_scrub_stamp":"2024-02-05T06:53:41.146141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:08.495898+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":1031,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.515577+0000","last_change":"2024-02-05T06:53:27.227987+0000","last_active":"2024-02-05T06:54:04.515577+0000","last_peered":"2024-02-05T06:54:04.515577+0000","last_clean":"2024-02-05T06:54:04.515577+0000","last_became_active":"2024-02-05T06:51:54.325427+0000","last_became_peered":"2024-02-05T06:51:54.325427+0000","last_unstale":"2024-02-05T06:54:04.515577+0000","last_undegraded":"2024-02-05T06:54:04.515577+0000","last_fullsized":"2024-02-05T06:54:04.515577+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:53:27.227870+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:48:28.657679+0000","last_clean_scrub_stamp":"2024-02-05T06:53:27.227870+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:27.245843+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"471'12704","reported_seq":12821,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:07.972541+0000","last_change":"2024-02-05T06:51:13.274482+0000","last_active":"2024-02-05T06:54:07.972541+0000","last_peered":"2024-02-05T06:54:07.972541+0000","last_clean":"2024-02-05T06:54:07.972541+0000","last_became_active":"2024-02-05T06:51:13.274182+0000","last_became_peered":"2024-02-05T06:51:13.274182+0000","last_unstale":"2024-02-05T06:54:07.972541+0000","last_undegraded":"2024-02-05T06:54:07.972541+0000","last_fullsized":"2024-02-05T06:54:07.972541+0000","mapping_epoch":418,"log_start":"453'8400","ondisk_log_start":"453'8400","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:51:12.249449+0000","objects_scrubbed":0,"log_size":4304,"log_dups_size":0,"ondisk_log_size":4304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:12.545547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203948032,"num_objects":3112,"num_object_clones":0,"num_object_copies":9336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12704,"num_write_kb":506112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":890,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.510320+0000","last_change":"2024-02-05T06:53:36.399616+0000","last_active":"2024-02-05T06:54:04.510320+0000","last_peered":"2024-02-05T06:54:04.510320+0000","last_clean":"2024-02-05T06:54:04.510320+0000","last_became_active":"2024-02-05T06:36:54.973387+0000","last_became_peered":"2024-02-05T06:36:54.973387+0000","last_unstale":"2024-02-05T06:54:04.510320+0000","last_undegraded":"2024-02-05T06:54:04.510320+0000","last_fullsized":"2024-02-05T06:54:04.510320+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:36.399508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:53:36.399508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:37.941444+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"465'64","reported_seq":843,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.504781+0000","last_change":"2024-02-05T06:53:36.797035+0000","last_active":"2024-02-05T06:54:04.504781+0000","last_peered":"2024-02-05T06:54:04.504781+0000","last_clean":"2024-02-05T06:54:04.504781+0000","last_became_active":"2024-02-05T06:40:08.604483+0000","last_became_peered":"2024-02-05T06:40:08.604483+0000","last_unstale":"2024-02-05T06:54:04.504781+0000","last_undegraded":"2024-02-05T06:54:04.504781+0000","last_fullsized":"2024-02-05T06:54:04.504781+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'52","last_scrub_stamp":"2024-02-05T06:53:36.796907+0000","last_deep_scrub":"326'52","last_deep_scrub_stamp":"2024-02-05T06:45:53.715482+0000","last_clean_scrub_stamp":"2024-02-05T06:53:36.796907+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:42.186110+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":415,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.509898+0000","last_change":"2024-02-05T06:52:47.127090+0000","last_active":"2024-02-05T06:54:04.509898+0000","last_peered":"2024-02-05T06:54:04.509898+0000","last_clean":"2024-02-05T06:54:04.509898+0000","last_became_active":"2024-02-05T06:45:09.776169+0000","last_became_peered":"2024-02-05T06:45:09.776169+0000","last_unstale":"2024-02-05T06:54:04.509898+0000","last_undegraded":"2024-02-05T06:54:04.509898+0000","last_fullsized":"2024-02-05T06:54:04.509898+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:47.126978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:47.126978+0000","last_clean_scrub_stamp":"2024-02-05T06:52:47.126978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"182'3","reported_seq":505,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.503804+0000","last_change":"2024-02-05T06:52:40.877231+0000","last_active":"2024-02-05T06:54:04.503804+0000","last_peered":"2024-02-05T06:54:04.503804+0000","last_clean":"2024-02-05T06:54:04.503804+0000","last_became_active":"2024-02-05T06:41:43.415806+0000","last_became_peered":"2024-02-05T06:41:43.415806+0000","last_unstale":"2024-02-05T06:54:04.503804+0000","last_undegraded":"2024-02-05T06:54:04.503804+0000","last_fullsized":"2024-02-05T06:54:04.503804+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'3","last_scrub_stamp":"2024-02-05T06:52:40.876912+0000","last_deep_scrub":"182'3","last_deep_scrub_stamp":"2024-02-05T06:39:33.373415+0000","last_clean_scrub_stamp":"2024-02-05T06:52:40.876912+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001220348,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":881,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.525962+0000","last_change":"2024-02-05T06:50:52.403321+0000","last_active":"2024-02-05T06:54:04.525962+0000","last_peered":"2024-02-05T06:54:04.525962+0000","last_clean":"2024-02-05T06:54:04.525962+0000","last_became_active":"2024-02-05T06:40:27.082440+0000","last_became_peered":"2024-02-05T06:40:27.082440+0000","last_unstale":"2024-02-05T06:54:04.525962+0000","last_undegraded":"2024-02-05T06:54:04.525962+0000","last_fullsized":"2024-02-05T06:54:04.525962+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:52.403179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:50:52.403179+0000","last_clean_scrub_stamp":"2024-02-05T06:50:52.403179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":926,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.503414+0000","last_change":"2024-02-05T06:52:17.849188+0000","last_active":"2024-02-05T06:54:04.503414+0000","last_peered":"2024-02-05T06:54:04.503414+0000","last_clean":"2024-02-05T06:54:04.503414+0000","last_became_active":"2024-02-05T06:51:54.325737+0000","last_became_peered":"2024-02-05T06:51:54.325737+0000","last_unstale":"2024-02-05T06:54:04.503414+0000","last_undegraded":"2024-02-05T06:54:04.503414+0000","last_fullsized":"2024-02-05T06:54:04.503414+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:52:17.849056+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:37.282442+0000","last_clean_scrub_stamp":"2024-02-05T06:52:17.849056+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"83'128","reported_seq":393,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.516108+0000","last_change":"2024-02-05T06:52:21.118216+0000","last_active":"2024-02-05T06:54:04.516108+0000","last_peered":"2024-02-05T06:54:04.516108+0000","last_clean":"2024-02-05T06:54:04.516108+0000","last_became_active":"2024-02-05T06:51:59.294208+0000","last_became_peered":"2024-02-05T06:51:59.294208+0000","last_unstale":"2024-02-05T06:54:04.516108+0000","last_undegraded":"2024-02-05T06:54:04.516108+0000","last_fullsized":"2024-02-05T06:54:04.516108+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:52:21.118090+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:38.778568+0000","last_clean_scrub_stamp":"2024-02-05T06:52:21.118090+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.4","version":"471'12508","reported_seq":12685,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:06.844838+0000","last_change":"2024-02-05T06:53:33.493384+0000","last_active":"2024-02-05T06:54:06.844838+0000","last_peered":"2024-02-05T06:54:06.844838+0000","last_clean":"2024-02-05T06:54:06.844838+0000","last_became_active":"2024-02-05T06:51:13.272748+0000","last_became_peered":"2024-02-05T06:51:13.272748+0000","last_unstale":"2024-02-05T06:54:06.844838+0000","last_undegraded":"2024-02-05T06:54:06.844838+0000","last_fullsized":"2024-02-05T06:54:06.844838+0000","mapping_epoch":418,"log_start":"450'7900","ondisk_log_start":"450'7900","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'10373","last_scrub_stamp":"2024-02-05T06:53:33.492667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:53:33.492667+0000","objects_scrubbed":5303,"log_size":4608,"log_dups_size":0,"ondisk_log_size":4608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:39.923205+0000","scrub_duration":1404,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204734464,"num_objects":3124,"num_object_clones":0,"num_object_copies":9372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12508,"num_write_kb":500224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"465'21","reported_seq":996,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.509733+0000","last_change":"2024-02-05T06:51:42.417785+0000","last_active":"2024-02-05T06:54:04.509733+0000","last_peered":"2024-02-05T06:54:04.509733+0000","last_clean":"2024-02-05T06:54:04.509733+0000","last_became_active":"2024-02-05T06:40:08.584204+0000","last_became_peered":"2024-02-05T06:40:08.584204+0000","last_unstale":"2024-02-05T06:54:04.509733+0000","last_undegraded":"2024-02-05T06:54:04.509733+0000","last_fullsized":"2024-02-05T06:54:04.509733+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:51:42.417532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:51:42.417532+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":501,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.503654+0000","last_change":"2024-02-05T06:52:32.883872+0000","last_active":"2024-02-05T06:54:04.503654+0000","last_peered":"2024-02-05T06:54:04.503654+0000","last_clean":"2024-02-05T06:54:04.503654+0000","last_became_active":"2024-02-05T06:41:43.412611+0000","last_became_peered":"2024-02-05T06:41:43.412611+0000","last_unstale":"2024-02-05T06:54:04.503654+0000","last_undegraded":"2024-02-05T06:54:04.503654+0000","last_fullsized":"2024-02-05T06:54:04.503654+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:32.883756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:41.617526+0000","last_clean_scrub_stamp":"2024-02-05T06:52:32.883756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":919,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.507684+0000","last_change":"2024-02-05T06:52:48.076179+0000","last_active":"2024-02-05T06:54:04.507684+0000","last_peered":"2024-02-05T06:54:04.507684+0000","last_clean":"2024-02-05T06:54:04.507684+0000","last_became_active":"2024-02-05T06:36:54.966016+0000","last_became_peered":"2024-02-05T06:36:54.966016+0000","last_unstale":"2024-02-05T06:54:04.507684+0000","last_undegraded":"2024-02-05T06:54:04.507684+0000","last_fullsized":"2024-02-05T06:54:04.507684+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:48.076059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:52:48.076059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:13.576887+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":886,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.513522+0000","last_change":"2024-02-05T06:53:50.717387+0000","last_active":"2024-02-05T06:54:04.513522+0000","last_peered":"2024-02-05T06:54:04.513522+0000","last_clean":"2024-02-05T06:54:04.513522+0000","last_became_active":"2024-02-05T06:40:27.129625+0000","last_became_peered":"2024-02-05T06:40:27.129625+0000","last_unstale":"2024-02-05T06:54:04.513522+0000","last_undegraded":"2024-02-05T06:54:04.513522+0000","last_fullsized":"2024-02-05T06:54:04.513522+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:50.717250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:53:50.717250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:08.463818+0000","scrub_duration":214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"83'128","reported_seq":1103,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.510814+0000","last_change":"2024-02-05T06:52:21.609713+0000","last_active":"2024-02-05T06:54:04.510814+0000","last_peered":"2024-02-05T06:54:04.510814+0000","last_clean":"2024-02-05T06:54:04.510814+0000","last_became_active":"2024-02-05T06:51:59.293451+0000","last_became_peered":"2024-02-05T06:51:59.293451+0000","last_unstale":"2024-02-05T06:54:04.510814+0000","last_undegraded":"2024-02-05T06:54:04.510814+0000","last_fullsized":"2024-02-05T06:54:04.510814+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:52:21.609593+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:48:25.725892+0000","last_clean_scrub_stamp":"2024-02-05T06:52:21.609593+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.5","version":"471'12736","reported_seq":12939,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:06.841713+0000","last_change":"2024-02-05T06:52:50.694891+0000","last_active":"2024-02-05T06:54:06.841713+0000","last_peered":"2024-02-05T06:54:06.841713+0000","last_clean":"2024-02-05T06:54:06.841713+0000","last_became_active":"2024-02-05T06:51:13.270923+0000","last_became_peered":"2024-02-05T06:51:13.270923+0000","last_unstale":"2024-02-05T06:54:06.841713+0000","last_undegraded":"2024-02-05T06:54:06.841713+0000","last_fullsized":"2024-02-05T06:54:06.841713+0000","mapping_epoch":418,"log_start":"450'8100","ondisk_log_start":"450'8100","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'7908","last_scrub_stamp":"2024-02-05T06:52:50.694691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:52:50.694691+0000","objects_scrubbed":7859,"log_size":4636,"log_dups_size":0,"ondisk_log_size":4636,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206700544,"num_objects":3154,"num_object_clones":0,"num_object_copies":9462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12736,"num_write_kb":508480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"465'60","reported_seq":1070,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.511821+0000","last_change":"2024-02-05T06:52:51.676140+0000","last_active":"2024-02-05T06:54:04.511821+0000","last_peered":"2024-02-05T06:54:04.511821+0000","last_clean":"2024-02-05T06:54:04.511821+0000","last_became_active":"2024-02-05T06:40:27.068417+0000","last_became_peered":"2024-02-05T06:40:27.068417+0000","last_unstale":"2024-02-05T06:54:04.511821+0000","last_undegraded":"2024-02-05T06:54:04.511821+0000","last_fullsized":"2024-02-05T06:54:04.511821+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'44","last_scrub_stamp":"2024-02-05T06:52:51.676057+0000","last_deep_scrub":"279'28","last_deep_scrub_stamp":"2024-02-05T06:42:50.148881+0000","last_clean_scrub_stamp":"2024-02-05T06:52:51.676057+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":600,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.516358+0000","last_change":"2024-02-05T06:54:01.596848+0000","last_active":"2024-02-05T06:54:04.516358+0000","last_peered":"2024-02-05T06:54:04.516358+0000","last_clean":"2024-02-05T06:54:04.516358+0000","last_became_active":"2024-02-05T06:41:43.418202+0000","last_became_peered":"2024-02-05T06:41:43.418202+0000","last_unstale":"2024-02-05T06:54:04.516358+0000","last_undegraded":"2024-02-05T06:54:04.516358+0000","last_fullsized":"2024-02-05T06:54:04.516358+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:54:01.596730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:54:01.596730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:01.961347+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":925,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.509972+0000","last_change":"2024-02-05T06:53:52.054596+0000","last_active":"2024-02-05T06:54:04.509972+0000","last_peered":"2024-02-05T06:54:04.509972+0000","last_clean":"2024-02-05T06:54:04.509972+0000","last_became_active":"2024-02-05T06:41:43.413095+0000","last_became_peered":"2024-02-05T06:41:43.413095+0000","last_unstale":"2024-02-05T06:54:04.509972+0000","last_undegraded":"2024-02-05T06:54:04.509972+0000","last_fullsized":"2024-02-05T06:54:04.509972+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:52.054463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:45.073681+0000","last_clean_scrub_stamp":"2024-02-05T06:53:52.054463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:21.447521+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":594,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.515127+0000","last_change":"2024-02-05T06:52:47.116225+0000","last_active":"2024-02-05T06:54:04.515127+0000","last_peered":"2024-02-05T06:54:04.515127+0000","last_clean":"2024-02-05T06:54:04.515127+0000","last_became_active":"2024-02-05T06:40:27.218081+0000","last_became_peered":"2024-02-05T06:40:27.218081+0000","last_unstale":"2024-02-05T06:54:04.515127+0000","last_undegraded":"2024-02-05T06:54:04.515127+0000","last_fullsized":"2024-02-05T06:54:04.515127+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:47.116118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:32.925000+0000","last_clean_scrub_stamp":"2024-02-05T06:52:47.116118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"83'128","reported_seq":1104,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.523848+0000","last_change":"2024-02-05T06:52:21.388903+0000","last_active":"2024-02-05T06:54:04.523848+0000","last_peered":"2024-02-05T06:54:04.523848+0000","last_clean":"2024-02-05T06:54:04.523848+0000","last_became_active":"2024-02-05T06:51:59.296988+0000","last_became_peered":"2024-02-05T06:51:59.296988+0000","last_unstale":"2024-02-05T06:54:04.523848+0000","last_undegraded":"2024-02-05T06:54:04.523848+0000","last_fullsized":"2024-02-05T06:54:04.523848+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:52:21.388783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:52:21.388783+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"471'12781","reported_seq":12950,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:06.520501+0000","last_change":"2024-02-05T06:53:46.461956+0000","last_active":"2024-02-05T06:54:06.520501+0000","last_peered":"2024-02-05T06:54:06.520501+0000","last_clean":"2024-02-05T06:54:06.520501+0000","last_became_active":"2024-02-05T06:51:13.267525+0000","last_became_peered":"2024-02-05T06:51:13.267525+0000","last_unstale":"2024-02-05T06:54:06.520501+0000","last_undegraded":"2024-02-05T06:54:06.520501+0000","last_fullsized":"2024-02-05T06:54:06.520501+0000","mapping_epoch":418,"log_start":"447'7600","ondisk_log_start":"447'7600","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"466'11503","last_scrub_stamp":"2024-02-05T06:53:46.461835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:53:46.461835+0000","objects_scrubbed":4404,"log_size":5181,"log_dups_size":0,"ondisk_log_size":5181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:58.262026+0000","scrub_duration":1947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200212480,"num_objects":3055,"num_object_clones":0,"num_object_copies":9165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12781,"num_write_kb":506752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":899,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.506369+0000","last_change":"2024-02-05T06:52:46.053806+0000","last_active":"2024-02-05T06:54:04.506369+0000","last_peered":"2024-02-05T06:54:04.506369+0000","last_clean":"2024-02-05T06:54:04.506369+0000","last_became_active":"2024-02-05T06:33:33.592546+0000","last_became_peered":"2024-02-05T06:33:33.592546+0000","last_unstale":"2024-02-05T06:54:04.506369+0000","last_undegraded":"2024-02-05T06:54:04.506369+0000","last_fullsized":"2024-02-05T06:54:04.506369+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:46.053684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:42.685240+0000","last_clean_scrub_stamp":"2024-02-05T06:52:46.053684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"465'24","reported_seq":754,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.503590+0000","last_change":"2024-02-05T06:52:26.996274+0000","last_active":"2024-02-05T06:54:04.503590+0000","last_peered":"2024-02-05T06:54:04.503590+0000","last_clean":"2024-02-05T06:54:04.503590+0000","last_became_active":"2024-02-05T06:45:09.783036+0000","last_became_peered":"2024-02-05T06:45:09.783036+0000","last_unstale":"2024-02-05T06:54:04.503590+0000","last_undegraded":"2024-02-05T06:54:04.503590+0000","last_fullsized":"2024-02-05T06:54:04.503590+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'18","last_scrub_stamp":"2024-02-05T06:52:26.996136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:52:26.996136+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":29,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":594,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.516512+0000","last_change":"2024-02-05T06:53:36.100637+0000","last_active":"2024-02-05T06:54:04.516512+0000","last_peered":"2024-02-05T06:54:04.516512+0000","last_clean":"2024-02-05T06:54:04.516512+0000","last_became_active":"2024-02-05T06:40:27.063300+0000","last_became_peered":"2024-02-05T06:40:27.063300+0000","last_unstale":"2024-02-05T06:54:04.516512+0000","last_undegraded":"2024-02-05T06:54:04.516512+0000","last_fullsized":"2024-02-05T06:54:04.516512+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:36.100400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:34.977997+0000","last_clean_scrub_stamp":"2024-02-05T06:53:36.100400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:45.258122+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":869,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.527392+0000","last_change":"2024-02-05T06:51:22.703991+0000","last_active":"2024-02-05T06:54:04.527392+0000","last_peered":"2024-02-05T06:54:04.527392+0000","last_clean":"2024-02-05T06:54:04.527392+0000","last_became_active":"2024-02-05T06:40:27.092613+0000","last_became_peered":"2024-02-05T06:40:27.092613+0000","last_unstale":"2024-02-05T06:54:04.527392+0000","last_undegraded":"2024-02-05T06:54:04.527392+0000","last_fullsized":"2024-02-05T06:54:04.527392+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:22.703865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:51:22.703865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":190,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"83'128","reported_seq":661,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.515906+0000","last_change":"2024-02-05T06:52:24.061520+0000","last_active":"2024-02-05T06:54:04.515906+0000","last_peered":"2024-02-05T06:54:04.515906+0000","last_clean":"2024-02-05T06:54:04.515906+0000","last_became_active":"2024-02-05T06:51:59.294191+0000","last_became_peered":"2024-02-05T06:51:59.294191+0000","last_unstale":"2024-02-05T06:54:04.515906+0000","last_undegraded":"2024-02-05T06:54:04.515906+0000","last_fullsized":"2024-02-05T06:54:04.515906+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:52:24.061383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:52:24.061383+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.7","version":"471'16128","reported_seq":16367,"reported_epoch":471,"state":"active+clean+scrubbing+deep","last_fresh":"2024-02-05T06:54:06.984055+0000","last_change":"2024-02-05T06:53:48.774956+0000","last_active":"2024-02-05T06:54:06.984055+0000","last_peered":"2024-02-05T06:54:06.984055+0000","last_clean":"2024-02-05T06:54:06.984055+0000","last_became_active":"2024-02-05T06:51:13.272066+0000","last_became_peered":"2024-02-05T06:51:13.272066+0000","last_unstale":"2024-02-05T06:54:06.984055+0000","last_undegraded":"2024-02-05T06:54:06.984055+0000","last_fullsized":"2024-02-05T06:54:06.984055+0000","mapping_epoch":418,"log_start":"465'11101","ondisk_log_start":"465'11101","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:51:12.249449+0000","objects_scrubbed":3374,"log_size":5027,"log_dups_size":0,"ondisk_log_size":5027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 18s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203685888,"num_objects":3108,"num_object_clones":0,"num_object_copies":9324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3108,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":507584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"208'12","reported_seq":762,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.516816+0000","last_change":"2024-02-05T06:52:33.103331+0000","last_active":"2024-02-05T06:54:04.516816+0000","last_peered":"2024-02-05T06:54:04.516816+0000","last_clean":"2024-02-05T06:54:04.516816+0000","last_became_active":"2024-02-05T06:40:27.064195+0000","last_became_peered":"2024-02-05T06:40:27.064195+0000","last_unstale":"2024-02-05T06:54:04.516816+0000","last_undegraded":"2024-02-05T06:54:04.516816+0000","last_fullsized":"2024-02-05T06:54:04.516816+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'12","last_scrub_stamp":"2024-02-05T06:52:33.103188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:52:33.103188+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"465'39","reported_seq":553,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.503276+0000","last_change":"2024-02-05T06:53:36.831417+0000","last_active":"2024-02-05T06:54:04.503276+0000","last_peered":"2024-02-05T06:54:04.503276+0000","last_clean":"2024-02-05T06:54:04.503276+0000","last_became_active":"2024-02-05T06:41:43.412579+0000","last_became_peered":"2024-02-05T06:41:43.412579+0000","last_unstale":"2024-02-05T06:54:04.503276+0000","last_undegraded":"2024-02-05T06:54:04.503276+0000","last_fullsized":"2024-02-05T06:54:04.503276+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'27","last_scrub_stamp":"2024-02-05T06:53:36.831295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:53:36.831295+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:03.891726+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"412'4","reported_seq":602,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.516923+0000","last_change":"2024-02-05T06:52:39.111257+0000","last_active":"2024-02-05T06:54:04.516923+0000","last_peered":"2024-02-05T06:54:04.516923+0000","last_clean":"2024-02-05T06:54:04.516923+0000","last_became_active":"2024-02-05T06:40:27.068279+0000","last_became_peered":"2024-02-05T06:40:27.068279+0000","last_unstale":"2024-02-05T06:54:04.516923+0000","last_undegraded":"2024-02-05T06:54:04.516923+0000","last_fullsized":"2024-02-05T06:54:04.516923+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'4","last_scrub_stamp":"2024-02-05T06:52:39.111151+0000","last_deep_scrub":"412'4","last_deep_scrub_stamp":"2024-02-05T06:50:51.841825+0000","last_clean_scrub_stamp":"2024-02-05T06:52:39.111151+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:08.393004+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":501,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.503800+0000","last_change":"2024-02-05T06:52:26.888326+0000","last_active":"2024-02-05T06:54:04.503800+0000","last_peered":"2024-02-05T06:54:04.503800+0000","last_clean":"2024-02-05T06:54:04.503800+0000","last_became_active":"2024-02-05T06:41:43.421231+0000","last_became_peered":"2024-02-05T06:41:43.421231+0000","last_unstale":"2024-02-05T06:54:04.503800+0000","last_undegraded":"2024-02-05T06:54:04.503800+0000","last_fullsized":"2024-02-05T06:54:04.503800+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:26.888098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:52:26.888098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"83'128","reported_seq":528,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.503557+0000","last_change":"2024-02-05T06:52:21.904768+0000","last_active":"2024-02-05T06:54:04.503557+0000","last_peered":"2024-02-05T06:54:04.503557+0000","last_clean":"2024-02-05T06:54:04.503557+0000","last_became_active":"2024-02-05T06:51:59.296150+0000","last_became_peered":"2024-02-05T06:51:59.296150+0000","last_unstale":"2024-02-05T06:54:04.503557+0000","last_undegraded":"2024-02-05T06:54:04.503557+0000","last_fullsized":"2024-02-05T06:54:04.503557+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:52:21.904642+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:48:28.657679+0000","last_clean_scrub_stamp":"2024-02-05T06:52:21.904642+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"465'21","reported_seq":930,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.526573+0000","last_change":"2024-02-05T06:53:51.658817+0000","last_active":"2024-02-05T06:54:04.526573+0000","last_peered":"2024-02-05T06:54:04.526573+0000","last_clean":"2024-02-05T06:54:04.526573+0000","last_became_active":"2024-02-05T06:40:27.091689+0000","last_became_peered":"2024-02-05T06:40:27.091689+0000","last_unstale":"2024-02-05T06:54:04.526573+0000","last_undegraded":"2024-02-05T06:54:04.526573+0000","last_fullsized":"2024-02-05T06:54:04.526573+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'21","last_scrub_stamp":"2024-02-05T06:53:51.658691+0000","last_deep_scrub":"412'15","last_deep_scrub_stamp":"2024-02-05T06:50:55.397689+0000","last_clean_scrub_stamp":"2024-02-05T06:53:51.658691+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:01.285256+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":919,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.510326+0000","last_change":"2024-02-05T06:53:55.061956+0000","last_active":"2024-02-05T06:54:04.510326+0000","last_peered":"2024-02-05T06:54:04.510326+0000","last_clean":"2024-02-05T06:54:04.510326+0000","last_became_active":"2024-02-05T06:41:43.412130+0000","last_became_peered":"2024-02-05T06:41:43.412130+0000","last_unstale":"2024-02-05T06:54:04.510326+0000","last_undegraded":"2024-02-05T06:54:04.510326+0000","last_fullsized":"2024-02-05T06:54:04.510326+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:55.061848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:53:55.061848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:24.423601+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":501,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.504047+0000","last_change":"2024-02-05T06:52:24.934323+0000","last_active":"2024-02-05T06:54:04.504047+0000","last_peered":"2024-02-05T06:54:04.504047+0000","last_clean":"2024-02-05T06:54:04.504047+0000","last_became_active":"2024-02-05T06:41:43.406094+0000","last_became_peered":"2024-02-05T06:41:43.406094+0000","last_unstale":"2024-02-05T06:54:04.504047+0000","last_undegraded":"2024-02-05T06:54:04.504047+0000","last_fullsized":"2024-02-05T06:54:04.504047+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:24.934198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:44:57.506909+0000","last_clean_scrub_stamp":"2024-02-05T06:52:24.934198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":593,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.516486+0000","last_change":"2024-02-05T06:53:54.796403+0000","last_active":"2024-02-05T06:54:04.516486+0000","last_peered":"2024-02-05T06:54:04.516486+0000","last_clean":"2024-02-05T06:54:04.516486+0000","last_became_active":"2024-02-05T06:40:27.063350+0000","last_became_peered":"2024-02-05T06:40:27.063350+0000","last_unstale":"2024-02-05T06:54:04.516486+0000","last_undegraded":"2024-02-05T06:54:04.516486+0000","last_fullsized":"2024-02-05T06:54:04.516486+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:54.796297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:53:54.796297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:17.620285+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"83'128","reported_seq":1071,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.503593+0000","last_change":"2024-02-05T06:54:02.691383+0000","last_active":"2024-02-05T06:54:04.503593+0000","last_peered":"2024-02-05T06:54:04.503593+0000","last_clean":"2024-02-05T06:54:04.503593+0000","last_became_active":"2024-02-05T06:51:59.303621+0000","last_became_peered":"2024-02-05T06:51:59.303621+0000","last_unstale":"2024-02-05T06:54:04.503593+0000","last_undegraded":"2024-02-05T06:54:04.503593+0000","last_fullsized":"2024-02-05T06:54:04.503593+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:54:02.691263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:54:02.691263+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:17.412431+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"465'21","reported_seq":955,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.509733+0000","last_change":"2024-02-05T06:53:37.421718+0000","last_active":"2024-02-05T06:54:04.509733+0000","last_peered":"2024-02-05T06:54:04.509733+0000","last_clean":"2024-02-05T06:54:04.509733+0000","last_became_active":"2024-02-05T06:40:27.101116+0000","last_became_peered":"2024-02-05T06:40:27.101116+0000","last_unstale":"2024-02-05T06:54:04.509733+0000","last_undegraded":"2024-02-05T06:54:04.509733+0000","last_fullsized":"2024-02-05T06:54:04.509733+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'19","last_scrub_stamp":"2024-02-05T06:53:37.421591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:53:37.421591+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:56.525787+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":593,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.516857+0000","last_change":"2024-02-05T06:53:44.983388+0000","last_active":"2024-02-05T06:54:04.516857+0000","last_peered":"2024-02-05T06:54:04.516857+0000","last_clean":"2024-02-05T06:54:04.516857+0000","last_became_active":"2024-02-05T06:40:27.064388+0000","last_became_peered":"2024-02-05T06:40:27.064388+0000","last_unstale":"2024-02-05T06:54:04.516857+0000","last_undegraded":"2024-02-05T06:54:04.516857+0000","last_fullsized":"2024-02-05T06:54:04.516857+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:44.983315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:53:44.983315+0000","last_clean_scrub_stamp":"2024-02-05T06:53:44.983315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:50.922462+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":886,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.526409+0000","last_change":"2024-02-05T06:52:26.443142+0000","last_active":"2024-02-05T06:54:04.526409+0000","last_peered":"2024-02-05T06:54:04.526409+0000","last_clean":"2024-02-05T06:54:04.526409+0000","last_became_active":"2024-02-05T06:40:27.091544+0000","last_became_peered":"2024-02-05T06:40:27.091544+0000","last_unstale":"2024-02-05T06:54:04.526409+0000","last_undegraded":"2024-02-05T06:54:04.526409+0000","last_fullsized":"2024-02-05T06:54:04.526409+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:26.443024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:52:26.443024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":926,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.511795+0000","last_change":"2024-02-05T06:52:45.010159+0000","last_active":"2024-02-05T06:54:04.511795+0000","last_peered":"2024-02-05T06:54:04.511795+0000","last_clean":"2024-02-05T06:54:04.511795+0000","last_became_active":"2024-02-05T06:40:27.090632+0000","last_became_peered":"2024-02-05T06:40:27.090632+0000","last_unstale":"2024-02-05T06:54:04.511795+0000","last_undegraded":"2024-02-05T06:54:04.511795+0000","last_fullsized":"2024-02-05T06:54:04.511795+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:45.010039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:45.010039+0000","last_clean_scrub_stamp":"2024-02-05T06:52:45.010039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:13.806168+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"83'128","reported_seq":528,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.503070+0000","last_change":"2024-02-05T06:52:20.872578+0000","last_active":"2024-02-05T06:54:04.503070+0000","last_peered":"2024-02-05T06:54:04.503070+0000","last_clean":"2024-02-05T06:54:04.503070+0000","last_became_active":"2024-02-05T06:51:59.290466+0000","last_became_peered":"2024-02-05T06:51:59.290466+0000","last_unstale":"2024-02-05T06:54:04.503070+0000","last_undegraded":"2024-02-05T06:54:04.503070+0000","last_fullsized":"2024-02-05T06:54:04.503070+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:52:20.872451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:52:20.872451+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"465'21","reported_seq":529,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.504688+0000","last_change":"2024-02-05T06:52:31.905022+0000","last_active":"2024-02-05T06:54:04.504688+0000","last_peered":"2024-02-05T06:54:04.504688+0000","last_clean":"2024-02-05T06:54:04.504688+0000","last_became_active":"2024-02-05T06:41:43.413708+0000","last_became_peered":"2024-02-05T06:41:43.413708+0000","last_unstale":"2024-02-05T06:54:04.504688+0000","last_undegraded":"2024-02-05T06:54:04.504688+0000","last_fullsized":"2024-02-05T06:54:04.504688+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:52:31.904813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:52:31.904813+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":905,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.509578+0000","last_change":"2024-02-05T06:53:43.562505+0000","last_active":"2024-02-05T06:54:04.509578+0000","last_peered":"2024-02-05T06:54:04.509578+0000","last_clean":"2024-02-05T06:54:04.509578+0000","last_became_active":"2024-02-05T06:40:27.088096+0000","last_became_peered":"2024-02-05T06:40:27.088096+0000","last_unstale":"2024-02-05T06:54:04.509578+0000","last_undegraded":"2024-02-05T06:54:04.509578+0000","last_fullsized":"2024-02-05T06:54:04.509578+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:43.562266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:53:43.562266+0000","last_clean_scrub_stamp":"2024-02-05T06:53:43.562266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:09.519621+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"208'10","reported_seq":1144,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.524639+0000","last_change":"2024-02-05T06:52:43.367615+0000","last_active":"2024-02-05T06:54:04.524639+0000","last_peered":"2024-02-05T06:54:04.524639+0000","last_clean":"2024-02-05T06:54:04.524639+0000","last_became_active":"2024-02-05T06:40:27.096584+0000","last_became_peered":"2024-02-05T06:40:27.096584+0000","last_unstale":"2024-02-05T06:54:04.524639+0000","last_undegraded":"2024-02-05T06:54:04.524639+0000","last_fullsized":"2024-02-05T06:54:04.524639+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-02-05T06:52:43.367496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:52:43.367496+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":641,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.508453+0000","last_change":"2024-02-05T06:52:44.242695+0000","last_active":"2024-02-05T06:54:04.508453+0000","last_peered":"2024-02-05T06:54:04.508453+0000","last_clean":"2024-02-05T06:54:04.508453+0000","last_became_active":"2024-02-05T06:45:09.775152+0000","last_became_peered":"2024-02-05T06:45:09.775152+0000","last_unstale":"2024-02-05T06:54:04.508453+0000","last_undegraded":"2024-02-05T06:54:04.508453+0000","last_fullsized":"2024-02-05T06:54:04.508453+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:44.242462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:41:38.559780+0000","last_clean_scrub_stamp":"2024-02-05T06:52:44.242462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:10.251365+0000","scrub_duration":228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"83'128","reported_seq":528,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.510594+0000","last_change":"2024-02-05T06:52:22.246184+0000","last_active":"2024-02-05T06:54:04.510594+0000","last_peered":"2024-02-05T06:54:04.510594+0000","last_clean":"2024-02-05T06:54:04.510594+0000","last_became_active":"2024-02-05T06:51:59.300112+0000","last_became_peered":"2024-02-05T06:51:59.300112+0000","last_unstale":"2024-02-05T06:54:04.510594+0000","last_undegraded":"2024-02-05T06:54:04.510594+0000","last_fullsized":"2024-02-05T06:54:04.510594+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:52:22.246060+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:37.282442+0000","last_clean_scrub_stamp":"2024-02-05T06:52:22.246060+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"465'30","reported_seq":971,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.524181+0000","last_change":"2024-02-05T06:53:26.322082+0000","last_active":"2024-02-05T06:54:04.524181+0000","last_peered":"2024-02-05T06:54:04.524181+0000","last_clean":"2024-02-05T06:54:04.524181+0000","last_became_active":"2024-02-05T06:40:27.093418+0000","last_became_peered":"2024-02-05T06:40:27.093418+0000","last_unstale":"2024-02-05T06:54:04.524181+0000","last_undegraded":"2024-02-05T06:54:04.524181+0000","last_fullsized":"2024-02-05T06:54:04.524181+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'20","last_scrub_stamp":"2024-02-05T06:53:26.321955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:53:26.321955+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:43.931325+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":594,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.516067+0000","last_change":"2024-02-05T06:53:43.055431+0000","last_active":"2024-02-05T06:54:04.516067+0000","last_peered":"2024-02-05T06:54:04.516067+0000","last_clean":"2024-02-05T06:54:04.516067+0000","last_became_active":"2024-02-05T06:40:27.098418+0000","last_became_peered":"2024-02-05T06:40:27.098418+0000","last_unstale":"2024-02-05T06:54:04.516067+0000","last_undegraded":"2024-02-05T06:54:04.516067+0000","last_fullsized":"2024-02-05T06:54:04.516067+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:43.055316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:53:43.055316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:12.814470+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"336'11","reported_seq":655,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.504555+0000","last_change":"2024-02-05T06:52:18.890279+0000","last_active":"2024-02-05T06:54:04.504555+0000","last_peered":"2024-02-05T06:54:04.504555+0000","last_clean":"2024-02-05T06:54:04.504555+0000","last_became_active":"2024-02-05T06:41:43.418753+0000","last_became_peered":"2024-02-05T06:41:43.418753+0000","last_unstale":"2024-02-05T06:54:04.504555+0000","last_undegraded":"2024-02-05T06:54:04.504555+0000","last_fullsized":"2024-02-05T06:54:04.504555+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'11","last_scrub_stamp":"2024-02-05T06:52:18.890167+0000","last_deep_scrub":"336'11","last_deep_scrub_stamp":"2024-02-05T06:46:55.721412+0000","last_clean_scrub_stamp":"2024-02-05T06:52:18.890167+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":877,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.509406+0000","last_change":"2024-02-05T06:52:30.384032+0000","last_active":"2024-02-05T06:54:04.509406+0000","last_peered":"2024-02-05T06:54:04.509406+0000","last_clean":"2024-02-05T06:54:04.509406+0000","last_became_active":"2024-02-05T06:40:27.096132+0000","last_became_peered":"2024-02-05T06:40:27.096132+0000","last_unstale":"2024-02-05T06:54:04.509406+0000","last_undegraded":"2024-02-05T06:54:04.509406+0000","last_fullsized":"2024-02-05T06:54:04.509406+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:30.383899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:52:30.383899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"465'32","reported_seq":766,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.512035+0000","last_change":"2024-02-05T06:53:32.977787+0000","last_active":"2024-02-05T06:54:04.512035+0000","last_peered":"2024-02-05T06:54:04.512035+0000","last_clean":"2024-02-05T06:54:04.512035+0000","last_became_active":"2024-02-05T06:45:09.784411+0000","last_became_peered":"2024-02-05T06:45:09.784411+0000","last_unstale":"2024-02-05T06:54:04.512035+0000","last_undegraded":"2024-02-05T06:54:04.512035+0000","last_fullsized":"2024-02-05T06:54:04.512035+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:53:32.977666+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:35:02.589229+0000","last_clean_scrub_stamp":"2024-02-05T06:53:32.977666+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:37.861865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":539,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.517138+0000","last_change":"2024-02-05T06:52:44.163784+0000","last_active":"2024-02-05T06:54:04.517138+0000","last_peered":"2024-02-05T06:54:04.517138+0000","last_clean":"2024-02-05T06:54:04.517138+0000","last_became_active":"2024-02-05T06:45:09.773490+0000","last_became_peered":"2024-02-05T06:45:09.773490+0000","last_unstale":"2024-02-05T06:54:04.517138+0000","last_undegraded":"2024-02-05T06:54:04.517138+0000","last_fullsized":"2024-02-05T06:54:04.517138+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:44.163661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:52:44.163661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:07.570342+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":910,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.504650+0000","last_change":"2024-02-05T06:53:53.666791+0000","last_active":"2024-02-05T06:54:04.504650+0000","last_peered":"2024-02-05T06:54:04.504650+0000","last_clean":"2024-02-05T06:54:04.504650+0000","last_became_active":"2024-02-05T06:41:43.421433+0000","last_became_peered":"2024-02-05T06:41:43.421433+0000","last_unstale":"2024-02-05T06:54:04.504650+0000","last_undegraded":"2024-02-05T06:54:04.504650+0000","last_fullsized":"2024-02-05T06:54:04.504650+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:53.666667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:53:53.666667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:17.847146+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":876,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.518330+0000","last_change":"2024-02-05T06:53:31.491982+0000","last_active":"2024-02-05T06:54:04.518330+0000","last_peered":"2024-02-05T06:54:04.518330+0000","last_clean":"2024-02-05T06:54:04.518330+0000","last_became_active":"2024-02-05T06:41:43.417155+0000","last_became_peered":"2024-02-05T06:41:43.417155+0000","last_unstale":"2024-02-05T06:54:04.518330+0000","last_undegraded":"2024-02-05T06:54:04.518330+0000","last_fullsized":"2024-02-05T06:54:04.518330+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:31.491724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:53:31.491724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:00.042636+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"465'21","reported_seq":679,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.509272+0000","last_change":"2024-02-05T06:52:45.194776+0000","last_active":"2024-02-05T06:54:04.509272+0000","last_peered":"2024-02-05T06:54:04.509272+0000","last_clean":"2024-02-05T06:54:04.509272+0000","last_became_active":"2024-02-05T06:39:13.244124+0000","last_became_peered":"2024-02-05T06:39:13.244124+0000","last_unstale":"2024-02-05T06:54:04.509272+0000","last_undegraded":"2024-02-05T06:54:04.509272+0000","last_fullsized":"2024-02-05T06:54:04.509272+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:52:45.194645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:52:45.194645+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":593,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.516196+0000","last_change":"2024-02-05T06:53:44.009695+0000","last_active":"2024-02-05T06:54:04.516196+0000","last_peered":"2024-02-05T06:54:04.516196+0000","last_clean":"2024-02-05T06:54:04.516196+0000","last_became_active":"2024-02-05T06:40:27.077701+0000","last_became_peered":"2024-02-05T06:40:27.077701+0000","last_unstale":"2024-02-05T06:54:04.516196+0000","last_undegraded":"2024-02-05T06:54:04.516196+0000","last_fullsized":"2024-02-05T06:54:04.516196+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:44.009607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:33.978941+0000","last_clean_scrub_stamp":"2024-02-05T06:53:44.009607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:13.642503+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":860,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.527142+0000","last_change":"2024-02-05T06:51:30.567546+0000","last_active":"2024-02-05T06:54:04.527142+0000","last_peered":"2024-02-05T06:54:04.527142+0000","last_clean":"2024-02-05T06:54:04.527142+0000","last_became_active":"2024-02-05T06:33:33.593901+0000","last_became_peered":"2024-02-05T06:33:33.593901+0000","last_unstale":"2024-02-05T06:54:04.527142+0000","last_undegraded":"2024-02-05T06:54:04.527142+0000","last_fullsized":"2024-02-05T06:54:04.527142+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:30.567403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:30.567403+0000","last_clean_scrub_stamp":"2024-02-05T06:51:30.567403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"336'2","reported_seq":895,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.508839+0000","last_change":"2024-02-05T06:52:46.298052+0000","last_active":"2024-02-05T06:54:04.508839+0000","last_peered":"2024-02-05T06:54:04.508839+0000","last_clean":"2024-02-05T06:54:04.508839+0000","last_became_active":"2024-02-05T06:41:43.409761+0000","last_became_peered":"2024-02-05T06:41:43.409761+0000","last_unstale":"2024-02-05T06:54:04.508839+0000","last_undegraded":"2024-02-05T06:54:04.508839+0000","last_fullsized":"2024-02-05T06:54:04.508839+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'2","last_scrub_stamp":"2024-02-05T06:52:46.297810+0000","last_deep_scrub":"336'2","last_deep_scrub_stamp":"2024-02-05T06:46:53.500155+0000","last_clean_scrub_stamp":"2024-02-05T06:52:46.297810+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"83'128","reported_seq":1100,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.507208+0000","last_change":"2024-02-05T06:53:47.062697+0000","last_active":"2024-02-05T06:54:04.507208+0000","last_peered":"2024-02-05T06:54:04.507208+0000","last_clean":"2024-02-05T06:54:04.507208+0000","last_became_active":"2024-02-05T06:51:59.292766+0000","last_became_peered":"2024-02-05T06:51:59.292766+0000","last_unstale":"2024-02-05T06:54:04.507208+0000","last_undegraded":"2024-02-05T06:54:04.507208+0000","last_fullsized":"2024-02-05T06:54:04.507208+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:53:47.062582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:53:47.062582+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:10.434191+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"465'32","reported_seq":966,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.518938+0000","last_change":"2024-02-05T06:51:32.530845+0000","last_active":"2024-02-05T06:54:04.518938+0000","last_peered":"2024-02-05T06:54:04.518938+0000","last_clean":"2024-02-05T06:54:04.518938+0000","last_became_active":"2024-02-05T06:40:27.080906+0000","last_became_peered":"2024-02-05T06:40:27.080906+0000","last_unstale":"2024-02-05T06:54:04.518938+0000","last_undegraded":"2024-02-05T06:54:04.518938+0000","last_fullsized":"2024-02-05T06:54:04.518938+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:51:32.530706+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-02-05T06:35:22.938675+0000","last_clean_scrub_stamp":"2024-02-05T06:51:32.530706+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":501,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.504126+0000","last_change":"2024-02-05T06:52:22.931134+0000","last_active":"2024-02-05T06:54:04.504126+0000","last_peered":"2024-02-05T06:54:04.504126+0000","last_clean":"2024-02-05T06:54:04.504126+0000","last_became_active":"2024-02-05T06:41:43.412654+0000","last_became_peered":"2024-02-05T06:41:43.412654+0000","last_unstale":"2024-02-05T06:54:04.504126+0000","last_undegraded":"2024-02-05T06:54:04.504126+0000","last_fullsized":"2024-02-05T06:54:04.504126+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:22.931061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:52:22.931061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":593,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.515050+0000","last_change":"2024-02-05T06:53:34.074529+0000","last_active":"2024-02-05T06:54:04.515050+0000","last_peered":"2024-02-05T06:54:04.515050+0000","last_clean":"2024-02-05T06:54:04.515050+0000","last_became_active":"2024-02-05T06:40:27.062975+0000","last_became_peered":"2024-02-05T06:40:27.062975+0000","last_unstale":"2024-02-05T06:54:04.515050+0000","last_undegraded":"2024-02-05T06:54:04.515050+0000","last_fullsized":"2024-02-05T06:54:04.515050+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:34.074458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:48:34.628182+0000","last_clean_scrub_stamp":"2024-02-05T06:53:34.074458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:41.569557+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":651,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.503984+0000","last_change":"2024-02-05T06:52:40.357841+0000","last_active":"2024-02-05T06:54:04.503984+0000","last_peered":"2024-02-05T06:54:04.503984+0000","last_clean":"2024-02-05T06:54:04.503984+0000","last_became_active":"2024-02-05T06:39:13.245684+0000","last_became_peered":"2024-02-05T06:39:13.245684+0000","last_unstale":"2024-02-05T06:54:04.503984+0000","last_undegraded":"2024-02-05T06:54:04.503984+0000","last_fullsized":"2024-02-05T06:54:04.503984+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:40.356857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:52:40.356857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":268,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"465'12","reported_seq":882,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.520048+0000","last_change":"2024-02-05T06:51:33.570857+0000","last_active":"2024-02-05T06:54:04.520048+0000","last_peered":"2024-02-05T06:54:04.520048+0000","last_clean":"2024-02-05T06:54:04.520048+0000","last_became_active":"2024-02-05T06:33:31.573410+0000","last_became_peered":"2024-02-05T06:33:31.573410+0000","last_unstale":"2024-02-05T06:54:04.520048+0000","last_undegraded":"2024-02-05T06:54:04.520048+0000","last_fullsized":"2024-02-05T06:54:04.520048+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'8","last_scrub_stamp":"2024-02-05T06:51:33.570665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:51:33.570665+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":897,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.509418+0000","last_change":"2024-02-05T06:53:47.483803+0000","last_active":"2024-02-05T06:54:04.509418+0000","last_peered":"2024-02-05T06:54:04.509418+0000","last_clean":"2024-02-05T06:54:04.509418+0000","last_became_active":"2024-02-05T06:41:43.424459+0000","last_became_peered":"2024-02-05T06:41:43.424459+0000","last_unstale":"2024-02-05T06:54:04.509418+0000","last_undegraded":"2024-02-05T06:54:04.509418+0000","last_fullsized":"2024-02-05T06:54:04.509418+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:47.483623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:49.117949+0000","last_clean_scrub_stamp":"2024-02-05T06:53:47.483623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:10.250852+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":931,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.510702+0000","last_change":"2024-02-05T06:53:41.132629+0000","last_active":"2024-02-05T06:54:04.510702+0000","last_peered":"2024-02-05T06:54:04.510702+0000","last_clean":"2024-02-05T06:54:04.510702+0000","last_became_active":"2024-02-05T06:40:27.060112+0000","last_became_peered":"2024-02-05T06:40:27.060112+0000","last_unstale":"2024-02-05T06:54:04.510702+0000","last_undegraded":"2024-02-05T06:54:04.510702+0000","last_fullsized":"2024-02-05T06:54:04.510702+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:41.132523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:48:33.848718+0000","last_clean_scrub_stamp":"2024-02-05T06:53:41.132523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:41.585442+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":599,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.515670+0000","last_change":"2024-02-05T06:53:28.246285+0000","last_active":"2024-02-05T06:54:04.515670+0000","last_peered":"2024-02-05T06:54:04.515670+0000","last_clean":"2024-02-05T06:54:04.515670+0000","last_became_active":"2024-02-05T06:44:54.625067+0000","last_became_peered":"2024-02-05T06:44:54.625067+0000","last_unstale":"2024-02-05T06:54:04.515670+0000","last_undegraded":"2024-02-05T06:54:04.515670+0000","last_fullsized":"2024-02-05T06:54:04.515670+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:28.246206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:48.512673+0000","last_clean_scrub_stamp":"2024-02-05T06:53:28.246206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:36.459451+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"465'32","reported_seq":991,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.511708+0000","last_change":"2024-02-05T06:52:23.643592+0000","last_active":"2024-02-05T06:54:04.511708+0000","last_peered":"2024-02-05T06:54:04.511708+0000","last_clean":"2024-02-05T06:54:04.511708+0000","last_became_active":"2024-02-05T06:41:43.419769+0000","last_became_peered":"2024-02-05T06:41:43.419769+0000","last_unstale":"2024-02-05T06:54:04.511708+0000","last_undegraded":"2024-02-05T06:54:04.511708+0000","last_fullsized":"2024-02-05T06:54:04.511708+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:52:23.643458+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:43:03.221466+0000","last_clean_scrub_stamp":"2024-02-05T06:52:23.643458+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":501,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.503173+0000","last_change":"2024-02-05T06:53:38.839606+0000","last_active":"2024-02-05T06:54:04.503173+0000","last_peered":"2024-02-05T06:54:04.503173+0000","last_clean":"2024-02-05T06:54:04.503173+0000","last_became_active":"2024-02-05T06:41:43.411043+0000","last_became_peered":"2024-02-05T06:41:43.411043+0000","last_unstale":"2024-02-05T06:54:04.503173+0000","last_undegraded":"2024-02-05T06:54:04.503173+0000","last_fullsized":"2024-02-05T06:54:04.503173+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:38.839486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:53:38.839486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:40.388758+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":792,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.525635+0000","last_change":"2024-02-05T06:53:48.540882+0000","last_active":"2024-02-05T06:54:04.525635+0000","last_peered":"2024-02-05T06:54:04.525635+0000","last_clean":"2024-02-05T06:54:04.525635+0000","last_became_active":"2024-02-05T06:40:27.097169+0000","last_became_peered":"2024-02-05T06:40:27.097169+0000","last_unstale":"2024-02-05T06:54:04.525635+0000","last_undegraded":"2024-02-05T06:54:04.525635+0000","last_fullsized":"2024-02-05T06:54:04.525635+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:48.540767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:08.713894+0000","last_clean_scrub_stamp":"2024-02-05T06:53:48.540767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:50.708856+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":904,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.504628+0000","last_change":"2024-02-05T06:52:29.108690+0000","last_active":"2024-02-05T06:54:04.504628+0000","last_peered":"2024-02-05T06:54:04.504628+0000","last_clean":"2024-02-05T06:54:04.504628+0000","last_became_active":"2024-02-05T06:41:43.414638+0000","last_became_peered":"2024-02-05T06:41:43.414638+0000","last_unstale":"2024-02-05T06:54:04.504628+0000","last_undegraded":"2024-02-05T06:54:04.504628+0000","last_fullsized":"2024-02-05T06:54:04.504628+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:29.108567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:50:42.316726+0000","last_clean_scrub_stamp":"2024-02-05T06:52:29.108567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"465'35","reported_seq":977,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.509723+0000","last_change":"2024-02-05T06:53:54.509653+0000","last_active":"2024-02-05T06:54:04.509723+0000","last_peered":"2024-02-05T06:54:04.509723+0000","last_clean":"2024-02-05T06:54:04.509723+0000","last_became_active":"2024-02-05T06:40:27.100610+0000","last_became_peered":"2024-02-05T06:40:27.100610+0000","last_unstale":"2024-02-05T06:54:04.509723+0000","last_undegraded":"2024-02-05T06:54:04.509723+0000","last_fullsized":"2024-02-05T06:54:04.509723+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'35","last_scrub_stamp":"2024-02-05T06:53:54.509362+0000","last_deep_scrub":"465'35","last_deep_scrub_stamp":"2024-02-05T06:53:54.509362+0000","last_clean_scrub_stamp":"2024-02-05T06:53:54.509362+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:00.417269+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":921,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.503694+0000","last_change":"2024-02-05T06:52:50.110926+0000","last_active":"2024-02-05T06:54:04.503694+0000","last_peered":"2024-02-05T06:54:04.503694+0000","last_clean":"2024-02-05T06:54:04.503694+0000","last_became_active":"2024-02-05T06:36:54.958414+0000","last_became_peered":"2024-02-05T06:36:54.958414+0000","last_unstale":"2024-02-05T06:54:04.503694+0000","last_undegraded":"2024-02-05T06:54:04.503694+0000","last_fullsized":"2024-02-05T06:54:04.503694+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:52:50.110793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:52:50.110793+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:14.374092+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"314'13","reported_seq":1205,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.510969+0000","last_change":"2024-02-05T06:52:22.597335+0000","last_active":"2024-02-05T06:54:04.510969+0000","last_peered":"2024-02-05T06:54:04.510969+0000","last_clean":"2024-02-05T06:54:04.510969+0000","last_became_active":"2024-02-05T06:40:27.063399+0000","last_became_peered":"2024-02-05T06:40:27.063399+0000","last_unstale":"2024-02-05T06:54:04.510969+0000","last_undegraded":"2024-02-05T06:54:04.510969+0000","last_fullsized":"2024-02-05T06:54:04.510969+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"314'13","last_scrub_stamp":"2024-02-05T06:52:22.597203+0000","last_deep_scrub":"314'13","last_deep_scrub_stamp":"2024-02-05T06:44:53.670923+0000","last_clean_scrub_stamp":"2024-02-05T06:52:22.597203+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"325'2","reported_seq":600,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.517303+0000","last_change":"2024-02-05T06:52:48.168560+0000","last_active":"2024-02-05T06:54:04.517303+0000","last_peered":"2024-02-05T06:54:04.517303+0000","last_clean":"2024-02-05T06:54:04.517303+0000","last_became_active":"2024-02-05T06:41:43.418640+0000","last_became_peered":"2024-02-05T06:41:43.418640+0000","last_unstale":"2024-02-05T06:54:04.517303+0000","last_undegraded":"2024-02-05T06:54:04.517303+0000","last_fullsized":"2024-02-05T06:54:04.517303+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'2","last_scrub_stamp":"2024-02-05T06:52:48.168445+0000","last_deep_scrub":"325'2","last_deep_scrub_stamp":"2024-02-05T06:45:46.242415+0000","last_clean_scrub_stamp":"2024-02-05T06:52:48.168445+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"465'29","reported_seq":972,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.525342+0000","last_change":"2024-02-05T06:52:37.441014+0000","last_active":"2024-02-05T06:54:04.525342+0000","last_peered":"2024-02-05T06:54:04.525342+0000","last_clean":"2024-02-05T06:54:04.525342+0000","last_became_active":"2024-02-05T06:36:54.954880+0000","last_became_peered":"2024-02-05T06:36:54.954880+0000","last_unstale":"2024-02-05T06:54:04.525342+0000","last_undegraded":"2024-02-05T06:54:04.525342+0000","last_fullsized":"2024-02-05T06:54:04.525342+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'23","last_scrub_stamp":"2024-02-05T06:52:37.440858+0000","last_deep_scrub":"445'23","last_deep_scrub_stamp":"2024-02-05T06:52:37.440858+0000","last_clean_scrub_stamp":"2024-02-05T06:52:37.440858+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":414,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.504506+0000","last_change":"2024-02-05T06:53:51.717338+0000","last_active":"2024-02-05T06:54:04.504506+0000","last_peered":"2024-02-05T06:54:04.504506+0000","last_clean":"2024-02-05T06:54:04.504506+0000","last_became_active":"2024-02-05T06:45:09.786579+0000","last_became_peered":"2024-02-05T06:45:09.786579+0000","last_unstale":"2024-02-05T06:54:04.504506+0000","last_undegraded":"2024-02-05T06:54:04.504506+0000","last_fullsized":"2024-02-05T06:54:04.504506+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:51.717205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:53:51.717205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:02.032086+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":673,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.510720+0000","last_change":"2024-02-05T06:53:51.117144+0000","last_active":"2024-02-05T06:54:04.510720+0000","last_peered":"2024-02-05T06:54:04.510720+0000","last_clean":"2024-02-05T06:54:04.510720+0000","last_became_active":"2024-02-05T06:39:13.248618+0000","last_became_peered":"2024-02-05T06:39:13.248618+0000","last_unstale":"2024-02-05T06:54:04.510720+0000","last_undegraded":"2024-02-05T06:54:04.510720+0000","last_fullsized":"2024-02-05T06:54:04.510720+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:51.116923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:34.551395+0000","last_clean_scrub_stamp":"2024-02-05T06:53:51.116923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:19.941766+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":869,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.526485+0000","last_change":"2024-02-05T06:51:21.832566+0000","last_active":"2024-02-05T06:54:04.526485+0000","last_peered":"2024-02-05T06:54:04.526485+0000","last_clean":"2024-02-05T06:54:04.526485+0000","last_became_active":"2024-02-05T06:41:43.411442+0000","last_became_peered":"2024-02-05T06:41:43.411442+0000","last_unstale":"2024-02-05T06:54:04.526485+0000","last_undegraded":"2024-02-05T06:54:04.526485+0000","last_fullsized":"2024-02-05T06:54:04.526485+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:21.832441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:51:21.832441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"465'46","reported_seq":986,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.511460+0000","last_change":"2024-02-05T06:53:33.046693+0000","last_active":"2024-02-05T06:54:04.511460+0000","last_peered":"2024-02-05T06:54:04.511460+0000","last_clean":"2024-02-05T06:54:04.511460+0000","last_became_active":"2024-02-05T06:33:31.569763+0000","last_became_peered":"2024-02-05T06:33:31.569763+0000","last_unstale":"2024-02-05T06:54:04.511460+0000","last_undegraded":"2024-02-05T06:54:04.511460+0000","last_fullsized":"2024-02-05T06:54:04.511460+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'34","last_scrub_stamp":"2024-02-05T06:53:33.046569+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:34:35.234636+0000","last_clean_scrub_stamp":"2024-02-05T06:53:33.046569+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:47.972370+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"281'2","reported_seq":932,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.509988+0000","last_change":"2024-02-05T06:53:30.452677+0000","last_active":"2024-02-05T06:54:04.509988+0000","last_peered":"2024-02-05T06:54:04.509988+0000","last_clean":"2024-02-05T06:54:04.509988+0000","last_became_active":"2024-02-05T06:40:27.073360+0000","last_became_peered":"2024-02-05T06:40:27.073360+0000","last_unstale":"2024-02-05T06:54:04.509988+0000","last_undegraded":"2024-02-05T06:54:04.509988+0000","last_fullsized":"2024-02-05T06:54:04.509988+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'2","last_scrub_stamp":"2024-02-05T06:53:30.452604+0000","last_deep_scrub":"281'2","last_deep_scrub_stamp":"2024-02-05T06:47:50.442159+0000","last_clean_scrub_stamp":"2024-02-05T06:53:30.452604+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:49.493466+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":896,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.504484+0000","last_change":"2024-02-05T06:53:52.698534+0000","last_active":"2024-02-05T06:54:04.504484+0000","last_peered":"2024-02-05T06:54:04.504484+0000","last_clean":"2024-02-05T06:54:04.504484+0000","last_became_active":"2024-02-05T06:36:54.958069+0000","last_became_peered":"2024-02-05T06:36:54.958069+0000","last_unstale":"2024-02-05T06:54:04.504484+0000","last_undegraded":"2024-02-05T06:54:04.504484+0000","last_fullsized":"2024-02-05T06:54:04.504484+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:52.698426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:38.793760+0000","last_clean_scrub_stamp":"2024-02-05T06:53:52.698426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:13.013123+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":642,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.510612+0000","last_change":"2024-02-05T06:53:36.022199+0000","last_active":"2024-02-05T06:54:04.510612+0000","last_peered":"2024-02-05T06:54:04.510612+0000","last_clean":"2024-02-05T06:54:04.510612+0000","last_became_active":"2024-02-05T06:45:09.770902+0000","last_became_peered":"2024-02-05T06:45:09.770902+0000","last_unstale":"2024-02-05T06:54:04.510612+0000","last_undegraded":"2024-02-05T06:54:04.510612+0000","last_fullsized":"2024-02-05T06:54:04.510612+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:36.022090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:53:36.022090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:39.332622+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"465'9","reported_seq":734,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.526971+0000","last_change":"2024-02-05T06:53:27.348848+0000","last_active":"2024-02-05T06:54:04.526971+0000","last_peered":"2024-02-05T06:54:04.526971+0000","last_clean":"2024-02-05T06:54:04.526971+0000","last_became_active":"2024-02-05T06:45:09.808663+0000","last_became_peered":"2024-02-05T06:45:09.808663+0000","last_unstale":"2024-02-05T06:54:04.526971+0000","last_undegraded":"2024-02-05T06:54:04.526971+0000","last_fullsized":"2024-02-05T06:54:04.526971+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'7","last_scrub_stamp":"2024-02-05T06:53:27.348728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:53:27.348728+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:51.281890+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":500,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.503684+0000","last_change":"2024-02-05T06:52:43.828744+0000","last_active":"2024-02-05T06:54:04.503684+0000","last_peered":"2024-02-05T06:54:04.503684+0000","last_clean":"2024-02-05T06:54:04.503684+0000","last_became_active":"2024-02-05T06:41:43.408428+0000","last_became_peered":"2024-02-05T06:41:43.408428+0000","last_unstale":"2024-02-05T06:54:04.503684+0000","last_undegraded":"2024-02-05T06:54:04.503684+0000","last_fullsized":"2024-02-05T06:54:04.503684+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:43.828638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:43.828638+0000","last_clean_scrub_stamp":"2024-02-05T06:52:43.828638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":601,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.515483+0000","last_change":"2024-02-05T06:53:58.717874+0000","last_active":"2024-02-05T06:54:04.515483+0000","last_peered":"2024-02-05T06:54:04.515483+0000","last_clean":"2024-02-05T06:54:04.515483+0000","last_became_active":"2024-02-05T06:40:27.071758+0000","last_became_peered":"2024-02-05T06:40:27.071758+0000","last_unstale":"2024-02-05T06:54:04.515483+0000","last_undegraded":"2024-02-05T06:54:04.515483+0000","last_fullsized":"2024-02-05T06:54:04.515483+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:58.717756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:37.152741+0000","last_clean_scrub_stamp":"2024-02-05T06:53:58.717756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:16.156426+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":501,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.503282+0000","last_change":"2024-02-05T06:53:34.839643+0000","last_active":"2024-02-05T06:54:04.503282+0000","last_peered":"2024-02-05T06:54:04.503282+0000","last_clean":"2024-02-05T06:54:04.503282+0000","last_became_active":"2024-02-05T06:41:43.407404+0000","last_became_peered":"2024-02-05T06:41:43.407404+0000","last_unstale":"2024-02-05T06:54:04.503282+0000","last_undegraded":"2024-02-05T06:54:04.503282+0000","last_fullsized":"2024-02-05T06:54:04.503282+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:34.839509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:21.998850+0000","last_clean_scrub_stamp":"2024-02-05T06:53:34.839509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:54.888016+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"465'25","reported_seq":742,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.523725+0000","last_change":"2024-02-05T06:52:31.409340+0000","last_active":"2024-02-05T06:54:04.523725+0000","last_peered":"2024-02-05T06:54:04.523725+0000","last_clean":"2024-02-05T06:54:04.523725+0000","last_became_active":"2024-02-05T06:45:09.787626+0000","last_became_peered":"2024-02-05T06:45:09.787626+0000","last_unstale":"2024-02-05T06:54:04.523725+0000","last_undegraded":"2024-02-05T06:54:04.523725+0000","last_fullsized":"2024-02-05T06:54:04.523725+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'19","last_scrub_stamp":"2024-02-05T06:52:31.409196+0000","last_deep_scrub":"92'13","last_deep_scrub_stamp":"2024-02-05T06:34:46.542893+0000","last_clean_scrub_stamp":"2024-02-05T06:52:31.409196+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":492,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.504659+0000","last_change":"2024-02-05T06:51:30.797965+0000","last_active":"2024-02-05T06:54:04.504659+0000","last_peered":"2024-02-05T06:54:04.504659+0000","last_clean":"2024-02-05T06:54:04.504659+0000","last_became_active":"2024-02-05T06:41:43.403746+0000","last_became_peered":"2024-02-05T06:41:43.403746+0000","last_unstale":"2024-02-05T06:54:04.504659+0000","last_undegraded":"2024-02-05T06:54:04.504659+0000","last_fullsized":"2024-02-05T06:54:04.504659+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:30.797854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:44.081318+0000","last_clean_scrub_stamp":"2024-02-05T06:51:30.797854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":441,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.527313+0000","last_change":"2024-02-05T06:53:34.584200+0000","last_active":"2024-02-05T06:54:04.527313+0000","last_peered":"2024-02-05T06:54:04.527313+0000","last_clean":"2024-02-05T06:54:04.527313+0000","last_became_active":"2024-02-05T06:42:33.129959+0000","last_became_peered":"2024-02-05T06:42:33.129959+0000","last_unstale":"2024-02-05T06:54:04.527313+0000","last_undegraded":"2024-02-05T06:54:04.527313+0000","last_fullsized":"2024-02-05T06:54:04.527313+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:34.583981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:45:51.217900+0000","last_clean_scrub_stamp":"2024-02-05T06:53:34.583981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:55.085329+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":885,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.509121+0000","last_change":"2024-02-05T06:52:51.154037+0000","last_active":"2024-02-05T06:54:04.509121+0000","last_peered":"2024-02-05T06:54:04.509121+0000","last_clean":"2024-02-05T06:54:04.509121+0000","last_became_active":"2024-02-05T06:40:27.068888+0000","last_became_peered":"2024-02-05T06:40:27.068888+0000","last_unstale":"2024-02-05T06:54:04.509121+0000","last_undegraded":"2024-02-05T06:54:04.509121+0000","last_fullsized":"2024-02-05T06:54:04.509121+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:51.153442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:44:55.125921+0000","last_clean_scrub_stamp":"2024-02-05T06:52:51.153442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:16.151191+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"465'46","reported_seq":940,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.519035+0000","last_change":"2024-02-05T06:52:27.389525+0000","last_active":"2024-02-05T06:54:04.519035+0000","last_peered":"2024-02-05T06:54:04.519035+0000","last_clean":"2024-02-05T06:54:04.519035+0000","last_became_active":"2024-02-05T06:33:31.572544+0000","last_became_peered":"2024-02-05T06:33:31.572544+0000","last_unstale":"2024-02-05T06:54:04.519035+0000","last_undegraded":"2024-02-05T06:54:04.519035+0000","last_fullsized":"2024-02-05T06:54:04.519035+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'34","last_scrub_stamp":"2024-02-05T06:52:27.389433+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:39:35.433539+0000","last_clean_scrub_stamp":"2024-02-05T06:52:27.389433+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":878,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.525780+0000","last_change":"2024-02-05T06:51:23.547319+0000","last_active":"2024-02-05T06:54:04.525780+0000","last_peered":"2024-02-05T06:54:04.525780+0000","last_clean":"2024-02-05T06:54:04.525780+0000","last_became_active":"2024-02-05T06:33:29.553864+0000","last_became_peered":"2024-02-05T06:33:29.553864+0000","last_unstale":"2024-02-05T06:54:04.525780+0000","last_undegraded":"2024-02-05T06:54:04.525780+0000","last_fullsized":"2024-02-05T06:54:04.525780+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:23.547188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:51:23.547188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":406,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.510422+0000","last_change":"2024-02-05T06:53:37.018077+0000","last_active":"2024-02-05T06:54:04.510422+0000","last_peered":"2024-02-05T06:54:04.510422+0000","last_clean":"2024-02-05T06:54:04.510422+0000","last_became_active":"2024-02-05T06:45:09.775037+0000","last_became_peered":"2024-02-05T06:45:09.775037+0000","last_unstale":"2024-02-05T06:54:04.510422+0000","last_undegraded":"2024-02-05T06:54:04.510422+0000","last_fullsized":"2024-02-05T06:54:04.510422+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:37.018004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:51.388156+0000","last_clean_scrub_stamp":"2024-02-05T06:53:37.018004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:52.016322+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":876,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.527222+0000","last_change":"2024-02-05T06:53:30.666137+0000","last_active":"2024-02-05T06:54:04.527222+0000","last_peered":"2024-02-05T06:54:04.527222+0000","last_clean":"2024-02-05T06:54:04.527222+0000","last_became_active":"2024-02-05T06:40:27.077485+0000","last_became_peered":"2024-02-05T06:40:27.077485+0000","last_unstale":"2024-02-05T06:54:04.527222+0000","last_undegraded":"2024-02-05T06:54:04.527222+0000","last_fullsized":"2024-02-05T06:54:04.527222+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:30.666019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:53:30.666019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:55.573254+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"465'25","reported_seq":826,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.500732+0000","last_change":"2024-02-05T06:53:25.983530+0000","last_active":"2024-02-05T06:54:04.500732+0000","last_peered":"2024-02-05T06:54:04.500732+0000","last_clean":"2024-02-05T06:54:04.500732+0000","last_became_active":"2024-02-05T06:36:54.959330+0000","last_became_peered":"2024-02-05T06:36:54.959330+0000","last_unstale":"2024-02-05T06:54:04.500732+0000","last_undegraded":"2024-02-05T06:54:04.500732+0000","last_fullsized":"2024-02-05T06:54:04.500732+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'19","last_scrub_stamp":"2024-02-05T06:53:25.983402+0000","last_deep_scrub":"247'13","last_deep_scrub_stamp":"2024-02-05T06:41:42.473424+0000","last_clean_scrub_stamp":"2024-02-05T06:53:25.983402+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:41.235478+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":31,"num_read_kb":25,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":898,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.509191+0000","last_change":"2024-02-05T06:53:48.501549+0000","last_active":"2024-02-05T06:54:04.509191+0000","last_peered":"2024-02-05T06:54:04.509191+0000","last_clean":"2024-02-05T06:54:04.509191+0000","last_became_active":"2024-02-05T06:41:43.421315+0000","last_became_peered":"2024-02-05T06:41:43.421315+0000","last_unstale":"2024-02-05T06:54:04.509191+0000","last_undegraded":"2024-02-05T06:54:04.509191+0000","last_fullsized":"2024-02-05T06:54:04.509191+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:48.501421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:53:48.501421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:01.128496+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":908,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.503789+0000","last_change":"2024-02-05T06:53:46.710740+0000","last_active":"2024-02-05T06:54:04.503789+0000","last_peered":"2024-02-05T06:54:04.503789+0000","last_clean":"2024-02-05T06:54:04.503789+0000","last_became_active":"2024-02-05T06:41:43.420342+0000","last_became_peered":"2024-02-05T06:41:43.420342+0000","last_unstale":"2024-02-05T06:54:04.503789+0000","last_undegraded":"2024-02-05T06:54:04.503789+0000","last_fullsized":"2024-02-05T06:54:04.503789+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:46.710631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:06.525012+0000","last_clean_scrub_stamp":"2024-02-05T06:53:46.710631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:09.216440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":637,"reported_epoch":471,"state":"active+clean","last_fresh":"2024-02-05T06:54:04.503908+0000","last_change":"2024-02-05T06:52:28.883912+0000","last_active":"2024-02-05T06:54:04.503908+0000","last_peered":"2024-02-05T06:54:04.503908+0000","last_clean":"2024-02-05T06:54:04.503908+0000","last_became_active":"2024-02-05T06:45:09.787134+0000","last_became_peered":"2024-02-05T06:45:09.787134+0000","last_unstale":"2024-02-05T06:54:04.503908+0000","last_undegraded":"2024-02-05T06:54:04.503908+0000","last_fullsized":"2024-02-05T06:54:04.503908+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:28.883819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:52:28.883819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":3236102168,"num_objects":49380,"num_object_clones":0,"num_object_copies":148140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49380,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":204548,"num_write_kb":8125633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9740562432,"data_stored":9740550216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75871,"ondisk_log_size":75871,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":825,"num_read_kb":820,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2688,"ondisk_log_size":2688,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1535,"num_read_kb":1292,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":373,"num_bytes_recovered":8234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1086,"ondisk_log_size":1086,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103419,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2025024,"kb_used_data":1213556,"kb_used_omap":1,"kb_used_meta":811390,"kb_avail":91720128,"statfs":{"total":95995035648,"available":93921411072,"internally_reserved":0,"allocated":1242681344,"data_stored":1242287510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":830863818},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:53:13 2024","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.2050000000000001,"15min":1.3999999999999999},"min":{"1min":0.23799999999999999,"5min":0.23400000000000001,"15min":0.21299999999999999},"max":{"1min":4.1029999999999998,"5min":5.2329999999999997,"15min":52.890999999999998},"last":0.91600000000000004},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.512,"15min":1.583},"min":{"1min":0.255,"5min":0.255,"15min":0.222},"max":{"1min":3.1400000000000001,"5min":41.109999999999999,"15min":53.353000000000002},"last":0.42999999999999999}]},{"osd":1,"last update":"Mon Feb 5 06:53:13 2024","interfaces":[{"interface":"back","average":{"1min":1.9670000000000001,"5min":1.6579999999999999,"15min":1.7909999999999999},"min":{"1min":0.318,"5min":0.26000000000000001,"15min":0.25},"max":{"1min":8.2569999999999997,"5min":23.716999999999999,"15min":84.216999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":1.492,"5min":1.7190000000000001,"15min":1.8740000000000001},"min":{"1min":0.24299999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":7.9450000000000003,"5min":41.167999999999999,"15min":84.126999999999995},"last":0.53600000000000003}]},{"osd":2,"last update":"Mon Feb 5 06:53:13 2024","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.5409999999999999,"15min":1.506},"min":{"1min":0.32700000000000001,"5min":0.249,"15min":0.249},"max":{"1min":8.4740000000000002,"5min":28.704999999999998,"15min":53.947000000000003},"last":0.78400000000000003},{"interface":"front","average":{"1min":1.54,"5min":1.6319999999999999,"15min":1.506},"min":{"1min":0.34100000000000003,"5min":0.28899999999999998,"15min":0.26800000000000002},"max":{"1min":11.452,"5min":23.678999999999998,"15min":53.299999999999997},"last":0.62}]},{"osd":3,"last update":"Mon Feb 5 06:53:13 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.462,"15min":1.794},"min":{"1min":0.39800000000000002,"5min":0.29099999999999998,"15min":0.28899999999999998},"max":{"1min":7.8250000000000002,"5min":23.199000000000002,"15min":73.283000000000001},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.72,"5min":1.8839999999999999,"15min":1.8560000000000001},"min":{"1min":0.307,"5min":0.29499999999999998,"15min":0.27400000000000002},"max":{"1min":7.9089999999999998,"5min":41.219000000000001,"15min":73.323999999999998},"last":0.69899999999999995}]},{"osd":4,"last update":"Mon Feb 5 06:53:13 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":1.0960000000000001,"15min":0.81299999999999994},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.1930000000000001,"5min":6.1980000000000004,"15min":6.1980000000000004},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.76100000000000001,"5min":1.4099999999999999,"15min":0.91400000000000003},"min":{"1min":0.318,"5min":0.25600000000000001,"15min":0.255},"max":{"1min":3.48,"5min":41.270000000000003,"15min":41.270000000000003},"last":0.71199999999999997}]},{"osd":5,"last update":"Mon Feb 5 06:53:13 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":2.4580000000000002,"15min":1.375},"min":{"1min":0.309,"5min":0.252,"15min":0.251},"max":{"1min":4.0170000000000003,"5min":109.482,"15min":109.482},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.88100000000000001,"5min":2.5270000000000001,"15min":1.458},"min":{"1min":0.30199999999999999,"5min":0.29999999999999999,"15min":0.252},"max":{"1min":3.1699999999999999,"5min":109.56399999999999,"15min":109.56399999999999},"last":0.80200000000000005}]},{"osd":6,"last update":"Mon Feb 5 06:53:13 2024","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":2.8769999999999998,"15min":1.5229999999999999},"min":{"1min":0.378,"5min":0.34899999999999998,"15min":0.28499999999999998},"max":{"1min":3.2490000000000001,"5min":113.914,"15min":113.914},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.79000000000000004,"5min":2.714,"15min":1.4350000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.307},"max":{"1min":3.0190000000000001,"5min":113.212,"15min":113.212},"last":0.67100000000000004}]}]},{"osd":6,"up_from":49,"seq":210453397761,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2357096,"kb_used_data":1599840,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":91388056,"statfs":{"total":95995035648,"available":93581369344,"internally_reserved":0,"allocated":1638236160,"data_stored":1637861381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":775420358},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:53:34 2024","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.482,"15min":2.3959999999999999},"min":{"1min":0.249,"5min":0.23699999999999999,"15min":0.224},"max":{"1min":10.039999999999999,"5min":15.401,"15min":98.918999999999997},"last":0.249},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.744,"15min":2.464},"min":{"1min":0.27400000000000002,"5min":0.223,"15min":0.223},"max":{"1min":10.23,"5min":15.388999999999999,"15min":98.775000000000006},"last":1.8420000000000001}]},{"osd":1,"last update":"Mon Feb 5 06:53:34 2024","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.6359999999999999,"15min":2.173},"min":{"1min":0.28199999999999997,"5min":0.254,"15min":0.24099999999999999},"max":{"1min":6.3730000000000002,"5min":47.348999999999997,"15min":98.843999999999994},"last":1.8620000000000001},{"interface":"front","average":{"1min":1.286,"5min":1.623,"15min":2.206},"min":{"1min":0.27800000000000002,"5min":0.24099999999999999,"15min":0.23100000000000001},"max":{"1min":6.3899999999999997,"5min":47.777999999999999,"15min":101.637},"last":1.5980000000000001}]},{"osd":2,"last update":"Mon Feb 5 06:53:34 2024","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.361,"15min":2.1150000000000002},"min":{"1min":0.29599999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":10.446,"5min":10.446,"15min":85.617000000000004},"last":2.2200000000000002},{"interface":"front","average":{"1min":1.607,"5min":1.6259999999999999,"15min":2.0779999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27000000000000002},"max":{"1min":16.516999999999999,"5min":16.516999999999999,"15min":85.335999999999999},"last":2.1709999999999998}]},{"osd":3,"last update":"Mon Feb 5 06:53:34 2024","interfaces":[{"interface":"back","average":{"1min":1.768,"5min":1.3779999999999999,"15min":2.5129999999999999},"min":{"1min":0.30299999999999999,"5min":0.27900000000000003,"15min":0.27000000000000002},"max":{"1min":14.648,"5min":14.648,"15min":99.269000000000005},"last":0.77600000000000002},{"interface":"front","average":{"1min":1.887,"5min":1.415,"15min":2.5870000000000002},"min":{"1min":0.31900000000000001,"5min":0.313,"15min":0.27800000000000002},"max":{"1min":14.6,"5min":14.6,"15min":101.75700000000001},"last":1.5169999999999999}]},{"osd":4,"last update":"Mon Feb 5 06:53:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.95399999999999996,"15min":0.72999999999999998},"min":{"1min":0.253,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":2.3220000000000001,"5min":5.5940000000000003,"15min":5.5940000000000003},"last":1.9370000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":1.018,"15min":0.745},"min":{"1min":0.33500000000000002,"5min":0.312,"15min":0.24299999999999999},"max":{"1min":1.3029999999999999,"5min":5.9500000000000002,"15min":5.9500000000000002},"last":0.83499999999999996}]},{"osd":5,"last update":"Mon Feb 5 06:53:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":1.0640000000000001,"15min":0.98299999999999998},"min":{"1min":0.30199999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":2.383,"5min":5.5570000000000004,"15min":36.548999999999999},"last":1.8999999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.80100000000000005,"15min":0.93000000000000005},"min":{"1min":0.34200000000000003,"5min":0.30299999999999999,"15min":0.28699999999999998},"max":{"1min":0.74099999999999999,"5min":3.75,"15min":36.433},"last":0.80900000000000005}]},{"osd":7,"last update":"Mon Feb 5 06:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.75,"15min":0.85299999999999998},"min":{"1min":0.31,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.177,"5min":3.6200000000000001,"15min":38.436999999999998},"last":1.569},{"interface":"front","average":{"1min":0.54600000000000004,"5min":1.036,"15min":0.95399999999999996},"min":{"1min":0.35899999999999999,"5min":0.29899999999999999,"15min":0.29699999999999999},"max":{"1min":1.244,"5min":5.3280000000000003,"15min":36.633000000000003},"last":1.968}]}]},{"osd":1,"up_from":13,"seq":55834575130,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2461084,"kb_used_data":1563604,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":91284068,"statfs":{"total":95995035648,"available":93474885632,"internally_reserved":0,"allocated":1601130496,"data_stored":1600704677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":919009733},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,3,7,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:53:32 2024","interfaces":[{"interface":"back","average":{"1min":7.1289999999999996,"5min":2.9929999999999999,"15min":2.145},"min":{"1min":0.41699999999999998,"5min":0.20899999999999999,"15min":0.16700000000000001},"max":{"1min":93.488,"5min":93.488,"15min":93.488},"last":1.821},{"interface":"front","average":{"1min":7.9020000000000001,"5min":3.0179999999999998,"15min":2.3079999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":93.689999999999998,"5min":93.689999999999998,"15min":93.689999999999998},"last":6.0419999999999998}]},{"osd":2,"last update":"Mon Feb 5 06:53:55 2024","interfaces":[{"interface":"back","average":{"1min":6.649,"5min":3.069,"15min":2.1179999999999999},"min":{"1min":0.38900000000000001,"5min":0.184,"15min":0.184},"max":{"1min":93.352999999999994,"5min":93.352999999999994,"15min":93.352999999999994},"last":3.3679999999999999},{"interface":"front","average":{"1min":6.5679999999999996,"5min":2.931,"15min":1.8959999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.22600000000000001},"max":{"1min":91.543000000000006,"5min":91.543000000000006,"15min":91.543000000000006},"last":0.33500000000000002}]},{"osd":3,"last update":"Mon Feb 5 06:53:13 2024","interfaces":[{"interface":"back","average":{"1min":2.3279999999999998,"5min":1.716,"15min":2.0179999999999998},"min":{"1min":0.50800000000000001,"5min":0.25900000000000001,"15min":0.23499999999999999},"max":{"1min":4.6159999999999997,"5min":7.9340000000000002,"15min":25.236999999999998},"last":3.218},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.76,"15min":1.782},"min":{"1min":0.255,"5min":0.255,"15min":0.23699999999999999},"max":{"1min":3.0649999999999999,"5min":11.018000000000001,"15min":43.354999999999997},"last":3.3100000000000001}]},{"osd":4,"last update":"Mon Feb 5 06:53:38 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.2989999999999999,"15min":1.151},"min":{"1min":0.32500000000000001,"5min":0.32000000000000001,"15min":0.27700000000000002},"max":{"1min":5.3360000000000003,"5min":7.4059999999999997,"15min":7.4059999999999997},"last":3.1000000000000001},{"interface":"front","average":{"1min":2.4209999999999998,"5min":1.6850000000000001,"15min":1.472},"min":{"1min":0.42299999999999999,"5min":0.33100000000000002,"15min":0.26300000000000001},"max":{"1min":9.8680000000000003,"5min":9.8680000000000003,"15min":10.419},"last":3.2509999999999999}]},{"osd":5,"last update":"Mon Feb 5 06:53:13 2024","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.607,"15min":1.425},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.29299999999999998},"max":{"1min":2.9860000000000002,"5min":11.058,"15min":43.414999999999999},"last":1.851},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.107,"15min":1.204},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":3.1480000000000001,"5min":8.1669999999999998,"15min":8.1669999999999998},"last":0.39400000000000002}]},{"osd":6,"last update":"Mon Feb 5 06:53:32 2024","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.5700000000000001,"15min":1.397},"min":{"1min":0.39900000000000002,"5min":0.38500000000000001,"15min":0.34799999999999998},"max":{"1min":4.4189999999999996,"5min":19.827999999999999,"15min":23.902999999999999},"last":0.42099999999999999},{"interface":"front","average":{"1min":1.554,"5min":2.0430000000000001,"15min":1.6060000000000001},"min":{"1min":0.44700000000000001,"5min":0.39300000000000002,"15min":0.35599999999999998},"max":{"1min":5.0049999999999999,"5min":19.526,"15min":43.475000000000001},"last":1.881}]},{"osd":7,"last update":"Mon Feb 5 06:53:55 2024","interfaces":[{"interface":"back","average":{"1min":1.7809999999999999,"5min":1.405,"15min":1.349},"min":{"1min":0.41199999999999998,"5min":0.377,"15min":0.32900000000000001},"max":{"1min":8.1600000000000001,"5min":8.1600000000000001,"15min":8.1600000000000001},"last":0.443},{"interface":"front","average":{"1min":2.4470000000000001,"5min":1.645,"15min":1.5349999999999999},"min":{"1min":0.43099999999999999,"5min":0.36599999999999999,"15min":0.35499999999999998},"max":{"1min":9.8230000000000004,"5min":9.8230000000000004,"15min":10.481},"last":3.286}]}]},{"osd":0,"up_from":8,"seq":34359738656,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1799948,"kb_used_data":1010628,"kb_used_omap":1,"kb_used_meta":789182,"kb_avail":91945204,"statfs":{"total":95995035648,"available":94151888896,"internally_reserved":0,"allocated":1034883072,"data_stored":1034472830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":808122821},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 06:53:24 2024","interfaces":[{"interface":"back","average":{"1min":2.48,"5min":2.012,"15min":1.6679999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.182},"max":{"1min":30.350999999999999,"5min":38.68,"15min":38.68},"last":0.54700000000000004},{"interface":"front","average":{"1min":3.335,"5min":2.5960000000000001,"15min":1.9910000000000001},"min":{"1min":0.41599999999999998,"5min":0.19500000000000001,"15min":0.18099999999999999},"max":{"1min":31.122,"5min":38.003,"15min":38.003},"last":0.59299999999999997}]},{"osd":2,"last update":"Mon Feb 5 06:53:45 2024","interfaces":[{"interface":"back","average":{"1min":2.1179999999999999,"5min":2.4740000000000002,"15min":1.7869999999999999},"min":{"1min":0.39400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.3559999999999999,"5min":27.312000000000001,"15min":27.312000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":1.6759999999999999,"5min":2.2589999999999999,"15min":1.6719999999999999},"min":{"1min":0.28299999999999997,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.298,"5min":27.359000000000002,"15min":27.359000000000002},"last":0.377}]},{"osd":3,"last update":"Mon Feb 5 06:53:14 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.869,"15min":1.8280000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":3.4420000000000002,"5min":31.777999999999999,"15min":43.302},"last":1.4610000000000001},{"interface":"front","average":{"1min":2.8210000000000002,"5min":2.101,"15min":2.1080000000000001},"min":{"1min":0.26100000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":9.1579999999999995,"5min":31.716000000000001,"15min":43.651000000000003},"last":1.5740000000000001}]},{"osd":4,"last update":"Mon Feb 5 06:53:37 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.8140000000000001,"15min":1.5589999999999999},"min":{"1min":0.33100000000000002,"5min":0.32200000000000001,"15min":0.27200000000000002},"max":{"1min":4.6959999999999997,"5min":10.952999999999999,"15min":40.789999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.94199999999999995,"5min":1.3200000000000001,"15min":1.3999999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.30099999999999999},"max":{"1min":3.5699999999999998,"5min":5.9459999999999997,"15min":40.773000000000003},"last":0.60499999999999998}]},{"osd":5,"last update":"Mon Feb 5 06:53:06 2024","interfaces":[{"interface":"back","average":{"1min":1.877,"5min":1.5529999999999999,"15min":1.609},"min":{"1min":0.35099999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":7.1239999999999997,"5min":7.1239999999999997,"15min":48.841999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.8400000000000001,"15min":1.6539999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28599999999999998},"max":{"1min":4.1859999999999999,"5min":11.156000000000001,"15min":48.450000000000003},"last":1.232}]},{"osd":6,"last update":"Mon Feb 5 06:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":1.2929999999999999,"15min":1.7589999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.33700000000000002},"max":{"1min":2.347,"5min":7.0049999999999999,"15min":136.76300000000001},"last":2.6789999999999998},{"interface":"front","average":{"1min":1.7350000000000001,"5min":1.78,"15min":2.0550000000000002},"min":{"1min":0.39800000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":6.0090000000000003,"5min":7.0720000000000001,"15min":135.92400000000001},"last":0.52900000000000003}]},{"osd":7,"last update":"Mon Feb 5 06:53:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.7470000000000001,"15min":1.4830000000000001},"min":{"1min":0.40699999999999997,"5min":0.35499999999999998,"15min":0.31900000000000001},"max":{"1min":5.2249999999999996,"5min":10.989000000000001,"15min":17.210000000000001},"last":1.5169999999999999},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.222,"15min":1.276},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.34899999999999998},"max":{"1min":3.6040000000000001,"5min":6.383,"15min":18.533999999999999},"last":0.57699999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411605,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1877496,"kb_used_data":1196592,"kb_used_omap":1,"kb_used_meta":680894,"kb_avail":91867656,"statfs":{"total":95995035648,"available":94072479744,"internally_reserved":0,"allocated":1225310208,"data_stored":1224920907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":697235909},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,0,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:53:55 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":3.6970000000000001,"15min":2.298},"min":{"1min":0.31,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.7489999999999997,"5min":101.67700000000001,"15min":101.67700000000001},"last":2.27},{"interface":"front","average":{"1min":0.99299999999999999,"5min":3.302,"15min":2.2290000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.17699999999999999},"max":{"1min":3.331,"5min":101.27800000000001,"15min":101.27800000000001},"last":0.83799999999999997}]},{"osd":1,"last update":"Mon Feb 5 06:53:55 2024","interfaces":[{"interface":"back","average":{"1min":2.1389999999999998,"5min":2.1920000000000002,"15min":1.7969999999999999},"min":{"1min":0.19500000000000001,"5min":0.185,"15min":0.185},"max":{"1min":6.7169999999999996,"5min":24.093,"15min":24.093},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.76,"5min":1.7969999999999999,"15min":1.5109999999999999},"min":{"1min":0.36499999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.6870000000000003,"5min":27.114000000000001,"15min":27.114000000000001},"last":3.1629999999999998}]},{"osd":3,"last update":"Mon Feb 5 06:53:22 2024","interfaces":[{"interface":"back","average":{"1min":2.069,"5min":2.871,"15min":2.0150000000000001},"min":{"1min":0.36299999999999999,"5min":0.21299999999999999,"15min":0.20899999999999999},"max":{"1min":5.4160000000000004,"5min":84.129000000000005,"15min":84.129000000000005},"last":2.246},{"interface":"front","average":{"1min":2.4929999999999999,"5min":2.7050000000000001,"15min":1.9410000000000001},"min":{"1min":0.34699999999999998,"5min":0.214,"15min":0.214},"max":{"1min":9.641,"5min":84.090999999999994,"15min":84.090999999999994},"last":1.048}]},{"osd":4,"last update":"Mon Feb 5 06:53:31 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.2629999999999999,"15min":1.149},"min":{"1min":0.39100000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":12.041},"last":2.2200000000000002},{"interface":"front","average":{"1min":0.82599999999999996,"5min":1.278,"15min":1.1319999999999999},"min":{"1min":0.35099999999999998,"5min":0.28000000000000003,"15min":0.26300000000000001},"max":{"1min":3.3679999999999999,"5min":5.4640000000000004,"15min":15.689},"last":0.98299999999999998}]},{"osd":5,"last update":"Mon Feb 5 06:53:05 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":2.5760000000000001,"15min":2.081},"min":{"1min":0.34999999999999998,"5min":0.314,"15min":0.314},"max":{"1min":3.673,"5min":78.497,"15min":86.614999999999995},"last":0.873},{"interface":"front","average":{"1min":1.3859999999999999,"5min":2.423,"15min":2.2069999999999999},"min":{"1min":0.378,"5min":0.32300000000000001,"15min":0.312},"max":{"1min":5.0330000000000004,"5min":78.774000000000001,"15min":86.543999999999997},"last":0.58099999999999996}]},{"osd":6,"last update":"Mon Feb 5 06:53:31 2024","interfaces":[{"interface":"back","average":{"1min":4.2809999999999997,"5min":2.1200000000000001,"15min":1.5029999999999999},"min":{"1min":0.372,"5min":0.36799999999999999,"15min":0.315},"max":{"1min":53.332000000000001,"5min":53.332000000000001,"15min":53.332000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":4.4530000000000003,"5min":2.1539999999999999,"15min":1.8},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.316},"max":{"1min":53.320999999999998,"5min":53.320999999999998,"15min":53.320999999999998},"last":0.629}]},{"osd":7,"last update":"Mon Feb 5 06:53:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8620000000000001,"5min":1.653,"15min":1.623},"min":{"1min":0.44500000000000001,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":5.04,"5min":9.3800000000000008,"15min":45.243000000000002},"last":0.66400000000000003},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.4510000000000001,"15min":1.5169999999999999},"min":{"1min":0.48399999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":5.593,"5min":7.5449999999999999,"15min":45.171999999999997},"last":2.1819999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149968,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2244104,"kb_used_data":1382528,"kb_used_omap":1,"kb_used_meta":861374,"kb_avail":91501048,"statfs":{"total":95995035648,"available":93697073152,"internally_reserved":0,"allocated":1415708672,"data_stored":1415302426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":882047431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:53:27 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.9319999999999999,"15min":1.8380000000000001},"min":{"1min":0.26100000000000001,"5min":0.20200000000000001,"15min":0.184},"max":{"1min":9.8849999999999998,"5min":38.686999999999998,"15min":53.656999999999996},"last":2.8900000000000001},{"interface":"front","average":{"1min":1.657,"5min":2.298,"15min":2.0110000000000001},"min":{"1min":0.38700000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.7210000000000001,"5min":35.506999999999998,"15min":53.338000000000001},"last":3.3239999999999998}]},{"osd":1,"last update":"Mon Feb 5 06:53:27 2024","interfaces":[{"interface":"back","average":{"1min":1.917,"5min":2.0339999999999998,"15min":1.7330000000000001},"min":{"1min":0.41899999999999998,"5min":0.22500000000000001,"15min":0.20499999999999999},"max":{"1min":8.9220000000000006,"5min":24.343,"15min":24.343},"last":0.44500000000000001},{"interface":"front","average":{"1min":1.8720000000000001,"5min":1.863,"15min":1.756},"min":{"1min":0.442,"5min":0.22600000000000001,"15min":0.215},"max":{"1min":6.4189999999999996,"5min":23.300999999999998,"15min":23.300999999999998},"last":0.30599999999999999}]},{"osd":2,"last update":"Mon Feb 5 06:53:27 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.962,"15min":1.806},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.23000000000000001},"max":{"1min":4.1879999999999997,"5min":14.746,"15min":41.529000000000003},"last":3.6760000000000002},{"interface":"front","average":{"1min":1.7509999999999999,"5min":1.744,"15min":1.6459999999999999},"min":{"1min":0.36099999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":10.217000000000001,"5min":14.734,"15min":41.759999999999998},"last":2.9140000000000001}]},{"osd":4,"last update":"Mon Feb 5 06:53:43 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.4099999999999999,"15min":1.268},"min":{"1min":0.41099999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":5.1100000000000003,"5min":16.59,"15min":16.59},"last":2.7869999999999999},{"interface":"front","average":{"1min":1.256,"5min":1.4870000000000001,"15min":1.4690000000000001},"min":{"1min":0.41699999999999998,"5min":0.29499999999999998,"15min":0.28599999999999998},"max":{"1min":4.5789999999999997,"5min":8.7260000000000009,"15min":19.359999999999999},"last":0.497}]},{"osd":5,"last update":"Mon Feb 5 06:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":1.361,"15min":1.4530000000000001},"min":{"1min":0.33000000000000002,"5min":0.32200000000000001,"15min":0.29299999999999998},"max":{"1min":2.238,"5min":11.458,"15min":20.257999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":1.048,"5min":1.554,"15min":1.3959999999999999},"min":{"1min":0.40999999999999998,"5min":0.33700000000000002,"15min":0.32800000000000001},"max":{"1min":3.6480000000000001,"5min":16.629999999999999,"15min":16.629999999999999},"last":2.831}]},{"osd":6,"last update":"Mon Feb 5 06:53:43 2024","interfaces":[{"interface":"back","average":{"1min":1.7490000000000001,"5min":1.835,"15min":1.522},"min":{"1min":0.42399999999999999,"5min":0.33100000000000002,"15min":0.28100000000000003},"max":{"1min":7.0279999999999996,"5min":16.643000000000001,"15min":16.643000000000001},"last":0.47499999999999998},{"interface":"front","average":{"1min":1.24,"5min":1.7210000000000001,"15min":1.49},"min":{"1min":0.41199999999999998,"5min":0.35299999999999998,"15min":0.318},"max":{"1min":2.448,"5min":12.75,"15min":12.75},"last":0.375}]},{"osd":7,"last update":"Mon Feb 5 06:54:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0110000000000001,"5min":1.671,"15min":1.462},"min":{"1min":0.46200000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":11.824,"5min":11.824,"15min":11.824},"last":2.8620000000000001},{"interface":"front","average":{"1min":0.96099999999999997,"5min":1.5840000000000001,"15min":1.4910000000000001},"min":{"1min":0.47499999999999998,"5min":0.39100000000000001,"15min":0.35899999999999999},"max":{"1min":2.2669999999999999,"5min":16.611999999999998,"15min":16.611999999999998},"last":0.52700000000000002}]}]},{"osd":4,"up_from":31,"seq":133143986442,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1574732,"kb_used_data":813312,"kb_used_omap":1,"kb_used_meta":761406,"kb_avail":92170420,"statfs":{"total":95995035648,"available":94382510080,"internally_reserved":0,"allocated":832831488,"data_stored":832424944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":779680197},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:53:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.252,"15min":1.2290000000000001},"min":{"1min":0.22900000000000001,"5min":0.216,"15min":0.20799999999999999},"max":{"1min":4.1150000000000002,"5min":7.6299999999999999,"15min":33.372},"last":0.45900000000000002},{"interface":"front","average":{"1min":1.167,"5min":1.3919999999999999,"15min":1.25},"min":{"1min":0.26400000000000001,"5min":0.26000000000000001,"15min":0.22700000000000001},"max":{"1min":10.130000000000001,"5min":10.130000000000001,"15min":33.323999999999998},"last":0.25700000000000001}]},{"osd":1,"last update":"Mon Feb 5 06:53:39 2024","interfaces":[{"interface":"back","average":{"1min":2.6859999999999999,"5min":1.5309999999999999,"15min":1.575},"min":{"1min":0.27900000000000003,"5min":0.254,"15min":0.21299999999999999},"max":{"1min":12.587,"5min":12.587,"15min":53.414000000000001},"last":1.7649999999999999},{"interface":"front","average":{"1min":1.446,"5min":1.3260000000000001,"15min":1.452},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23000000000000001},"max":{"1min":12.41,"5min":12.41,"15min":53.719999999999999},"last":3.9129999999999998}]},{"osd":2,"last update":"Mon Feb 5 06:53:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.298,"15min":1.0489999999999999},"min":{"1min":0.318,"5min":0.27300000000000002,"15min":0.26000000000000001},"max":{"1min":3.7440000000000002,"5min":7.5830000000000002,"15min":17.379999999999999},"last":0.78700000000000003},{"interface":"front","average":{"1min":1.7849999999999999,"5min":1.2729999999999999,"15min":1.2390000000000001},"min":{"1min":0.33900000000000002,"5min":0.28399999999999997,"15min":0.26000000000000001},"max":{"1min":4.7830000000000004,"5min":4.7830000000000004,"15min":18.077000000000002},"last":2.323}]},{"osd":3,"last update":"Mon Feb 5 06:53:39 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.3360000000000001,"15min":1.859},"min":{"1min":0.33700000000000002,"5min":0.28499999999999998,"15min":0.252},"max":{"1min":2.7559999999999998,"5min":7.3769999999999998,"15min":44.601999999999997},"last":2.1800000000000002},{"interface":"front","average":{"1min":2.0209999999999999,"5min":1.3839999999999999,"15min":1.865},"min":{"1min":0.307,"5min":0.307,"15min":0.29799999999999999},"max":{"1min":7.6619999999999999,"5min":7.6619999999999999,"15min":44.511000000000003},"last":1.744}]},{"osd":5,"last update":"Mon Feb 5 06:54:03 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.86299999999999999,"15min":0.80300000000000005},"min":{"1min":0.26000000000000001,"5min":0.25,"15min":0.24399999999999999},"max":{"1min":2.4409999999999998,"5min":4.649,"15min":5.6059999999999999},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.92000000000000004,"15min":0.79200000000000004},"min":{"1min":0.26300000000000001,"5min":0.255,"15min":0.255},"max":{"1min":3.2639999999999998,"5min":7.7290000000000001,"15min":7.7290000000000001},"last":0.56499999999999995}]},{"osd":6,"last update":"Mon Feb 5 06:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":1.022,"15min":0.93600000000000005},"min":{"1min":0.33800000000000002,"5min":0.29099999999999998,"15min":0.246},"max":{"1min":2.782,"5min":7.6950000000000003,"15min":11.868},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.746,"5min":0.93999999999999995,"15min":0.89100000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.29499999999999998},"max":{"1min":2.8039999999999998,"5min":4.0750000000000002,"15min":11.787000000000001},"last":0.44700000000000001}]},{"osd":7,"last update":"Mon Feb 5 06:53:45 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.85499999999999998,"15min":0.78000000000000003},"min":{"1min":0.33400000000000002,"5min":0.32700000000000001,"15min":0.27400000000000002},"max":{"1min":3.9980000000000002,"5min":4.1369999999999996,"15min":5.1500000000000004},"last":0.41199999999999998},{"interface":"front","average":{"1min":1.0880000000000001,"5min":0.94499999999999995,"15min":0.85899999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.28899999999999998},"max":{"1min":5.4809999999999999,"5min":5.4809999999999999,"15min":5.5890000000000004},"last":0.48899999999999999}]}]},{"osd":5,"up_from":40,"seq":171798692101,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1650548,"kb_used_data":802988,"kb_used_omap":1,"kb_used_meta":847486,"kb_avail":92094604,"statfs":{"total":95995035648,"available":94304874496,"internally_reserved":0,"allocated":822259712,"data_stored":821874377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":867826118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2,0,0,1,0,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:53:57 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":1.7330000000000001,"15min":1.603},"min":{"1min":0.29399999999999998,"5min":0.23100000000000001,"15min":0.223},"max":{"1min":2.6589999999999998,"5min":22.326000000000001,"15min":55.704999999999998},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.996,"5min":1.5509999999999999,"15min":1.6000000000000001},"min":{"1min":0.27300000000000002,"5min":0.26700000000000002,"15min":0.22600000000000001},"max":{"1min":2.7810000000000001,"5min":22.247,"15min":56.789000000000001},"last":0.48999999999999999}]},{"osd":1,"last update":"Mon Feb 5 06:53:57 2024","interfaces":[{"interface":"back","average":{"1min":5.0090000000000003,"5min":5.4539999999999997,"15min":2.8479999999999999},"min":{"1min":0.255,"5min":0.248,"15min":0.248},"max":{"1min":77.811999999999998,"5min":317.53399999999999,"15min":317.53399999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":5.3339999999999996,"5min":5.5670000000000002,"15min":2.867},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":77.373999999999995,"5min":317.80500000000001,"15min":317.80500000000001},"last":0.255}]},{"osd":2,"last update":"Mon Feb 5 06:53:57 2024","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.6399999999999999,"15min":1.3680000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.23000000000000001},"max":{"1min":7.0789999999999997,"5min":22.635999999999999,"15min":30.052},"last":0.32800000000000001},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.141,"15min":1.175},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":4.8890000000000002,"5min":7.0540000000000003,"15min":30.472999999999999},"last":0.64900000000000002}]},{"osd":3,"last update":"Mon Feb 5 06:53:57 2024","interfaces":[{"interface":"back","average":{"1min":3.1819999999999999,"5min":2.282,"15min":2.0289999999999999},"min":{"1min":0.308,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":43.442999999999998,"5min":43.442999999999998,"15min":62.508000000000003},"last":0.61899999999999999},{"interface":"front","average":{"1min":3.1379999999999999,"5min":2.117,"15min":2.1030000000000002},"min":{"1min":0.33300000000000002,"5min":0.29899999999999999,"15min":0.27400000000000002},"max":{"1min":43.545000000000002,"5min":43.545000000000002,"15min":62.460000000000001},"last":0.31}]},{"osd":4,"last update":"Mon Feb 5 06:53:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.82499999999999996,"15min":0.77200000000000002},"min":{"1min":0.25700000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":2.7669999999999999,"5min":9.3629999999999995,"15min":9.3629999999999995},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.89600000000000002,"15min":0.80100000000000005},"min":{"1min":0.26900000000000002,"5min":0.25600000000000001,"15min":0.252},"max":{"1min":2.4660000000000002,"5min":9.1839999999999993,"15min":9.1839999999999993},"last":0.40600000000000003}]},{"osd":6,"last update":"Mon Feb 5 06:53:35 2024","interfaces":[{"interface":"back","average":{"1min":3.8860000000000001,"5min":1.7,"15min":1.1299999999999999},"min":{"1min":0.35599999999999998,"5min":0.27700000000000002,"15min":0.23799999999999999},"max":{"1min":68.316999999999993,"5min":68.316999999999993,"15min":68.316999999999993},"last":0.46300000000000002},{"interface":"front","average":{"1min":4.1790000000000003,"5min":1.7949999999999999,"15min":1.196},"min":{"1min":0.371,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":70.352000000000004,"5min":70.352000000000004,"15min":70.352000000000004},"last":0.37}]},{"osd":7,"last update":"Mon Feb 5 06:53:57 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.95999999999999996,"15min":0.96899999999999997},"min":{"1min":0.33600000000000002,"5min":0.317,"15min":0.25600000000000001},"max":{"1min":3.6000000000000001,"5min":11.691000000000001,"15min":11.691000000000001},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.90500000000000003,"15min":0.91800000000000004},"min":{"1min":0.373,"5min":0.34399999999999997,"15min":0.27100000000000002},"max":{"1min":2.8250000000000002,"5min":4.9939999999999998,"15min":6.5839999999999996},"last":0.442}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1031602176,"data_stored":1031602176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1592328192,"data_stored":1592328192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1212616704,"data_stored":1212612632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1402273792,"data_stored":1402273792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":819662848,"data_stored":819658776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":819396608,"data_stored":819396608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1629818880,"data_stored":1629814808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1232863232,"data_stored":1232863232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T06:54:11.841 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T06:54:11.909 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:11.950 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:11 smithi028 ceph-mon[29442]: pgmap v993: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 911 op/s 2024-02-05T06:54:11.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:11 smithi028 ceph-mon[37984]: pgmap v993: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 911 op/s 2024-02-05T06:54:11.953 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:11.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:11.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:11.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:11 smithi167 ceph-mon[32774]: pgmap v993: 168 pgs: 1 active+clean+scrubbing+deep, 167 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 911 op/s 2024-02-05T06:54:12.055 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:12.097 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:12.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:12.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:12.199 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:12.229 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:12.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:12.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:12.332 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:12.349 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-02-05T06:54:12.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:12.391 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:12.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:12.491 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:12.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:12.548 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:12.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:12.627 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:54:12 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:54:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:54:12.649 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:12.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:12.682 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:12.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:12.782 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:12.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:12.831 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:12.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:12.932 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:12.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:12.966 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:12.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:13.066 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:13.095 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:13.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:13.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:13.098 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:54:13.197 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:13.244 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:13.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:13.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:13.333 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:54:13.346 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:13.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:13.417 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:13.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:13.518 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:13.573 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:13.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:13.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:13.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:13.719 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:13.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:13.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:13.821 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:13.855 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:13.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:13.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:13.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:13 smithi028 ceph-mon[29442]: pgmap v994: 168 pgs: 168 active+clean; 2.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-02-05T06:54:13.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:13 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:54:13.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:13 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:54:13.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:13 smithi028 ceph-mon[37984]: pgmap v994: 168 pgs: 168 active+clean; 2.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-02-05T06:54:13.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:13 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:54:13.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:13 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:54:13.957 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:13.963 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:54:13.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:13 smithi167 ceph-mon[32774]: pgmap v994: 168 pgs: 168 active+clean; 2.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-02-05T06:54:13.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:13 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:54:13.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:13 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:54:13.984 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:54:13 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:54:13.985 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:13.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:13.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:14.087 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:14.127 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:14.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:14.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:14.229 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:14.265 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:14.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:14.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:14.366 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:14.409 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:14.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:14.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:14.511 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:14.559 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:14.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:14.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:14.661 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:14.704 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:14.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:14.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:14.807 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:14.842 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:14.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:14.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:14.859 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:14 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:54:14.859 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:14 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:54:14.943 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:14.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:14 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:54:14.984 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:14.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:14.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:15.086 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:15.129 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:15.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:15.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:15.230 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:15.265 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:15.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:15.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:15.367 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:15.402 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:15.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:15.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:15.503 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:15.538 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:15.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:15.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:15.639 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:15.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:15.670 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:15.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:15.771 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:15.799 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:15.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:15.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:15.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:15.913 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:15 smithi028 ceph-mon[29442]: pgmap v995: 168 pgs: 168 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s 2024-02-05T06:54:15.918 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:15 smithi028 ceph-mon[37984]: pgmap v995: 168 pgs: 168 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s 2024-02-05T06:54:15.946 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:15.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:15.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:15.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:15 smithi167 ceph-mon[32774]: pgmap v995: 168 pgs: 168 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s 2024-02-05T06:54:16.048 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:16.077 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:16.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:16.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:16.179 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:16.210 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:16.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:16.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:16.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:16.351 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:16.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:16.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:16.453 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:16.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:16.485 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:16.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:16.586 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:16.619 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:16.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:16.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:16.721 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:16.784 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:16.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:16.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:16.885 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:16.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:16.917 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:16.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:17.018 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:17.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:17.066 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:17.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:17.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:17.204 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:17.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:17.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:17.240 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:54:17.244 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-05T06:54:17.306 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:17.335 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:17.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:17.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:17.437 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:17.477 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:17.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:17.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:17.579 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:17.613 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:17.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:17.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:17.715 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:17.747 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:17.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:17.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:17.848 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:17.890 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:17.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:17.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:17.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:17 smithi028 ceph-mon[29442]: pgmap v996: 168 pgs: 168 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-02-05T06:54:17.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:17 smithi028 ceph-mon[37984]: pgmap v996: 168 pgs: 168 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-02-05T06:54:17.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:17 smithi167 ceph-mon[32774]: pgmap v996: 168 pgs: 168 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-02-05T06:54:17.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:18.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:18.034 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:18.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:18.136 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:18.183 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:18.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:18.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:18.285 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:18.322 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:18.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:18.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:18.424 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:18.428 INFO:teuthology.orchestra.run.smithi028.stdout:ERROR: (22) Invalid argument 2024-02-05T06:54:18.428 INFO:teuthology.orchestra.run.smithi028.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:54:18.453 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:18.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:18.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:18.554 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:18.588 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:18.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:18.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:18.690 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:18.726 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:18.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:18.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:18.828 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:18.865 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:18.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:18.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:18.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:18 smithi028 ceph-mon[29442]: from='client.16158 v1:172.21.15.28:0/2294329119' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:54:18.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:18 smithi028 ceph-mon[37984]: from='client.16158 v1:172.21.15.28:0/2294329119' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:54:18.968 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:18.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:18 smithi167 ceph-mon[32774]: from='client.16158 v1:172.21.15.28:0/2294329119' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:54:18.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:18.999 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:18.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:19.100 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:19.143 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:19.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:19.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:19.198 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-02-05T06:54:19.246 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:19.284 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:19.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:19.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:19.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:19.433 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:19.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:19.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:19.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:19.579 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:19.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:19.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:19.639 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:54:19.681 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:19.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:19 smithi028 ceph-mon[29442]: pgmap v997: 168 pgs: 168 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s 2024-02-05T06:54:19.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:19 smithi028 ceph-mon[37984]: pgmap v997: 168 pgs: 168 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s 2024-02-05T06:54:19.716 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:19.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:19.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:19.818 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:19.850 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:19.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:19.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:19.952 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:19.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:19 smithi167 ceph-mon[32774]: pgmap v997: 168 pgs: 168 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.18k op/s 2024-02-05T06:54:19.993 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:19.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:19.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:20.095 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:20.132 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:20.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:20.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:20.234 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:20.235 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":996,"stamp":"2024-02-05T06:54:16.609891+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2836666740,"num_objects":43474,"num_object_clones":0,"num_object_copies":130422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43474,"num_whiteouts":0,"num_read":2380,"num_read_kb":2132,"num_write":211660,"num_write_kb":8125678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":411,"num_bytes_recovered":276079,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79944,"ondisk_log_size":79944,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":14733160,"kb_used_data":8276584,"kb_used_omap":12,"kb_used_meta":6456435,"kb_avail":735228056,"statfs":{"total":767960285184,"available":752873529344,"internally_reserved":0,"allocated":8475222016,"data_stored":8472395828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":6611390001},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,3,11,6,9,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-718340096,"num_objects":-10961,"num_object_clones":0,"num_object_copies":-32883,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10961,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10961,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.006064"},"pg_stats":[{"pgid":"11.3","version":"472'17192","reported_seq":17578,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:12.261983+0000","last_change":"2024-02-05T06:53:28.707185+0000","last_active":"2024-02-05T06:54:12.261983+0000","last_peered":"2024-02-05T06:54:12.261983+0000","last_clean":"2024-02-05T06:54:12.261983+0000","last_became_active":"2024-02-05T06:51:13.269833+0000","last_became_peered":"2024-02-05T06:51:13.269833+0000","last_unstale":"2024-02-05T06:54:12.261983+0000","last_undegraded":"2024-02-05T06:54:12.261983+0000","last_fullsized":"2024-02-05T06:54:12.261983+0000","mapping_epoch":418,"log_start":"456'12203","ondisk_log_start":"456'12203","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"462'14268","last_scrub_stamp":"2024-02-05T06:53:28.706913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:53:28.706913+0000","objects_scrubbed":5685,"log_size":4989,"log_dups_size":0,"ondisk_log_size":4989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:45.571701+0000","scrub_duration":1678,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176881664,"num_objects":2699,"num_object_clones":0,"num_object_copies":8097,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13165,"num_write_kb":507648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"465'35","reported_seq":759,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.691636+0000","last_change":"2024-02-05T06:52:45.303488+0000","last_active":"2024-02-05T06:54:08.691636+0000","last_peered":"2024-02-05T06:54:08.691636+0000","last_clean":"2024-02-05T06:54:08.691636+0000","last_became_active":"2024-02-05T06:45:09.796902+0000","last_became_peered":"2024-02-05T06:45:09.796902+0000","last_unstale":"2024-02-05T06:54:08.691636+0000","last_undegraded":"2024-02-05T06:54:08.691636+0000","last_fullsized":"2024-02-05T06:54:08.691636+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'25","last_scrub_stamp":"2024-02-05T06:52:45.303356+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-02-05T06:35:04.584657+0000","last_clean_scrub_stamp":"2024-02-05T06:52:45.303356+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":891,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.693309+0000","last_change":"2024-02-05T06:52:48.393899+0000","last_active":"2024-02-05T06:54:08.693309+0000","last_peered":"2024-02-05T06:54:08.693309+0000","last_clean":"2024-02-05T06:54:08.693309+0000","last_became_active":"2024-02-05T06:36:54.971304+0000","last_became_peered":"2024-02-05T06:36:54.971304+0000","last_unstale":"2024-02-05T06:54:08.693309+0000","last_undegraded":"2024-02-05T06:54:08.693309+0000","last_fullsized":"2024-02-05T06:54:08.693309+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:48.393628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:52:48.393628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:12.546963+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"249'13","reported_seq":1207,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.684777+0000","last_change":"2024-02-05T06:53:56.010563+0000","last_active":"2024-02-05T06:54:08.684777+0000","last_peered":"2024-02-05T06:54:08.684777+0000","last_clean":"2024-02-05T06:54:08.684777+0000","last_became_active":"2024-02-05T06:41:43.413139+0000","last_became_peered":"2024-02-05T06:41:43.413139+0000","last_unstale":"2024-02-05T06:54:08.684777+0000","last_undegraded":"2024-02-05T06:54:08.684777+0000","last_fullsized":"2024-02-05T06:54:08.684777+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'13","last_scrub_stamp":"2024-02-05T06:53:56.009983+0000","last_deep_scrub":"92'9","last_deep_scrub_stamp":"2024-02-05T06:34:52.283780+0000","last_clean_scrub_stamp":"2024-02-05T06:53:56.009983+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:08.067138+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":418,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.697936+0000","last_change":"2024-02-05T06:52:46.097175+0000","last_active":"2024-02-05T06:54:08.697936+0000","last_peered":"2024-02-05T06:54:08.697936+0000","last_clean":"2024-02-05T06:54:08.697936+0000","last_became_active":"2024-02-05T06:45:09.784229+0000","last_became_peered":"2024-02-05T06:45:09.784229+0000","last_unstale":"2024-02-05T06:54:08.697936+0000","last_undegraded":"2024-02-05T06:54:08.697936+0000","last_fullsized":"2024-02-05T06:54:08.697936+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:46.096965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:30.057099+0000","last_clean_scrub_stamp":"2024-02-05T06:52:46.096965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":1009,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.686266+0000","last_change":"2024-02-05T06:52:25.102060+0000","last_active":"2024-02-05T06:54:08.686266+0000","last_peered":"2024-02-05T06:54:08.686266+0000","last_clean":"2024-02-05T06:54:08.686266+0000","last_became_active":"2024-02-05T06:40:30.095202+0000","last_became_peered":"2024-02-05T06:40:30.095202+0000","last_unstale":"2024-02-05T06:54:08.686266+0000","last_undegraded":"2024-02-05T06:54:08.686266+0000","last_fullsized":"2024-02-05T06:54:08.686266+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:52:25.101936+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:41:50.706159+0000","last_clean_scrub_stamp":"2024-02-05T06:52:25.101936+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:13.579481+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.2","version":"472'13333","reported_seq":13540,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:12.342838+0000","last_change":"2024-02-05T06:52:53.157729+0000","last_active":"2024-02-05T06:54:12.342838+0000","last_peered":"2024-02-05T06:54:12.342838+0000","last_clean":"2024-02-05T06:54:12.342838+0000","last_became_active":"2024-02-05T06:51:13.275602+0000","last_became_peered":"2024-02-05T06:51:13.275602+0000","last_unstale":"2024-02-05T06:54:12.342838+0000","last_undegraded":"2024-02-05T06:54:12.342838+0000","last_fullsized":"2024-02-05T06:54:12.342838+0000","mapping_epoch":418,"log_start":"447'7500","ondisk_log_start":"447'7500","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'8067","last_scrub_stamp":"2024-02-05T06:52:53.155060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:52:53.155060+0000","objects_scrubbed":8022,"log_size":5833,"log_dups_size":0,"ondisk_log_size":5833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:15.955985+0000","scrub_duration":2122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178454528,"num_objects":2723,"num_object_clones":0,"num_object_copies":8169,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13333,"num_write_kb":513792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"465'53","reported_seq":785,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.683958+0000","last_change":"2024-02-05T06:52:40.987741+0000","last_active":"2024-02-05T06:54:08.683958+0000","last_peered":"2024-02-05T06:54:08.683958+0000","last_clean":"2024-02-05T06:54:08.683958+0000","last_became_active":"2024-02-05T06:45:09.782645+0000","last_became_peered":"2024-02-05T06:45:09.782645+0000","last_unstale":"2024-02-05T06:54:08.683958+0000","last_undegraded":"2024-02-05T06:54:08.683958+0000","last_fullsized":"2024-02-05T06:54:08.683958+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'39","last_scrub_stamp":"2024-02-05T06:52:40.987572+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-02-05T06:35:26.274944+0000","last_clean_scrub_stamp":"2024-02-05T06:52:40.987572+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:15.952684+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":920,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.691592+0000","last_change":"2024-02-05T06:53:45.072923+0000","last_active":"2024-02-05T06:54:08.691592+0000","last_peered":"2024-02-05T06:54:08.691592+0000","last_clean":"2024-02-05T06:54:08.691592+0000","last_became_active":"2024-02-05T06:39:13.240038+0000","last_became_peered":"2024-02-05T06:39:13.240038+0000","last_unstale":"2024-02-05T06:54:08.691592+0000","last_undegraded":"2024-02-05T06:54:08.691592+0000","last_fullsized":"2024-02-05T06:54:08.691592+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:45.072812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:53:45.072812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:02.736389+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":899,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.693538+0000","last_change":"2024-02-05T06:52:38.073950+0000","last_active":"2024-02-05T06:54:08.693538+0000","last_peered":"2024-02-05T06:54:08.693538+0000","last_clean":"2024-02-05T06:54:08.693538+0000","last_became_active":"2024-02-05T06:36:54.958082+0000","last_became_peered":"2024-02-05T06:36:54.958082+0000","last_unstale":"2024-02-05T06:54:08.693538+0000","last_undegraded":"2024-02-05T06:54:08.693538+0000","last_fullsized":"2024-02-05T06:54:08.693538+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:38.072995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:38.072995+0000","last_clean_scrub_stamp":"2024-02-05T06:52:38.072995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:12.747342+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":920,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.684144+0000","last_change":"2024-02-05T06:53:29.258337+0000","last_active":"2024-02-05T06:54:08.684144+0000","last_peered":"2024-02-05T06:54:08.684144+0000","last_clean":"2024-02-05T06:54:08.684144+0000","last_became_active":"2024-02-05T06:36:54.958805+0000","last_became_peered":"2024-02-05T06:36:54.958805+0000","last_unstale":"2024-02-05T06:54:08.684144+0000","last_undegraded":"2024-02-05T06:54:08.684144+0000","last_fullsized":"2024-02-05T06:54:08.684144+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:29.258153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:41.833719+0000","last_clean_scrub_stamp":"2024-02-05T06:53:29.258153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:38.792895+0000","scrub_duration":273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"83'128","reported_seq":929,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.689170+0000","last_change":"2024-02-05T06:53:55.916525+0000","last_active":"2024-02-05T06:54:08.689170+0000","last_peered":"2024-02-05T06:54:08.689170+0000","last_clean":"2024-02-05T06:54:08.689170+0000","last_became_active":"2024-02-05T06:41:46.053324+0000","last_became_peered":"2024-02-05T06:41:46.053324+0000","last_unstale":"2024-02-05T06:54:08.689170+0000","last_undegraded":"2024-02-05T06:54:08.689170+0000","last_fullsized":"2024-02-05T06:54:08.689170+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:53:55.916390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:53:55.916390+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:19.660693+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"83'128","reported_seq":1066,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.690552+0000","last_change":"2024-02-05T06:52:21.969321+0000","last_active":"2024-02-05T06:54:08.690552+0000","last_peered":"2024-02-05T06:54:08.690552+0000","last_clean":"2024-02-05T06:54:08.690552+0000","last_became_active":"2024-02-05T06:51:59.303575+0000","last_became_peered":"2024-02-05T06:51:59.303575+0000","last_unstale":"2024-02-05T06:54:08.690552+0000","last_undegraded":"2024-02-05T06:54:08.690552+0000","last_fullsized":"2024-02-05T06:54:08.690552+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:52:21.969067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:52:21.969067+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"472'13064","reported_seq":13276,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:12.284395+0000","last_change":"2024-02-05T06:53:32.058096+0000","last_active":"2024-02-05T06:54:12.284395+0000","last_peered":"2024-02-05T06:54:12.284395+0000","last_clean":"2024-02-05T06:54:12.284395+0000","last_became_active":"2024-02-05T06:51:13.273200+0000","last_became_peered":"2024-02-05T06:51:13.273200+0000","last_unstale":"2024-02-05T06:54:12.284395+0000","last_undegraded":"2024-02-05T06:54:12.284395+0000","last_fullsized":"2024-02-05T06:54:12.284395+0000","mapping_epoch":418,"log_start":"451'8100","ondisk_log_start":"451'8100","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"463'10360","last_scrub_stamp":"2024-02-05T06:53:32.057849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:53:32.057849+0000","objects_scrubbed":5494,"log_size":4964,"log_dups_size":0,"ondisk_log_size":4964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:01.054911+0000","scrub_duration":2042,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178651136,"num_objects":2726,"num_object_clones":0,"num_object_copies":8178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13064,"num_write_kb":505280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"465'32","reported_seq":769,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.690339+0000","last_change":"2024-02-05T06:53:31.987710+0000","last_active":"2024-02-05T06:54:08.690339+0000","last_peered":"2024-02-05T06:54:08.690339+0000","last_clean":"2024-02-05T06:54:08.690339+0000","last_became_active":"2024-02-05T06:45:09.774868+0000","last_became_peered":"2024-02-05T06:45:09.774868+0000","last_unstale":"2024-02-05T06:54:08.690339+0000","last_undegraded":"2024-02-05T06:54:08.690339+0000","last_fullsized":"2024-02-05T06:54:08.690339+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:53:31.987585+0000","last_deep_scrub":"115'16","last_deep_scrub_stamp":"2024-02-05T06:36:34.793243+0000","last_clean_scrub_stamp":"2024-02-05T06:53:31.987585+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:40.316102+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":615,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.689623+0000","last_change":"2024-02-05T06:53:56.618075+0000","last_active":"2024-02-05T06:54:08.689623+0000","last_peered":"2024-02-05T06:54:08.689623+0000","last_clean":"2024-02-05T06:54:08.689623+0000","last_became_active":"2024-02-05T06:40:27.086808+0000","last_became_peered":"2024-02-05T06:40:27.086808+0000","last_unstale":"2024-02-05T06:54:08.689623+0000","last_undegraded":"2024-02-05T06:54:08.689623+0000","last_fullsized":"2024-02-05T06:54:08.689623+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:56.617959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:41.860956+0000","last_clean_scrub_stamp":"2024-02-05T06:53:56.617959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:03.244501+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":503,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.689393+0000","last_change":"2024-02-05T06:52:19.870040+0000","last_active":"2024-02-05T06:54:08.689393+0000","last_peered":"2024-02-05T06:54:08.689393+0000","last_clean":"2024-02-05T06:54:08.689393+0000","last_became_active":"2024-02-05T06:41:43.417343+0000","last_became_peered":"2024-02-05T06:41:43.417343+0000","last_unstale":"2024-02-05T06:54:08.689393+0000","last_undegraded":"2024-02-05T06:54:08.689393+0000","last_fullsized":"2024-02-05T06:54:08.689393+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:19.869888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:42:55.610276+0000","last_clean_scrub_stamp":"2024-02-05T06:52:19.869888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:10.919416+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":885,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.691329+0000","last_change":"2024-02-05T06:54:02.502678+0000","last_active":"2024-02-05T06:54:08.691329+0000","last_peered":"2024-02-05T06:54:08.691329+0000","last_clean":"2024-02-05T06:54:08.691329+0000","last_became_active":"2024-02-05T06:41:43.410600+0000","last_became_peered":"2024-02-05T06:41:43.410600+0000","last_unstale":"2024-02-05T06:54:08.691329+0000","last_undegraded":"2024-02-05T06:54:08.691329+0000","last_fullsized":"2024-02-05T06:54:08.691329+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:54:02.502543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:58.479208+0000","last_clean_scrub_stamp":"2024-02-05T06:54:02.502543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:20.782865+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"472'17310","reported_seq":17706,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:12.973108+0000","last_change":"2024-02-05T06:53:34.387508+0000","last_active":"2024-02-05T06:54:12.973108+0000","last_peered":"2024-02-05T06:54:12.973108+0000","last_clean":"2024-02-05T06:54:12.973108+0000","last_became_active":"2024-02-05T06:51:13.273376+0000","last_became_peered":"2024-02-05T06:51:13.273376+0000","last_unstale":"2024-02-05T06:54:12.973108+0000","last_undegraded":"2024-02-05T06:54:12.973108+0000","last_fullsized":"2024-02-05T06:54:12.973108+0000","mapping_epoch":418,"log_start":"457'12903","ondisk_log_start":"457'12903","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'14766","last_scrub_stamp":"2024-02-05T06:53:34.387362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:53:34.387362+0000","objects_scrubbed":5240,"log_size":4407,"log_dups_size":0,"ondisk_log_size":4407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:38.451839+0000","scrub_duration":1825,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175046656,"num_objects":2671,"num_object_clones":0,"num_object_copies":8013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13081,"num_write_kb":504064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"465'46","reported_seq":712,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.678680+0000","last_change":"2024-02-05T06:52:38.220941+0000","last_active":"2024-02-05T06:54:08.678680+0000","last_peered":"2024-02-05T06:54:08.678680+0000","last_clean":"2024-02-05T06:54:08.678680+0000","last_became_active":"2024-02-05T06:39:13.257978+0000","last_became_peered":"2024-02-05T06:39:13.257978+0000","last_unstale":"2024-02-05T06:54:08.678680+0000","last_undegraded":"2024-02-05T06:54:08.678680+0000","last_fullsized":"2024-02-05T06:54:08.678680+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'34","last_scrub_stamp":"2024-02-05T06:52:38.220772+0000","last_deep_scrub":"295'34","last_deep_scrub_stamp":"2024-02-05T06:46:44.355563+0000","last_clean_scrub_stamp":"2024-02-05T06:52:38.220772+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":891,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.691160+0000","last_change":"2024-02-05T06:52:30.417398+0000","last_active":"2024-02-05T06:54:08.691160+0000","last_peered":"2024-02-05T06:54:08.691160+0000","last_clean":"2024-02-05T06:54:08.691160+0000","last_became_active":"2024-02-05T06:40:27.213439+0000","last_became_peered":"2024-02-05T06:40:27.213439+0000","last_unstale":"2024-02-05T06:54:08.691160+0000","last_undegraded":"2024-02-05T06:54:08.691160+0000","last_fullsized":"2024-02-05T06:54:08.691160+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:30.417237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:52.482566+0000","last_clean_scrub_stamp":"2024-02-05T06:52:30.417237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:12.546791+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"319'13","reported_seq":811,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:09.838855+0000","last_change":"2024-02-05T06:52:49.830770+0000","last_active":"2024-02-05T06:54:09.838855+0000","last_peered":"2024-02-05T06:54:09.838855+0000","last_clean":"2024-02-05T06:54:09.838855+0000","last_became_active":"2024-02-05T06:45:09.766220+0000","last_became_peered":"2024-02-05T06:45:09.766220+0000","last_unstale":"2024-02-05T06:54:09.838855+0000","last_undegraded":"2024-02-05T06:54:09.838855+0000","last_fullsized":"2024-02-05T06:54:09.838855+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'13","last_scrub_stamp":"2024-02-05T06:52:49.830615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:52:49.830615+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:13.660513+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":512,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.690702+0000","last_change":"2024-02-05T06:53:47.006909+0000","last_active":"2024-02-05T06:54:08.690702+0000","last_peered":"2024-02-05T06:54:08.690702+0000","last_clean":"2024-02-05T06:54:08.690702+0000","last_became_active":"2024-02-05T06:41:43.418066+0000","last_became_peered":"2024-02-05T06:41:43.418066+0000","last_unstale":"2024-02-05T06:54:08.690702+0000","last_undegraded":"2024-02-05T06:54:08.690702+0000","last_fullsized":"2024-02-05T06:54:08.690702+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:47.006807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:48:34.623889+0000","last_clean_scrub_stamp":"2024-02-05T06:53:47.006807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:54.416716+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":925,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.689380+0000","last_change":"2024-02-05T06:53:44.829783+0000","last_active":"2024-02-05T06:54:08.689380+0000","last_peered":"2024-02-05T06:54:08.689380+0000","last_clean":"2024-02-05T06:54:08.689380+0000","last_became_active":"2024-02-05T06:41:46.053993+0000","last_became_peered":"2024-02-05T06:41:46.053993+0000","last_unstale":"2024-02-05T06:54:08.689380+0000","last_undegraded":"2024-02-05T06:54:08.689380+0000","last_fullsized":"2024-02-05T06:54:08.689380+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:53:44.829662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:53:44.829662+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:49.174258+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"472'13434","reported_seq":13614,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:11.840540+0000","last_change":"2024-02-05T06:53:36.664242+0000","last_active":"2024-02-05T06:54:11.840540+0000","last_peered":"2024-02-05T06:54:11.840540+0000","last_clean":"2024-02-05T06:54:11.840540+0000","last_became_active":"2024-02-05T06:51:13.274262+0000","last_became_peered":"2024-02-05T06:51:13.274262+0000","last_unstale":"2024-02-05T06:54:11.840540+0000","last_undegraded":"2024-02-05T06:54:11.840540+0000","last_fullsized":"2024-02-05T06:54:11.840540+0000","mapping_epoch":418,"log_start":"454'8800","ondisk_log_start":"454'8800","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'10930","last_scrub_stamp":"2024-02-05T06:53:36.664069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:53:36.664069+0000","objects_scrubbed":5464,"log_size":4634,"log_dups_size":0,"ondisk_log_size":4634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:43.016542+0000","scrub_duration":1595,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189661184,"num_objects":2894,"num_object_clones":0,"num_object_copies":8682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13434,"num_write_kb":522496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":551,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.692521+0000","last_change":"2024-02-05T06:53:50.578263+0000","last_active":"2024-02-05T06:54:08.692521+0000","last_peered":"2024-02-05T06:54:08.692521+0000","last_clean":"2024-02-05T06:54:08.692521+0000","last_became_active":"2024-02-05T06:45:09.778609+0000","last_became_peered":"2024-02-05T06:45:09.778609+0000","last_unstale":"2024-02-05T06:54:08.692521+0000","last_undegraded":"2024-02-05T06:54:08.692521+0000","last_fullsized":"2024-02-05T06:54:08.692521+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:50.578152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:53:50.578152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:55.388149+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"465'50","reported_seq":992,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.691239+0000","last_change":"2024-02-05T06:51:40.451314+0000","last_active":"2024-02-05T06:54:08.691239+0000","last_peered":"2024-02-05T06:54:08.691239+0000","last_clean":"2024-02-05T06:54:08.691239+0000","last_became_active":"2024-02-05T06:40:27.277747+0000","last_became_peered":"2024-02-05T06:40:27.277747+0000","last_unstale":"2024-02-05T06:54:08.691239+0000","last_undegraded":"2024-02-05T06:54:08.691239+0000","last_fullsized":"2024-02-05T06:54:08.691239+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"424'38","last_scrub_stamp":"2024-02-05T06:51:40.451087+0000","last_deep_scrub":"424'38","last_deep_scrub_stamp":"2024-02-05T06:51:40.451087+0000","last_clean_scrub_stamp":"2024-02-05T06:51:40.451087+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:12.546676+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":897,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.692406+0000","last_change":"2024-02-05T06:52:53.475658+0000","last_active":"2024-02-05T06:54:08.692406+0000","last_peered":"2024-02-05T06:54:08.692406+0000","last_clean":"2024-02-05T06:54:08.692406+0000","last_became_active":"2024-02-05T06:40:27.093739+0000","last_became_peered":"2024-02-05T06:40:27.093739+0000","last_unstale":"2024-02-05T06:54:08.692406+0000","last_undegraded":"2024-02-05T06:54:08.692406+0000","last_fullsized":"2024-02-05T06:54:08.692406+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:53.475540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:53.475540+0000","last_clean_scrub_stamp":"2024-02-05T06:52:53.475540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:23.045094+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":410,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.696933+0000","last_change":"2024-02-05T06:53:28.389799+0000","last_active":"2024-02-05T06:54:08.696933+0000","last_peered":"2024-02-05T06:54:08.696933+0000","last_clean":"2024-02-05T06:54:08.696933+0000","last_became_active":"2024-02-05T06:45:09.780390+0000","last_became_peered":"2024-02-05T06:45:09.780390+0000","last_unstale":"2024-02-05T06:54:08.696933+0000","last_undegraded":"2024-02-05T06:54:08.696933+0000","last_fullsized":"2024-02-05T06:54:08.696933+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:28.389689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:45:49.145117+0000","last_clean_scrub_stamp":"2024-02-05T06:53:28.389689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:51.789095+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":669,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.683751+0000","last_change":"2024-02-05T06:53:40.036388+0000","last_active":"2024-02-05T06:54:08.683751+0000","last_peered":"2024-02-05T06:54:08.683751+0000","last_clean":"2024-02-05T06:54:08.683751+0000","last_became_active":"2024-02-05T06:51:54.322200+0000","last_became_peered":"2024-02-05T06:51:54.322200+0000","last_unstale":"2024-02-05T06:54:08.683751+0000","last_undegraded":"2024-02-05T06:54:08.683751+0000","last_fullsized":"2024-02-05T06:54:08.683751+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:53:40.036268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:53:40.036268+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:59.840080+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"472'13380","reported_seq":13600,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:12.962042+0000","last_change":"2024-02-05T06:53:36.641829+0000","last_active":"2024-02-05T06:54:12.962042+0000","last_peered":"2024-02-05T06:54:12.962042+0000","last_clean":"2024-02-05T06:54:12.962042+0000","last_became_active":"2024-02-05T06:51:13.267644+0000","last_became_peered":"2024-02-05T06:51:13.267644+0000","last_unstale":"2024-02-05T06:54:12.962042+0000","last_undegraded":"2024-02-05T06:54:12.962042+0000","last_fullsized":"2024-02-05T06:54:12.962042+0000","mapping_epoch":418,"log_start":"456'9000","ondisk_log_start":"456'9000","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'10815","last_scrub_stamp":"2024-02-05T06:53:36.641665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:53:36.641665+0000","objects_scrubbed":5282,"log_size":4380,"log_dups_size":0,"ondisk_log_size":4380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:40.621164+0000","scrub_duration":1905,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174194688,"num_objects":2658,"num_object_clones":0,"num_object_copies":7974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13380,"num_write_kb":513216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"326'11","reported_seq":1152,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.691391+0000","last_change":"2024-02-05T06:53:59.536292+0000","last_active":"2024-02-05T06:54:08.691391+0000","last_peered":"2024-02-05T06:54:08.691391+0000","last_clean":"2024-02-05T06:54:08.691391+0000","last_became_active":"2024-02-05T06:36:54.954819+0000","last_became_peered":"2024-02-05T06:36:54.954819+0000","last_unstale":"2024-02-05T06:54:08.691391+0000","last_undegraded":"2024-02-05T06:54:08.691391+0000","last_fullsized":"2024-02-05T06:54:08.691391+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'11","last_scrub_stamp":"2024-02-05T06:53:59.535905+0000","last_deep_scrub":"326'11","last_deep_scrub_stamp":"2024-02-05T06:45:56.212177+0000","last_clean_scrub_stamp":"2024-02-05T06:53:59.535905+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:19.671764+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"465'42","reported_seq":835,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.688354+0000","last_change":"2024-02-05T06:54:02.913296+0000","last_active":"2024-02-05T06:54:08.688354+0000","last_peered":"2024-02-05T06:54:08.688354+0000","last_clean":"2024-02-05T06:54:08.688354+0000","last_became_active":"2024-02-05T06:45:09.764696+0000","last_became_peered":"2024-02-05T06:45:09.764696+0000","last_unstale":"2024-02-05T06:54:08.688354+0000","last_undegraded":"2024-02-05T06:54:08.688354+0000","last_fullsized":"2024-02-05T06:54:08.688354+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'42","last_scrub_stamp":"2024-02-05T06:54:02.913160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:54:02.913160+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:26.597570+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"95'2","reported_seq":910,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.685197+0000","last_change":"2024-02-05T06:53:41.525189+0000","last_active":"2024-02-05T06:54:08.685197+0000","last_peered":"2024-02-05T06:54:08.685197+0000","last_clean":"2024-02-05T06:54:08.685197+0000","last_became_active":"2024-02-05T06:36:54.975093+0000","last_became_peered":"2024-02-05T06:36:54.975093+0000","last_unstale":"2024-02-05T06:54:08.685197+0000","last_undegraded":"2024-02-05T06:54:08.685197+0000","last_fullsized":"2024-02-05T06:54:08.685197+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-05T06:53:41.524836+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-05T06:35:25.616708+0000","last_clean_scrub_stamp":"2024-02-05T06:53:41.524836+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:59.895714+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":596,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.686437+0000","last_change":"2024-02-05T06:53:53.881450+0000","last_active":"2024-02-05T06:54:08.686437+0000","last_peered":"2024-02-05T06:54:08.686437+0000","last_clean":"2024-02-05T06:54:08.686437+0000","last_became_active":"2024-02-05T06:40:27.099168+0000","last_became_peered":"2024-02-05T06:40:27.099168+0000","last_unstale":"2024-02-05T06:54:08.686437+0000","last_undegraded":"2024-02-05T06:54:08.686437+0000","last_fullsized":"2024-02-05T06:54:08.686437+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:53.881337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:53:53.881337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:10.513794+0000","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":1093,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.691185+0000","last_change":"2024-02-05T06:52:18.376581+0000","last_active":"2024-02-05T06:54:08.691185+0000","last_peered":"2024-02-05T06:54:08.691185+0000","last_clean":"2024-02-05T06:54:08.691185+0000","last_became_active":"2024-02-05T06:51:54.332037+0000","last_became_peered":"2024-02-05T06:51:54.332037+0000","last_unstale":"2024-02-05T06:54:08.691185+0000","last_undegraded":"2024-02-05T06:54:08.691185+0000","last_fullsized":"2024-02-05T06:54:08.691185+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:52:18.376463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:52:18.376463+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:12.546735+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"472'13038","reported_seq":13268,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:12.296130+0000","last_change":"2024-02-05T06:53:30.282702+0000","last_active":"2024-02-05T06:54:12.296130+0000","last_peered":"2024-02-05T06:54:12.296130+0000","last_clean":"2024-02-05T06:54:12.296130+0000","last_became_active":"2024-02-05T06:51:13.272662+0000","last_became_peered":"2024-02-05T06:51:13.272662+0000","last_unstale":"2024-02-05T06:54:12.296130+0000","last_undegraded":"2024-02-05T06:54:12.296130+0000","last_fullsized":"2024-02-05T06:54:12.296130+0000","mapping_epoch":418,"log_start":"447'7200","ondisk_log_start":"447'7200","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"463'10228","last_scrub_stamp":"2024-02-05T06:53:30.282379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:53:30.282379+0000","objects_scrubbed":5646,"log_size":5838,"log_dups_size":0,"ondisk_log_size":5838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:40.107663+0000","scrub_duration":2319,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180944920,"num_objects":2762,"num_object_clones":0,"num_object_copies":8286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2762,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13038,"num_write_kb":505537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"335'11","reported_seq":1159,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:09.839754+0000","last_change":"2024-02-05T06:52:40.188002+0000","last_active":"2024-02-05T06:54:09.839754+0000","last_peered":"2024-02-05T06:54:09.839754+0000","last_clean":"2024-02-05T06:54:09.839754+0000","last_became_active":"2024-02-05T06:41:43.423638+0000","last_became_peered":"2024-02-05T06:41:43.423638+0000","last_unstale":"2024-02-05T06:54:09.839754+0000","last_undegraded":"2024-02-05T06:54:09.839754+0000","last_fullsized":"2024-02-05T06:54:09.839754+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"335'11","last_scrub_stamp":"2024-02-05T06:52:40.187864+0000","last_deep_scrub":"335'11","last_deep_scrub_stamp":"2024-02-05T06:52:40.187864+0000","last_clean_scrub_stamp":"2024-02-05T06:52:40.187864+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"465'35","reported_seq":1013,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.690788+0000","last_change":"2024-02-05T06:54:00.633400+0000","last_active":"2024-02-05T06:54:08.690788+0000","last_peered":"2024-02-05T06:54:08.690788+0000","last_clean":"2024-02-05T06:54:08.690788+0000","last_became_active":"2024-02-05T06:41:43.421030+0000","last_became_peered":"2024-02-05T06:41:43.421030+0000","last_unstale":"2024-02-05T06:54:08.690788+0000","last_undegraded":"2024-02-05T06:54:08.690788+0000","last_fullsized":"2024-02-05T06:54:08.690788+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'35","last_scrub_stamp":"2024-02-05T06:54:00.633175+0000","last_deep_scrub":"424'25","last_deep_scrub_stamp":"2024-02-05T06:51:39.813235+0000","last_clean_scrub_stamp":"2024-02-05T06:54:00.633175+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:18.151735+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":503,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.690784+0000","last_change":"2024-02-05T06:53:32.828121+0000","last_active":"2024-02-05T06:54:08.690784+0000","last_peered":"2024-02-05T06:54:08.690784+0000","last_clean":"2024-02-05T06:54:08.690784+0000","last_became_active":"2024-02-05T06:41:43.411732+0000","last_became_peered":"2024-02-05T06:41:43.411732+0000","last_unstale":"2024-02-05T06:54:08.690784+0000","last_undegraded":"2024-02-05T06:54:08.690784+0000","last_fullsized":"2024-02-05T06:54:08.690784+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:32.828031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:28.353956+0000","last_clean_scrub_stamp":"2024-02-05T06:53:32.828031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:39.494277+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":877,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.678610+0000","last_change":"2024-02-05T06:53:52.550097+0000","last_active":"2024-02-05T06:54:08.678610+0000","last_peered":"2024-02-05T06:54:08.678610+0000","last_clean":"2024-02-05T06:54:08.678610+0000","last_became_active":"2024-02-05T06:33:35.600485+0000","last_became_peered":"2024-02-05T06:33:35.600485+0000","last_unstale":"2024-02-05T06:54:08.678610+0000","last_undegraded":"2024-02-05T06:54:08.678610+0000","last_fullsized":"2024-02-05T06:54:08.678610+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:52.549976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:53:52.549976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:17.433853+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":1098,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.690667+0000","last_change":"2024-02-05T06:52:19.674161+0000","last_active":"2024-02-05T06:54:08.690667+0000","last_peered":"2024-02-05T06:54:08.690667+0000","last_clean":"2024-02-05T06:54:08.690667+0000","last_became_active":"2024-02-05T06:51:54.332371+0000","last_became_peered":"2024-02-05T06:51:54.332371+0000","last_unstale":"2024-02-05T06:54:08.690667+0000","last_undegraded":"2024-02-05T06:54:08.690667+0000","last_fullsized":"2024-02-05T06:54:08.690667+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:52:19.674004+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:48:25.725892+0000","last_clean_scrub_stamp":"2024-02-05T06:52:19.674004+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:11.067973+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"472'12881","reported_seq":13085,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:12.942844+0000","last_change":"2024-02-05T06:52:53.084302+0000","last_active":"2024-02-05T06:54:12.942844+0000","last_peered":"2024-02-05T06:54:12.942844+0000","last_clean":"2024-02-05T06:54:12.942844+0000","last_became_active":"2024-02-05T06:51:13.269283+0000","last_became_peered":"2024-02-05T06:51:13.269283+0000","last_unstale":"2024-02-05T06:54:12.942844+0000","last_undegraded":"2024-02-05T06:54:12.942844+0000","last_fullsized":"2024-02-05T06:54:12.942844+0000","mapping_epoch":418,"log_start":"456'8500","ondisk_log_start":"456'8500","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'7755","last_scrub_stamp":"2024-02-05T06:52:53.084023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:52:53.084023+0000","objects_scrubbed":7733,"log_size":4381,"log_dups_size":0,"ondisk_log_size":4381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:15.548872+0000","scrub_duration":1955,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169803776,"num_objects":2591,"num_object_clones":0,"num_object_copies":7773,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2591,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12881,"num_write_kb":495104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":906,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.691566+0000","last_change":"2024-02-05T06:52:20.642047+0000","last_active":"2024-02-05T06:54:08.691566+0000","last_peered":"2024-02-05T06:54:08.691566+0000","last_clean":"2024-02-05T06:54:08.691566+0000","last_became_active":"2024-02-05T06:41:43.411550+0000","last_became_peered":"2024-02-05T06:41:43.411550+0000","last_unstale":"2024-02-05T06:54:08.691566+0000","last_undegraded":"2024-02-05T06:54:08.691566+0000","last_fullsized":"2024-02-05T06:54:08.691566+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:20.641923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:52:20.641923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:11.067905+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"465'39","reported_seq":1046,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.679770+0000","last_change":"2024-02-05T06:52:50.055893+0000","last_active":"2024-02-05T06:54:08.679770+0000","last_peered":"2024-02-05T06:54:08.679770+0000","last_clean":"2024-02-05T06:54:08.679770+0000","last_became_active":"2024-02-05T06:41:43.409664+0000","last_became_peered":"2024-02-05T06:41:43.409664+0000","last_unstale":"2024-02-05T06:54:08.679770+0000","last_undegraded":"2024-02-05T06:54:08.679770+0000","last_fullsized":"2024-02-05T06:54:08.679770+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'29","last_scrub_stamp":"2024-02-05T06:52:50.055600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:52:50.055600+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"445'2","reported_seq":922,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.693439+0000","last_change":"2024-02-05T06:52:37.034737+0000","last_active":"2024-02-05T06:54:08.693439+0000","last_peered":"2024-02-05T06:54:08.693439+0000","last_clean":"2024-02-05T06:54:08.693439+0000","last_became_active":"2024-02-05T06:41:43.411404+0000","last_became_peered":"2024-02-05T06:41:43.411404+0000","last_unstale":"2024-02-05T06:54:08.693439+0000","last_undegraded":"2024-02-05T06:54:08.693439+0000","last_fullsized":"2024-02-05T06:54:08.693439+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'2","last_scrub_stamp":"2024-02-05T06:52:37.034607+0000","last_deep_scrub":"445'2","last_deep_scrub_stamp":"2024-02-05T06:52:37.034607+0000","last_clean_scrub_stamp":"2024-02-05T06:52:37.034607+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:12.747494+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":883,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.690005+0000","last_change":"2024-02-05T06:53:38.739502+0000","last_active":"2024-02-05T06:54:08.690005+0000","last_peered":"2024-02-05T06:54:08.690005+0000","last_clean":"2024-02-05T06:54:08.690005+0000","last_became_active":"2024-02-05T06:33:35.664858+0000","last_became_peered":"2024-02-05T06:33:35.664858+0000","last_unstale":"2024-02-05T06:54:08.690005+0000","last_undegraded":"2024-02-05T06:54:08.690005+0000","last_fullsized":"2024-02-05T06:54:08.690005+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:38.739377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:53:38.739377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:50.477820+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":1079,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.691218+0000","last_change":"2024-02-05T06:52:18.642202+0000","last_active":"2024-02-05T06:54:08.691218+0000","last_peered":"2024-02-05T06:54:08.691218+0000","last_clean":"2024-02-05T06:54:08.691218+0000","last_became_active":"2024-02-05T06:51:54.330806+0000","last_became_peered":"2024-02-05T06:51:54.330806+0000","last_unstale":"2024-02-05T06:54:08.691218+0000","last_undegraded":"2024-02-05T06:54:08.691218+0000","last_fullsized":"2024-02-05T06:54:08.691218+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:52:18.642076+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:38.778568+0000","last_clean_scrub_stamp":"2024-02-05T06:52:18.642076+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:11.068021+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"472'13187","reported_seq":13306,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:12.973762+0000","last_change":"2024-02-05T06:51:13.271458+0000","last_active":"2024-02-05T06:54:12.973762+0000","last_peered":"2024-02-05T06:54:12.973762+0000","last_clean":"2024-02-05T06:54:12.973762+0000","last_became_active":"2024-02-05T06:51:13.270994+0000","last_became_peered":"2024-02-05T06:51:13.270994+0000","last_unstale":"2024-02-05T06:54:12.973762+0000","last_undegraded":"2024-02-05T06:54:12.973762+0000","last_fullsized":"2024-02-05T06:54:12.973762+0000","mapping_epoch":418,"log_start":"456'8900","ondisk_log_start":"456'8900","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:51:12.249449+0000","objects_scrubbed":0,"log_size":4287,"log_dups_size":0,"ondisk_log_size":4287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180420608,"num_objects":2753,"num_object_clones":0,"num_object_copies":8259,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2753,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13187,"num_write_kb":510080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":503,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.689556+0000","last_change":"2024-02-05T06:52:50.894323+0000","last_active":"2024-02-05T06:54:08.689556+0000","last_peered":"2024-02-05T06:54:08.689556+0000","last_clean":"2024-02-05T06:54:08.689556+0000","last_became_active":"2024-02-05T06:41:43.417751+0000","last_became_peered":"2024-02-05T06:41:43.417751+0000","last_unstale":"2024-02-05T06:54:08.689556+0000","last_undegraded":"2024-02-05T06:54:08.689556+0000","last_fullsized":"2024-02-05T06:54:08.689556+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:50.894195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:52:50.894195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:10.919771+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"465'36","reported_seq":938,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.685158+0000","last_change":"2024-02-05T06:52:52.080789+0000","last_active":"2024-02-05T06:54:08.685158+0000","last_peered":"2024-02-05T06:54:08.685158+0000","last_clean":"2024-02-05T06:54:08.685158+0000","last_became_active":"2024-02-05T06:33:31.564604+0000","last_became_peered":"2024-02-05T06:33:31.564604+0000","last_unstale":"2024-02-05T06:54:08.685158+0000","last_undegraded":"2024-02-05T06:54:08.685158+0000","last_fullsized":"2024-02-05T06:54:08.685158+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:52:52.080633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:52:52.080633+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"355'6","reported_seq":1421,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:10.492154+0000","last_change":"2024-02-05T06:52:35.156214+0000","last_active":"2024-02-05T06:54:10.492154+0000","last_peered":"2024-02-05T06:54:10.492154+0000","last_clean":"2024-02-05T06:54:10.492154+0000","last_became_active":"2024-02-05T06:40:27.071509+0000","last_became_peered":"2024-02-05T06:40:27.071509+0000","last_unstale":"2024-02-05T06:54:10.492154+0000","last_undegraded":"2024-02-05T06:54:10.492154+0000","last_fullsized":"2024-02-05T06:54:10.492154+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'6","last_scrub_stamp":"2024-02-05T06:52:35.156085+0000","last_deep_scrub":"355'6","last_deep_scrub_stamp":"2024-02-05T06:47:46.642454+0000","last_clean_scrub_stamp":"2024-02-05T06:52:35.156085+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:13.579367+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00074151600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":830,"num_read_kb":825,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":901,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.678297+0000","last_change":"2024-02-05T06:53:33.469434+0000","last_active":"2024-02-05T06:54:08.678297+0000","last_peered":"2024-02-05T06:54:08.678297+0000","last_clean":"2024-02-05T06:54:08.678297+0000","last_became_active":"2024-02-05T06:36:54.972264+0000","last_became_peered":"2024-02-05T06:36:54.972264+0000","last_unstale":"2024-02-05T06:54:08.678297+0000","last_undegraded":"2024-02-05T06:54:08.678297+0000","last_fullsized":"2024-02-05T06:54:08.678297+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:33.469051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:53:33.469051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:45.519225+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":903,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.693119+0000","last_change":"2024-02-05T06:53:47.688370+0000","last_active":"2024-02-05T06:54:08.693119+0000","last_peered":"2024-02-05T06:54:08.693119+0000","last_clean":"2024-02-05T06:54:08.693119+0000","last_became_active":"2024-02-05T06:41:43.410910+0000","last_became_peered":"2024-02-05T06:41:43.410910+0000","last_unstale":"2024-02-05T06:54:08.693119+0000","last_undegraded":"2024-02-05T06:54:08.693119+0000","last_fullsized":"2024-02-05T06:54:08.693119+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:47.688112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:53:47.688112+0000","last_clean_scrub_stamp":"2024-02-05T06:53:47.688112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:00.832816+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":1058,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.693737+0000","last_change":"2024-02-05T06:52:22.946733+0000","last_active":"2024-02-05T06:54:08.693737+0000","last_peered":"2024-02-05T06:54:08.693737+0000","last_clean":"2024-02-05T06:54:08.693737+0000","last_became_active":"2024-02-05T06:51:54.323320+0000","last_became_peered":"2024-02-05T06:51:54.323320+0000","last_unstale":"2024-02-05T06:54:08.693737+0000","last_undegraded":"2024-02-05T06:54:08.693737+0000","last_fullsized":"2024-02-05T06:54:08.693737+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:52:22.946483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:52:22.946483+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:12.747372+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"472'13298","reported_seq":13417,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:12.955579+0000","last_change":"2024-02-05T06:51:13.274707+0000","last_active":"2024-02-05T06:54:12.955579+0000","last_peered":"2024-02-05T06:54:12.955579+0000","last_clean":"2024-02-05T06:54:12.955579+0000","last_became_active":"2024-02-05T06:51:13.274381+0000","last_became_peered":"2024-02-05T06:51:13.274381+0000","last_unstale":"2024-02-05T06:54:12.955579+0000","last_undegraded":"2024-02-05T06:54:12.955579+0000","last_fullsized":"2024-02-05T06:54:12.955579+0000","mapping_epoch":418,"log_start":"457'9000","ondisk_log_start":"457'9000","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:51:12.249449+0000","objects_scrubbed":0,"log_size":4298,"log_dups_size":0,"ondisk_log_size":4298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":177209344,"num_objects":2704,"num_object_clones":0,"num_object_copies":8112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13298,"num_write_kb":512064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":610,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.686791+0000","last_change":"2024-02-05T06:54:02.593564+0000","last_active":"2024-02-05T06:54:08.686791+0000","last_peered":"2024-02-05T06:54:08.686791+0000","last_clean":"2024-02-05T06:54:08.686791+0000","last_became_active":"2024-02-05T06:41:43.418198+0000","last_became_peered":"2024-02-05T06:41:43.418198+0000","last_unstale":"2024-02-05T06:54:08.686791+0000","last_undegraded":"2024-02-05T06:54:08.686791+0000","last_fullsized":"2024-02-05T06:54:08.686791+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:54:02.593328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:54:02.593328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:12.160360+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"465'52","reported_seq":1311,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.692191+0000","last_change":"2024-02-05T06:52:39.574360+0000","last_active":"2024-02-05T06:54:08.692191+0000","last_peered":"2024-02-05T06:54:08.692191+0000","last_clean":"2024-02-05T06:54:08.692191+0000","last_became_active":"2024-02-05T06:40:27.061111+0000","last_became_peered":"2024-02-05T06:40:27.061111+0000","last_unstale":"2024-02-05T06:54:08.692191+0000","last_undegraded":"2024-02-05T06:54:08.692191+0000","last_fullsized":"2024-02-05T06:54:08.692191+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'40","last_scrub_stamp":"2024-02-05T06:52:39.574272+0000","last_deep_scrub":"413'40","last_deep_scrub_stamp":"2024-02-05T06:50:56.446985+0000","last_clean_scrub_stamp":"2024-02-05T06:52:39.574272+0000","objects_scrubbed":8,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:11.068057+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":891,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.696612+0000","last_change":"2024-02-05T06:53:38.592980+0000","last_active":"2024-02-05T06:54:08.696612+0000","last_peered":"2024-02-05T06:54:08.696612+0000","last_clean":"2024-02-05T06:54:08.696612+0000","last_became_active":"2024-02-05T06:41:43.411335+0000","last_became_peered":"2024-02-05T06:41:43.411335+0000","last_unstale":"2024-02-05T06:54:08.696612+0000","last_undegraded":"2024-02-05T06:54:08.696612+0000","last_fullsized":"2024-02-05T06:54:08.696612+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:38.592868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:53:38.592868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:54.703288+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1033,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.686328+0000","last_change":"2024-02-05T06:54:03.548187+0000","last_active":"2024-02-05T06:54:08.686328+0000","last_peered":"2024-02-05T06:54:08.686328+0000","last_clean":"2024-02-05T06:54:08.686328+0000","last_became_active":"2024-02-05T06:51:54.326585+0000","last_became_peered":"2024-02-05T06:51:54.326585+0000","last_unstale":"2024-02-05T06:54:08.686328+0000","last_undegraded":"2024-02-05T06:54:08.686328+0000","last_fullsized":"2024-02-05T06:54:08.686328+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:54:03.548068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:54:03.548068+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:18.630524+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":504,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.689474+0000","last_change":"2024-02-05T06:51:40.969197+0000","last_active":"2024-02-05T06:54:08.689474+0000","last_peered":"2024-02-05T06:54:08.689474+0000","last_clean":"2024-02-05T06:54:08.689474+0000","last_became_active":"2024-02-05T06:41:43.408773+0000","last_became_peered":"2024-02-05T06:41:43.408773+0000","last_unstale":"2024-02-05T06:54:08.689474+0000","last_undegraded":"2024-02-05T06:54:08.689474+0000","last_fullsized":"2024-02-05T06:54:08.689474+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:40.969057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:51:40.969057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:10.919391+0000","scrub_duration":137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"472'13190","reported_seq":13309,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:12.970486+0000","last_change":"2024-02-05T06:51:13.274836+0000","last_active":"2024-02-05T06:54:12.970486+0000","last_peered":"2024-02-05T06:54:12.970486+0000","last_clean":"2024-02-05T06:54:12.970486+0000","last_became_active":"2024-02-05T06:51:13.272948+0000","last_became_peered":"2024-02-05T06:51:13.272948+0000","last_unstale":"2024-02-05T06:54:12.970486+0000","last_undegraded":"2024-02-05T06:54:12.970486+0000","last_fullsized":"2024-02-05T06:54:12.970486+0000","mapping_epoch":418,"log_start":"457'8900","ondisk_log_start":"457'8900","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:51:12.249449+0000","objects_scrubbed":0,"log_size":4290,"log_dups_size":0,"ondisk_log_size":4290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174325760,"num_objects":2660,"num_object_clones":0,"num_object_copies":7980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13190,"num_write_kb":507200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":941,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.684475+0000","last_change":"2024-02-05T06:53:54.096200+0000","last_active":"2024-02-05T06:54:08.684475+0000","last_peered":"2024-02-05T06:54:08.684475+0000","last_clean":"2024-02-05T06:54:08.684475+0000","last_became_active":"2024-02-05T06:41:43.412387+0000","last_became_peered":"2024-02-05T06:41:43.412387+0000","last_unstale":"2024-02-05T06:54:08.684475+0000","last_undegraded":"2024-02-05T06:54:08.684475+0000","last_fullsized":"2024-02-05T06:54:08.684475+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:54.095702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:34.871805+0000","last_clean_scrub_stamp":"2024-02-05T06:53:54.095702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:14.492332+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"465'21","reported_seq":992,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.690117+0000","last_change":"2024-02-05T06:52:44.087732+0000","last_active":"2024-02-05T06:54:08.690117+0000","last_peered":"2024-02-05T06:54:08.690117+0000","last_clean":"2024-02-05T06:54:08.690117+0000","last_became_active":"2024-02-05T06:41:43.420458+0000","last_became_peered":"2024-02-05T06:41:43.420458+0000","last_unstale":"2024-02-05T06:54:08.690117+0000","last_undegraded":"2024-02-05T06:54:08.690117+0000","last_fullsized":"2024-02-05T06:54:08.690117+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'17","last_scrub_stamp":"2024-02-05T06:52:44.087598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:52:44.087598+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:12.747430+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":926,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.683521+0000","last_change":"2024-02-05T06:53:44.055170+0000","last_active":"2024-02-05T06:54:08.683521+0000","last_peered":"2024-02-05T06:54:08.683521+0000","last_clean":"2024-02-05T06:54:08.683521+0000","last_became_active":"2024-02-05T06:33:28.542977+0000","last_became_peered":"2024-02-05T06:33:28.542977+0000","last_unstale":"2024-02-05T06:54:08.683521+0000","last_undegraded":"2024-02-05T06:54:08.683521+0000","last_fullsized":"2024-02-05T06:54:08.683521+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:44.054895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:53:44.054895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:11.925676+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.00091640799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":896,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.689572+0000","last_change":"2024-02-05T06:53:39.709665+0000","last_active":"2024-02-05T06:54:08.689572+0000","last_peered":"2024-02-05T06:54:08.689572+0000","last_clean":"2024-02-05T06:54:08.689572+0000","last_became_active":"2024-02-05T06:33:29.551549+0000","last_became_peered":"2024-02-05T06:33:29.551549+0000","last_unstale":"2024-02-05T06:54:08.689572+0000","last_undegraded":"2024-02-05T06:54:08.689572+0000","last_fullsized":"2024-02-05T06:54:08.689572+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:39.709556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:53:39.709556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:06.007029+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":596,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.686782+0000","last_change":"2024-02-05T06:53:41.146247+0000","last_active":"2024-02-05T06:54:08.686782+0000","last_peered":"2024-02-05T06:54:08.686782+0000","last_clean":"2024-02-05T06:54:08.686782+0000","last_became_active":"2024-02-05T06:40:27.077966+0000","last_became_peered":"2024-02-05T06:40:27.077966+0000","last_unstale":"2024-02-05T06:54:08.686782+0000","last_undegraded":"2024-02-05T06:54:08.686782+0000","last_fullsized":"2024-02-05T06:54:08.686782+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:41.146141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:45.774441+0000","last_clean_scrub_stamp":"2024-02-05T06:53:41.146141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:08.495898+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":1033,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.686131+0000","last_change":"2024-02-05T06:53:27.227987+0000","last_active":"2024-02-05T06:54:08.686131+0000","last_peered":"2024-02-05T06:54:08.686131+0000","last_clean":"2024-02-05T06:54:08.686131+0000","last_became_active":"2024-02-05T06:51:54.325427+0000","last_became_peered":"2024-02-05T06:51:54.325427+0000","last_unstale":"2024-02-05T06:54:08.686131+0000","last_undegraded":"2024-02-05T06:54:08.686131+0000","last_fullsized":"2024-02-05T06:54:08.686131+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:53:27.227870+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:48:28.657679+0000","last_clean_scrub_stamp":"2024-02-05T06:53:27.227870+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:27.245843+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"472'13072","reported_seq":13191,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:12.935742+0000","last_change":"2024-02-05T06:51:13.274482+0000","last_active":"2024-02-05T06:54:12.935742+0000","last_peered":"2024-02-05T06:54:12.935742+0000","last_clean":"2024-02-05T06:54:12.935742+0000","last_became_active":"2024-02-05T06:51:13.274182+0000","last_became_peered":"2024-02-05T06:51:13.274182+0000","last_unstale":"2024-02-05T06:54:12.935742+0000","last_undegraded":"2024-02-05T06:54:12.935742+0000","last_fullsized":"2024-02-05T06:54:12.935742+0000","mapping_epoch":418,"log_start":"456'8800","ondisk_log_start":"456'8800","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:51:12.249449+0000","objects_scrubbed":0,"log_size":4272,"log_dups_size":0,"ondisk_log_size":4272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179830784,"num_objects":2744,"num_object_clones":0,"num_object_copies":8232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13072,"num_write_kb":506112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":892,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.678509+0000","last_change":"2024-02-05T06:53:36.399616+0000","last_active":"2024-02-05T06:54:08.678509+0000","last_peered":"2024-02-05T06:54:08.678509+0000","last_clean":"2024-02-05T06:54:08.678509+0000","last_became_active":"2024-02-05T06:36:54.973387+0000","last_became_peered":"2024-02-05T06:36:54.973387+0000","last_unstale":"2024-02-05T06:54:08.678509+0000","last_undegraded":"2024-02-05T06:54:08.678509+0000","last_fullsized":"2024-02-05T06:54:08.678509+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:36.399508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:53:36.399508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:37.941444+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"465'64","reported_seq":846,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.693072+0000","last_change":"2024-02-05T06:53:36.797035+0000","last_active":"2024-02-05T06:54:08.693072+0000","last_peered":"2024-02-05T06:54:08.693072+0000","last_clean":"2024-02-05T06:54:08.693072+0000","last_became_active":"2024-02-05T06:40:08.604483+0000","last_became_peered":"2024-02-05T06:40:08.604483+0000","last_unstale":"2024-02-05T06:54:08.693072+0000","last_undegraded":"2024-02-05T06:54:08.693072+0000","last_fullsized":"2024-02-05T06:54:08.693072+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'52","last_scrub_stamp":"2024-02-05T06:53:36.796907+0000","last_deep_scrub":"326'52","last_deep_scrub_stamp":"2024-02-05T06:45:53.715482+0000","last_clean_scrub_stamp":"2024-02-05T06:53:36.796907+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:42.186110+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":417,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.679593+0000","last_change":"2024-02-05T06:52:47.127090+0000","last_active":"2024-02-05T06:54:08.679593+0000","last_peered":"2024-02-05T06:54:08.679593+0000","last_clean":"2024-02-05T06:54:08.679593+0000","last_became_active":"2024-02-05T06:45:09.776169+0000","last_became_peered":"2024-02-05T06:45:09.776169+0000","last_unstale":"2024-02-05T06:54:08.679593+0000","last_undegraded":"2024-02-05T06:54:08.679593+0000","last_fullsized":"2024-02-05T06:54:08.679593+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:47.126978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:47.126978+0000","last_clean_scrub_stamp":"2024-02-05T06:52:47.126978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"182'3","reported_seq":507,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.689298+0000","last_change":"2024-02-05T06:52:40.877231+0000","last_active":"2024-02-05T06:54:08.689298+0000","last_peered":"2024-02-05T06:54:08.689298+0000","last_clean":"2024-02-05T06:54:08.689298+0000","last_became_active":"2024-02-05T06:41:43.415806+0000","last_became_peered":"2024-02-05T06:41:43.415806+0000","last_unstale":"2024-02-05T06:54:08.689298+0000","last_undegraded":"2024-02-05T06:54:08.689298+0000","last_fullsized":"2024-02-05T06:54:08.689298+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'3","last_scrub_stamp":"2024-02-05T06:52:40.876912+0000","last_deep_scrub":"182'3","last_deep_scrub_stamp":"2024-02-05T06:39:33.373415+0000","last_clean_scrub_stamp":"2024-02-05T06:52:40.876912+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:10.919640+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001220348,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":883,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.692112+0000","last_change":"2024-02-05T06:50:52.403321+0000","last_active":"2024-02-05T06:54:08.692112+0000","last_peered":"2024-02-05T06:54:08.692112+0000","last_clean":"2024-02-05T06:54:08.692112+0000","last_became_active":"2024-02-05T06:40:27.082440+0000","last_became_peered":"2024-02-05T06:40:27.082440+0000","last_unstale":"2024-02-05T06:54:08.692112+0000","last_undegraded":"2024-02-05T06:54:08.692112+0000","last_fullsized":"2024-02-05T06:54:08.692112+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:50:52.403179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:50:52.403179+0000","last_clean_scrub_stamp":"2024-02-05T06:50:52.403179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:12.545609+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":928,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.688399+0000","last_change":"2024-02-05T06:52:17.849188+0000","last_active":"2024-02-05T06:54:08.688399+0000","last_peered":"2024-02-05T06:54:08.688399+0000","last_clean":"2024-02-05T06:54:08.688399+0000","last_became_active":"2024-02-05T06:51:54.325737+0000","last_became_peered":"2024-02-05T06:51:54.325737+0000","last_unstale":"2024-02-05T06:54:08.688399+0000","last_undegraded":"2024-02-05T06:54:08.688399+0000","last_fullsized":"2024-02-05T06:54:08.688399+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:52:17.849056+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:37.282442+0000","last_clean_scrub_stamp":"2024-02-05T06:52:17.849056+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:10.919612+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"83'128","reported_seq":395,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.686617+0000","last_change":"2024-02-05T06:52:21.118216+0000","last_active":"2024-02-05T06:54:08.686617+0000","last_peered":"2024-02-05T06:54:08.686617+0000","last_clean":"2024-02-05T06:54:08.686617+0000","last_became_active":"2024-02-05T06:51:59.294208+0000","last_became_peered":"2024-02-05T06:51:59.294208+0000","last_unstale":"2024-02-05T06:54:08.686617+0000","last_undegraded":"2024-02-05T06:54:08.686617+0000","last_fullsized":"2024-02-05T06:54:08.686617+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:52:21.118090+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:38.778568+0000","last_clean_scrub_stamp":"2024-02-05T06:52:21.118090+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:13.579410+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.4","version":"472'12850","reported_seq":13030,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:11.832084+0000","last_change":"2024-02-05T06:53:33.493384+0000","last_active":"2024-02-05T06:54:11.832084+0000","last_peered":"2024-02-05T06:54:11.832084+0000","last_clean":"2024-02-05T06:54:11.832084+0000","last_became_active":"2024-02-05T06:51:13.272748+0000","last_became_peered":"2024-02-05T06:51:13.272748+0000","last_unstale":"2024-02-05T06:54:11.832084+0000","last_undegraded":"2024-02-05T06:54:11.832084+0000","last_fullsized":"2024-02-05T06:54:11.832084+0000","mapping_epoch":418,"log_start":"453'8300","ondisk_log_start":"453'8300","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'10373","last_scrub_stamp":"2024-02-05T06:53:33.492667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:53:33.492667+0000","objects_scrubbed":5303,"log_size":4550,"log_dups_size":0,"ondisk_log_size":4550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:39.923205+0000","scrub_duration":1404,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182321152,"num_objects":2782,"num_object_clones":0,"num_object_copies":8346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2782,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12850,"num_write_kb":500224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"465'21","reported_seq":998,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.679441+0000","last_change":"2024-02-05T06:51:42.417785+0000","last_active":"2024-02-05T06:54:08.679441+0000","last_peered":"2024-02-05T06:54:08.679441+0000","last_clean":"2024-02-05T06:54:08.679441+0000","last_became_active":"2024-02-05T06:40:08.584204+0000","last_became_peered":"2024-02-05T06:40:08.584204+0000","last_unstale":"2024-02-05T06:54:08.679441+0000","last_undegraded":"2024-02-05T06:54:08.679441+0000","last_fullsized":"2024-02-05T06:54:08.679441+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:51:42.417532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:51:42.417532+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":503,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.689113+0000","last_change":"2024-02-05T06:52:32.883872+0000","last_active":"2024-02-05T06:54:08.689113+0000","last_peered":"2024-02-05T06:54:08.689113+0000","last_clean":"2024-02-05T06:54:08.689113+0000","last_became_active":"2024-02-05T06:41:43.412611+0000","last_became_peered":"2024-02-05T06:41:43.412611+0000","last_unstale":"2024-02-05T06:54:08.689113+0000","last_undegraded":"2024-02-05T06:54:08.689113+0000","last_fullsized":"2024-02-05T06:54:08.689113+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:32.883756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:41.617526+0000","last_clean_scrub_stamp":"2024-02-05T06:52:32.883756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:10.919746+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":921,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.682311+0000","last_change":"2024-02-05T06:52:48.076179+0000","last_active":"2024-02-05T06:54:08.682311+0000","last_peered":"2024-02-05T06:54:08.682311+0000","last_clean":"2024-02-05T06:54:08.682311+0000","last_became_active":"2024-02-05T06:36:54.966016+0000","last_became_peered":"2024-02-05T06:36:54.966016+0000","last_unstale":"2024-02-05T06:54:08.682311+0000","last_undegraded":"2024-02-05T06:54:08.682311+0000","last_fullsized":"2024-02-05T06:54:08.682311+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:48.076059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:52:48.076059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:13.576887+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":888,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.683559+0000","last_change":"2024-02-05T06:53:50.717387+0000","last_active":"2024-02-05T06:54:08.683559+0000","last_peered":"2024-02-05T06:54:08.683559+0000","last_clean":"2024-02-05T06:54:08.683559+0000","last_became_active":"2024-02-05T06:40:27.129625+0000","last_became_peered":"2024-02-05T06:40:27.129625+0000","last_unstale":"2024-02-05T06:54:08.683559+0000","last_undegraded":"2024-02-05T06:54:08.683559+0000","last_fullsized":"2024-02-05T06:54:08.683559+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:50.717250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:53:50.717250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:08.463818+0000","scrub_duration":214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"83'128","reported_seq":1106,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.690525+0000","last_change":"2024-02-05T06:52:21.609713+0000","last_active":"2024-02-05T06:54:08.690525+0000","last_peered":"2024-02-05T06:54:08.690525+0000","last_clean":"2024-02-05T06:54:08.690525+0000","last_became_active":"2024-02-05T06:51:59.293451+0000","last_became_peered":"2024-02-05T06:51:59.293451+0000","last_unstale":"2024-02-05T06:54:08.690525+0000","last_undegraded":"2024-02-05T06:54:08.690525+0000","last_fullsized":"2024-02-05T06:54:08.690525+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:52:21.609593+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:48:25.725892+0000","last_clean_scrub_stamp":"2024-02-05T06:52:21.609593+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.5","version":"472'13097","reported_seq":13302,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:11.809377+0000","last_change":"2024-02-05T06:52:50.694891+0000","last_active":"2024-02-05T06:54:11.809377+0000","last_peered":"2024-02-05T06:54:11.809377+0000","last_clean":"2024-02-05T06:54:11.809377+0000","last_became_active":"2024-02-05T06:51:13.270923+0000","last_became_peered":"2024-02-05T06:51:13.270923+0000","last_unstale":"2024-02-05T06:54:11.809377+0000","last_undegraded":"2024-02-05T06:54:11.809377+0000","last_fullsized":"2024-02-05T06:54:11.809377+0000","mapping_epoch":418,"log_start":"454'8500","ondisk_log_start":"454'8500","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'7908","last_scrub_stamp":"2024-02-05T06:52:50.694691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:52:50.694691+0000","objects_scrubbed":7859,"log_size":4597,"log_dups_size":0,"ondisk_log_size":4597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:13.579572+0000","scrub_duration":1523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183042048,"num_objects":2793,"num_object_clones":0,"num_object_copies":8379,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2793,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13097,"num_write_kb":508480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"465'60","reported_seq":1072,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.691530+0000","last_change":"2024-02-05T06:52:51.676140+0000","last_active":"2024-02-05T06:54:08.691530+0000","last_peered":"2024-02-05T06:54:08.691530+0000","last_clean":"2024-02-05T06:54:08.691530+0000","last_became_active":"2024-02-05T06:40:27.068417+0000","last_became_peered":"2024-02-05T06:40:27.068417+0000","last_unstale":"2024-02-05T06:54:08.691530+0000","last_undegraded":"2024-02-05T06:54:08.691530+0000","last_fullsized":"2024-02-05T06:54:08.691530+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'44","last_scrub_stamp":"2024-02-05T06:52:51.676057+0000","last_deep_scrub":"279'28","last_deep_scrub_stamp":"2024-02-05T06:42:50.148881+0000","last_clean_scrub_stamp":"2024-02-05T06:52:51.676057+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:11.068124+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":602,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.686961+0000","last_change":"2024-02-05T06:54:01.596848+0000","last_active":"2024-02-05T06:54:08.686961+0000","last_peered":"2024-02-05T06:54:08.686961+0000","last_clean":"2024-02-05T06:54:08.686961+0000","last_became_active":"2024-02-05T06:41:43.418202+0000","last_became_peered":"2024-02-05T06:41:43.418202+0000","last_unstale":"2024-02-05T06:54:08.686961+0000","last_undegraded":"2024-02-05T06:54:08.686961+0000","last_fullsized":"2024-02-05T06:54:08.686961+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:54:01.596730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:54:01.596730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:01.961347+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":927,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.683834+0000","last_change":"2024-02-05T06:53:52.054596+0000","last_active":"2024-02-05T06:54:08.683834+0000","last_peered":"2024-02-05T06:54:08.683834+0000","last_clean":"2024-02-05T06:54:08.683834+0000","last_became_active":"2024-02-05T06:41:43.413095+0000","last_became_peered":"2024-02-05T06:41:43.413095+0000","last_unstale":"2024-02-05T06:54:08.683834+0000","last_undegraded":"2024-02-05T06:54:08.683834+0000","last_fullsized":"2024-02-05T06:54:08.683834+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:52.054463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:45.073681+0000","last_clean_scrub_stamp":"2024-02-05T06:53:52.054463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:21.447521+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":596,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.686114+0000","last_change":"2024-02-05T06:52:47.116225+0000","last_active":"2024-02-05T06:54:08.686114+0000","last_peered":"2024-02-05T06:54:08.686114+0000","last_clean":"2024-02-05T06:54:08.686114+0000","last_became_active":"2024-02-05T06:40:27.218081+0000","last_became_peered":"2024-02-05T06:40:27.218081+0000","last_unstale":"2024-02-05T06:54:08.686114+0000","last_undegraded":"2024-02-05T06:54:08.686114+0000","last_fullsized":"2024-02-05T06:54:08.686114+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:47.116118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:32.925000+0000","last_clean_scrub_stamp":"2024-02-05T06:52:47.116118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:13.579331+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"83'128","reported_seq":1107,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.691131+0000","last_change":"2024-02-05T06:52:21.388903+0000","last_active":"2024-02-05T06:54:08.691131+0000","last_peered":"2024-02-05T06:54:08.691131+0000","last_clean":"2024-02-05T06:54:08.691131+0000","last_became_active":"2024-02-05T06:51:59.296988+0000","last_became_peered":"2024-02-05T06:51:59.296988+0000","last_unstale":"2024-02-05T06:54:08.691131+0000","last_undegraded":"2024-02-05T06:54:08.691131+0000","last_fullsized":"2024-02-05T06:54:08.691131+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:52:21.388783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:52:21.388783+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"472'13456","reported_seq":13627,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:16.519193+0000","last_change":"2024-02-05T06:53:46.461956+0000","last_active":"2024-02-05T06:54:16.519193+0000","last_peered":"2024-02-05T06:54:16.519193+0000","last_clean":"2024-02-05T06:54:16.519193+0000","last_became_active":"2024-02-05T06:51:13.267525+0000","last_became_peered":"2024-02-05T06:51:13.267525+0000","last_unstale":"2024-02-05T06:54:16.519193+0000","last_undegraded":"2024-02-05T06:54:16.519193+0000","last_fullsized":"2024-02-05T06:54:16.519193+0000","mapping_epoch":418,"log_start":"452'8200","ondisk_log_start":"452'8200","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"466'11503","last_scrub_stamp":"2024-02-05T06:53:46.461835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:53:46.461835+0000","objects_scrubbed":4404,"log_size":5256,"log_dups_size":0,"ondisk_log_size":5256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:58.262026+0000","scrub_duration":1947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155975680,"num_objects":2380,"num_object_clones":0,"num_object_copies":7140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13456,"num_write_kb":506752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":901,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.683362+0000","last_change":"2024-02-05T06:52:46.053806+0000","last_active":"2024-02-05T06:54:08.683362+0000","last_peered":"2024-02-05T06:54:08.683362+0000","last_clean":"2024-02-05T06:54:08.683362+0000","last_became_active":"2024-02-05T06:33:33.592546+0000","last_became_peered":"2024-02-05T06:33:33.592546+0000","last_unstale":"2024-02-05T06:54:08.683362+0000","last_undegraded":"2024-02-05T06:54:08.683362+0000","last_fullsized":"2024-02-05T06:54:08.683362+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:46.053684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:42.685240+0000","last_clean_scrub_stamp":"2024-02-05T06:52:46.053684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"465'24","reported_seq":756,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.689721+0000","last_change":"2024-02-05T06:52:26.996274+0000","last_active":"2024-02-05T06:54:08.689721+0000","last_peered":"2024-02-05T06:54:08.689721+0000","last_clean":"2024-02-05T06:54:08.689721+0000","last_became_active":"2024-02-05T06:45:09.783036+0000","last_became_peered":"2024-02-05T06:45:09.783036+0000","last_unstale":"2024-02-05T06:54:08.689721+0000","last_undegraded":"2024-02-05T06:54:08.689721+0000","last_fullsized":"2024-02-05T06:54:08.689721+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'18","last_scrub_stamp":"2024-02-05T06:52:26.996136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:52:26.996136+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:12.747463+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":29,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":596,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.687196+0000","last_change":"2024-02-05T06:53:36.100637+0000","last_active":"2024-02-05T06:54:08.687196+0000","last_peered":"2024-02-05T06:54:08.687196+0000","last_clean":"2024-02-05T06:54:08.687196+0000","last_became_active":"2024-02-05T06:40:27.063300+0000","last_became_peered":"2024-02-05T06:40:27.063300+0000","last_unstale":"2024-02-05T06:54:08.687196+0000","last_undegraded":"2024-02-05T06:54:08.687196+0000","last_fullsized":"2024-02-05T06:54:08.687196+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:36.100400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:34.977997+0000","last_clean_scrub_stamp":"2024-02-05T06:53:36.100400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:45.258122+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":871,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.693616+0000","last_change":"2024-02-05T06:51:22.703991+0000","last_active":"2024-02-05T06:54:08.693616+0000","last_peered":"2024-02-05T06:54:08.693616+0000","last_clean":"2024-02-05T06:54:08.693616+0000","last_became_active":"2024-02-05T06:40:27.092613+0000","last_became_peered":"2024-02-05T06:40:27.092613+0000","last_unstale":"2024-02-05T06:54:08.693616+0000","last_undegraded":"2024-02-05T06:54:08.693616+0000","last_fullsized":"2024-02-05T06:54:08.693616+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:22.703865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:51:22.703865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:12.545722+0000","scrub_duration":190,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"83'128","reported_seq":663,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.686534+0000","last_change":"2024-02-05T06:52:24.061520+0000","last_active":"2024-02-05T06:54:08.686534+0000","last_peered":"2024-02-05T06:54:08.686534+0000","last_clean":"2024-02-05T06:54:08.686534+0000","last_became_active":"2024-02-05T06:51:59.294191+0000","last_became_peered":"2024-02-05T06:51:59.294191+0000","last_unstale":"2024-02-05T06:54:08.686534+0000","last_undegraded":"2024-02-05T06:54:08.686534+0000","last_fullsized":"2024-02-05T06:54:08.686534+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:52:24.061383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:52:24.061383+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:13.579445+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.7","version":"472'16777","reported_seq":17060,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:11.917101+0000","last_change":"2024-02-05T06:54:08.528393+0000","last_active":"2024-02-05T06:54:11.917101+0000","last_peered":"2024-02-05T06:54:11.917101+0000","last_clean":"2024-02-05T06:54:11.917101+0000","last_became_active":"2024-02-05T06:51:13.272066+0000","last_became_peered":"2024-02-05T06:51:13.272066+0000","last_unstale":"2024-02-05T06:54:11.917101+0000","last_undegraded":"2024-02-05T06:54:11.917101+0000","last_fullsized":"2024-02-05T06:54:11.917101+0000","mapping_epoch":418,"log_start":"467'11701","ondisk_log_start":"467'11701","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"471'16513","last_scrub_stamp":"2024-02-05T06:54:08.528248+0000","last_deep_scrub":"471'16513","last_deep_scrub_stamp":"2024-02-05T06:54:08.528248+0000","last_clean_scrub_stamp":"2024-02-05T06:54:08.528248+0000","objects_scrubbed":3651,"log_size":5076,"log_dups_size":0,"ondisk_log_size":5076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":20,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:36.731421+0000","scrub_duration":19752,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179306496,"num_objects":2736,"num_object_clones":0,"num_object_copies":8208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2736,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13126,"num_write_kb":507584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"208'12","reported_seq":765,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.686328+0000","last_change":"2024-02-05T06:52:33.103331+0000","last_active":"2024-02-05T06:54:08.686328+0000","last_peered":"2024-02-05T06:54:08.686328+0000","last_clean":"2024-02-05T06:54:08.686328+0000","last_became_active":"2024-02-05T06:40:27.064195+0000","last_became_peered":"2024-02-05T06:40:27.064195+0000","last_unstale":"2024-02-05T06:54:08.686328+0000","last_undegraded":"2024-02-05T06:54:08.686328+0000","last_fullsized":"2024-02-05T06:54:08.686328+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'12","last_scrub_stamp":"2024-02-05T06:52:33.103188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:52:33.103188+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:13.579542+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"465'39","reported_seq":555,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.689079+0000","last_change":"2024-02-05T06:53:36.831417+0000","last_active":"2024-02-05T06:54:08.689079+0000","last_peered":"2024-02-05T06:54:08.689079+0000","last_clean":"2024-02-05T06:54:08.689079+0000","last_became_active":"2024-02-05T06:41:43.412579+0000","last_became_peered":"2024-02-05T06:41:43.412579+0000","last_unstale":"2024-02-05T06:54:08.689079+0000","last_undegraded":"2024-02-05T06:54:08.689079+0000","last_fullsized":"2024-02-05T06:54:08.689079+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'27","last_scrub_stamp":"2024-02-05T06:53:36.831295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:53:36.831295+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:03.891726+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"412'4","reported_seq":604,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.687024+0000","last_change":"2024-02-05T06:52:39.111257+0000","last_active":"2024-02-05T06:54:08.687024+0000","last_peered":"2024-02-05T06:54:08.687024+0000","last_clean":"2024-02-05T06:54:08.687024+0000","last_became_active":"2024-02-05T06:40:27.068279+0000","last_became_peered":"2024-02-05T06:40:27.068279+0000","last_unstale":"2024-02-05T06:54:08.687024+0000","last_undegraded":"2024-02-05T06:54:08.687024+0000","last_fullsized":"2024-02-05T06:54:08.687024+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'4","last_scrub_stamp":"2024-02-05T06:52:39.111151+0000","last_deep_scrub":"412'4","last_deep_scrub_stamp":"2024-02-05T06:50:51.841825+0000","last_clean_scrub_stamp":"2024-02-05T06:52:39.111151+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:13.579628+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":503,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.689083+0000","last_change":"2024-02-05T06:52:26.888326+0000","last_active":"2024-02-05T06:54:08.689083+0000","last_peered":"2024-02-05T06:54:08.689083+0000","last_clean":"2024-02-05T06:54:08.689083+0000","last_became_active":"2024-02-05T06:41:43.421231+0000","last_became_peered":"2024-02-05T06:41:43.421231+0000","last_unstale":"2024-02-05T06:54:08.689083+0000","last_undegraded":"2024-02-05T06:54:08.689083+0000","last_fullsized":"2024-02-05T06:54:08.689083+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:26.888098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:52:26.888098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:10.919691+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"83'128","reported_seq":530,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.688505+0000","last_change":"2024-02-05T06:52:21.904768+0000","last_active":"2024-02-05T06:54:08.688505+0000","last_peered":"2024-02-05T06:54:08.688505+0000","last_clean":"2024-02-05T06:54:08.688505+0000","last_became_active":"2024-02-05T06:51:59.296150+0000","last_became_peered":"2024-02-05T06:51:59.296150+0000","last_unstale":"2024-02-05T06:54:08.688505+0000","last_undegraded":"2024-02-05T06:54:08.688505+0000","last_fullsized":"2024-02-05T06:54:08.688505+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:52:21.904642+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:48:28.657679+0000","last_clean_scrub_stamp":"2024-02-05T06:52:21.904642+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:10.919583+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"465'21","reported_seq":932,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.692760+0000","last_change":"2024-02-05T06:53:51.658817+0000","last_active":"2024-02-05T06:54:08.692760+0000","last_peered":"2024-02-05T06:54:08.692760+0000","last_clean":"2024-02-05T06:54:08.692760+0000","last_became_active":"2024-02-05T06:40:27.091689+0000","last_became_peered":"2024-02-05T06:40:27.091689+0000","last_unstale":"2024-02-05T06:54:08.692760+0000","last_undegraded":"2024-02-05T06:54:08.692760+0000","last_fullsized":"2024-02-05T06:54:08.692760+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'21","last_scrub_stamp":"2024-02-05T06:53:51.658691+0000","last_deep_scrub":"412'15","last_deep_scrub_stamp":"2024-02-05T06:50:55.397689+0000","last_clean_scrub_stamp":"2024-02-05T06:53:51.658691+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:01.285256+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":921,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.690322+0000","last_change":"2024-02-05T06:53:55.061956+0000","last_active":"2024-02-05T06:54:08.690322+0000","last_peered":"2024-02-05T06:54:08.690322+0000","last_clean":"2024-02-05T06:54:08.690322+0000","last_became_active":"2024-02-05T06:41:43.412130+0000","last_became_peered":"2024-02-05T06:41:43.412130+0000","last_unstale":"2024-02-05T06:54:08.690322+0000","last_undegraded":"2024-02-05T06:54:08.690322+0000","last_fullsized":"2024-02-05T06:54:08.690322+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:55.061848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:53:55.061848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:24.423601+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":503,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.688947+0000","last_change":"2024-02-05T06:52:24.934323+0000","last_active":"2024-02-05T06:54:08.688947+0000","last_peered":"2024-02-05T06:54:08.688947+0000","last_clean":"2024-02-05T06:54:08.688947+0000","last_became_active":"2024-02-05T06:41:43.406094+0000","last_became_peered":"2024-02-05T06:41:43.406094+0000","last_unstale":"2024-02-05T06:54:08.688947+0000","last_undegraded":"2024-02-05T06:54:08.688947+0000","last_fullsized":"2024-02-05T06:54:08.688947+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:24.934198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:44:57.506909+0000","last_clean_scrub_stamp":"2024-02-05T06:52:24.934198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:10.919525+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":595,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.687051+0000","last_change":"2024-02-05T06:53:54.796403+0000","last_active":"2024-02-05T06:54:08.687051+0000","last_peered":"2024-02-05T06:54:08.687051+0000","last_clean":"2024-02-05T06:54:08.687051+0000","last_became_active":"2024-02-05T06:40:27.063350+0000","last_became_peered":"2024-02-05T06:40:27.063350+0000","last_unstale":"2024-02-05T06:54:08.687051+0000","last_undegraded":"2024-02-05T06:54:08.687051+0000","last_fullsized":"2024-02-05T06:54:08.687051+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:54.796297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:53:54.796297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:17.620285+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"83'128","reported_seq":1073,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.690408+0000","last_change":"2024-02-05T06:54:02.691383+0000","last_active":"2024-02-05T06:54:08.690408+0000","last_peered":"2024-02-05T06:54:08.690408+0000","last_clean":"2024-02-05T06:54:08.690408+0000","last_became_active":"2024-02-05T06:51:59.303621+0000","last_became_peered":"2024-02-05T06:51:59.303621+0000","last_unstale":"2024-02-05T06:54:08.690408+0000","last_undegraded":"2024-02-05T06:54:08.690408+0000","last_fullsized":"2024-02-05T06:54:08.690408+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:54:02.691263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:54:02.691263+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:17.412431+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"465'21","reported_seq":957,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.685317+0000","last_change":"2024-02-05T06:53:37.421718+0000","last_active":"2024-02-05T06:54:08.685317+0000","last_peered":"2024-02-05T06:54:08.685317+0000","last_clean":"2024-02-05T06:54:08.685317+0000","last_became_active":"2024-02-05T06:40:27.101116+0000","last_became_peered":"2024-02-05T06:40:27.101116+0000","last_unstale":"2024-02-05T06:54:08.685317+0000","last_undegraded":"2024-02-05T06:54:08.685317+0000","last_fullsized":"2024-02-05T06:54:08.685317+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'19","last_scrub_stamp":"2024-02-05T06:53:37.421591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:53:37.421591+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:56.525787+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":595,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.686358+0000","last_change":"2024-02-05T06:53:44.983388+0000","last_active":"2024-02-05T06:54:08.686358+0000","last_peered":"2024-02-05T06:54:08.686358+0000","last_clean":"2024-02-05T06:54:08.686358+0000","last_became_active":"2024-02-05T06:40:27.064388+0000","last_became_peered":"2024-02-05T06:40:27.064388+0000","last_unstale":"2024-02-05T06:54:08.686358+0000","last_undegraded":"2024-02-05T06:54:08.686358+0000","last_fullsized":"2024-02-05T06:54:08.686358+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:44.983315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:53:44.983315+0000","last_clean_scrub_stamp":"2024-02-05T06:53:44.983315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:50.922462+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":888,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.692595+0000","last_change":"2024-02-05T06:52:26.443142+0000","last_active":"2024-02-05T06:54:08.692595+0000","last_peered":"2024-02-05T06:54:08.692595+0000","last_clean":"2024-02-05T06:54:08.692595+0000","last_became_active":"2024-02-05T06:40:27.091544+0000","last_became_peered":"2024-02-05T06:40:27.091544+0000","last_unstale":"2024-02-05T06:54:08.692595+0000","last_undegraded":"2024-02-05T06:54:08.692595+0000","last_fullsized":"2024-02-05T06:54:08.692595+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:26.443024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:52:26.443024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:12.546876+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":928,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.684710+0000","last_change":"2024-02-05T06:52:45.010159+0000","last_active":"2024-02-05T06:54:08.684710+0000","last_peered":"2024-02-05T06:54:08.684710+0000","last_clean":"2024-02-05T06:54:08.684710+0000","last_became_active":"2024-02-05T06:40:27.090632+0000","last_became_peered":"2024-02-05T06:40:27.090632+0000","last_unstale":"2024-02-05T06:54:08.684710+0000","last_undegraded":"2024-02-05T06:54:08.684710+0000","last_fullsized":"2024-02-05T06:54:08.684710+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:45.010039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:45.010039+0000","last_clean_scrub_stamp":"2024-02-05T06:52:45.010039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:13.806168+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"83'128","reported_seq":530,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.688165+0000","last_change":"2024-02-05T06:52:20.872578+0000","last_active":"2024-02-05T06:54:08.688165+0000","last_peered":"2024-02-05T06:54:08.688165+0000","last_clean":"2024-02-05T06:54:08.688165+0000","last_became_active":"2024-02-05T06:51:59.290466+0000","last_became_peered":"2024-02-05T06:51:59.290466+0000","last_unstale":"2024-02-05T06:54:08.688165+0000","last_undegraded":"2024-02-05T06:54:08.688165+0000","last_fullsized":"2024-02-05T06:54:08.688165+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:52:20.872451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:52:20.872451+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:10.919446+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"465'21","reported_seq":531,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.690954+0000","last_change":"2024-02-05T06:52:31.905022+0000","last_active":"2024-02-05T06:54:08.690954+0000","last_peered":"2024-02-05T06:54:08.690954+0000","last_clean":"2024-02-05T06:54:08.690954+0000","last_became_active":"2024-02-05T06:41:43.413708+0000","last_became_peered":"2024-02-05T06:41:43.413708+0000","last_unstale":"2024-02-05T06:54:08.690954+0000","last_undegraded":"2024-02-05T06:54:08.690954+0000","last_fullsized":"2024-02-05T06:54:08.690954+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:52:31.904813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:52:31.904813+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:10.919721+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":907,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.678910+0000","last_change":"2024-02-05T06:53:43.562505+0000","last_active":"2024-02-05T06:54:08.678910+0000","last_peered":"2024-02-05T06:54:08.678910+0000","last_clean":"2024-02-05T06:54:08.678910+0000","last_became_active":"2024-02-05T06:40:27.088096+0000","last_became_peered":"2024-02-05T06:40:27.088096+0000","last_unstale":"2024-02-05T06:54:08.678910+0000","last_undegraded":"2024-02-05T06:54:08.678910+0000","last_fullsized":"2024-02-05T06:54:08.678910+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:43.562266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:53:43.562266+0000","last_clean_scrub_stamp":"2024-02-05T06:53:43.562266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:09.519621+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"208'10","reported_seq":1147,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.691845+0000","last_change":"2024-02-05T06:52:43.367615+0000","last_active":"2024-02-05T06:54:08.691845+0000","last_peered":"2024-02-05T06:54:08.691845+0000","last_clean":"2024-02-05T06:54:08.691845+0000","last_became_active":"2024-02-05T06:40:27.096584+0000","last_became_peered":"2024-02-05T06:40:27.096584+0000","last_unstale":"2024-02-05T06:54:08.691845+0000","last_undegraded":"2024-02-05T06:54:08.691845+0000","last_fullsized":"2024-02-05T06:54:08.691845+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-02-05T06:52:43.367496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:52:43.367496+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:12.546993+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":643,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.684053+0000","last_change":"2024-02-05T06:52:44.242695+0000","last_active":"2024-02-05T06:54:08.684053+0000","last_peered":"2024-02-05T06:54:08.684053+0000","last_clean":"2024-02-05T06:54:08.684053+0000","last_became_active":"2024-02-05T06:45:09.775152+0000","last_became_peered":"2024-02-05T06:45:09.775152+0000","last_unstale":"2024-02-05T06:54:08.684053+0000","last_undegraded":"2024-02-05T06:54:08.684053+0000","last_fullsized":"2024-02-05T06:54:08.684053+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:44.242462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:41:38.559780+0000","last_clean_scrub_stamp":"2024-02-05T06:52:44.242462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:10.251365+0000","scrub_duration":228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"83'128","reported_seq":531,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.678009+0000","last_change":"2024-02-05T06:52:22.246184+0000","last_active":"2024-02-05T06:54:08.678009+0000","last_peered":"2024-02-05T06:54:08.678009+0000","last_clean":"2024-02-05T06:54:08.678009+0000","last_became_active":"2024-02-05T06:51:59.300112+0000","last_became_peered":"2024-02-05T06:51:59.300112+0000","last_unstale":"2024-02-05T06:54:08.678009+0000","last_undegraded":"2024-02-05T06:54:08.678009+0000","last_fullsized":"2024-02-05T06:54:08.678009+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:52:22.246060+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:37.282442+0000","last_clean_scrub_stamp":"2024-02-05T06:52:22.246060+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:20.455643+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"465'30","reported_seq":973,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.691415+0000","last_change":"2024-02-05T06:53:26.322082+0000","last_active":"2024-02-05T06:54:08.691415+0000","last_peered":"2024-02-05T06:54:08.691415+0000","last_clean":"2024-02-05T06:54:08.691415+0000","last_became_active":"2024-02-05T06:40:27.093418+0000","last_became_peered":"2024-02-05T06:40:27.093418+0000","last_unstale":"2024-02-05T06:54:08.691415+0000","last_undegraded":"2024-02-05T06:54:08.691415+0000","last_fullsized":"2024-02-05T06:54:08.691415+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'20","last_scrub_stamp":"2024-02-05T06:53:26.321955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:53:26.321955+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:43.931325+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":596,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.686463+0000","last_change":"2024-02-05T06:53:43.055431+0000","last_active":"2024-02-05T06:54:08.686463+0000","last_peered":"2024-02-05T06:54:08.686463+0000","last_clean":"2024-02-05T06:54:08.686463+0000","last_became_active":"2024-02-05T06:40:27.098418+0000","last_became_peered":"2024-02-05T06:40:27.098418+0000","last_unstale":"2024-02-05T06:54:08.686463+0000","last_undegraded":"2024-02-05T06:54:08.686463+0000","last_fullsized":"2024-02-05T06:54:08.686463+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:43.055316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:53:43.055316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:12.814470+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"336'11","reported_seq":658,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.690655+0000","last_change":"2024-02-05T06:52:18.890279+0000","last_active":"2024-02-05T06:54:08.690655+0000","last_peered":"2024-02-05T06:54:08.690655+0000","last_clean":"2024-02-05T06:54:08.690655+0000","last_became_active":"2024-02-05T06:41:43.418753+0000","last_became_peered":"2024-02-05T06:41:43.418753+0000","last_unstale":"2024-02-05T06:54:08.690655+0000","last_undegraded":"2024-02-05T06:54:08.690655+0000","last_fullsized":"2024-02-05T06:54:08.690655+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'11","last_scrub_stamp":"2024-02-05T06:52:18.890167+0000","last_deep_scrub":"336'11","last_deep_scrub_stamp":"2024-02-05T06:46:55.721412+0000","last_clean_scrub_stamp":"2024-02-05T06:52:18.890167+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:10.919554+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":879,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.685104+0000","last_change":"2024-02-05T06:52:30.384032+0000","last_active":"2024-02-05T06:54:08.685104+0000","last_peered":"2024-02-05T06:54:08.685104+0000","last_clean":"2024-02-05T06:54:08.685104+0000","last_became_active":"2024-02-05T06:40:27.096132+0000","last_became_peered":"2024-02-05T06:40:27.096132+0000","last_unstale":"2024-02-05T06:54:08.685104+0000","last_undegraded":"2024-02-05T06:54:08.685104+0000","last_fullsized":"2024-02-05T06:54:08.685104+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:30.383899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:52:30.383899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"465'32","reported_seq":768,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.690579+0000","last_change":"2024-02-05T06:53:32.977787+0000","last_active":"2024-02-05T06:54:08.690579+0000","last_peered":"2024-02-05T06:54:08.690579+0000","last_clean":"2024-02-05T06:54:08.690579+0000","last_became_active":"2024-02-05T06:45:09.784411+0000","last_became_peered":"2024-02-05T06:45:09.784411+0000","last_unstale":"2024-02-05T06:54:08.690579+0000","last_undegraded":"2024-02-05T06:54:08.690579+0000","last_fullsized":"2024-02-05T06:54:08.690579+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:53:32.977666+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:35:02.589229+0000","last_clean_scrub_stamp":"2024-02-05T06:53:32.977666+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:37.861865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":541,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.686623+0000","last_change":"2024-02-05T06:52:44.163784+0000","last_active":"2024-02-05T06:54:08.686623+0000","last_peered":"2024-02-05T06:54:08.686623+0000","last_clean":"2024-02-05T06:54:08.686623+0000","last_became_active":"2024-02-05T06:45:09.773490+0000","last_became_peered":"2024-02-05T06:45:09.773490+0000","last_unstale":"2024-02-05T06:54:08.686623+0000","last_undegraded":"2024-02-05T06:54:08.686623+0000","last_fullsized":"2024-02-05T06:54:08.686623+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:44.163661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:52:44.163661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:13.579599+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":912,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.690701+0000","last_change":"2024-02-05T06:53:53.666791+0000","last_active":"2024-02-05T06:54:08.690701+0000","last_peered":"2024-02-05T06:54:08.690701+0000","last_clean":"2024-02-05T06:54:08.690701+0000","last_became_active":"2024-02-05T06:41:43.421433+0000","last_became_peered":"2024-02-05T06:41:43.421433+0000","last_unstale":"2024-02-05T06:54:08.690701+0000","last_undegraded":"2024-02-05T06:54:08.690701+0000","last_fullsized":"2024-02-05T06:54:08.690701+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:53.666667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:53:53.666667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:17.847146+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":878,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.692086+0000","last_change":"2024-02-05T06:53:31.491982+0000","last_active":"2024-02-05T06:54:08.692086+0000","last_peered":"2024-02-05T06:54:08.692086+0000","last_clean":"2024-02-05T06:54:08.692086+0000","last_became_active":"2024-02-05T06:41:43.417155+0000","last_became_peered":"2024-02-05T06:41:43.417155+0000","last_unstale":"2024-02-05T06:54:08.692086+0000","last_undegraded":"2024-02-05T06:54:08.692086+0000","last_fullsized":"2024-02-05T06:54:08.692086+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:31.491724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:53:31.491724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:00.042636+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"465'21","reported_seq":681,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.678487+0000","last_change":"2024-02-05T06:52:45.194776+0000","last_active":"2024-02-05T06:54:08.678487+0000","last_peered":"2024-02-05T06:54:08.678487+0000","last_clean":"2024-02-05T06:54:08.678487+0000","last_became_active":"2024-02-05T06:39:13.244124+0000","last_became_peered":"2024-02-05T06:39:13.244124+0000","last_unstale":"2024-02-05T06:54:08.678487+0000","last_undegraded":"2024-02-05T06:54:08.678487+0000","last_fullsized":"2024-02-05T06:54:08.678487+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:52:45.194645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:52:45.194645+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":595,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.686883+0000","last_change":"2024-02-05T06:53:44.009695+0000","last_active":"2024-02-05T06:54:08.686883+0000","last_peered":"2024-02-05T06:54:08.686883+0000","last_clean":"2024-02-05T06:54:08.686883+0000","last_became_active":"2024-02-05T06:40:27.077701+0000","last_became_peered":"2024-02-05T06:40:27.077701+0000","last_unstale":"2024-02-05T06:54:08.686883+0000","last_undegraded":"2024-02-05T06:54:08.686883+0000","last_fullsized":"2024-02-05T06:54:08.686883+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:44.009607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:33.978941+0000","last_clean_scrub_stamp":"2024-02-05T06:53:44.009607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:13.642503+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":862,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.693384+0000","last_change":"2024-02-05T06:51:30.567546+0000","last_active":"2024-02-05T06:54:08.693384+0000","last_peered":"2024-02-05T06:54:08.693384+0000","last_clean":"2024-02-05T06:54:08.693384+0000","last_became_active":"2024-02-05T06:33:33.593901+0000","last_became_peered":"2024-02-05T06:33:33.593901+0000","last_unstale":"2024-02-05T06:54:08.693384+0000","last_undegraded":"2024-02-05T06:54:08.693384+0000","last_fullsized":"2024-02-05T06:54:08.693384+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:30.567403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:30.567403+0000","last_clean_scrub_stamp":"2024-02-05T06:51:30.567403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:12.545695+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"336'2","reported_seq":897,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.678501+0000","last_change":"2024-02-05T06:52:46.298052+0000","last_active":"2024-02-05T06:54:08.678501+0000","last_peered":"2024-02-05T06:54:08.678501+0000","last_clean":"2024-02-05T06:54:08.678501+0000","last_became_active":"2024-02-05T06:41:43.409761+0000","last_became_peered":"2024-02-05T06:41:43.409761+0000","last_unstale":"2024-02-05T06:54:08.678501+0000","last_undegraded":"2024-02-05T06:54:08.678501+0000","last_fullsized":"2024-02-05T06:54:08.678501+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'2","last_scrub_stamp":"2024-02-05T06:52:46.297810+0000","last_deep_scrub":"336'2","last_deep_scrub_stamp":"2024-02-05T06:46:53.500155+0000","last_clean_scrub_stamp":"2024-02-05T06:52:46.297810+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"83'128","reported_seq":1102,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.683947+0000","last_change":"2024-02-05T06:53:47.062697+0000","last_active":"2024-02-05T06:54:08.683947+0000","last_peered":"2024-02-05T06:54:08.683947+0000","last_clean":"2024-02-05T06:54:08.683947+0000","last_became_active":"2024-02-05T06:51:59.292766+0000","last_became_peered":"2024-02-05T06:51:59.292766+0000","last_unstale":"2024-02-05T06:54:08.683947+0000","last_undegraded":"2024-02-05T06:54:08.683947+0000","last_fullsized":"2024-02-05T06:54:08.683947+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:53:47.062582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:53:47.062582+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:10.434191+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"465'32","reported_seq":968,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.692609+0000","last_change":"2024-02-05T06:51:32.530845+0000","last_active":"2024-02-05T06:54:08.692609+0000","last_peered":"2024-02-05T06:54:08.692609+0000","last_clean":"2024-02-05T06:54:08.692609+0000","last_became_active":"2024-02-05T06:40:27.080906+0000","last_became_peered":"2024-02-05T06:40:27.080906+0000","last_unstale":"2024-02-05T06:54:08.692609+0000","last_undegraded":"2024-02-05T06:54:08.692609+0000","last_fullsized":"2024-02-05T06:54:08.692609+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:51:32.530706+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-02-05T06:35:22.938675+0000","last_clean_scrub_stamp":"2024-02-05T06:51:32.530706+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:12.546644+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":503,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.689029+0000","last_change":"2024-02-05T06:52:22.931134+0000","last_active":"2024-02-05T06:54:08.689029+0000","last_peered":"2024-02-05T06:54:08.689029+0000","last_clean":"2024-02-05T06:54:08.689029+0000","last_became_active":"2024-02-05T06:41:43.412654+0000","last_became_peered":"2024-02-05T06:41:43.412654+0000","last_unstale":"2024-02-05T06:54:08.689029+0000","last_undegraded":"2024-02-05T06:54:08.689029+0000","last_fullsized":"2024-02-05T06:54:08.689029+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:22.931061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:52:22.931061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:10.919472+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":595,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.686107+0000","last_change":"2024-02-05T06:53:34.074529+0000","last_active":"2024-02-05T06:54:08.686107+0000","last_peered":"2024-02-05T06:54:08.686107+0000","last_clean":"2024-02-05T06:54:08.686107+0000","last_became_active":"2024-02-05T06:40:27.062975+0000","last_became_peered":"2024-02-05T06:40:27.062975+0000","last_unstale":"2024-02-05T06:54:08.686107+0000","last_undegraded":"2024-02-05T06:54:08.686107+0000","last_fullsized":"2024-02-05T06:54:08.686107+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:34.074458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:48:34.628182+0000","last_clean_scrub_stamp":"2024-02-05T06:53:34.074458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:41.569557+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":653,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.693646+0000","last_change":"2024-02-05T06:52:40.357841+0000","last_active":"2024-02-05T06:54:08.693646+0000","last_peered":"2024-02-05T06:54:08.693646+0000","last_clean":"2024-02-05T06:54:08.693646+0000","last_became_active":"2024-02-05T06:39:13.245684+0000","last_became_peered":"2024-02-05T06:39:13.245684+0000","last_unstale":"2024-02-05T06:54:08.693646+0000","last_undegraded":"2024-02-05T06:54:08.693646+0000","last_fullsized":"2024-02-05T06:54:08.693646+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:40.356857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:52:40.356857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:12.747519+0000","scrub_duration":268,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"465'12","reported_seq":884,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.691018+0000","last_change":"2024-02-05T06:51:33.570857+0000","last_active":"2024-02-05T06:54:08.691018+0000","last_peered":"2024-02-05T06:54:08.691018+0000","last_clean":"2024-02-05T06:54:08.691018+0000","last_became_active":"2024-02-05T06:33:31.573410+0000","last_became_peered":"2024-02-05T06:33:31.573410+0000","last_unstale":"2024-02-05T06:54:08.691018+0000","last_undegraded":"2024-02-05T06:54:08.691018+0000","last_fullsized":"2024-02-05T06:54:08.691018+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'8","last_scrub_stamp":"2024-02-05T06:51:33.570665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:51:33.570665+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:12.546706+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":899,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.679050+0000","last_change":"2024-02-05T06:53:47.483803+0000","last_active":"2024-02-05T06:54:08.679050+0000","last_peered":"2024-02-05T06:54:08.679050+0000","last_clean":"2024-02-05T06:54:08.679050+0000","last_became_active":"2024-02-05T06:41:43.424459+0000","last_became_peered":"2024-02-05T06:41:43.424459+0000","last_unstale":"2024-02-05T06:54:08.679050+0000","last_undegraded":"2024-02-05T06:54:08.679050+0000","last_fullsized":"2024-02-05T06:54:08.679050+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:47.483623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:49.117949+0000","last_clean_scrub_stamp":"2024-02-05T06:53:47.483623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:10.250852+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":933,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.691290+0000","last_change":"2024-02-05T06:53:41.132629+0000","last_active":"2024-02-05T06:54:08.691290+0000","last_peered":"2024-02-05T06:54:08.691290+0000","last_clean":"2024-02-05T06:54:08.691290+0000","last_became_active":"2024-02-05T06:40:27.060112+0000","last_became_peered":"2024-02-05T06:40:27.060112+0000","last_unstale":"2024-02-05T06:54:08.691290+0000","last_undegraded":"2024-02-05T06:54:08.691290+0000","last_fullsized":"2024-02-05T06:54:08.691290+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:41.132523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:48:33.848718+0000","last_clean_scrub_stamp":"2024-02-05T06:53:41.132523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:41.585442+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":601,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.686227+0000","last_change":"2024-02-05T06:53:28.246285+0000","last_active":"2024-02-05T06:54:08.686227+0000","last_peered":"2024-02-05T06:54:08.686227+0000","last_clean":"2024-02-05T06:54:08.686227+0000","last_became_active":"2024-02-05T06:44:54.625067+0000","last_became_peered":"2024-02-05T06:44:54.625067+0000","last_unstale":"2024-02-05T06:54:08.686227+0000","last_undegraded":"2024-02-05T06:54:08.686227+0000","last_fullsized":"2024-02-05T06:54:08.686227+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:28.246206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:48.512673+0000","last_clean_scrub_stamp":"2024-02-05T06:53:28.246206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:36.459451+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"465'32","reported_seq":993,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.690757+0000","last_change":"2024-02-05T06:52:23.643592+0000","last_active":"2024-02-05T06:54:08.690757+0000","last_peered":"2024-02-05T06:54:08.690757+0000","last_clean":"2024-02-05T06:54:08.690757+0000","last_became_active":"2024-02-05T06:41:43.419769+0000","last_became_peered":"2024-02-05T06:41:43.419769+0000","last_unstale":"2024-02-05T06:54:08.690757+0000","last_undegraded":"2024-02-05T06:54:08.690757+0000","last_fullsized":"2024-02-05T06:54:08.690757+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:52:23.643458+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:43:03.221466+0000","last_clean_scrub_stamp":"2024-02-05T06:52:23.643458+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:11.067938+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":503,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.688836+0000","last_change":"2024-02-05T06:53:38.839606+0000","last_active":"2024-02-05T06:54:08.688836+0000","last_peered":"2024-02-05T06:54:08.688836+0000","last_clean":"2024-02-05T06:54:08.688836+0000","last_became_active":"2024-02-05T06:41:43.411043+0000","last_became_peered":"2024-02-05T06:41:43.411043+0000","last_unstale":"2024-02-05T06:54:08.688836+0000","last_undegraded":"2024-02-05T06:54:08.688836+0000","last_fullsized":"2024-02-05T06:54:08.688836+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:38.839486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:53:38.839486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:40.388758+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":795,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.691770+0000","last_change":"2024-02-05T06:53:48.540882+0000","last_active":"2024-02-05T06:54:08.691770+0000","last_peered":"2024-02-05T06:54:08.691770+0000","last_clean":"2024-02-05T06:54:08.691770+0000","last_became_active":"2024-02-05T06:40:27.097169+0000","last_became_peered":"2024-02-05T06:40:27.097169+0000","last_unstale":"2024-02-05T06:54:08.691770+0000","last_undegraded":"2024-02-05T06:54:08.691770+0000","last_fullsized":"2024-02-05T06:54:08.691770+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:48.540767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:08.713894+0000","last_clean_scrub_stamp":"2024-02-05T06:53:48.540767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:50.708856+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":906,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.691299+0000","last_change":"2024-02-05T06:52:29.108690+0000","last_active":"2024-02-05T06:54:08.691299+0000","last_peered":"2024-02-05T06:54:08.691299+0000","last_clean":"2024-02-05T06:54:08.691299+0000","last_became_active":"2024-02-05T06:41:43.414638+0000","last_became_peered":"2024-02-05T06:41:43.414638+0000","last_unstale":"2024-02-05T06:54:08.691299+0000","last_undegraded":"2024-02-05T06:54:08.691299+0000","last_fullsized":"2024-02-05T06:54:08.691299+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:29.108567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:50:42.316726+0000","last_clean_scrub_stamp":"2024-02-05T06:52:29.108567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:12.747280+0000","scrub_duration":199,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"465'35","reported_seq":979,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.685253+0000","last_change":"2024-02-05T06:53:54.509653+0000","last_active":"2024-02-05T06:54:08.685253+0000","last_peered":"2024-02-05T06:54:08.685253+0000","last_clean":"2024-02-05T06:54:08.685253+0000","last_became_active":"2024-02-05T06:40:27.100610+0000","last_became_peered":"2024-02-05T06:40:27.100610+0000","last_unstale":"2024-02-05T06:54:08.685253+0000","last_undegraded":"2024-02-05T06:54:08.685253+0000","last_fullsized":"2024-02-05T06:54:08.685253+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'35","last_scrub_stamp":"2024-02-05T06:53:54.509362+0000","last_deep_scrub":"465'35","last_deep_scrub_stamp":"2024-02-05T06:53:54.509362+0000","last_clean_scrub_stamp":"2024-02-05T06:53:54.509362+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:00.417269+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":923,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.689821+0000","last_change":"2024-02-05T06:52:50.110926+0000","last_active":"2024-02-05T06:54:08.689821+0000","last_peered":"2024-02-05T06:54:08.689821+0000","last_clean":"2024-02-05T06:54:08.689821+0000","last_became_active":"2024-02-05T06:36:54.958414+0000","last_became_peered":"2024-02-05T06:36:54.958414+0000","last_unstale":"2024-02-05T06:54:08.689821+0000","last_undegraded":"2024-02-05T06:54:08.689821+0000","last_fullsized":"2024-02-05T06:54:08.689821+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:52:50.110793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:52:50.110793+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:14.374092+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"314'13","reported_seq":1208,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.690617+0000","last_change":"2024-02-05T06:52:22.597335+0000","last_active":"2024-02-05T06:54:08.690617+0000","last_peered":"2024-02-05T06:54:08.690617+0000","last_clean":"2024-02-05T06:54:08.690617+0000","last_became_active":"2024-02-05T06:40:27.063399+0000","last_became_peered":"2024-02-05T06:40:27.063399+0000","last_unstale":"2024-02-05T06:54:08.690617+0000","last_undegraded":"2024-02-05T06:54:08.690617+0000","last_fullsized":"2024-02-05T06:54:08.690617+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"314'13","last_scrub_stamp":"2024-02-05T06:52:22.597203+0000","last_deep_scrub":"314'13","last_deep_scrub_stamp":"2024-02-05T06:44:53.670923+0000","last_clean_scrub_stamp":"2024-02-05T06:52:22.597203+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:11.068090+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"325'2","reported_seq":602,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.686895+0000","last_change":"2024-02-05T06:52:48.168560+0000","last_active":"2024-02-05T06:54:08.686895+0000","last_peered":"2024-02-05T06:54:08.686895+0000","last_clean":"2024-02-05T06:54:08.686895+0000","last_became_active":"2024-02-05T06:41:43.418640+0000","last_became_peered":"2024-02-05T06:41:43.418640+0000","last_unstale":"2024-02-05T06:54:08.686895+0000","last_undegraded":"2024-02-05T06:54:08.686895+0000","last_fullsized":"2024-02-05T06:54:08.686895+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'2","last_scrub_stamp":"2024-02-05T06:52:48.168445+0000","last_deep_scrub":"325'2","last_deep_scrub_stamp":"2024-02-05T06:45:46.242415+0000","last_clean_scrub_stamp":"2024-02-05T06:52:48.168445+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:13.579512+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"465'29","reported_seq":974,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.692020+0000","last_change":"2024-02-05T06:52:37.441014+0000","last_active":"2024-02-05T06:54:08.692020+0000","last_peered":"2024-02-05T06:54:08.692020+0000","last_clean":"2024-02-05T06:54:08.692020+0000","last_became_active":"2024-02-05T06:36:54.954880+0000","last_became_peered":"2024-02-05T06:36:54.954880+0000","last_unstale":"2024-02-05T06:54:08.692020+0000","last_undegraded":"2024-02-05T06:54:08.692020+0000","last_fullsized":"2024-02-05T06:54:08.692020+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'23","last_scrub_stamp":"2024-02-05T06:52:37.440858+0000","last_deep_scrub":"445'23","last_deep_scrub_stamp":"2024-02-05T06:52:37.440858+0000","last_clean_scrub_stamp":"2024-02-05T06:52:37.440858+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:12.546937+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":416,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.690559+0000","last_change":"2024-02-05T06:53:51.717338+0000","last_active":"2024-02-05T06:54:08.690559+0000","last_peered":"2024-02-05T06:54:08.690559+0000","last_clean":"2024-02-05T06:54:08.690559+0000","last_became_active":"2024-02-05T06:45:09.786579+0000","last_became_peered":"2024-02-05T06:45:09.786579+0000","last_unstale":"2024-02-05T06:54:08.690559+0000","last_undegraded":"2024-02-05T06:54:08.690559+0000","last_fullsized":"2024-02-05T06:54:08.690559+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:51.717205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:53:51.717205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:02.032086+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":675,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.691001+0000","last_change":"2024-02-05T06:53:51.117144+0000","last_active":"2024-02-05T06:54:08.691001+0000","last_peered":"2024-02-05T06:54:08.691001+0000","last_clean":"2024-02-05T06:54:08.691001+0000","last_became_active":"2024-02-05T06:39:13.248618+0000","last_became_peered":"2024-02-05T06:39:13.248618+0000","last_unstale":"2024-02-05T06:54:08.691001+0000","last_undegraded":"2024-02-05T06:54:08.691001+0000","last_fullsized":"2024-02-05T06:54:08.691001+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:51.116923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:34.551395+0000","last_clean_scrub_stamp":"2024-02-05T06:53:51.116923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:19.941766+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":871,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.692674+0000","last_change":"2024-02-05T06:51:21.832566+0000","last_active":"2024-02-05T06:54:08.692674+0000","last_peered":"2024-02-05T06:54:08.692674+0000","last_clean":"2024-02-05T06:54:08.692674+0000","last_became_active":"2024-02-05T06:41:43.411442+0000","last_became_peered":"2024-02-05T06:41:43.411442+0000","last_unstale":"2024-02-05T06:54:08.692674+0000","last_undegraded":"2024-02-05T06:54:08.692674+0000","last_fullsized":"2024-02-05T06:54:08.692674+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:21.832441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:51:21.832441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:12.545637+0000","scrub_duration":295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"465'46","reported_seq":988,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.684346+0000","last_change":"2024-02-05T06:53:33.046693+0000","last_active":"2024-02-05T06:54:08.684346+0000","last_peered":"2024-02-05T06:54:08.684346+0000","last_clean":"2024-02-05T06:54:08.684346+0000","last_became_active":"2024-02-05T06:33:31.569763+0000","last_became_peered":"2024-02-05T06:33:31.569763+0000","last_unstale":"2024-02-05T06:54:08.684346+0000","last_undegraded":"2024-02-05T06:54:08.684346+0000","last_fullsized":"2024-02-05T06:54:08.684346+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'34","last_scrub_stamp":"2024-02-05T06:53:33.046569+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:34:35.234636+0000","last_clean_scrub_stamp":"2024-02-05T06:53:33.046569+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:47.972370+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"281'2","reported_seq":934,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.679679+0000","last_change":"2024-02-05T06:53:30.452677+0000","last_active":"2024-02-05T06:54:08.679679+0000","last_peered":"2024-02-05T06:54:08.679679+0000","last_clean":"2024-02-05T06:54:08.679679+0000","last_became_active":"2024-02-05T06:40:27.073360+0000","last_became_peered":"2024-02-05T06:40:27.073360+0000","last_unstale":"2024-02-05T06:54:08.679679+0000","last_undegraded":"2024-02-05T06:54:08.679679+0000","last_fullsized":"2024-02-05T06:54:08.679679+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'2","last_scrub_stamp":"2024-02-05T06:53:30.452604+0000","last_deep_scrub":"281'2","last_deep_scrub_stamp":"2024-02-05T06:47:50.442159+0000","last_clean_scrub_stamp":"2024-02-05T06:53:30.452604+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:49.493466+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":898,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.691152+0000","last_change":"2024-02-05T06:53:52.698534+0000","last_active":"2024-02-05T06:54:08.691152+0000","last_peered":"2024-02-05T06:54:08.691152+0000","last_clean":"2024-02-05T06:54:08.691152+0000","last_became_active":"2024-02-05T06:36:54.958069+0000","last_became_peered":"2024-02-05T06:36:54.958069+0000","last_unstale":"2024-02-05T06:54:08.691152+0000","last_undegraded":"2024-02-05T06:54:08.691152+0000","last_fullsized":"2024-02-05T06:54:08.691152+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:52.698426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:38.793760+0000","last_clean_scrub_stamp":"2024-02-05T06:53:52.698426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:13.013123+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":644,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.691029+0000","last_change":"2024-02-05T06:53:36.022199+0000","last_active":"2024-02-05T06:54:08.691029+0000","last_peered":"2024-02-05T06:54:08.691029+0000","last_clean":"2024-02-05T06:54:08.691029+0000","last_became_active":"2024-02-05T06:45:09.770902+0000","last_became_peered":"2024-02-05T06:45:09.770902+0000","last_unstale":"2024-02-05T06:54:08.691029+0000","last_undegraded":"2024-02-05T06:54:08.691029+0000","last_fullsized":"2024-02-05T06:54:08.691029+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:36.022090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:53:36.022090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:39.332622+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"465'9","reported_seq":736,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.693218+0000","last_change":"2024-02-05T06:53:27.348848+0000","last_active":"2024-02-05T06:54:08.693218+0000","last_peered":"2024-02-05T06:54:08.693218+0000","last_clean":"2024-02-05T06:54:08.693218+0000","last_became_active":"2024-02-05T06:45:09.808663+0000","last_became_peered":"2024-02-05T06:45:09.808663+0000","last_unstale":"2024-02-05T06:54:08.693218+0000","last_undegraded":"2024-02-05T06:54:08.693218+0000","last_fullsized":"2024-02-05T06:54:08.693218+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'7","last_scrub_stamp":"2024-02-05T06:53:27.348728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:53:27.348728+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:51.281890+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":502,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.689183+0000","last_change":"2024-02-05T06:52:43.828744+0000","last_active":"2024-02-05T06:54:08.689183+0000","last_peered":"2024-02-05T06:54:08.689183+0000","last_clean":"2024-02-05T06:54:08.689183+0000","last_became_active":"2024-02-05T06:41:43.408428+0000","last_became_peered":"2024-02-05T06:41:43.408428+0000","last_unstale":"2024-02-05T06:54:08.689183+0000","last_undegraded":"2024-02-05T06:54:08.689183+0000","last_fullsized":"2024-02-05T06:54:08.689183+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:43.828638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:43.828638+0000","last_clean_scrub_stamp":"2024-02-05T06:52:43.828638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:10.919666+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":603,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.686095+0000","last_change":"2024-02-05T06:53:58.717874+0000","last_active":"2024-02-05T06:54:08.686095+0000","last_peered":"2024-02-05T06:54:08.686095+0000","last_clean":"2024-02-05T06:54:08.686095+0000","last_became_active":"2024-02-05T06:40:27.071758+0000","last_became_peered":"2024-02-05T06:40:27.071758+0000","last_unstale":"2024-02-05T06:54:08.686095+0000","last_undegraded":"2024-02-05T06:54:08.686095+0000","last_fullsized":"2024-02-05T06:54:08.686095+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:58.717756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:37.152741+0000","last_clean_scrub_stamp":"2024-02-05T06:53:58.717756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:16.156426+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":503,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.688300+0000","last_change":"2024-02-05T06:53:34.839643+0000","last_active":"2024-02-05T06:54:08.688300+0000","last_peered":"2024-02-05T06:54:08.688300+0000","last_clean":"2024-02-05T06:54:08.688300+0000","last_became_active":"2024-02-05T06:41:43.407404+0000","last_became_peered":"2024-02-05T06:41:43.407404+0000","last_unstale":"2024-02-05T06:54:08.688300+0000","last_undegraded":"2024-02-05T06:54:08.688300+0000","last_fullsized":"2024-02-05T06:54:08.688300+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:34.839509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:21.998850+0000","last_clean_scrub_stamp":"2024-02-05T06:53:34.839509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:54.888016+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"465'25","reported_seq":744,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.696756+0000","last_change":"2024-02-05T06:52:31.409340+0000","last_active":"2024-02-05T06:54:08.696756+0000","last_peered":"2024-02-05T06:54:08.696756+0000","last_clean":"2024-02-05T06:54:08.696756+0000","last_became_active":"2024-02-05T06:45:09.787626+0000","last_became_peered":"2024-02-05T06:45:09.787626+0000","last_unstale":"2024-02-05T06:54:08.696756+0000","last_undegraded":"2024-02-05T06:54:08.696756+0000","last_fullsized":"2024-02-05T06:54:08.696756+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'19","last_scrub_stamp":"2024-02-05T06:52:31.409196+0000","last_deep_scrub":"92'13","last_deep_scrub_stamp":"2024-02-05T06:34:46.542893+0000","last_clean_scrub_stamp":"2024-02-05T06:52:31.409196+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:12.546907+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":494,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.690870+0000","last_change":"2024-02-05T06:51:30.797965+0000","last_active":"2024-02-05T06:54:08.690870+0000","last_peered":"2024-02-05T06:54:08.690870+0000","last_clean":"2024-02-05T06:54:08.690870+0000","last_became_active":"2024-02-05T06:41:43.403746+0000","last_became_peered":"2024-02-05T06:41:43.403746+0000","last_unstale":"2024-02-05T06:54:08.690870+0000","last_undegraded":"2024-02-05T06:54:08.690870+0000","last_fullsized":"2024-02-05T06:54:08.690870+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:30.797854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:44.081318+0000","last_clean_scrub_stamp":"2024-02-05T06:51:30.797854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:10.919360+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":443,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.693540+0000","last_change":"2024-02-05T06:53:34.584200+0000","last_active":"2024-02-05T06:54:08.693540+0000","last_peered":"2024-02-05T06:54:08.693540+0000","last_clean":"2024-02-05T06:54:08.693540+0000","last_became_active":"2024-02-05T06:42:33.129959+0000","last_became_peered":"2024-02-05T06:42:33.129959+0000","last_unstale":"2024-02-05T06:54:08.693540+0000","last_undegraded":"2024-02-05T06:54:08.693540+0000","last_fullsized":"2024-02-05T06:54:08.693540+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:34.583981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:45:51.217900+0000","last_clean_scrub_stamp":"2024-02-05T06:53:34.583981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:55.085329+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":887,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.678768+0000","last_change":"2024-02-05T06:52:51.154037+0000","last_active":"2024-02-05T06:54:08.678768+0000","last_peered":"2024-02-05T06:54:08.678768+0000","last_clean":"2024-02-05T06:54:08.678768+0000","last_became_active":"2024-02-05T06:40:27.068888+0000","last_became_peered":"2024-02-05T06:40:27.068888+0000","last_unstale":"2024-02-05T06:54:08.678768+0000","last_undegraded":"2024-02-05T06:54:08.678768+0000","last_fullsized":"2024-02-05T06:54:08.678768+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:51.153442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:44:55.125921+0000","last_clean_scrub_stamp":"2024-02-05T06:52:51.153442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:16.151191+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"465'46","reported_seq":942,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.692707+0000","last_change":"2024-02-05T06:52:27.389525+0000","last_active":"2024-02-05T06:54:08.692707+0000","last_peered":"2024-02-05T06:54:08.692707+0000","last_clean":"2024-02-05T06:54:08.692707+0000","last_became_active":"2024-02-05T06:33:31.572544+0000","last_became_peered":"2024-02-05T06:33:31.572544+0000","last_unstale":"2024-02-05T06:54:08.692707+0000","last_undegraded":"2024-02-05T06:54:08.692707+0000","last_fullsized":"2024-02-05T06:54:08.692707+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'34","last_scrub_stamp":"2024-02-05T06:52:27.389433+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:39:35.433539+0000","last_clean_scrub_stamp":"2024-02-05T06:52:27.389433+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:12.546820+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":880,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.691894+0000","last_change":"2024-02-05T06:51:23.547319+0000","last_active":"2024-02-05T06:54:08.691894+0000","last_peered":"2024-02-05T06:54:08.691894+0000","last_clean":"2024-02-05T06:54:08.691894+0000","last_became_active":"2024-02-05T06:33:29.553864+0000","last_became_peered":"2024-02-05T06:33:29.553864+0000","last_unstale":"2024-02-05T06:54:08.691894+0000","last_undegraded":"2024-02-05T06:54:08.691894+0000","last_fullsized":"2024-02-05T06:54:08.691894+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:23.547188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:51:23.547188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:12.545749+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":408,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.690411+0000","last_change":"2024-02-05T06:53:37.018077+0000","last_active":"2024-02-05T06:54:08.690411+0000","last_peered":"2024-02-05T06:54:08.690411+0000","last_clean":"2024-02-05T06:54:08.690411+0000","last_became_active":"2024-02-05T06:45:09.775037+0000","last_became_peered":"2024-02-05T06:45:09.775037+0000","last_unstale":"2024-02-05T06:54:08.690411+0000","last_undegraded":"2024-02-05T06:54:08.690411+0000","last_fullsized":"2024-02-05T06:54:08.690411+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:37.018004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:51.388156+0000","last_clean_scrub_stamp":"2024-02-05T06:53:37.018004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:52.016322+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":878,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.693457+0000","last_change":"2024-02-05T06:53:30.666137+0000","last_active":"2024-02-05T06:54:08.693457+0000","last_peered":"2024-02-05T06:54:08.693457+0000","last_clean":"2024-02-05T06:54:08.693457+0000","last_became_active":"2024-02-05T06:40:27.077485+0000","last_became_peered":"2024-02-05T06:40:27.077485+0000","last_unstale":"2024-02-05T06:54:08.693457+0000","last_undegraded":"2024-02-05T06:54:08.693457+0000","last_fullsized":"2024-02-05T06:54:08.693457+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:30.666019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:53:30.666019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:55.573254+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"465'25","reported_seq":828,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.672040+0000","last_change":"2024-02-05T06:53:25.983530+0000","last_active":"2024-02-05T06:54:08.672040+0000","last_peered":"2024-02-05T06:54:08.672040+0000","last_clean":"2024-02-05T06:54:08.672040+0000","last_became_active":"2024-02-05T06:36:54.959330+0000","last_became_peered":"2024-02-05T06:36:54.959330+0000","last_unstale":"2024-02-05T06:54:08.672040+0000","last_undegraded":"2024-02-05T06:54:08.672040+0000","last_fullsized":"2024-02-05T06:54:08.672040+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'19","last_scrub_stamp":"2024-02-05T06:53:25.983402+0000","last_deep_scrub":"247'13","last_deep_scrub_stamp":"2024-02-05T06:41:42.473424+0000","last_clean_scrub_stamp":"2024-02-05T06:53:25.983402+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:41.235478+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":31,"num_read_kb":25,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":901,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.678854+0000","last_change":"2024-02-05T06:53:48.501549+0000","last_active":"2024-02-05T06:54:08.678854+0000","last_peered":"2024-02-05T06:54:08.678854+0000","last_clean":"2024-02-05T06:54:08.678854+0000","last_became_active":"2024-02-05T06:41:43.421315+0000","last_became_peered":"2024-02-05T06:41:43.421315+0000","last_unstale":"2024-02-05T06:54:08.678854+0000","last_undegraded":"2024-02-05T06:54:08.678854+0000","last_fullsized":"2024-02-05T06:54:08.678854+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:48.501421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:53:48.501421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:01.128496+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":910,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.689919+0000","last_change":"2024-02-05T06:53:46.710740+0000","last_active":"2024-02-05T06:54:08.689919+0000","last_peered":"2024-02-05T06:54:08.689919+0000","last_clean":"2024-02-05T06:54:08.689919+0000","last_became_active":"2024-02-05T06:41:43.420342+0000","last_became_peered":"2024-02-05T06:41:43.420342+0000","last_unstale":"2024-02-05T06:54:08.689919+0000","last_undegraded":"2024-02-05T06:54:08.689919+0000","last_fullsized":"2024-02-05T06:54:08.689919+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:46.710631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:06.525012+0000","last_clean_scrub_stamp":"2024-02-05T06:53:46.710631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:09.216440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":639,"reported_epoch":472,"state":"active+clean","last_fresh":"2024-02-05T06:54:08.689462+0000","last_change":"2024-02-05T06:52:28.883912+0000","last_active":"2024-02-05T06:54:08.689462+0000","last_peered":"2024-02-05T06:54:08.689462+0000","last_clean":"2024-02-05T06:54:08.689462+0000","last_became_active":"2024-02-05T06:45:09.787134+0000","last_became_peered":"2024-02-05T06:45:09.787134+0000","last_unstale":"2024-02-05T06:54:08.689462+0000","last_undegraded":"2024-02-05T06:54:08.689462+0000","last_fullsized":"2024-02-05T06:54:08.689462+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:28.883819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:52:28.883819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:10.919497+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":2836070424,"num_objects":43276,"num_object_clones":0,"num_object_copies":129828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43276,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":210652,"num_write_kb":8125633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8390782976,"data_stored":8390770760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76052,"ondisk_log_size":76052,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":830,"num_read_kb":825,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2688,"ondisk_log_size":2688,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1535,"num_read_kb":1292,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":373,"num_bytes_recovered":8234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1086,"ondisk_log_size":1086,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103420,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1879824,"kb_used_data":1076744,"kb_used_omap":1,"kb_used_meta":803070,"kb_avail":91865328,"statfs":{"total":95995035648,"available":94070095872,"internally_reserved":0,"allocated":1102585856,"data_stored":1102237761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":822344138},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:53:13 2024","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.2050000000000001,"15min":1.3999999999999999},"min":{"1min":0.23799999999999999,"5min":0.23400000000000001,"15min":0.21299999999999999},"max":{"1min":4.1029999999999998,"5min":5.2329999999999997,"15min":52.890999999999998},"last":1.147},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.512,"15min":1.583},"min":{"1min":0.255,"5min":0.255,"15min":0.222},"max":{"1min":3.1400000000000001,"5min":41.109999999999999,"15min":53.353000000000002},"last":2.0720000000000001}]},{"osd":1,"last update":"Mon Feb 5 06:53:13 2024","interfaces":[{"interface":"back","average":{"1min":1.9670000000000001,"5min":1.6579999999999999,"15min":1.7909999999999999},"min":{"1min":0.318,"5min":0.26000000000000001,"15min":0.25},"max":{"1min":8.2569999999999997,"5min":23.716999999999999,"15min":84.216999999999999},"last":2.6890000000000001},{"interface":"front","average":{"1min":1.492,"5min":1.7190000000000001,"15min":1.8740000000000001},"min":{"1min":0.24299999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":7.9450000000000003,"5min":41.167999999999999,"15min":84.126999999999995},"last":0.34000000000000002}]},{"osd":2,"last update":"Mon Feb 5 06:53:13 2024","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.5409999999999999,"15min":1.506},"min":{"1min":0.32700000000000001,"5min":0.249,"15min":0.249},"max":{"1min":8.4740000000000002,"5min":28.704999999999998,"15min":53.947000000000003},"last":1.363},{"interface":"front","average":{"1min":1.54,"5min":1.6319999999999999,"15min":1.506},"min":{"1min":0.34100000000000003,"5min":0.28899999999999998,"15min":0.26800000000000002},"max":{"1min":11.452,"5min":23.678999999999998,"15min":53.299999999999997},"last":1.264}]},{"osd":3,"last update":"Mon Feb 5 06:53:13 2024","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.462,"15min":1.794},"min":{"1min":0.39800000000000002,"5min":0.29099999999999998,"15min":0.28899999999999998},"max":{"1min":7.8250000000000002,"5min":23.199000000000002,"15min":73.283000000000001},"last":1.4379999999999999},{"interface":"front","average":{"1min":1.72,"5min":1.8839999999999999,"15min":1.8560000000000001},"min":{"1min":0.307,"5min":0.29499999999999998,"15min":0.27400000000000002},"max":{"1min":7.9089999999999998,"5min":41.219000000000001,"15min":73.323999999999998},"last":1.821}]},{"osd":4,"last update":"Mon Feb 5 06:53:13 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":1.0960000000000001,"15min":0.81299999999999994},"min":{"1min":0.252,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.1930000000000001,"5min":6.1980000000000004,"15min":6.1980000000000004},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.76100000000000001,"5min":1.4099999999999999,"15min":0.91400000000000003},"min":{"1min":0.318,"5min":0.25600000000000001,"15min":0.255},"max":{"1min":3.48,"5min":41.270000000000003,"15min":41.270000000000003},"last":0.28899999999999998}]},{"osd":5,"last update":"Mon Feb 5 06:53:13 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":2.4580000000000002,"15min":1.375},"min":{"1min":0.309,"5min":0.252,"15min":0.251},"max":{"1min":4.0170000000000003,"5min":109.482,"15min":109.482},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.88100000000000001,"5min":2.5270000000000001,"15min":1.458},"min":{"1min":0.30199999999999999,"5min":0.29999999999999999,"15min":0.252},"max":{"1min":3.1699999999999999,"5min":109.56399999999999,"15min":109.56399999999999},"last":0.377}]},{"osd":6,"last update":"Mon Feb 5 06:53:13 2024","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":2.8769999999999998,"15min":1.5229999999999999},"min":{"1min":0.378,"5min":0.34899999999999998,"15min":0.28499999999999998},"max":{"1min":3.2490000000000001,"5min":113.914,"15min":113.914},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.79000000000000004,"5min":2.714,"15min":1.4350000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.307},"max":{"1min":3.0190000000000001,"5min":113.212,"15min":113.212},"last":0.31900000000000001}]}]},{"osd":6,"up_from":49,"seq":210453397762,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2169404,"kb_used_data":1416308,"kb_used_omap":1,"kb_used_meta":753086,"kb_avail":91575748,"statfs":{"total":95995035648,"available":93773565952,"internally_reserved":0,"allocated":1450299392,"data_stored":1449970352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":771160518},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:53:34 2024","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.482,"15min":2.3959999999999999},"min":{"1min":0.249,"5min":0.23699999999999999,"15min":0.224},"max":{"1min":10.039999999999999,"5min":15.401,"15min":98.918999999999997},"last":0.42899999999999999},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.744,"15min":2.464},"min":{"1min":0.27400000000000002,"5min":0.223,"15min":0.223},"max":{"1min":10.23,"5min":15.388999999999999,"15min":98.775000000000006},"last":3.9670000000000001}]},{"osd":1,"last update":"Mon Feb 5 06:53:34 2024","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.6359999999999999,"15min":2.173},"min":{"1min":0.28199999999999997,"5min":0.254,"15min":0.24099999999999999},"max":{"1min":6.3730000000000002,"5min":47.348999999999997,"15min":98.843999999999994},"last":6.0549999999999997},{"interface":"front","average":{"1min":1.286,"5min":1.623,"15min":2.206},"min":{"1min":0.27800000000000002,"5min":0.24099999999999999,"15min":0.23100000000000001},"max":{"1min":6.3899999999999997,"5min":47.777999999999999,"15min":101.637},"last":4.1580000000000004}]},{"osd":2,"last update":"Mon Feb 5 06:53:34 2024","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.361,"15min":2.1150000000000002},"min":{"1min":0.29599999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":10.446,"5min":10.446,"15min":85.617000000000004},"last":0.373},{"interface":"front","average":{"1min":1.607,"5min":1.6259999999999999,"15min":2.0779999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27000000000000002},"max":{"1min":16.516999999999999,"5min":16.516999999999999,"15min":85.335999999999999},"last":0.38800000000000001}]},{"osd":3,"last update":"Mon Feb 5 06:53:34 2024","interfaces":[{"interface":"back","average":{"1min":1.768,"5min":1.3779999999999999,"15min":2.5129999999999999},"min":{"1min":0.30299999999999999,"5min":0.27900000000000003,"15min":0.27000000000000002},"max":{"1min":14.648,"5min":14.648,"15min":99.269000000000005},"last":5.0839999999999996},{"interface":"front","average":{"1min":1.887,"5min":1.415,"15min":2.5870000000000002},"min":{"1min":0.31900000000000001,"5min":0.313,"15min":0.27800000000000002},"max":{"1min":14.6,"5min":14.6,"15min":101.75700000000001},"last":6.7510000000000003}]},{"osd":4,"last update":"Mon Feb 5 06:53:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.95399999999999996,"15min":0.72999999999999998},"min":{"1min":0.253,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":2.3220000000000001,"5min":5.5940000000000003,"15min":5.5940000000000003},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":1.018,"15min":0.745},"min":{"1min":0.33500000000000002,"5min":0.312,"15min":0.24299999999999999},"max":{"1min":1.3029999999999999,"5min":5.9500000000000002,"15min":5.9500000000000002},"last":0.40300000000000002}]},{"osd":5,"last update":"Mon Feb 5 06:53:34 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":1.0640000000000001,"15min":0.98299999999999998},"min":{"1min":0.30199999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":2.383,"5min":5.5570000000000004,"15min":36.548999999999999},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.80100000000000005,"15min":0.93000000000000005},"min":{"1min":0.34200000000000003,"5min":0.30299999999999999,"15min":0.28699999999999998},"max":{"1min":0.74099999999999999,"5min":3.75,"15min":36.433},"last":0.44}]},{"osd":7,"last update":"Mon Feb 5 06:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.75,"15min":0.85299999999999998},"min":{"1min":0.31,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.177,"5min":3.6200000000000001,"15min":38.436999999999998},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":1.036,"15min":0.95399999999999996},"min":{"1min":0.35899999999999999,"5min":0.29899999999999999,"15min":0.29699999999999999},"max":{"1min":1.244,"5min":5.3280000000000003,"15min":36.633000000000003},"last":0.41799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575131,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2311152,"kb_used_data":1381672,"kb_used_omap":1,"kb_used_meta":929406,"kb_avail":91434000,"statfs":{"total":95995035648,"available":93628416000,"internally_reserved":0,"allocated":1414832128,"data_stored":1414452048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":951712197},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:53:32 2024","interfaces":[{"interface":"back","average":{"1min":7.1289999999999996,"5min":2.9929999999999999,"15min":2.145},"min":{"1min":0.41699999999999998,"5min":0.20899999999999999,"15min":0.16700000000000001},"max":{"1min":93.488,"5min":93.488,"15min":93.488},"last":0.67100000000000004},{"interface":"front","average":{"1min":7.9020000000000001,"5min":3.0179999999999998,"15min":2.3079999999999998},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":93.689999999999998,"5min":93.689999999999998,"15min":93.689999999999998},"last":0.38500000000000001}]},{"osd":2,"last update":"Mon Feb 5 06:53:55 2024","interfaces":[{"interface":"back","average":{"1min":6.649,"5min":3.069,"15min":2.1179999999999999},"min":{"1min":0.38900000000000001,"5min":0.184,"15min":0.184},"max":{"1min":93.352999999999994,"5min":93.352999999999994,"15min":93.352999999999994},"last":2.5339999999999998},{"interface":"front","average":{"1min":6.5679999999999996,"5min":2.931,"15min":1.8959999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.22600000000000001},"max":{"1min":91.543000000000006,"5min":91.543000000000006,"15min":91.543000000000006},"last":0.52500000000000002}]},{"osd":3,"last update":"Mon Feb 5 06:53:13 2024","interfaces":[{"interface":"back","average":{"1min":2.3279999999999998,"5min":1.716,"15min":2.0179999999999998},"min":{"1min":0.50800000000000001,"5min":0.25900000000000001,"15min":0.23499999999999999},"max":{"1min":4.6159999999999997,"5min":7.9340000000000002,"15min":25.236999999999998},"last":0.26700000000000002},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.76,"15min":1.782},"min":{"1min":0.255,"5min":0.255,"15min":0.23699999999999999},"max":{"1min":3.0649999999999999,"5min":11.018000000000001,"15min":43.354999999999997},"last":0.95899999999999996}]},{"osd":4,"last update":"Mon Feb 5 06:53:38 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.2989999999999999,"15min":1.151},"min":{"1min":0.32500000000000001,"5min":0.32000000000000001,"15min":0.27700000000000002},"max":{"1min":5.3360000000000003,"5min":7.4059999999999997,"15min":7.4059999999999997},"last":0.58299999999999996},{"interface":"front","average":{"1min":2.4209999999999998,"5min":1.6850000000000001,"15min":1.472},"min":{"1min":0.42299999999999999,"5min":0.33100000000000002,"15min":0.26300000000000001},"max":{"1min":9.8680000000000003,"5min":9.8680000000000003,"15min":10.419},"last":0.83499999999999996}]},{"osd":5,"last update":"Mon Feb 5 06:53:13 2024","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.607,"15min":1.425},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.29299999999999998},"max":{"1min":2.9860000000000002,"5min":11.058,"15min":43.414999999999999},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.107,"15min":1.204},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":3.1480000000000001,"5min":8.1669999999999998,"15min":8.1669999999999998},"last":0.55900000000000005}]},{"osd":6,"last update":"Mon Feb 5 06:53:32 2024","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.5700000000000001,"15min":1.397},"min":{"1min":0.39900000000000002,"5min":0.38500000000000001,"15min":0.34799999999999998},"max":{"1min":4.4189999999999996,"5min":19.827999999999999,"15min":23.902999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":1.554,"5min":2.0430000000000001,"15min":1.6060000000000001},"min":{"1min":0.44700000000000001,"5min":0.39300000000000002,"15min":0.35599999999999998},"max":{"1min":5.0049999999999999,"5min":19.526,"15min":43.475000000000001},"last":2.609}]},{"osd":7,"last update":"Mon Feb 5 06:53:55 2024","interfaces":[{"interface":"back","average":{"1min":1.7809999999999999,"5min":1.405,"15min":1.349},"min":{"1min":0.41199999999999998,"5min":0.377,"15min":0.32900000000000001},"max":{"1min":8.1600000000000001,"5min":8.1600000000000001,"15min":8.1600000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":2.4470000000000001,"5min":1.645,"15min":1.5349999999999999},"min":{"1min":0.43099999999999999,"5min":0.36599999999999999,"15min":0.35499999999999998},"max":{"1min":9.8230000000000004,"5min":9.8230000000000004,"15min":10.481},"last":0.88600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738658,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1562784,"kb_used_data":777752,"kb_used_omap":1,"kb_used_meta":785022,"kb_avail":92182368,"statfs":{"total":95995035648,"available":94394744832,"internally_reserved":0,"allocated":796418048,"data_stored":796053545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":803862981},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 06:53:24 2024","interfaces":[{"interface":"back","average":{"1min":2.48,"5min":2.012,"15min":1.6679999999999999},"min":{"1min":0.188,"5min":0.188,"15min":0.182},"max":{"1min":30.350999999999999,"5min":38.68,"15min":38.68},"last":0.50600000000000001},{"interface":"front","average":{"1min":3.335,"5min":2.5960000000000001,"15min":1.9910000000000001},"min":{"1min":0.41599999999999998,"5min":0.19500000000000001,"15min":0.18099999999999999},"max":{"1min":31.122,"5min":38.003,"15min":38.003},"last":0.28999999999999998}]},{"osd":2,"last update":"Mon Feb 5 06:53:45 2024","interfaces":[{"interface":"back","average":{"1min":2.1179999999999999,"5min":2.4740000000000002,"15min":1.7869999999999999},"min":{"1min":0.39400000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.3559999999999999,"5min":27.312000000000001,"15min":27.312000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.6759999999999999,"5min":2.2589999999999999,"15min":1.6719999999999999},"min":{"1min":0.28299999999999997,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.298,"5min":27.359000000000002,"15min":27.359000000000002},"last":1.0740000000000001}]},{"osd":3,"last update":"Mon Feb 5 06:53:14 2024","interfaces":[{"interface":"back","average":{"1min":1.0680000000000001,"5min":1.869,"15min":1.8280000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":3.4420000000000002,"5min":31.777999999999999,"15min":43.302},"last":1.123},{"interface":"front","average":{"1min":2.8210000000000002,"5min":2.101,"15min":2.1080000000000001},"min":{"1min":0.26100000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":9.1579999999999995,"5min":31.716000000000001,"15min":43.651000000000003},"last":0.61499999999999999}]},{"osd":4,"last update":"Mon Feb 5 06:53:37 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.8140000000000001,"15min":1.5589999999999999},"min":{"1min":0.33100000000000002,"5min":0.32200000000000001,"15min":0.27200000000000002},"max":{"1min":4.6959999999999997,"5min":10.952999999999999,"15min":40.789999999999999},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.94199999999999995,"5min":1.3200000000000001,"15min":1.3999999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.30099999999999999},"max":{"1min":3.5699999999999998,"5min":5.9459999999999997,"15min":40.773000000000003},"last":0.63600000000000001}]},{"osd":5,"last update":"Mon Feb 5 06:54:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.5609999999999999,"15min":1.635},"min":{"1min":0.379,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":4.6799999999999997,"5min":7.1239999999999997,"15min":48.841999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.746,"15min":1.621},"min":{"1min":0.35699999999999998,"5min":0.28899999999999998,"15min":0.28599999999999998},"max":{"1min":4.7279999999999998,"5min":11.156000000000001,"15min":48.450000000000003},"last":0.46000000000000002}]},{"osd":6,"last update":"Mon Feb 5 06:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":1.2929999999999999,"15min":1.7589999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.33700000000000002},"max":{"1min":2.347,"5min":7.0049999999999999,"15min":136.76300000000001},"last":1.0609999999999999},{"interface":"front","average":{"1min":1.7350000000000001,"5min":1.78,"15min":2.0550000000000002},"min":{"1min":0.39800000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":6.0090000000000003,"5min":7.0720000000000001,"15min":135.92400000000001},"last":2.681}]},{"osd":7,"last update":"Mon Feb 5 06:53:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.7470000000000001,"15min":1.4830000000000001},"min":{"1min":0.40699999999999997,"5min":0.35499999999999998,"15min":0.31900000000000001},"max":{"1min":5.2249999999999996,"5min":10.989000000000001,"15min":17.210000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.222,"15min":1.276},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.34899999999999998},"max":{"1min":3.6040000000000001,"5min":6.383,"15min":18.533999999999999},"last":0.98099999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411606,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1749964,"kb_used_data":1060740,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":91995188,"statfs":{"total":95995035648,"available":94203072512,"internally_reserved":0,"allocated":1086197760,"data_stored":1085854198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:53:55 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":3.6970000000000001,"15min":2.298},"min":{"1min":0.31,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.7489999999999997,"5min":101.67700000000001,"15min":101.67700000000001},"last":1.1539999999999999},{"interface":"front","average":{"1min":0.99299999999999999,"5min":3.302,"15min":2.2290000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.17699999999999999},"max":{"1min":3.331,"5min":101.27800000000001,"15min":101.27800000000001},"last":0.20200000000000001}]},{"osd":1,"last update":"Mon Feb 5 06:53:55 2024","interfaces":[{"interface":"back","average":{"1min":2.1389999999999998,"5min":2.1920000000000002,"15min":1.7969999999999999},"min":{"1min":0.19500000000000001,"5min":0.185,"15min":0.185},"max":{"1min":6.7169999999999996,"5min":24.093,"15min":24.093},"last":0.48299999999999998},{"interface":"front","average":{"1min":1.76,"5min":1.7969999999999999,"15min":1.5109999999999999},"min":{"1min":0.36499999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.6870000000000003,"5min":27.114000000000001,"15min":27.114000000000001},"last":1.294}]},{"osd":3,"last update":"Mon Feb 5 06:53:22 2024","interfaces":[{"interface":"back","average":{"1min":2.069,"5min":2.871,"15min":2.0150000000000001},"min":{"1min":0.36299999999999999,"5min":0.21299999999999999,"15min":0.20899999999999999},"max":{"1min":5.4160000000000004,"5min":84.129000000000005,"15min":84.129000000000005},"last":1.1930000000000001},{"interface":"front","average":{"1min":2.4929999999999999,"5min":2.7050000000000001,"15min":1.9410000000000001},"min":{"1min":0.34699999999999998,"5min":0.214,"15min":0.214},"max":{"1min":9.641,"5min":84.090999999999994,"15min":84.090999999999994},"last":0.39500000000000002}]},{"osd":4,"last update":"Mon Feb 5 06:53:31 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.2629999999999999,"15min":1.149},"min":{"1min":0.39100000000000001,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":5.4539999999999997,"5min":5.4539999999999997,"15min":12.041},"last":1.2609999999999999},{"interface":"front","average":{"1min":0.82599999999999996,"5min":1.278,"15min":1.1319999999999999},"min":{"1min":0.35099999999999998,"5min":0.28000000000000003,"15min":0.26300000000000001},"max":{"1min":3.3679999999999999,"5min":5.4640000000000004,"15min":15.689},"last":0.29699999999999999}]},{"osd":5,"last update":"Mon Feb 5 06:54:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":2.2469999999999999,"15min":2.0369999999999999},"min":{"1min":0.32600000000000001,"5min":0.314,"15min":0.314},"max":{"1min":4.0449999999999999,"5min":78.497,"15min":86.614999999999995},"last":0.32600000000000001},{"interface":"front","average":{"1min":1.425,"5min":2.1480000000000001,"15min":2.1499999999999999},"min":{"1min":0.38900000000000001,"5min":0.32300000000000001,"15min":0.312},"max":{"1min":4.1909999999999998,"5min":78.774000000000001,"15min":86.543999999999997},"last":0.51700000000000002}]},{"osd":6,"last update":"Mon Feb 5 06:53:31 2024","interfaces":[{"interface":"back","average":{"1min":4.2809999999999997,"5min":2.1200000000000001,"15min":1.5029999999999999},"min":{"1min":0.372,"5min":0.36799999999999999,"15min":0.315},"max":{"1min":53.332000000000001,"5min":53.332000000000001,"15min":53.332000000000001},"last":0.36599999999999999},{"interface":"front","average":{"1min":4.4530000000000003,"5min":2.1539999999999999,"15min":1.8},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.316},"max":{"1min":53.320999999999998,"5min":53.320999999999998,"15min":53.320999999999998},"last":0.42599999999999999}]},{"osd":7,"last update":"Mon Feb 5 06:53:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8620000000000001,"5min":1.653,"15min":1.623},"min":{"1min":0.44500000000000001,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":5.04,"5min":9.3800000000000008,"15min":45.243000000000002},"last":0.46000000000000002},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.4510000000000001,"15min":1.5169999999999999},"min":{"1min":0.48399999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":5.593,"5min":7.5449999999999999,"15min":45.171999999999997},"last":1.2270000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149969,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2095324,"kb_used_data":1225620,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":91649828,"statfs":{"total":95995035648,"available":93849423872,"internally_reserved":0,"allocated":1255034880,"data_stored":1254674373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":890567111},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:53:27 2024","interfaces":[{"interface":"back","average":{"1min":1.694,"5min":1.9319999999999999,"15min":1.8380000000000001},"min":{"1min":0.26100000000000001,"5min":0.20200000000000001,"15min":0.184},"max":{"1min":9.8849999999999998,"5min":38.686999999999998,"15min":53.656999999999996},"last":0.68000000000000005},{"interface":"front","average":{"1min":1.657,"5min":2.298,"15min":2.0110000000000001},"min":{"1min":0.38700000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":5.7210000000000001,"5min":35.506999999999998,"15min":53.338000000000001},"last":0.48299999999999998}]},{"osd":1,"last update":"Mon Feb 5 06:53:27 2024","interfaces":[{"interface":"back","average":{"1min":1.917,"5min":2.0339999999999998,"15min":1.7330000000000001},"min":{"1min":0.41899999999999998,"5min":0.22500000000000001,"15min":0.20499999999999999},"max":{"1min":8.9220000000000006,"5min":24.343,"15min":24.343},"last":0.54600000000000004},{"interface":"front","average":{"1min":1.8720000000000001,"5min":1.863,"15min":1.756},"min":{"1min":0.442,"5min":0.22600000000000001,"15min":0.215},"max":{"1min":6.4189999999999996,"5min":23.300999999999998,"15min":23.300999999999998},"last":3.613}]},{"osd":2,"last update":"Mon Feb 5 06:53:27 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.962,"15min":1.806},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.23000000000000001},"max":{"1min":4.1879999999999997,"5min":14.746,"15min":41.529000000000003},"last":3.109},{"interface":"front","average":{"1min":1.7509999999999999,"5min":1.744,"15min":1.6459999999999999},"min":{"1min":0.36099999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":10.217000000000001,"5min":14.734,"15min":41.759999999999998},"last":1.6950000000000001}]},{"osd":4,"last update":"Mon Feb 5 06:53:43 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.4099999999999999,"15min":1.268},"min":{"1min":0.41099999999999998,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":5.1100000000000003,"5min":16.59,"15min":16.59},"last":0.60099999999999998},{"interface":"front","average":{"1min":1.256,"5min":1.4870000000000001,"15min":1.4690000000000001},"min":{"1min":0.41699999999999998,"5min":0.29499999999999998,"15min":0.28599999999999998},"max":{"1min":4.5789999999999997,"5min":8.7260000000000009,"15min":19.359999999999999},"last":0.51700000000000002}]},{"osd":5,"last update":"Mon Feb 5 06:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":1.361,"15min":1.4530000000000001},"min":{"1min":0.33000000000000002,"5min":0.32200000000000001,"15min":0.29299999999999998},"max":{"1min":2.238,"5min":11.458,"15min":20.257999999999999},"last":3.5760000000000001},{"interface":"front","average":{"1min":1.048,"5min":1.554,"15min":1.3959999999999999},"min":{"1min":0.40999999999999998,"5min":0.33700000000000002,"15min":0.32800000000000001},"max":{"1min":3.6480000000000001,"5min":16.629999999999999,"15min":16.629999999999999},"last":0.65400000000000003}]},{"osd":6,"last update":"Mon Feb 5 06:53:43 2024","interfaces":[{"interface":"back","average":{"1min":1.7490000000000001,"5min":1.835,"15min":1.522},"min":{"1min":0.42399999999999999,"5min":0.33100000000000002,"15min":0.28100000000000003},"max":{"1min":7.0279999999999996,"5min":16.643000000000001,"15min":16.643000000000001},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.24,"5min":1.7210000000000001,"15min":1.49},"min":{"1min":0.41199999999999998,"5min":0.35299999999999998,"15min":0.318},"max":{"1min":2.448,"5min":12.75,"15min":12.75},"last":2.7069999999999999}]},{"osd":7,"last update":"Mon Feb 5 06:54:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0110000000000001,"5min":1.671,"15min":1.462},"min":{"1min":0.46200000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":11.824,"5min":11.824,"15min":11.824},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.96099999999999997,"5min":1.5840000000000001,"15min":1.4910000000000001},"min":{"1min":0.47499999999999998,"5min":0.39100000000000001,"15min":0.35899999999999999},"max":{"1min":2.2669999999999999,"5min":16.611999999999998,"15min":16.611999999999998},"last":2.6709999999999998}]}]},{"osd":4,"up_from":31,"seq":133143986444,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1385180,"kb_used_data":627924,"kb_used_omap":1,"kb_used_meta":757246,"kb_avail":92359972,"statfs":{"total":95995035648,"available":94576611328,"internally_reserved":0,"allocated":642994176,"data_stored":642633371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":775420357},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:53:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.252,"15min":1.2290000000000001},"min":{"1min":0.22900000000000001,"5min":0.216,"15min":0.20799999999999999},"max":{"1min":4.1150000000000002,"5min":7.6299999999999999,"15min":33.372},"last":0.878},{"interface":"front","average":{"1min":1.167,"5min":1.3919999999999999,"15min":1.25},"min":{"1min":0.26400000000000001,"5min":0.26000000000000001,"15min":0.22700000000000001},"max":{"1min":10.130000000000001,"5min":10.130000000000001,"15min":33.323999999999998},"last":0.624}]},{"osd":1,"last update":"Mon Feb 5 06:53:39 2024","interfaces":[{"interface":"back","average":{"1min":2.6859999999999999,"5min":1.5309999999999999,"15min":1.575},"min":{"1min":0.27900000000000003,"5min":0.254,"15min":0.21299999999999999},"max":{"1min":12.587,"5min":12.587,"15min":53.414000000000001},"last":1.0409999999999999},{"interface":"front","average":{"1min":1.446,"5min":1.3260000000000001,"15min":1.452},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23000000000000001},"max":{"1min":12.41,"5min":12.41,"15min":53.719999999999999},"last":0.42499999999999999}]},{"osd":2,"last update":"Mon Feb 5 06:53:39 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.298,"15min":1.0489999999999999},"min":{"1min":0.318,"5min":0.27300000000000002,"15min":0.26000000000000001},"max":{"1min":3.7440000000000002,"5min":7.5830000000000002,"15min":17.379999999999999},"last":1.121},{"interface":"front","average":{"1min":1.7849999999999999,"5min":1.2729999999999999,"15min":1.2390000000000001},"min":{"1min":0.33900000000000002,"5min":0.28399999999999997,"15min":0.26000000000000001},"max":{"1min":4.7830000000000004,"5min":4.7830000000000004,"15min":18.077000000000002},"last":0.55900000000000005}]},{"osd":3,"last update":"Mon Feb 5 06:53:39 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.3360000000000001,"15min":1.859},"min":{"1min":0.33700000000000002,"5min":0.28499999999999998,"15min":0.252},"max":{"1min":2.7559999999999998,"5min":7.3769999999999998,"15min":44.601999999999997},"last":0.59999999999999998},{"interface":"front","average":{"1min":2.0209999999999999,"5min":1.3839999999999999,"15min":1.865},"min":{"1min":0.307,"5min":0.307,"15min":0.29799999999999999},"max":{"1min":7.6619999999999999,"5min":7.6619999999999999,"15min":44.511000000000003},"last":2.0379999999999998}]},{"osd":5,"last update":"Mon Feb 5 06:54:03 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.86299999999999999,"15min":0.80300000000000005},"min":{"1min":0.26000000000000001,"5min":0.25,"15min":0.24399999999999999},"max":{"1min":2.4409999999999998,"5min":4.649,"15min":5.6059999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.92000000000000004,"15min":0.79200000000000004},"min":{"1min":0.26300000000000001,"5min":0.255,"15min":0.255},"max":{"1min":3.2639999999999998,"5min":7.7290000000000001,"15min":7.7290000000000001},"last":0.45600000000000002}]},{"osd":6,"last update":"Mon Feb 5 06:53:39 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":1.022,"15min":0.93600000000000005},"min":{"1min":0.33800000000000002,"5min":0.29099999999999998,"15min":0.246},"max":{"1min":2.782,"5min":7.6950000000000003,"15min":11.868},"last":0.503},{"interface":"front","average":{"1min":0.746,"5min":0.93999999999999995,"15min":0.89100000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.29499999999999998},"max":{"1min":2.8039999999999998,"5min":4.0750000000000002,"15min":11.787000000000001},"last":0.96899999999999997}]},{"osd":7,"last update":"Mon Feb 5 06:53:45 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.85499999999999998,"15min":0.78000000000000003},"min":{"1min":0.33400000000000002,"5min":0.32700000000000001,"15min":0.27400000000000002},"max":{"1min":3.9980000000000002,"5min":4.1369999999999996,"15min":5.1500000000000004},"last":1.008},{"interface":"front","average":{"1min":1.0880000000000001,"5min":0.94499999999999995,"15min":0.85899999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.28899999999999998},"max":{"1min":5.4809999999999999,"5min":5.4809999999999999,"15min":5.5890000000000004},"last":0.52500000000000002}]}]},{"osd":5,"up_from":40,"seq":171798692102,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1579528,"kb_used_data":709824,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":92165624,"statfs":{"total":95995035648,"available":94377598976,"internally_reserved":0,"allocated":726859776,"data_stored":726520180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":890567110},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,8,3,9,5],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:53:57 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":1.7330000000000001,"15min":1.603},"min":{"1min":0.29399999999999998,"5min":0.23100000000000001,"15min":0.223},"max":{"1min":2.6589999999999998,"5min":22.326000000000001,"15min":55.704999999999998},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.996,"5min":1.5509999999999999,"15min":1.6000000000000001},"min":{"1min":0.27300000000000002,"5min":0.26700000000000002,"15min":0.22600000000000001},"max":{"1min":2.7810000000000001,"5min":22.247,"15min":56.789000000000001},"last":2.8100000000000001}]},{"osd":1,"last update":"Mon Feb 5 06:53:57 2024","interfaces":[{"interface":"back","average":{"1min":5.0090000000000003,"5min":5.4539999999999997,"15min":2.8479999999999999},"min":{"1min":0.255,"5min":0.248,"15min":0.248},"max":{"1min":77.811999999999998,"5min":317.53399999999999,"15min":317.53399999999999},"last":1.8660000000000001},{"interface":"front","average":{"1min":5.3339999999999996,"5min":5.5670000000000002,"15min":2.867},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":77.373999999999995,"5min":317.80500000000001,"15min":317.80500000000001},"last":2.0049999999999999}]},{"osd":2,"last update":"Mon Feb 5 06:53:57 2024","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.6399999999999999,"15min":1.3680000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.23000000000000001},"max":{"1min":7.0789999999999997,"5min":22.635999999999999,"15min":30.052},"last":0.67500000000000004},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.141,"15min":1.175},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":4.8890000000000002,"5min":7.0540000000000003,"15min":30.472999999999999},"last":0.60799999999999998}]},{"osd":3,"last update":"Mon Feb 5 06:53:57 2024","interfaces":[{"interface":"back","average":{"1min":3.1819999999999999,"5min":2.282,"15min":2.0289999999999999},"min":{"1min":0.308,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":43.442999999999998,"5min":43.442999999999998,"15min":62.508000000000003},"last":0.35199999999999998},{"interface":"front","average":{"1min":3.1379999999999999,"5min":2.117,"15min":2.1030000000000002},"min":{"1min":0.33300000000000002,"5min":0.29899999999999999,"15min":0.27400000000000002},"max":{"1min":43.545000000000002,"5min":43.545000000000002,"15min":62.460000000000001},"last":3.3500000000000001}]},{"osd":4,"last update":"Mon Feb 5 06:53:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.82499999999999996,"15min":0.77200000000000002},"min":{"1min":0.25700000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":2.7669999999999999,"5min":9.3629999999999995,"15min":9.3629999999999995},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.89600000000000002,"15min":0.80100000000000005},"min":{"1min":0.26900000000000002,"5min":0.25600000000000001,"15min":0.252},"max":{"1min":2.4660000000000002,"5min":9.1839999999999993,"15min":9.1839999999999993},"last":0.38600000000000001}]},{"osd":6,"last update":"Mon Feb 5 06:53:35 2024","interfaces":[{"interface":"back","average":{"1min":3.8860000000000001,"5min":1.7,"15min":1.1299999999999999},"min":{"1min":0.35599999999999998,"5min":0.27700000000000002,"15min":0.23799999999999999},"max":{"1min":68.316999999999993,"5min":68.316999999999993,"15min":68.316999999999993},"last":0.374},{"interface":"front","average":{"1min":4.1790000000000003,"5min":1.7949999999999999,"15min":1.196},"min":{"1min":0.371,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":70.352000000000004,"5min":70.352000000000004,"15min":70.352000000000004},"last":0.432}]},{"osd":7,"last update":"Mon Feb 5 06:53:57 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.95999999999999996,"15min":0.96899999999999997},"min":{"1min":0.33600000000000002,"5min":0.317,"15min":0.25600000000000001},"max":{"1min":3.6000000000000001,"5min":11.691000000000001,"15min":11.691000000000001},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.90500000000000003,"15min":0.91800000000000004},"min":{"1min":0.373,"5min":0.34399999999999997,"15min":0.27100000000000002},"max":{"1min":2.8250000000000002,"5min":4.9939999999999998,"15min":6.5839999999999996},"last":0.33200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":794099712,"data_stored":794099712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1404370944,"data_stored":1404370944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1073876992,"data_stored":1073872920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1237385216,"data_stored":1237385216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":625741824,"data_stored":625737752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":723648512,"data_stored":723648512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1438584832,"data_stored":1438580760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1093074944,"data_stored":1093074944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T06:54:20.242 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-05T06:54:20.242 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T06:54:20.242 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-02-05T06:54:20.243 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 1.000000 2024-02-05T06:54:20.243 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 3 1 2024-02-05T06:54:20.276 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:20.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:20.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:20.364 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:54:20.377 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:20.445 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:20.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:20.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:20.546 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:20.592 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:20.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:20.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:20.694 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:20.743 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:20.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:20.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:20.845 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:20.877 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:20.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:20.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:20.978 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:21.010 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:21.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:21.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:21.112 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:21.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:21.149 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:21.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:21.250 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:21.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:21.287 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:21.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:21.334 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is set 2024-02-05T06:54:21.388 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:21.418 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:21.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:21.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:21.520 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:21.550 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:21.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:21.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:21.564 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:21 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/164780625' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:54:21.564 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:21 smithi028 ceph-mon[37984]: pgmap v998: 168 pgs: 168 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 857 B/s rd, 0 B/s wr, 987 op/s 2024-02-05T06:54:21.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:21 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/164780625' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:54:21.564 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:21 smithi028 ceph-mon[29442]: pgmap v998: 168 pgs: 168 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 857 B/s rd, 0 B/s wr, 987 op/s 2024-02-05T06:54:21.652 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:21.690 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:21.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:21.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:21.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:21 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/164780625' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:54:21.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:21 smithi167 ceph-mon[32774]: pgmap v998: 168 pgs: 168 active+clean; 2.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 857 B/s rd, 0 B/s wr, 987 op/s 2024-02-05T06:54:21.791 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:21.828 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:21.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:21.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:21.929 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:21.937 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:54:21.989 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:21.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:21.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:22.091 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:22.120 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:22.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:22.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:22.221 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:22.256 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:22.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:22.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:22.357 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:22.393 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:22.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:22.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:22.495 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:22.528 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:22.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:22.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:22.565 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:22 smithi167 ceph-mon[32774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:54:22.565 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:22 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/164780625' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:54:22.565 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:22 smithi167 ceph-mon[32774]: osdmap e473: 8 total, 8 up, 8 in 2024-02-05T06:54:22.575 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:22 smithi028 ceph-mon[37984]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:54:22.575 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:22 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/164780625' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:54:22.575 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:22 smithi028 ceph-mon[37984]: osdmap e473: 8 total, 8 up, 8 in 2024-02-05T06:54:22.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:22 smithi028 ceph-mon[29442]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:54:22.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:22 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/164780625' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-02-05T06:54:22.575 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:22 smithi028 ceph-mon[29442]: osdmap e473: 8 total, 8 up, 8 in 2024-02-05T06:54:22.576 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:54:22 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:54:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:54:22.630 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:22.658 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:22.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:22.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:22.760 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:22.801 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:22.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:22.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:22.903 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:22.926 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:22.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:22.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:23.028 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:23.054 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:54:23.054 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:54:23.068 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:23.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:23.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:23.170 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:23.208 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:23.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:23.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:23.310 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:23.342 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:23.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:23.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:23.444 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:23.485 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:23.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:23.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:23.586 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:23.609 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:23.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:23.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:23.612 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:23 smithi167 ceph-mon[32774]: pgmap v1000: 168 pgs: 168 active+clean; 2.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-02-05T06:54:23.687 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:23 smithi028 ceph-mon[37984]: pgmap v1000: 168 pgs: 168 active+clean; 2.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-02-05T06:54:23.687 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:23 smithi028 ceph-mon[29442]: pgmap v1000: 168 pgs: 168 active+clean; 2.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-02-05T06:54:23.711 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:23.750 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:23.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:23.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:23.852 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:23.889 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:23.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:23.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:23.983 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:54:23 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:54:23.991 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:24.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:24.035 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:24.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:24.048 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-02-05T06:54:24.136 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:24.171 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:24.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:24.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:24.272 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:24.288 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:54:24.288 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:54:24.288 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:54:24.288 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:54:24.288 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:54:24.288 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:54:24.289 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:54:24.289 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:54:24.289 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:54:24.289 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:54:24.289 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:54:24.289 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:54:24.289 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:54:24.289 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:54:24.289 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:54:24.289 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:54:24.305 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:24.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:24.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:24.339 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:24 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/202643005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:54:24.407 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:24.428 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:54:24.449 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:24.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:24.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:24.551 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:24.584 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:24.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:24.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:24.644 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:24 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:54:24.687 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:24.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:24 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/202643005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:54:24.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:24 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:54:24.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:24 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/202643005' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:54:24.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:24 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:54:24.722 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:24.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:24.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:24.823 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:24.860 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:24.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:24.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:24.961 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:24.998 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:24.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:24.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:25.099 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:25.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:25.131 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:25.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:25.232 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:25.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:25.272 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:25.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:25.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:25.407 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:25.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:25.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:25.509 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:25.540 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:25.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:25.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:25.641 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:25.676 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:25.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:25.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:25.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:25 smithi028 ceph-mon[37984]: pgmap v1001: 168 pgs: 168 active+clean; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-02-05T06:54:25.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:25 smithi028 ceph-mon[29442]: pgmap v1001: 168 pgs: 168 active+clean; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-02-05T06:54:25.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:25 smithi167 ceph-mon[32774]: pgmap v1001: 168 pgs: 168 active+clean; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-02-05T06:54:25.778 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:25.808 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:25.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:25.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:25.909 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:25.945 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:25.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:25.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:26.047 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:26.077 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:26.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:26.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:26.178 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:26.218 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:26.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:26.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:26.320 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:26.354 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:26.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:26.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:26.455 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:26.478 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:26.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:26.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:26.580 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:26.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:26.635 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:26.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:26.736 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:26.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:26.766 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:26.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:26.867 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:26.891 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:26.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:26.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:26.943 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:54:26.943 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-02-05T06:54:26.944 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-02-05T06:54:26.944 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-02-05T06:54:26.944 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:54:26.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:27.015 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:27.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:27.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:27.117 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:27.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:27.154 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:27.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:27.255 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:27.292 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:27.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:27.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:27.394 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:27.430 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:27.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:27.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:27.532 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:27.569 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:27.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:27.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:27.671 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:27.711 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:27.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:27.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:27.722 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-02-05T06:54:27.813 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:27.871 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:27.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:27.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:27.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:27 smithi028 ceph-mon[37984]: pgmap v1002: 168 pgs: 168 active+clean; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-02-05T06:54:27.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:27 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:54:27.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:27 smithi028 ceph-mon[29442]: pgmap v1002: 168 pgs: 168 active+clean; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-02-05T06:54:27.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:27 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:54:27.973 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:27.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:27 smithi167 ceph-mon[32774]: pgmap v1002: 168 pgs: 168 active+clean; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s 2024-02-05T06:54:27.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:27 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:54:28.035 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:28.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:28.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:28.136 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:54:28.138 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:28.173 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:28.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:28.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:28.275 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:28.313 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:28.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:28.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:28.415 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:28.452 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:28.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:28.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:28.554 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:28.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:28.585 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:28.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:28.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:28.721 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:28.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:28.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:28.823 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:28.855 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:28.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:28.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:28.957 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:28.989 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:28.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:28.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:29.091 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:29.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:29.128 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:29.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:29.229 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:29.259 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:29.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:29.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:29.361 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:29.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:29.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:29.501 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:29.538 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:29.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:29.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:29.639 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:29.671 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:29.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:29.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:29.772 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:29.810 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:29.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:29.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:29.856 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T06:54:29.901 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:29 smithi028 ceph-mon[29442]: pgmap v1003: 168 pgs: 168 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-05T06:54:29.902 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:29 smithi028 ceph-mon[37984]: pgmap v1003: 168 pgs: 168 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-05T06:54:29.911 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:29.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:29.969 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:29.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:29.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:29 smithi167 ceph-mon[32774]: pgmap v1003: 168 pgs: 168 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-05T06:54:30.070 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:30.108 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:30.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:30.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:30.197 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:54:30.197 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-02-05T06:54:30.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-02-05T06:54:30.197 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-02-05T06:54:30.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.197 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.45587.0:209921 10.c e408/345 10:329d43f6:::benchmark_data_smithi028_154709_object88920:head v 408'12862, mlcod=408'12860)", 2024-02-05T06:54:30.197 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:50:33.136656+0000", 2024-02-05T06:54:30.197 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 237.05162985800001, 2024-02-05T06:54:30.197 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.035684118000000001, 2024-02-05T06:54:30.197 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:54:30.197 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:54:30.197 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:54:30.197 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:54:30.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.136656+0000", 2024-02-05T06:54:30.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:54:30.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.136656+0000", 2024-02-05T06:54:30.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.198 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.198 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:54:30.198 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.136657+0000", 2024-02-05T06:54:30.198 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.367e-06 2024-02-05T06:54:30.198 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:54:30.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.136664+0000", 2024-02-05T06:54:30.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9859999999999997e-06 2024-02-05T06:54:30.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:54:30.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.136675+0000", 2024-02-05T06:54:30.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0749e-05 2024-02-05T06:54:30.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.199 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.199 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:54:30.199 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.136682+0000", 2024-02-05T06:54:30.199 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5619999999999998e-06 2024-02-05T06:54:30.199 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:54:30.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.136747+0000", 2024-02-05T06:54:30.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5055000000000006e-05 2024-02-05T06:54:30.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:54:30.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.137416+0000", 2024-02-05T06:54:30.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00066860599999999997 2024-02-05T06:54:30.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.200 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.200 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:54:30.200 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.172293+0000", 2024-02-05T06:54:30.200 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.034877189000000003 2024-02-05T06:54:30.200 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.201 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:54:30.201 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.172340+0000", 2024-02-05T06:54:30.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6604000000000003e-05 2024-02-05T06:54:30.201 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.201 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:54:30.201 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.201 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.201 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.201 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.45587.0:209929 10.e e408/345 10:77891d7c:::benchmark_data_smithi028_154709_object88928:head v 408'13330, mlcod=408'13328)", 2024-02-05T06:54:30.201 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:50:33.144148+0000", 2024-02-05T06:54:30.201 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 237.04413762999999, 2024-02-05T06:54:30.201 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.034022338999999999, 2024-02-05T06:54:30.201 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:54:30.202 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:54:30.202 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:54:30.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:54:30.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.144148+0000", 2024-02-05T06:54:30.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.202 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.202 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:54:30.202 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.144148+0000", 2024-02-05T06:54:30.202 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.202 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:54:30.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.144149+0000", 2024-02-05T06:54:30.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.392e-06 2024-02-05T06:54:30.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:54:30.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.144158+0000", 2024-02-05T06:54:30.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5080000000000005e-06 2024-02-05T06:54:30.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.203 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.203 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:54:30.203 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.144167+0000", 2024-02-05T06:54:30.203 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.1870000000000003e-06 2024-02-05T06:54:30.203 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:54:30.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.144174+0000", 2024-02-05T06:54:30.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.182e-06 2024-02-05T06:54:30.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:54:30.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.144258+0000", 2024-02-05T06:54:30.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4027e-05 2024-02-05T06:54:30.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.204 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.204 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:54:30.204 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.144279+0000", 2024-02-05T06:54:30.204 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0925999999999999e-05 2024-02-05T06:54:30.204 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:54:30.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.177334+0000", 2024-02-05T06:54:30.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.033054596999999998 2024-02-05T06:54:30.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.205 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:54:30.205 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.178170+0000", 2024-02-05T06:54:30.205 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00083651999999999999 2024-02-05T06:54:30.205 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.205 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:54:30.205 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.205 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.205 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.205 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.45587.0:209933 10.e e408/345 10:7ff443b1:::benchmark_data_smithi028_154709_object88932:head v 408'13331, mlcod=408'13328)", 2024-02-05T06:54:30.205 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:50:33.145780+0000", 2024-02-05T06:54:30.205 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 237.042505517, 2024-02-05T06:54:30.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.034673046999999999, 2024-02-05T06:54:30.206 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:54:30.206 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:54:30.206 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:54:30.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:54:30.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.145780+0000", 2024-02-05T06:54:30.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:54:30.206 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.145780+0000", 2024-02-05T06:54:30.206 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.206 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.206 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.206 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:54:30.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.145781+0000", 2024-02-05T06:54:30.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.15e-06 2024-02-05T06:54:30.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:54:30.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.145788+0000", 2024-02-05T06:54:30.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2099999999999996e-06 2024-02-05T06:54:30.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:54:30.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.172299+0000", 2024-02-05T06:54:30.207 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.026510617 2024-02-05T06:54:30.207 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.207 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.207 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:54:30.207 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.172311+0000", 2024-02-05T06:54:30.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1578e-05 2024-02-05T06:54:30.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:54:30.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.174930+0000", 2024-02-05T06:54:30.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00261911 2024-02-05T06:54:30.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:54:30.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.174949+0000", 2024-02-05T06:54:30.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9046e-05 2024-02-05T06:54:30.208 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.208 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.208 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:54:30.208 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.180415+0000", 2024-02-05T06:54:30.208 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0054665069999999998 2024-02-05T06:54:30.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.209 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:54:30.209 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.180453+0000", 2024-02-05T06:54:30.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7829e-05 2024-02-05T06:54:30.209 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.209 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:54:30.209 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.209 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.209 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.209 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.45587.0:209938 10.c e408/345 10:379d7ec8:::benchmark_data_smithi028_154709_object88937:head v 408'12864, mlcod=408'12860)", 2024-02-05T06:54:30.209 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:50:33.147866+0000", 2024-02-05T06:54:30.209 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 237.04041939999999, 2024-02-05T06:54:30.209 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.030309638999999999, 2024-02-05T06:54:30.209 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:54:30.209 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:54:30.210 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:30.210 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:54:30.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:54:30.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.147866+0000", 2024-02-05T06:54:30.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:54:30.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.147866+0000", 2024-02-05T06:54:30.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.211 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.211 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:54:30.211 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.147868+0000", 2024-02-05T06:54:30.211 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.595e-06 2024-02-05T06:54:30.211 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:54:30.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.147876+0000", 2024-02-05T06:54:30.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5059999999999995e-06 2024-02-05T06:54:30.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:54:30.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.172293+0000", 2024-02-05T06:54:30.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024416473000000001 2024-02-05T06:54:30.212 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.212 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.212 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:54:30.212 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.172306+0000", 2024-02-05T06:54:30.212 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.363e-05 2024-02-05T06:54:30.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:54:30.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.172484+0000", 2024-02-05T06:54:30.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00017794800000000001 2024-02-05T06:54:30.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:54:30.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.173493+0000", 2024-02-05T06:54:30.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0010082489999999999 2024-02-05T06:54:30.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.213 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.213 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:54:30.213 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.178147+0000", 2024-02-05T06:54:30.213 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0046546130000000002 2024-02-05T06:54:30.213 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.214 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:54:30.214 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.178176+0000", 2024-02-05T06:54:30.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.8625000000000001e-05 2024-02-05T06:54:30.214 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.214 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:54:30.214 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.214 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.214 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.214 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.45587.0:210825 10.c e408/345 10:3727982e:::benchmark_data_smithi028_154709_object89824:head v 408'12922, mlcod=408'12917)", 2024-02-05T06:54:30.214 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:50:33.853962+0000", 2024-02-05T06:54:30.214 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 236.334323969, 2024-02-05T06:54:30.214 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.034219205000000003, 2024-02-05T06:54:30.214 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:54:30.215 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:54:30.215 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:54:30.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:54:30.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.853962+0000", 2024-02-05T06:54:30.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:54:30.215 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.853962+0000", 2024-02-05T06:54:30.215 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.215 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.215 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.215 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:54:30.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.853963+0000", 2024-02-05T06:54:30.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.051e-06 2024-02-05T06:54:30.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:54:30.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.853976+0000", 2024-02-05T06:54:30.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3429e-05 2024-02-05T06:54:30.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.216 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:54:30.216 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.853990+0000", 2024-02-05T06:54:30.216 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4197e-05 2024-02-05T06:54:30.216 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.216 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:54:30.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.853998+0000", 2024-02-05T06:54:30.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2690000000000002e-06 2024-02-05T06:54:30.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:54:30.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.861481+0000", 2024-02-05T06:54:30.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0074829839999999998 2024-02-05T06:54:30.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.217 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.217 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:54:30.217 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.861498+0000", 2024-02-05T06:54:30.217 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.783e-05 2024-02-05T06:54:30.217 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:54:30.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.888090+0000", 2024-02-05T06:54:30.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.02659166 2024-02-05T06:54:30.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.218 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:54:30.218 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:50:33.888181+0000", 2024-02-05T06:54:30.218 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.0785000000000006e-05 2024-02-05T06:54:30.218 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.218 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:54:30.218 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.218 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.218 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.218 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.45878.0:42086 11.7 e424/418 11:e9f7bdb4:::benchmark_data_smithi028_177697_object42085:head v 424'2570, mlcod=424'2568)", 2024-02-05T06:54:30.218 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:51:40.913611+0000", 2024-02-05T06:54:30.219 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 169.274674589, 2024-02-05T06:54:30.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.037406109999999999, 2024-02-05T06:54:30.219 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:54:30.219 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:54:30.219 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:54:30.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:54:30.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:51:40.913611+0000", 2024-02-05T06:54:30.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.219 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.219 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.219 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:54:30.219 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:51:40.913611+0000", 2024-02-05T06:54:30.219 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:54:30.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:51:40.913612+0000", 2024-02-05T06:54:30.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3900000000000004e-07 2024-02-05T06:54:30.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:54:30.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:51:40.913637+0000", 2024-02-05T06:54:30.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5655000000000001e-05 2024-02-05T06:54:30.220 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.220 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.220 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:54:30.220 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:51:40.913644+0000", 2024-02-05T06:54:30.220 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.4559999999999997e-06 2024-02-05T06:54:30.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:54:30.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:51:40.913649+0000", 2024-02-05T06:54:30.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.1050000000000003e-06 2024-02-05T06:54:30.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:54:30.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:51:40.913723+0000", 2024-02-05T06:54:30.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4328999999999999e-05 2024-02-05T06:54:30.221 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.221 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.221 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:54:30.221 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:51:40.913743+0000", 2024-02-05T06:54:30.221 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0071000000000001e-05 2024-02-05T06:54:30.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:54:30.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:51:40.950980+0000", 2024-02-05T06:54:30.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.037236391000000001 2024-02-05T06:54:30.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.222 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:54:30.222 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:51:40.951017+0000", 2024-02-05T06:54:30.222 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7463999999999999e-05 2024-02-05T06:54:30.222 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.222 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:54:30.222 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.222 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.222 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.223 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.45878.0:127540 11.7 e449/418 11:eccb5fa3:::benchmark_data_smithi028_177697_object574:head v 449'7963, mlcod=449'7961)", 2024-02-05T06:54:30.223 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:52:53.323242+0000", 2024-02-05T06:54:30.223 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 96.865043514000007, 2024-02-05T06:54:30.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.047176458999999997, 2024-02-05T06:54:30.223 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:54:30.223 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:54:30.223 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:54:30.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:54:30.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:52:53.323242+0000", 2024-02-05T06:54:30.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.223 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.223 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:54:30.223 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:52:53.323242+0000", 2024-02-05T06:54:30.223 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.223 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:54:30.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:52:53.323244+0000", 2024-02-05T06:54:30.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.609e-06 2024-02-05T06:54:30.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:54:30.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:52:53.323254+0000", 2024-02-05T06:54:30.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0223e-05 2024-02-05T06:54:30.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.224 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.224 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:54:30.224 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:52:53.323264+0000", 2024-02-05T06:54:30.224 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0492000000000001e-05 2024-02-05T06:54:30.224 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:54:30.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:52:53.323272+0000", 2024-02-05T06:54:30.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7349999999999996e-06 2024-02-05T06:54:30.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:54:30.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:52:53.323559+0000", 2024-02-05T06:54:30.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000287087 2024-02-05T06:54:30.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.225 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:54:30.225 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:52:53.323601+0000", 2024-02-05T06:54:30.225 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1962999999999998e-05 2024-02-05T06:54:30.225 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.225 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:54:30.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:52:53.370382+0000", 2024-02-05T06:54:30.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.046780414999999999 2024-02-05T06:54:30.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.226 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:54:30.226 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:52:53.370419+0000", 2024-02-05T06:54:30.226 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.6934999999999998e-05 2024-02-05T06:54:30.226 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.226 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:54:30.226 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.226 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.226 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.226 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(osd.5.0:132197 11.7 e466/418 11:e0695d3a:::benchmark_data_smithi028_177697_object124773:head v 466'11660, mlcod=466'11648)", 2024-02-05T06:54:30.227 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:53:49.307316+0000", 2024-02-05T06:54:30.227 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 40.880969853000003, 2024-02-05T06:54:30.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.030995987999999999, 2024-02-05T06:54:30.227 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:54:30.227 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:54:30.227 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:54:30.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:54:30.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:49.307316+0000", 2024-02-05T06:54:30.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.227 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.227 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:54:30.227 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:49.307316+0000", 2024-02-05T06:54:30.227 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.227 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:54:30.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:49.307317+0000", 2024-02-05T06:54:30.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3120000000000001e-06 2024-02-05T06:54:30.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:54:30.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:49.307325+0000", 2024-02-05T06:54:30.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9389999999999997e-06 2024-02-05T06:54:30.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.228 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.228 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:54:30.228 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:49.307339+0000", 2024-02-05T06:54:30.228 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3934000000000001e-05 2024-02-05T06:54:30.228 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:54:30.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:49.307346+0000", 2024-02-05T06:54:30.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3649999999999998e-06 2024-02-05T06:54:30.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:54:30.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:49.312323+0000", 2024-02-05T06:54:30.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0049764470000000002 2024-02-05T06:54:30.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.229 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.229 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:54:30.229 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:49.312343+0000", 2024-02-05T06:54:30.229 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0137999999999999e-05 2024-02-05T06:54:30.229 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:54:30.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:49.338234+0000", 2024-02-05T06:54:30.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025891009999999999 2024-02-05T06:54:30.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.230 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:54:30.230 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:49.338312+0000", 2024-02-05T06:54:30.230 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7843000000000007e-05 2024-02-05T06:54:30.230 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.230 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:54:30.230 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.230 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.230 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.230 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(osd.5.0:132198 11.7 e466/418 11:e06a3f33:::benchmark_data_smithi028_177697_object116659:head v 466'11661, mlcod=466'11648)", 2024-02-05T06:54:30.230 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:53:49.310032+0000", 2024-02-05T06:54:30.231 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 40.878253088000001, 2024-02-05T06:54:30.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.028309239, 2024-02-05T06:54:30.231 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:54:30.231 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:54:30.231 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:54:30.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:54:30.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:49.310032+0000", 2024-02-05T06:54:30.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.231 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.231 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:54:30.231 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:49.310032+0000", 2024-02-05T06:54:30.231 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.231 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:54:30.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:49.310034+0000", 2024-02-05T06:54:30.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6449999999999999e-06 2024-02-05T06:54:30.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:54:30.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:49.310044+0000", 2024-02-05T06:54:30.232 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.5489999999999995e-06 2024-02-05T06:54:30.232 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.232 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.232 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:54:30.232 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:49.310055+0000", 2024-02-05T06:54:30.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1260000000000001e-05 2024-02-05T06:54:30.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:54:30.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:49.310063+0000", 2024-02-05T06:54:30.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3990000000000004e-06 2024-02-05T06:54:30.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:54:30.233 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:49.316657+0000", 2024-02-05T06:54:30.233 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0065939919999999999 2024-02-05T06:54:30.233 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.233 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.233 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:54:30.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:49.316676+0000", 2024-02-05T06:54:30.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8828999999999999e-05 2024-02-05T06:54:30.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:54:30.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:49.338318+0000", 2024-02-05T06:54:30.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021641474000000001 2024-02-05T06:54:30.234 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.234 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.234 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:54:30.234 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:49.338342+0000", 2024-02-05T06:54:30.234 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.4091e-05 2024-02-05T06:54:30.234 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.234 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:54:30.235 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.235 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(osd.5.0:133347 11.7 e468/418 11:e40194e6:::benchmark_data_smithi028_177697_object117297:head v 468'12810, mlcod=468'12793)", 2024-02-05T06:54:30.235 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:53:53.691702+0000", 2024-02-05T06:54:30.235 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 36.496583323999999, 2024-02-05T06:54:30.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029508171, 2024-02-05T06:54:30.235 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:54:30.235 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:54:30.235 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:54:30.235 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.235 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:54:30.235 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:53.691702+0000", 2024-02-05T06:54:30.235 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.235 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:54:30.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:53.691702+0000", 2024-02-05T06:54:30.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:54:30.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:53.691703+0000", 2024-02-05T06:54:30.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2300000000000001e-06 2024-02-05T06:54:30.236 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.236 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.236 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:54:30.236 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:53.691711+0000", 2024-02-05T06:54:30.236 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5399999999999998e-06 2024-02-05T06:54:30.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:54:30.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:53.691724+0000", 2024-02-05T06:54:30.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3413e-05 2024-02-05T06:54:30.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:54:30.237 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:53.691731+0000", 2024-02-05T06:54:30.237 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9650000000000002e-06 2024-02-05T06:54:30.237 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.237 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.237 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:54:30.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:53.693861+0000", 2024-02-05T06:54:30.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021292070000000001 2024-02-05T06:54:30.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:54:30.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:53.693880+0000", 2024-02-05T06:54:30.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9704e-05 2024-02-05T06:54:30.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:54:30.238 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:53.721145+0000", 2024-02-05T06:54:30.238 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.027264371999999999 2024-02-05T06:54:30.238 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.238 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.238 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:54:30.239 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:53.721210+0000", 2024-02-05T06:54:30.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.5740000000000004e-05 2024-02-05T06:54:30.239 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.239 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:54:30.239 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.239 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.239 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.45878.0:191570 11.7 e468/418 11:e2f03dcf:::benchmark_data_smithi028_177697_object64604:head v 468'13084, mlcod=468'13080)", 2024-02-05T06:54:30.239 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:53:54.736898+0000", 2024-02-05T06:54:30.239 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 35.451387171, 2024-02-05T06:54:30.239 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029579373999999999, 2024-02-05T06:54:30.239 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:54:30.239 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:54:30.239 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:54:30.239 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:54:30.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:54.736898+0000", 2024-02-05T06:54:30.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:54:30.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:54.736898+0000", 2024-02-05T06:54:30.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.240 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:54:30.240 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:54.736900+0000", 2024-02-05T06:54:30.240 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4300000000000001e-06 2024-02-05T06:54:30.240 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.240 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:54:30.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:54.736907+0000", 2024-02-05T06:54:30.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8390000000000001e-06 2024-02-05T06:54:30.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:54:30.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:54.736921+0000", 2024-02-05T06:54:30.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4031e-05 2024-02-05T06:54:30.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:54:30.241 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:54.736928+0000", 2024-02-05T06:54:30.241 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.576e-06 2024-02-05T06:54:30.241 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.241 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.241 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:54:30.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:54.737012+0000", 2024-02-05T06:54:30.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.4116999999999994e-05 2024-02-05T06:54:30.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:54:30.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:54.737031+0000", 2024-02-05T06:54:30.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8825999999999999e-05 2024-02-05T06:54:30.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:54:30.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:54.766393+0000", 2024-02-05T06:54:30.242 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029361491999999999 2024-02-05T06:54:30.242 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.242 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.242 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:54:30.242 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:54.766478+0000", 2024-02-05T06:54:30.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5062999999999995e-05 2024-02-05T06:54:30.243 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.243 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:54:30.243 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.243 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.243 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.45878.0:196724 11.7 e469/418 11:e04ff1a0:::benchmark_data_smithi028_177697_object69758:head v 469'14289, mlcod=469'14282)", 2024-02-05T06:54:30.243 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:53:59.705224+0000", 2024-02-05T06:54:30.243 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 30.483061815999999, 2024-02-05T06:54:30.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.028687317, 2024-02-05T06:54:30.243 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:54:30.243 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:54:30.243 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:54:30.243 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.243 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:54:30.243 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:59.705224+0000", 2024-02-05T06:54:30.243 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:54:30.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:59.705224+0000", 2024-02-05T06:54:30.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:54:30.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:59.705225+0000", 2024-02-05T06:54:30.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3540000000000001e-06 2024-02-05T06:54:30.244 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.244 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.244 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:54:30.244 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:59.705233+0000", 2024-02-05T06:54:30.244 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.8050000000000007e-06 2024-02-05T06:54:30.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:54:30.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:59.705247+0000", 2024-02-05T06:54:30.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.4358999999999999e-05 2024-02-05T06:54:30.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:54:30.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:59.705255+0000", 2024-02-05T06:54:30.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9470000000000003e-06 2024-02-05T06:54:30.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.245 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.245 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:54:30.245 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:59.708887+0000", 2024-02-05T06:54:30.245 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0036322300000000002 2024-02-05T06:54:30.245 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:54:30.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:59.708910+0000", 2024-02-05T06:54:30.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2952999999999999e-05 2024-02-05T06:54:30.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:54:30.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:59.733836+0000", 2024-02-05T06:54:30.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.024925222 2024-02-05T06:54:30.246 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.246 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.246 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:54:30.246 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:59.733911+0000", 2024-02-05T06:54:30.246 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5446999999999998e-05 2024-02-05T06:54:30.246 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.246 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:54:30.246 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.247 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.45878.0:196888 11.7 e469/418 11:e8b23f5b:::benchmark_data_smithi028_177697_object69922:head v 469'14327, mlcod=469'14320)", 2024-02-05T06:54:30.247 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:53:59.852277+0000", 2024-02-05T06:54:30.247 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 30.336008335999999, 2024-02-05T06:54:30.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029354543, 2024-02-05T06:54:30.247 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:54:30.247 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:54:30.247 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:54:30.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:54:30.247 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:59.852277+0000", 2024-02-05T06:54:30.247 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.247 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.247 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.247 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:54:30.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:59.852277+0000", 2024-02-05T06:54:30.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:54:30.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:59.852279+0000", 2024-02-05T06:54:30.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.584e-06 2024-02-05T06:54:30.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:54:30.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:59.852286+0000", 2024-02-05T06:54:30.248 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.041e-06 2024-02-05T06:54:30.248 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.248 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.248 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:54:30.248 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:59.852295+0000", 2024-02-05T06:54:30.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4029999999999996e-06 2024-02-05T06:54:30.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.249 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:54:30.249 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:59.852303+0000", 2024-02-05T06:54:30.249 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9540000000000004e-06 2024-02-05T06:54:30.249 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.249 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:54:30.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:59.852451+0000", 2024-02-05T06:54:30.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014731 2024-02-05T06:54:30.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.250 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:54:30.250 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:59.852495+0000", 2024-02-05T06:54:30.250 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3995000000000001e-05 2024-02-05T06:54:30.250 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.250 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:54:30.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:59.881592+0000", 2024-02-05T06:54:30.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.029097115 2024-02-05T06:54:30.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.251 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:54:30.251 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:59.881632+0000", 2024-02-05T06:54:30.251 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0141000000000001e-05 2024-02-05T06:54:30.251 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.251 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:54:30.251 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.251 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.251 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.251 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(osd.5.0:135066 11.7 e470/418 11:e943140c:::benchmark_data_smithi028_177697_object87711:head v 470'14529, mlcod=470'14514)", 2024-02-05T06:54:30.251 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:54:00.693279+0000", 2024-02-05T06:54:30.251 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 29.495006252, 2024-02-05T06:54:30.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.042712364000000003, 2024-02-05T06:54:30.252 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:54:30.252 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:54:30.252 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:54:30.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:54:30.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.693279+0000", 2024-02-05T06:54:30.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:54:30.252 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.693279+0000", 2024-02-05T06:54:30.252 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.252 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.252 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.252 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:54:30.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.693281+0000", 2024-02-05T06:54:30.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.277e-06 2024-02-05T06:54:30.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:54:30.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.693288+0000", 2024-02-05T06:54:30.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9539999999999998e-06 2024-02-05T06:54:30.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:54:30.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.693303+0000", 2024-02-05T06:54:30.253 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5228999999999999e-05 2024-02-05T06:54:30.253 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.253 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.253 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:54:30.253 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.693310+0000", 2024-02-05T06:54:30.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.464e-06 2024-02-05T06:54:30.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:54:30.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.699281+0000", 2024-02-05T06:54:30.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0059711749999999996 2024-02-05T06:54:30.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:54:30.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.699301+0000", 2024-02-05T06:54:30.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9142000000000001e-05 2024-02-05T06:54:30.254 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.254 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.254 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:54:30.254 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.735952+0000", 2024-02-05T06:54:30.254 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036651230999999999 2024-02-05T06:54:30.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.255 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:54:30.255 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.735992+0000", 2024-02-05T06:54:30.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.9891999999999997e-05 2024-02-05T06:54:30.255 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.255 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:54:30.255 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.255 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.255 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.255 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(osd.5.0:135067 11.7 e470/418 11:e9432ad1:::benchmark_data_smithi028_177697_object123491:head v 470'14530, mlcod=470'14514)", 2024-02-05T06:54:30.255 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:54:00.696299+0000", 2024-02-05T06:54:30.255 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 29.491986464, 2024-02-05T06:54:30.255 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039716673000000001, 2024-02-05T06:54:30.255 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:54:30.255 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:54:30.255 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:54:30.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:54:30.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.696299+0000", 2024-02-05T06:54:30.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:54:30.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.696299+0000", 2024-02-05T06:54:30.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.256 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:54:30.256 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.696300+0000", 2024-02-05T06:54:30.256 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2860000000000001e-06 2024-02-05T06:54:30.256 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.256 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:54:30.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.696307+0000", 2024-02-05T06:54:30.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.567e-06 2024-02-05T06:54:30.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:54:30.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.696316+0000", 2024-02-05T06:54:30.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9609999999999994e-06 2024-02-05T06:54:30.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.257 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.257 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:54:30.257 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.696323+0000", 2024-02-05T06:54:30.257 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.0949999999999998e-06 2024-02-05T06:54:30.257 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:54:30.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.701990+0000", 2024-02-05T06:54:30.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0056673019999999999 2024-02-05T06:54:30.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:54:30.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.702012+0000", 2024-02-05T06:54:30.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1968999999999999e-05 2024-02-05T06:54:30.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.258 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.258 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:54:30.258 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.735994+0000", 2024-02-05T06:54:30.258 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.033982046000000002 2024-02-05T06:54:30.258 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.259 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:54:30.259 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.736016+0000", 2024-02-05T06:54:30.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1447e-05 2024-02-05T06:54:30.259 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.259 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:54:30.259 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.259 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.259 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.259 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(osd.5.0:135068 11.7 e470/418 11:e943faaf:::benchmark_data_smithi028_177697_object81675:head v 470'14531, mlcod=470'14514)", 2024-02-05T06:54:30.259 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:54:00.700934+0000", 2024-02-05T06:54:30.259 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 29.487351183000001, 2024-02-05T06:54:30.259 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039959925, 2024-02-05T06:54:30.259 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:54:30.259 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:54:30.260 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:54:30.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:54:30.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.700934+0000", 2024-02-05T06:54:30.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:54:30.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.700934+0000", 2024-02-05T06:54:30.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.260 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.260 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:54:30.260 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.700936+0000", 2024-02-05T06:54:30.260 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.31e-06 2024-02-05T06:54:30.260 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:54:30.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.700943+0000", 2024-02-05T06:54:30.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4410000000000004e-06 2024-02-05T06:54:30.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:54:30.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.700955+0000", 2024-02-05T06:54:30.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1459e-05 2024-02-05T06:54:30.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.261 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:54:30.261 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.700962+0000", 2024-02-05T06:54:30.261 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9410000000000001e-06 2024-02-05T06:54:30.261 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.261 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:54:30.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.707033+0000", 2024-02-05T06:54:30.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.006071796 2024-02-05T06:54:30.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:54:30.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.707053+0000", 2024-02-05T06:54:30.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9338999999999999e-05 2024-02-05T06:54:30.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:54:30.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.740853+0000", 2024-02-05T06:54:30.262 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.033800561999999999 2024-02-05T06:54:30.262 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.262 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.262 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:54:30.262 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.740894+0000", 2024-02-05T06:54:30.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.1077e-05 2024-02-05T06:54:30.263 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.263 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:54:30.263 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.263 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.263 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(osd.5.0:135069 11.7 e470/418 11:e9455342:::benchmark_data_smithi028_177697_object85635:head v 470'14532, mlcod=470'14514)", 2024-02-05T06:54:30.263 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:54:00.701853+0000", 2024-02-05T06:54:30.263 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 29.486432663999999, 2024-02-05T06:54:30.263 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.039066686000000003, 2024-02-05T06:54:30.263 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:54:30.263 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:54:30.263 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:54:30.263 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.263 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:54:30.263 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.701853+0000", 2024-02-05T06:54:30.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:54:30.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.701853+0000", 2024-02-05T06:54:30.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:54:30.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.701854+0000", 2024-02-05T06:54:30.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.282e-06 2024-02-05T06:54:30.264 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.264 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.264 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:54:30.264 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.701861+0000", 2024-02-05T06:54:30.264 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9859999999999997e-06 2024-02-05T06:54:30.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:54:30.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.701875+0000", 2024-02-05T06:54:30.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3814e-05 2024-02-05T06:54:30.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:54:30.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.701882+0000", 2024-02-05T06:54:30.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1799999999999999e-06 2024-02-05T06:54:30.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.265 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.265 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:54:30.265 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.709720+0000", 2024-02-05T06:54:30.265 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0078381879999999994 2024-02-05T06:54:30.265 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:54:30.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.709740+0000", 2024-02-05T06:54:30.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9675999999999999e-05 2024-02-05T06:54:30.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:54:30.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.740898+0000", 2024-02-05T06:54:30.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.031157482 2024-02-05T06:54:30.266 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.266 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.266 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:54:30.266 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.740920+0000", 2024-02-05T06:54:30.266 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2078000000000001e-05 2024-02-05T06:54:30.266 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.266 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:54:30.267 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.267 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(osd.5.0:135070 11.7 e470/418 11:e948e093:::benchmark_data_smithi028_177697_object117345:head v 470'14533, mlcod=470'14514)", 2024-02-05T06:54:30.267 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:54:00.705986+0000", 2024-02-05T06:54:30.267 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 29.482299889, 2024-02-05T06:54:30.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.034956295999999998, 2024-02-05T06:54:30.267 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:54:30.267 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:54:30.267 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:54:30.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:54:30.267 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.705986+0000", 2024-02-05T06:54:30.267 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.267 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.267 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.267 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:54:30.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.705986+0000", 2024-02-05T06:54:30.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:54:30.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.705987+0000", 2024-02-05T06:54:30.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.471e-06 2024-02-05T06:54:30.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:54:30.268 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.705995+0000", 2024-02-05T06:54:30.268 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0809999999999994e-06 2024-02-05T06:54:30.268 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.268 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.268 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:54:30.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.706004+0000", 2024-02-05T06:54:30.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7290000000000006e-06 2024-02-05T06:54:30.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:54:30.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.706017+0000", 2024-02-05T06:54:30.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3284e-05 2024-02-05T06:54:30.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:54:30.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.712549+0000", 2024-02-05T06:54:30.269 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0065316979999999998 2024-02-05T06:54:30.269 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.269 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.269 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:54:30.269 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.712568+0000", 2024-02-05T06:54:30.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9097999999999999e-05 2024-02-05T06:54:30.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:54:30.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.740922+0000", 2024-02-05T06:54:30.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.028353548999999999 2024-02-05T06:54:30.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.270 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:54:30.270 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.740942+0000", 2024-02-05T06:54:30.270 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0386000000000001e-05 2024-02-05T06:54:30.270 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.270 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:54:30.270 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.270 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.270 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.271 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(osd.5.0:135071 11.7 e470/418 11:e9493e89:::benchmark_data_smithi028_177697_object116942:head v 470'14534, mlcod=470'14514)", 2024-02-05T06:54:30.271 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:54:00.707553+0000", 2024-02-05T06:54:30.271 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 29.480732785000001, 2024-02-05T06:54:30.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.033417369000000002, 2024-02-05T06:54:30.271 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:54:30.271 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:54:30.271 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:54:30.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:54:30.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.707553+0000", 2024-02-05T06:54:30.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.271 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.271 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:54:30.271 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.707553+0000", 2024-02-05T06:54:30.271 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.271 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:54:30.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.707554+0000", 2024-02-05T06:54:30.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.2600000000000001e-07 2024-02-05T06:54:30.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:54:30.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.707559+0000", 2024-02-05T06:54:30.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4750000000000001e-06 2024-02-05T06:54:30.272 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.272 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.272 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:54:30.272 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.707568+0000", 2024-02-05T06:54:30.272 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9509999999999995e-06 2024-02-05T06:54:30.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:54:30.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.707573+0000", 2024-02-05T06:54:30.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3720000000000004e-06 2024-02-05T06:54:30.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:54:30.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.715202+0000", 2024-02-05T06:54:30.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0076290000000000004 2024-02-05T06:54:30.273 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.273 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.273 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:54:30.273 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.715222+0000", 2024-02-05T06:54:30.273 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9752999999999999e-05 2024-02-05T06:54:30.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:54:30.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.740944+0000", 2024-02-05T06:54:30.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.025721999999999998 2024-02-05T06:54:30.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.274 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:54:30.274 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.740970+0000", 2024-02-05T06:54:30.274 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5891999999999999e-05 2024-02-05T06:54:30.274 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.274 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:54:30.274 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.274 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.274 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.274 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(osd.5.0:135072 11.7 e470/418 11:e94b9aa4:::benchmark_data_smithi028_177697_object88391:head v 470'14535, mlcod=470'14514)", 2024-02-05T06:54:30.275 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:54:00.710383+0000", 2024-02-05T06:54:30.275 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 29.477902316000002, 2024-02-05T06:54:30.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.030609331, 2024-02-05T06:54:30.275 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:54:30.275 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:54:30.275 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:54:30.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:54:30.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.710383+0000", 2024-02-05T06:54:30.275 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.275 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.275 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.275 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:54:30.275 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.710383+0000", 2024-02-05T06:54:30.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:30.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:54:30.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.710384+0000", 2024-02-05T06:54:30.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1599999999999999e-06 2024-02-05T06:54:30.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:54:30.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.710392+0000", 2024-02-05T06:54:30.276 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4019999999999996e-06 2024-02-05T06:54:30.276 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.276 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.276 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:54:30.276 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.710402+0000", 2024-02-05T06:54:30.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0057e-05 2024-02-05T06:54:30.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:54:30.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.710409+0000", 2024-02-05T06:54:30.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.2899999999999997e-06 2024-02-05T06:54:30.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:54:30.277 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.719915+0000", 2024-02-05T06:54:30.277 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0095057649999999994 2024-02-05T06:54:30.277 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.277 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.277 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:54:30.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.719935+0000", 2024-02-05T06:54:30.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0412999999999999e-05 2024-02-05T06:54:30.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:54:30.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.740973+0000", 2024-02-05T06:54:30.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021037323 2024-02-05T06:54:30.278 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:30.278 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:30.278 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:54:30.278 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:00.740993+0000", 2024-02-05T06:54:30.278 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9921000000000001e-05 2024-02-05T06:54:30.278 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.278 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:54:30.278 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.279 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:30.279 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:54:30.279 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:54:30.283 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:30.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:30.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:30.385 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:30.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:30.439 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:30.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:30.536 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:54:30.540 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:30.570 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:30.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:30.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:30.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:30.712 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:30.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:30.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:30.814 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:30.861 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:30.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:30.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:30.962 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:30.998 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:30.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:30.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:31.067 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:54:31.099 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:31.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:31.157 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:31.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:31.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:31.305 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:31.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:31.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:31.377 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-02-05T06:54:31.407 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:31.437 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:31.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:31.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:31.539 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:31.577 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:31.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:31.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:31.679 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:31.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:31 smithi028 ceph-mon[29442]: pgmap v1004: 168 pgs: 168 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-05T06:54:31.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:31 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/78534314' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T06:54:31.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:31 smithi028 ceph-mon[29442]: from='client.46055 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T06:54:31.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:31 smithi028 ceph-mon[37984]: pgmap v1004: 168 pgs: 168 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-05T06:54:31.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:31 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/78534314' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T06:54:31.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:31 smithi028 ceph-mon[37984]: from='client.46055 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T06:54:31.720 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:31.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:31.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:31.724 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:31 smithi167 ceph-mon[32774]: pgmap v1004: 168 pgs: 168 active+clean; 1.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-05T06:54:31.724 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:31 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/78534314' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T06:54:31.724 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:31 smithi167 ceph-mon[32774]: from='client.46055 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T06:54:31.766 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-02-05T06:54:31.824 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:31.864 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:31.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:31.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:31.966 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:32.001 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:32.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:32.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:32.103 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:32.140 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:32.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:32.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:32.144 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:54:32.242 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:32.273 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:32.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:32.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:32.375 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:32.409 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:32.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:32.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:32.511 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:32.541 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:32.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:32.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:32.587 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:32 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:54:32.587 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:32 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:54:32.587 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:32 smithi167 ceph-mon[32774]: from='client.46055 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-05T06:54:32.587 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:32 smithi167 ceph-mon[32774]: osdmap e474: 8 total, 8 up, 8 in 2024-02-05T06:54:32.643 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:32.676 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:32.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:32.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:32.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:32 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:54:32.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:32 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:54:32.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:32 smithi028 ceph-mon[29442]: from='client.46055 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-05T06:54:32.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:32 smithi028 ceph-mon[29442]: osdmap e474: 8 total, 8 up, 8 in 2024-02-05T06:54:32.700 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:54:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:54:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:54:32.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:32 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:54:32.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:32 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:54:32.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:32 smithi028 ceph-mon[37984]: from='client.46055 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-05T06:54:32.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:32 smithi028 ceph-mon[37984]: osdmap e474: 8 total, 8 up, 8 in 2024-02-05T06:54:32.778 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:32.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:32.818 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:32.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:32.919 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:32.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:32.954 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:32.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:33.055 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:33.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:33.101 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:33.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:33.202 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:33.232 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:33.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:33.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:33.334 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:33.371 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:33.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:33.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:33.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:33.499 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:33.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:33.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:33.600 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:33.623 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:33 smithi167 ceph-mon[32774]: pgmap v1006: 168 pgs: 168 active+clean; 1.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-05T06:54:33.635 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:33.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:33.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:33.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:33 smithi028 ceph-mon[29442]: pgmap v1006: 168 pgs: 168 active+clean; 1.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-05T06:54:33.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:33 smithi028 ceph-mon[37984]: pgmap v1006: 168 pgs: 168 active+clean; 1.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-02-05T06:54:33.737 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:33.772 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:33.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:33.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:33.874 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:33.901 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:33.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:33.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:33.983 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:54:33 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:54:34.003 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:34.034 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:34.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:34.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:34.135 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:34.168 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:34.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:34.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:34.269 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:34.307 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:34.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:34.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:34.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:34.448 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:34.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:34.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:34.470 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:34 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:54:34.470 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:34 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:54:34.470 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:34 smithi167 ceph-mon[32774]: osdmap e475: 8 total, 8 up, 8 in 2024-02-05T06:54:34.470 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:34 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:54:34.548 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:54:34.550 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:34.579 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:34.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:34.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:34.681 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:34.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:34 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:54:34.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:34 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:54:34.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:34 smithi028 ceph-mon[29442]: osdmap e475: 8 total, 8 up, 8 in 2024-02-05T06:54:34.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:34 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:54:34.684 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:34 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:54:34.685 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:34 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:54:34.685 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:34 smithi028 ceph-mon[37984]: osdmap e475: 8 total, 8 up, 8 in 2024-02-05T06:54:34.685 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:34 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:54:34.721 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:34.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:34.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:34.823 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:34.860 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:34.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:34.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:34.942 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:54:34.942 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-02-05T06:54:34.942 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:34.942 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.45878.0:236554 11.8 e475/418 11:1e42c73a:::benchmark_data_smithi028_177697_object109588:head v 475'14783, mlcod=475'14780)", 2024-02-05T06:54:34.942 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:54:34.930831+0000", 2024-02-05T06:54:34.942 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0042148250000000002, 2024-02-05T06:54:34.942 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0042460739999999999, 2024-02-05T06:54:34.942 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:54:34.942 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-02-05T06:54:34.942 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:54:34.942 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:34.942 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:54:34.943 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:34.930831+0000", 2024-02-05T06:54:34.943 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:34.943 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:34.943 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:34.943 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:54:34.943 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:34.930831+0000", 2024-02-05T06:54:34.943 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:34.943 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:34.943 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:34.943 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:54:34.943 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:34.930833+0000", 2024-02-05T06:54:34.943 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2440000000000001e-06 2024-02-05T06:54:34.943 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:34.943 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:34.943 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:54:34.943 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:34.930852+0000", 2024-02-05T06:54:34.944 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9449e-05 2024-02-05T06:54:34.944 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:34.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:34.944 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:54:34.944 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:34.930866+0000", 2024-02-05T06:54:34.944 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3614e-05 2024-02-05T06:54:34.944 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:34.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:34.944 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:54:34.944 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:34.930873+0000", 2024-02-05T06:54:34.944 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.1239999999999999e-06 2024-02-05T06:54:34.944 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:34.944 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:34.944 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:54:34.944 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:34.930935+0000", 2024-02-05T06:54:34.945 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1728000000000003e-05 2024-02-05T06:54:34.945 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:34.945 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:34.945 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:54:34.945 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:34.930953+0000", 2024-02-05T06:54:34.945 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8142999999999999e-05 2024-02-05T06:54:34.945 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:34.945 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:54:34.945 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:34.945 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:34.945 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:34.945 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_repop(client.45878.0:236555 11.8 e475/418 11:18564cc3:::benchmark_data_smithi028_177697_object109589:head v 475'14784, mlcod=475'14781)", 2024-02-05T06:54:34.945 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:54:34.933246+0000", 2024-02-05T06:54:34.945 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.001799733, 2024-02-05T06:54:34.946 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019130600000000001, 2024-02-05T06:54:34.946 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:54:34.946 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-02-05T06:54:34.946 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:54:34.946 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:34.946 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:54:34.946 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:34.933246+0000", 2024-02-05T06:54:34.946 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:34.946 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:34.946 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:34.946 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:54:34.946 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:34.933246+0000", 2024-02-05T06:54:34.946 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:34.946 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:34.946 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:34.947 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:54:34.947 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:34.933248+0000", 2024-02-05T06:54:34.947 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2419999999999999e-06 2024-02-05T06:54:34.947 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:34.947 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:34.947 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:54:34.947 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:34.933255+0000", 2024-02-05T06:54:34.947 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.9789999999999996e-06 2024-02-05T06:54:34.947 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:34.947 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:34.947 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:54:34.947 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:34.933267+0000", 2024-02-05T06:54:34.947 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2648e-05 2024-02-05T06:54:34.948 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:34.948 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:34.948 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:54:34.948 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:34.933285+0000", 2024-02-05T06:54:34.948 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.7152000000000001e-05 2024-02-05T06:54:34.948 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:34.948 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:34.948 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:54:34.948 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:34.933343+0000", 2024-02-05T06:54:34.948 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8199999999999998e-05 2024-02-05T06:54:34.948 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:34.948 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:34.948 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:54:34.948 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:34.933364+0000", 2024-02-05T06:54:34.948 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1637999999999998e-05 2024-02-05T06:54:34.949 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:34.949 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:54:34.949 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:34.949 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:34.949 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:34.949 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.45878.0:236558 11.7 11:ee2e8cb2:::benchmark_data_smithi028_177697_object109592:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e475)", 2024-02-05T06:54:34.949 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:54:34.933286+0000", 2024-02-05T06:54:34.949 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.0017604610000000001, 2024-02-05T06:54:34.949 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019376910000000001, 2024-02-05T06:54:34.949 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:54:34.949 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "waiting for sub ops", 2024-02-05T06:54:34.949 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:54:34.949 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.45878", 2024-02-05T06:54:34.949 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/3059201968", 2024-02-05T06:54:34.949 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 236558 2024-02-05T06:54:34.950 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:34.950 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:54:34.950 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:34.950 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:54:34.950 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:34.933286+0000", 2024-02-05T06:54:34.950 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:34.950 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:34.950 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:34.950 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:54:34.950 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:34.933286+0000", 2024-02-05T06:54:34.950 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:54:34.950 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:34.950 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:34.950 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:54:34.950 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:34.933288+0000", 2024-02-05T06:54:34.950 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.9970000000000001e-06 2024-02-05T06:54:34.951 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:34.951 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:34.951 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:54:34.951 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:34.933293+0000", 2024-02-05T06:54:34.951 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5300000000000004e-06 2024-02-05T06:54:34.951 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:34.951 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:34.951 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:54:34.951 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:34.933305+0000", 2024-02-05T06:54:34.951 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1745e-05 2024-02-05T06:54:34.951 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:34.951 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:34.951 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:54:34.951 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:34.933313+0000", 2024-02-05T06:54:34.952 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5449999999999998e-06 2024-02-05T06:54:34.952 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:34.952 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:34.952 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:54:34.952 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:34.933371+0000", 2024-02-05T06:54:34.952 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8303000000000001e-05 2024-02-05T06:54:34.952 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:34.952 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:34.952 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:54:34.952 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:34.933491+0000", 2024-02-05T06:54:34.952 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120525 2024-02-05T06:54:34.952 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:54:34.952 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:54:34.952 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,4", 2024-02-05T06:54:34.953 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:54:34.933601+0000", 2024-02-05T06:54:34.953 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010998 2024-02-05T06:54:34.953 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:34.953 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:54:34.953 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:34.953 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:54:34.953 INFO:teuthology.orchestra.run.smithi167.stdout: ], 2024-02-05T06:54:34.953 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 3 2024-02-05T06:54:34.953 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:54:34.961 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:34.993 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:34.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:34.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:35.094 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:35.124 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:35.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:35.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:35.226 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:35.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:35.267 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:35.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:35.368 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:35.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:35.410 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:35.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:35.511 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:35.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:35.549 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:35.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:35.634 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:35 smithi028 ceph-mon[29442]: pgmap v1008: 168 pgs: 168 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-02-05T06:54:35.635 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:35 smithi028 ceph-mon[37984]: pgmap v1008: 168 pgs: 168 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-02-05T06:54:35.650 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:35.687 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:35.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:35.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:35.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:35 smithi167 ceph-mon[32774]: pgmap v1008: 168 pgs: 168 active+clean; 1.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-02-05T06:54:35.789 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:35.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:35.819 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:35.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:35.920 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:35.960 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:35.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:35.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:36.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:36.085 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-02-05T06:54:36.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:36.101 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:36.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:36.202 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:36.231 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:36.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:36.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:36.332 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:36.371 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:36.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:36.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:36.408 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-02-05T06:54:36.435 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:54:36.473 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:36.518 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:36.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:36.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:36.620 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:36.658 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:36.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:36.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:36.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:36 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/469532624' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:54:36.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:36 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/469532624' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:54:36.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:36 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/469532624' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:54:36.759 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:36.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:36.799 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:36.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:36.900 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:36.990 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:36.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:36.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:37.092 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:37.127 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:37.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:37.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:37.229 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:37.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:37.269 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:37.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:37.370 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:37.411 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:37.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:37.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:37.486 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:37 smithi028 ceph-mon[29442]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:54:37.486 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:37 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/469532624' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:54:37.486 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:37 smithi028 ceph-mon[29442]: osdmap e476: 8 total, 8 up, 8 in 2024-02-05T06:54:37.486 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:37 smithi028 ceph-mon[29442]: pgmap v1010: 168 pgs: 168 active+clean; 1.2 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s 2024-02-05T06:54:37.486 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:37 smithi028 ceph-mon[29442]: 6.8 scrub starts 2024-02-05T06:54:37.486 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:37 smithi028 ceph-mon[29442]: 6.8 scrub ok 2024-02-05T06:54:37.486 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:37 smithi028 ceph-mon[37984]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:54:37.486 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:37 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/469532624' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:54:37.486 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:37 smithi028 ceph-mon[37984]: osdmap e476: 8 total, 8 up, 8 in 2024-02-05T06:54:37.487 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:37 smithi028 ceph-mon[37984]: pgmap v1010: 168 pgs: 168 active+clean; 1.2 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s 2024-02-05T06:54:37.487 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:37 smithi028 ceph-mon[37984]: 6.8 scrub starts 2024-02-05T06:54:37.487 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:37 smithi028 ceph-mon[37984]: 6.8 scrub ok 2024-02-05T06:54:37.513 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:37.550 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:37.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:37.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:37.652 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:37.653 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:37 smithi167 ceph-mon[32774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:54:37.653 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:37 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/469532624' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:54:37.653 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:37 smithi167 ceph-mon[32774]: osdmap e476: 8 total, 8 up, 8 in 2024-02-05T06:54:37.653 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:37 smithi167 ceph-mon[32774]: pgmap v1010: 168 pgs: 168 active+clean; 1.2 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s 2024-02-05T06:54:37.653 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:37 smithi167 ceph-mon[32774]: 6.8 scrub starts 2024-02-05T06:54:37.653 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:37 smithi167 ceph-mon[32774]: 6.8 scrub ok 2024-02-05T06:54:37.698 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:37.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:37.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:37.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:37.841 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:37.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:37.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:37.943 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:37.979 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:37.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:37.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:38.081 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:38.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:38.120 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:38.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:38.221 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:38.255 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:38.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:38.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:38.312 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:54:38.312 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:54:38.312 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:54:38.312 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:54:38.313 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:54:38.313 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:54:38.313 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:54:38.313 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:54:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:54:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:54:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:54:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:54:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:54:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:54:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:54:38.313 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:54:38.329 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:54:38.330 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:54:38.357 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:38.393 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:38.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:38.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:38.494 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:38.528 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:38.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:38.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:38.630 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:38.662 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:38 smithi167 ceph-mon[32774]: 3.1d scrub starts 2024-02-05T06:54:38.662 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:38 smithi167 ceph-mon[32774]: 3.1d scrub ok 2024-02-05T06:54:38.662 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:38 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3747872505' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:54:38.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:38.668 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:38.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:38.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:38 smithi028 ceph-mon[37984]: 3.1d scrub starts 2024-02-05T06:54:38.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:38 smithi028 ceph-mon[37984]: 3.1d scrub ok 2024-02-05T06:54:38.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:38 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3747872505' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:54:38.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:38 smithi028 ceph-mon[29442]: 3.1d scrub starts 2024-02-05T06:54:38.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:38 smithi028 ceph-mon[29442]: 3.1d scrub ok 2024-02-05T06:54:38.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:38 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3747872505' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:54:38.769 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:38.803 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:38.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:38.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:38.904 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:38.935 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 4, 7, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-02-05T06:54:38.935 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-02-05T06:54:38.935 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-02-05T06:54:38.935 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd dump -f json-pretty 2024-02-05T06:54:38.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:38.946 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:38.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:39.032 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-02-05T06:54:39.047 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:39.094 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:39.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:39.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:39.196 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:39.228 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:39.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:39.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:39.330 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:39.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:39.372 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:39.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:39.391 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:54:39.419 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:39 smithi167 ceph-mon[32774]: 5.8 scrub starts 2024-02-05T06:54:39.419 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:39 smithi167 ceph-mon[32774]: 5.8 scrub ok 2024-02-05T06:54:39.419 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:39 smithi167 ceph-mon[32774]: 11.b scrub starts 2024-02-05T06:54:39.419 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:39 smithi167 ceph-mon[32774]: pgmap v1011: 168 pgs: 168 active+clean; 902 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.62k op/s 2024-02-05T06:54:39.419 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:39 smithi167 ceph-mon[32774]: 11.b scrub ok 2024-02-05T06:54:39.472 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:39.511 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:39.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:39.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:39.612 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:39.646 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:39.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:39.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:39.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:39 smithi028 ceph-mon[37984]: 5.8 scrub starts 2024-02-05T06:54:39.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:39 smithi028 ceph-mon[37984]: 5.8 scrub ok 2024-02-05T06:54:39.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:39 smithi028 ceph-mon[37984]: 11.b scrub starts 2024-02-05T06:54:39.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:39 smithi028 ceph-mon[37984]: pgmap v1011: 168 pgs: 168 active+clean; 902 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.62k op/s 2024-02-05T06:54:39.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:39 smithi028 ceph-mon[37984]: 11.b scrub ok 2024-02-05T06:54:39.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:39 smithi028 ceph-mon[29442]: 5.8 scrub starts 2024-02-05T06:54:39.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:39 smithi028 ceph-mon[29442]: 5.8 scrub ok 2024-02-05T06:54:39.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:39 smithi028 ceph-mon[29442]: 11.b scrub starts 2024-02-05T06:54:39.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:39 smithi028 ceph-mon[29442]: pgmap v1011: 168 pgs: 168 active+clean; 902 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.62k op/s 2024-02-05T06:54:39.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:39 smithi028 ceph-mon[29442]: 11.b scrub ok 2024-02-05T06:54:39.748 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:39.782 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:39.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:39.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:39.884 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:39.919 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:39.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:39.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:40.021 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:40.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:40.076 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:40.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:40.136 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T06:54:40.177 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:54:40.178 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:40.245 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:40.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:40.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:40.347 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:40.389 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:40.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:40.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:40.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:40 smithi028 ceph-mon[29442]: 4.d scrub starts 2024-02-05T06:54:40.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:40 smithi028 ceph-mon[29442]: 4.d scrub ok 2024-02-05T06:54:40.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:40 smithi028 ceph-mon[29442]: 1.f scrub starts 2024-02-05T06:54:40.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:40 smithi028 ceph-mon[29442]: 1.f scrub ok 2024-02-05T06:54:40.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:40 smithi028 ceph-mon[29442]: 5.5 scrub starts 2024-02-05T06:54:40.451 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:40 smithi028 ceph-mon[29442]: 5.5 scrub ok 2024-02-05T06:54:40.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:40 smithi028 ceph-mon[37984]: 4.d scrub starts 2024-02-05T06:54:40.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:40 smithi028 ceph-mon[37984]: 4.d scrub ok 2024-02-05T06:54:40.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:40 smithi028 ceph-mon[37984]: 1.f scrub starts 2024-02-05T06:54:40.451 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:40 smithi028 ceph-mon[37984]: 1.f scrub ok 2024-02-05T06:54:40.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:40 smithi028 ceph-mon[37984]: 5.5 scrub starts 2024-02-05T06:54:40.452 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:40 smithi028 ceph-mon[37984]: 5.5 scrub ok 2024-02-05T06:54:40.491 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:40.521 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:40.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:40.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:40.623 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:40.625 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:40 smithi167 ceph-mon[32774]: 4.d scrub starts 2024-02-05T06:54:40.625 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:40 smithi167 ceph-mon[32774]: 4.d scrub ok 2024-02-05T06:54:40.625 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:40 smithi167 ceph-mon[32774]: 1.f scrub starts 2024-02-05T06:54:40.626 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:40 smithi167 ceph-mon[32774]: 1.f scrub ok 2024-02-05T06:54:40.626 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:40 smithi167 ceph-mon[32774]: 5.5 scrub starts 2024-02-05T06:54:40.626 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:40 smithi167 ceph-mon[32774]: 5.5 scrub ok 2024-02-05T06:54:40.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:40.665 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:40.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:40.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:40.811 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:40.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:40.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:40.913 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:40.952 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:40.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:40.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:41.053 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:41.085 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:41.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:41.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:41.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:41.232 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:41.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:41.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:41.334 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:41.369 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:41.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:41.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:41.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:41.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:41.510 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:41.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:41.543 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:54:41.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:41.657 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:54:41.657 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:54:41.661 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:41.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:41.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:41.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:41 smithi028 ceph-mon[29442]: 11.2 scrub starts 2024-02-05T06:54:41.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:41 smithi028 ceph-mon[29442]: 5.2 scrub starts 2024-02-05T06:54:41.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:41 smithi028 ceph-mon[29442]: 5.2 scrub ok 2024-02-05T06:54:41.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:41 smithi028 ceph-mon[29442]: 11.a scrub starts 2024-02-05T06:54:41.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:41 smithi028 ceph-mon[29442]: pgmap v1012: 168 pgs: 168 active+clean; 902 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-05T06:54:41.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:41 smithi028 ceph-mon[29442]: 6.1d scrub starts 2024-02-05T06:54:41.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:41 smithi028 ceph-mon[29442]: 6.1d scrub ok 2024-02-05T06:54:41.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:41 smithi028 ceph-mon[29442]: 11.a scrub ok 2024-02-05T06:54:41.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:41 smithi028 ceph-mon[37984]: 11.2 scrub starts 2024-02-05T06:54:41.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:41 smithi028 ceph-mon[37984]: 5.2 scrub starts 2024-02-05T06:54:41.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:41 smithi028 ceph-mon[37984]: 5.2 scrub ok 2024-02-05T06:54:41.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:41 smithi028 ceph-mon[37984]: 11.a scrub starts 2024-02-05T06:54:41.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:41 smithi028 ceph-mon[37984]: pgmap v1012: 168 pgs: 168 active+clean; 902 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-05T06:54:41.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:41 smithi028 ceph-mon[37984]: 6.1d scrub starts 2024-02-05T06:54:41.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:41 smithi028 ceph-mon[37984]: 6.1d scrub ok 2024-02-05T06:54:41.692 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:41 smithi028 ceph-mon[37984]: 11.a scrub ok 2024-02-05T06:54:41.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:41 smithi167 ceph-mon[32774]: 11.2 scrub starts 2024-02-05T06:54:41.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:41 smithi167 ceph-mon[32774]: 5.2 scrub starts 2024-02-05T06:54:41.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:41 smithi167 ceph-mon[32774]: 5.2 scrub ok 2024-02-05T06:54:41.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:41 smithi167 ceph-mon[32774]: 11.a scrub starts 2024-02-05T06:54:41.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:41 smithi167 ceph-mon[32774]: pgmap v1012: 168 pgs: 168 active+clean; 902 MiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.06k op/s 2024-02-05T06:54:41.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:41 smithi167 ceph-mon[32774]: 6.1d scrub starts 2024-02-05T06:54:41.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:41 smithi167 ceph-mon[32774]: 6.1d scrub ok 2024-02-05T06:54:41.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:41 smithi167 ceph-mon[32774]: 11.a scrub ok 2024-02-05T06:54:41.764 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:41.795 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:41.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:41.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:41.896 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:41.936 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:41.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:41.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:41.976 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-02-05T06:54:42.038 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:42.089 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:42.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:42.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:42.191 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:42.233 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:42.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:42.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:42.335 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:42.380 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:42.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:42.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:42.480 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-02-05T06:54:42.482 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:42.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:42.524 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:42.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:42.625 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:42.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:42.661 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:42.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:42.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:42 smithi028 ceph-mon[37984]: 11.2 scrub ok 2024-02-05T06:54:42.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:42 smithi028 ceph-mon[37984]: 6.7 scrub starts 2024-02-05T06:54:42.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:42 smithi028 ceph-mon[37984]: 6.7 scrub ok 2024-02-05T06:54:42.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:42 smithi028 ceph-mon[37984]: 4.b scrub starts 2024-02-05T06:54:42.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:42 smithi028 ceph-mon[37984]: 4.b scrub ok 2024-02-05T06:54:42.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:42 smithi028 ceph-mon[29442]: 11.2 scrub ok 2024-02-05T06:54:42.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:42 smithi028 ceph-mon[29442]: 6.7 scrub starts 2024-02-05T06:54:42.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:42 smithi028 ceph-mon[29442]: 6.7 scrub ok 2024-02-05T06:54:42.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:42 smithi028 ceph-mon[29442]: 4.b scrub starts 2024-02-05T06:54:42.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:42 smithi028 ceph-mon[29442]: 4.b scrub ok 2024-02-05T06:54:42.699 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:54:42 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:54:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:54:42.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:42 smithi167 ceph-mon[32774]: 11.2 scrub ok 2024-02-05T06:54:42.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:42 smithi167 ceph-mon[32774]: 6.7 scrub starts 2024-02-05T06:54:42.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:42 smithi167 ceph-mon[32774]: 6.7 scrub ok 2024-02-05T06:54:42.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:42 smithi167 ceph-mon[32774]: 4.b scrub starts 2024-02-05T06:54:42.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:42 smithi167 ceph-mon[32774]: 4.b scrub ok 2024-02-05T06:54:42.763 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:42.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:42.796 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:42.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:42.873 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:54:42.897 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:42.941 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:42.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:42.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:43.043 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:54:43.068 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-02-05T06:54:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 476, 2024-02-05T06:54:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: "fsid": "9801e956-c3ef-11ee-95b6-87774f69a715", 2024-02-05T06:54:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: "created": "2024-02-05T06:27:59.919226+0000", 2024-02-05T06:54:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: "modified": "2024-02-05T06:54:36.380707+0000", 2024-02-05T06:54:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: "last_up_change": "2024-02-05T06:33:15.833573+0000", 2024-02-05T06:54:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: "last_in_change": "2024-02-05T06:41:42.383087+0000", 2024-02-05T06:54:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-02-05T06:54:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_num": 5804032, 2024-02-05T06:54:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_set": [ 2024-02-05T06:54:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: "nodeep-scrub", 2024-02-05T06:54:43.068 INFO:teuthology.orchestra.run.smithi028.stdout: "pglog_hardlimit", 2024-02-05T06:54:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: "purged_snapdirs", 2024-02-05T06:54:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_deletes", 2024-02-05T06:54:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: "sortbitwise" 2024-02-05T06:54:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:54:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_version": 18, 2024-02-05T06:54:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: "full_ratio": 0.94999998807907104, 2024-02-05T06:54:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-02-05T06:54:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: "nearfull_ratio": 0.85000002384185791, 2024-02-05T06:54:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_snapshot": "", 2024-02-05T06:54:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_max": 11, 2024-02-05T06:54:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd": 8, 2024-02-05T06:54:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: "require_min_compat_client": "luminous", 2024-02-05T06:54:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: "min_compat_client": "luminous", 2024-02-05T06:54:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: "require_osd_release": "squid", 2024-02-05T06:54:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_crimson": false, 2024-02-05T06:54:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: "pools": [ 2024-02-05T06:54:43.069 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 1, 2024-02-05T06:54:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".mgr", 2024-02-05T06:54:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:31:10.441424+0000", 2024-02-05T06:54:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-02-05T06:54:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-02-05T06:54:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:54:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:54:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:54:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:54:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:54:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:54:43.070 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:54:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:54:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:54:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:54:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 21, 2024-02-05T06:54:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 21, 2024-02-05T06:54:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 21, 2024-02-05T06:54:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 21, 2024-02-05T06:54:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 21, 2024-02-05T06:54:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:54:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:54:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:54:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:54:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:54:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:54:43.071 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:54:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "431", 2024-02-05T06:54:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:54:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:54:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "429", 2024-02-05T06:54:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:54:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:54:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-02-05T06:54:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-02-05T06:54:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:54:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:54:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:54:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:54:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:54:43.072 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:54:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:54:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:54:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:54:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:54:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:54:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:54:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:54:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:54:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:54:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:54:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:54:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:54:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:54:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:54:43.073 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:54:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:54:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:54:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:54:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:54:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:54:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:54:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:54:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:54:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:54:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-02-05T06:54:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_max": 32, 2024-02-05T06:54:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_min": 1 2024-02-05T06:54:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.074 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:54:43.075 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": {} 2024-02-05T06:54:43.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.075 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:54:43.075 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:54:43.075 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.8999999761581421, 2024-02-05T06:54:43.075 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.8999999761581421, 2024-02-05T06:54:43.075 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-02-05T06:54:43.075 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.8999999761581421, 2024-02-05T06:54:43.075 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.8999999761581421, 2024-02-05T06:54:43.075 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.88084030151367188, 2024-02-05T06:54:43.075 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.87999999523162842, 2024-02-05T06:54:43.075 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-02-05T06:54:43.075 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.076 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.076 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 2, 2024-02-05T06:54:43.076 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "datapool", 2024-02-05T06:54:43.076 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:33:27.397416+0000", 2024-02-05T06:54:43.076 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 8193, 2024-02-05T06:54:43.076 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-02-05T06:54:43.076 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:54:43.076 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:54:43.076 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:54:43.076 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:54:43.076 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:54:43.076 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:54:43.077 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:54:43.077 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:54:43.077 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:54:43.077 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:54:43.077 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 3, 2024-02-05T06:54:43.077 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 3, 2024-02-05T06:54:43.077 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 3, 2024-02-05T06:54:43.077 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 3, 2024-02-05T06:54:43.077 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 3, 2024-02-05T06:54:43.077 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:54:43.077 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:54:43.077 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:54:43.077 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:54:43.077 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:54:43.077 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:54:43.077 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:54:43.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.078 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "69", 2024-02-05T06:54:43.078 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:54:43.078 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:54:43.078 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T06:54:43.078 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:54:43.078 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:54:43.078 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 2, 2024-02-05T06:54:43.078 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 69, 2024-02-05T06:54:43.078 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:54:43.078 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:54:43.078 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:54:43.078 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:54:43.078 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:54:43.079 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:54:43.079 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:54:43.079 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:54:43.079 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:54:43.079 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:54:43.079 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:54:43.079 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:54:43.079 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:54:43.079 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:54:43.079 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:54:43.079 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:54:43.079 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:54:43.079 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:54:43.079 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:54:43.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.080 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:54:43.080 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:54:43.080 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:54:43.080 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:54:43.080 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:54:43.080 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:54:43.080 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:54:43.080 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:54:43.080 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:54:43.080 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:54:43.080 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:54:43.080 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-02-05T06:54:43.080 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:54:43.080 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd": {} 2024-02-05T06:54:43.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.081 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:54:43.081 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:54:43.081 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.6500000953674316, 2024-02-05T06:54:43.081 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.6500000953674316, 2024-02-05T06:54:43.081 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 0.87999999523162842, 2024-02-05T06:54:43.081 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2.3299999237060547, 2024-02-05T06:54:43.081 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2.3299999237060547, 2024-02-05T06:54:43.081 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.88084030151367188, 2024-02-05T06:54:43.081 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.87999999523162842, 2024-02-05T06:54:43.081 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-02-05T06:54:43.081 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.081 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.082 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 3, 2024-02-05T06:54:43.082 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": ".rgw.root", 2024-02-05T06:54:43.082 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:33:28.090506+0000", 2024-02-05T06:54:43.082 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-02-05T06:54:43.082 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-02-05T06:54:43.082 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:54:43.082 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:54:43.082 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:54:43.082 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:54:43.082 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:54:43.082 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:54:43.082 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:54:43.082 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:54:43.082 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:54:43.082 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:54:43.082 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-02-05T06:54:43.082 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-02-05T06:54:43.083 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-02-05T06:54:43.083 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-02-05T06:54:43.083 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-02-05T06:54:43.083 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:54:43.083 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:54:43.083 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:54:43.083 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:54:43.083 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:54:43.083 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:54:43.083 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:54:43.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.083 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "67", 2024-02-05T06:54:43.083 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:54:43.083 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:54:43.084 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T06:54:43.084 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:54:43.084 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:54:43.084 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-02-05T06:54:43.084 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-02-05T06:54:43.084 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:54:43.084 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:54:43.084 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:54:43.084 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:54:43.084 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:54:43.084 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:54:43.084 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:54:43.084 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:54:43.084 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:54:43.085 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:54:43.085 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:54:43.085 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:54:43.085 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:54:43.085 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:54:43.085 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:54:43.085 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:54:43.085 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:54:43.085 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:54:43.085 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:54:43.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.085 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:54:43.085 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:54:43.085 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:54:43.085 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:54:43.086 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:54:43.086 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:54:43.086 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:54:43.086 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:54:43.086 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:54:43.086 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:54:43.086 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:54:43.086 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-02-05T06:54:43.086 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:54:43.086 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-02-05T06:54:43.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.086 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:54:43.086 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:54:43.086 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.75, 2024-02-05T06:54:43.086 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.75, 2024-02-05T06:54:43.087 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-02-05T06:54:43.087 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.75, 2024-02-05T06:54:43.087 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.75, 2024-02-05T06:54:43.087 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.88084030151367188, 2024-02-05T06:54:43.087 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.87999999523162842, 2024-02-05T06:54:43.087 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-02-05T06:54:43.087 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.087 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 4, 2024-02-05T06:54:43.087 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.log", 2024-02-05T06:54:43.087 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:33:29.866249+0000", 2024-02-05T06:54:43.087 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-02-05T06:54:43.087 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-02-05T06:54:43.087 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:54:43.088 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:54:43.088 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:54:43.088 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:54:43.088 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:54:43.088 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:54:43.088 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:54:43.088 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:54:43.088 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:54:43.088 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:54:43.088 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-02-05T06:54:43.088 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-02-05T06:54:43.088 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-02-05T06:54:43.088 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-02-05T06:54:43.088 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-02-05T06:54:43.089 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:54:43.089 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:54:43.089 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:54:43.089 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:54:43.089 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:54:43.089 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:54:43.089 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:54:43.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.089 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "69", 2024-02-05T06:54:43.089 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:54:43.089 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:54:43.089 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T06:54:43.089 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:54:43.089 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:54:43.089 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-02-05T06:54:43.090 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-02-05T06:54:43.090 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:54:43.090 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:54:43.090 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:54:43.090 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:54:43.090 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:54:43.090 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:54:43.090 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:54:43.090 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:54:43.090 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:54:43.090 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:54:43.090 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:54:43.090 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:54:43.090 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:54:43.090 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:54:43.091 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:54:43.091 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:54:43.091 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:54:43.091 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:54:43.091 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:54:43.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.091 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:54:43.091 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:54:43.091 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:54:43.091 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:54:43.091 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:54:43.091 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:54:43.091 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:54:43.091 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:54:43.091 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:54:43.092 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:54:43.092 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:54:43.092 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-02-05T06:54:43.092 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:54:43.092 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-02-05T06:54:43.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.092 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:54:43.092 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:54:43.092 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2.5, 2024-02-05T06:54:43.092 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2.5, 2024-02-05T06:54:43.092 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-02-05T06:54:43.092 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2.5, 2024-02-05T06:54:43.092 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2.5, 2024-02-05T06:54:43.092 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.88084030151367188, 2024-02-05T06:54:43.093 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.87999999523162842, 2024-02-05T06:54:43.093 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-02-05T06:54:43.093 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.093 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 5, 2024-02-05T06:54:43.093 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.control", 2024-02-05T06:54:43.093 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:33:31.826006+0000", 2024-02-05T06:54:43.093 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-02-05T06:54:43.093 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-02-05T06:54:43.093 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:54:43.093 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:54:43.093 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:54:43.093 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:54:43.093 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:54:43.093 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:54:43.094 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:54:43.094 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:54:43.094 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:54:43.094 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:54:43.094 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-02-05T06:54:43.094 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-02-05T06:54:43.094 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-02-05T06:54:43.094 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-02-05T06:54:43.094 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-02-05T06:54:43.094 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:54:43.094 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:54:43.094 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:54:43.094 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:54:43.094 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:54:43.094 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:54:43.094 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:54:43.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.095 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "71", 2024-02-05T06:54:43.095 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:54:43.095 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:54:43.095 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T06:54:43.095 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:54:43.095 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:54:43.095 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-02-05T06:54:43.095 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-02-05T06:54:43.095 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:54:43.095 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:54:43.095 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:54:43.095 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:54:43.095 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:54:43.095 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:54:43.096 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:54:43.096 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:54:43.096 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:54:43.096 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:54:43.096 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:54:43.096 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:54:43.096 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:54:43.096 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:54:43.096 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:54:43.096 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:54:43.096 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:54:43.096 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:54:43.096 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:54:43.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.097 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:54:43.097 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:54:43.097 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:54:43.097 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:54:43.097 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:54:43.097 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:54:43.097 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:54:43.097 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:54:43.097 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:54:43.097 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:54:43.097 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:54:43.097 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-02-05T06:54:43.097 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:54:43.097 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-02-05T06:54:43.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.098 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:54:43.098 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:54:43.098 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.75, 2024-02-05T06:54:43.098 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.75, 2024-02-05T06:54:43.098 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-02-05T06:54:43.098 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.75, 2024-02-05T06:54:43.098 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.75, 2024-02-05T06:54:43.098 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.88084030151367188, 2024-02-05T06:54:43.098 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.87999999523162842, 2024-02-05T06:54:43.099 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-02-05T06:54:43.099 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.099 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 6, 2024-02-05T06:54:43.099 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "default.rgw.meta", 2024-02-05T06:54:43.099 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:33:33.890835+0000", 2024-02-05T06:54:43.099 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-02-05T06:54:43.099 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-02-05T06:54:43.099 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:54:43.099 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:54:43.099 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:54:43.099 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:54:43.100 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:54:43.100 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:54:43.100 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:54:43.100 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:54:43.100 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:54:43.100 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:54:43.100 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 32, 2024-02-05T06:54:43.100 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 32, 2024-02-05T06:54:43.100 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 32, 2024-02-05T06:54:43.100 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 32, 2024-02-05T06:54:43.100 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 32, 2024-02-05T06:54:43.100 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:54:43.100 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:54:43.101 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:54:43.101 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:54:43.101 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:54:43.101 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:54:43.101 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:54:43.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.101 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "73", 2024-02-05T06:54:43.101 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:54:43.101 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:54:43.101 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T06:54:43.101 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:54:43.101 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:54:43.101 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-02-05T06:54:43.102 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-02-05T06:54:43.102 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:54:43.102 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:54:43.102 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:54:43.102 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:54:43.102 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:54:43.102 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:54:43.102 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:54:43.102 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:54:43.102 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:54:43.102 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:54:43.102 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:54:43.102 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:54:43.102 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:54:43.102 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:54:43.102 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:54:43.103 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:54:43.103 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:54:43.103 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:54:43.103 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:54:43.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.103 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:54:43.103 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:54:43.103 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:54:43.103 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:54:43.103 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:54:43.103 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:54:43.103 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:54:43.103 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:54:43.103 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:54:43.103 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:54:43.103 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:54:43.104 INFO:teuthology.orchestra.run.smithi028.stdout: "options": { 2024-02-05T06:54:43.104 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_bias": 4 2024-02-05T06:54:43.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.104 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:54:43.104 INFO:teuthology.orchestra.run.smithi028.stdout: "rgw": {} 2024-02-05T06:54:43.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.104 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:54:43.104 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:54:43.104 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 1.75, 2024-02-05T06:54:43.104 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 1.75, 2024-02-05T06:54:43.104 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-02-05T06:54:43.104 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 1.75, 2024-02-05T06:54:43.104 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 1.75, 2024-02-05T06:54:43.104 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.88084030151367188, 2024-02-05T06:54:43.104 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.87999999523162842, 2024-02-05T06:54:43.105 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-02-05T06:54:43.105 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.105 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 11, 2024-02-05T06:54:43.105 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": "unique_pool_4", 2024-02-05T06:54:43.105 INFO:teuthology.orchestra.run.smithi028.stdout: "create_time": "2024-02-05T06:51:11.916055+0000", 2024-02-05T06:54:43.105 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-02-05T06:54:43.105 INFO:teuthology.orchestra.run.smithi028.stdout: "flags_names": "hashpspool", 2024-02-05T06:54:43.105 INFO:teuthology.orchestra.run.smithi028.stdout: "type": 1, 2024-02-05T06:54:43.105 INFO:teuthology.orchestra.run.smithi028.stdout: "size": 3, 2024-02-05T06:54:43.105 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": 2, 2024-02-05T06:54:43.105 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_rule": 0, 2024-02-05T06:54:43.105 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_count": 0, 2024-02-05T06:54:43.105 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_target": 0, 2024-02-05T06:54:43.105 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_barrier": 0, 2024-02-05T06:54:43.106 INFO:teuthology.orchestra.run.smithi028.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-02-05T06:54:43.106 INFO:teuthology.orchestra.run.smithi028.stdout: "object_hash": 2, 2024-02-05T06:54:43.106 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscale_mode": "off", 2024-02-05T06:54:43.106 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": 16, 2024-02-05T06:54:43.106 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num": 16, 2024-02-05T06:54:43.106 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_placement_num_target": 16, 2024-02-05T06:54:43.106 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_target": 16, 2024-02-05T06:54:43.106 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num_pending": 16, 2024-02-05T06:54:43.106 INFO:teuthology.orchestra.run.smithi028.stdout: "last_pg_merge_meta": { 2024-02-05T06:54:43.106 INFO:teuthology.orchestra.run.smithi028.stdout: "source_pgid": "0.0", 2024-02-05T06:54:43.106 INFO:teuthology.orchestra.run.smithi028.stdout: "ready_epoch": 0, 2024-02-05T06:54:43.106 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_started": 0, 2024-02-05T06:54:43.106 INFO:teuthology.orchestra.run.smithi028.stdout: "last_epoch_clean": 0, 2024-02-05T06:54:43.106 INFO:teuthology.orchestra.run.smithi028.stdout: "source_version": "0'0", 2024-02-05T06:54:43.106 INFO:teuthology.orchestra.run.smithi028.stdout: "target_version": "0'0" 2024-02-05T06:54:43.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.107 INFO:teuthology.orchestra.run.smithi028.stdout: "last_change": "422", 2024-02-05T06:54:43.107 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend": "0", 2024-02-05T06:54:43.107 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_prenautilus": "0", 2024-02-05T06:54:43.107 INFO:teuthology.orchestra.run.smithi028.stdout: "last_force_op_resend_preluminous": "0", 2024-02-05T06:54:43.107 INFO:teuthology.orchestra.run.smithi028.stdout: "auid": 0, 2024-02-05T06:54:43.107 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_mode": "selfmanaged", 2024-02-05T06:54:43.107 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_seq": 0, 2024-02-05T06:54:43.107 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_epoch": 0, 2024-02-05T06:54:43.107 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_snaps": [], 2024-02-05T06:54:43.107 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps": "[]", 2024-02-05T06:54:43.107 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_bytes": 0, 2024-02-05T06:54:43.107 INFO:teuthology.orchestra.run.smithi028.stdout: "quota_max_objects": 0, 2024-02-05T06:54:43.107 INFO:teuthology.orchestra.run.smithi028.stdout: "tiers": [], 2024-02-05T06:54:43.107 INFO:teuthology.orchestra.run.smithi028.stdout: "tier_of": -1, 2024-02-05T06:54:43.107 INFO:teuthology.orchestra.run.smithi028.stdout: "read_tier": -1, 2024-02-05T06:54:43.108 INFO:teuthology.orchestra.run.smithi028.stdout: "write_tier": -1, 2024-02-05T06:54:43.108 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_mode": "none", 2024-02-05T06:54:43.108 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_bytes": 0, 2024-02-05T06:54:43.108 INFO:teuthology.orchestra.run.smithi028.stdout: "target_max_objects": 0, 2024-02-05T06:54:43.108 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-02-05T06:54:43.108 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-02-05T06:54:43.108 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_target_full_ratio_micro": 800000, 2024-02-05T06:54:43.108 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_flush_age": 0, 2024-02-05T06:54:43.108 INFO:teuthology.orchestra.run.smithi028.stdout: "cache_min_evict_age": 0, 2024-02-05T06:54:43.108 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profile": "", 2024-02-05T06:54:43.108 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_params": { 2024-02-05T06:54:43.108 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "none" 2024-02-05T06:54:43.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.108 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_period": 0, 2024-02-05T06:54:43.108 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_count": 0, 2024-02-05T06:54:43.109 INFO:teuthology.orchestra.run.smithi028.stdout: "use_gmt_hitset": true, 2024-02-05T06:54:43.109 INFO:teuthology.orchestra.run.smithi028.stdout: "min_read_recency_for_promote": 0, 2024-02-05T06:54:43.109 INFO:teuthology.orchestra.run.smithi028.stdout: "min_write_recency_for_promote": 0, 2024-02-05T06:54:43.109 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_grade_decay_rate": 0, 2024-02-05T06:54:43.109 INFO:teuthology.orchestra.run.smithi028.stdout: "hit_set_search_last_n": 0, 2024-02-05T06:54:43.109 INFO:teuthology.orchestra.run.smithi028.stdout: "grade_table": [], 2024-02-05T06:54:43.109 INFO:teuthology.orchestra.run.smithi028.stdout: "stripe_width": 0, 2024-02-05T06:54:43.109 INFO:teuthology.orchestra.run.smithi028.stdout: "expected_num_objects": 0, 2024-02-05T06:54:43.109 INFO:teuthology.orchestra.run.smithi028.stdout: "fast_read": false, 2024-02-05T06:54:43.109 INFO:teuthology.orchestra.run.smithi028.stdout: "options": {}, 2024-02-05T06:54:43.109 INFO:teuthology.orchestra.run.smithi028.stdout: "application_metadata": { 2024-02-05T06:54:43.109 INFO:teuthology.orchestra.run.smithi028.stdout: "rados": {} 2024-02-05T06:54:43.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.109 INFO:teuthology.orchestra.run.smithi028.stdout: "read_balance": { 2024-02-05T06:54:43.109 INFO:teuthology.orchestra.run.smithi028.stdout: "score_type": "Fair distribution", 2024-02-05T06:54:43.109 INFO:teuthology.orchestra.run.smithi028.stdout: "score_acting": 2, 2024-02-05T06:54:43.110 INFO:teuthology.orchestra.run.smithi028.stdout: "score_stable": 2, 2024-02-05T06:54:43.110 INFO:teuthology.orchestra.run.smithi028.stdout: "optimal_score": 1, 2024-02-05T06:54:43.110 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_acting": 2, 2024-02-05T06:54:43.110 INFO:teuthology.orchestra.run.smithi028.stdout: "raw_score_stable": 2, 2024-02-05T06:54:43.110 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity_weighted": 0.88084030151367188, 2024-02-05T06:54:43.110 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity": 0.87999999523162842, 2024-02-05T06:54:43.110 INFO:teuthology.orchestra.run.smithi028.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-02-05T06:54:43.110 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.110 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.110 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:54:43.110 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-02-05T06:54:43.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.110 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-02-05T06:54:43.110 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "c19fdd5b-09f6-45d4-a519-0a938cf1f2a6", 2024-02-05T06:54:43.110 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:54:43.110 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:54:43.110 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:54:43.111 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:54:43.111 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:54:43.111 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:54:43.111 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 8, 2024-02-05T06:54:43.111 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 434, 2024-02-05T06:54:43.111 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:54:43.111 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:54:43.111 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:54:43.111 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:54:43.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.111 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:54:43.111 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-02-05T06:54:43.111 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 301279106 2024-02-05T06:54:43.111 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.111 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.112 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:54:43.112 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:54:43.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.112 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:54:43.112 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6802", 2024-02-05T06:54:43.112 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 301279106 2024-02-05T06:54:43.112 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.112 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.112 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:54:43.112 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:54:43.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.112 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:54:43.112 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6804", 2024-02-05T06:54:43.113 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 301279106 2024-02-05T06:54:43.113 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.113 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.113 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:54:43.113 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:54:43.113 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.113 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:54:43.113 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6803", 2024-02-05T06:54:43.113 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 301279106 2024-02-05T06:54:43.113 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.113 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.113 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6801/301279106", 2024-02-05T06:54:43.113 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6802/301279106", 2024-02-05T06:54:43.113 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6804/301279106", 2024-02-05T06:54:43.114 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6803/301279106", 2024-02-05T06:54:43.114 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:54:43.114 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:54:43.114 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:54:43.114 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.114 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-02-05T06:54:43.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:43.115 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:43.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:43.116 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "52245f6e-eb96-4d0d-985b-dbb6fd3ce8de", 2024-02-05T06:54:43.116 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:54:43.116 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:54:43.116 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:54:43.116 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:54:43.116 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:54:43.116 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:54:43.116 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 13, 2024-02-05T06:54:43.117 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 434, 2024-02-05T06:54:43.117 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:54:43.117 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:54:43.117 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:54:43.117 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:54:43.117 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.117 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:54:43.117 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6805", 2024-02-05T06:54:43.117 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1795224649 2024-02-05T06:54:43.117 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.117 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.118 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:54:43.118 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:54:43.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.118 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:54:43.118 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6806", 2024-02-05T06:54:43.118 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1795224649 2024-02-05T06:54:43.118 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.118 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.118 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:54:43.118 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:54:43.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.119 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:54:43.119 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6808", 2024-02-05T06:54:43.119 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1795224649 2024-02-05T06:54:43.119 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.119 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.119 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:54:43.119 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:54:43.119 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.119 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:54:43.119 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6807", 2024-02-05T06:54:43.119 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1795224649 2024-02-05T06:54:43.120 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.120 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.120 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6805/1795224649", 2024-02-05T06:54:43.120 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6806/1795224649", 2024-02-05T06:54:43.120 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6808/1795224649", 2024-02-05T06:54:43.120 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6807/1795224649", 2024-02-05T06:54:43.120 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:54:43.120 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:54:43.120 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:54:43.120 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.120 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-02-05T06:54:43.121 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "1e18b630-afdf-4de9-a1a2-37bdfdd01c67", 2024-02-05T06:54:43.121 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:54:43.121 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:54:43.121 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:54:43.121 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:54:43.121 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:54:43.121 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:54:43.121 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 18, 2024-02-05T06:54:43.121 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 434, 2024-02-05T06:54:43.121 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:54:43.121 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:54:43.121 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:54:43.122 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:54:43.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.122 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:54:43.122 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6809", 2024-02-05T06:54:43.122 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1993280018 2024-02-05T06:54:43.122 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.122 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.122 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:54:43.122 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:54:43.122 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.122 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:54:43.122 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6810", 2024-02-05T06:54:43.122 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1993280018 2024-02-05T06:54:43.123 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.123 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.123 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:54:43.123 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:54:43.123 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.123 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:54:43.123 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6812", 2024-02-05T06:54:43.123 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1993280018 2024-02-05T06:54:43.123 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.123 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.123 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:54:43.124 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:54:43.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.124 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:54:43.124 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6811", 2024-02-05T06:54:43.124 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1993280018 2024-02-05T06:54:43.124 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.124 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.124 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6809/1993280018", 2024-02-05T06:54:43.124 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6810/1993280018", 2024-02-05T06:54:43.124 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6812/1993280018", 2024-02-05T06:54:43.124 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6811/1993280018", 2024-02-05T06:54:43.124 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:54:43.125 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:54:43.125 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:54:43.125 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.125 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.125 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-02-05T06:54:43.125 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "3ae795af-3d59-41a6-95eb-b8447ff61282", 2024-02-05T06:54:43.125 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:54:43.125 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:54:43.125 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:54:43.125 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:54:43.125 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:54:43.125 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:54:43.126 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 26, 2024-02-05T06:54:43.126 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 434, 2024-02-05T06:54:43.126 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:54:43.126 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:54:43.126 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:54:43.126 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:54:43.126 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.126 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:54:43.126 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6813", 2024-02-05T06:54:43.126 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1294621398 2024-02-05T06:54:43.127 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.127 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.127 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:54:43.127 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:54:43.127 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.127 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:54:43.127 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6814", 2024-02-05T06:54:43.127 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1294621398 2024-02-05T06:54:43.127 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.127 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.127 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:54:43.128 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:54:43.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.128 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:54:43.128 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6816", 2024-02-05T06:54:43.128 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1294621398 2024-02-05T06:54:43.128 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.128 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.128 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:54:43.128 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:54:43.128 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.128 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:54:43.128 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6815", 2024-02-05T06:54:43.129 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1294621398 2024-02-05T06:54:43.129 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.129 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.129 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.28:6813/1294621398", 2024-02-05T06:54:43.129 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.28:6814/1294621398", 2024-02-05T06:54:43.129 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.28:6816/1294621398", 2024-02-05T06:54:43.129 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.28:6815/1294621398", 2024-02-05T06:54:43.129 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:54:43.129 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:54:43.129 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:54:43.129 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.129 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.130 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-02-05T06:54:43.130 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "fc6bd83d-570c-48c0-964d-e09b4608bc32", 2024-02-05T06:54:43.130 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:54:43.130 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:54:43.130 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:54:43.130 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 0.046722412109375, 2024-02-05T06:54:43.130 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:54:43.130 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:54:43.130 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 31, 2024-02-05T06:54:43.130 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 267, 2024-02-05T06:54:43.130 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:54:43.130 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:54:43.130 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:54:43.131 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:54:43.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.131 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:54:43.131 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6800", 2024-02-05T06:54:43.131 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4052090595 2024-02-05T06:54:43.131 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.131 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.131 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:54:43.131 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:54:43.131 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.131 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:54:43.131 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6801", 2024-02-05T06:54:43.131 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4052090595 2024-02-05T06:54:43.132 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.132 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.132 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:54:43.132 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:54:43.132 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.132 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:54:43.132 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6803", 2024-02-05T06:54:43.132 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4052090595 2024-02-05T06:54:43.132 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.132 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.132 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:54:43.133 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:54:43.133 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.133 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:54:43.133 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6802", 2024-02-05T06:54:43.133 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4052090595 2024-02-05T06:54:43.133 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.133 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.133 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.167:6800/4052090595", 2024-02-05T06:54:43.133 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.167:6801/4052090595", 2024-02-05T06:54:43.133 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.167:6803/4052090595", 2024-02-05T06:54:43.133 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.167:6802/4052090595", 2024-02-05T06:54:43.133 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:54:43.133 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:54:43.133 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:54:43.134 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.134 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.134 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-02-05T06:54:43.134 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "895bf465-2531-4d9f-bc23-1e75fd969f07", 2024-02-05T06:54:43.134 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:54:43.134 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:54:43.134 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:54:43.134 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:54:43.134 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:54:43.134 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:54:43.134 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 40, 2024-02-05T06:54:43.134 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 434, 2024-02-05T06:54:43.135 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:54:43.135 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:54:43.135 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:54:43.135 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:54:43.135 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.135 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:54:43.135 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6804", 2024-02-05T06:54:43.135 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1631080500 2024-02-05T06:54:43.135 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.135 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.135 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.135 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:54:43.135 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:54:43.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.136 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:54:43.136 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6805", 2024-02-05T06:54:43.136 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1631080500 2024-02-05T06:54:43.136 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.136 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.136 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:54:43.136 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:54:43.136 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.136 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:54:43.136 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6807", 2024-02-05T06:54:43.137 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1631080500 2024-02-05T06:54:43.137 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.137 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.137 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:54:43.137 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:54:43.137 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.137 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:54:43.137 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6806", 2024-02-05T06:54:43.137 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1631080500 2024-02-05T06:54:43.137 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.137 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.137 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.167:6804/1631080500", 2024-02-05T06:54:43.137 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.167:6805/1631080500", 2024-02-05T06:54:43.138 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.167:6807/1631080500", 2024-02-05T06:54:43.138 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.167:6806/1631080500", 2024-02-05T06:54:43.138 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:54:43.138 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:54:43.138 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:54:43.138 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.138 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.138 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-02-05T06:54:43.138 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "ce0a9300-87a4-4e37-b702-d667f168c0e0", 2024-02-05T06:54:43.138 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:54:43.138 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:54:43.138 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:54:43.138 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:54:43.139 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:54:43.139 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:54:43.139 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 49, 2024-02-05T06:54:43.139 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 434, 2024-02-05T06:54:43.139 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:54:43.139 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:54:43.139 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:54:43.139 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:54:43.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.139 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:54:43.139 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6808", 2024-02-05T06:54:43.139 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1644050617 2024-02-05T06:54:43.139 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.140 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.140 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:54:43.140 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:54:43.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.140 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:54:43.140 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6809", 2024-02-05T06:54:43.140 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1644050617 2024-02-05T06:54:43.140 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.140 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.140 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:54:43.140 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:54:43.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.141 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:54:43.141 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6811", 2024-02-05T06:54:43.141 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1644050617 2024-02-05T06:54:43.141 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.141 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.141 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:54:43.141 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:54:43.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.141 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:54:43.141 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6810", 2024-02-05T06:54:43.141 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1644050617 2024-02-05T06:54:43.141 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.142 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.142 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.167:6808/1644050617", 2024-02-05T06:54:43.142 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.167:6809/1644050617", 2024-02-05T06:54:43.142 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.167:6811/1644050617", 2024-02-05T06:54:43.142 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.167:6810/1644050617", 2024-02-05T06:54:43.142 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:54:43.142 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:54:43.142 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:54:43.142 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.142 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-02-05T06:54:43.142 INFO:teuthology.orchestra.run.smithi028.stdout: "uuid": "0c545cbc-0873-4232-8807-4af9299ef95e", 2024-02-05T06:54:43.143 INFO:teuthology.orchestra.run.smithi028.stdout: "up": 1, 2024-02-05T06:54:43.143 INFO:teuthology.orchestra.run.smithi028.stdout: "in": 1, 2024-02-05T06:54:43.143 INFO:teuthology.orchestra.run.smithi028.stdout: "weight": 1, 2024-02-05T06:54:43.143 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_affinity": 1, 2024-02-05T06:54:43.143 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_begin": 0, 2024-02-05T06:54:43.143 INFO:teuthology.orchestra.run.smithi028.stdout: "last_clean_end": 0, 2024-02-05T06:54:43.143 INFO:teuthology.orchestra.run.smithi028.stdout: "up_from": 58, 2024-02-05T06:54:43.143 INFO:teuthology.orchestra.run.smithi028.stdout: "up_thru": 434, 2024-02-05T06:54:43.143 INFO:teuthology.orchestra.run.smithi028.stdout: "down_at": 0, 2024-02-05T06:54:43.143 INFO:teuthology.orchestra.run.smithi028.stdout: "lost_at": 0, 2024-02-05T06:54:43.143 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addrs": { 2024-02-05T06:54:43.143 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:54:43.143 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.144 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:54:43.144 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6812", 2024-02-05T06:54:43.144 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4031475764 2024-02-05T06:54:43.144 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.144 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.144 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addrs": { 2024-02-05T06:54:43.144 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:54:43.144 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.144 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:54:43.144 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6813", 2024-02-05T06:54:43.144 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4031475764 2024-02-05T06:54:43.144 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.144 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.145 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addrs": { 2024-02-05T06:54:43.145 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:54:43.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.145 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:54:43.145 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6815", 2024-02-05T06:54:43.145 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4031475764 2024-02-05T06:54:43.145 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.145 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.145 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addrs": { 2024-02-05T06:54:43.145 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-02-05T06:54:43.145 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.145 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-02-05T06:54:43.146 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.167:6814", 2024-02-05T06:54:43.146 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 4031475764 2024-02-05T06:54:43.146 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.146 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.146 INFO:teuthology.orchestra.run.smithi028.stdout: "public_addr": "172.21.15.167:6812/4031475764", 2024-02-05T06:54:43.146 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_addr": "172.21.15.167:6813/4031475764", 2024-02-05T06:54:43.146 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_back_addr": "172.21.15.167:6815/4031475764", 2024-02-05T06:54:43.146 INFO:teuthology.orchestra.run.smithi028.stdout: "heartbeat_front_addr": "172.21.15.167:6814/4031475764", 2024-02-05T06:54:43.146 INFO:teuthology.orchestra.run.smithi028.stdout: "state": [ 2024-02-05T06:54:43.146 INFO:teuthology.orchestra.run.smithi028.stdout: "exists", 2024-02-05T06:54:43.146 INFO:teuthology.orchestra.run.smithi028.stdout: "up" 2024-02-05T06:54:43.146 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.146 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.147 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:54:43.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_xinfo": [ 2024-02-05T06:54:43.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 0, 2024-02-05T06:54:43.147 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:54:43.147 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:54:43.147 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:54:43.147 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:54:43.147 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:54:43.147 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:54:30.807613+0000", 2024-02-05T06:54:43.147 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:54:43.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.147 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.147 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 1, 2024-02-05T06:54:43.148 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:54:43.148 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:54:43.148 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:54:43.148 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:54:43.148 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:54:43.148 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:53:49.195031+0000", 2024-02-05T06:54:43.148 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:54:43.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.148 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.148 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 2, 2024-02-05T06:54:43.148 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:54:43.148 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:54:43.148 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:54:43.149 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:54:43.149 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:54:43.149 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:54:32.860646+0000", 2024-02-05T06:54:43.149 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:54:43.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.149 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.149 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 3, 2024-02-05T06:54:43.149 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:54:43.149 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:54:43.149 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:54:43.149 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:54:43.149 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:54:43.149 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:53:57.360490+0000", 2024-02-05T06:54:43.149 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:54:43.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.150 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 4, 2024-02-05T06:54:43.150 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:54:43.150 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:54:43.150 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:54:43.150 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:54:43.150 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:54:43.150 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:54:08.054438+0000", 2024-02-05T06:54:43.150 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:54:43.150 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.150 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.150 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 5, 2024-02-05T06:54:43.150 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:54:43.150 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:54:43.150 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:54:43.151 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:54:43.151 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:54:43.151 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:53:42.626706+0000", 2024-02-05T06:54:43.151 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:54:43.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.151 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 6, 2024-02-05T06:54:43.151 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:54:43.151 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:54:43.151 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:54:43.151 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:54:43.151 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:54:43.151 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:53:32.376633+0000", 2024-02-05T06:54:43.151 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:54:43.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.151 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.152 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": 7, 2024-02-05T06:54:43.152 INFO:teuthology.orchestra.run.smithi028.stdout: "down_stamp": "0.000000", 2024-02-05T06:54:43.152 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_probability": 0, 2024-02-05T06:54:43.152 INFO:teuthology.orchestra.run.smithi028.stdout: "laggy_interval": 0, 2024-02-05T06:54:43.152 INFO:teuthology.orchestra.run.smithi028.stdout: "features": 4540701272860131327, 2024-02-05T06:54:43.152 INFO:teuthology.orchestra.run.smithi028.stdout: "old_weight": 0, 2024-02-05T06:54:43.152 INFO:teuthology.orchestra.run.smithi028.stdout: "last_purged_snaps_scrub": "2024-02-05T06:53:59.722393+0000", 2024-02-05T06:54:43.152 INFO:teuthology.orchestra.run.smithi028.stdout: "dead_epoch": 0 2024-02-05T06:54:43.152 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.152 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:54:43.152 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap": [ 2024-02-05T06:54:43.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.152 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "3.18", 2024-02-05T06:54:43.153 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-02-05T06:54:43.153 INFO:teuthology.orchestra.run.smithi028.stdout: 5, 2024-02-05T06:54:43.153 INFO:teuthology.orchestra.run.smithi028.stdout: 2, 2024-02-05T06:54:43.153 INFO:teuthology.orchestra.run.smithi028.stdout: 4 2024-02-05T06:54:43.153 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.153 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "5.1b", 2024-02-05T06:54:43.153 INFO:teuthology.orchestra.run.smithi028.stdout: "osds": [ 2024-02-05T06:54:43.153 INFO:teuthology.orchestra.run.smithi028.stdout: 3, 2024-02-05T06:54:43.153 INFO:teuthology.orchestra.run.smithi028.stdout: 0, 2024-02-05T06:54:43.153 INFO:teuthology.orchestra.run.smithi028.stdout: 1 2024-02-05T06:54:43.153 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.153 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.153 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:54:43.153 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_items": [ 2024-02-05T06:54:43.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.154 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "3.9", 2024-02-05T06:54:43.154 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:54:43.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.154 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 4, 2024-02-05T06:54:43.154 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 1 2024-02-05T06:54:43.154 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.154 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.154 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "3.a", 2024-02-05T06:54:43.154 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:54:43.154 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.154 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 4, 2024-02-05T06:54:43.154 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-02-05T06:54:43.154 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.155 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.155 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.f", 2024-02-05T06:54:43.155 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:54:43.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.155 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-02-05T06:54:43.155 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 0 2024-02-05T06:54:43.155 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.155 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.155 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.12", 2024-02-05T06:54:43.155 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:54:43.155 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.155 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-02-05T06:54:43.155 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 0 2024-02-05T06:54:43.156 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.156 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.156 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.14", 2024-02-05T06:54:43.156 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:54:43.156 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.156 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-02-05T06:54:43.156 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-02-05T06:54:43.156 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.157 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.157 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.15", 2024-02-05T06:54:43.157 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:54:43.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.157 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 3, 2024-02-05T06:54:43.157 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-02-05T06:54:43.157 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.157 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.157 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "4.1d", 2024-02-05T06:54:43.157 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:54:43.157 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.157 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 1, 2024-02-05T06:54:43.158 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 2 2024-02-05T06:54:43.158 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.159 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.159 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "5.1c", 2024-02-05T06:54:43.159 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:54:43.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.159 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 4, 2024-02-05T06:54:43.159 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 6 2024-02-05T06:54:43.159 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.159 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.159 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "5.1e", 2024-02-05T06:54:43.159 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:54:43.159 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.160 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 0, 2024-02-05T06:54:43.160 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 3 2024-02-05T06:54:43.160 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.160 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.160 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "6.12", 2024-02-05T06:54:43.160 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:54:43.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.160 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 2, 2024-02-05T06:54:43.160 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 1 2024-02-05T06:54:43.160 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.160 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.160 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 4, 2024-02-05T06:54:43.161 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 0 2024-02-05T06:54:43.161 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.161 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.161 INFO:teuthology.orchestra.run.smithi028.stdout: "pgid": "6.13", 2024-02-05T06:54:43.161 INFO:teuthology.orchestra.run.smithi028.stdout: "mappings": [ 2024-02-05T06:54:43.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-02-05T06:54:43.161 INFO:teuthology.orchestra.run.smithi028.stdout: "from": 3, 2024-02-05T06:54:43.161 INFO:teuthology.orchestra.run.smithi028.stdout: "to": 1 2024-02-05T06:54:43.161 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.161 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-02-05T06:54:43.161 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.161 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-02-05T06:54:43.161 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_upmap_primaries": [], 2024-02-05T06:54:43.162 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_temp": [], 2024-02-05T06:54:43.162 INFO:teuthology.orchestra.run.smithi028.stdout: "primary_temp": [], 2024-02-05T06:54:43.162 INFO:teuthology.orchestra.run.smithi028.stdout: "blocklist": { 2024-02-05T06:54:43.162 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/1001442173": "2024-02-06T06:34:10.127914+0000", 2024-02-05T06:54:43.162 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3160026720": "2024-02-06T06:34:10.127914+0000", 2024-02-05T06:54:43.162 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/558171292": "2024-02-06T06:34:10.127914+0000", 2024-02-05T06:54:43.162 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2832437766": "2024-02-06T06:34:10.127914+0000", 2024-02-05T06:54:43.162 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/4125764599": "2024-02-06T06:28:54.105170+0000", 2024-02-05T06:54:43.162 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1081543653": "2024-02-06T06:28:54.105170+0000", 2024-02-05T06:54:43.162 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3794420554": "2024-02-06T06:28:27.553034+0000", 2024-02-05T06:54:43.162 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/1560471317": "2024-02-06T06:34:10.127914+0000", 2024-02-05T06:54:43.162 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:6800/4266601437": "2024-02-06T06:28:27.553034+0000", 2024-02-05T06:54:43.162 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/388893831": "2024-02-06T06:28:27.553034+0000", 2024-02-05T06:54:43.162 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/460817910": "2024-02-06T06:28:27.553034+0000", 2024-02-05T06:54:43.162 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/2728305743": "2024-02-06T06:28:54.105170+0000", 2024-02-05T06:54:43.163 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3149394045": "2024-02-06T06:34:10.127914+0000", 2024-02-05T06:54:43.163 INFO:teuthology.orchestra.run.smithi028.stdout: "172.21.15.28:0/3605013594": "2024-02-06T06:28:54.105170+0000" 2024-02-05T06:54:43.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.163 INFO:teuthology.orchestra.run.smithi028.stdout: "range_blocklist": {}, 2024-02-05T06:54:43.163 INFO:teuthology.orchestra.run.smithi028.stdout: "erasure_code_profiles": { 2024-02-05T06:54:43.163 INFO:teuthology.orchestra.run.smithi028.stdout: "default": { 2024-02-05T06:54:43.163 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-failure-domain": "osd", 2024-02-05T06:54:43.163 INFO:teuthology.orchestra.run.smithi028.stdout: "k": "2", 2024-02-05T06:54:43.163 INFO:teuthology.orchestra.run.smithi028.stdout: "m": "1", 2024-02-05T06:54:43.163 INFO:teuthology.orchestra.run.smithi028.stdout: "plugin": "jerasure", 2024-02-05T06:54:43.163 INFO:teuthology.orchestra.run.smithi028.stdout: "technique": "reed_sol_van" 2024-02-05T06:54:43.163 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.163 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-02-05T06:54:43.163 INFO:teuthology.orchestra.run.smithi028.stdout: "removed_snaps_queue": [], 2024-02-05T06:54:43.163 INFO:teuthology.orchestra.run.smithi028.stdout: "new_removed_snaps": [], 2024-02-05T06:54:43.164 INFO:teuthology.orchestra.run.smithi028.stdout: "new_purged_snaps": [], 2024-02-05T06:54:43.164 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_node_flags": {}, 2024-02-05T06:54:43.164 INFO:teuthology.orchestra.run.smithi028.stdout: "device_class_flags": {}, 2024-02-05T06:54:43.164 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode": { 2024-02-05T06:54:43.164 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_enabled": false, 2024-02-05T06:54:43.164 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_bucket_count": 0, 2024-02-05T06:54:43.164 INFO:teuthology.orchestra.run.smithi028.stdout: "degraded_stretch_mode": 0, 2024-02-05T06:54:43.164 INFO:teuthology.orchestra.run.smithi028.stdout: "recovering_stretch_mode": 0, 2024-02-05T06:54:43.164 INFO:teuthology.orchestra.run.smithi028.stdout: "stretch_mode_bucket": 0 2024-02-05T06:54:43.164 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-02-05T06:54:43.164 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-02-05T06:54:43.217 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:43.257 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:43.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:43.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:43.359 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:43.394 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:43.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:43.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:43.496 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:43.530 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:43.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:43.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:43.631 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:43.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:43.673 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:43.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:43.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:43 smithi167 ceph-mon[32774]: 1.11 scrub starts 2024-02-05T06:54:43.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:43 smithi167 ceph-mon[32774]: 1.11 scrub ok 2024-02-05T06:54:43.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:43 smithi167 ceph-mon[32774]: pgmap v1013: 168 pgs: 168 active+clean; 704 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 977 op/s 2024-02-05T06:54:43.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:43 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:54:43.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:43 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2640080966' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T06:54:43.733 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:54:43 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:54:43.774 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:43.777 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:43 smithi028 ceph-mon[29442]: 1.11 scrub starts 2024-02-05T06:54:43.777 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:43 smithi028 ceph-mon[29442]: 1.11 scrub ok 2024-02-05T06:54:43.777 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:43 smithi028 ceph-mon[29442]: pgmap v1013: 168 pgs: 168 active+clean; 704 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 977 op/s 2024-02-05T06:54:43.777 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:43 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:54:43.777 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:43 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2640080966' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T06:54:43.778 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:43 smithi028 ceph-mon[37984]: 1.11 scrub starts 2024-02-05T06:54:43.778 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:43 smithi028 ceph-mon[37984]: 1.11 scrub ok 2024-02-05T06:54:43.778 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:43 smithi028 ceph-mon[37984]: pgmap v1013: 168 pgs: 168 active+clean; 704 MiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 977 op/s 2024-02-05T06:54:43.778 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:43 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:54:43.778 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:43 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2640080966' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-02-05T06:54:43.805 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:43.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:43.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:43.906 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:43.938 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:43.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:43.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:44.039 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:44.070 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:44.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:44.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:44.172 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:44.207 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:44.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:44.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:44.309 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:44.344 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:44.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:44.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:44.446 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:44.485 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:44.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:44.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:44.587 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:44.618 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:44.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:44.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:44.720 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:44.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:44 smithi167 ceph-mon[32774]: 5.f scrub starts 2024-02-05T06:54:44.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:44 smithi167 ceph-mon[32774]: 5.f scrub ok 2024-02-05T06:54:44.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:44 smithi167 ceph-mon[32774]: 4.1f scrub starts 2024-02-05T06:54:44.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:44 smithi167 ceph-mon[32774]: 4.1f scrub ok 2024-02-05T06:54:44.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:44 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:54:44.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:44 smithi167 ceph-mon[32774]: 1.b scrub starts 2024-02-05T06:54:44.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:44 smithi167 ceph-mon[32774]: 1.b scrub ok 2024-02-05T06:54:44.752 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:44.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:44.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:44.853 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:44.882 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:44.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:44.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:44.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:44 smithi028 ceph-mon[29442]: 5.f scrub starts 2024-02-05T06:54:44.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:44 smithi028 ceph-mon[29442]: 5.f scrub ok 2024-02-05T06:54:44.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:44 smithi028 ceph-mon[29442]: 4.1f scrub starts 2024-02-05T06:54:44.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:44 smithi028 ceph-mon[29442]: 4.1f scrub ok 2024-02-05T06:54:44.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:44 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:54:44.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:44 smithi028 ceph-mon[29442]: 1.b scrub starts 2024-02-05T06:54:44.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:44 smithi028 ceph-mon[29442]: 1.b scrub ok 2024-02-05T06:54:44.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:44 smithi028 ceph-mon[37984]: 5.f scrub starts 2024-02-05T06:54:44.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:44 smithi028 ceph-mon[37984]: 5.f scrub ok 2024-02-05T06:54:44.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:44 smithi028 ceph-mon[37984]: 4.1f scrub starts 2024-02-05T06:54:44.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:44 smithi028 ceph-mon[37984]: 4.1f scrub ok 2024-02-05T06:54:44.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:44 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:54:44.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:44 smithi028 ceph-mon[37984]: 1.b scrub starts 2024-02-05T06:54:44.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:44 smithi028 ceph-mon[37984]: 1.b scrub ok 2024-02-05T06:54:44.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:45.038 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:45.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:45.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:45.140 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:45.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:45.174 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:45.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:45.275 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:45.313 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:45.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:45.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:45.415 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:45.465 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:45.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:45.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:45.567 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:45.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:45.609 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:45.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:45.621 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:45 smithi167 ceph-mon[32774]: 6.17 scrub starts 2024-02-05T06:54:45.622 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:45 smithi167 ceph-mon[32774]: 6.17 scrub ok 2024-02-05T06:54:45.622 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:45 smithi167 ceph-mon[32774]: 6.6 scrub starts 2024-02-05T06:54:45.622 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:45 smithi167 ceph-mon[32774]: pgmap v1014: 168 pgs: 168 active+clean; 552 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s 2024-02-05T06:54:45.622 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:45 smithi167 ceph-mon[32774]: 6.6 scrub ok 2024-02-05T06:54:45.622 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:45 smithi167 ceph-mon[32774]: 5.c scrub starts 2024-02-05T06:54:45.622 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:45 smithi167 ceph-mon[32774]: 5.c scrub ok 2024-02-05T06:54:45.710 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:45.731 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:45 smithi028 ceph-mon[29442]: 6.17 scrub starts 2024-02-05T06:54:45.731 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:45 smithi028 ceph-mon[29442]: 6.17 scrub ok 2024-02-05T06:54:45.731 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:45 smithi028 ceph-mon[29442]: 6.6 scrub starts 2024-02-05T06:54:45.732 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:45 smithi028 ceph-mon[29442]: pgmap v1014: 168 pgs: 168 active+clean; 552 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s 2024-02-05T06:54:45.732 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:45 smithi028 ceph-mon[29442]: 6.6 scrub ok 2024-02-05T06:54:45.732 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:45 smithi028 ceph-mon[29442]: 5.c scrub starts 2024-02-05T06:54:45.732 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:45 smithi028 ceph-mon[29442]: 5.c scrub ok 2024-02-05T06:54:45.732 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:45 smithi028 ceph-mon[37984]: 6.17 scrub starts 2024-02-05T06:54:45.732 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:45 smithi028 ceph-mon[37984]: 6.17 scrub ok 2024-02-05T06:54:45.732 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:45 smithi028 ceph-mon[37984]: 6.6 scrub starts 2024-02-05T06:54:45.732 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:45 smithi028 ceph-mon[37984]: pgmap v1014: 168 pgs: 168 active+clean; 552 MiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s 2024-02-05T06:54:45.732 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:45 smithi028 ceph-mon[37984]: 6.6 scrub ok 2024-02-05T06:54:45.732 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:45 smithi028 ceph-mon[37984]: 5.c scrub starts 2024-02-05T06:54:45.732 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:45 smithi028 ceph-mon[37984]: 5.c scrub ok 2024-02-05T06:54:45.750 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:45.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:45.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:45.804 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:54:45.804 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:54:45.852 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:45.890 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:45.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:45.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:45.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:46.033 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:46.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:46.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:46.135 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:46.172 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:46.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:46.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:46.223 INFO:tasks.thrashosds.thrasher:j is {'epoch': 476, 'fsid': '9801e956-c3ef-11ee-95b6-87774f69a715', 'created': '2024-02-05T06:27:59.919226+0000', 'modified': '2024-02-05T06:54:36.380707+0000', 'last_up_change': '2024-02-05T06:33:15.833573+0000', 'last_in_change': '2024-02-05T06:41:42.383087+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-05T06:31:10.441424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '431', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '429', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.899999976158142, 'score_stable': 1.899999976158142, 'optimal_score': 1, 'raw_score_acting': 1.899999976158142, 'raw_score_stable': 1.899999976158142, 'primary_affinity_weighted': 0.8808403015136719, 'average_primary_affinity': 0.8799999952316284, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-02-05T06:33:27.397416+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.8808403015136719, 'average_primary_affinity': 0.8799999952316284, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-02-05T06:33:28.090506+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8808403015136719, 'average_primary_affinity': 0.8799999952316284, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-02-05T06:33:29.866249+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.8808403015136719, 'average_primary_affinity': 0.8799999952316284, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-02-05T06:33:31.826006+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8808403015136719, 'average_primary_affinity': 0.8799999952316284, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-02-05T06:33:33.890835+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.8808403015136719, 'average_primary_affinity': 0.8799999952316284, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-02-05T06:51:11.916055+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '422', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8808403015136719, 'average_primary_affinity': 0.8799999952316284, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': 'c19fdd5b-09f6-45d4-a519-0a938cf1f2a6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 434, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6801', 'nonce': 301279106}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6802', 'nonce': 301279106}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6804', 'nonce': 301279106}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6803', 'nonce': 301279106}]}, 'public_addr': '172.21.15.28:6801/301279106', 'cluster_addr': '172.21.15.28:6802/301279106', 'heartbeat_back_addr': '172.21.15.28:6804/301279106', 'heartbeat_front_addr': '172.21.15.28:6803/301279106', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '52245f6e-eb96-4d0d-985b-dbb6fd3ce8de', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 434, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6805', 'nonce': 1795224649}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6806', 'nonce': 1795224649}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6808', 'nonce': 1795224649}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6807', 'nonce': 1795224649}]}, 'public_addr': '172.21.15.28:6805/1795224649', 'cluster_addr': '172.21.15.28:6806/1795224649', 'heartbeat_back_addr': '172.21.15.28:6808/1795224649', 'heartbeat_front_addr': '172.21.15.28:6807/1795224649', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '1e18b630-afdf-4de9-a1a2-37bdfdd01c67', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 434, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6809', 'nonce': 1993280018}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6810', 'nonce': 1993280018}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6812', 'nonce': 1993280018}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6811', 'nonce': 1993280018}]}, 'public_addr': '172.21.15.28:6809/1993280018', 'cluster_addr': '172.21.15.28:6810/1993280018', 'heartbeat_back_addr': '172.21.15.28:6812/1993280018', 'heartbeat_front_addr': '172.21.15.28:6811/1993280018', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '3ae795af-3d59-41a6-95eb-b8447ff61282', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 434, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6813', 'nonce': 1294621398}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6814', 'nonce': 1294621398}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6816', 'nonce': 1294621398}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.28:6815', 'nonce': 1294621398}]}, 'public_addr': '172.21.15.28:6813/1294621398', 'cluster_addr': '172.21.15.28:6814/1294621398', 'heartbeat_back_addr': '172.21.15.28:6816/1294621398', 'heartbeat_front_addr': '172.21.15.28:6815/1294621398', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'fc6bd83d-570c-48c0-964d-e09b4608bc32', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.046722412109375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 267, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6800', 'nonce': 4052090595}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6801', 'nonce': 4052090595}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6803', 'nonce': 4052090595}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6802', 'nonce': 4052090595}]}, 'public_addr': '172.21.15.167:6800/4052090595', 'cluster_addr': '172.21.15.167:6801/4052090595', 'heartbeat_back_addr': '172.21.15.167:6803/4052090595', 'heartbeat_front_addr': '172.21.15.167:6802/4052090595', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '895bf465-2531-4d9f-bc23-1e75fd969f07', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 434, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6804', 'nonce': 1631080500}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6805', 'nonce': 1631080500}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6807', 'nonce': 1631080500}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6806', 'nonce': 1631080500}]}, 'public_addr': '172.21.15.167:6804/1631080500', 'cluster_addr': '172.21.15.167:6805/1631080500', 'heartbeat_back_addr': '172.21.15.167:6807/1631080500', 'heartbeat_front_addr': '172.21.15.167:6806/1631080500', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'ce0a9300-87a4-4e37-b702-d667f168c0e0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 434, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6808', 'nonce': 1644050617}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6809', 'nonce': 1644050617}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6811', 'nonce': 1644050617}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6810', 'nonce': 1644050617}]}, 'public_addr': '172.21.15.167:6808/1644050617', 'cluster_addr': '172.21.15.167:6809/1644050617', 'heartbeat_back_addr': '172.21.15.167:6811/1644050617', 'heartbeat_front_addr': '172.21.15.167:6810/1644050617', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '0c545cbc-0873-4232-8807-4af9299ef95e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 434, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6812', 'nonce': 4031475764}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6813', 'nonce': 4031475764}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6815', 'nonce': 4031475764}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.167:6814', 'nonce': 4031475764}]}, 'public_addr': '172.21.15.167:6812/4031475764', 'cluster_addr': '172.21.15.167:6813/4031475764', 'heartbeat_back_addr': '172.21.15.167:6815/4031475764', 'heartbeat_front_addr': '172.21.15.167:6814/4031475764', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:54:30.807613+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:53:49.195031+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:54:32.860646+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:53:57.360490+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:54:08.054438+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:53:42.626706+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:53:32.376633+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701272860131327, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-02-05T06:53:59.722393+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.18', 'osds': [5, 2, 4]}, {'pgid': '5.1b', 'osds': [3, 0, 1]}], 'pg_upmap_items': [{'pgid': '3.9', 'mappings': [{'from': 4, 'to': 1}]}, {'pgid': '3.a', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '4.14', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.15', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.1c', 'mappings': [{'from': 4, 'to': 6}]}, {'pgid': '5.1e', 'mappings': [{'from': 0, 'to': 3}]}, {'pgid': '6.12', 'mappings': [{'from': 2, 'to': 1}, {'from': 4, 'to': 0}]}, {'pgid': '6.13', 'mappings': [{'from': 3, 'to': 1}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.28:6800/1001442173': '2024-02-06T06:34:10.127914+0000', '172.21.15.28:0/3160026720': '2024-02-06T06:34:10.127914+0000', '172.21.15.28:0/558171292': '2024-02-06T06:34:10.127914+0000', '172.21.15.28:0/2832437766': '2024-02-06T06:34:10.127914+0000', '172.21.15.28:6800/4125764599': '2024-02-06T06:28:54.105170+0000', '172.21.15.28:0/1081543653': '2024-02-06T06:28:54.105170+0000', '172.21.15.28:0/3794420554': '2024-02-06T06:28:27.553034+0000', '172.21.15.28:0/1560471317': '2024-02-06T06:34:10.127914+0000', '172.21.15.28:6800/4266601437': '2024-02-06T06:28:27.553034+0000', '172.21.15.28:0/388893831': '2024-02-06T06:28:27.553034+0000', '172.21.15.28:0/460817910': '2024-02-06T06:28:27.553034+0000', '172.21.15.28:0/2728305743': '2024-02-06T06:28:54.105170+0000', '172.21.15.28:0/3149394045': '2024-02-06T06:34:10.127914+0000', '172.21.15.28:0/3605013594': '2024-02-06T06:28:54.105170+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-02-05T06:54:46.223 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T06:54:46.274 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:46.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:46.320 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:46.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:46.421 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:46.449 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:46.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:46.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:46.491 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-02-05T06:54:46.551 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:46.601 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:46.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:46.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:46.628 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:54:46.703 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:46.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:46 smithi167 ceph-mon[32774]: 1.10 scrub starts 2024-02-05T06:54:46.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:46 smithi167 ceph-mon[32774]: 1.10 scrub ok 2024-02-05T06:54:46.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:46 smithi167 ceph-mon[32774]: 5.a scrub starts 2024-02-05T06:54:46.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:46 smithi167 ceph-mon[32774]: 5.a scrub ok 2024-02-05T06:54:46.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:46 smithi167 ceph-mon[32774]: 4.18 scrub starts 2024-02-05T06:54:46.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:46 smithi167 ceph-mon[32774]: 4.18 scrub ok 2024-02-05T06:54:46.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:46 smithi167 ceph-mon[32774]: 1.7 scrub starts 2024-02-05T06:54:46.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:46 smithi167 ceph-mon[32774]: 1.7 scrub ok 2024-02-05T06:54:46.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:46 smithi167 ceph-mon[32774]: 2.0 scrub starts 2024-02-05T06:54:46.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:46 smithi167 ceph-mon[32774]: 2.0 scrub ok 2024-02-05T06:54:46.749 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:46.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:46.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:46.775 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:46 smithi028 ceph-mon[29442]: 1.10 scrub starts 2024-02-05T06:54:46.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:46 smithi028 ceph-mon[29442]: 1.10 scrub ok 2024-02-05T06:54:46.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:46 smithi028 ceph-mon[29442]: 5.a scrub starts 2024-02-05T06:54:46.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:46 smithi028 ceph-mon[29442]: 5.a scrub ok 2024-02-05T06:54:46.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:46 smithi028 ceph-mon[29442]: 4.18 scrub starts 2024-02-05T06:54:46.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:46 smithi028 ceph-mon[29442]: 4.18 scrub ok 2024-02-05T06:54:46.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:46 smithi028 ceph-mon[29442]: 1.7 scrub starts 2024-02-05T06:54:46.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:46 smithi028 ceph-mon[29442]: 1.7 scrub ok 2024-02-05T06:54:46.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:46 smithi028 ceph-mon[29442]: 2.0 scrub starts 2024-02-05T06:54:46.776 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:46 smithi028 ceph-mon[29442]: 2.0 scrub ok 2024-02-05T06:54:46.776 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:46 smithi028 ceph-mon[37984]: 1.10 scrub starts 2024-02-05T06:54:46.776 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:46 smithi028 ceph-mon[37984]: 1.10 scrub ok 2024-02-05T06:54:46.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:46 smithi028 ceph-mon[37984]: 5.a scrub starts 2024-02-05T06:54:46.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:46 smithi028 ceph-mon[37984]: 5.a scrub ok 2024-02-05T06:54:46.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:46 smithi028 ceph-mon[37984]: 4.18 scrub starts 2024-02-05T06:54:46.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:46 smithi028 ceph-mon[37984]: 4.18 scrub ok 2024-02-05T06:54:46.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:46 smithi028 ceph-mon[37984]: 1.7 scrub starts 2024-02-05T06:54:46.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:46 smithi028 ceph-mon[37984]: 1.7 scrub ok 2024-02-05T06:54:46.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:46 smithi028 ceph-mon[37984]: 2.0 scrub starts 2024-02-05T06:54:46.777 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:46 smithi028 ceph-mon[37984]: 2.0 scrub ok 2024-02-05T06:54:46.848 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:54:46.851 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:46.897 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:46.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:46.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:46.999 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:47.031 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:47.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:47.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:47.133 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:47.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:47.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:47.276 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:47.316 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:47.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:47.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:47.417 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:47.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:47.456 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:47.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:47.461 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:54:47.537 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-02-05T06:54:47.557 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:47.585 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:47.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:47.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:47.686 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:47.724 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:47.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:47.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:47.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:47 smithi167 ceph-mon[32774]: 3.16 scrub starts 2024-02-05T06:54:47.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:47 smithi167 ceph-mon[32774]: 3.16 scrub ok 2024-02-05T06:54:47.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:47 smithi167 ceph-mon[32774]: 1.6 scrub starts 2024-02-05T06:54:47.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:47 smithi167 ceph-mon[32774]: 1.6 scrub ok 2024-02-05T06:54:47.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:47 smithi167 ceph-mon[32774]: pgmap v1015: 168 pgs: 168 active+clean; 531 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1001 B/s rd, 0 B/s wr, 1.15k op/s 2024-02-05T06:54:47.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:47 smithi167 ceph-mon[32774]: 1.3 scrub starts 2024-02-05T06:54:47.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:47 smithi167 ceph-mon[32774]: 1.3 scrub ok 2024-02-05T06:54:47.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:47 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2245052143' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:54:47.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:47 smithi167 ceph-mon[32774]: from='client.46079 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:54:47.825 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:47.853 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:47.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:47.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:47.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:47 smithi028 ceph-mon[37984]: 3.16 scrub starts 2024-02-05T06:54:47.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:47 smithi028 ceph-mon[37984]: 3.16 scrub ok 2024-02-05T06:54:47.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:47 smithi028 ceph-mon[37984]: 1.6 scrub starts 2024-02-05T06:54:47.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:47 smithi028 ceph-mon[37984]: 1.6 scrub ok 2024-02-05T06:54:47.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:47 smithi028 ceph-mon[37984]: pgmap v1015: 168 pgs: 168 active+clean; 531 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1001 B/s rd, 0 B/s wr, 1.15k op/s 2024-02-05T06:54:47.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:47 smithi028 ceph-mon[37984]: 1.3 scrub starts 2024-02-05T06:54:47.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:47 smithi028 ceph-mon[37984]: 1.3 scrub ok 2024-02-05T06:54:47.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:47 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2245052143' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:54:47.885 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:47 smithi028 ceph-mon[37984]: from='client.46079 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:54:47.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:47 smithi028 ceph-mon[29442]: 3.16 scrub starts 2024-02-05T06:54:47.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:47 smithi028 ceph-mon[29442]: 3.16 scrub ok 2024-02-05T06:54:47.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:47 smithi028 ceph-mon[29442]: 1.6 scrub starts 2024-02-05T06:54:47.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:47 smithi028 ceph-mon[29442]: 1.6 scrub ok 2024-02-05T06:54:47.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:47 smithi028 ceph-mon[29442]: pgmap v1015: 168 pgs: 168 active+clean; 531 MiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1001 B/s rd, 0 B/s wr, 1.15k op/s 2024-02-05T06:54:47.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:47 smithi028 ceph-mon[29442]: 1.3 scrub starts 2024-02-05T06:54:47.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:47 smithi028 ceph-mon[29442]: 1.3 scrub ok 2024-02-05T06:54:47.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:47 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2245052143' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:54:47.886 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:47 smithi028 ceph-mon[29442]: from='client.46079 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:54:47.954 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:47.989 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:47.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:47.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:48.090 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:48.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:48.121 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:48.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:48.222 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:48.263 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:48.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:48.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:48.365 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:48.402 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:48.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:48.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:48.504 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:48.536 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:48.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:48.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:48.611 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:48 smithi167 ceph-mon[32774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:54:48.611 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:48 smithi167 ceph-mon[32774]: Cluster is now healthy 2024-02-05T06:54:48.611 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:48 smithi167 ceph-mon[32774]: from='client.46079 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:54:48.611 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:48 smithi167 ceph-mon[32774]: osdmap e477: 8 total, 8 up, 8 in 2024-02-05T06:54:48.611 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:48 smithi167 ceph-mon[32774]: 1.e deep-scrub starts 2024-02-05T06:54:48.611 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:48 smithi167 ceph-mon[32774]: 1.e deep-scrub ok 2024-02-05T06:54:48.638 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:48.681 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:48.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:48.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:48.783 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:48.800 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:54:48.800 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:54:48.819 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:48.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:48.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:48.921 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:48.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:48 smithi028 ceph-mon[29442]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:54:48.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:48 smithi028 ceph-mon[29442]: Cluster is now healthy 2024-02-05T06:54:48.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:48 smithi028 ceph-mon[29442]: from='client.46079 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:54:48.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:48 smithi028 ceph-mon[29442]: osdmap e477: 8 total, 8 up, 8 in 2024-02-05T06:54:48.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:48 smithi028 ceph-mon[29442]: 1.e deep-scrub starts 2024-02-05T06:54:48.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:48 smithi028 ceph-mon[29442]: 1.e deep-scrub ok 2024-02-05T06:54:48.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:48 smithi028 ceph-mon[37984]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-02-05T06:54:48.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:48 smithi028 ceph-mon[37984]: Cluster is now healthy 2024-02-05T06:54:48.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:48 smithi028 ceph-mon[37984]: from='client.46079 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:54:48.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:48 smithi028 ceph-mon[37984]: osdmap e477: 8 total, 8 up, 8 in 2024-02-05T06:54:48.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:48 smithi028 ceph-mon[37984]: 1.e deep-scrub starts 2024-02-05T06:54:48.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:48 smithi028 ceph-mon[37984]: 1.e deep-scrub ok 2024-02-05T06:54:48.955 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:48.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:48.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:49.057 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:49.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:49.098 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:49.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:49.199 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:49.236 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:49.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:49.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:49.338 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:49.369 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:49.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:49.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:49.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:49.513 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:49.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:49.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:49.615 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:49.654 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:49.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:49.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:49.756 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:49.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:49.791 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:49.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:49.892 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:49.932 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:49.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:49.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:49.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:49 smithi028 ceph-mon[29442]: 4.5 scrub starts 2024-02-05T06:54:49.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:49 smithi028 ceph-mon[29442]: 4.5 scrub ok 2024-02-05T06:54:49.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:49 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:54:49.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:49 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:54:49.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:49 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:54:49.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:49 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:54:49.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:49 smithi028 ceph-mon[29442]: osdmap e478: 8 total, 8 up, 8 in 2024-02-05T06:54:49.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:49 smithi028 ceph-mon[29442]: 6.1f scrub starts 2024-02-05T06:54:49.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:49 smithi028 ceph-mon[29442]: pgmap v1018: 168 pgs: 168 active+clean; 185 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-02-05T06:54:49.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:49 smithi028 ceph-mon[29442]: 1.d scrub starts 2024-02-05T06:54:49.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:49 smithi028 ceph-mon[29442]: 1.d scrub ok 2024-02-05T06:54:49.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:49 smithi028 ceph-mon[29442]: 6.e scrub starts 2024-02-05T06:54:49.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:49 smithi028 ceph-mon[29442]: 6.1f scrub ok 2024-02-05T06:54:49.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:49 smithi028 ceph-mon[29442]: 6.e scrub ok 2024-02-05T06:54:49.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:49 smithi028 ceph-mon[37984]: 4.5 scrub starts 2024-02-05T06:54:49.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:49 smithi028 ceph-mon[37984]: 4.5 scrub ok 2024-02-05T06:54:49.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:49 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:54:49.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:49 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:54:49.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:49 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:54:49.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:49 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:54:49.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:49 smithi028 ceph-mon[37984]: osdmap e478: 8 total, 8 up, 8 in 2024-02-05T06:54:49.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:49 smithi028 ceph-mon[37984]: 6.1f scrub starts 2024-02-05T06:54:49.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:49 smithi028 ceph-mon[37984]: pgmap v1018: 168 pgs: 168 active+clean; 185 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-02-05T06:54:49.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:49 smithi028 ceph-mon[37984]: 1.d scrub starts 2024-02-05T06:54:49.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:49 smithi028 ceph-mon[37984]: 1.d scrub ok 2024-02-05T06:54:49.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:49 smithi028 ceph-mon[37984]: 6.e scrub starts 2024-02-05T06:54:49.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:49 smithi028 ceph-mon[37984]: 6.1f scrub ok 2024-02-05T06:54:49.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:49 smithi028 ceph-mon[37984]: 6.e scrub ok 2024-02-05T06:54:49.959 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Removed 126963 objects 2024-02-05T06:54:49.959 INFO:tasks.radosbench.radosbench.0.smithi028.stdout:Clean up completed and total clean up time :117.565 2024-02-05T06:54:49.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:49 smithi167 ceph-mon[32774]: 4.5 scrub starts 2024-02-05T06:54:49.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:49 smithi167 ceph-mon[32774]: 4.5 scrub ok 2024-02-05T06:54:49.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:49 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:54:49.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:49 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:54:49.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:49 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:54:49.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:49 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:54:49.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:49 smithi167 ceph-mon[32774]: osdmap e478: 8 total, 8 up, 8 in 2024-02-05T06:54:49.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:49 smithi167 ceph-mon[32774]: 6.1f scrub starts 2024-02-05T06:54:49.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:49 smithi167 ceph-mon[32774]: pgmap v1018: 168 pgs: 168 active+clean; 185 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-02-05T06:54:49.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:49 smithi167 ceph-mon[32774]: 1.d scrub starts 2024-02-05T06:54:49.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:49 smithi167 ceph-mon[32774]: 1.d scrub ok 2024-02-05T06:54:49.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:49 smithi167 ceph-mon[32774]: 6.e scrub starts 2024-02-05T06:54:49.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:49 smithi167 ceph-mon[32774]: 6.1f scrub ok 2024-02-05T06:54:49.984 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:49 smithi167 ceph-mon[32774]: 6.e scrub ok 2024-02-05T06:54:50.034 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:50.057 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:50.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:50.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:50.061 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-02-05T06:54:50.159 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:50.183 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:50.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:50.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:50.284 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:50.299 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd set noscrub 2024-02-05T06:54:50.304 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:54:50.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:50.309 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:50.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:50.410 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:50.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:50.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:50.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:50.558 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:50.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:50.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:50.659 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:50.684 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:50.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:50.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:50.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:50 smithi028 ceph-mon[29442]: 6.16 scrub starts 2024-02-05T06:54:50.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:50 smithi028 ceph-mon[37984]: 6.16 scrub starts 2024-02-05T06:54:50.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:50 smithi167 ceph-mon[32774]: 6.16 scrub starts 2024-02-05T06:54:50.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:50.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:50.810 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:50.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:50.910 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:50.934 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:50.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:50.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:51.035 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:51.060 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:51.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:51.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:51.161 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:51.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:51.185 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:51.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:51.286 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:51.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:51.314 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:51.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:51.415 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:51.438 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:51.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:51.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:51.540 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:51.563 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:51.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:51.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:51.665 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:51.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:51 smithi028 ceph-mon[29442]: 6.16 scrub ok 2024-02-05T06:54:51.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:51 smithi028 ceph-mon[29442]: 6.1a scrub starts 2024-02-05T06:54:51.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:51 smithi028 ceph-mon[29442]: 6.1a scrub ok 2024-02-05T06:54:51.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:51 smithi028 ceph-mon[29442]: 5.1f scrub starts 2024-02-05T06:54:51.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:51 smithi028 ceph-mon[29442]: 5.1f scrub ok 2024-02-05T06:54:51.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:51 smithi028 ceph-mon[29442]: pgmap v1019: 168 pgs: 168 active+clean; 185 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.04k op/s 2024-02-05T06:54:51.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:51 smithi028 ceph-mon[29442]: 1.9 deep-scrub starts 2024-02-05T06:54:51.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:51 smithi028 ceph-mon[29442]: 1.9 deep-scrub ok 2024-02-05T06:54:51.677 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:51 smithi028 ceph-mon[37984]: 6.16 scrub ok 2024-02-05T06:54:51.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:51 smithi028 ceph-mon[37984]: 6.1a scrub starts 2024-02-05T06:54:51.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:51 smithi028 ceph-mon[37984]: 6.1a scrub ok 2024-02-05T06:54:51.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:51 smithi028 ceph-mon[37984]: 5.1f scrub starts 2024-02-05T06:54:51.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:51 smithi028 ceph-mon[37984]: 5.1f scrub ok 2024-02-05T06:54:51.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:51 smithi028 ceph-mon[37984]: pgmap v1019: 168 pgs: 168 active+clean; 185 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.04k op/s 2024-02-05T06:54:51.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:51 smithi028 ceph-mon[37984]: 1.9 deep-scrub starts 2024-02-05T06:54:51.678 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:51 smithi028 ceph-mon[37984]: 1.9 deep-scrub ok 2024-02-05T06:54:51.690 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:51.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:51.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:51.723 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:51 smithi167 ceph-mon[32774]: 6.16 scrub ok 2024-02-05T06:54:51.723 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:51 smithi167 ceph-mon[32774]: 6.1a scrub starts 2024-02-05T06:54:51.724 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:51 smithi167 ceph-mon[32774]: 6.1a scrub ok 2024-02-05T06:54:51.724 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:51 smithi167 ceph-mon[32774]: 5.1f scrub starts 2024-02-05T06:54:51.724 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:51 smithi167 ceph-mon[32774]: 5.1f scrub ok 2024-02-05T06:54:51.724 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:51 smithi167 ceph-mon[32774]: pgmap v1019: 168 pgs: 168 active+clean; 185 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.04k op/s 2024-02-05T06:54:51.724 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:51 smithi167 ceph-mon[32774]: 1.9 deep-scrub starts 2024-02-05T06:54:51.724 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:51 smithi167 ceph-mon[32774]: 1.9 deep-scrub ok 2024-02-05T06:54:51.791 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:51.815 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:51.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:51.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:51.916 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:51.941 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:51.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:51.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:51.957 INFO:teuthology.orchestra.run.smithi167.stdout:ERROR: (22) Invalid argument 2024-02-05T06:54:51.958 INFO:teuthology.orchestra.run.smithi167.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-02-05T06:54:52.042 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:52.065 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:52.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:52.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:52.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:52.191 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:52.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:52.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:52.292 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:52.317 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:52.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:52.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:52.419 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:52.443 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:52.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:52.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:52.545 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:52.570 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:52.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:52.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:52.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:52.687 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-02-05T06:54:52.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:52.697 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:52.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:52.798 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:52.810 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:54:52 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:54:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:54:52.823 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:52.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:52.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:52.924 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:52.929 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:54:52.948 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:52.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:52.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:53.050 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:53.073 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:53.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:53.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:53.175 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:53.199 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:53.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:53.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:53.301 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:53.324 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:53.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:53.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:53.426 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:53.433 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:54:53.450 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:53.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:53.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:53.453 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:54:53.472 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:54:53.494 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:54:53.515 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:54:53.535 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:54:53.552 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:53.555 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:54:53.574 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:54:53.576 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:54:53.576 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:54:53.576 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:54:53.576 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:54:53.576 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:54:53.576 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:54:53.576 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:54:53.576 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:54:53.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:53.577 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:53.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:53.678 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:53.703 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:53.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:53.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:53.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:53 smithi028 ceph-mon[29442]: 11.8 scrub starts 2024-02-05T06:54:53.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:53 smithi028 ceph-mon[29442]: 11.8 scrub ok 2024-02-05T06:54:53.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:53 smithi028 ceph-mon[29442]: pgmap v1020: 168 pgs: 168 active+clean; 63 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 979 op/s 2024-02-05T06:54:53.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:53 smithi028 ceph-mon[29442]: 5.16 scrub starts 2024-02-05T06:54:53.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:53 smithi028 ceph-mon[29442]: 5.16 scrub ok 2024-02-05T06:54:53.715 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:53 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2692575300' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:54:53.715 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:53 smithi028 ceph-mon[37984]: 11.8 scrub starts 2024-02-05T06:54:53.715 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:53 smithi028 ceph-mon[37984]: 11.8 scrub ok 2024-02-05T06:54:53.716 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:53 smithi028 ceph-mon[37984]: pgmap v1020: 168 pgs: 168 active+clean; 63 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 979 op/s 2024-02-05T06:54:53.716 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:53 smithi028 ceph-mon[37984]: 5.16 scrub starts 2024-02-05T06:54:53.716 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:53 smithi028 ceph-mon[37984]: 5.16 scrub ok 2024-02-05T06:54:53.716 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:53 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2692575300' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:54:53.804 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:53.828 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:53.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:53.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:53.930 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:53.940 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:54:53.943 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-05T06:54:53.956 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:53.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:53.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:53.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:53 smithi167 ceph-mon[32774]: 11.8 scrub starts 2024-02-05T06:54:53.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:53 smithi167 ceph-mon[32774]: 11.8 scrub ok 2024-02-05T06:54:53.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:53 smithi167 ceph-mon[32774]: pgmap v1020: 168 pgs: 168 active+clean; 63 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 979 op/s 2024-02-05T06:54:53.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:53 smithi167 ceph-mon[32774]: 5.16 scrub starts 2024-02-05T06:54:53.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:53 smithi167 ceph-mon[32774]: 5.16 scrub ok 2024-02-05T06:54:53.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:53 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2692575300' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:54:53.983 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:54:53 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:54:54.058 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:54.082 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:54.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:54.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:54.183 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:54.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:54.208 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:54.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:54.308 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:54.332 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:54.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:54.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:54.434 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:54.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:54.458 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:54.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:54.559 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:54.559 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:54:54.582 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:54.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:54.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:54.683 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:54.694 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:54 smithi167 ceph-mon[32774]: 6.14 scrub starts 2024-02-05T06:54:54.694 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:54 smithi167 ceph-mon[32774]: 6.14 scrub ok 2024-02-05T06:54:54.707 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:54.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:54.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:54.807 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:54:54.808 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-02-05T06:54:54.808 INFO:teuthology.orchestra.run.smithi167.stdout: "num_ops": 0 2024-02-05T06:54:54.808 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:54:54.808 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:54.830 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-02-05T06:54:54.830 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-02-05T06:54:54.831 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:54.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:54.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:54.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:54 smithi028 ceph-mon[29442]: 6.14 scrub starts 2024-02-05T06:54:54.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:54 smithi028 ceph-mon[29442]: 6.14 scrub ok 2024-02-05T06:54:54.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:54 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:54:54.855 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:54 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:54:54.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:54 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:54:54.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:54 smithi028 ceph-mon[29442]: osdmap e479: 8 total, 8 up, 8 in 2024-02-05T06:54:54.856 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:54 smithi028 ceph-mon[29442]: from='client.16203 v1:172.21.15.28:0/954161356' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:54:54.856 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:54 smithi028 ceph-mon[37984]: 6.14 scrub starts 2024-02-05T06:54:54.856 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:54 smithi028 ceph-mon[37984]: 6.14 scrub ok 2024-02-05T06:54:54.856 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:54 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:54:54.856 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:54 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:54:54.856 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:54 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:54:54.856 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:54 smithi028 ceph-mon[37984]: osdmap e479: 8 total, 8 up, 8 in 2024-02-05T06:54:54.856 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:54 smithi028 ceph-mon[37984]: from='client.16203 v1:172.21.15.28:0/954161356' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:54:54.933 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:54.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:54.957 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:54.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:54.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:54 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:54:54.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:54 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:54:54.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:54 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:54:54.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:54 smithi167 ceph-mon[32774]: osdmap e479: 8 total, 8 up, 8 in 2024-02-05T06:54:54.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:54 smithi167 ceph-mon[32774]: from='client.16203 v1:172.21.15.28:0/954161356' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:54:55.057 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:55.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:55.082 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:55.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:55.183 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:55.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:55.207 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:55.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:55.308 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:55.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:55.333 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:55.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:55.434 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:55.457 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:55.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:55.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:55.559 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:55.583 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:55.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:55.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:55.684 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:55.708 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:55.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:55.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:55.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:55 smithi167 ceph-mon[32774]: pgmap v1022: 168 pgs: 168 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 720 B/s rd, 0 B/s wr, 783 op/s 2024-02-05T06:54:55.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:55 smithi167 ceph-mon[32774]: 4.a scrub starts 2024-02-05T06:54:55.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:55 smithi167 ceph-mon[32774]: 4.a scrub ok 2024-02-05T06:54:55.809 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:55.834 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:55.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:55.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:55.869 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":1021,"stamp":"2024-02-05T06:54:53.676170+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":2420,"num_read_kb":2172,"num_write":254936,"num_write_kb":8125678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":411,"num_bytes_recovered":276079,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79920,"ondisk_log_size":79920,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6624160,"kb_used_data":8028,"kb_used_omap":12,"kb_used_meta":6616051,"kb_avail":743337056,"statfs":{"total":767960285184,"available":761177145344,"internally_reserved":0,"allocated":8220672,"data_stored":5162316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":6774836785},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-556335128,"num_objects":-8490,"num_object_clones":0,"num_object_copies":-25470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8490,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8490,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.055843"},"pg_stats":[{"pgid":"11.3","version":"478'19891","reported_seq":20289,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:49.944645+0000","last_change":"2024-02-05T06:53:28.707185+0000","last_active":"2024-02-05T06:54:49.944645+0000","last_peered":"2024-02-05T06:54:49.944645+0000","last_clean":"2024-02-05T06:54:49.944645+0000","last_became_active":"2024-02-05T06:51:13.269833+0000","last_became_peered":"2024-02-05T06:51:13.269833+0000","last_unstale":"2024-02-05T06:54:49.944645+0000","last_undegraded":"2024-02-05T06:54:49.944645+0000","last_fullsized":"2024-02-05T06:54:49.944645+0000","mapping_epoch":418,"log_start":"465'14903","ondisk_log_start":"465'14903","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"462'14268","last_scrub_stamp":"2024-02-05T06:53:28.706913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:53:28.706913+0000","objects_scrubbed":5685,"log_size":4988,"log_dups_size":0,"ondisk_log_size":4988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1678,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15864,"num_write_kb":507648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.c","version":"465'35","reported_seq":771,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.547664+0000","last_change":"2024-02-05T06:52:45.303488+0000","last_active":"2024-02-05T06:54:48.547664+0000","last_peered":"2024-02-05T06:54:48.547664+0000","last_clean":"2024-02-05T06:54:48.547664+0000","last_became_active":"2024-02-05T06:45:09.796902+0000","last_became_peered":"2024-02-05T06:45:09.796902+0000","last_unstale":"2024-02-05T06:54:48.547664+0000","last_undegraded":"2024-02-05T06:54:48.547664+0000","last_fullsized":"2024-02-05T06:54:48.547664+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'25","last_scrub_stamp":"2024-02-05T06:52:45.303356+0000","last_deep_scrub":"93'15","last_deep_scrub_stamp":"2024-02-05T06:35:04.584657+0000","last_clean_scrub_stamp":"2024-02-05T06:52:45.303356+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":903,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.519522+0000","last_change":"2024-02-05T06:52:48.393899+0000","last_active":"2024-02-05T06:54:48.519522+0000","last_peered":"2024-02-05T06:54:48.519522+0000","last_clean":"2024-02-05T06:54:48.519522+0000","last_became_active":"2024-02-05T06:36:54.971304+0000","last_became_peered":"2024-02-05T06:36:54.971304+0000","last_unstale":"2024-02-05T06:54:48.519522+0000","last_undegraded":"2024-02-05T06:54:48.519522+0000","last_fullsized":"2024-02-05T06:54:48.519522+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:48.393628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:52:48.393628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"249'13","reported_seq":1227,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.549391+0000","last_change":"2024-02-05T06:53:56.010563+0000","last_active":"2024-02-05T06:54:48.549391+0000","last_peered":"2024-02-05T06:54:48.549391+0000","last_clean":"2024-02-05T06:54:48.549391+0000","last_became_active":"2024-02-05T06:41:43.413139+0000","last_became_peered":"2024-02-05T06:41:43.413139+0000","last_unstale":"2024-02-05T06:54:48.549391+0000","last_undegraded":"2024-02-05T06:54:48.549391+0000","last_fullsized":"2024-02-05T06:54:48.549391+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'13","last_scrub_stamp":"2024-02-05T06:53:56.009983+0000","last_deep_scrub":"92'9","last_deep_scrub_stamp":"2024-02-05T06:34:52.283780+0000","last_clean_scrub_stamp":"2024-02-05T06:53:56.009983+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:08.067138+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":438,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.882464+0000","last_change":"2024-02-05T06:54:48.882501+0000","last_active":"2024-02-05T06:54:48.882464+0000","last_peered":"2024-02-05T06:54:48.882464+0000","last_clean":"2024-02-05T06:54:48.882464+0000","last_became_active":"2024-02-05T06:45:09.784229+0000","last_became_peered":"2024-02-05T06:45:09.784229+0000","last_unstale":"2024-02-05T06:54:48.882464+0000","last_undegraded":"2024-02-05T06:54:48.882464+0000","last_fullsized":"2024-02-05T06:54:48.882464+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:54:48.882390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:30.057099+0000","last_clean_scrub_stamp":"2024-02-05T06:54:48.882390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:13.938741+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":1029,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:50.735061+0000","last_change":"2024-02-05T06:54:50.735085+0000","last_active":"2024-02-05T06:54:50.735061+0000","last_peered":"2024-02-05T06:54:50.735061+0000","last_clean":"2024-02-05T06:54:50.735061+0000","last_became_active":"2024-02-05T06:40:30.095202+0000","last_became_peered":"2024-02-05T06:40:30.095202+0000","last_unstale":"2024-02-05T06:54:50.735061+0000","last_undegraded":"2024-02-05T06:54:50.735061+0000","last_fullsized":"2024-02-05T06:54:50.735061+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:54:50.734974+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:54:50.734974+0000","last_clean_scrub_stamp":"2024-02-05T06:54:50.734974+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:02.927938+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.2","version":"478'16056","reported_seq":16292,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:49.927630+0000","last_change":"2024-02-05T06:54:40.794085+0000","last_active":"2024-02-05T06:54:49.927630+0000","last_peered":"2024-02-05T06:54:49.927630+0000","last_clean":"2024-02-05T06:54:49.927630+0000","last_became_active":"2024-02-05T06:51:13.275602+0000","last_became_peered":"2024-02-05T06:51:13.275602+0000","last_unstale":"2024-02-05T06:54:49.927630+0000","last_undegraded":"2024-02-05T06:54:49.927630+0000","last_fullsized":"2024-02-05T06:54:49.927630+0000","mapping_epoch":418,"log_start":"462'10200","ondisk_log_start":"462'10200","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"476'15404","last_scrub_stamp":"2024-02-05T06:54:40.793794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:54:40.793794+0000","objects_scrubbed":668,"log_size":5856,"log_dups_size":0,"ondisk_log_size":5856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:55.090224+0000","scrub_duration":683,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16056,"num_write_kb":513792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"465'53","reported_seq":805,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.549093+0000","last_change":"2024-02-05T06:54:39.136134+0000","last_active":"2024-02-05T06:54:48.549093+0000","last_peered":"2024-02-05T06:54:48.549093+0000","last_clean":"2024-02-05T06:54:48.549093+0000","last_became_active":"2024-02-05T06:45:09.782645+0000","last_became_peered":"2024-02-05T06:45:09.782645+0000","last_unstale":"2024-02-05T06:54:48.549093+0000","last_undegraded":"2024-02-05T06:54:48.549093+0000","last_fullsized":"2024-02-05T06:54:48.549093+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'53","last_scrub_stamp":"2024-02-05T06:54:39.135603+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-02-05T06:35:26.274944+0000","last_clean_scrub_stamp":"2024-02-05T06:54:39.135603+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:08.754175+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":932,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.535269+0000","last_change":"2024-02-05T06:53:45.072923+0000","last_active":"2024-02-05T06:54:48.535269+0000","last_peered":"2024-02-05T06:54:48.535269+0000","last_clean":"2024-02-05T06:54:48.535269+0000","last_became_active":"2024-02-05T06:39:13.240038+0000","last_became_peered":"2024-02-05T06:39:13.240038+0000","last_unstale":"2024-02-05T06:54:48.535269+0000","last_undegraded":"2024-02-05T06:54:48.535269+0000","last_fullsized":"2024-02-05T06:54:48.535269+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:45.072812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:53:45.072812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:02.736389+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":919,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.550446+0000","last_change":"2024-02-05T06:54:44.749384+0000","last_active":"2024-02-05T06:54:48.550446+0000","last_peered":"2024-02-05T06:54:48.550446+0000","last_clean":"2024-02-05T06:54:48.550446+0000","last_became_active":"2024-02-05T06:36:54.958082+0000","last_became_peered":"2024-02-05T06:36:54.958082+0000","last_unstale":"2024-02-05T06:54:48.550446+0000","last_undegraded":"2024-02-05T06:54:48.550446+0000","last_fullsized":"2024-02-05T06:54:48.550446+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:54:44.749016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:38.072995+0000","last_clean_scrub_stamp":"2024-02-05T06:54:44.749016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:06.501677+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":933,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.550010+0000","last_change":"2024-02-05T06:53:29.258337+0000","last_active":"2024-02-05T06:54:48.550010+0000","last_peered":"2024-02-05T06:54:48.550010+0000","last_clean":"2024-02-05T06:54:48.550010+0000","last_became_active":"2024-02-05T06:36:54.958805+0000","last_became_peered":"2024-02-05T06:36:54.958805+0000","last_unstale":"2024-02-05T06:54:48.550010+0000","last_undegraded":"2024-02-05T06:54:48.550010+0000","last_fullsized":"2024-02-05T06:54:48.550010+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:29.258153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:41.833719+0000","last_clean_scrub_stamp":"2024-02-05T06:53:29.258153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":273,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"83'128","reported_seq":942,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.525720+0000","last_change":"2024-02-05T06:53:55.916525+0000","last_active":"2024-02-05T06:54:48.525720+0000","last_peered":"2024-02-05T06:54:48.525720+0000","last_clean":"2024-02-05T06:54:48.525720+0000","last_became_active":"2024-02-05T06:41:46.053324+0000","last_became_peered":"2024-02-05T06:41:46.053324+0000","last_unstale":"2024-02-05T06:54:48.525720+0000","last_undegraded":"2024-02-05T06:54:48.525720+0000","last_fullsized":"2024-02-05T06:54:48.525720+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:53:55.916390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:53:55.916390+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:19.660693+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"83'128","reported_seq":1086,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.539066+0000","last_change":"2024-02-05T06:54:43.705342+0000","last_active":"2024-02-05T06:54:48.539066+0000","last_peered":"2024-02-05T06:54:48.539066+0000","last_clean":"2024-02-05T06:54:48.539066+0000","last_became_active":"2024-02-05T06:51:59.303575+0000","last_became_peered":"2024-02-05T06:51:59.303575+0000","last_unstale":"2024-02-05T06:54:48.539066+0000","last_undegraded":"2024-02-05T06:54:48.539066+0000","last_fullsized":"2024-02-05T06:54:48.539066+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:54:43.704150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:54:43.704150+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:03.925720+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"478'15790","reported_seq":16014,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:49.945325+0000","last_change":"2024-02-05T06:53:32.058096+0000","last_active":"2024-02-05T06:54:49.945325+0000","last_peered":"2024-02-05T06:54:49.945325+0000","last_clean":"2024-02-05T06:54:49.945325+0000","last_became_active":"2024-02-05T06:51:13.273200+0000","last_became_peered":"2024-02-05T06:51:13.273200+0000","last_unstale":"2024-02-05T06:54:49.945325+0000","last_undegraded":"2024-02-05T06:54:49.945325+0000","last_fullsized":"2024-02-05T06:54:49.945325+0000","mapping_epoch":418,"log_start":"465'10800","ondisk_log_start":"465'10800","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"463'10360","last_scrub_stamp":"2024-02-05T06:53:32.057849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:53:32.057849+0000","objects_scrubbed":5494,"log_size":4990,"log_dups_size":0,"ondisk_log_size":4990,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:01.054911+0000","scrub_duration":2042,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15790,"num_write_kb":505280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"465'32","reported_seq":781,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.535195+0000","last_change":"2024-02-05T06:53:31.987710+0000","last_active":"2024-02-05T06:54:48.535195+0000","last_peered":"2024-02-05T06:54:48.535195+0000","last_clean":"2024-02-05T06:54:48.535195+0000","last_became_active":"2024-02-05T06:45:09.774868+0000","last_became_peered":"2024-02-05T06:45:09.774868+0000","last_unstale":"2024-02-05T06:54:48.535195+0000","last_undegraded":"2024-02-05T06:54:48.535195+0000","last_fullsized":"2024-02-05T06:54:48.535195+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:53:31.987585+0000","last_deep_scrub":"115'16","last_deep_scrub_stamp":"2024-02-05T06:36:34.793243+0000","last_clean_scrub_stamp":"2024-02-05T06:53:31.987585+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":627,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.538774+0000","last_change":"2024-02-05T06:53:56.618075+0000","last_active":"2024-02-05T06:54:48.538774+0000","last_peered":"2024-02-05T06:54:48.538774+0000","last_clean":"2024-02-05T06:54:48.538774+0000","last_became_active":"2024-02-05T06:40:27.086808+0000","last_became_peered":"2024-02-05T06:40:27.086808+0000","last_unstale":"2024-02-05T06:54:48.538774+0000","last_undegraded":"2024-02-05T06:54:48.538774+0000","last_fullsized":"2024-02-05T06:54:48.538774+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:56.617959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:41.860956+0000","last_clean_scrub_stamp":"2024-02-05T06:53:56.617959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:03.244501+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":523,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.527099+0000","last_change":"2024-02-05T06:54:42.788077+0000","last_active":"2024-02-05T06:54:48.527099+0000","last_peered":"2024-02-05T06:54:48.527099+0000","last_clean":"2024-02-05T06:54:48.527099+0000","last_became_active":"2024-02-05T06:41:43.417343+0000","last_became_peered":"2024-02-05T06:41:43.417343+0000","last_unstale":"2024-02-05T06:54:48.527099+0000","last_undegraded":"2024-02-05T06:54:48.527099+0000","last_fullsized":"2024-02-05T06:54:48.527099+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:54:42.787745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:42:55.610276+0000","last_clean_scrub_stamp":"2024-02-05T06:54:42.787745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:59.389974+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":897,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.519560+0000","last_change":"2024-02-05T06:54:02.502678+0000","last_active":"2024-02-05T06:54:48.519560+0000","last_peered":"2024-02-05T06:54:48.519560+0000","last_clean":"2024-02-05T06:54:48.519560+0000","last_became_active":"2024-02-05T06:41:43.410600+0000","last_became_peered":"2024-02-05T06:41:43.410600+0000","last_unstale":"2024-02-05T06:54:48.519560+0000","last_undegraded":"2024-02-05T06:54:48.519560+0000","last_fullsized":"2024-02-05T06:54:48.519560+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:54:02.502543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:58.479208+0000","last_clean_scrub_stamp":"2024-02-05T06:54:02.502543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:20.782865+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.0","version":"478'19981","reported_seq":20389,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:49.949323+0000","last_change":"2024-02-05T06:53:34.387508+0000","last_active":"2024-02-05T06:54:49.949323+0000","last_peered":"2024-02-05T06:54:49.949323+0000","last_clean":"2024-02-05T06:54:49.949323+0000","last_became_active":"2024-02-05T06:51:13.273376+0000","last_became_peered":"2024-02-05T06:51:13.273376+0000","last_unstale":"2024-02-05T06:54:49.949323+0000","last_undegraded":"2024-02-05T06:54:49.949323+0000","last_fullsized":"2024-02-05T06:54:49.949323+0000","mapping_epoch":418,"log_start":"466'15603","ondisk_log_start":"466'15603","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'14766","last_scrub_stamp":"2024-02-05T06:53:34.387362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:53:34.387362+0000","objects_scrubbed":5240,"log_size":4378,"log_dups_size":0,"ondisk_log_size":4378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1825,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15752,"num_write_kb":504064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"465'46","reported_seq":724,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.519019+0000","last_change":"2024-02-05T06:52:38.220941+0000","last_active":"2024-02-05T06:54:48.519019+0000","last_peered":"2024-02-05T06:54:48.519019+0000","last_clean":"2024-02-05T06:54:48.519019+0000","last_became_active":"2024-02-05T06:39:13.257978+0000","last_became_peered":"2024-02-05T06:39:13.257978+0000","last_unstale":"2024-02-05T06:54:48.519019+0000","last_undegraded":"2024-02-05T06:54:48.519019+0000","last_fullsized":"2024-02-05T06:54:48.519019+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'34","last_scrub_stamp":"2024-02-05T06:52:38.220772+0000","last_deep_scrub":"295'34","last_deep_scrub_stamp":"2024-02-05T06:46:44.355563+0000","last_clean_scrub_stamp":"2024-02-05T06:52:38.220772+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":903,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.549120+0000","last_change":"2024-02-05T06:52:30.417398+0000","last_active":"2024-02-05T06:54:48.549120+0000","last_peered":"2024-02-05T06:54:48.549120+0000","last_clean":"2024-02-05T06:54:48.549120+0000","last_became_active":"2024-02-05T06:40:27.213439+0000","last_became_peered":"2024-02-05T06:40:27.213439+0000","last_unstale":"2024-02-05T06:54:48.549120+0000","last_undegraded":"2024-02-05T06:54:48.549120+0000","last_fullsized":"2024-02-05T06:54:48.549120+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:30.417237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:52.482566+0000","last_clean_scrub_stamp":"2024-02-05T06:52:30.417237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"319'13","reported_seq":831,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.526561+0000","last_change":"2024-02-05T06:52:49.830770+0000","last_active":"2024-02-05T06:54:48.526561+0000","last_peered":"2024-02-05T06:54:48.526561+0000","last_clean":"2024-02-05T06:54:48.526561+0000","last_became_active":"2024-02-05T06:45:09.766220+0000","last_became_peered":"2024-02-05T06:45:09.766220+0000","last_unstale":"2024-02-05T06:54:48.526561+0000","last_undegraded":"2024-02-05T06:54:48.526561+0000","last_fullsized":"2024-02-05T06:54:48.526561+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'13","last_scrub_stamp":"2024-02-05T06:52:49.830615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:52:49.830615+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":524,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.526341+0000","last_change":"2024-02-05T06:53:47.006909+0000","last_active":"2024-02-05T06:54:48.526341+0000","last_peered":"2024-02-05T06:54:48.526341+0000","last_clean":"2024-02-05T06:54:48.526341+0000","last_became_active":"2024-02-05T06:41:43.418066+0000","last_became_peered":"2024-02-05T06:41:43.418066+0000","last_unstale":"2024-02-05T06:54:48.526341+0000","last_undegraded":"2024-02-05T06:54:48.526341+0000","last_fullsized":"2024-02-05T06:54:48.526341+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:47.006807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:48:34.623889+0000","last_clean_scrub_stamp":"2024-02-05T06:53:47.006807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:54.416716+0000","scrub_duration":187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":938,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.526440+0000","last_change":"2024-02-05T06:53:44.829783+0000","last_active":"2024-02-05T06:54:48.526440+0000","last_peered":"2024-02-05T06:54:48.526440+0000","last_clean":"2024-02-05T06:54:48.526440+0000","last_became_active":"2024-02-05T06:41:46.053993+0000","last_became_peered":"2024-02-05T06:41:46.053993+0000","last_unstale":"2024-02-05T06:54:48.526440+0000","last_undegraded":"2024-02-05T06:54:48.526440+0000","last_fullsized":"2024-02-05T06:54:48.526440+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:53:44.829662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:53:44.829662+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:49.174258+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.f","version":"478'16328","reported_seq":16524,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:49.937686+0000","last_change":"2024-02-05T06:53:36.664242+0000","last_active":"2024-02-05T06:54:49.937686+0000","last_peered":"2024-02-05T06:54:49.937686+0000","last_clean":"2024-02-05T06:54:49.937686+0000","last_became_active":"2024-02-05T06:51:13.274262+0000","last_became_peered":"2024-02-05T06:51:13.274262+0000","last_unstale":"2024-02-05T06:54:49.937686+0000","last_undegraded":"2024-02-05T06:54:49.937686+0000","last_fullsized":"2024-02-05T06:54:49.937686+0000","mapping_epoch":418,"log_start":"466'11700","ondisk_log_start":"466'11700","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'10930","last_scrub_stamp":"2024-02-05T06:53:36.664069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:53:36.664069+0000","objects_scrubbed":5464,"log_size":4628,"log_dups_size":0,"ondisk_log_size":4628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1595,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16328,"num_write_kb":522496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":563,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.552874+0000","last_change":"2024-02-05T06:53:50.578263+0000","last_active":"2024-02-05T06:54:48.552874+0000","last_peered":"2024-02-05T06:54:48.552874+0000","last_clean":"2024-02-05T06:54:48.552874+0000","last_became_active":"2024-02-05T06:45:09.778609+0000","last_became_peered":"2024-02-05T06:45:09.778609+0000","last_unstale":"2024-02-05T06:54:48.552874+0000","last_undegraded":"2024-02-05T06:54:48.552874+0000","last_fullsized":"2024-02-05T06:54:48.552874+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:50.578152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:53:50.578152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:55.388149+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"465'50","reported_seq":1004,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.549210+0000","last_change":"2024-02-05T06:51:40.451314+0000","last_active":"2024-02-05T06:54:48.549210+0000","last_peered":"2024-02-05T06:54:48.549210+0000","last_clean":"2024-02-05T06:54:48.549210+0000","last_became_active":"2024-02-05T06:40:27.277747+0000","last_became_peered":"2024-02-05T06:40:27.277747+0000","last_unstale":"2024-02-05T06:54:48.549210+0000","last_undegraded":"2024-02-05T06:54:48.549210+0000","last_fullsized":"2024-02-05T06:54:48.549210+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"424'38","last_scrub_stamp":"2024-02-05T06:51:40.451087+0000","last_deep_scrub":"424'38","last_deep_scrub_stamp":"2024-02-05T06:51:40.451087+0000","last_clean_scrub_stamp":"2024-02-05T06:51:40.451087+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":909,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.548181+0000","last_change":"2024-02-05T06:52:53.475658+0000","last_active":"2024-02-05T06:54:48.548181+0000","last_peered":"2024-02-05T06:54:48.548181+0000","last_clean":"2024-02-05T06:54:48.548181+0000","last_became_active":"2024-02-05T06:40:27.093739+0000","last_became_peered":"2024-02-05T06:40:27.093739+0000","last_unstale":"2024-02-05T06:54:48.548181+0000","last_undegraded":"2024-02-05T06:54:48.548181+0000","last_fullsized":"2024-02-05T06:54:48.548181+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:53.475540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:53.475540+0000","last_clean_scrub_stamp":"2024-02-05T06:52:53.475540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":422,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.552679+0000","last_change":"2024-02-05T06:53:28.389799+0000","last_active":"2024-02-05T06:54:48.552679+0000","last_peered":"2024-02-05T06:54:48.552679+0000","last_clean":"2024-02-05T06:54:48.552679+0000","last_became_active":"2024-02-05T06:45:09.780390+0000","last_became_peered":"2024-02-05T06:45:09.780390+0000","last_unstale":"2024-02-05T06:54:48.552679+0000","last_undegraded":"2024-02-05T06:54:48.552679+0000","last_fullsized":"2024-02-05T06:54:48.552679+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:28.389689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:45:49.145117+0000","last_clean_scrub_stamp":"2024-02-05T06:53:28.389689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:51.789095+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":681,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.529468+0000","last_change":"2024-02-05T06:53:40.036388+0000","last_active":"2024-02-05T06:54:48.529468+0000","last_peered":"2024-02-05T06:54:48.529468+0000","last_clean":"2024-02-05T06:54:48.529468+0000","last_became_active":"2024-02-05T06:51:54.322200+0000","last_became_peered":"2024-02-05T06:51:54.322200+0000","last_unstale":"2024-02-05T06:54:48.529468+0000","last_undegraded":"2024-02-05T06:54:48.529468+0000","last_fullsized":"2024-02-05T06:54:48.529468+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:53:40.036268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:53:40.036268+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:59.840080+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"478'16038","reported_seq":16270,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:49.922061+0000","last_change":"2024-02-05T06:53:36.641829+0000","last_active":"2024-02-05T06:54:49.922061+0000","last_peered":"2024-02-05T06:54:49.922061+0000","last_clean":"2024-02-05T06:54:49.922061+0000","last_became_active":"2024-02-05T06:51:13.267644+0000","last_became_peered":"2024-02-05T06:51:13.267644+0000","last_unstale":"2024-02-05T06:54:49.922061+0000","last_undegraded":"2024-02-05T06:54:49.922061+0000","last_fullsized":"2024-02-05T06:54:49.922061+0000","mapping_epoch":418,"log_start":"466'11600","ondisk_log_start":"466'11600","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'10815","last_scrub_stamp":"2024-02-05T06:53:36.641665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:53:36.641665+0000","objects_scrubbed":5282,"log_size":4438,"log_dups_size":0,"ondisk_log_size":4438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1905,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16038,"num_write_kb":513216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"326'11","reported_seq":1172,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.549418+0000","last_change":"2024-02-05T06:53:59.536292+0000","last_active":"2024-02-05T06:54:48.549418+0000","last_peered":"2024-02-05T06:54:48.549418+0000","last_clean":"2024-02-05T06:54:48.549418+0000","last_became_active":"2024-02-05T06:36:54.954819+0000","last_became_peered":"2024-02-05T06:36:54.954819+0000","last_unstale":"2024-02-05T06:54:48.549418+0000","last_undegraded":"2024-02-05T06:54:48.549418+0000","last_fullsized":"2024-02-05T06:54:48.549418+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'11","last_scrub_stamp":"2024-02-05T06:53:59.535905+0000","last_deep_scrub":"326'11","last_deep_scrub_stamp":"2024-02-05T06:45:56.212177+0000","last_clean_scrub_stamp":"2024-02-05T06:53:59.535905+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:19.671764+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"465'42","reported_seq":847,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.525563+0000","last_change":"2024-02-05T06:54:02.913296+0000","last_active":"2024-02-05T06:54:48.525563+0000","last_peered":"2024-02-05T06:54:48.525563+0000","last_clean":"2024-02-05T06:54:48.525563+0000","last_became_active":"2024-02-05T06:45:09.764696+0000","last_became_peered":"2024-02-05T06:45:09.764696+0000","last_unstale":"2024-02-05T06:54:48.525563+0000","last_undegraded":"2024-02-05T06:54:48.525563+0000","last_fullsized":"2024-02-05T06:54:48.525563+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'42","last_scrub_stamp":"2024-02-05T06:54:02.913160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:54:02.913160+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:26.597570+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"95'2","reported_seq":922,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.519351+0000","last_change":"2024-02-05T06:53:41.525189+0000","last_active":"2024-02-05T06:54:48.519351+0000","last_peered":"2024-02-05T06:54:48.519351+0000","last_clean":"2024-02-05T06:54:48.519351+0000","last_became_active":"2024-02-05T06:36:54.975093+0000","last_became_peered":"2024-02-05T06:36:54.975093+0000","last_unstale":"2024-02-05T06:54:48.519351+0000","last_undegraded":"2024-02-05T06:54:48.519351+0000","last_fullsized":"2024-02-05T06:54:48.519351+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-05T06:53:41.524836+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-05T06:35:25.616708+0000","last_clean_scrub_stamp":"2024-02-05T06:53:41.524836+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:59.895714+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":608,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.532432+0000","last_change":"2024-02-05T06:53:53.881450+0000","last_active":"2024-02-05T06:54:48.532432+0000","last_peered":"2024-02-05T06:54:48.532432+0000","last_clean":"2024-02-05T06:54:48.532432+0000","last_became_active":"2024-02-05T06:40:27.099168+0000","last_became_peered":"2024-02-05T06:40:27.099168+0000","last_unstale":"2024-02-05T06:54:48.532432+0000","last_undegraded":"2024-02-05T06:54:48.532432+0000","last_fullsized":"2024-02-05T06:54:48.532432+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:53.881337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:53:53.881337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:10.513794+0000","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":1106,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.519393+0000","last_change":"2024-02-05T06:52:18.376581+0000","last_active":"2024-02-05T06:54:48.519393+0000","last_peered":"2024-02-05T06:54:48.519393+0000","last_clean":"2024-02-05T06:54:48.519393+0000","last_became_active":"2024-02-05T06:51:54.332037+0000","last_became_peered":"2024-02-05T06:51:54.332037+0000","last_unstale":"2024-02-05T06:54:48.519393+0000","last_undegraded":"2024-02-05T06:54:48.519393+0000","last_fullsized":"2024-02-05T06:54:48.519393+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:52:18.376463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:52:18.376463+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.d","version":"478'15800","reported_seq":16042,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:49.951850+0000","last_change":"2024-02-05T06:53:30.282702+0000","last_active":"2024-02-05T06:54:49.951850+0000","last_peered":"2024-02-05T06:54:49.951850+0000","last_clean":"2024-02-05T06:54:49.951850+0000","last_became_active":"2024-02-05T06:51:13.272662+0000","last_became_peered":"2024-02-05T06:51:13.272662+0000","last_unstale":"2024-02-05T06:54:49.951850+0000","last_undegraded":"2024-02-05T06:54:49.951850+0000","last_fullsized":"2024-02-05T06:54:49.951850+0000","mapping_epoch":418,"log_start":"462'10000","ondisk_log_start":"462'10000","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"463'10228","last_scrub_stamp":"2024-02-05T06:53:30.282379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:53:30.282379+0000","objects_scrubbed":5646,"log_size":5800,"log_dups_size":0,"ondisk_log_size":5800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2319,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15800,"num_write_kb":505537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"335'11","reported_seq":1179,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.536575+0000","last_change":"2024-02-05T06:52:40.188002+0000","last_active":"2024-02-05T06:54:48.536575+0000","last_peered":"2024-02-05T06:54:48.536575+0000","last_clean":"2024-02-05T06:54:48.536575+0000","last_became_active":"2024-02-05T06:41:43.423638+0000","last_became_peered":"2024-02-05T06:41:43.423638+0000","last_unstale":"2024-02-05T06:54:48.536575+0000","last_undegraded":"2024-02-05T06:54:48.536575+0000","last_fullsized":"2024-02-05T06:54:48.536575+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"335'11","last_scrub_stamp":"2024-02-05T06:52:40.187864+0000","last_deep_scrub":"335'11","last_deep_scrub_stamp":"2024-02-05T06:52:40.187864+0000","last_clean_scrub_stamp":"2024-02-05T06:52:40.187864+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"465'35","reported_seq":1026,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.547987+0000","last_change":"2024-02-05T06:54:00.633400+0000","last_active":"2024-02-05T06:54:48.547987+0000","last_peered":"2024-02-05T06:54:48.547987+0000","last_clean":"2024-02-05T06:54:48.547987+0000","last_became_active":"2024-02-05T06:41:43.421030+0000","last_became_peered":"2024-02-05T06:41:43.421030+0000","last_unstale":"2024-02-05T06:54:48.547987+0000","last_undegraded":"2024-02-05T06:54:48.547987+0000","last_fullsized":"2024-02-05T06:54:48.547987+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'35","last_scrub_stamp":"2024-02-05T06:54:00.633175+0000","last_deep_scrub":"424'25","last_deep_scrub_stamp":"2024-02-05T06:51:39.813235+0000","last_clean_scrub_stamp":"2024-02-05T06:54:00.633175+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:18.151735+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":516,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.526419+0000","last_change":"2024-02-05T06:53:32.828121+0000","last_active":"2024-02-05T06:54:48.526419+0000","last_peered":"2024-02-05T06:54:48.526419+0000","last_clean":"2024-02-05T06:54:48.526419+0000","last_became_active":"2024-02-05T06:41:43.411732+0000","last_became_peered":"2024-02-05T06:41:43.411732+0000","last_unstale":"2024-02-05T06:54:48.526419+0000","last_undegraded":"2024-02-05T06:54:48.526419+0000","last_fullsized":"2024-02-05T06:54:48.526419+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:32.828031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:28.353956+0000","last_clean_scrub_stamp":"2024-02-05T06:53:32.828031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":889,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.536225+0000","last_change":"2024-02-05T06:53:52.550097+0000","last_active":"2024-02-05T06:54:48.536225+0000","last_peered":"2024-02-05T06:54:48.536225+0000","last_clean":"2024-02-05T06:54:48.536225+0000","last_became_active":"2024-02-05T06:33:35.600485+0000","last_became_peered":"2024-02-05T06:33:35.600485+0000","last_unstale":"2024-02-05T06:54:48.536225+0000","last_undegraded":"2024-02-05T06:54:48.536225+0000","last_fullsized":"2024-02-05T06:54:48.536225+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:52.549976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:53:52.549976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:17.433853+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":1118,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.535569+0000","last_change":"2024-02-05T06:54:45.307556+0000","last_active":"2024-02-05T06:54:48.535569+0000","last_peered":"2024-02-05T06:54:48.535569+0000","last_clean":"2024-02-05T06:54:48.535569+0000","last_became_active":"2024-02-05T06:51:54.332371+0000","last_became_peered":"2024-02-05T06:51:54.332371+0000","last_unstale":"2024-02-05T06:54:48.535569+0000","last_undegraded":"2024-02-05T06:54:48.535569+0000","last_fullsized":"2024-02-05T06:54:48.535569+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:54:45.307473+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:48:25.725892+0000","last_clean_scrub_stamp":"2024-02-05T06:54:45.307473+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:13.569251+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.c","version":"478'15472","reported_seq":15688,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:49.940693+0000","last_change":"2024-02-05T06:52:53.084302+0000","last_active":"2024-02-05T06:54:49.940693+0000","last_peered":"2024-02-05T06:54:49.940693+0000","last_clean":"2024-02-05T06:54:49.940693+0000","last_became_active":"2024-02-05T06:51:13.269283+0000","last_became_peered":"2024-02-05T06:51:13.269283+0000","last_unstale":"2024-02-05T06:54:49.940693+0000","last_undegraded":"2024-02-05T06:54:49.940693+0000","last_fullsized":"2024-02-05T06:54:49.940693+0000","mapping_epoch":418,"log_start":"466'11100","ondisk_log_start":"466'11100","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'7755","last_scrub_stamp":"2024-02-05T06:52:53.084023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:52:53.084023+0000","objects_scrubbed":7733,"log_size":4372,"log_dups_size":0,"ondisk_log_size":4372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1955,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15472,"num_write_kb":495104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":927,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.535180+0000","last_change":"2024-02-05T06:54:40.272482+0000","last_active":"2024-02-05T06:54:48.535180+0000","last_peered":"2024-02-05T06:54:48.535180+0000","last_clean":"2024-02-05T06:54:48.535180+0000","last_became_active":"2024-02-05T06:41:43.411550+0000","last_became_peered":"2024-02-05T06:41:43.411550+0000","last_unstale":"2024-02-05T06:54:48.535180+0000","last_undegraded":"2024-02-05T06:54:48.535180+0000","last_fullsized":"2024-02-05T06:54:48.535180+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:54:40.272407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:54:40.272407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:52.724960+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"465'39","reported_seq":1059,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.536848+0000","last_change":"2024-02-05T06:52:50.055893+0000","last_active":"2024-02-05T06:54:48.536848+0000","last_peered":"2024-02-05T06:54:48.536848+0000","last_clean":"2024-02-05T06:54:48.536848+0000","last_became_active":"2024-02-05T06:41:43.409664+0000","last_became_peered":"2024-02-05T06:41:43.409664+0000","last_unstale":"2024-02-05T06:54:48.536848+0000","last_undegraded":"2024-02-05T06:54:48.536848+0000","last_fullsized":"2024-02-05T06:54:48.536848+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'29","last_scrub_stamp":"2024-02-05T06:52:50.055600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:52:50.055600+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"445'2","reported_seq":935,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.550363+0000","last_change":"2024-02-05T06:52:37.034737+0000","last_active":"2024-02-05T06:54:48.550363+0000","last_peered":"2024-02-05T06:54:48.550363+0000","last_clean":"2024-02-05T06:54:48.550363+0000","last_became_active":"2024-02-05T06:41:43.411404+0000","last_became_peered":"2024-02-05T06:41:43.411404+0000","last_unstale":"2024-02-05T06:54:48.550363+0000","last_undegraded":"2024-02-05T06:54:48.550363+0000","last_fullsized":"2024-02-05T06:54:48.550363+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'2","last_scrub_stamp":"2024-02-05T06:52:37.034607+0000","last_deep_scrub":"445'2","last_deep_scrub_stamp":"2024-02-05T06:52:37.034607+0000","last_clean_scrub_stamp":"2024-02-05T06:52:37.034607+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":896,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.538975+0000","last_change":"2024-02-05T06:53:38.739502+0000","last_active":"2024-02-05T06:54:48.538975+0000","last_peered":"2024-02-05T06:54:48.538975+0000","last_clean":"2024-02-05T06:54:48.538975+0000","last_became_active":"2024-02-05T06:33:35.664858+0000","last_became_peered":"2024-02-05T06:33:35.664858+0000","last_unstale":"2024-02-05T06:54:48.538975+0000","last_undegraded":"2024-02-05T06:54:48.538975+0000","last_fullsized":"2024-02-05T06:54:48.538975+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:38.739377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:53:38.739377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:50.477820+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":1099,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.535687+0000","last_change":"2024-02-05T06:54:46.317478+0000","last_active":"2024-02-05T06:54:48.535687+0000","last_peered":"2024-02-05T06:54:48.535687+0000","last_clean":"2024-02-05T06:54:48.535687+0000","last_became_active":"2024-02-05T06:51:54.330806+0000","last_became_peered":"2024-02-05T06:51:54.330806+0000","last_unstale":"2024-02-05T06:54:48.535687+0000","last_undegraded":"2024-02-05T06:54:48.535687+0000","last_fullsized":"2024-02-05T06:54:48.535687+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:54:46.317397+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:38.778568+0000","last_clean_scrub_stamp":"2024-02-05T06:54:46.317397+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:00.114963+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.9","version":"478'15940","reported_seq":16071,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:49.948679+0000","last_change":"2024-02-05T06:51:13.271458+0000","last_active":"2024-02-05T06:54:49.948679+0000","last_peered":"2024-02-05T06:54:49.948679+0000","last_clean":"2024-02-05T06:54:49.948679+0000","last_became_active":"2024-02-05T06:51:13.270994+0000","last_became_peered":"2024-02-05T06:51:13.270994+0000","last_unstale":"2024-02-05T06:54:49.948679+0000","last_undegraded":"2024-02-05T06:54:49.948679+0000","last_fullsized":"2024-02-05T06:54:49.948679+0000","mapping_epoch":418,"log_start":"467'11700","ondisk_log_start":"467'11700","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:51:12.249449+0000","objects_scrubbed":0,"log_size":4240,"log_dups_size":0,"ondisk_log_size":4240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15940,"num_write_kb":510080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":516,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.527170+0000","last_change":"2024-02-05T06:52:50.894323+0000","last_active":"2024-02-05T06:54:48.527170+0000","last_peered":"2024-02-05T06:54:48.527170+0000","last_clean":"2024-02-05T06:54:48.527170+0000","last_became_active":"2024-02-05T06:41:43.417751+0000","last_became_peered":"2024-02-05T06:41:43.417751+0000","last_unstale":"2024-02-05T06:54:48.527170+0000","last_undegraded":"2024-02-05T06:54:48.527170+0000","last_fullsized":"2024-02-05T06:54:48.527170+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:50.894195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:52:50.894195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"465'36","reported_seq":950,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.519293+0000","last_change":"2024-02-05T06:52:52.080789+0000","last_active":"2024-02-05T06:54:48.519293+0000","last_peered":"2024-02-05T06:54:48.519293+0000","last_clean":"2024-02-05T06:54:48.519293+0000","last_became_active":"2024-02-05T06:33:31.564604+0000","last_became_peered":"2024-02-05T06:33:31.564604+0000","last_unstale":"2024-02-05T06:54:48.519293+0000","last_undegraded":"2024-02-05T06:54:48.519293+0000","last_fullsized":"2024-02-05T06:54:48.519293+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:52:52.080633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:52:52.080633+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"355'6","reported_seq":1481,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:50.550706+0000","last_change":"2024-02-05T06:54:45.918935+0000","last_active":"2024-02-05T06:54:50.550706+0000","last_peered":"2024-02-05T06:54:50.550706+0000","last_clean":"2024-02-05T06:54:50.550706+0000","last_became_active":"2024-02-05T06:40:27.071509+0000","last_became_peered":"2024-02-05T06:40:27.071509+0000","last_unstale":"2024-02-05T06:54:50.550706+0000","last_undegraded":"2024-02-05T06:54:50.550706+0000","last_fullsized":"2024-02-05T06:54:50.550706+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'6","last_scrub_stamp":"2024-02-05T06:54:45.918813+0000","last_deep_scrub":"355'6","last_deep_scrub_stamp":"2024-02-05T06:47:46.642454+0000","last_clean_scrub_stamp":"2024-02-05T06:54:45.918813+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:49.511329+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00074151600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":870,"num_read_kb":865,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":913,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.536650+0000","last_change":"2024-02-05T06:53:33.469434+0000","last_active":"2024-02-05T06:54:48.536650+0000","last_peered":"2024-02-05T06:54:48.536650+0000","last_clean":"2024-02-05T06:54:48.536650+0000","last_became_active":"2024-02-05T06:36:54.972264+0000","last_became_peered":"2024-02-05T06:36:54.972264+0000","last_unstale":"2024-02-05T06:54:48.536650+0000","last_undegraded":"2024-02-05T06:54:48.536650+0000","last_fullsized":"2024-02-05T06:54:48.536650+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:33.469051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:53:33.469051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":916,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.539179+0000","last_change":"2024-02-05T06:53:47.688370+0000","last_active":"2024-02-05T06:54:48.539179+0000","last_peered":"2024-02-05T06:54:48.539179+0000","last_clean":"2024-02-05T06:54:48.539179+0000","last_became_active":"2024-02-05T06:41:43.410910+0000","last_became_peered":"2024-02-05T06:41:43.410910+0000","last_unstale":"2024-02-05T06:54:48.539179+0000","last_undegraded":"2024-02-05T06:54:48.539179+0000","last_fullsized":"2024-02-05T06:54:48.539179+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:47.688112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:53:47.688112+0000","last_clean_scrub_stamp":"2024-02-05T06:53:47.688112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:00.832816+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":1079,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.549192+0000","last_change":"2024-02-05T06:54:46.836244+0000","last_active":"2024-02-05T06:54:48.549192+0000","last_peered":"2024-02-05T06:54:48.549192+0000","last_clean":"2024-02-05T06:54:48.549192+0000","last_became_active":"2024-02-05T06:51:54.323320+0000","last_became_peered":"2024-02-05T06:51:54.323320+0000","last_unstale":"2024-02-05T06:54:48.549192+0000","last_undegraded":"2024-02-05T06:54:48.549192+0000","last_fullsized":"2024-02-05T06:54:48.549192+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:54:46.835972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:54:46.835972+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:01.073894+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"478'16002","reported_seq":16149,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:49.953130+0000","last_change":"2024-02-05T06:54:40.967225+0000","last_active":"2024-02-05T06:54:49.953130+0000","last_peered":"2024-02-05T06:54:49.953130+0000","last_clean":"2024-02-05T06:54:49.953130+0000","last_became_active":"2024-02-05T06:51:13.274381+0000","last_became_peered":"2024-02-05T06:51:13.274381+0000","last_unstale":"2024-02-05T06:54:49.953130+0000","last_undegraded":"2024-02-05T06:54:49.953130+0000","last_fullsized":"2024-02-05T06:54:49.953130+0000","mapping_epoch":418,"log_start":"467'11700","ondisk_log_start":"467'11700","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"476'15303","last_scrub_stamp":"2024-02-05T06:54:40.966761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:54:40.966761+0000","objects_scrubbed":719,"log_size":4302,"log_dups_size":0,"ondisk_log_size":4302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:57.781837+0000","scrub_duration":389,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16002,"num_write_kb":512064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":622,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.532952+0000","last_change":"2024-02-05T06:54:02.593564+0000","last_active":"2024-02-05T06:54:48.532952+0000","last_peered":"2024-02-05T06:54:48.532952+0000","last_clean":"2024-02-05T06:54:48.532952+0000","last_became_active":"2024-02-05T06:41:43.418198+0000","last_became_peered":"2024-02-05T06:41:43.418198+0000","last_unstale":"2024-02-05T06:54:48.532952+0000","last_undegraded":"2024-02-05T06:54:48.532952+0000","last_fullsized":"2024-02-05T06:54:48.532952+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:54:02.593328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:54:02.593328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:12.160360+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"465'52","reported_seq":1340,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.535575+0000","last_change":"2024-02-05T06:54:48.333397+0000","last_active":"2024-02-05T06:54:48.535575+0000","last_peered":"2024-02-05T06:54:48.535575+0000","last_clean":"2024-02-05T06:54:48.535575+0000","last_became_active":"2024-02-05T06:40:27.061111+0000","last_became_peered":"2024-02-05T06:40:27.061111+0000","last_unstale":"2024-02-05T06:54:48.535575+0000","last_undegraded":"2024-02-05T06:54:48.535575+0000","last_fullsized":"2024-02-05T06:54:48.535575+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'52","last_scrub_stamp":"2024-02-05T06:54:48.333272+0000","last_deep_scrub":"413'40","last_deep_scrub_stamp":"2024-02-05T06:50:56.446985+0000","last_clean_scrub_stamp":"2024-02-05T06:54:48.333272+0000","objects_scrubbed":8,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:13.924804+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":904,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.550099+0000","last_change":"2024-02-05T06:53:38.592980+0000","last_active":"2024-02-05T06:54:48.550099+0000","last_peered":"2024-02-05T06:54:48.550099+0000","last_clean":"2024-02-05T06:54:48.550099+0000","last_became_active":"2024-02-05T06:41:43.411335+0000","last_became_peered":"2024-02-05T06:41:43.411335+0000","last_unstale":"2024-02-05T06:54:48.550099+0000","last_undegraded":"2024-02-05T06:54:48.550099+0000","last_fullsized":"2024-02-05T06:54:48.550099+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:38.592868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:53:38.592868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:54.703288+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1046,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.532179+0000","last_change":"2024-02-05T06:54:03.548187+0000","last_active":"2024-02-05T06:54:48.532179+0000","last_peered":"2024-02-05T06:54:48.532179+0000","last_clean":"2024-02-05T06:54:48.532179+0000","last_became_active":"2024-02-05T06:51:54.326585+0000","last_became_peered":"2024-02-05T06:51:54.326585+0000","last_unstale":"2024-02-05T06:54:48.532179+0000","last_undegraded":"2024-02-05T06:54:48.532179+0000","last_fullsized":"2024-02-05T06:54:48.532179+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:54:03.548068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:54:03.548068+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:18.630524+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":524,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.527136+0000","last_change":"2024-02-05T06:54:40.980461+0000","last_active":"2024-02-05T06:54:48.527136+0000","last_peered":"2024-02-05T06:54:48.527136+0000","last_clean":"2024-02-05T06:54:48.527136+0000","last_became_active":"2024-02-05T06:41:43.408773+0000","last_became_peered":"2024-02-05T06:41:43.408773+0000","last_unstale":"2024-02-05T06:54:48.527136+0000","last_undegraded":"2024-02-05T06:54:48.527136+0000","last_fullsized":"2024-02-05T06:54:48.527136+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:54:40.980349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:54:40.980349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:42.429623+0000","scrub_duration":186,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.8","version":"478'15850","reported_seq":15989,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:52.485375+0000","last_change":"2024-02-05T06:54:52.485422+0000","last_active":"2024-02-05T06:54:52.485375+0000","last_peered":"2024-02-05T06:54:52.485375+0000","last_clean":"2024-02-05T06:54:52.485375+0000","last_became_active":"2024-02-05T06:51:13.272948+0000","last_became_peered":"2024-02-05T06:51:13.272948+0000","last_unstale":"2024-02-05T06:54:52.485375+0000","last_undegraded":"2024-02-05T06:54:52.485375+0000","last_fullsized":"2024-02-05T06:54:52.485375+0000","mapping_epoch":418,"log_start":"466'11600","ondisk_log_start":"466'11600","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"478'15850","last_scrub_stamp":"2024-02-05T06:54:52.485284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:54:52.485284+0000","objects_scrubbed":0,"log_size":4250,"log_dups_size":0,"ondisk_log_size":4250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:01.719295+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15850,"num_write_kb":507200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":953,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.548892+0000","last_change":"2024-02-05T06:53:54.096200+0000","last_active":"2024-02-05T06:54:48.548892+0000","last_peered":"2024-02-05T06:54:48.548892+0000","last_clean":"2024-02-05T06:54:48.548892+0000","last_became_active":"2024-02-05T06:41:43.412387+0000","last_became_peered":"2024-02-05T06:41:43.412387+0000","last_unstale":"2024-02-05T06:54:48.548892+0000","last_undegraded":"2024-02-05T06:54:48.548892+0000","last_fullsized":"2024-02-05T06:54:48.548892+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:54.095702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:34.871805+0000","last_clean_scrub_stamp":"2024-02-05T06:53:54.095702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:14.492332+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"465'21","reported_seq":1004,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.548943+0000","last_change":"2024-02-05T06:52:44.087732+0000","last_active":"2024-02-05T06:54:48.548943+0000","last_peered":"2024-02-05T06:54:48.548943+0000","last_clean":"2024-02-05T06:54:48.548943+0000","last_became_active":"2024-02-05T06:41:43.420458+0000","last_became_peered":"2024-02-05T06:41:43.420458+0000","last_unstale":"2024-02-05T06:54:48.548943+0000","last_undegraded":"2024-02-05T06:54:48.548943+0000","last_fullsized":"2024-02-05T06:54:48.548943+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'17","last_scrub_stamp":"2024-02-05T06:52:44.087598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:52:44.087598+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":939,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.528982+0000","last_change":"2024-02-05T06:53:44.055170+0000","last_active":"2024-02-05T06:54:48.528982+0000","last_peered":"2024-02-05T06:54:48.528982+0000","last_clean":"2024-02-05T06:54:48.528982+0000","last_became_active":"2024-02-05T06:33:28.542977+0000","last_became_peered":"2024-02-05T06:33:28.542977+0000","last_unstale":"2024-02-05T06:54:48.528982+0000","last_undegraded":"2024-02-05T06:54:48.528982+0000","last_fullsized":"2024-02-05T06:54:48.528982+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:44.054895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:53:44.054895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:11.925676+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.00091640799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":908,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.538851+0000","last_change":"2024-02-05T06:53:39.709665+0000","last_active":"2024-02-05T06:54:48.538851+0000","last_peered":"2024-02-05T06:54:48.538851+0000","last_clean":"2024-02-05T06:54:48.538851+0000","last_became_active":"2024-02-05T06:33:29.551549+0000","last_became_peered":"2024-02-05T06:33:29.551549+0000","last_unstale":"2024-02-05T06:54:48.538851+0000","last_undegraded":"2024-02-05T06:54:48.538851+0000","last_fullsized":"2024-02-05T06:54:48.538851+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:39.709556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:53:39.709556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:06.007029+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":608,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.532131+0000","last_change":"2024-02-05T06:53:41.146247+0000","last_active":"2024-02-05T06:54:48.532131+0000","last_peered":"2024-02-05T06:54:48.532131+0000","last_clean":"2024-02-05T06:54:48.532131+0000","last_became_active":"2024-02-05T06:40:27.077966+0000","last_became_peered":"2024-02-05T06:40:27.077966+0000","last_unstale":"2024-02-05T06:54:48.532131+0000","last_undegraded":"2024-02-05T06:54:48.532131+0000","last_fullsized":"2024-02-05T06:54:48.532131+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:41.146141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:45.774441+0000","last_clean_scrub_stamp":"2024-02-05T06:53:41.146141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:08.495898+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":1045,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.532591+0000","last_change":"2024-02-05T06:53:27.227987+0000","last_active":"2024-02-05T06:54:48.532591+0000","last_peered":"2024-02-05T06:54:48.532591+0000","last_clean":"2024-02-05T06:54:48.532591+0000","last_became_active":"2024-02-05T06:51:54.325427+0000","last_became_peered":"2024-02-05T06:51:54.325427+0000","last_unstale":"2024-02-05T06:54:48.532591+0000","last_undegraded":"2024-02-05T06:54:48.532591+0000","last_fullsized":"2024-02-05T06:54:48.532591+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:53:27.227870+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:48:28.657679+0000","last_clean_scrub_stamp":"2024-02-05T06:53:27.227870+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"478'15816","reported_seq":15964,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:49.948677+0000","last_change":"2024-02-05T06:54:38.837824+0000","last_active":"2024-02-05T06:54:49.948677+0000","last_peered":"2024-02-05T06:54:49.948677+0000","last_clean":"2024-02-05T06:54:49.948677+0000","last_became_active":"2024-02-05T06:51:13.274182+0000","last_became_peered":"2024-02-05T06:51:13.274182+0000","last_unstale":"2024-02-05T06:54:49.948677+0000","last_undegraded":"2024-02-05T06:54:49.948677+0000","last_fullsized":"2024-02-05T06:54:49.948677+0000","mapping_epoch":418,"log_start":"466'11500","ondisk_log_start":"466'11500","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"476'15003","last_scrub_stamp":"2024-02-05T06:54:38.835528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:54:38.835528+0000","objects_scrubbed":819,"log_size":4316,"log_dups_size":0,"ondisk_log_size":4316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:54.485575+0000","scrub_duration":314,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15816,"num_write_kb":506112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":913,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.519111+0000","last_change":"2024-02-05T06:54:39.400056+0000","last_active":"2024-02-05T06:54:48.519111+0000","last_peered":"2024-02-05T06:54:48.519111+0000","last_clean":"2024-02-05T06:54:48.519111+0000","last_became_active":"2024-02-05T06:36:54.973387+0000","last_became_peered":"2024-02-05T06:36:54.973387+0000","last_unstale":"2024-02-05T06:54:48.519111+0000","last_undegraded":"2024-02-05T06:54:48.519111+0000","last_fullsized":"2024-02-05T06:54:48.519111+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:54:39.399928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:54:39.399928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:43.721335+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"465'64","reported_seq":867,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.550095+0000","last_change":"2024-02-05T06:53:36.797035+0000","last_active":"2024-02-05T06:54:48.550095+0000","last_peered":"2024-02-05T06:54:48.550095+0000","last_clean":"2024-02-05T06:54:48.550095+0000","last_became_active":"2024-02-05T06:40:08.604483+0000","last_became_peered":"2024-02-05T06:40:08.604483+0000","last_unstale":"2024-02-05T06:54:48.550095+0000","last_undegraded":"2024-02-05T06:54:48.550095+0000","last_fullsized":"2024-02-05T06:54:48.550095+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'52","last_scrub_stamp":"2024-02-05T06:53:36.796907+0000","last_deep_scrub":"326'52","last_deep_scrub_stamp":"2024-02-05T06:45:53.715482+0000","last_clean_scrub_stamp":"2024-02-05T06:53:36.796907+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":429,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.536749+0000","last_change":"2024-02-05T06:52:47.127090+0000","last_active":"2024-02-05T06:54:48.536749+0000","last_peered":"2024-02-05T06:54:48.536749+0000","last_clean":"2024-02-05T06:54:48.536749+0000","last_became_active":"2024-02-05T06:45:09.776169+0000","last_became_peered":"2024-02-05T06:45:09.776169+0000","last_unstale":"2024-02-05T06:54:48.536749+0000","last_undegraded":"2024-02-05T06:54:48.536749+0000","last_fullsized":"2024-02-05T06:54:48.536749+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:47.126978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:47.126978+0000","last_clean_scrub_stamp":"2024-02-05T06:52:47.126978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"182'3","reported_seq":520,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.526358+0000","last_change":"2024-02-05T06:52:40.877231+0000","last_active":"2024-02-05T06:54:48.526358+0000","last_peered":"2024-02-05T06:54:48.526358+0000","last_clean":"2024-02-05T06:54:48.526358+0000","last_became_active":"2024-02-05T06:41:43.415806+0000","last_became_peered":"2024-02-05T06:41:43.415806+0000","last_unstale":"2024-02-05T06:54:48.526358+0000","last_undegraded":"2024-02-05T06:54:48.526358+0000","last_fullsized":"2024-02-05T06:54:48.526358+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'3","last_scrub_stamp":"2024-02-05T06:52:40.876912+0000","last_deep_scrub":"182'3","last_deep_scrub_stamp":"2024-02-05T06:39:33.373415+0000","last_clean_scrub_stamp":"2024-02-05T06:52:40.876912+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001220348,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":903,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.548472+0000","last_change":"2024-02-05T06:54:44.628472+0000","last_active":"2024-02-05T06:54:48.548472+0000","last_peered":"2024-02-05T06:54:48.548472+0000","last_clean":"2024-02-05T06:54:48.548472+0000","last_became_active":"2024-02-05T06:40:27.082440+0000","last_became_peered":"2024-02-05T06:40:27.082440+0000","last_unstale":"2024-02-05T06:54:48.548472+0000","last_undegraded":"2024-02-05T06:54:48.548472+0000","last_fullsized":"2024-02-05T06:54:48.548472+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:54:44.628253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:50:52.403179+0000","last_clean_scrub_stamp":"2024-02-05T06:54:44.628253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:04.875202+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":941,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.525663+0000","last_change":"2024-02-05T06:52:17.849188+0000","last_active":"2024-02-05T06:54:48.525663+0000","last_peered":"2024-02-05T06:54:48.525663+0000","last_clean":"2024-02-05T06:54:48.525663+0000","last_became_active":"2024-02-05T06:51:54.325737+0000","last_became_peered":"2024-02-05T06:51:54.325737+0000","last_unstale":"2024-02-05T06:54:48.525663+0000","last_undegraded":"2024-02-05T06:54:48.525663+0000","last_fullsized":"2024-02-05T06:54:48.525663+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:52:17.849056+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:37.282442+0000","last_clean_scrub_stamp":"2024-02-05T06:52:17.849056+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"83'128","reported_seq":416,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.531816+0000","last_change":"2024-02-05T06:54:47.850076+0000","last_active":"2024-02-05T06:54:48.531816+0000","last_peered":"2024-02-05T06:54:48.531816+0000","last_clean":"2024-02-05T06:54:48.531816+0000","last_became_active":"2024-02-05T06:51:59.294208+0000","last_became_peered":"2024-02-05T06:51:59.294208+0000","last_unstale":"2024-02-05T06:54:48.531816+0000","last_undegraded":"2024-02-05T06:54:48.531816+0000","last_fullsized":"2024-02-05T06:54:48.531816+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:54:47.849978+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:54:47.849978+0000","last_clean_scrub_stamp":"2024-02-05T06:54:47.849978+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:08.074950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.4","version":"478'15632","reported_seq":15824,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:49.928000+0000","last_change":"2024-02-05T06:53:33.493384+0000","last_active":"2024-02-05T06:54:49.928000+0000","last_peered":"2024-02-05T06:54:49.928000+0000","last_clean":"2024-02-05T06:54:49.928000+0000","last_became_active":"2024-02-05T06:51:13.272748+0000","last_became_peered":"2024-02-05T06:51:13.272748+0000","last_unstale":"2024-02-05T06:54:49.928000+0000","last_undegraded":"2024-02-05T06:54:49.928000+0000","last_fullsized":"2024-02-05T06:54:49.928000+0000","mapping_epoch":418,"log_start":"465'11000","ondisk_log_start":"465'11000","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'10373","last_scrub_stamp":"2024-02-05T06:53:33.492667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:53:33.492667+0000","objects_scrubbed":5303,"log_size":4632,"log_dups_size":0,"ondisk_log_size":4632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1404,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15632,"num_write_kb":500224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"465'21","reported_seq":1018,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.536659+0000","last_change":"2024-02-05T06:54:41.381913+0000","last_active":"2024-02-05T06:54:48.536659+0000","last_peered":"2024-02-05T06:54:48.536659+0000","last_clean":"2024-02-05T06:54:48.536659+0000","last_became_active":"2024-02-05T06:40:08.584204+0000","last_became_peered":"2024-02-05T06:40:08.584204+0000","last_unstale":"2024-02-05T06:54:48.536659+0000","last_undegraded":"2024-02-05T06:54:48.536659+0000","last_fullsized":"2024-02-05T06:54:48.536659+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'21","last_scrub_stamp":"2024-02-05T06:54:41.381119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:54:41.381119+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:47.623986+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":515,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.526301+0000","last_change":"2024-02-05T06:52:32.883872+0000","last_active":"2024-02-05T06:54:48.526301+0000","last_peered":"2024-02-05T06:54:48.526301+0000","last_clean":"2024-02-05T06:54:48.526301+0000","last_became_active":"2024-02-05T06:41:43.412611+0000","last_became_peered":"2024-02-05T06:41:43.412611+0000","last_unstale":"2024-02-05T06:54:48.526301+0000","last_undegraded":"2024-02-05T06:54:48.526301+0000","last_fullsized":"2024-02-05T06:54:48.526301+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:32.883756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:41.617526+0000","last_clean_scrub_stamp":"2024-02-05T06:52:32.883756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":941,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.529396+0000","last_change":"2024-02-05T06:54:45.120009+0000","last_active":"2024-02-05T06:54:48.529396+0000","last_peered":"2024-02-05T06:54:48.529396+0000","last_clean":"2024-02-05T06:54:48.529396+0000","last_became_active":"2024-02-05T06:36:54.966016+0000","last_became_peered":"2024-02-05T06:36:54.966016+0000","last_unstale":"2024-02-05T06:54:48.529396+0000","last_undegraded":"2024-02-05T06:54:48.529396+0000","last_fullsized":"2024-02-05T06:54:48.529396+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:54:45.119901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:54:45.119901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:02.245310+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":900,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.536736+0000","last_change":"2024-02-05T06:53:50.717387+0000","last_active":"2024-02-05T06:54:48.536736+0000","last_peered":"2024-02-05T06:54:48.536736+0000","last_clean":"2024-02-05T06:54:48.536736+0000","last_became_active":"2024-02-05T06:40:27.129625+0000","last_became_peered":"2024-02-05T06:40:27.129625+0000","last_unstale":"2024-02-05T06:54:48.536736+0000","last_undegraded":"2024-02-05T06:54:48.536736+0000","last_fullsized":"2024-02-05T06:54:48.536736+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:50.717250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:53:50.717250+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:08.463818+0000","scrub_duration":214,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"83'128","reported_seq":1126,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.535375+0000","last_change":"2024-02-05T06:54:39.264389+0000","last_active":"2024-02-05T06:54:48.535375+0000","last_peered":"2024-02-05T06:54:48.535375+0000","last_clean":"2024-02-05T06:54:48.535375+0000","last_became_active":"2024-02-05T06:51:59.293451+0000","last_became_peered":"2024-02-05T06:51:59.293451+0000","last_unstale":"2024-02-05T06:54:48.535375+0000","last_undegraded":"2024-02-05T06:54:48.535375+0000","last_fullsized":"2024-02-05T06:54:48.535375+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:54:39.264267+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:48:25.725892+0000","last_clean_scrub_stamp":"2024-02-05T06:54:39.264267+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:42.358751+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.5","version":"478'15890","reported_seq":16107,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:49.929421+0000","last_change":"2024-02-05T06:52:50.694891+0000","last_active":"2024-02-05T06:54:49.929421+0000","last_peered":"2024-02-05T06:54:49.929421+0000","last_clean":"2024-02-05T06:54:49.929421+0000","last_became_active":"2024-02-05T06:51:13.270923+0000","last_became_peered":"2024-02-05T06:51:13.270923+0000","last_unstale":"2024-02-05T06:54:49.929421+0000","last_undegraded":"2024-02-05T06:54:49.929421+0000","last_fullsized":"2024-02-05T06:54:49.929421+0000","mapping_epoch":418,"log_start":"466'11300","ondisk_log_start":"466'11300","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'7908","last_scrub_stamp":"2024-02-05T06:52:50.694691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:52:50.694691+0000","objects_scrubbed":7859,"log_size":4590,"log_dups_size":0,"ondisk_log_size":4590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15890,"num_write_kb":508480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"465'60","reported_seq":1084,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.535069+0000","last_change":"2024-02-05T06:52:51.676140+0000","last_active":"2024-02-05T06:54:48.535069+0000","last_peered":"2024-02-05T06:54:48.535069+0000","last_clean":"2024-02-05T06:54:48.535069+0000","last_became_active":"2024-02-05T06:40:27.068417+0000","last_became_peered":"2024-02-05T06:40:27.068417+0000","last_unstale":"2024-02-05T06:54:48.535069+0000","last_undegraded":"2024-02-05T06:54:48.535069+0000","last_fullsized":"2024-02-05T06:54:48.535069+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'44","last_scrub_stamp":"2024-02-05T06:52:51.676057+0000","last_deep_scrub":"279'28","last_deep_scrub_stamp":"2024-02-05T06:42:50.148881+0000","last_clean_scrub_stamp":"2024-02-05T06:52:51.676057+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":614,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.532633+0000","last_change":"2024-02-05T06:54:01.596848+0000","last_active":"2024-02-05T06:54:48.532633+0000","last_peered":"2024-02-05T06:54:48.532633+0000","last_clean":"2024-02-05T06:54:48.532633+0000","last_became_active":"2024-02-05T06:41:43.418202+0000","last_became_peered":"2024-02-05T06:41:43.418202+0000","last_unstale":"2024-02-05T06:54:48.532633+0000","last_undegraded":"2024-02-05T06:54:48.532633+0000","last_fullsized":"2024-02-05T06:54:48.532633+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:54:01.596730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:54:01.596730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:01.961347+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":939,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.529448+0000","last_change":"2024-02-05T06:53:52.054596+0000","last_active":"2024-02-05T06:54:48.529448+0000","last_peered":"2024-02-05T06:54:48.529448+0000","last_clean":"2024-02-05T06:54:48.529448+0000","last_became_active":"2024-02-05T06:41:43.413095+0000","last_became_peered":"2024-02-05T06:41:43.413095+0000","last_unstale":"2024-02-05T06:54:48.529448+0000","last_undegraded":"2024-02-05T06:54:48.529448+0000","last_fullsized":"2024-02-05T06:54:48.529448+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:52.054463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:45.073681+0000","last_clean_scrub_stamp":"2024-02-05T06:53:52.054463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:21.447521+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":616,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.532038+0000","last_change":"2024-02-05T06:54:36.823255+0000","last_active":"2024-02-05T06:54:48.532038+0000","last_peered":"2024-02-05T06:54:48.532038+0000","last_clean":"2024-02-05T06:54:48.532038+0000","last_became_active":"2024-02-05T06:40:27.218081+0000","last_became_peered":"2024-02-05T06:40:27.218081+0000","last_unstale":"2024-02-05T06:54:48.532038+0000","last_undegraded":"2024-02-05T06:54:48.532038+0000","last_fullsized":"2024-02-05T06:54:48.532038+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:54:36.823146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:32.925000+0000","last_clean_scrub_stamp":"2024-02-05T06:54:36.823146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:04.476515+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"83'128","reported_seq":1119,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.519283+0000","last_change":"2024-02-05T06:52:21.388903+0000","last_active":"2024-02-05T06:54:48.519283+0000","last_peered":"2024-02-05T06:54:48.519283+0000","last_clean":"2024-02-05T06:54:48.519283+0000","last_became_active":"2024-02-05T06:51:59.296988+0000","last_became_peered":"2024-02-05T06:51:59.296988+0000","last_unstale":"2024-02-05T06:54:48.519283+0000","last_undegraded":"2024-02-05T06:54:48.519283+0000","last_fullsized":"2024-02-05T06:54:48.519283+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:52:21.388783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:52:21.388783+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.6","version":"478'15836","reported_seq":16019,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:49.950126+0000","last_change":"2024-02-05T06:53:46.461956+0000","last_active":"2024-02-05T06:54:49.950126+0000","last_peered":"2024-02-05T06:54:49.950126+0000","last_clean":"2024-02-05T06:54:49.950126+0000","last_became_active":"2024-02-05T06:51:13.267525+0000","last_became_peered":"2024-02-05T06:51:13.267525+0000","last_unstale":"2024-02-05T06:54:49.950126+0000","last_undegraded":"2024-02-05T06:54:49.950126+0000","last_fullsized":"2024-02-05T06:54:49.950126+0000","mapping_epoch":418,"log_start":"464'10600","ondisk_log_start":"464'10600","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"466'11503","last_scrub_stamp":"2024-02-05T06:53:46.461835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:12.249449+0000","last_clean_scrub_stamp":"2024-02-05T06:53:46.461835+0000","objects_scrubbed":4404,"log_size":5236,"log_dups_size":0,"ondisk_log_size":5236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:58.262026+0000","scrub_duration":1947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15836,"num_write_kb":506752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":921,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.529017+0000","last_change":"2024-02-05T06:54:38.176682+0000","last_active":"2024-02-05T06:54:48.529017+0000","last_peered":"2024-02-05T06:54:48.529017+0000","last_clean":"2024-02-05T06:54:48.529017+0000","last_became_active":"2024-02-05T06:33:33.592546+0000","last_became_peered":"2024-02-05T06:33:33.592546+0000","last_unstale":"2024-02-05T06:54:48.529017+0000","last_undegraded":"2024-02-05T06:54:48.529017+0000","last_fullsized":"2024-02-05T06:54:48.529017+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:54:38.176554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:42.685240+0000","last_clean_scrub_stamp":"2024-02-05T06:54:38.176554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:44.667553+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"465'24","reported_seq":768,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.538803+0000","last_change":"2024-02-05T06:52:26.996274+0000","last_active":"2024-02-05T06:54:48.538803+0000","last_peered":"2024-02-05T06:54:48.538803+0000","last_clean":"2024-02-05T06:54:48.538803+0000","last_became_active":"2024-02-05T06:45:09.783036+0000","last_became_peered":"2024-02-05T06:45:09.783036+0000","last_unstale":"2024-02-05T06:54:48.538803+0000","last_undegraded":"2024-02-05T06:54:48.538803+0000","last_fullsized":"2024-02-05T06:54:48.538803+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'18","last_scrub_stamp":"2024-02-05T06:52:26.996136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:52:26.996136+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":29,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":608,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.533132+0000","last_change":"2024-02-05T06:53:36.100637+0000","last_active":"2024-02-05T06:54:48.533132+0000","last_peered":"2024-02-05T06:54:48.533132+0000","last_clean":"2024-02-05T06:54:48.533132+0000","last_became_active":"2024-02-05T06:40:27.063300+0000","last_became_peered":"2024-02-05T06:40:27.063300+0000","last_unstale":"2024-02-05T06:54:48.533132+0000","last_undegraded":"2024-02-05T06:54:48.533132+0000","last_fullsized":"2024-02-05T06:54:48.533132+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:36.100400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:34.977997+0000","last_clean_scrub_stamp":"2024-02-05T06:53:36.100400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":883,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.548835+0000","last_change":"2024-02-05T06:51:22.703991+0000","last_active":"2024-02-05T06:54:48.548835+0000","last_peered":"2024-02-05T06:54:48.548835+0000","last_clean":"2024-02-05T06:54:48.548835+0000","last_became_active":"2024-02-05T06:40:27.092613+0000","last_became_peered":"2024-02-05T06:40:27.092613+0000","last_unstale":"2024-02-05T06:54:48.548835+0000","last_undegraded":"2024-02-05T06:54:48.548835+0000","last_fullsized":"2024-02-05T06:54:48.548835+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:22.703865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:51:22.703865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":190,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"83'128","reported_seq":684,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.837190+0000","last_change":"2024-02-05T06:54:48.837234+0000","last_active":"2024-02-05T06:54:48.837190+0000","last_peered":"2024-02-05T06:54:48.837190+0000","last_clean":"2024-02-05T06:54:48.837190+0000","last_became_active":"2024-02-05T06:51:59.294191+0000","last_became_peered":"2024-02-05T06:51:59.294191+0000","last_unstale":"2024-02-05T06:54:48.837190+0000","last_undegraded":"2024-02-05T06:54:48.837190+0000","last_fullsized":"2024-02-05T06:54:48.837190+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:54:48.837109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:54:48.837109+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:01.932553+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.7","version":"478'19513","reported_seq":19808,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:49.933462+0000","last_change":"2024-02-05T06:54:08.528393+0000","last_active":"2024-02-05T06:54:49.933462+0000","last_peered":"2024-02-05T06:54:49.933462+0000","last_clean":"2024-02-05T06:54:49.933462+0000","last_became_active":"2024-02-05T06:51:13.272066+0000","last_became_peered":"2024-02-05T06:51:13.272066+0000","last_unstale":"2024-02-05T06:54:49.933462+0000","last_undegraded":"2024-02-05T06:54:49.933462+0000","last_fullsized":"2024-02-05T06:54:49.933462+0000","mapping_epoch":418,"log_start":"470'14501","ondisk_log_start":"470'14501","created":418,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"471'16513","last_scrub_stamp":"2024-02-05T06:54:08.528248+0000","last_deep_scrub":"471'16513","last_deep_scrub_stamp":"2024-02-05T06:54:08.528248+0000","last_clean_scrub_stamp":"2024-02-05T06:54:08.528248+0000","objects_scrubbed":3651,"log_size":5012,"log_dups_size":0,"ondisk_log_size":5012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":20,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:36.731421+0000","scrub_duration":19752,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15862,"num_write_kb":507584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"208'12","reported_seq":785,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.532307+0000","last_change":"2024-02-05T06:52:33.103331+0000","last_active":"2024-02-05T06:54:48.532307+0000","last_peered":"2024-02-05T06:54:48.532307+0000","last_clean":"2024-02-05T06:54:48.532307+0000","last_became_active":"2024-02-05T06:40:27.064195+0000","last_became_peered":"2024-02-05T06:40:27.064195+0000","last_unstale":"2024-02-05T06:54:48.532307+0000","last_undegraded":"2024-02-05T06:54:48.532307+0000","last_fullsized":"2024-02-05T06:54:48.532307+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'12","last_scrub_stamp":"2024-02-05T06:52:33.103188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:52:33.103188+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"465'39","reported_seq":568,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.525767+0000","last_change":"2024-02-05T06:53:36.831417+0000","last_active":"2024-02-05T06:54:48.525767+0000","last_peered":"2024-02-05T06:54:48.525767+0000","last_clean":"2024-02-05T06:54:48.525767+0000","last_became_active":"2024-02-05T06:41:43.412579+0000","last_became_peered":"2024-02-05T06:41:43.412579+0000","last_unstale":"2024-02-05T06:54:48.525767+0000","last_undegraded":"2024-02-05T06:54:48.525767+0000","last_fullsized":"2024-02-05T06:54:48.525767+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'27","last_scrub_stamp":"2024-02-05T06:53:36.831295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:53:36.831295+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:03.891726+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"412'4","reported_seq":616,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.533258+0000","last_change":"2024-02-05T06:52:39.111257+0000","last_active":"2024-02-05T06:54:48.533258+0000","last_peered":"2024-02-05T06:54:48.533258+0000","last_clean":"2024-02-05T06:54:48.533258+0000","last_became_active":"2024-02-05T06:40:27.068279+0000","last_became_peered":"2024-02-05T06:40:27.068279+0000","last_unstale":"2024-02-05T06:54:48.533258+0000","last_undegraded":"2024-02-05T06:54:48.533258+0000","last_fullsized":"2024-02-05T06:54:48.533258+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'4","last_scrub_stamp":"2024-02-05T06:52:39.111151+0000","last_deep_scrub":"412'4","last_deep_scrub_stamp":"2024-02-05T06:50:51.841825+0000","last_clean_scrub_stamp":"2024-02-05T06:52:39.111151+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":516,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.526269+0000","last_change":"2024-02-05T06:52:26.888326+0000","last_active":"2024-02-05T06:54:48.526269+0000","last_peered":"2024-02-05T06:54:48.526269+0000","last_clean":"2024-02-05T06:54:48.526269+0000","last_became_active":"2024-02-05T06:41:43.421231+0000","last_became_peered":"2024-02-05T06:41:43.421231+0000","last_unstale":"2024-02-05T06:54:48.526269+0000","last_undegraded":"2024-02-05T06:54:48.526269+0000","last_fullsized":"2024-02-05T06:54:48.526269+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:26.888098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:52:26.888098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"83'128","reported_seq":543,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.525887+0000","last_change":"2024-02-05T06:52:21.904768+0000","last_active":"2024-02-05T06:54:48.525887+0000","last_peered":"2024-02-05T06:54:48.525887+0000","last_clean":"2024-02-05T06:54:48.525887+0000","last_became_active":"2024-02-05T06:51:59.296150+0000","last_became_peered":"2024-02-05T06:51:59.296150+0000","last_unstale":"2024-02-05T06:54:48.525887+0000","last_undegraded":"2024-02-05T06:54:48.525887+0000","last_fullsized":"2024-02-05T06:54:48.525887+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:52:21.904642+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:48:28.657679+0000","last_clean_scrub_stamp":"2024-02-05T06:52:21.904642+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"465'21","reported_seq":944,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.549729+0000","last_change":"2024-02-05T06:53:51.658817+0000","last_active":"2024-02-05T06:54:48.549729+0000","last_peered":"2024-02-05T06:54:48.549729+0000","last_clean":"2024-02-05T06:54:48.549729+0000","last_became_active":"2024-02-05T06:40:27.091689+0000","last_became_peered":"2024-02-05T06:40:27.091689+0000","last_unstale":"2024-02-05T06:54:48.549729+0000","last_undegraded":"2024-02-05T06:54:48.549729+0000","last_fullsized":"2024-02-05T06:54:48.549729+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'21","last_scrub_stamp":"2024-02-05T06:53:51.658691+0000","last_deep_scrub":"412'15","last_deep_scrub_stamp":"2024-02-05T06:50:55.397689+0000","last_clean_scrub_stamp":"2024-02-05T06:53:51.658691+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:01.285256+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":934,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.534799+0000","last_change":"2024-02-05T06:53:55.061956+0000","last_active":"2024-02-05T06:54:48.534799+0000","last_peered":"2024-02-05T06:54:48.534799+0000","last_clean":"2024-02-05T06:54:48.534799+0000","last_became_active":"2024-02-05T06:41:43.412130+0000","last_became_peered":"2024-02-05T06:41:43.412130+0000","last_unstale":"2024-02-05T06:54:48.534799+0000","last_undegraded":"2024-02-05T06:54:48.534799+0000","last_fullsized":"2024-02-05T06:54:48.534799+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:55.061848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:53:55.061848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:24.423601+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":515,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.526659+0000","last_change":"2024-02-05T06:52:24.934323+0000","last_active":"2024-02-05T06:54:48.526659+0000","last_peered":"2024-02-05T06:54:48.526659+0000","last_clean":"2024-02-05T06:54:48.526659+0000","last_became_active":"2024-02-05T06:41:43.406094+0000","last_became_peered":"2024-02-05T06:41:43.406094+0000","last_unstale":"2024-02-05T06:54:48.526659+0000","last_undegraded":"2024-02-05T06:54:48.526659+0000","last_fullsized":"2024-02-05T06:54:48.526659+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:24.934198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:44:57.506909+0000","last_clean_scrub_stamp":"2024-02-05T06:52:24.934198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":607,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.532748+0000","last_change":"2024-02-05T06:53:54.796403+0000","last_active":"2024-02-05T06:54:48.532748+0000","last_peered":"2024-02-05T06:54:48.532748+0000","last_clean":"2024-02-05T06:54:48.532748+0000","last_became_active":"2024-02-05T06:40:27.063350+0000","last_became_peered":"2024-02-05T06:40:27.063350+0000","last_unstale":"2024-02-05T06:54:48.532748+0000","last_undegraded":"2024-02-05T06:54:48.532748+0000","last_fullsized":"2024-02-05T06:54:48.532748+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:54.796297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:53:54.796297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:17.620285+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"83'128","reported_seq":1086,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.539065+0000","last_change":"2024-02-05T06:54:02.691383+0000","last_active":"2024-02-05T06:54:48.539065+0000","last_peered":"2024-02-05T06:54:48.539065+0000","last_clean":"2024-02-05T06:54:48.539065+0000","last_became_active":"2024-02-05T06:51:59.303621+0000","last_became_peered":"2024-02-05T06:51:59.303621+0000","last_unstale":"2024-02-05T06:54:48.539065+0000","last_undegraded":"2024-02-05T06:54:48.539065+0000","last_fullsized":"2024-02-05T06:54:48.539065+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:54:02.691263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:54:02.691263+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:17.412431+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"465'21","reported_seq":969,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.536465+0000","last_change":"2024-02-05T06:53:37.421718+0000","last_active":"2024-02-05T06:54:48.536465+0000","last_peered":"2024-02-05T06:54:48.536465+0000","last_clean":"2024-02-05T06:54:48.536465+0000","last_became_active":"2024-02-05T06:40:27.101116+0000","last_became_peered":"2024-02-05T06:40:27.101116+0000","last_unstale":"2024-02-05T06:54:48.536465+0000","last_undegraded":"2024-02-05T06:54:48.536465+0000","last_fullsized":"2024-02-05T06:54:48.536465+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'19","last_scrub_stamp":"2024-02-05T06:53:37.421591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:53:37.421591+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:56.525787+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":608,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.532351+0000","last_change":"2024-02-05T06:53:44.983388+0000","last_active":"2024-02-05T06:54:48.532351+0000","last_peered":"2024-02-05T06:54:48.532351+0000","last_clean":"2024-02-05T06:54:48.532351+0000","last_became_active":"2024-02-05T06:40:27.064388+0000","last_became_peered":"2024-02-05T06:40:27.064388+0000","last_unstale":"2024-02-05T06:54:48.532351+0000","last_undegraded":"2024-02-05T06:54:48.532351+0000","last_fullsized":"2024-02-05T06:54:48.532351+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:44.983315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:53:44.983315+0000","last_clean_scrub_stamp":"2024-02-05T06:53:44.983315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:50.922462+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":900,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.549798+0000","last_change":"2024-02-05T06:52:26.443142+0000","last_active":"2024-02-05T06:54:48.549798+0000","last_peered":"2024-02-05T06:54:48.549798+0000","last_clean":"2024-02-05T06:54:48.549798+0000","last_became_active":"2024-02-05T06:40:27.091544+0000","last_became_peered":"2024-02-05T06:40:27.091544+0000","last_unstale":"2024-02-05T06:54:48.549798+0000","last_undegraded":"2024-02-05T06:54:48.549798+0000","last_fullsized":"2024-02-05T06:54:48.549798+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:26.443024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:52:26.443024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":941,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.550696+0000","last_change":"2024-02-05T06:52:45.010159+0000","last_active":"2024-02-05T06:54:48.550696+0000","last_peered":"2024-02-05T06:54:48.550696+0000","last_clean":"2024-02-05T06:54:48.550696+0000","last_became_active":"2024-02-05T06:40:27.090632+0000","last_became_peered":"2024-02-05T06:40:27.090632+0000","last_unstale":"2024-02-05T06:54:48.550696+0000","last_undegraded":"2024-02-05T06:54:48.550696+0000","last_fullsized":"2024-02-05T06:54:48.550696+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:45.010039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:45.010039+0000","last_clean_scrub_stamp":"2024-02-05T06:52:45.010039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"83'128","reported_seq":551,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.525039+0000","last_change":"2024-02-05T06:54:44.747161+0000","last_active":"2024-02-05T06:54:48.525039+0000","last_peered":"2024-02-05T06:54:48.525039+0000","last_clean":"2024-02-05T06:54:48.525039+0000","last_became_active":"2024-02-05T06:51:59.290466+0000","last_became_peered":"2024-02-05T06:51:59.290466+0000","last_unstale":"2024-02-05T06:54:48.525039+0000","last_undegraded":"2024-02-05T06:54:48.525039+0000","last_fullsized":"2024-02-05T06:54:48.525039+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:54:44.747058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:54:44.747058+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:14.240812+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"465'21","reported_seq":543,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.526574+0000","last_change":"2024-02-05T06:52:31.905022+0000","last_active":"2024-02-05T06:54:48.526574+0000","last_peered":"2024-02-05T06:54:48.526574+0000","last_clean":"2024-02-05T06:54:48.526574+0000","last_became_active":"2024-02-05T06:41:43.413708+0000","last_became_peered":"2024-02-05T06:41:43.413708+0000","last_unstale":"2024-02-05T06:54:48.526574+0000","last_undegraded":"2024-02-05T06:54:48.526574+0000","last_fullsized":"2024-02-05T06:54:48.526574+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:52:31.904813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:52:31.904813+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":920,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.519291+0000","last_change":"2024-02-05T06:53:43.562505+0000","last_active":"2024-02-05T06:54:48.519291+0000","last_peered":"2024-02-05T06:54:48.519291+0000","last_clean":"2024-02-05T06:54:48.519291+0000","last_became_active":"2024-02-05T06:40:27.088096+0000","last_became_peered":"2024-02-05T06:40:27.088096+0000","last_unstale":"2024-02-05T06:54:48.519291+0000","last_undegraded":"2024-02-05T06:54:48.519291+0000","last_fullsized":"2024-02-05T06:54:48.519291+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:43.562266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:53:43.562266+0000","last_clean_scrub_stamp":"2024-02-05T06:53:43.562266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:09.519621+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"208'10","reported_seq":1168,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.547930+0000","last_change":"2024-02-05T06:52:43.367615+0000","last_active":"2024-02-05T06:54:48.547930+0000","last_peered":"2024-02-05T06:54:48.547930+0000","last_clean":"2024-02-05T06:54:48.547930+0000","last_became_active":"2024-02-05T06:40:27.096584+0000","last_became_peered":"2024-02-05T06:40:27.096584+0000","last_unstale":"2024-02-05T06:54:48.547930+0000","last_undegraded":"2024-02-05T06:54:48.547930+0000","last_fullsized":"2024-02-05T06:54:48.547930+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-02-05T06:52:43.367496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:52:43.367496+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":663,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.549924+0000","last_change":"2024-02-05T06:54:44.325128+0000","last_active":"2024-02-05T06:54:48.549924+0000","last_peered":"2024-02-05T06:54:48.549924+0000","last_clean":"2024-02-05T06:54:48.549924+0000","last_became_active":"2024-02-05T06:45:09.775152+0000","last_became_peered":"2024-02-05T06:45:09.775152+0000","last_unstale":"2024-02-05T06:54:48.549924+0000","last_undegraded":"2024-02-05T06:54:48.549924+0000","last_fullsized":"2024-02-05T06:54:48.549924+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:54:44.325013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:41:38.559780+0000","last_clean_scrub_stamp":"2024-02-05T06:54:44.325013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:11.814367+0000","scrub_duration":198,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.11","version":"83'128","reported_seq":551,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.518834+0000","last_change":"2024-02-05T06:54:42.344402+0000","last_active":"2024-02-05T06:54:48.518834+0000","last_peered":"2024-02-05T06:54:48.518834+0000","last_clean":"2024-02-05T06:54:48.518834+0000","last_became_active":"2024-02-05T06:51:59.300112+0000","last_became_peered":"2024-02-05T06:51:59.300112+0000","last_unstale":"2024-02-05T06:54:48.518834+0000","last_undegraded":"2024-02-05T06:54:48.518834+0000","last_fullsized":"2024-02-05T06:54:48.518834+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:54:42.343810+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:37.282442+0000","last_clean_scrub_stamp":"2024-02-05T06:54:42.343810+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:58.465408+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"465'30","reported_seq":986,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.519662+0000","last_change":"2024-02-05T06:53:26.322082+0000","last_active":"2024-02-05T06:54:48.519662+0000","last_peered":"2024-02-05T06:54:48.519662+0000","last_clean":"2024-02-05T06:54:48.519662+0000","last_became_active":"2024-02-05T06:40:27.093418+0000","last_became_peered":"2024-02-05T06:40:27.093418+0000","last_unstale":"2024-02-05T06:54:48.519662+0000","last_undegraded":"2024-02-05T06:54:48.519662+0000","last_fullsized":"2024-02-05T06:54:48.519662+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'20","last_scrub_stamp":"2024-02-05T06:53:26.321955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:53:26.321955+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":608,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.532515+0000","last_change":"2024-02-05T06:53:43.055431+0000","last_active":"2024-02-05T06:54:48.532515+0000","last_peered":"2024-02-05T06:54:48.532515+0000","last_clean":"2024-02-05T06:54:48.532515+0000","last_became_active":"2024-02-05T06:40:27.098418+0000","last_became_peered":"2024-02-05T06:40:27.098418+0000","last_unstale":"2024-02-05T06:54:48.532515+0000","last_undegraded":"2024-02-05T06:54:48.532515+0000","last_fullsized":"2024-02-05T06:54:48.532515+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:43.055316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:53:43.055316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:12.814470+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"336'11","reported_seq":678,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.526294+0000","last_change":"2024-02-05T06:52:18.890279+0000","last_active":"2024-02-05T06:54:48.526294+0000","last_peered":"2024-02-05T06:54:48.526294+0000","last_clean":"2024-02-05T06:54:48.526294+0000","last_became_active":"2024-02-05T06:41:43.418753+0000","last_became_peered":"2024-02-05T06:41:43.418753+0000","last_unstale":"2024-02-05T06:54:48.526294+0000","last_undegraded":"2024-02-05T06:54:48.526294+0000","last_fullsized":"2024-02-05T06:54:48.526294+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'11","last_scrub_stamp":"2024-02-05T06:52:18.890167+0000","last_deep_scrub":"336'11","last_deep_scrub_stamp":"2024-02-05T06:46:55.721412+0000","last_clean_scrub_stamp":"2024-02-05T06:52:18.890167+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":899,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:49.580670+0000","last_change":"2024-02-05T06:54:49.580727+0000","last_active":"2024-02-05T06:54:49.580670+0000","last_peered":"2024-02-05T06:54:49.580670+0000","last_clean":"2024-02-05T06:54:49.580670+0000","last_became_active":"2024-02-05T06:40:27.096132+0000","last_became_peered":"2024-02-05T06:40:27.096132+0000","last_unstale":"2024-02-05T06:54:49.580670+0000","last_undegraded":"2024-02-05T06:54:49.580670+0000","last_fullsized":"2024-02-05T06:54:49.580670+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:54:49.580578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:54:49.580578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:00.847366+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"465'32","reported_seq":781,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.535407+0000","last_change":"2024-02-05T06:53:32.977787+0000","last_active":"2024-02-05T06:54:48.535407+0000","last_peered":"2024-02-05T06:54:48.535407+0000","last_clean":"2024-02-05T06:54:48.535407+0000","last_became_active":"2024-02-05T06:45:09.784411+0000","last_became_peered":"2024-02-05T06:45:09.784411+0000","last_unstale":"2024-02-05T06:54:48.535407+0000","last_undegraded":"2024-02-05T06:54:48.535407+0000","last_fullsized":"2024-02-05T06:54:48.535407+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:53:32.977666+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:35:02.589229+0000","last_clean_scrub_stamp":"2024-02-05T06:53:32.977666+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":554,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.532828+0000","last_change":"2024-02-05T06:52:44.163784+0000","last_active":"2024-02-05T06:54:48.532828+0000","last_peered":"2024-02-05T06:54:48.532828+0000","last_clean":"2024-02-05T06:54:48.532828+0000","last_became_active":"2024-02-05T06:45:09.773490+0000","last_became_peered":"2024-02-05T06:45:09.773490+0000","last_unstale":"2024-02-05T06:54:48.532828+0000","last_undegraded":"2024-02-05T06:54:48.532828+0000","last_fullsized":"2024-02-05T06:54:48.532828+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:44.163661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:52:44.163661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":925,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.539193+0000","last_change":"2024-02-05T06:53:53.666791+0000","last_active":"2024-02-05T06:54:48.539193+0000","last_peered":"2024-02-05T06:54:48.539193+0000","last_clean":"2024-02-05T06:54:48.539193+0000","last_became_active":"2024-02-05T06:41:43.421433+0000","last_became_peered":"2024-02-05T06:41:43.421433+0000","last_unstale":"2024-02-05T06:54:48.539193+0000","last_undegraded":"2024-02-05T06:54:48.539193+0000","last_fullsized":"2024-02-05T06:54:48.539193+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:53.666667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:53:53.666667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:17.847146+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":891,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.549214+0000","last_change":"2024-02-05T06:53:31.491982+0000","last_active":"2024-02-05T06:54:48.549214+0000","last_peered":"2024-02-05T06:54:48.549214+0000","last_clean":"2024-02-05T06:54:48.549214+0000","last_became_active":"2024-02-05T06:41:43.417155+0000","last_became_peered":"2024-02-05T06:41:43.417155+0000","last_unstale":"2024-02-05T06:54:48.549214+0000","last_undegraded":"2024-02-05T06:54:48.549214+0000","last_fullsized":"2024-02-05T06:54:48.549214+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:31.491724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:53:31.491724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:00.042636+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"465'21","reported_seq":693,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.536271+0000","last_change":"2024-02-05T06:52:45.194776+0000","last_active":"2024-02-05T06:54:48.536271+0000","last_peered":"2024-02-05T06:54:48.536271+0000","last_clean":"2024-02-05T06:54:48.536271+0000","last_became_active":"2024-02-05T06:39:13.244124+0000","last_became_peered":"2024-02-05T06:39:13.244124+0000","last_unstale":"2024-02-05T06:54:48.536271+0000","last_undegraded":"2024-02-05T06:54:48.536271+0000","last_fullsized":"2024-02-05T06:54:48.536271+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'15","last_scrub_stamp":"2024-02-05T06:52:45.194645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:52:45.194645+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":607,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.532216+0000","last_change":"2024-02-05T06:53:44.009695+0000","last_active":"2024-02-05T06:54:48.532216+0000","last_peered":"2024-02-05T06:54:48.532216+0000","last_clean":"2024-02-05T06:54:48.532216+0000","last_became_active":"2024-02-05T06:40:27.077701+0000","last_became_peered":"2024-02-05T06:40:27.077701+0000","last_unstale":"2024-02-05T06:54:48.532216+0000","last_undegraded":"2024-02-05T06:54:48.532216+0000","last_fullsized":"2024-02-05T06:54:48.532216+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:44.009607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:33.978941+0000","last_clean_scrub_stamp":"2024-02-05T06:53:44.009607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:13.642503+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":875,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.519612+0000","last_change":"2024-02-05T06:51:30.567546+0000","last_active":"2024-02-05T06:54:48.519612+0000","last_peered":"2024-02-05T06:54:48.519612+0000","last_clean":"2024-02-05T06:54:48.519612+0000","last_became_active":"2024-02-05T06:33:33.593901+0000","last_became_peered":"2024-02-05T06:33:33.593901+0000","last_unstale":"2024-02-05T06:54:48.519612+0000","last_undegraded":"2024-02-05T06:54:48.519612+0000","last_fullsized":"2024-02-05T06:54:48.519612+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:30.567403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:30.567403+0000","last_clean_scrub_stamp":"2024-02-05T06:51:30.567403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"336'2","reported_seq":910,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.518668+0000","last_change":"2024-02-05T06:52:46.298052+0000","last_active":"2024-02-05T06:54:48.518668+0000","last_peered":"2024-02-05T06:54:48.518668+0000","last_clean":"2024-02-05T06:54:48.518668+0000","last_became_active":"2024-02-05T06:41:43.409761+0000","last_became_peered":"2024-02-05T06:41:43.409761+0000","last_unstale":"2024-02-05T06:54:48.518668+0000","last_undegraded":"2024-02-05T06:54:48.518668+0000","last_fullsized":"2024-02-05T06:54:48.518668+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'2","last_scrub_stamp":"2024-02-05T06:52:46.297810+0000","last_deep_scrub":"336'2","last_deep_scrub_stamp":"2024-02-05T06:46:53.500155+0000","last_clean_scrub_stamp":"2024-02-05T06:52:46.297810+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"83'128","reported_seq":1115,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.529440+0000","last_change":"2024-02-05T06:53:47.062697+0000","last_active":"2024-02-05T06:54:48.529440+0000","last_peered":"2024-02-05T06:54:48.529440+0000","last_clean":"2024-02-05T06:54:48.529440+0000","last_became_active":"2024-02-05T06:51:59.292766+0000","last_became_peered":"2024-02-05T06:51:59.292766+0000","last_unstale":"2024-02-05T06:54:48.529440+0000","last_undegraded":"2024-02-05T06:54:48.529440+0000","last_fullsized":"2024-02-05T06:54:48.529440+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:53:47.062582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:53:47.062582+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:10.434191+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"465'32","reported_seq":981,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.552961+0000","last_change":"2024-02-05T06:51:32.530845+0000","last_active":"2024-02-05T06:54:48.552961+0000","last_peered":"2024-02-05T06:54:48.552961+0000","last_clean":"2024-02-05T06:54:48.552961+0000","last_became_active":"2024-02-05T06:40:27.080906+0000","last_became_peered":"2024-02-05T06:40:27.080906+0000","last_unstale":"2024-02-05T06:54:48.552961+0000","last_undegraded":"2024-02-05T06:54:48.552961+0000","last_fullsized":"2024-02-05T06:54:48.552961+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'24","last_scrub_stamp":"2024-02-05T06:51:32.530706+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-02-05T06:35:22.938675+0000","last_clean_scrub_stamp":"2024-02-05T06:51:32.530706+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":523,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.526747+0000","last_change":"2024-02-05T06:54:45.722177+0000","last_active":"2024-02-05T06:54:48.526747+0000","last_peered":"2024-02-05T06:54:48.526747+0000","last_clean":"2024-02-05T06:54:48.526747+0000","last_became_active":"2024-02-05T06:41:43.412654+0000","last_became_peered":"2024-02-05T06:41:43.412654+0000","last_unstale":"2024-02-05T06:54:48.526747+0000","last_undegraded":"2024-02-05T06:54:48.526747+0000","last_fullsized":"2024-02-05T06:54:48.526747+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:54:45.722062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:54:45.722062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:06.239298+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":608,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.531934+0000","last_change":"2024-02-05T06:53:34.074529+0000","last_active":"2024-02-05T06:54:48.531934+0000","last_peered":"2024-02-05T06:54:48.531934+0000","last_clean":"2024-02-05T06:54:48.531934+0000","last_became_active":"2024-02-05T06:40:27.062975+0000","last_became_peered":"2024-02-05T06:40:27.062975+0000","last_unstale":"2024-02-05T06:54:48.531934+0000","last_undegraded":"2024-02-05T06:54:48.531934+0000","last_fullsized":"2024-02-05T06:54:48.531934+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:34.074458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:48:34.628182+0000","last_clean_scrub_stamp":"2024-02-05T06:53:34.074458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":665,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.548399+0000","last_change":"2024-02-05T06:52:40.357841+0000","last_active":"2024-02-05T06:54:48.548399+0000","last_peered":"2024-02-05T06:54:48.548399+0000","last_clean":"2024-02-05T06:54:48.548399+0000","last_became_active":"2024-02-05T06:39:13.245684+0000","last_became_peered":"2024-02-05T06:39:13.245684+0000","last_unstale":"2024-02-05T06:54:48.548399+0000","last_undegraded":"2024-02-05T06:54:48.548399+0000","last_fullsized":"2024-02-05T06:54:48.548399+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:40.356857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:52:40.356857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":268,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"465'12","reported_seq":897,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.549016+0000","last_change":"2024-02-05T06:51:33.570857+0000","last_active":"2024-02-05T06:54:48.549016+0000","last_peered":"2024-02-05T06:54:48.549016+0000","last_clean":"2024-02-05T06:54:48.549016+0000","last_became_active":"2024-02-05T06:33:31.573410+0000","last_became_peered":"2024-02-05T06:33:31.573410+0000","last_unstale":"2024-02-05T06:54:48.549016+0000","last_undegraded":"2024-02-05T06:54:48.549016+0000","last_fullsized":"2024-02-05T06:54:48.549016+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'8","last_scrub_stamp":"2024-02-05T06:51:33.570665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:51:33.570665+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":911,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.519327+0000","last_change":"2024-02-05T06:53:47.483803+0000","last_active":"2024-02-05T06:54:48.519327+0000","last_peered":"2024-02-05T06:54:48.519327+0000","last_clean":"2024-02-05T06:54:48.519327+0000","last_became_active":"2024-02-05T06:41:43.424459+0000","last_became_peered":"2024-02-05T06:41:43.424459+0000","last_unstale":"2024-02-05T06:54:48.519327+0000","last_undegraded":"2024-02-05T06:54:48.519327+0000","last_fullsized":"2024-02-05T06:54:48.519327+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:47.483623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:49.117949+0000","last_clean_scrub_stamp":"2024-02-05T06:53:47.483623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:10.250852+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":946,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.535471+0000","last_change":"2024-02-05T06:53:41.132629+0000","last_active":"2024-02-05T06:54:48.535471+0000","last_peered":"2024-02-05T06:54:48.535471+0000","last_clean":"2024-02-05T06:54:48.535471+0000","last_became_active":"2024-02-05T06:40:27.060112+0000","last_became_peered":"2024-02-05T06:40:27.060112+0000","last_unstale":"2024-02-05T06:54:48.535471+0000","last_undegraded":"2024-02-05T06:54:48.535471+0000","last_fullsized":"2024-02-05T06:54:48.535471+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:41.132523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:48:33.848718+0000","last_clean_scrub_stamp":"2024-02-05T06:53:41.132523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":614,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.532682+0000","last_change":"2024-02-05T06:53:28.246285+0000","last_active":"2024-02-05T06:54:48.532682+0000","last_peered":"2024-02-05T06:54:48.532682+0000","last_clean":"2024-02-05T06:54:48.532682+0000","last_became_active":"2024-02-05T06:44:54.625067+0000","last_became_peered":"2024-02-05T06:44:54.625067+0000","last_unstale":"2024-02-05T06:54:48.532682+0000","last_undegraded":"2024-02-05T06:54:48.532682+0000","last_fullsized":"2024-02-05T06:54:48.532682+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:28.246206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:48.512673+0000","last_clean_scrub_stamp":"2024-02-05T06:53:28.246206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"465'32","reported_seq":1014,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.535659+0000","last_change":"2024-02-05T06:54:43.340495+0000","last_active":"2024-02-05T06:54:48.535659+0000","last_peered":"2024-02-05T06:54:48.535659+0000","last_clean":"2024-02-05T06:54:48.535659+0000","last_became_active":"2024-02-05T06:41:43.419769+0000","last_became_peered":"2024-02-05T06:41:43.419769+0000","last_unstale":"2024-02-05T06:54:48.535659+0000","last_undegraded":"2024-02-05T06:54:48.535659+0000","last_fullsized":"2024-02-05T06:54:48.535659+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'32","last_scrub_stamp":"2024-02-05T06:54:43.340368+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:43:03.221466+0000","last_clean_scrub_stamp":"2024-02-05T06:54:43.340368+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:51.319051+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":515,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.525580+0000","last_change":"2024-02-05T06:53:38.839606+0000","last_active":"2024-02-05T06:54:48.525580+0000","last_peered":"2024-02-05T06:54:48.525580+0000","last_clean":"2024-02-05T06:54:48.525580+0000","last_became_active":"2024-02-05T06:41:43.411043+0000","last_became_peered":"2024-02-05T06:41:43.411043+0000","last_unstale":"2024-02-05T06:54:48.525580+0000","last_undegraded":"2024-02-05T06:54:48.525580+0000","last_fullsized":"2024-02-05T06:54:48.525580+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:38.839486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:53:38.839486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":807,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.548577+0000","last_change":"2024-02-05T06:53:48.540882+0000","last_active":"2024-02-05T06:54:48.548577+0000","last_peered":"2024-02-05T06:54:48.548577+0000","last_clean":"2024-02-05T06:54:48.548577+0000","last_became_active":"2024-02-05T06:40:27.097169+0000","last_became_peered":"2024-02-05T06:40:27.097169+0000","last_unstale":"2024-02-05T06:54:48.548577+0000","last_undegraded":"2024-02-05T06:54:48.548577+0000","last_fullsized":"2024-02-05T06:54:48.548577+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:48.540767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:08.713894+0000","last_clean_scrub_stamp":"2024-02-05T06:53:48.540767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:50.708856+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":926,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.548877+0000","last_change":"2024-02-05T06:54:40.912247+0000","last_active":"2024-02-05T06:54:48.548877+0000","last_peered":"2024-02-05T06:54:48.548877+0000","last_clean":"2024-02-05T06:54:48.548877+0000","last_became_active":"2024-02-05T06:41:43.414638+0000","last_became_peered":"2024-02-05T06:41:43.414638+0000","last_unstale":"2024-02-05T06:54:48.548877+0000","last_undegraded":"2024-02-05T06:54:48.548877+0000","last_fullsized":"2024-02-05T06:54:48.548877+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:54:40.912135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:50:42.316726+0000","last_clean_scrub_stamp":"2024-02-05T06:54:40.912135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:46.882954+0000","scrub_duration":184,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"465'35","reported_seq":991,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.536458+0000","last_change":"2024-02-05T06:53:54.509653+0000","last_active":"2024-02-05T06:54:48.536458+0000","last_peered":"2024-02-05T06:54:48.536458+0000","last_clean":"2024-02-05T06:54:48.536458+0000","last_became_active":"2024-02-05T06:40:27.100610+0000","last_became_peered":"2024-02-05T06:40:27.100610+0000","last_unstale":"2024-02-05T06:54:48.536458+0000","last_undegraded":"2024-02-05T06:54:48.536458+0000","last_fullsized":"2024-02-05T06:54:48.536458+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'35","last_scrub_stamp":"2024-02-05T06:53:54.509362+0000","last_deep_scrub":"465'35","last_deep_scrub_stamp":"2024-02-05T06:53:54.509362+0000","last_clean_scrub_stamp":"2024-02-05T06:53:54.509362+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:00.417269+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":935,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.538874+0000","last_change":"2024-02-05T06:52:50.110926+0000","last_active":"2024-02-05T06:54:48.538874+0000","last_peered":"2024-02-05T06:54:48.538874+0000","last_clean":"2024-02-05T06:54:48.538874+0000","last_became_active":"2024-02-05T06:36:54.958414+0000","last_became_peered":"2024-02-05T06:36:54.958414+0000","last_unstale":"2024-02-05T06:54:48.538874+0000","last_undegraded":"2024-02-05T06:54:48.538874+0000","last_fullsized":"2024-02-05T06:54:48.538874+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:52:50.110793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:52:50.110793+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"314'13","reported_seq":1237,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:50.361999+0000","last_change":"2024-02-05T06:54:50.362023+0000","last_active":"2024-02-05T06:54:50.361999+0000","last_peered":"2024-02-05T06:54:50.361999+0000","last_clean":"2024-02-05T06:54:50.361999+0000","last_became_active":"2024-02-05T06:40:27.063399+0000","last_became_peered":"2024-02-05T06:40:27.063399+0000","last_unstale":"2024-02-05T06:54:50.361999+0000","last_undegraded":"2024-02-05T06:54:50.361999+0000","last_fullsized":"2024-02-05T06:54:50.361999+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"314'13","last_scrub_stamp":"2024-02-05T06:54:50.361942+0000","last_deep_scrub":"314'13","last_deep_scrub_stamp":"2024-02-05T06:44:53.670923+0000","last_clean_scrub_stamp":"2024-02-05T06:54:50.361942+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:50.529427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"325'2","reported_seq":614,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.533017+0000","last_change":"2024-02-05T06:52:48.168560+0000","last_active":"2024-02-05T06:54:48.533017+0000","last_peered":"2024-02-05T06:54:48.533017+0000","last_clean":"2024-02-05T06:54:48.533017+0000","last_became_active":"2024-02-05T06:41:43.418640+0000","last_became_peered":"2024-02-05T06:41:43.418640+0000","last_unstale":"2024-02-05T06:54:48.533017+0000","last_undegraded":"2024-02-05T06:54:48.533017+0000","last_fullsized":"2024-02-05T06:54:48.533017+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'2","last_scrub_stamp":"2024-02-05T06:52:48.168445+0000","last_deep_scrub":"325'2","last_deep_scrub_stamp":"2024-02-05T06:45:46.242415+0000","last_clean_scrub_stamp":"2024-02-05T06:52:48.168445+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"465'29","reported_seq":986,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.548091+0000","last_change":"2024-02-05T06:52:37.441014+0000","last_active":"2024-02-05T06:54:48.548091+0000","last_peered":"2024-02-05T06:54:48.548091+0000","last_clean":"2024-02-05T06:54:48.548091+0000","last_became_active":"2024-02-05T06:36:54.954880+0000","last_became_peered":"2024-02-05T06:36:54.954880+0000","last_unstale":"2024-02-05T06:54:48.548091+0000","last_undegraded":"2024-02-05T06:54:48.548091+0000","last_fullsized":"2024-02-05T06:54:48.548091+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'23","last_scrub_stamp":"2024-02-05T06:52:37.440858+0000","last_deep_scrub":"445'23","last_deep_scrub_stamp":"2024-02-05T06:52:37.440858+0000","last_clean_scrub_stamp":"2024-02-05T06:52:37.440858+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":428,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.539127+0000","last_change":"2024-02-05T06:53:51.717338+0000","last_active":"2024-02-05T06:54:48.539127+0000","last_peered":"2024-02-05T06:54:48.539127+0000","last_clean":"2024-02-05T06:54:48.539127+0000","last_became_active":"2024-02-05T06:45:09.786579+0000","last_became_peered":"2024-02-05T06:45:09.786579+0000","last_unstale":"2024-02-05T06:54:48.539127+0000","last_undegraded":"2024-02-05T06:54:48.539127+0000","last_fullsized":"2024-02-05T06:54:48.539127+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:51.717205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:53:51.717205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:02.032086+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":688,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.535217+0000","last_change":"2024-02-05T06:53:51.117144+0000","last_active":"2024-02-05T06:54:48.535217+0000","last_peered":"2024-02-05T06:54:48.535217+0000","last_clean":"2024-02-05T06:54:48.535217+0000","last_became_active":"2024-02-05T06:39:13.248618+0000","last_became_peered":"2024-02-05T06:39:13.248618+0000","last_unstale":"2024-02-05T06:54:48.535217+0000","last_undegraded":"2024-02-05T06:54:48.535217+0000","last_fullsized":"2024-02-05T06:54:48.535217+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:51.116923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:34.551395+0000","last_clean_scrub_stamp":"2024-02-05T06:53:51.116923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:19.941766+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":892,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.854377+0000","last_change":"2024-02-05T06:54:48.854417+0000","last_active":"2024-02-05T06:54:48.854377+0000","last_peered":"2024-02-05T06:54:48.854377+0000","last_clean":"2024-02-05T06:54:48.854377+0000","last_became_active":"2024-02-05T06:41:43.411442+0000","last_became_peered":"2024-02-05T06:41:43.411442+0000","last_unstale":"2024-02-05T06:54:48.854377+0000","last_undegraded":"2024-02-05T06:54:48.854377+0000","last_fullsized":"2024-02-05T06:54:48.854377+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:54:48.854297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:54:48.854297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:12.956719+0000","scrub_duration":299,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"465'46","reported_seq":1000,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.550399+0000","last_change":"2024-02-05T06:53:33.046693+0000","last_active":"2024-02-05T06:54:48.550399+0000","last_peered":"2024-02-05T06:54:48.550399+0000","last_clean":"2024-02-05T06:54:48.550399+0000","last_became_active":"2024-02-05T06:33:31.569763+0000","last_became_peered":"2024-02-05T06:33:31.569763+0000","last_unstale":"2024-02-05T06:54:48.550399+0000","last_undegraded":"2024-02-05T06:54:48.550399+0000","last_fullsized":"2024-02-05T06:54:48.550399+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'34","last_scrub_stamp":"2024-02-05T06:53:33.046569+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:34:35.234636+0000","last_clean_scrub_stamp":"2024-02-05T06:53:33.046569+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"281'2","reported_seq":946,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.536769+0000","last_change":"2024-02-05T06:53:30.452677+0000","last_active":"2024-02-05T06:54:48.536769+0000","last_peered":"2024-02-05T06:54:48.536769+0000","last_clean":"2024-02-05T06:54:48.536769+0000","last_became_active":"2024-02-05T06:40:27.073360+0000","last_became_peered":"2024-02-05T06:40:27.073360+0000","last_unstale":"2024-02-05T06:54:48.536769+0000","last_undegraded":"2024-02-05T06:54:48.536769+0000","last_fullsized":"2024-02-05T06:54:48.536769+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'2","last_scrub_stamp":"2024-02-05T06:53:30.452604+0000","last_deep_scrub":"281'2","last_deep_scrub_stamp":"2024-02-05T06:47:50.442159+0000","last_clean_scrub_stamp":"2024-02-05T06:53:30.452604+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:49.493466+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":910,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.548730+0000","last_change":"2024-02-05T06:53:52.698534+0000","last_active":"2024-02-05T06:54:48.548730+0000","last_peered":"2024-02-05T06:54:48.548730+0000","last_clean":"2024-02-05T06:54:48.548730+0000","last_became_active":"2024-02-05T06:36:54.958069+0000","last_became_peered":"2024-02-05T06:36:54.958069+0000","last_unstale":"2024-02-05T06:54:48.548730+0000","last_undegraded":"2024-02-05T06:54:48.548730+0000","last_fullsized":"2024-02-05T06:54:48.548730+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:52.698426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:38.793760+0000","last_clean_scrub_stamp":"2024-02-05T06:53:52.698426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:13.013123+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":656,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.535594+0000","last_change":"2024-02-05T06:53:36.022199+0000","last_active":"2024-02-05T06:54:48.535594+0000","last_peered":"2024-02-05T06:54:48.535594+0000","last_clean":"2024-02-05T06:54:48.535594+0000","last_became_active":"2024-02-05T06:45:09.770902+0000","last_became_peered":"2024-02-05T06:45:09.770902+0000","last_unstale":"2024-02-05T06:54:48.535594+0000","last_undegraded":"2024-02-05T06:54:48.535594+0000","last_fullsized":"2024-02-05T06:54:48.535594+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:36.022090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:53:36.022090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1b","version":"465'9","reported_seq":748,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.519429+0000","last_change":"2024-02-05T06:53:27.348848+0000","last_active":"2024-02-05T06:54:48.519429+0000","last_peered":"2024-02-05T06:54:48.519429+0000","last_clean":"2024-02-05T06:54:48.519429+0000","last_became_active":"2024-02-05T06:45:09.808663+0000","last_became_peered":"2024-02-05T06:45:09.808663+0000","last_unstale":"2024-02-05T06:54:48.519429+0000","last_undegraded":"2024-02-05T06:54:48.519429+0000","last_fullsized":"2024-02-05T06:54:48.519429+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'7","last_scrub_stamp":"2024-02-05T06:53:27.348728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:53:27.348728+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:51.281890+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":515,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.526382+0000","last_change":"2024-02-05T06:52:43.828744+0000","last_active":"2024-02-05T06:54:48.526382+0000","last_peered":"2024-02-05T06:54:48.526382+0000","last_clean":"2024-02-05T06:54:48.526382+0000","last_became_active":"2024-02-05T06:41:43.408428+0000","last_became_peered":"2024-02-05T06:41:43.408428+0000","last_unstale":"2024-02-05T06:54:48.526382+0000","last_undegraded":"2024-02-05T06:54:48.526382+0000","last_fullsized":"2024-02-05T06:54:48.526382+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:43.828638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:43.828638+0000","last_clean_scrub_stamp":"2024-02-05T06:52:43.828638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":615,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.532756+0000","last_change":"2024-02-05T06:53:58.717874+0000","last_active":"2024-02-05T06:54:48.532756+0000","last_peered":"2024-02-05T06:54:48.532756+0000","last_clean":"2024-02-05T06:54:48.532756+0000","last_became_active":"2024-02-05T06:40:27.071758+0000","last_became_peered":"2024-02-05T06:40:27.071758+0000","last_unstale":"2024-02-05T06:54:48.532756+0000","last_undegraded":"2024-02-05T06:54:48.532756+0000","last_fullsized":"2024-02-05T06:54:48.532756+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:58.717756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:37.152741+0000","last_clean_scrub_stamp":"2024-02-05T06:53:58.717756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:16.156426+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":515,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.525477+0000","last_change":"2024-02-05T06:53:34.839643+0000","last_active":"2024-02-05T06:54:48.525477+0000","last_peered":"2024-02-05T06:54:48.525477+0000","last_clean":"2024-02-05T06:54:48.525477+0000","last_became_active":"2024-02-05T06:41:43.407404+0000","last_became_peered":"2024-02-05T06:41:43.407404+0000","last_unstale":"2024-02-05T06:54:48.525477+0000","last_undegraded":"2024-02-05T06:54:48.525477+0000","last_fullsized":"2024-02-05T06:54:48.525477+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:34.839509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:21.998850+0000","last_clean_scrub_stamp":"2024-02-05T06:53:34.839509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:54.888016+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"465'25","reported_seq":756,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.552532+0000","last_change":"2024-02-05T06:52:31.409340+0000","last_active":"2024-02-05T06:54:48.552532+0000","last_peered":"2024-02-05T06:54:48.552532+0000","last_clean":"2024-02-05T06:54:48.552532+0000","last_became_active":"2024-02-05T06:45:09.787626+0000","last_became_peered":"2024-02-05T06:45:09.787626+0000","last_unstale":"2024-02-05T06:54:48.552532+0000","last_undegraded":"2024-02-05T06:54:48.552532+0000","last_fullsized":"2024-02-05T06:54:48.552532+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'19","last_scrub_stamp":"2024-02-05T06:52:31.409196+0000","last_deep_scrub":"92'13","last_deep_scrub_stamp":"2024-02-05T06:34:46.542893+0000","last_clean_scrub_stamp":"2024-02-05T06:52:31.409196+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":515,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.526493+0000","last_change":"2024-02-05T06:54:36.822689+0000","last_active":"2024-02-05T06:54:48.526493+0000","last_peered":"2024-02-05T06:54:48.526493+0000","last_clean":"2024-02-05T06:54:48.526493+0000","last_became_active":"2024-02-05T06:41:43.403746+0000","last_became_peered":"2024-02-05T06:41:43.403746+0000","last_unstale":"2024-02-05T06:54:48.526493+0000","last_undegraded":"2024-02-05T06:54:48.526493+0000","last_fullsized":"2024-02-05T06:54:48.526493+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:54:36.822620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:44.081318+0000","last_clean_scrub_stamp":"2024-02-05T06:54:36.822620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:49.760597+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":455,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.548545+0000","last_change":"2024-02-05T06:53:34.584200+0000","last_active":"2024-02-05T06:54:48.548545+0000","last_peered":"2024-02-05T06:54:48.548545+0000","last_clean":"2024-02-05T06:54:48.548545+0000","last_became_active":"2024-02-05T06:42:33.129959+0000","last_became_peered":"2024-02-05T06:42:33.129959+0000","last_unstale":"2024-02-05T06:54:48.548545+0000","last_undegraded":"2024-02-05T06:54:48.548545+0000","last_fullsized":"2024-02-05T06:54:48.548545+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:34.583981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:45:51.217900+0000","last_clean_scrub_stamp":"2024-02-05T06:53:34.583981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:55.085329+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":899,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.518872+0000","last_change":"2024-02-05T06:52:51.154037+0000","last_active":"2024-02-05T06:54:48.518872+0000","last_peered":"2024-02-05T06:54:48.518872+0000","last_clean":"2024-02-05T06:54:48.518872+0000","last_became_active":"2024-02-05T06:40:27.068888+0000","last_became_peered":"2024-02-05T06:40:27.068888+0000","last_unstale":"2024-02-05T06:54:48.518872+0000","last_undegraded":"2024-02-05T06:54:48.518872+0000","last_fullsized":"2024-02-05T06:54:48.518872+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:52:51.153442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:44:55.125921+0000","last_clean_scrub_stamp":"2024-02-05T06:52:51.153442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"465'46","reported_seq":954,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.553055+0000","last_change":"2024-02-05T06:52:27.389525+0000","last_active":"2024-02-05T06:54:48.553055+0000","last_peered":"2024-02-05T06:54:48.553055+0000","last_clean":"2024-02-05T06:54:48.553055+0000","last_became_active":"2024-02-05T06:33:31.572544+0000","last_became_peered":"2024-02-05T06:33:31.572544+0000","last_unstale":"2024-02-05T06:54:48.553055+0000","last_undegraded":"2024-02-05T06:54:48.553055+0000","last_fullsized":"2024-02-05T06:54:48.553055+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'34","last_scrub_stamp":"2024-02-05T06:52:27.389433+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:39:35.433539+0000","last_clean_scrub_stamp":"2024-02-05T06:52:27.389433+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":892,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.548250+0000","last_change":"2024-02-05T06:51:23.547319+0000","last_active":"2024-02-05T06:54:48.548250+0000","last_peered":"2024-02-05T06:54:48.548250+0000","last_clean":"2024-02-05T06:54:48.548250+0000","last_became_active":"2024-02-05T06:33:29.553864+0000","last_became_peered":"2024-02-05T06:33:29.553864+0000","last_unstale":"2024-02-05T06:54:48.548250+0000","last_undegraded":"2024-02-05T06:54:48.548250+0000","last_fullsized":"2024-02-05T06:54:48.548250+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:51:23.547188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:51:23.547188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":421,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.534845+0000","last_change":"2024-02-05T06:53:37.018077+0000","last_active":"2024-02-05T06:54:48.534845+0000","last_peered":"2024-02-05T06:54:48.534845+0000","last_clean":"2024-02-05T06:54:48.534845+0000","last_became_active":"2024-02-05T06:45:09.775037+0000","last_became_peered":"2024-02-05T06:45:09.775037+0000","last_unstale":"2024-02-05T06:54:48.534845+0000","last_undegraded":"2024-02-05T06:54:48.534845+0000","last_fullsized":"2024-02-05T06:54:48.534845+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:37.018004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:51.388156+0000","last_clean_scrub_stamp":"2024-02-05T06:53:37.018004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":890,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.548453+0000","last_change":"2024-02-05T06:53:30.666137+0000","last_active":"2024-02-05T06:54:48.548453+0000","last_peered":"2024-02-05T06:54:48.548453+0000","last_clean":"2024-02-05T06:54:48.548453+0000","last_became_active":"2024-02-05T06:40:27.077485+0000","last_became_peered":"2024-02-05T06:40:27.077485+0000","last_unstale":"2024-02-05T06:54:48.548453+0000","last_undegraded":"2024-02-05T06:54:48.548453+0000","last_fullsized":"2024-02-05T06:54:48.548453+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:30.666019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:53:30.666019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:54:55.573254+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"465'25","reported_seq":848,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.515816+0000","last_change":"2024-02-05T06:54:45.143674+0000","last_active":"2024-02-05T06:54:48.515816+0000","last_peered":"2024-02-05T06:54:48.515816+0000","last_clean":"2024-02-05T06:54:48.515816+0000","last_became_active":"2024-02-05T06:36:54.959330+0000","last_became_peered":"2024-02-05T06:36:54.959330+0000","last_unstale":"2024-02-05T06:54:48.515816+0000","last_undegraded":"2024-02-05T06:54:48.515816+0000","last_fullsized":"2024-02-05T06:54:48.515816+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'25","last_scrub_stamp":"2024-02-05T06:54:45.143519+0000","last_deep_scrub":"247'13","last_deep_scrub_stamp":"2024-02-05T06:41:42.473424+0000","last_clean_scrub_stamp":"2024-02-05T06:54:45.143519+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:47.089807+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":31,"num_read_kb":25,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":913,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.519126+0000","last_change":"2024-02-05T06:53:48.501549+0000","last_active":"2024-02-05T06:54:48.519126+0000","last_peered":"2024-02-05T06:54:48.519126+0000","last_clean":"2024-02-05T06:54:48.519126+0000","last_became_active":"2024-02-05T06:41:43.421315+0000","last_became_peered":"2024-02-05T06:41:43.421315+0000","last_unstale":"2024-02-05T06:54:48.519126+0000","last_undegraded":"2024-02-05T06:54:48.519126+0000","last_fullsized":"2024-02-05T06:54:48.519126+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:48.501421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:53:48.501421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:01.128496+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":922,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:48.538902+0000","last_change":"2024-02-05T06:53:46.710740+0000","last_active":"2024-02-05T06:54:48.538902+0000","last_peered":"2024-02-05T06:54:48.538902+0000","last_clean":"2024-02-05T06:54:48.538902+0000","last_became_active":"2024-02-05T06:41:43.420342+0000","last_became_peered":"2024-02-05T06:41:43.420342+0000","last_unstale":"2024-02-05T06:54:48.538902+0000","last_undegraded":"2024-02-05T06:54:48.538902+0000","last_fullsized":"2024-02-05T06:54:48.538902+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:53:46.710631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:06.525012+0000","last_clean_scrub_stamp":"2024-02-05T06:53:46.710631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:55:09.216440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":659,"reported_epoch":478,"state":"active+clean","last_fresh":"2024-02-05T06:54:49.773378+0000","last_change":"2024-02-05T06:54:49.773416+0000","last_active":"2024-02-05T06:54:49.773378+0000","last_peered":"2024-02-05T06:54:49.773378+0000","last_clean":"2024-02-05T06:54:49.773378+0000","last_became_active":"2024-02-05T06:45:09.787134+0000","last_became_peered":"2024-02-05T06:45:09.787134+0000","last_unstale":"2024-02-05T06:54:49.773378+0000","last_undegraded":"2024-02-05T06:54:49.773378+0000","last_fullsized":"2024-02-05T06:54:49.773378+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:54:49.773291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:54:49.773291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:18.933128+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":253928,"num_write_kb":8125633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76028,"ondisk_log_size":76028,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":870,"num_read_kb":865,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2688,"ondisk_log_size":2688,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1535,"num_read_kb":1292,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":373,"num_bytes_recovered":8234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1086,"ondisk_log_size":1086,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103428,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":838552,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":837758,"kb_avail":92906600,"statfs":{"total":95995035648,"available":95136358400,"internally_reserved":0,"allocated":802816,"data_stored":424173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":857864650},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:54:15 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.2589999999999999,"15min":1.2190000000000001},"min":{"1min":0.26400000000000001,"5min":0.23400000000000001,"15min":0.21299999999999999},"max":{"1min":4.282,"5min":5.2329999999999997,"15min":32.567},"last":0.57599999999999996},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.52,"15min":1.385},"min":{"1min":0.249,"5min":0.249,"15min":0.222},"max":{"1min":3.9220000000000002,"5min":41.109999999999999,"15min":41.109999999999999},"last":0.42299999999999999}]},{"osd":1,"last update":"Mon Feb 5 06:54:15 2024","interfaces":[{"interface":"back","average":{"1min":1.7270000000000001,"5min":1.669,"15min":1.8440000000000001},"min":{"1min":0.27700000000000002,"5min":0.26000000000000001,"15min":0.25},"max":{"1min":5.3170000000000002,"5min":23.716999999999999,"15min":84.216999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":1.6379999999999999,"5min":1.9179999999999999,"15min":1.9259999999999999},"min":{"1min":0.29599999999999999,"5min":0.24299999999999999,"15min":0.22800000000000001},"max":{"1min":4.8689999999999998,"5min":41.167999999999999,"15min":84.126999999999995},"last":0.58599999999999997}]},{"osd":2,"last update":"Mon Feb 5 06:54:15 2024","interfaces":[{"interface":"back","average":{"1min":1.6339999999999999,"5min":1.5860000000000001,"15min":1.359},"min":{"1min":0.28799999999999998,"5min":0.249,"15min":0.249},"max":{"1min":4.6059999999999999,"5min":28.704999999999998,"15min":28.704999999999998},"last":0.66500000000000004},{"interface":"front","average":{"1min":1.2829999999999999,"5min":1.6479999999999999,"15min":1.3009999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.26800000000000002},"max":{"1min":4.1379999999999999,"5min":23.678999999999998,"15min":23.678999999999998},"last":0.504}]},{"osd":3,"last update":"Mon Feb 5 06:54:15 2024","interfaces":[{"interface":"back","average":{"1min":3.5760000000000001,"5min":1.954,"15min":1.976},"min":{"1min":0.312,"5min":0.29199999999999998,"15min":0.28899999999999998},"max":{"1min":34.374000000000002,"5min":34.374000000000002,"15min":73.283000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":3.7469999999999999,"5min":2.383,"15min":2.0230000000000001},"min":{"1min":0.35299999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":34.267000000000003,"5min":41.219000000000001,"15min":73.323999999999998},"last":0.59499999999999997}]},{"osd":4,"last update":"Mon Feb 5 06:54:15 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.089,"15min":0.84499999999999997},"min":{"1min":0.29499999999999998,"5min":0.252,"15min":0.20200000000000001},"max":{"1min":3.8879999999999999,"5min":5.4470000000000001,"15min":6.1980000000000004},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.995,"5min":1.4179999999999999,"15min":0.94399999999999995},"min":{"1min":0.26400000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":4.1870000000000003,"5min":41.270000000000003,"15min":41.270000000000003},"last":0.29599999999999999}]},{"osd":5,"last update":"Mon Feb 5 06:54:15 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.3280000000000001,"15min":1.4059999999999999},"min":{"1min":0.27300000000000002,"5min":0.252,"15min":0.251},"max":{"1min":4.1280000000000001,"5min":41.255000000000003,"15min":109.482},"last":0.316},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.3839999999999999,"15min":1.49},"min":{"1min":0.36499999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":4.0039999999999996,"5min":41.203000000000003,"15min":109.56399999999999},"last":0.35799999999999998}]},{"osd":6,"last update":"Mon Feb 5 06:54:15 2024","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.625,"15min":1.554},"min":{"1min":0.317,"5min":0.317,"15min":0.28499999999999998},"max":{"1min":4.8419999999999996,"5min":41.326999999999998,"15min":113.914},"last":0.32900000000000001},{"interface":"front","average":{"1min":1.107,"5min":1.518,"15min":1.4690000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.307},"max":{"1min":4.202,"5min":41.384,"15min":113.212},"last":0.34200000000000003}]}]},{"osd":6,"up_from":49,"seq":210453397770,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":772224,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":771134,"kb_avail":92972928,"statfs":{"total":95995035648,"available":95204278272,"internally_reserved":0,"allocated":1105920,"data_stored":750404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":789641670},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:54:39 2024","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.49,"15min":2.3130000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.224},"max":{"1min":3.8300000000000001,"5min":15.401,"15min":98.918999999999997},"last":0.29899999999999999},{"interface":"front","average":{"1min":1.752,"5min":1.7190000000000001,"15min":2.4620000000000002},"min":{"1min":0.248,"5min":0.223,"15min":0.223},"max":{"1min":6.226,"5min":15.388999999999999,"15min":98.775000000000006},"last":1.3340000000000001}]},{"osd":1,"last update":"Mon Feb 5 06:54:39 2024","interfaces":[{"interface":"back","average":{"1min":1.579,"5min":1.7969999999999999,"15min":2.2130000000000001},"min":{"1min":0.27900000000000003,"5min":0.254,"15min":0.24099999999999999},"max":{"1min":6.0549999999999997,"5min":47.348999999999997,"15min":98.843999999999994},"last":1.694},{"interface":"front","average":{"1min":1.645,"5min":1.806,"15min":2.2519999999999998},"min":{"1min":0.26700000000000002,"5min":0.24099999999999999,"15min":0.23100000000000001},"max":{"1min":4.1580000000000004,"5min":47.777999999999999,"15min":101.637},"last":0.25900000000000001}]},{"osd":2,"last update":"Mon Feb 5 06:54:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.4099999999999999,"15min":2.0499999999999998},"min":{"1min":0.27700000000000002,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":3.9100000000000001,"5min":10.446,"15min":85.617000000000004},"last":1.28},{"interface":"front","average":{"1min":1.4750000000000001,"5min":1.5660000000000001,"15min":2.0830000000000002},"min":{"1min":0.33100000000000002,"5min":0.27300000000000002,"15min":0.27000000000000002},"max":{"1min":6.8540000000000001,"5min":16.516999999999999,"15min":85.335999999999999},"last":1.3879999999999999}]},{"osd":3,"last update":"Mon Feb 5 06:54:39 2024","interfaces":[{"interface":"back","average":{"1min":1.5489999999999999,"5min":1.5069999999999999,"15min":2.1699999999999999},"min":{"1min":0.32100000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":5.0839999999999996,"5min":14.648,"15min":98.888999999999996},"last":4.3150000000000004},{"interface":"front","average":{"1min":1.976,"5min":1.5880000000000001,"15min":2.2189999999999999},"min":{"1min":0.443,"5min":0.313,"15min":0.27800000000000002},"max":{"1min":6.7510000000000003,"5min":14.6,"15min":98.969999999999999},"last":0.32900000000000001}]},{"osd":4,"last update":"Mon Feb 5 06:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.85999999999999999,"15min":0.75700000000000001},"min":{"1min":0.27500000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":5.4720000000000004,"5min":5.5940000000000003,"15min":5.5940000000000003},"last":1.3},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.89200000000000002,"15min":0.77100000000000002},"min":{"1min":0.35999999999999999,"5min":0.312,"15min":0.24299999999999999},"max":{"1min":5.3109999999999999,"5min":5.9500000000000002,"15min":5.9500000000000002},"last":1.7270000000000001}]},{"osd":5,"last update":"Mon Feb 5 06:54:39 2024","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.97699999999999998,"15min":1.012},"min":{"1min":0.315,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":5.4320000000000004,"5min":5.4320000000000004,"15min":36.548999999999999},"last":1.3660000000000001},{"interface":"front","average":{"1min":1.2729999999999999,"5min":0.85799999999999998,"15min":0.98099999999999998},"min":{"1min":0.39500000000000002,"5min":0.32600000000000001,"15min":0.28699999999999998},"max":{"1min":10.345000000000001,"5min":10.345000000000001,"15min":36.433},"last":1.7509999999999999}]},{"osd":7,"last update":"Mon Feb 5 06:53:50 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.75,"15min":0.85299999999999998},"min":{"1min":0.31,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.177,"5min":3.6200000000000001,"15min":38.436999999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":1.036,"15min":0.95399999999999996},"min":{"1min":0.35899999999999999,"5min":0.29899999999999999,"15min":0.29699999999999999},"max":{"1min":1.244,"5min":5.3280000000000003,"15min":36.633000000000003},"last":1.4099999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575139,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":938552,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":92806600,"statfs":{"total":95995035648,"available":95033958400,"internally_reserved":0,"allocated":835584,"data_stored":424956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":960231877},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:54:34 2024","interfaces":[{"interface":"back","average":{"1min":1.75,"5min":2.9399999999999999,"15min":2.1480000000000001},"min":{"1min":0.35499999999999998,"5min":0.20899999999999999,"15min":0.183},"max":{"1min":5.0709999999999997,"5min":93.488,"15min":93.488},"last":2.3799999999999999},{"interface":"front","average":{"1min":2.2770000000000001,"5min":3.0859999999999999,"15min":2.3599999999999999},"min":{"1min":0.27800000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.9440000000000008,"5min":93.689999999999998,"15min":93.689999999999998},"last":5.2469999999999999}]},{"osd":2,"last update":"Mon Feb 5 06:53:55 2024","interfaces":[{"interface":"back","average":{"1min":6.649,"5min":3.069,"15min":2.1179999999999999},"min":{"1min":0.38900000000000001,"5min":0.184,"15min":0.184},"max":{"1min":93.352999999999994,"5min":93.352999999999994,"15min":93.352999999999994},"last":2.4209999999999998},{"interface":"front","average":{"1min":6.5679999999999996,"5min":2.931,"15min":1.8959999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.22600000000000001},"max":{"1min":91.543000000000006,"5min":91.543000000000006,"15min":91.543000000000006},"last":1.083}]},{"osd":3,"last update":"Mon Feb 5 06:54:14 2024","interfaces":[{"interface":"back","average":{"1min":6.718,"5min":2.7370000000000001,"15min":2.246},"min":{"1min":0.26700000000000002,"5min":0.25900000000000001,"15min":0.23499999999999999},"max":{"1min":93.459999999999994,"5min":93.459999999999994,"15min":93.459999999999994},"last":5.3399999999999999},{"interface":"front","average":{"1min":6.7690000000000001,"5min":2.8500000000000001,"15min":2.1139999999999999},"min":{"1min":0.39300000000000002,"5min":0.255,"15min":0.23699999999999999},"max":{"1min":93.406000000000006,"5min":93.406000000000006,"15min":93.406000000000006},"last":1.4059999999999999}]},{"osd":4,"last update":"Mon Feb 5 06:54:39 2024","interfaces":[{"interface":"back","average":{"1min":1.8169999999999999,"5min":1.407,"15min":1.1739999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.27700000000000002},"max":{"1min":8.0820000000000007,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.34899999999999998},{"interface":"front","average":{"1min":1.333,"5min":1.6859999999999999,"15min":1.454},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.26300000000000001},"max":{"1min":4.0289999999999999,"5min":9.8680000000000003,"15min":10.419},"last":5.5250000000000004}]},{"osd":5,"last update":"Mon Feb 5 06:54:14 2024","interfaces":[{"interface":"back","average":{"1min":1.6919999999999999,"5min":1.6140000000000001,"15min":1.4830000000000001},"min":{"1min":0.40200000000000002,"5min":0.34999999999999998,"15min":0.29399999999999998},"max":{"1min":5.851,"5min":11.058,"15min":43.414999999999999},"last":1.5349999999999999},{"interface":"front","average":{"1min":1.663,"5min":1.206,"15min":1.216},"min":{"1min":0.39400000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":8.1080000000000005,"5min":8.1669999999999998,"15min":8.1669999999999998},"last":0.48599999999999999}]},{"osd":6,"last update":"Mon Feb 5 06:54:34 2024","interfaces":[{"interface":"back","average":{"1min":2.0619999999999998,"5min":1.698,"15min":1.399},"min":{"1min":0.40300000000000002,"5min":0.38500000000000001,"15min":0.34799999999999998},"max":{"1min":8.1329999999999991,"5min":19.827999999999999,"15min":23.902999999999999},"last":3.2389999999999999},{"interface":"front","average":{"1min":1.8260000000000001,"5min":2.0230000000000001,"15min":1.643},"min":{"1min":0.42099999999999999,"5min":0.40999999999999998,"15min":0.35599999999999998},"max":{"1min":5.4240000000000004,"5min":19.526,"15min":43.475000000000001},"last":3.2850000000000001}]},{"osd":7,"last update":"Mon Feb 5 06:53:55 2024","interfaces":[{"interface":"back","average":{"1min":1.7809999999999999,"5min":1.405,"15min":1.349},"min":{"1min":0.41199999999999998,"5min":0.377,"15min":0.32900000000000001},"max":{"1min":8.1600000000000001,"5min":8.1600000000000001,"15min":8.1600000000000001},"last":1.052},{"interface":"front","average":{"1min":2.4470000000000001,"5min":1.645,"15min":1.5349999999999999},"min":{"1min":0.43099999999999999,"5min":0.36599999999999999,"15min":0.35499999999999998},"max":{"1min":9.8230000000000004,"5min":9.8230000000000004,"15min":10.481},"last":5.2969999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738665,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":798632,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":797502,"kb_avail":92946520,"statfs":{"total":95995035648,"available":95177236480,"internally_reserved":0,"allocated":1146880,"data_stored":751829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":816642501},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 06:54:25 2024","interfaces":[{"interface":"back","average":{"1min":2.444,"5min":2.085,"15min":1.7390000000000001},"min":{"1min":0.32800000000000001,"5min":0.188,"15min":0.182},"max":{"1min":16.963000000000001,"5min":38.68,"15min":38.68},"last":0.28199999999999997},{"interface":"front","average":{"1min":1.99,"5min":2.532,"15min":1.962},"min":{"1min":0.218,"5min":0.19500000000000001,"15min":0.18099999999999999},"max":{"1min":16.263999999999999,"5min":38.003,"15min":38.003},"last":0.30199999999999999}]},{"osd":2,"last update":"Mon Feb 5 06:54:50 2024","interfaces":[{"interface":"back","average":{"1min":1.554,"5min":2.0609999999999999,"15min":1.7170000000000001},"min":{"1min":0.27300000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":3.5249999999999999,"5min":27.312000000000001,"15min":27.312000000000001},"last":0.36899999999999999},{"interface":"front","average":{"1min":1.5049999999999999,"5min":2.0529999999999999,"15min":1.5980000000000001},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":4.2359999999999998,"5min":27.359000000000002,"15min":27.359000000000002},"last":0.41299999999999998}]},{"osd":3,"last update":"Mon Feb 5 06:54:17 2024","interfaces":[{"interface":"back","average":{"1min":2.3140000000000001,"5min":1.889,"15min":1.8009999999999999},"min":{"1min":0.27900000000000003,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":13.551,"5min":31.777999999999999,"15min":43.302},"last":0.38900000000000001},{"interface":"front","average":{"1min":2.6299999999999999,"5min":2.359,"15min":2.1040000000000001},"min":{"1min":0.32300000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":13.567,"5min":31.716000000000001,"15min":43.651000000000003},"last":0.33600000000000002}]},{"osd":4,"last update":"Mon Feb 5 06:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.95399999999999996,"5min":1.3999999999999999,"15min":1.504},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.27200000000000002},"max":{"1min":2.786,"5min":9.157,"15min":40.789999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":1.6379999999999999,"5min":1.252,"15min":1.4159999999999999},"min":{"1min":0.40799999999999997,"5min":0.316,"15min":0.316},"max":{"1min":5.681,"5min":5.8289999999999997,"15min":40.773000000000003},"last":0.59499999999999997}]},{"osd":5,"last update":"Mon Feb 5 06:54:09 2024","interfaces":[{"interface":"back","average":{"1min":1.4770000000000001,"5min":1.5609999999999999,"15min":1.635},"min":{"1min":0.379,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":4.6799999999999997,"5min":7.1239999999999997,"15min":48.841999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.746,"15min":1.621},"min":{"1min":0.35699999999999998,"5min":0.28899999999999998,"15min":0.28599999999999998},"max":{"1min":4.7279999999999998,"5min":11.156000000000001,"15min":48.450000000000003},"last":0.73699999999999999}]},{"osd":6,"last update":"Mon Feb 5 06:54:25 2024","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.1659999999999999,"15min":1.7849999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":4.8609999999999998,"5min":4.8609999999999998,"15min":136.76300000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.7370000000000001,"15min":2.0409999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":4.1689999999999996,"5min":7.0720000000000001,"15min":135.92400000000001},"last":0.66300000000000003}]},{"osd":7,"last update":"Mon Feb 5 06:53:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.7470000000000001,"15min":1.4830000000000001},"min":{"1min":0.40699999999999997,"5min":0.35499999999999998,"15min":0.31900000000000001},"max":{"1min":5.2249999999999996,"5min":10.989000000000001,"15min":17.210000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.222,"15min":1.276},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.34899999999999998},"max":{"1min":3.6040000000000001,"5min":6.383,"15min":18.533999999999999},"last":0.64400000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411614,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":699988,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":93045164,"statfs":{"total":95995035648,"available":95278247936,"internally_reserved":0,"allocated":1056768,"data_stored":686730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":715717061},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:53:55 2024","interfaces":[{"interface":"back","average":{"1min":1.7989999999999999,"5min":3.6970000000000001,"15min":2.298},"min":{"1min":0.31,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.7489999999999997,"5min":101.67700000000001,"15min":101.67700000000001},"last":3.004},{"interface":"front","average":{"1min":0.99299999999999999,"5min":3.302,"15min":2.2290000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.17699999999999999},"max":{"1min":3.331,"5min":101.27800000000001,"15min":101.27800000000001},"last":0.215}]},{"osd":1,"last update":"Mon Feb 5 06:53:55 2024","interfaces":[{"interface":"back","average":{"1min":2.1389999999999998,"5min":2.1920000000000002,"15min":1.7969999999999999},"min":{"1min":0.19500000000000001,"5min":0.185,"15min":0.185},"max":{"1min":6.7169999999999996,"5min":24.093,"15min":24.093},"last":2.2410000000000001},{"interface":"front","average":{"1min":1.76,"5min":1.7969999999999999,"15min":1.5109999999999999},"min":{"1min":0.36499999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":6.6870000000000003,"5min":27.114000000000001,"15min":27.114000000000001},"last":2.5739999999999998}]},{"osd":3,"last update":"Mon Feb 5 06:54:23 2024","interfaces":[{"interface":"back","average":{"1min":5.0250000000000004,"5min":3.5649999999999999,"15min":2.1549999999999998},"min":{"1min":0.30099999999999999,"5min":0.21299999999999999,"15min":0.20899999999999999},"max":{"1min":38.393000000000001,"5min":84.129000000000005,"15min":84.129000000000005},"last":3.081},{"interface":"front","average":{"1min":4.6310000000000002,"5min":3.343,"15min":2.089},"min":{"1min":0.32700000000000001,"5min":0.214,"15min":0.214},"max":{"1min":35.229999999999997,"5min":84.090999999999994,"15min":84.090999999999994},"last":2.8999999999999999}]},{"osd":4,"last update":"Mon Feb 5 06:54:34 2024","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.2450000000000001,"15min":1.1180000000000001},"min":{"1min":0.33500000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":7.726,"5min":7.726,"15min":12.041},"last":2.6219999999999999},{"interface":"front","average":{"1min":0.90100000000000002,"5min":1.143,"15min":1.083},"min":{"1min":0.29699999999999999,"5min":0.28000000000000003,"15min":0.26300000000000001},"max":{"1min":2.5779999999999998,"5min":5.4640000000000004,"15min":15.689},"last":0.38500000000000001}]},{"osd":5,"last update":"Mon Feb 5 06:54:08 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":2.2469999999999999,"15min":2.0369999999999999},"min":{"1min":0.32600000000000001,"5min":0.314,"15min":0.314},"max":{"1min":4.0449999999999999,"5min":78.497,"15min":86.614999999999995},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.425,"5min":2.1480000000000001,"15min":2.1499999999999999},"min":{"1min":0.38900000000000001,"5min":0.32300000000000001,"15min":0.312},"max":{"1min":4.1909999999999998,"5min":78.774000000000001,"15min":86.543999999999997},"last":0.443}]},{"osd":6,"last update":"Mon Feb 5 06:54:34 2024","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.976,"15min":1.462},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.315},"max":{"1min":3.8290000000000002,"5min":53.332000000000001,"15min":53.332000000000001},"last":0.33300000000000002},{"interface":"front","average":{"1min":1.3160000000000001,"5min":2.0249999999999999,"15min":1.7110000000000001},"min":{"1min":0.36099999999999999,"5min":0.33000000000000002,"15min":0.316},"max":{"1min":4.2309999999999999,"5min":53.320999999999998,"15min":53.320999999999998},"last":0.40200000000000002}]},{"osd":7,"last update":"Mon Feb 5 06:53:55 2024","interfaces":[{"interface":"back","average":{"1min":1.8620000000000001,"5min":1.653,"15min":1.623},"min":{"1min":0.44500000000000001,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":5.04,"5min":9.3800000000000008,"15min":45.243000000000002},"last":0.501},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.4510000000000001,"15min":1.5169999999999999},"min":{"1min":0.48399999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":5.593,"5min":7.5449999999999999,"15min":45.171999999999997},"last":2.7599999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149977,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":889124,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":92856028,"statfs":{"total":95995035648,"available":95084572672,"internally_reserved":0,"allocated":1404928,"data_stored":1013873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":909048263},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:54:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.444,"15min":1.7769999999999999},"min":{"1min":0.219,"5min":0.20200000000000001,"15min":0.184},"max":{"1min":7.1440000000000001,"5min":9.8849999999999998,"15min":53.656999999999996},"last":0.42999999999999999},{"interface":"front","average":{"1min":1.274,"5min":1.7869999999999999,"15min":1.895},"min":{"1min":0.25900000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":3.6230000000000002,"5min":12.997,"15min":53.338000000000001},"last":0.495}]},{"osd":1,"last update":"Mon Feb 5 06:54:28 2024","interfaces":[{"interface":"back","average":{"1min":2.6480000000000001,"5min":2.3069999999999999,"15min":1.804},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":11.417999999999999,"5min":24.343,"15min":24.343},"last":0.45600000000000002},{"interface":"front","average":{"1min":1.669,"5min":1.9990000000000001,"15min":1.744},"min":{"1min":0.221,"5min":0.221,"15min":0.215},"max":{"1min":11.526999999999999,"5min":23.300999999999998,"15min":23.300999999999998},"last":0.47399999999999998}]},{"osd":2,"last update":"Mon Feb 5 06:54:28 2024","interfaces":[{"interface":"back","average":{"1min":1.5409999999999999,"5min":1.7270000000000001,"15min":1.7649999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23000000000000001},"max":{"1min":4.2510000000000003,"5min":12.699999999999999,"15min":41.529000000000003},"last":0.55800000000000005},{"interface":"front","average":{"1min":1.5269999999999999,"5min":1.609,"15min":1.6399999999999999},"min":{"1min":0.28000000000000003,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":10.220000000000001,"5min":10.220000000000001,"15min":41.759999999999998},"last":0.50700000000000001}]},{"osd":4,"last update":"Mon Feb 5 06:54:43 2024","interfaces":[{"interface":"back","average":{"1min":1.675,"5min":1.4330000000000001,"15min":1.276},"min":{"1min":0.38100000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":10.15,"5min":16.59,"15min":16.59},"last":0.89300000000000002},{"interface":"front","average":{"1min":1.6619999999999999,"5min":1.544,"15min":1.5089999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.28599999999999998},"max":{"1min":12.103999999999999,"5min":12.103999999999999,"15min":19.359999999999999},"last":0.76600000000000001}]},{"osd":5,"last update":"Mon Feb 5 06:54:02 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":1.361,"15min":1.4530000000000001},"min":{"1min":0.33000000000000002,"5min":0.32200000000000001,"15min":0.29299999999999998},"max":{"1min":2.238,"5min":11.458,"15min":20.257999999999999},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.048,"5min":1.554,"15min":1.3959999999999999},"min":{"1min":0.40999999999999998,"5min":0.33700000000000002,"15min":0.32800000000000001},"max":{"1min":3.6480000000000001,"5min":16.629999999999999,"15min":16.629999999999999},"last":0.754}]},{"osd":6,"last update":"Mon Feb 5 06:54:43 2024","interfaces":[{"interface":"back","average":{"1min":1.825,"5min":1.9019999999999999,"15min":1.5660000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.28100000000000003},"max":{"1min":12.157999999999999,"5min":16.643000000000001,"15min":16.643000000000001},"last":0.746},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.5609999999999999,"15min":1.4550000000000001},"min":{"1min":0.375,"5min":0.35299999999999998,"15min":0.318},"max":{"1min":3.6800000000000002,"5min":12.75,"15min":12.75},"last":0.84399999999999997}]},{"osd":7,"last update":"Mon Feb 5 06:54:02 2024","interfaces":[{"interface":"back","average":{"1min":2.0110000000000001,"5min":1.671,"15min":1.462},"min":{"1min":0.46200000000000002,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":11.824,"5min":11.824,"15min":11.824},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.96099999999999997,"5min":1.5840000000000001,"15min":1.4910000000000001},"min":{"1min":0.47499999999999998,"5min":0.39100000000000001,"15min":0.35899999999999999},"max":{"1min":2.2669999999999999,"5min":16.611999999999998,"15min":16.611999999999998},"last":0.88300000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986451,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":802464,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":801662,"kb_avail":92942688,"statfs":{"total":95995035648,"available":95173312512,"internally_reserved":0,"allocated":811008,"data_stored":423727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":820902341},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:54:41 2024","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.248,"15min":1.23},"min":{"1min":0.23100000000000001,"5min":0.22900000000000001,"15min":0.20799999999999999},"max":{"1min":4.069,"5min":7.6299999999999999,"15min":33.372},"last":0.59499999999999997},{"interface":"front","average":{"1min":1.26,"5min":1.25,"15min":1.266},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.22700000000000001},"max":{"1min":3.8050000000000002,"5min":10.130000000000001,"15min":33.323999999999998},"last":0.66300000000000003}]},{"osd":1,"last update":"Mon Feb 5 06:54:41 2024","interfaces":[{"interface":"back","average":{"1min":3.5920000000000001,"5min":2.0419999999999998,"15min":1.758},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.21299999999999999},"max":{"1min":48.360999999999997,"5min":48.360999999999997,"15min":53.414000000000001},"last":0.55300000000000005},{"interface":"front","average":{"1min":3.653,"5min":1.901,"15min":1.6020000000000001},"min":{"1min":0.26400000000000001,"5min":0.23999999999999999,"15min":0.23000000000000001},"max":{"1min":48.670000000000002,"5min":48.670000000000002,"15min":53.719999999999999},"last":0.63}]},{"osd":2,"last update":"Mon Feb 5 06:54:41 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.3180000000000001,"15min":1.056},"min":{"1min":0.317,"5min":0.27300000000000002,"15min":0.26000000000000001},"max":{"1min":4.5220000000000002,"5min":7.5830000000000002,"15min":17.379999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":1.087,"5min":1.236,"15min":1.212},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.26000000000000001},"max":{"1min":3.1549999999999998,"5min":4.7830000000000004,"15min":18.077000000000002},"last":0.61799999999999999}]},{"osd":3,"last update":"Mon Feb 5 06:54:41 2024","interfaces":[{"interface":"back","average":{"1min":3.3420000000000001,"5min":1.6739999999999999,"15min":1.9730000000000001},"min":{"1min":0.32300000000000001,"5min":0.28499999999999998,"15min":0.252},"max":{"1min":54.228999999999999,"5min":54.228999999999999,"15min":54.228999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":3.419,"5min":1.8620000000000001,"15min":1.956},"min":{"1min":0.35599999999999998,"5min":0.307,"15min":0.29799999999999999},"max":{"1min":47.670999999999999,"5min":47.670999999999999,"15min":47.670999999999999},"last":0.58199999999999996}]},{"osd":5,"last update":"Mon Feb 5 06:54:03 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.86299999999999999,"15min":0.80300000000000005},"min":{"1min":0.26000000000000001,"5min":0.25,"15min":0.24399999999999999},"max":{"1min":2.4409999999999998,"5min":4.649,"15min":5.6059999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.92000000000000004,"15min":0.79200000000000004},"min":{"1min":0.26300000000000001,"5min":0.255,"15min":0.255},"max":{"1min":3.2639999999999998,"5min":7.7290000000000001,"15min":7.7290000000000001},"last":0.40500000000000003}]},{"osd":6,"last update":"Mon Feb 5 06:54:41 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.96099999999999997,"15min":0.90700000000000003},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.246},"max":{"1min":3.5790000000000002,"5min":7.6950000000000003,"15min":10.039999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.85299999999999998,"15min":0.84399999999999997},"min":{"1min":0.32200000000000001,"5min":0.312,"15min":0.29499999999999998},"max":{"1min":2.0910000000000002,"5min":4.0750000000000002,"15min":10.074},"last":0.51000000000000001}]},{"osd":7,"last update":"Mon Feb 5 06:54:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.83699999999999997,"15min":0.79300000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.27400000000000002},"max":{"1min":2.0659999999999998,"5min":4.1369999999999996,"15min":5.1500000000000004},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.755,"5min":0.90300000000000002,"15min":0.86499999999999999},"min":{"1min":0.41799999999999998,"5min":0.29799999999999999,"15min":0.28899999999999998},"max":{"1min":1.952,"5min":5.4809999999999999,"15min":5.5890000000000004},"last":0.497}]}]},{"osd":5,"up_from":40,"seq":171798692110,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":884624,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":883582,"kb_avail":92860528,"statfs":{"total":95995035648,"available":95089180672,"internally_reserved":0,"allocated":1056768,"data_stored":686624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":904788422},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:53:57 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":1.7330000000000001,"15min":1.603},"min":{"1min":0.29399999999999998,"5min":0.23100000000000001,"15min":0.223},"max":{"1min":2.6589999999999998,"5min":22.326000000000001,"15min":55.704999999999998},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.996,"5min":1.5509999999999999,"15min":1.6000000000000001},"min":{"1min":0.27300000000000002,"5min":0.26700000000000002,"15min":0.22600000000000001},"max":{"1min":2.7810000000000001,"5min":22.247,"15min":56.789000000000001},"last":0.69099999999999995}]},{"osd":1,"last update":"Mon Feb 5 06:53:57 2024","interfaces":[{"interface":"back","average":{"1min":5.0090000000000003,"5min":5.4539999999999997,"15min":2.8479999999999999},"min":{"1min":0.255,"5min":0.248,"15min":0.248},"max":{"1min":77.811999999999998,"5min":317.53399999999999,"15min":317.53399999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":5.3339999999999996,"5min":5.5670000000000002,"15min":2.867},"min":{"1min":0.255,"5min":0.22,"15min":0.22},"max":{"1min":77.373999999999995,"5min":317.80500000000001,"15min":317.80500000000001},"last":0.70299999999999996}]},{"osd":2,"last update":"Mon Feb 5 06:53:57 2024","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.6399999999999999,"15min":1.3680000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.23000000000000001},"max":{"1min":7.0789999999999997,"5min":22.635999999999999,"15min":30.052},"last":0.60699999999999998},{"interface":"front","average":{"1min":1.1970000000000001,"5min":1.141,"15min":1.175},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":4.8890000000000002,"5min":7.0540000000000003,"15min":30.472999999999999},"last":0.74399999999999999}]},{"osd":3,"last update":"Mon Feb 5 06:53:57 2024","interfaces":[{"interface":"back","average":{"1min":3.1819999999999999,"5min":2.282,"15min":2.0289999999999999},"min":{"1min":0.308,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":43.442999999999998,"5min":43.442999999999998,"15min":62.508000000000003},"last":0.57799999999999996},{"interface":"front","average":{"1min":3.1379999999999999,"5min":2.117,"15min":2.1030000000000002},"min":{"1min":0.33300000000000002,"5min":0.29899999999999999,"15min":0.27400000000000002},"max":{"1min":43.545000000000002,"5min":43.545000000000002,"15min":62.460000000000001},"last":0.65100000000000002}]},{"osd":4,"last update":"Mon Feb 5 06:53:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.82499999999999996,"15min":0.77200000000000002},"min":{"1min":0.25700000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":2.7669999999999999,"5min":9.3629999999999995,"15min":9.3629999999999995},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.89600000000000002,"15min":0.80100000000000005},"min":{"1min":0.26900000000000002,"5min":0.25600000000000001,"15min":0.252},"max":{"1min":2.4660000000000002,"5min":9.1839999999999993,"15min":9.1839999999999993},"last":0.47899999999999998}]},{"osd":6,"last update":"Mon Feb 5 06:54:36 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":1.595,"15min":1.145},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.23799999999999999},"max":{"1min":2.823,"5min":68.316999999999993,"15min":68.316999999999993},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.872,"5min":1.736,"15min":1.21},"min":{"1min":0.30599999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.472,"5min":70.352000000000004,"15min":70.352000000000004},"last":0.49299999999999999}]},{"osd":7,"last update":"Mon Feb 5 06:53:57 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.95999999999999996,"15min":0.96899999999999997},"min":{"1min":0.33600000000000002,"5min":0.317,"15min":0.25600000000000001},"max":{"1min":3.6000000000000001,"5min":11.691000000000001,"15min":11.691000000000001},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.90500000000000003,"15min":0.91800000000000004},"min":{"1min":0.373,"5min":0.34399999999999997,"15min":0.27100000000000002},"max":{"1min":2.8250000000000002,"5min":4.9939999999999998,"15min":6.5839999999999996},"last":0.46100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T06:54:55.876 INFO:tasks.thrashosds.thrasher:Setting 4.2 to [0, 2, 5] 2024-02-05T06:54:55.876 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '4.2', '0', '2', '5'] 2024-02-05T06:54:55.876 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd pg-upmap 4.2 0 2 5 2024-02-05T06:54:55.925 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:54:55.935 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:55.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:55 smithi028 ceph-mon[29442]: pgmap v1022: 168 pgs: 168 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 720 B/s rd, 0 B/s wr, 783 op/s 2024-02-05T06:54:55.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:55 smithi028 ceph-mon[29442]: 4.a scrub starts 2024-02-05T06:54:55.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:55 smithi028 ceph-mon[29442]: 4.a scrub ok 2024-02-05T06:54:55.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:55 smithi028 ceph-mon[37984]: pgmap v1022: 168 pgs: 168 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 720 B/s rd, 0 B/s wr, 783 op/s 2024-02-05T06:54:55.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:55 smithi028 ceph-mon[37984]: 4.a scrub starts 2024-02-05T06:54:55.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:55 smithi028 ceph-mon[37984]: 4.a scrub ok 2024-02-05T06:54:55.960 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:55.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:55.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:56.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:56.085 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:56.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:56.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:56.182 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-02-05T06:54:56.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:56.210 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:56.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:56.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:56.311 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:56.335 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:56.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:56.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:56.433 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:54:56.437 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:56.460 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:56.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:56.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:56.562 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:56.565 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-02-05T06:54:56.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:56.587 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:56.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:56.688 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:54:56.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:56.716 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:56.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:54:56.800 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:56 smithi028 ceph-mon[29442]: 4.13 scrub starts 2024-02-05T06:54:56.800 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:56 smithi028 ceph-mon[29442]: 4.13 scrub ok 2024-02-05T06:54:56.800 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:56 smithi028 ceph-mon[29442]: 4.f scrub starts 2024-02-05T06:54:56.801 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:56 smithi028 ceph-mon[29442]: 4.f scrub ok 2024-02-05T06:54:56.801 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:56 smithi028 ceph-mon[37984]: 4.13 scrub starts 2024-02-05T06:54:56.801 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:56 smithi028 ceph-mon[37984]: 4.13 scrub ok 2024-02-05T06:54:56.801 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:56 smithi028 ceph-mon[37984]: 4.f scrub starts 2024-02-05T06:54:56.801 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:56 smithi028 ceph-mon[37984]: 4.f scrub ok 2024-02-05T06:54:56.817 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:56.841 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:56.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:56.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:56.942 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:56.968 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:56.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:56.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:56.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:56 smithi167 ceph-mon[32774]: 4.13 scrub starts 2024-02-05T06:54:56.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:56 smithi167 ceph-mon[32774]: 4.13 scrub ok 2024-02-05T06:54:56.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:56 smithi167 ceph-mon[32774]: 4.f scrub starts 2024-02-05T06:54:56.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:56 smithi167 ceph-mon[32774]: 4.f scrub ok 2024-02-05T06:54:57.069 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:57.093 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:57.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:57.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:57.194 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:57.219 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:57.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:57.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:57.320 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:57.345 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:57.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:57.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:57.446 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:54:57.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:57.471 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:57.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:54:57.572 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:57.597 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:57.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:57.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:57.698 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:57.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:57.722 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:57.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:57.823 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:57.846 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:57.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:57.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:57.913 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:57 smithi167 ceph-mon[32774]: pgmap v1023: 168 pgs: 168 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 631 B/s rd, 0 B/s wr, 364 op/s 2024-02-05T06:54:57.913 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:57 smithi167 ceph-mon[32774]: 6.1e scrub starts 2024-02-05T06:54:57.913 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:57 smithi167 ceph-mon[32774]: 6.1e scrub ok 2024-02-05T06:54:57.948 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:57.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:57 smithi028 ceph-mon[29442]: pgmap v1023: 168 pgs: 168 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 631 B/s rd, 0 B/s wr, 364 op/s 2024-02-05T06:54:57.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:57 smithi028 ceph-mon[29442]: 6.1e scrub starts 2024-02-05T06:54:57.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:57 smithi028 ceph-mon[29442]: 6.1e scrub ok 2024-02-05T06:54:57.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:57 smithi028 ceph-mon[37984]: pgmap v1023: 168 pgs: 168 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 631 B/s rd, 0 B/s wr, 364 op/s 2024-02-05T06:54:57.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:57 smithi028 ceph-mon[37984]: 6.1e scrub starts 2024-02-05T06:54:57.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:57 smithi028 ceph-mon[37984]: 6.1e scrub ok 2024-02-05T06:54:57.971 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:57.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:57.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:58.073 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:58.096 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:58.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:58.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:58.185 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:54:58.185 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [], 2024-02-05T06:54:58.185 INFO:teuthology.orchestra.run.smithi167.stdout: "complaint_time": 30, 2024-02-05T06:54:58.186 INFO:teuthology.orchestra.run.smithi167.stdout: "num_blocked_ops": 0 2024-02-05T06:54:58.186 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:54:58.198 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:58.222 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:58.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:58.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:58.324 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:54:58.348 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:58.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:58.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:54:58.450 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:58.473 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:58.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:58.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:58.574 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:58.598 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:58.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:58.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:58.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:58.725 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:58.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:58.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:58.827 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:54:58.832 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-02-05T06:54:58.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:58.850 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:58.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:54:58.855 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:58 smithi167 ceph-mon[32774]: 6.1c scrub starts 2024-02-05T06:54:58.856 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:58 smithi167 ceph-mon[32774]: 6.1c scrub ok 2024-02-05T06:54:58.856 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:58 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:54:58.856 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:58 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:54:58.856 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:58 smithi167 ceph-mon[32774]: 5.15 deep-scrub starts 2024-02-05T06:54:58.856 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:58 smithi167 ceph-mon[32774]: 5.15 deep-scrub ok 2024-02-05T06:54:58.856 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:58 smithi167 ceph-mon[32774]: 4.e scrub starts 2024-02-05T06:54:58.856 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:58 smithi167 ceph-mon[32774]: 4.e scrub ok 2024-02-05T06:54:58.950 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:58.974 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:58.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:58.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:59.015 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:58 smithi028 ceph-mon[37984]: 6.1c scrub starts 2024-02-05T06:54:59.015 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:58 smithi028 ceph-mon[37984]: 6.1c scrub ok 2024-02-05T06:54:59.015 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:58 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:54:59.015 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:58 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:54:59.015 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:58 smithi028 ceph-mon[37984]: 5.15 deep-scrub starts 2024-02-05T06:54:59.015 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:58 smithi028 ceph-mon[37984]: 5.15 deep-scrub ok 2024-02-05T06:54:59.015 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:58 smithi028 ceph-mon[37984]: 4.e scrub starts 2024-02-05T06:54:59.015 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:58 smithi028 ceph-mon[37984]: 4.e scrub ok 2024-02-05T06:54:59.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:58 smithi028 ceph-mon[29442]: 6.1c scrub starts 2024-02-05T06:54:59.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:58 smithi028 ceph-mon[29442]: 6.1c scrub ok 2024-02-05T06:54:59.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:58 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:54:59.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:58 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:54:59.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:58 smithi028 ceph-mon[29442]: 5.15 deep-scrub starts 2024-02-05T06:54:59.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:58 smithi028 ceph-mon[29442]: 5.15 deep-scrub ok 2024-02-05T06:54:59.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:58 smithi028 ceph-mon[29442]: 4.e scrub starts 2024-02-05T06:54:59.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:58 smithi028 ceph-mon[29442]: 4.e scrub ok 2024-02-05T06:54:59.073 INFO:teuthology.orchestra.run.smithi167.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.b/config 2024-02-05T06:54:59.076 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:59.102 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:59.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:59.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:59.203 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:59.227 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:59.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:59.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:59.233 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:54:59.330 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:54:59.353 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:54:59.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:59.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:54:59.455 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:54:59.481 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:59.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:59.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:54:59.582 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:59.606 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:59.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:59.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:59.707 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:59.721 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is set 2024-02-05T06:54:59.731 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:59.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:59.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:54:59.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:59 smithi167 ceph-mon[32774]: 5.13 scrub starts 2024-02-05T06:54:59.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:59 smithi167 ceph-mon[32774]: 5.13 scrub ok 2024-02-05T06:54:59.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:59 smithi167 ceph-mon[32774]: pgmap v1024: 168 pgs: 168 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 295 op/s 2024-02-05T06:54:59.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:59 smithi167 ceph-mon[32774]: 5.9 scrub starts 2024-02-05T06:54:59.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:59 smithi167 ceph-mon[32774]: 5.9 scrub ok 2024-02-05T06:54:59.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:59 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2839235742' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:54:59.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:59 smithi167 ceph-mon[32774]: 5.3 scrub starts 2024-02-05T06:54:59.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:54:59 smithi167 ceph-mon[32774]: 5.3 scrub ok 2024-02-05T06:54:59.833 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:54:59.857 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:59.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:59.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:54:59.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:59 smithi028 ceph-mon[29442]: 5.13 scrub starts 2024-02-05T06:54:59.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:59 smithi028 ceph-mon[29442]: 5.13 scrub ok 2024-02-05T06:54:59.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:59 smithi028 ceph-mon[29442]: pgmap v1024: 168 pgs: 168 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 295 op/s 2024-02-05T06:54:59.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:59 smithi028 ceph-mon[29442]: 5.9 scrub starts 2024-02-05T06:54:59.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:59 smithi028 ceph-mon[29442]: 5.9 scrub ok 2024-02-05T06:54:59.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:59 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2839235742' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:54:59.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:59 smithi028 ceph-mon[29442]: 5.3 scrub starts 2024-02-05T06:54:59.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:54:59 smithi028 ceph-mon[29442]: 5.3 scrub ok 2024-02-05T06:54:59.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:59 smithi028 ceph-mon[37984]: 5.13 scrub starts 2024-02-05T06:54:59.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:59 smithi028 ceph-mon[37984]: 5.13 scrub ok 2024-02-05T06:54:59.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:59 smithi028 ceph-mon[37984]: pgmap v1024: 168 pgs: 168 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 295 op/s 2024-02-05T06:54:59.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:59 smithi028 ceph-mon[37984]: 5.9 scrub starts 2024-02-05T06:54:59.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:59 smithi028 ceph-mon[37984]: 5.9 scrub ok 2024-02-05T06:54:59.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:59 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2839235742' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-02-05T06:54:59.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:59 smithi028 ceph-mon[37984]: 5.3 scrub starts 2024-02-05T06:54:59.950 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:54:59 smithi028 ceph-mon[37984]: 5.3 scrub ok 2024-02-05T06:54:59.958 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:54:59.981 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:54:59.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:54:59.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:55:00.083 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:55:00.107 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:55:00.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:55:00.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:55:00.210 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:55:00.233 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:55:00.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:55:00.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:55:00.335 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:55:00.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:55:00.361 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:55:00.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:55:00.461 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:55:00.485 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:55:00.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:55:00.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:55:00.586 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5 2024-02-05T06:55:00.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:55:00.611 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:55:00.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.5' 2024-02-05T06:55:00.712 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3 2024-02-05T06:55:00.730 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'unique_pool_4' removed 2024-02-05T06:55:00.749 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:55:00.766 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:55:00.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:55:00.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.3' 2024-02-05T06:55:00.868 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:55:00.916 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:55:00.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:55:00.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:55:00.961 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:00 smithi167 ceph-mon[32774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:55:00.961 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:00 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2839235742' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:55:00.961 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:00 smithi167 ceph-mon[32774]: osdmap e480: 8 total, 8 up, 8 in 2024-02-05T06:55:00.961 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:00 smithi167 ceph-mon[32774]: 1.12 scrub starts 2024-02-05T06:55:00.961 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:00 smithi167 ceph-mon[32774]: 1.12 scrub ok 2024-02-05T06:55:00.961 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:00 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3149218006' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T06:55:00.961 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:00 smithi167 ceph-mon[32774]: from='client.46103 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T06:55:01.018 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1 2024-02-05T06:55:01.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:55:01.042 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:55:01.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.1' 2024-02-05T06:55:01.047 INFO:teuthology.orchestra.run.smithi167.stdout:{ 2024-02-05T06:55:01.047 INFO:teuthology.orchestra.run.smithi167.stdout: "size": 20, 2024-02-05T06:55:01.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 600, 2024-02-05T06:55:01.048 INFO:teuthology.orchestra.run.smithi167.stdout: "ops": [ 2024-02-05T06:55:01.048 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.048 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:121278 9.3 9:c5b97c0a:::benchmark_data_smithi028_129096_object121277:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-02-05T06:55:01.048 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:01.606264+0000", 2024-02-05T06:55:01.048 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 599.43262743699995, 2024-02-05T06:55:01.048 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.19801748599999999, 2024-02-05T06:55:01.048 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:55:01.048 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:55:01.048 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:55:01.048 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:55:01.048 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:55:01.048 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 121278 2024-02-05T06:55:01.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.049 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:55:01.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:55:01.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.606264+0000", 2024-02-05T06:55:01.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:55:01.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.606264+0000", 2024-02-05T06:55:01.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.049 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.049 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.049 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:55:01.049 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.606266+0000", 2024-02-05T06:55:01.049 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.637e-06 2024-02-05T06:55:01.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:55:01.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.606300+0000", 2024-02-05T06:55:01.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3850000000000003e-05 2024-02-05T06:55:01.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:55:01.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.606317+0000", 2024-02-05T06:55:01.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6960999999999999e-05 2024-02-05T06:55:01.050 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.050 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.050 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:55:01.050 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.606324+0000", 2024-02-05T06:55:01.050 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7939999999999997e-06 2024-02-05T06:55:01.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:55:01.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.627530+0000", 2024-02-05T06:55:01.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021206827000000001 2024-02-05T06:55:01.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:55:01.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.627632+0000", 2024-02-05T06:55:01.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010118 2024-02-05T06:55:01.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.051 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.051 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:55:01.051 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.627764+0000", 2024-02-05T06:55:01.051 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013247399999999999 2024-02-05T06:55:01.051 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:55:01.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.638046+0000", 2024-02-05T06:55:01.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.010282224 2024-02-05T06:55:01.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:55:01.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.672097+0000", 2024-02-05T06:55:01.052 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.034050992000000002 2024-02-05T06:55:01.052 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.052 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.052 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:55:01.052 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.804158+0000", 2024-02-05T06:55:01.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13206031300000001 2024-02-05T06:55:01.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:55:01.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.804214+0000", 2024-02-05T06:55:01.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6702e-05 2024-02-05T06:55:01.053 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.053 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.053 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:55:01.053 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.804282+0000", 2024-02-05T06:55:01.053 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7532000000000003e-05 2024-02-05T06:55:01.053 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.053 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:55:01.053 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.054 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.054 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.054 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:121284 9.3 9:cc770777:::benchmark_data_smithi028_129096_object121283:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-02-05T06:55:01.054 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:01.607527+0000", 2024-02-05T06:55:01.054 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 599.43136512000001, 2024-02-05T06:55:01.054 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.204634289, 2024-02-05T06:55:01.054 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:55:01.054 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:55:01.054 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:55:01.054 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:55:01.054 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:55:01.055 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 121284 2024-02-05T06:55:01.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.055 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:55:01.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:55:01.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.607527+0000", 2024-02-05T06:55:01.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.055 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:55:01.055 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.607527+0000", 2024-02-05T06:55:01.055 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.055 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.055 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:55:01.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.607528+0000", 2024-02-05T06:55:01.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2869999999999999e-06 2024-02-05T06:55:01.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:55:01.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.607578+0000", 2024-02-05T06:55:01.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9703999999999997e-05 2024-02-05T06:55:01.056 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.056 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.056 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:55:01.056 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.607588+0000", 2024-02-05T06:55:01.056 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0623e-05 2024-02-05T06:55:01.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.057 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:55:01.057 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.607594+0000", 2024-02-05T06:55:01.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.1990000000000003e-06 2024-02-05T06:55:01.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.057 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:55:01.057 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.630799+0000", 2024-02-05T06:55:01.057 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.023204758999999998 2024-02-05T06:55:01.057 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.057 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.057 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:55:01.057 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.630920+0000", 2024-02-05T06:55:01.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120815 2024-02-05T06:55:01.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:55:01.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.631209+0000", 2024-02-05T06:55:01.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00028938500000000002 2024-02-05T06:55:01.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.058 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:55:01.058 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.638095+0000", 2024-02-05T06:55:01.058 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.006885408 2024-02-05T06:55:01.058 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.058 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:55:01.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.672194+0000", 2024-02-05T06:55:01.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.034099061 2024-02-05T06:55:01.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:55:01.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.811988+0000", 2024-02-05T06:55:01.059 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.13979428599999999 2024-02-05T06:55:01.059 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.059 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.059 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:55:01.059 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.812093+0000", 2024-02-05T06:55:01.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010472299999999999 2024-02-05T06:55:01.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.060 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:55:01.060 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.812161+0000", 2024-02-05T06:55:01.060 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8039000000000004e-05 2024-02-05T06:55:01.060 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.060 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:55:01.060 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.060 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.060 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.060 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:121294 9.3 9:c54f2ea6:::benchmark_data_smithi028_129096_object121293:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-02-05T06:55:01.061 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:01.615831+0000", 2024-02-05T06:55:01.061 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 599.42306028400003, 2024-02-05T06:55:01.061 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20027431800000001, 2024-02-05T06:55:01.061 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:55:01.061 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:55:01.061 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:55:01.061 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:55:01.061 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:55:01.061 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 121294 2024-02-05T06:55:01.061 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.061 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:55:01.061 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.061 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:55:01.061 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.615831+0000", 2024-02-05T06:55:01.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:55:01.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.615831+0000", 2024-02-05T06:55:01.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:55:01.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.615833+0000", 2024-02-05T06:55:01.062 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.362e-06 2024-02-05T06:55:01.062 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.062 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.062 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:55:01.062 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.615907+0000", 2024-02-05T06:55:01.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4153000000000006e-05 2024-02-05T06:55:01.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:55:01.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.615917+0000", 2024-02-05T06:55:01.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0443e-05 2024-02-05T06:55:01.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:55:01.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.615925+0000", 2024-02-05T06:55:01.063 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4930000000000001e-06 2024-02-05T06:55:01.063 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.063 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.063 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:55:01.063 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.635225+0000", 2024-02-05T06:55:01.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.019300033000000001 2024-02-05T06:55:01.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:55:01.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.635370+0000", 2024-02-05T06:55:01.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014553100000000001 2024-02-05T06:55:01.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:55:01.064 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.635524+0000", 2024-02-05T06:55:01.064 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015344299999999999 2024-02-05T06:55:01.064 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.064 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.064 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:55:01.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.672146+0000", 2024-02-05T06:55:01.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.036622447000000002 2024-02-05T06:55:01.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:55:01.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.672198+0000", 2024-02-05T06:55:01.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2027000000000002e-05 2024-02-05T06:55:01.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.065 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.065 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:55:01.065 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.815828+0000", 2024-02-05T06:55:01.065 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14362993299999999 2024-02-05T06:55:01.065 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.066 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:55:01.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.816045+0000", 2024-02-05T06:55:01.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000216899 2024-02-05T06:55:01.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.066 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.066 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:55:01.066 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.816106+0000", 2024-02-05T06:55:01.066 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.0554000000000003e-05 2024-02-05T06:55:01.066 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.066 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:55:01.066 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.066 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.067 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.15390.0:121296 9.3 9:cf583f78:::benchmark_data_smithi028_129096_object121295:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e315)", 2024-02-05T06:55:01.067 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:45:01.616489+0000", 2024-02-05T06:55:01.067 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 599.422402498, 2024-02-05T06:55:01.067 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.19977958800000001, 2024-02-05T06:55:01.067 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:55:01.067 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:55:01.067 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:55:01.067 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.15390", 2024-02-05T06:55:01.067 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/2776619285", 2024-02-05T06:55:01.067 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 121296 2024-02-05T06:55:01.067 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.067 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:55:01.067 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:55:01.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.616489+0000", 2024-02-05T06:55:01.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:55:01.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.616489+0000", 2024-02-05T06:55:01.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.068 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.068 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:55:01.068 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.616490+0000", 2024-02-05T06:55:01.068 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.19e-06 2024-02-05T06:55:01.068 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:55:01.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.616543+0000", 2024-02-05T06:55:01.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2602000000000002e-05 2024-02-05T06:55:01.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:55:01.069 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.616554+0000", 2024-02-05T06:55:01.069 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0669999999999999e-05 2024-02-05T06:55:01.069 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.069 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.069 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:55:01.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.616561+0000", 2024-02-05T06:55:01.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3119999999999996e-06 2024-02-05T06:55:01.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:55:01.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.638128+0000", 2024-02-05T06:55:01.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.021566703999999999 2024-02-05T06:55:01.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:55:01.070 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.638253+0000", 2024-02-05T06:55:01.070 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012526700000000001 2024-02-05T06:55:01.070 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.070 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.070 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,3", 2024-02-05T06:55:01.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.638404+0000", 2024-02-05T06:55:01.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000150989 2024-02-05T06:55:01.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:55:01.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.672186+0000", 2024-02-05T06:55:01.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.033781904000000001 2024-02-05T06:55:01.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.071 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.071 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:55:01.071 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.672202+0000", 2024-02-05T06:55:01.071 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6025e-05 2024-02-05T06:55:01.071 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:55:01.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.816148+0000", 2024-02-05T06:55:01.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.14394652399999999 2024-02-05T06:55:01.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:55:01.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.816223+0000", 2024-02-05T06:55:01.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5007999999999997e-05 2024-02-05T06:55:01.072 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.072 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.072 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:55:01.072 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:45:01.816269+0000", 2024-02-05T06:55:01.072 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5392999999999997e-05 2024-02-05T06:55:01.073 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.073 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:55:01.073 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.073 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.073 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.45587.0:103385 10.1 10:887dabf2:::benchmark_data_smithi028_154709_object103384:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-02-05T06:55:01.073 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:48:51.195870+0000", 2024-02-05T06:55:01.073 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 369.84302167599998, 2024-02-05T06:55:01.073 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.23867543399999999, 2024-02-05T06:55:01.073 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:55:01.073 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-02-05T06:55:01.073 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:55:01.073 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.45587", 2024-02-05T06:55:01.073 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:55:01.073 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 103385 2024-02-05T06:55:01.073 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.074 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:55:01.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:55:01.074 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.195870+0000", 2024-02-05T06:55:01.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:55:01.074 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.195870+0000", 2024-02-05T06:55:01.074 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.074 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.074 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.074 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:55:01.074 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.195872+0000", 2024-02-05T06:55:01.075 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2230000000000001e-06 2024-02-05T06:55:01.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:55:01.076 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.195905+0000", 2024-02-05T06:55:01.076 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.2787000000000003e-05 2024-02-05T06:55:01.076 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.076 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.076 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:55:01.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.196396+0000", 2024-02-05T06:55:01.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00049096600000000002 2024-02-05T06:55:01.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:55:01.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.196408+0000", 2024-02-05T06:55:01.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2435e-05 2024-02-05T06:55:01.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.077 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:55:01.077 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.200078+0000", 2024-02-05T06:55:01.077 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0036696799999999998 2024-02-05T06:55:01.077 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.077 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:55:01.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.200210+0000", 2024-02-05T06:55:01.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013139900000000001 2024-02-05T06:55:01.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,4", 2024-02-05T06:55:01.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.200350+0000", 2024-02-05T06:55:01.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00014017700000000001 2024-02-05T06:55:01.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.078 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:55:01.078 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.207265+0000", 2024-02-05T06:55:01.078 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0069154899999999998 2024-02-05T06:55:01.078 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.078 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:55:01.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.434499+0000", 2024-02-05T06:55:01.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.227233609 2024-02-05T06:55:01.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.079 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:55:01.079 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.434546+0000", 2024-02-05T06:55:01.079 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6668000000000001e-05 2024-02-05T06:55:01.079 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.079 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:55:01.079 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.079 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.079 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.079 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.45587.0:103394 10.1 10:85e12e54:::benchmark_data_smithi028_154709_object103393:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-02-05T06:55:01.079 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:48:51.207491+0000", 2024-02-05T06:55:01.079 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 369.83140072399999, 2024-02-05T06:55:01.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.22720298, 2024-02-05T06:55:01.080 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:55:01.080 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-02-05T06:55:01.080 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:55:01.080 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.45587", 2024-02-05T06:55:01.080 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:55:01.080 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 103394 2024-02-05T06:55:01.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.080 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:55:01.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:55:01.080 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.207491+0000", 2024-02-05T06:55:01.080 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.080 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.080 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.080 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:55:01.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.207491+0000", 2024-02-05T06:55:01.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:55:01.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.207496+0000", 2024-02-05T06:55:01.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.904e-06 2024-02-05T06:55:01.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:55:01.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.207533+0000", 2024-02-05T06:55:01.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7540999999999999e-05 2024-02-05T06:55:01.081 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.081 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.081 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:55:01.081 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.207547+0000", 2024-02-05T06:55:01.081 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.3348e-05 2024-02-05T06:55:01.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.082 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:55:01.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.207557+0000", 2024-02-05T06:55:01.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0501e-05 2024-02-05T06:55:01.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.082 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:55:01.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.207625+0000", 2024-02-05T06:55:01.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7348000000000006e-05 2024-02-05T06:55:01.082 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.082 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.082 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:55:01.082 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.207780+0000", 2024-02-05T06:55:01.082 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000155072 2024-02-05T06:55:01.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,4", 2024-02-05T06:55:01.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.207943+0000", 2024-02-05T06:55:01.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000162798 2024-02-05T06:55:01.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:55:01.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.225685+0000", 2024-02-05T06:55:01.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.017742581 2024-02-05T06:55:01.083 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.083 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.083 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:55:01.083 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.434659+0000", 2024-02-05T06:55:01.083 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20897402400000001 2024-02-05T06:55:01.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.084 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:55:01.084 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.434694+0000", 2024-02-05T06:55:01.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.4863000000000001e-05 2024-02-05T06:55:01.084 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.084 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:55:01.084 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.084 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.084 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.084 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.45587.0:103385 10.11 10:887dabf2:::benchmark_data_smithi028_154709_object103384:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-02-05T06:55:01.084 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:48:51.224250+0000", 2024-02-05T06:55:01.084 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 369.814642123, 2024-02-05T06:55:01.084 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.042329407, 2024-02-05T06:55:01.084 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:55:01.085 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-02-05T06:55:01.085 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:55:01.085 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.45587", 2024-02-05T06:55:01.085 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:55:01.085 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 103385 2024-02-05T06:55:01.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.085 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:55:01.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:55:01.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.224250+0000", 2024-02-05T06:55:01.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.085 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.085 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:55:01.085 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.224250+0000", 2024-02-05T06:55:01.085 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.085 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:55:01.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.224252+0000", 2024-02-05T06:55:01.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.7329999999999999e-06 2024-02-05T06:55:01.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:55:01.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.224297+0000", 2024-02-05T06:55:01.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4518000000000003e-05 2024-02-05T06:55:01.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.086 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.086 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:55:01.086 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.224314+0000", 2024-02-05T06:55:01.086 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6767999999999999e-05 2024-02-05T06:55:01.086 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:55:01.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.224324+0000", 2024-02-05T06:55:01.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.0834e-05 2024-02-05T06:55:01.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:55:01.087 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.435212+0000", 2024-02-05T06:55:01.087 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.21088786300000001 2024-02-05T06:55:01.087 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.087 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.087 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:55:01.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.435220+0000", 2024-02-05T06:55:01.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.7109999999999995e-06 2024-02-05T06:55:01.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:55:01.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:52.266506+0000", 2024-02-05T06:55:01.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.831285736 2024-02-05T06:55:01.088 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.088 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.088 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:55:01.088 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:52.266579+0000", 2024-02-05T06:55:01.088 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.3244000000000005e-05 2024-02-05T06:55:01.088 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.088 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:55:01.089 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.089 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.089 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.45587.0:103394 10.1 10:85e12e54:::benchmark_data_smithi028_154709_object103393:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-02-05T06:55:01.089 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:48:51.224345+0000", 2024-02-05T06:55:01.089 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 369.814547022, 2024-02-05T06:55:01.089 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.031986635, 2024-02-05T06:55:01.089 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:55:01.089 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-02-05T06:55:01.089 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:55:01.089 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.45587", 2024-02-05T06:55:01.089 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:55:01.089 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 103394 2024-02-05T06:55:01.089 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.090 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:55:01.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.090 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:55:01.090 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.224345+0000", 2024-02-05T06:55:01.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.090 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:55:01.090 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.224345+0000", 2024-02-05T06:55:01.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.090 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.090 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:55:01.090 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.224346+0000", 2024-02-05T06:55:01.090 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.9299999999999997e-07 2024-02-05T06:55:01.090 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:55:01.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.224384+0000", 2024-02-05T06:55:01.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8577e-05 2024-02-05T06:55:01.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:55:01.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.224391+0000", 2024-02-05T06:55:01.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8410000000000002e-06 2024-02-05T06:55:01.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.091 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:55:01.091 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.224397+0000", 2024-02-05T06:55:01.091 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.8050000000000003e-06 2024-02-05T06:55:01.091 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.091 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:55:01.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.225792+0000", 2024-02-05T06:55:01.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0013949100000000001 2024-02-05T06:55:01.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op must wait for map", 2024-02-05T06:55:01.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.225797+0000", 2024-02-05T06:55:01.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6690000000000003e-06 2024-02-05T06:55:01.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.092 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.092 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:55:01.092 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.435045+0000", 2024-02-05T06:55:01.092 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.20924769800000001 2024-02-05T06:55:01.092 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:55:01.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.435057+0000", 2024-02-05T06:55:01.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2384e-05 2024-02-05T06:55:01.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:55:01.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:52.248123+0000", 2024-02-05T06:55:01.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.81306562400000004 2024-02-05T06:55:01.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.093 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for degraded object", 2024-02-05T06:55:01.093 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:52.249507+0000", 2024-02-05T06:55:01.093 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.001383699 2024-02-05T06:55:01.093 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.093 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:55:01.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:52.256040+0000", 2024-02-05T06:55:01.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.006533662 2024-02-05T06:55:01.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.094 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:55:01.094 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:52.256331+0000", 2024-02-05T06:55:01.094 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00029097299999999998 2024-02-05T06:55:01.094 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.094 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:55:01.094 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.094 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.094 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.094 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.45587.0:103410 10.1 10:821e21d6:::benchmark_data_smithi028_154709_object103409:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-02-05T06:55:01.094 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:48:51.443595+0000", 2024-02-05T06:55:01.095 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 369.59529702600003, 2024-02-05T06:55:01.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.81676914599999995, 2024-02-05T06:55:01.095 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:55:01.095 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:55:01.095 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:55:01.095 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.45587", 2024-02-05T06:55:01.095 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:55:01.095 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 103410 2024-02-05T06:55:01.095 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.095 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:55:01.095 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.095 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:55:01.095 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.443595+0000", 2024-02-05T06:55:01.095 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:55:01.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.443595+0000", 2024-02-05T06:55:01.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.096 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:55:01.096 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.443604+0000", 2024-02-05T06:55:01.096 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.8570000000000001e-06 2024-02-05T06:55:01.096 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.096 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:55:01.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.443664+0000", 2024-02-05T06:55:01.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.9953e-05 2024-02-05T06:55:01.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:55:01.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.443693+0000", 2024-02-05T06:55:01.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.9918999999999999e-05 2024-02-05T06:55:01.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.097 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:55:01.097 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.443709+0000", 2024-02-05T06:55:01.097 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5549e-05 2024-02-05T06:55:01.097 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.097 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:55:01.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.443830+0000", 2024-02-05T06:55:01.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00012127499999999999 2024-02-05T06:55:01.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:55:01.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:51.443853+0000", 2024-02-05T06:55:01.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.2331e-05 2024-02-05T06:55:01.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.098 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:55:01.098 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:52.249550+0000", 2024-02-05T06:55:01.098 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.80569765000000004 2024-02-05T06:55:01.098 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.098 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:55:01.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:52.249700+0000", 2024-02-05T06:55:01.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015017599999999999 2024-02-05T06:55:01.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 0,4", 2024-02-05T06:55:01.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:52.250090+0000", 2024-02-05T06:55:01.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00038907699999999999 2024-02-05T06:55:01.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.099 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.099 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:55:01.099 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:52.258894+0000", 2024-02-05T06:55:01.099 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0088044409999999997 2024-02-05T06:55:01.099 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:55:01.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:52.259693+0000", 2024-02-05T06:55:01.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00079862200000000003 2024-02-05T06:55:01.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:55:01.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:52.260259+0000", 2024-02-05T06:55:01.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00056657099999999998 2024-02-05T06:55:01.100 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.100 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.100 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:55:01.100 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:52.260326+0000", 2024-02-05T06:55:01.100 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7107000000000006e-05 2024-02-05T06:55:01.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.101 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:55:01.101 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:52.260364+0000", 2024-02-05T06:55:01.101 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7617999999999998e-05 2024-02-05T06:55:01.101 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.101 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:55:01.101 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.101 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.101 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.101 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.45587.0:104378 10.10 10.ce60b010 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-02-05T06:55:01.101 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:48:53.256083+0000", 2024-02-05T06:55:01.101 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 367.78280912399998, 2024-02-05T06:55:01.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96550274800000002, 2024-02-05T06:55:01.102 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:55:01.102 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-02-05T06:55:01.102 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:55:01.102 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.45587", 2024-02-05T06:55:01.102 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:55:01.102 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 104378 2024-02-05T06:55:01.102 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.102 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:55:01.102 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.102 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:55:01.102 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:53.256083+0000", 2024-02-05T06:55:01.102 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:55:01.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:53.256083+0000", 2024-02-05T06:55:01.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:55:01.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:53.256085+0000", 2024-02-05T06:55:01.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.3319999999999999e-06 2024-02-05T06:55:01.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.103 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.103 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:55:01.103 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:53.256119+0000", 2024-02-05T06:55:01.103 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.3701000000000001e-05 2024-02-05T06:55:01.103 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:55:01.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:53.256137+0000", 2024-02-05T06:55:01.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.8097999999999999e-05 2024-02-05T06:55:01.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:55:01.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:53.256146+0000", 2024-02-05T06:55:01.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.4550000000000001e-06 2024-02-05T06:55:01.104 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.104 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.104 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:55:01.104 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:53.256220+0000", 2024-02-05T06:55:01.104 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4141999999999996e-05 2024-02-05T06:55:01.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:55:01.105 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:53.256232+0000", 2024-02-05T06:55:01.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1882000000000001e-05 2024-02-05T06:55:01.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:55:01.105 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:54.221574+0000", 2024-02-05T06:55:01.105 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.96534206300000003 2024-02-05T06:55:01.105 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.105 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.105 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:55:01.106 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:54.221585+0000", 2024-02-05T06:55:01.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1075e-05 2024-02-05T06:55:01.106 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.106 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:55:01.106 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.106 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.106 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.106 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.45587.0:104383 10.10 10.6a674bd0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-02-05T06:55:01.106 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:48:53.273637+0000", 2024-02-05T06:55:01.106 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 367.765254395, 2024-02-05T06:55:01.106 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.94804730000000004, 2024-02-05T06:55:01.106 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:55:01.106 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-02-05T06:55:01.106 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:55:01.106 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.45587", 2024-02-05T06:55:01.107 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:55:01.107 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 104383 2024-02-05T06:55:01.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.107 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:55:01.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:55:01.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:53.273637+0000", 2024-02-05T06:55:01.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.107 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.107 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:55:01.107 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:53.273637+0000", 2024-02-05T06:55:01.107 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.107 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:55:01.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:53.273640+0000", 2024-02-05T06:55:01.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.6900000000000001e-06 2024-02-05T06:55:01.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:55:01.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:53.273693+0000", 2024-02-05T06:55:01.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2497000000000002e-05 2024-02-05T06:55:01.108 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.108 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.108 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:55:01.108 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:53.273714+0000", 2024-02-05T06:55:01.108 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.1149999999999999e-05 2024-02-05T06:55:01.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:55:01.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:53.273725+0000", 2024-02-05T06:55:01.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1124999999999999e-05 2024-02-05T06:55:01.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:55:01.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:53.273788+0000", 2024-02-05T06:55:01.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3106000000000002e-05 2024-02-05T06:55:01.109 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.109 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.109 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:55:01.109 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:53.273797+0000", 2024-02-05T06:55:01.109 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5969999999999992e-06 2024-02-05T06:55:01.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:55:01.110 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:54.221675+0000", 2024-02-05T06:55:01.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.94787816199999997 2024-02-05T06:55:01.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.110 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.110 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:55:01.110 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:54.221685+0000", 2024-02-05T06:55:01.110 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.9729999999999992e-06 2024-02-05T06:55:01.110 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.110 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:55:01.110 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.110 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.111 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.111 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.45587.0:104402 10.10 10.cd82d790 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-02-05T06:55:01.111 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:48:53.293526+0000", 2024-02-05T06:55:01.111 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 367.745365784, 2024-02-05T06:55:01.111 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.92821340399999996, 2024-02-05T06:55:01.111 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:55:01.111 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "reached pg", 2024-02-05T06:55:01.111 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:55:01.111 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.45587", 2024-02-05T06:55:01.111 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/3603251292", 2024-02-05T06:55:01.111 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 104402 2024-02-05T06:55:01.111 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.111 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:55:01.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:55:01.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:53.293526+0000", 2024-02-05T06:55:01.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:55:01.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:53.293526+0000", 2024-02-05T06:55:01.112 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.112 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.112 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.112 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:55:01.112 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:53.293528+0000", 2024-02-05T06:55:01.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6109999999999999e-06 2024-02-05T06:55:01.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:55:01.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:53.293581+0000", 2024-02-05T06:55:01.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3804000000000002e-05 2024-02-05T06:55:01.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:55:01.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:53.293594+0000", 2024-02-05T06:55:01.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2174999999999999e-05 2024-02-05T06:55:01.113 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.113 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.113 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:55:01.113 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:53.293600+0000", 2024-02-05T06:55:01.113 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.6989999999999997e-06 2024-02-05T06:55:01.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:55:01.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:53.293652+0000", 2024-02-05T06:55:01.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.2210999999999999e-05 2024-02-05T06:55:01.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for peered", 2024-02-05T06:55:01.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:53.293658+0000", 2024-02-05T06:55:01.114 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.4210000000000003e-06 2024-02-05T06:55:01.114 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.114 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.114 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:55:01.114 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:54.221731+0000", 2024-02-05T06:55:01.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.92807275 2024-02-05T06:55:01.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.115 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:55:01.115 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:48:54.221739+0000", 2024-02-05T06:55:01.115 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7329999999999992e-06 2024-02-05T06:55:01.115 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.115 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:55:01.115 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.115 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.115 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.116 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.45878.0:53122 11.5 11:a0bac0cb:::benchmark_data_smithi028_177697_object53121:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-02-05T06:55:01.116 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:51:48.356932+0000", 2024-02-05T06:55:01.116 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 192.681959302, 2024-02-05T06:55:01.116 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.19665516, 2024-02-05T06:55:01.116 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:55:01.116 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:55:01.116 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:55:01.116 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.45878", 2024-02-05T06:55:01.116 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/3059201968", 2024-02-05T06:55:01.116 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 53122 2024-02-05T06:55:01.116 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.116 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:55:01.116 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:55:01.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:51:48.356932+0000", 2024-02-05T06:55:01.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:55:01.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:51:48.356932+0000", 2024-02-05T06:55:01.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.117 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.117 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:55:01.117 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:51:48.356933+0000", 2024-02-05T06:55:01.117 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5899999999999995e-07 2024-02-05T06:55:01.117 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:55:01.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:51:48.356996+0000", 2024-02-05T06:55:01.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2434999999999994e-05 2024-02-05T06:55:01.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:55:01.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:51:48.357004+0000", 2024-02-05T06:55:01.118 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.5210000000000002e-06 2024-02-05T06:55:01.118 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.118 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.118 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:55:01.118 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:51:48.357009+0000", 2024-02-05T06:55:01.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.9089999999999999e-06 2024-02-05T06:55:01.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:55:01.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:51:48.357049+0000", 2024-02-05T06:55:01.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.0014e-05 2024-02-05T06:55:01.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:55:01.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:51:48.357136+0000", 2024-02-05T06:55:01.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.7237999999999994e-05 2024-02-05T06:55:01.119 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.119 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.119 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 5,6", 2024-02-05T06:55:01.119 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:51:48.357219+0000", 2024-02-05T06:55:01.119 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.2737000000000004e-05 2024-02-05T06:55:01.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:55:01.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:51:48.359281+0000", 2024-02-05T06:55:01.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020618670000000002 2024-02-05T06:55:01.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:55:01.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:51:48.359298+0000", 2024-02-05T06:55:01.120 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6643999999999999e-05 2024-02-05T06:55:01.120 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.120 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.120 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:55:01.120 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:51:48.553463+0000", 2024-02-05T06:55:01.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.19416557100000001 2024-02-05T06:55:01.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:55:01.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:51:48.553544+0000", 2024-02-05T06:55:01.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.1238999999999999e-05 2024-02-05T06:55:01.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.121 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.121 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:55:01.121 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:51:48.553588+0000", 2024-02-05T06:55:01.121 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.3226000000000003e-05 2024-02-05T06:55:01.121 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.121 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:55:01.121 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.121 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.122 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.45878.0:168295 11.f 11:f13a7919:::benchmark_data_smithi028_177697_object41329:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e464)", 2024-02-05T06:55:01.122 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:53:33.469752+0000", 2024-02-05T06:55:01.122 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 87.569139449999994, 2024-02-05T06:55:01.122 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.30596847599999999, 2024-02-05T06:55:01.122 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:55:01.122 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:55:01.122 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:55:01.122 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.45878", 2024-02-05T06:55:01.122 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/3059201968", 2024-02-05T06:55:01.122 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 168295 2024-02-05T06:55:01.122 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.122 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:55:01.122 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:55:01.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:33.469752+0000", 2024-02-05T06:55:01.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:55:01.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:33.469752+0000", 2024-02-05T06:55:01.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:55:01.123 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:33.469754+0000", 2024-02-05T06:55:01.123 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.778e-06 2024-02-05T06:55:01.123 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.123 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.123 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:55:01.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:33.469760+0000", 2024-02-05T06:55:01.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6439999999999997e-06 2024-02-05T06:55:01.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:55:01.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:33.469770+0000", 2024-02-05T06:55:01.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.04e-05 2024-02-05T06:55:01.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:55:01.124 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:33.469779+0000", 2024-02-05T06:55:01.124 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.3809999999999999e-06 2024-02-05T06:55:01.124 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.124 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.124 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:55:01.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:33.469916+0000", 2024-02-05T06:55:01.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00013733500000000001 2024-02-05T06:55:01.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:55:01.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:33.470037+0000", 2024-02-05T06:55:01.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000120956 2024-02-05T06:55:01.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,2", 2024-02-05T06:55:01.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:33.470151+0000", 2024-02-05T06:55:01.125 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113892 2024-02-05T06:55:01.125 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.125 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.125 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:55:01.125 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:33.472789+0000", 2024-02-05T06:55:01.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002638559 2024-02-05T06:55:01.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:55:01.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:33.474179+0000", 2024-02-05T06:55:01.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0013893969999999999 2024-02-05T06:55:01.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:55:01.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:33.775629+0000", 2024-02-05T06:55:01.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.30145002999999998 2024-02-05T06:55:01.126 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.126 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.126 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:55:01.126 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:33.775674+0000", 2024-02-05T06:55:01.126 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.5176000000000003e-05 2024-02-05T06:55:01.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.127 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:55:01.127 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:33.775721+0000", 2024-02-05T06:55:01.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6928000000000002e-05 2024-02-05T06:55:01.127 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.127 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:55:01.127 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.127 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.127 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.127 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.45878.0:168314 11.f 11:f7129641:::benchmark_data_smithi028_177697_object41348:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e464)", 2024-02-05T06:55:01.127 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:53:33.490792+0000", 2024-02-05T06:55:01.127 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 87.548099582999996, 2024-02-05T06:55:01.127 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.29699945100000003, 2024-02-05T06:55:01.127 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:55:01.127 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:55:01.128 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:55:01.128 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.45878", 2024-02-05T06:55:01.128 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/3059201968", 2024-02-05T06:55:01.128 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 168314 2024-02-05T06:55:01.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.128 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:55:01.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:55:01.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:33.490792+0000", 2024-02-05T06:55:01.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.128 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.128 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.128 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:55:01.128 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:33.490792+0000", 2024-02-05T06:55:01.128 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:55:01.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:33.490793+0000", 2024-02-05T06:55:01.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.7699999999999992e-07 2024-02-05T06:55:01.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:55:01.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:33.490797+0000", 2024-02-05T06:55:01.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8829999999999999e-06 2024-02-05T06:55:01.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.129 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.129 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:55:01.129 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:33.490809+0000", 2024-02-05T06:55:01.129 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2016e-05 2024-02-05T06:55:01.129 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:55:01.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:33.490815+0000", 2024-02-05T06:55:01.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2169999999999999e-06 2024-02-05T06:55:01.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:55:01.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:33.490866+0000", 2024-02-05T06:55:01.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.0537000000000001e-05 2024-02-05T06:55:01.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.130 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.130 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:55:01.130 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:33.490946+0000", 2024-02-05T06:55:01.130 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.0604000000000003e-05 2024-02-05T06:55:01.130 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,2", 2024-02-05T06:55:01.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:33.491047+0000", 2024-02-05T06:55:01.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000100425 2024-02-05T06:55:01.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:55:01.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:33.493153+0000", 2024-02-05T06:55:01.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0021057480000000002 2024-02-05T06:55:01.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.131 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.131 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:55:01.131 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:33.494906+0000", 2024-02-05T06:55:01.131 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0017531059999999999 2024-02-05T06:55:01.131 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:55:01.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:33.787697+0000", 2024-02-05T06:55:01.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.29279097199999998 2024-02-05T06:55:01.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:55:01.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:33.787753+0000", 2024-02-05T06:55:01.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5995999999999997e-05 2024-02-05T06:55:01.132 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.132 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.132 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:55:01.132 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:33.787792+0000", 2024-02-05T06:55:01.132 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8970000000000001e-05 2024-02-05T06:55:01.133 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.133 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:55:01.133 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.133 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.133 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.45878.0:168729 11.4 11:27dca3c5:::benchmark_data_smithi028_177697_object41763:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e464)", 2024-02-05T06:55:01.133 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:53:34.162451+0000", 2024-02-05T06:55:01.133 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 86.876440821000003, 2024-02-05T06:55:01.133 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.28764783999999999, 2024-02-05T06:55:01.133 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:55:01.133 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:55:01.133 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:55:01.133 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.45878", 2024-02-05T06:55:01.133 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/3059201968", 2024-02-05T06:55:01.133 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 168729 2024-02-05T06:55:01.133 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.134 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:55:01.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:55:01.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.162451+0000", 2024-02-05T06:55:01.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:55:01.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.162451+0000", 2024-02-05T06:55:01.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.134 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.134 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.134 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:55:01.134 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.162452+0000", 2024-02-05T06:55:01.134 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1519999999999999e-06 2024-02-05T06:55:01.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:55:01.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.162456+0000", 2024-02-05T06:55:01.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.8489999999999999e-06 2024-02-05T06:55:01.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:55:01.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.162467+0000", 2024-02-05T06:55:01.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1127e-05 2024-02-05T06:55:01.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.135 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.135 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:55:01.135 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.162475+0000", 2024-02-05T06:55:01.135 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.96e-06 2024-02-05T06:55:01.135 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:55:01.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.162532+0000", 2024-02-05T06:55:01.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6920999999999999e-05 2024-02-05T06:55:01.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:55:01.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.162645+0000", 2024-02-05T06:55:01.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000113239 2024-02-05T06:55:01.136 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.136 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.136 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 2,6", 2024-02-05T06:55:01.136 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.162766+0000", 2024-02-05T06:55:01.136 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.000121106 2024-02-05T06:55:01.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:55:01.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.165296+0000", 2024-02-05T06:55:01.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.002529552 2024-02-05T06:55:01.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:55:01.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.165451+0000", 2024-02-05T06:55:01.137 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00015486800000000001 2024-02-05T06:55:01.137 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.137 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.137 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:55:01.137 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.450036+0000", 2024-02-05T06:55:01.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.28458510399999998 2024-02-05T06:55:01.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:55:01.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.450074+0000", 2024-02-05T06:55:01.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 3.7710999999999999e-05 2024-02-05T06:55:01.138 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.138 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.138 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:55:01.138 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.450099+0000", 2024-02-05T06:55:01.138 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.5250999999999999e-05 2024-02-05T06:55:01.138 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.138 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:55:01.139 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.139 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.45878.0:168750 11.f 11:f2712193:::benchmark_data_smithi028_177697_object41784:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e464)", 2024-02-05T06:55:01.139 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:53:34.184289+0000", 2024-02-05T06:55:01.139 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 86.854602677000003, 2024-02-05T06:55:01.139 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.27541354200000001, 2024-02-05T06:55:01.139 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:55:01.139 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:55:01.139 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:55:01.139 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.45878", 2024-02-05T06:55:01.139 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/3059201968", 2024-02-05T06:55:01.139 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 168750 2024-02-05T06:55:01.139 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.139 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:55:01.139 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:55:01.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.184289+0000", 2024-02-05T06:55:01.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:55:01.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.184289+0000", 2024-02-05T06:55:01.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.140 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:55:01.140 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.184291+0000", 2024-02-05T06:55:01.140 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.0559999999999999e-06 2024-02-05T06:55:01.140 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.140 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:55:01.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.184296+0000", 2024-02-05T06:55:01.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.6290000000000001e-06 2024-02-05T06:55:01.141 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.141 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.141 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:55:01.141 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.184307+0000", 2024-02-05T06:55:01.141 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1671000000000001e-05 2024-02-05T06:55:01.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:55:01.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.184314+0000", 2024-02-05T06:55:01.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.3799999999999999e-06 2024-02-05T06:55:01.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:55:01.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.184363+0000", 2024-02-05T06:55:01.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.8807999999999998e-05 2024-02-05T06:55:01.142 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.142 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.142 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:55:01.142 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.184454+0000", 2024-02-05T06:55:01.142 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 9.179e-05 2024-02-05T06:55:01.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,2", 2024-02-05T06:55:01.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.184529+0000", 2024-02-05T06:55:01.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.4720000000000003e-05 2024-02-05T06:55:01.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:55:01.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.186496+0000", 2024-02-05T06:55:01.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0019663929999999999 2024-02-05T06:55:01.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.143 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:55:01.143 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.186908+0000", 2024-02-05T06:55:01.143 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00041271800000000001 2024-02-05T06:55:01.143 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.143 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:55:01.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.459588+0000", 2024-02-05T06:55:01.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.27268011199999997 2024-02-05T06:55:01.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:55:01.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.459646+0000", 2024-02-05T06:55:01.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.7586000000000001e-05 2024-02-05T06:55:01.144 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.144 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.144 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:55:01.144 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.459703+0000", 2024-02-05T06:55:01.144 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.6678999999999997e-05 2024-02-05T06:55:01.144 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.144 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:55:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.45878.0:168755 11.f 11:f2a1f944:::benchmark_data_smithi028_177697_object41789:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e464)", 2024-02-05T06:55:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:53:34.209239+0000", 2024-02-05T06:55:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 86.829653065000002, 2024-02-05T06:55:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.25060943099999999, 2024-02-05T06:55:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:55:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "commit sent; apply or cleanup", 2024-02-05T06:55:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:55:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.45878", 2024-02-05T06:55:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.28:0/3059201968", 2024-02-05T06:55:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 168755 2024-02-05T06:55:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:55:01.145 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:55:01.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.209239+0000", 2024-02-05T06:55:01.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:55:01.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.209239+0000", 2024-02-05T06:55:01.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.146 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:55:01.146 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.209240+0000", 2024-02-05T06:55:01.146 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.5629999999999999e-06 2024-02-05T06:55:01.146 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.146 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:55:01.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.209247+0000", 2024-02-05T06:55:01.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.8249999999999999e-06 2024-02-05T06:55:01.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:55:01.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.209258+0000", 2024-02-05T06:55:01.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.1352000000000001e-05 2024-02-05T06:55:01.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:55:01.147 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.209265+0000", 2024-02-05T06:55:01.147 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.2990000000000002e-06 2024-02-05T06:55:01.147 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.147 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.147 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:55:01.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.209310+0000", 2024-02-05T06:55:01.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.4790999999999999e-05 2024-02-05T06:55:01.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:55:01.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.209399+0000", 2024-02-05T06:55:01.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 8.9040000000000001e-05 2024-02-05T06:55:01.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "waiting for subops from 1,2", 2024-02-05T06:55:01.148 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.209474+0000", 2024-02-05T06:55:01.148 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 7.5399000000000001e-05 2024-02-05T06:55:01.148 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.148 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.148 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "op_commit", 2024-02-05T06:55:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.211522+0000", 2024-02-05T06:55:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.0020477759999999998 2024-02-05T06:55:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:55:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.212382+0000", 2024-02-05T06:55:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00086019600000000003 2024-02-05T06:55:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "sub_op_commit_rec", 2024-02-05T06:55:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.459751+0000", 2024-02-05T06:55:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.24736945199999999 2024-02-05T06:55:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "commit_sent", 2024-02-05T06:55:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.459805+0000", 2024-02-05T06:55:01.149 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.3936e-05 2024-02-05T06:55:01.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.150 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:55:01.150 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:53:34.459848+0000", 2024-02-05T06:55:01.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 4.2802000000000001e-05 2024-02-05T06:55:01.150 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.150 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:55:01.150 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.150 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.150 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.150 INFO:teuthology.orchestra.run.smithi167.stdout: "description": "osd_op(client.14568.0:1275 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e479)", 2024-02-05T06:55:01.150 INFO:teuthology.orchestra.run.smithi167.stdout: "initiated_at": "2024-02-05T06:55:00.562999+0000", 2024-02-05T06:55:01.150 INFO:teuthology.orchestra.run.smithi167.stdout: "age": 0.47589312700000003, 2024-02-05T06:55:01.150 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00027951799999999998, 2024-02-05T06:55:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: "type_data": { 2024-02-05T06:55:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: "flag_point": "started", 2024-02-05T06:55:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: "client_info": { 2024-02-05T06:55:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: "client": "client.14568", 2024-02-05T06:55:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: "client_addr": "v1:172.21.15.167:0/939366379", 2024-02-05T06:55:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: "tid": 1275 2024-02-05T06:55:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: "events": [ 2024-02-05T06:55:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "initiated", 2024-02-05T06:55:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:55:00.562999+0000", 2024-02-05T06:55:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "header_read", 2024-02-05T06:55:01.151 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:55:00.562999+0000", 2024-02-05T06:55:01.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0 2024-02-05T06:55:01.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "throttled", 2024-02-05T06:55:01.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:55:00.563001+0000", 2024-02-05T06:55:01.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 2.131e-06 2024-02-05T06:55:01.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "all_read", 2024-02-05T06:55:01.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:55:00.563017+0000", 2024-02-05T06:55:01.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6209999999999999e-05 2024-02-05T06:55:01.152 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.152 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.152 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "dispatched", 2024-02-05T06:55:01.152 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:55:00.563033+0000", 2024-02-05T06:55:01.152 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.6302000000000001e-05 2024-02-05T06:55:01.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "queued_for_pg", 2024-02-05T06:55:01.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:55:00.563046+0000", 2024-02-05T06:55:01.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 1.2999e-05 2024-02-05T06:55:01.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "reached_pg", 2024-02-05T06:55:01.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:55:00.563155+0000", 2024-02-05T06:55:01.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 0.00010857900000000001 2024-02-05T06:55:01.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.153 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.153 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "started", 2024-02-05T06:55:01.153 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:55:00.563222+0000", 2024-02-05T06:55:01.153 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 6.7440000000000005e-05 2024-02-05T06:55:01.153 INFO:teuthology.orchestra.run.smithi167.stdout: }, 2024-02-05T06:55:01.154 INFO:teuthology.orchestra.run.smithi167.stdout: { 2024-02-05T06:55:01.154 INFO:teuthology.orchestra.run.smithi167.stdout: "event": "done", 2024-02-05T06:55:01.154 INFO:teuthology.orchestra.run.smithi167.stdout: "time": "2024-02-05T06:55:00.563278+0000", 2024-02-05T06:55:01.154 INFO:teuthology.orchestra.run.smithi167.stdout: "duration": 5.5856999999999997e-05 2024-02-05T06:55:01.154 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.154 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:55:01.154 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.154 INFO:teuthology.orchestra.run.smithi167.stdout: } 2024-02-05T06:55:01.154 INFO:teuthology.orchestra.run.smithi167.stdout: ] 2024-02-05T06:55:01.154 INFO:teuthology.orchestra.run.smithi167.stdout:} 2024-02-05T06:55:01.154 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0 2024-02-05T06:55:01.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:55:01.178 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:55:01.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.0' 2024-02-05T06:55:01.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:00 smithi028 ceph-mon[37984]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:55:01.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:00 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2839235742' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:55:01.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:00 smithi028 ceph-mon[37984]: osdmap e480: 8 total, 8 up, 8 in 2024-02-05T06:55:01.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:00 smithi028 ceph-mon[37984]: 1.12 scrub starts 2024-02-05T06:55:01.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:00 smithi028 ceph-mon[37984]: 1.12 scrub ok 2024-02-05T06:55:01.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:00 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3149218006' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T06:55:01.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:00 smithi028 ceph-mon[37984]: from='client.46103 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T06:55:01.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:00 smithi028 ceph-mon[29442]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-02-05T06:55:01.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:00 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2839235742' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-02-05T06:55:01.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:00 smithi028 ceph-mon[29442]: osdmap e480: 8 total, 8 up, 8 in 2024-02-05T06:55:01.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:00 smithi028 ceph-mon[29442]: 1.12 scrub starts 2024-02-05T06:55:01.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:00 smithi028 ceph-mon[29442]: 1.12 scrub ok 2024-02-05T06:55:01.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:00 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3149218006' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T06:55:01.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:00 smithi028 ceph-mon[29442]: from='client.46103 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-05T06:55:01.280 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:55:01.303 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:55:01.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:55:01.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:55:01.405 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6 2024-02-05T06:55:01.428 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:55:01.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:55:01.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.6' 2024-02-05T06:55:01.530 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4 2024-02-05T06:55:01.553 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:55:01.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:55:01.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.4' 2024-02-05T06:55:01.655 DEBUG:teuthology.orchestra.run.smithi028:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2 2024-02-05T06:55:01.679 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: docker: command not found 2024-02-05T06:55:01.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:55:01.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi028 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.2' 2024-02-05T06:55:01.781 DEBUG:teuthology.orchestra.run.smithi167:> sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7 2024-02-05T06:55:01.805 INFO:teuthology.orchestra.run.smithi167.stderr:sudo: docker: command not found 2024-02-05T06:55:01.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T06:55:01.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi167 with status 1: 'sudo docker kill -s 1 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd.7' 2024-02-05T06:55:01.905 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-02-05T06:55:01.916 INFO:tasks.thrashosds:joining thrashosds 2024-02-05T06:55:01.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:01 smithi167 ceph-mon[32774]: pgmap v1026: 168 pgs: 168 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 124 op/s 2024-02-05T06:55:01.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:01 smithi167 ceph-mon[32774]: from='client.46103 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T06:55:01.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:01 smithi167 ceph-mon[32774]: osdmap e481: 8 total, 8 up, 8 in 2024-02-05T06:55:02.052 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:01 smithi028 ceph-mon[29442]: pgmap v1026: 168 pgs: 168 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 124 op/s 2024-02-05T06:55:02.052 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:01 smithi028 ceph-mon[29442]: from='client.46103 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T06:55:02.052 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:01 smithi028 ceph-mon[29442]: osdmap e481: 8 total, 8 up, 8 in 2024-02-05T06:55:02.052 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:01 smithi028 ceph-mon[37984]: pgmap v1026: 168 pgs: 168 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 124 op/s 2024-02-05T06:55:02.052 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:01 smithi028 ceph-mon[37984]: from='client.46103 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-02-05T06:55:02.053 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:01 smithi028 ceph-mon[37984]: osdmap e481: 8 total, 8 up, 8 in 2024-02-05T06:55:02.679 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:55:02 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:55:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:55:02.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:02 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:55:02.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:02 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2006898075' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:55:02.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:02 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:55:02.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:02 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2006898075' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:55:02.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:02 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:55:02.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:02 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2006898075' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-02-05T06:55:03.060 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0: {} 2024-02-05T06:55:03.060 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1: {} 2024-02-05T06:55:03.061 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2: {} 2024-02-05T06:55:03.061 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3: {} 2024-02-05T06:55:03.061 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4: {} 2024-02-05T06:55:03.061 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5: {} 2024-02-05T06:55:03.061 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6: {} 2024-02-05T06:55:03.061 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7: {} 2024-02-05T06:55:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:osd.0: osd_enable_op_tracker = '' 2024-02-05T06:55:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:osd.1: osd_enable_op_tracker = '' 2024-02-05T06:55:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:osd.2: osd_enable_op_tracker = '' 2024-02-05T06:55:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:osd.3: osd_enable_op_tracker = '' 2024-02-05T06:55:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:osd.4: osd_enable_op_tracker = '' 2024-02-05T06:55:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:osd.5: osd_enable_op_tracker = '' 2024-02-05T06:55:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:osd.6: osd_enable_op_tracker = '' 2024-02-05T06:55:03.062 INFO:teuthology.orchestra.run.smithi028.stderr:osd.7: osd_enable_op_tracker = '' 2024-02-05T06:55:03.485 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset noscrub 2024-02-05T06:55:03.761 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:03 smithi028 ceph-mon[29442]: pgmap v1028: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:55:03.761 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:03 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:55:03.761 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:03 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:55:03.761 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:03 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:55:03.761 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:03 smithi028 ceph-mon[37984]: pgmap v1028: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:55:03.761 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:03 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:55:03.761 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:03 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:55:03.761 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:03 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:55:03.983 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:55:03 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:55:03.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:03 smithi167 ceph-mon[32774]: pgmap v1028: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:55:03.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:03 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:55:03.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:03 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:55:03.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:03 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:55:04.275 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:55:04.826 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:04 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:55:04.826 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:04 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:55:04.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:04 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:55:05.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:05 smithi167 ceph-mon[32774]: pgmap v1029: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:55:06.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:05 smithi028 ceph-mon[29442]: pgmap v1029: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:55:06.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:05 smithi028 ceph-mon[37984]: pgmap v1029: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:55:06.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:06 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1077801230' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.2", "id": [0, 2, 5]}]: dispatch 2024-02-05T06:55:06.742 INFO:teuthology.orchestra.run.smithi028.stderr:set 4.2 pg_upmap mapping to [0,2,5] 2024-02-05T06:55:07.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:06 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1077801230' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.2", "id": [0, 2, 5]}]: dispatch 2024-02-05T06:55:07.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:06 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1077801230' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.2", "id": [0, 2, 5]}]: dispatch 2024-02-05T06:55:07.733 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:55:07 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a[63665]: logger=cleanup t=2024-02-05T06:55:07.453419636Z level=info msg="Completed cleanup jobs" duration=42.149039ms 2024-02-05T06:55:07.746 INFO:teuthology.orchestra.run.smithi028.stderr:noscrub is unset 2024-02-05T06:55:07.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:07 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:55:07.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:07 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:55:07.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:07 smithi028 ceph-mon[29442]: pgmap v1030: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-05T06:55:07.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:07 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1077801230' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.2", "id": [0, 2, 5]}]': finished 2024-02-05T06:55:07.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:07 smithi028 ceph-mon[29442]: osdmap e482: 8 total, 8 up, 8 in 2024-02-05T06:55:07.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:07 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1809802418' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:55:07.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:07 smithi028 ceph-mon[29442]: from='client.46121 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:55:07.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:07 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:55:07.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:07 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:55:07.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:07 smithi028 ceph-mon[37984]: pgmap v1030: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-05T06:55:07.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:07 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1077801230' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.2", "id": [0, 2, 5]}]': finished 2024-02-05T06:55:07.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:07 smithi028 ceph-mon[37984]: osdmap e482: 8 total, 8 up, 8 in 2024-02-05T06:55:07.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:07 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1809802418' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:55:07.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:07 smithi028 ceph-mon[37984]: from='client.46121 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:55:08.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:07 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:55:08.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:07 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:55:08.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:07 smithi167 ceph-mon[32774]: pgmap v1030: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-02-05T06:55:08.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:07 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1077801230' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.2", "id": [0, 2, 5]}]': finished 2024-02-05T06:55:08.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:07 smithi167 ceph-mon[32774]: osdmap e482: 8 total, 8 up, 8 in 2024-02-05T06:55:08.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:07 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1809802418' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:55:08.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:07 smithi167 ceph-mon[32774]: from='client.46121 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-02-05T06:55:08.703 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd unset nodeep-scrub 2024-02-05T06:55:08.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:08 smithi028 ceph-mon[29442]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-02-05T06:55:08.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:08 smithi028 ceph-mon[29442]: Cluster is now healthy 2024-02-05T06:55:08.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:08 smithi028 ceph-mon[29442]: from='client.46121 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:55:08.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:08 smithi028 ceph-mon[29442]: osdmap e483: 8 total, 8 up, 8 in 2024-02-05T06:55:08.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:08 smithi028 ceph-mon[37984]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-02-05T06:55:08.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:08 smithi028 ceph-mon[37984]: Cluster is now healthy 2024-02-05T06:55:08.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:08 smithi028 ceph-mon[37984]: from='client.46121 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:55:08.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:08 smithi028 ceph-mon[37984]: osdmap e483: 8 total, 8 up, 8 in 2024-02-05T06:55:08.960 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:55:09.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:08 smithi167 ceph-mon[32774]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-02-05T06:55:09.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:08 smithi167 ceph-mon[32774]: Cluster is now healthy 2024-02-05T06:55:09.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:08 smithi167 ceph-mon[32774]: from='client.46121 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-02-05T06:55:09.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:08 smithi167 ceph-mon[32774]: osdmap e483: 8 total, 8 up, 8 in 2024-02-05T06:55:09.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:09 smithi028 ceph-mon[29442]: pgmap v1033: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T06:55:09.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:09 smithi028 ceph-mon[29442]: 4.3 deep-scrub starts 2024-02-05T06:55:09.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:09 smithi028 ceph-mon[29442]: 4.3 deep-scrub ok 2024-02-05T06:55:09.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:09 smithi028 ceph-mon[37984]: pgmap v1033: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T06:55:09.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:09 smithi028 ceph-mon[37984]: 4.3 deep-scrub starts 2024-02-05T06:55:09.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:09 smithi028 ceph-mon[37984]: 4.3 deep-scrub ok 2024-02-05T06:55:10.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:09 smithi167 ceph-mon[32774]: pgmap v1033: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T06:55:10.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:09 smithi167 ceph-mon[32774]: 4.3 deep-scrub starts 2024-02-05T06:55:10.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:09 smithi167 ceph-mon[32774]: 4.3 deep-scrub ok 2024-02-05T06:55:11.347 INFO:teuthology.orchestra.run.smithi028.stderr:nodeep-scrub is unset 2024-02-05T06:55:11.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:11 smithi028 ceph-mon[37984]: 3.a scrub starts 2024-02-05T06:55:11.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:11 smithi028 ceph-mon[37984]: 3.a scrub ok 2024-02-05T06:55:11.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:11 smithi028 ceph-mon[37984]: 4.6 scrub starts 2024-02-05T06:55:11.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:11 smithi028 ceph-mon[37984]: 4.6 scrub ok 2024-02-05T06:55:11.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:11 smithi028 ceph-mon[37984]: pgmap v1034: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:55:11.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:11 smithi028 ceph-mon[37984]: 3.14 scrub starts 2024-02-05T06:55:11.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:11 smithi028 ceph-mon[37984]: 3.14 scrub ok 2024-02-05T06:55:11.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:11 smithi028 ceph-mon[37984]: 4.7 scrub starts 2024-02-05T06:55:11.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:11 smithi028 ceph-mon[37984]: 4.7 scrub ok 2024-02-05T06:55:11.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:11 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3216146161' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:55:11.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:11 smithi028 ceph-mon[37984]: from='client.46127 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:55:11.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:11 smithi028 ceph-mon[29442]: 3.a scrub starts 2024-02-05T06:55:11.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:11 smithi028 ceph-mon[29442]: 3.a scrub ok 2024-02-05T06:55:11.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:11 smithi028 ceph-mon[29442]: 4.6 scrub starts 2024-02-05T06:55:11.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:11 smithi028 ceph-mon[29442]: 4.6 scrub ok 2024-02-05T06:55:11.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:11 smithi028 ceph-mon[29442]: pgmap v1034: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:55:11.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:11 smithi028 ceph-mon[29442]: 3.14 scrub starts 2024-02-05T06:55:11.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:11 smithi028 ceph-mon[29442]: 3.14 scrub ok 2024-02-05T06:55:11.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:11 smithi028 ceph-mon[29442]: 4.7 scrub starts 2024-02-05T06:55:11.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:11 smithi028 ceph-mon[29442]: 4.7 scrub ok 2024-02-05T06:55:11.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:11 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3216146161' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:55:11.701 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:11 smithi028 ceph-mon[29442]: from='client.46127 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:55:11.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:11 smithi167 ceph-mon[32774]: 3.a scrub starts 2024-02-05T06:55:11.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:11 smithi167 ceph-mon[32774]: 3.a scrub ok 2024-02-05T06:55:11.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:11 smithi167 ceph-mon[32774]: 4.6 scrub starts 2024-02-05T06:55:11.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:11 smithi167 ceph-mon[32774]: 4.6 scrub ok 2024-02-05T06:55:11.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:11 smithi167 ceph-mon[32774]: pgmap v1034: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:55:11.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:11 smithi167 ceph-mon[32774]: 3.14 scrub starts 2024-02-05T06:55:11.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:11 smithi167 ceph-mon[32774]: 3.14 scrub ok 2024-02-05T06:55:11.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:11 smithi167 ceph-mon[32774]: 4.7 scrub starts 2024-02-05T06:55:11.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:11 smithi167 ceph-mon[32774]: 4.7 scrub ok 2024-02-05T06:55:11.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:11 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3216146161' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:55:11.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:11 smithi167 ceph-mon[32774]: from='client.46127 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-02-05T06:55:12.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:12 smithi028 ceph-mon[37984]: 5.11 scrub starts 2024-02-05T06:55:12.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:12 smithi028 ceph-mon[37984]: 5.11 scrub ok 2024-02-05T06:55:12.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:12 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:55:12.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:12 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:55:12.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:12 smithi028 ceph-mon[37984]: 6.f scrub starts 2024-02-05T06:55:12.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:12 smithi028 ceph-mon[37984]: from='client.46127 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:55:12.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:12 smithi028 ceph-mon[37984]: osdmap e484: 8 total, 8 up, 8 in 2024-02-05T06:55:12.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:12 smithi028 ceph-mon[37984]: 6.f scrub ok 2024-02-05T06:55:12.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:12 smithi028 ceph-mon[37984]: 3.f scrub starts 2024-02-05T06:55:12.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:12 smithi028 ceph-mon[37984]: 3.f scrub ok 2024-02-05T06:55:12.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:12 smithi028 ceph-mon[37984]: 4.9 scrub starts 2024-02-05T06:55:12.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:12 smithi028 ceph-mon[37984]: 4.9 scrub ok 2024-02-05T06:55:12.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:12 smithi028 ceph-mon[29442]: 5.11 scrub starts 2024-02-05T06:55:12.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:12 smithi028 ceph-mon[29442]: 5.11 scrub ok 2024-02-05T06:55:12.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:12 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:55:12.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:12 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:55:12.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:12 smithi028 ceph-mon[29442]: 6.f scrub starts 2024-02-05T06:55:12.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:12 smithi028 ceph-mon[29442]: from='client.46127 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:55:12.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:12 smithi028 ceph-mon[29442]: osdmap e484: 8 total, 8 up, 8 in 2024-02-05T06:55:12.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:12 smithi028 ceph-mon[29442]: 6.f scrub ok 2024-02-05T06:55:12.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:12 smithi028 ceph-mon[29442]: 3.f scrub starts 2024-02-05T06:55:12.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:12 smithi028 ceph-mon[29442]: 3.f scrub ok 2024-02-05T06:55:12.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:12 smithi028 ceph-mon[29442]: 4.9 scrub starts 2024-02-05T06:55:12.450 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:12 smithi028 ceph-mon[29442]: 4.9 scrub ok 2024-02-05T06:55:12.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:12 smithi167 ceph-mon[32774]: 5.11 scrub starts 2024-02-05T06:55:12.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:12 smithi167 ceph-mon[32774]: 5.11 scrub ok 2024-02-05T06:55:12.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:12 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:55:12.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:12 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:55:12.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:12 smithi167 ceph-mon[32774]: 6.f scrub starts 2024-02-05T06:55:12.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:12 smithi167 ceph-mon[32774]: from='client.46127 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-02-05T06:55:12.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:12 smithi167 ceph-mon[32774]: osdmap e484: 8 total, 8 up, 8 in 2024-02-05T06:55:12.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:12 smithi167 ceph-mon[32774]: 6.f scrub ok 2024-02-05T06:55:12.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:12 smithi167 ceph-mon[32774]: 3.f scrub starts 2024-02-05T06:55:12.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:12 smithi167 ceph-mon[32774]: 3.f scrub ok 2024-02-05T06:55:12.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:12 smithi167 ceph-mon[32774]: 4.9 scrub starts 2024-02-05T06:55:12.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:12 smithi167 ceph-mon[32774]: 4.9 scrub ok 2024-02-05T06:55:12.896 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-02-05T06:55:12.896 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd pool set .mgr pgp_num 21 2024-02-05T06:55:12.927 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:55:12 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:55:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:55:13.160 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:55:13.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:13 smithi028 ceph-mon[29442]: pgmap v1036: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T06:55:13.608 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:13 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:55:13.608 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:13 smithi028 ceph-mon[37984]: pgmap v1036: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T06:55:13.608 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:13 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:55:13.657 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:13 smithi167 ceph-mon[32774]: pgmap v1036: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T06:55:13.658 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:13 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:55:13.983 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:55:13 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:55:14.664 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:14 smithi028 ceph-mon[37984]: 1.2 deep-scrub starts 2024-02-05T06:55:14.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:14 smithi028 ceph-mon[37984]: 1.2 deep-scrub ok 2024-02-05T06:55:14.665 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:14 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:55:14.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:14 smithi028 ceph-mon[29442]: 1.2 deep-scrub starts 2024-02-05T06:55:14.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:14 smithi028 ceph-mon[29442]: 1.2 deep-scrub ok 2024-02-05T06:55:14.665 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:14 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:55:14.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:14 smithi167 ceph-mon[32774]: 1.2 deep-scrub starts 2024-02-05T06:55:14.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:14 smithi167 ceph-mon[32774]: 1.2 deep-scrub ok 2024-02-05T06:55:14.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:14 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:55:15.354 INFO:teuthology.orchestra.run.smithi028.stderr:set pool 1 pgp_num to 21 2024-02-05T06:55:15.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:15 smithi028 ceph-mon[29442]: 1.1 scrub starts 2024-02-05T06:55:15.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:15 smithi028 ceph-mon[29442]: 1.1 scrub ok 2024-02-05T06:55:15.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:15 smithi028 ceph-mon[29442]: 6.b deep-scrub starts 2024-02-05T06:55:15.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:15 smithi028 ceph-mon[29442]: 6.b deep-scrub ok 2024-02-05T06:55:15.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:15 smithi028 ceph-mon[29442]: pgmap v1037: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T06:55:15.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:15 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2802579537' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-02-05T06:55:15.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:15 smithi028 ceph-mon[37984]: 1.1 scrub starts 2024-02-05T06:55:15.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:15 smithi028 ceph-mon[37984]: 1.1 scrub ok 2024-02-05T06:55:15.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:15 smithi028 ceph-mon[37984]: 6.b deep-scrub starts 2024-02-05T06:55:15.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:15 smithi028 ceph-mon[37984]: 6.b deep-scrub ok 2024-02-05T06:55:15.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:15 smithi028 ceph-mon[37984]: pgmap v1037: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T06:55:15.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:15 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2802579537' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-02-05T06:55:15.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:15 smithi167 ceph-mon[32774]: 1.1 scrub starts 2024-02-05T06:55:15.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:15 smithi167 ceph-mon[32774]: 1.1 scrub ok 2024-02-05T06:55:15.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:15 smithi167 ceph-mon[32774]: 6.b deep-scrub starts 2024-02-05T06:55:15.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:15 smithi167 ceph-mon[32774]: 6.b deep-scrub ok 2024-02-05T06:55:15.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:15 smithi167 ceph-mon[32774]: pgmap v1037: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T06:55:15.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:15 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2802579537' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-02-05T06:55:16.068 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-02-05T06:55:16.364 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:55:16.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:16 smithi028 ceph-mon[37984]: 2.2 scrub starts 2024-02-05T06:55:16.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:16 smithi028 ceph-mon[37984]: 2.2 scrub ok 2024-02-05T06:55:16.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:16 smithi028 ceph-mon[37984]: 4.1c scrub starts 2024-02-05T06:55:16.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:16 smithi028 ceph-mon[37984]: 4.1c scrub ok 2024-02-05T06:55:16.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:16 smithi028 ceph-mon[37984]: 3.3 scrub starts 2024-02-05T06:55:16.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:16 smithi028 ceph-mon[37984]: 3.3 scrub ok 2024-02-05T06:55:16.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:16 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2802579537' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-02-05T06:55:16.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:16 smithi028 ceph-mon[37984]: osdmap e485: 8 total, 8 up, 8 in 2024-02-05T06:55:16.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:16 smithi028 ceph-mon[29442]: 2.2 scrub starts 2024-02-05T06:55:16.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:16 smithi028 ceph-mon[29442]: 2.2 scrub ok 2024-02-05T06:55:16.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:16 smithi028 ceph-mon[29442]: 4.1c scrub starts 2024-02-05T06:55:16.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:16 smithi028 ceph-mon[29442]: 4.1c scrub ok 2024-02-05T06:55:16.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:16 smithi028 ceph-mon[29442]: 3.3 scrub starts 2024-02-05T06:55:16.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:16 smithi028 ceph-mon[29442]: 3.3 scrub ok 2024-02-05T06:55:16.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:16 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2802579537' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-02-05T06:55:16.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:16 smithi028 ceph-mon[29442]: osdmap e485: 8 total, 8 up, 8 in 2024-02-05T06:55:16.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:16 smithi167 ceph-mon[32774]: 2.2 scrub starts 2024-02-05T06:55:16.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:16 smithi167 ceph-mon[32774]: 2.2 scrub ok 2024-02-05T06:55:16.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:16 smithi167 ceph-mon[32774]: 4.1c scrub starts 2024-02-05T06:55:16.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:16 smithi167 ceph-mon[32774]: 4.1c scrub ok 2024-02-05T06:55:16.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:16 smithi167 ceph-mon[32774]: 3.3 scrub starts 2024-02-05T06:55:16.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:16 smithi167 ceph-mon[32774]: 3.3 scrub ok 2024-02-05T06:55:16.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:16 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2802579537' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-02-05T06:55:16.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:16 smithi167 ceph-mon[32774]: osdmap e485: 8 total, 8 up, 8 in 2024-02-05T06:55:17.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:17 smithi028 ceph-mon[29442]: 5.18 scrub starts 2024-02-05T06:55:17.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:17 smithi028 ceph-mon[29442]: 5.18 scrub ok 2024-02-05T06:55:17.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:17 smithi028 ceph-mon[29442]: 3.1e scrub starts 2024-02-05T06:55:17.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:17 smithi028 ceph-mon[29442]: 3.1e scrub ok 2024-02-05T06:55:17.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:17 smithi028 ceph-mon[29442]: pgmap v1039: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T06:55:17.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:17 smithi028 ceph-mon[29442]: 3.4 scrub starts 2024-02-05T06:55:17.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:17 smithi028 ceph-mon[29442]: 3.4 scrub ok 2024-02-05T06:55:17.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:17 smithi028 ceph-mon[37984]: 5.18 scrub starts 2024-02-05T06:55:17.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:17 smithi028 ceph-mon[37984]: 5.18 scrub ok 2024-02-05T06:55:17.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:17 smithi028 ceph-mon[37984]: 3.1e scrub starts 2024-02-05T06:55:17.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:17 smithi028 ceph-mon[37984]: 3.1e scrub ok 2024-02-05T06:55:17.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:17 smithi028 ceph-mon[37984]: pgmap v1039: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T06:55:17.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:17 smithi028 ceph-mon[37984]: 3.4 scrub starts 2024-02-05T06:55:17.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:17 smithi028 ceph-mon[37984]: 3.4 scrub ok 2024-02-05T06:55:17.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:17 smithi167 ceph-mon[32774]: 5.18 scrub starts 2024-02-05T06:55:17.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:17 smithi167 ceph-mon[32774]: 5.18 scrub ok 2024-02-05T06:55:17.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:17 smithi167 ceph-mon[32774]: 3.1e scrub starts 2024-02-05T06:55:17.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:17 smithi167 ceph-mon[32774]: 3.1e scrub ok 2024-02-05T06:55:17.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:17 smithi167 ceph-mon[32774]: pgmap v1039: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T06:55:17.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:17 smithi167 ceph-mon[32774]: 3.4 scrub starts 2024-02-05T06:55:17.734 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:17 smithi167 ceph-mon[32774]: 3.4 scrub ok 2024-02-05T06:55:18.171 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a: {} 2024-02-05T06:55:18.181 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b: {} 2024-02-05T06:55:18.191 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c: {} 2024-02-05T06:55:18.192 INFO:teuthology.orchestra.run.smithi028.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-05T06:55:18.192 INFO:teuthology.orchestra.run.smithi028.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-05T06:55:18.193 INFO:teuthology.orchestra.run.smithi028.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-02-05T06:55:18.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:18 smithi028 ceph-mon[29442]: 1.5 scrub starts 2024-02-05T06:55:18.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:18 smithi028 ceph-mon[29442]: 1.5 scrub ok 2024-02-05T06:55:18.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:18 smithi028 ceph-mon[29442]: 6.10 scrub starts 2024-02-05T06:55:18.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:18 smithi028 ceph-mon[29442]: 6.10 scrub ok 2024-02-05T06:55:18.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:18 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1978259668' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T06:55:18.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:18 smithi028 ceph-mon[37984]: 1.5 scrub starts 2024-02-05T06:55:18.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:18 smithi028 ceph-mon[37984]: 1.5 scrub ok 2024-02-05T06:55:18.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:18 smithi028 ceph-mon[37984]: 6.10 scrub starts 2024-02-05T06:55:18.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:18 smithi028 ceph-mon[37984]: 6.10 scrub ok 2024-02-05T06:55:18.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:18 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1978259668' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T06:55:18.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:18 smithi167 ceph-mon[32774]: 1.5 scrub starts 2024-02-05T06:55:18.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:18 smithi167 ceph-mon[32774]: 1.5 scrub ok 2024-02-05T06:55:18.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:18 smithi167 ceph-mon[32774]: 6.10 scrub starts 2024-02-05T06:55:18.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:18 smithi167 ceph-mon[32774]: 6.10 scrub ok 2024-02-05T06:55:18.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:18 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1978259668' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-05T06:55:18.788 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd reweight 0 1 2024-02-05T06:55:19.037 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:55:19.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:19 smithi028 ceph-mon[37984]: 1.14 deep-scrub starts 2024-02-05T06:55:19.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:19 smithi028 ceph-mon[37984]: 1.14 deep-scrub ok 2024-02-05T06:55:19.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:19 smithi028 ceph-mon[37984]: 4.11 scrub starts 2024-02-05T06:55:19.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:19 smithi028 ceph-mon[37984]: 4.11 scrub ok 2024-02-05T06:55:19.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:19 smithi028 ceph-mon[37984]: pgmap v1040: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T06:55:19.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:19 smithi028 ceph-mon[29442]: 1.14 deep-scrub starts 2024-02-05T06:55:19.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:19 smithi028 ceph-mon[29442]: 1.14 deep-scrub ok 2024-02-05T06:55:19.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:19 smithi028 ceph-mon[29442]: 4.11 scrub starts 2024-02-05T06:55:19.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:19 smithi028 ceph-mon[29442]: 4.11 scrub ok 2024-02-05T06:55:19.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:19 smithi028 ceph-mon[29442]: pgmap v1040: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T06:55:19.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:19 smithi167 ceph-mon[32774]: 1.14 deep-scrub starts 2024-02-05T06:55:19.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:19 smithi167 ceph-mon[32774]: 1.14 deep-scrub ok 2024-02-05T06:55:19.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:19 smithi167 ceph-mon[32774]: 4.11 scrub starts 2024-02-05T06:55:19.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:19 smithi167 ceph-mon[32774]: 4.11 scrub ok 2024-02-05T06:55:19.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:19 smithi167 ceph-mon[32774]: pgmap v1040: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T06:55:20.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:20 smithi028 ceph-mon[29442]: 6.13 scrub starts 2024-02-05T06:55:20.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:20 smithi028 ceph-mon[29442]: 6.13 scrub ok 2024-02-05T06:55:20.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:20 smithi028 ceph-mon[29442]: 5.d scrub starts 2024-02-05T06:55:20.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:20 smithi028 ceph-mon[29442]: 5.d scrub ok 2024-02-05T06:55:20.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:20 smithi028 ceph-mon[29442]: 6.12 scrub starts 2024-02-05T06:55:20.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:20 smithi028 ceph-mon[29442]: 6.12 scrub ok 2024-02-05T06:55:20.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:20 smithi028 ceph-mon[37984]: 6.13 scrub starts 2024-02-05T06:55:20.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:20 smithi028 ceph-mon[37984]: 6.13 scrub ok 2024-02-05T06:55:20.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:20 smithi028 ceph-mon[37984]: 5.d scrub starts 2024-02-05T06:55:20.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:20 smithi028 ceph-mon[37984]: 5.d scrub ok 2024-02-05T06:55:20.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:20 smithi028 ceph-mon[37984]: 6.12 scrub starts 2024-02-05T06:55:20.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:20 smithi028 ceph-mon[37984]: 6.12 scrub ok 2024-02-05T06:55:20.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:20 smithi167 ceph-mon[32774]: 6.13 scrub starts 2024-02-05T06:55:20.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:20 smithi167 ceph-mon[32774]: 6.13 scrub ok 2024-02-05T06:55:20.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:20 smithi167 ceph-mon[32774]: 5.d scrub starts 2024-02-05T06:55:20.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:20 smithi167 ceph-mon[32774]: 5.d scrub ok 2024-02-05T06:55:20.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:20 smithi167 ceph-mon[32774]: 6.12 scrub starts 2024-02-05T06:55:20.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:20 smithi167 ceph-mon[32774]: 6.12 scrub ok 2024-02-05T06:55:21.371 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.0 to 1 (10000) 2024-02-05T06:55:21.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:21 smithi028 ceph-mon[29442]: pgmap v1041: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T06:55:21.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:21 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2297126042' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T06:55:21.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:21 smithi028 ceph-mon[29442]: from='client.46151 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T06:55:21.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:21 smithi028 ceph-mon[37984]: pgmap v1041: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T06:55:21.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:21 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2297126042' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T06:55:21.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:21 smithi028 ceph-mon[37984]: from='client.46151 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T06:55:21.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:21 smithi167 ceph-mon[32774]: pgmap v1041: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T06:55:21.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:21 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2297126042' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T06:55:21.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:21 smithi167 ceph-mon[32774]: from='client.46151 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T06:55:22.241 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 0 1 2024-02-05T06:55:22.491 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:55:22.501 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:22 smithi028 ceph-mon[37984]: 4.12 scrub starts 2024-02-05T06:55:22.501 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:22 smithi028 ceph-mon[37984]: 4.12 scrub ok 2024-02-05T06:55:22.501 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:22 smithi028 ceph-mon[37984]: from='client.46151 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T06:55:22.501 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:22 smithi028 ceph-mon[37984]: osdmap e486: 8 total, 8 up, 8 in 2024-02-05T06:55:22.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:22 smithi028 ceph-mon[29442]: 4.12 scrub starts 2024-02-05T06:55:22.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:22 smithi028 ceph-mon[29442]: 4.12 scrub ok 2024-02-05T06:55:22.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:22 smithi028 ceph-mon[29442]: from='client.46151 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T06:55:22.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:22 smithi028 ceph-mon[29442]: osdmap e486: 8 total, 8 up, 8 in 2024-02-05T06:55:22.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:22 smithi167 ceph-mon[32774]: 4.12 scrub starts 2024-02-05T06:55:22.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:22 smithi167 ceph-mon[32774]: 4.12 scrub ok 2024-02-05T06:55:22.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:22 smithi167 ceph-mon[32774]: from='client.46151 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-02-05T06:55:22.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:22 smithi167 ceph-mon[32774]: osdmap e486: 8 total, 8 up, 8 in 2024-02-05T06:55:22.831 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:55:22 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:55:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:55:23.668 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:23 smithi167 ceph-mon[32774]: 3.1c scrub starts 2024-02-05T06:55:23.669 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:23 smithi167 ceph-mon[32774]: 3.1c scrub ok 2024-02-05T06:55:23.669 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:23 smithi167 ceph-mon[32774]: 3.19 scrub starts 2024-02-05T06:55:23.669 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:23 smithi167 ceph-mon[32774]: 3.19 scrub ok 2024-02-05T06:55:23.669 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:23 smithi167 ceph-mon[32774]: 5.1c scrub starts 2024-02-05T06:55:23.669 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:23 smithi167 ceph-mon[32774]: 5.1c scrub ok 2024-02-05T06:55:23.669 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:23 smithi167 ceph-mon[32774]: 6.18 scrub starts 2024-02-05T06:55:23.670 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:23 smithi167 ceph-mon[32774]: 6.18 scrub ok 2024-02-05T06:55:23.670 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:23 smithi167 ceph-mon[32774]: 2.1 scrub starts 2024-02-05T06:55:23.670 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:23 smithi167 ceph-mon[32774]: 2.1 scrub ok 2024-02-05T06:55:23.670 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:23 smithi167 ceph-mon[32774]: pgmap v1043: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:55:23.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:23 smithi028 ceph-mon[29442]: 3.1c scrub starts 2024-02-05T06:55:23.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:23 smithi028 ceph-mon[29442]: 3.1c scrub ok 2024-02-05T06:55:23.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:23 smithi028 ceph-mon[29442]: 3.19 scrub starts 2024-02-05T06:55:23.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:23 smithi028 ceph-mon[29442]: 3.19 scrub ok 2024-02-05T06:55:23.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:23 smithi028 ceph-mon[29442]: 5.1c scrub starts 2024-02-05T06:55:23.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:23 smithi028 ceph-mon[29442]: 5.1c scrub ok 2024-02-05T06:55:23.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:23 smithi028 ceph-mon[29442]: 6.18 scrub starts 2024-02-05T06:55:23.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:23 smithi028 ceph-mon[29442]: 6.18 scrub ok 2024-02-05T06:55:23.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:23 smithi028 ceph-mon[29442]: 2.1 scrub starts 2024-02-05T06:55:23.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:23 smithi028 ceph-mon[29442]: 2.1 scrub ok 2024-02-05T06:55:23.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:23 smithi028 ceph-mon[29442]: pgmap v1043: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:55:23.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:23 smithi028 ceph-mon[37984]: 3.1c scrub starts 2024-02-05T06:55:23.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:23 smithi028 ceph-mon[37984]: 3.1c scrub ok 2024-02-05T06:55:23.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:23 smithi028 ceph-mon[37984]: 3.19 scrub starts 2024-02-05T06:55:23.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:23 smithi028 ceph-mon[37984]: 3.19 scrub ok 2024-02-05T06:55:23.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:23 smithi028 ceph-mon[37984]: 5.1c scrub starts 2024-02-05T06:55:23.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:23 smithi028 ceph-mon[37984]: 5.1c scrub ok 2024-02-05T06:55:23.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:23 smithi028 ceph-mon[37984]: 6.18 scrub starts 2024-02-05T06:55:23.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:23 smithi028 ceph-mon[37984]: 6.18 scrub ok 2024-02-05T06:55:23.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:23 smithi028 ceph-mon[37984]: 2.1 scrub starts 2024-02-05T06:55:23.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:23 smithi028 ceph-mon[37984]: 2.1 scrub ok 2024-02-05T06:55:23.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:23 smithi028 ceph-mon[37984]: pgmap v1043: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:55:23.983 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:55:23 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:55:24.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:24 smithi028 ceph-mon[29442]: 5.6 scrub starts 2024-02-05T06:55:24.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:24 smithi028 ceph-mon[29442]: 5.6 scrub ok 2024-02-05T06:55:24.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:24 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:55:24.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:24 smithi028 ceph-mon[37984]: 5.6 scrub starts 2024-02-05T06:55:24.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:24 smithi028 ceph-mon[37984]: 5.6 scrub ok 2024-02-05T06:55:24.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:24 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:55:24.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:24 smithi167 ceph-mon[32774]: 5.6 scrub starts 2024-02-05T06:55:24.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:24 smithi167 ceph-mon[32774]: 5.6 scrub ok 2024-02-05T06:55:24.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:24 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:55:25.387 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-02-05T06:55:25.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:25 smithi028 ceph-mon[29442]: 6.a scrub starts 2024-02-05T06:55:25.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:25 smithi028 ceph-mon[29442]: 6.a scrub ok 2024-02-05T06:55:25.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:25 smithi028 ceph-mon[29442]: 4.0 scrub starts 2024-02-05T06:55:25.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:25 smithi028 ceph-mon[29442]: 4.0 scrub ok 2024-02-05T06:55:25.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:25 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3816226564' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T06:55:25.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:25 smithi028 ceph-mon[29442]: pgmap v1044: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:55:25.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:25 smithi028 ceph-mon[29442]: 5.10 scrub starts 2024-02-05T06:55:25.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:25 smithi028 ceph-mon[29442]: 5.10 scrub ok 2024-02-05T06:55:25.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:25 smithi028 ceph-mon[37984]: 6.a scrub starts 2024-02-05T06:55:25.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:25 smithi028 ceph-mon[37984]: 6.a scrub ok 2024-02-05T06:55:25.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:25 smithi028 ceph-mon[37984]: 4.0 scrub starts 2024-02-05T06:55:25.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:25 smithi028 ceph-mon[37984]: 4.0 scrub ok 2024-02-05T06:55:25.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:25 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3816226564' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T06:55:25.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:25 smithi028 ceph-mon[37984]: pgmap v1044: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:55:25.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:25 smithi028 ceph-mon[37984]: 5.10 scrub starts 2024-02-05T06:55:25.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:25 smithi028 ceph-mon[37984]: 5.10 scrub ok 2024-02-05T06:55:25.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:25 smithi167 ceph-mon[32774]: 6.a scrub starts 2024-02-05T06:55:25.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:25 smithi167 ceph-mon[32774]: 6.a scrub ok 2024-02-05T06:55:25.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:25 smithi167 ceph-mon[32774]: 4.0 scrub starts 2024-02-05T06:55:25.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:25 smithi167 ceph-mon[32774]: 4.0 scrub ok 2024-02-05T06:55:25.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:25 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3816226564' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-02-05T06:55:25.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:25 smithi167 ceph-mon[32774]: pgmap v1044: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:55:25.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:25 smithi167 ceph-mon[32774]: 5.10 scrub starts 2024-02-05T06:55:25.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:25 smithi167 ceph-mon[32774]: 5.10 scrub ok 2024-02-05T06:55:26.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd reweight 1 1 2024-02-05T06:55:26.380 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:55:26.635 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:26 smithi028 ceph-mon[29442]: 4.15 scrub starts 2024-02-05T06:55:26.635 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:26 smithi028 ceph-mon[29442]: 4.15 scrub ok 2024-02-05T06:55:26.635 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:26 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3816226564' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-05T06:55:26.635 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:26 smithi028 ceph-mon[29442]: osdmap e487: 8 total, 8 up, 8 in 2024-02-05T06:55:26.635 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:26 smithi028 ceph-mon[29442]: 3.e scrub starts 2024-02-05T06:55:26.635 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:26 smithi028 ceph-mon[29442]: 3.e scrub ok 2024-02-05T06:55:26.636 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:26 smithi028 ceph-mon[37984]: 4.15 scrub starts 2024-02-05T06:55:26.636 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:26 smithi028 ceph-mon[37984]: 4.15 scrub ok 2024-02-05T06:55:26.636 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:26 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3816226564' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-05T06:55:26.636 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:26 smithi028 ceph-mon[37984]: osdmap e487: 8 total, 8 up, 8 in 2024-02-05T06:55:26.636 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:26 smithi028 ceph-mon[37984]: 3.e scrub starts 2024-02-05T06:55:26.636 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:26 smithi028 ceph-mon[37984]: 3.e scrub ok 2024-02-05T06:55:26.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:26 smithi167 ceph-mon[32774]: 4.15 scrub starts 2024-02-05T06:55:26.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:26 smithi167 ceph-mon[32774]: 4.15 scrub ok 2024-02-05T06:55:26.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:26 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3816226564' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-02-05T06:55:26.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:26 smithi167 ceph-mon[32774]: osdmap e487: 8 total, 8 up, 8 in 2024-02-05T06:55:26.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:26 smithi167 ceph-mon[32774]: 3.e scrub starts 2024-02-05T06:55:26.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:26 smithi167 ceph-mon[32774]: 3.e scrub ok 2024-02-05T06:55:27.602 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:27 smithi028 ceph-mon[37984]: 3.c scrub starts 2024-02-05T06:55:27.602 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:27 smithi028 ceph-mon[37984]: 3.c scrub ok 2024-02-05T06:55:27.602 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:27 smithi028 ceph-mon[37984]: 4.10 scrub starts 2024-02-05T06:55:27.602 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:27 smithi028 ceph-mon[37984]: 4.10 scrub ok 2024-02-05T06:55:27.603 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:27 smithi028 ceph-mon[37984]: pgmap v1046: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:55:27.603 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:27 smithi028 ceph-mon[37984]: 3.11 scrub starts 2024-02-05T06:55:27.603 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:27 smithi028 ceph-mon[37984]: 3.11 scrub ok 2024-02-05T06:55:27.603 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:27 smithi028 ceph-mon[29442]: 3.c scrub starts 2024-02-05T06:55:27.603 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:27 smithi028 ceph-mon[29442]: 3.c scrub ok 2024-02-05T06:55:27.603 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:27 smithi028 ceph-mon[29442]: 4.10 scrub starts 2024-02-05T06:55:27.603 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:27 smithi028 ceph-mon[29442]: 4.10 scrub ok 2024-02-05T06:55:27.603 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:27 smithi028 ceph-mon[29442]: pgmap v1046: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:55:27.603 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:27 smithi028 ceph-mon[29442]: 3.11 scrub starts 2024-02-05T06:55:27.603 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:27 smithi028 ceph-mon[29442]: 3.11 scrub ok 2024-02-05T06:55:27.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:27 smithi167 ceph-mon[32774]: 3.c scrub starts 2024-02-05T06:55:27.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:27 smithi167 ceph-mon[32774]: 3.c scrub ok 2024-02-05T06:55:27.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:27 smithi167 ceph-mon[32774]: 4.10 scrub starts 2024-02-05T06:55:27.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:27 smithi167 ceph-mon[32774]: 4.10 scrub ok 2024-02-05T06:55:27.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:27 smithi167 ceph-mon[32774]: pgmap v1046: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:55:27.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:27 smithi167 ceph-mon[32774]: 3.11 scrub starts 2024-02-05T06:55:27.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:27 smithi167 ceph-mon[32774]: 3.11 scrub ok 2024-02-05T06:55:28.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:28 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:55:28.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:28 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:55:28.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:28 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:55:28.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:28 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:55:28.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:28 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:55:28.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:28 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:55:28.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:28 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:55:28.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:28 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:55:28.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:28 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:55:29.404 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.1 to 1 (10000) 2024-02-05T06:55:29.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:29 smithi028 ceph-mon[37984]: 5.7 scrub starts 2024-02-05T06:55:29.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:29 smithi028 ceph-mon[37984]: 5.7 scrub ok 2024-02-05T06:55:29.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:29 smithi028 ceph-mon[37984]: 3.1 deep-scrub starts 2024-02-05T06:55:29.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:29 smithi028 ceph-mon[37984]: 3.1 deep-scrub ok 2024-02-05T06:55:29.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:29 smithi028 ceph-mon[37984]: osdmap e488: 8 total, 8 up, 8 in 2024-02-05T06:55:29.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:29 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2672213342' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T06:55:29.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:29 smithi028 ceph-mon[37984]: from='client.46163 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T06:55:29.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:29 smithi028 ceph-mon[37984]: 5.b scrub starts 2024-02-05T06:55:29.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:29 smithi028 ceph-mon[37984]: 5.b scrub ok 2024-02-05T06:55:29.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:29 smithi028 ceph-mon[37984]: pgmap v1048: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-05T06:55:29.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:29 smithi028 ceph-mon[29442]: 5.7 scrub starts 2024-02-05T06:55:29.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:29 smithi028 ceph-mon[29442]: 5.7 scrub ok 2024-02-05T06:55:29.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:29 smithi028 ceph-mon[29442]: 3.1 deep-scrub starts 2024-02-05T06:55:29.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:29 smithi028 ceph-mon[29442]: 3.1 deep-scrub ok 2024-02-05T06:55:29.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:29 smithi028 ceph-mon[29442]: osdmap e488: 8 total, 8 up, 8 in 2024-02-05T06:55:29.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:29 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2672213342' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T06:55:29.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:29 smithi028 ceph-mon[29442]: from='client.46163 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T06:55:29.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:29 smithi028 ceph-mon[29442]: 5.b scrub starts 2024-02-05T06:55:29.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:29 smithi028 ceph-mon[29442]: 5.b scrub ok 2024-02-05T06:55:29.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:29 smithi028 ceph-mon[29442]: pgmap v1048: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-05T06:55:29.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:29 smithi167 ceph-mon[32774]: 5.7 scrub starts 2024-02-05T06:55:29.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:29 smithi167 ceph-mon[32774]: 5.7 scrub ok 2024-02-05T06:55:29.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:29 smithi167 ceph-mon[32774]: 3.1 deep-scrub starts 2024-02-05T06:55:29.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:29 smithi167 ceph-mon[32774]: 3.1 deep-scrub ok 2024-02-05T06:55:29.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:29 smithi167 ceph-mon[32774]: osdmap e488: 8 total, 8 up, 8 in 2024-02-05T06:55:29.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:29 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2672213342' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T06:55:29.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:29 smithi167 ceph-mon[32774]: from='client.46163 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T06:55:29.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:29 smithi167 ceph-mon[32774]: 5.b scrub starts 2024-02-05T06:55:29.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:29 smithi167 ceph-mon[32774]: 5.b scrub ok 2024-02-05T06:55:29.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:29 smithi167 ceph-mon[32774]: pgmap v1048: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-05T06:55:30.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 1 1 2024-02-05T06:55:30.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:30 smithi028 ceph-mon[37984]: 5.e scrub starts 2024-02-05T06:55:30.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:30 smithi028 ceph-mon[37984]: 5.e scrub ok 2024-02-05T06:55:30.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:30 smithi028 ceph-mon[37984]: 4.4 scrub starts 2024-02-05T06:55:30.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:30 smithi028 ceph-mon[37984]: 4.4 scrub ok 2024-02-05T06:55:30.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:30 smithi028 ceph-mon[37984]: from='client.46163 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T06:55:30.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:30 smithi028 ceph-mon[37984]: osdmap e489: 8 total, 8 up, 8 in 2024-02-05T06:55:30.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:30 smithi028 ceph-mon[37984]: 3.d scrub starts 2024-02-05T06:55:30.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:30 smithi028 ceph-mon[37984]: 3.d scrub ok 2024-02-05T06:55:30.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:30 smithi028 ceph-mon[29442]: 5.e scrub starts 2024-02-05T06:55:30.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:30 smithi028 ceph-mon[29442]: 5.e scrub ok 2024-02-05T06:55:30.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:30 smithi028 ceph-mon[29442]: 4.4 scrub starts 2024-02-05T06:55:30.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:30 smithi028 ceph-mon[29442]: 4.4 scrub ok 2024-02-05T06:55:30.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:30 smithi028 ceph-mon[29442]: from='client.46163 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T06:55:30.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:30 smithi028 ceph-mon[29442]: osdmap e489: 8 total, 8 up, 8 in 2024-02-05T06:55:30.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:30 smithi028 ceph-mon[29442]: 3.d scrub starts 2024-02-05T06:55:30.449 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:30 smithi028 ceph-mon[29442]: 3.d scrub ok 2024-02-05T06:55:30.493 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:55:30.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:30 smithi167 ceph-mon[32774]: 5.e scrub starts 2024-02-05T06:55:30.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:30 smithi167 ceph-mon[32774]: 5.e scrub ok 2024-02-05T06:55:30.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:30 smithi167 ceph-mon[32774]: 4.4 scrub starts 2024-02-05T06:55:30.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:30 smithi167 ceph-mon[32774]: 4.4 scrub ok 2024-02-05T06:55:30.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:30 smithi167 ceph-mon[32774]: from='client.46163 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-02-05T06:55:30.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:30 smithi167 ceph-mon[32774]: osdmap e489: 8 total, 8 up, 8 in 2024-02-05T06:55:30.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:30 smithi167 ceph-mon[32774]: 3.d scrub starts 2024-02-05T06:55:30.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:30 smithi167 ceph-mon[32774]: 3.d scrub ok 2024-02-05T06:55:31.660 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:31 smithi028 ceph-mon[37984]: 6.1 scrub starts 2024-02-05T06:55:31.660 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:31 smithi028 ceph-mon[37984]: 6.1 scrub ok 2024-02-05T06:55:31.660 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:31 smithi028 ceph-mon[37984]: 3.1b scrub starts 2024-02-05T06:55:31.660 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:31 smithi028 ceph-mon[37984]: 3.1b scrub ok 2024-02-05T06:55:31.660 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:31 smithi028 ceph-mon[37984]: 1.c scrub starts 2024-02-05T06:55:31.660 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:31 smithi028 ceph-mon[37984]: 1.c scrub ok 2024-02-05T06:55:31.660 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:31 smithi028 ceph-mon[37984]: pgmap v1050: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:55:31.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:31 smithi028 ceph-mon[29442]: 6.1 scrub starts 2024-02-05T06:55:31.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:31 smithi028 ceph-mon[29442]: 6.1 scrub ok 2024-02-05T06:55:31.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:31 smithi028 ceph-mon[29442]: 3.1b scrub starts 2024-02-05T06:55:31.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:31 smithi028 ceph-mon[29442]: 3.1b scrub ok 2024-02-05T06:55:31.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:31 smithi028 ceph-mon[29442]: 1.c scrub starts 2024-02-05T06:55:31.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:31 smithi028 ceph-mon[29442]: 1.c scrub ok 2024-02-05T06:55:31.661 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:31 smithi028 ceph-mon[29442]: pgmap v1050: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:55:31.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:31 smithi167 ceph-mon[32774]: 6.1 scrub starts 2024-02-05T06:55:31.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:31 smithi167 ceph-mon[32774]: 6.1 scrub ok 2024-02-05T06:55:31.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:31 smithi167 ceph-mon[32774]: 3.1b scrub starts 2024-02-05T06:55:31.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:31 smithi167 ceph-mon[32774]: 3.1b scrub ok 2024-02-05T06:55:31.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:31 smithi167 ceph-mon[32774]: 1.c scrub starts 2024-02-05T06:55:31.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:31 smithi167 ceph-mon[32774]: 1.c scrub ok 2024-02-05T06:55:31.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:31 smithi167 ceph-mon[32774]: pgmap v1050: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:55:32.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:32 smithi028 ceph-mon[29442]: 3.5 scrub starts 2024-02-05T06:55:32.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:32 smithi028 ceph-mon[29442]: 3.5 scrub ok 2024-02-05T06:55:32.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:32 smithi028 ceph-mon[29442]: 4.16 scrub starts 2024-02-05T06:55:32.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:32 smithi028 ceph-mon[29442]: 4.16 scrub ok 2024-02-05T06:55:32.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:32 smithi028 ceph-mon[29442]: 6.5 scrub starts 2024-02-05T06:55:32.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:32 smithi028 ceph-mon[29442]: 6.5 scrub ok 2024-02-05T06:55:32.699 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:55:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:55:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:55:32.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:32 smithi028 ceph-mon[37984]: 3.5 scrub starts 2024-02-05T06:55:32.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:32 smithi028 ceph-mon[37984]: 3.5 scrub ok 2024-02-05T06:55:32.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:32 smithi028 ceph-mon[37984]: 4.16 scrub starts 2024-02-05T06:55:32.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:32 smithi028 ceph-mon[37984]: 4.16 scrub ok 2024-02-05T06:55:32.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:32 smithi028 ceph-mon[37984]: 6.5 scrub starts 2024-02-05T06:55:32.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:32 smithi028 ceph-mon[37984]: 6.5 scrub ok 2024-02-05T06:55:32.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:32 smithi167 ceph-mon[32774]: 3.5 scrub starts 2024-02-05T06:55:32.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:32 smithi167 ceph-mon[32774]: 3.5 scrub ok 2024-02-05T06:55:32.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:32 smithi167 ceph-mon[32774]: 4.16 scrub starts 2024-02-05T06:55:32.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:32 smithi167 ceph-mon[32774]: 4.16 scrub ok 2024-02-05T06:55:32.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:32 smithi167 ceph-mon[32774]: 6.5 scrub starts 2024-02-05T06:55:32.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:32 smithi167 ceph-mon[32774]: 6.5 scrub ok 2024-02-05T06:55:33.417 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-02-05T06:55:33.679 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:33 smithi167 ceph-mon[32774]: 6.4 scrub starts 2024-02-05T06:55:33.679 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:33 smithi167 ceph-mon[32774]: 6.4 scrub ok 2024-02-05T06:55:33.679 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:33 smithi167 ceph-mon[32774]: 3.8 scrub starts 2024-02-05T06:55:33.679 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:33 smithi167 ceph-mon[32774]: 3.8 scrub ok 2024-02-05T06:55:33.679 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:33 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3253849559' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T06:55:33.679 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:33 smithi167 ceph-mon[32774]: from='client.46169 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T06:55:33.679 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:33 smithi167 ceph-mon[32774]: pgmap v1051: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-05T06:55:33.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:33 smithi028 ceph-mon[29442]: 6.4 scrub starts 2024-02-05T06:55:33.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:33 smithi028 ceph-mon[29442]: 6.4 scrub ok 2024-02-05T06:55:33.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:33 smithi028 ceph-mon[29442]: 3.8 scrub starts 2024-02-05T06:55:33.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:33 smithi028 ceph-mon[29442]: 3.8 scrub ok 2024-02-05T06:55:33.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:33 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3253849559' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T06:55:33.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:33 smithi028 ceph-mon[29442]: from='client.46169 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T06:55:33.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:33 smithi028 ceph-mon[29442]: pgmap v1051: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-05T06:55:33.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:33 smithi028 ceph-mon[37984]: 6.4 scrub starts 2024-02-05T06:55:33.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:33 smithi028 ceph-mon[37984]: 6.4 scrub ok 2024-02-05T06:55:33.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:33 smithi028 ceph-mon[37984]: 3.8 scrub starts 2024-02-05T06:55:33.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:33 smithi028 ceph-mon[37984]: 3.8 scrub ok 2024-02-05T06:55:33.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:33 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3253849559' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T06:55:33.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:33 smithi028 ceph-mon[37984]: from='client.46169 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-02-05T06:55:33.700 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:33 smithi028 ceph-mon[37984]: pgmap v1051: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-02-05T06:55:33.982 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:55:33 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:55:34.155 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd reweight 2 1 2024-02-05T06:55:34.402 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:55:34.667 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:34 smithi028 ceph-mon[29442]: 3.18 scrub starts 2024-02-05T06:55:34.667 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:34 smithi028 ceph-mon[29442]: 3.18 scrub ok 2024-02-05T06:55:34.667 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:34 smithi028 ceph-mon[29442]: 4.19 scrub starts 2024-02-05T06:55:34.667 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:34 smithi028 ceph-mon[29442]: 4.19 scrub ok 2024-02-05T06:55:34.667 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:34 smithi028 ceph-mon[29442]: from='client.46169 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-05T06:55:34.667 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:34 smithi028 ceph-mon[29442]: osdmap e490: 8 total, 8 up, 8 in 2024-02-05T06:55:34.667 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:34 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:55:34.667 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:34 smithi028 ceph-mon[29442]: 6.3 scrub starts 2024-02-05T06:55:34.668 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:34 smithi028 ceph-mon[29442]: 6.3 scrub ok 2024-02-05T06:55:34.668 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:34 smithi028 ceph-mon[37984]: 3.18 scrub starts 2024-02-05T06:55:34.668 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:34 smithi028 ceph-mon[37984]: 3.18 scrub ok 2024-02-05T06:55:34.668 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:34 smithi028 ceph-mon[37984]: 4.19 scrub starts 2024-02-05T06:55:34.668 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:34 smithi028 ceph-mon[37984]: 4.19 scrub ok 2024-02-05T06:55:34.668 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:34 smithi028 ceph-mon[37984]: from='client.46169 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-05T06:55:34.668 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:34 smithi028 ceph-mon[37984]: osdmap e490: 8 total, 8 up, 8 in 2024-02-05T06:55:34.668 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:34 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:55:34.668 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:34 smithi028 ceph-mon[37984]: 6.3 scrub starts 2024-02-05T06:55:34.669 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:34 smithi028 ceph-mon[37984]: 6.3 scrub ok 2024-02-05T06:55:34.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:34 smithi167 ceph-mon[32774]: 3.18 scrub starts 2024-02-05T06:55:34.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:34 smithi167 ceph-mon[32774]: 3.18 scrub ok 2024-02-05T06:55:34.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:34 smithi167 ceph-mon[32774]: 4.19 scrub starts 2024-02-05T06:55:34.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:34 smithi167 ceph-mon[32774]: 4.19 scrub ok 2024-02-05T06:55:34.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:34 smithi167 ceph-mon[32774]: from='client.46169 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-02-05T06:55:34.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:34 smithi167 ceph-mon[32774]: osdmap e490: 8 total, 8 up, 8 in 2024-02-05T06:55:34.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:34 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:55:34.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:34 smithi167 ceph-mon[32774]: 6.3 scrub starts 2024-02-05T06:55:34.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:34 smithi167 ceph-mon[32774]: 6.3 scrub ok 2024-02-05T06:55:35.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:35 smithi028 ceph-mon[29442]: 6.d scrub starts 2024-02-05T06:55:35.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:35 smithi028 ceph-mon[29442]: 6.d scrub ok 2024-02-05T06:55:35.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:35 smithi028 ceph-mon[29442]: 3.6 scrub starts 2024-02-05T06:55:35.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:35 smithi028 ceph-mon[29442]: 3.6 scrub ok 2024-02-05T06:55:35.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:35 smithi028 ceph-mon[29442]: 1.4 scrub starts 2024-02-05T06:55:35.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:35 smithi028 ceph-mon[29442]: 1.4 scrub ok 2024-02-05T06:55:35.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:35 smithi028 ceph-mon[29442]: pgmap v1053: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 819 B/s rd, 0 op/s 2024-02-05T06:55:35.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:35 smithi028 ceph-mon[37984]: 6.d scrub starts 2024-02-05T06:55:35.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:35 smithi028 ceph-mon[37984]: 6.d scrub ok 2024-02-05T06:55:35.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:35 smithi028 ceph-mon[37984]: 3.6 scrub starts 2024-02-05T06:55:35.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:35 smithi028 ceph-mon[37984]: 3.6 scrub ok 2024-02-05T06:55:35.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:35 smithi028 ceph-mon[37984]: 1.4 scrub starts 2024-02-05T06:55:35.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:35 smithi028 ceph-mon[37984]: 1.4 scrub ok 2024-02-05T06:55:35.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:35 smithi028 ceph-mon[37984]: pgmap v1053: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 819 B/s rd, 0 op/s 2024-02-05T06:55:35.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:35 smithi167 ceph-mon[32774]: 6.d scrub starts 2024-02-05T06:55:35.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:35 smithi167 ceph-mon[32774]: 6.d scrub ok 2024-02-05T06:55:35.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:35 smithi167 ceph-mon[32774]: 3.6 scrub starts 2024-02-05T06:55:35.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:35 smithi167 ceph-mon[32774]: 3.6 scrub ok 2024-02-05T06:55:35.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:35 smithi167 ceph-mon[32774]: 1.4 scrub starts 2024-02-05T06:55:35.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:35 smithi167 ceph-mon[32774]: 1.4 scrub ok 2024-02-05T06:55:35.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:35 smithi167 ceph-mon[32774]: pgmap v1053: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 819 B/s rd, 0 op/s 2024-02-05T06:55:36.571 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.2 to 1 (10000) 2024-02-05T06:55:36.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:36 smithi028 ceph-mon[29442]: 4.1e scrub starts 2024-02-05T06:55:36.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:36 smithi028 ceph-mon[29442]: 4.1e scrub ok 2024-02-05T06:55:36.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:36 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3428118202' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T06:55:36.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:36 smithi028 ceph-mon[29442]: from='client.46175 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T06:55:36.727 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:36 smithi028 ceph-mon[37984]: 4.1e scrub starts 2024-02-05T06:55:36.727 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:36 smithi028 ceph-mon[37984]: 4.1e scrub ok 2024-02-05T06:55:36.727 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:36 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3428118202' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T06:55:36.727 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:36 smithi028 ceph-mon[37984]: from='client.46175 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T06:55:36.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:36 smithi167 ceph-mon[32774]: 4.1e scrub starts 2024-02-05T06:55:36.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:36 smithi167 ceph-mon[32774]: 4.1e scrub ok 2024-02-05T06:55:36.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:36 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3428118202' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T06:55:36.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:36 smithi167 ceph-mon[32774]: from='client.46175 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T06:55:37.213 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 2 1 2024-02-05T06:55:37.462 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:55:37.916 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:37 smithi028 ceph-mon[29442]: 3.10 deep-scrub starts 2024-02-05T06:55:37.916 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:37 smithi028 ceph-mon[29442]: 3.10 deep-scrub ok 2024-02-05T06:55:37.916 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:37 smithi028 ceph-mon[29442]: from='client.46175 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T06:55:37.916 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:37 smithi028 ceph-mon[29442]: osdmap e491: 8 total, 8 up, 8 in 2024-02-05T06:55:37.916 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:37 smithi028 ceph-mon[29442]: pgmap v1055: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 706 B/s rd, 0 op/s 2024-02-05T06:55:37.916 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:37 smithi028 ceph-mon[37984]: 3.10 deep-scrub starts 2024-02-05T06:55:37.916 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:37 smithi028 ceph-mon[37984]: 3.10 deep-scrub ok 2024-02-05T06:55:37.917 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:37 smithi028 ceph-mon[37984]: from='client.46175 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T06:55:37.917 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:37 smithi028 ceph-mon[37984]: osdmap e491: 8 total, 8 up, 8 in 2024-02-05T06:55:37.917 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:37 smithi028 ceph-mon[37984]: pgmap v1055: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 706 B/s rd, 0 op/s 2024-02-05T06:55:37.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:37 smithi167 ceph-mon[32774]: 3.10 deep-scrub starts 2024-02-05T06:55:37.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:37 smithi167 ceph-mon[32774]: 3.10 deep-scrub ok 2024-02-05T06:55:37.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:37 smithi167 ceph-mon[32774]: from='client.46175 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-02-05T06:55:37.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:37 smithi167 ceph-mon[32774]: osdmap e491: 8 total, 8 up, 8 in 2024-02-05T06:55:37.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:37 smithi167 ceph-mon[32774]: pgmap v1055: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 706 B/s rd, 0 op/s 2024-02-05T06:55:38.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:38 smithi028 ceph-mon[29442]: 6.19 scrub starts 2024-02-05T06:55:38.938 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:38 smithi028 ceph-mon[29442]: 6.19 scrub ok 2024-02-05T06:55:38.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:38 smithi028 ceph-mon[37984]: 6.19 scrub starts 2024-02-05T06:55:38.938 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:38 smithi028 ceph-mon[37984]: 6.19 scrub ok 2024-02-05T06:55:38.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:38 smithi167 ceph-mon[32774]: 6.19 scrub starts 2024-02-05T06:55:38.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:38 smithi167 ceph-mon[32774]: 6.19 scrub ok 2024-02-05T06:55:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-02-05T06:55:39.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:39 smithi167 ceph-mon[32774]: 4.c deep-scrub starts 2024-02-05T06:55:39.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:39 smithi167 ceph-mon[32774]: 4.c deep-scrub ok 2024-02-05T06:55:39.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:39 smithi167 ceph-mon[32774]: pgmap v1056: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:55:39.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:39 smithi167 ceph-mon[32774]: 5.4 scrub starts 2024-02-05T06:55:39.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:39 smithi167 ceph-mon[32774]: 5.4 scrub ok 2024-02-05T06:55:39.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:39 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1680998391' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T06:55:40.100 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:39 smithi028 ceph-mon[37984]: 4.c deep-scrub starts 2024-02-05T06:55:40.100 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:39 smithi028 ceph-mon[37984]: 4.c deep-scrub ok 2024-02-05T06:55:40.100 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:39 smithi028 ceph-mon[37984]: pgmap v1056: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:55:40.101 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:39 smithi028 ceph-mon[37984]: 5.4 scrub starts 2024-02-05T06:55:40.101 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:39 smithi028 ceph-mon[37984]: 5.4 scrub ok 2024-02-05T06:55:40.101 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:39 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1680998391' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T06:55:40.101 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:39 smithi028 ceph-mon[29442]: 4.c deep-scrub starts 2024-02-05T06:55:40.101 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:39 smithi028 ceph-mon[29442]: 4.c deep-scrub ok 2024-02-05T06:55:40.101 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:39 smithi028 ceph-mon[29442]: pgmap v1056: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:55:40.101 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:39 smithi028 ceph-mon[29442]: 5.4 scrub starts 2024-02-05T06:55:40.101 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:39 smithi028 ceph-mon[29442]: 5.4 scrub ok 2024-02-05T06:55:40.101 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:39 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1680998391' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-02-05T06:55:40.722 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd reweight 3 1 2024-02-05T06:55:40.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:40 smithi028 ceph-mon[29442]: 5.17 scrub starts 2024-02-05T06:55:40.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:40 smithi028 ceph-mon[29442]: 5.17 scrub ok 2024-02-05T06:55:40.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:40 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1680998391' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-05T06:55:40.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:40 smithi028 ceph-mon[29442]: osdmap e492: 8 total, 8 up, 8 in 2024-02-05T06:55:40.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:40 smithi028 ceph-mon[29442]: 3.13 scrub starts 2024-02-05T06:55:40.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:40 smithi028 ceph-mon[29442]: 3.13 scrub ok 2024-02-05T06:55:40.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:40 smithi028 ceph-mon[37984]: 5.17 scrub starts 2024-02-05T06:55:40.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:40 smithi028 ceph-mon[37984]: 5.17 scrub ok 2024-02-05T06:55:40.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:40 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1680998391' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-05T06:55:40.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:40 smithi028 ceph-mon[37984]: osdmap e492: 8 total, 8 up, 8 in 2024-02-05T06:55:40.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:40 smithi028 ceph-mon[37984]: 3.13 scrub starts 2024-02-05T06:55:40.949 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:40 smithi028 ceph-mon[37984]: 3.13 scrub ok 2024-02-05T06:55:40.974 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:55:40.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:40 smithi167 ceph-mon[32774]: 5.17 scrub starts 2024-02-05T06:55:40.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:40 smithi167 ceph-mon[32774]: 5.17 scrub ok 2024-02-05T06:55:40.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:40 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1680998391' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-02-05T06:55:40.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:40 smithi167 ceph-mon[32774]: osdmap e492: 8 total, 8 up, 8 in 2024-02-05T06:55:40.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:40 smithi167 ceph-mon[32774]: 3.13 scrub starts 2024-02-05T06:55:40.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:40 smithi167 ceph-mon[32774]: 3.13 scrub ok 2024-02-05T06:55:41.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:41 smithi167 ceph-mon[32774]: pgmap v1058: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 708 B/s rd, 0 op/s 2024-02-05T06:55:41.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:41 smithi167 ceph-mon[32774]: 3.1f scrub starts 2024-02-05T06:55:41.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:41 smithi167 ceph-mon[32774]: 3.1f scrub ok 2024-02-05T06:55:42.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:41 smithi028 ceph-mon[29442]: pgmap v1058: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 708 B/s rd, 0 op/s 2024-02-05T06:55:42.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:41 smithi028 ceph-mon[29442]: 3.1f scrub starts 2024-02-05T06:55:42.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:41 smithi028 ceph-mon[29442]: 3.1f scrub ok 2024-02-05T06:55:42.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:41 smithi028 ceph-mon[37984]: pgmap v1058: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 708 B/s rd, 0 op/s 2024-02-05T06:55:42.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:41 smithi028 ceph-mon[37984]: 3.1f scrub starts 2024-02-05T06:55:42.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:41 smithi028 ceph-mon[37984]: 3.1f scrub ok 2024-02-05T06:55:42.765 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:42 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:55:42.765 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:42 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:55:42.766 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:55:42 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:55:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:55:42.766 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:42 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:55:42.766 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:42 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:55:42.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:42 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:55:42.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:42 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:55:43.722 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.3 to 1 (10000) 2024-02-05T06:55:43.982 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:55:43 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:55:43.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:43 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:55:43.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:43 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:55:43.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:43 smithi167 ceph-mon[32774]: pgmap v1059: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:55:43.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:43 smithi167 ceph-mon[32774]: osdmap e493: 8 total, 8 up, 8 in 2024-02-05T06:55:43.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:43 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:55:43.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:43 smithi167 ceph-mon[32774]: 1.0 scrub starts 2024-02-05T06:55:43.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:43 smithi167 ceph-mon[32774]: 1.0 scrub ok 2024-02-05T06:55:43.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:43 smithi167 ceph-mon[32774]: 3.1a scrub starts 2024-02-05T06:55:43.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:43 smithi167 ceph-mon[32774]: 3.1a scrub ok 2024-02-05T06:55:43.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:43 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/756287506' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T06:55:44.036 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:43 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:55:44.036 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:43 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:55:44.036 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:43 smithi028 ceph-mon[37984]: pgmap v1059: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:55:44.037 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:43 smithi028 ceph-mon[37984]: osdmap e493: 8 total, 8 up, 8 in 2024-02-05T06:55:44.037 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:43 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:55:44.037 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:43 smithi028 ceph-mon[37984]: 1.0 scrub starts 2024-02-05T06:55:44.037 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:43 smithi028 ceph-mon[37984]: 1.0 scrub ok 2024-02-05T06:55:44.037 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:43 smithi028 ceph-mon[37984]: 3.1a scrub starts 2024-02-05T06:55:44.037 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:43 smithi028 ceph-mon[37984]: 3.1a scrub ok 2024-02-05T06:55:44.038 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:43 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/756287506' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T06:55:44.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:43 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:55:44.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:43 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:55:44.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:43 smithi028 ceph-mon[29442]: pgmap v1059: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:55:44.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:43 smithi028 ceph-mon[29442]: osdmap e493: 8 total, 8 up, 8 in 2024-02-05T06:55:44.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:43 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:55:44.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:43 smithi028 ceph-mon[29442]: 1.0 scrub starts 2024-02-05T06:55:44.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:43 smithi028 ceph-mon[29442]: 1.0 scrub ok 2024-02-05T06:55:44.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:43 smithi028 ceph-mon[29442]: 3.1a scrub starts 2024-02-05T06:55:44.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:43 smithi028 ceph-mon[29442]: 3.1a scrub ok 2024-02-05T06:55:44.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:43 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/756287506' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T06:55:44.480 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 3 1 2024-02-05T06:55:44.733 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:55:44.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:44 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:55:44.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:44 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/756287506' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-05T06:55:44.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:44 smithi167 ceph-mon[32774]: osdmap e494: 8 total, 8 up, 8 in 2024-02-05T06:55:44.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:44 smithi167 ceph-mon[32774]: 3.15 scrub starts 2024-02-05T06:55:44.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:44 smithi167 ceph-mon[32774]: 3.15 scrub ok 2024-02-05T06:55:44.986 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:44 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:55:44.986 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:44 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/756287506' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-05T06:55:44.986 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:44 smithi028 ceph-mon[29442]: osdmap e494: 8 total, 8 up, 8 in 2024-02-05T06:55:44.987 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:44 smithi028 ceph-mon[29442]: 3.15 scrub starts 2024-02-05T06:55:44.987 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:44 smithi028 ceph-mon[29442]: 3.15 scrub ok 2024-02-05T06:55:44.987 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:44 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:55:44.987 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:44 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/756287506' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-02-05T06:55:44.987 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:44 smithi028 ceph-mon[37984]: osdmap e494: 8 total, 8 up, 8 in 2024-02-05T06:55:44.987 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:44 smithi028 ceph-mon[37984]: 3.15 scrub starts 2024-02-05T06:55:44.987 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:44 smithi028 ceph-mon[37984]: 3.15 scrub ok 2024-02-05T06:55:45.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:45 smithi028 ceph-mon[29442]: pgmap v1062: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:55:45.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:45 smithi028 ceph-mon[29442]: 5.1a scrub starts 2024-02-05T06:55:45.895 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:45 smithi028 ceph-mon[29442]: 5.1a scrub ok 2024-02-05T06:55:45.895 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:45 smithi028 ceph-mon[37984]: pgmap v1062: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:55:45.895 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:45 smithi028 ceph-mon[37984]: 5.1a scrub starts 2024-02-05T06:55:45.895 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:45 smithi028 ceph-mon[37984]: 5.1a scrub ok 2024-02-05T06:55:45.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:45 smithi167 ceph-mon[32774]: pgmap v1062: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:55:45.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:45 smithi167 ceph-mon[32774]: 5.1a scrub starts 2024-02-05T06:55:45.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:45 smithi167 ceph-mon[32774]: 5.1a scrub ok 2024-02-05T06:55:46.735 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-02-05T06:55:47.098 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:46 smithi028 ceph-mon[29442]: 5.8 scrub starts 2024-02-05T06:55:47.099 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:46 smithi028 ceph-mon[29442]: 5.8 scrub ok 2024-02-05T06:55:47.099 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:46 smithi028 ceph-mon[29442]: 4.1a scrub starts 2024-02-05T06:55:47.099 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:46 smithi028 ceph-mon[29442]: 4.1a scrub ok 2024-02-05T06:55:47.099 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:46 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/368681108' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T06:55:47.099 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:46 smithi028 ceph-mon[37984]: 5.8 scrub starts 2024-02-05T06:55:47.099 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:46 smithi028 ceph-mon[37984]: 5.8 scrub ok 2024-02-05T06:55:47.099 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:46 smithi028 ceph-mon[37984]: 4.1a scrub starts 2024-02-05T06:55:47.099 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:46 smithi028 ceph-mon[37984]: 4.1a scrub ok 2024-02-05T06:55:47.099 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:46 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/368681108' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T06:55:47.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:46 smithi167 ceph-mon[32774]: 5.8 scrub starts 2024-02-05T06:55:47.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:46 smithi167 ceph-mon[32774]: 5.8 scrub ok 2024-02-05T06:55:47.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:46 smithi167 ceph-mon[32774]: 4.1a scrub starts 2024-02-05T06:55:47.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:46 smithi167 ceph-mon[32774]: 4.1a scrub ok 2024-02-05T06:55:47.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:46 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/368681108' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-02-05T06:55:47.431 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd reweight 4 1 2024-02-05T06:55:47.679 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:55:48.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:47 smithi028 ceph-mon[29442]: pgmap v1063: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 737 B/s rd, 0 op/s 2024-02-05T06:55:48.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:47 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/368681108' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-05T06:55:48.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:47 smithi028 ceph-mon[29442]: osdmap e495: 8 total, 8 up, 8 in 2024-02-05T06:55:48.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:47 smithi028 ceph-mon[29442]: 3.9 scrub starts 2024-02-05T06:55:48.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:47 smithi028 ceph-mon[29442]: 3.9 scrub ok 2024-02-05T06:55:48.015 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:47 smithi028 ceph-mon[37984]: pgmap v1063: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 737 B/s rd, 0 op/s 2024-02-05T06:55:48.015 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:47 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/368681108' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-05T06:55:48.015 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:47 smithi028 ceph-mon[37984]: osdmap e495: 8 total, 8 up, 8 in 2024-02-05T06:55:48.015 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:47 smithi028 ceph-mon[37984]: 3.9 scrub starts 2024-02-05T06:55:48.015 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:47 smithi028 ceph-mon[37984]: 3.9 scrub ok 2024-02-05T06:55:48.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:47 smithi167 ceph-mon[32774]: pgmap v1063: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 737 B/s rd, 0 op/s 2024-02-05T06:55:48.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:47 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/368681108' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-02-05T06:55:48.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:47 smithi167 ceph-mon[32774]: osdmap e495: 8 total, 8 up, 8 in 2024-02-05T06:55:48.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:47 smithi167 ceph-mon[32774]: 3.9 scrub starts 2024-02-05T06:55:48.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:47 smithi167 ceph-mon[32774]: 3.9 scrub ok 2024-02-05T06:55:49.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:48 smithi028 ceph-mon[29442]: 6.15 scrub starts 2024-02-05T06:55:49.003 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:48 smithi028 ceph-mon[29442]: 6.15 scrub ok 2024-02-05T06:55:49.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:48 smithi028 ceph-mon[37984]: 6.15 scrub starts 2024-02-05T06:55:49.004 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:48 smithi028 ceph-mon[37984]: 6.15 scrub ok 2024-02-05T06:55:49.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:48 smithi167 ceph-mon[32774]: 6.15 scrub starts 2024-02-05T06:55:49.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:48 smithi167 ceph-mon[32774]: 6.15 scrub ok 2024-02-05T06:55:49.746 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.4 to 1 (10000) 2024-02-05T06:55:50.059 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:49 smithi028 ceph-mon[29442]: pgmap v1065: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:55:50.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:49 smithi028 ceph-mon[29442]: 3.0 deep-scrub starts 2024-02-05T06:55:50.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:49 smithi028 ceph-mon[29442]: 3.0 deep-scrub ok 2024-02-05T06:55:50.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:49 smithi028 ceph-mon[29442]: 1.f scrub starts 2024-02-05T06:55:50.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:49 smithi028 ceph-mon[29442]: 1.f scrub ok 2024-02-05T06:55:50.060 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:49 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3570046743' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T06:55:50.060 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:49 smithi028 ceph-mon[37984]: pgmap v1065: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:55:50.060 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:49 smithi028 ceph-mon[37984]: 3.0 deep-scrub starts 2024-02-05T06:55:50.060 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:49 smithi028 ceph-mon[37984]: 3.0 deep-scrub ok 2024-02-05T06:55:50.060 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:49 smithi028 ceph-mon[37984]: 1.f scrub starts 2024-02-05T06:55:50.060 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:49 smithi028 ceph-mon[37984]: 1.f scrub ok 2024-02-05T06:55:50.060 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:49 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3570046743' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T06:55:50.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:49 smithi167 ceph-mon[32774]: pgmap v1065: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:55:50.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:49 smithi167 ceph-mon[32774]: 3.0 deep-scrub starts 2024-02-05T06:55:50.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:49 smithi167 ceph-mon[32774]: 3.0 deep-scrub ok 2024-02-05T06:55:50.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:49 smithi167 ceph-mon[32774]: 1.f scrub starts 2024-02-05T06:55:50.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:49 smithi167 ceph-mon[32774]: 1.f scrub ok 2024-02-05T06:55:50.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:49 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3570046743' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T06:55:50.474 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 4 1 2024-02-05T06:55:50.735 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:55:50.998 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:50 smithi028 ceph-mon[29442]: 4.1d scrub starts 2024-02-05T06:55:50.999 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:50 smithi028 ceph-mon[29442]: 4.1d scrub ok 2024-02-05T06:55:50.999 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:50 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3570046743' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-05T06:55:50.999 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:50 smithi028 ceph-mon[29442]: osdmap e496: 8 total, 8 up, 8 in 2024-02-05T06:55:50.999 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:50 smithi028 ceph-mon[29442]: 2.0 scrub starts 2024-02-05T06:55:50.999 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:50 smithi028 ceph-mon[29442]: 2.0 scrub ok 2024-02-05T06:55:50.999 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:50 smithi028 ceph-mon[29442]: 6.9 scrub starts 2024-02-05T06:55:50.999 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:50 smithi028 ceph-mon[37984]: 4.1d scrub starts 2024-02-05T06:55:50.999 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:50 smithi028 ceph-mon[37984]: 4.1d scrub ok 2024-02-05T06:55:50.999 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:50 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3570046743' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-05T06:55:51.000 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:50 smithi028 ceph-mon[37984]: osdmap e496: 8 total, 8 up, 8 in 2024-02-05T06:55:51.000 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:50 smithi028 ceph-mon[37984]: 2.0 scrub starts 2024-02-05T06:55:51.000 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:50 smithi028 ceph-mon[37984]: 2.0 scrub ok 2024-02-05T06:55:51.000 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:50 smithi028 ceph-mon[37984]: 6.9 scrub starts 2024-02-05T06:55:51.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:50 smithi167 ceph-mon[32774]: 4.1d scrub starts 2024-02-05T06:55:51.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:50 smithi167 ceph-mon[32774]: 4.1d scrub ok 2024-02-05T06:55:51.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:50 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3570046743' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-02-05T06:55:51.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:50 smithi167 ceph-mon[32774]: osdmap e496: 8 total, 8 up, 8 in 2024-02-05T06:55:51.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:50 smithi167 ceph-mon[32774]: 2.0 scrub starts 2024-02-05T06:55:51.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:50 smithi167 ceph-mon[32774]: 2.0 scrub ok 2024-02-05T06:55:51.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:50 smithi167 ceph-mon[32774]: 6.9 scrub starts 2024-02-05T06:55:52.094 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:51 smithi028 ceph-mon[29442]: 4.18 scrub starts 2024-02-05T06:55:52.094 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:51 smithi028 ceph-mon[29442]: 4.18 scrub ok 2024-02-05T06:55:52.094 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:51 smithi028 ceph-mon[29442]: pgmap v1067: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2024-02-05T06:55:52.094 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:51 smithi028 ceph-mon[29442]: 6.9 scrub ok 2024-02-05T06:55:52.094 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:51 smithi028 ceph-mon[29442]: 5.1f scrub starts 2024-02-05T06:55:52.094 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:51 smithi028 ceph-mon[29442]: 5.1f scrub ok 2024-02-05T06:55:52.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:51 smithi028 ceph-mon[37984]: 4.18 scrub starts 2024-02-05T06:55:52.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:51 smithi028 ceph-mon[37984]: 4.18 scrub ok 2024-02-05T06:55:52.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:51 smithi028 ceph-mon[37984]: pgmap v1067: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2024-02-05T06:55:52.095 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:51 smithi028 ceph-mon[37984]: 6.9 scrub ok 2024-02-05T06:55:52.095 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:51 smithi028 ceph-mon[37984]: 5.1f scrub starts 2024-02-05T06:55:52.095 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:51 smithi028 ceph-mon[37984]: 5.1f scrub ok 2024-02-05T06:55:52.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:51 smithi167 ceph-mon[32774]: 4.18 scrub starts 2024-02-05T06:55:52.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:51 smithi167 ceph-mon[32774]: 4.18 scrub ok 2024-02-05T06:55:52.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:51 smithi167 ceph-mon[32774]: pgmap v1067: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2024-02-05T06:55:52.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:51 smithi167 ceph-mon[32774]: 6.9 scrub ok 2024-02-05T06:55:52.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:51 smithi167 ceph-mon[32774]: 5.1f scrub starts 2024-02-05T06:55:52.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:51 smithi167 ceph-mon[32774]: 5.1f scrub ok 2024-02-05T06:55:52.698 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:55:52 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:55:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:55:52.756 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-02-05T06:55:53.051 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:52 smithi028 ceph-mon[37984]: 3.b scrub starts 2024-02-05T06:55:53.051 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:52 smithi028 ceph-mon[37984]: 3.b scrub ok 2024-02-05T06:55:53.051 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:52 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3352457077' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T06:55:53.051 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:52 smithi028 ceph-mon[29442]: 3.b scrub starts 2024-02-05T06:55:53.051 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:52 smithi028 ceph-mon[29442]: 3.b scrub ok 2024-02-05T06:55:53.051 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:52 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3352457077' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T06:55:53.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:52 smithi167 ceph-mon[32774]: 3.b scrub starts 2024-02-05T06:55:53.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:52 smithi167 ceph-mon[32774]: 3.b scrub ok 2024-02-05T06:55:53.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:52 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3352457077' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-02-05T06:55:53.463 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd reweight 5 1 2024-02-05T06:55:53.710 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:55:53.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:53 smithi167 ceph-mon[32774]: pgmap v1068: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:55:53.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:53 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3352457077' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-05T06:55:53.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:53 smithi167 ceph-mon[32774]: osdmap e497: 8 total, 8 up, 8 in 2024-02-05T06:55:53.983 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:55:53 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:55:54.055 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:53 smithi028 ceph-mon[29442]: pgmap v1068: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:55:54.055 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:53 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3352457077' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-05T06:55:54.056 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:53 smithi028 ceph-mon[29442]: osdmap e497: 8 total, 8 up, 8 in 2024-02-05T06:55:54.056 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:53 smithi028 ceph-mon[37984]: pgmap v1068: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:55:54.056 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:53 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3352457077' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-02-05T06:55:54.056 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:53 smithi028 ceph-mon[37984]: osdmap e497: 8 total, 8 up, 8 in 2024-02-05T06:55:54.793 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:54 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:55:54.794 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:54 smithi028 ceph-mon[37984]: osdmap e498: 8 total, 8 up, 8 in 2024-02-05T06:55:54.794 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:54 smithi028 ceph-mon[37984]: 5.19 scrub starts 2024-02-05T06:55:54.794 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:54 smithi028 ceph-mon[37984]: 5.19 scrub ok 2024-02-05T06:55:54.794 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:54 smithi028 ceph-mon[37984]: 4.1f scrub starts 2024-02-05T06:55:54.794 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:54 smithi028 ceph-mon[37984]: 4.1f scrub ok 2024-02-05T06:55:54.794 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:54 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:55:54.794 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:54 smithi028 ceph-mon[29442]: osdmap e498: 8 total, 8 up, 8 in 2024-02-05T06:55:54.794 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:54 smithi028 ceph-mon[29442]: 5.19 scrub starts 2024-02-05T06:55:54.794 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:54 smithi028 ceph-mon[29442]: 5.19 scrub ok 2024-02-05T06:55:54.794 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:54 smithi028 ceph-mon[29442]: 4.1f scrub starts 2024-02-05T06:55:54.794 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:54 smithi028 ceph-mon[29442]: 4.1f scrub ok 2024-02-05T06:55:55.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:54 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:55:55.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:54 smithi167 ceph-mon[32774]: osdmap e498: 8 total, 8 up, 8 in 2024-02-05T06:55:55.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:54 smithi167 ceph-mon[32774]: 5.19 scrub starts 2024-02-05T06:55:55.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:54 smithi167 ceph-mon[32774]: 5.19 scrub ok 2024-02-05T06:55:55.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:54 smithi167 ceph-mon[32774]: 4.1f scrub starts 2024-02-05T06:55:55.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:54 smithi167 ceph-mon[32774]: 4.1f scrub ok 2024-02-05T06:55:56.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:55 smithi028 ceph-mon[29442]: 1.e scrub starts 2024-02-05T06:55:56.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:55 smithi028 ceph-mon[29442]: 1.e scrub ok 2024-02-05T06:55:56.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:55 smithi028 ceph-mon[29442]: pgmap v1071: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:55:56.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:55 smithi028 ceph-mon[29442]: 1.13 scrub starts 2024-02-05T06:55:56.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:55 smithi028 ceph-mon[29442]: 1.13 scrub ok 2024-02-05T06:55:56.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:55 smithi028 ceph-mon[29442]: 5.2 scrub starts 2024-02-05T06:55:56.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:55 smithi028 ceph-mon[29442]: 5.2 scrub ok 2024-02-05T06:55:56.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:55 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2695422618' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T06:55:56.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:55 smithi028 ceph-mon[37984]: 1.e scrub starts 2024-02-05T06:55:56.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:55 smithi028 ceph-mon[37984]: 1.e scrub ok 2024-02-05T06:55:56.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:55 smithi028 ceph-mon[37984]: pgmap v1071: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:55:56.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:55 smithi028 ceph-mon[37984]: 1.13 scrub starts 2024-02-05T06:55:56.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:55 smithi028 ceph-mon[37984]: 1.13 scrub ok 2024-02-05T06:55:56.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:55 smithi028 ceph-mon[37984]: 5.2 scrub starts 2024-02-05T06:55:56.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:55 smithi028 ceph-mon[37984]: 5.2 scrub ok 2024-02-05T06:55:56.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:55 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2695422618' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T06:55:56.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:55 smithi167 ceph-mon[32774]: 1.e scrub starts 2024-02-05T06:55:56.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:55 smithi167 ceph-mon[32774]: 1.e scrub ok 2024-02-05T06:55:56.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:55 smithi167 ceph-mon[32774]: pgmap v1071: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:55:56.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:55 smithi167 ceph-mon[32774]: 1.13 scrub starts 2024-02-05T06:55:56.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:55 smithi167 ceph-mon[32774]: 1.13 scrub ok 2024-02-05T06:55:56.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:55 smithi167 ceph-mon[32774]: 5.2 scrub starts 2024-02-05T06:55:56.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:55 smithi167 ceph-mon[32774]: 5.2 scrub ok 2024-02-05T06:55:56.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:55 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2695422618' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T06:55:56.353 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.5 to 1 (10000) 2024-02-05T06:55:56.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:56 smithi028 ceph-mon[37984]: 5.14 scrub starts 2024-02-05T06:55:56.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:56 smithi028 ceph-mon[37984]: 5.14 scrub ok 2024-02-05T06:55:56.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:56 smithi028 ceph-mon[37984]: 1.5 scrub starts 2024-02-05T06:55:56.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:56 smithi028 ceph-mon[37984]: 1.5 scrub ok 2024-02-05T06:55:56.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:56 smithi028 ceph-mon[37984]: 1.a scrub starts 2024-02-05T06:55:56.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:56 smithi028 ceph-mon[37984]: 1.a scrub ok 2024-02-05T06:55:56.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:56 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2695422618' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-05T06:55:56.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:56 smithi028 ceph-mon[37984]: osdmap e499: 8 total, 8 up, 8 in 2024-02-05T06:55:56.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:56 smithi028 ceph-mon[29442]: 5.14 scrub starts 2024-02-05T06:55:56.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:56 smithi028 ceph-mon[29442]: 5.14 scrub ok 2024-02-05T06:55:56.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:56 smithi028 ceph-mon[29442]: 1.5 scrub starts 2024-02-05T06:55:56.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:56 smithi028 ceph-mon[29442]: 1.5 scrub ok 2024-02-05T06:55:56.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:56 smithi028 ceph-mon[29442]: 1.a scrub starts 2024-02-05T06:55:56.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:56 smithi028 ceph-mon[29442]: 1.a scrub ok 2024-02-05T06:55:56.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:56 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2695422618' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-05T06:55:56.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:56 smithi028 ceph-mon[29442]: osdmap e499: 8 total, 8 up, 8 in 2024-02-05T06:55:57.019 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 5 1 2024-02-05T06:55:57.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:56 smithi167 ceph-mon[32774]: 5.14 scrub starts 2024-02-05T06:55:57.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:56 smithi167 ceph-mon[32774]: 5.14 scrub ok 2024-02-05T06:55:57.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:56 smithi167 ceph-mon[32774]: 1.5 scrub starts 2024-02-05T06:55:57.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:56 smithi167 ceph-mon[32774]: 1.5 scrub ok 2024-02-05T06:55:57.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:56 smithi167 ceph-mon[32774]: 1.a scrub starts 2024-02-05T06:55:57.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:56 smithi167 ceph-mon[32774]: 1.a scrub ok 2024-02-05T06:55:57.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:56 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2695422618' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-02-05T06:55:57.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:56 smithi167 ceph-mon[32774]: osdmap e499: 8 total, 8 up, 8 in 2024-02-05T06:55:57.273 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:55:57.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:57 smithi028 ceph-mon[37984]: 3.7 scrub starts 2024-02-05T06:55:57.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:57 smithi028 ceph-mon[37984]: 3.7 scrub ok 2024-02-05T06:55:57.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:57 smithi028 ceph-mon[37984]: 3.12 scrub starts 2024-02-05T06:55:57.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:57 smithi028 ceph-mon[37984]: 3.12 scrub ok 2024-02-05T06:55:57.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:57 smithi028 ceph-mon[37984]: pgmap v1073: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-02-05T06:55:57.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:57 smithi028 ceph-mon[37984]: 5.1d deep-scrub starts 2024-02-05T06:55:57.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:57 smithi028 ceph-mon[37984]: 5.1d deep-scrub ok 2024-02-05T06:55:57.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:57 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:55:57.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:57 smithi028 ceph-mon[29442]: 3.7 scrub starts 2024-02-05T06:55:57.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:57 smithi028 ceph-mon[29442]: 3.7 scrub ok 2024-02-05T06:55:57.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:57 smithi028 ceph-mon[29442]: 3.12 scrub starts 2024-02-05T06:55:57.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:57 smithi028 ceph-mon[29442]: 3.12 scrub ok 2024-02-05T06:55:57.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:57 smithi028 ceph-mon[29442]: pgmap v1073: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-02-05T06:55:57.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:57 smithi028 ceph-mon[29442]: 5.1d deep-scrub starts 2024-02-05T06:55:57.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:57 smithi028 ceph-mon[29442]: 5.1d deep-scrub ok 2024-02-05T06:55:57.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:57 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:55:58.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:57 smithi167 ceph-mon[32774]: 3.7 scrub starts 2024-02-05T06:55:58.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:57 smithi167 ceph-mon[32774]: 3.7 scrub ok 2024-02-05T06:55:58.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:57 smithi167 ceph-mon[32774]: 3.12 scrub starts 2024-02-05T06:55:58.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:57 smithi167 ceph-mon[32774]: 3.12 scrub ok 2024-02-05T06:55:58.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:57 smithi167 ceph-mon[32774]: pgmap v1073: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-02-05T06:55:58.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:57 smithi167 ceph-mon[32774]: 5.1d deep-scrub starts 2024-02-05T06:55:58.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:57 smithi167 ceph-mon[32774]: 5.1d deep-scrub ok 2024-02-05T06:55:58.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:57 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:55:59.071 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:58 smithi028 ceph-mon[29442]: 4.14 deep-scrub starts 2024-02-05T06:55:59.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:58 smithi028 ceph-mon[29442]: 4.14 deep-scrub ok 2024-02-05T06:55:59.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:58 smithi028 ceph-mon[29442]: 3.17 deep-scrub starts 2024-02-05T06:55:59.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:58 smithi028 ceph-mon[29442]: 3.17 deep-scrub ok 2024-02-05T06:55:59.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:58 smithi028 ceph-mon[29442]: 4.8 scrub starts 2024-02-05T06:55:59.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:58 smithi028 ceph-mon[29442]: 4.8 scrub ok 2024-02-05T06:55:59.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:58 smithi028 ceph-mon[29442]: 1.d scrub starts 2024-02-05T06:55:59.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:58 smithi028 ceph-mon[29442]: 1.d scrub ok 2024-02-05T06:55:59.072 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:58 smithi028 ceph-mon[29442]: 5.12 scrub starts 2024-02-05T06:55:59.073 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:58 smithi028 ceph-mon[29442]: 5.12 scrub ok 2024-02-05T06:55:59.073 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:58 smithi028 ceph-mon[37984]: 4.14 deep-scrub starts 2024-02-05T06:55:59.073 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:58 smithi028 ceph-mon[37984]: 4.14 deep-scrub ok 2024-02-05T06:55:59.073 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:58 smithi028 ceph-mon[37984]: 3.17 deep-scrub starts 2024-02-05T06:55:59.073 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:58 smithi028 ceph-mon[37984]: 3.17 deep-scrub ok 2024-02-05T06:55:59.073 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:58 smithi028 ceph-mon[37984]: 4.8 scrub starts 2024-02-05T06:55:59.073 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:58 smithi028 ceph-mon[37984]: 4.8 scrub ok 2024-02-05T06:55:59.073 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:58 smithi028 ceph-mon[37984]: 1.d scrub starts 2024-02-05T06:55:59.073 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:58 smithi028 ceph-mon[37984]: 1.d scrub ok 2024-02-05T06:55:59.073 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:58 smithi028 ceph-mon[37984]: 5.12 scrub starts 2024-02-05T06:55:59.073 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:58 smithi028 ceph-mon[37984]: 5.12 scrub ok 2024-02-05T06:55:59.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:58 smithi167 ceph-mon[32774]: 4.14 deep-scrub starts 2024-02-05T06:55:59.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:58 smithi167 ceph-mon[32774]: 4.14 deep-scrub ok 2024-02-05T06:55:59.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:58 smithi167 ceph-mon[32774]: 3.17 deep-scrub starts 2024-02-05T06:55:59.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:58 smithi167 ceph-mon[32774]: 3.17 deep-scrub ok 2024-02-05T06:55:59.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:58 smithi167 ceph-mon[32774]: 4.8 scrub starts 2024-02-05T06:55:59.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:58 smithi167 ceph-mon[32774]: 4.8 scrub ok 2024-02-05T06:55:59.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:58 smithi167 ceph-mon[32774]: 1.d scrub starts 2024-02-05T06:55:59.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:58 smithi167 ceph-mon[32774]: 1.d scrub ok 2024-02-05T06:55:59.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:58 smithi167 ceph-mon[32774]: 5.12 scrub starts 2024-02-05T06:55:59.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:58 smithi167 ceph-mon[32774]: 5.12 scrub ok 2024-02-05T06:55:59.791 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-02-05T06:56:00.110 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:59 smithi028 ceph-mon[37984]: 5.1e deep-scrub starts 2024-02-05T06:56:00.110 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:59 smithi028 ceph-mon[37984]: 5.1e deep-scrub ok 2024-02-05T06:56:00.111 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:59 smithi028 ceph-mon[37984]: pgmap v1074: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-02-05T06:56:00.111 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:59 smithi028 ceph-mon[37984]: 1.9 deep-scrub starts 2024-02-05T06:56:00.111 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:59 smithi028 ceph-mon[37984]: 1.9 deep-scrub ok 2024-02-05T06:56:00.111 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:59 smithi028 ceph-mon[37984]: 6.1d deep-scrub starts 2024-02-05T06:56:00.111 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:59 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/841249423' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T06:56:00.111 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:55:59 smithi028 ceph-mon[37984]: from='client.46217 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T06:56:00.111 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:59 smithi028 ceph-mon[29442]: 5.1e deep-scrub starts 2024-02-05T06:56:00.111 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:59 smithi028 ceph-mon[29442]: 5.1e deep-scrub ok 2024-02-05T06:56:00.111 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:59 smithi028 ceph-mon[29442]: pgmap v1074: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-02-05T06:56:00.111 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:59 smithi028 ceph-mon[29442]: 1.9 deep-scrub starts 2024-02-05T06:56:00.111 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:59 smithi028 ceph-mon[29442]: 1.9 deep-scrub ok 2024-02-05T06:56:00.111 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:59 smithi028 ceph-mon[29442]: 6.1d deep-scrub starts 2024-02-05T06:56:00.112 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:59 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/841249423' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T06:56:00.112 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:55:59 smithi028 ceph-mon[29442]: from='client.46217 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T06:56:00.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:59 smithi167 ceph-mon[32774]: 5.1e deep-scrub starts 2024-02-05T06:56:00.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:59 smithi167 ceph-mon[32774]: 5.1e deep-scrub ok 2024-02-05T06:56:00.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:59 smithi167 ceph-mon[32774]: pgmap v1074: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-02-05T06:56:00.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:59 smithi167 ceph-mon[32774]: 1.9 deep-scrub starts 2024-02-05T06:56:00.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:59 smithi167 ceph-mon[32774]: 1.9 deep-scrub ok 2024-02-05T06:56:00.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:59 smithi167 ceph-mon[32774]: 6.1d deep-scrub starts 2024-02-05T06:56:00.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:59 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/841249423' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T06:56:00.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:55:59 smithi167 ceph-mon[32774]: from='client.46217 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-02-05T06:56:00.553 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd reweight 6 1 2024-02-05T06:56:00.809 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:56:01.062 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:00 smithi028 ceph-mon[37984]: 6.1d deep-scrub ok 2024-02-05T06:56:01.062 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:00 smithi028 ceph-mon[37984]: 3.2 scrub starts 2024-02-05T06:56:01.063 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:00 smithi028 ceph-mon[37984]: 3.2 scrub ok 2024-02-05T06:56:01.063 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:00 smithi028 ceph-mon[37984]: from='client.46217 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-05T06:56:01.063 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:00 smithi028 ceph-mon[37984]: osdmap e500: 8 total, 8 up, 8 in 2024-02-05T06:56:01.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:00 smithi028 ceph-mon[29442]: 6.1d deep-scrub ok 2024-02-05T06:56:01.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:00 smithi028 ceph-mon[29442]: 3.2 scrub starts 2024-02-05T06:56:01.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:00 smithi028 ceph-mon[29442]: 3.2 scrub ok 2024-02-05T06:56:01.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:00 smithi028 ceph-mon[29442]: from='client.46217 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-05T06:56:01.063 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:00 smithi028 ceph-mon[29442]: osdmap e500: 8 total, 8 up, 8 in 2024-02-05T06:56:01.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:00 smithi167 ceph-mon[32774]: 6.1d deep-scrub ok 2024-02-05T06:56:01.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:00 smithi167 ceph-mon[32774]: 3.2 scrub starts 2024-02-05T06:56:01.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:00 smithi167 ceph-mon[32774]: 3.2 scrub ok 2024-02-05T06:56:01.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:00 smithi167 ceph-mon[32774]: from='client.46217 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-02-05T06:56:01.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:00 smithi167 ceph-mon[32774]: osdmap e500: 8 total, 8 up, 8 in 2024-02-05T06:56:01.900 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:01 smithi028 ceph-mon[29442]: pgmap v1076: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-02-05T06:56:01.900 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:01 smithi028 ceph-mon[29442]: 6.0 scrub starts 2024-02-05T06:56:01.901 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:01 smithi028 ceph-mon[29442]: 1.8 scrub starts 2024-02-05T06:56:01.901 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:01 smithi028 ceph-mon[29442]: 1.8 scrub ok 2024-02-05T06:56:01.901 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:01 smithi028 ceph-mon[29442]: 6.0 scrub ok 2024-02-05T06:56:01.901 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:01 smithi028 ceph-mon[37984]: pgmap v1076: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-02-05T06:56:01.901 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:01 smithi028 ceph-mon[37984]: 6.0 scrub starts 2024-02-05T06:56:01.901 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:01 smithi028 ceph-mon[37984]: 1.8 scrub starts 2024-02-05T06:56:01.901 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:01 smithi028 ceph-mon[37984]: 1.8 scrub ok 2024-02-05T06:56:01.901 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:01 smithi028 ceph-mon[37984]: 6.0 scrub ok 2024-02-05T06:56:02.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:01 smithi167 ceph-mon[32774]: pgmap v1076: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 0 B/s, 0 objects/s recovering 2024-02-05T06:56:02.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:01 smithi167 ceph-mon[32774]: 6.0 scrub starts 2024-02-05T06:56:02.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:01 smithi167 ceph-mon[32774]: 1.8 scrub starts 2024-02-05T06:56:02.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:01 smithi167 ceph-mon[32774]: 1.8 scrub ok 2024-02-05T06:56:02.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:01 smithi167 ceph-mon[32774]: 6.0 scrub ok 2024-02-05T06:56:02.797 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:56:02 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:56:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:56:02.803 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.6 to 1 (10000) 2024-02-05T06:56:03.166 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:02 smithi028 ceph-mon[37984]: 5.1b scrub starts 2024-02-05T06:56:03.166 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:02 smithi028 ceph-mon[37984]: 5.1b scrub ok 2024-02-05T06:56:03.166 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:02 smithi028 ceph-mon[37984]: 4.2 scrub starts 2024-02-05T06:56:03.166 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:02 smithi028 ceph-mon[37984]: 4.2 scrub ok 2024-02-05T06:56:03.166 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:02 smithi028 ceph-mon[37984]: 5.a scrub starts 2024-02-05T06:56:03.166 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:02 smithi028 ceph-mon[37984]: 5.a scrub ok 2024-02-05T06:56:03.166 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:02 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2441703611' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T06:56:03.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:02 smithi028 ceph-mon[29442]: 5.1b scrub starts 2024-02-05T06:56:03.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:02 smithi028 ceph-mon[29442]: 5.1b scrub ok 2024-02-05T06:56:03.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:02 smithi028 ceph-mon[29442]: 4.2 scrub starts 2024-02-05T06:56:03.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:02 smithi028 ceph-mon[29442]: 4.2 scrub ok 2024-02-05T06:56:03.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:02 smithi028 ceph-mon[29442]: 5.a scrub starts 2024-02-05T06:56:03.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:02 smithi028 ceph-mon[29442]: 5.a scrub ok 2024-02-05T06:56:03.167 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:02 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2441703611' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T06:56:03.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:02 smithi167 ceph-mon[32774]: 5.1b scrub starts 2024-02-05T06:56:03.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:02 smithi167 ceph-mon[32774]: 5.1b scrub ok 2024-02-05T06:56:03.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:02 smithi167 ceph-mon[32774]: 4.2 scrub starts 2024-02-05T06:56:03.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:02 smithi167 ceph-mon[32774]: 4.2 scrub ok 2024-02-05T06:56:03.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:02 smithi167 ceph-mon[32774]: 5.a scrub starts 2024-02-05T06:56:03.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:02 smithi167 ceph-mon[32774]: 5.a scrub ok 2024-02-05T06:56:03.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:02 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2441703611' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T06:56:03.604 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 6 1 2024-02-05T06:56:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:56:03.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:03 smithi028 ceph-mon[29442]: pgmap v1077: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T06:56:03.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:03 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2441703611' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-05T06:56:03.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:03 smithi028 ceph-mon[29442]: osdmap e501: 8 total, 8 up, 8 in 2024-02-05T06:56:03.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:03 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:56:03.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:03 smithi028 ceph-mon[37984]: pgmap v1077: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T06:56:03.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:03 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2441703611' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-05T06:56:03.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:03 smithi028 ceph-mon[37984]: osdmap e501: 8 total, 8 up, 8 in 2024-02-05T06:56:03.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:03 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:56:03.982 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:56:03 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:56:03.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:03 smithi167 ceph-mon[32774]: pgmap v1077: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-02-05T06:56:03.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:03 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2441703611' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-02-05T06:56:03.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:03 smithi167 ceph-mon[32774]: osdmap e501: 8 total, 8 up, 8 in 2024-02-05T06:56:03.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:03 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:56:05.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:04 smithi028 ceph-mon[29442]: 4.1b scrub starts 2024-02-05T06:56:05.142 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:04 smithi028 ceph-mon[29442]: 4.1b scrub ok 2024-02-05T06:56:05.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:04 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:56:05.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:04 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:56:05.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:04 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:56:05.143 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:04 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:56:05.143 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:04 smithi028 ceph-mon[37984]: 4.1b scrub starts 2024-02-05T06:56:05.143 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:04 smithi028 ceph-mon[37984]: 4.1b scrub ok 2024-02-05T06:56:05.143 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:04 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:56:05.143 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:04 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:56:05.143 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:04 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:56:05.143 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:04 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:56:05.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:04 smithi167 ceph-mon[32774]: 4.1b scrub starts 2024-02-05T06:56:05.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:04 smithi167 ceph-mon[32774]: 4.1b scrub ok 2024-02-05T06:56:05.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:04 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:56:05.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:04 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:56:05.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:04 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:56:05.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:04 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:56:06.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:05 smithi028 ceph-mon[29442]: 5.5 scrub starts 2024-02-05T06:56:06.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:05 smithi028 ceph-mon[29442]: 5.5 scrub ok 2024-02-05T06:56:06.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:05 smithi028 ceph-mon[29442]: pgmap v1079: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:56:06.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:05 smithi028 ceph-mon[29442]: 6.7 scrub starts 2024-02-05T06:56:06.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:05 smithi028 ceph-mon[29442]: 1.b scrub starts 2024-02-05T06:56:06.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:05 smithi028 ceph-mon[29442]: 1.b scrub ok 2024-02-05T06:56:06.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:05 smithi028 ceph-mon[29442]: 6.7 scrub ok 2024-02-05T06:56:06.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:05 smithi028 ceph-mon[29442]: 1.6 scrub starts 2024-02-05T06:56:06.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:05 smithi028 ceph-mon[29442]: 1.6 scrub ok 2024-02-05T06:56:06.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:05 smithi028 ceph-mon[37984]: 5.5 scrub starts 2024-02-05T06:56:06.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:05 smithi028 ceph-mon[37984]: 5.5 scrub ok 2024-02-05T06:56:06.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:05 smithi028 ceph-mon[37984]: pgmap v1079: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:56:06.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:05 smithi028 ceph-mon[37984]: 6.7 scrub starts 2024-02-05T06:56:06.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:05 smithi028 ceph-mon[37984]: 1.b scrub starts 2024-02-05T06:56:06.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:05 smithi028 ceph-mon[37984]: 1.b scrub ok 2024-02-05T06:56:06.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:05 smithi028 ceph-mon[37984]: 6.7 scrub ok 2024-02-05T06:56:06.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:05 smithi028 ceph-mon[37984]: 1.6 scrub starts 2024-02-05T06:56:06.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:05 smithi028 ceph-mon[37984]: 1.6 scrub ok 2024-02-05T06:56:06.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:05 smithi167 ceph-mon[32774]: 5.5 scrub starts 2024-02-05T06:56:06.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:05 smithi167 ceph-mon[32774]: 5.5 scrub ok 2024-02-05T06:56:06.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:05 smithi167 ceph-mon[32774]: pgmap v1079: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:56:06.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:05 smithi167 ceph-mon[32774]: 6.7 scrub starts 2024-02-05T06:56:06.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:05 smithi167 ceph-mon[32774]: 1.b scrub starts 2024-02-05T06:56:06.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:05 smithi167 ceph-mon[32774]: 1.b scrub ok 2024-02-05T06:56:06.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:05 smithi167 ceph-mon[32774]: 6.7 scrub ok 2024-02-05T06:56:06.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:05 smithi167 ceph-mon[32774]: 1.6 scrub starts 2024-02-05T06:56:06.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:05 smithi167 ceph-mon[32774]: 1.6 scrub ok 2024-02-05T06:56:06.904 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-02-05T06:56:07.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:06 smithi028 ceph-mon[29442]: 6.1b scrub starts 2024-02-05T06:56:07.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:06 smithi028 ceph-mon[29442]: 6.1b scrub ok 2024-02-05T06:56:07.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:06 smithi028 ceph-mon[29442]: 1.3 scrub starts 2024-02-05T06:56:07.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:06 smithi028 ceph-mon[29442]: 1.3 scrub ok 2024-02-05T06:56:07.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:06 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/4017809325' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T06:56:07.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:06 smithi028 ceph-mon[29442]: 6.8 scrub starts 2024-02-05T06:56:07.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:06 smithi028 ceph-mon[29442]: 6.8 scrub ok 2024-02-05T06:56:07.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:06 smithi028 ceph-mon[37984]: 6.1b scrub starts 2024-02-05T06:56:07.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:06 smithi028 ceph-mon[37984]: 6.1b scrub ok 2024-02-05T06:56:07.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:06 smithi028 ceph-mon[37984]: 1.3 scrub starts 2024-02-05T06:56:07.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:06 smithi028 ceph-mon[37984]: 1.3 scrub ok 2024-02-05T06:56:07.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:06 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/4017809325' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T06:56:07.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:06 smithi028 ceph-mon[37984]: 6.8 scrub starts 2024-02-05T06:56:07.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:06 smithi028 ceph-mon[37984]: 6.8 scrub ok 2024-02-05T06:56:07.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:06 smithi167 ceph-mon[32774]: 6.1b scrub starts 2024-02-05T06:56:07.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:06 smithi167 ceph-mon[32774]: 6.1b scrub ok 2024-02-05T06:56:07.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:06 smithi167 ceph-mon[32774]: 1.3 scrub starts 2024-02-05T06:56:07.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:06 smithi167 ceph-mon[32774]: 1.3 scrub ok 2024-02-05T06:56:07.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:06 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/4017809325' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-02-05T06:56:07.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:06 smithi167 ceph-mon[32774]: 6.8 scrub starts 2024-02-05T06:56:07.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:06 smithi167 ceph-mon[32774]: 6.8 scrub ok 2024-02-05T06:56:07.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd reweight 7 1 2024-02-05T06:56:07.914 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:56:08.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:07 smithi028 ceph-mon[29442]: pgmap v1080: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:56:08.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:07 smithi028 ceph-mon[29442]: 5.c scrub starts 2024-02-05T06:56:08.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:07 smithi028 ceph-mon[29442]: 5.c scrub ok 2024-02-05T06:56:08.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:07 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/4017809325' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-05T06:56:08.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:07 smithi028 ceph-mon[29442]: osdmap e502: 8 total, 8 up, 8 in 2024-02-05T06:56:08.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:07 smithi028 ceph-mon[37984]: pgmap v1080: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:56:08.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:07 smithi028 ceph-mon[37984]: 5.c scrub starts 2024-02-05T06:56:08.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:07 smithi028 ceph-mon[37984]: 5.c scrub ok 2024-02-05T06:56:08.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:07 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/4017809325' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-05T06:56:08.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:07 smithi028 ceph-mon[37984]: osdmap e502: 8 total, 8 up, 8 in 2024-02-05T06:56:08.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:07 smithi167 ceph-mon[32774]: pgmap v1080: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:56:08.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:07 smithi167 ceph-mon[32774]: 5.c scrub starts 2024-02-05T06:56:08.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:07 smithi167 ceph-mon[32774]: 5.c scrub ok 2024-02-05T06:56:08.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:07 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/4017809325' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-02-05T06:56:08.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:07 smithi167 ceph-mon[32774]: osdmap e502: 8 total, 8 up, 8 in 2024-02-05T06:56:09.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:08 smithi028 ceph-mon[29442]: 6.2 scrub starts 2024-02-05T06:56:09.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:08 smithi028 ceph-mon[29442]: 6.2 scrub ok 2024-02-05T06:56:09.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:08 smithi028 ceph-mon[37984]: 6.2 scrub starts 2024-02-05T06:56:09.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:08 smithi028 ceph-mon[37984]: 6.2 scrub ok 2024-02-05T06:56:09.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:08 smithi167 ceph-mon[32774]: 6.2 scrub starts 2024-02-05T06:56:09.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:08 smithi167 ceph-mon[32774]: 6.2 scrub ok 2024-02-05T06:56:10.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:09 smithi028 ceph-mon[29442]: 6.11 scrub starts 2024-02-05T06:56:10.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:09 smithi028 ceph-mon[29442]: 6.11 scrub ok 2024-02-05T06:56:10.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:09 smithi028 ceph-mon[29442]: pgmap v1082: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:56:10.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:09 smithi028 ceph-mon[37984]: 6.11 scrub starts 2024-02-05T06:56:10.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:09 smithi028 ceph-mon[37984]: 6.11 scrub ok 2024-02-05T06:56:10.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:09 smithi028 ceph-mon[37984]: pgmap v1082: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:56:10.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:09 smithi167 ceph-mon[32774]: 6.11 scrub starts 2024-02-05T06:56:10.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:09 smithi167 ceph-mon[32774]: 6.11 scrub ok 2024-02-05T06:56:10.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:09 smithi167 ceph-mon[32774]: pgmap v1082: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:56:10.918 INFO:teuthology.orchestra.run.smithi028.stderr:reweighted osd.7 to 1 (10000) 2024-02-05T06:56:11.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:10 smithi028 ceph-mon[37984]: 4.17 scrub starts 2024-02-05T06:56:11.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:10 smithi028 ceph-mon[37984]: 4.17 scrub ok 2024-02-05T06:56:11.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:10 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:56:11.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:10 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:56:11.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:10 smithi028 ceph-mon[37984]: 3.1d scrub starts 2024-02-05T06:56:11.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:10 smithi028 ceph-mon[37984]: 3.1d scrub ok 2024-02-05T06:56:11.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:10 smithi028 ceph-mon[37984]: osdmap e503: 8 total, 8 up, 8 in 2024-02-05T06:56:11.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:10 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/4095138053' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T06:56:11.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:10 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:56:11.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:10 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:56:11.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:10 smithi028 ceph-mon[37984]: 4.b scrub starts 2024-02-05T06:56:11.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:10 smithi028 ceph-mon[37984]: 4.b scrub ok 2024-02-05T06:56:11.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:10 smithi028 ceph-mon[29442]: 4.17 scrub starts 2024-02-05T06:56:11.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:10 smithi028 ceph-mon[29442]: 4.17 scrub ok 2024-02-05T06:56:11.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:10 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:56:11.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:10 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:56:11.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:10 smithi028 ceph-mon[29442]: 3.1d scrub starts 2024-02-05T06:56:11.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:10 smithi028 ceph-mon[29442]: 3.1d scrub ok 2024-02-05T06:56:11.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:10 smithi028 ceph-mon[29442]: osdmap e503: 8 total, 8 up, 8 in 2024-02-05T06:56:11.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:10 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/4095138053' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T06:56:11.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:10 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:56:11.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:10 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:56:11.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:10 smithi028 ceph-mon[29442]: 4.b scrub starts 2024-02-05T06:56:11.200 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:10 smithi028 ceph-mon[29442]: 4.b scrub ok 2024-02-05T06:56:11.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:10 smithi167 ceph-mon[32774]: 4.17 scrub starts 2024-02-05T06:56:11.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:10 smithi167 ceph-mon[32774]: 4.17 scrub ok 2024-02-05T06:56:11.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:10 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:56:11.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:10 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:56:11.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:10 smithi167 ceph-mon[32774]: 3.1d scrub starts 2024-02-05T06:56:11.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:10 smithi167 ceph-mon[32774]: 3.1d scrub ok 2024-02-05T06:56:11.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:10 smithi167 ceph-mon[32774]: osdmap e503: 8 total, 8 up, 8 in 2024-02-05T06:56:11.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:10 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/4095138053' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T06:56:11.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:10 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:56:11.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:10 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:56:11.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:10 smithi167 ceph-mon[32774]: 4.b scrub starts 2024-02-05T06:56:11.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:10 smithi167 ceph-mon[32774]: 4.b scrub ok 2024-02-05T06:56:11.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd primary-affinity 7 1 2024-02-05T06:56:11.996 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:56:12.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:11 smithi028 ceph-mon[29442]: 5.1 scrub starts 2024-02-05T06:56:12.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:11 smithi028 ceph-mon[29442]: 5.1 scrub ok 2024-02-05T06:56:12.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:11 smithi028 ceph-mon[29442]: pgmap v1084: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 652 B/s rd, 0 op/s 2024-02-05T06:56:12.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:11 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/4095138053' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-05T06:56:12.117 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:11 smithi028 ceph-mon[29442]: osdmap e504: 8 total, 8 up, 8 in 2024-02-05T06:56:12.117 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:11 smithi028 ceph-mon[37984]: 5.1 scrub starts 2024-02-05T06:56:12.117 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:11 smithi028 ceph-mon[37984]: 5.1 scrub ok 2024-02-05T06:56:12.117 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:11 smithi028 ceph-mon[37984]: pgmap v1084: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 652 B/s rd, 0 op/s 2024-02-05T06:56:12.117 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:11 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/4095138053' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-05T06:56:12.118 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:11 smithi028 ceph-mon[37984]: osdmap e504: 8 total, 8 up, 8 in 2024-02-05T06:56:12.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:11 smithi167 ceph-mon[32774]: 5.1 scrub starts 2024-02-05T06:56:12.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:11 smithi167 ceph-mon[32774]: 5.1 scrub ok 2024-02-05T06:56:12.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:11 smithi167 ceph-mon[32774]: pgmap v1084: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 652 B/s rd, 0 op/s 2024-02-05T06:56:12.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:11 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/4095138053' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-02-05T06:56:12.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:11 smithi167 ceph-mon[32774]: osdmap e504: 8 total, 8 up, 8 in 2024-02-05T06:56:12.698 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:56:12 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:56:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:56:13.184 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:12 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:56:13.184 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:12 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:56:13.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:12 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:56:13.982 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:56:13 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:56:13.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:13 smithi167 ceph-mon[32774]: 4.1 scrub starts 2024-02-05T06:56:13.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:13 smithi167 ceph-mon[32774]: 4.1 scrub ok 2024-02-05T06:56:13.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:13 smithi167 ceph-mon[32774]: pgmap v1086: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-05T06:56:13.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:13 smithi167 ceph-mon[32774]: 3.f deep-scrub starts 2024-02-05T06:56:13.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:13 smithi167 ceph-mon[32774]: 3.f deep-scrub ok 2024-02-05T06:56:14.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:13 smithi028 ceph-mon[29442]: 4.1 scrub starts 2024-02-05T06:56:14.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:13 smithi028 ceph-mon[29442]: 4.1 scrub ok 2024-02-05T06:56:14.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:13 smithi028 ceph-mon[29442]: pgmap v1086: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-05T06:56:14.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:13 smithi028 ceph-mon[29442]: 3.f deep-scrub starts 2024-02-05T06:56:14.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:13 smithi028 ceph-mon[29442]: 3.f deep-scrub ok 2024-02-05T06:56:14.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:13 smithi028 ceph-mon[37984]: 4.1 scrub starts 2024-02-05T06:56:14.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:13 smithi028 ceph-mon[37984]: 4.1 scrub ok 2024-02-05T06:56:14.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:13 smithi028 ceph-mon[37984]: pgmap v1086: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-02-05T06:56:14.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:13 smithi028 ceph-mon[37984]: 3.f deep-scrub starts 2024-02-05T06:56:14.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:13 smithi028 ceph-mon[37984]: 3.f deep-scrub ok 2024-02-05T06:56:14.930 INFO:teuthology.orchestra.run.smithi028.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-02-05T06:56:15.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:14 smithi028 ceph-mon[37984]: 6.17 scrub starts 2024-02-05T06:56:15.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:14 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:56:15.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:14 smithi028 ceph-mon[37984]: 6.17 scrub ok 2024-02-05T06:56:15.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:14 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1275603785' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T06:56:15.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:14 smithi028 ceph-mon[37984]: 4.a scrub starts 2024-02-05T06:56:15.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:14 smithi028 ceph-mon[37984]: 4.a scrub ok 2024-02-05T06:56:15.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:14 smithi028 ceph-mon[37984]: 1.11 scrub starts 2024-02-05T06:56:15.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:14 smithi028 ceph-mon[37984]: 1.11 scrub ok 2024-02-05T06:56:15.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:14 smithi028 ceph-mon[37984]: 6.f scrub starts 2024-02-05T06:56:15.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:14 smithi028 ceph-mon[29442]: 6.17 scrub starts 2024-02-05T06:56:15.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:14 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:56:15.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:14 smithi028 ceph-mon[29442]: 6.17 scrub ok 2024-02-05T06:56:15.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:14 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1275603785' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T06:56:15.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:14 smithi028 ceph-mon[29442]: 4.a scrub starts 2024-02-05T06:56:15.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:14 smithi028 ceph-mon[29442]: 4.a scrub ok 2024-02-05T06:56:15.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:14 smithi028 ceph-mon[29442]: 1.11 scrub starts 2024-02-05T06:56:15.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:14 smithi028 ceph-mon[29442]: 1.11 scrub ok 2024-02-05T06:56:15.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:14 smithi028 ceph-mon[29442]: 6.f scrub starts 2024-02-05T06:56:15.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:14 smithi167 ceph-mon[32774]: 6.17 scrub starts 2024-02-05T06:56:15.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:14 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:56:15.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:14 smithi167 ceph-mon[32774]: 6.17 scrub ok 2024-02-05T06:56:15.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:14 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1275603785' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-02-05T06:56:15.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:14 smithi167 ceph-mon[32774]: 4.a scrub starts 2024-02-05T06:56:15.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:14 smithi167 ceph-mon[32774]: 4.a scrub ok 2024-02-05T06:56:15.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:14 smithi167 ceph-mon[32774]: 1.11 scrub starts 2024-02-05T06:56:15.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:14 smithi167 ceph-mon[32774]: 1.11 scrub ok 2024-02-05T06:56:15.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:14 smithi167 ceph-mon[32774]: 6.f scrub starts 2024-02-05T06:56:15.703 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-02-05T06:56:15.703 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-02-05T06:56:15.703 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-02-05T06:56:15.703 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-02-05T06:56:15.703 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-05T06:56:15.703 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd dump --format=json 2024-02-05T06:56:15.955 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:56:16.186 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:15 smithi028 ceph-mon[37984]: pgmap v1087: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 660 B/s rd, 0 op/s 2024-02-05T06:56:16.186 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:15 smithi028 ceph-mon[37984]: 5.0 scrub starts 2024-02-05T06:56:16.186 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:15 smithi028 ceph-mon[37984]: 5.0 scrub ok 2024-02-05T06:56:16.186 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:15 smithi028 ceph-mon[37984]: 6.f scrub ok 2024-02-05T06:56:16.186 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:15 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1275603785' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-05T06:56:16.187 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:15 smithi028 ceph-mon[37984]: osdmap e505: 8 total, 8 up, 8 in 2024-02-05T06:56:16.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:15 smithi028 ceph-mon[29442]: pgmap v1087: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 660 B/s rd, 0 op/s 2024-02-05T06:56:16.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:15 smithi028 ceph-mon[29442]: 5.0 scrub starts 2024-02-05T06:56:16.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:15 smithi028 ceph-mon[29442]: 5.0 scrub ok 2024-02-05T06:56:16.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:15 smithi028 ceph-mon[29442]: 6.f scrub ok 2024-02-05T06:56:16.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:15 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1275603785' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-05T06:56:16.187 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:15 smithi028 ceph-mon[29442]: osdmap e505: 8 total, 8 up, 8 in 2024-02-05T06:56:16.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:15 smithi167 ceph-mon[32774]: pgmap v1087: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 660 B/s rd, 0 op/s 2024-02-05T06:56:16.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:15 smithi167 ceph-mon[32774]: 5.0 scrub starts 2024-02-05T06:56:16.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:15 smithi167 ceph-mon[32774]: 5.0 scrub ok 2024-02-05T06:56:16.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:15 smithi167 ceph-mon[32774]: 6.f scrub ok 2024-02-05T06:56:16.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:15 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1275603785' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-02-05T06:56:16.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:15 smithi167 ceph-mon[32774]: osdmap e505: 8 total, 8 up, 8 in 2024-02-05T06:56:17.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:16 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:56:17.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:16 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:56:17.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:16 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:56:17.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:16 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:56:17.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:16 smithi028 ceph-mon[29442]: 1.10 scrub starts 2024-02-05T06:56:17.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:16 smithi028 ceph-mon[29442]: 1.10 scrub ok 2024-02-05T06:56:17.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:16 smithi028 ceph-mon[29442]: osdmap e506: 8 total, 8 up, 8 in 2024-02-05T06:56:17.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:16 smithi028 ceph-mon[29442]: 6.16 scrub starts 2024-02-05T06:56:17.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:16 smithi028 ceph-mon[29442]: 6.16 scrub ok 2024-02-05T06:56:17.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:16 smithi028 ceph-mon[29442]: 6.14 scrub starts 2024-02-05T06:56:17.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:16 smithi028 ceph-mon[29442]: 6.14 scrub ok 2024-02-05T06:56:17.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:16 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:56:17.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:16 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:56:17.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:16 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:56:17.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:16 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:56:17.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:16 smithi028 ceph-mon[37984]: 1.10 scrub starts 2024-02-05T06:56:17.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:16 smithi028 ceph-mon[37984]: 1.10 scrub ok 2024-02-05T06:56:17.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:16 smithi028 ceph-mon[37984]: osdmap e506: 8 total, 8 up, 8 in 2024-02-05T06:56:17.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:16 smithi028 ceph-mon[37984]: 6.16 scrub starts 2024-02-05T06:56:17.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:16 smithi028 ceph-mon[37984]: 6.16 scrub ok 2024-02-05T06:56:17.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:16 smithi028 ceph-mon[37984]: 6.14 scrub starts 2024-02-05T06:56:17.199 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:16 smithi028 ceph-mon[37984]: 6.14 scrub ok 2024-02-05T06:56:17.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:16 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:56:17.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:16 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:56:17.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:16 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:56:17.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:16 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:56:17.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:16 smithi167 ceph-mon[32774]: 1.10 scrub starts 2024-02-05T06:56:17.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:16 smithi167 ceph-mon[32774]: 1.10 scrub ok 2024-02-05T06:56:17.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:16 smithi167 ceph-mon[32774]: osdmap e506: 8 total, 8 up, 8 in 2024-02-05T06:56:17.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:16 smithi167 ceph-mon[32774]: 6.16 scrub starts 2024-02-05T06:56:17.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:16 smithi167 ceph-mon[32774]: 6.16 scrub ok 2024-02-05T06:56:17.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:16 smithi167 ceph-mon[32774]: 6.14 scrub starts 2024-02-05T06:56:17.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:16 smithi167 ceph-mon[32774]: 6.14 scrub ok 2024-02-05T06:56:17.974 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:56:17.975 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":506,"fsid":"9801e956-c3ef-11ee-95b6-87774f69a715","created":"2024-02-05T06:27:59.919226+0000","modified":"2024-02-05T06:56:15.925736+0000","last_up_change":"2024-02-05T06:33:15.833573+0000","last_in_change":"2024-02-05T06:41:42.383087+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-05T06:31:10.441424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":21,"pg_placement_num":21,"pg_placement_num_target":21,"pg_num_target":21,"pg_num_pending":21,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"485","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"429","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8999999761581421,"score_stable":1.8999999761581421,"optimal_score":1,"raw_score_acting":1.8999999761581421,"raw_score_stable":1.8999999761581421,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-02-05T06:33:27.397416+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":69,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-02-05T06:33:28.090506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-05T06:33:29.866249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-05T06:33:31.826006+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-05T06:33:33.890835+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c19fdd5b-09f6-45d4-a519-0a938cf1f2a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":482,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6801","nonce":301279106}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6802","nonce":301279106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6804","nonce":301279106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6803","nonce":301279106}]},"public_addr":"172.21.15.28:6801/301279106","cluster_addr":"172.21.15.28:6802/301279106","heartbeat_back_addr":"172.21.15.28:6804/301279106","heartbeat_front_addr":"172.21.15.28:6803/301279106","state":["exists","up"]},{"osd":1,"uuid":"52245f6e-eb96-4d0d-985b-dbb6fd3ce8de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":434,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6805","nonce":1795224649}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6806","nonce":1795224649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6808","nonce":1795224649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6807","nonce":1795224649}]},"public_addr":"172.21.15.28:6805/1795224649","cluster_addr":"172.21.15.28:6806/1795224649","heartbeat_back_addr":"172.21.15.28:6808/1795224649","heartbeat_front_addr":"172.21.15.28:6807/1795224649","state":["exists","up"]},{"osd":2,"uuid":"1e18b630-afdf-4de9-a1a2-37bdfdd01c67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":434,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6809","nonce":1993280018}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6810","nonce":1993280018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6812","nonce":1993280018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6811","nonce":1993280018}]},"public_addr":"172.21.15.28:6809/1993280018","cluster_addr":"172.21.15.28:6810/1993280018","heartbeat_back_addr":"172.21.15.28:6812/1993280018","heartbeat_front_addr":"172.21.15.28:6811/1993280018","state":["exists","up"]},{"osd":3,"uuid":"3ae795af-3d59-41a6-95eb-b8447ff61282","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":434,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6813","nonce":1294621398}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6814","nonce":1294621398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6816","nonce":1294621398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6815","nonce":1294621398}]},"public_addr":"172.21.15.28:6813/1294621398","cluster_addr":"172.21.15.28:6814/1294621398","heartbeat_back_addr":"172.21.15.28:6816/1294621398","heartbeat_front_addr":"172.21.15.28:6815/1294621398","state":["exists","up"]},{"osd":4,"uuid":"fc6bd83d-570c-48c0-964d-e09b4608bc32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":497,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6800","nonce":4052090595}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6801","nonce":4052090595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6803","nonce":4052090595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6802","nonce":4052090595}]},"public_addr":"172.21.15.167:6800/4052090595","cluster_addr":"172.21.15.167:6801/4052090595","heartbeat_back_addr":"172.21.15.167:6803/4052090595","heartbeat_front_addr":"172.21.15.167:6802/4052090595","state":["exists","up"]},{"osd":5,"uuid":"895bf465-2531-4d9f-bc23-1e75fd969f07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":434,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6804","nonce":1631080500}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6805","nonce":1631080500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6807","nonce":1631080500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6806","nonce":1631080500}]},"public_addr":"172.21.15.167:6804/1631080500","cluster_addr":"172.21.15.167:6805/1631080500","heartbeat_back_addr":"172.21.15.167:6807/1631080500","heartbeat_front_addr":"172.21.15.167:6806/1631080500","state":["exists","up"]},{"osd":6,"uuid":"ce0a9300-87a4-4e37-b702-d667f168c0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":434,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6808","nonce":1644050617}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6809","nonce":1644050617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6811","nonce":1644050617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6810","nonce":1644050617}]},"public_addr":"172.21.15.167:6808/1644050617","cluster_addr":"172.21.15.167:6809/1644050617","heartbeat_back_addr":"172.21.15.167:6811/1644050617","heartbeat_front_addr":"172.21.15.167:6810/1644050617","state":["exists","up"]},{"osd":7,"uuid":"0c545cbc-0873-4232-8807-4af9299ef95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":434,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6812","nonce":4031475764}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6813","nonce":4031475764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6815","nonce":4031475764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6814","nonce":4031475764}]},"public_addr":"172.21.15.167:6812/4031475764","cluster_addr":"172.21.15.167:6813/4031475764","heartbeat_back_addr":"172.21.15.167:6815/4031475764","heartbeat_front_addr":"172.21.15.167:6814/4031475764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:55:42.264031+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:56:15.234070+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:55:41.820201+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:56:09.820966+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:55:11.194726+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:56:15.699369+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:56:10.347384+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:55:27.618373+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"3.18","osds":[5,2,4]},{"pgid":"4.2","osds":[0,2,5]},{"pgid":"5.1b","osds":[3,0,1]}],"pg_upmap_items":[{"pgid":"3.9","mappings":[{"from":4,"to":1}]},{"pgid":"3.a","mappings":[{"from":4,"to":2}]},{"pgid":"4.f","mappings":[{"from":1,"to":0}]},{"pgid":"4.12","mappings":[{"from":1,"to":0}]},{"pgid":"4.14","mappings":[{"from":1,"to":2}]},{"pgid":"4.15","mappings":[{"from":3,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]},{"pgid":"5.1c","mappings":[{"from":4,"to":6}]},{"pgid":"5.1e","mappings":[{"from":0,"to":3}]},{"pgid":"6.12","mappings":[{"from":2,"to":1},{"from":4,"to":0}]},{"pgid":"6.13","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/1001442173":"2024-02-06T06:34:10.127914+0000","172.21.15.28:0/3160026720":"2024-02-06T06:34:10.127914+0000","172.21.15.28:0/558171292":"2024-02-06T06:34:10.127914+0000","172.21.15.28:0/2832437766":"2024-02-06T06:34:10.127914+0000","172.21.15.28:6800/4125764599":"2024-02-06T06:28:54.105170+0000","172.21.15.28:0/1081543653":"2024-02-06T06:28:54.105170+0000","172.21.15.28:0/3794420554":"2024-02-06T06:28:27.553034+0000","172.21.15.28:0/1560471317":"2024-02-06T06:34:10.127914+0000","172.21.15.28:6800/4266601437":"2024-02-06T06:28:27.553034+0000","172.21.15.28:0/388893831":"2024-02-06T06:28:27.553034+0000","172.21.15.28:0/460817910":"2024-02-06T06:28:27.553034+0000","172.21.15.28:0/2728305743":"2024-02-06T06:28:54.105170+0000","172.21.15.28:0/3149394045":"2024-02-06T06:34:10.127914+0000","172.21.15.28:0/3605013594":"2024-02-06T06:28:54.105170+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-05T06:56:18.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:17 smithi028 ceph-mon[37984]: pgmap v1090: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:56:18.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:17 smithi028 ceph-mon[37984]: 5.16 scrub starts 2024-02-05T06:56:18.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:17 smithi028 ceph-mon[37984]: 5.16 scrub ok 2024-02-05T06:56:18.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:17 smithi028 ceph-mon[37984]: 1.7 deep-scrub starts 2024-02-05T06:56:18.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:17 smithi028 ceph-mon[37984]: 1.7 deep-scrub ok 2024-02-05T06:56:18.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:17 smithi028 ceph-mon[37984]: 5.3 scrub starts 2024-02-05T06:56:18.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:17 smithi028 ceph-mon[37984]: 5.3 scrub ok 2024-02-05T06:56:18.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:17 smithi028 ceph-mon[29442]: pgmap v1090: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:56:18.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:17 smithi028 ceph-mon[29442]: 5.16 scrub starts 2024-02-05T06:56:18.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:17 smithi028 ceph-mon[29442]: 5.16 scrub ok 2024-02-05T06:56:18.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:17 smithi028 ceph-mon[29442]: 1.7 deep-scrub starts 2024-02-05T06:56:18.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:17 smithi028 ceph-mon[29442]: 1.7 deep-scrub ok 2024-02-05T06:56:18.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:17 smithi028 ceph-mon[29442]: 5.3 scrub starts 2024-02-05T06:56:18.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:17 smithi028 ceph-mon[29442]: 5.3 scrub ok 2024-02-05T06:56:18.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:17 smithi167 ceph-mon[32774]: pgmap v1090: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:56:18.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:17 smithi167 ceph-mon[32774]: 5.16 scrub starts 2024-02-05T06:56:18.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:17 smithi167 ceph-mon[32774]: 5.16 scrub ok 2024-02-05T06:56:18.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:17 smithi167 ceph-mon[32774]: 1.7 deep-scrub starts 2024-02-05T06:56:18.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:17 smithi167 ceph-mon[32774]: 1.7 deep-scrub ok 2024-02-05T06:56:18.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:17 smithi167 ceph-mon[32774]: 5.3 scrub starts 2024-02-05T06:56:18.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:17 smithi167 ceph-mon[32774]: 5.3 scrub ok 2024-02-05T06:56:18.665 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-05T06:56:18.665 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd dump --format=json 2024-02-05T06:56:18.916 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:56:19.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:18 smithi028 ceph-mon[29442]: 4.d scrub starts 2024-02-05T06:56:19.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:18 smithi028 ceph-mon[29442]: 4.d scrub ok 2024-02-05T06:56:19.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:18 smithi028 ceph-mon[29442]: 5.f scrub starts 2024-02-05T06:56:19.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:18 smithi028 ceph-mon[29442]: 5.f scrub ok 2024-02-05T06:56:19.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:18 smithi028 ceph-mon[29442]: 1.2 scrub starts 2024-02-05T06:56:19.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:18 smithi028 ceph-mon[29442]: 1.2 scrub ok 2024-02-05T06:56:19.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:18 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/3749228424' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T06:56:19.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:18 smithi028 ceph-mon[37984]: 4.d scrub starts 2024-02-05T06:56:19.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:18 smithi028 ceph-mon[37984]: 4.d scrub ok 2024-02-05T06:56:19.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:18 smithi028 ceph-mon[37984]: 5.f scrub starts 2024-02-05T06:56:19.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:18 smithi028 ceph-mon[37984]: 5.f scrub ok 2024-02-05T06:56:19.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:18 smithi028 ceph-mon[37984]: 1.2 scrub starts 2024-02-05T06:56:19.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:18 smithi028 ceph-mon[37984]: 1.2 scrub ok 2024-02-05T06:56:19.200 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:18 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/3749228424' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T06:56:19.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:18 smithi167 ceph-mon[32774]: 4.d scrub starts 2024-02-05T06:56:19.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:18 smithi167 ceph-mon[32774]: 4.d scrub ok 2024-02-05T06:56:19.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:18 smithi167 ceph-mon[32774]: 5.f scrub starts 2024-02-05T06:56:19.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:18 smithi167 ceph-mon[32774]: 5.f scrub ok 2024-02-05T06:56:19.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:18 smithi167 ceph-mon[32774]: 1.2 scrub starts 2024-02-05T06:56:19.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:18 smithi167 ceph-mon[32774]: 1.2 scrub ok 2024-02-05T06:56:19.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:18 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/3749228424' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T06:56:20.150 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:19 smithi028 ceph-mon[37984]: pgmap v1091: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T06:56:20.150 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:19 smithi028 ceph-mon[37984]: 6.c scrub starts 2024-02-05T06:56:20.150 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:19 smithi028 ceph-mon[37984]: 6.c scrub ok 2024-02-05T06:56:20.150 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:19 smithi028 ceph-mon[37984]: 3.16 deep-scrub starts 2024-02-05T06:56:20.150 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:19 smithi028 ceph-mon[37984]: 3.16 deep-scrub ok 2024-02-05T06:56:20.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:19 smithi028 ceph-mon[29442]: pgmap v1091: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T06:56:20.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:19 smithi028 ceph-mon[29442]: 6.c scrub starts 2024-02-05T06:56:20.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:19 smithi028 ceph-mon[29442]: 6.c scrub ok 2024-02-05T06:56:20.150 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:19 smithi028 ceph-mon[29442]: 3.16 deep-scrub starts 2024-02-05T06:56:20.151 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:19 smithi028 ceph-mon[29442]: 3.16 deep-scrub ok 2024-02-05T06:56:20.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:19 smithi167 ceph-mon[32774]: pgmap v1091: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T06:56:20.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:19 smithi167 ceph-mon[32774]: 6.c scrub starts 2024-02-05T06:56:20.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:19 smithi167 ceph-mon[32774]: 6.c scrub ok 2024-02-05T06:56:20.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:19 smithi167 ceph-mon[32774]: 3.16 deep-scrub starts 2024-02-05T06:56:20.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:19 smithi167 ceph-mon[32774]: 3.16 deep-scrub ok 2024-02-05T06:56:20.890 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:56:20.890 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":506,"fsid":"9801e956-c3ef-11ee-95b6-87774f69a715","created":"2024-02-05T06:27:59.919226+0000","modified":"2024-02-05T06:56:15.925736+0000","last_up_change":"2024-02-05T06:33:15.833573+0000","last_in_change":"2024-02-05T06:41:42.383087+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-05T06:31:10.441424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":21,"pg_placement_num":21,"pg_placement_num_target":21,"pg_num_target":21,"pg_num_pending":21,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"485","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"429","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8999999761581421,"score_stable":1.8999999761581421,"optimal_score":1,"raw_score_acting":1.8999999761581421,"raw_score_stable":1.8999999761581421,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-02-05T06:33:27.397416+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":69,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-02-05T06:33:28.090506+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-02-05T06:33:29.866249+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-02-05T06:33:31.826006+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-02-05T06:33:33.890835+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c19fdd5b-09f6-45d4-a519-0a938cf1f2a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":482,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6801","nonce":301279106}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6802","nonce":301279106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6804","nonce":301279106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6803","nonce":301279106}]},"public_addr":"172.21.15.28:6801/301279106","cluster_addr":"172.21.15.28:6802/301279106","heartbeat_back_addr":"172.21.15.28:6804/301279106","heartbeat_front_addr":"172.21.15.28:6803/301279106","state":["exists","up"]},{"osd":1,"uuid":"52245f6e-eb96-4d0d-985b-dbb6fd3ce8de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":434,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6805","nonce":1795224649}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6806","nonce":1795224649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6808","nonce":1795224649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6807","nonce":1795224649}]},"public_addr":"172.21.15.28:6805/1795224649","cluster_addr":"172.21.15.28:6806/1795224649","heartbeat_back_addr":"172.21.15.28:6808/1795224649","heartbeat_front_addr":"172.21.15.28:6807/1795224649","state":["exists","up"]},{"osd":2,"uuid":"1e18b630-afdf-4de9-a1a2-37bdfdd01c67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":434,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6809","nonce":1993280018}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6810","nonce":1993280018}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6812","nonce":1993280018}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6811","nonce":1993280018}]},"public_addr":"172.21.15.28:6809/1993280018","cluster_addr":"172.21.15.28:6810/1993280018","heartbeat_back_addr":"172.21.15.28:6812/1993280018","heartbeat_front_addr":"172.21.15.28:6811/1993280018","state":["exists","up"]},{"osd":3,"uuid":"3ae795af-3d59-41a6-95eb-b8447ff61282","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":434,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6813","nonce":1294621398}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6814","nonce":1294621398}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6816","nonce":1294621398}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.28:6815","nonce":1294621398}]},"public_addr":"172.21.15.28:6813/1294621398","cluster_addr":"172.21.15.28:6814/1294621398","heartbeat_back_addr":"172.21.15.28:6816/1294621398","heartbeat_front_addr":"172.21.15.28:6815/1294621398","state":["exists","up"]},{"osd":4,"uuid":"fc6bd83d-570c-48c0-964d-e09b4608bc32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":497,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6800","nonce":4052090595}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6801","nonce":4052090595}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6803","nonce":4052090595}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6802","nonce":4052090595}]},"public_addr":"172.21.15.167:6800/4052090595","cluster_addr":"172.21.15.167:6801/4052090595","heartbeat_back_addr":"172.21.15.167:6803/4052090595","heartbeat_front_addr":"172.21.15.167:6802/4052090595","state":["exists","up"]},{"osd":5,"uuid":"895bf465-2531-4d9f-bc23-1e75fd969f07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":434,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6804","nonce":1631080500}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6805","nonce":1631080500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6807","nonce":1631080500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6806","nonce":1631080500}]},"public_addr":"172.21.15.167:6804/1631080500","cluster_addr":"172.21.15.167:6805/1631080500","heartbeat_back_addr":"172.21.15.167:6807/1631080500","heartbeat_front_addr":"172.21.15.167:6806/1631080500","state":["exists","up"]},{"osd":6,"uuid":"ce0a9300-87a4-4e37-b702-d667f168c0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":434,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6808","nonce":1644050617}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6809","nonce":1644050617}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6811","nonce":1644050617}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6810","nonce":1644050617}]},"public_addr":"172.21.15.167:6808/1644050617","cluster_addr":"172.21.15.167:6809/1644050617","heartbeat_back_addr":"172.21.15.167:6811/1644050617","heartbeat_front_addr":"172.21.15.167:6810/1644050617","state":["exists","up"]},{"osd":7,"uuid":"0c545cbc-0873-4232-8807-4af9299ef95e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":434,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6812","nonce":4031475764}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6813","nonce":4031475764}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6815","nonce":4031475764}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.167:6814","nonce":4031475764}]},"public_addr":"172.21.15.167:6812/4031475764","cluster_addr":"172.21.15.167:6813/4031475764","heartbeat_back_addr":"172.21.15.167:6815/4031475764","heartbeat_front_addr":"172.21.15.167:6814/4031475764","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:55:42.264031+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:56:15.234070+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:55:41.820201+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:56:09.820966+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:55:11.194726+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:56:15.699369+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:56:10.347384+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2024-02-05T06:55:27.618373+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"3.18","osds":[5,2,4]},{"pgid":"4.2","osds":[0,2,5]},{"pgid":"5.1b","osds":[3,0,1]}],"pg_upmap_items":[{"pgid":"3.9","mappings":[{"from":4,"to":1}]},{"pgid":"3.a","mappings":[{"from":4,"to":2}]},{"pgid":"4.f","mappings":[{"from":1,"to":0}]},{"pgid":"4.12","mappings":[{"from":1,"to":0}]},{"pgid":"4.14","mappings":[{"from":1,"to":2}]},{"pgid":"4.15","mappings":[{"from":3,"to":2}]},{"pgid":"4.1d","mappings":[{"from":1,"to":2}]},{"pgid":"5.1c","mappings":[{"from":4,"to":6}]},{"pgid":"5.1e","mappings":[{"from":0,"to":3}]},{"pgid":"6.12","mappings":[{"from":2,"to":1},{"from":4,"to":0}]},{"pgid":"6.13","mappings":[{"from":3,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6800/1001442173":"2024-02-06T06:34:10.127914+0000","172.21.15.28:0/3160026720":"2024-02-06T06:34:10.127914+0000","172.21.15.28:0/558171292":"2024-02-06T06:34:10.127914+0000","172.21.15.28:0/2832437766":"2024-02-06T06:34:10.127914+0000","172.21.15.28:6800/4125764599":"2024-02-06T06:28:54.105170+0000","172.21.15.28:0/1081543653":"2024-02-06T06:28:54.105170+0000","172.21.15.28:0/3794420554":"2024-02-06T06:28:27.553034+0000","172.21.15.28:0/1560471317":"2024-02-06T06:34:10.127914+0000","172.21.15.28:6800/4266601437":"2024-02-06T06:28:27.553034+0000","172.21.15.28:0/388893831":"2024-02-06T06:28:27.553034+0000","172.21.15.28:0/460817910":"2024-02-06T06:28:27.553034+0000","172.21.15.28:0/2728305743":"2024-02-06T06:28:54.105170+0000","172.21.15.28:0/3149394045":"2024-02-06T06:34:10.127914+0000","172.21.15.28:0/3605013594":"2024-02-06T06:28:54.105170+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-05T06:56:21.157 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:20 smithi028 ceph-mon[29442]: 6.e scrub starts 2024-02-05T06:56:21.157 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:20 smithi028 ceph-mon[29442]: 6.e scrub ok 2024-02-05T06:56:21.157 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:20 smithi028 ceph-mon[29442]: 4.5 deep-scrub starts 2024-02-05T06:56:21.157 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:20 smithi028 ceph-mon[29442]: 4.5 deep-scrub ok 2024-02-05T06:56:21.157 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:20 smithi028 ceph-mon[29442]: 4.3 scrub starts 2024-02-05T06:56:21.157 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:20 smithi028 ceph-mon[29442]: 4.3 scrub ok 2024-02-05T06:56:21.157 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:20 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1278970354' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T06:56:21.158 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:20 smithi028 ceph-mon[37984]: 6.e scrub starts 2024-02-05T06:56:21.158 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:20 smithi028 ceph-mon[37984]: 6.e scrub ok 2024-02-05T06:56:21.158 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:20 smithi028 ceph-mon[37984]: 4.5 deep-scrub starts 2024-02-05T06:56:21.158 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:20 smithi028 ceph-mon[37984]: 4.5 deep-scrub ok 2024-02-05T06:56:21.158 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:20 smithi028 ceph-mon[37984]: 4.3 scrub starts 2024-02-05T06:56:21.158 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:20 smithi028 ceph-mon[37984]: 4.3 scrub ok 2024-02-05T06:56:21.158 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:20 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1278970354' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T06:56:21.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:20 smithi167 ceph-mon[32774]: 6.e scrub starts 2024-02-05T06:56:21.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:20 smithi167 ceph-mon[32774]: 6.e scrub ok 2024-02-05T06:56:21.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:20 smithi167 ceph-mon[32774]: 4.5 deep-scrub starts 2024-02-05T06:56:21.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:20 smithi167 ceph-mon[32774]: 4.5 deep-scrub ok 2024-02-05T06:56:21.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:20 smithi167 ceph-mon[32774]: 4.3 scrub starts 2024-02-05T06:56:21.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:20 smithi167 ceph-mon[32774]: 4.3 scrub ok 2024-02-05T06:56:21.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:20 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1278970354' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-05T06:56:21.565 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-05T06:56:21.565 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-05T06:56:21.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-05T06:56:21.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-05T06:56:21.567 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-05T06:56:21.567 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-05T06:56:21.568 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-05T06:56:21.568 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-05T06:56:22.155 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:56:22.156 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:56:22.160 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:56:22.160 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:56:22.163 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:56:22.164 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:56:22.166 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:56:22.166 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:56:22.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:21 smithi167 ceph-mon[32774]: pgmap v1092: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:56:22.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:21 smithi167 ceph-mon[32774]: 6.6 scrub starts 2024-02-05T06:56:22.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:21 smithi167 ceph-mon[32774]: 6.6 scrub ok 2024-02-05T06:56:22.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:21 smithi167 ceph-mon[32774]: 1.12 scrub starts 2024-02-05T06:56:22.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:21 smithi167 ceph-mon[32774]: 1.12 scrub ok 2024-02-05T06:56:22.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:21 smithi167 ceph-mon[32774]: 4.f deep-scrub starts 2024-02-05T06:56:22.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:21 smithi167 ceph-mon[32774]: 4.f deep-scrub ok 2024-02-05T06:56:22.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:21 smithi028 ceph-mon[29442]: pgmap v1092: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:56:22.295 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:21 smithi028 ceph-mon[29442]: 6.6 scrub starts 2024-02-05T06:56:22.296 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:21 smithi028 ceph-mon[29442]: 6.6 scrub ok 2024-02-05T06:56:22.296 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:21 smithi028 ceph-mon[29442]: 1.12 scrub starts 2024-02-05T06:56:22.296 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:21 smithi028 ceph-mon[29442]: 1.12 scrub ok 2024-02-05T06:56:22.296 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:21 smithi028 ceph-mon[29442]: 4.f deep-scrub starts 2024-02-05T06:56:22.296 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:21 smithi028 ceph-mon[29442]: 4.f deep-scrub ok 2024-02-05T06:56:22.296 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:21 smithi028 ceph-mon[37984]: pgmap v1092: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:56:22.296 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:21 smithi028 ceph-mon[37984]: 6.6 scrub starts 2024-02-05T06:56:22.296 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:21 smithi028 ceph-mon[37984]: 6.6 scrub ok 2024-02-05T06:56:22.296 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:21 smithi028 ceph-mon[37984]: 1.12 scrub starts 2024-02-05T06:56:22.296 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:21 smithi028 ceph-mon[37984]: 1.12 scrub ok 2024-02-05T06:56:22.296 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:21 smithi028 ceph-mon[37984]: 4.f deep-scrub starts 2024-02-05T06:56:22.296 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:21 smithi028 ceph-mon[37984]: 4.f deep-scrub ok 2024-02-05T06:56:22.613 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:56:22 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:56:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:56:23.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:22 smithi167 ceph-mon[32774]: 6.1f scrub starts 2024-02-05T06:56:23.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:22 smithi167 ceph-mon[32774]: 6.1f scrub ok 2024-02-05T06:56:23.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:22 smithi167 ceph-mon[32774]: 3.a scrub starts 2024-02-05T06:56:23.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:22 smithi167 ceph-mon[32774]: 3.a scrub ok 2024-02-05T06:56:23.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:22 smithi167 ceph-mon[32774]: 4.6 scrub starts 2024-02-05T06:56:23.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:22 smithi167 ceph-mon[32774]: 4.6 scrub ok 2024-02-05T06:56:23.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:22 smithi028 ceph-mon[29442]: 6.1f scrub starts 2024-02-05T06:56:23.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:22 smithi028 ceph-mon[29442]: 6.1f scrub ok 2024-02-05T06:56:23.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:22 smithi028 ceph-mon[29442]: 3.a scrub starts 2024-02-05T06:56:23.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:22 smithi028 ceph-mon[29442]: 3.a scrub ok 2024-02-05T06:56:23.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:22 smithi028 ceph-mon[29442]: 4.6 scrub starts 2024-02-05T06:56:23.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:22 smithi028 ceph-mon[29442]: 4.6 scrub ok 2024-02-05T06:56:23.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:22 smithi028 ceph-mon[37984]: 6.1f scrub starts 2024-02-05T06:56:23.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:22 smithi028 ceph-mon[37984]: 6.1f scrub ok 2024-02-05T06:56:23.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:22 smithi028 ceph-mon[37984]: 3.a scrub starts 2024-02-05T06:56:23.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:22 smithi028 ceph-mon[37984]: 3.a scrub ok 2024-02-05T06:56:23.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:22 smithi028 ceph-mon[37984]: 4.6 scrub starts 2024-02-05T06:56:23.444 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:22 smithi028 ceph-mon[37984]: 4.6 scrub ok 2024-02-05T06:56:23.982 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:56:23 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:56:23.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:23 smithi167 ceph-mon[32774]: 6.1e scrub starts 2024-02-05T06:56:23.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:23 smithi167 ceph-mon[32774]: 6.1e scrub ok 2024-02-05T06:56:23.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:23 smithi167 ceph-mon[32774]: pgmap v1093: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:56:23.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:23 smithi167 ceph-mon[32774]: 2.2 scrub starts 2024-02-05T06:56:23.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:23 smithi167 ceph-mon[32774]: 2.2 scrub ok 2024-02-05T06:56:23.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:23 smithi167 ceph-mon[32774]: 4.1c scrub starts 2024-02-05T06:56:23.983 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:23 smithi167 ceph-mon[32774]: 4.1c scrub ok 2024-02-05T06:56:24.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:23 smithi028 ceph-mon[37984]: 6.1e scrub starts 2024-02-05T06:56:24.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:23 smithi028 ceph-mon[37984]: 6.1e scrub ok 2024-02-05T06:56:24.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:23 smithi028 ceph-mon[37984]: pgmap v1093: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:56:24.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:23 smithi028 ceph-mon[37984]: 2.2 scrub starts 2024-02-05T06:56:24.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:23 smithi028 ceph-mon[37984]: 2.2 scrub ok 2024-02-05T06:56:24.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:23 smithi028 ceph-mon[37984]: 4.1c scrub starts 2024-02-05T06:56:24.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:23 smithi028 ceph-mon[37984]: 4.1c scrub ok 2024-02-05T06:56:24.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:23 smithi028 ceph-mon[29442]: 6.1e scrub starts 2024-02-05T06:56:24.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:23 smithi028 ceph-mon[29442]: 6.1e scrub ok 2024-02-05T06:56:24.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:23 smithi028 ceph-mon[29442]: pgmap v1093: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:56:24.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:23 smithi028 ceph-mon[29442]: 2.2 scrub starts 2024-02-05T06:56:24.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:23 smithi028 ceph-mon[29442]: 2.2 scrub ok 2024-02-05T06:56:24.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:23 smithi028 ceph-mon[29442]: 4.1c scrub starts 2024-02-05T06:56:24.199 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:23 smithi028 ceph-mon[29442]: 4.1c scrub ok 2024-02-05T06:56:25.224 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:24 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:56:25.224 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:24 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:56:25.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:24 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:56:26.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:25 smithi028 ceph-mon[37984]: pgmap v1094: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T06:56:26.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:25 smithi028 ceph-mon[37984]: 5.13 scrub starts 2024-02-05T06:56:26.094 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:25 smithi028 ceph-mon[37984]: 5.13 scrub ok 2024-02-05T06:56:26.094 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:25 smithi028 ceph-mon[29442]: pgmap v1094: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T06:56:26.094 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:25 smithi028 ceph-mon[29442]: 5.13 scrub starts 2024-02-05T06:56:26.094 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:25 smithi028 ceph-mon[29442]: 5.13 scrub ok 2024-02-05T06:56:26.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:25 smithi167 ceph-mon[32774]: pgmap v1094: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-05T06:56:26.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:25 smithi167 ceph-mon[32774]: 5.13 scrub starts 2024-02-05T06:56:26.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:25 smithi167 ceph-mon[32774]: 5.13 scrub ok 2024-02-05T06:56:27.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:26 smithi167 ceph-mon[32774]: 3.19 scrub starts 2024-02-05T06:56:27.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:26 smithi167 ceph-mon[32774]: 3.19 scrub ok 2024-02-05T06:56:27.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:26 smithi028 ceph-mon[29442]: 3.19 scrub starts 2024-02-05T06:56:27.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:26 smithi028 ceph-mon[29442]: 3.19 scrub ok 2024-02-05T06:56:27.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:26 smithi028 ceph-mon[37984]: 3.19 scrub starts 2024-02-05T06:56:27.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:26 smithi028 ceph-mon[37984]: 3.19 scrub ok 2024-02-05T06:56:28.197 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:27 smithi028 ceph-mon[29442]: pgmap v1095: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T06:56:28.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:27 smithi028 ceph-mon[29442]: 5.11 scrub starts 2024-02-05T06:56:28.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:27 smithi028 ceph-mon[29442]: 5.11 scrub ok 2024-02-05T06:56:28.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:27 smithi028 ceph-mon[29442]: 5.d scrub starts 2024-02-05T06:56:28.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:27 smithi028 ceph-mon[29442]: 5.d scrub ok 2024-02-05T06:56:28.198 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:27 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:56:28.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:27 smithi028 ceph-mon[37984]: pgmap v1095: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T06:56:28.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:27 smithi028 ceph-mon[37984]: 5.11 scrub starts 2024-02-05T06:56:28.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:27 smithi028 ceph-mon[37984]: 5.11 scrub ok 2024-02-05T06:56:28.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:27 smithi028 ceph-mon[37984]: 5.d scrub starts 2024-02-05T06:56:28.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:27 smithi028 ceph-mon[37984]: 5.d scrub ok 2024-02-05T06:56:28.198 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:27 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:56:28.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:27 smithi167 ceph-mon[32774]: pgmap v1095: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-02-05T06:56:28.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:27 smithi167 ceph-mon[32774]: 5.11 scrub starts 2024-02-05T06:56:28.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:27 smithi167 ceph-mon[32774]: 5.11 scrub ok 2024-02-05T06:56:28.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:27 smithi167 ceph-mon[32774]: 5.d scrub starts 2024-02-05T06:56:28.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:27 smithi167 ceph-mon[32774]: 5.d scrub ok 2024-02-05T06:56:28.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:27 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:56:29.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:28 smithi167 ceph-mon[32774]: 4.13 scrub starts 2024-02-05T06:56:29.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:28 smithi167 ceph-mon[32774]: 4.13 scrub ok 2024-02-05T06:56:29.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:28 smithi167 ceph-mon[32774]: 4.11 scrub starts 2024-02-05T06:56:29.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:28 smithi167 ceph-mon[32774]: 4.11 scrub ok 2024-02-05T06:56:29.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:28 smithi167 ceph-mon[32774]: 1.14 scrub starts 2024-02-05T06:56:29.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:28 smithi167 ceph-mon[32774]: 1.14 scrub ok 2024-02-05T06:56:29.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:28 smithi167 ceph-mon[32774]: 4.7 deep-scrub starts 2024-02-05T06:56:29.233 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:28 smithi167 ceph-mon[32774]: 4.7 deep-scrub ok 2024-02-05T06:56:29.251 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:28 smithi028 ceph-mon[29442]: 4.13 scrub starts 2024-02-05T06:56:29.252 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:28 smithi028 ceph-mon[29442]: 4.13 scrub ok 2024-02-05T06:56:29.252 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:28 smithi028 ceph-mon[29442]: 4.11 scrub starts 2024-02-05T06:56:29.252 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:28 smithi028 ceph-mon[29442]: 4.11 scrub ok 2024-02-05T06:56:29.252 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:28 smithi028 ceph-mon[29442]: 1.14 scrub starts 2024-02-05T06:56:29.252 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:28 smithi028 ceph-mon[29442]: 1.14 scrub ok 2024-02-05T06:56:29.252 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:28 smithi028 ceph-mon[29442]: 4.7 deep-scrub starts 2024-02-05T06:56:29.252 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:28 smithi028 ceph-mon[29442]: 4.7 deep-scrub ok 2024-02-05T06:56:29.252 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:28 smithi028 ceph-mon[37984]: 4.13 scrub starts 2024-02-05T06:56:29.252 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:28 smithi028 ceph-mon[37984]: 4.13 scrub ok 2024-02-05T06:56:29.252 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:28 smithi028 ceph-mon[37984]: 4.11 scrub starts 2024-02-05T06:56:29.252 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:28 smithi028 ceph-mon[37984]: 4.11 scrub ok 2024-02-05T06:56:29.252 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:28 smithi028 ceph-mon[37984]: 1.14 scrub starts 2024-02-05T06:56:29.253 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:28 smithi028 ceph-mon[37984]: 1.14 scrub ok 2024-02-05T06:56:29.253 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:28 smithi028 ceph-mon[37984]: 4.7 deep-scrub starts 2024-02-05T06:56:29.253 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:28 smithi028 ceph-mon[37984]: 4.7 deep-scrub ok 2024-02-05T06:56:30.083 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:29 smithi028 ceph-mon[37984]: pgmap v1096: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:56:30.084 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:29 smithi028 ceph-mon[37984]: 4.e scrub starts 2024-02-05T06:56:30.084 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:29 smithi028 ceph-mon[37984]: 4.e scrub ok 2024-02-05T06:56:30.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:29 smithi028 ceph-mon[29442]: pgmap v1096: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:56:30.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:29 smithi028 ceph-mon[29442]: 4.e scrub starts 2024-02-05T06:56:30.084 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:29 smithi028 ceph-mon[29442]: 4.e scrub ok 2024-02-05T06:56:30.116 INFO:teuthology.orchestra.run.smithi028.stdout:249108103447 2024-02-05T06:56:30.116 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-05T06:56:30.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:29 smithi167 ceph-mon[32774]: pgmap v1096: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:56:30.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:29 smithi167 ceph-mon[32774]: 4.e scrub starts 2024-02-05T06:56:30.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:29 smithi167 ceph-mon[32774]: 4.e scrub ok 2024-02-05T06:56:31.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:31 smithi028 ceph-mon[29442]: 5.15 scrub starts 2024-02-05T06:56:31.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:31 smithi028 ceph-mon[29442]: 5.15 scrub ok 2024-02-05T06:56:31.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:31 smithi028 ceph-mon[29442]: 6.1a scrub starts 2024-02-05T06:56:31.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:31 smithi028 ceph-mon[29442]: 6.1a scrub ok 2024-02-05T06:56:31.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:31 smithi028 ceph-mon[29442]: pgmap v1097: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:56:31.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:31 smithi028 ceph-mon[29442]: 6.1c scrub starts 2024-02-05T06:56:31.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:31 smithi028 ceph-mon[29442]: 6.1c scrub ok 2024-02-05T06:56:31.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:31 smithi028 ceph-mon[37984]: 5.15 scrub starts 2024-02-05T06:56:31.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:31 smithi028 ceph-mon[37984]: 5.15 scrub ok 2024-02-05T06:56:31.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:31 smithi028 ceph-mon[37984]: 6.1a scrub starts 2024-02-05T06:56:31.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:31 smithi028 ceph-mon[37984]: 6.1a scrub ok 2024-02-05T06:56:31.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:31 smithi028 ceph-mon[37984]: pgmap v1097: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:56:31.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:31 smithi028 ceph-mon[37984]: 6.1c scrub starts 2024-02-05T06:56:31.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:31 smithi028 ceph-mon[37984]: 6.1c scrub ok 2024-02-05T06:56:31.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:31 smithi167 ceph-mon[32774]: 5.15 scrub starts 2024-02-05T06:56:31.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:31 smithi167 ceph-mon[32774]: 5.15 scrub ok 2024-02-05T06:56:31.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:31 smithi167 ceph-mon[32774]: 6.1a scrub starts 2024-02-05T06:56:31.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:31 smithi167 ceph-mon[32774]: 6.1a scrub ok 2024-02-05T06:56:31.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:31 smithi167 ceph-mon[32774]: pgmap v1097: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:56:31.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:31 smithi167 ceph-mon[32774]: 6.1c scrub starts 2024-02-05T06:56:31.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:31 smithi167 ceph-mon[32774]: 6.1c scrub ok 2024-02-05T06:56:32.414 INFO:teuthology.orchestra.run.smithi028.stdout:77309411634 2024-02-05T06:56:32.415 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-05T06:56:32.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:32 smithi028 ceph-mon[29442]: 3.c scrub starts 2024-02-05T06:56:32.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:32 smithi028 ceph-mon[29442]: 3.c scrub ok 2024-02-05T06:56:32.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:32 smithi028 ceph-mon[29442]: 5.18 scrub starts 2024-02-05T06:56:32.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:32 smithi028 ceph-mon[29442]: 5.18 scrub ok 2024-02-05T06:56:32.639 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:32 smithi028 ceph-mon[37984]: 3.c scrub starts 2024-02-05T06:56:32.639 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:32 smithi028 ceph-mon[37984]: 3.c scrub ok 2024-02-05T06:56:32.639 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:32 smithi028 ceph-mon[37984]: 5.18 scrub starts 2024-02-05T06:56:32.639 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:32 smithi028 ceph-mon[37984]: 5.18 scrub ok 2024-02-05T06:56:32.639 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:56:32 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:56:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:56:32.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:32 smithi167 ceph-mon[32774]: 3.c scrub starts 2024-02-05T06:56:32.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:32 smithi167 ceph-mon[32774]: 3.c scrub ok 2024-02-05T06:56:32.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:32 smithi167 ceph-mon[32774]: 5.18 scrub starts 2024-02-05T06:56:32.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:32 smithi167 ceph-mon[32774]: 5.18 scrub ok 2024-02-05T06:56:33.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:33 smithi028 ceph-mon[29442]: 5.6 scrub starts 2024-02-05T06:56:33.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:33 smithi028 ceph-mon[29442]: 5.6 scrub ok 2024-02-05T06:56:33.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:33 smithi028 ceph-mon[29442]: pgmap v1098: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:56:33.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:33 smithi028 ceph-mon[29442]: 6.13 deep-scrub starts 2024-02-05T06:56:33.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:33 smithi028 ceph-mon[29442]: 5.9 deep-scrub starts 2024-02-05T06:56:33.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:33 smithi028 ceph-mon[29442]: 5.9 deep-scrub ok 2024-02-05T06:56:33.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:33 smithi028 ceph-mon[37984]: 5.6 scrub starts 2024-02-05T06:56:33.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:33 smithi028 ceph-mon[37984]: 5.6 scrub ok 2024-02-05T06:56:33.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:33 smithi028 ceph-mon[37984]: pgmap v1098: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:56:33.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:33 smithi028 ceph-mon[37984]: 6.13 deep-scrub starts 2024-02-05T06:56:33.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:33 smithi028 ceph-mon[37984]: 5.9 deep-scrub starts 2024-02-05T06:56:33.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:33 smithi028 ceph-mon[37984]: 5.9 deep-scrub ok 2024-02-05T06:56:33.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:33 smithi167 ceph-mon[32774]: 5.6 scrub starts 2024-02-05T06:56:33.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:33 smithi167 ceph-mon[32774]: 5.6 scrub ok 2024-02-05T06:56:33.730 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:33 smithi167 ceph-mon[32774]: pgmap v1098: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:56:33.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:33 smithi167 ceph-mon[32774]: 6.13 deep-scrub starts 2024-02-05T06:56:33.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:33 smithi167 ceph-mon[32774]: 5.9 deep-scrub starts 2024-02-05T06:56:33.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:33 smithi167 ceph-mon[32774]: 5.9 deep-scrub ok 2024-02-05T06:56:33.984 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:56:33 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:56:34.155 INFO:teuthology.orchestra.run.smithi028.stdout:111669149997 2024-02-05T06:56:34.155 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-05T06:56:34.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:34 smithi028 ceph-mon[37984]: 6.13 deep-scrub ok 2024-02-05T06:56:34.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:34 smithi028 ceph-mon[37984]: 6.10 scrub starts 2024-02-05T06:56:34.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:34 smithi028 ceph-mon[37984]: 6.10 scrub ok 2024-02-05T06:56:34.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:34 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:56:34.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:34 smithi028 ceph-mon[29442]: 6.13 deep-scrub ok 2024-02-05T06:56:34.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:34 smithi028 ceph-mon[29442]: 6.10 scrub starts 2024-02-05T06:56:34.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:34 smithi028 ceph-mon[29442]: 6.10 scrub ok 2024-02-05T06:56:34.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:34 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:56:34.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:34 smithi167 ceph-mon[32774]: 6.13 deep-scrub ok 2024-02-05T06:56:34.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:34 smithi167 ceph-mon[32774]: 6.10 scrub starts 2024-02-05T06:56:34.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:34 smithi167 ceph-mon[32774]: 6.10 scrub ok 2024-02-05T06:56:34.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:34 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:56:34.949 INFO:teuthology.orchestra.run.smithi028.stdout:133143986472 2024-02-05T06:56:34.950 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-05T06:56:35.578 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:35 smithi028 ceph-mon[37984]: 3.1c scrub starts 2024-02-05T06:56:35.579 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:35 smithi028 ceph-mon[37984]: 3.1c scrub ok 2024-02-05T06:56:35.579 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:35 smithi028 ceph-mon[37984]: 6.18 scrub starts 2024-02-05T06:56:35.579 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:35 smithi028 ceph-mon[37984]: 6.18 scrub ok 2024-02-05T06:56:35.579 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:35 smithi028 ceph-mon[37984]: pgmap v1099: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-05T06:56:35.579 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:35 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:56:35.579 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:35 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:56:35.579 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:35 smithi028 ceph-mon[37984]: 3.1e scrub starts 2024-02-05T06:56:35.579 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:35 smithi028 ceph-mon[37984]: 3.1e scrub ok 2024-02-05T06:56:35.579 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:35 smithi028 ceph-mon[29442]: 3.1c scrub starts 2024-02-05T06:56:35.579 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:35 smithi028 ceph-mon[29442]: 3.1c scrub ok 2024-02-05T06:56:35.579 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:35 smithi028 ceph-mon[29442]: 6.18 scrub starts 2024-02-05T06:56:35.579 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:35 smithi028 ceph-mon[29442]: 6.18 scrub ok 2024-02-05T06:56:35.580 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:35 smithi028 ceph-mon[29442]: pgmap v1099: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-05T06:56:35.580 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:35 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:56:35.580 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:35 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:56:35.580 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:35 smithi028 ceph-mon[29442]: 3.1e scrub starts 2024-02-05T06:56:35.580 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:35 smithi028 ceph-mon[29442]: 3.1e scrub ok 2024-02-05T06:56:35.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:35 smithi167 ceph-mon[32774]: 3.1c scrub starts 2024-02-05T06:56:35.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:35 smithi167 ceph-mon[32774]: 3.1c scrub ok 2024-02-05T06:56:35.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:35 smithi167 ceph-mon[32774]: 6.18 scrub starts 2024-02-05T06:56:35.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:35 smithi167 ceph-mon[32774]: 6.18 scrub ok 2024-02-05T06:56:35.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:35 smithi167 ceph-mon[32774]: pgmap v1099: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-02-05T06:56:35.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:35 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:56:35.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:35 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:56:35.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:35 smithi167 ceph-mon[32774]: 3.1e scrub starts 2024-02-05T06:56:35.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:35 smithi167 ceph-mon[32774]: 3.1e scrub ok 2024-02-05T06:56:35.912 INFO:teuthology.orchestra.run.smithi028.stdout:210453397791 2024-02-05T06:56:35.913 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-05T06:56:36.307 INFO:teuthology.orchestra.run.smithi028.stdout:34359738686 2024-02-05T06:56:36.307 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-05T06:56:36.644 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:36 smithi028 ceph-mon[37984]: 3.6 scrub starts 2024-02-05T06:56:36.644 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:36 smithi028 ceph-mon[37984]: 3.6 scrub ok 2024-02-05T06:56:36.644 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:36 smithi028 ceph-mon[37984]: osdmap e507: 8 total, 8 up, 8 in 2024-02-05T06:56:36.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:36 smithi028 ceph-mon[29442]: 3.6 scrub starts 2024-02-05T06:56:36.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:36 smithi028 ceph-mon[29442]: 3.6 scrub ok 2024-02-05T06:56:36.644 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:36 smithi028 ceph-mon[29442]: osdmap e507: 8 total, 8 up, 8 in 2024-02-05T06:56:36.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:36 smithi167 ceph-mon[32774]: 3.6 scrub starts 2024-02-05T06:56:36.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:36 smithi167 ceph-mon[32774]: 3.6 scrub ok 2024-02-05T06:56:36.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:36 smithi167 ceph-mon[32774]: osdmap e507: 8 total, 8 up, 8 in 2024-02-05T06:56:37.088 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:56:37.089 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:56:37.130 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:56:37.653 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:37 smithi028 ceph-mon[29442]: pgmap v1101: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 613 B/s rd, 0 op/s 2024-02-05T06:56:37.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:37 smithi028 ceph-mon[29442]: 4.0 scrub starts 2024-02-05T06:56:37.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:37 smithi028 ceph-mon[29442]: 4.0 scrub ok 2024-02-05T06:56:37.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:37 smithi028 ceph-mon[37984]: pgmap v1101: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 613 B/s rd, 0 op/s 2024-02-05T06:56:37.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:37 smithi028 ceph-mon[37984]: 4.0 scrub starts 2024-02-05T06:56:37.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:37 smithi028 ceph-mon[37984]: 4.0 scrub ok 2024-02-05T06:56:37.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:37 smithi167 ceph-mon[32774]: pgmap v1101: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 613 B/s rd, 0 op/s 2024-02-05T06:56:37.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:37 smithi167 ceph-mon[32774]: 4.0 scrub starts 2024-02-05T06:56:37.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:37 smithi167 ceph-mon[32774]: 4.0 scrub ok 2024-02-05T06:56:38.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:38 smithi028 ceph-mon[29442]: 4.15 scrub starts 2024-02-05T06:56:38.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:38 smithi028 ceph-mon[29442]: 4.15 scrub ok 2024-02-05T06:56:38.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:38 smithi028 ceph-mon[29442]: 6.4 scrub starts 2024-02-05T06:56:38.690 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:38 smithi028 ceph-mon[29442]: 6.4 scrub ok 2024-02-05T06:56:38.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:38 smithi028 ceph-mon[37984]: 4.15 scrub starts 2024-02-05T06:56:38.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:38 smithi028 ceph-mon[37984]: 4.15 scrub ok 2024-02-05T06:56:38.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:38 smithi028 ceph-mon[37984]: 6.4 scrub starts 2024-02-05T06:56:38.690 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:38 smithi028 ceph-mon[37984]: 6.4 scrub ok 2024-02-05T06:56:38.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:38 smithi167 ceph-mon[32774]: 4.15 scrub starts 2024-02-05T06:56:38.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:38 smithi167 ceph-mon[32774]: 4.15 scrub ok 2024-02-05T06:56:38.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:38 smithi167 ceph-mon[32774]: 6.4 scrub starts 2024-02-05T06:56:38.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:38 smithi167 ceph-mon[32774]: 6.4 scrub ok 2024-02-05T06:56:39.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:39 smithi028 ceph-mon[29442]: pgmap v1102: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:56:39.654 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:39 smithi028 ceph-mon[37984]: pgmap v1102: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:56:39.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:39 smithi167 ceph-mon[32774]: pgmap v1102: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:56:40.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:40 smithi028 ceph-mon[29442]: 5.1c scrub starts 2024-02-05T06:56:40.685 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:40 smithi028 ceph-mon[29442]: 5.1c scrub ok 2024-02-05T06:56:40.686 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:40 smithi028 ceph-mon[37984]: 5.1c scrub starts 2024-02-05T06:56:40.686 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:40 smithi028 ceph-mon[37984]: 5.1c scrub ok 2024-02-05T06:56:40.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:40 smithi167 ceph-mon[32774]: 5.1c scrub starts 2024-02-05T06:56:40.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:40 smithi167 ceph-mon[32774]: 5.1c scrub ok 2024-02-05T06:56:41.021 INFO:teuthology.orchestra.run.smithi028.stdout:77309411636 2024-02-05T06:56:41.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:41 smithi028 ceph-mon[29442]: 3.18 scrub starts 2024-02-05T06:56:41.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:41 smithi028 ceph-mon[29442]: 3.18 scrub ok 2024-02-05T06:56:41.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:41 smithi028 ceph-mon[29442]: 4.9 scrub starts 2024-02-05T06:56:41.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:41 smithi028 ceph-mon[29442]: 4.9 scrub ok 2024-02-05T06:56:41.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:41 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:56:41.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:41 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:56:41.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:41 smithi028 ceph-mon[29442]: osdmap e508: 8 total, 8 up, 8 in 2024-02-05T06:56:41.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:41 smithi028 ceph-mon[29442]: pgmap v1104: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:56:41.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:41 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/164166575' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-05T06:56:41.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:41 smithi028 ceph-mon[37984]: 3.18 scrub starts 2024-02-05T06:56:41.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:41 smithi028 ceph-mon[37984]: 3.18 scrub ok 2024-02-05T06:56:41.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:41 smithi028 ceph-mon[37984]: 4.9 scrub starts 2024-02-05T06:56:41.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:41 smithi028 ceph-mon[37984]: 4.9 scrub ok 2024-02-05T06:56:41.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:41 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:56:41.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:41 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:56:41.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:41 smithi028 ceph-mon[37984]: osdmap e508: 8 total, 8 up, 8 in 2024-02-05T06:56:41.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:41 smithi028 ceph-mon[37984]: pgmap v1104: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:56:41.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:41 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/164166575' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-05T06:56:41.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:41 smithi167 ceph-mon[32774]: 3.18 scrub starts 2024-02-05T06:56:41.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:41 smithi167 ceph-mon[32774]: 3.18 scrub ok 2024-02-05T06:56:41.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:41 smithi167 ceph-mon[32774]: 4.9 scrub starts 2024-02-05T06:56:41.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:41 smithi167 ceph-mon[32774]: 4.9 scrub ok 2024-02-05T06:56:41.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:41 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:56:41.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:41 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:56:41.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:41 smithi167 ceph-mon[32774]: osdmap e508: 8 total, 8 up, 8 in 2024-02-05T06:56:41.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:41 smithi167 ceph-mon[32774]: pgmap v1104: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:56:41.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:41 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/164166575' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-05T06:56:42.491 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:56:42.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:42 smithi028 ceph-mon[37984]: 3.14 scrub starts 2024-02-05T06:56:42.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:42 smithi028 ceph-mon[37984]: 3.14 scrub ok 2024-02-05T06:56:42.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:42 smithi028 ceph-mon[29442]: 3.14 scrub starts 2024-02-05T06:56:42.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:42 smithi028 ceph-mon[29442]: 3.14 scrub ok 2024-02-05T06:56:42.698 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:56:42 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:56:42] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:56:42.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:42 smithi167 ceph-mon[32774]: 3.14 scrub starts 2024-02-05T06:56:42.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:42 smithi167 ceph-mon[32774]: 3.14 scrub ok 2024-02-05T06:56:42.980 INFO:teuthology.orchestra.run.smithi028.stdout:171798692132 2024-02-05T06:56:42.980 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-05T06:56:43.645 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:43 smithi028 ceph-mon[37984]: pgmap v1105: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:56:43.645 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:43 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:56:43.645 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:43 smithi028 ceph-mon[37984]: 3.e deep-scrub starts 2024-02-05T06:56:43.645 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:43 smithi028 ceph-mon[37984]: 3.e deep-scrub ok 2024-02-05T06:56:43.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:43 smithi028 ceph-mon[29442]: pgmap v1105: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:56:43.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:43 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:56:43.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:43 smithi028 ceph-mon[29442]: 3.e deep-scrub starts 2024-02-05T06:56:43.645 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:43 smithi028 ceph-mon[29442]: 3.e deep-scrub ok 2024-02-05T06:56:43.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:43 smithi167 ceph-mon[32774]: pgmap v1105: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:56:43.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:43 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:56:43.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:43 smithi167 ceph-mon[32774]: 3.e deep-scrub starts 2024-02-05T06:56:43.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:43 smithi167 ceph-mon[32774]: 3.e deep-scrub ok 2024-02-05T06:56:44.232 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:56:43 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:56:44.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:44 smithi028 ceph-mon[29442]: 3.3 scrub starts 2024-02-05T06:56:44.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:44 smithi028 ceph-mon[29442]: 3.3 scrub ok 2024-02-05T06:56:44.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:44 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:56:44.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:44 smithi028 ceph-mon[29442]: 5.17 scrub starts 2024-02-05T06:56:44.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:44 smithi028 ceph-mon[29442]: 5.17 scrub ok 2024-02-05T06:56:44.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:44 smithi028 ceph-mon[37984]: 3.3 scrub starts 2024-02-05T06:56:44.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:44 smithi028 ceph-mon[37984]: 3.3 scrub ok 2024-02-05T06:56:44.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:44 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:56:44.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:44 smithi028 ceph-mon[37984]: 5.17 scrub starts 2024-02-05T06:56:44.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:44 smithi028 ceph-mon[37984]: 5.17 scrub ok 2024-02-05T06:56:44.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:44 smithi167 ceph-mon[32774]: 3.3 scrub starts 2024-02-05T06:56:44.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:44 smithi167 ceph-mon[32774]: 3.3 scrub ok 2024-02-05T06:56:44.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:44 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:56:44.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:44 smithi167 ceph-mon[32774]: 5.17 scrub starts 2024-02-05T06:56:44.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:44 smithi167 ceph-mon[32774]: 5.17 scrub ok 2024-02-05T06:56:45.098 INFO:teuthology.orchestra.run.smithi028.stdout:55834575161 2024-02-05T06:56:45.098 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-05T06:56:45.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:45 smithi028 ceph-mon[29442]: 6.1 scrub starts 2024-02-05T06:56:45.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:45 smithi028 ceph-mon[29442]: 6.1 scrub ok 2024-02-05T06:56:45.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:45 smithi028 ceph-mon[29442]: 5.b scrub starts 2024-02-05T06:56:45.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:45 smithi028 ceph-mon[29442]: 5.b scrub ok 2024-02-05T06:56:45.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:45 smithi028 ceph-mon[29442]: pgmap v1106: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:56:45.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:45 smithi028 ceph-mon[29442]: 6.b scrub starts 2024-02-05T06:56:45.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:45 smithi028 ceph-mon[29442]: 6.b scrub ok 2024-02-05T06:56:45.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:45 smithi028 ceph-mon[37984]: 6.1 scrub starts 2024-02-05T06:56:45.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:45 smithi028 ceph-mon[37984]: 6.1 scrub ok 2024-02-05T06:56:45.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:45 smithi028 ceph-mon[37984]: 5.b scrub starts 2024-02-05T06:56:45.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:45 smithi028 ceph-mon[37984]: 5.b scrub ok 2024-02-05T06:56:45.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:45 smithi028 ceph-mon[37984]: pgmap v1106: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:56:45.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:45 smithi028 ceph-mon[37984]: 6.b scrub starts 2024-02-05T06:56:45.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:45 smithi028 ceph-mon[37984]: 6.b scrub ok 2024-02-05T06:56:45.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:45 smithi167 ceph-mon[32774]: 6.1 scrub starts 2024-02-05T06:56:45.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:45 smithi167 ceph-mon[32774]: 6.1 scrub ok 2024-02-05T06:56:45.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:45 smithi167 ceph-mon[32774]: 5.b scrub starts 2024-02-05T06:56:45.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:45 smithi167 ceph-mon[32774]: 5.b scrub ok 2024-02-05T06:56:45.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:45 smithi167 ceph-mon[32774]: pgmap v1106: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-02-05T06:56:45.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:45 smithi167 ceph-mon[32774]: 6.b scrub starts 2024-02-05T06:56:45.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:45 smithi167 ceph-mon[32774]: 6.b scrub ok 2024-02-05T06:56:46.390 INFO:teuthology.orchestra.run.smithi028.stdout:249108103451 2024-02-05T06:56:46.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:46 smithi028 ceph-mon[37984]: 1.1 scrub starts 2024-02-05T06:56:46.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:46 smithi028 ceph-mon[37984]: 1.1 scrub ok 2024-02-05T06:56:46.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:46 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:56:46.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:46 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:56:46.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:46 smithi028 ceph-mon[37984]: osdmap e509: 8 total, 8 up, 8 in 2024-02-05T06:56:46.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:46 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/584618614' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-05T06:56:46.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:46 smithi028 ceph-mon[29442]: 1.1 scrub starts 2024-02-05T06:56:46.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:46 smithi028 ceph-mon[29442]: 1.1 scrub ok 2024-02-05T06:56:46.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:46 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:56:46.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:46 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:56:46.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:46 smithi028 ceph-mon[29442]: osdmap e509: 8 total, 8 up, 8 in 2024-02-05T06:56:46.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:46 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/584618614' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-05T06:56:46.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:46 smithi167 ceph-mon[32774]: 1.1 scrub starts 2024-02-05T06:56:46.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:46 smithi167 ceph-mon[32774]: 1.1 scrub ok 2024-02-05T06:56:46.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:46 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:56:46.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:46 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:56:46.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:46 smithi167 ceph-mon[32774]: osdmap e509: 8 total, 8 up, 8 in 2024-02-05T06:56:46.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:46 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/584618614' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-05T06:56:47.115 INFO:teuthology.orchestra.run.smithi028.stdout:210453397793 2024-02-05T06:56:47.487 INFO:tasks.cephadm.ceph_manager.ceph:need seq 249108103447 got 249108103451 for osd.7 2024-02-05T06:56:47.487 DEBUG:teuthology.parallel:result is None 2024-02-05T06:56:47.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:47 smithi028 ceph-mon[37984]: 3.1b scrub starts 2024-02-05T06:56:47.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:47 smithi028 ceph-mon[37984]: 3.1b scrub ok 2024-02-05T06:56:47.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:47 smithi028 ceph-mon[37984]: pgmap v1108: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:56:47.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:47 smithi028 ceph-mon[37984]: 1.c scrub starts 2024-02-05T06:56:47.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:47 smithi028 ceph-mon[37984]: 1.c scrub ok 2024-02-05T06:56:47.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:47 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2525844469' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-05T06:56:47.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:47 smithi028 ceph-mon[29442]: 3.1b scrub starts 2024-02-05T06:56:47.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:47 smithi028 ceph-mon[29442]: 3.1b scrub ok 2024-02-05T06:56:47.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:47 smithi028 ceph-mon[29442]: pgmap v1108: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:56:47.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:47 smithi028 ceph-mon[29442]: 1.c scrub starts 2024-02-05T06:56:47.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:47 smithi028 ceph-mon[29442]: 1.c scrub ok 2024-02-05T06:56:47.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:47 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2525844469' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-05T06:56:47.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:47 smithi167 ceph-mon[32774]: 3.1b scrub starts 2024-02-05T06:56:47.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:47 smithi167 ceph-mon[32774]: 3.1b scrub ok 2024-02-05T06:56:47.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:47 smithi167 ceph-mon[32774]: pgmap v1108: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-02-05T06:56:47.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:47 smithi167 ceph-mon[32774]: 1.c scrub starts 2024-02-05T06:56:47.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:47 smithi167 ceph-mon[32774]: 1.c scrub ok 2024-02-05T06:56:47.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:47 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2525844469' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-05T06:56:48.132 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453397791 got 210453397793 for osd.6 2024-02-05T06:56:48.132 DEBUG:teuthology.parallel:result is None 2024-02-05T06:56:48.483 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:56:48.517 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411634 got 77309411636 for osd.2 2024-02-05T06:56:48.517 DEBUG:teuthology.parallel:result is None 2024-02-05T06:56:48.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:48 smithi028 ceph-mon[37984]: 3.5 deep-scrub starts 2024-02-05T06:56:48.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:48 smithi028 ceph-mon[37984]: 3.5 deep-scrub ok 2024-02-05T06:56:48.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:48 smithi028 ceph-mon[37984]: 5.10 scrub starts 2024-02-05T06:56:48.699 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:48 smithi028 ceph-mon[37984]: 5.10 scrub ok 2024-02-05T06:56:48.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:48 smithi028 ceph-mon[29442]: 3.5 deep-scrub starts 2024-02-05T06:56:48.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:48 smithi028 ceph-mon[29442]: 3.5 deep-scrub ok 2024-02-05T06:56:48.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:48 smithi028 ceph-mon[29442]: 5.10 scrub starts 2024-02-05T06:56:48.700 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:48 smithi028 ceph-mon[29442]: 5.10 scrub ok 2024-02-05T06:56:48.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:48 smithi167 ceph-mon[32774]: 3.5 deep-scrub starts 2024-02-05T06:56:48.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:48 smithi167 ceph-mon[32774]: 3.5 deep-scrub ok 2024-02-05T06:56:48.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:48 smithi167 ceph-mon[32774]: 5.10 scrub starts 2024-02-05T06:56:48.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:48 smithi167 ceph-mon[32774]: 5.10 scrub ok 2024-02-05T06:56:49.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:49 smithi028 ceph-mon[29442]: pgmap v1109: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:56:49.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:49 smithi028 ceph-mon[37984]: pgmap v1109: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:56:49.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:49 smithi167 ceph-mon[32774]: pgmap v1109: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:56:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:56:50.024 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:56:51.566 INFO:teuthology.orchestra.run.smithi028.stdout:133143986475 2024-02-05T06:56:51.567 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:51 smithi028 ceph-mon[37984]: pgmap v1110: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:56:51.567 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:51 smithi028 ceph-mon[37984]: 3.4 scrub starts 2024-02-05T06:56:51.567 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:51 smithi028 ceph-mon[37984]: 3.4 scrub ok 2024-02-05T06:56:51.567 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:51 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2545569046' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-05T06:56:51.567 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:51 smithi028 ceph-mon[37984]: 3.d scrub starts 2024-02-05T06:56:51.567 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:51 smithi028 ceph-mon[37984]: 3.d scrub ok 2024-02-05T06:56:51.567 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:51 smithi028 ceph-mon[29442]: pgmap v1110: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:56:51.567 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:51 smithi028 ceph-mon[29442]: 3.4 scrub starts 2024-02-05T06:56:51.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:51 smithi028 ceph-mon[29442]: 3.4 scrub ok 2024-02-05T06:56:51.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:51 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2545569046' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-05T06:56:51.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:51 smithi028 ceph-mon[29442]: 3.d scrub starts 2024-02-05T06:56:51.568 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:51 smithi028 ceph-mon[29442]: 3.d scrub ok 2024-02-05T06:56:51.674 INFO:teuthology.orchestra.run.smithi028.stdout:111669150001 2024-02-05T06:56:51.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:51 smithi167 ceph-mon[32774]: pgmap v1110: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:56:51.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:51 smithi167 ceph-mon[32774]: 3.4 scrub starts 2024-02-05T06:56:51.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:51 smithi167 ceph-mon[32774]: 3.4 scrub ok 2024-02-05T06:56:51.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:51 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2545569046' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-05T06:56:51.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:51 smithi167 ceph-mon[32774]: 3.d scrub starts 2024-02-05T06:56:51.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:51 smithi167 ceph-mon[32774]: 3.d scrub ok 2024-02-05T06:56:52.217 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986472 got 133143986475 for osd.4 2024-02-05T06:56:52.217 DEBUG:teuthology.parallel:result is None 2024-02-05T06:56:52.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:52 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/1450157272' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-05T06:56:52.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:52 smithi028 ceph-mon[29442]: 3.8 deep-scrub starts 2024-02-05T06:56:52.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:52 smithi028 ceph-mon[29442]: 3.8 deep-scrub ok 2024-02-05T06:56:52.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:52 smithi028 ceph-mon[29442]: 6.3 scrub starts 2024-02-05T06:56:52.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:52 smithi028 ceph-mon[29442]: 6.3 scrub ok 2024-02-05T06:56:52.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:52 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/1450157272' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-05T06:56:52.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:52 smithi028 ceph-mon[37984]: 3.8 deep-scrub starts 2024-02-05T06:56:52.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:52 smithi028 ceph-mon[37984]: 3.8 deep-scrub ok 2024-02-05T06:56:52.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:52 smithi028 ceph-mon[37984]: 6.3 scrub starts 2024-02-05T06:56:52.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:52 smithi028 ceph-mon[37984]: 6.3 scrub ok 2024-02-05T06:56:52.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:52 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/1450157272' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-05T06:56:52.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:52 smithi167 ceph-mon[32774]: 3.8 deep-scrub starts 2024-02-05T06:56:52.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:52 smithi167 ceph-mon[32774]: 3.8 deep-scrub ok 2024-02-05T06:56:52.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:52 smithi167 ceph-mon[32774]: 6.3 scrub starts 2024-02-05T06:56:52.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:52 smithi167 ceph-mon[32774]: 6.3 scrub ok 2024-02-05T06:56:52.809 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:56:52 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:56:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:56:53.046 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:56:53.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:53 smithi028 ceph-mon[29442]: pgmap v1111: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:56:53.497 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:53 smithi028 ceph-mon[29442]: 4.10 scrub starts 2024-02-05T06:56:53.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:53 smithi028 ceph-mon[29442]: 4.10 scrub ok 2024-02-05T06:56:53.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:53 smithi028 ceph-mon[29442]: 5.4 scrub starts 2024-02-05T06:56:53.498 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:53 smithi028 ceph-mon[29442]: 5.4 scrub ok 2024-02-05T06:56:53.498 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:53 smithi028 ceph-mon[37984]: pgmap v1111: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:56:53.498 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:53 smithi028 ceph-mon[37984]: 4.10 scrub starts 2024-02-05T06:56:53.498 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:53 smithi028 ceph-mon[37984]: 4.10 scrub ok 2024-02-05T06:56:53.498 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:53 smithi028 ceph-mon[37984]: 5.4 scrub starts 2024-02-05T06:56:53.498 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:53 smithi028 ceph-mon[37984]: 5.4 scrub ok 2024-02-05T06:56:53.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:53 smithi167 ceph-mon[32774]: pgmap v1111: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:56:53.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:53 smithi167 ceph-mon[32774]: 4.10 scrub starts 2024-02-05T06:56:53.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:53 smithi167 ceph-mon[32774]: 4.10 scrub ok 2024-02-05T06:56:53.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:53 smithi167 ceph-mon[32774]: 5.4 scrub starts 2024-02-05T06:56:53.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:53 smithi167 ceph-mon[32774]: 5.4 scrub ok 2024-02-05T06:56:54.015 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149997 got 111669150001 for osd.3 2024-02-05T06:56:54.015 DEBUG:teuthology.parallel:result is None 2024-02-05T06:56:54.232 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:56:53 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:56:54.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:54 smithi028 ceph-mon[29442]: 5.7 deep-scrub starts 2024-02-05T06:56:54.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:54 smithi028 ceph-mon[29442]: 5.7 deep-scrub ok 2024-02-05T06:56:54.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:54 smithi028 ceph-mon[29442]: 5.e scrub starts 2024-02-05T06:56:54.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:54 smithi028 ceph-mon[29442]: 5.e scrub ok 2024-02-05T06:56:54.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:54 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:56:54.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:54 smithi028 ceph-mon[29442]: 1.4 scrub starts 2024-02-05T06:56:54.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:54 smithi028 ceph-mon[29442]: 1.4 scrub ok 2024-02-05T06:56:54.637 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:54 smithi028 ceph-mon[29442]: 6.12 scrub starts 2024-02-05T06:56:54.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:54 smithi028 ceph-mon[29442]: 6.12 scrub ok 2024-02-05T06:56:54.638 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:54 smithi028 ceph-mon[37984]: 5.7 deep-scrub starts 2024-02-05T06:56:54.638 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:54 smithi028 ceph-mon[37984]: 5.7 deep-scrub ok 2024-02-05T06:56:54.638 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:54 smithi028 ceph-mon[37984]: 5.e scrub starts 2024-02-05T06:56:54.638 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:54 smithi028 ceph-mon[37984]: 5.e scrub ok 2024-02-05T06:56:54.638 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:54 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:56:54.638 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:54 smithi028 ceph-mon[37984]: 1.4 scrub starts 2024-02-05T06:56:54.638 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:54 smithi028 ceph-mon[37984]: 1.4 scrub ok 2024-02-05T06:56:54.639 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:54 smithi028 ceph-mon[37984]: 6.12 scrub starts 2024-02-05T06:56:54.639 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:54 smithi028 ceph-mon[37984]: 6.12 scrub ok 2024-02-05T06:56:54.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:54 smithi167 ceph-mon[32774]: 5.7 deep-scrub starts 2024-02-05T06:56:54.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:54 smithi167 ceph-mon[32774]: 5.7 deep-scrub ok 2024-02-05T06:56:54.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:54 smithi167 ceph-mon[32774]: 5.e scrub starts 2024-02-05T06:56:54.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:54 smithi167 ceph-mon[32774]: 5.e scrub ok 2024-02-05T06:56:54.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:54 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:56:54.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:54 smithi167 ceph-mon[32774]: 1.4 scrub starts 2024-02-05T06:56:54.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:54 smithi167 ceph-mon[32774]: 1.4 scrub ok 2024-02-05T06:56:54.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:54 smithi167 ceph-mon[32774]: 6.12 scrub starts 2024-02-05T06:56:54.733 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:54 smithi167 ceph-mon[32774]: 6.12 scrub ok 2024-02-05T06:56:55.365 INFO:teuthology.orchestra.run.smithi028.stdout:55834575163 2024-02-05T06:56:55.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:55 smithi028 ceph-mon[29442]: 4.12 scrub starts 2024-02-05T06:56:55.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:55 smithi028 ceph-mon[29442]: 4.12 scrub ok 2024-02-05T06:56:55.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:55 smithi028 ceph-mon[29442]: 2.1 scrub starts 2024-02-05T06:56:55.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:55 smithi028 ceph-mon[29442]: 2.1 scrub ok 2024-02-05T06:56:55.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:55 smithi028 ceph-mon[29442]: pgmap v1112: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:56:55.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:55 smithi028 ceph-mon[29442]: 3.11 deep-scrub starts 2024-02-05T06:56:55.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:55 smithi028 ceph-mon[29442]: 3.11 deep-scrub ok 2024-02-05T06:56:55.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:55 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/2752257951' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-05T06:56:55.541 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:55 smithi028 ceph-mon[37984]: 4.12 scrub starts 2024-02-05T06:56:55.541 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:55 smithi028 ceph-mon[37984]: 4.12 scrub ok 2024-02-05T06:56:55.541 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:55 smithi028 ceph-mon[37984]: 2.1 scrub starts 2024-02-05T06:56:55.541 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:55 smithi028 ceph-mon[37984]: 2.1 scrub ok 2024-02-05T06:56:55.541 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:55 smithi028 ceph-mon[37984]: pgmap v1112: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:56:55.541 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:55 smithi028 ceph-mon[37984]: 3.11 deep-scrub starts 2024-02-05T06:56:55.542 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:55 smithi028 ceph-mon[37984]: 3.11 deep-scrub ok 2024-02-05T06:56:55.542 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:55 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/2752257951' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-05T06:56:55.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:55 smithi167 ceph-mon[32774]: 4.12 scrub starts 2024-02-05T06:56:55.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:55 smithi167 ceph-mon[32774]: 4.12 scrub ok 2024-02-05T06:56:55.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:55 smithi167 ceph-mon[32774]: 2.1 scrub starts 2024-02-05T06:56:55.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:55 smithi167 ceph-mon[32774]: 2.1 scrub ok 2024-02-05T06:56:55.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:55 smithi167 ceph-mon[32774]: pgmap v1112: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:56:55.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:55 smithi167 ceph-mon[32774]: 3.11 deep-scrub starts 2024-02-05T06:56:55.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:55 smithi167 ceph-mon[32774]: 3.11 deep-scrub ok 2024-02-05T06:56:55.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:55 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/2752257951' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-05T06:56:56.439 INFO:teuthology.orchestra.run.smithi028.stdout:171798692135 2024-02-05T06:56:56.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:56 smithi028 ceph-mon[29442]: 4.1a scrub starts 2024-02-05T06:56:56.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:56 smithi028 ceph-mon[29442]: 4.1a scrub ok 2024-02-05T06:56:56.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:56 smithi028 ceph-mon[29442]: 6.5 scrub starts 2024-02-05T06:56:56.631 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:56 smithi028 ceph-mon[29442]: 6.5 scrub ok 2024-02-05T06:56:56.632 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:56 smithi028 ceph-mon[37984]: 4.1a scrub starts 2024-02-05T06:56:56.632 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:56 smithi028 ceph-mon[37984]: 4.1a scrub ok 2024-02-05T06:56:56.632 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:56 smithi028 ceph-mon[37984]: 6.5 scrub starts 2024-02-05T06:56:56.632 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:56 smithi028 ceph-mon[37984]: 6.5 scrub ok 2024-02-05T06:56:56.649 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575161 got 55834575163 for osd.1 2024-02-05T06:56:56.649 DEBUG:teuthology.parallel:result is None 2024-02-05T06:56:56.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:56 smithi167 ceph-mon[32774]: 4.1a scrub starts 2024-02-05T06:56:56.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:56 smithi167 ceph-mon[32774]: 4.1a scrub ok 2024-02-05T06:56:56.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:56 smithi167 ceph-mon[32774]: 6.5 scrub starts 2024-02-05T06:56:56.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:56 smithi167 ceph-mon[32774]: 6.5 scrub ok 2024-02-05T06:56:57.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:57 smithi028 ceph-mon[29442]: 6.a scrub starts 2024-02-05T06:56:57.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:57 smithi028 ceph-mon[29442]: 6.a scrub ok 2024-02-05T06:56:57.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:57 smithi028 ceph-mon[29442]: 3.1f deep-scrub starts 2024-02-05T06:56:57.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:57 smithi028 ceph-mon[29442]: 3.1f deep-scrub ok 2024-02-05T06:56:57.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:57 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/462383209' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-05T06:56:57.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:57 smithi028 ceph-mon[29442]: pgmap v1113: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 993 B/s rd, 0 op/s 2024-02-05T06:56:57.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:57 smithi028 ceph-mon[37984]: 6.a scrub starts 2024-02-05T06:56:57.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:57 smithi028 ceph-mon[37984]: 6.a scrub ok 2024-02-05T06:56:57.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:57 smithi028 ceph-mon[37984]: 3.1f deep-scrub starts 2024-02-05T06:56:57.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:57 smithi028 ceph-mon[37984]: 3.1f deep-scrub ok 2024-02-05T06:56:57.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:57 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/462383209' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-05T06:56:57.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:57 smithi028 ceph-mon[37984]: pgmap v1113: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 993 B/s rd, 0 op/s 2024-02-05T06:56:57.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:57 smithi167 ceph-mon[32774]: 6.a scrub starts 2024-02-05T06:56:57.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:57 smithi167 ceph-mon[32774]: 6.a scrub ok 2024-02-05T06:56:57.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:57 smithi167 ceph-mon[32774]: 3.1f deep-scrub starts 2024-02-05T06:56:57.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:57 smithi167 ceph-mon[32774]: 3.1f deep-scrub ok 2024-02-05T06:56:57.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:57 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/462383209' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-05T06:56:57.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:57 smithi167 ceph-mon[32774]: pgmap v1113: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 993 B/s rd, 0 op/s 2024-02-05T06:56:57.903 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798692132 got 171798692135 for osd.5 2024-02-05T06:56:57.903 DEBUG:teuthology.parallel:result is None 2024-02-05T06:56:58.669 INFO:teuthology.orchestra.run.smithi028.stdout:34359738691 2024-02-05T06:56:58.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:58 smithi028 ceph-mon[29442]: 6.19 deep-scrub starts 2024-02-05T06:56:58.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:58 smithi028 ceph-mon[29442]: 6.19 deep-scrub ok 2024-02-05T06:56:58.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:58 smithi028 ceph-mon[29442]: 4.4 scrub starts 2024-02-05T06:56:58.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:58 smithi028 ceph-mon[29442]: 4.4 scrub ok 2024-02-05T06:56:58.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:58 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:56:58.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:58 smithi028 ceph-mon[29442]: 5.1a scrub starts 2024-02-05T06:56:58.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:58 smithi028 ceph-mon[29442]: 5.1a scrub ok 2024-02-05T06:56:58.670 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:58 smithi028 ceph-mon[37984]: 6.19 deep-scrub starts 2024-02-05T06:56:58.670 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:58 smithi028 ceph-mon[37984]: 6.19 deep-scrub ok 2024-02-05T06:56:58.670 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:58 smithi028 ceph-mon[37984]: 4.4 scrub starts 2024-02-05T06:56:58.670 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:58 smithi028 ceph-mon[37984]: 4.4 scrub ok 2024-02-05T06:56:58.670 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:58 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:56:58.671 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:58 smithi028 ceph-mon[37984]: 5.1a scrub starts 2024-02-05T06:56:58.671 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:58 smithi028 ceph-mon[37984]: 5.1a scrub ok 2024-02-05T06:56:58.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:58 smithi167 ceph-mon[32774]: 6.19 deep-scrub starts 2024-02-05T06:56:58.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:58 smithi167 ceph-mon[32774]: 6.19 deep-scrub ok 2024-02-05T06:56:58.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:58 smithi167 ceph-mon[32774]: 4.4 scrub starts 2024-02-05T06:56:58.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:58 smithi167 ceph-mon[32774]: 4.4 scrub ok 2024-02-05T06:56:58.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:58 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-05T06:56:58.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:58 smithi167 ceph-mon[32774]: 5.1a scrub starts 2024-02-05T06:56:58.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:58 smithi167 ceph-mon[32774]: 5.1a scrub ok 2024-02-05T06:56:59.410 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738686 got 34359738691 for osd.0 2024-02-05T06:56:59.410 DEBUG:teuthology.parallel:result is None 2024-02-05T06:56:59.411 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-02-05T06:56:59.411 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T06:56:59.661 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:56:59.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:59 smithi028 ceph-mon[29442]: 3.1 scrub starts 2024-02-05T06:56:59.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:59 smithi028 ceph-mon[29442]: 3.1 scrub ok 2024-02-05T06:56:59.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:59 smithi028 ceph-mon[29442]: pgmap v1114: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:56:59.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:59 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/939332676' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-05T06:56:59.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:59 smithi028 ceph-mon[29442]: 6.15 scrub starts 2024-02-05T06:56:59.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:56:59 smithi028 ceph-mon[29442]: 6.15 scrub ok 2024-02-05T06:56:59.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:59 smithi028 ceph-mon[37984]: 3.1 scrub starts 2024-02-05T06:56:59.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:59 smithi028 ceph-mon[37984]: 3.1 scrub ok 2024-02-05T06:56:59.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:59 smithi028 ceph-mon[37984]: pgmap v1114: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:56:59.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:59 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/939332676' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-05T06:56:59.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:59 smithi028 ceph-mon[37984]: 6.15 scrub starts 2024-02-05T06:56:59.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:56:59 smithi028 ceph-mon[37984]: 6.15 scrub ok 2024-02-05T06:56:59.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:59 smithi167 ceph-mon[32774]: 3.1 scrub starts 2024-02-05T06:56:59.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:59 smithi167 ceph-mon[32774]: 3.1 scrub ok 2024-02-05T06:56:59.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:59 smithi167 ceph-mon[32774]: pgmap v1114: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:56:59.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:59 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/939332676' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-05T06:56:59.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:59 smithi167 ceph-mon[32774]: 6.15 scrub starts 2024-02-05T06:56:59.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:56:59 smithi167 ceph-mon[32774]: 6.15 scrub ok 2024-02-05T06:57:00.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:00 smithi028 ceph-mon[29442]: 3.7 scrub starts 2024-02-05T06:57:00.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:00 smithi028 ceph-mon[29442]: 3.7 scrub ok 2024-02-05T06:57:00.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:00 smithi028 ceph-mon[37984]: 3.7 scrub starts 2024-02-05T06:57:00.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:00 smithi028 ceph-mon[37984]: 3.7 scrub ok 2024-02-05T06:57:00.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:00 smithi167 ceph-mon[32774]: 3.7 scrub starts 2024-02-05T06:57:00.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:00 smithi167 ceph-mon[32774]: 3.7 scrub ok 2024-02-05T06:57:01.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:01 smithi028 ceph-mon[29442]: 4.8 scrub starts 2024-02-05T06:57:01.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:01 smithi028 ceph-mon[29442]: 4.8 scrub ok 2024-02-05T06:57:01.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:01 smithi028 ceph-mon[29442]: 4.1f scrub starts 2024-02-05T06:57:01.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:01 smithi028 ceph-mon[29442]: 4.1f scrub ok 2024-02-05T06:57:01.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:01 smithi028 ceph-mon[29442]: 3.0 deep-scrub starts 2024-02-05T06:57:01.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:01 smithi028 ceph-mon[29442]: 3.0 deep-scrub ok 2024-02-05T06:57:01.487 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:01 smithi028 ceph-mon[29442]: pgmap v1115: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:57:01.487 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:01 smithi028 ceph-mon[37984]: 4.8 scrub starts 2024-02-05T06:57:01.487 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:01 smithi028 ceph-mon[37984]: 4.8 scrub ok 2024-02-05T06:57:01.488 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:01 smithi028 ceph-mon[37984]: 4.1f scrub starts 2024-02-05T06:57:01.488 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:01 smithi028 ceph-mon[37984]: 4.1f scrub ok 2024-02-05T06:57:01.488 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:01 smithi028 ceph-mon[37984]: 3.0 deep-scrub starts 2024-02-05T06:57:01.488 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:01 smithi028 ceph-mon[37984]: 3.0 deep-scrub ok 2024-02-05T06:57:01.488 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:01 smithi028 ceph-mon[37984]: pgmap v1115: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:57:01.717 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:57:01.721 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-05T06:57:01.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:01 smithi167 ceph-mon[32774]: 4.8 scrub starts 2024-02-05T06:57:01.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:01 smithi167 ceph-mon[32774]: 4.8 scrub ok 2024-02-05T06:57:01.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:01 smithi167 ceph-mon[32774]: 4.1f scrub starts 2024-02-05T06:57:01.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:01 smithi167 ceph-mon[32774]: 4.1f scrub ok 2024-02-05T06:57:01.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:01 smithi167 ceph-mon[32774]: 3.0 deep-scrub starts 2024-02-05T06:57:01.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:01 smithi167 ceph-mon[32774]: 3.0 deep-scrub ok 2024-02-05T06:57:01.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:01 smithi167 ceph-mon[32774]: pgmap v1115: 152 pgs: 152 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-02-05T06:57:02.504 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":1115,"stamp":"2024-02-05T06:57:00.660948+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":2536,"num_read_kb":2288,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":417,"num_bytes_recovered":276079,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3912,"ondisk_log_size":3912,"up":456,"acting":456,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":456,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6590812,"kb_used_data":7964,"kb_used_omap":12,"kb_used_meta":6582771,"kb_avail":743370404,"statfs":{"total":767960285184,"available":761211293696,"internally_reserved":0,"allocated":8155136,"data_stored":5013020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":6740758065},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003440"},"pg_stats":[{"pgid":"4.c","version":"465'35","reported_seq":776,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.355055+0000","last_change":"2024-02-05T06:55:53.783967+0000","last_active":"2024-02-05T06:56:46.355055+0000","last_peered":"2024-02-05T06:56:46.355055+0000","last_clean":"2024-02-05T06:56:46.355055+0000","last_became_active":"2024-02-05T06:55:53.782172+0000","last_became_peered":"2024-02-05T06:55:53.782172+0000","last_unstale":"2024-02-05T06:56:46.355055+0000","last_undegraded":"2024-02-05T06:56:46.355055+0000","last_fullsized":"2024-02-05T06:56:46.355055+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'35","last_scrub_stamp":"2024-02-05T06:55:38.386848+0000","last_deep_scrub":"465'35","last_deep_scrub_stamp":"2024-02-05T06:55:38.386848+0000","last_clean_scrub_stamp":"2024-02-05T06:55:38.386848+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:05.861117+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":974,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.849829+0000","last_change":"2024-02-05T06:55:51.461343+0000","last_active":"2024-02-05T06:56:46.849829+0000","last_peered":"2024-02-05T06:56:46.849829+0000","last_clean":"2024-02-05T06:56:46.849829+0000","last_became_active":"2024-02-05T06:36:54.971304+0000","last_became_peered":"2024-02-05T06:36:54.971304+0000","last_unstale":"2024-02-05T06:56:46.849829+0000","last_undegraded":"2024-02-05T06:56:46.849829+0000","last_fullsized":"2024-02-05T06:56:46.849829+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:51.461255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:55:51.461255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:09.634058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"249'13","reported_seq":1331,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:49.866908+0000","last_change":"2024-02-05T06:56:27.571237+0000","last_active":"2024-02-05T06:56:49.866908+0000","last_peered":"2024-02-05T06:56:49.866908+0000","last_clean":"2024-02-05T06:56:49.866908+0000","last_became_active":"2024-02-05T06:41:43.413139+0000","last_became_peered":"2024-02-05T06:41:43.413139+0000","last_unstale":"2024-02-05T06:56:49.866908+0000","last_undegraded":"2024-02-05T06:56:49.866908+0000","last_fullsized":"2024-02-05T06:56:49.866908+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'13","last_scrub_stamp":"2024-02-05T06:56:27.571150+0000","last_deep_scrub":"92'9","last_deep_scrub_stamp":"2024-02-05T06:34:52.283780+0000","last_clean_scrub_stamp":"2024-02-05T06:56:27.571150+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:47.132645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":467,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.356089+0000","last_change":"2024-02-05T06:56:19.585516+0000","last_active":"2024-02-05T06:56:46.356089+0000","last_peered":"2024-02-05T06:56:46.356089+0000","last_clean":"2024-02-05T06:56:46.356089+0000","last_became_active":"2024-02-05T06:55:53.775928+0000","last_became_peered":"2024-02-05T06:55:53.775928+0000","last_unstale":"2024-02-05T06:56:46.356089+0000","last_undegraded":"2024-02-05T06:56:46.356089+0000","last_fullsized":"2024-02-05T06:56:46.356089+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:19.585408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:30.057099+0000","last_clean_scrub_stamp":"2024-02-05T06:56:19.585408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:28.152706+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":1101,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.358055+0000","last_change":"2024-02-05T06:55:58.815445+0000","last_active":"2024-02-05T06:56:46.358055+0000","last_peered":"2024-02-05T06:56:46.358055+0000","last_clean":"2024-02-05T06:56:46.358055+0000","last_became_active":"2024-02-05T06:40:30.095202+0000","last_became_peered":"2024-02-05T06:40:30.095202+0000","last_unstale":"2024-02-05T06:56:46.358055+0000","last_undegraded":"2024-02-05T06:56:46.358055+0000","last_fullsized":"2024-02-05T06:56:46.358055+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:55:58.815341+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:55:58.815341+0000","last_clean_scrub_stamp":"2024-02-05T06:55:58.815341+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:27.464644+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"465'53","reported_seq":810,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.355838+0000","last_change":"2024-02-05T06:56:17.513167+0000","last_active":"2024-02-05T06:56:46.355838+0000","last_peered":"2024-02-05T06:56:46.355838+0000","last_clean":"2024-02-05T06:56:46.355838+0000","last_became_active":"2024-02-05T06:55:53.772755+0000","last_became_peered":"2024-02-05T06:55:53.772755+0000","last_unstale":"2024-02-05T06:56:46.355838+0000","last_undegraded":"2024-02-05T06:56:46.355838+0000","last_fullsized":"2024-02-05T06:56:46.355838+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'53","last_scrub_stamp":"2024-02-05T06:56:17.513011+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-02-05T06:35:26.274944+0000","last_clean_scrub_stamp":"2024-02-05T06:56:17.513011+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:24.597959+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":58,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1010,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.852585+0000","last_change":"2024-02-05T06:56:22.237388+0000","last_active":"2024-02-05T06:56:46.852585+0000","last_peered":"2024-02-05T06:56:46.852585+0000","last_clean":"2024-02-05T06:56:46.852585+0000","last_became_active":"2024-02-05T06:39:13.240038+0000","last_became_peered":"2024-02-05T06:39:13.240038+0000","last_unstale":"2024-02-05T06:56:46.852585+0000","last_undegraded":"2024-02-05T06:56:46.852585+0000","last_fullsized":"2024-02-05T06:56:46.852585+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:22.237307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:56:22.237307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:52.041823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":990,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.353011+0000","last_change":"2024-02-05T06:56:06.821933+0000","last_active":"2024-02-05T06:56:46.353011+0000","last_peered":"2024-02-05T06:56:46.353011+0000","last_clean":"2024-02-05T06:56:46.353011+0000","last_became_active":"2024-02-05T06:36:54.958082+0000","last_became_peered":"2024-02-05T06:36:54.958082+0000","last_unstale":"2024-02-05T06:56:46.353011+0000","last_undegraded":"2024-02-05T06:56:46.353011+0000","last_fullsized":"2024-02-05T06:56:46.353011+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:06.821835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:38.072995+0000","last_clean_scrub_stamp":"2024-02-05T06:56:06.821835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:10.479382+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1012,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.873459+0000","last_change":"2024-02-05T06:56:14.793063+0000","last_active":"2024-02-05T06:56:46.873459+0000","last_peered":"2024-02-05T06:56:46.873459+0000","last_clean":"2024-02-05T06:56:46.873459+0000","last_became_active":"2024-02-05T06:36:54.958805+0000","last_became_peered":"2024-02-05T06:36:54.958805+0000","last_unstale":"2024-02-05T06:56:46.873459+0000","last_undegraded":"2024-02-05T06:56:46.873459+0000","last_fullsized":"2024-02-05T06:56:46.873459+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:14.792963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:41.833719+0000","last_clean_scrub_stamp":"2024-02-05T06:56:14.792963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:30.282801+0000","scrub_duration":188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"83'128","reported_seq":1014,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.356776+0000","last_change":"2024-02-05T06:56:00.723471+0000","last_active":"2024-02-05T06:56:46.356776+0000","last_peered":"2024-02-05T06:56:46.356776+0000","last_clean":"2024-02-05T06:56:46.356776+0000","last_became_active":"2024-02-05T06:41:46.053324+0000","last_became_peered":"2024-02-05T06:41:46.053324+0000","last_unstale":"2024-02-05T06:56:46.356776+0000","last_undegraded":"2024-02-05T06:56:46.356776+0000","last_fullsized":"2024-02-05T06:56:46.356776+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:56:00.723365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:56:00.723365+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:17.238743+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"83'128","reported_seq":1158,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.352265+0000","last_change":"2024-02-05T06:56:04.850133+0000","last_active":"2024-02-05T06:56:46.352265+0000","last_peered":"2024-02-05T06:56:46.352265+0000","last_clean":"2024-02-05T06:56:46.352265+0000","last_became_active":"2024-02-05T06:51:59.303575+0000","last_became_peered":"2024-02-05T06:51:59.303575+0000","last_unstale":"2024-02-05T06:56:46.352265+0000","last_undegraded":"2024-02-05T06:56:46.352265+0000","last_fullsized":"2024-02-05T06:56:46.352265+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:56:04.850042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:56:04.850042+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:07.119585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"465'32","reported_seq":794,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.356133+0000","last_change":"2024-02-05T06:56:29.400573+0000","last_active":"2024-02-05T06:56:46.356133+0000","last_peered":"2024-02-05T06:56:46.356133+0000","last_clean":"2024-02-05T06:56:46.356133+0000","last_became_active":"2024-02-05T06:55:53.784136+0000","last_became_peered":"2024-02-05T06:55:53.784136+0000","last_unstale":"2024-02-05T06:56:46.356133+0000","last_undegraded":"2024-02-05T06:56:46.356133+0000","last_fullsized":"2024-02-05T06:56:46.356133+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'32","last_scrub_stamp":"2024-02-05T06:56:29.400436+0000","last_deep_scrub":"115'16","last_deep_scrub_stamp":"2024-02-05T06:36:34.793243+0000","last_clean_scrub_stamp":"2024-02-05T06:56:29.400436+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:38.167636+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":698,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.350946+0000","last_change":"2024-02-05T06:55:46.892950+0000","last_active":"2024-02-05T06:56:46.350946+0000","last_peered":"2024-02-05T06:56:46.350946+0000","last_clean":"2024-02-05T06:56:46.350946+0000","last_became_active":"2024-02-05T06:40:27.086808+0000","last_became_peered":"2024-02-05T06:40:27.086808+0000","last_unstale":"2024-02-05T06:56:46.350946+0000","last_undegraded":"2024-02-05T06:56:46.350946+0000","last_fullsized":"2024-02-05T06:56:46.350946+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:46.892872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:41.860956+0000","last_clean_scrub_stamp":"2024-02-05T06:55:46.892872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:10.157808+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":594,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.358012+0000","last_change":"2024-02-05T06:56:17.867908+0000","last_active":"2024-02-05T06:56:46.358012+0000","last_peered":"2024-02-05T06:56:46.358012+0000","last_clean":"2024-02-05T06:56:46.358012+0000","last_became_active":"2024-02-05T06:41:43.417343+0000","last_became_peered":"2024-02-05T06:41:43.417343+0000","last_unstale":"2024-02-05T06:56:46.358012+0000","last_undegraded":"2024-02-05T06:56:46.358012+0000","last_fullsized":"2024-02-05T06:56:46.358012+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:17.867792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:42:55.610276+0000","last_clean_scrub_stamp":"2024-02-05T06:56:17.867792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:47.760263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":968,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.850214+0000","last_change":"2024-02-05T06:56:18.662817+0000","last_active":"2024-02-05T06:56:46.850214+0000","last_peered":"2024-02-05T06:56:46.850214+0000","last_clean":"2024-02-05T06:56:46.850214+0000","last_became_active":"2024-02-05T06:41:43.410600+0000","last_became_peered":"2024-02-05T06:41:43.410600+0000","last_unstale":"2024-02-05T06:56:46.850214+0000","last_undegraded":"2024-02-05T06:56:46.850214+0000","last_fullsized":"2024-02-05T06:56:46.850214+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:18.662678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:58.479208+0000","last_clean_scrub_stamp":"2024-02-05T06:56:18.662678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:32.474005+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"465'46","reported_seq":803,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.352321+0000","last_change":"2024-02-05T06:56:21.285225+0000","last_active":"2024-02-05T06:56:46.352321+0000","last_peered":"2024-02-05T06:56:46.352321+0000","last_clean":"2024-02-05T06:56:46.352321+0000","last_became_active":"2024-02-05T06:39:13.257978+0000","last_became_peered":"2024-02-05T06:39:13.257978+0000","last_unstale":"2024-02-05T06:56:46.352321+0000","last_undegraded":"2024-02-05T06:56:46.352321+0000","last_fullsized":"2024-02-05T06:56:46.352321+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'46","last_scrub_stamp":"2024-02-05T06:56:21.285115+0000","last_deep_scrub":"465'46","last_deep_scrub_stamp":"2024-02-05T06:56:21.285115+0000","last_clean_scrub_stamp":"2024-02-05T06:56:21.285115+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:23.718297+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":982,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:51.921230+0000","last_change":"2024-02-05T06:56:51.921252+0000","last_active":"2024-02-05T06:56:51.921230+0000","last_peered":"2024-02-05T06:56:51.921230+0000","last_clean":"2024-02-05T06:56:51.921230+0000","last_became_active":"2024-02-05T06:40:27.213439+0000","last_became_peered":"2024-02-05T06:40:27.213439+0000","last_unstale":"2024-02-05T06:56:51.921230+0000","last_undegraded":"2024-02-05T06:56:51.921230+0000","last_fullsized":"2024-02-05T06:56:51.921230+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:51.921171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:56:51.921171+0000","last_clean_scrub_stamp":"2024-02-05T06:56:51.921171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:18.559378+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"498'14","reported_seq":604,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:53.238731+0000","last_change":"2024-02-05T06:56:53.238731+0000","last_active":"2024-02-05T06:56:53.238731+0000","last_peered":"2024-02-05T06:56:53.238731+0000","last_clean":"2024-02-05T06:56:53.238731+0000","last_became_active":"2024-02-05T06:55:53.778696+0000","last_became_peered":"2024-02-05T06:55:53.778696+0000","last_unstale":"2024-02-05T06:56:53.238731+0000","last_undegraded":"2024-02-05T06:56:53.238731+0000","last_fullsized":"2024-02-05T06:56:53.238731+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'14","last_scrub_stamp":"2024-02-05T06:56:53.238634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:56:53.238634+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:13.635891+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":595,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.357402+0000","last_change":"2024-02-05T06:55:33.998106+0000","last_active":"2024-02-05T06:56:46.357402+0000","last_peered":"2024-02-05T06:56:46.357402+0000","last_clean":"2024-02-05T06:56:46.357402+0000","last_became_active":"2024-02-05T06:41:43.418066+0000","last_became_peered":"2024-02-05T06:41:43.418066+0000","last_unstale":"2024-02-05T06:56:46.357402+0000","last_undegraded":"2024-02-05T06:56:46.357402+0000","last_fullsized":"2024-02-05T06:56:46.357402+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:33.998012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:48:34.623889+0000","last_clean_scrub_stamp":"2024-02-05T06:55:33.998012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:00.278236+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":1010,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.357133+0000","last_change":"2024-02-05T06:55:55.737355+0000","last_active":"2024-02-05T06:56:46.357133+0000","last_peered":"2024-02-05T06:56:46.357133+0000","last_clean":"2024-02-05T06:56:46.357133+0000","last_became_active":"2024-02-05T06:41:46.053993+0000","last_became_peered":"2024-02-05T06:41:46.053993+0000","last_unstale":"2024-02-05T06:56:46.357133+0000","last_undegraded":"2024-02-05T06:56:46.357133+0000","last_fullsized":"2024-02-05T06:56:46.357133+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:55:55.737264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:55:55.737264+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:21.901412+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":487,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.354728+0000","last_change":"2024-02-05T06:56:10.639208+0000","last_active":"2024-02-05T06:56:46.354728+0000","last_peered":"2024-02-05T06:56:46.354728+0000","last_clean":"2024-02-05T06:56:46.354728+0000","last_became_active":"2024-02-05T06:55:53.780475+0000","last_became_peered":"2024-02-05T06:55:53.780475+0000","last_unstale":"2024-02-05T06:56:46.354728+0000","last_undegraded":"2024-02-05T06:56:46.354728+0000","last_fullsized":"2024-02-05T06:56:46.354728+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:10.639107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:56:10.639107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:20.408170+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"465'50","reported_seq":1083,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.849739+0000","last_change":"2024-02-05T06:56:36.871929+0000","last_active":"2024-02-05T06:56:46.849739+0000","last_peered":"2024-02-05T06:56:46.849739+0000","last_clean":"2024-02-05T06:56:46.849739+0000","last_became_active":"2024-02-05T06:40:27.277747+0000","last_became_peered":"2024-02-05T06:40:27.277747+0000","last_unstale":"2024-02-05T06:56:46.849739+0000","last_undegraded":"2024-02-05T06:56:46.849739+0000","last_fullsized":"2024-02-05T06:56:46.849739+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'50","last_scrub_stamp":"2024-02-05T06:56:36.871842+0000","last_deep_scrub":"424'38","last_deep_scrub_stamp":"2024-02-05T06:51:40.451087+0000","last_clean_scrub_stamp":"2024-02-05T06:56:36.871842+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:02.712961+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":980,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.850319+0000","last_change":"2024-02-05T06:55:56.478591+0000","last_active":"2024-02-05T06:56:46.850319+0000","last_peered":"2024-02-05T06:56:46.850319+0000","last_clean":"2024-02-05T06:56:46.850319+0000","last_became_active":"2024-02-05T06:40:27.093739+0000","last_became_peered":"2024-02-05T06:40:27.093739+0000","last_unstale":"2024-02-05T06:56:46.850319+0000","last_undegraded":"2024-02-05T06:56:46.850319+0000","last_fullsized":"2024-02-05T06:56:46.850319+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:56.478502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:53.475540+0000","last_clean_scrub_stamp":"2024-02-05T06:55:56.478502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:57.696502+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":467,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.354725+0000","last_change":"2024-02-05T06:56:07.678259+0000","last_active":"2024-02-05T06:56:46.354725+0000","last_peered":"2024-02-05T06:56:46.354725+0000","last_clean":"2024-02-05T06:56:46.354725+0000","last_became_active":"2024-02-05T06:55:53.777838+0000","last_became_peered":"2024-02-05T06:55:53.777838+0000","last_unstale":"2024-02-05T06:56:46.354725+0000","last_undegraded":"2024-02-05T06:56:46.354725+0000","last_fullsized":"2024-02-05T06:56:46.354725+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:07.678156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:45:49.145117+0000","last_clean_scrub_stamp":"2024-02-05T06:56:07.678156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:08.897862+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":745,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.354955+0000","last_change":"2024-02-05T06:55:55.522216+0000","last_active":"2024-02-05T06:56:46.354955+0000","last_peered":"2024-02-05T06:56:46.354955+0000","last_clean":"2024-02-05T06:56:46.354955+0000","last_became_active":"2024-02-05T06:55:53.782397+0000","last_became_peered":"2024-02-05T06:55:53.782397+0000","last_unstale":"2024-02-05T06:56:46.354955+0000","last_undegraded":"2024-02-05T06:56:46.354955+0000","last_fullsized":"2024-02-05T06:56:46.354955+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:55:55.522125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:55:55.522125+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:22.394591+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"326'11","reported_seq":1268,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:49.866882+0000","last_change":"2024-02-05T06:56:14.663006+0000","last_active":"2024-02-05T06:56:49.866882+0000","last_peered":"2024-02-05T06:56:49.866882+0000","last_clean":"2024-02-05T06:56:49.866882+0000","last_became_active":"2024-02-05T06:36:54.954819+0000","last_became_peered":"2024-02-05T06:36:54.954819+0000","last_unstale":"2024-02-05T06:56:49.866882+0000","last_undegraded":"2024-02-05T06:56:49.866882+0000","last_fullsized":"2024-02-05T06:56:49.866882+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'11","last_scrub_stamp":"2024-02-05T06:56:14.662916+0000","last_deep_scrub":"326'11","last_deep_scrub_stamp":"2024-02-05T06:45:56.212177+0000","last_clean_scrub_stamp":"2024-02-05T06:56:14.662916+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:40.301441+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"465'42","reported_seq":853,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.354542+0000","last_change":"2024-02-05T06:56:12.621051+0000","last_active":"2024-02-05T06:56:46.354542+0000","last_peered":"2024-02-05T06:56:46.354542+0000","last_clean":"2024-02-05T06:56:46.354542+0000","last_became_active":"2024-02-05T06:55:53.781507+0000","last_became_peered":"2024-02-05T06:55:53.781507+0000","last_unstale":"2024-02-05T06:56:46.354542+0000","last_undegraded":"2024-02-05T06:56:46.354542+0000","last_fullsized":"2024-02-05T06:56:46.354542+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'42","last_scrub_stamp":"2024-02-05T06:56:12.620935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:56:12.620935+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:34.962046+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":53,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"95'2","reported_seq":1001,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.352345+0000","last_change":"2024-02-05T06:56:35.127594+0000","last_active":"2024-02-05T06:56:46.352345+0000","last_peered":"2024-02-05T06:56:46.352345+0000","last_clean":"2024-02-05T06:56:46.352345+0000","last_became_active":"2024-02-05T06:36:54.975093+0000","last_became_peered":"2024-02-05T06:36:54.975093+0000","last_unstale":"2024-02-05T06:56:46.352345+0000","last_undegraded":"2024-02-05T06:56:46.352345+0000","last_fullsized":"2024-02-05T06:56:46.352345+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-05T06:56:35.127506+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-05T06:35:25.616708+0000","last_clean_scrub_stamp":"2024-02-05T06:56:35.127506+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:42.999494+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":687,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:52.183530+0000","last_change":"2024-02-05T06:56:52.183555+0000","last_active":"2024-02-05T06:56:52.183530+0000","last_peered":"2024-02-05T06:56:52.183530+0000","last_clean":"2024-02-05T06:56:52.183530+0000","last_became_active":"2024-02-05T06:40:27.099168+0000","last_became_peered":"2024-02-05T06:40:27.099168+0000","last_unstale":"2024-02-05T06:56:52.183530+0000","last_undegraded":"2024-02-05T06:56:52.183530+0000","last_fullsized":"2024-02-05T06:56:52.183530+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:52.183451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:56:52.183451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:56.102342+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":1186,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:53.835113+0000","last_change":"2024-02-05T06:56:53.835136+0000","last_active":"2024-02-05T06:56:53.835113+0000","last_peered":"2024-02-05T06:56:53.835113+0000","last_clean":"2024-02-05T06:56:53.835113+0000","last_became_active":"2024-02-05T06:51:54.332037+0000","last_became_peered":"2024-02-05T06:51:54.332037+0000","last_unstale":"2024-02-05T06:56:53.835113+0000","last_undegraded":"2024-02-05T06:56:53.835113+0000","last_fullsized":"2024-02-05T06:56:53.835113+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:56:53.835057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:56:53.835057+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:18.020028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"335'11","reported_seq":1283,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:49.866897+0000","last_change":"2024-02-05T06:56:17.384021+0000","last_active":"2024-02-05T06:56:49.866897+0000","last_peered":"2024-02-05T06:56:49.866897+0000","last_clean":"2024-02-05T06:56:49.866897+0000","last_became_active":"2024-02-05T06:41:43.423638+0000","last_became_peered":"2024-02-05T06:41:43.423638+0000","last_unstale":"2024-02-05T06:56:49.866897+0000","last_undegraded":"2024-02-05T06:56:49.866897+0000","last_fullsized":"2024-02-05T06:56:49.866897+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"335'11","last_scrub_stamp":"2024-02-05T06:56:17.383944+0000","last_deep_scrub":"335'11","last_deep_scrub_stamp":"2024-02-05T06:52:40.187864+0000","last_clean_scrub_stamp":"2024-02-05T06:56:17.383944+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:19.860181+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"465'35","reported_seq":88,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.351368+0000","last_change":"2024-02-05T06:56:01.659934+0000","last_active":"2024-02-05T06:56:46.351368+0000","last_peered":"2024-02-05T06:56:46.351368+0000","last_clean":"2024-02-05T06:56:46.351368+0000","last_became_active":"2024-02-05T06:55:08.236796+0000","last_became_peered":"2024-02-05T06:55:08.236796+0000","last_unstale":"2024-02-05T06:56:46.351368+0000","last_undegraded":"2024-02-05T06:56:46.351368+0000","last_fullsized":"2024-02-05T06:56:46.351368+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'35","last_scrub_stamp":"2024-02-05T06:56:01.659841+0000","last_deep_scrub":"424'25","last_deep_scrub_stamp":"2024-02-05T06:51:39.813235+0000","last_clean_scrub_stamp":"2024-02-05T06:56:01.659841+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:25.835842+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":595,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.875182+0000","last_change":"2024-02-05T06:56:46.875206+0000","last_active":"2024-02-05T06:56:46.875182+0000","last_peered":"2024-02-05T06:56:46.875182+0000","last_clean":"2024-02-05T06:56:46.875182+0000","last_became_active":"2024-02-05T06:41:43.411732+0000","last_became_peered":"2024-02-05T06:41:43.411732+0000","last_unstale":"2024-02-05T06:56:46.875182+0000","last_undegraded":"2024-02-05T06:56:46.875182+0000","last_fullsized":"2024-02-05T06:56:46.875182+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:46.875115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:56:46.875115+0000","last_clean_scrub_stamp":"2024-02-05T06:56:46.875115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:01.487040+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":960,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.350744+0000","last_change":"2024-02-05T06:56:00.748685+0000","last_active":"2024-02-05T06:56:46.350744+0000","last_peered":"2024-02-05T06:56:46.350744+0000","last_clean":"2024-02-05T06:56:46.350744+0000","last_became_active":"2024-02-05T06:33:35.600485+0000","last_became_peered":"2024-02-05T06:33:35.600485+0000","last_unstale":"2024-02-05T06:56:46.350744+0000","last_undegraded":"2024-02-05T06:56:46.350744+0000","last_fullsized":"2024-02-05T06:56:46.350744+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:00.748582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:56:00.748582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:29.338413+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":1189,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.853027+0000","last_change":"2024-02-05T06:56:17.299742+0000","last_active":"2024-02-05T06:56:46.853027+0000","last_peered":"2024-02-05T06:56:46.853027+0000","last_clean":"2024-02-05T06:56:46.853027+0000","last_became_active":"2024-02-05T06:51:54.332371+0000","last_became_peered":"2024-02-05T06:51:54.332371+0000","last_unstale":"2024-02-05T06:56:46.853027+0000","last_undegraded":"2024-02-05T06:56:46.853027+0000","last_fullsized":"2024-02-05T06:56:46.853027+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:56:17.299642+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:56:17.299642+0000","last_clean_scrub_stamp":"2024-02-05T06:56:17.299642+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:40.989374+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":997,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.852466+0000","last_change":"2024-02-05T06:55:55.155977+0000","last_active":"2024-02-05T06:56:46.852466+0000","last_peered":"2024-02-05T06:56:46.852466+0000","last_clean":"2024-02-05T06:56:46.852466+0000","last_became_active":"2024-02-05T06:41:43.411550+0000","last_became_peered":"2024-02-05T06:41:43.411550+0000","last_unstale":"2024-02-05T06:56:46.852466+0000","last_undegraded":"2024-02-05T06:56:46.852466+0000","last_fullsized":"2024-02-05T06:56:46.852466+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:55.155885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:55:55.155885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:02.646148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"483'47","reported_seq":1146,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.353243+0000","last_change":"2024-02-05T06:56:20.326060+0000","last_active":"2024-02-05T06:56:46.353243+0000","last_peered":"2024-02-05T06:56:46.353243+0000","last_clean":"2024-02-05T06:56:46.353243+0000","last_became_active":"2024-02-05T06:41:43.409664+0000","last_became_peered":"2024-02-05T06:41:43.409664+0000","last_unstale":"2024-02-05T06:56:46.353243+0000","last_undegraded":"2024-02-05T06:56:46.353243+0000","last_fullsized":"2024-02-05T06:56:46.353243+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"483'47","last_scrub_stamp":"2024-02-05T06:56:20.325962+0000","last_deep_scrub":"483'47","last_deep_scrub_stamp":"2024-02-05T06:55:09.338593+0000","last_clean_scrub_stamp":"2024-02-05T06:56:20.325962+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:30.784476+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"445'2","reported_seq":1014,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:50.675774+0000","last_change":"2024-02-05T06:56:50.675799+0000","last_active":"2024-02-05T06:56:50.675774+0000","last_peered":"2024-02-05T06:56:50.675774+0000","last_clean":"2024-02-05T06:56:50.675774+0000","last_became_active":"2024-02-05T06:41:43.411404+0000","last_became_peered":"2024-02-05T06:41:43.411404+0000","last_unstale":"2024-02-05T06:56:50.675774+0000","last_undegraded":"2024-02-05T06:56:50.675774+0000","last_fullsized":"2024-02-05T06:56:50.675774+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'2","last_scrub_stamp":"2024-02-05T06:56:50.675717+0000","last_deep_scrub":"445'2","last_deep_scrub_stamp":"2024-02-05T06:52:37.034607+0000","last_clean_scrub_stamp":"2024-02-05T06:56:50.675717+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:03.738104+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":975,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.351989+0000","last_change":"2024-02-05T06:56:44.545932+0000","last_active":"2024-02-05T06:56:46.351989+0000","last_peered":"2024-02-05T06:56:46.351989+0000","last_clean":"2024-02-05T06:56:46.351989+0000","last_became_active":"2024-02-05T06:33:35.664858+0000","last_became_peered":"2024-02-05T06:33:35.664858+0000","last_unstale":"2024-02-05T06:56:46.351989+0000","last_undegraded":"2024-02-05T06:56:46.351989+0000","last_fullsized":"2024-02-05T06:56:46.351989+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:44.545830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:56:44.545830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:06.913870+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":1170,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.852946+0000","last_change":"2024-02-05T06:56:05.282642+0000","last_active":"2024-02-05T06:56:46.852946+0000","last_peered":"2024-02-05T06:56:46.852946+0000","last_clean":"2024-02-05T06:56:46.852946+0000","last_became_active":"2024-02-05T06:51:54.330806+0000","last_became_peered":"2024-02-05T06:51:54.330806+0000","last_unstale":"2024-02-05T06:56:46.852946+0000","last_undegraded":"2024-02-05T06:56:46.852946+0000","last_fullsized":"2024-02-05T06:56:46.852946+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:56:05.282558+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:38.778568+0000","last_clean_scrub_stamp":"2024-02-05T06:56:05.282558+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:27.374792+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":595,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:52.915542+0000","last_change":"2024-02-05T06:56:52.915542+0000","last_active":"2024-02-05T06:56:52.915542+0000","last_peered":"2024-02-05T06:56:52.915542+0000","last_clean":"2024-02-05T06:56:52.915542+0000","last_became_active":"2024-02-05T06:41:43.417751+0000","last_became_peered":"2024-02-05T06:41:43.417751+0000","last_unstale":"2024-02-05T06:56:52.915542+0000","last_undegraded":"2024-02-05T06:56:52.915542+0000","last_fullsized":"2024-02-05T06:56:52.915542+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:52.915434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:56:52.915434+0000","last_clean_scrub_stamp":"2024-02-05T06:56:52.915434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:15.880903+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"465'36","reported_seq":1029,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.352239+0000","last_change":"2024-02-05T06:56:22.259254+0000","last_active":"2024-02-05T06:56:46.352239+0000","last_peered":"2024-02-05T06:56:46.352239+0000","last_clean":"2024-02-05T06:56:46.352239+0000","last_became_active":"2024-02-05T06:33:31.564604+0000","last_became_peered":"2024-02-05T06:33:31.564604+0000","last_unstale":"2024-02-05T06:56:46.352239+0000","last_undegraded":"2024-02-05T06:56:46.352239+0000","last_fullsized":"2024-02-05T06:56:46.352239+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'36","last_scrub_stamp":"2024-02-05T06:56:22.259172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:56:22.259172+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:35.295571+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"355'6","reported_seq":1677,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:55.743398+0000","last_change":"2024-02-05T06:55:49.773020+0000","last_active":"2024-02-05T06:56:55.743398+0000","last_peered":"2024-02-05T06:56:55.743398+0000","last_clean":"2024-02-05T06:56:55.743398+0000","last_became_active":"2024-02-05T06:40:27.071509+0000","last_became_peered":"2024-02-05T06:40:27.071509+0000","last_unstale":"2024-02-05T06:56:55.743398+0000","last_undegraded":"2024-02-05T06:56:55.743398+0000","last_fullsized":"2024-02-05T06:56:55.743398+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'6","last_scrub_stamp":"2024-02-05T06:55:49.772867+0000","last_deep_scrub":"355'6","last_deep_scrub_stamp":"2024-02-05T06:47:46.642454+0000","last_clean_scrub_stamp":"2024-02-05T06:55:49.772867+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:01.906169+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00074151600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":995,"num_read_kb":990,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":984,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.351345+0000","last_change":"2024-02-05T06:55:28.341488+0000","last_active":"2024-02-05T06:56:46.351345+0000","last_peered":"2024-02-05T06:56:46.351345+0000","last_clean":"2024-02-05T06:56:46.351345+0000","last_became_active":"2024-02-05T06:36:54.972264+0000","last_became_peered":"2024-02-05T06:36:54.972264+0000","last_unstale":"2024-02-05T06:56:46.351345+0000","last_undegraded":"2024-02-05T06:56:46.351345+0000","last_fullsized":"2024-02-05T06:56:46.351345+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:28.341411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:55:28.341411+0000","last_clean_scrub_stamp":"2024-02-05T06:55:28.341411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:52.900329+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":995,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.352357+0000","last_change":"2024-02-05T06:56:37.699285+0000","last_active":"2024-02-05T06:56:46.352357+0000","last_peered":"2024-02-05T06:56:46.352357+0000","last_clean":"2024-02-05T06:56:46.352357+0000","last_became_active":"2024-02-05T06:41:43.410910+0000","last_became_peered":"2024-02-05T06:41:43.410910+0000","last_unstale":"2024-02-05T06:56:46.352357+0000","last_undegraded":"2024-02-05T06:56:46.352357+0000","last_fullsized":"2024-02-05T06:56:46.352357+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:37.699195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:53:47.688112+0000","last_clean_scrub_stamp":"2024-02-05T06:56:37.699195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:46.615453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":1151,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.352505+0000","last_change":"2024-02-05T06:56:05.843711+0000","last_active":"2024-02-05T06:56:46.352505+0000","last_peered":"2024-02-05T06:56:46.352505+0000","last_clean":"2024-02-05T06:56:46.352505+0000","last_became_active":"2024-02-05T06:51:54.323320+0000","last_became_peered":"2024-02-05T06:51:54.323320+0000","last_unstale":"2024-02-05T06:56:46.352505+0000","last_undegraded":"2024-02-05T06:56:46.352505+0000","last_fullsized":"2024-02-05T06:56:46.352505+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:56:05.843622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:56:05.843622+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:16.650034+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":701,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:53.023202+0000","last_change":"2024-02-05T06:56:53.023224+0000","last_active":"2024-02-05T06:56:53.023202+0000","last_peered":"2024-02-05T06:56:53.023202+0000","last_clean":"2024-02-05T06:56:53.023202+0000","last_became_active":"2024-02-05T06:41:43.418198+0000","last_became_peered":"2024-02-05T06:41:43.418198+0000","last_unstale":"2024-02-05T06:56:53.023202+0000","last_undegraded":"2024-02-05T06:56:53.023202+0000","last_fullsized":"2024-02-05T06:56:53.023202+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:53.023134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:56:53.023134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:21.357434+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"465'52","reported_seq":1435,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:49.867166+0000","last_change":"2024-02-05T06:56:20.296600+0000","last_active":"2024-02-05T06:56:49.867166+0000","last_peered":"2024-02-05T06:56:49.867166+0000","last_clean":"2024-02-05T06:56:49.867166+0000","last_became_active":"2024-02-05T06:40:27.061111+0000","last_became_peered":"2024-02-05T06:40:27.061111+0000","last_unstale":"2024-02-05T06:56:49.867166+0000","last_undegraded":"2024-02-05T06:56:49.867166+0000","last_fullsized":"2024-02-05T06:56:49.867166+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'52","last_scrub_stamp":"2024-02-05T06:56:20.296465+0000","last_deep_scrub":"465'52","last_deep_scrub_stamp":"2024-02-05T06:56:20.296465+0000","last_clean_scrub_stamp":"2024-02-05T06:56:20.296465+0000","objects_scrubbed":8,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:23.715637+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":975,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.849702+0000","last_change":"2024-02-05T06:55:59.575573+0000","last_active":"2024-02-05T06:56:46.849702+0000","last_peered":"2024-02-05T06:56:46.849702+0000","last_clean":"2024-02-05T06:56:46.849702+0000","last_became_active":"2024-02-05T06:41:43.411335+0000","last_became_peered":"2024-02-05T06:41:43.411335+0000","last_unstale":"2024-02-05T06:56:46.849702+0000","last_undegraded":"2024-02-05T06:56:46.849702+0000","last_fullsized":"2024-02-05T06:56:46.849702+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:59.575492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:55:59.575492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:02.307484+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1118,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.358935+0000","last_change":"2024-02-05T06:55:42.761231+0000","last_active":"2024-02-05T06:56:46.358935+0000","last_peered":"2024-02-05T06:56:46.358935+0000","last_clean":"2024-02-05T06:56:46.358935+0000","last_became_active":"2024-02-05T06:51:54.326585+0000","last_became_peered":"2024-02-05T06:51:54.326585+0000","last_unstale":"2024-02-05T06:56:46.358935+0000","last_undegraded":"2024-02-05T06:56:46.358935+0000","last_fullsized":"2024-02-05T06:56:46.358935+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:55:42.761093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:55:42.761093+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:01.946539+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":595,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.358045+0000","last_change":"2024-02-05T06:56:04.871500+0000","last_active":"2024-02-05T06:56:46.358045+0000","last_peered":"2024-02-05T06:56:46.358045+0000","last_clean":"2024-02-05T06:56:46.358045+0000","last_became_active":"2024-02-05T06:41:43.408773+0000","last_became_peered":"2024-02-05T06:41:43.408773+0000","last_unstale":"2024-02-05T06:56:46.358045+0000","last_undegraded":"2024-02-05T06:56:46.358045+0000","last_fullsized":"2024-02-05T06:56:46.358045+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:04.871418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:56:04.871418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:14.338347+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1032,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.873315+0000","last_change":"2024-02-05T06:56:32.642024+0000","last_active":"2024-02-05T06:56:46.873315+0000","last_peered":"2024-02-05T06:56:46.873315+0000","last_clean":"2024-02-05T06:56:46.873315+0000","last_became_active":"2024-02-05T06:41:43.412387+0000","last_became_peered":"2024-02-05T06:41:43.412387+0000","last_unstale":"2024-02-05T06:56:46.873315+0000","last_undegraded":"2024-02-05T06:56:46.873315+0000","last_fullsized":"2024-02-05T06:56:46.873315+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:32.641914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:34.871805+0000","last_clean_scrub_stamp":"2024-02-05T06:56:32.641914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:41.179139+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"506'26","reported_seq":1088,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.352818+0000","last_change":"2024-02-05T06:56:28.880945+0000","last_active":"2024-02-05T06:56:46.352818+0000","last_peered":"2024-02-05T06:56:46.352818+0000","last_clean":"2024-02-05T06:56:46.352818+0000","last_became_active":"2024-02-05T06:41:43.420458+0000","last_became_peered":"2024-02-05T06:41:43.420458+0000","last_unstale":"2024-02-05T06:56:46.352818+0000","last_undegraded":"2024-02-05T06:56:46.352818+0000","last_fullsized":"2024-02-05T06:56:46.352818+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"506'26","last_scrub_stamp":"2024-02-05T06:56:28.880824+0000","last_deep_scrub":"506'26","last_deep_scrub_stamp":"2024-02-05T06:56:28.880824+0000","last_clean_scrub_stamp":"2024-02-05T06:56:28.880824+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:50.242950+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:54.555119+0000","last_change":"2024-02-05T06:56:54.555138+0000","last_active":"2024-02-05T06:56:54.555119+0000","last_peered":"2024-02-05T06:56:54.555119+0000","last_clean":"2024-02-05T06:56:54.555119+0000","last_became_active":"2024-02-05T06:33:28.542977+0000","last_became_peered":"2024-02-05T06:33:28.542977+0000","last_unstale":"2024-02-05T06:56:54.555119+0000","last_undegraded":"2024-02-05T06:56:54.555119+0000","last_fullsized":"2024-02-05T06:56:54.555119+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:54.555061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:56:54.555061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:09.046543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091640799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":979,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.351103+0000","last_change":"2024-02-05T06:55:48.861478+0000","last_active":"2024-02-05T06:56:46.351103+0000","last_peered":"2024-02-05T06:56:46.351103+0000","last_clean":"2024-02-05T06:56:46.351103+0000","last_became_active":"2024-02-05T06:33:29.551549+0000","last_became_peered":"2024-02-05T06:33:29.551549+0000","last_unstale":"2024-02-05T06:56:46.351103+0000","last_undegraded":"2024-02-05T06:56:46.351103+0000","last_fullsized":"2024-02-05T06:56:46.351103+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:48.861372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:55:48.861372+0000","last_clean_scrub_stamp":"2024-02-05T06:55:48.861372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:56.583303+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":687,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:56.104477+0000","last_change":"2024-02-05T06:56:56.104502+0000","last_active":"2024-02-05T06:56:56.104477+0000","last_peered":"2024-02-05T06:56:56.104477+0000","last_clean":"2024-02-05T06:56:56.104477+0000","last_became_active":"2024-02-05T06:40:27.077966+0000","last_became_peered":"2024-02-05T06:40:27.077966+0000","last_unstale":"2024-02-05T06:56:56.104477+0000","last_undegraded":"2024-02-05T06:56:56.104477+0000","last_fullsized":"2024-02-05T06:56:56.104477+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:56.104390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:45.774441+0000","last_clean_scrub_stamp":"2024-02-05T06:56:56.104390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:59.852125+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":1125,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.358330+0000","last_change":"2024-02-05T06:56:17.945481+0000","last_active":"2024-02-05T06:56:46.358330+0000","last_peered":"2024-02-05T06:56:46.358330+0000","last_clean":"2024-02-05T06:56:46.358330+0000","last_became_active":"2024-02-05T06:51:54.325427+0000","last_became_peered":"2024-02-05T06:51:54.325427+0000","last_unstale":"2024-02-05T06:56:46.358330+0000","last_undegraded":"2024-02-05T06:56:46.358330+0000","last_fullsized":"2024-02-05T06:56:46.358330+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:56:17.945381+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:55:13.625211+0000","last_clean_scrub_stamp":"2024-02-05T06:56:17.945381+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:37.743131+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":984,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.352143+0000","last_change":"2024-02-05T06:56:04.632453+0000","last_active":"2024-02-05T06:56:46.352143+0000","last_peered":"2024-02-05T06:56:46.352143+0000","last_clean":"2024-02-05T06:56:46.352143+0000","last_became_active":"2024-02-05T06:36:54.973387+0000","last_became_peered":"2024-02-05T06:36:54.973387+0000","last_unstale":"2024-02-05T06:56:46.352143+0000","last_undegraded":"2024-02-05T06:56:46.352143+0000","last_fullsized":"2024-02-05T06:56:46.352143+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:04.632368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:56:04.632368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:11.107575+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"465'64","reported_seq":971,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:57.676743+0000","last_change":"2024-02-05T06:56:57.676806+0000","last_active":"2024-02-05T06:56:57.676743+0000","last_peered":"2024-02-05T06:56:57.676743+0000","last_clean":"2024-02-05T06:56:57.676743+0000","last_became_active":"2024-02-05T06:40:08.604483+0000","last_became_peered":"2024-02-05T06:40:08.604483+0000","last_unstale":"2024-02-05T06:56:57.676743+0000","last_undegraded":"2024-02-05T06:56:57.676743+0000","last_fullsized":"2024-02-05T06:56:57.676743+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'64","last_scrub_stamp":"2024-02-05T06:56:57.676658+0000","last_deep_scrub":"326'52","last_deep_scrub_stamp":"2024-02-05T06:45:53.715482+0000","last_clean_scrub_stamp":"2024-02-05T06:56:57.676658+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:59.770868+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":467,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.355034+0000","last_change":"2024-02-05T06:56:43.336295+0000","last_active":"2024-02-05T06:56:46.355034+0000","last_peered":"2024-02-05T06:56:46.355034+0000","last_clean":"2024-02-05T06:56:46.355034+0000","last_became_active":"2024-02-05T06:55:53.782266+0000","last_became_peered":"2024-02-05T06:55:53.782266+0000","last_unstale":"2024-02-05T06:56:46.355034+0000","last_undegraded":"2024-02-05T06:56:46.355034+0000","last_fullsized":"2024-02-05T06:56:46.355034+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:43.336174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:47.126978+0000","last_clean_scrub_stamp":"2024-02-05T06:56:43.336174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:48.067076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"182'3","reported_seq":599,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.357004+0000","last_change":"2024-02-05T06:56:22.817189+0000","last_active":"2024-02-05T06:56:46.357004+0000","last_peered":"2024-02-05T06:56:46.357004+0000","last_clean":"2024-02-05T06:56:46.357004+0000","last_became_active":"2024-02-05T06:41:43.415806+0000","last_became_peered":"2024-02-05T06:41:43.415806+0000","last_unstale":"2024-02-05T06:56:46.357004+0000","last_undegraded":"2024-02-05T06:56:46.357004+0000","last_fullsized":"2024-02-05T06:56:46.357004+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'3","last_scrub_stamp":"2024-02-05T06:56:22.817098+0000","last_deep_scrub":"182'3","last_deep_scrub_stamp":"2024-02-05T06:39:33.373415+0000","last_clean_scrub_stamp":"2024-02-05T06:56:22.817098+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:49.617085+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001220348,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":974,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.850304+0000","last_change":"2024-02-05T06:56:20.730377+0000","last_active":"2024-02-05T06:56:46.850304+0000","last_peered":"2024-02-05T06:56:46.850304+0000","last_clean":"2024-02-05T06:56:46.850304+0000","last_became_active":"2024-02-05T06:40:27.082440+0000","last_became_peered":"2024-02-05T06:40:27.082440+0000","last_unstale":"2024-02-05T06:56:46.850304+0000","last_undegraded":"2024-02-05T06:56:46.850304+0000","last_fullsized":"2024-02-05T06:56:46.850304+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:20.730298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:50:52.403179+0000","last_clean_scrub_stamp":"2024-02-05T06:56:20.730298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:38.646090+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.356865+0000","last_change":"2024-02-05T06:56:44.888318+0000","last_active":"2024-02-05T06:56:46.356865+0000","last_peered":"2024-02-05T06:56:46.356865+0000","last_clean":"2024-02-05T06:56:46.356865+0000","last_became_active":"2024-02-05T06:51:54.325737+0000","last_became_peered":"2024-02-05T06:51:54.325737+0000","last_unstale":"2024-02-05T06:56:46.356865+0000","last_undegraded":"2024-02-05T06:56:46.356865+0000","last_fullsized":"2024-02-05T06:56:46.356865+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:56:44.888186+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:37.282442+0000","last_clean_scrub_stamp":"2024-02-05T06:56:44.888186+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:59.607640+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"83'128","reported_seq":342,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.355978+0000","last_change":"2024-02-05T06:55:54.571484+0000","last_active":"2024-02-05T06:56:46.355978+0000","last_peered":"2024-02-05T06:56:46.355978+0000","last_clean":"2024-02-05T06:56:46.355978+0000","last_became_active":"2024-02-05T06:55:53.781272+0000","last_became_peered":"2024-02-05T06:55:53.781272+0000","last_unstale":"2024-02-05T06:56:46.355978+0000","last_undegraded":"2024-02-05T06:56:46.355978+0000","last_fullsized":"2024-02-05T06:56:46.355978+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:55:54.571345+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:54:47.849978+0000","last_clean_scrub_stamp":"2024-02-05T06:55:54.571345+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:19.045414+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"465'21","reported_seq":1089,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.353092+0000","last_change":"2024-02-05T06:56:10.441093+0000","last_active":"2024-02-05T06:56:46.353092+0000","last_peered":"2024-02-05T06:56:46.353092+0000","last_clean":"2024-02-05T06:56:46.353092+0000","last_became_active":"2024-02-05T06:40:08.584204+0000","last_became_peered":"2024-02-05T06:40:08.584204+0000","last_unstale":"2024-02-05T06:56:46.353092+0000","last_undegraded":"2024-02-05T06:56:46.353092+0000","last_fullsized":"2024-02-05T06:56:46.353092+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'21","last_scrub_stamp":"2024-02-05T06:56:10.441001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:56:10.441001+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:17.612654+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":594,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.357417+0000","last_change":"2024-02-05T06:56:30.847013+0000","last_active":"2024-02-05T06:56:46.357417+0000","last_peered":"2024-02-05T06:56:46.357417+0000","last_clean":"2024-02-05T06:56:46.357417+0000","last_became_active":"2024-02-05T06:41:43.412611+0000","last_became_peered":"2024-02-05T06:41:43.412611+0000","last_unstale":"2024-02-05T06:56:46.357417+0000","last_undegraded":"2024-02-05T06:56:46.357417+0000","last_fullsized":"2024-02-05T06:56:46.357417+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:30.846924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:41.617526+0000","last_clean_scrub_stamp":"2024-02-05T06:56:30.846924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:31.441121+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1012,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.872792+0000","last_change":"2024-02-05T06:56:02.579265+0000","last_active":"2024-02-05T06:56:46.872792+0000","last_peered":"2024-02-05T06:56:46.872792+0000","last_clean":"2024-02-05T06:56:46.872792+0000","last_became_active":"2024-02-05T06:36:54.966016+0000","last_became_peered":"2024-02-05T06:36:54.966016+0000","last_unstale":"2024-02-05T06:56:46.872792+0000","last_undegraded":"2024-02-05T06:56:46.872792+0000","last_fullsized":"2024-02-05T06:56:46.872792+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:02.579186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:56:02.579186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:17.018541+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":971,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.351537+0000","last_change":"2024-02-05T06:55:50.646138+0000","last_active":"2024-02-05T06:56:46.351537+0000","last_peered":"2024-02-05T06:56:46.351537+0000","last_clean":"2024-02-05T06:56:46.351537+0000","last_became_active":"2024-02-05T06:40:27.129625+0000","last_became_peered":"2024-02-05T06:40:27.129625+0000","last_unstale":"2024-02-05T06:56:46.351537+0000","last_undegraded":"2024-02-05T06:56:46.351537+0000","last_fullsized":"2024-02-05T06:56:46.351537+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:50.646032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:55:50.646032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:55.101476+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"83'128","reported_seq":1197,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.852918+0000","last_change":"2024-02-05T06:55:49.199226+0000","last_active":"2024-02-05T06:56:46.852918+0000","last_peered":"2024-02-05T06:56:46.852918+0000","last_clean":"2024-02-05T06:56:46.852918+0000","last_became_active":"2024-02-05T06:51:59.293451+0000","last_became_peered":"2024-02-05T06:51:59.293451+0000","last_unstale":"2024-02-05T06:56:46.852918+0000","last_undegraded":"2024-02-05T06:56:46.852918+0000","last_fullsized":"2024-02-05T06:56:46.852918+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:55:49.199108+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:48:25.725892+0000","last_clean_scrub_stamp":"2024-02-05T06:55:49.199108+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:18.160870+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"465'60","reported_seq":1162,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.852430+0000","last_change":"2024-02-05T06:56:14.234074+0000","last_active":"2024-02-05T06:56:46.852430+0000","last_peered":"2024-02-05T06:56:46.852430+0000","last_clean":"2024-02-05T06:56:46.852430+0000","last_became_active":"2024-02-05T06:40:27.068417+0000","last_became_peered":"2024-02-05T06:40:27.068417+0000","last_unstale":"2024-02-05T06:56:46.852430+0000","last_undegraded":"2024-02-05T06:56:46.852430+0000","last_fullsized":"2024-02-05T06:56:46.852430+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'60","last_scrub_stamp":"2024-02-05T06:56:14.233941+0000","last_deep_scrub":"279'28","last_deep_scrub_stamp":"2024-02-05T06:42:50.148881+0000","last_clean_scrub_stamp":"2024-02-05T06:56:14.233941+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:15.288233+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":693,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:51.027887+0000","last_change":"2024-02-05T06:56:51.027910+0000","last_active":"2024-02-05T06:56:51.027887+0000","last_peered":"2024-02-05T06:56:51.027887+0000","last_clean":"2024-02-05T06:56:51.027887+0000","last_became_active":"2024-02-05T06:41:43.418202+0000","last_became_peered":"2024-02-05T06:41:43.418202+0000","last_unstale":"2024-02-05T06:56:51.027887+0000","last_undegraded":"2024-02-05T06:56:51.027887+0000","last_fullsized":"2024-02-05T06:56:51.027887+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:51.027789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:56:51.027789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:52.404063+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.872893+0000","last_change":"2024-02-05T06:56:44.583092+0000","last_active":"2024-02-05T06:56:46.872893+0000","last_peered":"2024-02-05T06:56:46.872893+0000","last_clean":"2024-02-05T06:56:46.872893+0000","last_became_active":"2024-02-05T06:41:43.413095+0000","last_became_peered":"2024-02-05T06:41:43.413095+0000","last_unstale":"2024-02-05T06:56:46.872893+0000","last_undegraded":"2024-02-05T06:56:46.872893+0000","last_fullsized":"2024-02-05T06:56:46.872893+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:44.582999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:45.073681+0000","last_clean_scrub_stamp":"2024-02-05T06:56:44.582999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:46.979530+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":687,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.358785+0000","last_change":"2024-02-05T06:56:05.923363+0000","last_active":"2024-02-05T06:56:46.358785+0000","last_peered":"2024-02-05T06:56:46.358785+0000","last_clean":"2024-02-05T06:56:46.358785+0000","last_became_active":"2024-02-05T06:40:27.218081+0000","last_became_peered":"2024-02-05T06:40:27.218081+0000","last_unstale":"2024-02-05T06:56:46.358785+0000","last_undegraded":"2024-02-05T06:56:46.358785+0000","last_fullsized":"2024-02-05T06:56:46.358785+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:05.923279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:32.925000+0000","last_clean_scrub_stamp":"2024-02-05T06:56:05.923279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:14.587523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"83'128","reported_seq":1199,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.851903+0000","last_change":"2024-02-05T06:56:46.851929+0000","last_active":"2024-02-05T06:56:46.851903+0000","last_peered":"2024-02-05T06:56:46.851903+0000","last_clean":"2024-02-05T06:56:46.851903+0000","last_became_active":"2024-02-05T06:51:59.296988+0000","last_became_peered":"2024-02-05T06:51:59.296988+0000","last_unstale":"2024-02-05T06:56:46.851903+0000","last_undegraded":"2024-02-05T06:56:46.851903+0000","last_fullsized":"2024-02-05T06:56:46.851903+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:56:46.851842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:56:46.851842+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:55.756240+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":992,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.872451+0000","last_change":"2024-02-05T06:55:45.424698+0000","last_active":"2024-02-05T06:56:46.872451+0000","last_peered":"2024-02-05T06:56:46.872451+0000","last_clean":"2024-02-05T06:56:46.872451+0000","last_became_active":"2024-02-05T06:33:33.592546+0000","last_became_peered":"2024-02-05T06:33:33.592546+0000","last_unstale":"2024-02-05T06:56:46.872451+0000","last_undegraded":"2024-02-05T06:56:46.872451+0000","last_fullsized":"2024-02-05T06:56:46.872451+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:45.424621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:42.685240+0000","last_clean_scrub_stamp":"2024-02-05T06:55:45.424621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:01.990910+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"465'24","reported_seq":781,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.354680+0000","last_change":"2024-02-05T06:56:40.310264+0000","last_active":"2024-02-05T06:56:46.354680+0000","last_peered":"2024-02-05T06:56:46.354680+0000","last_clean":"2024-02-05T06:56:46.354680+0000","last_became_active":"2024-02-05T06:55:53.777933+0000","last_became_peered":"2024-02-05T06:55:53.777933+0000","last_unstale":"2024-02-05T06:56:46.354680+0000","last_undegraded":"2024-02-05T06:56:46.354680+0000","last_fullsized":"2024-02-05T06:56:46.354680+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'24","last_scrub_stamp":"2024-02-05T06:56:40.310150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:56:40.310150+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:55.385961+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":37,"num_read_kb":28,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":687,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.359203+0000","last_change":"2024-02-05T06:56:43.034625+0000","last_active":"2024-02-05T06:56:46.359203+0000","last_peered":"2024-02-05T06:56:46.359203+0000","last_clean":"2024-02-05T06:56:46.359203+0000","last_became_active":"2024-02-05T06:40:27.063300+0000","last_became_peered":"2024-02-05T06:40:27.063300+0000","last_unstale":"2024-02-05T06:56:46.359203+0000","last_undegraded":"2024-02-05T06:56:46.359203+0000","last_fullsized":"2024-02-05T06:56:46.359203+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:43.034489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:56:43.034489+0000","last_clean_scrub_stamp":"2024-02-05T06:56:43.034489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:44.201374+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":962,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.850096+0000","last_change":"2024-02-05T06:56:44.963204+0000","last_active":"2024-02-05T06:56:46.850096+0000","last_peered":"2024-02-05T06:56:46.850096+0000","last_clean":"2024-02-05T06:56:46.850096+0000","last_became_active":"2024-02-05T06:40:27.092613+0000","last_became_peered":"2024-02-05T06:40:27.092613+0000","last_unstale":"2024-02-05T06:56:46.850096+0000","last_undegraded":"2024-02-05T06:56:46.850096+0000","last_fullsized":"2024-02-05T06:56:46.850096+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:44.963072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:55:14.568023+0000","last_clean_scrub_stamp":"2024-02-05T06:56:44.963072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:02.151427+0000","scrub_duration":165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"83'128","reported_seq":756,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.358386+0000","last_change":"2024-02-05T06:55:57.812292+0000","last_active":"2024-02-05T06:56:46.358386+0000","last_peered":"2024-02-05T06:56:46.358386+0000","last_clean":"2024-02-05T06:56:46.358386+0000","last_became_active":"2024-02-05T06:51:59.294191+0000","last_became_peered":"2024-02-05T06:51:59.294191+0000","last_unstale":"2024-02-05T06:56:46.358386+0000","last_undegraded":"2024-02-05T06:56:46.358386+0000","last_fullsized":"2024-02-05T06:56:46.358386+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:55:57.812200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:55:57.812200+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:15.888287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"506'13","reported_seq":890,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:49.867168+0000","last_change":"2024-02-05T06:56:32.958495+0000","last_active":"2024-02-05T06:56:49.867168+0000","last_peered":"2024-02-05T06:56:49.867168+0000","last_clean":"2024-02-05T06:56:49.867168+0000","last_became_active":"2024-02-05T06:40:27.064195+0000","last_became_peered":"2024-02-05T06:40:27.064195+0000","last_unstale":"2024-02-05T06:56:49.867168+0000","last_undegraded":"2024-02-05T06:56:49.867168+0000","last_fullsized":"2024-02-05T06:56:49.867168+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"506'13","last_scrub_stamp":"2024-02-05T06:56:32.958410+0000","last_deep_scrub":"506'13","last_deep_scrub_stamp":"2024-02-05T06:56:32.958410+0000","last_clean_scrub_stamp":"2024-02-05T06:56:32.958410+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:56.948169+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"465'39","reported_seq":639,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.356803+0000","last_change":"2024-02-05T06:55:57.754233+0000","last_active":"2024-02-05T06:56:46.356803+0000","last_peered":"2024-02-05T06:56:46.356803+0000","last_clean":"2024-02-05T06:56:46.356803+0000","last_became_active":"2024-02-05T06:41:43.412579+0000","last_became_peered":"2024-02-05T06:41:43.412579+0000","last_unstale":"2024-02-05T06:56:46.356803+0000","last_undegraded":"2024-02-05T06:56:46.356803+0000","last_fullsized":"2024-02-05T06:56:46.356803+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'39","last_scrub_stamp":"2024-02-05T06:55:57.754110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:55:57.754110+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:59.869330+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"412'4","reported_seq":695,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.359287+0000","last_change":"2024-02-05T06:56:13.027321+0000","last_active":"2024-02-05T06:56:46.359287+0000","last_peered":"2024-02-05T06:56:46.359287+0000","last_clean":"2024-02-05T06:56:46.359287+0000","last_became_active":"2024-02-05T06:40:27.068279+0000","last_became_peered":"2024-02-05T06:40:27.068279+0000","last_unstale":"2024-02-05T06:56:46.359287+0000","last_undegraded":"2024-02-05T06:56:46.359287+0000","last_fullsized":"2024-02-05T06:56:46.359287+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'4","last_scrub_stamp":"2024-02-05T06:56:13.027147+0000","last_deep_scrub":"412'4","last_deep_scrub_stamp":"2024-02-05T06:56:13.027147+0000","last_clean_scrub_stamp":"2024-02-05T06:56:13.027147+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:26.844350+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":595,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:55.969297+0000","last_change":"2024-02-05T06:56:55.969297+0000","last_active":"2024-02-05T06:56:55.969297+0000","last_peered":"2024-02-05T06:56:55.969297+0000","last_clean":"2024-02-05T06:56:55.969297+0000","last_became_active":"2024-02-05T06:41:43.421231+0000","last_became_peered":"2024-02-05T06:41:43.421231+0000","last_unstale":"2024-02-05T06:56:55.969297+0000","last_undegraded":"2024-02-05T06:56:55.969297+0000","last_fullsized":"2024-02-05T06:56:55.969297+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:55.969201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:56:55.969201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:19.486105+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"83'128","reported_seq":623,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.356636+0000","last_change":"2024-02-05T06:56:20.806816+0000","last_active":"2024-02-05T06:56:46.356636+0000","last_peered":"2024-02-05T06:56:46.356636+0000","last_clean":"2024-02-05T06:56:46.356636+0000","last_became_active":"2024-02-05T06:51:59.296150+0000","last_became_peered":"2024-02-05T06:51:59.296150+0000","last_unstale":"2024-02-05T06:56:46.356636+0000","last_undegraded":"2024-02-05T06:56:46.356636+0000","last_fullsized":"2024-02-05T06:56:46.356636+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:56:20.806725+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:48:28.657679+0000","last_clean_scrub_stamp":"2024-02-05T06:56:20.806725+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:32.420942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"465'21","reported_seq":1015,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.850570+0000","last_change":"2024-02-05T06:56:09.572815+0000","last_active":"2024-02-05T06:56:46.850570+0000","last_peered":"2024-02-05T06:56:46.850570+0000","last_clean":"2024-02-05T06:56:46.850570+0000","last_became_active":"2024-02-05T06:40:27.091689+0000","last_became_peered":"2024-02-05T06:40:27.091689+0000","last_unstale":"2024-02-05T06:56:46.850570+0000","last_undegraded":"2024-02-05T06:56:46.850570+0000","last_fullsized":"2024-02-05T06:56:46.850570+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'21","last_scrub_stamp":"2024-02-05T06:56:09.572716+0000","last_deep_scrub":"412'15","last_deep_scrub_stamp":"2024-02-05T06:50:55.397689+0000","last_clean_scrub_stamp":"2024-02-05T06:56:09.572716+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:27.603023+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1004,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.852053+0000","last_change":"2024-02-05T06:55:36.193765+0000","last_active":"2024-02-05T06:56:46.852053+0000","last_peered":"2024-02-05T06:56:46.852053+0000","last_clean":"2024-02-05T06:56:46.852053+0000","last_became_active":"2024-02-05T06:41:43.412130+0000","last_became_peered":"2024-02-05T06:41:43.412130+0000","last_unstale":"2024-02-05T06:56:46.852053+0000","last_undegraded":"2024-02-05T06:56:46.852053+0000","last_fullsized":"2024-02-05T06:56:46.852053+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:36.193661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:55:36.193661+0000","last_clean_scrub_stamp":"2024-02-05T06:55:36.193661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:55.759566+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":594,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.357758+0000","last_change":"2024-02-05T06:56:16.881001+0000","last_active":"2024-02-05T06:56:46.357758+0000","last_peered":"2024-02-05T06:56:46.357758+0000","last_clean":"2024-02-05T06:56:46.357758+0000","last_became_active":"2024-02-05T06:41:43.406094+0000","last_became_peered":"2024-02-05T06:41:43.406094+0000","last_unstale":"2024-02-05T06:56:46.357758+0000","last_undegraded":"2024-02-05T06:56:46.357758+0000","last_fullsized":"2024-02-05T06:56:46.357758+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:16.880873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:44:57.506909+0000","last_clean_scrub_stamp":"2024-02-05T06:56:16.880873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:18.999720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":678,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.358977+0000","last_change":"2024-02-05T06:55:47.843581+0000","last_active":"2024-02-05T06:56:46.358977+0000","last_peered":"2024-02-05T06:56:46.358977+0000","last_clean":"2024-02-05T06:56:46.358977+0000","last_became_active":"2024-02-05T06:40:27.063350+0000","last_became_peered":"2024-02-05T06:40:27.063350+0000","last_unstale":"2024-02-05T06:56:46.358977+0000","last_undegraded":"2024-02-05T06:56:46.358977+0000","last_fullsized":"2024-02-05T06:56:46.358977+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:47.843476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:55:47.843476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:52.499266+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"83'128","reported_seq":1158,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.352164+0000","last_change":"2024-02-05T06:55:54.882340+0000","last_active":"2024-02-05T06:56:46.352164+0000","last_peered":"2024-02-05T06:56:46.352164+0000","last_clean":"2024-02-05T06:56:46.352164+0000","last_became_active":"2024-02-05T06:51:59.303621+0000","last_became_peered":"2024-02-05T06:51:59.303621+0000","last_unstale":"2024-02-05T06:56:46.352164+0000","last_undegraded":"2024-02-05T06:56:46.352164+0000","last_fullsized":"2024-02-05T06:56:46.352164+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:55:54.882237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:55:54.882237+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:11.595754+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"465'21","reported_seq":1040,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.352543+0000","last_change":"2024-02-05T06:55:31.338533+0000","last_active":"2024-02-05T06:56:46.352543+0000","last_peered":"2024-02-05T06:56:46.352543+0000","last_clean":"2024-02-05T06:56:46.352543+0000","last_became_active":"2024-02-05T06:40:27.101116+0000","last_became_peered":"2024-02-05T06:40:27.101116+0000","last_unstale":"2024-02-05T06:56:46.352543+0000","last_undegraded":"2024-02-05T06:56:46.352543+0000","last_fullsized":"2024-02-05T06:56:46.352543+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'21","last_scrub_stamp":"2024-02-05T06:55:31.338415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:55:31.338415+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:53.035265+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":687,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:55.060483+0000","last_change":"2024-02-05T06:56:55.060511+0000","last_active":"2024-02-05T06:56:55.060483+0000","last_peered":"2024-02-05T06:56:55.060483+0000","last_clean":"2024-02-05T06:56:55.060483+0000","last_became_active":"2024-02-05T06:40:27.064388+0000","last_became_peered":"2024-02-05T06:40:27.064388+0000","last_unstale":"2024-02-05T06:56:55.060483+0000","last_undegraded":"2024-02-05T06:56:55.060483+0000","last_fullsized":"2024-02-05T06:56:55.060483+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:55.060402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:56:55.060402+0000","last_clean_scrub_stamp":"2024-02-05T06:56:55.060402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:11.784859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":979,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.850437+0000","last_change":"2024-02-05T06:56:43.822790+0000","last_active":"2024-02-05T06:56:46.850437+0000","last_peered":"2024-02-05T06:56:46.850437+0000","last_clean":"2024-02-05T06:56:46.850437+0000","last_became_active":"2024-02-05T06:40:27.091544+0000","last_became_peered":"2024-02-05T06:40:27.091544+0000","last_unstale":"2024-02-05T06:56:46.850437+0000","last_undegraded":"2024-02-05T06:56:46.850437+0000","last_fullsized":"2024-02-05T06:56:46.850437+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:43.822664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:56:43.822664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:01.135753+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.873205+0000","last_change":"2024-02-05T06:56:16.612678+0000","last_active":"2024-02-05T06:56:46.873205+0000","last_peered":"2024-02-05T06:56:46.873205+0000","last_clean":"2024-02-05T06:56:46.873205+0000","last_became_active":"2024-02-05T06:40:27.090632+0000","last_became_peered":"2024-02-05T06:40:27.090632+0000","last_unstale":"2024-02-05T06:56:46.873205+0000","last_undegraded":"2024-02-05T06:56:46.873205+0000","last_fullsized":"2024-02-05T06:56:46.873205+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:16.612559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:45.010039+0000","last_clean_scrub_stamp":"2024-02-05T06:56:16.612559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:44.852656+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"83'128","reported_seq":623,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.356502+0000","last_change":"2024-02-05T06:56:15.929990+0000","last_active":"2024-02-05T06:56:46.356502+0000","last_peered":"2024-02-05T06:56:46.356502+0000","last_clean":"2024-02-05T06:56:46.356502+0000","last_became_active":"2024-02-05T06:51:59.290466+0000","last_became_peered":"2024-02-05T06:51:59.290466+0000","last_unstale":"2024-02-05T06:56:46.356502+0000","last_undegraded":"2024-02-05T06:56:46.356502+0000","last_fullsized":"2024-02-05T06:56:46.356502+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:56:15.929877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:56:15.929877+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:36.354409+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"465'21","reported_seq":622,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.357535+0000","last_change":"2024-02-05T06:56:37.005030+0000","last_active":"2024-02-05T06:56:46.357535+0000","last_peered":"2024-02-05T06:56:46.357535+0000","last_clean":"2024-02-05T06:56:46.357535+0000","last_became_active":"2024-02-05T06:41:43.413708+0000","last_became_peered":"2024-02-05T06:41:43.413708+0000","last_unstale":"2024-02-05T06:56:46.357535+0000","last_undegraded":"2024-02-05T06:56:46.357535+0000","last_fullsized":"2024-02-05T06:56:46.357535+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'21","last_scrub_stamp":"2024-02-05T06:56:37.004902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:56:37.004902+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:03.581406+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":991,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.351704+0000","last_change":"2024-02-05T06:55:56.582636+0000","last_active":"2024-02-05T06:56:46.351704+0000","last_peered":"2024-02-05T06:56:46.351704+0000","last_clean":"2024-02-05T06:56:46.351704+0000","last_became_active":"2024-02-05T06:40:27.088096+0000","last_became_peered":"2024-02-05T06:40:27.088096+0000","last_unstale":"2024-02-05T06:56:46.351704+0000","last_undegraded":"2024-02-05T06:56:46.351704+0000","last_fullsized":"2024-02-05T06:56:46.351704+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:56.582541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:53:43.562266+0000","last_clean_scrub_stamp":"2024-02-05T06:55:56.582541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:18.592287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"208'10","reported_seq":1264,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:54.868113+0000","last_change":"2024-02-05T06:55:55.469497+0000","last_active":"2024-02-05T06:56:54.868113+0000","last_peered":"2024-02-05T06:56:54.868113+0000","last_clean":"2024-02-05T06:56:54.868113+0000","last_became_active":"2024-02-05T06:40:27.096584+0000","last_became_peered":"2024-02-05T06:40:27.096584+0000","last_unstale":"2024-02-05T06:56:54.868113+0000","last_undegraded":"2024-02-05T06:56:54.868113+0000","last_fullsized":"2024-02-05T06:56:54.868113+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-02-05T06:55:55.469423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:55:55.469423+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:24.678194+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":487,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.356454+0000","last_change":"2024-02-05T06:56:13.794946+0000","last_active":"2024-02-05T06:56:46.356454+0000","last_peered":"2024-02-05T06:56:46.356454+0000","last_clean":"2024-02-05T06:56:46.356454+0000","last_became_active":"2024-02-05T06:55:53.776494+0000","last_became_peered":"2024-02-05T06:55:53.776494+0000","last_unstale":"2024-02-05T06:56:46.356454+0000","last_undegraded":"2024-02-05T06:56:46.356454+0000","last_fullsized":"2024-02-05T06:56:46.356454+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:13.794838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:41:38.559780+0000","last_clean_scrub_stamp":"2024-02-05T06:56:13.794838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:27.534724+0000","scrub_duration":158,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"83'128","reported_seq":623,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.351237+0000","last_change":"2024-02-05T06:56:14.365404+0000","last_active":"2024-02-05T06:56:46.351237+0000","last_peered":"2024-02-05T06:56:46.351237+0000","last_clean":"2024-02-05T06:56:46.351237+0000","last_became_active":"2024-02-05T06:51:59.300112+0000","last_became_peered":"2024-02-05T06:51:59.300112+0000","last_unstale":"2024-02-05T06:56:46.351237+0000","last_undegraded":"2024-02-05T06:56:46.351237+0000","last_fullsized":"2024-02-05T06:56:46.351237+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:56:14.365284+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:37.282442+0000","last_clean_scrub_stamp":"2024-02-05T06:56:14.365284+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:42.568686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"499'35","reported_seq":1062,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.850279+0000","last_change":"2024-02-05T06:55:57.523940+0000","last_active":"2024-02-05T06:56:46.850279+0000","last_peered":"2024-02-05T06:56:46.850279+0000","last_clean":"2024-02-05T06:56:46.850279+0000","last_became_active":"2024-02-05T06:40:27.093418+0000","last_became_peered":"2024-02-05T06:40:27.093418+0000","last_unstale":"2024-02-05T06:56:46.850279+0000","last_undegraded":"2024-02-05T06:56:46.850279+0000","last_fullsized":"2024-02-05T06:56:46.850279+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"499'35","last_scrub_stamp":"2024-02-05T06:55:57.523847+0000","last_deep_scrub":"499'35","last_deep_scrub_stamp":"2024-02-05T06:55:57.523847+0000","last_clean_scrub_stamp":"2024-02-05T06:55:57.523847+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:16.202240+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":679,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.358612+0000","last_change":"2024-02-05T06:55:39.823122+0000","last_active":"2024-02-05T06:56:46.358612+0000","last_peered":"2024-02-05T06:56:46.358612+0000","last_clean":"2024-02-05T06:56:46.358612+0000","last_became_active":"2024-02-05T06:40:27.098418+0000","last_became_peered":"2024-02-05T06:40:27.098418+0000","last_unstale":"2024-02-05T06:56:46.358612+0000","last_undegraded":"2024-02-05T06:56:46.358612+0000","last_fullsized":"2024-02-05T06:56:46.358612+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:39.823016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:55:39.823016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:53.345711+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"336'11","reported_seq":782,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:49.867107+0000","last_change":"2024-02-05T06:56:29.838510+0000","last_active":"2024-02-05T06:56:49.867107+0000","last_peered":"2024-02-05T06:56:49.867107+0000","last_clean":"2024-02-05T06:56:49.867107+0000","last_became_active":"2024-02-05T06:41:43.418753+0000","last_became_peered":"2024-02-05T06:41:43.418753+0000","last_unstale":"2024-02-05T06:56:49.867107+0000","last_undegraded":"2024-02-05T06:56:49.867107+0000","last_fullsized":"2024-02-05T06:56:49.867107+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'11","last_scrub_stamp":"2024-02-05T06:56:29.838382+0000","last_deep_scrub":"336'11","last_deep_scrub_stamp":"2024-02-05T06:54:57.729829+0000","last_clean_scrub_stamp":"2024-02-05T06:56:29.838382+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:51.634238+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":970,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.352959+0000","last_change":"2024-02-05T06:56:16.427380+0000","last_active":"2024-02-05T06:56:46.352959+0000","last_peered":"2024-02-05T06:56:46.352959+0000","last_clean":"2024-02-05T06:56:46.352959+0000","last_became_active":"2024-02-05T06:40:27.096132+0000","last_became_peered":"2024-02-05T06:40:27.096132+0000","last_unstale":"2024-02-05T06:56:46.352959+0000","last_undegraded":"2024-02-05T06:56:46.352959+0000","last_fullsized":"2024-02-05T06:56:46.352959+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:16.427279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:56:16.427279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:36.550984+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"465'32","reported_seq":794,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.355147+0000","last_change":"2024-02-05T06:56:27.429784+0000","last_active":"2024-02-05T06:56:46.355147+0000","last_peered":"2024-02-05T06:56:46.355147+0000","last_clean":"2024-02-05T06:56:46.355147+0000","last_became_active":"2024-02-05T06:55:53.782604+0000","last_became_peered":"2024-02-05T06:55:53.782604+0000","last_unstale":"2024-02-05T06:56:46.355147+0000","last_undegraded":"2024-02-05T06:56:46.355147+0000","last_fullsized":"2024-02-05T06:56:46.355147+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'32","last_scrub_stamp":"2024-02-05T06:56:27.429673+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:35:02.589229+0000","last_clean_scrub_stamp":"2024-02-05T06:56:27.429673+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:33.564171+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":479,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.354941+0000","last_change":"2024-02-05T06:56:41.348089+0000","last_active":"2024-02-05T06:56:46.354941+0000","last_peered":"2024-02-05T06:56:46.354941+0000","last_clean":"2024-02-05T06:56:46.354941+0000","last_became_active":"2024-02-05T06:55:53.781768+0000","last_became_peered":"2024-02-05T06:55:53.781768+0000","last_unstale":"2024-02-05T06:56:46.354941+0000","last_undegraded":"2024-02-05T06:56:46.354941+0000","last_fullsized":"2024-02-05T06:56:46.354941+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:41.347992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:56:41.347992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:06.192865+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":996,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.351716+0000","last_change":"2024-02-05T06:55:57.928797+0000","last_active":"2024-02-05T06:56:46.351716+0000","last_peered":"2024-02-05T06:56:46.351716+0000","last_clean":"2024-02-05T06:56:46.351716+0000","last_became_active":"2024-02-05T06:41:43.421433+0000","last_became_peered":"2024-02-05T06:41:43.421433+0000","last_unstale":"2024-02-05T06:56:46.351716+0000","last_undegraded":"2024-02-05T06:56:46.351716+0000","last_fullsized":"2024-02-05T06:56:46.351716+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:57.928712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:55:57.928712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:26.367767+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":962,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.850384+0000","last_change":"2024-02-05T06:56:08.621910+0000","last_active":"2024-02-05T06:56:46.850384+0000","last_peered":"2024-02-05T06:56:46.850384+0000","last_clean":"2024-02-05T06:56:46.850384+0000","last_became_active":"2024-02-05T06:41:43.417155+0000","last_became_peered":"2024-02-05T06:41:43.417155+0000","last_unstale":"2024-02-05T06:56:46.850384+0000","last_undegraded":"2024-02-05T06:56:46.850384+0000","last_fullsized":"2024-02-05T06:56:46.850384+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:08.621797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:56:08.621797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:25.140001+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"465'21","reported_seq":772,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:54.223995+0000","last_change":"2024-02-05T06:56:54.224021+0000","last_active":"2024-02-05T06:56:54.223995+0000","last_peered":"2024-02-05T06:56:54.223995+0000","last_clean":"2024-02-05T06:56:54.223995+0000","last_became_active":"2024-02-05T06:39:13.244124+0000","last_became_peered":"2024-02-05T06:39:13.244124+0000","last_unstale":"2024-02-05T06:56:54.223995+0000","last_undegraded":"2024-02-05T06:56:54.223995+0000","last_fullsized":"2024-02-05T06:56:54.223995+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'21","last_scrub_stamp":"2024-02-05T06:56:54.223932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:56:54.223932+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:06.048010+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":678,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.358805+0000","last_change":"2024-02-05T06:55:43.787798+0000","last_active":"2024-02-05T06:56:46.358805+0000","last_peered":"2024-02-05T06:56:46.358805+0000","last_clean":"2024-02-05T06:56:46.358805+0000","last_became_active":"2024-02-05T06:40:27.077701+0000","last_became_peered":"2024-02-05T06:40:27.077701+0000","last_unstale":"2024-02-05T06:56:46.358805+0000","last_undegraded":"2024-02-05T06:56:46.358805+0000","last_fullsized":"2024-02-05T06:56:46.358805+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:43.787663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:33.978941+0000","last_clean_scrub_stamp":"2024-02-05T06:55:43.787663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:13.399182+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":954,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.849947+0000","last_change":"2024-02-05T06:56:24.785042+0000","last_active":"2024-02-05T06:56:46.849947+0000","last_peered":"2024-02-05T06:56:46.849947+0000","last_clean":"2024-02-05T06:56:46.849947+0000","last_became_active":"2024-02-05T06:33:33.593901+0000","last_became_peered":"2024-02-05T06:33:33.593901+0000","last_unstale":"2024-02-05T06:56:46.849947+0000","last_undegraded":"2024-02-05T06:56:46.849947+0000","last_fullsized":"2024-02-05T06:56:46.849947+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:24.784899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:30.567403+0000","last_clean_scrub_stamp":"2024-02-05T06:56:24.784899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:36.743655+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"336'2","reported_seq":989,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.351288+0000","last_change":"2024-02-05T06:56:33.267158+0000","last_active":"2024-02-05T06:56:46.351288+0000","last_peered":"2024-02-05T06:56:46.351288+0000","last_clean":"2024-02-05T06:56:46.351288+0000","last_became_active":"2024-02-05T06:41:43.409761+0000","last_became_peered":"2024-02-05T06:41:43.409761+0000","last_unstale":"2024-02-05T06:56:46.351288+0000","last_undegraded":"2024-02-05T06:56:46.351288+0000","last_fullsized":"2024-02-05T06:56:46.351288+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'2","last_scrub_stamp":"2024-02-05T06:56:33.267001+0000","last_deep_scrub":"336'2","last_deep_scrub_stamp":"2024-02-05T06:46:53.500155+0000","last_clean_scrub_stamp":"2024-02-05T06:56:33.267001+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:39.312822+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"83'128","reported_seq":1195,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.872634+0000","last_change":"2024-02-05T06:56:28.621133+0000","last_active":"2024-02-05T06:56:46.872634+0000","last_peered":"2024-02-05T06:56:46.872634+0000","last_clean":"2024-02-05T06:56:46.872634+0000","last_became_active":"2024-02-05T06:51:59.292766+0000","last_became_peered":"2024-02-05T06:51:59.292766+0000","last_unstale":"2024-02-05T06:56:46.872634+0000","last_undegraded":"2024-02-05T06:56:46.872634+0000","last_fullsized":"2024-02-05T06:56:46.872634+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:56:28.621052+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:55:18.345171+0000","last_clean_scrub_stamp":"2024-02-05T06:56:28.621052+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:50.591989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"465'32","reported_seq":1060,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.850591+0000","last_change":"2024-02-05T06:56:27.854385+0000","last_active":"2024-02-05T06:56:46.850591+0000","last_peered":"2024-02-05T06:56:46.850591+0000","last_clean":"2024-02-05T06:56:46.850591+0000","last_became_active":"2024-02-05T06:40:27.080906+0000","last_became_peered":"2024-02-05T06:40:27.080906+0000","last_unstale":"2024-02-05T06:56:46.850591+0000","last_undegraded":"2024-02-05T06:56:46.850591+0000","last_fullsized":"2024-02-05T06:56:46.850591+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'32","last_scrub_stamp":"2024-02-05T06:56:27.854239+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-02-05T06:35:22.938675+0000","last_clean_scrub_stamp":"2024-02-05T06:56:27.854239+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:49.114925+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":594,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.357873+0000","last_change":"2024-02-05T06:56:18.833609+0000","last_active":"2024-02-05T06:56:46.357873+0000","last_peered":"2024-02-05T06:56:46.357873+0000","last_clean":"2024-02-05T06:56:46.357873+0000","last_became_active":"2024-02-05T06:41:43.412654+0000","last_became_peered":"2024-02-05T06:41:43.412654+0000","last_unstale":"2024-02-05T06:56:46.357873+0000","last_undegraded":"2024-02-05T06:56:46.357873+0000","last_fullsized":"2024-02-05T06:56:46.357873+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:18.833517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:56:18.833517+0000","last_clean_scrub_stamp":"2024-02-05T06:56:18.833517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:41.974670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":687,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:48.012375+0000","last_change":"2024-02-05T06:56:48.012400+0000","last_active":"2024-02-05T06:56:48.012375+0000","last_peered":"2024-02-05T06:56:48.012375+0000","last_clean":"2024-02-05T06:56:48.012375+0000","last_became_active":"2024-02-05T06:40:27.062975+0000","last_became_peered":"2024-02-05T06:40:27.062975+0000","last_unstale":"2024-02-05T06:56:48.012375+0000","last_undegraded":"2024-02-05T06:56:48.012375+0000","last_fullsized":"2024-02-05T06:56:48.012375+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:48.012302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:48:34.628182+0000","last_clean_scrub_stamp":"2024-02-05T06:56:48.012302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:01.312488+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":744,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.352418+0000","last_change":"2024-02-05T06:56:32.970339+0000","last_active":"2024-02-05T06:56:46.352418+0000","last_peered":"2024-02-05T06:56:46.352418+0000","last_clean":"2024-02-05T06:56:46.352418+0000","last_became_active":"2024-02-05T06:39:13.245684+0000","last_became_peered":"2024-02-05T06:39:13.245684+0000","last_unstale":"2024-02-05T06:56:46.352418+0000","last_undegraded":"2024-02-05T06:56:46.352418+0000","last_fullsized":"2024-02-05T06:56:46.352418+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:32.970223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:56:32.970223+0000","last_clean_scrub_stamp":"2024-02-05T06:56:32.970223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:41.045865+0000","scrub_duration":205,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"465'12","reported_seq":976,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:52.882417+0000","last_change":"2024-02-05T06:56:52.882417+0000","last_active":"2024-02-05T06:56:52.882417+0000","last_peered":"2024-02-05T06:56:52.882417+0000","last_clean":"2024-02-05T06:56:52.882417+0000","last_became_active":"2024-02-05T06:33:31.573410+0000","last_became_peered":"2024-02-05T06:33:31.573410+0000","last_unstale":"2024-02-05T06:56:52.882417+0000","last_undegraded":"2024-02-05T06:56:52.882417+0000","last_fullsized":"2024-02-05T06:56:52.882417+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'12","last_scrub_stamp":"2024-02-05T06:56:52.882323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:56:52.882323+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:00.891218+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":982,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.352547+0000","last_change":"2024-02-05T06:55:57.629987+0000","last_active":"2024-02-05T06:56:46.352547+0000","last_peered":"2024-02-05T06:56:46.352547+0000","last_clean":"2024-02-05T06:56:46.352547+0000","last_became_active":"2024-02-05T06:41:43.424459+0000","last_became_peered":"2024-02-05T06:41:43.424459+0000","last_unstale":"2024-02-05T06:56:46.352547+0000","last_undegraded":"2024-02-05T06:56:46.352547+0000","last_fullsized":"2024-02-05T06:56:46.352547+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:57.629857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:55:57.629857+0000","last_clean_scrub_stamp":"2024-02-05T06:55:57.629857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:18.238975+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1024,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.852373+0000","last_change":"2024-02-05T06:56:27.239627+0000","last_active":"2024-02-05T06:56:46.852373+0000","last_peered":"2024-02-05T06:56:46.852373+0000","last_clean":"2024-02-05T06:56:46.852373+0000","last_became_active":"2024-02-05T06:40:27.060112+0000","last_became_peered":"2024-02-05T06:40:27.060112+0000","last_unstale":"2024-02-05T06:56:46.852373+0000","last_undegraded":"2024-02-05T06:56:46.852373+0000","last_fullsized":"2024-02-05T06:56:46.852373+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:27.239527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:48:33.848718+0000","last_clean_scrub_stamp":"2024-02-05T06:56:27.239527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:33.141945+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":693,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:54.067359+0000","last_change":"2024-02-05T06:56:54.067382+0000","last_active":"2024-02-05T06:56:54.067359+0000","last_peered":"2024-02-05T06:56:54.067359+0000","last_clean":"2024-02-05T06:56:54.067359+0000","last_became_active":"2024-02-05T06:44:54.625067+0000","last_became_peered":"2024-02-05T06:44:54.625067+0000","last_unstale":"2024-02-05T06:56:54.067359+0000","last_undegraded":"2024-02-05T06:56:54.067359+0000","last_fullsized":"2024-02-05T06:56:54.067359+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:54.067271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:48.512673+0000","last_clean_scrub_stamp":"2024-02-05T06:56:54.067271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:56.974525+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"465'32","reported_seq":1084,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.853088+0000","last_change":"2024-02-05T06:55:54.152110+0000","last_active":"2024-02-05T06:56:46.853088+0000","last_peered":"2024-02-05T06:56:46.853088+0000","last_clean":"2024-02-05T06:56:46.853088+0000","last_became_active":"2024-02-05T06:41:43.419769+0000","last_became_peered":"2024-02-05T06:41:43.419769+0000","last_unstale":"2024-02-05T06:56:46.853088+0000","last_undegraded":"2024-02-05T06:56:46.853088+0000","last_fullsized":"2024-02-05T06:56:46.853088+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'32","last_scrub_stamp":"2024-02-05T06:55:54.151969+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:43:03.221466+0000","last_clean_scrub_stamp":"2024-02-05T06:55:54.151969+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:54.770452+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":594,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.356595+0000","last_change":"2024-02-05T06:56:39.896425+0000","last_active":"2024-02-05T06:56:46.356595+0000","last_peered":"2024-02-05T06:56:46.356595+0000","last_clean":"2024-02-05T06:56:46.356595+0000","last_became_active":"2024-02-05T06:41:43.411043+0000","last_became_peered":"2024-02-05T06:41:43.411043+0000","last_unstale":"2024-02-05T06:56:46.356595+0000","last_undegraded":"2024-02-05T06:56:46.356595+0000","last_fullsized":"2024-02-05T06:56:46.356595+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:39.896331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:56:39.896331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:42.545286+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":878,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.850056+0000","last_change":"2024-02-05T06:55:58.538469+0000","last_active":"2024-02-05T06:56:46.850056+0000","last_peered":"2024-02-05T06:56:46.850056+0000","last_clean":"2024-02-05T06:56:46.850056+0000","last_became_active":"2024-02-05T06:40:27.097169+0000","last_became_peered":"2024-02-05T06:40:27.097169+0000","last_unstale":"2024-02-05T06:56:46.850056+0000","last_undegraded":"2024-02-05T06:56:46.850056+0000","last_fullsized":"2024-02-05T06:56:46.850056+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:58.538370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:55:58.538370+0000","last_clean_scrub_stamp":"2024-02-05T06:55:58.538370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:13.347458+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":997,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.351878+0000","last_change":"2024-02-05T06:55:59.073655+0000","last_active":"2024-02-05T06:56:46.351878+0000","last_peered":"2024-02-05T06:56:46.351878+0000","last_clean":"2024-02-05T06:56:46.351878+0000","last_became_active":"2024-02-05T06:41:43.414638+0000","last_became_peered":"2024-02-05T06:41:43.414638+0000","last_unstale":"2024-02-05T06:56:46.351878+0000","last_undegraded":"2024-02-05T06:56:46.351878+0000","last_fullsized":"2024-02-05T06:56:46.351878+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:59.073554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:55:59.073554+0000","last_clean_scrub_stamp":"2024-02-05T06:55:59.073554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:12.525337+0000","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"465'35","reported_seq":1062,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.352445+0000","last_change":"2024-02-05T06:55:36.324828+0000","last_active":"2024-02-05T06:56:46.352445+0000","last_peered":"2024-02-05T06:56:46.352445+0000","last_clean":"2024-02-05T06:56:46.352445+0000","last_became_active":"2024-02-05T06:40:27.100610+0000","last_became_peered":"2024-02-05T06:40:27.100610+0000","last_unstale":"2024-02-05T06:56:46.352445+0000","last_undegraded":"2024-02-05T06:56:46.352445+0000","last_fullsized":"2024-02-05T06:56:46.352445+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'35","last_scrub_stamp":"2024-02-05T06:55:36.324742+0000","last_deep_scrub":"465'35","last_deep_scrub_stamp":"2024-02-05T06:53:54.509362+0000","last_clean_scrub_stamp":"2024-02-05T06:55:36.324742+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:57.356490+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":1014,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.351822+0000","last_change":"2024-02-05T06:56:26.916792+0000","last_active":"2024-02-05T06:56:46.351822+0000","last_peered":"2024-02-05T06:56:46.351822+0000","last_clean":"2024-02-05T06:56:46.351822+0000","last_became_active":"2024-02-05T06:36:54.958414+0000","last_became_peered":"2024-02-05T06:36:54.958414+0000","last_unstale":"2024-02-05T06:56:46.351822+0000","last_undegraded":"2024-02-05T06:56:46.351822+0000","last_fullsized":"2024-02-05T06:56:46.351822+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:56:26.916703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:56:26.916703+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:53.830736+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"314'13","reported_seq":1332,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:49.867103+0000","last_change":"2024-02-05T06:55:51.185124+0000","last_active":"2024-02-05T06:56:49.867103+0000","last_peered":"2024-02-05T06:56:49.867103+0000","last_clean":"2024-02-05T06:56:49.867103+0000","last_became_active":"2024-02-05T06:40:27.063399+0000","last_became_peered":"2024-02-05T06:40:27.063399+0000","last_unstale":"2024-02-05T06:56:49.867103+0000","last_undegraded":"2024-02-05T06:56:49.867103+0000","last_fullsized":"2024-02-05T06:56:49.867103+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"314'13","last_scrub_stamp":"2024-02-05T06:55:51.185019+0000","last_deep_scrub":"314'13","last_deep_scrub_stamp":"2024-02-05T06:44:53.670923+0000","last_clean_scrub_stamp":"2024-02-05T06:55:51.185019+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:19.687971+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"325'2","reported_seq":693,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.358392+0000","last_change":"2024-02-05T06:56:31.025946+0000","last_active":"2024-02-05T06:56:46.358392+0000","last_peered":"2024-02-05T06:56:46.358392+0000","last_clean":"2024-02-05T06:56:46.358392+0000","last_became_active":"2024-02-05T06:41:43.418640+0000","last_became_peered":"2024-02-05T06:41:43.418640+0000","last_unstale":"2024-02-05T06:56:46.358392+0000","last_undegraded":"2024-02-05T06:56:46.358392+0000","last_fullsized":"2024-02-05T06:56:46.358392+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'2","last_scrub_stamp":"2024-02-05T06:56:31.025780+0000","last_deep_scrub":"325'2","last_deep_scrub_stamp":"2024-02-05T06:45:46.242415+0000","last_clean_scrub_stamp":"2024-02-05T06:56:31.025780+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:32.257939+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"465'29","reported_seq":1057,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.849904+0000","last_change":"2024-02-05T06:55:49.420800+0000","last_active":"2024-02-05T06:56:46.849904+0000","last_peered":"2024-02-05T06:56:46.849904+0000","last_clean":"2024-02-05T06:56:46.849904+0000","last_became_active":"2024-02-05T06:36:54.954880+0000","last_became_peered":"2024-02-05T06:36:54.954880+0000","last_unstale":"2024-02-05T06:56:46.849904+0000","last_undegraded":"2024-02-05T06:56:46.849904+0000","last_fullsized":"2024-02-05T06:56:46.849904+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'29","last_scrub_stamp":"2024-02-05T06:55:49.420713+0000","last_deep_scrub":"445'23","last_deep_scrub_stamp":"2024-02-05T06:52:37.440858+0000","last_clean_scrub_stamp":"2024-02-05T06:55:49.420713+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:04.491402+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":459,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.354482+0000","last_change":"2024-02-05T06:55:53.778513+0000","last_active":"2024-02-05T06:56:46.354482+0000","last_peered":"2024-02-05T06:56:46.354482+0000","last_clean":"2024-02-05T06:56:46.354482+0000","last_became_active":"2024-02-05T06:55:53.778204+0000","last_became_peered":"2024-02-05T06:55:53.778204+0000","last_unstale":"2024-02-05T06:56:46.354482+0000","last_undegraded":"2024-02-05T06:56:46.354482+0000","last_fullsized":"2024-02-05T06:56:46.354482+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:42.887647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:55:42.887647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:07.990322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":766,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.852733+0000","last_change":"2024-02-05T06:56:39.329168+0000","last_active":"2024-02-05T06:56:46.852733+0000","last_peered":"2024-02-05T06:56:46.852733+0000","last_clean":"2024-02-05T06:56:46.852733+0000","last_became_active":"2024-02-05T06:39:13.248618+0000","last_became_peered":"2024-02-05T06:39:13.248618+0000","last_unstale":"2024-02-05T06:56:46.852733+0000","last_undegraded":"2024-02-05T06:56:46.852733+0000","last_fullsized":"2024-02-05T06:56:46.852733+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:39.329060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:34.551395+0000","last_clean_scrub_stamp":"2024-02-05T06:56:39.329060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:59.994846+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":963,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.850488+0000","last_change":"2024-02-05T06:56:22.064149+0000","last_active":"2024-02-05T06:56:46.850488+0000","last_peered":"2024-02-05T06:56:46.850488+0000","last_clean":"2024-02-05T06:56:46.850488+0000","last_became_active":"2024-02-05T06:41:43.411442+0000","last_became_peered":"2024-02-05T06:41:43.411442+0000","last_unstale":"2024-02-05T06:56:46.850488+0000","last_undegraded":"2024-02-05T06:56:46.850488+0000","last_fullsized":"2024-02-05T06:56:46.850488+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:22.063996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:56:22.063996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:41.197788+0000","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"465'46","reported_seq":1079,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.872966+0000","last_change":"2024-02-05T06:56:23.584296+0000","last_active":"2024-02-05T06:56:46.872966+0000","last_peered":"2024-02-05T06:56:46.872966+0000","last_clean":"2024-02-05T06:56:46.872966+0000","last_became_active":"2024-02-05T06:33:31.569763+0000","last_became_peered":"2024-02-05T06:33:31.569763+0000","last_unstale":"2024-02-05T06:56:46.872966+0000","last_undegraded":"2024-02-05T06:56:46.872966+0000","last_fullsized":"2024-02-05T06:56:46.872966+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'46","last_scrub_stamp":"2024-02-05T06:56:23.584185+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:34:35.234636+0000","last_clean_scrub_stamp":"2024-02-05T06:56:23.584185+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:45.464217+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"281'2","reported_seq":1025,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.353191+0000","last_change":"2024-02-05T06:56:46.134964+0000","last_active":"2024-02-05T06:56:46.353191+0000","last_peered":"2024-02-05T06:56:46.353191+0000","last_clean":"2024-02-05T06:56:46.353191+0000","last_became_active":"2024-02-05T06:40:27.073360+0000","last_became_peered":"2024-02-05T06:40:27.073360+0000","last_unstale":"2024-02-05T06:56:46.353191+0000","last_undegraded":"2024-02-05T06:56:46.353191+0000","last_fullsized":"2024-02-05T06:56:46.353191+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'2","last_scrub_stamp":"2024-02-05T06:56:46.134886+0000","last_deep_scrub":"281'2","last_deep_scrub_stamp":"2024-02-05T06:47:50.442159+0000","last_clean_scrub_stamp":"2024-02-05T06:56:46.134886+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:49.357469+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":981,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.351689+0000","last_change":"2024-02-05T06:55:56.885657+0000","last_active":"2024-02-05T06:56:46.351689+0000","last_peered":"2024-02-05T06:56:46.351689+0000","last_clean":"2024-02-05T06:56:46.351689+0000","last_became_active":"2024-02-05T06:36:54.958069+0000","last_became_peered":"2024-02-05T06:36:54.958069+0000","last_unstale":"2024-02-05T06:56:46.351689+0000","last_undegraded":"2024-02-05T06:56:46.351689+0000","last_fullsized":"2024-02-05T06:56:46.351689+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:56.885555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:55:56.885555+0000","last_clean_scrub_stamp":"2024-02-05T06:55:56.885555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:18.760179+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":488,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.356172+0000","last_change":"2024-02-05T06:56:22.520719+0000","last_active":"2024-02-05T06:56:46.356172+0000","last_peered":"2024-02-05T06:56:46.356172+0000","last_clean":"2024-02-05T06:56:46.356172+0000","last_became_active":"2024-02-05T06:55:53.782074+0000","last_became_peered":"2024-02-05T06:55:53.782074+0000","last_unstale":"2024-02-05T06:56:46.356172+0000","last_undegraded":"2024-02-05T06:56:46.356172+0000","last_fullsized":"2024-02-05T06:56:46.356172+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:22.520590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:56:22.520590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:28.502171+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"465'9","reported_seq":762,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.354921+0000","last_change":"2024-02-05T06:56:03.591900+0000","last_active":"2024-02-05T06:56:46.354921+0000","last_peered":"2024-02-05T06:56:46.354921+0000","last_clean":"2024-02-05T06:56:46.354921+0000","last_became_active":"2024-02-05T06:55:53.777564+0000","last_became_peered":"2024-02-05T06:55:53.777564+0000","last_unstale":"2024-02-05T06:56:46.354921+0000","last_undegraded":"2024-02-05T06:56:46.354921+0000","last_fullsized":"2024-02-05T06:56:46.354921+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'9","last_scrub_stamp":"2024-02-05T06:56:03.591805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:56:03.591805+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:13.128737+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":594,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.357470+0000","last_change":"2024-02-05T06:56:33.939933+0000","last_active":"2024-02-05T06:56:46.357470+0000","last_peered":"2024-02-05T06:56:46.357470+0000","last_clean":"2024-02-05T06:56:46.357470+0000","last_became_active":"2024-02-05T06:41:43.408428+0000","last_became_peered":"2024-02-05T06:41:43.408428+0000","last_unstale":"2024-02-05T06:56:46.357470+0000","last_undegraded":"2024-02-05T06:56:46.357470+0000","last_fullsized":"2024-02-05T06:56:46.357470+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:33.939832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:43.828638+0000","last_clean_scrub_stamp":"2024-02-05T06:56:33.939832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:51.049407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":686,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.358207+0000","last_change":"2024-02-05T06:55:44.778965+0000","last_active":"2024-02-05T06:56:46.358207+0000","last_peered":"2024-02-05T06:56:46.358207+0000","last_clean":"2024-02-05T06:56:46.358207+0000","last_became_active":"2024-02-05T06:40:27.071758+0000","last_became_peered":"2024-02-05T06:40:27.071758+0000","last_unstale":"2024-02-05T06:56:46.358207+0000","last_undegraded":"2024-02-05T06:56:46.358207+0000","last_fullsized":"2024-02-05T06:56:46.358207+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:44.778865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:37.152741+0000","last_clean_scrub_stamp":"2024-02-05T06:55:44.778865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:49.468816+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":594,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:56.957526+0000","last_change":"2024-02-05T06:56:56.957552+0000","last_active":"2024-02-05T06:56:56.957526+0000","last_peered":"2024-02-05T06:56:56.957526+0000","last_clean":"2024-02-05T06:56:56.957526+0000","last_became_active":"2024-02-05T06:41:43.407404+0000","last_became_peered":"2024-02-05T06:41:43.407404+0000","last_unstale":"2024-02-05T06:56:56.957526+0000","last_undegraded":"2024-02-05T06:56:56.957526+0000","last_fullsized":"2024-02-05T06:56:56.957526+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:56.957431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:56:56.957431+0000","last_clean_scrub_stamp":"2024-02-05T06:56:56.957431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:24.884518+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"465'25","reported_seq":770,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:55.225140+0000","last_change":"2024-02-05T06:56:55.225180+0000","last_active":"2024-02-05T06:56:55.225140+0000","last_peered":"2024-02-05T06:56:55.225140+0000","last_clean":"2024-02-05T06:56:55.225140+0000","last_became_active":"2024-02-05T06:55:53.773462+0000","last_became_peered":"2024-02-05T06:55:53.773462+0000","last_unstale":"2024-02-05T06:56:55.225140+0000","last_undegraded":"2024-02-05T06:56:55.225140+0000","last_fullsized":"2024-02-05T06:56:55.225140+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'25","last_scrub_stamp":"2024-02-05T06:56:55.225074+0000","last_deep_scrub":"92'13","last_deep_scrub_stamp":"2024-02-05T06:34:46.542893+0000","last_clean_scrub_stamp":"2024-02-05T06:56:55.225074+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:20.780809+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":586,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.357555+0000","last_change":"2024-02-05T06:56:09.824293+0000","last_active":"2024-02-05T06:56:46.357555+0000","last_peered":"2024-02-05T06:56:46.357555+0000","last_clean":"2024-02-05T06:56:46.357555+0000","last_became_active":"2024-02-05T06:41:43.403746+0000","last_became_peered":"2024-02-05T06:41:43.403746+0000","last_unstale":"2024-02-05T06:56:46.357555+0000","last_undegraded":"2024-02-05T06:56:46.357555+0000","last_fullsized":"2024-02-05T06:56:46.357555+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:09.824199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:44.081318+0000","last_clean_scrub_stamp":"2024-02-05T06:56:09.824199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:17.829232+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":526,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.850216+0000","last_change":"2024-02-05T06:56:01.576625+0000","last_active":"2024-02-05T06:56:46.850216+0000","last_peered":"2024-02-05T06:56:46.850216+0000","last_clean":"2024-02-05T06:56:46.850216+0000","last_became_active":"2024-02-05T06:42:33.129959+0000","last_became_peered":"2024-02-05T06:42:33.129959+0000","last_unstale":"2024-02-05T06:56:46.850216+0000","last_undegraded":"2024-02-05T06:56:46.850216+0000","last_fullsized":"2024-02-05T06:56:46.850216+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:01.576520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:45:51.217900+0000","last_clean_scrub_stamp":"2024-02-05T06:56:01.576520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:21.910242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":978,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.352958+0000","last_change":"2024-02-05T06:56:34.229094+0000","last_active":"2024-02-05T06:56:46.352958+0000","last_peered":"2024-02-05T06:56:46.352958+0000","last_clean":"2024-02-05T06:56:46.352958+0000","last_became_active":"2024-02-05T06:40:27.068888+0000","last_became_peered":"2024-02-05T06:40:27.068888+0000","last_unstale":"2024-02-05T06:56:46.352958+0000","last_undegraded":"2024-02-05T06:56:46.352958+0000","last_fullsized":"2024-02-05T06:56:46.352958+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:34.228993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:44:55.125921+0000","last_clean_scrub_stamp":"2024-02-05T06:56:34.228993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:34.450816+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"465'46","reported_seq":1025,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.850546+0000","last_change":"2024-02-05T06:55:33.311337+0000","last_active":"2024-02-05T06:56:46.850546+0000","last_peered":"2024-02-05T06:56:46.850546+0000","last_clean":"2024-02-05T06:56:46.850546+0000","last_became_active":"2024-02-05T06:33:31.572544+0000","last_became_peered":"2024-02-05T06:33:31.572544+0000","last_unstale":"2024-02-05T06:56:46.850546+0000","last_undegraded":"2024-02-05T06:56:46.850546+0000","last_fullsized":"2024-02-05T06:56:46.850546+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'46","last_scrub_stamp":"2024-02-05T06:55:33.311188+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:39:35.433539+0000","last_clean_scrub_stamp":"2024-02-05T06:55:33.311188+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:02.453477+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":971,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.850155+0000","last_change":"2024-02-05T06:56:34.909979+0000","last_active":"2024-02-05T06:56:46.850155+0000","last_peered":"2024-02-05T06:56:46.850155+0000","last_clean":"2024-02-05T06:56:46.850155+0000","last_became_active":"2024-02-05T06:33:29.553864+0000","last_became_peered":"2024-02-05T06:33:29.553864+0000","last_unstale":"2024-02-05T06:56:46.850155+0000","last_undegraded":"2024-02-05T06:56:46.850155+0000","last_fullsized":"2024-02-05T06:56:46.850155+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:34.909897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:56:34.909897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:53.241517+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":475,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.354267+0000","last_change":"2024-02-05T06:56:31.319212+0000","last_active":"2024-02-05T06:56:46.354267+0000","last_peered":"2024-02-05T06:56:46.354267+0000","last_clean":"2024-02-05T06:56:46.354267+0000","last_became_active":"2024-02-05T06:55:53.782687+0000","last_became_peered":"2024-02-05T06:55:53.782687+0000","last_unstale":"2024-02-05T06:56:46.354267+0000","last_undegraded":"2024-02-05T06:56:46.354267+0000","last_fullsized":"2024-02-05T06:56:46.354267+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:31.319116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:51.388156+0000","last_clean_scrub_stamp":"2024-02-05T06:56:31.319116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:57.045141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":961,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.850000+0000","last_change":"2024-02-05T06:56:05.798253+0000","last_active":"2024-02-05T06:56:46.850000+0000","last_peered":"2024-02-05T06:56:46.850000+0000","last_clean":"2024-02-05T06:56:46.850000+0000","last_became_active":"2024-02-05T06:40:27.077485+0000","last_became_peered":"2024-02-05T06:40:27.077485+0000","last_unstale":"2024-02-05T06:56:46.850000+0000","last_undegraded":"2024-02-05T06:56:46.850000+0000","last_fullsized":"2024-02-05T06:56:46.850000+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:05.798158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:56:05.798158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:16.675686+0000","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"465'25","reported_seq":919,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.354205+0000","last_change":"2024-02-05T06:55:50.585094+0000","last_active":"2024-02-05T06:56:46.354205+0000","last_peered":"2024-02-05T06:56:46.354205+0000","last_clean":"2024-02-05T06:56:46.354205+0000","last_became_active":"2024-02-05T06:36:54.959330+0000","last_became_peered":"2024-02-05T06:36:54.959330+0000","last_unstale":"2024-02-05T06:56:46.354205+0000","last_undegraded":"2024-02-05T06:56:46.354205+0000","last_fullsized":"2024-02-05T06:56:46.354205+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'25","last_scrub_stamp":"2024-02-05T06:55:50.584974+0000","last_deep_scrub":"247'13","last_deep_scrub_stamp":"2024-02-05T06:41:42.473424+0000","last_clean_scrub_stamp":"2024-02-05T06:55:50.584974+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:18.668660+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":31,"num_read_kb":25,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":992,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:56.230428+0000","last_change":"2024-02-05T06:56:56.230448+0000","last_active":"2024-02-05T06:56:56.230428+0000","last_peered":"2024-02-05T06:56:56.230428+0000","last_clean":"2024-02-05T06:56:56.230428+0000","last_became_active":"2024-02-05T06:41:43.421315+0000","last_became_peered":"2024-02-05T06:41:43.421315+0000","last_unstale":"2024-02-05T06:56:56.230428+0000","last_undegraded":"2024-02-05T06:56:56.230428+0000","last_fullsized":"2024-02-05T06:56:56.230428+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:56.230362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:56:56.230362+0000","last_clean_scrub_stamp":"2024-02-05T06:56:56.230362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:16.510025+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":993,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.351912+0000","last_change":"2024-02-05T06:55:53.896587+0000","last_active":"2024-02-05T06:56:46.351912+0000","last_peered":"2024-02-05T06:56:46.351912+0000","last_clean":"2024-02-05T06:56:46.351912+0000","last_became_active":"2024-02-05T06:41:43.420342+0000","last_became_peered":"2024-02-05T06:41:43.420342+0000","last_unstale":"2024-02-05T06:56:46.351912+0000","last_undegraded":"2024-02-05T06:56:46.351912+0000","last_fullsized":"2024-02-05T06:56:46.351912+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:53.896510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:06.525012+0000","last_clean_scrub_stamp":"2024-02-05T06:55:53.896510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:07.570199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":485,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.354804+0000","last_change":"2024-02-05T06:56:30.406241+0000","last_active":"2024-02-05T06:56:46.354804+0000","last_peered":"2024-02-05T06:56:46.354804+0000","last_clean":"2024-02-05T06:56:46.354804+0000","last_became_active":"2024-02-05T06:55:53.777092+0000","last_became_peered":"2024-02-05T06:55:53.777092+0000","last_unstale":"2024-02-05T06:56:46.354804+0000","last_undegraded":"2024-02-05T06:56:46.354804+0000","last_fullsized":"2024-02-05T06:56:46.354804+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:30.406132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:56:30.406132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:51.776695+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":995,"num_read_kb":990,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2688,"ondisk_log_size":2688,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1527,"num_read_kb":1284,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":378,"num_bytes_recovered":8234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1104,"ondisk_log_size":1104,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103454,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":821904,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":821118,"kb_avail":92923248,"statfs":{"total":95995035648,"available":95153405952,"internally_reserved":0,"allocated":794624,"data_stored":405511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":840825290},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.96599999999999997,"15min":1.1910000000000001},"min":{"1min":0.38600000000000001,"5min":0.23799999999999999,"15min":0.21299999999999999},"max":{"1min":0.84499999999999997,"5min":4.282,"15min":32.567},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.96999999999999997,"15min":1.365},"min":{"1min":0.40000000000000002,"5min":0.249,"15min":0.222},"max":{"1min":7.4260000000000002,"5min":7.4260000000000002,"15min":41.109999999999999},"last":0.68200000000000005}]},{"osd":1,"last update":"Mon Feb 5 06:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":1.2869999999999999,"15min":1.786},"min":{"1min":0.34599999999999997,"5min":0.25600000000000001,"15min":0.25},"max":{"1min":1.024,"5min":8.2569999999999997,"15min":84.216999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.079,"15min":1.778},"min":{"1min":0.379,"5min":0.24299999999999999,"15min":0.22800000000000001},"max":{"1min":0.751,"5min":7.9450000000000003,"15min":84.126999999999995},"last":0.70999999999999996}]},{"osd":2,"last update":"Mon Feb 5 06:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":1.1060000000000001,"15min":1.284},"min":{"1min":0.41399999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.82599999999999996,"5min":8.4740000000000002,"15min":28.704999999999998},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.65700000000000003,"5min":1.1259999999999999,"15min":1.286},"min":{"1min":0.49299999999999999,"5min":0.30099999999999999,"15min":0.26800000000000002},"max":{"1min":1.0609999999999999,"5min":11.452,"15min":23.678999999999998},"last":0.61699999999999999}]},{"osd":3,"last update":"Mon Feb 5 06:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":1.702,"15min":1.845},"min":{"1min":0.39900000000000002,"5min":0.312,"15min":0.28899999999999998},"max":{"1min":7.4050000000000002,"5min":34.374000000000002,"15min":73.283000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.69399999999999995,"5min":1.655,"15min":1.9570000000000001},"min":{"1min":0.46899999999999997,"5min":0.307,"15min":0.29499999999999998},"max":{"1min":1.1120000000000001,"5min":34.267000000000003,"15min":73.323999999999998},"last":0.755}]},{"osd":4,"last update":"Mon Feb 5 06:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.78900000000000003,"15min":0.84699999999999998},"min":{"1min":0.309,"5min":0.252,"15min":0.20200000000000001},"max":{"1min":0.72699999999999998,"5min":4.8310000000000004,"15min":6.1980000000000004},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.751,"15min":0.96099999999999997},"min":{"1min":0.40200000000000002,"5min":0.26400000000000001,"15min":0.25600000000000001},"max":{"1min":0.70799999999999996,"5min":4.1870000000000003,"15min":41.270000000000003},"last":0.51600000000000001}]},{"osd":5,"last update":"Mon Feb 5 06:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.82099999999999995,"15min":1.409},"min":{"1min":0.376,"5min":0.252,"15min":0.251},"max":{"1min":1.0900000000000001,"5min":4.6200000000000001,"15min":109.482},"last":0.496},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.78000000000000003,"15min":1.4910000000000001},"min":{"1min":0.42199999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.78400000000000003,"5min":4.0039999999999996,"15min":109.56399999999999},"last":0.63200000000000001}]},{"osd":6,"last update":"Mon Feb 5 06:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.96599999999999997,"15min":1.5600000000000001},"min":{"1min":0.35299999999999998,"5min":0.317,"15min":0.28499999999999998},"max":{"1min":0.81000000000000005,"5min":4.8419999999999996,"15min":113.914},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.82899999999999996,"15min":1.478},"min":{"1min":0.36799999999999999,"5min":0.317,"15min":0.307},"max":{"1min":0.75600000000000001,"5min":4.202,"15min":113.212},"last":0.59699999999999998}]}]},{"osd":6,"up_from":49,"seq":210453397796,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":772216,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":771134,"kb_avail":92972936,"statfs":{"total":95995035648,"available":95204286464,"internally_reserved":0,"allocated":1097728,"data_stored":731742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":789641670},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:56:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.05,"15min":1.8959999999999999},"min":{"1min":0.34000000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.68799999999999994,"5min":10.039999999999999,"15min":98.918999999999997},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":1.149,"15min":2.056},"min":{"1min":0.33900000000000002,"5min":0.248,"15min":0.223},"max":{"1min":0.83199999999999996,"5min":10.23,"15min":98.775000000000006},"last":0.48299999999999998}]},{"osd":1,"last update":"Mon Feb 5 06:56:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":1.0980000000000001,"15min":1.931},"min":{"1min":0.36099999999999999,"5min":0.27900000000000003,"15min":0.249},"max":{"1min":0.74199999999999999,"5min":6.3730000000000002,"15min":98.843999999999994},"last":0.379},{"interface":"front","average":{"1min":0.56499999999999995,"5min":1.1220000000000001,"15min":1.988},"min":{"1min":0.29199999999999998,"5min":0.24099999999999999,"15min":0.23100000000000001},"max":{"1min":0.73399999999999999,"5min":6.3899999999999997,"15min":101.637},"last":0.52200000000000002}]},{"osd":2,"last update":"Mon Feb 5 06:56:41 2024","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.228,"15min":1.5620000000000001},"min":{"1min":0.379,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":19.619,"5min":19.619,"15min":74.454999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":1.169,"15min":1.5209999999999999},"min":{"1min":0.40500000000000003,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.73799999999999999,"5min":16.516999999999999,"15min":74.546000000000006},"last":0.45800000000000002}]},{"osd":3,"last update":"Mon Feb 5 06:56:41 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":1.3029999999999999,"15min":1.8029999999999999},"min":{"1min":0.40699999999999997,"5min":0.30299999999999999,"15min":0.27900000000000003},"max":{"1min":0.83499999999999996,"5min":14.648,"15min":98.888999999999996},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.623,"5min":1.323,"15min":1.7809999999999999},"min":{"1min":0.47299999999999998,"5min":0.313,"15min":0.27800000000000002},"max":{"1min":0.81200000000000006,"5min":14.6,"15min":98.969999999999999},"last":0.442}]},{"osd":4,"last update":"Mon Feb 5 06:56:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.64400000000000002,"15min":0.76200000000000001},"min":{"1min":0.27300000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.77600000000000002,"5min":5.4720000000000004,"15min":5.5940000000000003},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.68300000000000005,"15min":0.78300000000000003},"min":{"1min":0.34999999999999998,"5min":0.313,"15min":0.24399999999999999},"max":{"1min":0.76500000000000001,"5min":5.9500000000000002,"15min":5.9500000000000002},"last":0.47099999999999997}]},{"osd":5,"last update":"Mon Feb 5 06:56:41 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.71899999999999997,"15min":1.02},"min":{"1min":0.376,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.753,"5min":5.4320000000000004,"15min":36.548999999999999},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.73899999999999999,"15min":0.99399999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.28699999999999998},"max":{"1min":0.81000000000000005,"5min":10.345000000000001,"15min":36.433},"last":0.501}]},{"osd":7,"last update":"Mon Feb 5 06:55:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.71899999999999997,"15min":0.877},"min":{"1min":0.33600000000000002,"5min":0.31,"15min":0.29299999999999998},"max":{"1min":0.76200000000000001,"5min":4.1200000000000001,"15min":38.436999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.96799999999999997,"15min":1.0189999999999999},"min":{"1min":0.42999999999999999,"5min":0.29899999999999999,"15min":0.29699999999999999},"max":{"1min":0.79000000000000004,"5min":11.454000000000001,"15min":36.633000000000003},"last":0.51100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575165,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":938544,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":92806608,"statfs":{"total":95995035648,"available":95033966592,"internally_reserved":0,"allocated":827392,"data_stored":406294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":960231877},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:56:39 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":2.4350000000000001,"15min":2.0939999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20899999999999999},"max":{"1min":0.51300000000000001,"5min":93.488,"15min":93.488},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.38900000000000001,"5min":2.7839999999999998,"15min":2.2000000000000002},"min":{"1min":0.25600000000000001,"5min":0.22600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54900000000000004,"5min":93.689999999999998,"15min":93.689999999999998},"last":0.5}]},{"osd":2,"last update":"Mon Feb 5 06:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":2.7469999999999999,"15min":2.085},"min":{"1min":0.24199999999999999,"5min":0.184,"15min":0.184},"max":{"1min":0.76300000000000001,"5min":93.352999999999994,"15min":93.352999999999994},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.374,"5min":2.5310000000000001,"15min":1.831},"min":{"1min":0.26200000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.78200000000000003,"5min":91.543000000000006,"15min":91.543000000000006},"last":0.41099999999999998}]},{"osd":3,"last update":"Mon Feb 5 06:56:17 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":2.8300000000000001,"15min":2.0800000000000001},"min":{"1min":0.29999999999999999,"5min":0.25900000000000001,"15min":0.23499999999999999},"max":{"1min":0.84099999999999997,"5min":93.459999999999994,"15min":93.459999999999994},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.44,"5min":2.516,"15min":2.097},"min":{"1min":0.31,"5min":0.255,"15min":0.255},"max":{"1min":0.86499999999999999,"5min":93.406000000000006,"15min":93.406000000000006},"last":0.52100000000000002}]},{"osd":4,"last update":"Mon Feb 5 06:56:45 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":1.222,"15min":1.1699999999999999},"min":{"1min":0.46400000000000002,"5min":0.309,"15min":0.27700000000000002},"max":{"1min":0.80700000000000005,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.66400000000000003,"5min":1.502,"15min":1.4279999999999999},"min":{"1min":0.53400000000000003,"5min":0.32100000000000001,"15min":0.30599999999999999},"max":{"1min":0.84699999999999998,"5min":9.8680000000000003,"15min":10.419},"last":0.64600000000000002}]},{"osd":5,"last update":"Mon Feb 5 06:56:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":1.3129999999999999,"15min":1.5049999999999999},"min":{"1min":0.44,"5min":0.34999999999999998,"15min":0.32900000000000001},"max":{"1min":0.92200000000000004,"5min":7.3680000000000003,"15min":43.414999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.60499999999999998,"5min":1.236,"15min":1.244},"min":{"1min":0.46899999999999997,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.90300000000000002,"5min":8.1080000000000005,"15min":8.1669999999999998},"last":0.54100000000000004}]},{"osd":6,"last update":"Mon Feb 5 06:56:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":1.3979999999999999,"15min":1.3959999999999999},"min":{"1min":0.45100000000000001,"5min":0.38500000000000001,"15min":0.34799999999999998},"max":{"1min":0.88900000000000001,"5min":8.1329999999999991,"15min":23.902999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.66400000000000003,"5min":1.5169999999999999,"15min":1.6599999999999999},"min":{"1min":0.48699999999999999,"5min":0.42099999999999999,"15min":0.35599999999999998},"max":{"1min":0.91600000000000004,"5min":7.4059999999999997,"15min":43.475000000000001},"last":0.67100000000000004}]},{"osd":7,"last update":"Mon Feb 5 06:55:59 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":1.3740000000000001,"15min":1.3400000000000001},"min":{"1min":0.42399999999999999,"5min":0.36499999999999999,"15min":0.32900000000000001},"max":{"1min":1.0669999999999999,"5min":8.1600000000000001,"15min":8.1600000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":1.651,"15min":1.5229999999999999},"min":{"1min":0.42699999999999999,"5min":0.41899999999999998,"15min":0.36599999999999999},"max":{"1min":1.002,"5min":9.8230000000000004,"15min":10.481},"last":0.55900000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738691,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":798624,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":797502,"kb_avail":92946528,"statfs":{"total":95995035648,"available":95177244672,"internally_reserved":0,"allocated":1138688,"data_stored":733167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":816642501},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 06:56:33 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":1.7969999999999999,"15min":1.552},"min":{"1min":0.24399999999999999,"5min":0.188,"15min":0.182},"max":{"1min":0.58099999999999996,"5min":38.68,"15min":38.68},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.376,"5min":2.036,"15min":1.7729999999999999},"min":{"1min":0.247,"5min":0.19500000000000001,"15min":0.18099999999999999},"max":{"1min":0.54800000000000004,"5min":38.003,"15min":38.003},"last":0.499}]},{"osd":2,"last update":"Mon Feb 5 06:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":1.7450000000000001,"15min":1.6180000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.20399999999999999},"max":{"1min":0.60399999999999998,"5min":27.312000000000001,"15min":27.312000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.39600000000000002,"5min":1.75,"15min":1.512},"min":{"1min":0.22500000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.58899999999999997,"5min":27.359000000000002,"15min":27.359000000000002},"last":0.52700000000000002}]},{"osd":3,"last update":"Mon Feb 5 06:56:22 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":1.4470000000000001,"15min":1.653},"min":{"1min":0.255,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.63200000000000001,"5min":31.777999999999999,"15min":43.302},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.39900000000000002,"5min":1.911,"15min":1.835},"min":{"1min":0.26300000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.57499999999999996,"5min":31.716000000000001,"15min":43.651000000000003},"last":0.56100000000000005}]},{"osd":4,"last update":"Mon Feb 5 06:56:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":1.0249999999999999,"15min":1.2350000000000001},"min":{"1min":0.34999999999999998,"5min":0.30599999999999999,"15min":0.27200000000000002},"max":{"1min":0.83899999999999997,"5min":9.157,"15min":10.952999999999999},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.97899999999999998,"15min":1.1619999999999999},"min":{"1min":0.40999999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.86499999999999999,"5min":5.681,"15min":8.4019999999999992},"last":0.97899999999999998}]},{"osd":5,"last update":"Mon Feb 5 06:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":1.2270000000000001,"15min":1.4990000000000001},"min":{"1min":0.41099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.88,"5min":7.1239999999999997,"15min":48.841999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.77000000000000002,"5min":1.0940000000000001,"15min":1.5309999999999999},"min":{"1min":0.43099999999999999,"5min":0.28899999999999998,"15min":0.28599999999999998},"max":{"1min":3.6299999999999999,"5min":9.1039999999999992,"15min":48.450000000000003},"last":0.94399999999999995}]},{"osd":6,"last update":"Mon Feb 5 06:56:33 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":1.0069999999999999,"15min":1.6839999999999999},"min":{"1min":0.47799999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.89900000000000002,"5min":5.71,"15min":136.76300000000001},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.71499999999999997,"5min":1.2949999999999999,"15min":1.944},"min":{"1min":0.46400000000000002,"5min":0.318,"15min":0.318},"max":{"1min":3.6459999999999999,"5min":7.0720000000000001,"15min":135.92400000000001},"last":0.80600000000000005}]},{"osd":7,"last update":"Mon Feb 5 06:55:54 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":1.208,"15min":1.383},"min":{"1min":0.46600000000000003,"5min":0.36399999999999999,"15min":0.31900000000000001},"max":{"1min":0.96499999999999997,"5min":9.25,"15min":10.989000000000001},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":1.004,"15min":1.2010000000000001},"min":{"1min":0.48499999999999999,"5min":0.378,"15min":0.34899999999999998},"max":{"1min":0.86899999999999999,"5min":5.7389999999999999,"15min":6.8470000000000004},"last":1.0189999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411640,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":699976,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":93045176,"statfs":{"total":95995035648,"available":95278260224,"internally_reserved":0,"allocated":1048576,"data_stored":668068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":715717061},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:56:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":1.306,"15min":2.2250000000000001},"min":{"1min":0.22,"5min":0.20599999999999999,"15min":0.17799999999999999},"max":{"1min":0.47199999999999998,"5min":6.5880000000000001,"15min":101.67700000000001},"last":1.05},{"interface":"front","average":{"1min":0.36599999999999999,"5min":1.25,"15min":2.1179999999999999},"min":{"1min":0.245,"5min":0.20200000000000001,"15min":0.17699999999999999},"max":{"1min":0.57499999999999996,"5min":7.4130000000000003,"15min":101.27800000000001},"last":0.82599999999999996}]},{"osd":1,"last update":"Mon Feb 5 06:56:03 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":2.113,"15min":1.861},"min":{"1min":0.24099999999999999,"5min":0.19500000000000001,"15min":0.185},"max":{"1min":0.52700000000000002,"5min":36.719000000000001,"15min":36.719000000000001},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.40000000000000002,"5min":2.0910000000000002,"15min":1.732},"min":{"1min":0.28599999999999998,"5min":0.24099999999999999,"15min":0.20200000000000001},"max":{"1min":0.55700000000000005,"5min":36.271999999999998,"15min":36.271999999999998},"last":1.135}]},{"osd":3,"last update":"Mon Feb 5 06:56:32 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":2.21,"15min":2.0459999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.20899999999999999},"max":{"1min":0.70899999999999996,"5min":38.393000000000001,"15min":84.129000000000005},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.41899999999999998,"5min":2.1070000000000002,"15min":1.9350000000000001},"min":{"1min":0.28799999999999998,"5min":0.28399999999999997,"15min":0.214},"max":{"1min":0.626,"5min":35.229999999999997,"15min":84.090999999999994},"last":1.0820000000000001}]},{"osd":4,"last update":"Mon Feb 5 06:56:37 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":1.129,"15min":1.1060000000000001},"min":{"1min":0.44500000000000001,"5min":0.307,"15min":0.27400000000000002},"max":{"1min":0.84199999999999997,"5min":7.726,"15min":12.041},"last":1.103},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.97399999999999998,"15min":1.0820000000000001},"min":{"1min":0.432,"5min":0.29699999999999999,"15min":0.26300000000000001},"max":{"1min":5.2939999999999996,"5min":6.0620000000000003,"15min":15.689},"last":1.226}]},{"osd":5,"last update":"Mon Feb 5 06:56:15 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.8120000000000001,"15min":1.6080000000000001},"min":{"1min":0.42899999999999999,"5min":0.318,"15min":0.314},"max":{"1min":0.81699999999999995,"5min":78.497,"15min":78.497},"last":1.3640000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":1.889,"15min":1.661},"min":{"1min":0.46500000000000002,"5min":0.36699999999999999,"15min":0.312},"max":{"1min":0.83399999999999996,"5min":78.774000000000001,"15min":78.774000000000001},"last":1.4359999999999999}]},{"osd":6,"last update":"Mon Feb 5 06:56:37 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":1.7889999999999999,"15min":1.4490000000000001},"min":{"1min":0.48999999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.91200000000000003,"5min":53.332000000000001,"15min":53.332000000000001},"last":1.2669999999999999},{"interface":"front","average":{"1min":0.67700000000000005,"5min":1.8460000000000001,"15min":1.629},"min":{"1min":0.41799999999999998,"5min":0.33000000000000002,"15min":0.316},"max":{"1min":0.90000000000000002,"5min":53.320999999999998,"15min":53.320999999999998},"last":1.3859999999999999}]},{"osd":7,"last update":"Mon Feb 5 06:56:03 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":1.3009999999999999,"15min":1.506},"min":{"1min":0.42199999999999999,"5min":0.41699999999999998,"15min":0.34100000000000003},"max":{"1min":0.81799999999999995,"5min":9.3800000000000008,"15min":45.243000000000002},"last":1.337},{"interface":"front","average":{"1min":0.66900000000000004,"5min":1.1579999999999999,"15min":1.496},"min":{"1min":0.47899999999999998,"5min":0.39100000000000001,"15min":0.36799999999999999},"max":{"1min":0.90400000000000003,"5min":6.1870000000000003,"15min":45.171999999999997},"last":1.413}]}]},{"osd":3,"up_from":26,"seq":111669150003,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":889116,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":92856036,"statfs":{"total":95995035648,"available":95084580864,"internally_reserved":0,"allocated":1396736,"data_stored":995211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":909048263},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:56:31 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":1.103,"15min":1.605},"min":{"1min":0.23499999999999999,"5min":0.20200000000000001,"15min":0.184},"max":{"1min":0.56499999999999995,"5min":9.8849999999999998,"15min":53.656999999999996},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.379,"5min":1.2450000000000001,"15min":1.806},"min":{"1min":0.27200000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.58499999999999996,"5min":12.792999999999999,"15min":53.338000000000001},"last":0.33100000000000002}]},{"osd":1,"last update":"Mon Feb 5 06:56:31 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":1.9239999999999999,"15min":1.7230000000000001},"min":{"1min":0.33600000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.51200000000000001,"5min":24.343,"15min":24.343},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.375,"5min":1.5660000000000001,"15min":1.5960000000000001},"min":{"1min":0.23699999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.45900000000000002,"5min":23.300999999999998,"15min":23.300999999999998},"last":0.29399999999999998}]},{"osd":2,"last update":"Mon Feb 5 06:56:31 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":1.208,"15min":1.5129999999999999},"min":{"1min":0.30399999999999999,"5min":0.23499999999999999,"15min":0.23000000000000001},"max":{"1min":0.60399999999999998,"5min":6.1689999999999996,"15min":14.746},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.40200000000000002,"5min":1.218,"15min":1.367},"min":{"1min":0.29499999999999998,"5min":0.26500000000000001,"15min":0.23200000000000001},"max":{"1min":0.60999999999999999,"5min":10.220000000000001,"15min":14.734},"last":0.377}]},{"osd":4,"last update":"Mon Feb 5 06:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":1.087,"15min":1.194},"min":{"1min":0.44500000000000001,"5min":0.38100000000000001,"15min":0.29799999999999999},"max":{"1min":0.79700000000000004,"5min":10.177,"15min":16.59},"last":0.754},{"interface":"front","average":{"1min":0.59499999999999997,"5min":1.1639999999999999,"15min":1.421},"min":{"1min":0.45800000000000002,"5min":0.29299999999999998,"15min":0.28599999999999998},"max":{"1min":0.82799999999999996,"5min":12.103999999999999,"15min":19.359999999999999},"last":0.60899999999999999}]},{"osd":5,"last update":"Mon Feb 5 06:56:05 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":1.1819999999999999,"15min":1.371},"min":{"1min":0.45500000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.83499999999999996,"5min":11.458,"15min":20.257999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.66100000000000003,"5min":1.4359999999999999,"15min":1.3340000000000001},"min":{"1min":0.44900000000000001,"5min":0.36899999999999999,"15min":0.33700000000000002},"max":{"1min":0.90700000000000003,"5min":16.629999999999999,"15min":16.629999999999999},"last":0.68500000000000005}]},{"osd":6,"last update":"Mon Feb 5 06:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":1.2869999999999999,"15min":1.4730000000000001},"min":{"1min":0.45700000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.81100000000000005,"5min":12.157999999999999,"15min":16.643000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.66300000000000003,"5min":1.157,"15min":1.383},"min":{"1min":0.53300000000000003,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.84499999999999997,"5min":5.8609999999999998,"15min":12.75},"last":0.55800000000000005}]},{"osd":7,"last update":"Mon Feb 5 06:56:05 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":1.5169999999999999,"15min":1.4259999999999999},"min":{"1min":0.46600000000000003,"5min":0.41799999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":11.824,"15min":11.824},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.69999999999999996,"5min":1.3360000000000001,"15min":1.4259999999999999},"min":{"1min":0.52000000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.85699999999999998,"5min":16.611999999999998,"15min":16.611999999999998},"last":0.69299999999999995}]}]},{"osd":4,"up_from":31,"seq":133143986477,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":785816,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":785022,"kb_avail":92959336,"statfs":{"total":95995035648,"available":95190360064,"internally_reserved":0,"allocated":802816,"data_stored":405065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":803862981},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:56:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":1.1850000000000001,"15min":1.1080000000000001},"min":{"1min":0.32500000000000001,"5min":0.22900000000000001,"15min":0.20799999999999999},"max":{"1min":0.74399999999999999,"5min":19.385999999999999,"15min":19.385999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.90000000000000002,"15min":1.0800000000000001},"min":{"1min":0.30499999999999999,"5min":0.25700000000000001,"15min":0.22700000000000001},"max":{"1min":0.83799999999999997,"5min":10.130000000000001,"15min":10.913},"last":0.69099999999999995}]},{"osd":1,"last update":"Mon Feb 5 06:56:46 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":1.663,"15min":1.587},"min":{"1min":0.36799999999999999,"5min":0.26900000000000002,"15min":0.21299999999999999},"max":{"1min":0.73199999999999998,"5min":48.360999999999997,"15min":53.414000000000001},"last":0.502},{"interface":"front","average":{"1min":0.58399999999999996,"5min":1.532,"15min":1.474},"min":{"1min":0.34999999999999998,"5min":0.23999999999999999,"15min":0.23699999999999999},"max":{"1min":0.86399999999999999,"5min":48.670000000000002,"15min":53.719999999999999},"last":0.53400000000000003}]},{"osd":2,"last update":"Mon Feb 5 06:56:46 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":1.2769999999999999,"15min":1.127},"min":{"1min":0.42699999999999999,"5min":0.29399999999999998,"15min":0.26000000000000001},"max":{"1min":0.78600000000000003,"5min":19.355,"15min":19.355},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.55700000000000005,"5min":1.056,"15min":1.1799999999999999},"min":{"1min":0.313,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.84399999999999997,"5min":4.7830000000000004,"15min":18.077000000000002},"last":0.59499999999999997}]},{"osd":3,"last update":"Mon Feb 5 06:56:46 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":1.4219999999999999,"15min":1.746},"min":{"1min":0.36599999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.80500000000000005,"5min":54.228999999999999,"15min":54.228999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.55700000000000005,"5min":1.585,"15min":1.706},"min":{"1min":0.38900000000000001,"5min":0.307,"15min":0.29799999999999999},"max":{"1min":0.70399999999999996,"5min":47.670999999999999,"15min":47.670999999999999},"last":0.56699999999999995}]},{"osd":5,"last update":"Mon Feb 5 06:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.59899999999999998,"15min":0.80900000000000005},"min":{"1min":0.33300000000000002,"5min":0.25,"15min":0.249},"max":{"1min":0.68100000000000005,"5min":2.677,"15min":5.6059999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.67600000000000005,"15min":0.79800000000000004},"min":{"1min":0.29099999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.78400000000000003,"5min":4.242,"15min":7.7290000000000001},"last":0.63200000000000001}]},{"osd":6,"last update":"Mon Feb 5 06:56:46 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.74299999999999999,"15min":0.92300000000000004},"min":{"1min":0.34899999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.89600000000000002,"5min":3.5790000000000002,"15min":10.039999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.68200000000000005,"15min":0.85099999999999998},"min":{"1min":0.38800000000000001,"5min":0.312,"15min":0.29499999999999998},"max":{"1min":0.81999999999999995,"5min":3.0739999999999998,"15min":10.074},"last":0.61799999999999999}]},{"osd":7,"last update":"Mon Feb 5 06:56:51 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.66800000000000004,"15min":0.77700000000000002},"min":{"1min":0.42199999999999999,"5min":0.32200000000000001,"15min":0.27400000000000002},"max":{"1min":0.75600000000000001,"5min":3.9980000000000002,"15min":4.1369999999999996},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.745,"15min":0.84899999999999998},"min":{"1min":0.42499999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.91600000000000004,"5min":5.4809999999999999,"15min":5.4809999999999999},"last":0.751}]}]},{"osd":5,"up_from":40,"seq":171798692136,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":884616,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":883582,"kb_avail":92860536,"statfs":{"total":95995035648,"available":95089188864,"internally_reserved":0,"allocated":1048576,"data_stored":667962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":904788422},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.88300000000000001,"15min":1.5669999999999999},"min":{"1min":0.307,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.80600000000000005,"5min":5.1299999999999999,"15min":55.704999999999998},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.91800000000000004,"15min":1.587},"min":{"1min":0.39600000000000002,"5min":0.27100000000000002,"15min":0.22600000000000001},"max":{"1min":0.74199999999999999,"5min":3.9489999999999998,"15min":56.789000000000001},"last":0.46800000000000003}]},{"osd":1,"last update":"Mon Feb 5 06:56:01 2024","interfaces":[{"interface":"back","average":{"1min":1.5960000000000001,"5min":2.0419999999999998,"15min":2.8260000000000001},"min":{"1min":0.378,"5min":0.248,"15min":0.248},"max":{"1min":16.859000000000002,"5min":77.811999999999998,"15min":317.53399999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":1.72,"15min":2.75},"min":{"1min":0.371,"5min":0.253,"15min":0.22},"max":{"1min":0.70299999999999996,"5min":77.373999999999995,"15min":317.80500000000001},"last":0.33300000000000002}]},{"osd":2,"last update":"Mon Feb 5 06:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":1.3169999999999999,"15min":1.302},"min":{"1min":0.35799999999999998,"5min":0.27200000000000002,"15min":0.23000000000000001},"max":{"1min":0.72099999999999997,"5min":22.635999999999999,"15min":22.635999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":1.044,"15min":1.127},"min":{"1min":0.39000000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.71999999999999997,"5min":9.952,"15min":9.952},"last":0.51800000000000002}]},{"osd":3,"last update":"Mon Feb 5 06:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":2.0950000000000002,"15min":1.9910000000000001},"min":{"1min":0.437,"5min":0.30099999999999999,"15min":0.29899999999999999},"max":{"1min":0.78300000000000003,"5min":43.442999999999998,"15min":62.508000000000003},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":1.8999999999999999,"15min":2.04},"min":{"1min":0.42699999999999999,"5min":0.29899999999999999,"15min":0.28100000000000003},"max":{"1min":0.78400000000000003,"5min":43.545000000000002,"15min":62.460000000000001},"last":0.55800000000000005}]},{"osd":4,"last update":"Mon Feb 5 06:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.623,"15min":0.78400000000000003},"min":{"1min":0.32400000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.82599999999999996,"5min":2.9399999999999999,"15min":9.3629999999999995},"last":0.38},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.69199999999999995,"15min":0.80800000000000005},"min":{"1min":0.372,"5min":0.25600000000000001,"15min":0.252},"max":{"1min":0.76900000000000002,"5min":3.4849999999999999,"15min":9.1839999999999993},"last":0.314}]},{"osd":6,"last update":"Mon Feb 5 06:56:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":1.369,"15min":1.1559999999999999},"min":{"1min":0.36699999999999999,"5min":0.27700000000000002,"15min":0.23799999999999999},"max":{"1min":0.745,"5min":68.316999999999993,"15min":68.316999999999993},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":1.462,"15min":1.224},"min":{"1min":0.499,"5min":0.30599999999999999,"15min":0.29899999999999999},"max":{"1min":0.71499999999999997,"5min":70.352000000000004,"15min":70.352000000000004},"last":0.45100000000000001}]},{"osd":7,"last update":"Mon Feb 5 06:56:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.79300000000000004,"15min":0.98299999999999998},"min":{"1min":0.41299999999999998,"5min":0.317,"15min":0.25600000000000001},"max":{"1min":0.73699999999999999,"5min":4.4130000000000003,"15min":11.691000000000001},"last":0.432},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.70199999999999996,"15min":0.92400000000000004},"min":{"1min":0.496,"5min":0.33200000000000002,"15min":0.27100000000000002},"max":{"1min":0.80800000000000005,"5min":2.9169999999999998,"15min":6.5839999999999996},"last":0.61299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T06:57:02.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph pg dump --format=json 2024-02-05T06:57:02.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:02 smithi028 ceph-mon[29442]: from='client.46355 v1:172.21.15.28:0/1684586077' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:57:02.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:02 smithi028 ceph-mon[29442]: 3.13 scrub starts 2024-02-05T06:57:02.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:02 smithi028 ceph-mon[29442]: 3.13 scrub ok 2024-02-05T06:57:02.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:02 smithi028 ceph-mon[37984]: from='client.46355 v1:172.21.15.28:0/1684586077' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:57:02.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:02 smithi028 ceph-mon[37984]: 3.13 scrub starts 2024-02-05T06:57:02.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:02 smithi028 ceph-mon[37984]: 3.13 scrub ok 2024-02-05T06:57:02.698 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:57:02 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:57:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:57:02.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:02 smithi167 ceph-mon[32774]: from='client.46355 v1:172.21.15.28:0/1684586077' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:57:02.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:02 smithi167 ceph-mon[32774]: 3.13 scrub starts 2024-02-05T06:57:02.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:02 smithi167 ceph-mon[32774]: 3.13 scrub ok 2024-02-05T06:57:02.769 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:57:03.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:03 smithi028 ceph-mon[29442]: 6.d scrub starts 2024-02-05T06:57:03.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:03 smithi028 ceph-mon[29442]: 6.d scrub ok 2024-02-05T06:57:03.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:03 smithi028 ceph-mon[29442]: 4.16 scrub starts 2024-02-05T06:57:03.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:03 smithi028 ceph-mon[29442]: 4.16 scrub ok 2024-02-05T06:57:03.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:03 smithi028 ceph-mon[29442]: pgmap v1116: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:57:03.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:03 smithi028 ceph-mon[29442]: 3.2 scrub starts 2024-02-05T06:57:03.448 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:03 smithi028 ceph-mon[29442]: 3.2 scrub ok 2024-02-05T06:57:03.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:03 smithi028 ceph-mon[37984]: 6.d scrub starts 2024-02-05T06:57:03.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:03 smithi028 ceph-mon[37984]: 6.d scrub ok 2024-02-05T06:57:03.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:03 smithi028 ceph-mon[37984]: 4.16 scrub starts 2024-02-05T06:57:03.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:03 smithi028 ceph-mon[37984]: 4.16 scrub ok 2024-02-05T06:57:03.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:03 smithi028 ceph-mon[37984]: pgmap v1116: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:57:03.448 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:03 smithi028 ceph-mon[37984]: 3.2 scrub starts 2024-02-05T06:57:03.449 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:03 smithi028 ceph-mon[37984]: 3.2 scrub ok 2024-02-05T06:57:03.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:03 smithi167 ceph-mon[32774]: 6.d scrub starts 2024-02-05T06:57:03.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:03 smithi167 ceph-mon[32774]: 6.d scrub ok 2024-02-05T06:57:03.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:03 smithi167 ceph-mon[32774]: 4.16 scrub starts 2024-02-05T06:57:03.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:03 smithi167 ceph-mon[32774]: 4.16 scrub ok 2024-02-05T06:57:03.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:03 smithi167 ceph-mon[32774]: pgmap v1116: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-02-05T06:57:03.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:03 smithi167 ceph-mon[32774]: 3.2 scrub starts 2024-02-05T06:57:03.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:03 smithi167 ceph-mon[32774]: 3.2 scrub ok 2024-02-05T06:57:04.231 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:57:03 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug there is no tcmu-runner data available 2024-02-05T06:57:04.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:04 smithi028 ceph-mon[37984]: 6.9 scrub starts 2024-02-05T06:57:04.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:04 smithi028 ceph-mon[37984]: 6.9 scrub ok 2024-02-05T06:57:04.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:04 smithi028 ceph-mon[37984]: 5.8 scrub starts 2024-02-05T06:57:04.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:04 smithi028 ceph-mon[37984]: 5.8 scrub ok 2024-02-05T06:57:04.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:04 smithi028 ceph-mon[37984]: purged_snaps scrub starts 2024-02-05T06:57:04.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:04 smithi028 ceph-mon[37984]: purged_snaps scrub ok 2024-02-05T06:57:04.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:04 smithi028 ceph-mon[37984]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:57:04.698 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:04 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:57:04.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:04 smithi028 ceph-mon[29442]: 6.9 scrub starts 2024-02-05T06:57:04.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:04 smithi028 ceph-mon[29442]: 6.9 scrub ok 2024-02-05T06:57:04.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:04 smithi028 ceph-mon[29442]: 5.8 scrub starts 2024-02-05T06:57:04.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:04 smithi028 ceph-mon[29442]: 5.8 scrub ok 2024-02-05T06:57:04.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:04 smithi028 ceph-mon[29442]: purged_snaps scrub starts 2024-02-05T06:57:04.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:04 smithi028 ceph-mon[29442]: purged_snaps scrub ok 2024-02-05T06:57:04.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:04 smithi028 ceph-mon[29442]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:57:04.699 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:04 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:57:04.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:04 smithi167 ceph-mon[32774]: 6.9 scrub starts 2024-02-05T06:57:04.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:04 smithi167 ceph-mon[32774]: 6.9 scrub ok 2024-02-05T06:57:04.731 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:04 smithi167 ceph-mon[32774]: 5.8 scrub starts 2024-02-05T06:57:04.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:04 smithi167 ceph-mon[32774]: 5.8 scrub ok 2024-02-05T06:57:04.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:04 smithi167 ceph-mon[32774]: purged_snaps scrub starts 2024-02-05T06:57:04.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:04 smithi167 ceph-mon[32774]: purged_snaps scrub ok 2024-02-05T06:57:04.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:04 smithi167 ceph-mon[32774]: from='client.24475 v1:172.21.15.167:0/539882411' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-02-05T06:57:04.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:04 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:57:04.759 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-02-05T06:57:04.762 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-02-05T06:57:05.480 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":1118,"stamp":"2024-02-05T06:57:04.662090+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596316,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":2541,"num_read_kb":2293,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":417,"num_bytes_recovered":276079,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3912,"ondisk_log_size":3912,"up":456,"acting":456,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":456,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6590812,"kb_used_data":7964,"kb_used_omap":12,"kb_used_meta":6582771,"kb_avail":743370404,"statfs":{"total":767960285184,"available":761211293696,"internally_reserved":0,"allocated":8155136,"data_stored":5013020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":6740758065},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002933"},"pg_stats":[{"pgid":"4.c","version":"465'35","reported_seq":776,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.355055+0000","last_change":"2024-02-05T06:55:53.783967+0000","last_active":"2024-02-05T06:56:46.355055+0000","last_peered":"2024-02-05T06:56:46.355055+0000","last_clean":"2024-02-05T06:56:46.355055+0000","last_became_active":"2024-02-05T06:55:53.782172+0000","last_became_peered":"2024-02-05T06:55:53.782172+0000","last_unstale":"2024-02-05T06:56:46.355055+0000","last_undegraded":"2024-02-05T06:56:46.355055+0000","last_fullsized":"2024-02-05T06:56:46.355055+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'35","last_scrub_stamp":"2024-02-05T06:55:38.386848+0000","last_deep_scrub":"465'35","last_deep_scrub_stamp":"2024-02-05T06:55:38.386848+0000","last_clean_scrub_stamp":"2024-02-05T06:55:38.386848+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:05.861117+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":974,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.849829+0000","last_change":"2024-02-05T06:55:51.461343+0000","last_active":"2024-02-05T06:56:46.849829+0000","last_peered":"2024-02-05T06:56:46.849829+0000","last_clean":"2024-02-05T06:56:46.849829+0000","last_became_active":"2024-02-05T06:36:54.971304+0000","last_became_peered":"2024-02-05T06:36:54.971304+0000","last_unstale":"2024-02-05T06:56:46.849829+0000","last_undegraded":"2024-02-05T06:56:46.849829+0000","last_fullsized":"2024-02-05T06:56:46.849829+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:51.461255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:55:51.461255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:09.634058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"249'13","reported_seq":1332,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:54.867904+0000","last_change":"2024-02-05T06:56:27.571237+0000","last_active":"2024-02-05T06:56:54.867904+0000","last_peered":"2024-02-05T06:56:54.867904+0000","last_clean":"2024-02-05T06:56:54.867904+0000","last_became_active":"2024-02-05T06:41:43.413139+0000","last_became_peered":"2024-02-05T06:41:43.413139+0000","last_unstale":"2024-02-05T06:56:54.867904+0000","last_undegraded":"2024-02-05T06:56:54.867904+0000","last_fullsized":"2024-02-05T06:56:54.867904+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"249'13","last_scrub_stamp":"2024-02-05T06:56:27.571150+0000","last_deep_scrub":"92'9","last_deep_scrub_stamp":"2024-02-05T06:34:52.283780+0000","last_clean_scrub_stamp":"2024-02-05T06:56:27.571150+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:47.132645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":467,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.356089+0000","last_change":"2024-02-05T06:56:19.585516+0000","last_active":"2024-02-05T06:56:46.356089+0000","last_peered":"2024-02-05T06:56:46.356089+0000","last_clean":"2024-02-05T06:56:46.356089+0000","last_became_active":"2024-02-05T06:55:53.775928+0000","last_became_peered":"2024-02-05T06:55:53.775928+0000","last_unstale":"2024-02-05T06:56:46.356089+0000","last_undegraded":"2024-02-05T06:56:46.356089+0000","last_fullsized":"2024-02-05T06:56:46.356089+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:19.585408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:30.057099+0000","last_clean_scrub_stamp":"2024-02-05T06:56:19.585408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:28.152706+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":1101,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.358055+0000","last_change":"2024-02-05T06:55:58.815445+0000","last_active":"2024-02-05T06:56:46.358055+0000","last_peered":"2024-02-05T06:56:46.358055+0000","last_clean":"2024-02-05T06:56:46.358055+0000","last_became_active":"2024-02-05T06:40:30.095202+0000","last_became_peered":"2024-02-05T06:40:30.095202+0000","last_unstale":"2024-02-05T06:56:46.358055+0000","last_undegraded":"2024-02-05T06:56:46.358055+0000","last_fullsized":"2024-02-05T06:56:46.358055+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:55:58.815341+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:55:58.815341+0000","last_clean_scrub_stamp":"2024-02-05T06:55:58.815341+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:27.464644+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"465'53","reported_seq":810,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.355838+0000","last_change":"2024-02-05T06:56:17.513167+0000","last_active":"2024-02-05T06:56:46.355838+0000","last_peered":"2024-02-05T06:56:46.355838+0000","last_clean":"2024-02-05T06:56:46.355838+0000","last_became_active":"2024-02-05T06:55:53.772755+0000","last_became_peered":"2024-02-05T06:55:53.772755+0000","last_unstale":"2024-02-05T06:56:46.355838+0000","last_undegraded":"2024-02-05T06:56:46.355838+0000","last_fullsized":"2024-02-05T06:56:46.355838+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'53","last_scrub_stamp":"2024-02-05T06:56:17.513011+0000","last_deep_scrub":"95'25","last_deep_scrub_stamp":"2024-02-05T06:35:26.274944+0000","last_clean_scrub_stamp":"2024-02-05T06:56:17.513011+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:24.597959+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":58,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1010,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.852585+0000","last_change":"2024-02-05T06:56:22.237388+0000","last_active":"2024-02-05T06:56:46.852585+0000","last_peered":"2024-02-05T06:56:46.852585+0000","last_clean":"2024-02-05T06:56:46.852585+0000","last_became_active":"2024-02-05T06:39:13.240038+0000","last_became_peered":"2024-02-05T06:39:13.240038+0000","last_unstale":"2024-02-05T06:56:46.852585+0000","last_undegraded":"2024-02-05T06:56:46.852585+0000","last_fullsized":"2024-02-05T06:56:46.852585+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:22.237307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:56:22.237307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:52.041823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":990,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.353011+0000","last_change":"2024-02-05T06:56:06.821933+0000","last_active":"2024-02-05T06:56:46.353011+0000","last_peered":"2024-02-05T06:56:46.353011+0000","last_clean":"2024-02-05T06:56:46.353011+0000","last_became_active":"2024-02-05T06:36:54.958082+0000","last_became_peered":"2024-02-05T06:36:54.958082+0000","last_unstale":"2024-02-05T06:56:46.353011+0000","last_undegraded":"2024-02-05T06:56:46.353011+0000","last_fullsized":"2024-02-05T06:56:46.353011+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:06.821835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:38.072995+0000","last_clean_scrub_stamp":"2024-02-05T06:56:06.821835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:10.479382+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1012,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.873459+0000","last_change":"2024-02-05T06:56:14.793063+0000","last_active":"2024-02-05T06:56:46.873459+0000","last_peered":"2024-02-05T06:56:46.873459+0000","last_clean":"2024-02-05T06:56:46.873459+0000","last_became_active":"2024-02-05T06:36:54.958805+0000","last_became_peered":"2024-02-05T06:36:54.958805+0000","last_unstale":"2024-02-05T06:56:46.873459+0000","last_undegraded":"2024-02-05T06:56:46.873459+0000","last_fullsized":"2024-02-05T06:56:46.873459+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:14.792963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:41.833719+0000","last_clean_scrub_stamp":"2024-02-05T06:56:14.792963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:30.282801+0000","scrub_duration":188,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"83'128","reported_seq":1014,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.356776+0000","last_change":"2024-02-05T06:56:00.723471+0000","last_active":"2024-02-05T06:56:46.356776+0000","last_peered":"2024-02-05T06:56:46.356776+0000","last_clean":"2024-02-05T06:56:46.356776+0000","last_became_active":"2024-02-05T06:41:46.053324+0000","last_became_peered":"2024-02-05T06:41:46.053324+0000","last_unstale":"2024-02-05T06:56:46.356776+0000","last_undegraded":"2024-02-05T06:56:46.356776+0000","last_fullsized":"2024-02-05T06:56:46.356776+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:56:00.723365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:56:00.723365+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:17.238743+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"83'128","reported_seq":1158,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.352265+0000","last_change":"2024-02-05T06:56:04.850133+0000","last_active":"2024-02-05T06:56:46.352265+0000","last_peered":"2024-02-05T06:56:46.352265+0000","last_clean":"2024-02-05T06:56:46.352265+0000","last_became_active":"2024-02-05T06:51:59.303575+0000","last_became_peered":"2024-02-05T06:51:59.303575+0000","last_unstale":"2024-02-05T06:56:46.352265+0000","last_undegraded":"2024-02-05T06:56:46.352265+0000","last_fullsized":"2024-02-05T06:56:46.352265+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:56:04.850042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:56:04.850042+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:07.119585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"465'32","reported_seq":794,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.356133+0000","last_change":"2024-02-05T06:56:29.400573+0000","last_active":"2024-02-05T06:56:46.356133+0000","last_peered":"2024-02-05T06:56:46.356133+0000","last_clean":"2024-02-05T06:56:46.356133+0000","last_became_active":"2024-02-05T06:55:53.784136+0000","last_became_peered":"2024-02-05T06:55:53.784136+0000","last_unstale":"2024-02-05T06:56:46.356133+0000","last_undegraded":"2024-02-05T06:56:46.356133+0000","last_fullsized":"2024-02-05T06:56:46.356133+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'32","last_scrub_stamp":"2024-02-05T06:56:29.400436+0000","last_deep_scrub":"115'16","last_deep_scrub_stamp":"2024-02-05T06:36:34.793243+0000","last_clean_scrub_stamp":"2024-02-05T06:56:29.400436+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:38.167636+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":698,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.350946+0000","last_change":"2024-02-05T06:55:46.892950+0000","last_active":"2024-02-05T06:56:46.350946+0000","last_peered":"2024-02-05T06:56:46.350946+0000","last_clean":"2024-02-05T06:56:46.350946+0000","last_became_active":"2024-02-05T06:40:27.086808+0000","last_became_peered":"2024-02-05T06:40:27.086808+0000","last_unstale":"2024-02-05T06:56:46.350946+0000","last_undegraded":"2024-02-05T06:56:46.350946+0000","last_fullsized":"2024-02-05T06:56:46.350946+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:46.892872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:41.860956+0000","last_clean_scrub_stamp":"2024-02-05T06:55:46.892872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:10.157808+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":594,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.358012+0000","last_change":"2024-02-05T06:56:17.867908+0000","last_active":"2024-02-05T06:56:46.358012+0000","last_peered":"2024-02-05T06:56:46.358012+0000","last_clean":"2024-02-05T06:56:46.358012+0000","last_became_active":"2024-02-05T06:41:43.417343+0000","last_became_peered":"2024-02-05T06:41:43.417343+0000","last_unstale":"2024-02-05T06:56:46.358012+0000","last_undegraded":"2024-02-05T06:56:46.358012+0000","last_fullsized":"2024-02-05T06:56:46.358012+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:17.867792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:42:55.610276+0000","last_clean_scrub_stamp":"2024-02-05T06:56:17.867792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:47.760263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":968,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.850214+0000","last_change":"2024-02-05T06:56:18.662817+0000","last_active":"2024-02-05T06:56:46.850214+0000","last_peered":"2024-02-05T06:56:46.850214+0000","last_clean":"2024-02-05T06:56:46.850214+0000","last_became_active":"2024-02-05T06:41:43.410600+0000","last_became_peered":"2024-02-05T06:41:43.410600+0000","last_unstale":"2024-02-05T06:56:46.850214+0000","last_undegraded":"2024-02-05T06:56:46.850214+0000","last_fullsized":"2024-02-05T06:56:46.850214+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:18.662678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:58.479208+0000","last_clean_scrub_stamp":"2024-02-05T06:56:18.662678+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:32.474005+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"465'46","reported_seq":803,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.352321+0000","last_change":"2024-02-05T06:56:21.285225+0000","last_active":"2024-02-05T06:56:46.352321+0000","last_peered":"2024-02-05T06:56:46.352321+0000","last_clean":"2024-02-05T06:56:46.352321+0000","last_became_active":"2024-02-05T06:39:13.257978+0000","last_became_peered":"2024-02-05T06:39:13.257978+0000","last_unstale":"2024-02-05T06:56:46.352321+0000","last_undegraded":"2024-02-05T06:56:46.352321+0000","last_fullsized":"2024-02-05T06:56:46.352321+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'46","last_scrub_stamp":"2024-02-05T06:56:21.285115+0000","last_deep_scrub":"465'46","last_deep_scrub_stamp":"2024-02-05T06:56:21.285115+0000","last_clean_scrub_stamp":"2024-02-05T06:56:21.285115+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:23.718297+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":982,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:51.921230+0000","last_change":"2024-02-05T06:56:51.921252+0000","last_active":"2024-02-05T06:56:51.921230+0000","last_peered":"2024-02-05T06:56:51.921230+0000","last_clean":"2024-02-05T06:56:51.921230+0000","last_became_active":"2024-02-05T06:40:27.213439+0000","last_became_peered":"2024-02-05T06:40:27.213439+0000","last_unstale":"2024-02-05T06:56:51.921230+0000","last_undegraded":"2024-02-05T06:56:51.921230+0000","last_fullsized":"2024-02-05T06:56:51.921230+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:51.921171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:56:51.921171+0000","last_clean_scrub_stamp":"2024-02-05T06:56:51.921171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:18.559378+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"498'14","reported_seq":605,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:54.868112+0000","last_change":"2024-02-05T06:56:53.238731+0000","last_active":"2024-02-05T06:56:54.868112+0000","last_peered":"2024-02-05T06:56:54.868112+0000","last_clean":"2024-02-05T06:56:54.868112+0000","last_became_active":"2024-02-05T06:55:53.778696+0000","last_became_peered":"2024-02-05T06:55:53.778696+0000","last_unstale":"2024-02-05T06:56:54.868112+0000","last_undegraded":"2024-02-05T06:56:54.868112+0000","last_fullsized":"2024-02-05T06:56:54.868112+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"498'14","last_scrub_stamp":"2024-02-05T06:56:53.238634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:56:53.238634+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:13.635891+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":599,"reported_epoch":509,"state":"active+clean+scrubbing","last_fresh":"2024-02-05T06:57:01.928512+0000","last_change":"2024-02-05T06:57:01.927424+0000","last_active":"2024-02-05T06:57:01.928512+0000","last_peered":"2024-02-05T06:57:01.928512+0000","last_clean":"2024-02-05T06:57:01.928512+0000","last_became_active":"2024-02-05T06:41:43.418066+0000","last_became_peered":"2024-02-05T06:41:43.418066+0000","last_unstale":"2024-02-05T06:57:01.928512+0000","last_undegraded":"2024-02-05T06:57:01.928512+0000","last_fullsized":"2024-02-05T06:57:01.928512+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:33.998012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:48:34.623889+0000","last_clean_scrub_stamp":"2024-02-05T06:55:33.998012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":1010,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.357133+0000","last_change":"2024-02-05T06:55:55.737355+0000","last_active":"2024-02-05T06:56:46.357133+0000","last_peered":"2024-02-05T06:56:46.357133+0000","last_clean":"2024-02-05T06:56:46.357133+0000","last_became_active":"2024-02-05T06:41:46.053993+0000","last_became_peered":"2024-02-05T06:41:46.053993+0000","last_unstale":"2024-02-05T06:56:46.357133+0000","last_undegraded":"2024-02-05T06:56:46.357133+0000","last_fullsized":"2024-02-05T06:56:46.357133+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":251,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:55:55.737264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:55:55.737264+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:21.901412+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":487,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.354728+0000","last_change":"2024-02-05T06:56:10.639208+0000","last_active":"2024-02-05T06:56:46.354728+0000","last_peered":"2024-02-05T06:56:46.354728+0000","last_clean":"2024-02-05T06:56:46.354728+0000","last_became_active":"2024-02-05T06:55:53.780475+0000","last_became_peered":"2024-02-05T06:55:53.780475+0000","last_unstale":"2024-02-05T06:56:46.354728+0000","last_undegraded":"2024-02-05T06:56:46.354728+0000","last_fullsized":"2024-02-05T06:56:46.354728+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:10.639107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:56:10.639107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:20.408170+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"465'50","reported_seq":1083,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.849739+0000","last_change":"2024-02-05T06:56:36.871929+0000","last_active":"2024-02-05T06:56:46.849739+0000","last_peered":"2024-02-05T06:56:46.849739+0000","last_clean":"2024-02-05T06:56:46.849739+0000","last_became_active":"2024-02-05T06:40:27.277747+0000","last_became_peered":"2024-02-05T06:40:27.277747+0000","last_unstale":"2024-02-05T06:56:46.849739+0000","last_undegraded":"2024-02-05T06:56:46.849739+0000","last_fullsized":"2024-02-05T06:56:46.849739+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'50","last_scrub_stamp":"2024-02-05T06:56:36.871842+0000","last_deep_scrub":"424'38","last_deep_scrub_stamp":"2024-02-05T06:51:40.451087+0000","last_clean_scrub_stamp":"2024-02-05T06:56:36.871842+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:02.712961+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":988,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:59.745153+0000","last_change":"2024-02-05T06:56:59.745175+0000","last_active":"2024-02-05T06:56:59.745153+0000","last_peered":"2024-02-05T06:56:59.745153+0000","last_clean":"2024-02-05T06:56:59.745153+0000","last_became_active":"2024-02-05T06:40:27.093739+0000","last_became_peered":"2024-02-05T06:40:27.093739+0000","last_unstale":"2024-02-05T06:56:59.745153+0000","last_undegraded":"2024-02-05T06:56:59.745153+0000","last_fullsized":"2024-02-05T06:56:59.745153+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:59.745099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:53.475540+0000","last_clean_scrub_stamp":"2024-02-05T06:56:59.745099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:00.186601+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":467,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.354725+0000","last_change":"2024-02-05T06:56:07.678259+0000","last_active":"2024-02-05T06:56:46.354725+0000","last_peered":"2024-02-05T06:56:46.354725+0000","last_clean":"2024-02-05T06:56:46.354725+0000","last_became_active":"2024-02-05T06:55:53.777838+0000","last_became_peered":"2024-02-05T06:55:53.777838+0000","last_unstale":"2024-02-05T06:56:46.354725+0000","last_undegraded":"2024-02-05T06:56:46.354725+0000","last_fullsized":"2024-02-05T06:56:46.354725+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:07.678156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:45:49.145117+0000","last_clean_scrub_stamp":"2024-02-05T06:56:07.678156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:08.897862+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":745,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.354955+0000","last_change":"2024-02-05T06:55:55.522216+0000","last_active":"2024-02-05T06:56:46.354955+0000","last_peered":"2024-02-05T06:56:46.354955+0000","last_clean":"2024-02-05T06:56:46.354955+0000","last_became_active":"2024-02-05T06:55:53.782397+0000","last_became_peered":"2024-02-05T06:55:53.782397+0000","last_unstale":"2024-02-05T06:56:46.354955+0000","last_undegraded":"2024-02-05T06:56:46.354955+0000","last_fullsized":"2024-02-05T06:56:46.354955+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:55:55.522125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:55:55.522125+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:22.394591+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"326'11","reported_seq":1269,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:54.867886+0000","last_change":"2024-02-05T06:56:14.663006+0000","last_active":"2024-02-05T06:56:54.867886+0000","last_peered":"2024-02-05T06:56:54.867886+0000","last_clean":"2024-02-05T06:56:54.867886+0000","last_became_active":"2024-02-05T06:36:54.954819+0000","last_became_peered":"2024-02-05T06:36:54.954819+0000","last_unstale":"2024-02-05T06:56:54.867886+0000","last_undegraded":"2024-02-05T06:56:54.867886+0000","last_fullsized":"2024-02-05T06:56:54.867886+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"326'11","last_scrub_stamp":"2024-02-05T06:56:14.662916+0000","last_deep_scrub":"326'11","last_deep_scrub_stamp":"2024-02-05T06:45:56.212177+0000","last_clean_scrub_stamp":"2024-02-05T06:56:14.662916+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:40.301441+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"465'42","reported_seq":853,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.354542+0000","last_change":"2024-02-05T06:56:12.621051+0000","last_active":"2024-02-05T06:56:46.354542+0000","last_peered":"2024-02-05T06:56:46.354542+0000","last_clean":"2024-02-05T06:56:46.354542+0000","last_became_active":"2024-02-05T06:55:53.781507+0000","last_became_peered":"2024-02-05T06:55:53.781507+0000","last_unstale":"2024-02-05T06:56:46.354542+0000","last_undegraded":"2024-02-05T06:56:46.354542+0000","last_fullsized":"2024-02-05T06:56:46.354542+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'42","last_scrub_stamp":"2024-02-05T06:56:12.620935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:56:12.620935+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:34.962046+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":53,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"95'2","reported_seq":1001,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.352345+0000","last_change":"2024-02-05T06:56:35.127594+0000","last_active":"2024-02-05T06:56:46.352345+0000","last_peered":"2024-02-05T06:56:46.352345+0000","last_clean":"2024-02-05T06:56:46.352345+0000","last_became_active":"2024-02-05T06:36:54.975093+0000","last_became_peered":"2024-02-05T06:36:54.975093+0000","last_unstale":"2024-02-05T06:56:46.352345+0000","last_undegraded":"2024-02-05T06:56:46.352345+0000","last_fullsized":"2024-02-05T06:56:46.352345+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"95'2","last_scrub_stamp":"2024-02-05T06:56:35.127506+0000","last_deep_scrub":"95'2","last_deep_scrub_stamp":"2024-02-05T06:35:25.616708+0000","last_clean_scrub_stamp":"2024-02-05T06:56:35.127506+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:42.999494+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":687,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:52.183530+0000","last_change":"2024-02-05T06:56:52.183555+0000","last_active":"2024-02-05T06:56:52.183530+0000","last_peered":"2024-02-05T06:56:52.183530+0000","last_clean":"2024-02-05T06:56:52.183530+0000","last_became_active":"2024-02-05T06:40:27.099168+0000","last_became_peered":"2024-02-05T06:40:27.099168+0000","last_unstale":"2024-02-05T06:56:52.183530+0000","last_undegraded":"2024-02-05T06:56:52.183530+0000","last_fullsized":"2024-02-05T06:56:52.183530+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:52.183451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:56:52.183451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:56.102342+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":1186,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:53.835113+0000","last_change":"2024-02-05T06:56:53.835136+0000","last_active":"2024-02-05T06:56:53.835113+0000","last_peered":"2024-02-05T06:56:53.835113+0000","last_clean":"2024-02-05T06:56:53.835113+0000","last_became_active":"2024-02-05T06:51:54.332037+0000","last_became_peered":"2024-02-05T06:51:54.332037+0000","last_unstale":"2024-02-05T06:56:53.835113+0000","last_undegraded":"2024-02-05T06:56:53.835113+0000","last_fullsized":"2024-02-05T06:56:53.835113+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:56:53.835057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:56:53.835057+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:18.020028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"335'11","reported_seq":1284,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:54.867904+0000","last_change":"2024-02-05T06:56:17.384021+0000","last_active":"2024-02-05T06:56:54.867904+0000","last_peered":"2024-02-05T06:56:54.867904+0000","last_clean":"2024-02-05T06:56:54.867904+0000","last_became_active":"2024-02-05T06:41:43.423638+0000","last_became_peered":"2024-02-05T06:41:43.423638+0000","last_unstale":"2024-02-05T06:56:54.867904+0000","last_undegraded":"2024-02-05T06:56:54.867904+0000","last_fullsized":"2024-02-05T06:56:54.867904+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"335'11","last_scrub_stamp":"2024-02-05T06:56:17.383944+0000","last_deep_scrub":"335'11","last_deep_scrub_stamp":"2024-02-05T06:52:40.187864+0000","last_clean_scrub_stamp":"2024-02-05T06:56:17.383944+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:19.860181+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"465'35","reported_seq":88,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.351368+0000","last_change":"2024-02-05T06:56:01.659934+0000","last_active":"2024-02-05T06:56:46.351368+0000","last_peered":"2024-02-05T06:56:46.351368+0000","last_clean":"2024-02-05T06:56:46.351368+0000","last_became_active":"2024-02-05T06:55:08.236796+0000","last_became_peered":"2024-02-05T06:55:08.236796+0000","last_unstale":"2024-02-05T06:56:46.351368+0000","last_undegraded":"2024-02-05T06:56:46.351368+0000","last_fullsized":"2024-02-05T06:56:46.351368+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'35","last_scrub_stamp":"2024-02-05T06:56:01.659841+0000","last_deep_scrub":"424'25","last_deep_scrub_stamp":"2024-02-05T06:51:39.813235+0000","last_clean_scrub_stamp":"2024-02-05T06:56:01.659841+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:25.835842+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":595,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.875182+0000","last_change":"2024-02-05T06:56:46.875206+0000","last_active":"2024-02-05T06:56:46.875182+0000","last_peered":"2024-02-05T06:56:46.875182+0000","last_clean":"2024-02-05T06:56:46.875182+0000","last_became_active":"2024-02-05T06:41:43.411732+0000","last_became_peered":"2024-02-05T06:41:43.411732+0000","last_unstale":"2024-02-05T06:56:46.875182+0000","last_undegraded":"2024-02-05T06:56:46.875182+0000","last_fullsized":"2024-02-05T06:56:46.875182+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:46.875115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:56:46.875115+0000","last_clean_scrub_stamp":"2024-02-05T06:56:46.875115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:01.487040+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":960,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.350744+0000","last_change":"2024-02-05T06:56:00.748685+0000","last_active":"2024-02-05T06:56:46.350744+0000","last_peered":"2024-02-05T06:56:46.350744+0000","last_clean":"2024-02-05T06:56:46.350744+0000","last_became_active":"2024-02-05T06:33:35.600485+0000","last_became_peered":"2024-02-05T06:33:35.600485+0000","last_unstale":"2024-02-05T06:56:46.350744+0000","last_undegraded":"2024-02-05T06:56:46.350744+0000","last_fullsized":"2024-02-05T06:56:46.350744+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:00.748582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:56:00.748582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:29.338413+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":1189,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.853027+0000","last_change":"2024-02-05T06:56:17.299742+0000","last_active":"2024-02-05T06:56:46.853027+0000","last_peered":"2024-02-05T06:56:46.853027+0000","last_clean":"2024-02-05T06:56:46.853027+0000","last_became_active":"2024-02-05T06:51:54.332371+0000","last_became_peered":"2024-02-05T06:51:54.332371+0000","last_unstale":"2024-02-05T06:56:46.853027+0000","last_undegraded":"2024-02-05T06:56:46.853027+0000","last_fullsized":"2024-02-05T06:56:46.853027+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:56:17.299642+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:56:17.299642+0000","last_clean_scrub_stamp":"2024-02-05T06:56:17.299642+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:40.989374+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":997,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.852466+0000","last_change":"2024-02-05T06:55:55.155977+0000","last_active":"2024-02-05T06:56:46.852466+0000","last_peered":"2024-02-05T06:56:46.852466+0000","last_clean":"2024-02-05T06:56:46.852466+0000","last_became_active":"2024-02-05T06:41:43.411550+0000","last_became_peered":"2024-02-05T06:41:43.411550+0000","last_unstale":"2024-02-05T06:56:46.852466+0000","last_undegraded":"2024-02-05T06:56:46.852466+0000","last_fullsized":"2024-02-05T06:56:46.852466+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:55.155885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:55:55.155885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:02.646148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"483'47","reported_seq":1146,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.353243+0000","last_change":"2024-02-05T06:56:20.326060+0000","last_active":"2024-02-05T06:56:46.353243+0000","last_peered":"2024-02-05T06:56:46.353243+0000","last_clean":"2024-02-05T06:56:46.353243+0000","last_became_active":"2024-02-05T06:41:43.409664+0000","last_became_peered":"2024-02-05T06:41:43.409664+0000","last_unstale":"2024-02-05T06:56:46.353243+0000","last_undegraded":"2024-02-05T06:56:46.353243+0000","last_fullsized":"2024-02-05T06:56:46.353243+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"483'47","last_scrub_stamp":"2024-02-05T06:56:20.325962+0000","last_deep_scrub":"483'47","last_deep_scrub_stamp":"2024-02-05T06:55:09.338593+0000","last_clean_scrub_stamp":"2024-02-05T06:56:20.325962+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:30.784476+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"445'2","reported_seq":1014,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:50.675774+0000","last_change":"2024-02-05T06:56:50.675799+0000","last_active":"2024-02-05T06:56:50.675774+0000","last_peered":"2024-02-05T06:56:50.675774+0000","last_clean":"2024-02-05T06:56:50.675774+0000","last_became_active":"2024-02-05T06:41:43.411404+0000","last_became_peered":"2024-02-05T06:41:43.411404+0000","last_unstale":"2024-02-05T06:56:50.675774+0000","last_undegraded":"2024-02-05T06:56:50.675774+0000","last_fullsized":"2024-02-05T06:56:50.675774+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'2","last_scrub_stamp":"2024-02-05T06:56:50.675717+0000","last_deep_scrub":"445'2","last_deep_scrub_stamp":"2024-02-05T06:52:37.034607+0000","last_clean_scrub_stamp":"2024-02-05T06:56:50.675717+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:03.738104+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":975,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.351989+0000","last_change":"2024-02-05T06:56:44.545932+0000","last_active":"2024-02-05T06:56:46.351989+0000","last_peered":"2024-02-05T06:56:46.351989+0000","last_clean":"2024-02-05T06:56:46.351989+0000","last_became_active":"2024-02-05T06:33:35.664858+0000","last_became_peered":"2024-02-05T06:33:35.664858+0000","last_unstale":"2024-02-05T06:56:46.351989+0000","last_undegraded":"2024-02-05T06:56:46.351989+0000","last_fullsized":"2024-02-05T06:56:46.351989+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:44.545830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:56:44.545830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:06.913870+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":1170,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.852946+0000","last_change":"2024-02-05T06:56:05.282642+0000","last_active":"2024-02-05T06:56:46.852946+0000","last_peered":"2024-02-05T06:56:46.852946+0000","last_clean":"2024-02-05T06:56:46.852946+0000","last_became_active":"2024-02-05T06:51:54.330806+0000","last_became_peered":"2024-02-05T06:51:54.330806+0000","last_unstale":"2024-02-05T06:56:46.852946+0000","last_undegraded":"2024-02-05T06:56:46.852946+0000","last_fullsized":"2024-02-05T06:56:46.852946+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:56:05.282558+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:38.778568+0000","last_clean_scrub_stamp":"2024-02-05T06:56:05.282558+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:27.374792+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":595,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:52.915542+0000","last_change":"2024-02-05T06:56:52.915542+0000","last_active":"2024-02-05T06:56:52.915542+0000","last_peered":"2024-02-05T06:56:52.915542+0000","last_clean":"2024-02-05T06:56:52.915542+0000","last_became_active":"2024-02-05T06:41:43.417751+0000","last_became_peered":"2024-02-05T06:41:43.417751+0000","last_unstale":"2024-02-05T06:56:52.915542+0000","last_undegraded":"2024-02-05T06:56:52.915542+0000","last_fullsized":"2024-02-05T06:56:52.915542+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:52.915434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:56:52.915434+0000","last_clean_scrub_stamp":"2024-02-05T06:56:52.915434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:15.880903+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"465'36","reported_seq":1029,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.352239+0000","last_change":"2024-02-05T06:56:22.259254+0000","last_active":"2024-02-05T06:56:46.352239+0000","last_peered":"2024-02-05T06:56:46.352239+0000","last_clean":"2024-02-05T06:56:46.352239+0000","last_became_active":"2024-02-05T06:33:31.564604+0000","last_became_peered":"2024-02-05T06:33:31.564604+0000","last_unstale":"2024-02-05T06:56:46.352239+0000","last_undegraded":"2024-02-05T06:56:46.352239+0000","last_fullsized":"2024-02-05T06:56:46.352239+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'36","last_scrub_stamp":"2024-02-05T06:56:22.259172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:56:22.259172+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:35.295571+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"355'6","reported_seq":1682,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:57:00.751009+0000","last_change":"2024-02-05T06:55:49.773020+0000","last_active":"2024-02-05T06:57:00.751009+0000","last_peered":"2024-02-05T06:57:00.751009+0000","last_clean":"2024-02-05T06:57:00.751009+0000","last_became_active":"2024-02-05T06:40:27.071509+0000","last_became_peered":"2024-02-05T06:40:27.071509+0000","last_unstale":"2024-02-05T06:57:00.751009+0000","last_undegraded":"2024-02-05T06:57:00.751009+0000","last_fullsized":"2024-02-05T06:57:00.751009+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"355'6","last_scrub_stamp":"2024-02-05T06:55:49.772867+0000","last_deep_scrub":"355'6","last_deep_scrub_stamp":"2024-02-05T06:47:46.642454+0000","last_clean_scrub_stamp":"2024-02-05T06:55:49.772867+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:01.906169+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00074151600000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1000,"num_read_kb":995,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":992,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:58.174065+0000","last_change":"2024-02-05T06:56:58.174091+0000","last_active":"2024-02-05T06:56:58.174065+0000","last_peered":"2024-02-05T06:56:58.174065+0000","last_clean":"2024-02-05T06:56:58.174065+0000","last_became_active":"2024-02-05T06:36:54.972264+0000","last_became_peered":"2024-02-05T06:36:54.972264+0000","last_unstale":"2024-02-05T06:56:58.174065+0000","last_undegraded":"2024-02-05T06:56:58.174065+0000","last_fullsized":"2024-02-05T06:56:58.174065+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:58.173983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:55:28.341411+0000","last_clean_scrub_stamp":"2024-02-05T06:56:58.173983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:58.912160+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":995,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.352357+0000","last_change":"2024-02-05T06:56:37.699285+0000","last_active":"2024-02-05T06:56:46.352357+0000","last_peered":"2024-02-05T06:56:46.352357+0000","last_clean":"2024-02-05T06:56:46.352357+0000","last_became_active":"2024-02-05T06:41:43.410910+0000","last_became_peered":"2024-02-05T06:41:43.410910+0000","last_unstale":"2024-02-05T06:56:46.352357+0000","last_undegraded":"2024-02-05T06:56:46.352357+0000","last_fullsized":"2024-02-05T06:56:46.352357+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:37.699195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:53:47.688112+0000","last_clean_scrub_stamp":"2024-02-05T06:56:37.699195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:46.615453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":1151,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.352505+0000","last_change":"2024-02-05T06:56:05.843711+0000","last_active":"2024-02-05T06:56:46.352505+0000","last_peered":"2024-02-05T06:56:46.352505+0000","last_clean":"2024-02-05T06:56:46.352505+0000","last_became_active":"2024-02-05T06:51:54.323320+0000","last_became_peered":"2024-02-05T06:51:54.323320+0000","last_unstale":"2024-02-05T06:56:46.352505+0000","last_undegraded":"2024-02-05T06:56:46.352505+0000","last_fullsized":"2024-02-05T06:56:46.352505+0000","mapping_epoch":134,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:56:05.843622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:56:05.843622+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:16.650034+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":701,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:53.023202+0000","last_change":"2024-02-05T06:56:53.023224+0000","last_active":"2024-02-05T06:56:53.023202+0000","last_peered":"2024-02-05T06:56:53.023202+0000","last_clean":"2024-02-05T06:56:53.023202+0000","last_became_active":"2024-02-05T06:41:43.418198+0000","last_became_peered":"2024-02-05T06:41:43.418198+0000","last_unstale":"2024-02-05T06:56:53.023202+0000","last_undegraded":"2024-02-05T06:56:53.023202+0000","last_fullsized":"2024-02-05T06:56:53.023202+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:53.023134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:56:53.023134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:21.357434+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"465'52","reported_seq":1436,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:54.868137+0000","last_change":"2024-02-05T06:56:20.296600+0000","last_active":"2024-02-05T06:56:54.868137+0000","last_peered":"2024-02-05T06:56:54.868137+0000","last_clean":"2024-02-05T06:56:54.868137+0000","last_became_active":"2024-02-05T06:40:27.061111+0000","last_became_peered":"2024-02-05T06:40:27.061111+0000","last_unstale":"2024-02-05T06:56:54.868137+0000","last_undegraded":"2024-02-05T06:56:54.868137+0000","last_fullsized":"2024-02-05T06:56:54.868137+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'52","last_scrub_stamp":"2024-02-05T06:56:20.296465+0000","last_deep_scrub":"465'52","last_deep_scrub_stamp":"2024-02-05T06:56:20.296465+0000","last_clean_scrub_stamp":"2024-02-05T06:56:20.296465+0000","objects_scrubbed":8,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:23.715637+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":983,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:57:02.763185+0000","last_change":"2024-02-05T06:57:02.763221+0000","last_active":"2024-02-05T06:57:02.763185+0000","last_peered":"2024-02-05T06:57:02.763185+0000","last_clean":"2024-02-05T06:57:02.763185+0000","last_became_active":"2024-02-05T06:41:43.411335+0000","last_became_peered":"2024-02-05T06:41:43.411335+0000","last_unstale":"2024-02-05T06:57:02.763185+0000","last_undegraded":"2024-02-05T06:57:02.763185+0000","last_fullsized":"2024-02-05T06:57:02.763185+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:57:02.763102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:57:02.763102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:27.550586+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1118,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.358935+0000","last_change":"2024-02-05T06:55:42.761231+0000","last_active":"2024-02-05T06:56:46.358935+0000","last_peered":"2024-02-05T06:56:46.358935+0000","last_clean":"2024-02-05T06:56:46.358935+0000","last_became_active":"2024-02-05T06:51:54.326585+0000","last_became_peered":"2024-02-05T06:51:54.326585+0000","last_unstale":"2024-02-05T06:56:46.358935+0000","last_undegraded":"2024-02-05T06:56:46.358935+0000","last_fullsized":"2024-02-05T06:56:46.358935+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:55:42.761093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:55:42.761093+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:01.946539+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":595,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.358045+0000","last_change":"2024-02-05T06:56:04.871500+0000","last_active":"2024-02-05T06:56:46.358045+0000","last_peered":"2024-02-05T06:56:46.358045+0000","last_clean":"2024-02-05T06:56:46.358045+0000","last_became_active":"2024-02-05T06:41:43.408773+0000","last_became_peered":"2024-02-05T06:41:43.408773+0000","last_unstale":"2024-02-05T06:56:46.358045+0000","last_undegraded":"2024-02-05T06:56:46.358045+0000","last_fullsized":"2024-02-05T06:56:46.358045+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:04.871418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:56:04.871418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:14.338347+0000","scrub_duration":125,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1032,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.873315+0000","last_change":"2024-02-05T06:56:32.642024+0000","last_active":"2024-02-05T06:56:46.873315+0000","last_peered":"2024-02-05T06:56:46.873315+0000","last_clean":"2024-02-05T06:56:46.873315+0000","last_became_active":"2024-02-05T06:41:43.412387+0000","last_became_peered":"2024-02-05T06:41:43.412387+0000","last_unstale":"2024-02-05T06:56:46.873315+0000","last_undegraded":"2024-02-05T06:56:46.873315+0000","last_fullsized":"2024-02-05T06:56:46.873315+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:32.641914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:34.871805+0000","last_clean_scrub_stamp":"2024-02-05T06:56:32.641914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:41.179139+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"506'26","reported_seq":1088,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.352818+0000","last_change":"2024-02-05T06:56:28.880945+0000","last_active":"2024-02-05T06:56:46.352818+0000","last_peered":"2024-02-05T06:56:46.352818+0000","last_clean":"2024-02-05T06:56:46.352818+0000","last_became_active":"2024-02-05T06:41:43.420458+0000","last_became_peered":"2024-02-05T06:41:43.420458+0000","last_unstale":"2024-02-05T06:56:46.352818+0000","last_undegraded":"2024-02-05T06:56:46.352818+0000","last_fullsized":"2024-02-05T06:56:46.352818+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"506'26","last_scrub_stamp":"2024-02-05T06:56:28.880824+0000","last_deep_scrub":"506'26","last_deep_scrub_stamp":"2024-02-05T06:56:28.880824+0000","last_clean_scrub_stamp":"2024-02-05T06:56:28.880824+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:50.242950+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:54.555119+0000","last_change":"2024-02-05T06:56:54.555138+0000","last_active":"2024-02-05T06:56:54.555119+0000","last_peered":"2024-02-05T06:56:54.555119+0000","last_clean":"2024-02-05T06:56:54.555119+0000","last_became_active":"2024-02-05T06:33:28.542977+0000","last_became_peered":"2024-02-05T06:33:28.542977+0000","last_unstale":"2024-02-05T06:56:54.555119+0000","last_undegraded":"2024-02-05T06:56:54.555119+0000","last_fullsized":"2024-02-05T06:56:54.555119+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:54.555061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:27.524724+0000","last_clean_scrub_stamp":"2024-02-05T06:56:54.555061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:09.046543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091640799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":987,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:57:00.545178+0000","last_change":"2024-02-05T06:57:00.545198+0000","last_active":"2024-02-05T06:57:00.545178+0000","last_peered":"2024-02-05T06:57:00.545178+0000","last_clean":"2024-02-05T06:57:00.545178+0000","last_became_active":"2024-02-05T06:33:29.551549+0000","last_became_peered":"2024-02-05T06:33:29.551549+0000","last_unstale":"2024-02-05T06:57:00.545178+0000","last_undegraded":"2024-02-05T06:57:00.545178+0000","last_fullsized":"2024-02-05T06:57:00.545178+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:57:00.545117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:57:00.545117+0000","last_clean_scrub_stamp":"2024-02-05T06:57:00.545117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:13.887965+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":687,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:56.104477+0000","last_change":"2024-02-05T06:56:56.104502+0000","last_active":"2024-02-05T06:56:56.104477+0000","last_peered":"2024-02-05T06:56:56.104477+0000","last_clean":"2024-02-05T06:56:56.104477+0000","last_became_active":"2024-02-05T06:40:27.077966+0000","last_became_peered":"2024-02-05T06:40:27.077966+0000","last_unstale":"2024-02-05T06:56:56.104477+0000","last_undegraded":"2024-02-05T06:56:56.104477+0000","last_fullsized":"2024-02-05T06:56:56.104477+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:56.104390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:45.774441+0000","last_clean_scrub_stamp":"2024-02-05T06:56:56.104390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:59.852125+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":1125,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.358330+0000","last_change":"2024-02-05T06:56:17.945481+0000","last_active":"2024-02-05T06:56:46.358330+0000","last_peered":"2024-02-05T06:56:46.358330+0000","last_clean":"2024-02-05T06:56:46.358330+0000","last_became_active":"2024-02-05T06:51:54.325427+0000","last_became_peered":"2024-02-05T06:51:54.325427+0000","last_unstale":"2024-02-05T06:56:46.358330+0000","last_undegraded":"2024-02-05T06:56:46.358330+0000","last_fullsized":"2024-02-05T06:56:46.358330+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:56:17.945381+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:55:13.625211+0000","last_clean_scrub_stamp":"2024-02-05T06:56:17.945381+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:37.743131+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":984,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.352143+0000","last_change":"2024-02-05T06:56:04.632453+0000","last_active":"2024-02-05T06:56:46.352143+0000","last_peered":"2024-02-05T06:56:46.352143+0000","last_clean":"2024-02-05T06:56:46.352143+0000","last_became_active":"2024-02-05T06:36:54.973387+0000","last_became_peered":"2024-02-05T06:36:54.973387+0000","last_unstale":"2024-02-05T06:56:46.352143+0000","last_undegraded":"2024-02-05T06:56:46.352143+0000","last_fullsized":"2024-02-05T06:56:46.352143+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:04.632368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:56:04.632368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:11.107575+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"465'64","reported_seq":972,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:57.676806+0000","last_change":"2024-02-05T06:56:57.676806+0000","last_active":"2024-02-05T06:56:57.676806+0000","last_peered":"2024-02-05T06:56:57.676806+0000","last_clean":"2024-02-05T06:56:57.676806+0000","last_became_active":"2024-02-05T06:40:08.604483+0000","last_became_peered":"2024-02-05T06:40:08.604483+0000","last_unstale":"2024-02-05T06:56:57.676806+0000","last_undegraded":"2024-02-05T06:56:57.676806+0000","last_fullsized":"2024-02-05T06:56:57.676806+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'64","last_scrub_stamp":"2024-02-05T06:56:57.676658+0000","last_deep_scrub":"326'52","last_deep_scrub_stamp":"2024-02-05T06:45:53.715482+0000","last_clean_scrub_stamp":"2024-02-05T06:56:57.676658+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:59.770868+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":467,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.355034+0000","last_change":"2024-02-05T06:56:43.336295+0000","last_active":"2024-02-05T06:56:46.355034+0000","last_peered":"2024-02-05T06:56:46.355034+0000","last_clean":"2024-02-05T06:56:46.355034+0000","last_became_active":"2024-02-05T06:55:53.782266+0000","last_became_peered":"2024-02-05T06:55:53.782266+0000","last_unstale":"2024-02-05T06:56:46.355034+0000","last_undegraded":"2024-02-05T06:56:46.355034+0000","last_fullsized":"2024-02-05T06:56:46.355034+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:43.336174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:47.126978+0000","last_clean_scrub_stamp":"2024-02-05T06:56:43.336174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:48.067076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"182'3","reported_seq":599,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.357004+0000","last_change":"2024-02-05T06:56:22.817189+0000","last_active":"2024-02-05T06:56:46.357004+0000","last_peered":"2024-02-05T06:56:46.357004+0000","last_clean":"2024-02-05T06:56:46.357004+0000","last_became_active":"2024-02-05T06:41:43.415806+0000","last_became_peered":"2024-02-05T06:41:43.415806+0000","last_unstale":"2024-02-05T06:56:46.357004+0000","last_undegraded":"2024-02-05T06:56:46.357004+0000","last_fullsized":"2024-02-05T06:56:46.357004+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'3","last_scrub_stamp":"2024-02-05T06:56:22.817098+0000","last_deep_scrub":"182'3","last_deep_scrub_stamp":"2024-02-05T06:39:33.373415+0000","last_clean_scrub_stamp":"2024-02-05T06:56:22.817098+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:49.617085+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001220348,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":974,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.850304+0000","last_change":"2024-02-05T06:56:20.730377+0000","last_active":"2024-02-05T06:56:46.850304+0000","last_peered":"2024-02-05T06:56:46.850304+0000","last_clean":"2024-02-05T06:56:46.850304+0000","last_became_active":"2024-02-05T06:40:27.082440+0000","last_became_peered":"2024-02-05T06:40:27.082440+0000","last_unstale":"2024-02-05T06:56:46.850304+0000","last_undegraded":"2024-02-05T06:56:46.850304+0000","last_fullsized":"2024-02-05T06:56:46.850304+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:20.730298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:50:52.403179+0000","last_clean_scrub_stamp":"2024-02-05T06:56:20.730298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:38.646090+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":1021,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.356865+0000","last_change":"2024-02-05T06:56:44.888318+0000","last_active":"2024-02-05T06:56:46.356865+0000","last_peered":"2024-02-05T06:56:46.356865+0000","last_clean":"2024-02-05T06:56:46.356865+0000","last_became_active":"2024-02-05T06:51:54.325737+0000","last_became_peered":"2024-02-05T06:51:54.325737+0000","last_unstale":"2024-02-05T06:56:46.356865+0000","last_undegraded":"2024-02-05T06:56:46.356865+0000","last_fullsized":"2024-02-05T06:56:46.356865+0000","mapping_epoch":250,"log_start":"0'0","ondisk_log_start":"0'0","created":129,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:56:44.888186+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:37.282442+0000","last_clean_scrub_stamp":"2024-02-05T06:56:44.888186+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:59.607640+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.e","version":"83'128","reported_seq":342,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.355978+0000","last_change":"2024-02-05T06:55:54.571484+0000","last_active":"2024-02-05T06:56:46.355978+0000","last_peered":"2024-02-05T06:56:46.355978+0000","last_clean":"2024-02-05T06:56:46.355978+0000","last_became_active":"2024-02-05T06:55:53.781272+0000","last_became_peered":"2024-02-05T06:55:53.781272+0000","last_unstale":"2024-02-05T06:56:46.355978+0000","last_undegraded":"2024-02-05T06:56:46.355978+0000","last_fullsized":"2024-02-05T06:56:46.355978+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:55:54.571345+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:54:47.849978+0000","last_clean_scrub_stamp":"2024-02-05T06:55:54.571345+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:19.045414+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"465'21","reported_seq":1089,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.353092+0000","last_change":"2024-02-05T06:56:10.441093+0000","last_active":"2024-02-05T06:56:46.353092+0000","last_peered":"2024-02-05T06:56:46.353092+0000","last_clean":"2024-02-05T06:56:46.353092+0000","last_became_active":"2024-02-05T06:40:08.584204+0000","last_became_peered":"2024-02-05T06:40:08.584204+0000","last_unstale":"2024-02-05T06:56:46.353092+0000","last_undegraded":"2024-02-05T06:56:46.353092+0000","last_fullsized":"2024-02-05T06:56:46.353092+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'21","last_scrub_stamp":"2024-02-05T06:56:10.441001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:56:10.441001+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:17.612654+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":594,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.357417+0000","last_change":"2024-02-05T06:56:30.847013+0000","last_active":"2024-02-05T06:56:46.357417+0000","last_peered":"2024-02-05T06:56:46.357417+0000","last_clean":"2024-02-05T06:56:46.357417+0000","last_became_active":"2024-02-05T06:41:43.412611+0000","last_became_peered":"2024-02-05T06:41:43.412611+0000","last_unstale":"2024-02-05T06:56:46.357417+0000","last_undegraded":"2024-02-05T06:56:46.357417+0000","last_fullsized":"2024-02-05T06:56:46.357417+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:30.846924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:41.617526+0000","last_clean_scrub_stamp":"2024-02-05T06:56:30.846924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:31.441121+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1012,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.872792+0000","last_change":"2024-02-05T06:56:02.579265+0000","last_active":"2024-02-05T06:56:46.872792+0000","last_peered":"2024-02-05T06:56:46.872792+0000","last_clean":"2024-02-05T06:56:46.872792+0000","last_became_active":"2024-02-05T06:36:54.966016+0000","last_became_peered":"2024-02-05T06:36:54.966016+0000","last_unstale":"2024-02-05T06:56:46.872792+0000","last_undegraded":"2024-02-05T06:56:46.872792+0000","last_fullsized":"2024-02-05T06:56:46.872792+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:02.579186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:56:02.579186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:17.018541+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":971,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.351537+0000","last_change":"2024-02-05T06:55:50.646138+0000","last_active":"2024-02-05T06:56:46.351537+0000","last_peered":"2024-02-05T06:56:46.351537+0000","last_clean":"2024-02-05T06:56:46.351537+0000","last_became_active":"2024-02-05T06:40:27.129625+0000","last_became_peered":"2024-02-05T06:40:27.129625+0000","last_unstale":"2024-02-05T06:56:46.351537+0000","last_undegraded":"2024-02-05T06:56:46.351537+0000","last_fullsized":"2024-02-05T06:56:46.351537+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:50.646032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:55:50.646032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:55.101476+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"83'128","reported_seq":1197,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.852918+0000","last_change":"2024-02-05T06:55:49.199226+0000","last_active":"2024-02-05T06:56:46.852918+0000","last_peered":"2024-02-05T06:56:46.852918+0000","last_clean":"2024-02-05T06:56:46.852918+0000","last_became_active":"2024-02-05T06:51:59.293451+0000","last_became_peered":"2024-02-05T06:51:59.293451+0000","last_unstale":"2024-02-05T06:56:46.852918+0000","last_undegraded":"2024-02-05T06:56:46.852918+0000","last_fullsized":"2024-02-05T06:56:46.852918+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:55:49.199108+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:48:25.725892+0000","last_clean_scrub_stamp":"2024-02-05T06:55:49.199108+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:18.160870+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"465'60","reported_seq":1162,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.852430+0000","last_change":"2024-02-05T06:56:14.234074+0000","last_active":"2024-02-05T06:56:46.852430+0000","last_peered":"2024-02-05T06:56:46.852430+0000","last_clean":"2024-02-05T06:56:46.852430+0000","last_became_active":"2024-02-05T06:40:27.068417+0000","last_became_peered":"2024-02-05T06:40:27.068417+0000","last_unstale":"2024-02-05T06:56:46.852430+0000","last_undegraded":"2024-02-05T06:56:46.852430+0000","last_fullsized":"2024-02-05T06:56:46.852430+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'60","last_scrub_stamp":"2024-02-05T06:56:14.233941+0000","last_deep_scrub":"279'28","last_deep_scrub_stamp":"2024-02-05T06:42:50.148881+0000","last_clean_scrub_stamp":"2024-02-05T06:56:14.233941+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:15.288233+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":693,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:51.027887+0000","last_change":"2024-02-05T06:56:51.027910+0000","last_active":"2024-02-05T06:56:51.027887+0000","last_peered":"2024-02-05T06:56:51.027887+0000","last_clean":"2024-02-05T06:56:51.027887+0000","last_became_active":"2024-02-05T06:41:43.418202+0000","last_became_peered":"2024-02-05T06:41:43.418202+0000","last_unstale":"2024-02-05T06:56:51.027887+0000","last_undegraded":"2024-02-05T06:56:51.027887+0000","last_fullsized":"2024-02-05T06:56:51.027887+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:51.027789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:56:51.027789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:52.404063+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1018,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.872893+0000","last_change":"2024-02-05T06:56:44.583092+0000","last_active":"2024-02-05T06:56:46.872893+0000","last_peered":"2024-02-05T06:56:46.872893+0000","last_clean":"2024-02-05T06:56:46.872893+0000","last_became_active":"2024-02-05T06:41:43.413095+0000","last_became_peered":"2024-02-05T06:41:43.413095+0000","last_unstale":"2024-02-05T06:56:46.872893+0000","last_undegraded":"2024-02-05T06:56:46.872893+0000","last_fullsized":"2024-02-05T06:56:46.872893+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:44.582999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:47:45.073681+0000","last_clean_scrub_stamp":"2024-02-05T06:56:44.582999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:46.979530+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":687,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.358785+0000","last_change":"2024-02-05T06:56:05.923363+0000","last_active":"2024-02-05T06:56:46.358785+0000","last_peered":"2024-02-05T06:56:46.358785+0000","last_clean":"2024-02-05T06:56:46.358785+0000","last_became_active":"2024-02-05T06:40:27.218081+0000","last_became_peered":"2024-02-05T06:40:27.218081+0000","last_unstale":"2024-02-05T06:56:46.358785+0000","last_undegraded":"2024-02-05T06:56:46.358785+0000","last_fullsized":"2024-02-05T06:56:46.358785+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:05.923279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:39:32.925000+0000","last_clean_scrub_stamp":"2024-02-05T06:56:05.923279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:14.587523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"83'128","reported_seq":1199,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.851903+0000","last_change":"2024-02-05T06:56:46.851929+0000","last_active":"2024-02-05T06:56:46.851903+0000","last_peered":"2024-02-05T06:56:46.851903+0000","last_clean":"2024-02-05T06:56:46.851903+0000","last_became_active":"2024-02-05T06:51:59.296988+0000","last_became_peered":"2024-02-05T06:51:59.296988+0000","last_unstale":"2024-02-05T06:56:46.851903+0000","last_undegraded":"2024-02-05T06:56:46.851903+0000","last_fullsized":"2024-02-05T06:56:46.851903+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:56:46.851842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:56:46.851842+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:55.756240+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":992,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.872451+0000","last_change":"2024-02-05T06:55:45.424698+0000","last_active":"2024-02-05T06:56:46.872451+0000","last_peered":"2024-02-05T06:56:46.872451+0000","last_clean":"2024-02-05T06:56:46.872451+0000","last_became_active":"2024-02-05T06:33:33.592546+0000","last_became_peered":"2024-02-05T06:33:33.592546+0000","last_unstale":"2024-02-05T06:56:46.872451+0000","last_undegraded":"2024-02-05T06:56:46.872451+0000","last_fullsized":"2024-02-05T06:56:46.872451+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:45.424621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:42.685240+0000","last_clean_scrub_stamp":"2024-02-05T06:55:45.424621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:01.990910+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"465'24","reported_seq":781,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.354680+0000","last_change":"2024-02-05T06:56:40.310264+0000","last_active":"2024-02-05T06:56:46.354680+0000","last_peered":"2024-02-05T06:56:46.354680+0000","last_clean":"2024-02-05T06:56:46.354680+0000","last_became_active":"2024-02-05T06:55:53.777933+0000","last_became_peered":"2024-02-05T06:55:53.777933+0000","last_unstale":"2024-02-05T06:56:46.354680+0000","last_undegraded":"2024-02-05T06:56:46.354680+0000","last_fullsized":"2024-02-05T06:56:46.354680+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'24","last_scrub_stamp":"2024-02-05T06:56:40.310150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:56:40.310150+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:55.385961+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":37,"num_read_kb":28,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":687,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.359203+0000","last_change":"2024-02-05T06:56:43.034625+0000","last_active":"2024-02-05T06:56:46.359203+0000","last_peered":"2024-02-05T06:56:46.359203+0000","last_clean":"2024-02-05T06:56:46.359203+0000","last_became_active":"2024-02-05T06:40:27.063300+0000","last_became_peered":"2024-02-05T06:40:27.063300+0000","last_unstale":"2024-02-05T06:56:46.359203+0000","last_undegraded":"2024-02-05T06:56:46.359203+0000","last_fullsized":"2024-02-05T06:56:46.359203+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:43.034489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:56:43.034489+0000","last_clean_scrub_stamp":"2024-02-05T06:56:43.034489+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:44.201374+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":962,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.850096+0000","last_change":"2024-02-05T06:56:44.963204+0000","last_active":"2024-02-05T06:56:46.850096+0000","last_peered":"2024-02-05T06:56:46.850096+0000","last_clean":"2024-02-05T06:56:46.850096+0000","last_became_active":"2024-02-05T06:40:27.092613+0000","last_became_peered":"2024-02-05T06:40:27.092613+0000","last_unstale":"2024-02-05T06:56:46.850096+0000","last_undegraded":"2024-02-05T06:56:46.850096+0000","last_fullsized":"2024-02-05T06:56:46.850096+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:44.963072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:55:14.568023+0000","last_clean_scrub_stamp":"2024-02-05T06:56:44.963072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:02.151427+0000","scrub_duration":165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"83'128","reported_seq":756,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.358386+0000","last_change":"2024-02-05T06:55:57.812292+0000","last_active":"2024-02-05T06:56:46.358386+0000","last_peered":"2024-02-05T06:56:46.358386+0000","last_clean":"2024-02-05T06:56:46.358386+0000","last_became_active":"2024-02-05T06:51:59.294191+0000","last_became_peered":"2024-02-05T06:51:59.294191+0000","last_unstale":"2024-02-05T06:56:46.358386+0000","last_undegraded":"2024-02-05T06:56:46.358386+0000","last_fullsized":"2024-02-05T06:56:46.358386+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:55:57.812200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:55:57.812200+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:15.888287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"506'13","reported_seq":891,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:54.868120+0000","last_change":"2024-02-05T06:56:32.958495+0000","last_active":"2024-02-05T06:56:54.868120+0000","last_peered":"2024-02-05T06:56:54.868120+0000","last_clean":"2024-02-05T06:56:54.868120+0000","last_became_active":"2024-02-05T06:40:27.064195+0000","last_became_peered":"2024-02-05T06:40:27.064195+0000","last_unstale":"2024-02-05T06:56:54.868120+0000","last_undegraded":"2024-02-05T06:56:54.868120+0000","last_fullsized":"2024-02-05T06:56:54.868120+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"506'13","last_scrub_stamp":"2024-02-05T06:56:32.958410+0000","last_deep_scrub":"506'13","last_deep_scrub_stamp":"2024-02-05T06:56:32.958410+0000","last_clean_scrub_stamp":"2024-02-05T06:56:32.958410+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:56.948169+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"465'39","reported_seq":647,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:59.925955+0000","last_change":"2024-02-05T06:56:59.926000+0000","last_active":"2024-02-05T06:56:59.925955+0000","last_peered":"2024-02-05T06:56:59.925955+0000","last_clean":"2024-02-05T06:56:59.925955+0000","last_became_active":"2024-02-05T06:41:43.412579+0000","last_became_peered":"2024-02-05T06:41:43.412579+0000","last_unstale":"2024-02-05T06:56:59.925955+0000","last_undegraded":"2024-02-05T06:56:59.925955+0000","last_fullsized":"2024-02-05T06:56:59.925955+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'39","last_scrub_stamp":"2024-02-05T06:56:59.925878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:56:59.925878+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:21.450493+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"412'4","reported_seq":695,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.359287+0000","last_change":"2024-02-05T06:56:13.027321+0000","last_active":"2024-02-05T06:56:46.359287+0000","last_peered":"2024-02-05T06:56:46.359287+0000","last_clean":"2024-02-05T06:56:46.359287+0000","last_became_active":"2024-02-05T06:40:27.068279+0000","last_became_peered":"2024-02-05T06:40:27.068279+0000","last_unstale":"2024-02-05T06:56:46.359287+0000","last_undegraded":"2024-02-05T06:56:46.359287+0000","last_fullsized":"2024-02-05T06:56:46.359287+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"412'4","last_scrub_stamp":"2024-02-05T06:56:13.027147+0000","last_deep_scrub":"412'4","last_deep_scrub_stamp":"2024-02-05T06:56:13.027147+0000","last_clean_scrub_stamp":"2024-02-05T06:56:13.027147+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:26.844350+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":595,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:55.969297+0000","last_change":"2024-02-05T06:56:55.969297+0000","last_active":"2024-02-05T06:56:55.969297+0000","last_peered":"2024-02-05T06:56:55.969297+0000","last_clean":"2024-02-05T06:56:55.969297+0000","last_became_active":"2024-02-05T06:41:43.421231+0000","last_became_peered":"2024-02-05T06:41:43.421231+0000","last_unstale":"2024-02-05T06:56:55.969297+0000","last_undegraded":"2024-02-05T06:56:55.969297+0000","last_fullsized":"2024-02-05T06:56:55.969297+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:55.969201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:56:55.969201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:19.486105+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"83'128","reported_seq":623,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.356636+0000","last_change":"2024-02-05T06:56:20.806816+0000","last_active":"2024-02-05T06:56:46.356636+0000","last_peered":"2024-02-05T06:56:46.356636+0000","last_clean":"2024-02-05T06:56:46.356636+0000","last_became_active":"2024-02-05T06:51:59.296150+0000","last_became_peered":"2024-02-05T06:51:59.296150+0000","last_unstale":"2024-02-05T06:56:46.356636+0000","last_undegraded":"2024-02-05T06:56:46.356636+0000","last_fullsized":"2024-02-05T06:56:46.356636+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:56:20.806725+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:48:28.657679+0000","last_clean_scrub_stamp":"2024-02-05T06:56:20.806725+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:32.420942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"465'21","reported_seq":1015,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.850570+0000","last_change":"2024-02-05T06:56:09.572815+0000","last_active":"2024-02-05T06:56:46.850570+0000","last_peered":"2024-02-05T06:56:46.850570+0000","last_clean":"2024-02-05T06:56:46.850570+0000","last_became_active":"2024-02-05T06:40:27.091689+0000","last_became_peered":"2024-02-05T06:40:27.091689+0000","last_unstale":"2024-02-05T06:56:46.850570+0000","last_undegraded":"2024-02-05T06:56:46.850570+0000","last_fullsized":"2024-02-05T06:56:46.850570+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'21","last_scrub_stamp":"2024-02-05T06:56:09.572716+0000","last_deep_scrub":"412'15","last_deep_scrub_stamp":"2024-02-05T06:50:55.397689+0000","last_clean_scrub_stamp":"2024-02-05T06:56:09.572716+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:27.603023+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1004,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.852053+0000","last_change":"2024-02-05T06:55:36.193765+0000","last_active":"2024-02-05T06:56:46.852053+0000","last_peered":"2024-02-05T06:56:46.852053+0000","last_clean":"2024-02-05T06:56:46.852053+0000","last_became_active":"2024-02-05T06:41:43.412130+0000","last_became_peered":"2024-02-05T06:41:43.412130+0000","last_unstale":"2024-02-05T06:56:46.852053+0000","last_undegraded":"2024-02-05T06:56:46.852053+0000","last_fullsized":"2024-02-05T06:56:46.852053+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:36.193661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:55:36.193661+0000","last_clean_scrub_stamp":"2024-02-05T06:55:36.193661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:55.759566+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":594,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.357758+0000","last_change":"2024-02-05T06:56:16.881001+0000","last_active":"2024-02-05T06:56:46.357758+0000","last_peered":"2024-02-05T06:56:46.357758+0000","last_clean":"2024-02-05T06:56:46.357758+0000","last_became_active":"2024-02-05T06:41:43.406094+0000","last_became_peered":"2024-02-05T06:41:43.406094+0000","last_unstale":"2024-02-05T06:56:46.357758+0000","last_undegraded":"2024-02-05T06:56:46.357758+0000","last_fullsized":"2024-02-05T06:56:46.357758+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:16.880873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:44:57.506909+0000","last_clean_scrub_stamp":"2024-02-05T06:56:16.880873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:18.999720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":686,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:59.092762+0000","last_change":"2024-02-05T06:56:59.092786+0000","last_active":"2024-02-05T06:56:59.092762+0000","last_peered":"2024-02-05T06:56:59.092762+0000","last_clean":"2024-02-05T06:56:59.092762+0000","last_became_active":"2024-02-05T06:40:27.063350+0000","last_became_peered":"2024-02-05T06:40:27.063350+0000","last_unstale":"2024-02-05T06:56:59.092762+0000","last_undegraded":"2024-02-05T06:56:59.092762+0000","last_fullsized":"2024-02-05T06:56:59.092762+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:59.092665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:56:59.092665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:20.274497+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"83'128","reported_seq":1158,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.352164+0000","last_change":"2024-02-05T06:55:54.882340+0000","last_active":"2024-02-05T06:56:46.352164+0000","last_peered":"2024-02-05T06:56:46.352164+0000","last_clean":"2024-02-05T06:56:46.352164+0000","last_became_active":"2024-02-05T06:51:59.303621+0000","last_became_peered":"2024-02-05T06:51:59.303621+0000","last_unstale":"2024-02-05T06:56:46.352164+0000","last_undegraded":"2024-02-05T06:56:46.352164+0000","last_fullsized":"2024-02-05T06:56:46.352164+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:55:54.882237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:55:54.882237+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:11.595754+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"465'21","reported_seq":1040,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.352543+0000","last_change":"2024-02-05T06:55:31.338533+0000","last_active":"2024-02-05T06:56:46.352543+0000","last_peered":"2024-02-05T06:56:46.352543+0000","last_clean":"2024-02-05T06:56:46.352543+0000","last_became_active":"2024-02-05T06:40:27.101116+0000","last_became_peered":"2024-02-05T06:40:27.101116+0000","last_unstale":"2024-02-05T06:56:46.352543+0000","last_undegraded":"2024-02-05T06:56:46.352543+0000","last_fullsized":"2024-02-05T06:56:46.352543+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'21","last_scrub_stamp":"2024-02-05T06:55:31.338415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:55:31.338415+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:53.035265+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":687,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:55.060483+0000","last_change":"2024-02-05T06:56:55.060511+0000","last_active":"2024-02-05T06:56:55.060483+0000","last_peered":"2024-02-05T06:56:55.060483+0000","last_clean":"2024-02-05T06:56:55.060483+0000","last_became_active":"2024-02-05T06:40:27.064388+0000","last_became_peered":"2024-02-05T06:40:27.064388+0000","last_unstale":"2024-02-05T06:56:55.060483+0000","last_undegraded":"2024-02-05T06:56:55.060483+0000","last_fullsized":"2024-02-05T06:56:55.060483+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:55.060402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:56:55.060402+0000","last_clean_scrub_stamp":"2024-02-05T06:56:55.060402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:11.784859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":979,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.850437+0000","last_change":"2024-02-05T06:56:43.822790+0000","last_active":"2024-02-05T06:56:46.850437+0000","last_peered":"2024-02-05T06:56:46.850437+0000","last_clean":"2024-02-05T06:56:46.850437+0000","last_became_active":"2024-02-05T06:40:27.091544+0000","last_became_peered":"2024-02-05T06:40:27.091544+0000","last_unstale":"2024-02-05T06:56:46.850437+0000","last_undegraded":"2024-02-05T06:56:46.850437+0000","last_fullsized":"2024-02-05T06:56:46.850437+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:43.822664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:56:43.822664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:01.135753+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1020,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.873205+0000","last_change":"2024-02-05T06:56:16.612678+0000","last_active":"2024-02-05T06:56:46.873205+0000","last_peered":"2024-02-05T06:56:46.873205+0000","last_clean":"2024-02-05T06:56:46.873205+0000","last_became_active":"2024-02-05T06:40:27.090632+0000","last_became_peered":"2024-02-05T06:40:27.090632+0000","last_unstale":"2024-02-05T06:56:46.873205+0000","last_undegraded":"2024-02-05T06:56:46.873205+0000","last_fullsized":"2024-02-05T06:56:46.873205+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:16.612559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:45.010039+0000","last_clean_scrub_stamp":"2024-02-05T06:56:16.612559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:44.852656+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"83'128","reported_seq":623,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.356502+0000","last_change":"2024-02-05T06:56:15.929990+0000","last_active":"2024-02-05T06:56:46.356502+0000","last_peered":"2024-02-05T06:56:46.356502+0000","last_clean":"2024-02-05T06:56:46.356502+0000","last_became_active":"2024-02-05T06:51:59.290466+0000","last_became_peered":"2024-02-05T06:51:59.290466+0000","last_unstale":"2024-02-05T06:56:46.356502+0000","last_undegraded":"2024-02-05T06:56:46.356502+0000","last_fullsized":"2024-02-05T06:56:46.356502+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:56:15.929877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:31:10.608186+0000","last_clean_scrub_stamp":"2024-02-05T06:56:15.929877+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:36.354409+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"465'21","reported_seq":622,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.357535+0000","last_change":"2024-02-05T06:56:37.005030+0000","last_active":"2024-02-05T06:56:46.357535+0000","last_peered":"2024-02-05T06:56:46.357535+0000","last_clean":"2024-02-05T06:56:46.357535+0000","last_became_active":"2024-02-05T06:41:43.413708+0000","last_became_peered":"2024-02-05T06:41:43.413708+0000","last_unstale":"2024-02-05T06:56:46.357535+0000","last_undegraded":"2024-02-05T06:56:46.357535+0000","last_fullsized":"2024-02-05T06:56:46.357535+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'21","last_scrub_stamp":"2024-02-05T06:56:37.004902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:56:37.004902+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:03.581406+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":991,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.351704+0000","last_change":"2024-02-05T06:55:56.582636+0000","last_active":"2024-02-05T06:56:46.351704+0000","last_peered":"2024-02-05T06:56:46.351704+0000","last_clean":"2024-02-05T06:56:46.351704+0000","last_became_active":"2024-02-05T06:40:27.088096+0000","last_became_peered":"2024-02-05T06:40:27.088096+0000","last_unstale":"2024-02-05T06:56:46.351704+0000","last_undegraded":"2024-02-05T06:56:46.351704+0000","last_fullsized":"2024-02-05T06:56:46.351704+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:56.582541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:53:43.562266+0000","last_clean_scrub_stamp":"2024-02-05T06:55:56.582541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:18.592287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"208'10","reported_seq":1265,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:58.051912+0000","last_change":"2024-02-05T06:55:55.469497+0000","last_active":"2024-02-05T06:56:58.051912+0000","last_peered":"2024-02-05T06:56:58.051912+0000","last_clean":"2024-02-05T06:56:58.051912+0000","last_became_active":"2024-02-05T06:40:27.096584+0000","last_became_peered":"2024-02-05T06:40:27.096584+0000","last_unstale":"2024-02-05T06:56:58.051912+0000","last_undegraded":"2024-02-05T06:56:58.051912+0000","last_fullsized":"2024-02-05T06:56:58.051912+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"208'10","last_scrub_stamp":"2024-02-05T06:55:55.469423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:55:55.469423+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:24.678194+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":487,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.356454+0000","last_change":"2024-02-05T06:56:13.794946+0000","last_active":"2024-02-05T06:56:46.356454+0000","last_peered":"2024-02-05T06:56:46.356454+0000","last_clean":"2024-02-05T06:56:46.356454+0000","last_became_active":"2024-02-05T06:55:53.776494+0000","last_became_peered":"2024-02-05T06:55:53.776494+0000","last_unstale":"2024-02-05T06:56:46.356454+0000","last_undegraded":"2024-02-05T06:56:46.356454+0000","last_fullsized":"2024-02-05T06:56:46.356454+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:13.794838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:41:38.559780+0000","last_clean_scrub_stamp":"2024-02-05T06:56:13.794838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:27.534724+0000","scrub_duration":158,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"83'128","reported_seq":623,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.351237+0000","last_change":"2024-02-05T06:56:14.365404+0000","last_active":"2024-02-05T06:56:46.351237+0000","last_peered":"2024-02-05T06:56:46.351237+0000","last_clean":"2024-02-05T06:56:46.351237+0000","last_became_active":"2024-02-05T06:51:59.300112+0000","last_became_peered":"2024-02-05T06:51:59.300112+0000","last_unstale":"2024-02-05T06:56:46.351237+0000","last_undegraded":"2024-02-05T06:56:46.351237+0000","last_fullsized":"2024-02-05T06:56:46.351237+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:56:14.365284+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:37:37.282442+0000","last_clean_scrub_stamp":"2024-02-05T06:56:14.365284+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:42.568686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"499'35","reported_seq":1062,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.850279+0000","last_change":"2024-02-05T06:55:57.523940+0000","last_active":"2024-02-05T06:56:46.850279+0000","last_peered":"2024-02-05T06:56:46.850279+0000","last_clean":"2024-02-05T06:56:46.850279+0000","last_became_active":"2024-02-05T06:40:27.093418+0000","last_became_peered":"2024-02-05T06:40:27.093418+0000","last_unstale":"2024-02-05T06:56:46.850279+0000","last_undegraded":"2024-02-05T06:56:46.850279+0000","last_fullsized":"2024-02-05T06:56:46.850279+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"499'35","last_scrub_stamp":"2024-02-05T06:55:57.523847+0000","last_deep_scrub":"499'35","last_deep_scrub_stamp":"2024-02-05T06:55:57.523847+0000","last_clean_scrub_stamp":"2024-02-05T06:55:57.523847+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:16.202240+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":679,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.358612+0000","last_change":"2024-02-05T06:55:39.823122+0000","last_active":"2024-02-05T06:56:46.358612+0000","last_peered":"2024-02-05T06:56:46.358612+0000","last_clean":"2024-02-05T06:56:46.358612+0000","last_became_active":"2024-02-05T06:40:27.098418+0000","last_became_peered":"2024-02-05T06:40:27.098418+0000","last_unstale":"2024-02-05T06:56:46.358612+0000","last_undegraded":"2024-02-05T06:56:46.358612+0000","last_fullsized":"2024-02-05T06:56:46.358612+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:39.823016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:55:39.823016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:53.345711+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"336'11","reported_seq":783,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:54.868165+0000","last_change":"2024-02-05T06:56:29.838510+0000","last_active":"2024-02-05T06:56:54.868165+0000","last_peered":"2024-02-05T06:56:54.868165+0000","last_clean":"2024-02-05T06:56:54.868165+0000","last_became_active":"2024-02-05T06:41:43.418753+0000","last_became_peered":"2024-02-05T06:41:43.418753+0000","last_unstale":"2024-02-05T06:56:54.868165+0000","last_undegraded":"2024-02-05T06:56:54.868165+0000","last_fullsized":"2024-02-05T06:56:54.868165+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'11","last_scrub_stamp":"2024-02-05T06:56:29.838382+0000","last_deep_scrub":"336'11","last_deep_scrub_stamp":"2024-02-05T06:54:57.729829+0000","last_clean_scrub_stamp":"2024-02-05T06:56:29.838382+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:51.634238+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":970,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.352959+0000","last_change":"2024-02-05T06:56:16.427380+0000","last_active":"2024-02-05T06:56:46.352959+0000","last_peered":"2024-02-05T06:56:46.352959+0000","last_clean":"2024-02-05T06:56:46.352959+0000","last_became_active":"2024-02-05T06:40:27.096132+0000","last_became_peered":"2024-02-05T06:40:27.096132+0000","last_unstale":"2024-02-05T06:56:46.352959+0000","last_undegraded":"2024-02-05T06:56:46.352959+0000","last_fullsized":"2024-02-05T06:56:46.352959+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:16.427279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:56:16.427279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:36.550984+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"465'32","reported_seq":794,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.355147+0000","last_change":"2024-02-05T06:56:27.429784+0000","last_active":"2024-02-05T06:56:46.355147+0000","last_peered":"2024-02-05T06:56:46.355147+0000","last_clean":"2024-02-05T06:56:46.355147+0000","last_became_active":"2024-02-05T06:55:53.782604+0000","last_became_peered":"2024-02-05T06:55:53.782604+0000","last_unstale":"2024-02-05T06:56:46.355147+0000","last_undegraded":"2024-02-05T06:56:46.355147+0000","last_fullsized":"2024-02-05T06:56:46.355147+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'32","last_scrub_stamp":"2024-02-05T06:56:27.429673+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:35:02.589229+0000","last_clean_scrub_stamp":"2024-02-05T06:56:27.429673+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:33.564171+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":479,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.354941+0000","last_change":"2024-02-05T06:56:41.348089+0000","last_active":"2024-02-05T06:56:46.354941+0000","last_peered":"2024-02-05T06:56:46.354941+0000","last_clean":"2024-02-05T06:56:46.354941+0000","last_became_active":"2024-02-05T06:55:53.781768+0000","last_became_peered":"2024-02-05T06:55:53.781768+0000","last_unstale":"2024-02-05T06:56:46.354941+0000","last_undegraded":"2024-02-05T06:56:46.354941+0000","last_fullsized":"2024-02-05T06:56:46.354941+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:41.347992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:56:41.347992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:06.192865+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":996,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.351716+0000","last_change":"2024-02-05T06:55:57.928797+0000","last_active":"2024-02-05T06:56:46.351716+0000","last_peered":"2024-02-05T06:56:46.351716+0000","last_clean":"2024-02-05T06:56:46.351716+0000","last_became_active":"2024-02-05T06:41:43.421433+0000","last_became_peered":"2024-02-05T06:41:43.421433+0000","last_unstale":"2024-02-05T06:56:46.351716+0000","last_undegraded":"2024-02-05T06:56:46.351716+0000","last_fullsized":"2024-02-05T06:56:46.351716+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:57.928712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:32.561974+0000","last_clean_scrub_stamp":"2024-02-05T06:55:57.928712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:26.367767+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":962,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.850384+0000","last_change":"2024-02-05T06:56:08.621910+0000","last_active":"2024-02-05T06:56:46.850384+0000","last_peered":"2024-02-05T06:56:46.850384+0000","last_clean":"2024-02-05T06:56:46.850384+0000","last_became_active":"2024-02-05T06:41:43.417155+0000","last_became_peered":"2024-02-05T06:41:43.417155+0000","last_unstale":"2024-02-05T06:56:46.850384+0000","last_undegraded":"2024-02-05T06:56:46.850384+0000","last_fullsized":"2024-02-05T06:56:46.850384+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:08.621797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:56:08.621797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:25.140001+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"465'21","reported_seq":772,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:54.223995+0000","last_change":"2024-02-05T06:56:54.224021+0000","last_active":"2024-02-05T06:56:54.223995+0000","last_peered":"2024-02-05T06:56:54.223995+0000","last_clean":"2024-02-05T06:56:54.223995+0000","last_became_active":"2024-02-05T06:39:13.244124+0000","last_became_peered":"2024-02-05T06:39:13.244124+0000","last_unstale":"2024-02-05T06:56:54.223995+0000","last_undegraded":"2024-02-05T06:56:54.223995+0000","last_fullsized":"2024-02-05T06:56:54.223995+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'21","last_scrub_stamp":"2024-02-05T06:56:54.223932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:56:54.223932+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:06.048010+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":678,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.358805+0000","last_change":"2024-02-05T06:55:43.787798+0000","last_active":"2024-02-05T06:56:46.358805+0000","last_peered":"2024-02-05T06:56:46.358805+0000","last_clean":"2024-02-05T06:56:46.358805+0000","last_became_active":"2024-02-05T06:40:27.077701+0000","last_became_peered":"2024-02-05T06:40:27.077701+0000","last_unstale":"2024-02-05T06:56:46.358805+0000","last_undegraded":"2024-02-05T06:56:46.358805+0000","last_fullsized":"2024-02-05T06:56:46.358805+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:43.787663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:33.978941+0000","last_clean_scrub_stamp":"2024-02-05T06:55:43.787663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:13.399182+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":954,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.849947+0000","last_change":"2024-02-05T06:56:24.785042+0000","last_active":"2024-02-05T06:56:46.849947+0000","last_peered":"2024-02-05T06:56:46.849947+0000","last_clean":"2024-02-05T06:56:46.849947+0000","last_became_active":"2024-02-05T06:33:33.593901+0000","last_became_peered":"2024-02-05T06:33:33.593901+0000","last_unstale":"2024-02-05T06:56:46.849947+0000","last_undegraded":"2024-02-05T06:56:46.849947+0000","last_fullsized":"2024-02-05T06:56:46.849947+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:24.784899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:30.567403+0000","last_clean_scrub_stamp":"2024-02-05T06:56:24.784899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:36.743655+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"336'2","reported_seq":989,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.351288+0000","last_change":"2024-02-05T06:56:33.267158+0000","last_active":"2024-02-05T06:56:46.351288+0000","last_peered":"2024-02-05T06:56:46.351288+0000","last_clean":"2024-02-05T06:56:46.351288+0000","last_became_active":"2024-02-05T06:41:43.409761+0000","last_became_peered":"2024-02-05T06:41:43.409761+0000","last_unstale":"2024-02-05T06:56:46.351288+0000","last_undegraded":"2024-02-05T06:56:46.351288+0000","last_fullsized":"2024-02-05T06:56:46.351288+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"336'2","last_scrub_stamp":"2024-02-05T06:56:33.267001+0000","last_deep_scrub":"336'2","last_deep_scrub_stamp":"2024-02-05T06:46:53.500155+0000","last_clean_scrub_stamp":"2024-02-05T06:56:33.267001+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:39.312822+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"83'128","reported_seq":1195,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.872634+0000","last_change":"2024-02-05T06:56:28.621133+0000","last_active":"2024-02-05T06:56:46.872634+0000","last_peered":"2024-02-05T06:56:46.872634+0000","last_clean":"2024-02-05T06:56:46.872634+0000","last_became_active":"2024-02-05T06:51:59.292766+0000","last_became_peered":"2024-02-05T06:51:59.292766+0000","last_unstale":"2024-02-05T06:56:46.872634+0000","last_undegraded":"2024-02-05T06:56:46.872634+0000","last_fullsized":"2024-02-05T06:56:46.872634+0000","mapping_epoch":434,"log_start":"0'0","ondisk_log_start":"0'0","created":429,"last_epoch_clean":435,"parent":"0.0","parent_split_bits":5,"last_scrub":"83'128","last_scrub_stamp":"2024-02-05T06:56:28.621052+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-02-05T06:55:18.345171+0000","last_clean_scrub_stamp":"2024-02-05T06:56:28.621052+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:50.591989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"465'32","reported_seq":1060,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.850591+0000","last_change":"2024-02-05T06:56:27.854385+0000","last_active":"2024-02-05T06:56:46.850591+0000","last_peered":"2024-02-05T06:56:46.850591+0000","last_clean":"2024-02-05T06:56:46.850591+0000","last_became_active":"2024-02-05T06:40:27.080906+0000","last_became_peered":"2024-02-05T06:40:27.080906+0000","last_unstale":"2024-02-05T06:56:46.850591+0000","last_undegraded":"2024-02-05T06:56:46.850591+0000","last_fullsized":"2024-02-05T06:56:46.850591+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'32","last_scrub_stamp":"2024-02-05T06:56:27.854239+0000","last_deep_scrub":"93'16","last_deep_scrub_stamp":"2024-02-05T06:35:22.938675+0000","last_clean_scrub_stamp":"2024-02-05T06:56:27.854239+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:49.114925+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":594,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.357873+0000","last_change":"2024-02-05T06:56:18.833609+0000","last_active":"2024-02-05T06:56:46.357873+0000","last_peered":"2024-02-05T06:56:46.357873+0000","last_clean":"2024-02-05T06:56:46.357873+0000","last_became_active":"2024-02-05T06:41:43.412654+0000","last_became_peered":"2024-02-05T06:41:43.412654+0000","last_unstale":"2024-02-05T06:56:46.357873+0000","last_undegraded":"2024-02-05T06:56:46.357873+0000","last_fullsized":"2024-02-05T06:56:46.357873+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:18.833517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:56:18.833517+0000","last_clean_scrub_stamp":"2024-02-05T06:56:18.833517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:41.974670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":687,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:48.012375+0000","last_change":"2024-02-05T06:56:48.012400+0000","last_active":"2024-02-05T06:56:48.012375+0000","last_peered":"2024-02-05T06:56:48.012375+0000","last_clean":"2024-02-05T06:56:48.012375+0000","last_became_active":"2024-02-05T06:40:27.062975+0000","last_became_peered":"2024-02-05T06:40:27.062975+0000","last_unstale":"2024-02-05T06:56:48.012375+0000","last_undegraded":"2024-02-05T06:56:48.012375+0000","last_fullsized":"2024-02-05T06:56:48.012375+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:48.012302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:48:34.628182+0000","last_clean_scrub_stamp":"2024-02-05T06:56:48.012302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:01.312488+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":744,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.352418+0000","last_change":"2024-02-05T06:56:32.970339+0000","last_active":"2024-02-05T06:56:46.352418+0000","last_peered":"2024-02-05T06:56:46.352418+0000","last_clean":"2024-02-05T06:56:46.352418+0000","last_became_active":"2024-02-05T06:39:13.245684+0000","last_became_peered":"2024-02-05T06:39:13.245684+0000","last_unstale":"2024-02-05T06:56:46.352418+0000","last_undegraded":"2024-02-05T06:56:46.352418+0000","last_fullsized":"2024-02-05T06:56:46.352418+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:32.970223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:56:32.970223+0000","last_clean_scrub_stamp":"2024-02-05T06:56:32.970223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:41.045865+0000","scrub_duration":205,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"465'12","reported_seq":976,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:52.882417+0000","last_change":"2024-02-05T06:56:52.882417+0000","last_active":"2024-02-05T06:56:52.882417+0000","last_peered":"2024-02-05T06:56:52.882417+0000","last_clean":"2024-02-05T06:56:52.882417+0000","last_became_active":"2024-02-05T06:33:31.573410+0000","last_became_peered":"2024-02-05T06:33:31.573410+0000","last_unstale":"2024-02-05T06:56:52.882417+0000","last_undegraded":"2024-02-05T06:56:52.882417+0000","last_fullsized":"2024-02-05T06:56:52.882417+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'12","last_scrub_stamp":"2024-02-05T06:56:52.882323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:56:52.882323+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:00.891218+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":982,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.352547+0000","last_change":"2024-02-05T06:55:57.629987+0000","last_active":"2024-02-05T06:56:46.352547+0000","last_peered":"2024-02-05T06:56:46.352547+0000","last_clean":"2024-02-05T06:56:46.352547+0000","last_became_active":"2024-02-05T06:41:43.424459+0000","last_became_peered":"2024-02-05T06:41:43.424459+0000","last_unstale":"2024-02-05T06:56:46.352547+0000","last_undegraded":"2024-02-05T06:56:46.352547+0000","last_fullsized":"2024-02-05T06:56:46.352547+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:57.629857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:55:57.629857+0000","last_clean_scrub_stamp":"2024-02-05T06:55:57.629857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:18.238975+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1024,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.852373+0000","last_change":"2024-02-05T06:56:27.239627+0000","last_active":"2024-02-05T06:56:46.852373+0000","last_peered":"2024-02-05T06:56:46.852373+0000","last_clean":"2024-02-05T06:56:46.852373+0000","last_became_active":"2024-02-05T06:40:27.060112+0000","last_became_peered":"2024-02-05T06:40:27.060112+0000","last_unstale":"2024-02-05T06:56:46.852373+0000","last_undegraded":"2024-02-05T06:56:46.852373+0000","last_fullsized":"2024-02-05T06:56:46.852373+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:27.239527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:48:33.848718+0000","last_clean_scrub_stamp":"2024-02-05T06:56:27.239527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:33.141945+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":693,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:54.067359+0000","last_change":"2024-02-05T06:56:54.067382+0000","last_active":"2024-02-05T06:56:54.067359+0000","last_peered":"2024-02-05T06:56:54.067359+0000","last_clean":"2024-02-05T06:56:54.067359+0000","last_became_active":"2024-02-05T06:44:54.625067+0000","last_became_peered":"2024-02-05T06:44:54.625067+0000","last_unstale":"2024-02-05T06:56:54.067359+0000","last_undegraded":"2024-02-05T06:56:54.067359+0000","last_fullsized":"2024-02-05T06:56:54.067359+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:54.067271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:43:48.512673+0000","last_clean_scrub_stamp":"2024-02-05T06:56:54.067271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:56.974525+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"465'32","reported_seq":1092,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:57:00.181253+0000","last_change":"2024-02-05T06:57:00.181287+0000","last_active":"2024-02-05T06:57:00.181253+0000","last_peered":"2024-02-05T06:57:00.181253+0000","last_clean":"2024-02-05T06:57:00.181253+0000","last_became_active":"2024-02-05T06:41:43.419769+0000","last_became_peered":"2024-02-05T06:41:43.419769+0000","last_unstale":"2024-02-05T06:57:00.181253+0000","last_undegraded":"2024-02-05T06:57:00.181253+0000","last_fullsized":"2024-02-05T06:57:00.181253+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'32","last_scrub_stamp":"2024-02-05T06:57:00.181185+0000","last_deep_scrub":"92'16","last_deep_scrub_stamp":"2024-02-05T06:43:03.221466+0000","last_clean_scrub_stamp":"2024-02-05T06:57:00.181185+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:11.366876+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":594,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.356595+0000","last_change":"2024-02-05T06:56:39.896425+0000","last_active":"2024-02-05T06:56:46.356595+0000","last_peered":"2024-02-05T06:56:46.356595+0000","last_clean":"2024-02-05T06:56:46.356595+0000","last_became_active":"2024-02-05T06:41:43.411043+0000","last_became_peered":"2024-02-05T06:41:43.411043+0000","last_unstale":"2024-02-05T06:56:46.356595+0000","last_undegraded":"2024-02-05T06:56:46.356595+0000","last_fullsized":"2024-02-05T06:56:46.356595+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:39.896331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:56:39.896331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:42.545286+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":878,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.850056+0000","last_change":"2024-02-05T06:55:58.538469+0000","last_active":"2024-02-05T06:56:46.850056+0000","last_peered":"2024-02-05T06:56:46.850056+0000","last_clean":"2024-02-05T06:56:46.850056+0000","last_became_active":"2024-02-05T06:40:27.097169+0000","last_became_peered":"2024-02-05T06:40:27.097169+0000","last_unstale":"2024-02-05T06:56:46.850056+0000","last_undegraded":"2024-02-05T06:56:46.850056+0000","last_fullsized":"2024-02-05T06:56:46.850056+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:58.538370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:55:58.538370+0000","last_clean_scrub_stamp":"2024-02-05T06:55:58.538370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:13.347458+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":997,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.351878+0000","last_change":"2024-02-05T06:55:59.073655+0000","last_active":"2024-02-05T06:56:46.351878+0000","last_peered":"2024-02-05T06:56:46.351878+0000","last_clean":"2024-02-05T06:56:46.351878+0000","last_became_active":"2024-02-05T06:41:43.414638+0000","last_became_peered":"2024-02-05T06:41:43.414638+0000","last_unstale":"2024-02-05T06:56:46.351878+0000","last_undegraded":"2024-02-05T06:56:46.351878+0000","last_fullsized":"2024-02-05T06:56:46.351878+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:59.073554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:55:59.073554+0000","last_clean_scrub_stamp":"2024-02-05T06:55:59.073554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:12.525337+0000","scrub_duration":132,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"465'35","reported_seq":1062,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.352445+0000","last_change":"2024-02-05T06:55:36.324828+0000","last_active":"2024-02-05T06:56:46.352445+0000","last_peered":"2024-02-05T06:56:46.352445+0000","last_clean":"2024-02-05T06:56:46.352445+0000","last_became_active":"2024-02-05T06:40:27.100610+0000","last_became_peered":"2024-02-05T06:40:27.100610+0000","last_unstale":"2024-02-05T06:56:46.352445+0000","last_undegraded":"2024-02-05T06:56:46.352445+0000","last_fullsized":"2024-02-05T06:56:46.352445+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":221,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'35","last_scrub_stamp":"2024-02-05T06:55:36.324742+0000","last_deep_scrub":"465'35","last_deep_scrub_stamp":"2024-02-05T06:53:54.509362+0000","last_clean_scrub_stamp":"2024-02-05T06:55:36.324742+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:56:57.356490+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":1014,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.351822+0000","last_change":"2024-02-05T06:56:26.916792+0000","last_active":"2024-02-05T06:56:46.351822+0000","last_peered":"2024-02-05T06:56:46.351822+0000","last_clean":"2024-02-05T06:56:46.351822+0000","last_became_active":"2024-02-05T06:36:54.958414+0000","last_became_peered":"2024-02-05T06:36:54.958414+0000","last_unstale":"2024-02-05T06:56:46.351822+0000","last_undegraded":"2024-02-05T06:56:46.351822+0000","last_fullsized":"2024-02-05T06:56:46.351822+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-02-05T06:56:26.916703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:56:26.916703+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:53.830736+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"314'13","reported_seq":1333,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:54.868102+0000","last_change":"2024-02-05T06:55:51.185124+0000","last_active":"2024-02-05T06:56:54.868102+0000","last_peered":"2024-02-05T06:56:54.868102+0000","last_clean":"2024-02-05T06:56:54.868102+0000","last_became_active":"2024-02-05T06:40:27.063399+0000","last_became_peered":"2024-02-05T06:40:27.063399+0000","last_unstale":"2024-02-05T06:56:54.868102+0000","last_undegraded":"2024-02-05T06:56:54.868102+0000","last_fullsized":"2024-02-05T06:56:54.868102+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"314'13","last_scrub_stamp":"2024-02-05T06:55:51.185019+0000","last_deep_scrub":"314'13","last_deep_scrub_stamp":"2024-02-05T06:44:53.670923+0000","last_clean_scrub_stamp":"2024-02-05T06:55:51.185019+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:19.687971+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"325'2","reported_seq":693,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.358392+0000","last_change":"2024-02-05T06:56:31.025946+0000","last_active":"2024-02-05T06:56:46.358392+0000","last_peered":"2024-02-05T06:56:46.358392+0000","last_clean":"2024-02-05T06:56:46.358392+0000","last_became_active":"2024-02-05T06:41:43.418640+0000","last_became_peered":"2024-02-05T06:41:43.418640+0000","last_unstale":"2024-02-05T06:56:46.358392+0000","last_undegraded":"2024-02-05T06:56:46.358392+0000","last_fullsized":"2024-02-05T06:56:46.358392+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'2","last_scrub_stamp":"2024-02-05T06:56:31.025780+0000","last_deep_scrub":"325'2","last_deep_scrub_stamp":"2024-02-05T06:45:46.242415+0000","last_clean_scrub_stamp":"2024-02-05T06:56:31.025780+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:32.257939+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"465'29","reported_seq":1057,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.849904+0000","last_change":"2024-02-05T06:55:49.420800+0000","last_active":"2024-02-05T06:56:46.849904+0000","last_peered":"2024-02-05T06:56:46.849904+0000","last_clean":"2024-02-05T06:56:46.849904+0000","last_became_active":"2024-02-05T06:36:54.954880+0000","last_became_peered":"2024-02-05T06:36:54.954880+0000","last_unstale":"2024-02-05T06:56:46.849904+0000","last_undegraded":"2024-02-05T06:56:46.849904+0000","last_fullsized":"2024-02-05T06:56:46.849904+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'29","last_scrub_stamp":"2024-02-05T06:55:49.420713+0000","last_deep_scrub":"445'23","last_deep_scrub_stamp":"2024-02-05T06:52:37.440858+0000","last_clean_scrub_stamp":"2024-02-05T06:55:49.420713+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:04.491402+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":459,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.354482+0000","last_change":"2024-02-05T06:55:53.778513+0000","last_active":"2024-02-05T06:56:46.354482+0000","last_peered":"2024-02-05T06:56:46.354482+0000","last_clean":"2024-02-05T06:56:46.354482+0000","last_became_active":"2024-02-05T06:55:53.778204+0000","last_became_peered":"2024-02-05T06:55:53.778204+0000","last_unstale":"2024-02-05T06:56:46.354482+0000","last_undegraded":"2024-02-05T06:56:46.354482+0000","last_fullsized":"2024-02-05T06:56:46.354482+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:42.887647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:55:42.887647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:07.990322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":766,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.852733+0000","last_change":"2024-02-05T06:56:39.329168+0000","last_active":"2024-02-05T06:56:46.852733+0000","last_peered":"2024-02-05T06:56:46.852733+0000","last_clean":"2024-02-05T06:56:46.852733+0000","last_became_active":"2024-02-05T06:39:13.248618+0000","last_became_peered":"2024-02-05T06:39:13.248618+0000","last_unstale":"2024-02-05T06:56:46.852733+0000","last_undegraded":"2024-02-05T06:56:46.852733+0000","last_fullsized":"2024-02-05T06:56:46.852733+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:39.329060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:51:34.551395+0000","last_clean_scrub_stamp":"2024-02-05T06:56:39.329060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:59.994846+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":963,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.850488+0000","last_change":"2024-02-05T06:56:22.064149+0000","last_active":"2024-02-05T06:56:46.850488+0000","last_peered":"2024-02-05T06:56:46.850488+0000","last_clean":"2024-02-05T06:56:46.850488+0000","last_became_active":"2024-02-05T06:41:43.411442+0000","last_became_peered":"2024-02-05T06:41:43.411442+0000","last_unstale":"2024-02-05T06:56:46.850488+0000","last_undegraded":"2024-02-05T06:56:46.850488+0000","last_fullsized":"2024-02-05T06:56:46.850488+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:22.063996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:56:22.063996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:41.197788+0000","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"465'46","reported_seq":1079,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.872966+0000","last_change":"2024-02-05T06:56:23.584296+0000","last_active":"2024-02-05T06:56:46.872966+0000","last_peered":"2024-02-05T06:56:46.872966+0000","last_clean":"2024-02-05T06:56:46.872966+0000","last_became_active":"2024-02-05T06:33:31.569763+0000","last_became_peered":"2024-02-05T06:33:31.569763+0000","last_unstale":"2024-02-05T06:56:46.872966+0000","last_undegraded":"2024-02-05T06:56:46.872966+0000","last_fullsized":"2024-02-05T06:56:46.872966+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'46","last_scrub_stamp":"2024-02-05T06:56:23.584185+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:34:35.234636+0000","last_clean_scrub_stamp":"2024-02-05T06:56:23.584185+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:45.464217+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"281'2","reported_seq":1025,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.353191+0000","last_change":"2024-02-05T06:56:46.134964+0000","last_active":"2024-02-05T06:56:46.353191+0000","last_peered":"2024-02-05T06:56:46.353191+0000","last_clean":"2024-02-05T06:56:46.353191+0000","last_became_active":"2024-02-05T06:40:27.073360+0000","last_became_peered":"2024-02-05T06:40:27.073360+0000","last_unstale":"2024-02-05T06:56:46.353191+0000","last_undegraded":"2024-02-05T06:56:46.353191+0000","last_fullsized":"2024-02-05T06:56:46.353191+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"281'2","last_scrub_stamp":"2024-02-05T06:56:46.134886+0000","last_deep_scrub":"281'2","last_deep_scrub_stamp":"2024-02-05T06:47:50.442159+0000","last_clean_scrub_stamp":"2024-02-05T06:56:46.134886+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:49.357469+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":981,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.351689+0000","last_change":"2024-02-05T06:55:56.885657+0000","last_active":"2024-02-05T06:56:46.351689+0000","last_peered":"2024-02-05T06:56:46.351689+0000","last_clean":"2024-02-05T06:56:46.351689+0000","last_became_active":"2024-02-05T06:36:54.958069+0000","last_became_peered":"2024-02-05T06:36:54.958069+0000","last_unstale":"2024-02-05T06:56:46.351689+0000","last_undegraded":"2024-02-05T06:56:46.351689+0000","last_fullsized":"2024-02-05T06:56:46.351689+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:56.885555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:55:56.885555+0000","last_clean_scrub_stamp":"2024-02-05T06:55:56.885555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:18.760179+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":488,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.356172+0000","last_change":"2024-02-05T06:56:22.520719+0000","last_active":"2024-02-05T06:56:46.356172+0000","last_peered":"2024-02-05T06:56:46.356172+0000","last_clean":"2024-02-05T06:56:46.356172+0000","last_became_active":"2024-02-05T06:55:53.782074+0000","last_became_peered":"2024-02-05T06:55:53.782074+0000","last_unstale":"2024-02-05T06:56:46.356172+0000","last_undegraded":"2024-02-05T06:56:46.356172+0000","last_fullsized":"2024-02-05T06:56:46.356172+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:22.520590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:56:22.520590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:28.502171+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"465'9","reported_seq":762,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.354921+0000","last_change":"2024-02-05T06:56:03.591900+0000","last_active":"2024-02-05T06:56:46.354921+0000","last_peered":"2024-02-05T06:56:46.354921+0000","last_clean":"2024-02-05T06:56:46.354921+0000","last_became_active":"2024-02-05T06:55:53.777564+0000","last_became_peered":"2024-02-05T06:55:53.777564+0000","last_unstale":"2024-02-05T06:56:46.354921+0000","last_undegraded":"2024-02-05T06:56:46.354921+0000","last_fullsized":"2024-02-05T06:56:46.354921+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'9","last_scrub_stamp":"2024-02-05T06:56:03.591805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:30.545651+0000","last_clean_scrub_stamp":"2024-02-05T06:56:03.591805+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:13.128737+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":594,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.357470+0000","last_change":"2024-02-05T06:56:33.939933+0000","last_active":"2024-02-05T06:56:46.357470+0000","last_peered":"2024-02-05T06:56:46.357470+0000","last_clean":"2024-02-05T06:56:46.357470+0000","last_became_active":"2024-02-05T06:41:43.408428+0000","last_became_peered":"2024-02-05T06:41:43.408428+0000","last_unstale":"2024-02-05T06:56:46.357470+0000","last_undegraded":"2024-02-05T06:56:46.357470+0000","last_fullsized":"2024-02-05T06:56:46.357470+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:33.939832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:43.828638+0000","last_clean_scrub_stamp":"2024-02-05T06:56:33.939832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:51.049407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":694,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:58.034257+0000","last_change":"2024-02-05T06:56:58.034281+0000","last_active":"2024-02-05T06:56:58.034257+0000","last_peered":"2024-02-05T06:56:58.034257+0000","last_clean":"2024-02-05T06:56:58.034257+0000","last_became_active":"2024-02-05T06:40:27.071758+0000","last_became_peered":"2024-02-05T06:40:27.071758+0000","last_unstale":"2024-02-05T06:56:58.034257+0000","last_undegraded":"2024-02-05T06:56:58.034257+0000","last_fullsized":"2024-02-05T06:56:58.034257+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:58.034183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:52:37.152741+0000","last_clean_scrub_stamp":"2024-02-05T06:56:58.034183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:15.962311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":594,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:56.957526+0000","last_change":"2024-02-05T06:56:56.957552+0000","last_active":"2024-02-05T06:56:56.957526+0000","last_peered":"2024-02-05T06:56:56.957526+0000","last_clean":"2024-02-05T06:56:56.957526+0000","last_became_active":"2024-02-05T06:41:43.407404+0000","last_became_peered":"2024-02-05T06:41:43.407404+0000","last_unstale":"2024-02-05T06:56:56.957526+0000","last_undegraded":"2024-02-05T06:56:56.957526+0000","last_fullsized":"2024-02-05T06:56:56.957526+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:56.957431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:56:56.957431+0000","last_clean_scrub_stamp":"2024-02-05T06:56:56.957431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:24.884518+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"465'25","reported_seq":770,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:55.225140+0000","last_change":"2024-02-05T06:56:55.225180+0000","last_active":"2024-02-05T06:56:55.225140+0000","last_peered":"2024-02-05T06:56:55.225140+0000","last_clean":"2024-02-05T06:56:55.225140+0000","last_became_active":"2024-02-05T06:55:53.773462+0000","last_became_peered":"2024-02-05T06:55:53.773462+0000","last_unstale":"2024-02-05T06:56:55.225140+0000","last_undegraded":"2024-02-05T06:56:55.225140+0000","last_fullsized":"2024-02-05T06:56:55.225140+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'25","last_scrub_stamp":"2024-02-05T06:56:55.225074+0000","last_deep_scrub":"92'13","last_deep_scrub_stamp":"2024-02-05T06:34:46.542893+0000","last_clean_scrub_stamp":"2024-02-05T06:56:55.225074+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:20.780809+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":586,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.357555+0000","last_change":"2024-02-05T06:56:09.824293+0000","last_active":"2024-02-05T06:56:46.357555+0000","last_peered":"2024-02-05T06:56:46.357555+0000","last_clean":"2024-02-05T06:56:46.357555+0000","last_became_active":"2024-02-05T06:41:43.403746+0000","last_became_peered":"2024-02-05T06:41:43.403746+0000","last_unstale":"2024-02-05T06:56:46.357555+0000","last_undegraded":"2024-02-05T06:56:46.357555+0000","last_fullsized":"2024-02-05T06:56:46.357555+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:09.824199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:34:44.081318+0000","last_clean_scrub_stamp":"2024-02-05T06:56:09.824199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:17.829232+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":526,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.850216+0000","last_change":"2024-02-05T06:56:01.576625+0000","last_active":"2024-02-05T06:56:46.850216+0000","last_peered":"2024-02-05T06:56:46.850216+0000","last_clean":"2024-02-05T06:56:46.850216+0000","last_became_active":"2024-02-05T06:42:33.129959+0000","last_became_peered":"2024-02-05T06:42:33.129959+0000","last_unstale":"2024-02-05T06:56:46.850216+0000","last_undegraded":"2024-02-05T06:56:46.850216+0000","last_fullsized":"2024-02-05T06:56:46.850216+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:01.576520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:45:51.217900+0000","last_clean_scrub_stamp":"2024-02-05T06:56:01.576520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:21.910242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":978,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.352958+0000","last_change":"2024-02-05T06:56:34.229094+0000","last_active":"2024-02-05T06:56:46.352958+0000","last_peered":"2024-02-05T06:56:46.352958+0000","last_clean":"2024-02-05T06:56:46.352958+0000","last_became_active":"2024-02-05T06:40:27.068888+0000","last_became_peered":"2024-02-05T06:40:27.068888+0000","last_unstale":"2024-02-05T06:56:46.352958+0000","last_undegraded":"2024-02-05T06:56:46.352958+0000","last_fullsized":"2024-02-05T06:56:46.352958+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:34.228993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:44:55.125921+0000","last_clean_scrub_stamp":"2024-02-05T06:56:34.228993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:34.450816+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"465'46","reported_seq":1025,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.850546+0000","last_change":"2024-02-05T06:55:33.311337+0000","last_active":"2024-02-05T06:56:46.850546+0000","last_peered":"2024-02-05T06:56:46.850546+0000","last_clean":"2024-02-05T06:56:46.850546+0000","last_became_active":"2024-02-05T06:33:31.572544+0000","last_became_peered":"2024-02-05T06:33:31.572544+0000","last_unstale":"2024-02-05T06:56:46.850546+0000","last_undegraded":"2024-02-05T06:56:46.850546+0000","last_fullsized":"2024-02-05T06:56:46.850546+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'46","last_scrub_stamp":"2024-02-05T06:55:33.311188+0000","last_deep_scrub":"88'22","last_deep_scrub_stamp":"2024-02-05T06:39:35.433539+0000","last_clean_scrub_stamp":"2024-02-05T06:55:33.311188+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:02.453477+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":971,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.850155+0000","last_change":"2024-02-05T06:56:34.909979+0000","last_active":"2024-02-05T06:56:46.850155+0000","last_peered":"2024-02-05T06:56:46.850155+0000","last_clean":"2024-02-05T06:56:46.850155+0000","last_became_active":"2024-02-05T06:33:29.553864+0000","last_became_peered":"2024-02-05T06:33:29.553864+0000","last_unstale":"2024-02-05T06:56:46.850155+0000","last_undegraded":"2024-02-05T06:56:46.850155+0000","last_fullsized":"2024-02-05T06:56:46.850155+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:34.909897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:28.530487+0000","last_clean_scrub_stamp":"2024-02-05T06:56:34.909897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:53.241517+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":475,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.354267+0000","last_change":"2024-02-05T06:56:31.319212+0000","last_active":"2024-02-05T06:56:46.354267+0000","last_peered":"2024-02-05T06:56:46.354267+0000","last_clean":"2024-02-05T06:56:46.354267+0000","last_became_active":"2024-02-05T06:55:53.782687+0000","last_became_peered":"2024-02-05T06:55:53.782687+0000","last_unstale":"2024-02-05T06:56:46.354267+0000","last_undegraded":"2024-02-05T06:56:46.354267+0000","last_fullsized":"2024-02-05T06:56:46.354267+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:31.319116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:46:51.388156+0000","last_clean_scrub_stamp":"2024-02-05T06:56:31.319116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:57.045141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":961,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.850000+0000","last_change":"2024-02-05T06:56:05.798253+0000","last_active":"2024-02-05T06:56:46.850000+0000","last_peered":"2024-02-05T06:56:46.850000+0000","last_clean":"2024-02-05T06:56:46.850000+0000","last_became_active":"2024-02-05T06:40:27.077485+0000","last_became_peered":"2024-02-05T06:40:27.077485+0000","last_unstale":"2024-02-05T06:56:46.850000+0000","last_undegraded":"2024-02-05T06:56:46.850000+0000","last_fullsized":"2024-02-05T06:56:46.850000+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:05.798158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:56:05.798158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:16.675686+0000","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"465'25","reported_seq":919,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.354205+0000","last_change":"2024-02-05T06:55:50.585094+0000","last_active":"2024-02-05T06:56:46.354205+0000","last_peered":"2024-02-05T06:56:46.354205+0000","last_clean":"2024-02-05T06:56:46.354205+0000","last_became_active":"2024-02-05T06:36:54.959330+0000","last_became_peered":"2024-02-05T06:36:54.959330+0000","last_unstale":"2024-02-05T06:56:46.354205+0000","last_undegraded":"2024-02-05T06:56:46.354205+0000","last_fullsized":"2024-02-05T06:56:46.354205+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"465'25","last_scrub_stamp":"2024-02-05T06:55:50.584974+0000","last_deep_scrub":"247'13","last_deep_scrub_stamp":"2024-02-05T06:41:42.473424+0000","last_clean_scrub_stamp":"2024-02-05T06:55:50.584974+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:18.668660+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":31,"num_read_kb":25,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":992,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:56.230428+0000","last_change":"2024-02-05T06:56:56.230448+0000","last_active":"2024-02-05T06:56:56.230428+0000","last_peered":"2024-02-05T06:56:56.230428+0000","last_clean":"2024-02-05T06:56:56.230428+0000","last_became_active":"2024-02-05T06:41:43.421315+0000","last_became_peered":"2024-02-05T06:41:43.421315+0000","last_unstale":"2024-02-05T06:56:56.230428+0000","last_undegraded":"2024-02-05T06:56:56.230428+0000","last_fullsized":"2024-02-05T06:56:56.230428+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:56.230362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:56:56.230362+0000","last_clean_scrub_stamp":"2024-02-05T06:56:56.230362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:58:16.510025+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":993,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.351912+0000","last_change":"2024-02-05T06:55:53.896587+0000","last_active":"2024-02-05T06:56:46.351912+0000","last_peered":"2024-02-05T06:56:46.351912+0000","last_clean":"2024-02-05T06:56:46.351912+0000","last_became_active":"2024-02-05T06:41:43.420342+0000","last_became_peered":"2024-02-05T06:41:43.420342+0000","last_unstale":"2024-02-05T06:56:46.351912+0000","last_undegraded":"2024-02-05T06:56:46.351912+0000","last_fullsized":"2024-02-05T06:56:46.351912+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:55:53.896510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:35:06.525012+0000","last_clean_scrub_stamp":"2024-02-05T06:55:53.896510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:07.570199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":485,"reported_epoch":509,"state":"active+clean","last_fresh":"2024-02-05T06:56:46.354804+0000","last_change":"2024-02-05T06:56:30.406241+0000","last_active":"2024-02-05T06:56:46.354804+0000","last_peered":"2024-02-05T06:56:46.354804+0000","last_clean":"2024-02-05T06:56:46.354804+0000","last_became_active":"2024-02-05T06:55:53.777092+0000","last_became_peered":"2024-02-05T06:55:53.777092+0000","last_unstale":"2024-02-05T06:56:46.354804+0000","last_undegraded":"2024-02-05T06:56:46.354804+0000","last_fullsized":"2024-02-05T06:56:46.354804+0000","mapping_epoch":497,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":498,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-05T06:56:30.406132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-05T06:33:34.572983+0000","last_clean_scrub_stamp":"2024-02-05T06:56:30.406132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-02-05T06:57:51.776695+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1000,"num_read_kb":995,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":262688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2688,"ondisk_log_size":2688,"up":63,"acting":63,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1527,"num_read_kb":1284,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":378,"num_bytes_recovered":8234,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1104,"ondisk_log_size":1104,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1133,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103455,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":821904,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":821118,"kb_avail":92923248,"statfs":{"total":95995035648,"available":95153405952,"internally_reserved":0,"allocated":794624,"data_stored":405511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":840825290},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.96599999999999997,"15min":1.1910000000000001},"min":{"1min":0.38600000000000001,"5min":0.23799999999999999,"15min":0.21299999999999999},"max":{"1min":0.84499999999999997,"5min":4.282,"15min":32.567},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.96999999999999997,"15min":1.365},"min":{"1min":0.40000000000000002,"5min":0.249,"15min":0.222},"max":{"1min":7.4260000000000002,"5min":7.4260000000000002,"15min":41.109999999999999},"last":0.40500000000000003}]},{"osd":1,"last update":"Mon Feb 5 06:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":1.2869999999999999,"15min":1.786},"min":{"1min":0.34599999999999997,"5min":0.25600000000000001,"15min":0.25},"max":{"1min":1.024,"5min":8.2569999999999997,"15min":84.216999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.079,"15min":1.778},"min":{"1min":0.379,"5min":0.24299999999999999,"15min":0.22800000000000001},"max":{"1min":0.751,"5min":7.9450000000000003,"15min":84.126999999999995},"last":0.46800000000000003}]},{"osd":2,"last update":"Mon Feb 5 06:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":1.1060000000000001,"15min":1.284},"min":{"1min":0.41399999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.82599999999999996,"5min":8.4740000000000002,"15min":28.704999999999998},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.65700000000000003,"5min":1.1259999999999999,"15min":1.286},"min":{"1min":0.49299999999999999,"5min":0.30099999999999999,"15min":0.26800000000000002},"max":{"1min":1.0609999999999999,"5min":11.452,"15min":23.678999999999998},"last":0.65100000000000002}]},{"osd":3,"last update":"Mon Feb 5 06:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":1.702,"15min":1.845},"min":{"1min":0.39900000000000002,"5min":0.312,"15min":0.28899999999999998},"max":{"1min":7.4050000000000002,"5min":34.374000000000002,"15min":73.283000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.69399999999999995,"5min":1.655,"15min":1.9570000000000001},"min":{"1min":0.46899999999999997,"5min":0.307,"15min":0.29499999999999998},"max":{"1min":1.1120000000000001,"5min":34.267000000000003,"15min":73.323999999999998},"last":0.68200000000000005}]},{"osd":4,"last update":"Mon Feb 5 06:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.78900000000000003,"15min":0.84699999999999998},"min":{"1min":0.309,"5min":0.252,"15min":0.20200000000000001},"max":{"1min":0.72699999999999998,"5min":4.8310000000000004,"15min":6.1980000000000004},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.751,"15min":0.96099999999999997},"min":{"1min":0.40200000000000002,"5min":0.26400000000000001,"15min":0.25600000000000001},"max":{"1min":0.70799999999999996,"5min":4.1870000000000003,"15min":41.270000000000003},"last":0.62}]},{"osd":5,"last update":"Mon Feb 5 06:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.82099999999999995,"15min":1.409},"min":{"1min":0.376,"5min":0.252,"15min":0.251},"max":{"1min":1.0900000000000001,"5min":4.6200000000000001,"15min":109.482},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.78000000000000003,"15min":1.4910000000000001},"min":{"1min":0.42199999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.78400000000000003,"5min":4.0039999999999996,"15min":109.56399999999999},"last":0.63500000000000001}]},{"osd":6,"last update":"Mon Feb 5 06:56:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.96599999999999997,"15min":1.5600000000000001},"min":{"1min":0.35299999999999998,"5min":0.317,"15min":0.28499999999999998},"max":{"1min":0.81000000000000005,"5min":4.8419999999999996,"15min":113.914},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.82899999999999996,"15min":1.478},"min":{"1min":0.36799999999999999,"5min":0.317,"15min":0.307},"max":{"1min":0.75600000000000001,"5min":4.202,"15min":113.212},"last":0.57199999999999995}]}]},{"osd":6,"up_from":49,"seq":210453397797,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":772216,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":771134,"kb_avail":92972936,"statfs":{"total":95995035648,"available":95204286464,"internally_reserved":0,"allocated":1097728,"data_stored":731742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":789641670},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:56:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":1.05,"15min":1.8959999999999999},"min":{"1min":0.34000000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.68799999999999994,"5min":10.039999999999999,"15min":98.918999999999997},"last":0.623},{"interface":"front","average":{"1min":0.55700000000000005,"5min":1.149,"15min":2.056},"min":{"1min":0.33900000000000002,"5min":0.248,"15min":0.223},"max":{"1min":0.83199999999999996,"5min":10.23,"15min":98.775000000000006},"last":0.57799999999999996}]},{"osd":1,"last update":"Mon Feb 5 06:56:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":1.0980000000000001,"15min":1.931},"min":{"1min":0.36099999999999999,"5min":0.27900000000000003,"15min":0.249},"max":{"1min":0.74199999999999999,"5min":6.3730000000000002,"15min":98.843999999999994},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.56499999999999995,"5min":1.1220000000000001,"15min":1.988},"min":{"1min":0.29199999999999998,"5min":0.24099999999999999,"15min":0.23100000000000001},"max":{"1min":0.73399999999999999,"5min":6.3899999999999997,"15min":101.637},"last":0.55500000000000005}]},{"osd":2,"last update":"Mon Feb 5 06:56:41 2024","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.228,"15min":1.5620000000000001},"min":{"1min":0.379,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":19.619,"5min":19.619,"15min":74.454999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.55400000000000005,"5min":1.169,"15min":1.5209999999999999},"min":{"1min":0.40500000000000003,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.73799999999999999,"5min":16.516999999999999,"15min":74.546000000000006},"last":0.63600000000000001}]},{"osd":3,"last update":"Mon Feb 5 06:56:41 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":1.3029999999999999,"15min":1.8029999999999999},"min":{"1min":0.40699999999999997,"5min":0.30299999999999999,"15min":0.27900000000000003},"max":{"1min":0.83499999999999996,"5min":14.648,"15min":98.888999999999996},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.623,"5min":1.323,"15min":1.7809999999999999},"min":{"1min":0.47299999999999998,"5min":0.313,"15min":0.27800000000000002},"max":{"1min":0.81200000000000006,"5min":14.6,"15min":98.969999999999999},"last":0.72099999999999997}]},{"osd":4,"last update":"Mon Feb 5 06:56:41 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.64400000000000002,"15min":0.76200000000000001},"min":{"1min":0.27300000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.77600000000000002,"5min":5.4720000000000004,"15min":5.5940000000000003},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.68300000000000005,"15min":0.78300000000000003},"min":{"1min":0.34999999999999998,"5min":0.313,"15min":0.24399999999999999},"max":{"1min":0.76500000000000001,"5min":5.9500000000000002,"15min":5.9500000000000002},"last":0.45700000000000002}]},{"osd":5,"last update":"Mon Feb 5 06:56:41 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.71899999999999997,"15min":1.02},"min":{"1min":0.376,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.753,"5min":5.4320000000000004,"15min":36.548999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.73899999999999999,"15min":0.99399999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.28699999999999998},"max":{"1min":0.81000000000000005,"5min":10.345000000000001,"15min":36.433},"last":0.53300000000000003}]},{"osd":7,"last update":"Mon Feb 5 06:56:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.64600000000000002,"15min":0.88},"min":{"1min":0.42499999999999999,"5min":0.31,"15min":0.29299999999999998},"max":{"1min":0.78700000000000003,"5min":4.1200000000000001,"15min":38.436999999999998},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.83899999999999997,"15min":1.0189999999999999},"min":{"1min":0.44400000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.879,"5min":11.454000000000001,"15min":36.633000000000003},"last":0.60799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575166,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":938544,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":92806608,"statfs":{"total":95995035648,"available":95033966592,"internally_reserved":0,"allocated":827392,"data_stored":406294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":960231877},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:56:39 2024","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":2.4350000000000001,"15min":2.0939999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20899999999999999},"max":{"1min":0.51300000000000001,"5min":93.488,"15min":93.488},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":2.7839999999999998,"15min":2.2000000000000002},"min":{"1min":0.25600000000000001,"5min":0.22600000000000001,"15min":0.19600000000000001},"max":{"1min":0.54900000000000004,"5min":93.689999999999998,"15min":93.689999999999998},"last":0.38400000000000001}]},{"osd":2,"last update":"Mon Feb 5 06:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":2.4300000000000002,"15min":1.9690000000000001},"min":{"1min":0.253,"5min":0.184,"15min":0.184},"max":{"1min":0.57499999999999996,"5min":93.352999999999994,"15min":93.352999999999994},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.39700000000000002,"5min":2.2040000000000002,"15min":1.7350000000000001},"min":{"1min":0.27700000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.55700000000000005,"5min":91.543000000000006,"15min":91.543000000000006},"last":0.34899999999999998}]},{"osd":3,"last update":"Mon Feb 5 06:56:17 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":2.8300000000000001,"15min":2.0800000000000001},"min":{"1min":0.29999999999999999,"5min":0.25900000000000001,"15min":0.23499999999999999},"max":{"1min":0.84099999999999997,"5min":93.459999999999994,"15min":93.459999999999994},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.44,"5min":2.516,"15min":2.097},"min":{"1min":0.31,"5min":0.255,"15min":0.255},"max":{"1min":0.86499999999999999,"5min":93.406000000000006,"15min":93.406000000000006},"last":0.498}]},{"osd":4,"last update":"Mon Feb 5 06:56:45 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":1.222,"15min":1.1699999999999999},"min":{"1min":0.46400000000000002,"5min":0.309,"15min":0.27700000000000002},"max":{"1min":0.80700000000000005,"5min":8.0820000000000007,"15min":8.0820000000000007},"last":0.44},{"interface":"front","average":{"1min":0.66400000000000003,"5min":1.502,"15min":1.4279999999999999},"min":{"1min":0.53400000000000003,"5min":0.32100000000000001,"15min":0.30599999999999999},"max":{"1min":0.84699999999999998,"5min":9.8680000000000003,"15min":10.419},"last":0.60899999999999999}]},{"osd":5,"last update":"Mon Feb 5 06:56:17 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":1.3129999999999999,"15min":1.5049999999999999},"min":{"1min":0.44,"5min":0.34999999999999998,"15min":0.32900000000000001},"max":{"1min":0.92200000000000004,"5min":7.3680000000000003,"15min":43.414999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.60499999999999998,"5min":1.236,"15min":1.244},"min":{"1min":0.46899999999999997,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.90300000000000002,"5min":8.1080000000000005,"15min":8.1669999999999998},"last":0.55200000000000005}]},{"osd":6,"last update":"Mon Feb 5 06:56:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":1.3979999999999999,"15min":1.3959999999999999},"min":{"1min":0.45100000000000001,"5min":0.38500000000000001,"15min":0.34799999999999998},"max":{"1min":0.88900000000000001,"5min":8.1329999999999991,"15min":23.902999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.66400000000000003,"5min":1.5169999999999999,"15min":1.6599999999999999},"min":{"1min":0.48699999999999999,"5min":0.42099999999999999,"15min":0.35599999999999998},"max":{"1min":0.91600000000000004,"5min":7.4059999999999997,"15min":43.475000000000001},"last":0.72799999999999998}]},{"osd":7,"last update":"Mon Feb 5 06:57:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":1.2689999999999999,"15min":1.2949999999999999},"min":{"1min":0.43099999999999999,"5min":0.36499999999999999,"15min":0.32900000000000001},"max":{"1min":0.84299999999999997,"5min":8.1600000000000001,"15min":8.1600000000000001},"last":0.626},{"interface":"front","average":{"1min":0.65200000000000002,"5min":1.52,"15min":1.4690000000000001},"min":{"1min":0.499,"5min":0.41899999999999998,"15min":0.36599999999999999},"max":{"1min":0.89400000000000002,"5min":9.8230000000000004,"15min":10.481},"last":0.65300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738692,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":798624,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":797502,"kb_avail":92946528,"statfs":{"total":95995035648,"available":95177244672,"internally_reserved":0,"allocated":1138688,"data_stored":733167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":816642501},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 5 06:56:33 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":1.7969999999999999,"15min":1.552},"min":{"1min":0.24399999999999999,"5min":0.188,"15min":0.182},"max":{"1min":0.58099999999999996,"5min":38.68,"15min":38.68},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.376,"5min":2.036,"15min":1.7729999999999999},"min":{"1min":0.247,"5min":0.19500000000000001,"15min":0.18099999999999999},"max":{"1min":0.54800000000000004,"5min":38.003,"15min":38.003},"last":0.44500000000000001}]},{"osd":2,"last update":"Mon Feb 5 06:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":1.603,"15min":1.5089999999999999},"min":{"1min":0.30299999999999999,"5min":0.245,"15min":0.20399999999999999},"max":{"1min":0.56399999999999995,"5min":27.312000000000001,"15min":27.312000000000001},"last":0.377},{"interface":"front","average":{"1min":0.443,"5min":1.625,"15min":1.4359999999999999},"min":{"1min":0.32900000000000001,"5min":0.218,"15min":0.20999999999999999},"max":{"1min":0.622,"5min":27.359000000000002,"15min":27.359000000000002},"last":0.39400000000000002}]},{"osd":3,"last update":"Mon Feb 5 06:56:22 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":1.4470000000000001,"15min":1.653},"min":{"1min":0.255,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.63200000000000001,"5min":31.777999999999999,"15min":43.302},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":1.911,"15min":1.835},"min":{"1min":0.26300000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.57499999999999996,"5min":31.716000000000001,"15min":43.651000000000003},"last":0.32700000000000001}]},{"osd":4,"last update":"Mon Feb 5 06:56:43 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":1.0249999999999999,"15min":1.2350000000000001},"min":{"1min":0.34999999999999998,"5min":0.30599999999999999,"15min":0.27200000000000002},"max":{"1min":0.83899999999999997,"5min":9.157,"15min":10.952999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.97899999999999998,"15min":1.1619999999999999},"min":{"1min":0.40999999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.86499999999999999,"5min":5.681,"15min":8.4019999999999992},"last":0.81599999999999995}]},{"osd":5,"last update":"Mon Feb 5 06:56:12 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":1.2270000000000001,"15min":1.4990000000000001},"min":{"1min":0.41099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.88,"5min":7.1239999999999997,"15min":48.841999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.77000000000000002,"5min":1.0940000000000001,"15min":1.5309999999999999},"min":{"1min":0.43099999999999999,"5min":0.28899999999999998,"15min":0.28599999999999998},"max":{"1min":3.6299999999999999,"5min":9.1039999999999992,"15min":48.450000000000003},"last":0.752}]},{"osd":6,"last update":"Mon Feb 5 06:56:33 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":1.0069999999999999,"15min":1.6839999999999999},"min":{"1min":0.47799999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.89900000000000002,"5min":5.71,"15min":136.76300000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.71499999999999997,"5min":1.2949999999999999,"15min":1.944},"min":{"1min":0.46400000000000002,"5min":0.318,"15min":0.318},"max":{"1min":3.6459999999999999,"5min":7.0720000000000001,"15min":135.92400000000001},"last":0.80300000000000005}]},{"osd":7,"last update":"Mon Feb 5 06:56:57 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":1.133,"15min":1.337},"min":{"1min":0.42499999999999999,"5min":0.36399999999999999,"15min":0.33700000000000002},"max":{"1min":0.88500000000000001,"5min":9.25,"15min":10.989000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.96999999999999997,"15min":1.1299999999999999},"min":{"1min":0.45900000000000002,"5min":0.378,"15min":0.378},"max":{"1min":1.0189999999999999,"5min":5.7389999999999999,"15min":6.8470000000000004},"last":0.84199999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411641,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":699976,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":93045176,"statfs":{"total":95995035648,"available":95278260224,"internally_reserved":0,"allocated":1048576,"data_stored":668068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":715717061},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:56:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":1.306,"15min":2.2250000000000001},"min":{"1min":0.22,"5min":0.20599999999999999,"15min":0.17799999999999999},"max":{"1min":0.47199999999999998,"5min":6.5880000000000001,"15min":101.67700000000001},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.36599999999999999,"5min":1.25,"15min":2.1179999999999999},"min":{"1min":0.245,"5min":0.20200000000000001,"15min":0.17699999999999999},"max":{"1min":0.57499999999999996,"5min":7.4130000000000003,"15min":101.27800000000001},"last":0.39300000000000002}]},{"osd":1,"last update":"Mon Feb 5 06:56:03 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":2.113,"15min":1.861},"min":{"1min":0.24099999999999999,"5min":0.19500000000000001,"15min":0.185},"max":{"1min":0.52700000000000002,"5min":36.719000000000001,"15min":36.719000000000001},"last":0.379},{"interface":"front","average":{"1min":0.40000000000000002,"5min":2.0910000000000002,"15min":1.732},"min":{"1min":0.28599999999999998,"5min":0.24099999999999999,"15min":0.20200000000000001},"max":{"1min":0.55700000000000005,"5min":36.271999999999998,"15min":36.271999999999998},"last":0.40999999999999998}]},{"osd":3,"last update":"Mon Feb 5 06:56:32 2024","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":2.21,"15min":2.0459999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.20899999999999999},"max":{"1min":0.70899999999999996,"5min":38.393000000000001,"15min":84.129000000000005},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.41899999999999998,"5min":2.1070000000000002,"15min":1.9350000000000001},"min":{"1min":0.28799999999999998,"5min":0.28399999999999997,"15min":0.214},"max":{"1min":0.626,"5min":35.229999999999997,"15min":84.090999999999994},"last":0.35699999999999998}]},{"osd":4,"last update":"Mon Feb 5 06:56:37 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":1.129,"15min":1.1060000000000001},"min":{"1min":0.44500000000000001,"5min":0.307,"15min":0.27400000000000002},"max":{"1min":0.84199999999999997,"5min":7.726,"15min":12.041},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.97399999999999998,"15min":1.0820000000000001},"min":{"1min":0.432,"5min":0.29699999999999999,"15min":0.26300000000000001},"max":{"1min":5.2939999999999996,"5min":6.0620000000000003,"15min":15.689},"last":0.52100000000000002}]},{"osd":5,"last update":"Mon Feb 5 06:56:15 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":1.8120000000000001,"15min":1.6080000000000001},"min":{"1min":0.42899999999999999,"5min":0.318,"15min":0.314},"max":{"1min":0.81699999999999995,"5min":78.497,"15min":78.497},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":1.889,"15min":1.661},"min":{"1min":0.46500000000000002,"5min":0.36699999999999999,"15min":0.312},"max":{"1min":0.83399999999999996,"5min":78.774000000000001,"15min":78.774000000000001},"last":0.56200000000000006}]},{"osd":6,"last update":"Mon Feb 5 06:56:37 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":1.7889999999999999,"15min":1.4490000000000001},"min":{"1min":0.48999999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.91200000000000003,"5min":53.332000000000001,"15min":53.332000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.67700000000000005,"5min":1.8460000000000001,"15min":1.629},"min":{"1min":0.41799999999999998,"5min":0.33000000000000002,"15min":0.316},"max":{"1min":0.90000000000000002,"5min":53.320999999999998,"15min":53.320999999999998},"last":0.64800000000000002}]},{"osd":7,"last update":"Mon Feb 5 06:56:03 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":1.3009999999999999,"15min":1.506},"min":{"1min":0.42199999999999999,"5min":0.41699999999999998,"15min":0.34100000000000003},"max":{"1min":0.81799999999999995,"5min":9.3800000000000008,"15min":45.243000000000002},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.66900000000000004,"5min":1.1579999999999999,"15min":1.496},"min":{"1min":0.47899999999999998,"5min":0.39100000000000001,"15min":0.36799999999999999},"max":{"1min":0.90400000000000003,"5min":6.1870000000000003,"15min":45.171999999999997},"last":0.54000000000000004}]}]},{"osd":3,"up_from":26,"seq":111669150004,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":889116,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":92856036,"statfs":{"total":95995035648,"available":95084580864,"internally_reserved":0,"allocated":1396736,"data_stored":995211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":909048263},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:56:31 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":1.103,"15min":1.605},"min":{"1min":0.23499999999999999,"5min":0.20200000000000001,"15min":0.184},"max":{"1min":0.56499999999999995,"5min":9.8849999999999998,"15min":53.656999999999996},"last":0.38},{"interface":"front","average":{"1min":0.379,"5min":1.2450000000000001,"15min":1.806},"min":{"1min":0.27200000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.58499999999999996,"5min":12.792999999999999,"15min":53.338000000000001},"last":0.48399999999999999}]},{"osd":1,"last update":"Mon Feb 5 06:56:31 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":1.9239999999999999,"15min":1.7230000000000001},"min":{"1min":0.33600000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.51200000000000001,"5min":24.343,"15min":24.343},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.375,"5min":1.5660000000000001,"15min":1.5960000000000001},"min":{"1min":0.23699999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.45900000000000002,"5min":23.300999999999998,"15min":23.300999999999998},"last":0.50700000000000001}]},{"osd":2,"last update":"Mon Feb 5 06:56:31 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":1.208,"15min":1.5129999999999999},"min":{"1min":0.30399999999999999,"5min":0.23499999999999999,"15min":0.23000000000000001},"max":{"1min":0.60399999999999998,"5min":6.1689999999999996,"15min":14.746},"last":0.433},{"interface":"front","average":{"1min":0.40200000000000002,"5min":1.218,"15min":1.367},"min":{"1min":0.29499999999999998,"5min":0.26500000000000001,"15min":0.23200000000000001},"max":{"1min":0.60999999999999999,"5min":10.220000000000001,"15min":14.734},"last":0.45600000000000002}]},{"osd":4,"last update":"Mon Feb 5 06:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":1.087,"15min":1.194},"min":{"1min":0.44500000000000001,"5min":0.38100000000000001,"15min":0.29799999999999999},"max":{"1min":0.79700000000000004,"5min":10.177,"15min":16.59},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.59499999999999997,"5min":1.1639999999999999,"15min":1.421},"min":{"1min":0.45800000000000002,"5min":0.29299999999999998,"15min":0.28599999999999998},"max":{"1min":0.82799999999999996,"5min":12.103999999999999,"15min":19.359999999999999},"last":0.64400000000000002}]},{"osd":5,"last update":"Mon Feb 5 06:56:05 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":1.1819999999999999,"15min":1.371},"min":{"1min":0.45500000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.83499999999999996,"5min":11.458,"15min":20.257999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.66100000000000003,"5min":1.4359999999999999,"15min":1.3340000000000001},"min":{"1min":0.44900000000000001,"5min":0.36899999999999999,"15min":0.33700000000000002},"max":{"1min":0.90700000000000003,"5min":16.629999999999999,"15min":16.629999999999999},"last":0.52900000000000003}]},{"osd":6,"last update":"Mon Feb 5 06:56:48 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":1.2869999999999999,"15min":1.4730000000000001},"min":{"1min":0.45700000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.81100000000000005,"5min":12.157999999999999,"15min":16.643000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":1.157,"15min":1.383},"min":{"1min":0.53300000000000003,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.84499999999999997,"5min":5.8609999999999998,"15min":12.75},"last":0.54800000000000004}]},{"osd":7,"last update":"Mon Feb 5 06:56:05 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":1.5169999999999999,"15min":1.4259999999999999},"min":{"1min":0.46600000000000003,"5min":0.41799999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":11.824,"15min":11.824},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.69999999999999996,"5min":1.3360000000000001,"15min":1.4259999999999999},"min":{"1min":0.52000000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.85699999999999998,"5min":16.611999999999998,"15min":16.611999999999998},"last":0.74199999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986478,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":785816,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":785022,"kb_avail":92959336,"statfs":{"total":95995035648,"available":95190360064,"internally_reserved":0,"allocated":802816,"data_stored":405065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":803862981},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:56:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":1.1850000000000001,"15min":1.1080000000000001},"min":{"1min":0.32500000000000001,"5min":0.22900000000000001,"15min":0.20799999999999999},"max":{"1min":0.74399999999999999,"5min":19.385999999999999,"15min":19.385999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.90000000000000002,"15min":1.0800000000000001},"min":{"1min":0.30499999999999999,"5min":0.25700000000000001,"15min":0.22700000000000001},"max":{"1min":0.83799999999999997,"5min":10.130000000000001,"15min":10.913},"last":0.63100000000000001}]},{"osd":1,"last update":"Mon Feb 5 06:56:46 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":1.663,"15min":1.587},"min":{"1min":0.36799999999999999,"5min":0.26900000000000002,"15min":0.21299999999999999},"max":{"1min":0.73199999999999998,"5min":48.360999999999997,"15min":53.414000000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.58399999999999996,"5min":1.532,"15min":1.474},"min":{"1min":0.34999999999999998,"5min":0.23999999999999999,"15min":0.23699999999999999},"max":{"1min":0.86399999999999999,"5min":48.670000000000002,"15min":53.719999999999999},"last":0.57299999999999995}]},{"osd":2,"last update":"Mon Feb 5 06:56:46 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":1.2769999999999999,"15min":1.127},"min":{"1min":0.42699999999999999,"5min":0.29399999999999998,"15min":0.26000000000000001},"max":{"1min":0.78600000000000003,"5min":19.355,"15min":19.355},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":1.056,"15min":1.1799999999999999},"min":{"1min":0.313,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.84399999999999997,"5min":4.7830000000000004,"15min":18.077000000000002},"last":0.52300000000000002}]},{"osd":3,"last update":"Mon Feb 5 06:56:46 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":1.4219999999999999,"15min":1.746},"min":{"1min":0.36599999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.80500000000000005,"5min":54.228999999999999,"15min":54.228999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.55700000000000005,"5min":1.585,"15min":1.706},"min":{"1min":0.38900000000000001,"5min":0.307,"15min":0.29799999999999999},"max":{"1min":0.70399999999999996,"5min":47.670999999999999,"15min":47.670999999999999},"last":0.61299999999999999}]},{"osd":5,"last update":"Mon Feb 5 06:56:06 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.59899999999999998,"15min":0.80900000000000005},"min":{"1min":0.33300000000000002,"5min":0.25,"15min":0.249},"max":{"1min":0.68100000000000005,"5min":2.677,"15min":5.6059999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.67600000000000005,"15min":0.79800000000000004},"min":{"1min":0.29099999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.78400000000000003,"5min":4.242,"15min":7.7290000000000001},"last":0.42199999999999999}]},{"osd":6,"last update":"Mon Feb 5 06:56:46 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.74299999999999999,"15min":0.92300000000000004},"min":{"1min":0.34899999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.89600000000000002,"5min":3.5790000000000002,"15min":10.039999999999999},"last":0.505},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.68200000000000005,"15min":0.85099999999999998},"min":{"1min":0.38800000000000001,"5min":0.312,"15min":0.29499999999999998},"max":{"1min":0.81999999999999995,"5min":3.0739999999999998,"15min":10.074},"last":0.46600000000000003}]},{"osd":7,"last update":"Mon Feb 5 06:56:51 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.66800000000000004,"15min":0.77700000000000002},"min":{"1min":0.42199999999999999,"5min":0.32200000000000001,"15min":0.27400000000000002},"max":{"1min":0.75600000000000001,"5min":3.9980000000000002,"15min":4.1369999999999996},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.627,"5min":0.745,"15min":0.84899999999999998},"min":{"1min":0.42499999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.91600000000000004,"5min":5.4809999999999999,"15min":5.4809999999999999},"last":0.44900000000000001}]}]},{"osd":5,"up_from":40,"seq":171798692137,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":884616,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":883582,"kb_avail":92860536,"statfs":{"total":95995035648,"available":95089188864,"internally_reserved":0,"allocated":1048576,"data_stored":667962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":904788422},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 5 06:57:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.82799999999999996,"15min":1.3},"min":{"1min":0.34899999999999998,"5min":0.255,"15min":0.23100000000000001},"max":{"1min":0.73599999999999999,"5min":3.6110000000000002,"15min":22.326000000000001},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.88500000000000001,"15min":1.2609999999999999},"min":{"1min":0.41699999999999998,"5min":0.27100000000000002,"15min":0.22600000000000001},"max":{"1min":0.82799999999999996,"5min":3.9489999999999998,"15min":22.247},"last":0.61299999999999999}]},{"osd":1,"last update":"Mon Feb 5 06:57:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":1.9630000000000001,"15min":2.5870000000000002},"min":{"1min":0.33200000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.80800000000000005,"5min":77.811999999999998,"15min":317.53399999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":1.665,"15min":2.5179999999999998},"min":{"1min":0.33300000000000002,"5min":0.253,"15min":0.22},"max":{"1min":0.82299999999999995,"5min":77.373999999999995,"15min":317.80500000000001},"last":0.54200000000000004}]},{"osd":2,"last update":"Mon Feb 5 06:57:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":1.2649999999999999,"15min":1.254},"min":{"1min":0.35799999999999998,"5min":0.27200000000000002,"15min":0.23000000000000001},"max":{"1min":0.81299999999999994,"5min":22.635999999999999,"15min":22.635999999999999},"last":0.439},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.97199999999999998,"15min":1.0600000000000001},"min":{"1min":0.40999999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.82899999999999996,"5min":9.952,"15min":9.952},"last":0.48499999999999999}]},{"osd":3,"last update":"Mon Feb 5 06:57:01 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":1.968,"15min":1.7549999999999999},"min":{"1min":0.40300000000000002,"5min":0.30099999999999999,"15min":0.29899999999999999},"max":{"1min":0.79300000000000004,"5min":43.442999999999998,"15min":62.508000000000003},"last":0.63},{"interface":"front","average":{"1min":0.63100000000000001,"5min":1.819,"15min":1.768},"min":{"1min":0.42599999999999999,"5min":0.31,"15min":0.28100000000000003},"max":{"1min":0.81599999999999995,"5min":43.545000000000002,"15min":62.460000000000001},"last":0.58099999999999996}]},{"osd":4,"last update":"Mon Feb 5 06:57:01 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.60699999999999998,"15min":0.78600000000000003},"min":{"1min":0.38,"5min":0.25700000000000001,"15min":0.24299999999999999},"max":{"1min":0.93799999999999994,"5min":2.9399999999999999,"15min":9.3629999999999995},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.66700000000000004,"15min":0.80900000000000005},"min":{"1min":0.314,"5min":0.25600000000000001,"15min":0.252},"max":{"1min":1.2250000000000001,"5min":3.4849999999999999,"15min":9.1839999999999993},"last":0.50900000000000001}]},{"osd":6,"last update":"Mon Feb 5 06:56:36 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":1.369,"15min":1.1559999999999999},"min":{"1min":0.36699999999999999,"5min":0.27700000000000002,"15min":0.23799999999999999},"max":{"1min":0.745,"5min":68.316999999999993,"15min":68.316999999999993},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":1.462,"15min":1.224},"min":{"1min":0.499,"5min":0.30599999999999999,"15min":0.29899999999999999},"max":{"1min":0.71499999999999997,"5min":70.352000000000004,"15min":70.352000000000004},"last":0.59399999999999997}]},{"osd":7,"last update":"Mon Feb 5 06:57:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.75600000000000001,"15min":0.97399999999999998},"min":{"1min":0.374,"5min":0.317,"15min":0.25600000000000001},"max":{"1min":0.878,"5min":4.4130000000000003,"15min":11.691000000000001},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.67300000000000004,"15min":0.91600000000000004},"min":{"1min":0.38500000000000001,"5min":0.33200000000000002,"15min":0.27100000000000002},"max":{"1min":1.2470000000000001,"5min":2.8250000000000002,"15min":6.5839999999999996},"last":0.65100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-05T06:57:05.486 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-02-05T06:57:05.486 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921 shell --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-02-05T06:57:05.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:05 smithi028 ceph-mon[37984]: osdmap e510: 8 total, 8 up, 8 in 2024-02-05T06:57:05.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:05 smithi028 ceph-mon[37984]: pgmap v1118: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:57:05.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:05 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:57:05.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:05 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:57:05.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:05 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:57:05.697 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:05 smithi028 ceph-mon[37984]: from='client.16473 v1:172.21.15.28:0/2037145528' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:57:05.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:05 smithi028 ceph-mon[29442]: osdmap e510: 8 total, 8 up, 8 in 2024-02-05T06:57:05.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:05 smithi028 ceph-mon[29442]: pgmap v1118: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:57:05.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:05 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:57:05.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:05 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:57:05.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:05 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:57:05.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:05 smithi028 ceph-mon[29442]: from='client.16473 v1:172.21.15.28:0/2037145528' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:57:05.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:05 smithi167 ceph-mon[32774]: osdmap e510: 8 total, 8 up, 8 in 2024-02-05T06:57:05.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:05 smithi167 ceph-mon[32774]: pgmap v1118: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:57:05.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:05 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:57:05.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:05 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:57:05.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:05 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:57:05.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:05 smithi167 ceph-mon[32774]: from='client.16473 v1:172.21.15.28:0/2037145528' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-05T06:57:05.734 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/mon.a/config 2024-02-05T06:57:06.902 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:06 smithi028 ceph-mon[37984]: 4.1e scrub starts 2024-02-05T06:57:06.902 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:06 smithi028 ceph-mon[37984]: 4.1e scrub ok 2024-02-05T06:57:06.902 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:06 smithi028 ceph-mon[29442]: 4.1e scrub starts 2024-02-05T06:57:06.902 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:06 smithi028 ceph-mon[29442]: 4.1e scrub ok 2024-02-05T06:57:06.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:06 smithi167 ceph-mon[32774]: 4.1e scrub starts 2024-02-05T06:57:06.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:06 smithi167 ceph-mon[32774]: 4.1e scrub ok 2024-02-05T06:57:07.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:07 smithi028 ceph-mon[29442]: pgmap v1119: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:57:07.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:07 smithi028 ceph-mon[29442]: 2.0 scrub starts 2024-02-05T06:57:07.947 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:07 smithi028 ceph-mon[29442]: 2.0 scrub ok 2024-02-05T06:57:07.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:07 smithi028 ceph-mon[37984]: pgmap v1119: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:57:07.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:07 smithi028 ceph-mon[37984]: 2.0 scrub starts 2024-02-05T06:57:07.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:07 smithi028 ceph-mon[37984]: 2.0 scrub ok 2024-02-05T06:57:07.948 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:57:07 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:57:07] ENGINE Bus STOPPING 2024-02-05T06:57:07.948 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:57:07 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:57:07] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-05T06:57:07.948 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:57:07 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:57:07] ENGINE Bus STOPPED 2024-02-05T06:57:07.948 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:57:07 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:57:07] ENGINE Bus STARTING 2024-02-05T06:57:07.948 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:57:07 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:57:07] ENGINE Serving on http://:::9283 2024-02-05T06:57:07.948 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:57:07 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:57:07] ENGINE Bus STARTED 2024-02-05T06:57:07.981 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:07 smithi167 ceph-mon[32774]: pgmap v1119: 152 pgs: 1 active+clean+scrubbing, 151 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-02-05T06:57:07.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:07 smithi167 ceph-mon[32774]: 2.0 scrub starts 2024-02-05T06:57:07.982 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:07 smithi167 ceph-mon[32774]: 2.0 scrub ok 2024-02-05T06:57:08.380 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-05T06:57:08.390 INFO:tasks.cephadm:Teardown begin 2024-02-05T06:57:08.391 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-05T06:57:08.441 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-05T06:57:08.563 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-05T06:57:08.564 DEBUG:teuthology.orchestra.run.smithi028:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-05T06:57:08.601 DEBUG:teuthology.orchestra.run.smithi167:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-05T06:57:08.627 INFO:tasks.cephadm:Stopping all daemons... 2024-02-05T06:57:08.627 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-02-05T06:57:08.627 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-9801e956-c3ef-11ee-95b6-87774f69a715@mon.a 2024-02-05T06:57:08.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:08 smithi167 ceph-mon[32774]: from='client.? v1:172.21.15.28:0/326099904' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-05T06:57:08.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:08 smithi167 ceph-mon[32774]: from='client.46367 ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-05T06:57:08.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:08 smithi167 ceph-mon[32774]: from='client.46367 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-02-05T06:57:08.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:08 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:57:08.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:08 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:57:08.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:08 smithi167 ceph-mon[32774]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:57:08.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:08 smithi167 ceph-mon[32774]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:57:08.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:08 smithi167 ceph-mon[32774]: 4.19 scrub starts 2024-02-05T06:57:08.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:08 smithi167 ceph-mon[32774]: 4.19 scrub ok 2024-02-05T06:57:08.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:08 smithi167 ceph-mon[32774]: 1.0 scrub starts 2024-02-05T06:57:08.732 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:08 smithi167 ceph-mon[32774]: 1.0 scrub ok 2024-02-05T06:57:08.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:08 smithi028 ceph-mon[37984]: from='client.? v1:172.21.15.28:0/326099904' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-05T06:57:08.947 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:08 smithi028 ceph-mon[37984]: from='client.46367 ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-05T06:57:08.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:08 smithi028 ceph-mon[37984]: from='client.46367 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-02-05T06:57:08.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:08 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:57:08.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:08 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:57:08.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:08 smithi028 ceph-mon[37984]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:57:08.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:08 smithi028 ceph-mon[37984]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:57:08.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:08 smithi028 ceph-mon[37984]: 4.19 scrub starts 2024-02-05T06:57:08.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:08 smithi028 ceph-mon[37984]: 4.19 scrub ok 2024-02-05T06:57:08.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:08 smithi028 ceph-mon[37984]: 1.0 scrub starts 2024-02-05T06:57:08.948 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:08 smithi028 ceph-mon[37984]: 1.0 scrub ok 2024-02-05T06:57:08.948 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:08 smithi028 ceph-mon[29442]: from='client.? v1:172.21.15.28:0/326099904' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-05T06:57:08.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:08 smithi028 ceph-mon[29442]: from='client.46367 ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-02-05T06:57:08.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:08 smithi028 ceph-mon[29442]: from='client.46367 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-02-05T06:57:08.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:08 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-05T06:57:08.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:08 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-05T06:57:08.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:08 smithi028 ceph-mon[29442]: from='mgr.44510 v1:172.21.15.28:0/285255165' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-05T06:57:08.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:08 smithi028 ceph-mon[29442]: from='mgr.44510 ' entity='mgr.y' 2024-02-05T06:57:08.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:08 smithi028 ceph-mon[29442]: 4.19 scrub starts 2024-02-05T06:57:08.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:08 smithi028 ceph-mon[29442]: 4.19 scrub ok 2024-02-05T06:57:08.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:08 smithi028 ceph-mon[29442]: 1.0 scrub starts 2024-02-05T06:57:08.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:08 smithi028 ceph-mon[29442]: 1.0 scrub ok 2024-02-05T06:57:08.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:08 smithi028 systemd[1]: Stopping Ceph mon.a for 9801e956-c3ef-11ee-95b6-87774f69a715... 2024-02-05T06:57:09.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:08 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-a[29438]: 2024-02-05T06:57:08.975+0000 7fa738a9e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T06:57:09.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:08 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-a[29438]: 2024-02-05T06:57:08.975+0000 7fa738a9e700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-02-05T06:57:09.447 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:09 smithi028 podman[206897]: 2024-02-05 06:57:09.191362006 +0000 UTC m=+0.322291534 container died edf115e6318f8233e18189108e4694ccd68c6d1df7ab9fb20a5065836e92af46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, ceph=True, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS) 2024-02-05T06:57:09.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:09 smithi028 podman[206897]: 2024-02-05 06:57:09.544997813 +0000 UTC m=+0.675927347 container cleanup edf115e6318f8233e18189108e4694ccd68c6d1df7ab9fb20a5065836e92af46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-a, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-05T06:57:09.809 INFO:journalctl@ceph.mon.a.smithi028.stdout:Feb 05 06:57:09 smithi028 bash[206897]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-a 2024-02-05T06:57:09.932 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@mon.a.service' 2024-02-05T06:57:09.982 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T06:57:09.983 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-02-05T06:57:09.983 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-02-05T06:57:09.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-9801e956-c3ef-11ee-95b6-87774f69a715@mon.c 2024-02-05T06:57:10.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:10 smithi028 systemd[1]: Stopping Ceph mon.c for 9801e956-c3ef-11ee-95b6-87774f69a715... 2024-02-05T06:57:10.549 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:10 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-c[37980]: 2024-02-05T06:57:10.299+0000 7fd8b2aa4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T06:57:10.549 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:10 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-c[37980]: 2024-02-05T06:57:10.299+0000 7fd8b2aa4700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-02-05T06:57:10.550 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:57:10 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:57:10] ENGINE Bus STOPPING 2024-02-05T06:57:10.550 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:57:10 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:57:10] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-05T06:57:10.550 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:57:10 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:57:10] ENGINE Bus STOPPED 2024-02-05T06:57:10.550 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:57:10 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:57:10] ENGINE Bus STARTING 2024-02-05T06:57:10.550 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:57:10 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:57:10] ENGINE Serving on http://:::9283 2024-02-05T06:57:10.550 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:57:10 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: [05/Feb/2024:06:57:10] ENGINE Bus STARTED 2024-02-05T06:57:10.856 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:10 smithi028 podman[207019]: 2024-02-05 06:57:10.54465259 +0000 UTC m=+0.383397369 container died 47eab7d888a5cf97d1466fb8adf694b7318a6f3a593d41f3558494308e86e76b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-c, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240131) 2024-02-05T06:57:11.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:11 smithi028 podman[207019]: 2024-02-05 06:57:11.000497903 +0000 UTC m=+0.839242690 container cleanup 47eab7d888a5cf97d1466fb8adf694b7318a6f3a593d41f3558494308e86e76b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-c, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-05T06:57:11.197 INFO:journalctl@ceph.mon.c.smithi028.stdout:Feb 05 06:57:11 smithi028 bash[207019]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-c 2024-02-05T06:57:11.526 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@mon.c.service' 2024-02-05T06:57:11.563 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T06:57:11.563 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-02-05T06:57:11.564 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-02-05T06:57:11.564 DEBUG:teuthology.orchestra.run.smithi167:> sudo systemctl stop ceph-9801e956-c3ef-11ee-95b6-87774f69a715@mon.b 2024-02-05T06:57:11.916 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:11 smithi167 systemd[1]: Stopping Ceph mon.b for 9801e956-c3ef-11ee-95b6-87774f69a715... 2024-02-05T06:57:12.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:11 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-b[32770]: 2024-02-05T06:57:11.910+0000 7f5824c51700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T06:57:12.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:11 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-b[32770]: 2024-02-05T06:57:11.910+0000 7f5824c51700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-02-05T06:57:12.232 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:12 smithi167 podman[152125]: 2024-02-05 06:57:12.145887966 +0000 UTC m=+0.359820105 container died 7cbbabf936992d710f8e034d38403e33c1c62092063512166f4a96f0df2efce2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-02-05T06:57:12.761 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:57:12 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: ::ffff:172.21.15.167 - - [05/Feb/2024:06:57:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-05T06:57:12.926 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:12 smithi167 podman[152125]: 2024-02-05 06:57:12.628112782 +0000 UTC m=+0.842044922 container cleanup 7cbbabf936992d710f8e034d38403e33c1c62092063512166f4a96f0df2efce2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_CLEAN=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0) 2024-02-05T06:57:12.926 INFO:journalctl@ceph.mon.b.smithi167.stdout:Feb 05 06:57:12 smithi167 bash[152125]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mon-b 2024-02-05T06:57:13.067 DEBUG:teuthology.orchestra.run.smithi167:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@mon.b.service' 2024-02-05T06:57:13.127 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T06:57:13.128 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-02-05T06:57:13.128 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-02-05T06:57:13.128 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-9801e956-c3ef-11ee-95b6-87774f69a715@mgr.y 2024-02-05T06:57:13.445 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:57:13 smithi028 systemd[1]: Stopping Ceph mgr.y for 9801e956-c3ef-11ee-95b6-87774f69a715... 2024-02-05T06:57:13.697 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:57:13 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:57:13.438+0000 7f656c325700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T06:57:13.697 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:57:13 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y[29675]: 2024-02-05T06:57:13.438+0000 7f656c325700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-02-05T06:57:13.698 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:57:13 smithi028 podman[207165]: 2024-02-05 06:57:13.471126798 +0000 UTC m=+0.162253985 container died c706bd03e503cf878164bd837c9a67fd401c9a94b2327f9e48f4eebcc8042e1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-02-05T06:57:14.104 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:57:13 smithi028 podman[207165]: 2024-02-05 06:57:13.890369659 +0000 UTC m=+0.581496854 container cleanup c706bd03e503cf878164bd837c9a67fd401c9a94b2327f9e48f4eebcc8042e1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:164d3be4a527be775f4d3c343ea591122f1b8921, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-02-05T06:57:14.104 INFO:journalctl@ceph.mgr.y.smithi028.stdout:Feb 05 06:57:13 smithi028 bash[207165]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-y 2024-02-05T06:57:14.244 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@mgr.y.service' 2024-02-05T06:57:14.282 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T06:57:14.283 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-02-05T06:57:14.283 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-02-05T06:57:14.283 DEBUG:teuthology.orchestra.run.smithi167:> sudo systemctl stop ceph-9801e956-c3ef-11ee-95b6-87774f69a715@mgr.x 2024-02-05T06:57:14.602 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:57:14 smithi167 systemd[1]: Stopping Ceph mgr.x for 9801e956-c3ef-11ee-95b6-87774f69a715... 2024-02-05T06:57:14.856 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:57:14 smithi167 podman[152259]: 2024-02-05 06:57:14.597435935 +0000 UTC m=+0.126452618 container died 06698b169813f574c848694bbc169f8c27760194bb96bc4942224fc5feb3147c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, ceph=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True) 2024-02-05T06:57:15.232 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:57:14 smithi167 podman[152259]: 2024-02-05 06:57:14.966963096 +0000 UTC m=+0.495979788 container cleanup 06698b169813f574c848694bbc169f8c27760194bb96bc4942224fc5feb3147c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be) 2024-02-05T06:57:15.232 INFO:journalctl@ceph.mgr.x.smithi167.stdout:Feb 05 06:57:14 smithi167 bash[152259]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715-mgr-x 2024-02-05T06:57:15.440 DEBUG:teuthology.orchestra.run.smithi167:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@mgr.x.service' 2024-02-05T06:57:15.478 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T06:57:15.478 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-02-05T06:57:15.478 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-02-05T06:57:15.478 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-9801e956-c3ef-11ee-95b6-87774f69a715@osd.0 2024-02-05T06:57:15.774 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 05 06:57:15 smithi028 systemd[1]: Stopping Ceph osd.0 for 9801e956-c3ef-11ee-95b6-87774f69a715... 2024-02-05T06:57:16.197 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 05 06:57:15 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-0[42047]: 2024-02-05T06:57:15.768+0000 7f6d2a457700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T06:57:16.197 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 05 06:57:15 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-0[42047]: 2024-02-05T06:57:15.768+0000 7f6d2a457700 -1 osd.0 510 *** Got signal Terminated *** 2024-02-05T06:57:16.197 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 05 06:57:15 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-0[42047]: 2024-02-05T06:57:15.768+0000 7f6d2a457700 -1 osd.0 510 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T06:57:17.232 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:57:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:57:16.861Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.28:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.28:8765: connect: connection refused" 2024-02-05T06:57:17.232 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:57:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:57:16.862Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.28:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.28:8765: connect: connection refused" 2024-02-05T06:57:17.232 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:57:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:57:16.865Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.28:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.28:8765: connect: connection refused" 2024-02-05T06:57:17.232 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:57:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:57:16.867Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.28:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.28:8765: connect: connection refused" 2024-02-05T06:57:21.617 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 05 06:57:21 smithi028 podman[207285]: 2024-02-05 06:57:21.341744785 +0000 UTC m=+5.671940342 container died f146066eb44666678e5752f7f7831223537ed7b07c7165bf9743ce77613f8205 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-0, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be) 2024-02-05T06:57:21.947 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 05 06:57:21 smithi028 podman[207285]: 2024-02-05 06:57:21.820250114 +0000 UTC m=+6.150445668 container cleanup f146066eb44666678e5752f7f7831223537ed7b07c7165bf9743ce77613f8205 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240131) 2024-02-05T06:57:21.947 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 05 06:57:21 smithi028 bash[207285]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-0 2024-02-05T06:57:22.403 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 05 06:57:22 smithi028 podman[207304]: 2024-02-05 06:57:22.088803275 +0000 UTC m=+0.738361661 container remove f146066eb44666678e5752f7f7831223537ed7b07c7165bf9743ce77613f8205 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-0, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0) 2024-02-05T06:57:22.697 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 05 06:57:22 smithi028 podman[207388]: 2024-02-05 06:57:22.301900613 +0000 UTC m=+0.018000375 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T06:57:22.697 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 05 06:57:22 smithi028 podman[207388]: 2024-02-05 06:57:22.514915288 +0000 UTC m=+0.231015093 container create 4969de19566b553fe3ef42a2fafe047478eb0817cd310843555b6b8708248c06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-05T06:57:23.197 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 05 06:57:22 smithi028 podman[207388]: 2024-02-05 06:57:22.746270422 +0000 UTC m=+0.462370182 container init 4969de19566b553fe3ef42a2fafe047478eb0817cd310843555b6b8708248c06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-05T06:57:23.197 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 05 06:57:22 smithi028 podman[207388]: 2024-02-05 06:57:22.749696037 +0000 UTC m=+0.465795803 container start 4969de19566b553fe3ef42a2fafe047478eb0817cd310843555b6b8708248c06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-05T06:57:23.197 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 05 06:57:22 smithi028 podman[207388]: 2024-02-05 06:57:22.892581846 +0000 UTC m=+0.608681618 container attach 4969de19566b553fe3ef42a2fafe047478eb0817cd310843555b6b8708248c06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-05T06:57:24.333 INFO:journalctl@ceph.osd.0.smithi028.stdout:Feb 05 06:57:24 smithi028 podman[207388]: 2024-02-05 06:57:24.031708442 +0000 UTC m=+1.747808208 container died 4969de19566b553fe3ef42a2fafe047478eb0817cd310843555b6b8708248c06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.build-date=20240131) 2024-02-05T06:57:24.742 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@osd.0.service' 2024-02-05T06:57:24.783 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T06:57:24.783 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-02-05T06:57:24.783 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-02-05T06:57:24.783 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-9801e956-c3ef-11ee-95b6-87774f69a715@osd.1 2024-02-05T06:57:24.990 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 05 06:57:24 smithi028 systemd[1]: Stopping Ceph osd.1 for 9801e956-c3ef-11ee-95b6-87774f69a715... 2024-02-05T06:57:25.447 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 05 06:57:25 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-1[48053]: 2024-02-05T06:57:25.149+0000 7f292529a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T06:57:25.447 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 05 06:57:25 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-1[48053]: 2024-02-05T06:57:25.149+0000 7f292529a700 -1 osd.1 510 *** Got signal Terminated *** 2024-02-05T06:57:25.447 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 05 06:57:25 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-1[48053]: 2024-02-05T06:57:25.149+0000 7f292529a700 -1 osd.1 510 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T06:57:30.984 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 05 06:57:30 smithi028 podman[207507]: 2024-02-05 06:57:30.695111626 +0000 UTC m=+5.677465493 container died dd6ed489ce49a92d1bf206add3b4e8e9730e86a64ec6a90fc2008bd538b06980 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-1, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-02-05T06:57:31.430 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 05 06:57:31 smithi028 podman[207507]: 2024-02-05 06:57:31.158435436 +0000 UTC m=+6.140789303 container cleanup dd6ed489ce49a92d1bf206add3b4e8e9730e86a64ec6a90fc2008bd538b06980 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-1, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=) 2024-02-05T06:57:31.431 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 05 06:57:31 smithi028 bash[207507]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-1 2024-02-05T06:57:31.697 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 05 06:57:31 smithi028 podman[207524]: 2024-02-05 06:57:31.426632364 +0000 UTC m=+0.721223127 container remove dd6ed489ce49a92d1bf206add3b4e8e9730e86a64ec6a90fc2008bd538b06980 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-1, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-02-05T06:57:31.984 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 05 06:57:31 smithi028 podman[207620]: 2024-02-05 06:57:31.627128379 +0000 UTC m=+0.018777772 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T06:57:31.984 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 05 06:57:31 smithi028 podman[207620]: 2024-02-05 06:57:31.735757873 +0000 UTC m=+0.127407269 container create 9a372693ac4d11082face8cada58f1a13852477a9f28f95059210eec48f1b0c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0) 2024-02-05T06:57:32.290 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 05 06:57:31 smithi028 podman[207620]: 2024-02-05 06:57:31.98020235 +0000 UTC m=+0.371851743 container init 9a372693ac4d11082face8cada58f1a13852477a9f28f95059210eec48f1b0c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-02-05T06:57:32.291 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 05 06:57:31 smithi028 podman[207620]: 2024-02-05 06:57:31.983327509 +0000 UTC m=+0.374976904 container start 9a372693ac4d11082face8cada58f1a13852477a9f28f95059210eec48f1b0c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0) 2024-02-05T06:57:32.291 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 05 06:57:32 smithi028 podman[207620]: 2024-02-05 06:57:32.055227994 +0000 UTC m=+0.446877416 container attach 9a372693ac4d11082face8cada58f1a13852477a9f28f95059210eec48f1b0c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux ) 2024-02-05T06:57:32.697 INFO:journalctl@ceph.osd.1.smithi028.stdout:Feb 05 06:57:32 smithi028 podman[207620]: 2024-02-05 06:57:32.286245626 +0000 UTC m=+0.677895027 container died 9a372693ac4d11082face8cada58f1a13852477a9f28f95059210eec48f1b0c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.2) 2024-02-05T06:57:33.073 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@osd.1.service' 2024-02-05T06:57:33.121 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T06:57:33.122 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-02-05T06:57:33.122 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-02-05T06:57:33.122 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-9801e956-c3ef-11ee-95b6-87774f69a715@osd.2 2024-02-05T06:57:33.408 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 05 06:57:33 smithi028 systemd[1]: Stopping Ceph osd.2 for 9801e956-c3ef-11ee-95b6-87774f69a715... 2024-02-05T06:57:33.697 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 05 06:57:33 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-2[53688]: 2024-02-05T06:57:33.402+0000 7fd6d6a85700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T06:57:33.697 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 05 06:57:33 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-2[53688]: 2024-02-05T06:57:33.402+0000 7fd6d6a85700 -1 osd.2 510 *** Got signal Terminated *** 2024-02-05T06:57:33.697 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 05 06:57:33 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-2[53688]: 2024-02-05T06:57:33.402+0000 7fd6d6a85700 -1 osd.2 510 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T06:57:39.343 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 05 06:57:39 smithi028 podman[207726]: 2024-02-05 06:57:39.000870435 +0000 UTC m=+5.701207520 container died 7192552aae2afd6493ff3b3ebd7e00eeb4d0464499c0a8414265e01e15680cfe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-2, org.label-schema.build-date=20240131, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-02-05T06:57:39.697 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 05 06:57:39 smithi028 podman[207726]: 2024-02-05 06:57:39.487434484 +0000 UTC m=+6.187771601 container cleanup 7192552aae2afd6493ff3b3ebd7e00eeb4d0464499c0a8414265e01e15680cfe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-2, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True) 2024-02-05T06:57:39.697 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 05 06:57:39 smithi028 bash[207726]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-2 2024-02-05T06:57:40.109 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 05 06:57:39 smithi028 podman[207743]: 2024-02-05 06:57:39.763963423 +0000 UTC m=+0.762357653 container remove 7192552aae2afd6493ff3b3ebd7e00eeb4d0464499c0a8414265e01e15680cfe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131) 2024-02-05T06:57:40.396 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 05 06:57:40 smithi028 podman[207834]: 2024-02-05 06:57:40.008445297 +0000 UTC m=+0.018695413 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T06:57:40.396 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 05 06:57:40 smithi028 podman[207834]: 2024-02-05 06:57:40.114825287 +0000 UTC m=+0.125075442 container create 66a38984b47306faeb512de3598b7962acb9620033305ed6dd1e35c263c8d1a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-02-05T06:57:40.396 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 05 06:57:40 smithi028 podman[207834]: 2024-02-05 06:57:40.308758514 +0000 UTC m=+0.319008628 container init 66a38984b47306faeb512de3598b7962acb9620033305ed6dd1e35c263c8d1a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-02-05T06:57:40.396 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 05 06:57:40 smithi028 podman[207834]: 2024-02-05 06:57:40.311768317 +0000 UTC m=+0.322018431 container start 66a38984b47306faeb512de3598b7962acb9620033305ed6dd1e35c263c8d1a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-2-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-02-05T06:57:40.397 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 05 06:57:40 smithi028 podman[207834]: 2024-02-05 06:57:40.392077232 +0000 UTC m=+0.402327359 container attach 66a38984b47306faeb512de3598b7962acb9620033305ed6dd1e35c263c8d1a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-05T06:57:40.697 INFO:journalctl@ceph.osd.2.smithi028.stdout:Feb 05 06:57:40 smithi028 podman[207834]: 2024-02-05 06:57:40.619238739 +0000 UTC m=+0.629488852 container died 66a38984b47306faeb512de3598b7962acb9620033305ed6dd1e35c263c8d1a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-02-05T06:57:41.192 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@osd.2.service' 2024-02-05T06:57:41.236 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T06:57:41.236 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-02-05T06:57:41.236 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-02-05T06:57:41.237 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-9801e956-c3ef-11ee-95b6-87774f69a715@osd.3 2024-02-05T06:57:41.446 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 05 06:57:41 smithi028 systemd[1]: Stopping Ceph osd.3 for 9801e956-c3ef-11ee-95b6-87774f69a715... 2024-02-05T06:57:41.947 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 05 06:57:41 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-3[59489]: 2024-02-05T06:57:41.564+0000 7f4bbaf89700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T06:57:41.947 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 05 06:57:41 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-3[59489]: 2024-02-05T06:57:41.564+0000 7f4bbaf89700 -1 osd.3 510 *** Got signal Terminated *** 2024-02-05T06:57:41.947 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 05 06:57:41 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-3[59489]: 2024-02-05T06:57:41.564+0000 7f4bbaf89700 -1 osd.3 510 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T06:57:43.164 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:42 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 2024-02-05T06:57:42.861+0000 7fd6791d7700 -1 osd.4 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:15.345786+0000 front 2024-02-05T06:57:15.345773+0000 (oldest deadline 2024-02-05T06:57:42.346240+0000) 2024-02-05T06:57:43.481 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:57:43 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:57:43.159+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:17.804342+0000 front 2024-02-05T06:57:17.804241+0000 (oldest deadline 2024-02-05T06:57:43.104006+0000) 2024-02-05T06:57:44.181 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:43 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 2024-02-05T06:57:43.902+0000 7fd6791d7700 -1 osd.4 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:15.345786+0000 front 2024-02-05T06:57:15.345773+0000 (oldest deadline 2024-02-05T06:57:42.346240+0000) 2024-02-05T06:57:44.181 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:57:44 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:57:44.070+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:57:44.481 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:57:44 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:57:44.203+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:17.804342+0000 front 2024-02-05T06:57:17.804241+0000 (oldest deadline 2024-02-05T06:57:43.104006+0000) 2024-02-05T06:57:44.947 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 05 06:57:44 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-3[59489]: 2024-02-05T06:57:44.616+0000 7f4bb7584700 -1 osd.3 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.933266+0000 front 2024-02-05T06:57:18.933212+0000 (oldest deadline 2024-02-05T06:57:43.633373+0000) 2024-02-05T06:57:45.231 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:44 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 2024-02-05T06:57:44.924+0000 7fd6791d7700 -1 osd.4 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:15.345786+0000 front 2024-02-05T06:57:15.345773+0000 (oldest deadline 2024-02-05T06:57:42.346240+0000) 2024-02-05T06:57:45.232 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:57:45 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:57:45.056+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:57:45.731 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:57:45 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:57:45.249+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:17.804342+0000 front 2024-02-05T06:57:17.804241+0000 (oldest deadline 2024-02-05T06:57:43.104006+0000) 2024-02-05T06:57:45.947 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 05 06:57:45 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-3[59489]: 2024-02-05T06:57:45.634+0000 7f4bb7584700 -1 osd.3 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.933266+0000 front 2024-02-05T06:57:18.933212+0000 (oldest deadline 2024-02-05T06:57:43.633373+0000) 2024-02-05T06:57:46.227 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:45 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 2024-02-05T06:57:45.955+0000 7fd6791d7700 -1 osd.4 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:15.345786+0000 front 2024-02-05T06:57:15.345773+0000 (oldest deadline 2024-02-05T06:57:42.346240+0000) 2024-02-05T06:57:46.227 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:57:46 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:57:46.087+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:57:46.481 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:57:46 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:57:46.222+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:17.804342+0000 front 2024-02-05T06:57:17.804241+0000 (oldest deadline 2024-02-05T06:57:43.104006+0000) 2024-02-05T06:57:47.206 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:46 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 2024-02-05T06:57:46.917+0000 7fd6791d7700 -1 osd.4 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:15.345786+0000 front 2024-02-05T06:57:15.345773+0000 (oldest deadline 2024-02-05T06:57:42.346240+0000) 2024-02-05T06:57:47.206 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:57:47 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:57:47.076+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:57:47.207 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:57:46 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:57:46.935+0000 7f56bef57700 -1 osd.5 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:20.660664+0000 front 2024-02-05T06:57:20.660894+0000 (oldest deadline 2024-02-05T06:57:45.960611+0000) 2024-02-05T06:57:47.481 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:57:47 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:57:47.201+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:17.804342+0000 front 2024-02-05T06:57:17.804241+0000 (oldest deadline 2024-02-05T06:57:43.104006+0000) 2024-02-05T06:57:47.482 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 05 06:57:47 smithi028 podman[207938]: 2024-02-05 06:57:47.195236997 +0000 UTC m=+5.726659048 container died 6ff8ca84d6bf9e5fe383ae999448b58e8d4ad1ffcd515c9fde6e7cfec5a7d695 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-3, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_CLEAN=True) 2024-02-05T06:57:47.937 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 05 06:57:47 smithi028 podman[207938]: 2024-02-05 06:57:47.640052755 +0000 UTC m=+6.171474805 container cleanup 6ff8ca84d6bf9e5fe383ae999448b58e8d4ad1ffcd515c9fde6e7cfec5a7d695 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-3, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240131, ceph=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-02-05T06:57:47.937 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 05 06:57:47 smithi028 bash[207938]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-3 2024-02-05T06:57:48.196 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 05 06:57:47 smithi028 podman[207967]: 2024-02-05 06:57:47.933336828 +0000 UTC m=+0.730117867 container remove 6ff8ca84d6bf9e5fe383ae999448b58e8d4ad1ffcd515c9fde6e7cfec5a7d695 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-3, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-05T06:57:48.231 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:47 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 2024-02-05T06:57:47.926+0000 7fd6791d7700 -1 osd.4 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:15.345786+0000 front 2024-02-05T06:57:15.345773+0000 (oldest deadline 2024-02-05T06:57:42.346240+0000) 2024-02-05T06:57:48.231 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:57:47 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:57:47.965+0000 7f56bef57700 -1 osd.5 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:20.660664+0000 front 2024-02-05T06:57:20.660894+0000 (oldest deadline 2024-02-05T06:57:45.960611+0000) 2024-02-05T06:57:48.232 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:57:48 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:57:48.101+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:57:48.502 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 05 06:57:48 smithi028 podman[208057]: 2024-02-05 06:57:48.150132669 +0000 UTC m=+0.018478086 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T06:57:48.502 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 05 06:57:48 smithi028 podman[208057]: 2024-02-05 06:57:48.259081537 +0000 UTC m=+0.127426944 container create 288af95454e4c2eec9a2613ad267520fa378eacfc5a9b20e303521435f0da8eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-3-deactivate, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-02-05T06:57:48.502 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 05 06:57:48 smithi028 podman[208057]: 2024-02-05 06:57:48.494950731 +0000 UTC m=+0.363296139 container init 288af95454e4c2eec9a2613ad267520fa378eacfc5a9b20e303521435f0da8eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True) 2024-02-05T06:57:48.502 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 05 06:57:48 smithi028 podman[208057]: 2024-02-05 06:57:48.497900171 +0000 UTC m=+0.366245580 container start 288af95454e4c2eec9a2613ad267520fa378eacfc5a9b20e303521435f0da8eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS) 2024-02-05T06:57:48.731 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:57:48 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:57:48.231+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:17.804342+0000 front 2024-02-05T06:57:17.804241+0000 (oldest deadline 2024-02-05T06:57:43.104006+0000) 2024-02-05T06:57:48.793 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 05 06:57:48 smithi028 podman[208057]: 2024-02-05 06:57:48.595135727 +0000 UTC m=+0.463481141 container attach 288af95454e4c2eec9a2613ad267520fa378eacfc5a9b20e303521435f0da8eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240131, ceph=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2) 2024-02-05T06:57:49.077 INFO:journalctl@ceph.osd.3.smithi028.stdout:Feb 05 06:57:48 smithi028 podman[208057]: 2024-02-05 06:57:48.789665073 +0000 UTC m=+0.658010491 container died 288af95454e4c2eec9a2613ad267520fa378eacfc5a9b20e303521435f0da8eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, CEPH_POINT_RELEASE=) 2024-02-05T06:57:49.231 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:48 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 2024-02-05T06:57:48.974+0000 7fd6791d7700 -1 osd.4 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:15.345786+0000 front 2024-02-05T06:57:15.345773+0000 (oldest deadline 2024-02-05T06:57:42.346240+0000) 2024-02-05T06:57:49.232 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:57:48 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:57:48.950+0000 7f56bef57700 -1 osd.5 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:20.660664+0000 front 2024-02-05T06:57:20.660894+0000 (oldest deadline 2024-02-05T06:57:45.960611+0000) 2024-02-05T06:57:49.232 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:57:49 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:57:49.120+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:57:49.423 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@osd.3.service' 2024-02-05T06:57:49.468 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T06:57:49.468 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-02-05T06:57:49.468 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-02-05T06:57:49.468 DEBUG:teuthology.orchestra.run.smithi167:> sudo systemctl stop ceph-9801e956-c3ef-11ee-95b6-87774f69a715@osd.4 2024-02-05T06:57:49.731 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:49 smithi167 systemd[1]: Stopping Ceph osd.4 for 9801e956-c3ef-11ee-95b6-87774f69a715... 2024-02-05T06:57:49.732 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:57:49 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:57:49.272+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:17.804342+0000 front 2024-02-05T06:57:17.804241+0000 (oldest deadline 2024-02-05T06:57:43.104006+0000) 2024-02-05T06:57:50.102 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:57:49 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:57:49.940+0000 7f56bef57700 -1 osd.5 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:20.660664+0000 front 2024-02-05T06:57:20.660894+0000 (oldest deadline 2024-02-05T06:57:45.960611+0000) 2024-02-05T06:57:50.102 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:49 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 2024-02-05T06:57:49.777+0000 7fd67cbdc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T06:57:50.102 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:49 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 2024-02-05T06:57:49.777+0000 7fd67cbdc700 -1 osd.4 510 *** Got signal Terminated *** 2024-02-05T06:57:50.102 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:49 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 2024-02-05T06:57:49.777+0000 7fd67cbdc700 -1 osd.4 510 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T06:57:50.102 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:50 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 2024-02-05T06:57:50.000+0000 7fd6791d7700 -1 osd.4 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:15.345786+0000 front 2024-02-05T06:57:15.345773+0000 (oldest deadline 2024-02-05T06:57:42.346240+0000) 2024-02-05T06:57:50.481 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:57:50 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:57:50.257+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:17.804342+0000 front 2024-02-05T06:57:17.804241+0000 (oldest deadline 2024-02-05T06:57:43.104006+0000) 2024-02-05T06:57:50.481 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:57:50 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:57:50.097+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:57:51.231 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:51 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 2024-02-05T06:57:51.028+0000 7fd6791d7700 -1 osd.4 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:15.345786+0000 front 2024-02-05T06:57:15.345773+0000 (oldest deadline 2024-02-05T06:57:42.346240+0000) 2024-02-05T06:57:51.232 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:57:50 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:57:50.906+0000 7f56bef57700 -1 osd.5 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:20.660664+0000 front 2024-02-05T06:57:20.660894+0000 (oldest deadline 2024-02-05T06:57:45.960611+0000) 2024-02-05T06:57:51.232 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:57:51 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:57:51.072+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:57:51.232 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:57:51 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:57:51.072+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:26.865994+0000 front 2024-02-05T06:57:26.866095+0000 (oldest deadline 2024-02-05T06:57:50.866589+0000) 2024-02-05T06:57:51.731 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:57:51 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:57:51.287+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:17.804342+0000 front 2024-02-05T06:57:17.804241+0000 (oldest deadline 2024-02-05T06:57:43.104006+0000) 2024-02-05T06:57:51.731 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:57:51 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:57:51.287+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:29.005152+0000 front 2024-02-05T06:57:29.005232+0000 (oldest deadline 2024-02-05T06:57:50.705036+0000) 2024-02-05T06:57:52.232 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:57:51 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:57:51.951+0000 7f56bef57700 -1 osd.5 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:20.660664+0000 front 2024-02-05T06:57:20.660894+0000 (oldest deadline 2024-02-05T06:57:45.960611+0000) 2024-02-05T06:57:52.232 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:52 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 2024-02-05T06:57:52.066+0000 7fd6791d7700 -1 osd.4 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:15.345786+0000 front 2024-02-05T06:57:15.345773+0000 (oldest deadline 2024-02-05T06:57:42.346240+0000) 2024-02-05T06:57:52.232 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:52 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 2024-02-05T06:57:52.066+0000 7fd6791d7700 -1 osd.4 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:25.247455+0000 front 2024-02-05T06:57:25.247478+0000 (oldest deadline 2024-02-05T06:57:51.147213+0000) 2024-02-05T06:57:52.232 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:57:52 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:57:52.086+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:57:52.232 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:57:52 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:57:52.086+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:26.865994+0000 front 2024-02-05T06:57:26.866095+0000 (oldest deadline 2024-02-05T06:57:50.866589+0000) 2024-02-05T06:57:52.731 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:57:52 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:57:52.240+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:17.804342+0000 front 2024-02-05T06:57:17.804241+0000 (oldest deadline 2024-02-05T06:57:43.104006+0000) 2024-02-05T06:57:52.732 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:57:52 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:57:52.240+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:29.005152+0000 front 2024-02-05T06:57:29.005232+0000 (oldest deadline 2024-02-05T06:57:50.705036+0000) 2024-02-05T06:57:53.268 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:53 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 2024-02-05T06:57:53.068+0000 7fd6791d7700 -1 osd.4 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:15.345786+0000 front 2024-02-05T06:57:15.345773+0000 (oldest deadline 2024-02-05T06:57:42.346240+0000) 2024-02-05T06:57:53.269 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:53 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 2024-02-05T06:57:53.068+0000 7fd6791d7700 -1 osd.4 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:25.247455+0000 front 2024-02-05T06:57:25.247478+0000 (oldest deadline 2024-02-05T06:57:51.147213+0000) 2024-02-05T06:57:53.269 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:57:53 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:57:52.999+0000 7f56bef57700 -1 osd.5 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:20.660664+0000 front 2024-02-05T06:57:20.660894+0000 (oldest deadline 2024-02-05T06:57:45.960611+0000) 2024-02-05T06:57:53.269 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:57:53 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:57:53.049+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:57:53.269 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:57:53 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:57:53.049+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:26.865994+0000 front 2024-02-05T06:57:26.866095+0000 (oldest deadline 2024-02-05T06:57:50.866589+0000) 2024-02-05T06:57:53.731 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:57:53 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:57:53.264+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:17.804342+0000 front 2024-02-05T06:57:17.804241+0000 (oldest deadline 2024-02-05T06:57:43.104006+0000) 2024-02-05T06:57:53.731 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:57:53 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:57:53.264+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:29.005152+0000 front 2024-02-05T06:57:29.005232+0000 (oldest deadline 2024-02-05T06:57:50.705036+0000) 2024-02-05T06:57:54.481 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 2024-02-05T06:57:54.104+0000 7fd6791d7700 -1 osd.4 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:15.345786+0000 front 2024-02-05T06:57:15.345773+0000 (oldest deadline 2024-02-05T06:57:42.346240+0000) 2024-02-05T06:57:54.482 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 2024-02-05T06:57:54.104+0000 7fd6791d7700 -1 osd.4 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:25.247455+0000 front 2024-02-05T06:57:25.247478+0000 (oldest deadline 2024-02-05T06:57:51.147213+0000) 2024-02-05T06:57:54.482 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:57:54.237+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:17.804342+0000 front 2024-02-05T06:57:17.804241+0000 (oldest deadline 2024-02-05T06:57:43.104006+0000) 2024-02-05T06:57:54.482 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:57:54.237+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:29.005152+0000 front 2024-02-05T06:57:29.005232+0000 (oldest deadline 2024-02-05T06:57:50.705036+0000) 2024-02-05T06:57:54.482 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:57:54.039+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:57:54.482 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:57:54.039+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:26.865994+0000 front 2024-02-05T06:57:26.866095+0000 (oldest deadline 2024-02-05T06:57:50.866589+0000) 2024-02-05T06:57:54.483 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:57:53 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:57:53.998+0000 7f56bef57700 -1 osd.5 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:20.660664+0000 front 2024-02-05T06:57:20.660894+0000 (oldest deadline 2024-02-05T06:57:45.960611+0000) 2024-02-05T06:57:54.483 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:57:53 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:57:53.998+0000 7f56bef57700 -1 osd.5 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:29.461641+0000 front 2024-02-05T06:57:29.461520+0000 (oldest deadline 2024-02-05T06:57:53.561520+0000) 2024-02-05T06:57:55.171 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:57:55 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:57:55.074+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:57:55.172 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:57:55 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:57:55.074+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:26.865994+0000 front 2024-02-05T06:57:26.866095+0000 (oldest deadline 2024-02-05T06:57:50.866589+0000) 2024-02-05T06:57:55.172 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:57:55 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:57:55.012+0000 7f56bef57700 -1 osd.5 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:20.660664+0000 front 2024-02-05T06:57:20.660894+0000 (oldest deadline 2024-02-05T06:57:45.960611+0000) 2024-02-05T06:57:55.172 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:57:55 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:57:55.012+0000 7f56bef57700 -1 osd.5 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:29.461641+0000 front 2024-02-05T06:57:29.461520+0000 (oldest deadline 2024-02-05T06:57:53.561520+0000) 2024-02-05T06:57:55.173 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: *** Caught signal (Segmentation fault) ** 2024-02-05T06:57:55.173 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: in thread 7fd6681b5700 thread_name:safe_timer 2024-02-05T06:57:55.173 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: ceph version 19.0.0-1241-g164d3be4 (164d3be4a527be775f4d3c343ea591122f1b8921) squid (dev) 2024-02-05T06:57:55.173 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd684032d20] 2024-02-05T06:57:55.173 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x5626d176e3da] 2024-02-05T06:57:55.173 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x210) [0x5626d17fd0c0] 2024-02-05T06:57:55.173 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x5626d13eb003] 2024-02-05T06:57:55.173 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x5626d1676871] 2024-02-05T06:57:55.174 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x421) [0x5626d1366441] 2024-02-05T06:57:55.174 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x5626d136849a] 2024-02-05T06:57:55.174 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x5626d136a443] 2024-02-05T06:57:55.174 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x5626d12cc37a] 2024-02-05T06:57:55.174 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 10: (CommonSafeTimer::timer_thread()+0x13f) [0x5626d191dbff] 2024-02-05T06:57:55.174 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 11: (CommonSafeTimerThread::entry()+0x11) [0x5626d191ef21] 2024-02-05T06:57:55.174 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fd6840281ca] 2024-02-05T06:57:55.174 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 13: clone() 2024-02-05T06:57:55.174 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 2024-02-05T06:57:54.876+0000 7fd6681b5700 -1 *** Caught signal (Segmentation fault) ** 2024-02-05T06:57:55.174 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: in thread 7fd6681b5700 thread_name:safe_timer 2024-02-05T06:57:55.174 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 2024-02-05T06:57:55.174 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: ceph version 19.0.0-1241-g164d3be4 (164d3be4a527be775f4d3c343ea591122f1b8921) squid (dev) 2024-02-05T06:57:55.175 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd684032d20] 2024-02-05T06:57:55.175 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x5626d176e3da] 2024-02-05T06:57:55.175 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x210) [0x5626d17fd0c0] 2024-02-05T06:57:55.175 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x5626d13eb003] 2024-02-05T06:57:55.175 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x5626d1676871] 2024-02-05T06:57:55.175 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x421) [0x5626d1366441] 2024-02-05T06:57:55.175 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x5626d136849a] 2024-02-05T06:57:55.175 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x5626d136a443] 2024-02-05T06:57:55.175 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x5626d12cc37a] 2024-02-05T06:57:55.175 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 10: (CommonSafeTimer::timer_thread()+0x13f) [0x5626d191dbff] 2024-02-05T06:57:55.175 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 11: (CommonSafeTimerThread::entry()+0x11) [0x5626d191ef21] 2024-02-05T06:57:55.175 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fd6840281ca] 2024-02-05T06:57:55.176 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 13: clone() 2024-02-05T06:57:55.176 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-02-05T06:57:55.176 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 2024-02-05T06:57:55.176 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: -3094> 2024-02-05T06:57:42.861+0000 7fd6791d7700 -1 osd.4 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:15.345786+0000 front 2024-02-05T06:57:15.345773+0000 (oldest deadline 2024-02-05T06:57:42.346240+0000) 2024-02-05T06:57:55.176 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: -2801> 2024-02-05T06:57:43.902+0000 7fd6791d7700 -1 osd.4 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:15.345786+0000 front 2024-02-05T06:57:15.345773+0000 (oldest deadline 2024-02-05T06:57:42.346240+0000) 2024-02-05T06:57:55.176 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: -2637> 2024-02-05T06:57:44.924+0000 7fd6791d7700 -1 osd.4 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:15.345786+0000 front 2024-02-05T06:57:15.345773+0000 (oldest deadline 2024-02-05T06:57:42.346240+0000) 2024-02-05T06:57:55.176 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: -1834> 2024-02-05T06:57:45.955+0000 7fd6791d7700 -1 osd.4 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:15.345786+0000 front 2024-02-05T06:57:15.345773+0000 (oldest deadline 2024-02-05T06:57:42.346240+0000) 2024-02-05T06:57:55.176 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: -1774> 2024-02-05T06:57:46.917+0000 7fd6791d7700 -1 osd.4 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:15.345786+0000 front 2024-02-05T06:57:15.345773+0000 (oldest deadline 2024-02-05T06:57:42.346240+0000) 2024-02-05T06:57:55.176 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: -1391> 2024-02-05T06:57:47.926+0000 7fd6791d7700 -1 osd.4 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:15.345786+0000 front 2024-02-05T06:57:15.345773+0000 (oldest deadline 2024-02-05T06:57:42.346240+0000) 2024-02-05T06:57:55.176 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: -1325> 2024-02-05T06:57:48.974+0000 7fd6791d7700 -1 osd.4 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:15.345786+0000 front 2024-02-05T06:57:15.345773+0000 (oldest deadline 2024-02-05T06:57:42.346240+0000) 2024-02-05T06:57:55.176 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: -1279> 2024-02-05T06:57:49.777+0000 7fd67cbdc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T06:57:55.176 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: -1278> 2024-02-05T06:57:49.777+0000 7fd67cbdc700 -1 osd.4 510 *** Got signal Terminated *** 2024-02-05T06:57:55.177 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: -1276> 2024-02-05T06:57:49.777+0000 7fd67cbdc700 -1 osd.4 510 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T06:57:55.177 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: -1255> 2024-02-05T06:57:50.000+0000 7fd6791d7700 -1 osd.4 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:15.345786+0000 front 2024-02-05T06:57:15.345773+0000 (oldest deadline 2024-02-05T06:57:42.346240+0000) 2024-02-05T06:57:55.177 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: -1123> 2024-02-05T06:57:51.028+0000 7fd6791d7700 -1 osd.4 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:15.345786+0000 front 2024-02-05T06:57:15.345773+0000 (oldest deadline 2024-02-05T06:57:42.346240+0000) 2024-02-05T06:57:55.177 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: -933> 2024-02-05T06:57:52.066+0000 7fd6791d7700 -1 osd.4 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:15.345786+0000 front 2024-02-05T06:57:15.345773+0000 (oldest deadline 2024-02-05T06:57:42.346240+0000) 2024-02-05T06:57:55.177 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: -932> 2024-02-05T06:57:52.066+0000 7fd6791d7700 -1 osd.4 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:25.247455+0000 front 2024-02-05T06:57:25.247478+0000 (oldest deadline 2024-02-05T06:57:51.147213+0000) 2024-02-05T06:57:55.177 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: -869> 2024-02-05T06:57:53.068+0000 7fd6791d7700 -1 osd.4 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:15.345786+0000 front 2024-02-05T06:57:15.345773+0000 (oldest deadline 2024-02-05T06:57:42.346240+0000) 2024-02-05T06:57:55.177 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: -868> 2024-02-05T06:57:53.068+0000 7fd6791d7700 -1 osd.4 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:25.247455+0000 front 2024-02-05T06:57:25.247478+0000 (oldest deadline 2024-02-05T06:57:51.147213+0000) 2024-02-05T06:57:55.177 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: -166> 2024-02-05T06:57:54.104+0000 7fd6791d7700 -1 osd.4 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:15.345786+0000 front 2024-02-05T06:57:15.345773+0000 (oldest deadline 2024-02-05T06:57:42.346240+0000) 2024-02-05T06:57:55.177 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: -165> 2024-02-05T06:57:54.104+0000 7fd6791d7700 -1 osd.4 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:25.247455+0000 front 2024-02-05T06:57:25.247478+0000 (oldest deadline 2024-02-05T06:57:51.147213+0000) 2024-02-05T06:57:55.177 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 0> 2024-02-05T06:57:54.876+0000 7fd6681b5700 -1 *** Caught signal (Segmentation fault) ** 2024-02-05T06:57:55.177 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: in thread 7fd6681b5700 thread_name:safe_timer 2024-02-05T06:57:55.177 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 2024-02-05T06:57:55.177 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: ceph version 19.0.0-1241-g164d3be4 (164d3be4a527be775f4d3c343ea591122f1b8921) squid (dev) 2024-02-05T06:57:55.178 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd684032d20] 2024-02-05T06:57:55.178 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x5626d176e3da] 2024-02-05T06:57:55.178 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x210) [0x5626d17fd0c0] 2024-02-05T06:57:55.178 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x5626d13eb003] 2024-02-05T06:57:55.178 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x5626d1676871] 2024-02-05T06:57:55.178 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x421) [0x5626d1366441] 2024-02-05T06:57:55.178 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x5626d136849a] 2024-02-05T06:57:55.178 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x5626d136a443] 2024-02-05T06:57:55.178 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x5626d12cc37a] 2024-02-05T06:57:55.178 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 10: (CommonSafeTimer::timer_thread()+0x13f) [0x5626d191dbff] 2024-02-05T06:57:55.178 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 11: (CommonSafeTimerThread::entry()+0x11) [0x5626d191ef21] 2024-02-05T06:57:55.178 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fd6840281ca] 2024-02-05T06:57:55.178 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 13: clone() 2024-02-05T06:57:55.178 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-02-05T06:57:55.178 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:54 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4[37694]: 2024-02-05T06:57:55.179 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:55 smithi167 podman[152396]: 2024-02-05 06:57:55.052317367 +0000 UTC m=+5.390187640 container died 7726687dfe66c2d97975414e770e31d96fa716e89cda199fc41df83365310e0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-05T06:57:55.481 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:57:55 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:57:55.282+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:17.804342+0000 front 2024-02-05T06:57:17.804241+0000 (oldest deadline 2024-02-05T06:57:43.104006+0000) 2024-02-05T06:57:55.481 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:57:55 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:57:55.282+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:29.005152+0000 front 2024-02-05T06:57:29.005232+0000 (oldest deadline 2024-02-05T06:57:50.705036+0000) 2024-02-05T06:57:55.834 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:55 smithi167 podman[152396]: 2024-02-05 06:57:55.503464634 +0000 UTC m=+5.841334909 container cleanup 7726687dfe66c2d97975414e770e31d96fa716e89cda199fc41df83365310e0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.license=GPLv2) 2024-02-05T06:57:55.834 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:55 smithi167 bash[152396]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4 2024-02-05T06:57:56.179 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:55 smithi167 podman[152411]: 2024-02-05 06:57:55.830104036 +0000 UTC m=+0.780088994 container remove 7726687dfe66c2d97975414e770e31d96fa716e89cda199fc41df83365310e0a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4, io.buildah.version=1.33.2, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-05T06:57:56.180 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:55 smithi167 systemd[1]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715@osd.4.service: Main process exited, code=exited, status=139/n/a 2024-02-05T06:57:56.180 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:57:55 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:57:55.972+0000 7f56bef57700 -1 osd.5 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:20.660664+0000 front 2024-02-05T06:57:20.660894+0000 (oldest deadline 2024-02-05T06:57:45.960611+0000) 2024-02-05T06:57:56.180 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:57:55 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:57:55.972+0000 7f56bef57700 -1 osd.5 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:29.461641+0000 front 2024-02-05T06:57:29.461520+0000 (oldest deadline 2024-02-05T06:57:53.561520+0000) 2024-02-05T06:57:56.180 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:57:56 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:57:56.066+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:57:56.180 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:57:56 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:57:56.066+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:26.865994+0000 front 2024-02-05T06:57:26.866095+0000 (oldest deadline 2024-02-05T06:57:50.866589+0000) 2024-02-05T06:57:56.430 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:57:56 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:57:56.317+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:17.804342+0000 front 2024-02-05T06:57:17.804241+0000 (oldest deadline 2024-02-05T06:57:43.104006+0000) 2024-02-05T06:57:56.430 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:57:56 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:57:56.317+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:29.005152+0000 front 2024-02-05T06:57:29.005232+0000 (oldest deadline 2024-02-05T06:57:50.705036+0000) 2024-02-05T06:57:56.430 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:56 smithi167 podman[152503]: 2024-02-05 06:57:56.078364278 +0000 UTC m=+0.017975983 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T06:57:56.430 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:56 smithi167 podman[152503]: 2024-02-05 06:57:56.206061647 +0000 UTC m=+0.145673339 container create d0ce53d6657df2c525850bdd0bd4e252b271cca7548416de2011fda8b80d002d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-02-05T06:57:56.731 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:56 smithi167 podman[152503]: 2024-02-05 06:57:56.425814541 +0000 UTC m=+0.365426248 container init d0ce53d6657df2c525850bdd0bd4e252b271cca7548416de2011fda8b80d002d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-05T06:57:56.731 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:56 smithi167 podman[152503]: 2024-02-05 06:57:56.429778478 +0000 UTC m=+0.369390178 container start d0ce53d6657df2c525850bdd0bd4e252b271cca7548416de2011fda8b80d002d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS) 2024-02-05T06:57:56.731 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:56 smithi167 podman[152503]: 2024-02-05 06:57:56.525242721 +0000 UTC m=+0.464854426 container attach d0ce53d6657df2c525850bdd0bd4e252b271cca7548416de2011fda8b80d002d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-02-05T06:57:57.015 INFO:journalctl@ceph.osd.4.smithi167.stdout:Feb 05 06:57:56 smithi167 podman[152538]: 2024-02-05 06:57:56.820138553 +0000 UTC m=+0.019279469 container died d0ce53d6657df2c525850bdd0bd4e252b271cca7548416de2011fda8b80d002d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-02-05T06:57:57.340 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:57:57 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:57:57.032+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:57:57.340 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:57:57 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:57:57.032+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:26.865994+0000 front 2024-02-05T06:57:26.866095+0000 (oldest deadline 2024-02-05T06:57:50.866589+0000) 2024-02-05T06:57:57.340 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:57:57 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:57:57.010+0000 7f56bef57700 -1 osd.5 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:20.660664+0000 front 2024-02-05T06:57:20.660894+0000 (oldest deadline 2024-02-05T06:57:45.960611+0000) 2024-02-05T06:57:57.340 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:57:57 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:57:57.010+0000 7f56bef57700 -1 osd.5 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:29.461641+0000 front 2024-02-05T06:57:29.461520+0000 (oldest deadline 2024-02-05T06:57:53.561520+0000) 2024-02-05T06:57:57.427 DEBUG:teuthology.orchestra.run.smithi167:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@osd.4.service' 2024-02-05T06:57:57.468 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T06:57:57.468 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-02-05T06:57:57.468 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-02-05T06:57:57.469 DEBUG:teuthology.orchestra.run.smithi167:> sudo systemctl stop ceph-9801e956-c3ef-11ee-95b6-87774f69a715@osd.5 2024-02-05T06:57:57.731 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:57:57 smithi167 systemd[1]: Stopping Ceph osd.5 for 9801e956-c3ef-11ee-95b6-87774f69a715... 2024-02-05T06:57:57.731 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:57:57 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:57:57.335+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:17.804342+0000 front 2024-02-05T06:57:17.804241+0000 (oldest deadline 2024-02-05T06:57:43.104006+0000) 2024-02-05T06:57:57.732 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:57:57 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:57:57.335+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:29.005152+0000 front 2024-02-05T06:57:29.005232+0000 (oldest deadline 2024-02-05T06:57:50.705036+0000) 2024-02-05T06:57:58.231 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:57:58 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:57:58.040+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:57:58.232 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:57:58 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:57:58.040+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:26.865994+0000 front 2024-02-05T06:57:26.866095+0000 (oldest deadline 2024-02-05T06:57:50.866589+0000) 2024-02-05T06:57:58.232 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:57:57 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:57:57.815+0000 7f56c295c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T06:57:58.232 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:57:57 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:57:57.815+0000 7f56c295c700 -1 osd.5 510 *** Got signal Terminated *** 2024-02-05T06:57:58.232 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:57:57 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:57:57.815+0000 7f56c295c700 -1 osd.5 510 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T06:57:58.232 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:57:57 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:57:57.967+0000 7f56bef57700 -1 osd.5 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:20.660664+0000 front 2024-02-05T06:57:20.660894+0000 (oldest deadline 2024-02-05T06:57:45.960611+0000) 2024-02-05T06:57:58.232 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:57:57 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:57:57.967+0000 7f56bef57700 -1 osd.5 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:29.461641+0000 front 2024-02-05T06:57:29.461520+0000 (oldest deadline 2024-02-05T06:57:53.561520+0000) 2024-02-05T06:57:58.731 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:57:58 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:57:58.350+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:17.804342+0000 front 2024-02-05T06:57:17.804241+0000 (oldest deadline 2024-02-05T06:57:43.104006+0000) 2024-02-05T06:57:58.731 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:57:58 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:57:58.350+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:29.005152+0000 front 2024-02-05T06:57:29.005232+0000 (oldest deadline 2024-02-05T06:57:50.705036+0000) 2024-02-05T06:57:59.349 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:57:58 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:57:58.987+0000 7f56bef57700 -1 osd.5 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:20.660664+0000 front 2024-02-05T06:57:20.660894+0000 (oldest deadline 2024-02-05T06:57:45.960611+0000) 2024-02-05T06:57:59.349 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:57:58 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:57:58.987+0000 7f56bef57700 -1 osd.5 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:29.461641+0000 front 2024-02-05T06:57:29.461520+0000 (oldest deadline 2024-02-05T06:57:53.561520+0000) 2024-02-05T06:57:59.349 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:57:59 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:57:59.063+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:57:59.349 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:57:59 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:57:59.063+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:26.865994+0000 front 2024-02-05T06:57:26.866095+0000 (oldest deadline 2024-02-05T06:57:50.866589+0000) 2024-02-05T06:57:59.731 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:57:59 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:57:59.344+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:17.804342+0000 front 2024-02-05T06:57:17.804241+0000 (oldest deadline 2024-02-05T06:57:43.104006+0000) 2024-02-05T06:57:59.731 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:57:59 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:57:59.344+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:29.005152+0000 front 2024-02-05T06:57:29.005232+0000 (oldest deadline 2024-02-05T06:57:50.705036+0000) 2024-02-05T06:58:00.372 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:58:00 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:58:00.028+0000 7f56bef57700 -1 osd.5 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:20.660664+0000 front 2024-02-05T06:57:20.660894+0000 (oldest deadline 2024-02-05T06:57:45.960611+0000) 2024-02-05T06:58:00.373 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:58:00 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:58:00.028+0000 7f56bef57700 -1 osd.5 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:29.461641+0000 front 2024-02-05T06:57:29.461520+0000 (oldest deadline 2024-02-05T06:57:53.561520+0000) 2024-02-05T06:58:00.373 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:58:00 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:58:00.028+0000 7f56bef57700 -1 osd.5 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:33.562133+0000 front 2024-02-05T06:57:33.562117+0000 (oldest deadline 2024-02-05T06:57:59.462001+0000) 2024-02-05T06:58:00.373 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:00 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:00.036+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:58:00.373 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:00 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:00.036+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:26.865994+0000 front 2024-02-05T06:57:26.866095+0000 (oldest deadline 2024-02-05T06:57:50.866589+0000) 2024-02-05T06:58:00.373 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:00 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:00.036+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:33.767539+0000 front 2024-02-05T06:57:33.767574+0000 (oldest deadline 2024-02-05T06:57:59.667586+0000) 2024-02-05T06:58:00.731 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:00 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:00.368+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:17.804342+0000 front 2024-02-05T06:57:17.804241+0000 (oldest deadline 2024-02-05T06:57:43.104006+0000) 2024-02-05T06:58:00.731 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:00 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:00.368+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:29.005152+0000 front 2024-02-05T06:57:29.005232+0000 (oldest deadline 2024-02-05T06:57:50.705036+0000) 2024-02-05T06:58:00.731 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:00 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:00.368+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:34.806334+0000 front 2024-02-05T06:57:34.806319+0000 (oldest deadline 2024-02-05T06:58:00.106157+0000) 2024-02-05T06:58:01.414 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:58:00 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:58:00.987+0000 7f56bef57700 -1 osd.5 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:20.660664+0000 front 2024-02-05T06:57:20.660894+0000 (oldest deadline 2024-02-05T06:57:45.960611+0000) 2024-02-05T06:58:01.414 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:58:00 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:58:00.987+0000 7f56bef57700 -1 osd.5 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:29.461641+0000 front 2024-02-05T06:57:29.461520+0000 (oldest deadline 2024-02-05T06:57:53.561520+0000) 2024-02-05T06:58:01.414 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:58:00 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:58:00.987+0000 7f56bef57700 -1 osd.5 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:33.562133+0000 front 2024-02-05T06:57:33.562117+0000 (oldest deadline 2024-02-05T06:57:59.462001+0000) 2024-02-05T06:58:01.414 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:01 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:01.071+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:58:01.414 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:01 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:01.071+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:26.865994+0000 front 2024-02-05T06:57:26.866095+0000 (oldest deadline 2024-02-05T06:57:50.866589+0000) 2024-02-05T06:58:01.415 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:01 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:01.071+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:33.767539+0000 front 2024-02-05T06:57:33.767574+0000 (oldest deadline 2024-02-05T06:57:59.667586+0000) 2024-02-05T06:58:01.731 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:01 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:01.408+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:17.804342+0000 front 2024-02-05T06:57:17.804241+0000 (oldest deadline 2024-02-05T06:57:43.104006+0000) 2024-02-05T06:58:01.731 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:01 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:01.408+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:29.005152+0000 front 2024-02-05T06:57:29.005232+0000 (oldest deadline 2024-02-05T06:57:50.705036+0000) 2024-02-05T06:58:01.731 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:01 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:01.408+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:34.806334+0000 front 2024-02-05T06:57:34.806319+0000 (oldest deadline 2024-02-05T06:58:00.106157+0000) 2024-02-05T06:58:02.231 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:58:01 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:58:01.947+0000 7f56bef57700 -1 osd.5 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:20.660664+0000 front 2024-02-05T06:57:20.660894+0000 (oldest deadline 2024-02-05T06:57:45.960611+0000) 2024-02-05T06:58:02.231 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:58:01 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:58:01.947+0000 7f56bef57700 -1 osd.5 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:29.461641+0000 front 2024-02-05T06:57:29.461520+0000 (oldest deadline 2024-02-05T06:57:53.561520+0000) 2024-02-05T06:58:02.231 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:58:01 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5[43605]: 2024-02-05T06:58:01.947+0000 7f56bef57700 -1 osd.5 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:33.562133+0000 front 2024-02-05T06:57:33.562117+0000 (oldest deadline 2024-02-05T06:57:59.462001+0000) 2024-02-05T06:58:02.232 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:02 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:02.052+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:58:02.232 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:02 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:02.052+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:26.865994+0000 front 2024-02-05T06:57:26.866095+0000 (oldest deadline 2024-02-05T06:57:50.866589+0000) 2024-02-05T06:58:02.232 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:02 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:02.052+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:33.767539+0000 front 2024-02-05T06:57:33.767574+0000 (oldest deadline 2024-02-05T06:57:59.667586+0000) 2024-02-05T06:58:02.731 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:02 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:02.380+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:17.804342+0000 front 2024-02-05T06:57:17.804241+0000 (oldest deadline 2024-02-05T06:57:43.104006+0000) 2024-02-05T06:58:02.731 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:02 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:02.380+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:29.005152+0000 front 2024-02-05T06:57:29.005232+0000 (oldest deadline 2024-02-05T06:57:50.705036+0000) 2024-02-05T06:58:02.731 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:02 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:02.380+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:34.806334+0000 front 2024-02-05T06:57:34.806319+0000 (oldest deadline 2024-02-05T06:58:00.106157+0000) 2024-02-05T06:58:03.384 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:03 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:03.101+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:58:03.384 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:03 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:03.101+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:26.865994+0000 front 2024-02-05T06:57:26.866095+0000 (oldest deadline 2024-02-05T06:57:50.866589+0000) 2024-02-05T06:58:03.384 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:03 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:03.101+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:33.767539+0000 front 2024-02-05T06:57:33.767574+0000 (oldest deadline 2024-02-05T06:57:59.667586+0000) 2024-02-05T06:58:03.645 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:58:03 smithi167 podman[152623]: 2024-02-05 06:58:03.380191941 +0000 UTC m=+5.694536403 container died de8ab29278e71f12ec87b43e29ac1ce020717cda5d77e6119d841b320fb9bbe1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_CLEAN=True) 2024-02-05T06:58:03.646 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:03 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:03.418+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:17.804342+0000 front 2024-02-05T06:57:17.804241+0000 (oldest deadline 2024-02-05T06:57:43.104006+0000) 2024-02-05T06:58:03.646 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:03 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:03.418+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:29.005152+0000 front 2024-02-05T06:57:29.005232+0000 (oldest deadline 2024-02-05T06:57:50.705036+0000) 2024-02-05T06:58:03.646 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:03 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:03.418+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:34.806334+0000 front 2024-02-05T06:57:34.806319+0000 (oldest deadline 2024-02-05T06:58:00.106157+0000) 2024-02-05T06:58:03.981 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:58:03 smithi167 podman[152623]: 2024-02-05 06:58:03.740698511 +0000 UTC m=+6.055042976 container cleanup de8ab29278e71f12ec87b43e29ac1ce020717cda5d77e6119d841b320fb9bbe1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be) 2024-02-05T06:58:03.981 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:58:03 smithi167 bash[152623]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5 2024-02-05T06:58:04.323 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:58:04 smithi167 podman[152640]: 2024-02-05 06:58:04.01912266 +0000 UTC m=+0.642140520 container remove de8ab29278e71f12ec87b43e29ac1ce020717cda5d77e6119d841b320fb9bbe1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-05T06:58:04.323 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:04 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:04.070+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:58:04.323 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:04 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:04.070+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:26.865994+0000 front 2024-02-05T06:57:26.866095+0000 (oldest deadline 2024-02-05T06:57:50.866589+0000) 2024-02-05T06:58:04.323 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:04 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:04.070+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:33.767539+0000 front 2024-02-05T06:57:33.767574+0000 (oldest deadline 2024-02-05T06:57:59.667586+0000) 2024-02-05T06:58:04.583 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:58:04 smithi167 podman[152731]: 2024-02-05 06:58:04.22253481 +0000 UTC m=+0.018806424 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T06:58:04.583 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:58:04 smithi167 podman[152731]: 2024-02-05 06:58:04.334800514 +0000 UTC m=+0.131072132 container create f8e592f339bcae63e40df9b8f6d7aba7724396717784385ac0111df310144fe5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131) 2024-02-05T06:58:04.584 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:04 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:04.383+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:17.804342+0000 front 2024-02-05T06:57:17.804241+0000 (oldest deadline 2024-02-05T06:57:43.104006+0000) 2024-02-05T06:58:04.584 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:04 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:04.383+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:29.005152+0000 front 2024-02-05T06:57:29.005232+0000 (oldest deadline 2024-02-05T06:57:50.705036+0000) 2024-02-05T06:58:04.584 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:04 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:04.383+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:34.806334+0000 front 2024-02-05T06:57:34.806319+0000 (oldest deadline 2024-02-05T06:58:00.106157+0000) 2024-02-05T06:58:04.876 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:58:04 smithi167 podman[152731]: 2024-02-05 06:58:04.579260408 +0000 UTC m=+0.375532023 container init f8e592f339bcae63e40df9b8f6d7aba7724396717784385ac0111df310144fe5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-02-05T06:58:04.877 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:58:04 smithi167 podman[152731]: 2024-02-05 06:58:04.582589357 +0000 UTC m=+0.378860972 container start f8e592f339bcae63e40df9b8f6d7aba7724396717784385ac0111df310144fe5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.license=GPLv2) 2024-02-05T06:58:04.877 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:58:04 smithi167 podman[152731]: 2024-02-05 06:58:04.662420015 +0000 UTC m=+0.458691631 container attach f8e592f339bcae63e40df9b8f6d7aba7724396717784385ac0111df310144fe5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-02-05T06:58:05.169 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:05.075+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:58:05.169 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:05.075+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:26.865994+0000 front 2024-02-05T06:57:26.866095+0000 (oldest deadline 2024-02-05T06:57:50.866589+0000) 2024-02-05T06:58:05.169 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:05.075+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:33.767539+0000 front 2024-02-05T06:57:33.767574+0000 (oldest deadline 2024-02-05T06:57:59.667586+0000) 2024-02-05T06:58:05.169 INFO:journalctl@ceph.osd.5.smithi167.stdout:Feb 05 06:58:04 smithi167 podman[152731]: 2024-02-05 06:58:04.872882611 +0000 UTC m=+0.669154233 container died f8e592f339bcae63e40df9b8f6d7aba7724396717784385ac0111df310144fe5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-02-05T06:58:05.470 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:05.353+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:17.804342+0000 front 2024-02-05T06:57:17.804241+0000 (oldest deadline 2024-02-05T06:57:43.104006+0000) 2024-02-05T06:58:05.470 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:05.353+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:29.005152+0000 front 2024-02-05T06:57:29.005232+0000 (oldest deadline 2024-02-05T06:57:50.705036+0000) 2024-02-05T06:58:05.470 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:05.353+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:34.806334+0000 front 2024-02-05T06:57:34.806319+0000 (oldest deadline 2024-02-05T06:58:00.106157+0000) 2024-02-05T06:58:05.490 DEBUG:teuthology.orchestra.run.smithi167:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@osd.5.service' 2024-02-05T06:58:05.531 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T06:58:05.531 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-02-05T06:58:05.531 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-02-05T06:58:05.531 DEBUG:teuthology.orchestra.run.smithi167:> sudo systemctl stop ceph-9801e956-c3ef-11ee-95b6-87774f69a715@osd.6 2024-02-05T06:58:05.731 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:05 smithi167 systemd[1]: Stopping Ceph osd.6 for 9801e956-c3ef-11ee-95b6-87774f69a715... 2024-02-05T06:58:06.231 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:05.852+0000 7fd864626700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T06:58:06.231 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:05.852+0000 7fd864626700 -1 osd.6 510 *** Got signal Terminated *** 2024-02-05T06:58:06.231 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:05 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:05.852+0000 7fd864626700 -1 osd.6 510 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T06:58:06.232 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:06.095+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:58:06.232 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:06.095+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:26.865994+0000 front 2024-02-05T06:57:26.866095+0000 (oldest deadline 2024-02-05T06:57:50.866589+0000) 2024-02-05T06:58:06.232 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:06.095+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:33.767539+0000 front 2024-02-05T06:57:33.767574+0000 (oldest deadline 2024-02-05T06:57:59.667586+0000) 2024-02-05T06:58:06.731 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:06.328+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:17.804342+0000 front 2024-02-05T06:57:17.804241+0000 (oldest deadline 2024-02-05T06:57:43.104006+0000) 2024-02-05T06:58:06.731 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:06.328+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:29.005152+0000 front 2024-02-05T06:57:29.005232+0000 (oldest deadline 2024-02-05T06:57:50.705036+0000) 2024-02-05T06:58:06.731 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:06 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:06.328+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:34.806334+0000 front 2024-02-05T06:57:34.806319+0000 (oldest deadline 2024-02-05T06:58:00.106157+0000) 2024-02-05T06:58:07.324 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:07 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:07.099+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:58:07.325 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:07 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:07.099+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:26.865994+0000 front 2024-02-05T06:57:26.866095+0000 (oldest deadline 2024-02-05T06:57:50.866589+0000) 2024-02-05T06:58:07.325 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:07 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:07.099+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:33.767539+0000 front 2024-02-05T06:57:33.767574+0000 (oldest deadline 2024-02-05T06:57:59.667586+0000) 2024-02-05T06:58:07.607 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:07 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:07.320+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:17.804342+0000 front 2024-02-05T06:57:17.804241+0000 (oldest deadline 2024-02-05T06:57:43.104006+0000) 2024-02-05T06:58:07.607 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:07 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:07.320+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:29.005152+0000 front 2024-02-05T06:57:29.005232+0000 (oldest deadline 2024-02-05T06:57:50.705036+0000) 2024-02-05T06:58:07.608 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:07 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:07.320+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:34.806334+0000 front 2024-02-05T06:57:34.806319+0000 (oldest deadline 2024-02-05T06:58:00.106157+0000) 2024-02-05T06:58:08.365 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:08 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:08.075+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:58:08.365 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:08 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:08.075+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:26.865994+0000 front 2024-02-05T06:57:26.866095+0000 (oldest deadline 2024-02-05T06:57:50.866589+0000) 2024-02-05T06:58:08.365 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:08 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:08.075+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:33.767539+0000 front 2024-02-05T06:57:33.767574+0000 (oldest deadline 2024-02-05T06:57:59.667586+0000) 2024-02-05T06:58:08.731 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:08 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:08.361+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:17.804342+0000 front 2024-02-05T06:57:17.804241+0000 (oldest deadline 2024-02-05T06:57:43.104006+0000) 2024-02-05T06:58:08.731 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:08 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:08.361+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:29.005152+0000 front 2024-02-05T06:57:29.005232+0000 (oldest deadline 2024-02-05T06:57:50.705036+0000) 2024-02-05T06:58:08.731 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:08 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:08.361+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:34.806334+0000 front 2024-02-05T06:57:34.806319+0000 (oldest deadline 2024-02-05T06:58:00.106157+0000) 2024-02-05T06:58:09.346 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:09 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:09.032+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:58:09.346 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:09 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:09.032+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:26.865994+0000 front 2024-02-05T06:57:26.866095+0000 (oldest deadline 2024-02-05T06:57:50.866589+0000) 2024-02-05T06:58:09.346 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:09 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:09.032+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:33.767539+0000 front 2024-02-05T06:57:33.767574+0000 (oldest deadline 2024-02-05T06:57:59.667586+0000) 2024-02-05T06:58:09.347 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:09 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:09.032+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-02-05T06:57:43.768495+0000 front 2024-02-05T06:57:43.768543+0000 (oldest deadline 2024-02-05T06:58:08.468483+0000) 2024-02-05T06:58:09.731 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:09 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:09.341+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:17.804342+0000 front 2024-02-05T06:57:17.804241+0000 (oldest deadline 2024-02-05T06:57:43.104006+0000) 2024-02-05T06:58:09.731 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:09 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:09.341+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:29.005152+0000 front 2024-02-05T06:57:29.005232+0000 (oldest deadline 2024-02-05T06:57:50.705036+0000) 2024-02-05T06:58:09.731 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:09 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:09.341+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:34.806334+0000 front 2024-02-05T06:57:34.806319+0000 (oldest deadline 2024-02-05T06:58:00.106157+0000) 2024-02-05T06:58:10.481 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:10 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:10.301+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:17.804342+0000 front 2024-02-05T06:57:17.804241+0000 (oldest deadline 2024-02-05T06:57:43.104006+0000) 2024-02-05T06:58:10.481 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:10 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:10.301+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:29.005152+0000 front 2024-02-05T06:57:29.005232+0000 (oldest deadline 2024-02-05T06:57:50.705036+0000) 2024-02-05T06:58:10.481 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:10 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:10.301+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:34.806334+0000 front 2024-02-05T06:57:34.806319+0000 (oldest deadline 2024-02-05T06:58:00.106157+0000) 2024-02-05T06:58:10.481 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:10 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6[49281]: 2024-02-05T06:58:10.301+0000 7fd85cc19700 -1 osd.6 510 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-02-05T06:57:45.407137+0000 front 2024-02-05T06:57:45.407219+0000 (oldest deadline 2024-02-05T06:58:09.507054+0000) 2024-02-05T06:58:10.482 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:10 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:10.062+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:58:10.482 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:10 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:10.062+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:26.865994+0000 front 2024-02-05T06:57:26.866095+0000 (oldest deadline 2024-02-05T06:57:50.866589+0000) 2024-02-05T06:58:10.482 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:10 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:10.062+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:33.767539+0000 front 2024-02-05T06:57:33.767574+0000 (oldest deadline 2024-02-05T06:57:59.667586+0000) 2024-02-05T06:58:10.482 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:10 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:10.062+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-02-05T06:57:43.768495+0000 front 2024-02-05T06:57:43.768543+0000 (oldest deadline 2024-02-05T06:58:08.468483+0000) 2024-02-05T06:58:11.481 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:11 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:11.089+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:58:11.481 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:11 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:11.089+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:26.865994+0000 front 2024-02-05T06:57:26.866095+0000 (oldest deadline 2024-02-05T06:57:50.866589+0000) 2024-02-05T06:58:11.481 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:11 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:11.089+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:33.767539+0000 front 2024-02-05T06:57:33.767574+0000 (oldest deadline 2024-02-05T06:57:59.667586+0000) 2024-02-05T06:58:11.481 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:11 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:11.089+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-02-05T06:57:43.768495+0000 front 2024-02-05T06:57:43.768543+0000 (oldest deadline 2024-02-05T06:58:08.468483+0000) 2024-02-05T06:58:11.966 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:11 smithi167 podman[152836]: 2024-02-05 06:58:11.519167464 +0000 UTC m=+5.774553404 container died a4b373a06f0ead4094530e5ddd384de03b6e7ef5ff91cf1107957ba10158c8ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_BRANCH=HEAD) 2024-02-05T06:58:12.231 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:12 smithi167 podman[152836]: 2024-02-05 06:58:12.061117841 +0000 UTC m=+6.316503791 container cleanup a4b373a06f0ead4094530e5ddd384de03b6e7ef5ff91cf1107957ba10158c8ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-02-05T06:58:12.231 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:12 smithi167 bash[152836]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6 2024-02-05T06:58:12.231 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:12 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:12.094+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:58:12.232 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:12 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:12.094+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:26.865994+0000 front 2024-02-05T06:57:26.866095+0000 (oldest deadline 2024-02-05T06:57:50.866589+0000) 2024-02-05T06:58:12.232 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:12 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:12.094+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:33.767539+0000 front 2024-02-05T06:57:33.767574+0000 (oldest deadline 2024-02-05T06:57:59.667586+0000) 2024-02-05T06:58:12.232 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:12 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:12.094+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-02-05T06:57:43.768495+0000 front 2024-02-05T06:57:43.768543+0000 (oldest deadline 2024-02-05T06:58:08.468483+0000) 2024-02-05T06:58:12.642 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:12 smithi167 podman[152854]: 2024-02-05 06:58:12.253959176 +0000 UTC m=+0.736737030 container remove a4b373a06f0ead4094530e5ddd384de03b6e7ef5ff91cf1107957ba10158c8ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-05T06:58:12.969 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:12 smithi167 podman[152946]: 2024-02-05 06:58:12.63841483 +0000 UTC m=+0.105544290 container create 31ec793b5f9cf2f84d78d887979ffe9c0ab922d263e7bbd5b8d41069a087e324 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux ) 2024-02-05T06:58:12.970 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:12 smithi167 podman[152946]: 2024-02-05 06:58:12.550809616 +0000 UTC m=+0.017939076 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T06:58:12.970 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:12 smithi167 podman[152946]: 2024-02-05 06:58:12.876071991 +0000 UTC m=+0.343201454 container init 31ec793b5f9cf2f84d78d887979ffe9c0ab922d263e7bbd5b8d41069a087e324 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-02-05T06:58:12.970 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:12 smithi167 podman[152946]: 2024-02-05 06:58:12.879563992 +0000 UTC m=+0.346693462 container start 31ec793b5f9cf2f84d78d887979ffe9c0ab922d263e7bbd5b8d41069a087e324 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-02-05T06:58:12.970 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:12 smithi167 podman[152946]: 2024-02-05 06:58:12.965649901 +0000 UTC m=+0.432779370 container attach 31ec793b5f9cf2f84d78d887979ffe9c0ab922d263e7bbd5b8d41069a087e324 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_CLEAN=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2) 2024-02-05T06:58:13.231 INFO:journalctl@ceph.osd.6.smithi167.stdout:Feb 05 06:58:13 smithi167 podman[152946]: 2024-02-05 06:58:13.169334231 +0000 UTC m=+0.636463692 container died 31ec793b5f9cf2f84d78d887979ffe9c0ab922d263e7bbd5b8d41069a087e324 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-02-05T06:58:13.232 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:13 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:13.092+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:58:13.232 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:13 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:13.092+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:26.865994+0000 front 2024-02-05T06:57:26.866095+0000 (oldest deadline 2024-02-05T06:57:50.866589+0000) 2024-02-05T06:58:13.232 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:13 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:13.092+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:33.767539+0000 front 2024-02-05T06:57:33.767574+0000 (oldest deadline 2024-02-05T06:57:59.667586+0000) 2024-02-05T06:58:13.232 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:13 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:13.092+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-02-05T06:57:43.768495+0000 front 2024-02-05T06:57:43.768543+0000 (oldest deadline 2024-02-05T06:58:08.468483+0000) 2024-02-05T06:58:13.824 DEBUG:teuthology.orchestra.run.smithi167:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@osd.6.service' 2024-02-05T06:58:13.866 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T06:58:13.867 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-02-05T06:58:13.867 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-02-05T06:58:13.867 DEBUG:teuthology.orchestra.run.smithi167:> sudo systemctl stop ceph-9801e956-c3ef-11ee-95b6-87774f69a715@osd.7 2024-02-05T06:58:14.126 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:13 smithi167 systemd[1]: Stopping Ceph osd.7 for 9801e956-c3ef-11ee-95b6-87774f69a715... 2024-02-05T06:58:14.390 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:14 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:14.120+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:58:14.390 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:14 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:14.120+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:26.865994+0000 front 2024-02-05T06:57:26.866095+0000 (oldest deadline 2024-02-05T06:57:50.866589+0000) 2024-02-05T06:58:14.390 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:14 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:14.120+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:33.767539+0000 front 2024-02-05T06:57:33.767574+0000 (oldest deadline 2024-02-05T06:57:59.667586+0000) 2024-02-05T06:58:14.390 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:14 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:14.120+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-02-05T06:57:43.768495+0000 front 2024-02-05T06:57:43.768543+0000 (oldest deadline 2024-02-05T06:58:08.468483+0000) 2024-02-05T06:58:14.390 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:14 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:14.221+0000 7f3c1e9c9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T06:58:14.390 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:14 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:14.221+0000 7f3c1e9c9700 -1 osd.7 510 *** Got signal Terminated *** 2024-02-05T06:58:14.390 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:14 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:14.221+0000 7f3c1e9c9700 -1 osd.7 510 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-05T06:58:15.481 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:15 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:15.156+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:58:15.481 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:15 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:15.156+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:26.865994+0000 front 2024-02-05T06:57:26.866095+0000 (oldest deadline 2024-02-05T06:57:50.866589+0000) 2024-02-05T06:58:15.481 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:15 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:15.156+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:33.767539+0000 front 2024-02-05T06:57:33.767574+0000 (oldest deadline 2024-02-05T06:57:59.667586+0000) 2024-02-05T06:58:15.481 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:15 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:15.156+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-02-05T06:57:43.768495+0000 front 2024-02-05T06:57:43.768543+0000 (oldest deadline 2024-02-05T06:58:08.468483+0000) 2024-02-05T06:58:16.481 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:16.157+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:58:16.481 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:16.157+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:26.865994+0000 front 2024-02-05T06:57:26.866095+0000 (oldest deadline 2024-02-05T06:57:50.866589+0000) 2024-02-05T06:58:16.481 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:16.157+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:33.767539+0000 front 2024-02-05T06:57:33.767574+0000 (oldest deadline 2024-02-05T06:57:59.667586+0000) 2024-02-05T06:58:16.481 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:16.157+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-02-05T06:57:43.768495+0000 front 2024-02-05T06:57:43.768543+0000 (oldest deadline 2024-02-05T06:58:08.468483+0000) 2024-02-05T06:58:17.194 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:58:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:58:16.861Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.28:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.28:8765: connect: connection refused" 2024-02-05T06:58:17.194 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:58:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:58:16.863Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.28:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.28:8765: connect: connection refused" 2024-02-05T06:58:17.194 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:58:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:58:16.865Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.28:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.28:8765: connect: connection refused" 2024-02-05T06:58:17.194 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:58:16 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:58:16.867Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.28:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.28:8765: connect: connection refused" 2024-02-05T06:58:17.481 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:17 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:17.189+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:58:17.481 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:17 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:17.189+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:26.865994+0000 front 2024-02-05T06:57:26.866095+0000 (oldest deadline 2024-02-05T06:57:50.866589+0000) 2024-02-05T06:58:17.481 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:17 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:17.189+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:33.767539+0000 front 2024-02-05T06:57:33.767574+0000 (oldest deadline 2024-02-05T06:57:59.667586+0000) 2024-02-05T06:58:17.481 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:17 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:17.189+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-02-05T06:57:43.768495+0000 front 2024-02-05T06:57:43.768543+0000 (oldest deadline 2024-02-05T06:58:08.468483+0000) 2024-02-05T06:58:18.481 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:18 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:18.170+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:58:18.481 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:18 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:18.170+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:26.865994+0000 front 2024-02-05T06:57:26.866095+0000 (oldest deadline 2024-02-05T06:57:50.866589+0000) 2024-02-05T06:58:18.481 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:18 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:18.170+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:33.767539+0000 front 2024-02-05T06:57:33.767574+0000 (oldest deadline 2024-02-05T06:57:59.667586+0000) 2024-02-05T06:58:18.481 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:18 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:18.170+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-02-05T06:57:43.768495+0000 front 2024-02-05T06:57:43.768543+0000 (oldest deadline 2024-02-05T06:58:08.468483+0000) 2024-02-05T06:58:19.481 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:19 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:19.180+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6803 osd.0 since back 2024-02-05T06:57:18.065137+0000 front 2024-02-05T06:57:18.064987+0000 (oldest deadline 2024-02-05T06:57:43.965017+0000) 2024-02-05T06:58:19.481 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:19 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:19.180+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6807 osd.1 since back 2024-02-05T06:57:26.865994+0000 front 2024-02-05T06:57:26.866095+0000 (oldest deadline 2024-02-05T06:57:50.866589+0000) 2024-02-05T06:58:19.482 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:19 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:19.180+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6811 osd.2 since back 2024-02-05T06:57:33.767539+0000 front 2024-02-05T06:57:33.767574+0000 (oldest deadline 2024-02-05T06:57:59.667586+0000) 2024-02-05T06:58:19.482 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:19 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:19.180+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.28:6815 osd.3 since back 2024-02-05T06:57:43.768495+0000 front 2024-02-05T06:57:43.768543+0000 (oldest deadline 2024-02-05T06:58:08.468483+0000) 2024-02-05T06:58:19.482 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:19 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7[55184]: 2024-02-05T06:58:19.180+0000 7f3c16fbc700 -1 osd.7 510 heartbeat_check: no reply from 172.21.15.167:6802 osd.4 since back 2024-02-05T06:57:54.770312+0000 front 2024-02-05T06:57:54.770337+0000 (oldest deadline 2024-02-05T06:58:18.270278+0000) 2024-02-05T06:58:19.981 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:19 smithi167 podman[153062]: 2024-02-05 06:58:19.72148291 +0000 UTC m=+5.630648028 container died bd7dbddc2c178a66c81fa061fd70ed7f507487c807578da71e70117b4b45565e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-02-05T06:58:20.361 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:20 smithi167 podman[153062]: 2024-02-05 06:58:20.080953809 +0000 UTC m=+5.990118926 container cleanup bd7dbddc2c178a66c81fa061fd70ed7f507487c807578da71e70117b4b45565e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131) 2024-02-05T06:58:20.361 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:20 smithi167 bash[153062]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7 2024-02-05T06:58:20.707 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:20 smithi167 podman[153080]: 2024-02-05 06:58:20.357113534 +0000 UTC m=+0.631636093 container remove bd7dbddc2c178a66c81fa061fd70ed7f507487c807578da71e70117b4b45565e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-05T06:58:20.981 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:20 smithi167 podman[153176]: 2024-02-05 06:58:20.606983104 +0000 UTC m=+0.018673571 image pull 8f4bf3e8b16aa617a3377544276cd5a5b73be8802746138ce876732ceeade8e1 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0 2024-02-05T06:58:20.982 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:20 smithi167 podman[153176]: 2024-02-05 06:58:20.725106131 +0000 UTC m=+0.136796589 container create b745fb0e8571292449f3bad7a8285c4a69246136c2778412a1a9a185fffb8de4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-02-05T06:58:20.982 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:20 smithi167 podman[153176]: 2024-02-05 06:58:20.919213546 +0000 UTC m=+0.330904005 container init b745fb0e8571292449f3bad7a8285c4a69246136c2778412a1a9a185fffb8de4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux ) 2024-02-05T06:58:20.982 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:20 smithi167 podman[153176]: 2024-02-05 06:58:20.922405702 +0000 UTC m=+0.334096164 container start b745fb0e8571292449f3bad7a8285c4a69246136c2778412a1a9a185fffb8de4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7-deactivate, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-02-05T06:58:21.481 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:20 smithi167 podman[153176]: 2024-02-05 06:58:20.993973784 +0000 UTC m=+0.405664259 container attach b745fb0e8571292449f3bad7a8285c4a69246136c2778412a1a9a185fffb8de4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be) 2024-02-05T06:58:21.481 INFO:journalctl@ceph.osd.7.smithi167.stdout:Feb 05 06:58:21 smithi167 podman[153176]: 2024-02-05 06:58:21.212558796 +0000 UTC m=+0.624249258 container died b745fb0e8571292449f3bad7a8285c4a69246136c2778412a1a9a185fffb8de4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-02-05T06:58:21.815 DEBUG:teuthology.orchestra.run.smithi167:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@osd.7.service' 2024-02-05T06:58:21.857 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T06:58:21.857 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-02-05T06:58:21.857 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-02-05T06:58:21.857 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-9801e956-c3ef-11ee-95b6-87774f69a715@rgw.foo.a 2024-02-05T06:58:22.161 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Feb 05 06:58:21 smithi028 systemd[1]: Stopping Ceph rgw.foo.a for 9801e956-c3ef-11ee-95b6-87774f69a715... 2024-02-05T06:58:22.430 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Feb 05 06:58:22 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-rgw-foo-a[64630]: 2024-02-05T06:58:22.155+0000 7f15163b8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-05T06:58:22.430 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Feb 05 06:58:22 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-rgw-foo-a[64630]: 2024-02-05T06:58:22.155+0000 7f1524c23ac0 -1 shutting down 2024-02-05T06:58:32.447 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Feb 05 06:58:32 smithi028 bash[208162]: time="2024-02-05T06:58:32Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-9801e956-c3ef-11ee-95b6-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-02-05T06:58:32.492 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Feb 05 06:58:32 smithi028 podman[208162]: 2024-02-05 06:58:32.188010707 +0000 UTC m=+10.137366538 container died 0c8a74bcd665cf060f92be62f0e89b552fec5fd25d64d7e0b28b71ad94fdd6f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-rgw-foo-a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2) 2024-02-05T06:58:32.875 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Feb 05 06:58:32 smithi028 podman[208162]: 2024-02-05 06:58:32.714820837 +0000 UTC m=+10.664176660 container cleanup 0c8a74bcd665cf060f92be62f0e89b552fec5fd25d64d7e0b28b71ad94fdd6f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-rgw-foo-a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-02-05T06:58:32.875 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Feb 05 06:58:32 smithi028 bash[208162]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715-rgw-foo-a 2024-02-05T06:58:33.130 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Feb 05 06:58:32 smithi028 podman[208190]: 2024-02-05 06:58:32.916022186 +0000 UTC m=+0.721524447 container remove 0c8a74bcd665cf060f92be62f0e89b552fec5fd25d64d7e0b28b71ad94fdd6f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-rgw-foo-a, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-05T06:58:33.130 INFO:journalctl@ceph.rgw.foo.a.smithi028.stdout:Feb 05 06:58:32 smithi028 systemd[1]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-02-05T06:58:33.145 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@rgw.foo.a.service' 2024-02-05T06:58:33.183 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T06:58:33.183 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-02-05T06:58:33.183 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-02-05T06:58:33.183 DEBUG:teuthology.orchestra.run.smithi167:> sudo systemctl stop ceph-9801e956-c3ef-11ee-95b6-87774f69a715@prometheus.a 2024-02-05T06:58:33.481 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:58:33 smithi167 systemd[1]: Stopping Ceph prometheus.a for 9801e956-c3ef-11ee-95b6-87774f69a715... 2024-02-05T06:58:33.829 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:58:33 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:58:33.532Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-05T06:58:33.829 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:58:33 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:58:33.533Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-02-05T06:58:33.829 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:58:33 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:58:33.533Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-02-05T06:58:33.829 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:58:33 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:58:33.533Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-05T06:58:33.829 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:58:33 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:58:33.533Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-02-05T06:58:33.829 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:58:33 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:58:33.533Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-02-05T06:58:33.830 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:58:33 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:58:33.533Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-02-05T06:58:33.830 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:58:33 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:58:33.533Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-02-05T06:58:33.830 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:58:33 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:58:33.544Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-02-05T06:58:33.830 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:58:33 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:58:33.547Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-02-05T06:58:33.830 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:58:33 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:58:33.547Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-02-05T06:58:33.830 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:58:33 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a[68593]: ts=2024-02-05T06:58:33.547Z caller=main.go:1147 level=info msg="See you next time!" 2024-02-05T06:58:33.830 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:58:33 smithi167 podman[153282]: 2024-02-05 06:58:33.553295544 +0000 UTC m=+0.144459016 container died 89e77b8f00daed45e30629aef5555c38dc6a653976263cef60036bd21e2c6119 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-05T06:58:34.212 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:58:33 smithi167 podman[153282]: 2024-02-05 06:58:33.93289012 +0000 UTC m=+0.524053587 container cleanup 89e77b8f00daed45e30629aef5555c38dc6a653976263cef60036bd21e2c6119 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-05T06:58:34.213 INFO:journalctl@ceph.prometheus.a.smithi167.stdout:Feb 05 06:58:33 smithi167 bash[153282]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715-prometheus-a 2024-02-05T06:58:34.342 DEBUG:teuthology.orchestra.run.smithi167:> sudo pkill -f 'journalctl -f -n 0 -u ceph-9801e956-c3ef-11ee-95b6-87774f69a715@prometheus.a.service' 2024-02-05T06:58:34.380 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-05T06:58:34.380 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-02-05T06:58:34.380 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 --force --keep-logs 2024-02-05T06:58:34.578 INFO:teuthology.orchestra.run.smithi028.stdout:Deleting cluster with fsid: 9801e956-c3ef-11ee-95b6-87774f69a715 2024-02-05T06:58:37.196 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:58:36 smithi028 systemd[1]: Stopping Ceph alertmanager.a for 9801e956-c3ef-11ee-95b6-87774f69a715... 2024-02-05T06:58:37.634 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:58:37 smithi028 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a[80834]: ts=2024-02-05T06:58:37.220Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-02-05T06:58:37.634 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:58:37 smithi028 podman[208667]: 2024-02-05 06:58:37.291634907 +0000 UTC m=+0.160771275 container died 696060c3ff8b7a9d0f55bed2f60178446b4a783b14be24ba5edb33e52b210ede (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T06:58:38.235 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:58:37 smithi028 podman[208667]: 2024-02-05 06:58:37.945297454 +0000 UTC m=+0.814433801 container cleanup 696060c3ff8b7a9d0f55bed2f60178446b4a783b14be24ba5edb33e52b210ede (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T06:58:38.235 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:58:37 smithi028 bash[208667]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a 2024-02-05T06:58:38.235 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:58:38 smithi028 podman[208681]: 2024-02-05 06:58:38.146713499 +0000 UTC m=+0.850541602 container remove 696060c3ff8b7a9d0f55bed2f60178446b4a783b14be24ba5edb33e52b210ede (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-05T06:58:38.235 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:58:38 smithi028 podman[208681]: 2024-02-05 06:58:38.231381837 +0000 UTC m=+0.935209938 volume remove 415ec61802d55f8d9c6de3bc0717252bed01ba231e2a17d0e2605a8be5e27827 2024-02-05T06:58:38.564 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:58:38 smithi028 systemd[1]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-05T06:58:38.564 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:58:38 smithi028 systemd[1]: Stopped Ceph alertmanager.a for 9801e956-c3ef-11ee-95b6-87774f69a715. 2024-02-05T06:58:38.564 INFO:journalctl@ceph.alertmanager.a.smithi028.stdout:Feb 05 06:58:38 smithi028 systemd[1]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715@alertmanager.a.service: Consumed 1.713s CPU time. 2024-02-05T06:58:38.839 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:58:38 smithi028 systemd[1]: Stopping Ceph node-exporter.a for 9801e956-c3ef-11ee-95b6-87774f69a715... 2024-02-05T06:58:39.096 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:58:38 smithi028 podman[208794]: 2024-02-05 06:58:38.835534556 +0000 UTC m=+0.124651151 container died f4251001d32cc3f587827207435d243e11ea8e4ae21e1dca8cf7818871588f33 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-05T06:58:39.405 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:58:39 smithi028 podman[208794]: 2024-02-05 06:58:39.225201284 +0000 UTC m=+0.514317879 container cleanup f4251001d32cc3f587827207435d243e11ea8e4ae21e1dca8cf7818871588f33 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-05T06:58:39.405 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:58:39 smithi028 bash[208794]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a 2024-02-05T06:58:39.696 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:58:39 smithi028 podman[208809]: 2024-02-05 06:58:39.401530358 +0000 UTC m=+0.559511629 container remove f4251001d32cc3f587827207435d243e11ea8e4ae21e1dca8cf7818871588f33 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-05T06:58:39.696 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:58:39 smithi028 systemd[1]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-02-05T06:58:39.696 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:58:39 smithi028 systemd[1]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-05T06:58:39.696 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:58:39 smithi028 systemd[1]: Stopped Ceph node-exporter.a for 9801e956-c3ef-11ee-95b6-87774f69a715. 2024-02-05T06:58:39.697 INFO:journalctl@ceph.node-exporter.a.smithi028.stdout:Feb 05 06:58:39 smithi028 systemd[1]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715@node-exporter.a.service: Consumed 4.879s CPU time. 2024-02-05T06:58:40.315 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 --force --keep-logs 2024-02-05T06:58:40.506 INFO:teuthology.orchestra.run.smithi167.stdout:Deleting cluster with fsid: 9801e956-c3ef-11ee-95b6-87774f69a715 2024-02-05T06:58:42.691 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:58:42 smithi167 systemd[1]: Stopping Ceph iscsi.iscsi.a for 9801e956-c3ef-11ee-95b6-87774f69a715... 2024-02-05T06:58:42.981 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:58:42 smithi167 ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a[60271]: debug Shutdown received 2024-02-05T06:58:52.981 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:58:52 smithi167 bash[153692]: time="2024-02-05T06:58:52Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-02-05T06:58:52.982 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:58:52 smithi167 podman[153692]: 2024-02-05 06:58:52.705678681 +0000 UTC m=+10.117071645 container died d9968e61e530f0c517bc3150699205f2c9a7f47dbb44f66010c26b32260181e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux ) 2024-02-05T06:58:53.316 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:58:53 smithi167 podman[153692]: 2024-02-05 06:58:53.102765517 +0000 UTC m=+10.514158522 container cleanup d9968e61e530f0c517bc3150699205f2c9a7f47dbb44f66010c26b32260181e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-02-05T06:58:53.316 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:58:53 smithi167 bash[153692]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a 2024-02-05T06:58:53.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:58:53 smithi167 podman[153720]: 2024-02-05 06:58:53.312480243 +0000 UTC m=+0.598421256 container remove d9968e61e530f0c517bc3150699205f2c9a7f47dbb44f66010c26b32260181e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:471dc037e7bbfe27d19519404cf9282a911af42da176a0f19461b166165464b0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-iscsi-iscsi-a, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-testing-3-2024-02-04-2254-164d3be, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-02-05T06:58:53.720 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:58:53 smithi167 systemd[1]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-02-05T06:58:53.720 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:58:53 smithi167 systemd[1]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-02-05T06:58:53.720 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:58:53 smithi167 systemd[1]: Stopped Ceph iscsi.iscsi.a for 9801e956-c3ef-11ee-95b6-87774f69a715. 2024-02-05T06:58:53.720 INFO:journalctl@ceph.iscsi.iscsi.a.smithi167.stdout:Feb 05 06:58:53 smithi167 systemd[1]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715@iscsi.iscsi.a.service: Consumed 3.356s CPU time. 2024-02-05T06:58:54.981 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:58:54 smithi167 systemd[1]: Stopping Ceph grafana.a for 9801e956-c3ef-11ee-95b6-87774f69a715... 2024-02-05T06:58:55.373 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:58:55 smithi167 podman[153982]: 2024-02-05 06:58:55.113011667 +0000 UTC m=+0.100991664 container died 5fe0a0c56ac4123c85388e99ddc8501f9940bbfd67dc30b0516653c75a70ede0 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, io.buildah.version=1.32.0, name=ubi8, build-date=2022-04-29T11:48:11.055897, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, version=8.5, vcs-type=git, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner ) 2024-02-05T06:58:55.667 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:58:55 smithi167 podman[153982]: 2024-02-05 06:58:55.452529377 +0000 UTC m=+0.440509374 container cleanup 5fe0a0c56ac4123c85388e99ddc8501f9940bbfd67dc30b0516653c75a70ede0 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a, build-date=2022-04-29T11:48:11.055897, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, io.buildah.version=1.32.0, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, version=8.5, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, vcs-type=git, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , vendor=Red Hat, Inc., io.openshift.tags=base rhel8, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-02-05T06:58:55.667 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:58:55 smithi167 bash[153982]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a 2024-02-05T06:58:55.981 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:58:55 smithi167 podman[153997]: 2024-02-05 06:58:55.67074716 +0000 UTC m=+0.556763007 container remove 5fe0a0c56ac4123c85388e99ddc8501f9940bbfd67dc30b0516653c75a70ede0 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-grafana-a, release=239.1651231664, version=8.5, build-date=2022-04-29T11:48:11.055897, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, io.openshift.expose-services=, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, distribution-scope=public, io.buildah.version=1.32.0, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git) 2024-02-05T06:58:55.981 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:58:55 smithi167 systemd[1]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-02-05T06:58:55.981 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:58:55 smithi167 systemd[1]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-05T06:58:55.981 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:58:55 smithi167 systemd[1]: Stopped Ceph grafana.a for 9801e956-c3ef-11ee-95b6-87774f69a715. 2024-02-05T06:58:55.981 INFO:journalctl@ceph.grafana.a.smithi167.stdout:Feb 05 06:58:55 smithi167 systemd[1]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715@grafana.a.service: Consumed 12.841s CPU time. 2024-02-05T06:58:56.312 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:58:56 smithi167 systemd[1]: Stopping Ceph node-exporter.b for 9801e956-c3ef-11ee-95b6-87774f69a715... 2024-02-05T06:58:56.619 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:58:56 smithi167 podman[154107]: 2024-02-05 06:58:56.309132037 +0000 UTC m=+0.111273279 container died 201f32f85bdf13c94d1df168b9acca0e666c49ab8286b26572587a9410b90f2b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-05T06:58:56.619 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:58:56 smithi167 podman[154107]: 2024-02-05 06:58:56.615317112 +0000 UTC m=+0.417458355 container cleanup 201f32f85bdf13c94d1df168b9acca0e666c49ab8286b26572587a9410b90f2b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-05T06:58:56.981 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:58:56 smithi167 bash[154107]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b 2024-02-05T06:58:56.981 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:58:56 smithi167 podman[154121]: 2024-02-05 06:58:56.791584633 +0000 UTC m=+0.486540705 container remove 201f32f85bdf13c94d1df168b9acca0e666c49ab8286b26572587a9410b90f2b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-9801e956-c3ef-11ee-95b6-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-05T06:58:56.981 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:58:56 smithi167 systemd[1]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-02-05T06:58:57.473 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:58:57 smithi167 systemd[1]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-05T06:58:57.473 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:58:57 smithi167 systemd[1]: Stopped Ceph node-exporter.b for 9801e956-c3ef-11ee-95b6-87774f69a715. 2024-02-05T06:58:57.474 INFO:journalctl@ceph.node-exporter.b.smithi167.stdout:Feb 05 06:58:57 smithi167 systemd[1]: ceph-9801e956-c3ef-11ee-95b6-87774f69a715@node-exporter.b.service: Consumed 4.421s CPU time. 2024-02-05T06:58:57.747 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-05T06:58:57.782 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-05T06:58:57.813 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-05T06:58:57.815 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/crash to /home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546690/remote/smithi028/crash 2024-02-05T06:58:57.815 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/crash -- . 2024-02-05T06:58:57.880 INFO:teuthology.orchestra.run.smithi028.stderr:tar: /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/crash: Cannot open: No such file or directory 2024-02-05T06:58:57.880 INFO:teuthology.orchestra.run.smithi028.stderr:tar: Error is not recoverable: exiting now 2024-02-05T06:58:57.883 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/crash to /home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546690/remote/smithi167/crash 2024-02-05T06:58:57.884 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar c -f - -C /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/crash -- . 2024-02-05T06:58:57.925 INFO:teuthology.orchestra.run.smithi167.stderr:tar: /var/lib/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/crash: Cannot open: No such file or directory 2024-02-05T06:58:57.925 INFO:teuthology.orchestra.run.smithi167.stderr:tar: Error is not recoverable: exiting now 2024-02-05T06:58:57.926 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-05T06:58:57.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-02-05T06:58:58.009 INFO:teuthology.orchestra.run.smithi028.stdout:2024-02-05T06:40:00.004239+0000 mon.a (mon.0) 1087 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-02-05T06:58:58.010 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-02-05T06:58:58.010 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[SEC\]' /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-02-05T06:58:58.039 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[ERR\]' /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-02-05T06:58:58.110 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[WRN\]' /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-02-05T06:58:58.180 INFO:teuthology.orchestra.run.smithi028.stdout:2024-02-05T06:40:00.004239+0000 mon.a (mon.0) 1087 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-02-05T06:58:58.180 INFO:tasks.cephadm:Compressing logs... 2024-02-05T06:58:58.180 DEBUG:teuthology.orchestra.run.smithi028:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-05T06:58:58.224 DEBUG:teuthology.orchestra.run.smithi167:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-05T06:58:58.279 INFO:teuthology.orchestra.run.smithi028.stderr:find: ‘/var/log/rbd-target-api’: No such file or directorygzip 2024-02-05T06:58:58.279 INFO:teuthology.orchestra.run.smithi028.stderr: -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-05T06:58:58.287 INFO:teuthology.orchestra.run.smithi167.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-05T06:58:58.287 INFO:teuthology.orchestra.run.smithi167.stderr:: No such file or directory 2024-02-05T06:58:58.289 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-mon.a.log 2024-02-05T06:58:58.289 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph.audit.log 2024-02-05T06:58:58.289 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-client.rgw.foo.a.log 2024-02-05T06:58:58.289 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-osd.1.log 2024-02-05T06:58:58.291 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-osd.3.log 2024-02-05T06:58:58.291 INFO:teuthology.orchestra.run.smithi028.stderr: 57.7% -- replaced with /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-02-05T06:58:58.291 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-mgr.y.log 2024-02-05T06:58:58.291 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-osd.0.log 2024-02-05T06:58:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph.cephadm.log 2024-02-05T06:58:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph.log 2024-02-05T06:58:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-osd.2.log 2024-02-05T06:58:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-volume.log 2024-02-05T06:58:58.292 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-mon.c.log 2024-02-05T06:58:58.301 INFO:teuthology.orchestra.run.smithi167.stderr:gzip -5 --verbose -- /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-osd.6.log 2024-02-05T06:58:58.301 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph.audit.log 2024-02-05T06:58:58.301 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-osd.4.log 2024-02-05T06:58:58.301 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-osd.7.log 2024-02-05T06:58:58.303 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-osd.5.log 2024-02-05T06:58:58.303 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-mgr.x.log 2024-02-05T06:58:58.303 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/tcmu-runner.log 2024-02-05T06:58:58.303 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph.cephadm.log 2024-02-05T06:58:58.303 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph.log 2024-02-05T06:58:58.303 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-volume.log 2024-02-05T06:58:58.304 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-mon.b.log 2024-02-05T06:58:58.304 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-volume.log: /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-mon.c.log: 91.5% -- replaced with /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph.log.gz 2024-02-05T06:58:58.399 INFO:teuthology.orchestra.run.smithi167.stderr:/var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-volume.log: /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-mon.b.log: 81.6% -- replaced with /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph.cephadm.log.gz 2024-02-05T06:58:58.410 INFO:teuthology.orchestra.run.smithi028.stderr: 91.1% -- replaced with /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-volume.log.gz 2024-02-05T06:58:58.412 INFO:teuthology.orchestra.run.smithi167.stderr: 67.6% -- replaced with /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/tcmu-runner.log.gz 2024-02-05T06:58:58.469 INFO:teuthology.orchestra.run.smithi028.stderr: 88.9% -- replaced with /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph.cephadm.log.gz 2024-02-05T06:58:58.506 INFO:teuthology.orchestra.run.smithi028.stderr: 93.9% -- replaced with /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph.audit.log.gz 2024-02-05T06:58:58.520 INFO:teuthology.orchestra.run.smithi167.stderr: 92.0% -- replaced with /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-mgr.x.log.gz 2024-02-05T06:58:58.527 INFO:teuthology.orchestra.run.smithi167.stderr: 91.2% -- replaced with /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-volume.log.gz 2024-02-05T06:58:58.527 INFO:teuthology.orchestra.run.smithi167.stderr: 84.9% -- replaced with /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph.log.gz 2024-02-05T06:58:58.660 INFO:teuthology.orchestra.run.smithi167.stderr: 89.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-05T06:58:58.661 INFO:teuthology.orchestra.run.smithi167.stderr: 90.0% -- replaced with /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph.audit.log.gz 2024-02-05T06:58:58.925 INFO:teuthology.orchestra.run.smithi028.stderr: 95.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-05T06:59:03.702 INFO:teuthology.orchestra.run.smithi028.stderr: 89.0% -- replaced with /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-mgr.y.log.gz 2024-02-05T06:59:06.123 INFO:teuthology.orchestra.run.smithi167.stderr: 90.7% -- replaced with /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-mon.b.log.gz 2024-02-05T06:59:06.877 INFO:teuthology.orchestra.run.smithi028.stderr: 90.7% -- replaced with /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-mon.c.log.gz 2024-02-05T06:59:15.410 INFO:teuthology.orchestra.run.smithi028.stderr: 90.2% -- replaced with /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-mon.a.log.gz 2024-02-05T07:00:25.760 INFO:teuthology.orchestra.run.smithi028.stderr: 92.8% -- replaced with /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-osd.2.log.gz 2024-02-05T07:01:38.442 INFO:teuthology.orchestra.run.smithi167.stderr: 93.4% -- replaced with /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-osd.6.log.gz 2024-02-05T07:01:43.145 INFO:teuthology.orchestra.run.smithi167.stderr: 93.7% -- replaced with /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-osd.4.log.gz 2024-02-05T07:03:18.531 INFO:teuthology.orchestra.run.smithi167.stderr: 93.8% -- replaced with /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-osd.5.log.gz 2024-02-05T07:03:21.797 INFO:teuthology.orchestra.run.smithi167.stderr: 93.8% -- replaced with /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-osd.7.log.gz 2024-02-05T07:03:21.799 INFO:teuthology.orchestra.run.smithi167.stderr: 2024-02-05T07:03:21.799 INFO:teuthology.orchestra.run.smithi167.stderr:real 4m23.562s 2024-02-05T07:03:21.799 INFO:teuthology.orchestra.run.smithi167.stderr:user 4m27.758s 2024-02-05T07:03:21.799 INFO:teuthology.orchestra.run.smithi167.stderr:sys 0m11.811s 2024-02-05T07:04:41.915 INFO:teuthology.orchestra.run.smithi028.stderr: 92.4% -- replaced with /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-osd.0.log.gz 2024-02-05T07:08:14.804 INFO:teuthology.orchestra.run.smithi028.stderr: 93.0% -- replaced with /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-osd.1.log.gz 2024-02-05T07:08:31.674 INFO:teuthology.orchestra.run.smithi028.stderr: 92.9% -- replaced with /var/log/ceph/9801e956-c3ef-11ee-95b6-87774f69a715/ceph-osd.3.log.gz 2024-02-05T07:08:31.676 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-02-05T07:08:31.676 INFO:teuthology.orchestra.run.smithi028.stderr:real 9m33.443s 2024-02-05T07:08:31.676 INFO:teuthology.orchestra.run.smithi028.stderr:user 7m10.816s 2024-02-05T07:08:31.676 INFO:teuthology.orchestra.run.smithi028.stderr:sys 0m17.740s 2024-02-05T07:08:31.677 INFO:tasks.cephadm:Archiving logs... 2024-02-05T07:08:31.678 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546690/remote/smithi028/log 2024-02-05T07:08:31.679 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-05T07:09:37.838 DEBUG:teuthology.misc:Transferring archived files from smithi167:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546690/remote/smithi167/log 2024-02-05T07:09:37.840 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-05T07:10:18.656 INFO:tasks.cephadm:Removing cluster... 2024-02-05T07:10:18.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 --force 2024-02-05T07:10:18.874 INFO:teuthology.orchestra.run.smithi028.stdout:Deleting cluster with fsid: 9801e956-c3ef-11ee-95b6-87774f69a715 2024-02-05T07:10:19.644 DEBUG:teuthology.orchestra.run.smithi167:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 9801e956-c3ef-11ee-95b6-87774f69a715 --force 2024-02-05T07:10:20.140 INFO:teuthology.orchestra.run.smithi167.stdout:Deleting cluster with fsid: 9801e956-c3ef-11ee-95b6-87774f69a715 2024-02-05T07:10:20.750 INFO:tasks.cephadm:Removing cephadm ... 2024-02-05T07:10:20.751 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-05T07:10:20.787 DEBUG:teuthology.orchestra.run.smithi167:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-05T07:10:20.812 INFO:tasks.cephadm:Teardown complete 2024-02-05T07:10:20.813 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-05T07:10:20.822 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-05T07:10:20.823 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-05T07:10:20.829 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-05T07:10:20.980 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-05T07:10:20.992 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-05T07:10:20.992 DEBUG:teuthology.orchestra.run.smithi028:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-05T07:10:20.995 DEBUG:teuthology.orchestra.run.smithi167:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-05T07:10:21.023 INFO:teuthology.orchestra.run.smithi167.stderr:bash: line 1: ntpq: command not found 2024-02-05T07:10:21.026 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-02-05T07:10:21.063 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-05T07:10:21.063 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-02-05T07:10:21.063 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 13 +832us[ +634us] +/- 92ms 2024-02-05T07:10:21.063 INFO:teuthology.orchestra.run.smithi028.stdout:^* hv02.front.sepia.ceph.com 3 6 377 12 -1351us[-1550us] +/- 49ms 2024-02-05T07:10:21.063 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 12 +1098us[ +900us] +/- 74ms 2024-02-05T07:10:21.063 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T07:10:21.067 INFO:teuthology.orchestra.run.smithi167.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-05T07:10:21.067 INFO:teuthology.orchestra.run.smithi167.stdout:=============================================================================== 2024-02-05T07:10:21.067 INFO:teuthology.orchestra.run.smithi167.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 11 +889us[ +889us] +/- 92ms 2024-02-05T07:10:21.067 INFO:teuthology.orchestra.run.smithi167.stdout:^* hv02.front.sepia.ceph.com 3 6 377 13 -1243us[-1418us] +/- 49ms 2024-02-05T07:10:21.067 INFO:teuthology.orchestra.run.smithi167.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 15 +1161us[ +986us] +/- 73ms 2024-02-05T07:10:21.067 INFO:teuthology.orchestra.run.smithi167.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-05T07:10:21.071 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-05T07:10:21.082 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-05T07:10:21.083 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-05T07:10:21.117 DEBUG:teuthology.orchestra.run.smithi028:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-05T07:10:21.209 DEBUG:teuthology.orchestra.run.smithi167:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-05T07:10:21.296 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-05T07:10:21.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:10:21.329 DEBUG:teuthology.orchestra.run.smithi167:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-05T07:10:21.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:10:21.357 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-05T07:10:21.374 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-05T07:10:21.449 INFO:teuthology.task.internal:Duration was 2983.491752 seconds 2024-02-05T07:10:21.449 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-05T07:10:21.493 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-05T07:10:21.493 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-05T07:10:21.496 DEBUG:teuthology.orchestra.run.smithi167:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-05T07:10:21.555 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-05T07:10:21.572 INFO:teuthology.orchestra.run.smithi167.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-05T07:10:22.043 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-05T07:10:22.043 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi028.front.sepia.ceph.com 2024-02-05T07:10:22.043 DEBUG:teuthology.orchestra.run.smithi028:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-05T07:10:22.066 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi167.front.sepia.ceph.com 2024-02-05T07:10:22.067 DEBUG:teuthology.orchestra.run.smithi167:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-05T07:10:22.100 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-05T07:10:22.100 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-05T07:10:22.110 DEBUG:teuthology.orchestra.run.smithi167:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-05T07:10:22.171 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-05T07:10:22.171 DEBUG:teuthology.orchestra.run.smithi028:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-05T07:10:22.180 DEBUG:teuthology.orchestra.run.smithi167:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-05T07:10:23.595 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-05T07:10:23.606 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-05T07:10:23.606 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-05T07:10:23.654 DEBUG:teuthology.orchestra.run.smithi167:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-05T07:10:23.685 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-05T07:10:23.698 DEBUG:teuthology.orchestra.run.smithi028:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-05T07:10:23.700 DEBUG:teuthology.orchestra.run.smithi167:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-05T07:10:23.732 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = core 2024-02-05T07:10:23.757 INFO:teuthology.orchestra.run.smithi167.stdout:kernel.core_pattern = core 2024-02-05T07:10:23.785 DEBUG:teuthology.orchestra.run.smithi028:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-05T07:10:23.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:10:23.825 DEBUG:teuthology.orchestra.run.smithi167:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-05T07:10:23.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-05T07:10:23.839 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-05T07:10:23.849 INFO:teuthology.task.internal:Transferring archived files... 2024-02-05T07:10:23.849 DEBUG:teuthology.misc:Transferring archived files from smithi028:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546690/remote/smithi028 2024-02-05T07:10:23.850 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-05T07:10:23.935 DEBUG:teuthology.misc:Transferring archived files from smithi167:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546690/remote/smithi167 2024-02-05T07:10:23.935 DEBUG:teuthology.orchestra.run.smithi167:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-05T07:10:23.976 INFO:teuthology.task.internal:Removing archive directory... 2024-02-05T07:10:23.976 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-05T07:10:23.978 DEBUG:teuthology.orchestra.run.smithi167:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-05T07:10:24.025 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-05T07:10:24.037 INFO:teuthology.task.internal:Not uploading archives. 2024-02-05T07:10:24.037 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-05T07:10:24.056 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-05T07:10:24.056 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-05T07:10:24.059 DEBUG:teuthology.orchestra.run.smithi167:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-05T07:10:24.070 INFO:teuthology.orchestra.run.smithi028.stdout: 265293 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 5 07:10 /home/ubuntu/cephtest 2024-02-05T07:10:24.079 INFO:teuthology.orchestra.run.smithi167.stdout: 265293 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 5 07:10 /home/ubuntu/cephtest 2024-02-05T07:10:24.080 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-05T07:10:24.142 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-05T07:10:24.175 INFO:teuthology.nuke:Checking targets against current locks 2024-02-05T07:10:24.206 DEBUG:teuthology.nuke:shortname: smithi028 2024-02-05T07:10:24.207 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-05T07:10:24.229 DEBUG:teuthology.nuke:shortname: smithi167 2024-02-05T07:10:24.230 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-05T07:10:24.234 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546690', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-05 06:11:36.392400', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBCA1FTL+IwOoJywhuWQEDUqUgZopUvLNHlNJXrX6CVXKNNh0+YvrNKCMk86c2yo6jBhtiVkzZ9WkUsbEM3HDBE='} 2024-02-05T07:10:24.249 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi167.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-02-05_05:57:01-orch:cephadm-wip-guits-testing-3-2024-02-04-2254-distro-default-smithi/7546690', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-05 06:11:36.394302', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMWP3yae/jbJ9gmk1hgW2izKA6ESL/12AEm5YAxTgVzCcWcHh690xyocMv//tt4sJMuVJXmzPpahdEBJiqLXN7c='} 2024-02-05T07:10:24.282 INFO:teuthology.orchestra.console.smithi028:Power off 2024-02-05T07:10:24.282 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-05T07:10:24.295 INFO:teuthology.orchestra.console.smithi167:Power off 2024-02-05T07:10:24.295 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-05T07:10:24.305 DEBUG:teuthology.orchestra.console.smithi028:power off output: Chassis Power Control: Down/Off 2024-02-05T07:10:24.306 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T07:10:24.318 DEBUG:teuthology.orchestra.console.smithi167:power off output: Chassis Power Control: Down/Off 2024-02-05T07:10:24.319 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T07:10:24.329 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2024-02-05T07:10:24.362 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is on 2024-02-05T07:10:28.330 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T07:10:28.363 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T07:10:28.440 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2024-02-05T07:10:28.474 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is on 2024-02-05T07:10:32.445 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T07:10:32.475 DEBUG:teuthology.orchestra.console.smithi167:pexpect command: ipmitool -H smithi167.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-05T07:10:32.556 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is off 2024-02-05T07:10:32.583 DEBUG:teuthology.orchestra.console.smithi167:check power output: Chassis Power is off 2024-02-05T07:10:32.658 INFO:teuthology.orchestra.console.smithi028:Power off completed 2024-02-05T07:10:32.684 INFO:teuthology.orchestra.console.smithi167:Power off completed 2024-02-05T07:10:32.788 INFO:teuthology.run:Summary data: description: orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} duration: 2983.491751909256 failure_reason: '"2024-02-05T06:40:00.004239+0000 mon.a (mon.0) 1087 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in cluster log' flavor: default owner: scheduled_gabrioux@teuthology success: false 2024-02-05T07:10:32.789 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-05T07:10:32.879 INFO:teuthology.run:FAIL